Next we will look at
the problem of web spam.
So the problem actually became very,
important with the rise of
the web search engines.
So let me first define what
do I mean web spam, right?
So spamming, it's really any
deliberate activity to boost the score
of a webpage or the webpage position
in the search engine results page.
All right?
And this means that basically what, what,
the goal is to kind of boost
the value of a given page.
so,spam, in some sense, are webpages
that are the result of spamming.
And actually, there is a whole
industry of people, in some sense,
that are trying to do this.
And this is what is called a search
engine optimization, right?
And there is a kind of grey line,
and these people might disagree.
But basically, the idea is,
how do we boost an importance of a,
of a given page on the web?
And make, make sure it appears,
let's say, on the first page of Google or
Bing results.
And it has been estimated that
around 10 to 15% of webpages on
the web are actually spam.
So how did spam web spam really work?
So early search engines
operated the following way.
They would go and crawl the web.
So basically they would go and
collect the web pages.
They would then index the web
pages by the words they contained.
And then whenever somebody searched,
they would basically look at
the given map page, see how much does it
mention the words in the search query, and
then it would rank the search results is
based on the number of times they
mention a given, search criteria.
So, what this means is that basically
the idea for the initial ranking was
to attempt toward the pages by matching
a a search query by importance.
Which means that basically
the search engines would,
would consider the number of times query
words appeared on the webpage, and
maybe also the prominence
of the word position,
whether that word was in the title or
in the header of a webpage.
So now that also is very, allows us to
nicely, nicely spam and exploit this idea.
So imagine the following.
Imagine that when peoples began using
the search engines for, on the web.
Those that, with the commercial interests.
Maybe people that are wanting
to sell something.
They want to exploit
the search engines and
make sure that people go and
visit their own pages.
Right?
So, the idea is kind of we want our
page to appear as high, as high on
the web search ranking as possible,
regardless of whether that is, that page
is really irrelevant to our given topic or
not, because driving traffic
to our website is good.
So lets look at the example where we
have our t-shirt reseller who wants to
appear high on the,
whenever people search about movies.
Maybe these t-shirt reseller reselling
movie related t-shirts, so he wants to
be as high as possible whenever
somebody searches for the word movies.
Right.
And the technique to achieving this,
boost in the relevance or
importance of our web page,
this is really what we mean by spam.
So here is how, how the first,
spammers did this.
This is called term spam.
So the idea is the following.
Right?
The idea is that for our t-shirt seller,
he creates the web page, and
then, wants to appear this web page
to really seem to be about movies.
So what the web seller can do,
is he can take the word movie, and
insert the word mo, movie,
1,000 times in the webpage.
And usually how this was done in
the early days on the web, is that you,
you would have a webpage where on
the top was the legitimate text, and
at the bottom was a huge list of words and
the ca,
the text color of these words was the same
as the color of the web background.
So kind of they wouldn't
bother the user who,
who comes to the web page, but the web
search engine would see these words and
would think that this particular web
page is all about movies, all right?
And now of course when somebody
will see and run a query on,
on movies, this web page would
appear to be all about movies, and
the person would go and
get it ranked higher.
So, this and similar techniques to
this one, are called term spam.
Let's look at how Google came up with
the solution to combat term spam.
So the basic idea, and
the basic intuition here is that,
rather than believing what
the page says about itself,
let's believe other people,
what they say about the page.
So in our context, this means that
we want to look at the anchor text.
So this is the blue text
underlying the hyperlink.
What does that text say about
the page that it is linked to?
So the, in some sense the idea is that
we are able to find web pages and, and
surface them out even if even for
queries for
the words of it that the web page itself
doesn't even mention, but other web
pages on the web may mention that word
when referring to the target page.
And of course, now we already know how
we can use PageRank, so we can think of
PageRank as a tool to measure the
importance of web pages on the web through
this idea of the anchor text, and the
question is why does this method works?
All right, in our hypothetical
t-shirt seller example, basically the,
we can combat him in a sense that now,
say, if it was very easy for
the t-shirt to say, to put the word
movies on the web page many times, and
say hey, I'm really about movies and
not about t-shirts.
But from the t-shirt seller's perspective,
it is very hard to convince other pages
on the web to link now to our teller's,
t-shirt seller's webpage, and
say that this webpage is about movies.
Of course, and
this is now where we are basically
coming to the notion of web spam.
Where the idea is that what a web
seller can do now, they can go and
create 1,000 web pages, that each link
to his, t-shirt seller web page and
each of these thousand web pages use
the word movie in, in the anchor text.
And this way, for example, now we
are having the notion of web spam where
the idea is that these web
pages have are creating false
links to our t-shirt seller web page, in
this way boost the importance of the page.
Of course, one thing is now that, for
example, these pages have no so they will
get very little web page page rank score,
and this page may still be linked low.
So the, the, the web shirt,
t-shirt seller person, it's very hard for
them, for them, for example to beat
a truly movie related webpage like IMDB.
Of course even this scheme can be beaten,
for example,
if there are big coordinated efforts
where people put a given anchor text and
link to a given target page, then what can
happen is that this given target page can
be ranked very high,
even though it doesn't mention a word.
Here is an example about
a coordinated effort,
about a given query,
in about a given target page.