1
00:00:00,330 --> 00:00:03,650
Next we will look at
the problem of web spam.

2
00:00:03,650 --> 00:00:07,760
So the problem actually became very,

3
00:00:07,760 --> 00:00:11,020
important with the rise of
the web search engines.

4
00:00:11,020 --> 00:00:13,780
So let me first define what
do I mean web spam, right?

5
00:00:13,780 --> 00:00:19,050
So spamming, it's really any
deliberate activity to boost the score

6
00:00:19,050 --> 00:00:23,695
of a webpage or the webpage position
in the search engine results page.

7
00:00:23,695 --> 00:00:25,150
All right?

8
00:00:25,150 --> 00:00:27,820
And this means that basically what, what,

9
00:00:27,820 --> 00:00:30,916
the goal is to kind of boost
the value of a given page.

10
00:00:30,916 --> 00:00:35,380
so,spam, in some sense, are webpages
that are the result of spamming.

11
00:00:35,380 --> 00:00:38,630
And actually, there is a whole
industry of people, in some sense,

12
00:00:38,630 --> 00:00:39,780
that are trying to do this.

13
00:00:39,780 --> 00:00:42,910
And this is what is called a search
engine optimization, right?

14
00:00:42,910 --> 00:00:46,140
And there is a kind of grey line,
and these people might disagree.

15
00:00:46,140 --> 00:00:49,370
But basically, the idea is,
how do we boost an importance of a,

16
00:00:49,370 --> 00:00:50,760
of a given page on the web?

17
00:00:50,760 --> 00:00:54,040
And make, make sure it appears,
let's say, on the first page of Google or

18
00:00:54,040 --> 00:00:54,655
Bing results.

19
00:00:54,655 --> 00:00:58,289
And it has been estimated that
around 10 to 15% of webpages on

20
00:00:58,289 --> 00:01:00,690
the web are actually spam.

21
00:01:00,690 --> 00:01:04,080
So how did spam web spam really work?

22
00:01:04,080 --> 00:01:07,710
So early search engines
operated the following way.

23
00:01:07,710 --> 00:01:09,200
They would go and crawl the web.

24
00:01:09,200 --> 00:01:11,970
So basically they would go and
collect the web pages.

25
00:01:11,970 --> 00:01:15,360
They would then index the web
pages by the words they contained.

26
00:01:15,360 --> 00:01:18,180
And then whenever somebody searched,
they would basically look at

27
00:01:18,180 --> 00:01:23,160
the given map page, see how much does it
mention the words in the search query, and

28
00:01:23,160 --> 00:01:25,860
then it would rank the search results is

29
00:01:25,860 --> 00:01:30,560
based on the number of times they
mention a given, search criteria.

30
00:01:30,560 --> 00:01:35,710
So, what this means is that basically
the idea for the initial ranking was

31
00:01:35,710 --> 00:01:41,070
to attempt toward the pages by matching
a a search query by importance.

32
00:01:41,070 --> 00:01:43,670
Which means that basically
the search engines would,

33
00:01:43,670 --> 00:01:47,760
would consider the number of times query
words appeared on the webpage, and

34
00:01:47,760 --> 00:01:50,870
maybe also the prominence
of the word position,

35
00:01:50,870 --> 00:01:54,830
whether that word was in the title or
in the header of a webpage.

36
00:01:54,830 --> 00:02:01,900
So now that also is very, allows us to
nicely, nicely spam and exploit this idea.

37
00:02:01,900 --> 00:02:03,590
So imagine the following.

38
00:02:03,590 --> 00:02:08,990
Imagine that when peoples began using
the search engines for, on the web.

39
00:02:08,990 --> 00:02:11,190
Those that, with the commercial interests.

40
00:02:11,190 --> 00:02:13,920
Maybe people that are wanting
to sell something.

41
00:02:13,920 --> 00:02:15,700
They want to exploit
the search engines and

42
00:02:15,700 --> 00:02:19,560
make sure that people go and
visit their own pages.

43
00:02:19,560 --> 00:02:22,390
Right?
So, the idea is kind of we want our

44
00:02:22,390 --> 00:02:26,720
page to appear as high, as high on
the web search ranking as possible,

45
00:02:26,720 --> 00:02:30,350
regardless of whether that is, that page
is really irrelevant to our given topic or

46
00:02:30,350 --> 00:02:33,020
not, because driving traffic
to our website is good.

47
00:02:33,020 --> 00:02:37,900
So lets look at the example where we
have our t-shirt reseller who wants to

48
00:02:37,900 --> 00:02:41,316
appear high on the,
whenever people search about movies.

49
00:02:41,316 --> 00:02:45,710
Maybe these t-shirt reseller reselling
movie related t-shirts, so he wants to

50
00:02:45,710 --> 00:02:50,380
be as high as possible whenever
somebody searches for the word movies.

51
00:02:50,380 --> 00:02:52,050
Right.
And the technique to achieving this,

52
00:02:53,180 --> 00:02:54,260
boost in the relevance or

53
00:02:54,260 --> 00:02:57,700
importance of our web page,
this is really what we mean by spam.

54
00:02:57,700 --> 00:03:01,440
So here is how, how the first,
spammers did this.

55
00:03:01,440 --> 00:03:02,650
This is called term spam.

56
00:03:02,650 --> 00:03:03,950
So the idea is the following.

57
00:03:03,950 --> 00:03:04,450
Right?

58
00:03:04,450 --> 00:03:08,590
The idea is that for our t-shirt seller,
he creates the web page, and

59
00:03:08,590 --> 00:03:13,260
then, wants to appear this web page
to really seem to be about movies.

60
00:03:13,260 --> 00:03:16,580
So what the web seller can do,
is he can take the word movie, and

61
00:03:16,580 --> 00:03:21,080
insert the word mo, movie,
1,000 times in the webpage.

62
00:03:21,080 --> 00:03:23,720
And usually how this was done in
the early days on the web, is that you,

63
00:03:23,720 --> 00:03:27,350
you would have a webpage where on
the top was the legitimate text, and

64
00:03:27,350 --> 00:03:31,020
at the bottom was a huge list of words and
the ca,

65
00:03:31,020 --> 00:03:34,580
the text color of these words was the same
as the color of the web background.

66
00:03:34,580 --> 00:03:36,405
So kind of they wouldn't
bother the user who,

67
00:03:36,405 --> 00:03:40,690
who comes to the web page, but the web
search engine would see these words and

68
00:03:40,690 --> 00:03:44,100
would think that this particular web
page is all about movies, all right?

69
00:03:45,200 --> 00:03:50,320
And now of course when somebody
will see and run a query on,

70
00:03:50,320 --> 00:03:54,230
on movies, this web page would
appear to be all about movies, and

71
00:03:54,230 --> 00:03:57,940
the person would go and
get it ranked higher.

72
00:03:57,940 --> 00:04:01,790
So, this and similar techniques to
this one, are called term spam.

73
00:04:03,290 --> 00:04:07,510
Let's look at how Google came up with
the solution to combat term spam.

74
00:04:07,510 --> 00:04:10,920
So the basic idea, and
the basic intuition here is that,

75
00:04:10,920 --> 00:04:14,290
rather than believing what
the page says about itself,

76
00:04:14,290 --> 00:04:17,440
let's believe other people,
what they say about the page.

77
00:04:17,440 --> 00:04:21,500
So in our context, this means that
we want to look at the anchor text.

78
00:04:21,500 --> 00:04:24,800
So this is the blue text
underlying the hyperlink.

79
00:04:24,800 --> 00:04:29,180
What does that text say about
the page that it is linked to?

80
00:04:29,180 --> 00:04:34,490
So the, in some sense the idea is that
we are able to find web pages and, and

81
00:04:34,490 --> 00:04:37,310
surface them out even if even for
queries for

82
00:04:37,310 --> 00:04:40,670
the words of it that the web page itself
doesn't even mention, but other web

83
00:04:40,670 --> 00:04:44,870
pages on the web may mention that word
when referring to the target page.

84
00:04:44,870 --> 00:04:48,530
And of course, now we already know how
we can use PageRank, so we can think of

85
00:04:48,530 --> 00:04:52,250
PageRank as a tool to measure the
importance of web pages on the web through

86
00:04:52,250 --> 00:04:57,680
this idea of the anchor text, and the
question is why does this method works?

87
00:04:57,680 --> 00:05:02,397
All right, in our hypothetical
t-shirt seller example, basically the,

88
00:05:02,397 --> 00:05:06,374
we can combat him in a sense that now,
say, if it was very easy for

89
00:05:06,374 --> 00:05:10,942
the t-shirt to say, to put the word
movies on the web page many times, and

90
00:05:10,942 --> 00:05:14,760
say hey, I'm really about movies and
not about t-shirts.

91
00:05:14,760 --> 00:05:17,060
But from the t-shirt seller's perspective,

92
00:05:17,060 --> 00:05:22,285
it is very hard to convince other pages
on the web to link now to our teller's,

93
00:05:22,285 --> 00:05:27,020
t-shirt seller's webpage, and
say that this webpage is about movies.

94
00:05:27,020 --> 00:05:27,790
Of course, and

95
00:05:27,790 --> 00:05:31,440
this is now where we are basically
coming to the notion of web spam.

96
00:05:31,440 --> 00:05:35,050
Where the idea is that what a web
seller can do now, they can go and

97
00:05:35,050 --> 00:05:41,080
create 1,000 web pages, that each link
to his, t-shirt seller web page and

98
00:05:41,080 --> 00:05:45,870
each of these thousand web pages use
the word movie in, in the anchor text.

99
00:05:45,870 --> 00:05:50,860
And this way, for example, now we
are having the notion of web spam where

100
00:05:50,860 --> 00:05:56,590
the idea is that these web
pages have are creating false

101
00:05:56,590 --> 00:06:01,380
links to our t-shirt seller web page, in
this way boost the importance of the page.

102
00:06:01,380 --> 00:06:06,772
Of course, one thing is now that, for
example, these pages have no so they will

103
00:06:06,772 --> 00:06:11,900
get very little web page page rank score,
and this page may still be linked low.

104
00:06:11,900 --> 00:06:18,200
So the, the, the web shirt,
t-shirt seller person, it's very hard for

105
00:06:18,200 --> 00:06:23,820
them, for them, for example to beat
a truly movie related webpage like IMDB.

106
00:06:23,820 --> 00:06:26,820
Of course even this scheme can be beaten,
for example,

107
00:06:26,820 --> 00:06:31,350
if there are big coordinated efforts
where people put a given anchor text and

108
00:06:31,350 --> 00:06:36,070
link to a given target page, then what can
happen is that this given target page can

109
00:06:36,070 --> 00:06:40,270
be ranked very high,
even though it doesn't mention a word.

110
00:06:40,270 --> 00:06:43,010
Here is an example about
a coordinated effort,

111
00:06:43,010 --> 00:06:46,760
about a given query,
in about a given target page.