1
00:00:07,745 --> 00:00:11,893
Another family of supervised learning
models that's related to linear

2
00:00:11,893 --> 00:00:15,701
classification models is
the Naive Bayes family of classifiers,

3
00:00:15,701 --> 00:00:20,325
which are based on simple probabilistic
models of how the data in each class might

4
00:00:20,325 --> 00:00:21,617
have been generated.

5
00:00:23,089 --> 00:00:26,785
Naive Bayes classifiers are called
naive because informally,

6
00:00:26,785 --> 00:00:31,715
they make the simplifying assumption that
each feature of an instance is independent

7
00:00:31,715 --> 00:00:33,850
of all the others, given the class.

8
00:00:35,290 --> 00:00:36,090
In practice, of course,

9
00:00:36,090 --> 00:00:40,320
this is not often the case,
features often are somewhat correlated.

10
00:00:40,320 --> 00:00:40,960
For example,

11
00:00:40,960 --> 00:00:46,290
in predicting whether a house is likely
to sell above the owner's asking price.

12
00:00:46,290 --> 00:00:50,450
Some features, such as the are of the
interior rooms are likely to be correlated

13
00:00:50,450 --> 00:00:54,950
with other features, such as the size of
the land that the house is built on or

14
00:00:54,950 --> 00:00:56,950
the number of bedrooms.

15
00:00:56,950 --> 00:00:59,840
And these features in turn might
be correlated with the location of

16
00:00:59,840 --> 00:01:00,820
the property, and so on.

17
00:01:02,880 --> 00:01:06,520
This naive simplifying assumption
means on the one hand,

18
00:01:06,520 --> 00:01:09,360
that learning a Naive Bayes
classifier is very fast.

19
00:01:09,360 --> 00:01:13,940
Because only simple per class
statistics need to be estimated for

20
00:01:13,940 --> 00:01:16,960
each feature and applied for
each feature independently.

21
00:01:18,320 --> 00:01:22,290
On the other hand, the penalty for
this efficiency is that the generalization

22
00:01:22,290 --> 00:01:26,900
performance of Naive Bayes Classifiers
can often be a bit worse than other

23
00:01:26,900 --> 00:01:31,080
more sophisticated methods, or
even linear models for classification.

24
00:01:32,220 --> 00:01:36,020
Even so, especially for
high dimensional data sets,

25
00:01:36,020 --> 00:01:40,450
Naive Bayes Classifiers can achieve
performance that's often competitive

26
00:01:40,450 --> 00:01:44,790
to other more sophisticated methods, like
support vector machines, for some tasks.

27
00:01:46,600 --> 00:01:50,480
There are three flavors of
Naive Bayes Classifier that are available

28
00:01:50,480 --> 00:01:52,097
in scikit learn.

29
00:01:52,097 --> 00:01:57,301
The Bernoulli Naive Bayes model uses
a set of binary occurrence features.

30
00:01:57,301 --> 00:02:00,450
When classifying texts document for
example,

31
00:02:00,450 --> 00:02:04,420
the Bernoulli Naive Bayes model is
quit handy because we could represent

32
00:02:05,680 --> 00:02:09,950
the presence or the absence of the given
word in the text with the binary feature.

33
00:02:11,050 --> 00:02:15,010
Of course this doesn't take into account
how often the word occurs in the text.

34
00:02:16,050 --> 00:02:20,670
So the Multinomial Naive Bayes model
uses a set of count base features

35
00:02:20,670 --> 00:02:25,090
each of which does account for how many
times a particular feature such as a word

36
00:02:25,090 --> 00:02:27,480
is observed in training
example like a document.

37
00:02:28,970 --> 00:02:31,900
In this lecture we won't have
time to cover the Bernoulli or

38
00:02:31,900 --> 00:02:34,710
Multinomial Naive Bayes models.

39
00:02:34,710 --> 00:02:38,770
However, those models are particularly
well suited to textual data,

40
00:02:38,770 --> 00:02:42,970
where each feature corresponds to
an observation for a particular word.

41
00:02:42,970 --> 00:02:46,440
And so you'll see Naive Bayes again,
including the Bernoulli and

42
00:02:46,440 --> 00:02:51,480
Multinomial models in more depth in the
text mining part of this specialization.

43
00:02:52,880 --> 00:02:57,150
This lecture will focus on
Gaussian Naive Bayes classifiers

44
00:02:57,150 --> 00:03:01,150
which assume features that
are continuous or real-valued.

45
00:03:01,150 --> 00:03:05,950
During training, the Gaussian Naive Bayes
Classifier estimates for each feature

46
00:03:05,950 --> 00:03:10,059
the mean and standard deviation of
the feature value for each class.

47
00:03:11,070 --> 00:03:14,780
For prediction, the classifier compares
the features of the example data point to

48
00:03:14,780 --> 00:03:18,740
be predicted with the feature
statistics for each class and

49
00:03:18,740 --> 00:03:21,550
selects the class that best
matches the data point.

50
00:03:23,190 --> 00:03:28,060
More specifically, the Gaussian Naive
Bayes Classifier assumes that the data for

51
00:03:28,060 --> 00:03:32,320
each class was generated by a simple
class specific Gaussian distribution.

52
00:03:33,350 --> 00:03:37,260
Predicting the class of a new data
point corresponds mathematically to

53
00:03:37,260 --> 00:03:41,740
estimating the probability that
each classes Gaussian distribution

54
00:03:41,740 --> 00:03:44,800
was most likely to have
generated the data point.

55
00:03:44,800 --> 00:03:48,370
Classifier then picks the class
that has the highest probability.

56
00:03:49,430 --> 00:03:52,210
Without going into
the mathematics involved,

57
00:03:52,210 --> 00:03:55,230
it can be shown that the decision
boundary between classes

58
00:03:55,230 --> 00:03:59,010
in the two class
Gaussian Naive Bayes Classifier.

59
00:03:59,010 --> 00:04:02,310
In general is a parabolic
curve between the classes.

60
00:04:03,642 --> 00:04:06,890
And in the special case where the variance
of these feature is the same for

61
00:04:06,890 --> 00:04:08,320
both classes.

62
00:04:08,320 --> 00:04:09,740
The decision boundary will be linear.

63
00:04:11,000 --> 00:04:14,870
Here's what that looks like, typically, on
a simple binary classification data set.

64
00:04:16,180 --> 00:04:19,720
The gray ellipses given idea of the shape
of the Gaussian distribution for

65
00:04:19,720 --> 00:04:22,030
each class,
as if we were looking down from above.

66
00:04:23,280 --> 00:04:25,850
You can see the centers of
the Gaussian's correspond

67
00:04:25,850 --> 00:04:28,609
to the mean value of each feature for
each class.

68
00:04:29,750 --> 00:04:30,420
More specifically,

69
00:04:30,420 --> 00:04:34,230
the gray ellipses show the contour
line of the Gaussian distribution for

70
00:04:34,230 --> 00:04:38,270
each class, that corresponds to about
two standard deviations from the mean.

71
00:04:40,220 --> 00:04:41,550
The line between the yellow and

72
00:04:41,550 --> 00:04:44,680
gray background areas represents
the decision boundary.

73
00:04:44,680 --> 00:04:47,820
And we can see that this
is indeed parabolic.

74
00:04:49,850 --> 00:04:52,740
To use the Gaussian Naive Bayes
classifier in Python,

75
00:04:53,880 --> 00:04:58,650
we just instantiate an instance
of the Gaussian NB class and call

76
00:04:58,650 --> 00:05:02,470
the fit method on the training data just
as we would with any other classifier.

77
00:05:03,980 --> 00:05:07,882
It's worth noting that the Naive Bayes
models are among a few classifiers in

78
00:05:07,882 --> 00:05:11,660
scikit learn that support
a method called partial fit,

79
00:05:12,840 --> 00:05:16,570
which can be used instead of fit to
train the classifier incrementally

80
00:05:16,570 --> 00:05:19,620
in case you're working with a huge
data set that doesn't fit into memory.

81
00:05:20,820 --> 00:05:24,750
More details on that are available in
the scikit learn documentation for

82
00:05:24,750 --> 00:05:25,430
Naive Bayes.

83
00:05:26,870 --> 00:05:30,960
For the Gaussian NB class there are no
special parameters to control the models

84
00:05:30,960 --> 00:05:31,510
complexity.

85
00:05:33,320 --> 00:05:37,600
Looking at one example in the notebook
from our synthetic two class dataset,

86
00:05:38,820 --> 00:05:42,450
we can see that, in fact, the Gaussian
Naive Bayes classifier achieves quite good

87
00:05:42,450 --> 00:05:45,060
performance on this simple
classification example.

88
00:05:45,060 --> 00:05:50,670
When the classes are no longer as
easily separable as with this second,

89
00:05:50,670 --> 00:05:53,480
more difficult binary example here.

90
00:05:53,480 --> 00:05:56,600
Like linear models,
Naive Bayes does not perform as well.

91
00:05:58,260 --> 00:06:01,600
On a real world example,
using the breast cancer data set,

92
00:06:01,600 --> 00:06:05,960
the Gaussian Naive Bayes Classifier also
does quite well, being quite competitive

93
00:06:05,960 --> 00:06:08,540
with other methods,
such as support vector classifiers.

94
00:06:10,730 --> 00:06:14,168
Typically, Gaussian Naive Bayes is
used for high-dimensional data.

95
00:06:14,168 --> 00:06:19,390
When each data instance has hundreds,
thousands or maybe even more features.

96
00:06:19,390 --> 00:06:23,985
And likewise the Bernoulli and Nultinomial
flavors of Naive Bayes are used for

97
00:06:23,985 --> 00:06:28,600
text classification where there are very
large number of distinct words is features

98
00:06:28,600 --> 00:06:31,760
and where the future vectors
are sparse because any given document

99
00:06:31,760 --> 00:06:34,400
uses only a small fraction
of the overall vocabulary.

100
00:06:35,780 --> 00:06:38,130
There's more in depth
material on the Bernoulli and

101
00:06:38,130 --> 00:06:41,930
Multinomial Naive Bayes Classifiers
in the text mining portion of

102
00:06:41,930 --> 00:06:42,820
this specialization.

103
00:06:45,070 --> 00:06:47,630
It can be shown that
Naive Bayes Classifiers are related

104
00:06:47,630 --> 00:06:50,910
mathematically to linear models,
so many of the pros and

105
00:06:50,910 --> 00:06:54,139
cons of linear models also
apply to Naive Bayes.

106
00:06:55,670 --> 00:07:00,200
On the positive side Naive Bayes
classifiers are fast to train and use for

107
00:07:00,200 --> 00:07:05,290
prediction and thus are well suitable to
high dimensional data including text.

108
00:07:05,290 --> 00:07:10,720
And the applications involving very large
data sets where efficiency is critical and

109
00:07:10,720 --> 00:07:13,960
computational costs rule out
other classification approaches.

110
00:07:15,610 --> 00:07:16,540
On the negative side,

111
00:07:16,540 --> 00:07:20,960
when the conditional independence
assumption about features doesn't hold.

112
00:07:20,960 --> 00:07:25,300
In other words, for a given class, there's
significant covariance among features,

113
00:07:25,300 --> 00:07:27,570
as is the case with many
real world datasets.

114
00:07:28,590 --> 00:07:32,000
Other more sophisticated classification
methods that can account for

115
00:07:32,000 --> 00:07:34,959
these dependencies are likely
to outperform Naive Bayes.

116
00:07:36,325 --> 00:07:38,730
And on a side note,
when getting confidence or

117
00:07:38,730 --> 00:07:42,010
probability estimates
associated with predictions,

118
00:07:42,010 --> 00:07:45,940
Naive Bayes classifiers produce
unreliable estimates, typically.

119
00:07:47,340 --> 00:07:52,239
Still, Naive Bayes Classifiers can
perform very competitively on some tasks,

120
00:07:52,239 --> 00:07:56,345
and are also often very useful as
baseline models against which more

121
00:07:56,345 --> 00:07:58,811
sophisticated models can be compared.