1
00:00:07,756 --> 00:00:11,102
When we began working with
supervised machine learning methods,

2
00:00:11,102 --> 00:00:14,341
we evaluated a classifier's
performance using its accuracy.

3
00:00:15,510 --> 00:00:16,870
Accuracy, as you might recall,

4
00:00:16,870 --> 00:00:20,490
is the fraction of samples that
were classified correctly.

5
00:00:20,490 --> 00:00:26,120
That is, where the classifier's predicted
label matched the correct or true label.

6
00:00:27,330 --> 00:00:30,960
We also evaluated a regression
model's performance using the default

7
00:00:30,960 --> 00:00:31,910
r squared metric.

8
00:00:33,340 --> 00:00:38,020
In this module, you'll learn why measures
like accuracy, which are simple and

9
00:00:38,020 --> 00:00:41,610
easy to understand, also have drawbacks.

10
00:00:41,610 --> 00:00:44,910
In that they don't give
a complete enough picture

11
00:00:44,910 --> 00:00:47,520
of a supervised learning
model's performance.

12
00:00:47,520 --> 00:00:51,350
And may not be the right metric for
measuring success in your application.

13
00:00:52,470 --> 00:00:57,650
So we're going to cover several additional
evaluation metrics beyond accuracy.

14
00:00:57,650 --> 00:01:01,600
We'll see how they're defined,
what the motivation is for using them, and

15
00:01:01,600 --> 00:01:06,050
how to use them scikit-learn to
get a better picture of how well

16
00:01:06,050 --> 00:01:08,735
A supervised model is
doing on a given data set.

17
00:01:08,735 --> 00:01:13,600
You'll also learn how about to choose
the right evaluation matrix for your

18
00:01:13,600 --> 00:01:18,300
application that can help you select the
best model or find the optimal parameters.

19
00:01:19,840 --> 00:01:20,670
So let's return for

20
00:01:20,670 --> 00:01:24,609
a moment to this workflow diagram that
we introduced earlier in the course.

21
00:01:25,610 --> 00:01:28,550
You see that evaluation
is a key part of this

22
00:01:28,550 --> 00:01:30,520
development cycle in
applied machine learning.

23
00:01:31,850 --> 00:01:36,020
Once a model is trained, the evaluation
step provides critical feedback

24
00:01:36,020 --> 00:01:39,200
on the trained model's
performance characteristics.

25
00:01:39,200 --> 00:01:41,850
Particularly those that might be
important for your application.

26
00:01:43,360 --> 00:01:47,700
The results of the evaluation step,
for example, might help you understand

27
00:01:47,700 --> 00:01:51,600
which data instances are being
classified or predicted incorrectly.

28
00:01:51,600 --> 00:01:55,400
Which might in turn suggest better
features or different kernel function or

29
00:01:55,400 --> 00:01:59,280
other refinements to your learning model
in the feature and model refinement phase.

30
00:02:00,950 --> 00:02:05,170
As we discussed earlier, the objective
function that's optimized during

31
00:02:05,170 --> 00:02:10,930
the training phase may be a different,
what's called a surrogate metric.

32
00:02:10,930 --> 00:02:15,440
That's easier to use in practice for
optimization purposes than what's used for

33
00:02:15,440 --> 00:02:17,360
the evaluation metric.

34
00:02:17,360 --> 00:02:21,950
For example, a commercial search engine
might use a ranking algorithm that is

35
00:02:21,950 --> 00:02:26,640
trained to recommend relevant web
pages that best match a query.

36
00:02:26,640 --> 00:02:31,482
In other words, trying to predict
a relevant label for a page.

37
00:02:31,482 --> 00:02:33,880
And that might be the objective
in the training phase.

38
00:02:33,880 --> 00:02:38,920
But there are many evaluation methods in
the evaluation phase that could be applied

39
00:02:38,920 --> 00:02:43,460
to measure aspects of that search engine's
performance using that ranking algorithm,

40
00:02:43,460 --> 00:02:47,290
that are important to the search
company's business, for example.

41
00:02:47,290 --> 00:02:50,500
Such as how many unique users
the system sees per day.

42
00:02:50,500 --> 00:02:53,840
Or how long the typical user
search session is and so on.

43
00:02:55,210 --> 00:02:58,340
So the evaluation measures
are the ones that in the end

44
00:02:58,340 --> 00:03:02,999
are used to select between different
trained models or settings.

45
00:03:04,260 --> 00:03:08,090
Actually commercial search
applications typically use a scorecard

46
00:03:08,090 --> 00:03:12,180
of multiple evaluation metrics to
make important business decisions.

47
00:03:12,180 --> 00:03:15,280
Or development decisions about
what models are chosen for use.

48
00:03:17,150 --> 00:03:19,760
So it's very important
to choose evaluation

49
00:03:19,760 --> 00:03:22,100
methods that match the goal
of your application.

50
00:03:23,270 --> 00:03:27,140
For predicting the correct digit
from a handwritten image, let's say,

51
00:03:27,140 --> 00:03:31,670
where each digit is equally likely,
then accuracy may be a sufficient metric.

52
00:03:32,770 --> 00:03:37,420
However, there are other possible aspects
of evaluation of model performance that

53
00:03:37,420 --> 00:03:40,310
are beyond average accuracy that
may be critical to measure.

54
00:03:40,310 --> 00:03:41,770
For example,

55
00:03:41,770 --> 00:03:46,210
in a health application that uses
a classifier to detect tumors in a medical

56
00:03:46,210 --> 00:03:50,030
image, we may want the classifier
to error on the side of caution.

57
00:03:50,030 --> 00:03:54,080
And flag anything that even has
a small chance of being cancerous.

58
00:03:54,080 --> 00:03:58,229
Even if it means sometimes incorrectly
classifying healthy tissue as diseased.

59
00:03:59,720 --> 00:04:02,220
In this case,
the classifier evaluation method

60
00:04:02,220 --> 00:04:05,430
would try to reduce what are called
false negative predictions.

61
00:04:06,550 --> 00:04:08,310
We'll look at this in more detail shortly.

62
00:04:09,900 --> 00:04:13,730
More generally, your application goal
might be based on very different metrics,

63
00:04:13,730 --> 00:04:18,310
such as user satisfaction,
amount of revenue to your page, or

64
00:04:18,310 --> 00:04:20,750
increase in patient survival rates.

65
00:04:21,790 --> 00:04:25,640
So in the end, you'll be selecting the
model or parameter settings that optimize

66
00:04:25,640 --> 00:04:29,770
those end evaluation metrics
that are important to your goal.

67
00:04:31,730 --> 00:04:32,949
So in this module,

68
00:04:32,949 --> 00:04:38,609
we'll be focusing first on the widely used
case of evaluating binary classification.

69
00:04:38,609 --> 00:04:40,433
And then we'll look at evaluation for

70
00:04:40,433 --> 00:04:43,970
the more general case of multi-class
evaluation as well as regression.

71
00:04:46,780 --> 00:04:49,920
Before we start defining and
using some evaluation metrics for

72
00:04:49,920 --> 00:04:54,250
binary classification, lets start by
looking at example of why just looking at

73
00:04:54,250 --> 00:04:58,870
accuracy may not be enough to gain a good
picture of what a classifier's doing.

74
00:05:00,570 --> 00:05:04,720
It'll also show us how knowing more about
the types of errors a learning algorithm

75
00:05:04,720 --> 00:05:08,270
makes can help us get a better picture
of a model's predictive performance.

76
00:05:09,280 --> 00:05:12,890
First, let's consider the case where
we have a binary classification task,

77
00:05:12,890 --> 00:05:16,440
where there are a lot of instances
labeled with the negative class.

78
00:05:16,440 --> 00:05:19,349
But only a few instances that
belong to the positive class.

79
00:05:20,480 --> 00:05:25,520
For example, we might see this scenario
in online search or recommender systems.

80
00:05:25,520 --> 00:05:29,240
Where the system has to predict whether or
not to display an advertisement or

81
00:05:29,240 --> 00:05:30,200
product suggestion.

82
00:05:31,280 --> 00:05:36,360
Or show a query suggestion or
item on a page that's likely to be

83
00:05:36,360 --> 00:05:40,550
relevant given a user's query and what
they clicked on in the past and so on.

84
00:05:40,550 --> 00:05:43,190
So those would be the positive examples.

85
00:05:43,190 --> 00:05:46,210
But of course there are many,
many irrelevant items

86
00:05:46,210 --> 00:05:49,790
that are in the negative class that
don't make sense to show a user.

87
00:05:49,790 --> 00:05:55,360
And so this is called
an imbalanced class scenario.

88
00:05:56,520 --> 00:06:00,270
Another example might be datasets
of credit card transactions.

89
00:06:00,270 --> 00:06:04,170
Where the vast majority of transactions
are classified as normal and not fraud

90
00:06:05,210 --> 00:06:09,500
with a small minority of transactions
that could be classified as fraudulent.

91
00:06:12,220 --> 00:06:16,520
These situations, which also apply to
multi-class classification problems,

92
00:06:16,520 --> 00:06:20,380
involve datasets that
have imbalanced classes.

93
00:06:20,380 --> 00:06:23,730
Imbalanced classes are very common
in machine learning scenarios, so

94
00:06:23,730 --> 00:06:25,800
it's important to understand
how to work with them.

95
00:06:27,850 --> 00:06:31,230
In particular, let's assume that
we have an e-commerce application.

96
00:06:31,230 --> 00:06:35,610
Where for every 1,000 randomly
sampled product items,

97
00:06:35,610 --> 00:06:40,639
one of them is relevant to a user's
need and the other 999 are not relevant.

98
00:06:42,000 --> 00:06:46,480
So recall that accuracy
computed over a set of

99
00:06:46,480 --> 00:06:50,958
instances is just the number of instances
where the classifier's label prediction

100
00:06:50,958 --> 00:06:53,960
was correct divided by
the total number of instances.

101
00:06:55,740 --> 00:07:01,101
Let's suppose you develop a classifier for
predicting relevant e-commerce items.

102
00:07:01,101 --> 00:07:04,623
And after you've finished the development,

103
00:07:04,623 --> 00:07:09,055
you measure its accuracy on
the test set to be 99.9%.

104
00:07:09,055 --> 00:07:11,960
At first, that might seem to
be amazingly good, right?

105
00:07:11,960 --> 00:07:13,730
That's incredibly close to perfect.

106
00:07:15,520 --> 00:07:19,110
But let's compare that
to a dummy classifier

107
00:07:19,110 --> 00:07:24,260
that always just predicts the most likely
class, namely, the not relevant class.

108
00:07:24,260 --> 00:07:27,670
In other words,
no matter what the actual instance is,

109
00:07:27,670 --> 00:07:31,320
the dummy classifier will always
predict that an item is not relevant.

110
00:07:32,550 --> 00:07:35,720
So if we have a test set
that has 1,000 items,

111
00:07:35,720 --> 00:07:40,390
on average 999 of them will
be not relevant anyway.

112
00:07:40,390 --> 00:07:46,292
So our dummy classifier will correctly
predict the not relevant label for

113
00:07:46,292 --> 00:07:48,493
all of those 999 items.

114
00:07:48,493 --> 00:07:54,811
And so the accuracy of the dummy
classifier is also going to be 99.9%.

115
00:07:54,811 --> 00:07:58,730
So in reality our own classifier's
performance isn't impressive at all.

116
00:07:58,730 --> 00:08:01,470
It's no better than just always predicting

117
00:08:01,470 --> 00:08:04,338
The majority class without
even looking at the data.

118
00:08:04,338 --> 00:08:08,641
Let's take a look at another example of
classification with imbalanced classes on

119
00:08:08,641 --> 00:08:10,490
a real dataset using our notebook.

120
00:08:11,810 --> 00:08:16,640
I'll start here using the digits dataset,
which has images of handwritten digits

121
00:08:16,640 --> 00:08:20,895
labeled with ten classes,
representing the digits zero through nine.

122
00:08:22,600 --> 00:08:27,063
As we can see by letting the dataset and
then computing the count of

123
00:08:27,063 --> 00:08:31,221
instances in each class,
using numpy's bin count method.

124
00:08:31,221 --> 00:08:34,260
There are roughly the same number
of instances in each class.

125
00:08:34,260 --> 00:08:36,660
So this dataset has balanced classes.

126
00:08:38,140 --> 00:08:40,740
However with this digits dataset,

127
00:08:40,740 --> 00:08:45,490
now what we're going to do is create a new
dataset with two imbalanced classes.

128
00:08:45,490 --> 00:08:50,804
By labelling all digits that are not
the digit 1 as the negative class

129
00:08:50,804 --> 00:08:56,316
with label 0, and digits that are 1
as the positive class, label 1.

130
00:08:56,316 --> 00:09:01,180
So what I've done here is dump the first
few entries from the original labels

131
00:09:01,180 --> 00:09:05,820
along with the new binary label, so
you can see the imbalance visually.

132
00:09:07,628 --> 00:09:12,998
Now when we use bincount, we can see that
there are about 1,600 negative examples,

133
00:09:12,998 --> 00:09:15,228
but only 182 positive examples.

134
00:09:15,228 --> 00:09:19,960
So indeed, we have a dataset
that is class imbalanced.

135
00:09:19,960 --> 00:09:24,613
Or as expected almost exactly
a nine to one ratio of negative to

136
00:09:24,613 --> 00:09:26,230
positive examples.

137
00:09:27,670 --> 00:09:31,347
Now let's create a train test
partition on this imbalance set.

138
00:09:31,347 --> 00:09:35,569
And then train a support vector
machine classifier with these binary

139
00:09:35,569 --> 00:09:38,780
labels using the radial
basis function as a kernel.

140
00:09:38,780 --> 00:09:42,397
We get the accuracy using
the score method, and

141
00:09:42,397 --> 00:09:45,140
we can see this is just over 90%.

142
00:09:45,140 --> 00:09:50,020
Again at first glance, 90% accuracy for
a classifier seems pretty good.

143
00:09:51,670 --> 00:09:54,600
However, now let's create
a Dummy Classifier

144
00:09:54,600 --> 00:09:59,099
that correctly reflect the class imbalance
to see if 90% really is that impressive.

145
00:10:00,765 --> 00:10:03,870
scikit-learn makes it easy
to create a dummy classifier

146
00:10:03,870 --> 00:10:07,480
just by using the DummyClassifier
class as shown here.

147
00:10:09,092 --> 00:10:10,603
Dummy classifiers, again,

148
00:10:10,603 --> 00:10:14,599
are called that because they don't even
look at the data to make a prediction.

149
00:10:14,599 --> 00:10:17,033
They simply use the strategy or

150
00:10:17,033 --> 00:10:22,810
rule of thumb that you instruct
them to use, when creating them.

151
00:10:22,810 --> 00:10:26,540
In fact, when you create the classifier,
you set the strategy argument to tell it

152
00:10:26,540 --> 00:10:28,823
what rule of thumb to use
to make its predictions.

153
00:10:28,823 --> 00:10:29,452
So here,

154
00:10:29,452 --> 00:10:34,579
we set this to the most frequent strategy
to predict the most frequent class.

155
00:10:36,432 --> 00:10:40,180
The DummyClassifier here is used
just like a regular classifier.

156
00:10:40,180 --> 00:10:44,951
So to prepare it for prediction,
we call the fit method on the x_train and

157
00:10:44,951 --> 00:10:49,070
y_train variables that hold
the training set instances and labels.

158
00:10:50,698 --> 00:10:55,280
Now this DummyClassifier won't actually be
looking at the individual data instances

159
00:10:55,280 --> 00:10:56,820
of those variables.

160
00:10:56,820 --> 00:11:01,014
But it does use the y train variable to
determine which class in the training data

161
00:11:01,014 --> 00:11:02,025
is most frequent.

162
00:11:04,182 --> 00:11:06,866
Finally, just like a regular classifier,

163
00:11:06,866 --> 00:11:10,939
we can call the predict method to
make predictions on the test set.

164
00:11:12,620 --> 00:11:15,990
This example shows the output of
the DummyClassifier's predictions.

165
00:11:15,990 --> 00:11:21,140
And as promised, you can see it's always
predicting 0 or the negative class for

166
00:11:21,140 --> 00:11:22,890
every instance in the test set.

167
00:11:24,613 --> 00:11:29,310
Now we can call the usual score method to
get the accuracy of the DummyClassifier's

168
00:11:29,310 --> 00:11:31,200
constant negative prediction.

169
00:11:32,350 --> 00:11:37,190
And we can see it's also 90%,
the same as our earlier support vector

170
00:11:37,190 --> 00:11:40,730
machine classifier with
radio bases function kernel.

171
00:11:42,210 --> 00:11:45,070
So that support vector classifier
was actually performing only

172
00:11:45,070 --> 00:11:47,345
very slightly better
than the DummyClassifier.

173
00:11:49,662 --> 00:11:53,558
The dummy classifier provides what
is called a null accuracy baseline.

174
00:11:53,558 --> 00:11:58,570
That is the accuracy that can be achieved
by always picking the most frequent class.

175
00:11:59,590 --> 00:12:03,270
You should not use a dummy classifier for
real classification problems, but

176
00:12:03,270 --> 00:12:06,400
it does provide a useful sanity
check in point of comparison.

177
00:12:08,120 --> 00:12:12,350
There are other types of dummy classifiers
that provide null base lines corresponding

178
00:12:12,350 --> 00:12:15,389
to other choices of the strategy
parameter as shown here.

179
00:12:16,555 --> 00:12:20,360
Most_frequent is the strategy we've
just seen that always predicts

180
00:12:20,360 --> 00:12:21,909
the most_frequent label.

181
00:12:21,909 --> 00:12:27,138
The stratified strategy, unlike
the constant most_frequent prediction

182
00:12:27,138 --> 00:12:32,620
is a random prediction that's
based on the class distributions.

183
00:12:32,620 --> 00:12:37,440
For example, if the positive class occurs
90% of the time in the training set.

184
00:12:37,440 --> 00:12:40,020
Then the stratified DummyClassifier

185
00:12:40,020 --> 00:12:43,660
will output the positive class
label with 90% probability.

186
00:12:43,660 --> 00:12:45,960
Otherwise, it will output
the negative class label.

187
00:12:47,580 --> 00:12:52,074
This can help ensure that metrics that
rely on having counts of both positive and

188
00:12:52,074 --> 00:12:55,176
negative class prediction
outcomes can be computed.

189
00:12:55,176 --> 00:12:59,681
The uniform strategy is another random
prediction method that will generate class

190
00:12:59,681 --> 00:13:02,440
predictions uniformly at random.

191
00:13:02,440 --> 00:13:06,230
That is, all classes have
an equal chance at being output

192
00:13:06,230 --> 00:13:09,350
as opposed to being weighed by their
frequency in the training set.

193
00:13:10,980 --> 00:13:13,890
This strategy may be useful for
gaining an accurate estimate of

194
00:13:13,890 --> 00:13:16,899
what the most common types of
prediction errors for each class.

195
00:13:18,620 --> 00:13:23,367
Finally, the constant strategy can be
useful when computing some metrics

196
00:13:23,367 --> 00:13:26,620
like F score,
which we will cover in a few minutes.

197
00:13:26,620 --> 00:13:28,650
Well, why is that?

198
00:13:28,650 --> 00:13:31,500
Well, when we have a binary
classification task where the most

199
00:13:31,500 --> 00:13:34,310
frequent class is the negative class.

200
00:13:34,310 --> 00:13:37,280
Turns out that using the most
frequent strategy will never predict

201
00:13:37,280 --> 00:13:38,280
the positive class.

202
00:13:38,280 --> 00:13:42,539
And will never be able to count the number
of positive instances that are correctly

203
00:13:42,539 --> 00:13:43,225
predicted.

204
00:13:43,225 --> 00:13:47,869
And so the overall count of such
positive correct predictions will be 0.

205
00:13:47,869 --> 00:13:50,639
So this in turn as you
will see in a few minutes,

206
00:13:50,639 --> 00:13:55,440
we'll cause some important metrics
like F scores to always be zero.

207
00:13:55,440 --> 00:14:00,341
So using the constant strategy,
we can force a dummy classifier to always

208
00:14:00,341 --> 00:14:05,648
predict the positive class even if it's
the minority class in a set of classes.

209
00:14:05,648 --> 00:14:09,676
And this will lead to more
meaningful computation of F-score.

210
00:14:12,543 --> 00:14:17,023
So what does it mean if we discover
that our classifier has close

211
00:14:17,023 --> 00:14:20,102
to the DummyClassifier's performance?

212
00:14:20,102 --> 00:14:25,040
While typically it means that the features
in our model may be ineffective,

213
00:14:25,040 --> 00:14:28,800
or erroneously computed or
missing for some reason,

214
00:14:28,800 --> 00:14:33,940
it could also be caused by a poor choice
of kernel or hyperparameter in the model.

215
00:14:35,100 --> 00:14:39,200
For example, if we change the support
vector classifier's kernel parameter to

216
00:14:39,200 --> 00:14:42,742
linear from rbf.

217
00:14:42,742 --> 00:14:46,451
And recompute the accuracy
on this retrain classifier,

218
00:14:46,451 --> 00:14:50,932
we can see that this leads to much
better performance of almost 98%

219
00:14:50,932 --> 00:14:54,887
compared to the most frequently
class based line of 90%.

220
00:14:54,887 --> 00:15:00,185
Finally, if you have accuracy that is
close to that of a dummy classifier,

221
00:15:00,185 --> 00:15:04,490
it could be because there is
indeed a large class imbalance.

222
00:15:04,490 --> 00:15:09,199
And the accuracy gains produced by
the classifier on the test set simply

223
00:15:09,199 --> 00:15:12,980
applied too few examples to
produce a significant gain.

224
00:15:14,440 --> 00:15:17,103
In general, for
imbalanced classification problems,

225
00:15:17,103 --> 00:15:19,216
you should use metrics
other than accuracy.

226
00:15:19,216 --> 00:15:25,510
We'll look at one shortly called AUC,
which is short for area under the curve.

227
00:15:25,510 --> 00:15:30,393
DummyRegressors, as you might guess, are
the counterpart to DummyClassifiers for

228
00:15:30,393 --> 00:15:31,230
regression.

229
00:15:31,230 --> 00:15:35,555
And they serve a similar role
as a null outcome baseline and

230
00:15:35,555 --> 00:15:38,389
sanity check for regression models.

231
00:15:38,389 --> 00:15:42,714
Since regression models have
continuous value prediction outputs.

232
00:15:42,714 --> 00:15:47,379
The strategy parameter for DummyRegressors
gives you a choice of function that

233
00:15:47,379 --> 00:15:52,766
you can apply to the distribution of
target values found in the training set.

234
00:15:52,766 --> 00:15:58,990
You can ask for the mean or
median value of the training set targets.

235
00:15:58,990 --> 00:16:03,196
The value corresponding to
the quantile that you provide Or

236
00:16:03,196 --> 00:16:05,031
a custom constant value.

237
00:16:07,993 --> 00:16:10,591
Now let's look more carefully
at the different types

238
00:16:10,591 --> 00:16:14,000
of outcomes we might see
using a binary classifier.

239
00:16:14,000 --> 00:16:16,650
This will give us some
insight into why using

240
00:16:16,650 --> 00:16:21,230
just accuracy doesn't give a complete
picture of the classifier's performance.

241
00:16:21,230 --> 00:16:22,730
And will motivate our definition and

242
00:16:22,730 --> 00:16:25,349
exploration of additional
evaluation metrics.

243
00:16:26,740 --> 00:16:30,850
With a positive and negative class,
there are four possible outcomes that we

244
00:16:30,850 --> 00:16:35,229
can break into two cases corresponding to
the first and second row of this matrix.

245
00:16:36,940 --> 00:16:40,989
If the true label for an instance is
negative, the classifier can predict

246
00:16:40,989 --> 00:16:44,660
either negative, which is correct,
and call the true negative.

247
00:16:45,760 --> 00:16:48,840
Or it can erroneously predict positive,
which is an error and

248
00:16:48,840 --> 00:16:50,090
called a false positive.

249
00:16:51,170 --> 00:16:56,178
If the true label for an instance is
positive, the classifier can predict

250
00:16:56,178 --> 00:17:00,640
either negative, which is an error and
called a false negative.

251
00:17:00,640 --> 00:17:04,089
Or it can predict positive, which is
correct and that's called a true positive.

252
00:17:05,582 --> 00:17:09,950
So maybe a quick way to remember this is
that the first word in these matrix cells

253
00:17:09,950 --> 00:17:15,300
is false, if it's a a classifier error,
or true if it's a classifier success.

254
00:17:16,380 --> 00:17:19,510
The second word is negative if
the true label is negative and

255
00:17:19,510 --> 00:17:21,650
positive if the true label is positive.

256
00:17:22,790 --> 00:17:23,460
Another name for

257
00:17:23,460 --> 00:17:28,480
a false positive that you might know
from statistics is a type one error.

258
00:17:28,480 --> 00:17:33,948
And another name for
a false negative is a type two error.

259
00:17:33,948 --> 00:17:39,023
We're going to use these two-letter
combinations, TN, FN, FP,

260
00:17:39,023 --> 00:17:45,970
and TP, as variable names, when defining
some new evaluation metrics shortly.

261
00:17:45,970 --> 00:17:50,015
We'll also use capital N here to denote
the total number of instances, of the sum

262
00:17:50,015 --> 00:17:53,900
of all the values in the matrix, the
number of data points we're looking at.

263
00:17:56,662 --> 00:18:00,003
This matrix of all combinations
of predicted label and

264
00:18:00,003 --> 00:18:02,550
true label is called a confusion matrix.

265
00:18:04,050 --> 00:18:07,150
We can take any classifier
prediction on a data instance and

266
00:18:07,150 --> 00:18:10,550
associate it with one of these
matrix cells, depending on

267
00:18:10,550 --> 00:18:14,310
the true label of the instance and
the classifier's predicted label.

268
00:18:16,560 --> 00:18:19,350
This also applies to
multi-class classification,

269
00:18:19,350 --> 00:18:23,718
in addition to the special case of
binary classification I've shown here.

270
00:18:23,718 --> 00:18:27,533
In the multi-class case with k classes,

271
00:18:27,533 --> 00:18:32,878
we simply have a k by k matrix
instead of a two by two matrix.

272
00:18:32,878 --> 00:18:37,560
Scikit-learn makes it easy to compute
a confusion matrix for your classifier.

273
00:18:37,560 --> 00:18:41,087
Let's take a look at the notebook.

274
00:18:41,087 --> 00:18:46,900
Here, we import the confusion
matrix class from sklearn.metrics.

275
00:18:46,900 --> 00:18:50,000
We're going to use the same training
set from the digits data set

276
00:18:50,000 --> 00:18:52,780
with the binary imbalance
labels that we created earlier.

277
00:18:54,540 --> 00:19:00,019
To get the confusion matrix, we simply
pass the y_test set of predicted labels

278
00:19:00,019 --> 00:19:04,865
and the y predicted set of predicted
labels and then print the output.

279
00:19:04,865 --> 00:19:08,813
The order of the cells of the little
matrix output here is the same as the one

280
00:19:08,813 --> 00:19:10,290
I just showed on the slide.

281
00:19:11,690 --> 00:19:16,010
True negative and false negative are in
the first column, and true positive and

282
00:19:16,010 --> 00:19:17,740
false positive are in the second column.

283
00:19:19,000 --> 00:19:23,330
In particular, the successful predictions
of the classifier are on the diagonal

284
00:19:23,330 --> 00:19:26,960
where the true class matches
the predicted class.

285
00:19:26,960 --> 00:19:30,460
The cells off the diagonal represent
errors of different types.

286
00:19:31,830 --> 00:19:35,760
Here, we compute the confusion matrices
for different choices of classifier in

287
00:19:35,760 --> 00:19:39,680
the problem so we can see how they shift
slightly with different choices of model.

288
00:19:41,910 --> 00:19:44,510
And this gives us some insight
into the nature of successes and

289
00:19:44,510 --> 00:19:46,580
failures observed for
each type of classifier.

290
00:19:48,620 --> 00:19:54,050
So first, we'll apply the most frequent
class DummyClassifier we saw earlier.

291
00:19:54,050 --> 00:19:56,428
What we can see here is
that the right column,

292
00:19:56,428 --> 00:20:01,140
that represent cases where the classifier
predicted the positive class, is all zero.

293
00:20:01,140 --> 00:20:05,755
Which makes sense for this dummy
classifier because it's always predicting

294
00:20:05,755 --> 00:20:08,642
the negative class, the most frequent one.

295
00:20:08,642 --> 00:20:11,754
We see that 407 instances
are true negatives, and

296
00:20:11,754 --> 00:20:14,598
there are 43 errors that
are false negatives.

297
00:20:16,178 --> 00:20:20,380
Here we apply the stratified
DummyClassifier that gives random output

298
00:20:20,380 --> 00:20:24,560
in proportion to the ratio
labels in the training set.

299
00:20:24,560 --> 00:20:27,640
Now the right column is no longer all zero

300
00:20:27,640 --> 00:20:32,450
because this DummyClassifier does predict
occasionally predict the positive class.

301
00:20:32,450 --> 00:20:37,105
If we add the numbers in the right column,
we see that 32 plus 6 equals 38 times

302
00:20:37,105 --> 00:20:41,820
the number of times the classifier
predicted the positive class.

303
00:20:41,820 --> 00:20:47,118
Of those times, in six cases, the lower
right diagonal, this was a true positive.

304
00:20:49,548 --> 00:20:54,240
In the next case, we'll apply a support
vector classifier with linear kernel and

305
00:20:54,240 --> 00:20:55,940
seed parameter equal to one.

306
00:20:57,270 --> 00:21:01,180
We note that looking along the diagonal
compared to the stratified dummy

307
00:21:01,180 --> 00:21:09,520
classifier above, which had a total of
375 plus 6, or 381 correct predictions.

308
00:21:09,520 --> 00:21:13,900
The support vector classifier
has a total of 402 plus 38,

309
00:21:13,900 --> 00:21:18,179
which is 440 correct predictions
on the same data set.

310
00:21:20,400 --> 00:21:24,484
Likewise, we can apply a logistic
regression classifier, and

311
00:21:24,484 --> 00:21:28,655
that obtains similar results to
the support vector classifier.

312
00:21:28,655 --> 00:21:32,110
And finally, we can apply
a decision tree classifier, and

313
00:21:32,110 --> 00:21:36,210
look at the confusion matrix
that results from that.

314
00:21:36,210 --> 00:21:40,240
One thing we notice is, that unlike
the support vector or logistic regression

315
00:21:40,240 --> 00:21:45,830
classifier, which had balanced numbers
of false negatives and false positives.

316
00:21:45,830 --> 00:21:50,709
The decision tree makes more than
twice as many false negative errors,

317
00:21:50,709 --> 00:21:55,766
17 of them actually, as false positive
errors, of which there are 7.

318
00:21:55,766 --> 00:22:00,372
Now that we've seen how a confusion matrix
can give us a little more information

319
00:22:00,372 --> 00:22:04,978
about the types of errors a classifier
makes, we're ready to move ahead and and

320
00:22:04,978 --> 00:22:09,036
define some new types of evaluation
metrics that use information from

321
00:22:09,036 --> 00:22:13,740
the computing matrix to give different
perspectives on classifier performance