1
00:00:10,157 --> 00:00:15,098
Now that we've looked at
evaluation of binary classifiers,

2
00:00:15,098 --> 00:00:19,848
let's take a look at how the more
general case of multi class

3
00:00:19,848 --> 00:00:23,375
classification is handled in evaluation.

4
00:00:23,375 --> 00:00:29,116
So in may respects, multi-class
evaluation is a straightforward

5
00:00:29,116 --> 00:00:33,760
extension of the methods we
use in binary evaluation.

6
00:00:33,760 --> 00:00:36,001
Instead of two classes,
we have multiple classes.

7
00:00:36,001 --> 00:00:40,992
So, the results for
multi-class evaluation amount to

8
00:00:40,992 --> 00:00:46,846
a collection of true verses
predicted binary outcome per class.

9
00:00:46,846 --> 00:00:49,687
And just as we saw in the binary case,

10
00:00:49,687 --> 00:00:54,646
you can generate confusion
matrices in the multi-class case.

11
00:00:54,646 --> 00:00:58,967
They're especially useful when
you have multiple classes,

12
00:00:58,967 --> 00:01:03,785
because there are many different
kinds of errors that result from one

13
00:01:03,785 --> 00:01:07,292
true class being predicted
as a different class.

14
00:01:07,292 --> 00:01:08,898
We'll look at an example of that.

15
00:01:08,898 --> 00:01:14,002
Classification reports that we saw in
the binary case are easy to generate for

16
00:01:14,002 --> 00:01:15,662
the multi-class case.

17
00:01:15,662 --> 00:01:20,808
Now the one area,
which is worth a little more examination

18
00:01:20,808 --> 00:01:24,915
is how averaging across
classes takes place.

19
00:01:26,908 --> 00:01:31,110
There are different ways to
average multi-class results that

20
00:01:31,110 --> 00:01:32,701
we'll cover shortly.

21
00:01:32,701 --> 00:01:37,229
And the support, the number of instances
for each class is important to consider.

22
00:01:37,229 --> 00:01:42,118
So just as we're all interested
in how to handle imbalance

23
00:01:42,118 --> 00:01:47,106
classes in the binary case,
it's important as you will see

24
00:01:47,106 --> 00:01:51,197
to consider similar issues
of how the support for

25
00:01:51,197 --> 00:01:57,228
classes might vary to a large or
small extent across multiple classes.

26
00:01:57,228 --> 00:02:01,335
There is a case of multi-label
classification in which each instance

27
00:02:01,335 --> 00:02:03,084
could have multiple labels.

28
00:02:03,084 --> 00:02:08,771
For example,
a web page might be labeled with different

29
00:02:08,771 --> 00:02:14,942
topics that come from a predefined
set of areas of interest.

30
00:02:14,942 --> 00:02:19,021
We won't cover multi-label
classification in this lecture.

31
00:02:19,021 --> 00:02:23,412
Instead, we'll focus exclusively
on multi-class evaluation.

32
00:02:23,412 --> 00:02:27,696
The multi-class confusion matrix is
a straightforward extension of the binary

33
00:02:27,696 --> 00:02:29,901
classifier two by two confusion matrix.

34
00:02:31,720 --> 00:02:35,590
For example, in our digits data set,
there are ten classes for

35
00:02:35,590 --> 00:02:38,010
the digits, zero through nine.

36
00:02:38,010 --> 00:02:41,990
So, the ten class confusion matrix is
a ten by ten matrix with the true digit

37
00:02:41,990 --> 00:02:46,720
class indexed by row and the predicted
digit class indexed by column.

38
00:02:48,090 --> 00:02:52,970
As with the two by two case, the correct
prediction is by the classifier where

39
00:02:52,970 --> 00:02:58,040
the true class matches the predicted
class are all along the diagonal and

40
00:02:58,040 --> 00:03:00,200
misclassifications are off the diagonal.

41
00:03:02,060 --> 00:03:05,550
In this example which was created
using the following notebook code

42
00:03:07,420 --> 00:03:12,530
based on a support vector classifier
with linear kernel, we can see that most

43
00:03:12,530 --> 00:03:17,210
of the predictions are correct with only
a few misclassifications here and there.

44
00:03:18,440 --> 00:03:23,730
The most frequent type of mistake here is
apparently misclassifying the true digit,

45
00:03:23,730 --> 00:03:27,080
eight as a predicted digit one
which happened three times.

46
00:03:28,350 --> 00:03:33,505
And indeed, the overall accuracy is high,
about 97% as shown here.

47
00:03:33,505 --> 00:03:38,202
As an aside, it's sometimes useful to
display a confusion matrix as a heat map

48
00:03:38,202 --> 00:03:42,918
in order to highlight the relative
frequencies of different types of errors.

49
00:03:42,918 --> 00:03:47,028
So, I've included the code
to generate that here.

50
00:03:47,028 --> 00:03:48,172
For comparison,

51
00:03:48,172 --> 00:03:52,596
I've also included a second confusion
matrix on the same dataset for

52
00:03:52,596 --> 00:03:57,642
another support vector classifier that
does much worse in a distinctive way.

53
00:03:57,642 --> 00:03:59,793
The only change is to use an RBF,

54
00:03:59,793 --> 00:04:03,710
radial basis function kernel
instead of a linear kernel.

55
00:04:05,845 --> 00:04:10,245
While we can see for the accuracy number
were about 43% below the confusion matrix

56
00:04:10,245 --> 00:04:14,435
that the classifier is doing much
worse than the delinear kernel,

57
00:04:14,435 --> 00:04:17,315
that single number doesn't
give much insight into why.

58
00:04:18,955 --> 00:04:24,143
Looking at the confusion matrix, however,
reveals that for every true digit class,

59
00:04:24,143 --> 00:04:28,136
a significant fraction of outcomes
are to predict the digit four.

60
00:04:28,136 --> 00:04:29,655
That's rather surprising.

61
00:04:29,655 --> 00:04:34,233
For example, of the 44 instances
of the true digit 2 in row 2,

62
00:04:34,233 --> 00:04:39,414
17 are classified correctly, but
27 are classified as the digit 4.

63
00:04:39,414 --> 00:04:43,989
Clearly, something is broken with this
model and I picked this second example

64
00:04:43,989 --> 00:04:48,648
just to show an extreme example of what
you might see when things go quite wrong.

65
00:04:48,648 --> 00:04:51,680
This digits dataset is
well-established and free of problems.

66
00:04:51,680 --> 00:04:54,987
But especially when developing
with a new dataset,

67
00:04:54,987 --> 00:04:59,875
seeing patterns like this in a confusion
matrix could give you valuable clues

68
00:04:59,875 --> 00:05:04,778
about possible problems, say in
the feature pre-processing for example.

69
00:05:04,778 --> 00:05:08,260
So as a general rule of thumb
as part of model evaluation,

70
00:05:08,260 --> 00:05:12,623
I suggest always looking at the confusion
matrix for your classifier.

71
00:05:12,623 --> 00:05:16,037
To get some insight into what
kind of errors it is making for

72
00:05:16,037 --> 00:05:20,776
each class including whether some classes
are much more prone to certain kinds of

73
00:05:20,776 --> 00:05:22,044
errors than others.

74
00:05:24,577 --> 00:05:26,708
Next, just as in the binary case,

75
00:05:26,708 --> 00:05:32,003
you can get a classification report that
summarizes multiple evaluation metrics for

76
00:05:32,003 --> 00:05:36,730
a multi-class classifier with an average
metric computed for each class.

77
00:05:38,824 --> 00:05:42,377
Now what I'm about to describe,
also applies to the binary class case,

78
00:05:42,377 --> 00:05:46,399
but it's easier to see when looking at
multi-class classification problem with

79
00:05:46,399 --> 00:05:47,350
several classes.

80
00:05:49,320 --> 00:05:55,848
So, here's an example of how to compute
macro-average precision and micro-average

81
00:05:55,848 --> 00:06:01,506
precision on a sample dataset that I
have extracted from our fruit dataset.

82
00:06:01,506 --> 00:06:02,669
In this example,

83
00:06:02,669 --> 00:06:07,642
we have three columns where the first
column is the true class of an example.

84
00:06:07,642 --> 00:06:12,503
The second column is the predictive
class from some classifier and

85
00:06:12,503 --> 00:06:16,756
the third column is a binary
variable that denotes whether

86
00:06:16,756 --> 00:06:20,080
the predictive class
matches the two class.

87
00:06:22,100 --> 00:06:25,840
And here, we have in our, this is
a multi-class classification problem.

88
00:06:27,410 --> 00:06:32,026
And so, we have three classes here.

89
00:06:32,026 --> 00:06:36,717
We have several instances there,
the orange class.

90
00:06:36,717 --> 00:06:39,024
We have two instances that
are the lemon class and

91
00:06:39,024 --> 00:06:41,280
we have two instances
that are the apple class.

92
00:06:43,460 --> 00:06:50,882
So in this first example,
we'll compute macro-average precision and

93
00:06:50,882 --> 00:06:59,169
the key aspect of macro-average precision
is that each class has equal weight.

94
00:06:59,169 --> 00:07:04,231
So in this case,
each of these classes will contribute

95
00:07:04,231 --> 00:07:10,957
one-third weight towards the final
macro-average precision value.

96
00:07:12,828 --> 00:07:16,509
So, there are two steps to
compute macro-average precision.

97
00:07:16,509 --> 00:07:19,660
The first one is to compute the metric.

98
00:07:19,660 --> 00:07:24,252
So in this case, we're going to
compute precision within each class.

99
00:07:24,252 --> 00:07:26,527
So, let's take a look at the orange class.

100
00:07:26,527 --> 00:07:31,240
There are five total examples
in the orange class and

101
00:07:31,240 --> 00:07:36,715
only one of them was predicted
correctly by the classifier.

102
00:07:36,715 --> 00:07:41,217
And so, that leads to a precision for

103
00:07:41,217 --> 00:07:46,602
the orange class of 1 out of 5 or 0.20.

104
00:07:46,602 --> 00:07:48,663
For the second class, the lemon class.

105
00:07:48,663 --> 00:07:52,549
There are a total of two instances and

106
00:07:52,549 --> 00:07:57,437
only one of them was predicted correctly,
and

107
00:07:57,437 --> 00:08:04,977
that leads to a precision of one-half or
0.50 for the lemon class.

108
00:08:04,977 --> 00:08:12,496
Let's write the precision for each of
the classes that we have calculated.

109
00:08:15,104 --> 00:08:18,049
This was.

110
00:08:21,415 --> 00:08:24,466
And for the third class, the apple class.

111
00:08:24,466 --> 00:08:28,338
The classifier predicted
both of these correctly.

112
00:08:28,338 --> 00:08:34,635
So, that's a precision of 2 out of 2 or
1.0.

113
00:08:38,087 --> 00:08:38,882
That's the first step.

114
00:08:38,882 --> 00:08:43,570
We've computed the precision
metric within each class.

115
00:08:43,570 --> 00:08:48,030
And then in the second step,
we simply average across these three to

116
00:08:48,030 --> 00:08:52,818
produce the final result,
to get our final macro-average precision.

117
00:08:52,818 --> 00:08:57,877
And so
we can simply compute the average of 0.2,

118
00:08:57,877 --> 00:09:02,701
0.5 and 1 and
we get our final macro-average

119
00:09:02,701 --> 00:09:07,419
precision for this set of results of 0.57.

120
00:09:07,419 --> 00:09:12,997
You'll notice here that no matter how
many instances they were in each class,

121
00:09:12,997 --> 00:09:16,972
because we computed position
within each class first,

122
00:09:16,972 --> 00:09:21,637
each class contributes equally
to the overall macro-average.

123
00:09:21,637 --> 00:09:25,480
So we could have had, for
example, a million examples and

124
00:09:25,480 --> 00:09:27,058
from the orange class.

125
00:09:27,058 --> 00:09:31,572
But that class would have still been
weighted equally, because we would have

126
00:09:31,572 --> 00:09:35,739
first computed precision for
the million orange examples and then that

127
00:09:35,739 --> 00:09:40,333
number would still get a third of the
weight compared to the other two classes.

128
00:09:40,333 --> 00:09:43,666
So, that's macro-average precision.

129
00:09:47,751 --> 00:09:54,192
Micro-average precision is
computed a little differently and

130
00:09:54,192 --> 00:10:00,401
it gives each instance in the data
results here equal weight.

131
00:10:00,401 --> 00:10:04,048
In micro-average precision,
we don't compute precision for

132
00:10:04,048 --> 00:10:05,470
each class separately.

133
00:10:05,470 --> 00:10:07,970
We treat the entire dataset,

134
00:10:07,970 --> 00:10:12,676
the entire set of results
here as an aggregate outcome.

135
00:10:12,676 --> 00:10:15,584
So to compute micro-average precision,

136
00:10:15,584 --> 00:10:18,742
we simply look at how
many of all the examples.

137
00:10:18,742 --> 00:10:21,957
We have nine examples here in total and

138
00:10:21,957 --> 00:10:27,486
micro-average precision will
simply compute the precision for

139
00:10:27,486 --> 00:10:32,724
all the examples,
regardless of class in the set of results.

140
00:10:32,724 --> 00:10:34,753
So out of these nine instances,

141
00:10:34,753 --> 00:10:39,121
we have found that the classifier
predicted four of them correctly.

142
00:10:41,440 --> 00:10:46,000
And so, the micro-average precision is

143
00:10:46,000 --> 00:10:50,435
simply computed as 4/9 or 0.44.

144
00:10:50,435 --> 00:10:57,178
And you'll notice here that if we had
a million instances of the orange class,

145
00:10:57,178 --> 00:11:01,778
for example,
that with micro-average precision,

146
00:11:01,778 --> 00:11:05,375
because each instance has equal weight.

147
00:11:05,375 --> 00:11:09,019
That would lead to the orange
class contributing many,

148
00:11:09,019 --> 00:11:13,218
many more instances to our
overall micro-average precision.

149
00:11:13,218 --> 00:11:17,787
And so, the effect of
micro-average precision is to give

150
00:11:17,787 --> 00:11:22,180
classes with a lot more
instances much more influence.

151
00:11:22,180 --> 00:11:26,777
So, the average here would have been
influenced much more by the million orange

152
00:11:26,777 --> 00:11:29,741
examples than by the two lemon and
apple examples.

153
00:11:31,799 --> 00:11:37,541
And so, that is the difference between
micro and macro-average precision.

154
00:11:37,541 --> 00:11:41,488
If the classes have about the same
number of instances, macro and

155
00:11:41,488 --> 00:11:43,948
micro-average will be about the same.

156
00:11:43,948 --> 00:11:48,690
If some classes are much larger,
have more instances than others and

157
00:11:48,690 --> 00:11:54,181
you want to weight your metric toward
the largest ones, use micro-averaging.

158
00:11:54,181 --> 00:12:00,407
If you want to weight your metric towards
the smallest classes, use macro-averaging.

159
00:12:00,407 --> 00:12:04,414
If the micro-average is much
lower than the macro-average,

160
00:12:04,414 --> 00:12:08,432
then examine the larger classes for
poor metric performance.

161
00:12:08,432 --> 00:12:13,281
If the macro-average is much lower than
the micro-average, then you should

162
00:12:13,281 --> 00:12:17,922
examine the smaller classes to see why
they have poor metric performance.

163
00:12:17,922 --> 00:12:21,749
Here, we use the average
parameter on the scoring function.

164
00:12:21,749 --> 00:12:24,715
In the first example,
we used the precision metric and

165
00:12:24,715 --> 00:12:28,844
specify whether we want micro-average
precision which is the first case or

166
00:12:28,844 --> 00:12:31,438
macro-average precision
in the second case.

167
00:12:44,494 --> 00:12:47,734
In the second example,
we use the f1 metric and

168
00:12:47,734 --> 00:12:50,577
compute micro and macro-averaged f1.

169
00:12:59,912 --> 00:13:02,242
Now that we've seen how
to compute these metrics,

170
00:13:02,242 --> 00:13:04,910
let's take a look at how to use
them to do model selection.