1
00:00:08,822 --> 00:00:13,470
Many classifiers in scikit learn can
provide information about the uncertainty

2
00:00:13,470 --> 00:00:18,050
associated with a particular prediction
either by using the decision function

3
00:00:18,050 --> 00:00:20,183
method or the predict proba method.

4
00:00:21,280 --> 00:00:25,940
When given a set of test points,
the decision function method provides for

5
00:00:25,940 --> 00:00:30,220
each one a classifier score value
that indicates how confidently

6
00:00:30,220 --> 00:00:33,080
classifier predicts the positive class.

7
00:00:33,080 --> 00:00:35,310
So there will be large
magnitude positive scores for

8
00:00:35,310 --> 00:00:38,560
those points, or
it predicts a negative class,

9
00:00:38,560 --> 00:00:41,729
there'll be large magnitude negative
scores for negative points.

10
00:00:43,270 --> 00:00:46,500
Here's an example in the notebook
showing the first few instances from our

11
00:00:46,500 --> 00:00:51,020
classification problem using
a logistic regression classifier.

12
00:00:51,020 --> 00:00:55,260
We can see the instances in the negative
class often have large magnitude

13
00:00:55,260 --> 00:00:56,780
negative scores.

14
00:00:56,780 --> 00:01:00,483
And indeed the instances in the positive
class has positive scores from

15
00:01:00,483 --> 00:01:02,529
the logistic regression classifier.

16
00:01:04,841 --> 00:01:08,821
Likewise, the predict proba function
provides predicted probabilities of

17
00:01:08,821 --> 00:01:09,870
class membership.

18
00:01:10,950 --> 00:01:14,640
Typically a classifier which
use the more likely class.

19
00:01:14,640 --> 00:01:16,774
That is in a binary classifier,

20
00:01:16,774 --> 00:01:20,656
you find the class with
probability greater than 50%.

21
00:01:20,656 --> 00:01:24,130
Adjusting this decision threshold affects
the prediction of the classifier.

22
00:01:25,350 --> 00:01:28,410
A higher threshold means that
a classifier has to be more confident

23
00:01:28,410 --> 00:01:30,330
in predicting the class.

24
00:01:30,330 --> 00:01:34,510
For example, we might predict class one
only if the estimated probability of class

25
00:01:34,510 --> 00:01:35,420
one was over 70%.

26
00:01:35,420 --> 00:01:38,570
And this results in a more
conservative classifier.

27
00:01:40,910 --> 00:01:43,540
Here's an example of getting these
prediction probabilities for

28
00:01:43,540 --> 00:01:46,550
the test instances for
the same logistic regression classifier.

29
00:01:48,060 --> 00:01:51,911
You can see that many entries
with a positive label of one,

30
00:01:51,911 --> 00:01:54,753
have a high probability like 0.995.

31
00:01:54,753 --> 00:02:00,819
While many negative label instances
have a very low prediction probability.

32
00:02:00,819 --> 00:02:05,070
Note that not all models provide useful
probability estimates of this type.

33
00:02:05,070 --> 00:02:08,190
For example, a model that was
over-fit to a trending set.

34
00:02:08,190 --> 00:02:12,140
Might provide overly optimistic high
probabilities that were in fact

35
00:02:12,140 --> 00:02:12,880
not accurate.

36
00:02:14,310 --> 00:02:18,210
Now, we can use these decision scores or
prediction probabilities for

37
00:02:18,210 --> 00:02:22,990
getting more complete evaluation
picture of a classifiers performance.

38
00:02:22,990 --> 00:02:26,720
For a particular application, we might
pick a specific decision threshold

39
00:02:26,720 --> 00:02:29,920
depending on whether we want
the classifier to be more or

40
00:02:29,920 --> 00:02:34,250
less conservative about making
false-positive or false-negative errors.

41
00:02:34,250 --> 00:02:37,080
It might not be entirely clear
when developing a new model,

42
00:02:37,080 --> 00:02:40,010
what the right decision
threshold would be, and

43
00:02:40,010 --> 00:02:43,770
how that choice will affect evaluation
metrics like precision and recall.

44
00:02:43,770 --> 00:02:44,980
So instead, what we'll do is,

45
00:02:44,980 --> 00:02:49,930
look at how classifier performs for
all possible decision thresholds.

46
00:02:49,930 --> 00:02:53,655
This example shows how that works.

47
00:02:53,655 --> 00:02:58,023
On the left here is a list of test
instances with their true label and

48
00:02:58,023 --> 00:02:59,350
classifier score.

49
00:03:00,830 --> 00:03:06,250
If we set a decision threshold, then
all the instances above that line, for

50
00:03:06,250 --> 00:03:12,387
example if we set the decision
threshold to be -20 here.

51
00:03:14,570 --> 00:03:18,830
Then, all the instances above the line
are below the threshold of -20.

52
00:03:18,830 --> 00:03:23,628
So -20 or less and
all the instances in this

53
00:03:23,628 --> 00:03:28,561
direction are above the threshold of -20.

54
00:03:28,561 --> 00:03:33,070
And so the ones below the threshold
will be predicted to be in the- class.

55
00:03:34,680 --> 00:03:38,814
And the ones above the threshold will
be predicted to be in the + class.

56
00:03:41,754 --> 00:03:47,160
So, if we pick the specific threshold,
so in this case, -20.

57
00:03:47,160 --> 00:03:51,220
And we partition the test
points in this way.

58
00:03:51,220 --> 00:03:52,830
We can compute partition and

59
00:03:52,830 --> 00:03:58,660
recall for the points that are predicted
to be in the positive class.

60
00:03:58,660 --> 00:04:06,389
So in this case, we have 12
instances here, 12 total instances.

61
00:04:07,600 --> 00:04:12,777
They're being predicted as positive and
only four of them, this one,

62
00:04:12,777 --> 00:04:17,264
this one, this one, and
this one are actually positive and so

63
00:04:17,264 --> 00:04:22,292
the precision here is 4 divided by 12 or
approximately 0.34.

64
00:04:24,982 --> 00:04:26,823
The recall on the other hand,

65
00:04:26,823 --> 00:04:31,981
there are four positive labeled instances
in the whole set of test examples here and

66
00:04:31,981 --> 00:04:36,610
we've found all of them with this
particular threshold setting.

67
00:04:36,610 --> 00:04:42,097
So the recall here is 4 out of 4, we
found all four positive labeled examples.

68
00:04:42,097 --> 00:04:45,957
And so, for
this particular threshold of -20,

69
00:04:45,957 --> 00:04:50,556
we can obtain precision on re
cost score for that threshold.

70
00:04:53,660 --> 00:04:58,580
Let's pick a different threshold let's
look at what happened when the threshold

71
00:04:58,580 --> 00:04:59,301
is -10?

72
00:04:59,301 --> 00:05:04,100
Right here, so again anything
below this line is treated and

73
00:05:04,100 --> 00:05:10,610
has a higher value than -10 here, so
those would be treated as + predictions.

74
00:05:11,750 --> 00:05:16,510
Things above the line have a score below
-10, so these would be predicted to be

75
00:05:18,200 --> 00:05:21,990
And again, we can compute a precision and
recall for

76
00:05:21,990 --> 00:05:27,200
this decision threshold setting, and
we can see here that there are a total

77
00:05:27,200 --> 00:05:32,460
of six instances in
the + prediction class.

78
00:05:32,460 --> 00:05:38,090
Of which four are actually
of the positive class, and

79
00:05:38,090 --> 00:05:43,250
so the precision here is 4 over 6 or
about 0.67.

80
00:05:43,250 --> 00:05:50,620
And again, the recall here is going to
be 4 out of 4, and it's going to be 1.0.

81
00:05:50,620 --> 00:05:58,459
Again, so that corresponds to this
point in the table over here.

82
00:05:58,459 --> 00:06:01,853
And then as were computing these
different precision and recalls for

83
00:06:01,853 --> 00:06:03,090
different Thresholds.

84
00:06:03,090 --> 00:06:06,210
We can also plot them on
this precision recall chart.

85
00:06:06,210 --> 00:06:11,891
So the first pair of precision
recall numbers that I got, 0.34 and

86
00:06:11,891 --> 00:06:16,912
1.0, we can plot on this point
in precision recall space.

87
00:06:16,912 --> 00:06:21,179
The second example, so
this was for the threshold of -20.

88
00:06:22,240 --> 00:06:26,020
When the threshold was -10,
we got precision of .67 and

89
00:06:26,020 --> 00:06:29,930
a recall of 1 corresponding to
this point that we can plot.

90
00:06:31,050 --> 00:06:36,590
And so you can see that if we do this for
a number of other thresholds, for

91
00:06:36,590 --> 00:06:41,230
example the threshold of 0,
we'll get a precision of 0.75.

92
00:06:41,230 --> 00:06:44,160
And a recall of 0.75 that
corresponds to this point.

93
00:06:45,780 --> 00:06:48,510
And in that choice of decision threshold.

94
00:06:48,510 --> 00:06:52,520
And we can keep doing that for
different thresholds.

95
00:06:52,520 --> 00:06:56,910
And we actually are plotting a series
of points through the space which

96
00:06:56,910 --> 00:06:58,329
we can be connected at as a curve.

97
00:06:59,440 --> 00:07:04,970
And so in this way, we can get a more
complete picture by varying the threshold

98
00:07:04,970 --> 00:07:07,920
of how the precision and
recall of the result and

99
00:07:07,920 --> 00:07:11,680
classifier output changes as
a function of the decision threshold.

100
00:07:13,820 --> 00:07:19,020
And this resulting chart here is
called a precision recall curve and

101
00:07:19,020 --> 00:07:20,710
we'll look at it in more detail next.