1
00:00:07,877 --> 00:00:11,404
We're now going to look at a second
supervised learning method that in

2
00:00:11,404 --> 00:00:14,692
spite of being called a regression
measure, is actually used for

3
00:00:14,692 --> 00:00:17,590
classification and
it’s called logistic regression.

4
00:00:18,910 --> 00:00:22,840
Logistic regression can be seen as
a kind of generalized linear model.

5
00:00:23,860 --> 00:00:28,640
And like ordinary least squares and other
regression methods, logistic regression

6
00:00:28,640 --> 00:00:33,150
takes a set input variables, the features,
and estimates a target value.

7
00:00:34,380 --> 00:00:37,010
However, unlike ordinary
linear regression,

8
00:00:37,010 --> 00:00:39,550
in it's most basic form
logistic repressions target

9
00:00:39,550 --> 00:00:42,600
value is a binary variable
instead of a continuous value.

10
00:00:44,780 --> 00:00:48,570
There are flavors of logistic regression
that can also be used in cases where

11
00:00:48,570 --> 00:00:52,510
the target value to be predicted is
a multi class categorical variable,

12
00:00:52,510 --> 00:00:54,010
not just binary.

13
00:00:54,010 --> 00:00:57,560
But for now we'll focus on the simple
binary case of logistic regression.

14
00:00:59,690 --> 00:01:02,310
Let's start by looking at linear
regression, which we saw earlier.

15
00:01:03,810 --> 00:01:08,450
Linear regression predicts a real valued
output y based on a weighted sum of

16
00:01:08,450 --> 00:01:12,450
input variables or features xi,
plus a constant b term.

17
00:01:13,570 --> 00:01:16,890
This diagram shows that
formula in graphical form.

18
00:01:16,890 --> 00:01:21,270
The square boxes on the left
represent the input features, xi.

19
00:01:21,270 --> 00:01:25,590
And the values above the arrows represent
the weights that each xi is multiplied by.

20
00:01:26,990 --> 00:01:29,810
The output variable y
in the box on the right

21
00:01:29,810 --> 00:01:33,230
is the sum of all the weighted
inputs that are connected into it.

22
00:01:34,540 --> 00:01:38,630
Note that we're adding b as a constant
term by treating it as the product of

23
00:01:38,630 --> 00:01:44,320
a special constant feature with value
1 multiplied by a weight of value b.

24
00:01:44,320 --> 00:01:47,770
This formula is summarized in
equation form below the diagram.

25
00:01:49,210 --> 00:01:51,840
The job of linear regression
is to estimate values for

26
00:01:51,840 --> 00:01:54,630
the model coefficients, wi hat and b hat.

27
00:01:55,760 --> 00:01:58,930
They give a model that best fit the
training data with minimal squared error.

28
00:02:00,450 --> 00:02:03,670
Logistic regression is similar
to linear regression, but

29
00:02:03,670 --> 00:02:05,130
with one critical addition.

30
00:02:06,200 --> 00:02:08,530
Here, we show the same type
of diagram that we showed for

31
00:02:08,530 --> 00:02:13,220
linear regression with the input
variables, xi in the left boxes and

32
00:02:13,220 --> 00:02:17,420
the model coefficients wi and
b above the arrows.

33
00:02:17,420 --> 00:02:21,462
The logistic regression model still
computes a weighted sum of the input

34
00:02:21,462 --> 00:02:26,030
features xi and the intercept term b,
but it runs this result

35
00:02:26,030 --> 00:02:30,460
through a special non-linear function f,
the logistic function

36
00:02:30,460 --> 00:02:34,710
represented by this new box in the middle
of the diagram to produce the output y.

37
00:02:36,480 --> 00:02:39,893
The logistic function itself is shown in
more detail on the plot on the right.

38
00:02:39,893 --> 00:02:43,442
It's an S shaped function
that gets closer and

39
00:02:43,442 --> 00:02:48,265
closer to 1 as the input value
increases above 0 and closer and

40
00:02:48,265 --> 00:02:52,457
closer to 0 as the input
value decreases far below 0.

41
00:02:52,457 --> 00:02:57,501
The effect of applying the logistic
function is to compress the output of

42
00:02:57,501 --> 00:03:02,391
the linear function so that it's
limited to a range between 0 and 1.

43
00:03:02,391 --> 00:03:07,325
Below the diagram, you can see the formula
for the predicted output y hat which

44
00:03:07,325 --> 00:03:11,214
first computes the same linear
combination of the inputs xi,

45
00:03:11,214 --> 00:03:15,850
model coefficient weights wi hat and
intercept b hat, but runs it through

46
00:03:15,850 --> 00:03:20,460
the additional step of applying
the logistic function to produce y hat.

47
00:03:22,230 --> 00:03:26,360
If we pick different values for
b hat and the w hat coefficients,

48
00:03:26,360 --> 00:03:29,390
we'll get different variants of
this s shaped logistic function,

49
00:03:29,390 --> 00:03:31,620
which again is always between 0 and 1.

50
00:03:33,600 --> 00:03:36,690
Because the job of basic
logistic regression is to predict

51
00:03:36,690 --> 00:03:41,560
a binary output value, you can see how
this might used for binary classification.

52
00:03:42,830 --> 00:03:45,730
We could identify data
instances with the target value

53
00:03:45,730 --> 00:03:49,010
of 0 as belonging to
the negative class and

54
00:03:49,010 --> 00:03:54,180
data instances with a target value of
1 belonging to the positive class.

55
00:03:54,180 --> 00:03:59,520
Then the value of y hat, that's the output
from the logistic regression formula,

56
00:03:59,520 --> 00:04:03,600
can be interpreted as the probability
that the input data instance belongs to

57
00:04:03,600 --> 00:04:06,190
the positive class,
given its input features.

58
00:04:07,370 --> 00:04:11,275
Let's look at a specific example of
logistic regression with one input

59
00:04:11,275 --> 00:04:11,933
variable.

60
00:04:13,577 --> 00:04:17,772
Suppose we want to whether or not
a student will pass a final exam based on

61
00:04:17,772 --> 00:04:22,910
a single input variable that's the number
of hours they spend studying for the exam.

62
00:04:24,620 --> 00:04:28,550
Students who end up failing the exam
are assigned to the negative class,

63
00:04:28,550 --> 00:04:30,605
which corresponds to a target value of 0.

64
00:04:31,660 --> 00:04:35,080
And students who pass the exam
are assigned to the positive class and

65
00:04:35,080 --> 00:04:36,895
associated with a target value of 1.

66
00:04:38,190 --> 00:04:40,990
This plot shows an example training set.

67
00:04:40,990 --> 00:04:44,152
The x-axis corresponds to
the number of hours studied and

68
00:04:44,152 --> 00:04:47,970
the y-axis corresponds to
the probability of passing the exam.

69
00:04:49,480 --> 00:04:51,000
The red points to the left,

70
00:04:51,000 --> 00:04:55,410
with a target value of 0 represent points
in the training set, which are examples of

71
00:04:55,410 --> 00:04:59,040
students who failed the exam, along with
the number of hours they spent studying.

72
00:05:00,340 --> 00:05:04,050
Likewise, the blue points with target
value 1 on the right represent points in

73
00:05:04,050 --> 00:05:07,800
the training set, corresponding
to students who passed the exam.

74
00:05:07,800 --> 00:05:11,250
With their x values representing
the number of hours those students spent

75
00:05:11,250 --> 00:05:11,769
studying.

76
00:05:13,100 --> 00:05:17,861
Using logistic regression, we can
estimate model coefficients for w hat and

77
00:05:17,861 --> 00:05:22,040
b hat that produce a logistic curve
that best fits these training points.

78
00:05:23,110 --> 00:05:26,310
In this example that logistic curve
might look something like this.

79
00:05:27,990 --> 00:05:31,430
Once the model coefficient has been
estimated, we now have a formula

80
00:05:31,430 --> 00:05:35,020
that can use the result logistic function
to estimate the probability that

81
00:05:35,020 --> 00:05:39,030
any given student will pass the exam,
given the number of hours they've studied.

82
00:05:40,300 --> 00:05:43,840
Students who estimated probability of
passing the exam is greater than or

83
00:05:43,840 --> 00:05:47,830
equal to 50% are predicted
to be in the positive class.

84
00:05:47,830 --> 00:05:50,310
Otherwise, they're predicted
to be in the negative class.

85
00:05:51,720 --> 00:05:53,980
So in this example we can see
that students who study for

86
00:05:53,980 --> 00:05:58,260
more than three hours will be
predicted to be in the positive class.

87
00:05:58,260 --> 00:06:02,150
Let's look at an example that uses two
input features now, instead of one.

88
00:06:03,510 --> 00:06:06,820
Here the plots show a training
set with two classes.

89
00:06:06,820 --> 00:06:08,940
Each data point has two features.

90
00:06:08,940 --> 00:06:14,780
Feature 1 corresponds to the x-axis, and
Feature 2 corresponds to the y-axis.

91
00:06:14,780 --> 00:06:17,491
The data points in the red
class on the left,

92
00:06:17,491 --> 00:06:21,758
form a cluster with low Feature 1 value,
and high Feature 2 value.

93
00:06:21,758 --> 00:06:25,262
And the points in the blue class have
intermediate Feature 1 value, and

94
00:06:25,262 --> 00:06:26,359
low Feature 2 value.

95
00:06:27,600 --> 00:06:31,300
We can apply logistic regression
to learn a binary classifier

96
00:06:31,300 --> 00:06:35,850
using this training set, using the same
idea we saw in the previous exam example.

97
00:06:37,620 --> 00:06:42,800
To do this, we'll add a third dimension
shown here as the vertical y-axis.

98
00:06:42,800 --> 00:06:45,869
Corresponding to the probability of
belonging to the positive class.

99
00:06:47,250 --> 00:06:50,440
We'll say that the red points
are associated with the negative class and

100
00:06:50,440 --> 00:06:54,800
have a target value of 0, and the blue
points are associated with the positive

101
00:06:54,800 --> 00:06:56,835
class and have a target value of 1.

102
00:06:58,440 --> 00:07:03,260
Then just as we did in the exam studying
example, we can estimate the w hat and

103
00:07:03,260 --> 00:07:07,940
b hat parameters of the logistic function
that best fits this training data.

104
00:07:07,940 --> 00:07:12,120
The only difference is that the logistic
function is now a function of two input

105
00:07:12,120 --> 00:07:14,230
features and not just one.

106
00:07:14,230 --> 00:07:18,640
So it forms something like a three
dimensional S shaped sheet in this space.

107
00:07:20,420 --> 00:07:24,010
Once this logistic function has been
estimated from the training data,

108
00:07:24,010 --> 00:07:26,740
we can use it to predict the class
membership for any point,

109
00:07:26,740 --> 00:07:31,119
given its Feature 1 and Feature 2 values,
same way we did for the exam example.

110
00:07:32,330 --> 00:07:36,760
Any data instances whose logistic
probability estimate y hat is greater than

111
00:07:36,760 --> 00:07:40,510
or equal to 0.5 are predicted to
be in the positive blue class,

112
00:07:42,280 --> 00:07:44,780
otherwise, in the other red class.

113
00:07:46,290 --> 00:07:49,580
Now if we imagine that there's
a plane representing y equals 0.5,

114
00:07:49,580 --> 00:07:53,800
as shown here,
that intersects this logistic function.

115
00:07:53,800 --> 00:08:00,630
It turns out that all the points that
have a value of y = 0.5, when you

116
00:08:00,630 --> 00:08:04,170
intersect that with a logistic function,
the points all lie along a straight line.

117
00:08:05,940 --> 00:08:10,290
In other words, using logistic regression
gives a linear decision boundary between

118
00:08:10,290 --> 00:08:11,600
the classes as shown here.

119
00:08:13,190 --> 00:08:17,580
If you imagine looking straight down on
the 3D logistic function on the left,

120
00:08:19,020 --> 00:08:21,760
you get the view that looks something
like something on the right.

121
00:08:21,760 --> 00:08:22,260
Here.

122
00:08:23,610 --> 00:08:29,160
The points with y greater or equal to 0.5
on the logistic function, lie in a region

123
00:08:29,160 --> 00:08:33,820
to the right of the straight line,
which is the dash line on the right here.

124
00:08:33,820 --> 00:08:38,800
And the points with y less than
0.5 on the logistic function

125
00:08:38,800 --> 00:08:42,747
would form a region to
the left of that dash line.

126
00:08:42,747 --> 00:08:48,290
Let's look at an example with
real data in Scikit-Learn.

127
00:08:48,290 --> 00:08:51,372
To perform logistic,
regression in Scikit-Learn,

128
00:08:51,372 --> 00:08:56,062
you import the logistic regression class
from the sklearn.linear model module,

129
00:08:56,062 --> 00:09:00,551
then create the object and call the fit
method using the training data just as you

130
00:09:00,551 --> 00:09:03,514
did for other class files
like k nearest neighbors.

131
00:09:05,190 --> 00:09:09,310
Here, the code also sets a parameter c
to 100, which we'll explain in a minute.

132
00:09:11,200 --> 00:09:15,350
The data set we're using here is
a modified form of our fruits data set,

133
00:09:15,350 --> 00:09:19,930
using only height and width as
the features, the features space, and

134
00:09:19,930 --> 00:09:25,290
with the target class value modified
into a binary classification problem

135
00:09:25,290 --> 00:09:30,040
predicting whether an object is an apple,
a positive class, or

136
00:09:30,040 --> 00:09:32,400
something other than an apple,
a negative class.

137
00:09:34,060 --> 00:09:36,092
Here is a graphical
display of the results.

138
00:09:36,092 --> 00:09:39,852
The x-axis corresponds to
the height feature and

139
00:09:39,852 --> 00:09:43,444
the y-axis corresponds
to the width feature.

140
00:09:43,444 --> 00:09:47,910
The black points represent the positive
apple class training points.

141
00:09:47,910 --> 00:09:51,030
And the yellow points are instances of
all the other fruits in the training set.

142
00:09:52,140 --> 00:09:56,420
The gray decision region represents
that area of the height and

143
00:09:56,420 --> 00:10:00,430
width feature space, where a fruit
would have an estimated probability

144
00:10:00,430 --> 00:10:02,570
greater than 0.5 of being an apple.

145
00:10:02,570 --> 00:10:06,880
And thus classified as an apple according
to the logistic regression function.

146
00:10:08,070 --> 00:10:13,380
The yellow decision region corresponds
to the region of feature space for

147
00:10:13,380 --> 00:10:17,530
objects that have an estimated probability
of less than 0.5 of being an apple.

148
00:10:18,730 --> 00:10:21,770
You can see the linear decision
boundary where the grey region

149
00:10:21,770 --> 00:10:25,350
meets the yellow region,
that results applying logistic regression.

150
00:10:26,610 --> 00:10:30,520
In fact, logistic regression results
are often quite similar to those you might

151
00:10:30,520 --> 00:10:33,420
obtain from a linear
support vector machine,

152
00:10:33,420 --> 00:10:36,470
another type of linear model
we explore for classification.

153
00:10:39,750 --> 00:10:44,474
Like ridge and lasso regression,
a regularization penalty on the model

154
00:10:44,474 --> 00:10:48,735
coefficients can also be applied
with logistic regression, and

155
00:10:48,735 --> 00:10:51,144
is controlled with the parameter C.

156
00:10:51,144 --> 00:10:55,872
In fact, the same L2
regularization penalty used for

157
00:10:55,872 --> 00:10:59,675
ridge regression is
turned on by default for

158
00:10:59,675 --> 00:11:04,011
logistic regression with
a default value C = 1.

159
00:11:04,011 --> 00:11:09,291
Note that for both Support Vector
machines and Logistic Regression,

160
00:11:09,291 --> 00:11:13,506
higher values of C correspond
to less regularization.

161
00:11:13,506 --> 00:11:14,679
With large values of C,

162
00:11:14,679 --> 00:11:18,870
logistic regression tries to fit
the training data as well as possible.

163
00:11:18,870 --> 00:11:24,013
While with small values of C, the model
tries harder to find model coefficients

164
00:11:24,013 --> 00:11:29,399
that are closer to 0, even if that model
fits the training data a little bit worse.

165
00:11:29,399 --> 00:11:32,896
You can see the effect of changing
the regularization parameter C for

166
00:11:32,896 --> 00:11:34,830
logistic regression in this visual.

167
00:11:36,620 --> 00:11:42,130
Using the same upper class of fire we
now vary C to take on values from 0.1 on

168
00:11:42,130 --> 00:11:49,160
the left to 1.0 in the middle,
and 100.0 on the right.

169
00:11:49,160 --> 00:11:53,619
Although the real power of
regularization doesn't become evident

170
00:11:53,619 --> 00:11:57,852
until we have data that has higher
dimensional feature spaces.

171
00:11:57,852 --> 00:12:02,277
You can get an idea of the trade off
that's happening between relying on

172
00:12:02,277 --> 00:12:07,144
a simpler model, one that puts more
emphasis on a single feature in this case,

173
00:12:07,144 --> 00:12:11,076
out of the two features, but
has lower training set accuracy.

174
00:12:11,076 --> 00:12:17,260
And that's an example as shown
on the left with C = 0.1.

175
00:12:17,260 --> 00:12:22,484
Or, better training data fit
on the right with C = 100.

176
00:12:22,484 --> 00:12:25,890
You can find the code that created this
example in the accompanying notebook.

177
00:12:28,550 --> 00:12:33,970
Finally, we show how logistic regression
again, with L2 regularization turned

178
00:12:33,970 --> 00:12:39,580
on by default, can be applied to a real
data set with many more features.

179
00:12:39,580 --> 00:12:40,900
The breast cancer data set.

180
00:12:42,010 --> 00:12:49,344
Here, logistic regression achieves both
training, and test set accuracy of 96%