1
00:00:07,796 --> 00:00:09,187
In this part of the course,

2
00:00:09,187 --> 00:00:12,394
you'll get an introduction to
the basics of neural networks.

3
00:00:12,394 --> 00:00:16,118
Which are a broad family of algorithms
that have formed the basis for

4
00:00:16,118 --> 00:00:20,177
the recent resurgence in the computational
field called deep learning.

5
00:00:20,177 --> 00:00:24,380
Early work on neural networks
actually began in the 1950s and 60s.

6
00:00:24,380 --> 00:00:27,285
And just recently,
has experienced a resurgence of interest,

7
00:00:27,285 --> 00:00:30,696
as deep learning has achieved
impressive state-of-the-art results.

8
00:00:30,696 --> 00:00:35,460
On specific tasks that range from
object classification in images,

9
00:00:35,460 --> 00:00:39,249
to fast accurate machine translation,
to gameplay.

10
00:00:40,730 --> 00:00:43,003
The topic of neural networks
requires its own course.

11
00:00:43,003 --> 00:00:45,203
And indeed,
if you're interested in more depth,

12
00:00:45,203 --> 00:00:47,465
you can check out the excellent
course on Coursera.

13
00:00:47,465 --> 00:00:51,247
Called Neural Networks for Machine
Learning, by a pioneer in this area,

14
00:00:51,247 --> 00:00:52,530
Professor Jeff Hinton.

15
00:00:53,860 --> 00:00:57,320
Here, we'll provide an introduction
to the basic concepts and

16
00:00:57,320 --> 00:01:00,580
algorithms that are foundation
of neural networks.

17
00:01:00,580 --> 00:01:03,814
And of the much more sophisticated
deep learning methods in use today.

18
00:01:03,814 --> 00:01:09,130
You'll learn about some basic models
called multi-layer perceptrons,

19
00:01:09,130 --> 00:01:13,160
supported by scikit-learn, that can be
used for classification and regression.

20
00:01:15,120 --> 00:01:18,720
Let's start by briefly reviewing simpler
methods we have already seen for

21
00:01:18,720 --> 00:01:21,110
regression and classification.

22
00:01:21,110 --> 00:01:25,239
Linear regression and logistic regression,
which we show graphically here.

23
00:01:26,870 --> 00:01:29,610
Linear regression predicts
a continuous output,

24
00:01:29,610 --> 00:01:32,450
y hat, shown as the box on the right.

25
00:01:32,450 --> 00:01:36,240
As a function as the sum
of the input variables xi,

26
00:01:36,240 --> 00:01:37,960
shown in the boxes on the left.

27
00:01:39,220 --> 00:01:43,991
Each weighted by
a corresponding coefficient,

28
00:01:43,991 --> 00:01:48,654
wi hat, plus an intercept or
bias term, b hat.

29
00:01:48,654 --> 00:01:52,031
We saw how various methods
like ordinary least squares,

30
00:01:52,031 --> 00:01:54,433
ridge regression or lasso regression.

31
00:01:54,433 --> 00:01:59,044
Could be used to estimate these
model coefficients, wi hat and

32
00:01:59,044 --> 00:02:04,004
b hat, shown above the arrows in
the diagram, from training data.

33
00:02:05,497 --> 00:02:08,135
Logistic regression takes
this one step further,

34
00:02:08,135 --> 00:02:12,670
by running the output of the linear
function of the input variables, xi.

35
00:02:12,670 --> 00:02:16,930
Through an additional nonlinear function,
the logistic function.

36
00:02:16,930 --> 00:02:21,250
Represented by the new box in the middle
of the diagram, to produce the output, y.

37
00:02:22,260 --> 00:02:24,100
Which, because of the logistic function,

38
00:02:24,100 --> 00:02:26,550
is now constrained to lie between zero and
one.

39
00:02:28,020 --> 00:02:31,070
We use logistical regression for
binary classification.

40
00:02:31,070 --> 00:02:33,650
Since we can interpret
y as the probability

41
00:02:33,650 --> 00:02:37,130
that a given input data instance
belongs to the positive class,

42
00:02:37,130 --> 00:02:39,580
in a two-class binary
classification scenario.

43
00:02:42,260 --> 00:02:44,200
Here's an example of
a simple neural network for

44
00:02:44,200 --> 00:02:47,150
regression, called
a multi-layer perceptron.

45
00:02:47,150 --> 00:02:49,430
Which I will sometimes abbreviate by MLP.

46
00:02:49,430 --> 00:02:53,710
These are also known as
feed-forward neural networks.

47
00:02:55,050 --> 00:02:59,200
MLPs take this idea of computing
weighted sums of the input features,

48
00:02:59,200 --> 00:03:01,280
like we saw in logistic regression.

49
00:03:01,280 --> 00:03:04,227
But it takes it a step
beyond logistic regression,

50
00:03:04,227 --> 00:03:07,938
by adding an additional processing
step called a hidden layer.

51
00:03:07,938 --> 00:03:14,239
Represented by this additional set of
boxes, h0, h1, and h2 in the diagram.

52
00:03:14,239 --> 00:03:18,411
These boxes, within the hidden layer,
are called hidden units.

53
00:03:18,411 --> 00:03:22,282
And each hidden unit in the hidden
layer computes a nonlinear function of

54
00:03:22,282 --> 00:03:24,509
the weighted sums of the input features.

55
00:03:24,509 --> 00:03:28,330
Resulting in intermediate output values,
v0, v1, v2.

56
00:03:30,210 --> 00:03:34,047
Then the MLP computes a weighted
sum of these hidden unit outputs,

57
00:03:34,047 --> 00:03:36,318
to form the final output value, Y hat.

58
00:03:38,931 --> 00:03:41,850
This nonlinear function that
the hidden unit applies.

59
00:03:41,850 --> 00:03:44,098
is called the activation function.

60
00:03:44,098 --> 00:03:49,321
In this example, your activation function
is the hyperbolic tangent function,

61
00:03:49,321 --> 00:03:52,129
which is related to the logistic function.

62
00:03:52,129 --> 00:03:56,709
You can see that the result of adding this
additional hidden layer processing step to

63
00:03:56,709 --> 00:03:59,237
the prediction model,
is a formula for y hat.

64
00:03:59,237 --> 00:04:04,354
That is already more involved than
the one for logistic regression.

65
00:04:04,354 --> 00:04:08,783
Now predicting y involves computing
a different initial weighted sum of

66
00:04:08,783 --> 00:04:11,694
the input feature values for
each hidden unit.

67
00:04:11,694 --> 00:04:14,337
Which applies a nonlinear
activation function.

68
00:04:14,337 --> 00:04:19,260
And then all of these nonlinear outputs
are combined, using another weighted sum,

69
00:04:19,260 --> 00:04:20,100
to produce y.

70
00:04:21,830 --> 00:04:25,850
In particular, there's one weight
between each input and each hidden unit.

71
00:04:27,050 --> 00:04:31,606
And one weight between each hidden
unit and the output variable.

72
00:04:31,606 --> 00:04:36,257
In fact, this addition and combination
of non-linear activation functions.

73
00:04:36,257 --> 00:04:40,072
Allows multi-layer perceptrons
to learn more complex functions.

74
00:04:40,072 --> 00:04:43,515
Than is possible with a simple linear or
logistic function.

75
00:04:43,515 --> 00:04:47,892
This additional expressive power enables
neural networks to perform more accurate

76
00:04:47,892 --> 00:04:48,639
prediction.

77
00:04:48,639 --> 00:04:52,220
When the relationship between the input
and output is itself complex.

78
00:04:53,280 --> 00:04:56,747
Of course, this complexity also means
that there are a lot more weights,

79
00:04:56,747 --> 00:05:00,230
model coefficients,
to estimate in the training phase.

80
00:05:00,230 --> 00:05:04,470
Which means that both more training data
and more computation are typically needed

81
00:05:04,470 --> 00:05:07,119
to learn in a neural network,
compared to a linear model.

82
00:05:08,850 --> 00:05:13,030
As an aside, there are a number of choices
for the activation function in a neural

83
00:05:13,030 --> 00:05:15,640
network, that gets
applied in hidden units.

84
00:05:15,640 --> 00:05:19,600
Here, the plot shows the input value
coming into the activation function,

85
00:05:19,600 --> 00:05:22,090
from the previous layer's
inputs on the x-axis.

86
00:05:22,090 --> 00:05:26,880
And the y-axis shows the resulting
output value for the function.

87
00:05:26,880 --> 00:05:32,129
This code to plot this example is
available in the accompanying notebook.

88
00:05:32,129 --> 00:05:35,787
The three main activation functions
we'll compare later in this lecture

89
00:05:35,787 --> 00:05:37,265
are the hyperbolic tangent.

90
00:05:37,265 --> 00:05:40,000
That's the S-shaped function in green.

91
00:05:40,000 --> 00:05:44,200
The rectified linear unit function,
which I'll abbreviate to relu,

92
00:05:44,200 --> 00:05:47,080
shown as the piecewise
linear function in blue.

93
00:05:48,330 --> 00:05:51,471
And the familiar logistic function,
which is shown in red.

94
00:05:51,471 --> 00:05:56,234
The relu activation function is
the default activation function for

95
00:05:56,234 --> 00:05:58,800
neural networks in scikit-learn.

96
00:05:58,800 --> 00:06:02,856
It maps any negative input values to zero.

97
00:06:02,856 --> 00:06:06,072
The hyperbolic tangent function,
or tanh function.

98
00:06:06,072 --> 00:06:10,445
Maps large positive input values
to outputs very close to one.

99
00:06:10,445 --> 00:06:16,061
And large negative input values,
to outputs very close to negative one.

100
00:06:16,061 --> 00:06:19,934
These differences in the activation
function can have some effect on the shape

101
00:06:19,934 --> 00:06:21,644
of regression prediction plots.

102
00:06:21,644 --> 00:06:26,262
Or classification decision boundaries
that neural networks learn.

103
00:06:26,262 --> 00:06:30,150
In general, we'll be using
either the hyperbolic tangent or

104
00:06:30,150 --> 00:06:33,675
the relu function as our
default activation function.

105
00:06:33,675 --> 00:06:36,020
Since these perform well for
most applications.

106
00:06:38,120 --> 00:06:41,633
Let's take a look at how we use
neural networks in scikit-learn for

107
00:06:41,633 --> 00:06:42,710
classification.

108
00:06:42,710 --> 00:06:46,060
Using the more complex synthetic
binary classification data set.

109
00:06:47,850 --> 00:06:50,351
To use a neural network classifier,

110
00:06:50,351 --> 00:06:55,932
you import the MLPClassifier class from
the sklearn.neural_network module.

111
00:06:55,932 --> 00:06:59,902
This code example shows the classifier
being fit to the training data,

112
00:06:59,902 --> 00:07:01,565
using a single hidden layer.

113
00:07:01,565 --> 00:07:06,352
With three different numbers
of hidden units in the layer,

114
00:07:06,352 --> 00:07:09,520
1 unit, 10 units and 100 units.

115
00:07:09,520 --> 00:07:12,271
As with all other classification
types we've seen,

116
00:07:12,271 --> 00:07:16,071
you can create the classifier objects
with the appropriate parameters.

117
00:07:16,071 --> 00:07:17,830
And call the fit method
on the training data.

118
00:07:20,050 --> 00:07:21,220
Here, the main parameter for

119
00:07:21,220 --> 00:07:25,075
a neural network classifier is this
parameter, hidden_layer_sizes.

120
00:07:26,190 --> 00:07:30,306
This parameter is a list,
with one element for each hidden layer,

121
00:07:30,306 --> 00:07:33,997
that gives the number of hidden
units to use for that layer.

122
00:07:33,997 --> 00:07:36,993
So here we're passing a list
with a single element.

123
00:07:36,993 --> 00:07:41,840
Meaning we want one hidden layer, using
the number in the variable called units.

124
00:07:42,950 --> 00:07:47,578
By default, if you don't specify
the hidden_layer_sizes parameter,

125
00:07:47,578 --> 00:07:52,287
scikit-learn will create a single
hidden layer with 100 hidden units.

126
00:07:52,287 --> 00:07:55,647
While a setting of 10 may work well for
simple data sets,

127
00:07:55,647 --> 00:07:57,785
like the one we use as examples here.

128
00:07:57,785 --> 00:08:02,988
For really complex data sets, the number
of hidden units could be in the thousands.

129
00:08:02,988 --> 00:08:05,415
It's also possible, as we'll see shortly,

130
00:08:05,415 --> 00:08:07,977
to create an MLP with more
than one hidden layer.

131
00:08:07,977 --> 00:08:12,461
By passing a hidden_layer_sizes
parameter with multiple entries.

132
00:08:13,981 --> 00:08:18,099
I want to also note the use of this
extra parameter, called solver.

133
00:08:18,099 --> 00:08:21,500
Which specifies the algorithm to use for
learning the weights of the network.

134
00:08:22,540 --> 00:08:26,140
Here, we're using the lbfgs algorithm.

135
00:08:26,140 --> 00:08:29,549
We'll discuss the solver parameter setting
further, at the end of this lecture.

136
00:08:31,060 --> 00:08:33,870
Also note that we're passing
in a random_state parameter,

137
00:08:33,870 --> 00:08:36,860
when creating the MLPClassifier object.

138
00:08:36,860 --> 00:08:39,540
Like we did for
the train-test split function.

139
00:08:39,540 --> 00:08:43,550
And we happened to set this random state
parameter to a fixed value of zero.

140
00:08:44,690 --> 00:08:48,670
This is because for neural networks,
their weights are initialized randomly,

141
00:08:48,670 --> 00:08:50,389
which can affect the model
that is learned.

142
00:08:51,560 --> 00:08:55,480
Because of this, even without changing
the key parameters on the same data set.

143
00:08:55,480 --> 00:08:59,150
The same neural network algorithm
might learn two different models.

144
00:08:59,150 --> 00:09:03,575
Depending on the value of the internal
random seed that is chosen.

145
00:09:03,575 --> 00:09:05,623
So by always setting the same value for

146
00:09:05,623 --> 00:09:08,250
the random seed used to
initialize the weights.

147
00:09:08,250 --> 00:09:10,746
We can assure the results
will always be the same, for

148
00:09:10,746 --> 00:09:12,309
everyone using these examples.

149
00:09:14,718 --> 00:09:17,181
This graphic plots the results
of running this code.

150
00:09:17,181 --> 00:09:21,464
To show how the number of hidden
units in a single layer in the neural

151
00:09:21,464 --> 00:09:25,383
network affects the model complexity for
classification.

152
00:09:25,383 --> 00:09:26,964
With a single hidden unit,

153
00:09:26,964 --> 00:09:30,821
the model is mathematically
equivalent to logistic regression.

154
00:09:30,821 --> 00:09:34,883
We see the classifier returns the familiar
simple linear decision boundary between

155
00:09:34,883 --> 00:09:35,750
the two classes.

156
00:09:36,900 --> 00:09:39,930
The training set score's low, and
the test score is not much better, so

157
00:09:39,930 --> 00:09:43,490
this network model is under-fitting.

158
00:09:43,490 --> 00:09:47,714
With ten hidden units, we can see that
the MLPClassifier is able to learn a more

159
00:09:47,714 --> 00:09:49,379
complete decision boundary.

160
00:09:49,379 --> 00:09:54,044
That captures more of the nonlinear,
cluster-oriented structure in the data,

161
00:09:54,044 --> 00:09:56,484
though the test set accuracy is still low.

162
00:09:57,590 --> 00:10:01,890
With 100 hidden units, the decision
boundary is even more detailed.

163
00:10:01,890 --> 00:10:05,430
And achieves much better accuracy,
on both the training and the test sets.

164
00:10:06,570 --> 00:10:10,110
Here's a graphical depiction of
a multi-layer perceptron with two

165
00:10:10,110 --> 00:10:10,889
hidden layers.

166
00:10:12,000 --> 00:10:15,740
Adding the second hidden layer further
increases the complexity of functions that

167
00:10:15,740 --> 00:10:18,430
the neural network can learn,
from more complex data sets.

168
00:10:19,700 --> 00:10:23,610
Taking this complexity further,
large architectures of neural networks,

169
00:10:23,610 --> 00:10:28,310
with many stages of computation, are why
deep learning methods are called deep.

170
00:10:28,310 --> 00:10:30,968
And we'll summarize deep learning,
in an upcoming lecture for this week.

171
00:10:34,085 --> 00:10:38,965
Here is an example in the notebook,
showing how we create a two-layer MLP,

172
00:10:38,965 --> 00:10:41,419
with 10 hidden units in each layer.

173
00:10:41,419 --> 00:10:44,500
We just set
the hidden_layer_sizes parameter,

174
00:10:44,500 --> 00:10:48,161
when creating the MLPClassifier,
to a two-element list.

175
00:10:48,161 --> 00:10:51,139
Indicating ten units,
in each of the two hidden layers.

176
00:10:52,950 --> 00:10:55,450
You can see the result of of
adding the second hidden layer,

177
00:10:55,450 --> 00:10:57,450
on the classification
problem we saw earlier.

178
00:10:58,570 --> 00:11:03,070
On the left is the original MLP,
with one hidden layer of ten units.

179
00:11:03,070 --> 00:11:04,960
And on the right is the same data set,

180
00:11:04,960 --> 00:11:09,630
using a new MLP with two hidden
layers of ten units each.

181
00:11:09,630 --> 00:11:14,230
You can see the MLP with two hidden layers
learned a more complex decision boundary.

182
00:11:14,230 --> 00:11:17,080
And achieved, in this case,
a much better fit on the training data,

183
00:11:17,080 --> 00:11:19,870
and slightly better
accuracy on the test data.

184
00:11:22,070 --> 00:11:25,610
Once we start adding more hidden layers,
with lots of hidden units.

185
00:11:25,610 --> 00:11:29,640
You can see that the number of weights,
or model coefficients, to estimate for

186
00:11:29,640 --> 00:11:31,320
a neural network can increase rapidly.

187
00:11:32,480 --> 00:11:35,430
So that more complex neural networks
could have many thousands of weights

188
00:11:35,430 --> 00:11:35,970
to estimate.

189
00:11:37,470 --> 00:11:42,210
We can control this model complexity, just
as we did with ridge and lasso regression.

190
00:11:42,210 --> 00:11:46,060
By adding an L2 regularization
penalty on the weights.

191
00:11:46,060 --> 00:11:50,810
Remember that L2 regularization penalizes
models that have a large sum of squares of

192
00:11:50,810 --> 00:11:52,730
all the weight values.

193
00:11:52,730 --> 00:11:53,836
With the effect being,

194
00:11:53,836 --> 00:11:57,556
that the neural network prefers models
with more weights shrunk close to zero.

195
00:12:00,025 --> 00:12:03,577
The regularization parameter for
MLPs is called alpha,

196
00:12:03,577 --> 00:12:06,194
like with the linear regression models.

197
00:12:06,194 --> 00:12:09,841
And in scikit-learn,
it's set to a small value by default,

198
00:12:09,841 --> 00:12:13,710
like 0.0001,
that gives a little bit of regularization.

199
00:12:15,280 --> 00:12:17,970
This code example shows
the effects of changing alpha for

200
00:12:17,970 --> 00:12:22,320
a larger MLP,
with 2 hidden layers of 100 nodes each.

201
00:12:22,320 --> 00:12:25,400
From a small value of 0.01,
to a larger value of 5.0.

202
00:12:25,400 --> 00:12:28,410
For variety here,

203
00:12:28,410 --> 00:12:32,940
we're also setting the activation function
to use the hyperbolic tangent function.

204
00:12:35,290 --> 00:12:38,090
Here's the graphical output
of this notebook code.

205
00:12:38,090 --> 00:12:41,460
You can see the effect of increasing
regularization with increasing alpha.

206
00:12:42,580 --> 00:12:45,160
In the left plot, when alpha is small,

207
00:12:45,160 --> 00:12:47,956
the decision boundaries are much
more complex and variable.

208
00:12:47,956 --> 00:12:50,671
And the classifier's over-fitting,

209
00:12:50,671 --> 00:12:55,475
as we can see from the very high
training set score, and low test score.

210
00:12:55,475 --> 00:13:01,178
On the other hand, the right plot uses the
largest value of alpha here, alpha 5.0.

211
00:13:01,178 --> 00:13:04,213
And that setting results in much
smoother decision boundaries,

212
00:13:04,213 --> 00:13:06,870
while still capturing the global
structure of the data.

213
00:13:06,870 --> 00:13:11,350
And this increased simplicity allows
it to generalize much better, and

214
00:13:11,350 --> 00:13:13,520
not over-fit to the training set.

215
00:13:13,520 --> 00:13:16,490
And this is evident from the much
higher test score, in this case.

216
00:13:18,640 --> 00:13:22,660
As with other supervised learning models,
like regularized regression and

217
00:13:22,660 --> 00:13:24,500
support vector machines.

218
00:13:24,500 --> 00:13:25,320
It can be critical,

219
00:13:25,320 --> 00:13:28,880
when using neural networks,
to properly normalize the input features.

220
00:13:30,930 --> 00:13:35,120
Let's apply the multi-layer perceptron
to the breast cancer data set.

221
00:13:35,120 --> 00:13:38,040
And notice that we first
apply the MinMaxScaler,

222
00:13:38,040 --> 00:13:39,910
to pre-process the input features.

223
00:13:40,930 --> 00:13:46,980
Here we'll combine a more complex network,
using 2 hidden layers with 100 units each.

224
00:13:46,980 --> 00:13:51,167
With a higher regularization
setting of alpha at 5.0, and

225
00:13:51,167 --> 00:13:53,270
using the lgbfs solver again.

226
00:13:54,800 --> 00:13:58,070
You can see, that with this multi-layer
perceptron, both the training and

227
00:13:58,070 --> 00:14:01,660
test set accuracy are among the highest
we have obtained on this data set.

228
00:14:03,070 --> 00:14:06,120
Like many of the other supervised
learning methods we've seen,

229
00:14:06,120 --> 00:14:10,400
you can also use multi-layer perceptrons
for regression, as well as classification.

230
00:14:11,940 --> 00:14:15,971
We're including MLP regression here,
as an example, for two reasons.

231
00:14:15,971 --> 00:14:18,830
First, because MLP
regression may be useful for

232
00:14:18,830 --> 00:14:21,069
some regression problems on its own.

233
00:14:21,069 --> 00:14:24,620
But more generally, because some deep
learning problems are regression problems.

234
00:14:24,620 --> 00:14:29,530
And so, as with classification, using
multi-layer perceptrons is a good starting

235
00:14:29,530 --> 00:14:32,670
point to learn about the more
complex architectures used for

236
00:14:32,670 --> 00:14:33,770
regression in deep learning.

237
00:14:35,125 --> 00:14:38,530
Here's the example of a simple MLP
regression model, in our notebook.

238
00:14:39,610 --> 00:14:43,900
You use the multi-layer
perceptron regressor by importing

239
00:14:43,900 --> 00:14:48,864
the MLPRegressor class from
the sklearn.neural_network module,

240
00:14:48,864 --> 00:14:52,072
and then creating the MLPRegressor object.

241
00:14:52,072 --> 00:14:55,153
When creating the object here, we're
setting the number of hidden layers and

242
00:14:55,153 --> 00:14:56,472
units within each hidden layer.

243
00:14:56,472 --> 00:15:00,720
Using the same hidden_layer_sizes
parameter that we used for classification.

244
00:15:01,760 --> 00:15:06,800
This example uses two hidden layers,
with 100 hidden nodes each.

245
00:15:06,800 --> 00:15:10,359
This notebook code has a loop that
cycles through different settings of

246
00:15:10,359 --> 00:15:14,765
the activation function parameter, and
the alpha parameter for L2 regularization.

247
00:15:16,985 --> 00:15:20,762
Here we've included regression
results that use, in the top row,

248
00:15:20,762 --> 00:15:24,290
the hyperbolic tangent
activation function.

249
00:15:24,290 --> 00:15:27,390
And in the bottom row,
the relu activation function.

250
00:15:29,340 --> 00:15:32,440
You can see the smoothness of the
activation function somewhat influences

251
00:15:32,440 --> 00:15:34,970
the smoothness of the corresponding
regression results.

252
00:15:36,470 --> 00:15:40,640
Along the columns, the plots also show the
effect of using different alpha settings,

253
00:15:40,640 --> 00:15:43,869
to increase the amount of L2
regularization from left to right.

254
00:15:44,890 --> 00:15:47,070
Again, as with classification,

255
00:15:47,070 --> 00:15:51,640
the effect of increasing the amount of
L2 regularization, by increasing alpha.

256
00:15:51,640 --> 00:15:55,690
Is to constrain the regression to use
simpler and simpler models, with fewer and

257
00:15:55,690 --> 00:15:56,810
fewer large weights.

258
00:15:57,820 --> 00:16:00,820
You can see this effect for
both activation functions, in the top and

259
00:16:00,820 --> 00:16:02,440
bottom rows.

260
00:16:02,440 --> 00:16:06,118
The regression line on the left has
higher variance than the much smoother,

261
00:16:06,118 --> 00:16:07,735
regularized model on the right.

262
00:16:10,316 --> 00:16:14,800
On the positive side, beyond these
simple examples we've shown here.

263
00:16:14,800 --> 00:16:18,101
Neural networks form the basis of
advanced learning architectures.

264
00:16:18,101 --> 00:16:21,786
That capture complex features, and
give state-of-the-art performance on

265
00:16:21,786 --> 00:16:24,593
an increasingly wide variety
of difficult learning tasks.

266
00:16:24,593 --> 00:16:28,527
From world-championship play for
the game of Go, to detailed and

267
00:16:28,527 --> 00:16:32,015
robust recognition of objects and images.

268
00:16:32,015 --> 00:16:35,540
However, with this increased power,
come increased costs.

269
00:16:35,540 --> 00:16:39,450
This larger and more complex models
typically require significant volumes of

270
00:16:39,450 --> 00:16:42,060
data, computation, and
training time to learn.

271
00:16:43,340 --> 00:16:48,754
In addition, careful pre-processing of the
input data is needed, to help ensure fast,

272
00:16:48,754 --> 00:16:53,043
stable, meaningful solutions to
finding the optimal set of weights.

273
00:16:53,043 --> 00:16:55,411
In general,
neural networks are a good choice,

274
00:16:55,411 --> 00:16:57,392
when the features are of similar types.

275
00:16:57,392 --> 00:17:01,050
For example,
all derived from the pixels of an image.

276
00:17:01,050 --> 00:17:05,658
And less of a good choice, when
the features are of very different types.

277
00:17:05,658 --> 00:17:09,544
Finally, let's review the key parameters
for the multi-layer perceptron in

278
00:17:09,544 --> 00:17:12,686
scikit-learn, that can be used
to control model complexity.

279
00:17:14,972 --> 00:17:17,931
The main way to control model
complexity for the MLP,

280
00:17:17,931 --> 00:17:21,618
is to control the hidden unit size and
structure.

281
00:17:21,618 --> 00:17:25,500
Using the hidden_layers_sizes parameter
that controls the number of hidden layers,

282
00:17:25,500 --> 00:17:27,300
and the number of units within each layer.

283
00:17:28,740 --> 00:17:31,950
Alpha controls the amount of
regularization that helps constrain

284
00:17:31,950 --> 00:17:33,610
the complexity of the model,

285
00:17:33,610 --> 00:17:35,937
by constraining the magnitude
of model weights.

286
00:17:37,030 --> 00:17:40,190
Finally, you can experiment with at
least three different choices for

287
00:17:40,190 --> 00:17:43,510
the nonlinear activation function,
by using the activation parameter.

288
00:17:44,560 --> 00:17:46,820
Earlier, we saw the solver parameter, for

289
00:17:46,820 --> 00:17:49,420
specifying the algorithm that
learns the network weights.

290
00:17:50,490 --> 00:17:54,553
Solver is the algorithm that actually
does the numerical work of finding

291
00:17:54,553 --> 00:17:55,826
the optimal weights.

292
00:17:55,826 --> 00:17:58,283
And one intuitive way of
visualizing this process.

293
00:17:58,283 --> 00:18:03,071
Is that all of the solver algorithms
have to do a kind of hill-climbing in

294
00:18:03,071 --> 00:18:06,560
a very bumpy landscape,
with lots of local minima.

295
00:18:06,560 --> 00:18:11,130
Where each local minimum corresponds
to a locally optimal set of weights.

296
00:18:11,130 --> 00:18:16,050
That is, a choice of weight setting that's
better than any nearby choices of weights.

297
00:18:16,050 --> 00:18:20,181
So across this whole landscape
of very bumpy local minima.

298
00:18:20,181 --> 00:18:24,530
Some will have higher validation scores on
the test data, and some will have lower.

299
00:18:24,530 --> 00:18:28,800
So depending on the initial random
initialization of the weights.

300
00:18:28,800 --> 00:18:31,420
And the nature of the trajectory
in the search path

301
00:18:31,420 --> 00:18:34,220
that a solver takes through
this bumpy landscape.

302
00:18:34,220 --> 00:18:37,390
The solver can end up at
different local minima,

303
00:18:37,390 --> 00:18:40,210
which can have different
validation scores.

304
00:18:40,210 --> 00:18:44,300
The default solver, adam,
tends to be both efficient and

305
00:18:44,300 --> 00:18:48,210
effective on large data sets,
with thousands of training examples.

306
00:18:48,210 --> 00:18:51,490
For small data sets, like many of
the ones we use in these examples,

307
00:18:51,490 --> 00:18:56,560
the lbfgs solver tends to be faster,
and find more effective weights.

308
00:18:56,560 --> 00:19:00,855
You can find further details on these more
advanced settings in the documentation for

309
00:19:00,855 --> 00:19:01,580
scikit-learn.