1
00:00:08,120 --> 00:00:12,986
Another tree based ensemble method that's
gain wide use in real world application is

2
00:00:12,986 --> 00:00:15,025
gradient boosted decision trees.

3
00:00:16,080 --> 00:00:20,220
Like random forest, gradient boosted trees
used an ensemble of multiple tress to

4
00:00:20,220 --> 00:00:24,180
create more powerful prediction models for
classification and regression.

5
00:00:25,300 --> 00:00:25,900
In this lecture,

6
00:00:25,900 --> 00:00:29,480
we'll provide a brief overview of
gradient boosted decision trees,

7
00:00:29,480 --> 00:00:32,770
along with the discussion of their key
parameters, the control model complexity.

8
00:00:34,090 --> 00:00:36,220
Unlike the random forest
method that builds and

9
00:00:36,220 --> 00:00:40,695
combines a forest of randomly different
trees in parallel, the key idea

10
00:00:40,695 --> 00:00:44,685
of gradient boosted decision trees is
that they build a series of trees.

11
00:00:44,685 --> 00:00:46,316
Where each tree is trained, so

12
00:00:46,316 --> 00:00:50,351
that it attempts to correct the mistakes
of the previous tree in the series.

13
00:00:51,825 --> 00:00:56,716
Typically, gradient boosted tree ensembles
use lots of shallow trees known in

14
00:00:56,716 --> 00:00:59,550
machine learning as weak learners.

15
00:00:59,550 --> 00:01:03,366
Built in a nonrandom way,
to create a model that makes fewer and

16
00:01:03,366 --> 00:01:05,822
fewer mistakes as more trees are added.

17
00:01:06,930 --> 00:01:10,355
Once the model is built,
making predictions with a gradient boosted

18
00:01:10,355 --> 00:01:12,963
tree models is fast and
doesn't use a lot of memory.

19
00:01:14,170 --> 00:01:17,532
Like random forests, the number of
estimators in the gradient boosted tree

20
00:01:17,532 --> 00:01:20,700
ensemble is an important parameter
in controlling model complexity.

21
00:01:21,920 --> 00:01:25,130
A new parameter that does not occur
with random forest is something called

22
00:01:25,130 --> 00:01:26,820
the learning rate.

23
00:01:26,820 --> 00:01:30,330
The learning rate controls how
the gradient boost the tree algorithms,

24
00:01:30,330 --> 00:01:32,250
builds a series of collective trees.

25
00:01:33,400 --> 00:01:36,290
When the learning rate is high,
each successive tree

26
00:01:36,290 --> 00:01:39,530
put strong emphases on correcting
the mistakes of its predecessor.

27
00:01:40,630 --> 00:01:43,472
And thus may result in a more
complex individual tree, and

28
00:01:43,472 --> 00:01:46,310
those overall are more complex model.

29
00:01:46,310 --> 00:01:50,133
With smaller settings of the learning
rate, there's less emphasis on thoroughly

30
00:01:50,133 --> 00:01:52,209
correcting the errors
of the previous step,

31
00:01:52,209 --> 00:01:54,520
which tends to lead to
simpler trees at each step.

32
00:01:57,330 --> 00:02:01,690
Here's an example showing how to use
gradient boosted trees in scikit-learn on

33
00:02:01,690 --> 00:02:06,322
our sample fruit classification test,
plotting the decision regions that result.

34
00:02:07,560 --> 00:02:10,935
The code is more or less the same
as what we used for random forests.

35
00:02:12,290 --> 00:02:14,652
But from the sklearn.ensemble module,

36
00:02:14,652 --> 00:02:17,625
we import
the GradientBoostingClassifier class.

37
00:02:18,690 --> 00:02:21,673
We then create
the GradientBoostingClassifier object, and

38
00:02:21,673 --> 00:02:24,780
fit it to the training
data in the usual way.

39
00:02:24,780 --> 00:02:30,644
By default, the learning rate parameter
is set to 0.1, the n_estimators parameter

40
00:02:30,644 --> 00:02:35,900
giving the number of trees to use is set
to 100, and the max depth is set to 3.

41
00:02:35,900 --> 00:02:39,517
As with random forests,
you can see the decision boundaries have

42
00:02:39,517 --> 00:02:44,399
that box-like shape that's characteristic
of decision trees or ensembles of trees.

43
00:02:46,640 --> 00:02:50,160
Now let's apply gradient boosted decision
trees to the breast cancer dataset.

44
00:02:51,640 --> 00:02:55,360
This code trains two different
gradient boosted classifiers.

45
00:02:55,360 --> 00:02:58,480
The first one uses the default settings.

46
00:02:58,480 --> 00:03:02,043
We can see that the first result has
perfect accuracy on the training set,

47
00:03:02,043 --> 00:03:04,530
which indicates the model
is likely overfitting.

48
00:03:05,950 --> 00:03:09,900
Two ways to learn a less complex
gradient boosted tree model are,

49
00:03:09,900 --> 00:03:13,800
to reduce the learning rate, so that each
tree doesn't try as hard to learn a more

50
00:03:13,800 --> 00:03:17,910
complex model, that fixes
the mistakes of its predecessor.

51
00:03:17,910 --> 00:03:22,295
And to reduce the max_depth parameter for
the individual trees in the ensemble.

52
00:03:23,740 --> 00:03:27,790
The second classifier example makes
these changes in the parameters.

53
00:03:27,790 --> 00:03:31,308
And you can see, that the training
set accuracy does decrease,

54
00:03:31,308 --> 00:03:34,056
while the test set accuracy
increases slightly.

55
00:03:36,360 --> 00:03:40,218
Gradient boosted decision trees are among
the best off-the-shelf supervised learning

56
00:03:40,218 --> 00:03:41,910
methods available.

57
00:03:41,910 --> 00:03:45,890
Achieving excellent accuracy with only
modest memory and runtime requirements to

58
00:03:45,890 --> 00:03:48,600
perform prediction,
once the model has been trained.

59
00:03:50,080 --> 00:03:53,310
Some major commercial applications of
machine learning have been based on

60
00:03:53,310 --> 00:03:54,980
gradient boosted decision trees.

61
00:03:56,600 --> 00:03:58,945
Like other decision tree
based learning methods,

62
00:03:58,945 --> 00:04:02,830
you don't need to apply feature
scaling for the algorithm to do well.

63
00:04:02,830 --> 00:04:06,850
And the futures can be a mix of binary,
categorical and continuous types.

64
00:04:09,150 --> 00:04:11,760
Boosted decision trees do
have several downsides.

65
00:04:11,760 --> 00:04:15,220
So like random forests,
ensembles of trees are very difficult for

66
00:04:15,220 --> 00:04:19,800
people to interpret,
compared to individual decision trees.

67
00:04:19,800 --> 00:04:21,380
However, this often may not matter for

68
00:04:21,380 --> 00:04:24,810
many applications where prediction
accuracy is the most important goal.

69
00:04:26,330 --> 00:04:29,669
Gradient boosted methods can require
careful tuning of the learning rate and

70
00:04:29,669 --> 00:04:31,010
other parameters.

71
00:04:31,010 --> 00:04:34,020
And the training process can
require a lot of computation.

72
00:04:35,150 --> 00:04:39,620
And like the other tree based methods we
saw, using gradient boosted methods for

73
00:04:39,620 --> 00:04:42,240
text classification or other scenarios.

74
00:04:42,240 --> 00:04:45,999
Where the featured space has thousands
of features with sparse values,

75
00:04:45,999 --> 00:04:49,895
is usually not a good choice for
accuracy and computational cost reasons.

76
00:04:52,950 --> 00:04:57,262
The key parameters controlling model
complexity for gradient boosted tree

77
00:04:57,262 --> 00:05:01,976
models are, n_estimators which sets the
number of small decisions trees the week

78
00:05:01,976 --> 00:05:05,090
learns to use in the ensemble,
and the learning rate.

79
00:05:06,150 --> 00:05:09,210
Typically, these two
parameters are tuned together.

80
00:05:09,210 --> 00:05:11,490
Since making the learning rates smaller,

81
00:05:11,490 --> 00:05:14,300
will require more trees to
maintain model complexity.

82
00:05:15,530 --> 00:05:20,260
Unlike random forest, increasing
an n_estimators can lead to overfeeding.

83
00:05:20,260 --> 00:05:24,690
So typically, the n_estimators setting
is chosen to best exploit the speed and

84
00:05:24,690 --> 00:05:27,790
memory capabilities of
the system during the training.

85
00:05:27,790 --> 00:05:31,380
And other parameters like
the learning rate are then adjusted,

86
00:05:31,380 --> 00:05:33,825
given that fixed an n_estimators setting.

87
00:05:34,868 --> 00:05:39,003
The max_depth parameter can also have
an effect of model complexity, but

88
00:05:39,003 --> 00:05:42,980
controlling the depth, and
has a complexity of the individual trees.

89
00:05:42,980 --> 00:05:47,040
The gradient boosting method assumes,
that each trees is a weak learner, and so

90
00:05:47,040 --> 00:05:51,280
the max_depth parameter is usually quite
small, on the order of three to five, for

91
00:05:51,280 --> 00:05:52,374
most applications.