1
00:00:07,670 --> 00:00:11,668
We saw that for classification, because
there were some scenarios like medical

2
00:00:11,668 --> 00:00:14,904
diagnostics predictions or
costumer facing web site features,

3
00:00:14,904 --> 00:00:19,770
where the consequences of false positive
were very different than false negatives.

4
00:00:19,770 --> 00:00:24,650
It made sense to distinguish these types
of errors and do a more detailed analysis.

5
00:00:24,650 --> 00:00:25,980
In evaluating classifiers for

6
00:00:25,980 --> 00:00:30,030
example we looked at plots like precision
recall curves that could show the trade

7
00:00:30,030 --> 00:00:34,210
offs a classifier could achieve between
making errors of those two types.

8
00:00:35,490 --> 00:00:38,730
In theory, we could apply the same
type of error analysis and

9
00:00:38,730 --> 00:00:42,920
more detailed evaluation to regression
that we applied for classification.

10
00:00:43,960 --> 00:00:48,050
For example, we could analyze
the regression model's predictions, and

11
00:00:48,050 --> 00:00:50,200
categorize errors of one type.

12
00:00:50,200 --> 00:00:53,160
Where the regression model's
predicted value was much larger

13
00:00:53,160 --> 00:00:54,760
than the target value.

14
00:00:54,760 --> 00:00:56,280
Compared to a second error type,

15
00:00:56,280 --> 00:00:59,040
where the predicted value was much
smaller than the target value.

16
00:01:00,850 --> 00:01:04,780
In practice though it turns out that for
most applications of regression,

17
00:01:04,780 --> 00:01:08,860
distinguishing between these types of
different errors is not as important.

18
00:01:10,100 --> 00:01:12,660
This simplifies evaluation for
regression quite a bit.

19
00:01:14,500 --> 00:01:19,810
In most cases, the default r squared
score that's available for regression and

20
00:01:19,810 --> 00:01:24,770
psychic learn and that summarizes how
well future instances will be predicted.

21
00:01:24,770 --> 00:01:26,450
It's adequate for most tasks.

22
00:01:27,800 --> 00:01:33,170
As a reminder, the r2_score for
perfect predictor is 1.0.

23
00:01:33,170 --> 00:01:38,133
And for a predictor that always output the
same constant value, the r2_score is 0.0.

24
00:01:38,133 --> 00:01:43,060
The r2_score despite the squared
in the name that suggests

25
00:01:43,060 --> 00:01:46,520
it's always positive does have
the potential to go negative for

26
00:01:46,520 --> 00:01:50,630
bad model fits, such as when fitting
non-linear functions to data.

27
00:01:52,510 --> 00:01:56,400
There are a few alternative
regression devaluation metrics

28
00:01:56,400 --> 00:02:01,010
you should be aware of that work
differently than the r2_score.

29
00:02:01,010 --> 00:02:05,700
Mean absolute error takes the mean
absolute difference between the target and

30
00:02:05,700 --> 00:02:07,750
predicted values.

31
00:02:07,750 --> 00:02:12,826
In machine running terms this corresponds
to the expected value of L1 norm laws.

32
00:02:12,826 --> 00:02:17,790
This is sometimes used for
example to asses focused outcomes for

33
00:02:17,790 --> 00:02:20,093
regression in time series analysis.

34
00:02:20,093 --> 00:02:26,020
Mean squared error takes the mean squared
difference between the target and

35
00:02:26,020 --> 00:02:32,220
predicted values and this corresponds to
the expected value of the L2 norm loss.

36
00:02:32,220 --> 00:02:35,340
This is widely used for
many regression problems and

37
00:02:35,340 --> 00:02:39,610
larger errors have correspondingly larger
squared contributions to the mean error.

38
00:02:41,030 --> 00:02:45,540
Like mean absolute error, mean squared
error doesn't distinguish between over and

39
00:02:45,540 --> 00:02:46,560
under estimates.

40
00:02:48,380 --> 00:02:51,250
Finally one situation that
does arise quite often,

41
00:02:51,250 --> 00:02:54,130
is the existence of outliers in the data,

42
00:02:54,130 --> 00:02:59,230
which can have unwanted influence on the
overall r squared or mean squared value.

43
00:02:59,230 --> 00:03:03,500
So in those cases,
when ignoring outlier is important,

44
00:03:03,500 --> 00:03:08,220
you can use the mean_absolute_error score,
which is robust with the presence of

45
00:03:08,220 --> 00:03:12,720
outliers because it uses the median of the
error distribution rather than the mean.

46
00:03:15,130 --> 00:03:18,230
We saw how using how dummy
classifiers could give us simple but

47
00:03:18,230 --> 00:03:22,380
useful baselines to compared against
when evaluating a classifier.

48
00:03:22,380 --> 00:03:25,150
The same functionality exist for
regression.

49
00:03:26,240 --> 00:03:29,600
There's a dummy regressor class
that provides predictions

50
00:03:29,600 --> 00:03:32,490
using simple strategies that
do not look at the input data.

51
00:03:34,110 --> 00:03:37,830
This example which is available as
the regression example from this lecture's

52
00:03:37,830 --> 00:03:43,520
notebook shows a scatter plot using
database on a single input variable,

53
00:03:43,520 --> 00:03:48,110
which is plotted along the x
axis from the diabetes data set.

54
00:03:49,630 --> 00:03:53,028
The points are the data instances
from the test split and

55
00:03:53,028 --> 00:03:57,158
form a cloud that looks like it may
trend down slightly to the right.

56
00:03:58,843 --> 00:04:02,706
The green line, which is also labeled
fitted model is the default linear

57
00:04:02,706 --> 00:04:06,050
regression that was fit
to the training points.

58
00:04:06,050 --> 00:04:09,240
We can see that it’s not a particularly
strong fit to the test data.

59
00:04:10,640 --> 00:04:15,640
The red line labeled dummy mean,
shows a linear model

60
00:04:15,640 --> 00:04:19,140
that uses the strategy of always
predicting the mean of the training data.

61
00:04:20,160 --> 00:04:22,614
So this is an example
of a dummy regressor.

62
00:04:24,550 --> 00:04:28,130
You can look at the notebook to see
that a dummy regressor is created and

63
00:04:28,130 --> 00:04:31,240
used just like a regular regression model.

64
00:04:31,240 --> 00:04:36,820
You create, fit with the training data,
and then call predict on the test data.

65
00:04:36,820 --> 00:04:40,850
Although again, like the dummy classifier
you should not use the dummy regressor for

66
00:04:40,850 --> 00:04:42,340
actual problems.

67
00:04:42,340 --> 00:04:45,617
Its only use is to provide a baseline for
comparison.

68
00:05:05,279 --> 00:05:08,778
Looking at the regression metrics output
from the linear model compared to

69
00:05:08,778 --> 00:05:10,070
the dummy model.

70
00:05:10,070 --> 00:05:14,010
We can see that as expected the dummy
regressor achieves an r squared

71
00:05:14,010 --> 00:05:14,700
score of 0.

72
00:05:14,700 --> 00:05:18,810
Since it always makes a constant
prediction without looking at the output.

73
00:05:20,220 --> 00:05:24,350
In this instance the linear model provides
only slightly better fit than the dummy

74
00:05:24,350 --> 00:05:28,820
regressor, according to both mean
squared error and the r2_score.

75
00:05:30,420 --> 00:05:33,870
Aside from the strategy of always
predicting the mean of the training target

76
00:05:33,870 --> 00:05:38,570
values, you could also create some other
flavors of dummy regressors that always

77
00:05:38,570 --> 00:05:43,380
predict the median of the training target
values, or a particular quantile of

78
00:05:43,380 --> 00:05:48,010
those values, or a specific custom
constant value that you provide.

79
00:05:49,430 --> 00:05:54,120
Although regression typically has simpler
evaluation needs than classification,

80
00:05:54,120 --> 00:05:57,810
it does pay to double check to make sure
the evaluation metric you choose for

81
00:05:57,810 --> 00:06:01,330
a regression problem does
penalize errors in a way

82
00:06:01,330 --> 00:06:03,580
that reflects the consequences
of those errors for

83
00:06:03,580 --> 00:06:07,530
the business, organizational, or
user needs of your application.