1 00:00:07,970 --> 00:00:11,670 Now that you've seen a number of different evaluation metrics 2 00:00:11,670 --> 00:00:14,482 for both binary and multiclass classification, 3 00:00:14,482 --> 00:00:17,310 let's take a look at how you can apply them as criteria for 4 00:00:17,310 --> 00:00:20,586 selecting the best classifier for your application, 5 00:00:20,586 --> 00:00:24,000 otherwise known as model selection. 6 00:00:24,000 --> 00:00:25,770 In previous lectures we've seen a number of 7 00:00:25,770 --> 00:00:30,300 different evaluation frameworks for potential model selection. 8 00:00:30,300 --> 00:00:34,084 First, we simply did training and testing on the same dataset, 9 00:00:34,084 --> 00:00:35,520 which as we well know, 10 00:00:35,520 --> 00:00:40,705 typically overfits badly and doesn't generalize well to new data. 11 00:00:40,705 --> 00:00:42,630 As a side note however, 12 00:00:42,630 --> 00:00:44,970 it can serve as a useful sanity check to make sure 13 00:00:44,970 --> 00:00:50,700 your software engineering and feature generation pipeline is working correctly. 14 00:00:50,700 --> 00:00:52,020 Second, we frequently use 15 00:00:52,020 --> 00:00:56,160 the single train-test split to produce a single evaluation metric. 16 00:00:56,160 --> 00:00:58,008 While fast and easy, 17 00:00:58,008 --> 00:01:00,300 this doesn't give as realistic a set of 18 00:01:00,300 --> 00:01:03,915 estimates for how well the model may work on future, new data. 19 00:01:03,915 --> 00:01:07,530 And we don't get a good picture for the variance in the evaluation metrics that may 20 00:01:07,530 --> 00:01:13,440 result as we do prediction on different test sets. 21 00:01:13,440 --> 00:01:18,765 Third, we used k-fold cross-validation to create K random train-test splits, 22 00:01:18,765 --> 00:01:23,485 where the evaluation metric was averaged across splits. 23 00:01:23,485 --> 00:01:26,905 This leads to models that are more reliable on unseen data. 24 00:01:26,905 --> 00:01:30,630 In particular, we can also use grid search using for 25 00:01:30,630 --> 00:01:35,130 example the GridSearchCV method within each cross-validation fold, 26 00:01:35,130 --> 00:01:40,660 to find optimal parameters for a model with respect to the evaluation metric. 27 00:01:40,660 --> 00:01:48,775 The default evaluation metric used for a cross-val score or GridSearchCV is accuracy. 28 00:01:48,775 --> 00:01:51,905 So how do you apply the new metrics you've learned about here like 29 00:01:51,905 --> 00:01:55,850 AUC in model selection? 30 00:01:55,850 --> 00:01:58,410 Scikit-learn makes this very easy. 31 00:01:58,410 --> 00:02:01,578 You simply add a scoring parameter that's set to 32 00:02:01,578 --> 00:02:06,055 the string with the name of the evaluation metric you want to use. 33 00:02:06,055 --> 00:02:10,005 Let's first look at an example using the scoring parameter for cross-validation, 34 00:02:10,005 --> 00:02:13,980 and then we'll take a look at the other primary method of model selection, grid search. 35 00:02:13,980 --> 00:02:18,780 In the notebook here we have a cross-validation example where we're running 36 00:02:18,780 --> 00:02:21,965 five folds using a support vector classifier 37 00:02:21,965 --> 00:02:26,140 with a linear kernel and C parameter set to one. 38 00:02:26,140 --> 00:02:33,780 The first call to cross-val score just uses default accuracy as the evaluation metric. 39 00:02:33,780 --> 00:02:40,709 The second call uses the scoring parameter using the string 'roc_auc', 40 00:02:40,709 --> 00:02:45,380 and this will use AUC as the evaluation metric. 41 00:02:45,380 --> 00:02:48,870 The third call sets the scoring parameter to 'recall', 42 00:02:48,870 --> 00:02:52,320 to use that as the evaluation metric. 43 00:02:52,320 --> 00:02:55,380 You can see the resulting list of five evaluation values, 44 00:02:55,380 --> 00:02:58,260 one per fold for each metric. 45 00:02:58,260 --> 00:03:01,470 Now, here we're not doing any parameter tuning we're simply evaluating 46 00:03:01,470 --> 00:03:08,270 our model's average performance across multiple folds. 47 00:03:08,270 --> 00:03:11,220 Now, in this grid search example we use 48 00:03:11,220 --> 00:03:15,753 a support vector classifier that uses a radio basis function kernel. 49 00:03:15,753 --> 00:03:19,200 And the critical parameter here is the gamma parameter that 50 00:03:19,200 --> 00:03:24,390 intuitively sets the radius or width of influence of the kernel. 51 00:03:24,390 --> 00:03:27,825 We use GridSearchCV to find the value of gamma 52 00:03:27,825 --> 00:03:32,300 that optimizes a given evaluation metric in two cases. 53 00:03:32,300 --> 00:03:36,509 In the first case, we just optimize for average accuracy; 54 00:03:36,509 --> 00:03:40,449 in the second case we optimize for AUC. 55 00:03:40,449 --> 00:03:44,624 In this particular case the optimal value of gamma happens to be the same, 56 00:03:44,624 --> 00:03:49,065 point zero-zero-one, for both evaluation metrics. 57 00:03:49,065 --> 00:03:51,080 As we'll see later in other cases, 58 00:03:51,080 --> 00:03:53,030 the optimal parameter value can be quite 59 00:03:53,030 --> 00:03:58,490 different depending on the evaluation metric used to optimize. 60 00:03:58,490 --> 00:04:00,320 You can see the complete list of names for 61 00:04:00,320 --> 00:04:03,590 the evaluation metric supported by the scoring parameter by 62 00:04:03,590 --> 00:04:09,352 running the following code that uses the score's variable imported from sklearn metrics. 63 00:04:09,352 --> 00:04:12,080 You can see metrics for classification such as the 64 00:04:12,080 --> 00:04:15,380 string 'precision_ micro' that represents 65 00:04:15,380 --> 00:04:18,890 micro-averaged precision as well as metrics for regression such as 66 00:04:18,890 --> 00:04:24,020 the R2 metric for R-squared regression loss. 67 00:04:24,020 --> 00:04:26,330 Let's take a look at a specific example that shows 68 00:04:26,330 --> 00:04:29,255 how a classifier's decision boundary changes 69 00:04:29,255 --> 00:04:33,890 when it's optimized for different evaluation metrics. 70 00:04:33,890 --> 00:04:35,870 This classification problem is based on 71 00:04:35,870 --> 00:04:38,240 the same binary digit classifier training and 72 00:04:38,240 --> 00:04:41,645 test sets we've been using as an example throughout the notebook. 73 00:04:41,645 --> 00:04:46,070 In these classification visualization examples, the positive examples, 74 00:04:46,070 --> 00:04:50,180 the digit one are shown as black points and the region of 75 00:04:50,180 --> 00:04:52,220 positive class prediction is shown in 76 00:04:52,220 --> 00:04:57,170 the light-colored or yellow region to the right of this decision boundary. 77 00:04:57,170 --> 00:04:59,795 The negative examples, all other digits, 78 00:04:59,795 --> 00:05:01,577 are shown as white points. 79 00:05:01,577 --> 00:05:03,889 And the region of negative class prediction here in 80 00:05:03,889 --> 00:05:08,112 these figures is to the left of the decision boundary. 81 00:05:08,112 --> 00:05:10,790 The data points have been plotted using two out of 82 00:05:10,790 --> 00:05:15,599 the 64 future values in the digits' dataset and have been jittered a little. 83 00:05:15,599 --> 00:05:19,400 That is, I've added a little bit of random noise so we can see more easily 84 00:05:19,400 --> 00:05:24,059 the density of examples in the feature space. 85 00:05:24,059 --> 00:05:28,505 Here's the scikit-learn code that produced this figure. 86 00:05:28,505 --> 00:05:32,530 We apply grid search here to explore different values of 87 00:05:32,530 --> 00:05:35,620 the optional class weight parameter that controls 88 00:05:35,620 --> 00:05:39,445 how much weight is given to each of the two classes during training. 89 00:05:39,445 --> 00:05:41,290 As it turns out, optimizing for 90 00:05:41,290 --> 00:05:43,660 different evaluation metrics results in 91 00:05:43,660 --> 00:05:46,914 different optimal values of the class weight parameter. 92 00:05:46,914 --> 00:05:49,694 As the class weight parameter increases, 93 00:05:49,694 --> 00:05:54,890 more emphasis will be given to correctly classifying the positive class instances. 94 00:05:54,890 --> 00:06:00,010 The precision-oriented classifier we see here with class weight of two, 95 00:06:00,010 --> 00:06:04,420 tries hard to reduce false negatives while increasing true positives. 96 00:06:04,420 --> 00:06:07,540 So it focuses on the cluster of positive class points in 97 00:06:07,540 --> 00:06:12,399 the lower right corner where there are relatively few negative class points. 98 00:06:12,399 --> 00:06:16,660 Here, precision is over 50 percent. 99 00:06:16,660 --> 00:06:20,950 In contrast, the recall-oriented classifier with class weight of 50, 100 00:06:20,950 --> 00:06:26,075 tries hard to reduce the number of false negatives while increasing true positives. 101 00:06:26,075 --> 00:06:28,120 That is, it tries to find most of 102 00:06:28,120 --> 00:06:33,205 the positive class points as part of its positive class predictions. 103 00:06:33,205 --> 00:06:36,100 We can also see that the decision boundary for 104 00:06:36,100 --> 00:06:40,030 the F1-oriented classifier has an optimal class weight of two, 105 00:06:40,030 --> 00:06:42,580 which is between the optimal class weight values for 106 00:06:42,580 --> 00:06:46,830 the precision and recall-oriented classifiers. 107 00:06:46,830 --> 00:06:50,905 Visually we can see that the F1-oriented classifier also has a kind of 108 00:06:50,905 --> 00:06:56,890 intermediate positioning between the precision and recall-oriented, decision boundaries. 109 00:06:56,890 --> 00:07:01,539 This makes sense given that F1 is the harmonic mean of precision and recall. 110 00:07:01,539 --> 00:07:05,665 The AUC-oriented classifier with 111 00:07:05,665 --> 00:07:11,230 optimal class weight to 5 has a similar decision boundary to the F1-oriented classifier, 112 00:07:11,230 --> 00:07:16,160 but shifted slightly in favor of higher recall. 113 00:07:16,160 --> 00:07:18,930 We can see the precision recall trade-off 114 00:07:18,930 --> 00:07:21,910 very clearly for this classification scenario in 115 00:07:21,910 --> 00:07:23,320 the precision recall curve for 116 00:07:23,320 --> 00:07:25,540 the default support vector classifier with 117 00:07:25,540 --> 00:07:29,095 linear kernel optimized for accuracy on the same dataset, 118 00:07:29,095 --> 00:07:33,835 and using the balanced option for the class weight parameter. 119 00:07:33,835 --> 00:07:35,230 Let's take a look at the code 120 00:07:35,230 --> 00:07:54,088 that generated this plot. 121 00:07:54,088 --> 00:08:08,870 Take a moment to 122 00:08:08,870 --> 00:08:11,120 imagine how the extreme lower right part of 123 00:08:11,120 --> 00:08:14,510 the curve on this precision recall curve represents 124 00:08:14,510 --> 00:08:16,010 a decision boundary that's highly 125 00:08:16,010 --> 00:08:20,210 precision-oriented in the lower right of the classification plot, 126 00:08:20,210 --> 00:08:22,759 where there's a cluster of positive examples. 127 00:08:22,759 --> 00:08:27,564 As the decision threshold is shifted to become less and less conservative, 128 00:08:27,564 --> 00:08:29,815 tracing the curve up into the left, 129 00:08:29,815 --> 00:08:32,510 the classifier becomes more and more like 130 00:08:32,510 --> 00:08:36,780 the recall-oriented support vector classifier example. 131 00:08:36,780 --> 00:08:39,215 Again, the red circle represents 132 00:08:39,215 --> 00:08:42,320 the precision recall trade-off achieved at the zero score mark, 133 00:08:42,320 --> 00:08:45,790 which is the actual decision boundary chosen for the trained classifier. 134 00:08:45,790 --> 00:08:49,630 For simplicity, we've often 135 00:08:49,630 --> 00:08:55,210 used a single train-test split in showing examples of evaluation scoring. 136 00:08:55,210 --> 00:08:58,240 However, using only cross-validation or a test set for 137 00:08:58,240 --> 00:09:01,330 model selection or parameter tuning may still lead 138 00:09:01,330 --> 00:09:04,045 to more subtle forms of overfitting and less 139 00:09:04,045 --> 00:09:08,780 optimistic evaluation estimates for future, unseen data. 140 00:09:08,780 --> 00:09:10,780 An intuitive explanation for this might be the 141 00:09:10,780 --> 00:09:13,780 following: "remember that the whole point of evaluating on 142 00:09:13,780 --> 00:09:16,510 a test set is to estimate how well 143 00:09:16,510 --> 00:09:21,580 a learning algorithm might perform on future, unseen data. 144 00:09:21,580 --> 00:09:24,610 The more information we see about our dataset as part of 145 00:09:24,610 --> 00:09:28,255 repeated cross-validation passes in choosing our model, 146 00:09:28,255 --> 00:09:31,360 the more influence any potential held-up test data 147 00:09:31,360 --> 00:09:33,685 has played into selecting the final model. 148 00:09:33,685 --> 00:09:37,720 Not merely evaluating it. 149 00:09:37,720 --> 00:09:40,620 This is sometimes called data leakage and 150 00:09:40,620 --> 00:09:44,730 we'll describe more about that phenomenon in another module. 151 00:09:44,730 --> 00:09:50,080 So, we haven't done an evaluation with a truly held-out test set unless we 152 00:09:50,080 --> 00:09:52,660 commit to holding back a test split that isn't seen by 153 00:09:52,660 --> 00:09:57,270 any process until the very end of the evaluation. 154 00:09:57,270 --> 00:10:01,450 So that's what's actually done in practice. 155 00:10:01,450 --> 00:10:05,515 There are three data splits: training for model building, 156 00:10:05,515 --> 00:10:11,955 validation for model selection and a test set for the final evaluation. 157 00:10:11,955 --> 00:10:15,220 The training and test sets are typically split out first, 158 00:10:15,220 --> 00:10:17,860 and then cross-validation is run using 159 00:10:17,860 --> 00:10:21,830 the training data to do model and parameter selection. 160 00:10:21,830 --> 00:10:27,370 Again, the test set is not seen until the very end of the evaluation process. 161 00:10:27,370 --> 00:10:31,090 Machine learning researchers take this protocol very seriously. 162 00:10:31,090 --> 00:10:33,400 The train-validate-test design is 163 00:10:33,400 --> 00:10:36,550 a very important universally applied framework for 164 00:10:36,550 --> 00:10:41,430 effective evaluation of machine learning models. 165 00:10:41,430 --> 00:10:43,670 That brings us to the end of this section 166 00:10:43,670 --> 00:10:46,795 of the course on evaluation for machine learning. 167 00:10:46,795 --> 00:10:49,160 You should now understand why accuracy only gives 168 00:10:49,160 --> 00:10:53,120 a partial picture of a classifier's performance and be more familiar with 169 00:10:53,120 --> 00:10:54,590 the motivation and definition of 170 00:10:54,590 --> 00:10:57,320 important alternative evaluation methods and 171 00:10:57,320 --> 00:11:00,575 metrics of machine learning like confusion matrices, 172 00:11:00,575 --> 00:11:06,345 precision, recall, F1 score and area under the RAC curve. 173 00:11:06,345 --> 00:11:08,840 You've also seen how to apply and choose 174 00:11:08,840 --> 00:11:11,420 these different evaluation metric alternatives in order to 175 00:11:11,420 --> 00:11:15,260 optimize model selection or parameter tuning for a classifier, 176 00:11:15,260 --> 00:11:19,370 to maximize a given evaluation metric. 177 00:11:19,370 --> 00:11:23,120 Finally, I'd like to leave you with a couple of points. 178 00:11:23,120 --> 00:11:25,850 First, simple accuracy may not often 179 00:11:25,850 --> 00:11:29,910 be the right goal for your particular machine learning application. 180 00:11:29,910 --> 00:11:34,535 As we saw for example with tumor detection or credit card fraud, 181 00:11:34,535 --> 00:11:37,670 false positives and false negatives might have 182 00:11:37,670 --> 00:11:42,590 very different real-world effects for users or for organization outcomes. 183 00:11:42,590 --> 00:11:45,110 So it's important to select an evaluation metric that 184 00:11:45,110 --> 00:11:49,850 reflects those user application or business needs. 185 00:11:49,850 --> 00:11:52,670 Second, there are a number of other dimensions along which it 186 00:11:52,670 --> 00:11:55,715 may be important to evaluate your machine learning algorithms, 187 00:11:55,715 --> 00:12:00,005 that we don't cover here but that are important for you to be aware of. 188 00:12:00,005 --> 00:12:03,420 I'll mention two specifically here. 189 00:12:03,420 --> 00:12:05,330 Learning curves are used to assess 190 00:12:05,330 --> 00:12:08,540 how a machine learning algorithm's evaluation metric changes or 191 00:12:08,540 --> 00:12:13,010 improves as the algorithm gets more training data. 192 00:12:13,010 --> 00:12:17,074 Learning curves may be useful as part of a cost-benefit analysis. 193 00:12:17,074 --> 00:12:19,010 Gathering training data in the form of 194 00:12:19,010 --> 00:12:22,880 labeled examples is often time-consuming and expensive. 195 00:12:22,880 --> 00:12:27,742 So being able to estimate the likely performance improvement of your classifier, 196 00:12:27,742 --> 00:12:30,740 if you say invest in doubling the amount of training data, 197 00:12:30,740 --> 00:12:34,670 can be a useful analysis. 198 00:12:34,670 --> 00:12:38,270 Second, sensitivity analysis amounts to looking at 199 00:12:38,270 --> 00:12:40,440 how an evaluation metric changes as 200 00:12:40,440 --> 00:12:44,660 small adjustments are made to important model parameters. 201 00:12:44,660 --> 00:12:49,365 This helps assess how robust the model is to choice of parameters. 202 00:12:49,365 --> 00:12:52,940 This may be important to perform especially if there are other costs such as 203 00:12:52,940 --> 00:12:57,890 runtime efficiency that are critical variables when deploying an operational system, 204 00:12:57,890 --> 00:13:00,440 that are correlated with different values of parameter. 205 00:13:00,440 --> 00:13:04,770 For example, decision tree depth or future value threshold. 206 00:13:04,770 --> 00:13:08,510 In this way, a more complete picture of the trade-offs achievable across 207 00:13:08,510 --> 00:13:11,180 different performance dimensions can help you make 208 00:13:11,180 --> 00:13:15,800 the best practical deployment decisions for your machine learning model.