1 00:00:08,130 --> 00:00:13,840 So, let's go back to the matrix of possible binary classification outcomes. 2 00:00:13,840 --> 00:00:20,179 This time filled out with the actual counts from the notebooks decision tree output. 3 00:00:20,179 --> 00:00:22,839 Remember our original motivation for creating 4 00:00:22,839 --> 00:00:26,440 this matrix was to go beyond a single number accuracy, 5 00:00:26,440 --> 00:00:29,170 to get more insight into the different types of 6 00:00:29,170 --> 00:00:33,875 prediction successes and failures of a given classifier. 7 00:00:33,875 --> 00:00:38,630 Now we have these four numbers that we can examine and compare manually. 8 00:00:38,630 --> 00:00:42,100 Let's look at this classification result visually to help 9 00:00:42,100 --> 00:00:46,539 us connect these four numbers to a classifier's performance. 10 00:00:46,539 --> 00:00:51,429 What I've done here is plot the data instances by using two specific feature values out 11 00:00:51,429 --> 00:00:56,914 of the total 64 feature values that make up each instance in the digits dataset. 12 00:00:56,914 --> 00:01:01,225 The black points here are the instances with true class positive 13 00:01:01,225 --> 00:01:06,377 namely the digit one and the white points have true class negative, 14 00:01:06,377 --> 00:01:10,685 that is, there are all the other digits except for one. 15 00:01:10,685 --> 00:01:12,655 The black line shows 16 00:01:12,655 --> 00:01:15,760 a hypothetical linear classifier's decision boundary 17 00:01:15,760 --> 00:01:17,859 for which any instance to the left of 18 00:01:17,859 --> 00:01:21,549 the decision boundary is predicted to be in the positive class and 19 00:01:21,549 --> 00:01:23,489 everything to the right of the decision boundary 20 00:01:23,489 --> 00:01:26,319 is predicted to be in the negative class. 21 00:01:26,319 --> 00:01:29,680 The true positive points are those black points in 22 00:01:29,680 --> 00:01:33,084 the positive prediction region and 23 00:01:33,084 --> 00:01:38,329 false positives are those white points in the positive prediction region. 24 00:01:38,329 --> 00:01:40,810 Likewise, true negatives are the white points in 25 00:01:40,810 --> 00:01:43,180 the negative prediction region and 26 00:01:43,180 --> 00:01:49,659 false negatives are black points in the negative prediction region. 27 00:01:49,659 --> 00:01:52,180 We've already seen one metric that can be derived from 28 00:01:52,180 --> 00:01:56,379 the confusion matrix counts namely accuracy. 29 00:01:56,379 --> 00:01:59,310 The successful predictions of the classifier, 30 00:01:59,310 --> 00:02:02,950 the ones where the predicted class matches 31 00:02:02,950 --> 00:02:07,480 the true class are along the diagonal of the confusion matrix. 32 00:02:07,480 --> 00:02:09,909 So, if we add up all the accounts along the diagonal, 33 00:02:09,909 --> 00:02:14,409 that will give us the total number of correct predictions across all classes, 34 00:02:14,409 --> 00:02:19,659 and dividing this sum by the total number of instances gives us accuracy. 35 00:02:19,659 --> 00:02:25,659 But, let's look at some other evaluation metrics we can compute from these four numbers. 36 00:02:25,659 --> 00:02:30,159 Well, a very simple related number that's sometimes used is classification error, 37 00:02:30,159 --> 00:02:32,965 which is the sum of the counts off the diagonal 38 00:02:32,965 --> 00:02:36,414 namely all of the errors divided by total instance count, 39 00:02:36,414 --> 00:02:42,550 and numerically, this is equivalent to just one minus the accuracy. 40 00:02:42,550 --> 00:02:45,009 Now, for a more interesting example, let's suppose, 41 00:02:45,009 --> 00:02:47,409 going back to our medical tumor detecting 42 00:02:47,409 --> 00:02:51,639 classifier that we wanted an evaluation metric that would give 43 00:02:51,639 --> 00:02:55,330 higher scores to classifiers that not only achieved the high number 44 00:02:55,330 --> 00:02:59,224 of true positives but also avoided false negatives. 45 00:02:59,224 --> 00:03:04,465 That is, that rarely failed to detect a true cancerous tumor. 46 00:03:04,465 --> 00:03:07,764 Recall, also known as the true positive rate, 47 00:03:07,764 --> 00:03:14,259 sensitivity or probability of detection is such an evaluation metric and it's obtained 48 00:03:14,259 --> 00:03:17,185 by dividing the number of true positives 49 00:03:17,185 --> 00:03:21,930 by the sum of true positives and false negatives. 50 00:03:21,930 --> 00:03:26,635 You can see from this formula that there are two ways to get a larger recall number. 51 00:03:26,635 --> 00:03:29,020 First, by either increasing the number of 52 00:03:29,020 --> 00:03:33,504 true positives or by reducing the number of false negatives. 53 00:03:33,504 --> 00:03:37,240 Since this will make the denominator smaller. 54 00:03:37,240 --> 00:03:40,509 In this example there are 26 true positives and 55 00:03:40,509 --> 00:03:47,004 17 false negatives which gives a recall of 0.6. 56 00:03:47,004 --> 00:03:49,655 Now suppose that we have a machine learning task, 57 00:03:49,655 --> 00:03:54,509 where it's really important to avoid false positives. 58 00:03:54,509 --> 00:03:58,250 In other words, we're fine with cases where not all true positive instances 59 00:03:58,250 --> 00:04:02,254 are detected but when the classifier does predict the positive class, 60 00:04:02,254 --> 00:04:05,460 we want to be very confident that it's correct. 61 00:04:05,460 --> 00:04:09,379 A lot of customer facing prediction problems are like this, for example, 62 00:04:09,379 --> 00:04:11,360 predicting when to show a user 63 00:04:11,360 --> 00:04:16,759 A query suggestion in a web search interface might be one such scenario. 64 00:04:16,759 --> 00:04:19,214 Users will often remember the failures of 65 00:04:19,214 --> 00:04:24,019 a machine learning prediction even when the majority of predictions are successes. 66 00:04:24,019 --> 00:04:29,375 So, precision is an evaluation metric that reflects the situation 67 00:04:29,375 --> 00:04:31,610 and is obtained by dividing the number of 68 00:04:31,610 --> 00:04:36,350 true positives by the sum of true positives and false positives. 69 00:04:36,350 --> 00:04:37,759 So to increase precision, 70 00:04:37,759 --> 00:04:43,100 we must either increase the number of true positives the classifier predicts or reduce 71 00:04:43,100 --> 00:04:45,529 the number of errors where the classifier incorrectly 72 00:04:45,529 --> 00:04:49,004 predicts that a negative instance is in the positive class. 73 00:04:49,004 --> 00:04:56,730 Here, the classifier has made seven false positive errors and so the precision is 0.79. 74 00:04:56,730 --> 00:05:02,562 Another related evaluation metric that will be useful is called the false positive rate, 75 00:05:02,562 --> 00:05:05,665 also known as specificity. 76 00:05:05,665 --> 00:05:08,050 This gives the fraction of all negative instances that 77 00:05:08,050 --> 00:05:12,310 the classifier incorrectly identifies as positive. 78 00:05:12,310 --> 00:05:14,220 Here, we have seven false positives, 79 00:05:14,220 --> 00:05:17,665 which out of a total of 407 negative instances, 80 00:05:17,665 --> 00:05:23,079 gives a false positive rate of 0.02. 81 00:05:23,079 --> 00:05:25,956 Going back to our classifier visualization, 82 00:05:25,956 --> 00:05:29,899 let's look at how precision and recall can be interpreted. 83 00:05:29,899 --> 00:05:32,379 The numbers that are in the confusion matrix here are 84 00:05:32,379 --> 00:05:35,795 derived from this classification scenario. 85 00:05:35,795 --> 00:05:41,930 We can see that a precision of 0.68 means that about 68 percent of the points in 86 00:05:41,930 --> 00:05:44,660 the positive prediction region to the left of 87 00:05:44,660 --> 00:05:52,329 the decision boundary or 13 out of the 19 instances are correctly labeled as positive. 88 00:05:52,329 --> 00:05:55,024 A recall of 0.87 means, 89 00:05:55,024 --> 00:05:57,796 that of all true positive instances, 90 00:05:57,796 --> 00:05:59,899 so all black points in the figure, 91 00:05:59,899 --> 00:06:08,694 the positive prediction region has 'found about 87 percent of them' or 13 out of 15. 92 00:06:08,694 --> 00:06:13,290 If we wanted a classifier that was oriented towards higher levels of 93 00:06:13,290 --> 00:06:18,620 precision like in the search engine query suggestion task, 94 00:06:18,620 --> 00:06:23,009 we might want a decision boundary instead that look like this. 95 00:06:23,009 --> 00:06:24,459 Now, all the points in 96 00:06:24,459 --> 00:06:28,355 the positive prediction region seven out of seven are true positives, 97 00:06:28,355 --> 00:06:33,459 giving us a perfect precision of 1.0. 98 00:06:33,459 --> 00:06:35,410 Now, this comes at a cost because out of 99 00:06:35,410 --> 00:06:40,101 the 15 total positive instances eight of them are now false negatives, 100 00:06:40,101 --> 00:06:44,189 in other words, they're incorrectly predicted as being negative. 101 00:06:44,189 --> 00:06:51,730 And so, recall drops to 7 divided by 15 or 0.47. 102 00:06:51,730 --> 00:06:57,199 On the other hand, if our classification task is like the tumor detection example, 103 00:06:57,199 --> 00:07:01,655 we want to minimize false negatives and obtain high recall. 104 00:07:01,655 --> 00:07:08,264 In which case, we would want the classifier's decision boundary to look more like this. 105 00:07:08,264 --> 00:07:11,019 Now, all 15 positive instances have 106 00:07:11,019 --> 00:07:14,004 been correctly predicted as being in the positive class, 107 00:07:14,004 --> 00:07:17,894 which means these tumors have all been detected. 108 00:07:17,894 --> 00:07:22,259 However, this also comes with a cost since the number of false positives, 109 00:07:22,259 --> 00:07:25,529 things that the detector triggers as 110 00:07:25,529 --> 00:07:30,139 possible tumors for example that are actually not, has gone up. 111 00:07:30,139 --> 00:07:40,800 So, recall is a perfect 1.0 score but the precision has dropped to 15 out of 42 or 0.36. 112 00:07:40,800 --> 00:07:43,199 These examples illustrate a classic trade-off 113 00:07:43,199 --> 00:07:46,080 that often appears in machine learning applications. 114 00:07:46,080 --> 00:07:48,689 Namely, that you can often increase the precision of 115 00:07:48,689 --> 00:07:55,097 a classifier but the downside is that you may reduce recall, 116 00:07:55,097 --> 00:08:00,870 or you could increase the recall of a classifier at the cost of reducing precision. 117 00:08:00,870 --> 00:08:05,790 Recall oriented machine learning tasks include medical and legal applications, 118 00:08:05,790 --> 00:08:12,089 where the consequences of not correctly identifying a positive example can be high. 119 00:08:12,089 --> 00:08:16,410 Often in these scenarios human experts are deployed to help filter out 120 00:08:16,410 --> 00:08:22,770 the false positives that almost inevitably increase with high recall applications. 121 00:08:22,770 --> 00:08:27,225 Many customer facing machine learning tasks, as I just mentioned, 122 00:08:27,225 --> 00:08:29,665 are often precision oriented since here 123 00:08:29,665 --> 00:08:33,570 the consequences of false positives can be high, for example, 124 00:08:33,570 --> 00:08:36,629 hurting the customer's experience on a website by 125 00:08:36,629 --> 00:08:40,714 providing incorrect or unhelpful information. 126 00:08:40,714 --> 00:08:43,860 Examples include, search engine ranking and 127 00:08:43,860 --> 00:08:50,159 classifying documents to annotate them with topic tags. 128 00:08:50,159 --> 00:08:52,860 When evaluating classifiers, it's often convenient 129 00:08:52,860 --> 00:08:55,965 to compute a quantity known as an F1 score, 130 00:08:55,965 --> 00:09:00,990 that combines precision and recall into a single number. 131 00:09:00,990 --> 00:09:03,690 Mathematically, this is based on 132 00:09:03,690 --> 00:09:08,210 the harmonic mean of precision and recall using this formula. 133 00:09:08,210 --> 00:09:09,514 After a little bit of algebra, 134 00:09:09,514 --> 00:09:12,809 we can rewrite the F1 score in terms of the quantities 135 00:09:12,809 --> 00:09:16,379 that we saw in the confusion matrix: true positives, 136 00:09:16,379 --> 00:09:19,860 false negatives and false positives. 137 00:09:19,860 --> 00:09:23,519 This F1 score is a special case of 138 00:09:23,519 --> 00:09:29,820 a more general evaluation metric known as an F score that introduces a parameter beta. 139 00:09:29,820 --> 00:09:32,460 By adjusting beta we can control how much emphasis 140 00:09:32,460 --> 00:09:36,629 an evaluation is given to precision versus recall. 141 00:09:36,629 --> 00:09:39,465 For example, if we have precision oriented users, 142 00:09:39,465 --> 00:09:42,644 we might say a beta equal to 0.5, 143 00:09:42,644 --> 00:09:47,750 since we want false positives to hurt performance more than false negatives. 144 00:09:47,750 --> 00:09:50,190 For recall oriented situations, 145 00:09:50,190 --> 00:09:53,299 we might set beta to a number larger than one, say two, 146 00:09:53,299 --> 00:09:59,335 to emphasize that false negatives should hurt performance more than false positives. 147 00:09:59,335 --> 00:10:02,460 The setting of beta equals one corresponds to the F1 score 148 00:10:02,460 --> 00:10:08,284 special case that we just saw that weights precision and recall equally. 149 00:10:08,284 --> 00:10:10,740 Let's take a look now at how we can compute 150 00:10:10,740 --> 00:10:17,679 these evaluation metrics in Python using scikit-learn. 151 00:10:17,679 --> 00:10:21,684 Scikit-learn metrics provides functions for computing accuracy, 152 00:10:21,684 --> 00:10:27,235 precision, recall, and F1 score as shown here in the notebook. 153 00:10:27,235 --> 00:10:29,320 The input to these functions is the same. 154 00:10:29,320 --> 00:10:32,870 The first argument here, y_test, 155 00:10:32,870 --> 00:10:36,759 is the array of true labels of the test set data instances 156 00:10:36,759 --> 00:10:42,549 and the second argument is the array of predicted labels for the test set data instances. 157 00:10:42,549 --> 00:10:46,225 Here we're using a variable called tree_predicted which are 158 00:10:46,225 --> 00:10:53,039 the predicted labels using the decision tree classifier in the previous notebook step. 159 00:10:53,039 --> 00:10:55,440 It's often useful when analyzing 160 00:10:55,440 --> 00:10:59,350 classifier performance to compute all of these metrics at once. 161 00:10:59,350 --> 00:11:05,309 So, sklearn metrics provides a handy classification report function. 162 00:11:05,309 --> 00:11:06,659 Like the previous core functions, 163 00:11:06,659 --> 00:11:08,879 classification report takes the true and 164 00:11:08,879 --> 00:11:13,309 predicted labels as the first two required arguments. 165 00:11:13,309 --> 00:11:17,820 It also takes some optional arguments that control the format of the output. 166 00:11:17,820 --> 00:11:24,894 Here, we use the target names option to label the classes in the output table. 167 00:11:24,894 --> 00:11:27,600 You can take a look at the scikit-learn documentation for 168 00:11:27,600 --> 00:11:31,705 more information on the other output options. 169 00:11:31,705 --> 00:11:33,495 The last column support, 170 00:11:33,495 --> 00:11:40,125 shows the number of instances in the test set that have that true label. 171 00:11:40,125 --> 00:11:42,759 Here we show classification reports for 172 00:11:42,759 --> 00:11:48,309 four different classifiers on the binary digit classification problem. 173 00:11:48,309 --> 00:11:51,970 The first set of results is from the dummy classifier and we can see 174 00:11:51,970 --> 00:11:55,779 that as expected both precision and recall for the positive class are very 175 00:11:55,779 --> 00:11:58,779 low since the dummy classifier is simply guessing 176 00:11:58,779 --> 00:12:00,970 randomly with low probability of 177 00:12:00,970 --> 00:12:05,000 predicting that positive class for the positive instances.