1 00:00:07,320 --> 00:00:11,714 Decision trees are a popular supervised learning method 2 00:00:11,714 --> 00:00:14,320 that like many other learning methods we've seen, 3 00:00:14,320 --> 00:00:17,559 can be used for both regression and classification. 4 00:00:17,559 --> 00:00:20,980 Decision trees are easy to use and understand and are often 5 00:00:20,980 --> 00:00:23,320 a good exploratory method if you're interested in getting 6 00:00:23,320 --> 00:00:27,250 a better idea about what the influential features are in your dataset. 7 00:00:27,250 --> 00:00:31,660 Basically, decision trees learn a series of explicit if then rules 8 00:00:31,660 --> 00:00:36,094 on feature values that result in a decision that predicts the target value. 9 00:00:36,094 --> 00:00:38,159 Here's a simple example. 10 00:00:38,159 --> 00:00:40,630 Suppose we're playing a game where one person is thinking of 11 00:00:40,630 --> 00:00:43,734 one of several possible objects so let's say, 12 00:00:43,734 --> 00:00:45,729 an automobile, a bus, 13 00:00:45,729 --> 00:00:47,240 an airplane, a bird, 14 00:00:47,240 --> 00:00:49,479 an elephant and a dog. 15 00:00:49,479 --> 00:00:53,965 A second person has to guess the identity of that thing by asking as few 16 00:00:53,965 --> 00:01:00,369 yes or no questions as possible up to a limit of no more than say, 10 questions. 17 00:01:00,369 --> 00:01:03,564 Suppose the first person secretly thinks of an automobile. 18 00:01:03,564 --> 00:01:09,484 So one of the first questions the second person might ask is, "Is it alive?" 19 00:01:09,484 --> 00:01:12,670 Intuitively, we know that this type of broad question is much more 20 00:01:12,670 --> 00:01:17,379 informative and helps eliminate a larger set of possibilities early on, 21 00:01:17,379 --> 00:01:21,010 compared to asking a more specific question right away like, 22 00:01:21,010 --> 00:01:25,069 "does it have orange fur with black stripes?" 23 00:01:25,069 --> 00:01:28,810 The more specific questions might be useful later once we've narrowed the class of 24 00:01:28,810 --> 00:01:33,625 things down to particular animals or vehicles, for example. 25 00:01:33,625 --> 00:01:37,540 If we think of the property of being alive as a binary feature of 26 00:01:37,540 --> 00:01:42,844 an object and the property of having orange fur with black stripes as another feature, 27 00:01:42,844 --> 00:01:46,280 we can say that the is a live feature is more informative at 28 00:01:46,280 --> 00:01:51,200 an early stage of guessing and thus would appear higher in our tree of questions. 29 00:01:51,200 --> 00:01:56,219 When the answer to the first question comes back as "No, it's not alive". 30 00:01:56,219 --> 00:01:59,239 A second question might be, "Does it fly?. 31 00:01:59,239 --> 00:02:01,439 When that answer comes back, no. 32 00:02:01,439 --> 00:02:03,515 We can continue to narrow down the set of 33 00:02:03,515 --> 00:02:08,300 possible answers by asking more and more specific questions such as, 34 00:02:08,300 --> 00:02:10,485 "Can it carry more than 10 people?". 35 00:02:10,485 --> 00:02:14,254 In this way any given object can be categorized as either 36 00:02:14,254 --> 00:02:18,110 matching the target object the first person is thinking of or not, 37 00:02:18,110 --> 00:02:24,134 according to its features as determined by asking the series of yes or no questions. 38 00:02:24,134 --> 00:02:30,590 We can form these questions into a tree with a node representing one question and the yes 39 00:02:30,590 --> 00:02:33,530 or no possible answers as the left and right branches 40 00:02:33,530 --> 00:02:37,219 from that node that connect the node to the next level of the tree. 41 00:02:37,219 --> 00:02:40,235 One question being answered at each level. 42 00:02:40,235 --> 00:02:43,669 At the bottom of the tree are nodes called leaf nodes that 43 00:02:43,669 --> 00:02:47,044 represent actual objects as the possible answers. 44 00:02:47,044 --> 00:02:51,139 For any object there's a path from the root of the tree to that object that is 45 00:02:51,139 --> 00:02:55,509 determined by the answers to the specific yes or no questions at each level. 46 00:02:55,509 --> 00:03:02,090 For example a dog is alive cannot fly and doesn't have a trunk. 47 00:03:02,090 --> 00:03:03,544 You can view this as a kind of 48 00:03:03,544 --> 00:03:07,254 simple decision tree for predicting the class of an object. 49 00:03:07,254 --> 00:03:11,150 We can generalize this idea of finding a set of rules that can learn to 50 00:03:11,150 --> 00:03:16,699 categorize an object into the correct category to many other classification tasks. 51 00:03:16,699 --> 00:03:20,150 For example, we're going to look at a classification task next that 52 00:03:20,150 --> 00:03:24,455 involves finding rules that can predict what species a particular flower is, 53 00:03:24,455 --> 00:03:27,780 based on measurements of certain parts of the flower. 54 00:03:27,780 --> 00:03:31,669 Rather than try to figure out these rules manually for every task, 55 00:03:31,669 --> 00:03:34,460 there are supervised algorithms that can learn them for 56 00:03:34,460 --> 00:03:37,235 us in a way that gets to an accurate decision quickly, 57 00:03:37,235 --> 00:03:39,134 which we'll look at now. 58 00:03:39,134 --> 00:03:45,409 Let's start by looking at a famous dataset in machine learning called the iris dataset. 59 00:03:45,409 --> 00:03:50,960 This dataset is included with scikit-learn in the datasets module. 60 00:03:50,960 --> 00:03:53,300 Each instance in the dataset represents one of 61 00:03:53,300 --> 00:03:56,990 three different species of Iris, a type of flower. 62 00:03:56,990 --> 00:04:00,634 There are four attributes or features for each instance, 63 00:04:00,634 --> 00:04:04,340 that represent measurements of different parts of the flower. 64 00:04:04,340 --> 00:04:06,514 The sepal length in centimeters, 65 00:04:06,514 --> 00:04:08,840 the sepal width in centimeters, 66 00:04:08,840 --> 00:04:12,780 petal length in centimeters and petal width in centimeters. 67 00:04:12,780 --> 00:04:15,189 The three species are called setosa, 68 00:04:15,189 --> 00:04:18,740 the versicolor and virginica. 69 00:04:18,740 --> 00:04:25,519 The data set has measurements for 150 flowers with 50 examples of each species. 70 00:04:25,519 --> 00:04:28,040 The classification task is to predict which of 71 00:04:28,040 --> 00:04:32,124 the three species and instances given these measurements. 72 00:04:32,124 --> 00:04:34,889 Unlike our simple example involving yes or 73 00:04:34,889 --> 00:04:38,384 no questions which were binary features of the object. 74 00:04:38,384 --> 00:04:41,850 The iris data sets features are continuous values. 75 00:04:41,850 --> 00:04:45,149 So the rules to be learned will be of the form for example, 76 00:04:45,149 --> 00:04:49,360 "Is Petal length greater than 2.3 centimeters?". 77 00:04:49,360 --> 00:04:53,964 Let's take a look at how a decision tree is built on this data. 78 00:04:53,964 --> 00:04:58,089 The goal when building a decision tree is to find the sequence of 79 00:04:58,089 --> 00:05:03,855 questions that has the best accuracy at classifying the data in the fewest steps. 80 00:05:03,855 --> 00:05:05,064 Looking at a decision tree, 81 00:05:05,064 --> 00:05:08,004 each decision splits the data into two branches 82 00:05:08,004 --> 00:05:12,345 based on some feature value being above or below a threshold. 83 00:05:12,345 --> 00:05:17,170 For example, whether the petal width is greater than one point two centimeters might 84 00:05:17,170 --> 00:05:24,279 be an example of a splitting rule that threshold is called a split point. 85 00:05:24,279 --> 00:05:28,879 An important concept is how informative a split of the data is. 86 00:05:28,879 --> 00:05:32,379 So intuitively an informative split of the data is one that does 87 00:05:32,379 --> 00:05:37,029 an excellent job at separating one class from the others. 88 00:05:37,029 --> 00:05:41,709 An example of an informative split might be to put all instances of the flowers with 89 00:05:41,709 --> 00:05:48,389 petal length less than 2.35 centimeters into one bin and the rest in the other bin. 90 00:05:48,389 --> 00:05:52,360 This is an informative split because at least for this training set, 91 00:05:52,360 --> 00:05:57,519 using that rule separates the setosa class completely from the others and 92 00:05:57,519 --> 00:06:03,225 allows us to predict the setosa class perfectly just based on this one measurement. 93 00:06:03,225 --> 00:06:06,120 A less informative split. 94 00:06:06,120 --> 00:06:09,915 Like a rule like sepal width less than or equal to three centimeters, 95 00:06:09,915 --> 00:06:15,230 would not produce as clear a separation of one class from the others. 96 00:06:15,230 --> 00:06:16,350 So for the best split, 97 00:06:16,350 --> 00:06:22,170 the results should produce as homogeneous a set of classes as possible. 98 00:06:22,170 --> 00:06:26,509 There are a number of mathematical ways to compute the best split. 99 00:06:26,509 --> 00:06:29,420 One criterion that's widely used for decision trees is 100 00:06:29,420 --> 00:06:33,401 called information game, for example. 101 00:06:33,401 --> 00:06:35,485 So to build the decision tree, 102 00:06:35,485 --> 00:06:38,620 the decision tree building algorithm starts by finding 103 00:06:38,620 --> 00:06:42,519 the feature that leads to the most informative split. 104 00:06:42,519 --> 00:06:46,250 For any given split of the data on a particular feature value, 105 00:06:46,250 --> 00:06:49,220 even for the best split it's likely that some examples will 106 00:06:49,220 --> 00:06:52,584 still be incorrectly classified or need further splitting. 107 00:06:52,584 --> 00:06:54,509 In the iris data set, 108 00:06:54,509 --> 00:06:56,810 if we look at all the flowers with petal length 109 00:06:56,810 --> 00:07:00,004 greater than 2.35 centimeters for example. 110 00:07:00,004 --> 00:07:04,160 Using that split leaves a pool of instances that are a combination 111 00:07:04,160 --> 00:07:09,060 of virginica and versicolor that still need to be distinguished further. 112 00:07:09,060 --> 00:07:11,360 So we can further improve the accuracy of 113 00:07:11,360 --> 00:07:14,149 the classification by continuing this process of 114 00:07:14,149 --> 00:07:19,189 finding the best split for the remaining subsets with the iris dataset. 115 00:07:19,189 --> 00:07:22,910 We can further split into flowers that have petal length less 116 00:07:22,910 --> 00:07:27,879 than versus greater than 4.95 centimeters for example. 117 00:07:27,879 --> 00:07:33,740 This second split is informative because the resulting subsets are more homogeneous that 118 00:07:33,740 --> 00:07:36,319 is split did a good job at dividing 119 00:07:36,319 --> 00:07:40,970 virginica from versicolor based on the second split test. 120 00:07:40,970 --> 00:07:45,589 We can continue this process recursively until we're left with leaves in 121 00:07:45,589 --> 00:07:48,035 the decision tree that all have the same 122 00:07:48,035 --> 00:07:53,120 or at least a predominant majority of a target value. 123 00:07:53,120 --> 00:07:57,740 Trees whose leaf nodes each have all the same target value are called pure, 124 00:07:57,740 --> 00:08:00,769 as opposed to mixed where the leaf nodes are 125 00:08:00,769 --> 00:08:04,654 allowed to contain at least some mixture of the classes. 126 00:08:04,654 --> 00:08:09,185 To predict the class of a new instance given its feature measurements, 127 00:08:09,185 --> 00:08:13,939 using the decision tree we simply start at the root of the decision tree and take 128 00:08:13,939 --> 00:08:16,160 the decision at each level based on 129 00:08:16,160 --> 00:08:19,725 the appropriate feature measurement until we get to a leafnode. 130 00:08:19,725 --> 00:08:27,129 The prediction is then just the majority class of the instances in that leafnode. 131 00:08:27,129 --> 00:08:29,084 So for the iris data for example, 132 00:08:29,084 --> 00:08:31,949 a flower that has a petal length of three centimeters, 133 00:08:31,949 --> 00:08:34,620 a petal width of two centimeters and a sepal width of 134 00:08:34,620 --> 00:08:37,735 two centimeters would end up at this leafnode, 135 00:08:37,735 --> 00:08:40,799 whose instances are all of the virginica class. 136 00:08:40,799 --> 00:08:42,809 So the prediction would be virginica. 137 00:08:42,809 --> 00:08:48,009 Decision trees can also be used for regression using the same process of testing 138 00:08:48,009 --> 00:08:50,470 the future values at each node and predicting 139 00:08:50,470 --> 00:08:53,769 the target value based on the contents of the leafnode. 140 00:08:53,769 --> 00:08:56,409 For regression, the leafnode prediction would be 141 00:08:56,409 --> 00:09:00,309 the mean value of the target values for the training points in that leaf. 142 00:09:00,309 --> 00:09:03,460 In scikit-learn, to build the decision tree you 143 00:09:03,460 --> 00:09:06,340 import the decision tree classifier class from 144 00:09:06,340 --> 00:09:09,789 the Sklearn tree module and fit it just as you would 145 00:09:09,789 --> 00:09:15,365 any classifier by creating the object and calling the fit method on the training data. 146 00:09:15,365 --> 00:09:18,789 This code first loads the iris data set and then computes 147 00:09:18,789 --> 00:09:23,304 the training and test accuracy in predicting the three classes. 148 00:09:23,304 --> 00:09:27,750 Notice that the training data here is predicted perfectly with an accuracy of 1.0. 149 00:09:27,750 --> 00:09:30,294 While the test data is a little bit worse. 150 00:09:30,294 --> 00:09:32,649 This is an indication that the tree is likely 151 00:09:32,649 --> 00:09:35,080 overfitting and in fact this is a problem with 152 00:09:35,080 --> 00:09:40,414 building decision trees in general that keep adding rules until the leafnodes are pure. 153 00:09:40,414 --> 00:09:45,154 Typically such trees are overly complex and essentially memorized the training data. 154 00:09:45,154 --> 00:09:46,884 So when building decision trees, 155 00:09:46,884 --> 00:09:51,680 we need to use some additional strategy to prevent this overfitting. 156 00:09:51,680 --> 00:09:56,149 One strategy to prevent overfitting is to prevent the tree from 157 00:09:56,149 --> 00:10:00,534 becoming really detailed and complex by stopping its growth early. 158 00:10:00,534 --> 00:10:03,330 This is called pre-pruning. 159 00:10:03,330 --> 00:10:05,629 Another strategy is to build a complete tree with 160 00:10:05,629 --> 00:10:09,580 pure leaves but then to prune back the tree into a simpler form. 161 00:10:09,580 --> 00:10:13,294 This is called post-pruning or sometimes just pruning. 162 00:10:13,294 --> 00:10:19,950 The decision tree implementation and scikit-learn only implements pre-pruning. 163 00:10:19,950 --> 00:10:22,279 We can control tree complexity via pruning by 164 00:10:22,279 --> 00:10:25,370 limiting either the maximum depth of the tree using 165 00:10:25,370 --> 00:10:27,500 the max depth parameter or 166 00:10:27,500 --> 00:10:32,029 the maximum number of leafnodes using the max leafnodes parameter. 167 00:10:32,029 --> 00:10:34,519 We could also set a threshold on the minimum number of 168 00:10:34,519 --> 00:10:38,240 instances that must be in a node to consider splitting it. 169 00:10:38,240 --> 00:10:42,440 And this would be using the min samples leaf parameter we can 170 00:10:42,440 --> 00:10:47,684 see the effect of pre-pruning by setting max depth to three on the iris dataset. 171 00:10:47,684 --> 00:10:49,909 Now the accuracy on the training data is slightly 172 00:10:49,909 --> 00:10:54,014 worse but the accuracy on the test data is slightly better. 173 00:10:54,014 --> 00:10:57,500 One great advantage of decision trees at least for the not too big, 174 00:10:57,500 --> 00:11:00,070 is that they're easy to interpret. 175 00:11:00,070 --> 00:11:02,509 Visualizing the entire tree can show you 176 00:11:02,509 --> 00:11:05,679 exactly how did the decision tree is making its predictions. 177 00:11:05,679 --> 00:11:08,304 So let's take a look at an example. 178 00:11:08,304 --> 00:11:12,375 In the shared utilities Python code for this course, 179 00:11:12,375 --> 00:11:16,080 we've provided a function call named Plot decision tree, 180 00:11:16,080 --> 00:11:17,850 that takes the classifier object, 181 00:11:17,850 --> 00:11:20,399 the feature names, and the class names as input. 182 00:11:20,399 --> 00:11:24,840 And uses the graphics library to visualize the tree. 183 00:11:24,840 --> 00:11:28,860 It works by calling the export graph its function in the scikit-learn 184 00:11:28,860 --> 00:11:34,125 tree module to create a dot file which is a text file description of the tree, 185 00:11:34,125 --> 00:11:39,684 and then using the graphics library to visualize the dot file creating an image. 186 00:11:39,684 --> 00:11:43,169 Here's an example of how to call the plot decision tree function using 187 00:11:43,169 --> 00:11:48,144 the classifier we learned on the iris dataset. 188 00:11:48,144 --> 00:11:51,960 Here's the resulting plot for the unpruned Iris dataset tree. 189 00:11:51,960 --> 00:11:56,860 The first line in a node indicates the decision rule being applied for that node. 190 00:11:56,860 --> 00:12:01,342 The second line indicates the total number of data instances for that node. 191 00:12:01,342 --> 00:12:05,490 The third line shows the class distribution among those instances. 192 00:12:05,490 --> 00:12:10,995 And the fourth line shows the majority class of that nodes data instances. 193 00:12:10,995 --> 00:12:18,039 For example in the root node the initial pool of data has 37 setosa examples, 194 00:12:18,039 --> 00:12:22,504 34 versicolor examples, and 41 virginica examples. 195 00:12:22,504 --> 00:12:27,549 After the first split into subsets that have petal length less than or equal to 2.35 on 196 00:12:27,549 --> 00:12:33,610 the left and those that have petal length greater than 2.35 on the right, 197 00:12:33,610 --> 00:12:38,544 we have a leafnode on the left that consists entirely of the 37 setosa examples 198 00:12:38,544 --> 00:12:45,080 and a decision note on the right that has 34 versicolor and 41 virginica examples. 199 00:12:45,080 --> 00:12:48,549 This node on the right then makes a second split based on 200 00:12:48,549 --> 00:12:53,289 petal length using a threshold of 4.95 centimeters and that creates 201 00:12:53,289 --> 00:12:56,860 two subsets of 36 samples on the left containing 202 00:12:56,860 --> 00:13:01,375 33 versicolor examples and 39 samples on the right, 203 00:13:01,375 --> 00:13:03,810 38 of which are virginica examples. 204 00:13:03,810 --> 00:13:08,110 We've used an option when plotting the decision tree to fill 205 00:13:08,110 --> 00:13:09,789 the nodes with colors that indicate 206 00:13:09,789 --> 00:13:13,899 the majority class of the data instances associated with that node. 207 00:13:13,899 --> 00:13:20,820 The intensity of the color indicates the strength of the majority count for that class. 208 00:13:20,820 --> 00:13:25,429 For larger trees that have say a depth of more than 5 or 10. 209 00:13:25,429 --> 00:13:28,419 Instead of trying to analyze all the paths in the tree it can be 210 00:13:28,419 --> 00:13:32,215 useful to see which paths most of the data takes. 211 00:13:32,215 --> 00:13:36,970 This can be done by looking for the largest samples values in the nodes. 212 00:13:36,970 --> 00:13:40,570 For example if we follow the largest samples values down this tree, 213 00:13:40,570 --> 00:13:44,860 we can see that a significant set of 35 virginica examples are classified 214 00:13:44,860 --> 00:13:51,919 perfectly when their petal length is greater than 5.05 centimeters. 215 00:13:51,919 --> 00:13:54,769 Another way of analyzing the tree instead of looking at the whole tree at 216 00:13:54,769 --> 00:13:59,600 once is to do what's called a feature important calculation. 217 00:13:59,600 --> 00:14:02,149 And this is one of the most useful and widely used types of 218 00:14:02,149 --> 00:14:05,860 summary analysis you can perform on a supervised learning model. 219 00:14:05,860 --> 00:14:08,629 Feature importance is typically a number 220 00:14:08,629 --> 00:14:12,605 between 0 and 1 that's assigned to an individual feature. 221 00:14:12,605 --> 00:14:17,269 It indicates how important that feature is to the overall prediction accuracy. 222 00:14:17,269 --> 00:14:22,679 A feature importance of zero means that the feature is not used at all in the prediction. 223 00:14:22,679 --> 00:14:24,830 A feature importance of one, 224 00:14:24,830 --> 00:14:28,009 means the feature perfectly predicts the target. 225 00:14:28,009 --> 00:14:30,830 Typically, feature importance numbers are always 226 00:14:30,830 --> 00:14:34,605 positive and they're normalized so they sum to one. 227 00:14:34,605 --> 00:14:39,490 In scikit-learn, feature importance values are stored as a list in 228 00:14:39,490 --> 00:14:44,328 an estimated property called feature_importances_. 229 00:14:44,328 --> 00:14:46,149 And note the underscore at the end of 230 00:14:46,149 --> 00:14:48,970 the name which indicates it's a property of the object 231 00:14:48,970 --> 00:14:54,835 that's set as a result of fitting the model and not say as a user defined property. 232 00:14:54,835 --> 00:15:00,009 The shared utilities python file contains a function called plot feature importances, 233 00:15:00,009 --> 00:15:04,139 that you can import and use to visualize future importance. 234 00:15:04,139 --> 00:15:07,179 It plots a horizontal bar chart with the features listed along 235 00:15:07,179 --> 00:15:12,819 the y axis by name and feature importance along the x axis. 236 00:15:12,819 --> 00:15:17,200 Here's the feature importance chart for the iris decision tree 237 00:15:17,200 --> 00:15:22,144 and this example for this particular train/test split of the iris data set. 238 00:15:22,144 --> 00:15:27,507 The pedal length feature easily has the largest feature importance weight. 239 00:15:27,507 --> 00:15:30,514 We can confirm this by looking at the decision tree 240 00:15:30,514 --> 00:15:32,389 that this is indeed corresponds to that 241 00:15:32,389 --> 00:15:34,909 features position at the top of the decision tree, 242 00:15:34,909 --> 00:15:38,809 showing that this first level just using the petal length feature does 243 00:15:38,809 --> 00:15:43,725 a good job splitting the turning data into separate classes. 244 00:15:43,725 --> 00:15:48,044 Note that if a feature has a low feature importance value, 245 00:15:48,044 --> 00:15:52,100 that doesn't necessarily mean that the feature is not important for prediction. 246 00:15:52,100 --> 00:15:55,200 It simply means that the particular feature wasn't chosen at 247 00:15:55,200 --> 00:15:58,679 an early level of the tree and this could be because the future may be 248 00:15:58,679 --> 00:16:00,960 identical or highly correlated with 249 00:16:00,960 --> 00:16:03,389 another informative feature and so 250 00:16:03,389 --> 00:16:06,669 doesn't provide any new additional signal for prediction. 251 00:16:06,669 --> 00:16:09,580 Feature importance values don't tell us which specific 252 00:16:09,580 --> 00:16:12,445 classes a feature might be especially predictive for, 253 00:16:12,445 --> 00:16:13,840 and they also don't indicate 254 00:16:13,840 --> 00:16:18,065 more complex relationships between features that may influence prediction. 255 00:16:18,065 --> 00:16:22,254 Even so the feature importance values provide an easy to understand summary 256 00:16:22,254 --> 00:16:27,159 that can give useful insight about individual features in the learning model. 257 00:16:27,159 --> 00:16:29,950 Because feature importance can vary depending on 258 00:16:29,950 --> 00:16:34,090 the specific model learned for a particular train/test split for example. 259 00:16:34,090 --> 00:16:37,090 It's common when computing feature importance to use 260 00:16:37,090 --> 00:16:39,965 an average over multiple train/test splits. 261 00:16:39,965 --> 00:16:43,965 For example when performing cross-validation. 262 00:16:43,965 --> 00:16:47,289 Finally let's apply decision trees to the breast cancer data 263 00:16:47,289 --> 00:16:52,293 set that we've been using across multiple supervised learning algorithms. 264 00:16:52,293 --> 00:16:54,889 Here we're controlling them all the complexity by 265 00:16:54,889 --> 00:16:58,365 setting the max depth and min samples leaf parameters. 266 00:16:58,365 --> 00:17:01,184 And we've included the visualization of the resulting tree 267 00:17:01,184 --> 00:17:04,549 here followed by a feature importance plot. 268 00:17:04,549 --> 00:17:08,450 As an exercise, try removing these two parameters to just use 269 00:17:08,450 --> 00:17:11,174 the default settings to see the effect on 270 00:17:11,174 --> 00:17:15,274 test set accuracy and the increase in overfitting that occurs. 271 00:17:15,274 --> 00:17:17,359 For this training set, 272 00:17:17,359 --> 00:17:21,904 the mean concave points feature gives the most informative initial split, 273 00:17:21,904 --> 00:17:27,740 followed by the worst area parameter. 274 00:17:27,740 --> 00:17:31,745 One of the major advantages of decision trees as a supervised learning method, 275 00:17:31,745 --> 00:17:34,730 is that the decision rules are easily visualized and interpreted 276 00:17:34,730 --> 00:17:39,130 by people including users without machine learning expertise. 277 00:17:39,130 --> 00:17:43,190 This makes decision trees a useful choice for getting an initial understanding of what 278 00:17:43,190 --> 00:17:48,025 some of the more important features are likely to be for a particular prediction task. 279 00:17:48,025 --> 00:17:51,059 Another advantage of decision trees is that you can use them 280 00:17:51,059 --> 00:17:54,555 without having to do feature pre-processing or normalization. 281 00:17:54,555 --> 00:17:57,779 Since each feature is processed independently and 282 00:17:57,779 --> 00:18:01,589 the splitting of the data doesn't depend on the absolute scale of the feature. 283 00:18:01,589 --> 00:18:06,555 The decision algorithm can operate on the original training data pretty much as is. 284 00:18:06,555 --> 00:18:08,700 So decision trees tend to work well with 285 00:18:08,700 --> 00:18:11,475 data sets that have a mixture of feature types-- binary, 286 00:18:11,475 --> 00:18:17,059 categorical or continuous and with features that are on very different scales. 287 00:18:17,059 --> 00:18:22,924 One drawback of decision trees is that despite the use of pruning they can still overfit 288 00:18:22,924 --> 00:18:25,460 all or parts of the data and may not achieve 289 00:18:25,460 --> 00:18:30,039 the best generalization performance compared to other methods. 290 00:18:30,039 --> 00:18:32,585 One way to overcome that problem is to create what's called 291 00:18:32,585 --> 00:18:35,464 an ensemble of decision trees which combines 292 00:18:35,464 --> 00:18:38,525 multiple decision trees to make a prediction and will 293 00:18:38,525 --> 00:18:42,889 look at ensembles of decision trees in the following lecture. 294 00:18:42,889 --> 00:18:47,125 So to recap, scikit-learn enables you to control 295 00:18:47,125 --> 00:18:52,220 the model complexity of your decision trees with three key parameters. 296 00:18:52,220 --> 00:18:54,319 First, max depth controls 297 00:18:54,319 --> 00:18:57,319 the maximum depth or the number of split points the decision can 298 00:18:57,319 --> 00:19:00,019 have and it's probably the most common parameter 299 00:19:00,019 --> 00:19:04,109 used to reduce tree complexity and thus reduce overfitting. 300 00:19:04,109 --> 00:19:08,329 The min samples leaf parameter is the threshold that controls what 301 00:19:08,329 --> 00:19:13,400 the minimum number of data instances has to be in a leaf to avoid splitting it further. 302 00:19:13,400 --> 00:19:16,707 This setting also reduces tree complexity. 303 00:19:16,707 --> 00:19:22,164 Finally, max leaf nodes limits the total number of nodes that are leaves of the tree. 304 00:19:22,164 --> 00:19:24,569 In effect, setting this parameter will indirectly 305 00:19:24,569 --> 00:19:28,095 influence the other two parameters and vice versa. 306 00:19:28,095 --> 00:19:30,240 So in practice, adjusting only one of 307 00:19:30,240 --> 00:19:33,404 these trees is typically enough to control most overfitting. 308 00:19:33,404 --> 00:19:36,599 Although even with the most optimized parameter settings, 309 00:19:36,599 --> 00:19:40,000 individual decision trees will still tend to overfit.