1 00:00:07,280 --> 00:00:14,671 Now we'll look at another way to estimate WNB for linear model, called Ridge Regression. 2 00:00:14,671 --> 00:00:18,545 Ridge regression uses the same least-squares criterion, 3 00:00:18,545 --> 00:00:21,000 but with one difference. 4 00:00:21,000 --> 00:00:22,184 During the training phase, 5 00:00:22,184 --> 00:00:24,480 it adds a penalty for feature weights, 6 00:00:24,480 --> 00:00:29,625 the WI values that are too large as shown in the equation here. 7 00:00:29,625 --> 00:00:33,060 You'll see that large weights means mathematically that 8 00:00:33,060 --> 00:00:37,990 the sum of their squared values is large. 9 00:00:37,990 --> 00:00:43,710 Once ridge regression has estimated the WNB parameters for the linear model, 10 00:00:43,710 --> 00:00:48,329 the prediction of Y values for new instances is exactly the same as in least squares. 11 00:00:48,329 --> 00:00:50,519 You just plug in your input feature values, 12 00:00:50,519 --> 00:00:53,130 the XIs and compute the sum of 13 00:00:53,130 --> 00:00:57,990 the weighted feature values plus B with the usual in your formula. 14 00:00:57,990 --> 00:01:01,585 So why would something like ridge regression be useful? 15 00:01:01,585 --> 00:01:04,079 This addition of a penalty term to 16 00:01:04,079 --> 00:01:08,909 a learning algorithm's objective function is called Regularisation. 17 00:01:08,909 --> 00:01:13,765 Regularisation is an extremely important concept in machine learning. 18 00:01:13,765 --> 00:01:16,724 It's a way to prevent overfitting, and thus, 19 00:01:16,724 --> 00:01:19,995 improve the likely generalization performance of a model, 20 00:01:19,995 --> 00:01:23,569 by restricting the models possible parameter settings. 21 00:01:23,569 --> 00:01:27,045 Usually the effect of this restriction from regularisation, 22 00:01:27,045 --> 00:01:31,750 is to reduce the complexity of the final estimated model. 23 00:01:31,750 --> 00:01:34,464 So how does this work with linear regression? 24 00:01:34,464 --> 00:01:38,694 The addition of the sum of squared parameter values that's shown in the box, 25 00:01:38,694 --> 00:01:41,670 to the least-squares objective means that models with 26 00:01:41,670 --> 00:01:48,114 larger feature weights (w) add more to the objective functions overall value. 27 00:01:48,114 --> 00:01:51,864 Because our goal is to minimize the overall objective function, 28 00:01:51,864 --> 00:01:55,008 the regularisation term acts as a penalty of 29 00:01:55,008 --> 00:01:59,359 models with lots of large feature weight values. 30 00:01:59,359 --> 00:02:01,215 In other words, all things being equal, 31 00:02:01,215 --> 00:02:02,605 if ridge regression finds 32 00:02:02,605 --> 00:02:07,125 two possible linear models that predict the training data values equally well, 33 00:02:07,125 --> 00:02:09,495 it will prefer the linear model that has 34 00:02:09,495 --> 00:02:13,810 a smaller overall sum of squared feature weights. 35 00:02:13,810 --> 00:02:16,439 The practical effect of using ridge regression, 36 00:02:16,439 --> 00:02:21,895 is to find the feature weights WI that fit the data well in at least square sense, 37 00:02:21,895 --> 00:02:27,030 and that set lots of the feature weights two values that are very small. 38 00:02:27,030 --> 00:02:31,000 We don't see this effect with a single variable linear regression example, 39 00:02:31,000 --> 00:02:34,694 but for regression problems with dozens or hundreds of features, 40 00:02:34,694 --> 00:02:36,780 the accuracy improvement from using 41 00:02:36,780 --> 00:02:42,469 regularized linear regression like ridge regression could be significant. 42 00:02:42,469 --> 00:02:47,599 The amount of regularisation to apply is controlled by the alpha parameter. 43 00:02:47,599 --> 00:02:50,150 Larger alpha means more regularization and 44 00:02:50,150 --> 00:02:54,039 simpler linear models with weights closer to zero. 45 00:02:54,039 --> 00:02:57,824 The default setting for alpha is 1.0. 46 00:02:57,824 --> 00:03:01,625 Notice that setting alpha to zero corresponds to the special case of 47 00:03:01,625 --> 00:03:06,169 ordinary least-squares linear regression that we saw earlier, 48 00:03:06,169 --> 00:03:07,909 that minimizes the total square here. 49 00:03:07,909 --> 00:03:11,449 In scikit learn, you use rich regression by importing 50 00:03:11,449 --> 00:03:15,590 the ridge class from sklearn.linear model. 51 00:03:15,590 --> 00:03:20,780 And then use that estimate or object just as you would for least-squares. 52 00:03:20,780 --> 00:03:23,180 The one difference is that you can specify 53 00:03:23,180 --> 00:03:26,344 the amount of the ridge regression regularisation penalty, 54 00:03:26,344 --> 00:03:28,669 which is called the L2 penalty, 55 00:03:28,669 --> 00:03:31,250 using the alpha parameter. 56 00:03:31,250 --> 00:03:35,419 Here, we're applying ridge regression to the crime data set. 57 00:03:35,419 --> 00:03:38,330 Now, you'll notice here that the results are not that impressive. 58 00:03:38,330 --> 00:03:40,759 The R-squared score on the test set is pretty 59 00:03:40,759 --> 00:03:44,974 comparable to what we got for least-squares regression. 60 00:03:44,974 --> 00:03:47,870 However there is something we can do in 61 00:03:47,870 --> 00:03:52,139 applying ridge regression that will improve the results dramatically. 62 00:03:52,139 --> 00:03:54,944 So now is the time for a brief digression about the need for 63 00:03:54,944 --> 00:03:58,965 feature preprocessing and normalization. 64 00:03:58,965 --> 00:04:00,800 Let's stop and think for a moment intuitively, 65 00:04:00,800 --> 00:04:03,074 what ridge regression is doing. 66 00:04:03,074 --> 00:04:06,379 It's regularizing the linear regression by imposing 67 00:04:06,379 --> 00:04:11,180 that sum of squares penalty on the size of the W coefficients. 68 00:04:11,180 --> 00:04:13,939 So the effect of increasing alpha is to 69 00:04:13,939 --> 00:04:18,064 shrink the AW coefficients toward zero and towards each other. 70 00:04:18,064 --> 00:04:20,300 But if the input variables, the features, 71 00:04:20,300 --> 00:04:22,235 have very different scales, 72 00:04:22,235 --> 00:04:26,105 then when this shrinkage happens of the coefficients, 73 00:04:26,105 --> 00:04:28,595 input variables with different scales will have 74 00:04:28,595 --> 00:04:31,395 different contributions to this L2 penalty, 75 00:04:31,395 --> 00:04:35,519 because the L2 penalty is a sum of squares of all the coefficients. 76 00:04:35,519 --> 00:04:37,819 So transforming the input features, 77 00:04:37,819 --> 00:04:39,350 so they're all on the same scale, 78 00:04:39,350 --> 00:04:44,300 means the ridge penalty is in some sense applied more fairly to all features 79 00:04:44,300 --> 00:04:49,900 without unduly weighting some more than others, 80 00:04:49,900 --> 00:04:53,310 just because of the difference in scales. 81 00:04:53,310 --> 00:04:57,899 So more generally, you'll see as we proceed through the course that feature 82 00:04:57,899 --> 00:05:01,814 normalization is important to perform for a number of different learning algorithms, 83 00:05:01,814 --> 00:05:04,615 beyond just regularized regression. 84 00:05:04,615 --> 00:05:06,660 This includes K-nearest neighbors, 85 00:05:06,660 --> 00:05:10,564 support vector machines, neural networks and others. 86 00:05:10,564 --> 00:05:12,509 The type of feature preprocessing and 87 00:05:12,509 --> 00:05:16,470 normalization that's needed can also depend on the data. 88 00:05:16,470 --> 00:05:19,410 For now, we're going to apply a widely used form of 89 00:05:19,410 --> 00:05:22,769 future normalization called MinMax Scaling, 90 00:05:22,769 --> 00:05:25,214 that will transform all the input variables, 91 00:05:25,214 --> 00:05:29,639 so they're all on the same scale between zero and one. 92 00:05:29,639 --> 00:05:31,170 To do this, we compute 93 00:05:31,170 --> 00:05:34,754 the minimum and maximum values for each feature on the training data, 94 00:05:34,754 --> 00:05:41,800 and then apply the minmax transformation for each feature as shown here. 95 00:05:41,800 --> 00:05:44,300 Here's an example of how it works with two features. 96 00:05:44,300 --> 00:05:47,779 Suppose we have one feature "height" whose values fall in 97 00:05:47,779 --> 00:05:52,154 a fairly narrow range between 1.5 and 2.5 units. 98 00:05:52,154 --> 00:05:53,404 But a second feature, 99 00:05:53,404 --> 00:05:57,725 "width" has a much wider range between five and 10 units. 100 00:05:57,725 --> 00:06:00,448 After applying minmax scaling, 101 00:06:00,448 --> 00:06:04,459 values for both features are transformed because they are on the same scale, 102 00:06:04,459 --> 00:06:07,235 with the minimum value getting mapped to zero, 103 00:06:07,235 --> 00:06:10,610 and the maximum value being transformed to one. 104 00:06:10,610 --> 00:06:15,230 And everything else getting transformed to a value between those two extremes. 105 00:06:15,230 --> 00:06:18,615 To apply minmax scaling, 106 00:06:18,615 --> 00:06:25,380 in scikit-learn, you import the minmax scalar object from sklearn.preprocessing. 107 00:06:25,380 --> 00:06:28,514 To prepare the scalar object for use, you create it, 108 00:06:28,514 --> 00:06:32,485 and then call the fit method using the training data Xtrain. 109 00:06:32,485 --> 00:06:34,740 This will compute the min and max feature values 110 00:06:34,740 --> 00:06:37,680 for each feature in this training dataset. 111 00:06:37,680 --> 00:06:39,509 Then to apply the scalar, 112 00:06:39,509 --> 00:06:41,144 you call it transform method, 113 00:06:41,144 --> 00:06:43,589 and pass in the data you want to rescale. 114 00:06:43,589 --> 00:06:47,459 The output will be the scale version of the input data. 115 00:06:47,459 --> 00:06:50,834 In this case, we want to scale the training data and save it 116 00:06:50,834 --> 00:06:53,962 in a new variable called Xtrain scaled. 117 00:06:53,962 --> 00:06:55,360 And the test data, 118 00:06:55,360 --> 00:06:58,710 saving that into a new variable called X-Tests-Scaled. 119 00:06:58,710 --> 00:07:00,750 Then, we just use 120 00:07:00,750 --> 00:07:05,595 these scaled versions of the feature data instead of the original feature data. 121 00:07:05,595 --> 00:07:08,779 Note that it could be more efficient to perform 122 00:07:08,779 --> 00:07:11,975 fitting and transforming in a single step on the training set, 123 00:07:11,975 --> 00:07:17,154 by using the scalers fit transform method as shown here. 124 00:07:17,154 --> 00:07:19,774 There's one last but very important point here, 125 00:07:19,774 --> 00:07:22,339 about how to apply minmax scaling or any kind of 126 00:07:22,339 --> 00:07:27,420 feature normalization in a learning scenario with training and test sets. 127 00:07:27,420 --> 00:07:29,735 You may have noticed two things here. 128 00:07:29,735 --> 00:07:35,404 First, that we're applying the same scalar object to both the training and the testing. 129 00:07:35,404 --> 00:07:37,189 And second, that we're training 130 00:07:37,189 --> 00:07:41,319 the scalar object on the training data and not on the test data. 131 00:07:41,319 --> 00:07:45,519 These are both critical aspects to feature normalization. 132 00:07:45,519 --> 00:07:49,920 If you don't apply the same scaling to training and test sets, 133 00:07:49,920 --> 00:07:52,514 you'll end up with more or less random data skew, 134 00:07:52,514 --> 00:07:54,959 which will invalidate your results. 135 00:07:54,959 --> 00:07:58,290 If you prepare the scaler or other normalization method by 136 00:07:58,290 --> 00:08:01,747 showing it the test data instead of the training data, 137 00:08:01,747 --> 00:08:04,840 this leads to a phenomenon called Data Leakage, 138 00:08:04,840 --> 00:08:08,894 where the training phase has information that is leaked from the test set. 139 00:08:08,894 --> 00:08:13,680 For example, like the distribution of extreme values for each feature in the test data, 140 00:08:13,680 --> 00:08:18,389 which the learner should never have access to during training. 141 00:08:18,389 --> 00:08:20,850 This in turn can cause the learning method to give 142 00:08:20,850 --> 00:08:24,060 unrealistically good estimates on the same test set. 143 00:08:24,060 --> 00:08:28,730 We'll look more at the phenomenon of data leakage later in the course. 144 00:08:28,730 --> 00:08:32,500 One downside to performing feature normalization is that 145 00:08:32,500 --> 00:08:36,950 the resulting model and the transformed features may be harder to interpret. 146 00:08:36,950 --> 00:08:38,200 Again, in the end, 147 00:08:38,200 --> 00:08:41,304 the type of feature normalization that's best to apply, 148 00:08:41,304 --> 00:08:43,003 can depend on the data set, 149 00:08:43,003 --> 00:08:47,330 learning task and learning algorithm to be used. 150 00:08:47,330 --> 00:08:50,610 We'll continue to touch on this issue throughout the course. 151 00:08:50,610 --> 00:08:54,210 Okay, let's return to 152 00:08:54,210 --> 00:09:00,139 ridge regression after we've added the code for minmax scaling of the input features. 153 00:09:00,139 --> 00:09:02,440 We can see the significant effect of 154 00:09:02,440 --> 00:09:05,850 minmax scaling on the performance of ridge regression. 155 00:09:05,850 --> 00:09:09,245 After the input features have been properly scaled, 156 00:09:09,245 --> 00:09:13,004 ridge regression achieves significantly better model fit 157 00:09:13,004 --> 00:09:17,485 with an R-squared value on the test set of about 0.6. 158 00:09:17,485 --> 00:09:19,184 Much better than without scaling, 159 00:09:19,184 --> 00:09:22,620 and much better now than ordinary least-squares. 160 00:09:22,620 --> 00:09:27,179 In fact if you apply the same minmax scaling with ordinary least-squares regression, 161 00:09:27,179 --> 00:09:29,820 you should find that it doesn't change the outcome at all. 162 00:09:29,820 --> 00:09:34,450 In general, regularisation works especially 163 00:09:34,450 --> 00:09:36,639 well when you have relatively small amounts of 164 00:09:36,639 --> 00:09:40,245 training data compared to the number of features in your model. 165 00:09:40,245 --> 00:09:47,289 Regularisation becomes less important as the amount of training data you have increases. 166 00:09:47,289 --> 00:09:49,389 We can see the effect of varying the amount of 167 00:09:49,389 --> 00:09:51,639 regularisation on the scale to training and 168 00:09:51,639 --> 00:09:56,693 test data using different settings for alpha in this example. 169 00:09:56,693 --> 00:10:02,724 The best R-squared value on the test set is achieved with an alpha setting of around 20. 170 00:10:02,724 --> 00:10:06,460 Significantly larger or smaller values of alpha, 171 00:10:06,460 --> 00:10:10,070 both lead to significantly worse model fit. 172 00:10:10,070 --> 00:10:12,919 This is another illustration of the general relationship between 173 00:10:12,919 --> 00:10:17,493 model complexity and test set performance that we saw earlier in this lecture. 174 00:10:17,493 --> 00:10:20,659 Where there's often an intermediate best value of a model of 175 00:10:20,659 --> 00:10:25,279 complexity parameter that does not lead to either under or overfitting. 176 00:10:25,279 --> 00:10:28,610 Another kind of regularized regression that you 177 00:10:28,610 --> 00:10:32,794 could use instead of ridge regression is called Lasso Regression. 178 00:10:32,794 --> 00:10:36,409 Like ridge regression, lasso regression adds 179 00:10:36,409 --> 00:10:40,203 a regularisation penalty term to the ordinary least-squares objective, 180 00:10:40,203 --> 00:10:44,620 that causes the model W-coefficients to shrink towards zero. 181 00:10:44,620 --> 00:10:51,041 Lasso regression uses a slightly different regularisation term called an L1 penalty, 182 00:10:51,041 --> 00:10:55,580 instead of ridge regression's L2 penalty as shown here. 183 00:10:55,580 --> 00:11:00,274 The L1 penalty looks kind of similar to the L2 penalty, 184 00:11:00,274 --> 00:11:04,695 in that it computes a sum over the coefficients but it's 185 00:11:04,695 --> 00:11:09,139 some of the absolute values of the W-coefficients instead of a sum of squares. 186 00:11:09,139 --> 00:11:11,855 And the results are noticeably different. 187 00:11:11,855 --> 00:11:17,220 With lasso regression, a subset of the coefficients are forced to be precisely zero. 188 00:11:17,220 --> 00:11:20,310 Which is a kind of automatic feature selection, 189 00:11:20,310 --> 00:11:22,302 since with the weight of zero the features are 190 00:11:22,302 --> 00:11:25,490 essentially ignored completely in the model. 191 00:11:25,490 --> 00:11:28,009 This sparse solution where only a subset of 192 00:11:28,009 --> 00:11:31,370 the most important features are left with non-zero weights, 193 00:11:31,370 --> 00:11:34,039 also makes the model easier to interpret. 194 00:11:34,039 --> 00:11:37,309 In cases where there are more than a few input variables. 195 00:11:37,309 --> 00:11:40,909 Like ridge regression, the amount of regularisation for 196 00:11:40,909 --> 00:11:44,149 the lasso regression is controlled by the parameter alpha, 197 00:11:44,149 --> 00:11:47,919 which by default is zero. 198 00:11:47,919 --> 00:11:49,482 Also like ridge regression, 199 00:11:49,482 --> 00:11:54,879 the purpose of using lasso regression is to estimate the WNB model coefficients. 200 00:11:54,879 --> 00:12:00,754 Once that's done, the prediction model formula is the same as for ordinary least-squares, 201 00:12:00,754 --> 00:12:03,034 you just use the linear model. 202 00:12:03,034 --> 00:12:07,210 In general, lasso regression is most helpful if you think there are 203 00:12:07,210 --> 00:12:11,950 only a few variables that have a medium or large effect on the output variable. 204 00:12:11,950 --> 00:12:17,830 Otherwise if there are lots of variables that contribute small or medium effects, 205 00:12:17,830 --> 00:12:21,529 ridge regression is typically the better choice. 206 00:12:21,529 --> 00:12:26,542 Let's take a look at lasso regression in scikit-learn using the notebook, 207 00:12:26,542 --> 00:12:29,330 using our communities in crime regression data set. 208 00:12:29,330 --> 00:12:31,776 To use lasso regression, 209 00:12:31,776 --> 00:12:37,365 you import the lasso class from sklearn.linear model, 210 00:12:37,365 --> 00:12:42,439 and then just use it as you would use an estimator like ridge regression. 211 00:12:42,439 --> 00:12:46,580 With some data sets you may occasionally get a convergence warning, 212 00:12:46,580 --> 00:12:51,549 in which case you can set the max_iter attribute to a larger value. 213 00:12:51,549 --> 00:12:55,519 So typically at least 20,000, or possibly more. 214 00:12:55,519 --> 00:13:01,679 Increasing the max inter-parameter will increase the computation time accordingly. 215 00:13:01,679 --> 00:13:03,919 In this example, we're applying lasso to 216 00:13:03,919 --> 00:13:08,654 a minmax scale version of the crime data set as we did for ridge regression. 217 00:13:08,654 --> 00:13:11,524 You can see that for Alpha set to 2.0, 218 00:13:11,524 --> 00:13:17,705 only 20 features with non-zero weights remain because with lasso regularisation, 219 00:13:17,705 --> 00:13:22,659 most of the features are set to have weights of exactly zero. 220 00:13:22,659 --> 00:13:25,730 I've listed the features with non-zero weights in 221 00:13:25,730 --> 00:13:30,065 order of their descending magnitude from the output. 222 00:13:30,065 --> 00:13:33,740 Although we need to be careful in interpreting 223 00:13:33,740 --> 00:13:37,205 any results for data on a complex problem like crime, 224 00:13:37,205 --> 00:13:40,700 the lasso regression results do help us see some of 225 00:13:40,700 --> 00:13:42,590 the strongest relationships between 226 00:13:42,590 --> 00:13:46,725 the input variables and outcomes for this particular data set. 227 00:13:46,725 --> 00:13:49,399 For example, looking at the top five features with 228 00:13:49,399 --> 00:13:53,210 non-zero weight that are found by lasso regression, 229 00:13:53,210 --> 00:13:58,309 we can see that location factors like percentage of people in dense housing, 230 00:13:58,309 --> 00:14:03,379 which indicates urban areas and socio economic variables like 231 00:14:03,379 --> 00:14:09,529 the fraction of vacant houses in an area are positively correlated with crime. 232 00:14:09,529 --> 00:14:12,230 And other variables like the percentage of families with 233 00:14:12,230 --> 00:14:16,580 two parents is negatively correlated. 234 00:14:16,580 --> 00:14:18,929 Finally, we can see the effect of tuning 235 00:14:18,929 --> 00:14:22,770 the regularisation parameter alpha for lasso regression. 236 00:14:22,770 --> 00:14:24,899 Like we saw with ridge regression, 237 00:14:24,899 --> 00:14:27,210 there's an optimal range for alpha that gives 238 00:14:27,210 --> 00:14:31,445 the best test set performance that neither under or over fits. 239 00:14:31,445 --> 00:14:35,445 Of course this best alpha value will be different for different data sets, 240 00:14:35,445 --> 00:14:38,335 and depends on various other factors such as the feature 241 00:14:38,335 --> 00:14:42,580 preprocessing methods being used. 242 00:14:42,580 --> 00:14:44,899 Let's suppose for a moment that we had a set of 243 00:14:44,899 --> 00:14:49,615 two-dimensional data points with features X0 and X1. 244 00:14:49,615 --> 00:14:53,629 Then we could transform each data point by adding additional features that 245 00:14:53,629 --> 00:14:58,220 were the three unique multiplicative combinations of X0 and X1. 246 00:14:58,220 --> 00:15:00,860 So, X0 squared, X0, 247 00:15:00,860 --> 00:15:03,659 X1 and X1 squared. 248 00:15:03,659 --> 00:15:06,919 So we've transformed our original two-dimensional points into a set of 249 00:15:06,919 --> 00:15:12,825 five-dimensional points that rely only on the information in the two-dimensional points. 250 00:15:12,825 --> 00:15:16,340 Now we can write a new regression problem that tries to predict 251 00:15:16,340 --> 00:15:22,929 the same output variable y-hat but using these five features instead of two. 252 00:15:22,929 --> 00:15:27,200 The critical insight here is that this is still a linear regression problem. 253 00:15:27,200 --> 00:15:30,610 The features are just numbers within a weighted sum. 254 00:15:30,610 --> 00:15:34,279 So we can use the same least-squares techniques to estimate 255 00:15:34,279 --> 00:15:37,100 the five model coefficients for these five features that we 256 00:15:37,100 --> 00:15:41,299 used in these simpler two-dimensional case. 257 00:15:41,299 --> 00:15:44,079 Now, why would we want to do this kind of transformation? 258 00:15:44,079 --> 00:15:47,809 Well, this is called polynomial future transformation that 259 00:15:47,809 --> 00:15:52,089 we can use to transform a problem into a higher dimensional regression space. 260 00:15:52,089 --> 00:15:56,659 And in effect, adding these extra polynomial features allows us a 261 00:15:56,659 --> 00:16:01,700 much richer set of complex functions that we can use to fit to the data. 262 00:16:01,700 --> 00:16:05,029 So you can think of this intuitively as allowing 263 00:16:05,029 --> 00:16:09,779 polynomials to be fit to the training data instead of simply a straight line, 264 00:16:09,779 --> 00:16:15,875 but using the same least-squares criterion that minimizes mean squared error. 265 00:16:15,875 --> 00:16:18,799 We'll see later that this approach of adding 266 00:16:18,799 --> 00:16:24,119 new features like polynomial features is also very effective with classification. 267 00:16:24,119 --> 00:16:25,850 And we'll look at this kind of transformation 268 00:16:25,850 --> 00:16:28,960 again in kernelized support vector machines. 269 00:16:28,960 --> 00:16:33,184 When we add these new polynomial features, 270 00:16:33,184 --> 00:16:36,875 we're essentially adding to the model's ability to capture interactions 271 00:16:36,875 --> 00:16:41,350 between the different variables by adding them as features to the linear model. 272 00:16:41,350 --> 00:16:44,330 For example, it may be that housing prices vary as 273 00:16:44,330 --> 00:16:48,320 a quadratic function of both the lat size that a house sits on, 274 00:16:48,320 --> 00:16:52,580 and the amount of taxes paid on the property as a theoretical example. 275 00:16:52,580 --> 00:16:56,345 A simple linear model could not capture this nonlinear relationship, 276 00:16:56,345 --> 00:17:01,100 but by adding nonlinear features like polynomials to the linear regression model, 277 00:17:01,100 --> 00:17:06,410 we can capture this nonlinearity. 278 00:17:06,410 --> 00:17:08,259 Or generally, we can use other types of 279 00:17:08,259 --> 00:17:12,170 nonlinear feature transformations beyond just polynomials. 280 00:17:12,170 --> 00:17:15,130 This is beyond the scope of this course but technically these are 281 00:17:15,130 --> 00:17:18,265 called nonlinear basis functions for regression, 282 00:17:18,265 --> 00:17:20,454 and are widely used. 283 00:17:20,454 --> 00:17:24,444 Of course, one side effect of adding lots of new features 284 00:17:24,444 --> 00:17:28,755 especially when we're taking every possible combination of K variables, 285 00:17:28,755 --> 00:17:32,829 is that these more complex models have the potential for overfitting. 286 00:17:32,829 --> 00:17:35,349 So in practice, polynomial regression is 287 00:17:35,349 --> 00:17:42,099 often done with a regularized learning method like ridge regression. 288 00:17:42,099 --> 00:17:47,095 Here's an example of polynomial regression using scikit-learn. 289 00:17:47,095 --> 00:17:50,740 There's already a handy class called polynomial features in the 290 00:17:50,740 --> 00:17:56,815 sklearn.preprocessing module that will generate these polynomial features for us. 291 00:17:56,815 --> 00:18:01,339 This example shows three regressions on a more complex regression dataset, 292 00:18:01,339 --> 00:18:06,069 that happens to have some quadratic interactions between variables. 293 00:18:06,069 --> 00:18:07,674 The first regression here, 294 00:18:07,674 --> 00:18:12,690 just uses least-squares regression without the polynomial feature transformation. 295 00:18:12,690 --> 00:18:18,890 The second regression creates the polynomial features object with degrees set to two, 296 00:18:18,890 --> 00:18:21,619 and then calls the fit transform method of 297 00:18:21,619 --> 00:18:25,850 the polynomial features object on the original XF1 features, 298 00:18:25,850 --> 00:18:31,144 to produce the new polynomial transform features XF1 poly. 299 00:18:31,144 --> 00:18:35,755 The code then calls ordinary least-squares linear regression. 300 00:18:35,755 --> 00:18:40,894 You can see indications of overfitting on this expanded feature representation, 301 00:18:40,894 --> 00:18:43,519 as the models r-squared score on the training set is 302 00:18:43,519 --> 00:18:47,480 close to one but much lower on the test set. 303 00:18:47,480 --> 00:18:51,230 So the third regression shows the effect of adding 304 00:18:51,230 --> 00:18:55,965 regularisation via ridge regression on this expanded feature set. 305 00:18:55,965 --> 00:19:00,605 Now, the training and tests r-squared scores are basically the same, 306 00:19:00,605 --> 00:19:02,329 with the test set score of 307 00:19:02,329 --> 00:19:04,579 the regularized polynomial regression 308 00:19:04,579 --> 00:19:08,859 performing the best of all three regression methods.