1 00:00:07,953 --> 00:00:12,240 One of the simplest kinds of supervised models are linear models. 2 00:00:13,400 --> 00:00:18,260 A linear model expresses the target output value in terms of a sum of weighted input 3 00:00:18,260 --> 00:00:18,849 variables. 4 00:00:20,310 --> 00:00:24,450 For example, our goal may be to predict the market value of a house, 5 00:00:24,450 --> 00:00:27,180 its expected sales price in the next month, for example. 6 00:00:28,820 --> 00:00:33,480 Suppose we're given two input variables, how much tax the properties assessed each 7 00:00:33,480 --> 00:00:37,119 year by the local government, and the age of the house in years. 8 00:00:38,390 --> 00:00:42,320 You can imagine that these two features of the house would each have some information 9 00:00:42,320 --> 00:00:45,410 that's helpful in predicting the market price. 10 00:00:45,410 --> 00:00:49,190 Because in most places, there's a positive correlation between the tax assessment on 11 00:00:49,190 --> 00:00:51,150 a house and its market value. 12 00:00:52,520 --> 00:00:55,910 Indeed the tax assessment is often partly based on market prices from 13 00:00:55,910 --> 00:00:56,609 previous years. 14 00:00:58,190 --> 00:01:01,960 And may be a negative correlation between its age in years and the market value, so 15 00:01:01,960 --> 00:01:05,080 older houses may need more repairs and upgrading, for example. 16 00:01:07,170 --> 00:01:12,130 One linear model, which I have made up as an example, could compute the expected 17 00:01:12,130 --> 00:01:17,845 market price in US dollars by starting with a constant term, here 212,000. 18 00:01:17,845 --> 00:01:23,290 And then adding some number, let's say 109 times the value of tax paid last year, 19 00:01:24,300 --> 00:01:28,520 and then subtracting 2,000 times the age of the house in years. 20 00:01:28,520 --> 00:01:34,050 So for example this linear model would estimate the market price of a house where 21 00:01:34,050 --> 00:01:39,570 the taxes estimate was $10,000 and that was 75 years old as 22 00:01:39,570 --> 00:01:44,210 about $1.2 million. 23 00:01:44,210 --> 00:01:48,720 Now, I just made up this particular linear model myself as an example but 24 00:01:48,720 --> 00:01:52,170 in general when we talk about training a linear model. 25 00:01:52,170 --> 00:01:55,848 We mean estimating values for the parameters of the model, or 26 00:01:55,848 --> 00:01:59,031 coefficients of the model as we sometimes call them, 27 00:01:59,031 --> 00:02:04,008 which are here the constant value 212,000 and the weights 109 and 20. 28 00:02:04,008 --> 00:02:09,079 In such a way that the resulting predictions for the outcome variable 29 00:02:09,079 --> 00:02:14,950 Yprice, for different houses are a good fit to the data from actual past sales. 30 00:02:14,950 --> 00:02:18,720 We'll discuss what good fit means shortly. 31 00:02:19,770 --> 00:02:24,940 Predicting house price is an example of a regression task using a linear model 32 00:02:24,940 --> 00:02:27,220 called, not surprisingly, linear regression. 33 00:02:28,480 --> 00:02:30,860 More generally, in a linear regression model, 34 00:02:30,860 --> 00:02:36,280 there may be multiple input variables, or features, which we'll denote x0, x1, etc. 35 00:02:37,860 --> 00:02:41,960 Each feature, xi, has a corresponding weight, wi. 36 00:02:43,090 --> 00:02:46,470 The predicted output, which we denote y hat, 37 00:02:47,790 --> 00:02:51,940 is a weighted sum of features plus a constant term b hat. 38 00:02:53,580 --> 00:02:57,090 I've put a hat over all the quantities here that are estimated 39 00:02:57,090 --> 00:02:59,710 during the regression training process. 40 00:02:59,710 --> 00:03:03,826 The w hat and b hat values which we call the train parameters or 41 00:03:03,826 --> 00:03:07,230 coefficients are estimated from training data. 42 00:03:07,230 --> 00:03:11,787 And y hat is estimated from the linear function of input feature values and 43 00:03:11,787 --> 00:03:13,840 the train parameters. 44 00:03:13,840 --> 00:03:20,797 For example, in the simple housing price example we just saw, 45 00:03:20,797 --> 00:03:26,049 w0 hat was 109, x0 represented tax paid, 46 00:03:26,049 --> 00:03:31,040 w1 hat was negative 20 x1 was house age and 47 00:03:31,040 --> 00:03:34,082 b hat was 212,000. 48 00:03:34,082 --> 00:03:39,306 We called these wi values model coefficients or sometimes future weights, 49 00:03:39,306 --> 00:03:43,560 and b hat is called the bias term or the intercept of the model. 50 00:03:46,340 --> 00:03:50,220 Here's an example of a linear regression model with just one input variable or 51 00:03:50,220 --> 00:03:54,920 feature x0 on a simple artificial example dataset. 52 00:03:56,280 --> 00:04:02,651 The blue cloud of points represents a training set of x0, y pairs. 53 00:04:03,770 --> 00:04:09,952 In this case, the formula for predicting the output y hat is just w0 hat times 54 00:04:09,952 --> 00:04:16,370 x0 + b hat, which you might recognize as the familiar slope intercept formula for 55 00:04:16,370 --> 00:04:21,930 a straight line, where w0 hat is the slope, and b hat is the y intercept. 56 00:04:23,690 --> 00:04:27,750 The grand red lines represent different possible linear regression models that 57 00:04:27,750 --> 00:04:31,500 could attempt to explain the relationship between x0 and y. 58 00:04:32,850 --> 00:04:35,950 You can see that some lines are a better fit than others. 59 00:04:37,130 --> 00:04:41,620 The better fitting models capture the approximately linear relationship where 60 00:04:41,620 --> 00:04:45,560 as x0 increases, y also increases in a linear fashion. 61 00:04:46,710 --> 00:04:49,130 The red line seemed specially good. 62 00:04:49,130 --> 00:04:53,540 Intuitively, there are not as many blue training points that are very far above or 63 00:04:53,540 --> 00:04:56,050 very far below the red linear model prediction. 64 00:04:57,430 --> 00:05:02,400 Let's take a look at a very simple form of linear regression model that just has 65 00:05:02,400 --> 00:05:07,370 one input variable, or feature to use for prediction. 66 00:05:08,780 --> 00:05:12,640 In this case, we have the vector x just has a single component, 67 00:05:12,640 --> 00:05:16,260 we'll call it x0, that's the input variable, input feature. 68 00:05:17,710 --> 00:05:22,149 And, in this case because there's just one variable, the predicted output is 69 00:05:22,149 --> 00:05:28,200 simply the product of the weight 70 00:05:28,200 --> 00:05:33,120 w0 with the input variable x0 plus a biased term b. 71 00:05:34,970 --> 00:05:39,575 So x0 is the value that's provided, it comes with the data and so 72 00:05:39,575 --> 00:05:43,007 the parameters we have to estimate are w0 and b, 73 00:05:43,007 --> 00:05:47,890 in order to obtain the parameters for this linear regression model. 74 00:05:49,230 --> 00:05:51,490 So this formula may look familiar, 75 00:05:51,490 --> 00:05:56,390 it's the formula for a line in terms of its slope. 76 00:05:56,390 --> 00:05:59,821 In this case, slope corresponds to the weight, w0, and 77 00:05:59,821 --> 00:06:03,194 b corresponds to the y intercept, we call the bias term. 78 00:06:06,682 --> 00:06:11,750 So here, the job of the model is to take as input. 79 00:06:11,750 --> 00:06:16,760 Let's pick a point here, on the x-axis so 80 00:06:16,760 --> 00:06:23,220 w0 corresponds to the slope of this line and 81 00:06:24,600 --> 00:06:29,360 b corresponds to the y intercept of the line. 82 00:06:29,360 --> 00:06:32,084 And so finding these two parameters, 83 00:06:32,084 --> 00:06:37,628 these two parameters together define a straight line in this feature space. 84 00:06:41,840 --> 00:06:45,942 Now the important thing to remember is that there's a training phase and 85 00:06:45,942 --> 00:06:47,200 a prediction phase. 86 00:06:47,200 --> 00:06:49,710 So the training phase, using the training data, 87 00:06:50,740 --> 00:06:54,430 is what we'll use to estimate w0 and b. 88 00:06:56,020 --> 00:07:01,153 So widely used method for estimating w and b for linear aggression problems is called 89 00:07:01,153 --> 00:07:06,018 least-squares linear regression, also known as ordinary least-squares. 90 00:07:06,018 --> 00:07:10,758 Least-squares linear regression finds the line through this cloud of 91 00:07:10,758 --> 00:07:15,741 points that minimizes what is called the means squared error of the model. 92 00:07:15,741 --> 00:07:20,157 The mean squared error of the model is essentially the sum of the squared 93 00:07:20,157 --> 00:07:23,397 differences between the predicted target value and 94 00:07:23,397 --> 00:07:27,310 the actual target value for all the points in the training set. 95 00:07:29,240 --> 00:07:32,010 This plot illustrates what that means. 96 00:07:33,210 --> 00:07:39,350 The blue points represent points in the training set, the red line here represents 97 00:07:39,350 --> 00:07:45,470 the least-squares models that was found through these cloud of training points. 98 00:07:45,470 --> 00:07:50,432 And these black lines show the difference between the y 99 00:07:50,432 --> 00:07:56,230 value that was predicted for training point based on it's x position, 100 00:07:56,230 --> 00:07:58,370 and the actual y value of the training point. 101 00:07:59,850 --> 00:08:06,320 So for example here, this point let's say has 102 00:08:06,320 --> 00:08:11,310 an x value of- 1.75. 103 00:08:11,310 --> 00:08:17,373 And if we plug it into the formula for this linear model, 104 00:08:17,373 --> 00:08:23,049 we get a prediction here, at this point on the line, 105 00:08:23,049 --> 00:08:27,824 which is somewhere around let's say 60. 106 00:08:34,047 --> 00:08:38,915 But the actual observed value in the training set for 107 00:08:38,915 --> 00:08:42,350 this point was maybe closer to 10. 108 00:08:42,350 --> 00:08:46,720 So, in this case, for this particular point, the squared difference between 109 00:08:46,720 --> 00:08:55,380 the predicted target and the actual target would be (60- 10) squared. 110 00:08:55,380 --> 00:08:59,379 So, we can do this calculation for every one of the points in the training set. 111 00:09:00,520 --> 00:09:04,960 We can compute this squared difference between 112 00:09:04,960 --> 00:09:09,710 the y value we observe in the training set for a point, and the y value that would be 113 00:09:09,710 --> 00:09:13,790 predicted by the linear model, given that training points x value. 114 00:09:16,690 --> 00:09:20,828 So each of these can be computed as the square difference can be computed, and 115 00:09:20,828 --> 00:09:26,636 then if we add all these up, And divide by the number of training points, 116 00:09:26,636 --> 00:09:30,680 take the average, that will be the mean squared error of the model. 117 00:09:33,490 --> 00:09:37,690 So the technique of least-squares, 118 00:09:37,690 --> 00:09:42,938 is designed to find the slope, the w value, and 119 00:09:42,938 --> 00:09:47,925 the b value of the y intercept, that minimize 120 00:09:47,925 --> 00:09:53,190 this squared error, this mean squared error. 121 00:09:54,630 --> 00:09:59,200 One thing to note about this linear regression model is 122 00:09:59,200 --> 00:10:01,680 that there are no parameters to control the model complexity. 123 00:10:03,520 --> 00:10:05,050 No matter what the value of w and b, 124 00:10:05,050 --> 00:10:08,270 the result is always going to be a straight line. 125 00:10:10,440 --> 00:10:13,900 This is both a strength and a weakness of the model as we'll see later. 126 00:10:15,900 --> 00:10:20,360 When you have a moment, compare this simple linear model to the more complex 127 00:10:20,360 --> 00:10:26,250 regression model learned with K nearest neighbors regression on the same dataset. 128 00:10:26,250 --> 00:10:29,590 You can see that linear models make a strong prior assumption 129 00:10:29,590 --> 00:10:32,710 about the relationship between the input x and output y. 130 00:10:34,290 --> 00:10:39,240 Linear models may seem simplistic, but for data with many features linear models can 131 00:10:39,240 --> 00:10:43,260 be very effective and generalize well to new data beyond the training set. 132 00:10:45,080 --> 00:10:49,340 Now the question is, how exactly do we estimate the near models w and 133 00:10:49,340 --> 00:10:52,290 b parameters so the model is a good fit? 134 00:10:53,640 --> 00:10:57,235 Well, the w and b parameters are estimated using the training data. 135 00:10:57,235 --> 00:11:01,056 And there are lots of different methods for estimating w and 136 00:11:01,056 --> 00:11:06,025 b depending on the criteria you'd like to use for the definition of what a good 137 00:11:06,025 --> 00:11:10,700 fit to the training data is and how you want to control model complexity. 138 00:11:12,240 --> 00:11:13,430 For linear models, 139 00:11:13,430 --> 00:11:17,940 model complexity is based on the nature of the weights w on the input features. 140 00:11:17,940 --> 00:11:21,625 Simpler linear models have a weight vector w that's closer to zero, 141 00:11:21,625 --> 00:11:26,330 i.e., where more features are either not used at all that have zero weight or 142 00:11:26,330 --> 00:11:28,980 have less influence on the outcome, a very small weight. 143 00:11:30,660 --> 00:11:34,390 Typically, given possible settings for the model parameters, 144 00:11:34,390 --> 00:11:38,480 the learning algorithm predicts the target value for each training example, and 145 00:11:38,480 --> 00:11:42,760 then computes what is called a loss function for each training example. 146 00:11:42,760 --> 00:11:46,000 That's a penalty value for incorrect predictions. 147 00:11:46,000 --> 00:11:50,110 The prediction's incorrect when the predicted target value is different 148 00:11:50,110 --> 00:11:52,100 than the actual target value in the training set. 149 00:11:53,160 --> 00:11:57,730 For example, a squared loss function would return the squared difference 150 00:11:57,730 --> 00:12:00,770 between the target value and the actual value as the penalty. 151 00:12:01,870 --> 00:12:08,010 The learning algorithm then computes or searches for the set of w, b parameters 152 00:12:08,010 --> 00:12:12,079 that minimize the total of this loss function over all training points. 153 00:12:13,540 --> 00:12:18,105 The most popular way to estimate w and b parameters is using what's called 154 00:12:18,105 --> 00:12:22,020 least-squares linear regression or ordinary least-squares. 155 00:12:22,020 --> 00:12:25,490 Least-squares finds the values of w and 156 00:12:25,490 --> 00:12:29,390 b that minimize the total sum of squared differences 157 00:12:29,390 --> 00:12:34,145 between the predicted y value and the actual y value in the training set. 158 00:12:34,145 --> 00:12:37,615 Or equivalently it minimizes the mean squared error of the model. 159 00:12:37,615 --> 00:12:42,590 Least-squares is based on the squared loss function mentioned before. 160 00:12:44,060 --> 00:12:46,010 This is illustrated graphically here, 161 00:12:46,010 --> 00:12:50,094 where I've zoomed in on the left lower portion of this simple regression dataset. 162 00:12:51,460 --> 00:12:54,910 The red line represents the least-squares solution for w and 163 00:12:54,910 --> 00:12:57,290 b through the training data. 164 00:12:57,290 --> 00:13:02,230 And the vertical lines represent the difference between the actual y 165 00:13:02,230 --> 00:13:04,869 value of a training point, xi, y and 166 00:13:04,869 --> 00:13:10,340 it's predicted y value given xi which lies on the red line where x equals xi. 167 00:13:12,580 --> 00:13:16,290 Adding up all the squared values of these differences for all the training points 168 00:13:16,290 --> 00:13:21,420 gives the total squared error and this is what the least-square solution minimizes. 169 00:13:22,640 --> 00:13:26,100 Here, there are no parameters to control model complexity. 170 00:13:26,100 --> 00:13:29,486 The linear model always uses all of the input variables and 171 00:13:29,486 --> 00:13:31,983 always is represented by a straight line. 172 00:13:35,093 --> 00:13:39,770 Another name for this quantity is the residual sum of squares. 173 00:13:40,940 --> 00:13:45,510 The actual target value is given in yi and the predicted y hat value for 174 00:13:45,510 --> 00:13:49,560 the same training example is given by the right side of the formula 175 00:13:49,560 --> 00:13:52,000 using the linear model with that parameters w and b. 176 00:13:53,690 --> 00:13:57,540 Let's look at how to implement this in Scikit-Learn. 177 00:13:57,540 --> 00:14:01,820 Linear regression in Scikit-Learn is implemented by the linear regression class 178 00:14:01,820 --> 00:14:04,580 in the sklearn.linear_model module. 179 00:14:06,120 --> 00:14:08,861 As we did with other estimators in Scikit-Learn, 180 00:14:08,861 --> 00:14:12,623 like the nearest neighbors classifier, and the regression models, 181 00:14:12,623 --> 00:14:15,955 we use the train test split function on the original data set. 182 00:14:15,955 --> 00:14:21,355 And then create and fit the linear regression object using the training 183 00:14:21,355 --> 00:14:27,600 data in X_train and the corresponding training data target values in Y_train. 184 00:14:27,600 --> 00:14:29,740 Here, note that we're doing the creation and 185 00:14:29,740 --> 00:14:32,930 fitting of the linear regression object in one line 186 00:14:32,930 --> 00:14:35,810 by chaining the fit method with the constructor for the new object. 187 00:14:37,770 --> 00:14:42,072 The linear regression fit method acts to estimate the future weights w, which 188 00:14:42,072 --> 00:14:48,419 are called the coefficients of the model and it stores this in the coeff_attribute. 189 00:14:48,419 --> 00:14:52,530 And the bias term, b, which is stored in the intercept_ attribute. 190 00:14:54,910 --> 00:14:59,550 Note that if a Scikit-Learn object attribute ends with an underscore, 191 00:14:59,550 --> 00:15:04,240 this means that these attributes were derived from training data, and not, say, 192 00:15:04,240 --> 00:15:06,180 quantities that were set by the user. 193 00:15:08,500 --> 00:15:13,750 If we dump the coef_ and intercept_ attributes for this simple example, 194 00:15:13,750 --> 00:15:18,167 we see that because there's only one input feature variable, 195 00:15:18,167 --> 00:15:23,027 there's only one element in the coeff_list, the value 45.7. 196 00:15:23,027 --> 00:15:27,796 The intercept attribute has a value of about 148.4. 197 00:15:27,796 --> 00:15:32,897 And we can see that indeed these correspond to the red line shown in 198 00:15:32,897 --> 00:15:38,794 the plot which has a slope of 45.7 and y intercept of about 148.4. 199 00:15:38,794 --> 00:15:41,760 Here is the same code in the notebook. 200 00:15:41,760 --> 00:15:46,124 With additional code to score the quality of the regression model, in the same way 201 00:15:46,124 --> 00:15:50,250 that we did for K nearest neighbors regression using the R-squared metric. 202 00:15:53,262 --> 00:15:58,369 And here is the notebook code we use to plot the least-squares linear solution for 203 00:15:58,369 --> 00:15:59,340 this dataset. 204 00:16:03,430 --> 00:16:06,235 Now that we have seen both K nearest neighbors regression and 205 00:16:06,235 --> 00:16:11,290 least-squares regression, it's interesting now to compare the least-squared 206 00:16:11,290 --> 00:16:13,981 linear regression results with the K nearest neighbors result. 207 00:16:15,400 --> 00:16:18,090 Here we can see how these two regression methods represent 208 00:16:18,090 --> 00:16:20,240 two complementary types of supervised learning. 209 00:16:21,638 --> 00:16:24,520 The K nearest neighbor regresser doesn't make a lot of assumptions about 210 00:16:24,520 --> 00:16:28,590 the structure of the data, and gives potentially accurate but sometimes 211 00:16:28,590 --> 00:16:32,360 unstable predictions that are sensitive to small changes in the training data. 212 00:16:33,760 --> 00:16:36,829 So it has a correspondingly higher training set, R-squared score, 213 00:16:36,829 --> 00:16:39,021 compared to least-squares linear regression. 214 00:16:39,021 --> 00:16:42,719 K-NN achieves an R-squared score of 0.72 and 215 00:16:42,719 --> 00:16:48,230 least-squares achieves an R-squared of 0.679 on the training set. 216 00:16:49,870 --> 00:16:53,880 On the other hand, linear models make strong assumptions about the structure of 217 00:16:53,880 --> 00:16:56,630 the data, in other words, that the target value 218 00:16:56,630 --> 00:17:00,340 can be predicted using a weighted sum of the input variables. 219 00:17:00,340 --> 00:17:03,900 And linear models give stable but potentially inaccurate predictions. 220 00:17:03,900 --> 00:17:07,990 However, in this case, it turns out that the linear model strong assumption 221 00:17:07,990 --> 00:17:10,330 that there's a linear relationship between the input and 222 00:17:10,330 --> 00:17:13,340 output variables happens to be a good fit for this dataset. 223 00:17:14,780 --> 00:17:18,120 And so it's better at more accurately predicting the y value for 224 00:17:18,120 --> 00:17:21,420 new x values that weren't seen during training. 225 00:17:21,420 --> 00:17:26,231 And we can see that the linear model gets a slightly better test set 226 00:17:26,231 --> 00:17:31,065 score of 0.492 versus 0.471 for K nearest neighbors. 227 00:17:31,065 --> 00:17:34,550 And this indicates its ability to better generalize and 228 00:17:34,550 --> 00:17:36,120 capture this global linear trend.