1
00:00:00,000 --> 00:00:04,055
So we talked about that the concept of
dependence, right?
2
00:00:04,055 --> 00:00:10,974
So X and Y are dependent upon one another
and if I tell you about X that gives you
3
00:00:10,974 --> 00:00:16,019
some information, additional information
about what Y is.
4
00:00:16,019 --> 00:00:23,352
Now, there are many, many, many different
kinds of dependencies that can exist in
5
00:00:23,352 --> 00:00:24,099
data.
Right?
6
00:00:24,099 --> 00:00:30,698
And the concept of covariance and
correlation measures only one special kind
7
00:00:30,698 --> 00:00:34,678
of dependency, and that's called linear
dependency.
8
00:00:34,678 --> 00:00:40,262
All right so covariance and correlation
measure the extent to which X and Y move
9
00:00:40,262 --> 00:00:43,376
together in a straight line fashion.
Okay.
10
00:00:43,376 --> 00:00:49,287
If there is a positive relationship then
when you plot when X versus Y you see an
11
00:00:49,287 --> 00:00:54,771
upward sloping linear relationship.
If there's negative covariance correlation
12
00:00:54,771 --> 00:00:59,771
and if you plot X versus Y then there's a
downward sloping linear relationship
13
00:00:59,771 --> 00:01:03,075
between them.
It is very important to understand that
14
00:01:03,075 --> 00:01:08,007
covariance and correlation only measures
linear association.
15
00:01:10,212 --> 00:01:13,026
You know, but linear association is just
one kind.
16
00:01:13,026 --> 00:01:17,094
You could have a non-linear association,
that is Y didn't have to be related in a
17
00:01:17,094 --> 00:01:21,654
quadratic form, or you know, a logarithmic
form, or an exponential form, or, or
18
00:01:21,654 --> 00:01:26,020
whatever.
Turns out that it's very difficult to
19
00:01:26,020 --> 00:01:30,004
define measures of non-linear dependence,
okay?
20
00:01:30,004 --> 00:01:35,088
There's just, there, you know, not because
there are so many ways of defining
21
00:01:35,088 --> 00:01:41,026
non-linear dependence there isn't a
general way to do it, and so when, we look
22
00:01:41,026 --> 00:01:46,361
at dependents with random variables, we
focus on linear dependents only.
23
00:01:46,361 --> 00:01:51,732
Now, this may seem like it's, it's highly
restrictive, but you know, one of the
24
00:01:51,732 --> 00:01:56,759
facts of the world is, a lot of
relationships are approximately linear
25
00:01:56,759 --> 00:02:00,889
over, you know, generally, you know,
viewed ranges of values.
26
00:02:00,889 --> 00:02:06,548
So even something that's nonlinear is
approximately linear over, you know, say,
27
00:02:06,548 --> 00:02:09,314
small intervals, or something like that.
So.
28
00:02:09,314 --> 00:02:14,102
Linear dependence could be a good
approximation for even non-linear
29
00:02:14,102 --> 00:02:19,556
dependence in, in many situations and when
we start looking at data and we, when we
30
00:02:19,556 --> 00:02:24,374
plot, you know, different values of X and
Y together, we'll often see that the
31
00:02:24,374 --> 00:02:29,661
dependence that comes from the data looks
pretty linear and so we don't actually
32
00:02:29,661 --> 00:02:33,108
have to go to something that's more
complicated.
33
00:02:33,372 --> 00:02:39,419
Alright, so what is covariance?
So, covariance is a measure of direction
34
00:02:39,419 --> 00:02:44,662
but not strength of the linear
relationship in the data, okay?
35
00:02:44,662 --> 00:02:50,904
So it's, so covariance, so we use the
Greek symbol sigma with the subscript x, y
36
00:02:50,904 --> 00:02:54,261
to represent the covariance between x and
y.
37
00:02:54,261 --> 00:02:59,771
The definition of covariance is the
expectation of x minus its mean multiplied
38
00:02:59,771 --> 00:03:03,964
by y minus its mean.
And so if we have a discreet random
39
00:03:03,964 --> 00:03:08,904
variable, we would take all values of X
and Y in the joint sample space and then
40
00:03:08,904 --> 00:03:13,708
we would take the product of X minus its
mean, Y minus its mean, weight by the
41
00:03:13,708 --> 00:03:19,972
probabilities, and add it all up. If we
have a continuous distribution, we would
42
00:03:19,972 --> 00:03:25,053
take from the summation into an integral.
We would take X minus the mean, Y minus
43
00:03:25,053 --> 00:03:30,074
the mean, weighted by the probability
curve and, and then compute that value.
44
00:03:30,074 --> 00:03:34,049
Alright?
Now, when you look at the formula it's not
45
00:03:34,049 --> 00:03:40,078
necessarily apparent that this is a
measure of linear association, right?
46
00:03:40,078 --> 00:03:47,024
And so what I want to do now is just show
you a graph and, and show graphically why
47
00:03:47,024 --> 00:03:53,024
this computation gives you a, a measure of
the direction of linear dependance.
48
00:03:55,058 --> 00:04:00,024
So.
Let's see.
49
00:04:02,002 --> 00:04:05,075
Here we go.
So you're going to do a graphical
50
00:04:05,075 --> 00:04:24,079
description of covariance.
So I'm going to do a graph.
51
00:04:24,079 --> 00:04:30,316
I'm going to take Y minus the mean of Y,
I'm going to plot on this axis, and I'm
52
00:04:30,316 --> 00:04:34,051
going to plot X minus the mean on this
axis.
53
00:04:34,051 --> 00:04:40,694
And what I'm going to do is, I'm going to
plot what I call a probability scatter
54
00:04:40,694 --> 00:04:44,654
plot.
So I'm going to look at you know, say like
55
00:04:44,654 --> 00:04:50,648
a discrete distribution between X and Y,
where every single point in that
56
00:04:50,648 --> 00:04:53,508
distribution is equally likely.
Okay?
57
00:04:53,508 --> 00:05:05,307
And so I'll give a, a distribution that's
going to look like this.
58
00:05:05,307 --> 00:05:09,607
Okay.
So looking at the probability scatter
59
00:05:09,607 --> 00:05:14,894
plot, you know, I mean, what does your
intuition tell you about the direction of
60
00:05:14,894 --> 00:05:18,143
linear dependence?
Is it positive or negative?
61
00:05:18,143 --> 00:05:21,027
It's positive.
And why is it positive?
62
00:05:22,098 --> 00:05:28,073
Yeah, so it's upward sloping.
So when we look at values of X above its
63
00:05:28,073 --> 00:05:33,049
mean, so when X minus mu is positive we're
up over here.
64
00:05:33,049 --> 00:05:37,032
Right?
So in this quadrant we have values of X
65
00:05:37,032 --> 00:05:44,096
above its mean and then notice that when X
is above its mean, in, this direction, Y
66
00:05:44,096 --> 00:05:48,525
also is above its mean.
So as X increases, Y increases.
67
00:05:48,525 --> 00:05:55,355
And then if we look down here, also as
well when X is below its mean then Y tends
68
00:05:55,355 --> 00:06:02,682
to be below its mean as well so that's
just showing you that X and Y move
69
00:06:02,682 --> 00:06:08,861
together in a positive way.
Now the so how does this give, rise to a
70
00:06:08,861 --> 00:06:17,453
positive covariance and remember
covariance is the expectation of X minus
71
00:06:17,453 --> 00:06:26,006
the mean of X times Y minus the mean of Y?
And, in this case, this would be a
72
00:06:26,006 --> 00:06:38,952
probability weighted average when we
multiplied by the probabilities between X
73
00:06:38,952 --> 00:06:43,043
and Y.
So let's just break up this probability
74
00:06:43,043 --> 00:06:50,281
scatter plot into four quadrants, so we'll
have quadrant one, quadrant two, quadrant
75
00:06:50,281 --> 00:06:57,005
three, and quadrant four.
Now in quadrant one, X is, tends to be
76
00:06:57,005 --> 00:07:01,091
below its mean, but Y tends to be above
its mean.
77
00:07:01,091 --> 00:07:05,598
Right?
So, in this quadrant we have X minus mu of
78
00:07:05,598 --> 00:07:11,074
X is less than zero, but Y minus mu of Y
is greater than zero.
79
00:07:11,074 --> 00:07:17,947
So we take the product of X minus mu of X,
and Y minus mu of Y, we get a negative
80
00:07:17,947 --> 00:07:24,051
number times a positive number.
And so we get a negative number.
81
00:07:25,037 --> 00:07:31,015
So if we see points in this quadrant, and
we want to know what is contribution to
82
00:07:31,015 --> 00:07:35,719
covariance of points in this quadrant,
it's equal to X minus the mean times Y
83
00:07:35,719 --> 00:07:40,453
minus the mean, weighted by the
probability, probabilities are always
84
00:07:40,453 --> 00:07:44,256
positive.
So the contributions to covariance in this
85
00:07:44,256 --> 00:07:50,100
quadrant are all negative values, okay?
Now notice in this graph, there are no
86
00:07:50,100 --> 00:07:52,687
contributions to covariance in this
quadrant.
87
00:07:52,687 --> 00:07:57,724
Now let's look at quadrant number two.
In quadrant number two, X is above it's
88
00:07:57,724 --> 00:08:00,916
mean.
And Y is above it's mean.
89
00:08:00,916 --> 00:08:07,720
So you get, X minus the mean of X is
positive, Y minus the mean of Y is
90
00:08:07,720 --> 00:08:13,503
positive, so when you take the product of
X minus its mean and Y minus its mean.
91
00:08:13,503 --> 00:08:17,912
You get a positive times a positive, so
that's a positive number.
92
00:08:17,912 --> 00:08:23,316
So these three blue dots here are equal to
X minus the mean, Y minus the mean.
93
00:08:23,316 --> 00:08:28,452
They're positive numbers, they're weighted
by a probability that's positive.
94
00:08:28,452 --> 00:08:34,613
So all these three dots here give a
positive contribution to covariance.Okay?
95
00:08:34,613 --> 00:08:39,314
And then you could do the last for the
quadrant number three.
96
00:08:39,314 --> 00:08:44,706
So X minus the mean is less than zero.
Y minus the mean is less than zero.
97
00:08:44,706 --> 00:08:50,524
So when you take the product of X minus
the mean, and Y minus the mean, you get a
98
00:08:50,524 --> 00:08:54,060
negative times a negative number, which is
positive.
99
00:08:55,046 --> 00:09:01,031
So in this quadrant down here, these three
blue dots give a positive contribution to
100
00:09:01,031 --> 00:09:04,058
covariance.
Because if you have multiplied two
101
00:09:04,058 --> 00:09:09,122
negative numbers.
Then finally in quadrant four, we have X
102
00:09:09,122 --> 00:09:16,907
minus the mean of X is a positive, but Y
minus the mean of Y is a negative.
103
00:09:16,907 --> 00:09:21,741
So we take the product, we get a negative
number.
104
00:09:21,741 --> 00:09:28,072
So that's a positive times a negative, so
that's less than zero.
105
00:09:28,072 --> 00:09:33,563
So, any dots in this range, give a
negative contribution to covariance.
106
00:09:33,563 --> 00:09:38,016
So what is covariance?
It's the expected value of the product, so
107
00:09:38,016 --> 00:09:43,390
its the sum of the product weighted by the
probabilities, so I take these three
108
00:09:43,390 --> 00:09:48,091
points, they're all positive, multiply by
positive numbers, its all positive.
109
00:09:48,091 --> 00:09:52,610
There's no contribution here, no
contribution here, positive contribution
110
00:09:52,610 --> 00:09:57,208
here, so we add everything up, we get a
positive co-variance.
111
00:09:57,208 --> 00:10:03,369
So this is a case where sigma X Y is
greater than zero.
112
00:10:03,369 --> 00:10:06,149
Okay?
Now covariance only gives you the
113
00:10:06,149 --> 00:10:08,591
direction of linear dependence.
Right?
114
00:10:08,591 --> 00:10:14,221
So if a covariance is positive we know X
and Y move together in a positive
115
00:10:14,221 --> 00:10:17,775
relationship.
Covariance doesn't tell you the strength
116
00:10:17,775 --> 00:10:19,496
of the relationship.
Okay.
117
00:10:19,496 --> 00:10:23,078
And that's where correlation comes into
play.
118
00:10:23,078 --> 00:10:27,029
Correlation tells you the direction and
strength.
119
00:10:31,058 --> 00:10:35,088
So correlation is defined to be the
covariance divided by the product of the
120
00:10:35,088 --> 00:10:39,038
standard deviations.
And it's just the scale value of the
121
00:10:39,038 --> 00:10:42,028
covariance.
So we now know what covariance is.
122
00:10:42,028 --> 00:10:47,012
Covariance measures the direction of
linear association between two random
123
00:10:47,012 --> 00:10:49,032
variables.
So, how do we compute it?
124
00:10:49,032 --> 00:10:54,032
So if we have a discrete distribution we
know that the covariance between X and Y,
125
00:10:54,032 --> 00:10:59,039
you just take the X values minus the mean,
Y values minus the mean, weight it by the
126
00:10:59,039 --> 00:11:03,072
probabilities, add them all up.
So you got a discrete distribution, things
127
00:11:03,072 --> 00:11:08,036
are, it's a brute-force calculation.
So with discreet distributions I said,
128
00:11:08,073 --> 00:11:11,060
computation is brute-force and, and, and
not fun.
129
00:11:12,003 --> 00:11:17,098
So here I have a, an example of computing
covariance.
130
00:11:17,098 --> 00:11:23,011
So, if you have a discreet distribution,
there's no real easy way to end up doing
131
00:11:23,011 --> 00:11:26,051
the computation.
So in a spreadsheet, you can take the
132
00:11:26,051 --> 00:11:30,010
values of X and the values of Y.
And the joint probabilities.
133
00:11:30,010 --> 00:11:34,054
And then you have to compute X minus the
mean and Y minus the mean, which you can
134
00:11:34,054 --> 00:11:37,045
do in a table.
And then you can take the product of X
135
00:11:37,045 --> 00:11:41,062
minus the mean and Y minus the mean, and
then weight that by the probability.
136
00:11:41,062 --> 00:11:44,020
And then add them all up at the end of the
day.
137
00:11:44,020 --> 00:11:46,094
So this is just doing the brute-force
calculation.
138
00:11:46,094 --> 00:11:50,095
The sum of X minus the mean, Y minus the
mean, weighted by the probability.
139
00:11:50,095 --> 00:11:55,012
So there, there's no easy way to do the
calculation, it, it's just brute-force.
140
00:11:55,062 --> 00:12:08,016
Now in the, for my discrete distribution
here, my example distribution one way of
141
00:12:08,016 --> 00:12:12,959
viewing relationships between the random
variables X and Y, is to do the
142
00:12:12,959 --> 00:12:16,551
probability scatter plot.
So this probability scatter plot, I have
143
00:12:16,551 --> 00:12:21,691
the values of Y here, and the values of X
here, and these bubbles here represent the
144
00:12:21,691 --> 00:12:25,650
probabilities associated with a joint
occurrence between X and Y.
145
00:12:25,650 --> 00:12:29,940
And the size of the bubble is related to
the magnitude of the probability.
146
00:12:29,940 --> 00:12:35,048
So we see that there's a bigger joint
probability for this point and this point
147
00:12:35,048 --> 00:12:39,112
than for these other points.
And you can sort of see that there's a
148
00:12:39,112 --> 00:12:42,553
upward sloping relationship in this
probability scatter plot.
149
00:12:42,553 --> 00:12:46,839
And that's indicating that it looks like
there's a positive relationship between X
150
00:12:46,839 --> 00:12:49,701
and Y.
So if there's any justice in the world, if
151
00:12:49,701 --> 00:12:54,499
we actually compute the covariance, we
should get a positive number.
152
00:12:54,499 --> 00:13:01,040
And so, this is what happens.
We compute the covariance between X and Y
153
00:13:01,040 --> 00:13:03,077
and it turns out to be 0.25.
Okay?
154
00:13:03,077 --> 00:13:10,013
So the covariance between X and Y is a
positive number and that means that as X
155
00:13:10,013 --> 00:13:15,087
goes up Y tends to go up as well.
And so that's just this number here which
156
00:13:15,087 --> 00:13:21,046
means that it computes the covariance and
do the group work calculation.
157
00:13:21,046 --> 00:13:24,087
It's a quarter 0.25.
Alright.
158
00:13:28,044 --> 00:13:33,356
Now, there are some important properties
of covariance that are very useful and
159
00:13:33,356 --> 00:13:37,002
we're going to be using them a lot in our
modeling.
160
00:13:37,002 --> 00:13:42,004
So that the first property is covariance
is a symmetric relationship, so if you
161
00:13:42,004 --> 00:13:47,019
look at the covariance between X and Y,
that's the same as the covariance between
162
00:13:47,019 --> 00:13:48,008
Y and X.
Right?
163
00:13:48,008 --> 00:13:53,062
So, covariance is a symmetric relationship
and you can get that just from the, the
164
00:13:53,062 --> 00:13:57,038
definition.
Now, another result if we take X we
165
00:13:57,038 --> 00:14:03,020
multiply it by some scalar a.
We take Y and multiply it by some scalar b
166
00:14:03,020 --> 00:14:09,552
so we have say 2X and 5Y.< /i>< /i>
The covariance of this scaled value of X
167
00:14:09,552 --> 00:14:16,357
and Y is equal to the scalars ab< /i>
times the covariance between X and Y.
168
00:14:16,357 --> 00:14:23,949
Okay, now this result comes from the, just
the definition of covariance so if you
169
00:14:23,949 --> 00:14:32,565
look at the covariance between aX and bY,
that's the expectation of aX minus a times
170
00:14:32,565 --> 00:14:40,017
the mean of X times bY - b times the mean
of Y, right?
171
00:14:40,086 --> 00:14:47,052
And so that's just the definition of
covariance between aX and bY.
172
00:14:47,052 --> 00:14:55,007
And so then this is equal to the
expectation of aX minus its mean and bY
173
00:14:55,007 --> 00:15:00,036
minus its mean.
And a and b are just numbers, so they come
174
00:15:00,036 --> 00:15:11,017
out of the expectation.
So then this is just A times B times the
175
00:15:11,017 --> 00:15:15,065
covariance between X and Y.
So, if you take a linear function, so if
176
00:15:15,065 --> 00:15:21,034
you just multiply, if you change the scale
of X and Y, then the covariance changes by
177
00:15:21,034 --> 00:15:24,094
these values.
Now this is very important because that
178
00:15:24,094 --> 00:15:30,056
partially tells you that I can make the
value of the covariance between X and Y
179
00:15:30,056 --> 00:15:35,037
anything that I want just by multiplying X
and Y by numbers a and b.
180
00:15:35,079 --> 00:15:41,083
Or, in other words, this is, this
relationship tells us that the magnitude
181
00:15:41,083 --> 00:15:47,087
of the covariance depends upon the units
of measurement between X and Y.
182
00:15:47,087 --> 00:15:50,476
Right?
So say X is return and Y is return right?
183
00:15:50,476 --> 00:15:55,472
So we get equal variance between X and Y,
what is the units of covariance?
184
00:15:55,472 --> 00:16:01,012
Well it's the product of returns, it's
return squared because I'm taking X minus
185
00:16:01,012 --> 00:16:04,358
its mean and multiplying it by Y minus its
mean.
186
00:16:04,358 --> 00:16:08,676
Now if I multiply returns by a 100, so I
have returns in percentage.
187
00:16:08,676 --> 00:16:13,124
So I, if a is a hundred and b is a 100,
then the covariance between X and Y
188
00:16:13,124 --> 00:16:16,699
becomes 100100 times the< /i>
covariance between X and Y.
189
00:16:16,699 --> 00:16:21,932
So I, I'll inflate the magnitude of the
covariance a lot, but I don't change the
190
00:16:21,932 --> 00:16:25,893
relationship between X and Y when I change
the scale, right?
191
00:16:25,893 --> 00:16:30,660
If I multiple X and Y by ten, both by ten,
that doesn't change the relationship that
192
00:16:30,660 --> 00:16:33,688
just changes the scale that we measure
things on.
193
00:16:33,688 --> 00:16:38,293
So this relationship tell us that, very
often we, we don't care about the
194
00:16:38,293 --> 00:16:43,906
magnitude of covariance by itself, we just
care weather or not it's a positive or
195
00:16:43,906 --> 00:16:49,550
negative number.
The coverage between X and itself, will
196
00:16:49,550 --> 00:16:54,432
have its X pull varied with itself but
that is just the variants of X, okay?
197
00:16:54,432 --> 00:16:59,335
If X and Y are independent there is no
relationship between X and Y at all.
198
00:16:59,335 --> 00:17:03,534
So, there is no linear relationship so the
covariance is equal to zero.
199
00:17:03,534 --> 00:17:08,235
Now the converse is not true.
If the covariance between X and Y is zero,
200
00:17:08,235 --> 00:17:12,845
we just know that there is no linear
relationship between X and Y.
201
00:17:12,845 --> 00:17:16,430
But X and Y can have a non-linear
relationship, alright?
202
00:17:16,430 --> 00:17:20,501
So knowledge that the covariance is zero
does not mean that X and Y are
203
00:17:20,501 --> 00:17:25,474
independent.
Now, the final result here is just a
204
00:17:25,474 --> 00:17:30,932
convenient way of computing covariance.
So it turns out that the covariance
205
00:17:30,932 --> 00:17:36,655
between X and Y could be also, could be
computed as the expectation of X times Y
206
00:17:36,655 --> 00:17:40,197
minus the expectation of X times the
expectation of Y.
207
00:17:40,197 --> 00:17:47,908
And, and this result just follows from you
know, brute force calculations.
208
00:17:47,908 --> 00:17:57,142
So if you want to calculate the covariance
between X and Y, the expectation of X
209
00:17:57,142 --> 00:18:04,053
minus its mean, Y minus its mean and then
you just multiply everything out.
210
00:18:04,053 --> 00:18:11,684
We get XY - X times the mean of Y - Y
times the mean of X, plus the mean of X
211
00:18:11,684 --> 00:18:19,455
times the mean of Y.
So this becomes expectation of XY minus mu
212
00:18:19,455 --> 00:18:28,826
of Y times the expectation of X minus the
expectation of Y times the mean of X + the
213
00:18:28,826 --> 00:18:35,945
mean of X times the mean of Y.
So this is the expectation of XY - UY
214
00:18:35,945 --> 00:18:44,213
times mu X Minus mu Y times mu X, plus mu
X times mu Y.
215
00:18:44,213 --> 00:18:50,210
So this is the expectation of XY minus the
mean of X times the mean of Y.
216
00:18:50,210 --> 00:18:56,594
So, by the definition of covariance, and
the linearity properties of expectation,
217
00:18:56,594 --> 00:19:03,683
we can get that covariance can also be
computed as the expectation of the product
218
00:19:03,683 --> 00:19:08,049
of X times Y, minus the mean of X times
the mean of Y.
219
00:19:08,049 --> 00:19:14,172
Now, this is often a more convenient way
to do the computation, you know, in