1
00:00:00,940 --> 00:00:05,370
Today, I'm going to finish up my lectures
on linear algebra.

2
00:00:05,370 --> 00:00:10,650
I'll start off by going over, or
introducing something called an orthogonal

3
00:00:10,650 --> 00:00:13,050
matrix, and then, I'm going to talk

4
00:00:13,050 --> 00:00:17,810
about three different factorizations, so
you can.

5
00:00:17,810 --> 00:00:22,950
Factor a, a matrix into something called a
single singular value factorization.

6
00:00:22,950 --> 00:00:25,798
You can also write it as product of

7
00:00:25,798 --> 00:00:28,800
eigenvectors and eigenvalues.

8
00:00:28,800 --> 00:00:31,094
And then there's a, a third factorization

9
00:00:31,094 --> 00:00:33,822
called QR factorization and I'm going to
use that

10
00:00:33,822 --> 00:00:38,400
to show you how you can or show you how
you can solve these squares problems.

11
00:00:38,400 --> 00:00:40,300
Using the q r factorization..

12
00:00:40,300 --> 00:00:45,363
And all of those factorization's basically
involve splitting a matrix

13
00:00:45,363 --> 00:00:50,840
up into an orthogonal matrix or, or one or
more orthogonal matrices.

14
00:00:50,840 --> 00:00:54,900
And then one other matrix that's designed
to have a special property.

15
00:00:57,310 --> 00:01:03,250
So I'm going to go back to the Talking
just about vectors for a second.

16
00:01:03,250 --> 00:01:06,730
So, suppose I have 2 vectors q and w.

17
00:01:08,420 --> 00:01:11,920
They're orthogonal, so they're
perpendicular to one another.

18
00:01:11,920 --> 00:01:15,773
If the inner product of q and w is equal
to 0.

19
00:01:21,950 --> 00:01:26,920
So now let's suppose I have a set, so here
a set of M vectors.

20
00:01:26,920 --> 00:01:29,630
So here, Q1 is not an element of Q.

21
00:01:29,630 --> 00:01:34,460
Q1 is actually a vector.
So I have Q1 up to QM.

22
00:01:34,460 --> 00:01:36,090
And remember these are vectors.

23
00:01:36,090 --> 00:01:38,330
I am still referring to the same vectors
from the

24
00:01:38,330 --> 00:01:40,660
first line so they are in an m dimensional
space.

25
00:01:40,660 --> 00:01:47,508
So, these are vectors of m elements, and I
have m of them, and this notation

26
00:01:47,508 --> 00:01:53,030
here means they live in m dimensional
space minus zero.

27
00:01:53,030 --> 00:01:55,360
So, this is the set minus operation.

28
00:01:55,360 --> 00:01:59,264
So I'm just trying to say there that the
zero vector is not allowed to be in the

29
00:01:59,264 --> 00:02:02,497
set because that sort of trivially
orthogonal to everything,

30
00:02:02,497 --> 00:02:04,595
because anything times 0 is going to be 0.

31
00:02:04,595 --> 00:02:05,175
[COUGH]

32
00:02:05,175 --> 00:02:11,610
So I have a set of m vectors, where the
zero vector is not allowed.

33
00:02:14,320 --> 00:02:17,030
And then I want to assume that every pair
of these vectors.

34
00:02:17,030 --> 00:02:19,010
So any, you know, for any i and any j

35
00:02:19,010 --> 00:02:22,370
Where i and j are different numbers
between one and m.

36
00:02:22,370 --> 00:02:27,220
If I take the inner product of those two
vectors I'm going to get 0.

37
00:02:27,220 --> 00:02:29,990
So that means every pair of vectors is
orthogonal.

38
00:02:31,110 --> 00:02:33,039
So every pair of vectors is perpendicular.

39
00:02:36,380 --> 00:02:42,422
And then I'm going to define a vector so
for each q j I'm going to divide q j

40
00:02:42,422 --> 00:02:48,399
by the length of q j and I'm going to call
that oops call that q tilda j.

41
00:02:50,720 --> 00:02:54,990
And so if I take a vector and I divide by
its length, I end up with something

42
00:02:54,990 --> 00:02:57,580
called a unit vector which points in the
same

43
00:02:57,580 --> 00:03:01,320
direction as the original vector, but it
has length one.

44
00:03:01,320 --> 00:03:01,890
So it doesn't matter.

45
00:03:01,890 --> 00:03:04,389
If, if QJ, if it was a really short
vector, then dividing

46
00:03:04,389 --> 00:03:06,837
by its length is going to make it longer
so that it's equal

47
00:03:06,837 --> 00:03:09,285
to 1, and if it's a really long vector,
dividing by its

48
00:03:09,285 --> 00:03:12,980
length is going to make it shorter, so the
length is equal to 1.

49
00:03:12,980 --> 00:03:15,800
So for any non zero vector this is going
to give me a vector

50
00:03:15,800 --> 00:03:18,810
that points in the same direction but that
has length one.

51
00:03:22,200 --> 00:03:24,930
And I'm going to call this new set of
vectors so

52
00:03:24,930 --> 00:03:27,720
q, I assume this little a was meant to be
a one.

53
00:03:27,720 --> 00:03:29,900
I don't know how I would have made that
typo.

54
00:03:31,460 --> 00:03:37,680
So q tilde one up to q tilde m are called
orthonormal fecta vectors.

55
00:03:37,680 --> 00:03:46,270
So orthogonal means that any pair of them
has a right angle in between them.

56
00:03:46,270 --> 00:03:47,275
And then when they

57
00:03:47,275 --> 00:03:48,570
also have length one.

58
00:03:49,730 --> 00:03:52,490
They get this extra little word, normal,
here.

59
00:03:52,490 --> 00:03:55,598
So orthonormal means a set of vectors who
are

60
00:03:55,598 --> 00:04:00,370
mutually perpendicular, and each vector
has to have length one.

61
00:04:00,370 --> 00:04:08,220
And now I want to make a matrix Q.

62
00:04:08,220 --> 00:04:12,520
That has these orthonormal vectors as its
columns.

63
00:04:12,520 --> 00:04:15,221
And think about what happens when I
consider

64
00:04:15,221 --> 00:04:17,990
product q, q transpose, or q transpose q.

65
00:04:20,600 --> 00:04:23,070
So the easier one is going to be q
transpose q.

66
00:04:23,070 --> 00:04:26,960
And what I'm going to have, is, in the
first row.

67
00:04:26,960 --> 00:04:32,340
So when I take the transpose of a matrix,
turns the columns into the rows.

68
00:04:32,340 --> 00:04:36,260
So q transpose has these orthonormal
vectors as its rows,

69
00:04:36,260 --> 00:04:40,660
and the original q has these orthonormal
vectors as its columns.

70
00:04:44,140 --> 00:04:49,650
Then when I make this product q transpose
q, what's going to happen?

71
00:04:49,650 --> 00:04:55,977
The first element, the, the 1,1 entry of
the product is going to

72
00:04:55,977 --> 00:05:02,230
end up being q 1 transpose, q tilde 1
transpose times q tilde 1.

73
00:05:02,230 --> 00:05:05,330
So that's just going to be the squared
length of this vector.

74
00:05:05,330 --> 00:05:09,290
But since the length is one, that's going
to be one.

75
00:05:09,290 --> 00:05:12,042
And then, for any other product in the

76
00:05:12,042 --> 00:05:15,568
first row, I'd have q tilde 1 transposed
times

77
00:05:15,568 --> 00:05:18,750
q tilde some other index, and because
these

78
00:05:18,750 --> 00:05:22,470
are orthogonal, that's going to be equal
to 0.

79
00:05:24,060 --> 00:05:28,563
So, what ends up happening, when I, I make
these products, in the

80
00:05:28,563 --> 00:05:33,700
off diagonal entries, I have qi Q tilde i
transpose q tilde j.

81
00:05:33,700 --> 00:05:34,570
And because

82
00:05:34,570 --> 00:05:39,442
those are orthogonal, that's how I, I made
this set of vectors,

83
00:05:39,442 --> 00:05:44,575
that's going to be 0, Except on the
diagonal where these vectors hit

84
00:05:44,575 --> 00:05:50,640
themselves.
So I have q tilde 1 transpose q tilde 1.

85
00:05:50,640 --> 00:05:53,188
And because it's an ortho normal set of
vectors, that's

86
00:05:53,188 --> 00:05:55,990
going to have, that's going to be equal to
1.

87
00:05:55,990 --> 00:05:59,240
So, what I end up with is the identity
matrix.

88
00:06:00,350 --> 00:06:00,654
So,

89
00:06:00,654 --> 00:06:03,010
I've found this matrix if I take this

90
00:06:03,010 --> 00:06:06,126
ortho normal set of vectors, make this
product q

91
00:06:06,126 --> 00:06:08,786
transposed q, then I end up with these dot

92
00:06:08,786 --> 00:06:12,840
products down the diagonal which are just
the length.

93
00:06:12,840 --> 00:06:15,900
Of the vectors, and then the mixed stock
products are

94
00:06:15,900 --> 00:06:18,580
going to be 0, so I, I get the identity
matrix.

95
00:06:18,580 --> 00:06:23,139
So in orthogonal an orthonormal set of
vectors when

96
00:06:23,139 --> 00:06:25,758
I put it into a matrix Q, has this

97
00:06:25,758 --> 00:06:30,070
property that q transpose q is the
identity matrix.

98
00:06:31,400 --> 00:06:34,040
And that holds for qq transpose as well.

99
00:06:35,250 --> 00:06:40,260
And so, what I'm trying to do here is, I,
I've now found a family of matrices.

100
00:06:40,260 --> 00:06:45,330
So, if this matrix has orthonormal
columns, then, q transpose is equal to q

101
00:06:45,330 --> 00:06:50,634
inverse, because I, I have a matrix where,
if I multiply either q transpose q,

102
00:06:50,634 --> 00:06:54,219
or q q transpose, I end up with the
identity matrix.

103
00:06:55,530 --> 00:06:57,810
So the reason that these orthogonal
matrices

104
00:06:57,810 --> 00:06:59,430
are going to be nice is because I can

105
00:06:59,430 --> 00:07:01,890
compute inverses just by taking the
transpose,

106
00:07:01,890 --> 00:07:04,130
so just by swapping the rows and columns.

107
00:07:08,780 --> 00:07:12,524
And so, this matrix has a, a special name,
so a square matrix

108
00:07:12,524 --> 00:07:16,219
q, so it has to be square, same number of
rows and columns.

109
00:07:17,310 --> 00:07:20,616
Is orthogonal if q transpose q is the
identity

110
00:07:20,616 --> 00:07:25,460
matrix and q q transpose is also the
identity matrix.

111
00:07:25,460 --> 00:07:28,570
And so it's a little bit strange the way
these got named.

112
00:07:28,570 --> 00:07:30,926
So vectors can be an orthogonal set as

113
00:07:30,926 --> 00:07:34,270
long as all of their, you know, their
pairwise

114
00:07:34,270 --> 00:07:37,901
dot products for two different vectors R
equals to

115
00:07:37,901 --> 00:07:41,150
zero, but it doesn't say anything about
the length.

116
00:07:41,150 --> 00:07:45,167
Orthonormal factors have their
parallelized

117
00:07:45,167 --> 00:07:48,950
perpendicular and have length one.

118
00:07:48,950 --> 00:07:51,342
But for some reason when we put those into
a matrix, the

119
00:07:51,342 --> 00:07:55,240
matrix we get isn't called the orthonormal
matrix it's called a orthogonal matrix.

120
00:07:55,240 --> 00:07:57,370
But it also has this condition.

121
00:07:57,370 --> 00:07:59,840
That it's not a diagonal matrix that I'm
getting.

122
00:07:59,840 --> 00:08:06,250
I'm getting actually the identity matrix.
So the columns of q have to have length 1.

123
00:08:06,250 --> 00:08:08,802
And it turns out that if you, if you

124
00:08:08,802 --> 00:08:12,674
actually think about what these are doing
in space,

125
00:08:12,674 --> 00:08:15,754
so it's easiest just to, to deal with the

126
00:08:15,754 --> 00:08:21,150
2 by 2 case Orthogonal matrices represent
rotations and reflections.

127
00:08:21,150 --> 00:08:25,023
So if I, if I have a matrix, you know.
I, I think I, I've shown you guys

128
00:08:25,023 --> 00:08:26,445
a rotation matrix already.

129
00:08:26,445 --> 00:08:26,910
[COUGH]

130
00:08:26,910 --> 00:08:29,142
So it turns out that if I, if I

131
00:08:29,142 --> 00:08:34,510
rotate the vector, that counts as an
orthogonal transformation.

132
00:08:34,510 --> 00:08:36,320
That's something that I can write.

133
00:08:36,320 --> 00:08:39,200
As an orthogonal matrix times a vector.

134
00:08:39,200 --> 00:08:42,400
And I can rotate that vector through a
certain angle.

135
00:08:42,400 --> 00:08:44,720
Also if I have a line, I can reflect

136
00:08:44,720 --> 00:08:48,400
a vector across that line using an
orthogonal matrix.

137
00:08:51,740 --> 00:08:57,260
So for example, let's Define Q to just be
this matrix.

138
00:08:57,260 --> 00:09:00,930
So cosine, negative sine, sine, cosine.

139
00:09:00,930 --> 00:09:04,850
And this rotates a vector in the xy plane
through an angle theta.

140
00:09:10,280 --> 00:09:13,335
And so you can see that this is going to
be an ortho, or you

141
00:09:13,335 --> 00:09:15,675
can check that this is an orthogonal
matrix

142
00:09:15,675 --> 00:09:18,070
just by making this product q transpose q.

143
00:09:19,220 --> 00:09:23,002
So q transpose, I'm just going to take the
first row of this, of q and put

144
00:09:23,002 --> 00:09:25,172
that in the first column of q transpose
and

145
00:09:25,172 --> 00:09:28,450
then the second row goes in the second
column.

146
00:09:28,450 --> 00:09:31,843
And then when I do the matrix
multiplication on

147
00:09:31,843 --> 00:09:35,671
the diagonal elements I'm going to get
cosign times

148
00:09:35,671 --> 00:09:40,690
cosign, so cosign squared Plus sine times
sine, so sine squared.

149
00:09:42,690 --> 00:09:46,794
So, it's a bit complicated, so on the
diagonal elements, I end up

150
00:09:46,794 --> 00:09:51,762
with cosine squared plus sine squared, and
on the off diagonal elements, it ends

151
00:09:51,762 --> 00:09:56,730
up being minus consine sine plus sine
cosine, so that cancels, each term cancels

152
00:09:56,730 --> 00:10:00,690
the other one out for every value of
theta, so this is always going

153
00:10:00,690 --> 00:10:02,010
to be equal to 0.

154
00:10:03,370 --> 00:10:07,850
And then I get a symmetric thing on the
other off-diagonal element.

155
00:10:07,850 --> 00:10:11,290
So cosine squared plus sine squared we all
know that's equal to 1.

156
00:10:11,290 --> 00:10:13,380
And this is 0 and this is 0.

157
00:10:13,380 --> 00:10:16,250
So this is going to give me the identity
matrix.

158
00:10:16,250 --> 00:10:19,980
So I had enough space at the bottom
right-hand corner to fit one more I in.

159
00:10:25,040 --> 00:10:32,450
And we also want to check that, so a q
transpose q equals qq transpose equals I.

160
00:10:32,450 --> 00:10:37,349
This was my condition for an orthogonal
matrix, but that implies that q inverse is

161
00:10:37,349 --> 00:10:39,834
equal to q transpose because that's also

162
00:10:39,834 --> 00:10:42,489
the definition for the inverse of a
matrix.

163
00:10:43,500 --> 00:10:48,350
So we need to check the q transpose is
actually the inverse of q.

164
00:10:51,350 --> 00:10:56,838
And so this is going to use a result, on
something called even and

165
00:10:56,838 --> 00:11:02,910
odd functions, so an even function is, if
I put in an minus Theta.

166
00:11:02,910 --> 00:11:04,310
I get the same value back.

167
00:11:04,310 --> 00:11:06,604
So in this case, cosine, if you think
about

168
00:11:06,604 --> 00:11:08,712
what that looks like, if I go a little bit

169
00:11:08,712 --> 00:11:13,360
positive, so I go to positive x or I go
the other way a little bit to negative x.

170
00:11:13,360 --> 00:11:16,522
Because that's a symmetric function around
0, I'm going to

171
00:11:16,522 --> 00:11:19,002
get the same value as long as I'm x away
from 0,

172
00:11:19,002 --> 00:11:23,360
and it doesn't matter which way I go,
positive direction or negative direction.

173
00:11:24,540 --> 00:11:29,640
So that means I could write cosine of
theta as cosine of minus theta.

174
00:11:29,640 --> 00:11:32,010
Both of those numbers are going to have
the same value.

175
00:11:33,470 --> 00:11:35,870
And sine is something that's called an odd
function.

176
00:11:39,640 --> 00:11:44,810
So, oops, did I mess this up?

177
00:11:48,756 --> 00:11:52,000
so sine, if you think it's going to do
exactly the opposite.

178
00:11:52,000 --> 00:11:55,230
So when I put in sine of, if I have sine
of theta.

179
00:11:56,584 --> 00:12:00,680
when I put in minus theta, I should get
minus sine of theta.

180
00:12:00,680 --> 00:12:04,110
Because it's, it's something that sort of
flipped over

181
00:12:04,110 --> 00:12:07,470
the origin rather than just reflected
across the y axis,

182
00:12:07,470 --> 00:12:09,290
so let me just check what I did So

183
00:12:09,290 --> 00:12:12,000
q should have the minus sign in the top
right.

184
00:12:13,830 --> 00:12:16,950
And so, when I put in the minus theta

185
00:12:16,950 --> 00:12:22,180
here that should've taken away this minus
sign here.

186
00:12:24,340 --> 00:12:25,680
So let's see if I got it right.

187
00:12:29,210 --> 00:12:35,230
Here okay these typos are killing me but
essentially what I'm, what I'm trying to

188
00:12:35,230 --> 00:12:41,085
show here is that q transpose is the same
thing as q just with a minus theta put in.

189
00:12:41,085 --> 00:12:41,590
[COUGH]

190
00:12:41,590 --> 00:12:46,943
And so what that's going to do is Q
rotates something through an

191
00:12:46,943 --> 00:12:52,770
angle theta, Q transpose rotates it
through an angle minus theta.

192
00:12:52,770 --> 00:12:55,286
So if I rotate through theta and though
minus

193
00:12:55,286 --> 00:12:57,258
theta I end up back at the same point

194
00:12:57,258 --> 00:12:59,774
and staying at the same point is exactly
what

195
00:12:59,774 --> 00:13:03,949
would've happened had I multiplied
something by the identity matrix.

196
00:13:05,700 --> 00:13:06,770
Okay.

197
00:13:06,770 --> 00:13:11,411
The other nice property that an Orthogonal
Matrix has is that

198
00:13:11,411 --> 00:13:15,751
it preserves dot products.
So if I have a dot product, x

199
00:13:15,751 --> 00:13:20,310
and y, and I multiply both x and y by an
orthogonal matrix, q.

200
00:13:21,810 --> 00:13:27,424
So I have qx .qy.
Or if I write that using

201
00:13:27,424 --> 00:13:32,128
matrix and vector notation rather than as
a .product.y

202
00:13:32,128 --> 00:13:38,720
I can write that as QX.product.quantity
transposed times the quantity QY.

203
00:13:39,740 --> 00:13:43,870
And then, I have a rule for how I can deal
with transposers of products of matrices.

204
00:13:43,870 --> 00:13:47,396
I take the transpose, transpose of each
element,

205
00:13:47,396 --> 00:13:50,102
and I put them in the opposite order, so

206
00:13:50,102 --> 00:13:54,530
the quantity Qx transposed becomes x
transposed Q transposed,

207
00:13:54,530 --> 00:13:57,350
and then I have, Qy just stays the same.

208
00:13:57,350 --> 00:14:00,990
And now what I end up with in the middle
here

209
00:14:00,990 --> 00:14:06,200
is this q transpose q which is equal to
the identity matrix.

210
00:14:07,580 --> 00:14:11,813
So I can just imaginary imagine,
multiplying either a Y

211
00:14:11,813 --> 00:14:16,560
by the identity matrix or X transpose
times the identity matrix.

212
00:14:16,560 --> 00:14:20,400
Either way it's going to give me Y or x
transpose back.

213
00:14:20,400 --> 00:14:22,670
And I end up with x transpose y.

214
00:14:22,670 --> 00:14:25,970
Which is exactly the definition of the dot
product of x and y.

215
00:14:27,550 --> 00:14:32,242
And the reason this is going to be really
useful is because

216
00:14:32,242 --> 00:14:37,490
the length of a vector would just be the
square root.

217
00:14:37,490 --> 00:14:42,976
Of, say, x dot x.
So

218
00:14:42,976 --> 00:14:48,061
that means that if I multiply x by an
orthogonal matrix

219
00:14:48,061 --> 00:14:55,180
q, so q is, q has to have, for, for this
multiplication to work.

220
00:14:55,180 --> 00:15:02,060
X is a vector of day M elements, Q has to
be an M by N, M by M matrix.

221
00:15:02,060 --> 00:15:06,140
So the product Qx is also going to be a
vector of M elements and

222
00:15:06,140 --> 00:15:12,310
the length of that vector is equal to the
length of the original vector X.

223
00:15:12,310 --> 00:15:13,296
And this makes sense

224
00:15:13,296 --> 00:15:16,319
again if you just think about it in terms
of a rotation matrix.

225
00:15:17,560 --> 00:15:20,760
Because if I rotate a vector, the length
is staying the same.

226
00:15:20,760 --> 00:15:24,595
If I take this and I just rotate it up
here, it's the distance from this

227
00:15:24,595 --> 00:15:28,810
point down to the origin is going to be
the same as for the vector I started with.