1
00:00:00,740 --> 00:00:04,077
Now that we have constructed Hilbert
spaces and ordinal

2
00:00:04,077 --> 00:00:07,523
bases we can see some of their
distinguishing features.

3
00:00:07,523 --> 00:00:09,971
One of them is norm conservation which is

4
00:00:09,971 --> 00:00:13,355
called Parseval's theorem and is an
extension of Peter

5
00:00:13,355 --> 00:00:14,003
[INAUDIBLE]

6
00:00:14,003 --> 00:00:18,090
famous ordinality theorem to infinite
dimensions.

7
00:00:18,090 --> 00:00:20,630
Then we will see the orthogonal projection
theorem.

8
00:00:20,630 --> 00:00:23,110
This is a powerful method to take a vector
from

9
00:00:23,110 --> 00:00:28,180
possibly an infinite dimensional space,
and project it onto a subspace.

10
00:00:28,180 --> 00:00:29,700
Once we have this, we will consider

11
00:00:29,700 --> 00:00:33,990
some examples of approximations and
othonormal basis.

12
00:00:33,990 --> 00:00:38,560
Module 3.3 Hilbert Space and
approximation.

13
00:00:41,020 --> 00:00:42,820
Your review of this sub module as
following:

14
00:00:42,820 --> 00:00:45,980
first I'm going to talk about the norm
conservation, when

15
00:00:45,980 --> 00:00:49,654
we have expansion into orthonormal basis,
and a

16
00:00:49,654 --> 00:00:55,470
so-called equivalence formula, which is
known as Parseval's formula.

17
00:00:55,470 --> 00:00:59,940
Then we talk about approximation by
projection, and we give an example.

18
00:01:02,280 --> 00:01:04,800
Parseval's theorem has a very simple form.

19
00:01:04,800 --> 00:01:10,300
It says that if you expand x in an
orthonormal basis, so we have a vector

20
00:01:10,300 --> 00:01:16,190
in a Hilbert space, we expand it in an
orthonormal basis with this vector w.

21
00:01:16,190 --> 00:01:20,360
Here, we are on the finite dimensional
case of dimension capital K.

22
00:01:20,360 --> 00:01:25,060
Then, for orthonormal basis, the square
norm of x

23
00:01:25,060 --> 00:01:29,660
is equal to the sum of squared
coefficients, okay?

24
00:01:29,660 --> 00:01:34,580
This of course a generalization of
Pythagorean theorem, which says that if I

25
00:01:34,580 --> 00:01:39,810
have the vector here, this is x, this

26
00:01:39,810 --> 00:01:45,174
is, so length in an orthonormal basis or,
that would be alpha 0

27
00:01:45,174 --> 00:01:50,362
in e 0, alpha with respect to

28
00:01:50,362 --> 00:01:55,065
e1, then x square is equal to alpha

29
00:01:55,065 --> 00:01:59,010
0 square plus alpha 1 square.
Okay?

30
00:01:59,010 --> 00:02:05,630
Very old result, very useful result.
Let's actually verify it algebraically.

31
00:02:06,640 --> 00:02:10,530
We start with the canonical basis, e0 and
e1, just as before.

32
00:02:11,650 --> 00:02:19,960
We write x as a linear combination of e0
and e1, and we go to a new basis.

33
00:02:19,960 --> 00:02:20,210
The new

34
00:02:20,210 --> 00:02:23,930
basis is v0 and v1.
It' s an autonomal basis.

35
00:02:23,930 --> 00:02:24,160
Okay.

36
00:02:24,160 --> 00:02:28,020
You can verify that these guys are at the
right angle.

37
00:02:28,020 --> 00:02:31,110
And this basis is given by v0 is cosine

38
00:02:31,110 --> 00:02:35,590
theta sine theta v1 is minus sine theta
cosine theta.

39
00:02:35,590 --> 00:02:38,330
You can see this here on the projections.

40
00:02:38,330 --> 00:02:44,090
And so x in the new basis is equal to beta
0 v 0.

41
00:02:44,090 --> 00:02:45,110
Vector 1,

42
00:02:45,110 --> 00:02:45,240
V1.

43
00:02:45,240 --> 00:02:49,330
What are the expressions of this, well we
know this because the

44
00:02:49,330 --> 00:02:54,842
basis is orthonormal so the expansion
coefficients beta 0 is equal to

45
00:02:54,842 --> 00:02:55,350
[UNKNOWN]

46
00:02:55,350 --> 00:03:01,400
product between V0 and X beta 1 the inner
product between V1 and X.

47
00:03:01,400 --> 00:03:03,320
Or in compact form.

48
00:03:03,320 --> 00:03:09,210
We can just write these inner products as
row, column, scaler products.

49
00:03:09,210 --> 00:03:15,648
OK, so the the zero is cosine theta.
Sin theta d one is

50
00:03:15,648 --> 00:03:20,630
minus sin theta cos theta as we have seen.
And so we write this as R

51
00:03:20,630 --> 00:03:24,820
times alpha where R is the rotation
matrix.

52
00:03:24,820 --> 00:03:25,830
Okay?

53
00:03:25,830 --> 00:03:28,800
A unitary matrix that corresponds to these
basis vectors.

54
00:03:29,910 --> 00:03:35,960
Alright, and it's not hard to verify that
the rotation matrix times this transpose

55
00:03:35,960 --> 00:03:41,220
is equal to identity, meaning it is equal
through transposition to its inverse.

56
00:03:41,220 --> 00:03:45,340
So it's a unitary matrix, as we probably
well know by now.

57
00:03:47,270 --> 00:03:51,070
Okay.
So let's look look at norm conservation.

58
00:03:51,070 --> 00:03:55,860
So the square norm in the canonical basis
is just as we announced,

59
00:03:55,860 --> 00:04:00,430
so x squared is equal to alpha 0 squared
plus alpha 1 squared.

60
00:04:00,430 --> 00:04:04,310
The square norm in the rotated basis is
the same, but with respect to beta.

61
00:04:04,310 --> 00:04:04,610
Okay.

62
00:04:04,610 --> 00:04:08,110
So we are going to show that these two
things are the same.

63
00:04:09,600 --> 00:04:11,280
Or, verify Parseval's formula.

64
00:04:12,810 --> 00:04:15,870
Well, one way to write beta 0 squared plus
beta 1 squared

65
00:04:15,870 --> 00:04:19,990
is to say it's a vector beta transpose
times beta, right, okay.

66
00:04:19,990 --> 00:04:24,210
I'm just, so I'm making myself absolutely
clear It's

67
00:04:24,210 --> 00:04:29,123
the scalar product of the vector beta,
okay, with itself.

68
00:04:29,123 --> 00:04:29,750
Okay.

69
00:04:29,750 --> 00:04:31,000
That's the thing here.

70
00:04:31,000 --> 00:04:34,340
Now, of course, beta is equal to R times
alpha.

71
00:04:34,340 --> 00:04:37,870
Beta transpose is R times alpha transpose.

72
00:04:37,870 --> 00:04:42,370
You go to the extra step of reordering
here the products, so

73
00:04:42,370 --> 00:04:47,670
the alpha transpose comes in front, R
transpose here times R times alpha.

74
00:04:47,670 --> 00:04:53,310
This of course simplified to identity that
we know because R is a unitary matrix.

75
00:04:53,310 --> 00:04:55,420
So this is equal to alpha transposed as
alpha.

76
00:04:56,420 --> 00:05:00,890
And of course we verify what we set out to
do, okay.

77
00:05:00,890 --> 00:05:02,650
Now we did this in two dimensions,

78
00:05:02,650 --> 00:05:07,480
it's obvious that this will hold in n
dimensions for an arbitrary n.

79
00:05:07,480 --> 00:05:13,060
It turns out it also tell it also holds
for infinite dimensional canonical basis.

80
00:05:13,060 --> 00:05:19,600
Okay, so that's Parseval's formula, very
important formula in signal processing.

81
00:05:20,990 --> 00:05:27,490
Okay, what it really means is that if you
have a vector X,

82
00:05:27,490 --> 00:05:32,920
and you look at this vector in, you know,
this basis,

83
00:05:34,850 --> 00:05:40,875
and you look at in some other basis, as we
just did, which is a rotation, because all

84
00:05:40,875 --> 00:05:41,680
[UNKNOWN]

85
00:05:41,680 --> 00:05:45,060
basis are essentially rotations of each
other and maybe

86
00:05:45,060 --> 00:05:49,795
some symmetry, then the length of the
vector doesn't change.

87
00:05:49,795 --> 00:05:54,200
Okay, so that's of course norm
conservational so means distance

88
00:05:54,200 --> 00:05:57,380
conservation through these transforms or

89
00:05:57,380 --> 00:06:00,310
through the expansion into orthonormal
basis.

90
00:06:03,120 --> 00:06:04,130
Alright.

91
00:06:04,130 --> 00:06:06,250
The next step we want to do is
approximation.

92
00:06:06,250 --> 00:06:10,300
We had briefly mentioned this at the
beginning of this module.

93
00:06:10,300 --> 00:06:13,880
So, we have a vector in R3 here.

94
00:06:13,880 --> 00:06:21,400
And, the vector X should be represented in
a subspace spanned by e0 and e1.

95
00:06:22,790 --> 00:06:28,435
Well, that's a subspace S of V.
Spanned by e0 e1,

96
00:06:28,435 --> 00:06:33,270
now shown in red, and the approximation is
x hat.

97
00:06:34,280 --> 00:06:37,790
It belongs to S, and it's the orthogonal
projection

98
00:06:37,790 --> 00:06:41,770
x to the plane spanned by e0 and e1.

99
00:06:41,770 --> 00:06:42,310
Okay?

100
00:06:42,310 --> 00:06:45,110
And that orthonormal projection we'll
denote by x hat.

101
00:06:47,720 --> 00:06:50,570
How can we do this?
It is very simple.

102
00:06:50,570 --> 00:06:53,375
We take an orthonormal basis for the
subspace.

103
00:06:53,375 --> 00:06:57,140
Okay, so remember we have the big space V.

104
00:06:57,140 --> 00:06:59,450
WE have the subspace c.

105
00:06:59,450 --> 00:07:02,310
So we take an orthonormal basis for the
subspace and

106
00:07:02,310 --> 00:07:05,148
the orthogonal projection is simply going
to be given by.

107
00:07:05,148 --> 00:07:12,650
X hat expanded in the ordinal basis that
spans the subspace, Okay?

108
00:07:12,650 --> 00:07:15,960
So the set of vectors sk, is the
orthogonal basis for

109
00:07:15,960 --> 00:07:20,190
s and here is the expansion form well in
this ordinal basis.

110
00:07:21,250 --> 00:07:24,330
This orthogonal projection is the best
approximation.

111
00:07:24,330 --> 00:07:29,900
Over S to the vector x.
And it's best in the l

112
00:07:29,900 --> 00:07:35,560
2 sense, or, it will minimize the
quadratic norm of the error.

113
00:07:37,550 --> 00:07:38,450
Okay.

114
00:07:38,450 --> 00:07:42,760
So in a word, orthogonal projection has
minimum-norm error.

115
00:07:42,760 --> 00:07:43,490
So.

116
00:07:43,490 --> 00:07:49,010
Among all vectors y that belong to s, the
one that minimizes

117
00:07:49,010 --> 00:07:53,000
the square of the difference here, so the
square norm of the

118
00:07:53,000 --> 00:07:57,790
difference x minus y is this vector x hat
that we have

119
00:07:57,790 --> 00:08:01,320
written out in terms of an ordinal basis
for the subspace s.

120
00:08:02,680 --> 00:08:05,590
Very important property is that the error
x

121
00:08:05,590 --> 00:08:10,430
minus x hat is orthogonal to the
approximation, okay?

122
00:08:10,430 --> 00:08:14,770
So we'll see it in the next slide, but
this is an extremely important formula.

123
00:08:14,770 --> 00:08:20,610
It's called the orthogonality principle.
It is used all over signal processing.

124
00:08:20,610 --> 00:08:25,180
When we minimize quadratic error.
All right.

125
00:08:25,180 --> 00:08:27,910
So let's see this very graphically; we
have

126
00:08:27,910 --> 00:08:34,100
s a subspace, this guy is subspace, so V,
V is the ambient space

127
00:08:34,100 --> 00:08:40,380
in this particle case R2 as is R1, it's a
one dimensional subspace.

128
00:08:40,380 --> 00:08:40,610
Okay.

129
00:08:40,610 --> 00:08:45,370
So we want to find the closest point to x.
So x is what we want to approximate.

130
00:08:45,370 --> 00:08:49,810
We'd like to find the closest point
somewhere here in S.

131
00:08:49,810 --> 00:08:50,430
Okay?

132
00:08:50,430 --> 00:08:53,010
How do we do this?
It has to be closest in the 2

133
00:08:53,010 --> 00:09:01,280
norm, so we create a circle around the tip
here of x, and we grow the circle.

134
00:09:01,280 --> 00:09:01,580
Okay.

135
00:09:01,580 --> 00:09:02,800
We are still not at s.

136
00:09:02,800 --> 00:09:05,028
And at some point, we reach s.

137
00:09:05,028 --> 00:09:08,310
This is cl-, either it is a closest one,
right?

138
00:09:08,310 --> 00:09:12,070
Because the circles measure quadratic
distance.

139
00:09:12,070 --> 00:09:15,260
And the first time we hit s, it is exactly
here.

140
00:09:15,260 --> 00:09:18,520
Okay.
So we have x-hat in blue.

141
00:09:18,520 --> 00:09:25,150
And we notice that x minus x-hat, the red
vector is orthogonal to the blue vector.

142
00:09:25,150 --> 00:09:25,735
That's your

143
00:09:25,735 --> 00:09:26,070
[INAUDIBLE]

144
00:09:26,070 --> 00:09:28,280
principle we have seen on the previous
slide.

145
00:09:31,430 --> 00:09:34,730
Let's look at a very concrete example.
It's polynomial approximation.

146
00:09:36,040 --> 00:09:38,995
So we go back to our favorite interval, -1
to 1.

147
00:09:38,995 --> 00:09:44,350
Okay, so we will get these guys, and for
this

148
00:09:44,350 --> 00:09:50,350
interval, we define.
As subspace pn minus one to one

149
00:09:50,350 --> 00:09:56,630
which are polynomials up to degree n minus
one on the interval minus one two,

150
00:09:56,630 --> 00:10:00,750
okay?
So a basis for this is simply to take

151
00:10:03,000 --> 00:10:08,670
as the successive monomials tk for k going
from zero to capital n minus one.

152
00:10:08,670 --> 00:10:10,660
Okay, so a naive basis here.

153
00:10:10,660 --> 00:10:17,260
Is really 1, t, t squared, t cubed,
etcetera.

154
00:10:17,260 --> 00:10:17,920
Okay.

155
00:10:17,920 --> 00:10:19,980
Now, these guys are not orthonormal to
each other.

156
00:10:19,980 --> 00:10:20,100
Okay?

157
00:10:20,100 --> 00:10:24,330
So this naive basis is not orthogonal.

158
00:10:24,330 --> 00:10:28,002
Okay, which is self-evident, because, for
example, On the

159
00:10:28,002 --> 00:10:32,860
interval -1 to 1.
Let's take this interval.

160
00:10:32,860 --> 00:10:34,800
We have the first element.

161
00:10:34,800 --> 00:10:36,070
That's this guy.
Okay.

162
00:10:36,070 --> 00:10:41,980
The second guy, he's orthogonal, because
the first one was symmetric, the second

163
00:10:41,980 --> 00:10:48,830
one is antisymmetric.
But the third guy is a quadratic function.

164
00:10:48,830 --> 00:10:51,930
And it's not properly scaled.

165
00:10:51,930 --> 00:10:53,320
I apologize.
But the quadratic function,

166
00:10:53,320 --> 00:10:57,030
of course, is also symmetric.
So for example, the inner product between

167
00:10:57,030 --> 00:11:04,130
1 and t square on the interval minus 1 to
1 is not equal to 0.

168
00:11:04,130 --> 00:11:05,530
Okay?

169
00:11:05,530 --> 00:11:07,520
Please check this more explicitly if

170
00:11:07,520 --> 00:11:10,520
you like, but it's fairly geometrically
evident.

171
00:11:12,220 --> 00:11:14,330
Okay, now we're are going to try to
approximate

172
00:11:14,330 --> 00:11:17,020
something that does not live on the
polynomial space.

173
00:11:17,020 --> 00:11:18,360
So that would be for

174
00:11:18,360 --> 00:11:19,910
example trigonometric functions.

175
00:11:19,910 --> 00:11:23,080
So take x, what we are going to
approximate as

176
00:11:23,080 --> 00:11:28,290
sin t over minus the interval minus one to
one.

177
00:11:28,290 --> 00:11:30,830
And we would like to approximate it on p
3.

178
00:11:30,830 --> 00:11:34,460
So up to polynomials of a third degree

179
00:11:37,320 --> 00:11:42,600
Okay, so the way to do it is, we build an
orthonormal basis from the naive basis.

180
00:11:42,600 --> 00:11:45,150
We project x over the orthonormal basis.

181
00:11:45,150 --> 00:11:48,250
We compute the approximation error.
Okay?

182
00:11:48,250 --> 00:11:52,990
So same thing as usual.
So S.

183
00:11:52,990 --> 00:11:57,160
Here is this P3 space.

184
00:11:57,160 --> 00:12:02,340
Our sin is somewhere out there, and we
want to compute this.

185
00:12:02,340 --> 00:12:07,554
To do this, we first construct an ordinal
basis for the sub spaces.

186
00:12:07,554 --> 00:12:08,727
Okay.

187
00:12:08,727 --> 00:12:10,008
We can compare this to the

188
00:12:10,008 --> 00:12:13,510
naive approximation, which would be Taylor
approximation.

189
00:12:13,510 --> 00:12:18,745
It's well known, very useful, but it's not
optimal, as we will see in just a minute.

190
00:12:18,745 --> 00:12:27,080
Okay, so from the naive basis, remember,
we have the naive basis, it's 1,

191
00:12:27,080 --> 00:12:33,180
t, T square, t cube etc.

192
00:12:33,180 --> 00:12:38,400
We can compute so that's a viral basis we
compute an orthogonal basis.

193
00:12:38,400 --> 00:12:43,550
There is a procedure to do this which is
called the Gram-Schmidt algorithm, okay.

194
00:12:43,550 --> 00:12:45,930
You explain in the appendix of this
lecture, we

195
00:12:45,930 --> 00:12:48,340
are not going to do it in the main
lecture.

196
00:12:48,340 --> 00:12:52,140
And it's a recursive procedure where you
take the first one You

197
00:12:52,140 --> 00:12:53,700
normalize it.
That's fine.

198
00:12:53,700 --> 00:12:57,860
You take the second one, and you make sure
it's ordinal to the first one.

199
00:12:57,860 --> 00:12:59,880
You normalize it, and you keep going.

200
00:12:59,880 --> 00:13:03,090
And the result of this is that you get
ordinal null

201
00:13:03,090 --> 00:13:06,400
vectors, u 0, which is just a scaled
version of 1.

202
00:13:06,400 --> 00:13:09,770
The second one, these two ordinals, you
don't

203
00:13:09,770 --> 00:13:12,630
have to change much except for the
scaling.

204
00:13:12,630 --> 00:13:16,151
The third one is a transformation of
t-square.

205
00:13:16,151 --> 00:13:17,460
T-square,

206
00:13:17,460 --> 00:13:20,570
I mean, like this, more explicit,
t-square.

207
00:13:20,570 --> 00:13:23,650
Into, you know, a second order point on u,
which

208
00:13:23,650 --> 00:13:28,980
is constructed such that u2 is orthogonal
to u0 and u1.

209
00:13:28,980 --> 00:13:31,800
And you can keep going like this.
It's a standard construction.

210
00:13:31,800 --> 00:13:33,440
It's called Legendre polynomials.

211
00:13:33,440 --> 00:13:35,640
And just from the name, you know this is

212
00:13:35,640 --> 00:13:39,610
a 19th century construction, so it's
extremely well known.

213
00:13:39,610 --> 00:13:42,760
And the appendix gives the details.
Okay, so now we have an

214
00:13:42,760 --> 00:13:44,630
ordinal basis for the subspace.

215
00:13:46,470 --> 00:13:50,340
And, let's just watch these Legendre's
polynomials, they're sort of cute,

216
00:13:50,340 --> 00:13:54,729
so the first one remember, it's, 1 over
square root of 2.

217
00:13:55,810 --> 00:13:58,840
So here we go.
That's 0.7 something.

218
00:13:58,840 --> 00:14:00,720
It's the black line.

219
00:14:00,720 --> 00:14:04,490
The second one is proportional to t, but
it has been scaled.

220
00:14:05,660 --> 00:14:08,270
The third

221
00:14:08,270 --> 00:14:12,700
one, which is quadratic, has been moved
about.

222
00:14:12,700 --> 00:14:15,160
So now it is actually orthogonal to

223
00:14:15,160 --> 00:14:18,150
It's automatically, the yellow color is
automatically

224
00:14:18,150 --> 00:14:20,660
orthogonal to the red curve because one

225
00:14:20,660 --> 00:14:23,050
is symmetrics the other one is
anti-symmetric.

226
00:14:23,050 --> 00:14:28,710
But the shift that was added if we go back
here, let me just show you if I

227
00:14:28,710 --> 00:14:33,560
shift here, this shift makes sure that the
inner product between the yellow curve and

228
00:14:33,560 --> 00:14:35,470
the black curve is zero.
Okay?

229
00:14:36,590 --> 00:14:41,450
And it's a fourth, is a third degree
polynomial, is the green guy.

230
00:14:42,510 --> 00:14:46,770
It is antisymmetric, so it will be
automatically orthogonal to the black one.

231
00:14:46,770 --> 00:14:53,470
The yellow one, but it has to be adjusted
so that is orthogonal to the red one.

232
00:14:53,470 --> 00:14:54,710
And we can keep going.

233
00:14:54,710 --> 00:14:58,660
Okay, so that's a four story guy.
Four order polynomial, and so on.

234
00:14:58,660 --> 00:15:01,270
So Legendre polynomials go off to
infinity, but

235
00:15:01,270 --> 00:15:02,850
we'll just look at a few of them.

236
00:15:02,850 --> 00:15:03,350
Okay?

237
00:15:04,900 --> 00:15:06,170
Here's a phase one.

238
00:15:06,170 --> 00:15:07,820
And it's a very cute picture.

239
00:15:07,820 --> 00:15:10,315
And this set of polynomials on this
interval

240
00:15:10,315 --> 00:15:14,950
-1 to 1, okay, they're important on this
interval.

241
00:15:14,950 --> 00:15:18,410
It's defined in such, it's constructed
actually

242
00:15:18,410 --> 00:15:21,280
in such a way that the inner product

243
00:15:24,200 --> 00:15:30,450
of two of these functions is equal to
delta i minus j.

244
00:15:30,450 --> 00:15:34,490
So it's equal to zero when i is different
from.

245
00:15:34,490 --> 00:15:38,590
j, and it's equal to one when, i is equal
to j, okay?

246
00:15:38,590 --> 00:15:42,550
It's not self-evident when you look at
the, at the functions

247
00:15:42,550 --> 00:15:46,090
except for the symmetries that I pointed
out, a minute ago.

248
00:15:47,850 --> 00:15:48,610
Alright.

249
00:15:48,610 --> 00:15:49,640
So now we can compute

250
00:15:49,640 --> 00:15:53,200
our expansion coefficients.
Remember, we want to write the formula

251
00:15:53,200 --> 00:15:59,200
where x hat is going to be some sum of
alpha-k.

252
00:15:59,200 --> 00:16:06,050
u k we call these guys here, and k goes
from 0 to capital K minus 1.

253
00:16:06,050 --> 00:16:10,950
So that's the orthogonal projection onto
the subspace spanned by the u k.

254
00:16:10,950 --> 00:16:14,458
Alright, so we have to take the integrals
between -1,

255
00:16:14,458 --> 00:16:15,020
1.

256
00:16:15,020 --> 00:16:17,607
Of the function, these polynomials, we
have just defined

257
00:16:17,607 --> 00:16:17,940
[UNKNOWN]

258
00:16:17,940 --> 00:16:22,200
polynomials and the function we want to
approximate sine t.

259
00:16:22,200 --> 00:16:23,200
Okay.

260
00:16:23,200 --> 00:16:29,070
So sin is of course an odd function as we
know

261
00:16:29,070 --> 00:16:33,280
as so alpha zero is going to be
automatically equal to zero.

262
00:16:33,280 --> 00:16:35,350
Alpha one is going to be different from
zero,

263
00:16:35,350 --> 00:16:41,090
because both u one and sine are odd
functions.

264
00:16:41,090 --> 00:16:43,020
So that's what you get if you

265
00:16:43,020 --> 00:16:46,951
do the integral, 0.73 something.

266
00:16:46,951 --> 00:16:49,830
And the third coefficient, alpha two, is
also

267
00:16:49,830 --> 00:16:52,550
equal to zero, because this character here
is.

268
00:16:53,680 --> 00:16:57,980
Symmetric, and this one is antisymmetric,
so it's automatically equal to 0.

269
00:16:57,980 --> 00:17:03,690
Okay, so what do we get?
So, when we do the orthogonal projection

270
00:17:03,690 --> 00:17:08,640
on these three basis vectors,

271
00:17:08,640 --> 00:17:14,220
so, u0, u1, u2, we get an approximation,
which is

272
00:17:14,220 --> 00:17:18,900
simply alpha 1, u1 And it's given by this
formula.

273
00:17:18,900 --> 00:17:22,530
If we do Taylor series, then the first
order approximation of

274
00:17:22,530 --> 00:17:28,860
Taylor series simply to take sin t equal
to t, okay.

275
00:17:28,860 --> 00:17:31,640
And so we're going to compare these two
approximates, they

276
00:17:31,640 --> 00:17:33,820
look very similar but one has been scaled
a little

277
00:17:33,820 --> 00:17:35,640
bit, okay.
Alright.

278
00:17:37,030 --> 00:17:42,570
So now we see that the approximation of
sine, which is the blue curve, the smooth

279
00:17:42,570 --> 00:17:48,790
blue curve here, t is the red curve, and
green is simply a scaled version.

280
00:17:48,790 --> 00:17:50,610
Doesn't look like a big deal.

281
00:17:50,610 --> 00:17:53,570
It's, you know, 10% smaller, but you can
immediately see that it's

282
00:17:53,570 --> 00:17:58,184
actually hugging the blue curve more
closely over the interval -1 to 1,

283
00:17:58,184 --> 00:17:58,850
right?

284
00:17:58,850 --> 00:18:00,860
So we are approximating this over this
interval.

285
00:18:00,860 --> 00:18:04,480
If we change the interval So, our
approximation would look different.

286
00:18:04,480 --> 00:18:05,040
Okay?

287
00:18:05,040 --> 00:18:07,420
But over this interval, as we can see

288
00:18:07,420 --> 00:18:11,060
here, we plugged the absolute value of the
difference.

289
00:18:11,060 --> 00:18:13,640
The red one is sin t minus t.

290
00:18:13,640 --> 00:18:17,590
Okay, it goes off quite a bit at the end
intervals here.

291
00:18:17,590 --> 00:18:21,850
It's very nice in the region around zero
region, okay.

292
00:18:21,850 --> 00:18:23,450
And we see that the

293
00:18:23,450 --> 00:18:28,810
green approximation which is sin t minus
our projection onto

294
00:18:28,810 --> 00:18:34,540
the subspace of the legorn-, the legendre
polynomials of orders zero

295
00:18:34,540 --> 00:18:40,300
one and two That error is overall, it is
smaller.

296
00:18:40,300 --> 00:18:42,630
Never goes goes off to these values, okay.

297
00:18:42,630 --> 00:18:47,460
Sometimes it's bigger, but overall, it
actually turns out to be smaller, okay?

298
00:18:48,540 --> 00:18:54,720
So, to compare this numerically, we can
compute the norm of

299
00:18:54,720 --> 00:19:00,900
sine t minus alpha 1 t, and it's 0.0337 In
Taylor

300
00:19:00,900 --> 00:19:06,110
serie, it's almost three times bigger.
It's 0.08.

301
00:19:06,110 --> 00:19:08,750
Necessarily we have to be as good or

302
00:19:08,750 --> 00:19:13,222
better than Taylor series because it's a
theorem.

303
00:19:13,222 --> 00:19:13,860
It's the orthogonal

304
00:19:13,860 --> 00:19:17,950
projection theorem we find the minimum
norm approximation.

305
00:19:17,950 --> 00:19:19,180
Okay?

306
00:19:19,180 --> 00:19:23,370
And with this, we have Showing on a very
practical

307
00:19:23,370 --> 00:19:28,450
example how to do orthogonal approximation
using an orthogonal basis.

308
00:19:30,470 --> 00:19:32,960
Now, this was all a lot of work.

309
00:19:32,960 --> 00:19:38,200
We defined Hilbert spaces, we had a lot of
definitions, and so on.

310
00:19:38,200 --> 00:19:42,340
So why do we do all this?
It's a great question.

311
00:19:42,340 --> 00:19:48,910
So, first is at both finite-length and
periodic signals live in C N.

312
00:19:48,910 --> 00:19:52,350
So we can use all of linear algebra and

313
00:19:52,350 --> 00:19:55,670
all the geometry of linear algebra to do
this.

314
00:19:55,670 --> 00:19:55,915
And

315
00:19:55,915 --> 00:20:01,130
Infinite-length signals, that we like for
general signal processing live in

316
00:20:01,130 --> 00:20:05,130
a more general Hilbert space, which is
small l2 of Z.

317
00:20:05,130 --> 00:20:08,300
Okay so we have a common geometric frame
work for

318
00:20:08,300 --> 00:20:13,210
both finite lengths, periodic signals and
infinite length sequences, okay.

319
00:20:13,210 --> 00:20:18,890
So we have one way to think about the
whole bunch of different problems, okay.

320
00:20:18,890 --> 00:20:21,300
Then we'll see that the expansion into
orthogonal

321
00:20:21,300 --> 00:20:24,380
bases is very central to signal
processing.

322
00:20:24,380 --> 00:20:27,830
So em, we can use different bases.

323
00:20:27,830 --> 00:20:30,240
As different observation tools for
signals.

324
00:20:30,240 --> 00:20:33,130
We're going to see something called the
Short

325
00:20:33,130 --> 00:20:37,860
Time Fourier Transform, to be defined in
Module 4.

326
00:20:37,860 --> 00:20:42,870
And the Short Time Fourier Transform as
the second half says, is

327
00:20:42,870 --> 00:20:46,330
something like a Fourier transform, but
it's a very particular way to look

328
00:20:46,330 --> 00:20:48,140
at signals.
It's very popular.

329
00:20:48,140 --> 00:20:51,350
For doing speech analysis and the like.
Okay?

330
00:20:51,350 --> 00:20:54,420
And when we do subspace projections, we
will

331
00:20:54,420 --> 00:20:58,110
see that we can do filtering, which will

332
00:20:58,110 --> 00:20:59,910
be explained, of course, in detail, in
this

333
00:20:59,910 --> 00:21:02,855
class, and we can do, for example, image
compression.

334
00:21:02,855 --> 00:21:04,230
Okay.

335
00:21:04,230 --> 00:21:10,030
So the notions we have seen, one was to
build bases, that's important.

336
00:21:10,030 --> 00:21:11,730
These are like our

337
00:21:11,730 --> 00:21:13,540
tools to look at signals.

338
00:21:13,540 --> 00:21:17,690
Okay, and another one was subspace
projection which was

339
00:21:17,690 --> 00:21:21,760
something that will come a lot when we do
approximation.

340
00:21:21,760 --> 00:21:23,990
And in particular when we do compression.

341
00:21:25,390 --> 00:21:28,070
Okay, let me just finish this properly
here.

342
00:21:28,070 --> 00:21:30,940
We have the origin, we have a sub space s,
we have

343
00:21:30,940 --> 00:21:36,120
x and we have the orthogonal projection
and we have the orthogonality principle.