>> Okay.
So, that completes the lecture.
So, just to summarize some main lessons
we've learned here.
There were three steps in deriving the
language models I've shown you.
The first step was to expand the joint
probability over a sequence of words, w1,
w2, up to wn, using the Chain rule of
probabilities.
This is what I showed you in the portion
of the lecture on Markov processes.
And the second step was to make Markov
independence assumptions.
In particular, assuming that the
probability of some word, wi, conditioned
on the entire previous sequence of i minus
1 previous words, actually depends only on
the previous two words in this sequence,
we call this a second-order Markov
assumption.
And the final step in deriving this, these
estimates was to smooth these diagram
estimates essentially using low order
accounts.
And that was done either through the
method of linear interpolation or through
this discounting method that I just showed
you.
So, just briefly, language modelling is a
huge industry and there's been a lot of
research in improved methods for language
modelling.
Some areas of particular interest are
methods that model the underlying topic of
documents or other long-range features of
a document.
So, conditioning on just the previous two
words is certainly limiting and in, in
some cases, you might want to condition on
the fact that a, a document is about
sports, or is about politics, or the
general topic that can influence the words
that are seen in the document and might be
important to condition on that.
Or we might condition on words which are
outside this two-word window, there's been
considerable interest in that problem.
Another type of model we'll see later in
the class, is language models built based
on syntactic models.
Language models that explicitly trying to
incorporate grammatical information,
information about what sentences a
grammatical versus non-grammatical in a
language.
And again, these models can often capture
the long range features, which fall
outside just a, a two-way window.
A rule though, it can be quite, quite
difficult to improve upon language models.
They're simple, they're very efficient and
they can get us a long way in many
problems.