Statistische Modellen van Natuurlijketaal verwerking

Statistical Models of Natural Language Processing

(Kunstmatige Intelligentie, Specialisatie Taalverwerking)

Instructor: Khalil Sima'an
Time: Second trimester (January-April): tuesday 11:00-13:00 hours
Place:   THE LECTURES HAS MOVED TO B.235 (my room is B234)


When computational models of language processing are not constructed in a purely linguistic context,
but aim at being relevant for practical applications or for psychological theory, they ought to be able to
perform tasks like disambiguation and prediction. For this reason, increasingly many models take statistical
properties of a sample corpus into account when they process new input. This course will give an overview
of the most important techniques used in statistical language processing. The course starts out by giving a
short introduction to probability theory, information theory and Bayesian learning, and continues with the
following topics: n-gram statistics and Markov models, smoothing techniques (Good-Turing and Katz
Discounting), Hidden-Markov Models (HMMs),  application of HMMs to part-of-speech  tagging,
Stochastic Context-Free Grammars (SCFGs) and stochastic parsing algorithms. The last two meetings
will be devoted to more advanced topics in stochastic parsing, possibly including: Bilexical-Dependency
models  and Data-Oriented Parsing (DOP) models.

Course material:

  Primary book:
     * Chris Manning and Hinrich Schütze. "Foundations of Statistical Natural Language Processing",
                 MIT Press. Cambridge, MA: May 1999.  (see )

  Secondary books (some chapters):
     * Daniel Jurafsky and  James H. Martin. `"SPEECH and LANGUAGE PROCESSING":
        An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition.
        Prentice-Hall, 2000.
     * Eugene Charniak.  "Statistical Language Learning", Cambridge, Mass, MIT Press, 1993.
     * Tom Mitchell. "Machine Learning", McGraw-Hill Series in Computer Science, 1997.

  Other material (see also list below):
     *  Joshua Goodman and Stanley Chen. "An empirical study of smoothing techniques for language modeling".
                      Technical report TR-10-98, Harvard University, August 1998.
                       see also
                       A correction of a small error in the statement of Katz formula in this report can be found here.
     * Various recent articles which will be supplied as handouts during the course.
     * Possibly: R. Bod.  "Beyond Grammar", Stanford CA, CSLI, 1998.

Examination: written exam or conclusive project

Some sources available on line (list is being exapanded):

Krenn, B. and Samuelsson, Ch. (1997).  ``The Linguist's Guide to Statistics".
Abney, S. (1996). ``Statistical Methods And Linguistics".
Charniak, E. (1997) `` Statistical techniques for natural language parsing'', AI Magazine
Shannon, C.E. (1948) ``A Mathematical Theory of Communication".
Pereira, F. (2000) Formal Grammar and Information Theory: Together Again?
          Philosophical Transactions of the Royal Society, 358(1769):1239-1253, April 2000.