11-761 Spring 2001 Course Syllabus

Special dates:
1/24: class starts at 11:00am; goes beyond 11:50 as needed.
1/29: no class
3/19: in-class midterm
4/3 4:00PM, WeH 5409: make-up lecture in lieu of 4/4
4/27 9:30AM, WeH 4601: special session to review midterm
5/2: in-class final

We will move at whatever pace we find comfortable. If there is time left at the end, we will either add topics (papers) or review project work.
Brush up on college level Prob & Stats: [mD] ch 1-5. (optional)

Topics	Dates	Required reading (due date)	Additional Background
Course Goals, Philosophy and Mechanics	1/15
Statistical Approach to Language: Overview and Historical Perspective (Lafferty's notes)	1/15		[MS] 1.1 - 1.3
Statistical Language Modeling, Computational Linguistics, Statistical Decision Making, the Source-Channel Paradigm	1/17, 1/22	[MS] 2.1 (1/24)
All About Words: Types, Tokens and Vocabularies	1/22	[MS] 1.4 (1/24)	[BCH] ch. 4
Unigrams: Statistical Estimation, Maximum Likelihood Estimates;	1/22, 1/24		[mD] ch.6, esp. 6.5 [MS] 6.2.1-6.2.2
Sparseness; Smoothing	1/31, 2/5
N-grams: linear interpolation; backoff	2/7, 2/12, 2/14	[MS] ch 6 ( 2/5) [sK] (2/5)
Measuring Success: Perplexity and Entropy	2/19		[MS] 2.2 IT notes Entropy of English
Clustering	2/21		[MS] 14.1, class LM, Lattice LM
Latent Variable Models; the EM Algorithm; examples	2/26, 2/28, 3/5	Derivation of EM for Gaussian mixture, EM derivation shortcut for exponential family	[MS] 14.2 EM notes
Hidden Markov Models	3/5		[MS] ch. 9, HMMs for speech
Decision Tree Language Models	3/7, 3/12		[MS] 16.1, [BBDM]
Maximum Entropy Modeling	3/12, 3/14, 3/21		[MS] 16.2, [rR], [BDD], slides
Mid-term exam; review of midterm	3/19, 3/21
Statistical Machine Translation (guest lectures by John Lafferty)	4/2, 4/3		[MS] 13.1- 13.3 Papers:1,2,3,4,5
Whole-sentence language models; Semantic coherence	4/9		[RCZ]
Stochastic Grammars, the Inside-Outside algorithm	4/11	Notes on Probabilistic Context Free Grammars	[MS] 11.1-11.4
Dimensionality Reduction; Latent Semantic Analysis	4/16	Bellegarda 99 (4/16)	Bellegarda 00
A Structured Language Model	4/18		ChelbaSlides 98, JelinekChelba 99
Language Model Adaptation	4/18
Statistical Information Retrieval (guest lecture by Jamie Callan)	4/23		[MS] 15.1- 15.4
Statistical Information Extraction (guest lecture by Andrew McCallum)	4/25		[MS] 15.5 McCallum paper 1, McCallum paper 2, last year paper
Review	4/30

Abbreviations (in order or appearance):

[MS] - Manning and Schutze, Foundations of statistical natural language processing.
[BCW] - Bell, Cleary and Witten , Text Compression.
[mD] - Morris Degroot, Probability and Statistics, 2nd edition.
[sK] - Slava M. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recognizer". IEEE-ASSP 35 no.3 p 400-401 1987.
[BBDM] - "A Tree-Based Statistical Language Model for Natural Language Speech Recognition", L. Bahl, P. Brown, P. deSouza and R. Mercer, IEEE Transcation on Acoustics, Speech and Signal Processing , volume 37, pages 1001--1008, 1989.
[rR] - "A Maximum Entropy Approach to Adaptive Statistical Language Modeling", Ronald Rosenfeld, Computer, Speech and Language, volume 10, pages 187--228, 1996.

Special dates: 1/24: class starts at 11:00am; goes beyond 11:50 as needed. 1/29: no class 3/19: in-class midterm 4/3 4:00PM, WeH 5409: make-up lecture in lieu of 4/4 4/27 9:30AM, WeH 4601: special session to review midterm 5/2: in-class final

Special dates:
1/24: class starts at 11:00am; goes beyond 11:50 as needed.
1/29: no class
3/19: in-class midterm
4/3 4:00PM, WeH 5409: make-up lecture in lieu of 4/4
4/27 9:30AM, WeH 4601: special session to review midterm
5/2: in-class final