16-721: Advanced Perception

Course Information

Time : Tuesdays and Thursdays, 10:30 am -- 11:50 am

Location : NSH 3002

Pre-requisites : Linear Algebra, Calculus, Undergraduate or Graduate level Vision

Web page : http://www.cs.cmu.edu/~srinivas/adv_perception/

Instructor

Prof. Srinivasa Narasimhan

http://www.cs.cmu.edu/~srinivas

Email: srinivas@cs.cmu.edu

Office: NSH 3211

Office Hours: Tuesdays 4:00 pm -- 5:00 pm

Teaching Assistant

TBD

Overview

This is a reading course on advanced topics in computer vision. The goal is to get graduate students (and advanced undergraduates) up to speed with current literature in the field of computer vision. This year's course plans to cover important papers in topics including sensors, physics-based models in computer vision and topics in the intersection of vision and graphics like image-based rendering. There will be introductory lectures in each of these topics, followed by presentations and writeups by students analyzing and comparing a few papers in an area. There will also be a few guest lectures by established researchers. We assume students have a rudimentary understanding of linear algebra, calculus, and have already taken a preliminary course in vision (either undergraduate level or graduate level). There will be no homework assignments other than readings and class presentations. There will be no exams. The readings will come from various journals and will be assigned as the course progresses. No specific text is required. This course is modeled along the lines of similar courses at Berkeley, Columbia and Stanford.

Topics

Sensors and Computational Imaging

Camera Fundamentals and Calibration
High Dynamic Range Imaging
Light Fields
Wide-angle and Panoramic Imaging
Demosaicing and Super-resolution
Motion Deblurring and High Speed Imaging
Projector-Camera Systems
3D Laser and Structured Light Range Scanning

Appearance modeling in Vision and Graphics

Reflectance (BRDF) Models
Inverse Rendering
Image-based Modeling and Rendering
Scattering in Vision and Graphics
Illumination Modeling
Environment Matting and Compositing
Appearance-Based Face and Object Recognition
Signal theoretic framework for reflection

Optional Texts

Robot Vision, B. K. P. Horn, MIT Press, 1986
Computer Vision: A Modern Approach, David A. Forsyth, Jean Ponce
Computer Vision, D. Ballard and C. Brown, Prentice-Hall, 1982, ISBN 0-13-165316-4

Grading

Three Class Presentations (atleast one in each of the two major areas described above) 45%
Class Attendance and Participation 15%
Writing a survey paper 40%
Project implementing a published paper (instead of the writeup) 40%

Note: Class Schedule is not finalized. Topics and dates may change depending on when invited speakers are able to visit.

PART I: Sensors and Computational Imaging

Class 1 : 01/11/2005
- Introduction (Part 1) + Course Mechanics

Class 2 : 01/13/2005

Introduction (Part 2)

Classes 3 and 4 : High Dynamic Range Imaging : 01/18/2005, 01/20/2005

B.C. Madden. Extended Intensity Range Image,Grasp,UPenn, 1996.
P. Devebec and J. Malik. Recovering High Dynamic Range Radiance Maps from Photographs, ACM SIGGRAPH, Vol 31, pp 369--378, 1997. ( assigned to Sanjeev )
M. Grossberg and S.K. Nayar, Determining the Camera Response from Images: What is Knowable? PAMI, Vol 25, pp 1455--1467, 2003.
S.K. Nayar and T. Mitsunaga, High Dynamic Range Imaging: Spatially Varying Pixel Exposures, CVPR, pp 472--479, 2000.
S. B. Kang, M. Uyttendaele, S. Winder and R.Szeliski, High Dynamic Range Video, ACM Transactions on Graphics, Vol. 22, pp 319--325, 2003.
Y. Y. Schechner and S. K. Nayar, Generalized Mosaicing, ICCV, July 2001. ( assigned to Ethan)
S. K. Nayar and V. Branzoi Adaptive Dynamic Range Imaging: Optical Control of Pixel Exposures Over Space and Time, ICCV, October 2003.
S. K. Nayar, V. Branzoi, and T. Boult, Programmable Imaging using a Digital Micromirror Array, CVPR, June 2004. (assigned to Tom)

Classes 5 and 6 : Light Fields : 01/25/2005, 01/27/2005

A. Gershun, Svetovoe Pole (The Light Field, in English), Journal of Mathematics and Physics, Vol. XVIII, pp 51--151, 1939.
E. H.Adelson and J. R.Bergen, The Plenoptic Function and the Elements of Early Vision, Computation Models of Visual Processing, pp 385--394, The MIT Press, 1991.
M. Levoy and P. Hanrahan, Light Field Rendering, Computer Graphics, Proc. SIGGRAPH, 1996. ( assigned to Jean-Francois )
S. Gortler, R. Grzeszczuk, R. Szeliski, M. Cohen, The Lumigraph. SIGGRAPH 96, pp 43-54.

Classes 7 and 8 : Wide-angle Imaging: 02/01/2005, 02/03/2005

K. Miyamoto, Fish Eye Lens, Journal of Optical Society of America, Vol. 54(8), pp 1060--1061, August 1964.
S. Baker and S. K. Nayar, A Theory of Catadioptric Image Formation, ICCV, pp 35--42, 1998.
S.K. Nayar and V.N. Peri, Folded Catadioptric Cameras, CVPR, Vol. 2, pp 217--223, 1999. ( assigned to Kenny )
R.A. Hicks and R.C. Bajcsy, Catadioptric Sensors that Approximate Wide-Angle Perspective Projections, CVPR, Vol. 1, pp 545--551, 2000. ( assigned to Jean-Francois )
J. Chahl and M. Srinivasan, Reflective surfaces for panoramic imaging, Applied Optics, Vol 36, Issue 31, pp 8275--8285, 1997.
R. Swaminathan, M. D. Grossberg, and S. K. Nayar, Caustics of Catadioptric Cameras, ICCV 2001.
K. Nishino and S. K. Nayar, The World in Eyes, CVPR 2004. ( assigned to Marek )

Special Lecture (VASC Seminar, 3:30pm) : Motion Deblurring : 02/07/2005

Special Lecture by Moshe Ben-Ezra (Siemens Corporate Research, Princeton).
M. Ben-Ezra and S. K. Nayar, Motion Deblurring using Hybrid Imaging, CVPR, Wisconsin, June 2003. ( Lecture )
M. Ben-Ezra, Assaf Zomet, and S. K. Nayar, Jitter Camera : High Resolution Video from a Low Resolution Detector, CVPR, Washington DC, June 2004. ( Lecture )

Classes 9 and 10 : Demosaicing and Superresolution: 02/08/2005, 02/10/2005

R. Ramanath, W. Snyder, G. Bilbro and W. Sander, Demosaicking methods for Bayer color arrays,Journal of Electronic Imaging, Vol 11, No. 3, pp 306--315, July 2002. ( assigned to Ankur )
S. G. Narasimhan and S. K. Nayar, Assorted Pixels: Multi-Sampled Imaging with Structural Models, ECCV 2002. ( assigned to Ankur )
M. Irani and S. Peleg, Improving Resolution by Image Registration. CVGIP: Graphical Models and Image Processing, Vol. 53, pp. 231-239, May 1991. ( assigned to Dinesh )
S. Baker and T. Kanade, Limits on Super-Resolution and How to Break Them. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 9, pp 1167--1183 (Section 1-3), September, 2002. (assigned to Sanjeev)
W.T. Freeman and E.C. Pasztor, Learning low-level vision, ICCV, Vol. 2, pp 1182--1189, 1999. (assigned to Ethan)

Class 11 : High Speed Imaging: 02/15/2005

Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, Mark Horowitz, High Speed Video Using a Dense Camera Array, CVPR 2004. ( assigned to Norman Papernick )

Classes 12 and 13 : Projector Camera Systems: 02/22/2005

R. Sukthankar, T.-J. Cham, G. Sukthankar. Dynamic Shadow Elimination for Multi-Projector Displays. Proceedings of Computer Vision and Pattern Recognition, 2001. ( assigned to Dinesh )
M. D. Grossberg, H. Peri, S. K. Nayar, and P. N. Belhumeur, Making One Object Look Like Another: Controlling Appearance Using a Projector-Camera System, CVPR, Washington DC, June 2004. (assigned to Philip)
T.-J. Cham, J. Rehg, R. Sukthankar, G. Sukthankar. Shadow Elimination and Occluder Light Suppression for Multi-Projector Displays. CVPR, 2003.
Shree K. Nayar, Harish Peri, Michael D. Grossberg, and Peter N. Belhumeur, A Projection System with Radiometric Compensation for Screen Imperfections, ICCV Workshop on Projector-Camera Systems (PROCAMS), Nice, France, October 2003.
Marc Levoy, Billy Chen, Vaibhav Vaish, Mark Horowitz, Ian McDowall, Mark Bolas, Synthetic aperture confocal imaging, SIGGRAPH 2004

Due to popular demand, the following topics on segmentation, tracking and recognition have also been included.

Special Class 14 : Human Tracking : 02/24/2005

Special Lecture by Simon Baker (CMU).

Readings for the lecture:
http://www.ri.cmu.edu/pubs/pub_4702.html
http://www.ri.cmu.edu/pubs/pub_4867.html

Special Class 15 : More Body Parts and Activity Recognition : 03/01/2005

Special Lecture by Alexei Efros (CMU).
Greg Mori, Xiaofeng Ren, Alexei A. Efros, Jitendra Malik, Recovering Human Body Configurations: Combining Segmentation and Recognition, CVPR 2004.
Alexei A. Efros, Alexander Berg, Greg Mori, Jitendra Malik, Recognizing Action at a Distance, ICCV 2003

Class 16 : Object Recognition : 03/03/2005

Henry Schneiderman and Takeo Kanade, Object Detection Using the Statistics of Parts, IJCV 2002. (assigned to Norman)
H. Schneiderman, T. Kanade. "A Statistical Method for 3D Object Detection Applied to Faces and Cars". IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2000)
P. Viola and M. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, CVPR 2001. (assigned to Marek)
A. Torralba, K. P. Murphy, W. T. Freeman and M. A. Rubin, Context-based vision system for place and object recognition, ICCV 2003. ( assigned to Ankur )
A. Torralba and A. Oliva, Statistics of Natural Image Categories, Network: Computation in Neural Systems. Vol. 14, 391-412.

Class 17 : Segmentation : 03/15/2005

Z.W. Tu and S.C. Zhu, Image Segmentation by Data-Driven Markov Chain Monte Carlo, IEEE Trans on PAMI, vol.24, no.5, pp. 657-673, May, 2002. ( assigned to --- )
E. Borenstein, E. Sharon, and S. Ullman, Combining Top-down and Bottom-up Segmentation, IEEE Workshop on Perceptual Organization in Computer Vision, POCV '04
J. Shi and J. Malik, Normalized Cuts and Image Segmentation, IEEE PAMI, Vol 22, No. 8, 2002. (assigned to Philipp)

Class 18 : Tracking : 03/17/2005

D. Comaniciu, V. Ramesh, P. Meer: Real-Time Tracking of Non-Rigid Objects using Mean Shift, CVPR 2000. (assigned to Mihail)
D. Comaniciu, V. Ramesh, P. Meer: Kernel-Based Object Tracking, IEEE Trans. PAMI., Vol. 25, No. 5, 564-575, 2003
Chris Stauffer and Eric Grimson, "Learning Patterns of Activity Using Real-Time Tracking", IEEE Transactions on PAMI, 22(8):747-757, 2000.
Chris Stauffer and Eric Grimson, "Adaptive background mixture models for real-time tracking", CVPR 1999.

Class 19 : 3D Range Scanning: 03/22/2005

Marc Levoy et al, The Digital Michelangelo Project: 3D Scanning of Large Statues, Siggraph 2000. ( assigned to Mihail )
Stamos, Ioannis and Peter K. Allen, Automatic registration of 2-D with 3-D imagery in urban environments, ICCV 2001, Vancouver. ( assigned to Jean-Francois )
Brian Curless and Marc Levoy, A Volumetric Method for Building Complex Models from Range Images Proc. SIGGRAPH '96. ( assigned to Jeremy )

PART II: Appearance modeling in Vision and Graphics

Class 20: Introduction + Overview : 03/24/2005
- F. E. Nicodemus, J.C. Richmond and J.J. Hsia, Geometrical Considerations and Nomenclature for Reflectance, Institute of Basic Standards, National Bureau of Standards, October 1977
- B.K.P. Horn, Robot Vision

Class 21: BRDF Modeling : 03/31/2005

M. Oren and S. Nayar, Generalization of the lambertian model and its implications for machine vision,, IJCV, Vol 14, pp 227--251 1996 (assigned to Sanjeev)
M. Oren and S. Nayar, Generalization of Lambert's Reflectance Model, SIGGRAPH 94, pp 239-246
K. Torrance and E. Sparrow, 1967. Theory for Off-Specular Reflection from Roughened Surfaces. Journal of the Optical Society of America, volume 57, number 9, pp 1105-1114. (assigned to Sanjeev)
J.J. Koenderink and A. J. van Doorn. Phenomenological Description of bidirectional surface reflection Journal of the Optical Society of America, volume 15, number 11, pp 2903-2912
K. Dana, B. Ginneken, S. Nayar and J. Koenderink. Reflectance and Texture of Real World Surfaces. TOG vol. 18, no. 1 pp 1-34.
E.P. Lafortune, S.C. Foo, K.E. Torrance and D.P. Greenberg. Non-Linear Approximation of Reflectance Functions SIGGRAPH 97, pp 117-126.
Initial versions of your survey papers are due. The goal is to get you to refine it after getting feedback from the instructor as well as other students.

Class 22: Inverse Rendering : 04/05/2005

R. Ramamoorthi and P. Hanrahan A Signal-Processing Framework for Inverse Rendering SIGGRAPH 01, pp 117-128.
S. Marschner, S. Westin, E. Lafortune, K. Torrance and D. Greenberg. Image-based BRDF Measurement Including Human Skin Eurographics Workshop on Rendering 2000, pp 139-152. (assigned to Mihail)
Y. Sato, M. Wheeler and K. Ikeuchi. Object shape and reflectance modeling from observation SIGGRAPH 97, pp 379-387.
Y. Yu, P. Debevec, J. Malik and T. Hawkins. Inverse global illumination: recovering reflectance models of real scenes from photographs SIGGRAPH 99, pp 215-224. (assigned to Jean-Francois)
S. Boivin and A. Gagalowicz. Image-based rendering of diffuse, specular and glossy surfaces from a single image SIGGRAPH 01, pp 107-116.

Class 23 : Special Lecture : 04/08/2005

Special Lecture and RI seminar by Ravi Ramamoorthi.

Classes 24 and 25: Image based Modeling and Rendering : 04/07/2005, 04/12/2005

P. Debevec et al. Acquiring the reflectance field of a human face . SIGGRAPH 00, pp 145-156. ( assigned to Marek )
M. Koudelka, S. Magda, P. Belhumeur and D. Kriegman Image-based Modeling and Rendering of Surfaces with Arbitrary BRDFs . CVPR 01, pp 568-575
D. Zongker, D. Werner, B. Curless and D. Salesin Environment Matting and Compositing . SIGGRAPH 99, pp 205-214. (assigned to Dinesh)
S. Chen and L. Williams View Interpolation for Image Synthesis . SIGGRAPH 93, pp 279-288. ( assigned to Ankur )
L. McMillan Plenoptic Modeling: An Image-Based Rendering System . SIGGRAPH 95, pp 39-46.
R. Szeliski, Stereo algorithms and representations for image-based rendering, British Machine Vision Conference (BMC'99), Volume 2, pg. 314--328, England, Sept 1999.
Andrew Fitzgibbon, Yonatan Wexler and Andrew Zisserman, Image-based rendering using image-based priors, ICCV 2003. (assigned to Philipp)

Classes 26 and 27: Illumination Modeling : 04/14/2005, 04/19/2005

R. Epstein, P. Hallinan, A. Yuille 5 +/- 2 eigenimages suffice: an empirical investigation of low-dimensional lighting models IEEE workshop on physics-based modeling in computer vision, pp 108-116, 1995.
P. Belhumeur and D. Kriegman What is the space of images under all possible lighting conditions? IJCV 28(3) pp 245-260, 1998.
T. Zickler, P. Belhumeur and D. Kriegman Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction ECCV 2002. ( assigned to Norm )
Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeur, A Theory of Multiplexed Illumination, ICCV, October 2003.

Classes 28 and 29: Scattering in Computer Vision and Computer Graphics : 04/21/2005, 04/26/2005 (Lectures)

S. Chandrasekhar, Radiative Transfer, Dover Publications, 1960 (Lecture)
S. G. Narasimhan and S. K. Nayar, Vision and the Atmosphere, IJCV 2001 (Lecture)
P. Hanrahan and W. Krueger. Reflection from layered surfaces due to subsurface scattering SIGGRAPH 93, pp 165-174. (assigned to Tom)
H. Jensen, S. Marschner, M. Levoy and P. Hanrahan A Practical model for subsurface light transport SIGGRAPH 01, pp 511-518. (assigned to Ethan)