15-869 Project
Revision 3, 1 Nov. 1999
The goal of the project is to give you practice at doing research and development in
the topic of the course, image-based modeling and rendering.
Steps:
- do literature search on related previous work,
- implement and test,
- give a presentation in class, and
- write up a description of your work, 8-12 pages in length, and create a web page that
summarizes your approach and showcases your results. The paper should in the form of
a conference paper, like the ones we have covered in class. Be sure to compare your
results to previous work and include references to at least three related papers.
The "implement and test" of phase (2) above could be application of existing
algorithms to a new problem; a variation on an existing algorithm; or design,
implementation, and testing of a new algorithm.
Schedule
- Meet with us to discuss your project idea: 27 or 28 Oct.
- Email a one-page proposal of what you plan to do by Fri,
29 Oct. to Steve AND Paul.
Try to narrow the scope so it's do-able in a month. Try to skim some of the relevant
references before you write up this proposal (If you can't find a paper, ask us; we might
have a copy.)
- Give a 3 minute progress report in class 15 Nov.
For this progress report, create a web page (no more than
the equivalent of two informative slides)
that you can show during your presentation.
Put web page at
classdir/pub/results/yourname/project/progress.html
before class that day.
Your progress report presentation should touch on:
(1) problem being solved,
(2) previous work you found,
(3) algorithm you're using,
(4) what's novel,
(5) how you plan to test your method (inputs & outputs).
Include only the most important information.
- Give a 20-minute presentation 29 Nov. or 1 Dec.
Plan on 15 minutes of talking, 5 minutes of questions.
- Submit written report: 3 Dec. and
put web page at
classdir/pub/results/yourname/project/index.html .
Project Topic Ideas
The following are some possible project topics. The list is not exclusive! You can also
modify an idea listed here, or propose something totally new. We've listed references that
I know of, but the list is incomplete, so you should plan on doing your own literature
search via web, journals, conference proceedings, and books.
- Quick scene reconstruction: Build a very quick user interface on top of
Criminisi's approach to permit a user to construct a simple model of a 3-D scene in a
matter of seconds, working from a single image. This project is best suited to someone
with a knack for user interfaces. Ideas for speeding up the interaction: (1) use
two-handed input (key press to change modes, mouse to point), (2) use edge-finding
algorithms to refine approximate feature locations chosen by user, (3) model with high
level primitives (blocks, pyramids), not triangles or rectangles, (4) optimize the code
for texture resampling.
- Shape from light fields: Input: a 4D light field.
Output: a 3D shape and reflectance model. You may wish to choose to constrain
the problem in certain ways (e.g., diffuse surfaces or known light sources or no
transparency, etc.) to make it more tractable. There is some light field data online
from Stanford.
- Model Newell-Simon Hall: Create complete 3D model of interior and
exterior.
-
Reflectance from video:
Shoot video of a road with a (steady) camera on a car.
Extract a model of the spatially varying reflectance as well
as shape (position and normal vectors) of the road.
To simplify things, you could assume the height of the camera above
the road is known, you could acquire the lighting environment
either by noting the time of day and compass heading
(to get sun position) or shoot at night and include light poles in
your video frames, and you could assume that the road is
approximately planar as an initial approximation.
To complicate things: shoot a wet road.
- Modeling from time lapse video: Working
from video of
Newell-Simon Hall shot from the same viewpoint but at different
times of day over many months, reconstruct shape and reflectance of
the building exterior. The user could assist in creating the
shape model by tapping on the corners of planar patches (providing a
coarse 2-D segmentation only) and your
system could solve for the orientations and reflectance parameters
using the images and estimates of the sun position for each
image.
See
Seitz's slides
on 3D Photography for quick overview and references on related
work on shape-from-shading and photometric stereo.
To test your method on other scenes,
you could shoot your own video or find outdoor cameras or time lapse
video on the web.
Some web cams:
Perceptual Robotics ,
Online Camera ,
Sondrio, Italy ,
Vienna ,
Florence ,
Moscow ,
San Francisco .
- Interactive registration for image mosaicing. Design a program that uses a
combination of user interaction and image processing to create an image mosaic. The user
specifies the registration approximately (within a few pixels) and the computer refines
the registration to sub-pixel accuracy in real time. Build many-image mosaics this way. To
the user, the warped image should appear to "snap" to the correct registration
when they get close to the minimum, analogous to snap-to-grid features of illustration
programs such as Adobe Illustrator. To achieve this, the minimization should run at 30Hz
or faster. The user assistance permits the search to be local and fast. Use of Shum's
3-parameter camera model will also help the optimization. Hardware texture mapping (a
high-end SGI machine or good PC graphics card, say) will also accelerate things.
References: See particularly Shum97, Pele97. Also Szel97, whose video shows the
optimization as taking about two seconds, but that's too slow! And Shum98. Szeliski
or Szeliski
and Coughlan use a global search approach (probably overkill); Scho83 uses a faster,
more local approach.
- Vision-assisted morph correspondence. This is a variation on the previous topic,
tailored to the specification of morph correspondences, which are scene-dependent, unlike
projective image warps. Design a vision-assisted morphing program that uses a combination
of image processing and user-interaction to specify the correspondence between two or more
images for morphing. For instance, the user could specify a feature in one image (a line,
curve, or image patch) and pull it towards its desired position in the second image. As
s/he gets close, the system "snaps" to the best match in the second image, by
performing a local search over the second image, and displays the resulting morph.
References: Beie92, Wolb90, other references above.
- Texture synthesis for hole filling. As we found in assignment 2, simple
hole-filling schemes leave much to be desired. Implement a technique for synthesizing
texture patterns by training on one or more images of natural textures (e.g., of pebbles,
wood grain, leaves, etc.). See work by Efros and Leung
and De Bonet for
examples of texture synthesis techniques. Explore how to use this to fill holes caused by
view-interpolation and other IBR techniques.
- Editing depth images: Depth images (RGB images with an extra depth "Z"
channel) provide a very concise representation of 3D shape and color and are easily
acquired from range scanners. Design a program like Photoshop for editing depth images.
You should include tools for painting new color and/or depth values (the depth channel
could be represented as a gray-scale image), cut/paste, warping, filtering, and other
effects. You should integrate your renderer from assignment 2 to freely change camera
viewpoint during the editing process.
References: Wil90, Hanr90.
- Multi-view metrology: Extend Criminisi's single view metrology technique to
integrate measurements from multiple input images, incorporating a way to specify
constraints. This could yield more accurate and more complete models. For example, you
should be able to specify: (1) a feature observed in two or more images has the same 3D
coordinates, (2) two line segments in an image have the same height in 3D.
- Image-based trees and plants: Design a system for modeling trees and other fauna
from images. The input would be one or more images of a tree (possibly including a
close-up of an individual leaf). The output would be images of many similar trees, or the
same tree under different viewing conditions. You may wish to include an interface to help
a user specify the branch structure of the tree (possibly different from the structure in
the photograph). Additional features: model different illuminations, change of seasons,
blowing wind. You may choose to do the modeling in 2D or 3D and may choose to use a
biologically based technique like L-systems (for a nice online tutorial by Przemyslaw Prusinkiewicz,
click here).
References: Prze94, Max95.
- Stereo panoramas: Create a panoramic stereo pair by waving a hand-held video
camera and stitching together strips from the resulting video stream. In particular,
suppose you captured 500 images, extracted the center column of pixels from each image,
and laid these out side-by-side to form a new 500 column mosaic image. Now do the
same thing, but create two mosaic images from two different off-center columns.
Composite these two images together as the red and blue channels of a new image and VOILA,
you have a stereo panorama, viewable in 3D using red-blue glasses. Your stitching
program should run at near-real time. Experiment with choosing different columns of
the images to fit the eye separation of different users. See the paper by Shmuel Peleg. Come up with interesting
extensions and enhancements to constitute a research project.
- Choose your own!!! Don't feel limited by the suggestions listed
here--these are just some ideas we've thought of.
References
- [Beie92]
- Thaddeus Beier and Shawn Neely, Feature-based Image Metamorphosis, Computer Graphics
(SIGGRAPH '92 Proceedings), 26 (2), July 1992, pp. 35-42
- [Hanr90]
- Pat Hanrahan and Paul Haeberli, Direct WYSIWYG painting and texturing on 3D shapes,
SIGGRAPH '90, pp. 215-223.
- [Max95]
- Max, Nelson, and Keiichi Ohsaki, Rendering
Trees from Precomputed Z-Buffer Views, 6th Eurographics Workshop on Rendering, June
1995.
- [Pele97]
- S. Peleg and J. Herman, Panoramic
Mosaicing with VideoBrush, DARPA Image Understanding Workshop, May 1997.
- [Pele97]
- S. Peleg and M. Ben-Ezra, Stereo Panorama
with a Single Camera. CVPR'99, pp. 395-401.
- [Prus94]
- P. Prusinkiewicz, M.
James, and R. Mech, Synthetic topiary, SIGGRAPH '94, pp. 351-358.
- [Scho83]
- Schowengerdt, Techniques for Image Processing and Classification in Remote Sensing 1983,
p. 101.
- [Shum97]
- H.-Y. Shum and R. Szeliski,
Panoramic image mosaicing. Technical Report MSR-TR-97-23, Microsoft Research, September
1997.
- [Shum98]
- H.-Y. Shum and R. Szeliski. Construction and refinement of panoramic mosaics with global
and local alignment. In Sixth International Conference on Computer Vision (ICCV'98), pages
953-958, Bombay, January 1998
- [Szel97]
- R. Szeliski and H.-Y. Shum. Creating full view panoramic image mosaics and
texture-mapped models. Computer Graphics (SIGGRAPH'97), pages 251-258, August 1997. (be
sure to see the video for this; Paul & Steve have copies)
- [Will90]
- Lance Williams, 3D Paint, Proceedings of the Symposium on Interactive 3D Graphics
Computer Graphics 24(2), 1990
- [Wolb90]
- George Wolberg, Digital Image Warping, IEEE Computer Society Press, 1990
Resources
Bibliographies and Online Paper Collections
15-869, Image-Based Modeling and Rendering
Steve Seitz and
Paul Heckbert
This file is http://www.cs.cmu.edu/~ph/869/www/project/project.html