|
|
|
|
Research
Ph.D. Thesis: Large Scale Scene Matching for Graphics and
Vision
Thesis Page
Our visual experience is extraordinarily varied and complex. The diversity
of the visual world makes it difficult for computer vision to understand
images and for computer graphics to synthesize visual content. But for all
its richness, it turns out that the space of "scenes" might not be
astronomically large. With access to imagery on an Internet scale,
regularities start to emerge - for most images, there exist numerous
examples of semantically and structurally similar scenes. Is it possible
to sample the space of scenes so densely that one can use similar scenes
to "brute force" otherwise difficult image understanding and manipulation
tasks? This thesis is focused on exploiting and refining large scale scene
matching to short circuit the typical computer vision and graphics
pipelines for image understanding and manipulation.
|
Image Sequence Geolocation with Human Travel Priors
Evangelos Kalogerakis, Olga Vesselova, James Hays, Alexei A. Efros, and Aaron Hertzmann.
IEEE International Conference on Computer Vision (ICCV '09)
Project Page
|
An empirical study of Context in Object Detection
Santosh Divvala, Derek Hoiem, James Hays, Alexei A. Efros, and Martial Hebert.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2009.
Project Page, Paper
|
IM2GPS: estimating geographic information from a single image
James Hays and Alexei Efros.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2008.
Project Page,
Paper,
Bibtex
Google Tech Talk.
Abstract:
Estimating geographic information from an image is an excellent, difficult high-level computer vision problem whose
time has come. The emergence of vast amounts of geographically-calibrated image data is a great reason for computer
vision to start looking globally . on the scale of the entire planet! In this paper, we propose a simple algorithm for
estimating a distribution over geographic locations from a single image using a purely data-driven scene matching
approach. For this task, we will leverage a dataset of over 6 million GPS-tagged images from the Internet. We represent
the estimated image location as a probability distribution over the Earth's surface. We quantitatively evaluate our
approach in several geolocation tasks and demonstrate encouraging performance (up to 30 times better than chance). We
show that geolocation estimates can provide the basis for numerous other image understanding tasks such as population
density estimation, land cover estimation or urban/rural classification.
|
Scene Completion Using Millions of Photographs
James Hays and Alexei Efros.
Transactions on Graphics (SIGGRAPH 2007). August 2007, vol. 26, No. 3.
Project Page,
SIGGRAPH Paper,
CACM Paper,
CACM Technical Perspective by Marc Levoy,
Bibtex
Abstract: What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of image completions and we allow users to select among them. We demonstrate the superiority of our algorithm over existing image completion approaches.
|
Interactive Tensor Field Design and Visualization on Surfaces
Eugene Zhang, James Hays,
and Greg Turk.
IEEE Transaction on Visualization and Computer Graphics, 2007, Vol 13(1), pp 94-107.
Project Page,
Paper, Bibtex
This research project was primarily Eugene's work and I played only a small role.
Abstract: Designing tensor fields in the plane and on surfaces is a necessary task in many graphics applications,
such as painterly rendering, pen-and-ink sketch of smooth surfaces, and anisotropic remeshing. In this
paper, we present an interactive design system that allows a user to create a wide variety of surface
tensor fields with control over the number and location of degenerate points. Our system combines basis
tensor fields to make an initial tensor field that satisfies a set of userspecifications. However, such
a field often contains unwanted degenerate points that cannot always be eliminated due to topological
constraints of the underlying surface. To reduce the artifacts caused by these degenerate points, our
system allows the user to move a degenerate point or to cancel a pair of degenerate points that have
opposite tensor indices.
|
Image De-fencing
Yanxi Liu,
Tamara Belkina,
James Hays,
and Roberto Lublinerman.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2008.
Project Page,
Paper, Bibtex
We introduce a novel image segmentation algorithm that uses translational symmetry as the
primary foreground/background separation cue. We investigate the process of identifying and
analyzing image regions that present approximate translational symmetry for the purpose of
image fourground/background separation. In conjunction with texture-based inpainting,
understanding the different see-through layers allows us to perform powerful image
manipulations such as recovering a mesh-occluded background (as much as 53% occluded area)
to achieve the effect of image and photo de-fencing.
|
Discovering Texture Regularity as a Higher-Order Correspondence Problem
James Hays, Marius Leordeanu,
Alexei Efros,
and Yanxi Liu.
European Conference on Computer Vision (ECCV) 2006.
Paper, Bibtex
We find arbitrarily distorted regular patterns in real images by treating lattice-finding as a higher-order
assignment problem. We leverage previous work
from Marius Leordeanu and Martial Hebert to approximate the optimal assignment under second-order constraints.
Source code available upon request!
|
Quantitative Evaluation of Near Regular Texture Synthesis Algorithms
Steve Lin,
James Hays, Chenyu Wu,
Vivek Kwatra,
and Yanxi Liu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2006
Paper, Bibtex
Quantitative evaluation is difficult for texture synthesis. Ground truth is not well defined.
But for certain textures you can objectively decide whether an algorithm has failed or not.
Regular and near-regular textures imply a definite structure that should be preserved.
We tested several popular algorithms on a large group of structured textures.
In addition to the CVPR 2006 paper, a more detailed technical report is available.
|
Near-Regular Texture Database - link
Online Database
We created a database of regular and near-regular textures for other researchers to use.
You can submit your own textures, as well, and help the database grow.
|
Digital Papercutting
Yanxi Liu, James Hays,
Ying-Qing Xu,
and Harry Shum
SIGGRAPH 2005 Sketch
Sketch, Bibtex
Papercutting is a widespread and ancient artform which, as far as we could tell, had no previous computational treatment. We developed algorithms to analyze the symmetry of papercut patterns and produce efficient folding and cutting plans.
|
Near-Regular Texture Analysis and Manipulation
Yanxi Liu,
Steve Lin, and James Hays.
SIGGRAPH 2004
Project page, Paper, Bibtex
Abstract: A near-regular texture deviates geometrically and photometrically from a regular congruent tiling. Although near-regular textures are ubiquitous in the man-made and natural world, they present computational challenges for state of the art texture analysis and synthesis algorithms. Using regular tiling as our anchor point, and with user-assisted lattice extraction, we can explicitly model the deformation of a near-regular texture with respect to geometry, lighting and color. We treat a deformation field both as a function that acts on a texture and as a texture that is acted upon, and develop a multi-modal framework where each deformation field is subject to analysis, synthesis and manipulation. Using this formalization, we are able to construct simple parametric models to faithfully synthesize the appearance of a near-regular texture and purposefully control its regularity.
See the project page for the paper and movies.
You know, they say grad school is like prison, but you actually spend all day looking at brick walls. -- Marty, former office mate.
|
Motion Analogies, Fall 2003
Transferring a "marching" style to a novel walking motion.
This was my final project for Data Driven Character Animation.
Basically it's Image Analogies except the feature
vectors are built from motion capture data. One problem that has to be solved is creating correspondence between A and A'.
However, B does not need to be registered with A or Ap, so the learned transformation is somewhat general. The
coherence parameter turns out to be useful for Motion Analogies as it was for Image Analogies. There are
potential inverse kinematics and temporal coherency issues that I did not address in my work.
Check out this SIGGRAPH 2005 paper "
Style Translation for Human Motion" by researchers at MIT which uses the same type of example motions to "learn" and transfer styles.
|
Image and Video Based Painterly Animation
James Hays and Irfan Essa.
NPAR 2004.
Project Page, Paper, Bibtex
We extend previous non-photorealistic rendering work to handle video significantly better by temporally constraining brush stroke properties in addition to other improvements. Have a look at the paper.
|
CV
|
|
|
|
