Harvey Mudd College
Computer Science 153
Assignment 5
Due Monday, December 11, by midnight

Back to Assignment 5, top-level page

Back to Assignment 5, Section 1: Eigenfaces


Section 2: image transfer

2d image transfer

Using at least four correspondences between two images of planar objects, it is possible to define the projective relationship between those two images. (More correspondences lead to better accuracy.) As a result, information from one image can be used to "fill in" regions that may be missing (or simply out of view) in the other image.

There are images in the directory 
/cs/cs153/Images/a5/2dtransfer
named store1.jpg, store2.jpg, and store3.jpg that are of the same (nearly planar) object. The store1.jpg image has a considerable specularity within a rectangle with an upper-left corner of 
  (x,y) = (363,501)
and a lower-right corner of 
  (x,y) = (544,687)
The goal of this section of the assignment is to fill in this missing information by using the information from either the store2.jpg or store3.jpg image (or both). You may want to check the notes from lectures 19-21 (they're posted here) in order to set up the equations you'll need to solve to determine the relationship between any two of the images.

The intent of this problem is largely to get comfortable with the linear algebra used in modeling image transformations in homogeneous coordinates. Thus, you should feel free to extract and set correspondences by hand.

Caution! These images are very large. While you might want to use these large images to actually obtain a detailed reconstruction of the missing part of store1.jpg, when including them here, perhaps use a half- or quarter-resolution image.

Results:

3d image transfer

Ultimately, the goal of vision is to extract information about the three-dimensional world from two-dimensional images. If the object being imaged is known (i.e., its feature points are in a known geometric configuration), it is possible from a single image to determine where additional 3d points would be located.

In the directory 

/cs/cs153/Images/a5/3dtransfer
there are several picures of a rubik's cube. Start with the image rcube1.jpg . Given that the cube is 3 units of length to a side, find the transformation from the cube's coordinate system to the image's pixel coordinate system using a number of (hand-chosen) point correspondences. In order to demonstrate the accuracy of your transformation, plot with a distinctly colored cross the points that are at the center of each little square's face on the rubik's cube. (Of course, you should do this not by actually finding those centers by hand (they're hard to determine exactly in any case), but by projecting their 3d cordinates into the image with the transformation you found using other points on the cube.

Extension possibilities

As one possible extension (unnecessary as this is extra credit, but in any case...), you might try to use the information in rcube1.jpg to fill in the missing portion of the image rcube2miss.jpg. There are a number of ways you could approach this, but it's trickier than in the transfer of 2d information, since the object is no longer planar. The "correct" solution (the original image) is rcube2.jpg. There is also another image, rcube3.jpg, available in the same directory.

Another direction for investigation is into the automatic detection of corresponding points in two images. Using the two store images store2.jpg and store3.jpg, develop a strategy that searches for matching points in the two images. In this case, do not assume that you know the relationship between the two. Develop a measure for the "strength" or "confidence" in a particular match, and using the ten most confident pairs, compute the transformation between the two images. Compare the results to the hand-picked points you used in section 1, above.

Results: