Harvey Mudd College
Computer Science 153
Problem Set 3
Due Friday, October 14, 5pm

This part based on an assignment by Z. Dodds.

Back to Problem Set 3 top-level page

Part 1: Object Modeling and Classification

In this section you will model and classify objects based on their color characteristics (and any other properties you wish to include). You will test your recognizer on images collected by the class, and then we will have a recognizer face off in class on Wednesday, October 19 on a new set of images, to be collected as part of this assignment.

Objects to Model

In the directory /cs/cs153/Images05/ModeledObjects

There are a number of objects for you to model. You must develop a model for 10 of the 13 objects in this directory. You will be developing modeling code that will (ideally) work to generate a model of your 10 objects.

In the above directory, in addition to the object images, there is also a .txt file for each image containing information about that image. (This is the .txt file you generated in problem set 2). This file contains, in the following order, each on their own line:

the filename of the image.
a single-word unique identifier for the dominant object.
Keywords = {inside, outside, people, animal, plant, manufactured, single-object, highlight, building}.
the bounding box of the dominant "object" in the image
Comments from the photographer

Developing an object model

For this assignment you will be generating an object model based on only a few examples of an object, so a statistical classifier will not work because we do not have enough data to get a good approximation of the class conditional density (although one possible extension is to gather more data to estimate the class conditional density for a few objects). So essentially our object "models" will represent one point in feature space that is (ideally) close to the center of where the given class of objects lie in that space.

Segmenting the object from the background: As we learned, segmentation is hard. Luckily, you know where the dominant object is in the image! Your first task will be to write the matlab code that extracts the dominant object from the rest of the scene using the coordinates supplied in the corresponding .txt file. Then you can focus on only the relevant object for the rest of the modeling process. (You can do this by hand if you want, but it'll be easier if you write a program to do it for you.)

Modeling with Color: We will start by using color as the feature in the object model. You will have the option of adding other features as an extension. The most natural way to model color is with a color histogram (just like you did when you segmented images using color). To generate a color histogram, you will need to decide two things: which color space you would like to work with (RGB or HSV or something else) and how you will divide up your bins. Once you decide on a colorspace and a strategy for binning it, create and plot at least one histogram of an image using that division of colorspace; include the results (and a brief explanation) here. Keep in mind that you may not want to divide colorspace uniformly along its axes. Note that these color histograms are models of a single object, and not histogram models for the class conditional densities that we talked about in class!

Results:

Combining color histograms

Next, you'll need to decide exactly what will constitute the model of an object. This problem is based around histograms, yes, but there are a variety of ways to use histograms to model, say, a "parrot." The simplest would be to take a single parrot image and use its histogram as the model for parrots in general. Or, you may want to combine the information available from additional images -- for example, by averaging. Alternatively, there may be information you don't want to use in your model of an object: for example, you might considering a more careful hand-segmentation of the objects when creating your object model.

Ultimately, you should have a function or script that will create an object model, given an image/images or, if you prefer, a list of pixels (or several lists of pixels, coming from several images). Include your object-modeling code, along with a brief explanation and description, below.

Results:

Model Matching

The final piece to your system is a means for comparing two object models to determine how similar they are. For this you might use histogram intersection (higher match scores then indicate better matches), perhaps with a scale factor used to eliminate any effect from the image size of the objects matched against. A second approach to consider is a histogram distance function, e.g., use a Euclidean distance where you compute the sum of the squared difference between the bins. In this case, low distance (small match scores) would indicate close matches of models. Again, consider a scale factor to eliminate any dependence on absolute image size. One problem with histogram comparison is that there is no account for bin distance (i.e. adjacent bins should be considered "more similar" than far away bins). You might try modifying your comparison function to try to account for this.

Regardless of how you choose to match object models, you will want to try matching several to see how it performs. In addition to the ModeledObjects directory, there is also a sibling UnmodeledObjects directory you may want to use to test histogram comparison. Don't use the UnmodeledObjects, however, to build the object representations, as there needs to be some independent data for use in testing the final system. As you test, you may want to consider a criterion for a "reject class," i.e., a class for objects that do not match any of the 12 modeled ones well.

Putting it all together

The end result of all this work should be the following (to which you should provide links here):

A set of models, one for each of the 10 object classes stored in a text file (whose format you will determine).
Two functions:
```
function models = loadModels( filename );
```
that reads in a set of models and builds your matlab representation of these models, and
```
function class = classify( image, models );
```
that takes the models output by the previous function and an image (of just the dominant object, as defined by its bounding box) and outputs the object's class.
A results file (can just be an html page), with the following information:
- First, for each image in the directory
  /cs/cs153/Images/a3/UnmodeledObjects
  create a row in a three-column list with
  1. The name of the image file
  2. The name of the object that your system considers the closest match (or "reject")
  3. The match score
- Second, with all of the images from both the ModeledObjects and UnmodeledObjects directories, create a confusion matrix that indicates how many of each object were correctly classified and what misclassifications occurred. The rows of the confusion matrix should indicate the actual objects appearing in the images, and the columns of the confusion matrix should indicate the resulting classifications. Each entry in the confusion matrix indicates how many images of a particular object were classified in each category.
- Finally, in addition to these results, comment briefly on the strengths and shortcomings of your object recognition system. Include at least one misclassified image and how the system might be changed to prevent at least that type of misclassification.

Possible Extensions

Remember that you need to pursue only one extension in one part of the assignment, and it can be anything you'd like to investigate. These are only suggestions.

We didn't have enough images to model class conditional densities. As an extension, you could choose a few objects (not necessarily from the list above) take many more pictures of those objects, then create a class conditional density of these objects by fitting a Gaussian distribution to their feature values (you can think of a specific color histogram as a point in an N-dimensional space, where N is the number of bins in your histogram). Compare the classification results created by your new approach to those that you get by applying the single-example classifier you created above.
Consider more features than color in your object model. You might try adding a shape feature, or a texture feature (or set of features). Compare the results of adding more features to the color-only classifier.
Rather than seeking individual objects, consider adapting your system to distinguish between outdoor and indoor scenes (the keywords later in the .txt files offer this information. Another distinction to pursue might be natural vs. manmade objects. Create a system that classifies input images into one of two categories, based on the color properties of those images, or any other features you choose to use. For this extension, you may want to use whole-image histograms, rather than the histograms of only the prominent objects in those images.

On to Part 2: PCA

Harvey Mudd College Computer Science 153 Problem Set 3 Due Friday, October 14, 5pm