In order to play the game of set, a player needs to match up three
cards so that for each of the 4 characteristics of the cards (shape,
color, number of objects, and shading) the cards are either all the
same, or all different. For a human player, the tricky part is
finding matches. For the computer, which can simply remember all the
attributes and run an exhaustive search rather quickly, the tricky
part is figuring out the attributes of the cards.
Color is fairly easy to determine for a picture of the card. The code simply adds up the number of pixels which register as each color, and then decides the color of the card based on that information. While there was a certain amount of parameter tuning because the lighting makes white turn up green, and blue is too close to black to be easily detected, this wasn't very difficult.
In order to determine the number of objects on the card, the code checks each column of the picture to see if it is empty or contains some colored pixels. Then it partitions the picture into blank sections and colored sections, and determines the number of objects from the number of sections. This can break easily if a color isn't detected properly, like blue.
To determine the shading of a card, we look at the number of colored pixels on a line inside the object (the smaller white line in pictures). If almost all the pixels are colored, the object is filled in. If none are colored, the object is empty, and if some are colored, the object is shaded.
Shape is the nastiest attribute to detect. Fortunately, we know from determining the number of objects where one of the shapes starts. When we draw a line a couple pixels further into the shape, we can determine which shape it is. If the color only appears for a few pixels in the center of the line, the shape is a diamond. If the color appears for most of the line, then the shape is an oval. If there are gaps, or the color only appears for a little bit at the top or bottom of the line, the shape is a squiggle.
And here are some screenshots of our code at work.
It got that one right.
And this one. Life is easy when you have a limited number of pictures the code must work for.
And it actually plays Set! properly now, although for a while there were some strange errors.
Although it can't detect matches if there aren't any there.