Sketch Recognition Research @ HMC

I am currently working on two projects related to sketch recognition. One is a long-term project to build and evaluate a sketch-based tool for digital circuit education. The other is a new project that involves building a sketch recognition system for military commanders to support planning.

Summer Research Opportunities, 2008

I am looking for 4-8 students to work with me this summer. Here I give some information about the projects and what I am looking for. You can read more about the projects below.

Project 1: Smarter Educational Software for Sketch recognition (4-6 positions)
Research on this project began in the summer of 2006 and will continue for the next several years. I have funding to hire students in the summer and I am happy to work with students in independent study projects during the year.

The work done in the summer of 2006 developed core recognition algorithms and a code base. In 2007 we improved those algorithms and build an initial prototype system. This summer we will continue to improve recognition algorithms and explore user interface issues. The goal is to have a fully functional prototype, ready for use in the classroom, by the start of fall 2008.

I also have one project specifically geared toward first-year women students. The goal of this project is to critically analyze different people's sketching styles under different conditions. This is important and exciting research that will have a big impact on how sketch recognition systems are built. You do not need much experience with computer science (you should be able to get by just fine with CS5), but a working knowledge of probability and statistics will help a lot. I would like to hire one or two students for this project.

Project 2: Recognizing Military Course of Action Diagrams (2 to 4 positions)
The goal of this project will be to recognize freely-drawn military course of action diagrams to support military commanders during battle planning. This project will focus mainly on sketch recognition algorithms, but may also include a user interface component. The unique and exciting part of this project is that we aim to achieve a working system with high recognition accuracy (>90%) on freely drawn data in only one year. Students will work in teams of two on narrowly defined problems to try to achieve the highest accuracy possible in the shortest time possible.

This work will be done in close collaboration with two of my colleagues: Tracy Hammond at Texas A&M University and Metin Sezgin at the University of Cambridge.

Required Background

For both projects, I am looking for motivated students who are as excited about this research as I am. It is not necessary for students to have any knowledge of sketch recognition or user interface design, but basic knowledge of AI is preferred. Students should have experience developing large software systems (at least CS 70, LSD preferred). For the first project, knowledge of digital design and computer architecture is a plus.

If you are interested, please apply through the CS department application procedure or come talk to me about an independent study. Feel free to contact me (alvarado@cs.hmc.edu) if you have any questions.

I also work to make positions available for first year women students. If you are a first year woman student who is interested in doing research with me (even if you are not sure you want to be a CS major), please apply even if you don't have the required background.

Project 1: Smarter Educational Software Through Sketch Recognition

Christine Alvarado and Sarah Harris
Harvey Mudd College

Project Summary

The central goal of this research is to construct and deploy computer simulation tools capable of understanding students' hand-drawn diagrams. The results of free-sketch recognition research cannot yet be incorporated into end-user applications for two reasons. First, free-sketch recognition is not sufficiently robust to incorporate into useful tools. Second, little is known about how to build usable interfaces that incorporate free-sketch recognition. This work will bridge the gap between free-sketch recognition technology and its end-users by focusing on its application to undergraduate engineering design. The outcome of this work will be improved techniques for free-sketch recognition, guidelines for incorporating free-sketch recognition into usable interfaces, and educational sketch-based simulation tools.

Software and Data for Download

A central goal of our project is to develop tools that others will find useful. Of course, our long term goal is to provide a sketch-based simulation program for digital circuit design. Eventually that tool will be made available. However, in the mean time, we are developing a number of tools that we believe people in the sketch recognition community will find useful.

We are also working to make several datasets available for developing and testing. These will be available in early summer 2007.

You can download our software and get more information from our download page.

Acknowledgments

This work is funded by an NSF CAREER award, under Grant Number IIS-0546809.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Project 2: Recognizing Military Course of Action Diagrams

The goal of this project is to robustly recognize military course of action symbols as drawn by commanders in the planning process. Although this project may seem like "just another sketch recognition domain" it is unique in that these algorithms need to achieve high accuracy rates (>90%) without placing any constraints on the way users draw their symbols. Achieving this goal will require using and developing of cutting edge recognition algorithms, tailored specifically for the sketching style of users in this domain.

This project is funded by DARPA through the Deep Green initiative. We are working closely with Tracy Hammond at Texas A&M University and Metin Sezgin at the University of Cambridge to develop these sketch recognition algorithms.

HMC Sketchers People

PI: Christine Alvarado, HMC Computer Science
co-PI: Sarah Harris, HMC Engineering

Current Students

Eric Doi: HMC '09, Computer Science
Jason Fennell: HMC '08, Joint Math/Computer Science
Joe Simons: HMC '08, Computer Science
Alice Zhu: HMC '10, Computer Science

Former Students

Chris Acon: HMC '07, Engineering
Ned Burns: Pomona '08, Computer Science
Howard Chen: HMC '07, Engineering
Andrew Danowitz: HMC '08, Engineering
Laurel Fullerton: HMC '07, Engineering
Sam Gordon: HMC '08, Engineering
Ellen Kephart, HMC '08, Computer Science
Michael Lazzereschi: Pomona '06, Computer Science
Max Pfluger: HMC '07, Engineering
Devin Smith: HMC '09, Computer Science
Mike Roberts: HMC '08, Computer Science
Raquel Robinson: HMC '10, Engineering
Paul Wais: HMC '07, Computer Sceince
Matt Weiner: HMC '08, Engineering
Aaron Wolin: HMC '07, Computer Science