I will discuss my research on developing technologies to enable machines to effectively interact with and sense the physical and social world. While Artificial Intelligence has succeeded in creating machines that exceed the performance of even the best humans in symbolic processing tasks (e.g. Chess and Jeopardy), progress on achieving human-level perceptual and motor abilities has lagged considerably behind. I will present three projects that are each aimed at either creating computational methods for extending the perceptual and motor capabilities of machines, or applying these methods to create machines that effectively interact with people. In the first part of the talk, I will discuss my work on the RUBI project in which we put together a robot that served as a teaching assistant in a San Diego preschool. RUBI combined touchscreen-based games with machine perception to interact with the children and teach them new words. Next, I will present work on utilizing online crowdsourcing services, such as Amazon’s Mechanical Turk, to address a daunting challenge for machine learning: obtaining large quantities of high-quality labeled data from which to learn. I will present a method of automatic quality control of human judgments obtained from such services that is able to automatically infer which labelers are more reliable and optimally combine their judgments to obtain an accurate consensus label for each piece of data. Lastly, I will present my work on using machine learning approaches to infer the goal of an agent from examples of its behavior. I will show how these new algorithms can be used as analytical tools for answering questions in the study of natural behavior as well as for interpreting the intentions of and imitating the behavior of humans.