Software framework turns small robots into controllers

ATLAS researchers have developed a software framework that enables developers who lack experience programming robots to repurpose off-the-shelf toy and educational robotic devices to serve as input-output devices for desktop applications.

Dan Szafir, assistant professor of computer science in the ATLAS Institute, Shaun Kane, assistant professor of computer science and ATLAS faculty fellow, and doctoral student Darren Guinness, presented the GUI (graphical user interfaces) robots framework at the Designing Interactive Systems (DIS 2017) conference in Edinburgh this summer.

The GUI Robots toolkit enables developers to transform low cost robots (often less than $100 USD) into wireless controllers for desktop applications. The toolkit supports connecting such devices with a wide range of software applications, including web browsers, 3D modeling tools and video games, extending the application user interfaces into the physical world in interesting new ways.   

With most software applications confined to screen-based graphical user interfaces, tactile input and haptic feedback are the next major frontier for interaction design.

Szafir says, “developers can use our framework to quickly prototype tangible interactions and attach them to existing applications. As consumer-oriented, wirelessly-connected robots become ubiquitous, our work can enable new user experiences in which an ecosystem of helpful robots extends traditional graphical user interface applications.”            

Similar to a Nintendo Wii console where the controller detects movements in three dimensions, providing realistic on-screen action games, users can repurpose devices to apply the same type of movement in any existing application. For instance, researchers configured a Sphero Ollie to be a controller for Rovio Entertainment’s popular Angry Birds game so that rolling the robot back and then moving it forward launched a bird.  

Robots can also be programmed to provide physical feedback, such as vibrating when a bird is launched. Or, a user could manipulate a 3D object on the screen by moving and rotating the robot controller in the air.

To test the GUI Robots toolkit, researchers asked twelve developers to build controllers for two applications: Angry Birds and Windows Movie Maker.  All of the developers were able to build working prototypes of the Angry Birds controller within a half hour; the Movie Maker controller took them a little longer.

As an extension to their work, the team is exploring the use of GUI Robots to provide haptic displays for visually impaired users, so that movement or tactile feedback from a robot could represent visual information such as charts, diagrams, movies or even interactive simulations. Other promising uses for their software are educational software, controllers for musical instruments and 3D modeling.

Read the research