Gesture-Based Computing

Scenario

Dr. Grote teaches urban studies and planning. Today, students in his seminar “Designing a Neighborhood Renaissance” are working with an immersive environment that they control by gesture. The environment consists of a U-shaped space with projections on three walls. Students can “walk” through the environment by stepping in place and motioning with their hands. Using similar hand and body gestures, students can simulate the construction of new buildings, initiate urban cleanup activities, and track expenditures for city services on a projected spreadsheet.

Will is the first to “walk” through the neighborhood they are studying, and he quickly finds himself involved in the scenario as it unfolds. Will leans over and “pinches” to pick up a bottle in a run-down neighborhood playground. This causes the spreadsheet on the opposite wall to flag sanitation costs, asking if more money is to be spent on cleanup. 

At first, students vote against spending any money on services, materials, or health, so they can see the effects of an austere economy. As Will thumbs through a calendar on the wall (to simulate time passing), trash piles grow in the cityscape. He and other students who walk through the environment comment that they feel nervous and depressed. The next day, Rita, who is now navigating, leans down to pick up a flattened soccer ball. But there is no place to dispose of trash. “Why are there no bins here?” she asks, prompting a discussion about sanitation services.

Over the next several days, students make improvements. They build an 11-story youth center, but find it blocks light on the street. Margaret, who is gesturing on behalf of the class, “pulls down” the top three stories one at a time until sunlight hits the street at 10:00 AM at the spring equinox.

Their goal over the semester is to find innovative solutions that are cost effective and that still address the problems of congested urban living. Because of the gesture interface, they feel they are making real progress, for it allows them to try quick changes and measure the value and cost of these changes within the environment. As an added bonus, using gesture to navigate a 3D immersive environment has turned out to be more seamless and intuitive—and thus more involving.

1. What is it?

Gesture-based computing refers to interfaces where the human body interacts with digital resources without using common input devices, such as a keyboard, mouse, game controller, or voice-entry mechanism. Such systems can scan a limited space, analyze human movement within it, and interpret the motions of hands and arms, head, face, or body as input. Touchscreen interfaces likewise enable gestures such as swipes or taps to navigate, play games, or organize data. Gesture-recognition technology is used in a wide array of applications, from those that parse facial expressions to determine human emotional reactions to complex, augmented-reality simulations that evaluate whole-body movement.

2. How does it work?

Motion may be used to input navigation commands or it may serve as data. When it is employed for navigation without a touchpad, users might zoom by stepping forward and backward. Or they could control a video stream: pointing left to replay, presenting the flat of a hand to stop, or pointing right to continue. The approach is widely used in interacting with games and to control entertainment systems, machinery, or robotic devices and to interact with smartphones and tablets. Systems such as Microsoft Kinect and Leap Motion use camera/scanner technologies to capture body movements, which are interpreted by software. These gesture-recognition systems can capture multiple points of interaction simultaneously, making them valuable where captured motion might be analyzed for mastery, as in dance, athletic technique, or physical therapy.

3. Who’s doing it?

Developers at numerous universities are experimenting with gesture-based technologies. The Tangible Media Group at the MIT Media Lab recently presented the inFORM interface, featuring a digital display and a pinscreen (a bed of dull pins that adapt to the shape of a 3D object pressed against it). The pins are attached to motors that lift in response to gestures made by a remote operator. The result is an arresting sight, giving the effect of hands reaching out from the display to manipulate the pins. Gesture-based technologies are also seeing direct employment in teaching. Researchers from the Technological Educational Institute of Crete and from Aalborg University in Copenhagen have teamed up to build a gesture-based game for teaching emotion-recognition skills to preschoolers. The game helps them label and recognize emotions as shown in human faces and to identify the causes of feelings in social situations. At Ohio University’s Language Resource Center, students can interact with an immersive “holodeck” environment created by images projected on multiple surfaces. Because gestures control zooming, panning, and sorting through images, the students find themselves working physically within the projected setting.

4. Why is it significant?

The introduction of the mouse allowed computer inputs to move beyond the linear dimension of the keyboard to two dimensions. Gesture-based computing is the next step in that evolution, enabling 3D input that involves users in the computing activity. The result can be less like talking about a molecule or DNA chain and more like lifting one to examine it from every angle. Combined with simulation, gesture-based interfaces allow learners to see results of actions taken, as when they gesture to control the movement of robotic devices, converse with a system using sign language, or hone their skills in a speech-therapy session. Tabletop systems that employ surface gestures may give small-group discussion a boost. Unlike a keyboard or mouse, where a single individual controls input, a gesture-based system enables input from multiple participants.

5. What are the downsides?

Because this is an emerging technology, accuracy varies greatly. While some systems might respond to the twitch of an eyelid, others have difficulty discerning fine motor movement. In some cases, gesture-based interfaces can provide expanded or easier access to users with disabilities, such as those with limited fine-motor control, but in other situations the technology may present new obstacles for accessibility. Health concerns caused by everyday use, such as stress issues and repetitive-motion injuries, need further examination, although it might eventually turn out that gross movements are less likely to cause stress injuries than the more restricted movements used with keyboard and mouse. Systems that recognize and interpret gestures might strike some users as invasive, sparking concerns about privacy. Finally, an interface that determines what gestures mean may require a new vocabulary for interaction and could conflict with gestures some cultures interpret as inappropriate.

6. Where is it going?

Many of the possibilities of gesture-based technology have yet to be explored, but the whole-body approach to computer interaction suggests that these interfaces could enable a more active and intuitive learning style that often seems like play. One particularly promising area is the combination of gesture-based computing with augmented reality. With properly equipped goggles or other immersive technology, students might work at performing medical procedures without risk to virtual “patients.” Safety drills in augmented reality could help learners in fields that handle dangerous chemicals or explosives, allowing them to acquire familiarity with safe procedures and to test their troubleshooting abilities before emergencies arise. In civil engineering, an immersive 3D setting might allow learners to construct a bridge by gesture, as if they are on site, using a specified set of material selections added one at a time. The resultant bridges could be tested with environmental scenarios to ensure that materials and design offer a sound approach within specified parameters of wear, climate, and soil conditions.

7. What are the implications for teaching and learning?

For hands-on work, such as applied engineering, design, or construction, gesture-based systems may offer new ways to interact with immersive 3D content and to investigate immersive scenarios. Gesture-based technologies hold enormous promise for psychomotor learning, too, making the acquisition of physical skills faster and the copying of best practices and techniques simpler and more immediate. Learners can watch a demonstration of a technique, copy the physical gestures, and hear a critique of their performance or observe a physical result that occurs in response to it. This kind of interaction can provide profound support for those in physical rehabilitation or for those perfecting techniques in dance or martial arts. Students might use such systems to acquire greater strength and speed in athletic endeavors, increased dexterity in playing musical instruments, or a broader expertise in the control of sophisticated equipment. Gestures might even take language acquisition beyond the realm of words and into areas of more accurate communication by enabling students to incorporate suitable facial expressions as they speak.

JANUARY 2014

© 2014 EDUCAUSE. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 License. EDUCAUSE is a nonprofit membership association created to support those who lead, manage, and use information technology to benefit higher education. The EDUCAUSE Learning Initiative is an EDUCAUSE community committed to the advancement of learning through the innovative application of technology. For more information about ELI, including membership, please contact us at info@educause.edu.