Computers learn to read "body language" down to finger movements

Almost half of all communication of people with each other falls on non-verbal contacts, often too inaccurately expressed. For a digital system, which is accustomed to receiving specific commands, this is an insurmountable obstacle in establishing a dialogue between a machine and a person. But the Robotics Institute at Carnegie Mellon University thinks they have found a way out.

It is not difficult to capture an arbitrary gesture with the help of the camera, all the problems are with their accurate and quick interpretation. The training system consists of three steps. First, the computer does not study the behavior of a specific individual; it selects individual gestures of the arms, legs, head turns and looks for their analogs on the object under study. For example, anxious hand waving - first we find a gesture, then we calculate who it belongs to.

To create a database of gestures and postures, a couple of graduate students grimaced in front of the camera for hours, showing the system in total several thousand combinations of body positions. Then came the third stage - using the Panoptic Studio installation of 500 cameras, each pose and individual gesture were filmed in detail from many angles, so that the system learned to accurately identify them in different situations.

It took 31 ultra-high-definition cameras to capture the movement of the fingers, so small and mobile for an electronic eye. But all this machinery and a lot of power are needed only at the stage of training the system; in the future, a simple webcam will be enough to "read" body language, scientists assure. They are currently working on combining 2D and 3D models and recognizing the gestures of individuals in a crowd.

The field of application of the new technology is not clearly delineated, due to the fact that it is problematic to prove the reliability of reading a gesture by a computer. It is one thing to put on a publicity show that recognizes the emotion of the crowd, and quite another to entrust the autopilot to drive the electric car, obeying the careless waves of the palm. Allowing the system to act as an assistant referee at sporting events could result in a scandal. Try explaining to the silicon brain that you weren't simulating collision and pain!