To read a person’s movements, now needed only one camera room

Researchers from Tokyo Institute of Technology and Carnegie Mellon University have jointly developed a new human motion capture system, which consists of one ultra-wide “fish eye” camera mounted on the user’s chest.

Conventional motion capture systems in specially equipped studios typically rely on multiple synchronized cameras attached to the ceiling and walls that capture the movements of a person wearing a protective suit equipped with multiple sensors. Such systems are often very expensive and limited in terms of space and environment in which the owner can move.

A team of researchers led by Hideki Koike of Tokyo Tech is introducing a new motion capture system, which consists of one ultra-wide fish eye camera mounted on the user’s chest. Their design not only helps to overcome the lack of space in existing systems, but is also cost-effective.

The system, called MonoEye, can capture the user’s body movements, as well as his or her perspective or “view window”. Our ultra-wide fisheye lens has a 280 degree field of view and can capture the user’s limbs, face and environment.

Text of study

To provide reliable multimodal motion capture, the system was designed with three deep neural networks capable of evaluating three-dimensional body posture, head posture and camera posture in real time.

The researchers have already trained these neural networks with an extensive data set consisting of 680,000 images of people with different body shapes, clothing, actions, background and lighting conditions, as well as 16,000 frames of photorealistic images.

The researchers suggest that in the future it will be possible to turn a chest camera into an everyday accessory such as a tie clip, brooch or sports equipment.