Not long after Pokémon Go made its record-breaking debut, new technology from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) demonstrated how these Pokémon creatures could be seen in augmented reality and affect their surroundings. MIT’s new technology is called Interactive Dynamic Video (IDV).

NBC’s article, “Want More Life in Your Pokemon? Now They Can React in the Real World,” highlighted the IDV technology, which lets you interact with a video instead of just watching it. You can “touch” the objects in the video and make them move, and so can Pikachu, Snorlax, Jigglypuff, and the other 148 creatures in Pokémon Go.

IDV brings Pokémon Go one step closer to dynamic augmented reality. This technology virtualizes real-world content and brings it into the virtual world in the form of a video simulation.

Image credit: Abe Davis/MIT CSAIL.

How does IDV Work?

Currently, augmented reality apps such as Pokémon Go simply superimpose virtual content into the scene you capture on your mobile device. The virtual content floats in front of the real objects on your screen.

IDV creates a simulation of the static objects in the scene. It enables the virtual character to apply forces, such as a push, pull or nudge, to the objects from the real world.

CSAIL Ph.D. student Abe Davis describes this technique in a paper he published earlier this year with Ph.D. student Justin G. Chen and professor Frédo Durand, “This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space. By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos.“

The team takes a short video of a stationary object. The video does not require specialized equipment: an iPhone video will work fine. Much of the complexity is removed since 2-dimensional images are used instead of a 3-dimensional model.

A slight force is applied. The table it was placed on was bumped for the wireman object above. The video captures the vibrations, even the most minute ones.

The key is in the analysis of the video. The resonant frequencies and locations of the vibrations are calculated. The video contains enough information to create a plausible physical model of the object.  The team developed an algorithm that can determine physical characteristics from this information.

IDV uses this information to create a simulation of the object. When you “apply a force” to a point on the object, the simulated response is shown. In the video below, a click of the mouse used to push or pull the object.

The IDV Algorithm

According to Abe Davis, “Our algorithm works in two parts: pre-processing, and simulation. The pre-processing step is where we analyze the video, extract vibration modes, and save them in a compact format for simulation. The pre-processing is done in MATLAB, and then the simulation is written in very efficient C++ and GLSL for speed.

“I chose MATLAB for pre-processing because it offered a lot of agility to test out new ideas. I almost always prototype things in MATLAB if I can get away with it. MATLAB is also great for running experiments — I just create a new script for each experiment I do, and it gives me a record that I can come back to later. It’s super valuable to be able to return to an experiment, months after the fact, and recreate the results with the press of a button.”

Other Applications

Even if interest in Pokémon Go wanes, IDV will surely find new uses. It has great potential in filmmaking, expanding the capabilities of CGI without incurring the high cost of today’s most sophisticated special effect platforms. IDV could also be used in engineering applications, such as structural health monitoring of buildings and bridges.

To learn more, check out this second video: