There are existing applications that turn a photo into a video. From Photoshop, to Premiere Pro, to After Effects, there are also plugins and other predictive algorithms that can do this task. This new algorithm however, aims to be the better version of the current applications for turning a photo to a video. Researchers at the University of Washington have developed a deep learning method that only needs a single photo to make a believable video, as DIY Photography details:
Listen beautiful relax classics on our Youtube channel.
Aleksander Hołyński explains that turning a photo into a video requires the algorithm to predict the future. “And in the real world, there are nearly infinite possibilities of what might happen next,” he adds. So, he and his team trained a neural network with thousands of videos of waterfalls, rivers, oceans, and other material with fluid motion. They would first ask the network to predict the motion of a video only by the first frame. Then, it would compare its prediction with the actual video, which helped it learn to identify clues that tell it what was going to happen next (such as ripples in a stream, for example).
The researchers tried to use “splatting,” a technique that moves each pixel according to its predicted motion. However, it posed another set of challenges. “Think about a flowing waterfall,” Hołyński told UW News. “If you just move the pixels down the waterfall, after a few frames of the video, you’ll have no pixels at the top!” So they had to come up with a solution for this, and they called it “symmetric splatting.” It doesn’t only predict the future, but also “the past” of an image, creating a seamless animation.
image credit: Sarah McQuate/University of Washington