Machine Learning Neural Network Musicvideo computer generated

This project by Damien Henry is an hour-long video set to music by Steve Reich. What you are seeing is not a style or filter applied to a video, but actually new footage generated by a neural network. This neural network is trained to videos recorded from train windows, with landscapes that moves from right to left. The algorithm uses a motion-prediction technique, basically it is trying to predict the next frame of the video. After training the neural network, it only needs one frame as an input to start generating new frames indefinitely.

Eerily enough, the predicted footage is capturing the feeling of riding a train pretty well. Even though the landscapes are more dreamlike than realistic, it is interesting enough that the algorithm itself figured out, what makes a train ride a train ride: For example that the background has to move slower than the foreground. It is important to note, that the resolution is currently pretty low due to the technical restriction neural networks have. But you can expect an increase in resolution and quality of such experiments in a not so distant future. Machine learning is still in children's shoes and engineers, artists and coders are trying to figure out, how they actually work and what they can and cannot do. I guess, now you have to cross "dreaming of train rides" off that list.

Next Post Previous Post