Optical flow estimation can be improved by using geometry of the scene (top left) or semantics that help segmentation (bottom left). Learning to match patches in the images (top right) can help to further improve optical flow. We can also use a complete end-to-end deep learning approach (bottom right) to estimate optical flow in real time.
Classical optical flow methods are based on comparing image pixels using a photometric loss with no information about the world and its geometry, or the statistics of image information effects optical flow estimation. We developed several methods that exploit the semantic information of the scene, utilize the geometry and approach the optical flow problem in a completely data-driven way using deep neural networks.
One of our Optical flow methods segments the image into various parts or layers based on semantic content [ ] . The optical flow for each of these segments is computed by using priors based on the semantic content in the segment, achieving state of the art results.
In contrast, MR-Flow [ ] uses geometric constraints in the world to apply priors on optical flow estimation. Our method breaks down an image in to static and moving parts estimates the depth of the static parts and the flow of the moving part. The optical flow is then obtained by compositing the two parts.
Further improvements in optical flow can be obtained by learning to match patches instead of applying a photometric loss on the pixels. Deep Discrete Flow [ ] uses a neural network for patch matching that can be used in optimization framework of estimating optical flow.
These methods rely on convex optimization for optical flow methods and tend to slow. We introduce SpyNet [ ], where we replace the optimization framework and estimate optical flow using an end-to-end deep neural network. This gives us real time optical flow estimation and is the smallest neural network in terms of memory for optical flow estimation.