Optical Flow Examples

flowHS_Sinus.zip is a workflow that does not need any additional data. Therefore only the workflow parameter file flowHS_Sinus.wrp is contained. The used sequence is a simple sinus test pattern generated using the sequence generator sequence.

First, we start with an overview of the used modules and the parts of the algorithm. The test sequence is generated using the module sequence in the top-left part of the workflow. The center part contains the actual optical flow estimation and the modules on the right visualize and display the results.

Now, we consider the details of the flow estimation. First, we need the derivatives of the input image pair wrt the x,y,t axes. This is done by the module diff. The image pair and its derivatives is then passed to the so called data term representing the Brightness Constant Constraint Equation(BCCE). In our implementation, this term has been generalized and divided into the three modules bm, mm and bcce. This generalized variant (GBCCE for Generalized BCCE) allows to use different brightness models (BM) and motion models (MM) which is not of scope here. Combined with the constant brightness model bm and the local constant motion model mm, the GBCCE represents the standard optical flow equation:

The second term used in the algorithm by Horn and Schunck is the so called spatial term or regularizer. It does not depend on the image input data and penalizes large derivatives of the optical flow field:

Using these two terms, the solver solver assembles a large linear equation system which is then passed to a library called PETSc to be solved. The connected roi is used to determine the considered image region to be used for flow estimation.

After execution, the result is visualized using the ArgosDisplay module. In the two tabs, the used sequence and the flow estimation result are shown. The expected result is a constant flow field to the top right. To see both frames of the input sequence, use the slider of the FrameSelector at the right half of the Display widget.

flowHS_Yosemite.zip is a workflow that depends on the (also included) image data consisting of two frames of the Yosemite Sequence from the famous Middlebury Datasets.

After execution, the result is visualized using the ArgosDisplay module. In the three tabs, the used sequence, the flow estimation result and the ground truth flow are shown.

Executing this workflow, yo may notice that the results in the lower parts of the sequence are quite bad. This is caused by large displacements between the two images (as visible regarding the two frames of the sequence). An iterated approach using pyramids would solve this but the example workflow here only shows the original approach from [HornSchunck81].

flowHS_Yosemite_Multi.zip is a workflow that depends on the (also included) image data consisting of five frames of the Yosemite Sequence from the famous Middlebury Datasets. Here, we used frame 10 to 14 of the dataset.

After execution, the result is visualized using the ArgosDisplay module. In the three tabs, the used sequence, the flow estimation result and the ground truth flow are shown. Use the frame selector to scroll through the five frames. Note that flow estimation takes place between the current and the subsequent image, so there are only four flow frames corresponding to the four time steps between five images.

This workflow demonstrates that the presented implementation of the Horn&Schunck algorithm is capable to estimate the optical flow on more than two consecutive frames of an image sequence.

The workflow presented in flowHS_Yosemite_IterCrop.zip uses the same data than the example before. The difference is that the five frames are not processed simultaneously but sliced into consecutive image pairs. These pairs are presented to the Horn&Schunk algorihtm in the center and the output is then assembled into an optical flow field with four time frames.

This workflow may serve as a base to estimate optical flow on multiple frames using algorithms only capable to handle two sequence frames at a time.

flowHS_Iterated_Yosemite.zip also uses two frames from the Yosemite Sequence (as above). The same algorithm by [HornSchunck81] is applied iteratively to the input sequence warped with the result of the previous iteration (starting with zero flow). The result is an flow increment and is added up onto the previous result.

After execution, the result is visualized using the ArgosDisplay module. In the two tabs, the flow estimation result and the ground truth are shown. Sequence visualization has been dropped, use the workflow from above to see the sequence frames.

Results in the lower left parts of the sequence look much better than without iterations, even if still no pyramid is used.

flowHS_Pyramid_Yosemite.zip also uses two frames from the Yosemite Sequence. There are two nested loops, the inner loop is exactly the iterative warping algoritm described before, but with only very few iterations. The outer loop implements the multiscale approach and resizes the input sequence to each pyramid level. Flow estimation starts at a low level with very small images, the last iteration uses the original size.

In the two tabs of the ArgosDisplay, the flow estimation result and the ground truth are shown. Sequence visualization has been dropped, use the workflow from above to see the sequence frames.

Results look very similar to the previous approach, but execution is much faster. This is caused by the reduced image size at lower pyramid levels and only a few iterations in the inner warping loop.

In flowHS_Pyramid_Rubberwhale.zip we use another of the Middlebury test sequences called RubberWhale. There only one loop selecting the pyramid level. Adding the flow increments is done explicitly using the add module. The pyramid uses very many levels (80) with a scaling factor of 95%.

Data input and output is done using the KittiReader and KittiWriter as the sequence and groundtruth is stored in the way proposed by the KITTI dataset The results are written to the res folder.