1991.Prob—Probability Distributions of Optical Flow (Simoncelli... 1991)

Link to paper

A Bayesian reworking of a linearized version of L&K, a local method. As outlined in F&W, the gradient constancy assumption can be worked into a matrix formula, where $\mathbf{v}$ is the predicted velocity at a point:

$$\mathbf{M}\cdot \mathbf{v}+\mathbf{b}=0,\mathbf{M}=\left(\begin{array}{cc}\sum f_x^2& \sum f_x{f}_y\\ \sum f_x{f}_y& \sum f_y^2\end{array}\right)=\mathbf{G}{\mathbf{G}}^T,\mathbf{b}=\left(\begin{array}{c}\sum f_x{f}_t\\ \sum f_y{f}_t\end{array}\right)$$

if $\mathbf{G}=\left(\begin{array}{ccc}{f}_{x_1}& f_{x_2}\cdots & f_{x_n}\\ f_{y_1}& f_{y_2}\cdots & f_{y_n}\end{array}\right)$. (The $f_x,f_y$ are partial derivatives of the video $f$ with respect to $x,y$.)

This doesn’t provide a certainty measure, however. By imposing Gaussian priors on $\mathbf{v}$, the noise on $\mathbf{v}$, and the noise on $f_t$, we can obtain an uncertainty measure which primarily depends on the magnitudes of $\mathbf{M}$, ${\mathbf{f}}_{\mathbf{s}}$, and $\mathbf{b}$:

$${\mu }_{\mathbf{v}}=-{\left[\frac{\mathbf{M}}{{\sigma }_1||{\mathbf{f}}_s|{|}^2+{\sigma }_2}+{\Lambda }_p^{-1}\right]}^{-1}\cdot \frac{\mathbf{b}}{{\sigma }_1||{\mathbf{f}}_s|{|}^2+{\sigma }_2},$$

where ${\sigma }_1,{\sigma }_2,{\Lambda }_p$ are defined in Outline of the approach and ${\mathbf{f}}_s=(f_x,f_y)$.

However, I think there are some suspicious parts in the math that I don’t really like. Additionally, since this is a local method and basically only involves direct calculation from the image (no iteration or anything), the uncertainty measures given by this method aren’t really tuned and don’t seem super ideal. I’m looking for a probabilistic formulation that allows an explicit (by the model) estimate of certainty for each point in the optical flow.

Tags §

Local methods Multi-scale pyramids, coarse-to-fine NOT deep learning Old school Optical flow Probabilistic flow Repeated warping Unsupervised Upscaling flow

Outline of the approach §

If we just take a single condition imposed by a single pixel, the linearized brightness constancy condition gives us

$${\mathbf{f}}_s\cdot \mathbf{v}+f_t=0.$$

Now, in reality we might actually have some noise. Suppose that $\mathbf{v}$ actually has a noise ${\mathbf{n}}_1$, and that $f_t$ actually has a noise $n_2$, so the above is not true; instead, we would have

$${\mathbf{f}}_s(\mathbf{v}-{\mathbf{n}}_1)+(f_t-n_2)=0⟹{\mathbf{f}}_{\mathbf{s}}\cdot \mathbf{v}+f_t={\mathbf{f}}_s\cdot {\mathbf{n}}_1+n_2,$$

in which we assume that ${\mathbf{n}}_i\sim \mathcal{N}(0,{\Lambda }_i)$ for $i=1,2$. I don’t know why ${\mathbf{f}}_s$ is assumed to be noiseless; it seems perfectly natural to have spatial noise. I guess maybe I’ll figure that later on.

Since we have the gradients $\nabla f$ and want to obtain $\mathbf{v}$, we want to figure out what the distribution $\mathbf{P}(\mathbf{v}|\nabla f)$ is. In order to do this, we use Bayes’ rule:

$$\mathbf{P}(\mathbf{v}\mid {\mathbf{f}}_s,f_t)=\frac{\mathbf{P}(f_t\mid \mathbf{v},{\mathbf{f}}_s)\cdot \mathbf{P}(\mathbf{v}\mid {\mathbf{f}}_s)}{\mathbf{P}(f_t\mid {\mathbf{f}}_s)}.$$

Then $\mathbf{P}(\mathbf{v}\mid {\mathbf{f}}_s)$ gets simplified to $\mathbf{P}(\mathbf{v})$, which doesn’t really make much sense to me (is this not just straight up wrong, and $\mathbf{v}$ clearly depends on ${\mathbf{f}}_s$?). The denominator gets reduced to $\mathbf{P}(f_t)$, which is fair as they’re independent.

We then need a prior for $\mathbf{P}(\mathbf{v})$, which the authors select to be Gaussian with covariance ${\mathbf{\Lambda }}_p$. Of course this is a reminder that now we are biasing towards $\mathbf{v}\to 0$.

Finally, with this, they do some math and get the following:

$$\begin{array}{rl}{\mathbf{\Lambda }}_{\mathbf{v}}& =[{\mathbf{f}}_s({\mathbf{f}}_s^T{\mathbf{\Lambda }}_1{\mathbf{f}}_s+{\mathbf{\Lambda }}_2{)}^{-1}{\mathbf{f}}_s^T+{\mathbf{\Lambda }}_p^{-1}{]}^{-1}\\ {\mu }_{\mathbf{v}}& =-{\mathbf{\Lambda }}_{\mathbf{v}}{\mathbf{f}}_s({\mathbf{f}}_s^T{\mathbf{\Lambda }}_1{\mathbf{f}}_s+{\mathbf{\Lambda }}_2{)}^{-1}{f}_t.\end{array}$$

I think this can probably just be directly calculated so I’m not really going to bother checking this. The diagonal form is given above in the introductory section, where ${\mathbf{\Lambda }}_1=I\cdot {\sigma }_1$ and ${\mathbf{\Lambda }}_2=[{\sigma }_2]$. In that form, the bias $\mathbf{v}\to 0$ is more clearly seen, and I’m not sure if I particularly like it.

Cited §

• 1981.H&S—Determining Optical Flow (Horn & Schunck, 1981)
• 1981.L&K—An Iterative Image Registration Technique with an Application to Stereo Vision (Lucas & Kanade, 1981)

Cited By §

• ! 2005.F&W—Optical Flow Estimation (Fleet & Weiss, 2005)

Return: index