Equations (1) and (2) in paper

[ambitionforcomputervision]

Hi,

Thank you for sharing the dataset and I look forward to see the code as well.

Regarding the Equations (1) and (2) in the paper: since the linear motion approximation combines the bidirectional flow, shouldn't the division in the equation be separated into two terms, such that the normalization is only applied to the appropriate flow. Specifically, -F_0t should only be normalized with w_0 in Equation (1) and not w_1. Analogous conditions apply for Equation (2).

[hjSim]

Hi. Thank you for your question.
I am afraid I don't understand your concerns perfectly. As I understand it, did you mean that, in the equation (1), the denominator should be "(1-t) * sum(w0)", not "(1-t) * sum(w0) + t * sum(w1)" ?

In the numerator of equation (1), (-F_0t^y) and F_1(1-t)^y are candidate flow vectors, and (1-t)sum(w0) and tsum(w1) are the corresponding coefficients (weighted sum) of the candidate flow vectors, respectively. Therefore we need to normalize the numerator with their coefficients, so that the resulting ~F_t0 has an appropriate length of flow vector. (Same for equation (2))

Also, Table 3 and Fig.7 support the validity of the equations (1) and (2).

[ambitionforcomputervision]

To clarify, what I mean is that in Equation (1), the candidate flow vector -F_0t^y should only be normalized with the weights w_0 and F_1(1-t)^t only normalized with w_1. In the current version of the paper, the candidate flow vectors are first combined and then divided by the sum of the weights. But in practice, the shapes of the of the flows vectors are not equal, thus the addition of the candidate flow vectors cannot be done. Instead, the complementary reversed flow F_t0^x = ((1-t) * sum(w_0) * (-F_0t^y) / (1-t) * sum(w_0)) + ((t * sum(w_1) * F_1(1-t)^y) / t * sum(w_1)).
Hopefully, this is clear.

[hjSim]

Thank you for the clarification.
First of all, more precisely, "(1-t)* " and "t* " in denominators should be dropped in your suggested equation. Because (1-t) and t are the weight terms to combine two flows to approximate F_t0^x. If not, the length of F_t0^x will be twice the length of the desired flow.
Hence, your equation should be :
F_t0^x = ((1-t) * sum(w_0) * (-F_0t^y) / (1-t)* sum(w_0)) + ((t * sum(w_1) * F_1(1-t)^y) / t* sum(w_1))

Then the above equation can be regarded as the linear combination of flow reversals, which is different from our proposed complementary flow reversal (CFR). (If you are not familiar with "flow reversal", please refer to "Quadratic video interpolation", NeurIPS, 2019)

We are aware that the linear combination of flow reversals can be another option for the flow approximation of F_t0. But in this approximation, the flows are reversed and then linearly combined simply through the two separated computations, which does not appropriately fill the holes appearing in the flow reversal. The flow vectors in the holes (i.e. when N_0 is empty) are zero vectors. These zeros vectors must replaced by the complementary flow vector if it exists (if N_1 is not empty). This is what we want to implement through the proposed CFR.

For example, suppose that two adjacent objects in the center of the image frame are moving away from each other along time t=0 to t=1. Then the optical flow F_01 should be diverging from the center, so that flow reversal of the F_01 reveals holes around the center of the image. In equation (1), N_0 is empty for the pixel x of the hole. However, F_10, we call a complementary flow, is converging flow to the center, so that the hole does not exist around the center of the image. Then N_1 is not the empty set for the pixel x in equation (1). Let's assume t=0.5, N_1={y} is a singleton and F_1(1-t)^y is an in-ward flow vector of the length 10.

In this case, in the linear combination of flow reversals, the zero flow vector (hole, 1st term) and the 10-length flow vector (2nd term) are linearly combined, so that F_t0^x = 5. The zero flow vector of the hole also affects the approximation.

In contrast, in our CFR, the zero vector of the hole can be ignored since N_0 is empty and the 10-length flow vector only contributes to obtain F_t0^x=10, which is desired. This is because we complementarily linearly combined two reversed flows at the same time with the single fractional term in equation (1), not in the separated two terms as in the linear combination of flow reversals.

To support our hypothesis, we evaluated the linear combination of flow reversals as in Table 3 in our paper. The end-point error of this approximation is 0.0807, which is worse than our CFR (0.0721).