Notes during memory use reduction

[xloem]

Hi,

For fun, kind of, I'm poking at reducing the memory usage of the standalone example, so more of it can run on my lower end system.

I've only skimmed the paper so far, but while looking at the code I'm encountering some small confusions or questions regarding the implementation, so I'm opening this pull request to connect a little bit.

I'll answer these questions myself when and if I find the answers.

Questions:

• I noticed the use of the kernel norm is detached from the gradient graph and cached between runs. How come this doesn't result in a disparate kernel norm as the kernels are updating during training?

• I noticed the use of F.interpolate does not specify align_corners, leaving it to default to False, which it looks like to me can leave some flat artefacts at the edges, and stretch the content between them by a subsample, when the interpolation is linear. Does this matter? My intuition would have been to do linear interpolation by dropping the last sample or wrapping to the first. In my changes, I had to add a constant of 1 to the input size to get the same interpolation output for truncated kernels.

• Why is it helpful to scale the weights of the kernels by their distance? Wouldn't the training process learn this scaling itself?

• Small changes to the backend can result in small (on the order of 1/100th) changes to outputs unless a lot of care is taken. How important is that kind of numerical stability?

Resolved:

• I understand that n=2*L in the fft is to avoid performing the circular convolution. (this took me some learning)

Idea:
Given the principles of this algorithm, it looks to me like it might be possible to run using very minimal ram, by using the fft to perform the kernel interpolation in frequency space, and streaming the convolution.

[xloem]

This is what I got done today. I'm kind of a crazy flake, so who knows what tomorrow holds, but here are some changes for less ram usage if they are of interest.