offload parameters during forward propagation but the CUDA memory actually be not freed

[zengzh95]

When I swap the parameters or activations of a module to CPU after its forward propagation, the CUDA memory actually be not freed.
But once the backward propagation of this module is performed, we can release the CUDA memory occupied by parameters or activations.

Is this related to the Autograd mechanism of PyTorch?
How can we free parameters or activations before backward propagation in PyTorch?