Fix backward_dense_test

[avbokovoy]

Attempt to fix the following five existing issues in the dense unit test:

1. OOM (only on A100, hard to catch due to randomness of UTs)
2. Memory access error (both MI300X and A100, hard to catch)
3. Assertions failure in PoolingMode.MEAN test (both ROCm and Nvidia)
4. Assertions failure in PoolingMode.SUM test (both ROCm and Nvidia)
5. Wrong indexing in vbe test

Issues 1, 2, 5 are also observed in pytorch/pytorch#141904

The initial intention of aligned_grad_output_tensor_for_cuda_backwards() function is unclear to me, so this fix particular might be "sub-optimal". Thus asking for some reviews

[netlify]

✅ Deploy Preview for pytorch-fbgemm-docs ready!

Name	Link
🔨 Latest commit	a83abef
🔍 Latest deploy log	https://app.netlify.com/sites/pytorch-fbgemm-docs/deploys/67b6f37ec70a520008ab3c7b
😎 Deploy Preview	https://deploy-preview-3702--pytorch-fbgemm-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[facebook-github-bot]

@q10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[sryap]

We should not do this every time. It will be costly. Is there a reason why you would like to do this?

[avbokovoy]

We should not do this every time. It will be costly. Is there a reason why you would like to do this?

@sryap Agree on that we shouldn't do it every time. However, with code bisecting, this allows to pass the unit test. I need some help from your side:

1. What is the intention of aligned_grad_output_tensor_for_cuda_backwards() function? I can assume that we get the grad_output without copy if the data is aligned to 16B, otherwise get the "aligned" tensor from input with potential memory copy. Is the tensor constructed with .contiguous() or empty_like is guaranteed to be aligned?
2. Could you please clarify what is tested here?:
   https://github.com/pytorch/FBGEMM/pull/3702/files#diff-dc94c00639d812c6bddd3a893aa08255d1ca5819cc8c3cfa524706d5a21a65baR331-R340
   We want to make sure that sequential call of bwd will produce the same gradient w.r.t. feature_requires_grad?
3. Is there any possible sync issues that might occur in this test scenario?
4. The parameter set to test the failure is:

(
    T=1,
    D=2,
    B=2,
    log_E=1,
    L=1,
    weights_precision=SparseType.FP16,
    weighted=False,
    mixed=False,
    mixed_B=True,
    long_segments=False,
    pooling_mode=PoolingMode.SUM,                                                                                                                        
    use_cpu=False,
    output_dtype=SparseType.FP32,
)

Also the random seed needs to be fixed at the start of test_backward_dense:

np.random.seed(2007)
torch.manual_seed(2007)

Are those parameters valid?