Fix Qwen2VL mrope for transformers 4.47.0

[li-plus]

Summary

Fix #461

Testing Done

• Hardware Type: A800-SXM4-80GB
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[ByronHsu]

Dude this is fast lol. I am curious why the tests can pass on AMD?

[li-plus]

No idea, but it seems I have no permission to run tests on nvidia gpu so they failed.

Run modal run dev.modal.tests
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Token missing. Could not authenticate client. If you have token credentials, │
│ see modal.com/docs/reference/modal.config for setup help. If you are a new   │
│ user, register an account at modal.com, then run `modal token new`.          │
╰──────────────────────────────────────────────────────────────────────────────╯

[austin362667]

Amazing! Thanks for super fast turnaround. @li-plus The Nvidia CI works as expected in my modal account.

I can help with reverting the workaround of #463 after this PR.

Wondering is this code change corresponds to https://github.com/huggingface/transformers/pull/34274/files#diff-09bc594f9680f1d042fd485106c68022d77b59831697a00b3b38f12a3e40f395R1698-R1715?

[li-plus]

@austin362667 Thanks! I haven't dug into the code diffs. From the model input perspective, before 4.47.0, the position_ids for each sample within a batch always starts from 0 and then increases, so I only accessed the first row of the cos/sin embed to save HBM bandwidth. However after 4.47.0, the position_ids is left padded by 1s and is different for each sample because the sequence length varies. I have to access the full rotary embeds to get the correct position for each sample. This change is compatible to transformers of older version.

[ByronHsu]

I can help with reverting the workaround of #463 after this PR.

@austin362667 Please do that. Thanks! Also, @li-plus i have added you as a maintainer, so you can directly push the the main repo's branch to run CI. We have disabled CI from external forks due to security reasons currently.