Cannot reproduce VC1(ViT-L) and VC1-base(ViT-B) in MuJoCo tasks?

[xiaoxiao0406]

Hi,thanks for the great job!I ran launch_all_jobs.sh under eai-vc/cortexbench/mujoco_vc/visual_imitation/ to replicate VC1 results in mujoco tasks. In the Adroit benchmark with VC-1(ViT-L), my two runs showed figures of 37.33 and 47.33 respectively, while the paper states 59.3 ± 5.2. Similarly, in the DMC benchmark with VC-1-base(ViT-B), my results were 40.91 and 60.88 respectively, whereas you reported 63.8 ± 2.7.

Additionally, I seek confirmation regarding the methodology applied in the benchmarks. Are results for each task averaged over three seeds (100, 200, 300) of highest_success in both the Adroit and Metaworld benchmarks? And in the DeepMind Control benchmark, is the average computed over three seeds of highest_score, followed by division by 10?

[xwen99]

You may see my comment for reference: #14 (comment)

[elicassion]

I'm not be able to reproduce the results for mujoco tasks, too. For DMC, I got 60.8 ± 0.93 for VC-1 (ViT-L), and 59.6 ± 0.89 for VC-1 (ViT-B). I'm using the normalization according to #14 (comment), i.e. dividing the raw scores by 10