Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
65 Open 252 Closed
65 Open 252 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Failed to download MMBench_V11.tsv error
#823 opened Mar 3, 2025 by OfirArviv
Ola模型自测与OpenCompass榜单差距很大
#822 opened Mar 3, 2025 by bobo0810
1
是否支持采样评估部分数据集？
#820 opened Mar 1, 2025 by zhangyimi
[BUG?] 使用同样的模型，在此 repo下跑推理与模型官方（例如 LLaVA）的推理 demo 结果不同
#817 opened Feb 28, 2025 by ZI-MA
Unable to reproduce Llama-3.2-11B-Vision-Instruct on MMstar
#814 opened Feb 27, 2025 by bollossom
3
Unable to reproduce Qwen2.5-VL-7B-Instruct
#813 opened Feb 27, 2025 by bollossom
@PhoenixZ810
1
关于使用 --config 的问题
#811 opened Feb 26, 2025 by yaogang2060
关于llava-cot的ChartQA的评估问题
#810 opened Feb 26, 2025 by nth2000
78B模型推理加速（78B Modeling Reasoning Acceleration）
#807 opened Feb 25, 2025 by zfr00
@PhoenixZ810
6
两次相同的测试结果，同一个评估模型，结果差异很大
#806 opened Feb 25, 2025 by slZheng077
@PhoenixZ810
9
会评测Baichuan-Omni-1.5吗？
#801 opened Feb 24, 2025 by bobo0810
How to run VITA1.0 on 4*80G GPUs?
#800 opened Feb 23, 2025 by Espere-1119-Song
@PhoenixZ810
14
自定义模型多卡推理问题，想一机多卡，每卡一个模型，并行推理一个测评
#799 opened Feb 22, 2025 by bjzhb666
@PhoenixZ810
6
Lmdeploy cannot be used to inference and eval
#798 opened Feb 22, 2025 by ligeng0197
@PhoenixZ810
1
qwen2.5VL-7B-Instruct 模型 MMMU(val)评测与官方不符
#797 opened Feb 21, 2025 by shawn0wang
@PhoenixZ810
8
使用qwen2.5_VL fine-tune后评测出现错误，是怎么回事
#791 opened Feb 20, 2025 by shawn0wang
1
Would VLLM be used to accelerate model inference?
#789 opened Feb 19, 2025 by ligeng0197
@PhoenixZ810
3
Could you support custom system prompt?
#785 opened Feb 16, 2025 by zhangzef
Is there a plan to support OCRBenchv2 in the near future?
#783 opened Feb 12, 2025 by Chrisding
关于AI2D评测是否有mask的问题问题
#781 opened Feb 11, 2025 by Super-Shen
Proposal to Add “Document Understanding” Section to the Open VLM Leaderboard
#769 opened Feb 5, 2025 by abrahamdaniels
videomme数据集验证结果问题
#767 opened Feb 3, 2025 by Yibo-0820
@FangXinyu-0913
1
fps and nframe should be set at least one valid value
#766 opened Feb 3, 2025 by 20201111139
@FangXinyu-0913
1
How is MMbench overall accuracy calculated?
#744 opened Jan 22, 2025 by Yesterday086
数据集图片压缩问题
#741 opened Jan 21, 2025 by Qzy568
ProTip! What’s not been updated in a month: updated:<2025-02-03.