v2.1.0

更新内容

• 发布自研多模数据能力标签模型PP-InsCapTagger；可用于数据的分析和过滤，试验案例表明在保持模型效果的条件下可减少50%的数据量，大幅提高训练效率。

• 新增Qwen2-VL、InternVL2、Stable Diffusion 3 (SD3)等前沿模型。

• 多模态大模型InternVL2、LLaVA、SD3、SDXL适配昇腾910B，提供国产计算芯片上的训推能力。

What's Changed

• 【pir 】modify dy2static Sd and 3. Grounding DINO model by @xiaoguoguo626807 in #689
• fix llava pretrain config by @pkhk-1 in #685
• Re-network the DIT, fix some parameters, and simplify the model networking code by @chang-wenbin in #632
• update DIT doc by @chang-wenbin in #693
• [NPU] Add llava npu doc by @Birdylx in #694
• sd3推理优化——避免同步 by @chang-wenbin in #695
• 减少重复拷贝，修复BUG by @chang-wenbin in #699
• Add Qwen2-VL infer codes by @nemonameless in #698
• [doc] Update requirements by @nemonameless in #703
• Llava bug by @LokeZhou in #704
• Fix is inference mode by @zhoutianzi666 in #711
• update readme by @lyuwenyu in #705
• update opensora video save method by @westfish in #712
• Limit the installed version of paddlenlp and fix bugs of llava-next. by @luyao-cv in #716
• SD3 transformer部分的优化 by @zhoutianzi666 in #713
• [wip] add mix scheme by @lyuwenyu in #664
• [NPU] InternVL2 supports npu training by @Birdylx in #714
• Add SD3 DreamBooth by @westfish in #686
• remove phi3 in internvl2 and refine format by @nemonameless in #715
• add flash_atten for qw2vl by @luyao-cv in #723
• [NPU] sdxl support NPU training by @wangna11BD in #719
• [NPU] sdxl lora support NPU training by @warrentdrew in #718
• Adapt fa for npu by @LielinJiang in #706
• [NPU] fix readme doc for SDXL LoRA training by @warrentdrew in #724
• [npu]sd3 dreambooth adapt for npu by @LielinJiang in #726
• add pp-inscaptagger by @pkhk-1 in #727
• ADD SD3 batch_parallel by @chang-wenbin in #731
• support auto parallel in dit and largedit by @jeff41404 in #551
• add env_run.sh and correct packages version by @luyao-cv in #733
• [NPU] Fix typo by @Birdylx in #696
• paddlemix v2.1 readme by @lyuwenyu in #734
• 修复paddlenlp develop版本适配错误_10-11 by @Xiaobin-Lu in #735
• 修复qwen2vl视频图像预处理 by @luyao-cv in #737
• [wip] update v2.1 readme by @lyuwenyu in #736
• fix internvl2 minimonkey dataset docs by @nemonameless in #741
• fix tests of evaclip and internvl2 by @nemonameless in #746
• image2text_generation rm use_fast by @LokeZhou in #744
• fix readme for llava_next_interleave by @luyao-cv in #748
• support Qwen2-VL sft training by @nemonameless in #739
• fix dit training by @nemonameless in #752
• fix tests by @nemonameless in #753
• remove use_fast in AutoTokenizer by @warrentdrew in #747
• fix dit weights convert to ppdiffusers by @nemonameless in #759
• [PPDiffusers]fix bugs and release 0.29.0 by @westfish in #742
• autolabel fix nltk download by @LokeZhou in #763
• [NPU] fix npu llava infer by @Birdylx in #757
• Add npu model list by @nepeplwu in #758
• Fix docs of by @nemonameless in #767
• merge upstream readme by @luyao-cv in #766
• correct huggingface_hub version by @luyao-cv in #771
• [NPU] Refine doc by @Birdylx in #774

New Contributors

• @xiaoguoguo626807 made their first contribution in #689
• @chang-wenbin made their first contribution in #632
• @wangna11BD made their first contribution in #719
• @LielinJiang made their first contribution in #706
• @jeff41404 made their first contribution in #551
• @Xiaobin-Lu made their first contribution in #735
• @nepeplwu made their first contribution in #758

Full Changelog: https://github.com/PaddlePaddle/PaddleMIX/commits/v2.1.0