Update llama32 vision (mllama) use attention bias #11316

yaoyu-33 · 2024-11-18T20:41:06Z

What does this PR do ?

Fix a previous workaround in llama32 vision to use attention bias in attention calculation.

Collection: [multimodal]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: yaoyu-33 <[email protected]>

Signed-off-by: Oliver Koenig <[email protected]>

Signed-off-by: yaoyu-33 <[email protected]>

scripts/vlm/mllama_finetune.py

Signed-off-by: yaoyu-33 <[email protected]>

…' into yuya/update_mllama_use_attn_bias

Signed-off-by: yaoyu-33 <[email protected]>

Signed-off-by: Yu Yao <[email protected]>

cuichenx

should we update 90B recipes to instruct model and the same lora config as 11B?

scripts/vlm/mllama_finetune.py

nemo/collections/vlm/recipes/mllama_11b.py

Signed-off-by: yaoyu-33 <[email protected]>

cuichenx

LGTM Thanks!

github-actions · 2024-11-22T19:38:19Z

beep boop 🤖: 🚨 The following files must be fixed before merge!

Your code was analyzed with PyLint. The following annotations have been identified:


------------------------------------
Your code has been rated at 10.00/10

Thank you for improving NeMo's documentation!

github-actions · 2024-11-23T01:13:55Z

[🤖]: Hi @yaoyu-33 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

* update recipe Signed-off-by: yaoyu-33 <[email protected]> * fix mllama mock ds Signed-off-by: yaoyu-33 <[email protected]> * update to use attention bias Signed-off-by: yaoyu-33 <[email protected]> * remove example Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring mock.py Signed-off-by: yaoyu-33 <[email protected]> * fix docstring language.py Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring language.py Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring mllama/base.py Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring mllama/language.py Signed-off-by: yaoyu-33 <[email protected]> * bump mcore Signed-off-by: Oliver Koenig <[email protected]> * Add scripts for mllama Signed-off-by: yaoyu-33 <[email protected]> * fix Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update script Signed-off-by: yaoyu-33 <[email protected]> * fix pylint Signed-off-by: yaoyu-33 <[email protected]> * revert Dockerfile.ci Signed-off-by: Yu Yao <[email protected]> * update script match recipe Signed-off-by: yaoyu-33 <[email protected]> * update recipes Signed-off-by: yaoyu-33 <[email protected]> * update mllama 90b recipe Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: Oliver Koenig <[email protected]> Signed-off-by: Yu Yao <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: Oliver Koenig <[email protected]>

* update recipe Signed-off-by: yaoyu-33 <[email protected]> * fix mllama mock ds Signed-off-by: yaoyu-33 <[email protected]> * update to use attention bias Signed-off-by: yaoyu-33 <[email protected]> * remove example Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring mock.py Signed-off-by: yaoyu-33 <[email protected]> * fix docstring language.py Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring language.py Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring mllama/base.py Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix docstring mllama/language.py Signed-off-by: yaoyu-33 <[email protected]> * bump mcore Signed-off-by: Oliver Koenig <[email protected]> * Add scripts for mllama Signed-off-by: yaoyu-33 <[email protected]> * fix Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update script Signed-off-by: yaoyu-33 <[email protected]> * fix pylint Signed-off-by: yaoyu-33 <[email protected]> * revert Dockerfile.ci Signed-off-by: Yu Yao <[email protected]> * update script match recipe Signed-off-by: yaoyu-33 <[email protected]> * update recipes Signed-off-by: yaoyu-33 <[email protected]> * update mllama 90b recipe Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: Oliver Koenig <[email protected]> Signed-off-by: Yu Yao <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: Oliver Koenig <[email protected]> Signed-off-by: Youngeun Kwon <[email protected]>

ko3n1g requested review from pablo-garay and ko3n1g as code owners November 19, 2024 10:53

ko3n1g added the Run CICD label Nov 19, 2024

yaoyu-33 and others added 15 commits November 19, 2024 11:53

update recipe

e3ca50f

Signed-off-by: yaoyu-33 <[email protected]>

fix mllama mock ds

c935e97

Signed-off-by: yaoyu-33 <[email protected]>

update to use attention bias

389f148

Signed-off-by: yaoyu-33 <[email protected]>

remove example

12e6a45

Signed-off-by: yaoyu-33 <[email protected]>

Apply isort and black reformatting

98ab3a5

Signed-off-by: yaoyu-33 <[email protected]>

fix docstring mock.py

bae1e22

Signed-off-by: yaoyu-33 <[email protected]>

fix docstring language.py

db755f2

Signed-off-by: yaoyu-33 <[email protected]>

Apply isort and black reformatting

3f5f478

Signed-off-by: yaoyu-33 <[email protected]>

fix docstring language.py

fbd71ab

Signed-off-by: yaoyu-33 <[email protected]>

Apply isort and black reformatting

d967646

Signed-off-by: yaoyu-33 <[email protected]>

fix docstring mllama/base.py

cd15326

Signed-off-by: yaoyu-33 <[email protected]>

Apply isort and black reformatting

1fea80d

Signed-off-by: yaoyu-33 <[email protected]>

Apply isort and black reformatting

01fccf4

Signed-off-by: yaoyu-33 <[email protected]>

fix docstring mllama/language.py

908079c

Signed-off-by: yaoyu-33 <[email protected]>

bump mcore

6373da2

Signed-off-by: Oliver Koenig <[email protected]>

ko3n1g force-pushed the yuya/update_mllama_use_attn_bias branch from db94715 to 6373da2 Compare November 19, 2024 10:54

ko3n1g added Run CICD and removed Run CICD labels Nov 19, 2024

Merge branch 'main' into yuya/update_mllama_use_attn_bias

3a3bc62

yaoyu-33 added Run CICD and removed Run CICD labels Nov 21, 2024

yaoyu-33 and others added 4 commits November 21, 2024 10:28

Merge branch 'main' into yuya/update_mllama_use_attn_bias

1fd657a

Add scripts for mllama

67bd26c

Signed-off-by: yaoyu-33 <[email protected]>

fix

f94915a

Signed-off-by: yaoyu-33 <[email protected]>

Apply isort and black reformatting

dcd8ac6

Signed-off-by: yaoyu-33 <[email protected]>

github-advanced-security bot found potential problems Nov 21, 2024

View reviewed changes

scripts/vlm/mllama_finetune.py Fixed Show fixed Hide fixed

update script

d88d182

Signed-off-by: yaoyu-33 <[email protected]>

yaoyu-33 added 2 commits November 21, 2024 13:33

Merge remote-tracking branch 'origin/yuya/update_mllama_use_attn_bias…

e76f25d

…' into yuya/update_mllama_use_attn_bias

fix pylint

dd72cf4

Signed-off-by: yaoyu-33 <[email protected]>

yaoyu-33 added Run CICD and removed Run CICD labels Nov 21, 2024

Merge branch 'main' into yuya/update_mllama_use_attn_bias

2c28ad6

yaoyu-33 added Run CICD and removed Run CICD labels Nov 22, 2024

revert Dockerfile.ci

ec1ebac

Signed-off-by: Yu Yao <[email protected]>

yaoyu-33 added Run CICD and removed Run CICD labels Nov 22, 2024

cuichenx reviewed Nov 22, 2024

View reviewed changes

scripts/vlm/mllama_finetune.py Show resolved Hide resolved

nemo/collections/vlm/recipes/mllama_11b.py Show resolved Hide resolved

yaoyu-33 added 3 commits November 22, 2024 11:19

update script match recipe

6b2b5e9

Signed-off-by: yaoyu-33 <[email protected]>

update recipes

29572dd

Signed-off-by: yaoyu-33 <[email protected]>

update mllama 90b recipe

05f3cb6

Signed-off-by: yaoyu-33 <[email protected]>

cuichenx approved these changes Nov 22, 2024

View reviewed changes

cuichenx added Run CICD and removed Run CICD labels Nov 22, 2024

yaoyu-33 added Run CICD and removed Run CICD labels Nov 22, 2024

yaoyu-33 merged commit 5094b2e into main Nov 25, 2024
179 of 329 checks passed

yaoyu-33 deleted the yuya/update_mllama_use_attn_bias branch November 25, 2024 01:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update llama32 vision (mllama) use attention bias #11316

Update llama32 vision (mllama) use attention bias #11316

yaoyu-33 commented Nov 18, 2024

cuichenx left a comment

cuichenx left a comment

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 23, 2024

Update llama32 vision (mllama) use attention bias #11316

Update llama32 vision (mllama) use attention bias #11316

Conversation

yaoyu-33 commented Nov 18, 2024

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

cuichenx left a comment

Choose a reason for hiding this comment

cuichenx left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 23, 2024