add cogvideox support for gaudi. #1600

nc-BobLee · 2024-12-12T09:07:36Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

nc-BobLee · 2024-12-12T09:31:17Z

test command：python cogvideo_generate.py --model_name_or_path /mnt/disk2/libo/hf_models/CogVideoX-2b/ --output_name
panda_gaudi.mp4
output video：https://github.com/user-attachments/assets/917511eb-9878-433e-96cc-37242c8e563c

nc-BobLee · 2024-12-12T09:32:47Z

panda_gaudi.mp4

examples/text-to-video/README.md

nc-BobLee · 2025-01-17T09:36:54Z

could you please help to review my change @shepark
thanks!

shepark · 2025-01-18T00:46:13Z

could you please help to review my change @shepark thanks!

I sent email to you, please response it. thanks.

Wei-Lin-Intel · 2025-01-23T02:29:18Z

Hi @regisss , could you please help to review this PR? Thanks!

dsocek

See comments for requested changes. Also, please run make style to fix code formatting

dsocek · 2025-01-23T20:20:18Z

examples/text-to-video/cogvideox_generate.py

@@ -0,0 +1,86 @@
+import argparse


Please remove this file and use text_to_video_generation.py script for this sample (need to switch pipeline based on model, similarly as one in text_to_image_generation.py)

this file has been removed.

dsocek · 2025-01-23T20:27:50Z

optimum/habana/diffusers/__init__.py

@@ -1,3 +1,4 @@
+from .pipelines.cogvideox.cogvideoX_gaudi import adapt_cogvideo_to_gaudi


remove from here

dsocek · 2025-01-23T20:31:43Z

optimum/habana/diffusers/pipelines/cogvideox/cogvideoX_gaudi.py

@@ -0,0 +1,468 @@
+from typing import Any, Dict, Optional, Tuple, Union


This file does not belong here. Either add these definitions to your cogvideox pipeline py file, or maybe put some of the attention functions under optimum/habana/diffusers/models/* and rest inside cogvideox pipeline py file.

this file has been removed.

dsocek · 2025-01-23T20:34:28Z

optimum/habana/diffusers/pipelines/cogvideox/cogvideoX_gaudi.py

+from diffusers.models.autoencoders.autoencoder_kl_cogvideox import AutoencoderKLCogVideoX
+
+
+class AutoencoderKLCogVideoXGaudi(AutoencoderKLCogVideoX):


Adapted from? Please put reference here

file and function removed.

dsocek · 2025-01-23T20:34:59Z

optimum/habana/diffusers/pipelines/cogvideox/cogvideoX_gaudi.py

+from diffusers.models.autoencoders.vae import DecoderOutput
+
+
+class CogVideoXCausalConv3dGaudi(nn.Module):


Adapted from? Please put reference here

function removed.

dsocek · 2025-01-23T20:50:49Z

optimum/habana/diffusers/pipelines/cogvideox/pipeline_cogvideox_gaudi.py

+    frames: torch.Tensor
+
+# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.retrieve_timesteps
+def retrieve_timesteps(


can we import this directly from diffusers?

this is for cogvideoX model does not copy.

dsocek · 2025-01-23T20:53:47Z

optimum/habana/diffusers/pipelines/cogvideox/pipeline_cogvideox_gaudi.py

+logger = logging.get_logger(__name__)  # pylint: disable=invalid-name
+
+
+class time_box_t():


Please use same timing mechanism as in other pipelines (need to deal with warmup etc.)

dsocek · 2025-01-23T20:54:33Z

optimum/habana/diffusers/pipelines/cogvideox/pipeline_cogvideox_gaudi.py

+                    htcore.mark_step()
+        time_box.show_time('transformer_hpu')
+
+        #HabanaProfile.stop()


Remove comment

dsocek · 2025-01-23T21:04:39Z

tests/test_diffusers.py

+        ).frames[0]
+
+        assert video is not None
+        assert 49 == len(video)


Use self.assertEqual here

dsocek · 2025-01-23T21:08:59Z

tests/test_diffusers.py

+            generator=torch.Generator(device="cpu").manual_seed(42),
+        ).frames[0]
+
+        assert video is not None


use self.assertIsNotNone here

astachowiczhabana · 2025-02-06T10:33:38Z

@regisss @nc-BobLee can we merge this change?

dsocek · 2025-02-06T13:39:33Z

@regisss @nc-BobLee can we merge this change?

@nc-BobLee can you, please, address some of the requested changes?

nc-BobLee · 2025-02-11T10:42:44Z

@regisss @nc-BobLee can we merge this change?

@nc-BobLee can you, please, address some of the requested changes?

I have addressed the requested changes. please help to review. @regisss

dsocek

Thanks for addressing the requested changes. Please rebase and resolve conflicts and run make style to fix code formatting.

Co-authored-by: regisss <[email protected]>

Wei-Lin-Intel · 2025-02-24T06:59:24Z

@regisss Please help to review the changes. All the codes are updated now. Thanks!

add cogvideox support for gaudi.

e87f823

nc-BobLee requested a review from regisss as a code owner December 12, 2024 09:07

nc-BobLee added 2 commits December 12, 2024 09:16

update README for cogvideX

fd556ca

import cogvideo module from optimumu lib

6a8e73d

nc-BobLee closed this Dec 12, 2024

nc-BobLee reopened this Dec 12, 2024

nc-BobLee and others added 8 commits December 12, 2024 09:54

refine test examples

7092a51

fix module import defect

13b86c8

update module import method

d125fe6

upgrade for diffusers version 0.31.0

4698ded

Merge branch 'huggingface:main' into cogvideox_dev

33a35da

add cogVideo test case.

21caddc

refine model default path

feff2a3

add required python lib for cogvideo

ae05af9

shepark reviewed Jan 7, 2025

View reviewed changes

examples/text-to-video/README.md Outdated Show resolved Hide resolved

shepark reviewed Jan 7, 2025

View reviewed changes

examples/text-to-video/README.md Outdated Show resolved Hide resolved

nc-BobLee and others added 3 commits January 13, 2025 10:53

refine README.MD

12badb8

Merge branch 'huggingface:main' into cogvideox_dev

731bd91

use gaudi implementation of apply rotary embedding.

7df1a6c

shepark approved these changes Jan 18, 2025

View reviewed changes

gyou2021 and others added 4 commits January 23, 2025 15:23

fix htcore defect

6919313

fix can't find htcore defect.

c15aa51

support for G3 on graph optimization

687caf9

clear debug code,

339e31f

dsocek suggested changes Jan 23, 2025

View reviewed changes

set autoencoder tiled decode gaudi wit setattr.

c3e253e

root and others added 5 commits February 7, 2025 16:46

move cogvideox conv3d to gaudi pipeline.

bb036d6

remove import gaudi function in __init__.py

65fb0ed

mv gaudi func to cogvideo pipelines.

cbf7ee1

refine cogvideox examples.

7148011

import cogvideo pipeline from OH diffusers.

1316cf2

refine some code style.

b49115c

dsocek approved these changes Feb 21, 2025

View reviewed changes

root and others added 7 commits February 24, 2025 11:04

fix code formmattings.

8fb2041

Fix PR doc upload workflow

46cf46a

Added Unit Test for Gemma-2-27b model (huggingface#1616)

c8acce7

Co-authored-by: regisss <[email protected]>

Remove debug for PR doc upload workflow

73b3bcf

Update TRL README.md to clean up models (huggingface#1706)

94503b1

add resolved cogvideox feature.

ae00346

add conflict free cogvideox support.

be81354

nc-BobLee requested review from ssarkar2, bhargaveede, vivekgoe, hlahkar, mandy-li and libinta as code owners February 24, 2025 06:47

Merge branch 'main' into cogvideox_dev

24ce12e

root and others added 2 commits February 24, 2025 15:03

refine T5Config typo.

aaa8b3f

Merge branch 'huggingface:main' into cogvideox_dev

5912278

libinta added the run-test Run CI for PRs from external contributors label Feb 26, 2025

Merge branch 'huggingface:main' into cogvideox_dev

bb84a65

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add cogvideox support for gaudi. #1600

add cogvideox support for gaudi. #1600

nc-BobLee commented Dec 12, 2024 •

edited

Loading

nc-BobLee commented Dec 12, 2024 •

edited

Loading

nc-BobLee commented Dec 12, 2024

nc-BobLee commented Jan 17, 2025

shepark commented Jan 18, 2025

Wei-Lin-Intel commented Jan 23, 2025

dsocek left a comment

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

dsocek Jan 23, 2025

nc-BobLee Feb 21, 2025

astachowiczhabana commented Feb 6, 2025

dsocek commented Feb 6, 2025

nc-BobLee commented Feb 11, 2025 •

edited

Loading

dsocek left a comment

Wei-Lin-Intel commented Feb 24, 2025

		@@ -1,3 +1,4 @@
		from .pipelines.cogvideox.cogvideoX_gaudi import adapt_cogvideo_to_gaudi

		@@ -0,0 +1,468 @@
		from typing import Any, Dict, Optional, Tuple, Union

		from diffusers.models.autoencoders.autoencoder_kl_cogvideox import AutoencoderKLCogVideoX


		class AutoencoderKLCogVideoXGaudi(AutoencoderKLCogVideoX):

		from diffusers.models.autoencoders.vae import DecoderOutput


		class CogVideoXCausalConv3dGaudi(nn.Module):

		logger = logging.get_logger(__name__) # pylint: disable=invalid-name


		class time_box_t():

add cogvideox support for gaudi. #1600

Are you sure you want to change the base?

add cogvideox support for gaudi. #1600

Conversation

nc-BobLee commented Dec 12, 2024 • edited Loading

What does this PR do?

Before submitting

nc-BobLee commented Dec 12, 2024 • edited Loading

nc-BobLee commented Dec 12, 2024

nc-BobLee commented Jan 17, 2025

shepark commented Jan 18, 2025

Wei-Lin-Intel commented Jan 23, 2025

dsocek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astachowiczhabana commented Feb 6, 2025

dsocek commented Feb 6, 2025

nc-BobLee commented Feb 11, 2025 • edited Loading

dsocek left a comment

Choose a reason for hiding this comment

Wei-Lin-Intel commented Feb 24, 2025

nc-BobLee commented Dec 12, 2024 •

edited

Loading

nc-BobLee commented Dec 12, 2024 •

edited

Loading

nc-BobLee commented Feb 11, 2025 •

edited

Loading