Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question about caching implementation #32

Open
Barry0121 opened this issue Mar 18, 2024 · 2 comments
Open

Question about caching implementation #32

Barry0121 opened this issue Mar 18, 2024 · 2 comments

Comments

@Barry0121
Copy link

Barry0121 commented Mar 18, 2024

Hi! I'm doing a deep dive into the implementation and have some questions.

  1. I see that the key caching-related codes are in DeepCache.sd.pipeline_stable_diffusion.py under StableDiffusionPipeline's call() around lines 726 to 740 (It says "7. Denoising loop" at the top, so I assume it must be around here). However, I am struggling to see how/where you are storing the cached up-sample block output? Are you doing this through sample_from_quad_center()? But this function is only for non-uniform sampling right?
  2. This question is related to Q1: So far all the hyperparameters belonging to the actual caching part of deepcache are: cache_interval, cache_layer_id (I believe this is the skip connection layer), cache_block_id (this should be the up-sampling block), center, and power (these are for non-uniform sampling). However, if we specify which block we want to cache, shouldn't that also specify which skip connection is being connected to the output? So, why do we need to specify both?
  3. Is there anywhere else I should check for stuff you have changed from the original stable_diffusion pipeline (outside of pipeline_stable_diffusion.py)?

There are parts of the paper I'm not sure about, so I apologize if my questions seem dumb. Thank you!

@horseee
Copy link
Owner

horseee commented Mar 19, 2024

Hi @Barry0121,

Q1: The variable responsible for storing cached features is named as prv_features at Line 723 and undergoes updates at Line 753. To understand the mechanism of feature caching, you can delve into the inference code of the UNet architecture. The function sample_from_quad_center exclusively serves to adjust the caching schedule.

Q2: Each combination of cache_layer_id and cache_block_id corresponds uniquely to a skip connection.

Q3: I documented the modifications compared to the original pipeline in ReadMe, available here. You can find detailed information there.

@Barry0121
Copy link
Author

Barry0121 commented Mar 19, 2024

Thank you for the response! Appreciate your insight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants