Remove _supports_static_cache = True for some model classes #34975
Conversation
| @@ -330,6 +330,8 @@ def forward(self, hidden_states): | |||
| ) # [num_tokens, num_experts] | |||
| gates = zeros.scatter(1, top_k_indices, 1) # [num_tokens, num_experts] | |||
| expert_size = gates.long().sum(0) # [num_experts,] | |||
| # (This cause torch.compile to fail with `torch._dynamo.exc.Unsupported: Backend compiler failed with a fake tensor exception at`) | |||
| # (and `DataDependentOutputException`) | |||
| expert_size = expert_size.tolist() | |||
There was a problem hiding this comment.
jimba has this line expert_size = expert_size.tolist() too and it has no _supports_static_cache = True. Let do the same for this model.
| if attention_mask.max() != 0: | ||
| raise ValueError("Custom 4D attention mask should be passed in inverted form with max==0`") |
There was a problem hiding this comment.
this is the only place I see an extra check attention_mask.max() != 0 within if attention_mask is not None and attention_mask.dim() == 4. Not sure if we really need it, but it gives another different error (different from what expert_size = expert_size.tolist() give above) if we use torch compile
There was a problem hiding this comment.
cc @mayank31398 you may know better if this check is really necessary
There was a problem hiding this comment.
it is not in granite modeling code though
| @@ -1155,7 +1156,7 @@ def forward( | |||
| elif position_ids is None: | |||
| position_ids = cache_position.unsqueeze(0) | |||
|
|
|||
| if (pixel_values, image_encoder_embeddings, perceiver_embeddings).count(None) != 2: | |||
There was a problem hiding this comment.
this will fail torch compile with another different type error.
ydshieh
changed the title
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
What does this PR do?
Remove
_supports_static_cache = Truefor some model classes. See the comments in changes.They were True before because it is set simply we can use static cache without torch.compile. But after #34247, static is kind tied to
torch.compileand we should say it works if it works with torch.compile