add better bettertransformers support #509
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR enhances BetterTransformer support across multiple transformer modules, particularly focusing on compatibility checks and proper initialization order for the nomic-ai/nomic-embed-text-v1.5 model.
- Added
check_if_bettertransformer_possiblefunction in/libs/infinity_emb/infinity_emb/transformer/acceleration.pyto verify model compatibility - Modified transformer modules to conditionally enable BetterTransformer only when both compatible and enabled
- Moved BetterTransformer conversion after quantization for proper operation order
- Standardized benchmark parameters in
Makefileto 50 requests/50 concurrent for both text and vision tests - Added 'eager' attention implementation setting when BetterTransformer is enabled and compatible
6 file(s) reviewed, 7 comment(s)
Edit PR Review Bot Settings | Greptile
| @@ -26,18 +26,14 @@ class TorchAudioModel(BaseAudioEmbedModel): | |||
| def __init__(self, *, engine_args: EngineArgs): | |||
| CHECK_TORCH.mark_required() | |||
| CHECK_TRANSFORMERS.mark_required() | |||
|
|
|||
| self.model = AutoModel.from_pretrained( | |||
| engine_args.model_name_or_path, | |||
| revision=engine_args.revision, | |||
| trust_remote_code=engine_args.trust_remote_code, | |||
| # attn_implementation="eager" if engine_args.bettertransformer else None, | |||
There was a problem hiding this comment.
logic: bettertransformer support appears incomplete - the commented line suggests it's not fully implemented yet
libs/infinity_emb/infinity_emb/transformer/crossencoder/torch.py
Outdated
Show resolved
Hide resolved
libs/infinity_emb/infinity_emb/transformer/embedder/sentence_transformer.py
Outdated
Show resolved
Hide resolved
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #509 +/- ##
==========================================
- Coverage 80.00% 79.85% -0.16%
==========================================
Files 43 43
Lines 3471 3484 +13
==========================================
+ Hits 2777 2782 +5
- Misses 694 702 +8 ☔ View full report in Codecov by Sentry. |
Related Issue
Checklist
Additional Notes
Add any other context about the PR here.