Release 0.1.0 #199

francoishernandez · 2025-01-24T16:47:13Z

francoishernandez
Jan 24, 2025
Maintainer

Hugging Face Dataset Streaming: Enables efficient streaming for processing large datasets. Hugging face dataset streaming support #177
Gemma2 Support Gemma2 support #160

Transformer Decoder Refactor: Major refactor and merging of previously separate LM/seq2seq transformer decoder classes. refactor transformer decoder and revamp the left padding attention mask #178
Hugging Face Model Conversion: Major refactor to better support of HF converter to facilitate further extensions. Major refactoring of convert HF #156

Self-Attention Edge Case: Addressed a specific edge case in self_attn_backend. [patch] handle self_attn_backend edge case #180
Max Length Handling: Resolved tokenization issues related to length limits in tokenize_id. patch max_length handling in tokenize_id #197
Vocabulary Update Fix: Ensures proper loading of parameters when updating vocabularies. fix update vocab param loading #184
Left Padding Attention Mask: Corrected unintended behavior from the recent refactor. hotfix post #178 #181

Code Cleanups: General improvements for code maintainability and readability. Code clean-ups #171
120-Column Limit: Optimized for modern screens, enhancing code readability. 120 columns makes more sense on modern screens #176

Pixtral and vision model support is still cooking. Basic pixtral support, paving the way for vision models 🖼️ #153
Streaming Inference should be investigated.
...