Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LoraX #2

Open
ghost opened this issue Dec 10, 2023 · 1 comment
Open

Add LoraX #2

ghost opened this issue Dec 10, 2023 · 1 comment

Comments

@ghost
Copy link

ghost commented Dec 10, 2023 •
edited by ghost
Loading

It is worth mentioning https://github.com/predibase/lorax on this list. It was built based on tgi v0.9.4 (when they still had the apache license) to enable dynamic lora adaptor loading on the inference time.

Related - maybe it's also worth mentioning dynamic lora adaptor loading as a feature itself. Two ideas I've seen so far are LoRaX (loading lora adaptor from disk at request time) and S-LoRA (pre-load adaptors in the GPU memory and route the compute at request time).

I can start a PR to add LoRaX to the table.
@lapp0
Copy link
Owner

lapp0 commented Dec 10, 2023

Sounds good to me, thanks for contributing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lapp0