Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting 'internal server error' while running ollama_demo file for lightrag for various smaller models. NEED HELP. #498

Open
shalini-agarwal opened this issue Dec 20, 2024 · 0 comments

Comments

@shalini-agarwal
Copy link

INFO:httpx:HTTP Request: POST http://127.0.0.1:11434/api/chat "HTTP/1.1 500 Internal Server Error" ....... ollama._types.ResponseError: POST predict: Post "http://127.0.0.1:52582/completion": EOF

I am trying to run the lightrag_ollama_demo.py file from examples folder in the GitHub repository. I have been constantly getting this error where Ollama encounters an internal server error and stops midway while doing entity extraction. I have tried Llama3.21b, TinyLlama, Phi, Qwen2.5:0.5b with nomic-embed-text, mxbai-embed-large and snowflake-arctic-embed:22m as embedding models. I have tried different combinations of LLM and the embedding models, but I get the same error for all these models. For Qwen, it did work a few times but other times I got this error again. I saw that others also got this error and some suggestions were to change the OLLAMA_KV_CACHE_TYPE to q8_0 and others suggested that after new changes, this error has been fixed. I tried changing the KV value to q8_0 through this command - launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0 in my terminal but even that didn't work. And I pulled all the recent changes only day before yesterday but I am still getting this error.
Here is my Ollama log if that helps -
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant