Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify to make rerank provider works to llama.cpp rerank API #9159

Closed
5 tasks done
ziyu4huang opened this issue Oct 10, 2024 · 1 comment · Fixed by #9160
Closed
5 tasks done

Modify to make rerank provider works to llama.cpp rerank API #9159

ziyu4huang opened this issue Oct 10, 2024 · 1 comment · Fixed by #9160
Labels
💪 enhancement New feature or request 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database.

Comments

@ziyu4huang
Copy link
Contributor

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Recently, llama.cpp support rerank , I modify provider to solve llama.cpp API problem not return text.

See llama.cpp issue

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
@dosubot dosubot bot added 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database. 💪 enhancement New feature or request labels Oct 10, 2024
@ziyu4huang
Copy link
Contributor Author

I have verify using latest llama.cpp, run with below option
./llama.cpp/build_hip/bin/llama-server --host 0.0.0.0 --port 8090 -c 8128 -m ./gguf/bge-reranker-v2-m3-Q4_K_M.gguf --reranking

Test use local-ai provider . It works well.

slot launch_slot_: id 0 | task 92 | processing task
slot update_slots: id 0 | task 92 | tokenizing prompt, len = 2
slot update_slots: id 0 | task 92 | prompt tokenized, n_ctx_slot = 8128, n_keep = 0, n_prompt_tokens = 48
slot update_slots: id 0 | task 92 | kv cache rm [0, end)
slot update_slots: id 0 | task 92 | prompt processing progress, n_past = 48, n_tokens = 48, progress = 1.000000
slot update_slots: id 0 | task 92 | prompt done, n_past = 48, n_tokens = 48
slot release: id 0 | task 92 | stop processing: n_past = 48, truncated = 0
slot launch_slot_: id 0 | task 93 | processing task
slot update_slots: id 0 | task 93 | tokenizing prompt, len = 2
slot update_slots: id 0 | task 93 | prompt tokenized, n_ctx_slot = 8128, n_keep = 0, n_prompt_tokens = 54
slot update_slots: id 0 | task 93 | kv cache rm [0, end)
slot update_slots: id 0 | task 93 | prompt processing progress, n_past = 54, n_tokens = 54, progress = 1.000000
slot update_slots: id 0 | task 93 | prompt done, n_past = 54, n_tokens = 54
slot release: id 0 | task 93 | stop processing: n_past = 54, truncated = 0
srv update_slots: all slots are idle
request: POST /rerank 172.30.0.7 200

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💪 enhancement New feature or request 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant