New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate to DuckDB #255

Open

timonv opened this issue Jan 30, 2025 · 0 comments

Labels

Member

timonv commented Jan 30, 2025

Unfortunately, lancedb has several limitations:

Datafusion is huge; both in bin size as in compile times
It only has ANN indexes, and these only work when there are more than 256 rows
Hybrid search requires an ANN index :(
It's hard to explore the data present in lancedb
It's hard to use lancedb for other storage as well

Migrating to duckdb should solve all this, and in addition we could:

Remove redb and use duckdb for caching as well
Use duckdb for other kinds of storage (i.e. LSP dumps)
Combine multiple duckdbs, to bring cross repo context to agents
Start evaluating the indexing/query pipelines (i.e. with ragas; swiftide supports it)

Swiftide integration is being worked on here bosun-ai/swiftide#578

timonv added the enhancement label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment