Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat(embeddings): Embeddable derive macro #59

Merged
merged 52 commits into from
Oct 18, 2024
Merged

Feat(embeddings): Embeddable derive macro #59

merged 52 commits into from
Oct 18, 2024

Conversation

marieaurore123
Copy link
Contributor

@marieaurore123 marieaurore123 commented Oct 15, 2024 •
edited
Loading

TODO: RENAME Embeddable trait to ToEmbedabble, ToEmbedTargets, Extract, ExtractEmbeddable, ...

This PR accomplishes:

  • Defines Embeddable trait which selects the portions of a struct that need ot be embedded. The embeddable() method returns a OneOrMany object which returns either one or many items to embed.
  • Write custom macro that derives the Embeddable trait for any user-defined object

@marieaurore123
feat: setup derive macro
b7d146f
@marieaurore123
test: test out writing embeddable macro
5904734
@marieaurore123
test: continue testing custom macro implementation
ee9b5c3
@marieaurore123
feat: macro generate trait bounds
c8c1e9c
@marieaurore123
refactor: split up macro into multiple files
dff0aeb
@marieaurore123
refactor: move macro derive crate inside rig-core
0d10011
@marieaurore123
Merge branch 'main' into feat(embeddings)/derive-macro
982a17b
@marieaurore123
feat: replace embedding logic with new embeddable trait and macro
79754aa
@marieaurore123
refactor: refactor rag examples, delete document embedding struct
3438fab
@marieaurore123
feat: remove document embedding from in memory store
597e6c3
@marieaurore123
revert: revert vector store to main
6c7ab8d
@marieaurore123
Merge branch 'refactor(vector-store)/in-memeory-vector-store' into fe…
d7d2c19 
…at(embeddings)/derive-macro
@marieaurore123
docs: update emebddings builder docstrings
bb712e3
@marieaurore123
refactor: derive macro
efa2b65
@marieaurore123
feat: add error type on embeddable trait
8325164
@marieaurore123
refactor: move embeddings to its own module and seperate embeddable
de022c4
@marieaurore123
refactor: split up macro into more files, fix all imports
220d9fc
@marieaurore123
Merge branch 'feat/embeddings-overhaul' into feat(embeddings)/derive-…
f5e60f5 
…macro
@marieaurore123
feat: handle tools with embeddingsbuilder
5a8c361
@marieaurore123
bug(macro): fix error when embed tags missing
dc89e54
@marieaurore123
style: cargo fmt
ae66d08
@marieaurore123
fix(tests): clippy
4305952
@marieaurore123
docs&revert: revert embeddable trait error type, add docstrings
24e3b98
@marieaurore123
style: cargo clippy
a7dbf6c
@marieaurore123
clippy(lancedb): fix unused function error
886ebcb
@marieaurore123
fix(test): remove useless assert false statement
79dea45
@marieaurore123 marieaurore123 changed the title Feat(embeddings)/derive macro Feat(embeddings): Embeddable derive macro Oct 15, 2024
@cvauclair
Copy link
Contributor

cvauclair commented Oct 16, 2024 •
edited
Loading

@garance-buricatu can you split the PR into two: 1) a PR that removes DocumentEmbeddings, adds the Embeddable trait.. anything not related to the macro; and 2) a PR specifically for the derive macro implementation?

That would make it much easier to review. Thank you!

marieaurore123
marieaurore123 commented Oct 17, 2024
View reviewed changes
rig-core/src/embeddings/embeddable.rs Outdated
pub fn rest(&self) -> Vec<T> {
self.rest.clone()
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I created the methods one and many to create objects of type OneOrMany instead of doing
impl From<T> for OneOrMany<T> and impl From<Vec<T>> for OneOrMany<T> because these are conflicting. See my comment below for why I would use impl From<Vec<T>> for OneOrMany<T> instead of impl TryFrom<Vec<T>> for OneOrMany<T>.

cvauclair
cvauclair requested changes Oct 17, 2024
View reviewed changes
Copy link
Contributor

@cvauclair cvauclair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not done reviewing, but there's a lot of stuff to change so here is the first part

rig-core/Cargo.toml Outdated Show resolved Hide resolved
rig-core/src/embeddings/mod.rs Outdated Show resolved Hide resolved
rig-core/examples/rag_dynamic_tools.rs Outdated Show resolved Hide resolved
rig-core/examples/rag.rs Outdated Show resolved Hide resolved
rig-core/examples/calculator_chatbot.rs Outdated Show resolved Hide resolved
rig-core/src/embeddings/embeddable.rs Outdated Show resolved Hide resolved
rig-core/src/embeddings/embeddable.rs Show resolved Hide resolved
rig-core/src/embeddings/embeddable.rs Outdated Show resolved Hide resolved
rig-core/src/embeddings/embeddable.rs Outdated Show resolved Hide resolved
rig-core/src/embeddings/embeddable.rs Outdated Show resolved Hide resolved
cvauclair
cvauclair requested changes Oct 17, 2024
View reviewed changes
rig-core/tests/embeddable_macro.rs Show resolved Hide resolved
rig-core/rig-core-derive/src/basic.rs Outdated Show resolved Hide resolved
rig-core/rig-core-derive/src/basic.rs Outdated Show resolved Hide resolved
rig-core/rig-core-derive/src/custom.rs Outdated Show resolved Hide resolved
rig-core/rig-core-derive/src/custom.rs Outdated Show resolved Hide resolved
cvauclair
cvauclair requested changes Oct 18, 2024
View reviewed changes
Copy link
Contributor

@cvauclair cvauclair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent! Just a tiny change to the OneOrMany error definition and then you can merge :)

rig-core/src/one_or_many.rs Outdated Show resolved Hide resolved
@marieaurore123 marieaurore123 merged commit 4039cf6 into feat/embeddings-overhaul Oct 18, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cvauclair cvauclair cvauclair requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@marieaurore123 @cvauclair