Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: generalized hash trie indexes for relational tuples #1503

Merged
merged 132 commits into from
Oct 29, 2024
Merged

feat: generalized hash trie indexes for relational tuples #1503

merged 132 commits into from
Oct 29, 2024

Conversation

jhellerstein
Copy link
Contributor

Generalized Hash Tries are part of the SIGMOD '23 FreeJoin paper by Wang/Willsey/Suciu. They provide a compressed ("factorized") representation of relations. By operating in the factorized domain, join algorithms can defer cross-products and achieve asymptotically optimal performance.

@jhellerstein @MingweiSamuel
initial GHT implementation. depends on #1245 and #1242
5dc3f61
@jhellerstein @MingweiSamuel
mostly working, some trouble with contains on references
57aab24
@jhellerstein @MingweiSamuel
passing tests
e639095
@jhellerstein @MingweiSamuel
syncing to github
1afb0b7
@jhellerstein @MingweiSamuel
working to_hash_trie and get()
2de3dda
@jhellerstein @MingweiSamuel
move GHT to lattices, complete get() and iter()
b0f8391
@jhellerstein @MingweiSamuel
WIP: Len trait working, Merge trait in progress
d631466
@jhellerstein @MingweiSamuel
Merge now working
8e2c3c8
@jhellerstein @MingweiSamuel
WIP: PartialOrd and Merge compiling, need to be tested
b3398a9
@jhellerstein @MingweiSamuel
WIP: added LatticeOrd, Eq, IsBot, IsTop traits
5e3ddc3
@jhellerstein @MingweiSamuel
WIP: lattice tests passing
a813cc9
@jhellerstein @MingweiSamuel
simplify ergonomics, test in hydroflow
a074bc6
@jhellerstein @MingweiSamuel
WIP: try to get bimorphisms and GHT to coexist
b0fea7a
@jhellerstein @MingweiSamuel
create a metadata trait for the whole GHT, old trait is now GHTNode
7d8ef49
@jhellerstein @MingweiSamuel
Ght wrapper now supports most lattice traits.
1d1c3b5
@jhellerstein @MingweiSamuel
WIP: fix keyed bimorphism
4948f50
@jhellerstein @MingweiSamuel
WIP: bimorphisms now working on GeneralizedHashTrieNode
7fb1140
@jhellerstein @MingweiSamuel
checkpoint -- largely working
1aeb8c0
@jhellerstein @MingweiSamuel
refactor lattices into multiple files and fix tests
fa0f2dd
@jhellerstein @MingweiSamuel
add ght tests
b37fcfe
@jhellerstein @MingweiSamuel
initial GHT implementation. depends on #1245 and #1242
78caf0c
@jhellerstein @MingweiSamuel
move GHT to lattices, complete get() and iter()
b506346
@jhellerstein @MingweiSamuel
complete rebase
6227aa0
@jhellerstein @MingweiSamuel
make clippy happier
36fd1b1
@jhellerstein @MingweiSamuel
add macro tuple! to convert variadic into tuple
1af4e00
@jhellerstein @MingweiSamuel
add tuple! macro to convert variadic to tuple
b673e19
@jhellerstein @MingweiSamuel
clean up tests removing println and asserting correctness
f44c87e
@jhellerstein @MingweiSamuel
generic join working for triangle query
3eadff3
@jhellerstein @MingweiSamuel
test for generic join working well
539173f
@jhellerstein @MingweiSamuel
freejoin functioning
686510b
@cloudflare-workers-and-pages Cloudflare Workers and Pages
Copy link

cloudflare-workers-and-pages bot commented Oct 22, 2024 •
edited
Loading

Deploying hydroflow with  Cloudflare Pages  Cloudflare Pages

Latest commit: 9b5fdab
Status: ✅  Deploy successful!
Preview URL: https://b4562122.hydroflow.pages.dev
Branch Preview URL: https://feature-ght.hydroflow.pages.dev

View logs

MingweiSamuel
MingweiSamuel reviewed Oct 23, 2024
View reviewed changes
Copy link
Member

@MingweiSamuel MingweiSamuel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review for variadics

variadics/src/variadic_collections.rs Outdated Show resolved Hide resolved
variadics_macro/src/lib.rs Show resolved Hide resolved
variadics_macro/README.md Outdated Show resolved Hide resolved
variadics_macro/src/lib.rs Outdated Show resolved Hide resolved
variadics_macro/src/lib.rs Outdated Show resolved Hide resolved
variadics_macro/src/lib.rs Outdated Show resolved Hide resolved
variadics_macro/README.md Show resolved Hide resolved
MingweiSamuel
MingweiSamuel reviewed Oct 25, 2024
View reviewed changes
variadics_macro/src/lib.rs Outdated Show resolved Hide resolved
variadics_macro/README.md Outdated Show resolved Hide resolved
lattices/src/lib.rs Outdated Show resolved Hide resolved
lattices/src/ght_macros.rs Outdated Show resolved Hide resolved
lattices/src/ght_lazy.rs Outdated Show resolved Hide resolved
lattices/src/ght_lazy.rs Outdated Show resolved Hide resolved
lattices/src/ght_lazy.rs Outdated Show resolved Hide resolved
lattices/src/ght_lazy.rs Outdated Show resolved Hide resolved
MingweiSamuel
MingweiSamuel reviewed Oct 25, 2024
View reviewed changes
lattices/Cargo.toml Outdated Show resolved Hide resolved
MingweiSamuel
MingweiSamuel reviewed Oct 25, 2024
View reviewed changes
Copy link
Member

@MingweiSamuel MingweiSamuel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static_height as an associated constant

lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght_test.rs Outdated Show resolved Hide resolved
MingweiSamuel
MingweiSamuel reviewed Oct 25, 2024
View reviewed changes
Copy link
Member

@MingweiSamuel MingweiSamuel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ght.rs

lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
lattices/src/ght.rs Outdated Show resolved Hide resolved
MingweiSamuel
MingweiSamuel reviewed Oct 25, 2024
View reviewed changes
Copy link
Member

@MingweiSamuel MingweiSamuel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest of GHT

Could we move the GHT stuff into its own submodule of lattices?

  • lattices/src
    • ght/
    • mod.rs (ght.rs)
    • lattice.rs
    • colt.rs (? for ght_lazy)
    • macros.rs

something like that does't have to be exactly that, what do you think?

lattices/src/ght_lattice.rs Outdated Show resolved Hide resolved
lattices/src/ght_lattice.rs Outdated Show resolved Hide resolved
lattices/src/ght_lattice.rs Outdated Show resolved Hide resolved
lattices/src/ght_lattice.rs Outdated Show resolved Hide resolved
lattices/src/ght_lazy.rs Outdated Show resolved Hide resolved
MingweiSamuel
MingweiSamuel approved these changes Oct 28, 2024
View reviewed changes
Copy link
Member

@MingweiSamuel MingweiSamuel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some dead code in comments, but looks good to go!

lattices/src/ght/mod.rs Outdated Show resolved Hide resolved
lattices/src/ght/mod.rs Outdated Show resolved Hide resolved
lattices/src/ght/mod.rs Outdated Show resolved Hide resolved
lattices/src/ght/mod.rs Outdated Show resolved Hide resolved
lattices/src/ght/mod.rs Outdated Show resolved Hide resolved
lattices/src/ght/mod.rs Outdated Show resolved Hide resolved
@MingweiSamuel
Copy link
Member

Good to merge as long as only latest-nightly tests are failing

@jhellerstein jhellerstein merged commit f7e740f into main Oct 29, 2024
39 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MingweiSamuel MingweiSamuel MingweiSamuel approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jhellerstein @MingweiSamuel @agiron123