Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add schema evolution to merge statement #3136

Merged
merged 1 commit into from
Feb 3, 2025

Conversation

JustinRush80
Copy link
Contributor

Description

Add schema evolution (only merge) to the MERGE statement. New columns are added based on the columns predicates in the MERGE operations (eg. target.id = source.id). Using when_not_matched_insert_all and when_matched_update_all will add any new column to the target schema

Related Issue(s)

Documentation

@github-actions github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels Jan 16, 2025
@JustinRush80 JustinRush80 marked this pull request as ready for review January 16, 2025 14:16
@ion-elgreco
Copy link
Collaborator

@JustinRush80 can you rebase your branch against main, or allow us to rebase it

I will do thorough review tomorrow then :)

@ion-elgreco
Copy link
Collaborator

@JustinRush80 could you rebase again, something went wrong since files changed is huge

Copy link

codecov bot commented Jan 18, 2025

Codecov Report

Attention: Patch coverage is 95.04762% with 52 lines in your changes missing coverage. Please review.

Project coverage is 72.18%. Comparing base (100b447) to head (b97cdb8).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/operations/merge/mod.rs 95.59% 3 Missing and 43 partials ⚠️
python/src/merge.rs 0.00% 4 Missing ⚠️
python/src/lib.rs 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3136      +/-   ##
==========================================
+ Coverage   71.74%   72.18%   +0.44%     
==========================================
  Files         138      138              
  Lines       44362    45251     +889     
  Branches    44362    45251     +889     
==========================================
+ Hits        31827    32666     +839     
- Misses      10495    10505      +10     
- Partials     2040     2080      +40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@ion-elgreco ion-elgreco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for picking this up! Just a couple modifications required but looks good so far!

python/deltalake/table.py Outdated Show resolved Hide resolved
python/tests/test_merge.py Show resolved Hide resolved
crates/core/src/operations/merge/mod.rs Outdated Show resolved Hide resolved
crates/core/src/operations/merge/mod.rs Outdated Show resolved Hide resolved
crates/core/src/operations/merge/mod.rs Outdated Show resolved Hide resolved
crates/core/src/operations/merge/mod.rs Outdated Show resolved Hide resolved
crates/core/src/operations/merge/mod.rs Outdated Show resolved Hide resolved
@ion-elgreco
Copy link
Collaborator

@JustinRush80 this is shaping up really nice!

@ion-elgreco
Copy link
Collaborator

@JustinRush80 Great work! Can you squash your commits? After that we are good to go to merge it :)

@JustinRush80 JustinRush80 force-pushed the feat/schema_evo branch 2 times, most recently from 48440db to 0444831 Compare February 3, 2025 01:30
@ion-elgreco ion-elgreco enabled auto-merge February 3, 2025 06:29
@ion-elgreco ion-elgreco added this pull request to the merge queue Feb 3, 2025
Merged via the queue into delta-io:main with commit 0718510 Feb 3, 2025
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Schema evolution on upsert (merge)
2 participants