chore: update to DataFusion 45.0.0 / arrow/parquet 54.0.0

[alamb]

Description

chore: upgrade to DataFusion 45.0.0

Related Issue(s)

• related to Release DataFusion 45.0.0 apache/datafusion#14008

Documentation

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[alamb]

It appears we will also have to update delta kernel with the new arrow first:
#delta_kernel = { version = "=0.6.0", features = ["default-engine"] }

[rtyler]

Yeah, I have this change which I would like to land to also make this easier

[alamb]

Yeah, I have this change which I would like to land to also make this easier

Turns out 5c5c65d makes it failry easy

[codecov]

Codecov Report

Attention: Patch coverage is 37.50000% with 10 lines in your changes missing coverage. Please review.

Project coverage is 72.18%. Comparing base (43edb9d) to head (2a9c991).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
python/src/filesystem.rs	0.00%	5 Missing ⚠️
python/src/lib.rs	0.00%	4 Missing ⚠️
crates/core/src/writer/json.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3175   +/-   ##
=======================================
  Coverage   72.17%   72.18%           
=======================================
  Files         138      138           
  Lines       45282    45292   +10     
  Branches    45282    45292   +10     
=======================================
+ Hits        32681    32692   +11     
- Misses      10533    10538    +5     
+ Partials     2068     2062    -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alamb]

I would say the tests all look good to me -- there looks to be some error with docs, but that doesn't look like any reason to avoid releasing DataFusion. Otherwise things are looking 👌

[ion-elgreco]

I would say the tests all look good to me -- there looks to be some error with docs, but that doesn't look like any reason to avoid releasing DataFusion. Otherwise things are looking 👌

Awesome! Thanks @alamb 🙏

[alamb]

I would say the tests all look good to me -- there looks to be some error with docs, but that doesn't look like any reason to avoid releasing DataFusion. Otherwise things are looking 👌

Awesome! Thanks @alamb 🙏

Thank you for your patience. BTW I still hope to get the fix for this issue into DataFusion 45 as well

• Type Coercion fails for List with inner type struct which has large/view types apache/datafusion#14154

[alamb]

I tried to fix the DCO / conventional commit. We'll see how it does once the tests run

[rtyler]

The git references will prevent us from cutting a release of the rust crates, but I think it's beneficial to land this into main and pile ontop 😄

[alamb]

Nice!

BTW I expect to publish the actual release to crates.io shortly (has the same content as the git branch)

[ion-elgreco]

@alamb I couldn't see, but did your coercion fix land in this release?

[alamb]

@alamb I couldn't see, but did your coercion fix land in this release?

The list is here: https://github.com/apache/datafusion/commits/branch-45/

There are two related fixes that did get in:

• Fix regression list Type Coercion List with inner type struct which has large/view types apache/datafusion#14385
• Do not rename struct fields when coercing types in CASE apache/datafusion#14384