fix: Parquet format output not working in CLI for show commands #25997
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BREAKING CHANGE: The short option -o, previously used for order-by in the table-list command, has been replaced and is now used for the output option.
hiltontj
reviewed
Feb 28, 2025
There was a problem hiding this comment.
He @Karribalu - I have a couple of suggestions in line. I think the breaking change is okay given the prior discussion about it, and that this makes it consistent with other CLIs that output parquet.
Comment on lines
+130
to
+143
| if let Some(path) = output_file_path { | ||
| let mut f = OpenOptions::new() | ||
| .write(true) | ||
| .create(true) | ||
| .truncate(true) | ||
| .open(path) | ||
| .await?; | ||
| f.write_all_buf(&mut bs).await?; | ||
| } else { | ||
| if output_format.is_parquet() { | ||
| Err(Error::NoOutputFileForParquet)? | ||
| } | ||
| println!("{}", String::from_utf8(bs.as_ref().to_vec()).unwrap()); | ||
| } |
There was a problem hiding this comment.
Could you move this code into a helper function since it is re-used in several places.
There was a problem hiding this comment.
It would be good to add a success test case that writes the parquet to a temp file, then reads it, and validates its contents.
Some helpful APIs that would enable that:
- We use the
tempfilecrate for temporary files in tests - There are APIs for reading parquet files into Arrow
RecordBatchs in theparquetcrate - There are helpers for visually asserting on the contents of those record batches in DataFusion, e.g.,
assert_batches_sorted_eq
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BREAKING CHANGE:
The short option -o, previously used for order-by in the table-list command, has been replaced and is now used for the output option.
Closes #25941
Describe your proposed changes here.