Allow HiveSplit info columns like '$file_size' and '$file_modified_time' to be queried in SQL #8800
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
aditi-pandit
force-pushed
the
hive_file_metadata
branch
from
38310c0 to
b7be310
Compare
aditi-pandit
force-pushed
the
hive_file_metadata
branch
from
b7be310 to
340ffbe
Compare
|
hey @aditi-pandit I also have a similar PR #7880 to let velox support query spark engine supported file metadata for hiveTables (file_path, file_size, file_name, file_modify_time, file_block_start, file_block_end) etc. Maybe we can work together to see if can let the change support for both engine presto and spark ? |
aditi-pandit
commented
Feb 22, 2024
aditi-pandit
force-pushed
the
hive_file_metadata
branch
from
340ffbe to
1974a77
Compare
|
hey @aditi-pandit may change the PR title to |
aditi-pandit
changed the title
aditi-pandit
force-pushed
the
hive_file_metadata
branch
from
1974a77 to
a67125c
Compare
Yuhta
reviewed
Feb 29, 2024
aditi-pandit
force-pushed
the
hive_file_metadata
branch
from
a67125c to
43fce91
Compare
aditi-pandit
force-pushed
the
hive_file_metadata
branch
from
43fce91 to
86ab66c
Compare
|
@Yuhta @majetideepak : PTAL. |
|
@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@Yuhta : Do you need help with the linter error ? Please can you give me more info about it. |
PHILO-HE
added a commit
to PHILO-HE/velox
that referenced
this pull request
Mar 7, 2024
…ified_time' to be queried in SQL (facebookincubator#8800)" This reverts commit b9afa14.
PHILO-HE
added a commit
to PHILO-HE/velox
that referenced
this pull request
Mar 7, 2024
…file_modified_time' to be queried in SQL (facebookincubator#8800)"" This reverts commit d3dc172.
tdcmeehan
reviewed
Mar 7, 2024
| @@ -364,11 +378,13 @@ std::shared_ptr<common::ScanSpec> makeScanSpec( | |||
| // SelectiveColumnReader doesn't support constant columns with filters, | |||
| // hence, we can't have a filter for a $path or $bucket column. | |||
| // | |||
| // Unfortunately, Presto happens to specify a filter for $path or | |||
| // $bucket column. This filter is redundant and needs to be removed. | |||
| // Unfortunately, Presto happens to specify a filter for $path, $file_size, | |||
There was a problem hiding this comment.
Just wondering if there is there an issue for this on Presto side?
Joe-Abraham
pushed a commit
to Joe-Abraham/velox
that referenced
this pull request
Jun 7, 2024
…me' to be queried in SQL (facebookincubator#8800) Summary: $file_size and $file_modified_time are queryable synthesized columns for Hive tables in Presto. Spark also has bunch of such queryable synthesized columns (facebookincubator#7880). The columns are passed by the co-ordinator to the worker in the HiveSplit. i) Velox HiveSplit needed to be enhanced to get filesize and file_modified_time metadata in a generic map data-structure of (column name, value) from Prestissimo. ii) These values should be populated by SplitReader into TableScanOperator output buffers. This also needs a Prestissimo change to populate the HiveSplit with this info sent in the fragment prestodb/presto#21965 Fixes prestodb/presto#21867 gaoyangxiaozhu will have a follow up PR on the Spark integration. Pull Request resolved: facebookincubator#8800 Reviewed By: mbasmanova Differential Revision: D54512245 Pulled By: Yuhta fbshipit-source-id: 190a97f9fcb1e869fff82e0a2264d57f9915376e
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
$file_size and $file_modified_time are queryable synthesized columns for Hive tables in Presto. Spark also has bunch of such queryable synthesized columns (#7880).
The columns are passed by the co-ordinator to the worker in the HiveSplit.
i) Velox HiveSplit needed to be enhanced to get filesize and file_modified_time metadata in a generic map data-structure of (column name, value) from Prestissimo.
ii) These values should be populated by SplitReader into TableScanOperator output buffers.
This also needs a Prestissimo change to populate the HiveSplit with this info sent in the fragment prestodb/presto#21965
Fixes prestodb/presto#21867
@gaoyangxiaozhu will have a follow up PR on the Spark integration.