Skip to content

Commit

Permalink
Add row size estimation support for selective ARRAY and MAP column re…
Browse files Browse the repository at this point in the history
…aders (#10622)

Summary:
Pull Request resolved: #10622

To do estimation on repeated types, we need the method to return not
just the byte size but also the row count of inner data, so that we can get
estimation on the average size of elements in one array or map.

Reviewed By: HuamengJiang

Differential Revision: D60477335

fbshipit-source-id: 2bec760d141106d48814ffd7ed3c24d720cc4122
  • Loading branch information
Yuhta authored and facebook-github-bot committed Aug 7, 2024
1 parent abbeef5 commit 58a80e5
Showing 1 changed file with 9 additions and 5 deletions.
14 changes: 9 additions & 5 deletions velox/dwio/common/SelectiveColumnReader.h
Original file line number Diff line number Diff line change
Expand Up @@ -477,11 +477,15 @@ class SelectiveColumnReader {
template <typename T>
void filterNulls(RowSet rows, bool isNull, bool extractValues);

// Temporary method for estimate in-memory row size (number of bits) of this
// column for Nimble. Will be removed once column statistics are added for
// Nimble.
virtual std::optional<size_t> estimatedRowBitSize() const {
return std::nullopt;
// Temporary method for estimate total in-memory byte size and row count of
// current encoding chunk on this column for Nimble. Will be removed once
// column statistics are added for Nimble. Note that the estimations are
// based on current encoding chunk, so in multi-chunk stripe this is not
// accurate. Other formats should not use this.
virtual bool estimateMaterializedSize(
size_t& /*byteSize*/,
size_t& /*rowCount*/) const {
return false;
}

StringView copyStringValueIfNeed(folly::StringPiece value) {
Expand Down

0 comments on commit 58a80e5

Please sign in to comment.