Skip to content

Commit

Permalink
Passdown fileLength when reading parquet footer to avoid call HDFS Na…
Browse files Browse the repository at this point in the history
…menode in executor side
  • Loading branch information
WangGuangxin committed Dec 18, 2024
1 parent accde83 commit 4053617
Showing 1 changed file with 5 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,11 @@ public static ParquetMetadata readFooter(
.build()
.getMetadataFilter();
}
return readFooter(configuration, file.toPath(), filter);
// The blockReplication and blockSize in FileStatus is useless here, since
// we only need to known the file length when reading parquet footer
FileStatus fileStatus = new FileStatus(
file.fileSize(), false, file.locations().length, 1, file.modificationTime(), file.toPath());
return readFooter(configuration, fileStatus, filter);
}

public static ParquetMetadata readFooter(Configuration configuration,
Expand Down

0 comments on commit 4053617

Please sign in to comment.