-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug fixes and improved support for Parquet TIMESTAMP #4801
Conversation
0870338
to
a48ea8d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Python changes LGTM.
extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ParquetFileReader.java
Show resolved
Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/transfer/DateTransfer.java
Show resolved
Hide resolved
extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ParquetFileReader.java
Outdated
Show resolved
Hide resolved
extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ParquetFileReader.java
Show resolved
Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/TypeInfos.java
Outdated
Show resolved
Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/transfer/DateTransfer.java
Show resolved
Hide resolved
@@ -981,6 +982,21 @@ public static long epochNanos(@Nullable final ZonedDateTime dateTime) { | |||
return safeComputeNanos(dateTime.toEpochSecond(), dateTime.getNano()); | |||
} | |||
|
|||
/** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In case my comments seem strong, this library is very exposed to users, so it needs to be ultra currated. Functions should only get added when there is a compelling reason.
- I am not a fan of having hard-coded methods for any specific timezone.
- The methods should accept a time zone as an input.
- If methods are added for
LocalDateTime
,LocalDateTime
signatures should be added to all relevant methods.
As part of #4421, we started throwing an exception on reading Parquet TIMESTAMP fields with isAdjustedToUTC set as false. After this change:
java.time.LocalDateTime
java.time.LocalDateTime
columns will be written as Parquet TIMESTAMP fields withisAdjustedToUTC=false
. Earlier they were written as binary data with a codec.Also, this PR fixes the bugs introduced in #4775 and #4755 that can lead to a
ClassCastException
in some cases on reading Parquet DATE and TIME columns.Related to #976