[VL] Enable Spark legacy date formatter if spark.sql.legacy.timeParserPolicy is set to 'LEGACY' #7375
Conversation
…cy is set to 'LEGACY'.
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
|
@rui-mo Could you help to take a look please? Thanks. |
| @@ -520,6 +520,11 @@ std::unordered_map<std::string, std::string> WholeStageResultIterator::getQueryC | |||
|
|
|||
| configs[velox::core::QueryConfig::kSparkPartitionId] = std::to_string(taskInfo_.partitionId); | |||
|
|
|||
| // Enable Spark legacy date formatter if spark.sql.legacy.timeParserPolicy is set to 'LEGACY'. | |||
| if (veloxCfg_->get<std::string>(kSparkLegacyTimeParserPolicy, "") == "LEGACY") { | |||
| configs[velox::core::QueryConfig::kSparkLegacyDateFormatter] = "true"; | |||
There was a problem hiding this comment.
Do we need to set this config as false if 'kSparkLegacyTimeParserPolicy' is not 'LEGACY'?
There was a problem hiding this comment.
Thanks for your feedback. It is necessary to do this. Updated.
|
Run Gluten Clickhouse CI |
|
@rui-mo Could you help to merge this PR? Thanks. |
|
@NEUpanning does this work for spark properties |
|
@surnaik No, this PR only works for |
What changes were proposed in this pull request?
Velox#10966 introduced simple date time formatter that aligns with Spark legacy date parsing/formatting behavior. This PR enables Velox simple date time formatter if
spark.sql.legacy.timeParserPolicyis set toLEGACY.Relates Velox issue: #10354
How was this patch tested?
Integration tests with Spark