Any plan to support schema evolution in Scan?

[rui-mo]

Description

Hi,
We noticed this issue when running Spark UT.
In this UT, a batch whose content is 1.0 | 1.23 | 1.23 was written into a Parquet file with type (DECIMAL(2, 1), DECIMAL(17, 2), DECIMAL(36, 2)), but was then read with type (DECIMAL(3, 2), DECIMAL(18, 3), DECIMAL(37, 3)). In this case, the created Velox plan node is as below, and incorrect result 0.10 | 0.123 | 0.123 was got.

-- TableScan[table: hive_table] -> a:DECIMAL(3,2), b:DECIMAL(18,3), c:DECIMAL(37,3)

We notice in HiveConnector.cpp#L472-L474, the fileType and readerOutputType_ are different, in which fileType is the written type and from file Footer at ReaderBase.h#L92-L93, while the readerOutputType_ is the reading type from the planner. Looks their names are compared to prevent missing column, but the type differences are not handled. Does Velox plan to support such type-casting behavior in Scan node in the future?

@mbasmanova Would you like to take a look on this issue? Thanks!

[rui-mo]

cc @FelixYBW @zhouyuan

[mbasmanova]

@rui-mo Rui, thank you for reporting this gap. We usually refer to this functionality as schema evolution. Table schema may evolve: new columns are added, existing column types are modified. TableScan needs to support this by being able to read the files written before the schema chance. Looks like we are missing support for column type change. It would be nice to add that.

CC: @Yuhta

[Yuhta]

Yes I notice this while we are reviewing #4040. Both names and types (including nested field names) should be able to overwrite from the planner instead of just reading from file. This is definitely something we should add in the future, probably in 2023 H2.

[rui-mo]

@mbasmanova @Yuhta Thanks for your answering! Glad to hear Velox plans to add it. Closing this question.

[zhli1142015]

Hello @mbasmanova and @Yuhta ,
May I ask if any progress for the schema evolution work?
Thanks.

[zhli1142015]

Please ignore this , found the RC of our issue in Gluten side. apache/incubator-gluten#4044