Should the deletion files share same read options as the data file?

Iceberg allows the format of delete file and data file to be different, but in most cases they are consistent.

add cases for multi deletions?

+1. At least, should cover UPDATE operation. MergeInto is nice to have.

@yma11 @YannByron While adding test cases, I discovered a bug in the Velox code. I will fix this bug first before updating the current PR. Later on, I will add test cases for MERGE INTO and UPDATE, as well as multiple DELETE operations.

Possible to pass deleteFiles without this map?

For normal SplitInfo, file information is stored in lists, it is unable to obtain the mapping relationship between the data file and the delete file. I have actually considered placing the delete file map under protobuf's LocalFiles. The current processing logic is to extract the map information into the IcebergReadOptions of each FileOrFiles in Java, and then combine them back into a map in C++. It is actually somewhat redundant, but this approach has the least impact on the current protobuf changes.
We can also add a new definition for table_format in LocalFiles, with the default being hive. It can also be hive-iceberg. When the format is hive-iceberg, the delete file map contained within will be read.

In current code, we put all the fields like paths, starts, lengths, etc of all files for each task together, with each as a list. I think we can refactor it by using a single list of file or split which contains path, start, length, a list of deleteFiles so that we won't need this map. But how will it affect the protobuf part? any idea?

@yma11 This is also a feasible solution, but not all data files have a delete file, and some data files may have multiple delete files. We would need a two-dimensional vector to maintain this relationship, and the two-dimensional vector would need to have empty vector for data files that do not have a corresponding delete file. Do you think this is a better solution? I'm OK to change to using this approach.

Yeah. scanInfo will have a list of SplitInfo and each SplitInfo contains its own path, start, length as well as a list of deleteFiles which may be empty. I think it will be clear in conception and without using idx anywhere. Spark-delta has a similar structure called AddFile which organizes like this. It's okay to merge this PR first and we do it as a follow up.

When will it happen that it's not a IcebergSplitInfo?

Since substrait::ReadRel_LocalFiles_FileOrFiles contains multiple files, each file will enter this function during iteration. The first time it enters as SplitInfo, and subsequently, it is replaced by IcebergSplitInfo.

maybe a stupid question: why it skips 3 and 4?

I removed some redundant fields and forgot to update the sequence numbers; it shouldn't skip 3 and 4.