Add Parquet user-defined predicate to push down StartsWith

[rdblue]

Parquet cannot currently push down startsWith predicates because it doesn't have a built-in implementation. Iceberg should add a user-defined predicate to push it down.

[waterlx]

Hi @rdblue has this issue addressed by PR #327 ? Could you please close this issue?

[aokolnychyi]

I think #327 provided a basis and #398 implemented startsWith in our dictionary and metrics filters in Parquet. @rdblue, is this what you mean by this issue?

[maqroll]

Sorry to jump in but maybe this issue refers to handle STARTS_WITH case with an UserDefinedPredicate here.

[rdblue]

When I opened this issue, I was thinking about the old read path. That's the one that @maqroll correctly pointed to. But we now use the new read path and the row group filters are ParquetMetricsRowGroupFilter and ParquetDictionaryRowGroupFilter. We needed to add support to those classes for the startsWith predicate added by #327, which was done by @aokolnychyi in #398.

I think the only remaining task for this (and for IN/NOT_IN) is to update the conversion from Spark filters to use the new Iceberg expressions.

[rdblue]

I'm going to close this one since I think it covers a case that we don't really need to worry about.

[bachar84]

In my humble opinion, issue #327 is the roots of persecution

[rdblue]

In my humble opinion, issue #327 is the roots of persecution

I'm not sure I understand you. What do you mean?

[aokolnychyi]

I think SparkFilters already converts startsWith correctly. However, we need to update it to use new expressions for IN as it still rewrites it as OR/AND.

[rdblue]

I agree. I opened #748 to track updating SparkFilters.