[VL] Support row type and fix subfield in filter push-down

[rui-mo]

What changes were proposed in this pull request?

Supports the push-down of IsNull and IsNotNull filter for row type.
Filter on subfield cannot be pushed-down, and we put them in the remaining filter.

How was this patch tested?

With unit test.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[rui-mo]

@FelixYBW @PHILO-HE Could you help take a review? Thanks.

[PHILO-HE]

Looks nice! cc @FelixYBW

[zhli1142015]

One question about the lgoic here.
I thought there is already logic to split filters to subfield filters and remaining filters in Velox: https://github.com/facebookincubator/velox/blob/main/velox/connectors/hive/HiveConnectorUtil.cpp#L790
I wonder what's the reason we maintain similar logic here or what's the difference between them. Thanks.

[rui-mo]

One question about the lgoic here. I thought there is already logic to split filters to subfield filters and remaining filters in Velox: https://github.com/facebookincubator/velox/blob/main/velox/connectors/hive/HiveConnectorUtil.cpp#L790 I wonder what's the reason we maintain similar logic here or what's the difference between them. Thanks.

@zhli1142015 Thanks for bringing this up. We were using our implementation from the start since Velox added the function extractFiltersFromRemainingFilter after we implemented it in Gluten. One distinction I see is that Gluten eliminates the filters to be pushed down from the call expression, whereas Velox does not. Before taking use of the Velox implementation, we might need to verify Gluten supported cases and fix the met issue if there is. We can then seamlessly transition to the Velox implementation. @FelixYBW Do you have any suggestion? Thanks.

[FelixYBW]

@zhli1142015 Thanks for bringing this up. We were using our implementation from the start since Velox added the function extractFiltersFromRemainingFilter after we implemented it in Gluten. One distinction I see is that Gluten eliminates the filters to be pushed down from the call expression, whereas Velox does not. Before taking use of the Velox implementation, we might need to verify Gluten supported cases and fix the met issue if there is. We can then seamlessly transition to the Velox implementation. @FelixYBW Do you have any suggestion? Thanks.

Compared to Velox's, Is there any logic we missed?

[FelixYBW]

Thank you, Rui for your fix.

[rui-mo]

Compared to Velox's, Is there any logic we missed?

@FelixYBW I notice Velox supports the pushdown of between and neq, while Gluten does not. But a previous quick trial shows Velox's implementation could create subfield filters that are not supported, which leads to runtime exception. We might need to do a full validation before using it.

[FelixYBW]

@FelixYBW I notice Velox supports the pushdown of between and neq, while Gluten does not. But a previous quick trial shows Velox's implementation could create subfield filters that are not supported, which leads to runtime exception. We might need to do a full validation before using it.

Thank you, Rui. Can you create an enhancement issue to track?

[rui-mo]

@FelixYBW Opened an issue to track #6666. Thanks.