[GLUTEN-6834][CORE] feat: add other join types from the official Substrait

[EpsilonPrime]

Adds the join types currently defined in Substrait to the Gluten copy. This is one of a vast set of changes aimed at reducing the differences from the official version in hope that one day there would be no differences.

[github-actions]

#6834

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[PHILO-HE]

@FelixYBW, @baibaichen

[zhztheplayer]

I suspect how it actually helps if we make such changes.

For example both Spark and Velox don't have a join type Right Anti at the moment (despite that whether they will add it in future or not). So it may look confusing to have it defined in the interchange protocol.

As commented in #6833 (comment), physical plans would vary more than logical plans in Gluten. We are already having to follow both Spark's and Velox/CH's plan protocols. It could have chance to start messing things up if we strictly follow another one in the middle layer.

[EpsilonPrime]

There are also physical relations in Substrait. The approach I am taking at unforking is to slowly chip away at the differences so we can swap the main instance in seamlessly. A swap of everything at once seems unlikely so I'm tackling what I can -- especially if there are minor code changes related to each small change.

[lgbo-ustc]

May add a new type for existence join ? existence join is transformed to left semi join at present
existence join is supported both at ch and velox now.

[EpsilonPrime]

May add a new type for existence join ? existence join is transformed to left semi join at present existence join is supported both at ch and velox now.

We are in the process of adding new join types to the specification at the moment so adding another is definitely a possibility. (Currently I have a pr out to add mark join.)

[github-actions]

Run Gluten Clickhouse CI

[zhztheplayer]

@EpsilonPrime Before we decide to take effort on migrating to mainstream Substrait, we may do some study on how much it could help us on next moves.

Thing is if we unfork Substrait, we should use it for supporting more backends in Gluten. Otherwise we get less benefit than cost.

Do you happen to know the progress of Substrait integration of some projects, for example, DuckDB, Arrow and Datafusion, if Gluten decide to add support for these libraries, are their Substrait consumer implementations reliable enough for us to use?

[FelixYBW]

As commented in #6833 (comment), physical plans would vary more than logical plans in Gluten. We are already having to follow both Spark's and Velox/CH's plan protocols. It could have chance to start messing things up if we strictly follow another one in the middle layer.

we should have the same protocol in long term, substrait should be able to fit all requirements from different frameworks, libraries and accelerators. It's substrait's goal. But now for Gluten it's not urgent.

[FelixYBW]

@JkSelf @rui-mo Do you happen to have the list of our modifications in Glute's substrait? I remember we have several pending PRs to upstream substrait but there is no active review, so we paused there.

@EpsilonPrime you may start to review the pending PRs in substrait if we have. Add the missing features Gluten needs to upstream.

[rui-mo]

Gluten modifications to Substrait: https://github.com/apache/incubator-gluten/blob/main/docs/developers/SubstraitModifications.md

[EpsilonPrime]

@JkSelf @rui-mo Do you happen to have the list of our modifications in Glute's substrait? I remember we have several pending PRs to upstream substrait but there is no active review, so we paused there.

@EpsilonPrime you may start to review the pending PRs in substrait if we have. Add the missing features Gluten needs to upstream.

All of the pending changes were merged last year (about the time I became a reviewer). I have been also merging changes upstream (for instance the equivalent of TextReadOptions was added last week).

[EpsilonPrime]

@EpsilonPrime Before we decide to take effort on migrating to mainstream Substrait, we may do some study on how much it could help us on next moves.

Thing is if we unfork Substrait, we should use it for supporting more backends in Gluten. Otherwise we get less benefit than cost.

Do you happen to know the progress of Substrait integration of some projects, for example, DuckDB, Arrow and Datafusion, if Gluten decide to add support for these libraries, are their Substrait consumer implementations reliable enough for us to use?

DuckDB has the best Substrait support of the three. Datafusion has a few issues which I'm hoping are addressed in their next release. Acero is in maintenance mode but has a working implementation but it's very strict about what it accepts.

Other benefits include tools which run on Substrait (like the validator and text plan format) which aren't really being used by Gluten at the moment.

The Spark proposal to move the Gluten communication logic (which may or may not have included Substrait) there was the reason I started looking into this effort.

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[zhztheplayer]

It seems CH CI failing at

15:08:17  /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/local-engine/Parser/JoinRelParser.cpp:272:80: error: no member named 'JoinRel_JoinType_JOIN_TYPE_ANTI' in namespace 'substrait'
15:08:17    272 |         if (join_opt_info.is_null_aware_anti_join && join.type() == substrait::JoinRel_JoinType_JOIN_TYPE_ANTI)
15:08:17        |                                                                     ~~~~~~~~~~~^
15:08:17  1 error generated.

[github-actions]

Run Gluten Clickhouse CI