[CORE] Add Gluten Project Improvement Proposals (GPIP) doc

[yikf]

What changes were proposed in this pull request?

This PR aims to add Gluten Project Improvement Proposals doc.

Gluten is growing rapidly, and many major optimizations are expected for it. To follow the Apache way, we should have a specification for major optimizations, and this documentation refers to Spark's SPIP.

How was this patch tested?

doc only

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[yikf]

@zhztheplayer @jackylee-ch When we were doing the #7750, found that the gluten project there is no specifications of significant change, so we want to add this document. cc @zjuwangg

[zhztheplayer]

Thank you for your proposal. Glad to see our development procedure gets more mature over time because of effort like this. +1 for the idea overall.

Additionally, as Gluten is now still under comparatively earlier stage of its lifetime[1], I'll also suggest we slow down a little bit on making something like GPIP a hard restriction of large contribution. I wonder contributor may hesitate on contributing critical features because of the workload of writing a comprehensive proposal. While filing a good document before conducting contribution could be highly encouraged and appreciated.

My key point is that we as a community may lower our posture a bit and thank for all kinds of contributions anyway. It doesn't mean lowering the bar of code review at all, since committers must take responsibility to keep the code healthy. I am just thinking if it's more feasible to let contributors make their own decisions on whether to provide a GPIP or brief information before opening a feature PR. Indeed, providing more information will help a lot on pushing the PR to merge since the reviewers could be clearer on the overall design and purpose of the change.

So at the moment I am thinking if we can remove or rephrase the following words (or similar words from other places), or make a clearer statement on the flexibility on whether to file the document.

When in doubt, if a committer thinks a change needs a GPIP, it does.

Thanks.

[1] Per my personal perspective. Say Velox is still under incubation either.

[yikf]

@zhztheplayer Thanks for your suggestion, I totally agree with you, Let's rephrase the documentation to make the community more healthy and sustainable.

[yikf]

Hi @zhztheplayer , Please take a look if you find a time, thanks

[PHILO-HE]

cc @zhouyuan @FelixYBW @baibaichen

[zhztheplayer]

Would you also mark out the source in the doc, perhaps in the first few of lines? Assuming it's originated from Apache Spark or another project? Thanks!

[zhouyuan]

@yikf Thanks for adding this and I like this idea. +1 from me.
I happen to read some Flink design/features recently and I personally think FLIP is also a good example to follow:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

thanks, -yuan

[yikf]

Would you also mark out the source in the doc, perhaps in the first few of lines? Assuming it's originated from Apache Spark or another project? Thanks!

Do you think it should be as a comment or as the first few lines of documentation?

[yikf]

@yikf Thanks for adding this and I like this idea. +1 from me. I happen to read some Flink design/features recently and I personally think FLIP is also a good example to follow: https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

thanks, -yuan

Thank you for your advice. I am open to both options.

[zhztheplayer]

Would you also mark out the source in the doc, perhaps in the first few of lines? Assuming it's originated from Apache Spark or another project? Thanks!

Do you think it should be as a comment or as the first few lines of documentation?

I think it's better to add to the documentation, namely improvement-proposals.md. Thanks.

[zhztheplayer]

👍

[PHILO-HE]

Looks good. Just some trivial comments. Thanks!

[PHILO-HE]

contribute -> contribution?

[PHILO-HE]

A GPIP

[PHILO-HE]

ditto. Please check all other places.

[PHILO-HE]

what does cross-cutting mean? Use cutting-edge instead?

[PHILO-HE]

Nit. Please break this long line of statements into multiple lines, which is friendly to reading. Please also check other places.

[yikf]

Thanks, addressed, PTAL.

[PHILO-HE]

Thanks!

[FelixYBW]

Thank you for your proposal! Yes, definitely we need better design doc, discusstion for the big features like the stage level resource management. We need to map the features to roadmap.

Also I think it's improtant we also need a single page to track all the proposals like Flink one: under discussion, accepted, released, discard. Looks it's pretty clear.

Meanwhile see if we have similar way to orgnize the issues. Currently we used several trackers like the #4652 but only 3 issues are be pin to top. Is there any good suggestions to do this? Is the cwiki a good way?

[yikf]

Thanks all for help on the PR. @FelixYBW @zhouyuan @PHILO-HE @jackylee-ch .

It does look very clear with a filing and categorization GPIP. A wiki is also a good choice.

Personal opinion, also for reference, is that it would be best to link the archiving and categorization actions to the GPIPs process, so developers don't need to update the archived content based on the lifecycle of GPIPs.

If a GPIP is initiated, an GitHub ISSUE will definitely be created to track it. We can record the link to the ISSUEs category in the GPIP documentation, such as "under discussion," which will link to a list that includes both the GPIP and "under discussion."