[CORE] Update ColumnarShuffleManager

[acvictor]

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

(Fixes: #ISSUE-ID)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[github-actions]

Run Gluten Clickhouse CI

[acvictor]

@ulysses-you removing using private lazy val sortShuffleManager instead of private [this] lazy val sortShuffleManager fixed the disk cleanup issue. I observed the same free disk graph before and after the change.

[acvictor]

@ulysses-you can you please review?

[jackylee-ch]

/Benchmark Velox TPCDS

[ulysses-you]

@ulysses-you removing using private lazy val sortShuffleManager instead of private [this] lazy val sortShuffleManager fixed the disk cleanup issue.

@acvictor Can you elaborate the reason ? why remove [this] fix the issue.

[acvictor]

@ulysses-you removing using private lazy val sortShuffleManager instead of private [this] lazy val sortShuffleManager fixed the disk cleanup issue.

@acvictor Can you elaborate the reason ? why remove [this] fix the issue.

I looked at Spark code but I am not 100% clear. I think object private was affecting clean up.

[acvictor]

/Benchmark Velox TPCDS

[github-actions]

Run Gluten Clickhouse CI

[acvictor]

@jackylee-ch did you observe any regressions?

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[acvictor]

@ulysses-you @jackylee-ch can you please review? I have seen no regression from my testing in Azure.

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.