Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Fix shuffle spill not reported to spark metric #6740

Merged
merged 4 commits into from
Aug 9, 2024

Conversation

marin-ma
Copy link
Contributor

@marin-ma marin-ma commented Aug 7, 2024

Will have Spill (Memory) and Spill (Disk) shown on UI
image

@github-actions github-actions bot added the VELOX label Aug 7, 2024
Copy link

github-actions bot commented Aug 7, 2024

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

FelixYBW
FelixYBW previously approved these changes Aug 7, 2024
FelixYBW
FelixYBW previously approved these changes Aug 8, 2024
@marin-ma marin-ma force-pushed the fix-spark-spill-metrics branch from 76fc1c8 to 9512482 Compare August 8, 2024 08:54
@marin-ma
Copy link
Contributor Author

marin-ma commented Aug 8, 2024

cc: @zhztheplayer Seems like spill from Velox pipeline is not reported to the TaskMetrics either.

@zhztheplayer
Copy link
Member

zhztheplayer commented Aug 9, 2024

cc: @zhztheplayer Seems like spill from Velox pipeline is not reported to the TaskMetrics either.

Hi @marin-ma, would you like to help file a issue for that? It was a known issue on which we are lack of tracking. Thanks.

@marin-ma
Copy link
Contributor Author

marin-ma commented Aug 9, 2024

Hi @marin-ma, would you like to help file a issue for that? It was a known issue on which we are lack of tracking. Thanks.

@zhztheplayer Filed issue here #6763

@marin-ma marin-ma merged commit 061efb1 into apache:main Aug 9, 2024
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants