[VL] Missing Spill (Memory) and Spill (Disk) metrics

[marin-ma]

Backend

VL (Velox)

Bug description

When spill is triggered, the spilled size is not reported to spark task metrics.

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[marin-ma]

Shuffle spill is added in #6740
Spill from velox pipeline is still missing hence the reported spill size can be smaller than the actual spill size.

[FelixYBW]

What's the spill reported from velox? memory spill or disk spill?

[marin-ma]

What's the spill reported from velox? memory spill or disk spill?

@FelixYBW Spill(Memory) is the in-memory raw data to be spilled, usually uncompressed. Spill(Disk) is the actual bytes spilled to disks, equals to the spill file size.
For velox pipeline, we may count Spill(Memory) from the memory allocation: before spill - after spill allocated memory size, and minus the shrunk bytes. And count Spill(Disk) from the spill file size.

[FelixYBW]

What's the current spill size reported from Velox?

[jinchengchenghh]

I have supported the metric in spill, can we get it on the UI? https://github.com/facebookincubator/velox/pull/6076/files