Spark 3.5: Display write metrics on SQL UI #11340
Conversation
manuzhang
force-pushed
the
spark-write-metrics
branch
3 times, most recently
from
8a561bd to
316dc40
Compare
|
Add
|
manuzhang
force-pushed
the
spark-write-metrics
branch
from
316dc40 to
99d4b65
Compare
|
|
||
| object MetricsUtils { | ||
|
|
||
| def postDriverMetrics(sparkContext: SparkContext, metricValues: java.util.Map[CustomMetric, Long]): Unit = { |
There was a problem hiding this comment.
This is not needed if we can support it at Spark side.
There was a problem hiding this comment.
Yes, but it will only be available in Spark 4+
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
manuzhang
force-pushed
the
spark-write-metrics
branch
from
99d4b65 to
f91d3f5
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
@manuzhang I am happy to see that someone is working on adding write-side Iceberg metrics to the Spark SQL UI! |
|
|
||
| @Override | ||
| public String description() { | ||
| return "number of total data files"; |
There was a problem hiding this comment.
This doesn't sound right.
"total" implies a number, so I think the description can just be "total data files".
If you really want to include "number", then "total number of data files".
|
@wypoon I plan to add metrics for all write operations, but I'd like to get the interfaces right at first. I'm not sure whether this is the best way to propagate a if (this.table instanceof BaseTable) {
this.metricsReporter = new InMemoryMetricsReporter();
((BaseTable) this.table).combineMetricsReporter(metricsReporter);
} |
No description provided.