Skip to content

Commit

Permalink
[CELEBORN-1519] Do not update estimated partition size if it is uncha…
Browse files Browse the repository at this point in the history
…nged

### What changes were proposed in this pull request?

We will not update the estimated partition size if it is unchanged.

### Why are the changes needed?

Celeborn currently triggers an workerinfo update even-though the estimated partition size is not changed. This leads to unnecessary logging and redundant worker info update operations.

Example log -

```
[master-partition-size-updater] WARN org.apache.celeborn.service.deploy.master.clustermeta.AbstractMetaManager - Celeborn cluster estimated partition size changed from 64.0 MiB to 64.0 MiB
```

### Does this PR introduce _any_ user-facing change?

NO

### How was this patch tested?
Existing UT's

Closes apache#2642 from s0nskar/CELEBORN-1519.

Authored-by: Sanskar Modi <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
  • Loading branch information
s0nskar authored and pan3793 committed Jul 25, 2024
1 parent ea6617c commit afe40e6
Showing 1 changed file with 6 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,12 @@ public void updatePartitionSize() {
} else {
estimatedPartitionSize = initialEstimatedPartitionSize;
}

// Do not trigger update is estimated partition size value is unchanged
if (estimatedPartitionSize == oldEstimatedPartitionSize) {
return;
}

LOG.warn(
"Celeborn cluster estimated partition size changed from {} to {}",
Utils.bytesToString(oldEstimatedPartitionSize),
Expand Down

0 comments on commit afe40e6

Please sign in to comment.