Skip to content

Commit

Permalink
Enhance batch size calculation
Browse files Browse the repository at this point in the history
Signed-off-by: Ansari, Mujammil <[email protected]>
  • Loading branch information
mujammil10 committed Dec 9, 2024
1 parent d04b850 commit a98e02f
Showing 1 changed file with 2 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -118,10 +118,9 @@ private static Future<StorageStatistics> fetchFromStorage(Marker marker, String

private static Future<StorageStatistics> fetchFromStorage(Marker marker, Connector storage, List<String> spaceIds) {

int concurrentRequests = spaceIds.size() > MAX_CONCURRENT_REQUEST_COUNT * MAX_SPACE_BATCH_SIZE ? MAX_CONCURRENT_REQUEST_COUNT
: spaceIds.size() / MAX_SPACE_BATCH_SIZE;
int targetSize = spaceIds.size() > MAX_CONCURRENT_REQUEST_COUNT * MAX_SPACE_BATCH_SIZE
? (int) Math.ceil((double) spaceIds.size() / MAX_CONCURRENT_REQUEST_COUNT) : MAX_SPACE_BATCH_SIZE;

int targetSize = (int) Math.ceil((float) spaceIds.size() / (float) concurrentRequests);
return CompositeFuture.all(Lists.partition(spaceIds, targetSize)
.stream()
.map(batchSpaceIds -> fetchFromStorageInBatches(marker, storage, batchSpaceIds))
Expand Down

0 comments on commit a98e02f

Please sign in to comment.