[improve] [broker] Make the estimated entry size more accurate

[poorbarcode]

Motivation

Background

• B1: Broker assumes the entry size is 1 when no publishing after the topic was loaded up at the latest time. Otherwise, it uses the average entries size published during the lastest time-range of the topic loaded up.
  • Issue: the average entry size is not prectice for the catch-up read after a topic is reloaded.
• B2: The feature managedLedgerMaxReadsInFlightSize assumes the entry size is 10kb when it read nothing. Otherwise, it uses the average entries size of the entries read out before.

Issue
Regarding scenario B2, it may cause an error Time-out elapsed while acquiring enough permits on the memory limiter to read from ledger 33, {tenant}/{ns}/persistent/{topic}-partition-0, estimated read size 1937 bytes for 1 entries (check managedLedgerMaxReadsInFlightSizeInMB) when there is hundreds topic being read and the exact entry size is very little.

More accurate solutions

• P0: Use the exact size of the entries that we attempt to read.
• P1: Use the exact size of the ledger that the entries were stored at to estimate the entries' size.
• P2: Use the size of the topic that the entries were stored at to estimate the entries' size.

Modifications

• Rather than use the P2 solution, change the solution to P1, and it will never return an inaccurate value such as 1 or 10k

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

• The feature managedLedgerMaxReadsInFlightSize assumes the entry size is 10kb when it read nothing. Otherwise, it uses the average entries size of the entries read out before.

After #23901 changes this case would be very rare since it would apply only for the first read in case no entries were produced to the topic.

[poorbarcode]

@lhotari

The feature managedLedgerMaxReadsInFlightSize assumes the entry size is 10kb when it read nothing. Otherwise, it uses the average entries size of the entries read out before.

After #23901 changes this case would be very rare since it would apply only for the first read in case no entries were produced to the topic.

Agree, but for the following case, the current solution is better

• There are 50k entries in the ledger
  • 1 entry was read out, the average size estimated is less accurate than the current implementation.
• The topic was sent for 30 days, and the customer changed the schema of the topic.
  • total size/ total count is less accurate than the current implementation
• Additionally, the current solution will never get an inaccurate value such as 1 or 10k, because it always calculates the exact average size for the entries and it will read a unique entry if no entries to read.

[poorbarcode]

/pulsarbot rerun-failure-checks