enhance: iterative download data during compaction to reduce memory cost

[tedxu]

See #37234

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tedxu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• internal/datanode/OWNERS [tedxu]
• internal/indexnode/OWNERS [tedxu]
• internal/storage/OWNERS [tedxu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@tedxu E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.39%. Comparing base (427b6a4) to head (18234af).
Report is 14 commits behind head on master.

❌ Your project check has failed because the head coverage (69.39%) is below the target coverage (77.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #39724   +/-   ##
=======================================
  Coverage   69.39%   69.39%           
=======================================
  Files         302      302           
  Lines       27070    27077    +7     
=======================================
+ Hits        18785    18790    +5     
- Misses       8285     8287    +2     

Components	Coverage Δ
Client	∅ <ø> (∅)
Core	69.39% <ø> (+<0.01%)	⬆️
Go	∅ <ø> (∅)

see 1 file with indirect coverage changes