Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for download via byte stream #20

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

elliotsayes
Copy link

A few problems I encountered that lead me to this particular solution:

  • When downloading in chunks via HTTP Range header, the arweave.net gateway will eventually rate limit your requests, even with large chunks. Gateway then does exponential backoff which can seriously impact download time.
  • Naively wrapping a stream result in WebWorkers and Isolates causes errors
  • dio streams are a lie - it downloads a buffer and wraps it in Stream.value
  • package:http on web still use XHR requests, also without real stream support: Migrate to the fetch API dart-lang/http#595

So to get around all of these issues:

  • Use a single HTTP request to prevent rate limiting
  • Use fetch on web (using fetch_client library instead of ardrive-http.ts to handle the more complex js/dart interop)
  • Use the low level HttpClient on mobile which just works

I've confirmed that the memory usage on web is stable. Not sure how to confirm on mobile

Caveats of this method:

  • Streamed responses do not run in a WebWorker or Dart Isolate
  • Lack of flow control in fetch. Seems that this cannot be fixed: Support for flow control? Zekfad/fetch_client#3
    • This means the possibility of unbounded memory usage, but only in circumstances where the download speed exceeds the save/decrypt speed. In practice this should be very rare (decrypt speed is 100s of MiB/s on my M1 air, disk speed even faster). Could crop up when saving to a slow flash drive or SD card? It may pay to warn users about this.

@thiagocarvalhodev thiagocarvalhodev self-assigned this Mar 21, 2023
@thiagocarvalhodev thiagocarvalhodev added the enhancement New feature or request label Mar 21, 2023
@karlprieb
Copy link
Collaborator

Another thing I noticed is that the code is not retrying using retryDelay and retryStatusCodes.
As you may know, the gateway can limit you or even return a 5xx error temporarily and that's why we retry.

karlprieb
karlprieb previously approved these changes Mar 23, 2023
@karlprieb karlprieb self-requested a review March 24, 2023 13:59
Copy link

@matibat matibat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

@@ -5,7 +5,7 @@ publish_to: none

environment:
sdk: '>=2.18.5 <3.0.0'
flutter: 3.3.9
flutter: 3.7.0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO for @matibat: check what are the downsides of bumping the version for localizations

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYR already fixed a bunch of localisation errors in the ArDrive-web PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants