Storage services like S3 will return responses with an HTTP 503 error when the number of requests being served exceeds its capacity. We can hit this error when, e.g., reading many parquet files through cudf (quickly):

  File "parquet.pyx", line 324, in pylibcudf.io.parquet.read_parquet
RuntimeError: curl_easy_perform() error near /opt/conda/conda-bld/work/cpp/src/remote_handle.cpp:255(The requested URL returned error: 503)

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html as the default retry policies of boto3 and this article has some general background.

I should be able to take a look at this today. We'll want to retry 429 and 503 codes, and possibly the other 50x codes too.