requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[dkbarn]

Describe the bug
In my application I use spotipy pretty heavily, and frequently I see the error:
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
It happens once every few hours. There are already some retry options exposed in the spotipy.Spotify.__init__, which I am making use of, but this retry functionality seems to only apply to HTTP errors, not low-level ConnectionErrors. Please could we consider adding retry support for any network-related error?

Your code
The error can happen on any one of the public API methods that ultimately results in communicating with Spotify's REST API server.

Expected behavior
The spotipy library doesn't not throw an error during regular use.

Output

  File "/opt/venv/lib/python3.10/site-packages/spotipy/client.py", line 376, in artists
    return self._get("artists/?ids=" + ",".join(tlist))
  File "/opt/venv/lib/python3.10/site-packages/spotipy/client.py", line 297, in _get
    return self._internal_call("GET", url, payload, kwargs)
  File "/opt/venv/lib/python3.10/site-packages/spotipy/client.py", line 221, in _internal_call
    headers = self._auth_headers()
  File "/opt/venv/lib/python3.10/site-packages/spotipy/client.py", line 212, in _auth_headers
    token = self.auth_manager.get_access_token(as_dict=False)
  File "/opt/venv/lib/python3.10/site-packages/spotipy/oauth2.py", line 238, in get_access_token
    token_info = self._request_access_token()
  File "/opt/venv/lib/python3.10/site-packages/spotipy/oauth2.py", line 257, in _request_access_token
    response = self._session.post(
  File "/opt/venv/lib/python3.10/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Environment:

• OS: Linux
• Python version: 3.10.4
• spotipy version: 2.20.0

[FrederikP]

👍

Looking at the code in client.py the retry stuff is initialized like this:

        retry = urllib3.Retry(
            total=self.retries,
            connect=None,
            read=False,
            allowed_methods=frozenset(['GET', 'POST', 'PUT', 'DELETE']),
            status=self.status_retries,
            backoff_factor=self.backoff_factor,
            status_forcelist=self.status_forcelist)

The docs say connect and read should be ints:

:param int connect:
    How many connection-related errors to retry on.

    These are errors raised before the request is sent to the remote server, which we assume has not triggered the server to process the request.

    Set to 0 to fail on the first retry of this type.

:param int read:
    How many times to retry on read errors.

    These errors are raised after the request was sent to the server, so the request may have side-effects.

    Set to 0 to fail on the first retry of this type.

I would really like to see the ability to provide a custom retry object. In my use case I'm seeing connection errors every time I have a longer running process. Probably connections are dropped one sided and we only find out when we try to do the next request.

[jbkze]

This is my workaround:

    def add_tracks_to_playlist(self, tracks: list, playlist_id: str):
        MAX_RETRIES = 10
        retry_count = 0

        for chunk in tqdm(chunk_list(tracks, 100)):
            while retry_count < MAX_RETRIES:
                try:
                    self.spotify.playlist_add_items(playlist_id, chunk)
                    break
                except ConnectionError as e:
                    print(f"Error encountered: {e}")
                    print(f"Retrying... (Attempt {retry_count + 1} of {MAX_RETRIES})")
                    retry_count += 1
                    time.sleep(1)
            else:
                print(f"Failed to add chunk to playlist after {MAX_RETRIES} attempts. Skipping...")
            retry_count = 0