Meta issue for JSONDecode errors we see often lately

[JOJ0]

Several users have reported similar randomly happening JSONDecode errors when retrieving arbitrary stuff via python3-discogs-client. I experienced it myself and post my experience soon. I'd like to summarize discussion and improvement suggestions in this issue here.

In my opinion it looks like api.discogs.com responds with unexpected data but only sometimes. I would guess it is a problem on their end for whatever reason, and they might be aware of and working on it already, or not..... In the end we have to live with it and find a workaround.

A second reason is that our backoff mechanism could be faulty or not compatible anymore since something might have changed at Discogs. It worked very well for us so far and it's in place since ...about 4 years already....#34

Related reports so far:

• JSON.decode errors #153
• Problems with main release #162

Did any of you solve the issue simply with try/except catchin jsondecode errors? If yes, share you experience, maybe a simple fix like that can be included in python3-discogs-client already.

Can this problem be reproduced when using api.discogs.com simply with Python requests module only?

[JOJ0]

This is what I do to catch such errors in an application using discogs_client. For example this loops through my sales inventory and imports into an sqlite database. Instead of crashing I'd rather have missing entries. This also includes an error counter, since I currently want to learn how often these errors happen.

            for listing in self.collection.me.inventory:
                try:
                    self.collection.create_sales_entry({
                        "d_sales_release_id": listing.release.id,
                        "d_sales_listing_id": listing.id,
                        "d_sales_release_url": listing.release.url,
                    })
                except JSONDecodeError as e:
                    log.error("Catched a JSONDecodeError. Not retrying! %s", e)
                    decode_err += 1
                except Exception as e:
                    log.error("Catched an Exception. Not retrying! %s", e)
                    other_err += 1

        print(f"Discogs JSONDecode errors : {decode_err}.")
        print(f"Other errors : {other_err}.")

I'm not 100% sure since this error is hard to reproduce but I'm pretty sure this must catch JSONDecodeErrors, no matter where they happen!

[JOJ0]

@AnssiAhola do you think we could include the catching of the JSONDecodeError (only! not other exceptions!) into the core of the client?

I'm not sure if simply wrapping the "main fetch wrapper":

discogs_client/discogs_client/fetchers.py

Lines 57 to 61 in 98059c0

	return request(
	method=method, url=url, data=data,
	headers=headers, params=params,
	timeout=(self.connect_timeout, self.read_timeout)
	)

in a try except is the best idea or an approach like the backoff decorator is a better solution:

discogs_client/discogs_client/utils.py

Lines 44 to 48 in 98059c0

	def backoff(f):
	"""
	Wraps the request method of the Fetcher class to provide
	exponential backoff if rate limit is hit.
	"""

or is all this a bad idea anyway? And of course we should add tests for it!