Fix the job jitter/delay around sidekiq retries

[kitallis]

Context

This is a pure technical debt requirement.

When retrying jobs via sidekiq_retry_in for example here. We compute a backoff value and then let sidekiq handle the retry. This generally works ok and keeps the code clean. But this is problematic for very large backoff values, which we often have, because the in-built jitter causes them to get very, very delayed.

Requirements & acceptance criteria

Move all Sidekiq job sidekiq_retry_in + sidekiq_retries_exhausted to actually normally rescue exceptions using rescue inside perform, and reenque the job via the backoff value, for example, here.

This will remove the in-built jitter and will ensure a bit more predictable delay in job enqueuing.

It will be interesting, to write a helper / concern or something that allow one to write the retry-reenqueue code easily without having to repeat too much in all jobs. Not sure if possible, worth exploring.

Additional Notes

This problem usually manifests as customer feedback where submission states are updated way too slowly, even though they have changed in App Store / Play Store.

[gitstart-connector]

Issue synced into GitStart. Please review, edit and hand it off to GitStart by following this link.

[gitstart-connector]

This issue has been handed off to GitStart: https://clients.gitstart.com/tramline/11653/tickets/TRAB-687