PTFE-1798 handling errors of no such index

[tcarmet]

If the redis instance happens to reboot during the runtime of the runner manager, redis will need for the Migrator().run() function to run to re-create the indexes. Otherwise we will face an interruption of service with the error:

redis.exceptions.ResponseError: runner-manager:runner_manager.models.runner_group.RunnerGroup:index: no such index

The following actions were taken in the code:

• Reduce the index interval from one hour to 15 minutes.
• Add more context to the indexing function, with this information.
• When failing to contact the redis instance, on the healtcheck, schedule a index run for a more dynamic self-healing approach.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.21%. Comparing base (ba90755) to head (c5e8b95).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #617      +/-   ##
==========================================
- Coverage   86.38%   86.21%   -0.17%     
==========================================
  Files          33       33              
  Lines        1381     1386       +5     
==========================================
+ Hits         1193     1195       +2     
- Misses        188      191       +3     

Flag	Coverage Δ
api	63.85% <88.88%> (+0.13%)	⬆️
unit	68.54% <44.44%> (-0.33%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.