Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTL breaks after computer goes to sleep #3177

Closed
matt2e opened this issue Oct 23, 2024 · 0 comments · Fixed by #3239
Closed

FTL breaks after computer goes to sleep #3177

matt2e opened this issue Oct 23, 2024 · 0 comments · Fixed by #3239
Assignees
Labels

Comments

@matt2e
Copy link
Collaborator

matt2e commented Oct 23, 2024

Steps to repro:

  • Run FTL dev locally, including modules with cron jobs
  • Put computer to sleep. Not sure how long is needed
  • Wake up computer

Expected: FTL continues working as normal
Actual:

  • FTL says that runners are dead after retrying too many times.
  • Cron jobs stop working because refs cant be resolved
info:cron: Frequent cron job triggered.
warn:lease:/system/scheduledtask/reap-async-calls: Lease expired
warn:lease:/system/scheduledtask/reap-call-events: Lease expired
warn:lease:/system/ftl-scaling/runner-creation: Lease expired
error:echo:localScaling: Runner failed too many times, not restarting: too many restarts
error:pubsub:localScaling: Runner failed too many times, not restarting: too many restarts
error:http:localScaling: Runner failed too many times, not restarting: too many restarts
error:fsm:localScaling: Runner failed too many times, not restarting: too many restarts
error:cron:localScaling: Runner failed too many times, not restarting: too many restarts
error:time:localScaling: Runner failed too many times, not restarting: too many restarts
warn:cron:cron:crn-cron-thirtySeconds-4qzorm587vy90f77:cron.thirtySeconds: Async call could not be called: not_found: could not resolve reference cron.thirtySeconds: not found
warn:cron:cron:crn-cron-thirtySeconds-4qzorm587vy90f77:cron.thirtySeconds: Async call could not be called: not_found: could not resolve reference cron.thirtySeconds: not found
warn:cron:cron:crn-cron-thirtySeconds-4qzorm587vy90f77:cron.thirtySeconds: Async call could not be called: not_found: could not resolve reference cron.thirtySeconds: not found
warn:cron:cron:crn-cron-thirtySeconds-4qzorm587vy90f77:cron.thirtySeconds: Async call could not be called: not_found: could not resolve reference cron.thirtySeconds: not found
warn:cron:cron:crn-cron-thirtySeconds-4qzorm587vy90f77:cron.thirtySeconds: Async call could not be called: not_found: could not resolve reference cron.thirtySeconds: not found
warn:cron:cron:crn-cron-thirtySeconds-4qzorm587vy90f77:cron.thirtySeconds: Async call could not be called: not_found: could not resolve reference cron.thirtySeconds: not found
@github-actions github-actions bot added the triage Issue needs triaging label Oct 23, 2024
@ftl-robot ftl-robot mentioned this issue Oct 23, 2024
@alecthomas alecthomas added the P2 label Oct 28, 2024
@alecthomas alecthomas added P1 and removed P2 labels Oct 28, 2024
@github-actions github-actions bot removed the triage Issue needs triaging label Oct 28, 2024
stuartwdouglas added a commit that referenced this issue Oct 30, 2024
When your laptop goes to sleep we were seeing lots of runner restarts using a cancelled lease context.

It is hard to reproduce reliably, but this should work around it for now.

I could not reproduce the issue by manually cancelling the lease, it only happens when the computer hibernates.

fixes: #3177
stuartwdouglas added a commit that referenced this issue Oct 30, 2024
When your laptop goes to sleep we were seeing lots of runner restarts using a cancelled lease context.

It is hard to reproduce reliably, but this should work around it for now.

I could not reproduce the issue by manually cancelling the lease, it only happens when the computer hibernates.

fixes: #3177
stuartwdouglas added a commit that referenced this issue Nov 6, 2024
When your laptop goes to sleep we were seeing lots of runner restarts
using a cancelled lease context.

It is hard to reproduce reliably, but this should work around it for
now.

I could not reproduce the issue by manually cancelling the lease, it
only happens when the computer hibernates.

fixes: #3177
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants