Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client: fix client blocking during garbage collection #25123

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

mismithhisler
Copy link
Member

@mismithhisler mismithhisler commented Feb 14, 2025

Description

This change removes any blocking calls to destroyAllocRunner(), which caused nomad clients to block when running allocations in when shutdown_delay was enabled at the group level. In addition, this change consolidates client GC by removing the MakeRoomFor() method, which is redundant to keepUsageBelowThreshold().

Testing & Reproduction steps

The trick to reproducing this issue is getting MakeRoomFor to trigger a GC. This can be done quickly via client settings of:

gc_max_allocs = 2 (or some low number)
gc_parallel_destroys = 1

MakeRoomFor contends with another method for triggering GC which is AllocStateUpdated, and normal GC timers. Firing periodic jobs every second can get MakeRoomFor to pop an alloc for GC.

job "hello-world" {
  type = "batch"
  
  periodic {
    cron             = "* * * * * * *"
    prohibit_overlap = true
  }

  group "servers" {
    // this at the *group* level is very important for triggering this behavior
    shutdown_delay = "1m"

    task "test" {
      driver = "docker"

      config {
        image   = "alpine:latest"
        command = "/bin/sh"
        args = ["-c", "echo", "123"]
      }

      resources {
        cpu    = 50
        memory = 64
      }
    }
  }
}

Links

Fixes GH #19917

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad website documentation to reflect this. Refer to
    the website README for docs guidelines. Please also consider whether the
    change requires notes within the upgrade guide.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.

This change removes any blocking calls to destroyAllocRunner, which
caused nomad clients to block when running allocations in certain
scenarios. In addition, this change consolidates client GC by removing
the MakeRoomFor method, which is redundent to keepUsageBelowThreshold.
@mismithhisler mismithhisler self-assigned this Feb 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

slow client GC prevents new allocs from running (periodic dispatch)
2 participants