Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport of fingerprint: initial fingerprint of Vault/Consul should be periodic into release/1.9.x #25112

Conversation

hc-github-team-nomad-core
Copy link
Contributor

Backport

This PR is auto-generated from #25102 to be assessed for backporting due to the inclusion of the label backport/1.9.x.

The below text is copied from the body of the original PR.


In #24526 we updated the Consul and Vault fingerprints so that they are no longer periodic. This fixed a problem that cluster admins reported where rolling updates of Vault or Consul would cause a thundering herd of fingerprint updates across the whole cluster.

But if Consul/Vault is not available during the initial fingerprint, it will never get fingerprinted again. This is challenging for cluster updates and black starts because the implicit service startup ordering may require reloads. Instead, have the fingerprinter run periodically but mark that it has made its first successful fingerprint of all Consul/Vault clusters. At that point, we can skip further periodic updates. The Reload method will reset the mark and allow the subsequent fingerprint to run normally.

Fixes: #25097
Ref: #24526
Ref: #24049

Testing

The following was run on a client agent's host where Nomad had not previously been run:

$ sudo systemctl stop consul
$ sudo systemctl start nomad
$ nomad node status
ID        Node Pool  DC        Name     Class      Drain  Eligibility  Status
847fba46  default    philly-1  client0  multipass  false  eligible     ready
$ nomad node status -verbose 847f | grep consul
$ # no fingerprint

Now start Consul and see that we eventually fingerprint as expected:

$ sudo systemctl start consul
$ # wait 15+ seconds
$ nomad node status -verbose 847f | grep consul
consul.connect                    = true
consul.datacenter                 = dc1
consul.dns.addr                   = 10.37.105.67
consul.dns.port                   = 8600
consul.ft.namespaces              = true
consul.grpc                       = 8502
consul.partition                  = default
consul.revision                   = 5781fc51
consul.server                     = false
consul.sku                        = ent
consul.version                    = 1.18.2+ent
unique.consul.name                = nomad0

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad website documentation to reflect this. Refer to
    the website README for docs guidelines. Please also consider whether the
    change requires notes within the upgrade guide.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.

Overview of commits

@hc-github-team-nomad-core hc-github-team-nomad-core force-pushed the backport/b-25097-fingerprint-consul-vault-periodic/similarly-subtle-cod branch from 7529f8a to 5482b13 Compare February 13, 2025 19:26
@tgross tgross merged commit e373ff3 into release/1.9.x Feb 13, 2025
25 checks passed
@tgross tgross deleted the backport/b-25097-fingerprint-consul-vault-periodic/similarly-subtle-cod branch February 13, 2025 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants