Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a warning about unique machine-ids #1910

Closed
wants to merge 1 commit into from

Conversation

pvdputte
Copy link

@pvdputte pvdputte commented Jul 6, 2022

No machines, especially controllers, should ever share machine-ids.

Cfr. #1547 (comment)

"The root cause seems to be that the controllers have matching machine IDs which k0s adds as --server-id."

Signed-off-by: pvdputte [email protected]

Description

I've been bitten by intermittent timeouts and general instability in my k0s testing with a HA control plane. It took me a long time to troubleshoot it. Eventually I found out that both my vagrant boxes and VMware templates were not resetting machine-id properly. A warning about the importance of having unique machine-ids when setting up HA could be helpful to others.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

How Has This Been Tested?

  • Manual test
  • Auto test added

Checklist:

  • My code follows the style guidelines of this project
  • My commit messages are signed-off
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • I have checked my code and corrected any misspellings

No machines, especially controllers, should ever share machine-ids.

Cfr. k0sproject#1547 (comment)

"The root cause seems to be that the controllers have matching machine IDs which k0s adds as --server-id."

Signed-off-by: pvdputte <[email protected]>
@pvdputte pvdputte requested a review from a team as a code owner July 6, 2022 13:07
@pvdputte pvdputte requested review from makhov and twz123 July 6, 2022 13:07
Copy link
Contributor

@trawler trawler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The /etc/machine-id file used by kubernetes is generated by the OS kernel when a machine is booted. An identical machine-id would not just cause a problem with Konnektivity but also with services such as DHCP.
In any case, the problem with issue #1547 was caused due to a cloned VM, that was not cleaned properly: #1547 (comment)
Cloning a VM without cleaning up machine-id is a bad practice and has nothing to do with k0s.

@trawler trawler closed this Jul 6, 2022
@pvdputte
Copy link
Author

pvdputte commented Jul 6, 2022

Well, if it wasn't for these Konnectivity issues we were having, we wouldn't have noticed our machine-id issue for a very long time to come.

In fact the problem has been lurking in our images and templates for years. No problems with DHCP in Vagrant/VirtualBox and also no issues with hundreds of instances in the datacenter (static IPs).

If I had been less persistent about troubleshooting while evaluating k0s, I could have just told myself I should use an alternative because it's no good. It was by pure chance I eventually discovered #1547.

I'm adamant about running k0s in production eventually so this was more about adding some potentially helpful 'troubleshooting tips' to the docs. Indeed it has nothing to do with k0s itself, I guess others will just find their way to closed github issues as well then.

@pvdputte
Copy link
Author

Cfr. k0sproject/k0sctl#435

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants