Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IPv6 ULA address is randomly generated on virtual interface of master and/or backup node #2551

Open
detzen opened this issue Feb 24, 2025 · 3 comments
Open

IPv6 ULA address is randomly generated on virtual interface of master and/or backup node #2551

detzen opened this issue Feb 24, 2025 · 3 comments

Comments

@detzen
Copy link

detzen commented Feb 24, 2025 •
edited
Loading

I have a VRRP setup with keepalived on 2 nodes. I use 2 VRRP instances (one for IPv4 and another for IPv6) with virtual MAC addresses. The instance for IPv4 works as desired, but the IPv6 related instance misbehaves which leads to network issues:
Shortly after master and backup election takes place, an IPv6 ULA address on the backup nodes virt. macvlan interface appears and sends out e.g. neighbor advertisements using the virtual MAC address which causes MAC flapps on the switch where both nodes are connected. Here you can see the relevant output of ip address show for both nodes.

Master node:

2: eth0@if182: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:53:eb:97 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.222.196/24 brd 192.168.222.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 20xx:xxxx:2de0::196/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe53:eb97/64 scope link
       valid_lft forever preferred_lft forever
9: vrrp.4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:5e:00:01:04 brd ff:ff:ff:ff:ff:ff
    inet 192.168.222.199/24 brd 192.168.222.255 scope global vrrp.4
       valid_lft forever preferred_lft forever
10: vrrp.6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:5e:00:02:06 brd ff:ff:ff:ff:ff:ff
    inet6 20xx:xxxx:2de0::199/64 scope global nodad
       valid_lft forever preferred_lft forever
    inet6 fe80::dead:beef/64 scope link nodad
       valid_lft forever preferred_lft forever

Backup node:

2: end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:1e:06:48:e4:bb brd ff:ff:ff:ff:ff:ff
    inet 192.168.222.248/24 brd 192.168.222.255 scope global end0
       valid_lft forever preferred_lft forever
    inet6 20xx:xxxx:2de0::248/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::21e:6ff:fe48:e4bb/64 scope link
       valid_lft forever preferred_lft forever
36: vrrp.4@end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:5e:00:01:04 brd ff:ff:ff:ff:ff:ff
37: vrrp.6@end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:5e:00:02:06 brd ff:ff:ff:ff:ff:ff
    inet6 fd04:db37:bcb0:f08c:200:5eff:fe00:206/64 scope global dynamic mngtmpaddr
       valid_lft 1775sec preferred_lft 1775sec

To Reproduce
Start keepalived and wait a while. After a short time, an IPv6 ULA adress on the vrrp. interface appears. and sends In my case, it's fd04:db37:bcb0:f08c:200:5eff:fe00:206/64

Expected behavior
I would expect that there is no IPv6 address on the backup node's MacVLAN virtual interface, as is the case immediately after starting keepalived
e.g.:

36: vrrp.4@end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:5e:00:01:04 brd ff:ff:ff:ff:ff:ff
37: vrrp.6@end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:5e:00:02:06 brd ff:ff:ff:ff:ff:ff

Keepalived version
Master

/usr/sbin/keepalived -v
Keepalived v2.2.7 (01/16,2022)

Copyright(C) 2001-2022 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 5.19.11
Running on Linux 6.8.12-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z)
Distro: Debian GNU/Linux 12 (bookworm)

configure options: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --enable-snmp --enable-sha1 --enable-snmp-rfcv2 --enable-snmp-rfcv3 --enable-dbus --enable-json --enable-bfd --enable-regex --with-init=systemd build_alias=x86_64-linux-gnu CFLAGS=-g -O2 -ffile-prefix-map=/build/keepalived-m8ENAG/keepalived-2.2.7=. -fstack-protector-strong -Wformat -Werror=format-security LDFLAGS=-Wl,-z,relro CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2

Config options:  NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 DBUS INIT=systemd SYSTEMD_NOTIFY

System options:  VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE VRF SO_MARK

Backup

Keepalived v2.2.7 (01/16,2022)

Copyright(C) 2001-2022 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 5.19.11
Running on Linux 6.1.0-31-arm64 #1 SMP Debian 6.1.128-1 (2025-02-07)
Distro: Debian GNU/Linux 12 (bookworm)

configure options: --build=aarch64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/aarch64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --enable-snmp --enable-sha1 --enable-snmp-rfcv2 --enable-snmp-rfcv3 --enable-dbus --enable-json --enable-bfd --enable-regex --with-init=systemd build_alias=aarch64-linux-gnu CFLAGS=-g -O2 -ffile-prefix-map=/build/keepalived-wSlR37/keepalived-2.2.7=. -fstack-protector-strong -Wformat -Werror=format-security LDFLAGS=-Wl,-z,relro CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2

Config options:  NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 DBUS INIT=systemd SYSTEMD_NOTIFY

System options:  VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE VRF SO_MARK

Distro (please complete the following information):

  • Name: Debian
  • Version: 12 (bookworm)
  • Architecture: x86_64 and ARM64

Configuration file:
Master

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 45.132.246.102
   smtp_connect_timeout 30
   vrrp_skip_check_adv_addr
   vrrp_version 3
}

vrrp_sync_group VI_46 {
    group {
        VI_4
        VI_6
    }
}

vrrp_instance VI_4 {
    state MASTER
    notify /etc/keepalived/keepalived_notify
    interface eth0
    virtual_router_id 4
    use_vmac
    vmac_xmit_base
    priority 255
    advert_int 1
    virtual_ipaddress {
        192.168.222.199/24 brd 192.168.222.255
    }
}

vrrp_instance VI_6 {
    state MASTER
    notify /etc/keepalived/keepalived_notify
    interface eth0
    virtual_router_id 6
    use_vmac
    vmac_xmit_base
    priority 255
    advert_int 1
    virtual_ipaddress {
        fe80::dead:beef/64
        20xx:xxxx:2de0::199/64
    }
}

Backup

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 45.132.246.102
   smtp_connect_timeout 30
   vrrp_skip_check_adv_addr
   vrrp_version 3
}

vrrp_sync_group VI_46 {
    group {
        VI_4
        VI_6
    }
    notify /etc/keepalived/keepalived_notify
    smtp_alert
}

vrrp_instance VI_4 {
    state BACKUP
    notify /etc/keepalived/keepalived_notify
    interface end0
    virtual_router_id 4
    priority 250
    use_vmac
    vmac_xmit_base
    advert_int 1
    virtual_ipaddress {
        192.168.222.199/24 brd 192.168.222.255
    }
}

vrrp_instance VI_6 {
    state BACKUP
    notify /etc/keepalived/keepalived_notify
    interface end0
    virtual_router_id 6
    use_vmac
    vmac_xmit_base
    priority 250
    advert_int 1
    virtual_ipaddress {
        fe80::dead:beef/64
        20xx:xxxx:2de0::199/64
    }
}

Notify and track scripts
/etc/keepalived/keepalived_notify

#!/bin/bash

# example output:
# INSTANCE:VI_6:BACKUP or INSTANCE:VI_6:MASTER

echo $1:$2:$3 > /var/run/keepalived.$1.$2.state

System Log entries
Master

Feb 24 20:40:02 adguard01 systemd[1]: Starting keepalived.service - Keepalive Daemon (LVS and VRRP)...
Feb 24 20:40:02 adguard01 Keepalived[15244]: Starting Keepalived v2.2.7 (01/16,2022)
Feb 24 20:40:02 adguard01 Keepalived[15244]: Running on Linux 6.8.12-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z) (built for Linux 5.19.11)
Feb 24 20:40:02 adguard01 Keepalived[15244]: Command line: '/usr/sbin/keepalived' '--dont-fork'
Feb 24 20:40:02 adguard01 Keepalived[15244]: Configuration file /etc/keepalived/keepalived.conf
Feb 24 20:40:02 adguard01 Keepalived[15244]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Feb 24 20:40:02 adguard01 Keepalived[15244]: Starting VRRP child process, pid=15245
Feb 24 20:40:02 adguard01 systemd[1]: keepalived.service: Got notification message from PID 15245, but reception only permitted for main PID 15244
Feb 24 20:40:02 adguard01 Keepalived_vrrp[15245]: Script user 'keepalived_script' does not exist
Feb 24 20:40:02 adguard01 Keepalived_vrrp[15245]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Feb 24 20:40:02 adguard01 Keepalived_vrrp[15245]: use_vmac or no_accept/strict specified, but no firewall configured - using nftables
Feb 24 20:40:02 adguard01 Keepalived_vrrp[15245]: NOTICE: setting sysctl net.ipv4.conf.default.rp_filter from 0 to 2
Feb 24 20:40:02 adguard01 Keepalived_vrrp[15245]: NOTICE: setting sysctl net.ipv4.conf.all.rp_filter from 2 to 0
Feb 24 20:40:02 adguard01 Keepalived[15244]: Startup complete
Feb 24 20:40:02 adguard01 systemd[1]: Started keepalived.service - Keepalive Daemon (LVS and VRRP).
Feb 24 20:40:02 adguard01 Keepalived_vrrp[15245]: (VI_4) Entering MASTER STATE
Feb 24 20:40:02 adguard01 Keepalived_vrrp[15245]: (VI_6) Entering MASTER STATE

Backup

Feb 24 20:39:43 c4 systemd[1]: Starting keepalived.service - Keepalive Daemon (LVS and VRRP)...
Feb 24 20:39:43 c4 Keepalived[426282]: Starting Keepalived v2.2.7 (01/16,2022)
Feb 24 20:39:43 c4 Keepalived[426282]: Running on Linux 6.1.0-31-arm64 #1 SMP Debian 6.1.128-1 (2025-02-07) (built for Linux 5.19.11)
Feb 24 20:39:43 c4 Keepalived[426282]: Command line: '/usr/sbin/keepalived' '--dont-fork'
Feb 24 20:39:43 c4 Keepalived[426282]: Configuration file /etc/keepalived/keepalived.conf
Feb 24 20:39:43 c4 Keepalived[426282]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Feb 24 20:39:43 c4 Keepalived[426282]: Starting VRRP child process, pid=426283
Feb 24 20:39:43 c4 systemd[1]: keepalived.service: Got notification message from PID 426283, but reception only permitted for main PID 426282
Feb 24 20:39:43 c4 Keepalived_vrrp[426283]: Script user 'keepalived_script' does not exist
Feb 24 20:39:43 c4 Keepalived_vrrp[426283]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Feb 24 20:39:43 c4 Keepalived_vrrp[426283]: use_vmac or no_accept/strict specified, but no firewall configured - using nftables
Feb 24 20:39:43 c4 Keepalived[426282]: Startup complete
Feb 24 20:39:43 c4 systemd[1]: Started keepalived.service - Keepalive Daemon (LVS and VRRP).
Feb 24 20:39:43 c4 Keepalived_vrrp[426283]: (VI_4) Entering BACKUP STATE (init)
Feb 24 20:39:43 c4 Keepalived_vrrp[426283]: (VI_6) Entering BACKUP STATE (init)
Feb 24 20:39:57 c4 Keepalived_vrrp[426283]: (VI_6) Entering MASTER STATE
Feb 24 20:39:57 c4 Keepalived_vrrp[426283]: VRRP_Group(VI_46) Syncing instances to MASTER state
Feb 24 20:39:57 c4 Keepalived_vrrp[426283]: (VI_4) Entering MASTER STATE
Feb 24 20:40:02 c4 Keepalived_vrrp[426283]: (VI_4) Master received advert from 192.168.222.196 with higher priority 255, ours 250
Feb 24 20:40:02 c4 Keepalived_vrrp[426283]: (VI_4) Entering BACKUP STATE
Feb 24 20:40:02 c4 Keepalived_vrrp[426283]: VRRP_Group(VI_46) Syncing instances to BACKUP state
Feb 24 20:40:02 c4 Keepalived_vrrp[426283]: (VI_6) Entering BACKUP STATE

Did keepalived coredump?
No
@pqarmitage
Copy link
Collaborator

The ULA address fd04:db37:bcb0:f08c:200:5eff:fe00:206/64 added on vrrp.6 on the backup node is not added by keepalived. keepalived adds IPv6 addresses with nodad configured, in order to avoid delays in the address becoming active. The ULA address added to vrrp.6 is derived from the MAC address of the interface, has mngtmpaddr set and a lifetime of 1775 seconds, none of which keepalived sets, unless it is configured to do so.

I have run your configurations in a Debian 12 VM and am not experiencing the problem of an added ULA, so I suspect that it is some process running on the backup node that is adding the address - perhaps when it notices the last ULA being removed from the interface.

@pqarmitage
Copy link
Collaborator

https://linux.debian.user.narkive.com/5RQyAQbd/mngtmpaddr-ip might give some clues about what is happening.

@detzen
Copy link
Author

detzen commented Feb 24, 2025 •
edited
Loading

Thanks for your reply.
I'm not aware of a process running on the backup node which add an ULA on an interface and in the meanwhile I also saw that the address was also added on the master node.

keepalived adds IPv6 addresses with nodad configured,
Thanks, I'll further investigate on OS level and hope to find the root cause.

edit:

https://linux.debian.user.narkive.com/5RQyAQbd/mngtmpaddr-ip might give some clues about what is happening.
I'll check and provide an update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pqarmitage @detzen