Set MACAddressPolicy=none for bridges/bonds/teams

Describe the bug
After creating bond interface via NetworkManager(nmcli, nmtui) it has autogenerated mac address.

Reproduction steps
Steps to reproduce the behavior:

1. Create bond via NetworkManager with at least 2 slaves
2. Inspect bond mac address

Expected behavior
Macaddress of bond interface is one of (first active) slaves.

Actual behavior
After creating bond interface via NetworkManager(nmcli, nmtui) it has autogenerated mac address , so this breaks IP address assigning via dhcp in OKD.

System details

• Bare Metal and QEMU
• Fedora CoreOS version : 34.20210725.3.0, 33.20210217.3.0

Ignition config

Additional information

[darkmuggle]

This is not a bug with either FCOS or Network-Manager. The default behavior is to use a random MAC address. If you require a specific Mac, you will need to configure it via nmcli or nmtui.

[dustymabe]

I guess it depends on how things are configured. According to https://wiki.linuxfoundation.org/networking/bonding#where_does_a_bonding_device_get_its_mac_address_from it should be the MAC of the first device added to the bond, which matches my previous experience here as well (from a previous lifetime before NetworkManager existed). I've played around with the various ethernet.cloned-mac-address NM settings trying to get it to use a non-autogenerated MAC for the bond (i.e. one derived from the subordinate devices) but haven't been able to.

Maybe @thom311 or @bengal could point us in the right direction by telling us which settings to tweak (or if there is a possible bug lurking).

[dustymabe]

Just checked against RHCOS (RHEL). It behaves the way I'd expect where the bond has the MAC of one of the subordinate devices. Notice that one of the entries below has permaddr and one doesn't:

[core@ignitionhost ~]$ rpm -q NetworkManager kernel
NetworkManager-1.30.0-9.el8_4.x86_64
kernel-4.18.0-305.10.2.el8_4.x86_64
[core@ignitionhost ~]$ 
[core@ignitionhost ~]$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:55:08:fa brd ff:ff:ff:ff:ff:ff
3: ens3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:55:08:fa brd ff:ff:ff:ff:ff:ff permaddr 52:54:00:26:3c:ce
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:55:08:fa brd ff:ff:ff:ff:ff:ff

[bengal]

Since systemd v242, udev sets a predictable MAC address (derived from the machine-id and the interface name) on virtual interfaces, including bond and bridges. As the bond has a MAC address set by userspace, kernel doesn't update the MAC when the first port is added to the bond.

See:

https://github.com/systemd/systemd/blob/v247/NEWS#L2332-L2358

This doesn't happen on RHEL8 because there is an older systemd version there.

To change the MAC of the bond from NetworkManager you should set the ethernet.cloned-mac-address property of the bond connection. That value will override the address set by udev. If this doesn't work, it's probably a bug.

And teaming works right (from my point of view) - takes mac address from first active intreface. But I cant change team runner type in kernel (dracut.cmdline) parameters during pxe installation of coreos. It sets always roundrobin. Bonding works like a charm, but this thing with mac addresses...

[dustymabe]

Thanks @bengal. That's not what I was expecting but certainly is the root cause here. It mentions

if a bridge interface is created without any slaves, and gains
a slave later, then now the bridge does not inherit slave's MAC.

Thinking out loud here:

Would it be an option for NM to create the bond/bridge in this case
with the subordinate devices from the start rather than adding them
subsequently?

@bengal, you mention setting ethernet.cloned-mac-address but there
is no way to set it to say "use subordinate device MAC" without
actually hardcoding that device's MAC address IIUC. Am I right?

@servsav Want to give the following a try to see if you get the behavior you
desire:

# /etc/systemd/network/98-bond-inherit-mac.link
[Match]
Type=bond

[Link]
MACAddressPolicy=none

[bengal]

But I cant change team runner type in kernel (dracut.cmdline) parameters during pxe installation of coreos. It sets always roundrobin.

Changing the team runner from kernel command line was implemented in dracut 51, but only in the network-legacy module:

dracutdevs/dracut@e4483e5

NetworkManager doesn't yet support the parameter. I filed an issue for that: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/774

@servsav Want to give the following a try to see if you get the behavior you
desire:

# /etc/systemd/network/98-bond-inherit-mac.link
[Match]
Type=bond

[Link]
MACAddressPolicy=none

Thanks a lot! This does work as I need!

[thom311]

IMO the default MACAddressPolicy is a serious misfeature on Fedora.
RHEL8 and RHEL9 doesn't do this.

[bengal]

Would it be an option for NM to create the bond/bridge in this case with the subordinate devices from the start rather than adding them subsequently?

No, it's not possible to create the bond with other interfaces already enslaved from the beginning, because in kernel API the master-slave relationship is tracked on the slave through a 'master' attribute. Therefore the master needs to exist initially without any slave, and then ports can be attached by setting their 'master' attribute via netlink.

[bengal]

@bengal, you mention setting ethernet.cloned-mac-address but there is no way to set it to say "use subordinate device MAC" without actually hardcoding that device's MAC address IIUC. Am I right?

Right, there isn't a way. I guess the reasons are:

• before the systemd change this feature wasn't needed; now, depending on how udev is configured[*], it might be useful; however nobody requested it. There were some complaints about this MAC address problem on bridges/bonds but the workaround of disabling MACAddressPolicy was considered enough;

• if NM is going to implement a new e.g. first-port value for the cloned MAC, it means that it won't set the MAC immediately during activation, but only at a later time (when the first port is added); this is something new and requires some thought.

[*] on RHEL, MACAddressPolicy is disabled

[dustymabe]

IMO the default MACAddressPolicy is a serious misfeature on Fedora.
RHEL8 and RHEL9 doesn't do this.

Is this enough of a misfeature that we should try to get it changed by talking with FESCO and submitting a change request?