initramfs: support configuring network on DHCP-less platforms

[bgilbert]

On Fedora CoreOS, Ignition performs the first-boot configuration starting from a running vanilla image and fetching the configuration to apply.
In some cases such configuration is coming from a network resource (e.g. a link-local metadata service, or a cloud bucket, or a cluster service like OpenShift MachineConfigServer).
This works fine in most cases, specifically on all platforms where there is a working DHCP and NetworkManager is able to configure the initramfs before Ignition runs.

There are however a few platforms where a machine is expected to auto-configure its own network using some other hints (i.e. not via DHCP):

• DigitalOcean does not provide DHCP for official images (no cloud agents: digitalocean #71 (comment)), network configuration is provided by an HTTP service on a link-local address.
• IBM Cloud Classic does not provide DHCP, network configuration is provided via a config-drive (see [WIP] providers: add experimental initrd network bootstrap afterburn#362).
• OpenNebula may not provide DHCP, in which case network configuration is provided via a config-drive (see Add OpenNebula provider afterburn#478).
• OpenStack may not provide DHCP, in which case network configuration is provided via a config-drive (see CoreOS does NOT configure network form configdrive network_data.json bugs#2645).
• Packet provides DHCP for the machine's public IPv4 address only, and exposes other details (e.g. bonding) via the metadata service.
• VMware may not provide DHCP, in which case we are currently using an Afterburn-specific guestinfo property (see https://github.com/coreos/afterburn/blob/master/docs/usage/initrd-network-cmdline.md#vmware).
• (Possibly some more platforms, likely derived from OpenStack config-drive)

In Container Linux, Afterburn runs in the initramfs, queries the metadata service, and writes networkd units for use by the real root filesystem. (On Packet they're written into /etc on first boot, and on DO they're written into /run on every boot.). This however introduces a lot of other troubles for the normal Ignition flow, like coreos/bugs#2205, and in general cannot work for cluster services like MachineConfigServer.

We'll need better functionality in Fedora CoreOS, so that NetworkManager in initramfs can properly configure the network in those cases, before Ignition runs. We should also account for transitioning such configuration to the real root (with a teardown in-between at the time of root-pivoting).

EDIT(lucab): reworded for clarity and expanded to reference all platforms where we have the same kind of troubles.

[dustymabe]

We could possibly start using nm-cloud-setup for this. See #320.

IIUC nm-cloud-setup talks to NetworkManager via dbus and dynamically applies configuration from cloud providers to the system. This would work for configuring the network in the real root. It currently would not work for applying the configuration in the initramfs (where Ignition runs) because NetworkManager runs in oneshot mode in the initramfs and talking to it via dbus is not an option. Getting NetworkManager to run in non oneshot mode in the initramfs (thus being able to be communicated with via dbus) is currently being worked on, so we'll hopefully get this in the future. The drawback is that currently if someone wants to grab things from remote resources in their Ignition config it won't work (this was a problem in CL and it was never solved).

Summary for short term strategy:

• On DO we can grab the ignition config from the link-local address in the initramfs
• On Packet we can grab the ignition config from the metadata service in the initramfs
• on DO remote references in Ignition configs won't work
• on Packet remote references in Ignition configs to machines via private packet networking won't work
• nm-cloud-setup will run in the real root and apply networking configuration

Longer term we'll hopefully be able to use nm-cloud-setup in the initramfs and get the full networking applied there as well.

We also need the glue code for Packet and DO to be written for nm-cloud-setup. Current implementations for AWS GCP and Azure are here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/master/clients/cloud-setup

[lucab]

There is also nm-initrd-generator which performs a similar lifting of external configuration details in initramfs.
Specifically, it currently knows about DeviceTree and iBFT sources.
See https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/1.26.2/src/initrd.

[cverna]

@thom311 would you be able to help us with this problem and weigh in on the design with your NM expertise ?