Network Management

[ajeddeloh]

Fedora uses NetworkManager for handling network configuration. Container Linux uses networkd. We need to decide on one. We don't want to carry both since that's just twice the maintainence and chance of breakage without much benefit.

NetworkManager is advantageous because it has wider adoption, especially within the Fedora (and RedHat) ecosystem. It is (to my knowledge) generally more stable than networkd. Unfortunately, it's also harder to write config files for. The nmstate project will help significantly (makes the configuation more declarative) but it still lacks the flexibility of networkd's configuration. nmstate would need to be rewritten in some compiled language (i.e. not python) for inclusion in FCOS.

networkd has a configuation format that lends itself nicely to Container Linux today. The ability to "layer" configs works well for having a default that can be overridden for cloud specific changes and user specified changes. This is especially powerful when combined with it's matching rules. It's configuration is very similar to systemd's in general. nmstate has a proposal for templates which would help, but they still aren't as flexible as networkd's configuation. Unfortunately, networkd tends to suffer regressions and isn't as actively maintained as the core of systemd or NetworkManager. It cannot handle config file changes without restarting the service, but that isn't an issue with FCOS since the nodes shouldn't be configured after first boot.

Finally, networkd has fewer dependencies than networkmanager (considering we're already shipping systemd), especially since Fedora enables most features. We could change this and repackage it for FCOS stripping out unneeded features, but that'd be another custom package to carry and maintain.

We don't have any visibilty into how existing CL or FAH are using networkd or NetworkManager (respectively). This makes determining what requirements we have for network configuration hard.

In my opinion, networkd is a better fit for FCOS, even if it is more regression-happy than we'd like. I'm also perhaps a bit biased coming from my CL background.

[mskarbek]

+1 for networkd.

[arithx]

It cannot handle config file changes without restarting the service, but that isn't an issue with FCOS since the nodes shouldn't be configured after first boot.

I don't think we can completely discount this issue as it also can affect us in the initramfs (like DO on CL).

[ajeddeloh]

I don't think we can completely discount this issue as it also can affect us in the initramfs (like DO on CL).

I think that's workable by doing something hacky like using some ip commands in a ExecPreStart to bootstrap enough networking to run coreos-metadata before trying to start systemd-networkd. Hell we could probably even use a network namespace if we didn't want it messing with the host networking at all. @bgilbert might know better though since he's poked at DO more.

[EdDev]

It cannot handle config file changes without restarting the service, but that isn't an issue with FCOS since the nodes shouldn't be configured after first boot.

I understand that this has not been a needed option so far, but with some use cases (like kubevirt), taking down a node is costly and unwanted.
Scenarios where the operator/admin adds a new network for its cluster usage should not interrupt the existing workload. At least, it would be smoother to add the vlan (for example) to the relevant nodes on-the-fly without any interruption.

Networkd vs NM also seems to have long term implications. Beside the shared maintenance burden, we have the need to change/evolve per the changes in requirements.
As more users get into this domain the more requirements they bring with them, the need to change seems to be stronger than ever.

[cgwalters]

This is a tricky discussion without taking into account layers higher level in the stack. Last I heard, OpenStack (at least RDO/RHOS) do not support NetworkManager (and since this is RHEL7, hence require legacy initscripts).

I'm not sure what the status of things is in the Kubernetes world in general - I imagine there are some things taking a hard dependency on NM, but I don't know.

At least, it would be smoother to add the vlan (for example) to the relevant nodes on-the-fly without any interruption.

I think it's an open question though whether such vlans are truly managed by NM - the way I think of it is you have "host networking" and "kubernetes networking" with the latter being a distinct layer.

[yanivlavi]

OpenStack are planning to support Network Manager as initscripts is being deprecated in the Fedora side over time (no added features and only major bug fixes).
Kubernetes networking is mainly overlay virtual networks that depend on the host networking to be defined prior to that, it also only handles the networking within Kube main scope. The host networking is a gap, specifically on the advanced use cases that are added by CRD multi-networks concepts (additional network devices to the main one).

The scope of the NM project is much wider in terms of what is manages and what is plans to manage.
It already allows for example to create OVS bridges for overlay to physical connectivity and SR-IOV pools management.
It also plans to add more advanced standardization like DPDK devices creation and management.

I don't think there is interest on the networkd side to handle this and even if there is the gap is wider.

You also need to keep in mind the the Fedora devs attention is on NM, not networkd, which will mean that Container Linux would need to maintain and test a completely different network stack, which is less stable by nature.

[thom311]

Hi, an NM guy here.

Discussing this topic is welcomed by us. Thank you for raising the issue. But the thread sounds like pressing for a decision quickly. We just recently started discussing missing gaps for adopting NetworkManager. Unsurprisingly, gaps were identified. All we can assure at this point is that these gaps can and will be filled. But I'd rather not have you take my word for it. Why not take the next months to address issues and provide a proof of concept. It seems better to make a decision after showing that NetworkManager actually works well in your environment.

Anyway, if you really want to force a decision now, then at least discuss it in terms of where NetworkManager can be in a near future. For example, claiming that it's "harder to write config files for" seems not a good argument. It's not clear what the exact issues are, or how hard it is to fix them. I'd even claim that this issue is not severe and NetworkManager will easily improve in this regard (e.g. https://bugzilla.gnome.org/show_bug.cgi?id=772414 )

In my opinion, networkd is a great piece of software and does what it does well. Both NetworkManager and networkd tick checkboxes in the feature matrix. If you purely decide based on that, networkd is very attractive: it works well today, why would you even change?

Yaniv elaborated already on the larger picture of why. Let me add, why we think NetworkManager should be used.

Our stance is that NetworkManager provides an API on Linux for configuring networking. Having multiple APIs is a burden. Look at how cockpit does not support networkd (cockpit-project/cockpit#7987). Can you image the effort to add a suitable API to networkd and leverage it from cockpit? And think further, integrating this API with the rest of the Linux ecosystem. It's really about integrating components on Linux. If you use NetworkManager today, you can use GUI, CLI, files-based configuration, ansible, cockpit. They all integrate with each other because they use the same underlying API. In the future, we aim for management systems (e.g. Openstack and libvirt) to also use the same API.

For example, I don't mind when systemd-timesyncd competes with chrony. Clientside NTP is simple from a configuration point of view. Not so a networking API. Integrating with either networkd or NetworkManager is a large effort. And it's almost unfeasable to find a powerful, common abstraction that targets multiple networking APIs. Sure, networkd or something else could take the role standard API. But NetworkManager is a defacto standard API today, it is quite suitable today, and it integrates with many components today. It will be simpler to fix the shortcomings of NetworkManager and focus on providing a suitable API than bending the rest of the ecosystem towards an API that does not even exist yet.

Granted, it's impossible for the generalist to be the perfect solution for all scenarios. But that is what both Linux kernel and systemd do very successfully. They don't only target a server, desktop, or embedded environment. Being a generalist is key to their success.

Regardless of whether NetworkManager will be adopted, NetworkManager will continue improving at becoming a generally suitable API. It's not that a decision needs to be made today. As NetworkManager keeps getting better, the door for adopting it is only opening further.

TL;DR:

• NetworkManager's value is in being well maintained, feature-rich and used widely (also by Fedora)
• It can already be stripped down considerably and configured by dropping in simple configuration files
• At the moment, switching to it could be difficult, but we're willing to work on Container Linux use cases
• We need time to understand Container Linux needs better, perhaps making NetworkManager more systemd-networkd-y along the way
• Before taking a NetworkManager vs networkd decision, can we list main NetworkManager issues and missing functionalities and see if/how we can adapt it in the next NetworkManager releases?

[ashcrow]

Anyway, if you really want to force a decision now, then at least discuss it in terms of where NetworkManager can be in a near future. For example, claiming that it's "harder to write config files for" seems not a good argument.

That's a fair statement. The true issue is more so around passing single configuration that can use hierarchical templates to produce the proper output for different clouds. If nmstate adds in stacking templates that would get us past minimum requirements for the configuration file.

There still is a bit of issue of running the configuration while in initramfs though. I don't think that's a killer, but is pretty limiting as we've moved to ignition.

The "it requires python" thing is also a problem, but nmstate is open to porting to a complied language and there is a workaround for using the python version as a "binary" while the port occurs.

Granted, it's impossible for the generalist to be the perfect solution for all scenarios.

Agreed.

Before taking a NetworkManager vs networkd decision, can we list main NetworkManager issues and missing functionalities and see if/how we can adapt it in the next NetworkManager releases?

I believe we attempted to do that in the last meeting we had. Keep in mind, it's not that "NetworkManager is wrong". Most of us use it on a daily basis and it's great! No matter what the decision is we should still get together over missing functionality to help widen network managers use cases.

[ajeddeloh]

Regarding the configuration: networkd's config is pretty great for our use case. There's literally nothing I'd want changed in terms of the configuration format. A big theme with networkd (and systemd for that matter) configuration is that it's fully decomposable into as many files as you'd like and every part of the configuration supports that. There are no special cases; this makes the configuration very clean. "Layering" configs is a fully supported case and first class citizen.

Supporting the networkd style configuration is somewhat antithetical to nmstate's goals from my understanding (please correct me if I'm misunderstanding). nmstate wants to have the running state of an interface and the configuration for that interface be expressed in the same manner (which is pretty cool). Describing the state of an interface is different than describing rules for what the state should be.

In networkd all the config files are parsed into a bunch of rules that get applied to the interface when interfaces appear to networkd. Networkd doesn't know what the state of the interface should be until the interface appears and the rules are applied. nmstate has a config file that describes exactly what the state should be but that needs to be generated somehow from a more generic source.

In the end, the desired state of the network is a function of some rules and an interface to apply them to. There's a few main components:

1. Configuration describing rules for how the network should be configured when an interface appears
2. Logic for parsing the configuration into a single unified set of rules (aka combining config files)
3. Logic for parsing those rules and determining what the state of the network should be given an interface
4. A way to apply that state to the host

networkd excels at #1 and #2 and combines 3 and 4 into one and the same (afaik networkd doesn't try to determine the ideal state first, it just applies rules until they are all applied). nmstate + nm does a great job at #4. The nmstate template proposal would tackle #3 (correct me if I'm wrong) but templates aren't as expressive or easily tweakable as networkd's configs. Looking at the proposal it looks like #3 would also be manual; we should make that automatic.

Misc thought experiment: what if there was something that took a networkd like set of configs and generate a nmstate config when an interface appeared?

Another difference (not saying this is better or worse, purely different) is that networkd does nothing automatically; everything is very explicit. This means you may need to do manual tweaking yourself but also means there are no surprises. I personally like this and think it goes well with the explicit, declarative nature of CL today, but could be convinced a little magic here and there is ok.

NetworkManager is great and supports a great many things but that also makes me a little nervous. More flexibility and features are not necesarily better in this case. Each feature someone uses is another chance of breakage, which is very painful when you're running automatic updates. Related question: does nm have any stability promises for configs? networkd doesn't make any promises to my knowledge, I'm just curious if NM does.

We also don't want users logging into their machines. The whole idea behind CL and FCOS is you have an Ignition config that defines what your machine should be, and then you don't touch it. The dbus API isn't helpful in this case since changing the config isn't useful. From what I understand (granted I don't understand kubernetes well) the host networking and the networking k8s sets up are non-overlapping, so we should just let each do it's own thing. That is, k8s shouldn't care what the host's networking stack is. @EdDev you talked about kubevirt wanting to make changes, can you elaborate on that?

As someone who doesn't use nm and has now been looking into it a lot, one thing that jumps out at me is since it's (arguable) primary use case is interactively configured the docs on how to configure it via just writing config files don't seem to be as good (comparatively). There's not a lot of examples for setting up nm without using nmcli or a gui tool. (Or maybe I'm just bad at finding them).

Finally, it seems the discussion seems to be "Can we get NetworkManager/nmstate to do enough to fill the role of networkd" and not "which is a better fit for FCOS". If we'd have to shoehorn things into nmstate or NetworkManager, we shouldn't use them. If we could incorporate the things we like from networkd that would be great, but I worry that that'd be a large undertaking to do right.

Sorry if that was a bit ramble-y. I had a lot of thoughts.

[leoluk]

I love NetworkManager for workstations, but I always uninstall it on servers. It has caused a fair amount of downtime for me in the past due to its dynamic nature/autoconfiguring unknown interfaces. Most recently, it suddenly felt responsible for configuring VLAN interfaces added by OpenStack Neutron, breaking an OpenStack cluster. Config management is pretty hard, too. Configuring something like bonded interfaces isn't straight-forward at all.

Yes, some of this can be disabled, but to me it feels much more complex and unpredictable than networkd.

[EdDev]

Describing the state of an interface is different than describing rules for what the state should be.

Right, I thought a template can be handy to cover the rules part without compromising the declarative state nature.

nmstate has a config file that describes exactly what the state should be but that needs to be generated somehow from a more generic source.

nmstate actually does not need the full state, it will take the requested config state and merge it with the current state to create a new full state. So you could define in the config that you are interested in setting the mtu of an interface, by specifying the interface name, type and the mtu itself and nmstate will internally read the full current state of that interface and overwrite the mtu with the one from the config that was input.

1. Configuration describing rules for how the network should be configured when an interface appears
2. Logic for parsing the configuration into a single unified set of rules (aka combining config files)
3. Logic for parsing those rules and determining what the state of the network should be given an interface
4. A way to apply that state to the host

1. nmsate indeed does not attempt to react to a new interface appearing, it assumes the network devices are post init and available, leaving that burden on the NM to handle. Having nmstate triggered when a new device is detected automatically and then processing the templates to generate a config for it is doable, but I would wander if that is really necessary when we could just wait until the system loaded all devices and only then run the "templating" once.
   It is also reasonable to request such an RFE from NM directly, to define template defaults per rules.
2. That is just merging dicts under the hood, if we define it well I see no problem doing that for both templates and state config files. In a simplistic solution, one is appended over the other and in case of collisions the last win. In my view, the challenge is just in the requirements and definitions of how this should work.

@EdDev you talked about kubevirt wanting to make changes, can you elaborate on that?

Kubevirt purpose is to run VM/s as applications on a k8s cluster and is an example of applications that are stronger bound to the env their run on (OS, HW) and more sensitive when moved from one node to another. As of that, changes to node networking settings (create a new bridge, change mtu, add a bond, replace/add an interface or vlan, etc) is preferred to occur without disturbing much (or at all) the applications that run on that node. As an example, adding a VLAN for a secondary network for the pods to consume should not take down the node with all its pods.

There's not a lot of examples for setting up nm without using nmcli or a gui tool.

I think you are referring to the ifcfg files. NM is consuming these config files and while NM is operational, it replaces the initscripts.

I think that the discussion is more towards: short term vs long term solution and supporting new requirements from the node.

It will be useful to provide an example scenario that incorporates most of the requirements you look for and consider a must. Then an estimation can be made to see the cost or provide alternatives for considerations.

[yanivlavi]

I love NetworkManager for workstations, but I always uninstall it on servers.

We have been working with the NetworkManager devs on improving the server use cases in recent Fedora releases. But keep in mind that the use case is initscripts running along side NetworkManager in OpenStack, I'm not sure that networkd running along side NetworkManager/initscripts would be any better.

1. Configuration describing rules for how the network should be configured when an interface appears
2. Logic for parsing the configuration into a single unified set of rules (aka combining config files)

I would appreciate operators use cases for what kinds of smartness they use after the defaults are applied where is decomposition is useful.

NetworkManager is great and supports a great many things but that also makes me a little nervous. More flexibility and features are not necesarily better in this case. Each feature someone uses is another chance of breakage, which is very painful when you're running automatic updates.
We also don't want users logging into their machines.

That doesn't mean due that these advanced use cases can be deprioritized. We are planning to provide a CRD that will lock support to the policies we expose on the cluster level, the intention is not to let customer SSH and do everything.

You should consider that maintaining something that is out of the Fedora eco-system will have maintenance costs that might be greater than improving nm and if nm is not used to solve these use cases than networkd would need to be able to do that.

host networking and the networking k8s sets up are non-overlapping, so we should just let each do it's own thing. That is, k8s shouldn't care what the host's networking stack is.

In the base installation that is true, but in that case ignition should probably only care about a single NIC or bond needed for K8s networking and scope mentioned here is bigger even for the basic use cases.
What we plan is to expose opt-in advanced capabilities for multiple host networks in a smart way from the cluster level, which is sensitive to the host networking stack and has a greater set of feature requirements.

You need to consider a possible near future where ignition does only care about a single NIC or bond needed for K8s networking and the rest is done after the cluster is up via smart cluster policies, providing a day 2 management for the non-base use cases.

[cgwalters]

Most recently, it suddenly felt responsible for configuring VLAN interfaces added by OpenStack Neutron, breaking an OpenStack cluster.

Yeah; see e.g. this bug.

[dcbw]

Note that all OpenShift installations, by default, run NetworkManager. That includes OpenShift Online (literally hundreds of nodes supporting tens of thousands of users). That is a huge number of systems that have proven that NetworkManager is capable of supporting OpenShift, Kubernetes, and container use-cases without interference or problems. In my 3+ years working with OpenShift and Kubernetes, I have only a few times (in 2015) had to debug an issue that involved NetworkManager.

eg NetworkManager is already widely deployed in mission-critical server environments and seems to work fairly well there for container-based use-cases.

[dcbw]

Yeah; see e.g. this bug.

@cgwalters not sure it's that one, for whatever reason we do not (and haven't) had problems with OpenShift and NM, and we extensively use veth interfaces there. NM does not touch them (though it does recognize them and show them in its CLI).