Policy for disabling SMT

[bgilbert]

There have been multiple rounds of CPU vulnerabilities (L1TF and MDS) which cannot be completely mitigated without disabling Simultaneous Multi-Threading on affected processors. Disabling SMT reduces system performance and changes the apparent number of processors on the system, which might confuse container orchestrators and the like, so it's difficult to do without warning in existing OSes -- especially on an upgrade of existing nodes. (Container Linux opted to leave SMT enabled but advise users to turn it off.) With Fedora CoreOS, we have the opportunity to choose a different default.

The kernel supports two relevant command-line arguments:

• mitigations=auto,nosmt disables SMT when needed to mitigate a vulnerability.
• nosmt unconditionally disables SMT.

There are also ways to disable SMT from userspace, but they're not connected to the kernel's vulnerability-detection logic. The only way to do conditional enablement is via the karg.

Proposal

Set mitigations=auto,nosmt by default on Fedora CoreOS.

Good parts:

• Better security by default
• Does not incur an unnecessary performance hit on CPUs that don't need it

Bad parts:

• Requires passing a karg, which isn't trivial to change via Ignition (though see Support default kernel arguments ostreedev/ostree#479)
• Causes a performance hit on vulnerable systems running workloads that aren't affected by the vulnerability, such as trusted workloads
• Does not protect systems with unknown SMT vulnerabilities, until those vulnerabilities become known and a kernel update is pushed out
• Updates to address future vulnerabilities can substantially reduce the performance of a node, as well as its apparent number of processors. I think this is a reasonable policy for a new OS, but we'd certainly need to document it.

Regardless of what we choose, we should document the default behavior and the fact that the user can change it.

Alternative defaults

• Unconditionally disable SMT. This provides predictable machine performance over the long term, but also penalizes processor families that have not had any public vulnerabilities.
• Leave SMT enabled. This leaves users exposed to known vulnerabilities, which is unsatisfactory.

[bgilbert]

Implementation merged in coreos/fedora-coreos-config#95. Design doc PR in #184.

[cgwalters]

This is...bold. I think it's probably right but things are going to get interesting if we start doing more things like this because it basically makes saying "I use Fedora" less meaningful (to the degree it is today), as the divergence between FCOS and other editions grows.

Also we're now diverging from RHCOS for that matter which...I have to wrap my head around this. Maybe we push to do this by default in the product too at some later release for new installs?

[bgilbert]

as the divergence between FCOS and other editions grows.

I'm comfortable with that divergence. 😄 The usage model for FCOS is already pretty substantially different from the other editions.

Maybe we push to do this by default in the product too at some later release for new installs?

Works for me! But I don't know that it's necessary. AIUI OpenShift is surfacing the SMT decision further up the stack, whereas FCOS doesn't have that ability.

[cgwalters]

The other aspect to this is I'd like to be able to say that Silverblue is based on FCOS...but for e.g. people compiling software I think this is a huge hit. (Obviously builds also happen on FCOS too, but it's more likely to be multi-tenant)

I wonder if there's been discussion about solving this less crudely in userspace - for example, there's no benefit to disabling SMT sharing between processes in the "same container" (definition of "container" can be up to userspace). Maybe ultimately some logic would need to be pushed to the scheduler though.

[bgilbert]

Silverblue should certainly use whatever default policy makes sense for them. Desktop use cases are an anti-goal for Fedora CoreOS so they shouldn't influence our default policy.

There has been some work on a kernel scheduler that co-schedules mutually trusted processes. From userspace, one could also imagine pinning workloads to specific cores. I don't think there's any tooling ready to go, though.

[dhawal55]

What is a good way to enable SMT from ignition config?

I'm trying to run FCOS on AWS which has deployed protections for MDS for all ec2 instances as mentioned here:
https://aws.amazon.com/security/security-bulletins/AWS-2019-004/

[cgwalters]

#318