Re: [PATCH RFC] KVM: arm64: allow ID_MMFR4_EL1 to be writable

From: Oliver Upton <oliver.upton@linux.dev>
To: Cornelia Huck <cohuck@redhat.com>
Cc: "Russell King (Oracle)" <linux@armlinux.org.uk>,
	Marc Zyngier <maz@kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will@kernel.org>, James Morse <james.morse@arm.com>,
	Suzuki K Poulose <suzuki.poulose@arm.com>,
	Zenghui Yu <yuzenghui@huawei.com>,
	linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev
Subject: Re: [PATCH RFC] KVM: arm64: allow ID_MMFR4_EL1 to be writable
Date: Wed, 8 May 2024 12:14:11 -0500	[thread overview]
Message-ID: <Zjuy40shK5a-yp95@linux.dev> (raw)
In-Reply-To: <87jzk4h5ir.fsf@redhat.com>

Hi Cornelia,

On Wed, May 08, 2024 at 02:06:36PM +0200, Cornelia Huck wrote:
> On Thu, May 02 2024, Oliver Upton <oliver.upton@linux.dev> wrote:
> > I think (1) should only be expected of VMMs that want rollback safety,
> > i.e. the ability to migrate state back to an older kernel. Our userspace
> > initializes vCPUs from a fixed set of feature ID register values that
> > prevents VMs on new kernels from picking up new CPU features.
> >
> > It is quite tedious, but necessary as rollback safety is very much a
> > non-goal of the KVM UAPI.
> 
> Depending on your use case, rollback safety might be quite
> important... have we ever stated exactly which guarantees the KVM UAPI
> is giving? IOW, can someone implementing a VMM look at a doc and see
> "oh, if I want to be able to go backwards, I need to be able to deal
> with x, y, and z coming up on the new kernel"?

The behavior of KVM/arm64 has always been that new VMs get the maximum
set of vCPU features supported by KVM / hardware modulo the ones we
require explicit opt-in from userspace (e.g. SVE, vPMU). This behavior
is described in the arm64 vCPU feature documentation [1].

The biggest benefit of this approach is that new vCPU features can be
added without a VMM change, as userspace can just treat the registers in
KVM_GET_REG_LIST as an opaque blob of data that needs to be migrated.

I'm willing to wager that the set of users who want to migrate state
from kernel N -> (N - 1) know the exact CPU definition they want to
expose to the guest, and in that case should be using a static set of
feature register values matching their intent.

Beyond the CPU architecture, KVM presents hypercall features to the VM
which userspace can _opt-out_ of on a per-feature basis using the
feature bitmap registers described in [2]. Like the feature ID
registers, we've preallocated a range of indices to be used for
hypercall bitmaps. So if an unexpected bitmap register appears on a new
kernel, userspace should write 0 to it to prevent new features from
silently creeping in.

Prescribing the exact combination of these UAPIs to achieve a
rollback-safe feature set is beyond the scope of the KVM documentation
and should be determined based on the minimum kernel version that needs
to work.

[1]: https://docs.kernel.org/virt/kvm/arm/vcpu-features.html#the-id-registers
[2]: https://docs.kernel.org/virt/kvm/arm/hypercalls.html

> >> I've been bitten before with KVM differences between kernel versions
> >> in the past - where the number of registers that userspace sees has
> >> changed despite being on the same hardware.
> >
> > This is intended behavior, as VMs are initialized to the maximum feature
> > set KVM is able to support. Forward-compatibility for the set of exposed
> > registers is tested, see the get-reg-list selftest.
> 
> I've seen this problem come up as well; if it is clear that this is
> something that KVM expects the VMM to handle if needed, that is fine;
> should we consider "it's tested in a selftest" as a canonical indicator
> of "this is what KVM supports compatibility wise"?

It certainly is the level of compatibility that gets actively tested :)

The canonical reason for this behavior, though, is that KVM/arm64
defaults to the maximum-possible feature set as discussed above.

-- 
Thanks,
Oliver

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel