From: Miguel Luis <miguel.luis@oracle.com>
To: "Russell King (Oracle)" <linux@armlinux.org.uk>
Cc: "linux-pm@vger.kernel.org" <linux-pm@vger.kernel.org>,
"loongarch@lists.linux.dev" <loongarch@lists.linux.dev>,
"linux-acpi@vger.kernel.org" <linux-acpi@vger.kernel.org>,
"linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>,
"linux-riscv@lists.infradead.org"
<linux-riscv@lists.infradead.org>,
"kvmarm@lists.linux.dev" <kvmarm@lists.linux.dev>,
"x86@kernel.org" <x86@kernel.org>,
"acpica-devel@lists.linuxfoundation.org"
<acpica-devel@lists.linuxfoundation.org>,
"linux-csky@vger.kernel.org" <linux-csky@vger.kernel.org>,
"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
"linux-ia64@vger.kernel.org" <linux-ia64@vger.kernel.org>,
"linux-parisc@vger.kernel.org" <linux-parisc@vger.kernel.org>,
Salil Mehta <salil.mehta@huawei.com>,
Jean-Philippe Brucker <jean-philippe@linaro.org>,
"jianyong.wu@arm.com" <jianyong.wu@arm.com>,
"justin.he@arm.com" <justin.he@arm.com>,
James Morse <james.morse@arm.com>
Subject: Re: [RFC PATCH v3 00/39] ACPI/arm64: add support for virtual cpuhotplug
Date: Mon, 30 Oct 2023 16:41:19 +0000 [thread overview]
Message-ID: <C2C5C292-AD2C-4D98-8225-39ABE68C5395@oracle.com> (raw)
In-Reply-To: <ZTffkAdOqL2pI2la@shell.armlinux.org.uk>
Hi Russell,
> On 24 Oct 2023, at 15:15, Russell King (Oracle) <linux@armlinux.org.uk> wrote:
>
> Hi,
>
> I'm posting James' patch set updated with most of the review comments
> from his RFC v2 series back in September. Individual patches have a
> changelog attached at the bottom of the commit message. Those which
> I have finished updating have my S-o-b on them, those which still have
> outstanding review comments from RFC v2 do not. In some of these cases
> I've asked questions and am waiting for responses.
>
> I'm posting this as RFC v3 because there's still some unaddressed
> comments and it's clearly not ready for merging. Even if it was ready
> to be merged, it is too late in this development cycle to be taking
> this change in, so there would be little point posting it non-RFC.
> Also James stated that he's waiting for confirmation from the
> Kubernetes/Kata folk - I have no idea what the status is there.
>
> I will be sending each patch individually to a wider audience
> appropriate for that patch - apologies to those missing out on this
> cover message. I have added more mailing lists to the series with the
> exception of the acpica list in a hope of this cover message also
> reaching those folk.
>
> The changes that aren't included are:
>
> 1. Updates for my patch that was merged via Thomas (thanks!):
> c4dd854f740c cpu-hotplug: Provide prototypes for arch CPU registration
> rather than having this change spread through James' patches.
>
> 2. New patch - simplification of PA-RISC's smp_prepare_boot_cpu()
>
> 3. Moved "ACPI: Use the acpi_device_is_present() helper in more places"
> and "ACPI: Rename acpi_scan_device_not_present() to be about
> enumeration" to the beginning of the series - these two patches are
> already queued up for merging into 6.7.
>
> 4. Moved "arm64, irqchip/gic-v3, ACPI: Move MADT GICC enabled check into
> a helper" to the beginning of the series, which has been submitted,
> but as yet the fate of that posting isn't known.
>
> The first four patches in this series are provided for completness only.
>
> There is an additional patch in James' git tree that isn't in the set
> of patches that James posted: "ACPI: processor: Only call
> arch_unregister_cpu() if HOTPLUG_CPU is selected" which looks to me to
> be a workaround for arch_unregister_cpu() being under the ifdef. I've
> commented on this on the RFC v2 posting making a suggestion, but as yet
> haven't had any response.
>
> I've included almost all of James' original covering body below the
> diffstat.
>
> The reason that I'm doing this is to help move this code forward so
> hopefully it can be merged - which is why I have been keen to dig out
> from James' patches anything that can be merged and submit it
> separately, since this is a feature for which some users have a
> definite need for.
>
> Please note that I haven't tested this beyond building for aarch64 at
> the present time.
>
> The series can be found at:
>
> git://git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/v6.6-rc7
>
> Documentation/arch/arm64/cpu-hotplug.rst | 79 +++++++++++++++
> Documentation/arch/arm64/index.rst | 1 +
> arch/arm64/Kconfig | 1 +
> arch/arm64/include/asm/acpi.h | 11 +++
> arch/arm64/include/asm/cpu.h | 1 -
> arch/arm64/kernel/acpi_numa.c | 11 ---
> arch/arm64/kernel/psci.c | 2 +-
> arch/arm64/kernel/setup.c | 13 +--
> arch/arm64/kernel/smp.c | 5 +-
> arch/ia64/Kconfig | 3 +
> arch/ia64/include/asm/acpi.h | 2 +-
> arch/ia64/include/asm/cpu.h | 6 --
> arch/ia64/kernel/acpi.c | 6 +-
> arch/ia64/kernel/setup.c | 2 +-
> arch/ia64/kernel/topology.c | 35 +------
> arch/loongarch/Kconfig | 2 +
> arch/loongarch/configs/loongson3_defconfig | 2 +-
> arch/loongarch/kernel/acpi.c | 4 +-
> arch/loongarch/kernel/topology.c | 38 +-------
> arch/parisc/kernel/smp.c | 8 +-
> arch/riscv/Kconfig | 1 +
> arch/riscv/kernel/setup.c | 19 +---
> arch/x86/Kconfig | 3 +
> arch/x86/include/asm/cpu.h | 4 -
> arch/x86/kernel/acpi/boot.c | 4 +-
> arch/x86/kernel/cpu/intel_epb.c | 2 +-
> arch/x86/kernel/topology.c | 27 +-----
> drivers/acpi/Kconfig | 14 ++-
> drivers/acpi/acpi_processor.c | 151 +++++++++++++++++++++++------
> drivers/acpi/bus.c | 16 +++
> drivers/acpi/device_pm.c | 2 +-
> drivers/acpi/device_sysfs.c | 2 +-
> drivers/acpi/internal.h | 1 -
> drivers/acpi/processor_core.c | 2 +-
> drivers/acpi/property.c | 2 +-
> drivers/acpi/scan.c | 148 ++++++++++++++++++----------
> drivers/base/arch_topology.c | 38 +++++---
> drivers/base/cpu.c | 44 +++++++--
> drivers/base/init.c | 2 +-
> drivers/base/node.c | 7 --
> drivers/firmware/psci/psci.c | 2 +
> drivers/irqchip/irq-gic-v3.c | 38 +++++---
> include/acpi/acpi_bus.h | 1 +
> include/acpi/actbl2.h | 1 +
> include/linux/acpi.h | 13 ++-
> include/linux/cpu.h | 4 +
> include/linux/cpumask.h | 25 +++++
> kernel/cpu.c | 3 +
> 48 files changed, 516 insertions(+), 292 deletions(-)
>
>
> On Wed, Sep 13, 2023 at 04:37:48PM +0000, James Morse wrote:
>> Hello!
>>
>> Changes since RFC-v1:
>> * riscv is new, ia64 is gone
>> * The KVM support is different, and upstream - no need to patch the host.
>>
>> ---
>>
>> This series adds what looks like cpuhotplug support to arm64 for use in
>> virtual machines. It does this by moving the cpu_register() calls for
>> architectures that support ACPI out of the arch code by using
>> GENERIC_CPU_DEVICES, then into the ACPI processor driver.
>>
>> The kubernetes folk really want to be able to add CPUs to an existing VM,
>> in exactly the same way they do on x86. The use-case is pre-booting guests
>> with one CPU, then adding the number that were actually needed when the
>> workload is provisioned.
>>
>> Wait? Doesn't arm64 support cpuhotplug already!?
>> In the arm world, cpuhotplug gets used to mean removing the power from a CPU.
>> The CPU is offline, and remains present. For x86, and ACPI, cpuhotplug
>> has the additional step of physically removing the CPU, so that it isn't
>> present anymore.
>>
>> Arm64 doesn't support this, and can't support it: CPUs are really a slice
>> of the SoC, and there is not enough information in the existing ACPI tables
>> to describe which bits of the slice also got removed. Without a reference
>> machine: adding this support to the spec is a wild goose chase.
>>
>> Critically: everything described in the firmware tables must remain present.
>>
>> For a virtual machine this is easy as all the other bits of 'virtual SoC'
>> are emulated, so they can (and do) remain present when a vCPU is 'removed'.
>>
>> On a system that supports cpuhotplug the MADT has to describe every possible
>> CPU at boot. Under KVM, the vGIC needs to know about every possible vCPU before
>> the guest is started.
>> With these constraints, virtual-cpuhotplug is really just a hypervisor/firmware
>> policy about which CPUs can be brought online.
>>
>> This series adds support for virtual-cpuhotplug as exactly that: firmware
>> policy. This may even work on a physical machine too; for a guest the part of
>> firmware is played by the VMM. (typically Qemu).
>>
>> PSCI support is modified to return 'DENIED' if the CPU can't be brought
>> online/enabled yet. The CPU object's _STA method's enabled bit is used to
>> indicate firmware's current disposition. If the CPU has its enabled bit clear,
>> it will not be registered with sysfs, and attempts to bring it online will
>> fail. The notifications that _STA has changed its value then work in the same
>> way as physical hotplug, and firmware can cause the CPU to be registered some
>> time later, allowing it to be brought online.
>>
>> This creates something that looks like cpuhotplug to user-space, as the sysfs
>> files appear and disappear, and the udev notifications look the same.
>>
>> One notable difference is the CPU present mask, which is exposed via sysfs.
>> Because the CPUs remain present throughout, they can still be seen in that mask.
>> This value does get used by webbrowsers to estimate the number of CPUs
>> as the CPU online mask is constantly changed on mobile phones.
>>
>> Linux is tolerant of PSCI returning errors, as its always been allowed to do
>> that. To avoid confusing OS that can't tolerate this, we needed an additional
>> bit in the MADT GICC flags. This series copies ACPI_MADT_ONLINE_CAPABLE, which
>> appears to be for this purpose, but calls it ACPI_MADT_GICC_CPU_CAPABLE as it
>> has a different bit position in the GICC.
>>
>> This code is unconditionally enabled for all ACPI architectures.
>> If there are problems with firmware tables on some devices, the CPUs will
>> already be online by the time the acpi_processor_make_enabled() is called.
>> A mismatch here causes a firmware-bug message and kernel taint. This should
>> only affect people with broken firmware who also boot with maxcpus=1, and
>> bring CPUs online later.
>>
>> I had a go at switching the remaining architectures over to GENERIC_CPU_DEVICES,
>> so that the Kconfig symbol can be removed, but I got stuck with powerpc
>> and s390.
>>
>> I've only build tested Loongarch and riscv. I've removed the ia64 specific
>> patches, but left the changes in other patches to make git-grep review of
>> renames easier.
>>
>> If folk want to play along at home, you'll need a copy of Qemu that supports this.
>> https://github.com/salil-mehta/qemu.git salil/virt-cpuhp-armv8/rfc-v2-rc6
>>
>> Replace your '-smp' argument with something like:
>> | -smp cpus=1,maxcpus=3,cores=3,threads=1,sockets=1
>>
>> then feed the following to the Qemu montior;
>> | (qemu) device_add driver=host-arm-cpu,core-id=1,id=cpu1
>> | (qemu) device_del cpu1
>>
>>
>> Why is this still an RFC? I'm still looking for confirmation from the
>> kubernetes/kata folk that this works for them. Because of this I've culled
>> the CC list...
>>
>>
>> This series is based on v6.6-rc1, and can be retrieved from:
>> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/ virtual_cpu_hotplug/rfc/v2
>>
>>
>> Thanks,
>>
>> James Morse (34):
>> ACPI: Move ACPI_HOTPLUG_CPU to be disabled on arm64 and riscv
>> drivers: base: Use present CPUs in GENERIC_CPU_DEVICES
>> drivers: base: Allow parts of GENERIC_CPU_DEVICES to be overridden
>> drivers: base: Move cpu_dev_init() after node_dev_init()
>> drivers: base: Print a warning instead of panic() when register_cpu()
>> fails
>> arm64: setup: Switch over to GENERIC_CPU_DEVICES using
>> arch_register_cpu()
>> x86: intel_epb: Don't rely on link order
>> x86/topology: Switch over to GENERIC_CPU_DEVICES
>> LoongArch: Switch over to GENERIC_CPU_DEVICES
>> riscv: Switch over to GENERIC_CPU_DEVICES
>> arch_topology: Make register_cpu_capacity_sysctl() tolerant to late
>> CPUs
>> ACPI: Use the acpi_device_is_present() helper in more places
>> ACPI: Rename acpi_scan_device_not_present() to be about enumeration
>> ACPI: Only enumerate enabled (or functional) devices
>> ACPI: processor: Add support for processors described as container
>> packages
>> ACPI: processor: Register CPUs that are online, but not described in
>> the DSDT
>> ACPI: processor: Register all CPUs from acpi_processor_get_info()
>> ACPI: Rename ACPI_HOTPLUG_CPU to include 'present'
>> ACPI: Move acpi_bus_trim_one() before acpi_scan_hot_remove()
>> ACPI: Rename acpi_processor_hotadd_init and remove pre-processor
>> guards
>> ACPI: Add post_eject to struct acpi_scan_handler for cpu hotplug
>> ACPI: Check _STA present bit before making CPUs not present
>> ACPI: Warn when the present bit changes but the feature is not enabled
>> drivers: base: Implement weak arch_unregister_cpu()
>> LoongArch: Use the __weak version of arch_unregister_cpu()
>> arm64: acpi: Move get_cpu_for_acpi_id() to a header
>> ACPICA: Add new MADT GICC flags fields [code first?]
>> arm64, irqchip/gic-v3, ACPI: Move MADT GICC enabled check into a
>> helper
>> irqchip/gic-v3: Don't return errors from gic_acpi_match_gicc()
>> irqchip/gic-v3: Add support for ACPI's disabled but 'online capable'
>> CPUs
>> ACPI: add support to register CPUs based on the _STA enabled bit
>> arm64: document virtual CPU hotplug's expectations
>> ACPI: Add _OSC bits to advertise OS support for toggling CPU
>> present/enabled
>> cpumask: Add enabled cpumask for present CPUs that can be brought
>> online
>>
>> Jean-Philippe Brucker (1):
>> arm64: psci: Ignore DENIED CPUs
>
Tested on QEMU, based on Salil's RFC v2 [1], running with KVM.
- boot
- hotplug up to 'maxcpus'
- hotunplug down to the number of boot cpus
- hotplug vcpus and migrate with vcpus offline
- hotplug vcpus and migrate with vcpus online
- hotplug vcpus then unplug vcpus then migrate
- successive live migrations (up until 6)
Feel free to add:
Tested-by: Miguel Luis <miguel.luis@oracle.com>
Thank you
Miguel
[1] https://lore.kernel.org/qemu-devel/20230926100436.28284-1-salil.mehta@huawei.com/
> --
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!
>
next prev parent reply other threads:[~2023-10-30 16:42 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-24 15:15 [RFC PATCH v3 00/39] ACPI/arm64: add support for virtual cpuhotplug Russell King (Oracle)
2023-10-24 15:16 ` [PATCH 01/39] parisc: simplify smp_prepare_boot_cpu() Russell King (Oracle)
2023-10-24 15:16 ` [PATCH 02/39] ACPI: Use the acpi_device_is_present() helper in more places Russell King
2023-10-24 15:16 ` [PATCH 03/39] ACPI: Rename acpi_scan_device_not_present() to be about enumeration Russell King
2023-10-24 15:16 ` [PATCH 04/39] arm64, irqchip/gic-v3, ACPI: Move MADT GICC enabled check into a helper Russell King
2023-10-24 15:16 ` [PATCH 05/39] ACPI: Move ACPI_HOTPLUG_CPU to be disabled on arm64 and riscv Russell King
2023-10-24 15:16 ` [PATCH 06/39] drivers: base: Use present CPUs in GENERIC_CPU_DEVICES Russell King
2023-10-24 15:16 ` [PATCH 07/39] drivers: base: Allow parts of GENERIC_CPU_DEVICES to be overridden Russell King
2023-10-24 15:16 ` [PATCH 08/39] drivers: base: Move cpu_dev_init() after node_dev_init() Russell King
2023-10-24 15:16 ` [PATCH 09/39] drivers: base: remove unnecessary call to register_cpu_under_node() Russell King (Oracle)
2023-10-24 15:17 ` [PATCH 10/39] drivers: base: Print a warning instead of panic() when register_cpu() fails Russell King
2023-10-24 15:17 ` [PATCH 11/39] arm64: setup: Switch over to GENERIC_CPU_DEVICES using arch_register_cpu() Russell King
2023-10-24 15:17 ` [PATCH 12/39] ia64/topology: Switch over to GENERIC_CPU_DEVICES Russell King
2023-10-24 15:17 ` [PATCH 13/39] x86: intel_epb: Don't rely on link order Russell King
2023-10-24 15:17 ` [PATCH 14/39] x86/topology: Switch over to GENERIC_CPU_DEVICES Russell King
2023-10-24 15:17 ` [PATCH 15/39] LoongArch: " Russell King
2023-10-24 15:17 ` [PATCH 16/39] riscv: " Russell King
2023-10-24 15:17 ` [PATCH 17/39] arch_topology: Make register_cpu_capacity_sysctl() tolerant to late CPUs Russell King
2023-10-24 15:17 ` [PATCH 18/39] ACPI: Only enumerate enabled (or functional) devices Russell King
2023-10-24 18:07 ` Rafael J. Wysocki
2023-10-24 15:17 ` [PATCH 19/39] ACPI: processor: Add support for processors described as container packages Russell King
2023-10-24 15:17 ` [PATCH 20/39] ACPI: processor: Register CPUs that are online, but not described in the DSDT Russell King
2023-10-24 15:17 ` [PATCH 21/39] ACPI: processor: Register all CPUs from acpi_processor_get_info() Russell King
2023-10-24 15:18 ` [PATCH 22/39] ACPI: Rename ACPI_HOTPLUG_CPU to include 'present' Russell King
2023-10-24 15:18 ` [PATCH 23/39] drivers: base: Implement weak arch_unregister_cpu() Russell King
2023-10-24 15:18 ` [PATCH 24/39] ACPI: Move acpi_bus_trim_one() before acpi_scan_hot_remove() Russell King
2023-10-24 15:18 ` [PATCH 25/39] ACPI: Rename acpi_processor_hotadd_init and remove pre-processor guards Russell King
2023-10-24 15:18 ` [PATCH 26/39] ACPI: Add post_eject to struct acpi_scan_handler for cpu hotplug Russell King
2023-10-24 15:18 ` [PATCH 27/39] ACPI: Check _STA present bit before making CPUs not present Russell King
2023-10-24 15:18 ` [PATCH 28/39] ACPI: Warn when the present bit changes but the feature is not enabled Russell King
2023-10-24 15:18 ` [PATCH 29/39] LoongArch: Use the __weak version of arch_unregister_cpu() Russell King
2023-10-24 15:18 ` [PATCH 30/39] arm64: acpi: Move get_cpu_for_acpi_id() to a header Russell King
2023-10-24 15:18 ` [PATCH 31/39] ACPICA: Add new MADT GICC flags fields Russell King
2023-10-24 15:18 ` [PATCH 32/39] irqchip/gic-v3: Don't return errors from gic_acpi_match_gicc() Russell King
2023-10-24 15:18 ` [PATCH 33/39] irqchip/gic-v3: Add support for ACPI's disabled but 'online capable' CPUs Russell King
2023-10-24 15:19 ` [PATCH 34/39] arm64: psci: Ignore DENIED CPUs Russell King
2023-11-16 7:45 ` Jianyong Wu
2023-11-20 9:24 ` Russell King (Oracle)
2023-11-20 9:36 ` Jianyong Wu
2023-11-20 9:58 ` Russell King (Oracle)
2023-11-21 1:44 ` Jianyong Wu
2023-10-24 15:19 ` [PATCH 35/39] ACPI: add support to register CPUs based on the _STA enabled bit Russell King
2023-10-24 15:19 ` [PATCH 36/39] arm64: document virtual CPU hotplug's expectations Russell King
2023-10-24 15:19 ` [PATCH 37/39] ACPI: Add _OSC bits to advertise OS support for toggling CPU present/enabled Russell King
2023-10-24 15:19 ` [PATCH 38/39] cpumask: Add enabled cpumask for present CPUs that can be brought online Russell King
2023-10-24 15:32 ` Andy Shevchenko
2023-10-24 16:02 ` Greg Kroah-Hartman
2023-10-24 16:05 ` Russell King (Oracle)
2023-10-24 15:19 ` [PATCH 39/39] ACPI: processor: Only call arch_unregister_cpu() if HOTPLUG_CPU is selected Russell King
2023-10-24 16:31 ` [RFC PATCH v3 00/39] ACPI/arm64: add support for virtual cpuhotplug Russell King (Oracle)
2023-10-24 18:26 ` Rafael J. Wysocki
2023-10-24 19:28 ` Russell King (Oracle)
2023-12-04 18:23 ` Russell King (Oracle)
2023-12-12 19:58 ` Russell King (Oracle)
2023-12-12 20:05 ` Rafael J. Wysocki
2023-12-12 20:16 ` Russell King (Oracle)
2023-10-30 16:41 ` Miguel Luis [this message]
2023-10-30 17:51 ` Russell King (Oracle)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=C2C5C292-AD2C-4D98-8225-39ABE68C5395@oracle.com \
--to=miguel.luis@oracle.com \
--cc=acpica-devel@lists.linuxfoundation.org \
--cc=james.morse@arm.com \
--cc=jean-philippe@linaro.org \
--cc=jianyong.wu@arm.com \
--cc=justin.he@arm.com \
--cc=kvmarm@lists.linux.dev \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-csky@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-ia64@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-parisc@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=linux@armlinux.org.uk \
--cc=loongarch@lists.linux.dev \
--cc=salil.mehta@huawei.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).