From: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
To: linux-pm@vger.kernel.org, loongarch@lists.linux.dev,
linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org,
linux-riscv@lists.infradead.org, kvmarm@lists.linux.dev,
x86@kernel.org, linux-csky@vger.kernel.org,
linux-doc@vger.kernel.org, linux-ia64@vger.kernel.org,
linux-parisc@vger.kernel.org
Cc: Salil Mehta <salil.mehta@huawei.com>,
Jean-Philippe Brucker <jean-philippe@linaro.org>,
jianyong.wu@arm.com, justin.he@arm.com,
James Morse <james.morse@arm.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>, Jonathan Corbet <corbet@lwn.net>
Subject: [PATCH 36/39] arm64: document virtual CPU hotplug's expectations
Date: Tue, 24 Oct 2023 16:19:14 +0100 [thread overview]
Message-ID: <E1qvJBa-00AqSK-Hb@rmk-PC.armlinux.org.uk> (raw)
In-Reply-To: <ZTffkAdOqL2pI2la@shell.armlinux.org.uk>
From: James Morse <james.morse@arm.com>
Add a description of physical and virtual CPU hotplug, explain the
differences and elaborate on what is required in ACPI for a working
virtual hotplug system.
Signed-off-by: James Morse <james.morse@arm.com>
---
Documentation/arch/arm64/cpu-hotplug.rst | 79 ++++++++++++++++++++++++
Documentation/arch/arm64/index.rst | 1 +
2 files changed, 80 insertions(+)
create mode 100644 Documentation/arch/arm64/cpu-hotplug.rst
diff --git a/Documentation/arch/arm64/cpu-hotplug.rst b/Documentation/arch/arm64/cpu-hotplug.rst
new file mode 100644
index 000000000000..76ba8d932c72
--- /dev/null
+++ b/Documentation/arch/arm64/cpu-hotplug.rst
@@ -0,0 +1,79 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. _cpuhp_index:
+
+====================
+CPU Hotplug and ACPI
+====================
+
+CPU hotplug in the arm64 world is commonly used to describe the kernel taking
+CPUs online/offline using PSCI. This document is about ACPI firmware allowing
+CPUs that were not available during boot to be added to the system later.
+
+``possible`` and ``present`` refer to the state of the CPU as seen by linux.
+
+
+CPU Hotplug on physical systems - CPUs not present at boot
+----------------------------------------------------------
+
+Physical systems need to mark a CPU that is ``possible`` but not ``present`` as
+being ``present``. An example would be a dual socket machine, where the package
+in one of the sockets can be replaced while the system is running.
+
+This is not supported.
+
+In the arm64 world CPUs are not a single device but a slice of the system.
+There are no systems that support the physical addition (or removal) of CPUs
+while the system is running, and ACPI is not able to sufficiently describe
+them.
+
+e.g. New CPUs come with new caches, but the platform's cache toplogy is
+described in a static table, the PPTT. How caches are shared between CPUs is
+not discoverable, and must be described by firmware.
+
+e.g. The GIC redistributor for each CPU must be accessed by the driver during
+boot to discover the system wide supported features. ACPI's MADT GICC
+structures can describe a redistributor associated with a disabled CPU, but
+can't describe whether the redistributor is accessible, only that it is not
+'always on'.
+
+arm64's ACPI tables assume that everything described is ``present``.
+
+
+CPU Hotplug on virtual systems - CPUs not enabled at boot
+---------------------------------------------------------
+
+Virtual systems have the advantage that all the properties the system will
+ever have can be described at boot. There are no power-domain considerations
+as such devices are emulated.
+
+CPU Hotplug on virtual systems is supported. It is distinct from physical
+CPU Hotplug as all resources are described as ``present``, but CPUs may be
+marked as disabled by firmware. Only the CPU's online/offline behaviour is
+influenced by firmware. An example is where a virtual machine boots with a
+single CPU, and additional CPUs are added once a cloud orchestrator deploys
+the workload.
+
+For a virtual machine, the VMM (e.g. Qemu) plays the part of firmware.
+
+Virtual hotplug is implemented as a firmware policy affecting which CPUs can be
+brought online. Firmware can enforce its policy via PSCI's return codes. e.g.
+``DENIED``.
+
+The ACPI tables must describe all the resources of the virtual machine. CPUs
+that firmware wishes to disable either from boot (or later) should not be
+``enabled`` in the MADT GICC structures, but should have the ``online capable``
+bit set, to indicate they can be enabled later. The boot CPU must be marked as
+``enabled``. The 'always on' GICR structure must be used to describe the
+redistributors.
+
+CPUs described as ``online capable`` but not ``enabled`` can be set to enabled
+by the DSDT's Processor object's _STA method. On virtual systems the _STA method
+must always report the CPU as ``present``. Changes to the firmware policy can
+be notified to the OS via device-check or eject-request.
+
+CPUs described as ``enabled`` in the static table, should not have their _STA
+modified dynamically by firmware. Soft-restart features such as kexec will
+re-read the static properties of the system from these static tables, and
+may malfunction if these no longer describe the running system. Linux will
+re-discover the dynamic properties of the system from the _STA method later
+during boot.
diff --git a/Documentation/arch/arm64/index.rst b/Documentation/arch/arm64/index.rst
index d08e924204bf..78544de0a8a9 100644
--- a/Documentation/arch/arm64/index.rst
+++ b/Documentation/arch/arm64/index.rst
@@ -13,6 +13,7 @@ ARM64 Architecture
asymmetric-32bit
booting
cpu-feature-registers
+ cpu-hotplug
elf_hwcaps
hugetlbpage
kdump
--
2.30.2
next prev parent reply other threads:[~2023-10-24 15:29 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-24 15:15 [RFC PATCH v3 00/39] ACPI/arm64: add support for virtual cpuhotplug Russell King (Oracle)
2023-10-24 15:16 ` [PATCH 01/39] parisc: simplify smp_prepare_boot_cpu() Russell King (Oracle)
2023-10-24 15:16 ` [PATCH 02/39] ACPI: Use the acpi_device_is_present() helper in more places Russell King
2023-10-24 15:16 ` [PATCH 03/39] ACPI: Rename acpi_scan_device_not_present() to be about enumeration Russell King
2023-10-24 15:16 ` [PATCH 04/39] arm64, irqchip/gic-v3, ACPI: Move MADT GICC enabled check into a helper Russell King
2023-10-24 15:16 ` [PATCH 05/39] ACPI: Move ACPI_HOTPLUG_CPU to be disabled on arm64 and riscv Russell King
2023-10-24 15:16 ` [PATCH 06/39] drivers: base: Use present CPUs in GENERIC_CPU_DEVICES Russell King
2023-10-24 15:16 ` [PATCH 07/39] drivers: base: Allow parts of GENERIC_CPU_DEVICES to be overridden Russell King
2023-10-24 15:16 ` [PATCH 08/39] drivers: base: Move cpu_dev_init() after node_dev_init() Russell King
2023-10-24 15:16 ` [PATCH 09/39] drivers: base: remove unnecessary call to register_cpu_under_node() Russell King (Oracle)
2023-10-24 15:17 ` [PATCH 10/39] drivers: base: Print a warning instead of panic() when register_cpu() fails Russell King
2023-10-24 15:17 ` [PATCH 11/39] arm64: setup: Switch over to GENERIC_CPU_DEVICES using arch_register_cpu() Russell King
2023-10-24 15:17 ` [PATCH 12/39] ia64/topology: Switch over to GENERIC_CPU_DEVICES Russell King
2023-10-24 15:17 ` [PATCH 13/39] x86: intel_epb: Don't rely on link order Russell King
2023-10-24 15:17 ` [PATCH 14/39] x86/topology: Switch over to GENERIC_CPU_DEVICES Russell King
2023-10-24 15:17 ` [PATCH 15/39] LoongArch: " Russell King
2023-10-24 15:17 ` [PATCH 16/39] riscv: " Russell King
2023-10-24 15:17 ` [PATCH 17/39] arch_topology: Make register_cpu_capacity_sysctl() tolerant to late CPUs Russell King
2023-10-24 15:17 ` [PATCH 18/39] ACPI: Only enumerate enabled (or functional) devices Russell King
2023-10-24 18:07 ` Rafael J. Wysocki
2023-10-24 15:17 ` [PATCH 19/39] ACPI: processor: Add support for processors described as container packages Russell King
2023-10-24 15:17 ` [PATCH 20/39] ACPI: processor: Register CPUs that are online, but not described in the DSDT Russell King
2023-10-24 15:17 ` [PATCH 21/39] ACPI: processor: Register all CPUs from acpi_processor_get_info() Russell King
2023-10-24 15:18 ` [PATCH 22/39] ACPI: Rename ACPI_HOTPLUG_CPU to include 'present' Russell King
2023-10-24 15:18 ` [PATCH 23/39] drivers: base: Implement weak arch_unregister_cpu() Russell King
2023-10-24 15:18 ` [PATCH 24/39] ACPI: Move acpi_bus_trim_one() before acpi_scan_hot_remove() Russell King
2023-10-24 15:18 ` [PATCH 25/39] ACPI: Rename acpi_processor_hotadd_init and remove pre-processor guards Russell King
2023-10-24 15:18 ` [PATCH 26/39] ACPI: Add post_eject to struct acpi_scan_handler for cpu hotplug Russell King
2023-10-24 15:18 ` [PATCH 27/39] ACPI: Check _STA present bit before making CPUs not present Russell King
2023-10-24 15:18 ` [PATCH 28/39] ACPI: Warn when the present bit changes but the feature is not enabled Russell King
2023-10-24 15:18 ` [PATCH 29/39] LoongArch: Use the __weak version of arch_unregister_cpu() Russell King
2023-10-24 15:18 ` [PATCH 30/39] arm64: acpi: Move get_cpu_for_acpi_id() to a header Russell King
2023-10-24 15:18 ` [PATCH 31/39] ACPICA: Add new MADT GICC flags fields Russell King
2023-10-24 15:18 ` [PATCH 32/39] irqchip/gic-v3: Don't return errors from gic_acpi_match_gicc() Russell King
2023-10-24 15:18 ` [PATCH 33/39] irqchip/gic-v3: Add support for ACPI's disabled but 'online capable' CPUs Russell King
2023-10-24 15:19 ` [PATCH 34/39] arm64: psci: Ignore DENIED CPUs Russell King
2023-11-16 7:45 ` Jianyong Wu
2023-11-20 9:24 ` Russell King (Oracle)
2023-11-20 9:36 ` Jianyong Wu
2023-11-20 9:58 ` Russell King (Oracle)
2023-11-21 1:44 ` Jianyong Wu
2023-10-24 15:19 ` [PATCH 35/39] ACPI: add support to register CPUs based on the _STA enabled bit Russell King
2023-10-24 15:19 ` Russell King [this message]
2023-10-24 15:19 ` [PATCH 37/39] ACPI: Add _OSC bits to advertise OS support for toggling CPU present/enabled Russell King
2023-10-24 15:19 ` [PATCH 38/39] cpumask: Add enabled cpumask for present CPUs that can be brought online Russell King
2023-10-24 15:32 ` Andy Shevchenko
2023-10-24 16:02 ` Greg Kroah-Hartman
2023-10-24 16:05 ` Russell King (Oracle)
2023-10-24 15:19 ` [PATCH 39/39] ACPI: processor: Only call arch_unregister_cpu() if HOTPLUG_CPU is selected Russell King
2023-10-24 16:31 ` [RFC PATCH v3 00/39] ACPI/arm64: add support for virtual cpuhotplug Russell King (Oracle)
2023-10-24 18:26 ` Rafael J. Wysocki
2023-10-24 19:28 ` Russell King (Oracle)
2023-12-04 18:23 ` Russell King (Oracle)
2023-12-12 19:58 ` Russell King (Oracle)
2023-12-12 20:05 ` Rafael J. Wysocki
2023-12-12 20:16 ` Russell King (Oracle)
2023-10-30 16:41 ` Miguel Luis
2023-10-30 17:51 ` Russell King (Oracle)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=E1qvJBa-00AqSK-Hb@rmk-PC.armlinux.org.uk \
--to=rmk+kernel@armlinux.org.uk \
--cc=catalin.marinas@arm.com \
--cc=corbet@lwn.net \
--cc=james.morse@arm.com \
--cc=jean-philippe@linaro.org \
--cc=jianyong.wu@arm.com \
--cc=justin.he@arm.com \
--cc=kvmarm@lists.linux.dev \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-csky@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-ia64@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-parisc@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=loongarch@lists.linux.dev \
--cc=salil.mehta@huawei.com \
--cc=will@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).