From: Hector Martin <marcan@marcan.st> To: Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Marc Zyngier <maz@kernel.org>, Mark Rutland <mark.rutland@arm.com> Cc: Zayd Qumsieh <zayd_qumsieh@apple.com>, Justin Lu <ih_justin@apple.com>, Ryan Houdek <Houdek.Ryan@fex-emu.org>, Mark Brown <broonie@kernel.org>, Ard Biesheuvel <ardb@kernel.org>, Mateusz Guzik <mjguzik@gmail.com>, Anshuman Khandual <anshuman.khandual@arm.com>, Oliver Upton <oliver.upton@linux.dev>, Miguel Luis <miguel.luis@oracle.com>, Joey Gouly <joey.gouly@arm.com>, Christoph Paasch <cpaasch@apple.com>, Kees Cook <keescook@chromium.org>, Sami Tolvanen <samitolvanen@google.com>, Baoquan He <bhe@redhat.com>, Joel Granados <j.granados@samsung.com>, Dawei Li <dawei.li@shingroup.cn>, Andrew Morton <akpm@linux-foundation.org>, Florent Revest <revest@chromium.org>, David Hildenbrand <david@redhat.com>, Stefan Roesch <shr@devkernel.io>, Andy Chiu <andy.chiu@sifive.com>, Josh Triplett <josh@joshtriplett.org>, Oleg Nesterov <oleg@redhat.com>, Helge Deller <deller@gmx.de>, Zev Weiss <zev@bewilderbeest.net>, Ondrej Mosnacek <omosnace@redhat.com>, Miguel Ojeda <ojeda@kernel.org>, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux <asahi@lists.linux.dev>, Hector Martin <marcan@marcan.st> Subject: [PATCH 4/4] arm64: Implement Apple IMPDEF TSO memory model control Date: Thu, 11 Apr 2024 09:51:23 +0900 [thread overview] Message-ID: <20240411-tso-v1-4-754f11abfbff@marcan.st> (raw) In-Reply-To: <20240411-tso-v1-0-754f11abfbff@marcan.st> Apple CPUs may implement the TSO memory model as an optional configurable mode. This allows x86 emulators to simplify their load/store handling, greatly increasing performance. Expose this via the prctl PR_SET_MEM_MODEL_TSO mechanism. We use the Apple IMPDEF AIDR_EL1 register to check for the availability of TSO mode, and enable this codepath on all CPUs with an Apple implementer. This relies on the ACTLR_EL1 thread state scaffolding introduced earlier. Signed-off-by: Hector Martin <marcan@marcan.st> --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/apple_cpufeature.h | 15 +++++++++++++++ arch/arm64/include/asm/cpufeature.h | 3 ++- arch/arm64/kernel/cpufeature_impdef.c | 23 +++++++++++++++++++++++ arch/arm64/kernel/process.c | 22 ++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 6 files changed, 65 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9b3593b34cce..2f3eedd955c9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2167,6 +2167,8 @@ endif # ARM64_PSEUDO_NMI config ARM64_MEMORY_MODEL_CONTROL bool "Runtime memory model control" + default ARCH_APPLE + select ARM64_ACTLR_STATE help Some ARM64 CPUs support runtime switching of the CPU memory model, which can be useful to emulate other CPU architectures diff --git a/arch/arm64/include/asm/apple_cpufeature.h b/arch/arm64/include/asm/apple_cpufeature.h new file mode 100644 index 000000000000..4370d91ffa3e --- /dev/null +++ b/arch/arm64/include/asm/apple_cpufeature.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 + +#ifndef __ASM_APPLE_CPUFEATURES_H +#define __ASM_APPLE_CPUFEATURES_H + +#include <linux/bits.h> +#include <asm/sysreg.h> + +#define AIDR_APPLE_TSO_SHIFT 9 +#define AIDR_APPLE_TSO BIT(9) + +#define ACTLR_APPLE_TSO_SHIFT 1 +#define ACTLR_APPLE_TSO BIT(1) + +#endif diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 46ab37f8f4d8..a191000d88c2 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -911,7 +911,8 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) static __always_inline bool system_has_actlr_state(void) { - return false; + return IS_ENABLED(CONFIG_ARM64_ACTLR_STATE) && + alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE); } s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); diff --git a/arch/arm64/kernel/cpufeature_impdef.c b/arch/arm64/kernel/cpufeature_impdef.c index bb04a8e3d79d..9325d1eb12f4 100644 --- a/arch/arm64/kernel/cpufeature_impdef.c +++ b/arch/arm64/kernel/cpufeature_impdef.c @@ -4,8 +4,21 @@ */ #include <asm/cpufeature.h> +#include <asm/apple_cpufeature.h> #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL +static bool has_apple_feature(const struct arm64_cpu_capabilities *entry, int scope) +{ + u64 val; + WARN_ON(scope != SCOPE_SYSTEM); + + if (read_cpuid_implementor() != ARM_CPU_IMP_APPLE) + return false; + + val = read_sysreg(aidr_el1); + return cpufeature_matches(val, entry); +} + static bool has_tso_fixed(const struct arm64_cpu_capabilities *entry, int scope) { /* List of CPUs that always use the TSO memory model */ @@ -22,6 +35,16 @@ static bool has_tso_fixed(const struct arm64_cpu_capabilities *entry, int scope) static const struct arm64_cpu_capabilities arm64_impdef_features[] = { #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL + { + .desc = "TSO memory model (Apple)", + .capability = ARM64_HAS_TSO_APPLE, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_apple_feature, + .field_pos = AIDR_APPLE_TSO_SHIFT, + .field_width = 1, + .sign = FTR_UNSIGNED, + .min_field_value = 1, + }, { .desc = "TSO memory model (Fixed)", .capability = ARM64_HAS_TSO_FIXED, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 117f80e16aac..34a19ecfb630 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -44,6 +44,7 @@ #include <linux/memory_ordering_model.h> #include <asm/alternative.h> +#include <asm/apple_cpufeature.h> #include <asm/compat.h> #include <asm/cpufeature.h> #include <asm/cacheflush.h> @@ -522,6 +523,10 @@ void update_sctlr_el1(u64 sctlr) #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL int arch_prctl_mem_model_get(struct task_struct *t) { + if (alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE) && + t->thread.actlr & ACTLR_APPLE_TSO) + return PR_SET_MEM_MODEL_TSO; + return PR_SET_MEM_MODEL_DEFAULT; } @@ -531,6 +536,23 @@ int arch_prctl_mem_model_set(struct task_struct *t, unsigned long val) val == PR_SET_MEM_MODEL_TSO) return 0; + if (alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE)) { + WARN_ON(!system_has_actlr_state()); + + switch (val) { + case PR_SET_MEM_MODEL_TSO: + t->thread.actlr |= ACTLR_APPLE_TSO; + break; + case PR_SET_MEM_MODEL_DEFAULT: + t->thread.actlr &= ~ACTLR_APPLE_TSO; + break; + default: + return -EINVAL; + } + write_sysreg(t->thread.actlr, actlr_el1); + return 0; + } + if (val == PR_SET_MEM_MODEL_DEFAULT) return 0; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index daa6b9495402..62f9ca9ce44b 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -52,6 +52,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_TSO_APPLE HAS_TSO_FIXED HAS_VA52 HAS_VIRT_HOST_EXTN -- 2.44.0
WARNING: multiple messages have this Message-ID (diff)
From: Hector Martin <marcan@marcan.st> To: Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Marc Zyngier <maz@kernel.org>, Mark Rutland <mark.rutland@arm.com> Cc: Zayd Qumsieh <zayd_qumsieh@apple.com>, Justin Lu <ih_justin@apple.com>, Ryan Houdek <Houdek.Ryan@fex-emu.org>, Mark Brown <broonie@kernel.org>, Ard Biesheuvel <ardb@kernel.org>, Mateusz Guzik <mjguzik@gmail.com>, Anshuman Khandual <anshuman.khandual@arm.com>, Oliver Upton <oliver.upton@linux.dev>, Miguel Luis <miguel.luis@oracle.com>, Joey Gouly <joey.gouly@arm.com>, Christoph Paasch <cpaasch@apple.com>, Kees Cook <keescook@chromium.org>, Sami Tolvanen <samitolvanen@google.com>, Baoquan He <bhe@redhat.com>, Joel Granados <j.granados@samsung.com>, Dawei Li <dawei.li@shingroup.cn>, Andrew Morton <akpm@linux-foundation.org>, Florent Revest <revest@chromium.org>, David Hildenbrand <david@redhat.com>, Stefan Roesch <shr@devkernel.io>, Andy Chiu <andy.chiu@sifive.com>, Josh Triplett <josh@joshtriplett.org>, Oleg Nesterov <oleg@redhat.com>, Helge Deller <deller@gmx.de>, Zev Weiss <zev@bewilderbeest.net>, Ondrej Mosnacek <omosnace@redhat.com>, Miguel Ojeda <ojeda@kernel.org>, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux <asahi@lists.linux.dev>, Hector Martin <marcan@marcan.st> Subject: [PATCH 4/4] arm64: Implement Apple IMPDEF TSO memory model control Date: Thu, 11 Apr 2024 09:51:23 +0900 [thread overview] Message-ID: <20240411-tso-v1-4-754f11abfbff@marcan.st> (raw) In-Reply-To: <20240411-tso-v1-0-754f11abfbff@marcan.st> Apple CPUs may implement the TSO memory model as an optional configurable mode. This allows x86 emulators to simplify their load/store handling, greatly increasing performance. Expose this via the prctl PR_SET_MEM_MODEL_TSO mechanism. We use the Apple IMPDEF AIDR_EL1 register to check for the availability of TSO mode, and enable this codepath on all CPUs with an Apple implementer. This relies on the ACTLR_EL1 thread state scaffolding introduced earlier. Signed-off-by: Hector Martin <marcan@marcan.st> --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/apple_cpufeature.h | 15 +++++++++++++++ arch/arm64/include/asm/cpufeature.h | 3 ++- arch/arm64/kernel/cpufeature_impdef.c | 23 +++++++++++++++++++++++ arch/arm64/kernel/process.c | 22 ++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 6 files changed, 65 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9b3593b34cce..2f3eedd955c9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2167,6 +2167,8 @@ endif # ARM64_PSEUDO_NMI config ARM64_MEMORY_MODEL_CONTROL bool "Runtime memory model control" + default ARCH_APPLE + select ARM64_ACTLR_STATE help Some ARM64 CPUs support runtime switching of the CPU memory model, which can be useful to emulate other CPU architectures diff --git a/arch/arm64/include/asm/apple_cpufeature.h b/arch/arm64/include/asm/apple_cpufeature.h new file mode 100644 index 000000000000..4370d91ffa3e --- /dev/null +++ b/arch/arm64/include/asm/apple_cpufeature.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 + +#ifndef __ASM_APPLE_CPUFEATURES_H +#define __ASM_APPLE_CPUFEATURES_H + +#include <linux/bits.h> +#include <asm/sysreg.h> + +#define AIDR_APPLE_TSO_SHIFT 9 +#define AIDR_APPLE_TSO BIT(9) + +#define ACTLR_APPLE_TSO_SHIFT 1 +#define ACTLR_APPLE_TSO BIT(1) + +#endif diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 46ab37f8f4d8..a191000d88c2 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -911,7 +911,8 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) static __always_inline bool system_has_actlr_state(void) { - return false; + return IS_ENABLED(CONFIG_ARM64_ACTLR_STATE) && + alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE); } s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); diff --git a/arch/arm64/kernel/cpufeature_impdef.c b/arch/arm64/kernel/cpufeature_impdef.c index bb04a8e3d79d..9325d1eb12f4 100644 --- a/arch/arm64/kernel/cpufeature_impdef.c +++ b/arch/arm64/kernel/cpufeature_impdef.c @@ -4,8 +4,21 @@ */ #include <asm/cpufeature.h> +#include <asm/apple_cpufeature.h> #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL +static bool has_apple_feature(const struct arm64_cpu_capabilities *entry, int scope) +{ + u64 val; + WARN_ON(scope != SCOPE_SYSTEM); + + if (read_cpuid_implementor() != ARM_CPU_IMP_APPLE) + return false; + + val = read_sysreg(aidr_el1); + return cpufeature_matches(val, entry); +} + static bool has_tso_fixed(const struct arm64_cpu_capabilities *entry, int scope) { /* List of CPUs that always use the TSO memory model */ @@ -22,6 +35,16 @@ static bool has_tso_fixed(const struct arm64_cpu_capabilities *entry, int scope) static const struct arm64_cpu_capabilities arm64_impdef_features[] = { #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL + { + .desc = "TSO memory model (Apple)", + .capability = ARM64_HAS_TSO_APPLE, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_apple_feature, + .field_pos = AIDR_APPLE_TSO_SHIFT, + .field_width = 1, + .sign = FTR_UNSIGNED, + .min_field_value = 1, + }, { .desc = "TSO memory model (Fixed)", .capability = ARM64_HAS_TSO_FIXED, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 117f80e16aac..34a19ecfb630 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -44,6 +44,7 @@ #include <linux/memory_ordering_model.h> #include <asm/alternative.h> +#include <asm/apple_cpufeature.h> #include <asm/compat.h> #include <asm/cpufeature.h> #include <asm/cacheflush.h> @@ -522,6 +523,10 @@ void update_sctlr_el1(u64 sctlr) #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL int arch_prctl_mem_model_get(struct task_struct *t) { + if (alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE) && + t->thread.actlr & ACTLR_APPLE_TSO) + return PR_SET_MEM_MODEL_TSO; + return PR_SET_MEM_MODEL_DEFAULT; } @@ -531,6 +536,23 @@ int arch_prctl_mem_model_set(struct task_struct *t, unsigned long val) val == PR_SET_MEM_MODEL_TSO) return 0; + if (alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE)) { + WARN_ON(!system_has_actlr_state()); + + switch (val) { + case PR_SET_MEM_MODEL_TSO: + t->thread.actlr |= ACTLR_APPLE_TSO; + break; + case PR_SET_MEM_MODEL_DEFAULT: + t->thread.actlr &= ~ACTLR_APPLE_TSO; + break; + default: + return -EINVAL; + } + write_sysreg(t->thread.actlr, actlr_el1); + return 0; + } + if (val == PR_SET_MEM_MODEL_DEFAULT) return 0; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index daa6b9495402..62f9ca9ce44b 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -52,6 +52,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_TSO_APPLE HAS_TSO_FIXED HAS_VA52 HAS_VIRT_HOST_EXTN -- 2.44.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2024-04-11 0:51 UTC|newest] Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-04-11 0:51 [PATCH 0/4] arm64: Support the TSO memory model Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` [PATCH 1/4] prctl: Introduce PR_{SET,GET}_MEM_MODEL Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` [PATCH 2/4] arm64: Implement PR_{GET,SET}_MEM_MODEL for always-TSO CPUs Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` [PATCH 3/4] arm64: Introduce scaffolding to add ACTLR_EL1 to thread state Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` Hector Martin [this message] 2024-04-11 0:51 ` [PATCH 4/4] arm64: Implement Apple IMPDEF TSO memory model control Hector Martin 2024-04-11 1:37 ` [PATCH 0/4] arm64: Support the TSO memory model Neal Gompa 2024-04-11 1:37 ` Neal Gompa 2024-04-11 13:28 ` Will Deacon 2024-04-11 13:28 ` Will Deacon 2024-04-11 14:19 ` Hector Martin 2024-04-11 14:19 ` Hector Martin 2024-04-11 18:43 ` Hector Martin 2024-04-11 18:43 ` Hector Martin 2024-04-16 2:22 ` Zayd Qumsieh 2024-04-16 2:22 ` Zayd Qumsieh 2024-04-19 16:58 ` Will Deacon 2024-04-19 16:58 ` Will Deacon 2024-04-19 18:05 ` Catalin Marinas 2024-04-19 18:05 ` Catalin Marinas 2024-04-19 16:58 ` Will Deacon 2024-04-19 16:58 ` Will Deacon 2024-04-20 11:37 ` Marc Zyngier 2024-04-20 11:37 ` Marc Zyngier 2024-05-02 0:10 ` Zayd Qumsieh 2024-05-02 0:10 ` Zayd Qumsieh 2024-05-02 13:25 ` Marc Zyngier 2024-05-02 13:25 ` Marc Zyngier 2024-05-06 8:20 ` Jonas Oberhauser 2024-05-06 8:20 ` Jonas Oberhauser 2024-04-20 12:13 ` Eric Curtin 2024-04-20 12:13 ` Eric Curtin 2024-04-20 12:15 ` Eric Curtin 2024-04-20 12:15 ` Eric Curtin 2024-05-06 11:21 ` Sergio Lopez Pascual 2024-05-06 11:21 ` Sergio Lopez Pascual 2024-05-06 16:12 ` Marc Zyngier 2024-05-06 16:12 ` Marc Zyngier 2024-05-06 16:20 ` Eric Curtin 2024-05-06 16:20 ` Eric Curtin 2024-05-06 22:04 ` Sergio Lopez Pascual 2024-05-06 22:04 ` Sergio Lopez Pascual 2024-05-02 0:16 ` Zayd Qumsieh 2024-05-02 0:16 ` Zayd Qumsieh 2024-05-07 10:24 ` Alex Bennée 2024-05-07 10:24 ` Alex Bennée 2024-05-07 14:52 ` Ard Biesheuvel 2024-05-07 14:52 ` Ard Biesheuvel 2024-05-09 11:13 ` Catalin Marinas 2024-05-09 11:13 ` Catalin Marinas 2024-05-09 12:31 ` Neal Gompa 2024-05-09 12:31 ` Neal Gompa 2024-05-09 12:56 ` Catalin Marinas 2024-05-09 12:56 ` Catalin Marinas 2024-04-16 2:11 ` Zayd Qumsieh 2024-04-16 2:11 ` Zayd Qumsieh
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240411-tso-v1-4-754f11abfbff@marcan.st \ --to=marcan@marcan.st \ --cc=Houdek.Ryan@fex-emu.org \ --cc=akpm@linux-foundation.org \ --cc=andy.chiu@sifive.com \ --cc=anshuman.khandual@arm.com \ --cc=ardb@kernel.org \ --cc=asahi@lists.linux.dev \ --cc=bhe@redhat.com \ --cc=broonie@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=cpaasch@apple.com \ --cc=david@redhat.com \ --cc=dawei.li@shingroup.cn \ --cc=deller@gmx.de \ --cc=ih_justin@apple.com \ --cc=j.granados@samsung.com \ --cc=joey.gouly@arm.com \ --cc=josh@joshtriplett.org \ --cc=keescook@chromium.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mark.rutland@arm.com \ --cc=maz@kernel.org \ --cc=miguel.luis@oracle.com \ --cc=mjguzik@gmail.com \ --cc=ojeda@kernel.org \ --cc=oleg@redhat.com \ --cc=oliver.upton@linux.dev \ --cc=omosnace@redhat.com \ --cc=revest@chromium.org \ --cc=samitolvanen@google.com \ --cc=shr@devkernel.io \ --cc=will@kernel.org \ --cc=zayd_qumsieh@apple.com \ --cc=zev@bewilderbeest.net \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.