From: Will Deacon <will@kernel.org> To: Hector Martin <marcan@marcan.st> Cc: Catalin Marinas <catalin.marinas@arm.com>, Marc Zyngier <maz@kernel.org>, Mark Rutland <mark.rutland@arm.com>, Zayd Qumsieh <zayd_qumsieh@apple.com>, Justin Lu <ih_justin@apple.com>, Ryan Houdek <Houdek.Ryan@fex-emu.org>, Mark Brown <broonie@kernel.org>, Ard Biesheuvel <ardb@kernel.org>, Mateusz Guzik <mjguzik@gmail.com>, Anshuman Khandual <anshuman.khandual@arm.com>, Oliver Upton <oliver.upton@linux.dev>, Miguel Luis <miguel.luis@oracle.com>, Joey Gouly <joey.gouly@arm.com>, Christoph Paasch <cpaasch@apple.com>, Kees Cook <keescook@chromium.org>, Sami Tolvanen <samitolvanen@google.com>, Baoquan He <bhe@redhat.com>, Joel Granados <j.granados@samsung.com>, Dawei Li <dawei.li@shingroup.cn>, Andrew Morton <akpm@linux-foundation.org>, Florent Revest <revest@chromium.org>, David Hildenbrand <david@redhat.com>, Stefan Roesch <shr@devkernel.io>, Andy Chiu <andy.chiu@sifive.com>, Josh Triplett <josh@joshtriplett.org>, Oleg Nesterov <oleg@redhat.com>, Helge Deller <deller@gmx.de>, Zev Weiss <zev@bewilderbeest.net>, Ondrej Mosnacek <omosnace@redhat.com>, Miguel Ojeda <ojeda@kernel.org>, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux <asahi@lists.linux.dev> Subject: Re: [PATCH 0/4] arm64: Support the TSO memory model Date: Fri, 19 Apr 2024 17:58:09 +0100 [thread overview] Message-ID: <20240419165809.GA4020@willie-the-truck> (raw) In-Reply-To: <28ab55b3-e699-4487-b332-f1f20a6b22a1@marcan.st> On Thu, Apr 11, 2024 at 11:19:13PM +0900, Hector Martin wrote: > On 2024/04/11 22:28, Will Deacon wrote: > > * Some binaries in a distribution exhibit instability which goes away > > in TSO mode, so a taskset-like program is used to run them with TSO > > enabled. > > Since the flag is cleared on execve, this third one isn't generally > possible as far as I know. Ah ok, I'd missed that. Thanks. > > In all these cases, we end up with native arm64 applications that will > > either fail to load or will crash in subtle ways on CPUs without the TSO > > feature. Assuming that the application cannot be fixed, a better > > approach would be to recompile using stronger instructions (e.g. > > LDAR/STLR) so that at least the resulting binary is portable. Now, it's > > true that some existing CPUs are TSO by design (this is a perfectly > > valid implementation of the arm64 memory model), but I think there's a > > big difference between quietly providing more ordering guarantees than > > software may be relying on and providing a mechanism to discover, > > request and ultimately rely upon the stronger behaviour. > > The problem is "just" using stronger instructions is much more > expensive, as emulators have demonstrated. If TSO didn't serve a > practical purpose I wouldn't be submitting this, but it does. This is > basically non-negotiable for x86 emulation; if this is rejected > upstream, it will forever live as a downstream patch used by the entire > gaming-on-Mac-Linux ecosystem (and this is an ecosystem we are very > explicitly targeting, given our efforts with microVMs for 4K page size > support and the upcoming Vulkan drivers). These microVMs sound quite interesting. What exactly are they? Are you running them under KVM? Ignoring the mechanism for the time being, would it solve your problem if you were able to run specific microVMs in TSO mode, or do you *really* need the VM to have finer-grained control than that? If the whole VM is running in TSO mode, then my concerns largely disappear, as that's indistinguishable from running on a hardware implementation that happens to be TSO. > That said, I have a pragmatic proposal here. The "fixed TSO" part of the > implementation should be harmless, since those CPUs would correctly run > poorly-written applications anyway so the API is moot. That leaves Apple > Silicon. Our native kernels are and likely always will be 16K page size, > due to a bunch of pain around 16K-only IOMMUs (4K kernels do boot > natively but with very broken functionality including no GPU > acceleration) plus performance differences that favor 16K. How about we > gate the TSO functionality to only be supported on 4K kernel builds? > This would make them only work in 4K VMs on Asahi Linux. We are very > explicitly discouraging people from trying to use the microVMs to work > around page size problems (which they can already do, another > fragmentation problem, anyway); any application which requires the 4K VM > to run that isn't an emulator is already clearly broken and advertising > that fact openly. So, adding TSO to this should be only a marginal risk > of further fragmentation, and it wouldn't allow apps to "sneakily" "just > work" on Apple machines by abusing TSO. I appreciate that you're trying to be constructive here, but I don't think we should tie this to the page size. It's an artifical limitation and I don't think it really addresses the underlying concerns that I have. Will
WARNING: multiple messages have this Message-ID (diff)
From: Will Deacon <will@kernel.org> To: Hector Martin <marcan@marcan.st> Cc: Catalin Marinas <catalin.marinas@arm.com>, Marc Zyngier <maz@kernel.org>, Mark Rutland <mark.rutland@arm.com>, Zayd Qumsieh <zayd_qumsieh@apple.com>, Justin Lu <ih_justin@apple.com>, Ryan Houdek <Houdek.Ryan@fex-emu.org>, Mark Brown <broonie@kernel.org>, Ard Biesheuvel <ardb@kernel.org>, Mateusz Guzik <mjguzik@gmail.com>, Anshuman Khandual <anshuman.khandual@arm.com>, Oliver Upton <oliver.upton@linux.dev>, Miguel Luis <miguel.luis@oracle.com>, Joey Gouly <joey.gouly@arm.com>, Christoph Paasch <cpaasch@apple.com>, Kees Cook <keescook@chromium.org>, Sami Tolvanen <samitolvanen@google.com>, Baoquan He <bhe@redhat.com>, Joel Granados <j.granados@samsung.com>, Dawei Li <dawei.li@shingroup.cn>, Andrew Morton <akpm@linux-foundation.org>, Florent Revest <revest@chromium.org>, David Hildenbrand <david@redhat.com>, Stefan Roesch <shr@devkernel.io>, Andy Chiu <andy.chiu@sifive.com>, Josh Triplett <josh@joshtriplett.org>, Oleg Nesterov <oleg@redhat.com>, Helge Deller <deller@gmx.de>, Zev Weiss <zev@bewilderbeest.net>, Ondrej Mosnacek <omosnace@redhat.com>, Miguel Ojeda <ojeda@kernel.org>, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux <asahi@lists.linux.dev> Subject: Re: [PATCH 0/4] arm64: Support the TSO memory model Date: Fri, 19 Apr 2024 17:58:09 +0100 [thread overview] Message-ID: <20240419165809.GA4020@willie-the-truck> (raw) In-Reply-To: <28ab55b3-e699-4487-b332-f1f20a6b22a1@marcan.st> On Thu, Apr 11, 2024 at 11:19:13PM +0900, Hector Martin wrote: > On 2024/04/11 22:28, Will Deacon wrote: > > * Some binaries in a distribution exhibit instability which goes away > > in TSO mode, so a taskset-like program is used to run them with TSO > > enabled. > > Since the flag is cleared on execve, this third one isn't generally > possible as far as I know. Ah ok, I'd missed that. Thanks. > > In all these cases, we end up with native arm64 applications that will > > either fail to load or will crash in subtle ways on CPUs without the TSO > > feature. Assuming that the application cannot be fixed, a better > > approach would be to recompile using stronger instructions (e.g. > > LDAR/STLR) so that at least the resulting binary is portable. Now, it's > > true that some existing CPUs are TSO by design (this is a perfectly > > valid implementation of the arm64 memory model), but I think there's a > > big difference between quietly providing more ordering guarantees than > > software may be relying on and providing a mechanism to discover, > > request and ultimately rely upon the stronger behaviour. > > The problem is "just" using stronger instructions is much more > expensive, as emulators have demonstrated. If TSO didn't serve a > practical purpose I wouldn't be submitting this, but it does. This is > basically non-negotiable for x86 emulation; if this is rejected > upstream, it will forever live as a downstream patch used by the entire > gaming-on-Mac-Linux ecosystem (and this is an ecosystem we are very > explicitly targeting, given our efforts with microVMs for 4K page size > support and the upcoming Vulkan drivers). These microVMs sound quite interesting. What exactly are they? Are you running them under KVM? Ignoring the mechanism for the time being, would it solve your problem if you were able to run specific microVMs in TSO mode, or do you *really* need the VM to have finer-grained control than that? If the whole VM is running in TSO mode, then my concerns largely disappear, as that's indistinguishable from running on a hardware implementation that happens to be TSO. > That said, I have a pragmatic proposal here. The "fixed TSO" part of the > implementation should be harmless, since those CPUs would correctly run > poorly-written applications anyway so the API is moot. That leaves Apple > Silicon. Our native kernels are and likely always will be 16K page size, > due to a bunch of pain around 16K-only IOMMUs (4K kernels do boot > natively but with very broken functionality including no GPU > acceleration) plus performance differences that favor 16K. How about we > gate the TSO functionality to only be supported on 4K kernel builds? > This would make them only work in 4K VMs on Asahi Linux. We are very > explicitly discouraging people from trying to use the microVMs to work > around page size problems (which they can already do, another > fragmentation problem, anyway); any application which requires the 4K VM > to run that isn't an emulator is already clearly broken and advertising > that fact openly. So, adding TSO to this should be only a marginal risk > of further fragmentation, and it wouldn't allow apps to "sneakily" "just > work" on Apple machines by abusing TSO. I appreciate that you're trying to be constructive here, but I don't think we should tie this to the page size. It's an artifical limitation and I don't think it really addresses the underlying concerns that I have. Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2024-04-19 16:58 UTC|newest] Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-04-11 0:51 [PATCH 0/4] arm64: Support the TSO memory model Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` [PATCH 1/4] prctl: Introduce PR_{SET,GET}_MEM_MODEL Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` [PATCH 2/4] arm64: Implement PR_{GET,SET}_MEM_MODEL for always-TSO CPUs Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` [PATCH 3/4] arm64: Introduce scaffolding to add ACTLR_EL1 to thread state Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 0:51 ` [PATCH 4/4] arm64: Implement Apple IMPDEF TSO memory model control Hector Martin 2024-04-11 0:51 ` Hector Martin 2024-04-11 1:37 ` [PATCH 0/4] arm64: Support the TSO memory model Neal Gompa 2024-04-11 1:37 ` Neal Gompa 2024-04-11 13:28 ` Will Deacon 2024-04-11 13:28 ` Will Deacon 2024-04-11 14:19 ` Hector Martin 2024-04-11 14:19 ` Hector Martin 2024-04-11 18:43 ` Hector Martin 2024-04-11 18:43 ` Hector Martin 2024-04-16 2:22 ` Zayd Qumsieh 2024-04-16 2:22 ` Zayd Qumsieh 2024-04-19 16:58 ` Will Deacon 2024-04-19 16:58 ` Will Deacon 2024-04-19 18:05 ` Catalin Marinas 2024-04-19 18:05 ` Catalin Marinas 2024-04-19 16:58 ` Will Deacon [this message] 2024-04-19 16:58 ` Will Deacon 2024-04-20 11:37 ` Marc Zyngier 2024-04-20 11:37 ` Marc Zyngier 2024-05-02 0:10 ` Zayd Qumsieh 2024-05-02 0:10 ` Zayd Qumsieh 2024-05-02 13:25 ` Marc Zyngier 2024-05-02 13:25 ` Marc Zyngier 2024-05-06 8:20 ` Jonas Oberhauser 2024-05-06 8:20 ` Jonas Oberhauser 2024-04-20 12:13 ` Eric Curtin 2024-04-20 12:13 ` Eric Curtin 2024-04-20 12:15 ` Eric Curtin 2024-04-20 12:15 ` Eric Curtin 2024-05-06 11:21 ` Sergio Lopez Pascual 2024-05-06 11:21 ` Sergio Lopez Pascual 2024-05-06 16:12 ` Marc Zyngier 2024-05-06 16:12 ` Marc Zyngier 2024-05-06 16:20 ` Eric Curtin 2024-05-06 16:20 ` Eric Curtin 2024-05-06 22:04 ` Sergio Lopez Pascual 2024-05-06 22:04 ` Sergio Lopez Pascual 2024-05-02 0:16 ` Zayd Qumsieh 2024-05-02 0:16 ` Zayd Qumsieh 2024-05-07 10:24 ` Alex Bennée 2024-05-07 10:24 ` Alex Bennée 2024-05-07 14:52 ` Ard Biesheuvel 2024-05-07 14:52 ` Ard Biesheuvel 2024-05-09 11:13 ` Catalin Marinas 2024-05-09 11:13 ` Catalin Marinas 2024-05-09 12:31 ` Neal Gompa 2024-05-09 12:31 ` Neal Gompa 2024-05-09 12:56 ` Catalin Marinas 2024-05-09 12:56 ` Catalin Marinas 2024-04-16 2:11 ` Zayd Qumsieh 2024-04-16 2:11 ` Zayd Qumsieh
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240419165809.GA4020@willie-the-truck \ --to=will@kernel.org \ --cc=Houdek.Ryan@fex-emu.org \ --cc=akpm@linux-foundation.org \ --cc=andy.chiu@sifive.com \ --cc=anshuman.khandual@arm.com \ --cc=ardb@kernel.org \ --cc=asahi@lists.linux.dev \ --cc=bhe@redhat.com \ --cc=broonie@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=cpaasch@apple.com \ --cc=david@redhat.com \ --cc=dawei.li@shingroup.cn \ --cc=deller@gmx.de \ --cc=ih_justin@apple.com \ --cc=j.granados@samsung.com \ --cc=joey.gouly@arm.com \ --cc=josh@joshtriplett.org \ --cc=keescook@chromium.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=marcan@marcan.st \ --cc=mark.rutland@arm.com \ --cc=maz@kernel.org \ --cc=miguel.luis@oracle.com \ --cc=mjguzik@gmail.com \ --cc=ojeda@kernel.org \ --cc=oleg@redhat.com \ --cc=oliver.upton@linux.dev \ --cc=omosnace@redhat.com \ --cc=revest@chromium.org \ --cc=samitolvanen@google.com \ --cc=shr@devkernel.io \ --cc=zayd_qumsieh@apple.com \ --cc=zev@bewilderbeest.net \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.