From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B25313775C; Fri, 19 Apr 2024 16:58:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713545899; cv=none; b=qXxa+dMJd/UrjgEvYrJDY8GpxOXo6o7MUjEcqLMZja5hKRbkkvjVLk7saO6N6NSDeEPIkeqG5h4f+Ceeq4n7zz6yYc4w6jUSbrR1lgXFtm+9SVkN9JPFMUepUUxT3YuyvnYjVPUosCHsADj+9XtJQptFu03W+9iBC//HGHbuS7w= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713545899; c=relaxed/simple; bh=fY83VWfSR+QZWsluWoCGu1XQqdbdOrpZq6vnbqfPttE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=kCuP7nQLtnJ2cnC84D48R6HvwArwt/nDXBvT9aCXwA8/43aVyUiEwL5FU/Hkzq92mZ39zi0xLDpQNU1OWC1KLbG299dOYbcqcgEljiF/FITDnQbpknvqngB+cVmfJbfKP15DTWxDm7Z6BItMmIUJLTa6bzhjbS2wZaetuCdu3fI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tpJ0e1ui; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tpJ0e1ui" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DE430C072AA; Fri, 19 Apr 2024 16:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713545898; bh=fY83VWfSR+QZWsluWoCGu1XQqdbdOrpZq6vnbqfPttE=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=tpJ0e1uitz/zirsN92RbRzCEyXBo/Sp+jhhHLhOhV65B4G+SHDf9/nub1A1xCHEb6 wmxY7JqARYjd/T+800sl1l4b7c4Y4VxUwIjDB8bogUJKKC6SoQSFfENoWdVkGPW5C3 LxPniduOqhzKLZ1tQoM/jfl86SklOQ2uJD0VPqrsFhsP2j6bezrqzOE81U2LNZt6EL geKn0NxgW56ElmeUibatHBPpi1WzGd00eTaHwJ6y9zmvK29R0rY7NImLmdCQTpumvW KBzhB8y28ryyn93ds5LCMGlzm0+1EEaHt1DnJ8JftKnQ3cKJ7XfeZF18ajyGekdNnS 0cQbqLo53DHTQ== Date: Fri, 19 Apr 2024 17:58:09 +0100 From: Will Deacon To: Hector Martin Cc: Catalin Marinas , Marc Zyngier , Mark Rutland , Zayd Qumsieh , Justin Lu , Ryan Houdek , Mark Brown , Ard Biesheuvel , Mateusz Guzik , Anshuman Khandual , Oliver Upton , Miguel Luis , Joey Gouly , Christoph Paasch , Kees Cook , Sami Tolvanen , Baoquan He , Joel Granados , Dawei Li , Andrew Morton , Florent Revest , David Hildenbrand , Stefan Roesch , Andy Chiu , Josh Triplett , Oleg Nesterov , Helge Deller , Zev Weiss , Ondrej Mosnacek , Miguel Ojeda , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux Subject: Re: [PATCH 0/4] arm64: Support the TSO memory model Message-ID: <20240419165809.GA4020@willie-the-truck> References: <20240411-tso-v1-0-754f11abfbff@marcan.st> <20240411132853.GA26481@willie-the-truck> <28ab55b3-e699-4487-b332-f1f20a6b22a1@marcan.st> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <28ab55b3-e699-4487-b332-f1f20a6b22a1@marcan.st> User-Agent: Mutt/1.10.1 (2018-07-13) On Thu, Apr 11, 2024 at 11:19:13PM +0900, Hector Martin wrote: > On 2024/04/11 22:28, Will Deacon wrote: > > * Some binaries in a distribution exhibit instability which goes away > > in TSO mode, so a taskset-like program is used to run them with TSO > > enabled. > > Since the flag is cleared on execve, this third one isn't generally > possible as far as I know. Ah ok, I'd missed that. Thanks. > > In all these cases, we end up with native arm64 applications that will > > either fail to load or will crash in subtle ways on CPUs without the TSO > > feature. Assuming that the application cannot be fixed, a better > > approach would be to recompile using stronger instructions (e.g. > > LDAR/STLR) so that at least the resulting binary is portable. Now, it's > > true that some existing CPUs are TSO by design (this is a perfectly > > valid implementation of the arm64 memory model), but I think there's a > > big difference between quietly providing more ordering guarantees than > > software may be relying on and providing a mechanism to discover, > > request and ultimately rely upon the stronger behaviour. > > The problem is "just" using stronger instructions is much more > expensive, as emulators have demonstrated. If TSO didn't serve a > practical purpose I wouldn't be submitting this, but it does. This is > basically non-negotiable for x86 emulation; if this is rejected > upstream, it will forever live as a downstream patch used by the entire > gaming-on-Mac-Linux ecosystem (and this is an ecosystem we are very > explicitly targeting, given our efforts with microVMs for 4K page size > support and the upcoming Vulkan drivers). These microVMs sound quite interesting. What exactly are they? Are you running them under KVM? Ignoring the mechanism for the time being, would it solve your problem if you were able to run specific microVMs in TSO mode, or do you *really* need the VM to have finer-grained control than that? If the whole VM is running in TSO mode, then my concerns largely disappear, as that's indistinguishable from running on a hardware implementation that happens to be TSO. > That said, I have a pragmatic proposal here. The "fixed TSO" part of the > implementation should be harmless, since those CPUs would correctly run > poorly-written applications anyway so the API is moot. That leaves Apple > Silicon. Our native kernels are and likely always will be 16K page size, > due to a bunch of pain around 16K-only IOMMUs (4K kernels do boot > natively but with very broken functionality including no GPU > acceleration) plus performance differences that favor 16K. How about we > gate the TSO functionality to only be supported on 4K kernel builds? > This would make them only work in 4K VMs on Asahi Linux. We are very > explicitly discouraging people from trying to use the microVMs to work > around page size problems (which they can already do, another > fragmentation problem, anyway); any application which requires the 4K VM > to run that isn't an emulator is already clearly broken and advertising > that fact openly. So, adding TSO to this should be only a marginal risk > of further fragmentation, and it wouldn't allow apps to "sneakily" "just > work" on Apple machines by abusing TSO. I appreciate that you're trying to be constructive here, but I don't think we should tie this to the page size. It's an artifical limitation and I don't think it really addresses the underlying concerns that I have. Will From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62886C04FF6 for ; Fri, 19 Apr 2024 16:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NmnCGvguFROgSgk/6g3Dn6HaFoRsYf5/QOgND4pRCRg=; b=s6EKR++MQUU8tv JWG4guUjf0OcN+ymrJ/q6BuXo7Hl0jeIknL0WsrMEtQcj/wggVi9rFaYIQ6Ov3Rd6btD2zeAq4k12 jJmoxMt+qTm69FJIummG8asM5C158/qg+1Tf/7Av6qRpcSLZSrdzDdskQJ00LpBphSb8beDGjw136 DfQiQrfGozrGA9GYeBcMApgjWTCEDfDJOZxcPDRsIGDUE5h/Rg1BRWBVJ2iMtHIPVGwQ1JTv41rIp lu9TtM3wDAjgu35ELO2Cj10Y8UtVSiRD6rIX7Eui/iODnKKXyqmglD/tpXsuHeWNc8Ohj2WtVJImK L/3jUxreBEeR52N1+hJw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxrZB-00000006Su9-1TGn; Fri, 19 Apr 2024 16:58:25 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxrZ8-00000006Ss9-1To5 for linux-arm-kernel@lists.infradead.org; Fri, 19 Apr 2024 16:58:24 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 89618CE1A9D; Fri, 19 Apr 2024 16:58:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DE430C072AA; Fri, 19 Apr 2024 16:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713545898; bh=fY83VWfSR+QZWsluWoCGu1XQqdbdOrpZq6vnbqfPttE=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=tpJ0e1uitz/zirsN92RbRzCEyXBo/Sp+jhhHLhOhV65B4G+SHDf9/nub1A1xCHEb6 wmxY7JqARYjd/T+800sl1l4b7c4Y4VxUwIjDB8bogUJKKC6SoQSFfENoWdVkGPW5C3 LxPniduOqhzKLZ1tQoM/jfl86SklOQ2uJD0VPqrsFhsP2j6bezrqzOE81U2LNZt6EL geKn0NxgW56ElmeUibatHBPpi1WzGd00eTaHwJ6y9zmvK29R0rY7NImLmdCQTpumvW KBzhB8y28ryyn93ds5LCMGlzm0+1EEaHt1DnJ8JftKnQ3cKJ7XfeZF18ajyGekdNnS 0cQbqLo53DHTQ== Date: Fri, 19 Apr 2024 17:58:09 +0100 From: Will Deacon To: Hector Martin Cc: Catalin Marinas , Marc Zyngier , Mark Rutland , Zayd Qumsieh , Justin Lu , Ryan Houdek , Mark Brown , Ard Biesheuvel , Mateusz Guzik , Anshuman Khandual , Oliver Upton , Miguel Luis , Joey Gouly , Christoph Paasch , Kees Cook , Sami Tolvanen , Baoquan He , Joel Granados , Dawei Li , Andrew Morton , Florent Revest , David Hildenbrand , Stefan Roesch , Andy Chiu , Josh Triplett , Oleg Nesterov , Helge Deller , Zev Weiss , Ondrej Mosnacek , Miguel Ojeda , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux Subject: Re: [PATCH 0/4] arm64: Support the TSO memory model Message-ID: <20240419165809.GA4020@willie-the-truck> References: <20240411-tso-v1-0-754f11abfbff@marcan.st> <20240411132853.GA26481@willie-the-truck> <28ab55b3-e699-4487-b332-f1f20a6b22a1@marcan.st> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <28ab55b3-e699-4487-b332-f1f20a6b22a1@marcan.st> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240419_095823_003612_97212416 X-CRM114-Status: GOOD ( 33.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Apr 11, 2024 at 11:19:13PM +0900, Hector Martin wrote: > On 2024/04/11 22:28, Will Deacon wrote: > > * Some binaries in a distribution exhibit instability which goes away > > in TSO mode, so a taskset-like program is used to run them with TSO > > enabled. > > Since the flag is cleared on execve, this third one isn't generally > possible as far as I know. Ah ok, I'd missed that. Thanks. > > In all these cases, we end up with native arm64 applications that will > > either fail to load or will crash in subtle ways on CPUs without the TSO > > feature. Assuming that the application cannot be fixed, a better > > approach would be to recompile using stronger instructions (e.g. > > LDAR/STLR) so that at least the resulting binary is portable. Now, it's > > true that some existing CPUs are TSO by design (this is a perfectly > > valid implementation of the arm64 memory model), but I think there's a > > big difference between quietly providing more ordering guarantees than > > software may be relying on and providing a mechanism to discover, > > request and ultimately rely upon the stronger behaviour. > > The problem is "just" using stronger instructions is much more > expensive, as emulators have demonstrated. If TSO didn't serve a > practical purpose I wouldn't be submitting this, but it does. This is > basically non-negotiable for x86 emulation; if this is rejected > upstream, it will forever live as a downstream patch used by the entire > gaming-on-Mac-Linux ecosystem (and this is an ecosystem we are very > explicitly targeting, given our efforts with microVMs for 4K page size > support and the upcoming Vulkan drivers). These microVMs sound quite interesting. What exactly are they? Are you running them under KVM? Ignoring the mechanism for the time being, would it solve your problem if you were able to run specific microVMs in TSO mode, or do you *really* need the VM to have finer-grained control than that? If the whole VM is running in TSO mode, then my concerns largely disappear, as that's indistinguishable from running on a hardware implementation that happens to be TSO. > That said, I have a pragmatic proposal here. The "fixed TSO" part of the > implementation should be harmless, since those CPUs would correctly run > poorly-written applications anyway so the API is moot. That leaves Apple > Silicon. Our native kernels are and likely always will be 16K page size, > due to a bunch of pain around 16K-only IOMMUs (4K kernels do boot > natively but with very broken functionality including no GPU > acceleration) plus performance differences that favor 16K. How about we > gate the TSO functionality to only be supported on 4K kernel builds? > This would make them only work in 4K VMs on Asahi Linux. We are very > explicitly discouraging people from trying to use the microVMs to work > around page size problems (which they can already do, another > fragmentation problem, anyway); any application which requires the 4K VM > to run that isn't an emulator is already clearly broken and advertising > that fact openly. So, adding TSO to this should be only a marginal risk > of further fragmentation, and it wouldn't allow apps to "sneakily" "just > work" on Apple machines by abusing TSO. I appreciate that you're trying to be constructive here, but I don't think we should tie this to the page size. It's an artifical limitation and I don't think it really addresses the underlying concerns that I have. Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel