From: Alexandru Elisei <alexandru.elisei@arm.com>
To: Andrew Jones <andrew.jones@linux.dev>
Cc: pbonzini@redhat.com, thuth@redhat.com, kvm@vger.kernel.org,
kvmarm@lists.cs.columbia.edu, nikos.nikoleris@arm.com
Subject: Re: [kvm-unit-tests RFC PATCH 05/19] lib/alloc_phys: Remove locking
Date: Mon, 26 Sep 2022 16:04:36 +0100 [thread overview]
Message-ID: <YzG/ZPLRsH4qwfnJ@monolith.localdoman> (raw)
In-Reply-To: <20220920145952.fnftt2v46daigtdt@kamzik>
Hi,
On Tue, Sep 20, 2022 at 04:59:52PM +0200, Andrew Jones wrote:
> On Tue, Sep 20, 2022 at 02:20:48PM +0100, Alexandru Elisei wrote:
> > Hi,
> >
> > On Tue, Sep 20, 2022 at 10:45:53AM +0200, Andrew Jones wrote:
> > > On Tue, Aug 09, 2022 at 10:15:44AM +0100, Alexandru Elisei wrote:
> > > > With powerpc moving the page allocator, there are no architectures left
> > > > which use the physical allocator after the boot setup: arm, arm64,
> > > > s390x and powerpc drain the physical allocator to initialize the page
> > > > allocator; and x86 calls setup_vm() to drain the allocator for each of
> > > > the tests that allocate memory.
> > >
> > > Please put the motivation for this change in the commit message. I looked
> > > ahead at the next patch to find it, but I'm not sure I agree with it. We
> > > should be able to keep the locking even when used early, since we probably
> > > need our locking to be something we can use early elsewhere anyway.
> >
> > You are correct, the commit message doesn't explain why locking is removed,
> > which makes the commit confusing. I will try to do a better job for the
> > next iteration (if we decide to keep this patch).
> >
> > I removed locking because the physical allocator by the end of the series
> > will end up being used only by arm64 to create the idmap, which is done on
>
> If only arm, and no unit tests, needs the phys allocator, then it can be
> integrated with whatever arm is using it for and removed from the general
> lib.
I kept the allocator in lib because I thought that RISC-V might have an use
for it. Since it's a RISC architecture, I was thinking that it also might
require software cache management around enabling/disabling the MMU. But in
the end it's up to you, it would be easy to move the physical allocator to
lib/arm if you think that is best.
>
> > the boot CPU and with the MMU off. After that, the translation table
> > allocator functions will use the page allocator, which can be used
> > concurrently.
> >
> > Looking at the spinlock implementation, spin_lock() doesn't protect from
> > the concurrent accesses when the MMU is disabled (lock->v is
> > unconditionally set to 1). Which means that spin_lock() does not work (in
> > the sense that it doesn't protect against concurrent accesses) on the boot
> > path, which doesn't need a spinlock anyway, because no secondaries are
> > online secondaries. It also means that spinlocks don't work when
> > AUXINFO_MMU_OFF is set. So for the purpose of simplicity I preferred to
> > drop it entirely.
>
> If other architectures or unit tests have / could have uses for the
> phys allocator then we should either document that it doesn't have
> locks or keep the locks, and arm will just know that they don't work,
> but also that they don't need to for its purposes.
I will write a comment explaining the baked in assumptions for the
allocator.
>
> Finally, if we drop the locks and arm doesn't have any other places where
> we use locks without the MMU enabled, then we can change the lock
> implementation to not have the no-mmu fallback - maybe by switching to the
> generic implementation as the other architectures have done.
The architecture mandates that load-acquire/store-release instructions are
supported only on Normal memory (to be more precise, Inner Shareable, Inner
Write-Back, Outer Write-Back Normal memory with Read allocation hints and
Write allocation hints and not transient and Outer Shareable, Inner
Write-Back, Outer Write-Back Normal memory with Read allocation hints and
Write allocation hints and not transient, ARM DDI 0487H.a, pages B2-211 and
B2-212).
If the AUXINFO_MMU_OFF flag is set, kvm-unit-tests doesn't enable the MMU
at boot, which means that all tests can be run with the MMU disabled. In
this case, all memory is Device-nGnRnE (instead of Normal). By using an
implementation that doesn't take into account that spin_lock() might be
called with the MMU disabled, kvm-unit-tests will end up using exclusive
access instructions on memory which doesn't support it. This can have
various effects, all rather unpleasant, like causing an external abort or
treating the exclusive access instruction as a NOP (ARM DDI 0487H.a, page
B2-212).
Tested this on my rockpro64 board, kvm-unit-tests built from current
master, with the mmu_disabled() path removed from spin_lock() (and
AUXINFO_MMU_OFF flag set), all tests hang indefinitely, that's because
phys_alloc_init() uses a spinlock. It is conceivable that we could rework
the setup code to remove the usage of spinlocks, but it's still the matter
of tests needing one for synchronization. It's also the matter of the uart
needing one for puts. And report. And probably other places.
Out of curiosity, without setting the AUXINFO_MMU_OFF flag, I tried using
the generic version of the spinlock (I assume you mean the one from
lib/asm-generic/spinlock.h, changed lib/arm64/asm/spinlock.h to include the
above header), selftest-setup hangs without displaying anything before
phys_alloc_init(), I have no idea why that is.
In the current implementation, when AUXINFO_MMU_OFF is set, tests that
actually use more than one thread might end up being incorrect some of the
time because spin_lock() doesn't protect against concurrent accesses.
That's pretty bad, but I think the alternative off all tests hanging
indefinitely is worse.
In my opinion, the current spinlock implementation is incorrect when the
MMU is disabled, but using a generic implementation is worse. I guess
another thing to put on the TODO list. Arm ARM recommends Lamport’s Bakery
algorithm for mutual exclusion and we could try to implement that for the
MMU disabled case, but I don't see much interest at the moment in running
tests with the MMU disabled.
Thanks,
Alex
>
> Thanks,
> drew
next prev parent reply other threads:[~2022-09-26 16:15 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-09 9:15 [kvm-unit-tests RFC PATCH 00/19] arm/arm64: Rework cache maintenance at boot Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 01/19] Makefile: Define __ASSEMBLY__ for assembly files Alexandru Elisei
2022-08-09 12:36 ` Nikos Nikoleris
2022-09-20 8:11 ` Andrew Jones
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 02/19] lib/alloc_phys: Initialize align_min Alexandru Elisei
2022-09-20 8:20 ` Andrew Jones
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 03/19] lib/alloc_phys: Use phys_alloc_aligned_safe and rename it to memalign_early Alexandru Elisei
2022-09-20 8:27 ` Andrew Jones
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 04/19] powerpc: Use the page allocator Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 05/19] lib/alloc_phys: Remove locking Alexandru Elisei
2022-09-20 8:45 ` Andrew Jones
2022-09-20 13:20 ` Alexandru Elisei
2022-09-20 14:59 ` Andrew Jones
2022-09-26 15:04 ` Alexandru Elisei [this message]
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 06/19] lib/alloc_phys: Remove allocation accounting Alexandru Elisei
2022-09-20 8:40 ` Andrew Jones
2022-09-20 13:19 ` Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 07/19] arm/arm64: Mark the phys_end parameter as unused in setup_mmu() Alexandru Elisei
2022-09-20 8:58 ` Andrew Jones
2022-09-26 11:01 ` Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 08/19] arm/arm64: Use pgd_alloc() to allocate mmu_idmap Alexandru Elisei
2022-09-20 9:05 ` Andrew Jones
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 09/19] arm/arm64: Zero secondary CPUs' stack Alexandru Elisei
2022-08-09 12:56 ` Nikos Nikoleris
2022-08-10 9:42 ` Alexandru Elisei
2022-08-10 10:00 ` Nikos Nikoleris
2022-09-20 9:24 ` Andrew Jones
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 10/19] arm/arm64: Enable the MMU early Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 11/19] arm/arm64: Map the UART when creating the translation tables Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 12/19] arm/arm64: assembler.h: Replace size with end address for dcache_by_line_op Alexandru Elisei
2022-08-09 13:01 ` Nikos Nikoleris
2022-09-20 9:37 ` Andrew Jones
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 13/19] arm: page.h: Add missing libcflat.h include Alexandru Elisei
2022-09-20 9:39 ` Andrew Jones
2022-09-26 11:02 ` Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 14/19] arm/arm64: Add C functions for doing cache maintenance Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 15/19] lib/alloc_phys: Add callback to perform " Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 16/19] arm/arm64: Allocate secondaries' stack using the page allocator Alexandru Elisei
2022-09-20 9:58 ` Andrew Jones
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 17/19] arm/arm64: Configure secondaries' stack before enabling the MMU Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 18/19] arm/arm64: Perform dcache maintenance at boot Alexandru Elisei
2022-08-09 9:15 ` [kvm-unit-tests RFC PATCH 19/19] arm/arm64: Rework the cache maintenance in asm_mmu_disable Alexandru Elisei
2022-08-09 13:53 ` Nikos Nikoleris
2022-08-09 14:22 ` Alexandru Elisei
2022-08-09 15:53 ` Nikos Nikoleris
2022-08-09 16:53 ` Alexandru Elisei
2022-08-09 19:48 ` Nikos Nikoleris
2022-08-10 8:52 ` Alexandru Elisei
2022-08-09 9:49 ` [kvm-unit-tests RFC PATCH 00/19] arm/arm64: Rework cache maintenance at boot Alexandru Elisei
2023-11-05 10:16 ` Alexandru Elisei
2023-11-06 9:37 ` Shaoqin Huang
2023-11-07 9:01 ` Alexandru Elisei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YzG/ZPLRsH4qwfnJ@monolith.localdoman \
--to=alexandru.elisei@arm.com \
--cc=andrew.jones@linux.dev \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=nikos.nikoleris@arm.com \
--cc=pbonzini@redhat.com \
--cc=thuth@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).