From: Leonardo Bras <leobras@redhat.com>
To: Guo Ren <guoren@kernel.org>
Cc: paul.walmsley@sifive.com, anup@brainfault.org,
peterz@infradead.org, mingo@redhat.com, will@kernel.org,
palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com,
tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org,
rdunlap@infradead.org, catalin.marinas@arm.com,
conor.dooley@microchip.com, xiaoguang.xing@sophgo.com,
bjorn@rivosinc.com, alexghiti@rivosinc.com,
keescook@chromium.org, greentime.hu@sifive.com,
ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com,
wuwei2016@iscas.ac.cn, linux-arch@vger.kernel.org,
linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org,
kvm@vger.kernel.org, virtualization@lists.linux-foundation.org,
linux-csky@vger.kernel.org, Guo Ren <guoren@linux.alibaba.com>
Subject: Re: [PATCH V11 08/17] riscv: qspinlock: Add virt_spin_lock() support for KVM guest
Date: Tue, 19 Sep 2023 02:30:23 -0300 [thread overview]
Message-ID: <ZQkx75LgsM3-UfaN@redhat.com> (raw)
In-Reply-To: <CAJF2gTR6USUQc7=TmFx+8HfhKa8whzb9qtNjLKR_FPZzN656Zg@mail.gmail.com>
On Sun, Sep 17, 2023 at 11:12:31PM +0800, Guo Ren wrote:
> On Thu, Sep 14, 2023 at 4:02 PM Leonardo Bras <leobras@redhat.com> wrote:
> >
> > On Sun, Sep 10, 2023 at 04:29:02AM -0400, guoren@kernel.org wrote:
> > > From: Guo Ren <guoren@linux.alibaba.com>
> > >
> > > Add a static key controlling whether virt_spin_lock() should be
> > > called or not. When running on bare metal set the new key to
> > > false.
> > >
> > > The KVM guests fall back to a Test-and-Set spinlock, because fair
> > > locks have horrible lock 'holder' preemption issues. The
> > > virt_spin_lock_key would shortcut for the
> > > queued_spin_lock_slowpath() function that allow virt_spin_lock to
> > > hijack it.
> > >
> > > Signed-off-by: Guo Ren <guoren@linux.alibaba.com>
> > > Signed-off-by: Guo Ren <guoren@kernel.org>
> > > ---
> > > .../admin-guide/kernel-parameters.txt | 4 +++
> > > arch/riscv/include/asm/sbi.h | 8 +++++
> > > arch/riscv/include/asm/spinlock.h | 22 ++++++++++++++
> > > arch/riscv/kernel/sbi.c | 2 +-
> > > arch/riscv/kernel/setup.c | 30 ++++++++++++++++++-
> > > 5 files changed, 64 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > > index 61cacb8dfd0e..f75bedc50e00 100644
> > > --- a/Documentation/admin-guide/kernel-parameters.txt
> > > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > > @@ -3927,6 +3927,10 @@
> > > no_uaccess_flush
> > > [PPC] Don't flush the L1-D cache after accessing user data.
> > >
> > > + no_virt_spin [RISC-V] Disable virt_spin_lock in KVM guest to use
> > > + native_queued_spinlock when the nopvspin option is enabled.
> > > + This would help vcpu=pcpu scenarios.
> > > +
> > > novmcoredd [KNL,KDUMP]
> > > Disable device dump. Device dump allows drivers to
> > > append dump data to vmcore so you can collect driver
> > > diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h
> > > index 501e06e52078..e0233b3d7a5f 100644
> > > --- a/arch/riscv/include/asm/sbi.h
> > > +++ b/arch/riscv/include/asm/sbi.h
> > > @@ -50,6 +50,13 @@ enum sbi_ext_base_fid {
> > > SBI_EXT_BASE_GET_MIMPID,
> > > };
> > >
> > > +enum sbi_ext_base_impl_id {
> > > + SBI_EXT_BASE_IMPL_ID_BBL = 0,
> > > + SBI_EXT_BASE_IMPL_ID_OPENSBI,
> > > + SBI_EXT_BASE_IMPL_ID_XVISOR,
> > > + SBI_EXT_BASE_IMPL_ID_KVM,
> > > +};
> > > +
> > > enum sbi_ext_time_fid {
> > > SBI_EXT_TIME_SET_TIMER = 0,
> > > };
> > > @@ -269,6 +276,7 @@ int sbi_console_getchar(void);
> > > long sbi_get_mvendorid(void);
> > > long sbi_get_marchid(void);
> > > long sbi_get_mimpid(void);
> > > +long sbi_get_firmware_id(void);
> > > void sbi_set_timer(uint64_t stime_value);
> > > void sbi_shutdown(void);
> > > void sbi_send_ipi(unsigned int cpu);
> > > diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h
> > > index 8ea0fee80652..6b38d6616f14 100644
> > > --- a/arch/riscv/include/asm/spinlock.h
> > > +++ b/arch/riscv/include/asm/spinlock.h
> > > @@ -4,6 +4,28 @@
> > > #define __ASM_RISCV_SPINLOCK_H
> > >
> > > #ifdef CONFIG_QUEUED_SPINLOCKS
> > > +/*
> > > + * The KVM guests fall back to a Test-and-Set spinlock, because fair locks
> > > + * have horrible lock 'holder' preemption issues. The virt_spin_lock_key
> > > + * would shortcut for the queued_spin_lock_slowpath() function that allow
> > > + * virt_spin_lock to hijack it.
> > > + */
> > > +DECLARE_STATIC_KEY_TRUE(virt_spin_lock_key);
> > > +
> > > +#define virt_spin_lock virt_spin_lock
> > > +static inline bool virt_spin_lock(struct qspinlock *lock)
> > > +{
> > > + if (!static_branch_likely(&virt_spin_lock_key))
> > > + return false;
> > > +
> > > + do {
> > > + while (atomic_read(&lock->val) != 0)
> > > + cpu_relax();
> > > + } while (atomic_cmpxchg(&lock->val, 0, _Q_LOCKED_VAL) != 0);
> > > +
> > > + return true;
> > > +}
> > > +
> > > #define _Q_PENDING_LOOPS (1 << 9)
> > > #endif
> > >
> > > diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c
> > > index 88eea3a99ee0..cdd45edc8db4 100644
> > > --- a/arch/riscv/kernel/sbi.c
> > > +++ b/arch/riscv/kernel/sbi.c
> > > @@ -555,7 +555,7 @@ static inline long sbi_get_spec_version(void)
> > > return __sbi_base_ecall(SBI_EXT_BASE_GET_SPEC_VERSION);
> > > }
> > >
> > > -static inline long sbi_get_firmware_id(void)
> > > +long sbi_get_firmware_id(void)
> > > {
> > > return __sbi_base_ecall(SBI_EXT_BASE_GET_IMP_ID);
> > > }
> > > diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
> > > index 0f084f037651..c57d15b05160 100644
> > > --- a/arch/riscv/kernel/setup.c
> > > +++ b/arch/riscv/kernel/setup.c
> > > @@ -26,6 +26,7 @@
> > > #include <asm/alternative.h>
> > > #include <asm/cacheflush.h>
> > > #include <asm/cpu_ops.h>
> > > +#include <asm/cpufeature.h>
> > > #include <asm/early_ioremap.h>
> > > #include <asm/pgtable.h>
> > > #include <asm/setup.h>
> > > @@ -283,16 +284,43 @@ DEFINE_STATIC_KEY_TRUE(combo_qspinlock_key);
> > > EXPORT_SYMBOL(combo_qspinlock_key);
> > > #endif
> > >
> > > +#ifdef CONFIG_QUEUED_SPINLOCKS
> > > +static bool no_virt_spin_key = false;
> >
> > I suggest no _key, also there is no need for "= false".
> > To be consistent with enable_qspinlock, I also suggest
> > adding __ro_after_init:
> >
> > static bool no_virt_spin __ro_after_init;
> okay.
>
> >
> >
> >
> > > +DEFINE_STATIC_KEY_TRUE(virt_spin_lock_key);
> > > +
> > > +static int __init no_virt_spin_setup(char *p)
> > > +{
> > > + no_virt_spin_key = true;
> > > +
> > > + return 0;
> > > +}
> > > +early_param("no_virt_spin", no_virt_spin_setup);
> > > +
> > > +static void __init virt_spin_lock_init(void)
> > > +{
> > > + if (sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM ||
> > > + no_virt_spin_key)
> > > + static_branch_disable(&virt_spin_lock_key);
> > > + else
> > > + pr_info("Enable virt_spin_lock\n");
> > > +}
> > > +#endif
> > > +
> >
> > A new virt_no_spin kernel parameter was introduced, but without
> > CONFIG_QUEUED_SPINLOCKS it will silently fail.
> >
> > I would suggest an #else clause here with a function to print an error /
> > warning message about no_virt_spin being invalid in this scenario.
> > It will probably help future debugging.
> If CONFIG_QUEUED_SPINLOCKS=n, no_virt_spin should be quiet. The
> no_virt_spin is one path of qspinlock.
IIUC having no_virt_spin being passed as parameter to a kernel with
CONFIG_QUEUED_SPINLOCKS=n is not supposed to have any warning this
parameter is useless.
I was just thinking it would be nice to have this warning during debugging,
but if it's standard practice then I am ok with this.
>
> >
> >
> > > static void __init riscv_spinlock_init(void)
> > > {
> > > #ifdef CONFIG_RISCV_COMBO_SPINLOCKS
> > > - if (!enable_qspinlock_key) {
> > > + if (!enable_qspinlock_key &&
> > > + (sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM)) {
> > > static_branch_disable(&combo_qspinlock_key);
> > > pr_info("Ticket spinlock: enabled\n");
> > > } else {
> > > pr_info("Queued spinlock: enabled\n");
> > > }
> > > #endif
> > > +
> > > +#ifdef CONFIG_QUEUED_SPINLOCKS
> > > + virt_spin_lock_init();
> > > +#endif
> > > }
> > >
> > > extern void __init init_rt_signal_env(void);
> > > --
> > > 2.36.1
> > >
> >
> > I am probably missing something out, but it looks to me that this patch is
> > causing 2 different changes:
> > 1 - Enabling no_virt_spin parameter
> > 2 - Disabling queued spinlocks for some firmware_id
> >
> > Wouldn't be better to split those changes in multiple patches?
> > Or am I missing the point on why they need to be together?
^ Want your input on this
Thanks!
Leo
> >
> > Thanks!
> > Leo
> >
>
>
> --
> Best Regards
> Guo Ren
>
next prev parent reply other threads:[~2023-09-19 5:31 UTC|newest]
Thread overview: 105+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-10 8:28 [PATCH V11 00/17] riscv: Add Native/Paravirt qspinlock support guoren
2023-09-10 8:28 ` [PATCH V11 01/17] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock guoren
2023-09-11 19:05 ` Leonardo Brás
2023-09-13 1:55 ` Guo Ren
2023-09-13 7:59 ` Leonardo Bras
2023-09-10 8:28 ` [PATCH V11 02/17] asm-generic: ticket-lock: Move into ticket_spinlock.h guoren
2023-09-13 8:15 ` Leonardo Bras
2023-09-10 8:28 ` [PATCH V11 03/17] riscv: Use Zicbop in arch_xchg when available guoren
2023-09-13 8:49 ` Leonardo Bras
2023-09-15 12:36 ` Guo Ren
2023-09-16 1:25 ` Leonardo Bras
2023-09-17 14:34 ` Guo Ren
2023-09-19 5:13 ` Leonardo Bras
2023-09-19 7:53 ` Guo Ren
2023-09-19 14:38 ` Leonardo Bras
2023-09-14 13:47 ` Andrew Jones
2023-09-15 8:22 ` Leonardo Bras
2023-09-15 11:07 ` Andrew Jones
2023-09-15 11:26 ` Conor Dooley
2023-09-15 12:22 ` Andrew Jones
2023-09-15 12:42 ` Conor Dooley
2023-09-16 0:05 ` Conor Dooley
2023-09-15 20:32 ` Leonardo Bras
2023-09-14 14:25 ` Andrew Jones
2023-09-14 14:47 ` Andrew Jones
2023-09-15 11:37 ` Conor Dooley
2023-09-15 12:14 ` Andrew Jones
2023-09-15 12:53 ` Conor Dooley
2023-12-31 8:29 ` guoren
2023-09-10 8:28 ` [PATCH V11 04/17] locking/qspinlock: Improve xchg_tail for number of cpus >= 16k guoren
2023-09-11 2:35 ` Waiman Long
2023-09-11 3:09 ` Guo Ren
2023-09-11 13:03 ` Waiman Long
2023-09-12 1:10 ` Guo Ren
2023-09-13 8:55 ` Leonardo Bras
2023-09-13 12:52 ` Guo Ren
2023-09-13 13:06 ` Waiman Long
2023-09-14 3:45 ` Guo Ren
2023-09-10 8:28 ` [PATCH V11 05/17] riscv: qspinlock: Add basic queued_spinlock support guoren
2023-09-13 20:28 ` Leonardo Bras
2023-09-14 4:46 ` Guo Ren
2023-09-14 9:43 ` Leonardo Bras
2023-09-15 2:10 ` Guo Ren
2023-09-15 9:08 ` Leonardo Bras
2023-09-17 15:02 ` Guo Ren
2023-09-19 5:20 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 06/17] riscv: qspinlock: Introduce combo spinlock guoren
2023-09-10 11:06 ` Guo Ren
2023-09-13 20:37 ` Leonardo Bras
2023-09-13 20:49 ` Leonardo Bras
2023-09-14 4:49 ` Guo Ren
2023-09-14 7:17 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 07/17] riscv: qspinlock: Introduce qspinlock param for command line guoren
2023-09-11 15:22 ` Waiman Long
2023-09-12 1:06 ` Guo Ren
2023-09-11 15:34 ` Waiman Long
2023-09-12 1:08 ` Guo Ren
2023-09-14 7:32 ` Leonardo Bras
2023-09-14 17:23 ` Waiman Long
2023-09-10 8:29 ` [PATCH V11 08/17] riscv: qspinlock: Add virt_spin_lock() support for KVM guest guoren
2023-09-14 8:02 ` Leonardo Bras
2023-09-17 15:12 ` Guo Ren
2023-09-19 5:30 ` Leonardo Bras [this message]
2023-09-19 8:04 ` Guo Ren
2023-09-19 14:40 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 09/17] riscv: qspinlock: errata: Add ERRATA_THEAD_WRITE_ONCE fixup guoren
2023-09-14 8:32 ` Leonardo Bras
2023-09-17 15:15 ` Guo Ren
2023-09-19 5:34 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 10/17] riscv: qspinlock: errata: Enable qspinlock for T-HEAD processors guoren
2023-09-14 9:36 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 11/17] RISC-V: paravirt: pvqspinlock: Add paravirt qspinlock skeleton guoren
2023-09-15 5:42 ` Leonardo Bras
2023-09-17 14:58 ` Guo Ren
2023-09-19 5:43 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 12/17] RISC-V: paravirt: pvqspinlock: Add nopvspin kernel parameter guoren
2023-09-15 6:05 ` Leonardo Bras
2023-09-17 15:03 ` Guo Ren
2023-09-19 5:44 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 13/17] RISC-V: paravirt: pvqspinlock: Add SBI implementation guoren
2023-09-15 6:23 ` Leonardo Bras
2023-09-17 15:06 ` Guo Ren
2023-09-19 5:45 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 14/17] RISC-V: paravirt: pvqspinlock: Add kconfig entry guoren
2023-09-15 6:25 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 15/17] RISC-V: paravirt: pvqspinlock: Add trace point for pv_kick/wait guoren
2023-09-15 6:33 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 16/17] RISC-V: paravirt: pvqspinlock: KVM: Add paravirt qspinlock skeleton guoren
2023-09-15 6:46 ` Leonardo Bras
2023-09-10 8:29 ` [PATCH V11 17/17] RISC-V: paravirt: pvqspinlock: KVM: Implement kvm_sbi_ext_pvlock_kick_cpu() guoren
2023-09-15 6:52 ` Leonardo Bras
2023-09-10 8:58 ` [PATCH V11 00/17] riscv: Add Native/Paravirt qspinlock support Conor Dooley
2023-09-10 9:16 ` Guo Ren
2023-09-10 9:20 ` Guo Ren
2023-09-10 9:31 ` Conor Dooley
2023-09-10 9:49 ` Guo Ren
2023-09-10 19:45 ` Conor Dooley
2023-09-11 3:36 ` Guo Ren
2023-09-11 12:52 ` Conor Dooley
2023-09-12 1:33 ` Guo Ren
2023-09-12 8:07 ` Conor Dooley
2023-09-12 10:58 ` Guo Ren
2023-11-06 20:42 ` Leonardo Bras
2023-11-12 4:23 ` Guo Ren
2023-11-13 10:19 ` Leonardo Bras Soares Passos
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZQkx75LgsM3-UfaN@redhat.com \
--to=leobras@redhat.com \
--cc=ajones@ventanamicro.com \
--cc=alexghiti@rivosinc.com \
--cc=anup@brainfault.org \
--cc=bjorn@rivosinc.com \
--cc=boqun.feng@gmail.com \
--cc=catalin.marinas@arm.com \
--cc=conor.dooley@microchip.com \
--cc=greentime.hu@sifive.com \
--cc=guoren@kernel.org \
--cc=guoren@linux.alibaba.com \
--cc=jszhang@kernel.org \
--cc=keescook@chromium.org \
--cc=kvm@vger.kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-csky@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=longman@redhat.com \
--cc=mingo@redhat.com \
--cc=palmer@rivosinc.com \
--cc=paul.walmsley@sifive.com \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=rdunlap@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=virtualization@lists.linux-foundation.org \
--cc=wefu@redhat.com \
--cc=will@kernel.org \
--cc=wuwei2016@iscas.ac.cn \
--cc=xiaoguang.xing@sophgo.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).