From: guoren@kernel.org To: guoren@kernel.org, anup.patel@wdc.com, palmerdabbelt@google.com, arnd@arndb.de, wens@csie.org, maxime@cerno.tech, drew@beagleboard.org, liush@allwinnertech.com, lazyparser@gmail.com, wefu@redhat.com Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-sunxi@lists.linux.dev, Guo Ren <guoren@linux.alibaba.com>, Christoph Hellwig <hch@lst.de>, Atish Patra <atish.patra@wdc.com> Subject: [PATCH V5 3/3] riscv: tlbflush: Optimize coding convention Date: Sun, 6 Jun 2021 09:04:01 +0000 [thread overview] Message-ID: <1622970249-50770-7-git-send-email-guoren@kernel.org> (raw) In-Reply-To: <1622970249-50770-1-git-send-email-guoren@kernel.org> From: Guo Ren <guoren@linux.alibaba.com> Passing the mm_struct as the first argument, as we can derive both the cpumask and asid from it instead of doing that in the callers. But more importantly, the static branch check can be moved deeper into the code to avoid a lot of duplication. Also add FIXME comment on the non-ASID code switches to a global flush once flushing more than a single page. Link: https://lore.kernel.org/linux-riscv/CAJF2gTQpDYtEdw6ZrTVZUYqxGdhLPs25RjuUiQtz=xN2oKs2fw@mail.gmail.com/T/#m30f7e8d02361f21f709bc3357b9f6ead1d47ed43 Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-Developed-by: Christoph Hellwig <hch@lst.de> Cc: Christoph Hellwig <hch@lst.de> Cc: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Anup Patel <anup.patel@wdc.com> Cc: Atish Patra <atish.patra@wdc.com> --- arch/riscv/mm/tlbflush.c | 91 ++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 50 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 87b4e52..facca6e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -12,56 +12,59 @@ void flush_tlb_all(void) } /* - * This function must not be called with cmask being null. + * This function must not be called with mm_cpumask(mm) being null. * Kernel may panic if cmask is NULL. */ -static void __sbi_tlb_flush_range(struct cpumask *cmask, unsigned long start, +static void __sbi_tlb_flush_range(struct mm_struct *mm, + unsigned long start, unsigned long size) { + struct cpumask *cmask = mm_cpumask(mm); struct cpumask hmask; unsigned int cpuid; + bool local; if (cpumask_empty(cmask)) return; cpuid = get_cpu(); - if (cpumask_any_but(cmask, cpuid) >= nr_cpu_ids) { - /* local cpu is the only cpu present in cpumask */ - if (size <= PAGE_SIZE) - local_flush_tlb_page(start); - else - local_flush_tlb_all(); - } else { - riscv_cpuid_to_hartid_mask(cmask, &hmask); - sbi_remote_sfence_vma(cpumask_bits(&hmask), start, size); - } + /* + * check if the tlbflush needs to be sent to other CPUs, local + * cpu is the only cpu present in cpumask. + */ + local = !(cpumask_any_but(cmask, cpuid) < nr_cpu_ids); - put_cpu(); -} - -static void __sbi_tlb_flush_range_asid(struct cpumask *cmask, - unsigned long start, - unsigned long size, - unsigned long asid) -{ - struct cpumask hmask; - unsigned int cpuid; - - if (cpumask_empty(cmask)) - return; - - cpuid = get_cpu(); + if (static_branch_likely(&use_asid_allocator)) { + unsigned long asid = atomic_long_read(&mm->context.id); - if (cpumask_any_but(cmask, cpuid) >= nr_cpu_ids) { - if (size == -1) - local_flush_tlb_all_asid(asid); - else - local_flush_tlb_range_asid(start, size, asid); + if (likely(local)) { + if (size == -1) + local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_asid(start, size, asid); + } else { + riscv_cpuid_to_hartid_mask(cmask, &hmask); + sbi_remote_sfence_vma_asid(cpumask_bits(&hmask), + start, size, asid); + } } else { - riscv_cpuid_to_hartid_mask(cmask, &hmask); - sbi_remote_sfence_vma_asid(cpumask_bits(&hmask), - start, size, asid); + if (likely(local)) { + /* + * FIXME: The non-ASID code switches to a global flush + * once flushing more than a single page. It's made by + * commit 6efb16b1d551 (RISC-V: Issue a tlb page flush + * if possible). + */ + if (size <= PAGE_SIZE) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); + } else { + riscv_cpuid_to_hartid_mask(cmask, &hmask); + sbi_remote_sfence_vma(cpumask_bits(&hmask), + start, size); + } } put_cpu(); @@ -69,28 +72,16 @@ static void __sbi_tlb_flush_range_asid(struct cpumask *cmask, void flush_tlb_mm(struct mm_struct *mm) { - if (static_branch_unlikely(&use_asid_allocator)) - __sbi_tlb_flush_range_asid(mm_cpumask(mm), 0, -1, - atomic_long_read(&mm->context.id)); - else - __sbi_tlb_flush_range(mm_cpumask(mm), 0, -1); + __sbi_tlb_flush_range(mm, 0, -1); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - if (static_branch_unlikely(&use_asid_allocator)) - __sbi_tlb_flush_range_asid(mm_cpumask(vma->vm_mm), addr, PAGE_SIZE, - atomic_long_read(&vma->vm_mm->context.id)); - else - __sbi_tlb_flush_range(mm_cpumask(vma->vm_mm), addr, PAGE_SIZE); + __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - if (static_branch_unlikely(&use_asid_allocator)) - __sbi_tlb_flush_range_asid(mm_cpumask(vma->vm_mm), start, end - start, - atomic_long_read(&vma->vm_mm->context.id)); - else - __sbi_tlb_flush_range(mm_cpumask(vma->vm_mm), start, end - start); + __sbi_tlb_flush_range(vma->vm_mm, start, end - start); } -- 2.7.4
WARNING: multiple messages have this Message-ID (diff)
From: guoren@kernel.org To: guoren@kernel.org, anup.patel@wdc.com, palmerdabbelt@google.com, arnd@arndb.de, wens@csie.org, maxime@cerno.tech, drew@beagleboard.org, liush@allwinnertech.com, lazyparser@gmail.com, wefu@redhat.com Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-sunxi@lists.linux.dev, Guo Ren <guoren@linux.alibaba.com>, Christoph Hellwig <hch@lst.de>, Atish Patra <atish.patra@wdc.com> Subject: [PATCH V5 3/3] riscv: tlbflush: Optimize coding convention Date: Sun, 6 Jun 2021 09:04:01 +0000 [thread overview] Message-ID: <1622970249-50770-7-git-send-email-guoren@kernel.org> (raw) In-Reply-To: <1622970249-50770-1-git-send-email-guoren@kernel.org> From: Guo Ren <guoren@linux.alibaba.com> Passing the mm_struct as the first argument, as we can derive both the cpumask and asid from it instead of doing that in the callers. But more importantly, the static branch check can be moved deeper into the code to avoid a lot of duplication. Also add FIXME comment on the non-ASID code switches to a global flush once flushing more than a single page. Link: https://lore.kernel.org/linux-riscv/CAJF2gTQpDYtEdw6ZrTVZUYqxGdhLPs25RjuUiQtz=xN2oKs2fw@mail.gmail.com/T/#m30f7e8d02361f21f709bc3357b9f6ead1d47ed43 Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-Developed-by: Christoph Hellwig <hch@lst.de> Cc: Christoph Hellwig <hch@lst.de> Cc: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Anup Patel <anup.patel@wdc.com> Cc: Atish Patra <atish.patra@wdc.com> --- arch/riscv/mm/tlbflush.c | 91 ++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 50 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 87b4e52..facca6e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -12,56 +12,59 @@ void flush_tlb_all(void) } /* - * This function must not be called with cmask being null. + * This function must not be called with mm_cpumask(mm) being null. * Kernel may panic if cmask is NULL. */ -static void __sbi_tlb_flush_range(struct cpumask *cmask, unsigned long start, +static void __sbi_tlb_flush_range(struct mm_struct *mm, + unsigned long start, unsigned long size) { + struct cpumask *cmask = mm_cpumask(mm); struct cpumask hmask; unsigned int cpuid; + bool local; if (cpumask_empty(cmask)) return; cpuid = get_cpu(); - if (cpumask_any_but(cmask, cpuid) >= nr_cpu_ids) { - /* local cpu is the only cpu present in cpumask */ - if (size <= PAGE_SIZE) - local_flush_tlb_page(start); - else - local_flush_tlb_all(); - } else { - riscv_cpuid_to_hartid_mask(cmask, &hmask); - sbi_remote_sfence_vma(cpumask_bits(&hmask), start, size); - } + /* + * check if the tlbflush needs to be sent to other CPUs, local + * cpu is the only cpu present in cpumask. + */ + local = !(cpumask_any_but(cmask, cpuid) < nr_cpu_ids); - put_cpu(); -} - -static void __sbi_tlb_flush_range_asid(struct cpumask *cmask, - unsigned long start, - unsigned long size, - unsigned long asid) -{ - struct cpumask hmask; - unsigned int cpuid; - - if (cpumask_empty(cmask)) - return; - - cpuid = get_cpu(); + if (static_branch_likely(&use_asid_allocator)) { + unsigned long asid = atomic_long_read(&mm->context.id); - if (cpumask_any_but(cmask, cpuid) >= nr_cpu_ids) { - if (size == -1) - local_flush_tlb_all_asid(asid); - else - local_flush_tlb_range_asid(start, size, asid); + if (likely(local)) { + if (size == -1) + local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_asid(start, size, asid); + } else { + riscv_cpuid_to_hartid_mask(cmask, &hmask); + sbi_remote_sfence_vma_asid(cpumask_bits(&hmask), + start, size, asid); + } } else { - riscv_cpuid_to_hartid_mask(cmask, &hmask); - sbi_remote_sfence_vma_asid(cpumask_bits(&hmask), - start, size, asid); + if (likely(local)) { + /* + * FIXME: The non-ASID code switches to a global flush + * once flushing more than a single page. It's made by + * commit 6efb16b1d551 (RISC-V: Issue a tlb page flush + * if possible). + */ + if (size <= PAGE_SIZE) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); + } else { + riscv_cpuid_to_hartid_mask(cmask, &hmask); + sbi_remote_sfence_vma(cpumask_bits(&hmask), + start, size); + } } put_cpu(); @@ -69,28 +72,16 @@ static void __sbi_tlb_flush_range_asid(struct cpumask *cmask, void flush_tlb_mm(struct mm_struct *mm) { - if (static_branch_unlikely(&use_asid_allocator)) - __sbi_tlb_flush_range_asid(mm_cpumask(mm), 0, -1, - atomic_long_read(&mm->context.id)); - else - __sbi_tlb_flush_range(mm_cpumask(mm), 0, -1); + __sbi_tlb_flush_range(mm, 0, -1); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - if (static_branch_unlikely(&use_asid_allocator)) - __sbi_tlb_flush_range_asid(mm_cpumask(vma->vm_mm), addr, PAGE_SIZE, - atomic_long_read(&vma->vm_mm->context.id)); - else - __sbi_tlb_flush_range(mm_cpumask(vma->vm_mm), addr, PAGE_SIZE); + __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - if (static_branch_unlikely(&use_asid_allocator)) - __sbi_tlb_flush_range_asid(mm_cpumask(vma->vm_mm), start, end - start, - atomic_long_read(&vma->vm_mm->context.id)); - else - __sbi_tlb_flush_range(mm_cpumask(vma->vm_mm), start, end - start); + __sbi_tlb_flush_range(vma->vm_mm, start, end - start); } -- 2.7.4 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv
next prev parent reply other threads:[~2021-06-06 9:05 UTC|newest] Thread overview: 106+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-06-06 9:03 [RFC PATCH v2 00/11] riscv: Add DMA_COHERENT support for Allwinner D1 guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:03 ` [RFC PATCH v2 01/11] riscv: asid: Use global mappings for kernel pages guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:03 ` [PATCH V5 1/3] riscv: " guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:03 ` [PATCH V5 2/3] riscv: Add ASID-based tlbflushing methods guoren 2021-06-06 9:03 ` guoren 2021-06-06 14:38 ` Christoph Hellwig 2021-06-06 14:38 ` Christoph Hellwig 2021-06-06 9:03 ` [RFC PATCH v2 02/11] riscv: asid: " guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 03/11] riscv: asid: Optimize tlbflush coding convention guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` guoren [this message] 2021-06-06 9:04 ` [PATCH V5 3/3] riscv: tlbflush: Optimize " guoren 2021-06-06 9:04 ` [RFC PATCH v2 04/11] riscv: pgtable: Fixup _PAGE_CHG_MASK usage guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 05/11] riscv: pgtable: Add custom protection_map init guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 06/11] riscv: pgtable: Add DMA_COHERENT with custom PTE attributes guoren 2021-06-06 9:04 ` guoren 2021-06-06 14:39 ` Christoph Hellwig 2021-06-06 14:39 ` Christoph Hellwig 2021-06-06 15:08 ` Guo Ren 2021-06-06 15:08 ` Guo Ren 2021-06-06 17:22 ` Nick Kossifidis 2021-06-06 17:22 ` Nick Kossifidis 2021-06-07 6:19 ` Christoph Hellwig 2021-06-07 6:19 ` Christoph Hellwig 2021-06-06 9:04 ` [RFC PATCH v2 07/11] riscv: cmo: Add dma-noncoherency support guoren 2021-06-06 9:04 ` guoren 2021-10-17 9:28 ` twd2 2021-10-17 9:28 ` twd2 2021-10-20 8:11 ` Guo Ren 2021-10-20 8:11 ` Guo Ren 2021-06-06 9:04 ` [RFC PATCH v2 08/11] riscv: cmo: Add vendor custom icache sync guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 09/11] riscv: soc: Initial DTS for Allwinner D1 NeZha board guoren 2021-06-06 9:04 ` guoren 2021-06-06 16:26 ` Jernej Škrabec 2021-06-06 16:26 ` Jernej Škrabec 2021-06-06 17:05 ` Guo Ren 2021-06-06 17:05 ` Guo Ren 2021-06-07 3:44 ` Guo Ren 2021-06-07 3:44 ` Guo Ren 2021-06-07 7:27 ` Maxime Ripard 2021-06-07 7:27 ` Maxime Ripard 2021-06-07 7:53 ` Guo Ren 2021-06-07 7:53 ` Guo Ren 2021-06-07 7:24 ` Maxime Ripard 2021-06-07 7:24 ` Maxime Ripard 2021-06-07 8:07 ` Guo Ren 2021-06-07 8:07 ` Guo Ren 2021-06-14 15:33 ` Maxime Ripard 2021-06-14 15:33 ` Maxime Ripard 2021-06-14 16:28 ` Guo Ren 2021-06-14 16:28 ` Guo Ren 2021-06-14 16:31 ` Jernej Škrabec 2021-06-14 16:31 ` Jernej Škrabec 2021-06-06 9:04 ` [RFC PATCH v2 10/11] riscv: soc: Add Allwinner SoC kconfig option guoren 2021-06-06 9:04 ` guoren 2021-06-07 7:19 ` Maxime Ripard 2021-06-07 7:19 ` Maxime Ripard 2021-06-07 7:27 ` Arnd Bergmann 2021-06-07 7:27 ` Arnd Bergmann 2021-06-07 7:45 ` Guo Ren 2021-06-07 7:45 ` Guo Ren 2021-06-07 7:43 ` Guo Ren 2021-06-07 7:43 ` Guo Ren 2021-06-07 12:12 ` Maxime Ripard 2021-06-07 12:12 ` Maxime Ripard 2021-06-07 12:39 ` Guo Ren 2021-06-07 12:39 ` Guo Ren 2021-06-06 9:04 ` [RFC PATCH v2 11/11] riscv: soc: Allwinner D1 GMAC driver only for temp use guoren 2021-06-06 9:04 ` guoren 2021-06-06 10:50 ` Andre Przywara 2021-06-06 10:50 ` Andre Przywara 2021-06-06 15:32 ` Guo Ren 2021-06-06 15:32 ` Guo Ren 2021-06-06 15:39 ` Jernej Škrabec 2021-06-06 15:39 ` Jernej Škrabec 2021-06-06 15:41 ` Guo Ren 2021-06-06 15:41 ` Guo Ren 2021-06-06 16:16 ` Arnd Bergmann 2021-06-06 16:16 ` Arnd Bergmann 2021-06-06 16:32 ` Jernej Škrabec 2021-06-06 16:32 ` Jernej Škrabec 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:29 ` [RFC PATCH v2 00/11] riscv: Add DMA_COHERENT support for Allwinner D1 Jernej Škrabec 2021-06-06 16:29 ` Jernej Škrabec 2021-06-06 16:54 ` Guo Ren 2021-06-06 16:54 ` Guo Ren 2021-06-06 17:14 ` Jernej Škrabec 2021-06-06 17:14 ` Jernej Škrabec 2021-06-06 23:42 ` Guo Ren 2021-06-06 23:42 ` Guo Ren 2021-06-07 3:44 ` Anup Patel 2021-06-07 3:44 ` Anup Patel 2021-06-07 4:36 ` Guo Ren 2021-06-07 4:36 ` Guo Ren -- strict thread matches above, loose matches on Subject: below -- 2021-05-30 16:49 [PATCH V5 0/3] riscv: Fixup asid_allocator remaining issues guoren 2021-05-30 16:49 ` [PATCH V5 3/3] riscv: tlbflush: Optimize coding convention guoren 2021-05-30 16:49 ` guoren
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1622970249-50770-7-git-send-email-guoren@kernel.org \ --to=guoren@kernel.org \ --cc=anup.patel@wdc.com \ --cc=arnd@arndb.de \ --cc=atish.patra@wdc.com \ --cc=drew@beagleboard.org \ --cc=guoren@linux.alibaba.com \ --cc=hch@lst.de \ --cc=lazyparser@gmail.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-riscv@lists.infradead.org \ --cc=linux-sunxi@lists.linux.dev \ --cc=liush@allwinnertech.com \ --cc=maxime@cerno.tech \ --cc=palmerdabbelt@google.com \ --cc=wefu@redhat.com \ --cc=wens@csie.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.