From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3005CC49361 for ; Wed, 16 Jun 2021 11:46:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1287161019 for ; Wed, 16 Jun 2021 11:46:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232026AbhFPLss (ORCPT ); Wed, 16 Jun 2021 07:48:48 -0400 Received: from mail.kernel.org ([198.145.29.99]:39386 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229563AbhFPLsr (ORCPT ); Wed, 16 Jun 2021 07:48:47 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 40C56610CD; Wed, 16 Jun 2021 11:46:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623844001; bh=4ivGHJ8D/40tMicOJyo3CcfWjreuzYEM57NqA4ib7zc=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=B7HetqJewVb8o+DcyhMKpL1EShrgL42I49PW/4dTZ1K22VMqw4kpgVnaDgZvoUw/W QU0PlDVqtaA+DuZsyXM/ELwISaPkYfGYs6egdy0zw9gp2/99SIe01a7Der33miQXlZ Q2Lj85QlbYrkxzAc1Vf/MBv5cD64RmOc5Rk6bnwj1VZy7pC0MEAdmMT/b6lvNpX8Fv ROXhh1cK/YxCpZDTGGUr7vvQ2/7DEtwQZbE8E1GBDf6xMzX+mH28E2+2aw1K2QjN+J pbVpg8JsfJlD01f35QvoZUOuZyxhFSLJUQJgtFg4qT9wSBFlelUdihYOZwF9DfMrfm KtTpllG00vprQ== Received: by mail-lj1-f176.google.com with SMTP id k8so3379764lja.4; Wed, 16 Jun 2021 04:46:41 -0700 (PDT) X-Gm-Message-State: AOAM533L0VkMwUfoim3zNB4QLfEiDka0bzF2XO02bg0hQSludDWkLYmm TGJ0+l4kHQ6xI38wc0W41DRxWuRfZUfdCIPCM5k= X-Google-Smtp-Source: ABdhPJxDdmACAhk/dgo0OCFyGlwnxDvugMor6ZLlNvgjHyFSnIharh9WIU8sXOxm8N+zt1fC27tv/w2OOyjGbJ1XTrU= X-Received: by 2002:a2e:4b11:: with SMTP id y17mr4042575lja.105.1623843999378; Wed, 16 Jun 2021 04:46:39 -0700 (PDT) MIME-Version: 1.0 References: <20210615023812.50885-1-mcroce@linux.microsoft.com> <20210615023812.50885-2-mcroce@linux.microsoft.com> In-Reply-To: <20210615023812.50885-2-mcroce@linux.microsoft.com> From: Guo Ren Date: Wed, 16 Jun 2021 19:46:27 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 1/3] riscv: optimized memcpy To: Matteo Croce Cc: linux-riscv , Linux Kernel Mailing List , linux-arch , Paul Walmsley , Palmer Dabbelt , Albert Ou , Atish Patra , Emil Renner Berthing , Akira Tsukamoto , Drew Fustini , Bin Meng Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Matteo, Have you tried Glibc generic implementation code? ref: https://lore.kernel.org/linux-arch/20190629053641.3iBfk9-I_D29cDp9yJnIdIg7oMtHNZlDmhLQPTumhEc@z/#t If Glibc codes have the same performance in your hardware, then you could give a generic implementation first. The current Linux generic implementation is so simple in lib/string.c: #ifndef __HAVE_ARCH_MEMCPY /** * memcpy - Copy one area of memory to another * @dest: Where to copy to * @src: Where to copy from * @count: The size of the area. * * You should not use this function to access IO space, use memcpy_toio() * or memcpy_fromio() instead. */ void *memcpy(void *dest, const void *src, size_t count) { char *tmp = dest; const char *s = src; while (count--) *tmp++ = *s++; return dest; } EXPORT_SYMBOL(memcpy); #endif On Tue, Jun 15, 2021 at 10:42 AM Matteo Croce wrote: > > From: Matteo Croce > > Write a C version of memcpy() which uses the biggest data size allowed, > without generating unaligned accesses. > > The procedure is made of three steps: > First copy data one byte at time until the destination buffer is aligned > to a long boundary. > Then copy the data one long at time shifting the current and the next u8 > to compose a long at every cycle. > Finally, copy the remainder one byte at time. > > On a BeagleV, the TCP RX throughput increased by 45%: > > before: > > $ iperf3 -c beaglev > Connecting to host beaglev, port 5201 > [ 5] local 192.168.85.6 port 44840 connected to 192.168.85.48 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 76.4 MBytes 641 Mbits/sec 27 624 KBytes > [ 5] 1.00-2.00 sec 72.5 MBytes 608 Mbits/sec 0 708 KBytes > [ 5] 2.00-3.00 sec 73.8 MBytes 619 Mbits/sec 10 451 KBytes > [ 5] 3.00-4.00 sec 72.5 MBytes 608 Mbits/sec 0 564 KBytes > [ 5] 4.00-5.00 sec 73.8 MBytes 619 Mbits/sec 0 658 KBytes > [ 5] 5.00-6.00 sec 73.8 MBytes 619 Mbits/sec 14 522 KBytes > [ 5] 6.00-7.00 sec 73.8 MBytes 619 Mbits/sec 0 621 KBytes > [ 5] 7.00-8.00 sec 72.5 MBytes 608 Mbits/sec 0 706 KBytes > [ 5] 8.00-9.00 sec 73.8 MBytes 619 Mbits/sec 20 580 KBytes > [ 5] 9.00-10.00 sec 73.8 MBytes 619 Mbits/sec 0 672 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 736 MBytes 618 Mbits/sec 71 sender > [ 5] 0.00-10.01 sec 733 MBytes 615 Mbits/sec receiver > > after: > > $ iperf3 -c beaglev > Connecting to host beaglev, port 5201 > [ 5] local 192.168.85.6 port 44864 connected to 192.168.85.48 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 109 MBytes 912 Mbits/sec 48 559 KBytes > [ 5] 1.00-2.00 sec 108 MBytes 902 Mbits/sec 0 690 KBytes > [ 5] 2.00-3.00 sec 106 MBytes 891 Mbits/sec 36 396 KBytes > [ 5] 3.00-4.00 sec 108 MBytes 902 Mbits/sec 0 567 KBytes > [ 5] 4.00-5.00 sec 106 MBytes 891 Mbits/sec 0 699 KBytes > [ 5] 5.00-6.00 sec 106 MBytes 891 Mbits/sec 32 414 KBytes > [ 5] 6.00-7.00 sec 106 MBytes 891 Mbits/sec 0 583 KBytes > [ 5] 7.00-8.00 sec 106 MBytes 891 Mbits/sec 0 708 KBytes > [ 5] 8.00-9.00 sec 106 MBytes 891 Mbits/sec 28 433 KBytes > [ 5] 9.00-10.00 sec 108 MBytes 902 Mbits/sec 0 591 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 1.04 GBytes 897 Mbits/sec 144 sender > [ 5] 0.00-10.01 sec 1.04 GBytes 894 Mbits/sec receiver > > And the decreased CPU time of the memcpy() is observable with perf top. > This is the `perf top -Ue task-clock` output when doing the test: > > before: > > Overhead Shared O Symbol > 42.22% [kernel] [k] memcpy > 35.00% [kernel] [k] __asm_copy_to_user > 3.50% [kernel] [k] sifive_l2_flush64_range > 2.30% [kernel] [k] stmmac_napi_poll_rx > 1.11% [kernel] [k] memset > > after: > > Overhead Shared O Symbol > 45.69% [kernel] [k] __asm_copy_to_user > 29.06% [kernel] [k] memcpy > 4.09% [kernel] [k] sifive_l2_flush64_range > 2.77% [kernel] [k] stmmac_napi_poll_rx > 1.24% [kernel] [k] memset > > Signed-off-by: Matteo Croce > --- > arch/riscv/include/asm/string.h | 8 ++- > arch/riscv/kernel/riscv_ksyms.c | 2 - > arch/riscv/lib/Makefile | 2 +- > arch/riscv/lib/memcpy.S | 108 -------------------------------- > arch/riscv/lib/string.c | 94 +++++++++++++++++++++++++++ > 5 files changed, 101 insertions(+), 113 deletions(-) > delete mode 100644 arch/riscv/lib/memcpy.S > create mode 100644 arch/riscv/lib/string.c > > diff --git a/arch/riscv/include/asm/string.h b/arch/riscv/include/asm/string.h > index 909049366555..6b5d6fc3eab4 100644 > --- a/arch/riscv/include/asm/string.h > +++ b/arch/riscv/include/asm/string.h > @@ -12,9 +12,13 @@ > #define __HAVE_ARCH_MEMSET > extern asmlinkage void *memset(void *, int, size_t); > extern asmlinkage void *__memset(void *, int, size_t); > + > +#ifdef CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE > #define __HAVE_ARCH_MEMCPY > -extern asmlinkage void *memcpy(void *, const void *, size_t); > -extern asmlinkage void *__memcpy(void *, const void *, size_t); > +extern void *memcpy(void *dest, const void *src, size_t count); > +extern void *__memcpy(void *dest, const void *src, size_t count); > +#endif > + > #define __HAVE_ARCH_MEMMOVE > extern asmlinkage void *memmove(void *, const void *, size_t); > extern asmlinkage void *__memmove(void *, const void *, size_t); > diff --git a/arch/riscv/kernel/riscv_ksyms.c b/arch/riscv/kernel/riscv_ksyms.c > index 5ab1c7e1a6ed..3f6d512a5b97 100644 > --- a/arch/riscv/kernel/riscv_ksyms.c > +++ b/arch/riscv/kernel/riscv_ksyms.c > @@ -10,8 +10,6 @@ > * Assembly functions that may be used (directly or indirectly) by modules > */ > EXPORT_SYMBOL(memset); > -EXPORT_SYMBOL(memcpy); > EXPORT_SYMBOL(memmove); > EXPORT_SYMBOL(__memset); > -EXPORT_SYMBOL(__memcpy); > EXPORT_SYMBOL(__memmove); > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile > index 25d5c9664e57..2ffe85d4baee 100644 > --- a/arch/riscv/lib/Makefile > +++ b/arch/riscv/lib/Makefile > @@ -1,9 +1,9 @@ > # SPDX-License-Identifier: GPL-2.0-only > lib-y += delay.o > -lib-y += memcpy.o > lib-y += memset.o > lib-y += memmove.o > lib-$(CONFIG_MMU) += uaccess.o > lib-$(CONFIG_64BIT) += tishift.o > +lib-$(CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE) += string.o > > obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o > diff --git a/arch/riscv/lib/memcpy.S b/arch/riscv/lib/memcpy.S > deleted file mode 100644 > index 51ab716253fa..000000000000 > --- a/arch/riscv/lib/memcpy.S > +++ /dev/null > @@ -1,108 +0,0 @@ > -/* SPDX-License-Identifier: GPL-2.0-only */ > -/* > - * Copyright (C) 2013 Regents of the University of California > - */ > - > -#include > -#include > - > -/* void *memcpy(void *, const void *, size_t) */ > -ENTRY(__memcpy) > -WEAK(memcpy) > - move t6, a0 /* Preserve return value */ > - > - /* Defer to byte-oriented copy for small sizes */ > - sltiu a3, a2, 128 > - bnez a3, 4f > - /* Use word-oriented copy only if low-order bits match */ > - andi a3, t6, SZREG-1 > - andi a4, a1, SZREG-1 > - bne a3, a4, 4f > - > - beqz a3, 2f /* Skip if already aligned */ > - /* > - * Round to nearest double word-aligned address > - * greater than or equal to start address > - */ > - andi a3, a1, ~(SZREG-1) > - addi a3, a3, SZREG > - /* Handle initial misalignment */ > - sub a4, a3, a1 > -1: > - lb a5, 0(a1) > - addi a1, a1, 1 > - sb a5, 0(t6) > - addi t6, t6, 1 > - bltu a1, a3, 1b > - sub a2, a2, a4 /* Update count */ > - > -2: > - andi a4, a2, ~((16*SZREG)-1) > - beqz a4, 4f > - add a3, a1, a4 > -3: > - REG_L a4, 0(a1) > - REG_L a5, SZREG(a1) > - REG_L a6, 2*SZREG(a1) > - REG_L a7, 3*SZREG(a1) > - REG_L t0, 4*SZREG(a1) > - REG_L t1, 5*SZREG(a1) > - REG_L t2, 6*SZREG(a1) > - REG_L t3, 7*SZREG(a1) > - REG_L t4, 8*SZREG(a1) > - REG_L t5, 9*SZREG(a1) > - REG_S a4, 0(t6) > - REG_S a5, SZREG(t6) > - REG_S a6, 2*SZREG(t6) > - REG_S a7, 3*SZREG(t6) > - REG_S t0, 4*SZREG(t6) > - REG_S t1, 5*SZREG(t6) > - REG_S t2, 6*SZREG(t6) > - REG_S t3, 7*SZREG(t6) > - REG_S t4, 8*SZREG(t6) > - REG_S t5, 9*SZREG(t6) > - REG_L a4, 10*SZREG(a1) > - REG_L a5, 11*SZREG(a1) > - REG_L a6, 12*SZREG(a1) > - REG_L a7, 13*SZREG(a1) > - REG_L t0, 14*SZREG(a1) > - REG_L t1, 15*SZREG(a1) > - addi a1, a1, 16*SZREG > - REG_S a4, 10*SZREG(t6) > - REG_S a5, 11*SZREG(t6) > - REG_S a6, 12*SZREG(t6) > - REG_S a7, 13*SZREG(t6) > - REG_S t0, 14*SZREG(t6) > - REG_S t1, 15*SZREG(t6) > - addi t6, t6, 16*SZREG > - bltu a1, a3, 3b > - andi a2, a2, (16*SZREG)-1 /* Update count */ > - > -4: > - /* Handle trailing misalignment */ > - beqz a2, 6f > - add a3, a1, a2 > - > - /* Use word-oriented copy if co-aligned to word boundary */ > - or a5, a1, t6 > - or a5, a5, a3 > - andi a5, a5, 3 > - bnez a5, 5f > -7: > - lw a4, 0(a1) > - addi a1, a1, 4 > - sw a4, 0(t6) > - addi t6, t6, 4 > - bltu a1, a3, 7b > - > - ret > - > -5: > - lb a4, 0(a1) > - addi a1, a1, 1 > - sb a4, 0(t6) > - addi t6, t6, 1 > - bltu a1, a3, 5b > -6: > - ret > -END(__memcpy) > diff --git a/arch/riscv/lib/string.c b/arch/riscv/lib/string.c > new file mode 100644 > index 000000000000..525f9ee25a74 > --- /dev/null > +++ b/arch/riscv/lib/string.c > @@ -0,0 +1,94 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * String functions optimized for hardware which doesn't > + * handle unaligned memory accesses efficiently. > + * > + * Copyright (C) 2021 Matteo Croce > + */ > + > +#include > +#include > + > +/* size below a classic byte at time copy is done */ > +#define MIN_THRESHOLD 64 > + > +/* convenience types to avoid cast between different pointer types */ > +union types { > + u8 *u8; > + unsigned long *ulong; > + uintptr_t uptr; > +}; > + > +union const_types { > + const u8 *u8; > + unsigned long *ulong; > +}; > + > +void *memcpy(void *dest, const void *src, size_t count) > +{ > + const int bytes_long = BITS_PER_LONG / 8; > +#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS > + const int mask = bytes_long - 1; > + const int distance = (src - dest) & mask; > +#endif > + union const_types s = { .u8 = src }; > + union types d = { .u8 = dest }; > + > +#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS > + if (count <= MIN_THRESHOLD) > + goto copy_remainder; > + > + /* copy a byte at time until destination is aligned */ > + for (; count && d.uptr & mask; count--) > + *d.u8++ = *s.u8++; > + > + if (distance) { > + unsigned long last, next; > + > + /* move s backward to the previous alignment boundary */ > + s.u8 -= distance; > + > + /* 32/64 bit wide copy from s to d. > + * d is aligned now but s is not, so read s alignment wise, > + * and do proper shift to get the right value. > + * Works only on Little Endian machines. > + */ > + for (next = s.ulong[0]; count >= bytes_long + mask; count -= bytes_long) { > + last = next; > + next = s.ulong[1]; > + > + d.ulong[0] = last >> (distance * 8) | > + next << ((bytes_long - distance) * 8); > + > + d.ulong++; > + s.ulong++; > + } > + > + /* restore s with the original offset */ > + s.u8 += distance; > + } else > +#endif > + { > + /* if the source and dest lower bits are the same, do a simple > + * 32/64 bit wide copy. > + */ > + for (; count >= bytes_long; count -= bytes_long) > + *d.ulong++ = *s.ulong++; > + } > + > + /* suppress warning when CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y */ > + goto copy_remainder; > + > +copy_remainder: > + while (count--) > + *d.u8++ = *s.u8++; > + > + return dest; > +} > +EXPORT_SYMBOL(memcpy); > + > +void *__memcpy(void *dest, const void *src, size_t count) > +{ > + return memcpy(dest, src, count); > +} > +EXPORT_SYMBOL(__memcpy); > -- > 2.31.1 > -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CED3C48BE6 for ; Wed, 16 Jun 2021 11:47:05 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D6B066115B for ; Wed, 16 Jun 2021 11:47:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D6B066115B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gkbIek5Men9sbMS3zSXn7OLSKB/bp47LCnNQlMJQn+8=; b=Vi8RB6BxnIozEo G9FBTR2MGNwWYHZXfYp0t6rldVcTHaSYwBclyHjN0rv92qkyxW6QjvepRu5WEXcV5aIrYGwi5gfkW dncnuGixnj4E0aocs3WwqsQUmudiTcnaede2nlsZ8Rm4H63911VVYy3F533pavlGAJkUV2sljStu6 3lN5qrgf7ZGBzjBLC6ApRnd5ghao09ANfLMDWlKeZii/dKAFmXd3T6Q7eEIyCt6MyUUsV59KaMW/5 KXSJWPSUeiUHWP4MXW3HVMZmfkOujZrmxHgorldjpCVKOaPN+944+2l3x2eYoudKBvJQbZQknsEQm nU4UsjJfGMzu9T5idv+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltU0M-0066dr-8x; Wed, 16 Jun 2021 11:46:46 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltU0I-0066bT-Dp for linux-riscv@lists.infradead.org; Wed, 16 Jun 2021 11:46:44 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 54F9261040 for ; Wed, 16 Jun 2021 11:46:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623844001; bh=4ivGHJ8D/40tMicOJyo3CcfWjreuzYEM57NqA4ib7zc=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=B7HetqJewVb8o+DcyhMKpL1EShrgL42I49PW/4dTZ1K22VMqw4kpgVnaDgZvoUw/W QU0PlDVqtaA+DuZsyXM/ELwISaPkYfGYs6egdy0zw9gp2/99SIe01a7Der33miQXlZ Q2Lj85QlbYrkxzAc1Vf/MBv5cD64RmOc5Rk6bnwj1VZy7pC0MEAdmMT/b6lvNpX8Fv ROXhh1cK/YxCpZDTGGUr7vvQ2/7DEtwQZbE8E1GBDf6xMzX+mH28E2+2aw1K2QjN+J pbVpg8JsfJlD01f35QvoZUOuZyxhFSLJUQJgtFg4qT9wSBFlelUdihYOZwF9DfMrfm KtTpllG00vprQ== Received: by mail-lj1-f171.google.com with SMTP id s22so3402480ljg.5 for ; Wed, 16 Jun 2021 04:46:41 -0700 (PDT) X-Gm-Message-State: AOAM5307dGygHVxyDMpsg3HSX8FSD7cfdLwROgRs2WboVgRted6RWo4q zt9OukvHc4jOGHXxTGhIheAFMPWu7t1vOQa3eaA= X-Google-Smtp-Source: ABdhPJxDdmACAhk/dgo0OCFyGlwnxDvugMor6ZLlNvgjHyFSnIharh9WIU8sXOxm8N+zt1fC27tv/w2OOyjGbJ1XTrU= X-Received: by 2002:a2e:4b11:: with SMTP id y17mr4042575lja.105.1623843999378; Wed, 16 Jun 2021 04:46:39 -0700 (PDT) MIME-Version: 1.0 References: <20210615023812.50885-1-mcroce@linux.microsoft.com> <20210615023812.50885-2-mcroce@linux.microsoft.com> In-Reply-To: <20210615023812.50885-2-mcroce@linux.microsoft.com> From: Guo Ren Date: Wed, 16 Jun 2021 19:46:27 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 1/3] riscv: optimized memcpy To: Matteo Croce Cc: linux-riscv , Linux Kernel Mailing List , linux-arch , Paul Walmsley , Palmer Dabbelt , Albert Ou , Atish Patra , Emil Renner Berthing , Akira Tsukamoto , Drew Fustini , Bin Meng X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210616_044642_563119_ACD2F5D8 X-CRM114-Status: GOOD ( 42.15 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Hi Matteo, Have you tried Glibc generic implementation code? ref: https://lore.kernel.org/linux-arch/20190629053641.3iBfk9-I_D29cDp9yJnIdIg7oMtHNZlDmhLQPTumhEc@z/#t If Glibc codes have the same performance in your hardware, then you could give a generic implementation first. The current Linux generic implementation is so simple in lib/string.c: #ifndef __HAVE_ARCH_MEMCPY /** * memcpy - Copy one area of memory to another * @dest: Where to copy to * @src: Where to copy from * @count: The size of the area. * * You should not use this function to access IO space, use memcpy_toio() * or memcpy_fromio() instead. */ void *memcpy(void *dest, const void *src, size_t count) { char *tmp = dest; const char *s = src; while (count--) *tmp++ = *s++; return dest; } EXPORT_SYMBOL(memcpy); #endif On Tue, Jun 15, 2021 at 10:42 AM Matteo Croce wrote: > > From: Matteo Croce > > Write a C version of memcpy() which uses the biggest data size allowed, > without generating unaligned accesses. > > The procedure is made of three steps: > First copy data one byte at time until the destination buffer is aligned > to a long boundary. > Then copy the data one long at time shifting the current and the next u8 > to compose a long at every cycle. > Finally, copy the remainder one byte at time. > > On a BeagleV, the TCP RX throughput increased by 45%: > > before: > > $ iperf3 -c beaglev > Connecting to host beaglev, port 5201 > [ 5] local 192.168.85.6 port 44840 connected to 192.168.85.48 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 76.4 MBytes 641 Mbits/sec 27 624 KBytes > [ 5] 1.00-2.00 sec 72.5 MBytes 608 Mbits/sec 0 708 KBytes > [ 5] 2.00-3.00 sec 73.8 MBytes 619 Mbits/sec 10 451 KBytes > [ 5] 3.00-4.00 sec 72.5 MBytes 608 Mbits/sec 0 564 KBytes > [ 5] 4.00-5.00 sec 73.8 MBytes 619 Mbits/sec 0 658 KBytes > [ 5] 5.00-6.00 sec 73.8 MBytes 619 Mbits/sec 14 522 KBytes > [ 5] 6.00-7.00 sec 73.8 MBytes 619 Mbits/sec 0 621 KBytes > [ 5] 7.00-8.00 sec 72.5 MBytes 608 Mbits/sec 0 706 KBytes > [ 5] 8.00-9.00 sec 73.8 MBytes 619 Mbits/sec 20 580 KBytes > [ 5] 9.00-10.00 sec 73.8 MBytes 619 Mbits/sec 0 672 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 736 MBytes 618 Mbits/sec 71 sender > [ 5] 0.00-10.01 sec 733 MBytes 615 Mbits/sec receiver > > after: > > $ iperf3 -c beaglev > Connecting to host beaglev, port 5201 > [ 5] local 192.168.85.6 port 44864 connected to 192.168.85.48 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 109 MBytes 912 Mbits/sec 48 559 KBytes > [ 5] 1.00-2.00 sec 108 MBytes 902 Mbits/sec 0 690 KBytes > [ 5] 2.00-3.00 sec 106 MBytes 891 Mbits/sec 36 396 KBytes > [ 5] 3.00-4.00 sec 108 MBytes 902 Mbits/sec 0 567 KBytes > [ 5] 4.00-5.00 sec 106 MBytes 891 Mbits/sec 0 699 KBytes > [ 5] 5.00-6.00 sec 106 MBytes 891 Mbits/sec 32 414 KBytes > [ 5] 6.00-7.00 sec 106 MBytes 891 Mbits/sec 0 583 KBytes > [ 5] 7.00-8.00 sec 106 MBytes 891 Mbits/sec 0 708 KBytes > [ 5] 8.00-9.00 sec 106 MBytes 891 Mbits/sec 28 433 KBytes > [ 5] 9.00-10.00 sec 108 MBytes 902 Mbits/sec 0 591 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 1.04 GBytes 897 Mbits/sec 144 sender > [ 5] 0.00-10.01 sec 1.04 GBytes 894 Mbits/sec receiver > > And the decreased CPU time of the memcpy() is observable with perf top. > This is the `perf top -Ue task-clock` output when doing the test: > > before: > > Overhead Shared O Symbol > 42.22% [kernel] [k] memcpy > 35.00% [kernel] [k] __asm_copy_to_user > 3.50% [kernel] [k] sifive_l2_flush64_range > 2.30% [kernel] [k] stmmac_napi_poll_rx > 1.11% [kernel] [k] memset > > after: > > Overhead Shared O Symbol > 45.69% [kernel] [k] __asm_copy_to_user > 29.06% [kernel] [k] memcpy > 4.09% [kernel] [k] sifive_l2_flush64_range > 2.77% [kernel] [k] stmmac_napi_poll_rx > 1.24% [kernel] [k] memset > > Signed-off-by: Matteo Croce > --- > arch/riscv/include/asm/string.h | 8 ++- > arch/riscv/kernel/riscv_ksyms.c | 2 - > arch/riscv/lib/Makefile | 2 +- > arch/riscv/lib/memcpy.S | 108 -------------------------------- > arch/riscv/lib/string.c | 94 +++++++++++++++++++++++++++ > 5 files changed, 101 insertions(+), 113 deletions(-) > delete mode 100644 arch/riscv/lib/memcpy.S > create mode 100644 arch/riscv/lib/string.c > > diff --git a/arch/riscv/include/asm/string.h b/arch/riscv/include/asm/string.h > index 909049366555..6b5d6fc3eab4 100644 > --- a/arch/riscv/include/asm/string.h > +++ b/arch/riscv/include/asm/string.h > @@ -12,9 +12,13 @@ > #define __HAVE_ARCH_MEMSET > extern asmlinkage void *memset(void *, int, size_t); > extern asmlinkage void *__memset(void *, int, size_t); > + > +#ifdef CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE > #define __HAVE_ARCH_MEMCPY > -extern asmlinkage void *memcpy(void *, const void *, size_t); > -extern asmlinkage void *__memcpy(void *, const void *, size_t); > +extern void *memcpy(void *dest, const void *src, size_t count); > +extern void *__memcpy(void *dest, const void *src, size_t count); > +#endif > + > #define __HAVE_ARCH_MEMMOVE > extern asmlinkage void *memmove(void *, const void *, size_t); > extern asmlinkage void *__memmove(void *, const void *, size_t); > diff --git a/arch/riscv/kernel/riscv_ksyms.c b/arch/riscv/kernel/riscv_ksyms.c > index 5ab1c7e1a6ed..3f6d512a5b97 100644 > --- a/arch/riscv/kernel/riscv_ksyms.c > +++ b/arch/riscv/kernel/riscv_ksyms.c > @@ -10,8 +10,6 @@ > * Assembly functions that may be used (directly or indirectly) by modules > */ > EXPORT_SYMBOL(memset); > -EXPORT_SYMBOL(memcpy); > EXPORT_SYMBOL(memmove); > EXPORT_SYMBOL(__memset); > -EXPORT_SYMBOL(__memcpy); > EXPORT_SYMBOL(__memmove); > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile > index 25d5c9664e57..2ffe85d4baee 100644 > --- a/arch/riscv/lib/Makefile > +++ b/arch/riscv/lib/Makefile > @@ -1,9 +1,9 @@ > # SPDX-License-Identifier: GPL-2.0-only > lib-y += delay.o > -lib-y += memcpy.o > lib-y += memset.o > lib-y += memmove.o > lib-$(CONFIG_MMU) += uaccess.o > lib-$(CONFIG_64BIT) += tishift.o > +lib-$(CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE) += string.o > > obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o > diff --git a/arch/riscv/lib/memcpy.S b/arch/riscv/lib/memcpy.S > deleted file mode 100644 > index 51ab716253fa..000000000000 > --- a/arch/riscv/lib/memcpy.S > +++ /dev/null > @@ -1,108 +0,0 @@ > -/* SPDX-License-Identifier: GPL-2.0-only */ > -/* > - * Copyright (C) 2013 Regents of the University of California > - */ > - > -#include > -#include > - > -/* void *memcpy(void *, const void *, size_t) */ > -ENTRY(__memcpy) > -WEAK(memcpy) > - move t6, a0 /* Preserve return value */ > - > - /* Defer to byte-oriented copy for small sizes */ > - sltiu a3, a2, 128 > - bnez a3, 4f > - /* Use word-oriented copy only if low-order bits match */ > - andi a3, t6, SZREG-1 > - andi a4, a1, SZREG-1 > - bne a3, a4, 4f > - > - beqz a3, 2f /* Skip if already aligned */ > - /* > - * Round to nearest double word-aligned address > - * greater than or equal to start address > - */ > - andi a3, a1, ~(SZREG-1) > - addi a3, a3, SZREG > - /* Handle initial misalignment */ > - sub a4, a3, a1 > -1: > - lb a5, 0(a1) > - addi a1, a1, 1 > - sb a5, 0(t6) > - addi t6, t6, 1 > - bltu a1, a3, 1b > - sub a2, a2, a4 /* Update count */ > - > -2: > - andi a4, a2, ~((16*SZREG)-1) > - beqz a4, 4f > - add a3, a1, a4 > -3: > - REG_L a4, 0(a1) > - REG_L a5, SZREG(a1) > - REG_L a6, 2*SZREG(a1) > - REG_L a7, 3*SZREG(a1) > - REG_L t0, 4*SZREG(a1) > - REG_L t1, 5*SZREG(a1) > - REG_L t2, 6*SZREG(a1) > - REG_L t3, 7*SZREG(a1) > - REG_L t4, 8*SZREG(a1) > - REG_L t5, 9*SZREG(a1) > - REG_S a4, 0(t6) > - REG_S a5, SZREG(t6) > - REG_S a6, 2*SZREG(t6) > - REG_S a7, 3*SZREG(t6) > - REG_S t0, 4*SZREG(t6) > - REG_S t1, 5*SZREG(t6) > - REG_S t2, 6*SZREG(t6) > - REG_S t3, 7*SZREG(t6) > - REG_S t4, 8*SZREG(t6) > - REG_S t5, 9*SZREG(t6) > - REG_L a4, 10*SZREG(a1) > - REG_L a5, 11*SZREG(a1) > - REG_L a6, 12*SZREG(a1) > - REG_L a7, 13*SZREG(a1) > - REG_L t0, 14*SZREG(a1) > - REG_L t1, 15*SZREG(a1) > - addi a1, a1, 16*SZREG > - REG_S a4, 10*SZREG(t6) > - REG_S a5, 11*SZREG(t6) > - REG_S a6, 12*SZREG(t6) > - REG_S a7, 13*SZREG(t6) > - REG_S t0, 14*SZREG(t6) > - REG_S t1, 15*SZREG(t6) > - addi t6, t6, 16*SZREG > - bltu a1, a3, 3b > - andi a2, a2, (16*SZREG)-1 /* Update count */ > - > -4: > - /* Handle trailing misalignment */ > - beqz a2, 6f > - add a3, a1, a2 > - > - /* Use word-oriented copy if co-aligned to word boundary */ > - or a5, a1, t6 > - or a5, a5, a3 > - andi a5, a5, 3 > - bnez a5, 5f > -7: > - lw a4, 0(a1) > - addi a1, a1, 4 > - sw a4, 0(t6) > - addi t6, t6, 4 > - bltu a1, a3, 7b > - > - ret > - > -5: > - lb a4, 0(a1) > - addi a1, a1, 1 > - sb a4, 0(t6) > - addi t6, t6, 1 > - bltu a1, a3, 5b > -6: > - ret > -END(__memcpy) > diff --git a/arch/riscv/lib/string.c b/arch/riscv/lib/string.c > new file mode 100644 > index 000000000000..525f9ee25a74 > --- /dev/null > +++ b/arch/riscv/lib/string.c > @@ -0,0 +1,94 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * String functions optimized for hardware which doesn't > + * handle unaligned memory accesses efficiently. > + * > + * Copyright (C) 2021 Matteo Croce > + */ > + > +#include > +#include > + > +/* size below a classic byte at time copy is done */ > +#define MIN_THRESHOLD 64 > + > +/* convenience types to avoid cast between different pointer types */ > +union types { > + u8 *u8; > + unsigned long *ulong; > + uintptr_t uptr; > +}; > + > +union const_types { > + const u8 *u8; > + unsigned long *ulong; > +}; > + > +void *memcpy(void *dest, const void *src, size_t count) > +{ > + const int bytes_long = BITS_PER_LONG / 8; > +#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS > + const int mask = bytes_long - 1; > + const int distance = (src - dest) & mask; > +#endif > + union const_types s = { .u8 = src }; > + union types d = { .u8 = dest }; > + > +#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS > + if (count <= MIN_THRESHOLD) > + goto copy_remainder; > + > + /* copy a byte at time until destination is aligned */ > + for (; count && d.uptr & mask; count--) > + *d.u8++ = *s.u8++; > + > + if (distance) { > + unsigned long last, next; > + > + /* move s backward to the previous alignment boundary */ > + s.u8 -= distance; > + > + /* 32/64 bit wide copy from s to d. > + * d is aligned now but s is not, so read s alignment wise, > + * and do proper shift to get the right value. > + * Works only on Little Endian machines. > + */ > + for (next = s.ulong[0]; count >= bytes_long + mask; count -= bytes_long) { > + last = next; > + next = s.ulong[1]; > + > + d.ulong[0] = last >> (distance * 8) | > + next << ((bytes_long - distance) * 8); > + > + d.ulong++; > + s.ulong++; > + } > + > + /* restore s with the original offset */ > + s.u8 += distance; > + } else > +#endif > + { > + /* if the source and dest lower bits are the same, do a simple > + * 32/64 bit wide copy. > + */ > + for (; count >= bytes_long; count -= bytes_long) > + *d.ulong++ = *s.ulong++; > + } > + > + /* suppress warning when CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y */ > + goto copy_remainder; > + > +copy_remainder: > + while (count--) > + *d.u8++ = *s.u8++; > + > + return dest; > +} > +EXPORT_SYMBOL(memcpy); > + > +void *__memcpy(void *dest, const void *src, size_t count) > +{ > + return memcpy(dest, src, count); > +} > +EXPORT_SYMBOL(__memcpy); > -- > 2.31.1 > -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv