From: Alexander Lobakin <aleksander.lobakin@intel.com>
To: Christoph Hellwig <hch@lst.de>
Cc: Alexander Lobakin <aleksander.lobakin@intel.com>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>,
Marek Szyprowski <m.szyprowski@samsung.com>,
Robin Murphy <robin.murphy@arm.com>,
Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
"Rafael J. Wysocki" <rafael@kernel.org>,
Magnus Karlsson <magnus.karlsson@intel.com>,
nex.sw.ncis.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org,
netdev@vger.kernel.org, iommu@lists.linux.dev,
linux-kernel@vger.kernel.org
Subject: [PATCH v6 6/7] page_pool: check for DMA sync shortcut earlier
Date: Tue, 7 May 2024 13:20:25 +0200 [thread overview]
Message-ID: <20240507112026.1803778-7-aleksander.lobakin@intel.com> (raw)
In-Reply-To: <20240507112026.1803778-1-aleksander.lobakin@intel.com>
We can save a couple more function calls in the Page Pool code if we
check for dma_need_sync() earlier, just when we test pp->p.dma_sync.
Move both these checks into an inline wrapper and call the PP wrapper
over the generic DMA sync function only when both are true.
You can't cache the result of dma_need_sync() in &page_pool, as it may
change anytime if an SWIOTLB buffer is allocated or mapped.
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
---
net/core/page_pool.c | 33 +++++++++++++++++++--------------
1 file changed, 19 insertions(+), 14 deletions(-)
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index c2819ff03dd2..8836aaaf2385 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -398,16 +398,26 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)
return page;
}
-static void page_pool_dma_sync_for_device(struct page_pool *pool,
- struct page *page,
- unsigned int dma_sync_size)
+static void __page_pool_dma_sync_for_device(const struct page_pool *pool,
+ const struct page *page,
+ u32 dma_sync_size)
{
+#if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC)
dma_addr_t dma_addr = page_pool_get_dma_addr(page);
dma_sync_size = min(dma_sync_size, pool->p.max_len);
- dma_sync_single_range_for_device(pool->p.dev, dma_addr,
- pool->p.offset, dma_sync_size,
- pool->p.dma_dir);
+ __dma_sync_single_for_device(pool->p.dev, dma_addr + pool->p.offset,
+ dma_sync_size, pool->p.dma_dir);
+#endif
+}
+
+static __always_inline void
+page_pool_dma_sync_for_device(const struct page_pool *pool,
+ const struct page *page,
+ u32 dma_sync_size)
+{
+ if (pool->dma_sync && dma_dev_need_sync(pool->p.dev))
+ __page_pool_dma_sync_for_device(pool, page, dma_sync_size);
}
static bool page_pool_dma_map(struct page_pool *pool, struct page *page)
@@ -429,8 +439,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page)
if (page_pool_set_dma_addr(page, dma))
goto unmap_failed;
- if (pool->dma_sync)
- page_pool_dma_sync_for_device(pool, page, pool->p.max_len);
+ page_pool_dma_sync_for_device(pool, page, pool->p.max_len);
return true;
@@ -699,9 +708,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page,
if (likely(__page_pool_page_can_be_recycled(page))) {
/* Read barrier done in page_ref_count / READ_ONCE */
- if (pool->dma_sync)
- page_pool_dma_sync_for_device(pool, page,
- dma_sync_size);
+ page_pool_dma_sync_for_device(pool, page, dma_sync_size);
if (allow_direct && in_softirq() &&
page_pool_recycle_in_cache(page, pool))
@@ -812,9 +819,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool,
return NULL;
if (__page_pool_page_can_be_recycled(page)) {
- if (pool->dma_sync)
- page_pool_dma_sync_for_device(pool, page, -1);
-
+ page_pool_dma_sync_for_device(pool, page, -1);
return page;
}
--
2.45.0
next prev parent reply other threads:[~2024-05-07 11:21 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-07 11:20 [PATCH v6 0/7] dma: skip calling no-op sync ops when possible Alexander Lobakin
2024-05-07 11:20 ` [PATCH v6 1/7] dma: compile-out DMA sync op calls when not used Alexander Lobakin
2024-05-07 11:20 ` [PATCH v6 2/7] dma: avoid redundant calls for sync operations Alexander Lobakin
2024-05-09 11:41 ` Marek Szyprowski
2024-05-09 11:44 ` Alexander Lobakin
2024-05-09 11:59 ` Alexander Lobakin
2024-05-09 12:02 ` Christoph Hellwig
2024-05-09 12:01 ` Christoph Hellwig
2024-05-09 13:43 ` Steven Price
2024-05-09 13:49 ` Christoph Hellwig
2024-05-09 14:33 ` Robin Murphy
2024-05-09 14:43 ` Alexander Lobakin
2024-05-07 11:20 ` [PATCH v6 3/7] iommu/dma: avoid expensive indirect " Alexander Lobakin
2024-05-07 11:20 ` [PATCH v6 4/7] page_pool: make sure frag API fields don't span between cachelines Alexander Lobakin
2024-05-07 11:20 ` [PATCH v6 5/7] page_pool: don't use driver-set flags field directly Alexander Lobakin
2024-05-07 11:20 ` Alexander Lobakin [this message]
2024-05-07 11:20 ` [PATCH v6 7/7] xsk: use generic DMA sync shortcut instead of a custom one Alexander Lobakin
2024-05-07 11:30 ` [PATCH v6 0/7] dma: skip calling no-op sync ops when possible Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240507112026.1803778-7-aleksander.lobakin@intel.com \
--to=aleksander.lobakin@intel.com \
--cc=bpf@vger.kernel.org \
--cc=edumazet@google.com \
--cc=hch@lst.de \
--cc=iommu@lists.linux.dev \
--cc=joro@8bytes.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=m.szyprowski@samsung.com \
--cc=magnus.karlsson@intel.com \
--cc=netdev@vger.kernel.org \
--cc=nex.sw.ncis.osdt.itp.upstreaming@intel.com \
--cc=rafael@kernel.org \
--cc=robin.murphy@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).