* [PATCH net-next v3 00/13] First try to replace page_frag with page_frag_cache
@ 2024-05-08 13:33 Yunsheng Lin
2024-05-08 13:34 ` [PATCH net-next v3 06/13] mm: page_frag: add '_va' suffix to page_frag API Yunsheng Lin
2024-05-08 13:34 ` [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache Yunsheng Lin
0 siblings, 2 replies; 8+ messages in thread
From: Yunsheng Lin @ 2024-05-08 13:33 UTC (permalink / raw
To: davem, kuba, pabeni
Cc: netdev, linux-kernel, Yunsheng Lin, Alexander Duyck,
Matthias Brugger, AngeloGioacchino Del Regno, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend,
linux-arm-kernel, linux-mediatek, bpf
After [1], there are still two implementations for page frag:
1. mm/page_alloc.c: net stack seems to be using it in the
rx part with 'struct page_frag_cache' and the main API
being page_frag_alloc_align().
2. net/core/sock.c: net stack seems to be using it in the
tx part with 'struct page_frag' and the main API being
skb_page_frag_refill().
This patchset tries to unfiy the page frag implementation
by replacing page_frag with page_frag_cache for sk_page_frag()
first. net_high_order_alloc_disable_key for the implementation
in net/core/sock.c doesn't seems matter that much now have
have pcp support for high-order pages in commit 44042b449872
("mm/page_alloc: allow high-order pages to be stored on the
per-cpu lists").
As the related change is mostly related to networking, so
targeting the net-next. And will try to replace the rest
of page_frag in the follow patchset.
After this patchset:
1. Unify the page frag implementation by taking the best out of
two the existing implementations: we are able to save some space
for the 'page_frag_cache' API user, and avoid 'get_page()' for
the old 'page_frag' API user.
2. Future bugfix and performance can be done in one place, hence
improving maintainability of page_frag's implementation.
Kernel Image changing:
Linux Kernel total | text data bss
------------------------------------------------------
after 44893035 | 27084547 17042328 766160
before 44896899 | 27088459 17042280 766160
delta -3864 | -3912 +48 +0
Performance validation:
1. Using micro-benchmark ko added in patch 1 to test aligned and
non-aligned API performance impact for the existing users, there
is no notiable performance degradation. Instead we have some minor
performance boot for both aligned and non-aligned API after this
patchset as below.
2. Use the below netcat test case, we also have some minor
performance boot for repalcing 'page_frag' with 'page_frag_cache'
after this patchset.
server: nc -l -k 1234 > /dev/null
client: perf stat -r 30 -- head -c 51200000000 /dev/zero | nc -N 127.0.0.1 1234
In order to avoid performance noise as much as possible, the testing
is done in system without any other laod and have enough iterations to
prove the data is stable enogh, complete log for testing is below:
*After* this patchset:
Performance counter stats for 'insmod ./page_frag_test.ko' (200 runs):
17.959047 task-clock (msec) # 0.001 CPUs utilized ( +- 0.16% )
7 context-switches # 0.371 K/sec ( +- 0.52% )
1 cpu-migrations # 0.037 K/sec ( +- 5.25% )
84 page-faults # 0.005 M/sec ( +- 0.10% )
46630340 cycles # 2.596 GHz ( +- 0.16% )
60991298 instructions # 1.31 insn per cycle ( +- 0.03% )
14778609 branches # 822.906 M/sec ( +- 0.03% )
21461 branch-misses # 0.15% of all branches ( +- 0.35% )
25.071309168 seconds time elapsed ( +- 0.42% )
Performance counter stats for 'insmod ./page_frag_test.ko test_align=1' (200 runs):
17.518324 task-clock (msec) # 0.001 CPUs utilized ( +- 0.22% )
7 context-switches # 0.397 K/sec ( +- 0.22% )
0 cpu-migrations # 0.014 K/sec ( +- 12.78% )
84 page-faults # 0.005 M/sec ( +- 0.10% )
45494458 cycles # 2.597 GHz ( +- 0.22% )
61059177 instructions # 1.34 insn per cycle ( +- 0.02% )
14792330 branches # 844.392 M/sec ( +- 0.02% )
21132 branch-misses # 0.14% of all branches ( +- 0.17% )
26.156288028 seconds time elapsed ( +- 0.41% )
perf stat -r 30 -- head -c 51200000000 /dev/zero | nc -N 127.0.0.1 1234
Performance counter stats for 'head -c 51200000000 /dev/zero' (30 runs):
107793.53 msec task-clock # 0.881 CPUs utilized ( +- 0.36% )
380421 context-switches # 3.529 K/sec ( +- 0.32% )
374 cpu-migrations # 3.470 /sec ( +- 1.31% )
74 page-faults # 0.686 /sec ( +- 0.28% )
92758718093 cycles # 0.861 GHz ( +- 0.48% ) (69.47%)
7035559641 stalled-cycles-frontend # 7.58% frontend cycles idle ( +- 1.19% ) (69.65%)
33668082825 stalled-cycles-backend # 36.30% backend cycles idle ( +- 0.84% ) (70.18%)
52424770535 instructions # 0.57 insn per cycle
# 0.64 stalled cycles per insn ( +- 0.26% ) (61.93%)
13240874953 branches # 122.836 M/sec ( +- 0.40% ) (60.36%)
208178019 branch-misses # 1.57% of all branches ( +- 0.65% ) (68.42%)
122.294 +- 0.402 seconds time elapsed ( +- 0.33% )
*Before* this patchset:
Performance counter stats for 'insmod ./page_frag_test_frag_page_v3_org.ko' (200 runs):
17.878013 task-clock (msec) # 0.001 CPUs utilized ( +- 0.02% )
7 context-switches # 0.384 K/sec ( +- 0.37% )
0 cpu-migrations # 0.002 K/sec ( +- 34.73% )
83 page-faults # 0.005 M/sec ( +- 0.11% )
46428731 cycles # 2.597 GHz ( +- 0.02% )
61080812 instructions # 1.32 insn per cycle ( +- 0.01% )
14802110 branches # 827.951 M/sec ( +- 0.01% )
33011 branch-misses # 0.22% of all branches ( +- 0.10% )
26.923903379 seconds time elapsed ( +- 0.32% )
Performance counter stats for 'insmod ./page_frag_test_frag_page_v3_org.ko test_align=1' (200 runs):
17.579978 task-clock (msec) # 0.001 CPUs utilized ( +- 0.05% )
7 context-switches # 0.393 K/sec ( +- 0.31% )
0 cpu-migrations # 0.014 K/sec ( +- 12.96% )
83 page-faults # 0.005 M/sec ( +- 0.10% )
45649145 cycles # 2.597 GHz ( +- 0.05% )
60999424 instructions # 1.34 insn per cycle ( +- 0.03% )
14778647 branches # 840.652 M/sec ( +- 0.03% )
33148 branch-misses # 0.22% of all branches ( +- 0.11% )
27.454413570 seconds time elapsed ( +- 0.48% )
perf stat -r 30 -- head -c 51200000000 /dev/zero | nc -N 127.0.0.1 1234
Performance counter stats for 'head -c 51200000000 /dev/zero' (30 runs):
110198.93 msec task-clock # 0.884 CPUs utilized ( +- 0.83% )
387680 context-switches # 3.518 K/sec ( +- 0.85% )
366 cpu-migrations # 3.321 /sec ( +- 11.38% )
74 page-faults # 0.672 /sec ( +- 0.27% )
92978008685 cycles # 0.844 GHz ( +- 0.49% ) (64.93%)
7339938950 stalled-cycles-frontend # 7.89% frontend cycles idle ( +- 1.48% ) (67.15%)
34783792329 stalled-cycles-backend # 37.41% backend cycles idle ( +- 1.52% ) (68.96%)
51704527141 instructions # 0.56 insn per cycle
# 0.67 stalled cycles per insn ( +- 0.37% ) (68.28%)
12865503633 branches # 116.748 M/sec ( +- 0.88% ) (66.11%)
212414695 branch-misses # 1.65% of all branches ( +- 0.45% ) (64.57%)
124.664 +- 0.990 seconds time elapsed ( +- 0.79% )
Note, ipv4-udp, ipv6-tcp and ipv6-udp is also tested with the below script:
nc -u -l -k 1234 > /dev/null
perf stat -r 4 -- head -c 51200000000 /dev/zero | nc -N -u 127.0.0.1 1234
nc -l6 -k 1234 > /dev/null
perf stat -r 4 -- head -c 51200000000 /dev/zero | nc -N ::1 1234
nc -l6 -k -u 1234 > /dev/null
perf stat -r 4 -- head -c 51200000000 /dev/zero | nc -u -N ::1 1234
CC: Alexander Duyck <alexander.duyck@gmail.com>
Change log:
v3:
1. Use new layout for 'struct page_frag_cache' as the discussion
with Alexander and other sugeestions from Alexander.
2. Add probe API to address Mat' comment about mptcp use case.
3. Some doc updating according to Bagas' suggestion.
v2:
1. reorder test module to patch 1.
2. split doc and maintainer updating to two patches.
3. refactor the page_frag before moving.
4. fix a type and 'static' warning in test module.
5. add a patch for xtensa arch to enable using get_order() in
BUILD_BUG_ON().
6. Add test case and performance data for the socket code.
Yunsheng Lin (13):
mm: page_frag: add a test module for page_frag
xtensa: remove the get_order() implementation
mm: page_frag: use free_unref_page() to free page fragment
mm: move the page fragment allocator from page_alloc into its own file
mm: page_frag: use initial zero offset for page_frag_alloc_align()
mm: page_frag: add '_va' suffix to page_frag API
mm: page_frag: avoid caller accessing 'page_frag_cache' directly
mm: page_frag: reuse existing space for 'size' and 'pfmemalloc'
net: introduce the skb_copy_to_va_nocache() helper
mm: page_frag: introduce prepare/probe/commit API
net: replace page_frag with page_frag_cache
mm: page_frag: update documentation for page_frag
mm: page_frag: add a entry in MAINTAINERS for page_frag
Documentation/mm/page_frags.rst | 156 ++++++-
MAINTAINERS | 11 +
arch/xtensa/include/asm/page.h | 18 -
.../chelsio/inline_crypto/chtls/chtls.h | 3 -
.../chelsio/inline_crypto/chtls/chtls_io.c | 100 ++---
.../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
drivers/net/ethernet/google/gve/gve_rx.c | 4 +-
drivers/net/ethernet/intel/ice/ice_txrx.c | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx.h | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx_lib.c | 2 +-
.../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 4 +-
.../marvell/octeontx2/nic/otx2_common.c | 2 +-
drivers/net/ethernet/mediatek/mtk_wed_wo.c | 4 +-
drivers/net/tun.c | 28 +-
drivers/nvme/host/tcp.c | 8 +-
drivers/nvme/target/tcp.c | 22 +-
drivers/vhost/net.c | 8 +-
include/linux/gfp.h | 22 -
include/linux/mm_types.h | 18 -
include/linux/page_frag_cache.h | 278 +++++++++++++
include/linux/sched.h | 4 +-
include/linux/skbuff.h | 3 +-
include/net/sock.h | 29 +-
kernel/bpf/cpumap.c | 2 +-
kernel/exit.c | 3 +-
kernel/fork.c | 3 +-
mm/Kconfig.debug | 8 +
mm/Makefile | 2 +
mm/page_alloc.c | 136 -------
mm/page_frag_cache.c | 337 ++++++++++++++++
mm/page_frag_test.c | 379 ++++++++++++++++++
net/core/skbuff.c | 56 +--
net/core/skmsg.c | 22 +-
net/core/sock.c | 46 ++-
net/core/xdp.c | 2 +-
net/ipv4/ip_output.c | 33 +-
net/ipv4/tcp.c | 35 +-
net/ipv4/tcp_output.c | 28 +-
net/ipv6/ip6_output.c | 33 +-
net/kcm/kcmsock.c | 30 +-
net/mptcp/protocol.c | 70 ++--
net/rxrpc/conn_object.c | 4 +-
net/rxrpc/local_object.c | 4 +-
net/rxrpc/txbuf.c | 15 +-
net/sched/em_meta.c | 2 +-
net/sunrpc/svcsock.c | 12 +-
net/tls/tls_device.c | 139 ++++---
47 files changed, 1576 insertions(+), 556 deletions(-)
create mode 100644 include/linux/page_frag_cache.h
create mode 100644 mm/page_frag_cache.c
create mode 100644 mm/page_frag_test.c
--
2.33.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next v3 06/13] mm: page_frag: add '_va' suffix to page_frag API
2024-05-08 13:33 [PATCH net-next v3 00/13] First try to replace page_frag with page_frag_cache Yunsheng Lin
@ 2024-05-08 13:34 ` Yunsheng Lin
2024-05-08 13:34 ` [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache Yunsheng Lin
1 sibling, 0 replies; 8+ messages in thread
From: Yunsheng Lin @ 2024-05-08 13:34 UTC (permalink / raw
To: davem, kuba, pabeni
Cc: netdev, linux-kernel, Yunsheng Lin, Alexander Duyck,
Jeroen de Borst, Praveen Kaligineedi, Shailend Chand,
Eric Dumazet, Jesse Brandeburg, Tony Nguyen, Sunil Goutham,
Geetha sowjanya, Subbaraya Sundeep, hariprasad, Felix Fietkau,
Sean Wang, Mark Lee, Lorenzo Bianconi, Matthias Brugger,
AngeloGioacchino Del Regno, Keith Busch, Jens Axboe,
Christoph Hellwig, Sagi Grimberg, Chaitanya Kulkarni,
Michael S. Tsirkin, Jason Wang, Andrew Morton, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Howells, Marc Dionne, Chuck Lever, Jeff Layton, Neil Brown,
Olga Kornievskaia, Dai Ngo, Tom Talpey, Trond Myklebust,
Anna Schumaker, intel-wired-lan, linux-arm-kernel, linux-mediatek,
linux-nvme, kvm, virtualization, linux-mm, bpf, linux-afs,
linux-nfs
Currently the page_frag API is returning 'virtual address'
or 'va' when allocing and expecting 'virtual address' or
'va' as input when freeing.
As we are about to support new use cases that the caller
need to deal with 'struct page' or need to deal with both
'va' and 'struct page'. In order to differentiate the API
handling between 'va' and 'struct page', add '_va' suffix
to the corresponding API mirroring the page_pool_alloc_va()
API of the page_pool. So that callers expecting to deal with
va, page or both va and page may call page_frag_alloc_va*,
page_frag_alloc_pg*, or page_frag_alloc* API accordingly.
CC: Alexander Duyck <alexander.duyck@gmail.com>
Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
---
drivers/net/ethernet/google/gve/gve_rx.c | 4 ++--
drivers/net/ethernet/intel/ice/ice_txrx.c | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx.h | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx_lib.c | 2 +-
.../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 4 ++--
.../marvell/octeontx2/nic/otx2_common.c | 2 +-
drivers/net/ethernet/mediatek/mtk_wed_wo.c | 4 ++--
drivers/nvme/host/tcp.c | 8 +++----
drivers/nvme/target/tcp.c | 22 +++++++++----------
drivers/vhost/net.c | 6 ++---
include/linux/page_frag_cache.h | 21 +++++++++---------
include/linux/skbuff.h | 2 +-
kernel/bpf/cpumap.c | 2 +-
mm/page_frag_cache.c | 12 +++++-----
mm/page_frag_test.c | 11 +++++-----
net/core/skbuff.c | 18 +++++++--------
net/core/xdp.c | 2 +-
net/rxrpc/txbuf.c | 15 +++++++------
net/sunrpc/svcsock.c | 6 ++---
19 files changed, 74 insertions(+), 71 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index acb73d4d0de6..b6c10100e462 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -729,7 +729,7 @@ static int gve_xdp_redirect(struct net_device *dev, struct gve_rx_ring *rx,
total_len = headroom + SKB_DATA_ALIGN(len) +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
- frame = page_frag_alloc(&rx->page_cache, total_len, GFP_ATOMIC);
+ frame = page_frag_alloc_va(&rx->page_cache, total_len, GFP_ATOMIC);
if (!frame) {
u64_stats_update_begin(&rx->statss);
rx->xdp_alloc_fails++;
@@ -742,7 +742,7 @@ static int gve_xdp_redirect(struct net_device *dev, struct gve_rx_ring *rx,
err = xdp_do_redirect(dev, &new, xdp_prog);
if (err)
- page_frag_free(frame);
+ page_frag_free_va(frame);
return err;
}
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c
index 8bb743f78fcb..399b317c509d 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ -126,7 +126,7 @@ ice_unmap_and_free_tx_buf(struct ice_tx_ring *ring, struct ice_tx_buf *tx_buf)
dev_kfree_skb_any(tx_buf->skb);
break;
case ICE_TX_BUF_XDP_TX:
- page_frag_free(tx_buf->raw_buf);
+ page_frag_free_va(tx_buf->raw_buf);
break;
case ICE_TX_BUF_XDP_XMIT:
xdp_return_frame(tx_buf->xdpf);
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h
index feba314a3fe4..6379f57d8228 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.h
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.h
@@ -148,7 +148,7 @@ static inline int ice_skb_pad(void)
* @ICE_TX_BUF_DUMMY: dummy Flow Director packet, unmap and kfree()
* @ICE_TX_BUF_FRAG: mapped skb OR &xdp_buff frag, only unmap DMA
* @ICE_TX_BUF_SKB: &sk_buff, unmap and consume_skb(), update stats
- * @ICE_TX_BUF_XDP_TX: &xdp_buff, unmap and page_frag_free(), stats
+ * @ICE_TX_BUF_XDP_TX: &xdp_buff, unmap and page_frag_free_va(), stats
* @ICE_TX_BUF_XDP_XMIT: &xdp_frame, unmap and xdp_return_frame(), stats
* @ICE_TX_BUF_XSK_TX: &xdp_buff on XSk queue, xsk_buff_free(), stats
*/
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx_lib.c b/drivers/net/ethernet/intel/ice/ice_txrx_lib.c
index 2719f0e20933..a1a41a14df0d 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx_lib.c
@@ -250,7 +250,7 @@ ice_clean_xdp_tx_buf(struct device *dev, struct ice_tx_buf *tx_buf,
switch (tx_buf->type) {
case ICE_TX_BUF_XDP_TX:
- page_frag_free(tx_buf->raw_buf);
+ page_frag_free_va(tx_buf->raw_buf);
break;
case ICE_TX_BUF_XDP_XMIT:
xdp_return_frame_bulk(tx_buf->xdpf, bq);
diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
index b938dc06045d..fcd1b149a45d 100644
--- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
+++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
@@ -303,7 +303,7 @@ static bool ixgbevf_clean_tx_irq(struct ixgbevf_q_vector *q_vector,
/* free the skb */
if (ring_is_xdp(tx_ring))
- page_frag_free(tx_buffer->data);
+ page_frag_free_va(tx_buffer->data);
else
napi_consume_skb(tx_buffer->skb, napi_budget);
@@ -2413,7 +2413,7 @@ static void ixgbevf_clean_tx_ring(struct ixgbevf_ring *tx_ring)
/* Free all the Tx ring sk_buffs */
if (ring_is_xdp(tx_ring))
- page_frag_free(tx_buffer->data);
+ page_frag_free_va(tx_buffer->data);
else
dev_kfree_skb_any(tx_buffer->skb);
diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c
index a85ac039d779..8eb5820b8a70 100644
--- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c
+++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c
@@ -553,7 +553,7 @@ static int __otx2_alloc_rbuf(struct otx2_nic *pfvf, struct otx2_pool *pool,
*dma = dma_map_single_attrs(pfvf->dev, buf, pool->rbsize,
DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
if (unlikely(dma_mapping_error(pfvf->dev, *dma))) {
- page_frag_free(buf);
+ page_frag_free_va(buf);
return -ENOMEM;
}
diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c
index 7063c78bd35f..c4228719f8a4 100644
--- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c
+++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c
@@ -142,8 +142,8 @@ mtk_wed_wo_queue_refill(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q,
dma_addr_t addr;
void *buf;
- buf = page_frag_alloc(&q->cache, q->buf_size,
- GFP_ATOMIC | GFP_DMA32);
+ buf = page_frag_alloc_va(&q->cache, q->buf_size,
+ GFP_ATOMIC | GFP_DMA32);
if (!buf)
break;
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index fdbcdcedcee9..79eddd74bfbb 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -500,7 +500,7 @@ static void nvme_tcp_exit_request(struct blk_mq_tag_set *set,
{
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
- page_frag_free(req->pdu);
+ page_frag_free_va(req->pdu);
}
static int nvme_tcp_init_request(struct blk_mq_tag_set *set,
@@ -514,7 +514,7 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set,
struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx];
u8 hdgst = nvme_tcp_hdgst_len(queue);
- req->pdu = page_frag_alloc(&queue->pf_cache,
+ req->pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
GFP_KERNEL | __GFP_ZERO);
if (!req->pdu)
@@ -1331,7 +1331,7 @@ static void nvme_tcp_free_async_req(struct nvme_tcp_ctrl *ctrl)
{
struct nvme_tcp_request *async = &ctrl->async_req;
- page_frag_free(async->pdu);
+ page_frag_free_va(async->pdu);
}
static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
@@ -1340,7 +1340,7 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
struct nvme_tcp_request *async = &ctrl->async_req;
u8 hdgst = nvme_tcp_hdgst_len(queue);
- async->pdu = page_frag_alloc(&queue->pf_cache,
+ async->pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
GFP_KERNEL | __GFP_ZERO);
if (!async->pdu)
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index a5422e2c979a..ea356ce22672 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1462,24 +1462,24 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
c->queue = queue;
c->req.port = queue->port->nport;
- c->cmd_pdu = page_frag_alloc(&queue->pf_cache,
+ c->cmd_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->cmd_pdu)
return -ENOMEM;
c->req.cmd = &c->cmd_pdu->cmd;
- c->rsp_pdu = page_frag_alloc(&queue->pf_cache,
+ c->rsp_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->rsp_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->rsp_pdu)
goto out_free_cmd;
c->req.cqe = &c->rsp_pdu->cqe;
- c->data_pdu = page_frag_alloc(&queue->pf_cache,
+ c->data_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->data_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->data_pdu)
goto out_free_rsp;
- c->r2t_pdu = page_frag_alloc(&queue->pf_cache,
+ c->r2t_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->r2t_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->r2t_pdu)
goto out_free_data;
@@ -1494,20 +1494,20 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
return 0;
out_free_data:
- page_frag_free(c->data_pdu);
+ page_frag_free_va(c->data_pdu);
out_free_rsp:
- page_frag_free(c->rsp_pdu);
+ page_frag_free_va(c->rsp_pdu);
out_free_cmd:
- page_frag_free(c->cmd_pdu);
+ page_frag_free_va(c->cmd_pdu);
return -ENOMEM;
}
static void nvmet_tcp_free_cmd(struct nvmet_tcp_cmd *c)
{
- page_frag_free(c->r2t_pdu);
- page_frag_free(c->data_pdu);
- page_frag_free(c->rsp_pdu);
- page_frag_free(c->cmd_pdu);
+ page_frag_free_va(c->r2t_pdu);
+ page_frag_free_va(c->data_pdu);
+ page_frag_free_va(c->rsp_pdu);
+ page_frag_free_va(c->cmd_pdu);
}
static int nvmet_tcp_alloc_cmds(struct nvmet_tcp_queue *queue)
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index f16279351db5..6691fac01e0d 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -686,8 +686,8 @@ static int vhost_net_build_xdp(struct vhost_net_virtqueue *nvq,
return -ENOSPC;
buflen += SKB_DATA_ALIGN(len + pad);
- buf = page_frag_alloc_align(&net->pf_cache, buflen, GFP_KERNEL,
- SMP_CACHE_BYTES);
+ buf = page_frag_alloc_va_align(&net->pf_cache, buflen, GFP_KERNEL,
+ SMP_CACHE_BYTES);
if (unlikely(!buf))
return -ENOMEM;
@@ -734,7 +734,7 @@ static int vhost_net_build_xdp(struct vhost_net_virtqueue *nvq,
return 0;
err:
- page_frag_free(buf);
+ page_frag_free_va(buf);
return ret;
}
diff --git a/include/linux/page_frag_cache.h b/include/linux/page_frag_cache.h
index 9da7cbd0ee47..a5747cf7a3a1 100644
--- a/include/linux/page_frag_cache.h
+++ b/include/linux/page_frag_cache.h
@@ -25,23 +25,24 @@ struct page_frag_cache {
void page_frag_cache_drain(struct page_frag_cache *nc);
void __page_frag_cache_drain(struct page *page, unsigned int count);
-void *__page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz,
- gfp_t gfp_mask, unsigned int align_mask);
+void *__page_frag_alloc_va_align(struct page_frag_cache *nc,
+ unsigned int fragsz, gfp_t gfp_mask,
+ unsigned int align_mask);
-static inline void *page_frag_alloc_align(struct page_frag_cache *nc,
- unsigned int fragsz, gfp_t gfp_mask,
- unsigned int align)
+static inline void *page_frag_alloc_va_align(struct page_frag_cache *nc,
+ unsigned int fragsz,
+ gfp_t gfp_mask, unsigned int align)
{
WARN_ON_ONCE(!is_power_of_2(align) || align > PAGE_SIZE);
- return __page_frag_alloc_align(nc, fragsz, gfp_mask, -align);
+ return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, -align);
}
-static inline void *page_frag_alloc(struct page_frag_cache *nc,
- unsigned int fragsz, gfp_t gfp_mask)
+static inline void *page_frag_alloc_va(struct page_frag_cache *nc,
+ unsigned int fragsz, gfp_t gfp_mask)
{
- return __page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u);
+ return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, ~0u);
}
-void page_frag_free(void *addr);
+void page_frag_free_va(void *addr);
#endif
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index ce077d14eab6..adaaa478fdce 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -3337,7 +3337,7 @@ static inline struct sk_buff *netdev_alloc_skb_ip_align(struct net_device *dev,
static inline void skb_free_frag(void *addr)
{
- page_frag_free(addr);
+ page_frag_free_va(addr);
}
void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask);
diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c
index a8e34416e960..3a6a237e7dd3 100644
--- a/kernel/bpf/cpumap.c
+++ b/kernel/bpf/cpumap.c
@@ -322,7 +322,7 @@ static int cpu_map_kthread_run(void *data)
/* Bring struct page memory area to curr CPU. Read by
* build_skb_around via page_is_pfmemalloc(), and when
- * freed written by page_frag_free call.
+ * freed written by page_frag_free_va call.
*/
prefetchw(page);
}
diff --git a/mm/page_frag_cache.c b/mm/page_frag_cache.c
index 152ae5dec58a..c0ecfa733727 100644
--- a/mm/page_frag_cache.c
+++ b/mm/page_frag_cache.c
@@ -61,9 +61,9 @@ void __page_frag_cache_drain(struct page *page, unsigned int count)
}
EXPORT_SYMBOL(__page_frag_cache_drain);
-void *__page_frag_alloc_align(struct page_frag_cache *nc,
- unsigned int fragsz, gfp_t gfp_mask,
- unsigned int align_mask)
+void *__page_frag_alloc_va_align(struct page_frag_cache *nc,
+ unsigned int fragsz, gfp_t gfp_mask,
+ unsigned int align_mask)
{
unsigned int size, offset;
struct page *page;
@@ -124,16 +124,16 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc,
return nc->va + offset;
}
-EXPORT_SYMBOL(__page_frag_alloc_align);
+EXPORT_SYMBOL(__page_frag_alloc_va_align);
/*
* Frees a page fragment allocated out of either a compound or order 0 page.
*/
-void page_frag_free(void *addr)
+void page_frag_free_va(void *addr)
{
struct page *page = virt_to_head_page(addr);
if (unlikely(put_page_testzero(page)))
free_unref_page(page, compound_order(page));
}
-EXPORT_SYMBOL(page_frag_free);
+EXPORT_SYMBOL(page_frag_free_va);
diff --git a/mm/page_frag_test.c b/mm/page_frag_test.c
index f1c861709551..92eb288aab75 100644
--- a/mm/page_frag_test.c
+++ b/mm/page_frag_test.c
@@ -265,7 +265,7 @@ static int page_frag_pop_thread(void *arg)
if (obj) {
nr--;
- page_frag_free(obj);
+ page_frag_free_va(obj);
} else {
cond_resched();
}
@@ -295,17 +295,18 @@ static int page_frag_push_thread(void *arg)
size = clamp(size, 1U, PAGE_SIZE);
if (test_align)
- va = page_frag_alloc_align(&test_frag, size, GFP_KERNEL,
- SMP_CACHE_BYTES);
+ va = page_frag_alloc_va_align(&test_frag, size,
+ GFP_KERNEL,
+ SMP_CACHE_BYTES);
else
- va = page_frag_alloc(&test_frag, size, GFP_KERNEL);
+ va = page_frag_alloc_va(&test_frag, size, GFP_KERNEL);
if (!va)
continue;
ret = objpool_push(va, pool);
if (ret) {
- page_frag_free(va);
+ page_frag_free_va(va);
cond_resched();
} else {
nr--;
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 466999a7515e..dca4e7445348 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -309,8 +309,8 @@ void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask)
fragsz = SKB_DATA_ALIGN(fragsz);
- return __page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC,
- align_mask);
+ return __page_frag_alloc_va_align(&nc->page, fragsz, GFP_ATOMIC,
+ align_mask);
}
EXPORT_SYMBOL(__napi_alloc_frag_align);
@@ -322,15 +322,15 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask)
if (in_hardirq() || irqs_disabled()) {
struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache);
- data = __page_frag_alloc_align(nc, fragsz, GFP_ATOMIC,
- align_mask);
+ data = __page_frag_alloc_va_align(nc, fragsz, GFP_ATOMIC,
+ align_mask);
} else {
struct napi_alloc_cache *nc;
local_bh_disable();
nc = this_cpu_ptr(&napi_alloc_cache);
- data = __page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC,
- align_mask);
+ data = __page_frag_alloc_va_align(&nc->page, fragsz, GFP_ATOMIC,
+ align_mask);
local_bh_enable();
}
return data;
@@ -740,12 +740,12 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len,
if (in_hardirq() || irqs_disabled()) {
nc = this_cpu_ptr(&netdev_alloc_cache);
- data = page_frag_alloc(nc, len, gfp_mask);
+ data = page_frag_alloc_va(nc, len, gfp_mask);
pfmemalloc = nc->pfmemalloc;
} else {
local_bh_disable();
nc = this_cpu_ptr(&napi_alloc_cache.page);
- data = page_frag_alloc(nc, len, gfp_mask);
+ data = page_frag_alloc_va(nc, len, gfp_mask);
pfmemalloc = nc->pfmemalloc;
local_bh_enable();
}
@@ -833,7 +833,7 @@ struct sk_buff *napi_alloc_skb(struct napi_struct *napi, unsigned int len)
} else {
len = SKB_HEAD_ALIGN(len);
- data = page_frag_alloc(&nc->page, len, gfp_mask);
+ data = page_frag_alloc_va(&nc->page, len, gfp_mask);
pfmemalloc = nc->page.pfmemalloc;
}
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 41693154e426..245a2d011aeb 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -391,7 +391,7 @@ void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,
page_pool_put_full_page(page->pp, page, napi_direct);
break;
case MEM_TYPE_PAGE_SHARED:
- page_frag_free(data);
+ page_frag_free_va(data);
break;
case MEM_TYPE_PAGE_ORDER0:
page = virt_to_page(data); /* Assumes order0 page*/
diff --git a/net/rxrpc/txbuf.c b/net/rxrpc/txbuf.c
index c3913d8a50d3..dccb0353ee84 100644
--- a/net/rxrpc/txbuf.c
+++ b/net/rxrpc/txbuf.c
@@ -33,8 +33,8 @@ struct rxrpc_txbuf *rxrpc_alloc_data_txbuf(struct rxrpc_call *call, size_t data_
data_align = umax(data_align, L1_CACHE_BYTES);
mutex_lock(&call->conn->tx_data_alloc_lock);
- buf = page_frag_alloc_align(&call->conn->tx_data_alloc, total, gfp,
- data_align);
+ buf = page_frag_alloc_va_align(&call->conn->tx_data_alloc, total, gfp,
+ data_align);
mutex_unlock(&call->conn->tx_data_alloc_lock);
if (!buf) {
kfree(txb);
@@ -96,17 +96,18 @@ struct rxrpc_txbuf *rxrpc_alloc_ack_txbuf(struct rxrpc_call *call, size_t sack_s
if (!txb)
return NULL;
- buf = page_frag_alloc(&call->local->tx_alloc,
- sizeof(*whdr) + sizeof(*ack) + 1 + 3 + sizeof(*trailer), gfp);
+ buf = page_frag_alloc_va(&call->local->tx_alloc,
+ sizeof(*whdr) + sizeof(*ack) + 1 + 3 + sizeof(*trailer), gfp);
if (!buf) {
kfree(txb);
return NULL;
}
if (sack_size) {
- buf2 = page_frag_alloc(&call->local->tx_alloc, sack_size, gfp);
+ buf2 = page_frag_alloc_va(&call->local->tx_alloc, sack_size,
+ gfp);
if (!buf2) {
- page_frag_free(buf);
+ page_frag_free_va(buf);
kfree(txb);
return NULL;
}
@@ -180,7 +181,7 @@ static void rxrpc_free_txbuf(struct rxrpc_txbuf *txb)
rxrpc_txbuf_free);
for (i = 0; i < txb->nr_kvec; i++)
if (txb->kvec[i].iov_base)
- page_frag_free(txb->kvec[i].iov_base);
+ page_frag_free_va(txb->kvec[i].iov_base);
kfree(txb);
atomic_dec(&rxrpc_nr_txbuf);
}
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 6b3f01beb294..42d20412c1c3 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1222,8 +1222,8 @@ static int svc_tcp_sendmsg(struct svc_sock *svsk, struct svc_rqst *rqstp,
/* The stream record marker is copied into a temporary page
* fragment buffer so that it can be included in rq_bvec.
*/
- buf = page_frag_alloc(&svsk->sk_frag_cache, sizeof(marker),
- GFP_KERNEL);
+ buf = page_frag_alloc_va(&svsk->sk_frag_cache, sizeof(marker),
+ GFP_KERNEL);
if (!buf)
return -ENOMEM;
memcpy(buf, &marker, sizeof(marker));
@@ -1235,7 +1235,7 @@ static int svc_tcp_sendmsg(struct svc_sock *svsk, struct svc_rqst *rqstp,
iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, rqstp->rq_bvec,
1 + count, sizeof(marker) + rqstp->rq_res.len);
ret = sock_sendmsg(svsk->sk_sock, &msg);
- page_frag_free(buf);
+ page_frag_free_va(buf);
if (ret < 0)
return ret;
*sentp += ret;
--
2.33.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache
2024-05-08 13:33 [PATCH net-next v3 00/13] First try to replace page_frag with page_frag_cache Yunsheng Lin
2024-05-08 13:34 ` [PATCH net-next v3 06/13] mm: page_frag: add '_va' suffix to page_frag API Yunsheng Lin
@ 2024-05-08 13:34 ` Yunsheng Lin
2024-05-09 16:22 ` Mat Martineau
1 sibling, 1 reply; 8+ messages in thread
From: Yunsheng Lin @ 2024-05-08 13:34 UTC (permalink / raw
To: davem, kuba, pabeni
Cc: netdev, linux-kernel, Yunsheng Lin, Alexander Duyck, Ayush Sawal,
Eric Dumazet, Willem de Bruijn, Jason Wang, Ingo Molnar,
Peter Zijlstra, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Steven Rostedt, Ben Segall, Mel Gorman,
Daniel Bristot de Oliveira, Valentin Schneider, John Fastabend,
Jakub Sitnicki, David Ahern, Matthieu Baerts, Mat Martineau,
Geliang Tang, Jamal Hadi Salim, Cong Wang, Jiri Pirko,
Boris Pismenny, bpf, mptcp
Use the newly introduced prepare/probe/commit API to
replace page_frag with page_frag_cache for sk_page_frag().
CC: Alexander Duyck <alexander.duyck@gmail.com>
Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
---
.../chelsio/inline_crypto/chtls/chtls.h | 3 -
.../chelsio/inline_crypto/chtls/chtls_io.c | 100 ++++---------
.../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
drivers/net/tun.c | 28 ++--
include/linux/sched.h | 4 +-
include/net/sock.h | 14 +-
kernel/exit.c | 3 +-
kernel/fork.c | 3 +-
net/core/skbuff.c | 32 ++--
net/core/skmsg.c | 22 +--
net/core/sock.c | 46 ++++--
net/ipv4/ip_output.c | 33 +++--
net/ipv4/tcp.c | 35 ++---
net/ipv4/tcp_output.c | 28 ++--
net/ipv6/ip6_output.c | 33 +++--
net/kcm/kcmsock.c | 30 ++--
net/mptcp/protocol.c | 70 +++++----
net/sched/em_meta.c | 2 +-
net/tls/tls_device.c | 139 ++++++++++--------
19 files changed, 331 insertions(+), 297 deletions(-)
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls.h b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls.h
index 7ff82b6778ba..fe2b6a8ef718 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls.h
+++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls.h
@@ -234,7 +234,6 @@ struct chtls_dev {
struct list_head list_node;
struct list_head rcu_node;
struct list_head na_node;
- unsigned int send_page_order;
int max_host_sndbuf;
u32 round_robin_cnt;
struct key_map kmap;
@@ -453,8 +452,6 @@ enum {
/* The ULP mode/submode of an skbuff */
#define skb_ulp_mode(skb) (ULP_SKB_CB(skb)->ulp_mode)
-#define TCP_PAGE(sk) (sk->sk_frag.page)
-#define TCP_OFF(sk) (sk->sk_frag.offset)
static inline struct chtls_dev *to_chtls_dev(struct tls_toe_device *tlsdev)
{
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c
index d567e42e1760..7820cae28c52 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c
@@ -825,12 +825,6 @@ void skb_entail(struct sock *sk, struct sk_buff *skb, int flags)
ULP_SKB_CB(skb)->flags = flags;
__skb_queue_tail(&csk->txq, skb);
sk->sk_wmem_queued += skb->truesize;
-
- if (TCP_PAGE(sk) && TCP_OFF(sk)) {
- put_page(TCP_PAGE(sk));
- TCP_PAGE(sk) = NULL;
- TCP_OFF(sk) = 0;
- }
}
static struct sk_buff *get_tx_skb(struct sock *sk, int size)
@@ -882,16 +876,12 @@ static void push_frames_if_head(struct sock *sk)
chtls_push_frames(csk, 1);
}
-static int chtls_skb_copy_to_page_nocache(struct sock *sk,
- struct iov_iter *from,
- struct sk_buff *skb,
- struct page *page,
- int off, int copy)
+static int chtls_skb_copy_to_va_nocache(struct sock *sk, struct iov_iter *from,
+ struct sk_buff *skb, char *va, int copy)
{
int err;
- err = skb_do_copy_data_nocache(sk, skb, from, page_address(page) +
- off, copy, skb->len);
+ err = skb_do_copy_data_nocache(sk, skb, from, va, copy, skb->len);
if (err)
return err;
@@ -1114,82 +1104,44 @@ int chtls_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
if (err)
goto do_fault;
} else {
+ struct page_frag_cache *pfrag = &sk->sk_frag;
int i = skb_shinfo(skb)->nr_frags;
- struct page *page = TCP_PAGE(sk);
- int pg_size = PAGE_SIZE;
- int off = TCP_OFF(sk);
- bool merge;
-
- if (page)
- pg_size = page_size(page);
- if (off < pg_size &&
- skb_can_coalesce(skb, i, page, off)) {
+ unsigned int offset, size;
+ bool merge = false;
+ struct page *page;
+ void *va;
+
+ size = 32U;
+ page = page_frag_alloc_prepare(pfrag, &offset, &size,
+ &va, sk->sk_allocation);
+ if (unlikely(!page))
+ goto wait_for_memory;
+
+ if (skb_can_coalesce(skb, i, page, offset))
merge = true;
- goto copy;
- }
- merge = false;
- if (i == (is_tls_tx(csk) ? (MAX_SKB_FRAGS - 1) :
- MAX_SKB_FRAGS))
+ else if (i == (is_tls_tx(csk) ? (MAX_SKB_FRAGS - 1) :
+ MAX_SKB_FRAGS))
goto new_buf;
- if (page && off == pg_size) {
- put_page(page);
- TCP_PAGE(sk) = page = NULL;
- pg_size = PAGE_SIZE;
- }
-
- if (!page) {
- gfp_t gfp = sk->sk_allocation;
- int order = cdev->send_page_order;
-
- if (order) {
- page = alloc_pages(gfp | __GFP_COMP |
- __GFP_NOWARN |
- __GFP_NORETRY,
- order);
- if (page)
- pg_size <<= order;
- }
- if (!page) {
- page = alloc_page(gfp);
- pg_size = PAGE_SIZE;
- }
- if (!page)
- goto wait_for_memory;
- off = 0;
- }
-copy:
- if (copy > pg_size - off)
- copy = pg_size - off;
+ copy = min_t(int, copy, size);
if (is_tls_tx(csk))
copy = min_t(int, copy, csk->tlshws.txleft);
- err = chtls_skb_copy_to_page_nocache(sk, &msg->msg_iter,
- skb, page,
- off, copy);
- if (unlikely(err)) {
- if (!TCP_PAGE(sk)) {
- TCP_PAGE(sk) = page;
- TCP_OFF(sk) = 0;
- }
+ err = chtls_skb_copy_to_va_nocache(sk, &msg->msg_iter,
+ skb, va, copy);
+ if (unlikely(err))
goto do_fault;
- }
+
/* Update the skb. */
if (merge) {
skb_frag_size_add(
&skb_shinfo(skb)->frags[i - 1],
copy);
+ page_frag_alloc_commit_noref(pfrag, copy);
} else {
- skb_fill_page_desc(skb, i, page, off, copy);
- if (off + copy < pg_size) {
- /* space left keep page */
- get_page(page);
- TCP_PAGE(sk) = page;
- } else {
- TCP_PAGE(sk) = NULL;
- }
+ skb_fill_page_desc(skb, i, page, offset, copy);
+ page_frag_alloc_commit(pfrag, copy);
}
- TCP_OFF(sk) = off + copy;
}
if (unlikely(skb->len == mss))
tx_skb_finalize(skb);
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_main.c b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_main.c
index 455a54708be4..ba88b2fc7cd8 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_main.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_main.c
@@ -34,7 +34,6 @@ static DEFINE_MUTEX(notify_mutex);
static RAW_NOTIFIER_HEAD(listen_notify_list);
static struct proto chtls_cpl_prot, chtls_cpl_protv6;
struct request_sock_ops chtls_rsk_ops, chtls_rsk_opsv6;
-static uint send_page_order = (14 - PAGE_SHIFT < 0) ? 0 : 14 - PAGE_SHIFT;
static void register_listen_notifier(struct notifier_block *nb)
{
@@ -273,8 +272,6 @@ static void *chtls_uld_add(const struct cxgb4_lld_info *info)
INIT_WORK(&cdev->deferq_task, process_deferq);
spin_lock_init(&cdev->listen_lock);
spin_lock_init(&cdev->idr_lock);
- cdev->send_page_order = min_t(uint, get_order(32768),
- send_page_order);
cdev->max_host_sndbuf = 48 * 1024;
if (lldi->vr->key.size)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 9254bca2813d..42f3ed13d604 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1598,8 +1598,8 @@ static bool tun_can_build_skb(struct tun_struct *tun, struct tun_file *tfile,
}
static struct sk_buff *__tun_build_skb(struct tun_file *tfile,
- struct page_frag *alloc_frag, char *buf,
- int buflen, int len, int pad)
+ struct page_frag_cache *alloc_frag,
+ char *buf, int buflen, int len, int pad)
{
struct sk_buff *skb = build_skb(buf, buflen);
@@ -1609,9 +1609,7 @@ static struct sk_buff *__tun_build_skb(struct tun_file *tfile,
skb_reserve(skb, pad);
skb_put(skb, len);
skb_set_owner_w(skb, tfile->socket.sk);
-
- get_page(alloc_frag->page);
- alloc_frag->offset += buflen;
+ page_frag_alloc_commit(alloc_frag, buflen);
return skb;
}
@@ -1660,9 +1658,10 @@ static struct sk_buff *tun_build_skb(struct tun_struct *tun,
struct virtio_net_hdr *hdr,
int len, int *skb_xdp)
{
- struct page_frag *alloc_frag = ¤t->task_frag;
+ struct page_frag_cache *alloc_frag = ¤t->task_frag;
struct bpf_prog *xdp_prog;
int buflen = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+ unsigned int size;
char *buf;
size_t copied;
int pad = TUN_RX_PAD;
@@ -1675,14 +1674,13 @@ static struct sk_buff *tun_build_skb(struct tun_struct *tun,
buflen += SKB_DATA_ALIGN(len + pad);
rcu_read_unlock();
- alloc_frag->offset = ALIGN((u64)alloc_frag->offset, SMP_CACHE_BYTES);
- if (unlikely(!skb_page_frag_refill(buflen, alloc_frag, GFP_KERNEL)))
+ size = buflen;
+ buf = page_frag_alloc_va_prepare_align(alloc_frag, &size, GFP_KERNEL,
+ SMP_CACHE_BYTES);
+ if (unlikely(!buf))
return ERR_PTR(-ENOMEM);
- buf = (char *)page_address(alloc_frag->page) + alloc_frag->offset;
- copied = copy_page_from_iter(alloc_frag->page,
- alloc_frag->offset + pad,
- len, from);
+ copied = copy_from_iter(buf + pad, len, from);
if (copied != len)
return ERR_PTR(-EFAULT);
@@ -1710,13 +1708,12 @@ static struct sk_buff *tun_build_skb(struct tun_struct *tun,
act = bpf_prog_run_xdp(xdp_prog, &xdp);
if (act == XDP_REDIRECT || act == XDP_TX) {
- get_page(alloc_frag->page);
- alloc_frag->offset += buflen;
+ page_frag_alloc_commit(alloc_frag, buflen);
}
err = tun_xdp_act(tun, xdp_prog, &xdp, act);
if (err < 0) {
if (act == XDP_REDIRECT || act == XDP_TX)
- put_page(alloc_frag->page);
+ page_frag_free_va(buf);
goto out;
}
@@ -1732,7 +1729,6 @@ static struct sk_buff *tun_build_skb(struct tun_struct *tun,
local_bh_enable();
return __tun_build_skb(tfile, alloc_frag, buf, buflen, len, pad);
-
out:
rcu_read_unlock();
local_bh_enable();
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 3c2abbc587b4..55c4b5fbe845 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -35,7 +35,6 @@
#include <linux/sched/types.h>
#include <linux/signal_types.h>
#include <linux/syscall_user_dispatch_types.h>
-#include <linux/mm_types_task.h>
#include <linux/task_io_accounting.h>
#include <linux/posix-timers_types.h>
#include <linux/restart_block.h>
@@ -45,6 +44,7 @@
#include <linux/rv.h>
#include <linux/livepatch_sched.h>
#include <linux/uidgid_types.h>
+#include <linux/page_frag_cache.h>
#include <asm/kmap_size.h>
/* task_struct member predeclarations (sorted alphabetically): */
@@ -1338,7 +1338,7 @@ struct task_struct {
/* Cache last used pipe for splice(): */
struct pipe_inode_info *splice_pipe;
- struct page_frag task_frag;
+ struct page_frag_cache task_frag;
#ifdef CONFIG_TASK_DELAY_ACCT
struct task_delay_info *delays;
diff --git a/include/net/sock.h b/include/net/sock.h
index 57421e680cce..ef06dce1d344 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -461,7 +461,7 @@ struct sock {
struct sk_buff_head sk_write_queue;
u32 sk_dst_pending_confirm;
u32 sk_pacing_status; /* see enum sk_pacing */
- struct page_frag sk_frag;
+ struct page_frag_cache sk_frag;
struct timer_list sk_timer;
unsigned long sk_pacing_rate; /* bytes per second */
@@ -2497,7 +2497,7 @@ static inline void sk_stream_moderate_sndbuf(struct sock *sk)
* Return: a per task page_frag if context allows that,
* otherwise a per socket one.
*/
-static inline struct page_frag *sk_page_frag(struct sock *sk)
+static inline struct page_frag_cache *sk_page_frag(struct sock *sk)
{
if (sk->sk_use_task_frag)
return ¤t->task_frag;
@@ -2505,7 +2505,15 @@ static inline struct page_frag *sk_page_frag(struct sock *sk)
return &sk->sk_frag;
}
-bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag);
+struct page *sk_page_frag_alloc_prepare(struct sock *sk,
+ struct page_frag_cache *pfrag,
+ unsigned int *size,
+ unsigned int *offset, void **va);
+
+struct page *sk_page_frag_alloc_pg_prepare(struct sock *sk,
+ struct page_frag_cache *pfrag,
+ unsigned int *size,
+ unsigned int *offset);
/*
* Default write policy as shown to user space via poll/select/SIGIO
diff --git a/kernel/exit.c b/kernel/exit.c
index 41a12630cbbc..8203275fd5ff 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -913,8 +913,7 @@ void __noreturn do_exit(long code)
if (tsk->splice_pipe)
free_pipe_info(tsk->splice_pipe);
- if (tsk->task_frag.page)
- put_page(tsk->task_frag.page);
+ page_frag_cache_drain(&tsk->task_frag);
exit_task_stack_account(tsk);
diff --git a/kernel/fork.c b/kernel/fork.c
index aebb3e6c96dc..a1c91949b45b 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -79,6 +79,7 @@
#include <linux/tty.h>
#include <linux/fs_struct.h>
#include <linux/magic.h>
+#include <linux/page_frag_cache.h>
#include <linux/perf_event.h>
#include <linux/posix-timers.h>
#include <linux/user-return-notifier.h>
@@ -1159,10 +1160,10 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
tsk->btrace_seq = 0;
#endif
tsk->splice_pipe = NULL;
- tsk->task_frag.page = NULL;
tsk->wake_q.next = NULL;
tsk->worker_private = NULL;
+ page_frag_cache_init(&tsk->task_frag);
kcov_task_init(tsk);
kmsan_task_create(tsk);
kmap_local_fork(tsk);
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index caee22db1cc7..02a2f05ce2e5 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3018,23 +3018,25 @@ static void sock_spd_release(struct splice_pipe_desc *spd, unsigned int i)
put_page(spd->pages[i]);
}
-static struct page *linear_to_page(struct page *page, unsigned int *len,
- unsigned int *offset,
- struct sock *sk)
+static struct page *linear_to_page(struct page_frag_cache *pfrag,
+ struct page *page, unsigned int *offset,
+ unsigned int *len, struct sock *sk)
{
- struct page_frag *pfrag = sk_page_frag(sk);
+ unsigned int new_len, new_offset;
+ struct page *frag_page;
+ void *va;
- if (!sk_page_frag_refill(sk, pfrag))
+ frag_page = sk_page_frag_alloc_prepare(sk, pfrag, &new_offset,
+ &new_len, &va);
+ if (!frag_page)
return NULL;
- *len = min_t(unsigned int, *len, pfrag->size - pfrag->offset);
+ *len = min_t(unsigned int, *len, new_len);
- memcpy(page_address(pfrag->page) + pfrag->offset,
- page_address(page) + *offset, *len);
- *offset = pfrag->offset;
- pfrag->offset += *len;
+ memcpy(va, page_address(page) + *offset, *len);
+ *offset = new_offset;
- return pfrag->page;
+ return frag_page;
}
static bool spd_can_coalesce(const struct splice_pipe_desc *spd,
@@ -3056,19 +3058,23 @@ static bool spd_fill_page(struct splice_pipe_desc *spd,
bool linear,
struct sock *sk)
{
+ struct page_frag_cache *pfrag = sk_page_frag(sk);
+
if (unlikely(spd->nr_pages == MAX_SKB_FRAGS))
return true;
if (linear) {
- page = linear_to_page(page, len, &offset, sk);
+ page = linear_to_page(pfrag, page, &offset, len, sk);
if (!page)
return true;
}
if (spd_can_coalesce(spd, page, offset)) {
spd->partial[spd->nr_pages - 1].len += *len;
+ page_frag_alloc_commit_noref(pfrag, *len);
return false;
}
- get_page(page);
+
+ page_frag_alloc_commit(pfrag, *len);
spd->pages[spd->nr_pages] = page;
spd->partial[spd->nr_pages].len = *len;
spd->partial[spd->nr_pages].offset = offset;
diff --git a/net/core/skmsg.c b/net/core/skmsg.c
index fd20aae30be2..ced167d5ba6c 100644
--- a/net/core/skmsg.c
+++ b/net/core/skmsg.c
@@ -27,23 +27,25 @@ static bool sk_msg_try_coalesce_ok(struct sk_msg *msg, int elem_first_coalesce)
int sk_msg_alloc(struct sock *sk, struct sk_msg *msg, int len,
int elem_first_coalesce)
{
- struct page_frag *pfrag = sk_page_frag(sk);
+ struct page_frag_cache *pfrag = sk_page_frag(sk);
u32 osize = msg->sg.size;
int ret = 0;
len -= msg->sg.size;
while (len > 0) {
+ unsigned int frag_offset, frag_len;
struct scatterlist *sge;
- u32 orig_offset;
+ struct page *page;
int use, i;
- if (!sk_page_frag_refill(sk, pfrag)) {
+ page = sk_page_frag_alloc_pg_prepare(sk, pfrag, &frag_offset,
+ &frag_len);
+ if (!page) {
ret = -ENOMEM;
goto msg_trim;
}
- orig_offset = pfrag->offset;
- use = min_t(int, len, pfrag->size - orig_offset);
+ use = min_t(int, len, frag_len);
if (!sk_wmem_schedule(sk, use)) {
ret = -ENOMEM;
goto msg_trim;
@@ -54,9 +56,10 @@ int sk_msg_alloc(struct sock *sk, struct sk_msg *msg, int len,
sge = &msg->sg.data[i];
if (sk_msg_try_coalesce_ok(msg, elem_first_coalesce) &&
- sg_page(sge) == pfrag->page &&
- sge->offset + sge->length == orig_offset) {
+ sg_page(sge) == page &&
+ sge->offset + sge->length == frag_offset) {
sge->length += use;
+ page_frag_alloc_commit_noref(pfrag, use);
} else {
if (sk_msg_full(msg)) {
ret = -ENOSPC;
@@ -65,14 +68,13 @@ int sk_msg_alloc(struct sock *sk, struct sk_msg *msg, int len,
sge = &msg->sg.data[msg->sg.end];
sg_unmark_end(sge);
- sg_set_page(sge, pfrag->page, use, orig_offset);
- get_page(pfrag->page);
+ sg_set_page(sge, page, use, frag_offset);
+ page_frag_alloc_commit(pfrag, use);
sk_msg_iter_next(msg, end);
}
sk_mem_charge(sk, use);
msg->sg.size += use;
- pfrag->offset += use;
len -= use;
}
diff --git a/net/core/sock.c b/net/core/sock.c
index 8d6e638b5426..8740d86c9b4c 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2191,10 +2191,7 @@ static void __sk_destruct(struct rcu_head *head)
pr_debug("%s: optmem leakage (%d bytes) detected\n",
__func__, atomic_read(&sk->sk_omem_alloc));
- if (sk->sk_frag.page) {
- put_page(sk->sk_frag.page);
- sk->sk_frag.page = NULL;
- }
+ page_frag_cache_drain(&sk->sk_frag);
/* We do not need to acquire sk->sk_peer_lock, we are the last user. */
put_cred(sk->sk_peer_cred);
@@ -2935,16 +2932,43 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
}
EXPORT_SYMBOL(skb_page_frag_refill);
-bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
+struct page *sk_page_frag_alloc_prepare(struct sock *sk,
+ struct page_frag_cache *pfrag,
+ unsigned int *offset,
+ unsigned int *size, void **va)
{
- if (likely(skb_page_frag_refill(32U, pfrag, sk->sk_allocation)))
- return true;
+ struct page *page;
+
+ *size = 32U;
+ page = page_frag_alloc_prepare(pfrag, offset, size, va,
+ sk->sk_allocation);
+ if (likely(page))
+ return page;
sk_enter_memory_pressure(sk);
sk_stream_moderate_sndbuf(sk);
- return false;
+ return NULL;
+}
+EXPORT_SYMBOL(sk_page_frag_alloc_prepare);
+
+struct page *sk_page_frag_alloc_pg_prepare(struct sock *sk,
+ struct page_frag_cache *pfrag,
+ unsigned int *offset,
+ unsigned int *size)
+{
+ struct page *page;
+
+ *size = 32U;
+ page = page_frag_alloc_pg_prepare(pfrag, offset, size,
+ sk->sk_allocation);
+ if (likely(page))
+ return page;
+
+ sk_enter_memory_pressure(sk);
+ sk_stream_moderate_sndbuf(sk);
+ return NULL;
}
-EXPORT_SYMBOL(sk_page_frag_refill);
+EXPORT_SYMBOL(sk_page_frag_alloc_pg_prepare);
void __lock_sock(struct sock *sk)
__releases(&sk->sk_lock.slock)
@@ -3478,8 +3502,8 @@ void sock_init_data_uid(struct socket *sock, struct sock *sk, kuid_t uid)
sk->sk_error_report = sock_def_error_report;
sk->sk_destruct = sock_def_destruct;
- sk->sk_frag.page = NULL;
- sk->sk_frag.offset = 0;
+ page_frag_cache_init(&sk->sk_frag);
+
sk->sk_peek_off = -1;
sk->sk_peer_pid = NULL;
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 9500031a1f55..eeb6a64a8250 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -952,7 +952,7 @@ static int __ip_append_data(struct sock *sk,
struct flowi4 *fl4,
struct sk_buff_head *queue,
struct inet_cork *cork,
- struct page_frag *pfrag,
+ struct page_frag_cache *pfrag,
int getfrag(void *from, char *to, int offset,
int len, int odd, struct sk_buff *skb),
void *from, int length, int transhdrlen,
@@ -1228,31 +1228,38 @@ static int __ip_append_data(struct sock *sk,
wmem_alloc_delta += copy;
} else if (!zc) {
int i = skb_shinfo(skb)->nr_frags;
+ unsigned int frag_offset, frag_size;
+ struct page *page;
+ void *va;
err = -ENOMEM;
- if (!sk_page_frag_refill(sk, pfrag))
+ page = sk_page_frag_alloc_prepare(sk, pfrag,
+ &frag_offset,
+ &frag_size, &va);
+ if (!page)
goto error;
skb_zcopy_downgrade_managed(skb);
- if (!skb_can_coalesce(skb, i, pfrag->page,
- pfrag->offset)) {
+ copy = min_t(int, copy, frag_size);
+
+ if (!skb_can_coalesce(skb, i, page, frag_offset)) {
err = -EMSGSIZE;
if (i == MAX_SKB_FRAGS)
goto error;
- __skb_fill_page_desc(skb, i, pfrag->page,
- pfrag->offset, 0);
+ __skb_fill_page_desc(skb, i, page, frag_offset,
+ copy);
skb_shinfo(skb)->nr_frags = ++i;
- get_page(pfrag->page);
+ page_frag_alloc_commit(pfrag, copy);
+ } else {
+ skb_frag_size_add(
+ &skb_shinfo(skb)->frags[i - 1], copy);
+ page_frag_alloc_commit_noref(pfrag, copy);
}
- copy = min_t(int, copy, pfrag->size - pfrag->offset);
- if (getfrag(from,
- page_address(pfrag->page) + pfrag->offset,
- offset, copy, skb->len, skb) < 0)
+
+ if (getfrag(from, va, offset, copy, skb->len, skb) < 0)
goto error_efault;
- pfrag->offset += copy;
- skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy);
skb_len_add(skb, copy);
wmem_alloc_delta += copy;
} else {
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index e1f0efbb29d6..8cf625a5b4ad 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1183,13 +1183,17 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
if (zc == 0) {
bool merge = true;
int i = skb_shinfo(skb)->nr_frags;
- struct page_frag *pfrag = sk_page_frag(sk);
-
- if (!sk_page_frag_refill(sk, pfrag))
+ struct page_frag_cache *pfrag = sk_page_frag(sk);
+ unsigned int offset, size;
+ struct page *page;
+ void *va;
+
+ page = sk_page_frag_alloc_prepare(sk, pfrag, &offset,
+ &size, &va);
+ if (!page)
goto wait_for_space;
- if (!skb_can_coalesce(skb, i, pfrag->page,
- pfrag->offset)) {
+ if (!skb_can_coalesce(skb, i, page, offset)) {
if (i >= READ_ONCE(net_hotdata.sysctl_max_skb_frags)) {
tcp_mark_push(tp, skb);
goto new_segment;
@@ -1197,7 +1201,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
merge = false;
}
- copy = min_t(int, copy, pfrag->size - pfrag->offset);
+ copy = min_t(int, copy, size);
if (unlikely(skb_zcopy_pure(skb) || skb_zcopy_managed(skb))) {
if (tcp_downgrade_zcopy_pure(sk, skb))
@@ -1209,22 +1213,19 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
if (!copy)
goto wait_for_space;
- err = skb_copy_to_page_nocache(sk, &msg->msg_iter, skb,
- pfrag->page,
- pfrag->offset,
- copy);
+ err = skb_copy_to_va_nocache(sk, &msg->msg_iter, skb,
+ va, copy);
if (err)
goto do_error;
/* Update the skb. */
if (merge) {
skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy);
+ page_frag_alloc_commit_noref(pfrag, copy);
} else {
- skb_fill_page_desc(skb, i, pfrag->page,
- pfrag->offset, copy);
- page_ref_inc(pfrag->page);
+ skb_fill_page_desc(skb, i, page, offset, copy);
+ page_frag_alloc_commit(pfrag, copy);
}
- pfrag->offset += copy;
} else if (zc == MSG_ZEROCOPY) {
/* First append to a fragless skb builds initial
* pure zerocopy skb
@@ -3122,11 +3123,7 @@ int tcp_disconnect(struct sock *sk, int flags)
WARN_ON(inet->inet_num && !icsk->icsk_bind_hash);
- if (sk->sk_frag.page) {
- put_page(sk->sk_frag.page);
- sk->sk_frag.page = NULL;
- sk->sk_frag.offset = 0;
- }
+ page_frag_cache_drain(&sk->sk_frag);
sk_error_report(sk);
return 0;
}
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 57edf66ff91b..133557b48b34 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -3966,9 +3966,12 @@ static int tcp_send_syn_data(struct sock *sk, struct sk_buff *syn)
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
struct tcp_fastopen_request *fo = tp->fastopen_req;
- struct page_frag *pfrag = sk_page_frag(sk);
+ struct page_frag_cache *pfrag = sk_page_frag(sk);
+ unsigned int offset, size;
struct sk_buff *syn_data;
int space, err = 0;
+ struct page *page;
+ void *va;
tp->rx_opt.mss_clamp = tp->advmss; /* If MSS is not cached */
if (!tcp_fastopen_cookie_check(sk, &tp->rx_opt.mss_clamp, &fo->cookie))
@@ -3987,30 +3990,31 @@ static int tcp_send_syn_data(struct sock *sk, struct sk_buff *syn)
space = min_t(size_t, space, fo->size);
- if (space &&
- !skb_page_frag_refill(min_t(size_t, space, PAGE_SIZE),
- pfrag, sk->sk_allocation))
- goto fallback;
+ if (space) {
+ size = min_t(size_t, space, PAGE_SIZE);
+ page = page_frag_alloc_prepare(pfrag, &offset, &size, &va,
+ sk->sk_allocation);
+ if (!page)
+ goto fallback;
+ }
+
syn_data = tcp_stream_alloc_skb(sk, sk->sk_allocation, false);
if (!syn_data)
goto fallback;
memcpy(syn_data->cb, syn->cb, sizeof(syn->cb));
if (space) {
- space = min_t(size_t, space, pfrag->size - pfrag->offset);
+ space = min_t(size_t, space, size);
space = tcp_wmem_schedule(sk, space);
}
if (space) {
- space = copy_page_from_iter(pfrag->page, pfrag->offset,
- space, &fo->data->msg_iter);
+ space = _copy_from_iter(va, space, &fo->data->msg_iter);
if (unlikely(!space)) {
tcp_skb_tsorted_anchor_cleanup(syn_data);
kfree_skb(syn_data);
goto fallback;
}
- skb_fill_page_desc(syn_data, 0, pfrag->page,
- pfrag->offset, space);
- page_ref_inc(pfrag->page);
- pfrag->offset += space;
+ skb_fill_page_desc(syn_data, 0, page, offset, space);
+ page_frag_alloc_commit(pfrag, space);
skb_len_add(syn_data, space);
skb_zcopy_set(syn_data, fo->uarg, NULL);
}
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 8f906e9fbc38..580b7deedbd6 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1404,7 +1404,7 @@ static int __ip6_append_data(struct sock *sk,
struct sk_buff_head *queue,
struct inet_cork_full *cork_full,
struct inet6_cork *v6_cork,
- struct page_frag *pfrag,
+ struct page_frag_cache *pfrag,
int getfrag(void *from, char *to, int offset,
int len, int odd, struct sk_buff *skb),
void *from, size_t length, int transhdrlen,
@@ -1745,32 +1745,39 @@ static int __ip6_append_data(struct sock *sk,
copy = err;
wmem_alloc_delta += copy;
} else if (!zc) {
+ unsigned int frag_offset, frag_size;
int i = skb_shinfo(skb)->nr_frags;
+ struct page *page;
+ void *va;
err = -ENOMEM;
- if (!sk_page_frag_refill(sk, pfrag))
+ page = sk_page_frag_alloc_prepare(sk, pfrag,
+ &frag_offset,
+ &frag_size, &va);
+ if (!page)
goto error;
skb_zcopy_downgrade_managed(skb);
- if (!skb_can_coalesce(skb, i, pfrag->page,
- pfrag->offset)) {
+ copy = min_t(int, copy, frag_size);
+
+ if (!skb_can_coalesce(skb, i, page, frag_offset)) {
err = -EMSGSIZE;
if (i == MAX_SKB_FRAGS)
goto error;
- __skb_fill_page_desc(skb, i, pfrag->page,
- pfrag->offset, 0);
+ __skb_fill_page_desc(skb, i, page, frag_offset,
+ copy);
skb_shinfo(skb)->nr_frags = ++i;
- get_page(pfrag->page);
+ page_frag_alloc_commit(pfrag, copy);
+ } else {
+ skb_frag_size_add(
+ &skb_shinfo(skb)->frags[i - 1], copy);
+ page_frag_alloc_commit_noref(pfrag, copy);
}
- copy = min_t(int, copy, pfrag->size - pfrag->offset);
- if (getfrag(from,
- page_address(pfrag->page) + pfrag->offset,
- offset, copy, skb->len, skb) < 0)
+
+ if (getfrag(from, va, offset, copy, skb->len, skb) < 0)
goto error_efault;
- pfrag->offset += copy;
- skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy);
skb->len += copy;
skb->data_len += copy;
skb->truesize += copy;
diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c
index 2f191e50d4fc..e52ddf716fa5 100644
--- a/net/kcm/kcmsock.c
+++ b/net/kcm/kcmsock.c
@@ -803,13 +803,17 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
while (msg_data_left(msg)) {
bool merge = true;
int i = skb_shinfo(skb)->nr_frags;
- struct page_frag *pfrag = sk_page_frag(sk);
-
- if (!sk_page_frag_refill(sk, pfrag))
+ struct page_frag_cache *pfrag = sk_page_frag(sk);
+ unsigned int offset, size;
+ struct page *page;
+ void *va;
+
+ page = sk_page_frag_alloc_prepare(sk, pfrag, &offset, &size,
+ &va);
+ if (!page)
goto wait_for_memory;
- if (!skb_can_coalesce(skb, i, pfrag->page,
- pfrag->offset)) {
+ if (!skb_can_coalesce(skb, i, page, offset)) {
if (i == MAX_SKB_FRAGS) {
struct sk_buff *tskb;
@@ -850,15 +854,12 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
if (head != skb)
head->truesize += copy;
} else {
- copy = min_t(int, msg_data_left(msg),
- pfrag->size - pfrag->offset);
+ copy = min_t(int, msg_data_left(msg), size);
if (!sk_wmem_schedule(sk, copy))
goto wait_for_memory;
- err = skb_copy_to_page_nocache(sk, &msg->msg_iter, skb,
- pfrag->page,
- pfrag->offset,
- copy);
+ err = skb_copy_to_va_nocache(sk, &msg->msg_iter, skb,
+ va, copy);
if (err)
goto out_error;
@@ -866,13 +867,12 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
if (merge) {
skb_frag_size_add(
&skb_shinfo(skb)->frags[i - 1], copy);
+ page_frag_alloc_commit_noref(pfrag, copy);
} else {
- skb_fill_page_desc(skb, i, pfrag->page,
- pfrag->offset, copy);
- get_page(pfrag->page);
+ skb_fill_page_desc(skb, i, page, offset, copy);
+ page_frag_alloc_commit(pfrag, copy);
}
- pfrag->offset += copy;
}
copied += copy;
diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index bb8f96f2b86f..ab844011d442 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -960,17 +960,18 @@ static bool mptcp_skb_can_collapse_to(u64 write_seq,
}
/* we can append data to the given data frag if:
- * - there is space available in the backing page_frag
- * - the data frag tail matches the current page_frag free offset
+ * - there is space available for the current page
+ * - the data frag tail matches the current page and offset
* - the data frag end sequence number matches the current write seq
*/
static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
- const struct page_frag *pfrag,
+ const struct page *page,
+ const unsigned int offset,
+ const unsigned int size,
const struct mptcp_data_frag *df)
{
- return df && pfrag->page == df->page &&
- pfrag->size - pfrag->offset > 0 &&
- pfrag->offset == (df->offset + df->data_len) &&
+ return df && size && page == df->page &&
+ offset == (df->offset + df->data_len) &&
df->data_seq + df->data_len == msk->write_seq;
}
@@ -1085,30 +1086,36 @@ static void mptcp_enter_memory_pressure(struct sock *sk)
/* ensure we get enough memory for the frag hdr, beyond some minimal amount of
* data
*/
-static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
+static struct page *mptcp_page_frag_alloc_prepare(struct sock *sk,
+ struct page_frag_cache *pfrag,
+ unsigned int *offset,
+ unsigned int *size, void **va)
{
- if (likely(skb_page_frag_refill(32U + sizeof(struct mptcp_data_frag),
- pfrag, sk->sk_allocation)))
- return true;
+ struct page *page;
+
+ page = page_frag_alloc_prepare(pfrag, offset, size, va,
+ sk->sk_allocation);
+ if (likely(page))
+ return page;
mptcp_enter_memory_pressure(sk);
- return false;
+ return NULL;
}
static struct mptcp_data_frag *
-mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag,
- int orig_offset)
+mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page *page,
+ unsigned int orig_offset)
{
int offset = ALIGN(orig_offset, sizeof(long));
struct mptcp_data_frag *dfrag;
- dfrag = (struct mptcp_data_frag *)(page_to_virt(pfrag->page) + offset);
+ dfrag = (struct mptcp_data_frag *)(page_to_virt(page) + offset);
dfrag->data_len = 0;
dfrag->data_seq = msk->write_seq;
dfrag->overhead = offset - orig_offset + sizeof(struct mptcp_data_frag);
dfrag->offset = offset + sizeof(struct mptcp_data_frag);
dfrag->already_sent = 0;
- dfrag->page = pfrag->page;
+ dfrag->page = page;
return dfrag;
}
@@ -1793,7 +1800,7 @@ static u32 mptcp_send_limit(const struct sock *sk)
static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
{
struct mptcp_sock *msk = mptcp_sk(sk);
- struct page_frag *pfrag;
+ struct page_frag_cache *pfrag;
size_t copied = 0;
int ret = 0;
long timeo;
@@ -1832,9 +1839,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
while (msg_data_left(msg)) {
int total_ts, frag_truesize = 0;
struct mptcp_data_frag *dfrag;
- bool dfrag_collapsed;
- size_t psize, offset;
+ bool dfrag_collapsed = false;
+ unsigned int offset, size;
+ struct page *page;
+ size_t psize;
u32 copy_limit;
+ void *va;
/* ensure fitting the notsent_lowat() constraint */
copy_limit = mptcp_send_limit(sk);
@@ -1845,21 +1855,26 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
* page allocator
*/
dfrag = mptcp_pending_tail(sk);
- dfrag_collapsed = mptcp_frag_can_collapse_to(msk, pfrag, dfrag);
+ page = page_frag_alloc_probe(pfrag, &offset, &size, &va);
+ dfrag_collapsed = mptcp_frag_can_collapse_to(msk, page, offset,
+ size, dfrag);
if (!dfrag_collapsed) {
- if (!mptcp_page_frag_refill(sk, pfrag))
+ size = 32U + sizeof(struct mptcp_data_frag);
+ page = mptcp_page_frag_alloc_prepare(sk, pfrag, &offset,
+ &size, &va);
+ if (!page)
goto wait_for_memory;
- dfrag = mptcp_carve_data_frag(msk, pfrag, pfrag->offset);
+ dfrag = mptcp_carve_data_frag(msk, page, offset);
frag_truesize = dfrag->overhead;
+ va += dfrag->overhead;
}
/* we do not bound vs wspace, to allow a single packet.
* memory accounting will prevent execessive memory usage
* anyway
*/
- offset = dfrag->offset + dfrag->data_len;
- psize = pfrag->size - offset;
+ psize = size - frag_truesize;
psize = min_t(size_t, psize, msg_data_left(msg));
psize = min_t(size_t, psize, copy_limit);
total_ts = psize + frag_truesize;
@@ -1867,8 +1882,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
if (!sk_wmem_schedule(sk, total_ts))
goto wait_for_memory;
- ret = do_copy_data_nocache(sk, psize, &msg->msg_iter,
- page_address(dfrag->page) + offset);
+ ret = do_copy_data_nocache(sk, psize, &msg->msg_iter, va);
if (ret)
goto do_error;
@@ -1877,7 +1891,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
copied += psize;
dfrag->data_len += psize;
frag_truesize += psize;
- pfrag->offset += frag_truesize;
WRITE_ONCE(msk->write_seq, msk->write_seq + psize);
/* charge data on mptcp pending queue to the msk socket
@@ -1885,11 +1898,14 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
*/
sk_wmem_queued_add(sk, frag_truesize);
if (!dfrag_collapsed) {
- get_page(dfrag->page);
+ page_frag_alloc_commit(pfrag, frag_truesize);
list_add_tail(&dfrag->list, &msk->rtx_queue);
if (!msk->first_pending)
WRITE_ONCE(msk->first_pending, dfrag);
+ } else {
+ page_frag_alloc_commit_noref(pfrag, frag_truesize);
}
+
pr_debug("msk=%p dfrag at seq=%llu len=%u sent=%u new=%d", msk,
dfrag->data_seq, dfrag->data_len, dfrag->already_sent,
!dfrag_collapsed);
diff --git a/net/sched/em_meta.c b/net/sched/em_meta.c
index 8996c73c9779..4da465af972f 100644
--- a/net/sched/em_meta.c
+++ b/net/sched/em_meta.c
@@ -590,7 +590,7 @@ META_COLLECTOR(int_sk_sendmsg_off)
*err = -1;
return;
}
- dst->value = sk->sk_frag.offset;
+ dst->value = page_frag_cache_page_offset(&sk->sk_frag);
}
META_COLLECTOR(int_sk_write_pend)
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index ab6e694f7bc2..46c3b1502cd2 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -256,25 +256,43 @@ static void tls_device_resync_tx(struct sock *sk, struct tls_context *tls_ctx,
clear_bit_unlock(TLS_TX_SYNC_SCHED, &tls_ctx->flags);
}
-static void tls_append_frag(struct tls_record_info *record,
- struct page_frag *pfrag,
- int size)
+static void tls_append_pfrag(struct tls_record_info *record,
+ struct page_frag_cache *pfrag, struct page *page,
+ unsigned int offset, unsigned int size)
{
skb_frag_t *frag;
frag = &record->frags[record->num_frags - 1];
- if (skb_frag_page(frag) == pfrag->page &&
- skb_frag_off(frag) + skb_frag_size(frag) == pfrag->offset) {
+ if (skb_frag_page(frag) == page &&
+ skb_frag_off(frag) + skb_frag_size(frag) == offset) {
skb_frag_size_add(frag, size);
+ page_frag_alloc_commit_noref(pfrag, size);
} else {
++frag;
- skb_frag_fill_page_desc(frag, pfrag->page, pfrag->offset,
- size);
+ skb_frag_fill_page_desc(frag, page, offset, size);
++record->num_frags;
- get_page(pfrag->page);
+ page_frag_alloc_commit(pfrag, size);
+ }
+
+ record->len += size;
+}
+
+static void tls_append_page(struct tls_record_info *record, struct page *page,
+ unsigned int offset, unsigned int size)
+{
+ skb_frag_t *frag;
+
+ frag = &record->frags[record->num_frags - 1];
+ if (skb_frag_page(frag) == page &&
+ skb_frag_off(frag) + skb_frag_size(frag) == offset) {
+ skb_frag_size_add(frag, size);
+ } else {
+ ++frag;
+ skb_frag_fill_page_desc(frag, page, offset, size);
+ ++record->num_frags;
+ get_page(page);
}
- pfrag->offset += size;
record->len += size;
}
@@ -315,11 +333,12 @@ static int tls_push_record(struct sock *sk,
static void tls_device_record_close(struct sock *sk,
struct tls_context *ctx,
struct tls_record_info *record,
- struct page_frag *pfrag,
+ struct page_frag_cache *pfrag,
unsigned char record_type)
{
struct tls_prot_info *prot = &ctx->prot_info;
- struct page_frag dummy_tag_frag;
+ unsigned int offset, size;
+ struct page *page;
/* append tag
* device will fill in the tag, we just need to append a placeholder
@@ -327,13 +346,14 @@ static void tls_device_record_close(struct sock *sk,
* increases frag count)
* if we can't allocate memory now use the dummy page
*/
- if (unlikely(pfrag->size - pfrag->offset < prot->tag_size) &&
- !skb_page_frag_refill(prot->tag_size, pfrag, sk->sk_allocation)) {
- dummy_tag_frag.page = dummy_page;
- dummy_tag_frag.offset = 0;
- pfrag = &dummy_tag_frag;
+ size = prot->tag_size;
+ page = page_frag_alloc_pg_prepare(pfrag, &offset, &size,
+ sk->sk_allocation);
+ if (unlikely(!page)) {
+ tls_append_page(record, dummy_page, 0, prot->tag_size);
+ } else {
+ tls_append_pfrag(record, pfrag, page, offset, prot->tag_size);
}
- tls_append_frag(record, pfrag, prot->tag_size);
/* fill prepend */
tls_fill_prepend(ctx, skb_frag_address(&record->frags[0]),
@@ -341,23 +361,33 @@ static void tls_device_record_close(struct sock *sk,
record_type);
}
-static int tls_create_new_record(struct tls_offload_context_tx *offload_ctx,
- struct page_frag *pfrag,
+static int tls_create_new_record(struct sock *sk,
+ struct tls_offload_context_tx *offload_ctx,
+ struct page_frag_cache *pfrag,
size_t prepend_size)
{
struct tls_record_info *record;
+ unsigned int offset, size;
+ struct page *page;
skb_frag_t *frag;
+ size = prepend_size;
+ page = page_frag_alloc_pg_prepare(pfrag, &offset, &size,
+ sk->sk_allocation);
+ if (!page) {
+ READ_ONCE(sk->sk_prot)->enter_memory_pressure(sk);
+ sk_stream_moderate_sndbuf(sk);
+ return -ENOMEM;
+ }
+
record = kmalloc(sizeof(*record), GFP_KERNEL);
if (!record)
return -ENOMEM;
frag = &record->frags[0];
- skb_frag_fill_page_desc(frag, pfrag->page, pfrag->offset,
- prepend_size);
+ skb_frag_fill_page_desc(frag, page, offset, prepend_size);
- get_page(pfrag->page);
- pfrag->offset += prepend_size;
+ page_frag_alloc_commit(pfrag, prepend_size);
record->num_frags = 1;
record->len = prepend_size;
@@ -365,33 +395,21 @@ static int tls_create_new_record(struct tls_offload_context_tx *offload_ctx,
return 0;
}
-static int tls_do_allocation(struct sock *sk,
- struct tls_offload_context_tx *offload_ctx,
- struct page_frag *pfrag,
- size_t prepend_size)
+static struct page *tls_do_allocation(struct sock *sk,
+ struct tls_offload_context_tx *ctx,
+ struct page_frag_cache *pfrag,
+ size_t prepend_size, unsigned int *offset,
+ unsigned int *size, void **va)
{
- int ret;
-
- if (!offload_ctx->open_record) {
- if (unlikely(!skb_page_frag_refill(prepend_size, pfrag,
- sk->sk_allocation))) {
- READ_ONCE(sk->sk_prot)->enter_memory_pressure(sk);
- sk_stream_moderate_sndbuf(sk);
- return -ENOMEM;
- }
+ if (!ctx->open_record) {
+ int ret;
- ret = tls_create_new_record(offload_ctx, pfrag, prepend_size);
+ ret = tls_create_new_record(sk, ctx, pfrag, prepend_size);
if (ret)
- return ret;
-
- if (pfrag->size > pfrag->offset)
- return 0;
+ return NULL;
}
- if (!sk_page_frag_refill(sk, pfrag))
- return -ENOMEM;
-
- return 0;
+ return sk_page_frag_alloc_prepare(sk, pfrag, offset, size, va);
}
static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i)
@@ -428,8 +446,8 @@ static int tls_push_data(struct sock *sk,
struct tls_prot_info *prot = &tls_ctx->prot_info;
struct tls_offload_context_tx *ctx = tls_offload_ctx_tx(tls_ctx);
struct tls_record_info *record;
+ struct page_frag_cache *pfrag;
int tls_push_record_flags;
- struct page_frag *pfrag;
size_t orig_size = size;
u32 max_open_record_len;
bool more = false;
@@ -466,8 +484,13 @@ static int tls_push_data(struct sock *sk,
max_open_record_len = TLS_MAX_PAYLOAD_SIZE +
prot->prepend_size;
do {
- rc = tls_do_allocation(sk, ctx, pfrag, prot->prepend_size);
- if (unlikely(rc)) {
+ unsigned int frag_offset, frag_size;
+ struct page *page;
+ void *va;
+
+ page = tls_do_allocation(sk, ctx, pfrag, prot->prepend_size,
+ &frag_offset, &frag_size, &va);
+ if (unlikely(!page)) {
rc = sk_stream_wait_memory(sk, &timeo);
if (!rc)
continue;
@@ -495,8 +518,8 @@ static int tls_push_data(struct sock *sk,
copy = min_t(size_t, size, max_open_record_len - record->len);
if (copy && (flags & MSG_SPLICE_PAGES)) {
- struct page_frag zc_pfrag;
- struct page **pages = &zc_pfrag.page;
+ struct page *splice_page;
+ struct page **pages = &splice_page;
size_t off;
rc = iov_iter_extract_pages(iter, &pages,
@@ -508,24 +531,22 @@ static int tls_push_data(struct sock *sk,
}
copy = rc;
- if (WARN_ON_ONCE(!sendpage_ok(zc_pfrag.page))) {
+ if (WARN_ON_ONCE(!sendpage_ok(splice_page))) {
iov_iter_revert(iter, copy);
rc = -EIO;
goto handle_error;
}
- zc_pfrag.offset = off;
- zc_pfrag.size = copy;
- tls_append_frag(record, &zc_pfrag, copy);
+ tls_append_page(record, splice_page, off, copy);
} else if (copy) {
- copy = min_t(size_t, copy, pfrag->size - pfrag->offset);
+ copy = min_t(size_t, copy, frag_size);
- rc = tls_device_copy_data(page_address(pfrag->page) +
- pfrag->offset, copy,
- iter);
+ rc = tls_device_copy_data(va, copy, iter);
if (rc)
goto handle_error;
- tls_append_frag(record, pfrag, copy);
+
+ tls_append_pfrag(record, pfrag, page, frag_offset,
+ copy);
}
size -= copy;
--
2.33.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache
2024-05-08 13:34 ` [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache Yunsheng Lin
@ 2024-05-09 16:22 ` Mat Martineau
2024-05-10 9:48 ` Yunsheng Lin
0 siblings, 1 reply; 8+ messages in thread
From: Mat Martineau @ 2024-05-09 16:22 UTC (permalink / raw
To: Yunsheng Lin
Cc: davem, kuba, pabeni, netdev, linux-kernel, Alexander Duyck,
Ayush Sawal, Eric Dumazet, Willem de Bruijn, Jason Wang,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Daniel Bristot de Oliveira, Valentin Schneider, John Fastabend,
Jakub Sitnicki, David Ahern, Matthieu Baerts, Geliang Tang,
Jamal Hadi Salim, Cong Wang, Jiri Pirko, Boris Pismenny, bpf,
mptcp
On Wed, 8 May 2024, Yunsheng Lin wrote:
> Use the newly introduced prepare/probe/commit API to
> replace page_frag with page_frag_cache for sk_page_frag().
>
> CC: Alexander Duyck <alexander.duyck@gmail.com>
> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
> ---
> .../chelsio/inline_crypto/chtls/chtls.h | 3 -
> .../chelsio/inline_crypto/chtls/chtls_io.c | 100 ++++---------
> .../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
> drivers/net/tun.c | 28 ++--
> include/linux/sched.h | 4 +-
> include/net/sock.h | 14 +-
> kernel/exit.c | 3 +-
> kernel/fork.c | 3 +-
> net/core/skbuff.c | 32 ++--
> net/core/skmsg.c | 22 +--
> net/core/sock.c | 46 ++++--
> net/ipv4/ip_output.c | 33 +++--
> net/ipv4/tcp.c | 35 ++---
> net/ipv4/tcp_output.c | 28 ++--
> net/ipv6/ip6_output.c | 33 +++--
> net/kcm/kcmsock.c | 30 ++--
> net/mptcp/protocol.c | 70 +++++----
> net/sched/em_meta.c | 2 +-
> net/tls/tls_device.c | 139 ++++++++++--------
> 19 files changed, 331 insertions(+), 297 deletions(-)
>
<snip>
> diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
> index bb8f96f2b86f..ab844011d442 100644
> --- a/net/mptcp/protocol.c
> +++ b/net/mptcp/protocol.c
> @@ -960,17 +960,18 @@ static bool mptcp_skb_can_collapse_to(u64 write_seq,
> }
>
> /* we can append data to the given data frag if:
> - * - there is space available in the backing page_frag
> - * - the data frag tail matches the current page_frag free offset
> + * - there is space available for the current page
> + * - the data frag tail matches the current page and offset
> * - the data frag end sequence number matches the current write seq
> */
> static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
> - const struct page_frag *pfrag,
> + const struct page *page,
> + const unsigned int offset,
> + const unsigned int size,
Hi Yunsheng -
Why add the 'size' parameter here? It's checked to be a nonzero value, but
it can only be 0 if page is also NULL. In this case "page == df->page"
will be false, so the function will return false even without checking
'size'.
Thanks,
Mat
> const struct mptcp_data_frag *df)
> {
> - return df && pfrag->page == df->page &&
> - pfrag->size - pfrag->offset > 0 &&
> - pfrag->offset == (df->offset + df->data_len) &&
> + return df && size && page == df->page &&
> + offset == (df->offset + df->data_len) &&
> df->data_seq + df->data_len == msk->write_seq;
> }
>
> @@ -1085,30 +1086,36 @@ static void mptcp_enter_memory_pressure(struct sock *sk)
> /* ensure we get enough memory for the frag hdr, beyond some minimal amount of
> * data
> */
> -static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
> +static struct page *mptcp_page_frag_alloc_prepare(struct sock *sk,
> + struct page_frag_cache *pfrag,
> + unsigned int *offset,
> + unsigned int *size, void **va)
> {
> - if (likely(skb_page_frag_refill(32U + sizeof(struct mptcp_data_frag),
> - pfrag, sk->sk_allocation)))
> - return true;
> + struct page *page;
> +
> + page = page_frag_alloc_prepare(pfrag, offset, size, va,
> + sk->sk_allocation);
> + if (likely(page))
> + return page;
>
> mptcp_enter_memory_pressure(sk);
> - return false;
> + return NULL;
> }
>
> static struct mptcp_data_frag *
> -mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag,
> - int orig_offset)
> +mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page *page,
> + unsigned int orig_offset)
> {
> int offset = ALIGN(orig_offset, sizeof(long));
> struct mptcp_data_frag *dfrag;
>
> - dfrag = (struct mptcp_data_frag *)(page_to_virt(pfrag->page) + offset);
> + dfrag = (struct mptcp_data_frag *)(page_to_virt(page) + offset);
> dfrag->data_len = 0;
> dfrag->data_seq = msk->write_seq;
> dfrag->overhead = offset - orig_offset + sizeof(struct mptcp_data_frag);
> dfrag->offset = offset + sizeof(struct mptcp_data_frag);
> dfrag->already_sent = 0;
> - dfrag->page = pfrag->page;
> + dfrag->page = page;
>
> return dfrag;
> }
> @@ -1793,7 +1800,7 @@ static u32 mptcp_send_limit(const struct sock *sk)
> static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
> {
> struct mptcp_sock *msk = mptcp_sk(sk);
> - struct page_frag *pfrag;
> + struct page_frag_cache *pfrag;
> size_t copied = 0;
> int ret = 0;
> long timeo;
> @@ -1832,9 +1839,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
> while (msg_data_left(msg)) {
> int total_ts, frag_truesize = 0;
> struct mptcp_data_frag *dfrag;
> - bool dfrag_collapsed;
> - size_t psize, offset;
> + bool dfrag_collapsed = false;
> + unsigned int offset, size;
> + struct page *page;
> + size_t psize;
> u32 copy_limit;
> + void *va;
>
> /* ensure fitting the notsent_lowat() constraint */
> copy_limit = mptcp_send_limit(sk);
> @@ -1845,21 +1855,26 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
> * page allocator
> */
> dfrag = mptcp_pending_tail(sk);
> - dfrag_collapsed = mptcp_frag_can_collapse_to(msk, pfrag, dfrag);
> + page = page_frag_alloc_probe(pfrag, &offset, &size, &va);
> + dfrag_collapsed = mptcp_frag_can_collapse_to(msk, page, offset,
> + size, dfrag);
> if (!dfrag_collapsed) {
> - if (!mptcp_page_frag_refill(sk, pfrag))
> + size = 32U + sizeof(struct mptcp_data_frag);
> + page = mptcp_page_frag_alloc_prepare(sk, pfrag, &offset,
> + &size, &va);
> + if (!page)
> goto wait_for_memory;
>
> - dfrag = mptcp_carve_data_frag(msk, pfrag, pfrag->offset);
> + dfrag = mptcp_carve_data_frag(msk, page, offset);
> frag_truesize = dfrag->overhead;
> + va += dfrag->overhead;
> }
>
> /* we do not bound vs wspace, to allow a single packet.
> * memory accounting will prevent execessive memory usage
> * anyway
> */
> - offset = dfrag->offset + dfrag->data_len;
> - psize = pfrag->size - offset;
> + psize = size - frag_truesize;
> psize = min_t(size_t, psize, msg_data_left(msg));
> psize = min_t(size_t, psize, copy_limit);
> total_ts = psize + frag_truesize;
> @@ -1867,8 +1882,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
> if (!sk_wmem_schedule(sk, total_ts))
> goto wait_for_memory;
>
> - ret = do_copy_data_nocache(sk, psize, &msg->msg_iter,
> - page_address(dfrag->page) + offset);
> + ret = do_copy_data_nocache(sk, psize, &msg->msg_iter, va);
> if (ret)
> goto do_error;
>
> @@ -1877,7 +1891,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
> copied += psize;
> dfrag->data_len += psize;
> frag_truesize += psize;
> - pfrag->offset += frag_truesize;
> WRITE_ONCE(msk->write_seq, msk->write_seq + psize);
>
> /* charge data on mptcp pending queue to the msk socket
> @@ -1885,11 +1898,14 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
> */
> sk_wmem_queued_add(sk, frag_truesize);
> if (!dfrag_collapsed) {
> - get_page(dfrag->page);
> + page_frag_alloc_commit(pfrag, frag_truesize);
> list_add_tail(&dfrag->list, &msk->rtx_queue);
> if (!msk->first_pending)
> WRITE_ONCE(msk->first_pending, dfrag);
> + } else {
> + page_frag_alloc_commit_noref(pfrag, frag_truesize);
> }
> +
> pr_debug("msk=%p dfrag at seq=%llu len=%u sent=%u new=%d", msk,
> dfrag->data_seq, dfrag->data_len, dfrag->already_sent,
> !dfrag_collapsed);
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache
2024-05-09 16:22 ` Mat Martineau
@ 2024-05-10 9:48 ` Yunsheng Lin
2024-05-10 17:29 ` Mat Martineau
0 siblings, 1 reply; 8+ messages in thread
From: Yunsheng Lin @ 2024-05-10 9:48 UTC (permalink / raw
To: Mat Martineau
Cc: davem, kuba, pabeni, netdev, linux-kernel, Alexander Duyck,
Ayush Sawal, Eric Dumazet, Willem de Bruijn, Jason Wang,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Daniel Bristot de Oliveira, Valentin Schneider, John Fastabend,
Jakub Sitnicki, David Ahern, Matthieu Baerts, Geliang Tang,
Jamal Hadi Salim, Cong Wang, Jiri Pirko, Boris Pismenny, bpf,
mptcp
On 2024/5/10 0:22, Mat Martineau wrote:
> On Wed, 8 May 2024, Yunsheng Lin wrote:
>
>> Use the newly introduced prepare/probe/commit API to
>> replace page_frag with page_frag_cache for sk_page_frag().
>>
>> CC: Alexander Duyck <alexander.duyck@gmail.com>
>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
>> ---
>> .../chelsio/inline_crypto/chtls/chtls.h | 3 -
>> .../chelsio/inline_crypto/chtls/chtls_io.c | 100 ++++---------
>> .../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
>> drivers/net/tun.c | 28 ++--
>> include/linux/sched.h | 4 +-
>> include/net/sock.h | 14 +-
>> kernel/exit.c | 3 +-
>> kernel/fork.c | 3 +-
>> net/core/skbuff.c | 32 ++--
>> net/core/skmsg.c | 22 +--
>> net/core/sock.c | 46 ++++--
>> net/ipv4/ip_output.c | 33 +++--
>> net/ipv4/tcp.c | 35 ++---
>> net/ipv4/tcp_output.c | 28 ++--
>> net/ipv6/ip6_output.c | 33 +++--
>> net/kcm/kcmsock.c | 30 ++--
>> net/mptcp/protocol.c | 70 +++++----
>> net/sched/em_meta.c | 2 +-
>> net/tls/tls_device.c | 139 ++++++++++--------
>> 19 files changed, 331 insertions(+), 297 deletions(-)
>>
>
> <snip>
>
>> diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
>> index bb8f96f2b86f..ab844011d442 100644
>> --- a/net/mptcp/protocol.c
>> +++ b/net/mptcp/protocol.c
>> @@ -960,17 +960,18 @@ static bool mptcp_skb_can_collapse_to(u64 write_seq,
>> }
>>
>> /* we can append data to the given data frag if:
>> - * - there is space available in the backing page_frag
>> - * - the data frag tail matches the current page_frag free offset
>> + * - there is space available for the current page
>> + * - the data frag tail matches the current page and offset
>> * - the data frag end sequence number matches the current write seq
>> */
>> static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
>> - const struct page_frag *pfrag,
>> + const struct page *page,
>> + const unsigned int offset,
>> + const unsigned int size,
>
> Hi Yunsheng -
>
> Why add the 'size' parameter here? It's checked to be a nonzero value, but it can only be 0 if page is also NULL. In this case "page == df->page" will be false, so the function will return false even without checking 'size'.
Is it possible that the pfrag->page is also NULL, which may cause
mptcp_frag_can_collapse_to() to return true?
I just found out that the 'size' is not set to zero when return
NULL for the implementation of probe API for the current version.
Perhaps it makes more sense to expect the API caller to make sure
the the returned 'page' not being NULL before using the 'offset',
'size' and 'va', like below:
df && page && page == df->page
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache
2024-05-10 9:48 ` Yunsheng Lin
@ 2024-05-10 17:29 ` Mat Martineau
2024-05-13 11:53 ` Yunsheng Lin
0 siblings, 1 reply; 8+ messages in thread
From: Mat Martineau @ 2024-05-10 17:29 UTC (permalink / raw
To: Yunsheng Lin
Cc: davem, kuba, pabeni, netdev, linux-kernel, Alexander Duyck,
Ayush Sawal, Eric Dumazet, Willem de Bruijn, Jason Wang,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Daniel Bristot de Oliveira, Valentin Schneider, John Fastabend,
Jakub Sitnicki, David Ahern, Matthieu Baerts, Geliang Tang,
Jamal Hadi Salim, Cong Wang, Jiri Pirko, Boris Pismenny, bpf,
mptcp
On Fri, 10 May 2024, Yunsheng Lin wrote:
> On 2024/5/10 0:22, Mat Martineau wrote:
>> On Wed, 8 May 2024, Yunsheng Lin wrote:
>>
>>> Use the newly introduced prepare/probe/commit API to
>>> replace page_frag with page_frag_cache for sk_page_frag().
>>>
>>> CC: Alexander Duyck <alexander.duyck@gmail.com>
>>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
>>> ---
>>> .../chelsio/inline_crypto/chtls/chtls.h | 3 -
>>> .../chelsio/inline_crypto/chtls/chtls_io.c | 100 ++++---------
>>> .../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
>>> drivers/net/tun.c | 28 ++--
>>> include/linux/sched.h | 4 +-
>>> include/net/sock.h | 14 +-
>>> kernel/exit.c | 3 +-
>>> kernel/fork.c | 3 +-
>>> net/core/skbuff.c | 32 ++--
>>> net/core/skmsg.c | 22 +--
>>> net/core/sock.c | 46 ++++--
>>> net/ipv4/ip_output.c | 33 +++--
>>> net/ipv4/tcp.c | 35 ++---
>>> net/ipv4/tcp_output.c | 28 ++--
>>> net/ipv6/ip6_output.c | 33 +++--
>>> net/kcm/kcmsock.c | 30 ++--
>>> net/mptcp/protocol.c | 70 +++++----
>>> net/sched/em_meta.c | 2 +-
>>> net/tls/tls_device.c | 139 ++++++++++--------
>>> 19 files changed, 331 insertions(+), 297 deletions(-)
>>>
>>
>> <snip>
>>
>>> diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
>>> index bb8f96f2b86f..ab844011d442 100644
>>> --- a/net/mptcp/protocol.c
>>> +++ b/net/mptcp/protocol.c
>>> @@ -960,17 +960,18 @@ static bool mptcp_skb_can_collapse_to(u64 write_seq,
>>> }
>>>
>>> /* we can append data to the given data frag if:
>>> - * - there is space available in the backing page_frag
>>> - * - the data frag tail matches the current page_frag free offset
>>> + * - there is space available for the current page
>>> + * - the data frag tail matches the current page and offset
>>> * - the data frag end sequence number matches the current write seq
>>> */
>>> static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
>>> - const struct page_frag *pfrag,
>>> + const struct page *page,
>>> + const unsigned int offset,
>>> + const unsigned int size,
>>
>> Hi Yunsheng -
>>
>> Why add the 'size' parameter here? It's checked to be a nonzero value,
>> but it can only be 0 if page is also NULL. In this case "page ==
>> df->page" will be false, so the function will return false even without
>> checking 'size'.
>
> Is it possible that the pfrag->page is also NULL, which may cause
> mptcp_frag_can_collapse_to() to return true?
Not sure. But I do know that df->page will never be NULL, so "page ==
df->page" will always be false when page == NULL.
>
> I just found out that the 'size' is not set to zero when return
> NULL for the implementation of probe API for the current version.
> Perhaps it makes more sense to expect the API caller to make sure
> the the returned 'page' not being NULL before using the 'offset',
> 'size' and 'va', like below:
>
> df && page && page == df->page
>
Given that df->page is never NULL, I don't think the extra "&& page" is
needed.
- Mat
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache
2024-05-10 17:29 ` Mat Martineau
@ 2024-05-13 11:53 ` Yunsheng Lin
2024-05-13 23:44 ` Mat Martineau
0 siblings, 1 reply; 8+ messages in thread
From: Yunsheng Lin @ 2024-05-13 11:53 UTC (permalink / raw
To: Mat Martineau
Cc: davem, kuba, pabeni, netdev, linux-kernel, Alexander Duyck,
Ayush Sawal, Eric Dumazet, Willem de Bruijn, Jason Wang,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Daniel Bristot de Oliveira, Valentin Schneider, John Fastabend,
Jakub Sitnicki, David Ahern, Matthieu Baerts, Geliang Tang,
Jamal Hadi Salim, Cong Wang, Jiri Pirko, Boris Pismenny, bpf,
mptcp
On 2024/5/11 1:29, Mat Martineau wrote:
> On Fri, 10 May 2024, Yunsheng Lin wrote:
>
>> On 2024/5/10 0:22, Mat Martineau wrote:
>>> On Wed, 8 May 2024, Yunsheng Lin wrote:
>>>
>>>> Use the newly introduced prepare/probe/commit API to
>>>> replace page_frag with page_frag_cache for sk_page_frag().
>>>>
>>>> CC: Alexander Duyck <alexander.duyck@gmail.com>
>>>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
>>>> ---
>>>> .../chelsio/inline_crypto/chtls/chtls.h | 3 -
>>>> .../chelsio/inline_crypto/chtls/chtls_io.c | 100 ++++---------
>>>> .../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
>>>> drivers/net/tun.c | 28 ++--
>>>> include/linux/sched.h | 4 +-
>>>> include/net/sock.h | 14 +-
>>>> kernel/exit.c | 3 +-
>>>> kernel/fork.c | 3 +-
>>>> net/core/skbuff.c | 32 ++--
>>>> net/core/skmsg.c | 22 +--
>>>> net/core/sock.c | 46 ++++--
>>>> net/ipv4/ip_output.c | 33 +++--
>>>> net/ipv4/tcp.c | 35 ++---
>>>> net/ipv4/tcp_output.c | 28 ++--
>>>> net/ipv6/ip6_output.c | 33 +++--
>>>> net/kcm/kcmsock.c | 30 ++--
>>>> net/mptcp/protocol.c | 70 +++++----
>>>> net/sched/em_meta.c | 2 +-
>>>> net/tls/tls_device.c | 139 ++++++++++--------
>>>> 19 files changed, 331 insertions(+), 297 deletions(-)
>>>>
>>>
>>> <snip>
>>>
>>>> diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
>>>> index bb8f96f2b86f..ab844011d442 100644
>>>> --- a/net/mptcp/protocol.c
>>>> +++ b/net/mptcp/protocol.c
>>>> @@ -960,17 +960,18 @@ static bool mptcp_skb_can_collapse_to(u64 write_seq,
>>>> }
>>>>
>>>> /* we can append data to the given data frag if:
>>>> - * - there is space available in the backing page_frag
>>>> - * - the data frag tail matches the current page_frag free offset
>>>> + * - there is space available for the current page
>>>> + * - the data frag tail matches the current page and offset
>>>> * - the data frag end sequence number matches the current write seq
>>>> */
>>>> static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
>>>> - const struct page_frag *pfrag,
>>>> + const struct page *page,
>>>> + const unsigned int offset,
>>>> + const unsigned int size,
>>>
>>> Hi Yunsheng -
>>>
>>> Why add the 'size' parameter here? It's checked to be a nonzero value, but it can only be 0 if page is also NULL. In this case "page == df->page" will be false, so the function will return false even without checking 'size'.
>>
>> Is it possible that the pfrag->page is also NULL, which may cause
>> mptcp_frag_can_collapse_to() to return true?
>
> Not sure. But I do know that df->page will never be NULL, so "page == df->page" will always be false when page == NULL.
>
>>
>> I just found out that the 'size' is not set to zero when return
>> NULL for the implementation of probe API for the current version.
>> Perhaps it makes more sense to expect the API caller to make sure
>> the the returned 'page' not being NULL before using the 'offset',
>> 'size' and 'va', like below:
>>
>> df && page && page == df->page
>>
>
> Given that df->page is never NULL, I don't think the extra "&& page" is needed.
Not checking the extra "&& page" seems to cause the below warning, it seems we
have the below options:
1. ignore the warning.
2. set offset to zero if there is no enough space when probe API asks for a specific
amount of available space as you suggested.
3. add the "&& page" in mptcp_frag_can_collapse_to()
what is your favour option? or any other better option?
net-mptcp-protocol.c:warning:variable-offset-is-used-uninitialized-whenever-if-condition-is-false
>
> - Mat
> .
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache
2024-05-13 11:53 ` Yunsheng Lin
@ 2024-05-13 23:44 ` Mat Martineau
0 siblings, 0 replies; 8+ messages in thread
From: Mat Martineau @ 2024-05-13 23:44 UTC (permalink / raw
To: Yunsheng Lin
Cc: davem, kuba, pabeni, netdev, linux-kernel, Alexander Duyck,
Ayush Sawal, Eric Dumazet, Willem de Bruijn, Jason Wang,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Daniel Bristot de Oliveira, Valentin Schneider, John Fastabend,
Jakub Sitnicki, David Ahern, Matthieu Baerts, Geliang Tang,
Jamal Hadi Salim, Cong Wang, Jiri Pirko, Boris Pismenny, bpf,
mptcp
On Mon, 13 May 2024, Yunsheng Lin wrote:
> On 2024/5/11 1:29, Mat Martineau wrote:
>> On Fri, 10 May 2024, Yunsheng Lin wrote:
>>
>>> On 2024/5/10 0:22, Mat Martineau wrote:
>>>> On Wed, 8 May 2024, Yunsheng Lin wrote:
>>>>
>>>>> Use the newly introduced prepare/probe/commit API to
>>>>> replace page_frag with page_frag_cache for sk_page_frag().
>>>>>
>>>>> CC: Alexander Duyck <alexander.duyck@gmail.com>
>>>>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
>>>>> ---
>>>>> .../chelsio/inline_crypto/chtls/chtls.h | 3 -
>>>>> .../chelsio/inline_crypto/chtls/chtls_io.c | 100 ++++---------
>>>>> .../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
>>>>> drivers/net/tun.c | 28 ++--
>>>>> include/linux/sched.h | 4 +-
>>>>> include/net/sock.h | 14 +-
>>>>> kernel/exit.c | 3 +-
>>>>> kernel/fork.c | 3 +-
>>>>> net/core/skbuff.c | 32 ++--
>>>>> net/core/skmsg.c | 22 +--
>>>>> net/core/sock.c | 46 ++++--
>>>>> net/ipv4/ip_output.c | 33 +++--
>>>>> net/ipv4/tcp.c | 35 ++---
>>>>> net/ipv4/tcp_output.c | 28 ++--
>>>>> net/ipv6/ip6_output.c | 33 +++--
>>>>> net/kcm/kcmsock.c | 30 ++--
>>>>> net/mptcp/protocol.c | 70 +++++----
>>>>> net/sched/em_meta.c | 2 +-
>>>>> net/tls/tls_device.c | 139 ++++++++++--------
>>>>> 19 files changed, 331 insertions(+), 297 deletions(-)
>>>>>
>>>>
>>>> <snip>
>>>>
>>>>> diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
>>>>> index bb8f96f2b86f..ab844011d442 100644
>>>>> --- a/net/mptcp/protocol.c
>>>>> +++ b/net/mptcp/protocol.c
>>>>> @@ -960,17 +960,18 @@ static bool mptcp_skb_can_collapse_to(u64 write_seq,
>>>>> }
>>>>>
>>>>> /* we can append data to the given data frag if:
>>>>> - * - there is space available in the backing page_frag
>>>>> - * - the data frag tail matches the current page_frag free offset
>>>>> + * - there is space available for the current page
>>>>> + * - the data frag tail matches the current page and offset
>>>>> * - the data frag end sequence number matches the current write seq
>>>>> */
>>>>> static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
>>>>> - const struct page_frag *pfrag,
>>>>> + const struct page *page,
>>>>> + const unsigned int offset,
>>>>> + const unsigned int size,
>>>>
>>>> Hi Yunsheng -
>>>>
>>>> Why add the 'size' parameter here? It's checked to be a nonzero value, but it can only be 0 if page is also NULL. In this case "page == df->page" will be false, so the function will return false even without checking 'size'.
>>>
>>> Is it possible that the pfrag->page is also NULL, which may cause
>>> mptcp_frag_can_collapse_to() to return true?
>>
>> Not sure. But I do know that df->page will never be NULL, so "page == df->page" will always be false when page == NULL.
>>
>>>
>>> I just found out that the 'size' is not set to zero when return
>>> NULL for the implementation of probe API for the current version.
>>> Perhaps it makes more sense to expect the API caller to make sure
>>> the the returned 'page' not being NULL before using the 'offset',
>>> 'size' and 'va', like below:
>>>
>>> df && page && page == df->page
>>>
>>
>> Given that df->page is never NULL, I don't think the extra "&& page" is needed.
>
> Not checking the extra "&& page" seems to cause the below warning, it seems we
> have the below options:
> 1. ignore the warning.
> 2. set offset to zero if there is no enough space when probe API asks for a specific
> amount of available space as you suggested.
> 3. add the "&& page" in mptcp_frag_can_collapse_to()
>
> what is your favour option? or any other better option?
>
> net-mptcp-protocol.c:warning:variable-offset-is-used-uninitialized-whenever-if-condition-is-false
>
Hi Yunsheng -
That static analyzer is correct that "offset" is *passed* uninitialized in
that scenario, but it doesn't recognize that "offset" is never compared
when page == NULL. So, it's a false positive in a way.
I don't think implementing fix #2 in the page_frag_alloc_probe() macro is
best, since the warning is specific to the MPTCP code and other
page_frag_cache users may have reasons to choose nonzero default values
for the offset. I suggest initializing offset = 0 where it is declared in
mptcp_sendmsg().
- Mat
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2024-05-13 23:44 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-08 13:33 [PATCH net-next v3 00/13] First try to replace page_frag with page_frag_cache Yunsheng Lin
2024-05-08 13:34 ` [PATCH net-next v3 06/13] mm: page_frag: add '_va' suffix to page_frag API Yunsheng Lin
2024-05-08 13:34 ` [PATCH net-next v3 11/13] net: replace page_frag with page_frag_cache Yunsheng Lin
2024-05-09 16:22 ` Mat Martineau
2024-05-10 9:48 ` Yunsheng Lin
2024-05-10 17:29 ` Mat Martineau
2024-05-13 11:53 ` Yunsheng Lin
2024-05-13 23:44 ` Mat Martineau
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).