From: Vaneet Narang <v.narang@samsung.com>
To: Daniel Borkmann <daniel@iogearbox.net>,
Maninder Singh <maninder1.s@samsung.com>,
Vaneet Narang <v.narang@samsung.com>
Cc: "davem@davemloft.net" <davem@davemloft.net>,
"willemb@google.com" <willemb@google.com>,
"edumazet@google.com" <edumazet@google.com>,
"eyal.birger@gmail.com" <eyal.birger@gmail.com>,
"tklauser@distanz.ch" <tklauser@distanz.ch>,
"fruggeri@aristanetworks.com" <fruggeri@aristanetworks.com>,
"dwmw2@infradead.org" <dwmw2@infradead.org>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
PANKAJ MISHRA <pankaj.m@samsung.com>,
Geon-ho Kim <gh007.kim@samsung.com>,
Hak-Bong Lee <hakbong5.lee@samsung.com>,
"ajeet.y@samsung.com" <ajeet.y@samsung.com>,
AKHILESH KUMAR <akhilesh.k@samsung.com>,
AMIT NAGAL <amit.nagal@samsung.com>
Subject: Re: [PATCH] af_packet: Raw socket destruction warning fix
Date: Wed, 10 Feb 2016 12:43:22 +0000 (GMT) [thread overview]
Message-ID: <230561650.749911455108197509.JavaMail.weblogic@epmlwas05a> (raw)
Hi,
>What driver are you using (is that in-tree)? Can you reproduce the same issue
>with a latest -net kernel, for example (or, a 'reasonably' recent one like 4.3 or
>4.4)? There has been quite a bit of changes in err queue handling (which also
>accounts rmem) as well. How reliably can you trigger the issue? Does it trigger
>with a completely different in-tree network driver as well with your tests? Would
>be useful to track/debug sk_rmem_alloc increases/decreases to see from which path
>new rmem is being charged in the time between packet_release() and packet_sock_destruct()
>for that socket ...
>
It seems race condition to us between packet_rcv and packet_close, we have tried to reproduce
this issue by adding delay in skb_set_owner_r and issue gets reproduced quite frequently.
we have added some traces and on analyzing we have realised following possible race condition.
packet_rcv
packet_close
skb_set_owner_r(skb, sk);
skb_queue_purge(&sk->sk_receive_queue);
spin_lock(&sk->sk_receive_queue.lock);
__skb_queue_tail(&sk->sk_receive_queue, skb);
spin_unlock(&sk->sk_receive_queue.lock);
Since packet was not added to receive queue so receive queue purge will no have any impact.
It will not free sk_buff stored in receive queue. So to fix this issue, we have make sure
skb_set_owner_r(skb, sk) & __skb_queue_tail(&sk->sk_receive_queue, skb) is called under
receive queue lock and we have moved receive queue purge from packet_release to
packet_sock_destruct.
we have added some traces in skb_set_owner_r & packet_sock_destruct. So this is
what we got
CPU 0
sk = db6d17c0 sk->sk_flags = 0x320 Size = 1984
Backtrace:
(dump_backtrace+0x0/0x128) from (show_stack+0x20/0x28)
(show_stack+0x0/0x28) from (dump_stack+0x24/0x28)
(dump_stack+0x0/0x28) from (packet_rcv+0x480/0x488)
(packet_rcv+0x0/0x488) from (__netif_receive_skb_core+0x53c/0x674)
(__netif_receive_skb_core+0x0/0x674) from (__netif_receive_skb+0x20/0x74)
(__netif_receive_skb+0x0/0x74) from (netif_receive_skb+0x2c/0xbc)
(netif_receive_skb+0x0/0xbc) from (napi_gro_receive+0x90/0xc0)
......
(net_rx_action+0x0/0x300) from(__do_softirq+0x160/0x340)
(__do_softirq+0x0/0x340) from (do_softirq+0xc4/0xe0)
(do_softirq+0x0/0xe0) from (irq_exit+0xc4/0xf8)
(irq_exit+0x0/0xf8) from (handle_IRQ+0x88/0x10c)
(handle_IRQ+0x0/0x10c) from (gic_handle_irq+0x64/0xac)
CPU 1
Backtrace:
sk = db6d17c0 sk->sk_rmem_alloc=1984 sk->sk_flags = 0x141
Receive Queue Empty = "Yes" Error queue empty = "Yes"
(packet_sock_destruct+0x0/0x1f4) from (__sk_free+0x28/0x18c)
(__sk_free+0x0/0x18c) from (sk_free+0x40/0x48)
(sk_free+0x0/0x48) from (packet_release+0x29c/0x310)
(packet_release+0x0/0x310) from (sock_release+0x30/0xb8)
(sock_release+0x0/0xb8) from (sock_close+0x1c/0x28)
(sock_close+0x0/0x28) from (__fput+0x9c/0x2b4)
(__fput+0x0/0x2b4) from (____fput+0x18/0x20)
(____fput+0x0/0x20) from (task_work_run+0xc0/0xfc)
(task_work_run+0x0/0xfc) from (do_work_pending+0x108/0x114)
(do_work_pending+0x0/0x114) from (work_pending+0xc/0x20)
>From this it appears packet_rcv was called when packet_release
was not done as sk_flags = 0x320 (SOCK_DEAD is not set) & packet_sock_destruct was called
when sk_rmem_alloc was increased but packet was not added to receive queue.
sk_rmem_alloc pending is same as size of last packet received on socket.
Kindly comment on the fix shared at following link.
http://www.spinics.net/lists/kernel/msg2184815.html
Thanks & Regards,
Vaneet Narang
next reply other threads:[~2016-02-10 12:43 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-10 12:43 Vaneet Narang [this message]
2016-02-10 14:56 ` [PATCH] af_packet: Raw socket destruction warning fix Eric Dumazet
-- strict thread matches above, loose matches on Subject: below --
2016-02-05 5:23 Vaneet Narang
2016-01-21 11:40 Maninder Singh
2016-01-26 0:13 ` Daniel Borkmann
2016-01-18 10:11 Vaneet Narang
2016-01-18 11:08 ` Daniel Borkmann
2016-01-18 6:37 Maninder Singh
2016-01-18 9:44 ` Daniel Borkmann
2016-01-18 10:29 ` Daniel Borkmann
2016-02-05 11:26 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=230561650.749911455108197509.JavaMail.weblogic@epmlwas05a \
--to=v.narang@samsung.com \
--cc=ajeet.y@samsung.com \
--cc=akhilesh.k@samsung.com \
--cc=amit.nagal@samsung.com \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=dwmw2@infradead.org \
--cc=edumazet@google.com \
--cc=eyal.birger@gmail.com \
--cc=fruggeri@aristanetworks.com \
--cc=gh007.kim@samsung.com \
--cc=hakbong5.lee@samsung.com \
--cc=linux-kernel@vger.kernel.org \
--cc=maninder1.s@samsung.com \
--cc=netdev@vger.kernel.org \
--cc=pankaj.m@samsung.com \
--cc=tklauser@distanz.ch \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).