From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751312AbcBJMn2 (ORCPT ); Wed, 10 Feb 2016 07:43:28 -0500 Received: from mailout2.samsung.com ([203.254.224.25]:55752 "EHLO mailout2.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751259AbcBJMnX (ORCPT ); Wed, 10 Feb 2016 07:43:23 -0500 X-AuditID: cbfee68e-f793c6d00000136c-56-56bb306a8040 Date: Wed, 10 Feb 2016 12:43:22 +0000 (GMT) From: Vaneet Narang Subject: Re: [PATCH] af_packet: Raw socket destruction warning fix To: Daniel Borkmann , Maninder Singh , Vaneet Narang Cc: "davem@davemloft.net" , "willemb@google.com" , "edumazet@google.com" , "eyal.birger@gmail.com" , "tklauser@distanz.ch" , "fruggeri@aristanetworks.com" , "dwmw2@infradead.org" , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , PANKAJ MISHRA , Geon-ho Kim , Hak-Bong Lee , "ajeet.y@samsung.com" , AKHILESH KUMAR , AMIT NAGAL Reply-to: v.narang@samsung.com MIME-version: 1.0 X-MTR: 20160210123925351@v.narang Msgkey: 20160210123925351@v.narang X-EPLocale: en_US.windows-1252 X-Priority: 3 X-EPWebmail-Msg-Type: personal X-EPWebmail-Reply-Demand: 0 X-EPApproval-Locale: X-EPHeader: ML X-MLAttribute: X-RootMTR: 20160210123925351@v.narang X-ParentMTR: X-ArchiveUser: X-CPGSPASS: N X-ConfirmMail: N,general Content-type: text/plain; charset=windows-1252 MIME-version: 1.0 Message-id: <230561650.749911455108197509.JavaMail.weblogic@epmlwas05a> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrBIsWRmVeSWpSXmKPExsWyRsSkTjfLYHeYwfW9whaXd81hc2D0+LxJ LoAxissmJTUnsyy1SN8ugSvj1tSFTAV31Cv6br5maWC8otbFyMkhJKAs0XntGiuILSFgInH+ z08WCFtM4sK99WxdjFxANUsZJRYsu8sGU3Tq4TMmiMQcRom25bPYQRIsAqoSx07MBZvEJqAt 8eZfL9gkYQEniTfdP1hBGkQEWhglGtdeZARxmAWusUo8n7OQDeIOOYm/C34xg9i8AoISJ2c+ gbpDUWLtv61QcSWJ3Vufs0PE5SSWTL3MBGHzSsxof8oCE5/2dQ0zhC0tcX7WBkaYfxZ/fwwV 55c4dnsHVK+AxNQzB6Fq1CRePfgBNZ9PYs3Ctyww9btOLWeG2XV/y1yoXgmJrS1PwD5mBrpz SvdDdgjbQOLIojms6H7hFXCXePnuFDgkJAQmckisvTSPaQKj0iwkdbOQzJqFZBaymgWMLKsY RVMLkguKk9KLjPSKE3OLS/PS9ZLzczcxAhPE6X/P+nYw3jxgfYhRgINRiYf3hsmuMCHWxLLi ytxDjKbAmJrILCWanA9MQ3kl8YbGZkYWpiamxkbmlmZK4rwJUj+DhQTSE0tSs1NTC1KL4otK c1KLDzEycXBKNTB2fhaYs1rqF6vFkuCqC5e+l051/xKf9CRsftnXc+4ZE+ecWVt7qDV2UZHB CtPbbF8PMhQ+/rpuV4/47tscKyvmtaTJbH6VY73naWF00iLBGQccDhWtEZumMWESRwVbsPbe WXmHtjJZRU5TTU/asGtHUWoLV1vsmQWpm7l+GNgKFLauaek+I9WnxFKckWioxVxUnAgARczb 5QsDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrJKsWRmVeSWpSXmKPExsVy+t/tXt0sg91hBr9fc1hc3jWHzYHR4/Mm uQDGqDSbjNTElNQihdS85PyUzLx0WyXv4HjneFMzA0NdQ0sLcyWFvMTcVFslF58AXbfMHKCh SgpliTmlQKGAxOJiJX07m6L80pJUhYz84hJbpWhDcyM9IwM9UyM9Q9NYK0MDAyNToJqEtIxb UxcyFdxRr+i7+ZqlgfGKWhcjJ4eQgLJE57VrrCC2hICJxKmHz5ggbDGJC/fWs3UxcgHVzGGU aFs+ix0kwSKgKnHsxFywBjYBbYk3/3pZQGxhASeJN90/WEEaRARaGCUa115kBHGYBa6xSjyf s5ANYp2cxN8Fv5hBbF4BQYmTM5+wQKxTlFj7bytUXEli99bn7BBxOYklUy9DncQrMaP9KQtM fNrXNcwQtrTE+VkbGGHOXvz9MVScX+LY7R1QvQISU88chKpRk3j14AfUfD6JNQvfssDU7zq1 nBlm1/0tc6F6JSS2tjwB+5gZ6M4p3Q/ZIWwDiSOL5rCi+4VXwF3i5btTrBMYZWchSc1C0j4L STuymgWMLKsYRVMLkguKk9IrjPWKE3OLS/PS9ZLzczcxgpPRs8U7GP+ftz7EKMDBqMTDe9Nk V5gQa2JZcWXuIUYJDmYlEd6/ErvDhHhTEiurUovy44tKc1KLDzGaAuNtIrOUaHI+MFHmlcQb GpuYmxqbWhgYmpubKYnzBvxdFyYkkJ5YkpqdmlqQWgTTx8TBKdXA6KleIHlP4cXODNGiQ7bG DtOL33ZFXuW5OmP18k3TE+9klG+7+PP8YmejV3u+Vs/be1Nf+Xo89/GL0/L3Tvy21PbKWqdH oer5OW4qQdO3rgt2cuvVS3hxaNb+4uU/JnfI5XnMCFDOZjdTclglYHdEyctGtHaG6e4QmSly 5vFMAecTxJbPerO9SYmlOCPRUIu5qDgRAC+sWmxcAwAA DLP-Filter: Pass X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id u1AChWmZ011161 Hi, >What driver are you using (is that in-tree)? Can you reproduce the same issue >with a latest -net kernel, for example (or, a 'reasonably' recent one like 4.3 or >4.4)? There has been quite a bit of changes in err queue handling (which also >accounts rmem) as well. How reliably can you trigger the issue? Does it trigger >with a completely different in-tree network driver as well with your tests? Would >be useful to track/debug sk_rmem_alloc increases/decreases to see from which path >new rmem is being charged in the time between packet_release() and packet_sock_destruct() >for that socket ... > It seems race condition to us between packet_rcv and packet_close, we have tried to reproduce this issue by adding delay in skb_set_owner_r and issue gets reproduced quite frequently. we have added some traces and on analyzing we have realised following possible race condition. packet_rcv packet_close skb_set_owner_r(skb, sk); skb_queue_purge(&sk->sk_receive_queue); spin_lock(&sk->sk_receive_queue.lock); __skb_queue_tail(&sk->sk_receive_queue, skb); spin_unlock(&sk->sk_receive_queue.lock); Since packet was not added to receive queue so receive queue purge will no have any impact. It will not free sk_buff stored in receive queue. So to fix this issue, we have make sure skb_set_owner_r(skb, sk) & __skb_queue_tail(&sk->sk_receive_queue, skb) is called under receive queue lock and we have moved receive queue purge from packet_release to packet_sock_destruct. we have added some traces in skb_set_owner_r & packet_sock_destruct. So this is what we got CPU 0 sk = db6d17c0 sk->sk_flags = 0x320 Size = 1984 Backtrace: (dump_backtrace+0x0/0x128) from (show_stack+0x20/0x28) (show_stack+0x0/0x28) from (dump_stack+0x24/0x28) (dump_stack+0x0/0x28) from (packet_rcv+0x480/0x488) (packet_rcv+0x0/0x488) from (__netif_receive_skb_core+0x53c/0x674) (__netif_receive_skb_core+0x0/0x674) from (__netif_receive_skb+0x20/0x74) (__netif_receive_skb+0x0/0x74) from (netif_receive_skb+0x2c/0xbc) (netif_receive_skb+0x0/0xbc) from (napi_gro_receive+0x90/0xc0) ...... (net_rx_action+0x0/0x300) from(__do_softirq+0x160/0x340) (__do_softirq+0x0/0x340) from (do_softirq+0xc4/0xe0) (do_softirq+0x0/0xe0) from (irq_exit+0xc4/0xf8) (irq_exit+0x0/0xf8) from (handle_IRQ+0x88/0x10c) (handle_IRQ+0x0/0x10c) from (gic_handle_irq+0x64/0xac) CPU 1 Backtrace: sk = db6d17c0 sk->sk_rmem_alloc=1984 sk->sk_flags = 0x141 Receive Queue Empty = "Yes" Error queue empty = "Yes" (packet_sock_destruct+0x0/0x1f4) from (__sk_free+0x28/0x18c) (__sk_free+0x0/0x18c) from (sk_free+0x40/0x48) (sk_free+0x0/0x48) from (packet_release+0x29c/0x310) (packet_release+0x0/0x310) from (sock_release+0x30/0xb8) (sock_release+0x0/0xb8) from (sock_close+0x1c/0x28) (sock_close+0x0/0x28) from (__fput+0x9c/0x2b4) (__fput+0x0/0x2b4) from (____fput+0x18/0x20) (____fput+0x0/0x20) from (task_work_run+0xc0/0xfc) (task_work_run+0x0/0xfc) from (do_work_pending+0x108/0x114) (do_work_pending+0x0/0x114) from (work_pending+0xc/0x20) >>From this it appears packet_rcv was called when packet_release was not done as sk_flags = 0x320 (SOCK_DEAD is not set) & packet_sock_destruct was called when sk_rmem_alloc was increased but packet was not added to receive queue. sk_rmem_alloc pending is same as size of last packet received on socket. Kindly comment on the fix shared at following link. http://www.spinics.net/lists/kernel/msg2184815.html Thanks & Regards, Vaneet Narang