From: Yi Zhang <yi.zhang@redhat.com>
To: Guoqing Jiang <guoqing.jiang@linux.dev>
Cc: RDMA mailing list <linux-rdma@vger.kernel.org>,
Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>,
Jason Gunthorpe <jgg@nvidia.com>,
leonro@nvidia.com
Subject: Re: [bug report] kmemleak in rdma_core observed during blktests nvme/rdma use siw
Date: Fri, 26 Apr 2024 13:56:04 +0800 [thread overview]
Message-ID: <CAHj4cs-RiZXAdz131ihQ=wsW8nsViphJeHAD_i6qi7_DtW7NwQ@mail.gmail.com> (raw)
In-Reply-To: <7de9793f-6805-1412-3fae-a5508910124b@linux.dev>
On Wed, Apr 24, 2024 at 9:28 PM Guoqing Jiang <guoqing.jiang@linux.dev> wrote:
>
> Hi,
>
> On 4/8/24 14:03, Yi Zhang wrote:
> > Hi
> > I found the below kmemleak issue during blktests nvme/rdma on the
> > latest linux-rdma/for-next, please help check it and let me know if
> > you need any info/testing for it, thanks.
>
> Could you share which test case caused the issue? I can't reproduce
> it with 6.9-rc3+ kernel (commit 586b5dfb51b) with the below.
It can be reproduced by [1], you can find more info from the symbol
info[2], I also attached the config file, maybe you can this config
file
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object 0xffff8883a87e8800 (size 192):
comm "rdma", pid 2355, jiffies 4294836069
hex dump (first 32 bytes):
32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2...............
10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~.....
backtrace (crc 4db191c4):
[<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0
[<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core]
[<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core]
[<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
[<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
[<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core]
[<ffffffffc24ac389>] 0xffffffffc24ac389
[<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
[<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]
[<ffffffffc208448c>]
rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[<ffffffff8e30e715>] netlink_unicast+0x445/0x710
[<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40
[<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420
[<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0
[<ffffffff8e9afad3>] do_syscall_64+0x93/0x180
[<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79
(gdb) l *(alloc_gid_entry+0x47)
0x2eff7 is in alloc_gid_entry (./include/linux/slab.h:628).
623
624 if (size > KMALLOC_MAX_CACHE_SIZE)
625 return kmalloc_large(size, flags);
626
627 index = kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
630 flags, size);
631 }
632 return __kmalloc(size, flags);
(gdb) l *(add_modify_gid+0x166)
0x30206 is in add_modify_gid (drivers/infiniband/core/cache.c:447).
442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry = alloc_gid_entry(attr);
448 if (!entry)
449 return -ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {
>
> use_siw=1 nvme_trtype=rdma ./check nvme/
>
> > # dmesg | grep kmemleak
> > [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem
> > pool available: 36041)
> > [ 67.130728] kmemleak: Automatic memory scanning thread started
> > [ 1051.771867] kmemleak: 2 new suspected memory leaks (see
> > /sys/kernel/debug/kmemleak)
> > [ 1832.796189] kmemleak: 8 new suspected memory leaks (see
> > /sys/kernel/debug/kmemleak)
> > [ 2578.189075] kmemleak: 17 new suspected memory leaks (see
> > /sys/kernel/debug/kmemleak)
> > [ 3330.710984] kmemleak: 4 new suspected memory leaks (see
> > /sys/kernel/debug/kmemleak)
> >
> > unreferenced object 0xffff88855da53400 (size 192):
> > comm "rdma", pid 10630, jiffies 4296575922
> > hex dump (first 32 bytes):
> > 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7...............
> > 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.]....
> > backtrace (crc 47f66721):
> > [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0
> > [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core]
> > [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]
>
> I guess add_modify_gid is called from config_non_roce_gid_cache, not sure
> why we don't check the return value of it here.
>
> Looks put_gid_entry is called in case add_modify_gid returns failure, it
> would
> trigger schedule_free_gid -> queue_work(ib_wq, &entry->del_work), then
> free_gid_work -> free_gid_entry_locked would free storage asynchronously by
> put_gid_ndev and also entry.
>
> > [<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
> > [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
> > [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core]
> > [<ffffffffc2a3d389>] 0xffffffffc2a3d389
> > [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
> > [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]
> > [<ffffffffc264648c>]
> > rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
> > [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710
> > [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40
> > [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420
> > [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0
> > [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180
> > [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79
>
> After ib_cache_setup_one failed, maybe ib_cache_cleanup_one is needed
> which flush ib_wq to ensure storage is freed. Could you try with the change?
Will try it later.
>
> --- a/drivers/infiniband/core/device.c
> +++ b/drivers/infiniband/core/device.c
> @@ -1388,7 +1388,7 @@ int ib_register_device(struct ib_device *device,
> const char *name,
> if (ret) {
> dev_warn(&device->dev,
> "Couldn't set up InfiniBand P_Key/GID cache\n");
> - return ret;
> + goto cache_cleanup;
> }
>
> Thanks,
> Guoqing
>
--
Best Regards,
Yi Zhang
next prev parent reply other threads:[~2024-04-26 5:56 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-08 6:03 [bug report] kmemleak in rdma_core observed during blktests nvme/rdma use siw Yi Zhang
2024-04-08 6:39 ` Leon Romanovsky
2024-04-08 6:57 ` Leon Romanovsky
2024-04-24 13:28 ` Guoqing Jiang
2024-04-26 5:56 ` Yi Zhang [this message]
2024-04-26 8:44 ` Yi Zhang
2024-04-28 2:54 ` Guoqing Jiang
2024-04-28 12:42 ` Yi Zhang
2024-04-29 0:53 ` Guoqing Jiang
2024-04-30 11:51 ` Yi Zhang
2024-05-08 13:08 ` [bug report][bisected] " Yi Zhang
2024-05-08 14:37 ` Zhu Yanjun
2024-05-08 15:31 ` Zhu Yanjun
2024-05-08 15:56 ` Jason Gunthorpe
2024-05-08 16:19 ` Chuck Lever
2024-05-10 1:41 ` Yi Zhang
2024-05-10 11:10 ` Zhu Yanjun
[not found] ` <a6b11edf-99e0-43bc-b49e-e0d6e201266b@linux.dev>
[not found] ` <CAHj4cs9XsHiKeVGNRjTCcg87ghWrDm972VMq+5w=8jvetjFG0w@mail.gmail.com>
2024-05-10 15:52 ` Zhu Yanjun
2024-04-24 15:57 ` [bug report] " Zhu Yanjun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAHj4cs-RiZXAdz131ihQ=wsW8nsViphJeHAD_i6qi7_DtW7NwQ@mail.gmail.com' \
--to=yi.zhang@redhat.com \
--cc=guoqing.jiang@linux.dev \
--cc=jgg@nvidia.com \
--cc=leonro@nvidia.com \
--cc=linux-rdma@vger.kernel.org \
--cc=shinichiro.kawasaki@wdc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.