From: Jason Gunthorpe <jgg@ziepe.ca>
To: Zhu Yanjun <yanjun.zhu@linux.dev>
Cc: Konstantin Taranov <kotaranov@microsoft.com>,
Brian Baboch <brian.baboch@gmail.com>,
Leon Romanovsky <leon@kernel.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
"stable@vger.kernel.org" <stable@vger.kernel.org>,
"florent.fourcot@wifirst.fr" <florent.fourcot@wifirst.fr>,
"brian.baboch@wifirst.fr" <brian.baboch@wifirst.fr>
Subject: Re: Excessive memory usage when infiniband config is enabled
Date: Tue, 7 May 2024 13:37:59 -0300 [thread overview]
Message-ID: <20240507163759.GF4718@ziepe.ca> (raw)
In-Reply-To: <fa606d14-c35b-4d27-95fe-93e2192f1f52@linux.dev>
On Tue, May 07, 2024 at 05:24:51PM +0200, Zhu Yanjun wrote:
> 在 2024/5/7 15:32, Konstantin Taranov 写道:
> > > Hello Leon,
> > >
> > > I feel that it's a bug because I don't understand why is this module/option
> > > allocating 6GB of RAM without any explicit configuration or usage from us.
> > > It's also worth mentioning that we are using the default linux-image from
> > > Debian bookworm, and it took us a long time to understand the reason
> > > behind this memory increase by bisecting the kernel's config file.
> > > Moreover the documentation of the module doesn't mention anything
> > > regarding additional memory usage, we're talking about an increase of 6Gb
> > > which is huge since we're not using the option.
> > > So is that an expected behavior, to have this much increase in the memory
> > > consumption, when activating the RDMA option even if we're not using it ? If
> > > that's the case, perhaps it would be good to mention this in the
> > > documentation.
> > >
> > > Thank you
> > >
> >
> > Hi Brian,
> >
> > I do not think it is a bug. The high memory usage seems to come from these lines:
> > rsrc_size = irdma_calc_mem_rsrc_size(rf);
> > rf->mem_rsrc = vzalloc(rsrc_size);
>
> Exactly. The memory usage is related with the number of QP.
> When on irdma, the Queue Pairs is 4092, Completion Queues is 8189, the
> memory usage is about 4194302.
>
> The command "modprobe irdma limits_sel" will change QP numbers.
> 0 means minimum, up to 124 QPs.
>
> Please use the command "modprobe irdma limits_sel=0" to make tests.
> Please let us know the test results.
It seems like a really unfortunate design choice in this driver to not
have dynamic memory allocation.
Burning 6G on every server that has your HW, regardless if any RDMA
apps are run, seems completely excessive.
Jason
next prev parent reply other threads:[~2024-05-07 16:38 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-06 15:15 Excessive memory usage when infiniband config is enabled Brian Baboch
2024-05-07 11:27 ` Leon Romanovsky
2024-05-07 13:15 ` Brian Baboch
2024-05-07 13:32 ` Konstantin Taranov
2024-05-07 15:24 ` Zhu Yanjun
2024-05-07 16:37 ` Jason Gunthorpe [this message]
2024-05-08 1:24 ` Saleem, Shiraz
2024-05-13 13:59 ` Brian Baboch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240507163759.GF4718@ziepe.ca \
--to=jgg@ziepe.ca \
--cc=brian.baboch@gmail.com \
--cc=brian.baboch@wifirst.fr \
--cc=florent.fourcot@wifirst.fr \
--cc=kotaranov@microsoft.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=stable@vger.kernel.org \
--cc=yanjun.zhu@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).