From: Tejun Heo <tj@kernel.org>
To: Parav Pandit <pandit.parav@gmail.com>
Cc: cgroups@vger.kernel.org, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
lizefan@huawei.com, Johannes Weiner <hannes@cmpxchg.org>,
Doug Ledford <dledford@redhat.com>,
Jonathan Corbet <corbet@lwn.net>,
james.l.morris@oracle.com, serge@hallyn.com,
Haggai Eran <haggaie@mellanox.com>,
Or Gerlitz <ogerlitz@mellanox.com>,
Matan Barak <matanb@mellanox.com>,
raindel@mellanox.com, akpm@linux-foundation.org,
linux-security-module@vger.kernel.org
Subject: Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource
Date: Fri, 11 Sep 2015 00:04:13 -0400 [thread overview]
Message-ID: <20150911040413.GA18850@htj.duckdns.org> (raw)
In-Reply-To: <CAG53R5WtuPA=J_GYPzNTAKbjB1r0K90qhXEDxLNf7vxYyxgrKA@mail.gmail.com>
Hello, Parav.
On Fri, Sep 11, 2015 at 09:09:58AM +0530, Parav Pandit wrote:
> The fact is that user level application uses hardware resources.
> Verbs layer is software abstraction for it. Drivers are hiding how
> they implement this QP or CQ or whatever hardware resource they
> project via API layer.
> For all of the userland on top of verb layer I mentioned above, the
> common resource abstraction is these resources AH, QP, CQ, MR etc.
> Hardware (and driver) might have different view of this resource in
> their real implementation.
> For example, verb layer can say that it has 100 QPs, but hardware
> might actually have 20 QPs that driver decide how to efficiently use
> it.
My uneducated suspicion is that the abstraction is just not developed
enough. It should be possible to virtualize these resources through,
most likely, time-sharing to the level where userland simply says "I
want this chunk transferred there" and OS schedules the transfer
prioritizing competing requests.
It could be that given the use cases rdma might not need such level of
abstraction - e.g. most users want to be and are pretty close to bare
metal, but, if that's true, it also kinda is weird to build
hierarchical resource distribution scheme on top of such bare
abstraction.
...
> > I don't know. What's proposed in this thread seems way too low level
> > to be useful anywhere else. Also, what if there are multiple devices?
> > Is that a problem to worry about?
>
> o.k. It doesn't have to be useful anywhere else. If it suffice the
> need of RDMA applications, its fine for near future.
> This patch allows limiting resources across multiple devices.
> As we go along the path, and if requirement come up to have knob on
> per device basis, thats something we can extend in future.
You kinda have to decide that upfront cuz it gets baked into the
interface.
> > I'm kinda doubtful we're gonna have too many of these. Hardware
> > details being exposed to userland this directly isn't common.
>
> Its common in RDMA applications. Again they may not be real hardware
> resource, its just API layer which defines those RDMA constructs.
It's still a very low level of abstraction which pretty much gets
decided by what the hardware and driver decide to do.
> > I'd say keep it simple and do the minimum. :)
>
> o.k. In that case new rdma cgroup controller which does rdma resource
> accounting is possibly the most simplest form?
> Make sense?
So, this fits cgroup's purpose to certain level but it feels like
we're trying to build too much on top of something which hasn't
developed sufficiently. I suppose it could be that this is the level
of development that rdma is gonna reach and dumb cgroup controller can
be useful for some use cases. I don't know, so, yeah, let's keep it
simple and avoid doing crazy stuff.
Thanks.
--
tejun
WARNING: multiple messages have this Message-ID (diff)
From: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
To: Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
lizefan-hv44wF8Li93QT0dZR+AlfA@public.gmane.org,
Johannes Weiner <hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org>,
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
Jonathan Corbet <corbet-T1hC0tSOHrs@public.gmane.org>,
james.l.morris-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org,
serge-A9i7LUbDfNHQT0dZR+AlfA@public.gmane.org,
Haggai Eran <haggaie-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
raindel-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org,
linux-security-module-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource
Date: Fri, 11 Sep 2015 00:04:13 -0400 [thread overview]
Message-ID: <20150911040413.GA18850@htj.duckdns.org> (raw)
In-Reply-To: <CAG53R5WtuPA=J_GYPzNTAKbjB1r0K90qhXEDxLNf7vxYyxgrKA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Hello, Parav.
On Fri, Sep 11, 2015 at 09:09:58AM +0530, Parav Pandit wrote:
> The fact is that user level application uses hardware resources.
> Verbs layer is software abstraction for it. Drivers are hiding how
> they implement this QP or CQ or whatever hardware resource they
> project via API layer.
> For all of the userland on top of verb layer I mentioned above, the
> common resource abstraction is these resources AH, QP, CQ, MR etc.
> Hardware (and driver) might have different view of this resource in
> their real implementation.
> For example, verb layer can say that it has 100 QPs, but hardware
> might actually have 20 QPs that driver decide how to efficiently use
> it.
My uneducated suspicion is that the abstraction is just not developed
enough. It should be possible to virtualize these resources through,
most likely, time-sharing to the level where userland simply says "I
want this chunk transferred there" and OS schedules the transfer
prioritizing competing requests.
It could be that given the use cases rdma might not need such level of
abstraction - e.g. most users want to be and are pretty close to bare
metal, but, if that's true, it also kinda is weird to build
hierarchical resource distribution scheme on top of such bare
abstraction.
...
> > I don't know. What's proposed in this thread seems way too low level
> > to be useful anywhere else. Also, what if there are multiple devices?
> > Is that a problem to worry about?
>
> o.k. It doesn't have to be useful anywhere else. If it suffice the
> need of RDMA applications, its fine for near future.
> This patch allows limiting resources across multiple devices.
> As we go along the path, and if requirement come up to have knob on
> per device basis, thats something we can extend in future.
You kinda have to decide that upfront cuz it gets baked into the
interface.
> > I'm kinda doubtful we're gonna have too many of these. Hardware
> > details being exposed to userland this directly isn't common.
>
> Its common in RDMA applications. Again they may not be real hardware
> resource, its just API layer which defines those RDMA constructs.
It's still a very low level of abstraction which pretty much gets
decided by what the hardware and driver decide to do.
> > I'd say keep it simple and do the minimum. :)
>
> o.k. In that case new rdma cgroup controller which does rdma resource
> accounting is possibly the most simplest form?
> Make sense?
So, this fits cgroup's purpose to certain level but it feels like
we're trying to build too much on top of something which hasn't
developed sufficiently. I suppose it could be that this is the level
of development that rdma is gonna reach and dumb cgroup controller can
be useful for some use cases. I don't know, so, yeah, let's keep it
simple and avoid doing crazy stuff.
Thanks.
--
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2015-09-11 4:04 UTC|newest]
Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-09-07 20:38 [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit
2015-09-07 20:38 ` Parav Pandit
2015-09-07 20:38 ` [PATCH 1/7] devcg: Added user option to rdma resource tracking Parav Pandit
2015-09-07 20:38 ` Parav Pandit
2015-09-07 20:38 ` [PATCH 2/7] devcg: Added rdma resource tracking module Parav Pandit
2015-09-07 20:38 ` Parav Pandit
2015-09-07 20:38 ` [PATCH 3/7] devcg: Added infrastructure for rdma device cgroup Parav Pandit
2015-09-08 5:31 ` Haggai Eran
2015-09-08 5:31 ` Haggai Eran
2015-09-08 7:02 ` Parav Pandit
2015-09-08 7:02 ` Parav Pandit
2015-09-07 20:38 ` [PATCH 4/7] devcg: Added rdma resource tracker object per task Parav Pandit
2015-09-08 5:48 ` Haggai Eran
2015-09-08 5:48 ` Haggai Eran
2015-09-08 7:04 ` Parav Pandit
2015-09-08 8:24 ` Haggai Eran
2015-09-08 8:24 ` Haggai Eran
2015-09-08 8:26 ` Parav Pandit
2015-09-07 20:38 ` [PATCH 5/7] devcg: device cgroup's extension for RDMA resource Parav Pandit
2015-09-07 20:38 ` Parav Pandit
2015-09-08 8:22 ` Haggai Eran
2015-09-08 8:22 ` Haggai Eran
2015-09-08 10:18 ` Parav Pandit
2015-09-08 13:50 ` Haggai Eran
2015-09-08 13:50 ` Haggai Eran
2015-09-08 14:13 ` Parav Pandit
2015-09-08 8:36 ` Haggai Eran
2015-09-08 8:36 ` Haggai Eran
2015-09-08 10:50 ` Parav Pandit
2015-09-08 10:50 ` Parav Pandit
2015-09-08 14:10 ` Haggai Eran
2015-09-08 14:10 ` Haggai Eran
2015-09-07 20:38 ` [PATCH 6/7] devcg: Added support to use RDMA device cgroup Parav Pandit
2015-09-08 8:40 ` Haggai Eran
2015-09-08 8:40 ` Haggai Eran
2015-09-08 10:22 ` Parav Pandit
2015-09-08 13:40 ` Haggai Eran
2015-09-08 13:40 ` Haggai Eran
2015-09-07 20:38 ` [PATCH 7/7] devcg: Added Documentation of " Parav Pandit
2015-09-07 20:38 ` Parav Pandit
2015-09-07 20:55 ` [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit
2015-09-08 12:45 ` Haggai Eran
2015-09-08 12:45 ` Haggai Eran
2015-09-08 15:23 ` Tejun Heo
2015-09-08 15:23 ` Tejun Heo
2015-09-09 3:57 ` Parav Pandit
2015-09-10 16:49 ` Tejun Heo
2015-09-10 17:46 ` Parav Pandit
2015-09-10 17:46 ` Parav Pandit
2015-09-10 20:22 ` Tejun Heo
2015-09-11 3:39 ` Parav Pandit
2015-09-11 4:04 ` Tejun Heo [this message]
2015-09-11 4:04 ` Tejun Heo
2015-09-11 4:24 ` Doug Ledford
2015-09-11 4:24 ` Doug Ledford
2015-09-11 14:52 ` Tejun Heo
2015-09-11 14:52 ` Tejun Heo
2015-09-11 16:26 ` Parav Pandit
2015-09-11 16:34 ` Tejun Heo
2015-09-11 16:34 ` Tejun Heo
2015-09-11 16:39 ` Parav Pandit
2015-09-11 16:39 ` Parav Pandit
2015-09-11 19:25 ` Tejun Heo
2015-09-14 10:18 ` Parav Pandit
2015-09-14 10:18 ` Parav Pandit
2015-09-11 16:47 ` Parav Pandit
2015-09-11 16:47 ` Parav Pandit
2015-09-11 19:05 ` Tejun Heo
2015-09-11 19:05 ` Tejun Heo
2015-09-11 19:22 ` Hefty, Sean
2015-09-11 19:43 ` Jason Gunthorpe
2015-09-11 19:43 ` Jason Gunthorpe
2015-09-11 20:06 ` Hefty, Sean
2015-09-14 11:09 ` Parav Pandit
2015-09-14 14:04 ` Parav Pandit
2015-09-14 15:21 ` Tejun Heo
2015-09-14 15:21 ` Tejun Heo
2015-09-14 17:28 ` Jason Gunthorpe
2015-09-14 17:28 ` Jason Gunthorpe
2015-09-14 18:54 ` Parav Pandit
2015-09-14 18:54 ` Parav Pandit
2015-09-14 20:18 ` Jason Gunthorpe
2015-09-15 3:08 ` Parav Pandit
2015-09-15 3:45 ` Jason Gunthorpe
2015-09-15 3:45 ` Jason Gunthorpe
2015-09-16 4:41 ` Parav Pandit
2015-09-16 4:41 ` Parav Pandit
2015-09-20 10:35 ` Haggai Eran
2015-09-20 10:35 ` Haggai Eran
2015-10-28 8:14 ` Parav Pandit
2015-10-28 8:14 ` Parav Pandit
2015-09-14 10:15 ` Parav Pandit
2015-09-11 4:43 ` Parav Pandit
2015-09-11 15:03 ` Tejun Heo
2015-09-10 17:48 ` Hefty, Sean
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150911040413.GA18850@htj.duckdns.org \
--to=tj@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=cgroups@vger.kernel.org \
--cc=corbet@lwn.net \
--cc=dledford@redhat.com \
--cc=haggaie@mellanox.com \
--cc=hannes@cmpxchg.org \
--cc=james.l.morris@oracle.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=linux-security-module@vger.kernel.org \
--cc=lizefan@huawei.com \
--cc=matanb@mellanox.com \
--cc=ogerlitz@mellanox.com \
--cc=pandit.parav@gmail.com \
--cc=raindel@mellanox.com \
--cc=serge@hallyn.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.