From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754890AbbIHMqN (ORCPT ); Tue, 8 Sep 2015 08:46:13 -0400 Received: from mail-am1on0070.outbound.protection.outlook.com ([157.56.112.70]:8192 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754521AbbIHMqE (ORCPT ); Tue, 8 Sep 2015 08:46:04 -0400 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=bestguesspass action=none header.from=mellanox.com; Subject: Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource To: Parav Pandit , , , , , , , , References: <1441658303-18081-1-git-send-email-pandit.parav@gmail.com> CC: , , , , , , , From: Haggai Eran Message-ID: <55EED87E.3000408@mellanox.com> Date: Tue, 8 Sep 2015 15:45:50 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <1441658303-18081-1-git-send-email-pandit.parav@gmail.com> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.0.52.254] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;DB3FFO11FD003;1:YQmOExhgcNXEI9gxtOSSJ961aOHlO+6iZU9I/DRC0oDDM4q+tLnMsdTW8cCvxbjbHfH1Q7g/2/De1ibZ5K//CAwVLghB0SW+Vdqp6j+ymq2ZVFB31tSMXplxk5opO/fTE5e2NdHTLLN9jofLyFNKG0TJRc6wd2zMPY72RF1bDSlMw4v3bwpJUwgPjy9BudaXfABtOXZ7+6M8PSaHA5CY2pRzW7nrw0jlasJ7qnPznYd4HLkViPnuK4MSK4B4bQOfKSVEXiwkU01uu2EHENp8GiM9MMCQrgFOkjV5kvYylCBAFntfD5GOAtzVUAdgZVUs8770hAL2TgAoI+AZyUaOkRj7dPLw+u8WZxHD+DO13TwHtpKwX5k8vXTC/2ljo+dCHoetAYAy+g28Zc5VhAFxLw== X-Forefront-Antispam-Report: CIP:193.47.165.134;CTRY:IL;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(438002)(199003)(479174004)(189002)(24454002)(6806004)(47776003)(54356999)(68736005)(62966003)(65956001)(65816999)(86362001)(23746002)(46102003)(59896002)(77096005)(2950100001)(76176999)(87936001)(50986999)(5007970100001)(87266999)(92566002)(77156002)(65806001)(36756003)(2201001)(106466001)(80316001)(97736004)(5001860100001)(50466002)(5004730100002)(189998001)(5001830100001)(33656002)(64706001)(5001770100001)(83506001)(4001350100001)(4001540100001)(64126003)(11100500001)(3940600001);DIR:OUT;SFP:1101;SCL:1;SRVR:AMSPR05MB344;H:mtlcas13.mtl.com;FPR:;SPF:Pass;PTR:ErrorRetry;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;AMSPR05MB344;2:eHLEZrLHNcx01tsKlEr26O1CbljYGpw6jHdaF+0bqXu+hJqZtRdyQuju7DFQlsHdC2kDf/6gxEXfKdblse5e614cK67v5TzsetOf2E/jHwPljqWLc4TQ7p/I8TTgSCP9tECzDyFYOOqkRTOg7rURv3SXd8SBo56gTOq0XGkAwzw=;3:CHw1IKonfjhDW5TC38nrZjReHF9D3cwgRvbPTmLU6pWCNV3lbiQVOUUOMj3kQ6urvpYk07AdO2efNoZpq8Xj467hk470S8TY48efKuGja1/iF2/j8LfqH3hukBYGNjp1aD0AUPqJo7uraZ7ABeC6BkdwZXVN7lai9vrpGUlLTUQwv1IGlSnmEHXrb5MFPnc6ipk51ebsZg4n1JKG0La1dia7qfc8Vow5Grwch9gtf3TOFHmuazYUTLRQxD/xS/Ae9N4A3i81dJi9z4eCxbyvrw==;25:bxGZ1opkg39+QyVVDwQAWoXM1eFLbqVeEQ7yRhLgP4bL0ra7mbL001hLJXCt2sWdmHrHda0cBrPEw6pRxfOm6B9ZBSji7e0bj7VtG1FxRQe5bRArZEeDY4MNOPrNUmJiaVzTo61+uauznBWgK7sgXzAez6lvlS68P9plc2M4YX3g5aWA0TB0RyxyXp5wp9Gi1VVm8r7OUEFwtl4hMgOF40dRAjHcBlIrdc+wxfpuImgjeNRZu8p2XaZfG2Dd+R2x/MQIZE9Mx+XfAbC0GSdgDw== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501001);SRVR:AMSPR05MB344; X-Microsoft-Exchange-Diagnostics: 1;AMSPR05MB344;20:3rsOPv2S5V/+Cc71uhxImOyUD7FxIvnT0he60yhFuEIXewsn2eMr1lGC66+PZDVLDv++8uqwA9vx492HJOeLIR2DGLBCv7Q9n2Q6+EPk/bUKJ6eyaEVMLIkmLfaaW6yZ5YVLD3FBk8vMvkboGCiinNNn2m3lPAUho93q6UueZD1QDF7n1mzPrKYoq4+rYFv7iN2yTEOrST8CjJFgC0lpliaGbP2Hgw1OnjLmnKe/wehD2z13AVFu8RtkhSwocMXaz542yfPyMe9qAejcoL+Pry82DekRgbei0CbnHzphA3TGwj9zFWrB5PEUvS7EZGg2yYHDyAIhLft2BefiHhsiBmZyN/Hjxp5rt22BtediKktZA0qE1m3YM1+hd+RAifm0UnWeLxmmm4gAgt0sMbwOdk1vtYYD+gjkfAp98z761f9r/bNW87VBV2ODorjoTd1aCTYC/j2Mm9WGqsTKhggVdqvAHEI5qoJ7E46LB/yC0KQ/ZqkvpPwxPI1ny1fzp271;4:S64UQ7kLZkwIMzrE4M+L55mrXBmBbJ6UrRwogPnPUI0pDUQjxl8YoYINN2cDLtnlUf26QRfWHtX4WU1G1VT/1sWxsD2OREkRu4lqFK9fM8bOzh6sAtueIfTTq80LvxlL0Ikjz7AtpXC8sIoZUeLFgH+VqyJFsv0qHUgg4GlLt3JsDSwhzKpPf94rJHoaeOrQ+M/wGOYpKX31sAyInuO7YPheTY/yNuh3XtV8obJ45RgS3XG2BQ0M6DBLsVLeW/fmn9Vp+0hco788WPe4nBSrx6dP1AIpiPzfJ9XGHHi3ghL4d5wVt6n2eUzqfHGQTdkC+iZ8jMvpOkgsDkQrjaoCvQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(8121501046)(5005006)(3002001);SRVR:AMSPR05MB344;BCL:0;PCL:0;RULEID:;SRVR:AMSPR05MB344; X-Forefront-PRVS: 069373DFB6 X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;AMSPR05MB344;23:l9dBUGu4G+rCAxHm7WpZXkq/w4jpjYYd7xOjaN?= =?Windows-1252?Q?XW+bABfrFY8DSLLzBH6nCgXDfCbRCf9OH9MUov7bV6alX4EN7QedElCV?= =?Windows-1252?Q?sMuK+8SOhKNjpNgvHNBWsvc69RK5sppLD5ahkpWPJTTxPLKCwdNTDdPT?= =?Windows-1252?Q?PoRlwx/QR3x+1oWQYCE+vLEXCqtCafVDds4vYAYswtMA/pKaQFeOpjsd?= =?Windows-1252?Q?QKU7F2/dPmLCyhxmMfVLFwGMc5TFx61QeWsDDzqhwHbMLJbdm/SyIzQq?= =?Windows-1252?Q?/pz01SHB1BGjnLEuM8mR2Z4xIQa73sALMKMW4fZTZcwejHVFlwTfBhhE?= =?Windows-1252?Q?1V6Mo1f4IcgDfZFOyY+onEIGw/hw9ymdFb7TYZbDaAUovvKHZQ0cE7+m?= =?Windows-1252?Q?+m4XikhZaGQdCkM7oVXyi0UFPLIvZn/NSRFzdlFt0i/qqFN9xXGSRj3t?= =?Windows-1252?Q?kXqZf0cQPnO76CkvBg0ODpUIC7K4V901d3FE9oR+K/bPagR61LB1NLS7?= =?Windows-1252?Q?67oIUJSu7LZUvOzIQWsXwSTNEDAPvBQisDhqaEZC03SF9cOWDGQwP7op?= =?Windows-1252?Q?7YBysr2esHaazFskGXktDC8/nM/ji3VbBeK5uqE4UPQZ9Z8Q9Hr7TLm/?= =?Windows-1252?Q?Mq8Ay4LNjXVuAH8O/1muGOLmXtPXH0uw6BN1onzLruvEvd+1geTF0K9m?= =?Windows-1252?Q?J63X5FTxBNHZytAkc/48imMZoV+gbRq2jTekO94M1tOcBJNm6cEVUu7Y?= =?Windows-1252?Q?MJC3+3Vxe+XxFYNTXit12FDRhWn3rXNpResAR3Ajbdx2yx5smKulPSfP?= =?Windows-1252?Q?Dj72hwuKtIU6phKAKyL87pVml/lPQpogYfbeFOZYx28ENRhSt7wH1BaO?= =?Windows-1252?Q?ryAQat5UgdCV9PM89CcQTMrKABYl83aNyLTFDcsjg4WHQgILF1TQSidI?= =?Windows-1252?Q?wsd25V3KGTb6P4myCjsbGqz8X/cIxpzHVhwtOQxqyQKJNJ57h4be56j1?= =?Windows-1252?Q?UVMf+pzLBretu9IpB5Cd8hfHe7mMZd5Mgw/TglSUDfz+l0xhFgZ4baAx?= =?Windows-1252?Q?gbAmqjCumKIQOTLW7RLxjHMBhuSrzNWA/9taD5j3HlychqmiywPutA+a?= =?Windows-1252?Q?K9DazfgSf6DsxZjNAi8MbfIaLA7XpndAhOhmHxWOzppA9izKdNEmmBgZ?= =?Windows-1252?Q?R3NRh6N+LUHs/OdgAghrnl/aD8OXMgP3A283u/DpjWvpIOMyhvJsaudL?= =?Windows-1252?Q?9b6EN3/pTqET9KvySkmwF/L/8MyVws781T1UOQmQmaJoyXuvOQOhGZC2?= =?Windows-1252?Q?FAoX4UNN74oIUCXqq/Ofmzy7KMsc5PPESBJcRxnQzK0Ac=3D?= X-Microsoft-Exchange-Diagnostics: 1;AMSPR05MB344;5:3rH+D0vD0qZcbon405yEDI+6XM9MpLyKpWeqJo7VrgAn9rsG8aoK03N6YjNpVYZofnBrovEE18sdE657UDj6wg/u5gdxf3HEGIfHiHRN1V8LvRxXl5qLWTBvaQrc3sGhNSLP2S0fkXOtuEizy1NAqQ==;24:JhHu88dizkigNdn7/QlIAfBOdHhGtlXpzSVkipp4u7M4aLqhpZj2HxFE1/07w2v5Bk6T5jJFtVBtQTfhf42mwhjY7haEgCHYHrSrlRDgz/A=;20:zl1QBGdaIr8bUISZC+0Fh21FIHtMqWgv4x0CK7VXGV0Gidm++6ufmaEOutACM5NbT/DlwrKLsK9pm6hcNRMdag== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2015 12:45:56.8464 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b;Ip=[193.47.165.134];Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMSPR05MB344 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/09/2015 23:38, Parav Pandit wrote: > Currently user space applications can easily take away all the rdma > device specific resources such as AH, CQ, QP, MR etc. Due to which other > applications in other cgroup or kernel space ULPs may not even get chance > to allocate any rdma resources. > > This patch-set allows limiting rdma resources to set of processes. > It extend device cgroup controller for limiting rdma device limits. I don't think extending the device cgroup is the right place for these limits. It is currently a very generic controller and adding various RDMA resources to it looks out of place. Why not create a new controller for rdma? Another thing I noticed is that all limits in this cgroup are global, while the resources they control are hardware device specific. I think it would be better if the cgroup controlled the limits of each device separately. > With this patch, user verbs module queries rdma device cgroup controller > to query process's limit to consume such resource. It uncharge resource > counter after resource is being freed. This is another reason why per-device limits would be better. Since limits are reflected to user-space when querying a specific device, it will show the same maximum limit on every device opened. If the user opens 3 devices they might expect to be able to open 3 times the number of the resources they actually can. Regards, Haggai From mboxrd@z Thu Jan 1 00:00:00 1970 From: Haggai Eran Subject: Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Date: Tue, 8 Sep 2015 15:45:50 +0300 Message-ID: <55EED87E.3000408@mellanox.com> References: <1441658303-18081-1-git-send-email-pandit.parav@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1441658303-18081-1-git-send-email-pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Parav Pandit , cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, lizefan-hv44wF8Li93QT0dZR+AlfA@public.gmane.org, hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org, dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org Cc: corbet-T1hC0tSOHrs@public.gmane.org, james.l.morris-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org, serge-A9i7LUbDfNHQT0dZR+AlfA@public.gmane.org, ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, raindel-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org, linux-security-module-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-rdma@vger.kernel.org On 07/09/2015 23:38, Parav Pandit wrote: > Currently user space applications can easily take away all the rdma > device specific resources such as AH, CQ, QP, MR etc. Due to which other > applications in other cgroup or kernel space ULPs may not even get chance > to allocate any rdma resources. > > This patch-set allows limiting rdma resources to set of processes. > It extend device cgroup controller for limiting rdma device limits. I don't think extending the device cgroup is the right place for these limits. It is currently a very generic controller and adding various RDMA resources to it looks out of place. Why not create a new controller for rdma? Another thing I noticed is that all limits in this cgroup are global, while the resources they control are hardware device specific. I think it would be better if the cgroup controlled the limits of each device separately. > With this patch, user verbs module queries rdma device cgroup controller > to query process's limit to consume such resource. It uncharge resource > counter after resource is being freed. This is another reason why per-device limits would be better. Since limits are reflected to user-space when querying a specific device, it will show the same maximum limit on every device opened. If the user opens 3 devices they might expect to be able to open 3 times the number of the resources they actually can. Regards, Haggai -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html