From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1761353AbbBIVcv (ORCPT ); Mon, 9 Feb 2015 16:32:51 -0500 Received: from prod-mail-xrelay02.akamai.com ([72.246.2.14]:33094 "EHLO prod-mail-xrelay02.akamai.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1761333AbbBIVct (ORCPT ); Mon, 9 Feb 2015 16:32:49 -0500 Message-ID: <54D92780.4000303@akamai.com> Date: Mon, 09 Feb 2015 16:32:48 -0500 From: Jason Baron User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Andy Lutomirski , peterz@infradead.org, mingo@redhat.com, viro@zeniv.linux.org.uk CC: akpm@linux-foundation.org, normalperson@yhbt.net, davidel@xmailserver.org, mtk.manpages@gmail.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH 2/2] epoll: introduce EPOLLEXCLUSIVE and EPOLLROUNDROBIN References: <68a0ad4a99551ea3bfff89da461bb490d63b0ca8.1423509605.git.jbaron@akamai.com> <54D915FC.7010003@amacapital.net> In-Reply-To: <54D915FC.7010003@amacapital.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/09/2015 03:18 PM, Andy Lutomirski wrote: > On 02/09/2015 12:06 PM, Jason Baron wrote: >> Epoll file descriptors that are added to a shared wakeup source are always >> added in a non-exclusive manner. That means that when we have multiple epoll >> fds attached to a shared wakeup source they are all woken up. This can >> lead to excessive cpu usage and uneven load distribution. >> >> This patch introduces two new 'events' flags that are intended to be used >> with EPOLL_CTL_ADD operations. EPOLLEXCLUSIVE, adds the epoll fd to the event >> source in an exclusive manner such that the minimum number of threads are >> woken. EPOLLROUNDROBIN, which depends on EPOLLEXCLUSIVE also being set, can >> also be added to the 'events' flag, such that we round robin around the set >> of waiting threads. >> >> An implementation note is that in the epoll wakeup routine, >> 'ep_poll_callback()', if EPOLLROUNDROBIN is set, we return 1, for a successful >> wakeup, only when there are current waiters. The idea is to use this additional >> heuristic in order minimize wakeup latencies. > > I don't understand what this is intended to do. > > If an event has EPOLLONESHOT, then this only one thread should be woken regardless, right? If not, isn't that just a bug that should be fixed? > hmm...so with EPOLLONESHOT you basically get notified once about an event. If i have multiple epoll fds (say 1 per-thread) attached to a single source in EPOLLONESHOT, then all threads will potentially get woken up once per event. Then, I would have to re-arm all of them. So I don't think this addresses this particular usecase...what I am trying to avoid is this mass wakeup or thundering herd for a shared event source. > If an event has EPOLLET, then the considerations are similar to EPOLLONESHOT, right? > EPOLLET is still going to cause this thundering herd. > If an event is a normal level-triggered non-one-shot event, then I don't understand how a round-robin wakeup makes any sense. It's level-triggered, after all. Yeah, so the current behavior is to wake up all of the threads. I'm trying to add a new mode where it load balances among the threads interested in the event. Perhaps, the test program I attached to 0/2 will show the issue better? Also, this originally came up in the context of a single listening socket which was attached to multiple epoll fds each in a separate thread. With the attached patch, I can measure a large decrease in cpu usage and better balancing behavior among the accepting threads. Thanks, -Jason