From: Stan Hu <stanhu@gmail.com> To: Eric Wong <e@yhbt.net> Cc: unicorn-public@yhbt.net Subject: Re: Sustained queuing on one listener can block requests from other listeners Date: Wed, 15 Apr 2020 22:46:58 -0700 [thread overview] Message-ID: <CAMBWrQmzXYjDobExF9GL1BkHAc=ibvMv1ciaSv_fLQ36MyZW0g@mail.gmail.com> (raw) In-Reply-To: <20200415052623.GA24409@dcvr> Thanks, Eric. That patch didn't work; it spun the CPU. I think this worked? diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index a52931a..aaa4955 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -708,7 +708,7 @@ def worker_loop(worker) # we're probably reasonably busy, so avoid calling select() # and do a speculative non-blocking accept() on ready listeners # before we sleep again in select(). - unless nr == 0 + if nr == readers.size tmp = ready.dup redo end On Tue, Apr 14, 2020 at 10:26 PM Eric Wong <e@yhbt.net> wrote: > > Stan Hu <stanhu@gmail.com> wrote: > > My unicorn.rb has two listeners: > > > > listen "127.0.0.1:8080", :tcp_nopush => false > > listen "/var/run/unicorn.socket", :backlog => 1024 > > Fwiw, lowering :backlog may make sense if you got other > hosts/instances. More below.. > > > We found that because of the greedy attempt to accept new connections > > before calling select() in > > https://github.com/defunkt/unicorn/blob/981f561a726bb4307d01e4a09a308edba8d69fe3/lib/unicorn/http_server.rb#L707-L714, > > listeners on another socket stall out until the first listener is > > drained. We would expect Unicorn to round-robin between the two > > listeners, but that doesn't happen as long as there is work to be done > > for the first listener. We've verified that deleting that `redo` block > > fixes the problem. > > > > What do you think about the various options? > > > > 1. Only running that redo block if there is one listener > > That seems reasonable, or if ready.size == nr_listeners > (proposed patch below) > > > 2. Removing the redo block entirely > > From what I recall ages ago, select() entry cost is pretty high > and I remember that redo helping a fair bit even in 2009 with > simple apps. Syscall cost is even higher now with CPU > vulnerability mitigations, and Ruby 1.9+ GVL release+reacquire > is also a penalty I didn't have when developing this on 1.8. > > Do you have time+hardware to benchmark either approach on a > simple app? I no longer have stable/reliable hardware for > benchmarking. Thanks. > > Totally untested patch to try approach #1 > > diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb > index a52931a..69f1f60 100644 > --- a/lib/unicorn/http_server.rb > +++ b/lib/unicorn/http_server.rb > @@ -686,6 +686,7 @@ def worker_loop(worker) > trap(:USR1) { nr = -65536 } > > ready = readers.dup > + nr_listeners = readers.size > @after_worker_ready.call(self, worker) > > begin > @@ -698,7 +699,6 @@ def worker_loop(worker) > # but that will return false > if client = sock.kgio_tryaccept > process_client(client) > - nr += 1 > worker.tick = time_now.to_i > end > break if nr < 0 > @@ -708,7 +708,7 @@ def worker_loop(worker) > # we're probably reasonably busy, so avoid calling select() > # and do a speculative non-blocking accept() on ready listeners > # before we sleep again in select(). > - unless nr == 0 > + if ready.size == nr_listeners > tmp = ready.dup > redo > end > > > > And `nr' can probably just be a boolean `reopen' flag if we're > not overloading it as a counter.
next prev parent reply other threads:[~2020-04-16 5:47 UTC|newest] Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-04-15 5:06 Stan Hu 2020-04-15 5:26 ` Eric Wong 2020-04-16 5:46 ` Stan Hu [this message] 2020-04-16 6:59 ` Eric Wong 2020-04-16 7:24 ` Stan Hu 2020-04-16 9:24 ` [PATCH] prevent single listener from monopolizing a worker Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://yhbt.net/unicorn/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CAMBWrQmzXYjDobExF9GL1BkHAc=ibvMv1ciaSv_fLQ36MyZW0g@mail.gmail.com' \ --to=stanhu@gmail.com \ --cc=e@yhbt.net \ --cc=unicorn-public@yhbt.net \ --subject='Re: Sustained queuing on one listener can block requests from other listeners' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: ../../unicorn.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).