raindrops RubyGem user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Eric Wong <normalperson@yhbt.net>
To: raindrops@librelist.com
Subject: Re: queued is always 0
Date: Thu, 1 Jul 2010 10:18:09 +0000	[thread overview]
Message-ID: <20100701101809.GA23938@dcvr.yhbt.net> (raw)
In-Reply-To: AANLkTikC4C6z3RoHNAYBcsa5iTx0-afh33imJJYM4cu8@mail.gmail.com

Ryan King <ryan@twitter.com> wrote:

Sorry for the late response, I was afk the entire day.

> On Tue, Jun 29, 2010 at 7:45 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Ryan King <ryan@twitter.com> wrote:
> >> On Mon, Jun 28, 2010 at 5:50 PM, Eric Wong <normalperson@yhbt.net> wrote:
<snip>

> > What happens when you run fewer Unicorn workers or increase concurrency
> > with ab?
> 
> I bumped the concurrency up to 2000 (with 16 workers) and still no
> queuing (according to raindrops).
> 
> Interestingly, when I ran this benchmark ab died because of timeouts.
> I lowered the concurrency back to 16 and ran it again, which quickly
> caused timeouts. When I looked at the unicorn processes it was obvious
> that they were still processing a backlog processes from the first ab
> run. I'm not sure if this is how it is supposed to work, but it was
> surprising to me.

They were buffered in the kernel, and Unicorn doesn't have a good way to
know if a client disconnected until Unicorn tries writing the response
(often not even the first packets).

> > How large are your responses?
> 
> About 43 KB.

That should be small enough to fit inside the default socket buffers
without ever blocking.

> > If you have small responses, Unicorn could've also finished writing the
> > response to the socket buffers and accepted another connection by the
> > time ab gets around to reading the socket.
> >
> > Under Linux, Unicorn (and Mongrel) also uses TCP_DEFER_ACCEPT with a
> > value of 1 (second)[1], so it won't even register in the queue if
> > ab opened the socket and didn't have a chance to write to it, yet...
> >
> > You could try commenting out the setsockopt call to set
> > TCP_DEFER_ACCEPT in lib/unicorn/socket_helper.rb for testing
> > and see if the queue jumps up, too..
> 
> We're not setting the tcp_nodelay option, so that code shouldn't be
> getting run, right?

It does run and we always set TCP_DEFER_ACCEPT (since the Mongrel days).
It is safe to disable for Unicorn if you're behind a reverse proxy,
but it shouldn't be a noticeable performance hit either way.

I'll definitely make this configurable in the future (since higher
values are desirable against some attacks), but there's no reason to
disable it for anything other than benchmarking/testing you're doing
now..

> > I noticed that the usage string for linux-tcp-listener-stats.rb was
> > wrong yesterday (and updated it in rainbows.git), but you can try
> > running it in a tight delay loop with "-d <seconds>":
> >
> >  ruby linux-tcp-listener-stats.rb -d 0.01 | awk '$3 != 0 { print $0}'
> >
> > The above will only print lines if there are queued connections.  You
> > can replace "-d 0.01" with a smaller number, but Raindrops (and the
> > underlying tcpdiag kernel module) can only give a snapshot of the
> > current queue size).
> 
> I had done basically the same thing in previous test runs. Never any
> queued requests.
> 
> I'm really hoping that I just misunderstand something here, but it
> seems that our unicorn processes are accepting connections long before
> they can service them.

Now that I think about it, it could be possible you're making a small
mistake in your linux-tcp-listener-stats invocation.

Unicorn listens to all interfaces on a specified port by default (e.g.
0.0.0.0:8080)  So you would need to point linux-tcp-listener-stats.rb
to "0.0.0.0:8080" and not "127.0.0.1:8080" (which may provide you
with a list of _client_ sockets, not server ones).


It looks like raindrops may double count localhost connections
because the client and server address are on the same host, and
I seem to recall tcpdiag not providing a way to filter those...
(It is very late at night for me, so I'll look at this again
sometime tomorrow.)

-- 
Eric Wong

  parent reply	other threads:[~2010-07-01 10:18 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-06-29  0:26 queued is always 0 Ryan King
2010-06-29  0:50 ` Eric Wong
2010-06-30  1:12   ` Ryan King
2010-06-30  2:45     ` Eric Wong
2010-06-30 18:51       ` Ryan King
2010-06-30 23:18         ` Ryan King
2010-07-01 10:18         ` Eric Wong [this message]
2010-07-01 18:37           ` Ryan King
2010-07-02  2:51             ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/raindrops/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20100701101809.GA23938@dcvr.yhbt.net \
    --to=normalperson@yhbt.net \
    --cc=raindrops@librelist.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/raindrops.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).