raindrops RubyGem user+dev discussion/patches/pulls/bugs/help
 help / color / Atom feed
From: Ryan King <ryan@twitter.com>
To: raindrops@librelist.com
Subject: Re: queued is always 0
Date: Wed, 30 Jun 2010 11:51:15 -0700
Message-ID: <AANLkTikC4C6z3RoHNAYBcsa5iTx0-afh33imJJYM4cu8@mail.gmail.com> (raw)
In-Reply-To: <20100630024549.GA9169@dcvr.yhbt.net>

On Tue, Jun 29, 2010 at 7:45 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Ryan King <ryan@twitter.com> wrote:
>> On Mon, Jun 28, 2010 at 5:50 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > Ryan King <ryan@twitter.com> wrote:
>> >> I'm trying to see if raindrops will help us instrument our stack, so
>> >> I'm doing some tests.
>> >>
>> >> I have 16 unicorns running our app with rainbows and am hitting it
>> >> with ab with a concurrency of 100. However, according to
>> >> linux-tcp-listener-stats.rb I have 0 queued requests. Maybe I'm
>> >> missing something, but shouldn't there be queued requests somewhere
>> >> here?
>> >
>> > Hi Ryan,
>> >
>> > The Unicorns could be accept()-ing connections fast enough and the
>> > queued connections aren't noticeable[1].
>
> <snip>
>
>> I did as you describe here and I got the expected results.
>>
>> I think I'm just confused at this point. We have 16 unicorns running,
>> with 100 clients, but no queued connections, which makes me assume
>> that we have accepted connections that we can't actually service yet.
>> Is this true?
>
> Not true with Unicorn.  Unicorn won't accept connections it can't
> service, ever.
>
> It could be ab itself isn't able to keep up with the requests/responses.
>
> How fast are your response times?

Pretty slow, actually. Running with concurrency of 16 (with 16 workers):


  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   1.0      0      12
  Processing:  1698 2444 351.8   2403    5885
  Waiting:     1698 2444 351.9   2403    5885
  Total:       1698 2444 352.0   2403    5886

  Percentage of the requests served within a certain time (ms)
    50%   2403
    66%   2533
    75%   2616
    80%   2670
    90%   2900
    95%   3085
    98%   3285
    99%   3478
   100%   5886 (longest request)


> What happens when you run fewer Unicorn workers or increase concurrency
> with ab?

I bumped the concurrency up to 2000 (with 16 workers) and still no
queuing (according to raindrops).

Interestingly, when I ran this benchmark ab died because of timeouts.
I lowered the concurrency back to 16 and ran it again, which quickly
caused timeouts. When I looked at the unicorn processes it was obvious
that they were still processing a backlog processes from the first ab
run. I'm not sure if this is how it is supposed to work, but it was
surprising to me.

> How large are your responses?

About 43 KB.

> If you have small responses, Unicorn could've also finished writing the
> response to the socket buffers and accepted another connection by the
> time ab gets around to reading the socket.
>
> Under Linux, Unicorn (and Mongrel) also uses TCP_DEFER_ACCEPT with a
> value of 1 (second)[1], so it won't even register in the queue if
> ab opened the socket and didn't have a chance to write to it, yet...
>
> You could try commenting out the setsockopt call to set
> TCP_DEFER_ACCEPT in lib/unicorn/socket_helper.rb for testing
> and see if the queue jumps up, too..

We're not setting the tcp_nodelay option, so that code shouldn't be
getting run, right?

> I noticed that the usage string for linux-tcp-listener-stats.rb was
> wrong yesterday (and updated it in rainbows.git), but you can try
> running it in a tight delay loop with "-d <seconds>":
>
>  ruby linux-tcp-listener-stats.rb -d 0.01 | awk '$3 != 0 { print $0}'
>
> The above will only print lines if there are queued connections.  You
> can replace "-d 0.01" with a smaller number, but Raindrops (and the
> underlying tcpdiag kernel module) can only give a snapshot of the
> current queue size).

I had done basically the same thing in previous test runs. Never any
queued requests.

I'm really hoping that I just misunderstand something here, but it
seems that our unicorn processes are accepting connections long before
they can service them.

-ryan

>
>
> [1] - I should actually make the defer timeout configurable, especially
>      for Rainbows!  nginx sets this to 60s and it seems to work fine.
>
> --
> Eric Wong
>

  reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-06-29  0:26 Ryan King
2010-06-29  0:50 ` Eric Wong
2010-06-30  1:12   ` Ryan King
2010-06-30  2:45     ` Eric Wong
2010-06-30 18:51       ` Ryan King [this message]
2010-06-30 23:18         ` Ryan King
2010-07-01 10:18         ` Eric Wong
2010-07-01 18:37           ` Ryan King
2010-07-02  2:51             ` Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/raindrops/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AANLkTikC4C6z3RoHNAYBcsa5iTx0-afh33imJJYM4cu8@mail.gmail.com \
    --to=ryan@twitter.com \
    --cc=raindrops@librelist.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

raindrops RubyGem user+dev discussion/patches/pulls/bugs/help

Archives are clonable:
	git clone --mirror https://yhbt.net/raindrops-public
	git clone --mirror http://ou63pmih66umazou.onion/raindrops-public

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.lang.ruby.raindrops
	nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.raindrops

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git