Number of simultaneous clients

Number of simultaneous clients

[Pierre Baillet]

Hello,

We've working on some improvement on our caching system and while
testing it, I encountered errors between Varnish and Unicorn. Namely,
Varnish claims that there is no backend connection and will not send
any data from Unicorn.

After some investigations (tcpdump), I found out that sometimes, I got
no reply from my backend server. See for example this complete
conversation between my Varnish server and my Unicorn server:

GET /items/422qq2nh931gk-aqXuxXfs/esi/card/sitemap/nil.xml/http HTTP/1.1
Accept: */*
Host: www.testing.ftnz.net
X-Forwarded-For: 127.0.0.1
Accept-Encoding: identity
X-Ftn-Is-Logged: no
X-Varnish: 1063393845


Yes, that's all. The last packet sent by the backend is an ACK to the
data packet sent by Varnish.

This happens when I really overload the Varnish with incoming
requests. I suspect it opens a lot of connection to our backend. But I
fail to see why Unicorn would not reply (even at a slow rate). Of
course, the issue can be reproduced more or less easily. Trying to
reproduce the issue by directly hitting the Unicorn has not succeeded.

Any thoughts ? Cheers,
-- 
Pierre Baillet <oct@fotopedia.com>
http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Number of simultaneous clients

[Eric Wong]

Pierre Baillet <oct@fotopedia.com> wrote:
> Hello,
> 
> We've working on some improvement on our caching system and while
> testing it, I encountered errors between Varnish and Unicorn. Namely,
> Varnish claims that there is no backend connection and will not send
> any data from Unicorn.
> 
> After some investigations (tcpdump), I found out that sometimes, I got
> no reply from my backend server. See for example this complete
> conversation between my Varnish server and my Unicorn server:
> 
> GET /items/422qq2nh931gk-aqXuxXfs/esi/card/sitemap/nil.xml/http HTTP/1.1
> Accept: */*
> Host: www.testing.ftnz.net
> X-Forwarded-For: 127.0.0.1
> Accept-Encoding: identity
> X-Ftn-Is-Logged: no
> X-Varnish: 1063393845
> 
> 
> Yes, that's all. The last packet sent by the backend is an ACK to the
> data packet sent by Varnish.

Hi Pierre,

Do you have any timeouts configured Varnish that would cause it to not
wait if Unicorn is overloaded?

Can you strace/truss the Unicorn process and see if it's actually
accepting that particular request and doing something with it?

Also, enable access logs for Unicorn (Rack::CommonLogger or Clogger[1])
and see if the requests are actually going through it.

To make it easier to track Varnish requests, you can easily customize
the Clogger format to include the X-Varnish header by adding
$http_x_varnish (or any other header) to the :format parameter:

  use Clogger,
        :format => Clogger::Format::Combined + ' $http_x_varnish',
        :logger => ::File.open("/path/to/log", "ab")
  run MyApp.new

[1] - http://clogger.rubyforge.org/

> This happens when I really overload the Varnish with incoming
> requests. I suspect it opens a lot of connection to our backend. But I
> fail to see why Unicorn would not reply (even at a slow rate). Of
> course, the issue can be reproduced more or less easily. Trying to
> reproduce the issue by directly hitting the Unicorn has not succeeded.
> 
> Any thoughts ? Cheers,

What's your listen :backlog parameter set to?

If the backlog fills up (the actual value inside the kernel is usually
higher than what we set it to), then the kernel will start rejecting
connections.  You got an ACK for one particular request, but perhaps
others are not getting it...

We default the listen backlog to 1024 which is pretty high, but for
benchmarking some apps you can increase that.  In Linux, you may need
to increase the net.core.somaxconn sysctl, too.

Keep in mind that if you through enough load at *anything*, it'll get
overloaded and stop working well.

For nginx users, I recommend using fail_timeout=0 so nginx will always
try requests anyways (it's cheap) and never mark it as down for any
period.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying