Re: Unicorn hangs on POST request

[Eric Wong <normalperson@yhbt.net>]

Tom Pesman <tom@tnux.net> wrote:
> > The request body doesn't seem to be there, presumably because Heroku
> > isn't sending it.
> >
> > Doe heroku fully buffer the request body before sending it to unicorn?
> > nginx fully buffers, and this is why I can only recommend nginx for slow
> > clients.
> >
> > The proxy -> unicorn transfer speed should _never_ be dependent by the
> > client -> proxy transfer speed.
> >
> > 1) client        ------------------ (slow) --------------> proxy
> > 2) proxy (nginx) --- (as fast as the connection goes) ---> unicorn
> >
> > With nginx, 1) and 2) are decoupled and serialized.  This adds latency,
> > but is the only way for multiprocess servers like unicorn to efficiently
> > handle slow clients.
> 
> I've some new information. Heroku buffers the headers of a HTTP
> request but it doesn't buffer the body of POST requests. Because of
> that I switched to Rainbows! and the responsiveness of the application
> increased dramatically.

Wow, the Heroku proxy is completely unsuitable for unicorn because of
the lack of body buffering.  Heroku really needs to do body buffering to
support unicorn properly (perhaps use nginx...).

It's a shame they charge people to deploy unicorn improperly like this.

> Right now I'm using this configuration:
> 
> worker_processes 4
> timeout 15
> preload_app true
> 
> Rainbows! do
>   use :EventMachine
>   worker_connections 50
>   client_max_body_size 5*1024*1024 # 5 megabytes
>   client_header_buffer_size 2 * 1024 # 2 kilobytes
>   timeout 10
> end
> 
> With high load the performance drops, is EventMachine the right choice
> for this situation (Rails application with slow POST requests and with
> up to 50% POST requests)? Will increasing the worker_connections help?

Probably not, at least it won't improve _consistency_ of performance
without changing your app.

The problem with buffering in Rainbows!+EM is the buffering happens after
the work is distributed to different worker processes (Rainbows!+EM is
still single-threaded).

> I'm planning to make a blog post about this and tell Heroku not to
> advise Unicorn for Rails applications but to use Rainbows and suggest
> a correct/optimised configuration.

The easiest solution is to use nginx with unicorn, I still believe this
is the best configuration for most web apps.

I don't believe there's a universal Rainbows! configuration which is
good for most apps.  With Rainbows!+EM, the buffering happens in a late
stage, after workload is distributed, so you end up with head-of-queue
blocking inside each process.

All of the modules _without_ "rack.input streaming" will fully buffer
responses (similar to how Rainbows!+EM) does it:
http://rainbows.rubyforge.org/Summary.html

However, CoolioThread{Pool/Spawn} should help you with the work load
distribution problem if your app is thread-safe, but these both do body
buffering.

However, the multithreaded options _with_ "rack.input streaming" still
give pretty good performance and let you get around the workload
distribution problem of the single-threaded + internal buffering
options.  You'll reduce disk seek overhead with input streaming.

For reference, nginx+unicorn is single-threaded, but with external
buffering (nginx does the buffering, not unicorn).
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying