Unicorn and HAProxy, 500 Internal errors after checks

Unicorn and HAProxy, 500 Internal errors after checks

[Pierre]

Hi,

This morning, while checking for a correct deployment, we found out
that the Unicorns we are using were sending 500 Internal errors very
frequently to the HAProxy that sits in front of them. After some
investigation, It turned out that HAProxy checks the backend by
opening and closing a connection to the unicorn. Unfortunately the
Unicorns we use ( v 0.990.0 ) will try to reply to this probe by a
500, resulting in the error being sent to the HAProxy when the proxy
has already close the connection. I think Unicorn should refrain from
sending an error in this case. There is no visible side effect,
excepted that there are a "lot" of 500s running on the wire that
nobody will ever read. Also, these errors are not sent in the Unicorn
log, making it quite difficult to understand what's going on :-)

Cheers,
--
Pierre <oct@fotopedia.com>
Server Shepherd at http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Clifton King]

Pierre,

Any specific reason you're using HAProxy for Unicorn?

Clifton

On Wed, Dec 1, 2010 at 5:59 AM, Pierre <oct@fotopedia.com> wrote:
> Hi,
>
> This morning, while checking for a correct deployment, we found out
> that the Unicorns we are using were sending 500 Internal errors very
> frequently to the HAProxy that sits in front of them. After some
> investigation, It turned out that HAProxy checks the backend by
> opening and closing a connection to the unicorn. Unfortunately the
> Unicorns we use ( v 0.990.0 ) will try to reply to this probe by a
> 500, resulting in the error being sent to the HAProxy when the proxy
> has already close the connection. I think Unicorn should refrain from
> sending an error in this case. There is no visible side effect,
> excepted that there are a "lot" of 500s running on the wire that
> nobody will ever read. Also, these errors are not sent in the Unicorn
> log, making it quite difficult to understand what's going on :-)
>
> Cheers,
> --
> Pierre <oct@fotopedia.com>
> Server Shepherd at http://www.fotopedia.com/
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Pierre]

Hi,

>
> Any specific reason you're using HAProxy for Unicorn?

Indeed, we have two front load-balancers instances running HAProxy and
several backends running Unicorns. We use HAProxy to amortize part of
the load and load-balance / failover between all the backends. We
especially use the leastconn feature of HAProxy that ensure that the
less powerful backends are queried less often. As we run on EC2
virtual machines, this feature has the please to help us maintaining a
good QoS.

Cheers,
-- 
Pierre <oct@fotopedia.com>
Server Shepherd at http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Eric Wong]

Pierre <oct@fotopedia.com> wrote:
> Hi,
> 
> This morning, while checking for a correct deployment, we found out
> that the Unicorns we are using were sending 500 Internal errors very
> frequently to the HAProxy that sits in front of them. After some
> investigation, It turned out that HAProxy checks the backend by
> opening and closing a connection to the unicorn. Unfortunately the
> Unicorns we use ( v 0.990.0 ) will try to reply to this probe by a
> 500, resulting in the error being sent to the HAProxy when the proxy
> has already close the connection.

Hi Pierre, HAProxy should be configured to send proper HTTP checks and
not just TCP connection checks, the problem will go away then.  Also, I
can not recommend HAProxy unless you're certain all your clients are on
a LAN and can be trusted to never trickle uploads nor reading large
responses.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Pierre]

Dear Eric,

On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Hi Pierre, HAProxy should be configured to send proper HTTP checks and
> not just TCP connection checks, the problem will go away then.

I understood this could be fixed this way and we will probably do that
soon. However, I think this is also the responsibility of Unicorn not
to reply anything when there's no request or at least log the error
somewhere :)

> Also, I
> can not recommend HAProxy unless you're certain all your clients are on
> a LAN and can be trusted to never trickle uploads nor reading large
> responses.

While I understand that uploads are very complicated to handle on the
stack (even nginx can be confused at upload sometimes), HAProxy proved
it was very good at managing tons of connections and high volume
traffic from the Internet. All the more so as it allows a very high
level of redundancy at a very small cost that cannot be achieved
simply otherwise. Do you have any pointers about your worrying
non-recommendation of HAProxy ?

As far as I'm concerned, I've used HAProxy in worldwide web context of
website serving up to 25 Millions pages with no problem, ever.

Thanks again for your support, :)
-- 
Pierre <oct@fotopedia.com>
Server Shepherd at http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Eric Wong]

Pierre <oct@fotopedia.com> wrote:
> On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Hi Pierre, HAProxy should be configured to send proper HTTP checks and
> > not just TCP connection checks, the problem will go away then.
> 
> I understood this could be fixed this way and we will probably do that
> soon. However, I think this is also the responsibility of Unicorn not
> to reply anything when there's no request or at least log the error
> somewhere :)

I'm not sure how Unicorn is actually replying to anything, does HAProxy
write *anything* to the socket?

Logging those bad connections is another DoS vector I'd rather avoid,
and for connections where nothing is written, not even possible...

If you have the TCP_DEFER_ACCEPT (Linux) or accept filters (FreeBSD),
it's highly likely the Unicorn would never see the connection if the
client never wrote to it.

> > Also, I
> > can not recommend HAProxy unless you're certain all your clients are on
> > a LAN and can be trusted to never trickle uploads nor reading large
> > responses.
> 
> While I understand that uploads are very complicated to handle on the
> stack (even nginx can be confused at upload sometimes), HAProxy proved
> it was very good at managing tons of connections and high volume
> traffic from the Internet. All the more so as it allows a very high
> level of redundancy at a very small cost that cannot be achieved
> simply otherwise. Do you have any pointers about your worrying
> non-recommendation of HAProxy ?

HAProxy starts writing request bodies to Unicorn as soon as the upload
starts, which means the Unicorn process will be bounded by the speed of
the original client connection.  If multiple clients upload slowly,
then you'll end up hogging many Unicorn workers.

nginx can also limit the size of client uploads (default 1M) to prevent
unnecessary I/O.

If you serve large responses that can't fit in kernel socket buffers,
then Unicorn will get stuck writing out to a client that isn't reading
fast enough.

AFAIK, HAProxy also does not yet maintain keep-alive connections to
clients, whereas nginx does. Keep-alive is important to client browsers,
they can halve their active connections to a site if keep-alive is
supported.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Lawrence Pit]

> HAProxy starts writing request bodies to Unicorn as soon as the upload
> starts,

Is that also true when the nginx upload module is used? (assuming nginx 
sits between HAProxy and the unicorns, otherwise it doesn't make sense 
anyways). Unicorn is only hit after the complete upload is finished by 
nginx. I would assume this wouldn't be different if HAProxy is put in 
front of all the nginx servers.

> AFAIK, HAProxy also does not yet maintain keep-alive connections to
> clients, whereas nginx does. Keep-alive is important to client browsers,
> they can halve their active connections to a site if keep-alive is
> supported.

keep-alive options are in HAProxy since 1.4

There is further new development on this front if you're using SSL. See 
the first quick news section at http://haproxy.1wt.eu/ :

"The main advantage over the x-forwarded-for patch is that it now 
supports keep-alive"



Lawrence
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Eric Wong]

Lawrence Pit <lawrence.pit@gmail.com> wrote:
> Eric Wong <normalperson@yhbt.net> wrote:
>> HAProxy starts writing request bodies to Unicorn as soon as the upload
>> starts,
>
> Is that also true when the nginx upload module is used? (assuming nginx  
> sits between HAProxy and the unicorns, otherwise it doesn't make sense  
> anyways). Unicorn is only hit after the complete upload is finished by  
> nginx. I would assume this wouldn't be different if HAProxy is put in  
> front of all the nginx servers.

I would setup a test and confirm.  You can do this trivially with the
following Rack app:

  use Rack::ContentLength
  use Rack::ContentType
  run lambda { |env|
    input = env['rack.input']
    buf = input.each do |line|
      # each line is printed as Unicorn receives it,
      # if Unicorn is behind nginx, the times for each
      # line should be roughly/exactly the same.
      puts "#{Time.now} #{line.inspect}"
    end
    [ 200, {}, [] ]
  }

And then telnetting or netcatting to your frontend, and typing the
following (no leading spaces, use a proper hostname for vhosts):

  PUT / HTTP/1.1
  Host: example.com
  Content-Length: 6

...and then hit Enter a few times while watching your Unicorn
terminal/logs.  If you delay between hitting the Enter nginx
should be buffering that.

> keep-alive options are in HAProxy since 1.4
>
> There is further new development on this front if you're using SSL. See  
> the first quick news section at http://haproxy.1wt.eu/ :
>
> "The main advantage over the x-forwarded-for patch is that it now  
> supports keep-alive"

Cool.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Pierre]

Hello,

On Wed, Dec 1, 2010 at 8:58 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Pierre <oct@fotopedia.com> wrote:
>> On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > Hi Pierre, HAProxy should be configured to send proper HTTP checks and
>> > not just TCP connection checks, the problem will go away then.
>>
>> I understood this could be fixed this way and we will probably do that
>> soon. However, I think this is also the responsibility of Unicorn not
>> to reply anything when there's no request or at least log the error
>> somewhere :)
>
> I'm not sure how Unicorn is actually replying to anything, does HAProxy
> write *anything* to the socket?
>

It does. To reproduce the behavior, start a unicorn, start a sniffing
tool (ngrep for example), connect to the unicorn and disconnect
immediately:

Connecting endpoint:

[08:22][virtual] root@infrabox:~# telnet localhost 2002
Trying 127.0.0.1...
Connected to infrabox.virtual.ftnz.net.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Sniffing Console:

[08:22][virtual] root@infrabox:~# ngrep "" port 2002 -d lo
interface: lo (127.0.0.0/255.0.0.0)
filter: (ip or ip6) and ( port 2002 )
#####
T 127.0.0.1:2002 -> 127.0.0.1:42874 [AFP]
  HTTP/1.1 500 Internal Server Error....
#


Further packet inspection reveals that the 500 Internal Server Error
is sent after the telnet has sent a FIN packet which means it will
just be discarded. I understand logging it could be a DoS vector, but
you must admit that seing 500's from the server without any obvious
reason can be frightening at first.

Re HAProxy, and to complete Laurence reply from my point of view,

> If you serve large responses that can't fit in kernel socket buffers,
> then Unicorn will get stuck writing out to a client that isn't reading
> fast enough.

That can definitely be a problem in that case. To work around that we
use a cache in the stack above the lower level of HAProxy (we have two
other levels of HAProxy in the stack).

Cheers,
-- 
Pierre <oct@fotopedia.com>
Server Shepherd at http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Eric Wong]

Pierre <oct@fotopedia.com> wrote:
> On Wed, Dec 1, 2010 at 8:58 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > I'm not sure how Unicorn is actually replying to anything, does HAProxy
> > write *anything* to the socket?
> 
> It does. To reproduce the behavior, start a unicorn, start a sniffing
> tool (ngrep for example), connect to the unicorn and disconnect
> immediately:
> 
> Connecting endpoint:
> 
> [08:22][virtual] root@infrabox:~# telnet localhost 2002
> Trying 127.0.0.1...
> Connected to infrabox.virtual.ftnz.net.
> Escape character is '^]'.
> ^]
> telnet> quit
> Connection closed.

Hi, you started writing an HTTP message ("quit" can be a valid HTTP
method name) and disconnected; don't do that.

> Further packet inspection reveals that the 500 Internal Server Error
> is sent after the telnet has sent a FIN packet which means it will
> just be discarded. I understand logging it could be a DoS vector, but
> you must admit that seing 500's from the server without any obvious
> reason can be frightening at first.

You can enable Ruby debugging (pass '-d' to unicorn) to see exactly
which exception you got.  Really, it shouldn't matter, you should be
sending real HTTP requests for health checks.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Pierre]

Hello,

On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
>>
>> Connecting endpoint:
>>
>> [08:22][virtual] root@infrabox:~# telnet localhost 2002
>> Trying 127.0.0.1...
>> Connected to infrabox.virtual.ftnz.net.
>> Escape character is '^]'.
>> ^]
>> telnet> quit
>> Connection closed.
>
> Hi, you started writing an HTTP message ("quit" can be a valid HTTP
> method name) and disconnected; don't do that.
>

No, quit is typed in the telnet console and not in the network stream.

When passing -d to unicorn, I get the exception, that's cool

Exception `EOFError' at
/usr/local/rvm/gems/ree-1.8.7-2010.02/gems/unicorn-0.990.0/lib/unicorn/http_request.rb:59
- end of file reached

Cheers,
-- 
Pierre <oct@fotopedia.com>
Server Shepherd at http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Eric Wong]

Pierre <oct@fotopedia.com> wrote:
> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Pierre <oct@fotopedia.com> wrote:
> >> Connecting endpoint:
> >>
> >> [08:22][virtual] root@infrabox:~# telnet localhost 2002
> >> Trying 127.0.0.1...
> >> Connected to infrabox.virtual.ftnz.net.
> >> Escape character is '^]'.
> >> ^]
> >> telnet> quit
> >> Connection closed.
> >
> > Hi, you started writing an HTTP message ("quit" can be a valid HTTP
> > method name) and disconnected; don't do that.
> 
> No, quit is typed in the telnet console and not in the network stream.

Are you sure?  With my telnet, "quit" definitely goes out to the server
since you're already connected.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[russell muetzelfeldt]

On 05/12/2010, at 10:38 AM, Eric Wong wrote:
> Pierre <oct@fotopedia.com> wrote:
>> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
>>> Pierre <oct@fotopedia.com> wrote:
>>>> Connecting endpoint:
>>>> 
>>>> [08:22][virtual] root@infrabox:~# telnet localhost 2002
>>>> Trying 127.0.0.1...
>>>> Connected to infrabox.virtual.ftnz.net.
>>>> Escape character is '^]'.
>>>> ^]
>>>> telnet> quit
>>>> Connection closed.
>>> 
>>> Hi, you started writing an HTTP message ("quit" can be a valid HTTP
>>> method name) and disconnected; don't do that.
>> 
>> No, quit is typed in the telnet console and not in the network stream.
> 
> Are you sure?  With my telnet, "quit" definitely goes out to the server
> since you're already connected.

he connects, then sends an escape ("^]") to the telnet process, then types "quit" at the prompt the telnet process shows to tell the local telnet process to close the connection.

the connection *might* see some traffic in attempted telnet option negotiation (unless the server end initiates that - I forget), but the "quit" string isn't transmitted to the remote end.


cheers

Russell

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Eric Wong]

russell muetzelfeldt <russm-rubyforge@slofith.org> wrote:
> On 05/12/2010, at 10:38 AM, Eric Wong wrote:
> > Pierre <oct@fotopedia.com> wrote:
> >> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
> >>> Pierre <oct@fotopedia.com> wrote:
> >>>> Connecting endpoint:
> >>>> 
> >>>> [08:22][virtual] root@infrabox:~# telnet localhost 2002
> >>>> Trying 127.0.0.1...
> >>>> Connected to infrabox.virtual.ftnz.net.
> >>>> Escape character is '^]'.
> >>>> ^]
> >>>> telnet> quit
> >>>> Connection closed.
> >>> 
> >>> Hi, you started writing an HTTP message ("quit" can be a valid HTTP
> >>> method name) and disconnected; don't do that.
> >> 
> >> No, quit is typed in the telnet console and not in the network stream.
> > 
> > Are you sure?  With my telnet, "quit" definitely goes out to the server
> > since you're already connected.

> he connects, then sends an escape ("^]") to the telnet process, then
> types "quit" at the prompt the telnet process shows to tell the local
> telnet process to close the connection.

Ah, I missed the "^]" :x

> the connection *might* see some traffic in attempted telnet option
> negotiation (unless the server end initiates that - I forget), but the
> "quit" string isn't transmitted to the remote end.

Due to TCP_DEFER_ACCEPT, older Linux kernels won't even wake up Unicorn.
Newer ones treat the TCP_DEFER_ARGUMENT as a timeout but it's just a
quick EOFError.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn and HAProxy, 500 Internal errors after checks

[Eric Wong]

Pierre <oct@fotopedia.com> wrote:
> On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Also, I
> > can not recommend HAProxy unless you're certain all your clients are on
> > a LAN and can be trusted to never trickle uploads nor reading large
> > responses.
> 
> While I understand that uploads are very complicated to handle on the
> stack (even nginx can be confused at upload sometimes), HAProxy proved
> it was very good at managing tons of connections and high volume
> traffic from the Internet. All the more so as it allows a very high
> level of redundancy at a very small cost that cannot be achieved
> simply otherwise. Do you have any pointers about your worrying
> non-recommendation of HAProxy ?

On a related note, LWN has a new article[1] explaining the large body
POST problem (more than a year since Slowloris, and ~3 years after
the largely-unknown david[2]).

[1] http://lwn.net/Articles/418017/ (subscribers)
    http://lwn.net/SubscriberLink/418017/1e63e408f34b5d68/ (non-subscribers)
[2] http://git.bogomips.org/cgit/david.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying