Our Unicorn Setup

Our Unicorn Setup

[Chris Wanstrath]

Hi list,

I've just published a post detailing our Unicorn setup, migration
process, and reasons for choosing it:
http://github.com/blog/517-unicorn

Thanks again!

-- 
Chris Wanstrath
http://github.com/defunkt

Re: Our Unicorn Setup

[Eric Wong]

Chris Wanstrath <chris@ozmm.org> wrote:
> Hi list,
> 
> I've just published a post detailing our Unicorn setup, migration
> process, and reasons for choosing it:
> http://github.com/blog/517-unicorn

Cool!

"The Unicorn master manages the workers and balancing"

Actually, the master never manages balancing, just the workers.  The
diagram is a little inaccurate as it looks like the master sees the
requests, it never does.

The request flow is like this:

           requests
              |
              |
              |
        shared socket(s)
             /|\
            / | \
           |  |  |
           |  |  |
         worker pool

While the shared socket is opened and configured by the master, but the
master does nothing else with the sockets.  You're completely right
about the pull balancing, it's one of the most distinctive differences
between Unicorn and other setups.



Also, for the 502s, do you get more 502s after the initial worker times
out?  I think nginx may (still)[1] mark a backend as completely dead/down
when a backend dies.  That may cause nginx to stop forwarding to that
backend entirely and throw more 502s for a few seconds until nginx
decides to actually try that backend again.

Setting fail_timeout=0 causes nginx to never mark backends as down and
always try to send a request to them:

  upstream github {
    server unix:/data/github/current/tmp/sockets/unicorn.sock fail_timeout=0;
  }

I'll add this bit somewhere in the Unicorn docs and release 0.93.3 with
the OpenBSD fix for Jeremy in a bit.

> Thanks again!

No problem :)


[1] - I'm not 100% sure if nginx still does this, but I don't see
      anything in the 0.6.x changelog that indicates otherwise.
      I don't see a huge amount of harm in doing this always, we've been
      using fail_timeout=0 for ~2 years now regardless of the backend.

-- 
Eric Wong

Re: Our Unicorn Setup

[Chris Wanstrath]

On Fri, Oct 9, 2009 at 1:30 PM, Eric Wong <normalperson@yhbt.net> wrote:

> "The Unicorn master manages the workers and balancing"
>
> Actually, the master never manages balancing, just the workers.  The
> diagram is a little inaccurate as it looks like the master sees the
> requests, it never does.
>
> The request flow is like this:
>
>           requests
>              |
>              |
>              |
>        shared socket(s)
>             /|\
>            / | \
>           |  |  |
>           |  |  |
>         worker pool
>
> While the shared socket is opened and configured by the master, but the
> master does nothing else with the sockets.  You're completely right
> about the pull balancing, it's one of the most distinctive differences
> between Unicorn and other setups.

Thanks! I've updated the diagram and some of the language to be accurate.

> Also, for the 502s, do you get more 502s after the initial worker times
> out?  I think nginx may (still)[1] mark a backend as completely dead/down
> when a backend dies.  That may cause nginx to stop forwarding to that
> backend entirely and throw more 502s for a few seconds until nginx
> decides to actually try that backend again.
>
> Setting fail_timeout=0 causes nginx to never mark backends as down and
> always try to send a request to them:
>
>  upstream github {
>    server unix:/data/github/current/tmp/sockets/unicorn.sock fail_timeout=0;
>  }

We'll try this out, thanks for letting us know about it.

Cheers,

-- 
Chris Wanstrath
http://github.com/defunkt

Re: Our Unicorn Setup

[Dusty Doris]

Thanks for this post Chris, it was very informative and has answered a
few questions that I've had in my head over the last couple of days.
I've been testing unicorn with a few apps for a couple days and
actually already moved one over to it.

I have a question for list.

We are currently setup with a load balancer that runs nginx and
haproxy.  Nginx, simply proxies to haproxy, which then balances that
across multiple mongrel or thin instances that span several servers.
We simply include the public directory on our load balancer so nginx
can serve static files right there.  We don't have nginx running on
the app servers, they are just mongrel or thin.

So, my question.  How would you do a Unicorn deployment when you have
multiple app servers?

1.  Simply use mongrels upstream and let it round-robin between all
the unicorn instances on the different servers?  Or, perhaps use the
fair-upstream plugin?

nginx -> [unicorns]

2.  Keep haproxy in the middle?

nginx -> haproxy -> [unicorns]

3.  Stick haproxy in front and have it balance between the app servers
that run their own nginx?

haproxy -> [nginxs] -> unicorn # could use socket instead of tcp in this case

I would love to hear any opinions.

Thanks!

Dusty Doris

On Fri, Oct 9, 2009 at 3:42 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> Hi list,
>
> I've just published a post detailing our Unicorn setup, migration
> process, and reasons for choosing it:
> http://github.com/blog/517-unicorn
>
> Thanks again!
>
> --
> Chris Wanstrath
> http://github.com/defunkt
> _______________________________________________
> mongrel-unicorn mailing list
> mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
>

Re: Our Unicorn Setup

[Eric Wong]

Dusty Doris <unicorn@dusty.name> wrote:
> Thanks for this post Chris, it was very informative and has answered a
> few questions that I've had in my head over the last couple of days.
> I've been testing unicorn with a few apps for a couple days and
> actually already moved one over to it.
> 
> I have a question for list.

First off, please don't top post, thanks :)

> We are currently setup with a load balancer that runs nginx and
> haproxy.  Nginx, simply proxies to haproxy, which then balances that
> across multiple mongrel or thin instances that span several servers.
> We simply include the public directory on our load balancer so nginx
> can serve static files right there.  We don't have nginx running on
> the app servers, they are just mongrel or thin.
> 
> So, my question.  How would you do a Unicorn deployment when you have
> multiple app servers?

For me, it depends on the amount of static files you serve with nginx
and also the traffic you hit.

Can I assume you're running Linux 2.6 (with epoll + awesome VFS layer)?

May I also assume your load balancer box is not very stressed right now?

> 1.  Simply use mongrels upstream and let it round-robin between all
> the unicorn instances on the different servers?  Or, perhaps use the
> fair-upstream plugin?
> 
> nginx -> [unicorns]

Based on your description of your current setup, this would be the best
way to go.  I would configure a lowish listen() :backlog for the
Unicorns, fail_timeout=0 in nginx for every server  This setup means
round-robin by default, but if one machine gets a :backlog overflow,
then nginx will automatically retry on a different backend.

> 2.  Keep haproxy in the middle?
> 
> nginx -> haproxy -> [unicorns]

This is probably not necessary, but it can't hurt a whole lot either.

Also an option for balancing.  If you're uncomfortable with the first
approach you can also configure haproxy as a backup server:

  upstream unicorn_failover {
    # round-robin between unicorn app servers on the LAN:
    server 192.168.0.1:8080 fail_timeout=0;
    server 192.168.0.2:8080 fail_timeout=0;
    server 192.168.0.3:8080 fail_timeout=0;

    # haproxy, configured the same way as you do now
    # the "backup" parameter means nginx won't hit haproxy unless
    # all the direct unicorn connections have backlog overflows
    # or other issues
    server 127.0.0.1:8080 fail_timeout=0 backup; # haproxy backup
  }

So your traffic flow may look like the first for the common case, but
you may have a slightly more balanced/queueing solution in case you're
completely overloaded.

> 3.  Stick haproxy in front and have it balance between the app servers
> that run their own nginx?
> 
> haproxy -> [nginxs] -> unicorn # could use socket instead of tcp in this case

This is probably only necessary if:

  1) you have a lot of static files that don't all fit in the VFS caches

  2) you handle a lot of large uploads/responses and nginx buffering will
     thrash one box

I know some sites that run this (or similar) config, but it's mainly
because this is what they've had for 5-10 years and don't have
time/resources to test new setups.

> I would love to hear any opinions.

You can also try the following, which is similar to what I describe in:

  http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31

Pretty much all the above setups are valid.  The important part is that
nginx must sit *somewhere* in between Unicorn and the rest of the world.

-- 
Eric Wong

Re: Our Unicorn Setup

[Dusty Doris]

On Fri, Oct 9, 2009 at 6:01 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Dusty Doris <unicorn@dusty.name> wrote:
>> Thanks for this post Chris, it was very informative and has answered a
>> few questions that I've had in my head over the last couple of days.
>> I've been testing unicorn with a few apps for a couple days and
>> actually already moved one over to it.
>>
>> I have a question for list.
>
> First off, please don't top post, thanks :)

Sorry about that.

>
>> We are currently setup with a load balancer that runs nginx and
>> haproxy.  Nginx, simply proxies to haproxy, which then balances that
>> across multiple mongrel or thin instances that span several servers.
>> We simply include the public directory on our load balancer so nginx
>> can serve static files right there.  We don't have nginx running on
>> the app servers, they are just mongrel or thin.
>>
>> So, my question.  How would you do a Unicorn deployment when you have
>> multiple app servers?
>
> For me, it depends on the amount of static files you serve with nginx
> and also the traffic you hit.
>
> Can I assume you're running Linux 2.6 (with epoll + awesome VFS layer)?
>
> May I also assume your load balancer box is not very stressed right now?
>

Yep.  We server our css, javascript, and some images out of the load
balancer.  But for the majority of our images and all the dynamically
created ones, we serve those from dedicated image servers that have
their own nginx instance running on each one.

>> 1.  Simply use mongrels upstream and let it round-robin between all
>> the unicorn instances on the different servers?  Or, perhaps use the
>> fair-upstream plugin?
>>
>> nginx -> [unicorns]
>
> Based on your description of your current setup, this would be the best
> way to go.  I would configure a lowish listen() :backlog for the
> Unicorns, fail_timeout=0 in nginx for every server  This setup means
> round-robin by default, but if one machine gets a :backlog overflow,
> then nginx will automatically retry on a different backend.
>

Thanks for the recommendation.  I was going to give that a shot first
to see how it went, as it would also be the easiest to manage.

When you say a lowish backlog?  What kind of numbers are you talking
about?  Say, we had 8 workers running that stayed pretty active.  They
are usually quick to respond, with an occasional 2 second response
(say 1/100) due to a bad sql query that we need to fix.  Would lowish
be 16, 32, 64, 128, 1024?

Oh and thanks for the tip on the fail_timeout.

>> 2.  Keep haproxy in the middle?
>>
>> nginx -> haproxy -> [unicorns]
>
> This is probably not necessary, but it can't hurt a whole lot either.
>
> Also an option for balancing.  If you're uncomfortable with the first
> approach you can also configure haproxy as a backup server:
>
>  upstream unicorn_failover {
>    # round-robin between unicorn app servers on the LAN:
>    server 192.168.0.1:8080 fail_timeout=0;
>    server 192.168.0.2:8080 fail_timeout=0;
>    server 192.168.0.3:8080 fail_timeout=0;
>
>    # haproxy, configured the same way as you do now
>    # the "backup" parameter means nginx won't hit haproxy unless
>    # all the direct unicorn connections have backlog overflows
>    # or other issues
>    server 127.0.0.1:8080 fail_timeout=0 backup; # haproxy backup
>  }
>
> So your traffic flow may look like the first for the common case, but
> you may have a slightly more balanced/queueing solution in case you're
> completely overloaded.
>
>> 3.  Stick haproxy in front and have it balance between the app servers
>> that run their own nginx?
>>
>> haproxy -> [nginxs] -> unicorn # could use socket instead of tcp in this case
>
> This is probably only necessary if:
>
>  1) you have a lot of static files that don't all fit in the VFS caches
>
>  2) you handle a lot of large uploads/responses and nginx buffering will
>     thrash one box
>
> I know some sites that run this (or similar) config, but it's mainly
> because this is what they've had for 5-10 years and don't have
> time/resources to test new setups.
>
>> I would love to hear any opinions.
>
> You can also try the following, which is similar to what I describe in:
>
>  http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31
>

Thats an interesting idea, thanks for sharing it.  I like how the
individual server also acts as a load balancer, but only if its having
trouble itself.  Otherwise, it just handles the requests through the
socket connection.

> Pretty much all the above setups are valid.  The important part is that
> nginx must sit *somewhere* in between Unicorn and the rest of the world.
>
> --
> Eric Wong
>

I appreciate your reply and especially for Unicorn.

Thanks!

Re: Our Unicorn Setup

[Eric Wong]

Dusty Doris <unicorn@dusty.name> wrote:
> On Fri, Oct 9, 2009 at 6:01 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Dusty Doris <unicorn@dusty.name> wrote:
> >> 1.  Simply use mongrels upstream and let it round-robin between all
> >> the unicorn instances on the different servers?  Or, perhaps use the
> >> fair-upstream plugin?
> >>
> >> nginx -> [unicorns]
> >
> > Based on your description of your current setup, this would be the best
> > way to go.  I would configure a lowish listen() :backlog for the
> > Unicorns, fail_timeout=0 in nginx for every server  This setup means
> > round-robin by default, but if one machine gets a :backlog overflow,
> > then nginx will automatically retry on a different backend.
> 
> Thanks for the recommendation.  I was going to give that a shot first
> to see how it went, as it would also be the easiest to manage.
> 
> When you say a lowish backlog?  What kind of numbers are you talking
> about?  Say, we had 8 workers running that stayed pretty active.  They
> are usually quick to respond, with an occasional 2 second response
> (say 1/100) due to a bad sql query that we need to fix.  Would lowish
> be 16, 32, 64, 128, 1024?

1024 is the default in Mongrel and Unicorn which is very generous.  5 is
the default value that Ruby initializes the sockets at, so picking
something in between is recommended.  It really depends on your app and
comfort level.  You can also tune and refine it over time safely
without worrying too much about dropping connections by configuring
multiple listeners per-instance (see below).

Keep in mind the backlog is rarely an exact setting, it's more of a
recommendation to the kernel (and the actual value is often higher
than specified).

> Oh and thanks for the tip on the fail_timeout.

No problem, I somehow thought it was widely-known by now...

> > You can also try the following, which is similar to what I describe in:
> >
> >  http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31
> >
> 
> Thats an interesting idea, thanks for sharing it.  I like how the
> individual server also acts as a load balancer, but only if its having
> trouble itself.  Otherwise, it just handles the requests through the
> socket connection.
> I appreciate your reply and especially for Unicorn.

You can also try a combination of (1) above and my proposed idea in
$gmane/31 by configuring two listeners per-Unicorn instance:

   # primary
   listen 8080, :backlog => 10, :tcp_nopush => true

   # only when all servers overflow the backlog=10 above
   listen 8081, :backlog => 1024, :tcp_nopush => true

And then putting the 8081s as a backup in nginx like this:

  upstream unicorn_failover {
    # round-robin between unicorns with small backlogs
    # as the primary option
    server 192.168.0.1:8080 fail_timeout=0;
    server 192.168.0.2:8080 fail_timeout=0;
    server 192.168.0.3:8080 fail_timeout=0;

    # the "backup" parameter means nginx won't ever try these
    # unless the set of listeners above fail.
    server 192.168.0.1:8081 fail_timeout=0 backup;
    server 192.168.0.2:8081 fail_timeout=0 backup;
    server 192.168.0.3:8081 fail_timeout=0 backup;
  }

You can monitor the nginx error logs and see how often it fails on the
low backlog listener, and then increment/decrement the backlog of
the primary listeners as needed to get better load-balancing.

-- 
Eric Wong

Re: Our Unicorn Setup

[Dusty Doris]

On Fri, Oct 9, 2009 at 7:11 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Dusty Doris <unicorn@dusty.name> wrote:
>> On Fri, Oct 9, 2009 at 6:01 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > Dusty Doris <unicorn@dusty.name> wrote:
>> >> 1.  Simply use mongrels upstream and let it round-robin between all
>> >> the unicorn instances on the different servers?  Or, perhaps use the
>> >> fair-upstream plugin?
>> >>
>> >> nginx -> [unicorns]
>> >
>> > Based on your description of your current setup, this would be the best
>> > way to go.  I would configure a lowish listen() :backlog for the
>> > Unicorns, fail_timeout=0 in nginx for every server  This setup means
>> > round-robin by default, but if one machine gets a :backlog overflow,
>> > then nginx will automatically retry on a different backend.
>>
>> Thanks for the recommendation.  I was going to give that a shot first
>> to see how it went, as it would also be the easiest to manage.
>>
>> When you say a lowish backlog?  What kind of numbers are you talking
>> about?  Say, we had 8 workers running that stayed pretty active.  They
>> are usually quick to respond, with an occasional 2 second response
>> (say 1/100) due to a bad sql query that we need to fix.  Would lowish
>> be 16, 32, 64, 128, 1024?
>
> 1024 is the default in Mongrel and Unicorn which is very generous.  5 is
> the default value that Ruby initializes the sockets at, so picking
> something in between is recommended.  It really depends on your app and
> comfort level.  You can also tune and refine it over time safely
> without worrying too much about dropping connections by configuring
> multiple listeners per-instance (see below).
>
> Keep in mind the backlog is rarely an exact setting, it's more of a
> recommendation to the kernel (and the actual value is often higher
> than specified).
>
>> Oh and thanks for the tip on the fail_timeout.
>
> No problem, I somehow thought it was widely-known by now...
>
>> > You can also try the following, which is similar to what I describe in:
>> >
>> >  http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31
>> >
>>
>> Thats an interesting idea, thanks for sharing it.  I like how the
>> individual server also acts as a load balancer, but only if its having
>> trouble itself.  Otherwise, it just handles the requests through the
>> socket connection.
>> I appreciate your reply and especially for Unicorn.
>
> You can also try a combination of (1) above and my proposed idea in
> $gmane/31 by configuring two listeners per-Unicorn instance:
>
>   # primary
>   listen 8080, :backlog => 10, :tcp_nopush => true
>
>   # only when all servers overflow the backlog=10 above
>   listen 8081, :backlog => 1024, :tcp_nopush => true
>
> And then putting the 8081s as a backup in nginx like this:
>
>   upstream unicorn_failover {
>     # round-robin between unicorns with small backlogs
>    # as the primary option
>     server 192.168.0.1:8080 fail_timeout=0;
>     server 192.168.0.2:8080 fail_timeout=0;
>     server 192.168.0.3:8080 fail_timeout=0;
>
>     # the "backup" parameter means nginx won't ever try these
>    # unless the set of listeners above fail.
>     server 192.168.0.1:8081 fail_timeout=0 backup;
>     server 192.168.0.2:8081 fail_timeout=0 backup;
>     server 192.168.0.3:8081 fail_timeout=0 backup;
>   }
>
> You can monitor the nginx error logs and see how often it fails on the
> low backlog listener, and then increment/decrement the backlog of
> the primary listeners as needed to get better load-balancing.
>
> --
> Eric Wong
>

Awesome!

I am going to give that a shot.