unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Eric Wong <normalperson@yhbt.net>
To: mongrel-unicorn@rubyforge.org
Cc: mongrel-users@rubyforge.org
Subject: Re: [Mongrel] scaling unicorn
Date: Mon, 21 Jun 2010 17:16:32 -0700	[thread overview]
Message-ID: <20100622001632.GA10082@dcvr.yhbt.net> (raw)
In-Reply-To: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>

snacktime <snacktime@gmail.com> wrote:
> Interested in some feeback on this (does it sound right?), or maybe
> this might be of interest to others.

Hi Chris,

I think you meant to post this to the mongrel-unicorn@rubyforge.org
list, not mongrel-users@rubyforge.org :>

> We are launching a new facebook app in a couple weeks and we did some
> load testing over the weekend on our unicorn web cluster.  The servers
> are 8 way xeon's with 24gb ram.  Our app ended up being primarily cpu
> bound.  So far the sweet spot for the number of unicorns seems to be
> around 40.  This seemed to yield the most requests per second without
> overloading the server or hitting memory bandwidth issues.  The
> backlog is at the somaxconn default of 128, I'm still not sure if we
> will bump that up or not.

The default backlog we try to specify is actually 1024 (same as
Mongrel).  But it's always a murky value anyways, as it's
kernel/sysctl-dependent.  With Unix domain sockets, some folks use
crazy values like 2048 to look better on synthetic benchmarks :)

> Increasing the number of unicorns beyond a
> certain point resulted in a noticable drop in the requests per second
> the server could handle.   I'm pretty sure the cause is the box
> running out of memory bandwidth.  The load average and resource usage
> in general (except for memory) would keep going down but so did the
> requests per second.  At 80 unicorns the requests per second dropped
> by more then half.  I'm going to disable hyperthreading and rerun some
> of the tests to see what impact that has.

That's "8 way xeon" _before_ hyperthreading, right?  Which family of
Xeons are you using, the Pentium4-based crap or the awesome new ones?

How much memory is each Unicorn worker using for your app?

40 workers for 8 physical cores sounds reasonable.  Depending on the
app, I think the reasonable range is anywhere from 2-8 workers per
physical core.  More if you're (unfortunately) limited by external
network calls, but since you claim to be CPU bound, less.

Do you have actual performance numbers you're able to share?
Mean/median request times/rates would be very useful.  If your requests
run very quickly, you may be limited by contention with the accept()
syscall on the listen socket, too.

I assume you're using nginx as the proxy, is this with Unix domain
sockets or TCP sockets?  Unix domain sockets should give a small
performance over TCP if it's all on the same box.

With TCP, you should also check to see you have enough local ports
available if you're hitting extremely high (and probably unrealistic :)
request rates.

Eric Wong
Unicorn mailing list - mongrel-unicorn@rubyforge.org
Do not quote signatures (like this one) or top post when replying

       reply	other threads:[~2010-06-22  0:16 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
2010-06-22  0:16 ` Eric Wong [this message]
2010-06-22  2:34   ` scaling unicorn Jamie Wilkinson
2010-06-22  4:53     ` Eric Wong
2010-06-22 18:03       ` snacktime
2010-06-22 18:57         ` Jamie Wilkinson
2010-06-23  9:32         ` Eric Wong
2010-06-22 19:18       ` Jamie Wilkinson
2010-06-22 17:30   ` [Mongrel] " snacktime

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20100622001632.GA10082@dcvr.yhbt.net \
    --to=normalperson@yhbt.net \
    --cc=mongrel-unicorn@rubyforge.org \
    --cc=mongrel-users@rubyforge.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).