load balancing with keepalive is hard, but we try anyways

From: Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>
To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
Subject: load balancing with keepalive is hard, but we try anyways
Date: Thu, 20 Jan 2011 11:15:35 +0000	[thread overview]
Message-ID: <20110120111535.GA27078@dcvr.yhbt.net> (raw)

(I've alluded to some of these problems in previous messages/release
notes/commit messages, but I've been meaning to write more on this
subject for ages, now, so here goes...)

>From what I can tell, all concurrency options suck at it.  Native
threads eat too much memory, event loops (that I've seen in the wild)
don't utilize multicore effectively, yadda yadda...

Disabling keepalive gives you the best load balancing for short
request-response cycles: this is what Unicorn does (with help from
nginx).  It's crap for slow/trickle stuff (what Rainbows! is for) and
also suboptimal for "hello world"-type applications that can do several
thousand requests a second (Rainbows! can do this, too, sorta).

And even in a perfect world where we could use teeny stacks, clone() and
no GVL, it all goes out the door when you have more than one machine.

How Rainbows! deals with keepalive load balancing now:

* Short keepalive timeout by default (5s).  This reduces the
  memory/cycles spent on idle clients.

* keepalive requests limited to 100 by default. This is to prevent
  aggressive clients from monopolizing a single thread or process.
  I considered allowing this to be a range and random per-client, but
  other factors (other clients, machine load, GC, different request
  profiles) provide enough randomness in my testing that it wasn't
  worth the extra code.

Both of these strategies force clients to reconnect more frequently than
they need to, allowing them to migrate to a different, potentially
less-loaded worker process/thread.  It ends up being a see-saw effect
that seems "good enough" load balancing to the mix but you're never in
an ideal (nor terrible) situation for long.

I'm working on yet another strategy, but it probably won't be useful
immediately under Ruby 1.9 and it won't be useful for multiple hosts...

Also keep in mind that hardly anybody uses nor needs Rainbows!, but
these issues affect other servers in other languages and platforms, too.

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying