Re: Maintaining capacity during deploys

From: seth.cousins@gmail.com
To: unicorn list <mongrel-unicorn@rubyforge.org>
Subject: Re: Maintaining capacity during deploys
Date: Thu, 29 Nov 2012 20:48:16 -0800	[thread overview]
Message-ID: <C638072D-9CB4-4C57-BB72-FD71859AD387@gmail.com> (raw)
In-Reply-To: <20121130012447.GA10092@dcvr.yhbt.net>

In my experience high loads and contention are a common issue when restarting the unicorn master process. In a previous project we dealt with this by 1) performing some warmup requests in the master before starting to fork workers; 2) replacing old workers slowly by having each new worker send a TTOU to the old master in after_fork and having the new master sleep for a couple of seconds between spawning workers.

It was a couple of years ago so the details are not fresh but iirc before tuning a restart took 5-10 seconds followed by load climbing to 10-20 (on a 4 proc machine) with a 2-5 minute slow recovery of long request times. In particularly pathological cases requests can start timing out which results in workers being killed and new workers needing to warm up on and already overloaded system. After tuning the rolling restart took 30-40 seconds but the load barely budged and the request processing times stayed constant.

.seth

On Nov 29, 2012, at 5:24 PM, Eric Wong <normalperson@yhbt.net> wrote:

> Tony Arcieri <tony.arcieri@gmail.com> wrote:
>> On Thu, Nov 29, 2012 at 3:34 PM, Lawrence Pit <lawrence.pit@gmail.com> wrote:
>>> 
>>> Perhaps it's possible to warm up the workers in the unicorn after_fork block?
>> 
>> Are people doing this in production (i.e. moving the termination of
>> the old master from before_fork to after_fork)? My worry is that
>> during this warming process you will have 2X the normal number of
>> Unicorn workers active at the same time, which could potentially lead
>> to exhausting of system resources (i.e. RAM)
> 
> I haven't done any terminations in the *_fork hooks for a long time.
> I just let 2x the normal workers run for a bit before sending SIGQUIT.
> 
> That said, I usually have plenty of RAM (and DB connections) to spare.
> Excessive CPU-bound loads are handled very well nowadays.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying