From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.6 required=5.0 tests=MSGID_FROM_MTA_HEADER, RP_MATCHES_RCVD,T_DKIM_INVALID shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Ryan Tomayko Newsgroups: gmane.comp.lang.ruby.unicorn.general Subject: Re: Fwd: Support for Soft Timeout in Unicorn Date: Fri, 18 Jun 2010 13:13:33 -0700 Message-ID: References: <20100603173749.GA19649@dcvr.yhbt.net> <20100603182246.GB19649@dcvr.yhbt.net> <20100603184730.GA2421@dcvr.yhbt.net> <20100604205925.GA7361@dcvr.yhbt.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1276892029 28345 80.91.229.12 (18 Jun 2010 20:13:49 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Fri, 18 Jun 2010 20:13:49 +0000 (UTC) Cc: Luke Melia To: unicorn list Original-X-From: mongrel-unicorn-bounces@rubyforge.org Fri Jun 18 22:13:44 2010 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=47UIdGFur+FQGYspGS8+h1BsOI2l+sLOZEohbzn5wKo=; b=TCMzJZEZCzFw7WxmsQDA37tbPfqVrILCmFUJ3e7PCshlHwbjMi6A8jg/MU6JnCQM6E sA1YceaR8Kz+Ae4sxUZiWtY2s0+EX4sE0rgDwXoDCmVCxNZPqnWs2cdLWpSoqF0UeROk zB0qOa7/2ji1ICx3SnmidHFm9rlMU+VXgvIG4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=s/B5dYCt+sxPfa14u52gv3wKa0Et5iKlZZ0FC3NPK3/wVSvUHczDmnnXqvOxio1PnM OK+V1qPVibnfWmPKHZJ8ftqjrmBOO/aU+uiCRJYyDcM9lX/6AK5n5AUO4iYaD6xnIjOj Hbx4SVa3W/D1cZyMqXHoOMmC6DSOfcbLIZluw= In-Reply-To: <20100604205925.GA7361@dcvr.yhbt.net> X-Google-Sender-Auth: Ao8gAZaXgVqumjBzpvSJwUQ1frs X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:591 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1OPhwd-0000u6-MX for gclrug-mongrel-unicorn@m.gmane.org; Fri, 18 Jun 2010 22:13:44 +0200 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 96FB3185835A; Fri, 18 Jun 2010 16:13:42 -0400 (EDT) Received: from mail-gw0-f50.google.com (mail-gw0-f50.google.com [74.125.83.50]) by rubyforge.org (Postfix) with ESMTP id 576E1185835A for ; Fri, 18 Jun 2010 16:13:34 -0400 (EDT) Received: by gwb1 with SMTP id 1so1129021gwb.23 for ; Fri, 18 Jun 2010 13:13:34 -0700 (PDT) Received: by 10.229.181.143 with SMTP id by15mr885302qcb.248.1276892013467; Fri, 18 Jun 2010 13:13:33 -0700 (PDT) Received: by 10.229.240.143 with HTTP; Fri, 18 Jun 2010 13:13:33 -0700 (PDT) On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong wrote: > Chris Wanstrath wrote: >> That's what we do at GitHub. We're running Rails 2.2.2 and have a >> custom config.ru, thanks to Unicorn: >> >> http://gist.github.com/424352 > > By the way, how's the OobGC middleware working for you guys? We rolled out the OobGC middleware along with a basic RailsBench GC config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms (~15%) off the average response time across the site (real traffic). The impact was significantly more for requests that allocate a lot of objects -- as much as 50% decreases in response time for the worst offenders. We saw no noticeable increase in CPU with OobGC set to run every 10 requests, and a fair increase in CPU with OobGC set to run every 5 requests. Because I rolled this stuff out somewhat non-scientifically, I've always wondered how much OobGC contributed to the overall savings vs. the RailsBench GC config. Disabling the OobGC middleware but leaving the RailsBench GC config in place, I get the following graph: http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png So we're spending ~1ms request time in GC with OobGC, and ~10ms request time in GC without it. Here's some system load graphs for the same time period just to show that OobGC has no adverse effect when set to GC every 10 requests: http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png I assume the RailsBench GC patches improve the effect of OobGC considerably by increasing the number of objects that can be allocated between GC runs, allowing more of the GC work to be deferred to in-between-requests time. Here's the RailsBench config we're using today, for the record: RUBY_HEAP_MIN_SLOTS=800000 RUBY_HEAP_FREE_MIN=100000 RUBY_HEAP_SLOTS_INCREMENT=300000 RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 RUBY_GC_MALLOC_LIMIT=79000000 This is only barely tuned for us. I stole most of the numbers from the Twitter and 37signals examples. I've also experimented with tuning the GC specifically to take advantage of OobGC: https://gist.github.com/87d574a19372c6043c5f # The following GC settings have been tuned for GitHub application web requests. # Most settings are significantly higher than the example configs published by # Twitter and 37signals. There's a couple reasons for this. First, the GitHub # app has a memory footprint that's 3x-4x larger than the standard Rails app # (roughly 200MB after first request compared to ~40MB-50MB). Second, because # Unicorn is such an exceptional piece of software, we're able to schedule GC # to run outside the context of requests so as not to effect response times. # As such, we try to allocate enough memory to service 5 requests without needing # GC and then run GC manually immediately after each fifth request has been # served but before the process starts accepting the next connection. The result # is higher memory use (~300MB per Unicorn worker process on average) and a # slight increase in CPU due to forced manual GC, but better response times. # ... Unfortunately, the bigger heap seems to cause a largish increase in the time needed for each GC, so the unicorn workers were spending too much time between requests. CPU and RES increases were even more than I'd expected. It also didn't eliminate in-request GC entirely as I'd hoped. I eventually abandoned the idea -- even if I could get it to behave, it's hardly worth the 1ms it would save. I mention it here because the general approach might work in situations where the base heap size is a bit smaller (say < 80MB) or perhaps I'm mistuning one of the parameters. Thanks, Ryan > Luke: did you also get a chance to try this in place of my original > monkey patch? > > Thanks in advance for any info you can share. > > -- > Eric Wong > _______________________________________________ > Unicorn mailing list - mongrel-unicorn@rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying > _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying