From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.5 required=3.0 tests=AWL,MSGID_FROM_MTA_HEADER, RP_MATCHES_RCVD shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Eric Wong Newsgroups: gmane.comp.lang.ruby.unicorn.general,gmane.comp.lang.ruby.rainbows.general Subject: Re: rainbows for sleepy/lethargic apps Date: Mon, 16 Nov 2009 13:26:41 -0800 Message-ID: <20091116212641.GA7093@dcvr.yhbt.net> References: <76461C1B-E671-4DA3-BED0-12F9E571125A@elctech.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1258406813 7515 80.91.229.12 (16 Nov 2009 21:26:53 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 16 Nov 2009 21:26:53 +0000 (UTC) Cc: rainbows-talk@rubyforge.org To: unicorn list Original-X-From: mongrel-unicorn-bounces@rubyforge.org Mon Nov 16 22:26:46 2009 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org Content-Disposition: inline In-Reply-To: <76461C1B-E671-4DA3-BED0-12F9E571125A@elctech.com> User-Agent: Mutt/1.5.18 (2008-05-17) X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:165 gmane.comp.lang.ruby.rainbows.general:28 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.50) id 1NA95w-0001E4-JS for gclrug-mongrel-unicorn@m.gmane.org; Mon, 16 Nov 2009 22:26:44 +0100 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id C22A218582D5; Mon, 16 Nov 2009 16:26:43 -0500 (EST) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by rubyforge.org (Postfix) with ESMTP id D216B18582D0; Mon, 16 Nov 2009 16:26:41 -0500 (EST) Received: from localhost (unknown [127.0.2.5]) by dcvr.yhbt.net (Postfix) with ESMTP id 3A1D11F449; Mon, 16 Nov 2009 21:26:41 +0000 (UTC) Dylan Stamat wrote: > I'm working on a project that has very bad application performance, > specifically, tons of long running requests due to view and > (specifically) database contention. > = > While I haven't tested our Unicorn setup under load yet, tweak backlog > counts, etc... I'm wondering if I'd be better off giving Rainbows! a > shot? In the Rainbows! AppPool diagram, I'm having a hard time > understanding the N:P relationship. When the P threshold is met, what > happens to N (the client)? Hi Dylan, With AppPool, request headers are fully parsed and end up getting queued up in userspace. No request bodies are read (since AppPool is only compatible with the Thread-based concurrency models at the moment). Rainbows is designed for apps that are intentionally/unavoidably sleepy. Since I imagine the database is within your control, I'd optimize that first and fall back to using Unicorn with a higher :backlog. Since you're hitting database contention, throwing more concurrency at the database with Rainbows! is probably the wrong thing to do... If your database is just on a high latency network for whatever reason, then maybe Rainbows! with Neverblock drivers can help. Otherwise if your database is bogged down because of memory/CPU/disk, then you probably want fewer things hitting it at once. Can you shard, cache or otherwise offload DB requests? With slow views, Unicorn is pretty effective at using all available cores already with multiple worker processes. Even with a Ruby VM with non-crippled native threads (Rubinius maybe, not MRI 1.9.1) I doubt it'll help with CPU/memory-bound rendering performance. > Also, with both Unicorn/Rainbows!, is there a explicit timeout number > set on the clients (and how does that differ between > Unicorn/Rainbows!)? For instance, in our Nginx config, I have > proxy_buffering turned on, with proxy_read_timeout and > proxy_send_timeout's, set pretty high. Based on your nginx config, the easiest thing would probably be to set a high :backlog for now and then work on tuning/optimizing/avoiding your database ASAP. Views are harder to optimize, Unicorn itself always uses as much resources as the OS can give it to render. A lot of slow views I've seen hit a lot of memory allocation, so check out tcmalloc (included with REE) or use 1.9 (possibly with tcmalloc, too). 1.9 has some good optimizations for allocating shorter strings (should be more effective on 64-bit) and small hashes/arrays, too. My personal style is also to use destructive methods as much as possible (concat/<< vs +/+=3D, map! vs map, gsub! vs gsub, tr! vs tr, etc...). Avoid creating lots of OpenStruct, too, excessive use of define_method/metadef allocates a lot of short-lived objects and thrashes memory. The timeout in Unicorn affects the entire request (including all I/O). The timeout in Rainbows only kicks in when the process is completely deadlocked. Since apparently lots of "normal" HTTP clients have ridiculous keepalive times, Rainbows! will be getting a separate keepalive_timeout directive soon that only affects reading HTTP headers. I don't plan enforcing a timeout when processing request bodies or app responses (those can probably be done on a more controlled manner in the app/middleware). > Lastly... =A1You rock Eric! Thanks, but much of the credit goes to random folks all over who have helped with this (and projects leading up to this), too :> -- = Eric Wong