From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.6 required=5.0 tests=AWL,MSGID_FROM_MTA_HEADER, RP_MATCHES_RCVD,T_DKIM_INVALID shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Dusty Doris Newsgroups: gmane.comp.lang.ruby.unicorn.general Subject: Re: Our Unicorn Setup Date: Fri, 9 Oct 2009 19:17:36 -0400 Message-ID: References: <8b73aaca0910091242u78ac787aj48fb1b63b5bf55bc@mail.gmail.com> <20091009220110.GB14137@dcvr.yhbt.net> <20091009231133.GC14137@dcvr.yhbt.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1255130269 8099 80.91.229.12 (9 Oct 2009 23:17:49 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 9 Oct 2009 23:17:49 +0000 (UTC) Cc: mongrel-unicorn@rubyforge.org To: Eric Wong Original-X-From: mongrel-unicorn-bounces@rubyforge.org Sat Oct 10 01:17:39 2009 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=WfX6P46cdN01Bp85C88jG+QwAyc134IVwTAaT/gCR/Y=; b=HMFP3W9QM9K98/7I5+jTpAXn79uPopc/JhkGCsjicTXgl8TJAK7ZZrWMqbkGcq/ceN u+kenUUOpNSgttmfoPJrD7DRutiHq72+u54ZG9YAQxMtaChddfPJGVAjbJgOHSiYxJ4s tvcYWOYmj25JZZYcQfFIYQGmou8WB9LpEFCvE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=nJSj1dZi7173Z+knTc7DU/YUS/BnbBBBhTOYvVO1fcXaNRl3CQJEKOVa8Hwx9Re8QM 7HCq/KPE3fKJaDFjRoc2bvlnooCUrIMqILZWvJ5jM/rDn1F0m60qVwwNByKZJg4ew3go v0Ll6P8QFB0yQLxRWKSQoKGJYx1Ns1DuvAjqI= In-Reply-To: <20091009231133.GC14137@dcvr.yhbt.net> X-Google-Sender-Auth: 0a560b6942ad61fe X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:74 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.50) id 1MwOiP-0006Fl-TT for gclrug-mongrel-unicorn@m.gmane.org; Sat, 10 Oct 2009 01:17:38 +0200 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 5DF7718581F7; Fri, 9 Oct 2009 19:17:37 -0400 (EDT) Received: from mail-yx0-f187.google.com (mail-yx0-f187.google.com [209.85.210.187]) by rubyforge.org (Postfix) with ESMTP id 844C418581F7 for ; Fri, 9 Oct 2009 19:17:36 -0400 (EDT) Received: by yxe17 with SMTP id 17so1016275yxe.33 for ; Fri, 09 Oct 2009 16:17:36 -0700 (PDT) Received: by 10.150.183.11 with SMTP id g11mr5836739ybf.50.1255130256258; Fri, 09 Oct 2009 16:17:36 -0700 (PDT) On Fri, Oct 9, 2009 at 7:11 PM, Eric Wong wrote: > Dusty Doris wrote: >> On Fri, Oct 9, 2009 at 6:01 PM, Eric Wong wrote: >> > Dusty Doris wrote: >> >> 1. =A0Simply use mongrels upstream and let it round-robin between all >> >> the unicorn instances on the different servers? =A0Or, perhaps use the >> >> fair-upstream plugin? >> >> >> >> nginx -> [unicorns] >> > >> > Based on your description of your current setup, this would be the best >> > way to go. =A0I would configure a lowish listen() :backlog for the >> > Unicorns, fail_timeout=3D0 in nginx for every server =A0This setup mea= ns >> > round-robin by default, but if one machine gets a :backlog overflow, >> > then nginx will automatically retry on a different backend. >> >> Thanks for the recommendation. =A0I was going to give that a shot first >> to see how it went, as it would also be the easiest to manage. >> >> When you say a lowish backlog? =A0What kind of numbers are you talking >> about? =A0Say, we had 8 workers running that stayed pretty active. =A0Th= ey >> are usually quick to respond, with an occasional 2 second response >> (say 1/100) due to a bad sql query that we need to fix. =A0Would lowish >> be 16, 32, 64, 128, 1024? > > 1024 is the default in Mongrel and Unicorn which is very generous. =A05 is > the default value that Ruby initializes the sockets at, so picking > something in between is recommended. =A0It really depends on your app and > comfort level. =A0You can also tune and refine it over time safely > without worrying too much about dropping connections by configuring > multiple listeners per-instance (see below). > > Keep in mind the backlog is rarely an exact setting, it's more of a > recommendation to the kernel (and the actual value is often higher > than specified). > >> Oh and thanks for the tip on the fail_timeout. > > No problem, I somehow thought it was widely-known by now... > >> > You can also try the following, which is similar to what I describe in: >> > >> > =A0http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31 >> > >> >> Thats an interesting idea, thanks for sharing it. =A0I like how the >> individual server also acts as a load balancer, but only if its having >> trouble itself. =A0Otherwise, it just handles the requests through the >> socket connection. >> I appreciate your reply and especially for Unicorn. > > You can also try a combination of (1) above and my proposed idea in > $gmane/31 by configuring two listeners per-Unicorn instance: > > =A0 # primary > =A0 listen 8080, :backlog =3D> 10, :tcp_nopush =3D> true > > =A0 # only when all servers overflow the backlog=3D10 above > =A0 listen 8081, :backlog =3D> 1024, :tcp_nopush =3D> true > > And then putting the 8081s as a backup in nginx like this: > > =A0=A0upstream unicorn_failover { > =A0=A0 =A0# round-robin between unicorns with small backlogs > =A0 =A0# as the primary option > =A0=A0 =A0server 192.168.0.1:8080 fail_timeout=3D0; > =A0=A0 =A0server 192.168.0.2:8080 fail_timeout=3D0; > =A0=A0 =A0server 192.168.0.3:8080 fail_timeout=3D0; > > =A0=A0 =A0# the "backup" parameter means nginx won't ever try these > =A0 =A0# unless the set of listeners above fail. > =A0=A0 =A0server 192.168.0.1:8081 fail_timeout=3D0 backup; > =A0=A0 =A0server 192.168.0.2:8081 fail_timeout=3D0 backup; > =A0=A0 =A0server 192.168.0.3:8081 fail_timeout=3D0 backup; > =A0=A0} > > You can monitor the nginx error logs and see how often it fails on the > low backlog listener, and then increment/decrement the backlog of > the primary listeners as needed to get better load-balancing. > > -- > Eric Wong > Awesome! I am going to give that a shot.