From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS33070 50.56.128.0/17 X-Spam-Status: No, score=0.0 required=3.0 tests=AWL,MSGID_FROM_MTA_HEADER, TVD_RCVD_IP shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Claudio Poli Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: How to manage growing memory with Rainbows! Date: Thu, 14 Feb 2013 07:58:17 +0100 Message-ID: References: <20130212050021.GA18443@dcvr.yhbt.net> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1360825114 9238 80.91.229.3 (14 Feb 2013 06:58:34 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 14 Feb 2013 06:58:34 +0000 (UTC) To: Rainbows! list Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Thu Feb 14 07:58:54 2013 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:sender:content-type:mime-version:subject:from :in-reply-to:date:content-transfer-encoding:message-id:references:to :x-mailer:x-gm-message-state; bh=YEq+DyU2RF7zlE9BIm13/nREI9EbtGeImdxgPg+qNkU=; b=emIQLCYwqNWA+TujU9xbfDRIk9AQwwoNMxmUR83PceA0IKyc5gF98mH/nnHucwy6PJ 2EsBbwoyQ44k7Yt2udtqjFbEIcWYTlcS37SBPdBItFWBIV/5m3oGbNyaQZk7gWGLEfS+ O94V14+ZOnNm1zS6T38Zou0G3zM3sVdcJPW9WykMMEJWGMwmwZIaMTHnrP/Cf1aMJF0y dShXBbT9k0LeGLFdtk+ZPwWTxKV2TsMv04hcbXQ7XDA7iplYsFbfsse+Nk4SStizoqM8 M5NW00zZoRpmUjzz9bZWQJ7dBsAhOCTfoUylwbyQaY2+YStX0lY0WtOx4xkOF3JhpQLN sqdA== X-Received: by 10.14.201.5 with SMTP id a5mr15541195eeo.17.1360825100697; Wed, 13 Feb 2013 22:58:20 -0800 (PST) In-Reply-To: <20130212050021.GA18443-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQnwT0YgKxT7M+HrNEj60OzioOHXr6dpbu0gPpQReshPTQc/BdxUPsMZ0x1UmafFvM1gRO7y X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:448 Archived-At: Received: from 50-56-192-79.static.cloud-ips.com ([50.56.192.79] helo=rubyforge.org) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1U5smO-0003vL-Eu for gclrrg-rainbows-talk@m.gmane.org; Thu, 14 Feb 2013 07:58:48 +0100 Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id DFCE52E099; Thu, 14 Feb 2013 06:58:27 +0000 (UTC) Received: from mail-ee0-f49.google.com (mail-ee0-f49.google.com [74.125.83.49]) by rubyforge.org (Postfix) with ESMTP id 477C12E099 for ; Thu, 14 Feb 2013 06:58:21 +0000 (UTC) Received: by mail-ee0-f49.google.com with SMTP id d4so1002671eek.8 for ; Wed, 13 Feb 2013 22:58:20 -0800 (PST) Received: from [192.168.5.103] ([151.64.208.92]) by mx.google.com with ESMTPS id m46sm12759052eeo.16.2013.02.13.22.58.18 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 13 Feb 2013 22:58:19 -0800 (PST) Il giorno 12/feb/2013, alle ore 06:00, Eric Wong ha scritto: Hello Eric, > I'm curious, what tweaks did you try? Measuring average memory in requests and tweaking RUBY_HEAP_MIN_SLOTS, RUBY_GC_MALLOC_LIMIT, RUBY_HEAP_FREE_MIN, etc. Using LD_PRELOAD with libtcmalloc Inviting Ruby (1.9.3) to perform GC after some heavy task. Profiling application under multiple ruby/jruby versions to no avail. Symbols vs string where applicable, etc. > What kind of workload are you running? (many disk writes at all?) No, it's quite a large Rails 3.2 app but we offload this kind of tasks to node.js in our architecture, which is able to operate under 80MB single instance. Our ruby app serves json requests (api) and does the frontend. > Which version of Ruby are you using? Tried all the 1.9.3 patchsets, railsexpress, falcon patches.. > Are you counting VMSize or RSS? Resident size > Are you on 64-bit? No, we were on 64bit but we switched to 32 bit. > Fwiw, virtual memory usage is very high on 64-bit Linux on newer > versions of glibc, but mostly harmless since the memory isn't actually > used (address space is nearly unlimited). > > You can try MALLOC_ARENA_MAX_=1 to limit the number of arenas if you > want. That might reduce fragmentation since the GVL in MRI means > it's unlikely to hit malloc lock contention (glibc uses multiple > malloc arenas to avoid contention by default). I didn't knew about this setting, might be worth a try, thanks. > OobGC is absolutely not recommended for Rainbows! (or anything doing > persistent connections or simultaneous clients within a process) Good to know. > However, you can safely send SIGQUIT to any Rainbows! worker (bypassing > master) whenever you feel memory usage is high, master will restart it. Will Rainbows! wait after the last request before restarting? > You can just put a simple counter in middleware to do it, something > like this: > > # nr is initialized to a number of your choice elsewhere > > nr -= 1 > if nr < 0 > Process.kill(:QUIT, $$) > end > Nice > The best solution is to fix your code/gems/Ruby :) > > I report and fix all the memory leaks I can find in gems+MRI. > > One thing to avoid is allocating too much memory in the first place > (always use LIMIT in SQL SELECT statements, read files in smaller > chunks, etc). It really takes only one poorly thought-out line of > code to either OOM or cause a swap storm. I agree, I'm not really saying I did everything possible but our project uses a lot of gems and I'm confident our ruby code is written fairly well (100% tested, although it doesn't mean anything in this case, easy methods, not really any black magic involved). Leak might be in some gem but so far I haven't been able to spot anything remotely useful. I fear installing new relic since every day I read obscure problems caused by it and I had some myself. > I haven't hit one of these problems in a while, but check out > commit f95113402b7239f225282806673e1b6424522b18 in > git://github.com/rack/rack.git for an example of how IO#gets > can ruin your app. Thanks for the example. What Rainbows! strategy would you run on 1.9.3 given that some API call might take 800ms/1200ms (uncached) and the number of requests is fairly high? Not only we are memory constrained but we are also trying to keep the costs down, the instance is a 4GB c1.medium on EC2, 1 core. Very underpowered as we tend to scale horizontally. Considering a powerful VPS instead at the moment, since we'll prolly have to support 300k users very soon. Thanks! _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying