From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: * X-Spam-ASN: AS33070 50.56.128.0/17 X-Spam-Status: No, score=1.0 required=3.0 tests=AWL,HK_RANDOM_FROM, MSGID_FROM_MTA_HEADER,TVD_RCVD_IP shortcircuit=no autolearn=no version=3.3.2 Path: news.gmane.org!not-for-mail From: Eric Wong Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: How to manage growing memory with Rainbows! Date: Thu, 14 Feb 2013 07:15:12 +0000 Message-ID: <20130214071512.GA10890@dcvr.yhbt.net> References: <20130212050021.GA18443@dcvr.yhbt.net> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1360826374 20363 80.91.229.3 (14 Feb 2013 07:19:34 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 14 Feb 2013 07:19:34 +0000 (UTC) To: Rainbows! list Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Thu Feb 14 08:19:56 2013 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:449 Archived-At: Received: from 50-56-192-79.static.cloud-ips.com ([50.56.192.79] helo=rubyforge.org) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1U5t6o-0006n8-4p for gclrrg-rainbows-talk@m.gmane.org; Thu, 14 Feb 2013 08:19:54 +0100 Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id CE54B2E09E; Thu, 14 Feb 2013 07:19:33 +0000 (UTC) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by rubyforge.org (Postfix) with ESMTP id 9462C2E09A for ; Thu, 14 Feb 2013 07:15:14 +0000 (UTC) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 0707B1F442; Thu, 14 Feb 2013 07:15:12 +0000 (UTC) Claudio Poli wrote: > Il giorno 12/feb/2013, alle ore 06:00, Eric Wong ha scritto: > > Using LD_PRELOAD with libtcmalloc Last I checked, tcmalloc never releases memory to the OS, so that could be a problem. (Giving memory back to the kernel and then getting it back soon afterwards is slow because the kernel must clear that memory, first, so most malloc implementations (including glibc) will try to keep memory for the process) > > You can try MALLOC_ARENA_MAX_=1 to limit the number of arenas if you > > want. That might reduce fragmentation since the GVL in MRI means > > it's unlikely to hit malloc lock contention (glibc uses multiple > > malloc arenas to avoid contention by default). > > I didn't knew about this setting, might be worth a try, thanks. You might also want to try MALLOC_MMAP_THRESHOLD_=131072 to disable the sliding sbrk/mmap window (you can try larger/smaller values). The latest Linux man-pages releases have better mallopt(3) documentation (but the arenas stuff was only documented on Dreppers blog, AFAIK (and glibc.git/malloc/*.[ch] comments) > > However, you can safely send SIGQUIT to any Rainbows! worker (bypassing > > master) whenever you feel memory usage is high, master will restart it. > > Will Rainbows! wait after the last request before restarting? Yes, SIGQUIT => graceful shutdown will always work for normal apps. The only exception is the new, optional rack.hijack (in rainbows.git). > What Rainbows! strategy would you run on 1.9.3 given that some API call might take 800ms/1200ms (uncached) and the number of requests is fairly high? It depends on your app. For standard Ruby API (Net::*), ThreadSpawn/Pool should be the most compatible and work well enough. These will work great if you're using Celluloid, too. EventMachine can scale very well but it requires your network-capable gems/libraries to use EM to benefit. Same issue with cool.io. I also like XEpollThread{Pool/Spawn}, but they're Linux-only and less-used. _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying