From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: * X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=1.0 required=3.0 tests=AWL,HK_RANDOM_FROM, MSGID_FROM_MTA_HEADER shortcircuit=no autolearn=no version=3.3.2 Path: news.gmane.org!not-for-mail From: Eric Wong Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: Page request roundtrip time increases substantially after a bit of use Date: Mon, 24 Jan 2011 19:50:48 -0800 Message-ID: <20110125035048.GA8124@dcvr.yhbt.net> References: <571697.98064.qm@web63303.mail.re1.yahoo.com> <20110124215440.GA25489@dcvr.yhbt.net> <20110125001107.GA1921@dcvr.yhbt.net> <288407.18061.qm@web63303.mail.re1.yahoo.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1295928593 5435 80.91.229.12 (25 Jan 2011 04:09:53 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Tue, 25 Jan 2011 04:09:53 +0000 (UTC) To: Rainbows! list Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Tue Jan 25 05:09:49 2011 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Content-Disposition: inline In-Reply-To: <288407.18061.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org> User-Agent: Mutt/1.5.18 (2008-05-17) X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:202 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1PhaE0-0003W7-Lq for gclrrg-rainbows-talk@m.gmane.org; Tue, 25 Jan 2011 05:09:48 +0100 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id A48AA1678320; Mon, 24 Jan 2011 23:09:47 -0500 (EST) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by rubyforge.org (Postfix) with ESMTP id 0ADC718582E2 for ; Mon, 24 Jan 2011 22:50:49 -0500 (EST) Received: from localhost (unknown [127.0.2.5]) by dcvr.yhbt.net (Postfix) with ESMTP id 360031F7E4; Tue, 25 Jan 2011 03:50:49 +0000 (UTC) chris mckenzie wrote: > chris mckenzie wrote: > > Hi Eric, > > > > I'll prepare a more formal response in a bit, but here is my test run: > > rainbows -c config.rb rackup.ru > > >Thanks, I'm trying to reproduce it now. Are you able to reproduce it > >more quickly by throwing some ab or httperf runs in the mix to make more > >requests? Reducing worker_processes usually help reproduce issues more > >quickly, too, especially if it's a resource leak. > > It's a finicky bug. That data below was perfect; exactly what I was > experiencing; but now it refuses to show its face again. Just to be sure; it's > the plateau that concerns me,not the occasional GC spikes ... not even those few > huge ones. Yeah, GC spikes are unavoidable in MRI (but so is Internet latency and users can't tell :). The plateau is strange/disturbing and I haven't been able to reproduce it. I'm still running my test and will probably run it for a few more hours to be on the safe side. > Another interesting note is that when I use the full stack that I have, the > bitrate throughput goes to the tube too. It may be a problem with chunked > transfer? I don't know really; probably ethtool and wireshark would tell us. Your test managed to reproduce over loopback, though. > > Also, do you have any iptables/firewall/QoS settings on that machine > > that could be interfering? > > No. This is a vanilla install. It's a desktop linux system and so it has X and > firefox and a few terminals; some minimal window manager; nothing extraordinary; > htop reveals 9GB ram free, although I just noticed that ruby is taking up double > digit cpu per core; perhaps it's just the nature of ruby but I'll see what I can > do about graphing that during a lifecycle. Aha! Double-digit CPU usage is definitely atypical for MRI. Is it some logrotate job hitting Rainbows! with USR1 signals repeatedly? Is it stuck in the double-digits or just spiking? Debian (and presumably Ubuntu) build Ruby 1.8 with the (non-default) --enable-pthreads option which can lead to this *occasionally* and also hurt performance. If you're stuck on 1.8, I would always build Ruby to not use pthreads (or try some of the patches from the guy at timetobleed.com that makes 1.8+pthreads better). > > I haven't noticed anything in CLOSE_WAIT since I started testing it a > > few minutes ago. Maybe it takes longer, but CLOSE_WAIT has always been > > a rarity to see in my years of working with TCP client/servers... > > We are looking at "TIME_WAIT" for a lot of them ... but after the browser is > down and the thread has been idling for 10 minutes ... it falls into > "CLOSE_WAIT" until I bring it down and back up. It's common to have many TIME_WAIT sockets lying around. Until you get many thousands of non-keepalive requests a second, they're mostly harmless and the kernel will GC them. > Let me upgrade my ruby to that today or early tomorrow ... I'm just using the > one that ubuntu gives me after apt-get update/upgrade (which is patch 72). I > probably can't go too custom as this has to be part of a deployable sdk; but > it's worth a shot. Thanks again. Definitely try that and going without --enable-pthreads if you have to use 1.8. I just "./configure --prefix=$HOME && make && make install" but I've heard RVM is popular these days. -- Eric Wong _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying