From: Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>
To: Rainbows! list <rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org>
Subject: Re: Page request roundtrip time increases substantially after a bit of use
Date: Mon, 24 Jan 2011 19:50:48 -0800 [thread overview]
Message-ID: <20110125035048.GA8124@dcvr.yhbt.net> (raw)
In-Reply-To: <288407.18061.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
chris mckenzie <kristopolous-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
> chris mckenzie <kristopolous at yahoo.com> wrote:
> > Hi Eric,
> >
> > I'll prepare a more formal response in a bit, but here is my test run:
> > rainbows -c config.rb rackup.ru
>
> >Thanks, I'm trying to reproduce it now. Are you able to reproduce it
> >more quickly by throwing some ab or httperf runs in the mix to make more
> >requests? Reducing worker_processes usually help reproduce issues more
> >quickly, too, especially if it's a resource leak.
>
> It's a finicky bug. That data below was perfect; exactly what I was
> experiencing; but now it refuses to show its face again. Just to be sure; it's
> the plateau that concerns me,not the occasional GC spikes ... not even those few
> huge ones.
Yeah, GC spikes are unavoidable in MRI (but so is Internet latency and
users can't tell :). The plateau is strange/disturbing and I haven't
been able to reproduce it.
I'm still running my test and will probably run it for a few more
hours to be on the safe side.
> Another interesting note is that when I use the full stack that I have, the
> bitrate throughput goes to the tube too. It may be a problem with chunked
> transfer? I don't know really; probably ethtool and wireshark would tell us.
Your test managed to reproduce over loopback, though.
> > Also, do you have any iptables/firewall/QoS settings on that machine
> > that could be interfering?
>
> No. This is a vanilla install. It's a desktop linux system and so it has X and
> firefox and a few terminals; some minimal window manager; nothing extraordinary;
> htop reveals 9GB ram free, although I just noticed that ruby is taking up double
> digit cpu per core; perhaps it's just the nature of ruby but I'll see what I can
> do about graphing that during a lifecycle.
Aha! Double-digit CPU usage is definitely atypical for MRI. Is it some
logrotate job hitting Rainbows! with USR1 signals repeatedly? Is it
stuck in the double-digits or just spiking?
Debian (and presumably Ubuntu) build Ruby 1.8 with the (non-default)
--enable-pthreads option which can lead to this *occasionally* and also
hurt performance. If you're stuck on 1.8, I would always build Ruby to
not use pthreads (or try some of the patches from the guy at
timetobleed.com that makes 1.8+pthreads better).
> > I haven't noticed anything in CLOSE_WAIT since I started testing it a
> > few minutes ago. Maybe it takes longer, but CLOSE_WAIT has always been
> > a rarity to see in my years of working with TCP client/servers...
>
> We are looking at "TIME_WAIT" for a lot of them ... but after the browser is
> down and the thread has been idling for 10 minutes ... it falls into
> "CLOSE_WAIT" until I bring it down and back up.
It's common to have many TIME_WAIT sockets lying around. Until you get
many thousands of non-keepalive requests a second, they're mostly
harmless and the kernel will GC them.
> Let me upgrade my ruby to that today or early tomorrow ... I'm just using the
> one that ubuntu gives me after apt-get update/upgrade (which is patch 72). I
> probably can't go too custom as this has to be part of a deployable sdk; but
> it's worth a shot. Thanks again.
Definitely try that and going without --enable-pthreads if you have to
use 1.8. I just "./configure --prefix=$HOME && make && make install"
but I've heard RVM is popular these days.
--
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
next prev parent reply other threads:[~2011-01-25 4:09 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-24 21:03 Page request roundtrip time increases substantially after a bit of use chris mckenzie
[not found] ` <571697.98064.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-24 21:54 ` Eric Wong
[not found] ` <20110124215440.GA25489-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 0:11 ` Eric Wong
[not found] ` <20110125001107.GA1921-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 1:14 ` chris mckenzie
[not found] ` <443004.19531.qm-9Fhxc66Fx5iB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25 2:02 ` Eric Wong
2011-01-25 2:55 ` chris mckenzie
[not found] ` <288407.18061.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25 3:50 ` Eric Wong [this message]
[not found] ` <20110125035048.GA8124-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 21:38 ` Eric Wong
[not found] ` <20110125213835.GA9421-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03 2:54 ` Eric Wong
[not found] ` <20110203025415.GA3812-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03 4:44 ` chris mckenzie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/rainbows/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110125035048.GA8124@dcvr.yhbt.net \
--to=normalperson-rmlxzr9ms24@public.gmane.org \
--cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/rainbows.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).