From: chris mckenzie <kristopolous-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
To: Rainbows! list <rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org>
Subject: Re: Page request roundtrip time increases substantially after a bit of use
Date: Mon, 24 Jan 2011 18:55:03 -0800 (PST) [thread overview]
Message-ID: <288407.18061.qm@web63303.mail.re1.yahoo.com> (raw)
In-Reply-To: <20110125001107.GA1921-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
chris mckenzie <kristopolous at yahoo.com> wrote:
> Hi Eric,
>
> I'll prepare a more formal response in a bit, but here is my test run:
> rainbows -c config.rb rackup.ru
>Thanks, I'm trying to reproduce it now. Are you able to reproduce it
>more quickly by throwing some ab or httperf runs in the mix to make more
>requests? Reducing worker_processes usually help reproduce issues more
>quickly, too, especially if it's a resource leak.
It's a finicky bug. That data below was perfect; exactly what I was
experiencing; but now it refuses to show its face again. Just to be sure; it's
the plateau that concerns me,not the occasional GC spikes ... not even those few
huge ones.
Another interesting note is that when I use the full stack that I have, the
bitrate throughput goes to the tube too. It may be a problem with chunked
transfer? I don't know really; probably ethtool and wireshark would tell us.
Here's the basic pattern though...
My application load pulls down < 100k JS. Usually it's about 60ms or so. When
the plateau hits, curl reports that the throughput goes from 10.9M/s to 32K/s
... however, this could be an unrelated problem.
If it's not, however, and If this problem was scarce resource acquisition
entirely, then I would probably think that a 10 byte file and an 80k file would
take about the same time ... say 2 seconds + however long it took to transfer.
> Also, do you have any iptables/firewall/QoS settings on that machine
> that could be interfering?
No. This is a vanilla install. It's a desktop linux system and so it has X and
firefox and a few terminals; some minimal window manager; nothing extraordinary;
htop reveals 9GB ram free, although I just noticed that ruby is taking up double
digit cpu per core; perhaps it's just the nature of ruby but I'll see what I can
do about graphing that during a lifecycle.
> I haven't noticed anything in CLOSE_WAIT since I started testing it a
> few minutes ago. Maybe it takes longer, but CLOSE_WAIT has always been
> a rarity to see in my years of working with TCP client/servers...
We are looking at "TIME_WAIT" for a lot of them ... but after the browser is
down and the thread has been idling for 10 minutes ... it falls into
"CLOSE_WAIT" until I bring it down and back up.
> The (CSV) output of the test can be seen here:
> http://qaa.ath.cx/single-request.csv.gz
> Just covering all my bases here, you don't have a super slow
> disk/filesystem that bogs down your entire system once your logs grow to
> a certain size, right?
I *thought* about this ... truly. I haven't factored it out to be honest but
no, the await from iostat is good, the machine itself is still responsive to my
cp and mv commands for the data during the tests and there is no noticable
slowdown of anything else. It's a two way quad core xeon with 12GB of memory
(t7500) so I don't think that the physical hardware is to blame (although it
could be memory related; who knows!)
> For those of you without any visualization software, I made a rudimentary graph
>
>
> from the data here:
>
> http://qaa.ath.cx/single-request.png
>
> You can clearly see how the delay increases and then doesn't ever go back down
> to previous levels.
> Very strange.
> I'm testing on my 32-bit machine with ruby 1.8.7 (2010-12-23 patchlevel
> 330) [i686-linux] straight off of ruby-lang.org . I usually use
> Rainbows! with 1.9.2-p136 and will try that if I can reproduce the bug
> under 1.8.7, too.
Let me upgrade my ruby to that today or early tomorrow ... I'm just using the
one that ubuntu gives me after apt-get update/upgrade (which is patch 72). I
probably can't go too custom as this has to be part of a deployable sdk; but
it's worth a shot. Thanks again.
~chris.
> I'll research answers to your previous questions now. Thanks for looking into
> this!
Alright, thanks!
--
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
next prev parent reply other threads:[~2011-01-25 3:01 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-24 21:03 Page request roundtrip time increases substantially after a bit of use chris mckenzie
[not found] ` <571697.98064.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-24 21:54 ` Eric Wong
[not found] ` <20110124215440.GA25489-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 0:11 ` Eric Wong
[not found] ` <20110125001107.GA1921-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 1:14 ` chris mckenzie
[not found] ` <443004.19531.qm-9Fhxc66Fx5iB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25 2:02 ` Eric Wong
2011-01-25 2:55 ` chris mckenzie [this message]
[not found] ` <288407.18061.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25 3:50 ` Eric Wong
[not found] ` <20110125035048.GA8124-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 21:38 ` Eric Wong
[not found] ` <20110125213835.GA9421-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03 2:54 ` Eric Wong
[not found] ` <20110203025415.GA3812-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03 4:44 ` chris mckenzie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/rainbows/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=288407.18061.qm@web63303.mail.re1.yahoo.com \
--to=kristopolous-/e1597as9lqavxtiumwx3w@public.gmane.org \
--cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/rainbows.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).