Rainbows! Rack HTTP server user/dev discussion
 help / color / mirror / code / Atom feed
From: chris mckenzie <kristopolous-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
To: Rainbows! list <rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org>
Subject: Re: Page request roundtrip time increases substantially after a bit of use
Date: Mon, 24 Jan 2011 18:55:03 -0800 (PST)	[thread overview]
Message-ID: <288407.18061.qm@web63303.mail.re1.yahoo.com> (raw)
In-Reply-To: <20110125001107.GA1921-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>

chris mckenzie <kristopolous at yahoo.com> wrote:
> Hi Eric,
> 
> I'll prepare a more formal response in a bit, but here is my test run:
> rainbows -c config.rb rackup.ru

>Thanks, I'm trying to reproduce it now.  Are you able to reproduce it
>more quickly by throwing some ab or httperf runs in the mix to make more
>requests?  Reducing worker_processes usually help reproduce issues more
>quickly, too, especially if it's a resource leak.

It's a finicky bug.  That data below was perfect; exactly what I was 
experiencing; but now it refuses to show its face again.  Just to be sure; it's 
the plateau that concerns me,not the occasional GC spikes ... not even those few 
huge ones.


Another interesting note is that when I use the full stack that I have, the 
bitrate throughput goes to the tube too.  It may be a problem with chunked 
transfer? I don't know really; probably ethtool and wireshark would tell us.

Here's the basic pattern though...

My application load pulls down < 100k JS.  Usually it's about 60ms or so.  When 
the plateau hits, curl reports that the throughput goes from 10.9M/s to 32K/s 
... however, this could be an unrelated problem.

If it's not, however, and If this problem was scarce resource acquisition 
entirely, then I would probably think that a 10 byte file and an 80k file would 
take about the same time ... say 2 seconds + however long it took to transfer.


> Also, do you have any iptables/firewall/QoS settings on that machine
> that could be interfering?

No.  This is a vanilla install.  It's a desktop linux system and so it has X and 
firefox and a few terminals; some minimal window manager; nothing extraordinary; 
htop reveals 9GB ram free, although I just noticed that ruby is taking up double 
digit cpu per core; perhaps it's just the nature of ruby but I'll see what I can 
do about graphing that during a lifecycle.




> I haven't noticed anything in CLOSE_WAIT since I started testing it a
> few minutes ago.  Maybe it takes longer, but CLOSE_WAIT has always been
> a rarity to see in my years of working with TCP client/servers...

We are looking at "TIME_WAIT" for a lot of them ... but after the browser is 
down and the thread has been idling for 10 minutes ... it falls into 
"CLOSE_WAIT" until I bring it down and back up.


> The (CSV) output of the test can be seen here:
> http://qaa.ath.cx/single-request.csv.gz

> Just covering all my bases here, you don't have a super slow
> disk/filesystem that bogs down your entire system once your logs grow to
> a certain size, right?

I *thought* about this ... truly.  I haven't factored it out to be honest but 
no, the await from iostat is good, the machine itself is still responsive to my 
cp and mv commands for the data during the tests and there is no noticable 
slowdown of anything else.  It's a two way quad core xeon with 12GB of memory 
(t7500) so I don't think that the physical hardware is to blame (although it 
could be memory related; who knows!)

> For those of you without any visualization software, I made a rudimentary graph 
>
>
> from the data here:
> 
> http://qaa.ath.cx/single-request.png
> 
> You can clearly see how the delay increases and then doesn't ever go back down 


> to previous levels.

> Very strange.

> I'm testing on my 32-bit machine with ruby 1.8.7 (2010-12-23 patchlevel
> 330) [i686-linux] straight off of ruby-lang.org .  I usually use
> Rainbows! with 1.9.2-p136 and will try that if I can reproduce the bug
> under 1.8.7, too.


Let me upgrade my ruby to that today or early tomorrow ... I'm just using the 
one that ubuntu gives me after apt-get update/upgrade (which is patch 72).  I 
probably can't go too custom as this has to be part of a deployable sdk; but 
it's worth a shot.  Thanks again.

~chris.

> I'll research answers to your previous questions now.  Thanks for looking into 


> this!

Alright, thanks!

-- 
Eric Wong



      
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


  parent reply	other threads:[~2011-01-25  3:01 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-01-24 21:03 Page request roundtrip time increases substantially after a bit of use chris mckenzie
     [not found] ` <571697.98064.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-24 21:54   ` Eric Wong
     [not found]     ` <20110124215440.GA25489-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25  0:11       ` Eric Wong
     [not found]         ` <20110125001107.GA1921-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25  1:14           ` chris mckenzie
     [not found]             ` <443004.19531.qm-9Fhxc66Fx5iB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25  2:02               ` Eric Wong
2011-01-25  2:55           ` chris mckenzie [this message]
     [not found]             ` <288407.18061.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25  3:50               ` Eric Wong
     [not found]                 ` <20110125035048.GA8124-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 21:38                   ` Eric Wong
     [not found]                     ` <20110125213835.GA9421-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03  2:54                       ` Eric Wong
     [not found]                         ` <20110203025415.GA3812-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03  4:44                           ` chris mckenzie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/rainbows/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=288407.18061.qm@web63303.mail.re1.yahoo.com \
    --to=kristopolous-/e1597as9lqavxtiumwx3w@public.gmane.org \
    --cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/rainbows.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).