From: chris mckenzie <kristopolous-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
Subject: Page request roundtrip time increases substantially after a bit of use
Date: Mon, 24 Jan 2011 13:03:25 -0800 (PST) [thread overview]
Message-ID: <571697.98064.qm@web63303.mail.re1.yahoo.com> (raw)
--- Note ---
I'm sorry if you get this twice. I was sending this email from Yahoo, and I
guess they defaulted to HTML for the mail. My apologies if this was the case.
If not, please ignore this message. Thank you for your time.
--- Note ---
Hi,
First of all, let me thank all of you for creating such a wonderful product.
Rainbows is a unique solution and is the perfect candidate to solve our complex
problems. I don't know where this current project could possibly be without
your fine work. :-)
Now about the problem. First a little background on the architecture, so you
can get the context:
I'm dealing with some code that I can't just legally paste for example (although
I can probably make a simple proof of concept if needed) ... Here's the design:
I have a cascading long-poll connection, which listens for various JSON
messages. The throughput is quite low (a few a second) and I have a policy of
falling over based on either a large amount of traffic, or a predefined amount
of time (30 seconds) lapsing.
Essentially I have a 30 second connection and at second 25, a new one opens up
... the first one closes, and that one lasts for 30 seconds, etc.
This is designed for web browsers and it's implemented through hidden iframes.
This is a problem that has persisted across Firefox 3.6/OS X and Firefox 3.6/XP
and Firefox 4.0b6/XP along with IE8/XP and Chrome/OS X and FF 3.6/Ubuntu so I
don't think that any nuance of a browser or OS could be considered culpable.
I'm also not using any middle webserver like nginx and am connecting directly to
rainbows!
*** The Problem ***
When serving static files along with my long polled connections, I will get a
round trip time of ten millisecond or so, usually. This is perfectly
acceptable.
Every now and then, however, it will be about 2.5 seconds. This will then be
followed by a bunch of the snappy millisecond level transaction times. This is
one browser with 1 persistent connection against rainbows configured with 10
worker_processes and 100 worker_connections.
Everything changes about 5-10 minutes into things.
Then every transaction takes about 2-4 seconds. Static files that are 10 bytes
in size, 2-4 seconds. Ruby code to emit "Hello World"? 2-4 seconds. Every
request. Still using just 1 browser.
After I exit all browsers and then do a netstat on the client machine to see
that the connections have closed, I can then do a curl command for a static
file; again 2-4 seconds.
On the machine running rainbows if I do a netstat, I get this:
tcp 1 0 10.10.192.12:7788 10.10.131.165:17443 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17352 CLOSE_WAIT
tcp 1 196 10.10.192.12:7788 10.10.131.165:17317 CLOSE_WAIT
tcp 1 196 10.10.192.12:7788 10.10.131.165:17310 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17437 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17366 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17410 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17447 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17357 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17449 CLOSE_WAIT
tcp 1 0 10.10.192.12:7788 10.10.131.165:17347 CLOSE_WAIT
^^^ rainbows is running on 7788 on this machine ^^^.
The reference machine, in this case, windows, claims that all the connections
are closed:
Proto Local Address Foreign Address State
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:443 0.0.0.0:0 LISTENING
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:6060 0.0.0.0:0 LISTENING
TCP 0.0.0.0:63508 0.0.0.0:0 LISTENING
TCP 10.10.131.165:139 0.0.0.0:0 LISTENING
TCP 10.10.131.165:16841 187.39.33.180:11827 ESTABLISHED
TCP 10.10.131.165:16844 69.63.181.105:5222 ESTABLISHED
TCP 10.10.131.165:16849 10.10.10.71:5222 ESTABLISHED
TCP 10.10.131.165:16883 64.12.29.50:443 ESTABLISHED
TCP 10.10.131.165:16904 64.12.28.222:443 ESTABLISHED
TCP 10.10.131.165:16915 64.12.165.99:443 ESTABLISHED
TCP 10.10.131.165:16918 64.12.202.37:443 ESTABLISHED
TCP 10.10.131.165:16958 205.188.248.151:443 ESTABLISHED
TCP 10.10.131.165:16961 205.188.254.83:443 ESTABLISHED
TCP 10.10.131.165:17102 10.10.131.136:22 ESTABLISHED
TCP 10.10.131.165:17466 10.10.192.12:22 ESTABLISHED
TCP 10.10.131.165:17470 10.0.0.29:515 SYN_SENT
TCP 127.0.0.1:1030 0.0.0.0:0 LISTENING
TCP 192.168.56.1:139 0.0.0.0:0 LISTENING
TCP [::]:135 [::]:0 LISTENING 0
I can disconnect the windows client machine; turn it off even, and this problem
persists.
I think that somewhere in the ruby stack, the connections are not closing. If I
increase my worker_process count and prolong the long poll, then yes, I'll
survive for 15 minutes instead of 5; but the problem will still eventually occur
and I will hit the wall.
I have yet to try to test unicorn or zbatery for this style of solution because
I need the keep-alive; and although I know that unicorn put in the keep-alive
support for rainbows, I haven't really taken the time necessary to know how to
invoke it. If you think this would be instructive, I'd be happy to do so.
As far as my ruby set-up, I'm using ruby1.8 and I have the ThreadSpawn model.
We are running on a modern version of Ubuntu without any serious customization.
If you think that another configuration would do the trick or if you know how to
squash this bug, it would be
very helpful. This problem has become of great concern for us. We love
rainbows and all that it is. Thanks for the project and keep up the good work.
Cheers,
~chris.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
next reply other threads:[~2011-01-24 21:18 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-24 21:03 chris mckenzie [this message]
[not found] ` <571697.98064.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-24 21:54 ` Page request roundtrip time increases substantially after a bit of use Eric Wong
[not found] ` <20110124215440.GA25489-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 0:11 ` Eric Wong
[not found] ` <20110125001107.GA1921-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 1:14 ` chris mckenzie
[not found] ` <443004.19531.qm-9Fhxc66Fx5iB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25 2:02 ` Eric Wong
2011-01-25 2:55 ` chris mckenzie
[not found] ` <288407.18061.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25 3:50 ` Eric Wong
[not found] ` <20110125035048.GA8124-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 21:38 ` Eric Wong
[not found] ` <20110125213835.GA9421-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03 2:54 ` Eric Wong
[not found] ` <20110203025415.GA3812-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03 4:44 ` chris mckenzie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://yhbt.net/rainbows/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=571697.98064.qm@web63303.mail.re1.yahoo.com \
--to=kristopolous-/e1597as9lqavxtiumwx3w@public.gmane.org \
--cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
http://yhbt.net/rainbows.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).