Rainbows! Rack HTTP server user/dev discussion
 help / color / mirror / code / Atom feed
From: Ilya Maykov <ivmaykov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
Subject: Rainbows! + EventMachine + Sinatra::Synchrony == pegged CPU when idle?
Date: Tue, 19 Jun 2012 02:06:19 -0700	[thread overview]
Message-ID: <CAAsVqN8=HxMbbQkfonojsoCHgJAPsDhC7hqqcBK9GnYsO2ww7g@mail.gmail.com> (raw)

Hi all,

We're using Rainbows + EventMachine + Sinatra::Synchrony to run a
fleet of RESTful web servers backed by a Cassandra cluster. We are
using the EventMachineTransport to talk to Cassandra with an
EM::Synchrony::ConnectionPool in each rainbows worker. We have a Storm
cluster pushing a large stream of real-time data into the Rainbows
fleet using HTTP PUT requests. We're running into some very strange
performance issues and need help figuring out what's going on.

Basically, when load is low, everything looks good. When we crank up
the load, all of a sudden the CPU gets pegged, request latencies go
waaaay up, and requests start timing out. Once this state is reached,
the high CPU usage (4 rainbows worker processes at ~50% each on a
2-core machine = nearly full load) remains even if we completely shut
off all incoming traffic. Taking a look with strace -p, it looks like
the rainbows processes are writing ascii NUL characters to file
descriptor 7 (which is a FIFO) as fast as the kernel will let them. My
guess is that the worker is trying to communicate with the rainbows
master process via the FIFO. Not sure what is triggering this
behavior, but would like to know if anyone else has ever seen
something like this. This thread sounded like it could've been a
similar issue, but died out without any conclusion:

Some details about the setup:

6-node cassandra cluster
3 nodes running rainbows web servers
4 rainbows workers per node
max of 50 cassandra connections per rainbows worker
rainbows.conf has:

Rainbows! do
  use :EventMachine
  worker_connections 50
  keepalive_requests 1000
  keepalive_timeout  10

So, each rainbows node can handle 4 * 50 = 200 simultaneous connections

12 Storm worker processes writing to the rainbows web servers
each Storm worker has max of 10 connections open to each of the 3 rainbows nodes

So, each rainbows node has 12 * 10 = 120 incoming connections from Storm.

Have been playing around with the numbers, the bug (assuming it is a
bug) seems to be easier to trigger when I increase the number of
incoming connections (from Storm workers), even if they are a lot less
than the rainbows servers can take (60-70% of the max connections is
usually enough). The bug is also easier to trigger when we increase
the volume of data we're pushing through Storm - hundreds or thousands
of requests per minute, no bug - hundreds of thousands of requests per
minute, yes bug. Cassandra is not the issue, it can easily take the
write load we're generating and is basically idle.

Any help in figuring this out would be greatly appreciated. Thanks,

-- Ilya
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
Do not quote signatures (like this one) or top post when replying

             reply	other threads:[~2012-06-19  9:06 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-06-19  9:06 Ilya Maykov [this message]
     [not found] ` <CAAsVqN8=HxMbbQkfonojsoCHgJAPsDhC7hqqcBK9GnYsO2ww7g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-06-19 17:54   ` Rainbows! + EventMachine + Sinatra::Synchrony == pegged CPU when idle? Eric Wong
     [not found]     ` <20120619175409.GA27303-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-07-23 23:34       ` Ilya Maykov
     [not found]         ` <CAAsVqN91_KiexPc4u9byuVstyknj7TgT4V7evs1x65W+6izzkQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-07-24  0:21           ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://yhbt.net/rainbows/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAAsVqN8=HxMbbQkfonojsoCHgJAPsDhC7hqqcBK9GnYsO2ww7g@mail.gmail.com' \
    --to=ivmaykov-re5jqeeqqe8avxtiumwx3w@public.gmane.org \
    --cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).