Rainbows! Rack HTTP server user/dev discussion
 help / color / mirror / code / Atom feed
From: "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org>
To: "Rainbows! list" <rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org>
Subject: Re: negative timeout in Rainbows::Fiber::Base
Date: Sat, 22 Sep 2012 17:52:11 +0800	[thread overview]
Message-ID: <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw@mail.gmail.com> (raw)
In-Reply-To: <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>

On Thu, Sep 6, 2012 at 7:27 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Simple: we don't read from the socket at all while processing a request
> Extra data the client sends gets buffered in the kernel, eventually TCP
> backoff will kick in and the Internet stays usable :>

Understood, thanks! I guess this is simple enough that it's a nice
trade off, since not every clients would do heavy pipelining.

> With the inability to easily stop read callbacks via EM, the socket
> buffers constantly get drained so the clients are able to keep sending
> data.  The only option I've found for Rainbows! + EM was to issue
> shutdown(SHUT_RD) if a client attempts to pipeline too much (which
> never happens with legitimate clients AFAIK).

And the buffer size you set: 0x1c000, I guess that's large enough
for most cases.

> It shouldn't matter for nginx, I don't think nginx will (ever) pipeline
> to a backend.  Nowadays nginx can do persistent connections to backends,
> though I'm not sure how much of a benefit it is for local sockets.

Here's a problem troubles me for a long time, and I failed to figure out
what's going on and cannot find a proper fix. I didn't mention this before,
because I don't think it's Zbatery/Rainbows' issue, but it turns out that
it might be related, and perhaps you would be interested to provide
some hints, I would much appreciate it.

So we're running all our apps on http://www.heroku.com
It's a hosting service that you simply `git push` to deploy your app,
worrying not all the complexity regarding deployment.

There's a very weird issue running Zbatery/Rainbows on it.
I don't remember if Unicorn would be suffering from that
since I don't run Unicorn directly for a while now.

The problem comes with Rails 3 and its "assets pipeline" feature.
It's basically a feature that let your assets (e.g. images, javascripts,
css, etc) no longer simple assets which could be served directly
via nginx or some other static web server.

Like, developers could name their coffeescripts files as:


And when a client is requesting "application.js", that "assets pipeline"
thing would compile the coffeescript file into a javascript file on the fly,
and return it to the client.

We can also "precompile" those assets before deploying to production.
Heroku would do this if it detects that the application is deploying is
built with Rails 3.

If this precompile thing is done successfully, then there's no problems,
at least in some of the cases.... But what if precompile failed? In my cases,
it would usually cause system stack overflow whenever it tries to compile
it on the fly.

Oh, suddenly I think maybe it's because I would wrap a fiber around
each request in some cases... and it just used too many method calls.
I am not sure about that... might check it next time I saw it.

Somehow I feel this might be related to persistent connections or
pipelined requests, because last time I didn't write the client class
correctly to handle pipelined requests as you pointed out, it would
also cause some issues regarding "assets pipeline" on Heroku.

I believe Heroku is running nginx in front off Zbatery, and if I didn't
get pipelined requests correctly, it would result timeout for many of
assets the client is asking about. I feel Heroku is using pipelined
requests if the client is using this.

And the most weird thing I can't understand is, I am trying to convince
one of my friend to try out Zbatery, but using ThreadSpawn model
would usually cause serving assets timeout, just like in my case
where I didn't get pipelined requests correct. Switching to EventMachine
solved the problem! Huh?

Moreover, once there are some assets timeout issues on EventMachine,
too. When I tried to debug this, I put some traces into Rainbows,
realizing that sometimes EventMachine didn't call `receive_data'
when receiving some pipelined requests. Could it be an eventmachine bug!?

Sorry that I wrote so much and it's unclear what's going on, since
I don't understand so I can't remember correctly...

If you have any idea for a specific case, I can try to provide all the
information I could collect. I can't get the nginx configuration Heroku
is using though :(

> I think your comment is unfortunately representative of a lot of
> software development nowadays.
> Embrace pessimism and let it be your guide :)

I hope I could understand/realize this sooner :)

Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
Do not quote signatures (like this one) or top post when replying

  parent reply	other threads:[~2012-09-22  9:52 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-08-23 20:36 negative timeout in Rainbows::Fiber::Base Lin Jen-Shin (godfat)
     [not found] ` <CAA2_N1unOXb7Z4Jr8oKoSLu266O9Ko4o=oWzAcMA1w3=9X74KA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-25  2:45   ` Eric Wong
     [not found]     ` <20120825024556.GA25977-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-08-26  0:12       ` Lin Jen-Shin (godfat)
     [not found]         ` <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-26  1:15           ` Eric Wong
2012-08-29 16:00           ` Lin Jen-Shin (godfat)
     [not found]             ` <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-29 21:17               ` Eric Wong
     [not found]                 ` <20120829211707.GA22726-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-08-30 21:33                   ` Lin Jen-Shin (godfat)
     [not found]                     ` <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-31  1:37                       ` Eric Wong
     [not found]                         ` <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-05 20:06                           ` Lin Jen-Shin (godfat)
     [not found]                             ` <CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-05 23:27                               ` Eric Wong
     [not found]                                 ` <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-22  9:52                                   ` Lin Jen-Shin (godfat) [this message]
     [not found]                                     ` <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-22 19:42                                       ` Eric Wong
     [not found]                                         ` <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-28 15:14                                           ` Lin Jen-Shin (godfat)
     [not found]                                             ` <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-28 19:11                                               ` Eric Wong
     [not found]                                                 ` <20120928191132.GA14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-28 19:24                                                   ` Eric Wong
     [not found]                                                     ` <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-10-31  0:14                                                       ` Lin Jen-Shin (godfat)
2012-12-18 11:09                                                       ` Lin Jen-Shin (godfat)
     [not found]                                                         ` <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-12-18 19:19                                                           ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://yhbt.net/rainbows/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw@mail.gmail.com' \
    --to=godfat-hoe/xeebyyidnm+yrofe0a@public.gmane.org \
    --cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).