Rainbows! Rack HTTP server user/dev discussion
 help / color / mirror / code / Atom feed
From: "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org>
To: "Rainbows! list" <rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org>
Subject: Re: negative timeout in Rainbows::Fiber::Base
Date: Fri, 28 Sep 2012 23:14:33 +0800	[thread overview]
Message-ID: <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw@mail.gmail.com> (raw)
In-Reply-To: <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>

On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
>> Moreover, once there are some assets timeout issues on EventMachine,
>> too. When I tried to debug this, I put some traces into Rainbows,
>> realizing that sometimes EventMachine didn't call `receive_data'
>> when receiving some pipelined requests. Could it be an eventmachine bug!?
> It could be the front-end proxy (incorrectly) detected the Rainbows!
> instance was down and stopped sending traffic to it.  Does this
> information get logged?

It would eventually show out as a timeout for 30 seconds. It's a log
from front-end proxy (router in their terminology). I am not sure
if this means it is sending traffic or not. If it's not, then I guess it

> Yeah, it's a bit much to understand.  Can you reproduce it consistently?

Yes, it could be reproduced consistently, but only in a certain environment
(e.g. on Heroku) which might not always be found.

> With the serving timeout for Zbatery+ThreadSpawn, can you ensure
> Content-Length/Transfer-Encoding:chunked is set in the response headers?

I just realized that this is not a timeout issue here. (doh, too many strange
issues) I just tried to curl it, and it would immediately return an error.

heroku[router]: Error H13 (Connection closed without response) -> GET
/assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1
queue= wait= service= status=503 bytes=

There is Content-Length: 98794 and no Transfer-Encoding.

I don't see if there's any difference between using EventMachine :(

By the way, weirdly that it seems there's no problems at all if we're
using Thin server. I guess they are testing against Thin, so Thin works
correctly... still can't I tell what's the difference.

> Since you mentioned stack overflows in response generation, perhaps
> whatever proxy Heroku is using doesn't handle crashed servers during
> the response correctly...

Probably. And I think I am 80% sure where it causes stack overflows now.
If I took out fibers, then it would be ok. So I guess that assets things are
using too much stack.

Too bad I can't switch to threads though :( If I switched to threads,
ActiveRecord would be complaining it could not get a connection from
the connection pool within 5 seconds. It's not convenient to increase the
size of connection pool on Heroku, either, and making it too large would
also cause other issues.


> Can you get stderr logs from Heroku?

Yes, Heroku would redirect both stdout and stderr to a place where we
could see. It's collapsed into one huge log though.

> I highly doubt nginx will pipeline requests, but we're not sure if
> they're really using nginx, yet.  With the problems you've described,
> it doesn't sound like they are, or they're using some broken version
> of it.

Umm... after reading the log, I think they are using another [router]
(in their terminology) in front of [nginx]. So it might be an issue in
their [router], I am not sure...

I'll keep you posted if you're interested. Thanks for all your help.

p.s. They might be using https://github.com/mochi/mochiweb
too, since if there's an issue, it would return this header:
Server: MochiWeb/1.0 (Any of you quaids got a smint?)

p.s.2. Perhaps I could even give you access to one of the apps.
I'll need to ask the owner though. Let me know if you're interested,
otherwise you could simply ignore this. I know it's too much to ask.
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
Do not quote signatures (like this one) or top post when replying

  parent reply	other threads:[~2012-09-28 15:26 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-08-23 20:36 negative timeout in Rainbows::Fiber::Base Lin Jen-Shin (godfat)
     [not found] ` <CAA2_N1unOXb7Z4Jr8oKoSLu266O9Ko4o=oWzAcMA1w3=9X74KA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-25  2:45   ` Eric Wong
     [not found]     ` <20120825024556.GA25977-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-08-26  0:12       ` Lin Jen-Shin (godfat)
     [not found]         ` <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-26  1:15           ` Eric Wong
2012-08-29 16:00           ` Lin Jen-Shin (godfat)
     [not found]             ` <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-29 21:17               ` Eric Wong
     [not found]                 ` <20120829211707.GA22726-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-08-30 21:33                   ` Lin Jen-Shin (godfat)
     [not found]                     ` <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-31  1:37                       ` Eric Wong
     [not found]                         ` <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-05 20:06                           ` Lin Jen-Shin (godfat)
     [not found]                             ` <CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-05 23:27                               ` Eric Wong
     [not found]                                 ` <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-22  9:52                                   ` Lin Jen-Shin (godfat)
     [not found]                                     ` <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-22 19:42                                       ` Eric Wong
     [not found]                                         ` <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-28 15:14                                           ` Lin Jen-Shin (godfat) [this message]
     [not found]                                             ` <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-28 19:11                                               ` Eric Wong
     [not found]                                                 ` <20120928191132.GA14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-28 19:24                                                   ` Eric Wong
     [not found]                                                     ` <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-10-31  0:14                                                       ` Lin Jen-Shin (godfat)
2012-12-18 11:09                                                       ` Lin Jen-Shin (godfat)
     [not found]                                                         ` <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-12-18 19:19                                                           ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://yhbt.net/rainbows/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw@mail.gmail.com \
    --to=godfat-hoe/xeebyyidnm+yrofe0a@public.gmane.org \
    --cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).