Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> Here's a problem troubles me for a long time, and I failed to figure out
> what's going on and cannot find a proper fix. I didn't mention this before,
> because I don't think it's Zbatery/Rainbows' issue, but it turns out that
> it might be related, and perhaps you would be interested to provide
> some hints, I would much appreciate it.
> 
> So we're running all our apps on http://www.heroku.com
> It's a hosting service that you simply `git push` to deploy your app,
> worrying not all the complexity regarding deployment.
> 
> There's a very weird issue running Zbatery/Rainbows on it.
> I don't remember if Unicorn would be suffering from that
> since I don't run Unicorn directly for a while now.
> 
> The problem comes with Rails 3 and its "assets pipeline" feature.
> It's basically a feature that let your assets (e.g. images, javascripts,
> css, etc) no longer simple assets which could be served directly
> via nginx or some other static web server.
> 
> Like, developers could name their coffeescripts files as:
> 
> application.coffee.js
> 
> And when a client is requesting "application.js", that "assets pipeline"
> thing would compile the coffeescript file into a javascript file on the fly,
> and return it to the client.
> 
> We can also "precompile" those assets before deploying to production.
> Heroku would do this if it detects that the application is deploying is
> built with Rails 3.
> 
> If this precompile thing is done successfully, then there's no problems,
> at least in some of the cases.... But what if precompile failed? In my cases,
> it would usually cause system stack overflow whenever it tries to compile
> it on the fly.
> 
> Oh, suddenly I think maybe it's because I would wrap a fiber around
> each request in some cases... and it just used too many method calls.
> I am not sure about that... might check it next time I saw it.
> 
> Somehow I feel this might be related to persistent connections or
> pipelined requests, because last time I didn't write the client class
> correctly to handle pipelined requests as you pointed out, it would
> also cause some issues regarding "assets pipeline" on Heroku.
> 
> I believe Heroku is running nginx in front off Zbatery, and if I didn't
> get pipelined requests correctly, it would result timeout for many of
> assets the client is asking about. I feel Heroku is using pipelined
> requests if the client is using this.
> 
> And the most weird thing I can't understand is, I am trying to convince
> one of my friend to try out Zbatery, but using ThreadSpawn model
> would usually cause serving assets timeout, just like in my case
> where I didn't get pipelined requests correct. Switching to EventMachine
> solved the problem! Huh?
> 
> Moreover, once there are some assets timeout issues on EventMachine,
> too. When I tried to debug this, I put some traces into Rainbows,
> realizing that sometimes EventMachine didn't call `receive_data'
> when receiving some pipelined requests. Could it be an eventmachine bug!?

It could be the front-end proxy (incorrectly) detected the Rainbows!
instance was down and stopped sending traffic to it.  Does this
information get logged?

> Sorry that I wrote so much and it's unclear what's going on, since
> I don't understand so I can't remember correctly...

Yeah, it's a bit much to understand.  Can you reproduce it consistently?

With the serving timeout for Zbatery+ThreadSpawn, can you ensure
Content-Length/Transfer-Encoding:chunked is set in the response headers?

Since you mentioned stack overflows in response generation, perhaps
whatever proxy Heroku is using doesn't handle crashed servers during
the response correctly...

Can you get stderr logs from Heroku?

> If you have any idea for a specific case, I can try to provide all the
> information I could collect. I can't get the nginx configuration Heroku
> is using though :(

I highly doubt nginx will pipeline requests, but we're not sure if
they're really using nginx, yet.  With the problems you've described,
it doesn't sound like they are, or they're using some broken version
of it.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying