From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: * X-Spam-ASN: AS33070 50.56.128.0/17 X-Spam-Status: No, score=1.0 required=3.0 tests=AWL,HK_RANDOM_FROM, MSGID_FROM_MTA_HEADER,TVD_RCVD_IP shortcircuit=no autolearn=no version=3.3.2 Path: news.gmane.org!not-for-mail From: Eric Wong Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: negative timeout in Rainbows::Fiber::Base Date: Fri, 28 Sep 2012 19:11:32 +0000 Message-ID: <20120928191132.GA14292@dcvr.yhbt.net> References: <20120829211707.GA22726@dcvr.yhbt.net> <20120831013731.GA16613@dcvr.yhbt.net> <20120905232739.GA25153@dcvr.yhbt.net> <20120922194222.GA6839@dcvr.yhbt.net> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1348860360 14517 80.91.229.3 (28 Sep 2012 19:26:00 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 28 Sep 2012 19:26:00 +0000 (UTC) To: Rainbows! list Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Fri Sep 28 21:26:06 2012 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:410 Archived-At: Received: from 50-56-192-79.static.cloud-ips.com ([50.56.192.79] helo=rubyforge.org) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1THgCH-00034W-AT for gclrrg-rainbows-talk@m.gmane.org; Fri, 28 Sep 2012 21:26:01 +0200 Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 10FBF2E06D; Fri, 28 Sep 2012 19:25:55 +0000 (UTC) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by rubyforge.org (Postfix) with ESMTP id 8C88D2E067 for ; Fri, 28 Sep 2012 19:11:34 +0000 (UTC) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id B4C171F42C; Fri, 28 Sep 2012 19:11:32 +0000 (UTC) "Lin Jen-Shin (godfat)" wrote: > On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong wrote: > [...] > >> Moreover, once there are some assets timeout issues on EventMachine, > >> too. When I tried to debug this, I put some traces into Rainbows, > >> realizing that sometimes EventMachine didn't call `receive_data' > >> when receiving some pipelined requests. Could it be an eventmachine bug!? > > > > It could be the front-end proxy (incorrectly) detected the Rainbows! > > instance was down and stopped sending traffic to it. Does this > > information get logged? > > It would eventually show out as a timeout for 30 seconds. It's a log > from front-end proxy (router in their terminology). I am not sure > if this means it is sending traffic or not. If it's not, then I guess it > explains... > > > Yeah, it's a bit much to understand. Can you reproduce it consistently? > > Yes, it could be reproduced consistently, but only in a certain environment > (e.g. on Heroku) which might not always be found. > > > With the serving timeout for Zbatery+ThreadSpawn, can you ensure > > Content-Length/Transfer-Encoding:chunked is set in the response headers? > > I just realized that this is not a timeout issue here. (doh, too many strange > issues) I just tried to curl it, and it would immediately return an error. > > heroku[router]: Error H13 (Connection closed without response) -> GET > /assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1 > queue= wait= service= status=503 bytes= > > There is Content-Length: 98794 and no Transfer-Encoding. So the _actual_ Content-Length that's sent is zero? Rainbows! should drop a connection if an exception is raised while sending the response body, perhaps the heroku router is confused by that? > I don't see if there's any difference between using EventMachine :( > > By the way, weirdly that it seems there's no problems at all if we're > using Thin server. I guess they are testing against Thin, so Thin works > correctly... still can't I tell what's the difference. Are you allowed to "strace -f" Thin and Rainbows! in an isolated (single thread/process instance)? I'd love to see the difference between how the servers handle this case differently... Can you try setting "keepalive_timeout 0" to disable persistent connections in Rainbows! and see if this helps? I don't think persistent connections make a big difference for LAN, especially not with typical HTTP response sizes (your 98K is pretty big :) > > Since you mentioned stack overflows in response generation, perhaps > > whatever proxy Heroku is using doesn't handle crashed servers during > > the response correctly... > > Probably. And I think I am 80% sure where it causes stack overflows now. > If I took out fibers, then it would be ok. So I guess that assets things are > using too much stack. > > Too bad I can't switch to threads though :( If I switched to threads, > ActiveRecord would be complaining it could not get a connection from > the connection pool within 5 seconds. It's not convenient to increase the > size of connection pool on Heroku, either, and making it too large would > also cause other issues. > > (sigh) > > > Can you get stderr logs from Heroku? > > Yes, Heroku would redirect both stdout and stderr to a place where we > could see. It's collapsed into one huge log though. > > > I highly doubt nginx will pipeline requests, but we're not sure if > > they're really using nginx, yet. With the problems you've described, > > it doesn't sound like they are, or they're using some broken version > > of it. > > Umm... after reading the log, I think they are using another [router] > (in their terminology) in front of [nginx]. So it might be an issue in > their [router], I am not sure... > > I'll keep you posted if you're interested. Thanks for all your help. Yes, please. > p.s. They might be using https://github.com/mochi/mochiweb > too, since if there's an issue, it would return this header: > Server: MochiWeb/1.0 (Any of you quaids got a smint?) > > p.s.2. Perhaps I could even give you access to one of the apps. > I'll need to ask the owner though. Let me know if you're interested, > otherwise you could simply ignore this. I know it's too much to ask. I think I can just walk you through the troubleshooting process here. (I'd rather avoid the potential for personal liability if given access to a server :) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying