From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.5 required=5.0 tests=AWL,MSGID_FROM_MTA_HEADER, RP_MATCHES_RCVD shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Eric Wong Newsgroups: gmane.comp.lang.ruby.unicorn.general Subject: Re: 502 bad gateway on nginx with recv() failed Date: Sat, 23 Oct 2010 16:22:31 -0700 Message-ID: <20101023232231.GC11427@dcvr.yhbt.net> References: <20101022211446.GA22603@dcvr.yhbt.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1287877231 10905 80.91.229.12 (23 Oct 2010 23:40:31 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sat, 23 Oct 2010 23:40:31 +0000 (UTC) To: unicorn list Original-X-From: mongrel-unicorn-bounces@rubyforge.org Sun Oct 24 01:40:27 2010 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:736 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1P9nhJ-0002Mo-Ef for gclrug-mongrel-unicorn@m.gmane.org; Sun, 24 Oct 2010 01:40:25 +0200 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id B87ED19783EC; Sat, 23 Oct 2010 19:40:22 -0400 (EDT) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by rubyforge.org (Postfix) with ESMTP id 174A119783CD for ; Sat, 23 Oct 2010 19:22:32 -0400 (EDT) Received: from localhost (unknown [127.0.2.5]) by dcvr.yhbt.net (Postfix) with ESMTP id 1F4721F78C; Sat, 23 Oct 2010 23:22:32 +0000 (UTC) Naresh V wrote: > On 23 October 2010 02:44, Eric Wong wrote: > > Naresh V wrote: > >> I'm serving the puppetmaster application with its config.ru through > >> unicorn - proxied by nginx. > >> I'm using unix sockets, 4 workers, and 2048 backlog. > >> > >> The clients - after their typical "puppet run" - send back a report to > >> the master in YAML. > >> Some clients whose reports tend to be large (close to 2mb) get a 502 > >> bad gateway error and error out. > >> > >> nginx log: > >> > >> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104: > >> Connection reset by peer) while reading response header from upstream, > >> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios > >> HTTP/1.1", upstream: > >> "http://unix:/tmp/.sock:/production/report/nagios", host: > >> "puppet:8140" > > > > Hi Naresh, do you see anything in the Unicorn stderr log file? > = > Hi Eric, I think I caught it: > = > E, [2010-10-22T23:03:30.207455 #10184] ERROR -- : worker=3D2 PID:10206 > timeout (60.207392s > 60s), killing > I, [2010-10-22T23:03:31.212533 #10184] INFO -- : reaped > # worker=3D2 > I, [2010-10-22T23:03:31.214768 #10490] INFO -- : worker=3D2 spawned pid= =3D10490 > I, [2010-10-22T23:03:31.221748 #10490] INFO -- : worker=3D2 ready > = > > Is the 2mb report part of the response or request? =A0Unicorn should > > have no problems accepting large requests (Rainbows! defaults the > > client_max_body_size to 1mb, just like nginx). > = > It's part of the PUT request, I guess. > = > > It could be Unicorn's internal (default 60s) timeout kicking > > in because puppet is slowly reading/generating the 2mb body. > = > I raised the timeout first to 120, then 180 - and I continued to get > the 502 (with the logs as above) > When I raised it upto 240, puppetd complained: Interesting. I'm not familiar with Puppet internals, but is there any valid reason it would be taking this long? Can you tell if the Unicorn worker is doing anything (using up CPU time in top) or just blocked on some socket connection to a database or DNS? (strace/truss will help). You should definitely talk to Puppet developers/users about why it's taking so long. HTTP requests taking anywhere near 60s is an eternity, I wonder if your Puppet is somehow misconfigured. -- = Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying