From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.2 required=5.0 tests=AWL,RP_MATCHES_RCVD shortcircuit=no autolearn=unavailable version=3.3.2 X-Original-To: archivist@yhbt.net Delivered-To: archivist@dcvr.yhbt.net Received: from rubyforge.org (rubyforge.org [205.234.109.19]) by dcvr.yhbt.net (Postfix) with ESMTP id 30920211AF for ; Thu, 8 Dec 2011 19:39:24 +0000 (UTC) Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id A59BA1858386; Thu, 8 Dec 2011 14:39:23 -0500 (EST) X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org X-Greylist: delayed 1056 seconds by postgrey-1.31 at rubyforge.org; Thu, 08 Dec 2011 13:37:11 EST Received: from gardner.asmallorange.com (gardner.asmallorange.com [184.173.73.186]) by rubyforge.org (Postfix) with ESMTP id B01161678364 for ; Thu, 8 Dec 2011 13:37:09 -0500 (EST) Received: from mail-qw0-f43.google.com ([209.85.216.43]) by gardner.asmallorange.com with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.69) (envelope-from ) id 1RYiZ8-004BZB-09 for mongrel-unicorn@rubyforge.org; Thu, 08 Dec 2011 13:19:30 -0500 Received: by qabg27 with SMTP id g27so818744qab.2 for ; Thu, 08 Dec 2011 10:19:27 -0800 (PST) Received: by 10.224.116.144 with SMTP id m16mr4138684qaq.19.1323368367275; Thu, 08 Dec 2011 10:19:27 -0800 (PST) MIME-Version: 1.0 Received: by 10.229.10.199 with HTTP; Thu, 8 Dec 2011 10:19:06 -0800 (PST) From: Jonathan del Strother Date: Thu, 8 Dec 2011 18:19:06 +0000 Message-ID: Subject: Master repeatedly killing workers due to timeouts To: mongrel-unicorn@rubyforge.org X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gardner.asmallorange.com X-AntiAbuse: Original Domain - rubyforge.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - steelskies.com X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Hi, We're using unicorn as a Rails server on Solaris, and it's been running great for several months. We've recently been having a few problems and I'm at a loss what might cause it. A number of times in the past few days, our unicorn slaves keep timing out & the master keeps restarting them. unicorn.log looks something like : E, [2011-12-08T18:11:32.912237 #26661] ERROR -- : worker=5 PID:15367 timeout (61s > 60s), killing E, [2011-12-08T18:11:32.952041 #26661] ERROR -- : reaped # worker=5 I, [2011-12-08T18:11:32.985925 #17824] INFO -- : worker=5 ready E, [2011-12-08T18:11:42.962869 #26661] ERROR -- : worker=3 PID:15499 timeout (61s > 60s), killing E, [2011-12-08T18:11:43.003741 #26661] ERROR -- : reaped # worker=3 I, [2011-12-08T18:11:43.043797 #18263] INFO -- : worker=3 ready E, [2011-12-08T18:11:51.016129 #26661] ERROR -- : worker=0 PID:16072 timeout (61s > 60s), killing E, [2011-12-08T18:11:51.053344 #26661] ERROR -- : reaped # worker=0 I, [2011-12-08T18:11:51.084365 #18532] INFO -- : worker=0 ready E, [2011-12-08T18:11:53.063737 #26661] ERROR -- : worker=4 PID:16126 timeout (61s > 60s), killing E, [2011-12-08T18:11:53.117573 #26661] ERROR -- : reaped # worker=4 I, [2011-12-08T18:11:53.154511 #18577] INFO -- : worker=4 ready E, [2011-12-08T18:11:59.130075 #26661] ERROR -- : worker=6 PID:16165 timeout (61s > 60s), killing E, [2011-12-08T18:11:59.176834 #26661] ERROR -- : reaped # worker=6 I, [2011-12-08T18:11:59.220115 #18814] INFO -- : worker=6 ready E, [2011-12-08T18:12:12.188372 #26661] ERROR -- : worker=7 PID:16931 timeout (61s > 60s), killing E, [2011-12-08T18:12:12.245647 #26661] ERROR -- : reaped # worker=7 I, [2011-12-08T18:12:12.282607 #19718] INFO -- : worker=7 ready E, [2011-12-08T18:12:29.256887 #26661] ERROR -- : worker=1 PID:17406 timeout (61s > 60s), killing E, [2011-12-08T18:12:29.302142 #26661] ERROR -- : reaped # worker=1 I, [2011-12-08T18:12:29.335366 #20731] INFO -- : worker=1 ready E, [2011-12-08T18:12:31.313734 #26661] ERROR -- : worker=2 PID:17659 timeout (61s > 60s), killing E, [2011-12-08T18:12:31.357066 #26661] ERROR -- : reaped # worker=2 I, [2011-12-08T18:12:31.390368 #20800] INFO -- : worker=2 ready - which seems like quite a lot of timeouts. However, our database connection, network, NFS etc all seem fine. There's no useful errors in the rails log files. The master still serves occasional requests, but drops a high percentage of them. Can anyone suggest where to look into this? I'm not even sure if that timeout is occurring during the initial fork of the worker process, or if it's timing out during a slow rails request. -Jonathan _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying