From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.6 required=5.0 tests=AWL,MSGID_FROM_MTA_HEADER, NORMAL_HTTP_TO_IP,RP_MATCHES_RCVD,WEIRD_PORT shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: John-Paul Bader Newsgroups: gmane.comp.lang.ruby.unicorn.general Subject: Re: "upstream timed out" after upgrades Date: Thu, 4 Feb 2010 09:48:50 +0100 Message-ID: <5D2B92C5-8D0A-4810-B86B-06F670C6EFDB@h3q.com> References: NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1265273965 25498 80.91.229.12 (4 Feb 2010 08:59:25 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 4 Feb 2010 08:59:25 +0000 (UTC) To: unicorn list Original-X-From: mongrel-unicorn-bounces@rubyforge.org Thu Feb 04 09:59:21 2010 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org X-Greylist: delayed 398 seconds by postgrey-1.31 at rubyforge.org; Thu, 04 Feb 2010 03:55:33 EST In-Reply-To: X-Mailer: Apple Mail (2.1077) X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:384 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1NcxYV-0004wC-Nh for gclrug-mongrel-unicorn@m.gmane.org; Thu, 04 Feb 2010 09:59:20 +0100 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 1559D1588033; Thu, 4 Feb 2010 03:59:19 -0500 (EST) Received: from mail.h3q.com (mail.h3q.com [213.73.89.199]) by rubyforge.org (Postfix) with ESMTP id 4399A18582E3 for ; Thu, 4 Feb 2010 03:55:33 -0500 (EST) Received: (qmail 70869 invoked from network); 4 Feb 2010 08:48:51 -0000 Received: from mail.h3q.com (HELO mail.h3q.com) (hukl) by mail.h3q.com with AES128-SHA encrypted SMTP; 4 Feb 2010 08:48:51 -0000 Hrm its getting weirder and weirder. I started unicorn_rails now manually in the app root which is now running o= n rails 2.3.3 again: = sudo unicorn_rails -E production -d -c /usr/local/etc/unicorn.rb Now first of all it throws lots of Exceptions it couldn't find several (imp= ortant) gems and then it produces this every second or so: Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2) When I request the javascript in question I get: Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn/http_response.rb:67 - Invalid argument Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2) This is the same no matter if I run it with -c or not. = I am puzzled. To give you the entire output of unicorn starting: [hukl@cccms /usr/local/www/cccms]$ sudo unicorn_rails -E production -d -p 9= 090 {:daemonize=3D>false, :unicorn_options=3D>{:listeners=3D>["0.0.0.0:9090"]}, :app=3D> #} Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - Fil= e exists - tmp/cache Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - Fil= e exists - tmp/pids Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - Fil= e exists - tmp/sessions Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - Fil= e exists - tmp/sockets I, [2010-02-04T09:42:08.141034 #38499] INFO -- : listening on addr=3D213.7= 3.89.122:9090 fd=3D3 Exception `LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_= require.rb:31 - no such file to load -- Win32API Exception `LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_= require.rb:38 - no such file to load -- Win32API I, [2010-02-04T09:42:08.158275 #38499] INFO -- : worker=3D0 spawning... I, [2010-02-04T09:42:08.159352 #38499] INFO -- : master process ready I, [2010-02-04T09:42:08.159471 #38501] INFO -- : worker=3D0 spawned pid=3D= 38501 I, [2010-02-04T09:42:08.197082 #38501] INFO -- : Refreshing Gem list Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb= :827 - Could not find RubyGem builder (~> 2.1.2) Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb= :827 - Could not find RubyGem memcache-client (>=3D 1.7.4) Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb= :827 - Could not find RubyGem tzinfo (~> 0.3.12) Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb= :827 - Could not find RubyGem i18n (~> 0.1.3) Exception `TypeError' at (eval):4 - can't modify frozen object Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:31 - no such file to load -- fast_xs Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:31 - no such file to load -- json Using c extension for JSON. Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb= :827 - Could not find RubyGem tmail (~> 1.2.3) Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:31 - no such file to load -- tmail/tmailscanner Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:38 - no such file to load -- tmail/tmailscanner Exception `MissingSourceFile' at /usr/local/lib/ruby/gems/1.8/gems/activesu= pport-2.3.3/lib/active_support/dependencies.rb:162 - no such file to load -= - tmail/tmailscanner Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb= :827 - Could not find RubyGem activerecord-postgresql-adapter (>=3D 0) Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:31 - no such file to load -- pg WARNING: Nokogiri was built against LibXML version 2.7.3, but has dynamical= ly loaded 2.7.6 Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:31 - no such file to load -- ruby-debug Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:31 - no such file to load -- classtree Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:38 - no such file to load -- classtree Exception `MissingSourceFile' at /usr/local/lib/ruby/gems/1.8/gems/activesu= pport-2.3.3/lib/active_support/dependencies.rb:162 - no such file to load -= - classtree Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:31 - no such file to load -- methodsig Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems= /custom_require.rb:38 - no such file to load -- methodsig Exception `MissingSourceFile' at /usr/local/lib/ruby/gems/1.8/gems/activesu= pport-2.3.3/lib/active_support/dependencies.rb:162 - no such file to load -= - methodsig =3D> Debugger enabled worker=3D0 ready Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2) Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn/http_response.rb:67 - Invalid argument Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2) Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn/http_response.rb:67 - Invalid argument Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2) Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn/http_response.rb:67 - Invalid argument Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95= .3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2) I seriously have no idea went wrong - It ran all so smooth couple of days a= go. What am I missing - there must be something plain wrong. I left the js = files out of the public site for now - it isn't needed that much and its mo= re important to serve the content for now. In case you are trying to get it. Kind regards, John On 04.02.2010, at 09:16, John-Paul Bader wrote: > Hey guys, > = > = > i recently updated several components of my website setup: = > = > varnish 2.0.5 -> 2.0.6 > nginx 7.4.2 -> 7.4.5 > unicorn 0.95.3 -> 0.96 > ruby18 to the latest patch level = > and rails from 2.3.2 > = > It all seemed to work great but recently I noticed that the website hangs= loading. The weird thing is that it hangs because 2 or 3 javascript files = take about a minute to finish and when they do they are incomplete as in cu= t of in the middle of the content. The nginx error log gives the following = output every few seconds: > = > 2010/02/04 08:58:41 [error] 29389#0: *471 upstream timed out (60: Operati= on timed out) while reading upstream, client: 213.73.89.122, server: www.cc= c.de, request: "GET /javascripts/jquery-1.3.2.min.js?1246657683 HTTP/1.1", = upstream: "http://unix:/var/run/unicorn.sock:/javascripts/jquery-1.3.2.min.= js?1246657683", host: "www.ccc.de" > = > I don't think its varnishs fault because i get Internal Server Errors (50= 0) if I run curl on the machine against nginx, not at first though - severa= l requests for that jquery file run just fine and then suddenly i get a 500= on a request with the content being cut of in the middle. > = > Next thing I did was configuring unicorn to listen on 213.73.89.122:9090 = rather than on a socket so i could run curl directly against it and even af= ter a fresh start of unicorn gives me: > = > = > curl http://213.73.89.122:9090/javascripts/jquery-1.3.2.min.js > (lots of javascript)=85,rowspan:"rowSpan",tabindex:"tabIndHTTP/1.1 500 In= ternal Server Error > = > curl: (18) transfer closed with 16256 bytes remaining to read > = > I also downgraded back to 0.95.3 but that didn't change anything. So now = after 8h of try and error I'm out of ideas and would be happy to hear some = suggestions. > = > Kind regards, John > _______________________________________________ > Unicorn mailing list - mongrel-unicorn@rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying > = _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying