From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.6 required=5.0 tests=AWL,FREEMAIL_FROM, MSGID_FROM_MTA_HEADER,RP_MATCHES_RCVD,T_DKIM_INVALID shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Lawrence Pit Newsgroups: gmane.comp.lang.ruby.unicorn.general Subject: Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks Date: Fri, 16 Jul 2010 10:47:10 +1000 Message-ID: <4C3FAC0E.9000508@gmail.com> References: <20100713201945.GB16925@dcvr.yhbt.net> <4C3D0140.6080806@gmail.com> <20100714003853.GA30485@dcvr.yhbt.net> <4C3D1D7C.6010200@gmail.com> <20100714023409.GB31092@dcvr.yhbt.net> <4C3D607F.2080701@gmail.com> <40ECA3FF-C778-4198-B162-D2A12230D85E@tramchase.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1279241247 27803 80.91.229.12 (16 Jul 2010 00:47:27 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Fri, 16 Jul 2010 00:47:27 +0000 (UTC) To: unicorn list Original-X-From: mongrel-unicorn-bounces@rubyforge.org Fri Jul 16 02:47:25 2010 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=1C1eXyiedxOYdbhf70GrI3UnJ6+CSmynbSQA903n0Jw=; b=oKiV4P2mZXeYI5VwLgWnGN9dTTQtNIpgPI2eNODsnSzC3CIkun1bhtNueVqVd1S67O zurdAZnZNGmXJqfgf2pnKrIQxG6dr1Y35r4LTZEIOmJstiUxvEHxFN6D8vWeLzWpSfVb 98qEts9jJ6ZDZ+LF5ogrLSpRj0rSZrCz8Phis= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=R3OqdmxyyqSDYSe6LIrzEAPyFBteEFvWBKRE1FdwARUiDEIu6+ShnMLMzkvZzkiBDZ bEW9uO68bNHdxL9MkYBIJ4BZVbu+0eXx5zDNVV4iAxot/4PX4isEnbjGB/UTmTgFB1kp OuSHIVX1Fbv0g9hdOH3yybM9ADXxYcan0rE0w= User-Agent: Thunderbird 2.0.0.24 (Macintosh/20100228) In-Reply-To: <40ECA3FF-C778-4198-B162-D2A12230D85E@tramchase.com> X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:651 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1OZZ5J-0007vO-2R for gclrug-mongrel-unicorn@m.gmane.org; Fri, 16 Jul 2010 02:47:25 +0200 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 5D944197828C; Thu, 15 Jul 2010 20:47:24 -0400 (EDT) Received: from mail-pv0-f178.google.com (mail-pv0-f178.google.com [74.125.83.178]) by rubyforge.org (Postfix) with ESMTP id D756A1858300 for ; Thu, 15 Jul 2010 20:47:16 -0400 (EDT) Received: by pva18 with SMTP id 18so716821pva.23 for ; Thu, 15 Jul 2010 17:47:15 -0700 (PDT) Received: by 10.114.81.11 with SMTP id e11mr364937wab.140.1279241235722; Thu, 15 Jul 2010 17:47:15 -0700 (PDT) Received: from copa.local ([124.149.45.23]) by mx.google.com with ESMTPS id s5sm12719371wak.12.2010.07.15.17.47.12 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 15 Jul 2010 17:47:14 -0700 (PDT) >> It appears unicorn rolls back by itself when it can't start the new master. Nice! >> >> Jamie also mentions to use the shared vendor/bundler_gems path. Though I do use that in all my projects I'm not so sure it's wise to do when you can't afford any downtime. I'll have to wait for unicorn v1.1.3 before I can test whether rolling back will indeed continue with unicorn v1.1.2 instead of v1.1.3 ;o >> > > Hi Lawrence, why does the shared vendor/bundler_gems cause you downtime? From not re-bundling during rollback? > > FWIW I made that recommendation because I ran into issues with unicorns not restarting correctly after running `cap deploy:cleanup`, since the `bundle exec` launches a binary with a path like /app/releases/201007125236/bin/unicorn, which goes missing after N deploys. I haven't tested if this applies to more recent versions > > Shared bundles are also significantly faster -- `bundle check || bundle install` is ~1s for me vs. several minutes to totally rebundle > Hi Jamie, I see what you mean. I haven't tested this yet, so I don't know for sure if it could cause downtime. I also always use bundle check || bundle install to get that time benefit. My worry was indeed that it might pick a newer unicorn to restart workers for the old master. I guess that can/should be prevented by having an on_rollback in my capistrano bundler task. What is the exact command you use to start unicorn? What I see is this, I start with: $ su -l deployer -c "cd /app/current ; bundle exec unicorn_rails -D -c /app/current/config/unicorn.rb" My after_fork and before_exec methods output the ENV vars to the unicorn log. I see for example: AFTER_FORK: VERSION 1 PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:.... GEM_HOME=/app/releases/20100714054705/vendor/bundler_gems GEM_PATH=/app/releases/20100714054705/vendor/bundler_gems:/app/releases/20100714054705/vendor/bundler_gems/bundler/gems BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile Note that /app/releases/20100714054705/vendor/bundler_gems is a symlink to /app/releases/shared/vendor/bundler_gems When I deploy a new version, symlink /app/current to the new release directory, and send USR2 : executing ["/app/releases/20100714054705/vendor/bundler_gems/bin/unicorn_rails", "-D", "-E", "staging", "-c", "/app/current/config/unicorn.rb"] (in /app/releases/20100714055624) BEFORE_EXEC: VERSION 1 PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:.... GEM_HOME=/app/releases/20100714054705/vendor/bundler_gems GEM_PATH=/app/releases/20100714054705/vendor/bundler_gems:/app/releases/20100714054705/vendor/bundler_gems/bundler/gems BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile I, [2010-07-15T23:31:47.765995 #23084] INFO -- : inherited addr=/tmp/.unicorn_sock fd=3 I, [2010-07-15T23:31:47.766646 #23084] INFO -- : Refreshing Gem list /app/releases/20100714055624/.bundle/environment.rb:175: warning: already initialized constant ENV_LOADED /app/releases/20100714055624/.bundle/environment.rb:176: warning: already initialized constant LOCKED_BY /app/releases/20100714055624/.bundle/environment.rb:177: warning: already initialized constant FINGERPRINT /app/releases/20100714055624/.bundle/environment.rb:178: warning: already initialized constant HOME /app/releases/20100714055624/.bundle/environment.rb:179: warning: already initialized constant AUTOREQUIRES /app/releases/20100714055624/.bundle/environment.rb:181: warning: already initialized constant SPECS AFTER_FORK: VERSION 2 PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:.... GEM_HOME=/app/releases/20100714055624/vendor/bundler_gems GEM_PATH=/app/releases/20100714055624/vendor/bundler_gems:/app/releases/20100714055624/vendor/bundler_gems/bundler/gems BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile What you see here is that the new worker does have correct GEM_HOME and GEM_PATH values, but the PATH and BUNDLE_GEMFILE values are pointing to the old dir. Are those bundler warnings something to worry about? The BUNDLE_GEMFILE value is worrying I think. I haven't tested this, but I'm pretty sure when Bundler.setup is called within your app it will actually setup using the old Gemfile then. So that would mean you need to reset it somehow? I don't see how at the moment... should unicorn provide a method similar to +working_directory+ to help ensure the application always uses the "current" Gemfile (eg "/app/current/Gemfile") ? Sound kind of strange unicorn should provide such a method, is there another way? What about that PATH value? What happens if 10 deployments later the dir /app/releases/20100714054705 is pruned? I tried. After removing that dir I could still send a HUP, but when I send a USR2 I get this: /app/releases/20100714054705/vendor/bundler_gems/gems/unicorn-1.1.2/lib/unicorn.rb:573:in `exec': No such file or directory - app/releases/20100714054705/vendor/bundler_gems/bin/unicorn_rails (Errno::ENOENT) from /app/releases/20100714054705/vendor/bundler_gems/gems/unicorn-1.1.2/lib/unicorn.rb:573:in `reexec' > since the `bundle exec` launches a binary with a path like /app/releases/201007125236/bin/unicorn, > which goes missing after N deploys That's what I'm seeing. How did using shared vendor/bundler_gems help you? Because I'm starting with bundle exec using shared vendor/bundler_gems as well. Cheers, Lawrence _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying