Unicorn fails to restart gracefully on capistrano deploy

Unicorn fails to restart gracefully on capistrano deploy

[Eirik Dentz Sinclair]

First off thanks very much for all the hard work on unicorn. Alas, we've encountered an issue where unicorn fails to spawn new workers that have loaded the incoming revision on a capistrano deploy. I'm not entirely sure the issue is due to unicorn as it appears that bundler was responsible for a similar issue in the past: http://github.com/carlhuda/bundler/issues/issue/259/#comment_180830

But presumably bundler has corrected that issue and any help in sorting out the issue would be greatly appreciated. 

We are following EngineYard's instructions for loading unicorn via bundle exec: http://docs.engineyard.com/appcloud/howtos/cutomization/customize-unicorn

So far we've tried deploying with the unicorn config set to both "preload_app true" and "preload_app false".
(unicorn.rb being used: https://gist.github.com/9a208e3d1d1e44161018)

When "preload_app true" is used the "uninitialized constant Prefix::OurCustomApi (NameError)" log entry/stacktrace appears once. And the workers never load the incoming code. So while the incoming code is in place and the symlinks are all updated we're still running the previous revision until a unicorn stop/start or reload is sent.

When "preload_app false" is used the "uninitialized constant Prefix::OurCustomApi (NameError)" log entry/stacktrace is written to the unicorn.stderr.log continuously as it keeps trying to respawn workers in a loop. This goes on until a unicorn stop command is sent. Then a unicorn start can be used to load the incoming revision successfully.

The common deployment stack and versions follow:

Capistrano 2.5.19
Bundler 1.0.0rc6
Rails 2.3.8
Unicorn 1.0.1 (tested under unicorn 1.1.3 with same results)

The deployments differ as follows:

The deployed versioned directory is 20100830230613
The incoming versioned directory is 20100902192111

Code in version 20100830230613 uses our_custom_api.gem version 1.0 via the Gemfile
Code in version 20100830230613 uses the constant OurCustomApi in config/initializers/our_custom_api.rb

Code in version 20100902192111 uses prefix_our_custom_api.gem version 2.0 via the Gemfile
Code in version 20100902192111 uses the constant Prefix::OurCustomApi in config/initializers/our_custom_api.rb

The relevant relevant unicorn.stderr.log can be found below and appear to show some sort of issue where both versioned directories are being loaded into the same process?

Unicorn 1.0.1:
https://gist.github.com/ddc852b23617fc8584e0

Unicorn 1.1.3:
https://gist.github.com/6314ea0e14ff78b4e8dc

Thanks in advance,

Eirik Dentz Sinclair
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn fails to restart gracefully on capistrano deploy

[Jamie Wilkinson]

On Sep 9, 2010, at 4:28 PM, Eirik Dentz Sinclair wrote:

> First off thanks very much for all the hard work on unicorn. Alas, we've encountered an issue where unicorn fails to spawn new workers that have loaded the incoming revision on a capistrano deploy. I'm not entirely sure the issue is due to unicorn as it appears that bundler was responsible for a similar issue in the past: http://github.com/carlhuda/bundler/issues/issue/259/#comment_180830
> 
> But presumably bundler has corrected that issue and any help in sorting out the issue would be greatly appreciated. 
> 
> [snip]
> 
> The deployed versioned directory is 20100830230613
> The incoming versioned directory is 20100902192111

I think I may have run into a potentially related bundler+unicorn+capistrano issue just today! 

With bundler+unicorn the shell environment does not really get cleared between USR2 restarts/forkings, which was the cause of the aforementioned GitHub ticket... PATH/RUBYOPT would just get infinitely appended to by bundler, and unicorn is happy to let you do whatever with the environment

The bug I found today cropped up while upgrading from rails 2.3.8 to rails 2.3.9 -- on deploy the new unicorn master would die with "hey you've already loaded rails 2.3.8!". Weird.

Some quick debugging showed my BUNDLE_GEMFILE environment variable was referencing a Gemfile from a (very) old capistrano release.... I'm actively debugging it now. I'm also using preload_app=true which might have contributed. 

Eirik, can you confirm any similar "old release" weirdnesses by dumping the ENV in your unicorn.rb's before_exec / before_fork?

	 STDERR.puts ENV.inspect
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn fails to restart gracefully on capistrano deploy

[Lawrence Pit]

See the before_exec in http://gist.github.com/534668


Cheers,
Lawrence

> On Sep 9, 2010, at 4:28 PM, Eirik Dentz Sinclair wrote:
>
>> First off thanks very much for all the hard work on unicorn. Alas, we've encountered an issue where unicorn fails to spawn new workers that have loaded the incoming revision on a capistrano deploy. I'm not entirely sure the issue is due to unicorn as it appears that bundler was responsible for a similar issue in the past: http://github.com/carlhuda/bundler/issues/issue/259/#comment_180830
>>
>> But presumably bundler has corrected that issue and any help in sorting out the issue would be greatly appreciated.
>>
>> [snip]
>>
>> The deployed versioned directory is 20100830230613
>> The incoming versioned directory is 20100902192111
> I think I may have run into a potentially related bundler+unicorn+capistrano issue just today!
>
> With bundler+unicorn the shell environment does not really get cleared between USR2 restarts/forkings, which was the cause of the aforementioned GitHub ticket... PATH/RUBYOPT would just get infinitely appended to by bundler, and unicorn is happy to let you do whatever with the environment
>
> The bug I found today cropped up while upgrading from rails 2.3.8 to rails 2.3.9 -- on deploy the new unicorn master would die with "hey you've already loaded rails 2.3.8!". Weird.
>
> Some quick debugging showed my BUNDLE_GEMFILE environment variable was referencing a Gemfile from a (very) old capistrano release.... I'm actively debugging it now. I'm also using preload_app=true which might have contributed.
>
> Eirik, can you confirm any similar "old release" weirdnesses by dumping the ENV in your unicorn.rb's before_exec / before_fork?
>
> 	 STDERR.puts ENV.inspect
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Unicorn fails to restart gracefully on capistrano deploy

[Eirik Dentz Sinclair]

Lawrence, Jamie

Thanks very much for your responses.

The before_exec block in Lawrence's gist was just what was needed. Not sure how many others are using Capistrano and bundler, but seems like that before_exec block would be a good addition to this page: http://unicorn.bogomips.org/Sandbox.html

Thanks again for the help.

-Eirik

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn fails to restart gracefully on capistrano deploy

[Eric Wong]

Eirik Dentz Sinclair <eirik@efficiency20.com> wrote:
> Lawrence, Jamie
> 
> Thanks very much for your responses.
> 
> The before_exec block in Lawrence's gist was just what was needed. Not sure how many others are using Capistrano and bundler, but seems like that before_exec block would be a good addition to this page: http://unicorn.bogomips.org/Sandbox.html
> 
> Thanks again for the help.

Thanks for confirming the fix.  I've updated the Sandbox document
with the following patch.

Patches/pull-requests to the mailing list for documentation are very
much appreciated, especially for Bundler and maybe other things I don't
use myself.  Lets try to treat the in-source documentation much like a
wiki, except anybody who grabs the source also has the latest up-to-date
docs ready for offline reading (and readable without a web browser :)

>From 1a75966a5d1a1f6307ed3386e2f91a28bbb72ca0 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Wed, 15 Sep 2010 14:42:54 -0700
Subject: [PATCH] doc: update Sandbox document for Bundler

Thanks to Lawrence Pit, Jamie Wilkinson, and Eirik Dentz Sinclair.

ref: mid.gmane.org/4C8986DA.7090603@gmail.com
ref: mid.gmane.org/5F1A02DB-CBDA-4302-9E26-8050C2D72433@efficiency20.com
---
 Sandbox |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/Sandbox b/Sandbox
index d2f7590..d101106 100644
--- a/Sandbox
+++ b/Sandbox
@@ -24,6 +24,9 @@ this:
 Then use HUP to reload, and then continue with the USR2+QUIT upgrade
 sequence.
 
+Environment variable pollution when exec-ing a new process (with USR2)
+is the primary issue with sandboxing tools such as Bundler and Isolate.
+
 == Bundler
 
 === Running
@@ -42,6 +45,12 @@ This is no longer be an issue as of bundler 0.9.17
 
 ref: http://mid.gmane.org/8FC34B23-5994-41CC-B5AF-7198EF06909E@tramchase.com
 
+=== Other ENV pollution issues
+
+You may need to set or reset BUNDLE_GEMFILE, GEM_HOME, GEM_PATH and PATH
+environment variables in the before_exec hook as illustrated by
+http://gist.github.com/534668
+
 == Isolate
 
 === Running
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying