[ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Eric Wong]

Changes:

Theses release fix a long-standing bug where the original PID
file is not restored when rolling back from a USR2 upgrade.
Presumably most upgrades aren't rolled back, so it took over a
year to notice this issue.  Thanks to Lawrence Pit for
discovering and reporting this issue.

About Unicorn:

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Lawrence Pit]

Hi Eric,

Thanks mate. Unfortunately this still doesn't seem to work for me.

I went through the process manually, you can see a transcript at 
http://pastie.org/1043347.txt
The indented text is what I see appearing in the unicorn stderr.log.

As you can see it reports two errors during the process, saying 
unicorn.pid is stale.

I wonder whether this is because in my unicorn config file I mention:

pid "/var/www/staging/current/tmp/pids/unicorn.pid"

whereas in your back_out_of_upgrade test you don't have such a line?



Cheers,
Lawrence

> Changes:
>
> Theses release fix a long-standing bug where the original PID
> file is not restored when rolling back from a USR2 upgrade.
> Presumably most upgrades aren't rolled back, so it took over a
> year to notice this issue.  Thanks to Lawrence Pit for
> discovering and reporting this issue.
>
> About Unicorn:
>
> Unicorn is an HTTP server for Rack applications designed to only serve
> fast clients on low-latency, high-bandwidth connections and take
> advantage of features in Unix/Unix-like kernels.  Slow clients should
> only be served by placing a reverse proxy capable of fully buffering
> both the the request and response in between Unicorn and slow clients.
>
> * http://unicorn.bogomips.org/
> * mongrel-unicorn@rubyforge.org
> * git://git.bogomips.org/unicorn.git
>
>   

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Eric Wong]

Lawrence Pit <lawrence.pit@gmail.com> wrote:
> Hi Eric,
>
> Thanks mate. Unfortunately this still doesn't seem to work for me.
>
> I went through the process manually, you can see a transcript at  
> http://pastie.org/1043347.txt
> The indented text is what I see appearing in the unicorn stderr.log.

Thanks Lawrence, the following line looks suspicious to me:

    reaped #<Process::Status: pid=4382,exited(0)> exec()-ed

Which makes it seem like
http://git.bogomips.org/cgit/unicorn.git/commit/?id=3f0f9d6 didn't get
applied (and I just checked the 1.1.2 gem and it looks good to me)

Can you dump out the ENV hash in your before_exec hook of your
unicorn.conf.rb?  I'm wondering if GEM_HOME or GEM_PATH is set
and doing anything funky, causing you to load an old version
of Unicorn:

  before_exec do |server|
    ENV.each do |key,value|
      server.logger.info "#{key}=#{value}"
    end
  end

> As you can see it reports two errors during the process, saying  
> unicorn.pid is stale.
>
> I wonder whether this is because in my unicorn config file I mention:
>
> pid "/var/www/staging/current/tmp/pids/unicorn.pid"
>
> whereas in your back_out_of_upgrade test you don't have such a line?

It does, I create a similar line in the unicorn_setup() function of
test-lib.sh

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Lawrence Pit]

Hi Eric,

At the top of my pastie http://pastie.org/1043347.txt you can see that 
when I request of list of gems it mentions only unicorn v1.1.2

I also added the suggested logging to the before_fork block and I also 
logged the unicorn version. Interesting output. First it runs the master 
with Unicorn v1.1.2. But when I send a USR2 it exec'ed with a Unicorn 
v1.1.1 for the new master. So yes, I still had a version lying around, 
namely in the $APP_ROOT/vendor/bundler_gems dir.

See also: http://pastie.org/1043487.txt

It does do this though after a USR2:

executing ["/usr/local/rvm/gems/ree-1.8.7-2010.02/bin/unicorn_rails", 
"-D", "-E", "staging", "-c", 
"/var/www/staging/current/config/unicorn/staging.rb"]

I.e., it tries to exec exactly how I start unicorn. That unicorn_rails 
binary in that path is definitely v1.1.2. So I don't quite understand 
why the reexeced one is picking the version from the vendor/bundler_gems 
dir.


I need to play a bit more with this..  I'll probably have to change my 
/etc/init.d/unicorn script so that it starts with the 
vendor/bundler_gems version instead of a system-wide version :o    I'll 
get back to you..



Cheers,
Lawrence

> Lawrence Pit <lawrence.pit@gmail.com> wrote:
>   
>> Hi Eric,
>>
>> Thanks mate. Unfortunately this still doesn't seem to work for me.
>>
>> I went through the process manually, you can see a transcript at  
>> http://pastie.org/1043347.txt
>> The indented text is what I see appearing in the unicorn stderr.log.
>>     
>
> Thanks Lawrence, the following line looks suspicious to me:
>
>     reaped #<Process::Status: pid=4382,exited(0)> exec()-ed
>
> Which makes it seem like
> http://git.bogomips.org/cgit/unicorn.git/commit/?id=3f0f9d6 didn't get
> applied (and I just checked the 1.1.2 gem and it looks good to me)
>
> Can you dump out the ENV hash in your before_exec hook of your
> unicorn.conf.rb?  I'm wondering if GEM_HOME or GEM_PATH is set
> and doing anything funky, causing you to load an old version
> of Unicorn:
>
>   before_exec do |server|
>     ENV.each do |key,value|
>       server.logger.info "#{key}=#{value}"
>     end
>   end
>
>   
>> As you can see it reports two errors during the process, saying  
>> unicorn.pid is stale.
>>
>> I wonder whether this is because in my unicorn config file I mention:
>>
>> pid "/var/www/staging/current/tmp/pids/unicorn.pid"
>>
>> whereas in your back_out_of_upgrade test you don't have such a line?
>>     
>
> It does, I create a similar line in the unicorn_setup() function of
> test-lib.sh
>
>   

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Eric Wong]

Lawrence Pit <lawrence.pit@gmail.com> wrote:
> Hi Eric,
>
> At the top of my pastie http://pastie.org/1043347.txt you can see that  
> when I request of list of gems it mentions only unicorn v1.1.2
>
> I also added the suggested logging to the before_fork block and I also  
> logged the unicorn version. Interesting output. First it runs the master  
> with Unicorn v1.1.2. But when I send a USR2 it exec'ed with a Unicorn  
> v1.1.1 for the new master. So yes, I still had a version lying around,  
> namely in the $APP_ROOT/vendor/bundler_gems dir.
>
> See also: http://pastie.org/1043487.txt
>
> It does do this though after a USR2:
>
> executing ["/usr/local/rvm/gems/ree-1.8.7-2010.02/bin/unicorn_rails",  
> "-D", "-E", "staging", "-c",  
> "/var/www/staging/current/config/unicorn/staging.rb"]
>
> I.e., it tries to exec exactly how I start unicorn. That unicorn_rails  
> binary in that path is definitely v1.1.2. So I don't quite understand  
> why the reexeced one is picking the version from the vendor/bundler_gems  
> dir.

It's probably loading 1.1.1 because of GEM_HOME/GEM_PATH being set.  The
"unicorn_rails"/"unicorn" wrappers just activates whatever gems it sees,
and GEM_HOME/GEM_PATH can influence the wrapper.

Managing RubyGems installations is still tricky :<

> I need to play a bit more with this..  I'll probably have to change my  
> /etc/init.d/unicorn script so that it starts with the  
> vendor/bundler_gems version instead of a system-wide version :o    I'll  
> get back to you..

Take a look at http://unicorn.bogomips.org/Sandbox.html  According to
Jamie, using "bundle exec unicorn_rails" is the correct way to go.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Lawrence Pit]

>> I need to play a bit more with this..  I'll probably have to change my  
>> /etc/init.d/unicorn script so that it starts with the  
>> vendor/bundler_gems version instead of a system-wide version :o    I'll  
>> get back to you..
>>     
>
> Take a look at http://unicorn.bogomips.org/Sandbox.html  According to
> Jamie, using "bundle exec unicorn_rails" is the correct way to go.
>   

Thanks.  Yes indeed, bundle exec is the way to go.  I've got it rolling 
back nicely now.

It appears unicorn rolls back by itself when it can't start the new 
master. Nice!

Jamie also mentions to use the shared vendor/bundler_gems path.  Though 
I do use that in all my projects I'm not so sure it's wise to do when 
you can't afford any downtime.  I'll have to wait for unicorn v1.1.3 
before I can test whether rolling back will indeed continue with unicorn 
v1.1.2 instead of v1.1.3 ;o


Cheers,
Lawrence

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Jamie Wilkinson]

On Jul 14, 2010, at 12:00 AM, Lawrence Pit wrote:

> It appears unicorn rolls back by itself when it can't start the new master. Nice!
> 
> Jamie also mentions to use the shared vendor/bundler_gems path.  Though I do use that in all my projects I'm not so sure it's wise to do when you can't afford any downtime.  I'll have to wait for unicorn v1.1.3 before I can test whether rolling back will indeed continue with unicorn v1.1.2 instead of v1.1.3 ;o

Hi Lawrence, why does the shared vendor/bundler_gems cause you downtime? From not re-bundling during rollback?

FWIW I made that recommendation because I ran into issues with unicorns not restarting correctly after running `cap deploy:cleanup`, since the `bundle exec` launches a binary with a path like /app/releases/201007125236/bin/unicorn, which goes missing after N deploys. I haven't tested if this applies to more recent versions

Shared bundles are also significantly faster -- `bundle check || bundle install` is ~1s for me vs. several minutes to totally rebundle

-jamie
http://jamiedubs.com
http://fffff.at
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Lawrence Pit]

>> It appears unicorn rolls back by itself when it can't start the new master. Nice!
>>
>> Jamie also mentions to use the shared vendor/bundler_gems path.  Though I do use that in all my projects I'm not so sure it's wise to do when you can't afford any downtime.  I'll have to wait for unicorn v1.1.3 before I can test whether rolling back will indeed continue with unicorn v1.1.2 instead of v1.1.3 ;o
>>     
>
> Hi Lawrence, why does the shared vendor/bundler_gems cause you downtime? From not re-bundling during rollback?
>
> FWIW I made that recommendation because I ran into issues with unicorns not restarting correctly after running `cap deploy:cleanup`, since the `bundle exec` launches a binary with a path like /app/releases/201007125236/bin/unicorn, which goes missing after N deploys. I haven't tested if this applies to more recent versions
>
> Shared bundles are also significantly faster -- `bundle check || bundle install` is ~1s for me vs. several minutes to totally rebundle
>   

Hi Jamie, I see what you mean. I haven't tested this yet, so I don't 
know for sure if it could cause downtime. I also always use bundle check 
|| bundle install to get that time benefit.

My worry was indeed that it might pick a newer unicorn to restart 
workers for the old master. I guess that can/should be prevented by 
having an on_rollback in my capistrano bundler task.


What is the exact command you use to start unicorn?


What I see is this, I start with:

$ su -l deployer -c "cd /app/current ; bundle exec unicorn_rails -D -c 
/app/current/config/unicorn.rb"

My after_fork and before_exec methods output the ENV vars to the unicorn 
log. I see for example:

AFTER_FORK: VERSION 1
PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:....
GEM_HOME=/app/releases/20100714054705/vendor/bundler_gems
GEM_PATH=/app/releases/20100714054705/vendor/bundler_gems:/app/releases/20100714054705/vendor/bundler_gems/bundler/gems
BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile

Note that /app/releases/20100714054705/vendor/bundler_gems is a symlink 
to /app/releases/shared/vendor/bundler_gems

When I deploy a new version, symlink /app/current to the new release 
directory, and send USR2 :

executing 
["/app/releases/20100714054705/vendor/bundler_gems/bin/unicorn_rails", 
"-D", "-E", "staging", "-c", "/app/current/config/unicorn.rb"] (in 
/app/releases/20100714055624)
BEFORE_EXEC: VERSION 1
PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:....
GEM_HOME=/app/releases/20100714054705/vendor/bundler_gems
GEM_PATH=/app/releases/20100714054705/vendor/bundler_gems:/app/releases/20100714054705/vendor/bundler_gems/bundler/gems
BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile
I, [2010-07-15T23:31:47.765995 #23084]  INFO -- : inherited 
addr=/tmp/.unicorn_sock fd=3
I, [2010-07-15T23:31:47.766646 #23084]  INFO -- : Refreshing Gem list
/app/releases/20100714055624/.bundle/environment.rb:175: warning: 
already initialized constant ENV_LOADED
/app/releases/20100714055624/.bundle/environment.rb:176: warning: 
already initialized constant LOCKED_BY
/app/releases/20100714055624/.bundle/environment.rb:177: warning: 
already initialized constant FINGERPRINT
/app/releases/20100714055624/.bundle/environment.rb:178: warning: 
already initialized constant HOME
/app/releases/20100714055624/.bundle/environment.rb:179: warning: 
already initialized constant AUTOREQUIRES
/app/releases/20100714055624/.bundle/environment.rb:181: warning: 
already initialized constant SPECS
AFTER_FORK: VERSION 2
PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:....
GEM_HOME=/app/releases/20100714055624/vendor/bundler_gems
GEM_PATH=/app/releases/20100714055624/vendor/bundler_gems:/app/releases/20100714055624/vendor/bundler_gems/bundler/gems
BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile


What you see here is that the new worker does have correct GEM_HOME and 
GEM_PATH values, but the PATH and BUNDLE_GEMFILE values are pointing to 
the old dir.

Are those bundler warnings something to worry about?

The BUNDLE_GEMFILE value is worrying I think. I haven't tested this, but 
I'm pretty sure when Bundler.setup is called within your app it will 
actually setup using the old Gemfile then. So that would mean you need 
to reset it somehow?  I don't see how at the moment...  should unicorn 
provide a method similar to +working_directory+ to help ensure the 
application always uses the "current" Gemfile (eg 
"/app/current/Gemfile") ?  Sound kind of strange unicorn should provide 
such a method, is there another way?

What about that PATH value?  What happens if 10 deployments later the 
dir /app/releases/20100714054705 is pruned? I tried. After removing that 
dir I could still send a HUP, but when I send a USR2 I get this:

/app/releases/20100714054705/vendor/bundler_gems/gems/unicorn-1.1.2/lib/unicorn.rb:573:in 
`exec': No such file or directory - 
app/releases/20100714054705/vendor/bundler_gems/bin/unicorn_rails 
(Errno::ENOENT)
        from 
/app/releases/20100714054705/vendor/bundler_gems/gems/unicorn-1.1.2/lib/unicorn.rb:573:in 
`reexec'


 > since the `bundle exec` launches a binary with a path like 
/app/releases/201007125236/bin/unicorn,
 > which goes missing after N deploys


That's what I'm seeing. How did using shared vendor/bundler_gems help 
you?  Because I'm starting with bundle exec using shared 
vendor/bundler_gems as well.




Cheers,
Lawrence
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: [ANN] unicorn 1.0.1 and 1.1.2 - fix rollbacks

[Lawrence Pit]

> What about that PATH value?  What happens if 10 deployments later the 
> dir /app/releases/20100714054705 is pruned? I tried. After removing 
> that dir I could still send a HUP, but when I send a USR2 I get this:

Eric had already pointed me to http://unicorn.bogomips.org/Sandbox.html 
before, I forgot to add that.. this does the trick:

    Unicorn::HttpServer::START_CTX[0] = 
"/app/shared/bundler_gems/bin/unicorn_rails"




Cheers,
Lawrence
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying