Should USR2 always work?

Should USR2 always work?

[Laurens Pit]

Hi,

When deploying new code we go through the USR2 QUIT sequence. This works very nicely and gives zero downtime.

My question is whether there are instances when this sequence is known to not work, and instead you should really send QUIT first and then start all over?

I didn't expect that to be the case, but in the past year occasionally I've experienced I had to resort to QUIT and start all over in order to get all components loaded correctly.

Specifically: yesterday I upgraded several projects to rails 3.0.11 and added a new i18n .yml file in the config dir. After the USR2 QUIT sequence all new code seemed to work fine. Except the new .yml file wasn't loaded. Another run of USR2 and QUIT didn't resolve it. Only after QUIT and start of unicorn was the new .yml file loaded.

This was not just on one machine, which might have been a fluke. This was on all machines for all projects, consistently.

Any ideas?



Cheers,
Lawrence
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Should USR2 always work?

[Eric Wong]

Laurens Pit <laurens.pit@mirror42.com> wrote:
> I didn't expect that to be the case, but in the past year occasionally
> I've experienced I had to resort to QUIT and start all over in order
> to get all components loaded correctly.
> 
> Specifically: yesterday I upgraded several projects to rails 3.0.11
> and added a new i18n .yml file in the config dir. After the USR2 QUIT
> sequence all new code seemed to work fine. Except the new .yml file
> wasn't loaded. Another run of USR2 and QUIT didn't resolve it. Only
> after QUIT and start of unicorn was the new .yml file loaded.
> 
> This was not just on one machine, which might have been a fluke. This
> was on all machines for all projects, consistently.
> 
> Any ideas?

Anything from stderr log files?  USR2 will fail if there's

* compile/load error when loading the app (if preload_app=true)

* unicorn/unicorn_rails executable script is missing
  (maybe Bundler is moving it?)

* Ruby installation got moved/shifted/changed

* Working directory got _moved_ (cap may cycle those out)
  You can set "working_directory" in your unicorn config
  to work around it.

  (and maybe other reasons I can't think of right now)

Come to think of it, the missing working directory case
could be the most common...

But stderr log files (stderr_path) should always tell you what's
going on.

== Linux(-only?) tip:

If the log files got deleted somehow, you may still be able to read it
via: "cat /proc/$PID/fd/2" on either the PID of the master or _any_
worker process.

To read some other log files, you can just replace the "2" with whatever
file descriptor.  Reading the symlinks ("ls -l /proc/$PID/fd/") will
tell you where each descriptor is pointed to, even if it is a deleted
file.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Should USR2 always work?

[Lawrence Pit]

> Anything from stderr log files?  USR2 will fail if there's

Right: 

> ruby: no such file to load -- bundler/setup (LoadError)


> * unicorn/unicorn_rails executable script is missing
>  (maybe Bundler is moving it?)


Bundler has been updated several times this year, I guess every time that happens and a new version gets installed a complete restart of unicorn is required? Am I the only one seeing this?

Perhaps I'm managing bundler wrong? Bundler and rake are the only two gems I install in the global gem space. With an update of bundler I do "gem uninstall bundler ; gem bundler install". Should I manage this differently?



Cheers,
Lawrence

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Should USR2 always work?

[Eric Wong]

Lawrence Pit <lawrence.pit@gmail.com> wrote:
> Bundler has been updated several times this year, I guess every time
> that happens and a new version gets installed a complete restart of
> unicorn is required? Am I the only one seeing this?

Managing bunder has always been tricky, unfortunately.

> Perhaps I'm managing bundler wrong? Bundler and rake are the only two
> gems I install in the global gem space. With an update of bundler I do
> "gem uninstall bundler ; gem bundler install". Should I manage this
> differently?

I brought up a "bring-your-own-executable" deployment a few months ago
but haven't heard any feedback.  Here it is for Bundler:

   http://mid.gmane.org/20110819022316.GA2951@dcvr.yhbt.net

I am also using something very similar with Isolate:

------------- /full/path/to/app_root/script/unicorn-byoe ---------
#!/home/ew/ruby-1.9.3/bin/ruby
# Unicorn: bring-your-own executable
require "isolate/now"
$LOAD_PATH << "./lib"
load Gem.bin_path("unicorn", "unicorn")
------------------------------- 8< -------------------------------

I run the path using the full pathname, so there's less ambiguity as to
what gets captured as $0 and executed on USR2.

I don't know of anybody else doing/trying this, though.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Should USR2 always work?

[Lawrence Pit]

> I brought up a "bring-your-own-executable" deployment a few months ago
> but haven't heard any feedback.  Here it is for Bundler:
> 
>   http://mid.gmane.org/20110819022316.GA2951@dcvr.yhbt.net


I tried this out by going back to a previous version of bundler, then deployed, unfortunately doesn't seem to work for me:

  executing ["/srv/ec/current/script/unicorn_byoe", "-D", "-E", "staging", "-c", "/srv/ec/current/config/unicorn/staging.rb"] (in /srv/ec/releases/20111123052840)
  /opt/ree/bin/ruby: no such file to load -- bundler/setup (LoadError)

Sending QUIT and then start anew works fine. Upgraded back to latest, tried again, and again no luck. QUIT + start is the only thing that works then.


Tried a slightly modified version of unicorn_byoe, I'm using rails:

#!/opt/ree/bin/ruby
require File.expand_path("../../config/boot.rb", __FILE__)
# Gem.bin_path(gem_name, executable_name, *version_requirements)
load Gem.bin_path("unicorn", "unicorn_rails")   



Cheers,
Lawrence


_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying