javascript disappears

javascript disappears

[Jimmy Soho]

Hi All,

When I deploy new code I always go through the USR2 - QUIT process.
However, after a few deployments it seems our base javascript file is
missing. I've experienced this a few times now. We use rails 2.3.8,
and have something like this in our layouts file:

<%= javascript_include_tag 'webtoolkit', 'jquery', 'jquery.metadata',
'jquery.url_utils', 'jquery.form', 'jquery.hotkeys',
'jquery.livequery', 'application', :cache => 'base'  %>

When it's missing it doesn't matter how often I kill with USR2 / QUIT,
it won't appear. If I stop the unicorn master process completely and
then start again, then it does create the base.js file again.

Anyone experienced this as well?  Anyone know what could be the reason for this?



Jimmy
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Jimmy Soho]

Btw, we also had this weird thing last week where after a new
deployment of code almost all field labels had the same value. We use
I18n v0.4.1 with a cache using memory_store, like so:

I18n::Backend::Simple.send(:include, I18n::Backend::Cache)
I18n.cache_store = ActiveSupport::Cache.lookup_store(:memory_store)
I18n.cache_store.silence!

I get the feeling that memory from the previous master/workers somehow
isn't released and is used by the new unicorn master and/or worker
processes.


Jimmy
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Eric Wong]

Jimmy Soho <jimmy.soho@gmail.com> wrote:
> Hi All,
> 
> When I deploy new code I always go through the USR2 - QUIT process.
> However, after a few deployments it seems our base javascript file is
> missing. I've experienced this a few times now. We use rails 2.3.8,
> and have something like this in our layouts file:
> 
> <%= javascript_include_tag 'webtoolkit', 'jquery', 'jquery.metadata',
> 'jquery.url_utils', 'jquery.form', 'jquery.hotkeys',
> 'jquery.livequery', 'application', :cache => 'base'  %>
> 
> When it's missing it doesn't matter how often I kill with USR2 / QUIT,
> it won't appear. If I stop the unicorn master process completely and
> then start again, then it does create the base.js file again.
> 
> Anyone experienced this as well?  Anyone know what could be the reason
> for this?

Hi Jimmy,   Are you deploying with Capistrano and you originally started
Unicorn in a working_directory that no longer exists on the filesystem?

Under Linux, you can check the working directory of any process
with `ls -l /proc/$PID/cwd`

That's the only thing I can think of at the moment.

In your other email:
> I get the feeling that memory from the previous master/workers somehow
> isn't released and is used by the new unicorn master and/or worker
> processes.

Shouldn't be possible unless your OS is broken.  A more likely
explanation is that Unicorn process is pinned to the working
directory of an old deploy.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Jimmy Soho]

> Hi Jimmy,   Are you deploying with Capistrano and you originally started
> Unicorn in a working_directory that no longer exists on the filesystem?
>
> Under Linux, you can check the working directory of any process
> with `ls -l /proc/$PID/cwd`

Ah! Yes. I use Capistrano, and it pruned releases. But definitely
/after/ a new version was deployed.

What I don't understand then though is that I have this in my unicorn
config file:

working_directory "/srv/app1/current"

How could the new master / workers still be pointing to the old
directory? Isn't what this working_directory directive is for?



Jimmy
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Eric Wong]

Jimmy Soho <jimmy.soho@gmail.com> wrote:
> > Hi Jimmy,   Are you deploying with Capistrano and you originally started
> > Unicorn in a working_directory that no longer exists on the filesystem?
> >
> > Under Linux, you can check the working directory of any process
> > with `ls -l /proc/$PID/cwd`
> 
> Ah! Yes. I use Capistrano, and it pruned releases. But definitely
> /after/ a new version was deployed.
> 
> What I don't understand then though is that I have this in my unicorn
> config file:
> 
> working_directory "/srv/app1/current"
> 
> How could the new master / workers still be pointing to the old
> directory? Isn't what this working_directory directive is for?

In your Capistrano deploy tasks, do you create/update the current
symlink before you send USR2?  You should have the symlink updated
before sending USR2..

Other folks here may have more experience with Capistrano+Unicorn,
maybe they have portable recipes they can share.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Jimmy Soho]

> In your Capistrano deploy tasks, do you create/update the current
> symlink before you send USR2?  You should have the symlink updated
> before sending USR2..

Yes I do.

Meanwhile I've been able to indeed reproduce the issue I was seeing if
I prune releases. This is what I do:

- ls /srv/app/releases shows only:

  /srv/app/releases/20100818022900

- stop unicorn
- start unicorn
- deploy new release (same code actually)
- unicorn now has a different PID, /proc/31761/cwd does point to the
latest release directory:

  /proc/31761/cwd -> /srv/app/releases/20100818024514

- refresh browser, ok, no issue.
- rm -Rf /srv/app/releases/20100818022900
- deploy another new release
- refresh browser: javascripts/base.js is missing

nothing in unicorn.stderr.log, but looking in unicorn.stdout.log it mentions:

  /srv/app/releases/20100818022900/Gemfile not found

The unicorn process PID hasn't changed, nor has proc/31761/cwd

My /srv/app/current dir now points to the latest release. Which I
guess is wrong as well, as unicorn failed to start with the latest
release; so it (= capistrano) needs to rollback somehow (which is a
different issue).

If I now peek in /proc/31761/environ then I notice these:

  GEM_HOME=/srv/app/releases/20100818022900/vendor/bundler_gems
  PATH=/srv/app/releases/20100818022900/vendor/bundler_gems/bin:[... etc.]
  GEM_PATH=/srv/app/releases/20100818022900/vendor/bundler_gems
  BUNDLE_GEMFILE=/srv/app/releases/20100818022900/Gemfile

which are all pointing to the working directory I pruned. Hence the
message about Gemfile not found I guess.

Any suggestions how / when to update those environment variables with
correct values?


> Other folks here may have more experience with Capistrano+Unicorn,
> maybe they have portable recipes they can share.

Maybe it's more a gem and/or bundler thing?  Though it seems like a
general issue to me that environment variables can have references to
old working directories, which need to be updated if you restart
Unicorn with USR2 all the time.  Is this something Unicorn could
possibly take care off?


Jimmy
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Eric Wong]

Jimmy Soho <jimmy.soho@gmail.com> wrote:
> If I now peek in /proc/31761/environ then I notice these:
> 
>   GEM_HOME=/srv/app/releases/20100818022900/vendor/bundler_gems
>   PATH=/srv/app/releases/20100818022900/vendor/bundler_gems/bin:[... etc.]
>   GEM_PATH=/srv/app/releases/20100818022900/vendor/bundler_gems
>   BUNDLE_GEMFILE=/srv/app/releases/20100818022900/Gemfile
> 
> which are all pointing to the working directory I pruned. Hence the
> message about Gemfile not found I guess.
> 
> Any suggestions how / when to update those environment variables with
> correct values?

Aha!  I believe there's an option for Bundler to use the /srv/app/shared
directory with Capistrano.  That's probably your best option.

> > Other folks here may have more experience with Capistrano+Unicorn,
> > maybe they have portable recipes they can share.
> 
> Maybe it's more a gem and/or bundler thing?  Though it seems like a
> general issue to me that environment variables can have references to
> old working directories, which need to be updated if you restart
> Unicorn with USR2 all the time.  Is this something Unicorn could
> possibly take care off?

You can also try force the environment variables with the
before_exec hook in your config file:

  before_exec do |server|
    ENV['GEM_HOME'] = '/srv/app/current/vendor/bundler_gems'
  end

But I think the best option is to use a shared directory for Bundler
since it should speed up deploys, too.  On the other hand, maybe Bundler
should be taught to respect symlink directories...

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Jimmy Soho]

> Aha!  I believe there's an option for Bundler to use the /srv/app/shared
> directory with Capistrano.  That's probably your best option.

I am using the shared bundler gems option, I use the example floating
around the internets:

run("cd #{release_path} && bundle install vendor/bundler_gems
--without development test --disable-shared-gems")

> You can also try force the environment variables with the
> before_exec hook in your config file:
>
>  before_exec do |server|
>    ENV['GEM_HOME'] = '/srv/app/current/vendor/bundler_gems'
>  end

After a few hours hacking I've come to the conclusion that's the only
way it can work, for now.

Having delved into bundler I think Bundler.setup needs to change. I'll
file an issue for that with them.


PS. I: for those interested, this is now the unicorn-conf.rb file I'm
using, so I can now happily prune all old releases:

http://gist.github.com/534668


PS. II: While I trying out various things I found that if I modified
the before_exec hook, then deployed that code, send USR2, then it
executes the before_exec hook of the old master, not the new master.
So I then had to send a USR2 a second time to see the new code in
before_exec taking effect. Is that intentional behaviour?


PS. III: Haven't tested it yet, but in case a USR2 fails, and unicorn
reverts back to the old master, does the old master then have the new
environment variable values? Or does it run the before_exec hook again
for the old master?



Jimmy
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: javascript disappears

[Eric Wong]

Jimmy Soho <jimmy.soho@gmail.com> wrote:
> PS. II: While I trying out various things I found that if I modified
> the before_exec hook, then deployed that code, send USR2, then it
> executes the before_exec hook of the old master, not the new master.
> So I then had to send a USR2 a second time to see the new code in
> before_exec taking effect. Is that intentional behaviour?

Yes.  USR2 doesn't reload the config file before by default.  Send a HUP
before USR2 if you want it to reload the config file.  Just keep in mind
if you add any new dependencies on your new deploy and HUP, things could
break, too, especially if you're using "preload_app true".

> PS. III: Haven't tested it yet, but in case a USR2 fails, and unicorn
> reverts back to the old master, does the old master then have the new
> environment variable values? Or does it run the before_exec hook again
> for the old master?

The old master doesn't get affected by environment (or any other local)
changes in before_exec.  before_exec only runs in the new master
process, so it can only affect the new master process (unless it sends
signals to the old master).

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying