Unicorn returns blank page after no use

Unicorn returns blank page after no use

[Farjad Adamjee]

Hello,

I am not sure if this is the correct place for this, I have googled 
around to see if anyone else has encountered such an issue, but I did 
not find anything.

I am having this issue where after a period of time of no use (on beta 
[production] environment), maybe 12 hours or so, the unicorn process 
sleeps. When I go and re-request the page, it returns blank.

I am using Apache proxy to Unicorn.

My unicorn.stderr.log states:
I, [2015-07-01T03:07:05.168511 #13258]  INFO -- : master done reopening logs
I, [2015-07-01T03:07:05.232546 #16222]  INFO -- : worker=2 done 
reopening logs
I, [2015-07-01T03:07:05.239934 #13308]  INFO -- : worker=0 done 
reopening logs
I, [2015-07-01T03:07:05.307616 #13311]  INFO -- : worker=1 done 
reopening logs


When I request the page again, it loads fine. It is only during the 
process of "reopening logs" when it returns a blank page.

My unicorn.rb file:

worker_processes 3

# Load app into the master before forking workers
# for super-fast worker spawn times
preload_app true

# Listen on
listen "127.0.0.1:8550", :backlog => 1024

# Restart any workers that haven't responded in 180 seconds
timeout 600

# PID for the Unicorn master
pid "/var/www/r/shared/pids/unicorn.pid"

# Unicorn Log paths
stderr_path "/var/www/r/shared/log/unicorn.stderr.log"
stdout_path "/var/www/r/shared/log/unicorn.stdout.log"

before_fork do |server, worker|
   ##
   # When sent a USR2, Unicorn will suffix its pidfile with .oldbin and
   # immediately start loading up a new version of itself (loaded with a new
   # version of our app). When this new Unicorn is completely loaded
   # it will begin spawning workers. The first worker spawned will check to
   # see if an .oldbin pidfile exists. If so, this means we've just 
booted up
   # a new Unicorn and need to tell the old one that it can now die. To 
do so
   # we send it a QUIT.
   #
   # Using this method we get 0 downtime deploys.
   defined?(ActiveRecord::Base) and 
ActiveRecord::Base.connection.disconnect!
   defined?(OathKeeper) and OathKeeper.disconnect!

   old_pid = '/var/www/r/shared/pids/unicorn.pid.oldbin'
   if File.exists?(old_pid) && server.pid != old_pid
     begin
       Process.kill("QUIT", File.read(old_pid).to_i)
     rescue Errno::ENOENT, Errno::ESRCH
       # someone else did our job for us
     end
   end
end

after_fork do |server, worker|
   ##
   #  # Unicorn master loads the app then forks off workers - because of 
the way
   #    # Unix forking works, we need to make sure we aren't using any 
of the parent's
   #      # sockets, e.g. db connection
   #
   defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
   defined?(OathKeeper) and OathKeeper.reconnect
end

Any help would be appreciated!
Thank you!

Farjad


-- 
Farjad Adamjee
844.246.4018

Re: Unicorn returns blank page after no use

[Eric Wong]

Farjad Adamjee <fadamjee@vailsys.com> wrote:
> Hello,
> 
> I am not sure if this is the correct place for this, I have googled
> around to see if anyone else has encountered such an issue, but I
> did not find anything.

This is the correct place :)  Plain-text email is the only way I
handle support for ease-of-archival and distribution.

> I am having this issue where after a period of time of no use (on
> beta [production] environment), maybe 12 hours or so, the unicorn
> process sleeps. When I go and re-request the page, it returns blank.

This sounds like a database (or any other persistent connection
backend) connection expiring due to an idle timeout.

This could also be a firewall/router timeout problem between the
server running unicorn and the database host.

Tony experienced this last year:
http://bogomips.org/unicorn-public/m/CAKM1sPNRsES6H6ByK6bO9Djwa8WvYV6HJ-rEaHopRUYBVFfuhg@mail.gmail.com

You can confirm by running lsof on a worker processes to look for
open sockets.  In case the connection loss isn't noticed by the
OS, use strace (or any other similar syscall tracer for your OS)
instead and watch connection activity.

With strace (or similar) knock yourself down to one worker
process via SIGTTOU to the master to ensure you're always tracing
the correct one.

> I am using Apache proxy to Unicorn.

By the way, this is not recommended for slow clients.
nginx (Open Source edition) is currently the only supported proxy.

>   # Using this method we get 0 downtime deploys.
>   defined?(ActiveRecord::Base) and
> ActiveRecord::Base.connection.disconnect!
>   defined?(OathKeeper) and OathKeeper.disconnect!

I suggest disabling any idle timeout you might have, enabling TCP
keepalive and enabling reconnects (see AR and your database driver
documentation).  Worst case (firewall/router problem you can't
fix via configuration), is to send a request to unicorn every
few minutes/hours to keep things going.

Perhaps OathKeeper has the same problem, too...
I've never heard of it until now.