Forking for background processing with rails/unicorn

Forking for background processing with rails/unicorn

[Brian Underwood]

Hello,

I've been working for a few days on a problem which, after tracking it
down, seems to be related to unicorn.  I'm hoping that somebody might
be able to help with information about what's going on.

In the Rails 3 application that I support, we sometimes fork during a
web request in order to do background processing behind the scenes
without the user (and the browser connection) to wait for it to
finish.  I realize that a better method would be to have a queue of
jobs to be processed (I even created a gem called job_boss which works
well for our case), but it will take us a while to get there, so I'm
trying to fix this for now.

The problem that I'm finding is that, when I fork, the HTTP connection
doesn't close.  IE doesn't handle this as well and the browser is left
sitting, waiting for the child process to finish rather than being
redirected.  This functionality worked fine in our application when we
were using Rails 2.3.5 and Unicorn 1.1.3, but broke when we moved to
Rails 3 and Unicorn 3.0.0 (we've since moved to Unicorn 3.4.0 and the
problem remains, I had some hope that Unicorn 3.3.1 might have
addressed it).

I've been able to solve this problem temporarily by closing two file
descriptors manually (#3 and, depending on the version of Rails #6 or
#7).  Obviously this is not ideal.  I found the UNICORN_FD environment
variable when looking through the Unicorn source, but that only seems
to indicate #3.  Where might #6/#7 be coming from?  Is there some way
that I can determine these without hard-coding the FD #?

Any help that you can provide would be greatly appreciated,
Brian Underwood
Software Systems Engineer

RBM Technologies, Inc.
215 1st St  ·  Cambridge, MA 02142
call  (617) 401-4666
visit www.rbmtechnologies.com


The Power Behind the Planogram™

"This email and any of its attachments may contain proprietary
information, which is privileged, confidential, or subject to
copyrights belonging to RBM Technologies, Inc. This email is intended
solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this email, you
are hereby notified that any dissemination, distribution, copying, or
action taken in relation to the contents of and attachments to this is
strictly prohibited and may be unlawful. If you have received this
email in error, please notify the sender immediately and permanently
delete the original and any copy of this email and any printout. Thank
you."
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Forking for background processing with rails/unicorn

[Eric Wong]

Brian Underwood <bunderwood@rbmtechnologies.com> wrote:
> In the Rails 3 application that I support, we sometimes fork during a
> web request in order to do background processing behind the scenes
> without the user (and the browser connection) to wait for it to
> finish.  I realize that a better method would be to have a queue of
> jobs to be processed (I even created a gem called job_boss which works
> well for our case), but it will take us a while to get there, so I'm
> trying to fix this for now.
> 
> The problem that I'm finding is that, when I fork, the HTTP connection
> doesn't close.  IE doesn't handle this as well and the browser is left
> sitting, waiting for the child process to finish rather than being
> redirected.  This functionality worked fine in our application when we
> were using Rails 2.3.5 and Unicorn 1.1.3, but broke when we moved to
> Rails 3 and Unicorn 3.0.0 (we've since moved to Unicorn 3.4.0 and the
> problem remains, I had some hope that Unicorn 3.3.1 might have
> addressed it).

I'm not able to reproduce any difference between 1.x and 3.x in this
regard in forking behavior.  I know that the client socket is never
closed on fork by default, so there's probably something else in your
app that changed...

> I've been able to solve this problem temporarily by closing two file
> descriptors manually (#3 and, depending on the version of Rails #6 or
> #7).  Obviously this is not ideal.  I found the UNICORN_FD environment
> variable when looking through the Unicorn source, but that only seems
> to indicate #3.  Where might #6/#7 be coming from?

Whatever is in UNICORN_FD is the listener socket (probably 3), so 6/7 is
probably the client socket, but you should normally close everything in
the forked child for background processing.

The IO.closefrom method in the io-extra[1] gem will get everything
including descriptors opened without a corresponding Ruby object.
If you know everything uses Ruby IO, then walking ObjectSpace works,
too:

  # close everything except stdin, stdout and stderr
  ObjectSpace.each_object(IO) { |s|
    begin
      s.close if s.fileno > 2
    rescue IOError, Errno::EBADF
      # EBADF can come from duplicate IOs due to FD inheritance
    end
  }

> Is there some way
> that I can determine these without hard-coding the FD #?

lsof or scanning /proc/$$/fd (Linux), but if you have any open socket
connections (memcached/activerecord/net-http-persistent/etc...), you
should close all of those and restart them if necessary in the child.

[1] http://rubyforge.org/projects/shards

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying