Forking non web processes

Forking non web processes

[Sam Saffron]

Hi Eric,

I have been trying to get unicorn to allow me to fork off non-web
processes like sidekiq/resque.

I got this working, except that I am constantly fighting with the
unicorn reaper. Any chance we can add some sort of api to fork off non
web processes? It helps save memory and cut down on master processes.

Cheers
Sam
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Forking non web processes

[Eric Wong]

Sam Saffron <sam.saffron@gmail.com> wrote:
> Hi Eric,
> 
> I have been trying to get unicorn to allow me to fork off non-web
> processes like sidekiq/resque.
> 
> I got this working, except that I am constantly fighting with the
> unicorn reaper. Any chance we can add some sort of api to fork off non
> web processes? It helps save memory and cut down on master processes.

I've been trying to avoid adding unicorn-specific APIs unless absolutely
necessary.

You're forking off from the master?  Worst case is you'll get a log
message about an unknown process, right?

I'm also wondering why... sidekiq/resque are standalone daemons
themselves.  Shouldn't that be done as part of the deploy/init process?
(unicorn isn't going to become init/upstart/systemd)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Forking non web processes

[Alex Sharp]

On Thu, Oct 24, 2013 at 9:17 AM, Eric Wong <normalperson@yhbt.net> wrote:
> I'm also wondering why... sidekiq/resque are standalone daemons
> themselves.  Shouldn't that be done as part of the deploy/init process?
> (unicorn isn't going to become init/upstart/systemd)

Agree with Eric here. You probably want to run unicorn and sidekiq /
resque in a way that they're not coupled to one another. They should
have different startup scripts and monitoring properties. And
eventually you may want to move your background worker processes to
another machine.

- alex sharp
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Forking non web processes

[Sam Saffron]

The only reason for this level of crazy is to get the 30% memory
saving you do when forking off a master in Ruby 2.0s CoW friendly GC.
I understand this is less traditional, but perfectly acceptable for
low cost VPS hosting.

The unicorn master does "reap/terminate" the child fork on startup, so
you are forced to hold off on forking.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Forking non web processes

[Eric Wong]

Sam Saffron <sam.saffron@gmail.com> wrote:
> The only reason for this level of crazy is to get the 30% memory
> saving you do when forking off a master in Ruby 2.0s CoW friendly GC.
> I understand this is less traditional, but perfectly acceptable for
> low cost VPS hosting.
> 
> The unicorn master does "reap/terminate" the child fork on startup, so
> you are forced to hold off on forking.

You can probably replace a worker process in the after_fork hook

(totally untested)

after_fork do |server, worker|
  if worker.nr == 1
    server.listeners = [] # drop listen sockets

    # keep ticking
    Thread.new { worker.tick = Time.now.to_i while sleep(server.timeout/2) }
    begin
      run_your_own_thing
    ensure
      exit
    end
  end
end

Or even just fork off via your Rack app.  This will be more portable in
case you want to try a different server.  unicorn workers do nothing
special with SIGCHLD, so your app can do whatever it wants with it.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Forking non web processes

[Sam Saffron]

Nice, will try this out, thank you

On Sat, Oct 26, 2013 at 7:44 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Sam Saffron <sam.saffron@gmail.com> wrote:
>> The only reason for this level of crazy is to get the 30% memory
>> saving you do when forking off a master in Ruby 2.0s CoW friendly GC.
>> I understand this is less traditional, but perfectly acceptable for
>> low cost VPS hosting.
>>
>> The unicorn master does "reap/terminate" the child fork on startup, so
>> you are forced to hold off on forking.
>
> You can probably replace a worker process in the after_fork hook
>
> (totally untested)
>
> after_fork do |server, worker|
>   if worker.nr == 1
>     server.listeners = [] # drop listen sockets
>
>     # keep ticking
>     Thread.new { worker.tick = Time.now.to_i while sleep(server.timeout/2) }
>     begin
>       run_your_own_thing
>     ensure
>       exit
>     end
>   end
> end
>
> Or even just fork off via your Rack app.  This will be more portable in
> case you want to try a different server.  unicorn workers do nothing
> special with SIGCHLD, so your app can do whatever it wants with it.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying