Re: Unicorn - "Still a couple bugs to knock out."

[Eric Wong <normalperson@yhbt.net>]

Dan Herrera <dan@revelationglobal.com> wrote:
> Hi Eric,
> 
> Thanks for getting in touch.  We were also pleasantly surprised by the
> performance of unicorn!
> 
> As to the blog post, we misspoke and will revise our language on the post to
> reflect that what we encountered weren't bugs, but issues with our own
> setup.

Thanks for the clarification, I was worried :x

> Our app has a memory leak, and we haven't yet settled on a monit recipe to
> restart unicorn when the sub processes that are started run over a certain
> memory threshold.  There were also some changes we needed to make with our
> nginx configuration, but again, they didn't reflect any problems with
> unicorn.

You can probably have monit just SIGQUIT the workers that run over the
memory threshold and just let the unicorn master restart them (note: I'm
not a monit user).

If your OS supports it, you can try setrlimit in your after_fork hook.
Modern Linux 2.6 doesn't support Process::RLIMIT_RSS, but you can still
do Process::RLIMIT_AS which is still somewhat useful[1].

  after_fork do |server, worker|
    Process.setrlimit(Process::RLIMIT_AS, size_in_kilobytes)
  end

Then have a simple Rack middleware that traps NoMemoryError:

  # XXX totally untested!, I'm not sure what to do if Process.kill
  # here hits NoMemoryError, either...
  class ExitOnOOM < Struct.new(:app)
    def call(env)
      begin
        app.call(env)
      rescue NoMemoryError
        Process.kill(:QUIT, 0) # graceful exit signal
        # or maybe just: exit!
      end
    end
  end

[1] On modern GNU/Linux systems that don't have RLIMIT_RSS, the VM size
    taken by the mmap()'ed shared libraries usually doesn't change over the
    lifetime of the process, so you can use that as a baseline and figure
    out how far you want to go from there....  Of course, your Unicorns
    shouldn't be swapping so swap usage won't have to be accounted for.

Of course the proper solution to all this is to fix your memory leaks :)

I posted some notes here a few months back on the Mongrel list:

  http://thread.gmane.org/gmane.comp.lang.ruby.mongrel.general/5760/focus=5766

> Engine Yard doesn't currently support unicorn on their slices, but once we
> have a satisfactory monit recipe and no other issues with our setup, we plan
> to work with them so that they'll hopefully change their mind.  I think
> this, and the monit recipe is what James meant when he said we wouldn't be
> using unicorn in the near future.

Ah, didn't know that.  I guess github was a special case where they
thought: "oh well, these guys are leaving us for another host anyways,
they can be our guinea pig!" :)

> Anyway, we are all great fans, and would like to thank all of you for all of
> your hard work.

No problem :)

-- 
Eric Wong