Unicorn - "Still a couple bugs to knock out."

Unicorn - "Still a couple bugs to knock out."

[Eric Wong]

Hi there,

We noticed your blog entry[1] and are pleasantly surprised with
the performance compared to existing servers.

However, this bit has us worried, care to elaborate so we can track
down and fix them?

> We aren't going to be jumping to unicorn in the near future.
> Still a couple bugs to knock out.

We released a fix in v0.93.2 last night to work around truncated POST
bodies in Rails.  Did you encounter something else?

Thanks!

[1] http://labs.revelationglobal.com/2009/10/06/mongrel_passenger_unicorn.html

-- 
Eric Wong

Re: Unicorn - "Still a couple bugs to knock out."

[Chris Wanstrath]

On Wed, Oct 7, 2009 at 5:31 PM, Eric Wong <normalperson@yhbt.net> wrote:

> We noticed your blog entry[1] and are pleasantly surprised with
> the performance compared to existing servers.

Note that the blog post features an embedded Gist, served via Unicorn ;)

-- 
Chris Wanstrath
http://github.com/defunkt

Re: Unicorn - "Still a couple bugs to knock out."

[Eric Wong]

Dan Herrera <dan@revelationglobal.com> wrote:
> Hi Eric,
> 
> Thanks for getting in touch.  We were also pleasantly surprised by the
> performance of unicorn!
> 
> As to the blog post, we misspoke and will revise our language on the post to
> reflect that what we encountered weren't bugs, but issues with our own
> setup.

Thanks for the clarification, I was worried :x

> Our app has a memory leak, and we haven't yet settled on a monit recipe to
> restart unicorn when the sub processes that are started run over a certain
> memory threshold.  There were also some changes we needed to make with our
> nginx configuration, but again, they didn't reflect any problems with
> unicorn.

You can probably have monit just SIGQUIT the workers that run over the
memory threshold and just let the unicorn master restart them (note: I'm
not a monit user).

If your OS supports it, you can try setrlimit in your after_fork hook.
Modern Linux 2.6 doesn't support Process::RLIMIT_RSS, but you can still
do Process::RLIMIT_AS which is still somewhat useful[1].

  after_fork do |server, worker|
    Process.setrlimit(Process::RLIMIT_AS, size_in_kilobytes)
  end

Then have a simple Rack middleware that traps NoMemoryError:

  # XXX totally untested!, I'm not sure what to do if Process.kill
  # here hits NoMemoryError, either...
  class ExitOnOOM < Struct.new(:app)
    def call(env)
      begin
        app.call(env)
      rescue NoMemoryError
        Process.kill(:QUIT, 0) # graceful exit signal
        # or maybe just: exit!
      end
    end
  end

[1] On modern GNU/Linux systems that don't have RLIMIT_RSS, the VM size
    taken by the mmap()'ed shared libraries usually doesn't change over the
    lifetime of the process, so you can use that as a baseline and figure
    out how far you want to go from there....  Of course, your Unicorns
    shouldn't be swapping so swap usage won't have to be accounted for.

Of course the proper solution to all this is to fix your memory leaks :)

I posted some notes here a few months back on the Mongrel list:

  http://thread.gmane.org/gmane.comp.lang.ruby.mongrel.general/5760/focus=5766

> Engine Yard doesn't currently support unicorn on their slices, but once we
> have a satisfactory monit recipe and no other issues with our setup, we plan
> to work with them so that they'll hopefully change their mind.  I think
> this, and the monit recipe is what James meant when he said we wouldn't be
> using unicorn in the near future.

Ah, didn't know that.  I guess github was a special case where they
thought: "oh well, these guys are leaving us for another host anyways,
they can be our guinea pig!" :)

> Anyway, we are all great fans, and would like to thank all of you for all of
> your hard work.

No problem :)

-- 
Eric Wong