Re: Exception `Errno::EAGAIN'

[Eric Wong <normalperson@yhbt.net>]

James Turnbull <james@lovedthanlost.net> wrote:
> I get this repeated error - about once a second:
> 
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> 
> Which seems to be related to:
> 
>         ready.each do |sock|
>           begin
>             process_client(sock.accept_nonblock)
>             nr += 1
>             alive.chmod(m = 0 == m ? 1 : 0)
>           rescue Errno::EAGAIN, Errno::ECONNABORTED
>           end
>           break if nr < 0
>         end

Hi James,

This is expected.  The kernel wakes up all the workers when there's
_one_ connection and they all race to accept() one client connection.
One wins and accepts the connection, 3 lose and go back to sleep.  With
lots of worker processes, this can be a thundering herd problem but
that's why we encourage Unicorn as a backend server, not as a frontend
server.

If we only have one listener, we could do a blocking accept() call to
avoid thundering herds under Linux, but testing on a 16-core box, I
don't remember being able to measure a performance improvement.  But
otherwise we have to use select() with multiple listeners.  nginx uses a
variety of non-portable locking mechanisms to implement its accept
mutex, but that's too much work under Ruby for little/no benefit.  In
the nginx case, sometimes even disabling the accept mutex entirely gives
better performance.

> My unicorn.conf is:
> 
> worker_processes 4
> working_directory "/etc/puppet"
> listen '/tmp/puppet.sock', :backlog => 1
> listen 8140, :tcp_nopush => true
> timeout 10
> pid "/tmp/puppet.pid"
> 
>   preload_app true
>   GC.respond_to?(:copy_on_write_friendly=) and
>   GC.copy_on_write_friendly = true
> 
>   before_fork do |server, worker|
>     # the following is recomended for Rails + "preload_app true"
>     # as there's no need for the master process to hold a connection
> #    defined?(ActiveRecord::Base) and
> #      ActiveRecord::Base.connection.disconnect!
> 
>     # the following allows a new master process to incrementally
>     # phase out the old master process with SIGTTOU to avoid a
>     # thundering herd (especially in the "preload_app false" case)
>     # when doing a transparent upgrade.  The last worker spawned
>     # will then kill off the old master process with a SIGQUIT.
>     old_pid = "#{server.config[:pid]}.oldbin"
>     if old_pid != server.pid
>       begin
>         sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
>         Process.kill(sig, File.read(old_pid).to_i)
>       rescue Errno::ENOENT, Errno::ESRCH
>       end
>     end
> 
>     # optionally throttle the master from forking too quickly by
> sleeping
>     sleep 1
>   end
> 
> The platform is Fedora 10, Ruby 1.8.6.
> 
> Oddly, if I comment out the preload I get:
> 
> Exception `Errno::EPERM' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639 -
> Operation not permitted - /tmp/0.520680132392046

Do you by any chance do user switching in your application
or config.ru?

Instead, since 0.94.0, I would do this:

	after_fork do |server, worker|
	  worker.user('user', 'group') if Process.euid == 0
	end

That way only the worker process drops permissions.  The master really
doesn't do anything interesting.   Of course *I* would just start
Unicorn as a regular user and forget the complexity of user switching
entirely...

-- 
Eric Wong