Exception `Errno::EAGAIN'

Exception `Errno::EAGAIN'

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I get this repeated error - about once a second:

Exception `Errno::EAGAIN' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)
Exception `Errno::EAGAIN' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)
Exception `Errno::EAGAIN' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)

Which seems to be related to:

        ready.each do |sock|
          begin
            process_client(sock.accept_nonblock)
            nr += 1
            alive.chmod(m = 0 == m ? 1 : 0)
          rescue Errno::EAGAIN, Errno::ECONNABORTED
          end
          break if nr < 0
        end

My unicorn.conf is:

worker_processes 4
working_directory "/etc/puppet"
listen '/tmp/puppet.sock', :backlog => 1
listen 8140, :tcp_nopush => true
timeout 10
pid "/tmp/puppet.pid"

  preload_app true
  GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

  before_fork do |server, worker|
    # the following is recomended for Rails + "preload_app true"
    # as there's no need for the master process to hold a connection
#    defined?(ActiveRecord::Base) and
#      ActiveRecord::Base.connection.disconnect!

    # the following allows a new master process to incrementally
    # phase out the old master process with SIGTTOU to avoid a
    # thundering herd (especially in the "preload_app false" case)
    # when doing a transparent upgrade.  The last worker spawned
    # will then kill off the old master process with a SIGQUIT.
    old_pid = "#{server.config[:pid]}.oldbin"
    if old_pid != server.pid
      begin
        sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
        Process.kill(sig, File.read(old_pid).to_i)
      rescue Errno::ENOENT, Errno::ESRCH
      end
    end

    # optionally throttle the master from forking too quickly by
sleeping
    sleep 1
  end

The platform is Fedora 10, Ruby 1.8.6.

Oddly, if I comment out the preload I get:

Exception `Errno::EPERM' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639 -
Operation not permitted - /tmp/0.520680132392046
Unhandled listen loop exception #<Errno::EPERM: Operation not
permitted - /tmp/0.520680132392046>.
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639:in `chmod'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639:in
`worker_loop'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in `fork'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:530:in `each'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:530:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:540:in
`maintain_worker_count'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:215:in `start'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:28:in `run'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/bin/unicorn:165
/usr/bin/unicorn:19:in `load'
/usr/bin/unicorn:19

Repeatedly.

Regards

James Turnbull

- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSwCM/iFa/lDkFHAyAQIiSggA5r3rFPGiY5AeratLgxPvUDE1OjnrhRGn
HxoRAcM901BNvXT5i7fX5fMTayW7vPXiKd0IazottCDgd40TYlFTQZNpU9hUDx+g
LRAKQ5M5gHTkFX6nqxFlys8WIA3xseDBBUBNxGnZ4YGbqUMg7tQ258b41hiU+4pW
fxQgzYN7JjRQ7/u1Px+86aEEdSGHFYNAOSwG15b+iE/7hQbiWrlFIL7p7XcZqT8M
UxDm6eTkhP1kry7zJ4wTxVsreXgCi75AaCJBfIIoKs0NMParJ/rBWB+42ZA+1t5/
aEaKXBOi47Kw3CmbsSh6Bi4zbTyuavCX6/tfOMNXx6jXygvykUzpgw==
=pJbE
-----END PGP SIGNATURE-----

Re: Exception `Errno::EAGAIN'

[Eric Wong]

James Turnbull <james@lovedthanlost.net> wrote:
> I get this repeated error - about once a second:
> 
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> 
> Which seems to be related to:
> 
>         ready.each do |sock|
>           begin
>             process_client(sock.accept_nonblock)
>             nr += 1
>             alive.chmod(m = 0 == m ? 1 : 0)
>           rescue Errno::EAGAIN, Errno::ECONNABORTED
>           end
>           break if nr < 0
>         end

Hi James,

This is expected.  The kernel wakes up all the workers when there's
_one_ connection and they all race to accept() one client connection.
One wins and accepts the connection, 3 lose and go back to sleep.  With
lots of worker processes, this can be a thundering herd problem but
that's why we encourage Unicorn as a backend server, not as a frontend
server.

If we only have one listener, we could do a blocking accept() call to
avoid thundering herds under Linux, but testing on a 16-core box, I
don't remember being able to measure a performance improvement.  But
otherwise we have to use select() with multiple listeners.  nginx uses a
variety of non-portable locking mechanisms to implement its accept
mutex, but that's too much work under Ruby for little/no benefit.  In
the nginx case, sometimes even disabling the accept mutex entirely gives
better performance.

> My unicorn.conf is:
> 
> worker_processes 4
> working_directory "/etc/puppet"
> listen '/tmp/puppet.sock', :backlog => 1
> listen 8140, :tcp_nopush => true
> timeout 10
> pid "/tmp/puppet.pid"
> 
>   preload_app true
>   GC.respond_to?(:copy_on_write_friendly=) and
>   GC.copy_on_write_friendly = true
> 
>   before_fork do |server, worker|
>     # the following is recomended for Rails + "preload_app true"
>     # as there's no need for the master process to hold a connection
> #    defined?(ActiveRecord::Base) and
> #      ActiveRecord::Base.connection.disconnect!
> 
>     # the following allows a new master process to incrementally
>     # phase out the old master process with SIGTTOU to avoid a
>     # thundering herd (especially in the "preload_app false" case)
>     # when doing a transparent upgrade.  The last worker spawned
>     # will then kill off the old master process with a SIGQUIT.
>     old_pid = "#{server.config[:pid]}.oldbin"
>     if old_pid != server.pid
>       begin
>         sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
>         Process.kill(sig, File.read(old_pid).to_i)
>       rescue Errno::ENOENT, Errno::ESRCH
>       end
>     end
> 
>     # optionally throttle the master from forking too quickly by
> sleeping
>     sleep 1
>   end
> 
> The platform is Fedora 10, Ruby 1.8.6.
> 
> Oddly, if I comment out the preload I get:
> 
> Exception `Errno::EPERM' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639 -
> Operation not permitted - /tmp/0.520680132392046

Do you by any chance do user switching in your application
or config.ru?

Instead, since 0.94.0, I would do this:

	after_fork do |server, worker|
	  worker.user('user', 'group') if Process.euid == 0
	end

That way only the worker process drops permissions.  The master really
doesn't do anything interesting.   Of course *I* would just start
Unicorn as a regular user and forget the complexity of user switching
entirely...

-- 
Eric Wong

Re: Exception `Errno::EAGAIN'

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eric Wong wrote:
> This is expected.  The kernel wakes up all the workers when there's
> _one_ connection and they all race to accept() one client connection.
> One wins and accepts the connection, 3 lose and go back to sleep.  With
> lots of worker processes, this can be a thundering herd problem but
> that's why we encourage Unicorn as a backend server, not as a frontend
> server.

Ah okay - makes sense.

> Do you by any chance do user switching in your application
> or config.ru?

In the application we do.

> Instead, since 0.94.0, I would do this:
> 
> 	after_fork do |server, worker|
> 	  worker.user('user', 'group') if Process.euid == 0
> 	end
> 
> That way only the worker process drops permissions.  The master really
> doesn't do anything interesting.   Of course *I* would just start
> Unicorn as a regular user and forget the complexity of user switching
> entirely...

Okay thanks - will give that a shot.

Regards

James Turnbull

- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSwCYGCFa/lDkFHAyAQIyPggA5XPzSOYgf4ekdBGoRcs4wp885eC8C/wY
BIjzdxilCEhA8mqBw4Hrjo5sOebglOukACFla7QaDqyk2t1velQEpxj8sR2StNEe
HHTbng/Jq3ryt4BxNucQWdKvdPTI3WJ3I9APqtGxNyh72FpY08uvAlUv02S8b/Aj
mtFqbqi2gu+tkbOoqW1AeJEabjJHd7HyPhf2YlHowBaHG61bzrFRnuhvqN8R6SAq
j3LI1kRpnTYdwKYEy3sLfR0OUSVmUuDkWjRkYt2tR2fAwCKkq2oupmcDUrvaT2Y2
fbQ8YJzoGjwVVvD1On/u6iNiTHZcOReYiHRsAhmtbaS+eHldSukHQw==
=EyRA
-----END PGP SIGNATURE-----