* Exception `Errno::EAGAIN'
@ 2009-11-15 23:21 James Turnbull
2009-11-15 23:47 ` Eric Wong
0 siblings, 1 reply; 3+ messages in thread
From: James Turnbull @ 2009-11-15 23:21 UTC (permalink / raw)
To: unicorn list
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I get this repeated error - about once a second:
Exception `Errno::EAGAIN' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)
Exception `Errno::EAGAIN' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)
Exception `Errno::EAGAIN' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)
Which seems to be related to:
ready.each do |sock|
begin
process_client(sock.accept_nonblock)
nr += 1
alive.chmod(m = 0 == m ? 1 : 0)
rescue Errno::EAGAIN, Errno::ECONNABORTED
end
break if nr < 0
end
My unicorn.conf is:
worker_processes 4
working_directory "/etc/puppet"
listen '/tmp/puppet.sock', :backlog => 1
listen 8140, :tcp_nopush => true
timeout 10
pid "/tmp/puppet.pid"
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
GC.copy_on_write_friendly = true
before_fork do |server, worker|
# the following is recomended for Rails + "preload_app true"
# as there's no need for the master process to hold a connection
# defined?(ActiveRecord::Base) and
# ActiveRecord::Base.connection.disconnect!
# the following allows a new master process to incrementally
# phase out the old master process with SIGTTOU to avoid a
# thundering herd (especially in the "preload_app false" case)
# when doing a transparent upgrade. The last worker spawned
# will then kill off the old master process with a SIGQUIT.
old_pid = "#{server.config[:pid]}.oldbin"
if old_pid != server.pid
begin
sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
Process.kill(sig, File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
end
end
# optionally throttle the master from forking too quickly by
sleeping
sleep 1
end
The platform is Fedora 10, Ruby 1.8.6.
Oddly, if I comment out the preload I get:
Exception `Errno::EPERM' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639 -
Operation not permitted - /tmp/0.520680132392046
Unhandled listen loop exception #<Errno::EPERM: Operation not
permitted - /tmp/0.520680132392046>.
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639:in `chmod'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639:in
`worker_loop'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in `fork'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:530:in `each'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:530:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:540:in
`maintain_worker_count'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:215:in `start'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:28:in `run'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/bin/unicorn:165
/usr/bin/unicorn:19:in `load'
/usr/bin/unicorn:19
Repeatedly.
Regards
James Turnbull
- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEVAwUBSwCM/iFa/lDkFHAyAQIiSggA5r3rFPGiY5AeratLgxPvUDE1OjnrhRGn
HxoRAcM901BNvXT5i7fX5fMTayW7vPXiKd0IazottCDgd40TYlFTQZNpU9hUDx+g
LRAKQ5M5gHTkFX6nqxFlys8WIA3xseDBBUBNxGnZ4YGbqUMg7tQ258b41hiU+4pW
fxQgzYN7JjRQ7/u1Px+86aEEdSGHFYNAOSwG15b+iE/7hQbiWrlFIL7p7XcZqT8M
UxDm6eTkhP1kry7zJ4wTxVsreXgCi75AaCJBfIIoKs0NMParJ/rBWB+42ZA+1t5/
aEaKXBOi47Kw3CmbsSh6Bi4zbTyuavCX6/tfOMNXx6jXygvykUzpgw==
=pJbE
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Exception `Errno::EAGAIN'
2009-11-15 23:21 Exception `Errno::EAGAIN' James Turnbull
@ 2009-11-15 23:47 ` Eric Wong
2009-11-16 0:08 ` James Turnbull
0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2009-11-15 23:47 UTC (permalink / raw)
To: unicorn list
James Turnbull <james@lovedthanlost.net> wrote:
> I get this repeated error - about once a second:
>
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
>
> Which seems to be related to:
>
> ready.each do |sock|
> begin
> process_client(sock.accept_nonblock)
> nr += 1
> alive.chmod(m = 0 == m ? 1 : 0)
> rescue Errno::EAGAIN, Errno::ECONNABORTED
> end
> break if nr < 0
> end
Hi James,
This is expected. The kernel wakes up all the workers when there's
_one_ connection and they all race to accept() one client connection.
One wins and accepts the connection, 3 lose and go back to sleep. With
lots of worker processes, this can be a thundering herd problem but
that's why we encourage Unicorn as a backend server, not as a frontend
server.
If we only have one listener, we could do a blocking accept() call to
avoid thundering herds under Linux, but testing on a 16-core box, I
don't remember being able to measure a performance improvement. But
otherwise we have to use select() with multiple listeners. nginx uses a
variety of non-portable locking mechanisms to implement its accept
mutex, but that's too much work under Ruby for little/no benefit. In
the nginx case, sometimes even disabling the accept mutex entirely gives
better performance.
> My unicorn.conf is:
>
> worker_processes 4
> working_directory "/etc/puppet"
> listen '/tmp/puppet.sock', :backlog => 1
> listen 8140, :tcp_nopush => true
> timeout 10
> pid "/tmp/puppet.pid"
>
> preload_app true
> GC.respond_to?(:copy_on_write_friendly=) and
> GC.copy_on_write_friendly = true
>
> before_fork do |server, worker|
> # the following is recomended for Rails + "preload_app true"
> # as there's no need for the master process to hold a connection
> # defined?(ActiveRecord::Base) and
> # ActiveRecord::Base.connection.disconnect!
>
> # the following allows a new master process to incrementally
> # phase out the old master process with SIGTTOU to avoid a
> # thundering herd (especially in the "preload_app false" case)
> # when doing a transparent upgrade. The last worker spawned
> # will then kill off the old master process with a SIGQUIT.
> old_pid = "#{server.config[:pid]}.oldbin"
> if old_pid != server.pid
> begin
> sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
> Process.kill(sig, File.read(old_pid).to_i)
> rescue Errno::ENOENT, Errno::ESRCH
> end
> end
>
> # optionally throttle the master from forking too quickly by
> sleeping
> sleep 1
> end
>
> The platform is Fedora 10, Ruby 1.8.6.
>
> Oddly, if I comment out the preload I get:
>
> Exception `Errno::EPERM' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639 -
> Operation not permitted - /tmp/0.520680132392046
Do you by any chance do user switching in your application
or config.ru?
Instead, since 0.94.0, I would do this:
after_fork do |server, worker|
worker.user('user', 'group') if Process.euid == 0
end
That way only the worker process drops permissions. The master really
doesn't do anything interesting. Of course *I* would just start
Unicorn as a regular user and forget the complexity of user switching
entirely...
--
Eric Wong
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Exception `Errno::EAGAIN'
2009-11-15 23:47 ` Eric Wong
@ 2009-11-16 0:08 ` James Turnbull
0 siblings, 0 replies; 3+ messages in thread
From: James Turnbull @ 2009-11-16 0:08 UTC (permalink / raw)
To: unicorn list
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Eric Wong wrote:
> This is expected. The kernel wakes up all the workers when there's
> _one_ connection and they all race to accept() one client connection.
> One wins and accepts the connection, 3 lose and go back to sleep. With
> lots of worker processes, this can be a thundering herd problem but
> that's why we encourage Unicorn as a backend server, not as a frontend
> server.
Ah okay - makes sense.
> Do you by any chance do user switching in your application
> or config.ru?
In the application we do.
> Instead, since 0.94.0, I would do this:
>
> after_fork do |server, worker|
> worker.user('user', 'group') if Process.euid == 0
> end
>
> That way only the worker process drops permissions. The master really
> doesn't do anything interesting. Of course *I* would just start
> Unicorn as a regular user and forget the complexity of user switching
> entirely...
Okay thanks - will give that a shot.
Regards
James Turnbull
- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEVAwUBSwCYGCFa/lDkFHAyAQIyPggA5XPzSOYgf4ekdBGoRcs4wp885eC8C/wY
BIjzdxilCEhA8mqBw4Hrjo5sOebglOukACFla7QaDqyk2t1velQEpxj8sR2StNEe
HHTbng/Jq3ryt4BxNucQWdKvdPTI3WJ3I9APqtGxNyh72FpY08uvAlUv02S8b/Aj
mtFqbqi2gu+tkbOoqW1AeJEabjJHd7HyPhf2YlHowBaHG61bzrFRnuhvqN8R6SAq
j3LI1kRpnTYdwKYEy3sLfR0OUSVmUuDkWjRkYt2tR2fAwCKkq2oupmcDUrvaT2Y2
fbQ8YJzoGjwVVvD1On/u6iNiTHZcOReYiHRsAhmtbaS+eHldSukHQw==
=EyRA
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2009-11-16 0:09 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-11-15 23:21 Exception `Errno::EAGAIN' James Turnbull
2009-11-15 23:47 ` Eric Wong
2009-11-16 0:08 ` James Turnbull
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).