* Re: unicorn 1.1.x never-ending listen loop IOError exceptions
2011-03-22 19:19 7% unicorn 1.1.x never-ending listen loop IOError exceptions David Nghiem
@ 2011-03-22 21:39 0% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2011-03-22 21:39 UTC (permalink / raw)
To: unicorn list
David Nghiem <david@zendesk.com> wrote:
> We run unicorn 1.1.5 behind nginx under Ruby Enterprise Edition
> 2011.02. We're using UNIX sockets instead of TCP sockets.
>
> Once in a while, one worker will throw exceptions in a loop, taking up
> lots of CPU and log space. Killing the worker removes the problem.
> Other workers are not affected.
>
> Here's the exception:
>
> E, [2011-01-10T10:13:13.518795 #15209] ERROR -- : Unhandled listen
> loop exception #<IOError: closed stream>.
> E, [2011-01-10T10:13:13.518881 #15209] ERROR -- :
> /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:732:in
> `select'
>
> I don't get any other exceptions. I've found other people have this
> "IOError: closed stream" error coupled with "Errno::EMFILE" but we
> have plenty of file-descriptors left.
No Errno::EBADF errors at all?
> Here's an lsof dump of the worker:
>
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
<snip> It looks like log reopening failed, below:
> ruby 15209 zendesk 1w REG 252,0 0
> 3015360 /data/zendesk/log/unicorn.stdout.log
> ruby 15209 zendesk 2w REG 252,0 24585
> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
> ruby 15209 zendesk 3w REG 252,0 61617590073
> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
Permissions issue during log rotation? Are you doing "killall -USR1
unicorn" by any chance or sending a bunch of USR1 signals faster than
unicorn can process them?
Try just sending one USR1 to the master process and letting the master
signal the workers.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* unicorn 1.1.x never-ending listen loop IOError exceptions
@ 2011-03-22 19:19 7% David Nghiem
2011-03-22 21:39 0% ` Eric Wong
0 siblings, 1 reply; 3+ results
From: David Nghiem @ 2011-03-22 19:19 UTC (permalink / raw)
To: mongrel-unicorn
We run unicorn 1.1.5 behind nginx under Ruby Enterprise Edition
2011.02. We're using UNIX sockets instead of TCP sockets.
Once in a while, one worker will throw exceptions in a loop, taking up
lots of CPU and log space. Killing the worker removes the problem.
Other workers are not affected.
Here's the exception:
E, [2011-01-10T10:13:13.518795 #15209] ERROR -- : Unhandled listen
loop exception #<IOError: closed stream>.
E, [2011-01-10T10:13:13.518881 #15209] ERROR -- :
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:732:in
`select'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:732:in
`worker_loop'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/newrelic_rpm-2.13.1/lib/new_relic/control/../agent/instrumentation/unicorn_instrumentation.rb:7:in
`call'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/newrelic_rpm-2.13.1/lib/new_relic/control/../agent/instrumentation/unicorn_instrumentation.rb:7:in
`worker_loop'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:603:in
`spawn_missing_workers'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:600:in
`fork'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:600:in
`spawn_missing_workers'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:596:in
`each'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:596:in
`spawn_missing_workers'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:610:in
`maintain_worker_count'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:407:in
`join'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:395:in
`loop'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:395:in
`join'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:29:in
`run'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/bin/unicorn_rails:210
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/bin/unicorn_rails:19:in
`load'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/bin/unicorn_rails:19
I don't get any other exceptions. I've found other people have this
"IOError: closed stream" error coupled with "Errno::EMFILE" but we
have plenty of file-descriptors left.
Here's an lsof dump of the worker:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 15209 zendesk cwd DIR 252,0 4096
2940970 /data/zendesk/releases/alpha
ruby 15209 zendesk rtd DIR 8,1 4096 2 /
ruby 15209 zendesk txt REG 8,1 2831149
246513 /usr/local/rvm/rubies/ree-1.8.7-2010.02/bin/ruby
ruby 15209 zendesk mem REG 8,1 10192
146145 /usr/lib/gconv/ISO8859-1.so
ruby 15209 zendesk mem REG 8,1 22848
263831 /lib/libnss_dns-2.9.so
ruby 15209 zendesk mem REG 8,1 9760
260915 /lib/libnss_mdns4_minimal.so.2
ruby 15209 zendesk mem REG 8,1 43472
263939 /lib/libnss_nis-2.9.so
ruby 15209 zendesk mem REG 8,1 35632
263929 /lib/libnss_compat-2.9.so
ruby 15209 zendesk mem REG 8,1 51616
263934 /lib/libnss_files-2.9.so
ruby 15209 zendesk mem REG 252,0 35772
3377231 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/em-http-request-0.2.7/lib/em_buffer.so
ruby 15209 zendesk mem REG 252,0 45360
3377237 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/em-http-request-0.2.7/lib/http11_client.so
ruby 15209 zendesk mem REG 252,0 1175256
3377180 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/eventmachine-0.12.10/lib/rubyeventmachine.so
ruby 15209 zendesk mem REG 8,1 35184
155430 /usr/lib/sasl2/libntlm.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18672
155418 /usr/lib/sasl2/libanonymous.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18672
155421 /usr/lib/sasl2/libcrammd5.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18640
155433 /usr/lib/sasl2/libplain.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18640
155427 /usr/lib/sasl2/liblogin.so.2.0.22
ruby 15209 zendesk mem REG 8,1 1297736
140958 /usr/lib/libdb-4.6.so
ruby 15209 zendesk mem REG 8,1 22616
155410 /usr/lib/sasl2/libsasldb.so.2.0.22
ruby 15209 zendesk mem REG 8,1 52000
155424 /usr/lib/sasl2/libdigestmd5.so.2.0.22
ruby 15209 zendesk mem REG 8,1 88824
263949 /lib/libresolv-2.9.so
ruby 15209 zendesk mem REG 8,1 105760
139781 /usr/lib/libsasl2.so.2.0.22
ruby 15209 zendesk mem REG 252,0 4110920
3384825 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/memcached-0.20.1/lib/rlibmemcached.so
ruby 15209 zendesk mem REG 8,1 97168
263938 /lib/libnsl-2.9.so
ruby 15209 zendesk mem REG 8,1 2086600
145819 /usr/lib/libmysqlclient.so.15.0.0
ruby 15209 zendesk mem REG 252,0 169011
3385005 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/mysql-2.8.1/lib/mysql_api.so
ruby 15209 zendesk mem REG 252,0 150950
3401474 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/yajl-ruby-0.7.6/lib/yajl_ext.so
ruby 15209 zendesk mem REG 252,0 29311
3367472 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/SystemTimer-1.2/lib/system_timer_native.so
ruby 15209 zendesk mem REG 8,1 13144
260768 /lib/libgpg-error.so.0.3.0
ruby 15209 zendesk mem REG 8,1 422664
260765 /lib/libgcrypt.so.11.4.4
ruby 15209 zendesk mem REG 8,1 1425368
141140 /usr/lib/libxml2.so.2.6.32
ruby 15209 zendesk mem REG 8,1 233152
439848 /usr/lib/libxslt.so.1.1.24
ruby 15209 zendesk mem REG 8,1 80792
439847 /usr/lib/libexslt.so.0.8.13
ruby 15209 zendesk mem REG 252,0 417564
3385563 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/nokogiri-1.4.1/lib/nokogiri/nokogiri.so
ruby 15209 zendesk mem REG 8,1 14220
252540 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/digest/md5.so
ruby 15209 zendesk mem REG 8,1 42715
252537 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/racc/cparse.so
ruby 15209 zendesk mem REG 252,0 121676
3465790 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/json-1.4.6/ext/json/ext/json/ext/generator.so
ruby 15209 zendesk mem REG 252,0 52561
3465795 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/json-1.4.6/ext/json/ext/json/ext/parser.so
ruby 15209 zendesk mem REG 252,0 20227
3401531 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/fast_xs-0.7.3/lib/fast_xs.so
ruby 15209 zendesk mem REG 8,1 46499
255504 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/site_ruby/1.8/x86_64-linux/iconv.so
ruby 15209 zendesk mem REG 8,1 139431
247137 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/bigdecimal.so
ruby 15209 zendesk mem REG 8,1 433200
247152 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/nkf.so
ruby 15209 zendesk mem REG 252,0 106488
3531787 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn_http.so
ruby 15209 zendesk mem REG 8,1 53645
247158 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/strscan.so
ruby 15209 zendesk mem REG 8,1 128208
247138 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/socket.so
ruby 15209 zendesk mem REG 8,1 12144
247141 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/fcntl.so
ruby 15209 zendesk mem REG 8,1 329504
260852 /lib/libssl.so.0.9.8
ruby 15209 zendesk mem REG 8,1 1002954
247156 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/openssl.so
ruby 15209 zendesk mem REG 8,1 110815
247146 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/zlib.so
ruby 15209 zendesk mem REG 8,1 35684
247140 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/digest.so
ruby 15209 zendesk mem REG 8,1 96768
260746 /lib/libz.so.1.2.3.3
ruby 15209 zendesk mem REG 8,1 1594824
260668 /lib/libcrypto.so.0.9.8
ruby 15209 zendesk mem REG 8,1 15204
252539 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/digest/sha1.so
ruby 15209 zendesk mem REG 8,1 12723
247159 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/rational.so
ruby 15209 zendesk mem REG 8,1 351320
247142 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/syck.so
ruby 15209 zendesk mem REG 8,1 57484
247154 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/stringio.so
ruby 15209 zendesk mem REG 8,1 29818
247150 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/etc.so
ruby 15209 zendesk mem REG 8,1 51878
247139 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/thread.so
ruby 15209 zendesk mem REG 8,1 130151
263933 /lib/libpthread-2.9.so
ruby 15209 zendesk mem REG 8,1 96560
260672 /lib/libgcc_s.so.1
ruby 15209 zendesk mem REG 8,1 1023448
139822 /usr/lib/libstdc++.so.6.0.10
ruby 15209 zendesk mem REG 8,1 1502512
263944 /lib/libc-2.9.so
ruby 15209 zendesk mem REG 8,1 542928
263942 /lib/libm-2.9.so
ruby 15209 zendesk mem REG 8,1 39112
263932 /lib/libcrypt-2.9.so
ruby 15209 zendesk mem REG 8,1 14608
263945 /lib/libdl-2.9.so
ruby 15209 zendesk mem REG 8,1 31656
263930 /lib/librt-2.9.so
ruby 15209 zendesk mem REG 8,1 728968
246427 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/libtcmalloc_minimal.so.0.0.0
ruby 15209 zendesk mem REG 8,1 135680
263936 /lib/ld-2.9.so
ruby 15209 zendesk mem REG 8,1 26048
139155 /usr/lib/gconv/gconv-modules.cache
ruby 15209 zendesk 0r CHR 1,3
3125 /dev/null
ruby 15209 zendesk 1w REG 252,0 0
3015360 /data/zendesk/log/unicorn.stdout.log
ruby 15209 zendesk 2w REG 252,0 24585
3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
ruby 15209 zendesk 3w REG 252,0 61617590073
3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
ruby 15209 zendesk 5u unix 0xffff88043bdf9080
49439 /var/run/zendesk/unicorn.sock
ruby 15209 zendesk 6w FIFO 0,6 5651378 pipe
ruby 15209 zendesk 7w REG 252,0 0
3015676 /data/zendesk/log/newrelic_agent.log
ruby 15209 zendesk 8w REG 252,0 68330048
3015673 /data/zendesk/log/production.log
ruby 15209 zendesk 9w REG 252,0 0
3015677 /data/zendesk/log/resque.log
ruby 15209 zendesk 10u IPv4 41707462
TCP 192.168.100.148:49364->192.168.100.183:mysql (ESTABLISHED)
ruby 15209 zendesk 11w REG 252,0 0
3015672 /data/zendesk/log/mail_processing_production.log
ruby 15209 zendesk 12u IPv4 55517
TCP 192.168.100.148:60198->192.168.100.182:mysql (ESTABLISHED)
ruby 15209 zendesk 13u IPv4 55521
TCP 192.168.100.148:42351->192.168.100.150:11211 (ESTABLISHED)
ruby 15209 zendesk 14u IPv4 55523
TCP 192.168.100.148:52409->192.168.100.148:11211 (ESTABLISHED)
ruby 15209 zendesk 15u IPv4 56105
TCP 192.168.100.148:58562->192.168.100.140:11211 (ESTABLISHED)
ruby 15209 zendesk 16u IPv4 56108
TCP 192.168.100.148:50500->192.168.100.144:11211 (ESTABLISHED)
ruby 15209 zendesk 17u IPv4 56110
TCP 192.168.100.148:46723->192.168.100.142:11211 (ESTABLISHED)
ruby 15209 zendesk 18u IPv4 56137
TCP 192.168.100.148:49568->192.168.100.146:11211 (ESTABLISHED)
ruby 15209 zendesk 19u IPv4 5845535
TCP 192.168.100.148:43836->192.168.100.183:mysql (ESTABLISHED)
ruby 15209 zendesk 20u IPv4 63603
TCP 192.168.100.148:41990->192.168.100.149:6379 (ESTABLISHED)
ruby 15209 zendesk 21u REG 8,1 0
90084 /tmp/0.961990208371916 (deleted)
ruby 15209 zendesk 22u IPv4 10795792
TCP 192.168.100.148:34587->192.168.100.183:mysql (ESTABLISHED)
ruby 15209 zendesk 23u IPv4 33013892
TCP 192.168.100.148:54382->192.168.100.183:mysql (ESTABLISHED)
I'm tempted to exit the worker when I get this exception N number of
times as a workaround; it tends to only happen when I'm sleeping =)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 7%]
* [ANN] Unicorn 2.0.0, 1.1.5, 1.0.2 released!
@ 2010-10-28 0:19 6% Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2010-10-28 0:19 UTC (permalink / raw)
To: mongrel-unicorn
tldr:
2.0.0 - cleanups for Rainbows!, but should be ready for general use
barring portability issues
1.1.5 - bug fixes
1.0.2 - bug fixes
Please send any and all feedback to the mailing list. Do not waste
bandwidth with HTML or signature attachments.
email: mongrel-unicorn@rubyforge.org
git: git://git.bogomips.org/unicorn.git
The longer change summaries/release notes:
== unicorn 2.0.0 - mostly internal cleanups
Despite the version number, this release mostly features
internal cleanups for future versions of Rainbows!. User
visible changes include reductions in CPU wakeups on idle sites
using high timeouts.
Barring possible portability issues due to the introduction of
the kgio library, this release should be ready for all to use.
However, 1.1.x (and possibly 1.0.x) will continue to be
maintained. Unicorn 1.1.5 and 1.0.2 have also been released
with bugfixes found during development of 2.0.0.
== unicorn 1.1.5
This maintenance release fixes several long-standing but
recently-noticed bugs. SIGHUP reloading now correctly restores
default values if they're erased or commented-out in the Unicorn
configuration file. Delays/slowdowns in signal handling since
0.990 are fixed, too.
== unicorn 1.0.2
This is the latest maintenance release of the 1.0.x series.
All users are encouraged to upgrade to 1.1.x stable series
and report bugs there.
Shortlog of changes since 1.0.1:
Eric Wong (8):
SIGTTIN works after SIGWINCH
fix delays in signal handling
Rakefile: don't post freshmeat on empty changelogs
Rakefile: capture prerelease tags
configurator: use "__send__" instead of "send"
configurator: reloading with unset values restores default
gemspec: depend on Isolate 3.0.0 for dev
doc: stop using deprecated rdoc CLI options
==
Have fun!
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2010-10-28 0:19 6% [ANN] Unicorn 2.0.0, 1.1.5, 1.0.2 released! Eric Wong
2011-03-22 19:19 7% unicorn 1.1.x never-ending listen loop IOError exceptions David Nghiem
2011-03-22 21:39 0% ` Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).