* unicorn 1.1.x never-ending listen loop IOError exceptions
@ 2011-03-22 19:19 David Nghiem
2011-03-22 21:39 ` Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: David Nghiem @ 2011-03-22 19:19 UTC (permalink / raw)
To: mongrel-unicorn
We run unicorn 1.1.5 behind nginx under Ruby Enterprise Edition
2011.02. We're using UNIX sockets instead of TCP sockets.
Once in a while, one worker will throw exceptions in a loop, taking up
lots of CPU and log space. Killing the worker removes the problem.
Other workers are not affected.
Here's the exception:
E, [2011-01-10T10:13:13.518795 #15209] ERROR -- : Unhandled listen
loop exception #<IOError: closed stream>.
E, [2011-01-10T10:13:13.518881 #15209] ERROR -- :
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:732:in
`select'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:732:in
`worker_loop'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/newrelic_rpm-2.13.1/lib/new_relic/control/../agent/instrumentation/unicorn_instrumentation.rb:7:in
`call'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/newrelic_rpm-2.13.1/lib/new_relic/control/../agent/instrumentation/unicorn_instrumentation.rb:7:in
`worker_loop'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:603:in
`spawn_missing_workers'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:600:in
`fork'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:600:in
`spawn_missing_workers'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:596:in
`each'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:596:in
`spawn_missing_workers'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:610:in
`maintain_worker_count'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:407:in
`join'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:395:in
`loop'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:395:in
`join'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:29:in
`run'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/bin/unicorn_rails:210
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/bin/unicorn_rails:19:in
`load'
/data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/bin/unicorn_rails:19
I don't get any other exceptions. I've found other people have this
"IOError: closed stream" error coupled with "Errno::EMFILE" but we
have plenty of file-descriptors left.
Here's an lsof dump of the worker:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 15209 zendesk cwd DIR 252,0 4096
2940970 /data/zendesk/releases/alpha
ruby 15209 zendesk rtd DIR 8,1 4096 2 /
ruby 15209 zendesk txt REG 8,1 2831149
246513 /usr/local/rvm/rubies/ree-1.8.7-2010.02/bin/ruby
ruby 15209 zendesk mem REG 8,1 10192
146145 /usr/lib/gconv/ISO8859-1.so
ruby 15209 zendesk mem REG 8,1 22848
263831 /lib/libnss_dns-2.9.so
ruby 15209 zendesk mem REG 8,1 9760
260915 /lib/libnss_mdns4_minimal.so.2
ruby 15209 zendesk mem REG 8,1 43472
263939 /lib/libnss_nis-2.9.so
ruby 15209 zendesk mem REG 8,1 35632
263929 /lib/libnss_compat-2.9.so
ruby 15209 zendesk mem REG 8,1 51616
263934 /lib/libnss_files-2.9.so
ruby 15209 zendesk mem REG 252,0 35772
3377231 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/em-http-request-0.2.7/lib/em_buffer.so
ruby 15209 zendesk mem REG 252,0 45360
3377237 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/em-http-request-0.2.7/lib/http11_client.so
ruby 15209 zendesk mem REG 252,0 1175256
3377180 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/eventmachine-0.12.10/lib/rubyeventmachine.so
ruby 15209 zendesk mem REG 8,1 35184
155430 /usr/lib/sasl2/libntlm.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18672
155418 /usr/lib/sasl2/libanonymous.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18672
155421 /usr/lib/sasl2/libcrammd5.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18640
155433 /usr/lib/sasl2/libplain.so.2.0.22
ruby 15209 zendesk mem REG 8,1 18640
155427 /usr/lib/sasl2/liblogin.so.2.0.22
ruby 15209 zendesk mem REG 8,1 1297736
140958 /usr/lib/libdb-4.6.so
ruby 15209 zendesk mem REG 8,1 22616
155410 /usr/lib/sasl2/libsasldb.so.2.0.22
ruby 15209 zendesk mem REG 8,1 52000
155424 /usr/lib/sasl2/libdigestmd5.so.2.0.22
ruby 15209 zendesk mem REG 8,1 88824
263949 /lib/libresolv-2.9.so
ruby 15209 zendesk mem REG 8,1 105760
139781 /usr/lib/libsasl2.so.2.0.22
ruby 15209 zendesk mem REG 252,0 4110920
3384825 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/memcached-0.20.1/lib/rlibmemcached.so
ruby 15209 zendesk mem REG 8,1 97168
263938 /lib/libnsl-2.9.so
ruby 15209 zendesk mem REG 8,1 2086600
145819 /usr/lib/libmysqlclient.so.15.0.0
ruby 15209 zendesk mem REG 252,0 169011
3385005 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/mysql-2.8.1/lib/mysql_api.so
ruby 15209 zendesk mem REG 252,0 150950
3401474 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/yajl-ruby-0.7.6/lib/yajl_ext.so
ruby 15209 zendesk mem REG 252,0 29311
3367472 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/SystemTimer-1.2/lib/system_timer_native.so
ruby 15209 zendesk mem REG 8,1 13144
260768 /lib/libgpg-error.so.0.3.0
ruby 15209 zendesk mem REG 8,1 422664
260765 /lib/libgcrypt.so.11.4.4
ruby 15209 zendesk mem REG 8,1 1425368
141140 /usr/lib/libxml2.so.2.6.32
ruby 15209 zendesk mem REG 8,1 233152
439848 /usr/lib/libxslt.so.1.1.24
ruby 15209 zendesk mem REG 8,1 80792
439847 /usr/lib/libexslt.so.0.8.13
ruby 15209 zendesk mem REG 252,0 417564
3385563 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/nokogiri-1.4.1/lib/nokogiri/nokogiri.so
ruby 15209 zendesk mem REG 8,1 14220
252540 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/digest/md5.so
ruby 15209 zendesk mem REG 8,1 42715
252537 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/racc/cparse.so
ruby 15209 zendesk mem REG 252,0 121676
3465790 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/json-1.4.6/ext/json/ext/json/ext/generator.so
ruby 15209 zendesk mem REG 252,0 52561
3465795 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/json-1.4.6/ext/json/ext/json/ext/parser.so
ruby 15209 zendesk mem REG 252,0 20227
3401531 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/fast_xs-0.7.3/lib/fast_xs.so
ruby 15209 zendesk mem REG 8,1 46499
255504 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/site_ruby/1.8/x86_64-linux/iconv.so
ruby 15209 zendesk mem REG 8,1 139431
247137 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/bigdecimal.so
ruby 15209 zendesk mem REG 8,1 433200
247152 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/nkf.so
ruby 15209 zendesk mem REG 252,0 106488
3531787 /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn_http.so
ruby 15209 zendesk mem REG 8,1 53645
247158 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/strscan.so
ruby 15209 zendesk mem REG 8,1 128208
247138 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/socket.so
ruby 15209 zendesk mem REG 8,1 12144
247141 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/fcntl.so
ruby 15209 zendesk mem REG 8,1 329504
260852 /lib/libssl.so.0.9.8
ruby 15209 zendesk mem REG 8,1 1002954
247156 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/openssl.so
ruby 15209 zendesk mem REG 8,1 110815
247146 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/zlib.so
ruby 15209 zendesk mem REG 8,1 35684
247140 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/digest.so
ruby 15209 zendesk mem REG 8,1 96768
260746 /lib/libz.so.1.2.3.3
ruby 15209 zendesk mem REG 8,1 1594824
260668 /lib/libcrypto.so.0.9.8
ruby 15209 zendesk mem REG 8,1 15204
252539 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/digest/sha1.so
ruby 15209 zendesk mem REG 8,1 12723
247159 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/rational.so
ruby 15209 zendesk mem REG 8,1 351320
247142 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/syck.so
ruby 15209 zendesk mem REG 8,1 57484
247154 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/stringio.so
ruby 15209 zendesk mem REG 8,1 29818
247150 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/etc.so
ruby 15209 zendesk mem REG 8,1 51878
247139 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/x86_64-linux/thread.so
ruby 15209 zendesk mem REG 8,1 130151
263933 /lib/libpthread-2.9.so
ruby 15209 zendesk mem REG 8,1 96560
260672 /lib/libgcc_s.so.1
ruby 15209 zendesk mem REG 8,1 1023448
139822 /usr/lib/libstdc++.so.6.0.10
ruby 15209 zendesk mem REG 8,1 1502512
263944 /lib/libc-2.9.so
ruby 15209 zendesk mem REG 8,1 542928
263942 /lib/libm-2.9.so
ruby 15209 zendesk mem REG 8,1 39112
263932 /lib/libcrypt-2.9.so
ruby 15209 zendesk mem REG 8,1 14608
263945 /lib/libdl-2.9.so
ruby 15209 zendesk mem REG 8,1 31656
263930 /lib/librt-2.9.so
ruby 15209 zendesk mem REG 8,1 728968
246427 /usr/local/rvm/rubies/ree-1.8.7-2010.02/lib/libtcmalloc_minimal.so.0.0.0
ruby 15209 zendesk mem REG 8,1 135680
263936 /lib/ld-2.9.so
ruby 15209 zendesk mem REG 8,1 26048
139155 /usr/lib/gconv/gconv-modules.cache
ruby 15209 zendesk 0r CHR 1,3
3125 /dev/null
ruby 15209 zendesk 1w REG 252,0 0
3015360 /data/zendesk/log/unicorn.stdout.log
ruby 15209 zendesk 2w REG 252,0 24585
3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
ruby 15209 zendesk 3w REG 252,0 61617590073
3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
ruby 15209 zendesk 5u unix 0xffff88043bdf9080
49439 /var/run/zendesk/unicorn.sock
ruby 15209 zendesk 6w FIFO 0,6 5651378 pipe
ruby 15209 zendesk 7w REG 252,0 0
3015676 /data/zendesk/log/newrelic_agent.log
ruby 15209 zendesk 8w REG 252,0 68330048
3015673 /data/zendesk/log/production.log
ruby 15209 zendesk 9w REG 252,0 0
3015677 /data/zendesk/log/resque.log
ruby 15209 zendesk 10u IPv4 41707462
TCP 192.168.100.148:49364->192.168.100.183:mysql (ESTABLISHED)
ruby 15209 zendesk 11w REG 252,0 0
3015672 /data/zendesk/log/mail_processing_production.log
ruby 15209 zendesk 12u IPv4 55517
TCP 192.168.100.148:60198->192.168.100.182:mysql (ESTABLISHED)
ruby 15209 zendesk 13u IPv4 55521
TCP 192.168.100.148:42351->192.168.100.150:11211 (ESTABLISHED)
ruby 15209 zendesk 14u IPv4 55523
TCP 192.168.100.148:52409->192.168.100.148:11211 (ESTABLISHED)
ruby 15209 zendesk 15u IPv4 56105
TCP 192.168.100.148:58562->192.168.100.140:11211 (ESTABLISHED)
ruby 15209 zendesk 16u IPv4 56108
TCP 192.168.100.148:50500->192.168.100.144:11211 (ESTABLISHED)
ruby 15209 zendesk 17u IPv4 56110
TCP 192.168.100.148:46723->192.168.100.142:11211 (ESTABLISHED)
ruby 15209 zendesk 18u IPv4 56137
TCP 192.168.100.148:49568->192.168.100.146:11211 (ESTABLISHED)
ruby 15209 zendesk 19u IPv4 5845535
TCP 192.168.100.148:43836->192.168.100.183:mysql (ESTABLISHED)
ruby 15209 zendesk 20u IPv4 63603
TCP 192.168.100.148:41990->192.168.100.149:6379 (ESTABLISHED)
ruby 15209 zendesk 21u REG 8,1 0
90084 /tmp/0.961990208371916 (deleted)
ruby 15209 zendesk 22u IPv4 10795792
TCP 192.168.100.148:34587->192.168.100.183:mysql (ESTABLISHED)
ruby 15209 zendesk 23u IPv4 33013892
TCP 192.168.100.148:54382->192.168.100.183:mysql (ESTABLISHED)
I'm tempted to exit the worker when I get this exception N number of
times as a workaround; it tends to only happen when I'm sleeping =)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unicorn 1.1.x never-ending listen loop IOError exceptions
2011-03-22 19:19 unicorn 1.1.x never-ending listen loop IOError exceptions David Nghiem
@ 2011-03-22 21:39 ` Eric Wong
2011-03-23 0:04 ` David Nghiem
0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2011-03-22 21:39 UTC (permalink / raw)
To: unicorn list
David Nghiem <david@zendesk.com> wrote:
> We run unicorn 1.1.5 behind nginx under Ruby Enterprise Edition
> 2011.02. We're using UNIX sockets instead of TCP sockets.
>
> Once in a while, one worker will throw exceptions in a loop, taking up
> lots of CPU and log space. Killing the worker removes the problem.
> Other workers are not affected.
>
> Here's the exception:
>
> E, [2011-01-10T10:13:13.518795 #15209] ERROR -- : Unhandled listen
> loop exception #<IOError: closed stream>.
> E, [2011-01-10T10:13:13.518881 #15209] ERROR -- :
> /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:732:in
> `select'
>
> I don't get any other exceptions. I've found other people have this
> "IOError: closed stream" error coupled with "Errno::EMFILE" but we
> have plenty of file-descriptors left.
No Errno::EBADF errors at all?
> Here's an lsof dump of the worker:
>
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
<snip> It looks like log reopening failed, below:
> ruby 15209 zendesk 1w REG 252,0 0
> 3015360 /data/zendesk/log/unicorn.stdout.log
> ruby 15209 zendesk 2w REG 252,0 24585
> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
> ruby 15209 zendesk 3w REG 252,0 61617590073
> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
Permissions issue during log rotation? Are you doing "killall -USR1
unicorn" by any chance or sending a bunch of USR1 signals faster than
unicorn can process them?
Try just sending one USR1 to the master process and letting the master
signal the workers.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unicorn 1.1.x never-ending listen loop IOError exceptions
2011-03-22 21:39 ` Eric Wong
@ 2011-03-23 0:04 ` David Nghiem
2011-03-23 1:09 ` Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: David Nghiem @ 2011-03-23 0:04 UTC (permalink / raw)
To: unicorn list
On Tue, Mar 22, 2011 at 2:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> I don't get any other exceptions. I've found other people have this
>> "IOError: closed stream" error coupled with "Errno::EMFILE" but we
>> have plenty of file-descriptors left.
>
> No Errno::EBADF errors at all?
I didn't check before or after the block of Errno::IOError errors,
there were enough IOError logs to fill up 40gb of disk. I'll capture
the entire deleted log next time.
> <snip> It looks like log reopening failed, below:
>
>> ruby 15209 zendesk 1w REG 252,0 0
>> 3015360 /data/zendesk/log/unicorn.stdout.log
>> ruby 15209 zendesk 2w REG 252,0 24585
>> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
>> ruby 15209 zendesk 3w REG 252,0 61617590073
>> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
>
> Permissions issue during log rotation? Are you doing "killall -USR1
> unicorn" by any chance or sending a bunch of USR1 signals faster than
> unicorn can process them?
>
> Try just sending one USR1 to the master process and letting the master
> signal the workers.
We rotate logs once a night using logrotate. There's a post-rotate
hook that sends USR1 once to the master. There are three log files
but logrotate is run with the sharedscripts option so only one hook
runs. The log dirs and unicorn processes are run as the same non-root
user.
I'll run an lsof on the healthy workers next time to see if they have
the same deleted log file descriptors.
Are the IOError exceptions and failure to rotate logs related? My
wild guess was this specific worker's unix socket got corrupted
somehow.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unicorn 1.1.x never-ending listen loop IOError exceptions
2011-03-23 0:04 ` David Nghiem
@ 2011-03-23 1:09 ` Eric Wong
[not found] ` <AANLkTim3xZMKxcmQzPjcB7vrJKiHoc2f8WoU8RatrDum@mail.gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2011-03-23 1:09 UTC (permalink / raw)
To: unicorn list
David Nghiem <david@zendesk.com> wrote:
> On Tue, Mar 22, 2011 at 2:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > No Errno::EBADF errors at all?
>
> I didn't check before or after the block of Errno::IOError errors,
> there were enough IOError logs to fill up 40gb of disk. I'll capture
> the entire deleted log next time.
Yikes. The first few lines of error messages should be enough.
> >> ruby 15209 zendesk 1w REG 252,0 0
> >> 3015360 /data/zendesk/log/unicorn.stdout.log
> >> ruby 15209 zendesk 2w REG 252,0 24585
> >> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
> >> ruby 15209 zendesk 3w REG 252,0 61617590073
> >> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
> >
> > Permissions issue during log rotation? Are you doing "killall -USR1
> > unicorn" by any chance or sending a bunch of USR1 signals faster than
> > unicorn can process them?
> >
> > Try just sending one USR1 to the master process and letting the master
> > signal the workers.
>
> Are the IOError exceptions and failure to rotate logs related? My
> wild guess was this specific worker's unix socket got corrupted
> somehow.
There's a good possibility. The USR1 signal closes a pipe we normally
select() on. The closed pipe triggers a wakeup from select(), which
triggers the log reopening (and reopening of the pipe).
Do you see: "worker=$NUM done reopening logs" messages in your
logs?
Another explanation is somehow you have a library that does something
along the lines of:
ObjectSpace.each_object(IO) { |io| io.close unless io.closed? }
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unicorn 1.1.x never-ending listen loop IOError exceptions
[not found] ` <AANLkTim3xZMKxcmQzPjcB7vrJKiHoc2f8WoU8RatrDum@mail.gmail.com>
@ 2011-03-29 16:24 ` Eric Wong
2011-03-29 16:54 ` [PATCH] add examples/logrotate.conf Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2011-03-29 16:24 UTC (permalink / raw)
To: David Nghiem; +Cc: unicorn list
David Nghiem <david@zendesk.com> wrote:
> I'm wondering if the problem is three logs for unicorn (unicorn.log, and the
> stdout, stderr logs) are rotated by logrotate, but the first log triggers
> the USR1 and in the middle of rotating the remaining logs, the unicorn log
> reopening gets confused.
>
> I'll make it so the USR1 signal is sent after all three logs have been
> rotated. Let this run for a couple months and report back.
That could be it. logrotate (without the scary/expensive
"copytruncate" option) works very quickly for individual files. Do you
have the "sharedscripts" option in logrotate enabled? You should enable
"sharedscripts"
I should post a logrotate example config so other users don't run into
the same problem...
> Thanks, Eric, for your help. I've been running unicorn since the pre-1.0
> days and it's my favorite ruby software.
Good to know!
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] add examples/logrotate.conf
2011-03-29 16:24 ` Eric Wong
@ 2011-03-29 16:54 ` Eric Wong
0 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2011-03-29 16:54 UTC (permalink / raw)
To: David Nghiem; +Cc: unicorn list
Eric Wong <normalperson@yhbt.net> wrote:
> That could be it. logrotate (without the scary/expensive
> "copytruncate" option) works very quickly for individual files. Do you
> have the "sharedscripts" option in logrotate enabled? You should enable
> "sharedscripts"
I actually went with lastaction in my example (I keep forgetting it over
the years and going to postrotate + sharedscripts for some reason..)
> I should post a logrotate example config so other users don't run into
> the same problem...
http://unicorn.bogomips.org/examples/logrotate.conf and pushed
this out to unicorn.git
>From c1ebb313735a280582d87c1ba44619aa47e00b06 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Tue, 29 Mar 2011 09:47:26 -0700
Subject: [PATCH] add examples/logrotate.conf
logrotate is the de facto tool for logrotation, so an
example config for highlighting important parts are in order.
Since our USR1 signal handling is part of the crusade against
the slow and lossy "copytruncate" option, be sure to
emphasize that :)
---
examples/logrotate.conf | 29 +++++++++++++++++++++++++++++
1 files changed, 29 insertions(+), 0 deletions(-)
create mode 100644 examples/logrotate.conf
diff --git a/examples/logrotate.conf b/examples/logrotate.conf
new file mode 100644
index 0000000..03fefc6
--- /dev/null
+++ b/examples/logrotate.conf
@@ -0,0 +1,29 @@
+# example logrotate config file, I usually keep this in
+# /etc/logrotate.d/unicorn_app on my Debian systems
+#
+# See the logrotate(8) manpage for more information:
+# http://linux.die.net/man/8/logrotate
+
+# Modify the following glob to match the logfiles your app writes to:
+/var/log/unicorn_app/*.log {
+ # this first block is mostly just personal preference, though
+ # I wish logrotate offered an "hourly" option...
+ daily
+ missingok
+ rotate 180
+ compress # must use with delaycompress below
+ dateext
+
+ # this is important if using "compress" since we need to call
+ # the "lastaction" script below before compressing:
+ delaycompress
+
+ # note the lack of the evil "copytruncate" option in this
+ # config. Unicorn supports the USR1 signal and we send it
+ # as our "lastaction" action:
+ lastaction
+ # assuming your pid file is in /var/run/unicorn_app/pid
+ pid=/var/run/unicorn_app/pid
+ test -s $pid && kill -USR1 "$(cat $pid)"
+ endscript
+}
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-03-29 17:00 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-03-22 19:19 unicorn 1.1.x never-ending listen loop IOError exceptions David Nghiem
2011-03-22 21:39 ` Eric Wong
2011-03-23 0:04 ` David Nghiem
2011-03-23 1:09 ` Eric Wong
[not found] ` <AANLkTim3xZMKxcmQzPjcB7vrJKiHoc2f8WoU8RatrDum@mail.gmail.com>
2011-03-29 16:24 ` Eric Wong
2011-03-29 16:54 ` [PATCH] add examples/logrotate.conf Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).