Re: unicorn 1.1.x never-ending listen loop IOError exceptions

[Eric Wong <normalperson@yhbt.net>]

David Nghiem <david@zendesk.com> wrote:
> On Tue, Mar 22, 2011 at 2:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > No Errno::EBADF errors at all?
> 
> I didn't check before or after the block of Errno::IOError errors,
> there were enough IOError logs to fill up 40gb of disk.  I'll capture
> the entire deleted log next time.

Yikes.  The first few lines of error messages should be enough.

> >> ruby    15209 zendesk    1w   REG              252,0           0
> >> 3015360 /data/zendesk/log/unicorn.stdout.log
> >> ruby    15209 zendesk    2w   REG              252,0       24585
> >> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
> >> ruby    15209 zendesk    3w   REG              252,0 61617590073
> >> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
> >
> > Permissions issue during log rotation?  Are you doing "killall -USR1
> > unicorn" by any chance or sending a bunch of USR1 signals faster than
> > unicorn can process them?
> >
> > Try just sending one USR1 to the master process and letting the master
> > signal the workers.
> 
> Are the IOError exceptions and failure to rotate logs related?  My
> wild guess was this specific worker's unix socket got corrupted
> somehow.

There's a good possibility.  The USR1 signal closes a pipe we normally
select() on.  The closed pipe triggers a wakeup from select(), which
triggers the log reopening (and reopening of the pipe).

Do you see:  "worker=$NUM done reopening logs" messages in your
logs?

Another explanation is somehow you have a library that does something
along the lines of:

 ObjectSpace.each_object(IO) { |io| io.close unless io.closed? }

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying