Weird random Zbatery exception

Weird random Zbatery exception

[Damian Janowski]

Hello everyone,

I'm giving Zbatery a try, currently testing with ThreadPool.

By the way, this instance is sitting behind Varnish.

I can't figure out precise steps to trigger this exception, but it
happens after a few GET/POST requests:

/usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_server.rb:361:in
`awaken_master': undefined method `kgio_trywrite' for nil:NilClass
(NoMethodError)
        from /usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_server.rb:130:in
`block in start'
        from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:32:in
`call'
        from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:32:in
`join'
        from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:32:in
`block in worker_loop'
        from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:30:in
`each'
        from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:30:in
`worker_loop'
        from /usr/local/lib/ruby/gems/1.9.1/gems/zbatery-4.1.1/lib/zbatery.rb:100:in
`join'
        from /usr/local/lib/ruby/gems/1.9.1/gems/zbatery-4.1.1/bin/zbatery:121:in
`<top (required)>'
        from /usr/local/bin/zbatery:19:in `load'
        from /usr/local/bin/zbatery:19:in `<main>'

(Needless to say, I googled already without much luck).

Any ideas?

D.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Weird random Zbatery exception

[Eric Wong]

Damian Janowski <jano-eWlILCNyKRBBDgjK7y7TUQ@public.gmane.org> wrote:
> I'm giving Zbatery a try, currently testing with ThreadPool.

Thanks for giving it a try! :)

> I can't figure out precise steps to trigger this exception, but it
> happens after a few GET/POST requests:

> /usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_server.rb:361:in
> `awaken_master': undefined method `kgio_trywrite' for nil:NilClass
> (NoMethodError)

Does your application fork/exec other processes?  (`` or "system()",
popen all included).

> (Needless to say, I googled already without much luck).

I think you found a new bug, does the following patch fix it?

diff --git a/lib/zbatery.rb b/lib/zbatery.rb
index 735b1e6..17d7c06 100644
--- a/lib/zbatery.rb
+++ b/lib/zbatery.rb
@@ -82,6 +82,7 @@ module Rainbows
       trap(:USR1) { Thread.new { reopen_logs } }
       trap(:USR2) { Thread.new { reexec } }
       trap(:HUP) { Thread.new { reexec; stop } }
+      trap(:CHLD, "IGNORE")
 
       # technically feasible in some cases, just not sanely supportable:
       %w(TTIN TTOU WINCH).each do |sig|
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Weird random Zbatery exception

[Eric Wong]

Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> diff --git a/lib/zbatery.rb b/lib/zbatery.rb
> index 735b1e6..17d7c06 100644
> --- a/lib/zbatery.rb
> +++ b/lib/zbatery.rb
> @@ -82,6 +82,7 @@ module Rainbows
>        trap(:USR1) { Thread.new { reopen_logs } }
>        trap(:USR2) { Thread.new { reexec } }
>        trap(:HUP) { Thread.new { reexec; stop } }
> +      trap(:CHLD, "IGNORE")

On second thought, "IGNORE" will probably break existing code.
trap(:CHLD, "DEFAULT") should be better.

Btw, this issue shouldn't affect Rainbows!, only Zbatery.
Can you confirm?
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Weird random Zbatery exception

[Eric Wong]

Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > +      trap(:CHLD, "IGNORE")
> 
> On second thought, "IGNORE" will probably break existing code.
> trap(:CHLD, "DEFAULT") should be better.

I just pushed out Zbatery 4.1.1.2.g3693 (prerelease) to RubyGems.org
with trap(:CHLD,"DEFAULT") + test cases.

You can install it with: gem install --pre zbatery

Let me know if it fixes your problem since I've been meaning to release
for 90d1d769b9af83d6d1 anyways (unlink oldbin pid file during graceful
shutdown).

(I also found https://redmine.ruby-lang.org/issues/5700 while working
 on this :o)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Weird random Zbatery exception

[Damian Janowski]

On Fri, Dec 2, 2011 at 10:46 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> I just pushed out Zbatery 4.1.1.2.g3693 (prerelease) to RubyGems.org
> with trap(:CHLD,"DEFAULT") + test cases.
>
> You can install it with: gem install --pre zbatery
>
> Let me know if it fixes your problem since I've been meaning to release
> for 90d1d769b9af83d6d1 anyways (unlink oldbin pid file during graceful
> shutdown).

It does seem to fix the problem.

Thank you very much!
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Weird random Zbatery exception

[Damian Janowski]

On Fri, Dec 2, 2011 at 8:40 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Btw, this issue shouldn't affect Rainbows!, only Zbatery.
> Can you confirm?

I can confirm Rainbows has been stable for a few days (whereas Zbatery
crashed after a few requests).

I'll test the pre release now and let you know, thank you again.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Weird random Zbatery exception

[Damian Janowski]

On Fri, Dec 2, 2011 at 7:47 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Thanks for giving it a try! :)

Thank you, Eric, for your prompt reply!

> Does your application fork/exec other processes?  (`` or "system()",
> popen all included).

Not in my application code, and I'm pretty sure not in any of the few
libraries I'm using.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying