rails 2 and slow external services

rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

Hi,

Some of my page loads (currently serviced by Unicorn) spend a great
deal of time waiting for external services (OpenID, OAuth, etc over
Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
Which concurrency model in Rainbows! is best suited for this sort of
page? I'm not totally clear on which parts if any of Rails are
thread-safe.

Thanks in advance!

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> Hi,
> 
> Some of my page loads (currently serviced by Unicorn) spend a great
> deal of time waiting for external services (OpenID, OAuth, etc over
> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
> Which concurrency model in Rainbows! is best suited for this sort of
> page? I'm not totally clear on which parts if any of Rails are
> thread-safe.

Hi, if you want compatibility over the greatest number of existing Gems,
then ThreadSpawn and ThreadPool are the safest options.

I'm fairly certain Rails 2.3 (or even 2.2) are _advertised_ as
thread-safe but I've never tested it heavily.  Hopefully other folks can
give some feedback here.

If you're running Ruby 1.9, then RevThreadSpawn and RevThreadPool should
be good options if you do heavy keepalive, too, but they're a bit iffy
under 1.8 due to incompatibilities with libev and the MRI 1.8 threading
model.

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Mon, Dec 13, 2010 at 2:39 AM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> Hi,
>>
>> Some of my page loads (currently serviced by Unicorn) spend a great
>> deal of time waiting for external services (OpenID, OAuth, etc over
>> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
>> Which concurrency model in Rainbows! is best suited for this sort of
>> page? I'm not totally clear on which parts if any of Rails are
>> thread-safe.
>
> Hi, if you want compatibility over the greatest number of existing Gems,
> then ThreadSpawn and ThreadPool are the safest options.
>
> I'm fairly certain Rails 2.3 (or even 2.2) are _advertised_ as
> thread-safe but I've never tested it heavily.  Hopefully other folks can
> give some feedback here.
>
> If you're running Ruby 1.9, then RevThreadSpawn and RevThreadPool should
> be good options if you do heavy keepalive, too, but they're a bit iffy
> under 1.8 due to incompatibilities with libev and the MRI 1.8 threading
> model.

This is Ruby 1.8.7 (REE). Is there any interesting difference between
ThreadPool and ThreadSpawn in this environment?

I also make use of a (heavily modified, which is another topic) OobGC.
Does anyone know if garbage collection in ruby 1.8.7 is reasonably
threadable? I expect not, but one can hope.

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> > ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> >> Some of my page loads (currently serviced by Unicorn) spend a great
> >> deal of time waiting for external services (OpenID, OAuth, etc over
> >> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
> 
> This is Ruby 1.8.7 (REE). Is there any interesting difference between
> ThreadPool and ThreadSpawn in this environment?

ThreadPool is generally more predictable, but ThreadSpawn has lower
memory usage if your traffic spikes are intermittent or low.

ThreadSpawn is much like the original Mongrel and ThreadPool was an
experiment with Ruby 1.9 in mind.  1.9 has more expensive (but slightly
more concurrent) threading.  If your bottlenecks are external HTTP
requests on 1.8, but first instinct would be to use ThreadSpawn.

Ruby 1.9 + ThreadPool would probably be well-suited for large file
serving to LAN clients with many slowish disks as it can use sendfile
via IO.copy_stream), but if you can afford the constant memory overhead,
it could be good in 1.8, too.

> I also make use of a (heavily modified, which is another topic) OobGC.
> Does anyone know if garbage collection in ruby 1.8.7 is reasonably
> threadable? I expect not, but one can hope.

It is not, the entire interpreter stops running every single thread for
GC.  I don't think using OobGC with any of the Rainbows! concurrency
models will work, only :Base and Unicorn.

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Mon, Dec 13, 2010 at 8:57 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> > ghazel@gmail.com wrote:
>> >> Some of my page loads (currently serviced by Unicorn) spend a great
>> >> deal of time waiting for external services (OpenID, OAuth, etc over
>> >> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
>>
>> This is Ruby 1.8.7 (REE). Is there any interesting difference between
>> ThreadPool and ThreadSpawn in this environment?
>
> ThreadPool is generally more predictable, but ThreadSpawn has lower
> memory usage if your traffic spikes are intermittent or low.
>
> ThreadSpawn is much like the original Mongrel and ThreadPool was an
> experiment with Ruby 1.9 in mind.  1.9 has more expensive (but slightly
> more concurrent) threading.  If your bottlenecks are external HTTP
> requests on 1.8, but first instinct would be to use ThreadSpawn.
>
> Ruby 1.9 + ThreadPool would probably be well-suited for large file
> serving to LAN clients with many slowish disks as it can use sendfile
> via IO.copy_stream), but if you can afford the constant memory overhead,
> it could be good in 1.8, too.

I don't mind constant memory overhead -  actually I prefer it to
spikey memory usage with an unknown peak. Otherwise they should be the
same?

>> I also make use of a (heavily modified, which is another topic) OobGC.
>> Does anyone know if garbage collection in ruby 1.8.7 is reasonably
>> threadable? I expect not, but one can hope.
>
> It is not, the entire interpreter stops running every single thread for
> GC.  I don't think using OobGC with any of the Rainbows! concurrency
> models will work, only :Base and Unicorn.

Well, not too surprising, but thanks for specifying.

I'm running a bit of my traffic through some Rainbows! right now, but I got:

2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
referrer: "https://foofoo.com"
2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"

From nginx in the error log. My proxy_read_timeout is 300, and my
listen backlog is 2048 (for now...). Basically my Rainbows! config is
identical to my Unicorn config, where I have not seen that happen,
except I added "Rainbows! { use :ThreadPool; worker_connections 100
}".

Any ideas?

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> On Mon, Dec 13, 2010 at 8:57 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> >> > ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> >> >> Some of my page loads (currently serviced by Unicorn) spend a great
> >> >> deal of time waiting for external services (OpenID, OAuth, etc over
> >> >> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
> >>
> >> This is Ruby 1.8.7 (REE). Is there any interesting difference between
> >> ThreadPool and ThreadSpawn in this environment?
> >
> > ThreadPool is generally more predictable, but ThreadSpawn has lower
> > memory usage if your traffic spikes are intermittent or low.
> >
> > ThreadSpawn is much like the original Mongrel and ThreadPool was an
> > experiment with Ruby 1.9 in mind.  1.9 has more expensive (but slightly
> > more concurrent) threading.  If your bottlenecks are external HTTP
> > requests on 1.8, but first instinct would be to use ThreadSpawn.
> >
> > Ruby 1.9 + ThreadPool would probably be well-suited for large file
> > serving to LAN clients with many slowish disks as it can use sendfile
> > via IO.copy_stream), but if you can afford the constant memory overhead,
> > it could be good in 1.8, too.
> 
> I don't mind constant memory overhead -  actually I prefer it to
> spikey memory usage with an unknown peak. Otherwise they should be the
> same?

Having idle threads with ThreadPool would affect GC performance,
actually.

> I'm running a bit of my traffic through some Rainbows! right now, but I got:
> 
> 2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
> Connection timed out) while reading response header from upstream,
> client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
> referrer: "https://foofoo.com"
> 2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
> Connection timed out) while reading response header from upstream,
> client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"
> 
> From nginx in the error log. My proxy_read_timeout is 300, and my
> listen backlog is 2048 (for now...). Basically my Rainbows! config is
> identical to my Unicorn config, where I have not seen that happen,
> except I added "Rainbows! { use :ThreadPool; worker_connections 100
> }".

Was your app hitting the Unicorn kill -9 timeout frequently before?  In
Rainbows!, the kill -9 timeout only kicks in when the entire
interpreter/VM is stuck due to a broken C extension or bug in Ruby.

If it's some component of your app taking a long time (and relying on
the Unicorn kill -9 timeout), you can try the Rainbows::ThreadTimeout
middleware: http://rainbows.rubyforge.org/Rainbows/ThreadTimeout.html

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Mon, Dec 13, 2010 at 10:35 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> On Mon, Dec 13, 2010 at 8:57 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > ghazel@gmail.com wrote:
>> I'm running a bit of my traffic through some Rainbows! right now, but I got:
>>
>> 2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
>> referrer: "https://foofoo.com"
>> 2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"
>>
>> From nginx in the error log. My proxy_read_timeout is 300, and my
>> listen backlog is 2048 (for now...). Basically my Rainbows! config is
>> identical to my Unicorn config, where I have not seen that happen,
>> except I added "Rainbows! { use :ThreadPool; worker_connections 100
>> }".
>
> Was your app hitting the Unicorn kill -9 timeout frequently before?  In
> Rainbows!, the kill -9 timeout only kicks in when the entire
> interpreter/VM is stuck due to a broken C extension or bug in Ruby.
>
> If it's some component of your app taking a long time (and relying on
> the Unicorn kill -9 timeout), you can try the Rainbows::ThreadTimeout
> middleware: http://rainbows.rubyforge.org/Rainbows/ThreadTimeout.html

I'm not sure how I would know that - I'm not actually sure which
timeout you mean. "timeout" in the config/unicorn.rb and
config/rainbows.rb is 300, same as proxy_read_timeout. Is that it? Are
you saying when that is hit that Rainbows! and Unicorn act
differently?

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> I'm not sure how I would know that - I'm not actually sure which
> timeout you mean. "timeout" in the config/unicorn.rb and
> config/rainbows.rb is 300, same as proxy_read_timeout. Is that it? Are
> you saying when that is hit that Rainbows! and Unicorn act
> differently?

Yes, the purpose of the "timeout" in the Unicorn config has always been
to kill unrecoverably stuck processes, Rainbows! just enforces that
more strictly.

The simple nature of Unicorn allows lazy people can use it to avoid
adding timeouts to their code without causing harm to other clients.

Since Rainbows! is designed to serve multiple clients in the same
process, killing a process would break all the clients on the process,
not just the one client that triggered the timeout.  So using Rainbows!
requires more discipline from the application authors than Unicorn.

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Mon, Dec 13, 2010 at 11:49 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> I'm not sure how I would know that - I'm not actually sure which
>> timeout you mean. "timeout" in the config/unicorn.rb and
>> config/rainbows.rb is 300, same as proxy_read_timeout. Is that it? Are
>> you saying when that is hit that Rainbows! and Unicorn act
>> differently?
>
> Yes, the purpose of the "timeout" in the Unicorn config has always been
> to kill unrecoverably stuck processes, Rainbows! just enforces that
> more strictly.
>
> The simple nature of Unicorn allows lazy people can use it to avoid
> adding timeouts to their code without causing harm to other clients.
>
> Since Rainbows! is designed to serve multiple clients in the same
> process, killing a process would break all the clients on the process,
> not just the one client that triggered the timeout.  So using Rainbows!
> requires more discipline from the application authors than Unicorn.

Hm. Well I was unaware that there was any timeout issue with my
application. When a Unicorn process timed out and died, how did the
request not timeout with nginx? Was it re-submitted to another worker?

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> Hm. Well I was unaware that there was any timeout issue with my
> application. When a Unicorn process timed out and died, how did the
> request not timeout with nginx? Was it re-submitted to another worker?

A timeout issue is one *possible* cause of the errors you were seeing
from nginx.  Of course you know the application better than I do, so,
I'm not certain it was a timeout issue, just a likely cause of the
errors.  Did your Unicorn error logs tell you if there were any
timeouts?

Anyways I'd always put timeouts around any code that accesses remote
services since the Internet is unreliable.  It's also pretty easy to
setup an evil OpenID provider server that can DoS web apps that don't
timeout themselves.

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Tue, Dec 14, 2010 at 9:27 AM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> Hm. Well I was unaware that there was any timeout issue with my
>> application. When a Unicorn process timed out and died, how did the
>> request not timeout with nginx? Was it re-submitted to another worker?
>
> A timeout issue is one *possible* cause of the errors you were seeing
> from nginx.  Of course you know the application better than I do, so,
> I'm not certain it was a timeout issue, just a likely cause of the
> errors.  Did your Unicorn error logs tell you if there were any
> timeouts?

Nothing in the Unicorn logs ever indicated an error like that. I
believe I found the cause though, and this did not seem to affect
Unicorn:

 - I was not using config.threadsafe! (should probably mention that
with the Rainbows! + Rails docs somewhere)
 - I was attempting to JSON serialize request.env. It seems there are
real objects in env, which when you try to serialize them cause your
application to hang forever. Bleh.

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
>  - I was not using config.threadsafe! (should probably mention that
> with the Rainbows! + Rails docs somewhere)

Done and pushed out.  Thanks for following up with your resolution
so others can benefit from your discoveries.

>From 7e2bb251228a30c0d4e66029b20bbbf85bc99a09 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>
Date: Tue, 14 Dec 2010 18:36:34 -0800
Subject: [PATCH] FAQ: add a note about config.threadsafe!

At least one user ran into it:
http://mid.gmane.org/AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
---
 FAQ |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/FAQ b/FAQ
index 7609d55..f45568b 100644
--- a/FAQ
+++ b/FAQ
@@ -98,3 +98,9 @@ For older versions of Rails, the following config.ru will work:
 
 One thing to watch out for is that RAILS_ENV will not be set in the
 environment for you, thus we set it to match RACK_ENV.
+
+=== I'm using threads and Rails is misbehaving!
+
+If you use any of the threaded concurrency models, you will need to use
+{config.threadsafe!}[http://m.onkey.org/thread-safety-for-your-rails]
+in your config/environments/$RAILS_ENV.rb
-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Tue, Dec 14, 2010 at 6:40 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>>  - I was not using config.threadsafe! (should probably mention that
>> with the Rainbows! + Rails docs somewhere)
>
> Done and pushed out.  Thanks for following up with your resolution
> so others can benefit from your discoveries.
>
> >From 7e2bb251228a30c0d4e66029b20bbbf85bc99a09 Mon Sep 17 00:00:00 2001
> From: Eric Wong <normalperson@yhbt.net>
> Date: Tue, 14 Dec 2010 18:36:34 -0800
> Subject: [PATCH] FAQ: add a note about config.threadsafe!
>
> At least one user ran into it:
> http://mid.gmane.org/AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com
> ---
>  FAQ |    6 ++++++
>  1 files changed, 6 insertions(+), 0 deletions(-)
>
> diff --git a/FAQ b/FAQ
> index 7609d55..f45568b 100644
> --- a/FAQ
> +++ b/FAQ
> @@ -98,3 +98,9 @@ For older versions of Rails, the following config.ru will work:
>
>  One thing to watch out for is that RAILS_ENV will not be set in the
>  environment for you, thus we set it to match RACK_ENV.
> +
> +=== I'm using threads and Rails is misbehaving!
> +
> +If you use any of the threaded concurrency models, you will need to use
> +{config.threadsafe!}[http://m.onkey.org/thread-safety-for-your-rails]
> +in your config/environments/$RAILS_ENV.rb

Sadly, also:
https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/3457-actioncontrollercachingsweeper-controller-instance-is-not-thread-safe

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Tue, Dec 14, 2010 at 6:01 PM,  <ghazel@gmail.com> wrote:
> On Tue, Dec 14, 2010 at 9:27 AM, Eric Wong <normalperson@yhbt.net> wrote:
>> ghazel@gmail.com wrote:
>>> Hm. Well I was unaware that there was any timeout issue with my
>>> application. When a Unicorn process timed out and died, how did the
>>> request not timeout with nginx? Was it re-submitted to another worker?
>>
>> A timeout issue is one *possible* cause of the errors you were seeing
>> from nginx.  Of course you know the application better than I do, so,
>> I'm not certain it was a timeout issue, just a likely cause of the
>> errors.  Did your Unicorn error logs tell you if there were any
>> timeouts?
>
> Nothing in the Unicorn logs ever indicated an error like that.

Also with Rainbows! I see a good number of:
2010/12/15 21:37:42 [error] 2104#0: *174851 sendfile() failed (32:
Broken pipe) while sending request to upstream,
2010/12/15 21:37:55 [error] 2090#0: *175527 readv() failed (104:
Connection reset by peer) while reading upstream,

on our photo upload page. These errors did not occur with Unicorn, and
the Unicorn logs said nothing about a failure either.

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Mon, Dec 13, 2010 at 10:35 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> I'm running a bit of my traffic through some Rainbows! right now, but I got:
>>
>> 2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
>> referrer: "https://foofoo.com"
>> 2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"
>>
>> From nginx in the error log. My proxy_read_timeout is 300, and my
>> listen backlog is 2048 (for now...). Basically my Rainbows! config is
>> identical to my Unicorn config, where I have not seen that happen,
>> except I added "Rainbows! { use :ThreadPool; worker_connections 100
>> }".
>
> Was your app hitting the Unicorn kill -9 timeout frequently before?  In
> Rainbows!, the kill -9 timeout only kicks in when the entire
> interpreter/VM is stuck due to a broken C extension or bug in Ruby.
>
> If it's some component of your app taking a long time (and relying on
> the Unicorn kill -9 timeout), you can try the Rainbows::ThreadTimeout
> middleware: http://rainbows.rubyforge.org/Rainbows/ThreadTimeout.html

I believe I found a bug in ThreadTimeout. It seems to be terminating a
request immediately every @timeout seconds. I'm using :ThreadSpawn and
my initializer is:
use Rainbows::ThreadTimeout, :timeout => 299

The observed behavior is that the request dies < 1 second later with
Rainbows::ThreadTimeout::ExecutionExpired. If I look 299 seconds back
in the log, I see another request killed prematurely. If I look 299
further back from that, I see the process started at this time.
Glancing at the code:

        now = Time.now
        @lock.synchronize do
          @active.delete_if do |thread, time|
            time >= now and thread.raise(ExecutionExpired).nil?
          end
        end

The "time >= now" seems incorrect to me. Since "time" is set to
"Time.now + @timeout" it will be greater than Time.now while the
request still has time left. I believe that should read "now >= time".

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> I believe I found a bug in ThreadTimeout. It seems to be terminating a
> request immediately every @timeout seconds. I'm using :ThreadSpawn and
> my initializer is:
> use Rainbows::ThreadTimeout, :timeout => 299
> 
> The observed behavior is that the request dies < 1 second later with
> Rainbows::ThreadTimeout::ExecutionExpired. If I look 299 seconds back
> in the log, I see another request killed prematurely. If I look 299
> further back from that, I see the process started at this time.
> Glancing at the code:
> 
>         now = Time.now
>         @lock.synchronize do
>           @active.delete_if do |thread, time|
>             time >= now and thread.raise(ExecutionExpired).nil?
>           end
>         end
> 
> The "time >= now" seems incorrect to me. Since "time" is set to
> "Time.now + @timeout" it will be greater than Time.now while the
> request still has time left. I believe that should read "now >= time".

Yup, pushing out a trivial fix in a bit along with a test case.
Thanks!

(sorry for the delayed response, been under the weather lately)
-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[Eric Wong]

ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> Also with Rainbows! I see a good number of:
> 2010/12/15 21:37:42 [error] 2104#0: *174851 sendfile() failed (32:
> Broken pipe) while sending request to upstream,
> 2010/12/15 21:37:55 [error] 2090#0: *175527 readv() failed (104:
> Connection reset by peer) while reading upstream,
> 
> on our photo upload page. These errors did not occur with Unicorn, and
> the Unicorn logs said nothing about a failure either.

Could this be related to the ThreadTimeout middleware bug?

Still working on making the tests run properly for that, but I seem to
be hitting another bug somewhere...

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: rails 2 and slow external services

[ghazel-Re5JQEeQqe8AvxtiuMwx3w]

On Sun, Dec 19, 2010 at 6:52 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
>> Also with Rainbows! I see a good number of:
>> 2010/12/15 21:37:42 [error] 2104#0: *174851 sendfile() failed (32:
>> Broken pipe) while sending request to upstream,
>> 2010/12/15 21:37:55 [error] 2090#0: *175527 readv() failed (104:
>> Connection reset by peer) while reading upstream,
>>
>> on our photo upload page. These errors did not occur with Unicorn, and
>> the Unicorn logs said nothing about a failure either.
>
> Could this be related to the ThreadTimeout middleware bug?

I don't think so. I added the ThreadTimeout middleware after many of
these had occurred.

-Greg
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying