Zbatery/Rainbows keepalive problem?

Zbatery/Rainbows keepalive problem?

[Jake Douglas]

I am using zbatery with ThreadPool and just upgraded to 0.5.0, which
subsequently upgraded rainbows to 2.0.0 and unicorn to 3.0.0. My
browser(s) now hangs for a while during each request. I noticed some
discussion about keep alive in the changelog. Putting
"keepalive_timeout 0" in the rainbows configuration fixes it. Am I
doing something wrong or is this a bug?

Thanks


-- 
Jake Douglas
206-795-9207
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Eric Wong]

Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I am using zbatery with ThreadPool and just upgraded to 0.5.0, which
> subsequently upgraded rainbows to 2.0.0 and unicorn to 3.0.0. My
> browser(s) now hangs for a while during each request. I noticed some
> discussion about keep alive in the changelog. Putting
> "keepalive_timeout 0" in the rainbows configuration fixes it. Am I
> doing something wrong or is this a bug?

Are you setting a Content-Length or Transfer-Encoding in your app?

You can add Rack::Chunked and Rack::ContentLength middlewares to your
config (some frameworks like Rails already do this, I think).

Which versions of Zbatery/Rainbows!/Unicorn were you using before which
exhibited this?  Rainbows!/Zbatery has supported keepalive for some time
and none of the three servers ever included
Rack::Chunked/Rack::ContentLength in its middleware stack by default.

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Jake Douglas]

I was using zbatery 0.4.0, Unicorn 2.0.1, unsure of the rainbows
version. This is a Rails 3.0.3 app and I am not setting any headers
myself. Are you saying that Rails is doing something that is causing
the problem? Please bear with me, as I am not familiar with the
details of how keep alive works.

Thanks

On Wed, Nov 24, 2010 at 1:17 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> I am using zbatery with ThreadPool and just upgraded to 0.5.0, which
>> subsequently upgraded rainbows to 2.0.0 and unicorn to 3.0.0. My
>> browser(s) now hangs for a while during each request. I noticed some
>> discussion about keep alive in the changelog. Putting
>> "keepalive_timeout 0" in the rainbows configuration fixes it. Am I
>> doing something wrong or is this a bug?
>
> Are you setting a Content-Length or Transfer-Encoding in your app?
>
> You can add Rack::Chunked and Rack::ContentLength middlewares to your
> config (some frameworks like Rails already do this, I think).
>
> Which versions of Zbatery/Rainbows!/Unicorn were you using before which
> exhibited this?  Rainbows!/Zbatery has supported keepalive for some time
> and none of the three servers ever included
> Rack::Chunked/Rack::ContentLength in its middleware stack by default.
>
> --
> Eric Wong
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>



-- 
Jake Douglas
206-795-9207
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Eric Wong]

Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Nov 24, 2010 at 1:17 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> I am using zbatery with ThreadPool and just upgraded to 0.5.0, which
> >> subsequently upgraded rainbows to 2.0.0 and unicorn to 3.0.0. My
> >> browser(s) now hangs for a while during each request. I noticed some
> >
> > Are you setting a Content-Length or Transfer-Encoding in your app?
> >
> > You can add Rack::Chunked and Rack::ContentLength middlewares to your
> > config (some frameworks like Rails already do this, I think).
> >
> > Which versions of Zbatery/Rainbows!/Unicorn were you using before which
> > exhibited this?  Rainbows!/Zbatery has supported keepalive for some time
> > and none of the three servers ever included
> > Rack::Chunked/Rack::ContentLength in its middleware stack by default.
>
> I was using zbatery 0.4.0, Unicorn 2.0.1, unsure of the rainbows
> version. This is a Rails 3.0.3 app and I am not setting any headers
> myself. Are you saying that Rails is doing something that is causing
> the problem? Please bear with me, as I am not familiar with the
> details of how keep alive works.

<top posting corrected>

Responses without keepalive don't need additional headers since the
server just closes the connection to notify the client.

HTTP/1.1 with keepalive requires a Content-Length (or Transfer-Encoding:
chunked) header to determine response boundaries, otherwise clients
would never know when one response ends and another begins.

The normal Rails "render" path already seems[1] to add Content-Length;
are you doing something else to send responses from Rails?

Including the Rack::Chunked/Rack::ContentLength middlewares (at the top
of the middleware stack) should solve all your problems.

If you only serve HTTP/1.1, you can probably get away with only
Rack::Chunked...


[1] - I haven't had a chance to test this myself

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Jake Douglas]

On Wed, Nov 24, 2010 at 2:14 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Wed, Nov 24, 2010 at 1:17 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
>> > Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >> I am using zbatery with ThreadPool and just upgraded to 0.5.0, which
>> >> subsequently upgraded rainbows to 2.0.0 and unicorn to 3.0.0. My
>> >> browser(s) now hangs for a while during each request. I noticed some
>> >
>> > Are you setting a Content-Length or Transfer-Encoding in your app?
>> >
>> > You can add Rack::Chunked and Rack::ContentLength middlewares to your
>> > config (some frameworks like Rails already do this, I think).
>> >
>> > Which versions of Zbatery/Rainbows!/Unicorn were you using before which
>> > exhibited this?  Rainbows!/Zbatery has supported keepalive for some time
>> > and none of the three servers ever included
>> > Rack::Chunked/Rack::ContentLength in its middleware stack by default.
>>
>> I was using zbatery 0.4.0, Unicorn 2.0.1, unsure of the rainbows
>> version. This is a Rails 3.0.3 app and I am not setting any headers
>> myself. Are you saying that Rails is doing something that is causing
>> the problem? Please bear with me, as I am not familiar with the
>> details of how keep alive works.
>
> <top posting corrected>
>
> Responses without keepalive don't need additional headers since the
> server just closes the connection to notify the client.
>
> HTTP/1.1 with keepalive requires a Content-Length (or Transfer-Encoding:
> chunked) header to determine response boundaries, otherwise clients
> would never know when one response ends and another begins.
>
> The normal Rails "render" path already seems[1] to add Content-Length;
> are you doing something else to send responses from Rails?
>
> Including the Rack::Chunked/Rack::ContentLength middlewares (at the top
> of the middleware stack) should solve all your problems.
>
> If you only serve HTTP/1.1, you can probably get away with only
> Rack::Chunked...
>
>
> [1] - I haven't had a chance to test this myself
>
> --
> Eric Wong


I am not doing anything special to send responses from Rails, only 'render'.

I can see in my browser that keep-alive is set, and Content-Length is
also set. It looks like the first couple requests finish quickly, and
then the remaining take an additional 5 seconds each. It seems the
browser is hanging for the default keep-alive timeout of 5 seconds on
each request before proceeding. This happens with different browsers.

Is there additional information I can provide to give better insight
into why this is happening?

Thanks
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Eric Wong]

Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Nov 24, 2010 at 2:14 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> >
> > The normal Rails "render" path already seems[1] to add Content-Length;
> > are you doing something else to send responses from Rails?
> 
> I am not doing anything special to send responses from Rails, only 'render'.
> 
> I can see in my browser that keep-alive is set, and Content-Length is
> also set. It looks like the first couple requests finish quickly, and
> then the remaining take an additional 5 seconds each. It seems the
> browser is hanging for the default keep-alive timeout of 5 seconds on
> each request before proceeding. This happens with different browsers.

Interesting.  I rarely actually do anything with a full web browser,
but I can confirm keepalive works as expected with curl...  Is there
anything in your error logs such as exceptions being thrown by your
app?

Can anybody else reproduce this behavior Jake is seeing?

> Is there additional information I can provide to give better insight
> into why this is happening?

tcpdump or similar output would be useful, thanks!

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Jake Douglas]

On Sat, Nov 27, 2010 at 12:50 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Wed, Nov 24, 2010 at 2:14 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
>> >
>> > The normal Rails "render" path already seems[1] to add Content-Length;
>> > are you doing something else to send responses from Rails?
>>
>> I am not doing anything special to send responses from Rails, only 'render'.
>>
>> I can see in my browser that keep-alive is set, and Content-Length is
>> also set. It looks like the first couple requests finish quickly, and
>> then the remaining take an additional 5 seconds each. It seems the
>> browser is hanging for the default keep-alive timeout of 5 seconds on
>> each request before proceeding. This happens with different browsers.
>
> Interesting.  I rarely actually do anything with a full web browser,
> but I can confirm keepalive works as expected with curl...  Is there
> anything in your error logs such as exceptions being thrown by your
> app?
>
> Can anybody else reproduce this behavior Jake is seeing?
>
>> Is there additional information I can provide to give better insight
>> into why this is happening?
>
> tcpdump or similar output would be useful, thanks!
>
> --
> Eric Wong
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>


It looks like the first issue is that by default Rails does not
actually use Rack::ContentLength or set the Content-Length itself.

The next issue is this bug, where Rack::ContentLength has some
conditions that cause it to skip handling of the
ActionDispatch::Response object from Rails. (ActionDispatch::Response
< Rack::Response)
http://rack.lighthouseapp.com/projects/22435/tickets/95-responsefinish-and-contentlength

Fixing and enabling Rack::ContentLength seems to fix the issue for a
single request to Rails from curl, but I am still getting the problem
in my browser(s). Zbatery logs say that the request finishes quickly
but the browser hangs for 5 seconds at a time. It is unclear to me
what the cause is. I will try to do some more research and let you
know the result.

Thanks
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Jake Douglas]

On Sat, Nov 27, 2010 at 1:55 PM, Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Sat, Nov 27, 2010 at 12:50 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
>> Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> On Wed, Nov 24, 2010 at 2:14 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
>>> >
>>> > The normal Rails "render" path already seems[1] to add Content-Length;
>>> > are you doing something else to send responses from Rails?
>>>
>>> I am not doing anything special to send responses from Rails, only 'render'.
>>>
>>> I can see in my browser that keep-alive is set, and Content-Length is
>>> also set. It looks like the first couple requests finish quickly, and
>>> then the remaining take an additional 5 seconds each. It seems the
>>> browser is hanging for the default keep-alive timeout of 5 seconds on
>>> each request before proceeding. This happens with different browsers.
>>
>> Interesting.  I rarely actually do anything with a full web browser,
>> but I can confirm keepalive works as expected with curl...  Is there
>> anything in your error logs such as exceptions being thrown by your
>> app?
>>
>> Can anybody else reproduce this behavior Jake is seeing?
>>
>>> Is there additional information I can provide to give better insight
>>> into why this is happening?
>>
>> tcpdump or similar output would be useful, thanks!
>>
>> --
>> Eric Wong
>> _______________________________________________
>> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
>> http://rubyforge.org/mailman/listinfo/rainbows-talk
>> Do not quote signatures (like this one) or top post when replying
>>
>
>
> It looks like the first issue is that by default Rails does not
> actually use Rack::ContentLength or set the Content-Length itself.
>
> The next issue is this bug, where Rack::ContentLength has some
> conditions that cause it to skip handling of the
> ActionDispatch::Response object from Rails. (ActionDispatch::Response
> < Rack::Response)
> http://rack.lighthouseapp.com/projects/22435/tickets/95-responsefinish-and-contentlength
>
> Fixing and enabling Rack::ContentLength seems to fix the issue for a
> single request to Rails from curl, but I am still getting the problem
> in my browser(s). Zbatery logs say that the request finishes quickly
> but the browser hangs for 5 seconds at a time. It is unclear to me
> what the cause is. I will try to do some more research and let you
> know the result.
>
> Thanks
>

After some more experimentation, it looks like the browser(s) fail to
close the connection after receiving the responses. Zbatery then waits
until the keepalive_timeout has expired, closes it, and accepts the
next connection. I gave the same application a try using Thin with
keep-alive and everything works fine. I spent a lot of time trying to
identify differences between the two, but the only useful thing I
managed to find is that the browser side never sends a FIN-ACK when
talking to Zbatery. For some reason, the browser doesn't think it's
finished. Any suggestions appreciated.

Thanks
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Eric Wong]

Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> After some more experimentation, it looks like the browser(s) fail to
> close the connection after receiving the responses. Zbatery then waits
> until the keepalive_timeout has expired, closes it, and accepts the
> next connection. I gave the same application a try using Thin with
> keep-alive and everything works fine. I spent a lot of time trying to
> identify differences between the two, but the only useful thing I
> managed to find is that the browser side never sends a FIN-ACK when
> talking to Zbatery. For some reason, the browser doesn't think it's
> finished. Any suggestions appreciated.

A shot in the dark, but I wonder if web browsers do anything special
with the Server: header with regard to keepalive.  Off the top of my
head, I know Thin includes the "Server:" header in all responses while
Rainbows! does not[1].

Can you try loading the Rainbows::ServerToken middleware to add
the Server header?

  use Rainbows::ServerToken, "foo/bar"

Any other differences in the headers the two servers send?

[1] - from what I can tell, rfc2616 does not require the Server: token,
      and recommends it to be configurable.

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Jake Douglas]

On Sun, Nov 28, 2010 at 2:07 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> After some more experimentation, it looks like the browser(s) fail to
>> close the connection after receiving the responses. Zbatery then waits
>> until the keepalive_timeout has expired, closes it, and accepts the
>> next connection. I gave the same application a try using Thin with
>> keep-alive and everything works fine. I spent a lot of time trying to
>> identify differences between the two, but the only useful thing I
>> managed to find is that the browser side never sends a FIN-ACK when
>> talking to Zbatery. For some reason, the browser doesn't think it's
>> finished. Any suggestions appreciated.
>
> A shot in the dark, but I wonder if web browsers do anything special
> with the Server: header with regard to keepalive.  Off the top of my
> head, I know Thin includes the "Server:" header in all responses while
> Rainbows! does not[1].
>
> Can you try loading the Rainbows::ServerToken middleware to add
> the Server header?
>
>  use Rainbows::ServerToken, "foo/bar"
>
> Any other differences in the headers the two servers send?
>
> [1] - from what I can tell, rfc2616 does not require the Server: token,
>      and recommends it to be configurable.
>
> --
> Eric Wong
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>


Adding the Server header does not change anything. The only other
difference I can see is that Rainbows! includes the Date header.
However, removing it also does not change the behavior. I am at a
complete loss now for what is causing this.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Eric Wong]

Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Adding the Server header does not change anything. The only other
> difference I can see is that Rainbows! includes the Date header.
> However, removing it also does not change the behavior. I am at a
> complete loss now for what is causing this.

A few more things, are these GET or POST requests (or something else)?
Is it done via AJAX/XmlHttpRequest?

Can you put together a redistributable test case as a standalone app
that I can hammer on locally?

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Jake Douglas]

On Mon, Nov 29, 2010 at 1:12 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Adding the Server header does not change anything. The only other
>> difference I can see is that Rainbows! includes the Date header.
>> However, removing it also does not change the behavior. I am at a
>> complete loss now for what is causing this.
>
> A few more things, are these GET or POST requests (or something else)?
> Is it done via AJAX/XmlHttpRequest?
>
> Can you put together a redistributable test case as a standalone app
> that I can hammer on locally?
>
> --
> Eric Wong
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>

They are normal GET requests, not XHRs. I'll try to get you a simple
test case sometime this week.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: Zbatery/Rainbows keepalive problem?

[Eric Wong]

Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Mon, Nov 29, 2010 at 1:12 PM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > Jake Douglas <jakecdouglas-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> Adding the Server header does not change anything. The only other
> >> difference I can see is that Rainbows! includes the Date header.
> >> However, removing it also does not change the behavior. I am at a
> >> complete loss now for what is causing this.
> >
> > A few more things, are these GET or POST requests (or something else)?
> > Is it done via AJAX/XmlHttpRequest?
> >
> > Can you put together a redistributable test case as a standalone app
> > that I can hammer on locally?
> >
> They are normal GET requests, not XHRs. I'll try to get you a simple
> test case sometime this week.

Hi Jake, is this still a problem?

I'm only testing with Rack::File, but can't seem to reproduce it
(using Firefox 3.6.12 (ugh)), either.

I just opened up http://rainbows.bogomips.org:8080/README.html
and started clicking some links.  I don't imagine it's too different
based on what the headers show...

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying