What happens when a client terminates a connection?

What happens when a client terminates a connection?

[Jesse Storimer]

I've been trying to understand what happens in Unicorn when a client
terminates a connection, and nginx logs a 499 response code.

In my debugging this can happen if the client is on a flaky connection,
or if they double-click a form submit button, the first request is
terminated and nginx logs a 499 response code.

It seems that in this case the Rails app actually aborts the request,
wherever it is in the course of it. The issue I ran into is that my app
made a destructive request to an external service in 
the context of a request, but the client disconnected before the app was
able to respond. So the external service returned its response but the
request was aborted before the app was able 
to commit its transaction to the database, confusion ensued.

Can you confirm that this is actually what happens in Unicorn when the
client disconnects? I'm not seeing anything in the logs to indicate the
actual behaviour.

In dealing with this I'm thinking about turning on
proxy_ignore_client_abort
(http://wiki.nginx.org/HttpProxyModule#proxy_ignore_client_abort) so
that requests that make it to the Rails 
app aren't aborted. Does anyone have experience with this? I can see it
causing its own sorts of confusion.

Jesse

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: What happens when a client terminates a connection?

[Eric Wong]

Jesse Storimer <jstorimer@gmail.com> wrote:
> I've been trying to understand what happens in Unicorn when a client
> terminates a connection, and nginx logs a 499 response code.

We'd have to read the nginx sources to answer what nginx does, but
of course I can answer what Unicorn does off the top of my head.

> In my debugging this can happen if the client is on a flaky connection,
> or if they double-click a form submit button, the first request is
> terminated and nginx logs a 499 response code.

<snip> (the snipped paragraph deserves independent observation/attention)

> Can you confirm that this is actually what happens in Unicorn when the
> client disconnects? I'm not seeing anything in the logs to indicate the
> actual behaviour.

It depends on when exactly the client (nginx) disconnect is detected.
Unicorn has 4 distinct states :

1) reading headers, if a client disconnects before it has written _all_
   of its request headers, the Rack app will never be called.

   Since no applicaton logic fired at this point.

2) inside Rack dispatch (rack.input reading)
   This will abort the Rack application dispatch if your client
   disconnects before _all_ of the request body is sent.  Unlike
   most servers, Unicorn lazily reads any request bodies.

   You can catch exceptions from env["rack.input"].{read,gets,each}
   to detect this.

   The Unicorn::PrereadInput middleware can minimize the time window for
   this state by reading the request body ASAP.

   You can also ignore this if your app isn't handling
   requests with bodies (POST/PUT), but since you mentioned form
   input...

3) inside Rack dispatch (after rack.input reading)
   Your app has no way of knowing your client disconnected at
   this stage.  You can hack Unicorn to IO.select in a separate
   thread, but there'll always be exposed windows leading up to
   4) so it's not worth it...

4) writing the response: Unicorn will abort whenever a socket
   error is detected.  Keep in mind that every single part of the
   Rack response array can be dynamically generated by the app.
   Your application can still be "running" even though the Rack app
   has returned its response for Unicorn to start writing.

   Clients/Rack middleware can be written to detect this in the
   response body "close" method by checking if body.each completed.

> In dealing with this I'm thinking about turning on
> proxy_ignore_client_abort
> (http://wiki.nginx.org/HttpProxyModule#proxy_ignore_client_abort) so
> that requests that make it to the Rails 
> app aren't aborted. Does anyone have experience with this? I can see it
> causing its own sorts of confusion.

I've never used it.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: What happens when a client terminates a connection?

[Eric Wong]

Jesse Storimer <jstorimer@gmail.com> wrote:
> It seems that in this case the Rails app actually aborts the request,
> wherever it is in the course of it. The issue I ran into is that my app
> made a destructive request to an external service in 
> the context of a request, but the client disconnected before the app was
> able to respond. So the external service returned its response but the
> request was aborted before the app was able 
> to commit its transaction to the database, confusion ensued.

You should make this request to the external service and wait for this
response inside state 3) as described in my other reply[1].

If you're affected by state 2), you should minimize the time in state 2)
by using PrereadInput middleware to jump to state 3) (and you'll avoid
making the external request entirely if the preread input fails due
to the client aborting).

Your goal is to minimize the time spent processing a non-idempotent
request/parts-of-the-request and isolate where a client can fail (and
maximize the time where they can gracefully recover).

If possible, the request you make to the external service should be
idempotent, but it doesn't seem to be...


I haven't encountered this problem myself in many years, but here's
what I did in the past for similar problems:

  Call the external service asynchronously and look at the various
  background job/queue libraries available to handle this.  The client
  should probably auto-refresh on a page (idempotently) until the async
  request is complete.

  I would even hold off on starting your external request until the client
  has hit the auto-refresh page.  This way you know the client has started
  the refresh cycle and you can fake the idempotency of the external
  request on your app side.


[1] - http://mid.gmane.org/20110808192824.GA5759@dcvr.yhbt.net

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: What happens when a client terminates a connection?

[Jesse Storimer]

On Mon, Aug 08, 2011 at 07:32:52PM +0000, Eric Wong wrote:
> (not sure if you're subscribed or not, actually, I don't see you in
> mailman...)
> 
> ----- Forwarded message from Eric Wong <normalperson@yhbt.net> -----
> 
> From: Eric Wong <normalperson@yhbt.net>
> To: unicorn list <mongrel-unicorn@rubyforge.org>
> Subject: Re: What happens when a client terminates a connection?
> 
> Jesse Storimer <jstorimer@gmail.com> wrote:
> > I've been trying to understand what happens in Unicorn when a client
> > terminates a connection, and nginx logs a 499 response code.
> 
> We'd have to read the nginx sources to answer what nginx does, but
> of course I can answer what Unicorn does off the top of my head.
> 
> > In my debugging this can happen if the client is on a flaky connection,
> > or if they double-click a form submit button, the first request is
> > terminated and nginx logs a 499 response code.
> 
> <snip> (the snipped paragraph deserves independent observation/attention)
> 
> > Can you confirm that this is actually what happens in Unicorn when the
> > client disconnects? I'm not seeing anything in the logs to indicate the
> > actual behaviour.
> 
> It depends on when exactly the client (nginx) disconnect is detected.
> Unicorn has 4 distinct states :
> 
> 1) reading headers, if a client disconnects before it has written _all_
>    of its request headers, the Rack app will never be called.
> 
>    Since no applicaton logic fired at this point.
> 
> 2) inside Rack dispatch (rack.input reading)
>    This will abort the Rack application dispatch if your client
>    disconnects before _all_ of the request body is sent.  Unlike
>    most servers, Unicorn lazily reads any request bodies.
> 
>    You can catch exceptions from env["rack.input"].{read,gets,each}
>    to detect this.
> 
>    The Unicorn::PrereadInput middleware can minimize the time window for
>    this state by reading the request body ASAP.
> 
>    You can also ignore this if your app isn't handling
>    requests with bodies (POST/PUT), but since you mentioned form
>    input...
> 
> 3) inside Rack dispatch (after rack.input reading)
>    Your app has no way of knowing your client disconnected at
>    this stage.  You can hack Unicorn to IO.select in a separate
>    thread, but there'll always be exposed windows leading up to
>    4) so it's not worth it...
> 
> 4) writing the response: Unicorn will abort whenever a socket
>    error is detected.  Keep in mind that every single part of the
>    Rack response array can be dynamically generated by the app.
>    Your application can still be "running" even though the Rack app
>    has returned its response for Unicorn to start writing.
> 
>    Clients/Rack middleware can be written to detect this in the
>    response body "close" method by checking if body.each completed.
> 
> > In dealing with this I'm thinking about turning on
> > proxy_ignore_client_abort
> > (http://wiki.nginx.org/HttpProxyModule#proxy_ignore_client_abort) so
> > that requests that make it to the Rails 
> > app aren't aborted. Does anyone have experience with this? I can see it
> > causing its own sorts of confusion.
> 
> I've never used it.
> 
> -- 
> Eric Wong
> 
> ----- End forwarded message -----
> 
> -- 
> Eric Wong

Thanks for that explanation. Just so I understand, once the Rack application 
enters 3) then it should be unaffected by a client disconnect, or any
socket error? I'll definitely give PrereadInput a try in that case.

When you say that Unicorn lazily reads request bodies, do you mean that
my Rails application might already be in the middle of processing the
request but Unicorn is still reading from the client socket? At that
point won't Rails have read all of the request body? Or does that only
apply if I stick a middleware at the front of the stack that does all
the heavy lifting?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: What happens when a client terminates a connection?

[Eric Wong]

Jesse Storimer <jstorimer@gmail.com> wrote:
> Eric Wong <normalperson@yhbt.net> wrote:
> > Unicorn has 4 distinct states :

<snip>

> > 3) inside Rack dispatch (after rack.input reading)
> >    Your app has no way of knowing your client disconnected at
> >    this stage.  You can hack Unicorn to IO.select in a separate
> >    thread, but there'll always be exposed windows leading up to
> >    4) so it's not worth it...
> > 
> > 4) writing the response: Unicorn will abort whenever a socket
> >    error is detected.  Keep in mind that every single part of the
> >    Rack response array can be dynamically generated by the app.
> >    Your application can still be "running" even though the Rack app
> >    has returned its response for Unicorn to start writing.
> > 
> >    Clients/Rack middleware can be written to detect this in the
> >    response body "close" method by checking if body.each completed.
> 
> Thanks for that explanation. Just so I understand, once the Rack application 
> enters 3) then it should be unaffected by a client disconnect, or any
> socket error? I'll definitely give PrereadInput a try in that case.

Yes, however I don't think I made it clear that 3) will /always/ transition
to state 4).  So you'll be able to use body.close to detect a client
write failure.

> When you say that Unicorn lazily reads request bodies, do you mean that
> my Rails application might already be in the middle of processing the
> request but Unicorn is still reading from the client socket?

Yes.  We use http://unicorn.bogomips.org/Unicorn::TeeInput.html by
default so it gives the Rack app a chance to reject a client
if it sees something it doesnt like

> At that point won't Rails have read all of the request body? Or does
> that only apply if I stick a middleware at the front of the stack that
> does all the heavy lifting?

I'm not certain about middlewares.  Rails may attempt to read all input
ASAP anyways depending on the request Content-Type/Encoding.  Check the
Rails/Rack sources for your versions of Rails/Rack to be sure.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying