"Read error: #<IOError: closed stream>" and uploads

"Read error: #<IOError: closed stream>" and uploads

[Alexey Bondar]

Hi. 

Unicorn 3.3.1 fails with "Read error: #<IOError: closed stream>" on request with attachments bigger than ~150kb. 

Ruby: 1.9.2-p0
Rails: 3.0.1

Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa
Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg

I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb.

Any ides why this happens?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: "Read error: #<IOError: closed stream>" and uploads

[Eric Wong]

Alexey Bondar <alexey.bondar@gmail.com> wrote:
> Hi. 
> 
> Unicorn 3.3.1 fails with "Read error: #<IOError: closed stream>" on request with attachments bigger than ~150kb. 
> 
> Ruby: 1.9.2-p0
> Rails: 3.0.1
> 
> Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa
> Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg
> 
> I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb.

Anything in the nginx error log?  Did this happen with Unicorn 1.1.x?
The input code changed in the 3.x series drastically to support
non-rewindable input, so some bugs could've crept in.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: "Read error: #<IOError: closed stream>" and uploads

[Eric Wong]

Alexey Bondar <alexey.bondar@gmail.com> wrote:
(back to mailing list)
> On Jan 24, 2011, at 21:19 , Eric Wong wrote:
> > Eric Wong <normalperson@yhbt.net> wrote:
> >> Alexey Bondar <alexey.bondar@gmail.com> wrote:
> >>> Hi. 
> >>> 
> >>> Unicorn 3.3.1 fails with "Read error: #<IOError: closed stream>" on request with attachments bigger than ~150kb. 
> >>> 
> >>> Ruby: 1.9.2-p0
> >>> Rails: 3.0.1
> >>> 
> >>> Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa
> >>> Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg
> >>> 
> >>> I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb.
> >> 
> >> Anything in the nginx error log?  Did this happen with Unicorn 1.1.x?
> >> The input code changed in the 3.x series drastically to support
> >> non-rewindable input, so some bugs could've crept in.
> 
> In nginx log only two messages: body buffered to disk and backend timeout. 

Wait, nginx buffered the body to disk when client_body_buffer_size is
128mb and you have trouble with a 150kb request?

> Same issue with 1.1.x

OK.  Which timeout exactly are you hitting in nginx?  How slow is your
client?  Is your request queue very huge and you don't have enough
Unicorn workers to handle the load?

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: "Read error: #<IOError: closed stream>" and uploads

[Eric Wong]

Alexey Bondar <alexey.bondar@gmail.com> wrote:
> On Jan 25, 2011, at 03:51 , Eric Wong wrote:
> > Wait, nginx buffered the body to disk when client_body_buffer_size is
> > 128mb and you have trouble with a 150kb request?
> 
> Oops, my fault: not body_buffer_size, but client_max_body_size.  client_body_buffer_size is 128k.
> 
> As I understand, in this case nginx should buffer whole request body, before sending to backend? 

Yes, nginx always buffers the whole request body

> >> Same issue with 1.1.x
> > 
> > OK.  Which timeout exactly are you hitting in nginx?  
> 
> Logs: 
> 2011/01/23 21:09:05 [warn] 79095#0: *261 a client request body is buffered to a temporary file /var/spool/nginx-client-body/3/00/0000000003

<snip>

> 2011/01/23 21:09:35 [error] 79095#0: *261 upstream timed out (60: Operation timed out) while sending request to upstream

<snip>

Very strange.  Do you process env["rack.input"] directly from your app
or do you let Rails/Rack handle everything?

This probably won't help, but can you try increasing the
proxy_send_timeout in nginx?  What's your Unicorn timeout?

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: "Read error: #<IOError: closed stream>" and uploads

[Eric Wong]

Alexey Bondar <alexey.bondar@gmail.com> wrote:
> Unicorn timeout is 75s. I just tried with 75s proxy_send_timeout: same
> issue.

What client are you using to send?  I wonder if Content-Length is
somehow screwed up because a client is trying to compress the data.
What happens when you try hitting Unicorn directly without nginx?

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: "Read error: #<IOError: closed stream>" and uploads

[Eric Wong]

Alexey Bondar <alexey.bondar@gmail.com> wrote:
> On Jan 25, 2011, at 06:53 , Eric Wong wrote:
> > What client are you using to send?  I wonder if Content-Length is
> > somehow screwed up because a client is trying to compress the data.
> 
> Different: Webkit based browsers, FF, Opera. 

OK.  I'm fairly certain those clients don't make compressed requests,
but I'm out of ideas as to what could be wrong...

> > What happens when you try hitting Unicorn directly without nginx?
> 
> Hm... How I can directly hit unicorn via unix socket?

Just have it listen on a TCP port, too (it can handle multiple
listen statements in the config).  Maybe we can narrow it down
to something in the nginx config...

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: "Read error: #<IOError: closed stream>" and uploads

[Eric Wong]

Alexey Bondar <alexey.bondar@gmail.com> wrote:
> On Jan 25, 2011, at 07:10 , Eric Wong wrote:
> > OK.  I'm fairly certain those clients don't make compressed requests,
> > but I'm out of ideas as to what could be wrong...
> > 
> > Just have it listen on a TCP port, too (it can handle multiple
> > listen statements in the config).  Maybe we can narrow it down
> > to something in the nginx config...
> 
> Hm. Yeah, direct access works well. 
> Uh, what can be wrong with nginx config in this case? :(

I would trim your config down the the bare minimum and use the defaults
wherever you can.  Then start adding things back in and see what broke
it.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

[PATCH] KNOWN_ISSUES: FreeBSD 8 and sendfile can be buggy

[Eric Wong]

Alexey Bondar <alexey.bondar@gmail.com> wrote:
> On Jan 25, 2011, at 07:54 , Eric Wong wrote:
> > Alexey Bondar <alexey.bondar@gmail.com> wrote:
> >> Hm. Yeah, direct access works well. 
> >> Uh, what can be wrong with nginx config in this case? :(
> > 
> > I would trim your config down the the bare minimum and use the defaults
> > wherever you can.  Then start adding things back in and see what broke
> > it.
> 
> Maxim Dounin from nginx mail list says that this issue caused by some issues with sendfile in
> FreeBSD 8. 
> 
> After setting sendfile to off uploads works well. 
> 
> You can add this info to known 3rd party issues. :)

Thanks Alexey!

>From 8ac0ae45a04f5f121f323c182403ef6eb0d8aa18 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Tue, 25 Jan 2011 13:30:21 -0800
Subject: [PATCH] KNOWN_ISSUES: FreeBSD 8 and sendfile can be buggy

Reported by Alexey Bondar.
---
 KNOWN_ISSUES |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index bc3dac5..95f4e43 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -6,6 +6,10 @@ acceptable solution.  Those issues are documented here.
 * For notes on sandboxing tools such as Bundler or Isolate,
   see the {Sandbox}[link:Sandbox.html] page.
 
+* nginx with "sendfile on" under FreeBSD 8 is broken when
+  uploads are buffered to disk.  Disabling sendfile is required to
+  work around this bug which should be fixed in newer versions of FreeBSD.
+
 * Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will
   segfault if called after forking.  This is fixed in trunk (r26936) and
   should be backported to the next 1.9.1 stable release (after p378).
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying