large uploads

large uploads

[John Joseph Bachir]

My application accepts uploads from users, which can be quite huge in
some cases. This of course requires setting the unicorn timeout to
something much higher than 60 seconds, more like 10 minutes.

Are there any drawbacks to doing this, other than the obvious drawback
of not killing off long-running requests that are illegitimate?

I've googled quite a bit about this and have found surprisingly little
-- I guess people who have apps that receive uploads just generally
don't use unicorn?

Thanks,
John
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: large uploads

[Eric Wong]

John Joseph Bachir <j@jjb.cc> wrote:
> My application accepts uploads from users, which can be quite huge in
> some cases. This of course requires setting the unicorn timeout to
> something much higher than 60 seconds, more like 10 minutes.
> 
> Are there any drawbacks to doing this, other than the obvious drawback
> of not killing off long-running requests that are illegitimate?

Are your users remote? (outside of your immediate LAN).

The upload speed (and thus timeout) for unicorn should be based on
the nginx <-> unicorn transfer rate.  Unicorn should never talk to users
directly and users can upload slowly to nginx which buffers requests to the
filesystem, first.

> I've googled quite a bit about this and have found surprisingly little
> -- I guess people who have apps that receive uploads just generally
> don't use unicorn?

I have a LAN-only application that regularly processes uploads several
hundreds of megabytes (via PUT) directly to Unicorn.  The local disk I/O
is often the limiting factor since the parallel requests fill up the
kernel buffers and wait on disk I/O.  The Rack application itself
unfortunately needs to seek/rewind so it must be in the filesystem.

Rainbows! with ThreadSpawn/ThreadPool can process uploads without
buffering them to disk first (but the Rack multipart parser may not).
I often stream several hundred megabytes of data directly to apps on
Rainbows! via PUT requests (curl -T), too.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: large uploads

[cliftonk]

You need to be buffering the upload before sending it to Unicorn. Are
you using nginx? We're stuck on Apache, so I just did all my uploading /
image-manipulation with nodejs instead.

Clifton
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: large uploads

[John Joseph Bachir]

On Thu, Sep 29, 2011 at 2:47 PM, Eric Wong <normalperson@yhbt.net> wrote:
> The upload speed (and thus timeout) for unicorn should be based on
> the nginx <-> unicorn transfer rate.  Unicorn should never talk to users
> directly and users can upload slowly to nginx which buffers requests to the
> filesystem, first.

On Thu, Sep 29, 2011 at 2:48 PM,  <cliftonk@gmail.com> wrote:
> You need to be buffering the upload before sending it to Unicorn. Are
> you using nginx?

Thanks for the responses! So, I've see this module:
https://github.com/vkholodkov/nginx-upload-module

Is that what you are referring to? Are there other solutions? Does
NGINX have inherent behavior which buffers requests? (Eric's email
seemed to suggest that).

Thanks,
john
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: large uploads

[Eric Wong]

John Joseph Bachir <j@jjb.cc> wrote:
> On Thu, Sep 29, 2011 at 2:47 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > The upload speed (and thus timeout) for unicorn should be based on
> > the nginx <-> unicorn transfer rate.  Unicorn should never talk to users
> > directly and users can upload slowly to nginx which buffers requests to the
> > filesystem, first.
> 
> On Thu, Sep 29, 2011 at 2:48 PM,  <cliftonk@gmail.com> wrote:
> > You need to be buffering the upload before sending it to Unicorn. Are
> > you using nginx?
> 
> Thanks for the responses! So, I've see this module:
> https://github.com/vkholodkov/nginx-upload-module

I think that's unrelated to what you're doing (it appears to be
for nginx-only setups).

> Is that what you are referring to? Are there other solutions? Does
> NGINX have inherent behavior which buffers requests? (Eric's email
> seemed to suggest that).

Yes, nginx buffers entire requests by default before sending them
to unicorn.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: large uploads

[Justin Giancola]

On Thu, Sep 29, 2011 at 4:05 PM, John Joseph Bachir <j@jjb.cc> wrote:
> Does NGINX have inherent behavior which buffers requests? (Eric's email
> seemed to suggest that).

Nginx always buffers the entire body of requests before calling the
upstream application. If the body exceeds client_body_buffer_size
Nginx will store it in a tempfile.

> Thanks for the responses! So, I've see this module:
> https://github.com/vkholodkov/nginx-upload-module

The upload module works a little differently. It allows you to
configure specific upload endpoints that Nginx will preprocess. Rather
than forwarding the entire request body to the upstream application,
Nginx will parse the multipart/form-data encoded files, store them in
tempfiles, and forward only the paths of the tempfiles to the upstream
application.


Justin
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: large uploads

[John Joseph Bachir]

So buffering the entire request and then sending it to the backend is
the default behavior of HttpUpstreamModule? That's fascinating. It
almost seems like that's a bad design choice, because it inhibits the
possibility of parallel work being done as the request comes in (like,
it's the opposite of how unix pipes work).
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: large uploads

[Eric Wong]

John Joseph Bachir <j@jjb.cc> wrote:
> So buffering the entire request and then sending it to the backend is
> the default behavior of HttpUpstreamModule? That's fascinating. It
> almost seems like that's a bad design choice, because it inhibits the
> possibility of parallel work being done as the request comes in (like,
> it's the opposite of how unix pipes work).

Yes, exactly.  It's a design trade-off; if a client is uploading slowly
(whether intentionally or maliciously) the process-per-client model of
Unicorn is /extremely/ inefficient and will easily be shut down
by slow clients.

If you want to be able to process uploads in a Rack application while
the client is uploading, you'd have to ensure your app is thread-safe
and use Rainbows! + ThreadSpawn/ThreadPool.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying