unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Sam Saffron <sam.saffron@gmail.com>
To: Eric Wong <e@80x24.org>
Cc: "Bráulio Bhavamitra" <braulio@eita.org.br>,
	unicorn-public <unicorn-public@bogomips.org>
Subject: Re: Dealing with big uploads and/or slow clients
Date: Fri, 6 Jun 2014 12:51:41 +1000	[thread overview]
Message-ID: <CAAtdryP0uxHJwUDNo-qNDLGcSpxpqTfga2bzb+d88tN_yj4DKA@mail.gmail.com> (raw)
In-Reply-To: <20140606012737.GA29097@dcvr.yhbt.net>

Wouldn't you just use Rack Hijack for this, you can easily pull the
socket out of the pipeline and then deal with the upload via
eventmachine or what not.

On Fri, Jun 6, 2014 at 11:27 AM, Eric Wong <e@80x24.org> wrote:
> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>> Hello all,
>
> Hello, please stop sending HTML email, my spam filters are configured to
> give HTML mail high spam scores and forces moderation, so you may not
> get a response for a long time.  Non-English signature also tends to get
> flagged as spam.  This is an English list.  (Fwiw, we get 10-20 spams
> each day to this list which are filtered)
>
>> How do you deal with big uploads and/or slow clients with unicorn? The only
>> solution I've found was to increase timeout, because nginx can't help too,
>> or can?
>
> nginx handles slow clients extremely well.
>
> How big are you talking about and how much RAM do you have for your OS
> page cache?  Big uploads are OK if you have enough memory for caching in
> Linux (or fast SSDs).  The Linux page cache is very effective if you can
> process the data fairly quickly.
>
> If your clients are very slow and uploads cannot fit in the Linux
> page cache, lower your vm.dirty_* knobs (see kernel docs) to force
> earlier write-out so you don't get a thundering herd of disk activity
> when you run out of RAM.
>
> I don't know about caching in other OSes.
>
> My earliest deployments of unicorn (on a LAN) handled several terabytes
> of uploads every day, so working with big uploads has always been
> a priority.
>
> The reason unicorn+nginx works well (or at all) is the attention paid to
> data costs:
>
> * Ruby processes are expensive, tens to hundreds of megabytes each
> * client connections are cheap, several kilobytes at most
>   (including kernel data structures)
> * networks are slow, so most clients trickle
>
> Thus we spend more time buffering in places where things are cheap, and
> spend as little time holding on to big Ruby processes as possible.
>
> .. And thus intense dislike of bloated HTML, top-posting and giant
> signatures is directly tied to my software design philosophy.
>

  reply	other threads:[~2014-06-06  2:51 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-06-05 17:39 Dealing with big uploads and/or slow clients Bráulio Bhavamitra
2014-06-06  1:27 ` Eric Wong
2014-06-06  2:51   ` Sam Saffron [this message]
2014-06-06  5:59     ` Eric Wong
2014-06-06 13:31   ` Bráulio Bhavamitra
2014-06-06 15:13     ` Michael Fischer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAAtdryP0uxHJwUDNo-qNDLGcSpxpqTfga2bzb+d88tN_yj4DKA@mail.gmail.com \
    --to=sam.saffron@gmail.com \
    --cc=braulio@eita.org.br \
    --cc=e@80x24.org \
    --cc=unicorn-public@bogomips.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).