unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: "Bráulio Bhavamitra" <braulio@eita.org.br>
To: Eric Wong <e@80x24.org>
Cc: unicorn-public <unicorn-public@bogomips.org>
Subject: Re: Dealing with big uploads and/or slow clients
Date: Fri, 6 Jun 2014 10:31:59 -0300	[thread overview]
Message-ID: <CAJri6_sisz8r1Hoon4V54y8E-gf0uMA0nq-orZnMq4Gue6Syyg@mail.gmail.com> (raw)
In-Reply-To: <20140606012737.GA29097@dcvr.yhbt.net>

Hello Eric,

Sorry, switched to plain text now...

So the idea is that nginx will buffer the upload request until all
data is received and then it will send it to unicorn? So unicorn only
receives the full request? Should I do some configuration to nginx for
this?

In fact, we are still using apache, but are planing to switch to nginx soon...

regards,
bráulio




On Thu, Jun 5, 2014 at 10:27 PM, Eric Wong <e@80x24.org> wrote:
> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>> Hello all,
>
> Hello, please stop sending HTML email, my spam filters are configured to
> give HTML mail high spam scores and forces moderation, so you may not
> get a response for a long time.  Non-English signature also tends to get
> flagged as spam.  This is an English list.  (Fwiw, we get 10-20 spams
> each day to this list which are filtered)
>
>> How do you deal with big uploads and/or slow clients with unicorn? The only
>> solution I've found was to increase timeout, because nginx can't help too,
>> or can?
>
> nginx handles slow clients extremely well.
>
> How big are you talking about and how much RAM do you have for your OS
> page cache?  Big uploads are OK if you have enough memory for caching in
> Linux (or fast SSDs).  The Linux page cache is very effective if you can
> process the data fairly quickly.
>
> If your clients are very slow and uploads cannot fit in the Linux
> page cache, lower your vm.dirty_* knobs (see kernel docs) to force
> earlier write-out so you don't get a thundering herd of disk activity
> when you run out of RAM.
>
> I don't know about caching in other OSes.
>
> My earliest deployments of unicorn (on a LAN) handled several terabytes
> of uploads every day, so working with big uploads has always been
> a priority.
>
> The reason unicorn+nginx works well (or at all) is the attention paid to
> data costs:
>
> * Ruby processes are expensive, tens to hundreds of megabytes each
> * client connections are cheap, several kilobytes at most
>   (including kernel data structures)
> * networks are slow, so most clients trickle
>
> Thus we spend more time buffering in places where things are cheap, and
> spend as little time holding on to big Ruby processes as possible.
>
> .. And thus intense dislike of bloated HTML, top-posting and giant
> signatures is directly tied to my software design philosophy.



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

  parent reply	other threads:[~2014-06-06 13:32 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-06-05 17:39 Dealing with big uploads and/or slow clients Bráulio Bhavamitra
2014-06-06  1:27 ` Eric Wong
2014-06-06  2:51   ` Sam Saffron
2014-06-06  5:59     ` Eric Wong
2014-06-06 13:31   ` Bráulio Bhavamitra [this message]
2014-06-06 15:13     ` Michael Fischer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAJri6_sisz8r1Hoon4V54y8E-gf0uMA0nq-orZnMq4Gue6Syyg@mail.gmail.com \
    --to=braulio@eita.org.br \
    --cc=e@80x24.org \
    --cc=unicorn-public@bogomips.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).