From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS15169 209.85.128.0/17 X-Spam-Status: No, score=0.5 required=3.0 tests=AWL,FREEMAIL_FROM shortcircuit=no autolearn=unavailable version=3.3.2 X-Original-To: unicorn-public@bogomips.org Received: from mail-qa0-f51.google.com (mail-qa0-f51.google.com [209.85.216.51]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id C462F1F5B1 for ; Fri, 6 Jun 2014 13:32:40 +0000 (UTC) Received: by mail-qa0-f51.google.com with SMTP id w8so3699944qac.38 for ; Fri, 06 Jun 2014 06:32:39 -0700 (PDT) X-Received: by 10.140.42.12 with SMTP id b12mr7970728qga.109.1402061559786; Fri, 06 Jun 2014 06:32:39 -0700 (PDT) MIME-Version: 1.0 Sender: brauliobo@gmail.com Received: by 10.140.94.85 with HTTP; Fri, 6 Jun 2014 06:31:59 -0700 (PDT) In-Reply-To: <20140606012737.GA29097@dcvr.yhbt.net> References: <20140606012737.GA29097@dcvr.yhbt.net> From: =?UTF-8?Q?Br=C3=A1ulio_Bhavamitra?= Date: Fri, 6 Jun 2014 10:31:59 -0300 X-Google-Sender-Auth: XHMopZ4xOH-6e7pEDFXc6lN0Q_E Message-ID: Subject: Re: Dealing with big uploads and/or slow clients To: Eric Wong Cc: unicorn-public Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: Hello Eric, Sorry, switched to plain text now... So the idea is that nginx will buffer the upload request until all data is received and then it will send it to unicorn? So unicorn only receives the full request? Should I do some configuration to nginx for this? In fact, we are still using apache, but are planing to switch to nginx soon= ... regards, br=C3=A1ulio On Thu, Jun 5, 2014 at 10:27 PM, Eric Wong wrote: > Br=C3=A1ulio Bhavamitra wrote: >> Hello all, > > Hello, please stop sending HTML email, my spam filters are configured to > give HTML mail high spam scores and forces moderation, so you may not > get a response for a long time. Non-English signature also tends to get > flagged as spam. This is an English list. (Fwiw, we get 10-20 spams > each day to this list which are filtered) > >> How do you deal with big uploads and/or slow clients with unicorn? The o= nly >> solution I've found was to increase timeout, because nginx can't help to= o, >> or can? > > nginx handles slow clients extremely well. > > How big are you talking about and how much RAM do you have for your OS > page cache? Big uploads are OK if you have enough memory for caching in > Linux (or fast SSDs). The Linux page cache is very effective if you can > process the data fairly quickly. > > If your clients are very slow and uploads cannot fit in the Linux > page cache, lower your vm.dirty_* knobs (see kernel docs) to force > earlier write-out so you don't get a thundering herd of disk activity > when you run out of RAM. > > I don't know about caching in other OSes. > > My earliest deployments of unicorn (on a LAN) handled several terabytes > of uploads every day, so working with big uploads has always been > a priority. > > The reason unicorn+nginx works well (or at all) is the attention paid to > data costs: > > * Ruby processes are expensive, tens to hundreds of megabytes each > * client connections are cheap, several kilobytes at most > (including kernel data structures) > * networks are slow, so most clients trickle > > Thus we spend more time buffering in places where things are cheap, and > spend as little time holding on to big Ruby processes as possible. > > .. And thus intense dislike of bloated HTML, top-posting and giant > signatures is directly tied to my software design philosophy. --=20 "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua ideologia. Morra por sua ideologia" P.R. Sarkar EITA - Educa=C3=A7=C3=A3o, Informa=C3=A7=C3=A3o e Tecnologias para Autogest= =C3=A3o http://cirandas.net/brauliobo http://eita.org.br "Paramapurusha =C3=A9 meu pai e Parama Prakriti =C3=A9 minha m=C3=A3e. O un= iverso =C3=A9 meu lar e todos n=C3=B3s somos cidad=C3=A3os deste cosmo. Este universo =C3= =A9 a imagina=C3=A7=C3=A3o da Mente Macroc=C3=B3smica, e todas as entidades est= =C3=A3o sendo criadas, preservadas e destru=C3=ADdas nas fases de extrovers=C3=A3o e introvers=C3=A3o do fluxo imaginativo c=C3=B3smico. No =C3=A2mbito pessoal,= quando uma pessoa imagina algo em sua mente, naquele momento, essa pessoa =C3=A9 a =C3=BAnica propriet=C3=A1ria daquilo que ela imagina, e ningu=C3=A9m mais. = Quando um ser humano criado mentalmente caminha por um milharal tamb=C3=A9m imaginado, a pessoa imaginada n=C3=A3o =C3=A9 a propriedade desse milharal,= pois ele pertence ao indiv=C3=ADduo que o est=C3=A1 imaginando. Este universo fo= i criado na imagina=C3=A7=C3=A3o de Brahma, a Entidade Suprema, por isso a propriedade deste universo =C3=A9 de Brahma, e n=C3=A3o dos microcosmos que tamb=C3=A9m foram criados pela imagina=C3=A7=C3=A3o de Brahma. Nenhuma prop= riedade deste mundo, mut=C3=A1vel ou imut=C3=A1vel, pertence a um indiv=C3=ADduo em particular; tudo =C3=A9 o patrim=C3=B4nio comum de todos." Restante do texto em http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia