* Dealing with big uploads and/or slow clients @ 2014-06-05 17:39 Bráulio Bhavamitra 2014-06-06 1:27 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Bráulio Bhavamitra @ 2014-06-05 17:39 UTC (permalink / raw) To: unicorn-public [-- Attachment #1: Type: text/plain, Size: 1558 bytes --] Hello all, How do you deal with big uploads and/or slow clients with unicorn? The only solution I've found was to increase timeout, because nginx can't help too, or can? regards, bráulio -- "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua ideologia. Morra por sua ideologia" P.R. Sarkar EITA - Educação, Informação e Tecnologias para Autogestão http://cirandas.net/brauliobo http://eita.org.br "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e destruídas nas fases de extroversão e introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a única proprietária daquilo que ela imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha por um milharal também imaginado, a pessoa imaginada não é a propriedade desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso a propriedade deste universo é de Brahma, e não dos microcosmos que também foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, mutável ou imutável, pertence a um indivíduo em particular; tudo é o patrimônio comum de todos." Restante do texto em http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia [-- Attachment #2: Type: message/external-body, Size: 84 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Dealing with big uploads and/or slow clients 2014-06-05 17:39 Dealing with big uploads and/or slow clients Bráulio Bhavamitra @ 2014-06-06 1:27 ` Eric Wong 2014-06-06 2:51 ` Sam Saffron 2014-06-06 13:31 ` Bráulio Bhavamitra 0 siblings, 2 replies; 6+ messages in thread From: Eric Wong @ 2014-06-06 1:27 UTC (permalink / raw) To: Bráulio Bhavamitra; +Cc: unicorn-public Bráulio Bhavamitra <braulio@eita.org.br> wrote: > Hello all, Hello, please stop sending HTML email, my spam filters are configured to give HTML mail high spam scores and forces moderation, so you may not get a response for a long time. Non-English signature also tends to get flagged as spam. This is an English list. (Fwiw, we get 10-20 spams each day to this list which are filtered) > How do you deal with big uploads and/or slow clients with unicorn? The only > solution I've found was to increase timeout, because nginx can't help too, > or can? nginx handles slow clients extremely well. How big are you talking about and how much RAM do you have for your OS page cache? Big uploads are OK if you have enough memory for caching in Linux (or fast SSDs). The Linux page cache is very effective if you can process the data fairly quickly. If your clients are very slow and uploads cannot fit in the Linux page cache, lower your vm.dirty_* knobs (see kernel docs) to force earlier write-out so you don't get a thundering herd of disk activity when you run out of RAM. I don't know about caching in other OSes. My earliest deployments of unicorn (on a LAN) handled several terabytes of uploads every day, so working with big uploads has always been a priority. The reason unicorn+nginx works well (or at all) is the attention paid to data costs: * Ruby processes are expensive, tens to hundreds of megabytes each * client connections are cheap, several kilobytes at most (including kernel data structures) * networks are slow, so most clients trickle Thus we spend more time buffering in places where things are cheap, and spend as little time holding on to big Ruby processes as possible. .. And thus intense dislike of bloated HTML, top-posting and giant signatures is directly tied to my software design philosophy. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Dealing with big uploads and/or slow clients 2014-06-06 1:27 ` Eric Wong @ 2014-06-06 2:51 ` Sam Saffron 2014-06-06 5:59 ` Eric Wong 2014-06-06 13:31 ` Bráulio Bhavamitra 1 sibling, 1 reply; 6+ messages in thread From: Sam Saffron @ 2014-06-06 2:51 UTC (permalink / raw) To: Eric Wong; +Cc: Bráulio Bhavamitra, unicorn-public Wouldn't you just use Rack Hijack for this, you can easily pull the socket out of the pipeline and then deal with the upload via eventmachine or what not. On Fri, Jun 6, 2014 at 11:27 AM, Eric Wong <e@80x24.org> wrote: > Bráulio Bhavamitra <braulio@eita.org.br> wrote: >> Hello all, > > Hello, please stop sending HTML email, my spam filters are configured to > give HTML mail high spam scores and forces moderation, so you may not > get a response for a long time. Non-English signature also tends to get > flagged as spam. This is an English list. (Fwiw, we get 10-20 spams > each day to this list which are filtered) > >> How do you deal with big uploads and/or slow clients with unicorn? The only >> solution I've found was to increase timeout, because nginx can't help too, >> or can? > > nginx handles slow clients extremely well. > > How big are you talking about and how much RAM do you have for your OS > page cache? Big uploads are OK if you have enough memory for caching in > Linux (or fast SSDs). The Linux page cache is very effective if you can > process the data fairly quickly. > > If your clients are very slow and uploads cannot fit in the Linux > page cache, lower your vm.dirty_* knobs (see kernel docs) to force > earlier write-out so you don't get a thundering herd of disk activity > when you run out of RAM. > > I don't know about caching in other OSes. > > My earliest deployments of unicorn (on a LAN) handled several terabytes > of uploads every day, so working with big uploads has always been > a priority. > > The reason unicorn+nginx works well (or at all) is the attention paid to > data costs: > > * Ruby processes are expensive, tens to hundreds of megabytes each > * client connections are cheap, several kilobytes at most > (including kernel data structures) > * networks are slow, so most clients trickle > > Thus we spend more time buffering in places where things are cheap, and > spend as little time holding on to big Ruby processes as possible. > > .. And thus intense dislike of bloated HTML, top-posting and giant > signatures is directly tied to my software design philosophy. > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Dealing with big uploads and/or slow clients 2014-06-06 2:51 ` Sam Saffron @ 2014-06-06 5:59 ` Eric Wong 0 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2014-06-06 5:59 UTC (permalink / raw) To: Sam Saffron; +Cc: Bráulio Bhavamitra, unicorn-public Sam Saffron <sam.saffron@gmail.com> wrote: > Wouldn't you just use Rack Hijack for this, you can easily pull the > socket out of the pipeline and then deal with the upload via > eventmachine or what not. Sure, but is that something that can Bráulio can drop in and use right away? I expect he's already running nginx. If he's going to be doing that, probably easier to just another server altogether. yahns[1] is pretty great for that. [1] http://yahns.yhbt.net/README (the unicorn site has been running on it since November) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Dealing with big uploads and/or slow clients 2014-06-06 1:27 ` Eric Wong 2014-06-06 2:51 ` Sam Saffron @ 2014-06-06 13:31 ` Bráulio Bhavamitra 2014-06-06 15:13 ` Michael Fischer 1 sibling, 1 reply; 6+ messages in thread From: Bráulio Bhavamitra @ 2014-06-06 13:31 UTC (permalink / raw) To: Eric Wong; +Cc: unicorn-public Hello Eric, Sorry, switched to plain text now... So the idea is that nginx will buffer the upload request until all data is received and then it will send it to unicorn? So unicorn only receives the full request? Should I do some configuration to nginx for this? In fact, we are still using apache, but are planing to switch to nginx soon... regards, bráulio On Thu, Jun 5, 2014 at 10:27 PM, Eric Wong <e@80x24.org> wrote: > Bráulio Bhavamitra <braulio@eita.org.br> wrote: >> Hello all, > > Hello, please stop sending HTML email, my spam filters are configured to > give HTML mail high spam scores and forces moderation, so you may not > get a response for a long time. Non-English signature also tends to get > flagged as spam. This is an English list. (Fwiw, we get 10-20 spams > each day to this list which are filtered) > >> How do you deal with big uploads and/or slow clients with unicorn? The only >> solution I've found was to increase timeout, because nginx can't help too, >> or can? > > nginx handles slow clients extremely well. > > How big are you talking about and how much RAM do you have for your OS > page cache? Big uploads are OK if you have enough memory for caching in > Linux (or fast SSDs). The Linux page cache is very effective if you can > process the data fairly quickly. > > If your clients are very slow and uploads cannot fit in the Linux > page cache, lower your vm.dirty_* knobs (see kernel docs) to force > earlier write-out so you don't get a thundering herd of disk activity > when you run out of RAM. > > I don't know about caching in other OSes. > > My earliest deployments of unicorn (on a LAN) handled several terabytes > of uploads every day, so working with big uploads has always been > a priority. > > The reason unicorn+nginx works well (or at all) is the attention paid to > data costs: > > * Ruby processes are expensive, tens to hundreds of megabytes each > * client connections are cheap, several kilobytes at most > (including kernel data structures) > * networks are slow, so most clients trickle > > Thus we spend more time buffering in places where things are cheap, and > spend as little time holding on to big Ruby processes as possible. > > .. And thus intense dislike of bloated HTML, top-posting and giant > signatures is directly tied to my software design philosophy. -- "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua ideologia. Morra por sua ideologia" P.R. Sarkar EITA - Educação, Informação e Tecnologias para Autogestão http://cirandas.net/brauliobo http://eita.org.br "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e destruídas nas fases de extroversão e introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a única proprietária daquilo que ela imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha por um milharal também imaginado, a pessoa imaginada não é a propriedade desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso a propriedade deste universo é de Brahma, e não dos microcosmos que também foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, mutável ou imutável, pertence a um indivíduo em particular; tudo é o patrimônio comum de todos." Restante do texto em http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Dealing with big uploads and/or slow clients 2014-06-06 13:31 ` Bráulio Bhavamitra @ 2014-06-06 15:13 ` Michael Fischer 0 siblings, 0 replies; 6+ messages in thread From: Michael Fischer @ 2014-06-06 15:13 UTC (permalink / raw) To: Bráulio Bhavamitra; +Cc: unicorn-public On Fri, Jun 6, 2014 at 6:31 AM, Bráulio Bhavamitra <braulio@eita.org.br> wrote: > So the idea is that nginx will buffer the upload request until all > data is received and then it will send it to unicorn? So unicorn only > receives the full request? Should I do some configuration to nginx for > this? Yes; there are a few tunables worth looking at. The "Buffers" section here offers some useful discussion: https://www.digitalocean.com/community/articles/how-to-optimize-nginx-configuration Best regards, --Michael ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-06-06 15:13 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-06-05 17:39 Dealing with big uploads and/or slow clients Bráulio Bhavamitra 2014-06-06 1:27 ` Eric Wong 2014-06-06 2:51 ` Sam Saffron 2014-06-06 5:59 ` Eric Wong 2014-06-06 13:31 ` Bráulio Bhavamitra 2014-06-06 15:13 ` Michael Fischer
Code repositories for project(s) associated with this public inbox https://yhbt.net/unicorn.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).