* Reserved workers not as webservers @ 2014-10-09 12:24 Bráulio Bhavamitra 2014-10-09 16:45 ` Michael Fischer 0 siblings, 1 reply; 13+ messages in thread From: Bráulio Bhavamitra @ 2014-10-09 12:24 UTC (permalink / raw) To: unicorn-public Hello all, I'm quite amazed of how preloading and fork and reduce memory usage. Making it use even less memory and restarting the app faster, the next step would be incorporate other daemons (also part of the app, in my case delayed_job and feed-updater) AS unicorn workers. This would be very interesting as restarting unicorn would also restart these daemons. Another benefit of incorporating daemons into unicorn is sharing pidfile. For it to work properly I would have to reserve some workers for these daemons. Today, I would simple do a condition on worker.nr and apply a specific code to start the daemon. Maybe better would be to have a different type of worker designed for this. But something I don't know how to do is to disable http requests to be forwared to these workers. How could I do it? Have you ever thought in doing this? best regards, bráulio ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 12:24 Reserved workers not as webservers Bráulio Bhavamitra @ 2014-10-09 16:45 ` Michael Fischer 2014-10-09 17:14 ` Devin Ben-Hur 2014-10-09 17:43 ` Bráulio Bhavamitra 0 siblings, 2 replies; 13+ messages in thread From: Michael Fischer @ 2014-10-09 16:45 UTC (permalink / raw) To: Bráulio Bhavamitra; +Cc: unicorn-public On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br> wrote: I'm quite amazed of how preloading and fork and reduce memory usage. > Making it use even less memory and restarting the app faster, the next > step would be incorporate other daemons (also part of the app, in my > case delayed_job and feed-updater) AS unicorn workers. As your friendly neighborhood service engineer, my experience is that separating the concerns (keeping asynchronous processors separate from your synchronouous web services) is worth the additional memory and processor cost. The two usually don't scale at the same rate, and you want to keep your failure domains separate as well: you don't want a bug in the async processor cause your web server to crash as well. And let's not even get into the programming and maintenance challenges. There is such as thing as being too cheap. :) --Michael ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 16:45 ` Michael Fischer @ 2014-10-09 17:14 ` Devin Ben-Hur 2014-10-09 17:17 ` Michael Fischer ` (2 more replies) 2014-10-09 17:43 ` Bráulio Bhavamitra 1 sibling, 3 replies; 13+ messages in thread From: Devin Ben-Hur @ 2014-10-09 17:14 UTC (permalink / raw) To: Michael Fischer, Bráulio Bhavamitra; +Cc: unicorn-public On 10/9/14, 9:45 AM, Michael Fischer wrote: > On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br> wrote: > > I'm quite amazed of how preloading and fork and reduce memory usage. >> Making it use even less memory and restarting the app faster, the next >> step would be incorporate other daemons (also part of the app, in my >> case delayed_job and feed-updater) AS unicorn workers. > > As your friendly neighborhood service engineer, my experience is that > separating the concerns (keeping asynchronous processors separate from your > synchronouous web services) is worth the additional memory and processor > cost. The two usually don't scale at the same rate, and you want to keep > your failure domains separate as well: you don't want a bug in the async > processor cause your web server to crash as well. And let's not even get > into the programming and maintenance challenges. Excellent points Michael. But to Bráulio's original request, it would be lovely to factor out the clean and robust process management parts of unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling) separate from the HTTP/Rack server. This would give us a robust supervisor for asynchronous workers that reduce memory footprint with CoW and responds to a well understood set of signals for management. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 17:14 ` Devin Ben-Hur @ 2014-10-09 17:17 ` Michael Fischer 2014-10-09 17:34 ` Eric Wong 2014-10-09 18:29 ` Bráulio Bhavamitra 2 siblings, 0 replies; 13+ messages in thread From: Michael Fischer @ 2014-10-09 17:17 UTC (permalink / raw) To: Devin Ben-Hur; +Cc: Bráulio Bhavamitra, unicorn-public On Thu, Oct 9, 2014 at 10:14 AM, Devin Ben-Hur <dbenhur@whitepages.com> wrote: > > Excellent points Michael. But to Bráulio's original request, it would be > lovely to factor out the clean and robust process management parts of > unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling) > separate from the HTTP/Rack server. This would give us a robust supervisor > for asynchronous workers that reduce memory footprint with CoW and responds > to a well understood set of signals for management. > Take a look at resque-pool (https://github.com/nevans/resque-pool) for one example of a pre-existing solution. I'm sure there are others. --Michael ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 17:14 ` Devin Ben-Hur 2014-10-09 17:17 ` Michael Fischer @ 2014-10-09 17:34 ` Eric Wong 2014-10-09 18:06 ` Bráulio Bhavamitra 2014-10-09 18:29 ` Bráulio Bhavamitra 2 siblings, 1 reply; 13+ messages in thread From: Eric Wong @ 2014-10-09 17:34 UTC (permalink / raw) To: Devin Ben-Hur; +Cc: Michael Fischer, Bráulio Bhavamitra, unicorn-public Devin Ben-Hur <dbenhur@whitepages.com> wrote: > Excellent points Michael. But to Bráulio's original request, it > would be lovely to factor out the clean and robust process > management parts of unicorn (daemonization, pidfile-mgmt, pre-load, > fork, reap, signaling) separate from the HTTP/Rack server. I think some projects exist nowadays (and were inspired by unicorn). I also cannot speak to the quality of them, though. My take is that stuff ends up being fairly specific to the type of app used and putting an abstraction around them makes it harder to learn and use than the lower-level primitives Ruby provides. The ordering of some things (e.g. writing pid file, preload, binding sockets, hooks, timeout checks) seems subject to the needs of the specific server; and it's easier to figure out the ordering of those things when the lower-level parts are right in front of you instead of abstracted away in a library. Having Ruby as an abstraction around C syscalls is great since I don't have to worry about things like buffer overruns or error-checking every single syscall. More abstraction than that ends up hiding too many important details, making programming and maintenance harder. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 17:34 ` Eric Wong @ 2014-10-09 18:06 ` Bráulio Bhavamitra 2014-10-09 18:15 ` Eric Wong 0 siblings, 1 reply; 13+ messages in thread From: Bráulio Bhavamitra @ 2014-10-09 18:06 UTC (permalink / raw) To: Eric Wong; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public Eric, how with a simple monkey patch will allow a worker to not connect with the kernel request queue? On Thu, Oct 9, 2014 at 2:34 PM, Eric Wong <e@80x24.org> wrote: > Devin Ben-Hur <dbenhur@whitepages.com> wrote: >> Excellent points Michael. But to Bráulio's original request, it >> would be lovely to factor out the clean and robust process >> management parts of unicorn (daemonization, pidfile-mgmt, pre-load, >> fork, reap, signaling) separate from the HTTP/Rack server. > > I think some projects exist nowadays (and were inspired by unicorn). > I also cannot speak to the quality of them, though. > > My take is that stuff ends up being fairly specific to the type of > app used and putting an abstraction around them makes it harder to learn > and use than the lower-level primitives Ruby provides. > > The ordering of some things (e.g. writing pid file, preload, binding > sockets, hooks, timeout checks) seems subject to the needs of the > specific server; and it's easier to figure out the ordering of those > things when the lower-level parts are right in front of you instead of > abstracted away in a library. > > Having Ruby as an abstraction around C syscalls is great since I don't > have to worry about things like buffer overruns or error-checking every > single syscall. More abstraction than that ends up hiding too many > important details, making programming and maintenance harder. -- "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua ideologia. Morra por sua ideologia" P.R. Sarkar EITA - Educação, Informação e Tecnologias para Autogestão http://cirandas.net/brauliobo http://eita.org.br "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e destruídas nas fases de extroversão e introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a única proprietária daquilo que ela imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha por um milharal também imaginado, a pessoa imaginada não é a propriedade desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso a propriedade deste universo é de Brahma, e não dos microcosmos que também foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, mutável ou imutável, pertence a um indivíduo em particular; tudo é o patrimônio comum de todos." Restante do texto em http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 18:06 ` Bráulio Bhavamitra @ 2014-10-09 18:15 ` Eric Wong 2014-10-11 3:35 ` Bráulio Bhavamitra 0 siblings, 1 reply; 13+ messages in thread From: Eric Wong @ 2014-10-09 18:15 UTC (permalink / raw) To: Bráulio Bhavamitra; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public Bráulio Bhavamitra <braulio@eita.org.br> wrote: > Eric, how with a simple monkey patch will allow a worker to not > connect with the kernel request queue? The listen socket is inherited by default. Closing it works. You can also keep the socket open and avoid calling any accept() wrappers, that is like a "pop" operation on the queue: kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib Also, please don't top-post (or send giant sigs). Thanks. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 18:15 ` Eric Wong @ 2014-10-11 3:35 ` Bráulio Bhavamitra 2014-10-13 0:10 ` Bráulio Bhavamitra 0 siblings, 1 reply; 13+ messages in thread From: Bráulio Bhavamitra @ 2014-10-11 3:35 UTC (permalink / raw) To: Eric Wong; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public On Thu, Oct 9, 2014 at 3:15 PM, Eric Wong <e@80x24.org> wrote: > Bráulio Bhavamitra <braulio@eita.org.br> wrote: >> Eric, how with a simple monkey patch will allow a worker to not >> connect with the kernel request queue? > > The listen socket is inherited by default. Closing it works. You can > also keep the socket open and avoid calling any accept() wrappers, that > is like a "pop" operation on the queue: > > kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib Done. If you have free time, please take a look at https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L139 > > Also, please don't top-post (or send giant sigs). Thanks. Sorry for that (again) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-11 3:35 ` Bráulio Bhavamitra @ 2014-10-13 0:10 ` Bráulio Bhavamitra 0 siblings, 0 replies; 13+ messages in thread From: Bráulio Bhavamitra @ 2014-10-13 0:10 UTC (permalink / raw) To: Eric Wong; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public I'm pretty happy to say this daemons setup is working beautifully on three production rails apps (10 workers each). It is really nice to have one pid/master process for the entire app, to know unicorn master restarts the daemons if they crash (which sometimes happens with delayed_job), and to restart the app and daemons all by once (with USR2 signal), and fast! Thanks Eric and other unicorn developers! cheers, bráulio On Sat, Oct 11, 2014 at 12:35 AM, Bráulio Bhavamitra <braulio@eita.org.br> wrote: > On Thu, Oct 9, 2014 at 3:15 PM, Eric Wong <e@80x24.org> wrote: >> Bráulio Bhavamitra <braulio@eita.org.br> wrote: >>> Eric, how with a simple monkey patch will allow a worker to not >>> connect with the kernel request queue? >> >> The listen socket is inherited by default. Closing it works. You can >> also keep the socket open and avoid calling any accept() wrappers, that >> is like a "pop" operation on the queue: >> >> kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib > Done. If you have free time, please take a look at > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L139 > >> >> Also, please don't top-post (or send giant sigs). Thanks. > Sorry for that (again) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 17:14 ` Devin Ben-Hur 2014-10-09 17:17 ` Michael Fischer 2014-10-09 17:34 ` Eric Wong @ 2014-10-09 18:29 ` Bráulio Bhavamitra 2 siblings, 0 replies; 13+ messages in thread From: Bráulio Bhavamitra @ 2014-10-09 18:29 UTC (permalink / raw) To: Devin Ben-Hur; +Cc: Michael Fischer, unicorn-public On Thu, Oct 9, 2014 at 2:14 PM, Devin Ben-Hur <dbenhur@whitepages.com> wrote: > On 10/9/14, 9:45 AM, Michael Fischer wrote: >> >> On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br> >> wrote: >> >> I'm quite amazed of how preloading and fork and reduce memory usage. >>> >>> Making it use even less memory and restarting the app faster, the next >>> step would be incorporate other daemons (also part of the app, in my >>> case delayed_job and feed-updater) AS unicorn workers. >> >> >> As your friendly neighborhood service engineer, my experience is that >> separating the concerns (keeping asynchronous processors separate from >> your >> synchronouous web services) is worth the additional memory and processor >> cost. The two usually don't scale at the same rate, and you want to keep >> your failure domains separate as well: you don't want a bug in the async >> processor cause your web server to crash as well. And let's not even get >> into the programming and maintenance challenges. > > > Excellent points Michael. But to Bráulio's original request, it would be > lovely to factor out the clean and robust process management parts of > unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling) > separate from the HTTP/Rack server. This would give us a robust supervisor > for asynchronous workers that reduce memory footprint with CoW and responds > to a well understood set of signals for management. That would be great! If unicorn's server except HTTP specific code could be extracted into a separate gem it would be fantastic :) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 16:45 ` Michael Fischer 2014-10-09 17:14 ` Devin Ben-Hur @ 2014-10-09 17:43 ` Bráulio Bhavamitra 2014-10-09 17:59 ` Michael Fischer 1 sibling, 1 reply; 13+ messages in thread From: Bráulio Bhavamitra @ 2014-10-09 17:43 UTC (permalink / raw) To: Michael Fischer; +Cc: unicorn-public Hello Michael, As being forked process, I never saw a worker problem crashing master or other workers. Also, unicorn takes care to restart workers if they die, and that is very useful for daemons to. I agree with your scaling point, but that could be solved with different unicorn configurations. cheers, bráulio On Thu, Oct 9, 2014 at 1:45 PM, Michael Fischer <mfischer@zendesk.com> wrote: > On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br> > wrote: > >> I'm quite amazed of how preloading and fork and reduce memory usage. >> Making it use even less memory and restarting the app faster, the next >> step would be incorporate other daemons (also part of the app, in my >> case delayed_job and feed-updater) AS unicorn workers. > > > As your friendly neighborhood service engineer, my experience is that > separating the concerns (keeping asynchronous processors separate from your > synchronouous web services) is worth the additional memory and processor > cost. The two usually don't scale at the same rate, and you want to keep > your failure domains separate as well: you don't want a bug in the async > processor cause your web server to crash as well. And let's not even get > into the programming and maintenance challenges. > > There is such as thing as being too cheap. :) > > --Michael > -- "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua ideologia. Morra por sua ideologia" P.R. Sarkar EITA - Educação, Informação e Tecnologias para Autogestão http://cirandas.net/brauliobo http://eita.org.br "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e destruídas nas fases de extroversão e introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a única proprietária daquilo que ela imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha por um milharal também imaginado, a pessoa imaginada não é a propriedade desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso a propriedade deste universo é de Brahma, e não dos microcosmos que também foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, mutável ou imutável, pertence a um indivíduo em particular; tudo é o patrimônio comum de todos." Restante do texto em http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 17:43 ` Bráulio Bhavamitra @ 2014-10-09 17:59 ` Michael Fischer 2014-10-09 18:01 ` Bráulio Bhavamitra 0 siblings, 1 reply; 13+ messages in thread From: Michael Fischer @ 2014-10-09 17:59 UTC (permalink / raw) To: Bráulio Bhavamitra; +Cc: unicorn-public On Thu, Oct 9, 2014 at 10:43 AM, Bráulio Bhavamitra <braulio@eita.org.br> wrote: As being forked process, I never saw a worker problem crashing master > or other workers. Process isolation is one thing, but that's not the same thing as system isolation. Imagine your worker process has a memory leak and in response, the kernel's OOM killer decides to kill your webserver children. --Michael ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Reserved workers not as webservers 2014-10-09 17:59 ` Michael Fischer @ 2014-10-09 18:01 ` Bráulio Bhavamitra 0 siblings, 0 replies; 13+ messages in thread From: Bráulio Bhavamitra @ 2014-10-09 18:01 UTC (permalink / raw) To: Michael Fischer; +Cc: unicorn-public Oh I see you are talking about the case you have many machines for a web app. The ones I admin is one VPS for all services. :) On Thu, Oct 9, 2014 at 2:59 PM, Michael Fischer <mfischer@zendesk.com> wrote: > On Thu, Oct 9, 2014 at 10:43 AM, Bráulio Bhavamitra <braulio@eita.org.br> > wrote: > >> As being forked process, I never saw a worker problem crashing master >> or other workers. > > > Process isolation is one thing, but that's not the same thing as system > isolation. Imagine your worker process has a memory leak and in response, > the kernel's OOM killer decides to kill your webserver children. > > --Michael -- "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua ideologia. Morra por sua ideologia" P.R. Sarkar EITA - Educação, Informação e Tecnologias para Autogestão http://cirandas.net/brauliobo http://eita.org.br "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e destruídas nas fases de extroversão e introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a única proprietária daquilo que ela imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha por um milharal também imaginado, a pessoa imaginada não é a propriedade desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso a propriedade deste universo é de Brahma, e não dos microcosmos que também foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, mutável ou imutável, pertence a um indivíduo em particular; tudo é o patrimônio comum de todos." Restante do texto em http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2014-10-13 0:10 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-10-09 12:24 Reserved workers not as webservers Bráulio Bhavamitra 2014-10-09 16:45 ` Michael Fischer 2014-10-09 17:14 ` Devin Ben-Hur 2014-10-09 17:17 ` Michael Fischer 2014-10-09 17:34 ` Eric Wong 2014-10-09 18:06 ` Bráulio Bhavamitra 2014-10-09 18:15 ` Eric Wong 2014-10-11 3:35 ` Bráulio Bhavamitra 2014-10-13 0:10 ` Bráulio Bhavamitra 2014-10-09 18:29 ` Bráulio Bhavamitra 2014-10-09 17:43 ` Bráulio Bhavamitra 2014-10-09 17:59 ` Michael Fischer 2014-10-09 18:01 ` Bráulio Bhavamitra
Code repositories for project(s) associated with this public inbox https://yhbt.net/unicorn.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).