unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
* Reserved workers not as webservers
@ 2014-10-09 12:24 Bráulio Bhavamitra
  2014-10-09 16:45 ` Michael Fischer
  0 siblings, 1 reply; 13+ messages in thread
From: Bráulio Bhavamitra @ 2014-10-09 12:24 UTC (permalink / raw)
  To: unicorn-public

Hello all,

I'm quite amazed of how preloading and fork and reduce memory usage.
Making it use even less memory and restarting the app faster, the next
step would be incorporate other daemons (also part of the app, in my
case delayed_job and feed-updater) AS unicorn workers.

This would be very interesting as restarting unicorn would also
restart these daemons.
Another benefit of incorporating daemons into unicorn is sharing pidfile.

For it to work properly I would have to reserve some workers for these
daemons. Today, I would simple do a condition on worker.nr and apply a
specific code to start the daemon. Maybe better would be to have a
different type of worker designed for this.

But something I don't know how to do is to disable http requests to be
forwared to these workers. How could I do it?

Have you ever thought in doing this?

best regards,
bráulio

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 12:24 Reserved workers not as webservers Bráulio Bhavamitra
@ 2014-10-09 16:45 ` Michael Fischer
  2014-10-09 17:14   ` Devin Ben-Hur
  2014-10-09 17:43   ` Bráulio Bhavamitra
  0 siblings, 2 replies; 13+ messages in thread
From: Michael Fischer @ 2014-10-09 16:45 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: unicorn-public

On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br>
wrote:

I'm quite amazed of how preloading and fork and reduce memory usage.
> Making it use even less memory and restarting the app faster, the next
> step would be incorporate other daemons (also part of the app, in my
> case delayed_job and feed-updater) AS unicorn workers.


As your friendly neighborhood service engineer, my experience is that
separating the concerns (keeping asynchronous processors separate from your
synchronouous web services) is worth the additional memory and processor
cost.  The two usually don't scale at the same rate, and you want to keep
your failure domains separate as well: you don't want a bug in the async
processor cause your web server to crash as well.   And let's not even get
into the programming and maintenance challenges.

There is such as thing as being too cheap. :)

--Michael


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 16:45 ` Michael Fischer
@ 2014-10-09 17:14   ` Devin Ben-Hur
  2014-10-09 17:17     ` Michael Fischer
                       ` (2 more replies)
  2014-10-09 17:43   ` Bráulio Bhavamitra
  1 sibling, 3 replies; 13+ messages in thread
From: Devin Ben-Hur @ 2014-10-09 17:14 UTC (permalink / raw)
  To: Michael Fischer, Bráulio Bhavamitra; +Cc: unicorn-public

On 10/9/14, 9:45 AM, Michael Fischer wrote:
> On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>
> I'm quite amazed of how preloading and fork and reduce memory usage.
>> Making it use even less memory and restarting the app faster, the next
>> step would be incorporate other daemons (also part of the app, in my
>> case delayed_job and feed-updater) AS unicorn workers.
>
> As your friendly neighborhood service engineer, my experience is that
> separating the concerns (keeping asynchronous processors separate from your
> synchronouous web services) is worth the additional memory and processor
> cost.  The two usually don't scale at the same rate, and you want to keep
> your failure domains separate as well: you don't want a bug in the async
> processor cause your web server to crash as well.   And let's not even get
> into the programming and maintenance challenges.

Excellent points Michael.  But to Bráulio's original request, it would 
be lovely to factor out the clean and robust process management parts of 
unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling) 
separate from the HTTP/Rack server.  This would give us a robust 
supervisor for asynchronous workers that reduce memory footprint with 
CoW and responds to a well understood set of signals for management.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 17:14   ` Devin Ben-Hur
@ 2014-10-09 17:17     ` Michael Fischer
  2014-10-09 17:34     ` Eric Wong
  2014-10-09 18:29     ` Bráulio Bhavamitra
  2 siblings, 0 replies; 13+ messages in thread
From: Michael Fischer @ 2014-10-09 17:17 UTC (permalink / raw)
  To: Devin Ben-Hur; +Cc: Bráulio Bhavamitra, unicorn-public

On Thu, Oct 9, 2014 at 10:14 AM, Devin Ben-Hur <dbenhur@whitepages.com>
wrote:

>
> Excellent points Michael.  But to Bráulio's original request, it would be
> lovely to factor out the clean and robust process management parts of
> unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling)
> separate from the HTTP/Rack server.  This would give us a robust supervisor
> for asynchronous workers that reduce memory footprint with CoW and responds
> to a well understood set of signals for management.
>

Take a look at resque-pool (https://github.com/nevans/resque-pool) for one
example of a pre-existing solution.  I'm sure there are others.

--Michael


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 17:14   ` Devin Ben-Hur
  2014-10-09 17:17     ` Michael Fischer
@ 2014-10-09 17:34     ` Eric Wong
  2014-10-09 18:06       ` Bráulio Bhavamitra
  2014-10-09 18:29     ` Bráulio Bhavamitra
  2 siblings, 1 reply; 13+ messages in thread
From: Eric Wong @ 2014-10-09 17:34 UTC (permalink / raw)
  To: Devin Ben-Hur; +Cc: Michael Fischer, Bráulio Bhavamitra, unicorn-public

Devin Ben-Hur <dbenhur@whitepages.com> wrote:
> Excellent points Michael.  But to Bráulio's original request, it
> would be lovely to factor out the clean and robust process
> management parts of unicorn (daemonization, pidfile-mgmt, pre-load,
> fork, reap, signaling) separate from the HTTP/Rack server.

I think some projects exist nowadays (and were inspired by unicorn).
I also cannot speak to the quality of them, though.

My take is that stuff ends up being fairly specific to the type of
app used and putting an abstraction around them makes it harder to learn
and use than the lower-level primitives Ruby provides.

The ordering of some things (e.g. writing pid file, preload, binding
sockets, hooks, timeout checks) seems subject to the needs of the
specific server; and it's easier to figure out the ordering of those
things when the lower-level parts are right in front of you instead of
abstracted away in a library.

Having Ruby as an abstraction around C syscalls is great since I don't
have to worry about things like buffer overruns or error-checking every
single syscall.  More abstraction than that ends up hiding too many
important details, making programming and maintenance harder.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 16:45 ` Michael Fischer
  2014-10-09 17:14   ` Devin Ben-Hur
@ 2014-10-09 17:43   ` Bráulio Bhavamitra
  2014-10-09 17:59     ` Michael Fischer
  1 sibling, 1 reply; 13+ messages in thread
From: Bráulio Bhavamitra @ 2014-10-09 17:43 UTC (permalink / raw)
  To: Michael Fischer; +Cc: unicorn-public

Hello Michael,

As being forked process, I never saw a worker problem crashing master
or other workers.

Also, unicorn takes care to restart workers if they die, and that is
very useful for daemons to.

I agree with your scaling point, but that could be solved with
different unicorn configurations.

cheers,
bráulio

On Thu, Oct 9, 2014 at 1:45 PM, Michael Fischer <mfischer@zendesk.com> wrote:
> On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> I'm quite amazed of how preloading and fork and reduce memory usage.
>> Making it use even less memory and restarting the app faster, the next
>> step would be incorporate other daemons (also part of the app, in my
>> case delayed_job and feed-updater) AS unicorn workers.
>
>
> As your friendly neighborhood service engineer, my experience is that
> separating the concerns (keeping asynchronous processors separate from your
> synchronouous web services) is worth the additional memory and processor
> cost.  The two usually don't scale at the same rate, and you want to keep
> your failure domains separate as well: you don't want a bug in the async
> processor cause your web server to crash as well.   And let's not even get
> into the programming and maintenance challenges.
>
> There is such as thing as being too cheap. :)
>
> --Michael
>



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 17:43   ` Bráulio Bhavamitra
@ 2014-10-09 17:59     ` Michael Fischer
  2014-10-09 18:01       ` Bráulio Bhavamitra
  0 siblings, 1 reply; 13+ messages in thread
From: Michael Fischer @ 2014-10-09 17:59 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: unicorn-public

On Thu, Oct 9, 2014 at 10:43 AM, Bráulio Bhavamitra <braulio@eita.org.br>
wrote:

As being forked process, I never saw a worker problem crashing master
> or other workers.


Process isolation is one thing, but that's not the same thing as system
isolation.  Imagine your worker process has a memory leak and in response,
the kernel's OOM killer decides to kill your webserver children.

--Michael


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 17:59     ` Michael Fischer
@ 2014-10-09 18:01       ` Bráulio Bhavamitra
  0 siblings, 0 replies; 13+ messages in thread
From: Bráulio Bhavamitra @ 2014-10-09 18:01 UTC (permalink / raw)
  To: Michael Fischer; +Cc: unicorn-public

Oh I see you are talking about the case you have many machines for a
web app. The ones I admin is one VPS for all services. :)

On Thu, Oct 9, 2014 at 2:59 PM, Michael Fischer <mfischer@zendesk.com> wrote:
> On Thu, Oct 9, 2014 at 10:43 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> As being forked process, I never saw a worker problem crashing master
>> or other workers.
>
>
> Process isolation is one thing, but that's not the same thing as system
> isolation.  Imagine your worker process has a memory leak and in response,
> the kernel's OOM killer decides to kill your webserver children.
>
> --Michael



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 17:34     ` Eric Wong
@ 2014-10-09 18:06       ` Bráulio Bhavamitra
  2014-10-09 18:15         ` Eric Wong
  0 siblings, 1 reply; 13+ messages in thread
From: Bráulio Bhavamitra @ 2014-10-09 18:06 UTC (permalink / raw)
  To: Eric Wong; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public

Eric, how with a simple monkey patch will allow a worker to not
connect with the kernel request queue?

On Thu, Oct 9, 2014 at 2:34 PM, Eric Wong <e@80x24.org> wrote:
> Devin Ben-Hur <dbenhur@whitepages.com> wrote:
>> Excellent points Michael.  But to Bráulio's original request, it
>> would be lovely to factor out the clean and robust process
>> management parts of unicorn (daemonization, pidfile-mgmt, pre-load,
>> fork, reap, signaling) separate from the HTTP/Rack server.
>
> I think some projects exist nowadays (and were inspired by unicorn).
> I also cannot speak to the quality of them, though.
>
> My take is that stuff ends up being fairly specific to the type of
> app used and putting an abstraction around them makes it harder to learn
> and use than the lower-level primitives Ruby provides.
>
> The ordering of some things (e.g. writing pid file, preload, binding
> sockets, hooks, timeout checks) seems subject to the needs of the
> specific server; and it's easier to figure out the ordering of those
> things when the lower-level parts are right in front of you instead of
> abstracted away in a library.
>
> Having Ruby as an abstraction around C syscalls is great since I don't
> have to worry about things like buffer overruns or error-checking every
> single syscall.  More abstraction than that ends up hiding too many
> important details, making programming and maintenance harder.



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 18:06       ` Bráulio Bhavamitra
@ 2014-10-09 18:15         ` Eric Wong
  2014-10-11  3:35           ` Bráulio Bhavamitra
  0 siblings, 1 reply; 13+ messages in thread
From: Eric Wong @ 2014-10-09 18:15 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public

Bráulio Bhavamitra <braulio@eita.org.br> wrote:
> Eric, how with a simple monkey patch will allow a worker to not
> connect with the kernel request queue?

The listen socket is inherited by default.  Closing it works.  You can
also keep the socket open and avoid calling any accept() wrappers, that
is like a "pop" operation on the queue:

  kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib

Also, please don't top-post (or send giant sigs).  Thanks.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 17:14   ` Devin Ben-Hur
  2014-10-09 17:17     ` Michael Fischer
  2014-10-09 17:34     ` Eric Wong
@ 2014-10-09 18:29     ` Bráulio Bhavamitra
  2 siblings, 0 replies; 13+ messages in thread
From: Bráulio Bhavamitra @ 2014-10-09 18:29 UTC (permalink / raw)
  To: Devin Ben-Hur; +Cc: Michael Fischer, unicorn-public

On Thu, Oct 9, 2014 at 2:14 PM, Devin Ben-Hur <dbenhur@whitepages.com> wrote:
> On 10/9/14, 9:45 AM, Michael Fischer wrote:
>>
>> On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br>
>> wrote:
>>
>> I'm quite amazed of how preloading and fork and reduce memory usage.
>>>
>>> Making it use even less memory and restarting the app faster, the next
>>> step would be incorporate other daemons (also part of the app, in my
>>> case delayed_job and feed-updater) AS unicorn workers.
>>
>>
>> As your friendly neighborhood service engineer, my experience is that
>> separating the concerns (keeping asynchronous processors separate from
>> your
>> synchronouous web services) is worth the additional memory and processor
>> cost.  The two usually don't scale at the same rate, and you want to keep
>> your failure domains separate as well: you don't want a bug in the async
>> processor cause your web server to crash as well.   And let's not even get
>> into the programming and maintenance challenges.
>
>
> Excellent points Michael.  But to Bráulio's original request, it would be
> lovely to factor out the clean and robust process management parts of
> unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling)
> separate from the HTTP/Rack server.  This would give us a robust supervisor
> for asynchronous workers that reduce memory footprint with CoW and responds
> to a well understood set of signals for management.

That would be great! If unicorn's server except HTTP specific code
could be extracted into a separate gem it would be fantastic :)

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-09 18:15         ` Eric Wong
@ 2014-10-11  3:35           ` Bráulio Bhavamitra
  2014-10-13  0:10             ` Bráulio Bhavamitra
  0 siblings, 1 reply; 13+ messages in thread
From: Bráulio Bhavamitra @ 2014-10-11  3:35 UTC (permalink / raw)
  To: Eric Wong; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public

On Thu, Oct 9, 2014 at 3:15 PM, Eric Wong <e@80x24.org> wrote:
> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>> Eric, how with a simple monkey patch will allow a worker to not
>> connect with the kernel request queue?
>
> The listen socket is inherited by default.  Closing it works.  You can
> also keep the socket open and avoid calling any accept() wrappers, that
> is like a "pop" operation on the queue:
>
>   kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib
Done. If you have free time, please take a look at
https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L139

>
> Also, please don't top-post (or send giant sigs).  Thanks.
Sorry for that (again)

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Reserved workers not as webservers
  2014-10-11  3:35           ` Bráulio Bhavamitra
@ 2014-10-13  0:10             ` Bráulio Bhavamitra
  0 siblings, 0 replies; 13+ messages in thread
From: Bráulio Bhavamitra @ 2014-10-13  0:10 UTC (permalink / raw)
  To: Eric Wong; +Cc: Devin Ben-Hur, Michael Fischer, unicorn-public

I'm pretty happy to say this daemons setup is working beautifully on
three production rails apps (10 workers each). It is really nice to
have one pid/master process for the entire app, to know unicorn master
restarts the daemons if they crash (which sometimes happens with
delayed_job), and to restart the app and daemons all by once (with
USR2 signal), and fast!

Thanks Eric and other unicorn developers!

cheers,
bráulio

On Sat, Oct 11, 2014 at 12:35 AM, Bráulio Bhavamitra
<braulio@eita.org.br> wrote:
> On Thu, Oct 9, 2014 at 3:15 PM, Eric Wong <e@80x24.org> wrote:
>> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>>> Eric, how with a simple monkey patch will allow a worker to not
>>> connect with the kernel request queue?
>>
>> The listen socket is inherited by default.  Closing it works.  You can
>> also keep the socket open and avoid calling any accept() wrappers, that
>> is like a "pop" operation on the queue:
>>
>>   kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib
> Done. If you have free time, please take a look at
> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L139
>
>>
>> Also, please don't top-post (or send giant sigs).  Thanks.
> Sorry for that (again)

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2014-10-13  0:10 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-10-09 12:24 Reserved workers not as webservers Bráulio Bhavamitra
2014-10-09 16:45 ` Michael Fischer
2014-10-09 17:14   ` Devin Ben-Hur
2014-10-09 17:17     ` Michael Fischer
2014-10-09 17:34     ` Eric Wong
2014-10-09 18:06       ` Bráulio Bhavamitra
2014-10-09 18:15         ` Eric Wong
2014-10-11  3:35           ` Bráulio Bhavamitra
2014-10-13  0:10             ` Bráulio Bhavamitra
2014-10-09 18:29     ` Bráulio Bhavamitra
2014-10-09 17:43   ` Bráulio Bhavamitra
2014-10-09 17:59     ` Michael Fischer
2014-10-09 18:01       ` Bráulio Bhavamitra

Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).