Reserved workers not as webservers

Reserved workers not as webservers

[Bráulio Bhavamitra]

Hello all,

I'm quite amazed of how preloading and fork and reduce memory usage.
Making it use even less memory and restarting the app faster, the next
step would be incorporate other daemons (also part of the app, in my
case delayed_job and feed-updater) AS unicorn workers.

This would be very interesting as restarting unicorn would also
restart these daemons.
Another benefit of incorporating daemons into unicorn is sharing pidfile.

For it to work properly I would have to reserve some workers for these
daemons. Today, I would simple do a condition on worker.nr and apply a
specific code to start the daemon. Maybe better would be to have a
different type of worker designed for this.

But something I don't know how to do is to disable http requests to be
forwared to these workers. How could I do it?

Have you ever thought in doing this?

best regards,
bráulio

Re: Reserved workers not as webservers

[Michael Fischer]

On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br>
wrote:

I'm quite amazed of how preloading and fork and reduce memory usage.
> Making it use even less memory and restarting the app faster, the next
> step would be incorporate other daemons (also part of the app, in my
> case delayed_job and feed-updater) AS unicorn workers.


As your friendly neighborhood service engineer, my experience is that
separating the concerns (keeping asynchronous processors separate from your
synchronouous web services) is worth the additional memory and processor
cost.  The two usually don't scale at the same rate, and you want to keep
your failure domains separate as well: you don't want a bug in the async
processor cause your web server to crash as well.   And let's not even get
into the programming and maintenance challenges.

There is such as thing as being too cheap. :)

--Michael

Re: Reserved workers not as webservers

[Devin Ben-Hur]

On 10/9/14, 9:45 AM, Michael Fischer wrote:
> On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>
> I'm quite amazed of how preloading and fork and reduce memory usage.
>> Making it use even less memory and restarting the app faster, the next
>> step would be incorporate other daemons (also part of the app, in my
>> case delayed_job and feed-updater) AS unicorn workers.
>
> As your friendly neighborhood service engineer, my experience is that
> separating the concerns (keeping asynchronous processors separate from your
> synchronouous web services) is worth the additional memory and processor
> cost.  The two usually don't scale at the same rate, and you want to keep
> your failure domains separate as well: you don't want a bug in the async
> processor cause your web server to crash as well.   And let's not even get
> into the programming and maintenance challenges.

Excellent points Michael.  But to Bráulio's original request, it would 
be lovely to factor out the clean and robust process management parts of 
unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling) 
separate from the HTTP/Rack server.  This would give us a robust 
supervisor for asynchronous workers that reduce memory footprint with 
CoW and responds to a well understood set of signals for management.

Re: Reserved workers not as webservers

[Michael Fischer]

On Thu, Oct 9, 2014 at 10:14 AM, Devin Ben-Hur <dbenhur@whitepages.com>
wrote:

>
> Excellent points Michael.  But to Bráulio's original request, it would be
> lovely to factor out the clean and robust process management parts of
> unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling)
> separate from the HTTP/Rack server.  This would give us a robust supervisor
> for asynchronous workers that reduce memory footprint with CoW and responds
> to a well understood set of signals for management.
>

Take a look at resque-pool (https://github.com/nevans/resque-pool) for one
example of a pre-existing solution.  I'm sure there are others.

--Michael

Re: Reserved workers not as webservers

[Eric Wong]

Devin Ben-Hur <dbenhur@whitepages.com> wrote:
> Excellent points Michael.  But to Bráulio's original request, it
> would be lovely to factor out the clean and robust process
> management parts of unicorn (daemonization, pidfile-mgmt, pre-load,
> fork, reap, signaling) separate from the HTTP/Rack server.

I think some projects exist nowadays (and were inspired by unicorn).
I also cannot speak to the quality of them, though.

My take is that stuff ends up being fairly specific to the type of
app used and putting an abstraction around them makes it harder to learn
and use than the lower-level primitives Ruby provides.

The ordering of some things (e.g. writing pid file, preload, binding
sockets, hooks, timeout checks) seems subject to the needs of the
specific server; and it's easier to figure out the ordering of those
things when the lower-level parts are right in front of you instead of
abstracted away in a library.

Having Ruby as an abstraction around C syscalls is great since I don't
have to worry about things like buffer overruns or error-checking every
single syscall.  More abstraction than that ends up hiding too many
important details, making programming and maintenance harder.

Re: Reserved workers not as webservers

[Bráulio Bhavamitra]

Hello Michael,

As being forked process, I never saw a worker problem crashing master
or other workers.

Also, unicorn takes care to restart workers if they die, and that is
very useful for daemons to.

I agree with your scaling point, but that could be solved with
different unicorn configurations.

cheers,
bráulio

On Thu, Oct 9, 2014 at 1:45 PM, Michael Fischer <mfischer@zendesk.com> wrote:
> On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> I'm quite amazed of how preloading and fork and reduce memory usage.
>> Making it use even less memory and restarting the app faster, the next
>> step would be incorporate other daemons (also part of the app, in my
>> case delayed_job and feed-updater) AS unicorn workers.
>
>
> As your friendly neighborhood service engineer, my experience is that
> separating the concerns (keeping asynchronous processors separate from your
> synchronouous web services) is worth the additional memory and processor
> cost.  The two usually don't scale at the same rate, and you want to keep
> your failure domains separate as well: you don't want a bug in the async
> processor cause your web server to crash as well.   And let's not even get
> into the programming and maintenance challenges.
>
> There is such as thing as being too cheap. :)
>
> --Michael
>



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

Re: Reserved workers not as webservers

[Michael Fischer]

On Thu, Oct 9, 2014 at 10:43 AM, Bráulio Bhavamitra <braulio@eita.org.br>
wrote:

As being forked process, I never saw a worker problem crashing master
> or other workers.


Process isolation is one thing, but that's not the same thing as system
isolation.  Imagine your worker process has a memory leak and in response,
the kernel's OOM killer decides to kill your webserver children.

--Michael

Re: Reserved workers not as webservers

[Bráulio Bhavamitra]

Oh I see you are talking about the case you have many machines for a
web app. The ones I admin is one VPS for all services. :)

On Thu, Oct 9, 2014 at 2:59 PM, Michael Fischer <mfischer@zendesk.com> wrote:
> On Thu, Oct 9, 2014 at 10:43 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> As being forked process, I never saw a worker problem crashing master
>> or other workers.
>
>
> Process isolation is one thing, but that's not the same thing as system
> isolation.  Imagine your worker process has a memory leak and in response,
> the kernel's OOM killer decides to kill your webserver children.
>
> --Michael



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

Re: Reserved workers not as webservers

[Bráulio Bhavamitra]

Eric, how with a simple monkey patch will allow a worker to not
connect with the kernel request queue?

On Thu, Oct 9, 2014 at 2:34 PM, Eric Wong <e@80x24.org> wrote:
> Devin Ben-Hur <dbenhur@whitepages.com> wrote:
>> Excellent points Michael.  But to Bráulio's original request, it
>> would be lovely to factor out the clean and robust process
>> management parts of unicorn (daemonization, pidfile-mgmt, pre-load,
>> fork, reap, signaling) separate from the HTTP/Rack server.
>
> I think some projects exist nowadays (and were inspired by unicorn).
> I also cannot speak to the quality of them, though.
>
> My take is that stuff ends up being fairly specific to the type of
> app used and putting an abstraction around them makes it harder to learn
> and use than the lower-level primitives Ruby provides.
>
> The ordering of some things (e.g. writing pid file, preload, binding
> sockets, hooks, timeout checks) seems subject to the needs of the
> specific server; and it's easier to figure out the ordering of those
> things when the lower-level parts are right in front of you instead of
> abstracted away in a library.
>
> Having Ruby as an abstraction around C syscalls is great since I don't
> have to worry about things like buffer overruns or error-checking every
> single syscall.  More abstraction than that ends up hiding too many
> important details, making programming and maintenance harder.



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

Re: Reserved workers not as webservers

[Eric Wong]

Bráulio Bhavamitra <braulio@eita.org.br> wrote:
> Eric, how with a simple monkey patch will allow a worker to not
> connect with the kernel request queue?

The listen socket is inherited by default.  Closing it works.  You can
also keep the socket open and avoid calling any accept() wrappers, that
is like a "pop" operation on the queue:

  kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib

Also, please don't top-post (or send giant sigs).  Thanks.

Re: Reserved workers not as webservers

[Bráulio Bhavamitra]

On Thu, Oct 9, 2014 at 2:14 PM, Devin Ben-Hur <dbenhur@whitepages.com> wrote:
> On 10/9/14, 9:45 AM, Michael Fischer wrote:
>>
>> On Thu, Oct 9, 2014 at 5:24 AM, Bráulio Bhavamitra <braulio@eita.org.br>
>> wrote:
>>
>> I'm quite amazed of how preloading and fork and reduce memory usage.
>>>
>>> Making it use even less memory and restarting the app faster, the next
>>> step would be incorporate other daemons (also part of the app, in my
>>> case delayed_job and feed-updater) AS unicorn workers.
>>
>>
>> As your friendly neighborhood service engineer, my experience is that
>> separating the concerns (keeping asynchronous processors separate from
>> your
>> synchronouous web services) is worth the additional memory and processor
>> cost.  The two usually don't scale at the same rate, and you want to keep
>> your failure domains separate as well: you don't want a bug in the async
>> processor cause your web server to crash as well.   And let's not even get
>> into the programming and maintenance challenges.
>
>
> Excellent points Michael.  But to Bráulio's original request, it would be
> lovely to factor out the clean and robust process management parts of
> unicorn (daemonization, pidfile-mgmt, pre-load, fork, reap, signaling)
> separate from the HTTP/Rack server.  This would give us a robust supervisor
> for asynchronous workers that reduce memory footprint with CoW and responds
> to a well understood set of signals for management.

That would be great! If unicorn's server except HTTP specific code
could be extracted into a separate gem it would be fantastic :)

Re: Reserved workers not as webservers

[Bráulio Bhavamitra]

On Thu, Oct 9, 2014 at 3:15 PM, Eric Wong <e@80x24.org> wrote:
> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>> Eric, how with a simple monkey patch will allow a worker to not
>> connect with the kernel request queue?
>
> The listen socket is inherited by default.  Closing it works.  You can
> also keep the socket open and avoid calling any accept() wrappers, that
> is like a "pop" operation on the queue:
>
>   kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib
Done. If you have free time, please take a look at
https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L139

>
> Also, please don't top-post (or send giant sigs).  Thanks.
Sorry for that (again)

Re: Reserved workers not as webservers

[Bráulio Bhavamitra]

I'm pretty happy to say this daemons setup is working beautifully on
three production rails apps (10 workers each). It is really nice to
have one pid/master process for the entire app, to know unicorn master
restarts the daemons if they crash (which sometimes happens with
delayed_job), and to restart the app and daemons all by once (with
USR2 signal), and fast!

Thanks Eric and other unicorn developers!

cheers,
bráulio

On Sat, Oct 11, 2014 at 12:35 AM, Bráulio Bhavamitra
<braulio@eita.org.br> wrote:
> On Thu, Oct 9, 2014 at 3:15 PM, Eric Wong <e@80x24.org> wrote:
>> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>>> Eric, how with a simple monkey patch will allow a worker to not
>>> connect with the kernel request queue?
>>
>> The listen socket is inherited by default.  Closing it works.  You can
>> also keep the socket open and avoid calling any accept() wrappers, that
>> is like a "pop" operation on the queue:
>>
>>   kgio_tryaccept in unicorn, accept/accept_nonblock/sysaccept in stdlib
> Done. If you have free time, please take a look at
> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L139
>
>>
>> Also, please don't top-post (or send giant sigs).  Thanks.
> Sorry for that (again)