unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: "Bráulio Bhavamitra" <braulio@eita.org.br>
To: Sarkis Varozian <svarozian@gmail.com>
Cc: Eric Wong <e@80x24.org>, Michael Fischer <mfischer@zendesk.com>,
	 unicorn-public <unicorn-public@bogomips.org>
Subject: Re: Request Queueing after deploy + USR2 restart
Date: Thu, 5 Mar 2015 14:31:47 -0300	[thread overview]
Message-ID: <CAJri6_svnT-XF546uUc25zRwSOWr8ZtdLAh7T1gHWLp9SC4_SQ@mail.gmail.com> (raw)
In-Reply-To: <CAGchx-JeYkVjKh2_B7a7oGSb_PDf2i=6KjwrhUXG_ckcrJEDeA@mail.gmail.com>

I would try to reproduce this locally with a production env and `ab -n 200
-c 2 http://localhost:3000/`



On Thu, Mar 5, 2015 at 2:28 PM, Sarkis Varozian <svarozian@gmail.com> wrote:

> Braulio,
>
> Are you referring to the vertical grey line? That is the deployment event.
> The part that spikes in the first graph is request queue which is a bit
> different on newrelic:
> http://blog.newrelic.com/2013/01/22/understanding-new-relic-queuing/
>
> We are using HAProxy to load balance (round robin) to 4 physical hosts
> running unicorn with 6 workers.
>
> I have not tried to reproduce this on 1 master - I assume this would be
> the same.
>
> I do in fact do the sleep now:
> https://gist.github.com/sarkis/1aa296044b1dfd3695ab#file-unicorn-rb-L37 -
> the deployment results above had the 1 second sleep in there.
>
> On Thu, Mar 5, 2015 at 9:13 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> In the graphs you posted, what is the grey part? It is not described in
>> the legend and it seems the problem is entirely there. What reverse proxy
>> are you using?
>>
>> Can you reproduce this with a single master instance?
>>
>> Could you try this sleep:
>> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91
>>
>>
>> On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <svarozian@gmail.com>
>> wrote:
>>
>>> Hey All,
>>>
>>> So I changed up my unicorn.rb a bit from my original post:
>>> https://gist.github.com/sarkis/1aa296044b1dfd3695ab
>>>
>>> I'm also still sending the USR2 signals on deploy staggered with 30
>>> second delay via capistrano:
>>>
>>> on roles(:web), in: :sequence, wait: 30
>>>
>>> As you can see I am now doing a warup via rack MockRequest (I hoped this
>>> would warmup the master). However, this is what a deploy looks like on
>>> newrelic:
>>>
>>>
>>> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0
>>>
>>>
>>> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0
>>>
>>> I'm running out of ideas to get rid of thse latency spikes. Would you
>>> guys recommend I try anything else at this point?
>>>
>>>
>>>
>>> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <svarozian@gmail.com>
>>> wrote:
>>>
>>>> Eric,
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is
>>>> the lazy loading - at least thats what all signs point to. I am going to
>>>> try and mock request some url endpoints. Currently, I can only think of
>>>> '/', as most other parts of the app require a session and auth. I'll report
>>>> back with results.
>>>>
>>>>
>>>>
>>>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
>>>>
>>>>> Sarkis Varozian <svarozian@gmail.com> wrote:
>>>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <
>>>>> mfischer@zendesk.com>
>>>>> > wrote:
>>>>> >
>>>>> > > I'm not exactly sure how preload_app works, but I suspect your app
>>>>> is
>>>>> > > lazy-loading a number of Ruby libraries while handling the first
>>>>> few
>>>>> > > requests that weren't automatically loaded during the preload
>>>>> process.
>>>>> > >
>>>>> > > Eric, your thoughts?
>>>>>
>>>>> (top-posting corrected)
>>>>>
>>>>> Yeah, preload_app won't help startup speed if much of the app is
>>>>> autoloaded.
>>>>>
>>>>> Sarkis: which Ruby version are you running?  IIRC, 1.9.2 had terrible
>>>>> startup performance compared to 1.9.3 and later in case you're stuck on
>>>>> 1.9.2
>>>>>
>>>>> > That does make sense - I was looking at another suggestion from a
>>>>> user here
>>>>> > (Braulio) of running a "warmup" using rack MockRequest:
>>>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>>>>> >
>>>>> > The only issue I am having with the above solution is it is
>>>>> happening in
>>>>> > the before_fork block - shouldn't I warmup the connection in
>>>>> after_fork?
>>>>>
>>>>> If preload_app is true, you can warmup in before_fork; otherwise it
>>>>> needs to be after_fork.
>>>>>
>>>>> > If
>>>>> > I follow the above gist properly it warms up the server with the old
>>>>> > activerecord base connection and then its turned off, then turned
>>>>> back on
>>>>> > in after_fork. I think I am not understanding the sequence of events
>>>>> > there...
>>>>>
>>>>> With preload_app and warmup, you need to ensure any stream connections
>>>>> (DB, memcached, redis, etc..) do not get shared between processes, so
>>>>> it's standard practice to disconnect in the parent and reconnect in the
>>>>> child.
>>>>>
>>>>> > If this is the case, I should warmup and also check/kill the old
>>>>> > master in the after_fork block after the new db, redis, neo4j
>>>>> connections
>>>>> > are all created. Thoughts?
>>>>>
>>>>> I've been leaving killing the master outside of the unicorn hooks
>>>>> and doing it as a separate step; seemed too fragile to do it in
>>>>> hooks from my perspective.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Sarkis Varozian*
>>>> svarozian@gmail.com
>>>>
>>>
>>>
>>>
>>> --
>>> *Sarkis Varozian*
>>> svarozian@gmail.com
>>>
>>
>>
>>
>> --
>> "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
>> ideologia. Morra por sua ideologia" P.R. Sarkar
>>
>> EITA - Educação, Informação e Tecnologias para Autogestão
>> http://cirandas.net/brauliobo
>> http://eita.org.br
>>
>> "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
>> lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
>> Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
>> destruídas nas fases de extroversão e introversão do fluxo imaginativo
>> cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
>> naquele momento, essa pessoa é a única proprietária daquilo que ela
>> imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
>> por um milharal também imaginado, a pessoa imaginada não é a propriedade
>> desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
>> universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
>> a propriedade deste universo é de Brahma, e não dos microcosmos que também
>> foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
>> mutável ou imutável, pertence a um indivíduo em particular; tudo é o
>> patrimônio comum de todos."
>> Restante do texto em
>> http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
>>
>
>
>
> --
> *Sarkis Varozian*
> svarozian@gmail.com
>



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia


  reply	other threads:[~2015-03-05 17:32 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-03-03 22:24 Sarkis Varozian
2015-03-03 22:32 ` Michael Fischer
2015-03-04 19:48   ` Sarkis Varozian
2015-03-04 19:51     ` Michael Fischer
2015-03-04 19:58       ` Sarkis Varozian
2015-03-04 20:17         ` Michael Fischer
2015-03-04 20:24           ` Sarkis Varozian
2015-03-04 20:27             ` Michael Fischer
2015-03-04 20:35             ` Eric Wong
2015-03-04 20:40               ` Sarkis Varozian
2015-03-05 17:07                 ` Sarkis Varozian
2015-03-05 17:13                   ` Bráulio Bhavamitra
2015-03-05 17:28                     ` Sarkis Varozian
2015-03-05 17:31                       ` Bráulio Bhavamitra [this message]
2015-03-05 17:32                       ` Bráulio Bhavamitra
2015-03-05 21:12                       ` Eric Wong
2015-03-03 22:47 ` Bráulio Bhavamitra
2015-03-04 19:50   ` Sarkis Varozian
     [not found] ` <CAJri6_vidE15Xor4THzQB3uxyqPdApxHoyWp47NAG8m8TQuw0Q@mail.gmail.com>
2015-09-13 15:12   ` Bráulio Bhavamitra
2015-09-14  2:14     ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAJri6_svnT-XF546uUc25zRwSOWr8ZtdLAh7T1gHWLp9SC4_SQ@mail.gmail.com \
    --to=braulio@eita.org.br \
    --cc=e@80x24.org \
    --cc=mfischer@zendesk.com \
    --cc=svarozian@gmail.com \
    --cc=unicorn-public@bogomips.org \
    --subject='Re: Request Queueing after deploy + USR2 restart' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	../../unicorn.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).