unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: "Bráulio Bhavamitra" <braulio@eita.org.br>
To: Sarkis Varozian <svarozian@gmail.com>
Cc: Eric Wong <e@80x24.org>, Michael Fischer <mfischer@zendesk.com>,
	 unicorn-public <unicorn-public@bogomips.org>
Subject: Re: Request Queueing after deploy + USR2 restart
Date: Thu, 5 Mar 2015 14:13:12 -0300	[thread overview]
Message-ID: <CAJri6_twFUFzJ3vFtCysy59apsgf+m_sMfGo39h=Jf36By50Dw@mail.gmail.com> (raw)
In-Reply-To: <CAGchx-LV6q3Ku+3akpdmwVbWozJ2C3651616+PC5OXD-TUd5YQ@mail.gmail.com>

In the graphs you posted, what is the grey part? It is not described in the
legend and it seems the problem is entirely there. What reverse proxy are
you using?

Can you reproduce this with a single master instance?

Could you try this sleep:
https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91


On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <svarozian@gmail.com> wrote:

> Hey All,
>
> So I changed up my unicorn.rb a bit from my original post:
> https://gist.github.com/sarkis/1aa296044b1dfd3695ab
>
> I'm also still sending the USR2 signals on deploy staggered with 30 second
> delay via capistrano:
>
> on roles(:web), in: :sequence, wait: 30
>
> As you can see I am now doing a warup via rack MockRequest (I hoped this
> would warmup the master). However, this is what a deploy looks like on
> newrelic:
>
>
> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0
>
>
> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0
>
> I'm running out of ideas to get rid of thse latency spikes. Would you guys
> recommend I try anything else at this point?
>
>
>
> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <svarozian@gmail.com>
> wrote:
>
>> Eric,
>>
>> Thanks for the quick reply.
>>
>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is the
>> lazy loading - at least thats what all signs point to. I am going to try
>> and mock request some url endpoints. Currently, I can only think of '/', as
>> most other parts of the app require a session and auth. I'll report back
>> with results.
>>
>>
>>
>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
>>
>>> Sarkis Varozian <svarozian@gmail.com> wrote:
>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <mfischer@zendesk.com
>>> >
>>> > wrote:
>>> >
>>> > > I'm not exactly sure how preload_app works, but I suspect your app is
>>> > > lazy-loading a number of Ruby libraries while handling the first few
>>> > > requests that weren't automatically loaded during the preload
>>> process.
>>> > >
>>> > > Eric, your thoughts?
>>>
>>> (top-posting corrected)
>>>
>>> Yeah, preload_app won't help startup speed if much of the app is
>>> autoloaded.
>>>
>>> Sarkis: which Ruby version are you running?  IIRC, 1.9.2 had terrible
>>> startup performance compared to 1.9.3 and later in case you're stuck on
>>> 1.9.2
>>>
>>> > That does make sense - I was looking at another suggestion from a user
>>> here
>>> > (Braulio) of running a "warmup" using rack MockRequest:
>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>>> >
>>> > The only issue I am having with the above solution is it is happening
>>> in
>>> > the before_fork block - shouldn't I warmup the connection in
>>> after_fork?
>>>
>>> If preload_app is true, you can warmup in before_fork; otherwise it
>>> needs to be after_fork.
>>>
>>> > If
>>> > I follow the above gist properly it warms up the server with the old
>>> > activerecord base connection and then its turned off, then turned back
>>> on
>>> > in after_fork. I think I am not understanding the sequence of events
>>> > there...
>>>
>>> With preload_app and warmup, you need to ensure any stream connections
>>> (DB, memcached, redis, etc..) do not get shared between processes, so
>>> it's standard practice to disconnect in the parent and reconnect in the
>>> child.
>>>
>>> > If this is the case, I should warmup and also check/kill the old
>>> > master in the after_fork block after the new db, redis, neo4j
>>> connections
>>> > are all created. Thoughts?
>>>
>>> I've been leaving killing the master outside of the unicorn hooks
>>> and doing it as a separate step; seemed too fragile to do it in
>>> hooks from my perspective.
>>>
>>
>>
>>
>> --
>> *Sarkis Varozian*
>> svarozian@gmail.com
>>
>
>
>
> --
> *Sarkis Varozian*
> svarozian@gmail.com
>



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia


  reply	other threads:[~2015-03-05 17:13 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-03-03 22:24 Sarkis Varozian
2015-03-03 22:32 ` Michael Fischer
2015-03-04 19:48   ` Sarkis Varozian
2015-03-04 19:51     ` Michael Fischer
2015-03-04 19:58       ` Sarkis Varozian
2015-03-04 20:17         ` Michael Fischer
2015-03-04 20:24           ` Sarkis Varozian
2015-03-04 20:27             ` Michael Fischer
2015-03-04 20:35             ` Eric Wong
2015-03-04 20:40               ` Sarkis Varozian
2015-03-05 17:07                 ` Sarkis Varozian
2015-03-05 17:13                   ` Bráulio Bhavamitra [this message]
2015-03-05 17:28                     ` Sarkis Varozian
2015-03-05 17:31                       ` Bráulio Bhavamitra
2015-03-05 17:32                       ` Bráulio Bhavamitra
2015-03-05 21:12                       ` Eric Wong
2015-03-03 22:47 ` Bráulio Bhavamitra
2015-03-04 19:50   ` Sarkis Varozian
     [not found] ` <CAJri6_vidE15Xor4THzQB3uxyqPdApxHoyWp47NAG8m8TQuw0Q@mail.gmail.com>
2015-09-13 15:12   ` Bráulio Bhavamitra
2015-09-14  2:14     ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJri6_twFUFzJ3vFtCysy59apsgf+m_sMfGo39h=Jf36By50Dw@mail.gmail.com' \
    --to=braulio@eita.org.br \
    --cc=e@80x24.org \
    --cc=mfischer@zendesk.com \
    --cc=svarozian@gmail.com \
    --cc=unicorn-public@bogomips.org \
    --subject='Re: Request Queueing after deploy + USR2 restart' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	../../unicorn.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).