From: Sarkis Varozian <email@example.com> To: "Bráulio Bhavamitra" <firstname.lastname@example.org> Cc: Eric Wong <email@example.com>, Michael Fischer <firstname.lastname@example.org>, unicorn-public <email@example.com> Subject: Re: Request Queueing after deploy + USR2 restart Date: Thu, 5 Mar 2015 09:28:40 -0800 [thread overview] Message-ID: <CAGchx-JeYkVjKh2_B7a7oGSb_PDf2i=6KjwrhUXG_ckcrJEDeA@mail.gmail.com> (raw) In-Reply-To: <CAJri6_twFUFzJ3vFtCysy59apsgf+m_sMfGo39h=Jf36By50Dw@mail.gmail.com> Braulio, Are you referring to the vertical grey line? That is the deployment event. The part that spikes in the first graph is request queue which is a bit different on newrelic: http://blog.newrelic.com/2013/01/22/understanding-new-relic-queuing/ We are using HAProxy to load balance (round robin) to 4 physical hosts running unicorn with 6 workers. I have not tried to reproduce this on 1 master - I assume this would be the same. I do in fact do the sleep now: https://gist.github.com/sarkis/1aa296044b1dfd3695ab#file-unicorn-rb-L37 - the deployment results above had the 1 second sleep in there. On Thu, Mar 5, 2015 at 9:13 AM, Bráulio Bhavamitra <firstname.lastname@example.org> wrote: > In the graphs you posted, what is the grey part? It is not described in > the legend and it seems the problem is entirely there. What reverse proxy > are you using? > > Can you reproduce this with a single master instance? > > Could you try this sleep: > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91 > > > On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <email@example.com> > wrote: > >> Hey All, >> >> So I changed up my unicorn.rb a bit from my original post: >> https://gist.github.com/sarkis/1aa296044b1dfd3695ab >> >> I'm also still sending the USR2 signals on deploy staggered with 30 >> second delay via capistrano: >> >> on roles(:web), in: :sequence, wait: 30 >> >> As you can see I am now doing a warup via rack MockRequest (I hoped this >> would warmup the master). However, this is what a deploy looks like on >> newrelic: >> >> >> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0 >> >> >> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0 >> >> I'm running out of ideas to get rid of thse latency spikes. Would you >> guys recommend I try anything else at this point? >> >> >> >> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <firstname.lastname@example.org> >> wrote: >> >>> Eric, >>> >>> Thanks for the quick reply. >>> >>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is the >>> lazy loading - at least thats what all signs point to. I am going to try >>> and mock request some url endpoints. Currently, I can only think of '/', as >>> most other parts of the app require a session and auth. I'll report back >>> with results. >>> >>> >>> >>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <email@example.com> wrote: >>> >>>> Sarkis Varozian <firstname.lastname@example.org> wrote: >>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer < >>>> email@example.com> >>>> > wrote: >>>> > >>>> > > I'm not exactly sure how preload_app works, but I suspect your app >>>> is >>>> > > lazy-loading a number of Ruby libraries while handling the first few >>>> > > requests that weren't automatically loaded during the preload >>>> process. >>>> > > >>>> > > Eric, your thoughts? >>>> >>>> (top-posting corrected) >>>> >>>> Yeah, preload_app won't help startup speed if much of the app is >>>> autoloaded. >>>> >>>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible >>>> startup performance compared to 1.9.3 and later in case you're stuck on >>>> 1.9.2 >>>> >>>> > That does make sense - I was looking at another suggestion from a >>>> user here >>>> > (Braulio) of running a "warmup" using rack MockRequest: >>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77 >>>> > >>>> > The only issue I am having with the above solution is it is happening >>>> in >>>> > the before_fork block - shouldn't I warmup the connection in >>>> after_fork? >>>> >>>> If preload_app is true, you can warmup in before_fork; otherwise it >>>> needs to be after_fork. >>>> >>>> > If >>>> > I follow the above gist properly it warms up the server with the old >>>> > activerecord base connection and then its turned off, then turned >>>> back on >>>> > in after_fork. I think I am not understanding the sequence of events >>>> > there... >>>> >>>> With preload_app and warmup, you need to ensure any stream connections >>>> (DB, memcached, redis, etc..) do not get shared between processes, so >>>> it's standard practice to disconnect in the parent and reconnect in the >>>> child. >>>> >>>> > If this is the case, I should warmup and also check/kill the old >>>> > master in the after_fork block after the new db, redis, neo4j >>>> connections >>>> > are all created. Thoughts? >>>> >>>> I've been leaving killing the master outside of the unicorn hooks >>>> and doing it as a separate step; seemed too fragile to do it in >>>> hooks from my perspective. >>>> >>> >>> >>> >>> -- >>> *Sarkis Varozian* >>> firstname.lastname@example.org >>> >> >> >> >> -- >> *Sarkis Varozian* >> email@example.com >> > > > > -- > "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua > ideologia. Morra por sua ideologia" P.R. Sarkar > > EITA - Educação, Informação e Tecnologias para Autogestão > http://cirandas.net/brauliobo > http://eita.org.br > > "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu > lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da > Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e > destruídas nas fases de extroversão e introversão do fluxo imaginativo > cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente, > naquele momento, essa pessoa é a única proprietária daquilo que ela > imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha > por um milharal também imaginado, a pessoa imaginada não é a propriedade > desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este > universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso > a propriedade deste universo é de Brahma, e não dos microcosmos que também > foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo, > mutável ou imutável, pertence a um indivíduo em particular; tudo é o > patrimônio comum de todos." > Restante do texto em > http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia > -- *Sarkis Varozian* firstname.lastname@example.org
next prev parent reply other threads:[~2015-03-05 17:28 UTC|newest] Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-03-03 22:24 Sarkis Varozian 2015-03-03 22:32 ` Michael Fischer 2015-03-04 19:48 ` Sarkis Varozian 2015-03-04 19:51 ` Michael Fischer 2015-03-04 19:58 ` Sarkis Varozian 2015-03-04 20:17 ` Michael Fischer 2015-03-04 20:24 ` Sarkis Varozian 2015-03-04 20:27 ` Michael Fischer 2015-03-04 20:35 ` Eric Wong 2015-03-04 20:40 ` Sarkis Varozian 2015-03-05 17:07 ` Sarkis Varozian 2015-03-05 17:13 ` Bráulio Bhavamitra 2015-03-05 17:28 ` Sarkis Varozian [this message] 2015-03-05 17:31 ` Bráulio Bhavamitra 2015-03-05 17:32 ` Bráulio Bhavamitra 2015-03-05 21:12 ` Eric Wong 2015-03-03 22:47 ` Bráulio Bhavamitra 2015-03-04 19:50 ` Sarkis Varozian [not found] ` <CAJri6_vidE15Xor4THzQB3uxyqPdApxHoyWp47NAG8m8TQuw0Q@mail.gmail.com> 2015-09-13 15:12 ` Bráulio Bhavamitra 2015-09-14 2:14 ` Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://yhbt.net/unicorn/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CAGchx-JeYkVjKh2_B7a7oGSb_PDf2i=6KjwrhUXG_ckcrJEDeA@mail.gmail.com' \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: Request Queueing after deploy + USR2 restart' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: ../../unicorn.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).