From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS15169 209.85.128.0/17 X-Spam-Status: No, score=-0.7 required=3.0 tests=AWL,BAYES_00,FREEMAIL_FROM, URIBL_BLOCKED,URIBL_DBL_ABUSE_REDIR shortcircuit=no autolearn=ham version=3.3.2 X-Original-To: unicorn-public@bogomips.org Received: from mail-ie0-f176.google.com (mail-ie0-f176.google.com [209.85.223.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 1305C1F49F for ; Wed, 4 Mar 2015 19:48:17 +0000 (UTC) Received: by iecar1 with SMTP id ar1so70138836iec.11 for ; Wed, 04 Mar 2015 11:48:06 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.107.15.68 with SMTP id x65mr14681380ioi.39.1425498486332; Wed, 04 Mar 2015 11:48:06 -0800 (PST) Received: by 10.107.43.73 with HTTP; Wed, 4 Mar 2015 11:48:06 -0800 (PST) In-Reply-To: References: Date: Wed, 4 Mar 2015 11:48:06 -0800 Message-ID: Subject: Re: Request Queueing after deploy + USR2 restart From: Sarkis Varozian To: Michael Fischer Cc: unicorn-public Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: PublicInbox::Filter 0.0.1 List-Id: Michael, Thanks for this - I have since changed the way we are restarting the unicorn servers after a deploy by changing capistrano task to do: in :sequence, wait: 30 We have 4 backends and the above will restart them sequentially, waiting 30s (which I think should be more than enough time), however, I still get the following latency spikes after a deploy: http://goo.gl/tYnLUJ This is what the individual servers look like for the same time interval: http://goo.gl/x7KcKq On Tue, Mar 3, 2015 at 2:32 PM, Michael Fischer wrote: > If the response times are falling a minute or so after the reload, I'd > chalk it up to a cold CPU cache. You will probably want to stagger your > reloads across backends to minimize the impact. > > --Michael > > On Tue, Mar 3, 2015 at 2:24 PM, Sarkis Varozian > wrote: > >> We have a rails application with the following unicorn.rb: >> http://goo.gl/qZ5NLn >> >> When we deploy to the application, a USR2 signal is sent to the unicorn >> master which spins up a new master and we use the before_fork in the >> unicorn.rb config above to send signals to the old master as the new >> workers come online. >> >> I've been trying to debug a weird issue that manifests as "Request >> Queueing" in our Newrelic APM. The graph shows what happens after a >> deployment (represented by the vertical lines). Here is the graph: >> http://goo.gl/iFZPMv . As you see from the graph, it is inconsistent - >> there is always a latency spike - however, at times Request Queueing is >> higher than previous deploys. >> >> Any ideas on what exactly is going on here? Any suggestions on >> tools/profilers to use to get to the bottom of this? Should we expect this >> to happen on each deploy? >> >> Thanks, >> >> -- >> *Sarkis Varozian* >> svarozian@gmail.com >> >> >> > -- *Sarkis Varozian* svarozian@gmail.com