From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS15169 74.125.0.0/16 X-Spam-Status: No, score=-2.9 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_PASS shortcircuit=no autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail-wm0-f51.google.com (mail-wm0-f51.google.com [74.125.82.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id C123520078 for ; Tue, 7 Jun 2016 15:17:49 +0000 (UTC) Received: by mail-wm0-f51.google.com with SMTP id k204so73986216wmk.0 for ; Tue, 07 Jun 2016 08:17:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=K6VvxgMxIrx30zW1ad0qHNnQpfRUaNqHSCS0K7ZI/us=; b=FTOBLu2CrRG3dzZp9kkZ3MgPr5HbZpW8LLTExU/AWeglmazYy3Fbn/1T2m8nm/h19L /ihbZDJsyYOxfTdymLCzdu4rHRb+5iLiUTF6muLHD5t+ewgGkM0BmkmpWzBhzBJjgm+j PUdc+dEw67feCNVuJtHdo5dRAr8+wWkW/F/6AMpyMK8dBzqS/0h+/gkcEEdT4jU6BYT8 7hbKu7buANHmfMgNhNZ3DxTvJK9NPe3/2Wpy2pqFSn1oEiJQcyDCWwUEf5Jri46ZOM4a jzuZG+ETnFok1wqGqckb7Eln3ISwO43j4Ot50P1VqpZdwCu8sa6R38uwYddUgPK3YoMF jBBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=K6VvxgMxIrx30zW1ad0qHNnQpfRUaNqHSCS0K7ZI/us=; b=KjY9/PjoRK9/R78xn11f+LFaNz0d0jkTQ/vkVCOucUtRS/DZdtpasKa3AB8DHUaM7R 2Ve7teU7bDczTxIhAti9Gz/6c0BIWoEPNSLiSEZLHChnhSo+3K4XqC6LTcNIcFZsazNh OBd5dJKVs9WhMk3BdI5wwJyiHr9gfcFpd2ncSMtg8U1vsoYlJ59wH5lVo+tcP7ygWe1g 373/VmH/1Qv3hTAWzy9UmPGPPcJ2s1JB4veD4YFqSmWk8F3atEE6Mh771GJPUWQ0J3qS 2PGnK8t4buM/sfAQQ8aa3EoauBVzHCHQRadMGy+m3UYBWEkOeKv3WjTF+iI6JzBs3qhl W0cQ== X-Gm-Message-State: ALyK8tLRK9fbAzDZJl5Fk7E0PT42Ty+DHsAko68m0r2yEMRUhJlPcVSSwjyF3HPpezMmukvH7B4AMFCFAV9asA== X-Received: by 10.28.93.80 with SMTP id r77mr3227443wmb.20.1465312668348; Tue, 07 Jun 2016 08:17:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.212.16 with HTTP; Tue, 7 Jun 2016 08:17:28 -0700 (PDT) In-Reply-To: <20160607134107.GA4500@dcvr.yhbt.net> References: <20160607134107.GA4500@dcvr.yhbt.net> From: =?UTF-8?Q?Jesper_R=C3=B8nn=2DJensen?= Date: Tue, 7 Jun 2016 17:17:28 +0200 Message-ID: Subject: Re: [PATCH] `unicorn upgrade` script resilience against exceptions To: Eric Wong Cc: unicorn-public@bogomips.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: Thanks for input. I will close this issue and wait for your patch later, then. Will also close pull-request I put at github with reference to this thread. PS. Thanks for tip on HTML vs Plain text mails. Didn't know Gmail could send plain text :) On Tue, Jun 7, 2016 at 3:41 PM, Eric Wong wrote: > Jesper R=C3=B8nn-Jensen wrote: >> This is actually a change proposal "by accident". But first, some >> background: > > Please don't send HTML mail. I guess the bounce message didn't > include the reason for the bounce, but should be fixed; now; > and the bounce will tell you to not send HTML. > > Anyways if you stick to old-school Usenet/mailing list posting > conventions, I'm more than happy to help you :) > >> My server uses a standard version of this `init.sh` script. I found that >> executing `unicorn upgrade` often gives me problems. > > Hm... I haven't looked at that script in a while given all > the new-fangled init replacements... > >> For example, it breaks my Capistrano deploy script to use `unicorn:upgra= de` >> to deploy. This is a shame, since it's the correct >> command for Unicorn to pick up any code changes. But unicorn upgrade in = the >> script fails with messages like: >> >> ``` >> sudo /etc/init.d/unicorn upgrade >> Couldn't upgrade, starting 'export HOME=3D/home/deploy ; cd >> /var/www/my_app/current && /home/deploy/.rvm/wrappers/my_app/bundle exec >> unicorn -D -c /var/www/my_app/shared/config/unicorn.rb -E production' >> instead >> master failed to start, check stderr log for details >> ``` >> >> And the stderror log: >> ``` >> E, [2016-06-07T09:03:27.517267 #27583] ERROR -- : reaped #> pid 27621 exit 1> exec()-ed >> /var/www/my_app/shared/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/u= nicorn/http_server.rb:195:in >> `pid=3D': Already running on PID:27583 (or >> pid=3D/var/www/my_app/shared/tmp/pids/unicorn.pid is stale) >> (ArgumentError) >> ``` >> >> I am proposing this change because I misread the documentation for the >> SIGNALS! >> I thought that Unicorn itself sends a QUIT after all subprocesses has en= ded >> via SIG USR2. >> >> The change is to remove the QUIT signal to the PID.oldbin process, and t= hat >> apparently stops all the failures I ran into. So I made this change on a >> server, and it works! >> >> I just don't understand why, since reading the documentation again it sa= ys: >> https://github.com/defunkt/unicorn/blob/d23d4713dc9ab9732c574f5aa34a4b67= 40b43164/SIGNALS#L35 >> >> > * USR2 - reexecute the running binary. A separate QUIT >> > should be sent to the original process once the child is verified to >> > be up and running. >> >> So clearly, I should send the QUIT signal. > > Yes. > >> My goal is to have a resilient, robust deployment, where unicorn picks u= p >> any code changes. > > Unfortunately, PID files have always been racy with USR2. > Nowadays systemd is fairly standardized and seems to work > pretty well for managing sockets and services. > > I'm still not a fan of some systemd things, but I think the socket > activation part is very nice. Example @.service and .socket files > are distributed nowadays: > > https://unicorn.bogomips.org/examples/unicorn@.service > https://unicorn.bogomips.org/examples/unicorn.socket > >> Can you help me and enlighten me as to why this proposed change works? > > Anyways, things are racy in that script and increasing the > "sleep 2" to a higher number will help if your system is really > overloaded. > >> --- >> examples/init.sh | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/examples/init.sh b/examples/init.sh >> index 1f0e035..cbadc11 100644 >> --- a/examples/init.sh >> +++ b/examples/init.sh >> @@ -45,7 +45,7 @@ restart|reload) >> $CMD >> ;; >> upgrade) >> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT >> + if sig USR2 && sleep 2 && sig 0 >> then >> n=3D$TIMEOUT >> while test -s $old_pid && test $n -ge 0 >> -- > > I actually wrote a better init script for a similar server, > patch coming in a bit (or later, close to falling asleep). --=20 Jesper R=C3=B8nn-Jensen Nine A/S Mobile: +45 2989 1822 Blog http://justaddwater.dk/ jesperrr@gmail.com (Private e-mail and Google Talk IM)