Re: Fwd: Zero-downtime deploys, signals, ruby-pg

From: Eric Wong <normalperson@yhbt.net>
To: Joe Van Dyk <joe@tanga.com>
Cc: mongrel-unicorn@rubyforge.org
Subject: Re: Fwd: Zero-downtime deploys, signals, ruby-pg
Date: Wed, 20 Nov 2013 09:47:17 +0000	[thread overview]
Message-ID: <20131120094717.GA17836@dcvr.yhbt.net> (raw)
In-Reply-To: <CACfv+p+4FK5BSL4D0uMi-Zgtf7VtcpTVu7sG4WXv2YXWyRbeWA@mail.gmail.com>

Joe Van Dyk <joe@tanga.com> wrote:
> Just sent this via gmail. Any ideas why I got this error?

It contained an HTML portion which Mailman couldn't strip or handle.
Forwarding to the list for now[1]

> ---------- Forwarded message ----------
> From: Joe Van Dyk <joe@tanga.com>
> To: unicorn list <mongrel-unicorn@rubyforge.org>
> Cc:
> Date: Tue, 19 Nov 2013 15:29:27 -0800
> Subject: Zero-downtime deploys, signals, ruby-pg
> Hi folks,
> 
> I think the upgrade to ruby-pg 0.16 broke the zero-downtime deployments on
> unicorn.
> 
> See mailing list post here:
> https://groups.google.com/d/msg/ruby-pg/5_ylGmog1S4/uvQ3EDl7MUUJ
> 
> I think the problem is that unicorn sends a signal to the worker thread
> when it's about to be shutdown. ruby-pg receives that signal and
> immediately cancels any actively-running queries. Leading to exceptions
> like:
> 
> PG::QueryCanceled: ERROR: canceling statement due to user request : UPDATE
> "orders" SET...
> from deep inside Sequel or ActiveRecord.
> 
> Michael Granger said it's a unicorn problem (apparently).

Unfortunately, there's no easy solution in unicorn.

ruby-pg is correct in interrupting the running query since it doesn't
know if the interrupt signal is for graceful unicorn shutdown or if the
user is thinking: "must stop query now or all data is destroyed!"

So I suspect ruby-pg favors the panicking user who accidentally issued
an UPDATE or DELETE without a proper WHERE clause :)

As far as workarounds on the unicorn/rack-side go:

1) retry on PG::QueryCanceled queries, this is analogous to the
   EINTR retries Ruby does for most interrupted syscalls.
   However, this takes time and you need to be careful in case
   you forgot transactions and what not.

2) Avoid SIGQUIT on the unicorn master completely, just send SIGKILL
   to the master.  The workers will slowly realize the master is dead
   and die gracefully.  This could increase RAM usage.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying