Re: Fwd: Zero-downtime deploys, signals, ruby-pg

Re: Fwd: Zero-downtime deploys, signals, ruby-pg

[Eric Wong]

Joe Van Dyk <joe@tanga.com> wrote:
> Just sent this via gmail. Any ideas why I got this error?

It contained an HTML portion which Mailman couldn't strip or handle.
Forwarding to the list for now[1]

> ---------- Forwarded message ----------
> From: Joe Van Dyk <joe@tanga.com>
> To: unicorn list <mongrel-unicorn@rubyforge.org>
> Cc:
> Date: Tue, 19 Nov 2013 15:29:27 -0800
> Subject: Zero-downtime deploys, signals, ruby-pg
> Hi folks,
> 
> I think the upgrade to ruby-pg 0.16 broke the zero-downtime deployments on
> unicorn.
> 
> See mailing list post here:
> https://groups.google.com/d/msg/ruby-pg/5_ylGmog1S4/uvQ3EDl7MUUJ
> 
> I think the problem is that unicorn sends a signal to the worker thread
> when it's about to be shutdown. ruby-pg receives that signal and
> immediately cancels any actively-running queries. Leading to exceptions
> like:
> 
> PG::QueryCanceled: ERROR: canceling statement due to user request : UPDATE
> "orders" SET...
> from deep inside Sequel or ActiveRecord.
> 
> Michael Granger said it's a unicorn problem (apparently).

Unfortunately, there's no easy solution in unicorn.

ruby-pg is correct in interrupting the running query since it doesn't
know if the interrupt signal is for graceful unicorn shutdown or if the
user is thinking: "must stop query now or all data is destroyed!"

So I suspect ruby-pg favors the panicking user who accidentally issued
an UPDATE or DELETE without a proper WHERE clause :)

As far as workarounds on the unicorn/rack-side go:

1) retry on PG::QueryCanceled queries, this is analogous to the
   EINTR retries Ruby does for most interrupted syscalls.
   However, this takes time and you need to be careful in case
   you forgot transactions and what not.

2) Avoid SIGQUIT on the unicorn master completely, just send SIGKILL
   to the master.  The workers will slowly realize the master is dead
   and die gracefully.  This could increase RAM usage.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Zero-downtime deploys, signals, ruby-pg

[Eric Wong]

Eric Wong <normalperson@yhbt.net> wrote:
> Joe Van Dyk <joe@tanga.com> wrote:
> > Just sent this via gmail. Any ideas why I got this error?
> 
> It contained an HTML portion which Mailman couldn't strip or handle.
> Forwarding to the list for now[1]

[1] - I forgot the footnote from the earlier email:

      I've been working on replacing this mailing list with something
      like a mailing list (but not exactly)[2].  Hopefully it'll do a
      better job of stripping HTML as well as having public logging
      of message rejections/failures.

[2] - This is needed because Rubyforge might not be around much longer.
      Rubyforge has been down for days at a time with no explanation.
      Running a real mailing list myself is probably too much effort,
      and relying on hard-to-admin centralized systems isn't good
      anyways, so I'm coming up with something slacker BOFHs like
      me can deal with without ending up on spam blacklists.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying