unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: Adam Fields <unicorn5958@street86.com>
Cc: unicorn-public@bogomips.org
Subject: Re: Dead PostgreSQL connections on worker restart
Date: Fri, 15 Apr 2016 05:42:02 +0000	[thread overview]
Message-ID: <20160415054202.GA4043@dcvr.yhbt.net> (raw)
In-Reply-To: <E6592304-8CB6-46AB-BDF6-CCAC1732219D@street86.com>

Adam Fields <unicorn5958@street86.com> wrote:
> We have discovered that every time a unicorn worker is restarted,
> rails throws "PG::ConnectionBad: connection is closed” errors.

<snip>

> We are using preload: true, and have before and after fork
> rules to close and reopen the connections:

<snip>

> before_fork do |server, worker|
>  defined?(ActiveRecord::Base) and
>      ActiveRecord::Base.connection.disconnect!

<snip>

> Yet it seems that something is holding onto a dead connection
> and trying to use it. 

I'm not familiar with the Rails side anymore, so hopefully
others can chime in.  However, from the OS perspective,
it's generic and not dependent on Ruby or Pg, just *nix...

> How can we troubleshoot this?

Use lsof to determine which sockets are shared.
lsof should be available and common on most platforms,
definitely Linux-based ones.

	lsof -p $PID_OF_MASTER
	lsof -p $PID_OF_WORKER

And compare the sockets shared between the processes based on
the output of lsof.  I'm mostly going off Linux lsof output,
here, but the idea is the same across all *nixes...

Sockets marked as "LISTEN" are intended to be shared by unicorn.
(UDP and other packet-oriented so sockets are fine to share, too;
 but most apps don't use them)

If your Pg connection is over TCP, look for TCP sockets
in the ESTABLISHED state and ensure each process has
a unique number in the DEVICE column (and/or local port).

Likewise if your Pg connection is over a Unix socket.  Just make
sure the path is the correct one (to distinguish from any Unix
sockets unicorn listens on).

In other words, no stream-oriented connections you make should
be shared.  There probably shouldn't be any stream-oriented
connections in your master process at all.

  reply	other threads:[~2016-04-15  5:42 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-04-15  3:26 Adam Fields
2016-04-15  5:42 ` Eric Wong [this message]
2016-04-21 19:42   ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160415054202.GA4043@dcvr.yhbt.net \
    --to=e@80x24.org \
    --cc=unicorn-public@bogomips.org \
    --cc=unicorn5958@street86.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this inbox:

	../../unicorn.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).