unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Daniel Condomitti <daniel@condomitti.com>
To: Tony Devlin <tonydevlin@gmail.com>
Cc: Eric Wong <e@80x24.org>, unicorn-public@bogomips.org
Subject: Re: Weird Unicorn Timeout Issues (Hibernation problem?)
Date: Wed, 6 Aug 2014 11:26:28 -0700	[thread overview]
Message-ID: <8E584028BCA3426ABB655EC529613E22@condomitti.com> (raw)
In-Reply-To: <CAKM1sPPsb24aNA-puN=E6aYn=RqAynMaNMBiGvWqHtyaFe4cew@mail.gmail.com>

This is exactly what happened to us and I should have been clearer. I wasn’t referring to the default Linux kernel settings causing the killing the connection; it was a network device between our application servers and the database server. It only affected certain applications as some were hit hundreds of times per second and would never be disconnected and the ones that would disconnect were only hit a few times per hour. I -believe- we just dropped the keepalive interval on both sides of the firewall below its idle timeout.  

On Wednesday, August 6, 2014 at 7:05 AM, Tony Devlin wrote:

> Eric,
> The problem is a firewall that sits between the servers and the database.  It is an idle session timeout of 30 minutes, so it is silently killing the connection.  I have reached out to our Network Engineering department but they are saying they can not change that idle session timeout, nor create a special rule to allow this connection to bypass that rule.   
> Currently, I setup a polling device that calls the applications URL every 20 minutes.  This causes the connection between the server and DB to refresh it's idle timeout.  This is obviously a very hacky way to handle it, so I am trying to look into AR and Oracle_Enhanced to see if they have some sort of keepalive option for the database.  I thought it would work with the reaping_frequency, but apparently that does not work out as I had expected when you are not running in pools or a thread.  So I'm still on the lookout for something to handle that.  
> On Wed, Aug 6, 2014 at 5:45 AM, Eric Wong <e@80x24.org (mailto:e@80x24.org)> wrote:
> >  
> > Any update?  It looks like your DB driver is not using/respecting any
> > timeout at all[1].  It is bad to not have a timeout there.  There should
> > be a way to set a timeout so you can at least tell the user the DB
> > connection dropped or maybe get your app to disconnect+retry once.
> >  
> > A better looking strace would be something like:
> >  
> >     write(fd, ...); => success
> >     (poll|select|ppoll) syscall ...
> >     read(fd, ...); /* only if (poll|select|ppoll) was successful[2] */
> >  
> > This goes for configuring all connections/services for any app.
> >  
> > [1] or if it's relying on SO_RCVTIMEO socket option(rare), that's set
> >     way too high.  Any timeout set for any external connection should
> >     be lower than the unicorn (last-resort) timeout feature.
> >  
> > [2] any read() syscall after (poll|select|ppoll) should be non-blocking,
> >     because (poll|select|ppoll) may spuriously wakeup.  

      reply	other threads:[~2014-08-06 18:56 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-08-04 18:12 Weird Unicorn Timeout Issues (Hibernation problem?) Tony Devlin
2014-08-04 18:39 ` Eric Wong
2014-08-04 18:41   ` Kapil Israni
2014-08-04 18:48     ` Eric Wong
2014-08-04 18:45   ` Tony Devlin
2014-08-04 19:11     ` Eric Wong
2014-08-04 19:21       ` Tony Devlin
2014-08-04 19:45         ` Eric Wong
2014-08-04 20:06           ` Michael Fischer
2014-08-04 18:55   ` Daniel Condomitti
2014-08-04 19:34     ` Eric Wong
2014-08-04 20:24       ` Tony Devlin
2014-08-04 20:44         ` Eric Wong
2014-08-04 20:46           ` Eric Wong
2014-08-05 14:46             ` Tony Devlin
2014-08-06  9:45               ` Eric Wong
2014-08-06 14:05                 ` Tony Devlin
2014-08-06 18:26                   ` Daniel Condomitti [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8E584028BCA3426ABB655EC529613E22@condomitti.com \
    --to=daniel@condomitti.com \
    --cc=e@80x24.org \
    --cc=tonydevlin@gmail.com \
    --cc=unicorn-public@bogomips.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).