From 34b400cbec2a05e9a1d9fad2d6bd34f54620fdcb Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Wed, 24 Aug 2011 17:59:55 -0700 Subject: doc: add Application Timeouts document Hopefully this leads to fewer worker processes being killed. --- .document | 1 + Application_Timeouts | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 Application_Timeouts diff --git a/.document b/.document index a3d7605..4092597 100644 --- a/.document +++ b/.document @@ -26,3 +26,4 @@ unicorn_rails_1 ISSUES Sandbox Links +Application_Timeouts diff --git a/Application_Timeouts b/Application_Timeouts new file mode 100644 index 0000000..5f0370d --- /dev/null +++ b/Application_Timeouts @@ -0,0 +1,77 @@ += Application Timeouts + +This article focuses on _application_ setup for Rack applications, but +can be expanded to all applications that connect to external resources +and expect short response times. + +This article is not specific to \Unicorn, but exists to discourage +the overuse of the built-in +{timeout}[link:Unicorn/Configurator.html#method-i-timeout] directive +in \Unicorn. + +== ALL External Resources Are Considered Unreliable + +Network reliability can _never_ be guaranteed. Network failures cannot +be detected reliably by the client (Rack application) in a reasonable +timeframe, not even on a LAN. + +Thus, application authors must configure timeouts when interacting with +external resources. + +Most database adapters allow configurable timeouts. + +Net::HTTP and Net::SMTP in the Ruby standard library allow +configurable timeouts. + +Even for things as fast as {memcached}[http://memcached.org/], +{dalli}[http://rubygems.org/gems/dalli], +{memcached}[http://rubygems.org/gems/memcached] and +{memcache-client}[http://rubygems.org/gems/memcache-client] RubyGems all +offer configurable timeouts. + +Consult the relevant documentation for the libraries you use on +how to configure these timeouts. + +== Rolling Your Own Socket Code + +Use non-blocking I/O and IO.select with a timeout to wait on sockets. + +== Timeout module in the Ruby standard library + +Ruby offers a Timeout module in its standard library. It has several +caveats and is not always reliable: + +* /Some/ Ruby C extensions are not interrupted/timed-out gracefully by + this module (report these bugs to extension authors, please) but + pure-Ruby components should be. + +* Long-running tasks may run inside `ensure' clauses after timeout + fires, causing the timeout to be ineffective. + +The Timeout module is a second-to-last-resort solution, timeouts using +IO.select (or similar) are more reliable. If you depend on libraries +that do not offer timeouts when connecting to external resources, kindly +ask those library authors to provide configurable timeouts. + +=== A Note About Filesystems + +Most operations to regular files on POSIX filesystems are NOT +interruptable. Thus, the "timeout" module in the Ruby standard library +can not reliably timeout systems with massive amounts of iowait. + +If your app relies on the filesystem, ensure all the data your +application works with is small enough to fit in the kernel page cache. +Otherwise increase the amount of physical memory you have to match, or +employ a fast, low-latency storage system (solid state). + +Volumes mounted over NFS (and thus a potentially unreliable network) +must be mounted with timeouts and applications must be prepared to +handle network/server failures. + +== The Last Line Of Defense + +The {timeout}[link:Unicorn/Configurator.html#method-i-timeout] mechanism +in \Unicorn is an extreme solution that should be avoided whenever +possible. It will help catch bugs in your application where and when +your application forgets to use timeouts, but it is expensive as it +kills and respawns a worker process. -- cgit v1.2.3-24-ge0c7