Logging errors from Unicorn

Logging errors from Unicorn

[Peter Kieltyka]

Hey guys,

I'm running a Rails 3 beta3 application using Ruby 1.9.2-preview3 and it has been running fairly well, however every now and then (usually when under high load), I see error messages like:

2010/06/04 14:34:29 [error] 1884#0: *8844 upstream prematurely closed connection while reading response header from upstream, client: 119.235.237.93, server: portal.crowdreel.com, request: "GET /post/18Vmy HTTP/1.1", upstream: "http://unix:/home/app/crowdreel-portal/tmp/sockets/unicorn.sock:/post/18Vmy", host: "crowdreel.com"

... a lot of these messages.. this is a pretty heavy duty server with lots of ram.

I have a setup similar to what I read up on github's blog as in: nginx -> unicorns via a socket upstream.

My unicorn.rb file looks like:

--------

# 16 workers and 1 master
worker_processes 16

# Load the app into the master before forking workers for super-fast worker spawn times
preload_app false

# Restart any workers that haven't responded in 30 seconds
timeout 45

# Listen on a Unix data socket
listen File.expand_path(File.dirname(__FILE__)+'/../tmp/sockets/unicorn.sock'), :backlog => 2048

----------

I don't preload the app because I start running into annoying issues where my workers use the same db connection, so instead I just have them load their own instance, its very fast to start up anyways.

I've read everything I could on this issue and exhausted Google, which leaves me to think the only thing left is some incompatibilities with 1.9.2. So I was wondering, how can I get logs from unicorn's workers/master to know what is happening to these workers, perhaps something is segfaulting?

Thx!

Regards,

Peter
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Logging errors from Unicorn

[Eric Wong]

Peter Kieltyka <peter.kieltyka@nulayer.com> wrote:
> Hey guys,
> 
> I'm running a Rails 3 beta3 application using Ruby 1.9.2-preview3 and
> it has been running fairly well, however every now and then (usually
> when under high load), I see error messages like:
> 
> 2010/06/04 14:34:29 [error] 1884#0: *8844 upstream prematurely closed
> connection while reading response header from upstream, client:
> 119.235.237.93, server: portal.crowdreel.com, request: "GET
> /post/18Vmy HTTP/1.1", upstream:
> "http://unix:/home/app/crowdreel-portal/tmp/sockets/unicorn.sock:/post/18Vmy",
> host: "crowdreel.com"
> 
> ... a lot of these messages.. this is a pretty heavy duty server with
> lots of ram.
> 
> I have a setup similar to what I read up on github's blog as in: nginx
> -> unicorns via a socket upstream.
> 
> My unicorn.rb file looks like:
> 
> --------
> 
> # 16 workers and 1 master
> worker_processes 16
> 
> # Load the app into the master before forking workers for super-fast worker spawn times
> preload_app false
> 
> # Restart any workers that haven't responded in 30 seconds
> timeout 45
> 
> # Listen on a Unix data socket
> listen File.expand_path(File.dirname(__FILE__)+'/../tmp/sockets/unicorn.sock'), :backlog => 2048
> 
> ----------

> I don't preload the app because I start running into annoying issues
> where my workers use the same db connection, so instead I just have
> them load their own instance, its very fast to start up anyways.
> 
> I've read everything I could on this issue and exhausted Google, which
> leaves me to think the only thing left is some incompatibilities with
> 1.9.2. So I was wondering, how can I get logs from unicorn's
> workers/master to know what is happening to these workers, perhaps
> something is segfaulting?

Hi Peter,

Set stderr_path in your unicorn config file so Unicorn can log
its errors to a file.  Otherwise Unicorn will send all errors to
/dev/null when daemonized.

I shall emphasize the importance of stderr_path if you're using the
default logger configuration in the documentation.

Any chance the "/post/18Vmy" request is hitting your timeout?  The only
way you can know is if Unicorn logged its errors.  You might also want
to check your Rails log for exceptions, too, as some uncaught errors
may show up there.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

[PATCH] doc: emphasize the importance of stderr_path

[Eric Wong]

Eric Wong <normalperson@yhbt.net> wrote:
> I shall emphasize the importance of stderr_path if you're using the
> default logger configuration in the documentation.

I've just pushed the following out to unicorn.git and website.

Comments/grammar/speling korections welcome as always.

>From 2d5a4b075801493a85c6e8d2dbdf0c95002e046d Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Fri, 4 Jun 2010 13:42:34 -0700
Subject: [PATCH] doc: emphasize the importance of stderr_path

While second nature to myself, stderr_path may be an
overlooked configuration parameter for some users.  Also,
add a minimal sample configuration file that is shorter
and hopefully less intimidating to new users.
---
 examples/unicorn.conf.minimal.rb |   13 +++++++++++++
 examples/unicorn.conf.rb         |   12 +++++++++---
 lib/unicorn/configurator.rb      |   22 ++++++++++++++++++----
 3 files changed, 40 insertions(+), 7 deletions(-)
 create mode 100644 examples/unicorn.conf.minimal.rb

diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb
new file mode 100644
index 0000000..2a47910
--- /dev/null
+++ b/examples/unicorn.conf.minimal.rb
@@ -0,0 +1,13 @@
+# Minimal sample configuration file for Unicorn (not Rack) when used
+# with daemonization (unicorn -D) started in your working directory.
+#
+# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
+# documentation.
+# See also http://unicorn.bogomips.org/examples/unicorn.conf.rb for
+# a more verbose configuration using more features.
+
+listen 2007 # by default Unicorn listens on port 8080
+worker_processes 2 # this should be >= nr_cpus
+pid "/path/to/app/shared/pids/unicorn.pid"
+stderr_path "/path/to/app/shared/log/unicorn.log"
+stdout_path "/path/to/app/shared/log/unicorn.log"
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index e209894..37c3e81 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -1,4 +1,9 @@
-# Sample configuration file for Unicorn (not Rack)
+# Sample verbose configuration file for Unicorn (not Rack)
+#
+# This configuration file documents many features of Unicorn
+# that may not be needed for some applications. See
+# http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
+# for a much simpler configuration file.
 #
 # See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
 # documentation.
@@ -22,8 +27,9 @@ timeout 30
 # feel free to point this anywhere accessible on the filesystem
 pid "/path/to/app/shared/pids/unicorn.pid"
 
-# some applications/frameworks log to stderr or stdout, so prevent
-# them from going to /dev/null when daemonized here:
+# By default, the Unicorn logger will write to stderr.
+# Additionally, ome applications/frameworks log to stderr or stdout,
+# so prevent them from going to /dev/null when daemonized here:
 stderr_path "/path/to/app/shared/log/unicorn.stderr.log"
 stdout_path "/path/to/app/shared/log/unicorn.stdout.log"
 
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 64a25e3..4fa745d 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -7,9 +7,11 @@ module Unicorn
 
   # Implements a simple DSL for configuring a Unicorn server.
   #
-  # See http://unicorn.bogomips.org/examples/unicorn.conf.rb for an
-  # example config file.  An example config file for use with nginx is
-  # also available at http://unicorn.bogomips.org/examples/nginx.conf
+  # See http://unicorn.bogomips.org/examples/unicorn.conf.rb and
+  # http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
+  # example configuration files.  An example config file for use with
+  # nginx is also available at
+  # http://unicorn.bogomips.org/examples/nginx.conf
   class Configurator < Struct.new(:set, :config_file, :after_reload)
 
     # Default settings for Unicorn
@@ -78,6 +80,10 @@ module Unicorn
     # sets object to the +new+ Logger-like object.  The new logger-like
     # object must respond to the following methods:
     #  +debug+, +info+, +warn+, +error+, +fatal+, +close+
+    # The default Logger will log its output to the path specified
+    # by +stderr_path+.  If you're running Unicorn daemonized, then
+    # you must specify a path to prevent error messages from going
+    # to /dev/null.
     def logger(new)
       %w(debug info warn error fatal close).each do |m|
         new.respond_to?(m) and next
@@ -310,11 +316,19 @@ module Unicorn
     # file will be opened with the File::APPEND flag and writes
     # synchronized to the kernel (but not necessarily to _disk_) so
     # multiple processes can safely append to it.
+    #
+    # If you are daemonizing and using the default +logger+, it is important
+    # to specify this as errors will otherwise be lost to /dev/null.
+    # Some applications/libraries may also triggering warnings that go to
+    # stderr, and they will end up here.
     def stderr_path(path)
       set_path(:stderr_path, path)
     end
 
-    # Same as stderr_path, except for $stdout
+    # Same as stderr_path, except for $stdout.  Not many Rack applications
+    # write to $stdout, but any that do will have their output written here.
+    # It is safe to point this to the same location a stderr_path.
+    # Like stderr_path, this defaults to /dev/null when daemonized.
     def stdout_path(path)
       set_path(:stdout_path, path)
     end
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Logging errors from Unicorn

[Peter]

Thanks Eric!

It really helps knowing the stderr_path configuration option :) doh. I haven't
yet run into the issue since, but I will have my unicorn logs ready when it
does.

By the way, have you ever thought about moving the unicorn mailing list to
a Google group? It's much easier to interact with, search, communicate, etc.
Just saying...

Even issues like trying to post this reply here, and getting "The following
errors were found: You have  lines longer than 80 characters. Fix that."
... hehe, thats pretty funny.

Cheers.

Peter


_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Logging errors from Unicorn

[Eric Wong]

Peter <peter.kieltyka@nulayer.com> wrote:
> By the way, have you ever thought about moving the unicorn mailing list to
> a Google group? It's much easier to interact with, search, communicate, etc.
> Just saying...

Google Groups get more spam, and last I checked there's no way to
download mail archives easily other than scraping the site.

The Rubyforge at least has that, but the searchability sucks (so we
mirror to gmane/mail-archive).

Librelist is nice, but didn't exist when I started this project and at
this point the pain of migration is more than the benefit.  I'll
consider Librelist if there's strong support on the mailing list here.

> Even issues like trying to post this reply here, and getting "The following
> errors were found: You have  lines longer than 80 characters. Fix that."
> ... hehe, thats pretty funny.

Where does it say that?  I usually wrap my replies at 72 characters, but
there should be nothing in the mailman config that prevents longer lines
from being posted (and some patches do go longer than 80 due to the +/-
prefixes).  Of course HTML email is hugely wasteful and that's not
allowed here.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying