From: Camilo Lopez <camilo@camilolopez.com>
To: mongrel-unicorn@rubyforge.org
Cc: Camilo Lopez <camilo@shopify.com>, admin@camilolopez.com
Subject: [PATCH] Kill lazy workers with TERM before KILL
Date: Wed, 16 Apr 2014 03:35:18 +0000 [thread overview]
Message-ID: <1397619318-28190-2-git-send-email-camilo@shopify.com> (raw)
In-Reply-To: <1397619318-28190-1-git-send-email-camilo@shopify.com>
We want to know what was going on with a lazy worker, KILL is un-trappable so we
are sending TERM instead, then to make sure the process really dies we are
KILLing a second later.
---
lib/unicorn/http_server.rb | 31 +++++++++++++++++++------------
lib/unicorn/worker.rb | 7 ++++++-
2 files changed, 25 insertions(+), 13 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 21cb9a1..1b2a33b 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -474,22 +474,29 @@ class Unicorn::HttpServer
end
# forcibly terminate all workers that haven't checked in in timeout seconds. The timeout is implemented using an unlinked File
+ # Workers will be killed with TERM first, so the backtrace can be printed, on
+ # the next call to murder_lazy_workers the worker will receive KILL.
def murder_lazy_workers
next_sleep = @timeout - 1
now = Time.now.to_i
WORKERS.dup.each_pair do |wpid, worker|
- tick = worker.tick
- 0 == tick and next # skip workers that haven't processed any clients
- diff = now - tick
- tmp = @timeout - diff
- if tmp >= 0
- next_sleep > tmp and next_sleep = tmp
- next
+ if worker.needs_sig_kill?
+ kill_worker(:KILL, wpid)
+ else
+ tick = worker.tick
+ 0 == tick and next # skip workers that haven't processed any clients
+ diff = now - tick
+ tmp = @timeout - diff
+ if tmp >= 0
+ next_sleep > tmp and next_sleep = tmp
+ next
+ end
+ next_sleep = 0
+ logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
+ "(#{diff}s > #{@timeout}s), killing"
+ kill_worker(:TERM, wpid)
+ worker.needs_sig_kill = true
end
- next_sleep = 0
- logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
- "(#{diff}s > #{@timeout}s), killing"
- kill_worker(:KILL, wpid) # take no prisoners for timeout violations
end
next_sleep <= 0 ? 1 : next_sleep
end
@@ -606,7 +613,7 @@ class Unicorn::HttpServer
def init_worker_process(worker)
worker.atfork_child
# we'll re-trap :QUIT later for graceful shutdown iff we accept clients
- EXIT_SIGS.each { |sig| trap(sig) { exit!(0) } }
+ EXIT_SIGS.each { |sig| trap(sig) { STDERR.puts(caller); exit!(0) } }
exit!(0) if (SIG_QUEUE & EXIT_SIGS)[0]
WORKER_QUEUE_SIGS.each { |sig| trap(sig, nil) }
trap(:CHLD, 'DEFAULT')
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index e74a1c9..4214500 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -11,7 +11,7 @@ require "raindrops"
class Unicorn::Worker
# :stopdoc:
attr_accessor :nr, :switched
- attr_writer :tmp
+ attr_writer :tmp, :needs_sig_kill
attr_reader :to_io # IO.select-compatible
PER_DROP = Raindrops::PAGE_SIZE / Raindrops::SIZE
@@ -25,6 +25,7 @@ class Unicorn::Worker
@nr = nr
@tmp = @switched = false
@to_io, @master = Unicorn.pipe
+ @needs_sig_kill = false
end
def atfork_child # :nodoc:
@@ -149,4 +150,8 @@ class Unicorn::Worker
Process.euid != uid and Process::UID.change_privilege(uid)
@switched = true
end
+
+ def needs_sig_kill?
+ @needs_sig_kill
+ end
end
--
1.9.2
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
next prev parent reply other threads:[~2014-04-16 3:58 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-16 3:35 [PATCH] Print stack for lazy workers Camilo Lopez
2014-04-16 3:35 ` Camilo Lopez [this message]
2014-04-16 8:44 ` [PATCH] Kill lazy workers with TERM before KILL Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/unicorn/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1397619318-28190-2-git-send-email-camilo@shopify.com \
--to=camilo@camilolopez.com \
--cc=admin@camilolopez.com \
--cc=camilo@shopify.com \
--cc=mongrel-unicorn@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).