yahns Ruby server user/dev discussion
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: yahns-public@yhbt.net
Subject: [PATCH 2/3] server: workaround Linux v5.5..v5.13 epoll bug
Date: Sat,  9 Oct 2021 02:24:45 +0000	[thread overview]
Message-ID: <20211009022446.705-3-e@80x24.org> (raw)
In-Reply-To: <20211009022446.705-1-e@80x24.org>

epoll_wait() wakeups from QueueQuitter got lost during graceful
shutdown since there's multiple worker threads operating off the
same FD.  Workaround the problem by re-arming the eventfd for
every worker thread reaped.

Link: https://yhbt.net/lore/lkml/20210405231025.33829-1-dave@stgolabs.net/
---
 lib/yahns/queue_epoll.rb |  4 ++++
 lib/yahns/server.rb      | 17 ++++++++++-------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/lib/yahns/queue_epoll.rb b/lib/yahns/queue_epoll.rb
index 9e4271a..a198fbf 100644
--- a/lib/yahns/queue_epoll.rb
+++ b/lib/yahns/queue_epoll.rb
@@ -32,6 +32,10 @@ def queue_mod(io, flags)
     epoll_ctl(Epoll::CTL_MOD, io, flags)
   end
 
+  def queue_del(io)
+    epoll_ctl(Epoll::CTL_DEL, io, 0)
+  end
+
   def thr_init
     Thread.current[:yahns_rbuf] = ''.dup
     Thread.current[:yahns_fdmap] = @fdmap
diff --git a/lib/yahns/server.rb b/lib/yahns/server.rb
index 208b5ee..74eeb7e 100644
--- a/lib/yahns/server.rb
+++ b/lib/yahns/server.rb
@@ -438,25 +438,28 @@ def quit_enter(alive)
   # This just injects the QueueQuitter object which acts like a
   # monkey wrench thrown into a perfectly good engine :)
   def quit_finish
-    quitter = Yahns::QueueQuitter.new
+    # we must not let quitters get GC-ed if we have any worker threads leftover
+    @quitter = Yahns::QueueQuitter.new
 
     # throw the monkey wrench into the worker threads
-    @queues.each { |q| q.queue_add(quitter, Yahns::Queue::QEV_QUIT) }
+    @queues.each { |q| q.queue_add(@quitter, Yahns::Queue::QEV_QUIT) }
 
     # watch the monkey wrench destroy all the threads!
     # Ugh, this may fail if we have dedicated threads trickling
     # response bodies out (e.g. "tail -F")  Oh well, have a timeout
     begin
       @wthr.delete_if { |t| t.join(0.01) }
+      # Workaround Linux 5.5+ bug (fixed in 5.13+)
+      # https://yhbt.net/lore/lkml/20210405231025.33829-1-dave@stgolabs.net/
+      @wthr[0] && @queues[0].respond_to?(:queue_del) and @queues.each do |q|
+        q.queue_del(@quitter)
+        q.queue_add(@quitter, Yahns::Queue::QEV_QUIT)
+      end
     end while @wthr[0] && Yahns.now <= @shutdown_expire
 
     # cleanup, our job is done
     @queues.each(&:close).clear
-
-    # we must not let quitter get GC-ed if we have any worker threads leftover
-    @quitter = quitter
-
-    quitter.close
+    @quitter.close # keep object around in case @wthr isn't empty
   rescue => e
     Yahns::Log.exception(@logger, "quit finish", e)
   ensure

  parent reply	other threads:[~2021-10-09  2:24 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-10-09  2:24 [PATCH 0/3] various minor bugfixes Eric Wong
2021-10-09  2:24 ` [PATCH 1/3] do not sleep if signals are pending Eric Wong
2021-10-09  2:24 ` Eric Wong [this message]
2021-10-09  2:24 ` [PATCH 3/3] gemspec: allow unicorn 6.x Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/yahns/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211009022446.705-3-e@80x24.org \
    --to=e@80x24.org \
    --cc=yahns-public@yhbt.net \
    --subject='Re: [PATCH 2/3] server: workaround Linux v5.5..v5.13 epoll bug' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	../../yahns.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).