queue_epoll: document epoll concurrency caveats

Multithreaded epoll is a tricky interface to use.  Fortunately it
should be hidden from users and yahns is no different from any other
multithreaded server from a user application programming
perspective (which requires a lot of careful programming in the
first place).

1 files changed, 16 insertions, 1 deletions

diff --git a/lib/yahns/queue_epoll.rb b/lib/yahns/queue_epoll.rb
index 3e6536b..854aba9 100644
--- a/lib/yahns/queue_epoll.rb
+++ b/lib/yahns/queue_epoll.rb
@@ -1,6 +1,10 @@
 # -*- encoding: binary -*-
 # Copyright (C) 2013, Eric Wong <normalperson@yhbt.net> and all contributors
 # License: GPLv3 or later (https://www.gnu.org/licenses/gpl-3.0.txt)
+#
+# This is the dangerous, low-level epoll interface for sleepy_penguin
+# It is safe as long as you're aware of all potential concurrency
+# issues given multithreading, GC, and epoll itself.
 class Yahns::Queue < SleepyPenguin::Epoll::IO # :nodoc:
   include SleepyPenguin
   attr_accessor :fdmap # Yahns::Fdmap
@@ -18,6 +22,8 @@ class Yahns::Queue < SleepyPenguin::Epoll::IO # :nodoc:
   # for HTTP and HTTPS servers, we rely on the io writing to us, first
   # flags: QEV_RD/QEV_WR (usually QEV_RD)
   def queue_add(io, flags)
+    # order is very important here, this thread cannot do anything with
+    # io once we've issued epoll_ctl() because another thread may use it
     @fdmap.add(io)
     epoll_ctl(Epoll::CTL_ADD, io, flags)
   end
@@ -28,9 +34,14 @@ class Yahns::Queue < SleepyPenguin::Epoll::IO # :nodoc:
   end
 
   # use only before hijacking, once hijacked, io may be unusable to us
+  # It is not safe to call this unless it is an unarmed EPOLLONESHOT
+  # object.
   def queue_del(io)
-    @fdmap.forget(io)
+    # order does not really matter here, however Epoll::CTL_DEL
+    # will free up ~200 bytes of unswappable kernel memory,
+    # so we call it first
     epoll_ctl(Epoll::CTL_DEL, io, 0)
+    @fdmap.forget(io)
   end
 
   # returns an array of infinitely running threads
@@ -39,6 +50,10 @@ class Yahns::Queue < SleepyPenguin::Epoll::IO # :nodoc:
       thr_init
       begin
         epoll_wait(max_events) do |_, io| # don't care for flags for now
+
+          # Note: we absolutely must not do anything with io after
+          # we've called epoll_ctl on it, io is exclusive to this
+          # thread only until epoll_ctl is called on it.
           case rv = io.yahns_step
           when :wait_readable
             epoll_ctl(Epoll::CTL_MOD, io, QEV_RD)