fdmap: prevent possible/theoretical race

With a GVL-free Ruby implementation, the following situation
may occur as FDs are recycled frequently with 3 threads
running:

expiry                 | normal    | acceptor
-----------------------+-----------+----------------------
yahns_expire(fd).enter |           |
                       | close(fd) |
                       |           | accept().return => fd
shutdown(fd) - WRONG   |           |
-----------------------+-----------+----------------------

So we must prevent expiry from running while another thread
is closing, otherwise there is a small chance a lack of
lock inside the Ruby implementation itself can lead to a
mis-issued close() to a fast-recycled FD.

1 files changed, 8 insertions, 5 deletions

diff --git a/lib/yahns/queue_epoll.rb b/lib/yahns/queue_epoll.rb
index 3d2e33f..665b87c 100644
--- a/lib/yahns/queue_epoll.rb
+++ b/lib/yahns/queue_epoll.rb
@@ -27,7 +27,9 @@ class Yahns::Queue < SleepyPenguin::Epoll::IO # :nodoc:
     Thread.current[:yahns_fdmap] = @fdmap
   end
 
+  # use only before hijacking, once hijacked, io may be unusable to us
   def queue_del(io)
+    @fdmap.forget(io)
     epoll_ctl(Epoll::CTL_DEL, io, 0)
   end
 
@@ -45,12 +47,13 @@ class Yahns::Queue < SleepyPenguin::Epoll::IO # :nodoc:
           when :wait_readwrite
             epoll_ctl(Epoll::CTL_MOD, io, QEV_RDWR)
           when :ignore # only used by rack.hijack
-            @fdmap.decr
+            # we cannot call Epoll::CTL_DEL after hijacking, the hijacker
+            # may have already closed it  Likewise, io.fileno is not
+            # expected to work, so we had to erase it from fdmap before hijack
           when nil, :close
-            # this is be the ONLY place where we call IO#close on
-            # things inside the queue
-            io.close
-            @fdmap.decr
+            # this must be the ONLY place where we call IO#close on
+            # things that got inside the queue
+            @fdmap.sync_close(io)
           else
             raise "BUG: #{io.inspect}#yahns_step returned: #{rv.inspect}"
           end