proxy_pass: fix race condition due to flawed hijack check

The entire idea of a one-shot-based design is all the mutual
exclusion is handled by the event dispatch mechanism (epoll or
kqueue) without burdening the user with extra locking.  However, the
way the hijack works means we check the Rack env for the
'rack.hijack_io' key which is shared across requests and may
be cleared.

Ideally, this would not be a problem if the Rack dispatch allowed
returning a special value (e.g. ":ignore") instead of the normal
status-headers-body array, much like what the non-standard
"async.callback" API Thin started.

We could also avoid this problem by disallowing our "unhijack-ing"
of the socket but at a significant cost of crippling code
reusability, including that of existing middleware.

Thus, we allocate a new, empty request object here to avoid a TOCTTOU
in the following timeline:

original thread:                                 | another thread
HttpClient#yahns_step                            |
r = k.app.call(env = @hs.env)  # socket hijacked into epoll queue
<thread is scheduled away>                       | epoll_wait readiness
                                                 | ReqRes#yahns_step
                                                 | proxy dispatch ...
                                                 | proxy_busy_mod_done
************************** DANGER BELOW ********************************
                                                 | HttpClient#yahns_step
                                                 | # clears env
 # sees empty env:                               |
return :ignore if env.include?('rack.hijack_io') |

In other words, we cannot ever touch the original env seen by the
original thread since it must see the 'rack.hijack_io' value because
both are operating in the same Yahns::HttpClient object.  This will
happen regardless of GVL existence.

Avoiding errors like this is absolutely critical to every one-shot
based design.

1 files changed, 32 insertions, 3 deletions

diff --git a/lib/yahns/proxy_http_response.rb b/lib/yahns/proxy_http_response.rb
index af8d8cc..4801008 100644
--- a/lib/yahns/proxy_http_response.rb
+++ b/lib/yahns/proxy_http_response.rb
@@ -227,11 +227,40 @@ module Yahns::HttpResponse # :nodoc:
     proxy_busy_mod_done(wbuf.wbuf_persist) # returns nil
   end
 
+  def proxy_wait_next(qflags)
+    # We must allocate a new, empty request object here to avoid a TOCTTOU
+    # in the following timeline
+    #
+    # original thread:                                 | another thread
+    # HttpClient#yahns_step                            |
+    # r = k.app.call(env = @hs.env)  # socket hijacked into epoll queue
+    # <thread is scheduled away>                       | epoll_wait readiness
+    #                                                  | ReqRes#yahns_step
+    #                                                  | proxy dispatch ...
+    #                                                  | proxy_busy_mod_done
+    # ************************** DANGER BELOW ********************************
+    #                                                  | HttpClient#yahns_step
+    #                                                  | # clears env
+    # sees empty env:                                  |
+    # return :ignore if env.include?('rack.hijack_io') |
+    #
+    # In other words, we cannot touch the original env seen by the
+    # original thread since it must see the 'rack.hijack_io' value
+    # because both are operating in the same Yahns::HttpClient object.
+    # This will happen regardless of GVL existence
+    hs = Unicorn::HttpRequest.new
+    hs.buf.replace(@hs.buf)
+    @hs = hs
+
+    # n.b. we may not touch anything in this object once we call queue_mod,
+    # another thread is likely to take it!
+    Thread.current[:yahns_queue].queue_mod(self, qflags)
+  end
+
   def proxy_busy_mod_done(alive)
-    q = Thread.current[:yahns_queue]
     case http_response_done(alive)
-    when :wait_readable then q.queue_mod(self, Yahns::Queue::QEV_RD)
-    when :wait_writable then q.queue_mod(self, Yahns::Queue::QEV_WR)
+    when :wait_readable then proxy_wait_next(Yahns::Queue::QEV_RD)
+    when :wait_writable then proxy_wait_next(Yahns::Queue::QEV_WR)
     when :close then Thread.current[:yahns_fdmap].sync_close(self)
     end