yahns Ruby server user/dev discussion
 help / color / mirror / code / Atom feed
* [PATCH] proxy_pass: fix race condition due to flawed hijack check
@ 2015-04-21  8:49 Eric Wong
  2015-04-21  9:14 ` Eric Wong
  0 siblings, 1 reply; 2+ messages in thread
From: Eric Wong @ 2015-04-21  8:49 UTC (permalink / raw)
  To: yahns-public

The entire idea of a one-shot-based design is all the mutual
exclusion is handled by the event dispatch mechanism (epoll or
kqueue) without burdening the user with extra locking.  However, the
way the hijack works means we check the Rack env for the
'rack.hijack_io' key which is shared across requests and may
be cleared.

Ideally, this would not be a problem if the Rack dispatch allowed
returning a special value (e.g. ":ignore") instead of the normal
status-headers-body array, much like what the non-standard
"async.callback" API Thin started.

We could also avoid this problem by disallowing our "unhijack-ing"
of the socket but at a significant cost of crippling code
reusability, including that of existing middleware.

Thus, we allocate a new, empty request object here to avoid a TOCTTOU
in the following timeline:

original thread:                                 | another thread
HttpClient#yahns_step                            |
r = k.app.call(env = @hs.env)  # socket hijacked into epoll queue
<thread is scheduled away>                       | epoll_wait readiness
                                                 | ReqRes#yahns_step
                                                 | proxy dispatch ...
                                                 | proxy_busy_mod_done
************************** DANGER BELOW ********************************
                                                 | HttpClient#yahns_step
						 | # clears env
 # sees empty env:                               |
return :ignore if env.include?('rack.hijack_io') |

In other words, we cannot ever touch the original env seen by the
original thread since it must see the 'rack.hijack_io' value because
both are operating in the same Yahns::HttpClient object.  This will
happen regardless of GVL existence.

Avoiding errors like this is absolutely critical to every one-shot
based design.
---
 lib/yahns/proxy_http_response.rb | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)

diff --git a/lib/yahns/proxy_http_response.rb b/lib/yahns/proxy_http_response.rb
index af8d8cc..6d2457b 100644
--- a/lib/yahns/proxy_http_response.rb
+++ b/lib/yahns/proxy_http_response.rb
@@ -227,11 +227,37 @@ module Yahns::HttpResponse # :nodoc:
     proxy_busy_mod_done(wbuf.wbuf_persist) # returns nil
   end
 
+  def proxy_wait_next(qflags)
+    # We must allocate a new, empty request object here to avoid a TOCTTOU
+    # in the following timeline
+    #
+    # original thread:                                 | another thread
+    # HttpClient#yahns_step                            |
+    # r = k.app.call(env = @hs.env)  # socket hijacked into epoll queue
+    # <thread is scheduled away>                       | epoll_wait readiness
+    #                                                  | ReqRes#yahns_step
+    #                                                  | proxy dispatch ...
+    #                                                  | proxy_busy_mod_done
+    # ************************** DANGER BELOW ********************************
+    #                                                  | HttpClient#yahns_step
+    #                                                  | # clears env
+    # sees empty env:                                  |
+    # return :ignore if env.include?('rack.hijack_io') |
+    #
+    # In other words, we cannot touch the original env seen by the
+    # original thread since it must see the 'rack.hijack_io' value
+    # because both are operating in the same Yahns::HttpClient object.
+    # This will happen regardless of GVL existence
+    hs = Unicorn::HttpRequest.new
+    hs.buf.replace(@hs.buf)
+    @hs = hs
+    Thread.current[:yahns_queue].queue_mod(self, qflags)
+  end
+
   def proxy_busy_mod_done(alive)
-    q = Thread.current[:yahns_queue]
     case http_response_done(alive)
-    when :wait_readable then q.queue_mod(self, Yahns::Queue::QEV_RD)
-    when :wait_writable then q.queue_mod(self, Yahns::Queue::QEV_WR)
+    when :wait_readable then proxy_wait_next(Yahns::Queue::QEV_RD)
+    when :wait_writable then proxy_wait_next(Yahns::Queue::QEV_WR)
     when :close then Thread.current[:yahns_fdmap].sync_close(self)
     end
 
-- 
EW


^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [PATCH] proxy_pass: fix race condition due to flawed hijack check
  2015-04-21  8:49 [PATCH] proxy_pass: fix race condition due to flawed hijack check Eric Wong
@ 2015-04-21  9:14 ` Eric Wong
  0 siblings, 0 replies; 2+ messages in thread
From: Eric Wong @ 2015-04-21  9:14 UTC (permalink / raw)
  To: yahns-public

Pushed as commit 5328992829b2ff76cd7cda6d1911ecad70f0e8c2
additional burst test for pipelining and additional comment about
queue_mod being a hand-off point for the object hitting another thread:

+    # n.b. we may not touch anything in this object once we call queue_mod,
+    # another thread is likely to take it!
     Thread.current[:yahns_queue].queue_mod(self, qflags)
   end

diff --git a/test/test_proxy_pass.rb b/test/test_proxy_pass.rb
index 840ad1c..a27a45e 100644
--- a/test/test_proxy_pass.rb
+++ b/test/test_proxy_pass.rb
@@ -437,6 +437,16 @@ class TestProxyPass < Testcase
       end
       re = /^Date:[^\r\n]+/
       assert_equal after_up.sub(re, ''), r1.sub(re, '')
+
+      pl.write "GET / HTTP/1.1\r\nHost: example.com"
+      sleep 0.1 # hope epoll wakes up and reads in this time
+      pl.write "\r\n\r\nGET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
+      burst = pl.readpartial(666)
+      until burst.scan(/^hi$/).size == 2
+        burst << pl.readpartial(666)
+      end
+      assert_equal 2, burst.scan(/^hi$/).size
+      assert_match %r{\r\n\r\nhi\n\z}, burst
     end
     r1 = r1.split("\r\n").reject { |x| x =~ /^Date: / }
     r2 = r2.split("\r\n").reject { |x| x =~ /^Date: / }

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2015-04-21  9:14 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-21  8:49 [PATCH] proxy_pass: fix race condition due to flawed hijack check Eric Wong
2015-04-21  9:14 ` Eric Wong

Code repositories for project(s) associated with this inbox:

	../../../yahns.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).