[RFC] initial cut at OpenSSL support

[RFC] initial cut at OpenSSL support

[Eric Wong]

The current CA model and code quality of OpenSSL have long put me off
from supporting TLS; however but efforts such as "Let's Encrypt"
and the fallout from Heartbleed give me hope for the future.

This implements, as much as possible, a "hands-off" approach to TLS
support via OpenSSL.  This implementation allows us to shift
responsibility away from us to users and upstreams (the Ruby 'openssl'
extension maintainers, software packagers, and OpenSSL project itself).

This is also perhaps the easiest way for now for us, while being most
powerful for users.  It requires users to configure their own OpenSSL
context object which we'll use as-is.

This context object is used as the :ssl_ctx parameter to the "listen"
directive in the yahns configuration file:

	require 'openssl' # we will not do this for the user, even
        ctx = OpenSSL::SSL::SSLContext.new
	# user must configure ctx here...
	listen 443, ssl_ctx: ctx

This way, in case we support GnuTLS or other TLS libraries, there'll
be less confusion as to what a user is actually using.
---
 lib/yahns/config.rb         |  2 ++
 lib/yahns/openssl_client.rb | 49 +++++++++++++++++++++++++++++++++
 lib/yahns/openssl_server.rb | 21 +++++++++++++++
 lib/yahns/server.rb         | 15 ++++++-----
 lib/yahns/socket_helper.rb  | 17 +++++++++---
 test/test_ssl.rb            | 66 +++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 159 insertions(+), 11 deletions(-)
 create mode 100644 lib/yahns/openssl_client.rb
 create mode 100644 lib/yahns/openssl_server.rb
 create mode 100644 test/test_ssl.rb

diff --git a/lib/yahns/config.rb b/lib/yahns/config.rb
index f354961..e880d92 100644
--- a/lib/yahns/config.rb
+++ b/lib/yahns/config.rb
@@ -202,6 +202,8 @@ class Yahns::Config # :nodoc:
         raise ArgumentError, "#{var}: not boolean: #{key}=#{value.inspect}"
     end
 
+    require_relative('openssl_server') if options[:ssl_ctx]
+
     options[:yahns_app_ctx] = @block.ctx
     @config_listeners.include?(address) and
       raise ArgumentError, "listen #{address} already in use"
diff --git a/lib/yahns/openssl_client.rb b/lib/yahns/openssl_client.rb
new file mode 100644
index 0000000..2043628
--- /dev/null
+++ b/lib/yahns/openssl_client.rb
@@ -0,0 +1,49 @@
+# Copyright (C) 2014, all contributors <yahns-public@yhbt.net>
+# License: GPLv3 or later (see COPYING for details)
+
+require_relative 'sendfile_compat'
+
+# this is to be included into a Kgio::Socket-derived class
+module Yahns::OpenSSLClient # :nodoc:
+  include Yahns::SendfileCompat
+
+  def yahns_init_ssl(ssl_ctx)
+    @need_accept = true
+    @ssl = OpenSSL::SSL::SSLSocket.new(self, ssl_ctx)
+  end
+
+  def kgio_trywrite(buf)
+    rv = @ssl.write_nonblock(buf, exception: false)
+    Integer === rv and
+      rv = buf.bytesize == rv ? nil : buf.byteslice(rv, buf.bytesize)
+    rv
+  end
+
+  def kgio_syssend(buf, flags)
+    kgio_trywrite(buf)
+  end
+
+  def kgio_tryread(len, buf)
+    if @need_accept
+      # most protocols require read before write, so we start the negotiation
+      # process here:
+      begin
+        @ssl.accept_nonblock
+      rescue IO::WaitReadable
+        return :wait_readable
+      rescue IO::WaitWritable
+        return :wait_writable
+      end
+      @need_accept = false
+    end
+    @ssl.read_nonblock(len, buf, exception: false)
+  end
+
+  def shutdown(*args)
+    @ssl.shutdown(*args)
+  end
+
+  def close
+    @ssl.close
+  end
+end
diff --git a/lib/yahns/openssl_server.rb b/lib/yahns/openssl_server.rb
new file mode 100644
index 0000000..3940892
--- /dev/null
+++ b/lib/yahns/openssl_server.rb
@@ -0,0 +1,21 @@
+# Copyright (C) 2014, all contributors <yahns-public@yhbt.net>
+# License: GPLv3 or later (see COPYING for details)
+
+require_relative 'acceptor'
+require_relative 'openssl_client'
+
+class Yahns::OpenSSLServer < Kgio::TCPServer # :nodoc:
+  include Yahns::Acceptor
+
+  def self.wrap(fd, ssl_ctx)
+    srv = for_fd(fd)
+    srv.instance_variable_set(:@ssl_ctx, ssl_ctx)
+    srv
+  end
+
+  def kgio_accept(klass, flags)
+    io = super
+    io.yahns_init_ssl(@ssl_ctx)
+    io
+  end
+end
diff --git a/lib/yahns/server.rb b/lib/yahns/server.rb
index 1196d2d..e05a0e4 100644
--- a/lib/yahns/server.rb
+++ b/lib/yahns/server.rb
@@ -177,10 +177,9 @@ class Yahns::Server # :nodoc:
     tries = 5
 
     begin
-      io = bind_listen(address, sock_opts(address))
-      unless Yahns::TCPServer === io || Yahns::UNIXServer === io
-        io = server_cast(io)
-      end
+      opts = sock_opts(address)
+      io = bind_listen(address, opts)
+      io = server_cast(io, opts) unless io.class.name.start_with?('Yahns::')
       @logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
       @listeners << io
       io
@@ -298,7 +297,7 @@ class Yahns::Server # :nodoc:
   end
 
   def sock_opts(io)
-    @config.config_listeners[sock_name(io)]
+    @config.config_listeners[sock_name(io)] || {}
   end
 
   def inherit_listeners!
@@ -315,9 +314,10 @@ class Yahns::Server # :nodoc:
     # accept4(2).
     inherited = ENV['YAHNS_FD'].to_s.split(',').map! do |fd|
       io = Socket.for_fd(fd.to_i)
-      set_server_sockopt(io, sock_opts(io))
+      opts = sock_opts(io)
+      set_server_sockopt(io, opts)
       @logger.info "inherited addr=#{sock_name(io)} fd=#{fd}"
-      server_cast(io)
+      server_cast(io, opts)
     end
 
     @listeners.replace(inherited)
@@ -368,6 +368,7 @@ class Yahns::Server # :nodoc:
       ctx.queue = queues[qegg] ||= qegg_vivify(qegg, fdmap)
       ctx = ctx.dup
       ctx.__send__(:include, l.expire_mod)
+      ctx.__send__(:include, Yahns::OpenSSLClient) if opts[:ssl_ctx]
       ctx_list << ctx
       # acceptors feed the the queues
       l.spawn_acceptor(opts[:threads] || 1, @logger, ctx)
diff --git a/lib/yahns/socket_helper.rb b/lib/yahns/socket_helper.rb
index 6e1830f..66df8b0 100644
--- a/lib/yahns/socket_helper.rb
+++ b/lib/yahns/socket_helper.rb
@@ -16,7 +16,7 @@ module Yahns::SocketHelper # :nodoc:
   end
 
   def set_server_sockopt(sock, opt)
-    opt = {backlog: 1024}.merge!(opt || {})
+    opt = {backlog: 1024}.merge!(opt)
     sock.close_on_exec = true
 
     TCPSocket === sock and sock.setsockopt(:IPPROTO_TCP, :TCP_NODELAY, 1)
@@ -97,7 +97,12 @@ module Yahns::SocketHelper # :nodoc:
 
     sock.bind(Socket.pack_sockaddr_in(port, addr))
     sock.autoclose = false
-    Yahns::TCPServer.for_fd(sock.fileno)
+
+    if ssl_ctx = opt[:ssl_ctx]
+      Yahns::OpenSSLServer.wrap(sock.fileno, ssl_ctx)
+    else
+      Yahns::TCPServer.for_fd(sock.fileno)
+    end
   end
 
   # returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6
@@ -128,11 +133,15 @@ module Yahns::SocketHelper # :nodoc:
   end
 
   # casts a given Socket to be a TCPServer or UNIXServer
-  def server_cast(sock)
+  def server_cast(sock, opts)
     sock.autoclose = false
     begin
       Socket.unpack_sockaddr_in(sock.getsockname)
-      Yahns::TCPServer.for_fd(sock.fileno)
+      if ssl_ctx = opts[:ssl_ctx]
+        Yahns::OpenSSLServer.wrap(sock.fileno, ssl_ctx)
+      else
+        Yahns::TCPServer.for_fd(sock.fileno)
+      end
     rescue ArgumentError
       Yahns::UNIXServer.for_fd(sock.fileno)
     end
diff --git a/test/test_ssl.rb b/test/test_ssl.rb
new file mode 100644
index 0000000..890cf58
--- /dev/null
+++ b/test/test_ssl.rb
@@ -0,0 +1,66 @@
+# Copyright (C) 2014,  all contributors <yahns-public@yhbt.net>
+# License: GPLv3 or later (https://www.gnu.org/licenses/gpl-3.0.txt)
+require_relative 'server_helper'
+require 'openssl'
+class TestSSL < Testcase
+  ENV["N"].to_i > 1 and parallelize_me!
+  include ServerHelper
+
+  # copied from test/openssl/utils.rb in Ruby:
+
+  TEST_KEY_DH1024 = OpenSSL::PKey::DH.new <<-_end_of_pem_
+-----BEGIN DH PARAMETERS-----
+MIGHAoGBAKnKQ8MNK6nYZzLrrcuTsLxuiJGXoOO5gT+tljOTbHBuiktdMTITzIY0
+pFxIvjG05D7HoBZQfrR0c92NGWPkAiCkhQKB8JCbPVzwNLDy6DZ0pmofDKrEsYHG
+AQjjxMXhwULlmuR/K+WwlaZPiLIBYalLAZQ7ZbOPeVkJ8ePao0eLAgEC
+-----END DH PARAMETERS-----
+  _end_of_pem_
+
+  TEST_KEY_DH1024.priv_key = OpenSSL::BN.new("48561834C67E65FFD2A9B47F41" \
+     "E5E78FDC95C387428FDB1E4B0188B64D1643C3A8D3455B945B7E8C4D166010C7C2" \
+     "CE23BFB9BEF43D0348FE7FA5284B0225E7FE1537546D114E3D8A4411B9B9351AB4" \
+     "51E1A358F50ED61B1F00DA29336EEBBD649980AC86D76AF8BBB065298C2052672E" \
+     "EF3EF13AB47A15275FC2836F3AC74CEA", 16)
+
+  def setup
+    server_helper_setup
+  end
+
+  def teardown
+    server_helper_teardown
+  end
+
+  def ssl_client(host, port)
+    ctx = OpenSSL::SSL::SSLContext.new
+    ctx.ciphers = "ADH"
+    s = TCPSocket.new(host, port)
+    ssl = OpenSSL::SSL::SSLSocket.new(s, ctx)
+    ssl.connect
+    ssl.sync_close = true
+    ssl
+  end
+
+  def test_ssl_basic
+    err, cfg, host, port = @err, Yahns::Config.new, @srv.addr[3], @srv.addr[1]
+    host, port = @srv.addr[3], @srv.addr[1]
+    ctx = OpenSSL::SSL::SSLContext.new
+    ctx.ciphers = "ADH"
+    ctx.tmp_dh_callback = proc { TEST_KEY_DH1024 }
+
+    pid = mkserver(cfg) do
+      cfg.instance_eval do
+        ru = lambda { |_| [ 200, {'Content-Length'=>'2'}, ['HI'] ] }
+        app(:rack, ru) { listen "#{host}:#{port}", ssl_ctx: ctx }
+        logger(Logger.new(err.path))
+      end
+    end
+    client = ssl_client(host, port)
+    client.write("GET / HTTP/1.0\r\n\r\n")
+    head, body = client.read.split("\r\n\r\n", 2)
+    assert_equal "HI", body
+    assert_match %r{\AHTTP/1\.\d 200 OK\r\n}, head
+  ensure
+    client.close if client
+    quit_wait(pid)
+  end
+end if defined?(OpenSSL)
-- 
EW

Re: [RFC] initial cut at OpenSSL support

[Eric Wong]

I think the following needs to be squashed, since the server
isn't using sync_close=true

--- a/lib/yahns/openssl_client.rb
+++ b/lib/yahns/openssl_client.rb
@@ -41,9 +41,11 @@ module Yahns::OpenSSLClient # :nodoc:
 
   def shutdown(*args)
     @ssl.shutdown(*args)
+    super # BasicSocket#shutdown
   end
 
   def close
     @ssl.close
+    super # IO#close
   end
 end

Re: [RFC] initial cut at OpenSSL support

[Eric Wong]

Pushed OpenSSL support with a few (unrelated) minor fixes and cleanups

Eric Wong (4):
      extras/autoindex: simplify checking non-.gz
      Rakefile: kill more useless gsub use
      initial cut at OpenSSL support
      test/test_ssl: skip test if SSL on older Rubies

 Rakefile                    |  4 +--
 extras/autoindex.rb         |  4 ++-
 lib/yahns/config.rb         |  2 ++
 lib/yahns/openssl_client.rb | 52 +++++++++++++++++++++++++++++
 lib/yahns/openssl_server.rb | 21 ++++++++++++
 lib/yahns/server.rb         | 15 +++++----
 lib/yahns/socket_helper.rb  | 17 +++++++---
 test/server_helper.rb       |  6 ++--
 test/test_ssl.rb            | 79 +++++++++++++++++++++++++++++++++++++++++++++
 9 files changed, 183 insertions(+), 17 deletions(-)

commit 71aea810c6e15ba8af662698adbcb28be7e2c395
Author: Eric Wong <e@80x24.org>
Date:   Tue Dec 2 02:21:15 2014 +0000

    test/test_ssl: skip test if SSL on older Rubies
    
    We rely on exception-free non-blocking I/O for performance,
    so it is easier for us to avoid supporting new features on
    old Rubies.

commit 65a903181cd5cdd78b4df7eacc1c574f0ef8e95c
Author: Eric Wong <e@80x24.org>
Date:   Sat Nov 29 04:08:54 2014 +0000

    initial cut at OpenSSL support
    
    The current CA model and code quality of OpenSSL have long put me off
    from supporting TLS; however but efforts such as "Let's Encrypt"
    and the fallout from Heartbleed give me hope for the future.
    
    This implements, as much as possible, a "hands-off" approach to TLS
    support via OpenSSL.  This implementation allows us to shift
    responsibility away from us to users and upstreams (the Ruby 'openssl'
    extension maintainers, software packagers, and OpenSSL project itself).
    
    This is also perhaps the easiest way for now for us, while being most
    powerful for users.  It requires users to configure their own OpenSSL
    context object which we'll use as-is.
    
    This context object is used as the :ssl_ctx parameter to the "listen"
    directive in the yahns configuration file:
    
    	require 'openssl' # we will not do this for the user, even
            ctx = OpenSSL::SSL::SSLContext.new
    	# user must configure ctx here...
    	listen 443, ssl_ctx: ctx
    
    This way, in case we support GnuTLS or other TLS libraries, there'll
    be less confusion as to what a user is actually using.
    
    Note: this feature requires Ruby 2.1 and later for non-kgio
    {read,write}_nonblock(.. exception: false) support.

commit a1dba8aa91a533870c44ec0b695391f16be9a71f
Author: Eric Wong <e@80x24.org>
Date:   Tue Dec 2 01:56:31 2014 +0000

    Rakefile: kill more useless gsub use
    
    It's wrong to use gsub when stripping text in those cases.

commit 7a552121db4937f00d2b9a8586a475d02a8f833c
Author: Eric Wong <e@80x24.org>
Date:   Tue Dec 2 01:47:52 2014 +0000

    extras/autoindex: simplify checking non-.gz
    
    We only want to strip one ".gz" suffix to check for the
    original, so avoid a needless use of gsub! and use sub!
    instead.
    
    While we're at it, note the use of "dup.sub!" (vs plain "sub")
    to ensure we only handle files with a .gz suffix.