| Date | Commit message (Collapse) |
|---|
|
Hopefully the last of the 1.12.x series, this release
fixes a few minor bugs mainly needed for testing.
No upgrade should be necessary for non-proxy_pass users.
4 changes since v1.12.4 from the "maint" branch at
git://yhbt.net/yahns.git
http_client: set state to :ignore before hijack callback
test/test_client_expire: fix for high RLIMIT_NOFILE
proxy_pass: do not chunk HTTP/1.0 with keep-alive
proxy_pass: X-Forwarded-For appends to existing list
lib/yahns/http_client.rb | 6 +++---
lib/yahns/proxy_http_response.rb | 8 ++++++--
lib/yahns/proxy_pass.rb | 5 ++++-
test/test_client_expire.rb | 13 +++++++++++--
test/test_proxy_pass.rb | 10 ++++++++++
5 files changed, 34 insertions(+), 8 deletions(-)
Note: the current "master" branch (at commit 5e211ea003d2)
includes refactorings and new features not included in this
release.
|
|
Ugh, this is a little slower, but some people will want to
forward through multiple proxies.
|
|
Instead, we must drop non-terminated responses since
HTTP/1.0 clients do not understand chunked encoding.
This is necessary for "ab -k" which still uses HTTP/1.0.
|
|
The ab(1) command we use for testing is limited to 20000 open
connections under Debian jessie; a perfectly reasonable limit
to avoid port exhaustion. I never noticed this limit before,
but systemd under Jessie seems to have upped the default
RLIMIT_NOFILE to 65536(!), causing ab to error out.
We don't even need 10K connections for testing,
we just need to hit *some* limit before we start expiring.
So lower the RLIMIT_NOFILE back to 1024 in the forked server
process so we can test more quickly without running out of
ports or memory, since exhausting the 65536 RLIMIT_NOFILE
limit is not going to happen with a single TCP address.
|
|
We need to set state as early as possible as any modification of our
HttpClient object is unsafe after it is handed over to the
underlying application. Otherwise, we could be clobbering a state
set inside the hijack but before we hit the case statement in
HttpClient#step_write.
This bug should not affect current (known) uses of rack.hijack;
but will be necessary for upcoming proxy_pass to support unbuffered
proxy responses.
|
|
This release fixes some resource leaks in uncommonly used parts
of yahns as well as including some documentation improvements.
No need to upgrade unless you rely on rack.hijack for responses
or use the (currently-undocumented) proxy_pass module(*).
9 non-merge changes since 1.12.3:
proxy_pass: honor wbuf_persist when ending response
proxy_http_response: fix non-terminated fast responses, too
test_proxy_pass: test for auto chunking on 1.0 backends
wbuf: drop persistence if writing to client fails
proxy_http_response: cleanup: avoid redundant setting of "alive"
proxy_http_response: do not persist upstream on slow clients
proxy_pass: drop resources immediately on errors
document Rack::Chunked/ContentLength semi-requirements
extras/exec_cgi: document cgit example
Documentation/yahns-rackup.pod | 10 ++++++++++
GIT-VERSION-GEN | 2 +-
examples/yahns_rack_basic.conf.rb | 6 ++++++
extras/exec_cgi.rb | 8 ++++++++
lib/yahns/proxy_http_response.rb | 40 ++++++++++++++++++++++-----------------
lib/yahns/proxy_pass.rb | 5 +++--
lib/yahns/wbuf_common.rb | 1 +
test/test_proxy_pass.rb | 15 +++++++++++++++
8 files changed, 67 insertions(+), 20 deletions(-)
(*) 1.13.0 will include refactoring in proxy_pass and possibly
documenting it as stable-enough-for-public use:
https://yhbt.net/yahns-public/20160220081619.GA10850@dcvr.yhbt.net/t/
|
|
* proxy_pass-fix:
proxy_pass: drop resources immediately on errors
proxy_http_response: do not persist upstream on slow clients
proxy_http_response: cleanup: avoid redundant setting of "alive"
wbuf: drop persistence if writing to client fails
test_proxy_pass: test for auto chunking on 1.0 backends
|
|
Apparently this can be useful to some people.
|
|
Ugh, it sucks that other servers are so tolerant of violations
of the Rack spec. Rainbows! had the same problem:
https://bogomips.org/rainbows-public/20140704195032.GA13152@dcvr.yhbt.net/
|
|
We don't want to wait on GC to reap sockets on errors,
generational GC in Ruby is less aggressive about reaping
long-lived objects such as long-lived HTTP connections.
|
|
For slow clients, we want to be able to drop the connection
to the upstream as soon as we are done buffering and not waste
resources by leaving it in an :ignore state. We also need to
remember the client for the fdmap to prevent shutdowns.
Ugh, this is really hard to test locally.
|
|
We already check for the truthiness of "alive" in the "if"
statement, so re-setting is pointless.
|
|
We cannot maintain a persistent connection to a client if
writing to the client fails; so we can't proceed to let
the app hijack the response.
This may happen in the unlikely case where a response
header needs to be buffered with a Wbuf (and the app
uses response hijacking).
|
|
These are followups to the following two commits:
* commit d16326723d
("proxy_http_response: fix non-terminated fast responses, too")
* commit 8c9f33a539
("proxy_http_response: workaround non-terminated backends")
|
|
Without this, non-terminated backends were not properly
supported if they gave tiny responses or responded faster
than we could stream the response to the client.
This is necessary to support fast responses from some non-Rack
HTTP/1.0-only backend servers which rely on connection
termination to terminate responses.
Tested manually with a Perl PSGI application running under
"plackup". Unlike Rack, the PSGI spec does not specify whether
the PSGI application or PSGI server should handle response
termination: git clone https://github.com/plack/psgi-specs.git
Follow-up-to: 8c9f33a5396d2 ("workaround non-terminated backends")
|
|
If a static file response gets truncated while writing, we
set wbuf_persist to false in wbuf_flush of lib/yahns/wbuf_common.rb
Thus, we must check wbuf_persist attribute as late as possible
before yielding control back to the caller in proxy_response_finish
|
|
This release only contains two changes since v1.12.2:
* proxy_http_response: workaround non-terminated backends
The first is a fix for the undocumented and unstable "proxy_pass"
feature:
https://yhbt.net/yahns-public/20160406062556.10988-1-e@80x24.org/t/
However, I'm considering supporting proxy_pass as a stable API:
https://yhbt.net/yahns-public/20160220081619.GA10850@dcvr.yhbt.net/t/
* doc: recommend "verify_mode: OpenSSL::SSL::VERIFY_NONE"
Only a documentation change prompted by the discovery that
some browsers/platforms will try to prompt users for client
certs:
https://yhbt.net/yahns-public/20160316003434.GA14791@dcvr.yhbt.net/t/
|
|
The Ruby default parameters on top of OpenSSL seem designed
for client usage. For server usage, requiring client-side
certificate verification is uncommon for HTTPS sites.
So follow what WEBrick does for HTTPS and use SSL_VERIFY_NONE
in our documentation.
Thanks-to: Shota Fukumori (sora_h) <her@sorah.jp>
on the unicorn list:
<CA+wiQwuE=ya6F4s4k3GCTUppk7mbBOYOVwVXhTsX2SP8mgdmNQ@mail.gmail.com>
|
|
Without this, we could only support persistent connections if
the backend gives a valid Content-Length or set
"Transfer-Encoding: chunked" in the response header.
Being good netizens, we want to use persistent connections as
much as possible if a remote client supports it; so perform
chunking ourselves when our remote clients are HTTP/1.1 and
able to decode chunked responses.
This is necessary to support some non-Rack HTTP/1.0-only
backend servers which rely on connection termination to
terminate responses.
Tested manually with a Perl PSGI application running under
"plackup". Unlike Rack, the PSGI spec does not specify whether
the PSGI application or PSGI server should handle response
termination: git clone https://github.com/plack/psgi-specs.git
|
|
This release ensures OpenSSL::SSL::SSLContext#session_id_context
is always set for OpenSSL users. It won't overwrite existing
settings, but setting it to a random value is necessary to
ensure clients do not get aborted connections when attempting to
use a session cache.
No need to actually upgrade if you're on 1.12.1, you may add the
following to your yahns_config(5) file where
OpenSSL::SSL::SSLContext is configured:
# recommended, not required. This sets safer defaults
# provided by Ruby on top of what OpenSSL gives:
ssl_ctx.set_params
# required, and done by default in v1.12.2:
ssl_ctx.session_id_context ||= OpenSSL::Random.random_bytes(32)
yahns gives you full control of of how OpenSSL::SSL::SSLContext is
configured. To avoid bugs, yahns only ensures
OpenSSL::SSL::SSLContext#session_id_context is set (if not previously
set by the user) and calls OpenSSL::SSL::SSLContext#setup before
spawning threads to avoid race conditions. yahns itself does not and
will not enforce any opinion on the compatibility/performance/security
trade-offs regarding TLS configuration.
Note: keep in mind using an SSL session cache may be less useful
with yahns because HTTP/1.1 persistent connections may live
forever :)
3 bug/doc fixes on top of v1.12.1:
document OpenSSL::SSL::SSLContext#set_params use
ssl: ensure is session_id_context is always set
test/*: fix mktmpdir usage for 1.9.3
|
|
We should not infinite loop, oops :x
Also, ensure 'yahns' is in the directory in case tests are
SIGKILL-ed and directories are left over.
|
|
When a client attempts to reuse a session, we must have a
session_id_context set or else handshakes fail. This problem
manifests only with clients which attempt to reuse stored
sessions. This is irrespective of any session caching
configured (even if explicitly disabled) in the server.
The SSL_CTX_set_session_id_context(3SSL) manpage states:
If the session id context is not set on an SSL/TLS server and
client certificates are used, stored sessions will not be reused
but a fatal error will be flagged and the handshake will fail.
|
|
I use whatever Ruby developers deem to be reasonable defaults.
Because compatibility with old systems is still valued, these
may not be the safest possible configuration; but ought to be
better than what OpenSSL upstream provides by default.
|
|
Most notably release fixes TLS output buffering for large
responses to slow clients. For Rack HTTPS users,
env['SERVER_PORT'] also defaults to 443 properly unless the
Host: request header specifies differently.
Also, the extras/autoindex change is to make our own directory
listing look nicer as we use Let's Encrypt and don't want to
waste space listing ".well-known/" directory contents on:
https://yahns.yhbt.net/
Yes, we really do care how our homepage looks!
6 changes since v1.12.1:
extras/autoindex: support hiding dotfiles
fix output buffering with SSL_write
https: ensure SERVER_PORT defaults to 443
test_ssl: check SERVER_PORT when parsed from Host: header
doc: mention kqueue/kevent alongside epoll
doc: more minor updates
|
|
Remove all pandoc references. We shouldn't need to clutter our
documentation with out-of-date references to pandoc, and pod2man is
probably widely-available enough that nobody should need to install
it.
Reduce HTTP redirects when linking to external sites.
It's also excessive to mention libkqueue as using the native
implementation (whether it be kqueue or epoll) is preferred
and easier.
|
|
epoll and kqueue are similar and we use them in a similar way;
so mention kqueue alongside epoll for users who may already be
familiar with kqueue on *BSD but not epoll under Linux.
epoll is a queue, too!
|
|
We need to ensure SERVER_PORT is still parsed from the Host:
header when it is given, there.
|
|
This helps Rack::Request#url and similar methods generate proper
URLs instead of the obviously wrong: "https://example.com:80/"
Note: we don't track the actual port the listener is bound to,
and it may not be worth it since the use of the Host: header
is long-established and Host: headers include the port number
if non-standard.
|
|
The underlying SSL_write called by the OpenSSL socket when
we use write_nonblock must get the same arguments after a
call returns SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE.
Ensure that by always passing a copy of the user-supplied
buffer to OpenSSL::SSL::SSLSocket#write_nonblock and retaining
our copy of the string internally as @ssl_blocked if we hit
EAGAIN on the socket.
String#dup is inexpensive in modern Ruby, as copying a
non-embedded string is implemented using copy-on-write.
We also prefer to use write_nonblock directly instead of
using our kgio-dependent sendfile emulation layer to avoid
allocating a new string on partial writes.
ref: https://bugs.ruby-lang.org/issues/12085
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/73882
http://mid.gmane.org/redmine.issue-12085.20160219020243.4b790a77f1cdd593@ruby-lang.org
|
|
Switch option initialization to using a keyword hash
since yet-another boolean is too much.
Using kwargs won't work under Ruby 1.9.3 which we still
support (for now).
Note: being a part of extras/, there's no API stability
guarantees but this should've maintained it.
|
|
Most notably, serving static files over HTTPS did not work
before this release with the "sendfile" gem installed. The
yahns_config(5) manpage is also updated with an example for
using OpenSSL::SSL::SSLContext objects. Users of
Rack::Request#scheme and env['rack.url_scheme'] should see
"https" properly set for HTTPS connections.
There's also a bunch of internal tweaks like taking advantage of
the file-level frozen_string_literal: directive in 2.3 and
explicitly clearing short-lived string buffers
TLS support is still in its early stages, but I'm experimenting
with Let's Encrypt (via getssl[1]) and hosting https://YHBT.net/
on it.
For now, I suggest using a separate yahns instance (with a
different master process) to avoid any potential data leaks
between HTTPS and HTTP instances. In the future, it may be
possible to isolate HTTPS from HTTP at the worker process level.
Supporting GnuTLS (alongside OpenSSL) may be in our future, too.
To paraphrase the warning in http://www.postfix.org/TLS_README.html
(which was written before Heartbleed):
WARNING
By turning on TLS support in yahns, you not only get the
ability to encrypt traffic and to authenticate remote
clients. You also turn on thousands and thousands of
lines of OpenSSL library code. Assuming that OpenSSL is
written as carefully as Eric's own code, every 1000 lines
introduce one additional bug into yahns.
I'm not nearly as careful with yahns as Wietse is with postfix,
either.
20 changes since v1.11.0:
README: updates for kqueue
add .gitattributes for Ruby method detection
nodoc internals
enable frozen_string_literal for Ruby 2.3+
copyright updates for 2016
extras/exec_cgi: fix frozen string error on slow responses
avoid StringIO#binmode for the next few years
use String#clear for short-lived buffers we create
gemspec: make rack a development dependency
build: install-gem forced to "--local" domain
acceptor: all subclasses of TCPServer use TCP_INFO
properly emulate sendfile for OpenSSL sockets
avoid race conditions in OpenSSL::SSL::SSLContext#setup
set HTTPS and rack.url_scheme in Rack env as appropriate
proxy_pass: pass X-Forwarded-Proto through
doc: switch to perlpod (from pandoc-flavored Markdown)
doc: trim down documentation slightly
doc: document ssl_ctx for "listen" directive
doc: various doc and linkification improvements
http_context: reduce constant lookup + bytecode
[1] git clone https://github.com/srvrco/getssl.git
|
|
This saves about 100 bytes of iseq overhead based
on my measurements.
|
|
Correctly link to subsections within the same page, and include
a link to mailing list archives.
Also, use "ssl_ctx" consistently as a local variable as
we internally use "ctx" for other purposes.
|
|
With the advent of Let's Encrypt, we'll see more users
interested in using yahns with OpenSSL support.
So document how a listener may be passed an SSLContext.
|
|
The "threads:" option for the "listen" directive is worthless.
Having a dedicated thread per-process is already more than enough
(and ideal) for a multi-process setup. Multiple acceptor threads
is still wrong for a single-process setup (even if we did not
have a GVL) as it still incurs contention with the worker
pool within the kernel.
So remove the documentation regarding "listen ... threads: ",
for now; at least until somebody can prove it's useful and not
taking up space.
Additionally, "atfork_parent" may be useful for restarting
background threads/connections if somebody wants to run
background jobs in the master process, so stop saying
it's completely useless.
|
|
pod2man(1) and pod2text(1) are already installed on most modern
GNU/Linix systems including Debian and RedHat-based systems;
pandoc(1) and Haskell are not, and we do not wish to waste
precious bandwidth and disk space of potential packagers.
perlpod(1) is also better standardized than any Markdown flavor,
especially when it comes to generating manpages.
Finally, I'm mildly proficient at Perl (it is similar to Ruby)
and can poke around at the source if I encounter breakage.
|
|
This allows backend application servers to set "rack.url_scheme"
as appropriate using Rack::Request#scheme.
Plack/PSGI users can also take advantage of this using
Plack::Middleware::ReverseProxy
|
|
env['HTTPS'] is not documented in rack SPEC, but appears to be
used by Rack::Request since 2010[*]. Also, set rack.url_scheme
as documented by rack SPEC.
[*] - commit 4defbe5d7c07b3ba721ff34a8ff59fde480a4a9f
("Improves performance by lazy loading the session.")
|
|
By explicitly calling OpenSSL::SSL::SSLContext#setup before
accepting connections. We cannot rely on "setup" being called
implicitly because any callbacks configured or objects
configured by the client may not be thread-safe.
We also avoid calling "setup" in the master process (if yahns is
configured to use worker processeses) in case the setup code
starts any TCP connections (e.g. to memcached for session
caching).
|
|
We cannot use the sendfile(2) syscall when serving static files
to TLS clients without breaking them. We currently rely on
OpenSSL to encrypt the data before it hits the socket, so it
must be read into userspace buffers before being written to the
socket.
|
|
This will allow Yahns::OpenSSLServer instances to take advantage
of TCP_INFO under Linux, saving us the overhead of method
invocations.
|
|
This avoids needless network traffic when installing
the locally-built gem.
|
|
We don't depend on rack directly, and unicorn 5.1 will make rack
optional. This seems reasonable for testing, but one day I could
imagine this being more than an HTTP or Rack server...
|
|
This should reduce memory pressure slightly as we can
have finer-grained control of memory usage for buffers which
can be several kilobytes large.
It is not safe to do this for output buffers we get from the
application, as they may reuse that memory themselves.
|
|
Apparently, StringIO#binmode has been totally broken in 1.9+ and
I've always hidden this bug with the combination of an explicit
string and magic "encoding: binary" comments :x
ref: https://bugs.ruby-lang.org/issues/11945
|
|
Oops, we need to duplicate our buffer in case the CGI executable
returns just the header :x
|
|
Using the 'update-copyright' script from gnulib[1]:
git ls-files | UPDATE_COPYRIGHT_HOLDER='all contributors' \
UPDATE_COPYRIGHT_USE_INTERVALS=2 \
xargs /path/to/gnulib/build-aux/update-copyright
We're also switching to 'GPL-3.0+' as recommended by SPDX
to be consistent with our gemspec and other metadata
(as opposed to the longer but equivalent "GPLv3 or later").
[1] git://git.savannah.gnu.org/gnulib.git
|
|
There are likely yet-to-be-discovered bugs in here.
Also, keeping explicit #freeze calls for 2.2 users, since most
users have not migrated to 2.3, yet.
|
|
We do not expose any sort of API beyond what's in the config file
manpage to our users. Do not mislead them into thinking we
currently have a stable API (though I'm considering one).
This avoids wasting disk space and installation time for users who
do not have a: "gem: --no-ri --no-rdoc" line in their ~/.gemrc
|
|
The "diff" function detection for C does not map well to
Ruby files, take advantage of gitattributes(5) to improve
method name detection in generated patches as well as
making "git diff -W" output more useful.
|