| Date | Commit message (Collapse) |
|---|
|
This release avoids confusing HTTP/1.1 clients with a
"Transfer-Encoding: chunked" header on bodyless responses.
The header was causing "curl -T" to wait indefinitely for
a response body after the server sent a 204.
This regression was introduced with autochunk introduced
with yahns 1.13(*).
rack.hijack was also broken for ancient "HTTP/0.9" requests,
and now fixed.
4 changes since 1.14.0:
queue_*: fix outdated comments
http_response: support rack.hijack on HTTP/0.9 responses
response: do not set chunked header on bodyless responses
proxy_pass: do not chunk on bodyless upstream responses
(*) https://yhbt.net/yahns-public/20160805-yahns-1.13.0-released@lucky13/
|
|
As with the previous commit
("response: do not set chunked header on bodyless responses"),
blindly setting "Transfer-Encoding: chunked" is wrong and
confuses "curl -T" on 204 responses, at least.
|
|
Setting "Transfer-Encoding: chunked" on responses will confuse
clients which see a 204 response and do not expect a body.
This follows Rack::Chunked behavior, as yahns should function
without Rack::Chunked middleware.
This regression appeared in yahns v1.13.0 (2016-08-05)
|
|
We still need to iterate through all response headers to support
response-only Rack hijacking. Previously, we only supported
full hijacking on so-called "HTTP/0.9" clients.
n.b. This diff will be easier to read with the
-b/--ignore-space-change option of git-diff(1) or GNU diff(1)
|
|
These comments have been outdated since 2013 with commit
cd84e2ccbdf2 ("ensure we stop all threads at exit")
|
|
There's minor feature removals for undocumented and
undefined features and behavior which are unlikely to
affect anybody unless they serve HTTPS.
Our website is now self-hosted with HTTPS support (HTTP remains
supported for legacy systems):
https://yhbt.net/yahns/README
See git history at git://yhbt.net/yahns.git for full details.
openssl_client: avoid undefined SSL_write behavior
move website to https://yhbt.net/yahns/
stream_file: remove #to_io support from responses
response: only stream "file" responses on known length
response: fixup compile error
req_res: do not send 502 on catchall error if response buffered
|
|
If we already started writing a response, we cannot be sending a
502 to inform a user a connection has failed. This should
prevent errors from the OpenSSL layer about mismatched buffers
due to the combination of slow clients and upstreams prematurely
aborting.
|
|
Introduced in commit 08cea89483e90c79daa2c8efe70da6bdeba71147
('response: only stream "file" responses on known length')
|
|
This means we need to check Content-Length and use it
properly (unless Content-Range is set).
|
|
This is not a part of any Rack specification, and can lead to
interesting bugs with wrapped classes for gzip and TLS. AFAIK,
.to_io was meant to support IO#wait_*able and IO.select methods,
not to actually perform read/writes on the return value.
|
|
HTTPS adds some level of privacy protection and helps marketing
(because we care soooo much about good marketing! :P).
Performance-wise, this reduces subjectAltName bloat when
negotiating connections and will also speed up occasional
certificate renewals when/if we drop the old name.
Also, not occupying the document root of a domain will make it
easier to add alternative site locations in the future, because
centralization sucks and I don't like the idea of anybody paying
ICANN or similar entities for domain names.
|
|
Sometimes apps may trigger zero-byte chunks in the response
body for whatever reason. We should maintain consistent
behavior with the rest of kgio; and Ruby OpenSSL::SSL::SSLSocket
should maintain consistent behavior with the core IO class:
https://bugs.ruby-lang.org/issues/12660
|
|
And probably a billion new regressions!
yahns now allows users to skip the Rack::Head, Rack::Chunked and
Rack::ContentLength middlewares to ease migrating from/to other
real-world Rack HTTP servers. Most notably, our chunked
encoding implementation is a bit faster than Rack::Chunked by
taking advantage of the writev(2) syscall:
https://yhbt.net/yahns-public/20160803031906.14553-4-e@80x24.org/
There's also rack 2.x fixes in the test case and extras/ section
(these incompatibilities did not affect existing users unless
they use the wonky extras/ section).
There's also some graceful shutdown fixes, the process title is
now changed to display the number of live FDs.
Of course, there's the usual round of documentation improvements
which are systemd and OpenSSL setup-related this time around.
However, the majority of changes (proxy_*, wbuf_lite), affect
currently-unadvertised functionality which is subject to removal
or incompatible config changes. However, they are used to serve
our mailing list archives at:
https://yhbt.net/yahns-public/
49 changes since yahns 1.12.5:
proxy_pass: simplify writing request bodies upstream
proxy_pass: hoist out proxy_res_headers method
proxy_pass: simplify proxy_http_response
proxy_pass: split out body and trailer reading in response
proxy_pass: trim down proxy_response_finish, too
proxy_pass: split out req_res into a separate file
proxy_pass: fix resumes after complete buffering is unblocked
proxy_pass: X-Forwarded-For appends to existing list
proxy_pass: pass entire object to proxy_http_response
proxy_pass: support "proxy_buffering: false"
proxy_pass: remove unnecessary rescue
req_res: store proxy_pass object here, instead
proxy_pass: redo "proxy_buffering: false"
wbuf: remove needless "busy" parameter
Merge branch 'maint'
extras/try_gzip_static: do not show backtrace on syscall errors
wbuf: remove tmpdir parameter
wbuf_lite: fix write retries for OpenSSL sockets
test_proxy_pass_no_buffering: fix racy test
queue_*: check for closed IO objects
cleanup graceful shutdown handling
proxy_pass: more descriptive error messages
proxy_pass: fix HTTP/1.0 backends on EOF w/o buffering
wbuf_common: reset offset counter when done
extras/try_gzip_static: resolve symlinks
test_ssl: remove unnecessary priv_key DH parameter
openssl_client: wrap shutdown for graceful termination
proxy_pass: keep trailer buffer on blocked client writes
proxy_pass: avoid TOCTTOU race when unbuffering, too
proxy_pass: avoid accessing logger in env after hijacking
proxy_pass: avoid stuck responses in "proxy_buffering: false"
extras: include status messages in responses
update init and add systemd examples
test_proxy_pass_no_buffering: exclude rb/ru files, too
wbuf_lite: use StringIO instead of TmpIO
wbuf_lite: truncate StringIO when done
wbuf_lite: prevent clobbering responses
wbuf_lite: unify EOF error handling
wbuf_lite: reset sf_offset/sf_count consistently
wbuf_lite: clear @busy flag when re-arming
http_response: drop bodies for non-compliant responses
fix rack 2.x compatibility bugs
doc: add session cache usage to OpenSSL example
test: skip some buffering tests on non-default values
response: drop clients after HTTP responses of unknown length
response: reduce stack overhead for parameter passing
response: support auto-chunking for HTTP/1.1
Revert "document Rack::Chunked/ContentLength semi-requirements"
extras/exec_cgi: fix for HTTPoxy vulnerability
|
|
Bad clients may set the Proxy: header in the response and
cause any CGI programs we execute to use the value of that
header as the HTTP proxy. This affects folks calling code
which respects the HTTP_PROXY environment variable in CGI
programs.
ref: https://httpoxy.org/
|
|
Actually, I guess I misread, rack (starting at) 1.0 stopped
requiring Content-Length/Chunked headers but I never noticed.
Oh well.
This reverts commit 4968041a7e1ff90b920704f50fccb9e7968d0d99.
|
|
We might as well do it since puma and thin both do(*),
and we can still do writev for now to get some speedups
by avoiding Rack::Chunked overhead.
timing runs of "curl --no-buffer http://127.0.0.1:9292/ >/dev/null"
results in a best case drop from ~260ms to ~205ms on one VM
by disabling Rack::Chunked in the below config.ru
$ ruby -I lib bin/yahns-rackup -E none config.ru
==> config.ru <==
class Body
STR = ' ' * 1024 * 16
def each
10000.times { yield STR }
end
end
use Rack::Chunked if ENV['RACK_CHUNKED']
run(lambda do |env|
[ 200, [ %w(Content-Type text/plain) ], Body.new ]
end)
(*) they can do Content-Length, but I don't think it's
worth the effort at the server level.
|
|
This makes it easier to add more parameters to
http_response_write and simplifies current callers.
|
|
Clients are not able to handle persistent connections unless
the client knows the length of the response.
|
|
It's too hard to reliably test output buffering behavior
with non-default values users sometimes set; so just skip
and warn about it for now.
ref: commit dad99b5ecd93cdf0a514ff9fb51d198f8aebb188
("test/test_proxy_pass: remove buffer size tuning")
|
|
Sometimes, one process is all you need :>
Fwiw, I am also experimenting with the following in my
yahns.conf.rb file:
scache_stats = lambda do |env|
s = ctx.session_cache_stats.inspect << "\n"
[200, [%W(Content-Length #{s.size}), %w(Content-Type text/plain)], [s]]
end
app(:rack, scache_stats, preload: true) do
listen "unix:/tmp/yahns-scache.#$$.sock"
end
Which allows me to get stats based on the master PID (not worker):
printf 'GET / HTTP/1.0\r\n\r\n' | socat - UNIX:/tmp/yahns-scache.$PID.sock
|
|
rack 2.x has some incompatible changes an deprecations; support
it but remain compatible with rack 1.x for the next few years.
|
|
Rack::Lint-compliant applications wouldn't have this problem;
but apparently public-facing Rack servers (webrick/puma/thin)
all implement this; so there is precedence for implementing
this in yahns itself.
|
|
This allows us to speed up subsequent calls to wbuf_write when
the client socket buffers are cleared. This doesn't affect
functionality outside of performance.
|
|
Since we use wbuf_lite for long, streaming requests, we need to
reset the offset and counts when aborting the existing wbuf and
not assume the wbuf goes out-of-scope and expires when we
are done using it. Fix stupid bugs in BUG notices while
we're at it :x
|
|
StringIO can never be truncated outside our control, so it is
a bug if we see EOF on trysendio, here.
|
|
All of our wbuf code assumes we append to existing buffers
(files) since sendfile cannot deal otherwise. We also
follow this pattern for StringIO to avoid extra data copies.
|
|
And explain why this is doable for StringIO and not TmpIO,
which is file-backed.
|
|
This allows us to work transparently with our OpenSSL
workaround[*] while allowing us to reuse our non-sendfile
compatibility code. Unfortunately, this means we duplicate a
lot of code from the normal wbuf code for now; but that should
be fairly stable at this point.
[*] https://bugs.ruby-lang.org/issues/12085
|
|
We may have temporary files lingering from concurrent
multi-threaded tests in our forked child since FD_CLOFORK
does not exist :P
|
|
Reduce raciness in the init script and add LSB tags.
However, the systemd examples should be race-free and
safer (if one feels safe using systemd :P)
|
|
This is mainly to benefit curl(1) users who forget to use '-f'
to show failures. Not sure if I want to keep this change, it
seems like bloat; but Rack::ShowStatus pages are totally
overkill...
|
|
Another critical bugfix for this yet-to-be-released feature.
By the time we call proxy_unbuffer in proxy_read_body,
the req_res socket may be completely drained of readable data
and a persistent-connection-capable backend will be waiting
for the next request (not knowing we do not yet support
persistent connections).
This affects both chunked and identity responses from the
upstream.
|
|
The HTTP state (@hs) object could be replaced in proxy_wait_next
causing @hs.env['rack.logger'] to become inaccessible.
|
|
proxy_unbuffer is vulnerable to the same race condition
we avoided in commit 5328992829b2
("proxy_pass: fix race condition due to flawed hijack check")
|
|
When breaking early out of the upstream response body loop, we
must ensure we know we've hit the parser body EOF and preserve
the buffer for trailer parsing. Otherwise, reentering
proxy_read_body will put us in a bad state and corrupt
responses.
This is a critical bugfix which only affects users of
the soon-to-be-released "proxy_buffering: false" feature
of proxy_pass.
|
|
When we call shutdown on bad upstream responses, FD limits,
or server termination, we need to ensure the TLS connection
is terminated properly by calling SSL_shutdown and avoiding
confusion on the client side (or violating TLS specs!).
|
|
It seems unnecessary to set it at all and it's deprecated
in current Ruby trunk.
|
|
Static gzip files may not exist for symlinks, but they
could resolve to a file for which a pre-gzipped file
exists.
|
|
This fixes a case where "proxy_buffering: false" users may
encounter a "upstream error: BUG: EOF on tmpio sf_offset="
as a wbuf may be reused.
Oddly, it took over one week of running the latest proxy_pass
as of commit 616e42c8d609905d9355bb5db726a5348303ffae
("proxy_pass: fix HTTP/1.0 backends on EOF w/o buffering")
|
|
We must ensure we properly close connections to HTTP/1.0
backends even if we blocked writing on outgoing data.
|
|
This should make it easier to figure out where certain
errors are coming from and perhaps fix problems with
upstreams, too.
This helped me track down the problem causing public-inbox WWW
component running under Perl v5.20.2 on my Debian jessie system
to break and drop connections going through
Plack::Middleware::Deflater with gzip:
https://public-inbox.org/meta/20160607071401.29325-1-e@80x24.org/
Perl 5.14.2 on Debian wheezy did not detect this problem :x
|
|
Using a 10ms tick was too little, use 100ms instead to avoid
burning CPU. Ideally, we would not tick at all during shutdown
(we're normally tickless); but the common case could be
slightly more expensive; and shutdowns are rare (I hope).
Then, change our process title to indicate we're shutting down,
and finally, cut down on repeated log spew during shutdown and
only log dropping changes.
This mean we could potentially take 90ms longer to notice when
we can do a graceful shutdown, but oh well...
While we're at it, add a test to ensure graceful shutdowns
work as intended with multiple processes.
|
|
Using a high max_events may mean some IO objects are closed
after they're retrieved from the kernel but before our Ruby
process has had a chance to get to them.
|
|
We can force output buffer files to a directory of our
choosing to avoid being confused by temporary files
from other tests polluting the process we care about.
|
|
OpenSSL can't handle write retries if we append to an existing
string. Thus we must preserve the same string object upon
retrying.
Do that by utilizing the underlying Wbuf class which could
already handles it transparently using trysendfile.
However, we still avoiding the subtlety of wbuf_close_common
reliance we previously used.
ref: commit 551e670281bea77e727a732ba94275265ccae5f6
("fix output buffering with SSL_write")
|
|
We can retrieve it when we actually need to create the
temporary file. This saves an ivar slot and method dispatch
parameters.
This patch is nice, unfortunately the patch which follows is
not :P
|
|
On ENAMETOOLONG and perhaps other system errors which we can do
nothing about, we should not spew a giant backtrace which could
be used as an easy DoS vector.
|
|
* maint:
yahns 1.12.5 - proxy_pass + rack.hijack fixes
proxy_pass: X-Forwarded-For appends to existing list
|
|
Hopefully the last of the 1.12.x series, this release
fixes a few minor bugs mainly needed for testing.
No upgrade should be necessary for non-proxy_pass users.
4 changes since v1.12.4 from the "maint" branch at
git://yhbt.net/yahns.git
http_client: set state to :ignore before hijack callback
test/test_client_expire: fix for high RLIMIT_NOFILE
proxy_pass: do not chunk HTTP/1.0 with keep-alive
proxy_pass: X-Forwarded-For appends to existing list
lib/yahns/http_client.rb | 6 +++---
lib/yahns/proxy_http_response.rb | 8 ++++++--
lib/yahns/proxy_pass.rb | 5 ++++-
test/test_client_expire.rb | 13 +++++++++++--
test/test_proxy_pass.rb | 10 ++++++++++
5 files changed, 34 insertions(+), 8 deletions(-)
Note: the current "master" branch (at commit 5e211ea003d2)
includes refactorings and new features not included in this
release.
|
|
Ugh, this is a little slower, but some people will want to
forward through multiple proxies.
|