* [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix
@ 2017-04-01 8:08 6% Eric Wong
0 siblings, 0 replies; 5+ results
From: Eric Wong @ 2017-04-01 8:08 UTC (permalink / raw)
To: ruby-talk, unicorn-public
Cc: Jeremy Evans, Simon Eskildsen, Dylan Thacker-Smith
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
unicorn 5.3.0
A couple of portability fixes from Dylan Thacker-Smith and
Jeremy Evans since 5.3.0.pre1 over a week ago, but this looks
ready for a stable release, today.
When I started this over 8 years ago, I wondered if this would
just end up being an April Fools' joke. Guess not. I guess I
somehow tricked people into using a terribly marketed web server
that cannot talk directly to untrusted clients :x Anyways,
unicorn won't be able to handle slow clients 8 years from now,
either, or 80 years from now. And I vow never to learn to use
new-fangled things like epoll, kqueue, or threads :P
Anyways, this is a largish release with several new features,
and no backwards incompatibilities.
Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170401&x=t
This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:
https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
Jeremy Evans contributed several new features. First he
implemented after_worker_exit to aid debugging:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170401&x=t#t
And then security-related features to isolate workers. Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170401&x=t#t
https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170401&x=t#t
Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170401&x=t#t
There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.
shortlog of changes since v5.2.0 (2016-10-31):
Dylan Thacker-Smith (1):
Check for Socket::TCP_INFO constant before trying to get TCP_INFO
Eric Wong (30):
drop rb_str_set_len compatibility replacement
TUNING: document THP caveat for Linux users
tee_input: simplify condition for IO#write
remove response_start_sent
http_request: freeze constant strings passed IO#write
Revert "remove response_start_sent"
t/t0012-reload-empty-config.sh: access ivars directly if needed
t0011-active-unix-socket.sh: fix race condition in test
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
unicorn_http: reduce rb_global_variable calls
oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
http_request: reduce insn size for check_client_connection
freebsd: avoid EINVAL when setting accept filter
test-lib: expr(1) portability fix
tests: keep disabled tests defined
test_exec: SO_KEEPALIVE value only needs to be true
doc: fix links to raindrops project
http_request: support proposed Raindrops::TCP states on non-Linux
ISSUES: expand on mail archive info + subscription disclaimer
test_ccc: use a pipe to synchronize test
doc: remove private email support address
input: update documentation and hide internals.
http_server: initialize @pid ivar
gemspec: remove olddoc from build dependency
doc: add version annotations for new features
unicorn 5.3.0.pre1
doc: note after_worker_exit is also 5.3.0+
test_exec: SO_KEEPALIVE value only needs to be true (take #2)
Jeremy Evans (7):
Add after_worker_exit configuration option
Fix code example in after_worker_exit documentation
Add support for chroot to Worker#user
Add after_worker_ready configuration option
Add worker_exec configuration option
Don't pass a block for fork when forking workers
Check for SocketError on first ccc attempt
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
--
Yes, this release is real despite the date.
^ permalink raw reply [relevance 6%]
* [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix
@ 2017-03-24 0:28 6% Eric Wong
0 siblings, 0 replies; 5+ results
From: Eric Wong @ 2017-03-24 0:28 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Jeremy Evans, Simon Eskildsen
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
This is a pre-release RubyGem intended for testing.
Changes:
unicorn 5.3.0.pre1
A largish release with several new features.
Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170324&x=t
This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:
https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
Jeremy Evans contributed several new features. First he
implemented after_worker_exit to aid debugging:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170324&x=t#t
And then security-related features to isolate workers. Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170324&x=t#t
https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170324&x=t#t
Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170324&x=t#t
There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.
34 changes since 5.2.0 (2016-10-31):
Eric Wong (27):
drop rb_str_set_len compatibility replacement
TUNING: document THP caveat for Linux users
tee_input: simplify condition for IO#write
remove response_start_sent
http_request: freeze constant strings passed IO#write
Revert "remove response_start_sent"
t/t0012-reload-empty-config.sh: access ivars directly if needed
t0011-active-unix-socket.sh: fix race condition in test
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
unicorn_http: reduce rb_global_variable calls
oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
http_request: reduce insn size for check_client_connection
freebsd: avoid EINVAL when setting accept filter
test-lib: expr(1) portability fix
tests: keep disabled tests defined
test_exec: SO_KEEPALIVE value only needs to be true
doc: fix links to raindrops project
http_request: support proposed Raindrops::TCP states on non-Linux
ISSUES: expand on mail archive info + subscription disclaimer
test_ccc: use a pipe to synchronize test
doc: remove private email support address
input: update documentation and hide internals.
http_server: initialize @pid ivar
gemspec: remove olddoc from build dependency
doc: add version annotations for new features
Jeremy Evans (6):
Add after_worker_exit configuration option
Fix code example in after_worker_exit documentation
Add support for chroot to Worker#user
Add after_worker_ready configuration option
Add worker_exec configuration option
Don't pass a block for fork when forking workers
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
--
5.3.0 in a week, maybe?
^ permalink raw reply [relevance 6%]
* Re: [PATCH 0/3] TCP_INFO check_client_connection followups
2017-03-08 6:02 7% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
2017-03-08 6:02 7% ` [PATCH 1/3] new test for check_client_connection Eric Wong
@ 2017-03-08 10:14 0% ` Simon Eskildsen
1 sibling, 0 replies; 5+ results
From: Simon Eskildsen @ 2017-03-08 10:14 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
This looks great Eric. Thanks for taking this to the finish line!
On Wed, Mar 8, 2017 at 1:02 AM, Eric Wong <e@80x24.org> wrote:
> This series goes on top of Simon's excellent work to get
> a TCP_INFO-based check_client_connection working under Linux.
>
> First off, add a test extracted from one of Simon's messages;
> then revert the signature changes to existing methods to
> avoid breaking tmm1/gctools, and finally add a more portable
> fallback for Ruby 2.2+ users (tested on FreeBSD).
>
> Further work will improve portability of raindrops for FreeBSD
> (and maybe other *BSDs incidentally, too). That will be sent to
> raindrops-public@bogomips => https://bogomips.org/raindrops-public/
>
> Eric Wong (3):
> new test for check_client_connection
> revert signature change to HttpServer#process_client
> support "struct tcp_info" on non-Linux and Ruby 2.2+
>
> lib/unicorn/http_request.rb | 42 +++++++++++++++++++----
> lib/unicorn/http_server.rb | 6 ++--
> lib/unicorn/oob_gc.rb | 4 +--
> lib/unicorn/socket_helper.rb | 16 +++++++--
> test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
> test/unit/test_request.rb | 28 +++++++--------
> 6 files changed, 150 insertions(+), 27 deletions(-)
> create mode 100644 test/unit/test_ccc.rb
>
> Also pushed to the ccc-tcp-v3 branch:
>
> The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:
>
> t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)
>
> are available in the git repository at:
>
> git://bogomips.org/unicorn ccc-tcp-v3
>
> for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:
>
> support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)
>
> ----------------------------------------------------------------
> Eric Wong (3):
> new test for check_client_connection
> revert signature change to HttpServer#process_client
> support "struct tcp_info" on non-Linux and Ruby 2.2+
>
> Simon Eskildsen (1):
> check_client_connection: use tcp state on linux
>
> lib/unicorn/http_request.rb | 73 ++++++++++++++++++++++++++++++++++++---
> lib/unicorn/socket_helper.rb | 16 +++++++--
> test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 163 insertions(+), 7 deletions(-)
> create mode 100644 test/unit/test_ccc.rb
>
> --
> EW
^ permalink raw reply [relevance 0%]
* [PATCH 1/3] new test for check_client_connection
2017-03-08 6:02 7% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
@ 2017-03-08 6:02 7% ` Eric Wong
2017-03-08 10:14 0% ` [PATCH 0/3] TCP_INFO check_client_connection followups Simon Eskildsen
1 sibling, 0 replies; 5+ results
From: Eric Wong @ 2017-03-08 6:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Simon Eskildsen
This was a bit tricky to test, but it's probably more reliable
now that we're relying on TCP_INFO.
Based on test by Simon Eskildsen <simon.eskildsen@shopify.com>:
https://bogomips.org/unicorn-public/CAO3HKM49+aLD=KLij3zhJqkWnR7bCWVan0mOvxD85xfrW8RXOw@mail.gmail.com/
---
test/unit/test_ccc.rb | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 81 insertions(+)
create mode 100644 test/unit/test_ccc.rb
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
new file mode 100644
index 0000000..22b1a9c
--- /dev/null
+++ b/test/unit/test_ccc.rb
@@ -0,0 +1,81 @@
+require 'socket'
+require 'unicorn'
+require 'io/wait'
+require 'tempfile'
+require 'test/unit'
+
+class TestCccTCPI < Test::Unit::TestCase
+ def test_ccc_tcpi
+ start_pid = $$
+ host = '127.0.0.1'
+ srv = TCPServer.new(host, 0)
+ port = srv.addr[1]
+ err = Tempfile.new('unicorn_ccc')
+ rd, wr = IO.pipe
+ pid = fork do
+ reqs = 0
+ rd.close
+ worker_pid = nil
+ app = lambda do |env|
+ worker_pid ||= begin
+ at_exit { wr.write(reqs.to_s) if worker_pid == $$ }
+ $$
+ end
+ reqs += 1
+ sleep(1) if env['PATH_INFO'] == '/sleep'
+ [ 200, [ %w(Content-Length 0), %w(Content-Type text/plain) ], [] ]
+ end
+ ENV['UNICORN_FD'] = srv.fileno.to_s
+ opts = {
+ listeners: [ "#{host}:#{port}" ],
+ stderr_path: err.path,
+ check_client_connection: true,
+ }
+ uni = Unicorn::HttpServer.new(app, opts)
+ uni.start.join
+ end
+ wr.close
+
+ # make sure the server is running, at least
+ client = TCPSocket.new(host, port)
+ client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
+ assert client.wait_readable(10), 'never got response from server'
+ res = client.read
+ assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first response'
+ assert_match %r{\r\n\r\n\z}, res, 'got end of response, server is ready'
+ client.close
+
+ # start a slow request...
+ sleeper = TCPSocket.new(host, port)
+ sleeper.write("GET /sleep HTTP/1.1\r\nHost: example.com\r\n\r\n")
+
+ # and a bunch of aborted ones
+ nr = 100
+ nr.times do |i|
+ client = TCPSocket.new(host, port)
+ client.write("GET /collections/#{rand(10000)} HTTP/1.1\r\n" \
+ "Host: example.com\r\n\r\n")
+ client.close
+ end
+ sleeper.close
+ kpid = pid
+ pid = nil
+ Process.kill(:QUIT, kpid)
+ _, status = Process.waitpid2(kpid)
+ assert status.success?
+ reqs = rd.read.to_i
+ warn "server got #{reqs} requests with #{nr} CCC aborted\n" if $DEBUG
+ assert_operator reqs, :<, nr
+ assert_operator reqs, :>=, 2, 'first 2 requests got through, at least'
+ ensure
+ return if start_pid != $$
+ srv.close if srv
+ if pid
+ Process.kill(:QUIT, pid)
+ _, status = Process.waitpid2(pid)
+ assert status.success?
+ end
+ err.close! if err
+ rd.close if rd
+ end
+end
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 0/3] TCP_INFO check_client_connection followups
@ 2017-03-08 6:02 7% Eric Wong
2017-03-08 6:02 7% ` [PATCH 1/3] new test for check_client_connection Eric Wong
2017-03-08 10:14 0% ` [PATCH 0/3] TCP_INFO check_client_connection followups Simon Eskildsen
0 siblings, 2 replies; 5+ results
From: Eric Wong @ 2017-03-08 6:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Simon Eskildsen
This series goes on top of Simon's excellent work to get
a TCP_INFO-based check_client_connection working under Linux.
First off, add a test extracted from one of Simon's messages;
then revert the signature changes to existing methods to
avoid breaking tmm1/gctools, and finally add a more portable
fallback for Ruby 2.2+ users (tested on FreeBSD).
Further work will improve portability of raindrops for FreeBSD
(and maybe other *BSDs incidentally, too). That will be sent to
raindrops-public@bogomips => https://bogomips.org/raindrops-public/
Eric Wong (3):
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
lib/unicorn/http_request.rb | 42 +++++++++++++++++++----
lib/unicorn/http_server.rb | 6 ++--
lib/unicorn/oob_gc.rb | 4 +--
lib/unicorn/socket_helper.rb | 16 +++++++--
test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
test/unit/test_request.rb | 28 +++++++--------
6 files changed, 150 insertions(+), 27 deletions(-)
create mode 100644 test/unit/test_ccc.rb
Also pushed to the ccc-tcp-v3 branch:
The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:
t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)
are available in the git repository at:
git://bogomips.org/unicorn ccc-tcp-v3
for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:
support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)
----------------------------------------------------------------
Eric Wong (3):
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
lib/unicorn/http_request.rb | 73 ++++++++++++++++++++++++++++++++++++---
lib/unicorn/socket_helper.rb | 16 +++++++--
test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 163 insertions(+), 7 deletions(-)
create mode 100644 test/unit/test_ccc.rb
--
EW
^ permalink raw reply [relevance 7%]
Results 1-5 of 5 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2017-03-08 6:02 7% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
2017-03-08 6:02 7% ` [PATCH 1/3] new test for check_client_connection Eric Wong
2017-03-08 10:14 0% ` [PATCH 0/3] TCP_INFO check_client_connection followups Simon Eskildsen
2017-03-24 0:28 6% [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix Eric Wong
2017-04-01 8:08 6% [ANN] unicorn 5.3.0 " Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).