* [PATCH 0/5..5/5] more tests to Perl 5 for stability
@ 2024-05-06 20:10 3% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2024-05-06 20:10 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]
It's far easier to maintain tests in a language that's been
"dead" for 20 years :P This is another step towards freeing us
up to make more internal changes, too; as well as avoiding slow
Ruby startup overhead. (Perl 5 startup is slow, too, but not
nearly as slow as Ruby)
Eric Wong (5):
GNUmakefile: build writes shebang-modified files
t/*.t: use write_file helper function
tests: port broken-app test to Perl 5
tests: move test/unit/test_request.rb to Perl 5
port test/unit/test_ccc.rb to Perl 5
GNUmakefile | 1 +
t/active-unix-socket.t | 13 +--
t/broken-app.ru | 13 ---
t/client_body_buffer_size.t | 5 +-
t/heartbeat-timeout.t | 4 +-
t/integration.ru | 11 +++
t/integration.t | 128 +++++++++++++++++++++++++--
t/lib.perl | 2 +-
t/reload-bad-config.t | 17 ++--
t/reopen-logs.t | 5 +-
t/t0009-broken-app.sh | 56 ------------
t/winch_ttin.t | 7 +-
t/working_directory.t | 16 +---
test/unit/test_ccc.rb | 92 -------------------
test/unit/test_request.rb | 170 ------------------------------------
15 files changed, 153 insertions(+), 387 deletions(-)
delete mode 100644 t/broken-app.ru
delete mode 100755 t/t0009-broken-app.sh
delete mode 100644 test/unit/test_ccc.rb
delete mode 100644 test/unit/test_request.rb
[-- Attachment #2: 0001-GNUmakefile-build-writes-shebang-modified-files.patch --]
[-- Type: text/x-diff, Size: 772 bytes --]
From 5e9dbfd071aa939677aaf3d269115fb88e606311 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:35 +0000
Subject: [PATCH 1/5] GNUmakefile: build writes shebang-modified files
This makes it easier to run individual integration tests via
prove(1) rather than all at once with gmake(1).
---
GNUmakefile | 1 +
1 file changed, 1 insertion(+)
diff --git a/GNUmakefile b/GNUmakefile
index 70e7e10..227842c 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -78,6 +78,7 @@ man1_bins := $(addsuffix .1, $(base_bins))
man1_paths := $(addprefix man/man1/, $(man1_bins))
tmp_bins = $(addprefix $(tmp_bin)/, unicorn unicorn_rails)
pid := $(shell echo $$PPID)
+build: $(tmp_bins)
$(tmp_bin)/%: bin/% | $(tmp_bin)
$(INSTALL) -m 755 $< $@.$(pid)
[-- Attachment #3: 0002-t-.t-use-write_file-helper-function.patch --]
[-- Type: text/x-diff, Size: 8088 bytes --]
From 9cbf87fd110acc36c3b6eec14231aed3be78ecf4 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:36 +0000
Subject: [PATCH 2/5] t/*.t: use write_file helper function
This shortens the tests a bit for readability.
---
t/active-unix-socket.t | 13 +++----------
t/client_body_buffer_size.t | 5 ++---
t/heartbeat-timeout.t | 4 +---
t/integration.t | 5 ++---
t/reload-bad-config.t | 17 ++++++-----------
t/reopen-logs.t | 5 +----
t/winch_ttin.t | 7 ++-----
t/working_directory.t | 16 ++++------------
8 files changed, 21 insertions(+), 51 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index ff731b5..ab3c973 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -11,29 +11,22 @@ END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
{
- open my $fh, '>', "$tmpdir/u1.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u1.conf.rb", <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
stderr_path "$err_log"
EOM
- close $fh;
-
- open $fh, '>', "$tmpdir/u2.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u2.conf.rb", <<EOM;
pid "$tmpdir/u.pid"
listen "$u2"
stderr_path "$tmpdir/err2.log"
EOM
- close $fh;
- open $fh, '>', "$tmpdir/u3.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u3.conf.rb", <<EOM;
pid "$tmpdir/u3.pid"
listen "$u1"
stderr_path "$tmpdir/err3.log"
EOM
- close $fh;
}
my @uarg = qw(-D -E none t/integration.ru);
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index d479901..c8e871d 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -4,11 +4,10 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-open my $conf_fh, '>', $u_conf;
-$conf_fh->autoflush(1);
-print $conf_fh <<EOM;
+my $conf_fh = write_file '>', $u_conf, <<EOM;
client_body_buffer_size 0
EOM
+$conf_fh->autoflush(1);
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf);
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index 694867a..0ae0764 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -6,14 +6,12 @@ use autodie;
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
mkdir "$tmpdir/alt";
my $srv = tcp_server();
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$tmpdir/pid"
preload_app true
stderr_path "$err_log"
timeout 3 # WORST FEATURE EVER
EOM
-close $fh;
my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
diff --git a/t/integration.t b/t/integration.t
index d17ace0..93480fa 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -18,13 +18,12 @@ if ('ensure Perl does not set SO_KEEPALIVE by default') {
$val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
}
my $t0 = time;
-open my $conf_fh, '>', $u_conf;
-$conf_fh->autoflush(1);
my $u1 = "$tmpdir/u1";
-print $conf_fh <<EOM;
+my $conf_fh = write_file '>', $u_conf, <<EOM;
early_hints true
listen "$u1"
EOM
+$conf_fh->autoflush(1);
my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
local $ENV{NO_PROXY} = '*'; # for curl
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index c023b88..4c17968 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -6,32 +6,27 @@ use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $ru = "$tmpdir/config.ru";
-my $u_conf = "$tmpdir/u.conf.rb";
-open my $fh, '>', $ru;
-print $fh <<'EOM';
+write_file '>', $ru, <<'EOM';
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
config = ru = "hello world\n" # check for config variable conflicts, too
run lambda { |env| [ 200, {}, [ ru.to_s ] ] }
EOM
-close $fh;
-open $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
preload_app true
stderr_path "$err_log"
EOM
-close $fh;
my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
is($bdy, "hello world\n", 'body matches expected');
-open $fh, '>>', $ru;
-say $fh '....this better be a syntax error in any version of ruby...';
-close $fh;
+write_file '>>', $ru, <<'EOM';
+....this better be a syntax error in any version of ruby...
+EOM
$ar->do_kill('HUP'); # reload
my @l;
@@ -42,7 +37,7 @@ for (1..1000) {
}
diag slurp($err_log) if $ENV{V};
ok(grep(/error reloading/, @l), 'got error reloading');
-open $fh, '>', $err_log;
+open my $fh, '>', $err_log; # truncate
close $fh;
($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index 76a4dbd..14bc6ef 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -4,14 +4,11 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
my $srv = tcp_server();
-my $u_conf = "$tmpdir/u.conf.rb";
my $out_log = "$tmpdir/out.log";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
stderr_path "$err_log"
stdout_path "$out_log"
EOM
-close $fh;
my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index c507959..3a3d430 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -4,13 +4,11 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
use POSIX qw(mkfifo);
-my $u_conf = "$tmpdir/u.conf.rb";
my $u_sock = "$tmpdir/u.sock";
my $fifo = "$tmpdir/fifo";
mkfifo($fifo, 0666) or die "mkfifo($fifo): $!";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$tmpdir/pid"
listen "$u_sock"
stderr_path "$err_log"
@@ -19,11 +17,10 @@ after_fork do |server, worker|
File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s }
end
EOM
-close $fh;
unicorn('-D', '-c', $u_conf, 't/integration.ru')->join;
is($?, 0, 'daemonized properly');
-open $fh, '<', "$tmpdir/pid";
+open my $fh, '<', "$tmpdir/pid";
chomp(my $pid = <$fh>);
ok(kill(0, $pid), 'daemonized PID works');
my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef };
diff --git a/t/working_directory.t b/t/working_directory.t
index f9254eb..f1c0a35 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -5,15 +5,13 @@ use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
mkdir "$tmpdir/alt";
my $ru = "$tmpdir/alt/config.ru";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$pid_file"
preload_app true
stderr_path "$err_log"
working_directory "$tmpdir/alt" # the whole point of this test
before_fork { |_,_| \$master_ppid = Process.ppid }
EOM
-close $fh;
my $common_ru = <<'EOM';
use Rack::ContentLength
@@ -21,12 +19,10 @@ use Rack::ContentType, 'text/plain'
run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] }
EOM
-open $fh, '>', $ru;
-print $fh <<EOM;
+write_file '>', $ru, <<EOM;
#\\--daemonize --listen $u_sock
$common_ru
EOM
-close $fh;
unicorn('-c', $u_conf)->join; # will daemonize
chomp($daemon_pid = slurp($pid_file));
@@ -39,9 +35,7 @@ check_stderr;
if ('test without CLI switches in config.ru') {
truncate $err_log, 0;
- open $fh, '>', $ru;
- print $fh $common_ru;
- close $fh;
+ write_file '>', $ru, $common_ru;
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
chomp($daemon_pid = slurp($pid_file));
@@ -68,8 +62,7 @@ if ('ensures broken working_directory (missing config.ru) is OK') {
if ('fooapp.rb (not config.ru) works with working_directory') {
truncate $err_log, 0;
my $fooapp = "$tmpdir/alt/fooapp.rb";
- open $fh, '>', $fooapp;
- print $fh <<EOM;
+ write_file '>', $fooapp, <<EOM;
class Fooapp
def self.call(env)
b = "dir=#{Dir.pwd}"
@@ -78,7 +71,6 @@ class Fooapp
end
end
EOM
- close $fh;
my $srv = tcp_server;
my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
{ -C => '/', 3 => $srv });
[-- Attachment #4: 0003-tests-port-broken-app-test-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4208 bytes --]
From e61a1152a613032927613b805a46c4d831bad00c Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:37 +0000
Subject: [PATCH 3/5] tests: port broken-app test to Perl 5
Save some inodes and startup time by folding it into the
integration test.
---
t/broken-app.ru | 13 ----------
t/integration.ru | 1 +
t/integration.t | 18 +++++++++++++-
t/lib.perl | 2 +-
t/t0009-broken-app.sh | 56 -------------------------------------------
5 files changed, 19 insertions(+), 71 deletions(-)
delete mode 100644 t/broken-app.ru
delete mode 100755 t/t0009-broken-app.sh
diff --git a/t/broken-app.ru b/t/broken-app.ru
deleted file mode 100644
index 5966bff..0000000
--- a/t/broken-app.ru
+++ /dev/null
@@ -1,13 +0,0 @@
-# frozen_string_literal: false
-# we do not want Rack::Lint or anything to protect us
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
-map "/" do
- run lambda { |env| [ 200, {}, [ "OK\n" ] ] }
-end
-map "/raise" do
- run lambda { |env| raise "BAD" }
-end
-map "/nil" do
- run lambda { |env| nil }
-end
diff --git a/t/integration.ru b/t/integration.ru
index 6df481c..3a0d99c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -100,6 +100,7 @@ def rack_input_tests(env)
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
when '/broken_app'; raise RuntimeError, 'hello'
+ when '/nil'; nil
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 93480fa..c9a7877 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -122,7 +122,23 @@ check_stderr;
($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0');
like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint');
is($bdy, undef, 'no response body after exception');
-truncate($errfh, 0);
+seek $errfh, 0, SEEK_SET;
+{
+ my $nxt;
+ while (!defined($nxt) && defined($_ = <$errfh>)) {
+ $nxt = <$errfh> if /app error/;
+ }
+ ok $nxt, 'got app error' and
+ like $nxt, qr/\bintegration\.ru/, 'got backtrace';
+}
+seek $errfh, 0, SEEK_SET;
+truncate $errfh, 0;
+
+($status, $hdr, $bdy) = do_req($srv, 'GET /nil HTTP/1.0');
+like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on nil endpoint');
+like slurp($err_log), qr/app error/, 'exception logged for nil';
+seek $errfh, 0, SEEK_SET;
+truncate $errfh, 0;
my $ck_early_hints = sub {
my ($note) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index 8c842b1..382f08c 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -30,7 +30,7 @@ $pid_file = "$tmpdir/pid";
$fifo = "$tmpdir/fifo";
$u_sock = "$tmpdir/u.sock";
$u_conf = "$tmpdir/u.conf.rb";
-open($errfh, '>>', $err_log);
+open($errfh, '+>>', $err_log);
if (my $t = $ENV{TAIL}) {
my @tail = $t =~ /tail/ ? split(/\s+/, $t) : (qw(tail -F));
diff --git a/t/t0009-broken-app.sh b/t/t0009-broken-app.sh
deleted file mode 100755
index 895b178..0000000
--- a/t/t0009-broken-app.sh
+++ /dev/null
@@ -1,56 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-
-t_plan 9 "graceful handling of broken apps"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -E none -D broken-app.ru -c $unicorn_config
- unicorn_wait_start
-}
-
-t_begin "normal response is alright" && {
- test xOK = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "app raised exception" && {
- curl -sSf http://$listen/raise 2> $tmp || :
- grep -F 500 $tmp
- > $tmp
-}
-
-t_begin "app exception logged and backtrace not swallowed" && {
- grep -F 'app error' $r_err
- grep -A1 -F 'app error' $r_err | tail -1 | grep broken-app.ru:
- dbgcat r_err
- > $r_err
-}
-
-t_begin "trigger bad response" && {
- curl -sSf http://$listen/nil 2> $tmp || :
- grep -F 500 $tmp
- > $tmp
-}
-
-t_begin "app exception logged" && {
- grep -F 'app error' $r_err
- > $r_err
-}
-
-t_begin "normal responses alright afterwards" && {
- > $tmp
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- wait
- test xOK = x$(sort < $tmp | uniq)
-}
-
-t_begin "teardown" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && check_stderr
-
-t_done
[-- Attachment #5: 0004-tests-move-test-unit-test_request.rb-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11298 bytes --]
From 01224642a20e91de5ea18c6f20856142158068a8 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:38 +0000
Subject: [PATCH 4/5] tests: move test/unit/test_request.rb to Perl 5
Another step towards having more freedom to change our internals
and having a more stable language for tests to reduce
maintenance overhead by avoiding Ruby incompatibilities.
---
t/integration.ru | 6 ++
t/integration.t | 72 +++++++++++++++-
test/unit/test_request.rb | 170 --------------------------------------
3 files changed, 75 insertions(+), 173 deletions(-)
delete mode 100644 test/unit/test_request.rb
diff --git a/t/integration.ru b/t/integration.ru
index 3a0d99c..a6b022a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,7 @@ def env_dump(env)
else
case k
when 'rack.version', 'rack.after_reply'; h[k] = v
+ when 'rack.input'; h[k] = v.class.to_s
end
end
end
@@ -112,6 +113,11 @@ def rack_input_tests(env)
when 'PUT'
case env['PATH_INFO']
when %r{\A/rack_input}; rack_input_tests(env)
+ when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
+ end
+ when 'OPTIONS'
+ case env['REQUEST_URI']
+ when '*'; [ 200, {}, [ env_dump(env) ] ]
end
end # case REQUEST_METHOD
end) # run
diff --git a/t/integration.t b/t/integration.t
index c9a7877..3b1d6df 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -103,14 +103,75 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- ($status, $hdr, my $json) = do_req $srv, 'GET /env_dump';
+ my $get_json = sub {
+ my (@req) = @_;
+ my @r = do_req $srv, @req;
+ my $env = eval { JSON::PP->new->decode($r[2]) };
+ diag "$@ (r[2]=$r[2])" if $@;
+ is ref($env), 'HASH', "@req response body is JSON";
+ (@r, $env)
+ };
+ ($status, $hdr, my $json, my $env) = $get_json->('GET /env_dump');
is($status, undef, 'no status for HTTP/0.9');
is($hdr, undef, 'no header for HTTP/0.9');
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
- my $env = JSON::PP->new->decode($json);
- is(ref($env), 'HASH', 'JSON decoded body to hashref');
is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+ is $env->{'rack.url_scheme'}, 'http', 'rack.url_scheme default';
+ is $env->{'rack.input'}, 'StringIO', 'StringIO for no content';
+
+ my $req = 'OPTIONS *';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '', "$req => PATH_INFO";
+ is $env->{REQUEST_URI}, '*', "$req => REQUEST_URI";
+
+ $req = 'GET http://e:3/env_dump?y=z';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, 'y=z', "$req => QUERY_STRING";
+
+ $req = 'GET http://e:3/env_dump#frag';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, '', "$req => QUERY_STRING";
+ is $env->{FRAGMENT}, 'frag', "$req => FRAGMENT";
+
+ $req = 'GET http://e:3/env_dump?a=b#frag';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, 'a=b', "$req => QUERY_STRING";
+ is $env->{FRAGMENT}, 'frag', "$req => FRAGMENT";
+
+ for my $proto (qw(https http)) {
+ $req = "X-Forwarded-Proto: $proto";
+ ($status, $hdr, $json, $env) = $get_json->(
+ "GET /env_dump HTTP/1.0\r\n".
+ "X-Forwarded-Proto: $proto");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{'rack.url_scheme'}, $proto, "$req => rack.url_scheme";
+ }
+
+ $req = 'X-Forwarded-Proto: ftp'; # invalid proto
+ ($status, $hdr, $json, $env) = $get_json->(
+ "GET /env_dump HTTP/1.0\r\n".
+ "X-Forwarded-Proto: ftp");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{'rack.url_scheme'}, 'http', "$req => rack.url_scheme";
+
+ ($status, $hdr, $json, $env) = $get_json->("PUT /env_dump HTTP/1.0\r\n".
+ 'Content-Length: 0');
+ is $env->{'rack.input'}, 'StringIO', 'content-length: 0 uses StringIO';
+
+ ($status, $hdr, $json, $env) = $get_json->("PUT /env_dump HTTP/1.0\r\n".
+ 'Content-Length: 1');
+ is $env->{'rack.input'}, 'Unicorn::TeeInput',
+ 'content-length: 1 uses TeeInput';
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
@@ -179,6 +240,11 @@ if ('bad requests') {
($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1';
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+ for my $abs_uri (qw(ssh+http://e/ ftp://e/x http+ssh://e/x)) {
+ ($status, $hdr) = do_req $srv, "GET $abs_uri HTTP/1.0";
+ like $status, qr!\AHTTP/1\.[01] 400 \b!, "400 on $abs_uri";
+ }
+
$c = tcp_start($srv);
print $c 'GET /';
my $buf = join('', (0..9), 'ab');
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
deleted file mode 100644
index 9d1b350..0000000
--- a/test/unit/test_request.rb
+++ /dev/null
@@ -1,170 +0,0 @@
-# -*- encoding: binary -*-
-# frozen_string_literal: false
-
-# Copyright (c) 2009 Eric Wong
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-
-require './test/test_helper'
-
-include Unicorn
-
-class RequestTest < Test::Unit::TestCase
-
- MockRequest = Class.new(StringIO)
-
- AI = Addrinfo.new(Socket.sockaddr_un('/unicorn/sucks'))
-
- def setup
- @request = HttpRequest.new
- @app = lambda do |env|
- [ 200, { 'content-length' => '0', 'content-type' => 'text/plain' }, [] ]
- end
- @lint = Rack::Lint.new(@app)
- end
-
- def test_options
- client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '', env['REQUEST_PATH']
- assert_equal '', env['PATH_INFO']
- assert_equal '*', env['REQUEST_URI']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_query
- client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal 'y=z', env['QUERY_STRING']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_fragment
- client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal '', env['QUERY_STRING']
- assert_equal 'frag', env['FRAGMENT']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_query_and_fragment
- client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal 'a=b', env['QUERY_STRING']
- assert_equal 'frag', env['FRAGMENT']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_unsupported_schemes
- %w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
- client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- assert_raises(HttpParserError) { @request.read_headers(client, AI) }
- end
- end
-
- def test_x_forwarded_proto_https
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: https\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "https", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_x_forwarded_proto_http
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: http\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_x_forwarded_proto_invalid
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: ftp\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_rack_lint_get
- client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_equal '127.0.0.1', env['REMOTE_ADDR']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_no_content_stringio
- client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal StringIO, env['rack.input'].class
- end
-
- def test_zero_content_stringio
- client = MockRequest.new("PUT / HTTP/1.1\r\n" \
- "Content-Length: 0\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal StringIO, env['rack.input'].class
- end
-
- def test_real_content_not_stringio
- client = MockRequest.new("PUT / HTTP/1.1\r\n" \
- "Content-Length: 1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal Unicorn::TeeInput, env['rack.input'].class
- end
-
- def test_rack_lint_put
- client = MockRequest.new(
- "PUT / HTTP/1.1\r\n" \
- "Host: foo\r\n" \
- "Content-Length: 5\r\n" \
- "\r\n" \
- "abcde")
- env = @request.read_headers(client, AI)
- assert ! env.include?(:http_body)
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_rack_lint_big_put
- count = 100
- bs = 0x10000
- buf = (' ' * bs).freeze
- length = bs * count
- client = Tempfile.new('big_put')
- client.syswrite(
- "PUT / HTTP/1.1\r\n" \
- "Host: foo\r\n" \
- "Content-Length: #{length}\r\n" \
- "\r\n")
- count.times { assert_equal bs, client.syswrite(buf) }
- assert_equal 0, client.sysseek(0)
- env = @request.read_headers(client, AI)
- assert ! env.include?(:http_body)
- assert_equal length, env['rack.input'].size
- count.times {
- tmp = env['rack.input'].read(bs)
- tmp << env['rack.input'].read(bs - tmp.size) if tmp.size != bs
- assert_equal buf, tmp
- }
- assert_nil env['rack.input'].read(bs)
- env['rack.input'].rewind
- assert_kind_of Array, @lint.call(env)
- end
-end
[-- Attachment #6: 0005-port-test-unit-test_ccc.rb-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6013 bytes --]
From d6d127f50f9225bf51ef6ce0abce9bad87efaae3 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:39 +0000
Subject: [PATCH 5/5] port test/unit/test_ccc.rb to Perl 5
We'll fold this into integration.t to reduce startup time
penalties and get the benefit of a stable language to reduce
maintenance overhead.
---
t/integration.ru | 4 ++
t/integration.t | 33 ++++++++++++++++
test/unit/test_ccc.rb | 92 -------------------------------------------
3 files changed, 37 insertions(+), 92 deletions(-)
delete mode 100644 test/unit/test_ccc.rb
diff --git a/t/integration.ru b/t/integration.ru
index a6b022a..aaed608 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -87,6 +87,7 @@ def rack_input_tests(env)
[ 200, h, [ dig.hexdigest ] ]
end
+$nr_aborts = 0
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -101,7 +102,10 @@ def rack_input_tests(env)
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
when '/broken_app'; raise RuntimeError, 'hello'
+ when '/aborted'; $nr_aborts += 1; [ 200, {}, [] ]
+ when '/nr_aborts'; [ 200, { 'nr-aborts' => "#$nr_aborts" }, [] ]
when '/nil'; nil
+ when '/read_fifo'; [ 200, {}, [ File.read(env['HTTP_READ_FIFO']) ] ]
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3b1d6df..2d448cd 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -405,6 +405,39 @@ EOM
my $wpid = readline($fifo_fh);
like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
$ck_early_hints->('ccc on');
+
+ $c = tcp_start $srv, 'GET /env_dump HTTP/1.0';
+ vec(my $rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT 'timed out env_dump';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ ok $hdr, 'got all headers';
+
+ # start a slow TCP request
+ my $rfifo = "$tmpdir/rfifo";
+ mkfifo_die $rfifo;
+ $c = tcp_start $srv, "GET /read_fifo HTTP/1.0\r\nRead-FIFO: $rfifo";
+ tcp_start $srv, 'GET /aborted HTTP/1.0' for (1..100);
+ write_file '>', $rfifo, 'TFIN';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ $bdy = <$c>;
+ is $bdy, 'TFIN', 'got slow response from TCP socket';
+
+ # slow Unix socket request
+ $c = unix_start $u1, "GET /read_fifo HTTP/1.0\r\nRead-FIFO: $rfifo";
+ vec($rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT 'timed out Unix CCC';
+ unix_start $u1, 'GET /aborted HTTP/1.0' for (1..100);
+ write_file '>', $rfifo, 'UFIN';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ $bdy = <$c>;
+ is $bdy, 'UFIN', 'got slow response from Unix socket';
+
+ ($status, $hdr, $bdy) = do_req $srv, 'GET /nr_aborts HTTP/1.0';
+ like "@$hdr", qr/nr-aborts: 0\b/,
+ 'aborted connections unseen by Rack app';
}
if ('max_header_len internal API') {
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
deleted file mode 100644
index a0a2bff..0000000
--- a/test/unit/test_ccc.rb
+++ /dev/null
@@ -1,92 +0,0 @@
-# frozen_string_literal: false
-require 'socket'
-require 'unicorn'
-require 'io/wait'
-require 'tempfile'
-require 'test/unit'
-require './test/test_helper'
-
-class TestCccTCPI < Test::Unit::TestCase
- def test_ccc_tcpi
- start_pid = $$
- host = '127.0.0.1'
- srv = TCPServer.new(host, 0)
- port = srv.addr[1]
- err = Tempfile.new('unicorn_ccc')
- rd, wr = IO.pipe
- sleep_pipe = IO.pipe
- pid = fork do
- sleep_pipe[1].close
- reqs = 0
- rd.close
- worker_pid = nil
- app = lambda do |env|
- worker_pid ||= begin
- at_exit { wr.write(reqs.to_s) if worker_pid == $$ }
- $$
- end
- reqs += 1
-
- # will wake up when writer closes
- sleep_pipe[0].read if env['PATH_INFO'] == '/sleep'
-
- [ 200, {'content-length'=>'0', 'content-type'=>'text/plain'}, [] ]
- end
- ENV['UNICORN_FD'] = srv.fileno.to_s
- opts = {
- listeners: [ "#{host}:#{port}" ],
- stderr_path: err.path,
- check_client_connection: true,
- }
- uni = Unicorn::HttpServer.new(app, opts)
- uni.start.join
- end
- wr.close
-
- # make sure the server is running, at least
- client = tcp_socket(host, port)
- client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
- assert client.wait(10), 'never got response from server'
- res = client.read
- assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first response'
- assert_match %r{\r\n\r\n\z}, res, 'got end of response, server is ready'
- client.close
-
- # start a slow request...
- sleeper = tcp_socket(host, port)
- sleeper.write("GET /sleep HTTP/1.1\r\nHost: example.com\r\n\r\n")
-
- # and a bunch of aborted ones
- nr = 100
- nr.times do |i|
- client = tcp_socket(host, port)
- client.write("GET /collections/#{rand(10000)} HTTP/1.1\r\n" \
- "Host: example.com\r\n\r\n")
- client.close
- end
- sleep_pipe[1].close # wake up the reader in the worker
- res = sleeper.read
- assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first sleeper response'
- assert_match %r{\r\n\r\n\z}, res, 'got end of sleeper response'
- sleeper.close
- kpid = pid
- pid = nil
- Process.kill(:QUIT, kpid)
- _, status = Process.waitpid2(kpid)
- assert status.success?
- reqs = rd.read.to_i
- warn "server got #{reqs} requests with #{nr} CCC aborted\n" if $DEBUG
- assert_operator reqs, :<, nr
- assert_operator reqs, :>=, 2, 'first 2 requests got through, at least'
- ensure
- return if start_pid != $$
- srv.close if srv
- if pid
- Process.kill(:QUIT, pid)
- _, status = Process.waitpid2(pid)
- assert status.success?
- end
- err.close! if err
- rd.close if rd
- end
-end
^ permalink raw reply related [relevance 3%]
* [PATCH 00..11/11] more tests to Perl 5..
@ 2023-09-10 20:08 2% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2023-09-10 20:08 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 3020 bytes --]
Hopefully this is less maintenance down the line since Ruby
introduces incompatibilities at a higher rate than Perl.
I don't fully trust Perl, either, but far more Ruby code gets
broken by new releases.
More to come at some point...
Note: attached patches are generated with --irreversible-delete
to save bandwidth.
Eric Wong (11):
tests: port some bad config tests to Perl 5
tests: port working_directory tests to Perl 5
tests: port t/heartbeat-timeout to Perl 5
tests: port reopen logs test over to Perl 5
tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
tests: introduce `do_req' helper sub
tests: use more common variable names between tests
tests: use Time::HiRes `sleep' and `time' everywhere
tests: fold SO_KEEPALIVE check to Perl 5 integration
tests: move broken app test to Perl 5 integration test
tests: fold early shutdown() tests into t/integration.t
t/active-unix-socket.t | 4 +-
t/client_body_buffer_size.t | 6 +-
t/heartbeat-timeout.ru | 2 +-
t/heartbeat-timeout.t | 62 +++++++++++++++
t/integration.ru | 1 +
t/integration.t | 82 +++++++++++++-------
t/lib.perl | 51 ++++++++++--
t/reload-bad-config.t | 54 +++++++++++++
t/{t0006.ru => reopen-logs.ru} | 0
t/reopen-logs.t | 39 ++++++++++
t/t0001-reload-bad-config.sh | 53 -------------
t/t0002-config-conflict.sh | 49 ------------
t/t0003-working_directory.sh | 51 ------------
t/t0004-heartbeat-timeout.sh | 69 -----------------
t/t0004-working_directory_broken.sh | 24 ------
t/t0005-working_directory_app.rb.sh | 40 ----------
t/t0006-reopen-logs.sh | 83 --------------------
t/t0007-working_directory_no_embed_cli.sh | 44 -----------
t/t0009-winch_ttin.sh | 59 --------------
t/winch_ttin.t | 67 ++++++++++++++++
t/working_directory.t | 94 +++++++++++++++++++++++
test/exec/test_exec.rb | 23 +-----
test/unit/test_server.rb | 67 ----------------
23 files changed, 424 insertions(+), 600 deletions(-)
create mode 100644 t/heartbeat-timeout.t
create mode 100644 t/reload-bad-config.t
rename t/{t0006.ru => reopen-logs.ru} (100%)
create mode 100644 t/reopen-logs.t
delete mode 100755 t/t0001-reload-bad-config.sh
delete mode 100755 t/t0002-config-conflict.sh
delete mode 100755 t/t0003-working_directory.sh
delete mode 100755 t/t0004-heartbeat-timeout.sh
delete mode 100755 t/t0004-working_directory_broken.sh
delete mode 100755 t/t0005-working_directory_app.rb.sh
delete mode 100755 t/t0006-reopen-logs.sh
delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
delete mode 100755 t/t0009-winch_ttin.sh
create mode 100644 t/winch_ttin.t
create mode 100644 t/working_directory.t
[-- Attachment #2: 0001-tests-port-some-bad-config-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3987 bytes --]
From f43c28ea10ca8d520b55f2fbb20710dd66fc4fb5 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 22:55:09 +0000
Subject: [PATCH 01/11] tests: port some bad config tests to Perl 5
We can fold some tests into one test to save on Perl startup
time (but Ruby startup time is a lost cause).
---
t/lib.perl | 12 ++++----
t/reload-bad-config.t | 58 ++++++++++++++++++++++++++++++++++++
t/t0001-reload-bad-config.sh | 53 --------------------------------
t/t0002-config-conflict.sh | 49 ------------------------------
4 files changed, 65 insertions(+), 107 deletions(-)
create mode 100644 t/reload-bad-config.t
delete mode 100755 t/t0001-reload-bad-config.sh
delete mode 100755 t/t0002-config-conflict.sh
diff --git a/t/lib.perl b/t/lib.perl
index fe3404ba..7de9e426 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -9,17 +9,19 @@ use Test::More;
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+our ($tmpdir, $errfh, $err_log);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
+ $tmpdir $errfh $err_log
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log");
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
+$err_log = "$tmpdir/err.log";
+open($errfh, '>>', $err_log);
+END { diag slurp($err_log) if $tmpdir };
sub check_stderr () {
- my @log = slurp("$tmpdir/err.log");
+ my @log = slurp($err_log);
diag("@log") if $ENV{V};
my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
@err = grep(!/failed to set accept_filter=/, @err);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
new file mode 100644
index 00000000..c7055c7e
--- /dev/null
+++ b/t/reload-bad-config.t
@@ -0,0 +1,58 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my $ru = "$tmpdir/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+
+open my $fh, '>', $ru;
+print $fh <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+config = ru = "hello world\n" # check for config variable conflicts, too
+run lambda { |env| [ 200, {}, [ ru.to_s ] ] }
+EOM
+close $fh;
+
+open $fh, '>', $u_conf;
+print $fh <<EOM;
+preload_app true
+stderr_path "$err_log"
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
+is($bdy, "hello world\n", 'body matches expected');
+
+open $fh, '>>', $ru;
+say $fh '....this better be a syntax error in any version of ruby...';
+close $fh;
+
+$ar->do_kill('HUP'); # reload
+my @l;
+for (1..1000) {
+ @l = grep(/(?:done|error) reloading/, slurp($err_log)) and
+ last;
+ select undef, undef, undef, 0.011;
+}
+diag slurp($err_log) if $ENV{V};
+ok(grep(/error reloading/, @l), 'got error reloading');
+open $fh, '>', $err_log;
+close $fh;
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
+is($bdy, "hello world\n", 'body matches expected after reload');
+
+check_stderr;
+undef $tmpdir; # quiet t/lib.perl END{}
+done_testing;
diff --git a/t/t0001-reload-bad-config.sh b/t/t0001-reload-bad-config.sh
deleted file mode 100755
index 55bb3555..00000000
diff --git a/t/t0002-config-conflict.sh b/t/t0002-config-conflict.sh
deleted file mode 100755
index d7b2181a..00000000
[-- Attachment #3: 0002-tests-port-working_directory-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4809 bytes --]
From d4514174ee7eadea89003f380acacf32d52acd9d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:18:16 +0000
Subject: [PATCH 02/11] tests: port working_directory tests to Perl 5
We can fold a bunch of them into one test to save startup
time, inodes, and FS activity.
---
t/t0003-working_directory.sh | 51 ---------
t/t0004-working_directory_broken.sh | 24 -----
t/t0005-working_directory_app.rb.sh | 40 -------
t/t0007-working_directory_no_embed_cli.sh | 44 --------
t/working_directory.t | 122 ++++++++++++++++++++++
5 files changed, 122 insertions(+), 159 deletions(-)
delete mode 100755 t/t0003-working_directory.sh
delete mode 100755 t/t0004-working_directory_broken.sh
delete mode 100755 t/t0005-working_directory_app.rb.sh
delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
create mode 100644 t/working_directory.t
diff --git a/t/t0003-working_directory.sh b/t/t0003-working_directory.sh
deleted file mode 100755
index 79988d8b..00000000
diff --git a/t/t0004-working_directory_broken.sh b/t/t0004-working_directory_broken.sh
deleted file mode 100755
index ca9d3825..00000000
diff --git a/t/t0005-working_directory_app.rb.sh b/t/t0005-working_directory_app.rb.sh
deleted file mode 100755
index 0fbab4fc..00000000
diff --git a/t/t0007-working_directory_no_embed_cli.sh b/t/t0007-working_directory_no_embed_cli.sh
deleted file mode 100755
index 77d67072..00000000
diff --git a/t/working_directory.t b/t/working_directory.t
new file mode 100644
index 00000000..e7ff43a5
--- /dev/null
+++ b/t/working_directory.t
@@ -0,0 +1,122 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+mkdir "$tmpdir/alt";
+my $u_sock = "$tmpdir/u.sock";
+my $ru = "$tmpdir/alt/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+working_directory "$tmpdir/alt" # the whole point of this test
+before_fork { |_,_| \$master_ppid = Process.ppid }
+EOM
+close $fh;
+
+my $common_ru = <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] }
+EOM
+
+open $fh, '>', $ru;
+print $fh <<EOM;
+#\\--daemonize --listen $u_sock
+$common_ru
+EOM
+close $fh;
+
+my $pid;
+my $stop_daemon = sub {
+ my ($is_END) = @_;
+ kill('TERM', $pid);
+ my $tries = 1000;
+ while (CORE::kill(0, $pid) && --$tries) {
+ select undef, undef, undef, 0.01;
+ }
+ if ($is_END && CORE::kill(0, $pid)) {
+ CORE::kill('KILL', $pid);
+ die "daemonized PID=$pid did not die";
+ } else {
+ ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
+ undef $pid;
+ }
+};
+
+END { $stop_daemon->(1) if defined $pid };
+
+unicorn('-c', $u_conf)->join; # will daemonize
+chomp($pid = slurp("$tmpdir/pid"));
+
+my $c = unix_start($u_sock, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+chomp(my $bdy = do { local $/; <$c> });
+is($bdy, 1, 'got expected $master_ppid');
+
+$stop_daemon->();
+check_stderr;
+
+if ('test without CLI switches in config.ru') {
+ truncate $err_log, 0;
+ open $fh, '>', $ru;
+ print $fh $common_ru;
+ close $fh;
+
+ unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
+ chomp($pid = slurp("$tmpdir/pid"));
+
+ $c = unix_start($u_sock, 'GET / HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ chomp($bdy = do { local $/; <$c> });
+ is($bdy, 1, 'got expected $master_ppid');
+
+ $stop_daemon->();
+ check_stderr;
+}
+
+if ('ensures broken working_directory (missing config.ru) is OK') {
+ truncate $err_log, 0;
+ unlink $ru;
+
+ my $auto_reap = unicorn('-c', $u_conf);
+ $auto_reap->join;
+ isnt($?, 0, 'exited with error due to missing config.ru');
+
+ like(slurp($err_log), qr/rackup file \Q(config.ru)\E not readable/,
+ 'noted unreadability of config.ru in stderr');
+}
+
+if ('fooapp.rb (not config.ru) works with working_directory') {
+ truncate $err_log, 0;
+ my $fooapp = "$tmpdir/alt/fooapp.rb";
+ open $fh, '>', $fooapp;
+ print $fh <<EOM;
+class Fooapp
+ def self.call(env)
+ b = "dir=#{Dir.pwd}"
+ h = { 'content-type' => 'text/plain', 'content-length' => b.bytesize.to_s }
+ [ 200, h, [ b ] ]
+ end
+end
+EOM
+ close $fh;
+ my $srv = tcp_server;
+ my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
+ { -C => '/', 3 => $srv });
+ $c = tcp_start($srv, 'GET / HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ chomp($bdy = do { local $/; <$c> });
+ is($bdy, "dir=$tmpdir/alt",
+ 'fooapp.rb (w/o config.ru) w/ working_directory');
+ close $c;
+ $auto_reap->join('TERM');
+ is($?, 0, 'fooapp.rb process exited');
+ check_stderr;
+}
+
+undef $tmpdir;
+done_testing;
[-- Attachment #4: 0003-tests-port-t-heartbeat-timeout-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3478 bytes --]
From d67284a692683bca59effd9c0670bd5dd47e4fa3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:53:58 +0000
Subject: [PATCH 03/11] tests: port t/heartbeat-timeout to Perl 5
I absolutely detest and regret adding this feature,
but I'm hell bent on supporting it until the end of days
because we don't break compatibility.
---
t/heartbeat-timeout.ru | 2 +-
t/heartbeat-timeout.t | 69 ++++++++++++++++++++++++++++++++++++
t/t0004-heartbeat-timeout.sh | 69 ------------------------------------
3 files changed, 70 insertions(+), 70 deletions(-)
create mode 100644 t/heartbeat-timeout.t
delete mode 100755 t/t0004-heartbeat-timeout.sh
diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index 20a79380..3eeb5d64 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -7,6 +7,6 @@
sleep # in case STOP signal is not received in time
[ 500, headers, [ "Should never get here\n" ] ]
else
- [ 200, headers, [ "#$$\n" ] ]
+ [ 200, headers, [ "#$$" ] ]
end
}
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
new file mode 100644
index 00000000..1fcf21a2
--- /dev/null
+++ b/t/heartbeat-timeout.t
@@ -0,0 +1,69 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
+mkdir "$tmpdir/alt";
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+timeout 3 # WORST FEATURE EVER
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
+
+my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $wpid = do { local $/; <$c> };
+like($wpid, qr/\A[0-9]+\z/, 'worker is running');
+
+my $t0 = clock_gettime(CLOCK_MONOTONIC);
+$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+vec(my $rvec = '', fileno($c), 1) = 1;
+is(select($rvec, undef, undef, 6), 1, 'got readiness');
+$c->blocking(0);
+is(sysread($c, my $buf, 128), 0, 'got EOF response');
+my $elapsed = clock_gettime(CLOCK_MONOTONIC) - $t0;
+ok($elapsed > 3, 'timeout took >3s');
+
+my @timeout_err = slurp($err_log);
+truncate($err_log, 0);
+is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
+ 'noted timeout error') or diag explain(\@timeout_err);
+
+# did it respawn?
+$c = tcp_start($srv, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $new_pid = do { local $/; <$c> };
+isnt($new_pid, $wpid, 'spawned new worker');
+
+diag 'SIGSTOP for 4 seconds...';
+$ar->do_kill('STOP');
+sleep 4;
+$ar->do_kill('CONT');
+for my $i (1..2) {
+ $c = tcp_start($srv, 'GET /pid HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!,
+ "PID request succeeds #$i after STOP+CONT");
+ my $spid = do { local $/; <$c> };
+ is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
+ if ($i == 1) {
+ diag 'sleeping 2s to ensure timeout is not delayed';
+ sleep 2;
+ }
+}
+
+$ar->join('TERM');
+check_stderr;
+undef $tmpdir;
+
+done_testing;
diff --git a/t/t0004-heartbeat-timeout.sh b/t/t0004-heartbeat-timeout.sh
deleted file mode 100755
index 29652837..00000000
[-- Attachment #5: 0004-tests-port-reopen-logs-test-over-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2317 bytes --]
From 1607ac966f604ec4cf383025c4c3ee296f638fff Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 07:13:11 +0000
Subject: [PATCH 04/11] tests: port reopen logs test over to Perl 5
Being able to do subsecond sleeps is one welcome advantage
over POSIX (not GNU) sleep(1) in portable Bourne sh.
---
t/{t0006.ru => reopen-logs.ru} | 0
t/reopen-logs.t | 43 ++++++++++++++++++
t/t0006-reopen-logs.sh | 83 ----------------------------------
3 files changed, 43 insertions(+), 83 deletions(-)
rename t/{t0006.ru => reopen-logs.ru} (100%)
create mode 100644 t/reopen-logs.t
delete mode 100755 t/t0006-reopen-logs.sh
diff --git a/t/t0006.ru b/t/reopen-logs.ru
similarity index 100%
rename from t/t0006.ru
rename to t/reopen-logs.ru
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
new file mode 100644
index 00000000..e1bf524c
--- /dev/null
+++ b/t/reopen-logs.t
@@ -0,0 +1,43 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+my $out_log = "$tmpdir/out.log";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+stderr_path "$err_log"
+stdout_path "$out_log"
+EOM
+close $fh;
+
+my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs opened');
+
+rename($err_log, "$err_log.rot");
+rename($out_log, "$out_log.rot");
+
+$auto_reap->do_kill('USR1');
+
+my $tries = 1000;
+while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+
+ok(-f $out_log, 'stdout_path recreated after USR1');
+ok(-f $err_log, 'stderr_path recreated after USR1');
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs reopened with sync==true');
+
+$auto_reap->join('QUIT');
+is($?, 0, 'no error on exit');
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/t0006-reopen-logs.sh b/t/t0006-reopen-logs.sh
deleted file mode 100755
index a6e7a17c..00000000
[-- Attachment #6: 0005-tests-rewrite-SIGWINCH-SIGTTIN-test-in-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2916 bytes --]
From 86aea575c331a3b5242db1c14a848928a37ff9e3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 08:27:04 +0000
Subject: [PATCH 05/11] tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
No need to deal with full second sleeps, here.
---
t/t0009-winch_ttin.sh | 59 -----------------------------------
t/winch_ttin.t | 72 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 72 insertions(+), 59 deletions(-)
delete mode 100755 t/t0009-winch_ttin.sh
create mode 100644 t/winch_ttin.t
diff --git a/t/t0009-winch_ttin.sh b/t/t0009-winch_ttin.sh
deleted file mode 100755
index 6e56e30c..00000000
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
new file mode 100644
index 00000000..1a198778
--- /dev/null
+++ b/t/winch_ttin.t
@@ -0,0 +1,72 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use POSIX qw(mkfifo);
+my $u_conf = "$tmpdir/u.conf.rb";
+my $u_sock = "$tmpdir/u.sock";
+my $fifo = "$tmpdir/fifo";
+mkfifo($fifo, 0666) or die "mkfifo($fifo): $!";
+
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+listen "$u_sock"
+stderr_path "$err_log"
+after_fork do |server, worker|
+ # test script will block while reading from $fifo,
+ File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s }
+end
+EOM
+close $fh;
+
+unicorn('-D', '-c', $u_conf, 't/integration.ru')->join;
+is($?, 0, 'daemonized properly');
+open $fh, '<', "$tmpdir/pid";
+chomp(my $pid = <$fh>);
+ok(kill(0, $pid), 'daemonized PID works');
+my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef };
+END { $quit->() };
+
+open $fh, '<', $fifo;
+my $worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'initial worker spawned');
+
+my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+my $worker_pid = do { local $/; <$c> };
+like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
+chomp $worker_pid;
+ok(kill(0, $worker_pid), 'worker_pid is valid');
+
+ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
+
+my $tries = 1000;
+while (CORE::kill(0, $worker_pid) && --$tries) {
+ select undef, undef, undef, 0.01;
+}
+ok(!CORE::kill(0, $worker_pid), 'worker not running');
+
+ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');
+
+open $fh, '<', $fifo;
+$worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'worker restarted');
+
+$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+chomp(my $new_worker_pid = do { local $/; <$c> });
+like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+ok(kill(0, $new_worker_pid), 'got a valid worker PID');
+isnt($worker_pid, $new_worker_pid, 'worker PID changed');
+
+$quit->();
+
+check_stderr;
+undef $tmpdir;
+done_testing;
[-- Attachment #7: 0006-tests-introduce-do_req-helper-sub.patch --]
[-- Type: text/x-diff, Size: 11556 bytes --]
From 29885f0d95aaa8e1d1f6cf3b791d9f08338a511e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:15:16 +0000
Subject: [PATCH 06/11] tests: introduce `do_req' helper sub
While early tests required fine-grained control in trickling
requests, many of our later tests can use a short one-liner
w/o having to spawn curl.
---
t/heartbeat-timeout.t | 12 +++---------
t/integration.t | 33 +++++++++++++--------------------
t/lib.perl | 16 +++++++++++++++-
t/reload-bad-config.t | 8 ++------
t/reopen-logs.t | 8 ++------
t/winch_ttin.t | 11 ++++-------
t/working_directory.t | 17 +++++------------
7 files changed, 44 insertions(+), 61 deletions(-)
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index 1fcf21a2..ce1f7e16 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -18,10 +18,8 @@ close $fh;
my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
-my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $wpid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $wpid = do { local $/; <$c> };
like($wpid, qr/\A[0-9]+\z/, 'worker is running');
my $t0 = clock_gettime(CLOCK_MONOTONIC);
@@ -39,10 +37,8 @@ is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
'noted timeout error') or diag explain(\@timeout_err);
# did it respawn?
-$c = tcp_start($srv, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_pid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $new_pid = do { local $/; <$c> };
isnt($new_pid, $wpid, 'spawned new worker');
diag 'SIGSTOP for 4 seconds...';
@@ -50,11 +46,9 @@ $ar->do_kill('STOP');
sleep 4;
$ar->do_kill('CONT');
for my $i (1..2) {
- $c = tcp_start($srv, 'GET /pid HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr, my $spid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!,
"PID request succeeds #$i after STOP+CONT");
- my $spid = do { local $/; <$c> };
is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
if ($i == 1) {
diag 'sleeping 2s to ensure timeout is not delayed';
diff --git a/t/integration.t b/t/integration.t
index bb2ab51b..13b07467 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -62,11 +62,10 @@ EOM
},
);
-my ($c, $status, $hdr);
+my ($c, $status, $hdr, $bdy);
# response header tests
-$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
is_deeply([ grep(/^X-R2: /, @$hdr) ],
@@ -84,16 +83,16 @@ SKIP: { # Date header check
};
-$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
'rack 3 array headers supported') or diag(explain($hdr));
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- my $c = tcp_start($srv, 'GET /env_dump');
- my $json = do { local $/; readline($c) };
+ ($status, $hdr, my $json) = do_req $srv, 'GET /env_dump';
+ is($status, undef, 'no status for HTTP/0.9');
+ is($hdr, undef, 'no header for HTTP/0.9');
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
my $env = JSON::PP->new->decode($json);
@@ -102,8 +101,7 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /nil-header-value HTTP/1.0');
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
@@ -128,12 +126,10 @@ my $ck_early_hints = sub {
$ck_early_hints->('ccc off'); # we'll retest later
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'POST /tweak-status-code HTTP/1.0';
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'POST /restore-status-code HTTP/1.0';
is($status, $orig_200_status, 'original status restored');
}
@@ -145,12 +141,11 @@ SKIP: {
}
if ('bad requests') {
- $c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1';
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
$c = tcp_start($srv);
- print $c 'GET /';;
+ print $c 'GET /';
my $buf = join('', (0..9), 'ab');
for (0..1023) { print $c $buf }
print $c " HTTP/1.0\r\n\r\n";
@@ -308,12 +303,10 @@ EOM
$wpid =~ s/\Apid=// or die;
ok(CORE::kill(0, $wpid), 'worker PID retrieved');
- $c = tcp_start($srv, $req);
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req($srv, $req);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
- $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req($srv, 'GET /xxxxxx HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
}
diff --git a/t/lib.perl b/t/lib.perl
index 7de9e426..13e390d6 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -12,7 +12,8 @@ use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh, $err_log);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log
- SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
+ do_req);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -182,6 +183,19 @@ sub unicorn {
UnicornTest::AutoReap->new($pid);
}
+sub do_req ($@) {
+ my ($dst, @req) = @_;
+ my $c = ref($dst) ? tcp_start($dst, @req) : unix_start($dst, @req);
+ return $c if !wantarray;
+ my ($status, $hdr);
+ # read headers iff HTTP/1.x request, HTTP/0.9 remains supported
+ my ($first) = (join('', @req) =~ m!\A([^\r\n]+)!);
+ ($status, $hdr) = slurp_hdr($c) if $first =~ m{\s*HTTP/\S+$};
+ my $bdy = do { local $/; <$c> };
+ close $c;
+ ($status, $hdr, $bdy);
+}
+
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index c7055c7e..543421da 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -25,9 +25,7 @@ EOM
close $fh;
my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
is($bdy, "hello world\n", 'body matches expected');
@@ -47,9 +45,7 @@ ok(grep(/error reloading/, @l), 'got error reloading');
open $fh, '>', $err_log;
close $fh;
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
is($bdy, "hello world\n", 'body matches expected after reload');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index e1bf524c..8a58c1b9 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -14,9 +14,7 @@ EOM
close $fh;
my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "true\n", 'logs opened');
rename($err_log, "$err_log.rot");
@@ -31,9 +29,7 @@ while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
ok(-f $out_log, 'stdout_path recreated after USR1');
ok(-f $err_log, 'stderr_path recreated after USR1');
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "true\n", 'logs reopened with sync==true');
$auto_reap->join('QUIT');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 1a198778..509b118f 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -34,10 +34,8 @@ my $worker_nr = <$fh>;
close $fh;
is($worker_nr, '0', 'initial worker spawned');
-my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
like($status, qr/ 200\b/, 'got 200 response');
-my $worker_pid = do { local $/; <$c> };
like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
chomp $worker_pid;
ok(kill(0, $worker_pid), 'worker_pid is valid');
@@ -57,11 +55,10 @@ $worker_nr = <$fh>;
close $fh;
is($worker_nr, '0', 'worker restarted');
-$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
like($status, qr/ 200\b/, 'got 200 response');
-chomp(my $new_worker_pid = do { local $/; <$c> });
-like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+like($new_worker_pid, qr/\A[0-9]+\n\z/, 'got new worker PID');
+chomp $new_worker_pid;
ok(kill(0, $new_worker_pid), 'got a valid worker PID');
isnt($worker_pid, $new_worker_pid, 'worker PID changed');
diff --git a/t/working_directory.t b/t/working_directory.t
index e7ff43a5..6c974720 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -52,10 +52,8 @@ END { $stop_daemon->(1) if defined $pid };
unicorn('-c', $u_conf)->join; # will daemonize
chomp($pid = slurp("$tmpdir/pid"));
-my $c = unix_start($u_sock, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-chomp(my $bdy = do { local $/; <$c> });
-is($bdy, 1, 'got expected $master_ppid');
+my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+is($bdy, "1\n", 'got expected $master_ppid');
$stop_daemon->();
check_stderr;
@@ -69,10 +67,8 @@ if ('test without CLI switches in config.ru') {
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
chomp($pid = slurp("$tmpdir/pid"));
- $c = unix_start($u_sock, 'GET / HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
- chomp($bdy = do { local $/; <$c> });
- is($bdy, 1, 'got expected $master_ppid');
+ ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+ is($bdy, "1\n", 'got expected $master_ppid');
$stop_daemon->();
check_stderr;
@@ -107,12 +103,9 @@ EOM
my $srv = tcp_server;
my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
{ -C => '/', 3 => $srv });
- $c = tcp_start($srv, 'GET / HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
- chomp($bdy = do { local $/; <$c> });
+ ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "dir=$tmpdir/alt",
'fooapp.rb (w/o config.ru) w/ working_directory');
- close $c;
$auto_reap->join('TERM');
is($?, 0, 'fooapp.rb process exited');
check_stderr;
[-- Attachment #8: 0007-tests-use-more-common-variable-names-between-tests.patch --]
[-- Type: text/x-diff, Size: 6507 bytes --]
From 948f78403172657590d690b9255467b9ccb968cd Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:31:44 +0000
Subject: [PATCH 07/11] tests: use more common variable names between tests
Stuff like $u_conf, $daemon_pid, $pid_file, etc. will
reduce cognitive overhead.
---
t/active-unix-socket.t | 2 +-
t/client_body_buffer_size.t | 6 ++----
t/heartbeat-timeout.t | 3 +--
t/integration.t | 5 ++---
t/lib.perl | 31 +++++++++++++++++++++++++++----
t/working_directory.t | 31 +++++--------------------------
6 files changed, 38 insertions(+), 40 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 4dcc8dc6..32cb0c2e 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -15,7 +15,7 @@ my $u2 = "$tmpdir/u2.sock";
print $fh <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
-stderr_path "$tmpdir/err.log"
+stderr_path "$err_log"
EOM
close $fh;
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index 3067f284..d4799012 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -4,16 +4,14 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-my $uconf = "$tmpdir/u.conf.rb";
-
-open my $conf_fh, '>', $uconf;
+open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
print $conf_fh <<EOM;
client_body_buffer_size 0
EOM
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
-my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf);
my $ar = unicorn(@uarg, { 3 => $srv });
my ($c, $status, $hdr);
my $mem_class = 'StringIO';
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index ce1f7e16..694867a4 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -6,7 +6,6 @@ use autodie;
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
mkdir "$tmpdir/alt";
my $srv = tcp_server();
-my $u_conf = "$tmpdir/u.conf.rb";
open my $fh, '>', $u_conf;
print $fh <<EOM;
pid "$tmpdir/pid"
@@ -23,7 +22,7 @@ like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
like($wpid, qr/\A[0-9]+\z/, 'worker is running');
my $t0 = clock_gettime(CLOCK_MONOTONIC);
-$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+my $c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
vec(my $rvec = '', fileno($c), 1) = 1;
is(select($rvec, undef, undef, 6), 1, 'got readiness');
$c->blocking(0);
diff --git a/t/integration.t b/t/integration.t
index 13b07467..eb40ffc7 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -10,15 +10,14 @@ use autodie;
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
my $t0 = time;
-my $conf = "$tmpdir/u.conf.rb";
-open my $conf_fh, '>', $conf;
+open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
my $u1 = "$tmpdir/u1";
print $conf_fh <<EOM;
early_hints true
listen "$u1"
EOM
-my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
+my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
my $fifo = "$tmpdir/fifo";
POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
diff --git a/t/lib.perl b/t/lib.perl
index 13e390d6..244972bc 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,20 +6,43 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
+use Time::HiRes qw(sleep);
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh, $err_log);
+our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
+ $pid_file);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
- $tmpdir $errfh $err_log
+ $tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req);
+ do_req stop_daemon);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
$err_log = "$tmpdir/err.log";
+$pid_file = "$tmpdir/pid";
+$u_sock = "$tmpdir/u.sock";
+$u_conf = "$tmpdir/u.conf.rb";
open($errfh, '>>', $err_log);
-END { diag slurp($err_log) if $tmpdir };
+
+sub stop_daemon (;$) {
+ my ($is_END) = @_;
+ kill('TERM', $daemon_pid);
+ my $tries = 1000;
+ while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) }
+ if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing
+ CORE::kill('KILL', $daemon_pid);
+ die "daemon_pid=$daemon_pid did not die";
+ } else {
+ ok(!CORE::kill(0, $daemon_pid), 'daemonized unicorn gone');
+ undef $daemon_pid;
+ }
+};
+
+END {
+ diag slurp($err_log) if $tmpdir;
+ stop_daemon(1) if defined $daemon_pid;
+};
sub check_stderr () {
my @log = slurp($err_log);
diff --git a/t/working_directory.t b/t/working_directory.t
index 6c974720..f9254eb8 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -4,12 +4,10 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
mkdir "$tmpdir/alt";
-my $u_sock = "$tmpdir/u.sock";
my $ru = "$tmpdir/alt/config.ru";
-my $u_conf = "$tmpdir/u.conf.rb";
open my $fh, '>', $u_conf;
print $fh <<EOM;
-pid "$tmpdir/pid"
+pid "$pid_file"
preload_app true
stderr_path "$err_log"
working_directory "$tmpdir/alt" # the whole point of this test
@@ -30,32 +28,13 @@ $common_ru
EOM
close $fh;
-my $pid;
-my $stop_daemon = sub {
- my ($is_END) = @_;
- kill('TERM', $pid);
- my $tries = 1000;
- while (CORE::kill(0, $pid) && --$tries) {
- select undef, undef, undef, 0.01;
- }
- if ($is_END && CORE::kill(0, $pid)) {
- CORE::kill('KILL', $pid);
- die "daemonized PID=$pid did not die";
- } else {
- ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
- undef $pid;
- }
-};
-
-END { $stop_daemon->(1) if defined $pid };
-
unicorn('-c', $u_conf)->join; # will daemonize
-chomp($pid = slurp("$tmpdir/pid"));
+chomp($daemon_pid = slurp($pid_file));
my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
is($bdy, "1\n", 'got expected $master_ppid');
-$stop_daemon->();
+stop_daemon;
check_stderr;
if ('test without CLI switches in config.ru') {
@@ -65,12 +44,12 @@ if ('test without CLI switches in config.ru') {
close $fh;
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
- chomp($pid = slurp("$tmpdir/pid"));
+ chomp($daemon_pid = slurp($pid_file));
($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
is($bdy, "1\n", 'got expected $master_ppid');
- $stop_daemon->();
+ stop_daemon;
check_stderr;
}
[-- Attachment #9: 0008-tests-use-Time-HiRes-sleep-and-time-everywhere.patch --]
[-- Type: text/x-diff, Size: 4106 bytes --]
From dd9f2efeebf20cfa1def0ce92cb4e35a8b5c1580 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:35:09 +0000
Subject: [PATCH 08/11] tests: use Time::HiRes `sleep' and `time' everywhere
The time(2) syscall use by CORE::time is inaccurate[1].
It's also easier to read `sleep 0.01' rather than the
longer `select' equivalent.
[1] a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01)
---
t/active-unix-socket.t | 2 +-
t/integration.t | 5 +++--
t/lib.perl | 4 ++--
t/reload-bad-config.t | 2 +-
t/reopen-logs.t | 4 ++--
t/winch_ttin.t | 4 +---
6 files changed, 10 insertions(+), 11 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 32cb0c2e..ff731b5f 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -86,7 +86,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
'fail to connect to u1');
for (1..50) { # wait for init process to reap worker
kill(0, $worker_pid) or last;
- select(undef, undef, undef, 0.011);
+ sleep 0.011;
}
ok(!kill(0, $worker_pid), 'worker gone after parent dies');
}
diff --git a/t/integration.t b/t/integration.t
index eb40ffc7..80485e44 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -77,8 +77,9 @@ SKIP: { # Date header check
eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
$d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
my $t = HTTP::Date::str2time($d[0]);
- ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
- diag(explain([$t, $!, \@d]));
+ my $now = time;
+ ok($t >= ($t0 - 1) && $t > 0 && $t <= ($now + 1), 'valid date') or
+ diag(explain(["t=$t t0=$t0 now=$now", $!, \@d]));
};
diff --git a/t/lib.perl b/t/lib.perl
index 244972bc..9254b23b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,7 +6,7 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
-use Time::HiRes qw(sleep);
+use Time::HiRes qw(sleep time);
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
@@ -15,7 +15,7 @@ our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req stop_daemon);
+ do_req stop_daemon sleep time);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index 543421da..c023b88c 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -38,7 +38,7 @@ my @l;
for (1..1000) {
@l = grep(/(?:done|error) reloading/, slurp($err_log)) and
last;
- select undef, undef, undef, 0.011;
+ sleep 0.011;
}
diag slurp($err_log) if $ENV{V};
ok(grep(/error reloading/, @l), 'got error reloading');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index 8a58c1b9..76a4dbdf 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -23,8 +23,8 @@ rename($out_log, "$out_log.rot");
$auto_reap->do_kill('USR1');
my $tries = 1000;
-while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
-while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $err_log && --$tries) { sleep 0.01 };
+while (!-f $out_log && --$tries) { sleep 0.01 };
ok(-f $out_log, 'stdout_path recreated after USR1');
ok(-f $err_log, 'stderr_path recreated after USR1');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 509b118f..c5079599 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -43,9 +43,7 @@ ok(kill(0, $worker_pid), 'worker_pid is valid');
ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
my $tries = 1000;
-while (CORE::kill(0, $worker_pid) && --$tries) {
- select undef, undef, undef, 0.01;
-}
+while (CORE::kill(0, $worker_pid) && --$tries) { sleep 0.01 }
ok(!CORE::kill(0, $worker_pid), 'worker not running');
ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');
[-- Attachment #10: 0009-tests-fold-SO_KEEPALIVE-check-to-Perl-5-integration.patch --]
[-- Type: text/x-diff, Size: 2675 bytes --]
From b588ccbbf73547487f54fd1a9d5396d6848e8661 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:21:05 +0000
Subject: [PATCH 09/11] tests: fold SO_KEEPALIVE check to Perl 5 integration
No need to startup more processes than necessary.
---
t/integration.t | 13 +++++++++++++
test/exec/test_exec.rb | 23 +----------------------
2 files changed, 14 insertions(+), 22 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index 80485e44..bea221ce 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,16 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
+use Socket qw(SOL_SOCKET SO_KEEPALIVE);
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
+
+if ('ensure Perl does not set SO_KEEPALIVE by default') {
+ my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+ unpack('i', $val) == 0 or
+ setsockopt($srv, SOL_SOCKET, SO_KEEPALIVE, pack('i', 0));
+ $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+}
my $t0 = time;
open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
@@ -71,6 +79,11 @@ is_deeply([ grep(/^X-R2: /, @$hdr) ],
[ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+{
+ my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+ is(unpack('i', $val), 1, 'SO_KEEPALIVE set on inherited socket');
+}
+
SKIP: { # Date header check
my @d = grep(/^Date: /i, @$hdr);
is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 55f828e7..84944520 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -1,6 +1,5 @@
# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
+# Don't add to this file, new tests are in Perl 5. See t/README
FLOCK_PATH = File.expand_path(__FILE__)
require './test/test_helper'
@@ -97,26 +96,6 @@ def teardown
end
end
- def test_inherit_listener_unspecified
- File.open("config.ru", "wb") { |fp| fp.write(HI) }
- sock = TCPServer.new(@addr, @port)
- sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
- pid = xfork do
- redirect_test_io do
- ENV['UNICORN_FD'] = sock.fileno.to_s
- exec($unicorn_bin, sock.fileno => sock.fileno)
- end
- end
- res = hit(["http://#@addr:#@port/"])
- assert_equal [ "HI\n" ], res
- assert_shutdown(pid)
- assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
- 'unicorn should always set SO_KEEPALIVE on inherited sockets'
- ensure
- sock.close if sock
- end
-
def test_working_directory_rel_path_config_file
other = Tempfile.new('unicorn.wd')
File.unlink(other.path)
[-- Attachment #11: 0010-tests-move-broken-app-test-to-Perl-5-integration-tes.patch --]
[-- Type: text/x-diff, Size: 2376 bytes --]
From 7160f1b519aece0fe645d22a7d8fb954a43ad6fb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:37:32 +0000
Subject: [PATCH 10/11] tests: move broken app test to Perl 5 integration test
Less Ruby means fewer incompatibilities to worry about with
every new version.
---
t/integration.ru | 1 +
t/integration.t | 6 ++++++
test/unit/test_server.rb | 14 --------------
3 files changed, 7 insertions(+), 14 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index 086126ab..888833a9 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -98,6 +98,7 @@ def rack_input_tests(env)
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
+ when '/broken_app'; raise RuntimeError, 'hello'
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index bea221ce..ba17dd9e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -118,6 +118,12 @@ SKIP: {
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
+check_stderr;
+($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0');
+like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint');
+is($bdy, undef, 'no response body after exception');
+truncate($errfh, 0);
+
my $ck_early_hints = sub {
my ($note) = @_;
$c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 0a710d12..2af12eac 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -127,20 +127,6 @@ def test_after_reply
sock.close
end
- def test_broken_app
- teardown
- app = lambda { |env| raise RuntimeError, "hello" }
- # [200, {}, []] }
- redirect_test_io do
- @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
- @server.start
- end
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("GET / HTTP/1.0\r\n\r\n")
- assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096)
- assert_nil sock.close
- end
-
def test_simple_server
results = hit(["http://localhost:#{@port}/test"])
assert_equal 'hello!\n', results[0], "Handler didn't really run"
[-- Attachment #12: 0011-tests-fold-early-shutdown-tests-into-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 4527 bytes --]
From 05028146b5e69c566663fdab9f8b92c6145a791a Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:52:03 +0000
Subject: [PATCH 11/11] tests: fold early shutdown() tests into t/integration.t
This means fewer redundant tests and more chances to notice
Ruby incompatibilities.
---
t/integration.t | 22 +++++++++++++++--
test/unit/test_server.rb | 53 ----------------------------------------
2 files changed, 20 insertions(+), 55 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index ba17dd9e..7310ff29 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,7 +7,7 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-use Socket qw(SOL_SOCKET SO_KEEPALIVE);
+use Socket qw(SOL_SOCKET SO_KEEPALIVE SHUT_WR);
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
@@ -209,6 +209,7 @@ SKIP: {
defined($opt{overwrite}) and
print { $c } ('x' x $opt{overwrite});
$c->flush or die $!;
+ shutdown($c, SHUT_WR);
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
};
@@ -225,6 +226,8 @@ SKIP: {
# ensure small overwrites don't get checksummed
$ck_hash->('identity', '/rack_input', -s => $blob_size,
overwrite => 1); # one extra byte
+ unlike(slurp($err_log), qr/ClientShutdown/,
+ 'no overreads after client SHUT_WR');
# excessive overwrite truncated
$c = tcp_start($srv);
@@ -238,8 +241,23 @@ SKIP: {
$! = 0;
while (print $c $buf and time < $end) { ++$n }
ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+ undef $c;
+ }
+
+ # client shutdown early
+ $c = tcp_start($srv);
+ $c->autoflush(0);
+ print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 16384\r\n\r\n";
+ if (1) {
+ local $SIG{PIPE} = 'IGNORE';
+ print $c 'too short body';
+ shutdown($c, SHUT_WR);
+ vec(my $rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT "timed out";
+ my $buf = <$c>;
+ is($buf, undef, 'server aborted after client SHUT_WR');
+ undef $c;
}
- undef $c;
$curl // skip 'no curl found in PATH', 1;
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 2af12eac..7ffa48f0 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -132,59 +132,6 @@ def test_simple_server
assert_equal 'hello!\n', results[0], "Handler didn't really run"
end
- def test_client_shutdown_writes
- bs = 15609315 * rand
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("PUT /hello HTTP/1.1\r\n")
- sock.syswrite("Host: example.com\r\n")
- sock.syswrite("Transfer-Encoding: chunked\r\n")
- sock.syswrite("Trailer: X-Foo\r\n")
- sock.syswrite("\r\n")
- sock.syswrite("%x\r\n" % [ bs ])
- sock.syswrite("F" * bs)
- sock.syswrite("\r\n0\r\nX-")
- "Foo: bar\r\n\r\n".each_byte do |x|
- sock.syswrite x.chr
- sleep 0.05
- end
- # we wrote the entire request before shutting down, server should
- # continue to process our request and never hit EOFError on our sock
- sock.shutdown(Socket::SHUT_WR)
- buf = sock.read
- assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
- next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
- assert_equal 'hello!\n', next_client
- lines = File.readlines("test_stderr.#$$.log")
- assert lines.grep(/^Unicorn::ClientShutdown: /).empty?
- assert_nil sock.close
- end
-
- def test_client_shutdown_write_truncates
- bs = 15609315 * rand
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("PUT /hello HTTP/1.1\r\n")
- sock.syswrite("Host: example.com\r\n")
- sock.syswrite("Transfer-Encoding: chunked\r\n")
- sock.syswrite("Trailer: X-Foo\r\n")
- sock.syswrite("\r\n")
- sock.syswrite("%x\r\n" % [ bs ])
- sock.syswrite("F" * (bs / 2.0))
-
- # shutdown prematurely, this will force the server to abort
- # processing on us even during app dispatch
- sock.shutdown(Socket::SHUT_WR)
- IO.select([sock], nil, nil, 60) or raise "Timed out"
- buf = sock.read
- assert_equal "", buf
- next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
- assert_equal 'hello!\n', next_client
- lines = File.readlines("test_stderr.#$$.log")
- lines = lines.grep(/^Unicorn::ClientShutdown: bytes_read=\d+/)
- assert_equal 1, lines.size
- assert_match %r{\AUnicorn::ClientShutdown: bytes_read=\d+ true$}, lines[0]
- assert_nil sock.close
- end
-
def test_client_malformed_body
bs = 15653984
sock = tcp_socket('127.0.0.1', @port)
^ permalink raw reply related [relevance 2%]
* [PATCH 00-23/23] start porting tests to Perl5
@ 2023-06-05 10:32 1% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2023-06-05 10:32 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 4179 bytes --]
Still a lot more work to do, but at least socat is no longer a
test dependency. Perl5 is installed on far more systems than
socat.
Ruby introduces breaking changes every year and I can't trust
tests to work as they were originally intended, anymore.
Perl 5 doesn't have perfect backwards compatibility, either; but
it's the least bad of any widely-installed scripting language.
Note: that 23/23 introduces a subtle bugfix which changes
behavior for systemd users
Patches are attached to reduce load on SMTP servers.
Some more patches to come as I deal with Ruby 3.x deprecation
warnings :<
Eric Wong (23):
switch unit/test_response.rb to Perl 5 integration test
support rack 3 multi-value headers
port t0018-write-on-close.sh to Perl 5
port t0000-http-basic.sh to Perl 5
port t0002-parser-error.sh to Perl 5
t/integration.t: use start_req to simplify test slighly
port t0011-active-unix-socket.sh to Perl 5
port t0100-rack-input-tests.sh to Perl 5
tests: use autodie to simplify error checking
port t0019-max_header_len.sh to Perl 5
test_exec: drop sd_listen_fds emulation test
test_exec: drop test_basic and test_config_ru_alt_path
tests: check_stderr consistently in Perl 5 tests
tests: consistent tcp_start and unix_start across Perl 5 tests
port t9000-preread-input.sh to Perl 5
port t/t0116-client_body_buffer_size.sh to Perl 5
tests: get rid of sha1sum.rb and rsha1() sh function
early_hints supports Rack 3 array headers
test_server: drop early_hints test
t/integration.t: switch PUT tests to MD5, reuse buffers
tests: move test_upload.rb tests to t/integration.t
drop redundant IO#close_on_exec=false calls
LISTEN_FDS-inherited sockets are immortal across SIGHUP
GNUmakefile | 7 +-
lib/unicorn/http_server.rb | 12 +-
t/README | 21 +-
t/active-unix-socket.t | 113 +++++++
t/bin/content-md5-put | 36 ---
t/bin/sha1sum.rb | 17 --
t/{t0116.ru => client_body_buffer_size.ru} | 2 -
t/client_body_buffer_size.t | 82 ++++++
t/integration.ru | 114 +++++++
t/integration.t | 326 +++++++++++++++++++++
t/lib.perl | 217 ++++++++++++++
t/preread_input.ru | 21 +-
t/rack-input-tests.ru | 21 --
t/t0000-http-basic.sh | 50 ----
t/t0002-parser-error.sh | 94 ------
t/t0011-active-unix-socket.sh | 79 -----
t/t0018-write-on-close.sh | 23 --
t/t0019-max_header_len.sh | 49 ----
t/t0100-rack-input-tests.sh | 124 --------
t/t0116-client_body_buffer_size.sh | 80 -----
t/t9000-preread-input.sh | 48 ---
t/test-lib.sh | 4 -
t/write-on-close.ru | 11 -
test/exec/test_exec.rb | 57 ----
test/unit/test_response.rb | 111 -------
test/unit/test_server.rb | 31 --
test/unit/test_upload.rb | 301 -------------------
27 files changed, 891 insertions(+), 1160 deletions(-)
create mode 100644 t/active-unix-socket.t
delete mode 100755 t/bin/content-md5-put
delete mode 100755 t/bin/sha1sum.rb
rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
create mode 100644 t/client_body_buffer_size.t
create mode 100644 t/integration.ru
create mode 100644 t/integration.t
create mode 100644 t/lib.perl
delete mode 100644 t/rack-input-tests.ru
delete mode 100755 t/t0000-http-basic.sh
delete mode 100755 t/t0002-parser-error.sh
delete mode 100755 t/t0011-active-unix-socket.sh
delete mode 100755 t/t0018-write-on-close.sh
delete mode 100755 t/t0019-max_header_len.sh
delete mode 100755 t/t0100-rack-input-tests.sh
delete mode 100755 t/t0116-client_body_buffer_size.sh
delete mode 100755 t/t9000-preread-input.sh
delete mode 100644 t/write-on-close.ru
delete mode 100644 test/unit/test_response.rb
delete mode 100644 test/unit/test_upload.rb
[-- Attachment #2: 0001-switch-unit-test_response.rb-to-Perl-5-integration-t.patch --]
[-- Type: text/x-diff, Size: 15667 bytes --]
From 086e397abc0126556af24df77a976671294df2ee Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:30 +0000
Subject: [PATCH 01/23] switch unit/test_response.rb to Perl 5 integration test
http_response_write may benefit from API changes for Rack 3
support.
Since there's no benefit I can see from using a unit test,
switch to an integration test to avoid having to maintain the
unit test if our internal http_response_write method changes.
Of course, I can't trust tests written in Ruby since I've had to
put up with a constant stream of incompatibilities over the past
two decades :< Perl is more widely installed than socat[1], and
nearly all the Perl I wrote 20 years ago still works
unmodified today.
[1] the rarest dependency of the Bourne shell integration tests
---
GNUmakefile | 5 +-
t/README | 24 +++--
t/integration.ru | 38 ++++++++
t/integration.t | 64 +++++++++++++
t/lib.perl | 189 +++++++++++++++++++++++++++++++++++++
test/unit/test_response.rb | 111 ----------------------
6 files changed, 313 insertions(+), 118 deletions(-)
create mode 100644 t/integration.ru
create mode 100644 t/integration.t
create mode 100644 t/lib.perl
delete mode 100644 test/unit/test_response.rb
diff --git a/GNUmakefile b/GNUmakefile
index 0e08ef0..5cca189 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -86,7 +86,7 @@ $(tmp_bin)/%: bin/% | $(tmp_bin)
bins: $(tmp_bins)
t_log := $(T_log) $(T_n_log)
-test: $(T) $(T_n)
+test: $(T) $(T_n) test-prove
@cat $(t_log) | $(MRI) test/aggregate.rb
@$(RM) $(t_log)
@@ -141,6 +141,9 @@ t/random_blob:
test-integration: $(T_sh)
+test-prove:
+ prove -vw
+
check: test-require test test-integration
test-all: check
diff --git a/t/README b/t/README
index 14de559..8a5243e 100644
--- a/t/README
+++ b/t/README
@@ -5,16 +5,24 @@ TCP ports or Unix domain sockets. They're all designed to run
concurrently with other tests to minimize test time, but tests may be
run independently as well.
-We write our tests in Bourne shell because that's what we're
-comfortable writing integration tests with.
+New tests are written in Perl 5 because we need a stable language
+to test real-world behavior and Ruby introduces incompatibilities
+at a far faster rate than Perl 5. Perl is Ruby's older cousin, so
+it should be easy-to-learn for Rubyists.
+
+Old tests are in Bourne shell, but the socat(1) dependency was probably
+too rare compared to Perl 5.
== Requirements
-* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!)
+* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
+* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
+
+The following requirements will eventually be dropped.
+
* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
-* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...)
We do not use bashisms or any non-portable, non-POSIX constructs
in our shell code. We use the "pipefail" option if available and
@@ -26,9 +34,13 @@ with {dash}[http://gondor.apana.org.au/~herbert/dash/] and
To run the entire test suite with 8 tests running at once:
- make -j8
+ make -j8 && prove -vw
+
+To run one individual test (Perl5):
+
+ prove -vw t/integration.t
-To run one individual test:
+To run one individual test (shell):
make t0000-simple-http.sh
diff --git a/t/integration.ru b/t/integration.ru
new file mode 100644
index 0000000..6ef873c
--- /dev/null
+++ b/t/integration.ru
@@ -0,0 +1,38 @@
+#!ruby
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+# this goes for t/integration.t We'll try to put as many tests
+# in here as possible to avoid startup overhead of Ruby.
+
+$orig_rack_200 = nil
+def tweak_status_code
+ $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
+ Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
+ [ 200, {}, [] ]
+end
+
+def restore_status_code
+ $orig_rack_200 or return [ 500, {}, [] ]
+ Rack::Utils::HTTP_STATUS_CODES[200] = $orig_rack_200
+ [ 200, {}, [] ]
+end
+
+run(lambda do |env|
+ case env['REQUEST_METHOD']
+ when 'GET'
+ case env['PATH_INFO']
+ when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+ when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
+ when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ end # case PATH_INFO (GET)
+ when 'POST'
+ case env['PATH_INFO']
+ when '/tweak-status-code'; tweak_status_code
+ when '/restore-status-code'; restore_status_code
+ end # case PATH_INFO (POST)
+ # ...
+ when 'PUT'
+ # ...
+ end # case REQUEST_METHOD
+end) # run
diff --git a/t/integration.t b/t/integration.t
new file mode 100644
index 0000000..5569155
--- /dev/null
+++ b/t/integration.t
@@ -0,0 +1,64 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+my $srv = tcp_server();
+my $t0 = time;
+my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+
+sub slurp_hdr {
+ my ($c) = @_;
+ local $/ = "\r\n\r\n"; # affects both readline+chomp
+ chomp(my $hdr = readline($c));
+ my ($status, @hdr) = split(/\r\n/, $hdr);
+ diag explain([ $status, \@hdr ]) if $ENV{V};
+ ($status, \@hdr);
+}
+
+my ($c, $status, $hdr);
+
+# response header tests
+$c = tcp_connect($srv);
+print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+my $orig_200_status = $status;
+is_deeply([ grep(/^X-R2: /, @$hdr) ],
+ [ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
+ 'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+
+SKIP: { # Date header check
+ my @d = grep(/^Date: /i, @$hdr);
+ is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
+ eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
+ $d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
+ my $t = HTTP::Date::str2time($d[0]);
+ ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
+ diag(explain([$t, $!, \@d]));
+};
+
+# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
+$c = tcp_connect($srv);
+print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
+ 'nil header value accepted for broken apps') or diag(explain($hdr));
+
+if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
+ $c = tcp_connect($srv);
+ print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
+
+ $c = tcp_connect($srv);
+ print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ is($status, $orig_200_status, 'original status restored');
+}
+
+
+# ... more stuff here
+undef $ar;
+diag slurp("$tmpdir/err.log") if $ENV{V};
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
new file mode 100644
index 0000000..dd9c6b7
--- /dev/null
+++ b/t/lib.perl
@@ -0,0 +1,189 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+package UnicornTest;
+use v5.14;
+use parent qw(Exporter);
+use Test::More;
+use IO::Socket::INET;
+use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
+use File::Temp 0.19 (); # 0.19 for ->newdir
+our ($tmpdir, $errfh);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
+ SEEK_SET);
+
+my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
+$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+
+sub tcp_server {
+ my %opt = (
+ ReuseAddr => 1,
+ Proto => 'tcp',
+ Type => SOCK_STREAM,
+ Listen => SOMAXCONN,
+ Blocking => 0,
+ @_,
+ );
+ eval {
+ die 'IPv4-only' if $ENV{TEST_IPV4_ONLY};
+ require IO::Socket::INET6;
+ IO::Socket::INET6->new(%opt, LocalAddr => '[::1]')
+ } || eval {
+ die 'IPv6-only' if $ENV{TEST_IPV6_ONLY};
+ IO::Socket::INET->new(%opt, LocalAddr => '127.0.0.1')
+ } || BAIL_OUT "failed to create TCP server: $! ($@)";
+}
+
+sub tcp_host_port {
+ my ($s) = @_;
+ my ($h, $p) = ($s->sockhost, $s->sockport);
+ my $ipv4 = $s->sockdomain == AF_INET;
+ if (wantarray) {
+ $ipv4 ? ($h, $p) : ("[$h]", $p);
+ } else {
+ $ipv4 ? "$h:$p" : "[$h]:$p";
+ }
+}
+
+sub tcp_connect {
+ my ($dest, %opt) = @_;
+ my $addr = tcp_host_port($dest);
+ my $s = ref($dest)->new(
+ Proto => 'tcp',
+ Type => SOCK_STREAM,
+ PeerAddr => $addr,
+ %opt,
+ ) or BAIL_OUT "failed to connect to $addr: $!";
+ $s->autoflush(1);
+ $s;
+}
+
+sub slurp {
+ open my $fh, '<', $_[0] or die "open($_[0]): $!";
+ local $/;
+ <$fh>;
+}
+
+sub spawn {
+ my $env = ref($_[0]) eq 'HASH' ? shift : undef;
+ my $opt = ref($_[-1]) eq 'HASH' ? pop : {};
+ my @cmd = @_;
+ my $old = POSIX::SigSet->new;
+ my $set = POSIX::SigSet->new;
+ $set->fillset or die "sigfillset: $!";
+ sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
+ pipe(my ($r, $w)) or die "pipe: $!";
+ my $pid = fork // die "fork: $!";
+ if ($pid == 0) {
+ close $r;
+ $SIG{__DIE__} = sub {
+ warn(@_);
+ syswrite($w, my $num = $! + 0);
+ _exit(1);
+ };
+
+ # pretend to be systemd (cf. sd_listen_fds(3))
+ my $cfd;
+ for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
+ my $io = $opt->{$cfd} // next;
+ my $pfd = fileno($io) // die "fileno($io): $!";
+ if ($pfd == $cfd) {
+ fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+ } else {
+ dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
+ }
+ }
+ if (($cfd - 3) > 0) {
+ $env->{LISTEN_PID} = $$;
+ $env->{LISTEN_FDS} = $cfd - 3;
+ }
+
+ if (defined(my $pgid = $opt->{pgid})) {
+ setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
+ }
+ $SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
+ if (defined(my $cd = $opt->{-C})) {
+ chdir $cd // die "chdir($cd): $!";
+ }
+ $old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
+ sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
+ @ENV{keys %$env} = values(%$env) if $env;
+ exec { $cmd[0] } @cmd;
+ die "exec @cmd: $!";
+ }
+ close $w;
+ sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK(old): $!";
+ if (my $cerrnum = do { local $/, <$r> }) {
+ $! = $cerrnum;
+ die "@cmd PID=$pid died: $!";
+ }
+ $pid;
+}
+
+sub which {
+ my ($file) = @_;
+ return $file if index($file, '/') >= 0;
+ for my $p (split(/:/, $ENV{PATH})) {
+ $p .= "/$file";
+ return $p if -x $p;
+ }
+ undef;
+}
+
+# returns an AutoReap object
+sub unicorn {
+ my %env;
+ if (ref($_[0]) eq 'HASH') {
+ my $e = shift;
+ %env = %$e;
+ }
+ my @args = @_;
+ push(@args, {}) if ref($args[-1]) ne 'HASH';
+ $args[-1]->{2} //= $errfh; # stderr default
+
+ state $ruby = which($ENV{RUBY} // 'ruby');
+ state $lib = File::Spec->rel2abs('lib');
+ state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
+ state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
+ state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
+ state $exe = File::Spec->rel2abs('bin/unicorn');
+ my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
+ UnicornTest::AutoReap->new($pid);
+}
+
+# automatically kill + reap children when this goes out-of-scope
+package UnicornTest::AutoReap;
+use v5.14;
+
+sub new {
+ my (undef, $pid) = @_;
+ bless { pid => $pid, owner => $$ }, __PACKAGE__
+}
+
+sub kill {
+ my ($self, $sig) = @_;
+ CORE::kill($sig // 'TERM', $self->{pid});
+}
+
+sub join {
+ my ($self, $sig) = @_;
+ my $pid = delete $self->{pid} or return;
+ CORE::kill($sig, $pid) if defined $sig;
+ my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+ $ret == $pid or die "BUG: waitpid($pid) != $ret";
+}
+
+sub DESTROY {
+ my ($self) = @_;
+ return if $self->{owner} != $$;
+ $self->join('TERM');
+}
+
+package main; # inject ourselves into the t/*.t script
+UnicornTest->import;
+Test::More->import;
+# try to ensure ->DESTROY fires:
+$SIG{TERM} = sub { exit(15 + 128) };
+$SIG{INT} = sub { exit(2 + 128) };
+1;
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
deleted file mode 100644
index fbe433f..0000000
--- a/test/unit/test_response.rb
+++ /dev/null
@@ -1,111 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2005 Zed A. Shaw
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-#
-# Additional work donated by contributors. See git history
-# for more information.
-
-require './test/test_helper'
-require 'time'
-
-include Unicorn
-
-class ResponseTest < Test::Unit::TestCase
- include Unicorn::HttpResponse
-
- def test_httpdate
- before = Time.now.to_i - 1
- str = httpdate
- assert_kind_of(String, str)
- middle = Time.parse(str).to_i
- after = Time.now.to_i
- assert before <= middle
- assert middle <= after
- end
-
- def test_response_headers
- out = StringIO.new
- http_response_write(out, 200, {"X-Whatever" => "stuff"}, ["cool"])
- assert ! out.closed?
-
- assert out.length > 0, "output didn't have data"
- end
-
- # ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
- def test_response_header_broken_nil
- out = StringIO.new
- http_response_write(out, 200, {"Nil" => nil}, %w(hysterical raisin))
- assert ! out.closed?
-
- assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted'
- end
-
- def test_response_string_status
- out = StringIO.new
- http_response_write(out,'200', {}, [])
- assert ! out.closed?
- assert out.length > 0, "output didn't have data"
- end
-
- def test_response_200
- io = StringIO.new
- http_response_write(io, 200, {}, [])
- assert ! io.closed?
- assert io.length > 0, "output didn't have data"
- end
-
- def test_response_with_default_reason
- code = 400
- io = StringIO.new
- http_response_write(io, code, {}, [])
- assert ! io.closed?
- lines = io.string.split(/\r\n/)
- assert_match(/.* Bad Request$/, lines.first,
- "wrong default reason phrase")
- end
-
- def test_rack_multivalue_headers
- out = StringIO.new
- http_response_write(out,200, {"X-Whatever" => "stuff\nbleh"}, [])
- assert ! out.closed?
- assert_match(/^X-Whatever: stuff\r\nX-Whatever: bleh\r\n/, out.string)
- end
-
- # Even though Rack explicitly forbids "Status" in the header hash,
- # some broken clients still rely on it
- def test_status_header_added
- out = StringIO.new
- http_response_write(out,200, {"X-Whatever" => "stuff"}, [])
- assert ! out.closed?
- end
-
- def test_unknown_status_pass_through
- out = StringIO.new
- http_response_write(out,"666 I AM THE BEAST", {}, [] )
- assert ! out.closed?
- headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/)
- assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0])
- end
-
- def test_modified_rack_http_status_codes_late
- r, w = IO.pipe
- pid = fork do
- r.close
- # Users may want to globally override the status text associated
- # with an HTTP status code in their app.
- Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
- http_response_write(w, 200, {}, [])
- w.close
- end
- w.close
- assert_equal "HTTP/1.1 200 HI\r\n", r.gets
- r.read # just drain the pipe
- pid, status = Process.waitpid2(pid)
- assert status.success?, status.inspect
- ensure
- r.close
- w.close unless w.closed?
- end
-end
[-- Attachment #3: 0002-support-rack-3-multi-value-headers.patch --]
[-- Type: text/x-diff, Size: 1710 bytes --]
From ea0559c700fa029044464de4bd572662c10b7273 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:31 +0000
Subject: [PATCH 02/23] support rack 3 multi-value headers
The first step in adding Rack 3 support. Rack supports
multi-value headers via array rather than newlines.
Tested-by: Martin Posthumus <martin.posthumus@gmail.com>
Link: https://yhbt.net/unicorn-public/7c851d8a-bc57-7df8-3240-2f5ab831c47c@gmail.com/
---
t/integration.ru | 1 +
t/integration.t | 9 +++++++++
2 files changed, 10 insertions(+)
diff --git a/t/integration.ru b/t/integration.ru
index 6ef873c..5183217 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -23,6 +23,7 @@ def restore_status_code
when 'GET'
case env['PATH_INFO']
when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+ when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
end # case PATH_INFO (GET)
diff --git a/t/integration.t b/t/integration.t
index 5569155..e876c71 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -38,6 +38,15 @@ SKIP: { # Date header check
diag(explain([$t, $!, \@d]));
};
+
+$c = tcp_connect($srv);
+print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([ grep(/^x-r3: /, @$hdr) ],
+ [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
+ 'rack 3 array headers supported') or diag(explain($hdr));
+
+
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
$c = tcp_connect($srv);
print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
[-- Attachment #4: 0003-port-t0018-write-on-close.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4091 bytes --]
From 295a6c616f8840bc04617a377c04c3422aeebddc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:32 +0000
Subject: [PATCH 03/23] port t0018-write-on-close.sh to Perl 5
This doesn't require restarting, so it's a perfect candidate.
---
t/integration.ru | 15 +++++++++++++++
t/integration.t | 14 +++++++++++++-
t/lib.perl | 2 +-
t/t0018-write-on-close.sh | 23 -----------------------
t/write-on-close.ru | 11 -----------
5 files changed, 29 insertions(+), 36 deletions(-)
delete mode 100755 t/t0018-write-on-close.sh
delete mode 100644 t/write-on-close.ru
diff --git a/t/integration.ru b/t/integration.ru
index 5183217..12f5d48 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -18,6 +18,20 @@ def restore_status_code
[ 200, {}, [] ]
end
+class WriteOnClose
+ def each(&block)
+ @callback = block
+ end
+
+ def close
+ @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
+ end
+end
+
+def write_on_close
+ [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -26,6 +40,7 @@ def restore_status_code
when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ when '/write_on_close'; write_on_close
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index e876c71..3ab5c90 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -4,6 +4,7 @@
use v5.14; BEGIN { require './t/lib.perl' };
my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
my $t0 = time;
my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
@@ -66,8 +67,19 @@ if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
is($status, $orig_200_status, 'original status restored');
}
+SKIP: {
+ eval { require HTTP::Tiny } or skip "HTTP::Tiny missing: $@", 1;
+ my $ht = HTTP::Tiny->new;
+ my $res = $ht->get("http://$host_port/write_on_close");
+ is($res->{content}, 'Goodbye', 'write-on-close body read');
+}
# ... more stuff here
undef $ar;
-diag slurp("$tmpdir/err.log") if $ENV{V};
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+is_deeply(\@err, [], 'no unexpected errors in stderr');
+is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index dd9c6b7..12deaf8 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET);
+ SEEK_SET tcp_host_port);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/t0018-write-on-close.sh b/t/t0018-write-on-close.sh
deleted file mode 100755
index 3afefea..0000000
--- a/t/t0018-write-on-close.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 4 "write-on-close tests for funky response-bodies"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -D -c $unicorn_config write-on-close.ru
- unicorn_wait_start
-}
-
-t_begin "write-on-close response body succeeds" && {
- test xGoodbye = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
diff --git a/t/write-on-close.ru b/t/write-on-close.ru
deleted file mode 100644
index 725c4d6..0000000
--- a/t/write-on-close.ru
+++ /dev/null
@@ -1,11 +0,0 @@
-class WriteOnClose
- def each(&block)
- @callback = block
- end
-
- def close
- @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
- end
-end
-use Rack::ContentType, "text/plain"
-run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] })
[-- Attachment #5: 0004-port-t0000-http-basic.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3372 bytes --]
From 1bb4362cee167ac7aeec910d3f52419e391f1e61 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:33 +0000
Subject: [PATCH 04/23] port t0000-http-basic.sh to Perl 5
One more socat dependency down...
---
t/integration.ru | 16 ++++++++++++++
t/integration.t | 11 ++++++++++
t/t0000-http-basic.sh | 50 -------------------------------------------
3 files changed, 27 insertions(+), 50 deletions(-)
delete mode 100755 t/t0000-http-basic.sh
diff --git a/t/integration.ru b/t/integration.ru
index 12f5d48..c0bef99 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -32,6 +32,21 @@ def write_on_close
[ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
end
+def env_dump(env)
+ require 'json'
+ h = {}
+ env.each do |k,v|
+ case v
+ when String, Integer, true, false; h[k] = v
+ else
+ case k
+ when 'rack.version', 'rack.after_reply'; h[k] = v
+ end
+ end
+ end
+ h.to_json
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -40,6 +55,7 @@ def write_on_close
when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3ab5c90..ee22e7e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -47,6 +47,17 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
'rack 3 array headers supported') or diag(explain($hdr));
+SKIP: {
+ eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
+ $c = tcp_connect($srv);
+ print $c "GET /env_dump\r\n" or die $!;
+ my $json = do { local $/; readline($c) };
+ unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
+ unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
+ my $env = JSON::PP->new->decode($json);
+ is(ref($env), 'HASH', 'JSON decoded body to hashref');
+ is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
$c = tcp_connect($srv);
diff --git a/t/t0000-http-basic.sh b/t/t0000-http-basic.sh
deleted file mode 100755
index 8ab58ac..0000000
--- a/t/t0000-http-basic.sh
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 8 "simple HTTP connection tests"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -D -c $unicorn_config env.ru
- unicorn_wait_start
-}
-
-t_begin "single request" && {
- curl -sSfv http://$listen/
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_begin "HTTP/0.9 request should not return headers" && {
- (
- printf 'GET /\r\n'
- cat $fifo > $tmp &
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
-}
-
-t_begin "env.inspect should've put everything on one line" && {
- test 1 -eq $(count_lines < $tmp)
-}
-
-t_begin "no headers in output" && {
- if grep ^Connection: $tmp
- then
- die "Connection header found in $tmp"
- elif grep ^HTTP/ $tmp
- then
- die "HTTP/ found in $tmp"
- fi
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_done
[-- Attachment #6: 0005-port-t0002-parser-error.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4875 bytes --]
From 2eb7b1662c291ab535ee5dabf5d96194ca6483d4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:34 +0000
Subject: [PATCH 05/23] port t0002-parser-error.sh to Perl 5
Another socat dependency down...
---
t/integration.t | 33 +++++++++++++++
t/lib.perl | 9 +++-
t/t0002-parser-error.sh | 94 -----------------------------------------
3 files changed, 41 insertions(+), 95 deletions(-)
delete mode 100755 t/t0002-parser-error.sh
diff --git a/t/integration.t b/t/integration.t
index ee22e7e..503b7eb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -85,6 +85,39 @@ SKIP: {
is($res->{content}, 'Goodbye', 'write-on-close body read');
}
+if ('bad requests') {
+ $c = start_req($srv, 'GET /env_dump HTTP/1/1');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /' or die $!;
+ my $buf = join('', (0..9), 'ab');
+ for (0..1023) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!,
+ '414 on REQUEST_PATH > (12 * 1024)');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /hello-world?a' or die $!;
+ $buf = join('', (0..9));
+ for (0..1023) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!,
+ '414 on QUERY_STRING > (10 * 1024)');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /hello-world#a' or die $!;
+ $buf = join('', (0..9), 'a'..'f');
+ for (0..63) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
+}
+
+
# ... more stuff here
undef $ar;
my @log = slurp("$tmpdir/err.log");
diff --git a/t/lib.perl b/t/lib.perl
index 12deaf8..7d712b5 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port);
+ SEEK_SET tcp_host_port start_req);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -59,6 +59,13 @@ sub tcp_connect {
$s;
}
+sub start_req {
+ my ($srv, @req) = @_;
+ my $c = tcp_connect($srv);
+ print $c @req, "\r\n\r\n" or die "print: $!";
+ $c;
+}
+
sub slurp {
open my $fh, '<', $_[0] or die "open($_[0]): $!";
local $/;
diff --git a/t/t0002-parser-error.sh b/t/t0002-parser-error.sh
deleted file mode 100755
index 9dc1cd2..0000000
--- a/t/t0002-parser-error.sh
+++ /dev/null
@@ -1,94 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "parser error test"
-
-t_begin "setup and startup" && {
- unicorn_setup
- unicorn -D env.ru -c $unicorn_config
- unicorn_wait_start
-}
-
-t_begin "send a bad request" && {
- (
- printf 'GET / HTTP/1/1\r\nHost: example.com\r\n\r\n'
- cat $fifo > $tmp &
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
-}
-
-dbgcat tmp
-
-t_begin "response should be a 400" && {
- grep -F 'HTTP/1.1 400 Bad Request' $tmp
-}
-
-t_begin "send a huge Request URI (REQUEST_PATH > (12 * 1024))" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
- do
- printf '0123456789ab'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (REQUEST_PATH)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /hello-world?a'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
- do
- printf '0123456789'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (QUERY_STRING)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (FRAGMENT > 1024)" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /hello-world#a'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<64;i++) print i}')
- do
- printf '0123456789abcdef'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (FRAGMENT)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "server stderr should be clean" && check_stderr
-
-t_begin "term signal sent" && kill $unicorn_pid
-
-t_done
[-- Attachment #7: 0006-t-integration.t-use-start_req-to-simplify-test-sligh.patch --]
[-- Type: text/x-diff, Size: 2556 bytes --]
From 0bb06cc0c8c4f5b76514858067bbb2871dda0d6e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:35 +0000
Subject: [PATCH 06/23] t/integration.t: use start_req to simplify test slighly
Less code is usually better.
---
t/integration.t | 18 ++++++------------
1 file changed, 6 insertions(+), 12 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index 503b7eb..b7ba1fb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -20,8 +20,7 @@ sub slurp_hdr {
my ($c, $status, $hdr);
# response header tests
-$c = tcp_connect($srv);
-print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
@@ -40,8 +39,7 @@ SKIP: { # Date header check
};
-$c = tcp_connect($srv);
-print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -49,8 +47,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- $c = tcp_connect($srv);
- print $c "GET /env_dump\r\n" or die $!;
+ my $c = start_req($srv, 'GET /env_dump');
my $json = do { local $/; readline($c) };
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -60,20 +57,17 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_connect($srv);
-print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = tcp_connect($srv);
- print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+ $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = tcp_connect($srv);
- print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+ $c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is($status, $orig_200_status, 'original status restored');
}
[-- Attachment #8: 0007-port-t0011-active-unix-socket.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6945 bytes --]
From 10c83beaca58df8b92d8228e798559069cd89beb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:36 +0000
Subject: [PATCH 07/23] port t0011-active-unix-socket.sh to Perl 5
Another socat dependency down... I've also started turning
FD_CLOEXEC off on a pipe as a mechanism to detect daemonized
process death in tests.
---
t/active-unix-socket.t | 117 ++++++++++++++++++++++++++++++++++
t/integration.ru | 1 +
t/t0011-active-unix-socket.sh | 79 -----------------------
3 files changed, 118 insertions(+), 79 deletions(-)
create mode 100644 t/active-unix-socket.t
delete mode 100755 t/t0011-active-unix-socket.sh
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
new file mode 100644
index 0000000..6b5c218
--- /dev/null
+++ b/t/active-unix-socket.t
@@ -0,0 +1,117 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use IO::Socket::UNIX;
+my %to_kill;
+END { kill('TERM', values(%to_kill)) if keys %to_kill }
+my $u1 = "$tmpdir/u1.sock";
+my $u2 = "$tmpdir/u2.sock";
+my $unix_req = sub {
+ my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
+ print $s @_, "\r\n\r\n" or die $!;
+ $s;
+};
+{
+ use autodie;
+ open my $fh, '>', "$tmpdir/u1.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u1"
+stderr_path "$tmpdir/err1.log"
+EOM
+ close $fh;
+
+ open $fh, '>', "$tmpdir/u2.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u2"
+stderr_path "$tmpdir/err2.log"
+EOM
+ close $fh;
+
+ open $fh, '>', "$tmpdir/u3.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u3.pid"
+listen "$u1"
+stderr_path "$tmpdir/err3.log"
+EOM
+ close $fh;
+}
+
+my @uarg = qw(-D -E none t/integration.ru);
+
+# this pipe will be used to notify us when all daemons die:
+pipe(my ($p0, $p1)) or die "pipe: $!";
+fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+
+# start the first instance
+unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+is($?, 0, 'daemonized 1st process');
+chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
+ok(kill(0, $worker_pid), 'worker is kill-able');
+
+
+# 2nd process conflicts on PID
+unicorn('-c', "$tmpdir/u2.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting PID file fails to start');
+
+chomp(my $pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
+
+chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID unchanged');
+
+
+# 3rd process conflicts on socket
+unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting UNIX socket fails to start');
+
+chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID still unchanged');
+
+chomp($pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
+
+{ # teardown initial process via SIGKILL
+ ok(kill('KILL', delete $to_kill{u1}), 'SIGKILL initial daemon');
+ close $p1;
+ vec(my $rvec = '', fileno($p0), 1) = 1;
+ is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+ is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+ ok(-f "$tmpdir/u.pid", 'pid file stayed after SIGKILL');
+ ok(-S $u1, 'socket stayed after SIGKILL');
+ is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef,
+ 'fail to connect to u1');
+ ok(!kill(0, $worker_pid), 'worker gone after parent dies');
+}
+
+# restart the first instance
+{
+ pipe(($p0, $p1)) or die "pipe: $!";
+ fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+ unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+ is($?, 0, 'daemonized 1st process');
+ chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+ like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+ chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+ like($pid2, qr/\A\d+\z/, 'worker running');
+
+ ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
+ close $p1;
+ vec(my $rvec = '', fileno($p0), 1) = 1;
+ is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+ is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+ ok(!-f "$tmpdir/u.pid", 'pid file gone after SIGTERM');
+ ok(-S $u1, 'socket stays after SIGTERM');
+}
+
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+done_testing;
diff --git a/t/integration.ru b/t/integration.ru
index c0bef99..21f5449 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -57,6 +57,7 @@ def env_dump(env)
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
+ when '/pid'; [ 200, {}, [ "#$$\n" ] ]
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh
deleted file mode 100755
index fae0b6c..0000000
--- a/t/t0011-active-unix-socket.sh
+++ /dev/null
@@ -1,79 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "existing UNIX domain socket check"
-
-read_pid_unix () {
- x=$(printf 'GET / HTTP/1.0\r\n\r\n' | \
- socat - UNIX:$unix_socket | \
- tail -1)
- test -n "$x"
- y="$(expr "$x" : '\([0-9][0-9]*\)')"
- test x"$x" = x"$y"
- test -n "$y"
- echo "$y"
-}
-
-t_begin "setup and start" && {
- rtmpfiles unix_socket unix_config
- rm -f $unix_socket
- unicorn_setup
- grep -v ^listen < $unicorn_config > $unix_config
- echo "listen '$unix_socket'" >> $unix_config
- unicorn -D -c $unix_config pid.ru
- unicorn_wait_start
- orig_master_pid=$unicorn_pid
-}
-
-t_begin "get pid of worker" && {
- worker_pid=$(read_pid_unix)
- t_info "worker_pid=$worker_pid"
-}
-
-t_begin "fails to start with existing pid file" && {
- rm -f $ok
- unicorn -D -c $unix_config pid.ru || echo ok > $ok
- test x"$(cat $ok)" = xok
-}
-
-t_begin "worker pid unchanged" && {
- test x"$(read_pid_unix)" = x$worker_pid
- > $r_err
-}
-
-t_begin "fails to start with listening UNIX domain socket bound" && {
- rm $ok $pid
- unicorn -D -c $unix_config pid.ru || echo ok > $ok
- test x"$(cat $ok)" = xok
- > $r_err
-}
-
-t_begin "worker pid unchanged (again)" && {
- test x"$(read_pid_unix)" = x$worker_pid
-}
-
-t_begin "nuking the existing Unicorn succeeds" && {
- kill -9 $unicorn_pid
- while kill -0 $unicorn_pid
- do
- sleep 1
- done
- check_stderr
-}
-
-t_begin "succeeds in starting with leftover UNIX domain socket bound" && {
- test -S $unix_socket
- unicorn -D -c $unix_config pid.ru
- unicorn_wait_start
-}
-
-t_begin "worker pid changed" && {
- test x"$(read_pid_unix)" != x$worker_pid
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "no errors" && check_stderr
-
-t_done
[-- Attachment #9: 0008-port-t0100-rack-input-tests.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11722 bytes --]
From b4ed148186295f2d5c8448eab7f2b201615d1e4e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:37 +0000
Subject: [PATCH 08/23] port t0100-rack-input-tests.sh to Perl 5
Yet another socat dependency gone \o/
---
t/bin/content-md5-put | 36 -----------
t/integration.ru | 27 +++++++-
t/integration.t | 97 +++++++++++++++++++++++++++-
t/lib.perl | 3 +-
t/rack-input-tests.ru | 21 ------
t/t0100-rack-input-tests.sh | 124 ------------------------------------
6 files changed, 124 insertions(+), 184 deletions(-)
delete mode 100755 t/bin/content-md5-put
delete mode 100644 t/rack-input-tests.ru
delete mode 100755 t/t0100-rack-input-tests.sh
diff --git a/t/bin/content-md5-put b/t/bin/content-md5-put
deleted file mode 100755
index 01da0bb..0000000
--- a/t/bin/content-md5-put
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# simple chunked HTTP PUT request generator (and just that),
-# it reads stdin and writes to stdout so socat can write to a
-# UNIX or TCP socket (or to another filter or file) along with
-# a Content-MD5 trailer.
-require 'digest/md5'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-
-bs = ENV['bs'] ? ENV['bs'].to_i : 4096
-
-if ARGV.grep("--no-headers").empty?
- $stdout.write(
- "PUT / HTTP/1.1\r\n" \
- "Host: example.com\r\n" \
- "Transfer-Encoding: chunked\r\n" \
- "Trailer: Content-MD5\r\n" \
- "\r\n"
- )
-end
-
-digest = Digest::MD5.new
-if buf = $stdin.readpartial(bs)
- begin
- digest.update(buf)
- $stdout.write("%x\r\n" % [ buf.size ])
- $stdout.write(buf)
- $stdout.write("\r\n")
- end while $stdin.read(bs, buf)
-end
-
-digest = [ digest.digest ].pack('m').strip
-$stdout.write("0\r\n")
-$stdout.write("Content-MD5: #{digest}\r\n\r\n")
diff --git a/t/integration.ru b/t/integration.ru
index 21f5449..98528f6 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,29 @@ def env_dump(env)
h.to_json
end
+def rack_input_tests(env)
+ return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
+ cap = 16384
+ require 'digest/sha1'
+ digest = Digest::SHA1.new
+ input = env['rack.input']
+ case env['PATH_INFO']
+ when '/rack_input/size_first'; input.size
+ when '/rack_input/rewind_first'; input.rewind
+ when '/rack_input'; # OK
+ else
+ abort "bad path: #{env['PATH_INFO']}"
+ end
+ if buf = input.read(rand(cap))
+ begin
+ raise "#{buf.size} > #{cap}" if buf.size > cap
+ digest.update(buf)
+ end while input.read(rand(cap), buf)
+ end
+ [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
+ [ digest.hexdigest ] ]
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -66,6 +89,8 @@ def env_dump(env)
end # case PATH_INFO (POST)
# ...
when 'PUT'
- # ...
+ case env['PATH_INFO']
+ when %r{\A/rack_input}; rack_input_tests(env)
+ end
end # case REQUEST_METHOD
end) # run
diff --git a/t/integration.t b/t/integration.t
index b7ba1fb..8cef561 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,13 +1,16 @@
#!perl -w
# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# this is the main integration test for things which don't require
+# restarting or signals
use v5.14; BEGIN { require './t/lib.perl' };
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
-
+my $curl = which('curl');
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub slurp_hdr {
my ($c) = @_;
local $/ = "\r\n\r\n"; # affects both readline+chomp
@@ -17,6 +20,48 @@ sub slurp_hdr {
($status, \@hdr);
}
+my %PUT = (
+ chunked_md5 => sub {
+ my ($in, $out, $path, %opt) = @_;
+ my $bs = $opt{bs} // 16384;
+ require Digest::MD5;
+ my $dig = Digest::MD5->new;
+ print $out <<EOM;
+PUT $path HTTP/1.1\r
+Transfer-Encoding: chunked\r
+Trailer: Content-MD5\r
+\r
+EOM
+ my ($buf, $r);
+ while (1) {
+ $r = read($in, $buf, $bs) // die "read: $!";
+ last if $r == 0;
+ printf $out "%x\r\n", length($buf);
+ print $out $buf, "\r\n";
+ $dig->add($buf);
+ }
+ print $out "0\r\nContent-MD5: ", $dig->b64digest, "\r\n\r\n";
+ },
+ identity => sub {
+ my ($in, $out, $path, %opt) = @_;
+ my $bs = $opt{bs} // 16384;
+ my $clen = $opt{-s} // -s $in;
+ print $out <<EOM;
+PUT $path HTTP/1.0\r
+Content-Length: $clen\r
+\r
+EOM
+ my ($buf, $r, $len);
+ while ($clen) {
+ $len = $clen > $bs ? $bs : $clen;
+ $r = read($in, $buf, $len) // die "read: $!";
+ die 'premature EOF' if $r == 0;
+ print $out $buf;
+ $clen -= $r;
+ }
+ },
+);
+
my ($c, $status, $hdr);
# response header tests
@@ -111,6 +156,55 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
}
+# input tests
+my ($blob_size, $blob_hash);
+SKIP: {
+ open(my $rh, '<', 't/random_blob') or
+ skip "t/random_blob not generated $!", 1;
+ $blob_size = -s $rh;
+ require Digest::SHA;
+ $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+
+ my $ck_hash = sub {
+ my ($sub, $path, %opt) = @_;
+ seek($rh, 0, SEEK_SET) // die "seek: $!";
+ $c = tcp_connect($srv);
+ $c->autoflush(0);
+ $PUT{$sub}->($rh, $c, $path, %opt);
+ $c->flush or die "flush: $!";
+ ($status, $hdr) = slurp_hdr($c);
+ is(readline($c), $blob_hash, "$sub $path");
+ };
+ $ck_hash->('identity', '/rack_input', -s => $blob_size);
+ $ck_hash->('chunked_md5', '/rack_input');
+ $ck_hash->('identity', '/rack_input/size_first', -s => $blob_size);
+ $ck_hash->('identity', '/rack_input/rewind_first', -s => $blob_size);
+ $ck_hash->('chunked_md5', '/rack_input/size_first');
+ $ck_hash->('chunked_md5', '/rack_input/rewind_first');
+
+
+ $curl // skip 'no curl found in PATH', 1;
+
+ my ($copt, $cout);
+ my $url = "http://$host_port/rack_input";
+ my $do_curl = sub {
+ my (@arg) = @_;
+ pipe(my $cout, $copt->{1}) or die "pipe: $!";
+ open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+ my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
+ close(delete $copt->{1}) or die "close: $!";
+ is(readline($cout), $blob_hash, "curl @arg response");
+ is(waitpid($cpid, 0), $cpid, "curl @arg exited");
+ is($?, 0, "no error from curl @arg");
+ is(slurp("$tmpdir/curl.err"), '', "no stderr from curl @arg");
+ };
+
+ $do_curl->(qw(-T t/random_blob));
+
+ seek($rh, 0, SEEK_SET) // die "seek: $!";
+ $copt->{0} = $rh;
+ $do_curl->('-T-');
+}
# ... more stuff here
undef $ar;
@@ -120,4 +214,5 @@ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
is_deeply(\@err, [], 'no unexpected errors in stderr');
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+undef $tmpdir;
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 7d712b5..ae9f197 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req);
+ SEEK_SET tcp_host_port start_req which spawn);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -193,4 +193,5 @@ Test::More->import;
# try to ensure ->DESTROY fires:
$SIG{TERM} = sub { exit(15 + 128) };
$SIG{INT} = sub { exit(2 + 128) };
+$SIG{PIPE} = sub { exit(13 + 128) };
1;
diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru
deleted file mode 100644
index 5459e85..0000000
--- a/t/rack-input-tests.ru
+++ /dev/null
@@ -1,21 +0,0 @@
-# SHA1 checksum generator
-require 'digest/sha1'
-use Rack::ContentLength
-cap = 16384
-app = lambda do |env|
- /\A100-continue\z/i =~ env['HTTP_EXPECT'] and
- return [ 100, {}, [] ]
- digest = Digest::SHA1.new
- input = env['rack.input']
- input.size if env["PATH_INFO"] == "/size_first"
- input.rewind if env["PATH_INFO"] == "/rewind_first"
- if buf = input.read(rand(cap))
- begin
- raise "#{buf.size} > #{cap}" if buf.size > cap
- digest.update(buf)
- end while input.read(rand(cap), buf)
- end
-
- [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ]
-end
-run app
diff --git a/t/t0100-rack-input-tests.sh b/t/t0100-rack-input-tests.sh
deleted file mode 100755
index ee7a437..0000000
--- a/t/t0100-rack-input-tests.sh
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-test -r random_blob || die "random_blob required, run with 'make $0'"
-
-t_plan 10 "rack.input read tests"
-
-t_begin "setup and startup" && {
- rtmpfiles curl_out curl_err
- unicorn_setup
- unicorn -E none -D rack-input-tests.ru -c $unicorn_config
- blob_sha1=$(rsha1 < random_blob)
- blob_size=$(count_bytes < random_blob)
- t_info "blob_sha1=$blob_sha1"
- unicorn_wait_start
-}
-
-t_begin "corked identity request" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT / HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- content-md5-put < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#size first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /size_first HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#rewind first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /rewind_first HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#size first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /size_first HTTP/1.1\r\n'
- printf 'Host: example.com\r\n'
- printf 'Transfer-Encoding: chunked\r\n'
- printf 'Trailer: Content-MD5\r\n'
- printf '\r\n'
- content-md5-put --no-headers < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#rewind first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /rewind_first HTTP/1.1\r\n'
- printf 'Host: example.com\r\n'
- printf 'Transfer-Encoding: chunked\r\n'
- printf 'Trailer: Content-MD5\r\n'
- printf '\r\n'
- content-md5-put --no-headers < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "regular request" && {
- curl -sSf -T random_blob http://$listen/ > $curl_out 2> $curl_err
- test x$blob_sha1 = x$(cat $curl_out)
- test ! -s $curl_err
-}
-
-t_begin "chunked request" && {
- curl -sSf -T- < random_blob http://$listen/ > $curl_out 2> $curl_err
- test x$blob_sha1 = x$(cat $curl_out)
- test ! -s $curl_err
-}
-
-dbgcat r_err
-
-t_begin "shutdown" && {
- kill $unicorn_pid
-}
-
-t_done
[-- Attachment #10: 0009-tests-use-autodie-to-simplify-error-checking.patch --]
[-- Type: text/x-diff, Size: 8495 bytes --]
From 3a1d015a3859b639d8e4463e9436a49f4f0f720e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:38 +0000
Subject: [PATCH 09/23] tests: use autodie to simplify error checking
autodie is bundled with Perl 5.10+ and simplifies error
checking in most cases. Some subroutines aren't perfectly
translatable and their call sites had to be tweaked, but
most of them are.
---
t/active-unix-socket.t | 13 +++++++------
t/integration.t | 37 +++++++++++++++++++------------------
t/lib.perl | 30 +++++++++++++++---------------
3 files changed, 41 insertions(+), 39 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 6b5c218..1241904 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -4,17 +4,18 @@
use v5.14; BEGIN { require './t/lib.perl' };
use IO::Socket::UNIX;
+use autodie;
+no autodie 'kill';
my %to_kill;
END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
my $unix_req = sub {
my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
- print $s @_, "\r\n\r\n" or die $!;
+ print $s @_, "\r\n\r\n";
$s;
};
{
- use autodie;
open my $fh, '>', "$tmpdir/u1.conf.rb";
print $fh <<EOM;
pid "$tmpdir/u.pid"
@@ -43,8 +44,8 @@ EOM
my @uarg = qw(-D -E none t/integration.ru);
# this pipe will be used to notify us when all daemons die:
-pipe(my ($p0, $p1)) or die "pipe: $!";
-fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+pipe(my $p0, my $p1);
+fcntl($p1, POSIX::F_SETFD, 0);
# start the first instance
unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
@@ -93,8 +94,8 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
# restart the first instance
{
- pipe(($p0, $p1)) or die "pipe: $!";
- fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+ pipe($p0, $p1);
+ fcntl($p1, POSIX::F_SETFD, 0);
unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
is($?, 0, 'daemonized 1st process');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
diff --git a/t/integration.t b/t/integration.t
index 8cef561..af17d51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -5,6 +5,7 @@
# restarting or signals
use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
@@ -34,7 +35,7 @@ Trailer: Content-MD5\r
EOM
my ($buf, $r);
while (1) {
- $r = read($in, $buf, $bs) // die "read: $!";
+ $r = read($in, $buf, $bs);
last if $r == 0;
printf $out "%x\r\n", length($buf);
print $out $buf, "\r\n";
@@ -54,7 +55,7 @@ EOM
my ($buf, $r, $len);
while ($clen) {
$len = $clen > $bs ? $bs : $clen;
- $r = read($in, $buf, $len) // die "read: $!";
+ $r = read($in, $buf, $len);
die 'premature EOF' if $r == 0;
print $out $buf;
$clen -= $r;
@@ -130,28 +131,28 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
$c = tcp_connect($srv);
- print $c 'GET /' or die $!;
+ print $c 'GET /';
my $buf = join('', (0..9), 'ab');
- for (0..1023) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..1023) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on REQUEST_PATH > (12 * 1024)');
$c = tcp_connect($srv);
- print $c 'GET /hello-world?a' or die $!;
+ print $c 'GET /hello-world?a';
$buf = join('', (0..9));
- for (0..1023) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..1023) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on QUERY_STRING > (10 * 1024)');
$c = tcp_connect($srv);
- print $c 'GET /hello-world#a' or die $!;
+ print $c 'GET /hello-world#a';
$buf = join('', (0..9), 'a'..'f');
- for (0..63) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..63) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
}
@@ -159,7 +160,7 @@ if ('bad requests') {
# input tests
my ($blob_size, $blob_hash);
SKIP: {
- open(my $rh, '<', 't/random_blob') or
+ CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
require Digest::SHA;
@@ -167,11 +168,11 @@ SKIP: {
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
- seek($rh, 0, SEEK_SET) // die "seek: $!";
+ seek($rh, 0, SEEK_SET);
$c = tcp_connect($srv);
$c->autoflush(0);
$PUT{$sub}->($rh, $c, $path, %opt);
- $c->flush or die "flush: $!";
+ $c->flush or die $!;
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
};
@@ -189,10 +190,10 @@ SKIP: {
my $url = "http://$host_port/rack_input";
my $do_curl = sub {
my (@arg) = @_;
- pipe(my $cout, $copt->{1}) or die "pipe: $!";
- open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+ pipe(my $cout, $copt->{1});
+ open $copt->{2}, '>', "$tmpdir/curl.err";
my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
- close(delete $copt->{1}) or die "close: $!";
+ close(delete $copt->{1});
is(readline($cout), $blob_hash, "curl @arg response");
is(waitpid($cpid, 0), $cpid, "curl @arg exited");
is($?, 0, "no error from curl @arg");
@@ -201,7 +202,7 @@ SKIP: {
$do_curl->(qw(-T t/random_blob));
- seek($rh, 0, SEEK_SET) // die "seek: $!";
+ seek($rh, 0, SEEK_SET);
$copt->{0} = $rh;
$do_curl->('-T-');
}
diff --git a/t/lib.perl b/t/lib.perl
index ae9f197..49632cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -4,6 +4,7 @@
package UnicornTest;
use v5.14;
use parent qw(Exporter);
+use autodie;
use Test::More;
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
@@ -14,7 +15,7 @@ our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+open($errfh, '>>', "$tmpdir/err.log");
sub tcp_server {
my %opt = (
@@ -62,14 +63,14 @@ sub tcp_connect {
sub start_req {
my ($srv, @req) = @_;
my $c = tcp_connect($srv);
- print $c @req, "\r\n\r\n" or die "print: $!";
+ print $c @req, "\r\n\r\n";
$c;
}
sub slurp {
- open my $fh, '<', $_[0] or die "open($_[0]): $!";
+ open my $fh, '<', $_[0];
local $/;
- <$fh>;
+ readline($fh);
}
sub spawn {
@@ -80,8 +81,8 @@ sub spawn {
my $set = POSIX::SigSet->new;
$set->fillset or die "sigfillset: $!";
sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
- pipe(my ($r, $w)) or die "pipe: $!";
- my $pid = fork // die "fork: $!";
+ pipe(my $r, my $w);
+ my $pid = fork;
if ($pid == 0) {
close $r;
$SIG{__DIE__} = sub {
@@ -94,9 +95,9 @@ sub spawn {
my $cfd;
for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
my $io = $opt->{$cfd} // next;
- my $pfd = fileno($io) // die "fileno($io): $!";
+ my $pfd = fileno($io);
if ($pfd == $cfd) {
- fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+ fcntl($io, F_SETFD, 0);
} else {
dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
}
@@ -110,9 +111,7 @@ sub spawn {
setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
}
$SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
- if (defined(my $cd = $opt->{-C})) {
- chdir $cd // die "chdir($cd): $!";
- }
+ if (defined(my $cd = $opt->{-C})) { chdir $cd }
$old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
@ENV{keys %$env} = values(%$env) if $env;
@@ -162,22 +161,23 @@ sub unicorn {
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
+use autodie;
sub new {
my (undef, $pid) = @_;
bless { pid => $pid, owner => $$ }, __PACKAGE__
}
-sub kill {
+sub do_kill {
my ($self, $sig) = @_;
- CORE::kill($sig // 'TERM', $self->{pid});
+ kill($sig // 'TERM', $self->{pid});
}
sub join {
my ($self, $sig) = @_;
my $pid = delete $self->{pid} or return;
- CORE::kill($sig, $pid) if defined $sig;
- my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+ kill($sig, $pid) if defined $sig;
+ my $ret = waitpid($pid, 0);
$ret == $pid or die "BUG: waitpid($pid) != $ret";
}
[-- Attachment #11: 0010-port-t0019-max_header_len.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 5571 bytes --]
From 43c7d73b8b9e6995b5a986b10a8623395e89a538 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:39 +0000
Subject: [PATCH 10/23] port t0019-max_header_len.sh to Perl 5
This was the final socat requirement for integration tests.
I think curl will remain an optional dependency for tests
since it's probably the most widely-installed HTTP client.
---
GNUmakefile | 2 +-
t/README | 7 +-----
t/integration.ru | 1 +
t/integration.t | 43 +++++++++++++++++++++++++++++++---
| 49 ---------------------------------------
5 files changed, 43 insertions(+), 59 deletions(-)
delete mode 100755 t/t0019-max_header_len.sh
diff --git a/GNUmakefile b/GNUmakefile
index 5cca189..eab9082 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -125,7 +125,7 @@ $(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore
t/trash/.gitignore : | t/trash
echo '*' >$@
-dependencies := socat curl
+dependencies := curl
deps := $(addprefix t/.dep+,$(dependencies))
$(deps): dep_bin = $(lastword $(subst +, ,$@))
$(deps):
diff --git a/t/README b/t/README
index 8a5243e..d09c715 100644
--- a/t/README
+++ b/t/README
@@ -10,18 +10,13 @@ to test real-world behavior and Ruby introduces incompatibilities
at a far faster rate than Perl 5. Perl is Ruby's older cousin, so
it should be easy-to-learn for Rubyists.
-Old tests are in Bourne shell, but the socat(1) dependency was probably
-too rare compared to Perl 5.
+Old tests are in Bourne shell and slowly being ported to Perl 5.
== Requirements
* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
-
-The following requirements will eventually be dropped.
-
-* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
We do not use bashisms or any non-portable, non-POSIX constructs
diff --git a/t/integration.ru b/t/integration.ru
index 98528f6..edc408c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -81,6 +81,7 @@ def rack_input_tests(env)
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+ else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index af17d51..c687655 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,15 +1,19 @@
#!perl -w
# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
-# this is the main integration test for things which don't require
-# restarting or signals
+
+# This is the main integration test for fast-ish things to minimize
+# Ruby startup time penalties.
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
-my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+my $conf = "$tmpdir/u.conf.rb";
+open my $conf_fh, '>', $conf;
+$conf_fh->autoflush(1);
+my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub slurp_hdr {
@@ -207,7 +211,40 @@ SKIP: {
$do_curl->('-T-');
}
+
# ... more stuff here
+
+# SIGHUP-able stuff goes here
+
+if ('max_header_len internal API') {
+ undef $c;
+ my $req = 'GET / HTTP/1.0';
+ my $len = length($req."\r\n\r\n");
+ my $fifo = "$tmpdir/fifo";
+ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+ print $conf_fh <<EOM;
+Unicorn::HttpParser.max_header_len = $len
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+ $ar->do_kill('HUP');
+ open my $fifo_fh, '<', $fifo;
+ my $wpid = readline($fifo_fh);
+ like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+ close $fifo_fh;
+ $wpid =~ s/\Apid=// or die;
+ ok(CORE::kill(0, $wpid), 'worker PID retrieved');
+
+ $c = start_req($srv, $req);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
+
+ $c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
+}
+
+
undef $ar;
my @log = slurp("$tmpdir/err.log");
diag("@log") if $ENV{V};
diff --git a/t/t0019-max_header_len.sh b/t/t0019-max_header_len.sh
deleted file mode 100755
index 6a355b4..0000000
--- a/t/t0019-max_header_len.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 5 "max_header_len setting (only intended for Rainbows!)"
-
-t_begin "setup and start" && {
- unicorn_setup
- req='GET / HTTP/1.0\r\n\r\n'
- len=$(printf "$req" | count_bytes)
- echo Unicorn::HttpParser.max_header_len = $len >> $unicorn_config
- unicorn -D -c $unicorn_config env.ru
- unicorn_wait_start
-}
-
-t_begin "minimal request succeeds" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf "$req"
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
-
- fgrep "HTTP/1.1 200 OK" $tmp
-}
-
-t_begin "big request fails" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'GET /xxxxxx HTTP/1.0\r\n\r\n'
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
- fgrep "HTTP/1.1 413" $tmp
-}
-
-dbgcat tmp
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
[-- Attachment #12: 0011-test_exec-drop-sd_listen_fds-emulation-test.patch --]
[-- Type: text/x-diff, Size: 1751 bytes --]
From 5d828a4ef7683345bcf2ff659442fed0a6fb7a97 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:40 +0000
Subject: [PATCH 11/23] test_exec: drop sd_listen_fds emulation test
The Perl 5 tests already rely on this implicitly, and there was
never a point when Perl 5 couldn't emulate systemd behavior.
---
test/exec/test_exec.rb | 33 ---------------------------------
1 file changed, 33 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 2929b2e..1d3a0fd 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -97,39 +97,6 @@ def teardown
end
end
- def test_sd_listen_fds_emulation
- # [ruby-core:69895] [Bug #11336] fixed by r51576
- return if RUBY_VERSION.to_f < 2.3
-
- File.open("config.ru", "wb") { |fp| fp.write(HI) }
- sock = TCPServer.new(@addr, @port)
-
- [ %W(-l #@addr:#@port), nil ].each do |l|
- sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
- pid = xfork do
- redirect_test_io do
- # pretend to be systemd
- ENV['LISTEN_PID'] = "#$$"
- ENV['LISTEN_FDS'] = '1'
-
- # 3 = SD_LISTEN_FDS_START
- args = [ $unicorn_bin ]
- args.concat(l) if l
- args << { 3 => sock }
- exec(*args)
- end
- end
- res = hit(["http://#@addr:#@port/"])
- assert_equal [ "HI\n" ], res
- assert_shutdown(pid)
- assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
- 'unicorn should always set SO_KEEPALIVE on inherited sockets'
- end
- ensure
- sock.close if sock
- end
-
def test_inherit_listener_unspecified
File.open("config.ru", "wb") { |fp| fp.write(HI) }
sock = TCPServer.new(@addr, @port)
[-- Attachment #13: 0012-test_exec-drop-test_basic-and-test_config_ru_alt_pat.patch --]
[-- Type: text/x-diff, Size: 1667 bytes --]
From 548593c6b3d52a4bebd52542ad9c423ed2b7252d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:41 +0000
Subject: [PATCH 12/23] test_exec: drop test_basic and test_config_ru_alt_path
We already have coverage for these basic things elsewhere.
---
test/exec/test_exec.rb | 24 ------------------------
1 file changed, 24 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 1d3a0fd..55f828e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -265,16 +265,6 @@ def test_exit_signals
end
end
- def test_basic
- File.open("config.ru", "wb") { |fp| fp.syswrite(HI) }
- pid = fork do
- redirect_test_io { exec($unicorn_bin, "-l", "#{@addr}:#{@port}") }
- end
- results = retry_hit(["http://#{@addr}:#{@port}/"])
- assert_equal String, results[0].class
- assert_shutdown(pid)
- end
-
def test_rack_env_unset
File.open("config.ru", "wb") { |fp| fp.syswrite(SHOW_RACK_ENV) }
pid = fork { redirect_test_io { exec($unicorn_bin, "-l#@addr:#@port") } }
@@ -638,20 +628,6 @@ def test_read_embedded_cli_switches
assert_shutdown(pid)
end
- def test_config_ru_alt_path
- config_path = "#{@tmpdir}/foo.ru"
- File.open(config_path, "wb") { |fp| fp.syswrite(HI) }
- pid = fork do
- redirect_test_io do
- Dir.chdir("/")
- exec($unicorn_bin, "-l#{@addr}:#{@port}", config_path)
- end
- end
- results = retry_hit(["http://#{@addr}:#{@port}/"])
- assert_equal String, results[0].class
- assert_shutdown(pid)
- end
-
def test_load_module
libdir = "#{@tmpdir}/lib"
FileUtils.mkpath([ libdir ])
[-- Attachment #14: 0013-tests-check_stderr-consistently-in-Perl-5-tests.patch --]
[-- Type: text/x-diff, Size: 2415 bytes --]
From cd7ee67fc8ebadec9bdd913d49ed3f214596ea47 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:42 +0000
Subject: [PATCH 13/23] tests: check_stderr consistently in Perl 5 tests
The Bourne shell tests did, so lets not let stuff sneak past us.
---
t/active-unix-socket.t | 5 ++---
t/integration.t | 7 ++-----
t/lib.perl | 10 +++++++++-
3 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 1241904..c132dc2 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -20,7 +20,7 @@ my $unix_req = sub {
print $fh <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
-stderr_path "$tmpdir/err1.log"
+stderr_path "$tmpdir/err.log"
EOM
close $fh;
@@ -113,6 +113,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
ok(-S $u1, 'socket stays after SIGTERM');
}
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
+check_stderr;
done_testing;
diff --git a/t/integration.t b/t/integration.t
index c687655..939dc24 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -246,11 +246,8 @@ EOM
undef $ar;
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
-my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
-is_deeply(\@err, [], 'no unexpected errors in stderr');
-is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
+check_stderr;
undef $tmpdir;
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 49632cf..315ef2d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,12 +11,20 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req which spawn);
+ SEEK_SET tcp_host_port start_req which spawn check_stderr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
open($errfh, '>>', "$tmpdir/err.log");
+sub check_stderr () {
+ my @log = slurp("$tmpdir/err.log");
+ diag("@log") if $ENV{V};
+ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+ is_deeply(\@err, [], 'no unexpected errors in stderr');
+ is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
[-- Attachment #15: 0014-tests-consistent-tcp_start-and-unix_start-across-Per.patch --]
[-- Type: text/x-diff, Size: 8017 bytes --]
From 0dcd8bd569813a175ad43837db3ab07019a95b99 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:43 +0000
Subject: [PATCH 14/23] tests: consistent tcp_start and unix_start across Perl
5 tests
I'll be using Unix sockets more in tests since there's no
risk of system-wide conflicts with TCP port allocation.
Furthermore, curl supports `--unix-socket' nowadays; so
there's little reason to rely on TCP sockets and the conflicts
they bring in tests.
---
t/active-unix-socket.t | 13 ++++---------
t/integration.t | 28 ++++++++++++++--------------
t/lib.perl | 30 ++++++++++++++++--------------
3 files changed, 34 insertions(+), 37 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index c132dc2..8723137 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -10,11 +10,6 @@ my %to_kill;
END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
-my $unix_req = sub {
- my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
- print $s @_, "\r\n\r\n";
- $s;
-};
{
open my $fh, '>', "$tmpdir/u1.conf.rb";
print $fh <<EOM;
@@ -53,7 +48,7 @@ is($?, 0, 'daemonized 1st process');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
-chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $worker_pid = readline(unix_start($u1, 'GET /pid')));
like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
ok(kill(0, $worker_pid), 'worker is kill-able');
@@ -65,7 +60,7 @@ isnt($?, 0, 'conflicting PID file fails to start');
chomp(my $pidf = slurp("$tmpdir/u.pid"));
is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
-chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $pid2 = readline(unix_start($u1, 'GET /pid')));
is($worker_pid, $pid2, 'worker PID unchanged');
@@ -73,7 +68,7 @@ is($worker_pid, $pid2, 'worker PID unchanged');
unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
isnt($?, 0, 'conflicting UNIX socket fails to start');
-chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
is($worker_pid, $pid2, 'worker PID still unchanged');
chomp($pidf = slurp("$tmpdir/u.pid"));
@@ -101,7 +96,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
- chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+ chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
like($pid2, qr/\A\d+\z/, 'worker running');
ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
diff --git a/t/integration.t b/t/integration.t
index 939dc24..b33e3c3 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -70,7 +70,7 @@ EOM
my ($c, $status, $hdr);
# response header tests
-$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
@@ -89,7 +89,7 @@ SKIP: { # Date header check
};
-$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -97,7 +97,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- my $c = start_req($srv, 'GET /env_dump');
+ my $c = tcp_start($srv, 'GET /env_dump');
my $json = do { local $/; readline($c) };
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -107,17 +107,17 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
+$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
+ $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
+ $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is($status, $orig_200_status, 'original status restored');
}
@@ -130,12 +130,12 @@ SKIP: {
}
if ('bad requests') {
- $c = start_req($srv, 'GET /env_dump HTTP/1/1');
+ $c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
- $c = tcp_connect($srv);
- print $c 'GET /';
+ $c = tcp_start($srv);
+ print $c 'GET /';;
my $buf = join('', (0..9), 'ab');
for (0..1023) { print $c $buf }
print $c " HTTP/1.0\r\n\r\n";
@@ -143,7 +143,7 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on REQUEST_PATH > (12 * 1024)');
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
print $c 'GET /hello-world?a';
$buf = join('', (0..9));
for (0..1023) { print $c $buf }
@@ -152,7 +152,7 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on QUERY_STRING > (10 * 1024)');
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
print $c 'GET /hello-world#a';
$buf = join('', (0..9), 'a'..'f');
for (0..63) { print $c $buf }
@@ -173,7 +173,7 @@ SKIP: {
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
seek($rh, 0, SEEK_SET);
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
$c->autoflush(0);
$PUT{$sub}->($rh, $c, $path, %opt);
$c->flush or die $!;
@@ -235,11 +235,11 @@ EOM
$wpid =~ s/\Apid=// or die;
ok(CORE::kill(0, $wpid), 'worker PID retrieved');
- $c = start_req($srv, $req);
+ $c = tcp_start($srv, $req);
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
- $c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+ $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
}
diff --git a/t/lib.perl b/t/lib.perl
index 315ef2d..1d6e78d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,8 +10,8 @@ use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req which spawn check_stderr);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -55,26 +55,28 @@ sub tcp_host_port {
}
}
-sub tcp_connect {
- my ($dest, %opt) = @_;
- my $addr = tcp_host_port($dest);
- my $s = ref($dest)->new(
+sub unix_start ($@) {
+ my ($dst, @req) = @_;
+ my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
+ BAIL_OUT "unix connect $dst: $!";
+ $s->autoflush(1);
+ print $s @req, "\r\n\r\n" if @req;
+ $s;
+}
+
+sub tcp_start ($@) {
+ my ($dst, @req) = @_;
+ my $addr = tcp_host_port($dst);
+ my $s = ref($dst)->new(
Proto => 'tcp',
Type => SOCK_STREAM,
PeerAddr => $addr,
- %opt,
) or BAIL_OUT "failed to connect to $addr: $!";
$s->autoflush(1);
+ print $s @req, "\r\n\r\n" if @req;
$s;
}
-sub start_req {
- my ($srv, @req) = @_;
- my $c = tcp_connect($srv);
- print $c @req, "\r\n\r\n";
- $c;
-}
-
sub slurp {
open my $fh, '<', $_[0];
local $/;
[-- Attachment #16: 0015-port-t9000-preread-input.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3856 bytes --]
From 1b8840d8d13491eecd2fa92e06f73c65eadd33ba Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:44 +0000
Subject: [PATCH 15/23] port t9000-preread-input.sh to Perl 5
Stuffing it into t/integration.t for now so we can save on
startup costs.
---
t/integration.t | 32 ++++++++++++++++++++++++---
t/lib.perl | 2 +-
t/preread_input.ru | 4 +---
t/t9000-preread-input.sh | 48 ----------------------------------------
4 files changed, 31 insertions(+), 55 deletions(-)
delete mode 100755 t/t9000-preread-input.sh
diff --git a/t/integration.t b/t/integration.t
index b33e3c3..f5afd5d 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,8 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-my $srv = tcp_server();
-my $host_port = tcp_host_port($srv);
+our $srv = tcp_server();
+our $host_port = tcp_host_port($srv);
my $t0 = time;
my $conf = "$tmpdir/u.conf.rb";
open my $conf_fh, '>', $conf;
@@ -209,8 +209,34 @@ SKIP: {
seek($rh, 0, SEEK_SET);
$copt->{0} = $rh;
$do_curl->('-T-');
-}
+ diag 'testing Unicorn::PrereadInput...';
+ local $srv = tcp_server();
+ local $host_port = tcp_host_port($srv);
+ check_stderr;
+ truncate($errfh, 0);
+
+ my $pri = unicorn(qw(-E none t/preread_input.ru), { 3 => $srv });
+ $url = "http://$host_port/";
+
+ $do_curl->(qw(-T t/random_blob));
+ seek($rh, 0, SEEK_SET);
+ $copt->{0} = $rh;
+ $do_curl->('-T-');
+
+ my @pr_err = slurp("$tmpdir/err.log");
+ is(scalar(grep(/app dispatch:/, @pr_err)), 2, 'app dispatched twice');
+
+ # abort a chunked request by blocking curl on a FIFO:
+ $c = tcp_start($srv, "PUT / HTTP/1.1\r\nTransfer-Encoding: chunked");
+ close $c;
+ @pr_err = slurp("$tmpdir/err.log");
+ is(scalar(grep(/app dispatch:/, @pr_err)), 2,
+ 'app did not dispatch on aborted request');
+ undef $pri;
+ check_stderr;
+ diag 'Unicorn::PrereadInput middleware tests done';
+}
# ... more stuff here
diff --git a/t/lib.perl b/t/lib.perl
index 1d6e78d..b6148cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -79,7 +79,7 @@ sub tcp_start ($@) {
sub slurp {
open my $fh, '<', $_[0];
- local $/;
+ local $/ if !wantarray;
readline($fh);
}
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 79685c4..f0a1748 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,8 +1,6 @@
#\-E none
require 'digest/sha1'
require 'unicorn/preread_input'
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
use Unicorn::PrereadInput
nr = 0
run lambda { |env|
@@ -13,5 +11,5 @@
dig.update(buf)
end
- [ 200, {}, [ "#{dig.hexdigest}\n" ] ]
+ [ 200, {}, [ dig.hexdigest ] ]
}
diff --git a/t/t9000-preread-input.sh b/t/t9000-preread-input.sh
deleted file mode 100755
index d6c73ab..0000000
--- a/t/t9000-preread-input.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 9 "PrereadInput middleware tests"
-
-t_begin "setup and start" && {
- random_blob_sha1=$(rsha1 < random_blob)
- unicorn_setup
- unicorn -D -c $unicorn_config preread_input.ru
- unicorn_wait_start
-}
-
-t_begin "single identity request" && {
- curl -sSf -T random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
- test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "single chunked request" && {
- curl -sSf -T- < random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
- test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "app only dispatched twice" && {
- test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "aborted chunked request" && {
- rm -f $tmp
- curl -sSf -T- < $fifo http://$listen/ > $tmp &
- curl_pid=$!
- kill -9 $curl_pid
- wait
-}
-
-t_begin "app only dispatched twice" && {
- test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "killing succeeds" && {
- kill -QUIT $unicorn_pid
-}
-
-t_done
[-- Attachment #17: 0016-port-t-t0116-client_body_buffer_size.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8861 bytes --]
From e9593301044f305d4a0e074f77eea35015ca0ec4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:45 +0000
Subject: [PATCH 16/23] port t/t0116-client_body_buffer_size.sh to Perl 5
While I'm fine with depending on curl for certain things,
there's no need for it here since unicorn has had lazy
rack.input for over a decade, at this point.
---
t/active-unix-socket.t | 1 +
t/{t0116.ru => client_body_buffer_size.ru} | 2 -
t/client_body_buffer_size.t | 83 ++++++++++++++++++++++
t/integration.t | 10 ---
t/lib.perl | 12 +++-
t/t0116-client_body_buffer_size.sh | 80 ---------------------
6 files changed, 95 insertions(+), 93 deletions(-)
rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
create mode 100644 t/client_body_buffer_size.t
delete mode 100755 t/t0116-client_body_buffer_size.sh
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 8723137..4e11837 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -109,4 +109,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
}
check_stderr;
+undef $tmpdir;
done_testing;
diff --git a/t/t0116.ru b/t/client_body_buffer_size.ru
similarity index 82%
rename from t/t0116.ru
rename to t/client_body_buffer_size.ru
index fab5fce..44161a5 100644
--- a/t/t0116.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,6 +1,4 @@
#\ -E none
-use Rack::ContentLength
-use Rack::ContentType, 'text/plain'
app = lambda do |env|
input = env['rack.input']
case env["PATH_INFO"]
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
new file mode 100644
index 0000000..b1a99f3
--- /dev/null
+++ b/t/client_body_buffer_size.t
@@ -0,0 +1,83 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $uconf = "$tmpdir/u.conf.rb";
+
+open my $conf_fh, '>', $uconf;
+$conf_fh->autoflush(1);
+print $conf_fh <<EOM;
+client_body_buffer_size 0
+EOM
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my $ar = unicorn(@uarg, { 3 => $srv });
+my ($c, $status, $hdr);
+my $mem_class = 'StringIO';
+my $fs_class = 'Unicorn::TmpIO';
+
+$c = tcp_start($srv, "PUT /input_class HTTP/1.0\r\nContent-Length: 0");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'zero-byte file is StringIO');
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+print $c '.';
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 byte file is filesystem-backed');
+
+
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+seek($conf_fh, 0, SEEK_SET);
+truncate($conf_fh, 0);
+print $conf_fh <<EOM;
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+$ar->do_kill('HUP');
+open my $fifo_fh, '<', $fifo;
+like(my $wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+ 'reloaded w/ default client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'class for a 1 byte file is memory-backed');
+
+
+my $one_meg = 1024 ** 2;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte file is FS-backed');
+
+# reload with bigger client_body_buffer_size
+say $conf_fh "client_body_buffer_size $one_meg";
+$ar->do_kill('HUP');
+open $fifo_fh, '<', $fifo;
+like($wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+ 'reloaded w/ bigger client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, '1 megabyte file is now memory-backed');
+
+my $too_big = $one_meg + 1;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $too_big");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte + 1 byte file is FS-backed');
+
+
+undef $ar;
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/integration.t b/t/integration.t
index f5afd5d..855c260 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -15,16 +15,6 @@ open my $conf_fh, '>', $conf;
$conf_fh->autoflush(1);
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
-sub slurp_hdr {
- my ($c) = @_;
- local $/ = "\r\n\r\n"; # affects both readline+chomp
- chomp(my $hdr = readline($c));
- my ($status, @hdr) = split(/\r\n/, $hdr);
- diag explain([ $status, \@hdr ]) if $ENV{V};
- ($status, \@hdr);
-}
-
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index b6148cf..2685c3b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,11 +11,12 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port which spawn check_stderr unix_start);
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
open($errfh, '>>', "$tmpdir/err.log");
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub check_stderr () {
my @log = slurp("$tmpdir/err.log");
@@ -25,6 +26,15 @@ sub check_stderr () {
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
}
+sub slurp_hdr {
+ my ($c) = @_;
+ local $/ = "\r\n\r\n"; # affects both readline+chomp
+ chomp(my $hdr = readline($c));
+ my ($status, @hdr) = split(/\r\n/, $hdr);
+ diag explain([ $status, \@hdr ]) if $ENV{V};
+ ($status, \@hdr);
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
diff --git a/t/t0116-client_body_buffer_size.sh b/t/t0116-client_body_buffer_size.sh
deleted file mode 100755
index c9e17c7..0000000
--- a/t/t0116-client_body_buffer_size.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 12 "client_body_buffer_size settings"
-
-t_begin "setup and start" && {
- unicorn_setup
- rtmpfiles unicorn_config_tmp one_meg
- dd if=/dev/zero bs=1M count=1 of=$one_meg
- cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
- File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
-end
-EOF
- cat $unicorn_config > $unicorn_config_tmp
- echo client_body_buffer_size 0 >> $unicorn_config
- unicorn -D -c $unicorn_config t0116.ru
- unicorn_wait_start
- fs_class=Unicorn::TmpIO
- mem_class=StringIO
-
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a zero-byte file should be StringIO" && {
- > $tmp
- test xStringIO = x"$(curl -T $tmp -sSf http://$listen/input_class)"
-}
-
-t_begin "class for a 1 byte file should be filesystem-backed" && {
- echo > $tmp
- test x$fs_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "reload with default client_body_buffer_size" && {
- mv $unicorn_config_tmp $unicorn_config
- kill -HUP $unicorn_pid
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a 1 byte file should be memory-backed" && {
- echo > $tmp
- test x$mem_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "class for a random blob file should be filesystem-backed" && {
- resp="$(curl -T random_blob -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "one megabyte file should be filesystem-backed" && {
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "reload with a big client_body_buffer_size" && {
- echo "client_body_buffer_size(1024 * 1024)" >> $unicorn_config
- kill -HUP $unicorn_pid
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "one megabyte file should be memory-backed" && {
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$mem_class = x"$resp"
-}
-
-t_begin "one megabyte + 1 byte file should be filesystem-backed" && {
- echo >> $one_meg
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
[-- Attachment #18: 0017-tests-get-rid-of-sha1sum.rb-and-rsha1-sh-function.patch --]
[-- Type: text/x-diff, Size: 1255 bytes --]
From b47912160f2336dde3901e588cc23fb2c2f8d9dc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:46 +0000
Subject: [PATCH 17/23] tests: get rid of sha1sum.rb and rsha1() sh function
These are no longer needed since Perl has long included
Digest::SHA
---
t/bin/sha1sum.rb | 17 -----------------
t/test-lib.sh | 4 ----
2 files changed, 21 deletions(-)
delete mode 100755 t/bin/sha1sum.rb
diff --git a/t/bin/sha1sum.rb b/t/bin/sha1sum.rb
deleted file mode 100755
index 53d68ce..0000000
--- a/t/bin/sha1sum.rb
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# Reads from stdin and outputs the SHA1 hex digest of the input
-
-require 'digest/sha1'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-bs = 16384
-digest = Digest::SHA1.new
-if buf = $stdin.read(bs)
- begin
- digest.update(buf)
- end while $stdin.read(bs, buf)
-end
-
-$stdout.syswrite("#{digest.hexdigest}\n")
diff --git a/t/test-lib.sh b/t/test-lib.sh
index e70d0c6..8613144 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -123,7 +123,3 @@ unicorn_wait_start () {
# no need to play tricks with FIFOs since we got "ready_pipe" now
unicorn_pid=$(cat $pid)
}
-
-rsha1 () {
- sha1sum.rb
-}
[-- Attachment #19: 0018-early_hints-supports-Rack-3-array-headers.patch --]
[-- Type: text/x-diff, Size: 4606 bytes --]
From 6ad9f4b54ee16ffecea7e16b710552b45db33a16 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:47 +0000
Subject: [PATCH 18/23] early_hints supports Rack 3 array headers
We can hoist out append_headers into a new method and use it in
both e103_response_write and http_response_write.
t/integration.t now tests early_hints with both possible
values of check_client_connection.
---
t/integration.ru | 7 +++++++
t/integration.t | 47 ++++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 49 insertions(+), 5 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index edc408c..dab384d 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -5,6 +5,11 @@
# this goes for t/integration.t We'll try to put as many tests
# in here as possible to avoid startup overhead of Ruby.
+def early_hints(env, val)
+ env['rack.early_hints'].call('link' => val) # val may be ary or string
+ [ 200, {}, [ val.class.to_s ] ]
+end
+
$orig_rack_200 = nil
def tweak_status_code
$orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
@@ -81,6 +86,8 @@ def rack_input_tests(env)
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+ when '/early_hints_rack2'; early_hints(env, "r\n2")
+ when '/early_hints_rack3'; early_hints(env, %w(r 3))
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 855c260..8433497 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -13,8 +13,16 @@ my $t0 = time;
my $conf = "$tmpdir/u.conf.rb";
open my $conf_fh, '>', $conf;
$conf_fh->autoflush(1);
+my $u1 = "$tmpdir/u1";
+print $conf_fh <<EOM;
+early_hints true
+listen "$u1"
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
@@ -102,6 +110,26 @@ $c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
+my $ck_early_hints = sub {
+ my ($note) = @_;
+ $c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 2 value');
+ is_deeply(['link: r', 'link: 2'], $hdr, 'rack 2 hints match '.$note);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+ is(readline($c), 'String', 'early hints used a String for rack 2');
+
+ $c = unix_start($u1, 'GET /early_hints_rack3 HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 3');
+ is_deeply(['link: r', 'link: 3'], $hdr, 'rack 3 hints match '.$note);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+ is(readline($c), 'Array', 'early hints used a String for rack 3');
+};
+$ck_early_hints->('ccc off'); # we'll retest later
+
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
@@ -154,6 +182,7 @@ if ('bad requests') {
# input tests
my ($blob_size, $blob_hash);
SKIP: {
+ skip 'SKIP_EXPENSIVE on', 1 if $ENV{SKIP_EXPENSIVE};
CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
@@ -232,16 +261,24 @@ SKIP: {
# SIGHUP-able stuff goes here
+if ('check_client_connection') {
+ print $conf_fh <<EOM; # appending to existing
+check_client_connection true
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+ $ar->do_kill('HUP');
+ open my $fifo_fh, '<', $fifo;
+ my $wpid = readline($fifo_fh);
+ like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+ $ck_early_hints->('ccc on');
+}
+
if ('max_header_len internal API') {
undef $c;
my $req = 'GET / HTTP/1.0';
my $len = length($req."\r\n\r\n");
- my $fifo = "$tmpdir/fifo";
- POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
- print $conf_fh <<EOM;
+ print $conf_fh <<EOM; # appending to existing
Unicorn::HttpParser.max_header_len = $len
-listen "$host_port" # TODO: remove this requirement for SIGHUP
-after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
EOM
$ar->do_kill('HUP');
open my $fifo_fh, '<', $fifo;
[-- Attachment #20: 0019-test_server-drop-early_hints-test.patch --]
[-- Type: text/x-diff, Size: 1737 bytes --]
From 3e6bc9fb589fd88469349a38a77704c3333623e0 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:48 +0000
Subject: [PATCH 19/23] test_server: drop early_hints test
t/integration.t already is more complete in that it tests
both Rack 2 and 3 along with both possible values of
check_client_connection.
---
test/unit/test_server.rb | 31 -------------------------------
1 file changed, 31 deletions(-)
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index fe98fcc..0a710d1 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,17 +23,6 @@ def call(env)
end
end
-class TestEarlyHintsHandler
- def call(env)
- while env['rack.input'].read(4096)
- end
- env['rack.early_hints'].call(
- "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
- )
- [200, { 'content-type' => 'text/plain' }, ['hello!\n']]
- end
-end
-
class TestRackAfterReply
def initialize
@called = false
@@ -112,26 +101,6 @@ def test_preload_app_config
tmp.close!
end
- def test_early_hints
- teardown
- redirect_test_io do
- @server = HttpServer.new(TestEarlyHintsHandler.new,
- :listeners => [ "127.0.0.1:#@port"],
- :early_hints => true)
- @server.start
- end
-
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("GET / HTTP/1.0\r\n\r\n")
-
- responses = sock.read(4096)
- assert_match %r{\AHTTP/1.[01] 103\b}, responses
- assert_match %r{^Link: </style\.css>}, responses
- assert_match %r{^Link: </script\.js>}, responses
-
- assert_match %r{^HTTP/1.[01] 200\b}, responses
- end
-
def test_after_reply
teardown
[-- Attachment #21: 0020-t-integration.t-switch-PUT-tests-to-MD5-reuse-buffer.patch --]
[-- Type: text/x-diff, Size: 3740 bytes --]
From cb826915cdd1881cbcfc1fb4e645d26244dfda71 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:49 +0000
Subject: [PATCH 20/23] t/integration.t: switch PUT tests to MD5, reuse buffers
MD5 is faster, and these tests aren't meant to be secure,
they're just for checking for data corruption.
Furthermore, Content-MD5 is a supported HTTP trailer and
we can verify that here to obsolete other tests.
Furthermore, we can reuse buffers on env['rack.input'].read
calls to avoid malloc(3) and GC overhead.
Combined, these give roughly a 3% speedup for t/integration.t
on my system.
---
t/integration.ru | 20 +++++++++++++++-----
t/integration.t | 5 ++---
t/preread_input.ru | 17 ++++++++++++-----
3 files changed, 29 insertions(+), 13 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index dab384d..086126a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -55,8 +55,8 @@ def env_dump(env)
def rack_input_tests(env)
return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
cap = 16384
- require 'digest/sha1'
- digest = Digest::SHA1.new
+ require 'digest/md5'
+ dig = Digest::MD5.new
input = env['rack.input']
case env['PATH_INFO']
when '/rack_input/size_first'; input.size
@@ -68,11 +68,21 @@ def rack_input_tests(env)
if buf = input.read(rand(cap))
begin
raise "#{buf.size} > #{cap}" if buf.size > cap
- digest.update(buf)
+ dig.update(buf)
end while input.read(rand(cap), buf)
+ buf.clear # remove this call if Ruby ever gets escape analysis
end
- [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
- [ digest.hexdigest ] ]
+ h = { 'content-type' => 'text/plain' }
+ if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+ cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+ cmd5_bin = cmd5_b64.unpack('m')[0]
+ if cmd5_bin != dig.digest
+ h['content-length'] = cmd5_b64.size.to_s
+ return [ 500, h, [ cmd5_b64 ] ]
+ end
+ end
+ h['content-length'] = '32'
+ [ 200, h, [ dig.hexdigest ] ]
end
run(lambda do |env|
diff --git a/t/integration.t b/t/integration.t
index 8433497..38a9675 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,7 +27,6 @@ my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
my $bs = $opt{bs} // 16384;
- require Digest::MD5;
my $dig = Digest::MD5->new;
print $out <<EOM;
PUT $path HTTP/1.1\r
@@ -186,8 +185,8 @@ SKIP: {
CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
- require Digest::SHA;
- $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+ require Digest::MD5;
+ $blob_hash = Digest::MD5->new->addfile($rh)->hexdigest;
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
diff --git a/t/preread_input.ru b/t/preread_input.ru
index f0a1748..18af221 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,15 +1,22 @@
#\-E none
-require 'digest/sha1'
+require 'digest/md5'
require 'unicorn/preread_input'
use Unicorn::PrereadInput
nr = 0
run lambda { |env|
$stderr.write "app dispatch: #{nr += 1}\n"
input = env["rack.input"]
- dig = Digest::SHA1.new
- while buf = input.read(16384)
- dig.update(buf)
+ dig = Digest::MD5.new
+ if buf = input.read(16384)
+ begin
+ dig.update(buf)
+ end while input.read(16384, buf)
+ buf.clear # remove this call if Ruby ever gets escape analysis
+ end
+ if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+ cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+ cmd5_bin = cmd5_b64.unpack('m')[0]
+ return [500, {}, [ cmd5_b64 ] ] if cmd5_bin != dig.digest
end
-
[ 200, {}, [ dig.hexdigest ] ]
}
[-- Attachment #22: 0021-tests-move-test_upload.rb-tests-to-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 12280 bytes --]
From 181e4b5b6339fc5e9c3ad7d3690b736f6bd038aa Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:50 +0000
Subject: [PATCH 21/23] tests: move test_upload.rb tests to t/integration.t
The overread tests are ported over, and checksumming alone
is enough to guard against data corruption.
Randomizing the size of `read' calls on the client side will
shake out any boundary bugs on the server side.
---
t/integration.t | 32 ++++-
test/unit/test_upload.rb | 301 ---------------------------------------
2 files changed, 27 insertions(+), 306 deletions(-)
delete mode 100644 test/unit/test_upload.rb
diff --git a/t/integration.t b/t/integration.t
index 38a9675..a568758 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -26,7 +26,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
- my $bs = $opt{bs} // 16384;
my $dig = Digest::MD5->new;
print $out <<EOM;
PUT $path HTTP/1.1\r
@@ -36,7 +35,7 @@ Trailer: Content-MD5\r
EOM
my ($buf, $r);
while (1) {
- $r = read($in, $buf, $bs);
+ $r = read($in, $buf, 999 + int(rand(0xffff)));
last if $r == 0;
printf $out "%x\r\n", length($buf);
print $out $buf, "\r\n";
@@ -46,15 +45,15 @@ EOM
},
identity => sub {
my ($in, $out, $path, %opt) = @_;
- my $bs = $opt{bs} // 16384;
my $clen = $opt{-s} // -s $in;
print $out <<EOM;
PUT $path HTTP/1.0\r
Content-Length: $clen\r
\r
EOM
- my ($buf, $r, $len);
+ my ($buf, $r, $len, $bs);
while ($clen) {
+ $bs = 999 + int(rand(0xffff));
$len = $clen > $bs ? $bs : $clen;
$r = read($in, $buf, $len);
die 'premature EOF' if $r == 0;
@@ -192,8 +191,10 @@ SKIP: {
my ($sub, $path, %opt) = @_;
seek($rh, 0, SEEK_SET);
$c = tcp_start($srv);
- $c->autoflush(0);
+ $c->autoflush($opt{sync} // 0);
$PUT{$sub}->($rh, $c, $path, %opt);
+ defined($opt{overwrite}) and
+ print { $c } ('x' x $opt{overwrite});
$c->flush or die $!;
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
@@ -205,6 +206,27 @@ SKIP: {
$ck_hash->('chunked_md5', '/rack_input/size_first');
$ck_hash->('chunked_md5', '/rack_input/rewind_first');
+ $ck_hash->('identity', '/rack_input', -s => $blob_size, sync => 1);
+ $ck_hash->('chunked_md5', '/rack_input', sync => 1);
+
+ # ensure small overwrites don't get checksummed
+ $ck_hash->('identity', '/rack_input', -s => $blob_size,
+ overwrite => 1); # one extra byte
+
+ # excessive overwrite truncated
+ $c = tcp_start($srv);
+ $c->autoflush(0);
+ print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 1\r\n\r\n";
+ if (1) {
+ local $SIG{PIPE} = 'IGNORE';
+ my $buf = "\0" x 8192;
+ my $n = 0;
+ my $end = time + 5;
+ $! = 0;
+ while (print $c $buf and time < $end) { ++$n }
+ ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+ }
+ undef $c;
$curl // skip 'no curl found in PATH', 1;
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
deleted file mode 100644
index 76e6c1c..0000000
--- a/test/unit/test_upload.rb
+++ /dev/null
@@ -1,301 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
-require './test/test_helper'
-require 'digest/md5'
-
-include Unicorn
-
-class UploadTest < Test::Unit::TestCase
-
- def setup
- @addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
- @port = unused_port
- @hdr = {'Content-Type' => 'text/plain', 'Content-Length' => '0'}
- @bs = 4096
- @count = 256
- @server = nil
-
- # we want random binary data to test 1.9 encoding-aware IO craziness
- @random = File.open('/dev/urandom','rb')
- @sha1 = Digest::SHA1.new
- @sha1_app = lambda do |env|
- input = env['rack.input']
- resp = {}
-
- @sha1.reset
- while buf = input.read(@bs)
- @sha1.update(buf)
- end
- resp[:sha1] = @sha1.hexdigest
-
- # rewind and read again
- input.rewind
- @sha1.reset
- while buf = input.read(@bs)
- @sha1.update(buf)
- end
-
- if resp[:sha1] == @sha1.hexdigest
- resp[:sysread_read_byte_match] = true
- end
-
- if expect_size = env['HTTP_X_EXPECT_SIZE']
- if expect_size.to_i == input.size
- resp[:expect_size_match] = true
- end
- end
- resp[:size] = input.size
- resp[:content_md5] = env['HTTP_CONTENT_MD5']
-
- [ 200, @hdr.merge({'X-Resp' => resp.inspect}), [] ]
- end
- end
-
- def teardown
- redirect_test_io { @server.stop(false) } if @server
- @random.close
- reset_sig_handlers
- end
-
- def test_put
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
- @count.times do |i|
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- sock.syswrite(buf)
- end
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_content_md5
- md5 = Digest::MD5.new
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- sock.syswrite("PUT / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n" \
- "Trailer: Content-MD5\r\n\r\n")
- @count.times do |i|
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- md5.update(buf)
- sock.syswrite("#{'%x' % buf.size}\r\n")
- sock.syswrite(buf << "\r\n")
- end
- sock.syswrite("0\r\n")
-
- content_md5 = [ md5.digest! ].pack('m').strip.freeze
- sock.syswrite("Content-MD5: #{content_md5}\r\n\r\n")
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- assert_equal content_md5, resp[:content_md5]
- end
-
- def test_put_trickle_small
- @count, @bs = 2, 128
- start_server(@sha1_app)
- assert_equal 256, length
- sock = tcp_socket(@addr, @port)
- hdr = "PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n"
- @count.times do
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- hdr << buf
- sock.syswrite(hdr)
- hdr = ''
- sleep 0.6
- end
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_keepalive_truncates_small_overwrite
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- to_upload = length + 1
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{to_upload}\r\n\r\n")
- @count.times do
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- sock.syswrite(buf)
- end
- sock.syswrite('12345') # write 4 bytes more than we expected
- @sha1.update('1')
-
- buf = sock.readpartial(4096)
- while buf !~ /\r\n\r\n/
- buf << sock.readpartial(4096)
- end
- read = buf.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal to_upload, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_excessive_overwrite_closed
- tmp = Tempfile.new('overwrite_check')
- tmp.sync = true
- start_server(lambda { |env|
- nr = 0
- while buf = env['rack.input'].read(65536)
- nr += buf.size
- end
- tmp.write(nr.to_s)
- [ 200, @hdr, [] ]
- })
- sock = tcp_socket(@addr, @port)
- buf = ' ' * @bs
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
-
- @count.times { sock.syswrite(buf) }
- assert_raise(Errno::ECONNRESET, Errno::EPIPE) do
- ::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) }
- end
- sock.gets
- tmp.rewind
- assert_equal length, tmp.read.to_i
- end
-
- # Despite reading numerous articles and inspecting the 1.9.1-p0 C
- # source, Eric Wong will never trust that we're always handling
- # encoding-aware IO objects correctly. Thus this test uses shell
- # utilities that should always operate on files/sockets on a
- # byte-level.
- def test_uncomfortable_with_onenine_encodings
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=#{@bs}", "count=#{@count}"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
-
- # small StringIO path
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=1024", "count=1"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- end
-
- def test_chunked_upload_via_curl
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=#{@bs}", "count=#{@count}"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- cmd = "curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
- -isSf --no-buffer -T- " \
- "http://#@addr:#@port/"
- resp = Tempfile.new('resp')
- resp.sync = true
-
- rd, wr = IO.pipe.each do |io|
- io.sync = io.close_on_exec = true
- end
- pid = spawn(*cmd, { 0 => rd, 1 => resp })
- rd.close
-
- tmp.rewind
- @count.times { |i|
- wr.write(tmp.read(@bs))
- sleep(rand / 10) if 0 == i % 8
- }
- wr.close
- pid, status = Process.waitpid2(pid)
-
- resp.rewind
- resp = resp.read
- assert status.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- assert_match(/expect_size_match/, resp)
- end
-
- def test_curl_chunked_small
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- # small StringIO path
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=1024", "count=1"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
- -isSf --no-buffer -T- http://#@addr:#@port/ < #{tmp.path}`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- assert_match(/expect_size_match/, resp)
- end
-
- private
-
- def length
- @bs * @count
- end
-
- def start_server(app)
- redirect_test_io do
- @server = HttpServer.new(app, :listeners => [ "#{@addr}:#{@port}" ] )
- @server.start
- end
- end
-
-end
[-- Attachment #23: 0022-drop-redundant-IO-close_on_exec-false-calls.patch --]
[-- Type: text/x-diff, Size: 891 bytes --]
From 841b9e756beb1aa00d0f89097a808adcbbf45397 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:51 +0000
Subject: [PATCH 22/23] drop redundant IO#close_on_exec=false calls
Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
---
lib/unicorn/http_server.rb | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 348e745..dd92b38 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -472,10 +472,7 @@ def worker_spawn(worker)
def listener_sockets
listener_fds = {}
- LISTENERS.each do |sock|
- sock.close_on_exec = false
- listener_fds[sock.fileno] = sock
- end
+ LISTENERS.each { |sock| listener_fds[sock.fileno] = sock }
listener_fds
end
[-- Attachment #24: 0023-LISTEN_FDS-inherited-sockets-are-immortal-across-SIG.patch --]
[-- Type: text/x-diff, Size: 3229 bytes --]
From 6ff8785c9277c5978e6dc01cb1b3da25d6bae2db Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:52 +0000
Subject: [PATCH 23/23] LISTEN_FDS-inherited sockets are immortal across SIGHUP
When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.
I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
---
lib/unicorn/http_server.rb | 7 ++++++-
t/client_body_buffer_size.t | 1 -
t/integration.t | 1 -
3 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index dd92b38..f1b4a54 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -77,6 +77,7 @@ def initialize(app, options = {})
options[:use_defaults] = true
self.config = Unicorn::Configurator.new(options)
self.listener_opts = {}
+ @immortal = [] # immortal inherited sockets from systemd
# We use @self_pipe differently in the master and worker processes:
#
@@ -158,6 +159,7 @@ def listeners=(listeners)
end
set_names = listener_names(listeners)
dead_names.concat(cur_names - set_names).uniq!
+ dead_names -= @immortal.map { |io| sock_name(io) }
LISTENERS.delete_if do |io|
if dead_names.include?(sock_name(io))
@@ -807,17 +809,20 @@ def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
# before they become Kgio::UNIXServer or Kgio::TCPServer
inherited = ENV['UNICORN_FD'].to_s.split(',')
+ immortal = []
# emulate sd_listen_fds() for systemd
sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
if sd_pid.to_i == $$ # n.b. $$ can never be zero
# 3 = SD_LISTEN_FDS_START
- inherited.concat((3...(3 + sd_fds.to_i)).to_a)
+ immortal = (3...(3 + sd_fds.to_i)).to_a
+ inherited.concat(immortal)
end
# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
inherited.map! do |fd|
io = Socket.for_fd(fd.to_i)
+ @immortal << io if immortal.include?(fd)
io.autoclose = false
io = server_cast(io)
set_server_sockopt(io, listener_opts[sock_name(io)])
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index b1a99f3..3067f28 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -36,7 +36,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
seek($conf_fh, 0, SEEK_SET);
truncate($conf_fh, 0);
print $conf_fh <<EOM;
-listen "$host_port" # TODO: remove this requirement for SIGHUP
after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
EOM
$ar->do_kill('HUP');
diff --git a/t/integration.t b/t/integration.t
index a568758..bb2ab51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -17,7 +17,6 @@ my $u1 = "$tmpdir/u1";
print $conf_fh <<EOM;
early_hints true
listen "$u1"
-listen "$host_port" # TODO: remove this requirement for SIGHUP
EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
^ permalink raw reply related [relevance 1%]
* [PATCH 0/2] drop Ruby 1.9.3 support, require 2.0+
@ 2021-09-14 23:39 6% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2021-09-14 23:39 UTC (permalink / raw)
To: unicorn-public
Eventually, I'd like to move towards newer versions of Ruby
(e.g. 2.3+ has more *_nonblock stuff). For now, this is a small
step and hopefully won't cause problems for maintainers of
legacy systems...
Eric Wong (2):
drop Ruby 1.9.3 support, require 2.0+ for now
drop unnecessary IO#close_on_exec=true assignment
HACKING | 2 +-
README | 3 +--
Sandbox | 2 +-
ext/unicorn_http/extconf.rb | 4 ++--
ext/unicorn_http/unicorn_http.rl | 14 +-------------
lib/unicorn.rb | 2 --
lib/unicorn/http_server.rb | 2 +-
t/README | 2 +-
unicorn.gemspec | 4 ++--
9 files changed, 10 insertions(+), 25 deletions(-)
^ permalink raw reply [relevance 6%]
* Re: Potential Unicorn vulnerability
[not found] ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
@ 2021-03-12 9:41 0% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2021-03-12 9:41 UTC (permalink / raw)
To: Dirkjan Bussink; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 6397 bytes --]
Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
>
> > On 11 Mar 2021, at 04:02, Eric Wong <normalperson@yhbt.net> wrote:
> >
> > Thanks for reaching out. Fwiw, I prefer if everything were made
> > public right away, but I'll leave it up to you if you're not
> > comfortable with it.
> >
> > I don't know much about security, anwyays; and don't like
> > classifying bugs (or classifying anything)...
>
> We reached out privately first out of care and to follow best practices
> around coordinated disclosure, in case you considered this a security
> vulnerability. We have no objections to moving this to the public mailing
> list. We are viewing this patch as a proactive hardening against race
> conditions more so than a vulnerability.
OK, I'm adding unicorn-public@yhbt.net to the Cc: of this mail.
Personally, I prefer everything be reported publicly ASAP.
There's a constant threat of power/network failures from
disasters and such that could cause messages to be delayed
too long or indefinitely.
> We are also reaching out to a private Rails Security coordination channel
> that we’re a part of to raise awareness of this behavior so other Unicorn
> users in this group can look for similar issues in their code.
OK.
> To move this discussion to the public list, would you prefer that you move
> this thread publicly, or that we resend the message or forward it to the
> public mailing list?
Attached are the initial two (previously private) emails in this
thread, so no need to resend. They have the correct
message/rfc822 MIME type set so they should be readable from
most MUAs and mail archives.
> > Ouch, so the hijack check we had in HttpParser_clear didn't fire...
>
> Yes, to our understanding it only would fire if explicitly set and that
> doesn’t happen here.
>
> >> While we understand and appreciate that Unicorn is not a multi-threaded web
> >> server, it feels like using the same `Hash` object between requests
> >> introduces the chance that a dependency like an external gem may use threads
> >> and thus potentially leak information between requests.
> >
> > Yes, there's likely problems in trying to use threads with a
> > codebase that was only intended to be single-threaded. And
> > using Thread.current[...] here wouldn't have made a. difference.
>
> For us it was also the difference between “requests are handled single threaded”
> vs “you can’t use threads for anything else either.” We were totally aware of the
> first, but the latter seems more accidental and has a much broader impact.
Agreed.
> > I worry some endpoints out there will suffer performance
> > degradation. Expensive endpoints probably won't notice,
> > but maybe the fastest ones will...
>
> That is a good point, but I think in practice most users do enough in most
> requests that the trade off is totally worth it. At least for us it definitely
> is. Maybe it would be an option to make the sharing somehow opt-in instead of
> default behavior? So that by default the safe behavior is used, but for those
> that want to, they can opt into the sharing if they know their app is safe
> enough to work with that.
I'm not in favor of new options since they add support costs
and increase the learning/maintenance curve.
What I've been thinking about is bumping the major version to 6.0
Although our internals are technically not supported stable API,
there may be odd stuff out there similar to OobGC that uses
instance_variable_get or similar things to reach into internals.
Added with the fact our internals haven't changed in many years;
I'm inclined to believe there are other OobGC-like things out
there that can break.
Also, with 6.0; users who completely avoid Threads can keep
using 5.x, while others can use 6.x
> >> For the sake of transparency to our users, we plan on publishing a public
> >> post next week on how this was part of the larger series of bugs that had a
> >> security impact at GitHub. We've also attached a suggested patch that removes
> >> the environment sharing, which is what we're running right now to reduce the
> >> risk of this ever happening again.
> >
> > Did you measure performance degradations in any endpoints you have?
>
> We did measure and there were zero noticeable performance degradations. That’s
> also because all our requests do a bunch of work and are not direct Rack apps,
> and use stuff like Rails or Sinatra. Those all usually wrap the `env` hash
> anyway with their own per request object and there’s a lot of other allocations
> happening anyway.
>
> In microbenchmarks we could see a difference, but even there, at least for us,
> we’d gladly pay the performance price for the safety if we’d have any endpoints
> where it would be measurable.
OK. Fwiw, there's still some stones left unturned that could
recover the lost speed if somebody _really_ cares for it(*)
(*) String#clear on response header buffer, swapping Ragel
for a faster HTTP parser, ...
> All in all, I think here that a safe default would be helpful for users, as
> mentioned earlier, and that maybe for those cases where the latest performance
> bit matters, the existing behavior could be opted into. Whether this option is
> worth it from a maintenance standpoint is something you’re better able to answer
> than we are though.
It's probably OK and I'm inclined to accept your patch for 6.0.
At this point, I'm more worried about potential breakage of some
3rd-party OobGC-like thing that reaches deep into our internals.
Btw, did you consider replacing the @request HttpRequest object
entirely instead of the env and buf elements?
I suppose that's more allocations, still; but could've
been a smaller change.
> > I'll see about finding a less-noisy/overloaded system to run
> > benchmarks against...
> >
> > I noticed some of the OobGC tests in t/ were failing (patch below),
> > but few users use the that version of OobGC.
>
> I wasn’t easily able to run the entire suite, but only parts of it which is why
> I didn’t have a complete fix there. I can add this to the patch then.
Oops, was that the integration tests in t/* ?
They can run separately via "make test-integration"
(I never trusted some Ruby behaviors to remain stable,
so I started writing tests in Bourne shell).
<snip> I have nothing to add to the rest of the mail
unicorn-public readers: see attachments
[-- Attachment #2: 0001-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 12105 bytes --]
[-- Attachment #2.1.1: Type: text/plain, Size: 2813 bytes --]
Hello Eric,
We're reaching out privately first on what we think could be classified as a
security issue in Unicorn. Since there may be other similarly impacted users,
this is out of an abundance of caution before sending it to the public
Unicorn mailing list.
The issue at hand is that the environment sharing and reuse between requests
in Unicorn, combined with other non thread safe code, resulted in a security
vulnerability where a very small number of user sessions tracked through
cookies got misrouted. For this reason, we logged everyone out of GitHub, see
also https://github.blog/2021-03-08-github-security-update-a-bug-related-to-handling-of-authenticated-sessions/.
The unsafe background thread code we had ended up triggering an exception at
rare times and the exception tracking logic was using a deferred block
executed through a Ruby Proc to gather information, including things from the
request at the time the logic started.
That thread captured something from the cookie jar among other things, and
the following code in Rack memoized that into the (shared) environment.
From https://github.com/rack/rack/blob/2.1.4/lib/rack/request.rb#L219-L229
def cookies
hash = fetch_header(RACK_REQUEST_COOKIE_HASH) do |k|
set_header(k, {})
end
string = get_header HTTP_COOKIE
return hash if string == get_header(RACK_REQUEST_COOKIE_STRING)
hash.replace Utils.parse_cookies_header string
set_header(RACK_REQUEST_COOKIE_STRING, string)
hash
end
Because of the environment sharing, the memoization ended up overriding what
would be memoized (with the RACK_REQUEST_COOKIE_STRING key) for the new
request and because a specific session cookie was bumped then with a new
timeout on each request, the wrong session cookie was serialized.
While we understand and appreciate that Unicorn is not a multi-threaded web
server, it feels like using the same `Hash` object between requests
introduces the chance that a dependency like an external gem may use threads
and thus potentially leak information between requests.
For the sake of transparency to our users, we plan on publishing a public
post next week on how this was part of the larger series of bugs that had a
security impact at GitHub. We've also attached a suggested patch that removes
the environment sharing, which is what we're running right now to reduce the
risk of this ever happening again.
We hope you're open to collaborating on a fix prior to any public detailed
disclosure of how this request environment sharing could lead to security
issues, if you feel that is desired.
I’ve added John & Kevin here on the CC since they’ve also worked on this and
that way we have some better timezone spread on our side if needed.
Cheers,
Dirkjan Bussink
[-- Attachment #2.1.2: 0001-Drop-reuse-of-Ruby-level-objects.patch --]
[-- Type: application/octet-stream, Size: 4067 bytes --]
From 44aa6b056e1b24d42ab3efba738b38f4cd54a068 Mon Sep 17 00:00:00 2001
From: Dirkjan Bussink <d.bussink@gmail.com>
Date: Mon, 8 Mar 2021 09:51:09 +0100
Subject: [PATCH] Drop reuse of Ruby level objects
Remove the reuse of environment and the buffer as Ruby level objects.
Reusing these is very risky in the context of running any other threads
within the unicorn process, also for threads that run background tasks.
This lead to a significant security incident where the problems would
not have happened if there was no reuse of the `env` object. From a
safety perspective, this also removes reuse of the buffer object as it's
also a Ruby object and could be grabbed and retained outside of the http
parsing logic.
The downside here is that we allocate two extra objects on each request,
but that is worth the trade off here and the security risk we otherwise
would carry to leaking wrong and incorrect data.
---
ext/unicorn_http/extconf.rb | 1 -
ext/unicorn_http/unicorn_http.rl | 30 +++---------------------------
test/unit/test_http_parser.rb | 3 +++
3 files changed, 6 insertions(+), 28 deletions(-)
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 95514bc..7e4775c 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -10,7 +10,6 @@
have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 1.9.3+ required'
-have_func("rb_hash_clear", "ruby.h") # Ruby 2.0+
have_func("gmtime_r", "time.h")
message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 21e09d6..9904d85 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -65,18 +65,6 @@ struct http_parser {
static ID id_set_backtrace, id_is_chunked_p;
static VALUE cHttpParser;
-#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
-# define my_hash_clear(h) (void)rb_hash_clear(h)
-#else /* !HAVE_RB_HASH_CLEAR - Ruby <= 1.9.3 */
-
-static ID id_clear;
-
-static void my_hash_clear(VALUE h)
-{
- rb_funcall(h, id_clear, 0);
-}
-#endif /* HAVE_RB_HASH_CLEAR */
-
static void finalize_header(struct http_parser *hp);
static void parser_raise(VALUE klass, const char *msg)
@@ -445,6 +433,8 @@ static void http_parser_init(struct http_parser *hp)
hp->cont = Qfalse; /* zero on MRI, should be optimized away by above */
%% write init;
hp->cs = cs;
+ hp->buf = rb_str_new(NULL, 0);
+ hp->env = rb_hash_new();
}
/** exec **/
@@ -628,8 +618,6 @@ static VALUE HttpParser_init(VALUE self)
struct http_parser *hp = data_get(self);
http_parser_init(hp);
- hp->buf = rb_str_new(NULL, 0);
- hp->env = rb_hash_new();
return self;
}
@@ -643,16 +631,7 @@ static VALUE HttpParser_init(VALUE self)
*/
static VALUE HttpParser_clear(VALUE self)
{
- struct http_parser *hp = data_get(self);
-
- /* we can't safely reuse .buf and .env if hijacked */
- if (HP_FL_TEST(hp, HIJACK))
- return HttpParser_init(self);
-
- http_parser_init(hp);
- my_hash_clear(hp->env);
-
- return self;
+ return HttpParser_init(self);
}
static void advance_str(VALUE str, off_t nr)
@@ -1025,9 +1004,6 @@ void Init_unicorn_http(void)
id_set_backtrace = rb_intern("set_backtrace");
init_unicorn_httpdate();
-#ifndef HAVE_RB_HASH_CLEAR
- id_clear = rb_intern("clear");
-#endif
id_is_chunked_p = rb_intern("is_chunked?");
}
#undef SET_GLOBAL
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 697af44..d3f9ae7 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -33,6 +33,9 @@ def test_parse_simple
parser.clear
req.clear
+ req = parser.env
+ http = parser.buf
+
http << "G"
assert_nil parser.parse
assert_equal "G", http
--
2.30.1
[-- Attachment #3: 0002-re-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 5635 bytes --]
From: Eric Wong <normalperson@yhbt.net>
To: Dirkjan Bussink <dbussink@github.com>
Cc: John Crepezzi <seejohnrun@github.com>, Kevin Sawicki <kevinsawicki@github.com>
Subject: Re: Potential Unicorn vulnerability
Date: Thu, 11 Mar 2021 03:02:50 +0000
Message-ID: <20210311030250.GA1266@dcvr>
Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
>
> We're reaching out privately first on what we think could be classified as a
> security issue in Unicorn. Since there may be other similarly impacted users,
> this is out of an abundance of caution before sending it to the public
> Unicorn mailing list.
Thanks for reaching out. Fwiw, I prefer if everything were made
public right away, but I'll leave it up to you if you're not
comfortable with it.
I don't know much about security, anwyays; and don't like
classifying bugs (or classifying anything)...
<snip>
> That thread captured something from the cookie jar among other things, and
> the following code in Rack memoized that into the (shared) environment.
Ouch, so the hijack check we had in HttpParser_clear didn't fire...
<snip>
> While we understand and appreciate that Unicorn is not a multi-threaded web
> server, it feels like using the same `Hash` object between requests
> introduces the chance that a dependency like an external gem may use threads
> and thus potentially leak information between requests.
Yes, there's likely problems in trying to use threads with a
codebase that was only intended to be single-threaded. And
using Thread.current[...] here wouldn't have made a. difference.
I worry some endpoints out there will suffer performance
degradation. Expensive endpoints probably won't notice,
but maybe the fastest ones will...
`buf' is particularly worrying to me since it's a 16384-byte
allocation for a socket read on headers that could amount
to lots of GC pressure and hurt locality all around.
env['rack.input'] may also hold onto `buf', too, since
input is lazily-consumed (though it may be possible to
workaround that...).
Losing `env' is probably less worrying since keys are all
fstrings in modern Rubies. Though losing the backing store (and
not having rb_hash_new_with_size in the C API) would mean more
rehashing with many headers.
The simple Rack apps I worked on back in-the-day which
benefitted from these object-reuse optimizations are unlikely to
be upgraded to newer versions of unicorn. Of course, there
may be other similar Rack apps out there that depend on these...
> For the sake of transparency to our users, we plan on publishing a public
> post next week on how this was part of the larger series of bugs that had a
> security impact at GitHub. We've also attached a suggested patch that removes
> the environment sharing, which is what we're running right now to reduce the
> risk of this ever happening again.
Did you measure performance degradations in any endpoints you have?
I'll see about finding a less-noisy/overloaded system to run
benchmarks against...
I noticed some of the OobGC tests in t/ were failing (patch below),
but few users use the that version of OobGC.
Also, t/t0200-rack-hijack.sh would go away.
> We hope you're open to collaborating on a fix prior to any public detailed
> disclosure of how this request environment sharing could lead to security
> issues, if you feel that is desired.
Sure, as long as everything is done via plain-text email so
I won't need working graphics drivers or modern hardware :>
Along the same lines, would you be OK with this entire mail thread
(including all mail headers) being eventually published in the
public mailbox at http://ou63pmih66umazou.onion/unicorn-public/
and https://yhbt.net/unicorn-public/ ?
> I’ve added John & Kevin here on the CC since they’ve also worked on this and
> that way we have some better timezone spread on our side if needed.
OK, I'm around/awake at pretty random times.
Aforementioned OomGC change:
-------8<-------
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 3b2f488..91a8e51 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -60,7 +60,6 @@ def self.new(app, interval = 5, path = %r{\A/})
self.const_set :OOBGC_INTERVAL, interval
ObjectSpace.each_object(Unicorn::HttpServer) do |s|
s.extend(self)
- self.const_set :OOBGC_ENV, s.instance_variable_get(:@request).env
end
app # pretend to be Rack middleware since it was in the past
end
@@ -68,9 +67,10 @@ def self.new(app, interval = 5, path = %r{\A/})
#:stopdoc:
def process_client(client)
super(client) # Unicorn::HttpServer#process_client
- if OOBGC_PATH =~ OOBGC_ENV['PATH_INFO'] && ((@@nr -= 1) <= 0)
+ env = instance_variable_get(:@request).env
+ if OOBGC_PATH =~ env['PATH_INFO'] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
- OOBGC_ENV.clear
+ env.clear
disabled = GC.enable
GC.start
GC.disable if disabled
^ permalink raw reply related [relevance 0%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
@ 2020-09-01 15:41 6% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2020-09-01 15:41 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> >> I don't really see any reason to protect against newer Ruby version.
> >
> > I do: Ruby does not have a good track record when it comes to
> > maintaining backwards compatibility.
>
> Regardless, preventing the gem installation before the version even exists
> cause massive pains to people trying to do the good thing of testing their
> app against ruby master.
>
> Until it is known that compatibility is broken, restricting the ruby version
> causes more harm than good. Just this morning I had to submit patches
> to a dozen gems.
That sucks. Perhaps adding warnings about untested+unsupported
versions to test_helper.rb and extconf.rb is a better way to go?
(nothing annoying at runtime after it's installed, though)
Then, maybe leave the version check out of the gemspec entirely.
Fwiw, the type of breakage from incompatibilities I'm worried
about is subtle things that don't show up immediately
(e.g. encodings, hash ordering, frozen strings, etc...).
Stuff that obviously breaks at startup is preferable
(e.g. I added some 1.9 symbol hash keys to signal the move
away from 1.8).
Thanks.
^ permalink raw reply [relevance 6%]
* Re: [ruby-core:99185] [Ruby master Bug#17023] How to prevent String memory to be relocated in ruby-ffi
@ 2020-07-15 23:49 7% ` Aaron Patterson
0 siblings, 0 replies; 73+ results
From: Aaron Patterson @ 2020-07-15 23:49 UTC (permalink / raw)
To: Ruby developers; +Cc: unicorn-public
> On Jul 15, 2020, at 4:35 PM, Eric Wong <normalperson@yhbt.net> wrote:
>
> tenderlove@ruby-lang.org wrote:
>> Right, that makes sense. I really need to document this (and
>> I apologize for not doing so already), but
>> `rb_gc_register_address` will pin your objects. When you know
>> you're done with the reference, you can release it with
>> `rb_gc_unregister_address`. Of course if you don't call the
>> unregister function, the reference will stay alive forever.
>
> Btw, does rb_gc_register_mark_object pin? A quick glance at
> gc.c tells me it doesn't, and I'll need to revert commit
> 2a6cb76d5010cb763ef5a2c305728465d15eb7c9 in unicorn:
> https://yhbt.net/unicorn-public/20181226050857.6413-1-e@80x24.org/
Yes, it does pin. I’m not super proud of this code, but here is where objects passed to rb_gc_register_mark_object get pinned:
https://github.com/ruby/ruby/blob/c2a6295ec04a191c689d22254ac1ad5d665e27ad/vm.c#L2307-L2320
I don’t know why the mark object array is an array of arrays (I assume so as not to waste space in the array buffer?). Maybe this could be a more friendly data structure.
I created a pinned list in compile.c so that objects allocated and used at compile time don’t move (they become free to move once iseq assembly is finished). It seems that might be a more generally useful thing, but so far I’ve only seen two places that need this feature.
^ permalink raw reply [relevance 7%]
* Re: Rack::Request#params EOFError
2017-03-21 19:06 5% ` Eric Wong
@ 2017-03-26 20:52 0% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2017-03-26 20:52 UTC (permalink / raw)
To: John Smart; +Cc: unicorn-public
> John Smart <smartj@gmail.com> wrote:
> > When multipart > 112KB, I noticed that Unicorn tees the input stream
> > to a temporary file. I was wondering: might Unicorn::TeeInput raise
> > an EOFError as part of normal operation when reaching the end of the
> > input stream? If so, this would break the Rack spec. I only tested
> > this on Darwin, still working on a self-contained repro.
Ping on this.
Eric Wong <e@80x24.org> wrote:
> No, it should not raise EOFError unless a client sent less than
> the Content-Length it declared in the header, or if it sent a
> short chunk with "Transfer-Encoding: chunked".
>
> EOFError should be raised to break out of the application
> processing entirely if and only if the client decides to disconnect
> prematurely. This is needed to allow unicorn to move onto other
> clients.
>
> What unicorn could (and maybe should) do is raise a different
> error which is not a subclass of EOFError; to prevent the error
> from being caught by Rack (or any other middlewares).
I'm still considering this, but I'm also wondering if it'll
break any existing code which relies on Unicorn::ClientShutdown
being a subclass of EOFError
> What client are you using?
>
> Is it sending "Transfer-Encoding: chunked" or a Content-Length?
>
> Is nginx in front of unicorn? If not, does it happen when nginx
> is in front of unicorn?
^ permalink raw reply [relevance 0%]
* Re: Rack::Request#params EOFError
@ 2017-03-21 19:06 5% ` Eric Wong
2017-03-26 20:52 0% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Eric Wong @ 2017-03-21 19:06 UTC (permalink / raw)
To: John Smart; +Cc: unicorn-public
John Smart <smartj@gmail.com> wrote:
> I found an interesting bug today. When sending a form multipart >
> 112KB, I noticed that Rack::Request#params was empty. This only occurs
> when using Unicorn, and does not occur when using Thin.
>
> I dug into Rack source and noticed that any EOFError raised while
> reading the body input stream will cause the post body input stream to
> be ignored:
>
> https://github.com/rack/rack/blob/cabaa58fe6ac355623746e287475af88c9395d66/lib/rack/request.rb#L357
EOFError should not happen with normal operations supported by
env['rack.input'] (read, gets, each, rewind).
So I'm not sure why rack does not let EOFError propagate up the
stack...
The equivalent Ruby methods (IO#{read,gets,each,rewind})
do not raise EOFError; read and gets should return nil on EOF...
> When multipart > 112KB, I noticed that Unicorn tees the input stream
> to a temporary file. I was wondering: might Unicorn::TeeInput raise
> an EOFError as part of normal operation when reaching the end of the
> input stream? If so, this would break the Rack spec. I only tested
> this on Darwin, still working on a self-contained repro.
No, it should not raise EOFError unless a client sent less than
the Content-Length it declared in the header, or if it sent a
short chunk with "Transfer-Encoding: chunked".
EOFError should be raised to break out of the application
processing entirely if and only if the client decides to disconnect
prematurely. This is needed to allow unicorn to move onto other
clients.
What unicorn could (and maybe should) do is raise a different
error which is not a subclass of EOFError; to prevent the error
from being caught by Rack (or any other middlewares).
What client are you using?
Is it sending "Transfer-Encoding: chunked" or a Content-Length?
Is nginx in front of unicorn? If not, does it happen when nginx
is in front of unicorn?
> I ended up raising client_body_buffer_size to 1MB as a work-around.
> I'm wondering if this is a bug?
Not sure, yet. unicorn isn't meant to handle unreliable
clients; it's designed to talk to nginx.
Thanks.
^ permalink raw reply [relevance 5%]
* Re: [PATCH] check_client_connection: use tcp state on linux
2017-02-27 11:44 3% ` Simon Eskildsen
@ 2017-02-28 21:12 0% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2017-02-28 21:12 UTC (permalink / raw)
To: Simon Eskildsen; +Cc: unicorn-public
Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
<snip>
> I would assume you would see TIME_WAIT and CLOSE. LAST_ACK_CLOSING it
> seems pretty unlikely to hit, but not impossible. As with CLOSING,
> I've included LAST_ACK_CLOSING for completeness.
Did you mean "LAST_ACK, and CLOSING"? (not joined by underscore)
Anyways, thanks for testing and adding
> <e@80x24.org> wrote:
> > Yep, we need to account for the UNIX socket case. I forget if
> > kgio even makes them different...
>
> I read the implementation and verified by dumping the class when
> testing on some test boxes. You are right—it's a simple Kgio::Socket
> object, not differentiating between Kgio::TCPSocket and
> Kgio::UnixSocket at the class level. Kgio only does this if they're
> explicitly passed to override the class returned from #try_accept.
> Unicorn doesn't do this.
>
> I've tried to find a way to determine the socket domain (INET vs.
> UNIX) on the socket object, but neither Ruby's Socket class nor Kgio
> seems to expose this. I'm not entirely sure what the simplest way to
> do this check would be. We could have the accept loop pass the correct
> class to #try_accept based on the listening socket that came back from
> #accept. If we passed the listening socket to #read after accept, we'd
> know.. but I don't like that the request knows about the listener
> either. Alternatively, we could expose the socket domain in Kgio, but
> that'll be problematic in the near-ish future as you've mentioned
> wanting to move away from Kgio as Ruby's IO library is at parity as
> per Ruby 2.4.
>
> What do you suggest pursuing here to check whether the client socket
> is a TCP socket?
I think passing the listening socket is the best way to go about
detecting whether a socket is INET or UNIX, for now.
You're right about kgio, I'd rather not make more changes to
kgio but we will still need it to for Ruby <2.2.x users.
And #read is an overloaded name, feel free to change it :)
> Below is a patch addressing the other concerns. I had to include
> require raindrops so the `defined?` check would do the right thing, as
> the only other file that requires Raindrops is the worker one which is
> loaded after http_request. I can change the load-order or require
> raindrops in lib/unicorn.rb if you prefer.
The require is fine. However we do not need a class variable,
below...
> # TODO: remove redundant names
> Unicorn.const_set(:HttpRequest, Unicorn::HttpParser)
> @@ -29,6 +30,7 @@ class Unicorn::HttpParser
> # 2.2+ optimizes hash assignments when used with literal string keys
> HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
> @@input_class = Unicorn::TeeInput
> + @@raindrops_tcp_info_defined = defined?(Raindrops::TCP_Info)
I prefer we avoid adding this cvar, instead...
> @@check_client_connection = false
>
> def self.input_class
> @@ -83,11 +85,7 @@ def read(socket)
> false until add_parse(socket.kgio_read!(16384))
> end
>
> - # detect if the socket is valid by writing a partial response:
> - if @@check_client_connection && headers?
> - self.response_start_sent = true
> - HTTP_RESPONSE_START.each { |c| socket.write(c) }
> - end
> + check_client_connection(socket) if @@check_client_connection
>
> e['rack.input'] = 0 == content_length ?
> NULL_IO : @@input_class.new(socket, self)
> @@ -108,4 +106,27 @@ def call
> def hijacked?
> env.include?('rack.hijack_io'.freeze)
> end
... we can have different methods defined:
if defined?(Raindrops::TCP_Info) # Linux, maybe FreeBSD
def check_client_connection(client, listener) # :nodoc:
...
end
else # portable version
def check_client_connection(client, listener) # :nodoc:
...
end
end
And eliminate the class variable entirely.
> +
> + private
I prefer to avoid marking methods as 'private' to ease any
ad-hoc unit testing which may come up. Instead, rely on :nodoc:
directives to discourage people from depending on it.
Thanks.
> + def check_client_connection(socket)
> + if @@raindrops_tcp_info_defined
> + tcp_info = Raindrops::TCP_Info.new(socket)
> + raise Errno::EPIPE, "client closed connection".freeze, [] if
> closed_state?(tcp_info.state)
> + elsif headers?
> + self.response_start_sent = true
> + HTTP_RESPONSE_START.each { |c| socket.write(c) }
> + end
> + end
> +
> + def closed_state?(state)
> + case state
> + when 1 # ESTABLISHED
> + false
> + when 6, 7, 8, 9, 11 # TIME_WAIT, CLOSE, CLOSE_WAIT, LAST_ACK, CLOSING
> + true
> + else
> + false
> + end
> + end
> end
closed_state? looks good to me, good call on short-circuiting
the common case of ESTABLISHED!
^ permalink raw reply [relevance 0%]
* Re: [PATCH] check_client_connection: use tcp state on linux
@ 2017-02-27 11:44 3% ` Simon Eskildsen
2017-02-28 21:12 0% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Simon Eskildsen @ 2017-02-27 11:44 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
> I prefer we use a hash or case statement. Both allow more
> optimization in the YARV VM of CRuby (opt_aref and
> opt_case_dispatch in insns.def). case _might_ be a little
> faster if there's no constant lookup overhead, but
> a microbench or dumping the bytecode will be necessary
> to be sure :)
>
> A hash or a case can also help portability-wise in case
> we hit a system where these numbers are non-sequential;
> or if we forgot something.
Good point. I double checked all the states on Linux and found that we
were missing TCP_CLOSING [1] [2]. This is a state where the other side
is closed, and you have buffered data on your side. It doesn't seem
like this would ever happen in Unicorn, but I think we should include
it for completeness. This also means the range becomes non-sequential.
I looked at Illumus (solaris-derived) [3] and BSD [4] and for the TCP
states we're interested in it also appears to have a non-continues
range.
My co-worker, Kir Shatrov, benchmarked a bunch of approaches to the
state check and found that case is a good solution [5]. Due to the
realness of non-sequential states in common operating systems, I think
case is the way to go here as you suggested. I've made sure to
short-circuit the common-case of TCP_ESTABLISHED. I've only seen
CLOSE_WAIT in testing, but in the wild-life of large production scale
I would assume you would see TIME_WAIT and CLOSE. LAST_ACK_CLOSING it
seems pretty unlikely to hit, but not impossible. As with CLOSING,
I've included LAST_ACK_CLOSING for completeness.
[1] https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/include/net/tcp_states.h#L27
[2] https://github.com/torvalds/linux/blob/ca78d3173cff3503bcd15723b049757f75762d15/net/ipv4/tcp.c#L228
[3] https://github.com/freebsd/freebsd/blob/386ddae58459341ec567604707805814a2128a57/sys/netinet/tcp_fsm.h
[4] https://github.com/illumos/illumos-gate/blob/f7877f5d39900cfd8b20dd673e5ccc1ef7cc7447/usr/src/uts/common/netinet/tcp_fsm.h
[5] https://gist.github.com/kirs/11ba4ce84c08188c9f7eba9c639616a5
> Yep, we need to account for the UNIX socket case. I forget if
> kgio even makes them different...
I read the implementation and verified by dumping the class when
testing on some test boxes. You are right—it's a simple Kgio::Socket
object, not differentiating between Kgio::TCPSocket and
Kgio::UnixSocket at the class level. Kgio only does this if they're
explicitly passed to override the class returned from #try_accept.
Unicorn doesn't do this.
I've tried to find a way to determine the socket domain (INET vs.
UNIX) on the socket object, but neither Ruby's Socket class nor Kgio
seems to expose this. I'm not entirely sure what the simplest way to
do this check would be. We could have the accept loop pass the correct
class to #try_accept based on the listening socket that came back from
#accept. If we passed the listening socket to #read after accept, we'd
know.. but I don't like that the request knows about the listener
either. Alternatively, we could expose the socket domain in Kgio, but
that'll be problematic in the near-ish future as you've mentioned
wanting to move away from Kgio as Ruby's IO library is at parity as
per Ruby 2.4.
What do you suggest pursuing here to check whether the client socket
is a TCP socket?
Below is a patch addressing the other concerns. I had to include
require raindrops so the `defined?` check would do the right thing, as
the only other file that requires Raindrops is the worker one which is
loaded after http_request. I can change the load-order or require
raindrops in lib/unicorn.rb if you prefer.
Missing is the socket type check. Thanks for your feedback!
---
lib/unicorn/http_request.rb | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 0c1f9bb..eedccac 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -2,6 +2,7 @@
# :enddoc:
# no stable API here
require 'unicorn_http'
+require 'raindrops'
# TODO: remove redundant names
Unicorn.const_set(:HttpRequest, Unicorn::HttpParser)
@@ -29,6 +30,7 @@ class Unicorn::HttpParser
# 2.2+ optimizes hash assignments when used with literal string keys
HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
@@input_class = Unicorn::TeeInput
+ @@raindrops_tcp_info_defined = defined?(Raindrops::TCP_Info)
@@check_client_connection = false
def self.input_class
@@ -83,11 +85,7 @@ def read(socket)
false until add_parse(socket.kgio_read!(16384))
end
- # detect if the socket is valid by writing a partial response:
- if @@check_client_connection && headers?
- self.response_start_sent = true
- HTTP_RESPONSE_START.each { |c| socket.write(c) }
- end
+ check_client_connection(socket) if @@check_client_connection
e['rack.input'] = 0 == content_length ?
NULL_IO : @@input_class.new(socket, self)
@@ -108,4 +106,27 @@ def call
def hijacked?
env.include?('rack.hijack_io'.freeze)
end
+
+ private
+
+ def check_client_connection(socket)
+ if @@raindrops_tcp_info_defined
+ tcp_info = Raindrops::TCP_Info.new(socket)
+ raise Errno::EPIPE, "client closed connection".freeze, [] if
closed_state?(tcp_info.state)
+ elsif headers?
+ self.response_start_sent = true
+ HTTP_RESPONSE_START.each { |c| socket.write(c) }
+ end
+ end
+
+ def closed_state?(state)
+ case state
+ when 1 # ESTABLISHED
+ false
+ when 6, 7, 8, 9, 11 # TIME_WAIT, CLOSE, CLOSE_WAIT, LAST_ACK, CLOSING
+ true
+ else
+ false
+ end
+ end
end
--
2.11.0
^ permalink raw reply related [relevance 3%]
* Re: check_client_connection using getsockopt(2)
2017-02-22 18:33 0% ` Eric Wong
@ 2017-02-22 20:09 0% ` Simon Eskildsen
0 siblings, 0 replies; 73+ results
From: Simon Eskildsen @ 2017-02-22 20:09 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public, raindrops-public
On Wed, Feb 22, 2017 at 1:33 PM, Eric Wong <e@80x24.org> wrote:
> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>
> <snip> great to know it's still working after all these years :>
>
>> This confirms Eric's comment that the existing
>> `check_client_connection` works perfectly on loopback, but as soon as
>> you put an actual network between the Unicorn and client—it's only
>> effective 20% of the time, even with `TCP_NODELAY`. I'm assuming, due
>> to buffering, even when disabling Nagle's. As we're changing our
>> architecture, we move from ngx (lb) -> ngx (host) -> unicorn to ngx
>> (lb) -> unicorn. That means this patch will no longer work for us.
>
> Side comment: I'm a bit curious how you guys arrived at removing
> nginx at the host level (if you're allowed to share that info)
>
> I've mainly kept nginx on the same host as unicorn, but used
> haproxy or similar (with minimal buffering) at the LB level.
> That allows better filesystem load distribution for large uploads.
Absolutely. We're starting to experiment with Kubernetes, and a result
we're interested in simplifying ingress as much as possible. We could
still run them, but if I can avoid explaining why they're there for
the next 5 years—I'll be happy :) As I see, the current use-cases we
have for the host nginx are (with why we can get rid of it):
* Keepalive between edge nginx LBs and host LBs to avoid excessive
network traffic. This is not a deal-breaker, so we'll just ignore this
problem. It's a _massive_ amount of traffic normally though.
* Out of rotation. We take nodes gracefully out of rotation by
touching a file that'll return 404 on a health-checking endpoint from
the edge LBs. Kubernetes implements this by just removing all the
containers on that node.
* Graceful deploys. When we deploy with containers, we take each
container out of rotation with the local host Nginx, wait for it to
come up, and put it back in rotation. We don't utilize Unicorns
graceful restarts because we want a Ruby upgrade deploy (replacing the
container) to be the same as an updated code rollout.
* File uploads. As you mention, currently we load-balance this between
them. I haven't finished the investigation on whether this is feasible
on the front LBs. Otherwise we may go for a 2-tier Nginx solution or
expand the edge. However, with Kubernetes I'd really like to avoid
having host nginxes. It's not very native to the Kubernetes paradigm.
Does it balance across the network, or only to the pods on that node?
* check_client_connection working. :-) This thread.
Slow clients and other advantages we find from the edge Nginxes.
>> I propose instead of the early `write(2)` hack, that we use
>> `getsockopt(2)` with the `TCP_INFO` flag and read the state of the
>> socket. If it's in `CLOSE_WAIT` or `CLOSE`, we kill the connection and
>> move on to the next.
>>
>> https://github.com/torvalds/linux/blob/8fa3b6f9392bf6d90cb7b908e07bd90166639f0a/include/uapi/linux/tcp.h#L163
>>
>> This is not going to be portable, but we can do this on only Linux
>> which I suspect is where most production deployments of Unicorn that
>> would benefit from this feature run. It's not useful in development
>> (which is likely more common to not be Linux). We could also introduce
>> it under a new configuration option if desired. In my testing, this
>> works to reject 100% of requests early when not on loopback.
>
> Good to know about it's effectiveness! I don't mind adding
> Linux-only features as long as existing functionality still
> works on the server-oriented *BSDs.
>
>> The code is essentially:
>>)?
>> def client_closed?
>> tcp_info = socket.getsockopt(Socket::SOL_TCP, Socket::TCP_INFO)
>> state = tcp_info.unpack("c")[0]
>> state == TCP_CLOSE || state == TCP_CLOSE_WAIT
>> end
>
> +Cc: raindrops-public@bogomips.org -> https://bogomips.org/raindrops-public/
>
> Fwiw, raindrops (already a dependency of unicorn) has TCP_INFO
> support under Linux:
>
> https://bogomips.org/raindrops.git/tree/ext/raindrops/linux_tcp_info.c
>
> I haven't used it, much, but FreeBSD also implements a subset of
> this feature, nowadays, and ought to be supportable, too. We
> can look into adding it for raindrops.
Cool, I think it makes sense to use Raindrops here, advantage being
it'd use the actual struct instead of a sketchy `#unpack`.
I meant to ask, in Raindrops why do you use the netlink API to get the
socket backlog instead of `getsockopt(2)` with `TCP_INFO` to get
`tcpi_unacked`? (as described in
http://www.ryanfrantz.com/posts/apache-tcp-backlog/) We use this to
monitor socket backlogs with a sidekick Ruby daemon. Although we're
looking to replace it with a middleware to simplify for Kubernetes.
It's one of our main metrics for monitoring performance, especially
around deploys.
>
> I don't know about other BSDs...
>
>> This could be called at the top of `#process_client` in `http_server.rb`.
>>
>> Would there be interest in this upstream? Any comments on this
>> proposed implementation? Currently, we're using a middleware with the
>> Rack hijack API, but this seems like it belongs at the webserver
>> level.
>
> I guess hijack means you have to discard other middlewares and
> the normal rack response handling in unicorn? If so, ouch!
>
> Unfortunately, without hijack, there's no portable way in Rack
> to get at the underlying IO object; so I guess this needs to
> be done at the server level...
>
> So yeah, I guess it belongs in the webserver.
I was going to use `env["unicorn.socket"]`/`env["puma.socket"]`, but
you could also do `env.delete("hijack_io")` after hijacking to allow
Unicorn to still render the response. Unfortunately the
`<webserver>.socket` key is not part of the Rack standard, so I'm
hesitant to use it. When this gets into Unicorn I'm planning to
propose it upstream to Puma as well.
>
> I think we can automatically use TCP_INFO if available for
> check_client_connection; then fall back to the old early write
> hack for Unix sockets and systems w/o TCP_INFO (which would set
> the response_start_sent flag).
>
> No need to introduce yet another configuration option.
Cool. How would you suggest I check for TCP_INFO compatible platforms
in Unicorn? Is `RUBY_PLATFORM.ends_with?("linux".freeze)` sufficient
or do you prefer another mechanism? I agree that we should fall back
to the write hack on other platforms.
^ permalink raw reply [relevance 0%]
* Re: check_client_connection using getsockopt(2)
2017-02-22 12:02 6% check_client_connection using getsockopt(2) Simon Eskildsen
@ 2017-02-22 18:33 0% ` Eric Wong
2017-02-22 20:09 0% ` Simon Eskildsen
0 siblings, 1 reply; 73+ results
From: Eric Wong @ 2017-02-22 18:33 UTC (permalink / raw)
To: Simon Eskildsen; +Cc: unicorn-public, raindrops-public
Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
<snip> great to know it's still working after all these years :>
> This confirms Eric's comment that the existing
> `check_client_connection` works perfectly on loopback, but as soon as
> you put an actual network between the Unicorn and client—it's only
> effective 20% of the time, even with `TCP_NODELAY`. I'm assuming, due
> to buffering, even when disabling Nagle's. As we're changing our
> architecture, we move from ngx (lb) -> ngx (host) -> unicorn to ngx
> (lb) -> unicorn. That means this patch will no longer work for us.
Side comment: I'm a bit curious how you guys arrived at removing
nginx at the host level (if you're allowed to share that info)
I've mainly kept nginx on the same host as unicorn, but used
haproxy or similar (with minimal buffering) at the LB level.
That allows better filesystem load distribution for large uploads.
> I propose instead of the early `write(2)` hack, that we use
> `getsockopt(2)` with the `TCP_INFO` flag and read the state of the
> socket. If it's in `CLOSE_WAIT` or `CLOSE`, we kill the connection and
> move on to the next.
>
> https://github.com/torvalds/linux/blob/8fa3b6f9392bf6d90cb7b908e07bd90166639f0a/include/uapi/linux/tcp.h#L163
>
> This is not going to be portable, but we can do this on only Linux
> which I suspect is where most production deployments of Unicorn that
> would benefit from this feature run. It's not useful in development
> (which is likely more common to not be Linux). We could also introduce
> it under a new configuration option if desired. In my testing, this
> works to reject 100% of requests early when not on loopback.
Good to know about it's effectiveness! I don't mind adding
Linux-only features as long as existing functionality still
works on the server-oriented *BSDs.
> The code is essentially:
>
> def client_closed?
> tcp_info = socket.getsockopt(Socket::SOL_TCP, Socket::TCP_INFO)
> state = tcp_info.unpack("c")[0]
> state == TCP_CLOSE || state == TCP_CLOSE_WAIT
> end
+Cc: raindrops-public@bogomips.org -> https://bogomips.org/raindrops-public/
Fwiw, raindrops (already a dependency of unicorn) has TCP_INFO
support under Linux:
https://bogomips.org/raindrops.git/tree/ext/raindrops/linux_tcp_info.c
I haven't used it, much, but FreeBSD also implements a subset of
this feature, nowadays, and ought to be supportable, too. We
can look into adding it for raindrops.
I don't know about other BSDs...
> This could be called at the top of `#process_client` in `http_server.rb`.
>
> Would there be interest in this upstream? Any comments on this
> proposed implementation? Currently, we're using a middleware with the
> Rack hijack API, but this seems like it belongs at the webserver
> level.
I guess hijack means you have to discard other middlewares and
the normal rack response handling in unicorn? If so, ouch!
Unfortunately, without hijack, there's no portable way in Rack
to get at the underlying IO object; so I guess this needs to
be done at the server level...
So yeah, I guess it belongs in the webserver.
I think we can automatically use TCP_INFO if available for
check_client_connection; then fall back to the old early write
hack for Unix sockets and systems w/o TCP_INFO (which would set
the response_start_sent flag).
No need to introduce yet another configuration option.
^ permalink raw reply [relevance 0%]
* check_client_connection using getsockopt(2)
@ 2017-02-22 12:02 6% Simon Eskildsen
2017-02-22 18:33 0% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Simon Eskildsen @ 2017-02-22 12:02 UTC (permalink / raw)
To: unicorn-public
Hello!
Almost five years ago Tom Burns contributed the patch in collaboration
with Eric that introduced the `check_client_connection` configuration
option. To summarize the patch, it was a solution to a problem we have
of rapid refreshes during sales where Unicorn would render a page 5
times if an eager customer refreshed Shopify 5 times, despite only
seeing one-rendering. This is a large amount of lost capacity. Four
of these connections would effectively be in a `CLOSE` state in the
backlog, get `accept(2)`ed and a response would be sent back only to
get an error that the client had closed its connection.
The patch solved this problem by instead of doing a single `write(2)`,
it would do a write of the generic HTTP version line, then jump into
the middleware stack and render the Rack response in a second write.
If the client had closed, the first `write(2)` of the HTTP version
header would usually throw an exception causing Unicorn to bail before
rendering the heavy Rack response. This saves a large amount of
capacity during spiky traffic.
A subsequent commit after testing by Eric revealed that:
> This only affects clients connecting over Unix domain sockets and TCP via loopback (127...*). It is unlikely to detect disconnects if the client is on a remote host (even on a fast LAN).
Thanks for that testing Eric. If we hadn't stumbled upon this in the
documentation proactively, this wouldn't have been easy to debug in
production.
In my testing, I can confirm Eric's tests. My testing essentially
consists of a snippet like the following to send rapid requests and
then closing the client. I've confirmed with Wireshark this is roughly
how popular browsers behave when refreshing fast on a slowly rendered
page:
100.times do |i|
client = TCPSocket.new("some-unicorn", 20_000)
client.write("GET /collections/#{rand(10000)}
HTTP/1.1\r\nHost:walrusser.myshopify.com\r\n\r\n")
client.close
end
This confirms Eric's comment that the existing
`check_client_connection` works perfectly on loopback, but as soon as
you put an actual network between the Unicorn and client—it's only
effective 20% of the time, even with `TCP_NODELAY`. I'm assuming, due
to buffering, even when disabling Nagle's. As we're changing our
architecture, we move from ngx (lb) -> ngx (host) -> unicorn to ngx
(lb) -> unicorn. That means this patch will no longer work for us.
I propose instead of the early `write(2)` hack, that we use
`getsockopt(2)` with the `TCP_INFO` flag and read the state of the
socket. If it's in `CLOSE_WAIT` or `CLOSE`, we kill the connection and
move on to the next.
https://github.com/torvalds/linux/blob/8fa3b6f9392bf6d90cb7b908e07bd90166639f0a/include/uapi/linux/tcp.h#L163
This is not going to be portable, but we can do this on only Linux
which I suspect is where most production deployments of Unicorn that
would benefit from this feature run. It's not useful in development
(which is likely more common to not be Linux). We could also introduce
it under a new configuration option if desired. In my testing, this
works to reject 100% of requests early when not on loopback.
The code is essentially:
def client_closed?
tcp_info = socket.getsockopt(Socket::SOL_TCP, Socket::TCP_INFO)
state = tcp_info.unpack("c")[0]
state == TCP_CLOSE || state == TCP_CLOSE_WAIT
end
This could be called at the top of `#process_client` in `http_server.rb`.
Would there be interest in this upstream? Any comments on this
proposed implementation? Currently, we're using a middleware with the
Rack hijack API, but this seems like it belongs at the webserver
level.
^ permalink raw reply [relevance 6%]
* [ANN] unicorn 5.0.0.pre1 - incompatible changes!
@ 2015-06-15 22:56 5% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2015-06-15 22:56 UTC (permalink / raw)
To: unicorn-public
This release finally drops Ruby 1.8 support and requires Ruby 1.9.3
or later. The horrible "Status:" header in our HTTP response is
finally gone, saving at least 16 precious bytes in every single HTTP
response.
Under Ruby 2.1 and later, the monotonic clock is used for timeout
handling for better accuracy.
Several experimental, unused and undocumented features are removed.
There's also tiny, minor performance and memory improvements from
dropping 1.8 compatibility, but probably nothing noticeable on a
typical real-life (bloated) app.
The biggest performance improvement we made was to our website by
switching to olddoc. Depending on connection speed, latency, and
renderer performance, it typically loads two to four times faster.
Finally, for the billionth time: unicorn must never be exposed
to slow clients, as it will never ever use new-fangled things
like non-blocking socket I/O, threads, epoll or kqueue. unicorn
must be used with a fully-buffering reverse proxy such as nginx
for slow clients.
I'll tag 5.0.0 final in a week or so if all goes well
= gem install --pre unicorn
= git clone git://bogomips.org/unicorn.git
= http://unicorn.bogomips.org/
* ISSUES: update with mailing list subscription
* GIT-VERSION-GEN: start 5.0.0 development
* http: remove xftrust options
* FAQ: add entry for Rails autoflush_log
* dev: remove isolate dependency
* unicorn.gemspec: depend on test-unit 3.0
* http_response: remove Status: header
* remove RubyForge and Freecode references
* remove mongrel.rubyforge.org references
* http: remove the keepalive requests limit
* http: reduce parser from 72 to 56 bytes on 64-bit
* examples: add run_once to before_fork hook example
* worker: remove old tmp accessor
* http_server: save 450+ bytes of memory on x86-64
* t/t0002-parser-error.sh: relax test for rack 1.6.0
* remove SSL support
* tmpio: drop the "size" method
* switch docs + website to olddoc
* README: clarify/reduce references to unicorn_rails
* gemspec: fixup olddoc migration
* use the monotonic clock under Ruby 2.1+
* http: -Wshorten-64-to-32 warnings on clang
* remove old inetd+git examples and exec_cgi
* http: standalone require + reduction in binary size
* GNUmakefile: fix clean gem build + reduce build cruft
* socket_helper: reduce constant lookups and caching
* remove 1.8, <= 1.9.1 fallback for missing IO#autoclose=
* favor IO#close_on_exec= over fcntl in 1.9+
* use require_relative to reduce syscalls at startup
* doc: update support status for Ruby versions
* fix uninstalled testing and reduce require paths
* test_socket_helper: do not depend on SO_REUSEPORT
* favor "a.b(&:c)" form over "a.b { |x| x.c }"
* ISSUES: add section for bugs in other projects
* http_server: favor ivars over constants
* explain 11 byte magic number for self-pipe
* const: drop constants used by Rainbows!
* reduce and localize constant string use
* Links: mark Rainbows! as historical, reference yahns
* save about 200 bytes of memory on x86-64
* http: remove deprecated reset method
* http: remove experimental dechunk! method
* socket_helper: update comments
* doc: document UNICORN_FD in manpage
* doc: document Etc.nprocessors for worker_processes
* favor more string literals for cold call sites
* tee_input: support for Rack::TempfileReaper middleware
* support TempfileReaper in deployment and development envs
* favor kgio_wait_readable for single FD over select
* Merge tag 'v4.9.0'
* http_request: support rack.hijack by default
* avoid extra allocation for hijack proc creation
* FAQ: add note about ECONNRESET errors from bodies
* process SIGWINCH unless stdin is a TTY
* ISSUES: discourage HTML mail strongly, welcome nyms
* http: use rb_hash_clear in Ruby 2.0+
* http_response: avoid special-casing for Rack < 1.5
* www: install NEWS.atom.xml properly
* http_server: remove a few more accessors and constants
* http_response: simplify regular expression
* move the socket into Rack env for hijacking
* http: move response_start_sent into the C ext
* FAQ: reorder bit on Rack 1.1.x and Rails 2.3.x
* ensure body is closed during hijack
--
EW
^ permalink raw reply [relevance 5%]
* [PATCH 2/2] http: move response_start_sent into the C ext
2015-06-06 1:58 7% [PATCH 0/2] eliminate generic ivars from HttpRequest class Eric Wong
2015-06-06 1:58 5% ` [PATCH 1/2] move the socket into Rack env for hijacking Eric Wong
@ 2015-06-06 1:58 4% ` Eric Wong
1 sibling, 0 replies; 73+ results
From: Eric Wong @ 2015-06-06 1:58 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Combined with the previous commit to eliminate the `@socket'
instance variable, this eliminates the last instance variable
in the Unicorn::HttpRequest class.
Eliminating the last instance variable avoids the creation of a
internal hash table used for implementing the "generic" instance
variables found in non-pure-Ruby classes. Method entry overhead
remains the same.
While this change doesn't do a whole lot for unicorn memory usage
where the HttpRequest is a singleton, it helps other HTTP servers
which rely on this code where thousands of clients may be connected.
---
ext/unicorn_http/unicorn_http.rl | 26 +++++++++++++++++++++++---
lib/unicorn/http_request.rb | 4 +---
test/unit/test_http_parser_ng.rb | 11 +++++++++++
3 files changed, 35 insertions(+), 6 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index bd45dd0..a5f069d 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -25,6 +25,7 @@ void init_unicorn_httpdate(void);
#define UH_FL_KAVERSION 0x80
#define UH_FL_HASHEADER 0x100
#define UH_FL_TO_CLEAR 0x200
+#define UH_FL_RESSTART 0x400 /* for check_client_connection */
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
@@ -60,7 +61,7 @@ struct http_parser {
} len;
};
-static ID id_set_backtrace, id_response_start_sent;
+static ID id_set_backtrace;
#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
# define my_hash_clear(h) (void)rb_hash_clear(h)
@@ -597,7 +598,6 @@ static VALUE HttpParser_clear(VALUE self)
http_parser_init(hp);
my_hash_clear(hp->env);
- rb_ivar_set(self, id_response_start_sent, Qfalse);
return self;
}
@@ -880,6 +880,25 @@ static VALUE HttpParser_filter_body(VALUE self, VALUE dst, VALUE src)
return src;
}
+static VALUE HttpParser_rssset(VALUE self, VALUE boolean)
+{
+ struct http_parser *hp = data_get(self);
+
+ if (RTEST(boolean))
+ HP_FL_SET(hp, RESSTART);
+ else
+ HP_FL_UNSET(hp, RESSTART);
+
+ return boolean; /* ignored by Ruby anyways */
+}
+
+static VALUE HttpParser_rssget(VALUE self)
+{
+ struct http_parser *hp = data_get(self);
+
+ return HP_FL_TEST(hp, RESSTART) ? Qtrue : Qfalse;
+}
+
#define SET_GLOBAL(var,str) do { \
var = find_common_field(str, sizeof(str) - 1); \
assert(!NIL_P(var) && "missed global field"); \
@@ -914,6 +933,8 @@ void Init_unicorn_http(void)
rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
rb_define_method(cHttpParser, "env", HttpParser_env, 0);
+ rb_define_method(cHttpParser, "response_start_sent=", HttpParser_rssset, 1);
+ rb_define_method(cHttpParser, "response_start_sent", HttpParser_rssget, 0);
/*
* The maximum size a single chunk when using chunked transfer encoding.
@@ -939,7 +960,6 @@ void Init_unicorn_http(void)
SET_GLOBAL(g_content_length, "CONTENT_LENGTH");
SET_GLOBAL(g_http_connection, "CONNECTION");
id_set_backtrace = rb_intern("set_backtrace");
- id_response_start_sent = rb_intern("@response_start_sent");
init_unicorn_httpdate();
#ifndef HAVE_RB_HASH_CLEAR
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index f5c6b5b..9339bce 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -25,8 +25,6 @@ class Unicorn::HttpParser
RACK_HIJACK_IO = "rack.hijack_io".freeze
NULL_IO = StringIO.new("")
- attr_accessor :response_start_sent
-
# :stopdoc:
# A frozen format for this is about 15% faster
# Drop these frozen strings when Ruby 2.2 becomes more prevalent,
@@ -92,7 +90,7 @@ class Unicorn::HttpParser
# detect if the socket is valid by writing a partial response:
if @@check_client_connection && headers?
- @response_start_sent = true
+ self.response_start_sent = true
HTTP_RESPONSE_START.each { |c| socket.write(c) }
end
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index efd82e1..d186f5a 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -34,6 +34,17 @@ class HttpParserNgTest < Test::Unit::TestCase
assert_equal false, @parser.response_start_sent
end
+ def test_response_start_sent
+ assert_equal false, @parser.response_start_sent, "default is false"
+ @parser.response_start_sent = true
+ assert_equal true, @parser.response_start_sent
+ @parser.response_start_sent = false
+ assert_equal false, @parser.response_start_sent
+ @parser.response_start_sent = true
+ @parser.clear
+ assert_equal false, @parser.response_start_sent
+ end
+
def test_connection_TE
@parser.buf << "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: TE\r\n"
@parser.buf << "TE: trailers\r\n\r\n"
--
EW
^ permalink raw reply related [relevance 4%]
* [PATCH 1/2] move the socket into Rack env for hijacking
2015-06-06 1:58 7% [PATCH 0/2] eliminate generic ivars from HttpRequest class Eric Wong
@ 2015-06-06 1:58 5% ` Eric Wong
2015-06-06 1:58 4% ` [PATCH 2/2] http: move response_start_sent into the C ext Eric Wong
1 sibling, 0 replies; 73+ results
From: Eric Wong @ 2015-06-06 1:58 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
This avoids the expensive generic instance variable for @socket
and exposes the socket as `env["unicorn.socket"]' to the Rack
application.
As as nice side-effect, applications may access
`env["unicorn.socket"]' as part of the API may be useful for
3rd-party bits such as Raindrops::TCP_Info for reading the tcp_info
struct on Linux-based systems.
Yes, `env["unicorn.socket"]' is a proprietary API in unicorn!
News at 11! But then again, unicorn is not the first Rack server
to implement `env["#{servername}.socket"]', either...
---
lib/unicorn/http_request.rb | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index b60e383..f5c6b5b 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -33,6 +33,7 @@ class Unicorn::HttpParser
# 2.2+ optimizes hash assignments when used with literal string keys
REMOTE_ADDR = 'REMOTE_ADDR'.freeze
RACK_INPUT = 'rack.input'.freeze
+ UNICORN_SOCKET = 'unicorn.socket'.freeze
HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
@@input_class = Unicorn::TeeInput
@@check_client_connection = false
@@ -99,7 +100,7 @@ class Unicorn::HttpParser
NULL_IO : @@input_class.new(socket, self)
# for Rack hijacking in Rack 1.5 and later
- @socket = socket
+ e[UNICORN_SOCKET] = socket
e[RACK_HIJACK] = self
e.merge!(DEFAULTS)
@@ -108,7 +109,7 @@ class Unicorn::HttpParser
# for rack.hijack, we respond to this method so no extra allocation
# of a proc object
def call
- env[RACK_HIJACK_IO] = @socket
+ env[RACK_HIJACK_IO] = env[UNICORN_SOCKET]
end
def hijacked?
--
EW
^ permalink raw reply related [relevance 5%]
* [PATCH 0/2] eliminate generic ivars from HttpRequest class
@ 2015-06-06 1:58 7% Eric Wong
2015-06-06 1:58 5% ` [PATCH 1/2] move the socket into Rack env for hijacking Eric Wong
2015-06-06 1:58 4% ` [PATCH 2/2] http: move response_start_sent into the C ext Eric Wong
0 siblings, 2 replies; 73+ results
From: Eric Wong @ 2015-06-06 1:58 UTC (permalink / raw)
To: unicorn-public
With the mainline Ruby VM, generic instance variables are implemented as
a st (hash) table for each object costing at least 192 bytes. This
isn't a huge problem for unicorn as it only ever allocates one
HttpRequest object, but still makes the HttpRequest class less suitable
for other servers.
While generic ivars will be less expensive when Ruby 2.3 is released in
December, we're still better off eliminating them entirely as they're
not going to be cheaper than T_OBJECT instance variables.
With this, I'll probably tag and release 5.0.0-rc1 soon.
Eric Wong (2):
move the socket into Rack env for hijacking
http: move response_start_sent into the C ext
ext/unicorn_http/unicorn_http.rl | 26 +++++++++++++++++++++++---
lib/unicorn/http_request.rb | 9 ++++-----
test/unit/test_http_parser_ng.rb | 11 +++++++++++
3 files changed, 38 insertions(+), 8 deletions(-)
Note: Yes, [PATCH 1/2] introduces a unicorn-specific field into the
Rack env, but unicorn is not the only server with
`env["#{servername}.socket"]' in the Rack env.
And [PATCH 2/2] isn't useful without [PATCH 1/2]
^ permalink raw reply [relevance 7%]
* Re: No, passenger 5.0 is not faster than unicorn :)
2014-12-03 9:56 6% ` Sam Saffron
@ 2014-12-03 9:57 0% ` Sam Saffron
0 siblings, 0 replies; 73+ results
From: Sam Saffron @ 2014-12-03 9:57 UTC (permalink / raw)
To: Bráulio Bhavamitra; +Cc: unicorn-public, Hitendra Hugo Melo
oops sent wrong link meant to send this
https://meta.discourse.org/t/raptor-web-server/21304/6
On Wed, Dec 3, 2014 at 8:56 PM, Sam Saffron <sam.saffron@gmail.com> wrote:
> I covered this here:
> http://discuss.topazlabs.com/t/amidst-blizzards-they-rest/1147
>
> it seems like an odd marketing move to me ... optimising a bit that
> needs very little help. heck ripping out hashie and the 50 frames
> omniauth injects would have a significantly bigger impact on rails
> apps out there than optimising the 0.5% that needs little optimising.
>
> On Wed, Dec 3, 2014 at 8:50 PM, Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>> Hello all,
>>
>> I've just tested a one instance each (one worker with unicorn and
>> --max-pool-size 1 passenger 5) on the rails app I work.
>>
>> And the results are just as I expected, no miracle at all: Unicorn is
>> still the fatest!
>> (the difference is only a few milliseconds less per request)
>>
>> The blocking design of unicorn is proving itself very efficient.
>>
>> cheers!
>> bráulio
>>
^ permalink raw reply [relevance 0%]
* Re: No, passenger 5.0 is not faster than unicorn :)
@ 2014-12-03 9:56 6% ` Sam Saffron
2014-12-03 9:57 0% ` Sam Saffron
0 siblings, 1 reply; 73+ results
From: Sam Saffron @ 2014-12-03 9:56 UTC (permalink / raw)
To: Bráulio Bhavamitra; +Cc: unicorn-public, Hitendra Hugo Melo
I covered this here:
http://discuss.topazlabs.com/t/amidst-blizzards-they-rest/1147
it seems like an odd marketing move to me ... optimising a bit that
needs very little help. heck ripping out hashie and the 50 frames
omniauth injects would have a significantly bigger impact on rails
apps out there than optimising the 0.5% that needs little optimising.
On Wed, Dec 3, 2014 at 8:50 PM, Bráulio Bhavamitra <braulio@eita.org.br> wrote:
> Hello all,
>
> I've just tested a one instance each (one worker with unicorn and
> --max-pool-size 1 passenger 5) on the rails app I work.
>
> And the results are just as I expected, no miracle at all: Unicorn is
> still the fatest!
> (the difference is only a few milliseconds less per request)
>
> The blocking design of unicorn is proving itself very efficient.
>
> cheers!
> bráulio
>
^ permalink raw reply [relevance 6%]
* Re: dropping Ruby 1.8 support for unicorn 5?
2014-09-27 8:32 5% dropping Ruby 1.8 support for unicorn 5? Eric Wong
@ 2014-09-27 8:37 0% ` Ernest W. Durbin III
0 siblings, 0 replies; 73+ results
From: Ernest W. Durbin III @ 2014-09-27 8:37 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]
As the submitter of the referenced 1.8.6 patch *still ashamed* I can say that I fully support a 4.x series with “maintenance” level support.
Since the language won’t be changing, the 4.x series should be a very quiet branch indeed.
Onward, to greater things.
-Ernest
On Sep 27, 2014, at 4:32 AM, Eric Wong <e@80x24.org> wrote:
> We've brought this up a few times, but I suppose we might as well drop
> 1.8 support in a major version change.
>
> We may still maintain unicorn 4.x for 1.8 users indefinitely; after all,
> we only accepted a patch for 1.8.6 compatibility less than a year
> ago(!)[1]. So I'll still feel a _little_ bad for dropping 1.8 :x
>
> One big reason for this is it looks like Ruby will move towards
> deprecating old Data_* macros for superior (1.9+-only) TypedData_*
> macros in the next few years[2]. The theme for unicorn 5 is mostly
> dropping old, unused crap anyways; and not gaining new bloat.
>
> Worst case is we support 1.8 and avoid deprecation warnings through
> the use of ifdefs in the HTTP parser, but I'm no fan of ifdefs.
>
>
> [1] commit 7e9e4c740aba24096f768f578779dc1053cb8b70
> (construct listener_fds Hash in 1.8.6 compatible way)
>
> [2] http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=47717
> This is due to type-checking issues like
> https://bugs.ruby-lang.org/issues/10296
>
^ permalink raw reply [relevance 0%]
* dropping Ruby 1.8 support for unicorn 5?
@ 2014-09-27 8:32 5% Eric Wong
2014-09-27 8:37 0% ` Ernest W. Durbin III
0 siblings, 1 reply; 73+ results
From: Eric Wong @ 2014-09-27 8:32 UTC (permalink / raw)
To: unicorn-public
We've brought this up a few times, but I suppose we might as well drop
1.8 support in a major version change.
We may still maintain unicorn 4.x for 1.8 users indefinitely; after all,
we only accepted a patch for 1.8.6 compatibility less than a year
ago(!)[1]. So I'll still feel a _little_ bad for dropping 1.8 :x
One big reason for this is it looks like Ruby will move towards
deprecating old Data_* macros for superior (1.9+-only) TypedData_*
macros in the next few years[2]. The theme for unicorn 5 is mostly
dropping old, unused crap anyways; and not gaining new bloat.
Worst case is we support 1.8 and avoid deprecation warnings through
the use of ifdefs in the HTTP parser, but I'm no fan of ifdefs.
[1] commit 7e9e4c740aba24096f768f578779dc1053cb8b70
(construct listener_fds Hash in 1.8.6 compatible way)
[2] http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=47717
This is due to type-checking issues like
https://bugs.ruby-lang.org/issues/10296
^ permalink raw reply [relevance 5%]
* Re: Rack encodings (was: Please move to github)
2014-08-05 5:56 14% Rack encodings (was: Please move to github) Gary Grossman
@ 2014-08-05 6:28 6% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2014-08-05 6:28 UTC (permalink / raw)
To: Gary Grossman; +Cc: hongli, unicorn-public, michael, mfischer
Gary Grossman <gary.grossman@gmail.com> wrote:
> It feels like we were getting some momentum here on an important but
> long-dormant issue here... maybe it's time to move this discussion
> to rack-devel?
Sure, rack-devel is a pretty dormant mailing list but there's been a
burst of activity a few weeks ago.
Unlike this list, subscription is required to post; and first posts
from newbies are moderated. For folks who do not login to Google
(crazies like me :P) subscription is possible without any login
or password: rack-devel+subscribe@googlegroups.com
> Perhaps there's another Rack luminary who can lead
> the charge, or at least see if there's some consensus after a few
> more years of shared experience on what "sane" encodings might
> look like.
At least there's other server implementers who'll probably
chime in.
> A lightweight way to move the implementation forward might be a
> simple Rack middleware gem which sets the new encodings on the
> environment, or adding the functionality to rack itself. Once
> developers were comfortable with the new regime, the app servers
> could follow suit and put those encodings in the env natively,
> and the Rubyland implementation of the new encodings could be
> dropped.
Sounds like a good plan. Thanks for bringing more attention to this.
^ permalink raw reply [relevance 6%]
* Re: Rack encodings (was: Please move to github)
@ 2014-08-05 5:56 14% Gary Grossman
2014-08-05 6:28 6% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Gary Grossman @ 2014-08-05 5:56 UTC (permalink / raw)
To: hongli; +Cc: unicorn-public, michael, e, mfischer, gary.grossman
It feels like we were getting some momentum here on an important but
long-dormant issue here... maybe it's time to move this discussion
to rack-devel? Perhaps there's another Rack luminary who can lead
the charge, or at least see if there's some consensus after a few
more years of shared experience on what "sane" encodings might
look like.
A lightweight way to move the implementation forward might be a
simple Rack middleware gem which sets the new encodings on the
environment, or adding the functionality to rack itself. Once
developers were comfortable with the new regime, the app servers
could follow suit and put those encodings in the env natively,
and the Rubyland implementation of the new encodings could be
dropped.
Gary
^ permalink raw reply [relevance 14%]
* Re: Rack encodings (was: Please move to github)
2014-08-04 8:48 6% ` Rack encodings (was: Please move to github) Eric Wong
@ 2014-08-04 9:46 6% ` Hongli Lai
0 siblings, 0 replies; 73+ results
From: Hongli Lai @ 2014-08-04 9:46 UTC (permalink / raw)
To: Eric Wong; +Cc: Michael Fischer, Gary Grossman, unicorn-public, Michael Grosser
On Mon, Aug 4, 2014 at 10:48 AM, Eric Wong <e@80x24.org> wrote:
> Fair enough. Would you/Phusion be comfortable taking the lead here?
> This feels like another "hot potato" issue :>
Unfortunately, we're too busy with a major project to be able to lead
this effort.
--
Phusion | Web Application deployment, scaling, and monitoring solutions
Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
^ permalink raw reply [relevance 6%]
* Rack encodings (was: Please move to github)
2014-08-04 7:22 5% ` Hongli Lai
@ 2014-08-04 8:48 6% ` Eric Wong
2014-08-04 9:46 6% ` Hongli Lai
0 siblings, 1 reply; 73+ results
From: Eric Wong @ 2014-08-04 8:48 UTC (permalink / raw)
To: Hongli Lai
Cc: Michael Fischer, Gary Grossman, unicorn-public, Michael Grosser
(Long overdue Subject: change)
Hongli Lai <hongli@phusion.nl> wrote:
> Hi guys. Phusion Passenger author here. I would very much support
> standardization of encoding issues.
> At this point, I don't really care what the standard is, as long as
> it's a sane standard that everybody can follow.
Fair enough. Would you/Phusion be comfortable taking the lead here?
This feels like another "hot potato" issue :>
> In my opinion, following Encoding.default_external is not helpful.
> Most users have absolutely no idea how to configure
> Encoding.default_external, or even know that it exists. I've also
> never, ever seen anybody who does *not* want default_external to be
> UTF-8. If it's not set to UTF-8, then it's always by accident (e.g.
> the user not knowing that it depends on LC_CTYPE, that LC_CTYPE is set
> differently in the shell than from an init script, or even what
> LC_CTYPE is).
Perhaps we need to educate users to set LC_CTYPE/LC_ALL/LANG so
Encoding.default_external works as intended? Adding another
option to Rack will just as likely to get missed.
Maybe servers could emit a big warning saying:
WARNING: Encoding.default_external is not UTF-8 ...
And add a --quiet-utf8-warning option for the few folks who really do
not want UTF-8.
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-02 19:33 6% ` Michael Fischer
@ 2014-08-04 7:22 5% ` Hongli Lai
2014-08-04 8:48 6% ` Rack encodings (was: Please move to github) Eric Wong
0 siblings, 1 reply; 73+ results
From: Hongli Lai @ 2014-08-04 7:22 UTC (permalink / raw)
To: Michael Fischer; +Cc: Gary Grossman, Eric Wong, unicorn-public, Michael Grosser
On Sat, Aug 2, 2014 at 9:33 PM, Michael Fischer <mfischer@zendesk.com> wrote:
> On Sat, Aug 2, 2014 at 12:07 PM, Gary Grossman <gary.grossman@gmail.com>
> wrote:
>> This might be one of those instances where it would be helpful for
>> implementation to lead specification. Unicorn is one of the leading
>> servers of its genre, if not the leader. If you supported a switch
>> that made the encoding regime more sane, I think other popular servers
>> like Thin and Passenger would swiftly follow and it might re-energize
>> the discussion about getting encodings into the Rack spec once and
>> for all, and give a base for experimentation and iteration for
>> getting the encodings in the spec right.
>>
>
> I agree with Gary here. It's often too easy to decide to preserve the
> status quo because things work well enough -- and then, eventually, time
> catches up with you and it no longer does.
Hi guys. Phusion Passenger author here. I would very much support
standardization of encoding issues. Every now and then, a user submits
a bug report on Phusion Passenger, mentioning an encoding problem. The
user would say that the problem occurs on Phusion Passenger but not on
Unicorn/Thin/etc. The Rack spec doesn't say anything about encodings
so strictly speaking it's not "our fault", but it's still hard to tell
users that it's "their fault" or "their framework's fault" based on
this alone. It's also not a helpful answer: users often have no idea
what to do about the issue.
At this point, I don't really care what the standard is, as long as
it's a sane standard that everybody can follow.
In my opinion, following Encoding.default_external is not helpful.
Most users have absolutely no idea how to configure
Encoding.default_external, or even know that it exists. I've also
never, ever seen anybody who does *not* want default_external to be
UTF-8. If it's not set to UTF-8, then it's always by accident (e.g.
the user not knowing that it depends on LC_CTYPE, that LC_CTYPE is set
differently in the shell than from an init script, or even what
LC_CTYPE is).
--
Phusion | Web Application deployment, scaling, and monitoring solutions
Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
^ permalink raw reply [relevance 5%]
* Re: Please move to github
2014-08-02 19:07 4% ` Gary Grossman
2014-08-02 19:33 6% ` Michael Fischer
@ 2014-08-02 20:15 6% ` Eric Wong
1 sibling, 0 replies; 73+ results
From: Eric Wong @ 2014-08-02 20:15 UTC (permalink / raw)
To: Gary Grossman; +Cc: unicorn-public, michael
Gary Grossman <gary.grossman@gmail.com> wrote:
> We'd pretty much need to introduce some kind of configuration
> switch, at least for the short term and maybe for the long term.
> The hope would be that it could become the default setting.
> Apps that don't use UTF8 should be able to set their desired default
> external encoding appropriately.
If possible, I would like to avoid an option and rely on
Encoding.default_external in a new major version. Too many ways to set
the same thing is confusing and requires extra documentation overhead.
> >The rack-devel mailing list had a discussion on this in September 2010
> >and a decision was never reached. You can search the archives at:
> >http://groups.google.com/group/rack-devel
>
> I came across this thread but didn't realize that was the last word
> so far when it came to Rack and encodings.
>
> This might be one of those instances where it would be helpful for
> implementation to lead specification. Unicorn is one of the leading
> servers of its genre, if not the leader. If you supported a switch
> that made the encoding regime more sane, I think other popular servers
> like Thin and Passenger would swiftly follow and it might re-energize
> the discussion about getting encodings into the Rack spec once and
> for all, and give a base for experimentation and iteration for
> getting the encodings in the spec right.
I might start with WEBrick (or the Rack/WEBrick handler). WEBrick is
distributed with Ruby and maintained by the core team. It's not used in
production much, but it the reference implementation which is usable
from all Ruby implementations.
naruse (from that rack-devel thread) is also active in Ruby core and
is very knowledgeable in these areas.
> Thanks again for reviewing the patch. I'll work on a new patch that
> incorporates your comments and has a switch for enabling/disabling
> the functionality, and I'll try to follow roughly what the spec
> group in 2010 thought would make sense in terms of encodings for
> the various strings in the env. And I'll see if I can ask the
> Rack folks to chime in.
Definitely get other Rack folks to chime in, even if it is a
unicorn-only change. This has been a problem for years already,
so taking more time to get things right won't hurt.
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-02 19:07 4% ` Gary Grossman
@ 2014-08-02 19:33 6% ` Michael Fischer
2014-08-04 7:22 5% ` Hongli Lai
2014-08-02 20:15 6% ` Please move to github Eric Wong
1 sibling, 1 reply; 73+ results
From: Michael Fischer @ 2014-08-02 19:33 UTC (permalink / raw)
To: Gary Grossman; +Cc: Eric Wong, unicorn-public, Michael Grosser
On Sat, Aug 2, 2014 at 12:07 PM, Gary Grossman <gary.grossman@gmail.com>
wrote:
This might be one of those instances where it would be helpful for
> implementation to lead specification. Unicorn is one of the leading
> servers of its genre, if not the leader. If you supported a switch
> that made the encoding regime more sane, I think other popular servers
> like Thin and Passenger would swiftly follow and it might re-energize
> the discussion about getting encodings into the Rack spec once and
> for all, and give a base for experimentation and iteration for
> getting the encodings in the spec right.
>
I agree with Gary here. It's often too easy to decide to preserve the
status quo because things work well enough -- and then, eventually, time
catches up with you and it no longer does.
If Gary's proposal makes sense, and improves matters without doing
significant harm -- despite it not adhering to the letter of Rack
compliance as it is currently specified today -- it would represent a major
step forward if implemented in Unicorn. (And as Gary suggested, the
specification and other implementors will probably catch up by necessity if
the behavior proves beneficial.)
The first step is to prove it's worth shaking the tree with some
benchmarks, though. :)
--Michael
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-02 8:50 4% ` Eric Wong
@ 2014-08-02 19:07 4% ` Gary Grossman
2014-08-02 19:33 6% ` Michael Fischer
2014-08-02 20:15 6% ` Please move to github Eric Wong
0 siblings, 2 replies; 73+ results
From: Gary Grossman @ 2014-08-02 19:07 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public, michael
Hi Eric,
Thanks for your reply and for reviewing the patch!
>Right, the Rack spec does not dictate this. Doing this out-of-spec has
>the ability to break existing apps as well as compatibility with other
>app servers.
It's true, my patch is too naive since it's a pretty drastic change
in behavior not behind any kind of switch.
>What do other app servers do?
I did a little survey. ASCII-8BIT is kind of the de facto standard
even if it's not mandated by the Rack specification. Phusion
Passenger, Thin and WEBrick all send mostly ASCII-8BIT strings in
the env.
>My main concern is having more different behavior between various Rack
>servers servers, making it harder to switch between them.
Very valid; Rack wouldn't be much of a standard if there were a bunch
of variants in use.
>Another concern is breaking apps which are already working around this
>but work with non-UTF-8 encodings.
We'd pretty much need to introduce some kind of configuration
switch, at least for the short term and maybe for the long term.
The hope would be that it could become the default setting.
Apps that don't use UTF8 should be able to set their desired default
external encoding appropriately.
>The rack-devel mailing list had a discussion on this in September 2010
>and a decision was never reached. You can search the archives at:
>http://groups.google.com/group/rack-devel
I came across this thread but didn't realize that was the last word
so far when it came to Rack and encodings.
This might be one of those instances where it would be helpful for
implementation to lead specification. Unicorn is one of the leading
servers of its genre, if not the leader. If you supported a switch
that made the encoding regime more sane, I think other popular servers
like Thin and Passenger would swiftly follow and it might re-energize
the discussion about getting encodings into the Rack spec once and
for all, and give a base for experimentation and iteration for
getting the encodings in the spec right.
There's a lot of developer pain here. Many apps probably are serving
up encoding-related 500 errors without knowing it. There are
stories of developers adding "# encoding" everywhere, setting
the external/internal encoding, and then "things are fine until it
blows up somewhere else." I heard recently that a very large company
has stuck with Ruby 1.8.7, probably to avoid these encoding issues
among other things. It would be nice to improve the situation.
>Disclaimer: I am not an encoding expert, so for that reason I prefer
>to let other Rack folks make the decision.
I'm not an encoding expert either! Most people aren't... which is
why it'd be nice if they didn't have to know so much about it when
they write a Rack app!
>Do you have performance measurements for doing this as pure-Ruby
>middleware vs your patch?
I don't have measurements currently but I'll get some.
Our app is several years old and so there's a lot of stuff in
request.env by the time we get around to forcing everything to
UTF8 encoding. I wouldn't be surprised if the hit on
every single request is small but significant for us.
>So it should be best if there were a way to do this for all Rack
>servers.
Thanks again for reviewing the patch. I'll work on a new patch that
incorporates your comments and has a switch for enabling/disabling
the functionality, and I'll try to follow roughly what the spec
group in 2010 thought would make sense in terms of encodings for
the various strings in the env. And I'll see if I can ask the
Rack folks to chime in.
Gary
^ permalink raw reply [relevance 4%]
* Re: Please move to github
2014-08-02 7:51 4% Gary Grossman
2014-08-02 7:54 6% ` Kapil Israni
@ 2014-08-02 8:50 4% ` Eric Wong
2014-08-02 19:07 4% ` Gary Grossman
1 sibling, 1 reply; 73+ results
From: Eric Wong @ 2014-08-02 8:50 UTC (permalink / raw)
To: Gary Grossman; +Cc: unicorn-public, michael
Gary Grossman <gary.grossman@gmail.com> wrote:
> Hi Eric,
>
> I work with Michael, and this discussion sure got off on the
> wrong foot... we love unicorn and use it heavily, and just
> want to contribute back to it.
No worries, cultural differences happen. Thanks for following up.
> To detail the encoding problem we were trying to fix, unicorn
> uses rb_str_new in several places to create Ruby strings.
> For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
> encoding.
>
> While the Rack specification doesn't dictate what encoding
> should be used for strings in the environment, many
> developers would probably expect the default external encoding
> setting in Encoding.default_external to be used.
Right, the Rack spec does not dictate this. Doing this out-of-spec has
the ability to break existing apps as well as compatibility with other
app servers.
What do other app servers do?
My main concern is having more different behavior between various Rack
servers servers, making it harder to switch between them.
Another concern is breaking apps which are already working around this
but work with non-UTF-8 encodings.
The rack-devel mailing list had a discussion on this in September 2010
and a decision was never reached. You can search the archives at:
http://groups.google.com/group/rack-devel
I've also saved the thread to a mbox at
http://80x24.org/rack-devel-encoding-2010.mbox.gz
since Google Groups archives are a bit painful to navigate.
Disclaimer: I am not an encoding expert, so for that reason I prefer
to let other Rack folks make the decision.
> Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
> in the env can lead to Encoding::CompatibilityErrors being
> raised when a UTF8 string and ASCII-8BIT string are concatenated,
> which happens frequently when properties like request.url are
> referenced in erb templates. To get around these problems,
> an app would have to force encoding on the strings in the env
> manually. It seems a shame to do this in slower Ruby code when
> it could be done up front by unicorn.
Yes, this existing behavior sucks on UTF-8-heavy apps. I would rather
not add more unicorn-only options which make switching between servers
harder.
Do you have performance measurements for doing this as pure-Ruby
middleware vs your patch?
My dislike of lock-in also applies to app servers. Application-visible
differences like these should be avoided so people can switch between
servers, too.
So it should be best if there were a way to do this for all Rack
servers.
> We'd like to propose that unicorn use rb_external_str_new to
> make strings instead of rb_str_new.
>
> Perhaps you have your reasons for continuing to use rb_str_new
> but we figured we'd run this by you.
If the Rack spec mandated encodings, I would do it in a heartbeat.
> Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
> rb_external_str_new instead of rb_str_new to create strings. The resulting
> strings will use the default external encoding. Continue using rb_str_new for
> older versions of Ruby.
A better, shorter, more direct subject would be:
Subject: use Encoding.default_external for header values
Commit message body is fine <snip>
> +#ifdef HAVE_RUBY_ENCODING_H
> +/* Use default external encoding for strings for Ruby 1.9+,
> + * fall back to rb_str_new when unavailable */
> +#define rb_str_new rb_external_str_new
> +#endif
This is too heavy-handed, as some strings (buffers) may
need to stay binary via rb_str_new. If we were to do this, it would
something like:
#ifdef HAVE_RUBY_ENCODING_H
# define env_val_new(ptr,len) rb_external_str_new((ptr),(len))
#else
# define env_val_new(ptr,len) rb_str_new((ptr),(len))
#endif
... And only making sure header values are set to external.
Last I checked the HTTP RFCs (it's been a while) header keys are
required to be US-ASCII-only (and our parser enforces that).
> + def test_encoding
> + if ''.respond_to?(:encoding)
> + client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
> + "Host: foo\r\n\r\n")
> + env = @request.read(client)
> + encoding = Encoding.default_external
> + assert_equal encoding, env['REQUEST_PATH'].encoding
> + assert_equal encoding, env['PATH_INFO'].encoding
> + assert_equal encoding, env['QUERY_STRING'].encoding
> + end
This would need to test and work with (and appropriately reject)
invalid requests with bad encodings, too.
^ permalink raw reply [relevance 4%]
* Re: Please move to github
2014-08-02 7:54 6% ` Kapil Israni
@ 2014-08-02 8:02 6% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2014-08-02 8:02 UTC (permalink / raw)
To: Kapil Israni; +Cc: unicorn-public
Kapil Israni <kapil.israni@gmail.com> wrote:
> How do I unsubscribe from this email list?
Send an email to: unicorn-public+unsubscribe@bogomips.org
(it should've been mentioned in the welcome message, and is in
every header).
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-02 7:51 4% Gary Grossman
@ 2014-08-02 7:54 6% ` Kapil Israni
2014-08-02 8:02 6% ` Eric Wong
2014-08-02 8:50 4% ` Eric Wong
1 sibling, 1 reply; 73+ results
From: Kapil Israni @ 2014-08-02 7:54 UTC (permalink / raw)
Cc: unicorn-public
How do I unsubscribe from this email list?
On Sat, Aug 2, 2014 at 12:51 AM, Gary Grossman <gary.grossman@gmail.com>
wrote:
> Hi Eric,
>
> I work with Michael, and this discussion sure got off on the
> wrong foot... we love unicorn and use it heavily, and just
> want to contribute back to it.
>
> To detail the encoding problem we were trying to fix, unicorn
> uses rb_str_new in several places to create Ruby strings.
> For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
> encoding.
>
> While the Rack specification doesn't dictate what encoding
> should be used for strings in the environment, many
> developers would probably expect the default external encoding
> setting in Encoding.default_external to be used.
>
> Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
> in the env can lead to Encoding::CompatibilityErrors being
> raised when a UTF8 string and ASCII-8BIT string are concatenated,
> which happens frequently when properties like request.url are
> referenced in erb templates. To get around these problems,
> an app would have to force encoding on the strings in the env
> manually. It seems a shame to do this in slower Ruby code when
> it could be done up front by unicorn.
>
> We'd like to propose that unicorn use rb_external_str_new to
> make strings instead of rb_str_new.
>
> Perhaps you have your reasons for continuing to use rb_str_new
> but we figured we'd run this by you.
>
> Here's a proposed patch.
>
> Gary
>
> From befb01530c8d930ba53cc58b979ddf42a4c12565 Mon Sep 17 00:00:00 2001
> From: Gary Grossman <gary.grossman@gmail.com>
> Date: Sat, 2 Aug 2014 00:19:30 -0700
> Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
> rb_external_str_new instead of rb_str_new to create strings. The resulting
> strings will use the default external encoding. Continue using rb_str_new
> for
> older versions of Ruby.
>
> Using the default external encoding instead of ASCII-8BIT for
> strings is more in line with developer expectations and will cause
> less unexpected bugs such as Encoding::CompatibilityErrors which
> result when, say, a UTF8 string and ASCII-8BIT string are
> concatenated together.
>
> Added a unit test to ensure that strings returned in the Rack
> environment conform to the default external encoding.
> ---
> ext/unicorn_http/ext_help.h | 6 ++++++
> test/unit/test_request.rb | 13 +++++++++++++
> 2 files changed, 19 insertions(+)
>
> diff --git a/ext/unicorn_http/ext_help.h b/ext/unicorn_http/ext_help.h
> index c87c272..6806f8e 100644
> --- a/ext/unicorn_http/ext_help.h
> +++ b/ext/unicorn_http/ext_help.h
> @@ -79,4 +79,10 @@ static int str_cstr_case_eq(VALUE val, const char *ptr,
> long len)
> #define STR_CSTR_CASE_EQ(val, const_str) \
> str_cstr_case_eq(val, const_str, sizeof(const_str) - 1)
>
> +#ifdef HAVE_RUBY_ENCODING_H
> +/* Use default external encoding for strings for Ruby 1.9+,
> + * fall back to rb_str_new when unavailable */
> +#define rb_str_new rb_external_str_new
> +#endif
> +
> #endif /* ext_help_h */
> diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
> index fbda1a2..0a105e0 100644
> --- a/test/unit/test_request.rb
> +++ b/test/unit/test_request.rb
> @@ -179,4 +179,17 @@ class RequestTest < Test::Unit::TestCase
> env['rack.input'].rewind
> res = @lint.call(env)
> end
> +
> + def test_encoding
> + if ''.respond_to?(:encoding)
> + client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
> + "Host: foo\r\n\r\n")
> + env = @request.read(client)
> + encoding = Encoding.default_external
> + assert_equal encoding, env['REQUEST_PATH'].encoding
> + assert_equal encoding, env['PATH_INFO'].encoding
> + assert_equal encoding, env['QUERY_STRING'].encoding
> + end
> + end
> +
> end
> --
> 1.9.1
>
>
>
--
Kapil
^ permalink raw reply [relevance 6%]
* Re: Please move to github
@ 2014-08-02 7:51 4% Gary Grossman
2014-08-02 7:54 6% ` Kapil Israni
2014-08-02 8:50 4% ` Eric Wong
0 siblings, 2 replies; 73+ results
From: Gary Grossman @ 2014-08-02 7:51 UTC (permalink / raw)
To: e; +Cc: unicorn-public, michael
Hi Eric,
I work with Michael, and this discussion sure got off on the
wrong foot... we love unicorn and use it heavily, and just
want to contribute back to it.
To detail the encoding problem we were trying to fix, unicorn
uses rb_str_new in several places to create Ruby strings.
For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
encoding.
While the Rack specification doesn't dictate what encoding
should be used for strings in the environment, many
developers would probably expect the default external encoding
setting in Encoding.default_external to be used.
Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
in the env can lead to Encoding::CompatibilityErrors being
raised when a UTF8 string and ASCII-8BIT string are concatenated,
which happens frequently when properties like request.url are
referenced in erb templates. To get around these problems,
an app would have to force encoding on the strings in the env
manually. It seems a shame to do this in slower Ruby code when
it could be done up front by unicorn.
We'd like to propose that unicorn use rb_external_str_new to
make strings instead of rb_str_new.
Perhaps you have your reasons for continuing to use rb_str_new
but we figured we'd run this by you.
Here's a proposed patch.
Gary
From befb01530c8d930ba53cc58b979ddf42a4c12565 Mon Sep 17 00:00:00 2001
From: Gary Grossman <gary.grossman@gmail.com>
Date: Sat, 2 Aug 2014 00:19:30 -0700
Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
rb_external_str_new instead of rb_str_new to create strings. The resulting
strings will use the default external encoding. Continue using rb_str_new for
older versions of Ruby.
Using the default external encoding instead of ASCII-8BIT for
strings is more in line with developer expectations and will cause
less unexpected bugs such as Encoding::CompatibilityErrors which
result when, say, a UTF8 string and ASCII-8BIT string are
concatenated together.
Added a unit test to ensure that strings returned in the Rack
environment conform to the default external encoding.
---
ext/unicorn_http/ext_help.h | 6 ++++++
test/unit/test_request.rb | 13 +++++++++++++
2 files changed, 19 insertions(+)
diff --git a/ext/unicorn_http/ext_help.h b/ext/unicorn_http/ext_help.h
index c87c272..6806f8e 100644
--- a/ext/unicorn_http/ext_help.h
+++ b/ext/unicorn_http/ext_help.h
@@ -79,4 +79,10 @@ static int str_cstr_case_eq(VALUE val, const char *ptr, long len)
#define STR_CSTR_CASE_EQ(val, const_str) \
str_cstr_case_eq(val, const_str, sizeof(const_str) - 1)
+#ifdef HAVE_RUBY_ENCODING_H
+/* Use default external encoding for strings for Ruby 1.9+,
+ * fall back to rb_str_new when unavailable */
+#define rb_str_new rb_external_str_new
+#endif
+
#endif /* ext_help_h */
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..0a105e0 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -179,4 +179,17 @@ class RequestTest < Test::Unit::TestCase
env['rack.input'].rewind
res = @lint.call(env)
end
+
+ def test_encoding
+ if ''.respond_to?(:encoding)
+ client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
+ "Host: foo\r\n\r\n")
+ env = @request.read(client)
+ encoding = Encoding.default_external
+ assert_equal encoding, env['REQUEST_PATH'].encoding
+ assert_equal encoding, env['PATH_INFO'].encoding
+ assert_equal encoding, env['QUERY_STRING'].encoding
+ end
+ end
+
end
--
1.9.1
^ permalink raw reply related [relevance 4%]
* Re: Please move to github
@ 2014-08-02 7:46 4% Gary Grossman
0 siblings, 0 replies; 73+ results
From: Gary Grossman @ 2014-08-02 7:46 UTC (permalink / raw)
To: e; +Cc: unicorn-public, michael
Hi Eric,
I work with Michael, and this discussion sure got off on the
wrong foot... we love unicorn and use it heavily, and just
want to contribute back to it.
To detail the encoding problem we were trying to fix, unicorn
uses rb_str_new in several places to create Ruby strings.
For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
encoding.
While the Rack specification doesn't dictate what encoding
should be used for strings in the environment, many
developers would probably expect the default external encoding
setting in Encoding.default_external to be used.
Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
in the env can lead to Encoding::CompatibilityErrors being
raised when a UTF8 string and ASCII-8BIT string are concatenated,
which happens frequently when properties like request.url are
referenced in erb templates. To get around these problems,
an app would have to force encoding on the strings in the env
manually. It seems a shame to do this in slower Ruby code when
it could be done up front by unicorn.
We'd like to propose that unicorn use rb_external_str_new to
make strings instead of rb_str_new.
Perhaps you have your reasons for continuing to use rb_str_new
but we figured we'd run this by you.
Here's a proposed patch.
From befb01530c8d930ba53cc58b979ddf42a4c12565 Mon Sep 17 00:00:00 2001
From: Gary Grossman <gary.grossman@gmail.com>
Date: Sat, 2 Aug 2014 00:19:30 -0700
Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
rb_external_str_new instead of rb_str_new to create strings. The resulting
strings will use the default external encoding. Continue using rb_str_new
for
older versions of Ruby.
Using the default external encoding instead of ASCII-8BIT for
strings is more in line with developer expectations and will cause
less unexpected bugs such as Encoding::CompatibilityErrors which
result when, say, a UTF8 string and ASCII-8BIT string are
concatenated together.
Added a unit test to ensure that strings returned in the Rack
environment conform to the default external encoding.
---
ext/unicorn_http/ext_help.h | 6 ++++++
test/unit/test_request.rb | 13 +++++++++++++
2 files changed, 19 insertions(+)
diff --git a/ext/unicorn_http/ext_help.h b/ext/unicorn_http/ext_help.h
index c87c272..6806f8e 100644
--- a/ext/unicorn_http/ext_help.h
+++ b/ext/unicorn_http/ext_help.h
@@ -79,4 +79,10 @@ static int str_cstr_case_eq(VALUE val, const char *ptr,
long len)
#define STR_CSTR_CASE_EQ(val, const_str) \
str_cstr_case_eq(val, const_str, sizeof(const_str) - 1)
+#ifdef HAVE_RUBY_ENCODING_H
+/* Use default external encoding for strings for Ruby 1.9+,
+ * fall back to rb_str_new when unavailable */
+#define rb_str_new rb_external_str_new
+#endif
+
#endif /* ext_help_h */
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..0a105e0 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -179,4 +179,17 @@ class RequestTest < Test::Unit::TestCase
env['rack.input'].rewind
res = @lint.call(env)
end
+
+ def test_encoding
+ if ''.respond_to?(:encoding)
+ client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
+ "Host: foo\r\n\r\n")
+ env = @request.read(client)
+ encoding = Encoding.default_external
+ assert_equal encoding, env['REQUEST_PATH'].encoding
+ assert_equal encoding, env['PATH_INFO'].encoding
+ assert_equal encoding, env['QUERY_STRING'].encoding
+ end
+ end
+
end
--
1.9.1
Gary
^ permalink raw reply related [relevance 4%]
* Re: Please move to github
2014-08-01 23:26 6% ` Daniel Evans
@ 2014-08-01 23:38 6% ` Michael Grosser
0 siblings, 0 replies; 73+ results
From: Michael Grosser @ 2014-08-01 23:38 UTC (permalink / raw)
To: Daniel Evans
Cc: Xavier Noria, Gabe da Silveira, Eric Wong,
unicorn-public@bogomips.org
Let me quickly tell the other 8.9M projects on GH that they are doing
it wrong ;)
I think having different ways of interacting with every project you
want to work on makes OS harder,
also contributing on PRs is much easier when using some collaboration
tool, viewing the source is easier and patches are simpler to read.
It also encourages nice docs and other things newbies need (does not
really matter for some backend thing like unicorn).
Sure crazy snowflakes like linux kernel might need something
different, but most other giant and small projects do just fine.
This discussion is going nowhere, so let's just stop now ;)
On Fri, Aug 1, 2014 at 4:26 PM, Daniel Evans <evans.daniel.n@gmail.com> wrote:
> This brings back memories of the BitKeeper debacle with the Linux kernel.
> One of the reasons git exists is because a large open source project trusted
> a proprietary product and it got ripped out from underneath them.
>
> On Fri, Aug 1, 2014 at 5:12 PM, Michael Grosser <michael@grosser.it> wrote:
>>
>> Patch coming soon, already pinpointed it, just wanted to look at the
>> issues to see if someone already solved it when I noticed that it's
>> not on github.
>>
>> But yeah otherwise, use whatever you like, chances are you do the most
>> work here anyway ;)
>>
>> As far as I am concerned any other OS/self-hosted tool like gitlab etc
>> would also be an improvement.
>>
>> On Fri, Aug 1, 2014 at 4:09 PM, Xavier Noria <fxn@hashref.com> wrote:
>> > Guys, Eric has obviously made a conscious choice by not hosting the source
>> > code on GitHub, and his rationale is clear.
>> >
>> > Eric I for one respect and understand your point of view, and think you have
>> > the right to do whatever you want with your projects. Thanks a lot for your
>> > open source and your integrity.
>> >
>>
>
>
>
> --
> Daniel Evans
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 23:09 6% ` Xavier Noria
2014-08-01 23:12 6% ` Michael Grosser
@ 2014-08-01 23:18 6% ` Aaron Suggs
1 sibling, 0 replies; 73+ results
From: Aaron Suggs @ 2014-08-01 23:18 UTC (permalink / raw)
To: Xavier Noria
Cc: Gabe da Silveira, Michael Grosser, Eric Wong,
unicorn-public@bogomips.org
> On Aug 1, 2014, at 7:09 PM, Xavier Noria <fxn@hashref.com> wrote:
>
> Guys, Eric has obviously made a conscious choice by not hosting the source
> code on GitHub, and his rationale is clear.
>
> Eric I for one respect and understand your point of view, and think you
> have the right to do whatever you want with your projects. Thanks a lot for
> your open source and your integrity.
Well said, Xavier. Thank you, Eric.
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 23:12 6% ` Michael Grosser
2014-08-01 23:24 6% ` Eric Wong
@ 2014-08-01 23:26 6% ` Daniel Evans
2014-08-01 23:38 6% ` Michael Grosser
1 sibling, 1 reply; 73+ results
From: Daniel Evans @ 2014-08-01 23:26 UTC (permalink / raw)
To: Michael Grosser
Cc: Xavier Noria, Gabe da Silveira, Eric Wong,
unicorn-public@bogomips.org
This brings back memories of the BitKeeper debacle with the Linux kernel.
One of the reasons git exists is because a large open source project trusted
a proprietary product and it got ripped out from underneath them.
On Fri, Aug 1, 2014 at 5:12 PM, Michael Grosser <michael@grosser.it> wrote:
>
> Patch coming soon, already pinpointed it, just wanted to look at the
> issues to see if someone already solved it when I noticed that it's
> not on github.
>
> But yeah otherwise, use whatever you like, chances are you do the most
> work here anyway ;)
>
> As far as I am concerned any other OS/self-hosted tool like gitlab etc
> would also be an improvement.
>
> On Fri, Aug 1, 2014 at 4:09 PM, Xavier Noria <fxn@hashref.com> wrote:
> > Guys, Eric has obviously made a conscious choice by not hosting the source
> > code on GitHub, and his rationale is clear.
> >
> > Eric I for one respect and understand your point of view, and think you have
> > the right to do whatever you want with your projects. Thanks a lot for your
> > open source and your integrity.
> >
>
--
Daniel Evans
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 23:12 6% ` Michael Grosser
@ 2014-08-01 23:24 6% ` Eric Wong
2014-08-01 23:26 6% ` Daniel Evans
1 sibling, 0 replies; 73+ results
From: Eric Wong @ 2014-08-01 23:24 UTC (permalink / raw)
To: Michael Grosser
Cc: Xavier Noria, Gabe da Silveira, Eric Wong,
unicorn-public@bogomips.org
Michael Grosser <michael@grosser.it> wrote:
> Patch coming soon, already pinpointed it, just wanted to look at the
> issues to see if someone already solved it when I noticed that it's
> not on github.
Thanks, I look forward to it.
> As far as I am concerned any other OS/self-hosted tool like gitlab etc
> would also be an improvement.
I do not like using web browsers or any sort of login/registration.
I made that clear when I first announced unicorn on the mongrel
list many years ago[1]. The only reason I can sustain working on
a project is if I don't have to deal with that stuff.
_Anybody_ can send plain-text email to this list, and it'll be archived
and viewable forever.
[1] http://mid.gmane.org/20090211230457.GB22926@dcvr.yhbt.net
(it turns out I can provide support the project, but only
as long as I stay inside my plain-text mail client).
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 23:09 6% ` Xavier Noria
@ 2014-08-01 23:12 6% ` Michael Grosser
2014-08-01 23:24 6% ` Eric Wong
2014-08-01 23:26 6% ` Daniel Evans
2014-08-01 23:18 6% ` Aaron Suggs
1 sibling, 2 replies; 73+ results
From: Michael Grosser @ 2014-08-01 23:12 UTC (permalink / raw)
To: Xavier Noria; +Cc: Gabe da Silveira, Eric Wong, unicorn-public@bogomips.org
Patch coming soon, already pinpointed it, just wanted to look at the
issues to see if someone already solved it when I noticed that it's
not on github.
But yeah otherwise, use whatever you like, chances are you do the most
work here anyway ;)
As far as I am concerned any other OS/self-hosted tool like gitlab etc
would also be an improvement.
On Fri, Aug 1, 2014 at 4:09 PM, Xavier Noria <fxn@hashref.com> wrote:
> Guys, Eric has obviously made a conscious choice by not hosting the source
> code on GitHub, and his rationale is clear.
>
> Eric I for one respect and understand your point of view, and think you have
> the right to do whatever you want with your projects. Thanks a lot for your
> open source and your integrity.
>
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 22:48 6% ` Gabe da Silveira
@ 2014-08-01 23:09 6% ` Xavier Noria
2014-08-01 23:12 6% ` Michael Grosser
2014-08-01 23:18 6% ` Aaron Suggs
0 siblings, 2 replies; 73+ results
From: Xavier Noria @ 2014-08-01 23:09 UTC (permalink / raw)
To: Gabe da Silveira; +Cc: Michael Grosser, Eric Wong, unicorn-public@bogomips.org
Guys, Eric has obviously made a conscious choice by not hosting the source
code on GitHub, and his rationale is clear.
Eric I for one respect and understand your point of view, and think you
have the right to do whatever you want with your projects. Thanks a lot for
your open source and your integrity.
^ permalink raw reply [relevance 6%]
* Re: Please move to github
[not found] ` <CAAms34NByMcXnbGQ3DvCZTsQuh6yBn8sMSNeTi7Ss4VmmYoOrQ@mail.gmail.com>
@ 2014-08-01 23:07 6% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2014-08-01 23:07 UTC (permalink / raw)
To: Michael Grosser; +Cc: unicorn-public
(please don't drop Cc-s, we do not require subscription)
Michael Grosser <michael@grosser.it> wrote:
> Most open-source projects need some kind of work done even if they are
> stable (atm we are trying to fix some encoding issue)
Please describe the issues you're experiencing, then.
> Easier access would mean more PRs vs people just monkey-patching it
> and not caring about giving back.
The better option is to encourage people to use non-proprietary
communications platforms.
As long as contributions are technically valid, I'll accept patches/pull
requests from anyone, anywhere. I do not even require people use real
names.
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 22:27 6% ` Michael Grosser
2014-08-01 22:41 6% ` Eric Wong
@ 2014-08-01 22:48 6% ` Gabe da Silveira
2014-08-01 23:09 6% ` Xavier Noria
1 sibling, 1 reply; 73+ results
From: Gabe da Silveira @ 2014-08-01 22:48 UTC (permalink / raw)
To: Michael Grosser; +Cc: Eric Wong, unicorn-public
I am a huge fan of GitHub and use it daily in my professional job, but both
the original message and now this comes off full of youthful arrogance and
hubris (in addition to abysmal grammar). GitHub is not the only way to do
open-source, and developers who refuse to use something other than GitHub
tend to be people impressed by superficial polish, but lacking
understanding in the type of deep engineering that allows for things like
the Linux kernel (or preforking app servers like Unicorn) to work. There
is more to software engineering than what is fashionable in the last 5
years. Please be more respectful of the maintainers of serious software
like Unicorn than to piss on their choice of project management tools and
techniques.
On Fri, Aug 1, 2014 at 5:27 PM, Michael Grosser <michael@grosser.it> wrote:
> Using github would mean more contributors, better software and
> especially helping more people (compare patches/issues/communication
> on unicorn to any other os project and github and the difference
> should be apparent),
> and I think that's what OS is about ... it's not ideal, but it's the
> best we got.
>
> On Fri, Aug 1, 2014 at 2:32 PM, Eric Wong <e@80x24.org> wrote:
> > Michael Grosser <michael@grosser.it> wrote:
> >> Current state makes it very hard to mange/search/fork/open-issues etc
> >> especially for newcomers,
> >> please move the project to github so we can have nice disussions
> >> forks/prs etc goodness.
> >
> > No. Never. Github is proprietary communications tool which requires
> > users to accept a terms of service and login. That gives power and
> > influence to a single entity (and a for-profit organization at that).
> >
> > Contributing to unicorn is *socially* as easy as contributing to git or
> > the Linux kernel. There is no need to signup for anything, no need to
> > ever touch a bloated web browser.
> >
> > The reason I contribute to Free Software is because I am against any
> > sort of lock-in or proprietary features. It absolutely sickens me to
> > encounter users who seem to be incapable of using git without a
> > proprietary communications tool.
>
>
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 22:27 6% ` Michael Grosser
@ 2014-08-01 22:41 6% ` Eric Wong
[not found] ` <CAAms34NByMcXnbGQ3DvCZTsQuh6yBn8sMSNeTi7Ss4VmmYoOrQ@mail.gmail.com>
2014-08-01 22:48 6% ` Gabe da Silveira
1 sibling, 1 reply; 73+ results
From: Eric Wong @ 2014-08-01 22:41 UTC (permalink / raw)
To: Michael Grosser; +Cc: unicorn-public
Michael Grosser <michael@grosser.it> wrote:
> Using github would mean more contributors, better software and
> especially helping more people (compare patches/issues/communication
> on unicorn to any other os project and github and the difference
> should be apparent),
> and I think that's what OS is about ... it's not ideal, but it's the
> best we got.
The problem is the Ruby community accepting non-Free solutions and
letting them become a standard. We need to fix that, but I'm not sure
how.
Also, does a pre-forking server based on ancient technology need much
work done on it?
^ permalink raw reply [relevance 6%]
* Re: Please move to github
@ 2014-08-01 22:32 6% Victor Kmita
0 siblings, 0 replies; 73+ results
From: Victor Kmita @ 2014-08-01 22:32 UTC (permalink / raw)
To: e; +Cc: unicorn-public, michael
"No. Never.=94 =93It absolutely sickens me to encounter users who seem =
to be incapable of using git without a proprietary communications tool."
Dude. You sicken me.=
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 21:32 6% ` Eric Wong
@ 2014-08-01 22:27 6% ` Michael Grosser
2014-08-01 22:41 6% ` Eric Wong
2014-08-01 22:48 6% ` Gabe da Silveira
0 siblings, 2 replies; 73+ results
From: Michael Grosser @ 2014-08-01 22:27 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Using github would mean more contributors, better software and
especially helping more people (compare patches/issues/communication
on unicorn to any other os project and github and the difference
should be apparent),
and I think that's what OS is about ... it's not ideal, but it's the
best we got.
On Fri, Aug 1, 2014 at 2:32 PM, Eric Wong <e@80x24.org> wrote:
> Michael Grosser <michael@grosser.it> wrote:
>> Current state makes it very hard to mange/search/fork/open-issues etc
>> especially for newcomers,
>> please move the project to github so we can have nice disussions
>> forks/prs etc goodness.
>
> No. Never. Github is proprietary communications tool which requires
> users to accept a terms of service and login. That gives power and
> influence to a single entity (and a for-profit organization at that).
>
> Contributing to unicorn is *socially* as easy as contributing to git or
> the Linux kernel. There is no need to signup for anything, no need to
> ever touch a bloated web browser.
>
> The reason I contribute to Free Software is because I am against any
> sort of lock-in or proprietary features. It absolutely sickens me to
> encounter users who seem to be incapable of using git without a
> proprietary communications tool.
^ permalink raw reply [relevance 6%]
* Re: Please move to github
2014-08-01 20:27 13% Please move to github Michael Grosser
@ 2014-08-01 21:32 6% ` Eric Wong
2014-08-01 22:27 6% ` Michael Grosser
0 siblings, 1 reply; 73+ results
From: Eric Wong @ 2014-08-01 21:32 UTC (permalink / raw)
To: Michael Grosser; +Cc: unicorn-public
Michael Grosser <michael@grosser.it> wrote:
> Current state makes it very hard to mange/search/fork/open-issues etc
> especially for newcomers,
> please move the project to github so we can have nice disussions
> forks/prs etc goodness.
No. Never. Github is proprietary communications tool which requires
users to accept a terms of service and login. That gives power and
influence to a single entity (and a for-profit organization at that).
Contributing to unicorn is *socially* as easy as contributing to git or
the Linux kernel. There is no need to signup for anything, no need to
ever touch a bloated web browser.
The reason I contribute to Free Software is because I am against any
sort of lock-in or proprietary features. It absolutely sickens me to
encounter users who seem to be incapable of using git without a
proprietary communications tool.
^ permalink raw reply [relevance 6%]
* Please move to github
@ 2014-08-01 20:27 13% Michael Grosser
2014-08-01 21:32 6% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Michael Grosser @ 2014-08-01 20:27 UTC (permalink / raw)
To: unicorn-public
Current state makes it very hard to mange/search/fork/open-issues etc
especially for newcomers,
please move the project to github so we can have nice disussions
forks/prs etc goodness.
^ permalink raw reply [relevance 13%]
* Re: Unicorn/Rainbows! mailing list migration
[not found] ` <20091029224426.GA12314-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
@ 2009-10-30 22:17 6% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2009-10-30 22:17 UTC (permalink / raw)
To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw,
rainbows-talk-GrnCvJ7WPxnNLxjTenLetw
Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> As I'm sure you've all heard by now, RubyForge will be moving to a
> read-only state and we'll have to migrate our mailing lists somewhere.
OK, it looks like we'll be able to keep our mailing lists on RubyForge.
http://tomcopeland.blogs.com/juniordeveloper/2009/10/things-to-keep-from-rubyforge.html
On the other hand, if anybody just *wants* to move to librelist, let us
know and we can consider it together, too.
Thanks.
--
Eric Wong
^ permalink raw reply [relevance 6%]
* unicorn mailing list moving
@ 2013-12-17 1:56 6% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2013-12-17 1:56 UTC (permalink / raw)
To: mongrel-unicorn
Heads up, some things will change regarding this mailing list.
Rubyforge has not been doing well lately and it looks like Rubyforge is
going away. This means the mailing list needs to move (and likely
evolve into something a little more distributed).
What will change:
* the email address (duh)
* hopefully better control of spam filtering/training
What won't change:
* no signups, nor list subscription requirement.
drive-by contributors will always be welcome
... more later :>
Note: I will never encourage nor promote the use of any non-Free Software.
So don't worry about that[1] :)
[1] hasbeghangryl, gb zbfg, guvf nyfb zrnaf: qba'g trg lbhe ubcrf hc :>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: Handling closed clients
[not found] ` <m21u2sjpc9.fsf@macdaddy.atl.damballa>
@ 2013-11-07 16:48 6% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2013-11-07 16:48 UTC (permalink / raw)
To: Andrew Hobson; +Cc: mongrel-unicorn
Andrew Hobson <ahobson@gmail.com> wrote:
> Eric Wong <normalperson@yhbt.net> writes:
>
> > (Please don't cull Cc:, I'm assuming you're not subscribed to the
> > mailing list since we don't require subscriptions)
>
> Sorry, that was unintentional.
No worries, and it is good to also send a copy to each recipient in the
thread in case Rubyforge is down (like it is right now). If it stays
down, I'll have to find/make a replacement myself.
...And move to something more decentralized and resilient to downtime
while I'm at it.
> > With my proposed patch to eliminate IO#close from StreamInput,
> > this test is no longer an accurate representation of unicorn behavior.
>
> I applied that one line patch a day and a half ago and we haven't seen
> the error in the field (yet). I am optimistic you have elegantly fixed
> the problem.
>
> If we do see an error, I will send another email to the list.
>
> Thanks again for your help,
No problem, I'll push that out later today.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: Forking non web processes
@ 2013-10-24 16:25 6% ` Alex Sharp
0 siblings, 0 replies; 73+ results
From: Alex Sharp @ 2013-10-24 16:25 UTC (permalink / raw)
To: unicorn list
On Thu, Oct 24, 2013 at 9:17 AM, Eric Wong <normalperson@yhbt.net> wrote:
> I'm also wondering why... sidekiq/resque are standalone daemons
> themselves. Shouldn't that be done as part of the deploy/init process?
> (unicorn isn't going to become init/upstart/systemd)
Agree with Eric here. You probably want to run unicorn and sidekiq /
resque in a way that they're not coupled to one another. They should
have different startup scripts and monitoring properties. And
eventually you may want to move your background worker processes to
another machine.
- alex sharp
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: [PATCH] preload_app can take an optional block for warmup
@ 2013-09-23 10:58 6% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2013-09-23 10:58 UTC (permalink / raw)
To: Aman Gupta; +Cc: unicorn list
Aman Gupta <aman@tmm1.net> wrote:
> > In particular, what benefit does this have over putting the same
> > code in config.ru or config/initializer.rb (or similar?)
>
> With my patch, preload_app yields a rack app object which includes the
> middleware stack. AFAIK there's no way to do this in the context of
> config.ru, since the app is still being built. My goal is to warm up
> the final application that will be serving traffic, including all
> middleware.
Good point. Perhaps this can be triggered via Rack::Builder#to_app
instead, so it can benefit _all_ Rack HTTP servers.
Care to move this discussion over to rack-devel?
There's the odd non-Rack::Builder case, too, but I think everybody just
uses Rack::Builder via config.ru, right?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: Unicorn_rails ignores USR2 signal
@ 2013-05-08 18:15 4% ` Ryan Jones (RyanonRails)
0 siblings, 0 replies; 73+ results
From: Ryan Jones (RyanonRails) @ 2013-05-08 18:15 UTC (permalink / raw)
To: mongrel-unicorn
We've run into the exact same problem. We spent a few good solid hours trying to
fix the problem with no avail (but we do have a 'solution').
Server specs:
Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-27-generic x86_64)
Rails & Ruby:
Rails 4.0.0.beta1
Ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]
Unicorn 4.6.2 - preload_app true
Capistrano
The first thing we tried was removing all of the gems (foreman, sidekiq, etc.)
and that didn't work. We tried preload_app false, and that worked (but this is
not the situation we want). We also tried searching for any gems that were
capturing USR2 (or any signals in general).
We tried monkey patching Kernel & Trap to see if we could pinpoint the problem,
but that didn't seem to turn up much. We then jumped in unicorn's http_server.rb
and threw in some logging. The only thing that seemed to show us was that USR2
was unable to get through to unicorn. Signals HUP, QUIT, etc. all worked fine
(and made their way through as per norm).
We then fired up strace with unicorn to see if we could 'see' the USR2 signal.
It looks almost identical to Jeffery's strace. It looks like the USR2 signal is
received by the process, but then dropped (or ignored). If we pass it a QUIT
signal it works just fine and we can see that in the strace.
So here's the chain of events that we think is occuring:
1. We start unicorn (a unicorn blessing)
2. before_fork is triggered (Signal trapping works here)
3. A unicorn is born (Signal trapping does not work here)
4. The after fork is triggered
In our situation. Once a unicorn is born, it can no longer received a USR2 (for
whatever reason). We never actually checked to see if the after fork received a
USR2.
After we managed to figure out the path of execution, we figured out a quick
fix. We actually call reexec method directly on the unicorn server in the
before_fork block in unicorn.rb. Here's our unicorn.rb
https://gist.github.com/RyanonRails/5541902
----- snip -----
before_fork do |server, worker|
Signal.trap 'USR2' do
puts 'Since a black hole is lurking and eating USR2s we will hit
http_server#reexec ourselves'
server.send(:reexec)
end
end
----- snip -----
This way we can actually force the USR2 directly on the server (since USR2 is
available inside of the before_fork).
This seems to be working well for us now. We're still frustrated that we weren't
able to find the cause of the problem, but we least we can move forward (with
minor disruption to the code/code base).
Lastly, here are our current theories as to why this could be occuring:
1. When unicorn is building it's native extensions something weird is happening
in regards to the USR2 signal
2. A gem is somehow gobbling up the signal (we were quite thorough, but we
might've missed something)
Hopefully this helps!
Thanks,
Ryan & Mike
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 4%]
* Re: Why doesn't SIGTERM quit gracefully?
@ 2013-04-25 11:02 5% ` Andreas Falk
0 siblings, 0 replies; 73+ results
From: Andreas Falk @ 2013-04-25 11:02 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn list
On Thu, Apr 25, 2013 at 10:51 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Andreas Falk <mail@andreasfalk.se> wrote:
>> I'm wondering why SIGINT and SIGTERM both were chosen for the quick
>> shutdown? I agree with SIGINT but not with SIGTERM. A lot of unix
>> tools send SIGTERM as default (kill, runit among some) and it seems to
>> be the standard way of telling a process to quit gracefully but not
>> among Ruby people (there are a few other ruby processes behaving the
>> same way). I just think it's weird that the default command will exit
>> without taking care of their current request.
>>
>> Also i'm not on the mailinglist so it would be great if you could cc
>> mail@andreasfalk.se
>
> I think it's weird, too. But that's what nginx does, and I based most
> of the UI decisions on nginx (so it's easy to reuse nginx scripts
> with unicorn).
Is it something you'd be willing to change? The developers behind
resque reasoned in a similar way in regards to nginx but changed it
after a while. You can read more about it here
<https://github.com/resque/resque/issues/368> and in the referenced
issues.
The change i'd like to see is to preferably have SIGTERM and SIGQUIT
swap places but at least move SIGTERM to do the same thing as SIGQUIT
do now (graceful exit).
I may be wrong but i feel that the change shouldn't completely break
anything (since it would still exit, just take a bit longer) and
switching some signals around in the nginx scripts shouldn't be that
much work? Also some people are probably using SIGINT already and
wouldn't be affected. I think the benefit in the long run of being in
line with the "standard" outweighs the hassle of converting a few
scripts.
Also perhaps with some luck the nginx developers will pick on the
changes in other software and switch theirs around too!
Thanks for a really great tool anyway!
Andreas
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 5%]
* Re: Unicorn hangs on POST request
2013-03-11 22:20 5% ` Tom Pesman
@ 2013-03-11 23:01 0% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2013-03-11 23:01 UTC (permalink / raw)
To: unicorn list
Tom Pesman <tom@tnux.net> wrote:
> > Tom Pesman <tom@tnux.net> wrote:
> >> Eric Wong wrote:
> >> > Tom Pesman <tom@tnux.net> wrote:
> >> >> I'm trying to fix a problem I'm experiencing with my Rails
> >> >> application hosted at Heroku. I've one POST request which hangs and
> >> >> with the help of a customized rack-timeout gem
> >> >> (https://github.com/tompesman/rack-timeout) I managed to get a
> >> >> stacktrace: https://gist.github.com/tompesman/7b13e02d349aacc720e0
> >> >>
> >> >> How can I debug this further to get to the bottom of this and is
> >> >> this a rack or a unicorn problem?
> >> >
> >> > It's a client or proxy problem.
> >> >
> >> > The request was too large to be transferred within your configured
> >> > timeout, or the client or proxy layer was too slow at transferring the
> >> > POST to unicorn, or the host running unicorn was too overloaded/slow
> >> > to buffer the request.
> >> >
> >> > Factors:
> >> > 1) Disk/filesystem/memory speed on the (client|proxy) talking to
> >> unicorn
> >> > 2) Disk/filesystem/memory speed on the host running unicorn.
> >> > 3) The network link between the (client|proxy) <-> unicorn.
> >> >
> >> > I don't know about Heroku, but nginx will fully buffer the request
> >> body
> >> > before sending to unicorn, so all 3 factors are within your control.
> >> >
> >> > Does Heroku limit (or allow limiting of) the size of request bodies?
> >> >
> >> > Maybe a bad client sent a gigantic request. nginx limits request
> >> bodies
> >> > to 1M by default (client_max_body_size config directive).
> >> >
> >> > [1] unicorn buffers request bodies to TMPDIR via TeeInput
> >> >
> >>
> >> I agree with you if the POST request has a file to upload, but the
> >> requests we're dealing with do not upload a file and are actually quite
> >> small.
> >
> > Do you have error logs from the proxy Heroku uses?
>
> The only output Heroku proxy (router called at Heroku) gives is something
> like this:
> 2013-03-11 15:55:58+00:00 heroku router - - at=info method=POST
> path=/api/v1/games/2257907/move.json host=aaa.bbb.ccc
> fwd="xxx.xxx.xxx.xxx/NX" dyno=web.1 queue=0 wait=8ms connect=9ms
> service=5019ms status=500 bytes=0
>
> > Even with small requests, clients/networks can fail to send the entire
> > request. nginx will log prematurely aborted client requests; check
> > if whatever proxy Heroku uses does the same.
>
> I think it's hard to debug with Heroku as the router doesn't give more
> output than this. I'm in contact with Heroku and I'll try to debug this
> with them. For me I have 2 problems with this issue, the waiting affects
> other requests and it's happening frequently (say multiple times per
> minute at 2k-4k rpm). How can I reduce the impact while not aborting valid
> requests?
The best way is to use nginx.
Otherwise, lowering your rack-timeout is probably the best way
More below...
> >> Can I modify the my customized rack-timeout gem to get more information
> >> to
> >> debug this problem?
> >> https://github.com/tompesman/rack-timeout/blob/master/lib/rack/timeout.rb
> >
> > Your env.inspect should show you @bytes_read in the Unicorn::TeeInput
> > object before the timeout was hit.
>
> This is the output of env.inspect:
> https://gist.github.com/tompesman/1217d4f02aa33efcd873
>
> It shows a CONTENT_LENGTH of 1351 and the "rack.input" =>
> Unicorn::TeeInput -> @bytes_read == 0.
>
> So what is happening here?
>
> 1. Heroku router/proxy sends the header to unicorn.
> 2. Unicorn receives the request and gives the request to Rack
> 3. Rack receives the request and asks Unicorn for the contents of the request
> 4. Unicorn should give the content to Rack. Now it gets interesting, how
> can I see the raw contents of the request? Or are we certain that it isn't
> there because @bytes_read == 0?
The request body doesn't seem to be there, presumably because Heroku
isn't sending it.
Doe heroku fully buffer the request body before sending it to unicorn?
nginx fully buffers, and this is why I can only recommend nginx for slow
clients.
The proxy -> unicorn transfer speed should _never_ be dependent by the
client -> proxy transfer speed.
1) client ------------------ (slow) --------------> proxy
2) proxy (nginx) --- (as fast as the connection goes) ---> unicorn
With nginx, 1) and 2) are decoupled and serialized. This adds latency,
but is the only way for multiprocess servers like unicorn to efficiently
handle slow clients.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: Unicorn hangs on POST request
@ 2013-03-11 22:20 5% ` Tom Pesman
2013-03-11 23:01 0% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Tom Pesman @ 2013-03-11 22:20 UTC (permalink / raw)
To: unicorn list
> Tom Pesman <tom@tnux.net> wrote:
>> Eric Wong wrote:
>> > Tom Pesman <tom@tnux.net> wrote:
>> >> I'm trying to fix a problem I'm experiencing with my Rails
>> >> application hosted at Heroku. I've one POST request which hangs and
>> >> with the help of a customized rack-timeout gem
>> >> (https://github.com/tompesman/rack-timeout) I managed to get a
>> >> stacktrace: https://gist.github.com/tompesman/7b13e02d349aacc720e0
>> >>
>> >> How can I debug this further to get to the bottom of this and is
>> >> this a rack or a unicorn problem?
>> >
>> > It's a client or proxy problem.
>> >
>> > The request was too large to be transferred within your configured
>> > timeout, or the client or proxy layer was too slow at transferring the
>> > POST to unicorn, or the host running unicorn was too overloaded/slow
>> > to buffer the request.
>> >
>> > Factors:
>> > 1) Disk/filesystem/memory speed on the (client|proxy) talking to
>> unicorn
>> > 2) Disk/filesystem/memory speed on the host running unicorn.
>> > 3) The network link between the (client|proxy) <-> unicorn.
>> >
>> > I don't know about Heroku, but nginx will fully buffer the request
>> body
>> > before sending to unicorn, so all 3 factors are within your control.
>> >
>> > Does Heroku limit (or allow limiting of) the size of request bodies?
>> >
>> > Maybe a bad client sent a gigantic request. nginx limits request
>> bodies
>> > to 1M by default (client_max_body_size config directive).
>> >
>> > [1] unicorn buffers request bodies to TMPDIR via TeeInput
>> >
>>
>> I agree with you if the POST request has a file to upload, but the
>> requests we're dealing with do not upload a file and are actually quite
>> small.
>
> Do you have error logs from the proxy Heroku uses?
The only output Heroku proxy (router called at Heroku) gives is something
like this:
2013-03-11 15:55:58+00:00 heroku router - - at=info method=POST
path=/api/v1/games/2257907/move.json host=aaa.bbb.ccc
fwd="xxx.xxx.xxx.xxx/NX" dyno=web.1 queue=0 wait=8ms connect=9ms
service=5019ms status=500 bytes=0
> Even with small requests, clients/networks can fail to send the entire
> request. nginx will log prematurely aborted client requests; check
> if whatever proxy Heroku uses does the same.
I think it's hard to debug with Heroku as the router doesn't give more
output than this. I'm in contact with Heroku and I'll try to debug this
with them. For me I have 2 problems with this issue, the waiting affects
other requests and it's happening frequently (say multiple times per
minute at 2k-4k rpm). How can I reduce the impact while not aborting valid
requests?
>> Can I modify the my customized rack-timeout gem to get more information
>> to
>> debug this problem?
>> https://github.com/tompesman/rack-timeout/blob/master/lib/rack/timeout.rb
>
> Your env.inspect should show you @bytes_read in the Unicorn::TeeInput
> object before the timeout was hit.
This is the output of env.inspect:
https://gist.github.com/tompesman/1217d4f02aa33efcd873
It shows a CONTENT_LENGTH of 1351 and the "rack.input" =>
Unicorn::TeeInput -> @bytes_read == 0.
So what is happening here?
1. Heroku router/proxy sends the header to unicorn.
2. Unicorn receives the request and gives the request to Rack
3. Rack receives the request and asks Unicorn for the contents of the request
4. Unicorn should give the content to Rack. Now it gets interesting, how
can I see the raw contents of the request? Or are we certain that it isn't
there because @bytes_read == 0?
Thanks!
--
Tom Pesman
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 5%]
* [PATCH] auto-generate Unicorn::Const::UNICORN_VERSION
@ 2013-02-08 18:55 4% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2013-02-08 18:55 UTC (permalink / raw)
To: unicorn list; +Cc: Maurizio De Santis
Maurizio De Santis <m.desantis@morganspa.com> wrote:
> Hello,
>
> I want to report that unicorn-4.6.0/lib/unicorn/const.rb declares
> UNICORN_VERSION = "4.5.0", which I think should be "4.6.0".
Oops, I've been meaning to move that constant over to an auto-generated file
using GIT-VERSION-GEN. This should work:
--------------------------------- 8< ------------------------------
>From cb0623f25db7f06660e563e8e746bfe0ae5ba9c5 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Fri, 8 Feb 2013 18:50:07 +0000
Subject: [PATCH] auto-generate Unicorn::Const::UNICORN_VERSION
This DRYs out our code and prevents snafus like the 4.6.0
release where UNICORN_VERSION stayed at 4.5.0
Reported-by: Maurizio De Santis <m.desantis@morganspa.com>
---
.gitignore | 1 +
GIT-VERSION-GEN | 69 ++++++++++++++++++++++++++--------------------------
GNUmakefile | 2 +-
lib/unicorn/const.rb | 4 +--
4 files changed, 37 insertions(+), 39 deletions(-)
diff --git a/.gitignore b/.gitignore
index 50c2736..19a82d6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,3 +22,4 @@ pkg/
/man
/tmp
/LATEST
+/lib/unicorn/version.rb
diff --git a/GIT-VERSION-GEN b/GIT-VERSION-GEN
index 56aef5f..e5d414a 100755
--- a/GIT-VERSION-GEN
+++ b/GIT-VERSION-GEN
@@ -1,40 +1,39 @@
-#!/bin/sh
-
-GVF=GIT-VERSION-FILE
-DEF_VER=v4.6.0
-
-LF='
-'
+#!/usr/bin/env ruby
+DEF_VER = "v4.6.0"
+CONSTANT = "Unicorn::Const::UNICORN_VERSION"
+RVF = "lib/unicorn/version.rb"
+GVF = "GIT-VERSION-FILE"
+vn = DEF_VER
# First see if there is a version file (included in release tarballs),
# then try git-describe, then default.
-if test -f version
-then
- VN=$(cat version) || VN="$DEF_VER"
-elif test -d .git -o -f .git &&
- VN=$(git describe --abbrev=4 HEAD 2>/dev/null) &&
- case "$VN" in
- *$LF*) (exit 1) ;;
- v[0-9]*)
- git update-index -q --refresh
- test -z "$(git diff-index --name-only HEAD --)" ||
- VN="$VN-dirty" ;;
- esac
-then
- VN=$(echo "$VN" | sed -e 's/-/./g');
-else
- VN="$DEF_VER"
-fi
+if File.exist?(".git")
+ describe = `git describe --abbrev=4 HEAD 2>/dev/null`.strip
+ case describe
+ when /\Av[0-9]*/
+ vn = describe
+ system(*%w(git update-index -q --refresh))
+ unless `git diff-index --name-only HEAD --`.chomp.empty?
+ vn << "-dirty"
+ end
+ vn.tr!('-', '.')
+ end
+end
+
+vn = vn.sub!(/\Av/, "")
+
+# generate the Ruby constant
+new_ruby_version = "#{CONSTANT} = '#{vn}'\n"
+cur_ruby_version = File.read(RVF) rescue nil
+if new_ruby_version != cur_ruby_version
+ File.open(RVF, "w") { |fp| fp.write(new_ruby_version) }
+end
-VN=$(expr "$VN" : v*'\(.*\)')
+# generate the makefile snippet
+new_make_version = "GIT_VERSION = #{vn}\n"
+cur_make_version = File.read(GVF) rescue nil
+if new_make_version != cur_make_version
+ File.open(GVF, "w") { |fp| fp.write(new_make_version) }
+end
-if test -r $GVF
-then
- VC=$(sed -e 's/^GIT_VERSION = //' <$GVF)
-else
- VC=unset
-fi
-test "$VN" = "$VC" || {
- echo >&2 "GIT_VERSION = $VN"
- echo "GIT_VERSION = $VN" >$GVF
-}
+puts vn if $0 == __FILE__
diff --git a/GNUmakefile b/GNUmakefile
index ea13486..34a2d95 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -155,7 +155,7 @@ clean:
man html:
$(MAKE) -C Documentation install-$@
-pkg_extra := GIT-VERSION-FILE ChangeLog LATEST NEWS \
+pkg_extra := GIT-VERSION-FILE lib/unicorn/version.rb ChangeLog LATEST NEWS \
$(ext)/unicorn_http.c $(man1_paths)
ChangeLog: GIT-VERSION-FILE .wrongdoc.yml
diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index fcc30c0..51d7394 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -7,9 +7,6 @@
# improvement over using the strings directly. Symbols did not really
# improve things much compared to constants.
module Unicorn::Const
-
- UNICORN_VERSION = "4.5.0"
-
# default TCP listen host address (0.0.0.0, all interfaces)
DEFAULT_HOST = "0.0.0.0"
@@ -44,3 +41,4 @@ module Unicorn::Const
# :startdoc:
end
+require 'unicorn/version'
--
1.8.1.2.526.gf51a757
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 4%]
* Re: Maintaining capacity during deploys
2012-11-30 1:28 5% ` Devin Ben-Hur
@ 2012-11-30 1:40 0% ` Tony Arcieri
0 siblings, 0 replies; 73+ results
From: Tony Arcieri @ 2012-11-30 1:40 UTC (permalink / raw)
To: unicorn list
On Thu, Nov 29, 2012 at 5:28 PM, Devin Ben-Hur <dbenhur@whitepages.com> wrote:
> A better solution is to use a profiler to identify what extra work is being
> done when an unwarm worker gets its first request and move that work into an
> initialization step which occurs before fork when run with app preload
> enabled.
I've done that, unfortunately that work is connection setup which must
happen after forking or otherwise file descriptors would wind up
shared between processes.
--
Tony Arcieri
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: Maintaining capacity during deploys
@ 2012-11-30 1:28 5% ` Devin Ben-Hur
2012-11-30 1:40 0% ` Tony Arcieri
0 siblings, 1 reply; 73+ results
From: Devin Ben-Hur @ 2012-11-30 1:28 UTC (permalink / raw)
To: unicorn list
On 11/29/12 3:34 PM, Lawrence Pit wrote:
>> Unfortunately, while the new workers are forking and begin processing
>> requests, we're still seeing significant spikes in our haproxy request
>> queue. It seems as if after we restart, the unwarmed workers get
>> swamped by the incoming requests.
>
> Perhaps it's possible to warm up the workers in the unicorn after_fork block?
I've successfully applied this methodology to a nasty rails app that had
a lot of latent initialization upon first request. Each worker gets a
unique private secondary listen port and each worker sends a warm-up
request to a prior worker in the after_fork hook. In our environment our
load balancer drains each host as it's being deployed, and this does
effect the length of deployment across many hosts in a cluster, but the
warmup bucket brigade is effective at making sure workers on that host
are responsive when they get added back to the available pool.
A better solution is to use a profiler to identify what extra work is
being done when an unwarm worker gets its first request and move that
work into an initialization step which occurs before fork when run with
app preload enabled.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 5%]
* Re: When a client terminates a connection
2012-11-28 13:32 5% When a client terminates a connection Andrew Stewart
@ 2012-11-28 21:10 0% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2012-11-28 21:10 UTC (permalink / raw)
To: unicorn list
Andrew Stewart <boss@airbladesoftware.com> wrote:
> Hello,
>
> I have run into the following situation several times in the past few months:
>
> (My stack is Nginx -- Unicorn -- Rails 3.0.12)
>
> 1. Client clicks a delete link in my webapp.
> 2. Rails starts processing the appropriate destroy action. This updates various things in the database (but not in a single db transaction).
> 3. Client terminates the connection.
> 4. Rails stops executing the destroy action wherever it has got to, leaving the latter part of the action unexecuted and my db inconsistent.
> 5. Rails and Unicorn don't write anything to their logs; Nginx writes a 499.
>
> This question has been asked before[1] but regrettably I don't
> entirely understand all the solutions proposed :)
>
> I plan to wrap all the changes made by my destroy action in a single
> db transaction. I'm assuming this will ensure db consistency, even
> when subject to awkward clients. Is that correct?
Yes (assuming all your DB actions in the transaction may be
rolled-back). I'm no expert in DBs, but I recommend you always
use transactions when data consistency is required.
> (I'll probably also move the action's work onto a background process,
> e.g. with delayed job.)
This also works, too, and is likely better if your DB actions take
a long time.
> Now apologies if I'm barking up the wrong tree here...in general is it
> possible to configure Unicorn to buffer the request and:
>
> - if the client terminates before the request is fully received, don't
> pass it to Rails at all;
> - else pass the complete request to Rails and let Rails execute it
> fully, even if client terminates in the meantime?
Yes. Using the Unicorn::PrereadInput middleware (before Rails or any
other middleware/framework touches env["rack.input"]) should do
everything you want.
> The thread[1] also mentioned Nginx's proxy_ignore_client_abort. If
> that's relevant, has anyone tried it?
I haven't, perhaps Jesse can respond?
> [1]
> http://rubyforge.org/pipermail/mongrel-unicorn/2011-August/001096.html
On a related note, Tom Burns has/had the same problem[2]. Unlike you,
he wanted to avoid processing the request in as many cases as possible.
The solution he came up with isn't 100% foolproof, either, but his
primary goal was to reduce load on an overloaded system.
I haven't heard back on results of our nasty/crazy solution, though.
If you do combine Tom's solution with yours, you still need to ensure
DB consistency in the app.
[2] http://mid.gmane.org/CAK4qKG32dGgNbzTzCb6NoH5hu_DrHMOFaFhk-6Xvy-T86++ukA@mail.gmail.com
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* When a client terminates a connection
@ 2012-11-28 13:32 5% Andrew Stewart
2012-11-28 21:10 0% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Andrew Stewart @ 2012-11-28 13:32 UTC (permalink / raw)
To: unicorn list
Hello,
I have run into the following situation several times in the past few months:
(My stack is Nginx -- Unicorn -- Rails 3.0.12)
1. Client clicks a delete link in my webapp.
2. Rails starts processing the appropriate destroy action. This updates various things in the database (but not in a single db transaction).
3. Client terminates the connection.
4. Rails stops executing the destroy action wherever it has got to, leaving the latter part of the action unexecuted and my db inconsistent.
5. Rails and Unicorn don't write anything to their logs; Nginx writes a 499.
This question has been asked before[1] but regrettably I don't entirely understand all the solutions proposed :)
I plan to wrap all the changes made by my destroy action in a single db transaction. I'm assuming this will ensure db consistency, even when subject to awkward clients. Is that correct?
(I'll probably also move the action's work onto a background process, e.g. with delayed job.)
Now apologies if I'm barking up the wrong tree here...in general is it possible to configure Unicorn to buffer the request and:
- if the client terminates before the request is fully received, don't pass it to Rails at all;
- else pass the complete request to Rails and let Rails execute it fully, even if client terminates in the meantime?
The thread[1] also mentioned Nginx's proxy_ignore_client_abort. If that's relevant, has anyone tried it?
[1] http://rubyforge.org/pipermail/mongrel-unicorn/2011-August/001096.html
Many thanks,
Andrew Stewart
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 5%]
* Re: pid file deleted briefly when doing hot restart
2012-11-26 23:56 7% ` Petteri Räty
@ 2012-11-27 0:35 0% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2012-11-27 0:35 UTC (permalink / raw)
To: unicorn list
Petteri Räty <betelgeuse@gentoo.org> wrote:
> On 26.11.2012 20.24, Eric Wong wrote:
> >
> >> The use case here is that with health monitors wouldn't have a window
> >> where a pid file does not exist. With a hot restart it should always be
> >> possible to have a pid file that points to either the old or the new master.
> >
> > Then, doesn't nginx have the same problem?
> >
>
> nginx doens't have the same problem as I show later. Even if it I
> restart/reload nginx very infrequently. Unicorn has to be hot restarted
> every time we change our application code and happens frequently.
>
> >>> I think unicorn differs a bit from nginx here:
> >>>
> >>> nginx uses rename() to clear the way for a new pid file. Like unicorn,
> >>> this still leaves a window where no pid file exists.
> >>>
> >>
> >> Looking at the inotify log it seems the reason pid file does not exist
> >> is an explicit delete and not due to rename. It happens a couple seconds
> >> earlier also so the window is possible to hit even with a periodic poller.
> >
> > Is matching nginx rename behavior enough to solve the problem?
> >
> > Matching nginx behavior can become the default if it solves your problem.
> >
>
> nginx does not explicitly unlink the old pid file before it renames it
> out of the way so yes matching nginx in that regard changes the behavior
> exactly how I originally asked but you were against that. Maybe the
> point is moot though.
Ah, I thought you wanted the pid file to be replaced without
a window where the file is non-existent (or empty).
> This is from a combo of USR2, WINCH, QUIT sent from htop to the master
> process:
>
> 2012-11-27 01:44:25 +0200
> [:moved_from, :move]
> "nginx.pid"
> 2012-11-27 01:44:25 +0200
> [:moved_to, :move]
> "nginx.pid.oldbin"
> 2012-11-27 01:44:25 +0200
OK, this is the window where file does not exist at all.
unicorn has the same problem here, but obviously unicorn is slower than
nginx.
> [:create]
> "nginx.pid"
> 2012-11-27 01:44:25 +0200
> [:open]
> "nginx.pid"
This part of nginx makes me uncomfortable since nginx.pid is empty.
> 2012-11-27 01:44:25 +0200
> [:modify]
> "nginx.pid"
> 2012-11-27 01:44:25 +0200
> [:close_write, :close]
> "nginx.pid"
> 2012-11-27 01:45:31 +0200
> [:delete]
> "nginx.pid.oldbin"
>
> The window here is much smaller than for the current unicorn behavior.
A small window is still a window.
Can't you make your health monitor check the state of the listening
ports as well? There's no point where a listening port will be
unavailable.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: pid file deleted briefly when doing hot restart
@ 2012-11-26 23:56 7% ` Petteri Räty
2012-11-27 0:35 0% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Petteri Räty @ 2012-11-26 23:56 UTC (permalink / raw)
To: mongrel-unicorn
On 26.11.2012 20.24, Eric Wong wrote:
>
>> The use case here is that with health monitors wouldn't have a window
>> where a pid file does not exist. With a hot restart it should always be
>> possible to have a pid file that points to either the old or the new master.
>
> Then, doesn't nginx have the same problem?
>
nginx doens't have the same problem as I show later. Even if it I
restart/reload nginx very infrequently. Unicorn has to be hot restarted
every time we change our application code and happens frequently.
>>> I think unicorn differs a bit from nginx here:
>>>
>>> nginx uses rename() to clear the way for a new pid file. Like unicorn,
>>> this still leaves a window where no pid file exists.
>>>
>>
>> Looking at the inotify log it seems the reason pid file does not exist
>> is an explicit delete and not due to rename. It happens a couple seconds
>> earlier also so the window is possible to hit even with a periodic poller.
>
> Is matching nginx rename behavior enough to solve the problem?
>
> Matching nginx behavior can become the default if it solves your problem.
>
nginx does not explicitly unlink the old pid file before it renames it
out of the way so yes matching nginx in that regard changes the behavior
exactly how I originally asked but you were against that. Maybe the
point is moot though.
This is from a combo of USR2, WINCH, QUIT sent from htop to the master
process:
2012-11-27 01:44:25 +0200
[:moved_from, :move]
"nginx.pid"
2012-11-27 01:44:25 +0200
[:moved_to, :move]
"nginx.pid.oldbin"
2012-11-27 01:44:25 +0200
[:create]
"nginx.pid"
2012-11-27 01:44:25 +0200
[:open]
"nginx.pid"
2012-11-27 01:44:25 +0200
[:modify]
"nginx.pid"
2012-11-27 01:44:25 +0200
[:close_write, :close]
"nginx.pid"
2012-11-27 01:45:31 +0200
[:delete]
"nginx.pid.oldbin"
The window here is much smaller than for the current unicorn behavior.
Regards,
Petteri
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 7%]
* pid file deleted briefly when doing hot restart
@ 2012-11-25 23:27 9% Petteri Räty
0 siblings, 1 reply; 73+ results
From: Petteri Räty @ 2012-11-25 23:27 UTC (permalink / raw)
To: mongrel-unicorn
What follows are all the write actions related to unicorn pid file when
doing a hot restart. Seems like a bug to me that unicorn is deleting the
pid file before writing the new file. Is there a reason for it? It seems
to go against that rename that aims for an atomic replace that would
always ensure the pid file is there.
Regards,
Petteri
2012-11-26 00:54:12 +0200
[:delete]
"unicorn_blue1.pid"
2012-11-26 00:54:12 +0200
[:create]
"0.18100578716567672.15108"
2012-11-26 00:54:12 +0200
[:open]
"0.18100578716567672.15108"
2012-11-26 00:54:12 +0200
[:modify]
"0.18100578716567672.15108"
2012-11-26 00:54:12 +0200
[:moved_from, :move]
"0.18100578716567672.15108"
2012-11-26 00:54:12 +0200
[:moved_to, :move]
"unicorn_blue1.pid.oldbin"
2012-11-26 00:54:12 +0200
[:close_write, :close]
"unicorn_blue1.pid.oldbin"
2012-11-26 00:54:14 +0200
[:create]
"0.9762316483892712.16822"
2012-11-26 00:54:14 +0200
[:open]
"0.9762316483892712.16822"
2012-11-26 00:54:14 +0200
[:modify]
"0.9762316483892712.16822"
2012-11-26 00:54:14 +0200
[:moved_from, :move]
"0.9762316483892712.16822"
2012-11-26 00:54:14 +0200
[:moved_to, :move]
"unicorn_blue1.pid"
2012-11-26 00:54:14 +0200
[:close_write, :close]
"unicorn_blue1.pid"
2012-11-26 00:54:14 +0200
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 9%]
* Re: Is a client uploading a file a slow client from unicorn's point of view?
@ 2012-10-09 1:58 5% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2012-10-09 1:58 UTC (permalink / raw)
To: unicorn list
Jimmy Soho <jimmy.soho@gmail.com> wrote:
> Hi All,
>
> I was wondering what would happen when large files were uploaded to
> our system in parallel to endpoints that don't process file uploads.
> In particular I was wondering if we're vulnerable to a simple DoS
> attack.
nginx will protect you by buffering large requests to disk, so slow
requests are taken care of (of course you may still run out of disk
space)
> The setup I tested with was nginx v1.2.4 with upload module (v2.2.0)
> configured only for location /uploads with 2 unicorn (v4.3.1) workers
> with timeout 30 secs, all running on 1 small unix box.
>
> In a few terminals I started this command 3 times in parallel:
>
> $ curl -i -F importer_input=@/Users/admin/largefile.tar.gz
> https://mywebserver.com/doesnotexist
>
> In a browser I then tried to go a page that would be served by a unicorn worker.
>
> My expectation was that I would not get to see the web page as all
> unicorn workers would be busy with receiving / saving the upload. As
> discussed in for example this article:
> http://stackoverflow.com/questions/9592664/unicorn-rails-large-uploads.
> Or as https://github.com/dwilkie/carrierwave_direct describes it:
>
> "Processing and saving file uploads are typically long running tasks
> and should be done in a background process."
That is true. It's good to move slow jobs to background processes if
possible if the bottleneck is either:
a) your application processing
b) the storage destination of your app (e.g. cloud storage)
However, if your only bottleneck is client <-> your app, then nginx
will take care of that part for you.
> But I don't see this. The page is served just fine in my setup. The
> requests for the file uploads appear in the nginx access log at the
> same time the curl upload command eventually finishes minutes later
> client side, and then it's handed off to a unicorn/rack worker
> process, which quickly returns a 404 page not found. Response times of
> less than 50ms.
>
> What am I missing here? I'm starting to wonder what's the use of the
> nginx upload module? My understanding was that it's use was to keep
> unicorn workers available as long as a file upload was in progress,
> but it seems that without that module it does the same thing.
I'm not familiar with the nginx upload module, but stock nginx will
already do full request buffering for you. It looks like the nginx
upload module[1] is mostly meant for standalone apps written for
nginx, and not when nginx is used as a proxy for Rails app...
[1] http://www.grid.net.ru/nginx/upload.en.html
> Another question (more an nginx question though I guess): is there a
> way to kill an upload request as early as possible if the request is
> not made against known / accepted URI locations, instead of waiting
> for it to be completely uploaded to our system and/or waiting for it
> to reach the unicorn workers?
I'm not sure if nginx has this functionality, but unicorn lazily buffers
uploads. So your upload will be fully read by nginx, but unicorn
will only read the uploaded request body if your application wants to
read it.
Unfortunately, I think most application frameworks (e.g. Rails) will
attempt to do all the multipart parsing up front. To get around this,
you'll probably want some middleware along the following lines (and
placed in front of whichever part of your stack calls
Rack::Multipart.parse_multipart)
class BadUploadStopper
def initialize(app)
@app = app
end
def call(env)
case env["REQUEST_METHOD"]
when "POST", "PUT"
case env["PATH_INFO"]
when "/upload_allowed"
@app.call(env) # forward to the app
else
# bad path, don't waste time with @app.call
[ 403, {}, [ "Go away\n" ] ]
end
else
@app.call(env) # forward to the app
end
end
end
------------------- config.ru ---------------------
use BadUploadStopper
run YourApp.new
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 5%]
* Re: Sinatra app fails to perform at scale
2012-06-15 18:09 4% Sinatra app fails to perform at scale Eliot Shepard
@ 2012-06-15 21:22 0% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2012-06-15 21:22 UTC (permalink / raw)
To: unicorn list; +Cc: Eliot Shepard
Eliot Shepard <eshepard@slower.net> wrote:
> Hello,
>
> We're attempting to move our set of Rails and Sinatra apps from
> Passenger/REE to Unicorn/1.9.3. We've started with one of the Sinatra
> apps, which typically sees about 50 req/s. It happens to talk to both
> Redis (Resque) and MongoDB GridFS.
Are you having problems at 50 req/s or higher than that with ab?
> The basic setup works fine when tested under ab, but we're having
> trouble getting the deploy into production. It performs fine for a
> bit, then the nginx write queue fills up and begins returning 502s.
Is ab hitting nginx or unicorn? You'll probably get more
accurate/consistent results using ab with keepalive (-k) and hitting
nginx with ab.
> A colleague has posted a more detailed description of the issue and our
> setup on ServerFault:
> http://serverfault.com/questions/398972/need-to-increase-nginx-throughput-to-an-upstream-unix-socket-linux-kernel-tun
Fwiw, I don't (and don't intend to) monitor external websites for
questions, especially when they require signups. Feel free to post
my response in part or full (or link back to this ML archive).
I'll quote the relevant parts
<snip> nginx config looks fine.
The Unicorn config would be helpful here, too.
Can you try setting the listen:backlog directive to a higher number?
Something like:
listen "/tmp/app.sock", :backlog => 8192
You'll want to stay within your net.core.somaxconn = 8192 value or raise
that sysctl to match unicorn. However, if you have multiple machines
and want to load balance, I recommend trying a lower backlog.
> The issue is, it just seems that past a certain amount of load, nginx
> can't get requests through the socket at a fast enough rate. It doesn't
> matter how many app server processes I set up, it doesn't even matter
> what the app is (tried it with a dummy app with just a single endpoint
> that returned an empty page with status 404). The bottleneck seems to
> be the socket, not the app.
> I'm getting a flood of these messages in the nginx error log:
> connect() to unix:/tmp/app.sock failed (11: Resource temporarily
> unavailable) wh ile connecting to upstream
> Many requests result in status code 502, and those that don't take a
> long time to complete. The nginx write queue stat hovers around 1000.
Can you also verify unicorn is correctly closing connections that nginx
opens to it? unicorn 4.3.1 should be fine in this regard...
> Additional information on the environment
> curbed@app1:~$ uname -a
> Linux app1 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC
> 2012 x86_64 x86_64 x86_64 GNU/Linux
> curbed@app1:~$ ruby -v
> ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
> curbed@app1:~$ unicorn -v
> unicorn v4.3.1
> curbed@app1:~$ nginx -V
> nginx version: nginx/1.2.1
I haven't tested with nginx 1.2.x, yet, but I expect it to be fine.
I know 1.2.x also supports keepalive to backend connections, but
I don't expect it to benefit on a fast LAN or local Unix socket...
> Any suggestions on configuration, kernel tuning, etc. would be
> welcomed (here or on SF). Please CC me if you answer through the list.
> Thanks for your time.
Thanks for reminding us to Cc: :)
I think Tom had a similar question a few years ago
http://mid.gmane.org/20090918064831.GA5285@dcvr.yhbt.net
However, hitting issues with the default backlog (1024) with 50 req/s
wouldn't be expected...
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Sinatra app fails to perform at scale
@ 2012-06-15 18:09 4% Eliot Shepard
2012-06-15 21:22 0% ` Eric Wong
0 siblings, 1 reply; 73+ results
From: Eliot Shepard @ 2012-06-15 18:09 UTC (permalink / raw)
To: mongrel-unicorn
Hello,
We're attempting to move our set of Rails and Sinatra apps from
Passenger/REE to Unicorn/1.9.3. We've started with one of the Sinatra
apps, which typically sees about 50 req/s. It happens to talk to both
Redis (Resque) and MongoDB GridFS.
The basic setup works fine when tested under ab, but we're having
trouble getting the deploy into production. It performs fine for a
bit, then the nginx write queue fills up and begins returning 502s. A
colleague has posted a more detailed description of the issue and our
setup on ServerFault:
http://serverfault.com/questions/398972/need-to-increase-nginx-throughput-to-an-upstream-unix-socket-linux-kernel-tun
Additional information on the environment
curbed@app1:~$ uname -a
Linux app1 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
curbed@app1:~$ ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
curbed@app1:~$ unicorn -v
unicorn v4.3.1
curbed@app1:~$ nginx -V
nginx version: nginx/1.2.1
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_ssl_module
--with-cc-opt=-Wno-error --with-http_gzip_static_module
--with-http_stub_status_module
--add-module=/home/curbed/src/nginx-modules/nginx-gridfs
--add-module=/home/curbed/src/nginx-modules/ngx_http_redis-0.3.6
--add-module=/home/curbed/src/nginx-modules/headers-more-nginx-module
Kernel tweaks:
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.route.flush = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.somaxconn = 8192
net.netfilter.nf_conntrack_max = 131072
Any suggestions on configuration, kernel tuning, etc. would be
welcomed (here or on SF). Please CC me if you answer through the list.
Thanks for your time.
Eliot
--
Eliot Shepard
Head of Tech, Curbed Network
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 4%]
* Re: Unicorn and streaming in Rails 3.1
@ 2011-06-25 20:33 5% ` Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2011-06-25 20:33 UTC (permalink / raw)
To: unicorn list
Xavier Noria <fxn@hashref.com> wrote:
> If it is a real bad idea, is the recommendation
> to Unicorn users that they should just ignore this new feature?
Another thing, Rainbows! + ThreadSpawn/ThreadPool concurrency may do the
trick (without needing nginx at all). The per-client overhead of
Rainbows! + threads (several hundred KB) is higher than nginx, but still
much lower than Unicorn.
All your Rails code must be thread-safe, though.
If you use Linux, XEpollThreadPool/XEpollThreadSpawn can be worth a
try, too. The cost of completely idle keepalive clients should be
roughly inline with nginx.
If you want to forgo thread-safety, Rainbows! + StreamResponseEpoll[1]
+ ForceStreaming middleware[2] may also be an option, too (needs nginx).
Keep in mind that I don't know of anybody using Rainbows! for any
serious sites, so there could still be major bugs :)
Rainbows! http://rainbows.rubyforge.org/
[1] currently in rainbows.git, will probably be released this weekend...
[2] I'll probably move this to Rainbows! instead of a Unicorn branch:
http://bogomips.org/unicorn.git/tree/lib/unicorn/force_streaming.rb?h=force_streaming
This is 100% untested, I've never run it.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 5%]
* Re: Confused classes
2010-02-06 12:33 5% Confused classes Warren Konkel
@ 2010-02-06 13:06 0% ` Iñaki Baz Castillo
0 siblings, 0 replies; 73+ results
From: Iñaki Baz Castillo @ 2010-02-06 13:06 UTC (permalink / raw)
To: mongrel-unicorn
El Sábado, 6 de Febrero de 2010, Warren Konkel escribió:
> I switched from mod_passenger to unicorn on a fairly high traffic site
> and ran into a strange problem that forced me to move back to
> mod_passenger... it seems as if classes would sometimes get mixed up
> with each other. If I had two Rails models:
>
> class Foo < ActiveRecord::Base; end
> class Bar < ActiveRecord::Base; end
>
> Normally Foo.inspect and Bar.inspect would return:
>
> Foo(field1: integer, field2: integer)
> Bar(field3: integer, field4: integer)
>
> When things were "broken" within a process, sometimes I would see:
>
> Foo(field3: integer, field4: integer) <--- note field3/field4
> actually belong to Bar, not Foo
>
> And because of that, wacky errors would appear in my logs like:
>
> Foo.find_by_field1(12345) --> not a method
> Foo.create(:field1 => 12345) --> column not found
>
> I also noticed the problem with field serializing in ActiveRecord... given:
>
> class Boz < ActiveRecord::Base
> serialize :some_data
> end
>
> When processes were working correctly, Boz.find(1).some_data would
> return an actual object (like a Hash). When things were broken, the
> raw serialized string from the database would be returned... almost as
> if the Boz class "forgot" that it's supposed to deserialize
> "some_data".
>
> Could it be that class attributes are somehow being co-mingled when
> unicorn is starting up under high concurrency? Perhaps a mutex is
> missing somewhere?
IMHO all your Unicorn workers are sharing the same DB connection (the same
ActiveRecord instances) so the problem arises.
Take a look to the configuration here:
http://unicorn.bogomips.org/examples/unicorn.conf.rb
You can see there how the ActiveRecord is disconnected at the beggining and
started for each worker later.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Confused classes
@ 2010-02-06 12:33 5% Warren Konkel
2010-02-06 13:06 0% ` Iñaki Baz Castillo
0 siblings, 1 reply; 73+ results
From: Warren Konkel @ 2010-02-06 12:33 UTC (permalink / raw)
To: mongrel-unicorn
I switched from mod_passenger to unicorn on a fairly high traffic site
and ran into a strange problem that forced me to move back to
mod_passenger... it seems as if classes would sometimes get mixed up
with each other. If I had two Rails models:
class Foo < ActiveRecord::Base; end
class Bar < ActiveRecord::Base; end
Normally Foo.inspect and Bar.inspect would return:
Foo(field1: integer, field2: integer)
Bar(field3: integer, field4: integer)
When things were "broken" within a process, sometimes I would see:
Foo(field3: integer, field4: integer) <--- note field3/field4
actually belong to Bar, not Foo
And because of that, wacky errors would appear in my logs like:
Foo.find_by_field1(12345) --> not a method
Foo.create(:field1 => 12345) --> column not found
I also noticed the problem with field serializing in ActiveRecord... given:
class Boz < ActiveRecord::Base
serialize :some_data
end
When processes were working correctly, Boz.find(1).some_data would
return an actual object (like a Hash). When things were broken, the
raw serialized string from the database would be returned... almost as
if the Boz class "forgot" that it's supposed to deserialize
"some_data".
Could it be that class attributes are somehow being co-mingled when
unicorn is starting up under high concurrency? Perhaps a mutex is
missing somewhere?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 5%]
* draft release notes (so far) for upcoming 0.93.0
@ 2009-10-01 8:13 4% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2009-10-01 8:13 UTC (permalink / raw)
To: mongrel-unicorn
The one minor bugfix is only for users who set RAILS_RELATIVE_URL_ROOT
in a config file. Users of the "--path" switch or those who set the
environment variable in the shell were unaffected by this bug. Note
that we still don't have relative URL root support for Rails < 2.3, and
are unlikely to bother with it unless there is visible demand for it. I
didn't even know people used/cared for relative URL roots before today
(it was a Rails 2.3.4 user).
New features (so far) includes support for :tries and :delay when
specifying a "listen" in an after_fork hook. This was inspired by Chris
Wanstrath's example of binding per-worker listen sockets in a loop while
migrating (or upgrading) Unicorn. Setting a negative value for :tries
means we'll retry the listen indefinitely until the socket becomes
available.
So you can do something like this in an after_fork hook:
after_fork do |server,worker|
addr = "127.0.0.1:#{9293 + worker.nr}"
server.listen(addr, :tries => -1, :delay => 5)
end
There's also the usual round of added documentation, packaging fixes,
code cleanups and minor performance improvements that are viewable
in the git log....
Shortlog since v0.92.0 (so far) below:
Eric Wong (45):
build: hardcode the canonical git URL
build: manifest dropped manpages
build: smaller ChangeLog
doc/LATEST: remove trailing newline
http: don't force -fPIC if it can't be used
.gitignore on *.rbc files Rubinius generates
README/gemspec: a better description, hopefully
GNUmakefile: add missing .manifest dep on test installs
Add HACKING document
configurator: fix user switch example in RDoc
local.mk.sample: time and perms enforcement
unicorn_rails: show "RAILS_ENV" in help message
gemspec: compatibility with older Rubygems
Split out KNOWN_ISSUES document
KNOWN_ISSUES: add notes about the "isolate" gem
gemspec: fix test_files regexp match
gemspec: remove tests that fork from test_files
test_signals: ensure we can parse pids in response
GNUmakefile: cleanup test/manifest generation
util: remove APPEND_FLAGS constant
http_request: simplify and remove handle_body method
http_response: simplify and remove const dependencies
local.mk.sample: fix .js times
TUNING: notes about benchmarking a high :backlog
HttpServer#listen accepts :tries and :delay parameters
"make install" avoids installing multiple .so objects
Use Configurator#expand_addr in HttpServer#listen
configurator: move initialization stuff to #initialize
Remove "Z" constant for binary strings
cgi_wrapper: don't warn about stdoutput usage
cgi_wrapper: simplify status handling in response
cgi_wrapper: use Array#concat instead of +=
server: correctly unset reexec_pid on child death
configurator: update and modernize examples
configurator: add colons in front of listen() options
configurator: remove DEFAULT_LOGGER constant
gemspec: clarify commented-out licenses section
Add makefile targets for non-release installs
cleanup: use question mark op for 1-byte comparisons
RDoc for Unicorn::HttpServer::Worker
small cleanup to pid file handling + documentation
rails: RAILS_RELATIVE_URL_ROOT may be set in Unicorn config
unicorn_rails: undeprecate --path switch
manpages: document environment variables
README: remove reference to different versions
--
Eric Wong
^ permalink raw reply [relevance 4%]
* [ANN] unicorn 0.9.0 (experimental release)
@ 2009-07-01 22:58 4% Eric Wong
0 siblings, 0 replies; 73+ results
From: Eric Wong @ 2009-07-01 22:58 UTC (permalink / raw)
To: mongrel-unicorn, ruby-talk, rack-devel; +Cc: mongrel-development
Unicorn is a Rack HTTP server for Unix, fast clients and nothing else[1]
We now have support for "Transfer-Encoding: chunked" bodies in
requests. Not only that, Rack applications reading input bodies
get that data streamed off to the client socket on an as-needed
basis. This allows the application to do things like upload
progress notification and even tunneling of arbitrary stream
protocols via bidirectional chunked encoding.
See Unicorn::App::Inetd and examples/git.ru (including the
comments) for an example of tunneling the git:// protocol over
HTTP.
This release also gives applications the ability to respond
positively to "Expect: 100-continue" headers before being rerun
without closing the socket connection. See Unicorn::App::Inetd
for an example of how this is used.
This release is NOT recommended for production use.
Eric Wong (43):
http_request: no need to reset the request
http_request: StringIO is binary for empty bodies (1.9)
http_request: fix typo for 1.9
Transfer-Encoding: chunked streaming input support
Unicorn::App::Inetd: reinventing Unix, poorly :)
README: update with mailing list info
local.mk.sample: publish_doc gzips all html, js, css
Put copyright text in new files, include GPL2 text
examples/cat-chunk-proxy: link to proposed curl(1) patch
Update TODO
Avoid duplicating the "Z" constant
Optimize body-less GET/HEAD requests (again)
tee_input: Don't expose the @rd object as a return value
exec_cgi: small cleanups
README: another note about older Sinatra
tee_input: avoid defining a @rd.size method
Make TeeInput easier to use
test_upload: add tests for chunked encoding
GNUmakefile: more stringent error checking in tests
test_upload: fix ECONNRESET with 1.9
GNUmakefile: allow TRACER= to be specified for tests
test_rails: workaround long-standing 1.9 bug
tee_input: avoid rereading fresh data
"Fix" tests that break with stream_input=false
inetd: fix broken constant references
configurator: provide stream_input (true|false) option
chunked_reader: simpler interface
http_request: force BUFFER to be Encoding::BINARY
ACK clients on "Expect: 100-continue" header
Only send "100 Continue" when no body has been sent
http_request: tighter Transfer-Encoding: "chunked" check
Add trailer_parser for parsing trailers
chunked_reader: Add test for chunk parse failure
TeeInput: use only one IO for tempfile
trailer_parser: set keys with "HTTP_" prefix
TrailerParser integration into ChunkedReader
Unbind listeners as before stopping workers
Retry listen() on EADDRINUSE 5 times ever 500ms
Re-add support for non-portable socket options
Move "Expect: 100-continue" handling to the app
tee_input: avoid ignoring initial body blob
Force streaming input onto apps by default
unicorn 0.9.0
* site: http://unicorn.bogomips.org/
* git: git://git.bogomips.org/unicorn.git
* http+git: http://git.bogomips.org:8080/unicorn.git [1]
* cgit: http://git.bogomips.org/cgit/unicorn.git/
* list: mongrel-unicorn@rubyforge.org
[1] - Actually, most of the new features in this release should in fact
work on non-Unix OSes so they can be adapted for other servers with
wider audiences.
[2] - This is the git:// protocol tunneled inside a bidirectional
"Transfer-Encoding: chunked" request/response using this latest
release of Unicorn.
--
Eric Wong
^ permalink raw reply [relevance 4%]
Results 1-73 of 73 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2009-07-01 22:58 4% [ANN] unicorn 0.9.0 (experimental release) Eric Wong
2009-10-01 8:13 4% draft release notes (so far) for upcoming 0.93.0 Eric Wong
2009-10-29 22:44 Unicorn/Rainbows! mailing list migration Eric Wong
[not found] ` <20091029224426.GA12314-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2009-10-30 22:17 6% ` Eric Wong
2010-02-06 12:33 5% Confused classes Warren Konkel
2010-02-06 13:06 0% ` Iñaki Baz Castillo
2011-06-25 16:08 Unicorn and streaming in Rails 3.1 Xavier Noria
2011-06-25 20:33 5% ` Eric Wong
2012-03-09 21:48 Unicorn_rails ignores USR2 signal Yeung, Jeffrey
2012-03-09 22:24 ` Eric Wong
2012-03-09 22:39 ` Yeung, Jeffrey
2012-03-10 0:02 ` Eric Wong
2012-03-10 1:07 ` Yeung, Jeffrey
2012-03-10 1:30 ` Eric Wong
2012-03-12 21:21 ` Eric Wong
2012-03-12 22:39 ` Yeung, Jeffrey
2012-03-12 22:44 ` Eric Wong
2012-03-20 19:57 ` Eric Wong
2012-03-30 22:16 ` Yeung, Jeffrey
2012-03-30 22:51 ` Alex Sharp
2013-05-08 18:15 4% ` Ryan Jones (RyanonRails)
2012-06-15 18:09 4% Sinatra app fails to perform at scale Eliot Shepard
2012-06-15 21:22 0% ` Eric Wong
2012-10-09 0:39 Is a client uploading a file a slow client from unicorn's point of view? Jimmy Soho
2012-10-09 1:58 5% ` Eric Wong
2012-11-25 23:27 9% pid file deleted briefly when doing hot restart Petteri Räty
2012-11-26 0:43 ` Eric Wong
2012-11-26 10:39 ` Petteri Räty
2012-11-26 18:24 ` Eric Wong
2012-11-26 23:56 7% ` Petteri Räty
2012-11-27 0:35 0% ` Eric Wong
2012-11-28 13:32 5% When a client terminates a connection Andrew Stewart
2012-11-28 21:10 0% ` Eric Wong
[not found] <CAHOTMV++otgxdru_oZLXuVuqHF7F4uMwd04O0QZBjxeqFR-=XQ@mail.gmail.com>
2012-11-29 23:05 ` Fwd: Maintaining capacity during deploys Tony Arcieri
2012-11-29 23:34 ` Lawrence Pit
2012-11-30 1:28 5% ` Devin Ben-Hur
2012-11-30 1:40 0% ` Tony Arcieri
2013-02-08 13:54 Unicorn 4.6.0 version is 4.5.0 Maurizio De Santis
2013-02-08 18:55 4% ` [PATCH] auto-generate Unicorn::Const::UNICORN_VERSION Eric Wong
2013-03-09 16:02 Unicorn hangs on POST request Tom Pesman
2013-03-09 21:02 ` Eric Wong
2013-03-10 16:22 ` Tom Pesman
2013-03-11 19:49 ` Eric Wong
2013-03-11 22:20 5% ` Tom Pesman
2013-03-11 23:01 0% ` Eric Wong
2013-04-25 8:02 Why doesn't SIGTERM quit gracefully? Andreas Falk
2013-04-25 8:51 ` Eric Wong
2013-04-25 11:02 5% ` Andreas Falk
2013-09-20 21:40 [PATCH] preload_app can take an optional block for warmup Aman Gupta
2013-09-21 8:49 ` Eric Wong
2013-09-21 23:10 ` Aman Gupta
2013-09-23 10:58 6% ` Eric Wong
2013-10-24 11:08 Forking non web processes Sam Saffron
2013-10-24 16:17 ` Eric Wong
2013-10-24 16:25 6% ` Alex Sharp
2013-11-05 14:46 Handling closed clients Andrew Hobson
2013-11-05 17:20 ` Eric Wong
[not found] ` <m2iow6k7nk.fsf@macdaddy.atl.damballa>
2013-11-05 20:51 ` Eric Wong
[not found] ` <m21u2sjpc9.fsf@macdaddy.atl.damballa>
2013-11-07 16:48 6% ` Eric Wong
2013-12-17 1:56 6% unicorn mailing list moving Eric Wong
2014-08-01 20:27 13% Please move to github Michael Grosser
2014-08-01 21:32 6% ` Eric Wong
2014-08-01 22:27 6% ` Michael Grosser
2014-08-01 22:41 6% ` Eric Wong
[not found] ` <CAAms34NByMcXnbGQ3DvCZTsQuh6yBn8sMSNeTi7Ss4VmmYoOrQ@mail.gmail.com>
2014-08-01 23:07 6% ` Eric Wong
2014-08-01 22:48 6% ` Gabe da Silveira
2014-08-01 23:09 6% ` Xavier Noria
2014-08-01 23:12 6% ` Michael Grosser
2014-08-01 23:24 6% ` Eric Wong
2014-08-01 23:26 6% ` Daniel Evans
2014-08-01 23:38 6% ` Michael Grosser
2014-08-01 23:18 6% ` Aaron Suggs
2014-08-01 22:32 6% Victor Kmita
2014-08-02 7:46 4% Gary Grossman
2014-08-02 7:51 4% Gary Grossman
2014-08-02 7:54 6% ` Kapil Israni
2014-08-02 8:02 6% ` Eric Wong
2014-08-02 8:50 4% ` Eric Wong
2014-08-02 19:07 4% ` Gary Grossman
2014-08-02 19:33 6% ` Michael Fischer
2014-08-04 7:22 5% ` Hongli Lai
2014-08-04 8:48 6% ` Rack encodings (was: Please move to github) Eric Wong
2014-08-04 9:46 6% ` Hongli Lai
2014-08-02 20:15 6% ` Please move to github Eric Wong
2014-08-05 5:56 14% Rack encodings (was: Please move to github) Gary Grossman
2014-08-05 6:28 6% ` Eric Wong
2014-09-27 8:32 5% dropping Ruby 1.8 support for unicorn 5? Eric Wong
2014-09-27 8:37 0% ` Ernest W. Durbin III
2014-12-03 9:50 No, passenger 5.0 is not faster than unicorn :) Bráulio Bhavamitra
2014-12-03 9:56 6% ` Sam Saffron
2014-12-03 9:57 0% ` Sam Saffron
2015-06-06 1:58 7% [PATCH 0/2] eliminate generic ivars from HttpRequest class Eric Wong
2015-06-06 1:58 5% ` [PATCH 1/2] move the socket into Rack env for hijacking Eric Wong
2015-06-06 1:58 4% ` [PATCH 2/2] http: move response_start_sent into the C ext Eric Wong
2015-06-15 22:56 5% [ANN] unicorn 5.0.0.pre1 - incompatible changes! Eric Wong
2017-02-22 12:02 6% check_client_connection using getsockopt(2) Simon Eskildsen
2017-02-22 18:33 0% ` Eric Wong
2017-02-22 20:09 0% ` Simon Eskildsen
2017-02-25 14:03 [PATCH] check_client_connection: use tcp state on linux Simon Eskildsen
2017-02-25 16:19 ` Simon Eskildsen
2017-02-25 23:12 ` Eric Wong
2017-02-27 11:44 3% ` Simon Eskildsen
2017-02-28 21:12 0% ` Eric Wong
2017-03-21 18:44 Rack::Request#params EOFError John Smart
2017-03-21 19:06 5% ` Eric Wong
2017-03-26 20:52 0% ` Eric Wong
[not found] <redmine.issue-17023.20200710175402.5550@ruby-lang.org>
[not found] ` <redmine.journal-86563.20200715200039.5550@ruby-lang.org>
2020-07-15 23:35 ` [ruby-core:99184] [Ruby master Bug#17023] How to prevent String memory to be relocated in ruby-ffi Eric Wong
2020-07-15 23:49 7% ` [ruby-core:99185] " Aaron Patterson
2020-09-01 12:17 [PATCH] Update ruby_version requirement to allow ruby 3.0 Jean Boussier
2020-09-01 14:48 ` Eric Wong
2020-09-01 15:04 ` Jean Boussier
2020-09-01 15:41 6% ` Eric Wong
[not found] <F6712BF3-A4DD-41EE-8252-B9799B35E618@github.com>
[not found] ` <20210311030250.GA1266@dcvr>
[not found] ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
2021-03-12 9:41 0% ` Potential Unicorn vulnerability Eric Wong
2021-09-14 23:39 6% [PATCH 0/2] drop Ruby 1.9.3 support, require 2.0+ Eric Wong
2023-06-05 10:32 1% [PATCH 00-23/23] start porting tests to Perl5 Eric Wong
2023-09-10 20:08 2% [PATCH 00..11/11] more tests to Perl 5 Eric Wong
2024-05-06 20:10 3% [PATCH 0/5..5/5] more tests to Perl 5 for stability Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).