unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 0/4] a small pile of patches
@ 2024-03-23 19:45  3% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-03-23 19:45 UTC (permalink / raw)
  To: unicorn-public

[-- Attachment #1: Type: text/plain, Size: 4446 bytes --]

Some stuff to future-proof against future Ruby incompatibilities.
More coming....

I've also pushed out preliminary work (started in 2021) to the
`pico' branch to switch the HTTP parser from Ragel to
picohttpparser.  It will simplify the build + maintenance,
especially when distros carry different Ragel versions (or don't
package it all, as some hackers can't afford bandwidth and disk
for a C++ toolchain).

Other notes: New releases will probably be hosted on yhbt.net if
the Rubygems.org MFA threshold is reached.  Caring about the
identity of hackers is totally misguided when we already show
our code (and even document it!).  If you can't audit the code
yourself, get an actual professional to do it and don't bother
amateurs like me.

Eric Wong (4):
  t/integration: disable proxies when running curl(1)
  tests: port back-out-of-upgrade to Perl 5
  doc: various updates and disclaimers
  treewide: future-proof frozen_string_literal changes

 HACKING                             |  13 +++-
 README                              |   9 +++
 Rakefile                            |   1 +
 TODO                                |   4 +-
 bin/unicorn                         |   1 +
 bin/unicorn_rails                   |   1 +
 examples/big_app_gc.rb              |   1 +
 examples/echo.ru                    |   1 +
 examples/logger_mp_safe.rb          |   1 +
 examples/unicorn.conf.minimal.rb    |   1 +
 examples/unicorn.conf.rb            |   1 +
 ext/unicorn_http/extconf.rb         |   1 +
 lib/unicorn.rb                      |   1 +
 lib/unicorn/app/old_rails.rb        |   1 +
 lib/unicorn/app/old_rails/static.rb |   1 +
 lib/unicorn/cgi_wrapper.rb          |   1 +
 lib/unicorn/configurator.rb         |   1 +
 lib/unicorn/const.rb                |   1 +
 lib/unicorn/http_request.rb         |   1 +
 lib/unicorn/http_response.rb        |   1 +
 lib/unicorn/http_server.rb          |   1 +
 lib/unicorn/launcher.rb             |   1 +
 lib/unicorn/oob_gc.rb               |   1 +
 lib/unicorn/preread_input.rb        |   1 +
 lib/unicorn/select_waiter.rb        |   1 +
 lib/unicorn/socket_helper.rb        |   1 +
 lib/unicorn/stream_input.rb         |   1 +
 lib/unicorn/tee_input.rb            |   1 +
 lib/unicorn/tmpio.rb                |   1 +
 lib/unicorn/util.rb                 |   1 +
 lib/unicorn/worker.rb               |   1 +
 setup.rb                            |   1 +
 t/back-out-of-upgrade.t             |  44 +++++++++++
 t/broken-app.ru                     |   1 +
 t/client_body_buffer_size.ru        |   1 +
 t/detach.ru                         |   1 +
 t/env.ru                            |   1 +
 t/fails-rack-lint.ru                |   1 +
 t/heartbeat-timeout.ru              |   1 +
 t/integration.ru                    |   1 +
 t/integration.t                     |   1 +
 t/lib.perl                          |  67 ++++++++++++++---
 t/listener_names.ru                 |   1 +
 t/oob_gc.ru                         |   1 +
 t/oob_gc_path.ru                    |   1 +
 t/pid.ru                            |   1 +
 t/preread_input.ru                  |   1 +
 t/reopen-logs.ru                    |   1 +
 t/t0008-back_out_of_upgrade.sh      | 110 ----------------------------
 t/t0013.ru                          |   1 +
 t/t0014.ru                          |   1 +
 t/t0301.ru                          |   1 +
 test/aggregate.rb                   |   1 +
 test/benchmark/dd.ru                |   1 +
 test/benchmark/ddstream.ru          |   1 +
 test/benchmark/readinput.ru         |   1 +
 test/benchmark/stack.ru             |   1 +
 test/exec/test_exec.rb              |   1 +
 test/test_helper.rb                 |   1 +
 test/unit/test_ccc.rb               |   1 +
 test/unit/test_configurator.rb      |   1 +
 test/unit/test_droplet.rb           |   1 +
 test/unit/test_http_parser.rb       |   1 +
 test/unit/test_http_parser_ng.rb    |   1 +
 test/unit/test_request.rb           |   1 +
 test/unit/test_server.rb            |   1 +
 test/unit/test_signals.rb           |   1 +
 test/unit/test_socket_helper.rb     |   1 +
 test/unit/test_stream_input.rb      |   1 +
 test/unit/test_tee_input.rb         |   1 +
 test/unit/test_util.rb              |   1 +
 test/unit/test_waiter.rb            |   1 +
 unicorn.gemspec                     |   1 +
 73 files changed, 188 insertions(+), 126 deletions(-)
 create mode 100644 t/back-out-of-upgrade.t
 delete mode 100755 t/t0008-back_out_of_upgrade.sh

[-- Attachment #2: 0001-t-integration-disable-proxies-when-running-curl-1.patch --]
[-- Type: text/x-diff, Size: 737 bytes --]

From f3acce5dce62ac4b0288d3c0ddf0a6db2cbd9e7f Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Tue, 9 Jan 2024 21:35:08 +0000
Subject: [PATCH 1/4] t/integration: disable proxies when running curl(1)

This was also done in t/test-lib.sh, but using '*' is more
encompassing.
---
 t/integration.t | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/integration.t b/t/integration.t
index 7310ff2..d17ace0 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,6 +27,7 @@ listen "$u1"
 EOM
 my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
 my $curl = which('curl');
+local $ENV{NO_PROXY} = '*'; # for curl
 my $fifo = "$tmpdir/fifo";
 POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
 my %PUT = (

[-- Attachment #3: 0002-tests-port-back-out-of-upgrade-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8889 bytes --]

From 724fb631c76f09964ec289ee8e144886ba15d380 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 6 Nov 2023 05:45:29 +0000
Subject: [PATCH 2/4] tests: port back-out-of-upgrade to Perl 5

Another place where we can be faster without adding more
dependencies on Ruby maintaining stable behavior.
---
 t/back-out-of-upgrade.t        |  44 +++++++++++++
 t/lib.perl                     |  67 +++++++++++++++++---
 t/t0008-back_out_of_upgrade.sh | 110 ---------------------------------
 3 files changed, 102 insertions(+), 119 deletions(-)
 create mode 100644 t/back-out-of-upgrade.t
 delete mode 100755 t/t0008-back_out_of_upgrade.sh

diff --git a/t/back-out-of-upgrade.t b/t/back-out-of-upgrade.t
new file mode 100644
index 0000000..cf3b09f
--- /dev/null
+++ b/t/back-out-of-upgrade.t
@@ -0,0 +1,44 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# test backing out of USR2 upgrade
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+mkfifo_die $fifo;
+write_file '>', $u_conf, <<EOM;
+preload_app true
+stderr_path "$err_log"
+pid "$pid_file"
+after_fork { |s,w| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+my $ar = unicorn(qw(-E none t/pid.ru -c), $u_conf, { 3 => $srv });
+
+like(my $wpid_orig_1 = slurp($fifo), qr/\Apid=\d+\z/a, 'got worker pid');
+
+ok $ar->do_kill('USR2'), 'USR2 to start upgrade';
+ok $ar->do_kill('WINCH'), 'drop old worker';
+
+like(my $wpid_new = slurp($fifo), qr/\Apid=\d+\z/a, 'got pid from new master');
+chomp(my $new_pid = slurp($pid_file));
+isnt $new_pid, $ar->{pid}, 'PID file changed';
+chomp(my $pid_oldbin = slurp("$pid_file.oldbin"));
+is $pid_oldbin, $ar->{pid}, '.oldbin PID valid';
+
+ok $ar->do_kill('HUP'), 'HUP old master';
+like(my $wpid_orig_2 = slurp($fifo), qr/\Apid=\d+\z/a, 'got worker new pid');
+ok kill('QUIT', $new_pid), 'abort old master';
+kill_until_dead $new_pid;
+
+my ($st, $hdr, $req_pid) = do_req $srv, 'GET /';
+chomp $req_pid;
+is $wpid_orig_2, "pid=$req_pid", 'new worker on old worker serves';
+
+ok !-f "$pid_file.oldbin", '.oldbin PID file gone';
+chomp(my $old_pid = slurp($pid_file));
+is $old_pid, $ar->{pid}, 'PID file restored';
+
+my @log = grep !/ERROR -- : reaped .*? exec\(\)-ed/, slurp($err_log);
+check_stderr @log;
+undef $tmpdir;
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 9254b23..b20a2c6 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,30 +6,58 @@ use v5.14;
 use parent qw(Exporter);
 use autodie;
 use Test::More;
+use Socket qw(SOMAXCONN);
 use Time::HiRes qw(sleep time);
 use IO::Socket::INET;
+use IO::Socket::UNIX;
+use Carp qw(croak);
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
-	$pid_file);
+	$pid_file, $wtest_sock, $fifo);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
 	$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
+	$wtest_sock $fifo
 	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
-	do_req stop_daemon sleep time);
+	do_req stop_daemon sleep time mkfifo_die kill_until_dead write_file);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+
+$wtest_sock = "$tmpdir/wtest.sock";
 $err_log = "$tmpdir/err.log";
 $pid_file = "$tmpdir/pid";
+$fifo = "$tmpdir/fifo";
 $u_sock = "$tmpdir/u.sock";
 $u_conf = "$tmpdir/u.conf.rb";
 open($errfh, '>>', $err_log);
 
+if (my $t = $ENV{TAIL}) {
+	my @tail = $t =~ /tail/ ? split(/\s+/, $t) : (qw(tail -F));
+	push @tail, $err_log;
+	my $pid = fork;
+	if ($pid == 0) {
+		open STDOUT, '>&', \*STDERR;
+		exec @tail;
+		die "exec(@tail): $!";
+	}
+	say "# @tail";
+	sleep 0.2;
+	UnicornTest::AutoReap->new($pid);
+}
+
+sub kill_until_dead ($;%) {
+	my ($pid, %opt) = @_;
+	my $tries = $opt{tries} // 1000;
+	my $sig = $opt{sig} // 0;
+	while (CORE::kill($sig, $pid) && --$tries) { sleep(0.01) }
+	$tries or croak "PID: $pid died after signal ($sig)";
+}
+
 sub stop_daemon (;$) {
 	my ($is_END) = @_;
 	kill('TERM', $daemon_pid);
-	my $tries = 1000;
-	while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) }
+	kill_until_dead $daemon_pid;
 	if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing
 		CORE::kill('KILL', $daemon_pid);
 		die "daemon_pid=$daemon_pid did not die";
@@ -44,8 +72,9 @@ END {
 	stop_daemon(1) if defined $daemon_pid;
 };
 
-sub check_stderr () {
-	my @log = slurp($err_log);
+sub check_stderr (@) {
+	my @log = @_;
+	slurp($err_log) if !@log;
 	diag("@log") if $ENV{V};
 	my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
 	@err = grep(!/failed to set accept_filter=/, @err);
@@ -63,6 +92,16 @@ sub slurp_hdr {
 	($status, \@hdr);
 }
 
+sub unix_server (;$@) {
+	my $l = shift // $u_sock;
+	IO::Socket::UNIX->new(Listen => SOMAXCONN, Local => $l, Blocking => 0,
+				Type => SOCK_STREAM, @_);
+}
+
+sub unix_connect ($) {
+	IO::Socket::UNIX->new(Peer => $_[0], Type => SOCK_STREAM);
+}
+
 sub tcp_server {
 	my %opt = (
 		ReuseAddr => 1,
@@ -95,8 +134,7 @@ sub tcp_host_port {
 
 sub unix_start ($@) {
 	my ($dst, @req) = @_;
-	my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
-		BAIL_OUT "unix connect $dst: $!";
+	my $s = unix_connect($dst) or BAIL_OUT "unix connect $dst: $!";
 	$s->autoflush(1);
 	print $s @req, "\r\n\r\n" if @req;
 	$s;
@@ -201,7 +239,7 @@ sub unicorn {
 	state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
 	state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
 	state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
-	state $exe = File::Spec->rel2abs('bin/unicorn');
+	state $exe = File::Spec->rel2abs("test/$eng-$ver/bin/unicorn");
 	my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
 	UnicornTest::AutoReap->new($pid);
 }
@@ -219,6 +257,17 @@ sub do_req ($@) {
 	($status, $hdr, $bdy);
 }
 
+sub mkfifo_die ($;$) {
+	POSIX::mkfifo($_[0], $_[1] // 0600) or croak "mkfifo: $!";
+}
+
+sub write_file ($$@) { # mode, filename, LIST (for print)
+	open(my $fh, shift, shift);
+	print $fh @_;
+	# return $fh for futher writes if user wants it:
+	defined(wantarray) && !wantarray ? $fh : close $fh;
+}
+
 # automatically kill + reap children when this goes out-of-scope
 package UnicornTest::AutoReap;
 use v5.14;
diff --git a/t/t0008-back_out_of_upgrade.sh b/t/t0008-back_out_of_upgrade.sh
deleted file mode 100755
index 96d4057..0000000
--- a/t/t0008-back_out_of_upgrade.sh
+++ /dev/null
@@ -1,110 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 13 "backout of USR2 upgrade"
-
-worker_wait_start () {
-	test xSTART = x"$(cat $fifo)"
-	unicorn_pid=$(cat $pid)
-}
-
-t_begin "setup and start" && {
-	unicorn_setup
-	rm -f $pid.oldbin
-
-cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
-  # test script will block while reading from $fifo,
-  # so notify the script on the first worker we spawn
-  # by opening the FIFO
-  if worker.nr == 0
-    File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
-  end
-end
-EOF
-	unicorn -D -c $unicorn_config pid.ru
-	worker_wait_start
-	orig_master_pid=$unicorn_pid
-}
-
-t_begin "read original worker pid" && {
-	orig_worker_pid=$(curl -sSf http://$listen/)
-	test -n "$orig_worker_pid" && kill -0 $orig_worker_pid
-}
-
-t_begin "upgrade to new master" && {
-	kill -USR2 $orig_master_pid
-}
-
-t_begin "kill old worker" && {
-	kill -WINCH $orig_master_pid
-}
-
-t_begin "wait for new worker to start" && {
-	worker_wait_start
-	test $unicorn_pid -ne $orig_master_pid
-	new_master_pid=$unicorn_pid
-}
-
-t_begin "old master pid is stashed in $pid.oldbin" && {
-	test -s "$pid.oldbin"
-	test $orig_master_pid -eq $(cat $pid.oldbin)
-}
-
-t_begin "ensure old worker is no longer running" && {
-	i=0
-	while kill -0 $orig_worker_pid 2>/dev/null
-	do
-		i=$(( $i + 1 ))
-		test $i -lt 600 || die "timed out"
-		sleep 1
-	done
-}
-
-t_begin "capture pid of new worker" && {
-	new_worker_pid=$(curl -sSf http://$listen/)
-}
-
-t_begin "reload old master process" && {
-	kill -HUP $orig_master_pid
-	worker_wait_start
-}
-
-t_begin "gracefully kill new master and ensure it dies" && {
-	kill -QUIT $new_master_pid
-	i=0
-	while kill -0 $new_worker_pid 2>/dev/null
-	do
-		i=$(( $i + 1 ))
-		test $i -lt 600 || die "timed out"
-		sleep 1
-	done
-}
-
-t_begin "ensure $pid.oldbin does not exist" && {
-	i=0
-	while test -s $pid.oldbin
-	do
-		i=$(( $i + 1 ))
-		test $i -lt 600 || die "timed out"
-		sleep 1
-	done
-	while ! test -s $pid
-	do
-		i=$(( $i + 1 ))
-		test $i -lt 600 || die "timed out"
-		sleep 1
-	done
-}
-
-t_begin "ensure $pid is correct" && {
-	cur_master_pid=$(cat $pid)
-	test $orig_master_pid -eq $cur_master_pid
-}
-
-t_begin "killing succeeds" && {
-	kill $orig_master_pid
-}
-
-dbgcat r_err
-
-t_done

[-- Attachment #4: 0003-doc-various-updates-and-disclaimers.patch --]
[-- Type: text/x-diff, Size: 3343 bytes --]

From 69d15a7a51a096b6acf00ccf23e1b988076d3b5f Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Mon, 1 Jan 2024 10:43:13 +0000
Subject: [PATCH 3/4] doc: various updates and disclaimers

Covering my ass from draconian legislation.
---
 HACKING | 13 +++++++++----
 README  |  9 +++++++++
 TODO    |  4 +---
 3 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/HACKING b/HACKING
index 5aca83e..777e75e 100644
--- a/HACKING
+++ b/HACKING
@@ -6,6 +6,8 @@ Like Mongrel, we use Ruby where it makes sense, and Ragel with C where
 it helps performance.  All of the code that actually runs your Rack
 application is written Ruby, Ragel or C.
 
+Ragel may be dropped in favor of a picohttpparser-based one in the future.
+
 As far as tests and documentation goes, we're not afraid to embrace Unix
 and use traditional Unix tools where they make sense and get the job
 done.
@@ -16,6 +18,9 @@ Tests are good, but slow tests make development slow, so we make tests
 faster (in parallel) with GNU make (instead of Rake) and avoiding
 RubyGems.
 
+New tests are written in Perl 5 and use TAP <https://testanything.org/>
+to ensure stability and immunity from Ruby incompatibilities.
+
 Users of GNU-based systems (such as GNU/Linux) usually have GNU make
 installed as "make" instead of "gmake".
 
@@ -69,10 +74,10 @@ supported by the versions of Ruby we target.
 
 === Ragel Compatibility
 
-We target the latest released version of Ragel and will update our code
-to keep up with new releases.  Packaged tarballs and gems include the
-generated source code so they will remain usable if compatibility is
-broken.
+We target the latest released version of Ragel in Debian and will update
+our code to keep up with new releases.  Packaged tarballs and gems
+include the generated source code so they will remain usable if
+compatibility is broken.
 
 == Contributing
 
diff --git a/README b/README
index 84c0fdf..b60ed00 100644
--- a/README
+++ b/README
@@ -122,6 +122,7 @@ supported.  Run `unicorn -h` to see command-line options.
 
 There is NO WARRANTY whatsoever if anything goes wrong, but
 {let us know}[link:ISSUES.html] and maybe someone can fix it.
+No commercial support will ever be provided by the amateur maintainer.
 
 unicorn is designed to only serve fast clients either on the local host
 or a fast LAN.  See the PHILOSOPHY and DESIGN documents for more details
@@ -132,6 +133,14 @@ damage done to the entire Ruby ecosystem.  Its unintentional popularity
 set Ruby back decades in parallelism, concurrency and robustness since
 it prolongs and proliferates the existence of poorly-written code.
 
+unicorn hackers are NOT responsible for your supply chain security:
+read and understand it yourself or get someone you trust to audit it.
+Malicious commits and releases will be made if under duress.  The only
+defense you'll ever have is from reviewing the source code.
+
+No user or contributor will ever be expected to sacrifice their own
+security by running JavaScript or revealing any personal information.
+
 == Contact
 
 All feedback (bug reports, user/development dicussion, patches, pull
diff --git a/TODO b/TODO
index ebbccdc..a3b18fd 100644
--- a/TODO
+++ b/TODO
@@ -1,3 +1 @@
-* Documentation improvements
-
-* improve test suite
+* improve test suite (port to Perl 5 for stability and maintainability)

[-- Attachment #5: 0004-treewide-future-proof-frozen_string_literal-changes.patch --]
[-- Type: text/x-diff, Size: 24100 bytes --]

From ccf2443901c18ffb26b2785f52d921005e862167 Mon Sep 17 00:00:00 2001
From: Eric Wong <bofh@yhbt.net>
Date: Thu, 8 Feb 2024 12:16:31 +0000
Subject: [PATCH 4/4] treewide: future-proof frozen_string_literal changes

Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1].  In order to avoid
incompatibilities in the future, I used a Perl script[2] to
prepend `frozen_string_literal: false' to every Ruby file.

Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.

[1] https://bugs.ruby-lang.org/issues/20205
[2] https://yhbt.net/add-fsl.git/74d7689/s/?b=add-fsl.perl
---
 Rakefile                            | 1 +
 bin/unicorn                         | 1 +
 bin/unicorn_rails                   | 1 +
 examples/big_app_gc.rb              | 1 +
 examples/echo.ru                    | 1 +
 examples/logger_mp_safe.rb          | 1 +
 examples/unicorn.conf.minimal.rb    | 1 +
 examples/unicorn.conf.rb            | 1 +
 ext/unicorn_http/extconf.rb         | 1 +
 lib/unicorn.rb                      | 1 +
 lib/unicorn/app/old_rails.rb        | 1 +
 lib/unicorn/app/old_rails/static.rb | 1 +
 lib/unicorn/cgi_wrapper.rb          | 1 +
 lib/unicorn/configurator.rb         | 1 +
 lib/unicorn/const.rb                | 1 +
 lib/unicorn/http_request.rb         | 1 +
 lib/unicorn/http_response.rb        | 1 +
 lib/unicorn/http_server.rb          | 1 +
 lib/unicorn/launcher.rb             | 1 +
 lib/unicorn/oob_gc.rb               | 1 +
 lib/unicorn/preread_input.rb        | 1 +
 lib/unicorn/select_waiter.rb        | 1 +
 lib/unicorn/socket_helper.rb        | 1 +
 lib/unicorn/stream_input.rb         | 1 +
 lib/unicorn/tee_input.rb            | 1 +
 lib/unicorn/tmpio.rb                | 1 +
 lib/unicorn/util.rb                 | 1 +
 lib/unicorn/worker.rb               | 1 +
 setup.rb                            | 1 +
 t/broken-app.ru                     | 1 +
 t/client_body_buffer_size.ru        | 1 +
 t/detach.ru                         | 1 +
 t/env.ru                            | 1 +
 t/fails-rack-lint.ru                | 1 +
 t/heartbeat-timeout.ru              | 1 +
 t/integration.ru                    | 1 +
 t/listener_names.ru                 | 1 +
 t/oob_gc.ru                         | 1 +
 t/oob_gc_path.ru                    | 1 +
 t/pid.ru                            | 1 +
 t/preread_input.ru                  | 1 +
 t/reopen-logs.ru                    | 1 +
 t/t0013.ru                          | 1 +
 t/t0014.ru                          | 1 +
 t/t0301.ru                          | 1 +
 test/aggregate.rb                   | 1 +
 test/benchmark/dd.ru                | 1 +
 test/benchmark/ddstream.ru          | 1 +
 test/benchmark/readinput.ru         | 1 +
 test/benchmark/stack.ru             | 1 +
 test/exec/test_exec.rb              | 1 +
 test/test_helper.rb                 | 1 +
 test/unit/test_ccc.rb               | 1 +
 test/unit/test_configurator.rb      | 1 +
 test/unit/test_droplet.rb           | 1 +
 test/unit/test_http_parser.rb       | 1 +
 test/unit/test_http_parser_ng.rb    | 1 +
 test/unit/test_request.rb           | 1 +
 test/unit/test_server.rb            | 1 +
 test/unit/test_signals.rb           | 1 +
 test/unit/test_socket_helper.rb     | 1 +
 test/unit/test_stream_input.rb      | 1 +
 test/unit/test_tee_input.rb         | 1 +
 test/unit/test_util.rb              | 1 +
 test/unit/test_waiter.rb            | 1 +
 unicorn.gemspec                     | 1 +
 66 files changed, 66 insertions(+)

diff --git a/Rakefile b/Rakefile
index 37569ce..fe1588b 100644
--- a/Rakefile
+++ b/Rakefile
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # optional rake-compiler support in case somebody needs to cross compile
 begin
   mk = "ext/unicorn_http/Makefile"
diff --git a/bin/unicorn b/bin/unicorn
index 00c8464..af8353c 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -1,5 +1,6 @@
 #!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 require 'unicorn/launcher'
 require 'optparse'
 
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 354c1df..374fd8e 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -1,5 +1,6 @@
 #!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 require 'unicorn/launcher'
 require 'optparse'
 require 'fileutils'
diff --git a/examples/big_app_gc.rb b/examples/big_app_gc.rb
index c1bae10..0baea26 100644
--- a/examples/big_app_gc.rb
+++ b/examples/big_app_gc.rb
@@ -1,2 +1,3 @@
+# frozen_string_literal: false
 # see {Unicorn::OobGC}[https://yhbt.net/unicorn/Unicorn/OobGC.html]
 # Unicorn::OobGC was broken in Unicorn v3.3.1 - v3.6.1 and fixed in v3.6.2
diff --git a/examples/echo.ru b/examples/echo.ru
index e982180..453a5e6 100644
--- a/examples/echo.ru
+++ b/examples/echo.ru
@@ -1,4 +1,5 @@
 #\-E none
+# frozen_string_literal: false
 #
 # Example application that echoes read data back to the HTTP client.
 # This emulates the old echo protocol people used to run.
diff --git a/examples/logger_mp_safe.rb b/examples/logger_mp_safe.rb
index 05ad3fa..f2c0500 100644
--- a/examples/logger_mp_safe.rb
+++ b/examples/logger_mp_safe.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # Multi-Processing-safe monkey patch for Logger
 #
 # This monkey patch fixes the case where "preload_app true" is used and
diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb
index 46fd634..4f96ede 100644
--- a/examples/unicorn.conf.minimal.rb
+++ b/examples/unicorn.conf.minimal.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # Minimal sample configuration file for Unicorn (not Rack) when used
 # with daemonization (unicorn -D) started in your working directory.
 #
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index d90bdc4..5bae830 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # Sample verbose configuration file for Unicorn (not Rack)
 #
 # This configuration file documents many features of Unicorn
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 11099cd..de896fe 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 require 'mkmf'
 
 have_func("rb_hash_clear", "ruby.h") or abort 'Ruby 2.0+ required'
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 564cb30..fb91679 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 require 'etc'
 require 'stringio'
 require 'raindrops'
diff --git a/lib/unicorn/app/old_rails.rb b/lib/unicorn/app/old_rails.rb
index 1e8c41a..54b3e69 100644
--- a/lib/unicorn/app/old_rails.rb
+++ b/lib/unicorn/app/old_rails.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # :enddoc:
 # This code is based on the original Rails handler in Mongrel
diff --git a/lib/unicorn/app/old_rails/static.rb b/lib/unicorn/app/old_rails/static.rb
index 2257270..cf34e02 100644
--- a/lib/unicorn/app/old_rails/static.rb
+++ b/lib/unicorn/app/old_rails/static.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 # :enddoc:
 # This code is based on the original Rails handler in Mongrel
 # Copyright (c) 2005 Zed A. Shaw
diff --git a/lib/unicorn/cgi_wrapper.rb b/lib/unicorn/cgi_wrapper.rb
index d9b7fe5..fb43605 100644
--- a/lib/unicorn/cgi_wrapper.rb
+++ b/lib/unicorn/cgi_wrapper.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # :enddoc:
 # This code is based on the original CGIWrapper from Mongrel
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index b21a01d..3c81596 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 require 'logger'
 
 # Implements a simple DSL for configuring a unicorn server.
diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index 33ab4ac..8032863 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 module Unicorn::Const # :nodoc:
   # default TCP listen host address (0.0.0.0, all interfaces)
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index ab3bd6e..a48dab7 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 # :enddoc:
 # no stable API here
 require 'unicorn_http'
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 0ed0ae3..3634165 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 # :enddoc:
 # Writes a Rack response to your client using the HTTP/1.1 specification.
 # You use it by simply doing:
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ed5bbf1..08fbe40 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # This is the process manager of Unicorn. This manages worker
 # processes which in turn handle the I/O and application process.
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 78e8f39..bd3324e 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # :enddoc:
 $stdout.sync = $stderr.sync = true
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index db9f2cb..efd9177 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # Strongly consider https://github.com/tmm1/gctools if using Ruby 2.1+
 # It is built on new APIs in Ruby 2.1, so it is more intelligent than
diff --git a/lib/unicorn/preread_input.rb b/lib/unicorn/preread_input.rb
index 12eb3e8..c62cc09 100644
--- a/lib/unicorn/preread_input.rb
+++ b/lib/unicorn/preread_input.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 module Unicorn
 # This middleware is used to ensure input is buffered to memory
diff --git a/lib/unicorn/select_waiter.rb b/lib/unicorn/select_waiter.rb
index cb84aab..d11ea57 100644
--- a/lib/unicorn/select_waiter.rb
+++ b/lib/unicorn/select_waiter.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # fallback for non-Linux and Linux <4.5 systems w/o EPOLLEXCLUSIVE
 class Unicorn::SelectWaiter # :nodoc:
   def get_readers(ready, readers, timeout) # :nodoc:
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 06ec2b2..986932f 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 # :enddoc:
 require 'socket'
 
diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb
index 9246f73..23a9976 100644
--- a/lib/unicorn/stream_input.rb
+++ b/lib/unicorn/stream_input.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # When processing uploads, unicorn may expose a StreamInput object under
 # "rack.input" of the Rack environment when
diff --git a/lib/unicorn/tee_input.rb b/lib/unicorn/tee_input.rb
index 2ccc2d9..b3c6535 100644
--- a/lib/unicorn/tee_input.rb
+++ b/lib/unicorn/tee_input.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # Acts like tee(1) on an input input to provide a input-like stream
 # while providing rewindable semantics through a File/StringIO backing
diff --git a/lib/unicorn/tmpio.rb b/lib/unicorn/tmpio.rb
index 0bbf6ec..deecd80 100644
--- a/lib/unicorn/tmpio.rb
+++ b/lib/unicorn/tmpio.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 # :stopdoc:
 require 'tmpdir'
 
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index b826de4..f28d929 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 require 'fcntl'
 module Unicorn::Util # :nodoc:
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 4af31be..d2445d5 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 require "raindrops"
 
 # This class and its members can be considered a stable interface
diff --git a/setup.rb b/setup.rb
index cf1abd9..96cf75a 100644
--- a/setup.rb
+++ b/setup.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 #
 # setup.rb
 #
diff --git a/t/broken-app.ru b/t/broken-app.ru
index d05d7ab..5966bff 100644
--- a/t/broken-app.ru
+++ b/t/broken-app.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # we do not want Rack::Lint or anything to protect us
 use Rack::ContentLength
 use Rack::ContentType, "text/plain"
diff --git a/t/client_body_buffer_size.ru b/t/client_body_buffer_size.ru
index 44161a5..1a0fb16 100644
--- a/t/client_body_buffer_size.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,4 +1,5 @@
 #\ -E none
+# frozen_string_literal: false
 app = lambda do |env|
   input = env['rack.input']
   case env["PATH_INFO"]
diff --git a/t/detach.ru b/t/detach.ru
index bbd998e..8d35951 100644
--- a/t/detach.ru
+++ b/t/detach.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 use Rack::ContentType, "text/plain"
 fifo_path = ENV["TEST_FIFO"] or abort "TEST_FIFO not set"
 run lambda { |env|
diff --git a/t/env.ru b/t/env.ru
index 388412e..86c3cfa 100644
--- a/t/env.ru
+++ b/t/env.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 use Rack::ContentLength
 use Rack::ContentType, "text/plain"
 run lambda { |env| [ 200, {}, [ env.inspect << "\n" ] ] }
diff --git a/t/fails-rack-lint.ru b/t/fails-rack-lint.ru
index 82bfb5f..8b8b5ec 100644
--- a/t/fails-rack-lint.ru
+++ b/t/fails-rack-lint.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # This rack app returns an invalid status code, which will cause
 # Rack::Lint to throw an exception if it is present.  This
 # is used to check whether Rack::Lint is in the stack or not.
diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index 3eeb5d6..ccc6a8e 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 use Rack::ContentLength
 headers = { 'content-type' => 'text/plain' }
 run lambda { |env|
diff --git a/t/integration.ru b/t/integration.ru
index 888833a..6df481c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -1,4 +1,5 @@
 #!ruby
+# frozen_string_literal: false
 # Copyright (C) unicorn hackers <unicorn-public@80x24.org>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
 
diff --git a/t/listener_names.ru b/t/listener_names.ru
index edb4e6a..f52c59b 100644
--- a/t/listener_names.ru
+++ b/t/listener_names.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 use Rack::ContentLength
 use Rack::ContentType, "text/plain"
 names = Unicorn.listener_names.inspect # rely on preload_app=true
diff --git a/t/oob_gc.ru b/t/oob_gc.ru
index 224cb06..2ae58a8 100644
--- a/t/oob_gc.ru
+++ b/t/oob_gc.ru
@@ -1,4 +1,5 @@
 #\-E none
+# frozen_string_literal: false
 require 'unicorn/oob_gc'
 use Rack::ContentLength
 use Rack::ContentType, "text/plain"
diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru
index 7f40601..5358222 100644
--- a/t/oob_gc_path.ru
+++ b/t/oob_gc_path.ru
@@ -1,4 +1,5 @@
 #\-E none
+# frozen_string_literal: false
 require 'unicorn/oob_gc'
 use Rack::ContentLength
 use Rack::ContentType, "text/plain"
diff --git a/t/pid.ru b/t/pid.ru
index f5fd31f..b49b137 100644
--- a/t/pid.ru
+++ b/t/pid.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 use Rack::ContentLength
 use Rack::ContentType, "text/plain"
 run lambda { |env| [ 200, {}, [ "#$$\n" ] ] }
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 18af221..5f68fe9 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,4 +1,5 @@
 #\-E none
+# frozen_string_literal: false
 require 'digest/md5'
 require 'unicorn/preread_input'
 use Unicorn::PrereadInput
diff --git a/t/reopen-logs.ru b/t/reopen-logs.ru
index c39e8f6..488da85 100644
--- a/t/reopen-logs.ru
+++ b/t/reopen-logs.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 use Rack::ContentLength
 use Rack::ContentType, "text/plain"
 run lambda { |env|
diff --git a/t/t0013.ru b/t/t0013.ru
index 48a3a34..e425093 100644
--- a/t/t0013.ru
+++ b/t/t0013.ru
@@ -1,4 +1,5 @@
 #\ -E none
+# frozen_string_literal: false
 use Rack::ContentLength
 use Rack::ContentType, 'text/plain'
 app = lambda do |env|
diff --git a/t/t0014.ru b/t/t0014.ru
index b0bd2b7..686d214 100644
--- a/t/t0014.ru
+++ b/t/t0014.ru
@@ -1,4 +1,5 @@
 #\ -E none
+# frozen_string_literal: false
 use Rack::ContentLength
 use Rack::ContentType, 'text/plain'
 app = lambda do |env|
diff --git a/t/t0301.ru b/t/t0301.ru
index ce68213..54929b1 100644
--- a/t/t0301.ru
+++ b/t/t0301.ru
@@ -1,4 +1,5 @@
 #\-N --debug
+# frozen_string_literal: false
 run(lambda do |env|
   case env['PATH_INFO']
   when '/vars'
diff --git a/test/aggregate.rb b/test/aggregate.rb
index 5eebbe5..0f32b2f 100755
--- a/test/aggregate.rb
+++ b/test/aggregate.rb
@@ -1,5 +1,6 @@
 #!/usr/bin/ruby -n
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 BEGIN { $tests = $assertions = $failures = $errors = 0 }
 
diff --git a/test/benchmark/dd.ru b/test/benchmark/dd.ru
index 111fa2e..5bd2739 100644
--- a/test/benchmark/dd.ru
+++ b/test/benchmark/dd.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # This benchmark is the simplest test of the I/O facilities in
 # unicorn.  It is meant to return a fixed-sized blob to test
 # the performance of things in Unicorn, _NOT_ the app.
diff --git a/test/benchmark/ddstream.ru b/test/benchmark/ddstream.ru
index b14c973..fd40ced 100644
--- a/test/benchmark/ddstream.ru
+++ b/test/benchmark/ddstream.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # This app is intended to test large HTTP responses with or without
 # a fully-buffering reverse proxy such as nginx. Without a fully-buffering
 # reverse proxy, unicorn will be unresponsive when client count exceeds
diff --git a/test/benchmark/readinput.ru b/test/benchmark/readinput.ru
index c91bec3..95c0226 100644
--- a/test/benchmark/readinput.ru
+++ b/test/benchmark/readinput.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 # This app is intended to test large HTTP requests with or without
 # a fully-buffering reverse proxy such as nginx. Without a fully-buffering
 # reverse proxy, unicorn will be unresponsive when client count exceeds
diff --git a/test/benchmark/stack.ru b/test/benchmark/stack.ru
index fc9193f..17a565b 100644
--- a/test/benchmark/stack.ru
+++ b/test/benchmark/stack.ru
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 run(lambda { |env|
   body = "#{caller.size}\n"
   h = {
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 8494452..807f724 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 # Don't add to this file, new tests are in Perl 5. See t/README
 FLOCK_PATH = File.expand_path(__FILE__)
 require './test/test_helper'
diff --git a/test/test_helper.rb b/test/test_helper.rb
index d86f83b..0bf3c90 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # Copyright (c) 2005 Zed A. Shaw
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
index f518230..a0a2bff 100644
--- a/test/unit/test_ccc.rb
+++ b/test/unit/test_ccc.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 require 'socket'
 require 'unicorn'
 require 'io/wait'
diff --git a/test/unit/test_configurator.rb b/test/unit/test_configurator.rb
index 1298f0e..1a89aca 100644
--- a/test/unit/test_configurator.rb
+++ b/test/unit/test_configurator.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 require 'test/unit'
 require 'tempfile'
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 81ad82b..4b2d2d0 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 require 'test/unit'
 require 'unicorn'
 
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 697af44..adcc84f 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # Copyright (c) 2005 Zed A. Shaw
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 425d5ad..fd47246 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 require './test/test_helper'
 require 'digest/md5'
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index 53ae944..9d1b350 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # Copyright (c) 2009 Eric Wong
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 7ffa48f..5a2252f 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # Copyright (c) 2005 Zed A. Shaw
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index 6c48754..49ff3c7 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 # Copyright (c) 2009 Eric Wong
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index a446f06..4363474 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 require './test/test_helper'
 require 'tempfile'
diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb
index 7986ca7..7ee98e4 100644
--- a/test/unit/test_stream_input.rb
+++ b/test/unit/test_stream_input.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 require 'test/unit'
 require 'digest/sha1'
diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb
index 607ce87..8f05c77 100644
--- a/test/unit/test_tee_input.rb
+++ b/test/unit/test_tee_input.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 require 'test/unit'
 require 'digest/sha1'
diff --git a/test/unit/test_util.rb b/test/unit/test_util.rb
index bc7b233..ce53b86 100644
--- a/test/unit/test_util.rb
+++ b/test/unit/test_util.rb
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 
 require './test/test_helper'
 require 'tempfile'
diff --git a/test/unit/test_waiter.rb b/test/unit/test_waiter.rb
index 0995de2..a20994b 100644
--- a/test/unit/test_waiter.rb
+++ b/test/unit/test_waiter.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: false
 require 'test/unit'
 require 'unicorn'
 require 'unicorn/select_waiter'
diff --git a/unicorn.gemspec b/unicorn.gemspec
index e7e3ef7..36700a8 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -1,4 +1,5 @@
 # -*- encoding: binary -*-
+# frozen_string_literal: false
 manifest = File.exist?('.manifest') ?
   IO.readlines('.manifest').map!(&:chomp!) : `git ls-files`.split("\n")
 

^ permalink raw reply related	[relevance 3%]

* [PATCH 00..11/11] more tests to Perl 5..
@ 2023-09-10 20:08  1% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-10 20:08 UTC (permalink / raw)
  To: unicorn-public

[-- Attachment #1: Type: text/plain, Size: 3020 bytes --]

Hopefully this is less maintenance down the line since Ruby
introduces incompatibilities at a higher rate than Perl.
I don't fully trust Perl, either, but far more Ruby code gets
broken by new releases.

More to come at some point...

Note: attached patches are generated with --irreversible-delete
to save bandwidth.

Eric Wong (11):
  tests: port some bad config tests to Perl 5
  tests: port working_directory tests to Perl 5
  tests: port t/heartbeat-timeout to Perl 5
  tests: port reopen logs test over to Perl 5
  tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
  tests: introduce `do_req' helper sub
  tests: use more common variable names between tests
  tests: use Time::HiRes `sleep' and `time' everywhere
  tests: fold SO_KEEPALIVE check to Perl 5 integration
  tests: move broken app test to Perl 5 integration test
  tests: fold early shutdown() tests into t/integration.t

 t/active-unix-socket.t                    |  4 +-
 t/client_body_buffer_size.t               |  6 +-
 t/heartbeat-timeout.ru                    |  2 +-
 t/heartbeat-timeout.t                     | 62 +++++++++++++++
 t/integration.ru                          |  1 +
 t/integration.t                           | 82 +++++++++++++-------
 t/lib.perl                                | 51 ++++++++++--
 t/reload-bad-config.t                     | 54 +++++++++++++
 t/{t0006.ru => reopen-logs.ru}            |  0
 t/reopen-logs.t                           | 39 ++++++++++
 t/t0001-reload-bad-config.sh              | 53 -------------
 t/t0002-config-conflict.sh                | 49 ------------
 t/t0003-working_directory.sh              | 51 ------------
 t/t0004-heartbeat-timeout.sh              | 69 -----------------
 t/t0004-working_directory_broken.sh       | 24 ------
 t/t0005-working_directory_app.rb.sh       | 40 ----------
 t/t0006-reopen-logs.sh                    | 83 --------------------
 t/t0007-working_directory_no_embed_cli.sh | 44 -----------
 t/t0009-winch_ttin.sh                     | 59 --------------
 t/winch_ttin.t                            | 67 ++++++++++++++++
 t/working_directory.t                     | 94 +++++++++++++++++++++++
 test/exec/test_exec.rb                    | 23 +-----
 test/unit/test_server.rb                  | 67 ----------------
 23 files changed, 424 insertions(+), 600 deletions(-)
 create mode 100644 t/heartbeat-timeout.t
 create mode 100644 t/reload-bad-config.t
 rename t/{t0006.ru => reopen-logs.ru} (100%)
 create mode 100644 t/reopen-logs.t
 delete mode 100755 t/t0001-reload-bad-config.sh
 delete mode 100755 t/t0002-config-conflict.sh
 delete mode 100755 t/t0003-working_directory.sh
 delete mode 100755 t/t0004-heartbeat-timeout.sh
 delete mode 100755 t/t0004-working_directory_broken.sh
 delete mode 100755 t/t0005-working_directory_app.rb.sh
 delete mode 100755 t/t0006-reopen-logs.sh
 delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
 delete mode 100755 t/t0009-winch_ttin.sh
 create mode 100644 t/winch_ttin.t
 create mode 100644 t/working_directory.t

[-- Attachment #2: 0001-tests-port-some-bad-config-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3987 bytes --]

From f43c28ea10ca8d520b55f2fbb20710dd66fc4fb5 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 22:55:09 +0000
Subject: [PATCH 01/11] tests: port some bad config tests to Perl 5

We can fold some tests into one test to save on Perl startup
time (but Ruby startup time is a lost cause).
---
 t/lib.perl                   | 12 ++++----
 t/reload-bad-config.t        | 58 ++++++++++++++++++++++++++++++++++++
 t/t0001-reload-bad-config.sh | 53 --------------------------------
 t/t0002-config-conflict.sh   | 49 ------------------------------
 4 files changed, 65 insertions(+), 107 deletions(-)
 create mode 100644 t/reload-bad-config.t
 delete mode 100755 t/t0001-reload-bad-config.sh
 delete mode 100755 t/t0002-config-conflict.sh

diff --git a/t/lib.perl b/t/lib.perl
index fe3404ba..7de9e426 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -9,17 +9,19 @@ use Test::More;
 use IO::Socket::INET;
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+our ($tmpdir, $errfh, $err_log);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
+	$tmpdir $errfh $err_log
 	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log");
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
+$err_log = "$tmpdir/err.log";
+open($errfh, '>>', $err_log);
+END { diag slurp($err_log) if $tmpdir };
 
 sub check_stderr () {
-	my @log = slurp("$tmpdir/err.log");
+	my @log = slurp($err_log);
 	diag("@log") if $ENV{V};
 	my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
 	@err = grep(!/failed to set accept_filter=/, @err);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
new file mode 100644
index 00000000..c7055c7e
--- /dev/null
+++ b/t/reload-bad-config.t
@@ -0,0 +1,58 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my $ru = "$tmpdir/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+
+open my $fh, '>', $ru;
+print $fh <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+config = ru = "hello world\n" # check for config variable conflicts, too
+run lambda { |env| [ 200, {}, [ ru.to_s ] ] }
+EOM
+close $fh;
+
+open $fh, '>', $u_conf;
+print $fh <<EOM;
+preload_app true
+stderr_path "$err_log"
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
+is($bdy, "hello world\n", 'body matches expected');
+
+open $fh, '>>', $ru;
+say $fh '....this better be a syntax error in any version of ruby...';
+close $fh;
+
+$ar->do_kill('HUP'); # reload
+my @l;
+for (1..1000) {
+	@l = grep(/(?:done|error) reloading/, slurp($err_log)) and
+		last;
+	select undef, undef, undef, 0.011;
+}
+diag slurp($err_log) if $ENV{V};
+ok(grep(/error reloading/, @l), 'got error reloading');
+open $fh, '>', $err_log;
+close $fh;
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
+is($bdy, "hello world\n", 'body matches expected after reload');
+
+check_stderr;
+undef $tmpdir; # quiet t/lib.perl END{}
+done_testing;
diff --git a/t/t0001-reload-bad-config.sh b/t/t0001-reload-bad-config.sh
deleted file mode 100755
index 55bb3555..00000000
diff --git a/t/t0002-config-conflict.sh b/t/t0002-config-conflict.sh
deleted file mode 100755
index d7b2181a..00000000

[-- Attachment #3: 0002-tests-port-working_directory-tests-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4809 bytes --]

From d4514174ee7eadea89003f380acacf32d52acd9d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:18:16 +0000
Subject: [PATCH 02/11] tests: port working_directory tests to Perl 5

We can fold a bunch of them into one test to save startup
time, inodes, and FS activity.
---
 t/t0003-working_directory.sh              |  51 ---------
 t/t0004-working_directory_broken.sh       |  24 -----
 t/t0005-working_directory_app.rb.sh       |  40 -------
 t/t0007-working_directory_no_embed_cli.sh |  44 --------
 t/working_directory.t                     | 122 ++++++++++++++++++++++
 5 files changed, 122 insertions(+), 159 deletions(-)
 delete mode 100755 t/t0003-working_directory.sh
 delete mode 100755 t/t0004-working_directory_broken.sh
 delete mode 100755 t/t0005-working_directory_app.rb.sh
 delete mode 100755 t/t0007-working_directory_no_embed_cli.sh
 create mode 100644 t/working_directory.t

diff --git a/t/t0003-working_directory.sh b/t/t0003-working_directory.sh
deleted file mode 100755
index 79988d8b..00000000
diff --git a/t/t0004-working_directory_broken.sh b/t/t0004-working_directory_broken.sh
deleted file mode 100755
index ca9d3825..00000000
diff --git a/t/t0005-working_directory_app.rb.sh b/t/t0005-working_directory_app.rb.sh
deleted file mode 100755
index 0fbab4fc..00000000
diff --git a/t/t0007-working_directory_no_embed_cli.sh b/t/t0007-working_directory_no_embed_cli.sh
deleted file mode 100755
index 77d67072..00000000
diff --git a/t/working_directory.t b/t/working_directory.t
new file mode 100644
index 00000000..e7ff43a5
--- /dev/null
+++ b/t/working_directory.t
@@ -0,0 +1,122 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+mkdir "$tmpdir/alt";
+my $u_sock = "$tmpdir/u.sock";
+my $ru = "$tmpdir/alt/config.ru";
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+working_directory "$tmpdir/alt" # the whole point of this test
+before_fork { |_,_| \$master_ppid = Process.ppid }
+EOM
+close $fh;
+
+my $common_ru = <<'EOM';
+use Rack::ContentLength
+use Rack::ContentType, 'text/plain'
+run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] }
+EOM
+
+open $fh, '>', $ru;
+print $fh <<EOM;
+#\\--daemonize --listen $u_sock
+$common_ru
+EOM
+close $fh;
+
+my $pid;
+my $stop_daemon = sub {
+	my ($is_END) = @_;
+	kill('TERM', $pid);
+	my $tries = 1000;
+	while (CORE::kill(0, $pid) && --$tries) {
+		select undef, undef, undef, 0.01;
+	}
+	if ($is_END && CORE::kill(0, $pid)) {
+		CORE::kill('KILL', $pid);
+		die "daemonized PID=$pid did not die";
+	} else {
+		ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
+		undef $pid;
+	}
+};
+
+END { $stop_daemon->(1) if defined $pid };
+
+unicorn('-c', $u_conf)->join; # will daemonize
+chomp($pid = slurp("$tmpdir/pid"));
+
+my $c = unix_start($u_sock, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+chomp(my $bdy = do { local $/; <$c> });
+is($bdy, 1, 'got expected $master_ppid');
+
+$stop_daemon->();
+check_stderr;
+
+if ('test without CLI switches in config.ru') {
+	truncate $err_log, 0;
+	open $fh, '>', $ru;
+	print $fh $common_ru;
+	close $fh;
+
+	unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
+	chomp($pid = slurp("$tmpdir/pid"));
+
+	$c = unix_start($u_sock, 'GET / HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	chomp($bdy = do { local $/; <$c> });
+	is($bdy, 1, 'got expected $master_ppid');
+
+	$stop_daemon->();
+	check_stderr;
+}
+
+if ('ensures broken working_directory (missing config.ru) is OK') {
+	truncate $err_log, 0;
+	unlink $ru;
+
+	my $auto_reap = unicorn('-c', $u_conf);
+	$auto_reap->join;
+	isnt($?, 0, 'exited with error due to missing config.ru');
+
+	like(slurp($err_log), qr/rackup file \Q(config.ru)\E not readable/,
+		'noted unreadability of config.ru in stderr');
+}
+
+if ('fooapp.rb (not config.ru) works with working_directory') {
+	truncate $err_log, 0;
+	my $fooapp = "$tmpdir/alt/fooapp.rb";
+	open $fh, '>', $fooapp;
+	print $fh <<EOM;
+class Fooapp
+  def self.call(env)
+    b = "dir=#{Dir.pwd}"
+    h = { 'content-type' => 'text/plain', 'content-length' => b.bytesize.to_s }
+    [ 200, h, [ b ] ]
+  end
+end
+EOM
+	close $fh;
+	my $srv = tcp_server;
+	my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
+				{ -C => '/', 3 => $srv });
+	$c = tcp_start($srv, 'GET / HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	chomp($bdy = do { local $/; <$c> });
+	is($bdy, "dir=$tmpdir/alt",
+		'fooapp.rb (w/o config.ru) w/ working_directory');
+	close $c;
+	$auto_reap->join('TERM');
+	is($?, 0, 'fooapp.rb process exited');
+	check_stderr;
+}
+
+undef $tmpdir;
+done_testing;

[-- Attachment #4: 0003-tests-port-t-heartbeat-timeout-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3478 bytes --]

From d67284a692683bca59effd9c0670bd5dd47e4fa3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Thu, 7 Sep 2023 23:53:58 +0000
Subject: [PATCH 03/11] tests: port t/heartbeat-timeout to Perl 5

I absolutely detest and regret adding this feature,
but I'm hell bent on supporting it until the end of days
because we don't break compatibility.
---
 t/heartbeat-timeout.ru       |  2 +-
 t/heartbeat-timeout.t        | 69 ++++++++++++++++++++++++++++++++++++
 t/t0004-heartbeat-timeout.sh | 69 ------------------------------------
 3 files changed, 70 insertions(+), 70 deletions(-)
 create mode 100644 t/heartbeat-timeout.t
 delete mode 100755 t/t0004-heartbeat-timeout.sh

diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru
index 20a79380..3eeb5d64 100644
--- a/t/heartbeat-timeout.ru
+++ b/t/heartbeat-timeout.ru
@@ -7,6 +7,6 @@
     sleep # in case STOP signal is not received in time
     [ 500, headers, [ "Should never get here\n" ] ]
   else
-    [ 200, headers, [ "#$$\n" ] ]
+    [ 200, headers, [ "#$$" ] ]
   end
 }
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
new file mode 100644
index 00000000..1fcf21a2
--- /dev/null
+++ b/t/heartbeat-timeout.t
@@ -0,0 +1,69 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
+mkdir "$tmpdir/alt";
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+preload_app true
+stderr_path "$err_log"
+timeout 3 # WORST FEATURE EVER
+EOM
+close $fh;
+
+my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
+
+my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $wpid = do { local $/; <$c> };
+like($wpid, qr/\A[0-9]+\z/, 'worker is running');
+
+my $t0 = clock_gettime(CLOCK_MONOTONIC);
+$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+vec(my $rvec = '', fileno($c), 1) = 1;
+is(select($rvec, undef, undef, 6), 1, 'got readiness');
+$c->blocking(0);
+is(sysread($c, my $buf, 128), 0, 'got EOF response');
+my $elapsed = clock_gettime(CLOCK_MONOTONIC) - $t0;
+ok($elapsed > 3, 'timeout took >3s');
+
+my @timeout_err = slurp($err_log);
+truncate($err_log, 0);
+is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
+    'noted timeout error') or diag explain(\@timeout_err);
+
+# did it respawn?
+$c = tcp_start($srv, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
+my $new_pid = do { local $/; <$c> };
+isnt($new_pid, $wpid, 'spawned new worker');
+
+diag 'SIGSTOP for 4 seconds...';
+$ar->do_kill('STOP');
+sleep 4;
+$ar->do_kill('CONT');
+for my $i (1..2) {
+	$c = tcp_start($srv, 'GET /pid HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200\b!,
+		"PID request succeeds #$i after STOP+CONT");
+	my $spid = do { local $/; <$c> };
+	is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
+	if ($i == 1) {
+		diag 'sleeping 2s to ensure timeout is not delayed';
+		sleep 2;
+	}
+}
+
+$ar->join('TERM');
+check_stderr;
+undef $tmpdir;
+
+done_testing;
diff --git a/t/t0004-heartbeat-timeout.sh b/t/t0004-heartbeat-timeout.sh
deleted file mode 100755
index 29652837..00000000

[-- Attachment #5: 0004-tests-port-reopen-logs-test-over-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2317 bytes --]

From 1607ac966f604ec4cf383025c4c3ee296f638fff Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 07:13:11 +0000
Subject: [PATCH 04/11] tests: port reopen logs test over to Perl 5

Being able to do subsecond sleeps is one welcome advantage
over POSIX (not GNU) sleep(1) in portable Bourne sh.
---
 t/{t0006.ru => reopen-logs.ru} |  0
 t/reopen-logs.t                | 43 ++++++++++++++++++
 t/t0006-reopen-logs.sh         | 83 ----------------------------------
 3 files changed, 43 insertions(+), 83 deletions(-)
 rename t/{t0006.ru => reopen-logs.ru} (100%)
 create mode 100644 t/reopen-logs.t
 delete mode 100755 t/t0006-reopen-logs.sh

diff --git a/t/t0006.ru b/t/reopen-logs.ru
similarity index 100%
rename from t/t0006.ru
rename to t/reopen-logs.ru
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
new file mode 100644
index 00000000..e1bf524c
--- /dev/null
+++ b/t/reopen-logs.t
@@ -0,0 +1,43 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $srv = tcp_server();
+my $u_conf = "$tmpdir/u.conf.rb";
+my $out_log = "$tmpdir/out.log";
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+stderr_path "$err_log"
+stdout_path "$out_log"
+EOM
+close $fh;
+
+my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
+my $c = tcp_start($srv, 'GET / HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+my $bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs opened');
+
+rename($err_log, "$err_log.rot");
+rename($out_log, "$out_log.rot");
+
+$auto_reap->do_kill('USR1');
+
+my $tries = 1000;
+while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+
+ok(-f $out_log, 'stdout_path recreated after USR1');
+ok(-f $err_log, 'stderr_path recreated after USR1');
+
+$c = tcp_start($srv, 'GET / HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+$bdy = do { local $/; <$c> };
+is($bdy, "true\n", 'logs reopened with sync==true');
+
+$auto_reap->join('QUIT');
+is($?, 0, 'no error on exit');
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/t0006-reopen-logs.sh b/t/t0006-reopen-logs.sh
deleted file mode 100755
index a6e7a17c..00000000

[-- Attachment #6: 0005-tests-rewrite-SIGWINCH-SIGTTIN-test-in-Perl-5.patch --]
[-- Type: text/x-diff, Size: 2916 bytes --]

From 86aea575c331a3b5242db1c14a848928a37ff9e3 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 08:27:04 +0000
Subject: [PATCH 05/11] tests: rewrite SIGWINCH && SIGTTIN test in Perl 5

No need to deal with full second sleeps, here.
---
 t/t0009-winch_ttin.sh | 59 -----------------------------------
 t/winch_ttin.t        | 72 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+), 59 deletions(-)
 delete mode 100755 t/t0009-winch_ttin.sh
 create mode 100644 t/winch_ttin.t

diff --git a/t/t0009-winch_ttin.sh b/t/t0009-winch_ttin.sh
deleted file mode 100755
index 6e56e30c..00000000
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
new file mode 100644
index 00000000..1a198778
--- /dev/null
+++ b/t/winch_ttin.t
@@ -0,0 +1,72 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+use POSIX qw(mkfifo);
+my $u_conf = "$tmpdir/u.conf.rb";
+my $u_sock = "$tmpdir/u.sock";
+my $fifo = "$tmpdir/fifo";
+mkfifo($fifo, 0666) or die "mkfifo($fifo): $!";
+
+open my $fh, '>', $u_conf;
+print $fh <<EOM;
+pid "$tmpdir/pid"
+listen "$u_sock"
+stderr_path "$err_log"
+after_fork do |server, worker|
+  # test script will block while reading from $fifo,
+  File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s }
+end
+EOM
+close $fh;
+
+unicorn('-D', '-c', $u_conf, 't/integration.ru')->join;
+is($?, 0, 'daemonized properly');
+open $fh, '<', "$tmpdir/pid";
+chomp(my $pid = <$fh>);
+ok(kill(0, $pid), 'daemonized PID works');
+my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef };
+END { $quit->() };
+
+open $fh, '<', $fifo;
+my $worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'initial worker spawned');
+
+my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+my ($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+my $worker_pid = do { local $/; <$c> };
+like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
+chomp $worker_pid;
+ok(kill(0, $worker_pid), 'worker_pid is valid');
+
+ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
+
+my $tries = 1000;
+while (CORE::kill(0, $worker_pid) && --$tries) {
+	select undef, undef, undef, 0.01;
+}
+ok(!CORE::kill(0, $worker_pid), 'worker not running');
+
+ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');
+
+open $fh, '<', $fifo;
+$worker_nr = <$fh>;
+close $fh;
+is($worker_nr, '0', 'worker restarted');
+
+$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
+($status, $hdr) = slurp_hdr($c);
+like($status, qr/ 200\b/, 'got 200 response');
+chomp(my $new_worker_pid = do { local $/; <$c> });
+like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+ok(kill(0, $new_worker_pid), 'got a valid worker PID');
+isnt($worker_pid, $new_worker_pid, 'worker PID changed');
+
+$quit->();
+
+check_stderr;
+undef $tmpdir;
+done_testing;

[-- Attachment #7: 0006-tests-introduce-do_req-helper-sub.patch --]
[-- Type: text/x-diff, Size: 11556 bytes --]

From 29885f0d95aaa8e1d1f6cf3b791d9f08338a511e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:15:16 +0000
Subject: [PATCH 06/11] tests: introduce `do_req' helper sub

While early tests required fine-grained control in trickling
requests, many of our later tests can use a short one-liner
w/o having to spawn curl.
---
 t/heartbeat-timeout.t | 12 +++---------
 t/integration.t       | 33 +++++++++++++--------------------
 t/lib.perl            | 16 +++++++++++++++-
 t/reload-bad-config.t |  8 ++------
 t/reopen-logs.t       |  8 ++------
 t/winch_ttin.t        | 11 ++++-------
 t/working_directory.t | 17 +++++------------
 7 files changed, 44 insertions(+), 61 deletions(-)

diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index 1fcf21a2..ce1f7e16 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -18,10 +18,8 @@ close $fh;
 
 my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv });
 
-my $c = tcp_start($srv, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $wpid) = do_req($srv, 'GET /pid HTTP/1.0');
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $wpid = do { local $/; <$c> };
 like($wpid, qr/\A[0-9]+\z/, 'worker is running');
 
 my $t0 = clock_gettime(CLOCK_MONOTONIC);
@@ -39,10 +37,8 @@ is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1,
     'noted timeout error') or diag explain(\@timeout_err);
 
 # did it respawn?
-$c = tcp_start($srv, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_pid) = do_req($srv, 'GET /pid HTTP/1.0');
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
-my $new_pid = do { local $/; <$c> };
 isnt($new_pid, $wpid, 'spawned new worker');
 
 diag 'SIGSTOP for 4 seconds...';
@@ -50,11 +46,9 @@ $ar->do_kill('STOP');
 sleep 4;
 $ar->do_kill('CONT');
 for my $i (1..2) {
-	$c = tcp_start($srv, 'GET /pid HTTP/1.0');
-	($status, $hdr) = slurp_hdr($c);
+	($status, $hdr, my $spid) = do_req($srv, 'GET /pid HTTP/1.0');
 	like($status, qr!\AHTTP/1\.[01] 200\b!,
 		"PID request succeeds #$i after STOP+CONT");
-	my $spid = do { local $/; <$c> };
 	is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i");
 	if ($i == 1) {
 		diag 'sleeping 2s to ensure timeout is not delayed';
diff --git a/t/integration.t b/t/integration.t
index bb2ab51b..13b07467 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -62,11 +62,10 @@ EOM
 	},
 );
 
-my ($c, $status, $hdr);
+my ($c, $status, $hdr, $bdy);
 
 # response header tests
-$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
 my $orig_200_status = $status;
 is_deeply([ grep(/^X-R2: /, @$hdr) ],
@@ -84,16 +83,16 @@ SKIP: { # Date header check
 };
 
 
-$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
 is_deeply([ grep(/^x-r3: /, @$hdr) ],
 	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
 	'rack 3 array headers supported') or diag(explain($hdr));
 
 SKIP: {
 	eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
-	my $c = tcp_start($srv, 'GET /env_dump');
-	my $json = do { local $/; readline($c) };
+	($status, $hdr, my $json) = do_req $srv, 'GET /env_dump';
+	is($status, undef, 'no status for HTTP/0.9');
+	is($hdr, undef, 'no header for HTTP/0.9');
 	unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
 	unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
 	my $env = JSON::PP->new->decode($json);
@@ -102,8 +101,7 @@ SKIP: {
 }
 
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr) = do_req($srv, 'GET /nil-header-value HTTP/1.0');
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
@@ -128,12 +126,10 @@ my $ck_early_hints = sub {
 $ck_early_hints->('ccc off'); # we'll retest later
 
 if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
-	$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
-	($status, $hdr) = slurp_hdr($c);
+	($status, $hdr) = do_req $srv, 'POST /tweak-status-code HTTP/1.0';
 	like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
 
-	$c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
-	($status, $hdr) = slurp_hdr($c);
+	($status, $hdr) = do_req $srv, 'POST /restore-status-code HTTP/1.0';
 	is($status, $orig_200_status, 'original status restored');
 }
 
@@ -145,12 +141,11 @@ SKIP: {
 }
 
 if ('bad requests') {
-	$c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
-	($status, $hdr) = slurp_hdr($c);
+	($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1';
 	like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
 
 	$c = tcp_start($srv);
-	print $c 'GET /';;
+	print $c 'GET /';
 	my $buf = join('', (0..9), 'ab');
 	for (0..1023) { print $c $buf }
 	print $c " HTTP/1.0\r\n\r\n";
@@ -308,12 +303,10 @@ EOM
 	$wpid =~ s/\Apid=// or die;
 	ok(CORE::kill(0, $wpid), 'worker PID retrieved');
 
-	$c = tcp_start($srv, $req);
-	($status, $hdr) = slurp_hdr($c);
+	($status, $hdr) = do_req($srv, $req);
 	like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
 
-	$c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
-	($status, $hdr) = slurp_hdr($c);
+	($status, $hdr) = do_req($srv, 'GET /xxxxxx HTTP/1.0');
 	like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
 }
 
diff --git a/t/lib.perl b/t/lib.perl
index 7de9e426..13e390d6 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -12,7 +12,8 @@ use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh, $err_log);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
 	$tmpdir $errfh $err_log
-	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
+	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
+	do_req);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -182,6 +183,19 @@ sub unicorn {
 	UnicornTest::AutoReap->new($pid);
 }
 
+sub do_req ($@) {
+	my ($dst, @req) = @_;
+	my $c = ref($dst) ? tcp_start($dst, @req) : unix_start($dst, @req);
+	return $c if !wantarray;
+	my ($status, $hdr);
+	# read headers iff HTTP/1.x request, HTTP/0.9 remains supported
+	my ($first) = (join('', @req) =~ m!\A([^\r\n]+)!);
+	($status, $hdr) = slurp_hdr($c) if $first =~ m{\s*HTTP/\S+$};
+	my $bdy = do { local $/; <$c> };
+	close $c;
+	($status, $hdr, $bdy);
+}
+
 # automatically kill + reap children when this goes out-of-scope
 package UnicornTest::AutoReap;
 use v5.14;
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index c7055c7e..543421da 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -25,9 +25,7 @@ EOM
 close $fh;
 
 my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv });
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start');
 is($bdy, "hello world\n", 'body matches expected');
 
@@ -47,9 +45,7 @@ ok(grep(/error reloading/, @l), 'got error reloading');
 open $fh, '>', $err_log;
 close $fh;
 
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload');
 is($bdy, "hello world\n", 'body matches expected after reload');
 
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index e1bf524c..8a58c1b9 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -14,9 +14,7 @@ EOM
 close $fh;
 
 my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } );
-my $c = tcp_start($srv, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-my $bdy = do { local $/; <$c> };
+my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
 is($bdy, "true\n", 'logs opened');
 
 rename($err_log, "$err_log.rot");
@@ -31,9 +29,7 @@ while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
 ok(-f $out_log, 'stdout_path recreated after USR1');
 ok(-f $err_log, 'stderr_path recreated after USR1');
 
-$c = tcp_start($srv, 'GET / HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
-$bdy = do { local $/; <$c> };
+($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
 is($bdy, "true\n", 'logs reopened with sync==true');
 
 $auto_reap->join('QUIT');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 1a198778..509b118f 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -34,10 +34,8 @@ my $worker_nr = <$fh>;
 close $fh;
 is($worker_nr, '0', 'initial worker spawned');
 
-my $c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
+my ($status, $hdr, $worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
 like($status, qr/ 200\b/, 'got 200 response');
-my $worker_pid = do { local $/; <$c> };
 like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response');
 chomp $worker_pid;
 ok(kill(0, $worker_pid), 'worker_pid is valid');
@@ -57,11 +55,10 @@ $worker_nr = <$fh>;
 close $fh;
 is($worker_nr, '0', 'worker restarted');
 
-$c = unix_start($u_sock, 'GET /pid HTTP/1.0');
-($status, $hdr) = slurp_hdr($c);
+($status, $hdr, my $new_worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0');
 like($status, qr/ 200\b/, 'got 200 response');
-chomp(my $new_worker_pid = do { local $/; <$c> });
-like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID');
+like($new_worker_pid, qr/\A[0-9]+\n\z/, 'got new worker PID');
+chomp $new_worker_pid;
 ok(kill(0, $new_worker_pid), 'got a valid worker PID');
 isnt($worker_pid, $new_worker_pid, 'worker PID changed');
 
diff --git a/t/working_directory.t b/t/working_directory.t
index e7ff43a5..6c974720 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -52,10 +52,8 @@ END { $stop_daemon->(1) if defined $pid };
 unicorn('-c', $u_conf)->join; # will daemonize
 chomp($pid = slurp("$tmpdir/pid"));
 
-my $c = unix_start($u_sock, 'GET / HTTP/1.0');
-my ($status, $hdr) = slurp_hdr($c);
-chomp(my $bdy = do { local $/; <$c> });
-is($bdy, 1, 'got expected $master_ppid');
+my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+is($bdy, "1\n", 'got expected $master_ppid');
 
 $stop_daemon->();
 check_stderr;
@@ -69,10 +67,8 @@ if ('test without CLI switches in config.ru') {
 	unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
 	chomp($pid = slurp("$tmpdir/pid"));
 
-	$c = unix_start($u_sock, 'GET / HTTP/1.0');
-	($status, $hdr) = slurp_hdr($c);
-	chomp($bdy = do { local $/; <$c> });
-	is($bdy, 1, 'got expected $master_ppid');
+	($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
+	is($bdy, "1\n", 'got expected $master_ppid');
 
 	$stop_daemon->();
 	check_stderr;
@@ -107,12 +103,9 @@ EOM
 	my $srv = tcp_server;
 	my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb),
 				{ -C => '/', 3 => $srv });
-	$c = tcp_start($srv, 'GET / HTTP/1.0');
-	($status, $hdr) = slurp_hdr($c);
-	chomp($bdy = do { local $/; <$c> });
+	($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0');
 	is($bdy, "dir=$tmpdir/alt",
 		'fooapp.rb (w/o config.ru) w/ working_directory');
-	close $c;
 	$auto_reap->join('TERM');
 	is($?, 0, 'fooapp.rb process exited');
 	check_stderr;

[-- Attachment #8: 0007-tests-use-more-common-variable-names-between-tests.patch --]
[-- Type: text/x-diff, Size: 6507 bytes --]

From 948f78403172657590d690b9255467b9ccb968cd Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:31:44 +0000
Subject: [PATCH 07/11] tests: use more common variable names between tests

Stuff like $u_conf, $daemon_pid, $pid_file, etc. will
reduce cognitive overhead.
---
 t/active-unix-socket.t      |  2 +-
 t/client_body_buffer_size.t |  6 ++----
 t/heartbeat-timeout.t       |  3 +--
 t/integration.t             |  5 ++---
 t/lib.perl                  | 31 +++++++++++++++++++++++++++----
 t/working_directory.t       | 31 +++++--------------------------
 6 files changed, 38 insertions(+), 40 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 4dcc8dc6..32cb0c2e 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -15,7 +15,7 @@ my $u2 = "$tmpdir/u2.sock";
 	print $fh <<EOM;
 pid "$tmpdir/u.pid"
 listen "$u1"
-stderr_path "$tmpdir/err.log"
+stderr_path "$err_log"
 EOM
 	close $fh;
 
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index 3067f284..d4799012 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -4,16 +4,14 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
-my $uconf = "$tmpdir/u.conf.rb";
-
-open my $conf_fh, '>', $uconf;
+open my $conf_fh, '>', $u_conf;
 $conf_fh->autoflush(1);
 print $conf_fh <<EOM;
 client_body_buffer_size 0
 EOM
 my $srv = tcp_server();
 my $host_port = tcp_host_port($srv);
-my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf);
 my $ar = unicorn(@uarg, { 3 => $srv });
 my ($c, $status, $hdr);
 my $mem_class = 'StringIO';
diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t
index ce1f7e16..694867a4 100644
--- a/t/heartbeat-timeout.t
+++ b/t/heartbeat-timeout.t
@@ -6,7 +6,6 @@ use autodie;
 use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
 mkdir "$tmpdir/alt";
 my $srv = tcp_server();
-my $u_conf = "$tmpdir/u.conf.rb";
 open my $fh, '>', $u_conf;
 print $fh <<EOM;
 pid "$tmpdir/pid"
@@ -23,7 +22,7 @@ like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds');
 like($wpid, qr/\A[0-9]+\z/, 'worker is running');
 
 my $t0 = clock_gettime(CLOCK_MONOTONIC);
-$c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
+my $c = tcp_start($srv, 'GET /block-forever HTTP/1.0');
 vec(my $rvec = '', fileno($c), 1) = 1;
 is(select($rvec, undef, undef, 6), 1, 'got readiness');
 $c->blocking(0);
diff --git a/t/integration.t b/t/integration.t
index 13b07467..eb40ffc7 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -10,15 +10,14 @@ use autodie;
 our $srv = tcp_server();
 our $host_port = tcp_host_port($srv);
 my $t0 = time;
-my $conf = "$tmpdir/u.conf.rb";
-open my $conf_fh, '>', $conf;
+open my $conf_fh, '>', $u_conf;
 $conf_fh->autoflush(1);
 my $u1 = "$tmpdir/u1";
 print $conf_fh <<EOM;
 early_hints true
 listen "$u1"
 EOM
-my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
+my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv });
 my $curl = which('curl');
 my $fifo = "$tmpdir/fifo";
 POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
diff --git a/t/lib.perl b/t/lib.perl
index 13e390d6..244972bc 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,20 +6,43 @@ use v5.14;
 use parent qw(Exporter);
 use autodie;
 use Test::More;
+use Time::HiRes qw(sleep);
 use IO::Socket::INET;
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
-our ($tmpdir, $errfh, $err_log);
+our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
+	$pid_file);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
-	$tmpdir $errfh $err_log
+	$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
 	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
-	do_req);
+	do_req stop_daemon);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
 $err_log = "$tmpdir/err.log";
+$pid_file = "$tmpdir/pid";
+$u_sock = "$tmpdir/u.sock";
+$u_conf = "$tmpdir/u.conf.rb";
 open($errfh, '>>', $err_log);
-END { diag slurp($err_log) if $tmpdir };
+
+sub stop_daemon (;$) {
+	my ($is_END) = @_;
+	kill('TERM', $daemon_pid);
+	my $tries = 1000;
+	while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) }
+	if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing
+		CORE::kill('KILL', $daemon_pid);
+		die "daemon_pid=$daemon_pid did not die";
+	} else {
+		ok(!CORE::kill(0, $daemon_pid), 'daemonized unicorn gone');
+		undef $daemon_pid;
+	}
+};
+
+END {
+	diag slurp($err_log) if $tmpdir;
+	stop_daemon(1) if defined $daemon_pid;
+};
 
 sub check_stderr () {
 	my @log = slurp($err_log);
diff --git a/t/working_directory.t b/t/working_directory.t
index 6c974720..f9254eb8 100644
--- a/t/working_directory.t
+++ b/t/working_directory.t
@@ -4,12 +4,10 @@
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
 mkdir "$tmpdir/alt";
-my $u_sock = "$tmpdir/u.sock";
 my $ru = "$tmpdir/alt/config.ru";
-my $u_conf = "$tmpdir/u.conf.rb";
 open my $fh, '>', $u_conf;
 print $fh <<EOM;
-pid "$tmpdir/pid"
+pid "$pid_file"
 preload_app true
 stderr_path "$err_log"
 working_directory "$tmpdir/alt" # the whole point of this test
@@ -30,32 +28,13 @@ $common_ru
 EOM
 close $fh;
 
-my $pid;
-my $stop_daemon = sub {
-	my ($is_END) = @_;
-	kill('TERM', $pid);
-	my $tries = 1000;
-	while (CORE::kill(0, $pid) && --$tries) {
-		select undef, undef, undef, 0.01;
-	}
-	if ($is_END && CORE::kill(0, $pid)) {
-		CORE::kill('KILL', $pid);
-		die "daemonized PID=$pid did not die";
-	} else {
-		ok(!CORE::kill(0, $pid), 'daemonized unicorn gone');
-		undef $pid;
-	}
-};
-
-END { $stop_daemon->(1) if defined $pid };
-
 unicorn('-c', $u_conf)->join; # will daemonize
-chomp($pid = slurp("$tmpdir/pid"));
+chomp($daemon_pid = slurp($pid_file));
 
 my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
 is($bdy, "1\n", 'got expected $master_ppid');
 
-$stop_daemon->();
+stop_daemon;
 check_stderr;
 
 if ('test without CLI switches in config.ru') {
@@ -65,12 +44,12 @@ if ('test without CLI switches in config.ru') {
 	close $fh;
 
 	unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize
-	chomp($pid = slurp("$tmpdir/pid"));
+	chomp($daemon_pid = slurp($pid_file));
 
 	($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0');
 	is($bdy, "1\n", 'got expected $master_ppid');
 
-	$stop_daemon->();
+	stop_daemon;
 	check_stderr;
 }
 

[-- Attachment #9: 0008-tests-use-Time-HiRes-sleep-and-time-everywhere.patch --]
[-- Type: text/x-diff, Size: 4106 bytes --]

From dd9f2efeebf20cfa1def0ce92cb4e35a8b5c1580 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 09:35:09 +0000
Subject: [PATCH 08/11] tests: use Time::HiRes `sleep' and `time' everywhere

The time(2) syscall use by CORE::time is inaccurate[1].
It's also easier to read `sleep 0.01' rather than the
longer `select' equivalent.

[1] a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01)
---
 t/active-unix-socket.t | 2 +-
 t/integration.t        | 5 +++--
 t/lib.perl             | 4 ++--
 t/reload-bad-config.t  | 2 +-
 t/reopen-logs.t        | 4 ++--
 t/winch_ttin.t         | 4 +---
 6 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 32cb0c2e..ff731b5f 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -86,7 +86,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 		'fail to connect to u1');
 	for (1..50) { # wait for init process to reap worker
 		kill(0, $worker_pid) or last;
-		select(undef, undef, undef, 0.011);
+		sleep 0.011;
 	}
 	ok(!kill(0, $worker_pid), 'worker gone after parent dies');
 }
diff --git a/t/integration.t b/t/integration.t
index eb40ffc7..80485e44 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -77,8 +77,9 @@ SKIP: { # Date header check
 	eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
 	$d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
 	my $t = HTTP::Date::str2time($d[0]);
-	ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
-		diag(explain([$t, $!, \@d]));
+	my $now = time;
+	ok($t >= ($t0 - 1) && $t > 0 && $t <= ($now + 1), 'valid date') or
+		diag(explain(["t=$t t0=$t0 now=$now", $!, \@d]));
 };
 
 
diff --git a/t/lib.perl b/t/lib.perl
index 244972bc..9254b23b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -6,7 +6,7 @@ use v5.14;
 use parent qw(Exporter);
 use autodie;
 use Test::More;
-use Time::HiRes qw(sleep);
+use Time::HiRes qw(sleep time);
 use IO::Socket::INET;
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
@@ -15,7 +15,7 @@ our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid,
 our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn
 	$tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file
 	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr
-	do_req stop_daemon);
+	do_req stop_daemon sleep time);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t
index 543421da..c023b88c 100644
--- a/t/reload-bad-config.t
+++ b/t/reload-bad-config.t
@@ -38,7 +38,7 @@ my @l;
 for (1..1000) {
 	@l = grep(/(?:done|error) reloading/, slurp($err_log)) and
 		last;
-	select undef, undef, undef, 0.011;
+	sleep 0.011;
 }
 diag slurp($err_log) if $ENV{V};
 ok(grep(/error reloading/, @l), 'got error reloading');
diff --git a/t/reopen-logs.t b/t/reopen-logs.t
index 8a58c1b9..76a4dbdf 100644
--- a/t/reopen-logs.t
+++ b/t/reopen-logs.t
@@ -23,8 +23,8 @@ rename($out_log, "$out_log.rot");
 $auto_reap->do_kill('USR1');
 
 my $tries = 1000;
-while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 };
-while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 };
+while (!-f $err_log && --$tries) { sleep 0.01 };
+while (!-f $out_log && --$tries) { sleep 0.01 };
 
 ok(-f $out_log, 'stdout_path recreated after USR1');
 ok(-f $err_log, 'stderr_path recreated after USR1');
diff --git a/t/winch_ttin.t b/t/winch_ttin.t
index 509b118f..c5079599 100644
--- a/t/winch_ttin.t
+++ b/t/winch_ttin.t
@@ -43,9 +43,7 @@ ok(kill(0, $worker_pid), 'worker_pid is valid');
 ok(kill('WINCH', $pid), 'SIGWINCH can be sent');
 
 my $tries = 1000;
-while (CORE::kill(0, $worker_pid) && --$tries) {
-	select undef, undef, undef, 0.01;
-}
+while (CORE::kill(0, $worker_pid) && --$tries) { sleep 0.01 }
 ok(!CORE::kill(0, $worker_pid), 'worker not running');
 
 ok(kill('TTIN', $pid), 'SIGTTIN to restart worker');

[-- Attachment #10: 0009-tests-fold-SO_KEEPALIVE-check-to-Perl-5-integration.patch --]
[-- Type: text/x-diff, Size: 2675 bytes --]

From b588ccbbf73547487f54fd1a9d5396d6848e8661 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:21:05 +0000
Subject: [PATCH 09/11] tests: fold SO_KEEPALIVE check to Perl 5 integration

No need to startup more processes than necessary.
---
 t/integration.t        | 13 +++++++++++++
 test/exec/test_exec.rb | 23 +----------------------
 2 files changed, 14 insertions(+), 22 deletions(-)

diff --git a/t/integration.t b/t/integration.t
index 80485e44..bea221ce 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,16 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
+use Socket qw(SOL_SOCKET SO_KEEPALIVE);
 our $srv = tcp_server();
 our $host_port = tcp_host_port($srv);
+
+if ('ensure Perl does not set SO_KEEPALIVE by default') {
+	my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+	unpack('i', $val) == 0 or
+		setsockopt($srv, SOL_SOCKET, SO_KEEPALIVE, pack('i', 0));
+	$val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+}
 my $t0 = time;
 open my $conf_fh, '>', $u_conf;
 $conf_fh->autoflush(1);
@@ -71,6 +79,11 @@ is_deeply([ grep(/^X-R2: /, @$hdr) ],
 	[ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
 	'rack 2 LF-delimited headers supported') or diag(explain($hdr));
 
+{
+	my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE);
+	is(unpack('i', $val), 1, 'SO_KEEPALIVE set on inherited socket');
+}
+
 SKIP: { # Date header check
 	my @d = grep(/^Date: /i, @$hdr);
 	is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 55f828e7..84944520 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -1,6 +1,5 @@
 # -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
+# Don't add to this file, new tests are in Perl 5. See t/README
 FLOCK_PATH = File.expand_path(__FILE__)
 require './test/test_helper'
 
@@ -97,26 +96,6 @@ def teardown
     end
   end
 
-  def test_inherit_listener_unspecified
-    File.open("config.ru", "wb") { |fp| fp.write(HI) }
-    sock = TCPServer.new(@addr, @port)
-    sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
-    pid = xfork do
-      redirect_test_io do
-        ENV['UNICORN_FD'] = sock.fileno.to_s
-        exec($unicorn_bin, sock.fileno => sock.fileno)
-      end
-    end
-    res = hit(["http://#@addr:#@port/"])
-    assert_equal [ "HI\n" ], res
-    assert_shutdown(pid)
-    assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
-                'unicorn should always set SO_KEEPALIVE on inherited sockets'
-  ensure
-    sock.close if sock
-  end
-
   def test_working_directory_rel_path_config_file
     other = Tempfile.new('unicorn.wd')
     File.unlink(other.path)

[-- Attachment #11: 0010-tests-move-broken-app-test-to-Perl-5-integration-tes.patch --]
[-- Type: text/x-diff, Size: 2376 bytes --]

From 7160f1b519aece0fe645d22a7d8fb954a43ad6fb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:37:32 +0000
Subject: [PATCH 10/11] tests: move broken app test to Perl 5 integration test

Less Ruby means fewer incompatibilities to worry about with
every new version.
---
 t/integration.ru         |  1 +
 t/integration.t          |  6 ++++++
 test/unit/test_server.rb | 14 --------------
 3 files changed, 7 insertions(+), 14 deletions(-)

diff --git a/t/integration.ru b/t/integration.ru
index 086126ab..888833a9 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -98,6 +98,7 @@ def rack_input_tests(env)
     when '/pid'; [ 200, {}, [ "#$$\n" ] ]
     when '/early_hints_rack2'; early_hints(env, "r\n2")
     when '/early_hints_rack3'; early_hints(env, %w(r 3))
+    when '/broken_app'; raise RuntimeError, 'hello'
     else '/'; [ 200, {}, [ env_dump(env) ] ]
     end # case PATH_INFO (GET)
   when 'POST'
diff --git a/t/integration.t b/t/integration.t
index bea221ce..ba17dd9e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -118,6 +118,12 @@ SKIP: {
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
+check_stderr;
+($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0');
+like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint');
+is($bdy, undef, 'no response body after exception');
+truncate($errfh, 0);
+
 my $ck_early_hints = sub {
 	my ($note) = @_;
 	$c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 0a710d12..2af12eac 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -127,20 +127,6 @@ def test_after_reply
     sock.close
   end
 
-  def test_broken_app
-    teardown
-    app = lambda { |env| raise RuntimeError, "hello" }
-    # [200, {}, []] }
-    redirect_test_io do
-      @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] )
-      @server.start
-    end
-    sock = tcp_socket('127.0.0.1', @port)
-    sock.syswrite("GET / HTTP/1.0\r\n\r\n")
-    assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096)
-    assert_nil sock.close
-  end
-
   def test_simple_server
     results = hit(["http://localhost:#{@port}/test"])
     assert_equal 'hello!\n', results[0], "Handler didn't really run"

[-- Attachment #12: 0011-tests-fold-early-shutdown-tests-into-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 4527 bytes --]

From 05028146b5e69c566663fdab9f8b92c6145a791a Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Sun, 10 Sep 2023 19:52:03 +0000
Subject: [PATCH 11/11] tests: fold early shutdown() tests into t/integration.t

This means fewer redundant tests and more chances to notice
Ruby incompatibilities.
---
 t/integration.t          | 22 +++++++++++++++--
 test/unit/test_server.rb | 53 ----------------------------------------
 2 files changed, 20 insertions(+), 55 deletions(-)

diff --git a/t/integration.t b/t/integration.t
index ba17dd9e..7310ff29 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,7 +7,7 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
-use Socket qw(SOL_SOCKET SO_KEEPALIVE);
+use Socket qw(SOL_SOCKET SO_KEEPALIVE SHUT_WR);
 our $srv = tcp_server();
 our $host_port = tcp_host_port($srv);
 
@@ -209,6 +209,7 @@ SKIP: {
 		defined($opt{overwrite}) and
 			print { $c } ('x' x $opt{overwrite});
 		$c->flush or die $!;
+		shutdown($c, SHUT_WR);
 		($status, $hdr) = slurp_hdr($c);
 		is(readline($c), $blob_hash, "$sub $path");
 	};
@@ -225,6 +226,8 @@ SKIP: {
 	# ensure small overwrites don't get checksummed
 	$ck_hash->('identity', '/rack_input', -s => $blob_size,
 			overwrite => 1); # one extra byte
+	unlike(slurp($err_log), qr/ClientShutdown/,
+		'no overreads after client SHUT_WR');
 
 	# excessive overwrite truncated
 	$c = tcp_start($srv);
@@ -238,8 +241,23 @@ SKIP: {
 		$! = 0;
 		while (print $c $buf and time < $end) { ++$n }
 		ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+		undef $c;
+	}
+
+	# client shutdown early
+	$c = tcp_start($srv);
+	$c->autoflush(0);
+	print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 16384\r\n\r\n";
+	if (1) {
+		local $SIG{PIPE} = 'IGNORE';
+		print $c 'too short body';
+		shutdown($c, SHUT_WR);
+		vec(my $rvec = '', fileno($c), 1) = 1;
+		select($rvec, undef, undef, 10) or BAIL_OUT "timed out";
+		my $buf = <$c>;
+		is($buf, undef, 'server aborted after client SHUT_WR');
+		undef $c;
 	}
-	undef $c;
 
 	$curl // skip 'no curl found in PATH', 1;
 
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 2af12eac..7ffa48f0 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -132,59 +132,6 @@ def test_simple_server
     assert_equal 'hello!\n', results[0], "Handler didn't really run"
   end
 
-  def test_client_shutdown_writes
-    bs = 15609315 * rand
-    sock = tcp_socket('127.0.0.1', @port)
-    sock.syswrite("PUT /hello HTTP/1.1\r\n")
-    sock.syswrite("Host: example.com\r\n")
-    sock.syswrite("Transfer-Encoding: chunked\r\n")
-    sock.syswrite("Trailer: X-Foo\r\n")
-    sock.syswrite("\r\n")
-    sock.syswrite("%x\r\n" % [ bs ])
-    sock.syswrite("F" * bs)
-    sock.syswrite("\r\n0\r\nX-")
-    "Foo: bar\r\n\r\n".each_byte do |x|
-      sock.syswrite x.chr
-      sleep 0.05
-    end
-    # we wrote the entire request before shutting down, server should
-    # continue to process our request and never hit EOFError on our sock
-    sock.shutdown(Socket::SHUT_WR)
-    buf = sock.read
-    assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
-    next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
-    assert_equal 'hello!\n', next_client
-    lines = File.readlines("test_stderr.#$$.log")
-    assert lines.grep(/^Unicorn::ClientShutdown: /).empty?
-    assert_nil sock.close
-  end
-
-  def test_client_shutdown_write_truncates
-    bs = 15609315 * rand
-    sock = tcp_socket('127.0.0.1', @port)
-    sock.syswrite("PUT /hello HTTP/1.1\r\n")
-    sock.syswrite("Host: example.com\r\n")
-    sock.syswrite("Transfer-Encoding: chunked\r\n")
-    sock.syswrite("Trailer: X-Foo\r\n")
-    sock.syswrite("\r\n")
-    sock.syswrite("%x\r\n" % [ bs ])
-    sock.syswrite("F" * (bs / 2.0))
-
-    # shutdown prematurely, this will force the server to abort
-    # processing on us even during app dispatch
-    sock.shutdown(Socket::SHUT_WR)
-    IO.select([sock], nil, nil, 60) or raise "Timed out"
-    buf = sock.read
-    assert_equal "", buf
-    next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
-    assert_equal 'hello!\n', next_client
-    lines = File.readlines("test_stderr.#$$.log")
-    lines = lines.grep(/^Unicorn::ClientShutdown: bytes_read=\d+/)
-    assert_equal 1, lines.size
-    assert_match %r{\AUnicorn::ClientShutdown: bytes_read=\d+ true$}, lines[0]
-    assert_nil sock.close
-  end
-
   def test_client_malformed_body
     bs = 15653984
     sock = tcp_socket('127.0.0.1', @port)

^ permalink raw reply related	[relevance 1%]

* [PATCH] add chunk_response config directive for compatibility
@ 2023-06-20 12:28  7% EW
  0 siblings, 0 replies; 200+ results
From: EW @ 2023-06-20 12:28 UTC (permalink / raw)
  To: unicorn-public

Since Rack::Chunked is gone in Rack 3.1+ and no longer loaded by
default, we've learned to chunk HTTP/1.1 responses w/o
Content-Length to allow clients to detect truncated responses.

Unfortunately, there exist broken HTTP clients which advertise
"HTTP/1.1" in the request but cannot parse chunked responses.
Thus, we must continue to send unchunked responses as we have in
prior releases if that's what clients expect.  That is, chunked
responses are opt-in unless RACK_ENV is "deployment" or
"development".

It doesn't matter if clients are in the wrong: they've worked
this way for 14 years and we must do everything in our power to
avoid breaking existing expectations on upgrades.

I hate adding config directives, but breaking changes are even
worse since users upgrading unicorn often have no easy way to
to fix broken clients even if they're on the same LAN.
---
  Of course, if users upgrading to unicorn could fix everything;
  they wouldn't need unicorn at all :P
  unicorn was created to support broken code and now exists to
  perpetuate broken code into eternity.

 Documentation/unicorn.1      |  1 +
 lib/unicorn.rb               |  2 +
 lib/unicorn/configurator.rb  | 17 ++++++-
 lib/unicorn/http_response.rb |  2 +-
 lib/unicorn/http_server.rb   |  3 +-
 t/integration.ru             |  7 +++
 t/integration.t              | 88 +++++++++++++++++++++++++++++++++---
 7 files changed, 110 insertions(+), 10 deletions(-)

diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1
index b2c5e70..502f44a 100644
--- a/Documentation/unicorn.1
+++ b/Documentation/unicorn.1
@@ -69,6 +69,7 @@ options.
 Disables loading middleware implied by RACK_ENV.  This bypasses the
 configuration documented in the RACK ENVIRONMENT section, but still
 allows RACK_ENV to be used for application/framework\-specific purposes.
+This also affects the "chunk_response" config file directive in unicorn 7.0+
 .RS
 .RE
 .SH RACKUP COMPATIBILITY OPTIONS
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b817b77..e7b2806 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -78,7 +78,9 @@ def self.builder(ru, op)
       # middlewares will need ContentLength middleware.
       case ENV["RACK_ENV"]
       when "development"
+        server.chunk_response = true if server.chunk_response.nil?
       when "deployment"
+        server.chunk_response = true if server.chunk_response.nil?
         middleware.delete(:ShowExceptions)
         middleware.delete(:Lint)
       else
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index ecdf03e..bbc0448 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -59,6 +59,7 @@ class Unicorn::Configurator
     :check_client_connection => false,
     :rewindable_input => true,
     :client_body_buffer_size => Unicorn::Const::MAX_BODY,
+    :chunk_response => nil,
   }
   #:startdoc:
 
@@ -129,6 +130,19 @@ def [](key) # :nodoc:
     set[key]
   end
 
+  # Whether or not to chunk eligible HTTP/1.1 responses.  This is
+  # necessary for Rack 3.1+ users where the Rack::Chunked middleware
+  # no longer exists.
+  # Default: +true+ if +default_middleware+ is +true+ AND
+  # RACK_ENV is either +development+ or +deployment+.
+  # It is +false+ otherwise to support broken clients advertising
+  # HTTP/1.1 but lacking the ability to parse chunked responses.
+  # +chunk_response+ only exists since unicorn 7.0+ (the first release
+  # with Rack 3.x support).
+  def chunk_response(bool)
+    set_bool(:chunk_response, bool)
+  end
+
   # sets object to the +obj+ Logger-like object.  The new Logger-like
   # object must respond to the following methods:
   # * debug
@@ -270,7 +284,8 @@ def worker_processes(nr)
   end
 
   # sets whether to add default middleware in the development and
-  # deployment RACK_ENVs.
+  # deployment RACK_ENVs.  This also affects the +chunk_response+
+  # directive in unicorn 7.0+
   #
   # default_middleware is only available in unicorn 5.5.0+
   def default_middleware(bool)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 0ed0ae3..c73efe9 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -68,7 +68,7 @@ def http_response_write(socket, status, headers, body,
           append_header(buf, key, value)
         end
       end
-      if !hijack && !term && req.chunkable_response?
+      if !hijack && !term && req.chunkable_response? && @chunk_response
         do_chunk = true
         buf << "Transfer-Encoding: chunked\r\n".freeze
       end
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index f1b4a54..93d7141 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user,
-                :default_middleware, :early_hints
+                :default_middleware, :early_hints, :chunk_response
   attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
@@ -69,6 +69,7 @@ class Unicorn::HttpServer
   # incoming requests on the socket.
   def initialize(app, options = {})
     @app = app
+    @chunk_response = nil
     @reexec_pid = 0
     @default_middleware = true
     options = options.dup
diff --git a/t/integration.ru b/t/integration.ru
index 086126a..d8a8178 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -85,6 +85,12 @@ def rack_input_tests(env)
   [ 200, h, [ dig.hexdigest ] ]
 end
 
+class NoArray
+  def each
+    yield "HI\n"
+  end
+end
+
 run(lambda do |env|
   case env['REQUEST_METHOD']
   when 'GET'
@@ -98,6 +104,7 @@ def rack_input_tests(env)
     when '/pid'; [ 200, {}, [ "#$$\n" ] ]
     when '/early_hints_rack2'; early_hints(env, "r\n2")
     when '/early_hints_rack3'; early_hints(env, %w(r 3))
+    when '/no-ary'; [ 200, {}, NoArray.new ]
     else '/'; [ 200, {}, [ env_dump(env) ] ]
     end # case PATH_INFO (GET)
   when 'POST'
diff --git a/t/integration.t b/t/integration.t
index bb2ab51..115a761 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -22,6 +22,13 @@ my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');
 my $fifo = "$tmpdir/fifo";
 POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+my $wait_fifo = sub {
+	open my $fifo_fh, '<', $fifo;
+	my $wpid = readline($fifo_fh);
+	like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+	$wpid;
+};
+
 my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
@@ -176,6 +183,78 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
 }
 
+my $tmp = { 3 => tcp_server() };
+my $no_ary = "GET /no-ary HTTP/1.1\r\nHost: example.com\r\n\r\n";
+if (diag('chunk_response is off by default w/ RACK_ENV=none') || 1) {
+	print { $c = tcp_start($srv) } $no_ary;
+	($status, $hdr) = slurp_hdr($c);
+	unlike("@$hdr", qr/Transfer-Encoding/i,
+		'no Transfer-Encoding for RACK_ENV=none despite HTTP/1.1');
+	local $/;
+	is(readline($c), "HI\n", 'unchunked body response');
+}
+
+# pretend we have Rack::Chunked for RACK_ENV=(deployment|development)
+for my $rack_env (qw(deployment development)) {
+	my $cfg = "$tmpdir/nochunk.conf.rb";
+	open my $fh, '>', $cfg;
+	my $u = unicorn('-E', $rack_env, qw(t/integration.ru -c), $cfg, $tmp);
+	$c = tcp_start($tmp->{3});
+	print $c $no_ary;
+	($status, $hdr) = slurp_hdr($c);
+	like("@$hdr", qr/Transfer-Encoding/i,
+		"Transfer-Encoding set by default for RACK_ENV=$rack_env");
+	is(do { local $/; readline($c) },
+		"3\r\nHI\n\r\n0\r\n\r\n", 'chunked body response');
+
+	print $fh <<EOM;
+chunk_response false
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+	close $fh;
+	$u->do_kill('HUP');
+	$wait_fifo->();
+	$c = tcp_start($tmp->{3});
+	print $c $no_ary;
+	($status, $hdr) = slurp_hdr($c);
+	unlike("@$hdr", qr/Transfer-Encoding/i,
+			"RACK_ENV=$rack_env w/o chunk_response");
+	is(do { local $/; readline($c) },
+		"HI\n", 'unchunked body response');
+}
+
+if (diag('chunk_response true w/ RACK_ENV=none') || 1) {
+	my $cfg = "$tmpdir/chunk.conf.rb";
+	open my $fh, '>', $cfg;
+	print $fh "chunk_response true\n";
+	close $fh;
+	my $u = unicorn(qw(-E none t/integration.ru -c), $cfg, $tmp);
+	$c = tcp_start($tmp->{3});
+	print $c $no_ary;
+	($status, $hdr) = slurp_hdr($c);
+	like("@$hdr", qr/Transfer-Encoding/i,
+		"Transfer-Encoding set by chunk_response false");
+	is(do { local $/; readline($c) },
+		"3\r\nHI\n\r\n0\r\n\r\n", 'chunked body response');
+
+	# reset to default:
+	open $fh, '>', $cfg;
+	print $fh <<EOM;
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+	close $fh;
+	$u->do_kill('HUP');
+	$wait_fifo->();
+
+	$c = tcp_start($tmp->{3});
+	print $c $no_ary;
+	($status, $hdr) = slurp_hdr($c);
+	unlike("@$hdr", qr/Transfer-Encoding/i,
+			'chunk_response false after HUP reset');
+	is(do { local $/; readline($c) },
+		"HI\n", 'unchunked body response after HUP reset');
+}
+
 # input tests
 my ($blob_size, $blob_hash);
 SKIP: {
@@ -287,9 +366,7 @@ check_client_connection true
 after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
 EOM
 	$ar->do_kill('HUP');
-	open my $fifo_fh, '<', $fifo;
-	my $wpid = readline($fifo_fh);
-	like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+	$wait_fifo->();
 	$ck_early_hints->('ccc on');
 }
 
@@ -301,10 +378,7 @@ if ('max_header_len internal API') {
 Unicorn::HttpParser.max_header_len = $len
 EOM
 	$ar->do_kill('HUP');
-	open my $fifo_fh, '<', $fifo;
-	my $wpid = readline($fifo_fh);
-	like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
-	close $fifo_fh;
+	my $wpid = $wait_fifo->();
 	$wpid =~ s/\Apid=// or die;
 	ok(CORE::kill(0, $wpid), 'worker PID retrieved');
 

^ permalink raw reply related	[relevance 7%]

* [PATCH] unicorn_http_common.rl: use only ASCII spaces for compatibility
@ 2023-06-20 10:46  4% EW
  0 siblings, 0 replies; 200+ results
From: EW @ 2023-06-20 10:46 UTC (permalink / raw)
  To: unicorn-public

Ragel 6.10 on FreeBSD 12.4 amd64 complains and fails on this, yet the same
Ragel version on Debian 11.x i386 and amd64 never has.  I suspect this can
fix compatibility on s390x, arm64, armel, and armhf Debian builds:

https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=s390x&ver=6.1.0-1&stamp=1687156375&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=arm64&ver=6.1.0-1&stamp=1687156478&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armel&ver=6.1.0-1&stamp=1687156619&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armhf&ver=6.1.0-1&stamp=1687156807&file=log

Fixes: d5fbbf547203061b (Add some tolerance (RFC2616 sec. 19.3), 2016-10-20)
---
 ext/unicorn_http/unicorn_http_common.rl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ext/unicorn_http/unicorn_http_common.rl b/ext/unicorn_http/unicorn_http_common.rl
index 0988b54..507e570 100644
--- a/ext/unicorn_http/unicorn_http_common.rl
+++ b/ext/unicorn_http/unicorn_http_common.rl
@@ -4,7 +4,7 @@
 
 #### HTTP PROTOCOL GRAMMAR
 # line endings
-  CRLF = ("\r\n" | "\n");
+  CRLF = ("\r\n" | "\n");
 
 # character types
   CTL = (cntrl | 127);

^ permalink raw reply related	[relevance 4%]

* [PATCH 00-23/23] start porting tests to Perl5
@ 2023-06-05 10:32  1% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-05 10:32 UTC (permalink / raw)
  To: unicorn-public

[-- Attachment #1: Type: text/plain, Size: 4179 bytes --]

Still a lot more work to do, but at least socat is no longer a
test dependency.  Perl5 is installed on far more systems than
socat.

Ruby introduces breaking changes every year and I can't trust
tests to work as they were originally intended, anymore.
Perl 5 doesn't have perfect backwards compatibility, either; but
it's the least bad of any widely-installed scripting language.

Note: that 23/23 introduces a subtle bugfix which changes
behavior for systemd users

Patches are attached to reduce load on SMTP servers.
Some more patches to come as I deal with Ruby 3.x deprecation
warnings :<

Eric Wong (23):
  switch unit/test_response.rb to Perl 5 integration test
  support rack 3 multi-value headers
  port t0018-write-on-close.sh to Perl 5
  port t0000-http-basic.sh to Perl 5
  port t0002-parser-error.sh to Perl 5
  t/integration.t: use start_req to simplify test slighly
  port t0011-active-unix-socket.sh to Perl 5
  port t0100-rack-input-tests.sh to Perl 5
  tests: use autodie to simplify error checking
  port t0019-max_header_len.sh to Perl 5
  test_exec: drop sd_listen_fds emulation test
  test_exec: drop test_basic and test_config_ru_alt_path
  tests: check_stderr consistently in Perl 5 tests
  tests: consistent tcp_start and unix_start across Perl 5 tests
  port t9000-preread-input.sh to Perl 5
  port t/t0116-client_body_buffer_size.sh to Perl 5
  tests: get rid of sha1sum.rb and rsha1() sh function
  early_hints supports Rack 3 array headers
  test_server: drop early_hints test
  t/integration.t: switch PUT tests to MD5, reuse buffers
  tests: move test_upload.rb tests to t/integration.t
  drop redundant IO#close_on_exec=false calls
  LISTEN_FDS-inherited sockets are immortal across SIGHUP

 GNUmakefile                                |   7 +-
 lib/unicorn/http_server.rb                 |  12 +-
 t/README                                   |  21 +-
 t/active-unix-socket.t                     | 113 +++++++
 t/bin/content-md5-put                      |  36 ---
 t/bin/sha1sum.rb                           |  17 --
 t/{t0116.ru => client_body_buffer_size.ru} |   2 -
 t/client_body_buffer_size.t                |  82 ++++++
 t/integration.ru                           | 114 +++++++
 t/integration.t                            | 326 +++++++++++++++++++++
 t/lib.perl                                 | 217 ++++++++++++++
 t/preread_input.ru                         |  21 +-
 t/rack-input-tests.ru                      |  21 --
 t/t0000-http-basic.sh                      |  50 ----
 t/t0002-parser-error.sh                    |  94 ------
 t/t0011-active-unix-socket.sh              |  79 -----
 t/t0018-write-on-close.sh                  |  23 --
 t/t0019-max_header_len.sh                  |  49 ----
 t/t0100-rack-input-tests.sh                | 124 --------
 t/t0116-client_body_buffer_size.sh         |  80 -----
 t/t9000-preread-input.sh                   |  48 ---
 t/test-lib.sh                              |   4 -
 t/write-on-close.ru                        |  11 -
 test/exec/test_exec.rb                     |  57 ----
 test/unit/test_response.rb                 | 111 -------
 test/unit/test_server.rb                   |  31 --
 test/unit/test_upload.rb                   | 301 -------------------
 27 files changed, 891 insertions(+), 1160 deletions(-)
 create mode 100644 t/active-unix-socket.t
 delete mode 100755 t/bin/content-md5-put
 delete mode 100755 t/bin/sha1sum.rb
 rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
 create mode 100644 t/client_body_buffer_size.t
 create mode 100644 t/integration.ru
 create mode 100644 t/integration.t
 create mode 100644 t/lib.perl
 delete mode 100644 t/rack-input-tests.ru
 delete mode 100755 t/t0000-http-basic.sh
 delete mode 100755 t/t0002-parser-error.sh
 delete mode 100755 t/t0011-active-unix-socket.sh
 delete mode 100755 t/t0018-write-on-close.sh
 delete mode 100755 t/t0019-max_header_len.sh
 delete mode 100755 t/t0100-rack-input-tests.sh
 delete mode 100755 t/t0116-client_body_buffer_size.sh
 delete mode 100755 t/t9000-preread-input.sh
 delete mode 100644 t/write-on-close.ru
 delete mode 100644 test/unit/test_response.rb
 delete mode 100644 test/unit/test_upload.rb

[-- Attachment #2: 0001-switch-unit-test_response.rb-to-Perl-5-integration-t.patch --]
[-- Type: text/x-diff, Size: 15667 bytes --]

From 086e397abc0126556af24df77a976671294df2ee Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:30 +0000
Subject: [PATCH 01/23] switch unit/test_response.rb to Perl 5 integration test

http_response_write may benefit from API changes for Rack 3
support.

Since there's no benefit I can see from using a unit test,
switch to an integration test to avoid having to maintain the
unit test if our internal http_response_write method changes.

Of course, I can't trust tests written in Ruby since I've had to
put up with a constant stream of incompatibilities over the past
two decades :<   Perl is more widely installed than socat[1], and
nearly all the Perl I wrote 20 years ago still works
unmodified today.

[1] the rarest dependency of the Bourne shell integration tests
---
 GNUmakefile                |   5 +-
 t/README                   |  24 +++--
 t/integration.ru           |  38 ++++++++
 t/integration.t            |  64 +++++++++++++
 t/lib.perl                 | 189 +++++++++++++++++++++++++++++++++++++
 test/unit/test_response.rb | 111 ----------------------
 6 files changed, 313 insertions(+), 118 deletions(-)
 create mode 100644 t/integration.ru
 create mode 100644 t/integration.t
 create mode 100644 t/lib.perl
 delete mode 100644 test/unit/test_response.rb

diff --git a/GNUmakefile b/GNUmakefile
index 0e08ef0..5cca189 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -86,7 +86,7 @@ $(tmp_bin)/%: bin/% | $(tmp_bin)
 bins: $(tmp_bins)
 
 t_log := $(T_log) $(T_n_log)
-test: $(T) $(T_n)
+test: $(T) $(T_n) test-prove
 	@cat $(t_log) | $(MRI) test/aggregate.rb
 	@$(RM) $(t_log)
 
@@ -141,6 +141,9 @@ t/random_blob:
 
 test-integration: $(T_sh)
 
+test-prove:
+	prove -vw
+
 check: test-require test test-integration
 test-all: check
 
diff --git a/t/README b/t/README
index 14de559..8a5243e 100644
--- a/t/README
+++ b/t/README
@@ -5,16 +5,24 @@ TCP ports or Unix domain sockets.  They're all designed to run
 concurrently with other tests to minimize test time, but tests may be
 run independently as well.
 
-We write our tests in Bourne shell because that's what we're
-comfortable writing integration tests with.
+New tests are written in Perl 5 because we need a stable language
+to test real-world behavior and Ruby introduces incompatibilities
+at a far faster rate than Perl 5.  Perl is Ruby's older cousin, so
+it should be easy-to-learn for Rubyists.
+
+Old tests are in Bourne shell, but the socat(1) dependency was probably
+too rare compared to Perl 5.
 
 == Requirements
 
-* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!)
+* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
+* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
 * {GNU make}[https://www.gnu.org/software/make/]
+
+The following requirements will eventually be dropped.
+
 * {socat}[http://www.dest-unreach.org/socat/]
 * {curl}[https://curl.haxx.se/]
-* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...)
 
 We do not use bashisms or any non-portable, non-POSIX constructs
 in our shell code.  We use the "pipefail" option if available and
@@ -26,9 +34,13 @@ with {dash}[http://gondor.apana.org.au/~herbert/dash/] and
 
 To run the entire test suite with 8 tests running at once:
 
-  make -j8
+  make -j8 && prove -vw
+
+To run one individual test (Perl5):
+
+  prove -vw t/integration.t
 
-To run one individual test:
+To run one individual test (shell):
 
   make t0000-simple-http.sh
 
diff --git a/t/integration.ru b/t/integration.ru
new file mode 100644
index 0000000..6ef873c
--- /dev/null
+++ b/t/integration.ru
@@ -0,0 +1,38 @@
+#!ruby
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+# this goes for t/integration.t  We'll try to put as many tests
+# in here as possible to avoid startup overhead of Ruby.
+
+$orig_rack_200 = nil
+def tweak_status_code
+  $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
+  Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
+  [ 200, {}, [] ]
+end
+
+def restore_status_code
+  $orig_rack_200 or return [ 500, {}, [] ]
+  Rack::Utils::HTTP_STATUS_CODES[200] = $orig_rack_200
+  [ 200, {}, [] ]
+end
+
+run(lambda do |env|
+  case env['REQUEST_METHOD']
+  when 'GET'
+    case env['PATH_INFO']
+    when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+    when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
+    when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+    end # case PATH_INFO (GET)
+  when 'POST'
+    case env['PATH_INFO']
+    when '/tweak-status-code'; tweak_status_code
+    when '/restore-status-code'; restore_status_code
+    end # case PATH_INFO (POST)
+    # ...
+  when 'PUT'
+    # ...
+  end # case REQUEST_METHOD
+end) # run
diff --git a/t/integration.t b/t/integration.t
new file mode 100644
index 0000000..5569155
--- /dev/null
+++ b/t/integration.t
@@ -0,0 +1,64 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+my $srv = tcp_server();
+my $t0 = time;
+my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+
+sub slurp_hdr {
+	my ($c) = @_;
+	local $/ = "\r\n\r\n"; # affects both readline+chomp
+	chomp(my $hdr = readline($c));
+	my ($status, @hdr) = split(/\r\n/, $hdr);
+	diag explain([ $status, \@hdr ]) if $ENV{V};
+	($status, \@hdr);
+}
+
+my ($c, $status, $hdr);
+
+# response header tests
+$c = tcp_connect($srv);
+print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+my $orig_200_status = $status;
+is_deeply([ grep(/^X-R2: /, @$hdr) ],
+	[ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
+	'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+
+SKIP: { # Date header check
+	my @d = grep(/^Date: /i, @$hdr);
+	is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
+	eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
+	$d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
+	my $t = HTTP::Date::str2time($d[0]);
+	ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
+		diag(explain([$t, $!, \@d]));
+};
+
+# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
+$c = tcp_connect($srv);
+print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
+	'nil header value accepted for broken apps') or diag(explain($hdr));
+
+if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
+	$c = tcp_connect($srv);
+	print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
+
+	$c = tcp_connect($srv);
+	print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	is($status, $orig_200_status, 'original status restored');
+}
+
+
+# ... more stuff here
+undef $ar;
+diag slurp("$tmpdir/err.log") if $ENV{V};
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
new file mode 100644
index 0000000..dd9c6b7
--- /dev/null
+++ b/t/lib.perl
@@ -0,0 +1,189 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+package UnicornTest;
+use v5.14;
+use parent qw(Exporter);
+use Test::More;
+use IO::Socket::INET;
+use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
+use File::Temp 0.19 (); # 0.19 for ->newdir
+our ($tmpdir, $errfh);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
+	SEEK_SET);
+
+my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
+$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+
+sub tcp_server {
+	my %opt = (
+		ReuseAddr => 1,
+		Proto => 'tcp',
+		Type => SOCK_STREAM,
+		Listen => SOMAXCONN,
+		Blocking => 0,
+		@_,
+	);
+	eval {
+		die 'IPv4-only' if $ENV{TEST_IPV4_ONLY};
+		require IO::Socket::INET6;
+		IO::Socket::INET6->new(%opt, LocalAddr => '[::1]')
+	} || eval {
+		die 'IPv6-only' if $ENV{TEST_IPV6_ONLY};
+		IO::Socket::INET->new(%opt, LocalAddr => '127.0.0.1')
+	} || BAIL_OUT "failed to create TCP server: $! ($@)";
+}
+
+sub tcp_host_port {
+	my ($s) = @_;
+	my ($h, $p) = ($s->sockhost, $s->sockport);
+	my $ipv4 = $s->sockdomain == AF_INET;
+	if (wantarray) {
+		$ipv4 ? ($h, $p) : ("[$h]", $p);
+	} else {
+		$ipv4 ? "$h:$p" : "[$h]:$p";
+	}
+}
+
+sub tcp_connect {
+	my ($dest, %opt) = @_;
+	my $addr = tcp_host_port($dest);
+	my $s = ref($dest)->new(
+		Proto => 'tcp',
+		Type => SOCK_STREAM,
+		PeerAddr => $addr,
+		%opt,
+	) or BAIL_OUT "failed to connect to $addr: $!";
+	$s->autoflush(1);
+	$s;
+}
+
+sub slurp {
+	open my $fh, '<', $_[0] or die "open($_[0]): $!";
+	local $/;
+	<$fh>;
+}
+
+sub spawn {
+	my $env = ref($_[0]) eq 'HASH' ? shift : undef;
+	my $opt = ref($_[-1]) eq 'HASH' ? pop : {};
+	my @cmd = @_;
+	my $old = POSIX::SigSet->new;
+	my $set = POSIX::SigSet->new;
+	$set->fillset or die "sigfillset: $!";
+	sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
+	pipe(my ($r, $w)) or die "pipe: $!";
+	my $pid = fork // die "fork: $!";
+	if ($pid == 0) {
+		close $r;
+		$SIG{__DIE__} = sub {
+			warn(@_);
+			syswrite($w, my $num = $! + 0);
+			_exit(1);
+		};
+
+		# pretend to be systemd (cf. sd_listen_fds(3))
+		my $cfd;
+		for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
+			my $io = $opt->{$cfd} // next;
+			my $pfd = fileno($io) // die "fileno($io): $!";
+			if ($pfd == $cfd) {
+				fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+			} else {
+				dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
+			}
+		}
+		if (($cfd - 3) > 0) {
+			$env->{LISTEN_PID} = $$;
+			$env->{LISTEN_FDS} = $cfd - 3;
+		}
+
+		if (defined(my $pgid = $opt->{pgid})) {
+			setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
+		}
+		$SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
+		if (defined(my $cd = $opt->{-C})) {
+			chdir $cd // die "chdir($cd): $!";
+		}
+		$old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
+		sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
+		@ENV{keys %$env} = values(%$env) if $env;
+		exec { $cmd[0] } @cmd;
+		die "exec @cmd: $!";
+	}
+	close $w;
+	sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK(old): $!";
+	if (my $cerrnum = do { local $/, <$r> }) {
+		$! = $cerrnum;
+		die "@cmd PID=$pid died: $!";
+	}
+	$pid;
+}
+
+sub which {
+	my ($file) = @_;
+	return $file if index($file, '/') >= 0;
+	for my $p (split(/:/, $ENV{PATH})) {
+		$p .= "/$file";
+		return $p if -x $p;
+	}
+	undef;
+}
+
+# returns an AutoReap object
+sub unicorn {
+	my %env;
+	if (ref($_[0]) eq 'HASH') {
+		my $e = shift;
+		%env = %$e;
+	}
+	my @args = @_;
+	push(@args, {}) if ref($args[-1]) ne 'HASH';
+	$args[-1]->{2} //= $errfh; # stderr default
+
+	state $ruby = which($ENV{RUBY} // 'ruby');
+	state $lib = File::Spec->rel2abs('lib');
+	state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
+	state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
+	state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
+	state $exe = File::Spec->rel2abs('bin/unicorn');
+	my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
+	UnicornTest::AutoReap->new($pid);
+}
+
+# automatically kill + reap children when this goes out-of-scope
+package UnicornTest::AutoReap;
+use v5.14;
+
+sub new {
+	my (undef, $pid) = @_;
+	bless { pid => $pid, owner => $$ }, __PACKAGE__
+}
+
+sub kill {
+	my ($self, $sig) = @_;
+	CORE::kill($sig // 'TERM', $self->{pid});
+}
+
+sub join {
+	my ($self, $sig) = @_;
+	my $pid = delete $self->{pid} or return;
+	CORE::kill($sig, $pid) if defined $sig;
+	my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+	$ret == $pid or die "BUG: waitpid($pid) != $ret";
+}
+
+sub DESTROY {
+	my ($self) = @_;
+	return if $self->{owner} != $$;
+	$self->join('TERM');
+}
+
+package main; # inject ourselves into the t/*.t script
+UnicornTest->import;
+Test::More->import;
+# try to ensure ->DESTROY fires:
+$SIG{TERM} = sub { exit(15 + 128) };
+$SIG{INT} = sub { exit(2 + 128) };
+1;
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
deleted file mode 100644
index fbe433f..0000000
--- a/test/unit/test_response.rb
+++ /dev/null
@@ -1,111 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2005 Zed A. Shaw
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-#
-# Additional work donated by contributors.  See git history
-# for more information.
-
-require './test/test_helper'
-require 'time'
-
-include Unicorn
-
-class ResponseTest < Test::Unit::TestCase
-  include Unicorn::HttpResponse
-
-  def test_httpdate
-    before = Time.now.to_i - 1
-    str = httpdate
-    assert_kind_of(String, str)
-    middle = Time.parse(str).to_i
-    after = Time.now.to_i
-    assert before <= middle
-    assert middle <= after
-  end
-
-  def test_response_headers
-    out = StringIO.new
-    http_response_write(out, 200, {"X-Whatever" => "stuff"}, ["cool"])
-    assert ! out.closed?
-
-    assert out.length > 0, "output didn't have data"
-  end
-
-  # ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-  def test_response_header_broken_nil
-    out = StringIO.new
-    http_response_write(out, 200, {"Nil" => nil}, %w(hysterical raisin))
-    assert ! out.closed?
-
-    assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted'
-  end
-
-  def test_response_string_status
-    out = StringIO.new
-    http_response_write(out,'200', {}, [])
-    assert ! out.closed?
-    assert out.length > 0, "output didn't have data"
-  end
-
-  def test_response_200
-    io = StringIO.new
-    http_response_write(io, 200, {}, [])
-    assert ! io.closed?
-    assert io.length > 0, "output didn't have data"
-  end
-
-  def test_response_with_default_reason
-    code = 400
-    io = StringIO.new
-    http_response_write(io, code, {}, [])
-    assert ! io.closed?
-    lines = io.string.split(/\r\n/)
-    assert_match(/.* Bad Request$/, lines.first,
-                 "wrong default reason phrase")
-  end
-
-  def test_rack_multivalue_headers
-    out = StringIO.new
-    http_response_write(out,200, {"X-Whatever" => "stuff\nbleh"}, [])
-    assert ! out.closed?
-    assert_match(/^X-Whatever: stuff\r\nX-Whatever: bleh\r\n/, out.string)
-  end
-
-  # Even though Rack explicitly forbids "Status" in the header hash,
-  # some broken clients still rely on it
-  def test_status_header_added
-    out = StringIO.new
-    http_response_write(out,200, {"X-Whatever" => "stuff"}, [])
-    assert ! out.closed?
-  end
-
-  def test_unknown_status_pass_through
-    out = StringIO.new
-    http_response_write(out,"666 I AM THE BEAST", {}, [] )
-    assert ! out.closed?
-    headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/)
-    assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0])
-  end
-
-  def test_modified_rack_http_status_codes_late
-    r, w = IO.pipe
-    pid = fork do
-      r.close
-      # Users may want to globally override the status text associated
-      # with an HTTP status code in their app.
-      Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
-      http_response_write(w, 200, {}, [])
-      w.close
-    end
-    w.close
-    assert_equal "HTTP/1.1 200 HI\r\n", r.gets
-    r.read # just drain the pipe
-    pid, status = Process.waitpid2(pid)
-    assert status.success?, status.inspect
-  ensure
-    r.close
-    w.close unless w.closed?
-  end
-end

[-- Attachment #3: 0002-support-rack-3-multi-value-headers.patch --]
[-- Type: text/x-diff, Size: 1710 bytes --]

From ea0559c700fa029044464de4bd572662c10b7273 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:31 +0000
Subject: [PATCH 02/23] support rack 3 multi-value headers

The first step in adding Rack 3 support.  Rack supports
multi-value headers via array rather than newlines.

Tested-by: Martin Posthumus <martin.posthumus@gmail.com>
Link: https://yhbt.net/unicorn-public/7c851d8a-bc57-7df8-3240-2f5ab831c47c@gmail.com/
---
 t/integration.ru | 1 +
 t/integration.t  | 9 +++++++++
 2 files changed, 10 insertions(+)

diff --git a/t/integration.ru b/t/integration.ru
index 6ef873c..5183217 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -23,6 +23,7 @@ def restore_status_code
   when 'GET'
     case env['PATH_INFO']
     when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+    when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
     when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
     end # case PATH_INFO (GET)
diff --git a/t/integration.t b/t/integration.t
index 5569155..e876c71 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -38,6 +38,15 @@ SKIP: { # Date header check
 		diag(explain([$t, $!, \@d]));
 };
 
+
+$c = tcp_connect($srv);
+print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([ grep(/^x-r3: /, @$hdr) ],
+	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
+	'rack 3 array headers supported') or diag(explain($hdr));
+
+
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
 $c = tcp_connect($srv);
 print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;

[-- Attachment #4: 0003-port-t0018-write-on-close.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4091 bytes --]

From 295a6c616f8840bc04617a377c04c3422aeebddc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:32 +0000
Subject: [PATCH 03/23] port t0018-write-on-close.sh to Perl 5

This doesn't require restarting, so it's a perfect candidate.
---
 t/integration.ru          | 15 +++++++++++++++
 t/integration.t           | 14 +++++++++++++-
 t/lib.perl                |  2 +-
 t/t0018-write-on-close.sh | 23 -----------------------
 t/write-on-close.ru       | 11 -----------
 5 files changed, 29 insertions(+), 36 deletions(-)
 delete mode 100755 t/t0018-write-on-close.sh
 delete mode 100644 t/write-on-close.ru

diff --git a/t/integration.ru b/t/integration.ru
index 5183217..12f5d48 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -18,6 +18,20 @@ def restore_status_code
   [ 200, {}, [] ]
 end
 
+class WriteOnClose
+  def each(&block)
+    @callback = block
+  end
+
+  def close
+    @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
+  end
+end
+
+def write_on_close
+  [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
+end
+
 run(lambda do |env|
   case env['REQUEST_METHOD']
   when 'GET'
@@ -26,6 +40,7 @@ def restore_status_code
     when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
     when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+    when '/write_on_close'; write_on_close
     end # case PATH_INFO (GET)
   when 'POST'
     case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index e876c71..3ab5c90 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -4,6 +4,7 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
 my $t0 = time;
 my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
 
@@ -66,8 +67,19 @@ if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
 	is($status, $orig_200_status, 'original status restored');
 }
 
+SKIP: {
+	eval { require HTTP::Tiny } or skip "HTTP::Tiny missing: $@", 1;
+	my $ht = HTTP::Tiny->new;
+	my $res = $ht->get("http://$host_port/write_on_close");
+	is($res->{content}, 'Goodbye', 'write-on-close body read');
+}
 
 # ... more stuff here
 undef $ar;
-diag slurp("$tmpdir/err.log") if $ENV{V};
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+is_deeply(\@err, [], 'no unexpected errors in stderr');
+is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
 done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index dd9c6b7..12deaf8 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET);
+	SEEK_SET tcp_host_port);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/t0018-write-on-close.sh b/t/t0018-write-on-close.sh
deleted file mode 100755
index 3afefea..0000000
--- a/t/t0018-write-on-close.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 4 "write-on-close tests for funky response-bodies"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	unicorn -D -c $unicorn_config write-on-close.ru
-	unicorn_wait_start
-}
-
-t_begin "write-on-close response body succeeds" && {
-	test xGoodbye = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
-	check_stderr
-}
-
-t_done
diff --git a/t/write-on-close.ru b/t/write-on-close.ru
deleted file mode 100644
index 725c4d6..0000000
--- a/t/write-on-close.ru
+++ /dev/null
@@ -1,11 +0,0 @@
-class WriteOnClose
-  def each(&block)
-    @callback = block
-  end
-
-  def close
-    @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
-  end
-end
-use Rack::ContentType, "text/plain"
-run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] })

[-- Attachment #5: 0004-port-t0000-http-basic.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3372 bytes --]

From 1bb4362cee167ac7aeec910d3f52419e391f1e61 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:33 +0000
Subject: [PATCH 04/23] port t0000-http-basic.sh to Perl 5

One more socat dependency down...
---
 t/integration.ru      | 16 ++++++++++++++
 t/integration.t       | 11 ++++++++++
 t/t0000-http-basic.sh | 50 -------------------------------------------
 3 files changed, 27 insertions(+), 50 deletions(-)
 delete mode 100755 t/t0000-http-basic.sh

diff --git a/t/integration.ru b/t/integration.ru
index 12f5d48..c0bef99 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -32,6 +32,21 @@ def write_on_close
   [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
 end
 
+def env_dump(env)
+  require 'json'
+  h = {}
+  env.each do |k,v|
+    case v
+    when String, Integer, true, false; h[k] = v
+    else
+      case k
+      when 'rack.version', 'rack.after_reply'; h[k] = v
+      end
+    end
+  end
+  h.to_json
+end
+
 run(lambda do |env|
   case env['REQUEST_METHOD']
   when 'GET'
@@ -40,6 +55,7 @@ def write_on_close
     when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
     when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+    when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
     end # case PATH_INFO (GET)
   when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3ab5c90..ee22e7e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -47,6 +47,17 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
 	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
 	'rack 3 array headers supported') or diag(explain($hdr));
 
+SKIP: {
+	eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
+	$c = tcp_connect($srv);
+	print $c "GET /env_dump\r\n" or die $!;
+	my $json = do { local $/; readline($c) };
+	unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
+	unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
+	my $env = JSON::PP->new->decode($json);
+	is(ref($env), 'HASH', 'JSON decoded body to hashref');
+	is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+}
 
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
 $c = tcp_connect($srv);
diff --git a/t/t0000-http-basic.sh b/t/t0000-http-basic.sh
deleted file mode 100755
index 8ab58ac..0000000
--- a/t/t0000-http-basic.sh
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 8 "simple HTTP connection tests"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	unicorn -D -c $unicorn_config env.ru
-	unicorn_wait_start
-}
-
-t_begin "single request" && {
-	curl -sSfv http://$listen/
-}
-
-t_begin "check stderr has no errors" && {
-	check_stderr
-}
-
-t_begin "HTTP/0.9 request should not return headers" && {
-	(
-		printf 'GET /\r\n'
-		cat $fifo > $tmp &
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-}
-
-t_begin "env.inspect should've put everything on one line" && {
-	test 1 -eq $(count_lines < $tmp)
-}
-
-t_begin "no headers in output" && {
-	if grep ^Connection: $tmp
-	then
-		die "Connection header found in $tmp"
-	elif grep ^HTTP/ $tmp
-	then
-		die "HTTP/ found in $tmp"
-	fi
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr has no errors" && {
-	check_stderr
-}
-
-t_done

[-- Attachment #6: 0005-port-t0002-parser-error.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4875 bytes --]

From 2eb7b1662c291ab535ee5dabf5d96194ca6483d4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:34 +0000
Subject: [PATCH 05/23] port t0002-parser-error.sh to Perl 5

Another socat dependency down...
---
 t/integration.t         | 33 +++++++++++++++
 t/lib.perl              |  9 +++-
 t/t0002-parser-error.sh | 94 -----------------------------------------
 3 files changed, 41 insertions(+), 95 deletions(-)
 delete mode 100755 t/t0002-parser-error.sh

diff --git a/t/integration.t b/t/integration.t
index ee22e7e..503b7eb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -85,6 +85,39 @@ SKIP: {
 	is($res->{content}, 'Goodbye', 'write-on-close body read');
 }
 
+if ('bad requests') {
+	$c = start_req($srv, 'GET /env_dump HTTP/1/1');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+
+	$c = tcp_connect($srv);
+	print $c 'GET /' or die $!;
+	my $buf = join('', (0..9), 'ab');
+	for (0..1023) { print $c $buf or die $! }
+	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 414 \b!,
+		'414 on REQUEST_PATH > (12 * 1024)');
+
+	$c = tcp_connect($srv);
+	print $c 'GET /hello-world?a' or die $!;
+	$buf = join('', (0..9));
+	for (0..1023) { print $c $buf or die $! }
+	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 414 \b!,
+		'414 on QUERY_STRING > (10 * 1024)');
+
+	$c = tcp_connect($srv);
+	print $c 'GET /hello-world#a' or die $!;
+	$buf = join('', (0..9), 'a'..'f');
+	for (0..63) { print $c $buf or die $! }
+	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
+}
+
+
 # ... more stuff here
 undef $ar;
 my @log = slurp("$tmpdir/err.log");
diff --git a/t/lib.perl b/t/lib.perl
index 12deaf8..7d712b5 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port);
+	SEEK_SET tcp_host_port start_req);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -59,6 +59,13 @@ sub tcp_connect {
 	$s;
 }
 
+sub start_req {
+	my ($srv, @req) = @_;
+	my $c = tcp_connect($srv);
+	print $c @req, "\r\n\r\n" or die "print: $!";
+	$c;
+}
+
 sub slurp {
 	open my $fh, '<', $_[0] or die "open($_[0]): $!";
 	local $/;
diff --git a/t/t0002-parser-error.sh b/t/t0002-parser-error.sh
deleted file mode 100755
index 9dc1cd2..0000000
--- a/t/t0002-parser-error.sh
+++ /dev/null
@@ -1,94 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "parser error test"
-
-t_begin "setup and startup" && {
-	unicorn_setup
-	unicorn -D env.ru -c $unicorn_config
-	unicorn_wait_start
-}
-
-t_begin "send a bad request" && {
-	(
-		printf 'GET / HTTP/1/1\r\nHost: example.com\r\n\r\n'
-		cat $fifo > $tmp &
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-	test xok = x$(cat $ok)
-}
-
-dbgcat tmp
-
-t_begin "response should be a 400" && {
-	grep -F 'HTTP/1.1 400 Bad Request' $tmp
-}
-
-t_begin "send a huge Request URI (REQUEST_PATH > (12 * 1024))" && {
-	rm -f $tmp
-	cat $fifo > $tmp &
-	(
-		set -e
-		trap 'echo ok > $ok' EXIT
-		printf 'GET /'
-		for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
-		do
-			printf '0123456789ab'
-		done
-		printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
-	) | socat - TCP:$listen > $fifo || :
-	test xok = x$(cat $ok)
-	wait
-}
-
-t_begin "response should be a 414 (REQUEST_PATH)" && {
-	grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && {
-	rm -f $tmp
-	cat $fifo > $tmp &
-	(
-		set -e
-		trap 'echo ok > $ok' EXIT
-		printf 'GET /hello-world?a'
-		for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
-		do
-			printf '0123456789'
-		done
-		printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
-	) | socat - TCP:$listen > $fifo || :
-	test xok = x$(cat $ok)
-	wait
-}
-
-t_begin "response should be a 414 (QUERY_STRING)" && {
-	grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (FRAGMENT > 1024)" && {
-	rm -f $tmp
-	cat $fifo > $tmp &
-	(
-		set -e
-		trap 'echo ok > $ok' EXIT
-		printf 'GET /hello-world#a'
-		for i in $(awk </dev/null 'BEGIN{for(i=0;i<64;i++) print i}')
-		do
-			printf '0123456789abcdef'
-		done
-		printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
-	) | socat - TCP:$listen > $fifo || :
-	test xok = x$(cat $ok)
-	wait
-}
-
-t_begin "response should be a 414 (FRAGMENT)" && {
-	grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "server stderr should be clean" && check_stderr
-
-t_begin "term signal sent" && kill $unicorn_pid
-
-t_done

[-- Attachment #7: 0006-t-integration.t-use-start_req-to-simplify-test-sligh.patch --]
[-- Type: text/x-diff, Size: 2556 bytes --]

From 0bb06cc0c8c4f5b76514858067bbb2871dda0d6e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:35 +0000
Subject: [PATCH 06/23] t/integration.t: use start_req to simplify test slighly

Less code is usually better.
---
 t/integration.t | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/t/integration.t b/t/integration.t
index 503b7eb..b7ba1fb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -20,8 +20,7 @@ sub slurp_hdr {
 my ($c, $status, $hdr);
 
 # response header tests
-$c = tcp_connect($srv);
-print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
 my $orig_200_status = $status;
@@ -40,8 +39,7 @@ SKIP: { # Date header check
 };
 
 
-$c = tcp_connect($srv);
-print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([ grep(/^x-r3: /, @$hdr) ],
 	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -49,8 +47,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
 
 SKIP: {
 	eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
-	$c = tcp_connect($srv);
-	print $c "GET /env_dump\r\n" or die $!;
+	my $c = start_req($srv, 'GET /env_dump');
 	my $json = do { local $/; readline($c) };
 	unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
 	unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -60,20 +57,17 @@ SKIP: {
 }
 
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_connect($srv);
-print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
 if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
-	$c = tcp_connect($srv);
-	print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+	$c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
 
-	$c = tcp_connect($srv);
-	print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+	$c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	is($status, $orig_200_status, 'original status restored');
 }

[-- Attachment #8: 0007-port-t0011-active-unix-socket.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6945 bytes --]

From 10c83beaca58df8b92d8228e798559069cd89beb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:36 +0000
Subject: [PATCH 07/23] port t0011-active-unix-socket.sh to Perl 5

Another socat dependency down...  I've also started turning
FD_CLOEXEC off on a pipe as a mechanism to detect daemonized
process death in tests.
---
 t/active-unix-socket.t        | 117 ++++++++++++++++++++++++++++++++++
 t/integration.ru              |   1 +
 t/t0011-active-unix-socket.sh |  79 -----------------------
 3 files changed, 118 insertions(+), 79 deletions(-)
 create mode 100644 t/active-unix-socket.t
 delete mode 100755 t/t0011-active-unix-socket.sh

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
new file mode 100644
index 0000000..6b5c218
--- /dev/null
+++ b/t/active-unix-socket.t
@@ -0,0 +1,117 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use IO::Socket::UNIX;
+my %to_kill;
+END { kill('TERM', values(%to_kill)) if keys %to_kill }
+my $u1 = "$tmpdir/u1.sock";
+my $u2 = "$tmpdir/u2.sock";
+my $unix_req = sub {
+	my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
+	print $s @_, "\r\n\r\n" or die $!;
+	$s;
+};
+{
+	use autodie;
+	open my $fh, '>', "$tmpdir/u1.conf.rb";
+	print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u1"
+stderr_path "$tmpdir/err1.log"
+EOM
+	close $fh;
+
+	open $fh, '>', "$tmpdir/u2.conf.rb";
+	print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u2"
+stderr_path "$tmpdir/err2.log"
+EOM
+	close $fh;
+
+	open $fh, '>', "$tmpdir/u3.conf.rb";
+	print $fh <<EOM;
+pid "$tmpdir/u3.pid"
+listen "$u1"
+stderr_path "$tmpdir/err3.log"
+EOM
+	close $fh;
+}
+
+my @uarg = qw(-D -E none t/integration.ru);
+
+# this pipe will be used to notify us when all daemons die:
+pipe(my ($p0, $p1)) or die "pipe: $!";
+fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+
+# start the first instance
+unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+is($?, 0, 'daemonized 1st process');
+chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
+ok(kill(0, $worker_pid), 'worker is kill-able');
+
+
+# 2nd process conflicts on PID
+unicorn('-c', "$tmpdir/u2.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting PID file fails to start');
+
+chomp(my $pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
+
+chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID unchanged');
+
+
+# 3rd process conflicts on socket
+unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting UNIX socket fails to start');
+
+chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID still unchanged');
+
+chomp($pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
+
+{ # teardown initial process via SIGKILL
+	ok(kill('KILL', delete $to_kill{u1}), 'SIGKILL initial daemon');
+	close $p1;
+	vec(my $rvec = '', fileno($p0), 1) = 1;
+	is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+	is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+	ok(-f "$tmpdir/u.pid", 'pid file stayed after SIGKILL');
+	ok(-S $u1, 'socket stayed after SIGKILL');
+	is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef,
+		'fail to connect to u1');
+	ok(!kill(0, $worker_pid), 'worker gone after parent dies');
+}
+
+# restart the first instance
+{
+	pipe(($p0, $p1)) or die "pipe: $!";
+	fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+	unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+	is($?, 0, 'daemonized 1st process');
+	chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+	like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+	chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+	like($pid2, qr/\A\d+\z/, 'worker running');
+
+	ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
+	close $p1;
+	vec(my $rvec = '', fileno($p0), 1) = 1;
+	is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+	is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+	ok(!-f "$tmpdir/u.pid", 'pid file gone after SIGTERM');
+	ok(-S $u1, 'socket stays after SIGTERM');
+}
+
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+done_testing;
diff --git a/t/integration.ru b/t/integration.ru
index c0bef99..21f5449 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -57,6 +57,7 @@ def env_dump(env)
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
     when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
+    when '/pid'; [ 200, {}, [ "#$$\n" ] ]
     end # case PATH_INFO (GET)
   when 'POST'
     case env['PATH_INFO']
diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh
deleted file mode 100755
index fae0b6c..0000000
--- a/t/t0011-active-unix-socket.sh
+++ /dev/null
@@ -1,79 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "existing UNIX domain socket check"
-
-read_pid_unix () {
-	x=$(printf 'GET / HTTP/1.0\r\n\r\n' | \
-	    socat - UNIX:$unix_socket | \
-	    tail -1)
-	test -n "$x"
-	y="$(expr "$x" : '\([0-9][0-9]*\)')"
-	test x"$x" = x"$y"
-	test -n "$y"
-	echo "$y"
-}
-
-t_begin "setup and start" && {
-	rtmpfiles unix_socket unix_config
-	rm -f $unix_socket
-	unicorn_setup
-	grep -v ^listen < $unicorn_config > $unix_config
-	echo "listen '$unix_socket'" >> $unix_config
-	unicorn -D -c $unix_config pid.ru
-	unicorn_wait_start
-	orig_master_pid=$unicorn_pid
-}
-
-t_begin "get pid of worker" && {
-	worker_pid=$(read_pid_unix)
-	t_info "worker_pid=$worker_pid"
-}
-
-t_begin "fails to start with existing pid file" && {
-	rm -f $ok
-	unicorn -D -c $unix_config pid.ru || echo ok > $ok
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "worker pid unchanged" && {
-	test x"$(read_pid_unix)" = x$worker_pid
-	> $r_err
-}
-
-t_begin "fails to start with listening UNIX domain socket bound" && {
-	rm $ok $pid
-	unicorn -D -c $unix_config pid.ru || echo ok > $ok
-	test x"$(cat $ok)" = xok
-	> $r_err
-}
-
-t_begin "worker pid unchanged (again)" && {
-	test x"$(read_pid_unix)" = x$worker_pid
-}
-
-t_begin "nuking the existing Unicorn succeeds" && {
-	kill -9 $unicorn_pid
-	while kill -0 $unicorn_pid
-	do
-		sleep 1
-	done
-	check_stderr
-}
-
-t_begin "succeeds in starting with leftover UNIX domain socket bound" && {
-	test -S $unix_socket
-	unicorn -D -c $unix_config pid.ru
-	unicorn_wait_start
-}
-
-t_begin "worker pid changed" && {
-	test x"$(read_pid_unix)" != x$worker_pid
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "no errors" && check_stderr
-
-t_done

[-- Attachment #9: 0008-port-t0100-rack-input-tests.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11722 bytes --]

From b4ed148186295f2d5c8448eab7f2b201615d1e4e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:37 +0000
Subject: [PATCH 08/23] port t0100-rack-input-tests.sh to Perl 5

Yet another socat dependency gone \o/
---
 t/bin/content-md5-put       |  36 -----------
 t/integration.ru            |  27 +++++++-
 t/integration.t             |  97 +++++++++++++++++++++++++++-
 t/lib.perl                  |   3 +-
 t/rack-input-tests.ru       |  21 ------
 t/t0100-rack-input-tests.sh | 124 ------------------------------------
 6 files changed, 124 insertions(+), 184 deletions(-)
 delete mode 100755 t/bin/content-md5-put
 delete mode 100644 t/rack-input-tests.ru
 delete mode 100755 t/t0100-rack-input-tests.sh

diff --git a/t/bin/content-md5-put b/t/bin/content-md5-put
deleted file mode 100755
index 01da0bb..0000000
--- a/t/bin/content-md5-put
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# simple chunked HTTP PUT request generator (and just that),
-# it reads stdin and writes to stdout so socat can write to a
-# UNIX or TCP socket (or to another filter or file) along with
-# a Content-MD5 trailer.
-require 'digest/md5'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-
-bs = ENV['bs'] ? ENV['bs'].to_i : 4096
-
-if ARGV.grep("--no-headers").empty?
-  $stdout.write(
-      "PUT / HTTP/1.1\r\n" \
-      "Host: example.com\r\n" \
-      "Transfer-Encoding: chunked\r\n" \
-      "Trailer: Content-MD5\r\n" \
-      "\r\n"
-    )
-end
-
-digest = Digest::MD5.new
-if buf = $stdin.readpartial(bs)
-  begin
-    digest.update(buf)
-    $stdout.write("%x\r\n" % [ buf.size ])
-    $stdout.write(buf)
-    $stdout.write("\r\n")
-  end while $stdin.read(bs, buf)
-end
-
-digest = [ digest.digest ].pack('m').strip
-$stdout.write("0\r\n")
-$stdout.write("Content-MD5: #{digest}\r\n\r\n")
diff --git a/t/integration.ru b/t/integration.ru
index 21f5449..98528f6 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,29 @@ def env_dump(env)
   h.to_json
 end
 
+def rack_input_tests(env)
+  return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
+  cap = 16384
+  require 'digest/sha1'
+  digest = Digest::SHA1.new
+  input = env['rack.input']
+  case env['PATH_INFO']
+  when '/rack_input/size_first'; input.size
+  when '/rack_input/rewind_first'; input.rewind
+  when '/rack_input'; # OK
+  else
+    abort "bad path: #{env['PATH_INFO']}"
+  end
+  if buf = input.read(rand(cap))
+    begin
+      raise "#{buf.size} > #{cap}" if buf.size > cap
+      digest.update(buf)
+    end while input.read(rand(cap), buf)
+  end
+  [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
+    [ digest.hexdigest ] ]
+end
+
 run(lambda do |env|
   case env['REQUEST_METHOD']
   when 'GET'
@@ -66,6 +89,8 @@ def env_dump(env)
     end # case PATH_INFO (POST)
     # ...
   when 'PUT'
-    # ...
+    case env['PATH_INFO']
+    when %r{\A/rack_input}; rack_input_tests(env)
+    end
   end # case REQUEST_METHOD
 end) # run
diff --git a/t/integration.t b/t/integration.t
index b7ba1fb..8cef561 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,13 +1,16 @@
 #!perl -w
 # Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# this is the main integration test for things which don't require
+# restarting or signals
 
 use v5.14; BEGIN { require './t/lib.perl' };
 my $srv = tcp_server();
 my $host_port = tcp_host_port($srv);
 my $t0 = time;
 my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
-
+my $curl = which('curl');
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
 sub slurp_hdr {
 	my ($c) = @_;
 	local $/ = "\r\n\r\n"; # affects both readline+chomp
@@ -17,6 +20,48 @@ sub slurp_hdr {
 	($status, \@hdr);
 }
 
+my %PUT = (
+	chunked_md5 => sub {
+		my ($in, $out, $path, %opt) = @_;
+		my $bs = $opt{bs} // 16384;
+		require Digest::MD5;
+		my $dig = Digest::MD5->new;
+		print $out <<EOM;
+PUT $path HTTP/1.1\r
+Transfer-Encoding: chunked\r
+Trailer: Content-MD5\r
+\r
+EOM
+		my ($buf, $r);
+		while (1) {
+			$r = read($in, $buf, $bs) // die "read: $!";
+			last if $r == 0;
+			printf $out "%x\r\n", length($buf);
+			print $out $buf, "\r\n";
+			$dig->add($buf);
+		}
+		print $out "0\r\nContent-MD5: ", $dig->b64digest, "\r\n\r\n";
+	},
+	identity => sub {
+		my ($in, $out, $path, %opt) = @_;
+		my $bs = $opt{bs} // 16384;
+		my $clen = $opt{-s} // -s $in;
+		print $out <<EOM;
+PUT $path HTTP/1.0\r
+Content-Length: $clen\r
+\r
+EOM
+		my ($buf, $r, $len);
+		while ($clen) {
+			$len = $clen > $bs ? $bs : $clen;
+			$r = read($in, $buf, $len) // die "read: $!";
+			die 'premature EOF' if $r == 0;
+			print $out $buf;
+			$clen -= $r;
+		}
+	},
+);
+
 my ($c, $status, $hdr);
 
 # response header tests
@@ -111,6 +156,55 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
 }
 
+# input tests
+my ($blob_size, $blob_hash);
+SKIP: {
+	open(my $rh, '<', 't/random_blob') or
+		skip "t/random_blob not generated $!", 1;
+	$blob_size = -s $rh;
+	require Digest::SHA;
+	$blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+
+	my $ck_hash = sub {
+		my ($sub, $path, %opt) = @_;
+		seek($rh, 0, SEEK_SET) // die "seek: $!";
+		$c = tcp_connect($srv);
+		$c->autoflush(0);
+		$PUT{$sub}->($rh, $c, $path, %opt);
+		$c->flush or die "flush: $!";
+		($status, $hdr) = slurp_hdr($c);
+		is(readline($c), $blob_hash, "$sub $path");
+	};
+	$ck_hash->('identity', '/rack_input', -s => $blob_size);
+	$ck_hash->('chunked_md5', '/rack_input');
+	$ck_hash->('identity', '/rack_input/size_first', -s => $blob_size);
+	$ck_hash->('identity', '/rack_input/rewind_first', -s => $blob_size);
+	$ck_hash->('chunked_md5', '/rack_input/size_first');
+	$ck_hash->('chunked_md5', '/rack_input/rewind_first');
+
+
+	$curl // skip 'no curl found in PATH', 1;
+
+	my ($copt, $cout);
+	my $url = "http://$host_port/rack_input";
+	my $do_curl = sub {
+		my (@arg) = @_;
+		pipe(my $cout, $copt->{1}) or die "pipe: $!";
+		open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+		my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
+		close(delete $copt->{1}) or die "close: $!";
+		is(readline($cout), $blob_hash, "curl @arg response");
+		is(waitpid($cpid, 0), $cpid, "curl @arg exited");
+		is($?, 0, "no error from curl @arg");
+		is(slurp("$tmpdir/curl.err"), '', "no stderr from curl @arg");
+	};
+
+	$do_curl->(qw(-T t/random_blob));
+
+	seek($rh, 0, SEEK_SET) // die "seek: $!";
+	$copt->{0} = $rh;
+	$do_curl->('-T-');
+}
 
 # ... more stuff here
 undef $ar;
@@ -120,4 +214,5 @@ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
 is_deeply(\@err, [], 'no unexpected errors in stderr');
 is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
 
+undef $tmpdir;
 done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 7d712b5..ae9f197 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port start_req);
+	SEEK_SET tcp_host_port start_req which spawn);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -193,4 +193,5 @@ Test::More->import;
 # try to ensure ->DESTROY fires:
 $SIG{TERM} = sub { exit(15 + 128) };
 $SIG{INT} = sub { exit(2 + 128) };
+$SIG{PIPE} = sub { exit(13 + 128) };
 1;
diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru
deleted file mode 100644
index 5459e85..0000000
--- a/t/rack-input-tests.ru
+++ /dev/null
@@ -1,21 +0,0 @@
-# SHA1 checksum generator
-require 'digest/sha1'
-use Rack::ContentLength
-cap = 16384
-app = lambda do |env|
-  /\A100-continue\z/i =~ env['HTTP_EXPECT'] and
-    return [ 100, {}, [] ]
-  digest = Digest::SHA1.new
-  input = env['rack.input']
-  input.size if env["PATH_INFO"] == "/size_first"
-  input.rewind if env["PATH_INFO"] == "/rewind_first"
-  if buf = input.read(rand(cap))
-    begin
-      raise "#{buf.size} > #{cap}" if buf.size > cap
-      digest.update(buf)
-    end while input.read(rand(cap), buf)
-  end
-
-  [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ]
-end
-run app
diff --git a/t/t0100-rack-input-tests.sh b/t/t0100-rack-input-tests.sh
deleted file mode 100755
index ee7a437..0000000
--- a/t/t0100-rack-input-tests.sh
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-test -r random_blob || die "random_blob required, run with 'make $0'"
-
-t_plan 10 "rack.input read tests"
-
-t_begin "setup and startup" && {
-	rtmpfiles curl_out curl_err
-	unicorn_setup
-	unicorn -E none -D rack-input-tests.ru -c $unicorn_config
-	blob_sha1=$(rsha1 < random_blob)
-	blob_size=$(count_bytes < random_blob)
-	t_info "blob_sha1=$blob_sha1"
-	unicorn_wait_start
-}
-
-t_begin "corked identity request" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT / HTTP/1.0\r\n'
-		printf 'Content-Length: %d\r\n\r\n' $blob_size
-		cat random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		content-md5-put < random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#size first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /size_first HTTP/1.0\r\n'
-		printf 'Content-Length: %d\r\n\r\n' $blob_size
-		cat random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#rewind first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /rewind_first HTTP/1.0\r\n'
-		printf 'Content-Length: %d\r\n\r\n' $blob_size
-		cat random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#size first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /size_first HTTP/1.1\r\n'
-		printf 'Host: example.com\r\n'
-		printf 'Transfer-Encoding: chunked\r\n'
-		printf 'Trailer: Content-MD5\r\n'
-		printf '\r\n'
-		content-md5-put --no-headers < random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#rewind first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /rewind_first HTTP/1.1\r\n'
-		printf 'Host: example.com\r\n'
-		printf 'Transfer-Encoding: chunked\r\n'
-		printf 'Trailer: Content-MD5\r\n'
-		printf '\r\n'
-		content-md5-put --no-headers < random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "regular request" && {
-	curl -sSf -T random_blob http://$listen/ > $curl_out 2> $curl_err
-        test x$blob_sha1 = x$(cat $curl_out)
-        test ! -s $curl_err
-}
-
-t_begin "chunked request" && {
-	curl -sSf -T- < random_blob http://$listen/ > $curl_out 2> $curl_err
-        test x$blob_sha1 = x$(cat $curl_out)
-        test ! -s $curl_err
-}
-
-dbgcat r_err
-
-t_begin "shutdown" && {
-	kill $unicorn_pid
-}
-
-t_done

[-- Attachment #10: 0009-tests-use-autodie-to-simplify-error-checking.patch --]
[-- Type: text/x-diff, Size: 8495 bytes --]

From 3a1d015a3859b639d8e4463e9436a49f4f0f720e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:38 +0000
Subject: [PATCH 09/23] tests: use autodie to simplify error checking

autodie is bundled with Perl 5.10+ and simplifies error
checking in most cases.  Some subroutines aren't perfectly
translatable and their call sites had to be tweaked, but
most of them are.
---
 t/active-unix-socket.t | 13 +++++++------
 t/integration.t        | 37 +++++++++++++++++++------------------
 t/lib.perl             | 30 +++++++++++++++---------------
 3 files changed, 41 insertions(+), 39 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 6b5c218..1241904 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -4,17 +4,18 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use IO::Socket::UNIX;
+use autodie;
+no autodie 'kill';
 my %to_kill;
 END { kill('TERM', values(%to_kill)) if keys %to_kill }
 my $u1 = "$tmpdir/u1.sock";
 my $u2 = "$tmpdir/u2.sock";
 my $unix_req = sub {
 	my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
-	print $s @_, "\r\n\r\n" or die $!;
+	print $s @_, "\r\n\r\n";
 	$s;
 };
 {
-	use autodie;
 	open my $fh, '>', "$tmpdir/u1.conf.rb";
 	print $fh <<EOM;
 pid "$tmpdir/u.pid"
@@ -43,8 +44,8 @@ EOM
 my @uarg = qw(-D -E none t/integration.ru);
 
 # this pipe will be used to notify us when all daemons die:
-pipe(my ($p0, $p1)) or die "pipe: $!";
-fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+pipe(my $p0, my $p1);
+fcntl($p1, POSIX::F_SETFD, 0);
 
 # start the first instance
 unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
@@ -93,8 +94,8 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 
 # restart the first instance
 {
-	pipe(($p0, $p1)) or die "pipe: $!";
-	fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+	pipe($p0, $p1);
+	fcntl($p1, POSIX::F_SETFD, 0);
 	unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
 	is($?, 0, 'daemonized 1st process');
 	chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
diff --git a/t/integration.t b/t/integration.t
index 8cef561..af17d51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -5,6 +5,7 @@
 # restarting or signals
 
 use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
 my $srv = tcp_server();
 my $host_port = tcp_host_port($srv);
 my $t0 = time;
@@ -34,7 +35,7 @@ Trailer: Content-MD5\r
 EOM
 		my ($buf, $r);
 		while (1) {
-			$r = read($in, $buf, $bs) // die "read: $!";
+			$r = read($in, $buf, $bs);
 			last if $r == 0;
 			printf $out "%x\r\n", length($buf);
 			print $out $buf, "\r\n";
@@ -54,7 +55,7 @@ EOM
 		my ($buf, $r, $len);
 		while ($clen) {
 			$len = $clen > $bs ? $bs : $clen;
-			$r = read($in, $buf, $len) // die "read: $!";
+			$r = read($in, $buf, $len);
 			die 'premature EOF' if $r == 0;
 			print $out $buf;
 			$clen -= $r;
@@ -130,28 +131,28 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
 
 	$c = tcp_connect($srv);
-	print $c 'GET /' or die $!;
+	print $c 'GET /';
 	my $buf = join('', (0..9), 'ab');
-	for (0..1023) { print $c $buf or die $! }
-	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	for (0..1023) { print $c $buf }
+	print $c " HTTP/1.0\r\n\r\n";
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on REQUEST_PATH > (12 * 1024)');
 
 	$c = tcp_connect($srv);
-	print $c 'GET /hello-world?a' or die $!;
+	print $c 'GET /hello-world?a';
 	$buf = join('', (0..9));
-	for (0..1023) { print $c $buf or die $! }
-	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	for (0..1023) { print $c $buf }
+	print $c " HTTP/1.0\r\n\r\n";
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on QUERY_STRING > (10 * 1024)');
 
 	$c = tcp_connect($srv);
-	print $c 'GET /hello-world#a' or die $!;
+	print $c 'GET /hello-world#a';
 	$buf = join('', (0..9), 'a'..'f');
-	for (0..63) { print $c $buf or die $! }
-	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	for (0..63) { print $c $buf }
+	print $c " HTTP/1.0\r\n\r\n";
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
 }
@@ -159,7 +160,7 @@ if ('bad requests') {
 # input tests
 my ($blob_size, $blob_hash);
 SKIP: {
-	open(my $rh, '<', 't/random_blob') or
+	CORE::open(my $rh, '<', 't/random_blob') or
 		skip "t/random_blob not generated $!", 1;
 	$blob_size = -s $rh;
 	require Digest::SHA;
@@ -167,11 +168,11 @@ SKIP: {
 
 	my $ck_hash = sub {
 		my ($sub, $path, %opt) = @_;
-		seek($rh, 0, SEEK_SET) // die "seek: $!";
+		seek($rh, 0, SEEK_SET);
 		$c = tcp_connect($srv);
 		$c->autoflush(0);
 		$PUT{$sub}->($rh, $c, $path, %opt);
-		$c->flush or die "flush: $!";
+		$c->flush or die $!;
 		($status, $hdr) = slurp_hdr($c);
 		is(readline($c), $blob_hash, "$sub $path");
 	};
@@ -189,10 +190,10 @@ SKIP: {
 	my $url = "http://$host_port/rack_input";
 	my $do_curl = sub {
 		my (@arg) = @_;
-		pipe(my $cout, $copt->{1}) or die "pipe: $!";
-		open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+		pipe(my $cout, $copt->{1});
+		open $copt->{2}, '>', "$tmpdir/curl.err";
 		my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
-		close(delete $copt->{1}) or die "close: $!";
+		close(delete $copt->{1});
 		is(readline($cout), $blob_hash, "curl @arg response");
 		is(waitpid($cpid, 0), $cpid, "curl @arg exited");
 		is($?, 0, "no error from curl @arg");
@@ -201,7 +202,7 @@ SKIP: {
 
 	$do_curl->(qw(-T t/random_blob));
 
-	seek($rh, 0, SEEK_SET) // die "seek: $!";
+	seek($rh, 0, SEEK_SET);
 	$copt->{0} = $rh;
 	$do_curl->('-T-');
 }
diff --git a/t/lib.perl b/t/lib.perl
index ae9f197..49632cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -4,6 +4,7 @@
 package UnicornTest;
 use v5.14;
 use parent qw(Exporter);
+use autodie;
 use Test::More;
 use IO::Socket::INET;
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
@@ -14,7 +15,7 @@ our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+open($errfh, '>>', "$tmpdir/err.log");
 
 sub tcp_server {
 	my %opt = (
@@ -62,14 +63,14 @@ sub tcp_connect {
 sub start_req {
 	my ($srv, @req) = @_;
 	my $c = tcp_connect($srv);
-	print $c @req, "\r\n\r\n" or die "print: $!";
+	print $c @req, "\r\n\r\n";
 	$c;
 }
 
 sub slurp {
-	open my $fh, '<', $_[0] or die "open($_[0]): $!";
+	open my $fh, '<', $_[0];
 	local $/;
-	<$fh>;
+	readline($fh);
 }
 
 sub spawn {
@@ -80,8 +81,8 @@ sub spawn {
 	my $set = POSIX::SigSet->new;
 	$set->fillset or die "sigfillset: $!";
 	sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
-	pipe(my ($r, $w)) or die "pipe: $!";
-	my $pid = fork // die "fork: $!";
+	pipe(my $r, my $w);
+	my $pid = fork;
 	if ($pid == 0) {
 		close $r;
 		$SIG{__DIE__} = sub {
@@ -94,9 +95,9 @@ sub spawn {
 		my $cfd;
 		for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
 			my $io = $opt->{$cfd} // next;
-			my $pfd = fileno($io) // die "fileno($io): $!";
+			my $pfd = fileno($io);
 			if ($pfd == $cfd) {
-				fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+				fcntl($io, F_SETFD, 0);
 			} else {
 				dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
 			}
@@ -110,9 +111,7 @@ sub spawn {
 			setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
 		}
 		$SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
-		if (defined(my $cd = $opt->{-C})) {
-			chdir $cd // die "chdir($cd): $!";
-		}
+		if (defined(my $cd = $opt->{-C})) { chdir $cd }
 		$old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
 		sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
 		@ENV{keys %$env} = values(%$env) if $env;
@@ -162,22 +161,23 @@ sub unicorn {
 # automatically kill + reap children when this goes out-of-scope
 package UnicornTest::AutoReap;
 use v5.14;
+use autodie;
 
 sub new {
 	my (undef, $pid) = @_;
 	bless { pid => $pid, owner => $$ }, __PACKAGE__
 }
 
-sub kill {
+sub do_kill {
 	my ($self, $sig) = @_;
-	CORE::kill($sig // 'TERM', $self->{pid});
+	kill($sig // 'TERM', $self->{pid});
 }
 
 sub join {
 	my ($self, $sig) = @_;
 	my $pid = delete $self->{pid} or return;
-	CORE::kill($sig, $pid) if defined $sig;
-	my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+	kill($sig, $pid) if defined $sig;
+	my $ret = waitpid($pid, 0);
 	$ret == $pid or die "BUG: waitpid($pid) != $ret";
 }
 

[-- Attachment #11: 0010-port-t0019-max_header_len.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 5571 bytes --]

From 43c7d73b8b9e6995b5a986b10a8623395e89a538 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:39 +0000
Subject: [PATCH 10/23] port t0019-max_header_len.sh to Perl 5

This was the final socat requirement for integration tests.
I think curl will remain an optional dependency for tests
since it's probably the most widely-installed HTTP client.
---
 GNUmakefile               |  2 +-
 t/README                  |  7 +-----
 t/integration.ru          |  1 +
 t/integration.t           | 43 +++++++++++++++++++++++++++++++---
 t/t0019-max_header_len.sh | 49 ---------------------------------------
 5 files changed, 43 insertions(+), 59 deletions(-)
 delete mode 100755 t/t0019-max_header_len.sh

diff --git a/GNUmakefile b/GNUmakefile
index 5cca189..eab9082 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -125,7 +125,7 @@ $(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore
 t/trash/.gitignore : | t/trash
 	echo '*' >$@
 
-dependencies := socat curl
+dependencies := curl
 deps := $(addprefix t/.dep+,$(dependencies))
 $(deps): dep_bin = $(lastword $(subst +, ,$@))
 $(deps):
diff --git a/t/README b/t/README
index 8a5243e..d09c715 100644
--- a/t/README
+++ b/t/README
@@ -10,18 +10,13 @@ to test real-world behavior and Ruby introduces incompatibilities
 at a far faster rate than Perl 5.  Perl is Ruby's older cousin, so
 it should be easy-to-learn for Rubyists.
 
-Old tests are in Bourne shell, but the socat(1) dependency was probably
-too rare compared to Perl 5.
+Old tests are in Bourne shell and slowly being ported to Perl 5.
 
 == Requirements
 
 * {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
 * {Perl 5.14+}[https://www.perl.org/] # your distro should have it
 * {GNU make}[https://www.gnu.org/software/make/]
-
-The following requirements will eventually be dropped.
-
-* {socat}[http://www.dest-unreach.org/socat/]
 * {curl}[https://curl.haxx.se/]
 
 We do not use bashisms or any non-portable, non-POSIX constructs
diff --git a/t/integration.ru b/t/integration.ru
index 98528f6..edc408c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -81,6 +81,7 @@ def rack_input_tests(env)
     when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
     when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+    else '/'; [ 200, {}, [ env_dump(env) ] ]
     end # case PATH_INFO (GET)
   when 'POST'
     case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index af17d51..c687655 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,15 +1,19 @@
 #!perl -w
 # Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
-# this is the main integration test for things which don't require
-# restarting or signals
+
+# This is the main integration test for fast-ish things to minimize
+# Ruby startup time penalties.
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
 my $srv = tcp_server();
 my $host_port = tcp_host_port($srv);
 my $t0 = time;
-my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+my $conf = "$tmpdir/u.conf.rb";
+open my $conf_fh, '>', $conf;
+$conf_fh->autoflush(1);
+my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');
 END { diag slurp("$tmpdir/err.log") if $tmpdir };
 sub slurp_hdr {
@@ -207,7 +211,40 @@ SKIP: {
 	$do_curl->('-T-');
 }
 
+
 # ... more stuff here
+
+# SIGHUP-able stuff goes here
+
+if ('max_header_len internal API') {
+	undef $c;
+	my $req = 'GET / HTTP/1.0';
+	my $len = length($req."\r\n\r\n");
+	my $fifo = "$tmpdir/fifo";
+	POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+	print $conf_fh <<EOM;
+Unicorn::HttpParser.max_header_len = $len
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+	$ar->do_kill('HUP');
+	open my $fifo_fh, '<', $fifo;
+	my $wpid = readline($fifo_fh);
+	like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+	close $fifo_fh;
+	$wpid =~ s/\Apid=// or die;
+	ok(CORE::kill(0, $wpid), 'worker PID retrieved');
+
+	$c = start_req($srv, $req);
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
+
+	$c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
+}
+
+
 undef $ar;
 my @log = slurp("$tmpdir/err.log");
 diag("@log") if $ENV{V};
diff --git a/t/t0019-max_header_len.sh b/t/t0019-max_header_len.sh
deleted file mode 100755
index 6a355b4..0000000
--- a/t/t0019-max_header_len.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 5 "max_header_len setting (only intended for Rainbows!)"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	req='GET / HTTP/1.0\r\n\r\n'
-	len=$(printf "$req" | count_bytes)
-	echo Unicorn::HttpParser.max_header_len = $len >> $unicorn_config
-	unicorn -D -c $unicorn_config env.ru
-	unicorn_wait_start
-}
-
-t_begin "minimal request succeeds" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf "$req"
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-	test xok = x$(cat $ok)
-
-	fgrep "HTTP/1.1 200 OK" $tmp
-}
-
-t_begin "big request fails" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'GET /xxxxxx HTTP/1.0\r\n\r\n'
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-	test xok = x$(cat $ok)
-	fgrep "HTTP/1.1 413" $tmp
-}
-
-dbgcat tmp
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
-	check_stderr
-}
-
-t_done

[-- Attachment #12: 0011-test_exec-drop-sd_listen_fds-emulation-test.patch --]
[-- Type: text/x-diff, Size: 1751 bytes --]

From 5d828a4ef7683345bcf2ff659442fed0a6fb7a97 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:40 +0000
Subject: [PATCH 11/23] test_exec: drop sd_listen_fds emulation test

The Perl 5 tests already rely on this implicitly, and there was
never a point when Perl 5 couldn't emulate systemd behavior.
---
 test/exec/test_exec.rb | 33 ---------------------------------
 1 file changed, 33 deletions(-)

diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 2929b2e..1d3a0fd 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -97,39 +97,6 @@ def teardown
     end
   end
 
-  def test_sd_listen_fds_emulation
-    # [ruby-core:69895] [Bug #11336] fixed by r51576
-    return if RUBY_VERSION.to_f < 2.3
-
-    File.open("config.ru", "wb") { |fp| fp.write(HI) }
-    sock = TCPServer.new(@addr, @port)
-
-    [ %W(-l #@addr:#@port), nil ].each do |l|
-      sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
-      pid = xfork do
-        redirect_test_io do
-          # pretend to be systemd
-          ENV['LISTEN_PID'] = "#$$"
-          ENV['LISTEN_FDS'] = '1'
-
-          # 3 = SD_LISTEN_FDS_START
-          args = [ $unicorn_bin ]
-          args.concat(l) if l
-          args << { 3 => sock }
-          exec(*args)
-        end
-      end
-      res = hit(["http://#@addr:#@port/"])
-      assert_equal [ "HI\n" ], res
-      assert_shutdown(pid)
-      assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
-                  'unicorn should always set SO_KEEPALIVE on inherited sockets'
-    end
-  ensure
-    sock.close if sock
-  end
-
   def test_inherit_listener_unspecified
     File.open("config.ru", "wb") { |fp| fp.write(HI) }
     sock = TCPServer.new(@addr, @port)

[-- Attachment #13: 0012-test_exec-drop-test_basic-and-test_config_ru_alt_pat.patch --]
[-- Type: text/x-diff, Size: 1667 bytes --]

From 548593c6b3d52a4bebd52542ad9c423ed2b7252d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:41 +0000
Subject: [PATCH 12/23] test_exec: drop test_basic and test_config_ru_alt_path

We already have coverage for these basic things elsewhere.
---
 test/exec/test_exec.rb | 24 ------------------------
 1 file changed, 24 deletions(-)

diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 1d3a0fd..55f828e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -265,16 +265,6 @@ def test_exit_signals
     end
   end
 
-  def test_basic
-    File.open("config.ru", "wb") { |fp| fp.syswrite(HI) }
-    pid = fork do
-      redirect_test_io { exec($unicorn_bin, "-l", "#{@addr}:#{@port}") }
-    end
-    results = retry_hit(["http://#{@addr}:#{@port}/"])
-    assert_equal String, results[0].class
-    assert_shutdown(pid)
-  end
-
   def test_rack_env_unset
     File.open("config.ru", "wb") { |fp| fp.syswrite(SHOW_RACK_ENV) }
     pid = fork { redirect_test_io { exec($unicorn_bin, "-l#@addr:#@port") } }
@@ -638,20 +628,6 @@ def test_read_embedded_cli_switches
     assert_shutdown(pid)
   end
 
-  def test_config_ru_alt_path
-    config_path = "#{@tmpdir}/foo.ru"
-    File.open(config_path, "wb") { |fp| fp.syswrite(HI) }
-    pid = fork do
-      redirect_test_io do
-        Dir.chdir("/")
-        exec($unicorn_bin, "-l#{@addr}:#{@port}", config_path)
-      end
-    end
-    results = retry_hit(["http://#{@addr}:#{@port}/"])
-    assert_equal String, results[0].class
-    assert_shutdown(pid)
-  end
-
   def test_load_module
     libdir = "#{@tmpdir}/lib"
     FileUtils.mkpath([ libdir ])

[-- Attachment #14: 0013-tests-check_stderr-consistently-in-Perl-5-tests.patch --]
[-- Type: text/x-diff, Size: 2415 bytes --]

From cd7ee67fc8ebadec9bdd913d49ed3f214596ea47 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:42 +0000
Subject: [PATCH 13/23] tests: check_stderr consistently in Perl 5 tests

The Bourne shell tests did, so lets not let stuff sneak past us.
---
 t/active-unix-socket.t |  5 ++---
 t/integration.t        |  7 ++-----
 t/lib.perl             | 10 +++++++++-
 3 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 1241904..c132dc2 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -20,7 +20,7 @@ my $unix_req = sub {
 	print $fh <<EOM;
 pid "$tmpdir/u.pid"
 listen "$u1"
-stderr_path "$tmpdir/err1.log"
+stderr_path "$tmpdir/err.log"
 EOM
 	close $fh;
 
@@ -113,6 +113,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 	ok(-S $u1, 'socket stays after SIGTERM');
 }
 
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
+check_stderr;
 done_testing;
diff --git a/t/integration.t b/t/integration.t
index c687655..939dc24 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -246,11 +246,8 @@ EOM
 
 
 undef $ar;
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
-my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
-is_deeply(\@err, [], 'no unexpected errors in stderr');
-is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
+check_stderr;
 
 undef $tmpdir;
 done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 49632cf..315ef2d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,12 +11,20 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port start_req which spawn);
+	SEEK_SET tcp_host_port start_req which spawn check_stderr);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
 open($errfh, '>>', "$tmpdir/err.log");
 
+sub check_stderr () {
+	my @log = slurp("$tmpdir/err.log");
+	diag("@log") if $ENV{V};
+	my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+	is_deeply(\@err, [], 'no unexpected errors in stderr');
+	is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+}
+
 sub tcp_server {
 	my %opt = (
 		ReuseAddr => 1,

[-- Attachment #15: 0014-tests-consistent-tcp_start-and-unix_start-across-Per.patch --]
[-- Type: text/x-diff, Size: 8017 bytes --]

From 0dcd8bd569813a175ad43837db3ab07019a95b99 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:43 +0000
Subject: [PATCH 14/23] tests: consistent tcp_start and unix_start across Perl
 5 tests

I'll be using Unix sockets more in tests since there's no
risk of system-wide conflicts with TCP port allocation.
Furthermore, curl supports `--unix-socket' nowadays; so
there's little reason to rely on TCP sockets and the conflicts
they bring in tests.
---
 t/active-unix-socket.t | 13 ++++---------
 t/integration.t        | 28 ++++++++++++++--------------
 t/lib.perl             | 30 ++++++++++++++++--------------
 3 files changed, 34 insertions(+), 37 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index c132dc2..8723137 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -10,11 +10,6 @@ my %to_kill;
 END { kill('TERM', values(%to_kill)) if keys %to_kill }
 my $u1 = "$tmpdir/u1.sock";
 my $u2 = "$tmpdir/u2.sock";
-my $unix_req = sub {
-	my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
-	print $s @_, "\r\n\r\n";
-	$s;
-};
 {
 	open my $fh, '>', "$tmpdir/u1.conf.rb";
 	print $fh <<EOM;
@@ -53,7 +48,7 @@ is($?, 0, 'daemonized 1st process');
 chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
 like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
 
-chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $worker_pid = readline(unix_start($u1, 'GET /pid')));
 like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
 ok(kill(0, $worker_pid), 'worker is kill-able');
 
@@ -65,7 +60,7 @@ isnt($?, 0, 'conflicting PID file fails to start');
 chomp(my $pidf = slurp("$tmpdir/u.pid"));
 is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
 
-chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $pid2 = readline(unix_start($u1, 'GET /pid')));
 is($worker_pid, $pid2, 'worker PID unchanged');
 
 
@@ -73,7 +68,7 @@ is($worker_pid, $pid2, 'worker PID unchanged');
 unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
 isnt($?, 0, 'conflicting UNIX socket fails to start');
 
-chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
 is($worker_pid, $pid2, 'worker PID still unchanged');
 
 chomp($pidf = slurp("$tmpdir/u.pid"));
@@ -101,7 +96,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 	chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
 	like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
 
-	chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+	chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
 	like($pid2, qr/\A\d+\z/, 'worker running');
 
 	ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
diff --git a/t/integration.t b/t/integration.t
index 939dc24..b33e3c3 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -70,7 +70,7 @@ EOM
 my ($c, $status, $hdr);
 
 # response header tests
-$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
 my $orig_200_status = $status;
@@ -89,7 +89,7 @@ SKIP: { # Date header check
 };
 
 
-$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([ grep(/^x-r3: /, @$hdr) ],
 	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -97,7 +97,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
 
 SKIP: {
 	eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
-	my $c = start_req($srv, 'GET /env_dump');
+	my $c = tcp_start($srv, 'GET /env_dump');
 	my $json = do { local $/; readline($c) };
 	unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
 	unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -107,17 +107,17 @@ SKIP: {
 }
 
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
+$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
 if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
-	$c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
+	$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
 
-	$c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
+	$c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	is($status, $orig_200_status, 'original status restored');
 }
@@ -130,12 +130,12 @@ SKIP: {
 }
 
 if ('bad requests') {
-	$c = start_req($srv, 'GET /env_dump HTTP/1/1');
+	$c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
 
-	$c = tcp_connect($srv);
-	print $c 'GET /';
+	$c = tcp_start($srv);
+	print $c 'GET /';;
 	my $buf = join('', (0..9), 'ab');
 	for (0..1023) { print $c $buf }
 	print $c " HTTP/1.0\r\n\r\n";
@@ -143,7 +143,7 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on REQUEST_PATH > (12 * 1024)');
 
-	$c = tcp_connect($srv);
+	$c = tcp_start($srv);
 	print $c 'GET /hello-world?a';
 	$buf = join('', (0..9));
 	for (0..1023) { print $c $buf }
@@ -152,7 +152,7 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on QUERY_STRING > (10 * 1024)');
 
-	$c = tcp_connect($srv);
+	$c = tcp_start($srv);
 	print $c 'GET /hello-world#a';
 	$buf = join('', (0..9), 'a'..'f');
 	for (0..63) { print $c $buf }
@@ -173,7 +173,7 @@ SKIP: {
 	my $ck_hash = sub {
 		my ($sub, $path, %opt) = @_;
 		seek($rh, 0, SEEK_SET);
-		$c = tcp_connect($srv);
+		$c = tcp_start($srv);
 		$c->autoflush(0);
 		$PUT{$sub}->($rh, $c, $path, %opt);
 		$c->flush or die $!;
@@ -235,11 +235,11 @@ EOM
 	$wpid =~ s/\Apid=// or die;
 	ok(CORE::kill(0, $wpid), 'worker PID retrieved');
 
-	$c = start_req($srv, $req);
+	$c = tcp_start($srv, $req);
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
 
-	$c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+	$c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
 }
diff --git a/t/lib.perl b/t/lib.perl
index 315ef2d..1d6e78d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,8 +10,8 @@ use IO::Socket::INET;
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port start_req which spawn check_stderr);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+	SEEK_SET tcp_host_port which spawn check_stderr unix_start);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -55,26 +55,28 @@ sub tcp_host_port {
 	}
 }
 
-sub tcp_connect {
-	my ($dest, %opt) = @_;
-	my $addr = tcp_host_port($dest);
-	my $s = ref($dest)->new(
+sub unix_start ($@) {
+	my ($dst, @req) = @_;
+	my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
+		BAIL_OUT "unix connect $dst: $!";
+	$s->autoflush(1);
+	print $s @req, "\r\n\r\n" if @req;
+	$s;
+}
+
+sub tcp_start ($@) {
+	my ($dst, @req) = @_;
+	my $addr = tcp_host_port($dst);
+	my $s = ref($dst)->new(
 		Proto => 'tcp',
 		Type => SOCK_STREAM,
 		PeerAddr => $addr,
-		%opt,
 	) or BAIL_OUT "failed to connect to $addr: $!";
 	$s->autoflush(1);
+	print $s @req, "\r\n\r\n" if @req;
 	$s;
 }
 
-sub start_req {
-	my ($srv, @req) = @_;
-	my $c = tcp_connect($srv);
-	print $c @req, "\r\n\r\n";
-	$c;
-}
-
 sub slurp {
 	open my $fh, '<', $_[0];
 	local $/;

[-- Attachment #16: 0015-port-t9000-preread-input.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3856 bytes --]

From 1b8840d8d13491eecd2fa92e06f73c65eadd33ba Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:44 +0000
Subject: [PATCH 15/23] port t9000-preread-input.sh to Perl 5

Stuffing it into t/integration.t for now so we can save on
startup costs.
---
 t/integration.t          | 32 ++++++++++++++++++++++++---
 t/lib.perl               |  2 +-
 t/preread_input.ru       |  4 +---
 t/t9000-preread-input.sh | 48 ----------------------------------------
 4 files changed, 31 insertions(+), 55 deletions(-)
 delete mode 100755 t/t9000-preread-input.sh

diff --git a/t/integration.t b/t/integration.t
index b33e3c3..f5afd5d 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,8 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
-my $srv = tcp_server();
-my $host_port = tcp_host_port($srv);
+our $srv = tcp_server();
+our $host_port = tcp_host_port($srv);
 my $t0 = time;
 my $conf = "$tmpdir/u.conf.rb";
 open my $conf_fh, '>', $conf;
@@ -209,8 +209,34 @@ SKIP: {
 	seek($rh, 0, SEEK_SET);
 	$copt->{0} = $rh;
 	$do_curl->('-T-');
-}
 
+	diag 'testing Unicorn::PrereadInput...';
+	local $srv = tcp_server();
+	local $host_port = tcp_host_port($srv);
+	check_stderr;
+	truncate($errfh, 0);
+
+	my $pri = unicorn(qw(-E none t/preread_input.ru), { 3 => $srv });
+	$url = "http://$host_port/";
+
+	$do_curl->(qw(-T t/random_blob));
+	seek($rh, 0, SEEK_SET);
+	$copt->{0} = $rh;
+	$do_curl->('-T-');
+
+	my @pr_err = slurp("$tmpdir/err.log");
+	is(scalar(grep(/app dispatch:/, @pr_err)), 2, 'app dispatched twice');
+
+	# abort a chunked request by blocking curl on a FIFO:
+	$c = tcp_start($srv, "PUT / HTTP/1.1\r\nTransfer-Encoding: chunked");
+	close $c;
+	@pr_err = slurp("$tmpdir/err.log");
+	is(scalar(grep(/app dispatch:/, @pr_err)), 2,
+			'app did not dispatch on aborted request');
+	undef $pri;
+	check_stderr;
+	diag 'Unicorn::PrereadInput middleware tests done';
+}
 
 # ... more stuff here
 
diff --git a/t/lib.perl b/t/lib.perl
index 1d6e78d..b6148cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -79,7 +79,7 @@ sub tcp_start ($@) {
 
 sub slurp {
 	open my $fh, '<', $_[0];
-	local $/;
+	local $/ if !wantarray;
 	readline($fh);
 }
 
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 79685c4..f0a1748 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,8 +1,6 @@
 #\-E none
 require 'digest/sha1'
 require 'unicorn/preread_input'
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
 use Unicorn::PrereadInput
 nr = 0
 run lambda { |env|
@@ -13,5 +11,5 @@
     dig.update(buf)
   end
 
-  [ 200, {}, [ "#{dig.hexdigest}\n" ] ]
+  [ 200, {}, [ dig.hexdigest ] ]
 }
diff --git a/t/t9000-preread-input.sh b/t/t9000-preread-input.sh
deleted file mode 100755
index d6c73ab..0000000
--- a/t/t9000-preread-input.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 9 "PrereadInput middleware tests"
-
-t_begin "setup and start" && {
-	random_blob_sha1=$(rsha1 < random_blob)
-	unicorn_setup
-	unicorn  -D -c $unicorn_config preread_input.ru
-	unicorn_wait_start
-}
-
-t_begin "single identity request" && {
-	curl -sSf -T random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
-	test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "single chunked request" && {
-	curl -sSf -T- < random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
-	test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "app only dispatched twice" && {
-	test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "aborted chunked request" && {
-	rm -f $tmp
-	curl -sSf -T- < $fifo http://$listen/ > $tmp &
-	curl_pid=$!
-	kill -9 $curl_pid
-	wait
-}
-
-t_begin "app only dispatched twice" && {
-	test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "killing succeeds" && {
-	kill -QUIT $unicorn_pid
-}
-
-t_done

[-- Attachment #17: 0016-port-t-t0116-client_body_buffer_size.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8861 bytes --]

From e9593301044f305d4a0e074f77eea35015ca0ec4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:45 +0000
Subject: [PATCH 16/23] port t/t0116-client_body_buffer_size.sh to Perl 5

While I'm fine with depending on curl for certain things,
there's no need for it here since unicorn has had lazy
rack.input for over a decade, at this point.
---
 t/active-unix-socket.t                     |  1 +
 t/{t0116.ru => client_body_buffer_size.ru} |  2 -
 t/client_body_buffer_size.t                | 83 ++++++++++++++++++++++
 t/integration.t                            | 10 ---
 t/lib.perl                                 | 12 +++-
 t/t0116-client_body_buffer_size.sh         | 80 ---------------------
 6 files changed, 95 insertions(+), 93 deletions(-)
 rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
 create mode 100644 t/client_body_buffer_size.t
 delete mode 100755 t/t0116-client_body_buffer_size.sh

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 8723137..4e11837 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -109,4 +109,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 }
 
 check_stderr;
+undef $tmpdir;
 done_testing;
diff --git a/t/t0116.ru b/t/client_body_buffer_size.ru
similarity index 82%
rename from t/t0116.ru
rename to t/client_body_buffer_size.ru
index fab5fce..44161a5 100644
--- a/t/t0116.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,6 +1,4 @@
 #\ -E none
-use Rack::ContentLength
-use Rack::ContentType, 'text/plain'
 app = lambda do |env|
   input = env['rack.input']
   case env["PATH_INFO"]
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
new file mode 100644
index 0000000..b1a99f3
--- /dev/null
+++ b/t/client_body_buffer_size.t
@@ -0,0 +1,83 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $uconf = "$tmpdir/u.conf.rb";
+
+open my $conf_fh, '>', $uconf;
+$conf_fh->autoflush(1);
+print $conf_fh <<EOM;
+client_body_buffer_size 0
+EOM
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my $ar = unicorn(@uarg, { 3 => $srv });
+my ($c, $status, $hdr);
+my $mem_class = 'StringIO';
+my $fs_class = 'Unicorn::TmpIO';
+
+$c = tcp_start($srv, "PUT /input_class HTTP/1.0\r\nContent-Length: 0");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'zero-byte file is StringIO');
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+print $c '.';
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 byte file is filesystem-backed');
+
+
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+seek($conf_fh, 0, SEEK_SET);
+truncate($conf_fh, 0);
+print $conf_fh <<EOM;
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+$ar->do_kill('HUP');
+open my $fifo_fh, '<', $fifo;
+like(my $wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+	'reloaded w/ default client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'class for a 1 byte file is memory-backed');
+
+
+my $one_meg = 1024 ** 2;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte file is FS-backed');
+
+# reload with bigger client_body_buffer_size
+say $conf_fh "client_body_buffer_size $one_meg";
+$ar->do_kill('HUP');
+open $fifo_fh, '<', $fifo;
+like($wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+	'reloaded w/ bigger client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, '1 megabyte file is now memory-backed');
+
+my $too_big = $one_meg + 1;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $too_big");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte + 1 byte file is FS-backed');
+
+
+undef $ar;
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/integration.t b/t/integration.t
index f5afd5d..855c260 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -15,16 +15,6 @@ open my $conf_fh, '>', $conf;
 $conf_fh->autoflush(1);
 my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
-sub slurp_hdr {
-	my ($c) = @_;
-	local $/ = "\r\n\r\n"; # affects both readline+chomp
-	chomp(my $hdr = readline($c));
-	my ($status, @hdr) = split(/\r\n/, $hdr);
-	diag explain([ $status, \@hdr ]) if $ENV{V};
-	($status, \@hdr);
-}
-
 my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index b6148cf..2685c3b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,11 +11,12 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port which spawn check_stderr unix_start);
+	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
 open($errfh, '>>', "$tmpdir/err.log");
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
 
 sub check_stderr () {
 	my @log = slurp("$tmpdir/err.log");
@@ -25,6 +26,15 @@ sub check_stderr () {
 	is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
 }
 
+sub slurp_hdr {
+	my ($c) = @_;
+	local $/ = "\r\n\r\n"; # affects both readline+chomp
+	chomp(my $hdr = readline($c));
+	my ($status, @hdr) = split(/\r\n/, $hdr);
+	diag explain([ $status, \@hdr ]) if $ENV{V};
+	($status, \@hdr);
+}
+
 sub tcp_server {
 	my %opt = (
 		ReuseAddr => 1,
diff --git a/t/t0116-client_body_buffer_size.sh b/t/t0116-client_body_buffer_size.sh
deleted file mode 100755
index c9e17c7..0000000
--- a/t/t0116-client_body_buffer_size.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 12 "client_body_buffer_size settings"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	rtmpfiles unicorn_config_tmp one_meg
-	dd if=/dev/zero bs=1M count=1 of=$one_meg
-	cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
-  File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
-end
-EOF
-	cat $unicorn_config > $unicorn_config_tmp
-	echo client_body_buffer_size 0 >> $unicorn_config
-	unicorn -D -c $unicorn_config t0116.ru
-	unicorn_wait_start
-	fs_class=Unicorn::TmpIO
-	mem_class=StringIO
-
-	test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a zero-byte file should be StringIO" && {
-	> $tmp
-	test xStringIO = x"$(curl -T $tmp -sSf http://$listen/input_class)"
-}
-
-t_begin "class for a 1 byte file should be filesystem-backed" && {
-	echo > $tmp
-	test x$fs_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "reload with default client_body_buffer_size" && {
-	mv $unicorn_config_tmp $unicorn_config
-	kill -HUP $unicorn_pid
-	test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a 1 byte file should be memory-backed" && {
-	echo > $tmp
-	test x$mem_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "class for a random blob file should be filesystem-backed" && {
-	resp="$(curl -T random_blob -sSf http://$listen/tmp_class)"
-	test x$fs_class = x"$resp"
-}
-
-t_begin "one megabyte file should be filesystem-backed" && {
-	resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
-	test x$fs_class = x"$resp"
-}
-
-t_begin "reload with a big client_body_buffer_size" && {
-	echo "client_body_buffer_size(1024 * 1024)" >> $unicorn_config
-	kill -HUP $unicorn_pid
-	test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "one megabyte file should be memory-backed" && {
-	resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
-	test x$mem_class = x"$resp"
-}
-
-t_begin "one megabyte + 1 byte file should be filesystem-backed" && {
-	echo >> $one_meg
-	resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
-	test x$fs_class = x"$resp"
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
-	check_stderr
-}
-
-t_done

[-- Attachment #18: 0017-tests-get-rid-of-sha1sum.rb-and-rsha1-sh-function.patch --]
[-- Type: text/x-diff, Size: 1255 bytes --]

From b47912160f2336dde3901e588cc23fb2c2f8d9dc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:46 +0000
Subject: [PATCH 17/23] tests: get rid of sha1sum.rb and rsha1() sh function

These are no longer needed since Perl has long included
Digest::SHA
---
 t/bin/sha1sum.rb | 17 -----------------
 t/test-lib.sh    |  4 ----
 2 files changed, 21 deletions(-)
 delete mode 100755 t/bin/sha1sum.rb

diff --git a/t/bin/sha1sum.rb b/t/bin/sha1sum.rb
deleted file mode 100755
index 53d68ce..0000000
--- a/t/bin/sha1sum.rb
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# Reads from stdin and outputs the SHA1 hex digest of the input
-
-require 'digest/sha1'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-bs = 16384
-digest = Digest::SHA1.new
-if buf = $stdin.read(bs)
-  begin
-    digest.update(buf)
-  end while $stdin.read(bs, buf)
-end
-
-$stdout.syswrite("#{digest.hexdigest}\n")
diff --git a/t/test-lib.sh b/t/test-lib.sh
index e70d0c6..8613144 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -123,7 +123,3 @@ unicorn_wait_start () {
 	# no need to play tricks with FIFOs since we got "ready_pipe" now
 	unicorn_pid=$(cat $pid)
 }
-
-rsha1 () {
-	sha1sum.rb
-}

[-- Attachment #19: 0018-early_hints-supports-Rack-3-array-headers.patch --]
[-- Type: text/x-diff, Size: 4606 bytes --]

From 6ad9f4b54ee16ffecea7e16b710552b45db33a16 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:47 +0000
Subject: [PATCH 18/23] early_hints supports Rack 3 array headers

We can hoist out append_headers into a new method and use it in
both e103_response_write and http_response_write.

t/integration.t now tests early_hints with both possible
values of check_client_connection.
---
 t/integration.ru |  7 +++++++
 t/integration.t  | 47 ++++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 49 insertions(+), 5 deletions(-)

diff --git a/t/integration.ru b/t/integration.ru
index edc408c..dab384d 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -5,6 +5,11 @@
 # this goes for t/integration.t  We'll try to put as many tests
 # in here as possible to avoid startup overhead of Ruby.
 
+def early_hints(env, val)
+  env['rack.early_hints'].call('link' => val) # val may be ary or string
+  [ 200, {}, [ val.class.to_s ] ]
+end
+
 $orig_rack_200 = nil
 def tweak_status_code
   $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
@@ -81,6 +86,8 @@ def rack_input_tests(env)
     when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
     when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+    when '/early_hints_rack2'; early_hints(env, "r\n2")
+    when '/early_hints_rack3'; early_hints(env, %w(r 3))
     else '/'; [ 200, {}, [ env_dump(env) ] ]
     end # case PATH_INFO (GET)
   when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 855c260..8433497 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -13,8 +13,16 @@ my $t0 = time;
 my $conf = "$tmpdir/u.conf.rb";
 open my $conf_fh, '>', $conf;
 $conf_fh->autoflush(1);
+my $u1 = "$tmpdir/u1";
+print $conf_fh <<EOM;
+early_hints true
+listen "$u1"
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+EOM
 my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
 my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
@@ -102,6 +110,26 @@ $c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
+my $ck_early_hints = sub {
+	my ($note) = @_;
+	$c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 2 value');
+	is_deeply(['link: r', 'link: 2'], $hdr, 'rack 2 hints match '.$note);
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+	is(readline($c), 'String', 'early hints used a String for rack 2');
+
+	$c = unix_start($u1, 'GET /early_hints_rack3 HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 3');
+	is_deeply(['link: r', 'link: 3'], $hdr, 'rack 3 hints match '.$note);
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+	is(readline($c), 'Array', 'early hints used a String for rack 3');
+};
+$ck_early_hints->('ccc off'); # we'll retest later
+
 if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
 	$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
@@ -154,6 +182,7 @@ if ('bad requests') {
 # input tests
 my ($blob_size, $blob_hash);
 SKIP: {
+	skip 'SKIP_EXPENSIVE on', 1 if $ENV{SKIP_EXPENSIVE};
 	CORE::open(my $rh, '<', 't/random_blob') or
 		skip "t/random_blob not generated $!", 1;
 	$blob_size = -s $rh;
@@ -232,16 +261,24 @@ SKIP: {
 
 # SIGHUP-able stuff goes here
 
+if ('check_client_connection') {
+	print $conf_fh <<EOM; # appending to existing
+check_client_connection true
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+	$ar->do_kill('HUP');
+	open my $fifo_fh, '<', $fifo;
+	my $wpid = readline($fifo_fh);
+	like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+	$ck_early_hints->('ccc on');
+}
+
 if ('max_header_len internal API') {
 	undef $c;
 	my $req = 'GET / HTTP/1.0';
 	my $len = length($req."\r\n\r\n");
-	my $fifo = "$tmpdir/fifo";
-	POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
-	print $conf_fh <<EOM;
+	print $conf_fh <<EOM; # appending to existing
 Unicorn::HttpParser.max_header_len = $len
-listen "$host_port" # TODO: remove this requirement for SIGHUP
-after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
 EOM
 	$ar->do_kill('HUP');
 	open my $fifo_fh, '<', $fifo;

[-- Attachment #20: 0019-test_server-drop-early_hints-test.patch --]
[-- Type: text/x-diff, Size: 1737 bytes --]

From 3e6bc9fb589fd88469349a38a77704c3333623e0 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:48 +0000
Subject: [PATCH 19/23] test_server: drop early_hints test

t/integration.t already is more complete in that it tests
both Rack 2 and 3 along with both possible values of
check_client_connection.
---
 test/unit/test_server.rb | 31 -------------------------------
 1 file changed, 31 deletions(-)

diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index fe98fcc..0a710d1 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,17 +23,6 @@ def call(env)
   end
 end
 
-class TestEarlyHintsHandler
-  def call(env)
-    while env['rack.input'].read(4096)
-    end
-    env['rack.early_hints'].call(
-      "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
-    )
-    [200, { 'content-type' => 'text/plain' }, ['hello!\n']]
-  end
-end
-
 class TestRackAfterReply
   def initialize
     @called = false
@@ -112,26 +101,6 @@ def test_preload_app_config
     tmp.close!
   end
 
-  def test_early_hints
-    teardown
-    redirect_test_io do
-      @server = HttpServer.new(TestEarlyHintsHandler.new,
-                               :listeners => [ "127.0.0.1:#@port"],
-                               :early_hints => true)
-      @server.start
-    end
-
-    sock = tcp_socket('127.0.0.1', @port)
-    sock.syswrite("GET / HTTP/1.0\r\n\r\n")
-
-    responses = sock.read(4096)
-    assert_match %r{\AHTTP/1.[01] 103\b}, responses
-    assert_match %r{^Link: </style\.css>}, responses
-    assert_match %r{^Link: </script\.js>}, responses
-
-    assert_match %r{^HTTP/1.[01] 200\b}, responses
-  end
-
   def test_after_reply
     teardown
 

[-- Attachment #21: 0020-t-integration.t-switch-PUT-tests-to-MD5-reuse-buffer.patch --]
[-- Type: text/x-diff, Size: 3740 bytes --]

From cb826915cdd1881cbcfc1fb4e645d26244dfda71 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:49 +0000
Subject: [PATCH 20/23] t/integration.t: switch PUT tests to MD5, reuse buffers

MD5 is faster, and these tests aren't meant to be secure,
they're just for checking for data corruption.
Furthermore, Content-MD5 is a supported HTTP trailer and
we can verify that here to obsolete other tests.

Furthermore, we can reuse buffers on env['rack.input'].read
calls to avoid malloc(3) and GC overhead.

Combined, these give roughly a 3% speedup for t/integration.t
on my system.
---
 t/integration.ru   | 20 +++++++++++++++-----
 t/integration.t    |  5 ++---
 t/preread_input.ru | 17 ++++++++++++-----
 3 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/t/integration.ru b/t/integration.ru
index dab384d..086126a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -55,8 +55,8 @@ def env_dump(env)
 def rack_input_tests(env)
   return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
   cap = 16384
-  require 'digest/sha1'
-  digest = Digest::SHA1.new
+  require 'digest/md5'
+  dig = Digest::MD5.new
   input = env['rack.input']
   case env['PATH_INFO']
   when '/rack_input/size_first'; input.size
@@ -68,11 +68,21 @@ def rack_input_tests(env)
   if buf = input.read(rand(cap))
     begin
       raise "#{buf.size} > #{cap}" if buf.size > cap
-      digest.update(buf)
+      dig.update(buf)
     end while input.read(rand(cap), buf)
+    buf.clear # remove this call if Ruby ever gets escape analysis
   end
-  [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
-    [ digest.hexdigest ] ]
+  h = { 'content-type' => 'text/plain' }
+  if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+    cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+    cmd5_bin = cmd5_b64.unpack('m')[0]
+    if cmd5_bin != dig.digest
+      h['content-length'] = cmd5_b64.size.to_s
+      return [ 500, h, [ cmd5_b64 ] ]
+    end
+  end
+  h['content-length'] = '32'
+  [ 200, h, [ dig.hexdigest ] ]
 end
 
 run(lambda do |env|
diff --git a/t/integration.t b/t/integration.t
index 8433497..38a9675 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,7 +27,6 @@ my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
 		my $bs = $opt{bs} // 16384;
-		require Digest::MD5;
 		my $dig = Digest::MD5->new;
 		print $out <<EOM;
 PUT $path HTTP/1.1\r
@@ -186,8 +185,8 @@ SKIP: {
 	CORE::open(my $rh, '<', 't/random_blob') or
 		skip "t/random_blob not generated $!", 1;
 	$blob_size = -s $rh;
-	require Digest::SHA;
-	$blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+	require Digest::MD5;
+	$blob_hash = Digest::MD5->new->addfile($rh)->hexdigest;
 
 	my $ck_hash = sub {
 		my ($sub, $path, %opt) = @_;
diff --git a/t/preread_input.ru b/t/preread_input.ru
index f0a1748..18af221 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,15 +1,22 @@
 #\-E none
-require 'digest/sha1'
+require 'digest/md5'
 require 'unicorn/preread_input'
 use Unicorn::PrereadInput
 nr = 0
 run lambda { |env|
   $stderr.write "app dispatch: #{nr += 1}\n"
   input = env["rack.input"]
-  dig = Digest::SHA1.new
-  while buf = input.read(16384)
-    dig.update(buf)
+  dig = Digest::MD5.new
+  if buf = input.read(16384)
+    begin
+      dig.update(buf)
+    end while input.read(16384, buf)
+    buf.clear # remove this call if Ruby ever gets escape analysis
+  end
+  if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+    cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+    cmd5_bin = cmd5_b64.unpack('m')[0]
+    return [500, {}, [ cmd5_b64 ] ] if cmd5_bin != dig.digest
   end
-
   [ 200, {}, [ dig.hexdigest ] ]
 }

[-- Attachment #22: 0021-tests-move-test_upload.rb-tests-to-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 12280 bytes --]

From 181e4b5b6339fc5e9c3ad7d3690b736f6bd038aa Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:50 +0000
Subject: [PATCH 21/23] tests: move test_upload.rb tests to t/integration.t

The overread tests are ported over, and checksumming alone
is enough to guard against data corruption.

Randomizing the size of `read' calls on the client side will
shake out any boundary bugs on the server side.
---
 t/integration.t          |  32 ++++-
 test/unit/test_upload.rb | 301 ---------------------------------------
 2 files changed, 27 insertions(+), 306 deletions(-)
 delete mode 100644 test/unit/test_upload.rb

diff --git a/t/integration.t b/t/integration.t
index 38a9675..a568758 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -26,7 +26,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
 my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
-		my $bs = $opt{bs} // 16384;
 		my $dig = Digest::MD5->new;
 		print $out <<EOM;
 PUT $path HTTP/1.1\r
@@ -36,7 +35,7 @@ Trailer: Content-MD5\r
 EOM
 		my ($buf, $r);
 		while (1) {
-			$r = read($in, $buf, $bs);
+			$r = read($in, $buf, 999 + int(rand(0xffff)));
 			last if $r == 0;
 			printf $out "%x\r\n", length($buf);
 			print $out $buf, "\r\n";
@@ -46,15 +45,15 @@ EOM
 	},
 	identity => sub {
 		my ($in, $out, $path, %opt) = @_;
-		my $bs = $opt{bs} // 16384;
 		my $clen = $opt{-s} // -s $in;
 		print $out <<EOM;
 PUT $path HTTP/1.0\r
 Content-Length: $clen\r
 \r
 EOM
-		my ($buf, $r, $len);
+		my ($buf, $r, $len, $bs);
 		while ($clen) {
+			$bs = 999 + int(rand(0xffff));
 			$len = $clen > $bs ? $bs : $clen;
 			$r = read($in, $buf, $len);
 			die 'premature EOF' if $r == 0;
@@ -192,8 +191,10 @@ SKIP: {
 		my ($sub, $path, %opt) = @_;
 		seek($rh, 0, SEEK_SET);
 		$c = tcp_start($srv);
-		$c->autoflush(0);
+		$c->autoflush($opt{sync} // 0);
 		$PUT{$sub}->($rh, $c, $path, %opt);
+		defined($opt{overwrite}) and
+			print { $c } ('x' x $opt{overwrite});
 		$c->flush or die $!;
 		($status, $hdr) = slurp_hdr($c);
 		is(readline($c), $blob_hash, "$sub $path");
@@ -205,6 +206,27 @@ SKIP: {
 	$ck_hash->('chunked_md5', '/rack_input/size_first');
 	$ck_hash->('chunked_md5', '/rack_input/rewind_first');
 
+	$ck_hash->('identity', '/rack_input', -s => $blob_size, sync => 1);
+	$ck_hash->('chunked_md5', '/rack_input', sync => 1);
+
+	# ensure small overwrites don't get checksummed
+	$ck_hash->('identity', '/rack_input', -s => $blob_size,
+			overwrite => 1); # one extra byte
+
+	# excessive overwrite truncated
+	$c = tcp_start($srv);
+	$c->autoflush(0);
+	print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 1\r\n\r\n";
+	if (1) {
+		local $SIG{PIPE} = 'IGNORE';
+		my $buf = "\0" x 8192;
+		my $n = 0;
+		my $end = time + 5;
+		$! = 0;
+		while (print $c $buf and time < $end) { ++$n }
+		ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+	}
+	undef $c;
 
 	$curl // skip 'no curl found in PATH', 1;
 
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
deleted file mode 100644
index 76e6c1c..0000000
--- a/test/unit/test_upload.rb
+++ /dev/null
@@ -1,301 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
-require './test/test_helper'
-require 'digest/md5'
-
-include Unicorn
-
-class UploadTest < Test::Unit::TestCase
-
-  def setup
-    @addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
-    @port = unused_port
-    @hdr = {'Content-Type' => 'text/plain', 'Content-Length' => '0'}
-    @bs = 4096
-    @count = 256
-    @server = nil
-
-    # we want random binary data to test 1.9 encoding-aware IO craziness
-    @random = File.open('/dev/urandom','rb')
-    @sha1 = Digest::SHA1.new
-    @sha1_app = lambda do |env|
-      input = env['rack.input']
-      resp = {}
-
-      @sha1.reset
-      while buf = input.read(@bs)
-        @sha1.update(buf)
-      end
-      resp[:sha1] = @sha1.hexdigest
-
-      # rewind and read again
-      input.rewind
-      @sha1.reset
-      while buf = input.read(@bs)
-        @sha1.update(buf)
-      end
-
-      if resp[:sha1] == @sha1.hexdigest
-        resp[:sysread_read_byte_match] = true
-      end
-
-      if expect_size = env['HTTP_X_EXPECT_SIZE']
-        if expect_size.to_i == input.size
-          resp[:expect_size_match] = true
-        end
-      end
-      resp[:size] = input.size
-      resp[:content_md5] = env['HTTP_CONTENT_MD5']
-
-      [ 200, @hdr.merge({'X-Resp' => resp.inspect}), [] ]
-    end
-  end
-
-  def teardown
-    redirect_test_io { @server.stop(false) } if @server
-    @random.close
-    reset_sig_handlers
-  end
-
-  def test_put
-    start_server(@sha1_app)
-    sock = tcp_socket(@addr, @port)
-    sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
-    @count.times do |i|
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      sock.syswrite(buf)
-    end
-    read = sock.read.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal length, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-  end
-
-  def test_put_content_md5
-    md5 = Digest::MD5.new
-    start_server(@sha1_app)
-    sock = tcp_socket(@addr, @port)
-    sock.syswrite("PUT / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n" \
-                  "Trailer: Content-MD5\r\n\r\n")
-    @count.times do |i|
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      md5.update(buf)
-      sock.syswrite("#{'%x' % buf.size}\r\n")
-      sock.syswrite(buf << "\r\n")
-    end
-    sock.syswrite("0\r\n")
-
-    content_md5 = [ md5.digest! ].pack('m').strip.freeze
-    sock.syswrite("Content-MD5: #{content_md5}\r\n\r\n")
-    read = sock.read.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal length, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-    assert_equal content_md5, resp[:content_md5]
-  end
-
-  def test_put_trickle_small
-    @count, @bs = 2, 128
-    start_server(@sha1_app)
-    assert_equal 256, length
-    sock = tcp_socket(@addr, @port)
-    hdr = "PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n"
-    @count.times do
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      hdr << buf
-      sock.syswrite(hdr)
-      hdr = ''
-      sleep 0.6
-    end
-    read = sock.read.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal length, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-  end
-
-  def test_put_keepalive_truncates_small_overwrite
-    start_server(@sha1_app)
-    sock = tcp_socket(@addr, @port)
-    to_upload = length + 1
-    sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{to_upload}\r\n\r\n")
-    @count.times do
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      sock.syswrite(buf)
-    end
-    sock.syswrite('12345') # write 4 bytes more than we expected
-    @sha1.update('1')
-
-    buf = sock.readpartial(4096)
-    while buf !~ /\r\n\r\n/
-      buf << sock.readpartial(4096)
-    end
-    read = buf.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal to_upload, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-  end
-
-  def test_put_excessive_overwrite_closed
-    tmp = Tempfile.new('overwrite_check')
-    tmp.sync = true
-    start_server(lambda { |env|
-      nr = 0
-      while buf = env['rack.input'].read(65536)
-        nr += buf.size
-      end
-      tmp.write(nr.to_s)
-      [ 200, @hdr, [] ]
-    })
-    sock = tcp_socket(@addr, @port)
-    buf = ' ' * @bs
-    sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
-
-    @count.times { sock.syswrite(buf) }
-    assert_raise(Errno::ECONNRESET, Errno::EPIPE) do
-      ::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) }
-    end
-    sock.gets
-    tmp.rewind
-    assert_equal length, tmp.read.to_i
-  end
-
-  # Despite reading numerous articles and inspecting the 1.9.1-p0 C
-  # source, Eric Wong will never trust that we're always handling
-  # encoding-aware IO objects correctly.  Thus this test uses shell
-  # utilities that should always operate on files/sockets on a
-  # byte-level.
-  def test_uncomfortable_with_onenine_encodings
-    # POSIX doesn't require all of these to be present on a system
-    which('curl') or return
-    which('sha1sum') or return
-    which('dd') or return
-
-    start_server(@sha1_app)
-
-    tmp = Tempfile.new('dd_dest')
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=#{@bs}", "count=#{@count}"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
-    assert $?.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-
-    # small StringIO path
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=1024", "count=1"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
-    assert $?.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-  end
-
-  def test_chunked_upload_via_curl
-    # POSIX doesn't require all of these to be present on a system
-    which('curl') or return
-    which('sha1sum') or return
-    which('dd') or return
-
-    start_server(@sha1_app)
-
-    tmp = Tempfile.new('dd_dest')
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=#{@bs}", "count=#{@count}"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    cmd = "curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
-           -isSf --no-buffer -T- " \
-          "http://#@addr:#@port/"
-    resp = Tempfile.new('resp')
-    resp.sync = true
-
-    rd, wr = IO.pipe.each do |io|
-      io.sync = io.close_on_exec = true
-    end
-    pid = spawn(*cmd, { 0 => rd, 1 => resp })
-    rd.close
-
-    tmp.rewind
-    @count.times { |i|
-      wr.write(tmp.read(@bs))
-      sleep(rand / 10) if 0 == i % 8
-    }
-    wr.close
-    pid, status = Process.waitpid2(pid)
-
-    resp.rewind
-    resp = resp.read
-    assert status.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-    assert_match(/expect_size_match/, resp)
-  end
-
-  def test_curl_chunked_small
-    # POSIX doesn't require all of these to be present on a system
-    which('curl') or return
-    which('sha1sum') or return
-    which('dd') or return
-
-    start_server(@sha1_app)
-
-    tmp = Tempfile.new('dd_dest')
-    # small StringIO path
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=1024", "count=1"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    resp = `curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
-            -isSf --no-buffer -T- http://#@addr:#@port/ < #{tmp.path}`
-    assert $?.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-    assert_match(/expect_size_match/, resp)
-  end
-
-  private
-
-  def length
-    @bs * @count
-  end
-
-  def start_server(app)
-    redirect_test_io do
-      @server = HttpServer.new(app, :listeners => [ "#{@addr}:#{@port}" ] )
-      @server.start
-    end
-  end
-
-end

[-- Attachment #23: 0022-drop-redundant-IO-close_on_exec-false-calls.patch --]
[-- Type: text/x-diff, Size: 891 bytes --]

From 841b9e756beb1aa00d0f89097a808adcbbf45397 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:51 +0000
Subject: [PATCH 22/23] drop redundant IO#close_on_exec=false calls

Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
---
 lib/unicorn/http_server.rb | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 348e745..dd92b38 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -472,10 +472,7 @@ def worker_spawn(worker)
 
   def listener_sockets
     listener_fds = {}
-    LISTENERS.each do |sock|
-      sock.close_on_exec = false
-      listener_fds[sock.fileno] = sock
-    end
+    LISTENERS.each { |sock| listener_fds[sock.fileno] = sock }
     listener_fds
   end
 

[-- Attachment #24: 0023-LISTEN_FDS-inherited-sockets-are-immortal-across-SIG.patch --]
[-- Type: text/x-diff, Size: 3229 bytes --]

From 6ff8785c9277c5978e6dc01cb1b3da25d6bae2db Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:52 +0000
Subject: [PATCH 23/23] LISTEN_FDS-inherited sockets are immortal across SIGHUP

When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.

I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
---
 lib/unicorn/http_server.rb  | 7 ++++++-
 t/client_body_buffer_size.t | 1 -
 t/integration.t             | 1 -
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index dd92b38..f1b4a54 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -77,6 +77,7 @@ def initialize(app, options = {})
     options[:use_defaults] = true
     self.config = Unicorn::Configurator.new(options)
     self.listener_opts = {}
+    @immortal = [] # immortal inherited sockets from systemd
 
     # We use @self_pipe differently in the master and worker processes:
     #
@@ -158,6 +159,7 @@ def listeners=(listeners)
     end
     set_names = listener_names(listeners)
     dead_names.concat(cur_names - set_names).uniq!
+    dead_names -= @immortal.map { |io| sock_name(io) }
 
     LISTENERS.delete_if do |io|
       if dead_names.include?(sock_name(io))
@@ -807,17 +809,20 @@ def inherit_listeners!
     # inherit sockets from parents, they need to be plain Socket objects
     # before they become Kgio::UNIXServer or Kgio::TCPServer
     inherited = ENV['UNICORN_FD'].to_s.split(',')
+    immortal = []
 
     # emulate sd_listen_fds() for systemd
     sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
     if sd_pid.to_i == $$ # n.b. $$ can never be zero
       # 3 = SD_LISTEN_FDS_START
-      inherited.concat((3...(3 + sd_fds.to_i)).to_a)
+      immortal = (3...(3 + sd_fds.to_i)).to_a
+      inherited.concat(immortal)
     end
     # to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
 
     inherited.map! do |fd|
       io = Socket.for_fd(fd.to_i)
+      @immortal << io if immortal.include?(fd)
       io.autoclose = false
       io = server_cast(io)
       set_server_sockopt(io, listener_opts[sock_name(io)])
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index b1a99f3..3067f28 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -36,7 +36,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
 seek($conf_fh, 0, SEEK_SET);
 truncate($conf_fh, 0);
 print $conf_fh <<EOM;
-listen "$host_port" # TODO: remove this requirement for SIGHUP
 after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
 EOM
 $ar->do_kill('HUP');
diff --git a/t/integration.t b/t/integration.t
index a568758..bb2ab51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -17,7 +17,6 @@ my $u1 = "$tmpdir/u1";
 print $conf_fh <<EOM;
 early_hints true
 listen "$u1"
-listen "$host_port" # TODO: remove this requirement for SIGHUP
 EOM
 my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');

^ permalink raw reply related	[relevance 1%]

* Re: unicorn worker being killed issue
  @ 2023-05-10 21:34  4%   ` subashkc1
  0 siblings, 0 replies; 200+ results
From: subashkc1 @ 2023-05-10 21:34 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

Hi Eric,

Thanks for you reply, I'll try to answer all questions below



> subashkc1 subashkc1@protonmail.com wrote:
>
> > I have a weird issue that I'm having trouble debugging. I've
> > created a sample bare bones rails app where there issue is
> > reproducible https://github.com/subashkc/sidekiq_webui_issue.
>
>
> That's too big and having https://enterprise.contribsys.com/ as
> a source in your Gemfile means I'm probably not able/allowed
> to inspect those gems.
>

I've removed the sidekiq pro form gemfile, it also happens with just the sidekiq gem

> Some general questions:
>
> * how are you starting unicorn and what is the config file?
> (mainly, is `preload_app' enabled? and any hooks?)` preload_app true' is the most common source of problems if
> the rest of your application (including all dependencies)
> isn't equipped to handle fork.

unicorn is started just with `bundle exec unicorn` from a Procfile using foreman, no other config options so its just 1 unicorn worker and its a very bare-bones rails app to reproduce the issue, all I did was add sidekiq gem, and load up the sidekiq webui to see the issue in action
>
> * What are the RACK_ENV / RAILS_ENV?
>

It is happening locally, so all env are development

> `unicorn_rails' is actually pointless nowadays, and the plain` unicorn' command is highly recommended.

I'm only using the unicorn gem, not touching unicorn_rails at all

>
> Use a single worker for debugging, especially if you want to
> strace/truss the process.

Yeah just using 1 unicorn worker with all the default configs, (this repo doesn't have a unicorn config file so all defaults)

>
> Fwiw, I haven't touched Rails roughly a decade (and barely at
> that). My only web apps for the past 15 years or so are on bare
> Rack and meant to be consumed via curl or w3m (nothing involving
> JavaScript or fancy browsers).
>
> Maybe others watching this mailbox can chime in....
>
> > After running the rails app, and load up the sidekiq web UI,
> > if I click on any tabs e.g. busy/queues then the request is
> > served right away with a 200 but it seems there is a 2nd
> > request (can't tell where from) which eventually times out and
> > kills the unicorn worker.
>
>
> I suspect a 2nd request comes from some JS code. Check for a
> network debugging tool included with your browser if it has one.
>

Yeah that's what I can't figure out, I can't see any other requests in the browser network tools

> Or use a single unicorn worker, then strace/truss that worker.
> Something like `strace -p $PID_OF_WORKER -f -v -s5000 -o /tmp/dump`
> (assuming Linux)

I did this, but the output was so verbose I couldn't make any sense of it as I wasn't sure what to look for

>
> Are you able to reproduce the issue with JS disabled?
> Preferably just via curl to minimize surprises.

Ok, it doesn't happen if I do the same request using curl, just doing `curl localhost:8080/sidekiq/busy` and if I do this right after clicking on the tab in the UI, this request doesn't get served right away, it waits until the request from the UI times out >60s, kills the unicorn worker and then the curl req is served

>
> > I originally opened the issue with sidekiq repo but Mike
> > (sidekiq owner) wasn't able to find anything wrong with
> > sidekiq code and he suggested it seemed unicorn was double
> > reading the request and the 2nd one dies after 60s timeout.
>
>
> Double-reading a request is impossible unless there's something
> very broken in Ruby or your networking stack. There'd be a
> hundreds of reports about it if that were true if Ruby or
> unicorn were double-reading requests.
>
> Are you going through some firewalls, or caching/scanning
> proxies which might send a request twice? I've seen that before.
>

No, this is all happening locally on my dev machine, no firewalls or proxies involved

> I have seen some folks accidentally enable multiple loggers,
> so it looks like a single request is happening multiple times.
>

No, just a very basic rails app with the default logger, haven't enabled or added anything else

> strace is usually my tool for debugging stuff like this; along
> with sprinkling `warn "#{LINE} ..."' statements throughout.
> And then perhaps commenting out lots of code and disabling
> as many dependencies as possible.

Yeah it is a rails app with just the sidekiq gem, no other dependencies and no code to disable, haven't added anything.

Let me know if that gives you some hints or if there are other things that I can try or check

Warm Regards

^ permalink raw reply	[relevance 4%]

* [PATCH] doc: add POP3 archive link, favor AUTH=ANONYMOUS for IMAP
@ 2022-08-16 18:44  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-16 18:44 UTC (permalink / raw)
  To: unicorn-public

POP3 access is less straightforward since it requires a UUID
cookie embedded into the username to track deletes, and also
requires embedding the mailbox name into the username (since
POP3 has a 1:1 user:folder mapping).  Of course, a Tor .onion
endpoint is supported for anonymity.

AUTH=ANONYMOUS is recommended for IMAP since bots were attempting
to use compromised logins to scrape the public mail archives for
private info and burning up my CPU.
---
 .olddoc.yml |  4 ++--
 ISSUES      |  7 ++++---
 README      | 11 ++++++-----
 3 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/.olddoc.yml b/.olddoc.yml
index 10ddc07..9780a83 100644
--- a/.olddoc.yml
+++ b/.olddoc.yml
@@ -14,8 +14,8 @@ noindex:
 - unicorn_rails_1
 public_email: unicorn-public@yhbt.net
 imap_url:
-- imaps://yhbt.net/inbox.comp.lang.ruby.unicorn.0
-- imap://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
+- imaps://;AUTH=ANONYMOUS@yhbt.net/inbox.comp.lang.ruby.unicorn.0
+- imap://;AUTH=ANONYMOUS@7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
 nntp_url:
 - nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
 - nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn
diff --git a/ISSUES b/ISSUES
index 015de37..083b1c8 100644
--- a/ISSUES
+++ b/ISSUES
@@ -81,14 +81,15 @@ Mail is publicly-archived, SMTP subscription is discouraged to avoid
 servers being a single-point-of-failure, so Cc: all participants.
 
 The HTTP(S) archives have links to per-thread Atom feeds and downloadable
-mboxes.  Read-only IMAP(S) folders and NNTP(S) newsgroups are also available.
+mboxes.  Read-only IMAP(S) folders, POP3, and NNTP(S) newsgroups are available.
 
 * https://yhbt.net/unicorn-public/
 * http://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/unicorn-public/
-* imaps://yhbt.net/inbox.comp.lang.ruby.unicorn.0
-* imap://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
+* imaps://;AUTH=ANONYMOUS@yhbt.net/inbox.comp.lang.ruby.unicorn.0
+* imap://;AUTH=ANONYMOUS@7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
 * nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
 * nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
+* https://yhbt.net/unicorn-public/_/text/help/#pop3
 
 Full Atom feeds:
 * https://yhbt.net/unicorn-public/new.atom
diff --git a/README b/README
index 22c27e8..5411003 100644
--- a/README
+++ b/README
@@ -140,11 +140,12 @@ Read-only NNTP access is available at:
 nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn and
 nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
 
-Read-only IMAP access is also avaialble at:
-imaps://yhbt.net/inbox.comp.lang.ruby.unicorn.0 and
-imap://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
-The AUTH=ANONYMOUS mechanism is supported, as is any username+password
-combination.
+Read-only IMAP access is also available at:
+imaps://;AUTH=ANONYMOUS@yhbt.net/inbox.comp.lang.ruby.unicorn.0 and
+imap://;AUTH=ANONYMOUS@7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.lang.ruby.unicorn.0
+
+Archives are also available over POP3, instructions at:
+https://yhbt.net/unicorn-public/_/text/help/#pop3
 
 For the latest on unicorn releases, you may also finger us at
 unicorn@yhbt.net or check our NEWS page (and subscribe to our Atom

^ permalink raw reply related	[relevance 4%]

* Bus Error with Unicorn 6.0 on OpenBSD-current (6.9) with Ruby 3.0 in minimal rails application
  2021-04-04 20:03  0%         ` Eric Wong
@ 2021-04-07 21:19  0%           ` vtamara
  0 siblings, 0 replies; 200+ results
From: vtamara @ 2021-04-07 21:19 UTC (permalink / raw)
  To: Eric Wong; +Cc: Jeremy Evans, unicorn-public, ports

Good evening

I found that the problem happens also:

1. In OpenBSD-current (what will be 6.9 soon) --for that reason I copy 
to ports-openbsd
2. With what I consider the minimal rails application,  started with the 
flag --minimal, running with sqlite, i.e:

rails new --minimal ap61

The configuration of gems in /var/www/bundler and the configuration of 
nginx is as I explained in the previous email  
https://yhbt.net/unicorn-public/20210404200358.GA32719@dcvr/T/#mcf469c2eb443db6c9e4fee55afa3b252606c27f2 
but  changing cor1440 with ap61.

In my previous email I forgot some details, like how to create self 
signed certificate and how to install nginx and ruby, here they go:

doas pkg_add nginx
doas pkg_add ruby-3.0.0p0
  ln -sf /usr/local/bin/ruby30 /usr/local/bin/ruby
  ln -sf /usr/local/bin/erb30 /usr/local/bin/erb
  ln -sf /usr/local/bin/irb30 /usr/local/bin/irb
  ln -sf /usr/local/bin/rdoc30 /usr/local/bin/racc
  ln -sf /usr/local/bin/rdoc30 /usr/local/bin/rdoc
  ln -sf /usr/local/bin/rdoc30 /usr/local/bin/rbs
  ln -sf /usr/local/bin/ri30 /usr/local/bin/ri
  ln -sf /usr/local/bin/rake30 /usr/local/bin/rake
  ln -sf /usr/local/bin/gem30 /usr/local/bin/gem
  ln -sf /usr/local/bin/bundle30 /usr/local/bin/bundle
  ln -sf /usr/local/bin/bundler30 /usr/local/bin/bundler
  ln -sf /usr/local/bin/typeprof30 /usr/local/bin/typeprof


# Creating self signed certificate
doas openssl genrsa -out /etc/ssl/private/server.key 2048
openssl req -new -key /etc/ssl/private/server.key \
   -out /etc/ssl/private/server.csr
openssl x509 -req -days 3650 -in /etc/ssl/private/server.csr \
   -signkey /etc/ssl/private/server.key -out /etc/ssl/server.crt

# Installing sqlite3
doas pkg_add sqlite3
doas gem install --install-dir /var/www/bundler/ruby/3.0/ sqlite3

# Installing rails globally
doas gem install rails
doas ln -s /usr/local/bin/rails30 /usr/local/bin/rails

rails new --minimal ap61
mv ap61 /var/www/htdocs/
cd /var/www/htdocs/ap61

# Configuring unicorn
cat <<EOF >> Gemfile
group :production do
   gem 'unicorn'
end
EOF
doas gem install --install-dir /var/www/bundler/ruby/3.0 unicorn
bundle
cat <<EOF > config/unicorn.rb
listen 2011
APP_PATH="/var/www/htdocs/ap61"
working_directory APP_PATH
worker_processes 2
pid APP_PATH + "/tmp/pids/unicorn.pid"
stderr_path APP_PATH + "/log/unicorn.log"
stdout_path APP_PATH + "/log/unicorn.log"
EOF

# Start the application
bundle exec /usr/local/bin/unicorn_rails -c config/unicorn.rb -E 
production -D


# Try to see the initial rails screen
doas pkg_add w3m
w3m https://127.0.0.1/ap61

It will not present the presentation screen, stopping w3m and checking 
the logfile I see the samen Segmentation faults:

I, [2021-04-07T15:27:34.289298 #75098]  INFO -- : listening on 
addr=0.0.0.0:2011 fd=12
I, [2021-04-07T15:27:34.290752 #75098]  INFO -- : worker=0 spawning...
I, [2021-04-07T15:27:34.303205 #75098]  INFO -- : worker=1 spawning...
I, [2021-04-07T15:27:34.308574 #98078]  INFO -- : worker=0 spawned 
pid=98078
I, [2021-04-07T15:27:34.310157 #98078]  INFO -- : Refreshing Gem list
I, [2021-04-07T15:27:34.329730 #75098]  INFO -- : master process ready
I, [2021-04-07T15:27:34.390398 #14713]  INFO -- : worker=1 spawned 
pid=14713
I, [2021-04-07T15:27:34.391984 #14713]  INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:86: [BUG] 
Segmentation fault at 0x0000000000000000
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]



-- Control frame information 
-----------------------------------------------
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG] 
Segmentation fault at 0x0000000000000001
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]


-- Control frame information 
-----------------------------------------------
E, [2021-04-07T15:27:50.430649 #75098] ERROR -- : reaped 
#<Process::Status: pid 98078 SIGSEGV (signal 11) (core dumped)> worker=0
I, [2021-04-07T15:27:50.430976 #75098]  INFO -- : worker=0 spawning...
I, [2021-04-07T15:27:50.439973 #19064]  INFO -- : worker=0 spawned 
pid=19064
I, [2021-04-07T15:27:50.441607 #19064]  INFO -- : Refreshing Gem list
E, [2021-04-07T15:27:51.490681 #75098] ERROR -- : reaped 
#<Process::Status: pid 14713 SIGSEGV (signal 11) (core dumped)> worker=1
I, [2021-04-07T15:27:51.491040 #75098]  INFO -- : worker=1 spawning...
I, [2021-04-07T15:27:51.500370 #66270]  INFO -- : worker=1 spawned 
pid=66270
I, [2021-04-07T15:27:51.501983 #66270]  INFO -- : Refreshing Gem list
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG] 
Segmentation fault at 0x0000030bbfc8d6f5
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]



-- Control frame information 
-----------------------------------------------
/var/www/bundler/ruby/3.0/gems/unicorn-6.0.0/lib/unicorn.rb:80: [BUG] 
Segmentation fault at 0x0000000000000001
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-openbsd]


-- Control frame information 
-----------------------------------------------
E, [2021-04-07T15:28:05.676716 #75098] ERROR -- : reaped 
#<Process::Status: pid 19064 SIGSEGV (signal 11) (core dumped)> worker=0
I, [2021-04-07T15:28:05.677110 #75098]  INFO -- : worker=0 spawning...
I, [2021-04-07T15:28:05.685956 #32548]  INFO -- : worker=0 spawned 
pid=32548
I, [2021-04-07T15:28:05.687658 #32548]  INFO -- : Refreshing Gem list
E, [2021-04-07T15:28:06.272044 #75098] ERROR -- : reaped 
#<Process::Status: pid 66270 SIGSEGV (signal 11) (core dumped)> worker=1
I, [2021-04-07T15:28:06.272459 #75098]  INFO -- : worker=1 spawning...
I, [2021-04-07T15:28:06.281695 #65711]  INFO -- : worker=1 spawned 
pid=65711
I, [2021-04-07T15:28:06.283312 #65711]  INFO -- : Refreshing Gem list



--------

A core file is generated:
$ ls -l ruby30.core
-rw-------  1 servidor  servidor  44485672 Apr  7 15:41 ruby30.core


Checking the trace

$ doas pkg_add gdb
$ egdb /usr/local/bin/ruby30 ruby30.core
...
  [New process 601176]
[New process 412220]
[New process 332011]
Core was generated by `ruby30'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000030bbfc70af8 in rb_vm_frame_method_entry ()
    from /usr/local/lib/libruby30.so
[Current thread is 1 (process 601176)]
(gdb) bt
#0  0x0000030bbfc70af8 in rb_vm_frame_method_entry ()
    from /usr/local/lib/libruby30.so
#1  0x0000030bbfca04cb in rb_vmdebug_stack_dump_raw ()
    from /usr/local/lib/libruby30.so
#2  0x0000030bbfca118a in rb_vm_bugreport () from 
/usr/local/lib/libruby30.so
#3  0x0000030bbfabd021 in rb_bug_for_fatal_signal ()
    from /usr/local/lib/libruby30.so
#4  0x0000030bbfbf4470 in sigsegv () from /usr/local/lib/libruby30.so
#5  <signal handler called>
#6  0x0000030bbfc78bb8 in vm_exec_core () from 
/usr/local/lib/libruby30.so
#7  0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#8  0x0000030bbfb20b0b in load_iseq_eval () from 
/usr/local/lib/libruby30.so
#9  0x0000030bbfb1ebaf in rb_load_internal () from 
/usr/local/lib/libruby30.so
#10 0x0000030bbfb20216 in rb_f_load () from /usr/local/lib/libruby30.so
#11 0x0000030bbfc9717e in vm_call_cfunc_with_frame ()
    from /usr/local/lib/libruby30.so
#12 0x0000030bbfc8e955 in vm_sendish () from /usr/local/lib/libruby30.so
#13 0x0000030bbfc77292 in vm_exec_core () from 
/usr/local/lib/libruby30.so
#14 0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#15 0x0000030bbfb20b0b in load_iseq_eval () from 
/usr/local/lib/libruby30.so
#16 0x0000030bbfb1ebaf in rb_load_internal () from 
/usr/local/lib/libruby30.so
#17 0x0000030bbfb20216 in rb_f_load () from /usr/local/lib/libruby30.so
#18 0x0000030bbfc9717e in vm_call_cfunc_with_frame ()
    from /usr/local/lib/libruby30.so
#19 0x0000030bbfc8e955 in vm_sendish () from /usr/local/lib/libruby30.so
#20 0x0000030bbfc77292 in vm_exec_core () from 
/usr/local/lib/libruby30.so
#21 0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#22 0x0000030bbfb20b0b in load_iseq_eval () from 
/usr/local/lib/libruby30.so
#23 0x0000030bbfb1ebaf in rb_load_internal () from 
/usr/local/lib/libruby30.so
#24 0x0000030bbfb20216 in rb_f_load () from /usr/local/lib/libruby30.so
#25 0x0000030bbfc9717e in vm_call_cfunc_with_frame ()
    from /usr/local/lib/libruby30.so
#26 0x0000030bbfc8e955 in vm_sendish () from /usr/local/lib/libruby30.so
#27 0x0000030bbfc77292 in vm_exec_core () from 
/usr/local/lib/libruby30.so
#28 0x0000030bbfc89d95 in rb_vm_exec () from /usr/local/lib/libruby30.so
#29 0x0000030bbfac5044 in rb_ec_exec_node () from 
/usr/local/lib/libruby30.so
#30 0x0000030bbfac4eca in ruby_run_node () from 
/usr/local/lib/libruby30.so
#31 0x0000030997a42cec in main ()


I still have not tried to use the .gdbinit file, but hope to do it soon.


Blessings.

El 2021-04-04 16:03, Eric Wong escribió:
> Jeremy Evans <code@jeremyevans.net> wrote:
>> Unfortunately, given the required amount of setup, I don't think I'll
>> have time to look into this.  The best way to debug this is to start
>> removing code/dependencies piece by piece until the problem goes away.
>> Once the problem goes away, add that code back, and make sure it fails
>> again, then start removing other code/dependencies.  Repeat this 
>> process
>> until you have the most minimal example, where removing any additional
>> code/dependencies will stop the code from failing.
> 
> The above is exactly what I would do, too.
> 
> I would also try to correlate the time of Bus Errors with logs
> of when + which requests are failing, and try replaying only
> those requests to narrow things down.
> 
> Also, do you have core dumps enabled?  Enable those
> ("ulimit -c unlimited" in sh), and run gdb on them as you get
> them.  The .gdbinit in the source tree of the Ruby version
> you're using is helpful not just for Ruby itself, but
> also C extensions, too.  I'm no a gdb expert, but it's
> well-documented and has built-in help.

-- 
Dios, gracias por tu amor infinito.
--
   Vladimir Támara Patiño.  http://vtamara.pasosdeJesus.org/
   http://www.pasosdejesus.org/dominio_publico_colombia.html

^ permalink raw reply	[relevance 0%]

* Re: Bus Error with Unicorn 6.0 on OpenBSD/adJ 6.8 with Ruby 3.0
  2021-04-04 16:43  5%       ` Jeremy Evans
@ 2021-04-04 20:03  0%         ` Eric Wong
  2021-04-07 21:19  0%           ` Bus Error with Unicorn 6.0 on OpenBSD-current (6.9) with Ruby 3.0 in minimal rails application vtamara
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-04 20:03 UTC (permalink / raw)
  To: vtamara; +Cc: Jeremy Evans, unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Unfortunately, given the required amount of setup, I don't think I'll
> have time to look into this.  The best way to debug this is to start
> removing code/dependencies piece by piece until the problem goes away.
> Once the problem goes away, add that code back, and make sure it fails
> again, then start removing other code/dependencies.  Repeat this process
> until you have the most minimal example, where removing any additional
> code/dependencies will stop the code from failing.

The above is exactly what I would do, too.

I would also try to correlate the time of Bus Errors with logs
of when + which requests are failing, and try replaying only
those requests to narrow things down.

Also, do you have core dumps enabled?  Enable those
("ulimit -c unlimited" in sh), and run gdb on them as you get
them.  The .gdbinit in the source tree of the Ruby version
you're using is helpful not just for Ruby itself, but
also C extensions, too.  I'm no a gdb expert, but it's
well-documented and has built-in help.

^ permalink raw reply	[relevance 0%]

* Re: Bus Error with Unicorn 6.0 on OpenBSD/adJ 6.8 with Ruby 3.0
  2021-04-04 11:06  4%     ` vtamara
@ 2021-04-04 16:43  5%       ` Jeremy Evans
  2021-04-04 20:03  0%         ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2021-04-04 16:43 UTC (permalink / raw)
  To: vtamara; +Cc: Eric Wong, unicorn-public

On 04/04 07:06, vtamara wrote:
> Hi
> 
> Thank you very much for the reply and the interest in helping.
> 
> > vtamara: did previous versions of Ruby, unicorn, or OpenBSD work
> > for your app?
> 
> The application was working withouth issue on OpenBSD/adJ amd64 6.7, ruby
> 2.7 and unicorn 6.0.  So there is something in conflict betweeen Unicorn 6.0
> with OpenBSD/adJ 6.8 and ruby 3.0.
> 
> > Is preload_app in use?  If so, does the problem manifest without
> > preload_app?
> 
> No
> 
> > Are you using Ractor or any other new/experimental stuff
> > in Ruby?
> 
> No
> 
> > Can you reproduce the problem with a simple "hello world"-style app?
> 
> I don't have a hello world example at hand, but some projects that present
> this problem when I run them on OpenBSD/adJ amd64 6.8, nginx 1.18.0, node
> 12.16.1, PostgreSQL 13.2, Ruby 3.0, Rails 6.1 and Unicorn 6.0 are:
> 
> https://github.com/pasosdeJesus/cor1440
> https://github.com/pasosdeJesus/cor1440_pdJ
> https://github.com/pasosdeJesus/sivel2
> https://github.com/pasosdeJesus/si_anzorc
> https://github.com/pasosdeJesus/si_asom
> https://github.com/pasosdeJesus/si_codacop
> https://github.com/pasosdeJesus/sivel2_mujeresindigenas/

Unfortunately, given the required amount of setup, I don't think I'll
have time to look into this.  The best way to debug this is to start
removing code/dependencies piece by piece until the problem goes away.
Once the problem goes away, add that code back, and make sure it fails
again, then start removing other code/dependencies.  Repeat this process
until you have the most minimal example, where removing any additional
code/dependencies will stop the code from failing.

Sorry,
Jeremy

> 
> The simplest of them is https://github.com/pasosdeJesus/cor1440, althoug it
> requires:
> * PostgreSQL
> * nginx
> * nodejs
> * yarn. I install it with: doas pkg_add bash; ftp -o-
> https://yarnpkg.com/install.sh | bash
> * A directory /var/www/bundler/ruby/3.0 where I install the gems with the
> permissions of a user (not root) that runs the application, and better
> running:
>   bundle config path /var/www/bundler/.
> * unicorn installed globally with:
>   doas gem install unicorn
>   doas ln -s /usr/local/bin/unicorn_rails30 /usr/local/bin/unicorn
> 
> To create the PostgreSQL user (in the following example sipdes) and a
> database for the production environment  (in the following example
> cor1440gen_pro):
> 
>   doas su - _postgresql
>   createuser -h /var/www/var/run/postgresql/ -U postgres -s sipdes
>   createdb -h /var/www/var/run/postgresql/ -U postgres -O sipdes
> cor1440gen_pro
>   psql -h /var/www/var/run/postgresql/ -U postgres
>   postgres=# alter user sipdes with password 'xyz';
>   postgres=# \q
>   exit
> 
> With nginx suppossng I will run the application in localhost over SSL, what
> I do is to choose an unused port (let's say 2009) and add to the
> /etc/nginx/nginx.conf configuration in its http block:
> 
>  upstream unicorncor1440 {
> 	  server 127.0.0.1:2009 fail_timeout=0;
>   }
> 
> I also add a server block inside the http block with:
> 
>   server {
>       listen 443 ssl;
>       ssl_certificate /etc/ssl/server.crt;
>       ssl_certificate_key /etc/ssl/private/server.key;
>       root /var/www/htdocs/cor1440/;
>       server_name 127.0.0.1;
>       error_log logs/cor1440.rror.log;
> 
>       location /cor1440 {
>         try_files $uri @unicorncor1440;
>       }
> 
>       location @unicorncor1440 {
>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>         proxy_set_header X-Forwarded-Proto $scheme;
>         proxy_set_header Host $http_host;
>         proxy_redirect off;
>         proxy_pass http://unicorncor1440;
>         error_page 500 502 503 504 /500.html;
>         keepalive_timeout 10;
>       }
> 
>       location ^~ /cor1440/assets/ {
>         gzip_static on;
>         expires max;
>         add_header Cache-Control public;
>         root /var/www/htdocs/cor1440/public/;
>       }
> 
>       location ^~ /cor1440/images/ {
>         gzip_static on;
>         expires max;
>         add_header Cache-Control public;
>         root /var/www/htdocs/cor1440/public/;
>       }
> 
>       location ^~ /cor1440/packs/ {
>         gzip_static on;
>         add_header Cache-Control public;
>         root /var/www/htdocs/cor1440/public/;
>       }
>     }
> 
> And restarted nginx:
> 
>   doas rcctl restart nginx
> 
> Then to configure:
> 
>   cd /tmp
>   git clone git@github.com:pasosdeJesus/cor1440
>   doas mv cor1440 /var/www/htdocs/
>   doas chown -R $USER:$USER /var/www/htdocs/cor1440
>   cd /var/www/htdocs/cor1440
>   mkdir .bundle
>   echo '---' > .bundle/config
>   echo 'BUNDLE_PATH: "/var/www/bundler"' >> .bundle/config
>   echo 'BUNDLE_DISABLE_SHARED_GEMS: "true"' >> .bundle/config
>   bundle
> 
> Here some gems will require installing with special permissions for example:
> 
>   doas gem install --install-dir /var/www/bundler/ruby/3.0 unicorn
> 
> Other gems that will require this kind of installation will be:
>   racc nio4r websocket-driver bcrypt bindex msgpack ffi redcarpet kgio
> libxml-ruby pg
>   nokogiri bootsnap puma sassc
> 
> Once bundle runs completely:
> 
>   cp .env.plantilla .env
>   vi .env
> 
> There I set the variables:
> * BD_USUARIO with the database user,
> * BD_CLAVE with the password
> * BD_PRO with the name of the production database
> * CONFIG_HOSTS=127.0.0.1
> * DIRAP with the path to the application
> * RAILS_ENV=production
> * PUERTOUNICORN with the port I choosed and set in nginx.conf
> * USUARIO_AP with the operating system user who will run the application
> 
> Prepare credentials and database:
> 
>   bin/rails credentials:edit
>   RAILS_ENV=production bin/rails credentials:edit
>   bin/railsp db:setup  db:migrate sip:indices
> 
> Precompile assets:
> 
>   CXX=c++ yarn
>   doas ln -s /usr/local/bin/node /tmp  # To overcome bug of node with rails
> on OpenBSD
>   bin/railsp assets:precompile
> 
> Run the application with:
>   SECRET_KEY_BASE=cf46 DIRAP=/var/www/htdocs/cor1440/ bin/u.sh
> 
> Trying to see the application running with a browser in
> https://127.0.0.1/cor1440 will produce the errors described in
> log/unicorn.log.
> 
> Blessings. Thank you.
> 
> El 2021-03-30 00:00, Jeremy Evans escribi??:
> > On 03/30 01:58, Eric Wong wrote:
> > > vtamara <vtamara@pasosdeJesus.org> wrote:
> > > > Hi
> > > >
> > > > Launching a rails application with Ruby 3, Unicorn 6 on OpenBSD/adJ 6.8, I'm
> > > > experiencing a lot of "Bus Error" and "Segmentation Faults" during
> > > > aproximately half hour.
> > > 
> > > +Cc: Jeremy Evans, who is more familiar with OpenBSD and Ruby 3.0
> > > than I
> > > cf. https://yhbt.net/unicorn-public/6acb1c84c7392d7b4a64572b20498549@pasosdeJesus.org/
> > 
> > I'll be happy to look into this if a minimal self-contained reproducible
> > example is posted.  Personally, I run my Ruby apps on OpenBSD amd64 with
> > Unicorn, and haven't seen similar errors.  However, I don't use
> > preload_app and use fairly minimal dependencies.
> > 
> > Thanks,
> > Jeremy
> 
> -- 
> Dios, gracias por tu amor infinito.
> --
>   Vladimir T??mara Pati??o.  http://vtamara.pasosdeJesus.org/
>   http://www.pasosdejesus.org/dominio_publico_colombia.html

^ permalink raw reply	[relevance 5%]

* Re: Bus Error with Unicorn 6.0 on OpenBSD/adJ 6.8 with Ruby 3.0
  @ 2021-04-04 11:06  4%     ` vtamara
  2021-04-04 16:43  5%       ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: vtamara @ 2021-04-04 11:06 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: Eric Wong, unicorn-public

Hi

Thank you very much for the reply and the interest in helping.

> vtamara: did previous versions of Ruby, unicorn, or OpenBSD work
> for your app?

The application was working withouth issue on OpenBSD/adJ amd64 6.7, 
ruby 2.7 and unicorn 6.0.  So there is something in conflict betweeen 
Unicorn 6.0 with OpenBSD/adJ 6.8 and ruby 3.0.

> Is preload_app in use?  If so, does the problem manifest without
> preload_app?

No

> Are you using Ractor or any other new/experimental stuff
> in Ruby?

No

> Can you reproduce the problem with a simple "hello world"-style app?

I don't have a hello world example at hand, but some projects that 
present this problem when I run them on OpenBSD/adJ amd64 6.8, nginx 
1.18.0, node 12.16.1, PostgreSQL 13.2, Ruby 3.0, Rails 6.1 and Unicorn 
6.0 are:

https://github.com/pasosdeJesus/cor1440
https://github.com/pasosdeJesus/cor1440_pdJ
https://github.com/pasosdeJesus/sivel2
https://github.com/pasosdeJesus/si_anzorc
https://github.com/pasosdeJesus/si_asom
https://github.com/pasosdeJesus/si_codacop
https://github.com/pasosdeJesus/sivel2_mujeresindigenas/

The simplest of them is https://github.com/pasosdeJesus/cor1440, althoug 
it requires:
* PostgreSQL
* nginx
* nodejs
* yarn. I install it with: doas pkg_add bash; ftp -o- 
https://yarnpkg.com/install.sh | bash
* A directory /var/www/bundler/ruby/3.0 where I install the gems with 
the permissions of a user (not root) that runs the application, and 
better running:
   bundle config path /var/www/bundler/.
* unicorn installed globally with:
   doas gem install unicorn
   doas ln -s /usr/local/bin/unicorn_rails30 /usr/local/bin/unicorn

To create the PostgreSQL user (in the following example sipdes) and a 
database for the production environment  (in the following example 
cor1440gen_pro):

   doas su - _postgresql
   createuser -h /var/www/var/run/postgresql/ -U postgres -s sipdes
   createdb -h /var/www/var/run/postgresql/ -U postgres -O sipdes 
cor1440gen_pro
   psql -h /var/www/var/run/postgresql/ -U postgres
   postgres=# alter user sipdes with password 'xyz';
   postgres=# \q
   exit

With nginx suppossng I will run the application in localhost over SSL, 
what I do is to choose an unused port (let's say 2009) and add to the 
/etc/nginx/nginx.conf configuration in its http block:

  upstream unicorncor1440 {
	  server 127.0.0.1:2009 fail_timeout=0;
   }

I also add a server block inside the http block with:

   server {
       listen 443 ssl;
       ssl_certificate /etc/ssl/server.crt;
       ssl_certificate_key /etc/ssl/private/server.key;
       root /var/www/htdocs/cor1440/;
       server_name 127.0.0.1;
       error_log logs/cor1440.rror.log;

       location /cor1440 {
         try_files $uri @unicorncor1440;
       }

       location @unicorncor1440 {
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_set_header Host $http_host;
         proxy_redirect off;
         proxy_pass http://unicorncor1440;
         error_page 500 502 503 504 /500.html;
         keepalive_timeout 10;
       }

       location ^~ /cor1440/assets/ {
         gzip_static on;
         expires max;
         add_header Cache-Control public;
         root /var/www/htdocs/cor1440/public/;
       }

       location ^~ /cor1440/images/ {
         gzip_static on;
         expires max;
         add_header Cache-Control public;
         root /var/www/htdocs/cor1440/public/;
       }

       location ^~ /cor1440/packs/ {
         gzip_static on;
         add_header Cache-Control public;
         root /var/www/htdocs/cor1440/public/;
       }
     }

And restarted nginx:

   doas rcctl restart nginx

Then to configure:

   cd /tmp
   git clone git@github.com:pasosdeJesus/cor1440
   doas mv cor1440 /var/www/htdocs/
   doas chown -R $USER:$USER /var/www/htdocs/cor1440
   cd /var/www/htdocs/cor1440
   mkdir .bundle
   echo '---' > .bundle/config
   echo 'BUNDLE_PATH: "/var/www/bundler"' >> .bundle/config
   echo 'BUNDLE_DISABLE_SHARED_GEMS: "true"' >> .bundle/config
   bundle

Here some gems will require installing with special permissions for 
example:

   doas gem install --install-dir /var/www/bundler/ruby/3.0 unicorn

Other gems that will require this kind of installation will be:
   racc nio4r websocket-driver bcrypt bindex msgpack ffi redcarpet kgio 
libxml-ruby pg
   nokogiri bootsnap puma sassc

Once bundle runs completely:

   cp .env.plantilla .env
   vi .env

There I set the variables:
* BD_USUARIO with the database user,
* BD_CLAVE with the password
* BD_PRO with the name of the production database
* CONFIG_HOSTS=127.0.0.1
* DIRAP with the path to the application
* RAILS_ENV=production
* PUERTOUNICORN with the port I choosed and set in nginx.conf
* USUARIO_AP with the operating system user who will run the application

Prepare credentials and database:

   bin/rails credentials:edit
   RAILS_ENV=production bin/rails credentials:edit
   bin/railsp db:setup  db:migrate sip:indices

Precompile assets:

   CXX=c++ yarn
   doas ln -s /usr/local/bin/node /tmp  # To overcome bug of node with 
rails on OpenBSD
   bin/railsp assets:precompile

Run the application with:
   SECRET_KEY_BASE=cf46 DIRAP=/var/www/htdocs/cor1440/ bin/u.sh

Trying to see the application running with a browser in 
https://127.0.0.1/cor1440 will produce the errors described in 
log/unicorn.log.

Blessings. Thank you.

El 2021-03-30 00:00, Jeremy Evans escribió:
> On 03/30 01:58, Eric Wong wrote:
>> vtamara <vtamara@pasosdeJesus.org> wrote:
>> > Hi
>> >
>> > Launching a rails application with Ruby 3, Unicorn 6 on OpenBSD/adJ 6.8, I'm
>> > experiencing a lot of "Bus Error" and "Segmentation Faults" during
>> > aproximately half hour.
>> 
>> +Cc: Jeremy Evans, who is more familiar with OpenBSD and Ruby 3.0 than 
>> I
>> cf. 
>> https://yhbt.net/unicorn-public/6acb1c84c7392d7b4a64572b20498549@pasosdeJesus.org/
> 
> I'll be happy to look into this if a minimal self-contained 
> reproducible
> example is posted.  Personally, I run my Ruby apps on OpenBSD amd64 
> with
> Unicorn, and haven't seen similar errors.  However, I don't use
> preload_app and use fairly minimal dependencies.
> 
> Thanks,
> Jeremy

-- 
Dios, gracias por tu amor infinito.
--
   Vladimir Támara Patiño.  http://vtamara.pasosdeJesus.org/
   http://www.pasosdejesus.org/dominio_publico_colombia.html

^ permalink raw reply	[relevance 4%]

* Re: Potential Unicorn vulnerability
  2021-03-12 11:14  5%       ` Dirkjan Bussink
@ 2021-03-12 12:00  0%         ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-12 12:00 UTC (permalink / raw)
  To: Dirkjan Bussink; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public

Dirkjan Bussink <dbussink@github.com> wrote:
> Hi Eric,
> 
> > On 12 Mar 2021, at 10:41, Eric Wong <normalperson@yhbt.net> wrote:
> > 
> > I'm not in favor of new options since they add support costs
> > and increase the learning/maintenance curve.
> > 
> > What I've been thinking about is bumping the major version to 6.0
> > 
> > Although our internals are technically not supported stable API,
> > there may be odd stuff out there similar to OobGC that uses
> > instance_variable_get or similar things to reach into internals.
> > Added with the fact our internals haven't changed in many years;
> > I'm inclined to believe there are other OobGC-like things out
> > there that can break.
> > 
> > Also, with 6.0; users who completely avoid Threads can keep
> > using 5.x, while others can use 6.x
> 
> That sounds like a good plan then. Once there’s a new version we can
> bump that on our side to remove the manual patch then.

OK.  I think it's safe to wait a few days for more comments
before releasing in case there's more last-minute revelations
(see below :x)

> > Btw, did you consider replacing the @request HttpRequest object
> > entirely instead of the env and buf elements?
> > I suppose that's more allocations, still; but could've
> > been a smaller change.
> 
> Ah, that’s a very good point. I think that would also have been a valid
> approach but it does indeed add more allocations. If that approach would
> be preferred, I think it can also be changed to that?
> 
> I don’t really have a strong preference on which approach to take here,
> do you?

I was going to say I didn't have a preference and the
current approach was fine...

However, I just now realized now that clobbering+replacing all
of @request is required.

That's because env['rack.input'] is (Stream|Tee)Input,
and that is lazily consumed and those objects keep state in
@request (as the historically-named @parser)

If we're to make env safe to be shipped off to another thread,
then @request still needs to stick around to maintain state
of env['rack.input'] until it's all consumed.

It probably doesn't affect most apps out there that just decode
forms via HTTP POST; but the streamed rack.input is something
that's critical for projects that feed unicorn with PUTs via
"curl -T"

> > Oops, was that the integration tests in t/* ?
> 
> Nope, looks like some platform behavior changes (tried on MacOS first).
> I was able to get the tests running and working on Debian Buster this
> morning before I sent a new version of the patch and they are all passing
> there for me locally.

Ah, no idea about MacOS or any proprietary OS; I've never
considered them supported.  But yeah, it should work on any
GNU/Linux and Free-as-in-speech *BSDs

^ permalink raw reply	[relevance 0%]

* Re: Potential Unicorn vulnerability
  2021-03-12  9:41  2%     ` Potential Unicorn vulnerability Eric Wong
@ 2021-03-12 11:14  5%       ` Dirkjan Bussink
  2021-03-12 12:00  0%         ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Dirkjan Bussink @ 2021-03-12 11:14 UTC (permalink / raw)
  To: Eric Wong; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public

Hi Eric,

> On 12 Mar 2021, at 10:41, Eric Wong <normalperson@yhbt.net> wrote:
> 
> I'm not in favor of new options since they add support costs
> and increase the learning/maintenance curve.
> 
> What I've been thinking about is bumping the major version to 6.0
> 
> Although our internals are technically not supported stable API,
> there may be odd stuff out there similar to OobGC that uses
> instance_variable_get or similar things to reach into internals.
> Added with the fact our internals haven't changed in many years;
> I'm inclined to believe there are other OobGC-like things out
> there that can break.
> 
> Also, with 6.0; users who completely avoid Threads can keep
> using 5.x, while others can use 6.x

That sounds like a good plan then. Once there’s a new version we can
bump that on our side to remove the manual patch then.

> Btw, did you consider replacing the @request HttpRequest object
> entirely instead of the env and buf elements?
> I suppose that's more allocations, still; but could've
> been a smaller change.

Ah, that’s a very good point. I think that would also have been a valid
approach but it does indeed add more allocations. If that approach would
be preferred, I think it can also be changed to that?

I don’t really have a strong preference on which approach to take here,
do you? 

> Oops, was that the integration tests in t/* ?

Nope, looks like some platform behavior changes (tried on MacOS first).
I was able to get the tests running and working on Debian Buster this
morning before I sent a new version of the patch and they are all passing
there for me locally.

Cheers,

Dirkjan

^ permalink raw reply	[relevance 5%]

* Re: Potential Unicorn vulnerability
       [not found]       ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
@ 2021-03-12  9:41  2%     ` Eric Wong
  2021-03-12 11:14  5%       ` Dirkjan Bussink
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-12  9:41 UTC (permalink / raw)
  To: Dirkjan Bussink; +Cc: John Crepezzi, Kevin Sawicki, unicorn-public

[-- Attachment #1: Type: text/plain, Size: 6397 bytes --]

Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
> 
> > On 11 Mar 2021, at 04:02, Eric Wong <normalperson@yhbt.net> wrote:
> > 
> > Thanks for reaching out.  Fwiw, I prefer if everything were made
> > public right away, but I'll leave it up to you if you're not
> > comfortable with it.
> > 
> > I don't know much about security, anwyays; and don't like
> > classifying bugs (or classifying anything)...
> 
> We reached out privately first out of care and to follow best practices
> around coordinated disclosure, in case you considered this a security
> vulnerability. We have no objections to moving this to the public mailing
> list. We are viewing this patch as a proactive hardening against race
> conditions more so than a vulnerability. 

OK, I'm adding unicorn-public@yhbt.net to the Cc: of this mail.

Personally, I prefer everything be reported publicly ASAP.
There's a constant threat of power/network failures from
disasters and such that could cause messages to be delayed
too long or indefinitely.

> We are also reaching out to a private Rails Security coordination channel
> that we’re a part of to raise awareness of this behavior so other Unicorn
> users in this group can look for similar issues in their code. 

OK.

> To move this discussion to the public list, would you prefer that you move
> this thread publicly, or that we resend the message or forward it to the
> public mailing list? 

Attached are the initial two (previously private) emails in this
thread, so no need to resend.  They have the correct
message/rfc822 MIME type set so they should be readable from
most MUAs and mail archives.

> > Ouch, so the hijack check we had in HttpParser_clear didn't fire...
> 
> Yes, to our understanding it only would fire if explicitly set and that
> doesn’t happen here.
> 
> >> While we understand and appreciate that Unicorn is not a multi-threaded web
> >> server, it feels like using the same `Hash` object between requests
> >> introduces the chance that a dependency like an external gem may use threads
> >> and thus potentially leak information between requests.
> > 
> > Yes, there's likely problems in trying to use threads with a
> > codebase that was only intended to be single-threaded.  And
> > using Thread.current[...] here wouldn't have made a. difference.
> 
> For us it was also the difference between “requests are handled single threaded”
> vs “you can’t use threads for anything else either.” We were totally aware of the
> first, but the latter seems more accidental and has a much broader impact. 

Agreed.

> > I worry some endpoints out there will suffer performance
> > degradation.  Expensive endpoints probably won't notice,
> > but maybe the fastest ones will...
> 
> That is a good point, but I think in practice most users do enough in most
> requests that the trade off is totally worth it. At least for us it definitely
> is. Maybe it would be an option to make the sharing somehow opt-in instead of
> default behavior? So that by default the safe behavior is used, but for those
> that want to, they can opt into the sharing if they know their app is safe
> enough to work with that. 

I'm not in favor of new options since they add support costs
and increase the learning/maintenance curve.

What I've been thinking about is bumping the major version to 6.0

Although our internals are technically not supported stable API,
there may be odd stuff out there similar to OobGC that uses
instance_variable_get or similar things to reach into internals.
Added with the fact our internals haven't changed in many years;
I'm inclined to believe there are other OobGC-like things out
there that can break.

Also, with 6.0; users who completely avoid Threads can keep
using 5.x, while others can use 6.x

> >> For the sake of transparency to our users, we plan on publishing a public
> >> post next week on how this was part of the larger series of bugs that had a
> >> security impact at GitHub. We've also attached a suggested patch that removes
> >> the environment sharing, which is what we're running right now to reduce the
> >> risk of this ever happening again.
> > 
> > Did you measure performance degradations in any endpoints you have?
> 
> We did measure and there were zero noticeable performance degradations. That’s
> also because all our requests do a bunch of work and are not direct Rack apps,
> and use stuff like Rails or Sinatra. Those all usually wrap the `env` hash
> anyway with their own per request object and there’s a lot of other allocations
> happening anyway.
> 
> In microbenchmarks we could see a difference, but even there, at least for us,
> we’d gladly pay the performance price for the safety if we’d have any endpoints
> where it would be measurable. 

OK.  Fwiw, there's still some stones left unturned that could
recover the lost speed if somebody _really_ cares for it(*)

(*) String#clear on response header buffer, swapping Ragel
    for a faster HTTP parser, ...

> All in all, I think here that a safe default would be helpful for users, as
> mentioned earlier, and that maybe for those cases where the latest performance
> bit matters, the existing behavior could be opted into. Whether this option is
> worth it from a maintenance standpoint is something you’re better able to answer
> than we are though.

It's probably OK and I'm inclined to accept your patch for 6.0.

At this point, I'm more worried about potential breakage of some
3rd-party OobGC-like thing that reaches deep into our internals.

Btw, did you consider replacing the @request HttpRequest object
entirely instead of the env and buf elements?
I suppose that's more allocations, still; but could've
been a smaller change.

> > I'll see about finding a less-noisy/overloaded system to run
> > benchmarks against...
> > 
> > I noticed some of the OobGC tests in t/ were failing (patch below),
> > but few users use the that version of OobGC.
> 
> I wasn’t easily able to run the entire suite, but only parts of it which is why
> I didn’t have a complete fix there. I can add this to the patch then.

Oops, was that the integration tests in t/* ?
They can run separately via "make test-integration"
(I never trusted some Ruby behaviors to remain stable,
 so I started writing tests in Bourne shell).

<snip> I have nothing to add to the rest of the mail

unicorn-public readers: see attachments

[-- Attachment #2: 0001-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 12105 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 2813 bytes --]

Hello Eric,

We're reaching out privately first on what we think could be classified as a
security issue in Unicorn. Since there may be other similarly impacted users,
this is out of an abundance of caution before sending it to the public
Unicorn mailing list.

The issue at hand is that the environment sharing and reuse between requests
in Unicorn, combined with other non thread safe code, resulted in a security
vulnerability where a very small number of user sessions tracked through
cookies got misrouted. For this reason, we logged everyone out of GitHub, see
also https://github.blog/2021-03-08-github-security-update-a-bug-related-to-handling-of-authenticated-sessions/. 

The unsafe background thread code we had ended up triggering an exception at
rare times and the exception tracking logic was using a deferred block
executed through a Ruby Proc to gather information, including things from the
request at the time the logic started.

That thread captured something from the cookie jar among other things, and
the following code in Rack memoized that into the (shared) environment.

From https://github.com/rack/rack/blob/2.1.4/lib/rack/request.rb#L219-L229

def cookies
  hash = fetch_header(RACK_REQUEST_COOKIE_HASH) do |k|
    set_header(k, {})
  end
  string = get_header HTTP_COOKIE

  return hash if string == get_header(RACK_REQUEST_COOKIE_STRING)
  hash.replace Utils.parse_cookies_header string
  set_header(RACK_REQUEST_COOKIE_STRING, string)
  hash
end

Because of the environment sharing, the memoization ended up overriding what
would be memoized (with the RACK_REQUEST_COOKIE_STRING key) for the new
request and because a specific session cookie was bumped then with a new
timeout on each request, the wrong session cookie was serialized.

While we understand and appreciate that Unicorn is not a multi-threaded web
server, it feels like using the same `Hash` object between requests
introduces the chance that a dependency like an external gem may use threads
and thus potentially leak information between requests.

For the sake of transparency to our users, we plan on publishing a public
post next week on how this was part of the larger series of bugs that had a
security impact at GitHub. We've also attached a suggested patch that removes
the environment sharing, which is what we're running right now to reduce the
risk of this ever happening again.

We hope you're open to collaborating on a fix prior to any public detailed
disclosure of how this request environment sharing could lead to security
issues, if you feel that is desired. 

I’ve added John & Kevin here on the CC since they’ve also worked on this and
that way we have some better timezone spread on our side if needed. 

Cheers,

Dirkjan Bussink


[-- Attachment #2.1.2: 0001-Drop-reuse-of-Ruby-level-objects.patch --]
[-- Type: application/octet-stream, Size: 4067 bytes --]

From 44aa6b056e1b24d42ab3efba738b38f4cd54a068 Mon Sep 17 00:00:00 2001
From: Dirkjan Bussink <d.bussink@gmail.com>
Date: Mon, 8 Mar 2021 09:51:09 +0100
Subject: [PATCH] Drop reuse of Ruby level objects

Remove the reuse of environment and the buffer as Ruby level objects.
Reusing these is very risky in the context of running any other threads
within the unicorn process, also for threads that run background tasks.

This lead to a significant security incident where the problems would
not have happened if there was no reuse of the `env` object. From a
safety perspective, this also removes reuse of the buffer object as it's
also a Ruby object and could be grabbed and retained outside of the http
parsing logic.

The downside here is that we allocate two extra objects on each request,
but that is worth the trade off here and the security risk we otherwise
would carry to leaking wrong and incorrect data.
---
 ext/unicorn_http/extconf.rb      |  1 -
 ext/unicorn_http/unicorn_http.rl | 30 +++---------------------------
 test/unit/test_http_parser.rb    |  3 +++
 3 files changed, 6 insertions(+), 28 deletions(-)

diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 95514bc..7e4775c 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -10,7 +10,6 @@
 have_macro("SIZEOF_SIZE_T", "ruby.h") or check_sizeof("size_t", "sys/types.h")
 have_macro("SIZEOF_LONG", "ruby.h") or check_sizeof("long", "sys/types.h")
 have_func("rb_str_set_len", "ruby.h") or abort 'Ruby 1.9.3+ required'
-have_func("rb_hash_clear", "ruby.h") # Ruby 2.0+
 have_func("gmtime_r", "time.h")
 
 message('checking if String#-@ (str_uminus) dedupes... ')
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 21e09d6..9904d85 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -65,18 +65,6 @@ struct http_parser {
 static ID id_set_backtrace, id_is_chunked_p;
 static VALUE cHttpParser;
 
-#ifdef HAVE_RB_HASH_CLEAR /* Ruby >= 2.0 */
-#  define my_hash_clear(h) (void)rb_hash_clear(h)
-#else /* !HAVE_RB_HASH_CLEAR - Ruby <= 1.9.3 */
-
-static ID id_clear;
-
-static void my_hash_clear(VALUE h)
-{
-  rb_funcall(h, id_clear, 0);
-}
-#endif /* HAVE_RB_HASH_CLEAR */
-
 static void finalize_header(struct http_parser *hp);
 
 static void parser_raise(VALUE klass, const char *msg)
@@ -445,6 +433,8 @@ static void http_parser_init(struct http_parser *hp)
   hp->cont = Qfalse; /* zero on MRI, should be optimized away by above */
   %% write init;
   hp->cs = cs;
+  hp->buf = rb_str_new(NULL, 0);
+  hp->env = rb_hash_new();
 }
 
 /** exec **/
@@ -628,8 +618,6 @@ static VALUE HttpParser_init(VALUE self)
   struct http_parser *hp = data_get(self);
 
   http_parser_init(hp);
-  hp->buf = rb_str_new(NULL, 0);
-  hp->env = rb_hash_new();
 
   return self;
 }
@@ -643,16 +631,7 @@ static VALUE HttpParser_init(VALUE self)
  */
 static VALUE HttpParser_clear(VALUE self)
 {
-  struct http_parser *hp = data_get(self);
-
-  /* we can't safely reuse .buf and .env if hijacked */
-  if (HP_FL_TEST(hp, HIJACK))
-    return HttpParser_init(self);
-
-  http_parser_init(hp);
-  my_hash_clear(hp->env);
-
-  return self;
+  return HttpParser_init(self);
 }
 
 static void advance_str(VALUE str, off_t nr)
@@ -1025,9 +1004,6 @@ void Init_unicorn_http(void)
   id_set_backtrace = rb_intern("set_backtrace");
   init_unicorn_httpdate();
 
-#ifndef HAVE_RB_HASH_CLEAR
-  id_clear = rb_intern("clear");
-#endif
   id_is_chunked_p = rb_intern("is_chunked?");
 }
 #undef SET_GLOBAL
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 697af44..d3f9ae7 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -33,6 +33,9 @@ def test_parse_simple
     parser.clear
     req.clear
 
+    req = parser.env
+    http = parser.buf
+
     http << "G"
     assert_nil parser.parse
     assert_equal "G", http
-- 
2.30.1


[-- Attachment #3: 0002-re-potential-unicorn-vulnerability.eml --]
[-- Type: message/rfc822, Size: 5635 bytes --]

From: Eric Wong <normalperson@yhbt.net>
To: Dirkjan Bussink <dbussink@github.com>
Cc: John Crepezzi <seejohnrun@github.com>, Kevin Sawicki <kevinsawicki@github.com>
Subject: Re: Potential Unicorn vulnerability
Date: Thu, 11 Mar 2021 03:02:50 +0000
Message-ID: <20210311030250.GA1266@dcvr>

Dirkjan Bussink <dbussink@github.com> wrote:
> Hello Eric,
> 
> We're reaching out privately first on what we think could be classified as a
> security issue in Unicorn. Since there may be other similarly impacted users,
> this is out of an abundance of caution before sending it to the public
> Unicorn mailing list.

Thanks for reaching out.  Fwiw, I prefer if everything were made
public right away, but I'll leave it up to you if you're not
comfortable with it.

I don't know much about security, anwyays; and don't like
classifying bugs (or classifying anything)...

<snip>

> That thread captured something from the cookie jar among other things, and
> the following code in Rack memoized that into the (shared) environment.

Ouch, so the hijack check we had in HttpParser_clear didn't fire...

<snip>

> While we understand and appreciate that Unicorn is not a multi-threaded web
> server, it feels like using the same `Hash` object between requests
> introduces the chance that a dependency like an external gem may use threads
> and thus potentially leak information between requests.

Yes, there's likely problems in trying to use threads with a
codebase that was only intended to be single-threaded.  And
using Thread.current[...] here wouldn't have made a. difference.

I worry some endpoints out there will suffer performance
degradation.  Expensive endpoints probably won't notice,
but maybe the fastest ones will...

`buf' is particularly worrying to me since it's a 16384-byte
allocation for a socket read on headers that could amount
to lots of GC pressure and hurt locality all around.

env['rack.input'] may also hold onto `buf', too, since
input is lazily-consumed (though it may be possible to
workaround that...).

Losing `env' is probably less worrying since keys are all
fstrings in modern Rubies.  Though losing the backing store (and
not having rb_hash_new_with_size in the C API) would mean more
rehashing with many headers.

The simple Rack apps I worked on back in-the-day which
benefitted from these object-reuse optimizations are unlikely to
be upgraded to newer versions of unicorn.  Of course, there
may be other similar Rack apps out there that depend on these...

> For the sake of transparency to our users, we plan on publishing a public
> post next week on how this was part of the larger series of bugs that had a
> security impact at GitHub. We've also attached a suggested patch that removes
> the environment sharing, which is what we're running right now to reduce the
> risk of this ever happening again.

Did you measure performance degradations in any endpoints you have?

I'll see about finding a less-noisy/overloaded system to run
benchmarks against...

I noticed some of the OobGC tests in t/ were failing (patch below),
but few users use the that version of OobGC.

Also, t/t0200-rack-hijack.sh would go away.

> We hope you're open to collaborating on a fix prior to any public detailed
> disclosure of how this request environment sharing could lead to security
> issues, if you feel that is desired. 

Sure, as long as everything is done via plain-text email so
I won't need working graphics drivers or modern hardware :>

Along the same lines, would you be OK with this entire mail thread
(including all mail headers) being eventually published in the
public mailbox at http://ou63pmih66umazou.onion/unicorn-public/
and https://yhbt.net/unicorn-public/ ?

> I’ve added John & Kevin here on the CC since they’ve also worked on this and
> that way we have some better timezone spread on our side if needed. 

OK, I'm around/awake at pretty random times.

Aforementioned OomGC change:
-------8<-------
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 3b2f488..91a8e51 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -60,7 +60,6 @@ def self.new(app, interval = 5, path = %r{\A/})
     self.const_set :OOBGC_INTERVAL, interval
     ObjectSpace.each_object(Unicorn::HttpServer) do |s|
       s.extend(self)
-      self.const_set :OOBGC_ENV, s.instance_variable_get(:@request).env
     end
     app # pretend to be Rack middleware since it was in the past
   end
@@ -68,9 +67,10 @@ def self.new(app, interval = 5, path = %r{\A/})
   #:stopdoc:
   def process_client(client)
     super(client) # Unicorn::HttpServer#process_client
-    if OOBGC_PATH =~ OOBGC_ENV['PATH_INFO'] && ((@@nr -= 1) <= 0)
+    env = instance_variable_get(:@request).env
+    if OOBGC_PATH =~ env['PATH_INFO'] && ((@@nr -= 1) <= 0)
       @@nr = OOBGC_INTERVAL
-      OOBGC_ENV.clear
+      env.clear
       disabled = GC.enable
       GC.start
       GC.disable if disabled

^ permalink raw reply related	[relevance 2%]

* Re: [PATCH] Add rack.after_reply functionality
  2020-12-09 22:18  9%         ` Eric Wong
@ 2020-12-09 23:44  6%           ` Blake Williams
  0 siblings, 0 replies; 200+ results
From: Blake Williams @ 2020-12-09 23:44 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public


> Thanks, pushed to https://yhbt.net/unicorn.git as
> commit 673c15e3f020bccc0336838617875b26c9a45f4e
> with Puma noted (see below) and extra `env' check added.
> 
> Anything else?  Will wait a few days/week for others to chime in
> and probably cut 5.8 in a week or so.

Thanks, that should be everything on my end. Looking forward to the
release.

> Btw, a sidenote about some strangeness in your mail replies:
> 
>  I'm not sure why, but your "Reply-To" header is set to the
>  unbracketed Message-ID ("20201209094344.GA25593@dcvr") of the
>  message, which makes no sense...
> 
>  I wonder if you or your mail client is confusing "Reply-To:"
>  with the "In-Reply-To:" header.  They're completely different:
>  "Reply-To:" should be an email address you control (not a
>  Message-ID) if it differs from what's in the "From:" header.
> 
>  In any case, your mail client already sets the "References:"
>  header correctly (probably w/o any interaction on your part),
>  and the "In-Reply-To:" header is not necessary (and it needs
>  angle brackets, anyways, since comments are allowed).
> 
>  cf. https://www.jwz.org/doc/threading.html

Strange, thanks for the heads up. I’ve updated the settings, so
hopefully that issue is resolved.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add rack.after_reply functionality
  2020-12-09 14:58  6%       ` Blake Williams
@ 2020-12-09 22:18  9%         ` Eric Wong
  2020-12-09 23:44  6%           ` Blake Williams
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-12-09 22:18 UTC (permalink / raw)
  To: Blake Williams; +Cc: unicorn-public

Blake Williams <blake@blakewilliams.me> wrote:
> Totally understood. I looked at Puma, Thin, and Webrick and only Puma
> supports this functionality without a third-party gem. There is at least one
> third party gem adding this functionality to web servers that don’t support it,
> but it doesn’t look like it’s being maintained any longer.

Alright, I guess that third-party gem is no longer relevant.

> I briefly dove into the Puma source and it looks like rack.after_reply has
> been supported since 2011. It’s also still in use by Puma for their common 
> logger middleware patch.

Thanks, pushed to https://yhbt.net/unicorn.git as
commit 673c15e3f020bccc0336838617875b26c9a45f4e
with Puma noted (see below) and extra `env' check added.

Anything else?  Will wait a few days/week for others to chime in
and probably cut 5.8 in a week or so.

Btw, a sidenote about some strangeness in your mail replies:

  I'm not sure why, but your "Reply-To" header is set to the
  unbracketed Message-ID ("20201209094344.GA25593@dcvr") of the
  message, which makes no sense...

  I wonder if you or your mail client is confusing "Reply-To:"
  with the "In-Reply-To:" header.  They're completely different:
  "Reply-To:" should be an email address you control (not a
  Message-ID) if it differs from what's in the "From:" header.

  In any case, your mail client already sets the "References:"
  header correctly (probably w/o any interaction on your part),
  and the "In-Reply-To:" header is not necessary (and it needs
  angle brackets, anyways, since comments are allowed).

  cf. https://www.jwz.org/doc/threading.html


Author: Blake Williams <blake@blakewilliams.me>
Date:   Tue Dec 8 16:47:16 2020 -0500

    Add rack.after_reply functionality
    
    This adds `rack.after_reply` functionality which allows rack middleware
    to pass lambdas that will be executed after the client connection has
    been closed.
    
    This was driven by a need to perform actions in a request that shouldn't
    block the request from completing but also don't make sense as
    background jobs.
    
    There is prior art of this being supported found in a few gems, as well
    as this functionality existing in other rack based servers (e.g. Puma).
    
    [ew: check if `env' is set in ensure statement]
    
    Acked-by: Eric Wong <e@80x24.org>

^ permalink raw reply	[relevance 9%]

* Re: [PATCH] Add rack.after_reply functionality
  2020-12-09  9:43  6%     ` Eric Wong
@ 2020-12-09 14:58  6%       ` Blake Williams
  2020-12-09 22:18  9%         ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Blake Williams @ 2020-12-09 14:58 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public


> OK.  Keep in mind this can be server-dependent since unicorn
> can't support persistent connections while (most?) other servers
> do (and they may disable TCP_CORK or Nagle, etc).

I see, I’ll look into this as well.

>>> Citation(s) needed.  Also, are there any proposal(s) to get this
>>> into the Rack specification?
>> 
>> Not that I’m aware of. But I agree that it and early_hints could both be
>> good additions to the spec.
> 
> By "Citation(s) needed", can you list some other (preferably
> well-known) Rack servers which also implement this feature?
> 
> We wouldn't want current unicorn users thinking we're adopting
> random new things which nobody else supports :>
> 
> Thanks.


Totally understood. I looked at Puma, Thin, and Webrick and only Puma
supports this functionality without a third-party gem. There is at least one
third party gem adding this functionality to web servers that don’t support it,
but it doesn’t look like it’s being maintained any longer.

I briefly dove into the Puma source and it looks like rack.after_reply has
been supported since 2011. It’s also still in use by Puma for their common 
logger middleware patch.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add rack.after_reply functionality
  2020-12-08 23:48  6%   ` Blake Williams
@ 2020-12-09  9:43  6%     ` Eric Wong
  2020-12-09 14:58  6%       ` Blake Williams
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-12-09  9:43 UTC (permalink / raw)
  To: Blake Williams; +Cc: unicorn-public

Blake Williams <blake@blakewilliams.me> wrote:
> 
> > In the past, I've used response_body.close in middleware for
> > similar things, and nowadays Rack::BodyProxy exists in rack, too.
> > 
> > So I'm a little skeptical if this is actually needed, but it's
> > probably too late to do anything about:
> 
> The first approach we took uses Rack::Events, which I believe is using
> Rack::BodyProxy under the hood. We ran into an issue where all but
> the last line of the response is written and the connection isn’t
> actually closed yet when the callbacks are called.

OK.  Keep in mind this can be server-dependent since unicorn
can't support persistent connections while (most?) other servers
do (and they may disable TCP_CORK or Nagle, etc).

> > Citation(s) needed.  Also, are there any proposal(s) to get this
> > into the Rack specification?
> 
> Not that I’m aware of. But I agree that it and early_hints could both be
> good additions to the spec.

By "Citation(s) needed", can you list some other (preferably
well-known) Rack servers which also implement this feature?

We wouldn't want current unicorn users thinking we're adopting
random new things which nobody else supports :>

Thanks.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add rack.after_reply functionality
  2020-12-08 22:46  5% ` Eric Wong
@ 2020-12-08 23:48  6%   ` Blake Williams
  2020-12-09  9:43  6%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Blake Williams @ 2020-12-08 23:48 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public


> In the past, I've used response_body.close in middleware for
> similar things, and nowadays Rack::BodyProxy exists in rack, too.
> 
> So I'm a little skeptical if this is actually needed, but it's
> probably too late to do anything about:

The first approach we took uses Rack::Events, which I believe is using
Rack::BodyProxy under the hood. We ran into an issue where all but
the last line of the response is written and the connection isn’t
actually closed yet when the callbacks are called.

> Citation(s) needed.  Also, are there any proposal(s) to get this
> into the Rack specification?

Not that I’m aware of. But I agree that it and early_hints could both be
good additions to the spec.

> I'll have to squash the following in, since an exception can be
> raised if a client truncates the request:

Good catch, sounds good to me.

> Everything else seems OK.  Thanks.

Awesome, thanks for the quick reply/feedback.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add rack.after_reply functionality
  2020-12-08 21:47  4% [PATCH] Add rack.after_reply functionality Blake Williams
@ 2020-12-08 22:46  5% ` Eric Wong
  2020-12-08 23:48  6%   ` Blake Williams
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-12-08 22:46 UTC (permalink / raw)
  To: Blake Williams; +Cc: unicorn-public

Blake Williams <blake@blakewilliams.me> wrote:
> This adds `rack.after_reply` functionality which allows rack middleware
> to pass lambdas that will be executed after the client connection has
> been closed.
> 
> This was driven by a need to perform actions in a request that shouldn't
> block the request from completing but also don't make sense as
> background jobs.

Thanks for the patch and explanation.

In the past, I've used response_body.close in middleware for
similar things, and nowadays Rack::BodyProxy exists in rack, too.

So I'm a little skeptical if this is actually needed, but it's
probably too late to do anything about:

> There is prior art of this being supported found in a few gems, as well
> as this functionality existing in other rack based servers.

Citation(s) needed.  Also, are there any proposal(s) to get this
into the Rack specification?

While I can't participate in discussions which requires a
GUI (JS/CAPTCHA) or accepting certain Terms-of-service,
the rack-devel@googlegroups.com mailing list still exists
and I can chime in there.

rack.early_hints could probably be in the Rack spec, too...

More below...

> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 05dad99..9889f55 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -629,6 +629,8 @@ def process_client(client)
>        end
>      end
>  
> +    env["rack.after_reply"] = []
> +
>      status, headers, body = @app.call(env)
>  
>      begin
> @@ -651,6 +653,8 @@ def process_client(client)
>      end
>    rescue => e
>      handle_error(client, e)
> +  ensure
> +    env["rack.after_reply"].each { |after_reply| after_reply.call }
>    end
>  
>    def nuke_listeners!(readers)

I'll have to squash the following in, since an exception can be
raised if a client truncates the request:

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 9889f55c..c0f14ba5 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -654,7 +654,7 @@ def process_client(client)
   rescue => e
     handle_error(client, e)
   ensure
-    env["rack.after_reply"].each { |after_reply| after_reply.call }
+    env["rack.after_reply"].each(&:call) if env
   end
 
   def nuke_listeners!(readers)

Everything else seems OK.  Thanks.

^ permalink raw reply related	[relevance 5%]

* [PATCH] Add rack.after_reply functionality
@ 2020-12-08 21:47  4% Blake Williams
  2020-12-08 22:46  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Blake Williams @ 2020-12-08 21:47 UTC (permalink / raw)
  To: unicorn-public

This adds `rack.after_reply` functionality which allows rack middleware
to pass lambdas that will be executed after the client connection has
been closed.

This was driven by a need to perform actions in a request that shouldn't
block the request from completing but also don't make sense as
background jobs.

There is prior art of this being supported found in a few gems, as well
as this functionality existing in other rack based servers.
---
 lib/unicorn/http_server.rb |  4 ++++
 test/unit/test_server.rb   | 44 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 05dad99..9889f55 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -629,6 +629,8 @@ def process_client(client)
       end
     end
 
+    env["rack.after_reply"] = []
+
     status, headers, body = @app.call(env)
 
     begin
@@ -651,6 +653,8 @@ def process_client(client)
     end
   rescue => e
     handle_error(client, e)
+  ensure
+    env["rack.after_reply"].each { |after_reply| after_reply.call }
   end
 
   def nuke_listeners!(readers)
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 384fa6b..781750d 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -34,6 +34,24 @@ def call(env)
   end
 end
 
+class TestRackAfterReply
+  def initialize
+    @called = false
+  end
+
+  def call(env)
+    while env['rack.input'].read(4096)
+    end
+
+    env["rack.after_reply"] << -> { @called = true }
+
+    [200, { 'Content-Type' => 'text/plain' }, ["after_reply_called: #{@called}"]]
+  rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e
+    $stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n")
+    raise e
+  end
+end
+
 class WebServerTest < Test::Unit::TestCase
 
   def setup
@@ -114,6 +132,32 @@ def test_early_hints
     assert_match %r{^HTTP/1.[01] 200\b}, responses
   end
 
+  def test_after_reply
+    teardown
+
+    redirect_test_io do
+      @server = HttpServer.new(TestRackAfterReply.new,
+                               :listeners => [ "127.0.0.1:#@port"])
+      @server.start
+    end
+
+    sock = TCPSocket.new('127.0.0.1', @port)
+    sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+    responses = sock.read(4096)
+    assert_match %r{\AHTTP/1.[01] 200\b}, responses
+    assert_match %r{^after_reply_called: false}, responses
+
+    sock = TCPSocket.new('127.0.0.1', @port)
+    sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+    responses = sock.read(4096)
+    assert_match %r{\AHTTP/1.[01] 200\b}, responses
+    assert_match %r{^after_reply_called: true}, responses
+
+    sock.close
+  end
+
   def test_broken_app
     teardown
     app = lambda { |env| raise RuntimeError, "hello" }
-- 
2.29.2



^ permalink raw reply related	[relevance 4%]

* [PATCH] PATCH: doc: add IMAP/IMAPS mailbox info
@ 2020-09-08  2:06  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-09-08  2:06 UTC (permalink / raw)
  To: unicorn-public

Apparently, an IMAP server happened at some point and somehow
stays running.  Maybe more users have IMAP clients than NNTP
clients...
---
 ISSUES | 5 ++++-
 README | 7 +++++++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/ISSUES b/ISSUES
index d11dc56..4513ad5 100644
--- a/ISSUES
+++ b/ISSUES
@@ -76,15 +76,18 @@ document distributed with git) on guidelines for patch submission.
 * public: mailto:unicorn-public@yhbt.net
 * nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
 * nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
+* imaps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn.0
 * https://yhbt.net/unicorn-public/
 * http://ou63pmih66umazou.onion/unicorn-public/
 
 Mailing list subscription is optional, so Cc: all participants.
 
-You can follow along via NNTP (read-only):
+You can follow along via NNTP or IMAP (read-only):
 
 	nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
 	nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
+	imaps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn.0
+	imap://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn.0
 
 Or Atom feeds:
 
diff --git a/README b/README
index 0e95f48..35a7388 100644
--- a/README
+++ b/README
@@ -136,10 +136,17 @@ requests) go to the mailing list/newsgroup.  See the ISSUES document for
 information on the {mailing list}[mailto:unicorn-public@yhbt.net].
 
 The mailing list is archived at https://yhbt.net/unicorn-public/
+
 Read-only NNTP access is available at:
 nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn and
 nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
 
+Read-only IMAP access is also avaialble at:
+imaps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn.0 and
+imap://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn.0
+AUTH=ANONYMOUS mechanism is supported, as is any username+password
+combination.
+
 For the latest on unicorn releases, you may also finger us at
 unicorn@yhbt.net or check our NEWS page (and subscribe to our Atom
 feed).

^ permalink raw reply related	[relevance 4%]

* Re: [PATCH] Add early hints support
  2020-07-20 11:53  6%                   ` Jean Boussier
@ 2020-07-20 20:27  6%                     ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-07-20 20:27 UTC (permalink / raw)
  To: Jean Boussier; +Cc: unicorn-public

Jean Boussier <jean.boussier@shopify.com> wrote:
> > upcase seems VERY compelling in a micro benchmark since it can
> > go straight into opt_case_dispatch.  But I worry the extra
> > garbage might have a different effect in a real app, especially
> > with more headers.
> 
> It is indeed debatable. The thing is that on match, that Regexp
> will generate a MatchData and a garbage string. So I'd think
> the extra GC pressure would be totally negligible.

Ah good point.  I take it Ruby doesn't lazy-allocate or have
optimizations to quickly recycle unused MatchData?  (Maybe it
could, if somebody who hacks on the VM is reading this).

> But I agree it's hard to tell without a larger more representative benchmark.

Yes, I would accept patches to speed these up with comprehensive
benchmark :)

> > Maybe:
> > 
> > 	 (val || ''.freeze).include?("\n".freeze)
> > 
> > Can work for those buggy apps, though...
> 
> Yes. or:
> 
>    vall && val.include?...

Yes.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-20 10:55  6%                 ` Eric Wong
@ 2020-07-20 11:53  6%                   ` Jean Boussier
  2020-07-20 20:27  6%                     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-07-20 11:53 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

> upcase seems VERY compelling in a micro benchmark since it can
> go straight into opt_case_dispatch.  But I worry the extra
> garbage might have a different effect in a real app, especially
> with more headers.

It is indeed debatable. The thing is that on match, that Regexp
will generate a MatchData and a garbage string. So I'd think
the extra GC pressure would be totally negligible.

But I agree it's hard to tell without a larger more representative benchmark.

> Maybe:
> 
> 	 (val || ''.freeze).include?("\n".freeze)
> 
> Can work for those buggy apps, though...

Yes. or:

   vall && val.include?...


^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-20 10:27  5%               ` Jean Boussier
@ 2020-07-20 10:55  6%                 ` Eric Wong
  2020-07-20 11:53  6%                   ` Jean Boussier
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-07-20 10:55 UTC (permalink / raw)
  To: Jean Boussier; +Cc: unicorn-public

Jean Boussier <jean.boussier@shopify.com> wrote:
> > `match?' is Ruby 2.4+, which is probably too big a jump since
> > we're still on Ruby 1.9.3 at the moment...
> 
> That's what I figured.
> 
> > String comparison as in `==' and `!='?  Would be interested to know
> > where and what improvements can be had.

Thanks for the benchmarks.

> One place that jumped to mind when I saw it is http_response_write.
> But there are many other places where Regexp are used to do case
> insensitive comparisons.
> 
> ```
> require 'benchmark/ips'
> 
> def http_response_write(headers)
>   headers.each do |key, value|
>     case key
>     when %r{\A(?:Date|Connection)\z}i
>       next
>     end
>   end
> end
> 
> def http_response_write_upcase(headers)
>   headers.each do |key, value|
>     case key.upcase
>     when 'DATE'.freeze, 'CONNECTION'.freeze
>       next
>     end
>   end
> end
> 
> def http_response_write_casecmp(headers)
>   headers.each do |key, value|
>     case key
>     when key.casecmp?('Date'.freeze) || key.casecmp?('Connection'.freeze)
>       next
>     end
>   end
> end
> 
> HEADERS = {
>   'Foo' => 'bar',
>   'Date' => 'plop',
>   'User-Agent' => 'blah',
> }
> 
> Benchmark.ips do |x|
>   x.report('original') { http_response_write(HEADERS) }
>   x.report('upcase') { http_response_write_upcase(HEADERS) }
>   x.report('casecmp?') { http_response_write_casecmp(HEADERS) }
>   x.compare!
> end
> ```
> 
> ```
> Warming up --------------------------------------
>             original    82.066k i/100ms
>               upcase   177.429k i/100ms
>             casecmp?    96.288k i/100ms
> Calculating -------------------------------------
>             original    831.610k (± 1.6%) i/s -      4.185M in   5.034146s
>               upcase      1.770M (± 1.6%) i/s -      8.871M in   5.013796s
>             casecmp?    979.618k (± 1.3%) i/s -      4.911M in   5.013678s
> 
> Comparison:
>               upcase:  1769883.2 i/s
>             casecmp?:   979618.3 i/s - 1.81x  (± 0.00) slower
>             original:   831610.2 i/s - 2.13x  (± 0.00) slower
> ```

upcase seems VERY compelling in a micro benchmark since it can
go straight into opt_case_dispatch.  But I worry the extra
garbage might have a different effect in a real app, especially
with more headers.

> Similarly, that method use `value =~ /\n/` which could be replaced
> favorably for `value.include?("\n".freeze)`

Yes, we tried that a few years ago and broke existing code
that had `nil' value, so it was promptly reverted:

https://yhbt.net/unicorn-public/CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com/

Maybe:

	 (val || ''.freeze).include?("\n".freeze)

Can work for those buggy apps, though...

> ```
> VAL = "foobar"
> Benchmark.ips do |x|
>   x.report('=~') { VAL =~ /\n/ }
>   x.report('include?') { VAL.include?("\n".freeze) }
>   x.compare!
> end

> > Ruby just seems hopeless performance-wise 
> 
> Well, the gap between 1.9.3 and 2.5+ is pretty big performance-wise.

True, but it's still pretty slow :>

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-20 10:09  6%             ` Eric Wong
@ 2020-07-20 10:27  5%               ` Jean Boussier
  2020-07-20 10:55  6%                 ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-07-20 10:27 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

> `match?' is Ruby 2.4+, which is probably too big a jump since
> we're still on Ruby 1.9.3 at the moment...

That's what I figured.

> String comparison as in `==' and `!='?  Would be interested to know
> where and what improvements can be had.

One place that jumped to mind when I saw it is http_response_write.
But there are many other places where Regexp are used to do case
insensitive comparisons.

```
require 'benchmark/ips'

def http_response_write(headers)
  headers.each do |key, value|
    case key
    when %r{\A(?:Date|Connection)\z}i
      next
    end
  end
end

def http_response_write_upcase(headers)
  headers.each do |key, value|
    case key.upcase
    when 'DATE'.freeze, 'CONNECTION'.freeze
      next
    end
  end
end

def http_response_write_casecmp(headers)
  headers.each do |key, value|
    case key
    when key.casecmp?('Date'.freeze) || key.casecmp?('Connection'.freeze)
      next
    end
  end
end

HEADERS = {
  'Foo' => 'bar',
  'Date' => 'plop',
  'User-Agent' => 'blah',
}

Benchmark.ips do |x|
  x.report('original') { http_response_write(HEADERS) }
  x.report('upcase') { http_response_write_upcase(HEADERS) }
  x.report('casecmp?') { http_response_write_casecmp(HEADERS) }
  x.compare!
end
```

```
Warming up --------------------------------------
            original    82.066k i/100ms
              upcase   177.429k i/100ms
            casecmp?    96.288k i/100ms
Calculating -------------------------------------
            original    831.610k (± 1.6%) i/s -      4.185M in   5.034146s
              upcase      1.770M (± 1.6%) i/s -      8.871M in   5.013796s
            casecmp?    979.618k (± 1.3%) i/s -      4.911M in   5.013678s

Comparison:
              upcase:  1769883.2 i/s
            casecmp?:   979618.3 i/s - 1.81x  (± 0.00) slower
            original:   831610.2 i/s - 2.13x  (± 0.00) slower
```

Similarly, that method use `value =~ /\n/` which could be replaced
favorably for `value.include?("\n".freeze)`

```
VAL = "foobar"
Benchmark.ips do |x|
  x.report('=~') { VAL =~ /\n/ }
  x.report('include?') { VAL.include?("\n".freeze) }
  x.compare!
end
```

```
Warming up --------------------------------------
                  =~   409.096k i/100ms
            include?     1.322M i/100ms
Calculating -------------------------------------
                  =~      4.083M (± 2.1%) i/s -     20.455M in   5.011539s
            include?     13.053M (± 1.5%) i/s -     66.125M in   5.067097s

Comparison:
            include?: 13052859.2 i/s
                  =~:  4083401.3 i/s - 3.20x  (± 0.00) slower

```

> Ruby just seems hopeless performance-wise 

Well, the gap between 1.9.3 and 2.5+ is pretty big performance-wise.



^ permalink raw reply	[relevance 5%]

* Re: [PATCH] Add early hints support
  2020-07-20  9:18  6%           ` Jean Boussier
@ 2020-07-20 10:09  6%             ` Eric Wong
  2020-07-20 10:27  5%               ` Jean Boussier
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-07-20 10:09 UTC (permalink / raw)
  To: Jean Boussier; +Cc: unicorn-public

Jean Boussier <jean.boussier@shopify.com> wrote:
> > I guess... I was wondering more in terms of the big picture with
> > HTTP parsing, I/O, and response generation taken into account.
> 
> I wasn't too sure how to benchmark the entire loop. But regardless
> 18/20M i/s shows that it should be meaningless. It's only executed
> once per request, so would "waste" 1 second for 18 million requests
> processed.

Yeah, it can't support persistent connections, so Unix sockets
are required to avoid TCP port exhaustion and ensure repeatable
results.

> > (and there may be improvements in those areas later this year,
> > assuming the world doesn't end sooner...)
> 
> If you are interested in general optimizations, I did spot a bunch
> of sub optimal patterns, such as `=~` in places where a string
> comparison, or `match?` would do.
> 
> However many optimizations are only available on more recent
> rubies.

`match?' is Ruby 2.4+, which is probably too big a jump since
we're still on Ruby 1.9.3 at the moment... (though maybe 2.3+
is/was on the horizon).

String comparison as in `==' and `!='?  Would be interested to know
where and what improvements can be had.

But yeah, Ruby just seems hopeless performance-wise (and
compatibility-wise :P); so unicorn mainly exists to keep legacy
projects alive.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-17  1:19  6%         ` Eric Wong
@ 2020-07-20  9:18  6%           ` Jean Boussier
  2020-07-20 10:09  6%             ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-07-20  9:18 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

> I guess... I was wondering more in terms of the big picture with
> HTTP parsing, I/O, and response generation taken into account.

I wasn't too sure how to benchmark the entire loop. But regardless
18/20M i/s shows that it should be meaningless. It's only executed
once per request, so would "waste" 1 second for 18 million requests
processed.

> (and there may be improvements in those areas later this year,
> assuming the world doesn't end sooner...)

If you are interested in general optimizations, I did spot a bunch
of sub optimal patterns, such as `=~` in places where a string
comparison, or `match?` would do.

However many optimizations are only available on more recent
rubies.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-16 12:24  5%       ` Jean Boussier
@ 2020-07-17  1:19  6%         ` Eric Wong
  2020-07-20  9:18  6%           ` Jean Boussier
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-07-17  1:19 UTC (permalink / raw)
  To: Jean Boussier; +Cc: unicorn-public

Jean Boussier <jean.boussier@shopify.com> wrote:
>                 ivar: 20934740.7 i/s
>                 attr: 18622122.9 i/s - 1.12x  (± 0.00) slower
>         ivar:missing: 18408273.1 i/s - 1.14x  (± 0.00) slower
> 
> So yeah, the difference between raw @ivar and an attr_reader
> is barely noticeable, and both are very fast.

I guess... I was wondering more in terms of the big picture with
HTTP parsing, I/O, and response generation taken into account.
(and there may be improvements in those areas later this year,
 assuming the world doesn't end sooner...)

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-16 12:16  6%     ` Eric Wong
@ 2020-07-16 12:24  5%       ` Jean Boussier
  2020-07-17  1:19  6%         ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-07-16 12:24 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

> Alright, numbers would've been helpful but I'll take your word
> for it.

My bad, I didn't get the hint:

```
require 'benchmark/ips'

class Foo
  attr_accessor :foo

  def initialize
    @foo = 1
  end

  def missing_ivar
    @bar
  end

  def ivar
    @foo
  end
  
  def accessor
    foo
  end
end

foo = Foo.new

Benchmark.ips do |x|
  x.report('ivar') { foo.ivar }
  x.report('attr') { foo.accessor }
  x.report('ivar:missing') { foo.missing_ivar }
  x.compare!
end
```

ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]

Warming up --------------------------------------
                ivar     2.105M i/100ms
                attr     1.824M i/100ms
        ivar:missing     1.795M i/100ms
Calculating -------------------------------------
                ivar     20.935M (± 1.5%) i/s -    105.248M in   5.028625s
                attr     18.622M (± 0.8%) i/s -     94.845M in   5.093471s
        ivar:missing     18.408M (± 1.1%) i/s -     93.357M in   5.072160s

Comparison:
                ivar: 20934740.7 i/s
                attr: 18622122.9 i/s - 1.12x  (± 0.00) slower
        ivar:missing: 18408273.1 i/s - 1.14x  (± 0.00) slower

So yeah, the difference between raw @ivar and an attr_reader is barely noticeable, and both are very fast.



^ permalink raw reply	[relevance 5%]

* Re: [PATCH] Add early hints support
  2020-07-16 11:41  6%   ` Jean Boussier
@ 2020-07-16 12:16  6%     ` Eric Wong
  2020-07-16 12:24  5%       ` Jean Boussier
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-07-16 12:16 UTC (permalink / raw)
  To: Jean Boussier; +Cc: unicorn-public

Jean Boussier <jean.boussier@shopify.com> wrote:
> Thanks for the very timely response.
> 
> > Since this RDoc ends up on the website, links to any relevant
> > Rails documentation and RFC 8297 would also be appropriate.
> > Otherwise non-Rails users like me might have no clue what
> > it's for.
> 
> I updated the documentation, let me know what you think of it.

It's fine, pushed for now.  Will release in a few days or a week
in case there's comments from others.

> > Are the method calls for .to_s necessary?
> 
> I don't think they are, I mostly took inspiration from the puma implementation
> that does all this defensive checks. Based on how that interface is
> used by Rails, we could assume both keys and values are strings already.
> 
> I simplified the implementation.

OK.

> > Eep, extra branch...  What's the performance impact for existing
> > users when not activated? (on Unix sockets).
> 
> Extremely small.

Alright, numbers would've been helpful but I'll take your word
for it.

If we get more branches in the main loop for other new features;
I wonder if generating the code for process_client dynamically
at initialize-time is worth it to avoid branches in the main
loop... (or if there's some way to hint MJIT to do that).

> > Perhaps bypassing the method and accessing the @early_hints ivar
> > directly can be slightly faster w/o method dispatch.  That
> > should also allow using attr_writer instead of attr_accessor,
> > I think.
> 
>  attr_reader is very optimized in MRI, it's barely slower than @early_hints.
> Also it ensure that it doesn't emit a warning in verbose mode if the variable
> isn't initialized.

Thanks again.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-16 10:50  5% ` Eric Wong
@ 2020-07-16 11:41  6%   ` Jean Boussier
  2020-07-16 12:16  6%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-07-16 11:41 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

Thanks for the very timely response.

> Since this RDoc ends up on the website, links to any relevant
> Rails documentation and RFC 8297 would also be appropriate.
> Otherwise non-Rails users like me might have no clue what
> it's for.

I updated the documentation, let me know what you think of it.

> Are the method calls for .to_s necessary?

I don't think they are, I mostly took inspiration from the puma implementation
that does all this defensive checks. Based on how that interface is
used by Rails, we could assume both keys and values are strings already.

I simplified the implementation.

> Eep, extra branch...  What's the performance impact for existing
> users when not activated? (on Unix sockets).

Extremely small.

> 
> Perhaps bypassing the method and accessing the @early_hints ivar
> directly can be slightly faster w/o method dispatch.  That
> should also allow using attr_writer instead of attr_accessor,
> I think.

 attr_reader is very optimized in MRI, it's barely slower than @early_hints.
Also it ensure that it doesn't emit a warning in verbose mode if the variable
isn't initialized.

From e0494e10de6549d1b513eef03e68bfa58a6b26ec Mon Sep 17 00:00:00 2001
From: Jean Boussier <jean.boussier@gmail.com>
Date: Thu, 16 Jul 2020 11:39:30 +0200
Subject: [PATCH] Add early hints support

While not part of the rack spec, this API is exposed
by both puma and falcon, and Rails use it when available.

The 103 Early Hints response code is specified in RFC 8297.
---
 lib/unicorn/configurator.rb |  9 +++++++++
 lib/unicorn/http_server.rb  | 31 +++++++++++++++++++++++++++++--
 test/unit/test_server.rb    | 30 ++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index c3a4f2d..b0606af 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -276,6 +276,15 @@ def default_middleware(bool)
     set_bool(:default_middleware, bool)
   end
 
+  # sets whether to enable the proposed early hints Rack API.
+  # If enabled, Rails 5.2+ will automatically send a 103 Early Hint
+  # for all the `javascript_include_tag` and `stylesheet_link_tag`
+  # in your response. See: https://api.rubyonrails.org/v5.2/classes/ActionDispatch/Request.html#method-i-send_early_hints
+  # See also https://tools.ietf.org/html/rfc8297
+  def early_hints(bool)
+    set_bool(:early_hints, bool)
+  end
+
   # sets listeners to the given +addresses+, replacing or augmenting the
   # current set.  This is for the global listener pool shared by all
   # worker processes.  For per-worker listeners, see the after_fork example
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 45a2e97..05dad99 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user,
-                :default_middleware
+                :default_middleware, :early_hints
   attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
@@ -588,6 +588,25 @@ def handle_error(client, e)
   rescue
   end
 
+  def e103_response_write(client, headers)
+    response = if @request.response_start_sent
+      "103 Early Hints\r\n"
+    else
+      "HTTP/1.1 103 Early Hints\r\n"
+    end
+
+    headers.each_pair do |k, vs|
+      next if !vs || vs.empty?
+      values = vs.to_s.split("\n".freeze)
+      values.each do |v|
+        response << "#{k}: #{v}\r\n"
+      end
+    end
+    response << "\r\n".freeze
+    response << "HTTP/1.1 ".freeze if @request.response_start_sent
+    client.write(response)
+  end
+
   def e100_response_write(client, env)
     # We use String#freeze to avoid allocations under Ruby 2.1+
     # Not many users hit this code path, so it's better to reduce the
@@ -602,7 +621,15 @@ def e100_response_write(client, env)
   # once a client is accepted, it is processed in its entirety here
   # in 3 easy steps: read request, call app, write app response
   def process_client(client)
-    status, headers, body = @app.call(env = @request.read(client))
+    env = @request.read(client)
+
+    if early_hints
+      env["rack.early_hints"] = lambda do |headers|
+        e103_response_write(client, headers)
+      end
+    end
+
+    status, headers, body = @app.call(env)
 
     begin
       return if @request.hijacked?
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 8096955..d706243 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,6 +23,16 @@ def call(env)
   end
 end
 
+class TestEarlyHintsHandler
+  def call(env)
+    while env['rack.input'].read(4096)
+    end
+    env['rack.early_hints'].call(
+      "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
+    )
+    [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
+  end
+end
 
 class WebServerTest < Test::Unit::TestCase
 
@@ -84,6 +94,26 @@ def test_preload_app_config
     tmp.close!
   end
 
+  def test_early_hints
+    teardown
+    redirect_test_io do
+      @server = HttpServer.new(TestEarlyHintsHandler.new,
+                               :listeners => [ "127.0.0.1:#@port"],
+                               :early_hints => true)
+      @server.start
+    end
+
+    sock = TCPSocket.new('127.0.0.1', @port)
+    sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+    responses = sock.sysread(4096)
+    assert_match %r{\AHTTP/1.[01] 103\b}, responses
+    assert_match %r{^Link: </style\.css>}, responses
+    assert_match %r{^Link: </script\.js>}, responses
+
+    assert_match %r{^HTTP/1.[01] 200\b}, responses
+  end
+
   def test_broken_app
     teardown
     app = lambda { |env| raise RuntimeError, "hello" }
-- 
2.26.2



^ permalink raw reply related	[relevance 6%]

* Re: [PATCH] Add early hints support
  2020-07-16 10:05  4% [PATCH] Add early hints support Jean Boussier
@ 2020-07-16 10:50  5% ` Eric Wong
  2020-07-16 11:41  6%   ` Jean Boussier
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-07-16 10:50 UTC (permalink / raw)
  To: Jean Boussier; +Cc: unicorn-public

Jean Boussier <jean.boussier@shopify.com> wrote:
> While not part of the rack spec, this API is exposed
> by both puma and falcon, and Rails use it when available.
> 
> The 103 Early Hints response code is specified in RFC 8297.

Thanks, I can probably accept it with some minor tweaks.
Some comment below...

> ---
>  lib/unicorn/configurator.rb |  5 +++++
>  lib/unicorn/http_server.rb  | 33 +++++++++++++++++++++++++++++++--
>  test/unit/test_server.rb    | 30 ++++++++++++++++++++++++++++++
>  3 files changed, 66 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
> index c3a4f2d..43e91f4 100644
> --- a/lib/unicorn/configurator.rb
> +++ b/lib/unicorn/configurator.rb
> @@ -276,6 +276,11 @@ def default_middleware(bool)
>      set_bool(:default_middleware, bool)
>    end
>  
> +  # sets wether to enable rack's early hints API.

spelling: "whether".

Since it's not officially part of Rack, yet, perhaps something
like:

	Enable the proposed early hints Rack API.

I'm no grammar expert, so I also rephrased that sentence to
avoid the apostrophe.

Since this RDoc ends up on the website, links to any relevant
Rails documentation and RFC 8297 would also be appropriate.
Otherwise non-Rails users like me might have no clue what
it's for.

> +  def early_hints(bool)
> +    set_bool(:early_hints, bool)
> +  end
> +
>    # sets listeners to the given +addresses+, replacing or augmenting the
>    # current set.  This is for the global listener pool shared by all
>    # worker processes.  For per-worker listeners, see the after_fork example
> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 45a2e97..3e0c9a4 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -15,7 +15,7 @@ class Unicorn::HttpServer
>                  :before_fork, :after_fork, :before_exec,
>                  :listener_opts, :preload_app,
>                  :orig_app, :config, :ready_pipe, :user,
> -                :default_middleware
> +                :default_middleware, :early_hints
>    attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
>  
>    attr_reader :pid, :logger
> @@ -588,6 +588,27 @@ def handle_error(client, e)
>    rescue
>    end
>  
> +  def e103_response_write(client, headers)
> +    response = if @request.response_start_sent
> +      "103 Early Hints\r\n"
> +    else
> +      "HTTP/1.1 103 Early Hints\r\n"
> +    end
> +
> +    headers.each_pair do |k, vs|
> +      if vs.respond_to?(:to_s) && !vs.to_s.empty?
> +        vs.to_s.split("\n".freeze).each do |v|

Are the method calls for .to_s necessary?  At least for regular
Rack responses, this bit from unicorn/http_response.rb seems to
handle odd apps which (improperly) give `nil' value:

          if value =~ /\n/
            # avoiding blank, key-only cookies with /\n+/
            value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" }
          else
            buf << "#{key}: #{value}\r\n"
          end

The split would never be attempted on nil since it wouldn't
match /\n/, and the /\n+/ was needed in the Rails 2.3.5 days:

https://public-inbox.org/rack-devel/20100105235845.GB3377@dcvr.yhbt.net/
https://yhbt.net/unicorn-public/52400de1c9e9437b5c9df899f273485f663bb5b5/s/

> +          response << "#{k}: #{v}\r\n"
> +        end
> +      else
> +        response << "#{k}: #{vs}\r\n"
> +      end
> +    end
> +    response << "\r\n".freeze
> +    response << "HTTP/1.1 ".freeze if @request.response_start_sent
> +    client.write(response)
> +  end
> +
>    def e100_response_write(client, env)
>      # We use String#freeze to avoid allocations under Ruby 2.1+
>      # Not many users hit this code path, so it's better to reduce the
> @@ -602,7 +623,15 @@ def e100_response_write(client, env)
>    # once a client is accepted, it is processed in its entirety here
>    # in 3 easy steps: read request, call app, write app response
>    def process_client(client)
> -    status, headers, body = @app.call(env = @request.read(client))
> +    env = @request.read(client)
> +
> +    if early_hints
> +      env["rack.early_hints"] = lambda do |headers|
> +        e103_response_write(client, headers)
> +      end
> +    end

Eep, extra branch...  What's the performance impact for existing
users when not activated? (on Unix sockets).

Perhaps bypassing the method and accessing the @early_hints ivar
directly can be slightly faster w/o method dispatch.  That
should also allow using attr_writer instead of attr_accessor,
I think.

Not sure how much people here care about minor performance
regressions, here...  I don't really upgrade or touch old
Rack apps, anymore; and I'm certainly never going to buy
a faster CPU.

> +    status, headers, body = @app.call(env)
>  
>      begin
>        return if @request.hijacked?

^ permalink raw reply	[relevance 5%]

* [PATCH] Add early hints support
@ 2020-07-16 10:05  4% Jean Boussier
  2020-07-16 10:50  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jean Boussier @ 2020-07-16 10:05 UTC (permalink / raw)
  To: unicorn-public

While not part of the rack spec, this API is exposed
by both puma and falcon, and Rails use it when available.

The 103 Early Hints response code is specified in RFC 8297.
---
 lib/unicorn/configurator.rb |  5 +++++
 lib/unicorn/http_server.rb  | 33 +++++++++++++++++++++++++++++++--
 test/unit/test_server.rb    | 30 ++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index c3a4f2d..43e91f4 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -276,6 +276,11 @@ def default_middleware(bool)
     set_bool(:default_middleware, bool)
   end
 
+  # sets wether to enable rack's early hints API.
+  def early_hints(bool)
+    set_bool(:early_hints, bool)
+  end
+
   # sets listeners to the given +addresses+, replacing or augmenting the
   # current set.  This is for the global listener pool shared by all
   # worker processes.  For per-worker listeners, see the after_fork example
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 45a2e97..3e0c9a4 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user,
-                :default_middleware
+                :default_middleware, :early_hints
   attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
@@ -588,6 +588,27 @@ def handle_error(client, e)
   rescue
   end
 
+  def e103_response_write(client, headers)
+    response = if @request.response_start_sent
+      "103 Early Hints\r\n"
+    else
+      "HTTP/1.1 103 Early Hints\r\n"
+    end
+
+    headers.each_pair do |k, vs|
+      if vs.respond_to?(:to_s) && !vs.to_s.empty?
+        vs.to_s.split("\n".freeze).each do |v|
+          response << "#{k}: #{v}\r\n"
+        end
+      else
+        response << "#{k}: #{vs}\r\n"
+      end
+    end
+    response << "\r\n".freeze
+    response << "HTTP/1.1 ".freeze if @request.response_start_sent
+    client.write(response)
+  end
+
   def e100_response_write(client, env)
     # We use String#freeze to avoid allocations under Ruby 2.1+
     # Not many users hit this code path, so it's better to reduce the
@@ -602,7 +623,15 @@ def e100_response_write(client, env)
   # once a client is accepted, it is processed in its entirety here
   # in 3 easy steps: read request, call app, write app response
   def process_client(client)
-    status, headers, body = @app.call(env = @request.read(client))
+    env = @request.read(client)
+
+    if early_hints
+      env["rack.early_hints"] = lambda do |headers|
+        e103_response_write(client, headers)
+      end
+    end
+
+    status, headers, body = @app.call(env)
 
     begin
       return if @request.hijacked?
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 8096955..d706243 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,6 +23,16 @@ def call(env)
   end
 end
 
+class TestEarlyHintsHandler
+  def call(env)
+    while env['rack.input'].read(4096)
+    end
+    env['rack.early_hints'].call(
+      "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
+    )
+    [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
+  end
+end
 
 class WebServerTest < Test::Unit::TestCase
 
@@ -84,6 +94,26 @@ def test_preload_app_config
     tmp.close!
   end
 
+  def test_early_hints
+    teardown
+    redirect_test_io do
+      @server = HttpServer.new(TestEarlyHintsHandler.new,
+                               :listeners => [ "127.0.0.1:#@port"],
+                               :early_hints => true)
+      @server.start
+    end
+
+    sock = TCPSocket.new('127.0.0.1', @port)
+    sock.syswrite("GET / HTTP/1.0\r\n\r\n")
+
+    responses = sock.sysread(4096)
+    assert_match %r{\AHTTP/1.[01] 103\b}, responses
+    assert_match %r{^Link: </style\.css>}, responses
+    assert_match %r{^Link: </script\.js>}, responses
+
+    assert_match %r{^HTTP/1.[01] 200\b}, responses
+  end
+
   def test_broken_app
     teardown
     app = lambda { |env| raise RuntimeError, "hello" }
-- 
2.26.2



^ permalink raw reply related	[relevance 4%]

* [PATCH] doc: s/bogomips.org/yhbt.net/g
  2020-01-14  7:46  4% [PATCH] doc: s/bogomips.org/yhbt.net/g Eric Wong
@ 2020-01-14  7:46  2% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2020-01-14  7:46 UTC (permalink / raw)
  To: unicorn-public

bogomips.org is due to expire, soon, and I'm not willing to pay
extortionist fees to Ethos Capital/PIR/ICANN to keep a .org.  So
it's at yhbt.net, for now, but it will change again to
whatever's affordable...  Identity is overrated.

Tor users can use .onions and kick ICANN to the curb:

	torsocks w3m http://unicorn.ou63pmih66umazou.onion/
	torsocks git clone http://ou63pmih66umazou.onion/unicorn.git/
	torsocks w3m http://ou63pmih66umazou.onion/unicorn-public/

While we're at it, `s/news.gmane.org/news.gmane.io/g', too.
(but I suspect that'll need to be resynched since our mail
"List-Id:" header is changing).
---
 .olddoc.yml                      | 19 ++++++++++++-------
 Documentation/unicorn.1          |  8 ++++----
 Documentation/unicorn_rails.1    |  8 ++++----
 FAQ                              |  2 +-
 GNUmakefile                      |  4 ++--
 HACKING                          |  2 +-
 ISSUES                           | 24 ++++++++++++------------
 KNOWN_ISSUES                     |  4 ++--
 Links                            | 10 +++++-----
 README                           | 12 ++++++------
 SIGNALS                          |  2 +-
 Sandbox                          |  4 ++--
 archive/slrnpull.conf            |  2 +-
 examples/big_app_gc.rb           |  2 +-
 examples/logrotate.conf          |  4 ++--
 examples/nginx.conf              |  2 +-
 examples/unicorn.conf.minimal.rb |  4 ++--
 examples/unicorn.conf.rb         |  4 ++--
 ext/unicorn_http/unicorn_http.rl |  2 +-
 lib/unicorn.rb                   |  2 +-
 lib/unicorn/configurator.rb      |  6 +++---
 lib/unicorn/http_server.rb       |  2 +-
 lib/unicorn/oob_gc.rb            |  4 ++--
 unicorn.gemspec                  |  4 ++--
 24 files changed, 71 insertions(+), 66 deletions(-)

diff --git a/.olddoc.yml b/.olddoc.yml
index d2d340f..0609bdb 100644
--- a/.olddoc.yml
+++ b/.olddoc.yml
@@ -1,8 +1,9 @@
 ---
-cgit_url: https://bogomips.org/unicorn.git
-git_url: https://bogomips.org/unicorn.git
-rdoc_url: https://bogomips.org/unicorn/
-ml_url: https://bogomips.org/unicorn-public/
+cgit_url: https://yhbt.net/unicorn.git
+rdoc_url: https://yhbt.net/unicorn/
+ml_url:
+- https://yhbt.net/unicorn-public/
+- http://ou63pmih66umazou.onion/unicorn-public/
 merge_html:
   unicorn_1: Documentation/unicorn.1.html
   unicorn_rails_1: Documentation/unicorn_rails.1.html
@@ -11,7 +12,11 @@ noindex:
 - LATEST
 - TODO
 - unicorn_rails_1
-public_email: unicorn-public@bogomips.org
+public_email: unicorn-public@yhbt.net
 nntp_url:
-  - nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
-  - nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+- nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
+- nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
+- nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
+source_code:
+- git clone https://yhbt.net/unicorn.git
+- torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1
index 3f8cb96..d76d40f 100644
--- a/Documentation/unicorn.1
+++ b/Documentation/unicorn.1
@@ -154,7 +154,7 @@ TTIN \- increment the number of worker processes by one
 .IP \[bu] 2
 TTOU \- decrement the number of worker processes by one
 .PP
-See the SIGNALS (https://bogomips.org/unicorn/SIGNALS.html) document for
+See the SIGNALS (https://yhbt.net/unicorn/SIGNALS.html) document for
 full description of all signals used by Unicorn.
 .SH RACK ENVIRONMENT
 .PP
@@ -204,11 +204,11 @@ the unicorn config file.
 \f[I]Rack::Builder\f[] ri/RDoc
 .IP \[bu] 2
 \f[I]Unicorn::Configurator\f[] ri/RDoc
-.UR https://bogomips.org/unicorn/Unicorn/Configurator.html
+.UR https://yhbt.net/unicorn/Unicorn/Configurator.html
 .UE
 .IP \[bu] 2
 unicorn RDoc
-.UR https://bogomips.org/unicorn/
+.UR https://yhbt.net/unicorn/
 .UE
 .IP \[bu] 2
 Rack RDoc
@@ -219,4 +219,4 @@ Rackup HowTo
 .UR https://github.com/rack/rack/wiki/(tutorial)-rackup-howto
 .UE
 .SH AUTHORS
-The Unicorn Community <unicorn-public@bogomips.org>.
+The Unicorn Community <unicorn-public@yhbt.net>.
diff --git a/Documentation/unicorn_rails.1 b/Documentation/unicorn_rails.1
index 71c80be..fec0a2a 100644
--- a/Documentation/unicorn_rails.1
+++ b/Documentation/unicorn_rails.1
@@ -180,7 +180,7 @@ TTIN \- increment the number of worker processes by one
 .IP \[bu] 2
 TTOU \- decrement the number of worker processes by one
 .PP
-See the SIGNALS (https://bogomips.org/unicorn/SIGNALS.html) document for
+See the SIGNALS (https://yhbt.net/unicorn/SIGNALS.html) document for
 full description of all signals used by Unicorn.
 .SH SEE ALSO
 .IP \[bu] 2
@@ -189,11 +189,11 @@ unicorn(1)
 \f[I]Rack::Builder\f[] ri/RDoc
 .IP \[bu] 2
 \f[I]Unicorn::Configurator\f[] ri/RDoc
-.UR https://bogomips.org/unicorn/Unicorn/Configurator.html
+.UR https://yhbt.net/unicorn/Unicorn/Configurator.html
 .UE
 .IP \[bu] 2
 unicorn RDoc
-.UR https://bogomips.org/unicorn/
+.UR https://yhbt.net/unicorn/
 .UE
 .IP \[bu] 2
 Rack RDoc
@@ -204,4 +204,4 @@ Rackup HowTo
 .UR https://github.com/rack/rack/wiki/(tutorial)-rackup-howto
 .UE
 .SH AUTHORS
-The Unicorn Community <unicorn-public@bogomips.org>.
+The Unicorn Community <unicorn-public@yhbt.net>.
diff --git a/FAQ b/FAQ
index 4ae2034..018ca92 100644
--- a/FAQ
+++ b/FAQ
@@ -7,7 +7,7 @@ drained entirely by the application.  This may happen when request
 bodies are gzipped, as unicorn reads request body data lazily to avoid
 overhead from bad requests.
 
-Ref: https://bogomips.org/unicorn-public/FC91211E-FD32-432C-92FC-0318714C2170@zendesk.com/
+Ref: https://yhbt.net/unicorn-public/FC91211E-FD32-432C-92FC-0318714C2170@zendesk.com/
 
 === Why aren't my Rails log files rotated when I use SIGUSR1?
 
diff --git a/GNUmakefile b/GNUmakefile
index 94c46ee..eac3473 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -193,13 +193,13 @@ doc: .document $(ext)/unicorn_http.c man html .olddoc.yml $(PLACEHOLDERS)
 	install -m644 $(man1_paths) doc/
 	tar cf - $$(git ls-files examples/) | (cd doc && tar xf -)
 
-# publishes docs to https://bogomips.org/unicorn/
+# publishes docs to https://yhbt.net/unicorn/
 publish_doc:
 	-git set-file-times
 	$(MAKE) doc
 	$(MAKE) doc_gz
 	chmod 644 $$(find doc -type f)
-	$(RSYNC) -av doc/ bogomips.org:/srv/bogomips/unicorn/
+	$(RSYNC) -av doc/ yhbt.net:/srv/yhbt/unicorn/
 	git ls-files | xargs touch
 
 # Create gzip variants of the same timestamp as the original so nginx
diff --git a/HACKING b/HACKING
index be1bb85..976bf92 100644
--- a/HACKING
+++ b/HACKING
@@ -57,7 +57,7 @@ Please wrap documentation at 72 characters-per-line or less (long URLs
 are exempt) so it is comfortably readable from terminals.
 
 When referencing mailing list posts, use
-<tt>https://bogomips.org/unicorn-public/$MESSAGE_ID/</tt> if possible
+<tt>https://yhbt.net/unicorn-public/$MESSAGE_ID/</tt> if possible
 since the Message-ID remains searchable even if a particular site
 becomes unavailable.
 
diff --git a/ISSUES b/ISSUES
index 473da2f..d11dc56 100644
--- a/ISSUES
+++ b/ISSUES
@@ -1,9 +1,9 @@
 = Issues
 
-mailto:unicorn-public@bogomips.org is the best place to report bugs,
+mailto:unicorn-public@yhbt.net is the best place to report bugs,
 submit patches and/or obtain support after you have searched the
-{email archives}[https://bogomips.org/unicorn-public/] and
-{documentation}[https://bogomips.org/unicorn/].
+{email archives}[https://yhbt.net/unicorn-public/] and
+{documentation}[https://yhbt.net/unicorn/].
 
 * No subscription will ever be required to email us
 * Cc: all participants in a thread or commit, as subscription is optional
@@ -12,12 +12,12 @@ submit patches and/or obtain support after you have searched the
 * Do not send HTML mail or images,
   they hurt reader privacy and will be flagged as spam
 * Anonymous and pseudonymous messages will ALWAYS be welcome
-* The email submission port (587) is enabled on the bogomips.org MX:
-  https://bogomips.org/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
+* The email submission port (587) is enabled on the yhbt.net MX:
+  https://yhbt.net/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
 
 We will never have a centralized or formal bug tracker.  Instead we
 can interoperate with any bug tracker which can Cc: us plain-text to
-mailto:unicorn-public@bogomips.org   This includes the Debian BTS
+mailto:unicorn-public@yhbt.net   This includes the Debian BTS
 at https://bugs.debian.org/unicorn and possibly others.
 
 If your issue is of a sensitive nature or you're just shy in public,
@@ -73,10 +73,10 @@ document distributed with git) on guidelines for patch submission.
 
 == Contact Info
 
-* public: mailto:unicorn-public@bogomips.org
-* nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+* public: mailto:unicorn-public@yhbt.net
+* nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
 * nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
-* https://bogomips.org/unicorn-public/
+* https://yhbt.net/unicorn-public/
 * http://ou63pmih66umazou.onion/unicorn-public/
 
 Mailing list subscription is optional, so Cc: all participants.
@@ -84,13 +84,13 @@ Mailing list subscription is optional, so Cc: all participants.
 You can follow along via NNTP (read-only):
 
 	nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
-	nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+	nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
 
 Or Atom feeds:
 
-	https://bogomips.org/unicorn-public/new.atom
+	https://yhbt.net/unicorn-public/new.atom
 	http://ou63pmih66umazou.onion/unicorn-public/new.atom
 
-	The HTML archives at https://bogomips.org/unicorn-public/
+	The HTML archives at https://yhbt.net/unicorn-public/
 	also has links to per-thread Atom feeds and downloadable
 	mboxes.
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index ebd4822..0017f20 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -9,7 +9,7 @@ acceptable solution.  Those issues are documented here.
   handlers.
 
 * Issues with FreeBSD jails can be worked around as documented by Tatsuya Ono:
-  https://bogomips.org/unicorn-public/CAHBuKRj09FdxAgzsefJWotexw-7JYZGJMtgUp_dhjPz9VbKD6Q@mail.gmail.com/
+  https://yhbt.net/unicorn-public/CAHBuKRj09FdxAgzsefJWotexw-7JYZGJMtgUp_dhjPz9VbKD6Q@mail.gmail.com/
 
 * PRNGs (pseudo-random number generators) loaded before forking
   (e.g. "preload_app true") may need to have their internal state
@@ -60,7 +60,7 @@ acceptable solution.  Those issues are documented here.
   application to use Rails 2.3.2 and you have no other choice, then
   you may edit your unicorn gemspec and remove the Rack dependency.
 
-  ref: https://bogomips.org/unicorn-public/20091014221552.GA30624@dcvr.yhbt.net/
+  ref: https://yhbt.net/unicorn-public/20091014221552.GA30624@dcvr.yhbt.net/
   Note: the workaround described in the article above only made
   the issue more subtle and we didn't notice them immediately.
 
diff --git a/Links b/Links
index 10551a6..f81142d 100644
--- a/Links
+++ b/Links
@@ -2,7 +2,7 @@
 
 If you're interested in unicorn, you may be interested in some of the projects
 listed below.  If you have any links to add/change/remove, please tell us at
-mailto:unicorn-public@bogomips.org!
+mailto:unicorn-public@yhbt.net!
 
 == Disclaimer
 
@@ -23,10 +23,10 @@ or services behind them.
 * {golden_brindle}[https://github.com/simonoff/golden_brindle] - tool to
   manage multiple unicorn instances/applications on a single server
 
-* {raindrops}[https://bogomips.org/raindrops/] - real-time stats for
+* {raindrops}[https://yhbt.net/raindrops/] - real-time stats for
   preforking Rack servers
 
-* {UnXF}[https://bogomips.org/unxf/]  Un-X-Forward* the Rack environment,
+* {UnXF}[https://yhbt.net/unxf/]  Un-X-Forward* the Rack environment,
   useful since unicorn is designed to be deployed behind a reverse proxy.
 
 === unicorn is written to work with
@@ -52,7 +52,7 @@ or services behind them.
 * {Mongrel}[https://rubygems.org/gems/mongrel] - the awesome webserver
   unicorn is based on.  A historical archive of the mongrel dev list
   featuring early discussions of unicorn is available at:
-  https://bogomips.org/mongrel-devel/
+  https://yhbt.net/mongrel-devel/
 
-* {david}[https://bogomips.org/david.git] - a tool to explain why you need
+* {david}[https://yhbt.net/david.git] - a tool to explain why you need
   nginx in front of unicorn
diff --git a/README b/README
index 89467fc..0e95f48 100644
--- a/README
+++ b/README
@@ -80,12 +80,12 @@ You may install it via RubyGems on RubyGems.org:
 You can get the latest source via git from the following locations
 (these versions may not be stable):
 
-  https://bogomips.org/unicorn.git
+  https://yhbt.net/unicorn.git
   https://repo.or.cz/unicorn.git (mirror)
 
 You may browse the code from the web:
 
-* https://bogomips.org/unicorn.git
+* https://yhbt.net/unicorn.git
 * https://repo.or.cz/w/unicorn.git (gitweb)
 
 See the HACKING guide on how to contribute and build prerelease gems
@@ -133,13 +133,13 @@ and libraries which run on top of it.
 
 All feedback (bug reports, user/development dicussion, patches, pull
 requests) go to the mailing list/newsgroup.  See the ISSUES document for
-information on the {mailing list}[mailto:unicorn-public@bogomips.org].
+information on the {mailing list}[mailto:unicorn-public@yhbt.net].
 
-The mailing list is archived at https://bogomips.org/unicorn-public/
+The mailing list is archived at https://yhbt.net/unicorn-public/
 Read-only NNTP access is available at:
 nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn and
-nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
 
 For the latest on unicorn releases, you may also finger us at
-unicorn@bogomips.org or check our NEWS page (and subscribe to our Atom
+unicorn@yhbt.net or check our NEWS page (and subscribe to our Atom
 feed).
diff --git a/SIGNALS b/SIGNALS
index 1af851d..7321f2b 100644
--- a/SIGNALS
+++ b/SIGNALS
@@ -8,7 +8,7 @@ should be possible to easily share process management scripts between
 Unicorn and nginx.
 
 One example init script is distributed with unicorn:
-https://bogomips.org/unicorn/examples/init.sh
+https://yhbt.net/unicorn/examples/init.sh
 
 === Master Process
 
diff --git a/Sandbox b/Sandbox
index d0f915e..651e5cd 100644
--- a/Sandbox
+++ b/Sandbox
@@ -34,7 +34,7 @@ is the primary issue with sandboxing tools such as Bundler and Isolate.
 If you're bundling unicorn, use "bundle exec unicorn" (or "bundle exec
 unicorn_rails") to start unicorn with the correct environment variables
 
-ref: https://bogomips.org/unicorn-public/9ECF07C4-5216-47BE-961D-AFC0F0C82060@internetfamo.us/
+ref: https://yhbt.net/unicorn-public/9ECF07C4-5216-47BE-961D-AFC0F0C82060@internetfamo.us/
 
 Otherwise (if you choose to not sandbox your unicorn installation), we
 expect the tips for Isolate (below) apply, too.
@@ -44,7 +44,7 @@ expect the tips for Isolate (below) apply, too.
 This is no longer be an issue as of bundler 0.9.17
 
 ref:
-https://bogomips.org/unicorn-public/8FC34B23-5994-41CC-B5AF-7198EF06909E@tramchase.com/
+https://yhbt.net/unicorn-public/8FC34B23-5994-41CC-B5AF-7198EF06909E@tramchase.com/
 
 === BUNDLE_GEMFILE for Capistrano users
 
diff --git a/archive/slrnpull.conf b/archive/slrnpull.conf
index fcfcafe..fd04f97 100644
--- a/archive/slrnpull.conf
+++ b/archive/slrnpull.conf
@@ -1,4 +1,4 @@
 # group_name                         max        expire     headers_only
 gmane.comp.lang.ruby.unicorn.general 1000000000 1000000000 0
 
-# usage: slrnpull -d $PWD -h news.gmane.org --no-post
+# usage: slrnpull -d $PWD -h news.gmane.io --no-post
diff --git a/examples/big_app_gc.rb b/examples/big_app_gc.rb
index 9d05719..c1bae10 100644
--- a/examples/big_app_gc.rb
+++ b/examples/big_app_gc.rb
@@ -1,2 +1,2 @@
-# see {Unicorn::OobGC}[https://bogomips.org/unicorn/Unicorn/OobGC.html]
+# see {Unicorn::OobGC}[https://yhbt.net/unicorn/Unicorn/OobGC.html]
 # Unicorn::OobGC was broken in Unicorn v3.3.1 - v3.6.1 and fixed in v3.6.2
diff --git a/examples/logrotate.conf b/examples/logrotate.conf
index 77a01b5..c3aa40d 100644
--- a/examples/logrotate.conf
+++ b/examples/logrotate.conf
@@ -5,7 +5,7 @@
 #    https://linux.die.net/man/8/logrotate
 #
 # public logrotate-related discussion in our archives:
-#    https://bogomips.org/unicorn-public/?q=logrotate
+#    https://yhbt.net/unicorn-public/?q=logrotate
 
 # Modify the following glob to match the logfiles your app writes to:
 /var/log/unicorn_app/*.log {
@@ -33,7 +33,7 @@
 		systemctl kill -s SIGUSR1 unicorn@2.service
 
 		# Examples for other process management systems appreciated
-		# Mail us at unicorn-public@bogomips.org
+		# Mail us at unicorn-public@yhbt.net
 		# (see above for archives)
 
 		# If you use a pid file and assuming your pid file
diff --git a/examples/nginx.conf b/examples/nginx.conf
index b6b69c1..c5026f9 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -113,7 +113,7 @@ http {
     # try_files directive appeared in in nginx 0.7.27 and has stabilized
     # over time.  Older versions of nginx (e.g. 0.6.x) requires
     # "if (!-f $request_filename)" which was less efficient:
-    # https://bogomips.org/unicorn.git/tree/examples/nginx.conf?id=v3.3.1#n127
+    # https://yhbt.net/unicorn.git/tree/examples/nginx.conf?id=v3.3.1#n127
     try_files $uri/index.html $uri.html $uri @app;
 
     location @app {
diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb
index 2d1bf0a..46fd634 100644
--- a/examples/unicorn.conf.minimal.rb
+++ b/examples/unicorn.conf.minimal.rb
@@ -1,9 +1,9 @@
 # Minimal sample configuration file for Unicorn (not Rack) when used
 # with daemonization (unicorn -D) started in your working directory.
 #
-# See https://bogomips.org/unicorn/Unicorn/Configurator.html for complete
+# See https://yhbt.net/unicorn/Unicorn/Configurator.html for complete
 # documentation.
-# See also https://bogomips.org/unicorn/examples/unicorn.conf.rb for
+# See also https://yhbt.net/unicorn/examples/unicorn.conf.rb for
 # a more verbose configuration using more features.
 
 listen 2007 # by default Unicorn listens on port 8080
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index d2897ef..d90bdc4 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -2,10 +2,10 @@
 #
 # This configuration file documents many features of Unicorn
 # that may not be needed for some applications. See
-# https://bogomips.org/unicorn/examples/unicorn.conf.minimal.rb
+# https://yhbt.net/unicorn/examples/unicorn.conf.minimal.rb
 # for a much simpler configuration file.
 #
-# See https://bogomips.org/unicorn/Unicorn/Configurator.html for complete
+# See https://yhbt.net/unicorn/Unicorn/Configurator.html for complete
 # documentation.
 
 # Use at least one worker per core if you're on a dedicated server,
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 8ef23bc..dfe3a63 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -487,7 +487,7 @@ static void set_url_scheme(VALUE env, VALUE *server_port)
      * and X-Forwarded-Proto handling from this parser?  We've had it
      * forever and nobody has said anything against it, either.
      * Anyways, please send comments to our public mailing list:
-     * unicorn-public@bogomips.org (no HTML mail, no subscription necessary)
+     * unicorn-public@yhbt.net (no HTML mail, no subscription necessary)
      */
     scheme = rb_hash_aref(env, g_http_x_forwarded_ssl);
     if (!NIL_P(scheme) && STR_CSTR_EQ(scheme, "on")) {
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index dd5dff4..d5991fe 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -96,7 +96,7 @@ def self.builder(ru, op)
 
   # returns an array of strings representing TCP listen socket addresses
   # and Unix domain socket paths.  This is useful for use with
-  # Raindrops::Middleware under Linux: https://bogomips.org/raindrops/
+  # Raindrops::Middleware under Linux: https://yhbt.net/raindrops/
   def self.listener_names
     Unicorn::HttpServer::LISTENERS.map do |io|
       Unicorn::SocketHelper.sock_name(io)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index e8b76f5..c3a4f2d 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -3,11 +3,11 @@
 
 # Implements a simple DSL for configuring a unicorn server.
 #
-# See https://bogomips.org/unicorn/examples/unicorn.conf.rb and
-# https://bogomips.org/unicorn/examples/unicorn.conf.minimal.rb
+# See https://yhbt.net/unicorn/examples/unicorn.conf.rb and
+# https://yhbt.net/unicorn/examples/unicorn.conf.minimal.rb
 # example configuration files.  An example config file for use with
 # nginx is also available at
-# https://bogomips.org/unicorn/examples/nginx.conf
+# https://yhbt.net/unicorn/examples/nginx.conf
 #
 # See the link:/TUNING.html document for more information on tuning unicorn.
 class Unicorn::Configurator
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 5334fa0..a52931a 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -6,7 +6,7 @@
 # forked worker children.
 #
 # Users do not need to know the internals of this class, but reading the
-# {source}[https://bogomips.org/unicorn.git/tree/lib/unicorn/http_server.rb]
+# {source}[https://yhbt.net/unicorn.git/tree/lib/unicorn/http_server.rb]
 # is education for programmers wishing to learn how unicorn works.
 # See Unicorn::Configurator for information on how to configure unicorn.
 class Unicorn::HttpServer
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index c4741a0..3b2f488 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -43,8 +43,8 @@
 #     use Unicorn::OobGC, 2, %r{\A/(?:expensive/foo|more_expensive/foo)}
 #
 # Feedback from users of early implementations of this module:
-# * https://bogomips.org/unicorn-public/0BFC98E9-072B-47EE-9A70-05478C20141B@lukemelia.com/
-# * https://bogomips.org/unicorn-public/AANLkTilUbgdyDv9W1bi-s_W6kq9sOhWfmuYkKLoKGOLj@mail.gmail.com/
+# * https://yhbt.net/unicorn-public/0BFC98E9-072B-47EE-9A70-05478C20141B@lukemelia.com/
+# * https://yhbt.net/unicorn-public/AANLkTilUbgdyDv9W1bi-s_W6kq9sOhWfmuYkKLoKGOLj@mail.gmail.com/
 
 module Unicorn::OobGC
 
diff --git a/unicorn.gemspec b/unicorn.gemspec
index ceea831..a189f8d 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -15,14 +15,14 @@
   s.authors = ['unicorn hackers']
   s.summary = 'Rack HTTP server for fast clients and Unix'
   s.description = File.read('README').split("\n\n")[1]
-  s.email = %q{unicorn-public@bogomips.org}
+  s.email = %q{unicorn-public@yhbt.net}
   s.executables = %w(unicorn unicorn_rails)
   s.extensions = %w(ext/unicorn_http/extconf.rb)
   s.extra_rdoc_files = IO.readlines('.document').map!(&:chomp!).keep_if do |f|
     File.exist?(f)
   end
   s.files = manifest
-  s.homepage = 'https://bogomips.org/unicorn/'
+  s.homepage = 'https://yhbt.net/unicorn/'
   s.test_files = test_files
 
   # technically we need ">= 1.9.3", too, but avoid the array here since

^ permalink raw reply related	[relevance 2%]

* [PATCH] doc: s/bogomips.org/yhbt.net/g
@ 2020-01-14  7:46  4% Eric Wong
  2020-01-14  7:46  2% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2020-01-14  7:46 UTC (permalink / raw)
  To: unicorn-public

bogomips.org is due to expire, soon, and I'm not willing to pay
extortionist fees to Ethos Capital/PIR/ICANN to keep a .org.  So
it's at yhbt.net, for now, but it will change again to
whatever's affordable...  Identity is overrated.

Tor users can use .onions and kick ICANN to the curb:

	torsocks w3m http://unicorn.ou63pmih66umazou.onion/
	torsocks git clone http://ou63pmih66umazou.onion/unicorn.git/
	torsocks w3m http://ou63pmih66umazou.onion/unicorn-public/

While we're at it, `s/news.gmane.org/news.gmane.io/g', too.
(but I suspect that'll need to be resynched since our mail
"List-Id:" header is changing).
---
 .olddoc.yml                      | 19 ++++++++++++-------
 Documentation/unicorn.1          |  8 ++++----
 Documentation/unicorn_rails.1    |  8 ++++----
 FAQ                              |  2 +-
 GNUmakefile                      |  4 ++--
 HACKING                          |  2 +-
 ISSUES                           | 24 ++++++++++++------------
 KNOWN_ISSUES                     |  4 ++--
 Links                            | 10 +++++-----
 README                           | 12 ++++++------
 SIGNALS                          |  2 +-
 Sandbox                          |  4 ++--
 archive/slrnpull.conf            |  2 +-
 examples/big_app_gc.rb           |  2 +-
 examples/logrotate.conf          |  4 ++--
 examples/nginx.conf              |  2 +-
 examples/unicorn.conf.minimal.rb |  4 ++--
 examples/unicorn.conf.rb         |  4 ++--
 ext/unicorn_http/unicorn_http.rl |  2 +-
 lib/unicorn.rb                   |  2 +-
 lib/unicorn/configurator.rb      |  6 +++---
 lib/unicorn/http_server.rb       |  2 +-
 lib/unicorn/oob_gc.rb            |  4 ++--
 unicorn.gemspec                  |  4 ++--
 24 files changed, 71 insertions(+), 66 deletions(-)

diff --git a/.olddoc.yml b/.olddoc.yml
index d2d340f..0609bdb 100644
--- a/.olddoc.yml
+++ b/.olddoc.yml
@@ -2,4 +2,5 @@
-cgit_url: https://bogomips.org/unicorn.git
-git_url: https://bogomips.org/unicorn.git
-rdoc_url: https://bogomips.org/unicorn/
-ml_url: https://bogomips.org/unicorn-public/
+cgit_url: https://yhbt.net/unicorn.git
+rdoc_url: https://yhbt.net/unicorn/
+ml_url:
+- https://yhbt.net/unicorn-public/
+- http://ou63pmih66umazou.onion/unicorn-public/
@@ -14 +15 @@ noindex:
-public_email: unicorn-public@bogomips.org
+public_email: unicorn-public@yhbt.net
@@ -16,2 +17,6 @@ nntp_url:
-  - nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
-  - nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+- nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
+- nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
+- nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
+source_code:
+- git clone https://yhbt.net/unicorn.git
+- torsocks git clone http://ou63pmih66umazou.onion/unicorn.git
diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1
index 3f8cb96..d76d40f 100644
--- a/Documentation/unicorn.1
+++ b/Documentation/unicorn.1
@@ -157 +157 @@ TTOU \- decrement the number of worker processes by one
-See the SIGNALS (https://bogomips.org/unicorn/SIGNALS.html) document for
+See the SIGNALS (https://yhbt.net/unicorn/SIGNALS.html) document for
@@ -207 +207 @@ the unicorn config file.
-.UR https://bogomips.org/unicorn/Unicorn/Configurator.html
+.UR https://yhbt.net/unicorn/Unicorn/Configurator.html
@@ -211 +211 @@ unicorn RDoc
-.UR https://bogomips.org/unicorn/
+.UR https://yhbt.net/unicorn/
@@ -222 +222 @@ Rackup HowTo
-The Unicorn Community <unicorn-public@bogomips.org>.
+The Unicorn Community <unicorn-public@yhbt.net>.
diff --git a/Documentation/unicorn_rails.1 b/Documentation/unicorn_rails.1
index 71c80be..fec0a2a 100644
--- a/Documentation/unicorn_rails.1
+++ b/Documentation/unicorn_rails.1
@@ -183 +183 @@ TTOU \- decrement the number of worker processes by one
-See the SIGNALS (https://bogomips.org/unicorn/SIGNALS.html) document for
+See the SIGNALS (https://yhbt.net/unicorn/SIGNALS.html) document for
@@ -192 +192 @@ unicorn(1)
-.UR https://bogomips.org/unicorn/Unicorn/Configurator.html
+.UR https://yhbt.net/unicorn/Unicorn/Configurator.html
@@ -196 +196 @@ unicorn RDoc
-.UR https://bogomips.org/unicorn/
+.UR https://yhbt.net/unicorn/
@@ -207 +207 @@ Rackup HowTo
-The Unicorn Community <unicorn-public@bogomips.org>.
+The Unicorn Community <unicorn-public@yhbt.net>.
diff --git a/FAQ b/FAQ
index 4ae2034..018ca92 100644
--- a/FAQ
+++ b/FAQ
@@ -10 +10 @@ overhead from bad requests.
-Ref: https://bogomips.org/unicorn-public/FC91211E-FD32-432C-92FC-0318714C2170@zendesk.com/
+Ref: https://yhbt.net/unicorn-public/FC91211E-FD32-432C-92FC-0318714C2170@zendesk.com/
diff --git a/GNUmakefile b/GNUmakefile
index 94c46ee..eac3473 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -196 +196 @@ doc: .document $(ext)/unicorn_http.c man html .olddoc.yml $(PLACEHOLDERS)
-# publishes docs to https://bogomips.org/unicorn/
+# publishes docs to https://yhbt.net/unicorn/
@@ -202 +202 @@ publish_doc:
-	$(RSYNC) -av doc/ bogomips.org:/srv/bogomips/unicorn/
+	$(RSYNC) -av doc/ yhbt.net:/srv/yhbt/unicorn/
diff --git a/HACKING b/HACKING
index be1bb85..976bf92 100644
--- a/HACKING
+++ b/HACKING
@@ -60 +60 @@ When referencing mailing list posts, use
-<tt>https://bogomips.org/unicorn-public/$MESSAGE_ID/</tt> if possible
+<tt>https://yhbt.net/unicorn-public/$MESSAGE_ID/</tt> if possible
diff --git a/ISSUES b/ISSUES
index 473da2f..d11dc56 100644
--- a/ISSUES
+++ b/ISSUES
@@ -3 +3 @@
-mailto:unicorn-public@bogomips.org is the best place to report bugs,
+mailto:unicorn-public@yhbt.net is the best place to report bugs,
@@ -5,2 +5,2 @@ submit patches and/or obtain support after you have searched the
-{email archives}[https://bogomips.org/unicorn-public/] and
-{documentation}[https://bogomips.org/unicorn/].
+{email archives}[https://yhbt.net/unicorn-public/] and
+{documentation}[https://yhbt.net/unicorn/].
@@ -15,2 +15,2 @@ submit patches and/or obtain support after you have searched the
-* The email submission port (587) is enabled on the bogomips.org MX:
-  https://bogomips.org/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
+* The email submission port (587) is enabled on the yhbt.net MX:
+  https://yhbt.net/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
@@ -20 +20 @@ can interoperate with any bug tracker which can Cc: us plain-text to
-mailto:unicorn-public@bogomips.org   This includes the Debian BTS
+mailto:unicorn-public@yhbt.net   This includes the Debian BTS
@@ -76,2 +76,2 @@ document distributed with git) on guidelines for patch submission.
-* public: mailto:unicorn-public@bogomips.org
-* nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+* public: mailto:unicorn-public@yhbt.net
+* nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
@@ -79 +79 @@ document distributed with git) on guidelines for patch submission.
-* https://bogomips.org/unicorn-public/
+* https://yhbt.net/unicorn-public/
@@ -87 +87 @@ You can follow along via NNTP (read-only):
-	nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+	nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
@@ -91 +91 @@ Or Atom feeds:
-	https://bogomips.org/unicorn-public/new.atom
+	https://yhbt.net/unicorn-public/new.atom
@@ -94 +94 @@ Or Atom feeds:
-	The HTML archives at https://bogomips.org/unicorn-public/
+	The HTML archives at https://yhbt.net/unicorn-public/
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index ebd4822..0017f20 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -12 +12 @@ acceptable solution.  Those issues are documented here.
-  https://bogomips.org/unicorn-public/CAHBuKRj09FdxAgzsefJWotexw-7JYZGJMtgUp_dhjPz9VbKD6Q@mail.gmail.com/
+  https://yhbt.net/unicorn-public/CAHBuKRj09FdxAgzsefJWotexw-7JYZGJMtgUp_dhjPz9VbKD6Q@mail.gmail.com/
@@ -63 +63 @@ acceptable solution.  Those issues are documented here.
-  ref: https://bogomips.org/unicorn-public/20091014221552.GA30624@dcvr.yhbt.net/
+  ref: https://yhbt.net/unicorn-public/20091014221552.GA30624@dcvr.yhbt.net/
diff --git a/Links b/Links
index 10551a6..f81142d 100644
--- a/Links
+++ b/Links
@@ -5 +5 @@ listed below.  If you have any links to add/change/remove, please tell us at
-mailto:unicorn-public@bogomips.org!
+mailto:unicorn-public@yhbt.net!
@@ -26 +26 @@ or services behind them.
-* {raindrops}[https://bogomips.org/raindrops/] - real-time stats for
+* {raindrops}[https://yhbt.net/raindrops/] - real-time stats for
@@ -29 +29 @@ or services behind them.
-* {UnXF}[https://bogomips.org/unxf/]  Un-X-Forward* the Rack environment,
+* {UnXF}[https://yhbt.net/unxf/]  Un-X-Forward* the Rack environment,
@@ -55 +55 @@ or services behind them.
-  https://bogomips.org/mongrel-devel/
+  https://yhbt.net/mongrel-devel/
@@ -57 +57 @@ or services behind them.
-* {david}[https://bogomips.org/david.git] - a tool to explain why you need
+* {david}[https://yhbt.net/david.git] - a tool to explain why you need
diff --git a/README b/README
index 89467fc..0e95f48 100644
--- a/README
+++ b/README
@@ -83 +83 @@ You can get the latest source via git from the following locations
-  https://bogomips.org/unicorn.git
+  https://yhbt.net/unicorn.git
@@ -88 +88 @@ You may browse the code from the web:
-* https://bogomips.org/unicorn.git
+* https://yhbt.net/unicorn.git
@@ -136 +136 @@ requests) go to the mailing list/newsgroup.  See the ISSUES document for
-information on the {mailing list}[mailto:unicorn-public@bogomips.org].
+information on the {mailing list}[mailto:unicorn-public@yhbt.net].
@@ -138 +138 @@ information on the {mailing list}[mailto:unicorn-public@bogomips.org].
-The mailing list is archived at https://bogomips.org/unicorn-public/
+The mailing list is archived at https://yhbt.net/unicorn-public/
@@ -141 +141 @@ nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn and
-nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+nntp://news.gmane.io/gmane.comp.lang.ruby.unicorn.general
@@ -144 +144 @@ For the latest on unicorn releases, you may also finger us at
-unicorn@bogomips.org or check our NEWS page (and subscribe to our Atom
+unicorn@yhbt.net or check our NEWS page (and subscribe to our Atom
diff --git a/SIGNALS b/SIGNALS
index 1af851d..7321f2b 100644
--- a/SIGNALS
+++ b/SIGNALS
@@ -11 +11 @@ One example init script is distributed with unicorn:
-https://bogomips.org/unicorn/examples/init.sh
+https://yhbt.net/unicorn/examples/init.sh
diff --git a/Sandbox b/Sandbox
index d0f915e..651e5cd 100644
--- a/Sandbox
+++ b/Sandbox
@@ -37 +37 @@ unicorn_rails") to start unicorn with the correct environment variables
-ref: https://bogomips.org/unicorn-public/9ECF07C4-5216-47BE-961D-AFC0F0C82060@internetfamo.us/
+ref: https://yhbt.net/unicorn-public/9ECF07C4-5216-47BE-961D-AFC0F0C82060@internetfamo.us/
@@ -47 +47 @@ ref:
-https://bogomips.org/unicorn-public/8FC34B23-5994-41CC-B5AF-7198EF06909E@tramchase.com/
+https://yhbt.net/unicorn-public/8FC34B23-5994-41CC-B5AF-7198EF06909E@tramchase.com/
diff --git a/archive/slrnpull.conf b/archive/slrnpull.conf
index fcfcafe..fd04f97 100644
--- a/archive/slrnpull.conf
+++ b/archive/slrnpull.conf
@@ -4 +4 @@ gmane.comp.lang.ruby.unicorn.general 1000000000 1000000000 0
-# usage: slrnpull -d $PWD -h news.gmane.org --no-post
+# usage: slrnpull -d $PWD -h news.gmane.io --no-post
diff --git a/examples/big_app_gc.rb b/examples/big_app_gc.rb
index 9d05719..c1bae10 100644
--- a/examples/big_app_gc.rb
+++ b/examples/big_app_gc.rb
@@ -1 +1 @@
-# see {Unicorn::OobGC}[https://bogomips.org/unicorn/Unicorn/OobGC.html]
+# see {Unicorn::OobGC}[https://yhbt.net/unicorn/Unicorn/OobGC.html]
diff --git a/examples/logrotate.conf b/examples/logrotate.conf
index 77a01b5..c3aa40d 100644
--- a/examples/logrotate.conf
+++ b/examples/logrotate.conf
@@ -8 +8 @@
-#    https://bogomips.org/unicorn-public/?q=logrotate
+#    https://yhbt.net/unicorn-public/?q=logrotate
@@ -36 +36 @@
-		# Mail us at unicorn-public@bogomips.org
+		# Mail us at unicorn-public@yhbt.net
diff --git a/examples/nginx.conf b/examples/nginx.conf
index b6b69c1..c5026f9 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -116 +116 @@ http {
-    # https://bogomips.org/unicorn.git/tree/examples/nginx.conf?id=v3.3.1#n127
+    # https://yhbt.net/unicorn.git/tree/examples/nginx.conf?id=v3.3.1#n127
diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb
index 2d1bf0a..46fd634 100644
--- a/examples/unicorn.conf.minimal.rb
+++ b/examples/unicorn.conf.minimal.rb
@@ -4 +4 @@
-# See https://bogomips.org/unicorn/Unicorn/Configurator.html for complete
+# See https://yhbt.net/unicorn/Unicorn/Configurator.html for complete
@@ -6 +6 @@
-# See also https://bogomips.org/unicorn/examples/unicorn.conf.rb for
+# See also https://yhbt.net/unicorn/examples/unicorn.conf.rb for
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index d2897ef..d90bdc4 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -5 +5 @@
-# https://bogomips.org/unicorn/examples/unicorn.conf.minimal.rb
+# https://yhbt.net/unicorn/examples/unicorn.conf.minimal.rb
@@ -8 +8 @@
-# See https://bogomips.org/unicorn/Unicorn/Configurator.html for complete
+# See https://yhbt.net/unicorn/Unicorn/Configurator.html for complete
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 8ef23bc..dfe3a63 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -490 +490 @@ static void set_url_scheme(VALUE env, VALUE *server_port)
-     * unicorn-public@bogomips.org (no HTML mail, no subscription necessary)
+     * unicorn-public@yhbt.net (no HTML mail, no subscription necessary)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index dd5dff4..d5991fe 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -99 +99 @@ def self.builder(ru, op)
-  # Raindrops::Middleware under Linux: https://bogomips.org/raindrops/
+  # Raindrops::Middleware under Linux: https://yhbt.net/raindrops/
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index e8b76f5..c3a4f2d 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -6,2 +6,2 @@
-# See https://bogomips.org/unicorn/examples/unicorn.conf.rb and
-# https://bogomips.org/unicorn/examples/unicorn.conf.minimal.rb
+# See https://yhbt.net/unicorn/examples/unicorn.conf.rb and
+# https://yhbt.net/unicorn/examples/unicorn.conf.minimal.rb
@@ -10 +10 @@
-# https://bogomips.org/unicorn/examples/nginx.conf
+# https://yhbt.net/unicorn/examples/nginx.conf
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 5334fa0..a52931a 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -9 +9 @@
-# {source}[https://bogomips.org/unicorn.git/tree/lib/unicorn/http_server.rb]
+# {source}[https://yhbt.net/unicorn.git/tree/lib/unicorn/http_server.rb]
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index c4741a0..3b2f488 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -46,2 +46,2 @@
-# * https://bogomips.org/unicorn-public/0BFC98E9-072B-47EE-9A70-05478C20141B@lukemelia.com/
-# * https://bogomips.org/unicorn-public/AANLkTilUbgdyDv9W1bi-s_W6kq9sOhWfmuYkKLoKGOLj@mail.gmail.com/
+# * https://yhbt.net/unicorn-public/0BFC98E9-072B-47EE-9A70-05478C20141B@lukemelia.com/
+# * https://yhbt.net/unicorn-public/AANLkTilUbgdyDv9W1bi-s_W6kq9sOhWfmuYkKLoKGOLj@mail.gmail.com/
diff --git a/unicorn.gemspec b/unicorn.gemspec
index ceea831..a189f8d 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -18 +18 @@
-  s.email = %q{unicorn-public@bogomips.org}
+  s.email = %q{unicorn-public@yhbt.net}
@@ -25 +25 @@
-  s.homepage = 'https://bogomips.org/unicorn/'
+  s.homepage = 'https://yhbt.net/unicorn/'

^ permalink raw reply related	[relevance 4%]

* Re: Traffic priority with Unicorn
  2019-12-17  5:12  3% ` Eric Wong
@ 2019-12-18 22:06  0%   ` Bertrand Paquet
  0 siblings, 0 replies; 200+ results
From: Bertrand Paquet @ 2019-12-18 22:06 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On Tue, 17 Dec 2019 at 06:12, Eric Wong <bofh@yhbt.net> wrote:
>
> Bertrand Paquet <bertrand.paquet@doctolib.com> wrote:
> > Hello,
> >
> > I would like to introduce some traffic priority in Unicorn. The goal
> > is to keep critical endpoints online even if the application is
> > slowing down a lot.
> >
> > The idea is to classify the request at nginx level (by vhost, http
> > path, header or whatever), and send the queries to two different
> > unicorn sockets (opened by the same unicorn instance): one for high
> > priority request, one for low priority request.
> > I need to do some small modifications [1] in the unicorn worker loop
> > to process high priority requests first. It seems to work:
> > - I launch a first apache bench toward the low priority port
> > - I launch a second apache bench toward the high priority port
> > - Unicorn handles the queries only for that one, and stop answering to
> > the low priority traffic
>
> > [1] https://github.com/bpaquet/unicorn/commit/58d6ba2805d4399f680f97eefff82c407e0ed30f#
>
> Easier to view locally w/o JS/CSS using "git show -W" for context:
>
> $ git remote add bpaquet https://github.com/bpaquet/unicorn
> $ git fetch bpaquet
> $ git show -W 58d6ba28
>   <snip>
> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 5334fa0c..976b728e 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -676,53 +676,56 @@ def reopen_worker_logs(worker_nr)
>    # runs inside each forked worker, this sits around and waits
>    # for connections and doesn't die until the parent dies (or is
>    # given a INT, QUIT, or TERM signal)
>    def worker_loop(worker)
>      ppid = @master_pid
>      readers = init_worker_process(worker)
>      nr = 0 # this becomes negative if we need to reopen logs
>
>      # this only works immediately if the master sent us the signal
>      # (which is the normal case)
>      trap(:USR1) { nr = -65536 }
>
>      ready = readers.dup
> +    high_priority_reader = readers.first
> +    last_processed_is_high_priority = false
>      @after_worker_ready.call(self, worker)
>
>      begin
>        nr < 0 and reopen_worker_logs(worker.nr)
>        nr = 0
>        worker.tick = time_now.to_i
>        tmp = ready.dup
>        while sock = tmp.shift
>          # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
>          # but that will return false
>          if client = sock.kgio_tryaccept
> +          last_processed_is_high_priority = sock == high_priority_reader
>            process_client(client)
>            nr += 1
>            worker.tick = time_now.to_i
>          end
> -        break if nr < 0
> +        break if nr < 0 || last_processed_is_high_priority
>        end
>
>        # make the following bet: if we accepted clients this round,
>        # we're probably reasonably busy, so avoid calling select()
>        # and do a speculative non-blocking accept() on ready listeners
>        # before we sleep again in select().
> -      unless nr == 0
> +      unless nr == 0 || !last_processed_is_high_priority
>          tmp = ready.dup
>          redo
>        end
>
>        ppid == Process.ppid or return
>
>        # timeout used so we can detect parent death:
>        worker.tick = time_now.to_i
>        ret = IO.select(readers, nil, nil, @timeout) and ready = ret[0]
>      rescue => e
>        redo if nr < 0 && readers[0]
>        Unicorn.log_error(@logger, "listen loop error", e) if readers[0]
>      end while readers[0]
>    end
>
>    # delivers a signal to a worker and fails gracefully if the worker
>    # is no longer running.
>
> > The tradeoff are
> > - No more "bet"[2] on low priority traffic. This is probably slowing
> > down a little bit the low  priority traffic.
>
> Yeah, but low priority is low priority, so it's fine to slow
> them down, right? :>
>
> > - This approach is only low / high. Not sure if I can extend it for 3
> > (or more) level of priority without a non negligible performance
> > impact (because of the "bet" above).
>
> I don't think it makes sense to have more than two levels of
> priority (zero, one, two, infinity rule?)
> <https://en.wikipedia.org/wiki/Zero_One_Infinity>
>
> > Do you think this approach is correct?
>
> readers order isn't guaranteed, especially when inheriting
> sockets from systemd or similar launchers.

Interesting point.

>
> I think some sort order could be defined via listen option...
>
> I'm not sure if inheriting multiple sockets from systemd or
> similar launchers using LISTEN_FDS env can guarantee ordering
> (or IO.select in Ruby, for that matter).
>
> It seems OK otherwise, I think...  Have you tested in real world?

Not yet. But I will probably test it soon.

>
> > Do you have any better idea to have some traffic prioritization?
> > (Another idea is to have dedicated workers for each priority class.
> > This approach has other downsides, I would like to avoid it).
> > Is it something we can  introduce in Unicorn (not as default
> > behaviour, but as a configuration option)?
>
> If you're willing to drop some low-priority requests, using a
> small listen :backlog value for a low-priority listen may work.
>
> I'm hesitant to put extra code in worker_loop method since
> it can slow down current users who don't need the feature.
>
> Instead, perhaps try replacing the worker_loop method entirely
> (similar to how oob_gc.rb wraps process_client) so users who
> don't enable the feature won't be penalized with extra code.
> Users who opt into the feature can get an entirely different
> method.

Ok I will try this approach. I'm a little bit annoyed by the code duplication.

>
> > Thx for any opinion.
>
> The best option would be to never get yourself in a situation
> where you're never overloaded by making everything fast :>
> Anything else seems pretty ugly...

On a system which handle 10k QPS, it's really difficult to never have
an issue somewhere :)

Thx

Bertrand

^ permalink raw reply	[relevance 0%]

* Re: Traffic priority with Unicorn
  @ 2019-12-17  5:12  3% ` Eric Wong
  2019-12-18 22:06  0%   ` Bertrand Paquet
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2019-12-17  5:12 UTC (permalink / raw)
  To: Bertrand Paquet; +Cc: unicorn-public

Bertrand Paquet <bertrand.paquet@doctolib.com> wrote:
> Hello,
> 
> I would like to introduce some traffic priority in Unicorn. The goal
> is to keep critical endpoints online even if the application is
> slowing down a lot.
> 
> The idea is to classify the request at nginx level (by vhost, http
> path, header or whatever), and send the queries to two different
> unicorn sockets (opened by the same unicorn instance): one for high
> priority request, one for low priority request.
> I need to do some small modifications [1] in the unicorn worker loop
> to process high priority requests first. It seems to work:
> - I launch a first apache bench toward the low priority port
> - I launch a second apache bench toward the high priority port
> - Unicorn handles the queries only for that one, and stop answering to
> the low priority traffic

> [1] https://github.com/bpaquet/unicorn/commit/58d6ba2805d4399f680f97eefff82c407e0ed30f#

Easier to view locally w/o JS/CSS using "git show -W" for context:

$ git remote add bpaquet https://github.com/bpaquet/unicorn
$ git fetch bpaquet
$ git show -W 58d6ba28
  <snip>
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 5334fa0c..976b728e 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -676,53 +676,56 @@ def reopen_worker_logs(worker_nr)
   # runs inside each forked worker, this sits around and waits
   # for connections and doesn't die until the parent dies (or is
   # given a INT, QUIT, or TERM signal)
   def worker_loop(worker)
     ppid = @master_pid
     readers = init_worker_process(worker)
     nr = 0 # this becomes negative if we need to reopen logs
 
     # this only works immediately if the master sent us the signal
     # (which is the normal case)
     trap(:USR1) { nr = -65536 }
 
     ready = readers.dup
+    high_priority_reader = readers.first
+    last_processed_is_high_priority = false
     @after_worker_ready.call(self, worker)
 
     begin
       nr < 0 and reopen_worker_logs(worker.nr)
       nr = 0
       worker.tick = time_now.to_i
       tmp = ready.dup
       while sock = tmp.shift
         # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
         # but that will return false
         if client = sock.kgio_tryaccept
+          last_processed_is_high_priority = sock == high_priority_reader
           process_client(client)
           nr += 1
           worker.tick = time_now.to_i
         end
-        break if nr < 0
+        break if nr < 0 || last_processed_is_high_priority
       end
 
       # make the following bet: if we accepted clients this round,
       # we're probably reasonably busy, so avoid calling select()
       # and do a speculative non-blocking accept() on ready listeners
       # before we sleep again in select().
-      unless nr == 0
+      unless nr == 0 || !last_processed_is_high_priority
         tmp = ready.dup
         redo
       end
 
       ppid == Process.ppid or return
 
       # timeout used so we can detect parent death:
       worker.tick = time_now.to_i
       ret = IO.select(readers, nil, nil, @timeout) and ready = ret[0]
     rescue => e
       redo if nr < 0 && readers[0]
       Unicorn.log_error(@logger, "listen loop error", e) if readers[0]
     end while readers[0]
   end
 
   # delivers a signal to a worker and fails gracefully if the worker
   # is no longer running.

> The tradeoff are
> - No more "bet"[2] on low priority traffic. This is probably slowing
> down a little bit the low  priority traffic.

Yeah, but low priority is low priority, so it's fine to slow
them down, right? :>

> - This approach is only low / high. Not sure if I can extend it for 3
> (or more) level of priority without a non negligible performance
> impact (because of the "bet" above).

I don't think it makes sense to have more than two levels of
priority (zero, one, two, infinity rule?)
<https://en.wikipedia.org/wiki/Zero_One_Infinity>

> Do you think this approach is correct?

readers order isn't guaranteed, especially when inheriting
sockets from systemd or similar launchers.

I think some sort order could be defined via listen option...

I'm not sure if inheriting multiple sockets from systemd or
similar launchers using LISTEN_FDS env can guarantee ordering
(or IO.select in Ruby, for that matter).

It seems OK otherwise, I think...  Have you tested in real world?

> Do you have any better idea to have some traffic prioritization?
> (Another idea is to have dedicated workers for each priority class.
> This approach has other downsides, I would like to avoid it).
> Is it something we can  introduce in Unicorn (not as default
> behaviour, but as a configuration option)?

If you're willing to drop some low-priority requests, using a
small listen :backlog value for a low-priority listen may work.

I'm hesitant to put extra code in worker_loop method since
it can slow down current users who don't need the feature.

Instead, perhaps try replacing the worker_loop method entirely
(similar to how oob_gc.rb wraps process_client) so users who
don't enable the feature won't be penalized with extra code.
Users who opt into the feature can get an entirely different
method.

> Thx for any opinion.

The best option would be to never get yourself in a situation
where you're never overloaded by making everything fast :>
Anything else seems pretty ugly...

^ permalink raw reply related	[relevance 3%]

* [PATCH 3/3] http: memoize more common fields
  @ 2019-07-04 22:01  5% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2019-07-04 22:01 UTC (permalink / raw)
  To: unicorn-public

"DNT" is common, nowadays.
"Forwarded" is... *shrug* It's an RFC, at least.
"Origin" is a CORS, and something I've seen.

I've seen "Upgrade-Insecure-Requests", "X-Forwarded-Host",
"X-Request-ID", and "X-Requested-With" in the wild, too;
so add those.
---
 ext/unicorn_http/common_fields.gperf | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/ext/unicorn_http/common_fields.gperf b/ext/unicorn_http/common_fields.gperf
index 179afe5..a10f4cd 100644
--- a/ext/unicorn_http/common_fields.gperf
+++ b/ext/unicorn_http/common_fields.gperf
@@ -28,7 +28,11 @@ CONTENT_LENGTH
 CONTENT_TYPE
 COOKIE
 DATE
+# Do Not Track
+DNT
 EXPECT
+# RFC 7239 (does anybody use Forwarded:?)
+FORWARDED
 FROM
 HOST
 IF_MATCH
@@ -39,6 +43,7 @@ IF_UNMODIFIED_SINCE
 # Firefox sends Keep-Alive (or maybe only old versions?)
 KEEP_ALIVE
 MAX_FORWARDS
+ORIGIN
 PRAGMA
 PROXY_AUTHORIZATION
 RANGE
@@ -47,10 +52,15 @@ TE
 TRAILER
 TRANSFER_ENCODING
 UPGRADE
+UPGRADE_INSECURE_REQUESTS
 USER_AGENT
 VIA
 # common proxies set some of these X- headers
 X_FORWARDED_FOR
+X_FORWARDED_HOST
 X_FORWARDED_PROTO
 X_REAL_IP
+X_REQUEST_ID
+# XMLHttpRequest
+X_REQUESTED_WITH
 WARNING
-- 
EW


^ permalink raw reply related	[relevance 5%]

* [PATCH] Links: add historical mongrel dev list mirror
@ 2018-12-28 21:45  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-12-28 21:45 UTC (permalink / raw)
  To: unicorn-public

unicorn has no future, but at least some of our past is archived.

This was taken from nntp://news.gmane.org/gmane.comp.lang.ruby.mongrel.devel
as the HTTP parts of gmane are dead.  Unfortunately, Gmane only
started archiving mongrel-development@rubyforge.org in 2008 and
discussions prior to that may be lost.  If anybody has mbox, Maildir
or MH archives from those days, I'd be glad to import them.
---
 Links | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Links b/Links
index baba9c7..10551a6 100644
--- a/Links
+++ b/Links
@@ -50,7 +50,9 @@ or services behind them.
 === Prior Work
 
 * {Mongrel}[https://rubygems.org/gems/mongrel] - the awesome webserver
-  unicorn is based on
+  unicorn is based on.  A historical archive of the mongrel dev list
+  featuring early discussions of unicorn is available at:
+  https://bogomips.org/mongrel-devel/
 
 * {david}[https://bogomips.org/david.git] - a tool to explain why you need
   nginx in front of unicorn
-- 
EW


^ permalink raw reply related	[relevance 5%]

* [ANN] unicorn 5.5.0.pre1 - Rack HTTP server for fast clients and Unix
@ 2018-12-20 22:28  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-12-20 22:28 UTC (permalink / raw)
  To: ruby-talk, unicorn-public; +Cc: Jeremy Evans

unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

Disclaimer:

Due to its ability to tolerate crashes and isolate clients, unicorn
is unfortunately known to prolong the existence of bugs in applications
and libraries which run on top of it.

* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone https://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

This is a pre-release RubyGem intended for testing.

Changes:

unicorn 5.5.0.pre1

Jeremy Evans contributed the "default_middleware" configuration option:

  https://bogomips.org/unicorn-public/20180913192055.GD48926@jeremyevans.local/

Jeremy also contributed the ability to use separate groups for the process
and log files:

  https://bogomips.org/unicorn-public/20180913192449.GE48926@jeremyevans.local/

There's also a couple of uninteresting minor optimizations and
documentation additions.

Eric Wong (10):
      remove random seed reset atfork
      use IO#wait instead of kgio_wait_readable
      Merge branch '5.4-stable'
      shrink pipes under Linux
      socket_helper: add hint for FreeBSD users for accf_http(9)
      tests: ensure -N/--no-default-middleware not supported in config.ru
      doc: update more URLs to use HTTPS and avoid redirects
      deduplicate strings VM-wide in Ruby 2.5+
      doc/ISSUES: add links to git clone-able mail archives of our dependencies
      README: minor updates and additional disclaimer

Jeremy Evans (2):
      Make Worker#user support different process primary group and log file group
      Support default_middleware configuration option
-- 

^ permalink raw reply	[relevance 5%]

* [PATCH] doc/ISSUES: add links to git clone-able mail archives of our dependencies
@ 2018-12-13  1:07  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-12-13  1:07 UTC (permalink / raw)
  To: unicorn-public

Archives are crucial to preserving history and knowledge in Free
Software projects, so promote them for projects we depend on.

Naq lrf, gur nepuviny fbsgjner qrirybcrq sbe nepuvivat gur
havpbea znvyvat yvfg unf ybat fhecnffrq gur hfrshyarff bs
havpbea vgfrys :C
---
 ISSUES | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/ISSUES b/ISSUES
index 610f011..c04ff54 100644
--- a/ISSUES
+++ b/ISSUES
@@ -39,6 +39,7 @@ https://bugs.ruby-lang.org/ and discuss+fix them on the ruby-core
 list at mailto:ruby-core@ruby-lang.org
 Subscription to post is required to ruby-core, unfortunately:
 mailto:ruby-core-request@ruby-lang.org?subject=subscribe
+Unofficial archives are available at: https://public-inbox.org/ruby-core/
 
 For uncommon bugs in Rack, we may forward bugs to
 mailto:rack-devel@googlegroups.com and discuss there.
@@ -46,6 +47,7 @@ Subscription (without any web UI or Google account) is possible via:
 mailto:rack-devel+subscribe@googlegroups.com
 Note: not everyone can use the proprietary bug tracker used by Rack,
 but their mailing list remains operational.
+Unofficial archives are available at: https://public-inbox.org/rack-devel/
 
 Uncommon bugs we encounter in the Linux kernel should be Cc:-ed to the
 Linux kernel mailing list (LKML) at mailto:linux-kernel@vger.kernel.org
@@ -54,11 +56,12 @@ and subsystem maintainers such as mailto:netdev@vger.kernel.org
 involved with any problematic commits (including those in the
 Signed-off-by: and other trailer lines).  No subscription is necessary,
 and the our mailing list follows the same conventions as LKML for
-interopability.  There is a kernel.org Bugzilla instance, but it is
-ignored by most developers.
+interopability. Archives are available at https://lore.kernel.org/lkml/
+There is a kernel.org Bugzilla instance, but it is ignored by most.
 
 Likewise for any rare glibc bugs we might encounter, we should Cc:
 mailto:libc-alpha@sourceware.org
+Unofficial archives are available at: https://public-inbox.org/libc-alpha/
 Keep in mind glibc upstream does use Bugzilla for tracking bugs:
 https://sourceware.org/bugzilla/
 

^ permalink raw reply related	[relevance 4%]

* Re: Support default_middleware configurator method
  @ 2018-09-21  0:21  5%           ` Jeremy Evans
  0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2018-09-21  0:21 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 09/19 07:39, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > OK. To implement that, I modified the bin/unicorn file so -N
> > is only respected while parsing ARGV, and not while parsing
> > embedded configuration file options.
> 
> Thanks.
> 
> Unfortunately, -N on the command-line was broken by your patch.
> I fixed configurator.rb ordering (below) to pass t0300
> integration test.
> 
> Also, using a non-config.ru .rb file (TestHandler in
> test/unit/test_server.rb) was broken because of missing
> parentheses.

Sorry about both of those issues and for the delay in responding to
this.  I've included your changes in the following squashed commit,
which also includes an integration test for the default_middleware
configuration option, based on the existing integration test for
the -N option.

Thanks,
Jeremy

From fdbbcc82b838a39abdc43448490eb83d80c4f763 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Thu, 13 Sep 2018 10:48:25 -0700
Subject: [PATCH] Support default_middleware configuration option

This allows for the equivalent of the
-N/--no-default_middleware command line option to be
specified in the configuration file so it doesn't
need to be specified on the command line every time
unicorn is executed.

It explicitly excludes the use of -N/--no-default_middleware
as an embedded configuration option in the rackup file, by
ignoring the options after ARGV is parsed.

In order to allow the configuration method to work, have
the lambda that Unicorn.builder returns accept two arguments.
Technically, only one argument is needed for the HttpServer
instance, but I'm guessing if the lambda accepts a single
argument, we expect that to be a rack application instead
of a lambda that returns a rack application.

The command line option option to disable default middleware
will take precedence over the unicorn configuration file option
if both are present.

For backwards compatibility, if the lambda passed to
HttpServer accepts 0 arguments, then call it without
arguments.
---
 bin/unicorn                         |  4 +++-
 lib/unicorn.rb                      |  8 ++------
 lib/unicorn/configurator.rb         | 11 +++++++++++
 lib/unicorn/http_server.rb          |  8 +++++---
 t/t0301-default-middleware-false.sh | 22 ++++++++++++++++++++++
 5 files changed, 43 insertions(+), 10 deletions(-)
 create mode 100644 t/t0301-default-middleware-false.sh

diff --git a/bin/unicorn b/bin/unicorn
index 3c5e5cb..00c8464 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -6,6 +6,7 @@
 ENV["RACK_ENV"] ||= "development"
 rackup_opts = Unicorn::Configurator::RACKUP
 options = rackup_opts[:options]
+set_no_default_middleware = true
 
 op = OptionParser.new("", 24, '  ') do |opts|
   cmd = File.basename($0)
@@ -60,7 +61,7 @@
 
   opts.on("-N", "--no-default-middleware",
           "do not load middleware implied by RACK_ENV") do |e|
-    rackup_opts[:no_default_middleware] = true
+    rackup_opts[:no_default_middleware] = true if set_no_default_middleware
   end
 
   opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
@@ -110,6 +111,7 @@
   opts.parse! ARGV
 end
 
+set_no_default_middleware = false
 app = Unicorn.builder(ARGV[0] || 'config.ru', op)
 op = nil
 
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b6dae36..5f2134d 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -45,12 +45,8 @@ def self.builder(ru, op)
       abort "rack and Rack::Builder must be available for processing #{ru}"
     end
 
-    # Op is going to get cleared before the returned lambda is called, so
-    # save this value so that it's still there when we need it:
-    no_default_middleware = op[:no_default_middleware]
-
     # always called after config file parsing, may be called after forking
-    lambda do ||
+    lambda do |_, server|
       inner_app = case ru
       when /\.ru$/
         raw = File.read(ru)
@@ -66,7 +62,7 @@ def self.builder(ru, op)
         pp({ :inner_app => inner_app })
       end
 
-      return inner_app if no_default_middleware
+      return inner_app unless server.default_middleware
 
       middleware = { # order matters
         ContentLength: nil,
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index f34d38b..d426edf 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -88,6 +88,9 @@ def reload(merge_defaults = true) #:nodoc:
     RACKUP[:set_listener] and
       set[:listeners] << "#{RACKUP[:host]}:#{RACKUP[:port]}"
 
+    RACKUP[:no_default_middleware] and
+      set[:default_middleware] = false
+
     # unicorn_rails creates dirs here after working_directory is bound
     after_reload.call if after_reload
 
@@ -265,6 +268,14 @@ def worker_processes(nr)
     set_int(:worker_processes, nr, 1)
   end
 
+  # sets whether to add default middleware in the development and
+  # deployment RACK_ENVs.
+  #
+  # default_middleware is only available in unicorn 5.5.0+
+  def default_middleware(bool)
+    set_bool(:default_middleware, bool)
+  end
+
   # sets listeners to the given +addresses+, replacing or augmenting the
   # current set.  This is for the global listener pool shared by all
   # worker processes.  For per-worker listeners, see the after_fork example
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index b2bbddb..62f6171 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -14,7 +14,8 @@ class Unicorn::HttpServer
   attr_accessor :app, :timeout, :worker_processes,
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
-                :orig_app, :config, :ready_pipe, :user
+                :orig_app, :config, :ready_pipe, :user,
+                :default_middleware
   attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
@@ -70,6 +71,7 @@ def initialize(app, options = {})
     @app = app
     @request = Unicorn::HttpRequest.new
     @reexec_pid = 0
+    @default_middleware = true
     options = options.dup
     @ready_pipe = options.delete(:ready_pipe)
     @init_listeners = options[:listeners] ? options[:listeners].dup : []
@@ -784,12 +786,12 @@ def listener_names(listeners = LISTENERS)
   end
 
   def build_app!
-    if app.respond_to?(:arity) && app.arity == 0
+    if app.respond_to?(:arity) && (app.arity == 0 || app.arity == 2)
       if defined?(Gem) && Gem.respond_to?(:refresh)
         logger.info "Refreshing Gem list"
         Gem.refresh
       end
-      self.app = app.call
+      self.app = app.arity == 0 ? app.call : app.call(nil, self)
     end
   end
 
diff --git a/t/t0301-default-middleware-false.sh b/t/t0301-default-middleware-false.sh
new file mode 100644
index 0000000..c505f1d
--- /dev/null
+++ b/t/t0301-default-middleware-false.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+. ./test-lib.sh
+t_plan 3 "test the default_middleware false configuration option"
+
+t_begin "setup and start" && {
+	unicorn_setup
+	echo default_middleware false >> $unicorn_config
+	unicorn -D -c $unicorn_config fails-rack-lint.ru
+	unicorn_wait_start
+}
+
+t_begin "check exit status with Rack::Lint not present" && {
+	test 42 -eq "$(curl -sf -o/dev/null -w'%{http_code}' http://$listen/)"
+}
+
+t_begin "killing succeeds" && {
+	kill $unicorn_pid
+	check_stderr
+}
+
+t_done
+
-- 
2.17.1


^ permalink raw reply related	[relevance 5%]

* Re: Support default_middleware configurator method
  2018-09-14 10:56  0%     ` Eric Wong
@ 2018-09-14 15:03  6%       ` Jeremy Evans
    0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2018-09-14 15:03 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 09/14 10:56, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > On 09/13 10:34, Eric Wong wrote:
> > > Worse case is we only support default_middleware as a config
> > > option (but I prefer not to add more options).  Embedded -N support
> > > is an anti-feature which leads to lock-in.
> > 
> > I don't really care about embedded -N support.  I actually didn't even
> > know embedded command file options were possible until I saw the
> > comment in Unicorn.builder.  I just want some way to specify not to
> > load default middleware without requiring a command line option each
> > time.
> 
> Alright, I can accept a patch for the unicorn-specific config
> option only; but it definitely must not be allowed to work in
> .ru files.  Thanks.

OK. To implement that, I modified the bin/unicorn file so -N
is only respected while parsing ARGV, and not while parsing
embedded configuration file options.

Thanks,
Jeremy

From d628cad9e5c4b6e15783f1089b3bd84f42061bfb Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Thu, 13 Sep 2018 10:48:25 -0700
Subject: [PATCH] Support default_middleware configuration option

This allows for the equivalent of the
-N/--no-default_middleware command line option to be
specified in the configuration file so it doesn't
need to be specified on the command line every time
unicorn is executed.

It explicitly excludes the use of -N/--no-default_middleware
as an embedded configuration option in the rackup file, by
ignoring the options after ARGV is parsed.

In order to allow the configuration method to work, have
the lambda that Unicorn.builder returns accept two arguments.
Technically, only one argument is needed for the HttpServer
instance, but I'm guessing if the lambda accepts a single
argument, we expect that to be a rack application instead
of a lambda that returns a rack application.

The command line option option to disable default middleware
will take precedence over the unicorn configuration file option
if both are present.

For backwards compatibility, if the lambda passed to
HttpServer accepts 0 arguments, then call it without
arguments.
---
 bin/unicorn                 |  4 +++-
 lib/unicorn.rb              |  8 ++------
 lib/unicorn/configurator.rb | 11 +++++++++++
 lib/unicorn/http_server.rb  |  8 +++++---
 4 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/bin/unicorn b/bin/unicorn
index 3c5e5cb..00c8464 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -6,6 +6,7 @@
 ENV["RACK_ENV"] ||= "development"
 rackup_opts = Unicorn::Configurator::RACKUP
 options = rackup_opts[:options]
+set_no_default_middleware = true
 
 op = OptionParser.new("", 24, '  ') do |opts|
   cmd = File.basename($0)
@@ -60,7 +61,7 @@
 
   opts.on("-N", "--no-default-middleware",
           "do not load middleware implied by RACK_ENV") do |e|
-    rackup_opts[:no_default_middleware] = true
+    rackup_opts[:no_default_middleware] = true if set_no_default_middleware
   end
 
   opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
@@ -110,6 +111,7 @@
   opts.parse! ARGV
 end
 
+set_no_default_middleware = false
 app = Unicorn.builder(ARGV[0] || 'config.ru', op)
 op = nil
 
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b6dae36..5f2134d 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -45,12 +45,8 @@ def self.builder(ru, op)
       abort "rack and Rack::Builder must be available for processing #{ru}"
     end
 
-    # Op is going to get cleared before the returned lambda is called, so
-    # save this value so that it's still there when we need it:
-    no_default_middleware = op[:no_default_middleware]
-
     # always called after config file parsing, may be called after forking
-    lambda do ||
+    lambda do |_, server|
       inner_app = case ru
       when /\.ru$/
         raw = File.read(ru)
@@ -66,7 +62,7 @@ def self.builder(ru, op)
         pp({ :inner_app => inner_app })
       end
 
-      return inner_app if no_default_middleware
+      return inner_app unless server.default_middleware
 
       middleware = { # order matters
         ContentLength: nil,
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index f34d38b..9c36dfe 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -265,6 +265,14 @@ def worker_processes(nr)
     set_int(:worker_processes, nr, 1)
   end
 
+  # sets whether to add default middleware in the development and
+  # deployment RACK_ENVs.
+  #
+  # default_middleware is only available in unicorn 5.5.0+
+  def default_middleware(bool)
+    set_bool(:default_middleware, bool)
+  end
+
   # sets listeners to the given +addresses+, replacing or augmenting the
   # current set.  This is for the global listener pool shared by all
   # worker processes.  For per-worker listeners, see the after_fork example
@@ -706,6 +714,9 @@ def parse_rackup_file # :nodoc:
 
     /^#\\(.*)/ =~ File.read(ru) or return
     RACKUP[:optparse].parse!($1.split(/\s+/))
+    if RACKUP[:no_default_middleware]
+      set[:default_middleware] = false
+    end
 
     if RACKUP[:daemonize]
       # unicorn_rails wants a default pid path, (not plain 'unicorn')
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index b2bbddb..7531886 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -14,7 +14,8 @@ class Unicorn::HttpServer
   attr_accessor :app, :timeout, :worker_processes,
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
-                :orig_app, :config, :ready_pipe, :user
+                :orig_app, :config, :ready_pipe, :user,
+                :default_middleware
   attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
@@ -70,6 +71,7 @@ def initialize(app, options = {})
     @app = app
     @request = Unicorn::HttpRequest.new
     @reexec_pid = 0
+    @default_middleware = true
     options = options.dup
     @ready_pipe = options.delete(:ready_pipe)
     @init_listeners = options[:listeners] ? options[:listeners].dup : []
@@ -784,12 +786,12 @@ def listener_names(listeners = LISTENERS)
   end
 
   def build_app!
-    if app.respond_to?(:arity) && app.arity == 0
+    if app.respond_to?(:arity) && app.arity == 0 || app.arity == 2
       if defined?(Gem) && Gem.respond_to?(:refresh)
         logger.info "Refreshing Gem list"
         Gem.refresh
       end
-      self.app = app.call
+      self.app = app.arity == 0 ? app.call : app.call(nil, self)
     end
   end
 
-- 
2.17.1


^ permalink raw reply related	[relevance 6%]

* Re: Support default_middleware configurator method
  2018-09-14  0:00  6%   ` Jeremy Evans
@ 2018-09-14 10:56  0%     ` Eric Wong
  2018-09-14 15:03  6%       ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2018-09-14 10:56 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> On 09/13 10:34, Eric Wong wrote:
> > Worse case is we only support default_middleware as a config
> > option (but I prefer not to add more options).  Embedded -N support
> > is an anti-feature which leads to lock-in.
> 
> I don't really care about embedded -N support.  I actually didn't even
> know embedded command file options were possible until I saw the
> comment in Unicorn.builder.  I just want some way to specify not to
> load default middleware without requiring a command line option each
> time.

Alright, I can accept a patch for the unicorn-specific config
option only; but it definitely must not be allowed to work in
.ru files.  Thanks.

^ permalink raw reply	[relevance 0%]

* Re: Support default_middleware configurator method
  2018-09-13 22:34  5% ` Eric Wong
@ 2018-09-14  0:00  6%   ` Jeremy Evans
  2018-09-14 10:56  0%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2018-09-14  0:00 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 09/13 10:34, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > I was originally looking for a way to turn off default middleware
> > without having to specify the -N command line option every time.
> > After reading the Unicorn.builder code, I thought it may be
> > possible to do this by specifying the option as an embedded option
> > in the rackup file, via the following line at the top of the file:
> > 
> > #\-N
> > 
> > Unfortunately, while this works for many other command line options,
> > because of Unicorn's internals, this doesn't work for -N, as
> > embedded options are not parsed until after the check for skipping
> > default middleware is made.
> 
> That was intentional, I think.  We should never be encouraging
> users to pollute config.ru and make it difficult to migrate/test
> other servers.
> 
> > This patch makes the embedded -N option work, as well as adds a
> > default_middleware configuration file option
> 
> I really hate the -N vs RACK_ENV=none redundancy...  Is -N still
> needed for Rails or something else while keeping RACK_ENV unset?

I use RACK_ENV to configure the environment for my web applications, and
for a few reasons have used development/test/production as the
environment names.  I try as much as possible to have the development
environment mirror the production environment unless there is a good
reason to be different, and would like to avoid adding all of the
middleware Unicorn adds in development mode.

Additionally, let's say I want to add Rack::CommonLogger to both
development and production environments.  So in my config.ru, I add:

  use Rack::CommonLogger, Logger.new($stderr)

Now I load up the development version.  Then I have two common loggers
in development, with all requests being logged twice.  I currently
work around this by doing something like:

  unless ENV['RACK_ENV'] == 'development'
    use Rack::CommonLogger, Logger.new($stderr)
  end

I would like to avoid having to do that.

> Worse case is we only support default_middleware as a config
> option (but I prefer not to add more options).  Embedded -N support
> is an anti-feature which leads to lock-in.

I don't really care about embedded -N support.  I actually didn't even
know embedded command file options were possible until I saw the
comment in Unicorn.builder.  I just want some way to specify not to
load default middleware without requiring a command line option each
time.

Thanks,
Jeremy

^ permalink raw reply	[relevance 6%]

* Re: Support default_middleware configurator method
  2018-09-13 19:20  5% Support default_middleware configurator method Jeremy Evans
@ 2018-09-13 22:34  5% ` Eric Wong
  2018-09-14  0:00  6%   ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2018-09-13 22:34 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> I was originally looking for a way to turn off default middleware
> without having to specify the -N command line option every time.
> After reading the Unicorn.builder code, I thought it may be
> possible to do this by specifying the option as an embedded option
> in the rackup file, via the following line at the top of the file:
> 
> #\-N
> 
> Unfortunately, while this works for many other command line options,
> because of Unicorn's internals, this doesn't work for -N, as
> embedded options are not parsed until after the check for skipping
> default middleware is made.

That was intentional, I think.  We should never be encouraging
users to pollute config.ru and make it difficult to migrate/test
other servers.

> This patch makes the embedded -N option work, as well as adds a
> default_middleware configuration file option

I really hate the -N vs RACK_ENV=none redundancy...  Is -N still
needed for Rails or something else while keeping RACK_ENV unset?

Worse case is we only support default_middleware as a config
option (but I prefer not to add more options).  Embedded -N support
is an anti-feature which leads to lock-in.

Thanks.

^ permalink raw reply	[relevance 5%]

* Support default_middleware configurator method
@ 2018-09-13 19:20  5% Jeremy Evans
  2018-09-13 22:34  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2018-09-13 19:20 UTC (permalink / raw)
  To: unicorn-public

I was originally looking for a way to turn off default middleware
without having to specify the -N command line option every time.
After reading the Unicorn.builder code, I thought it may be
possible to do this by specifying the option as an embedded option
in the rackup file, via the following line at the top of the file:

#\-N

Unfortunately, while this works for many other command line options,
because of Unicorn's internals, this doesn't work for -N, as
embedded options are not parsed until after the check for skipping
default middleware is made.

This patch makes the embedded -N option work, as well as adds a
default_middleware configuration file option.  It does this by
passing the HttpServer instance to the lambda that returns the
rack application.  It uses arity checking to do this, like the
previous approach, but support an arity of 2 instead of just an
arity of 0 (which is still supported for backwards compatibility).

An alternative approach would be to parse the embedded options
inside Unicorn.builder, but that results in them getting parsed
twice as well as duplicating some code.  Additionally, embedded
options are less understandable than unicorn configuration file
options.  If you want a patch for that simpler approach, I have one.

Thanks,
Jeremy


From 90cca6ff773d58657a4b466a03e8a21b486c913f Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Thu, 13 Sep 2018 10:48:25 -0700
Subject: [PATCH] Support default_middleware configuration option

This allows for the equivalent of the
-N/--no-default_middleware command line option to be
specified in the configuration file.  It
also allows the -N/--no-default_middleware embedded
configuration option in the rackup file, which should
have worked previously but did not because embedded
configuration options weren't parsed until after the
app was built.

In order to allow the configuration method to work, have
the lambda that Unicorn.builder returns accept two arguments.
Technically, only one argument is needed for the HttpServer
instance, but I'm guessing if the lambda accepts a single
argument, we expect that to be a rack application instead
of a lambda that returns a rack application.

The command line option (or rackup embedded configuration
option) to disable default middleware will take precedence
over the unicorn configuration file option if both are
present.

For backwards compatibility, if the lambda passed to
HttpServer accepts 0 arguments, then call it without
arguments.
---
 lib/unicorn.rb              |  8 ++------
 lib/unicorn/configurator.rb | 11 +++++++++++
 lib/unicorn/http_server.rb  |  8 +++++---
 3 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b6dae36..5f2134d 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -45,12 +45,8 @@ def self.builder(ru, op)
       abort "rack and Rack::Builder must be available for processing #{ru}"
     end
 
-    # Op is going to get cleared before the returned lambda is called, so
-    # save this value so that it's still there when we need it:
-    no_default_middleware = op[:no_default_middleware]
-
     # always called after config file parsing, may be called after forking
-    lambda do ||
+    lambda do |_, server|
       inner_app = case ru
       when /\.ru$/
         raw = File.read(ru)
@@ -66,7 +62,7 @@ def self.builder(ru, op)
         pp({ :inner_app => inner_app })
       end
 
-      return inner_app if no_default_middleware
+      return inner_app unless server.default_middleware
 
       middleware = { # order matters
         ContentLength: nil,
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index f34d38b..9c36dfe 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -265,6 +265,14 @@ def worker_processes(nr)
     set_int(:worker_processes, nr, 1)
   end
 
+  # sets whether to add default middleware in the development and
+  # deployment RACK_ENVs.
+  #
+  # default_middleware is only available in unicorn 5.5.0+
+  def default_middleware(bool)
+    set_bool(:default_middleware, bool)
+  end
+
   # sets listeners to the given +addresses+, replacing or augmenting the
   # current set.  This is for the global listener pool shared by all
   # worker processes.  For per-worker listeners, see the after_fork example
@@ -706,6 +714,9 @@ def parse_rackup_file # :nodoc:
 
     /^#\\(.*)/ =~ File.read(ru) or return
     RACKUP[:optparse].parse!($1.split(/\s+/))
+    if RACKUP[:no_default_middleware]
+      set[:default_middleware] = false
+    end
 
     if RACKUP[:daemonize]
       # unicorn_rails wants a default pid path, (not plain 'unicorn')
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index b2bbddb..7531886 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -14,7 +14,8 @@ class Unicorn::HttpServer
   attr_accessor :app, :timeout, :worker_processes,
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
-                :orig_app, :config, :ready_pipe, :user
+                :orig_app, :config, :ready_pipe, :user,
+                :default_middleware
   attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
@@ -70,6 +71,7 @@ def initialize(app, options = {})
     @app = app
     @request = Unicorn::HttpRequest.new
     @reexec_pid = 0
+    @default_middleware = true
     options = options.dup
     @ready_pipe = options.delete(:ready_pipe)
     @init_listeners = options[:listeners] ? options[:listeners].dup : []
@@ -784,12 +786,12 @@ def listener_names(listeners = LISTENERS)
   end
 
   def build_app!
-    if app.respond_to?(:arity) && app.arity == 0
+    if app.respond_to?(:arity) && app.arity == 0 || app.arity == 2
       if defined?(Gem) && Gem.respond_to?(:refresh)
         logger.info "Refreshing Gem list"
         Gem.refresh
       end
-      self.app = app.call
+      self.app = app.arity == 0 ? app.call : app.call(nil, self)
     end
   end
 
-- 
2.17.1


^ permalink raw reply related	[relevance 5%]

* [PATCH] socket_helper: add hint for FreeBSD users for accf_http(9)
@ 2018-08-20 20:42  6% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-08-20 20:42 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong

Because I forget to load accf_http on new FreeBSD installs,
too :x
---
 lib/unicorn/socket_helper.rb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index b540302..8a6f6ee 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -83,6 +83,7 @@ def set_tcp_sockopt(sock, opt)
         rescue => e
           logger.error("#{sock_name(sock)} " \
                        "failed to set accept_filter=#{name} (#{e.inspect})")
+          logger.error("perhaps accf_http(9) needs to be loaded".freeze)
         end if arg != got
       end
     end
-- 
EW


^ permalink raw reply related	[relevance 6%]

* Re: KGIO issues under WSL
  @ 2018-08-14  3:14  4% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2018-08-14  3:14 UTC (permalink / raw)
  To: Sam Saffron; +Cc: unicorn-public

Sam Saffron <sam.saffron@gmail.com> wrote:
> Howdy,
> 
> For some crazy reason we really want to run Unicorn on Windows
> Subsytem for Linux. We only use Unicorn in production and some of our
> devs like using WSL for "crazy reasons TM".
> 
> I guess the on-boarding is somewhat easier for Windows users vs
> spinning up a proper VM.

IMHO, you're holding your developers back and they'll constantly
be running into new problems in Ruby development.

> WSL overall works fine but is missing some socket options that mean
> Kgio is toast and you can not accept sockets.

WSL needs to add support for those Linux options if they intend
to pass themselves off as Linux-compatible.  And it seems you
guys pay for MS licenses, so they they should be fixing stuff
for you.

<snip>

> I was thinking since we are going to be giving up kgio longer term
> anyway, is there a way of having some flag for running unicorn without
> kgio or maybe even with minimal socket options given setsockoptions
> and getsockoptions are patchy. A monkey patch would be fine a well
> cause this is just for development and Puma seems to be ok-ish.

No flags or new options for this.  Ruby 2.3+ should be OK to run
entirely w/o kgio using the `exception: false` keyword.  I don't
know if you want to take a stab at it, there's not a whole lot
of kgio use in unicorn, even.

However, Ruby 2.0 and even 1.9.3 support should be maintained;
but maybe that can lazy-load kgio and fall back to expensive
exceptions if kgio isn't found.

There should be no increase in the bytecode or stack footprint
for 2.3+ users, but it's fine for users on older Rubies.

I also wouldn't worry about exception costs for exception paths
where kgio_trywrite tries to write something but gives up if it
doesn't work; but the accept_nonblock codepath is pretty critical
for being kept garbage-free.


Anyways, we will never advertise support for any proprietary
systems, if it works it's incidental.

(Abg gung jr nqiregvfr ng nyy :C)

^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 5.4.0 - Rack HTTP server for fast clients and Unix
@ 2017-12-23 23:42  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-23 23:42 UTC (permalink / raw)
  To: ruby-talk, unicorn-public; +Cc: James P Robinson Jr, Sam Saffron

unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

Changes:

Rack hijack support improves as the app code can capture and use
the Rack `env' privately without copying it (to avoid clobbering
by another client).  Thanks to Sam Saffron for reporting and
testing this new feature:
  https://bogomips.org/unicorn-public/CAAtdryPG3nLuyo0jxfYW1YHu1Q+ZpkLkd4KdWC8vA46B5haZxw@mail.gmail.com/T/

We also now support $DEBUG being set by the Rack app (instead of
relying on the "-d" CLI switch).  Thanks to James P Robinson Jr
for reporting this bug:
  https://bogomips.org/unicorn-public/D6324CB4.7BC3E%25james.robinson3@cigna.com/T/
  (Coincidentally, this fix will be irrelevant for Ruby 2.5
   which requires 'pp' by default)

There's a few minor test cleanups and documentation updates, too.

All commits since v5.3.1 (2017-10-03):

    reduce method calls with String#start_with?
    require 'pp' if $DEBUG is set by Rack app
    avoid reusing env on hijack
    tests: cleanup some unused variable warnings
    ISSUES: add a note about Debian BTS interopability

Roughly all mailing discussions since the last release:

  https://bogomips.org/unicorn-public/?q=d:20171004..20171223

^ permalink raw reply	[relevance 4%]

* [PATCH] ISSUES: add a note about Debian BTS interopability
@ 2017-12-23 23:06  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-23 23:06 UTC (permalink / raw)
  To: unicorn-public

This isn't anything new, just explicitly stating what's always
been the case.

In other news, I hate formal things.
---
 ISSUES | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/ISSUES b/ISSUES
index 10df4d6..610f011 100644
--- a/ISSUES
+++ b/ISSUES
@@ -15,6 +15,11 @@ submit patches and/or obtain support after you have searched the
 * The email submission port (587) is enabled on the bogomips.org MX:
   https://bogomips.org/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
 
+We will never have a centralized or formal bug tracker.  Instead we
+can interoperate with any bug tracker which can Cc: us plain-text to
+mailto:unicorn-public@bogomips.org   This includes the Debian BTS
+at https://bugs.debian.org/unicorn and possibly others.
+
 If your issue is of a sensitive nature or you're just shy in public,
 use anonymity tools such as Tor or Mixmaster; and rely on the public
 mail archives for responses.  Be sure to scrub sensitive log messages
-- 
EW


^ permalink raw reply related	[relevance 5%]

* [PATCH] tests: cleanup some unused variable warnings
@ 2017-12-22  3:17  3% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-22  3:17 UTC (permalink / raw)
  To: unicorn-public

Add a new "check-warnings" target to the GNUmakefile to make
checking for this easier.  Warnings aren't fatal, and newer
versions of Ruby tend to increase warnings.
---
 GNUmakefile               |  5 +++++
 test/unit/test_droplet.rb |  2 +-
 test/unit/test_request.rb | 20 ++++++++++----------
 3 files changed, 16 insertions(+), 11 deletions(-)

diff --git a/GNUmakefile b/GNUmakefile
index 51045d4..2505e1f 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -249,5 +249,10 @@ endif
 $(PLACEHOLDERS):
 	echo olddoc_placeholder > $@
 
+check-warnings:
+	@(for i in $$(git ls-files '*.rb' bin | grep -v '^setup\.rb$$'); \
+	  do $(RUBY) --disable-gems -d -W2 -c \
+	  $$i; done) | grep -v '^Syntax OK$$' || :
+
 .PHONY: .FORCE-GIT-VERSION-FILE doc $(T) $(slow_tests) man
 .PHONY: test-install
diff --git a/test/unit/test_droplet.rb b/test/unit/test_droplet.rb
index 73cf38c..81ad82b 100644
--- a/test/unit/test_droplet.rb
+++ b/test/unit/test_droplet.rb
@@ -4,7 +4,7 @@
 class TestDroplet < Test::Unit::TestCase
   def test_create_many_droplets
     now = Time.now.to_i
-    tmp = (0..1024).map do |i|
+    (0..1024).each do |i|
       droplet = Unicorn::Worker.new(i)
       assert droplet.respond_to?(:tick)
       assert_equal 0, droplet.tick
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index f0ccaf7..6cb0268 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -34,7 +34,7 @@ def test_options
     assert_equal '', env['REQUEST_PATH']
     assert_equal '', env['PATH_INFO']
     assert_equal '*', env['REQUEST_URI']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_absolute_uri_with_query
@@ -44,7 +44,7 @@ def test_absolute_uri_with_query
     assert_equal '/x', env['REQUEST_PATH']
     assert_equal '/x', env['PATH_INFO']
     assert_equal 'y=z', env['QUERY_STRING']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_absolute_uri_with_fragment
@@ -55,7 +55,7 @@ def test_absolute_uri_with_fragment
     assert_equal '/x', env['PATH_INFO']
     assert_equal '', env['QUERY_STRING']
     assert_equal 'frag', env['FRAGMENT']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_absolute_uri_with_query_and_fragment
@@ -66,7 +66,7 @@ def test_absolute_uri_with_query_and_fragment
     assert_equal '/x', env['PATH_INFO']
     assert_equal 'a=b', env['QUERY_STRING']
     assert_equal 'frag', env['FRAGMENT']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_absolute_uri_unsupported_schemes
@@ -83,7 +83,7 @@ def test_x_forwarded_proto_https
                              "Host: foo\r\n\r\n")
     env = @request.read(client)
     assert_equal "https", env['rack.url_scheme']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_x_forwarded_proto_http
@@ -92,7 +92,7 @@ def test_x_forwarded_proto_http
                              "Host: foo\r\n\r\n")
     env = @request.read(client)
     assert_equal "http", env['rack.url_scheme']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_x_forwarded_proto_invalid
@@ -101,7 +101,7 @@ def test_x_forwarded_proto_invalid
                              "Host: foo\r\n\r\n")
     env = @request.read(client)
     assert_equal "http", env['rack.url_scheme']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_rack_lint_get
@@ -109,7 +109,7 @@ def test_rack_lint_get
     env = @request.read(client)
     assert_equal "http", env['rack.url_scheme']
     assert_equal '127.0.0.1', env['REMOTE_ADDR']
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_no_content_stringio
@@ -143,7 +143,7 @@ def test_rack_lint_put
       "abcde")
     env = @request.read(client)
     assert ! env.include?(:http_body)
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 
   def test_rack_lint_big_put
@@ -177,6 +177,6 @@ def client.kgio_read!(*args)
     }
     assert_nil env['rack.input'].read(bs)
     env['rack.input'].rewind
-    res = @lint.call(env)
+    assert_kind_of Array, @lint.call(env)
   end
 end
-- 
EW

^ permalink raw reply related	[relevance 3%]

* Re: Auto scaling workers with unicorn
  2017-12-04 23:42  5% Auto scaling workers with unicorn Sam Saffron
@ 2017-12-05  1:51  0% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-12-05  1:51 UTC (permalink / raw)
  To: Sam Saffron; +Cc: unicorn-public

Sam Saffron <sam.saffron@gmail.com> wrote:
> I would like to amend Discourse so we "automatically" absorb certain
> traffic spikes. As it stands we can only configure unicorn with
> num_workers and use TTIN and TTOUT to tune the number on the fly.
> 
> I was wondering if you would be open to patching unicorn to allow it
> to perform auto-tuning based on raindrops info.

I'm no fan of this or auto-tuning systems in general.
More explanation below.

> How it could work
> 
> 1. configure unicorn with min_workers, max_workers, wince_delay, scale_up_delay
> 
> 2. If queued requests is over 0 for N samples over scale_up_delay, add
> a worker up until max_workers
> 
> 3. If queued requests is 0 for N samples over wince_delay scale down
> until you reach min workers

This adds more complexity to configuration: increasing the
likelyhood of getting these numbers completely wrong.  GC and
malloc tuning is tricky and error-prone enough, already.

Mainly, this tends to hide problems for later; instead of
forcing you to deal with your resource limitations up front.

It becomes more difficult to forsee resource limitations down
the line.  Before I worked on unicorn, I've seen auto-scaling
Apache workers mistuned far too often and running out of DB
connections or memory; and that happens at the worst time:
when your site is under heavy load (when you have the most to
lose (or gain)).

> Having this system in place can heavily optimize memory in large
> deployments and simplifies provisioning logic quite a lot.

My philosophy remains to tune for the worst case possible.

If you really need to do something like run an expensive
off-peak cronjob, maybe have it TTIN at the beginning and TTOU
again at the end.

Fwiw, the most useful thing I've found TTIN/TTOU for is cutting
down to one worker so I know which one to strace when tracking
down a problem; not auto-scaling.

> Wondering what you think about this and if you think unicorn should
> provide this option?

Fwiw, my position has been consistent on this throughout the years.

Also, digging through the archives, Ben Somers came up with
alicorn a while back and it might be up your alley:

https://bogomips.org/unicorn-public/CAO1NZApo0TLJY2KgSg+Fjt1jEcuPfq=UCC0SCvvnuGDnr39w8w@mail.gmail.com/

^ permalink raw reply	[relevance 0%]

* Auto scaling workers with unicorn
@ 2017-12-04 23:42  5% Sam Saffron
  2017-12-05  1:51  0% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Sam Saffron @ 2017-12-04 23:42 UTC (permalink / raw)
  To: unicorn-public

I would like to amend Discourse so we "automatically" absorb certain
traffic spikes. As it stands we can only configure unicorn with
num_workers and use TTIN and TTOUT to tune the number on the fly.

I was wondering if you would be open to patching unicorn to allow it
to perform auto-tuning based on raindrops info.

How it could work

1. configure unicorn with min_workers, max_workers, wince_delay, scale_up_delay

2. If queued requests is over 0 for N samples over scale_up_delay, add
a worker up until max_workers

3. If queued requests is 0 for N samples over wince_delay scale down
until you reach min workers

Having this system in place can heavily optimize memory in large
deployments and simplifies provisioning logic quite a lot.

Wondering what you think about this and if you think unicorn should
provide this option?

^ permalink raw reply	[relevance 5%]

* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
  2017-10-03 14:52  5% ` Xuanzhong Wei
@ 2017-10-03 17:15  0%   ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-10-03 17:15 UTC (permalink / raw)
  To: Xuanzhong Wei; +Cc: code, maillist, pere.joan, philip, unicorn-public

Xuanzhong Wei <azrlew@gmail.com> wrote:
> We have the same issue here.
> 
> IMHO, it is a bug introduced by 979ebcf91705709be5041a3be4514e5f1f6ec02c.
> The `mark_ary` get GCed before we add it to the ruby's global_list
> since we are doing memory allocations before calling rb_global_variable.
> 
> A simple test can be found here:
> https://github.com/azrle/ruby_c_ext_test

Thanks, which compiler and version did you use?

> I will try to submit a patch later.

https://bogomips.org/unicorn-public/20171003145718.30404-1-azrlew@gmail.com/raw

Yes, seems corect since the compiler doesn't need to keep
mark_ary anymore once it only needs the address (&mark_ary).
OBJ_FREEZE is an inline which does nothing to prevent
the compiler from only keeping RBasic->flags around and
not the actual VALUE.

^ permalink raw reply	[relevance 0%]

* [PATCH] fix GC issue on rb_global_variable array
@ 2017-10-03 14:57  5% Xuanzhong Wei
  0 siblings, 0 replies; 200+ results
From: Xuanzhong Wei @ 2017-10-03 14:57 UTC (permalink / raw)
  To: unicorn-public; +Cc: Xuanzhong Wei

We need to add the array to ruby's global_list right after created it;
otherwise it probably gets GCed.
---
 ext/unicorn_http/unicorn_http.rl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 6fc3498..357440b 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -921,6 +921,7 @@ void Init_unicorn_http(void)
   VALUE mUnicorn, cHttpParser;
 
   mark_ary = rb_ary_new();
+  rb_global_variable(&mark_ary);
   mUnicorn = rb_define_module("Unicorn");
   cHttpParser = rb_define_class_under(mUnicorn, "HttpParser", rb_cObject);
   eHttpParserError =
@@ -976,7 +977,6 @@ void Init_unicorn_http(void)
   init_unicorn_httpdate(mark_ary);
 
   OBJ_FREEZE(mark_ary);
-  rb_global_variable(&mark_ary);
 
 #ifndef HAVE_RB_HASH_CLEAR
   id_clear = rb_intern("clear");
-- 
2.14.1


^ permalink raw reply related	[relevance 5%]

* Re: Random crash when sending USR2 + QUIT signals to Unicorn process
  @ 2017-10-03 14:52  5% ` Xuanzhong Wei
  2017-10-03 17:15  0%   ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Xuanzhong Wei @ 2017-10-03 14:52 UTC (permalink / raw)
  To: e; +Cc: code, maillist, pere.joan, philip, unicorn-public, Xuanzhong Wei

We have the same issue here.

IMHO, it is a bug introduced by 979ebcf91705709be5041a3be4514e5f1f6ec02c.
The `mark_ary` get GCed before we add it to the ruby's global_list
since we are doing memory allocations before calling rb_global_variable.

A simple test can be found here:
https://github.com/azrle/ruby_c_ext_test

I will try to submit a patch later.

^ permalink raw reply	[relevance 5%]

* Re: Patch: Add support for chroot to Worker#user V2
  2017-04-05  3:44  6% ` Eric Wong
@ 2017-04-05  4:57  6%   ` Jeremy Evans
  0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2017-04-05  4:57 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 04/05 03:44, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > @@ -134,6 +136,11 @@ def user(user, group = nil)
> >        Process.initgroups(user, gid)
> >        Process::GID.change_privilege(gid)
> >      end
> > +    if chroot
> > +      chroot = Dir.pwd if chroot == true
> > +      Dir.chroot(chroot)
> > +      Dir.chdir('/')
> > +    end
> 
> By the way, I noticed in configurator.rb (for
> working_directory), we also update ENV['PWD'] after chdir.
> 
> Perhaps we should do so, here?

That makes sense to me, though I don't use ENV['PWD'] personally.

> For working_directory, I preserved ENV['PWD'] in case Dir.pwd
> wasn't aware of symlinks; or there's code which relies on
> env['PWD'] without caring for making the syscalls required for
> Dir.pwd...
> 
> I'm not sure how much it matters in practice...

I'm not either, but I don't see how it could hurt.

Thanks,
Jeremy

^ permalink raw reply	[relevance 6%]

* Re: Patch: Add support for chroot to Worker#user V2
  2017-02-23 18:46  8% Patch: Add support for chroot to Worker#user V2 Jeremy Evans
@ 2017-04-05  3:44  6% ` Eric Wong
  2017-04-05  4:57  6%   ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-04-05  3:44 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> @@ -134,6 +136,11 @@ def user(user, group = nil)
>        Process.initgroups(user, gid)
>        Process::GID.change_privilege(gid)
>      end
> +    if chroot
> +      chroot = Dir.pwd if chroot == true
> +      Dir.chroot(chroot)
> +      Dir.chdir('/')
> +    end

By the way, I noticed in configurator.rb (for
working_directory), we also update ENV['PWD'] after chdir.

Perhaps we should do so, here?

For working_directory, I preserved ENV['PWD'] in case Dir.pwd
wasn't aware of symlinks; or there's code which relies on
env['PWD'] without caring for making the syscalls required for
Dir.pwd...

I'm not sure how much it matters in practice...

^ permalink raw reply	[relevance 6%]

* Re: after_worker_exit on murder
  2017-04-04 14:08  5% after_worker_exit on murder Simon Eskildsen
  2017-04-04 14:32  0% ` Jeremy Evans
@ 2017-04-05  1:19  4% ` Eric Wong
  1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2017-04-05  1:19 UTC (permalink / raw)
  To: Simon Eskildsen; +Cc: unicorn-public, Jeremy Evans

Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
> With Jeremy Evans' work on `after_worker_exit`, I was hoping I could
> replace our internal fork which has a `before_murder` hook to report
> to monitoring systems when workers are forcibly killed. However, the
> `after_worker_exit` is only called on reaping—not when murdering.

Hi Simon, it looks like Jeremy clarified after_worker_exit for you.

Anyways...  I remember rejecting patches to add more to timeout
support in unicorn over the years since I did not want it to be
a crutch for application developers, or worse; a reason for
people to feel locked into unicorn.  Instead I wrote things like
https://bogomips.org/unicorn/Application_Timeouts.html to
discourage relying on unicorn's built-in `timeout' feature.

But, it seems like there's still a reliance on the built-in
timeout...

Why is that?  (If you're allowed to disclose)

I don't mean to put you guys (Shopify) on the spot,
as I'm sure other folks do it, too; but you're here :)

Anyways, is this something that could or should be improved
in Rack or ruby itself?  Or are there buggy external libraries
or even external dependencies like NFS to blame?

(Or, perhaps "I don't know" is fine :)

The Ruby timeout handling in stdlib could be way better if
done natively, I think.  Perhaps `ensure` could be better
declared, too; but there's still the problem of getting
external code to work well with it...


Anyways I am of two minds; on one hand, buggy code is fine!
unicorn handles it by nuking a process.  But on the other
hand... it's irritatingly inefficient and shoving things under
the rug still rots and stinks eventually.

^ permalink raw reply	[relevance 4%]

* Re: after_worker_exit on murder
  2017-04-04 14:08  5% after_worker_exit on murder Simon Eskildsen
@ 2017-04-04 14:32  0% ` Jeremy Evans
  2017-04-05  1:19  4% ` Eric Wong
  1 sibling, 0 replies; 200+ results
From: Jeremy Evans @ 2017-04-04 14:32 UTC (permalink / raw)
  To: Simon Eskildsen; +Cc: unicorn-public

On 04/04 10:08, Simon Eskildsen wrote:
> Hi!
> 
> With Jeremy Evans' work on `after_worker_exit`, I was hoping I could
> replace our internal fork which has a `before_murder` hook to report
> to monitoring systems when workers are forcibly killed. However, the
> `after_worker_exit` is only called on reaping???not when murdering.
> 
> How would you feel about executing this on the murder path?
> Alternatively, we could add an `after_murder` hook for this path.

I checked and after_worker_exit is called when reaping the worker
after murdering, so I don't think any changes are necessary. I would
guess if you added it when murdering, it would be called twice on the
same worker.

Thanks,
Jeremy

^ permalink raw reply	[relevance 0%]

* after_worker_exit on murder
@ 2017-04-04 14:08  5% Simon Eskildsen
  2017-04-04 14:32  0% ` Jeremy Evans
  2017-04-05  1:19  4% ` Eric Wong
  0 siblings, 2 replies; 200+ results
From: Simon Eskildsen @ 2017-04-04 14:08 UTC (permalink / raw)
  To: unicorn-public, Jeremy Evans

Hi!

With Jeremy Evans' work on `after_worker_exit`, I was hoping I could
replace our internal fork which has a `before_murder` hook to report
to monitoring systems when workers are forcibly killed. However, the
`after_worker_exit` is only called on reaping—not when murdering.

How would you feel about executing this on the murder path?
Alternatively, we could add an `after_murder` hook for this path.

^ permalink raw reply	[relevance 5%]

* [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix
@ 2017-04-01  8:08  6% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-04-01  8:08 UTC (permalink / raw)
  To: ruby-talk, unicorn-public
  Cc: Jeremy Evans, Simon Eskildsen, Dylan Thacker-Smith

unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

Changes:

unicorn 5.3.0

A couple of portability fixes from Dylan Thacker-Smith and
Jeremy Evans since 5.3.0.pre1 over a week ago, but this looks
ready for a stable release, today.

When I started this over 8 years ago, I wondered if this would
just end up being an April Fools' joke.  Guess not.  I guess I
somehow tricked people into using a terribly marketed web server
that cannot talk directly to untrusted clients :x  Anyways,
unicorn won't be able to handle slow clients 8 years from now,
either, or 80 years from now.  And I vow never to learn to use
new-fangled things like epoll, kqueue, or threads :P

Anyways, this is a largish release with several new features,
and no backwards incompatibilities.

Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
  https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170401&x=t

This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:

   https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u

Jeremy Evans contributed several new features.  First he
implemented after_worker_exit to aid debugging:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
  https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170401&x=t#t

And then security-related features to isolate workers.  Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
  https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
  https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170401&x=t#t
  https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170401&x=t#t

Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
  https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170401&x=t#t

There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.

shortlog of changes since v5.2.0 (2016-10-31):

Dylan Thacker-Smith (1):
      Check for Socket::TCP_INFO constant before trying to get TCP_INFO

Eric Wong (30):
      drop rb_str_set_len compatibility replacement
      TUNING: document THP caveat for Linux users
      tee_input: simplify condition for IO#write
      remove response_start_sent
      http_request: freeze constant strings passed IO#write
      Revert "remove response_start_sent"
      t/t0012-reload-empty-config.sh: access ivars directly if needed
      t0011-active-unix-socket.sh: fix race condition in test
      new test for check_client_connection
      revert signature change to HttpServer#process_client
      support "struct tcp_info" on non-Linux and Ruby 2.2+
      unicorn_http: reduce rb_global_variable calls
      oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
      http_request: reduce insn size for check_client_connection
      freebsd: avoid EINVAL when setting accept filter
      test-lib: expr(1) portability fix
      tests: keep disabled tests defined
      test_exec: SO_KEEPALIVE value only needs to be true
      doc: fix links to raindrops project
      http_request: support proposed Raindrops::TCP states on non-Linux
      ISSUES: expand on mail archive info + subscription disclaimer
      test_ccc: use a pipe to synchronize test
      doc: remove private email support address
      input: update documentation and hide internals.
      http_server: initialize @pid ivar
      gemspec: remove olddoc from build dependency
      doc: add version annotations for new features
      unicorn 5.3.0.pre1
      doc: note after_worker_exit is also 5.3.0+
      test_exec: SO_KEEPALIVE value only needs to be true (take #2)

Jeremy Evans (7):
      Add after_worker_exit configuration option
      Fix code example in after_worker_exit documentation
      Add support for chroot to Worker#user
      Add after_worker_ready configuration option
      Add worker_exec configuration option
      Don't pass a block for fork when forking workers
      Check for SocketError on first ccc attempt

Simon Eskildsen (1):
      check_client_connection: use tcp state on linux

-- 
Yes, this release is real despite the date.

^ permalink raw reply	[relevance 6%]

* [PATCH] doc: note after_worker_exit is also 5.3.0+
@ 2017-03-24  0:33  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-24  0:33 UTC (permalink / raw)
  To: unicorn-public; +Cc: Jeremy Evans

Followup-to: 650e01ab0b118803486b56f3ee59521d59042dae
  ("doc: add version annotations for new features")
---
 Oops, missed the 5.3.0.pre1 release; I managed to squash in
 the worker_exec change for that, though.

 lib/unicorn/configurator.rb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 5203537b..7fb9030f 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -172,6 +172,8 @@ def after_fork(*args, &block)
   #      server.logger.error("worker process failure: #{status.inspect}")
   #    end
   #  end
+  #
+  # after_worker_exit is only available in unicorn 5.3.0+
   def after_worker_exit(*args, &block)
     set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
   end
-- 
EW

^ permalink raw reply related	[relevance 5%]

* [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix
@ 2017-03-24  0:28  6% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-24  0:28 UTC (permalink / raw)
  To: ruby-talk, unicorn-public; +Cc: Jeremy Evans, Simon Eskildsen

unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

This is a pre-release RubyGem intended for testing.

Changes:

unicorn 5.3.0.pre1

A largish release with several new features.

Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
  https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170324&x=t

This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:

  https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u

Jeremy Evans contributed several new features.  First he
implemented after_worker_exit to aid debugging:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
  https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170324&x=t#t

And then security-related features to isolate workers.  Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
  https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
  https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170324&x=t#t
  https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170324&x=t#t

Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
  https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170324&x=t#t

There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.

34 changes since 5.2.0 (2016-10-31):

Eric Wong (27):
      drop rb_str_set_len compatibility replacement
      TUNING: document THP caveat for Linux users
      tee_input: simplify condition for IO#write
      remove response_start_sent
      http_request: freeze constant strings passed IO#write
      Revert "remove response_start_sent"
      t/t0012-reload-empty-config.sh: access ivars directly if needed
      t0011-active-unix-socket.sh: fix race condition in test
      new test for check_client_connection
      revert signature change to HttpServer#process_client
      support "struct tcp_info" on non-Linux and Ruby 2.2+
      unicorn_http: reduce rb_global_variable calls
      oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
      http_request: reduce insn size for check_client_connection
      freebsd: avoid EINVAL when setting accept filter
      test-lib: expr(1) portability fix
      tests: keep disabled tests defined
      test_exec: SO_KEEPALIVE value only needs to be true
      doc: fix links to raindrops project
      http_request: support proposed Raindrops::TCP states on non-Linux
      ISSUES: expand on mail archive info + subscription disclaimer
      test_ccc: use a pipe to synchronize test
      doc: remove private email support address
      input: update documentation and hide internals.
      http_server: initialize @pid ivar
      gemspec: remove olddoc from build dependency
      doc: add version annotations for new features

Jeremy Evans (6):
      Add after_worker_exit configuration option
      Fix code example in after_worker_exit documentation
      Add support for chroot to Worker#user
      Add after_worker_ready configuration option
      Add worker_exec configuration option
      Don't pass a block for fork when forking workers

Simon Eskildsen (1):
      check_client_connection: use tcp state on linux

-- 
5.3.0 in a week, maybe?

^ permalink raw reply	[relevance 6%]

* [ANN] raindrops 0.18.0 - real-time stats for preforking Rack servers
@ 2017-03-23  2:48  3% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-23  2:48 UTC (permalink / raw)
  To: ruby-talk, raindrops-public; +Cc: unicorn-public, Simon Eskildsen, Jeremy Evans

raindrops is a real-time stats toolkit to show statistics for Rack HTTP
servers.  It is designed for preforking servers such as unicorn, but
should support any Rack HTTP server on platforms supporting POSIX shared
memory.  It may also be used as a generic scoreboard for sharing atomic
counters across multiple processes.

* https://bogomips.org/raindrops/
* No subscription necessary, no HTML mail:
  raindrops-public@bogomips.org
* mail archives: https://bogomips.org/raindrops-public/
  http://ou63pmih66umazou.onion/raindrops-public/
  nntp://news.public-inbox.org/inbox.comp.lang.ruby.raindrops
  nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.raindrops
* git clone git://bogomips.org/raindrops.git
* https://bogomips.org/raindrops/NEWS.atom.xml
* Demo site: https://raindrops-demo.bogomips.org:8443/

Changes:

    raindrops 0.18.0
    
    The most notable feature of this release is the addition of
    FreeBSD and OpenBSD TCP_INFO support.  This includes the
    Raindrops::TCP for portably mapping TCP state names to
    platform-dependent numeric values:
    
      https://bogomips.org/raindrops/Raindrops.html#TCP
    
    Thanks to Jeremy Evans and Simon Eskildsen on the
    unicorn-public@bogomips.org mailing list for inspiring
    these changes to raindrops.
    
    There's also a few internal cleanups, and documentation
    improvements, including some fixes to the largely-forgotten
    Raindrops::Aggreage::PMQ class:
    
      https://bogomips.org/raindrops/Raindrops/Aggregate/PMQ.html
    
    20 changes since 0.17.0:
    
          test_inet_diag_socket: fix Fixnum deprecation warning
          TODO: add item for IPv6 breakage
          ext: fix documentation for C ext-defined classes
          TCP_Info: custom documentation for #get!
          TypedData C-API conversion
          test_watcher: disable test correctly when aggregate is missing
          tcp_info: support this struct under FreeBSD
          define Raindrops::TCP hash for TCP states
          linux_inet_diag: reduce stack usage and simplify
          avoid reading errno repeatedly
          aggregate/pmq: avoid false sharing of lock buffers
          aggregate/pmq: remove io-extra requirement
          aggregate/pmq: avoid File#stat allocation
          Merge remote-tracking branch 'origin/freebsd'
          Merge remote-tracking branch 'origin/aggregate-pmq'
          doc: remove private email support address
          doc: update location of TCP_INFO-related stuff
          build: avoid olddoc for building the RubyGem
          doc: document Raindrops::TCP hash
          aggregate/pmq: update version numbers for Ruby and Linux

^ permalink raw reply	[relevance 3%]

* Re: mail queue breakage :x
  2017-03-21 23:10  5% ` Eric Wong
@ 2017-03-21 23:24  0%   ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-21 23:24 UTC (permalink / raw)
  To: unicorn-public

Eric Wong <e@80x24.org> wrote:
> Anyways, feel free to exit/run it yourself, with or without tor
> and add mirrors and whatnot.  And you can hammer the hell out

Just be warned that attached validation script checks against
deliveries to mail-archive.com; so no, don't hammer the
mail-archive.com servers; I don't know how they set their server
up...

But yeah, anything running off bogomips.org is designed to
handle being on the frontpage of major news aggregators and
then some.

^ permalink raw reply	[relevance 0%]

* Re: mail queue breakage :x
  @ 2017-03-21 23:10  5% ` Eric Wong
  2017-03-21 23:24  0%   ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-21 23:10 UTC (permalink / raw)
  To: unicorn-public

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

Maybe the attached script will help detect mirror mismatches
in the future (run via cron).  But of course the original
mail failure was due to the crontab of the mlmmj user
getting clobbered :x

Anyways, feel free to exit/run it yourself, with or without tor
and add mirrors and whatnot.  And you can hammer the hell out
of https://bogomips.org/unicorn-public/new.atom or
http://ou63pmih66umazou.onion/unicorn-public/new.atom or
nntp://news.public-inbox.org/

(it doesn't run unicorn or nginx :P)

[-- Attachment #2: bogomips-mail-check.rb --]
[-- Type: application/x-ruby, Size: 1465 bytes --]

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] Add worker_exec configuration option V2
  2017-03-13 15:32  5%           ` Jeremy Evans
@ 2017-03-13 19:18  6%             ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-13 19:18 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Here's a patch to fix the stack depth issue:

Thanks, applied with some trailing whitespace fixups.

(I suggest enabling the pre-commit hook distributed with
 git in $GIT_DIR/hooks/pre-commit, it'll run "git diff-index --check ..."
 to check that)

> From aa4914846a0870e5b01530a47d6dbfe09aeb39ce Mon Sep 17 00:00:00 2001

Btw, you can omit the above line and replace it with "---8<---"
(no quotes) so the applier can use "git am --scissors".

Thanks.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add worker_exec configuration option V2
  2017-03-11  7:18  6%         ` Eric Wong
@ 2017-03-13 15:32  5%           ` Jeremy Evans
  2017-03-13 19:18  6%             ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-03-13 15:32 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 03/11 07:18, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > On 03/10 09:19, Eric Wong wrote:
> > > Jeremy Evans <code@jeremyevans.net> wrote:
> > > > -      if pid = fork
> > > > -        @workers[pid] = worker
> > > > -        worker.atfork_parent
> > > > +
> > > > +      pid = if @worker_exec
> > > > +        worker_spawn(worker)
> > > >        else
> > > > -        after_fork_internal
> > > > -        worker_loop(worker)
> > > > -        exit
> > > > +        fork do
> > > > +          after_fork_internal
> > > > +          worker_loop(worker)
> > > > +          exit
> > > > +        end
> > > 
> > > I prefer to avoid the block with fork.  The block deepens the
> > > stack for the running app, so it can affect GC efficiency.
> > > 
> > > Can be fixed in a separate patch...
> > 
> > That makes sense.  If you would like me to send a separate patch to fix
> > it, I can do that.
> 
> Yes, please.
> 
> Not sure if we should automate the stack depth test...
> I resisted it in the past since it might be too fragile
> w.r.t changes to Ruby (even using "unicorn" via the
> RubyGems-generated wrapper deepens it by 2).

Here's a patch to fix the stack depth issue:

From aa4914846a0870e5b01530a47d6dbfe09aeb39ce Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Mon, 13 Mar 2017 08:28:54 -0700
Subject: [PATCH] Don't pass a block for fork when forking workers

This reduces the stack depth, making GC more efficient.
---
 lib/unicorn/http_server.rb | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index a5bd2c4..023df10 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -541,14 +541,12 @@ def spawn_missing_workers
       worker = Unicorn::Worker.new(worker_nr)
       before_fork.call(self, worker)
 
-      pid = if @worker_exec
-        worker_spawn(worker)
-      else
-        fork do
-          after_fork_internal
-          worker_loop(worker)
-          exit
-        end
+      pid = @worker_exec ?  worker_spawn(worker) : fork
+      
+      unless pid
+        after_fork_internal
+        worker_loop(worker)
+        exit
       end
 
       @workers[pid] = worker
-- 
2.11.0


^ permalink raw reply related	[relevance 5%]

* Re: [PATCH] Add worker_exec configuration option V2
  2017-03-11  5:26  6%       ` Jeremy Evans
@ 2017-03-11  7:18  6%         ` Eric Wong
  2017-03-13 15:32  5%           ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-11  7:18 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> On 03/10 09:19, Eric Wong wrote:
> > tests, later, or you can.  I also had some FreeBSD test fixes
> > (which might apply to OpenBSD) on a VM somewhere which I'll Cc:
> > you on: there was also just SO_KEEPALIVE fix I posted:
> > 
> >   https://bogomips.org/unicorn-public/20170310203431.28067-1-e@80x24.org/raw
> 
> The C test code also returns 8 on OpenBSD, FWIW.  I'm happy to test any
> test fixes on OpenBSD, just let me know.

Thanks. I'll be happy to help fix any OpenBSD failures you see
from "gmake check"  I don't think the accept_filter fixes apply
to OpenBSD, but I guess the expr(1) fix did.

Thank you.

Same goes for NetBSD, DragonflyBSD or any other completely
Free/Open Source OSes anybody else here uses.

> > Jeremy Evans <code@jeremyevans.net> wrote:
> > > -      if pid = fork
> > > -        @workers[pid] = worker
> > > -        worker.atfork_parent
> > > +
> > > +      pid = if @worker_exec
> > > +        worker_spawn(worker)
> > >        else
> > > -        after_fork_internal
> > > -        worker_loop(worker)
> > > -        exit
> > > +        fork do
> > > +          after_fork_internal
> > > +          worker_loop(worker)
> > > +          exit
> > > +        end
> > 
> > I prefer to avoid the block with fork.  The block deepens the
> > stack for the running app, so it can affect GC efficiency.
> > 
> > Can be fixed in a separate patch...
> 
> That makes sense.  If you would like me to send a separate patch to fix
> it, I can do that.

Yes, please.

Not sure if we should automate the stack depth test...
I resisted it in the past since it might be too fragile
w.r.t changes to Ruby (even using "unicorn" via the
RubyGems-generated wrapper deepens it by 2).

Thanks again.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add worker_exec configuration option V2
  2017-03-10 21:19 11%     ` Eric Wong
@ 2017-03-11  5:26  6%       ` Jeremy Evans
  2017-03-11  7:18  6%         ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-03-11  5:26 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 03/10 09:19, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > Here's V2 of the patch, which I think should address all of the issues
> > you pointed out.
> 
> Thanks.  Pushed to git://80x24.org/unicorn worker_exec I can add
> tests, later, or you can.  I also had some FreeBSD test fixes
> (which might apply to OpenBSD) on a VM somewhere which I'll Cc:
> you on: there was also just SO_KEEPALIVE fix I posted:
> 
>   https://bogomips.org/unicorn-public/20170310203431.28067-1-e@80x24.org/raw

The C test code also returns 8 on OpenBSD, FWIW.  I'm happy to test any
test fixes on OpenBSD, just let me know.

> 
> >      worker_nr = -1
> >      until (worker_nr += 1) == @worker_processes
> >        @workers.value?(worker_nr) and next
> >        worker = Unicorn::Worker.new(worker_nr)
> >        before_fork.call(self, worker)
> > -      if pid = fork
> > -        @workers[pid] = worker
> > -        worker.atfork_parent
> > +
> > +      pid = if @worker_exec
> > +        worker_spawn(worker)
> >        else
> > -        after_fork_internal
> > -        worker_loop(worker)
> > -        exit
> > +        fork do
> > +          after_fork_internal
> > +          worker_loop(worker)
> > +          exit
> > +        end
> 
> I prefer to avoid the block with fork.  The block deepens the
> stack for the running app, so it can affect GC efficiency.
> 
> Can be fixed in a separate patch...

That makes sense.  If you would like me to send a separate patch to fix
it, I can do that.

Thanks,
Jeremy

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add worker_exec configuration option V2
  2017-03-09 19:41  5%   ` [PATCH] Add worker_exec configuration option V2 Jeremy Evans
@ 2017-03-10 21:19 11%     ` Eric Wong
  2017-03-11  5:26  6%       ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-10 21:19 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Here's V2 of the patch, which I think should address all of the issues
> you pointed out.

Thanks.  Pushed to git://80x24.org/unicorn worker_exec I can add
tests, later, or you can.  I also had some FreeBSD test fixes
(which might apply to OpenBSD) on a VM somewhere which I'll Cc:
you on: there was also just SO_KEEPALIVE fix I posted:

  https://bogomips.org/unicorn-public/20170310203431.28067-1-e@80x24.org/raw

>      worker_nr = -1
>      until (worker_nr += 1) == @worker_processes
>        @workers.value?(worker_nr) and next
>        worker = Unicorn::Worker.new(worker_nr)
>        before_fork.call(self, worker)
> -      if pid = fork
> -        @workers[pid] = worker
> -        worker.atfork_parent
> +
> +      pid = if @worker_exec
> +        worker_spawn(worker)
>        else
> -        after_fork_internal
> -        worker_loop(worker)
> -        exit
> +        fork do
> +          after_fork_internal
> +          worker_loop(worker)
> +          exit
> +        end

I prefer to avoid the block with fork.  The block deepens the
stack for the running app, so it can affect GC efficiency.

Can be fixed in a separate patch...

^ permalink raw reply	[relevance 11%]

* Re: [PATCH] Add worker_exec configuration option V2
  2017-03-08 20:02  4% ` Eric Wong
  2017-03-09  4:52  6%   ` Jeremy Evans
@ 2017-03-09 19:41  5%   ` Jeremy Evans
  2017-03-10 21:19 11%     ` Eric Wong
  1 sibling, 1 reply; 200+ results
From: Jeremy Evans @ 2017-03-09 19:41 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 03/08 08:02, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > The worker_exec configuration option makes all worker processes
> > exec after forking.  This initializes the worker processes with
> > separate memory layouts, defeating address space discovery
> > attacks on operating systems supporting address space layout
> > randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
> > Solaris.
> > 
> > Support for execing workers is very similar to support for reexecing
> > the master process.  The main difference is the worker's to_i and
> > master pipes also need to be inherited after worker exec just as the
> > listening sockets need to be inherited after reexec.
> 
> Thanks, this seems like an acceptable feature.

Eric,

Here's V2 of the patch, which I think should address all of the issues
you pointed out.

Thanks,
Jeremy

From 8e68bf8c6a8b91704f31dd9b9a62e6f2e330e380 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Wed, 8 Mar 2017 10:19:02 -0800
Subject: [PATCH] Add worker_exec configuration option

The worker_exec configuration option makes all worker processes
exec after forking.  This initializes the worker processes with
separate memory layouts, defeating address space discovery
attacks on operating systems supporting address space layout
randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
Solaris.

Support for execing workers is very similar to support for reexecing
the master process.  The main difference is the worker's to_i and
master pipes also need to be inherited after worker exec just as the
listening sockets need to be inherited after reexec.

Because execing working is similar to reexecing the master, this
extracts a couple of methods from reexec (listener_sockets and
close_sockets_on_exec), so they can be reused in worker_spawn.
---
 lib/unicorn/configurator.rb | 10 ++++++
 lib/unicorn/http_server.rb  | 83 ++++++++++++++++++++++++++++++++++-----------
 lib/unicorn/worker.rb       |  5 +--
 3 files changed, 77 insertions(+), 21 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7ed5ffa..f69f220 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -53,6 +53,7 @@ class Unicorn::Configurator
         server.logger.info("worker=#{worker.nr} ready")
       },
     :pid => nil,
+    :worker_exec => false,
     :preload_app => false,
     :check_client_connection => false,
     :rewindable_input => true, # for Rack 2.x: (Rack::VERSION[0] <= 1),
@@ -239,6 +240,15 @@ def timeout(seconds)
     set[:timeout] = seconds > max ? max : seconds
   end
 
+  # Whether to exec in each worker process after forking.  This changes the
+  # memory layout of each worker process, which is a security feature designed
+  # to defeat possible address space discovery attacks.  Note that using
+  # worker_exec only makes sense if you are not preloading the application,
+  # and will result in higher memory usage.
+  def worker_exec(bool)
+    set_bool(:worker_exec, bool)
+  end
+
   # sets the current number of worker_processes to +nr+.  Each worker
   # process will serve exactly one client at a time.  You can
   # increment or decrement this value at runtime by sending SIGTTIN
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ef897ad..a5bd2c4 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user
-  attr_writer   :after_worker_exit, :after_worker_ready
+  attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
@@ -105,6 +105,14 @@ def initialize(app, options = {})
     # list of signals we care about and trap in master.
     @queue_sigs = [
       :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
+
+    @worker_data = if worker_data = ENV['UNICORN_WORKER']
+      worker_data = worker_data.split(',').map!(&:to_i)
+      worker_data[1] = worker_data.slice!(1..2).map do |i|
+        Kgio::Pipe.for_fd(i)
+      end
+      worker_data
+    end
   end
 
   # Runs the thing.  Returns self so you can run join on it
@@ -113,7 +121,7 @@ def start
     # this pipe is used to wake us up from select(2) in #join when signals
     # are trapped.  See trap_deferred.
     @self_pipe.replace(Unicorn.pipe)
-    @master_pid = $$
+    @master_pid = @worker_data ? Process.ppid : $$
 
     # setup signal handlers before writing pid file in case people get
     # trigger happy and send signals as soon as the pid file exists.
@@ -430,11 +438,7 @@ def reexec
     end
 
     @reexec_pid = fork do
-      listener_fds = {}
-      LISTENERS.each do |sock|
-        sock.close_on_exec = false
-        listener_fds[sock.fileno] = sock
-      end
+      listener_fds = listener_sockets
       ENV['UNICORN_FD'] = listener_fds.keys.join(',')
       Dir.chdir(START_CTX[:cwd])
       cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
@@ -442,12 +446,7 @@ def reexec
       # avoid leaking FDs we don't know about, but let before_exec
       # unset FD_CLOEXEC, if anything else in the app eventually
       # relies on FD inheritence.
-      (3..1024).each do |io|
-        next if listener_fds.include?(io)
-        io = IO.for_fd(io) rescue next
-        io.autoclose = false
-        io.close_on_exec = true
-      end
+      close_sockets_on_exec(listener_fds)
 
       # exec(command, hash) works in at least 1.9.1+, but will only be
       # required in 1.9.4/2.0.0 at earliest.
@@ -459,6 +458,40 @@ def reexec
     proc_name 'master (old)'
   end
 
+  def worker_spawn(worker)
+    listener_fds = listener_sockets
+    env = {}
+    env['UNICORN_FD'] = listener_fds.keys.join(',')
+
+    listener_fds[worker.to_io.fileno] = worker.to_io
+    listener_fds[worker.master.fileno] = worker.master
+
+    worker_info = [worker.nr, worker.to_io.fileno, worker.master.fileno]
+    env['UNICORN_WORKER'] = worker_info.join(',')
+
+    close_sockets_on_exec(listener_fds)
+
+    Process.spawn(env, START_CTX[0], *START_CTX[:argv], listener_fds)
+  end
+
+  def listener_sockets
+    listener_fds = {}
+    LISTENERS.each do |sock|
+      sock.close_on_exec = false
+      listener_fds[sock.fileno] = sock
+    end
+    listener_fds
+  end
+
+  def close_sockets_on_exec(sockets)
+    (3..1024).each do |io|
+      next if sockets.include?(io)
+      io = IO.for_fd(io) rescue next
+      io.autoclose = false
+      io.close_on_exec = true
+    end
+  end
+
   # forcibly terminate all workers that haven't checked in in timeout seconds.  The timeout is implemented using an unlinked File
   def murder_lazy_workers
     next_sleep = @timeout - 1
@@ -495,19 +528,31 @@ def after_fork_internal
   end
 
   def spawn_missing_workers
+    if @worker_data
+      worker = Unicorn::Worker.new(*@worker_data)
+      after_fork_internal
+      worker_loop(worker)
+      exit
+    end
+
     worker_nr = -1
     until (worker_nr += 1) == @worker_processes
       @workers.value?(worker_nr) and next
       worker = Unicorn::Worker.new(worker_nr)
       before_fork.call(self, worker)
-      if pid = fork
-        @workers[pid] = worker
-        worker.atfork_parent
+
+      pid = if @worker_exec
+        worker_spawn(worker)
       else
-        after_fork_internal
-        worker_loop(worker)
-        exit
+        fork do
+          after_fork_internal
+          worker_loop(worker)
+          exit
+        end
       end
+
+      @workers[pid] = worker
+      worker.atfork_parent
     end
     rescue => e
       @logger.error(e) rescue nil
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index e22c1bf..8bbac5e 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -12,18 +12,19 @@ class Unicorn::Worker
   # :stopdoc:
   attr_accessor :nr, :switched
   attr_reader :to_io # IO.select-compatible
+  attr_reader :master
 
   PER_DROP = Raindrops::PAGE_SIZE / Raindrops::SIZE
   DROPS = []
 
-  def initialize(nr)
+  def initialize(nr, pipe=nil)
     drop_index = nr / PER_DROP
     @raindrop = DROPS[drop_index] ||= Raindrops.new(PER_DROP)
     @offset = nr % PER_DROP
     @raindrop[@offset] = 0
     @nr = nr
     @switched = false
-    @to_io, @master = Unicorn.pipe
+    @to_io, @master = pipe || Unicorn.pipe
   end
 
   def atfork_child # :nodoc:
-- 
2.11.0


^ permalink raw reply related	[relevance 5%]

* Re: [PATCH] Add worker_exec configuration option
  2017-03-09  4:52  6%   ` Jeremy Evans
@ 2017-03-09 13:57  6%     ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-09 13:57 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Thanks for your detailed review.  I will work on an updated patch and
> try to send it tomorrow or Friday.

No problem and no rush; I might be out a bit the next few days.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add worker_exec configuration option
  2017-03-08 20:02  4% ` Eric Wong
@ 2017-03-09  4:52  6%   ` Jeremy Evans
  2017-03-09 13:57  6%     ` Eric Wong
  2017-03-09 19:41  5%   ` [PATCH] Add worker_exec configuration option V2 Jeremy Evans
  1 sibling, 1 reply; 200+ results
From: Jeremy Evans @ 2017-03-09  4:52 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 03/08 08:02, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > The worker_exec configuration option makes all worker processes
> > exec after forking.  This initializes the worker processes with
> > separate memory layouts, defeating address space discovery
> > attacks on operating systems supporting address space layout
> > randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
> > Solaris.
> > 
> > Support for execing workers is very similar to support for reexecing
> > the master process.  The main difference is the worker's to_i and
> > master pipes also need to be inherited after worker exec just as the
> > listening sockets need to be inherited after reexec.
> 
> Thanks, this seems like an acceptable feature.

<snip>

Eric,

Thanks for your detailed review.  I will work on an updated patch and
try to send it tomorrow or Friday.

Thanks,
Jeremy

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add worker_exec configuration option
  2017-03-08 18:44  3% [PATCH] Add worker_exec configuration option Jeremy Evans
@ 2017-03-08 20:02  4% ` Eric Wong
  2017-03-09  4:52  6%   ` Jeremy Evans
  2017-03-09 19:41  5%   ` [PATCH] Add worker_exec configuration option V2 Jeremy Evans
  0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2017-03-08 20:02 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> The worker_exec configuration option makes all worker processes
> exec after forking.  This initializes the worker processes with
> separate memory layouts, defeating address space discovery
> attacks on operating systems supporting address space layout
> randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
> Solaris.
> 
> Support for execing workers is very similar to support for reexecing
> the master process.  The main difference is the worker's to_i and
> master pipes also need to be inherited after worker exec just as the
> listening sockets need to be inherited after reexec.

Thanks, this seems like an acceptable feature.
Some comments below....

> +++ b/lib/unicorn/http_server.rb
> @@ -105,6 +105,14 @@ def initialize(app, options = {})
>      # list of signals we care about and trap in master.
>      @queue_sigs = [
>        :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
> +
> +    if worker_nr = ENV['UNICORN_WORKER_NR']
> +      @worker_nr = worker_nr.to_i
> +      @master_pid = Process.ppid
> +    else
> +      @master_pid = $$
> +    end
> +
>    end

We should nil @worker_nr in the normal case to avoid introducing
new warnings on uninitialized ivars.

>    # Runs the thing.  Returns self so you can run join on it
> @@ -113,7 +121,6 @@ def start
>      # this pipe is used to wake us up from select(2) in #join when signals
>      # are trapped.  See trap_deferred.
>      @self_pipe.replace(Unicorn.pipe)
> -    @master_pid = $$

Any particular reason setting @master_pid was moved to initialize?

Maybe we (or anybody else) has tests or other code which do:

	  foo = Unicorn::HttpServer.new ...
	  pid = fork do
	    foo.start.join
	  end

Which would be broken by the movement from start => initialize

Again, I don't consider this a stable API and don't want people
depending on our internals; but I will also avoid breaking
existing behavior whenever possible.

> @@ -459,6 +466,39 @@ def reexec
>      proc_name 'master (old)'
>    end
>  
> +  def worker_exec(worker)
> +    listener_fds = {}
> +    LISTENERS.each do |sock|
> +      sock.close_on_exec = false
> +      listener_fds[sock.fileno] = sock
> +    end
> +    ENV['UNICORN_FD'] = listener_fds.keys.join(',')
> +    ENV['UNICORN_WORKER_NR'] = worker.nr.to_s

> +    ENV['UNICORN_WORKER_TO_IO'] = worker.to_io.fileno.to_s
> +    ENV['UNICORN_WORKER_MASTER'] = worker.master.fileno.to_s

We can cram these two FDs into one UNICORN_WORKER_FD
comma-delimited variable like UNICORN_FD.  The underlying
getenv/setenv/putenv functions are all O(n), so keeping env
smaller would be nice.

And we can even avoid modifying ENV on Ruby 1.9+ by
using exec(env, cmd, options...) or even Process.spawn (see below)

> +    Dir.chdir(START_CTX[:cwd])
> +    cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
> +
> +    listener_fds[worker.to_io.fileno] = worker.to_io
> +    listener_fds[worker.master.fileno] = worker.master
> +
> +    # avoid leaking FDs we don't know about, but let before_exec
> +    # unset FD_CLOEXEC, if anything else in the app eventually
> +    # relies on FD inheritence.

Comment seems blindly copy+pasted, since before_exec does not
seem to exist, for this...

> +    (3..1024).each do |io|
> +      next if listener_fds.include?(io)
> +      io = IO.for_fd(io) rescue next
> +      io.autoclose = false
> +      io.close_on_exec = true
> +    end

I suppose we should probably use Process.getrlimit, here,
maybe skip the loop entirely for Ruby 2.0.0+ (which made
close-on-exec the default).

This will be a separate patch...


Since there's no before_exec hook as with SIGUSR2, I think we
can do all of this setup in master and use Process.spawn instead
of fork+exec explicitly.  Process.spawn transparently allows the use
of vfork+exec to avoid fork overhead on some systems.

> +    # exec(command, hash) works in at least 1.9.1+, but will only be
> +    # required in 1.9.4/2.0.0 at earliest.

I guess it's time to update that comment, since 1.9.4 never
happened :)

> +    cmd << listener_fds
> +    logger.info "executing worker #{cmd.inspect} (in #{Dir.pwd})"
> +    exec(*cmd)
> +  end
> +
>    # forcibly terminate all workers that haven't checked in in timeout seconds.  The timeout is implemented using an unlinked File
>    def murder_lazy_workers
>      next_sleep = @timeout - 1
> @@ -495,6 +535,12 @@ def after_fork_internal
>    end
>  
>    def spawn_missing_workers
> +    if @worker_nr
> +      worker = Unicorn::Worker.new(@worker_nr, [ENV["UNICORN_WORKER_TO_IO"], ENV["UNICORN_WORKER_MASTER"]].map{|i| Kgio::Pipe.for_fd(i.to_i)})

Please keep new code wrapped to 80 chars or less.  I'm still
using hardware that predates this project and my eyes will
only get worse with age.

> @@ -505,7 +551,11 @@ def spawn_missing_workers
>          worker.atfork_parent
>        else
>          after_fork_internal
> -        worker_loop(worker)
> +        if @worker_exec
> +          worker_exec(worker)
> +        else
> +          worker_loop(worker)
> +        end

I prefer the ternary operator for simple cases like this.  It is
more compact in terms of screen space and VM bytecode.  But this
may not be changed at all if we switch to using Process.spawn,
instead.

The rest looks fine.  Thanks.

^ permalink raw reply	[relevance 4%]

* [PATCH] Add worker_exec configuration option
@ 2017-03-08 18:44  3% Jeremy Evans
  2017-03-08 20:02  4% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-03-08 18:44 UTC (permalink / raw)
  To: unicorn-public

The worker_exec configuration option makes all worker processes
exec after forking.  This initializes the worker processes with
separate memory layouts, defeating address space discovery
attacks on operating systems supporting address space layout
randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
Solaris.

Support for execing workers is very similar to support for reexecing
the master process.  The main difference is the worker's to_i and
master pipes also need to be inherited after worker exec just as the
listening sockets need to be inherited after reexec.
---
 lib/unicorn/configurator.rb | 10 ++++++++
 lib/unicorn/http_server.rb  | 56 ++++++++++++++++++++++++++++++++++++++++++---
 lib/unicorn/worker.rb       |  5 ++--
 3 files changed, 66 insertions(+), 5 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7ed5ffa..f69f220 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -53,6 +53,7 @@ class Unicorn::Configurator
         server.logger.info("worker=#{worker.nr} ready")
       },
     :pid => nil,
+    :worker_exec => false,
     :preload_app => false,
     :check_client_connection => false,
     :rewindable_input => true, # for Rack 2.x: (Rack::VERSION[0] <= 1),
@@ -239,6 +240,15 @@ def timeout(seconds)
     set[:timeout] = seconds > max ? max : seconds
   end
 
+  # Whether to exec in each worker process after forking.  This changes the
+  # memory layout of each worker process, which is a security feature designed
+  # to defeat possible address space discovery attacks.  Note that using
+  # worker_exec only makes sense if you are not preloading the application,
+  # and will result in higher memory usage.
+  def worker_exec(bool)
+    set_bool(:worker_exec, bool)
+  end
+
   # sets the current number of worker_processes to +nr+.  Each worker
   # process will serve exactly one client at a time.  You can
   # increment or decrement this value at runtime by sending SIGTTIN
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ef897ad..d68d7d8 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user
-  attr_writer   :after_worker_exit, :after_worker_ready
+  attr_writer   :after_worker_exit, :after_worker_ready, :worker_exec
 
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
@@ -105,6 +105,14 @@ def initialize(app, options = {})
     # list of signals we care about and trap in master.
     @queue_sigs = [
       :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
+
+    if worker_nr = ENV['UNICORN_WORKER_NR']
+      @worker_nr = worker_nr.to_i
+      @master_pid = Process.ppid
+    else
+      @master_pid = $$
+    end
+
   end
 
   # Runs the thing.  Returns self so you can run join on it
@@ -113,7 +121,6 @@ def start
     # this pipe is used to wake us up from select(2) in #join when signals
     # are trapped.  See trap_deferred.
     @self_pipe.replace(Unicorn.pipe)
-    @master_pid = $$
 
     # setup signal handlers before writing pid file in case people get
     # trigger happy and send signals as soon as the pid file exists.
@@ -459,6 +466,39 @@ def reexec
     proc_name 'master (old)'
   end
 
+  def worker_exec(worker)
+    listener_fds = {}
+    LISTENERS.each do |sock|
+      sock.close_on_exec = false
+      listener_fds[sock.fileno] = sock
+    end
+    ENV['UNICORN_FD'] = listener_fds.keys.join(',')
+    ENV['UNICORN_WORKER_NR'] = worker.nr.to_s
+    ENV['UNICORN_WORKER_TO_IO'] = worker.to_io.fileno.to_s
+    ENV['UNICORN_WORKER_MASTER'] = worker.master.fileno.to_s
+    Dir.chdir(START_CTX[:cwd])
+    cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
+
+    listener_fds[worker.to_io.fileno] = worker.to_io
+    listener_fds[worker.master.fileno] = worker.master
+
+    # avoid leaking FDs we don't know about, but let before_exec
+    # unset FD_CLOEXEC, if anything else in the app eventually
+    # relies on FD inheritence.
+    (3..1024).each do |io|
+      next if listener_fds.include?(io)
+      io = IO.for_fd(io) rescue next
+      io.autoclose = false
+      io.close_on_exec = true
+    end
+
+    # exec(command, hash) works in at least 1.9.1+, but will only be
+    # required in 1.9.4/2.0.0 at earliest.
+    cmd << listener_fds
+    logger.info "executing worker #{cmd.inspect} (in #{Dir.pwd})"
+    exec(*cmd)
+  end
+
   # forcibly terminate all workers that haven't checked in in timeout seconds.  The timeout is implemented using an unlinked File
   def murder_lazy_workers
     next_sleep = @timeout - 1
@@ -495,6 +535,12 @@ def after_fork_internal
   end
 
   def spawn_missing_workers
+    if @worker_nr
+      worker = Unicorn::Worker.new(@worker_nr, [ENV["UNICORN_WORKER_TO_IO"], ENV["UNICORN_WORKER_MASTER"]].map{|i| Kgio::Pipe.for_fd(i.to_i)})
+      worker_loop(worker)
+      exit
+    end
+
     worker_nr = -1
     until (worker_nr += 1) == @worker_processes
       @workers.value?(worker_nr) and next
@@ -505,7 +551,11 @@ def spawn_missing_workers
         worker.atfork_parent
       else
         after_fork_internal
-        worker_loop(worker)
+        if @worker_exec
+          worker_exec(worker)
+        else
+          worker_loop(worker)
+        end
         exit
       end
     end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index e22c1bf..8bbac5e 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -12,18 +12,19 @@ class Unicorn::Worker
   # :stopdoc:
   attr_accessor :nr, :switched
   attr_reader :to_io # IO.select-compatible
+  attr_reader :master
 
   PER_DROP = Raindrops::PAGE_SIZE / Raindrops::SIZE
   DROPS = []
 
-  def initialize(nr)
+  def initialize(nr, pipe=nil)
     drop_index = nr / PER_DROP
     @raindrop = DROPS[drop_index] ||= Raindrops.new(PER_DROP)
     @offset = nr % PER_DROP
     @raindrop[@offset] = 0
     @nr = nr
     @switched = false
-    @to_io, @master = Unicorn.pipe
+    @to_io, @master = pipe || Unicorn.pipe
   end
 
   def atfork_child # :nodoc:
-- 
2.11.0


^ permalink raw reply related	[relevance 3%]

* Re: [PATCH 0/3] TCP_INFO check_client_connection followups
  2017-03-08  6:02  5% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
@ 2017-03-08 10:14  0% ` Simon Eskildsen
  0 siblings, 0 replies; 200+ results
From: Simon Eskildsen @ 2017-03-08 10:14 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

This looks great Eric. Thanks for taking this to the finish line!

On Wed, Mar 8, 2017 at 1:02 AM, Eric Wong <e@80x24.org> wrote:
> This series goes on top of Simon's excellent work to get
> a TCP_INFO-based check_client_connection working under Linux.
>
> First off, add a test extracted from one of Simon's messages;
> then revert the signature changes to existing methods to
> avoid breaking tmm1/gctools, and finally add a more portable
> fallback for Ruby 2.2+ users (tested on FreeBSD).
>
> Further work will improve portability of raindrops for FreeBSD
> (and maybe other *BSDs incidentally, too).  That will be sent to
> raindrops-public@bogomips => https://bogomips.org/raindrops-public/
>
> Eric Wong (3):
>   new test for check_client_connection
>   revert signature change to HttpServer#process_client
>   support "struct tcp_info" on non-Linux and Ruby 2.2+
>
>  lib/unicorn/http_request.rb  | 42 +++++++++++++++++++----
>  lib/unicorn/http_server.rb   |  6 ++--
>  lib/unicorn/oob_gc.rb        |  4 +--
>  lib/unicorn/socket_helper.rb | 16 +++++++--
>  test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
>  test/unit/test_request.rb    | 28 +++++++--------
>  6 files changed, 150 insertions(+), 27 deletions(-)
>  create mode 100644 test/unit/test_ccc.rb
>
> Also pushed to the ccc-tcp-v3 branch:
>
> The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:
>
>   t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)
>
> are available in the git repository at:
>
>   git://bogomips.org/unicorn ccc-tcp-v3
>
> for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:
>
>   support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)
>
> ----------------------------------------------------------------
> Eric Wong (3):
>       new test for check_client_connection
>       revert signature change to HttpServer#process_client
>       support "struct tcp_info" on non-Linux and Ruby 2.2+
>
> Simon Eskildsen (1):
>       check_client_connection: use tcp state on linux
>
>  lib/unicorn/http_request.rb  | 73 ++++++++++++++++++++++++++++++++++++---
>  lib/unicorn/socket_helper.rb | 16 +++++++--
>  test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 163 insertions(+), 7 deletions(-)
>  create mode 100644 test/unit/test_ccc.rb
>
> --
> EW

^ permalink raw reply	[relevance 0%]

* [PATCH] doc: add version annotations for new features
  2017-03-08  7:29 13%   ` Eric Wong
@ 2017-03-08  7:44  9%     ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-03-08  7:44 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

I suppose this is a good idea, too.

Will merge before the 5.3.0 RCs and release (soonish, I think...)

-------8<--------
Subject: [PATCH] doc: add version annotations for new features

We will inevitably have people running old unicorn versions
for many years to come; but they may be reading the latest
documentation online.

Annotate when the new features (will) appear to avoid misleading
users on old versions.
---
 lib/unicorn/configurator.rb | 2 ++
 lib/unicorn/worker.rb       | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7ed5ffa..3eb8c22 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -186,6 +186,8 @@ def after_worker_exit(*args, &block)
   #
   # Do not use Configurator#user if you rely on changing users in the
   # after_worker_ready hook.
+  #
+  # after_worker_ready is only available in unicorn 5.3.0+
   def after_worker_ready(*args, &block)
     set_hook(:after_worker_ready, block_given? ? block : args[0])
   end
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index e22c1bf..2f5b6a6 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -124,7 +124,10 @@ def close # :nodoc:
   # Any and all errors raised within this method will be propagated
   # directly back to the caller (usually the +after_fork+ hook.
   # These errors commonly include ArgumentError for specifying an
-  # invalid user/group and Errno::EPERM for insufficient privileges
+  # invalid user/group and Errno::EPERM for insufficient privileges.
+  #
+  # chroot support is only available in unicorn 5.3.0+
+  # user and group switching appeared in unicorn 0.94.0 (2009-11-05)
   def user(user, group = nil, chroot = false)
     # we do not protect the caller, checking Process.euid == 0 is
     # insufficient because modern systems have fine-grained
-- 
EW

^ permalink raw reply related	[relevance 9%]

* Re: Patch: Add after_worker_ready configuration option V2
  2017-02-23 20:29 13% ` Eric Wong
@ 2017-03-08  7:29 13%   ` Eric Wong
  2017-03-08  7:44  9%     ` [PATCH] doc: add version annotations for new features Eric Wong
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-08  7:29 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Eric Wong <e@80x24.org> wrote:
> Thanks, I've pushed this and the chroot patch out to the
> 'chroot' branch.  Willl wait a bit for comments from others
> before merging into 'master'.

No comments, so no objections, so merged and pushed to 'master'
as commit ff13ad38ba9f83e0dd298be451aac7c75145d33b

    Merge remote-tracking branch 'origin/chroot'

    * origin/chroot:
      Add after_worker_ready configuration option
      Add support for chroot to Worker#user

Thanks.

^ permalink raw reply	[relevance 13%]

* [PATCH 0/3] TCP_INFO check_client_connection followups
@ 2017-03-08  6:02  5% Eric Wong
  2017-03-08 10:14  0% ` Simon Eskildsen
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-03-08  6:02 UTC (permalink / raw)
  To: unicorn-public; +Cc: Simon Eskildsen

This series goes on top of Simon's excellent work to get
a TCP_INFO-based check_client_connection working under Linux.

First off, add a test extracted from one of Simon's messages;
then revert the signature changes to existing methods to
avoid breaking tmm1/gctools, and finally add a more portable
fallback for Ruby 2.2+ users (tested on FreeBSD).

Further work will improve portability of raindrops for FreeBSD
(and maybe other *BSDs incidentally, too).  That will be sent to
raindrops-public@bogomips => https://bogomips.org/raindrops-public/

Eric Wong (3):
  new test for check_client_connection
  revert signature change to HttpServer#process_client
  support "struct tcp_info" on non-Linux and Ruby 2.2+

 lib/unicorn/http_request.rb  | 42 +++++++++++++++++++----
 lib/unicorn/http_server.rb   |  6 ++--
 lib/unicorn/oob_gc.rb        |  4 +--
 lib/unicorn/socket_helper.rb | 16 +++++++--
 test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
 test/unit/test_request.rb    | 28 +++++++--------
 6 files changed, 150 insertions(+), 27 deletions(-)
 create mode 100644 test/unit/test_ccc.rb

Also pushed to the ccc-tcp-v3 branch:

The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:

  t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)

are available in the git repository at:

  git://bogomips.org/unicorn ccc-tcp-v3

for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:

  support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)

----------------------------------------------------------------
Eric Wong (3):
      new test for check_client_connection
      revert signature change to HttpServer#process_client
      support "struct tcp_info" on non-Linux and Ruby 2.2+

Simon Eskildsen (1):
      check_client_connection: use tcp state on linux

 lib/unicorn/http_request.rb  | 73 ++++++++++++++++++++++++++++++++++++---
 lib/unicorn/socket_helper.rb | 16 +++++++--
 test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 163 insertions(+), 7 deletions(-)
 create mode 100644 test/unit/test_ccc.rb

-- 
EW

^ permalink raw reply	[relevance 5%]

* Re: Patch: Add after_worker_ready configuration option V2
  2017-02-23 18:49  7% Patch: Add after_worker_ready configuration option V2 Jeremy Evans
@ 2017-02-23 20:29 13% ` Eric Wong
  2017-03-08  7:29 13%   ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-23 20:29 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Here's V2 of the after_worker_ready patch.  This adds some more
> documentation, and tries to give a better description of the
> advantages in the commit message.

Thanks, I've pushed this and the chroot patch out to the
'chroot' branch.  Willl wait a bit for comments from others
before merging into 'master'.

The following changes since commit c8f06be298d667ba85573668ee916680a258c2c7:

  Fix code example in after_worker_exit documentation (2017-02-23 19:26:30 +0000)

are available in the git repository at:

  git://bogomips.org/unicorn chroot

for you to fetch changes up to d322345251e15125df896bb8f0a5b53b49a1bf3f:

  Add after_worker_ready configuration option (2017-02-23 20:23:44 +0000)

----------------------------------------------------------------
Jeremy Evans (2):
      Add support for chroot to Worker#user
      Add after_worker_ready configuration option

 lib/unicorn/configurator.rb | 22 ++++++++++++++++++++++
 lib/unicorn/http_server.rb  |  4 ++--
 lib/unicorn/worker.rb       | 13 ++++++++++---
 3 files changed, 34 insertions(+), 5 deletions(-)

^ permalink raw reply	[relevance 13%]

* Re: Patch: Fix code example in after_worker_exit documentation
  @ 2017-02-23 20:21  5% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-02-23 20:21 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Here's a separate commit for the documentation fix.

Thanks, pushed as c8f06be298d667ba85573668ee916680a258c2c7
to 'master'.  I added a "Fixes: " tag to link it the original.

> From 6efb4b8e07a0459180579b72fac5003e424837c0 Mon Sep 17 00:00:00 2001

Hint: you can omit the above line, and add "----8<----" so the
applier can use "git am --scissors" without having to just part
of the message.  It's no big deal, but I guess I'll do what I
can to promote --scissors as it's not too well-known.
(and usually, the entire email is the patch for "git am")

^ permalink raw reply	[relevance 5%]

* Re: Patch: Add after_worker_ready configuration option
  2017-02-23  9:23  4%     ` Eric Wong
@ 2017-02-23 20:00  5%       ` Jeremy Evans
  0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2017-02-23 20:00 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 02/23 09:23, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > On 02/23 02:32, Eric Wong wrote:
> > > Jeremy Evans <code@jeremyevans.net> wrote:
> > > Anyways, I'm somewhat inclined to accept this as well, but will
> > > think about it a bit, too.
> > > 
> > > One problem I have with this change is that it requires users
> > > read more documentation and know more caveats to use.
> >  
> > It's unfortunate, but I think it's necessary in this case. Unicorn's
> > default behavior works for most users, but this makes it so you
> > don't have to override HttpServer#init_worker_process to get similar
> > functionality.
> 
> Yeah.  Unfortunately, with every option comes more
> support/maintenance overhead.  So I'm trying to make sure we've
> explored every possible avenue before adding this new option.

I agree.  If I knew of a way to get this functionality without
overriding a private method, I would use it.  I'm not sure there is a
way currently, which I why I worked on the patch.

> > >     So I'm wondering if there can be some of that which could be
> > >     trimmed out someday or made optional, somehow.
> > 
> > I'm not sure how open you are to a plugin-like system for Unicorn, where
> > you can have separate files for separate features, and users can choose
> > which features the want to use.  This limits memory usage so that you
> > don't pay a memory cost for features you don't use.  This is the
> > approach used by Sequel, Roda, and Rodauth (other libraries I maintain),
> > and if you are interested in a similar system for Unicorn, I would be
> > happy to work on one.
> 
> Not really.  As I've stated many times in the past, unicorn is
> just one of many servers available for Rack users; so I've been
> against people building too many dependencies on unicorn, that
> comes at the expense of other servers.
 
True, but there are features that only unicorn provides.  If you want
those features, you have to use unicorn anyway. 

> What I had in mind was having a shim process which does socket
> setup, user switching, etc. before exec-ing unicorn in cases
> where a systemd-like spawner is not used.  But it's probably too
> small to make an impact on real apps anyways, since Ruby still
> has tons of methods which never get called, either.

I could see that decreasing memory usage slightly, but I agree it's
unlikely to make a significant impact.

> > I do have a couple of additional patches I plan to send, but I think it
> > may be better to ask here first.
> > 
> > One is that SIGUSR1 handling doesn't work well with chroot if the log
> > file is outside of the chroot, since the worker process may not have
> > access to reopen the file.  One simple fix is to reopen the logs in
> > the master process, then send SIGQUIT instead of SIGUSR1 to the child
> > processes.  Are you open to a patch that does that?
> 
> Right now, the worker exits with some log spew if reopening
> logs fails, and the worker respawns anyways, correct?
> 
> Perhaps that mapping can be made automatic if chroot is
> enabled.  I wonder if that's too much magic...
> Or the log spew could be suppressed when the error
> is ENOENT and chroot is enabled.
> 
> The sysadmin could also send SIGHUP to reload everything
> and respawn all children.
> 
> So, I'm wondering if using SIGHUP is too much of a burden, or if
> automatic USR1 => QUIT mapping for chroot users is too magical.

SIGHUP is actually fine.  I didn't realize it reopened log files
in the master process, but apparently it does, so I can just use
SIGHUP instead of SIGUSR1.

> > The second feature is more exotic security feature called fork+exec.
> > The current issue with unicorn's direct forking of children is that
> > forked children share the memory layout of the server.  This is good
> > from a memory usage standpoint due to CoW memory.  However, if there
> > is a memory vulnerability in the unicorn application, this makes it
> > easier to exploit, since if the a unicorn worker process crashes, the
> > master process will fork a child with pretty much the same memory
> > layout.  This allows address space discovery attacks.  Using fork+exec,
> > each child process would have a unique address space for ASLR, on
> > most Unix-like systems (Linux, MacOS X, NetBSD, OpenBSD, Solaris).
> > There are additional security advantages from fork+exec on OpenBSD, but
> > all Unix-like systems that use ASLR can benefit from the unique
> > memory layout per process.
> 
> It depends how intrusive the implementation is...

That I don't know yet.  I haven't implemented fork+exec before, so I'm
not sure how many changes are needed. I probably won't be able to work
on this right away.

> Is this merely calling exec in the worker?
> 
> I'm not sure if your use of "fork+exec" is merely the two
> well-known syscalls combined, or if there's something else;
> I'll assume the former, since that's enough to have a
> unique address space for Ruby bytecode.
> 
> Can this be done in an after_fork/after_worker_ready hook?
> 
> Socket inheritance can be done the same way it handles USR2
> upgrades and systemd socket inheritance.

I expect in addition to socket inheritance, we will need to handle pipe
inheritance for the @to_io and @master pipes in the worker.

Thanks,
Jeremy

^ permalink raw reply	[relevance 5%]

* Patch: Add after_worker_ready configuration option V2
@ 2017-02-23 18:49  7% Jeremy Evans
  2017-02-23 20:29 13% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-23 18:49 UTC (permalink / raw)
  To: unicorn-public

Here's V2 of the after_worker_ready patch.  This adds some more
documentation, and tries to give a better description of the
advantages in the commit message.

From cbc6fe845ade8946b7db2ecd2b86a0bd8e18bbb8 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Tue, 21 Feb 2017 16:33:09 -0800
Subject: [PATCH] Add after_worker_ready configuration option

This adds a hook that is called after the application has
been loaded by the worker process, directly before it starts
accepting requests.  This hook is necessary if your application
needs to gain access to resources during initialization,
and then drop privileges before serving requests.

This is especially useful in conjunction with chroot support
so the app can load all the normal ruby libraries it needs
to function, and then chroot before accepting requests.

If you are preloading the app, it's possible to drop privileges
or chroot in after_fork, but if you are not preloading the app,
the only way to currently do this is to override the private
HttpServer#init_worker_process method, and overriding private
methods is a recipe for future breakage if the internals are
modified.  This hook allows for such functionality to be
supported and not break in future versions of Unicorn.
---
 lib/unicorn/configurator.rb | 22 ++++++++++++++++++++++
 lib/unicorn/http_server.rb  |  4 ++--
 2 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 1e2b6e4..7ed5ffa 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -49,6 +49,9 @@ class Unicorn::Configurator
           server.logger.error(m)
         end
       },
+    :after_worker_ready => lambda { |server, worker|
+        server.logger.info("worker=#{worker.nr} ready")
+      },
     :pid => nil,
     :preload_app => false,
     :check_client_connection => false,
@@ -172,6 +175,21 @@ def after_worker_exit(*args, &block)
     set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
   end
 
+  # sets after_worker_ready hook to a given block.  This block will be called
+  # by a worker process after it has been fully loaded, directly before it
+  # starts responding to requests:
+  #
+  #  after_worker_ready do |server,worker|
+  #    server.logger.info("worker #{worker.nr} ready, dropping privileges")
+  #    worker.user('username', 'groupname')
+  #  end
+  #
+  # Do not use Configurator#user if you rely on changing users in the
+  # after_worker_ready hook.
+  def after_worker_ready(*args, &block)
+    set_hook(:after_worker_ready, block_given? ? block : args[0])
+  end
+
   # sets before_fork got be a given Proc object.  This Proc
   # object will be called by the master process before forking
   # each worker.
@@ -569,6 +587,10 @@ def working_directory(path)
   # This switch will occur after calling the after_fork hook, and only
   # if the Worker#user method is not called in the after_fork hook
   # +group+ is optional and will not change if unspecified.
+  #
+  # Do not use Configurator#user if you rely on changing users in the
+  # after_worker_ready hook.  Instead, you need to call Worker#user
+  # directly in after_worker_ready.
   def user(user, group = nil)
     # raises ArgumentError on invalid user/group
     Etc.getpwnam(user)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index c2086cb..ef897ad 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user
-  attr_writer   :after_worker_exit
+  attr_writer   :after_worker_exit, :after_worker_ready
 
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
@@ -644,7 +644,7 @@ def worker_loop(worker)
     trap(:USR1) { nr = -65536 }
 
     ready = readers.dup
-    @logger.info "worker=#{worker.nr} ready"
+    @after_worker_ready.call(self, worker)
 
     begin
       nr < 0 and reopen_worker_logs(worker.nr)
-- 
2.11.0


^ permalink raw reply related	[relevance 7%]

* Patch: Add support for chroot to Worker#user V2
@ 2017-02-23 18:46  8% Jeremy Evans
  2017-04-05  3:44  6% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-23 18:46 UTC (permalink / raw)
  To: unicorn-public

Here's V2 of the chroot support patch.

This changes the commit message language, and supports chrooting to
a directory that isn't the current directory.

From 9bd82792d57f54a868c9a0e9af2bd452f3ef298d Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Tue, 21 Feb 2017 08:44:34 -0800
Subject: [PATCH] Add support for chroot to Worker#user

Any chrooting would need to happen inside Worker#user, because
you can't chroot until after you have parsed the list of groups,
and you must chroot before dropping root privileges.

chroot adds an extra layer of security, so that if the unicorn
process is exploited, file system access is limited to the chroot
directory instead of the entire file system.
---
 lib/unicorn/worker.rb | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 6748a2f..e22c1bf 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -111,9 +111,11 @@ def close # :nodoc:
   # In most cases, you should be using the Unicorn::Configurator#user
   # directive instead.  This method should only be used if you need
   # fine-grained control of exactly when you want to change permissions
-  # in your after_fork hooks.
+  # in your after_fork or after_worker_ready hooks, or if you want to
+  # use the chroot support.
   #
-  # Changes the worker process to the specified +user+ and +group+
+  # Changes the worker process to the specified +user+ and +group+,
+  # and chroots to the current working directory if +chroot+ is set.
   # This is only intended to be called from within the worker
   # process from the +after_fork+ hook.  This should be called in
   # the +after_fork+ hook after any privileged functions need to be
@@ -123,7 +125,7 @@ def close # :nodoc:
   # directly back to the caller (usually the +after_fork+ hook.
   # These errors commonly include ArgumentError for specifying an
   # invalid user/group and Errno::EPERM for insufficient privileges
-  def user(user, group = nil)
+  def user(user, group = nil, chroot = false)
     # we do not protect the caller, checking Process.euid == 0 is
     # insufficient because modern systems have fine-grained
     # capabilities.  Let the caller handle any and all errors.
@@ -134,6 +136,11 @@ def user(user, group = nil)
       Process.initgroups(user, gid)
       Process::GID.change_privilege(gid)
     end
+    if chroot
+      chroot = Dir.pwd if chroot == true
+      Dir.chroot(chroot)
+      Dir.chdir('/')
+    end
     Process.euid != uid and Process::UID.change_privilege(uid)
     @switched = true
   end
-- 
2.11.0


^ permalink raw reply related	[relevance 8%]

* Re: Patch: Add after_worker_ready configuration option
  2017-02-23  3:43  8%   ` Jeremy Evans
@ 2017-02-23  9:23  4%     ` Eric Wong
  2017-02-23 20:00  5%       ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-23  9:23 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:

<snip> acknowledging everything above.

> On 02/23 02:32, Eric Wong wrote:
> > Jeremy Evans <code@jeremyevans.net> wrote:
> > Anyways, I'm somewhat inclined to accept this as well, but will
> > think about it a bit, too.
> > 
> > One problem I have with this change is that it requires users
> > read more documentation and know more caveats to use.
>  
> It's unfortunate, but I think it's necessary in this case. Unicorn's
> default behavior works for most users, but this makes it so you
> don't have to override HttpServer#init_worker_process to get similar
> functionality.

Yeah.  Unfortunately, with every option comes more
support/maintenance overhead.  So I'm trying to make sure we've
explored every possible avenue before adding this new option.

<snip>

> > (*) Fwiw, I wasn't thrilled to add user switching to unicorn back
> >     in 2009/2010, as it should not need to run privileged ports.
> 
> I think had you not added it, there wouldn't be a need for a chroot
> patch, since otherwise one would assume any user that wanted to
> drop privileges would just run the necessary code themselves.
> 
> One thing to keep in mind is that if you did not add the user switching
> code, developers who need it would implement it themselves, and would
> likely miss things like initgroups.

Heh.  As I was mentioning above, every feature can come with
new, unintended consequences :>  But yeah, maybe push things
up to Ruby stdlib, or systemd...

> >     So I'm wondering if there can be some of that which could be
> >     trimmed out someday or made optional, somehow.
> 
> I'm not sure how open you are to a plugin-like system for Unicorn, where
> you can have separate files for separate features, and users can choose
> which features the want to use.  This limits memory usage so that you
> don't pay a memory cost for features you don't use.  This is the
> approach used by Sequel, Roda, and Rodauth (other libraries I maintain),
> and if you are interested in a similar system for Unicorn, I would be
> happy to work on one.

Not really.  As I've stated many times in the past, unicorn is
just one of many servers available for Rack users; so I've been
against people building too many dependencies on unicorn, that
comes at the expense of other servers.

What I had in mind was having a shim process which does socket
setup, user switching, etc. before exec-ing unicorn in cases
where a systemd-like spawner is not used.  But it's probably too
small to make an impact on real apps anyways, since Ruby still
has tons of methods which never get called, either.

> I do have a couple of additional patches I plan to send, but I think it
> may be better to ask here first.
> 
> One is that SIGUSR1 handling doesn't work well with chroot if the log
> file is outside of the chroot, since the worker process may not have
> access to reopen the file.  One simple fix is to reopen the logs in
> the master process, then send SIGQUIT instead of SIGUSR1 to the child
> processes.  Are you open to a patch that does that?

Right now, the worker exits with some log spew if reopening
logs fails, and the worker respawns anyways, correct?

Perhaps that mapping can be made automatic if chroot is
enabled.  I wonder if that's too much magic...
Or the log spew could be suppressed when the error
is ENOENT and chroot is enabled.

The sysadmin could also send SIGHUP to reload everything
and respawn all children.

So, I'm wondering if using SIGHUP is too much of a burden, or if
automatic USR1 => QUIT mapping for chroot users is too magical.

(trying to avoid adding more options + documentation, as always :)

> The second feature is more exotic security feature called fork+exec.
> The current issue with unicorn's direct forking of children is that
> forked children share the memory layout of the server.  This is good
> from a memory usage standpoint due to CoW memory.  However, if there
> is a memory vulnerability in the unicorn application, this makes it
> easier to exploit, since if the a unicorn worker process crashes, the
> master process will fork a child with pretty much the same memory
> layout.  This allows address space discovery attacks.  Using fork+exec,
> each child process would have a unique address space for ASLR, on
> most Unix-like systems (Linux, MacOS X, NetBSD, OpenBSD, Solaris).
> There are additional security advantages from fork+exec on OpenBSD, but
> all Unix-like systems that use ASLR can benefit from the unique
> memory layout per process.

It depends how intrusive the implementation is...

Is this merely calling exec in the worker?

I'm not sure if your use of "fork+exec" is merely the two
well-known syscalls combined, or if there's something else;
I'll assume the former, since that's enough to have a
unique address space for Ruby bytecode.

Can this be done in an after_fork/after_worker_ready hook?

Socket inheritance can be done the same way it handles USR2
upgrades and systemd socket inheritance.

^ permalink raw reply	[relevance 4%]

* Re: Patch: Add after_worker_ready configuration option
  2017-02-23  2:32  9% ` Eric Wong
  2017-02-23  2:34  6%   ` Eric Wong
@ 2017-02-23  3:43  8%   ` Jeremy Evans
  2017-02-23  9:23  4%     ` Eric Wong
  1 sibling, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-23  3:43 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 02/23 02:32, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > This adds a hook that is called after the application has
> > been loaded by the worker process, directly before it starts
> > accepting requests.  This hook is necessary if your application
> > needs to get gain access to resources during initialization,
> > and then drop privileges before serving requests.
> > 
> > This is especially useful in conjunction with chroot support
> > so the app can load all the normal ruby libraries it needs
> > to function, and then chroot before accepting requests.
> > 
> > If you are preloading the app, it's possible to drop privileges
> > or chroot in after_fork, but if you are not preloading the app,
> > there is not currently a properly place to handle this, hence
> > the reason for this feature.
> 
> This means using Unicorn::Worker#user directly,
> and not Unicorn::Configurator#user.  OK...

Correct.  We could add configuration support to specify when to call
Worker#user, but that seems like overkill.
 
> Also, were you planning to send a v2 for chroot support?
> I'm fine with either.

Sorry about that, I will send a v2 for chroot support tomorrow.

> 
> > --- a/lib/unicorn/configurator.rb
> > +++ b/lib/unicorn/configurator.rb
> 
> > @@ -162,7 +165,7 @@ def after_fork(*args, &block)
> >    # sets after_worker_exit hook to a given block.  This block will be called
> >    # by the master process after a worker exits:
> >    #
> > -  #  after_fork do |server,worker,status|
> > +  #  after_worker_exit do |server,worker,status|
> >    #    # status is a Process::Status instance for the exited worker process
> >    #    unless status.success?
> >    #      server.logger.error("worker process failure: #{status.inspect}")
> 
> This looks like an unrelated hunk which belongs in a separate
> patch.  I can squeeze in a separate fix for this (credited to
> you) or you can send a separate patch.
 
Correct, this was an unrelated bug in my last patch.  I'll separate this
hunk into a separate patch.

> > @@ -172,6 +175,18 @@ def after_worker_exit(*args, &block)
> >      set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
> >    end
> >  
> > +  # sets after_worker_ready hook to a given block.  This block will be called
> > +  # by a worker process after it has been fully loaded, directly before it
> > +  # starts responding to requests:
> > +  #
> > +  #  after_worker_ready do |server,worker|
> > +  #    server.logger.info("worker #{worker.nr} ready, dropping privileges")
> > +  #    worker.user('username', 'groupname')
> > +  #  end
> 
> The documentation should probably say something along the lines
> of:
> 
>     Do not use Unicorn::Configurator#user if you rely on
>     changing users in the after_worker_ready hook.
> 
> And maybe corresponding documentation for Unicorn::Configurator#user,
> too.

OK, I'll update the documentation to be more complete.

> Anyways, I'm somewhat inclined to accept this as well, but will
> think about it a bit, too.
> 
> One problem I have with this change is that it requires users
> read more documentation and know more caveats to use.
 
It's unfortunate, but I think it's necessary in this case. Unicorn's
default behavior works for most users, but this makes it so you
don't have to override HttpServer#init_worker_process to get similar
functionality.

> Adding to that, Unicorn::Configurator#user has been favored
> (documentation-wise) over Unicorn::Worker#user in the past;
> so it's a bit of a reversal(*)

I think Configurator#user still makes sense for most users.  As the
documentation mentions, directly calling Worker#user should only
be done if Configurator#user is not sufficient.

> Based on the series of changes you're making, I'm sensing you're
> working on some complex system which might need more security
> than a typical app.

These patches are related to all applications using Unicorn that I run
on OpenBSD (about 20 separate applications).  They are designed to
to reduce the attack surface if an attacker is able to find a
remote code execution vulnerability in one of the applications.

In addition to chroot(2), after chrooting, these applications also
use pledge(2), which is an OpenBSD-specific system call filter
(similar in principle to seccomp(2) in Linux).

> I know you use OpenBSD, and I don't know much about the init
> system or how deployment works, there.
 
It's a very simple sh(1) based-init system, similar to traditional Unix.
Most OpenBSD system daemons will start as root, chroot to a specific
directory, drop privileges (possibly separate privileges in separate
daemon processes connected via IPC), and finally use pledge(2) to limit
what each process can actually use in terms of system calls.

> Nowadays, it seems more common things (binding sockets,
> switching users, chrooting, etc..) is handled by init systems
> (at least systemd); and having that functionality in unicorn
> would mean redundant bloat.

I believe only Linux uses a systemd-like approach, and while most
Linux systems use systemd these days, usage is not universal
(notable exceptions are Slackware, Void, and Gentoo).  But my
knowledge of other operating systems is not very broad.

> It would be useful to me and folks who won't use these features
> to understand how and why they're used.  That can help justify
> the (admittedly small) memory overhead which comes with it.

True.  I'm sorry the commit messages didn't provide a more thorough
explanation.  I will try to improve that in the future.

> (*) Fwiw, I wasn't thrilled to add user switching to unicorn back
>     in 2009/2010, as it should not need to run privileged ports.

I think had you not added it, there wouldn't be a need for a chroot
patch, since otherwise one would assume any user that wanted to
drop privileges would just run the necessary code themselves.

One thing to keep in mind is that if you did not add the user switching
code, developers who need it would implement it themselves, and would
likely miss things like initgroups.

>     So I'm wondering if there can be some of that which could be
>     trimmed out someday or made optional, somehow.

I'm not sure how open you are to a plugin-like system for Unicorn, where
you can have separate files for separate features, and users can choose
which features the want to use.  This limits memory usage so that you
don't pay a memory cost for features you don't use.  This is the
approach used by Sequel, Roda, and Rodauth (other libraries I maintain),
and if you are interested in a similar system for Unicorn, I would be
happy to work on one.

I do have a couple of additional patches I plan to send, but I think it
may be better to ask here first.

One is that SIGUSR1 handling doesn't work well with chroot if the log
file is outside of the chroot, since the worker process may not have
access to reopen the file.  One simple fix is to reopen the logs in
the master process, then send SIGQUIT instead of SIGUSR1 to the child
processes.  Are you open to a patch that does that?

The second feature is more exotic security feature called fork+exec.
The current issue with unicorn's direct forking of children is that
forked children share the memory layout of the server.  This is good
from a memory usage standpoint due to CoW memory.  However, if there
is a memory vulnerability in the unicorn application, this makes it
easier to exploit, since if the a unicorn worker process crashes, the
master process will fork a child with pretty much the same memory
layout.  This allows address space discovery attacks.  Using fork+exec,
each child process would have a unique address space for ASLR, on
most Unix-like systems (Linux, MacOS X, NetBSD, OpenBSD, Solaris).
There are additional security advantages from fork+exec on OpenBSD, but
all Unix-like systems that use ASLR can benefit from the unique
memory layout per process.

Thanks,
Jeremy

^ permalink raw reply	[relevance 8%]

* Re: check_client_connection using getsockopt(2)
  2017-02-23  1:42  4%     ` Eric Wong
@ 2017-02-23  2:42  0%       ` Simon Eskildsen
  0 siblings, 0 replies; 200+ results
From: Simon Eskildsen @ 2017-02-23  2:42 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public, raindrops-public

On Wed, Feb 22, 2017 at 8:42 PM, Eric Wong <e@80x24.org> wrote:
> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>
> <snip> Thanks for the writeup!
>
> Another sidenote: It seems nginx <-> unicorn is a bit odd
> for deployment in a containerized environment(*).
>
>> I meant to ask, in Raindrops why do you use the netlink API to get the
>> socket backlog instead of `getsockopt(2)` with `TCP_INFO` to get
>> `tcpi_unacked`? (as described in
>> http://www.ryanfrantz.com/posts/apache-tcp-backlog/) We use this to
>> monitor socket backlogs with a sidekick Ruby daemon. Although we're
>> looking to replace it with a middleware to simplify for Kubernetes.
>> It's one of our main metrics for monitoring performance, especially
>> around deploys.
>
> The netlink API allows independently-spawned processes to
> monitor others; so it can be system-wide.  TCP_INFO requires the
> process doing the checking to also have the socket open.
>
> I guess this reasoning for using netlink is invalid for containers,
> though...

If you namespace the network it's problematic, yeah. I'm considering
right now putting Raindrops in a middleware with the netlink API
inside the container, but it feels weird. That said, if you consider
the alternative of using `getsockopt(2)` on the listening socket, I
don't know how you'd get access to the Unicorn listening socket from a
middleware. Would it be nuts to expose a hook in Unicorn that allows
periodic execution for monitoring listening stats from Raindrops on
the listening socket? It seems somewhat of a narrow use-case, but on
the other hand I'm also not a fan of doing
`Raindrops::Linux.tcp_listener_stats("localhost:#{ENV["PORT"}")`, but
that might be less ugly.

>
>> I was going to use `env["unicorn.socket"]`/`env["puma.socket"]`, but
>> you could also do `env.delete("hijack_io")` after hijacking to allow
>> Unicorn to still render the response. Unfortunately the
>> `<webserver>.socket` key is not part of the Rack standard, so I'm
>> hesitant to use it. When this gets into Unicorn I'm planning to
>> propose it upstream to Puma as well.
>
> I was going to say env.delete("rack.hijack_io") is dangerous
> (since env could be copied by middleware); but apparently not:
> rack.hijack won't work with a copied env, either.
> You only need to delete with the same env object you call
> rack.hijack with.
>
> But calling rack.hijack followed by env.delete may still
> have unintended side-effects in other servers; so I guess
> we (again) cannot rely on hijack working portably.

Exactly, it gives the illusion of portability but e.g. Puma stores an
instance variable to check whether a middleware hijacked, rendering
the `env#delete` option useless.

>
>> Cool. How would you suggest I check for TCP_INFO compatible platforms
>> in Unicorn? Is `RUBY_PLATFORM.ends_with?("linux".freeze)` sufficient
>> or do you prefer another mechanism? I agree that we should fall back
>> to the write hack on other platforms.
>
> The Raindrops::TCP_Info class should be undefined on unsupported
> platforms, so I think you can check for that, instead.  Then it
> should be transparent to add FreeBSD support from unicorn's
> perspective.

Perfect. I'll start working on a patch.

>
>
> (*) So I've been wondering if adding a "unicorn-mode" to an
>     existing C10K, slow-client-capable Ruby/Rack server +
>     reverse proxy makes sense for containerized deploys.
>     Maybe...

I'd love to hear more about this idea. What are you contemplating?

^ permalink raw reply	[relevance 0%]

* Re: Patch: Add after_worker_ready configuration option
  2017-02-23  2:32  9% ` Eric Wong
@ 2017-02-23  2:34  6%   ` Eric Wong
  2017-02-23  3:43  8%   ` Jeremy Evans
  1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2017-02-23  2:34 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Eric Wong <e@80x24.org> wrote:
> Also, were you planning to send a v2 for chroot support?

+ Or did you want me to squash the explicit path change in?

> I'm fine with either.

^ permalink raw reply	[relevance 6%]

* Re: Patch: Add after_worker_ready configuration option
  2017-02-23  0:49  4% Patch: Add after_worker_ready configuration option Jeremy Evans
@ 2017-02-23  2:32  9% ` Eric Wong
  2017-02-23  2:34  6%   ` Eric Wong
  2017-02-23  3:43  8%   ` Jeremy Evans
  0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2017-02-23  2:32 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> This adds a hook that is called after the application has
> been loaded by the worker process, directly before it starts
> accepting requests.  This hook is necessary if your application
> needs to get gain access to resources during initialization,
> and then drop privileges before serving requests.
> 
> This is especially useful in conjunction with chroot support
> so the app can load all the normal ruby libraries it needs
> to function, and then chroot before accepting requests.
> 
> If you are preloading the app, it's possible to drop privileges
> or chroot in after_fork, but if you are not preloading the app,
> there is not currently a properly place to handle this, hence
> the reason for this feature.

This means using Unicorn::Worker#user directly,
and not Unicorn::Configurator#user.  OK...

Also, were you planning to send a v2 for chroot support?
I'm fine with either.

> --- a/lib/unicorn/configurator.rb
> +++ b/lib/unicorn/configurator.rb

> @@ -162,7 +165,7 @@ def after_fork(*args, &block)
>    # sets after_worker_exit hook to a given block.  This block will be called
>    # by the master process after a worker exits:
>    #
> -  #  after_fork do |server,worker,status|
> +  #  after_worker_exit do |server,worker,status|
>    #    # status is a Process::Status instance for the exited worker process
>    #    unless status.success?
>    #      server.logger.error("worker process failure: #{status.inspect}")

This looks like an unrelated hunk which belongs in a separate
patch.  I can squeeze in a separate fix for this (credited to
you) or you can send a separate patch.

> @@ -172,6 +175,18 @@ def after_worker_exit(*args, &block)
>      set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
>    end
>  
> +  # sets after_worker_ready hook to a given block.  This block will be called
> +  # by a worker process after it has been fully loaded, directly before it
> +  # starts responding to requests:
> +  #
> +  #  after_worker_ready do |server,worker|
> +  #    server.logger.info("worker #{worker.nr} ready, dropping privileges")
> +  #    worker.user('username', 'groupname')
> +  #  end

The documentation should probably say something along the lines
of:

    Do not use Unicorn::Configurator#user if you rely on
    changing users in the after_worker_ready hook.

And maybe corresponding documentation for Unicorn::Configurator#user,
too.

Anyways, I'm somewhat inclined to accept this as well, but will
think about it a bit, too.

One problem I have with this change is that it requires users
read more documentation and know more caveats to use.

Adding to that, Unicorn::Configurator#user has been favored
(documentation-wise) over Unicorn::Worker#user in the past;
so it's a bit of a reversal(*)

Based on the series of changes you're making, I'm sensing you're
working on some complex system which might need more security
than a typical app.

I know you use OpenBSD, and I don't know much about the init
system or how deployment works, there.

Nowadays, it seems more common things (binding sockets,
switching users, chrooting, etc..) is handled by init systems
(at least systemd); and having that functionality in unicorn
would mean redundant bloat.

It would be useful to me and folks who won't use these features
to understand how and why they're used.  That can help justify
the (admittedly small) memory overhead which comes with it.

Thanks.


(*) Fwiw, I wasn't thrilled to add user switching to unicorn back
    in 2009/2010, as it should not need to run privileged ports.
    So I'm wondering if there can be some of that which could be
    trimmed out someday or made optional, somehow.

^ permalink raw reply	[relevance 9%]

* Re: check_client_connection using getsockopt(2)
  @ 2017-02-23  1:42  4%     ` Eric Wong
  2017-02-23  2:42  0%       ` Simon Eskildsen
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-23  1:42 UTC (permalink / raw)
  To: Simon Eskildsen; +Cc: unicorn-public, raindrops-public

Simon Eskildsen <simon.eskildsen@shopify.com> wrote:

<snip> Thanks for the writeup!

Another sidenote: It seems nginx <-> unicorn is a bit odd
for deployment in a containerized environment(*).

> I meant to ask, in Raindrops why do you use the netlink API to get the
> socket backlog instead of `getsockopt(2)` with `TCP_INFO` to get
> `tcpi_unacked`? (as described in
> http://www.ryanfrantz.com/posts/apache-tcp-backlog/) We use this to
> monitor socket backlogs with a sidekick Ruby daemon. Although we're
> looking to replace it with a middleware to simplify for Kubernetes.
> It's one of our main metrics for monitoring performance, especially
> around deploys.

The netlink API allows independently-spawned processes to
monitor others; so it can be system-wide.  TCP_INFO requires the
process doing the checking to also have the socket open.

I guess this reasoning for using netlink is invalid for containers,
though...

> I was going to use `env["unicorn.socket"]`/`env["puma.socket"]`, but
> you could also do `env.delete("hijack_io")` after hijacking to allow
> Unicorn to still render the response. Unfortunately the
> `<webserver>.socket` key is not part of the Rack standard, so I'm
> hesitant to use it. When this gets into Unicorn I'm planning to
> propose it upstream to Puma as well.

I was going to say env.delete("rack.hijack_io") is dangerous
(since env could be copied by middleware); but apparently not:
rack.hijack won't work with a copied env, either.
You only need to delete with the same env object you call
rack.hijack with.

But calling rack.hijack followed by env.delete may still
have unintended side-effects in other servers; so I guess
we (again) cannot rely on hijack working portably.

> Cool. How would you suggest I check for TCP_INFO compatible platforms
> in Unicorn? Is `RUBY_PLATFORM.ends_with?("linux".freeze)` sufficient
> or do you prefer another mechanism? I agree that we should fall back
> to the write hack on other platforms.

The Raindrops::TCP_Info class should be undefined on unsupported
platforms, so I think you can check for that, instead.  Then it
should be transparent to add FreeBSD support from unicorn's
perspective.


(*) So I've been wondering if adding a "unicorn-mode" to an
    existing C10K, slow-client-capable Ruby/Rack server +
    reverse proxy makes sense for containerized deploys.
    Maybe...

^ permalink raw reply	[relevance 4%]

* Patch: Add after_worker_ready configuration option
@ 2017-02-23  0:49  4% Jeremy Evans
  2017-02-23  2:32  9% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-23  0:49 UTC (permalink / raw)
  To: unicorn-public

This adds a hook that is called after the application has
been loaded by the worker process, directly before it starts
accepting requests.  This hook is necessary if your application
needs to get gain access to resources during initialization,
and then drop privileges before serving requests.

This is especially useful in conjunction with chroot support
so the app can load all the normal ruby libraries it needs
to function, and then chroot before accepting requests.

If you are preloading the app, it's possible to drop privileges
or chroot in after_fork, but if you are not preloading the app,
there is not currently a properly place to handle this, hence
the reason for this feature.
---
 lib/unicorn/configurator.rb | 17 ++++++++++++++++-
 lib/unicorn/http_server.rb  |  4 ++--
 2 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 5bad925..6df5a30 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -49,6 +49,9 @@ class Unicorn::Configurator
           server.logger.error(m)
         end
       },
+    :after_worker_ready => lambda { |server, worker|
+        server.logger.info("worker=#{worker.nr} ready")
+      },
     :pid => nil,
     :preload_app => false,
     :check_client_connection => false,
@@ -162,7 +165,7 @@ def after_fork(*args, &block)
   # sets after_worker_exit hook to a given block.  This block will be called
   # by the master process after a worker exits:
   #
-  #  after_fork do |server,worker,status|
+  #  after_worker_exit do |server,worker,status|
   #    # status is a Process::Status instance for the exited worker process
   #    unless status.success?
   #      server.logger.error("worker process failure: #{status.inspect}")
@@ -172,6 +175,18 @@ def after_worker_exit(*args, &block)
     set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
   end
 
+  # sets after_worker_ready hook to a given block.  This block will be called
+  # by a worker process after it has been fully loaded, directly before it
+  # starts responding to requests:
+  #
+  #  after_worker_ready do |server,worker|
+  #    server.logger.info("worker #{worker.nr} ready, dropping privileges")
+  #    worker.user('username', 'groupname')
+  #  end
+  def after_worker_ready(*args, &block)
+    set_hook(:after_worker_ready, block_given? ? block : args[0])
+  end
+
   # sets before_fork got be a given Proc object.  This Proc
   # object will be called by the master process before forking
   # each worker.
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index c2086cb..ef897ad 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,7 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user
-  attr_writer   :after_worker_exit
+  attr_writer   :after_worker_exit, :after_worker_ready
 
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
@@ -644,7 +644,7 @@ def worker_loop(worker)
     trap(:USR1) { nr = -65536 }
 
     ready = readers.dup
-    @logger.info "worker=#{worker.nr} ready"
+    @after_worker_ready.call(self, worker)
 
     begin
       nr < 0 and reopen_worker_logs(worker.nr)
-- 
2.11.0


^ permalink raw reply related	[relevance 4%]

* Re: Patch: Add after_worker_exit configuration option
  2017-02-21 20:49  8%       ` Jeremy Evans
@ 2017-02-21 21:38  6%         ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2017-02-21 21:38 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Here's a revised patch that should address the issues you identified:

Thanks.  Pushed as 2af91a1fef70d6546ee03760011c170a082db667

I needed one change to an integration test to workaround
the lack of reader attribute, but I'd rather pay the cost
in the test than the server, itself:

https://bogomips.org/unicorn.git/patch?id=2c6aa5878d052abb
("t/t0012-reload-empty-config.sh: access ivars directly if needed")

^ permalink raw reply	[relevance 6%]

* Re: Patch: Add after_worker_exit configuration option
  2017-02-21 20:15  6%     ` Eric Wong
@ 2017-02-21 20:49  8%       ` Jeremy Evans
  2017-02-21 21:38  6%         ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-21 20:49 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 02/21 08:15, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > On 02/21 07:43, Eric Wong wrote:
> > > Jeremy Evans <code@jeremyevans.net> wrote:
> > > > This option can be used to implement custom behavior for handling
> > > > worker exits.  For example, let's say you have a specific request
> > > > that crashes a worker process, which you expect to be due to a
> > > > improperly programmed C extension. By modifying your worker to
> > > > save request related data in a temporary file and using this option,
> > > > you can get a record of what request is crashing the application,
> > > > which will make debugging easier.
> > > > 
> > > > This is not a complete patch as it doesn't include tests, but
> > > > before writing tests I wanted to see if this is something you'd
> > > > consider including in unicorn.
> > > 
> > > What advantage does this have over Ruby's builtin at_exit?
> > 
> > at_exit is not called if the interpreter crashes:
> > 
> > ruby -e 'at_exit{File.write('a.txt', 'a')}; Process.kill :SEGV, $$' 2>/dev/null
> > ([ -f a.txt ] && echo at_exit called) || echo at_exit not called
> 
> Ah, thanks.  I didn't read the code carefully, enough.
> The commit message and documentation should reflect that it's
> called in the master and not the worker.
> 
> Anyways, this is probably OK since I can't think of a less
> intrusive or more generic (across all Rack servers) way of doing
> it.  So I'm inclined to accept some version of this.
> 
> > --- a/lib/unicorn/http_server.rb
> > +++ b/lib/unicorn/http_server.rb
> > @@ -14,7 +14,8 @@ class Unicorn::HttpServer
> >    attr_accessor :app, :timeout, :worker_processes,
> >                  :before_fork, :after_fork, :before_exec,
> >                  :listener_opts, :preload_app,
> > -                :orig_app, :config, :ready_pipe, :user
> > +                :orig_app, :config, :ready_pipe, :user,
> > +                :after_worker_exit
> 
> I've been trying to reduce the method entry overhead across
> the board.  Since this is a new field, can it be attr_writer
> solely for configurator?
> 
> I don't think there's reader dependency other than below...
> 
> > @@ -395,8 +396,7 @@ def reap_all_workers
> >          proc_name 'master'
> >        else
> >          worker = @workers.delete(wpid) and worker.close rescue nil
> > -        m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
> > -        status.success? ? logger.info(m) : logger.error(m)
> > +        after_worker_exit.call(self, worker, status)
> 
> Which can be made to access the ivar directly:
> 
> 	   @after_worker_exit.call(self, worker, status)

Here's a revised patch that should address the issues you identified:

From 91b95652e4bdb7fc9af77e9ac06ad7400faef796 Mon Sep 17 00:00:00 2001
From: Jeremy Evans <code@jeremyevans.net>
Date: Fri, 17 Feb 2017 16:12:33 -0800
Subject: [PATCH] Add after_worker_exit configuration option

This option is executed in the master process following all
worker process exits.  It is most useful in the case where
the worker process crashes the ruby interpreter, as the worker
process may not be able to send error notifications
appropriately.

For example, let's say you have a specific request that crashes a
worker process, which you expect to be due to a improperly
programmed C extension. By modifying your worker to save request
related data in a temporary file and using this option, you can get
a record of what request is crashing the application, which will
make debugging easier.

Example:

after_worker_exit do |server, worker, status|
  server.logger.info "worker #{status.success? ? 'exit' : 'crash'}: #{status}"

  file = "request.#{status.pid}.txt"
  if File.exist?(file)
    do_something_with(File.read(file)) unless status.success?
    File.delete(file)
  end
end
---
 lib/unicorn/configurator.rb | 21 +++++++++++++++++++++
 lib/unicorn/http_server.rb  |  4 ++--
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 3329c10..5bad925 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -41,6 +41,14 @@ class Unicorn::Configurator
     :before_exec => lambda { |server|
         server.logger.info("forked child re-executing...")
       },
+    :after_worker_exit => lambda { |server, worker, status|
+        m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
+        if status.success?
+          server.logger.info(m)
+        else
+          server.logger.error(m)
+        end
+      },
     :pid => nil,
     :preload_app => false,
     :check_client_connection => false,
@@ -151,6 +159,19 @@ def after_fork(*args, &block)
     set_hook(:after_fork, block_given? ? block : args[0])
   end
 
+  # sets after_worker_exit hook to a given block.  This block will be called
+  # by the master process after a worker exits:
+  #
+  #  after_fork do |server,worker,status|
+  #    # status is a Process::Status instance for the exited worker process
+  #    unless status.success?
+  #      server.logger.error("worker process failure: #{status.inspect}")
+  #    end
+  #  end
+  def after_worker_exit(*args, &block)
+    set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
+  end
+
   # sets before_fork got be a given Proc object.  This Proc
   # object will be called by the master process before forking
   # each worker.
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 35bd100..c2086cb 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -15,6 +15,7 @@ class Unicorn::HttpServer
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
                 :orig_app, :config, :ready_pipe, :user
+  attr_writer   :after_worker_exit
 
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
@@ -395,8 +396,7 @@ def reap_all_workers
         proc_name 'master'
       else
         worker = @workers.delete(wpid) and worker.close rescue nil
-        m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
-        status.success? ? logger.info(m) : logger.error(m)
+        @after_worker_exit.call(self, worker, status)
       end
     rescue Errno::ECHILD
       break
-- 
2.11.0


^ permalink raw reply related	[relevance 8%]

* Re: Patch: Add after_worker_exit configuration option
  2017-02-21 20:02  6%   ` Jeremy Evans
@ 2017-02-21 20:15  6%     ` Eric Wong
  2017-02-21 20:49  8%       ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-21 20:15 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> On 02/21 07:43, Eric Wong wrote:
> > Jeremy Evans <code@jeremyevans.net> wrote:
> > > This option can be used to implement custom behavior for handling
> > > worker exits.  For example, let's say you have a specific request
> > > that crashes a worker process, which you expect to be due to a
> > > improperly programmed C extension. By modifying your worker to
> > > save request related data in a temporary file and using this option,
> > > you can get a record of what request is crashing the application,
> > > which will make debugging easier.
> > > 
> > > This is not a complete patch as it doesn't include tests, but
> > > before writing tests I wanted to see if this is something you'd
> > > consider including in unicorn.
> > 
> > What advantage does this have over Ruby's builtin at_exit?
> 
> at_exit is not called if the interpreter crashes:
> 
> ruby -e 'at_exit{File.write('a.txt', 'a')}; Process.kill :SEGV, $$' 2>/dev/null
> ([ -f a.txt ] && echo at_exit called) || echo at_exit not called

Ah, thanks.  I didn't read the code carefully, enough.
The commit message and documentation should reflect that it's
called in the master and not the worker.

Anyways, this is probably OK since I can't think of a less
intrusive or more generic (across all Rack servers) way of doing
it.  So I'm inclined to accept some version of this.

> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -14,7 +14,8 @@ class Unicorn::HttpServer
>    attr_accessor :app, :timeout, :worker_processes,
>                  :before_fork, :after_fork, :before_exec,
>                  :listener_opts, :preload_app,
> -                :orig_app, :config, :ready_pipe, :user
> +                :orig_app, :config, :ready_pipe, :user,
> +                :after_worker_exit

I've been trying to reduce the method entry overhead across
the board.  Since this is a new field, can it be attr_writer
solely for configurator?

I don't think there's reader dependency other than below...

> @@ -395,8 +396,7 @@ def reap_all_workers
>          proc_name 'master'
>        else
>          worker = @workers.delete(wpid) and worker.close rescue nil
> -        m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
> -        status.success? ? logger.info(m) : logger.error(m)
> +        after_worker_exit.call(self, worker, status)

Which can be made to access the ivar directly:

	   @after_worker_exit.call(self, worker, status)

^ permalink raw reply	[relevance 6%]

* Re: Patch: Add support for chroot to Worker#user
  2017-02-21 19:53  6% ` Eric Wong
@ 2017-02-21 20:15  6%   ` Jeremy Evans
  0 siblings, 0 replies; 200+ results
From: Jeremy Evans @ 2017-02-21 20:15 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 02/21 07:53, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > Any chrooting would need to happen inside Worker#user, because
> > you can't chroot until after you have parsed the list of groups,
> > and you must chroot before dropping root privileges.
> > 
> > chroot is an important security feature, so that if the unicorn
> > process is exploited, file system access is limited to the chroot
> > directory instead of the entire system.
> 
> I'm hesitant to document this as "an important security feature".
> 
> Perhaps "an extra layer of security" is more accurate,
> as there are ways of breaking out of a chroot.

That's fine.  Note that while breaking out of a chroot is easy if you
have root privileges, it is difficult to break out of a chroot
if root privileges have been dropped.  There are a couple of
possibilities listed at https://github.com/earthquake/chw00t, neither of
which is likely in environments where unicorn is typically deployed.

> 
> > This is not a complete patch as it does not include tests. I can
> > add tests if you would consider accepting this feature.
> 
> I wouldn't worry about automated tests if elevated privileges
> are required.
> 
> > -  # Changes the worker process to the specified +user+ and +group+
> > +  # Changes the worker process to the specified +user+ and +group+,
> > +  # and chroots to the current working directory if +chroot+ is set.
> >    # This is only intended to be called from within the worker
> >    # process from the +after_fork+ hook.  This should be called in
> >    # the +after_fork+ hook after any privileged functions need to be
> > @@ -123,7 +124,7 @@ def close # :nodoc:
> >    # directly back to the caller (usually the +after_fork+ hook.
> >    # These errors commonly include ArgumentError for specifying an
> >    # invalid user/group and Errno::EPERM for insufficient privileges
> > -  def user(user, group = nil)
> > +  def user(user, group = nil, chroot = false)
> >      # we do not protect the caller, checking Process.euid == 0 is
> >      # insufficient because modern systems have fine-grained
> >      # capabilities.  Let the caller handle any and all errors.
> > @@ -134,6 +135,10 @@ def user(user, group = nil)
> >        Process.initgroups(user, gid)
> >        Process::GID.change_privilege(gid)
> >      end
> > +    if chroot
> > +      Dir.chroot(Dir.pwd)
> > +      Dir.chdir('/')
> > +    end
> 
> Perhaps this can be made more flexible by allowing chroot to
> any specified directory, not just pwd.  Something like:
> 
>     chroot = Dir.pwd if chroot == true
>     if chroot
>       Dir.chroot(chroot)
>       Dir.chdir('/')
>     end
> 
> Anyways I'm inclined to accept this feature.

I had basically the same code originally, but wanted to minimize the API
surface to increase the likelihood for acceptance.  So I'm definitely in
favor of that change.

Thanks,
Jeremy


^ permalink raw reply	[relevance 6%]

* Re: Patch: Add after_worker_exit configuration option
  2017-02-21 19:43  6% ` Eric Wong
@ 2017-02-21 20:02  6%   ` Jeremy Evans
  2017-02-21 20:15  6%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-21 20:02 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 02/21 07:43, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > This option can be used to implement custom behavior for handling
> > worker exits.  For example, let's say you have a specific request
> > that crashes a worker process, which you expect to be due to a
> > improperly programmed C extension. By modifying your worker to
> > save request related data in a temporary file and using this option,
> > you can get a record of what request is crashing the application,
> > which will make debugging easier.
> > 
> > This is not a complete patch as it doesn't include tests, but
> > before writing tests I wanted to see if this is something you'd
> > consider including in unicorn.
> 
> What advantage does this have over Ruby's builtin at_exit?

at_exit is not called if the interpreter crashes:

ruby -e 'at_exit{File.write('a.txt', 'a')}; Process.kill :SEGV, $$' 2>/dev/null
([ -f a.txt ] && echo at_exit called) || echo at_exit not called

> 
> AFAIK, at_exit may be registered inside after_fork hooks to the
> same effect.
> 
> Thanks.

^ permalink raw reply	[relevance 6%]

* Re: Patch: Add support for chroot to Worker#user
  2017-02-21 19:24  8% Patch: Add support for chroot to Worker#user Jeremy Evans
@ 2017-02-21 19:53  6% ` Eric Wong
  2017-02-21 20:15  6%   ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-21 19:53 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> Any chrooting would need to happen inside Worker#user, because
> you can't chroot until after you have parsed the list of groups,
> and you must chroot before dropping root privileges.
> 
> chroot is an important security feature, so that if the unicorn
> process is exploited, file system access is limited to the chroot
> directory instead of the entire system.

I'm hesitant to document this as "an important security feature".

Perhaps "an extra layer of security" is more accurate,
as there are ways of breaking out of a chroot.

> This is not a complete patch as it does not include tests. I can
> add tests if you would consider accepting this feature.

I wouldn't worry about automated tests if elevated privileges
are required.

> -  # Changes the worker process to the specified +user+ and +group+
> +  # Changes the worker process to the specified +user+ and +group+,
> +  # and chroots to the current working directory if +chroot+ is set.
>    # This is only intended to be called from within the worker
>    # process from the +after_fork+ hook.  This should be called in
>    # the +after_fork+ hook after any privileged functions need to be
> @@ -123,7 +124,7 @@ def close # :nodoc:
>    # directly back to the caller (usually the +after_fork+ hook.
>    # These errors commonly include ArgumentError for specifying an
>    # invalid user/group and Errno::EPERM for insufficient privileges
> -  def user(user, group = nil)
> +  def user(user, group = nil, chroot = false)
>      # we do not protect the caller, checking Process.euid == 0 is
>      # insufficient because modern systems have fine-grained
>      # capabilities.  Let the caller handle any and all errors.
> @@ -134,6 +135,10 @@ def user(user, group = nil)
>        Process.initgroups(user, gid)
>        Process::GID.change_privilege(gid)
>      end
> +    if chroot
> +      Dir.chroot(Dir.pwd)
> +      Dir.chdir('/')
> +    end

Perhaps this can be made more flexible by allowing chroot to
any specified directory, not just pwd.  Something like:

    chroot = Dir.pwd if chroot == true
    if chroot
      Dir.chroot(chroot)
      Dir.chdir('/')
    end

Anyways I'm inclined to accept this feature.

^ permalink raw reply	[relevance 6%]

* Re: Patch: Add after_worker_exit configuration option
  2017-02-21 19:19  4% Patch: Add after_worker_exit configuration option Jeremy Evans
@ 2017-02-21 19:43  6% ` Eric Wong
  2017-02-21 20:02  6%   ` Jeremy Evans
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2017-02-21 19:43 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> This option can be used to implement custom behavior for handling
> worker exits.  For example, let's say you have a specific request
> that crashes a worker process, which you expect to be due to a
> improperly programmed C extension. By modifying your worker to
> save request related data in a temporary file and using this option,
> you can get a record of what request is crashing the application,
> which will make debugging easier.
> 
> This is not a complete patch as it doesn't include tests, but
> before writing tests I wanted to see if this is something you'd
> consider including in unicorn.

What advantage does this have over Ruby's builtin at_exit?

AFAIK, at_exit may be registered inside after_fork hooks to the
same effect.

Thanks.

^ permalink raw reply	[relevance 6%]

* Patch: Add support for chroot to Worker#user
@ 2017-02-21 19:24  8% Jeremy Evans
  2017-02-21 19:53  6% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-21 19:24 UTC (permalink / raw)
  To: unicorn-public

Any chrooting would need to happen inside Worker#user, because
you can't chroot until after you have parsed the list of groups,
and you must chroot before dropping root privileges.

chroot is an important security feature, so that if the unicorn
process is exploited, file system access is limited to the chroot
directory instead of the entire system.

This is not a complete patch as it does not include tests. I can
add tests if you would consider accepting this feature.
---
 lib/unicorn/worker.rb | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index 6748a2f..db418ea 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -111,9 +111,10 @@ def close # :nodoc:
   # In most cases, you should be using the Unicorn::Configurator#user
   # directive instead.  This method should only be used if you need
   # fine-grained control of exactly when you want to change permissions
-  # in your after_fork hooks.
+  # in your after_fork hooks, or if you want to use the chroot support.
   #
-  # Changes the worker process to the specified +user+ and +group+
+  # Changes the worker process to the specified +user+ and +group+,
+  # and chroots to the current working directory if +chroot+ is set.
   # This is only intended to be called from within the worker
   # process from the +after_fork+ hook.  This should be called in
   # the +after_fork+ hook after any privileged functions need to be
@@ -123,7 +124,7 @@ def close # :nodoc:
   # directly back to the caller (usually the +after_fork+ hook.
   # These errors commonly include ArgumentError for specifying an
   # invalid user/group and Errno::EPERM for insufficient privileges
-  def user(user, group = nil)
+  def user(user, group = nil, chroot = false)
     # we do not protect the caller, checking Process.euid == 0 is
     # insufficient because modern systems have fine-grained
     # capabilities.  Let the caller handle any and all errors.
@@ -134,6 +135,10 @@ def user(user, group = nil)
       Process.initgroups(user, gid)
       Process::GID.change_privilege(gid)
     end
+    if chroot
+      Dir.chroot(Dir.pwd)
+      Dir.chdir('/')
+    end
     Process.euid != uid and Process::UID.change_privilege(uid)
     @switched = true
   end
-- 
2.11.0


^ permalink raw reply related	[relevance 8%]

* Patch: Add after_worker_exit configuration option
@ 2017-02-21 19:19  4% Jeremy Evans
  2017-02-21 19:43  6% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jeremy Evans @ 2017-02-21 19:19 UTC (permalink / raw)
  To: unicorn-public

This option can be used to implement custom behavior for handling
worker exits.  For example, let's say you have a specific request
that crashes a worker process, which you expect to be due to a
improperly programmed C extension. By modifying your worker to
save request related data in a temporary file and using this option,
you can get a record of what request is crashing the application,
which will make debugging easier.

This is not a complete patch as it doesn't include tests, but
before writing tests I wanted to see if this is something you'd
consider including in unicorn.

Example:

after_worker_exit do |server, worker, status|
  server.logger.info "worker #{status.success? ? 'exit' : 'crash'}: #{status}"

  file = "request.#{status.pid}.txt"
  if File.exist?(file)
    do_something_with(File.read(file)) unless status.success?
    File.delete(file)
  end
end
---
 lib/unicorn/configurator.rb | 14 ++++++++++++++
 lib/unicorn/http_server.rb  |  6 +++---
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 3329c10..81589b0 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -41,6 +41,14 @@ class Unicorn::Configurator
     :before_exec => lambda { |server|
         server.logger.info("forked child re-executing...")
       },
+    :after_worker_exit => lambda { |server, worker, status|
+        m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
+        if status.success?
+          server.logger.info(m)
+        else
+          server.logger.error(m)
+        end
+      },
     :pid => nil,
     :preload_app => false,
     :check_client_connection => false,
@@ -151,6 +159,12 @@ def after_fork(*args, &block)
     set_hook(:after_fork, block_given? ? block : args[0])
   end
 
+  # sets after_worker_exit hook to a given block.  This block will be called
+  # by the master process after a worker exits.
+  def after_worker_exit(*args, &block)
+    set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
+  end
+
   # sets before_fork got be a given Proc object.  This Proc
   # object will be called by the master process before forking
   # each worker.
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 35bd100..567ee0e 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -14,7 +14,8 @@ class Unicorn::HttpServer
   attr_accessor :app, :timeout, :worker_processes,
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
-                :orig_app, :config, :ready_pipe, :user
+                :orig_app, :config, :ready_pipe, :user,
+                :after_worker_exit
 
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
@@ -395,8 +396,7 @@ def reap_all_workers
         proc_name 'master'
       else
         worker = @workers.delete(wpid) and worker.close rescue nil
-        m = "reaped #{status.inspect} worker=#{worker.nr rescue 'unknown'}"
-        status.success? ? logger.info(m) : logger.error(m)
+        after_worker_exit.call(self, worker, status)
       end
     rescue Errno::ECHILD
       break
-- 
2.11.0


^ permalink raw reply related	[relevance 4%]

* [ANN] unicorn 5.2.0 - Rack HTTP server for fast clients and *nix
       [not found]     <20161031-unicorn-5.2.0-released@bogomips.org>
@ 2016-10-31 20:04  4% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-10-31 20:04 UTC (permalink / raw)
  To: ruby-talk, unicorn-public; +Cc: Mishael A Sibiryakov

unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

Changes:

    Most notably, this release allows us to support requests with
    lines delimited by LF-only, as opposed to the standard CRLF
    pair and allowed by RFC 2616 sec 19.3.

    Thanks to Mishael A Sibiryakov for the explanation and change:

      https://bogomips.org/unicorn-public/1476954332.1736.156.camel@junki.org/

    Thanks to Let's Encrypt, the website also moves to HTTPS
    <https://bogomips.org/unicorn/> to improve reader privacy.  The
    "unicorn.bogomips.org" subdomain will be retired soon to reduce
    subjectAltName bloat and speed up certificate renewals.

    There's also the usual round of documentation and example
    updates, too.

    Eric Wong (7):
          examples/init.sh: update to reduce upgrade raciness
          doc: systemd should only kill master in example
          examples/logrotate.conf: update example for systemd
          doc: update gmane URLs to point to our own archives
          relocate website to https://bogomips.org/unicorn/
          TODO: remove Rack 2.x item
          build: "install-gem" target avoids network

    Mishael A Sibiryakov (1):
          Add some tolerance (RFC2616 sec. 19.3)

^ permalink raw reply	[relevance 4%]

* Re: [PATCH] Add some tolerance (RFC2616 sec. 19.3)
  2016-10-20 20:50  6%     ` Eric Wong
@ 2016-10-20 21:03  6%       ` Mishael A Sibiryakov
  0 siblings, 0 replies; 200+ results
From: Mishael A Sibiryakov @ 2016-10-20 21:03 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On Thu, 2016-10-20 at 20:50 +0000, Eric Wong wrote:

> > > > +  def test_multiline_header_0d0a
> > > > +    parser = HttpParser.new
> > > > +    parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
> > > > bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
> > > 
> > > I expect code to be wrapped at 80 lines or less.  Fixed locally.
> > > (I need big fonts, even 80 is a compromise, I really prefer 64)
> > 
> > Line 221 with "ssl bullshit" guided me to this.
> 
> Ah, that's an old test from Mongrel, probably not worth fixing;
> just trying to keep new code cleaner.

It's a good desire :) I'll comply next time.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add some tolerance (RFC2616 sec. 19.3)
  2016-10-20 20:25  6%   ` Mishael A Sibiryakov
@ 2016-10-20 20:50  6%     ` Eric Wong
  2016-10-20 21:03  6%       ` Mishael A Sibiryakov
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-10-20 20:50 UTC (permalink / raw)
  To: Mishael A Sibiryakov; +Cc: unicorn-public

Mishael A Sibiryakov <death@junki.org> wrote:
> It's just a temporary fork and I'm too lazy ))

<snip>

> Whoops, my fault. 

No worries :)

> > > +  def test_multiline_header_0d0a
> > > +    parser = HttpParser.new
> > > +    parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
> > > bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
> > 
> > I expect code to be wrapped at 80 lines or less.  Fixed locally.
> > (I need big fonts, even 80 is a compromise, I really prefer 64)
> 
> Line 221 with "ssl bullshit" guided me to this.

Ah, that's an old test from Mongrel, probably not worth fixing;
just trying to keep new code cleaner.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add some tolerance (RFC2616 sec. 19.3)
  2016-10-20 17:55  5% ` Eric Wong
@ 2016-10-20 20:25  6%   ` Mishael A Sibiryakov
  2016-10-20 20:50  6%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Mishael A Sibiryakov @ 2016-10-20 20:25 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On Thu, 2016-10-20 at 17:55 +0000, Eric Wong wrote:

> Thanks for that useful explanation.  Aside from the unnecessary,
> "Hi all,", that is an informative commit message which justifies
> the usefulness of that patch.
> 
> > 
> > Github commit https://github.com/uno4ki/unicorn/commit/ed127b66e162aaf1
> > 76de05720f6be758f8b41b1f
> 
> Unfortunately, the commit message in your git repo is lacking.
> I've used the text at the top of your email.

It's just a temporary fork and I'm too lazy ))

> Eeep, Evolution does some strange things with formatting
> whitespaces.  It looks like instructions for making it nicer are
> in the Linux kernel:
> 
> https://bogomips.org/mirrors/linux.git/plain/Documentation/email-clients.txt?h=v4.8

Whoops, my fault. 

> > +  def test_multiline_header_0d0a
> > +    parser = HttpParser.new
> > +    parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
> > bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
> 
> I expect code to be wrapped at 80 lines or less.  Fixed locally.
> (I need big fonts, even 80 is a compromise, I really prefer 64)

Line 221 with "ssl bullshit" guided me to this.

> Anyways, pushed to the "rfc2616-sec19.3" branch.
> 
> I've also uploaded a prerelease 5.1.0.4.gd5fbb to RubyGems
> for folks without Ragel.
> 
> 	gem install --pre unicorn -v 5.1.0.4.gd5fbb
> 
> Anything else?  Expect a 5.2.0 release in a few days or so.
> Thanks.

Thanks for the quick acceptance.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add some tolerance (RFC2616 sec. 19.3)
  2016-10-20  9:05  4% [PATCH] Add some tolerance (RFC2616 sec. 19.3) Mishael A Sibiryakov
@ 2016-10-20 17:55  5% ` Eric Wong
  2016-10-20 20:25  6%   ` Mishael A Sibiryakov
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2016-10-20 17:55 UTC (permalink / raw)
  To: Mishael A Sibiryakov; +Cc: unicorn-public

Mishael A Sibiryakov <death@junki.org> wrote:
> Hi all.
> 
> We're implementing client certificate authentication with nginx and
> unicorn. 
> 
> Nginx configured in the following way:
> 
> proxy_set_header X-SSL-Client-Cert $ssl_client_cert;
> 
> When client submits certificate and nginx passes it to the unicorn,
> unicorn responds with 400 (Bad Request). This caused because nginx
> doesn't use "\r\n" they using just "\n" and multilne headers is failed
> to parse (I've added test).
> 
> Accorording to RFC2616 section 19.3:
> https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.3
> 
> "The line terminator for message-header fields is the sequence CRLF.
> However, we recommend that applications, when parsing such headers,
> recognize a single LF as a line terminator and ignore the leading CR."
> 
> CRLF changed to ("\r\n" | "\n")

Thanks for that useful explanation.  Aside from the unnecessary,
"Hi all,", that is an informative commit message which justifies
the usefulness of that patch.

> Github commit https://github.com/uno4ki/unicorn/commit/ed127b66e162aaf1
> 76de05720f6be758f8b41b1f

Unfortunately, the commit message in your git repo is lacking.
I've used the text at the top of your email.

> PS: Googling "nginx unicorn ssl_client_cert" shows the problem. 
> 
> ---
>  ext/unicorn_http/unicorn_http_common.rl |  2 +-
>  test/unit/test_http_parser.rb           | 16 ++++++++++++++++
>  2 files changed, 17 insertions(+), 1 deletion(-)

Eeep, Evolution does some strange things with formatting
whitespaces.  It looks like instructions for making it nicer are
in the Linux kernel:

https://bogomips.org/mirrors/linux.git/plain/Documentation/email-clients.txt?h=v4.8

<snip>

> +  def test_multiline_header_0d0a
> +    parser = HttpParser.new
> +    parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
> bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"

I expect code to be wrapped at 80 lines or less.  Fixed locally.
(I need big fonts, even 80 is a compromise, I really prefer 64)

Anyways, pushed to the "rfc2616-sec19.3" branch.

I've also uploaded a prerelease 5.1.0.4.gd5fbb to RubyGems
for folks without Ragel.

	gem install --pre unicorn -v 5.1.0.4.gd5fbb

Anything else?  Expect a 5.2.0 release in a few days or so.
Thanks.

^ permalink raw reply	[relevance 5%]

* [PATCH] Add some tolerance (RFC2616 sec. 19.3)
@ 2016-10-20  9:05  4% Mishael A Sibiryakov
  2016-10-20 17:55  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Mishael A Sibiryakov @ 2016-10-20  9:05 UTC (permalink / raw)
  To: unicorn-public

Hi all.

We're implementing client certificate authentication with nginx and
unicorn. 

Nginx configured in the following way:

proxy_set_header X-SSL-Client-Cert $ssl_client_cert;

When client submits certificate and nginx passes it to the unicorn,
unicorn responds with 400 (Bad Request). This caused because nginx
doesn't use "\r\n" they using just "\n" and multilne headers is failed
to parse (I've added test).

Accorording to RFC2616 section 19.3:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.3

"The line terminator for message-header fields is the sequence CRLF.
However, we recommend that applications, when parsing such headers,
recognize a single LF as a line terminator and ignore the leading CR."

CRLF changed to ("\r\n" | "\n")

Github commit https://github.com/uno4ki/unicorn/commit/ed127b66e162aaf1
76de05720f6be758f8b41b1f


PS: Googling "nginx unicorn ssl_client_cert" shows the problem. 

---
 ext/unicorn_http/unicorn_http_common.rl |  2 +-
 test/unit/test_http_parser.rb           | 16 ++++++++++++++++
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/ext/unicorn_http/unicorn_http_common.rl
b/ext/unicorn_http/unicorn_http_common.rl
index cc1d455..507e570 100644
--- a/ext/unicorn_http/unicorn_http_common.rl
+++ b/ext/unicorn_http/unicorn_http_common.rl
@@ -4,7 +4,7 @@
 
 #### HTTP PROTOCOL GRAMMAR
 # line endings
-  CRLF = "\r\n";
+  CRLF = ("\r\n" | "\n");
 
 # character types
   CTL = (cntrl | 127);
diff --git a/test/unit/test_http_parser.rb
b/test/unit/test_http_parser.rb
index c72f7f2..4b1a16e 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -230,6 +230,22 @@ def test_nasty_pound_header
     assert_equal expect, req['HTTP_X_SSL_BULLSHIT']
   end
 
+  def test_multiline_header_0d0a
+    parser = HttpParser.new
+    parser.buf << "GET / HTTP/1.0\r\nX-Multiline-Header: foo
bar\r\n\tcha cha\r\n\tzha zha\r\n\r\n"
+    req = parser.env
+    assert_equal req, parser.parse
+    assert_equal 'foo bar cha cha zha zha',
req['HTTP_X_MULTILINE_HEADER']
+  end
+
+  def test_multiline_header_0a
+    parser = HttpParser.new
+    parser.buf << "GET / HTTP/1.0\nX-Multiline-Header: foo bar\n\tcha
cha\n\tzha zha\n\n"
+    req = parser.env
+    assert_equal req, parser.parse
+    assert_equal 'foo bar cha cha zha zha',
req['HTTP_X_MULTILINE_HEADER']
+  end
+
   def test_continuation_eats_leading_spaces
     parser = HttpParser.new
     header = "GET / HTTP/1.1\r\n" \
-- 
2.10.1

^ permalink raw reply related	[relevance 4%]

* Re: Please join the Rack Specification discussion for `env['upgrade.websocket']`
  2016-08-06  3:09  3%   ` Boaz Segev
@ 2016-08-06  4:59  0%     ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-08-06  4:59 UTC (permalink / raw)
  To: Boaz Segev; +Cc: unicorn-public

Boaz Segev <boaz@2be.co.il> wrote:
> Running two (or more) event loops, each with it's own
> resources, is wasteful and promotes needless context switches.
> 
> There is no reason to hijack a socket when the server can
> easily provide callbacks for IO related events using it's
> existing established event loop.

As Michael said, WS is a really bad fit for unicorn :)

> This alone should provide a performance boost. There are other
> considerations as well, but I think they all derive from this
> core principle of having the web server retain control over
> all network IO concerns.

For anybody else writing an epoll-based server with multiple
epoll-based event loops in different threads, the following is
good ordering; relatively cheap and scalable:

	epoll_ctl(original_epfd, EPOLL_CTL_DEL, client_fd, NULL);
	epoll_ctl(websocket_epfd, EPOLL_CTL_ADD, client_fd, &ev);

Change the order, and it's slower and less scalable:

	/* in other words, don't do this in this order */
	epoll_ctl(websocket_epfd, EPOLL_CTL_ADD, client_fd, &ev);
	epoll_ctl(original_epfd, EPOLL_CTL_DEL, client_fd, NULL);

The former ensures a simple topology where the file behind
client_fd only exists in one eventpoll rbtree at a time.  The
latter creates a complex topology which needs to do additional
locking and scanning to prevent infinite loops.

This is true since Linux 3.13, commit 67347fe4e632633
("epoll: do not take global 'epmutex' for simple topologies")
 https://bogomips.org/mirrors/linux.git/commit?id=67347fe4e632633

> like hijacking, it's optional.
> 
> Unicorn was design for very specific use cases, so I
> definitely understand if this might not be as interesting for
> you. 

Yes, unicorn itself will never have a multi-client-aware event
loop.  It is out-of-scope for its design, it seems WebSockets
usually have idle times which would cause any server with the
design of unicorn to fall over.  unicorn cannot even handle
HTTP/1.1 persistent connections.

> However, I do think your experience as developers could help
> enrich us all. If you have any comments or anything to add
> regarding the proposed `websocket.upgrade` specification, your
> voice is welcome:

Sorry, not interested, and definitely not touching centralized
message boards :>

^ permalink raw reply	[relevance 0%]

* Re: Please join the Rack Specification discussion for `env['upgrade.websocket']`
  @ 2016-08-06  3:09  3%   ` Boaz Segev
  2016-08-06  4:59  0%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Boaz Segev @ 2016-08-06  3:09 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

Eric, thank you for your review and thoughts on the matter.

You raised a few questions, please allow me to both answer and clarify the intention behind the proposed extension.

>> I represent an effort to extend Rack so that it allows server-side
>> websocket upgrade implementation support and pure Rack websocket
>> applications.
>> 
>> This new Rack feature proposal is gaining support, with over 42
>> support votes in the [original Rack discussion
>> thread](https://github.com/rack/rack/issues/1093).
> 
> You mention performance several times, but I am not sure what
> you mean.  unicorn completely stops caring for a socket after
> it's hijacked by the app.  In other words:
> 
>  Your Rack app could take the socket and inject it into an
>  event loop running in a separate thread.  That event loop
>  could be 100% C code running without GVL for all unicorn
>  cares.


Running two (or more) event loops, each with it's own resources, is wasteful and promotes needless context switches.

There is no reason to hijack a socket when the server can easily provide callbacks for IO related events using it's existing established event loop.

This alone should provide a performance boost. There are other considerations as well, but I think they all derive from this core principle of having the web server retain control over all network IO concerns.

> Also, Ruby IO.select also supports arbitrary number of
> descriptors in some cases (Linux for sure, and probably most
> server-oriented *BSDs).  Of course, the malloc usage + O(n)
> behavior still suck, but no, select(2) is not always limited to
> <=1024 FDs.


Although this isn't as relevant to the proposed specification, it is a good example for how intricate network programming details are unknown by most Ruby programmers and shows why it would be better for the web server to retain control over all network IO concerns.

For example, the Linux man page expressly states that "To monitor file descriptors greater than 1023, use poll(2) instead"

This is true even when using select for a single FD that has a value of more then 1023, since `select` is often implemented using a bit vector map of 1024 bits that are controlled using the FD_(*) macros.

See, for example: http://linux-tips.org/t/is-it-possible-to-listen-file-descriptor-greater-than-1024-with-select/45/2

Ruby's IO.select uses this same system call and suffers the same issues that are present in the OS - each OS with it's own quirks.

I don't think application developers should worry about these details when these issues were already resolved during the server's designed.

By having the server provide callbacks - instead of hijacking - we eliminate a hoard of potential bugs and considerations.

>  unicorn would only parse the first HTTP request (with the
>  Upgrade header) before the Rack app hijacks it.


Of course, not every server has to offer this feature - just like hijacking, it's optional.

Unicorn was design for very specific use cases, so I definitely understand if this might not be as interesting for you. 

However, I do think your experience as developers could help enrich us all. If you have any comments or anything to add regarding the proposed `websocket.upgrade` specification, your voice is welcome:

https://github.com/boazsegev/iodine/issues/6



^ permalink raw reply	[relevance 3%]

* [PATCH] examples/logrotate.conf: update example for systemd
@ 2016-06-20 20:00  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-06-20 20:00 UTC (permalink / raw)
  To: unicorn-public

...And add placeholders for other systems
---
  Leaving out the part for using a backup service :>

	for i in /var/log/unicorn_app/*.log.gz
	do
		curl -sSf -T $i cloud.nsa.gov/store && rm $i
	done

 examples/logrotate.conf | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/examples/logrotate.conf b/examples/logrotate.conf
index 03fefc6..437f6c6 100644
--- a/examples/logrotate.conf
+++ b/examples/logrotate.conf
@@ -3,6 +3,9 @@
 #
 # See the logrotate(8) manpage for more information:
 #    http://linux.die.net/man/8/logrotate
+#
+# public logrotate-related discussion in our archives:
+#    https://bogomips.org/unicorn-public/?q=logrotate
 
 # Modify the following glob to match the logfiles your app writes to:
 /var/log/unicorn_app/*.log {
@@ -22,7 +25,19 @@
 	# config.  Unicorn supports the USR1 signal and we send it
 	# as our "lastaction" action:
 	lastaction
-		# assuming your pid file is in /var/run/unicorn_app/pid
+		# For systemd users, assuming you use two services
+		# (as recommended) to allow zero-downtime upgrades.
+		# Only one service needs to be started, but signaling
+		# both here is harmless as long as they're both enabled
+		systemctl kill -s SIGUSR1 unicorn@1.service
+		systemctl kill -s SIGUSR1 unicorn@2.service
+
+		# Examples for other process management systems appreciated
+		# Mail us at unicorn-public@bogomips.org
+		# (see above for archives)
+
+		# If you use a pid file and assuming your pid file
+		# is in /var/run/unicorn_app/pid
 		pid=/var/run/unicorn_app/pid
 		test -s $pid && kill -USR1 "$(cat $pid)"
 	endscript

^ permalink raw reply related	[relevance 4%]

* [PATCH] examples/init.sh: update to reduce upgrade raciness
  @ 2016-06-07 21:13  3%     ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-06-07 21:13 UTC (permalink / raw)
  To: Jesper Rønn-Jensen; +Cc: unicorn-public

Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> On Tue, Jun 7, 2016 at 3:41 PM, Eric Wong <e@80x24.org> wrote:
> > Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
> >> +++ b/examples/init.sh
> >> @@ -45,7 +45,7 @@ restart|reload)
> >>   $CMD
> >>   ;;
> >>  upgrade)
> >> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
> >> + if sig USR2 && sleep 2 && sig 0
> >>   then
> >>   n=$TIMEOUT
> >>   while test -s $old_pid && test $n -ge 0
> >> --
> >
> > I actually wrote a better init script for a similar server,
> > patch coming in a bit (or later, close to falling asleep).
>
> Thanks for input.
> 
> I will close this issue and wait for your patch later, then.
 
Proposed patch below (for "git am --scissors" users)
I've pushed this out to the "jr/init" branch on the git repo
and you can download the blob directly, here:

https://bogomips.org/unicorn.git/plain/examples/init.sh?h=jr/init

Will rebase/amend this branch as necessary until merged into master.

> Will also close pull-request I put at github with reference to this thread.

Please do, thank you.

Their terms-of-service doesn't allow bots to auto-close and redirect
things; and I can't support any centralized messaging system.

> >         Anyways if you stick to old-school Usenet/mailing list posting
> >         conventions, I'm more than happy to help you :)
 
> PS. Thanks for tip on HTML vs Plain text mails. Didn't know Gmail
> could send plain text :)

I think the only problem with gmail I've heard was from mobile
phone app users.  At least the git ML and LKML gets text-only
gmail posts all the time.

We old-school conventions mean we also trim our quotes and
avoid top-posting :)

------------8<----------------
Subject: [PATCH] examples/init.sh: update to reduce upgrade raciness
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Rework the "upgrade" target to only read the PID files once to
avoid misreading the wrong PID files in the middle of the
upgrade.

Additionally, introduce the UPGRADE_DELAY environment parameter
so users can increase/decrease according to their application
startup time.

PID files are inherently racy and people should be using a
process manager (systemd or similar) instead, but this should
mitigate most of the problems with the old target.

While we're at it, add LSB tags for systems which complain
about the lack of them and modernize things a bit using
$(command) construct instead of the more fragile `command`.

Thanks-to: Jesper Rønn-Jensen <jesperrr@gmail.com>
---
 examples/init.sh | 44 ++++++++++++++++++++++++++++++++++++--------
 1 file changed, 36 insertions(+), 8 deletions(-)

diff --git a/examples/init.sh b/examples/init.sh
index 1f0e035..4ef6cdc 100644
--- a/examples/init.sh
+++ b/examples/init.sh
@@ -1,7 +1,16 @@
 #!/bin/sh
 set -e
+### BEGIN INIT INFO
+# Provides:          unicorn
+# Required-Start:    $local_fs $network
+# Required-Stop:     $local_fs $network
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# Short-Description: Start/stop unicorn Rack app server
+### END INIT INFO
+
 # Example init script, this can be used with nginx, too,
-# since nginx and unicorn accept the same signals
+# since nginx and unicorn accept the same signals.
 
 # Feel free to change any of the following variables for your app:
 TIMEOUT=${TIMEOUT-60}
@@ -9,21 +18,22 @@ APP_ROOT=/home/x/my_app/current
 PID=$APP_ROOT/tmp/pids/unicorn.pid
 CMD="/usr/bin/unicorn -D -c $APP_ROOT/config/unicorn.rb"
 INIT_CONF=$APP_ROOT/config/init.conf
+UPGRADE_DELAY=${UPGRADE_DELAY-2}
 action="$1"
 set -u
 
 test -f "$INIT_CONF" && . $INIT_CONF
 
-old_pid="$PID.oldbin"
+OLD="$PID.oldbin"
 
 cd $APP_ROOT || exit 1
 
 sig () {
-	test -s "$PID" && kill -$1 `cat $PID`
+	test -s "$PID" && kill -$1 $(cat $PID)
 }
 
 oldsig () {
-	test -s $old_pid && kill -$1 `cat $old_pid`
+	test -s "$OLD" && kill -$1 $(cat $OLD)
 }
 
 case $action in
@@ -45,18 +55,36 @@ restart|reload)
 	$CMD
 	;;
 upgrade)
-	if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
+	if oldsig 0
+	then
+		echo >&2 "Old upgraded process still running with $OLD"
+		exit 1
+	fi
+
+	cur_pid=
+	if test -s "$PID"
+	then
+		cur_pid=$(cat $PID)
+	fi
+
+	if test -n "$cur_pid" &&
+			kill -USR2 "$cur_pid" &&
+			sleep $UPGRADE_DELAY &&
+			new_pid=$(cat $PID) &&
+			test x"$new_pid" != x"$cur_pid" &&
+			kill -0 "$new_pid" &&
+			kill -QUIT "$cur_pid"
 	then
 		n=$TIMEOUT
-		while test -s $old_pid && test $n -ge 0
+		while kill -0 "$cur_pid" 2>/dev/null && test $n -ge 0
 		do
 			printf '.' && sleep 1 && n=$(( $n - 1 ))
 		done
 		echo
 
-		if test $n -lt 0 && test -s $old_pid
+		if test $n -lt 0 && kill -0 "$cur_pid" 2>/dev/null
 		then
-			echo >&2 "$old_pid still exists after $TIMEOUT seconds"
+			echo >&2 "$cur_pid still running after $TIMEOUT seconds"
 			exit 1
 		fi
 		exit 0
-- 
EW

^ permalink raw reply related	[relevance 3%]

* Re: if you want to unsubscribe
  @ 2016-06-07 14:14  4% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-06-07 14:14 UTC (permalink / raw)
  To: unicorn-public

Eric Wong <e@80x24.org> wrote:
> mlmmj[1] needs to know YOUR email address when you email:
> 
> 	unicorn-public+unsubscribe@bogomips.org
> 
> Do NOT use "unroll.me" or similar services.
> 
> They are meant for unsubscribing from marketing lists with unique
> unsubscription addresses for each recipient.  That's not how
> discussion list managers like mlmmj, ezmlm, or Mailman work.

Actually, if the unique URL is in the List-Unsubscribe header,
it won't be accidentally quoted in discussions.
So I just enabled some gory changes which should add a unique,
per-user URL to the List-Unsubscribe: header.

You should see something like the following in your headers:

List-Unsubscribe: <mailto:unicorn-public+unsubscribe@bogomips.org>,
   <https://bogomips.org/u/2my2pvPLJPZlWCLwyy5VMiO9t/unicorn-public>

The https:// link should be one-click, no-confirmation,
unsubscribe which hopefully makes SORBS and similar services happy.
Yes, that's a destructive GET request with no confirmation.

And barely-tested :)

> [1] http://mlmmj.org/
>
> Dealing with bounces and unsubscription is why I prefer people
> read mailing lists over NNTP or HTTP:

Soon, it will be possible to subscribe using POP3, too,
totally anonymously using a UUID you generate.
It actually makes sense with all the webmail providers which
support it.

> nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
> nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
> http://bogomips.org/unicorn-public/
> 
> You'll never need a subscription or even a valid reply
> address to mail this list.

HTTPS is also available, as long as Let's Encrypt runs:
https://bogomips.org/unicorn-public/

And Tor hidden service:

http://ou63pmih66umazou.onion/unicorn-public/
nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn
torsocks git clone git://ou63pmih66umazou.onion/unicorn # for code

I don't know how stable the hidden service names will be given
how short they are; I know Tor devs are planning a transition
to longer, harder-to-collide ones.
All I know is, there will be more hidden service mirrors.

[1] https://public-inbox.org/meta/20160520213655.19381-1-e@80x24.org/T/
    git clone git://80x24.org/public-inbox

^ permalink raw reply	[relevance 4%]

* [PUSHED] various documentation updates
@ 2016-01-07  3:41  2% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2016-01-07  3:41 UTC (permalink / raw)
  To: unicorn-public

* add nntp_url to the olddoc website footer
* update legacy support status for 4.x (not 4.8.x)
* update copyright range to 2016
* note all of our development tools are Free Software, too
* remove cgit mention; it may not always be cgit
  (but URLs should remain compatible).
* discourage downloading snapshot tarballs;
  "git clone" + periodic "git fetch" is more efficient
* remove most mentions of unicorn_rails as that
  was meant for ancient Rails 1.x/2.x users
* update path reference to Ruby 2.3.0
* fix nginx upstream module link to avoid redirect
* shorten Message-ID example to avoid redirects
  and inadvertant linkage
---
  Also pushed to the website http://unicorn.bogomips.org/
  (using olddoc.git @ c98abe82b6b3 from git://80x24.org/olddoc.git)

  Curious, does anybody out there use Rails 2.x or earlier?

 .olddoc.yml                       |  1 +
 Documentation/unicorn.1.txt       |  1 -
 Documentation/unicorn_rails.1.txt |  2 +-
 HACKING                           |  2 +-
 README                            | 17 +++++------------
 lib/unicorn/configurator.rb       |  5 +++--
 lib/unicorn/http_server.rb        |  4 ++--
 7 files changed, 13 insertions(+), 19 deletions(-)

diff --git a/.olddoc.yml b/.olddoc.yml
index 063c1c6..cda8ac3 100644
--- a/.olddoc.yml
+++ b/.olddoc.yml
@@ -13,3 +13,4 @@ noindex:
 - unicorn_rails_1
 public_email: unicorn-public@bogomips.org
 private_email: unicorn@bogomips.org
+nntp_url: nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
diff --git a/Documentation/unicorn.1.txt b/Documentation/unicorn.1.txt
index efdda4b..3f20a9a 100644
--- a/Documentation/unicorn.1.txt
+++ b/Documentation/unicorn.1.txt
@@ -175,7 +175,6 @@ the unicorn config file.
 
 # SEE ALSO
 
-* unicorn_rails(1)
 * *Rack::Builder* ri/RDoc
 * *Unicorn::Configurator* ri/RDoc
 * [Unicorn RDoc][1]
diff --git a/Documentation/unicorn_rails.1.txt b/Documentation/unicorn_rails.1.txt
index bff703e..2ce7501 100644
--- a/Documentation/unicorn_rails.1.txt
+++ b/Documentation/unicorn_rails.1.txt
@@ -4,7 +4,7 @@
 
 # NAME
 
-unicorn_rails - a script/server-like command to launch the Unicorn HTTP server
+unicorn_rails - unicorn launcher for Rails 1.x and 2.x users
 
 # SYNOPSIS
 
diff --git a/HACKING b/HACKING
index 6c5f897..0fa22cd 100644
--- a/HACKING
+++ b/HACKING
@@ -57,7 +57,7 @@ Please wrap documentation at 72 characters-per-line or less (long URLs
 are exempt) so it is comfortably readable from terminals.
 
 When referencing mailing list posts, use
-"http://bogomips.org/unicorn-public/m/$MESSAGE_ID.html" if possible
+<tt>http://bogomips.org/unicorn-public/$MESSAGE_ID/</tt> if possible
 since the Message-ID remains searchable even if a particular site
 becomes unavailable.
 
diff --git a/README b/README
index db9f0d4..11de938 100644
--- a/README
+++ b/README
@@ -13,7 +13,7 @@ both the the request and response in between unicorn and slow clients.
   {nginx}[http://nginx.org/] or {Rack}[http://rack.github.io/].
 
 * Compatible with Ruby 1.9.3 and later.
-  unicorn 4.8.x will remain supported for Ruby 1.8 users.
+  unicorn 4.x remains supported for Ruby 1.8 users.
 
 * Process management: unicorn will reap and restart workers that
   die from broken apps.  There is no need to manage multiple processes
@@ -60,7 +60,7 @@ both the the request and response in between unicorn and slow clients.
 
 == License
 
-unicorn is copyright 2009 by all contributors (see logs in git).
+unicorn is copyright 2009-2016 by all contributors (see logs in git).
 It is based on Mongrel 1.1.5.
 Mongrel is copyright 2007 Zed A. Shaw and contributors.
 
@@ -68,7 +68,7 @@ unicorn is licensed under (your choice) of the GPLv2 or later
 (GPLv3+ preferred), or Ruby (1.8)-specific terms.
 See the included LICENSE file for details.
 
-unicorn is 100% Free Software.
+unicorn is 100% Free Software (including all development tools used).
 
 == Install
 
@@ -85,10 +85,9 @@ You can get the latest source via git from the following locations
   git://bogomips.org/unicorn.git
   git://repo.or.cz/unicorn.git (mirror)
 
-You may browse the code from the web and download the latest snapshot
-tarballs here:
+You may browse the code from the web:
 
-* http://bogomips.org/unicorn.git (cgit)
+* http://bogomips.org/unicorn.git
 * http://repo.or.cz/w/unicorn.git (gitweb)
 
 See the HACKING guide on how to contribute and build prerelease gems
@@ -102,12 +101,6 @@ In APP_ROOT, run:
 
   unicorn
 
-=== Ancient Rails 1.2 - 2.x versions
-
-In RAILS_ROOT, run:
-
-  unicorn_rails
-
 unicorn will bind to all interfaces on TCP port 8080 by default.
 You may use the +--listen/-l+ switch to bind to a different
 address:port or a UNIX socket.
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 4da19bb..948c6e3 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -181,8 +181,6 @@ def before_exec(*args, &block)
   # to have nginx always retry backends that may have had workers
   # SIGKILL-ed due to timeouts.
   #
-  #    # See http://wiki.nginx.org/NginxHttpUpstreamModule for more details
-  #    # on nginx upstream configuration:
   #    upstream unicorn_backend {
   #      # for UNIX domain socket setups:
   #      server unix:/path/to/.unicorn.sock fail_timeout=0;
@@ -192,6 +190,9 @@ def before_exec(*args, &block)
   #      server 192.168.0.8:8080 fail_timeout=0;
   #      server 192.168.0.9:8080 fail_timeout=0;
   #    }
+  #
+  # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html
+  # for more details on nginx upstream configuration.
   def timeout(seconds)
     set_int(:timeout, seconds, 3)
     # POSIX says 31 days is the smallest allowed maximum timeout for select()
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ca56ed3..741cca5 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -36,7 +36,7 @@ class Unicorn::HttpServer
   # or even different installations of the same applications without
   # downtime.  Keys of this constant Hash are described as follows:
   #
-  # * 0 - the path to the unicorn/unicorn_rails executable
+  # * 0 - the path to the unicorn executable
   # * :argv - a deep copy of the ARGV array the executable originally saw
   # * :cwd - the working directory of the application, this is where
   # you originally started Unicorn.
@@ -45,7 +45,7 @@ class Unicorn::HttpServer
   # you can set the following in your Unicorn config file, HUP and then
   # continue with the traditional USR2 + QUIT upgrade steps:
   #
-  #   Unicorn::HttpServer::START_CTX[0] = "/home/bofh/2.2.0/bin/unicorn"
+  #   Unicorn::HttpServer::START_CTX[0] = "/home/bofh/2.3.0/bin/unicorn"
   START_CTX = {
     :argv => ARGV.map(&:dup),
     0 => $0.dup,
-- 
EW

^ permalink raw reply related	[relevance 2%]

* [PATCH] add .gitattributes for Ruby method detection
@ 2015-11-17 22:06  6% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-11-17 22:06 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong

The "diff" function detection for C does not map well to
Ruby files, take advantage of gitattributes(5) to improve
method name detection in generated patches as well as
making "git diff -W" output more useful.
---
 .gitattributes | 5 +++++
 1 file changed, 5 insertions(+)
 create mode 100644 .gitattributes

diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..21e0bd4
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,5 @@
+*.gemspec diff=ruby
+*.rb diff=ruby
+*.ru diff=ruby
+Rakefile diff=ruby
+bin/* diff=ruby
-- 
EW


^ permalink raw reply related	[relevance 6%]

* [PATCH] examples: add systemd socket and service files
@ 2015-11-17 21:45  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-11-17 21:45 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong

Since we have init scripts, we ought to have the equivalent for
systemd users who cannot upgrade via the normal SIGUSR2 mechanism;
but can use multiple services: "unicorn@1" + h"unicorn@2" to
accomplish the same thing.

Based on examples by Christos Trochalakis <yatiohi@ideopolis.gr>

ref:
http://bogomips.org/unicorn-public/20150708130821.GA1361@luke.ws.skroutz.gr/
---
 examples/unicorn.socket   | 11 +++++++++++
 examples/unicorn@.service | 26 ++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
 create mode 100644 examples/unicorn.socket
 create mode 100644 examples/unicorn@.service

diff --git a/examples/unicorn.socket b/examples/unicorn.socket
new file mode 100644
index 0000000..7d5f773
--- /dev/null
+++ b/examples/unicorn.socket
@@ -0,0 +1,11 @@
+# ==> /etc/systemd/system/unicorn.socket <==
+[Unit]
+Description = unicorn sockets
+
+[Socket]
+ListenStream = 127.0.0.1:8080
+ListenStream = /tmp/path/to/.unicorn.sock
+Service = unicorn@1.service
+
+[Install]
+WantedBy = sockets.target
diff --git a/examples/unicorn@.service b/examples/unicorn@.service
new file mode 100644
index 0000000..b058da5
--- /dev/null
+++ b/examples/unicorn@.service
@@ -0,0 +1,26 @@
+# ==> /etc/systemd/system/unicorn@.service <==
+# Since SIGUSR2 upgrades do not work under systemd, this service file
+# allows starting two simultaneous services during upgrade time
+# (e.g. unicorn@1 unicorn@2) with the intention that they take
+# turns running in-between upgrades.  This should allow upgrading
+# without downtime.
+
+[Unit]
+Description = unicorn Rack application server %i
+Wants = unicorn.socket
+After = unicorn.socket
+
+[Service]
+ExecStart = /usr/bin/unicorn -c /path/to/unicorn.conf.rb /path/to/config.ru
+Sockets = unicorn.socket
+KillSignal = SIGQUIT
+User = nobody
+Group = nogroup
+ExecReload = /bin/kill -HUP $MAINPID
+
+# This is based on the Unicorn::Configurator#timeout directive,
+# adding a few seconds for scheduling differences:
+TimeoutStopSec = 62
+
+[Install]
+WantedBy = multi-user.target
-- 
EW


^ permalink raw reply related	[relevance 5%]

* [PATCH 3/3] doc updates
  @ 2015-11-01  8:37  4% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-11-01  8:37 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong

ISSUES: note images are considered spam as well as HTML.

Links: Clarify we may only endorse the Free versions of nginx, not the
non-Free versions.

Add a link to Starman as a unicorn derivative, as I even use Starman
myself.  Remove yahns, since it's really the complete opposite of
unicorn and probably not appropriate to place next to Starman and
gunicorn
---
 ISSUES | 2 +-
 Links  | 7 ++++---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/ISSUES b/ISSUES
index 7c91555..394c852 100644
--- a/ISSUES
+++ b/ISSUES
@@ -9,7 +9,7 @@ submit patches and/or obtain support after you have searched the
 * Cc: all participants in a thread or commit, as subscription is optional
 * Do not {top post}[http://catb.org/jargon/html/T/top-post.html] in replies
 * Quote as little as possible of the message you're replying to
-* Do not send HTML mail, it will be flagged as spam
+* Do not send HTML mail or images, it will be flagged as spam
 * Anonymous and pseudonymous messages will always be welcome.
 * The email submission port (587) is enabled on the bogomips.org MX:
   http://bogomips.org/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
diff --git a/Links b/Links
index 16c9467..6474a9d 100644
--- a/Links
+++ b/Links
@@ -37,14 +37,15 @@ or services behind them.
 * {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
   Rack and unicorn
 
-* {nginx}[http://nginx.org/] - the reverse proxy for use with unicorn
+* {nginx}[http://nginx.org/] (Free versions) -
+  the reverse proxy for use with unicorn
 
 === Derivatives
 
 * {Green Unicorn}[http://gunicorn.org/] - a Python version of unicorn
 
-* {yahns}[http://yahns.yhbt.net/] - the complete opposite of unicorn in
-  every imaginable way.  Designed for energy efficiency on idle sites.
+* {Starman}[http://search.cpan.org/dist/Starman/] - Plack/PSGI version
+  of unicorn
 
 === Prior Work
 
-- 
EW


^ permalink raw reply related	[relevance 4%]

* [PATCH 2/2] inheriting sockets from UNICORN_FD does not close them
  @ 2015-10-27  3:33  3% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-10-27  3:33 UTC (permalink / raw)
  To: unicorn-public

For some reason, I thought invalid descriptors passed to UNICORN_FD
would be automatically closed by the master process; but apparently
this hasn't been the case.  On the other hand, this bug has been
around for over 6 years now and nobody noticed or cared enough to
tell us, so fixing it might break existing setups.

Since there may be users relying on this behavior, we cannot change
the behavior anymore; so update the documentation and write at test
to ensure we can never "fix" this bug at the expense of breaking
any working setups which may be out there.

Keep in mind that a before_exec hook may always be used to modify
the UNICORN_FD environment by setting the close_on_exec flag and
removing the appropriate descriptor from the environment.

I originally intended to add the ability to inherit new listeners
without a config file specification so systemd users can avoid
repeating themselves in the systemd and unicorn config files,
but apparently there is nothing to change in our code.
---
 Documentation/unicorn.1.txt |  4 +---
 test/exec/test_exec.rb      | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/Documentation/unicorn.1.txt b/Documentation/unicorn.1.txt
index 193860f..5b82ad5 100644
--- a/Documentation/unicorn.1.txt
+++ b/Documentation/unicorn.1.txt
@@ -166,9 +166,7 @@ variable internally when doing transparent upgrades.
 UNICORN_FD is a comma-delimited list of one or more file descriptors
 used to implement USR2 upgrades.  Init systems may bind listen sockets
 itself and spawn unicorn with UNICORN_FD set to the file descriptor
-numbers of the listen socket(s).  The unicorn CONFIG_FILE must still
-have the inherited listen socket parameters defined as in a normal
-startup, otherwise the socket will be closed.
+numbers of the listen socket(s).
 
 # SEE ALSO
 
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index af6f151..ca0b7bc 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -128,6 +128,26 @@ def test_sd_listen_fds_emulation
     # [ruby-core:69895] [Bug #11336] fixed by r51576
   end if RUBY_VERSION.to_f >= 2.3
 
+  def test_inherit_listener_unspecified
+    File.open("config.ru", "wb") { |fp| fp.write(HI) }
+    sock = TCPServer.new(@addr, @port)
+    sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
+
+    pid = xfork do
+      redirect_test_io do
+        ENV['UNICORN_FD'] = sock.fileno.to_s
+        exec($unicorn_bin, sock.fileno => sock.fileno)
+      end
+    end
+    res = hit(["http://#@addr:#@port/"])
+    assert_equal [ "HI\n" ], res
+    assert_shutdown(pid)
+    assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
+                'unicorn should always set SO_KEEPALIVE on inherited sockets'
+  ensure
+    sock.close if sock
+  end
+
   def test_working_directory_rel_path_config_file
     other = Tempfile.new('unicorn.wd')
     File.unlink(other.path)
-- 
EW


^ permalink raw reply related	[relevance 3%]

* [PATCH] doc: update mail archive info
@ 2015-10-05  2:43  7% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-10-05  2:43 UTC (permalink / raw)
  To: unicorn-public

public-inbox supports read-only NNTP access nowadays to make it
easier to follow archives.  It is read-only to encourage Cc:-ing
all participants (which avoids reliance on the few-points-of-failure
behavior of NNTP).  Unlike email, NNTP also lacks good anti-spam
filtering.

Additionally, the gmane group also got redirected to the
bogomips.org address at some point since RubyForge died.

While we're at it, link to my post about enabling the submission
port (587).  It's been a year and nothing bad has happened, yet.

Finally, remove most of the documentation for ssoma since it's
unlikely anybody will use it given the existence of NNTP access.
It did little besides clutter the page.  However, git:// (used
by ssoma) remains strictly more efficient than NNTP.

Vebavpnyyl, gur AAGC freire sbe choyvp-vaobk pna unaqyr
gubhfnaqf bs fybj pyvragf.  Fbzrguvat havpbea jvyy arire or noyr
gb qb :C
---
 ISSUES | 45 ++++++++++++++++++---------------------------
 README |  5 +++++
 2 files changed, 23 insertions(+), 27 deletions(-)

diff --git a/ISSUES b/ISSUES
index b172f8a..7c91555 100644
--- a/ISSUES
+++ b/ISSUES
@@ -11,6 +11,8 @@ submit patches and/or obtain support after you have searched the
 * Quote as little as possible of the message you're replying to
 * Do not send HTML mail, it will be flagged as spam
 * Anonymous and pseudonymous messages will always be welcome.
+* The email submission port (587) is enabled on the bogomips.org MX:
+  http://bogomips.org/unicorn-public/20141004232241.GA23908@dcvr.yhbt.net/t/
 
 If your issue is of a sensitive nature or you're just shy in public,
 then feel free to email us privately at mailto:unicorn@bogomips.org
@@ -63,39 +65,28 @@ document distributed with git) on guidelines for patch submission.
 
 * public: mailto:unicorn-public@bogomips.org
 * private: mailto:unicorn@bogomips.org
+* nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
+* http://bogomips.org/unicorn-public/
 
 We operate a {public-inbox}[http://public-inbox.org/] which
-feeds the mailing list.  You may subscribe either using
-{ssoma}[http://ssoma.public-inbox.org/] or by sending a mail
-to mailto:unicorn-public+subscribe@bogomips.org
+feeds the mailing list.  Subscription is optional, so Cc:
+all participants.
 
-ssoma is a mail archiver/fetcher using git.  It operates in a similar
-fashion to tools such as slrnpull, fetchmail, or getmail.  ssoma
-subscription instructions:
+You can follow along via NNTP:
 
-	URL=git://bogomips.org/unicorn-public
-	LISTNAME=unicorn
+	nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
+	nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
 
-	# to initialize a maildir (this may be a new or existing maildir,
-	# ssoma will not touch existing messages)
-	# If you prefer mbox, use mbox:/path/to/mbox as the last argument
-	# You may also use imap://$MAILSERVER/INBOX for an IMAP account
-	# or imaps:// for an IMAPS account, as well.
-	ssoma add $LISTNAME $URL maildir:/path/to/maildir
+Or Atom feeds:
 
-	# read with your favorite MUA (only using mutt as an example)
-	mutt -f /path/to/maildir # (or /path/to/mbox)
+	http://bogomips.org/unicorn-public/new.atom
 
-	# to keep your mbox or maildir up-to-date, periodically run the following:
-	ssoma sync $LISTNAME
+	The HTML archives at http://bogomips.org/unicorn-public/
+	also has links to per-thread Atom feeds and downloadable
+	mboxes.
 
-	# your MUA may modify and delete messages from the maildir or mbox,
-	# this does not affect ssoma functionality at all
+You may also subscribe via plain-text email:
 
-	# to sync all your ssoma subscriptions
-	ssoma sync
-
-	# You may wish to sync in your cronjob
-	ssoma sync --cron
-
-HTML archives are available here: http://bogomips.org/unicorn-public/
+	mailto:unicorn-public+subscribe@bogomips.org
+	(and confirming the auto-reply)
diff --git a/README b/README
index dc121d3..db9f0d4 100644
--- a/README
+++ b/README
@@ -140,6 +140,11 @@ All feedback (bug reports, user/development dicussion, patches, pull
 requests) go to the mailing list/newsgroup.  See the ISSUES document for
 information on the {mailing list}[mailto:unicorn-public@bogomips.org].
 
+The mailing list is archived at http://bogomips.org/unicorn-public/
+Read-only NNTP access is available at:
+nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn and
+nntp://news.gmane.org/gmane.comp.lang.ruby.unicorn.general
+
 For the latest on unicorn releases, you may also finger us at
 unicorn@bogomips.org or check our NEWS page (and subscribe to our Atom
 feed).
-- 
EW


^ permalink raw reply related	[relevance 7%]

* [PATCH] doc: remove references to old servers
@ 2015-07-15 22:05  2% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-07-15 22:05 UTC (permalink / raw)
  To: unicorn-public

They'll continue to be maintained, but we're no longer advertising
them.  Also, favor lowercase "unicorn" while we're at it since that
matches the executable and gem name to avoid unnecessary escaping
for RDoc.
---
 Application_Timeouts             |  6 +++---
 KNOWN_ISSUES                     | 14 +++++++-------
 Links                            | 30 ++++++++++++++----------------
 PHILOSOPHY                       |  6 ------
 README                           | 32 ++++++++++++++++----------------
 Sandbox                          |  2 +-
 TUNING                           | 10 +++++-----
 examples/nginx.conf              | 21 ++++++++++-----------
 ext/unicorn_http/unicorn_http.rl |  2 +-
 lib/unicorn.rb                   |  6 +++---
 lib/unicorn/configurator.rb      | 24 ++++++++++--------------
 lib/unicorn/http_server.rb       |  4 ++--
 lib/unicorn/socket_helper.rb     |  8 +++-----
 lib/unicorn/util.rb              |  2 +-
 lib/unicorn/worker.rb            |  4 ++--
 15 files changed, 78 insertions(+), 93 deletions(-)

diff --git a/Application_Timeouts b/Application_Timeouts
index 5f0370d..561a1cc 100644
--- a/Application_Timeouts
+++ b/Application_Timeouts
@@ -4,10 +4,10 @@ This article focuses on _application_ setup for Rack applications, but
 can be expanded to all applications that connect to external resources
 and expect short response times.
 
-This article is not specific to \Unicorn, but exists to discourage
+This article is not specific to unicorn, but exists to discourage
 the overuse of the built-in
 {timeout}[link:Unicorn/Configurator.html#method-i-timeout] directive
-in \Unicorn.
+in unicorn.
 
 == ALL External Resources Are Considered Unreliable
 
@@ -71,7 +71,7 @@ handle network/server failures.
 == The Last Line Of Defense
 
 The {timeout}[link:Unicorn/Configurator.html#method-i-timeout] mechanism
-in \Unicorn is an extreme solution that should be avoided whenever
+in unicorn is an extreme solution that should be avoided whenever
 possible.  It will help catch bugs in your application where and when
 your application forgets to use timeouts, but it is expensive as it
 kills and respawns a worker process.
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index 1950223..6b80517 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -13,7 +13,7 @@ acceptable solution.  Those issues are documented here.
 
 * PRNGs (pseudo-random number generators) loaded before forking
   (e.g. "preload_app true") may need to have their internal state
-  reset in the after_fork hook.  Starting with \Unicorn 3.6.1, we
+  reset in the after_fork hook.  Starting with unicorn 3.6.1, we
   have builtin workarounds for Kernel#rand and OpenSSL::Random users,
   but applications may use other PRNGs.
 
@@ -36,13 +36,13 @@ acceptable solution.  Those issues are documented here.
 
 * Under some versions of Ruby 1.8, it is necessary to call +srand+ in an
   after_fork hook to get correct random number generation.  We have a builtin
-  workaround for this starting with \Unicorn 3.6.1
+  workaround for this starting with unicorn 3.6.1
 
   See http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/36450
 
 * On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
   stdio that causes failure for file uploads larger than 112K.  Upgrade
-  your version of Ruby or continue using Unicorn 1.x/3.4.x.
+  your version of Ruby or continue using unicorn 1.x/3.4.x.
 
 * Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will
   segfault if called after forking.  Upgrade to Ruby 1.9.2 or call
@@ -53,12 +53,12 @@ acceptable solution.  Those issues are documented here.
 
 * Rails 2.3.2 bundles its own version of Rack.  This may cause subtle
   bugs when simultaneously loaded with the system-wide Rack Rubygem
-  which Unicorn depends on.  Upgrading to Rails 2.3.4 (or later) is
+  which unicorn depends on.  Upgrading to Rails 2.3.4 (or later) is
   strongly recommended for all Rails 2.3.x users for this (and security
   reasons).  Rails 2.2.x series (or before) did not bundle Rack and are
   should be unnaffected.  If there is any reason which forces your
   application to use Rails 2.3.2 and you have no other choice, then
-  you may edit your Unicorn gemspec and remove the Rack dependency.
+  you may edit your unicorn gemspec and remove the Rack dependency.
 
   ref: http://mid.gmane.org/20091014221552.GA30624@dcvr.yhbt.net
   Note: the workaround described in the article above only made
@@ -71,9 +71,9 @@ acceptable solution.  Those issues are documented here.
     set :env, :production
     set :run, false
   Since this is no longer an issue with Sinatra 0.9.x apps, this will not be
-  fixed on our end.  Since Unicorn is itself the application launcher, the
+  fixed on our end.  Since unicorn is itself the application launcher, the
   at_exit handler used in old Sinatra always caused Mongrel to be launched
-  whenever a Unicorn worker was about to exit.
+  whenever a unicorn worker was about to exit.
 
   Also remember we're capable of replacing the running binary without dropping
   any connections regardless of framework :)
diff --git a/Links b/Links
index 5a586c1..ad05afd 100644
--- a/Links
+++ b/Links
@@ -1,13 +1,13 @@
 = Related Projects
 
-If you're interested in \Unicorn, you may be interested in some of the projects
+If you're interested in unicorn, you may be interested in some of the projects
 listed below.  If you have any links to add/change/remove, please tell us at
 mailto:unicorn-public@bogomips.org!
 
 == Disclaimer
 
-The \Unicorn project is not responsible for the content in these links.
-Furthermore, the \Unicorn project has never, does not and will never endorse:
+The unicorn project is not responsible for the content in these links.
+Furthermore, the unicorn project has never, does not and will never endorse:
 
 * any for-profit entities or services
 * any non-{Free Software}[http://www.gnu.org/philosophy/free-sw.html]
@@ -15,13 +15,13 @@ Furthermore, the \Unicorn project has never, does not and will never endorse:
 The existence of these links does not imply endorsement of any entities
 or services behind them.
 
-=== For use with \Unicorn
+=== For use with unicorn
 
 * {Bluepill}[https://github.com/arya/bluepill] -
   a simple process monitoring tool written in Ruby
 
 * {golden_brindle}[https://github.com/simonoff/golden_brindle] - tool to
-  manage multiple \Unicorn instances/applications on a single server
+  manage multiple unicorn instances/applications on a single server
 
 * {raindrops}[http://raindrops.bogomips.org/] - real-time stats for
   preforking Rack servers
@@ -29,27 +29,25 @@ or services behind them.
 * {UnXF}[http://bogomips.org/unxf/]  Un-X-Forward* the Rack environment,
   useful since unicorn is designed to be deployed behind a reverse proxy.
 
-=== \Unicorn is written to work with
+=== unicorn is written to work with
 
 * {Rack}[http://rack.github.io/] - a minimal interface between webservers
   supporting Ruby and Ruby frameworks
 
 * {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
-  Rack and \Unicorn
+  Rack and unicorn
 
-* {nginx}[http://nginx.org/] - the reverse proxy for use with \Unicorn
+* {nginx}[http://nginx.org/] - the reverse proxy for use with unicorn
 
-* {kgio}[http://bogomips.org/kgio/] - the I/O library written for \Unicorn
+* {kgio}[http://bogomips.org/kgio/] - the I/O library written for unicorn
+  (deprecated and functionality being mainlined into Ruby)
 
 === Derivatives
 
-* {Green Unicorn}[http://gunicorn.org/] - a Python version of \Unicorn
+* {Green Unicorn}[http://gunicorn.org/] - a Python version of unicorn
 
-* {Rainbows!}[http://rainbows.bogomips.org/] - \Unicorn for sleepy
-  apps and slow clients (historical).
-
-* {yahns}[http://yahns.yhbt.net/] - like Rainbows!, but with fewer options
-  and designed for energy efficiency on idle sites.
+* {yahns}[http://yahns.yhbt.net/] - the complete opposite of unicorn in
+  every imaginable way.  Designed for energy efficiency on idle sites.
 
 === Prior Work
 
@@ -57,4 +55,4 @@ or services behind them.
   unicorn is based on
 
 * {david}[http://bogomips.org/david.git] - a tool to explain why you need
-  nginx in front of \Unicorn
+  nginx in front of unicorn
diff --git a/PHILOSOPHY b/PHILOSOPHY
index 18b2d82..feb83d9 100644
--- a/PHILOSOPHY
+++ b/PHILOSOPHY
@@ -137,9 +137,3 @@ unicorn is highly inefficient for Comet/reverse-HTTP/push applications
 where the HTTP connection spends a large amount of time idle.
 Nevertheless, the ease of troubleshooting, debugging, and management of
 unicorn may still outweigh the drawbacks for these applications.
-
-The {Rainbows!}[http://rainbows.bogomips.org/] aims to fill the gap for
-odd corner cases where the nginx + unicorn combination is not enough.
-While Rainbows! management/administration is largely identical to
-unicorn, Rainbows! is far more ambitious and has seen little real-world
-usage.
diff --git a/README b/README
index bd626e9..dc121d3 100644
--- a/README
+++ b/README
@@ -1,10 +1,10 @@
-= Unicorn: Rack HTTP server for fast clients and Unix
+= unicorn: Rack HTTP server for fast clients and Unix
 
-\Unicorn is an HTTP server for Rack applications designed to only serve
+unicorn is an HTTP server for Rack applications designed to only serve
 fast clients on low-latency, high-bandwidth connections and take
 advantage of features in Unix/Unix-like kernels.  Slow clients should
 only be served by placing a reverse proxy capable of fully buffering
-both the the request and response in between \Unicorn and slow clients.
+both the the request and response in between unicorn and slow clients.
 
 == Features
 
@@ -15,9 +15,9 @@ both the the request and response in between \Unicorn and slow clients.
 * Compatible with Ruby 1.9.3 and later.
   unicorn 4.8.x will remain supported for Ruby 1.8 users.
 
-* Process management: \Unicorn will reap and restart workers that
+* Process management: unicorn will reap and restart workers that
   die from broken apps.  There is no need to manage multiple processes
-  or ports yourself.  \Unicorn can spawn and manage any number of
+  or ports yourself.  unicorn can spawn and manage any number of
   worker processes you choose to scale to your backend.
 
 * Load balancing is done entirely by the operating system kernel.
@@ -33,11 +33,11 @@ both the the request and response in between \Unicorn and slow clients.
 * Builtin reopening of all log files in your application via
   USR1 signal.  This allows logrotate to rotate files atomically and
   quickly via rename instead of the racy and slow copytruncate method.
-  \Unicorn also takes steps to ensure multi-line log entries from one
+  unicorn also takes steps to ensure multi-line log entries from one
   request all stay within the same file.
 
 * nginx-style binary upgrades without losing connections.
-  You can upgrade \Unicorn, your entire application, libraries
+  You can upgrade unicorn, your entire application, libraries
   and even your Ruby interpreter without dropping clients.
 
 * before_fork and after_fork hooks in case your application
@@ -60,15 +60,15 @@ both the the request and response in between \Unicorn and slow clients.
 
 == License
 
-\Unicorn is copyright 2009 by all contributors (see logs in git).
+unicorn is copyright 2009 by all contributors (see logs in git).
 It is based on Mongrel 1.1.5.
 Mongrel is copyright 2007 Zed A. Shaw and contributors.
 
-\Unicorn is licensed under (your choice) of the GPLv2 or later
+unicorn is licensed under (your choice) of the GPLv2 or later
 (GPLv3+ preferred), or Ruby (1.8)-specific terms.
 See the included LICENSE file for details.
 
-\Unicorn is 100% Free Software.
+unicorn is 100% Free Software.
 
 == Install
 
@@ -108,17 +108,17 @@ In RAILS_ROOT, run:
 
   unicorn_rails
 
-\Unicorn will bind to all interfaces on TCP port 8080 by default.
+unicorn will bind to all interfaces on TCP port 8080 by default.
 You may use the +--listen/-l+ switch to bind to a different
 address:port or a UNIX socket.
 
 === Configuration File(s)
 
-\Unicorn will look for the config.ru file used by rackup in APP_ROOT.
+unicorn will look for the config.ru file used by rackup in APP_ROOT.
 
-For deployments, it can use a config file for \Unicorn-specific options
+For deployments, it can use a config file for unicorn-specific options
 specified by the +--config-file/-c+ command-line switch.  See
-Unicorn::Configurator for the syntax of the \Unicorn-specific options.
+Unicorn::Configurator for the syntax of the unicorn-specific options.
 The default settings are designed for maximum out-of-the-box
 compatibility with existing applications.
 
@@ -130,7 +130,7 @@ supported.  Run `unicorn -h` to see command-line options.
 There is NO WARRANTY whatsoever if anything goes wrong, but
 {let us know}[link:ISSUES.html] and we'll try our best to fix it.
 
-\Unicorn is designed to only serve fast clients either on the local host
+unicorn is designed to only serve fast clients either on the local host
 or a fast LAN.  See the PHILOSOPHY and DESIGN documents for more details
 regarding this.
 
@@ -140,6 +140,6 @@ All feedback (bug reports, user/development dicussion, patches, pull
 requests) go to the mailing list/newsgroup.  See the ISSUES document for
 information on the {mailing list}[mailto:unicorn-public@bogomips.org].
 
-For the latest on \Unicorn releases, you may also finger us at
+For the latest on unicorn releases, you may also finger us at
 unicorn@bogomips.org or check our NEWS page (and subscribe to our Atom
 feed).
diff --git a/Sandbox b/Sandbox
index a6c3fe7..997b92f 100644
--- a/Sandbox
+++ b/Sandbox
@@ -1,4 +1,4 @@
-= Tips for using \Unicorn with Sandbox installation tools
+= Tips for using unicorn with Sandbox installation tools
 
 Since unicorn includes executables and is usually used to start a Ruby
 process, there are certain caveats to using it with tools that sandbox
diff --git a/TUNING b/TUNING
index 6a6d7db..247090b 100644
--- a/TUNING
+++ b/TUNING
@@ -1,10 +1,10 @@
-= Tuning \Unicorn
+= Tuning unicorn
 
-\Unicorn performance is generally as good as a (mostly) Ruby web server
+unicorn performance is generally as good as a (mostly) Ruby web server
 can provide.  Most often the performance bottleneck is in the web
 application running on Unicorn rather than Unicorn itself.
 
-== \Unicorn Configuration
+== unicorn Configuration
 
 See Unicorn::Configurator for details on the config file format.
 +worker_processes+ is the most-commonly needed tuning parameter.
@@ -14,7 +14,7 @@ See Unicorn::Configurator for details on the config file format.
 * worker_processes should be scaled to the number of processes your
   backend system(s) can support.  DO NOT scale it to the number of
   external network clients your application expects to be serving.
-  \Unicorn is NOT for serving slow clients, that is the job of nginx.
+  unicorn is NOT for serving slow clients, that is the job of nginx.
 
 * worker_processes should be *at* *least* the number of CPU cores on
   a dedicated server (unless you do not have enough memory).
@@ -58,7 +58,7 @@ See Unicorn::Configurator for details on the config file format.
 * UNIX domain sockets are slightly faster than TCP sockets, but only
   work if nginx is on the same machine.
 
-== Other \Unicorn settings
+== Other unicorn settings
 
 * Setting "preload_app true" can allow copy-on-write-friendly GC to
   be used to save memory.  It will probably not work out of the box with
diff --git a/examples/nginx.conf b/examples/nginx.conf
index a68fe6f..0583c1f 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -1,5 +1,5 @@
 # This is example contains the bare mininum to get nginx going with
-# Unicorn or Rainbows! servers.  Generally these configuration settings
+# unicorn servers.  Generally these configuration settings
 # are applicable to other HTTP application servers (and not just Ruby
 # ones), so if you have one working well for proxying another app
 # server, feel free to continue using it.
@@ -44,8 +44,8 @@ http {
   # click tracking!
   access_log /path/to/nginx.access.log combined;
 
-  # you generally want to serve static files with nginx since neither
-  # Unicorn nor Rainbows! is optimized for it at the moment
+  # you generally want to serve static files with nginx since
+  # unicorn is not and will never be optimized for it
   sendfile on;
 
   tcp_nopush on; # off may be better for *some* Comet/long-poll stuff
@@ -67,10 +67,10 @@ http {
              text/javascript application/x-javascript
              application/atom+xml;
 
-  # this can be any application server, not just Unicorn/Rainbows!
+  # this can be any application server, not just unicorn
   upstream app_server {
     # fail_timeout=0 means we always retry an upstream even if it failed
-    # to return a good HTTP response (in case the Unicorn master nukes a
+    # to return a good HTTP response (in case the unicorn master nukes a
     # single worker for timing out).
 
     # for UNIX domain socket setups:
@@ -132,12 +132,11 @@ http {
       # redirects, we set the Host: header above already.
       proxy_redirect off;
 
-      # set "proxy_buffering off" *only* for Rainbows! when doing
-      # Comet/long-poll/streaming.  It's also safe to set if you're using
-      # only serving fast clients with Unicorn + nginx, but not slow
-      # clients.  You normally want nginx to buffer responses to slow
-      # clients, even with Rails 3.1 streaming because otherwise a slow
-      # client can become a bottleneck of Unicorn.
+      # It's also safe to set if you're using only serving fast clients
+      # with unicorn + nginx, but not slow clients.  You normally want
+      # nginx to buffer responses to slow clients, even with Rails 3.1
+      # streaming because otherwise a slow client can become a bottleneck
+      # of unicorn.
       #
       # The Rack application may also set "X-Accel-Buffering (yes|no)"
       # in the response headers do disable/enable buffering on a
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index a5f069d..046ccb5 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -38,7 +38,7 @@ static VALUE set_maxhdrlen(VALUE self, VALUE len)
   return UINT2NUM(MAX_HEADER_LEN = NUM2UINT(len));
 }
 
-/* keep this small for Rainbows! since every client has one */
+/* keep this small for other servers (e.g. yahns) since every client has one */
 struct http_parser {
   int cs; /* Ragel internal state */
   unsigned int flags;
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 9fdcb8e..b0e6bd1 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -10,10 +10,10 @@ require 'kgio'
 # enough functionality to service web application requests fast as possible.
 # :startdoc:
 
-# \Unicorn exposes very little of an user-visible API and most of its
-# internals are subject to change.  \Unicorn is designed to host Rack
+# unicorn exposes very little of an user-visible API and most of its
+# internals are subject to change.  unicorn is designed to host Rack
 # applications, so applications should be written against the Rack SPEC
-# and not \Unicorn internals.
+# and not unicorn internals.
 module Unicorn
 
   # Raised inside TeeInput when a client closes the socket inside the
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 02f6b6b..4da19bb 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,7 +1,7 @@
 # -*- encoding: binary -*-
 require 'logger'
 
-# Implements a simple DSL for configuring a \Unicorn server.
+# Implements a simple DSL for configuring a unicorn server.
 #
 # See http://unicorn.bogomips.org/examples/unicorn.conf.rb and
 # http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
@@ -282,20 +282,19 @@ class Unicorn::Configurator
   #   Setting this to +true+ can make streaming responses in Rails 3.1
   #   appear more quickly at the cost of slightly higher bandwidth usage.
   #   The effect of this option is most visible if nginx is not used,
-  #   but nginx remains highly recommended with \Unicorn.
+  #   but nginx remains highly recommended with unicorn.
   #
   #   This has no effect on UNIX sockets.
   #
-  #   Default: +true+ (Nagle's algorithm disabled) in \Unicorn,
-  #   +true+ in Rainbows!  This defaulted to +false+ in \Unicorn
-  #   3.x
+  #   Default: +true+ (Nagle's algorithm disabled) in unicorn
+  #   This defaulted to +false+ in unicorn 3.x
   #
   # [:tcp_nopush => true or false]
   #
   #   Enables/disables TCP_CORK in Linux or TCP_NOPUSH in FreeBSD
   #
   #   This prevents partial TCP frames from being sent out and reduces
-  #   wakeups in nginx if it is on a different machine.  Since \Unicorn
+  #   wakeups in nginx if it is on a different machine.  Since unicorn
   #   is only designed for applications that send the response body
   #   quickly without keepalive, sockets will always be flushed on close
   #   to prevent delays.
@@ -303,7 +302,7 @@ class Unicorn::Configurator
   #   This has no effect on UNIX sockets.
   #
   #   Default: +false+
-  #   This defaulted to +true+ in \Unicorn 3.4 - 3.7
+  #   This defaulted to +true+ in unicorn 3.4 - 3.7
   #
   # [:ipv6only => true or false]
   #
@@ -387,12 +386,10 @@ class Unicorn::Configurator
   #   and +false+ or +nil+ is synonymous for a value of zero.
   #
   #   A value of +1+ is a good optimization for local networks
-  #   and trusted clients.  For Rainbows! and Zbatery users, a higher
-  #   value (e.g. +60+) provides more protection against some
-  #   denial-of-service attacks.  There is no good reason to ever
-  #   disable this with a +zero+ value when serving HTTP.
+  #   and trusted clients.  There is no good reason to ever
+  #   disable this with a +zero+ value with unicorn.
   #
-  #   Default: 1 retransmit for \Unicorn, 60 for Rainbows! 0.95.0\+
+  #   Default: 1
   #
   # [:accept_filter => String]
   #
@@ -401,8 +398,7 @@ class Unicorn::Configurator
   #   This enables either the "dataready" or (default) "httpready"
   #   accept() filter under FreeBSD.  This is intended as an
   #   optimization to reduce context switches with common GET/HEAD
-  #   requests.  For Rainbows! and Zbatery users, this provides
-  #   some protection against certain denial-of-service attacks, too.
+  #   requests.
   #
   #   There is no good reason to change from the default.
   #
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 0f97516..3dbfd3e 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -7,8 +7,8 @@
 #
 # Users do not need to know the internals of this class, but reading the
 # {source}[http://bogomips.org/unicorn.git/tree/lib/unicorn/http_server.rb]
-# is education for programmers wishing to learn how \Unicorn works.
-# See Unicorn::Configurator for information on how to configure \Unicorn.
+# is education for programmers wishing to learn how unicorn works.
+# See Unicorn::Configurator for information on how to configure unicorn.
 class Unicorn::HttpServer
   # :stopdoc:
   attr_accessor :app, :timeout, :worker_processes,
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 812ac53..df8315e 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -5,14 +5,12 @@ require 'socket'
 module Unicorn
   module SocketHelper
 
-    # internal interface, only used by Rainbows!/Zbatery
+    # internal interface
     DEFAULTS = {
       # The semantics for TCP_DEFER_ACCEPT changed in Linux 2.6.32+
       # with commit d1b99ba41d6c5aa1ed2fc634323449dd656899e9
-      # This change shouldn't affect Unicorn users behind nginx (a
-      # value of 1 remains an optimization), but Rainbows! users may
-      # want to use a higher value on Linux 2.6.32+ to protect against
-      # denial-of-service attacks
+      # This change shouldn't affect unicorn users behind nginx (a
+      # value of 1 remains an optimization).
       :tcp_defer_accept => 1,
 
       # FreeBSD, we need to override this to 'dataready' if we
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index c7784bd..2f8bfeb 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -1,7 +1,7 @@
 # -*- encoding: binary -*-
 
 require 'fcntl'
-module Unicorn::Util
+module Unicorn::Util # :nodoc:
 
 # :stopdoc:
   def self.is_log?(fp)
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index b3f8afe..6748a2f 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -3,8 +3,8 @@ require "raindrops"
 
 # This class and its members can be considered a stable interface
 # and will not change in a backwards-incompatible fashion between
-# releases of \Unicorn.  Knowledge of this class is generally not
-# not needed for most users of \Unicorn.
+# releases of unicorn.  Knowledge of this class is generally not
+# not needed for most users of unicorn.
 #
 # Some users may want to access it in the before_fork/after_fork hooks.
 # See the Unicorn::Configurator RDoc for examples.
-- 
EW


^ permalink raw reply related	[relevance 2%]

* Re: [DRE-maint] unicorn: native systemd service
  2015-06-27  3:05  5%       ` Eric Wong
  2015-06-27  4:01  4%         ` [RFC] emulate sd_listen_fds for systemd support Eric Wong
@ 2015-06-30  9:20  0%         ` Christos Trochalakis
  1 sibling, 0 replies; 200+ results
From: Christos Trochalakis @ 2015-06-30  9:20 UTC (permalink / raw)
  To: Eric Wong; +Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

On Sat, Jun 27, 2015 at 03:05:01AM +0000, Eric Wong wrote:
>Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
>> On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
>> >With socket activation, you should just be able to kill unicorn using
>> >SIGQUIT (just master, or even all workers) and restart without ever
>> >dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
>> >since that'll cause the master to kill all workers ASAP.
>>
>> Yes, you are right socket activation is also an option! I have made some
>> experiments with a simple rack app to test it.
>>
>> systemd uses the LISTEN_FDS env variable that is an integer indicating the
>> number of inherited file descriptors. Those FDs have consecutive numbers
>> starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
>>
>> So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
>> simple wrapper script for that. Here is the full configuration:
>
>OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
>directly so the wrapper is unnecessary.
>
><snip>
>
>> KillMode=mixed
>
>I don't think KillMode=mixed is necessary, here.  systemd can send
>SIGQUIT to workers.
>

Perhaps there is a race here, if a worker receives SIGQUIT first, and the
master respawns a new worker before receiving/handling its own SIGQUIT.
This is definitely a longshot, and will probably never happen, but
even then every process will eventually quit gracefully.

><snip>
>
>> TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
>> === sock is false). systemd socket files have support for most options now but
>> we might want unicorn to `setsockopt` them as well. For example,
>> 'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
>> they are not available in jessie (v215).
>
>They are now :)
>
>http://bogomips.org/unicorn-public/m/1435373879-4299-1-git-send-email-e@80x24.org.txt
>
>We don't have KeepAliveIntervalSec
>(aka TCP_KEEPINTVL) since it's not-portable and probably over
>Are you sure about that?

I added 'KeepAliveIntervalSec' by mistake. `KeepAlive`, a supported
option by unicorn, is included in jessie (systemd v215). Either way with
your last patch those options are enforced by unicorn.

>
>> socket activation is a really interesting setup, but personally I would not run
>> it with a large application. Clients would have to wait for the new master to
>> be up and running before a reply is returned, and that could take tenths of
>> seconds. The USR2 rexec solves that problem since both old and new workers are
>> accepting on the socket and we can kill the old ones when we are ready. In that
>> case the PIDFile trick is handy to support zero downtime restarts with no
>> latency.
>
>Is it possible to make systemd fire up two unicorn masters?
>That would be a nice feature to have with socket activation.

That would be great! I am a bit surprised that specifying multiple
services ('Service=' option) is not supported.



^ permalink raw reply	[relevance 0%]

* [RFC] emulate sd_listen_fds for systemd support
  2015-06-27  3:05  5%       ` Eric Wong
@ 2015-06-27  4:01  4%         ` Eric Wong
  2015-06-30  9:20  0%         ` [DRE-maint] unicorn: native systemd service Christos Trochalakis
  1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2015-06-27  4:01 UTC (permalink / raw)
  To: Christos Trochalakis
  Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

Unfortunately, testing this is tricky because FD=3
(SD_LISTEN_FDS_START) tends to be grabbed by (MRI) Ruby 1.9.3
and onwards for the internal self-pipe.

So for now, I've manually tested this with a systemd installation

Disclaimer: this is also my first experience using systemd
---
  Eric Wong <e@80x24.org> wrote:
  > OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
  > directly so the wrapper is unnecessary.

 lib/unicorn/http_server.rb | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 297b9f9..0f97516 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -766,12 +766,22 @@ def redirect_io(io, path)
   def inherit_listeners!
     # inherit sockets from parents, they need to be plain Socket objects
     # before they become Kgio::UNIXServer or Kgio::TCPServer
-    inherited = ENV['UNICORN_FD'].to_s.split(',').map do |fd|
-      io = Socket.for_fd(fd.to_i)
+    inherited = ENV['UNICORN_FD'].to_s.split(',')
+
+    # emulate sd_listen_fds() for systemd
+    sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
+    if sd_pid && sd_pid.to_i == $$
+      # 3 = SD_LISTEN_FDS_START
+      inherited.concat((3...(3 + sd_fds.to_i)).map { |fd| Socket.for_fd(fd) })
+    end
+    # to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
+
+    inherited.map! do |fd|
+      io = String === fd ? Socket.for_fd(fd.to_i) : fd
       io.autoclose = false
       io = server_cast(io)
       set_server_sockopt(io, listener_opts[sock_name(io)])
-      logger.info "inherited addr=#{sock_name(io)} fd=#{fd}"
+      logger.info "inherited addr=#{sock_name(io)} fd=#{io.fileno}"
       io
     end
 
-- 
EW

^ permalink raw reply related	[relevance 4%]

* Re: [DRE-maint] unicorn: native systemd service
  @ 2015-06-27  3:05  5%       ` Eric Wong
  2015-06-27  4:01  4%         ` [RFC] emulate sd_listen_fds for systemd support Eric Wong
  2015-06-30  9:20  0%         ` [DRE-maint] unicorn: native systemd service Christos Trochalakis
  0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2015-06-27  3:05 UTC (permalink / raw)
  To: Christos Trochalakis
  Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
> On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
> >With socket activation, you should just be able to kill unicorn using
> >SIGQUIT (just master, or even all workers) and restart without ever
> >dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
> >since that'll cause the master to kill all workers ASAP.
> 
> Yes, you are right socket activation is also an option! I have made some
> experiments with a simple rack app to test it.
> 
> systemd uses the LISTEN_FDS env variable that is an integer indicating the
> number of inherited file descriptors. Those FDs have consecutive numbers
> starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
> 
> So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
> simple wrapper script for that. Here is the full configuration:

OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
directly so the wrapper is unnecessary.

<snip>

> KillMode=mixed

I don't think KillMode=mixed is necessary, here.  systemd can send
SIGQUIT to workers.

<snip>

> TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
> === sock is false). systemd socket files have support for most options now but
> we might want unicorn to `setsockopt` them as well. For example,
> 'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
> they are not available in jessie (v215).

They are now :)

http://bogomips.org/unicorn-public/m/1435373879-4299-1-git-send-email-e@80x24.org.txt

We don't have KeepAliveIntervalSec
(aka TCP_KEEPINTVL) since it's not-portable and probably over
Are you sure about that?

> socket activation is a really interesting setup, but personally I would not run
> it with a large application. Clients would have to wait for the new master to
> be up and running before a reply is returned, and that could take tenths of
> seconds. The USR2 rexec solves that problem since both old and new workers are
> accepting on the socket and we can kill the old ones when we are ready. In that
> case the PIDFile trick is handy to support zero downtime restarts with no
> latency.

Is it possible to make systemd fire up two unicorn masters?
That would be a nice feature to have with socket activation.

^ permalink raw reply	[relevance 5%]

* Re: Does the the environment is preserved on USR2?
  2015-06-26 23:14  0%   ` Eric Wong
@ 2015-06-26 23:38  0%     ` Bráulio Bhavamitra
  0 siblings, 0 replies; 200+ results
From: Bráulio Bhavamitra @ 2015-06-26 23:38 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On Fri, Jun 26, 2015 at 8:14 PM, Eric Wong <e@80x24.org> wrote:
> Eric Wong <e@80x24.org> wrote:
>> USR2 should not change environment variables besides UNICORN_FD
>> and PWD (if working_directory is used).
>>
>> On most Linux versions, you can check the initial environment of a
>> running process by inspecting the environ file for the process:
>>
>>       tr '\0' '\n' < /proc/$PID/environ
>>
>> This may not reflect environment changes after the process is started.
>
> Actually, I noticed this doesn't even reflect ENV changes on
> fork from Ruby (but system("env") displays the correct result
> from a forked process) using Linux 4.0.6
>
>> However, you can use ENV inside Ruby code to check that.  Maybe add a
>> private Rack endpoint to show ENV.inspect output to your app to check
>> this...
>
> So use ENV from Ruby instead :)

Thanks Eric, will try this!

^ permalink raw reply	[relevance 0%]

* Re: Does the the environment is preserved on USR2?
  2015-06-26 22:31  5% ` Eric Wong
@ 2015-06-26 23:14  0%   ` Eric Wong
  2015-06-26 23:38  0%     ` Bráulio Bhavamitra
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-06-26 23:14 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: unicorn-public

Eric Wong <e@80x24.org> wrote:
> USR2 should not change environment variables besides UNICORN_FD
> and PWD (if working_directory is used).
> 
> On most Linux versions, you can check the initial environment of a
> running process by inspecting the environ file for the process:
> 
> 	tr '\0' '\n' < /proc/$PID/environ
> 
> This may not reflect environment changes after the process is started.

Actually, I noticed this doesn't even reflect ENV changes on
fork from Ruby (but system("env") displays the correct result
from a forked process) using Linux 4.0.6

> However, you can use ENV inside Ruby code to check that.  Maybe add a
> private Rack endpoint to show ENV.inspect output to your app to check
> this...

So use ENV from Ruby instead :)

^ permalink raw reply	[relevance 0%]

* Re: Does the the environment is preserved on USR2?
  @ 2015-06-26 22:31  5% ` Eric Wong
  2015-06-26 23:14  0%   ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-06-26 22:31 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: unicorn-public

USR2 should not change environment variables besides UNICORN_FD
and PWD (if working_directory is used).

On most Linux versions, you can check the initial environment of a
running process by inspecting the environ file for the process:

	tr '\0' '\n' < /proc/$PID/environ

This may not reflect environment changes after the process is started.

However, you can use ENV inside Ruby code to check that.  Maybe add a
private Rack endpoint to show ENV.inspect output to your app to check
this...

^ permalink raw reply	[relevance 5%]

* [ANN] unicorn 5.0.0.pre1 - incompatible changes!
@ 2015-06-15 22:56  6% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-15 22:56 UTC (permalink / raw)
  To: unicorn-public

This release finally drops Ruby 1.8 support and requires Ruby 1.9.3
or later.  The horrible "Status:" header in our HTTP response is
finally gone, saving at least 16 precious bytes in every single HTTP
response.

Under Ruby 2.1 and later, the monotonic clock is used for timeout
handling for better accuracy.

Several experimental, unused and undocumented features are removed.

There's also tiny, minor performance and memory improvements from
dropping 1.8 compatibility, but probably nothing noticeable on a
typical real-life (bloated) app.

The biggest performance improvement we made was to our website by
switching to olddoc.  Depending on connection speed, latency, and
renderer performance, it typically loads two to four times faster.

Finally, for the billionth time: unicorn must never be exposed
to slow clients, as it will never ever use new-fangled things
like non-blocking socket I/O, threads, epoll or kqueue.  unicorn
must be used with a fully-buffering reverse proxy such as nginx
for slow clients.

I'll tag 5.0.0 final in a week or so if all goes well

= gem install --pre unicorn
= git clone git://bogomips.org/unicorn.git
= http://unicorn.bogomips.org/

* ISSUES: update with mailing list subscription
* GIT-VERSION-GEN: start 5.0.0 development
* http: remove xftrust options
* FAQ: add entry for Rails autoflush_log
* dev: remove isolate dependency
* unicorn.gemspec: depend on test-unit 3.0
* http_response: remove Status: header
* remove RubyForge and Freecode references
* remove mongrel.rubyforge.org references
* http: remove the keepalive requests limit
* http: reduce parser from 72 to 56 bytes on 64-bit
* examples: add run_once to before_fork hook example
* worker: remove old tmp accessor
* http_server: save 450+ bytes of memory on x86-64
* t/t0002-parser-error.sh: relax test for rack 1.6.0
* remove SSL support
* tmpio: drop the "size" method
* switch docs + website to olddoc
* README: clarify/reduce references to unicorn_rails
* gemspec: fixup olddoc migration
* use the monotonic clock under Ruby 2.1+
* http: -Wshorten-64-to-32 warnings on clang
* remove old inetd+git examples and exec_cgi
* http: standalone require + reduction in binary size
* GNUmakefile: fix clean gem build + reduce build cruft
* socket_helper: reduce constant lookups and caching
* remove 1.8, <= 1.9.1 fallback for missing IO#autoclose=
* favor IO#close_on_exec= over fcntl in 1.9+
* use require_relative to reduce syscalls at startup
* doc: update support status for Ruby versions
* fix uninstalled testing and reduce require paths
* test_socket_helper: do not depend on SO_REUSEPORT
* favor "a.b(&:c)" form over "a.b { |x| x.c }"
* ISSUES: add section for bugs in other projects
* http_server: favor ivars over constants
* explain 11 byte magic number for self-pipe
* const: drop constants used by Rainbows!
* reduce and localize constant string use
* Links: mark Rainbows! as historical, reference yahns
* save about 200 bytes of memory on x86-64
* http: remove deprecated reset method
* http: remove experimental dechunk! method
* socket_helper: update comments
* doc: document UNICORN_FD in manpage
* doc: document Etc.nprocessors for worker_processes
* favor more string literals for cold call sites
* tee_input: support for Rack::TempfileReaper middleware
* support TempfileReaper in deployment and development envs
* favor kgio_wait_readable for single FD over select
* Merge tag 'v4.9.0'
* http_request: support rack.hijack by default
* avoid extra allocation for hijack proc creation
* FAQ: add note about ECONNRESET errors from bodies
* process SIGWINCH unless stdin is a TTY
* ISSUES: discourage HTML mail strongly, welcome nyms
* http: use rb_hash_clear in Ruby 2.0+
* http_response: avoid special-casing for Rack < 1.5
* www: install NEWS.atom.xml properly
* http_server: remove a few more accessors and constants
* http_response: simplify regular expression
* move the socket into Rack env for hijacking
* http: move response_start_sent into the C ext
* FAQ: reorder bit on Rack 1.1.x and Rails 2.3.x
* ensure body is closed during hijack

-- 
EW

^ permalink raw reply	[relevance 6%]

* [PATCH] http_server: remove a few more accessors and constants
@ 2015-06-04  1:39  2% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-06-04  1:39 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong

Unnecessarily exposed accessors and constants take up unnecessary
memory in constant/method tables as well as using extra space in
instruction sequences.

Preforking servers like unicorn are a bloated pigs anyways,
but saving a few hundred bytes here and there can add up and
make them marginally less bad.
---
 lib/unicorn/http_server.rb | 52 ++++++++++++++++++++++------------------------
 test/test_helper.rb        |  5 +++--
 2 files changed, 28 insertions(+), 29 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index cf8e122..2657c29 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -11,25 +11,23 @@
 # See Unicorn::Configurator for information on how to configure \Unicorn.
 class Unicorn::HttpServer
   # :stopdoc:
-  attr_accessor :app, :request, :timeout, :worker_processes,
+  attr_accessor :app, :timeout, :worker_processes,
                 :before_fork, :after_fork, :before_exec,
                 :listener_opts, :preload_app,
-                :reexec_pid, :orig_app, :init_listeners,
-                :master_pid, :config, :ready_pipe, :user
+                :orig_app, :config, :ready_pipe, :user
 
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
   include Unicorn::HttpResponse
 
   # all bound listener sockets
+  # note: this is public used by raindrops, but not recommended for use
+  # in new projects
   LISTENERS = []
 
   # listeners we have yet to bind
   NEW_LISTENERS = []
 
-  # list of signals we care about and trap in master.
-  QUEUE_SIGS = [ :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
-
   # :startdoc:
   # We populate this at startup so we can figure out how to reexecute
   # and upgrade the currently running instance of Unicorn
@@ -70,7 +68,7 @@ class Unicorn::HttpServer
   def initialize(app, options = {})
     @app = app
     @request = Unicorn::HttpRequest.new
-    self.reexec_pid = 0
+    @reexec_pid = 0
     options = options.dup
     @ready_pipe = options.delete(:ready_pipe)
     @init_listeners = options[:listeners] ? options[:listeners].dup : []
@@ -102,7 +100,10 @@ class Unicorn::HttpServer
     # monitoring tools may also rely on pid files existing before we
     # attempt to connect to the listener(s)
     config.commit!(self, :skip => [:listeners, :pid])
-    self.orig_app = app
+    @orig_app = app
+    # list of signals we care about and trap in master.
+    @queue_sigs = [
+      :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, :TTIN, :TTOU ]
   end
 
   # Runs the thing.  Returns self so you can run join on it
@@ -116,7 +117,7 @@ class Unicorn::HttpServer
     # setup signal handlers before writing pid file in case people get
     # trigger happy and send signals as soon as the pid file exists.
     # Note that signals don't actually get handled until the #join method
-    QUEUE_SIGS.each { |sig| trap(sig) { @sig_queue << sig; awaken_master } }
+    @queue_sigs.each { |sig| trap(sig) { @sig_queue << sig; awaken_master } }
     trap(:CHLD) { awaken_master }
 
     # write pid early for Mongrel compatibility if we're not inheriting sockets
@@ -186,7 +187,7 @@ class Unicorn::HttpServer
     if path
       if x = valid_pid?(path)
         return path if pid && path == pid && x == $$
-        if x == reexec_pid && pid.end_with?('.oldbin')
+        if x == @reexec_pid && pid.end_with?('.oldbin')
           logger.warn("will not set pid=#{path} while reexec-ed "\
                       "child is running PID:#{x}")
           return
@@ -387,9 +388,9 @@ class Unicorn::HttpServer
     begin
       wpid, status = Process.waitpid2(-1, Process::WNOHANG)
       wpid or return
-      if reexec_pid == wpid
+      if @reexec_pid == wpid
         logger.error "reaped #{status.inspect} exec()-ed"
-        self.reexec_pid = 0
+        @reexec_pid = 0
         self.pid = pid.chomp('.oldbin') if pid
         proc_name 'master'
       else
@@ -404,13 +405,13 @@ class Unicorn::HttpServer
 
   # reexecutes the START_CTX with a new binary
   def reexec
-    if reexec_pid > 0
+    if @reexec_pid > 0
       begin
-        Process.kill(0, reexec_pid)
-        logger.error "reexec-ed child already running PID:#{reexec_pid}"
+        Process.kill(0, @reexec_pid)
+        logger.error "reexec-ed child already running PID:#@reexec_pid"
         return
       rescue Errno::ESRCH
-        self.reexec_pid = 0
+        @reexec_pid = 0
       end
     end
 
@@ -428,7 +429,7 @@ class Unicorn::HttpServer
       end
     end
 
-    self.reexec_pid = fork do
+    @reexec_pid = fork do
       listener_fds = {}
       LISTENERS.each do |sock|
         sock.close_on_exec = false
@@ -576,9 +577,6 @@ class Unicorn::HttpServer
     handle_error(client, e)
   end
 
-  EXIT_SIGS = [ :QUIT, :TERM, :INT ]
-  WORKER_QUEUE_SIGS = QUEUE_SIGS - EXIT_SIGS
-
   def nuke_listeners!(readers)
     # only called from the worker, ordering is important here
     tmp = readers.dup
@@ -593,9 +591,10 @@ class Unicorn::HttpServer
   def init_worker_process(worker)
     worker.atfork_child
     # we'll re-trap :QUIT later for graceful shutdown iff we accept clients
-    EXIT_SIGS.each { |sig| trap(sig) { exit!(0) } }
-    exit!(0) if (@sig_queue & EXIT_SIGS)[0]
-    WORKER_QUEUE_SIGS.each { |sig| trap(sig, nil) }
+    exit_sigs = [ :QUIT, :TERM, :INT ]
+    exit_sigs.each { |sig| trap(sig) { exit!(0) } }
+    exit!(0) if (@sig_queue & exit_sigs)[0]
+    (@queue_sigs - exit_sigs).each { |sig| trap(sig, nil) }
     trap(:CHLD, 'DEFAULT')
     @sig_queue.clear
     proc_name "worker[#{worker.nr}]"
@@ -629,7 +628,7 @@ class Unicorn::HttpServer
   # for connections and doesn't die until the parent dies (or is
   # given a INT, QUIT, or TERM signal)
   def worker_loop(worker)
-    ppid = master_pid
+    ppid = @master_pid
     readers = init_worker_process(worker)
     nr = 0 # this becomes negative if we need to reopen logs
 
@@ -723,7 +722,7 @@ class Unicorn::HttpServer
     config.commit!(self)
     soft_kill_each_worker(:QUIT)
     Unicorn::Util.reopen_logs
-    self.app = orig_app
+    self.app = @orig_app
     build_app! if preload_app
     logger.info "done reloading config_file=#{config.config_file}"
   rescue StandardError, LoadError, SyntaxError => e
@@ -788,9 +787,8 @@ class Unicorn::HttpServer
   # call only after calling inherit_listeners!
   # This binds any listeners we did NOT inherit from the parent
   def bind_new_listeners!
-    NEW_LISTENERS.each { |addr| listen(addr) }
+    NEW_LISTENERS.each { |addr| listen(addr) }.clear
     raise ArgumentError, "no listeners" if LISTENERS.empty?
-    NEW_LISTENERS.clear
   end
 
   # try to use the monotonic clock in Ruby >= 2.1, it is immune to clock
diff --git a/test/test_helper.rb b/test/test_helper.rb
index c4fe07a..c21f75d 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -292,6 +292,7 @@ def chunked_spawn(stdout, *cmd)
 end
 
 def reset_sig_handlers
-  sigs = %w(CHLD).concat(Unicorn::HttpServer::QUEUE_SIGS)
-  sigs.each { |sig| trap(sig, "DEFAULT") }
+  %w(WINCH QUIT INT TERM USR1 USR2 HUP TTIN TTOU CHLD).each do |sig|
+    trap(sig, "DEFAULT")
+  end
 end
-- 
EW


^ permalink raw reply related	[relevance 2%]

* [PATCH] FAQ: add note about ECONNRESET errors from bodies
@ 2015-05-18 21:34  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-05-18 21:34 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong, Michael Fischer, Gabe Martin-Dempesy

Thanks to Michael Fischer and Gabe Martin-Dempesy for bringing this
to light on the mailing list.

Ref: <CABHxtY7Sn5yaiR5a3gDk1G4XySE+UtfuqUTcOSdmwneXLD5rcg@mail.gmail.com>
Ref: <FC91211E-FD32-432C-92FC-0318714C2170@zendesk.com>

Cc: Michael Fischer <mfischer@zendesk.com>
Cc: Gabe Martin-Dempesy <gabe@zendesk.com>
---
 FAQ | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/FAQ b/FAQ
index 66f1a09..fdf1400 100644
--- a/FAQ
+++ b/FAQ
@@ -1,5 +1,14 @@
 = Frequently Asked Questions about Unicorn
 
+=== Why is nginx getting ECONNRESET as a reverse proxy?
+
+Request body data (commonly from POST and PUT requests) may not be
+drained entirely by the application.  This may happen when request
+bodies are gzipped, as unicorn reads request body data lazily to avoid
+overhead from bad requests.
+
+Ref: http://mid.gmane.org/FC91211E-FD32-432C-92FC-0318714C2170@zendesk.com
+
 === Why aren't my Rails log files rotated when I use SIGUSR1?
 
 The Rails autoflush_log option must remain disabled with multiprocess
-- 
EW


^ permalink raw reply related	[relevance 5%]

* [ANN] unicorn 4.9.0 - Rack HTTP server for fast clients and *nix
@ 2015-04-24  3:17  6% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-04-24  3:17 UTC (permalink / raw)
  To: ruby-talk; +Cc: unicorn-public, Mike Mulvaney

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* http://unicorn.bogomips.org/
* public list: unicorn-public@bogomips.org
* mail archives: http://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml

Changes:

  unicorn 4.9.0 - TempfileReaper support in Rack 1.6

  This release supports the Rack::TempfileReaper middleware found
  in rack 1.6 for cleaning up disk space used by temporary files.
  We also use Rack::TempfileReaper for cleaning up large temporary
  files buffered with TeeInput.  Users on rack 1.5 and earlier
  will see no changes.

  There's also a bunch of documentation/build system improvements.

  This is likely to be the last Ruby 1.8-compatible release, unicorn 5.x
  will require 1.9.3 or later as well as dropping lots of cruft (the
  stupid "Status:" header in responses being the most notable).

  21 changes backported from master:

        ISSUES: update with mailing list subscription
        FAQ: add entry for Rails autoflush_log
        dev: remove isolate dependency
        unicorn.gemspec: depend on test-unit 3.0
        remove RubyForge and Freecode references
        remove mongrel.rubyforge.org references
        examples: add run_once to before_fork hook example
        t/t0002-parser-error.sh: relax test for rack 1.6.0
        switch docs + website to olddoc
        README: clarify/reduce references to unicorn_rails
        gemspec: fixup olddoc migration
        GNUmakefile: fix clean gem build + reduce build cruft
        doc: update support status for Ruby versions
        fix uninstalled testing and reduce require paths
        test_socket_helper: do not depend on SO_REUSEPORT
        ISSUES: add section for bugs in other projects
        explain 11 byte magic number for self-pipe
        Links: mark Rainbows! as historical, reference yahns
        doc: document UNICORN_FD in manpage
        tee_input: support for Rack::TempfileReaper middleware
        support TempfileReaper in deployment and development envs
-- 
EW

^ permalink raw reply	[relevance 6%]

* unicorn 4.8.x-stable branch pushed to git
@ 2015-04-22 19:02  6% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-04-22 19:02 UTC (permalink / raw)
  To: unicorn-public; +Cc: Mulvaney, Mike

Only backporting documentation + build/test system fixes, but I'll
probably apply and push the TeeInput patch in:
http://bogomips.org/unicorn-public/m/20150422183808.GA5277@dcvr.yhbt.net.txt

The following changes since commit 7087bb7ed5a1b9d9f24069cb92707d086668b6dc:

  unicorn 4.8.3 - the end of an era (2014-05-07 07:49:19 +0000)

are available in the git repository at:

  git://bogomips.org/unicorn 4.8.x-stable

for you to fetch changes up to 548e1e67d314f6ebd17df37ece0ee20632462f6f:

  doc: document UNICORN_FD in manpage (2015-04-22 18:57:39 +0000)

----------------------------------------------------------------
Eric Wong (19):
      ISSUES: update with mailing list subscription
      FAQ: add entry for Rails autoflush_log
      dev: remove isolate dependency
      unicorn.gemspec: depend on test-unit 3.0
      remove RubyForge and Freecode references
      remove mongrel.rubyforge.org references
      examples: add run_once to before_fork hook example
      t/t0002-parser-error.sh: relax test for rack 1.6.0
      switch docs + website to olddoc
      README: clarify/reduce references to unicorn_rails
      gemspec: fixup olddoc migration
      GNUmakefile: fix clean gem build + reduce build cruft
      doc: update support status for Ruby versions
      fix uninstalled testing and reduce require paths
      test_socket_helper: do not depend on SO_REUSEPORT
      ISSUES: add section for bugs in other projects
      explain 11 byte magic number for self-pipe
      Links: mark Rainbows! as historical, reference yahns
      doc: document UNICORN_FD in manpage

 .document                             |  1 -
 .gitignore                            |  4 +-
 .wrongdoc.yml => .olddoc.yml          |  6 ++-
 Documentation/unicorn.1.txt           |  7 ++++
 FAQ                                   | 10 ++++-
 GNUmakefile                           | 71 +++++++++++++----------------------
 HACKING                               | 31 ++++-----------
 ISSUES                                | 46 +++++++++++++++++++++--
 KNOWN_ISSUES                          | 20 +++++-----
 Links                                 |  5 ++-
 README                                | 10 ++---
 Rakefile                              | 44 ----------------------
 Sandbox                               |  2 +-
 examples/unicorn.conf.rb              | 11 ++++++
 lib/unicorn/configurator.rb           |  2 -
 lib/unicorn/http_server.rb            |  6 ++-
 local.mk.sample                       | 59 -----------------------------
 script/isolate_for_tests              | 31 ---------------
 t/GNUmakefile                         |  6 +--
 t/README                              |  2 +-
 t/t0002-parser-error.sh               |  6 +--
 test/exec/test_exec.rb                |  2 +-
 test/test_helper.rb                   |  4 +-
 test/unit/test_http_parser.rb         |  6 +--
 test/unit/test_http_parser_ng.rb      |  2 +-
 test/unit/test_http_parser_xftrust.rb |  2 +-
 test/unit/test_request.rb             |  2 +-
 test/unit/test_response.rb            |  6 +--
 test/unit/test_server.rb              |  6 +--
 test/unit/test_signals.rb             |  2 +-
 test/unit/test_socket_helper.rb       |  8 ++--
 test/unit/test_upload.rb              |  2 +-
 test/unit/test_util.rb                |  2 +-
 unicorn.gemspec                       | 13 +++----
 34 files changed, 167 insertions(+), 270 deletions(-)
 rename .wrongdoc.yml => .olddoc.yml (85%)
 delete mode 100644 local.mk.sample
 delete mode 100755 script/isolate_for_tests

^ permalink raw reply	[relevance 6%]

* Re: $USER and $HOME shell variables not set
  @ 2015-03-27 18:34  4% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-03-27 18:34 UTC (permalink / raw)
  To: Russell Jennings; +Cc: unicorn-public

Russell Jennings <violentpurr@gmail.com> wrote:
> Hello,
> 
> I am running into some issues with these variables being set - since I
> am spawning a script from a unicorn worker (via a rails controller) I
> figured I’d ask here. 
> 
> Here is the stackoverflow with the full background:
> http://stackoverflow.com/questions/29233181/why-is-envhome-nil-in-my-rake-task
> 
> in short: does unicorn have anything to do with $HOME or $USER not
> being defined?

Nope, unicorn itself does not change these variables.  We only set
UNICORN_FD (for SIGUSR2 upgrades), PWD (if working_directory is used).

Your init system may change users and clobber these options via
sudo/su/env and similar wrappers, so I think it has to do with how
you're starting unicorn.  If you're using sudo anywhere, the env_*
options from the sudoers files will also affect which envs get
clobbered/preserved/added.

> From what I can tell, that its unicorn is the only
> thing thats different versus running the same ruby via a rails console
> (which does indeed set those shell variables correctly)
> 
> in no mans land, so any hep or insight would be greatly appreciated. 

Can you add something to log the contents of ENV.inspect to
a log file.  Perhaps something like:

	Rails.logger.debug("env: #{ENV.inspect}")

It would also be helpful to show the snippet of code from where you're
running Rake in case you're accidentally setting an option wrong.


Under Linux, you can also inspect the original environment of any running
process from /proc/$PID/environ  In most cases/kernel versions, it won't
keep up with env changes once a process is running.
I use tr to replace '\0' with '\n' (newline) to help with readability

	tr '\0' '\n' </proc/$PID/environ

^ permalink raw reply	[relevance 4%]

* Re: [PATCH] ISSUES: add section for bugs in other projects
  2015-02-10 17:06  4% [PATCH] ISSUES: add section for bugs in other projects Eric Wong
@ 2015-02-10 17:26  5% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2015-02-10 17:26 UTC (permalink / raw)
  To: unicorn-public

Squashing the following, since we don't want things sinkholed
or missed:

--- a/ISSUES
+++ b/ISSUES
@@ -6,7 +6,7 @@ submit patches and/or obtain support after you have searched the
 {documentation}[http://unicorn.bogomips.org/].
 
 * No subscription will ever be required to email the public inbox.
-* Please Cc: all participants in a thread, as subscription is optional
+* Cc: all participants in a thread or commit, as subscription is optional
 * Do not {top post}[http://catb.org/jargon/html/T/top-post.html] in replies
 * Quote as little as possible of the message you're replying to
 * Do not send HTML mail, it will likely be flagged as spam
@@ -39,9 +39,13 @@ but their mailing list remains operational.
 
 Uncommon bugs we encounter in the Linux kernel should be Cc:-ed to the
 Linux kernel mailing list (LKML) at mailto:linux-kernel@vger.kernel.org
-and other kernel lists such as mailto:netdev@vger.kernel.org
-(for networking issues).  No subscription is necessary, and the our
-mailing list follows the same conventions as LKML for interopability.
+and subsystem maintainers such as mailto:netdev@vger.kernel.org
+(for networking issues).  It is expected practice to Cc: anybody
+involved with any problematic commits (including those in the
+Signed-off-by: and other trailer lines).  No subscription is necessary,
+and the our mailing list follows the same conventions as LKML for
+interopability.  There is a kernel.org Bugzilla instance, but it is
+ignored by most developers.
 
 Likewise for any rare glibc bugs we might encounter, we should Cc:
 mailto:libc-alpha@sourceware.org

^ permalink raw reply	[relevance 5%]

* [PATCH] ISSUES: add section for bugs in other projects
@ 2015-02-10 17:06  4% Eric Wong
  2015-02-10 17:26  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2015-02-10 17:06 UTC (permalink / raw)
  To: unicorn-public

This is not anything new, just documenting what has been going
on since the beginning.

There's been a small number of generic networking (or mm) bugs in
the kernel which affect unicorn, but are usually found and fixed
with more popular, non-Ruby servers, first.

Aside from generic performance problems, I don't think there's ever
been a glibc bug which affected unicorn.
---
 ISSUES | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/ISSUES b/ISSUES
index f66d14a..343cab4 100644
--- a/ISSUES
+++ b/ISSUES
@@ -18,6 +18,36 @@ instead and your issue will be handled discreetly.
 If you don't get a response within a few days, we may have forgotten
 about it so feel free to ask again.
 
+== Bugs in related projects
+
+unicorn is sometimes affected by bugs in its dependencies.  Bugs
+triggered by unicorn in mainline Ruby, rack, GNU C library (glibc),
+or the Linux kernel will be reported upstream and fixed.
+
+For bugs in Ruby itself, we may forward bugs to
+https://bugs.ruby-lang.org/ and discuss+fix them on the ruby-core
+list at mailto:ruby-core@ruby-lang.org
+Subscription to post is required to ruby-core, unfortunately:
+mailto:ruby-core-request@ruby-lang.org?subject=subscribe
+
+For uncommon bugs in Rack, we may forward bugs to
+mailto:rack-devel@googlegroups.com and discuss there.
+Subscription (without any web UI or Google account) is possible via:
+mailto:rack-devel+subscribe@googlegroups.com
+Note: not everyone can use the proprietary bug tracker used by Rack,
+but their mailing list remains operational.
+
+Uncommon bugs we encounter in the Linux kernel should be Cc:-ed to the
+Linux kernel mailing list (LKML) at mailto:linux-kernel@vger.kernel.org
+and other kernel lists such as mailto:netdev@vger.kernel.org
+(for networking issues).  No subscription is necessary, and the our
+mailing list follows the same conventions as LKML for interopability.
+
+Likewise for any rare glibc bugs we might encounter, we should Cc:
+mailto:libc-alpha@sourceware.org
+Keep in mind glibc upstream does use Bugzilla for tracking bugs:
+https://sourceware.org/bugzilla/
+
 == Submitting Patches
 
 See the HACKING document (and additionally, the
-- 
EW

^ permalink raw reply related	[relevance 4%]

* Re: No, passenger 5.0 is not faster than unicorn :)
  2014-12-03 11:00  4% ` Hongli Lai
  2014-12-03 11:05  0%   ` Sam Saffron
@ 2014-12-03 14:10  0%   ` Bráulio Bhavamitra
  1 sibling, 0 replies; 200+ results
From: Bráulio Bhavamitra @ 2014-12-03 14:10 UTC (permalink / raw)
  To: Hongli Lai; +Cc: unicorn-public, Hitendra Hugo Melo

Hello Hongli,

Thank you the guide, I've already learned a bit from it.

We already use nginx for static files and ssl and varnish for caching
public pages, so maybe turbocaching won't help too much.

In this test I've tested passenger in standalone mode (--max-pool-size
1) and unicorn with one worker. On a slow page, the variation was
minimal (~8.26 req/s in unicorn and ~8.11 in passenger). I haven't
tested fast and cacheable page.

Also, I've used ab for benchmarking. Next time will try wrk.

cheers,
bráulio

On Wed, Dec 3, 2014 at 8:00 AM, Hongli Lai <hongli@phusion.nl> wrote:
> Unicorn *is* in general very good and very efficient, no doubt about that.
> Eric Wong has made great design choices and is an excellent programmer.
>
> Having said that, in certain specific cases there's still room for
> improvement. That's why we focused so much on microoptimizations and
> specific optimizations like turbocaching. Have you followed Phusion
> Passenger's Server Optimization Guide?
> https://www.phusionpassenger.com/documentation/ServerOptimizationGuide.html
>
> Also, you have to ensure that your Rails app sets the correct caching
> headers. By default, Rails sets "Cache-Control: private, no-store" so that
> the turbocache cannot kick in. You should see very different results if you
> add "headers['Cache-Control'] = 'public'" to your Rails app. If you need any
> help with this, please feel free to contact me off-list. I'd be happy to
> help. We have also a benchmarking kit so that you can double check the
> results; email me if you're interested in this.
>
> As Sam said, most of the time will be spent in the Rails app. But
> turbocaching is one notable exception: it's the one feature that can speed
> things up even if your app is slow - provided that you set HTTP caching
> headers correctly.
>
> Unicorn is excellent at what it does: it's a minimal server with a specific
> I/O model that is supposed to be used behind a buffering reverse proxy.
> There is nothing wrong with that, and for the workloads that it's designed
> for, it's great. Phusion Passenger has merely chosen a non-generalist
> approach that aims to squeeze additional performance from specific cases. Of
> course, nothing's a silver bullet. Like any tool, it only works if you use
> it correctly.
>
> On Wed, Dec 3, 2014 at 10:50 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>>
>> Hello all,
>>
>> I've just tested a one instance each (one worker with unicorn and
>> --max-pool-size 1 passenger 5) on the rails app I work.
>>
>> And the results are just as I expected, no miracle at all: Unicorn is
>> still the fatest!
>> (the difference is only a few milliseconds less per request)
>>
>> The blocking design of unicorn is proving itself very efficient.
>>
>> cheers!
>> bráulio
>>
>
>
>
> --
> Phusion | Web Application deployment, scaling, and monitoring solutions
>
> Web: http://www.phusion.nl/
> E-mail: info@phusion.nl
> Chamber of commerce no: 08173483 (The Netherlands)



-- 
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da Mente Macrocósmica, e todas as entidades estão sendo
criadas, preservadas e destruídas nas fases de extroversão e
introversão do fluxo imaginativo cósmico. No âmbito pessoal, quando
uma pessoa imagina algo em sua mente, naquele momento, essa pessoa é a
única proprietária daquilo que ela imagina, e ninguém mais. Quando um
ser humano criado mentalmente caminha por um milharal também
imaginado, a pessoa imaginada não é a propriedade desse milharal, pois
ele pertence ao indivíduo que o está imaginando. Este universo foi
criado na imaginação de Brahma, a Entidade Suprema, por isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também foram criados pela imaginação de Brahma. Nenhuma propriedade
deste mundo, mutável ou imutável, pertence a um indivíduo em
particular; tudo é o patrimônio comum de todos."
Restante do texto em
http://cirandas.net/brauliobo/blog/a-problematica-de-hoje-em-dia

^ permalink raw reply	[relevance 0%]

* Re: No, passenger 5.0 is not faster than unicorn :)
  2014-12-03 11:00  4% ` Hongli Lai
@ 2014-12-03 11:05  0%   ` Sam Saffron
  2014-12-03 14:10  0%   ` Bráulio Bhavamitra
  1 sibling, 0 replies; 200+ results
From: Sam Saffron @ 2014-12-03 11:05 UTC (permalink / raw)
  To: Hongli Lai; +Cc: Bráulio Bhavamitra, unicorn-public, Hitendra Hugo Melo

Yeah, anonymous caching is super critical, we monkey it in here:
https://github.com/discourse/discourse/blob/master/lib/middleware/anonymous_cache.rb
to be honest this really should be part of rails.

On Wed, Dec 3, 2014 at 10:00 PM, Hongli Lai <hongli@phusion.nl> wrote:
> Unicorn *is* in general very good and very efficient, no doubt about that.
> Eric Wong has made great design choices and is an excellent programmer.
>
> Having said that, in certain specific cases there's still room for
> improvement. That's why we focused so much on microoptimizations and
> specific optimizations like turbocaching. Have you followed Phusion
> Passenger's Server Optimization Guide?
> https://www.phusionpassenger.com/documentation/ServerOptimizationGuide.html
>
> Also, you have to ensure that your Rails app sets the correct caching
> headers. By default, Rails sets "Cache-Control: private, no-store" so that
> the turbocache cannot kick in. You should see very different results if you
> add "headers['Cache-Control'] = 'public'" to your Rails app. If you need
> any help with this, please feel free to contact me off-list. I'd be happy
> to help. We have also a benchmarking kit so that you can double check the
> results; email me if you're interested in this.
>
> As Sam said, most of the time will be spent in the Rails app. But
> turbocaching is one notable exception: it's the one feature that can speed
> things up even if your app is slow - provided that you set HTTP caching
> headers correctly.
>
> Unicorn is excellent at what it does: it's a minimal server with a specific
> I/O model that is supposed to be used behind a buffering reverse proxy.
> There is nothing wrong with that, and for the workloads that it's designed
> for, it's great. Phusion Passenger has merely chosen a non-generalist
> approach that aims to squeeze additional performance from specific cases.
> Of course, nothing's a silver bullet. Like any tool, it only works if you
> use it correctly.
>
> On Wed, Dec 3, 2014 at 10:50 AM, Bráulio Bhavamitra <braulio@eita.org.br>
> wrote:
>
>> Hello all,
>>
>> I've just tested a one instance each (one worker with unicorn and
>> --max-pool-size 1 passenger 5) on the rails app I work.
>>
>> And the results are just as I expected, no miracle at all: Unicorn is
>> still the fatest!
>> (the difference is only a few milliseconds less per request)
>>
>> The blocking design of unicorn is proving itself very efficient.
>>
>> cheers!
>> bráulio
>>
>>
>
>
> --
> Phusion | Web Application deployment, scaling, and monitoring solutions
>
> Web: http://www.phusion.nl/
> E-mail: info@phusion.nl
> Chamber of commerce no: 08173483 (The Netherlands)
>
>

^ permalink raw reply	[relevance 0%]

* Re: No, passenger 5.0 is not faster than unicorn :)
  @ 2014-12-03 11:00  4% ` Hongli Lai
  2014-12-03 11:05  0%   ` Sam Saffron
  2014-12-03 14:10  0%   ` Bráulio Bhavamitra
  0 siblings, 2 replies; 200+ results
From: Hongli Lai @ 2014-12-03 11:00 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: unicorn-public, Hitendra Hugo Melo

Unicorn *is* in general very good and very efficient, no doubt about that.
Eric Wong has made great design choices and is an excellent programmer.

Having said that, in certain specific cases there's still room for
improvement. That's why we focused so much on microoptimizations and
specific optimizations like turbocaching. Have you followed Phusion
Passenger's Server Optimization Guide?
https://www.phusionpassenger.com/documentation/ServerOptimizationGuide.html

Also, you have to ensure that your Rails app sets the correct caching
headers. By default, Rails sets "Cache-Control: private, no-store" so that
the turbocache cannot kick in. You should see very different results if you
add "headers['Cache-Control'] = 'public'" to your Rails app. If you need
any help with this, please feel free to contact me off-list. I'd be happy
to help. We have also a benchmarking kit so that you can double check the
results; email me if you're interested in this.

As Sam said, most of the time will be spent in the Rails app. But
turbocaching is one notable exception: it's the one feature that can speed
things up even if your app is slow - provided that you set HTTP caching
headers correctly.

Unicorn is excellent at what it does: it's a minimal server with a specific
I/O model that is supposed to be used behind a buffering reverse proxy.
There is nothing wrong with that, and for the workloads that it's designed
for, it's great. Phusion Passenger has merely chosen a non-generalist
approach that aims to squeeze additional performance from specific cases.
Of course, nothing's a silver bullet. Like any tool, it only works if you
use it correctly.

On Wed, Dec 3, 2014 at 10:50 AM, Bráulio Bhavamitra <braulio@eita.org.br>
wrote:

> Hello all,
>
> I've just tested a one instance each (one worker with unicorn and
> --max-pool-size 1 passenger 5) on the rails app I work.
>
> And the results are just as I expected, no miracle at all: Unicorn is
> still the fatest!
> (the difference is only a few milliseconds less per request)
>
> The blocking design of unicorn is proving itself very efficient.
>
> cheers!
> bráulio
>
>


-- 
Phusion | Web Application deployment, scaling, and monitoring solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)


^ permalink raw reply	[relevance 4%]

* [ANN] email submission port (587) enabled
@ 2014-10-04 23:22  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-10-04 23:22 UTC (permalink / raw)
  To: unicorn-public

For those of you unfortunate enough to be stuck behind firewalls
or rely on crap webmail interfaces, you can pass the:

   --smtp-server-port=587 --smtp-server=bogomips.org

options to "git send-email" when sending patches/plain-text mail
to unicorn-public@bogomips.org

You'll still need a valid From: address to receive replies and
participate in the discussion if you're not subscribed to the
mailing list.

Note: "git send-email" can send regular plain-text emails with
valid Subject: and From: headers.  If you're not familiar with
email message headers, they're nearly identical to HTTP, and
you can use "git format-patch"-generated patches as a guide.

Caveats:

- This will expose your client IP address to the world
  (but many existing mail setups do that anyways)

- This will not work for Cc-ing others in replies directly,
  but works for all @bogomips.org email addresses,
  so you may want to add "--suppress-cc=all" or similar.

Patchbomb away! :D

ref: http://www.ietf.org/rfc/rfc4409.txt
  https://www.kernel.org/pub/software/scm/git/docs/git-send-email.html

^ permalink raw reply	[relevance 4%]

* Re: Master hooks needed
  @ 2014-10-04  2:25  5%             ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-10-04  2:25 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: unicorn-public

Bráulio Bhavamitra <braulio@eita.org.br> wrote:
> Documentation is an excelent solution :)

OK, pushed to the website[1] and unicorn.git as
commit 3318b070c6513a3baa7ce7ac26f4835f46ccff1f

    examples: add run_once to before_fork hook example

    There may be code in a before_fork hook which should run only once,
    document an example using a guard variable since it may not be
    immediately obvious to all users.

    Inspired-by: Bráulio Bhavamitra <braulio@eita.org.br>

http://bogomips.org/unicorn-public/m/20141004015707.GA1951@dcvr.yhbt.net.html

[1] http://unicorn.bogomips.org/examples/

^ permalink raw reply	[relevance 5%]

* Re: Master hooks needed
  2014-10-03 12:22  4% ` Eric Wong
  2014-10-03 12:36  0%   ` Valentin Mihov
@ 2014-10-04  0:53  0%   ` Bráulio Bhavamitra
    1 sibling, 1 reply; 200+ results
From: Bráulio Bhavamitra @ 2014-10-04  0:53 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

The problem is actually worser, and the worker.nr == 0 can't be used.
I had to do something like this:

master_run = true

before_fork do |server, worker|
  if master_run
     #warm up server...

     #kill old pid...

     #disconnect active record

     master_run = false
  end

  # other stuff for each worker
end

In the last example, using worker.nr == 0 would make the server crash
in case the worker 0 was killed/restarted.

Also the AR's disconnect and *many other stuff* people put on
before_fork should only be run once the master was preloaded, not for
every worker.

So I still think at least a hook like master_preloaded(server) is necessary.

cheers,
bráulio

On Fri, Oct 3, 2014 at 9:22 AM, Eric Wong <e@80x24.org> wrote:
> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
>> Hello all,
>>
>> If I need to hook something after master load, I'm currently doing:
>>
>> before_fork do |server, worker|
>>   # worker 0 is the first to init, so hold the master here
>>   if worker.nr == 0
>>      #warm up server...
>>
>>      #kill old pid...
>>   end
>>
>>   # other stuff for each worker
>> end
>>
>> Both operations I currently do (server warm up and old pid kill) need
>> to be run only once, and not for every worker as before_fork does,
>> that's why I had to put the condition seen above.
>
> The above is fine if your first worker never dies.  I think you can add
> a local variable to ensure it only runs the first time worker.nr == 0 is
> started, in case a worker dies.  Something like:
>
>     first = true
>     before_fork do |server, worker|
>       # worker 0 is the first to init, so hold the master here
>       if worker.nr == 0 && first
>          first = false
>          #warm up server...
>
>          #kill old pid...
>       end
>
>       # other stuff for each worker
>     end
>
> For what it's worth, I'm not a fan of auto-killing the old PID in the
> unicorn config and regret having it in the example config.  It's only
> for the most memory-constrained configs and fragile (because anything
> with pid files is always fragile).
>
>> So hooks for master is needed, something like
>> master_after_load(server) and master_init(server).
>>
>> What do you think?
>
> rack.git also has a Rack::Builder#warmup method.  Aman originally
> proposed it for unicorn, but it's useful outside of unicorn so
> we moved it to Rack.
>
> In general, I'm against adding new hooks/options because they tend to
> make maintainability and documentation harder for ops folks.
> I still have nightmares of some Capistrano config filled with hooks
> from years ago :x
>
> Features like these also makes migrating away from unicorn harder, so
> that is another reason we ended up adding #warmup to Rack and not
> unicorn.

^ permalink raw reply	[relevance 0%]

* Re: Master hooks needed
  2014-10-03 12:22  4% ` Eric Wong
@ 2014-10-03 12:36  0%   ` Valentin Mihov
  2014-10-04  0:53  0%   ` Bráulio Bhavamitra
  1 sibling, 0 replies; 200+ results
From: Valentin Mihov @ 2014-10-03 12:36 UTC (permalink / raw)
  To: Eric Wong; +Cc: Bráulio Bhavamitra, unicorn-public

On Fri, Oct 3, 2014 at 3:22 PM, Eric Wong <e@80x24.org> wrote:
>
> Bráulio Bhavamitra <braulio@eita.org.br> wrote:
> > Hello all,
> >
> > If I need to hook something after master load, I'm currently doing:
> >
> > before_fork do |server, worker|
> >   # worker 0 is the first to init, so hold the master here
> >   if worker.nr == 0
> >      #warm up server...
> >
> >      #kill old pid...
> >   end
> >
> >   # other stuff for each worker
> > end
> >
> > Both operations I currently do (server warm up and old pid kill) need
> > to be run only once, and not for every worker as before_fork does,
> > that's why I had to put the condition seen above.
>
> The above is fine if your first worker never dies.  I think you can add
> a local variable to ensure it only runs the first time worker.nr == 0 is
> started, in case a worker dies.  Something like:
>
>     first = true
>     before_fork do |server, worker|
>       # worker 0 is the first to init, so hold the master here
>       if worker.nr == 0 && first
>          first = false
>          #warm up server...
>
>          #kill old pid...
>       end
>
>       # other stuff for each worker
>     end
>
> For what it's worth, I'm not a fan of auto-killing the old PID in the
> unicorn config and regret having it in the example config.  It's only
> for the most memory-constrained configs and fragile (because anything
> with pid files is always fragile).
>
> > So hooks for master is needed, something like
> > master_after_load(server) and master_init(server).
> >
> > What do you think?
>
> rack.git also has a Rack::Builder#warmup method.  Aman originally
> proposed it for unicorn, but it's useful outside of unicorn so
> we moved it to Rack.
>
> In general, I'm against adding new hooks/options because they tend to
> make maintainability and documentation harder for ops folks.
> I still have nightmares of some Capistrano config filled with hooks
> from years ago :x
>
> Features like these also makes migrating away from unicorn harder, so
> that is another reason we ended up adding #warmup to Rack and not
> unicorn.
>

Isn't it much better to do the warmup in an initializer instead in
unicorn? This way you can preload_app=true and the master will execute
the warmup code and fork. Killing the old pid is probably stopping you
from do that, right?

--Valentin

^ permalink raw reply	[relevance 0%]

* Re: Master hooks needed
  @ 2014-10-03 12:22  4% ` Eric Wong
  2014-10-03 12:36  0%   ` Valentin Mihov
  2014-10-04  0:53  0%   ` Bráulio Bhavamitra
  0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2014-10-03 12:22 UTC (permalink / raw)
  To: Bráulio Bhavamitra; +Cc: unicorn-public

Bráulio Bhavamitra <braulio@eita.org.br> wrote:
> Hello all,
> 
> If I need to hook something after master load, I'm currently doing:
> 
> before_fork do |server, worker|
>   # worker 0 is the first to init, so hold the master here
>   if worker.nr == 0
>      #warm up server...
> 
>      #kill old pid...
>   end
> 
>   # other stuff for each worker
> end
> 
> Both operations I currently do (server warm up and old pid kill) need
> to be run only once, and not for every worker as before_fork does,
> that's why I had to put the condition seen above.

The above is fine if your first worker never dies.  I think you can add
a local variable to ensure it only runs the first time worker.nr == 0 is
started, in case a worker dies.  Something like:

    first = true
    before_fork do |server, worker|
      # worker 0 is the first to init, so hold the master here
      if worker.nr == 0 && first
         first = false
         #warm up server...

         #kill old pid...
      end

      # other stuff for each worker
    end

For what it's worth, I'm not a fan of auto-killing the old PID in the
unicorn config and regret having it in the example config.  It's only
for the most memory-constrained configs and fragile (because anything
with pid files is always fragile).

> So hooks for master is needed, something like
> master_after_load(server) and master_init(server).
> 
> What do you think?

rack.git also has a Rack::Builder#warmup method.  Aman originally
proposed it for unicorn, but it's useful outside of unicorn so
we moved it to Rack.

In general, I'm against adding new hooks/options because they tend to
make maintainability and documentation harder for ops folks.
I still have nightmares of some Capistrano config filled with hooks
from years ago :x

Features like these also makes migrating away from unicorn harder, so
that is another reason we ended up adding #warmup to Rack and not
unicorn.

^ permalink raw reply	[relevance 4%]

* Rack encodings (was: Please move to github)
  @ 2014-08-04  8:48  5%         ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-08-04  8:48 UTC (permalink / raw)
  To: Hongli Lai
  Cc: Michael Fischer, Gary Grossman, unicorn-public, Michael Grosser

(Long overdue Subject: change)

Hongli Lai <hongli@phusion.nl> wrote:
> Hi guys. Phusion Passenger author here. I would very much support
> standardization of encoding issues.

> At this point, I don't really care what the standard is, as long as
> it's a sane standard that everybody can follow.

Fair enough.  Would you/Phusion be comfortable taking the lead here?
This feels like another "hot potato" issue :>

> In my opinion, following Encoding.default_external is not helpful.
> Most users have absolutely no idea how to configure
> Encoding.default_external, or even know that it exists. I've also
> never, ever seen anybody who does *not* want default_external to be
> UTF-8. If it's not set to UTF-8, then it's always by accident (e.g.
> the user not knowing that it depends on LC_CTYPE, that LC_CTYPE is set
> differently in the shell than from an init script, or even what
> LC_CTYPE is).

Perhaps we need to educate users to set LC_CTYPE/LC_ALL/LANG so
Encoding.default_external works as intended?  Adding another
option to Rack will just as likely to get missed.

Maybe servers could emit a big warning saying:

    WARNING: Encoding.default_external is not UTF-8 ...

And add a --quiet-utf8-warning option for the few folks who really do
not want UTF-8.

^ permalink raw reply	[relevance 5%]

* Re: Please move to github
  @ 2014-08-02  8:50  3% ` Eric Wong
    0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2014-08-02  8:50 UTC (permalink / raw)
  To: Gary Grossman; +Cc: unicorn-public, michael

Gary Grossman <gary.grossman@gmail.com> wrote:
> Hi Eric,
> 
> I work with Michael, and this discussion sure got off on the
> wrong foot... we love unicorn and use it heavily, and just
> want to contribute back to it.

No worries, cultural differences happen.  Thanks for following up.

> To detail the encoding problem we were trying to fix, unicorn
> uses rb_str_new in several places to create Ruby strings.
> For Ruby 1.9 and later, these strings are assigned ASCII-8BIT
> encoding.
> 
> While the Rack specification doesn't dictate what encoding
> should be used for strings in the environment, many
> developers would probably expect the default external encoding
> setting in Encoding.default_external to be used.

Right, the Rack spec does not dictate this.  Doing this out-of-spec has
the ability to break existing apps as well as compatibility with other
app servers.

What do other app servers do?

My main concern is having more different behavior between various Rack
servers servers, making it harder to switch between them.

Another concern is breaking apps which are already working around this
but work with non-UTF-8 encodings.

The rack-devel mailing list had a discussion on this in September 2010
and a decision was never reached. You can search the archives at:
http://groups.google.com/group/rack-devel

I've also saved the thread to a mbox at
http://80x24.org/rack-devel-encoding-2010.mbox.gz
since Google Groups archives are a bit painful to navigate.

Disclaimer: I am not an encoding expert, so for that reason I prefer
to let other Rack folks make the decision.

> Many Rails applications use UTF8 heavily. The use of ASCII-8BIT
> in the env can lead to Encoding::CompatibilityErrors being
> raised when a UTF8 string and ASCII-8BIT string are concatenated,
> which happens frequently when properties like request.url are
> referenced in erb templates. To get around these problems,
> an app would have to force encoding on the strings in the env
> manually. It seems a shame to do this in slower Ruby code when
> it could be done up front by unicorn.

Yes, this existing behavior sucks on UTF-8-heavy apps.  I would rather
not add more unicorn-only options which make switching between servers
harder.

Do you have performance measurements for doing this as pure-Ruby
middleware vs your patch?

My dislike of lock-in also applies to app servers.  Application-visible
differences like these should be avoided so people can switch between
servers, too.

So it should be best if there were a way to do this for all Rack
servers.

> We'd like to propose that unicorn use rb_external_str_new to
> make strings instead of rb_str_new.
> 
> Perhaps you have your reasons for continuing to use rb_str_new
> but we figured we'd run this by you.

If the Rack spec mandated encodings, I would do it in a heartbeat.

> Subject: [PATCH] If unicorn is used with Ruby 1.9 or later, use
>  rb_external_str_new instead of rb_str_new to create strings. The resulting
>  strings will use the default external encoding. Continue using rb_str_new for
>  older versions of Ruby.

A better, shorter, more direct subject would be:

Subject: use Encoding.default_external for header values

Commit message body is fine <snip>

> +#ifdef HAVE_RUBY_ENCODING_H
> +/* Use default external encoding for strings for Ruby 1.9+,
> + * fall back to rb_str_new when unavailable */
> +#define rb_str_new rb_external_str_new
> +#endif

This is too heavy-handed, as some strings (buffers) may
need to stay binary via rb_str_new.  If we were to do this, it would
something like:

#ifdef HAVE_RUBY_ENCODING_H
#  define env_val_new(ptr,len) rb_external_str_new((ptr),(len))
#else
#  define env_val_new(ptr,len) rb_str_new((ptr),(len))
#endif

... And only making sure header values are set to external.

Last I checked the HTTP RFCs (it's been a while) header keys are
required to be US-ASCII-only (and our parser enforces that).

> +  def test_encoding
> +    if ''.respond_to?(:encoding)
> +      client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
> +                               "Host: foo\r\n\r\n")
> +      env = @request.read(client)
> +      encoding = Encoding.default_external
> +      assert_equal encoding, env['REQUEST_PATH'].encoding
> +      assert_equal encoding, env['PATH_INFO'].encoding
> +      assert_equal encoding, env['QUERY_STRING'].encoding
> +    end

This would need to test and work with (and appropriately reject)
invalid requests with bad encodings, too.

^ permalink raw reply	[relevance 3%]

* [PATCH] FAQ: add entry for Rails autoflush_log
  @ 2014-07-03 19:28  5%   ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-07-03 19:28 UTC (permalink / raw)
  To: unicorn-public; +Cc: Cedric Maion

Thanks to Cedric Maion for bringing this up on the mailing list:
http://bogomips.org/unicorn-public/m/20140703144048.GA6674@cedric-maion.com
---
 This updates http://unicorn.bogomips.org/FAQ.html

 FAQ | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/FAQ b/FAQ
index c84a8af..66f1a09 100644
--- a/FAQ
+++ b/FAQ
@@ -1,6 +1,14 @@
 = Frequently Asked Questions about Unicorn
 
-===  I've installed Rack 1.1.x, why can't Unicorn load Rails (2.3.5)?
+=== Why aren't my Rails log files rotated when I use SIGUSR1?
+
+The Rails autoflush_log option must remain disabled with multiprocess
+servers such as unicorn.  Buffering in userspace may cause lines to be
+partially written and lead to corruption in the presence of multiple
+processes.  With reasonable amounts of logging, the performance impact
+of autoflush_log should be negligible on Linux and other modern kernels.
+
+=== I've installed Rack 1.1.x, why can't Unicorn load Rails (2.3.5)?
 
 Rails 2.3.5 is not compatible with Rack 1.1.x.  Unicorn is compatible
 with both Rack 1.1.x and Rack 1.0.x, and RubyGems will load the latest
-- 
EW

^ permalink raw reply related	[relevance 5%]

* subscription to new ML probably working...
@ 2014-05-08  8:43  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-05-08  8:43 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: unicorn-public

I'll migrate everybody else over around 20140514T20:00Z, otherwise you
might get duplicate messages from being subscribed to both lists.
If you're willing to deal with dupes for a few days, you can try
subscribe/unsubscribe/help commands:

	unicorn-public+subscribe@bogomips.org
	unicorn-public+unsubscribe@bogomips.org
	unicorn-public+help@bogomips.org
	(help will tell you the rest)

Stop reading here unless you're interested in implementation details :>

I only have mlmmj configured to handle the +commands, outgoing SMTP,
and bounces.  It does not receive normal list traffic like a standard
mlmmj install.  Instead, I have it configured to replay data from ssoma
every minute, so I used (as the mlmmj user):

    NAME=unicorn-public
    URL=git://bogomips.org/unicorn-public.git
    ssoma add $NAME $URL "command:ssoma-replay -L $HOME/spool/$NAME"

Where ssoma-replay is a tiny shell script which reads from stdin:
------------------------------8<--------------------------
#!/bin/sh
MSG=$(mktemp -t ssoma-replay.orig.$USER.XXXXXX || exit 1)
cat > "$MSG"
exec /usr/bin/mlmmj-send "$@" -m "$MSG"
------------------------------8<--------------------------
This means it should be easy to have outgoing SMTP on a different server
and allow other folks to run the same service.

Other notes: mlmmj has defaults I like out-of-the-box (unlike mailman,
ugh).  However, archive retrieval with mlmmj is rather painful,
but we have public-inbox and ssoma :)

[1] http://mlmmj.org/

^ permalink raw reply	[relevance 4%]

* Re: [ANN] unicorn 4.8.3 - the end of an era
  @ 2014-05-07  9:46  4%   ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-05-07  9:46 UTC (permalink / raw)
  To: Jérémy Lecour; +Cc: Liste Unicorn, unicorn-public

Jérémy Lecour <jeremy.lecour@gmail.com> wrote:
> Hi Eric and all,
> 
> I'm sorry if I'm the dumbest guy in the room, but I absolutely don't
> understand what I have to do to stay in the mailing list.

Sorry about the confusion, I'll try to clarify...

> Would you mind explaining what a regular subscriber has to do to keep
> receiving those emails in their inbox?

You use ssoma[1] to to import mail into your inbox.  This is like how
slrnpull works with NNTP or getmail/fetchmail works with POP.  It's
only a one-way sync, but you can import it into an IMAP folder.

Currently there's no SMTP delivery component, but I could probably set
one up this week if there's enough demand for one.  The subscriber list
will be public, though.

[1] http://ssoma.public-inbox.org/

Taken from http://unicorn.bogomips.org/ISSUES
based on what I wrote in:
http://bogomips.org/unicorn-public/m/20140421183728.GA27047@dcvr.yhbt.net.html

	URL=git://bogomips.org/unicorn-public
	LISTNAME=unicorn

	# to initialize a maildir (this may be a new or existing maildir,
	# ssoma will not touch existing messages)
	# If you prefer mbox, use mbox:/path/to/mbox as the last argument
	# You may also use imap://$MAILSERVER/INBOX for an IMAP account
	# or imaps:// for an IMAPS account, as well.
	ssoma add $LISTNAME $URL maildir:/path/to/maildir

	# read with your favorite MUA (only using mutt as an example)
	mutt -f /path/to/maildir # (or /path/to/mbox)

	# to keep your mbox or maildir up-to-date, periodically run the following:
	ssoma sync $LISTNAME

	# your MUA may modify and delete messages from the maildir or mbox,
	# this does not affect ssoma functionality at all

	# to sync all your ssoma subscriptions
	ssoma sync

	# You may wish to sync in your cronjob
	ssoma sync --cron

^ permalink raw reply	[relevance 4%]

* [PATCH] add slrnpull.conf example for gmane archives
@ 2014-04-21 19:18  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-04-21 19:18 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: unicorn-public

In case anybody else wants to verify/check the archive or
use this for other projects, we'll document what we did here.
---
  Note: for the public-inbox import, I also merged in some emails
  from my personal archives which gmane didn't catch.

 archive/.gitignore    | 3 +++
 archive/slrnpull.conf | 4 ++++
 2 files changed, 7 insertions(+)
 create mode 100644 archive/.gitignore
 create mode 100644 archive/slrnpull.conf

diff --git a/archive/.gitignore b/archive/.gitignore
new file mode 100644
index 0000000..bd7ad57
--- /dev/null
+++ b/archive/.gitignore
@@ -0,0 +1,3 @@
+/data
+/news
+/requests
diff --git a/archive/slrnpull.conf b/archive/slrnpull.conf
new file mode 100644
index 0000000..fcfcafe
--- /dev/null
+++ b/archive/slrnpull.conf
@@ -0,0 +1,4 @@
+# group_name                         max        expire     headers_only
+gmane.comp.lang.ruby.unicorn.general 1000000000 1000000000 0
+
+# usage: slrnpull -d $PWD -h news.gmane.org --no-post
-- 
Eric Wong

^ permalink raw reply related	[relevance 5%]

* Re: [ANN] public-inbox - "archives first" approach to mailing lists
  @ 2014-04-21 19:07  5% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2014-04-21 19:07 UTC (permalink / raw)
  To: mongrel-unicorn

Eric Wong <normalperson@yhbt.net> wrote:
> I'm importing the unicorn ML archives, now, so all of our future
> emails will be archived with this.

HTML archives: http://bogomips.org/unicorn-public/
(an ugly work-in-progress, but all the data is there,
 bikeshed with pretty gooeys to your content :)

Or once you have ssoma[1] installed:

	URL=git://bogomips.org/unicorn-public
	ssoma add unicorn $URL mbox:/tmp/mbox
	ssoma sync unicorn
	mutt -f /tmp/mbox
	# of course, any mailer will do, and IMAP is supported, too

[1] http://ssoma.public-inbox.org/

^ permalink raw reply	[relevance 5%]

* [ANN] unicorn 4.7.0 - minor updates, license tweak
@ 2013-11-04  7:57  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-11-04  7:57 UTC (permalink / raw)
  To: mongrel-unicorn

* git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml

Changes:

* support SO_REUSEPORT on new listeners (:reuseport)

This allows users to start an independent instance of unicorn on
a the same port as a running unicorn (as long as both instances
use :reuseport).

ref: https://lwn.net/Articles/542629/

* unicorn is now GPLv2-or-later and Ruby 1.8-licensed
(instead of GPLv2-only, GPLv3-only, and Ruby 1.8-licensed)

This changes nothing at the moment.  Once the FSF publishes the next
version of the GPL, users may choose the newer GPL version without the
unicorn BDFL approving it.  Two years ago when I got permission to add
GPLv3 to the license options, I also got permission from all past
contributors to approve future versions of the GPL.  So now I'm
approving all future versions of the GPL for use with unicorn.

Reasoning below:

In case the GPLv4 arrives and I am not alive to approve/review it,
the lesser of evils is have give blanket approval of all future GPL
versions (as published by the FSF).  The worse evil is to be stuck
with a license which cannot guarantee the Free-ness of this project
in the future.

This unfortunately means the FSF can theoretically come out with
license terms I do not agree with, but the GPLv2 and GPLv3 will
always be an option to all users.

Note: we currently prefer GPLv3

Two improvements thanks to Ernest W. Durbin III:

* USR2 redirects fixed for Ruby 1.8.6 (broken since 4.1.0)
* unicorn(1) and unicorn_rails(1) enforces valid integer for -p/--port

A few more odd, minor tweaks and fixes:

* attempt to rename PID file when possible (on USR2)
* workaround reopen atomicity issues for stdio vs non-stdio
* improve handling of client-triggerable socket errors

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: [PATCH] license: allow all future versions of the GNU GPL
  @ 2013-10-26 10:09  5%   ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-10-26 10:09 UTC (permalink / raw)
  To: unicorn list

Hongli Lai <hongli@phusion.nl> wrote:
> On Sat, Oct 26, 2013 at 9:58 AM, Eric Wong <normalperson@yhbt.net> wrote:
> > There is currently no GPLv4, so this change has no effect at the
> > moment.
> >
> > In case the GPLv4 arrives and I am not alive to approve/review it,
> > the lesser of evils is have give blanket approval of all future GPL
> > versions (as published by the FSF).  The worse evil is to be stuck
> > with a license which cannot guarantee the Free-ness of this project
> > in the future.
> >
> > This unfortunately means the FSF can theoretically come out with
> > license terms I do not agree with, but the GPLv2 and GPLv3 will
> > always be an option to all users.
> 
> Don't you need approval from all contributors, including Zed, to be
> able to do something like this?

I already got this a few years ago when I got permission for the GPLv3:

- The unicorn project leader (Eric Wong) reserves the right to add future
- versions of the GPL (and no other licenses) as published by the FSF to
- the licensing terms.

So I'm adding all future versions of the GPL.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* [PATCH] license: allow all future versions of the GNU GPL
@ 2013-10-26  7:58  5% Eric Wong
    0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-10-26  7:58 UTC (permalink / raw)
  To: mongrel-unicorn

There is currently no GPLv4, so this change has no effect at the
moment.

In case the GPLv4 arrives and I am not alive to approve/review it,
the lesser of evils is have give blanket approval of all future GPL
versions (as published by the FSF).  The worse evil is to be stuck
with a license which cannot guarantee the Free-ness of this project
in the future.

This unfortunately means the FSF can theoretically come out with
license terms I do not agree with, but the GPLv2 and GPLv3 will
always be an option to all users.
---
 LICENSE                          | 17 ++++++++++-------
 README                           |  5 +++--
 ext/unicorn_http/unicorn_http.rl |  2 +-
 lib/unicorn/app/inetd.rb         |  2 +-
 lib/unicorn/app/old_rails.rb     |  2 +-
 lib/unicorn/cgi_wrapper.rb       |  2 +-
 test/test_helper.rb              |  2 +-
 test/unit/test_http_parser.rb    |  2 +-
 test/unit/test_request.rb        |  2 +-
 test/unit/test_response.rb       |  2 +-
 test/unit/test_server.rb         |  2 +-
 test/unit/test_signals.rb        |  2 +-
 unicorn.gemspec                  |  2 +-
 13 files changed, 24 insertions(+), 20 deletions(-)

diff --git a/LICENSE b/LICENSE
index 099110a..5b6458e 100644
--- a/LICENSE
+++ b/LICENSE
@@ -3,15 +3,18 @@ revision control for names and email addresses of all of them.
 
 You can redistribute it and/or modify it under either the terms of the
 GNU General Public License (GPL) as published by the Free Software
-Foundation (FSF), version {3.0}[http://www.gnu.org/licenses/gpl-3.0.txt]
-or version {2.0}[http://www.gnu.org/licenses/gpl-2.0.txt]
-or the Ruby-specific license terms (see below).
+Foundation (FSF), either version 2 of the License, or (at your option)
+any later version.  We currently prefer the GPLv3 or later for
+derivative works, but the GPLv2 is fine.
 
-The unicorn project leader (Eric Wong) reserves the right to add future
-versions of the GPL (and no other licenses) as published by the FSF to
-the licensing terms.
+The complete texts of the GPLv2 and GPLv3 are below:
+GPLv2 - http://www.gnu.org/licenses/gpl-2.0.txt
+GPLv3 - http://www.gnu.org/licenses/gpl-3.0.txt
 
-=== Ruby-specific terms (if you're not using the GPLv2 or GPLv3)
+You may (against our _preference_) also use the Ruby 1.8 license terms
+which we inherited from the original Mongrel project when we forked it:
+
+=== Ruby 1.8-specific terms (if you're not using the GPL)
 
   1. You may make and give away verbatim copies of the source form of the
      software without restriction, provided that you duplicate all of the
diff --git a/README b/README
index 5dde0d7..42167c8 100644
--- a/README
+++ b/README
@@ -63,8 +63,9 @@ both the the request and response in between \Unicorn and slow clients.
 It is based on Mongrel 1.1.5.
 Mongrel is copyright 2007 Zed A. Shaw and contributors.
 
-\Unicorn is tri-licensed under (your choice) of the GPLv3, GPLv2 or
-Ruby (1.8)-specific terms.  See the included LICENSE file for details.
+\Unicorn is licensed under (your choice) of the GPLv2 or later
+(GPLv3+ preferred), or Ruby (1.8)-specific terms.
+See the included LICENSE file for details.
 
 \Unicorn is 100% Free Software.
 
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 3529740..6293033 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -2,7 +2,7 @@
  * Copyright (c) 2009 Eric Wong (all bugs are Eric's fault)
  * Copyright (c) 2005 Zed A. Shaw
  * You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
- * the GPLv3
+ * the GPLv2+ (GPLv3+ preferred)
  */
 #include "ruby.h"
 #include "ext_help.h"
diff --git a/lib/unicorn/app/inetd.rb b/lib/unicorn/app/inetd.rb
index b851214..13b6624 100644
--- a/lib/unicorn/app/inetd.rb
+++ b/lib/unicorn/app/inetd.rb
@@ -2,7 +2,7 @@
 # :enddoc:
 # Copyright (c) 2009 Eric Wong
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 
 # this class *must* be used with Rack::Chunked
 module Unicorn::App
diff --git a/lib/unicorn/app/old_rails.rb b/lib/unicorn/app/old_rails.rb
index ad1ca8e..1e8c41a 100644
--- a/lib/unicorn/app/old_rails.rb
+++ b/lib/unicorn/app/old_rails.rb
@@ -5,7 +5,7 @@
 # Copyright (c) 2005 Zed A. Shaw
 # Copyright (c) 2009 Eric Wong
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 # Additional work donated by contributors.  See CONTRIBUTORS for more info.
 require 'unicorn/cgi_wrapper'
 require 'dispatcher'
diff --git a/lib/unicorn/cgi_wrapper.rb b/lib/unicorn/cgi_wrapper.rb
index d0175d0..d9b7fe5 100644
--- a/lib/unicorn/cgi_wrapper.rb
+++ b/lib/unicorn/cgi_wrapper.rb
@@ -5,7 +5,7 @@
 # Copyright (c) 2005 Zed A. Shaw
 # Copyright (c) 2009 Eric Wong
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 #
 # Additional work donated by contributors.  See CONTRIBUTORS for more info.
 
diff --git a/test/test_helper.rb b/test/test_helper.rb
index cf98996..c65f2f3 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -2,7 +2,7 @@
 
 # Copyright (c) 2005 Zed A. Shaw 
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 #
 # Additional work donated by contributors.  See http://mongrel.rubyforge.org/attributions.html 
 # for more information.
diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 64146e7..8d5b251 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -2,7 +2,7 @@
 
 # Copyright (c) 2005 Zed A. Shaw 
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 #
 # Additional work donated by contributors.  See http://mongrel.rubyforge.org/attributions.html
 # for more information.
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index a57cbcd..fbda1a2 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -2,7 +2,7 @@
 
 # Copyright (c) 2009 Eric Wong
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 
 require 'test/test_helper'
 
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
index 054c3dd..85ac085 100644
--- a/test/unit/test_response.rb
+++ b/test/unit/test_response.rb
@@ -2,7 +2,7 @@
 
 # Copyright (c) 2005 Zed A. Shaw 
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 #
 # Additional work donated by contributors.  See http://mongrel.rubyforge.org/attributions.html 
 # for more information.
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index a821790..e5b335f 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -2,7 +2,7 @@
 
 # Copyright (c) 2005 Zed A. Shaw 
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 #
 # Additional work donated by contributors.  See http://mongrel.rubyforge.org/attributions.html
 # for more information.
diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb
index f1d8bb3..443c736 100644
--- a/test/unit/test_signals.rb
+++ b/test/unit/test_signals.rb
@@ -2,7 +2,7 @@
 
 # Copyright (c) 2009 Eric Wong
 # You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv3
+# the GPLv2+ (GPLv3+ preferred)
 #
 # Ensure we stay sane in the face of signals being sent to us
 
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 0453eb0..4619a89 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -40,5 +40,5 @@ Gem::Specification.new do |s|
   s.add_development_dependency('isolate', '~> 3.2')
   s.add_development_dependency('wrongdoc', '~> 1.6.1')
 
-  s.licenses = ["GPLv2", "GPLv3", "Ruby 1.8"]
+  s.licenses = ["GPLv2+", "Ruby 1.8"]
 end
-- 
1.8.4.483.g7fe67e6.dirty
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 5%]

* Re: Forking non web processes
  2013-10-24 11:08  5% Forking non web processes Sam Saffron
@ 2013-10-24 16:17  0% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-10-24 16:17 UTC (permalink / raw)
  To: unicorn list

Sam Saffron <sam.saffron@gmail.com> wrote:
> Hi Eric,
> 
> I have been trying to get unicorn to allow me to fork off non-web
> processes like sidekiq/resque.
> 
> I got this working, except that I am constantly fighting with the
> unicorn reaper. Any chance we can add some sort of api to fork off non
> web processes? It helps save memory and cut down on master processes.

I've been trying to avoid adding unicorn-specific APIs unless absolutely
necessary.

You're forking off from the master?  Worst case is you'll get a log
message about an unknown process, right?

I'm also wondering why... sidekiq/resque are standalone daemons
themselves.  Shouldn't that be done as part of the deploy/init process?
(unicorn isn't going to become init/upstart/systemd)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Forking non web processes
@ 2013-10-24 11:08  5% Sam Saffron
  2013-10-24 16:17  0% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Sam Saffron @ 2013-10-24 11:08 UTC (permalink / raw)
  To: mongrel-unicorn

Hi Eric,

I have been trying to get unicorn to allow me to fork off non-web
processes like sidekiq/resque.

I got this working, except that I am constantly fighting with the
unicorn reaper. Any chance we can add some sort of api to fork off non
web processes? It helps save memory and cut down on master processes.

Cheers
Sam
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Re: More unexplained timeouts
  2013-09-29 20:13  4% More " nick
@ 2013-09-30  0:06  0% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-09-30  0:06 UTC (permalink / raw)
  To: unicorn list

nick@auger.net wrote:
> We're still suffering from unexplained workers timing out.  We
> recently upgraded to the latest unicorn 4.6.3 (while still on REE
> 1.8.7) in the hopes that it would solve our issues.  Unfortunately,
> this seemed to exacerbate the problem, with timeouts happening more
> frequently, but that could be related to greater precision in timeouts
> in newer versions of unicorn.  (In our unicorn 3.6.2, a timeout set to
> 120s might not ACTUALLY timeout until 180s or more, thus allowing a
> bit more time for Ruby to finish whatever it was choking on.)

Yes, there were some fixes in 4.x to improve the timeout accuracy.

> We dropped the timeout down to 65s (to make sure it was triggered) and
> then tried to add greater logging (per
> http://permalink.gmane.org/gmane.comp.lang.ruby.unicorn.general/1269.)
> The START/FINISH approach confirms it's not an issue with our
> application code, ie:
> 
> HH:MM:SS- S/F[PID]- /PATH
> 15:21:01- START-25904- /pathA
> 15:21:01- FINISH-25904- /pathA
> 15:21:01- START-25904- /pathB
> 15:21:01- FINISH-25904- /pathB
> 15:21:01- START-25904- /pathC
> 15:21:01- FINISH-25904- /pathC
> worker=11 PID:25904 timeout (66s > 65s), killing
> reaped #<Process::Status: pid=25904,signaled(SIGKILL=9)> worker=11
> 
> For each START we always get a corresponding FINISH and then the
> worker is killed.  Additionally, our nginx logs confirm that this last
> request was sent back to the client.  No 'upstream' errors in our
> nginx log, either.
> 
> When we tried the Thread sleep approach, nothing actually appeared in
> the logs.  I imagine this means that ruby or some C extension is
> misbehaving.

Sounds like it.  1.8 and old C extensions could easily lock up the
interpreter on blocking calls.

Another problem could be using new versions of C extensions that are no
longer tested under 1.8.  I admit I haven't tested recent versions of
unicorn/kgio/raindrops on 1.8 lately, either, but I'm _fairly_ sure they
still work since they haven't changed much.

> Unfortunately, it's been impossible for us to recreate this in
> development.  

Are you running any different gems/extensions in development vs
production?

> Thoughts?
> 
> RHEL 5.6
> REE 1.8.7 2011.12
> Unicorn 4.6.3
> 16 unicorn workers on 8 cores
> No swap activity, no peaks in load

What other gems/extensions do you use?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* More unexplained timeouts
@ 2013-09-29 20:13  4% nick
  2013-09-30  0:06  0% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: nick @ 2013-09-29 20:13 UTC (permalink / raw)
  To: mongrel-unicorn

We're still suffering from unexplained workers timing out.  We recently upgraded to the latest unicorn 4.6.3 (while still on REE 1.8.7) in the hopes that it would solve our issues.  Unfortunately, this seemed to exacerbate the problem, with timeouts happening more frequently, but that could be related to greater precision in timeouts in newer versions of unicorn.  (In our unicorn 3.6.2, a timeout set to 120s might not ACTUALLY timeout until 180s or more, thus allowing a bit more time for Ruby to finish whatever it was choking on.)

We dropped the timeout down to 65s (to make sure it was triggered) and then tried to add greater logging (per http://permalink.gmane.org/gmane.comp.lang.ruby.unicorn.general/1269.)  The START/FINISH approach confirms it's not an issue with our application code, ie:

HH:MM:SS- S/F[PID]- /PATH
15:21:01- START-25904- /pathA
15:21:01- FINISH-25904- /pathA
15:21:01- START-25904- /pathB
15:21:01- FINISH-25904- /pathB
15:21:01- START-25904- /pathC
15:21:01- FINISH-25904- /pathC
worker=11 PID:25904 timeout (66s > 65s), killing
reaped #<Process::Status: pid=25904,signaled(SIGKILL=9)> worker=11

For each START we always get a corresponding FINISH and then the worker is killed.  Additionally, our nginx logs confirm that this last request was sent back to the client.  No 'upstream' errors in our nginx log, either.

When we tried the Thread sleep approach, nothing actually appeared in the logs.  I imagine this means that ruby or some C extension is misbehaving.

Unfortunately, it's been impossible for us to recreate this in development.  

Thoughts?

RHEL 5.6
REE 1.8.7 2011.12
Unicorn 4.6.3
16 unicorn workers on 8 cores
No swap activity, no peaks in load

Again, thanks for all your help!

-Nick

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: A barrage of unexplained timeouts
  2013-08-20 21:32  0%                 ` Eric Wong
@ 2013-08-21 13:33  0%                   ` nick
  0 siblings, 0 replies; 200+ results
From: nick @ 2013-08-21 13:33 UTC (permalink / raw)
  To: unicorn list

"Eric Wong" <normalperson@yhbt.net> said:
> nick@auger.net wrote:
>> "Eric Wong" <normalperson@yhbt.net> said:
>> > nick@auger.net wrote:
>> >> "Eric Wong" <normalperson@yhbt.net> said:
>> > I'm stumped :<
>>
>> I was afraid you'd say that :(.
> 
> Actually, another potential issue is DNS lookups timing out.  But they
> shouldn't take *that* long...
> 
>> > Do you have any background threads running that could be hanging the
>> > workers?   This is Ruby 1.8, after all, so there's more likely to be
>> > some blocking call hanging the entire process.  AFAIK, some monitoring
>> > software runs a background thread in the unicorn worker and maybe the
>> > OpenSSL extension doesn't work as well if it encountered network
>> > problems under Ruby 1.8
>>
>> We don't explicitly create any threads in our rails code.  We do
>> communicate with backgroundrb worker processes, although, none of the
>> strangeness today involved any routes that would hit backgroundrb
>> workers.
> 
> I proactively audit every piece of code (including external
> libraries/gems) loaded by an app for potentially blocking calls (hits to
> the filesystem, socket calls w/o timeout/blocking).   I use strace to
> help me find that sometimes...
> 
>> Is there any instrumentation that I could add that might help
>> debugging in the future? ($request_time and $upstream_response_time
>> are now in my nginx logs.)  We have noticed these "unexplainable
>> timeouts" before, but typically for a single worker.  If there's some
>> debugging that could be added I might be able to track it down during
>> these one-off events.
> 
> As an experiment, can you replay traffic a few minutes leading up to and
> including that 7m period in a test setup with only one straced worker?

I've replayed this in a development environment without triggering the issue.  Lack of POST payloads and slight differences between the two environments, make this difficult to test.

> Run "strace -T -f -o $FILE -p $PID_OF_WORKER" and see if there's any
> unexpected/surprising dependencies (connect() to unrecognized addresses,
> open() to networked filesystems, fcntl locks, etc...).
> 
> You can play around with some other strace options (-v/-s SIZE/-e filters)

I'll do my best to look at the output for anything out of the ordinary.  Unfortunately, strace is a new tool for me and is a bit out of my wheelhouse.


_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: A barrage of unexplained timeouts
  2013-08-20 21:19  5%               ` nick
@ 2013-08-20 21:32  0%                 ` Eric Wong
  2013-08-21 13:33  0%                   ` nick
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-08-20 21:32 UTC (permalink / raw)
  To: unicorn list

nick@auger.net wrote:
> "Eric Wong" <normalperson@yhbt.net> said:
> > nick@auger.net wrote:
> >> "Eric Wong" <normalperson@yhbt.net> said:
> > I'm stumped :<
> 
> I was afraid you'd say that :(.

Actually, another potential issue is DNS lookups timing out.  But they
shouldn't take *that* long...

> > Do you have any background threads running that could be hanging the
> > workers?   This is Ruby 1.8, after all, so there's more likely to be
> > some blocking call hanging the entire process.  AFAIK, some monitoring
> > software runs a background thread in the unicorn worker and maybe the
> > OpenSSL extension doesn't work as well if it encountered network
> > problems under Ruby 1.8
> 
> We don't explicitly create any threads in our rails code.  We do
> communicate with backgroundrb worker processes, although, none of the
> strangeness today involved any routes that would hit backgroundrb
> workers.

I proactively audit every piece of code (including external
libraries/gems) loaded by an app for potentially blocking calls (hits to
the filesystem, socket calls w/o timeout/blocking).   I use strace to
help me find that sometimes...

> Is there any instrumentation that I could add that might help
> debugging in the future? ($request_time and $upstream_response_time
> are now in my nginx logs.)  We have noticed these "unexplainable
> timeouts" before, but typically for a single worker.  If there's some
> debugging that could be added I might be able to track it down during
> these one-off events.

As an experiment, can you replay traffic a few minutes leading up to and
including that 7m period in a test setup with only one straced worker?

Run "strace -T -f -o $FILE -p $PID_OF_WORKER" and see if there's any
unexpected/surprising dependencies (connect() to unrecognized addresses,
open() to networked filesystems, fcntl locks, etc...).

You can play around with some other strace options (-v/-s SIZE/-e filters)

Maybe you'll find something, there.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: A barrage of unexplained timeouts
  @ 2013-08-20 21:19  5%               ` nick
  2013-08-20 21:32  0%                 ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: nick @ 2013-08-20 21:19 UTC (permalink / raw)
  To: unicorn list

"Eric Wong" <normalperson@yhbt.net> said:

> nick@auger.net wrote:
>> "Eric Wong" <normalperson@yhbt.net> said:
>> > This is really strange.  This was only really bad for a 7s period?
>>
>> It was a 7 minute period.  All of the workers would become busy and
>> exceed their >120s timeout.  Master would kill and re-spawn them,
>> they'd start to respond to a handful of requests (anywhere from 5-50)
>> after which they'd become "busy" again, and get force killed by
>> master.  This pattern happened 3 times over a 7 minute period.
>>
>> > Has it happened again?
>>
>> No
>>
>> > Anything else going on with the system at that time?  Swapping,
>> > particularly...
>>
>> No swap activity or high load.  Our munin graphs indicate a peak of
>> web/app server disk latency around that time, although our graphs show
>> many other similar peaks, without incident.
> 
> I'm stumped :<

I was afraid you'd say that :(.

> Do you have any background threads running that could be hanging the
> workers?   This is Ruby 1.8, after all, so there's more likely to be
> some blocking call hanging the entire process.  AFAIK, some monitoring
> software runs a background thread in the unicorn worker and maybe the
> OpenSSL extension doesn't work as well if it encountered network
> problems under Ruby 1.8

We don't explicitly create any threads in our rails code.  We do communicate with backgroundrb worker processes, although, none of the strangeness today involved any routes that would hit backgroundrb workers.

Is there any instrumentation that I could add that might help debugging in the future? ($request_time and $upstream_response_time are now in my nginx logs.)  We have noticed these "unexplainable timeouts" before, but typically for a single worker.  If there's some debugging that could be added I might be able to track it down during these one-off events.

Thoughts?

I absolutely appreciate all your help!

-Nick

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* [PATCH] unicorn_forever: new executable to respawn masters
@ 2013-07-24  3:11  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2013-07-24  3:11 UTC (permalink / raw)
  To: mongrel-unicorn

Comments/reports of success/failure appreciated.

(Bcc-ing the user who contacted me privately about daemontools :)
--------------------------------8<------------------------------
From: Eric Wong <normalperson@yhbt.net>
Subject: [PATCH] unicorn_forever: new executable to respawn masters

Warning: lightly tested (and not under daemontools/systemd/etc)

This may be useful for daemontools and similar init replacements
which behave badly when the master process is replaced during the
normal SIGUSR2 && SIGQUIT routine.

Usage:

  unicorn_forever EXISTING UNICORN COMMAND-LINE

Example:

  unicorn_forever unicorn -c /path/to/unicorn_config.rb config.ru

It can also be used to keep Rainbows! processes alive as long as
you check for "Rainbows!" constant references in your config file.

  unicorn_forever rainbows -c /path/to/rainbows_config.rb config.ru

Supported signals:

SIGKILL - really kill the unicorn_forever process (unblockable)

SIGSTOP - pause the process, this prevent unicorn_forever from detecting
          or respawning a dead master

SIGTSTP - same as SIGSTOP

SIGCONT - resumes a process stopped by SIGCONT

Those signals above were really implicit to everything, the following
two should be familiar to existing unicorn users.

SIGHUP  - reloads the config (just like regular unicorn).
          This does not touch the existing master process, but allows
	  future masters to be spawned with a different set of listen
	  sockets.

SIGUSR1 - reopens existing log files, this signal is forwarded to the
          regular unicorn master (and thus any workers it has)

All other normal unicorn signals are logged and otherwise ignored.
They are not forwarded to the unicorn master.

To upgrade a unicorn application, just send SIGQUIT (not SIGUSR2) to the
existing master and unicorn_forever will automatically respawn.

There is no way to gracefully upgrade unicorn_forever without losing
connections.  Doing graceful upgrades of unicorn_forever would defeat
the purpose and cause parents (e.g. daemontools) to notice a child
death.

unicorn_forever is probably unnecessary for systemd.  The use of cgroups
with systemd prevents daemons from "escaping" the control of systemd, so
a daemonized unicorn probably remains visible to systemd.

Implementation:

unicorn_forever is stripped down version of unicorn (and the
Unicorn::HttpServer class) which contains enough to:

* parse the config file for listeners (and general validation)
* bind listen sockets
* issue chdir for the working_directory
* set the UNICORN_FD environment variable
* exec the real process (unicorn/rainbows/whatever...)

It does not load nor validate the application.
---
 bin/unicorn_forever    | 126 +++++++++++++++++++++
 lib/unicorn/forever.rb | 289 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 415 insertions(+)
 create mode 100755 bin/unicorn_forever
 create mode 100644 lib/unicorn/forever.rb

diff --git a/bin/unicorn_forever b/bin/unicorn_forever
new file mode 100755
index 0000000..bef3a5f
--- /dev/null
+++ b/bin/unicorn_forever
@@ -0,0 +1,126 @@
+#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
+# -*- encoding: binary -*-
+require 'unicorn'
+require 'unicorn/forever'
+require 'optparse'
+
+rackup_opts = Unicorn::Configurator::RACKUP
+options = rackup_opts[:options]
+
+op = OptionParser.new("", 24, '  ') do |opts|
+  cmd = File.basename($0)
+  opts.banner = "Usage: #{cmd} " \
+                "[ruby options] [#{cmd} options] [rackup config file]"
+  opts.separator "Ruby options:"
+
+  lineno = 1
+  opts.on("-e", "--eval LINE", "evaluate a LINE of code") do |line|
+    eval line, TOPLEVEL_BINDING, "-e", lineno
+    lineno += 1
+  end
+
+  opts.on("-d", "--debug", "set debugging flags (set $DEBUG to true)") do
+    $DEBUG = true
+  end
+
+  opts.on("-w", "--warn", "turn warnings on for your script") do
+    $-w = true
+  end
+
+  opts.on("-I", "--include PATH",
+          "specify $LOAD_PATH (may be used more than once)") do |path|
+    $LOAD_PATH.unshift(*path.split(/:/))
+  end
+
+  opts.on("-r", "--require LIBRARY",
+          "require the library, before executing your script") do |library|
+    require library
+  end
+
+  opts.separator "#{cmd} options:"
+
+  # some of these switches exist for rackup command-line compatibility,
+
+  opts.on("-o", "--host HOST",
+          "listen on HOST (default: #{Unicorn::Const::DEFAULT_HOST})") do |h|
+    rackup_opts[:host] = h
+    rackup_opts[:set_listener] = true
+  end
+
+  opts.on("-p", "--port PORT",
+          "use PORT (default: #{Unicorn::Const::DEFAULT_PORT})") do |p|
+    rackup_opts[:port] = p.to_i
+    rackup_opts[:set_listener] = true
+  end
+
+  opts.on("-E", "--env RACK_ENV",
+          "use RACK_ENV for defaults (default: development)") do |e|
+    ENV["RACK_ENV"] = e
+  end
+
+  opts.on("-N", "--no-default-middleware",
+          "do not load middleware implied by RACK_ENV") do |e|
+    rackup_opts[:no_default_middleware] = true
+  end
+
+  opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
+    rackup_opts[:daemonize] = !!d
+  end
+
+  opts.on("-P", "--pid FILE", "DEPRECATED") do |f|
+    warn %q{Use of --pid/-P is strongly discouraged}
+    warn %q{Use the 'pid' directive in the Unicorn config file instead}
+    options[:pid] = f
+  end
+
+  opts.on("-s", "--server SERVER",
+          "this flag only exists for compatibility") do |s|
+    warn "-s/--server only exists for compatibility with rackup"
+  end
+
+  # Unicorn-specific stuff
+  opts.on("-l", "--listen {HOST:PORT|PATH}",
+          "listen on HOST:PORT or PATH",
+          "this may be specified multiple times",
+          "(default: #{Unicorn::Const::DEFAULT_LISTEN})") do |address|
+    options[:listeners] << address
+  end
+
+  opts.on("-c", "--config-file FILE", "Unicorn-specific config file") do |f|
+    options[:config_file] = f
+  end
+
+  # I'm avoiding Unicorn-specific config options on the command-line.
+  # IMNSHO, config options on the command-line are redundant given
+  # config files and make things unnecessarily complicated with multiple
+  # places to look for a config option.
+
+  opts.separator "Common options:"
+
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts opts.to_s.gsub(/^.*DEPRECATED.*$/s, '')
+    exit
+  end
+
+  opts.on_tail("-v", "--version", "Show version") do
+    puts "#{cmd} v#{Unicorn::Const::UNICORN_VERSION}"
+    exit
+  end
+
+  opts.parse! ARGV
+end
+
+# ARGV[0] is usually "unicorn", but "unicorn_rails" or custom
+# BYOE wrappers work, too
+ru = ARGV[1] || 'config.ru'
+Unicorn::Configurator::RACKUP.merge!(:file => ru, :optparse => op)
+op = nil
+
+if $DEBUG
+  require 'pp'
+  pp({
+    :unicorn_options => options,
+    :daemonize => rackup_opts[:daemonize],
+  })
+end
+Unicorn::Forever.new(options).start.join
diff --git a/lib/unicorn/forever.rb b/lib/unicorn/forever.rb
new file mode 100644
index 0000000..3f6c5fe
--- /dev/null
+++ b/lib/unicorn/forever.rb
@@ -0,0 +1,289 @@
+# -*- encoding: binary -*-
+#
+# Normally the unicorn master process can handle all the restarting,
+# however with init replacements becoming process managers, we may want
+# to use something which never dies and restarts the master.
+class Unicorn::Forever
+  # :stopdoc:
+  include Unicorn::SocketHelper
+
+  attr_accessor :listener_opts, :init_listeners, :config, :logger
+
+  START_CTX = {
+    :argv => ARGV.map { |arg| arg.dup },
+  }
+
+  # We favor ENV['PWD'] since it is (usually) symlink aware for Capistrano
+  # and like systems
+  START_CTX[:cwd] = begin
+    a = File.stat(pwd = ENV['PWD'])
+    b = File.stat(Dir.pwd)
+    a.ino == b.ino && a.dev == b.dev ? pwd : Dir.pwd
+  rescue
+    Dir.pwd
+  end
+
+  def initialize(options = {})
+    @listeners = []
+    @self_pipe = []
+    @new_listeners = []
+    @init_listeners = options[:listeners] ? options[:listeners].dup : []
+    options[:use_defaults] = true
+    @config = Unicorn::Configurator.new(options)
+    @listener_opts = {}
+    @config.commit!(self, :skip => [:listeners])
+    @respawn = true
+    @self_pipe = Kgio::Pipe.new
+    @self_pipe.each { |io| io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
+    # signal queue used for self-piping
+    @sig_queue = []
+  end
+
+  def setup_sighandlers
+    %w(CHLD HUP QUIT TERM INT USR1 USR2 TTIN TTOU WINCH).each do |s|
+      trap(s) { sig_handler(s.to_sym) }
+    end
+  end
+
+  # Runs the thing.  Returns self so you can run join on it
+  def start
+    # no inheriting, just start the default if needed
+    config_listeners = @config[:listeners].dup
+    if config_listeners.empty?
+      config_listeners << Unicorn::Const::DEFAULT_LISTEN
+      @init_listeners << Unicorn::Const::DEFAULT_LISTEN
+      START_CTX[:argv] << "-l#{Unicorn::Const::DEFAULT_LISTEN}"
+    end
+    @new_listeners.replace(config_listeners)
+
+    bind_new_listeners!
+    setup_sighandlers
+    do_exec
+    self
+  end
+
+  # replaces current listener set with +listeners+.  This will
+  # close the socket if it will not exist in the new listener set
+  def listeners=(listeners)
+    cur_names, dead_names = [], []
+    listener_names.each do |name|
+      if ?/ == name[0]
+        # mark unlinked sockets as dead so we can rebind them
+        (File.socket?(name) ? cur_names : dead_names) << name
+      else
+        cur_names << name
+      end
+    end
+    set_names = listener_names(listeners)
+    dead_names.concat(cur_names - set_names).uniq!
+
+    @listeners.delete_if do |io|
+      if dead_names.include?(sock_name(io))
+        IO_PURGATORY.delete_if do |pio|
+          pio.fileno == io.fileno && (pio.close rescue nil).nil? # true
+        end
+        (io.close rescue nil).nil? # true
+      else
+        set_server_sockopt(io, listener_opts[sock_name(io)])
+        false
+      end
+    end
+
+    (set_names - cur_names).each { |addr| listen(addr) }
+  end
+
+  def stdout_path=(path); redirect_io($stdout, path); end
+  def stderr_path=(path); redirect_io($stderr, path); end
+
+  # for Configurator compatibility:
+  def noop(arg = nil); end
+  alias client_body_buffer_size= noop
+  alias client_body_buffer_size noop
+  alias check_client_connection= noop
+  alias check_client_connection noop
+  alias pid= noop
+  alias pid noop
+  alias preload_app= noop
+  alias preload_app noop
+  alias timeout= noop
+  alias timeout noop
+  alias trust_x_forwarded= noop
+  alias trust_x_forwarded noop
+  alias rewindable_input= noop
+  alias rewindable_input noop
+  alias worker_processes= noop
+  alias worker_processes noop
+  alias after_fork= noop
+  alias after_fork noop
+  alias before_fork= noop
+  alias before_fork noop
+  alias before_exec= noop
+  alias before_exec noop
+
+  # add a given address to the +listeners+ set, idempotently
+  # Allows workers to add a private, per-process listener via the
+  # after_fork hook.  Very useful for debugging and testing.
+  # +:tries+ may be specified as an option for the number of times
+  # to retry, and +:delay+ may be specified as the time in seconds
+  # to delay between retries.
+  # A negative value for +:tries+ indicates the listen will be
+  # retried indefinitely, this is useful when workers belonging to
+  # different masters are spawned during a transparent upgrade.
+  def listen(address, opt = {}.merge(listener_opts[address] || {}))
+    address = config.expand_addr(address)
+    return if String === address && listener_names.include?(address)
+
+    delay = opt[:delay] || 0.5
+    tries = opt[:tries] || 5
+    begin
+      io = bind_listen(address, opt)
+      unless Kgio::TCPServer === io || Kgio::UNIXServer === io
+        IO_PURGATORY << io
+        io = server_cast(io)
+      end
+      @logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
+      @listeners << io
+      io
+    rescue Errno::EADDRINUSE => err
+      @logger.error "adding listener failed addr=#{address} (in use)"
+      raise err if tries == 0
+      tries -= 1
+      @logger.error "retrying in #{delay} seconds " \
+                    "(#{tries < 0 ? 'infinite' : tries} tries left)"
+      sleep(delay)
+      retry
+    rescue => err
+      @logger.fatal "error adding listener addr=#{address}"
+      raise err
+    end
+  end
+
+  # reaps all unreaped processes
+  def reap_all
+    begin
+      pid, status = Process.waitpid2(-1, Process::WNOHANG)
+      pid or return
+      @logger.error "reaped #{status.inspect}"
+      @exec_pid = nil if pid == @exec_pid
+    rescue Errno::ECHILD
+      break
+    end while true
+  end
+
+  # Monitors child and receives signals forever (or until SIGKILL is sent).
+  # This handles signals one-at-a-time time.
+  # Send SIGSTOP to this process to prevent it from respawning
+  def join
+    begin
+      IO.select([@self_pipe[0]])
+      @self_pipe[0].kgio_tryread(11)
+      reap_all
+      sig = @sig_queue.shift
+
+      case sig
+      when nil # spurious wakeup
+      when :USR1 # rotate logs
+        @logger.info "unicorn-forever reopening logs..."
+        Unicorn::Util.reopen_logs
+        @logger.info "unicorn-forever done reopening logs"
+
+        # this is the only signal we always forward
+        Process.kill(sig, @exec_pid) if @exec_pid
+      when :HUP
+        # we only implement SIGHUP because we need to be aware of new
+        # sockets from updated configs
+        if @config.config_file
+          load_config!
+        else # exec binary and exit if there's no config file
+          @logger.info "SIGHUP received but config file not defined"
+        end
+      else
+        @logger.info "unhandled signal: SIG#{sig} received"
+        zero = START_CTX[:argv][0]
+        if @exec_pid
+          @logger.info("did you mean to signal `#{zero}' at PID:#@exec_pid?")
+        else
+          @logger.info("`#{zero}' not running")
+        end
+      end
+      unless @exec_pid
+        do_exec
+
+        # throttle, in case the master keeps dying, we don't want to burn
+        # cycles and fill up logs by constant respawning
+        sleep 1
+      end
+    rescue => e
+      Unicorn.log_error(@logger, "forever loop error", e)
+    end while true
+    # We don't go down easily, use SIGKILL
+  end
+
+  def sig_handler(s)
+    @sig_queue << s
+    @self_pipe[1].kgio_trywrite('^') # wakeup ourselves from IO.select
+  end
+
+  def do_exec
+    @exec_pid = fork do # This will become the unicorn master process
+      # Don't let actions on any TTY -forever may have influence us
+      Process.setsid
+
+      listener_fds = Hash[@listeners.map do |sock|
+        # IO#close_on_exec= will be available on any future version of
+        # Ruby that sets FD_CLOEXEC by default on new file descriptors
+        # ref: http://redmine.ruby-lang.org/issues/5041
+        sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
+        [ sock.fileno, sock ]
+      end]
+      ENV['UNICORN_FD'] = listener_fds.keys.join(',')
+      Dir.chdir(START_CTX[:cwd])
+      cmd = START_CTX[:argv]
+
+      # avoid leaking FDs we don't know about, but let before_exec
+      # unset FD_CLOEXEC, if anything else in the app eventually
+      # relies on FD inheritence.
+      (3..1024).each do |io|
+        next if listener_fds.include?(io)
+        io = IO.for_fd(io) rescue next
+        IO_PURGATORY << io
+        io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
+      end
+
+      # exec(command, hash) works in at least 1.9.1+, but will only be
+      # required in 1.9.4/2.0.0 at earliest.
+      cmd << listener_fds if RUBY_VERSION >= "1.9.1"
+      @logger.info "executing #{cmd.inspect} (in #{Dir.pwd})"
+      exec(*cmd)
+    end
+  end
+
+  def load_config!
+    @logger.info "reloading config_file=#{@config.config_file}"
+    @config[:listeners].replace(@init_listeners)
+    @config.reload
+    @config.commit!(self)
+    Unicorn::Util.reopen_logs
+    @logger.info "done reloading config_file=#{@config.config_file}"
+  rescue StandardError, LoadError, SyntaxError => e
+    Unicorn.log_error(@logger,
+        "error reloading config_file=#{@config.config_file}", e)
+  end
+
+  # returns an array of string names for the given listener array
+  def listener_names(listeners = @listeners)
+    listeners.map { |io| sock_name(io) }
+  end
+
+  def redirect_io(io, path)
+    File.open(path, 'ab') { |fp| io.reopen(fp) } if path
+    io.sync = true
+  end
+
+  # This binds any listeners we did NOT inherit from the parent
+  def bind_new_listeners!
+    @new_listeners.each { |addr| listen(addr) }
+    raise ArgumentError, "no listeners" if @listeners.empty?
+    @new_listeners.clear
+  end
+end
-- 
Eric Wong

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 5%]

* Re: HEAD responses contain body
  2013-06-13 19:28  0%       ` Jonathan Rudenberg
@ 2013-06-13 19:34  0%         ` Jonathan Rudenberg
  0 siblings, 0 replies; 200+ results
From: Jonathan Rudenberg @ 2013-06-13 19:34 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list


On Jun 13, 2013, at 3:28 PM, Jonathan Rudenberg <jonathan@titanous.com> wrote:

> 
> On Jun 13, 2013, at 3:21 PM, Eric Wong <normalperson@yhbt.net> wrote:
> 
>> Jonathan Rudenberg <jonathan@titanous.com> wrote:
>>> On Jun 13, 2013, at 2:22 PM, Eric Wong <normalperson@yhbt.net> wrote:
>>>> Jonathan Rudenberg <jonathan@titanous.com> wrote:
>>>>> RFC 2616 section 9.4[1] states:
>>>>> 
>>>>>> The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.
>>>>> 
>>>>> A HEAD request against this simple Rack app running on unicorn-4.6.2:
>>>>> 
>>>>>  require 'rack'
>>>>> 
>>>> 
>>>> +     use Rack::Head
>>>> 
>>>>>  run lambda { |env| [200, {}, []] }
>>>> 
>>>> The Rack::Head middleware should be used to correctly strip HEAD
>>>> responses of their bodies (frameworks such as Rails/Sinatra should
>>>> already add Rack::Head to the middleware stack for you)
>>> 
>>> This does not change the result, as the Rack::Head implementation looks like this:
>>> 
>>>   def call(env)
>>>     status, headers, body = @app.call(env)
>>> 
>>>     if env["REQUEST_METHOD"] == "HEAD"
>>>       body.close if body.respond_to? :close
>>>       [status, headers, []]
>>>     else
>>>       [status, headers, body]
>>>     end
>>>   end
>> 
>> OK, I think you were hitting another problem because you were lacking
>> Rack::ContentType
>> 
>> Try the following:
>> -----------------------8<---------------------
>> require 'rack'
>> use Rack::ContentLength # less ambiguous than Rack::Chunked adding '0'
>> use Rack::Head
>> use Rack::ContentType
>> run lambda { |env| [200, {}, []] }
>> -----------------------8<---------------------
> 
> Thanks, this stack works.
> 
>> I added the Rack::ContentLength (it's already in the default middleware
>> stack) since I believe Rack::Chunked adding the '0' is a violation of
>> rfc2616... I'll need to read more closely to be sure.
> 
> Hmm, so this is a bug in Rack::Chunked? My reading of the spec says that the '0' is incorrect.

Actually, the solution is that Rack::Head needs to come before Rack::Chunked. Perhaps Rack's default development stack should include this?

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: HEAD responses contain body
  2013-06-13 18:42  0%   ` Jonathan Rudenberg
@ 2013-06-13 19:21  0%     ` Eric Wong
  2013-06-13 19:28  0%       ` Jonathan Rudenberg
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-06-13 19:21 UTC (permalink / raw)
  To: Jonathan Rudenberg; +Cc: unicorn list

Jonathan Rudenberg <jonathan@titanous.com> wrote:
> On Jun 13, 2013, at 2:22 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Jonathan Rudenberg <jonathan@titanous.com> wrote:
> >> RFC 2616 section 9.4[1] states:
> >> 
> >>> The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.
> >> 
> >> A HEAD request against this simple Rack app running on unicorn-4.6.2:
> >> 
> >>    require 'rack'
> >> 
> > 
> > +     use Rack::Head
> > 
> >>    run lambda { |env| [200, {}, []] }
> > 
> > The Rack::Head middleware should be used to correctly strip HEAD
> > responses of their bodies (frameworks such as Rails/Sinatra should
> > already add Rack::Head to the middleware stack for you)
> 
> This does not change the result, as the Rack::Head implementation looks like this:
> 
>     def call(env)
>       status, headers, body = @app.call(env)
> 
>       if env["REQUEST_METHOD"] == "HEAD"
>         body.close if body.respond_to? :close
>         [status, headers, []]
>       else
>         [status, headers, body]
>       end
>     end

OK, I think you were hitting another problem because you were lacking
Rack::ContentType

Try the following:
-----------------------8<---------------------
require 'rack'
use Rack::ContentLength # less ambiguous than Rack::Chunked adding '0'
use Rack::Head
use Rack::ContentType
run lambda { |env| [200, {}, []] }
-----------------------8<---------------------

I added the Rack::ContentLength (it's already in the default middleware
stack) since I believe Rack::Chunked adding the '0' is a violation of
rfc2616... I'll need to read more closely to be sure.

> >>    HTTP/1.1 500 Internal Server Error
> > 
> >> As you can see, not only is there a zero-length chunked encoding body,
> >> but for some unknown reason there is a 500 response with no body as
> >> well.
> > 
> > Try using "-d" on the command-line to enable debugging to see what the
> > error is (and check the logs/stderr output).
> 
>     Exception `Errno::ENOTCONN' at /Users/titanous/.gem/ruby/1.9.3/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:565 - Socket is not connected

Ugh, that's an unfortunate side effect of the client closing the
connection, first :/

> > Also, what RACK_ENV (or -E/--env) are you using?  It could be the
> > incorrect HEAD response tripping Rack::Lint under development mode.
> 
> None, specified, I'm booting unicorn with no configuration or flags specified.

That defaults the RACK_ENV to to "development", so you got
Rack::ContentLength, Rack::Chunked, Rack::CommonLogger,
Rack::ShowExceptions and Rack::Lint
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: HEAD responses contain body
  2013-06-13 19:21  0%     ` Eric Wong
@ 2013-06-13 19:28  0%       ` Jonathan Rudenberg
  2013-06-13 19:34  0%         ` Jonathan Rudenberg
  0 siblings, 1 reply; 200+ results
From: Jonathan Rudenberg @ 2013-06-13 19:28 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list


On Jun 13, 2013, at 3:21 PM, Eric Wong <normalperson@yhbt.net> wrote:

> Jonathan Rudenberg <jonathan@titanous.com> wrote:
>> On Jun 13, 2013, at 2:22 PM, Eric Wong <normalperson@yhbt.net> wrote:
>>> Jonathan Rudenberg <jonathan@titanous.com> wrote:
>>>> RFC 2616 section 9.4[1] states:
>>>> 
>>>>> The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.
>>>> 
>>>> A HEAD request against this simple Rack app running on unicorn-4.6.2:
>>>> 
>>>>   require 'rack'
>>>> 
>>> 
>>> +     use Rack::Head
>>> 
>>>>   run lambda { |env| [200, {}, []] }
>>> 
>>> The Rack::Head middleware should be used to correctly strip HEAD
>>> responses of their bodies (frameworks such as Rails/Sinatra should
>>> already add Rack::Head to the middleware stack for you)
>> 
>> This does not change the result, as the Rack::Head implementation looks like this:
>> 
>>    def call(env)
>>      status, headers, body = @app.call(env)
>> 
>>      if env["REQUEST_METHOD"] == "HEAD"
>>        body.close if body.respond_to? :close
>>        [status, headers, []]
>>      else
>>        [status, headers, body]
>>      end
>>    end
> 
> OK, I think you were hitting another problem because you were lacking
> Rack::ContentType
> 
> Try the following:
> -----------------------8<---------------------
> require 'rack'
> use Rack::ContentLength # less ambiguous than Rack::Chunked adding '0'
> use Rack::Head
> use Rack::ContentType
> run lambda { |env| [200, {}, []] }
> -----------------------8<---------------------

Thanks, this stack works.

> I added the Rack::ContentLength (it's already in the default middleware
> stack) since I believe Rack::Chunked adding the '0' is a violation of
> rfc2616... I'll need to read more closely to be sure.

Hmm, so this is a bug in Rack::Chunked? My reading of the spec says that the '0' is incorrect.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: HEAD responses contain body
  2013-06-13 18:22  5% ` Eric Wong
@ 2013-06-13 18:42  0%   ` Jonathan Rudenberg
  2013-06-13 19:21  0%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Jonathan Rudenberg @ 2013-06-13 18:42 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list, "wrote:"


On Jun 13, 2013, at 2:22 PM, Eric Wong <normalperson@yhbt.net> wrote:

> Jonathan Rudenberg <jonathan@titanous.com> wrote:
>> RFC 2616 section 9.4[1] states:
>> 
>>> The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.
>> 
>> A HEAD request against this simple Rack app running on unicorn-4.6.2:
>> 
>>    require 'rack'
>> 
> 
> +     use Rack::Head
> 
>>    run lambda { |env| [200, {}, []] }
> 
> The Rack::Head middleware should be used to correctly strip HEAD
> responses of their bodies (frameworks such as Rails/Sinatra should
> already add Rack::Head to the middleware stack for you)

This does not change the result, as the Rack::Head implementation looks like this:

    def call(env)
      status, headers, body = @app.call(env)

      if env["REQUEST_METHOD"] == "HEAD"
        body.close if body.respond_to? :close
        [status, headers, []]
      else
        [status, headers, body]
      end
    end

>> Looks like this on the wire:
>> 
>>    HEAD / HTTP/1.1
>>    User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5
>>    Host: localhost:8080
>>    Accept: */*
>> 
>>    HTTP/1.1 200 OK
>>    Date: Thu, 13 Jun 2013 16:04:55 GMT
>>    Status: 200 OK
>>    Connection: close
>>    Transfer-Encoding: chunked
>> 
>>    0
>> 
>>    HTTP/1.1 500 Internal Server Error
> 
>> As you can see, not only is there a zero-length chunked encoding body,
>> but for some unknown reason there is a 500 response with no body as
>> well.
> 
> Try using "-d" on the command-line to enable debugging to see what the
> error is (and check the logs/stderr output).

    Exception `Errno::ENOTCONN' at /Users/titanous/.gem/ruby/1.9.3/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:565 - Socket is not connected

> Also, what RACK_ENV (or -E/--env) are you using?  It could be the
> incorrect HEAD response tripping Rack::Lint under development mode.

None, specified, I'm booting unicorn with no configuration or flags specified.

> 
>> Please cc any responses directly to me, as I do not subscribe to this list.
> 
> Done :>

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: HEAD responses contain body
  @ 2013-06-13 18:22  5% ` Eric Wong
  2013-06-13 18:42  0%   ` Jonathan Rudenberg
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-06-13 18:22 UTC (permalink / raw)
  To: unicorn list; +Cc: Jonathan Rudenberg, "wrote:"

Jonathan Rudenberg <jonathan@titanous.com> wrote:
> RFC 2616 section 9.4[1] states:
> 
> > The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.
> 
> A HEAD request against this simple Rack app running on unicorn-4.6.2:
> 
>     require 'rack'
> 

+     use Rack::Head

>     run lambda { |env| [200, {}, []] }

The Rack::Head middleware should be used to correctly strip HEAD
responses of their bodies (frameworks such as Rails/Sinatra should
already add Rack::Head to the middleware stack for you)

> Looks like this on the wire:
> 
>     HEAD / HTTP/1.1
>     User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5
>     Host: localhost:8080
>     Accept: */*
> 
>     HTTP/1.1 200 OK
>     Date: Thu, 13 Jun 2013 16:04:55 GMT
>     Status: 200 OK
>     Connection: close
>     Transfer-Encoding: chunked
> 
>     0
> 
>     HTTP/1.1 500 Internal Server Error

> As you can see, not only is there a zero-length chunked encoding body,
> but for some unknown reason there is a 500 response with no body as
> well.

Try using "-d" on the command-line to enable debugging to see what the
error is (and check the logs/stderr output).

Also, what RACK_ENV (or -E/--env) are you using?  It could be the
incorrect HEAD response tripping Rack::Lint under development mode.

> Please cc any responses directly to me, as I do not subscribe to this list.

Done :>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Re: add HTTP response headers
  2013-05-12 19:20  5% ` Eric Wong
  2013-05-12 20:25  6%   ` pille
@ 2013-05-12 21:37  6%   ` Alejandro Riera
  1 sibling, 0 replies; 200+ results
From: Alejandro Riera @ 2013-05-12 21:37 UTC (permalink / raw)
  To: unicorn list; +Cc: unicorn list, pille

Wow! Awesome answer, you rock!

Enviado desde el movil

El 12/05/2013, a las 21:20, Eric Wong <normalperson@yhbt.net> escribió:

> pille <pille+unicorn+mailinglist@struction.de> wrote:
>> hi,
>> 
>> is it possible to add static and dynamic HTTP response headers in
>> unicorn.conf.rb?
> 
> No, but it is easily possible with a Rack config.ru
> 
>> i'd like to add the hostname of the worker for debugging and a
>> timestamp, when the request was worked on.
>> 
>> currently this is done in nginx, which should be stripped from the stack.
>> i'd like to keep it out of the webapp itself, because it's infrastructure.
> 
> Can you consider config.ru infrastructure?  unicorn tries to do as
> much generically via Rack as possible.
> 
> The following (totally untested) middleware should work:
> ------------------------8<------------------------
> require 'time'
> require 'socket'
> require 'rack/utils'
> 
> # Usage (in config.ru)
> #   require 'name/of/this/file'
> #   use ExtraHeaders
> #   # other middlewares ...
> #   run YourApp.new
> class ExtraHeaders
>  def initialize(app)
>    @app = app
>  end
> 
>  def call(env)
>    start = Time.now.httpdate
>    status, headers, body = @app.call(env)
>    headers = Rack::Utils::HeaderHash.new(headers)
>    headers["X-Hostname"] = Socket.gethostname
>    headers["X-Start"] = start
>    [ status, headers, body ]
>  end
> end
> ------------------------8<------------------------
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: add HTTP response headers
  2013-05-12 19:20  5% ` Eric Wong
@ 2013-05-12 20:25  6%   ` pille
  2013-05-12 21:37  6%   ` Alejandro Riera
  1 sibling, 0 replies; 200+ results
From: pille @ 2013-05-12 20:25 UTC (permalink / raw)
  To: unicorn list

eric, thanks for your comprehensive help!

  pille
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: add HTTP response headers
  2013-05-12 18:12 13% add HTTP response headers pille
@ 2013-05-12 19:20  5% ` Eric Wong
  2013-05-12 20:25  6%   ` pille
  2013-05-12 21:37  6%   ` Alejandro Riera
  0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2013-05-12 19:20 UTC (permalink / raw)
  To: unicorn list; +Cc: pille

pille <pille+unicorn+mailinglist@struction.de> wrote:
> hi,
> 
> is it possible to add static and dynamic HTTP response headers in
> unicorn.conf.rb?

No, but it is easily possible with a Rack config.ru

> i'd like to add the hostname of the worker for debugging and a
> timestamp, when the request was worked on.
> 
> currently this is done in nginx, which should be stripped from the stack.
> i'd like to keep it out of the webapp itself, because it's infrastructure.

Can you consider config.ru infrastructure?  unicorn tries to do as
much generically via Rack as possible.

The following (totally untested) middleware should work:
------------------------8<------------------------
require 'time'
require 'socket'
require 'rack/utils'

# Usage (in config.ru)
#   require 'name/of/this/file'
#   use ExtraHeaders
#   # other middlewares ...
#   run YourApp.new
class ExtraHeaders
  def initialize(app)
    @app = app
  end

  def call(env)
    start = Time.now.httpdate
    status, headers, body = @app.call(env)
    headers = Rack::Utils::HeaderHash.new(headers)
    headers["X-Hostname"] = Socket.gethostname
    headers["X-Start"] = start
    [ status, headers, body ]
  end
end
------------------------8<------------------------
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* add HTTP response headers
@ 2013-05-12 18:12 13% pille
  2013-05-12 19:20  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: pille @ 2013-05-12 18:12 UTC (permalink / raw)
  To: mongrel-unicorn

hi,

is it possible to add static and dynamic HTTP response headers in
unicorn.conf.rb?

i'd like to add the hostname of the worker for debugging and a
timestamp, when the request was worked on.

currently this is done in nginx, which should be stripped from the stack.
i'd like to keep it out of the webapp itself, because it's infrastructure.

cheers
  pille
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 13%]

* Re: [PATCH] Add -N or --no-default-middleware option.
  2013-01-29  3:52  5% ` Eric Wong
  2013-01-29  4:02  5%   ` Eric Wong
@ 2013-01-29  4:03  6%   ` Lin Jen-Shin (godfat)
  1 sibling, 0 replies; 200+ results
From: Lin Jen-Shin (godfat) @ 2013-01-29  4:03 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list

On Tue, Jan 29, 2013 at 11:52 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Lin Jen-Shin <godfat@godfat.org> wrote:
>> +  opts.on("-N", "--no-default-middleware",
>> +          "no default middleware even if RACK_ENV is development") do |e|
>
> RACK_ENV=deployment also loads middleware, so I think it's more accurate
> with:
>
>         do not load middleware implied by RACK_ENV
>
> This also puts us back within the 80-column limit imposed by "test_help"
> in test/exec/test_exec.rb

Oh right, I only remembered to keep the source within 80-column limit,
but forgot that the output might be longer than source. Thanks for the
better wordings. I updated the patch for Zbatery, but it's too late
for Rainbows though.

> Will sign-off and push that change squashed unless there are objections.

Surely, thanks!

> I also just bumped master to rack 1.5.1, which fixes the Rack::Lint
> test regression in 1.5.0.
>
> Will pull in the "hijack" branch into master soon (and release 4.6.0)
> since all my questions about it have been answered in rack-devel.

I haven't had a chance to read hijack more, looking forward to it :)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add -N or --no-default-middleware option.
  2013-01-29  3:52  5% ` Eric Wong
@ 2013-01-29  4:02  5%   ` Eric Wong
  2013-01-29  4:03  6%   ` Lin Jen-Shin (godfat)
  1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2013-01-29  4:02 UTC (permalink / raw)
  To: unicorn list; +Cc: Lin Jen-Shin

Eric Wong <normalperson@yhbt.net> wrote:
> Lin Jen-Shin <godfat@godfat.org> wrote:
> > +  opts.on("-N", "--no-default-middleware",
> > +          "no default middleware even if RACK_ENV is development") do |e|
> 
> RACK_ENV=deployment also loads middleware, so I think it's more accurate
> with:
> 
> 	do not load middleware implied by RACK_ENV
> 
> This also puts us back within the 80-column limit imposed by "test_help"
> in test/exec/test_exec.rb

Heh, turns out there's a bug in test_exec.rb (but I still don't think
mentioning "development" for this help is correct)

>From ee3ada3c697311ab0a799fcdc492ae062e7d5128 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Tue, 29 Jan 2013 03:56:16 +0000
Subject: [PATCH] test_exec: do not count '\n' as column width

This off-by-one error was incorrectly rejecting a line which
would've been readable without wrapping on an 80-column terminal.
---
 test/exec/test_exec.rb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 1cee2b7..10a1bae 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -323,6 +323,7 @@ EOF
     # mobile phone or netbook on a slow connection :)
     assert lines.size <= 24, "help height fits in an ANSI terminal window"
     lines.each do |line|
+      line.chomp!
       assert line.size <= 80, "help width fits in an ANSI terminal window"
     end
   end
-- 
1.8.1.1.253.g2934a48

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 5%]

* Re: [PATCH] Add -N or --no-default-middleware option.
  2013-01-29  3:21  8% [PATCH] Add -N or --no-default-middleware option Lin Jen-Shin
@ 2013-01-29  3:52  5% ` Eric Wong
  2013-01-29  4:02  5%   ` Eric Wong
  2013-01-29  4:03  6%   ` Lin Jen-Shin (godfat)
  0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2013-01-29  3:52 UTC (permalink / raw)
  To: unicorn list; +Cc: Lin Jen-Shin

Lin Jen-Shin <godfat@godfat.org> wrote:
> +  opts.on("-N", "--no-default-middleware",
> +          "no default middleware even if RACK_ENV is development") do |e|

RACK_ENV=deployment also loads middleware, so I think it's more accurate
with:

	do not load middleware implied by RACK_ENV

This also puts us back within the 80-column limit imposed by "test_help"
in test/exec/test_exec.rb

Will sign-off and push that change squashed unless there are objections.

I also just bumped master to rack 1.5.1, which fixes the Rack::Lint
test regression in 1.5.0.

Will pull in the "hijack" branch into master soon (and release 4.6.0)
since all my questions about it have been answered in rack-devel.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* [PATCH] Add -N or --no-default-middleware option.
@ 2013-01-29  3:21  8% Lin Jen-Shin
  2013-01-29  3:52  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Lin Jen-Shin @ 2013-01-29  3:21 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Lin Jen-Shin

This would prevent Unicorn from adding default middleware,
as if RACK_ENV were always none. (not development nor deployment)

This should also be applied to `rainbows' and `zbatery' as well.

One of the reasons to add this is to avoid conflicting
RAILS_ENV and RACK_ENV. It would be helpful in the case
where a Rails application and Rack application are composed
together, while we want Rails app runs under development
and Rack app runs under none (if we don't want those default
middleware), and we don't really want to make RAILS_ENV
set to development and RACK_ENV to none because it might be
confusing. Note that Rails would also look into RACK_ENV.

Another reason for this is that only `rackup' would be
inserting those default middleware. Both `thin' and `puma'
would not do this, nor does Rack::Handler.get.run which is
used in Sinatra.

So using this option would make it work differently from
`rackup' but somehow more similar to `thin' or `puma'.

Discussion thread on the mailing list:
http://rubyforge.org/pipermail/mongrel-unicorn/2013-January/001675.html
---
 bin/unicorn    | 5 +++++
 lib/unicorn.rb | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/bin/unicorn b/bin/unicorn
index 9962b58..415d164 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -58,6 +58,11 @@ op = OptionParser.new("", 24, '  ') do |opts|
     ENV["RACK_ENV"] = e
   end
 
+  opts.on("-N", "--no-default-middleware",
+          "no default middleware even if RACK_ENV is development") do |e|
+    rackup_opts[:no_default_middleware] = true
+  end
+
   opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
     rackup_opts[:daemonize] = !!d
   end
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index d96ff91..f0ceffe 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -49,6 +49,8 @@ module Unicorn
 
       pp({ :inner_app => inner_app }) if $DEBUG
 
+      return inner_app if op[:no_default_middleware]
+
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
       # and Zbatery both do.  Users accustomed to the Rack::Server default
-- 
1.8.1.1

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 8%]

* Re: No middleware without touching RACK_ENV
  2013-01-25 18:39  0% ` Eric Wong
@ 2013-01-28 14:43  0%   ` Lin Jen-Shin (godfat)
  0 siblings, 0 replies; 200+ results
From: Lin Jen-Shin (godfat) @ 2013-01-28 14:43 UTC (permalink / raw)
  To: unicorn list

On Sat, Jan 26, 2013 at 2:39 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Doesn't Rails favor RAILS_ENV over RACK_ENV?  unicorn ignores RAILS_ENV
> (unicorn_rails respects RAILS_ENV, but unicorn_rails isn't recommended
> for modern (Rack-enabled) Rails)

Yes, it is. But it would still look into RACK_ENV if RAILS_ENV is not available.
One solution would be setting RAILS_ENV=development while setting
RACK_ENV=none. I don't really want to do this for several reasons:

* It would be better to keep RAILS_ENV == RACK_ENV to avoid confusion.
* It might be tricky to make RAILS_ENV default to development while setting
  RACK_ENV since Rails would be looking into it. So we need to make sure
  that Rails is loaded and default to development and then we can setup
  RACK_ENV to avoid messing up the original Rails default.

> Is there a benefit which RACK_ENV=development gives you for Rails
> that RAILS_ENV=development does not give you?

The main reason is that I want to avoid confusion. The other reason
is that that's also how Heroku tries to use. I think it's also because of
consistency. (so that we don't have to distinguish RACK_ENV and RAILS_ENV)

The other reason would be making them different might cause some
issues as Rails would also be looking at RACK_ENV.

> Fwiw, the times I worked on Rails, I used customized dev environments
> anyways, so I would use RAILS_ENV=dev_local or RAILS_ENV=dev_$NETWORK
> for switching between a localhost-only vs networked setup.

I would also do this I guess, but since I am not the only one working on
the application, I would avoid making such changes. We have a lot of
configurations depending on the word "development" already.

>> I know this is somehow a spec from Rack, but I guess I
>> don't like this behaviour. One workaround would be
>> providing a `after_app_load' hook, and we add this to
>> the bottom of config.ru:
>>
>>   ENV['RACK_ENV_ORIGINAL'] = ENV['RACK_ENV']
>>   ENV['RACK_ENV'] = 'none'
>>
>> and add this to the unicorn configuration file:
>>
>>   after_app_load do |_|
>>     ENV['RACK_ENV'] = ENV['RACK_ENV_ORIGINAL']
>>   end
>>
>> This is probably a stupid hack, but I wonder if after_app_load
>> hook would be useful for other cases?
>
> I don't see how this hook has benefits over putting this in config.ru
> (or somewhere else in your app, since your app is already loaded...)

It's better to me because it's clear on the loading order.
The point where to change RACK_ENV=none would be
after Rails loaded (to avoid messing Rails up), and
_before Unicorn inserting middleware_. And then I would
like to switch RACK_ENV=RAILS_ENV to avoid confusion.
It's hard to tell where I can put this code.

> I'd imagine users wanting the same app-wide settings going between
> different application servers, too.

Speaking to this, I might want to complain about the other severs :P
As far as I know, only if you use `rackup' would you get those middleware.

That is, `unicorn' is compatible with `rackup'. That's a good thing.
But it seems to me that both `thin' and `puma' (and maybe including
others, I didn't check) would not do this if you use their command line
tools. They even have different options to enable Rack::CommonLogger.

Actually this is also the reason why sometimes `thin' and `unicorn'
worked differently when people switching from Thin to Unicorn.
And you know a lot of people are using Thin in the first place,
thus sometimes blaming Unicorn works differently.

I am not sure how many people are using `rackup', but according to
my observation, not too much. If people are using Sinatra directly,
eventually they would launch the server via Rack::Handler.get.run,
which does not try to wrap additional middleware as what `rackup'
would do if I read correctly.

So I am not sure if a lot of people would be expecting this actually.
I used to use `rackup' a lot to get consistent behaviour, but it looks to
me not many people even know the existence of rackup :(

> I'm against adding more command-line or config options, especially
> around loading middleware.  RACK_ENV is already a source of confusion,
> and Rails also has/(or had?) it's own middleware loading scheme
> independent of config.ru.

Yes, I hate that, too. I can't complain about Rails less :P I don't understand
why people are building those middleware scheme on their own way,
even Sinatra does this. I guess they want more control over middleware,
so that they can reorder, delete, insert more/less middleware somehow.
I guess that makes sense in some way... but they should be really in
Rack instead of each web framework IMHO.

> git format-patch + git send-email is the recommended way if your
> mail client cannot be taught to leave formatting alone.  Also this
> is easy for sending multiple patches in the same thread.
>
> git format-patch --stdout with a good mail client which will not wrap
> messages is also great (especially for single patches).
>
> Attachments (generated using git format-patch) for text/x-diff should be
> accepted by Mailman.  Attached patches are a last resort, I prefer
> inline patches since I can quote/reply to them inline.
>
> (same patch submission guidelines as git or Linux kernel)

Last time I tried to setup SMTP for git send-email failed.
I'll try again next time. Thanks!

> Commit messages should explain why a change is made/needed, not just
> what it does.

Yeah, I know, but I can understand if none of them is acceptable,
(that's why I didn't bring this up in the first place) and the reason to
build those features might be arguing. So instead of writing them
carefully, I decided to bring this up sooner and then we can work out
the true reason later on :D

Thanks a lot for your time.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: No middleware without touching RACK_ENV
  2013-01-25 10:52  6% No middleware without touching RACK_ENV Lin Jen-Shin (godfat)
@ 2013-01-25 18:39  0% ` Eric Wong
  2013-01-28 14:43  0%   ` Lin Jen-Shin (godfat)
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2013-01-25 18:39 UTC (permalink / raw)
  To: unicorn list

"Lin Jen-Shin (godfat)" <godfat@godfat.org> wrote:
> Hi,
> 
> This might be a bit silly, but finally I decided to bring this up.
> 
> We're running a Rails app along with another Rack app in
> the same config.ru, and what I want to do is telling Unicorn
> that we don't want any middleware which Unicorn might
> insert with RACK_ENV=development and RACK_ENV=
> deployment, because we're adding Rack::CommonLogger
> and other middleware by ourselves, while we're using
> RACK_ENV=development for Rails when developing.

Doesn't Rails favor RAILS_ENV over RACK_ENV?  unicorn ignores RAILS_ENV
(unicorn_rails respects RAILS_ENV, but unicorn_rails isn't recommended
for modern (Rack-enabled) Rails)

> That is, we want to use RACK_ENV=development for Rails
> because that's how it used to be, but we also don't want
> those additional middleware got inserted automatically.
> This has no problem with RACK_ENV=production.

Is there a benefit which RACK_ENV=development gives you for Rails
that RAILS_ENV=development does not give you?

Fwiw, the times I worked on Rails, I used customized dev environments
anyways, so I would use RAILS_ENV=dev_local or RAILS_ENV=dev_$NETWORK
for switching between a localhost-only vs networked setup.

> I know this is somehow a spec from Rack, but I guess I
> don't like this behaviour. One workaround would be
> providing a `after_app_load' hook, and we add this to
> the bottom of config.ru:
> 
>   ENV['RACK_ENV_ORIGINAL'] = ENV['RACK_ENV']
>   ENV['RACK_ENV'] = 'none'
> 
> and add this to the unicorn configuration file:
> 
>   after_app_load do |_|
>     ENV['RACK_ENV'] = ENV['RACK_ENV_ORIGINAL']
>   end
> 
> This is probably a stupid hack, but I wonder if after_app_load
> hook would be useful for other cases?

I don't see how this hook has benefits over putting this in config.ru
(or somewhere else in your app, since your app is already loaded...)

I'd imagine users wanting the same app-wide settings going between
different application servers, too.

> Or if we could have an option to turn off this Rack behaviour
> simulation, like:
> 
>     default_middleware false
> 
> That might be more straightforward for what we want. Oh but
> it seems it's not that easy to add this. What about an option
> for unicorn?
> 
>     unicorn -E development -N
> 
> or
> 
>     unicorn -E development --no-default-middleware

I'm against adding more command-line or config options, especially
around loading middleware.  RACK_ENV is already a source of confusion,
and Rails also has/(or had?) it's own middleware loading scheme
independent of config.ru.

> This would need to be applied to `rainbows' and `zbatery', too,
> though. Patches below for consideration:
> (Sorry if Gmail messed the format up, but from last time
> I tried, it doesn't accept attachments :( What's the best way?
> Links to raw patches?)

git format-patch + git send-email is the recommended way if your
mail client cannot be taught to leave formatting alone.  Also this
is easy for sending multiple patches in the same thread.

git format-patch --stdout with a good mail client which will not wrap
messages is also great (especially for single patches).

Attachments (generated using git format-patch) for text/x-diff should be
accepted by Mailman.  Attached patches are a last resort, I prefer
inline patches since I can quote/reply to them inline.

(same patch submission guidelines as git or Linux kernel)

> https://github.com/godfat/unicorn/pull/2
> 
> commit 95de5abf38a81a76af15476d4705713d2d644664
> Author: Lin Jen-Shin <godfat@godfat.org>
> Date:   Fri Jan 25 18:18:21 2013 +0800
> 
>     Add `after_app_load' hook.
> 
>     The hook would be called right after application is loaded.

Commit messages should explain why a change is made/needed, not just
what it does.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* No middleware without touching RACK_ENV
@ 2013-01-25 10:52  6% Lin Jen-Shin (godfat)
  2013-01-25 18:39  0% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Lin Jen-Shin (godfat) @ 2013-01-25 10:52 UTC (permalink / raw)
  To: unicorn list

Hi,

This might be a bit silly, but finally I decided to bring this up.

We're running a Rails app along with another Rack app in
the same config.ru, and what I want to do is telling Unicorn
that we don't want any middleware which Unicorn might
insert with RACK_ENV=development and RACK_ENV=
deployment, because we're adding Rack::CommonLogger
and other middleware by ourselves, while we're using
RACK_ENV=development for Rails when developing.

That is, we want to use RACK_ENV=development for Rails
because that's how it used to be, but we also don't want
those additional middleware got inserted automatically.
This has no problem with RACK_ENV=production.

I know this is somehow a spec from Rack, but I guess I
don't like this behaviour. One workaround would be
providing a `after_app_load' hook, and we add this to
the bottom of config.ru:

  ENV['RACK_ENV_ORIGINAL'] = ENV['RACK_ENV']
  ENV['RACK_ENV'] = 'none'

and add this to the unicorn configuration file:

  after_app_load do |_|
    ENV['RACK_ENV'] = ENV['RACK_ENV_ORIGINAL']
  end

This is probably a stupid hack, but I wonder if after_app_load
hook would be useful for other cases?

Or if we could have an option to turn off this Rack behaviour
simulation, like:

    default_middleware false

That might be more straightforward for what we want. Oh but
it seems it's not that easy to add this. What about an option
for unicorn?

    unicorn -E development -N

or

    unicorn -E development --no-default-middleware

This would need to be applied to `rainbows' and `zbatery', too,
though. Patches below for consideration:
(Sorry if Gmail messed the format up, but from last time
I tried, it doesn't accept attachments :( What's the best way?
Links to raw patches?)

https://github.com/godfat/unicorn/pull/2

commit 95de5abf38a81a76af15476d4705713d2d644664
Author: Lin Jen-Shin <godfat@godfat.org>
Date:   Fri Jan 25 18:18:21 2013 +0800

    Add `after_app_load' hook.

    The hook would be called right after application is loaded.

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7651093..332bdbc 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -43,6 +43,9 @@ class Unicorn::Configurator
     :before_exec => lambda { |server|
         server.logger.info("forked child re-executing...")
       },
+    :after_app_load => lambda { |server|
+        server.logger.info("application loaded")
+      },
     :pid => nil,
     :preload_app => false,
     :check_client_connection => false,
@@ -171,6 +174,13 @@ class Unicorn::Configurator
     set_hook(:before_exec, block_given? ? block : args[0], 1)
   end

+  # sets the after_app_load hook to a given Proc object.  This
+  # Proc object will be called by the master process right
+  # after application loaded.
+  def after_app_load(*args, &block)
+    set_hook(:after_app_load, block_given? ? block : args[0], 1)
+  end
+
   # sets the timeout of worker processes to +seconds+.  Workers
   # handling the request/app.call/response cycle taking longer than
   # this time period will be forcibly killed (via SIGKILL).  This
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index aa98aeb..a3b30ee 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -14,6 +14,7 @@ class Unicorn::HttpServer
   # :stopdoc:
   attr_accessor :app, :request, :timeout, :worker_processes,
                 :before_fork, :after_fork, :before_exec,
+                :after_app_load,
                 :listener_opts, :preload_app,
                 :reexec_pid, :orig_app, :init_listeners,
                 :master_pid, :config, :ready_pipe, :user
@@ -716,6 +717,7 @@ class Unicorn::HttpServer
         Gem.refresh
       end
       self.app = app.call
+      config.after_app_load.call(self)
     end
   end






And --no-default-middleware
https://github.com/godfat/unicorn/pull/3

commit e3575db2a36e3ca2acda18bfee97bf95609a9860
Author: Lin Jen-Shin <godfat@godfat.org>
Date:   Fri Jan 25 18:38:52 2013 +0800

    Add -N or --no-default-middleware option.

    This would prevent Unicorn from adding default middleware,
    as if RACK_ENV is always none. (not development nor deployment)

    This should also apply to `rainbows' and `zbatery'.

diff --git a/bin/unicorn b/bin/unicorn
index 9962b58..415d164 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -58,6 +58,11 @@ op = OptionParser.new("", 24, '  ') do |opts|
     ENV["RACK_ENV"] = e
   end

+  opts.on("-N", "--no-default-middleware",
+          "no default middleware even if RACK_ENV is development") do |e|
+    rackup_opts[:no_default_middleware] = true
+  end
+
   opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
     rackup_opts[:daemonize] = !!d
   end
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index d96ff91..f0ceffe 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -49,6 +49,8 @@ module Unicorn

       pp({ :inner_app => inner_app }) if $DEBUG

+      return inner_app if op[:no_default_middleware]
+
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
       # and Zbatery both do.  Users accustomed to the Rack::Server default
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 6%]

* [ANN] unicorn 4.5.0 (final) - check_client_connection option
@ 2012-12-08  0:17  4% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-12-08  0:17 UTC (permalink / raw)
  To: mongrel-unicorn

Changes:

The new check_client_connection option allows unicorn to detect
most disconnected local clients before potentially expensive
application processing begins.

This feature is useful for applications experiencing spikes of
traffic leading to undesirable queue times, as clients will
disconnect (and perhaps even retry, compounding the problem)
before unicorn can even start processing the request.

To enable this feature, add the following line to a unicorn
config file:

      check_client_connection true

This feature only works when nginx (or any other HTTP/1.0+
client) is on the same machine as unicorn.

A huge thanks to Tom Burns for implementing and testing this
change in production with real traffic (including mitigating
an unexpected DoS attack).

ref: http://mid.gmane.org/CAK4qKG3rkfVYLyeqEqQyuNEh_nZ8yw0X_cwTxJfJ+TOU+y8F+w@mail.gmail.com

This release fixes broken Rainbows! compatibility in 4.5.0pre1.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: Add licenses to gemspec
       [not found]           ` <20121204113041.GA17535@dcvr.yhbt.net>
@ 2012-12-04 11:38  6%         ` Peter Marsh
  0 siblings, 0 replies; 200+ results
From: Peter Marsh @ 2012-12-04 11:38 UTC (permalink / raw)
  To: Eric Wong; +Cc: mongrel-unicorn

I made a bit of a mess of emailing in the patch, I'm sorry. I'll
submit an inline patch to mongrel-unicorn@rubyforge.org next time.

Cheers,

Pete

On 4 December 2012 11:30, Eric Wong <normalperson@yhbt.net> wrote:
> Peter Marsh <pete@modeltwozero.com> wrote:
>> Hi Eric,
>>
>> The changes look great, thanks for letting me know about the Ruby
>> license, I wasn't aware of that.
>
> Thanks for the patch and followup.  Pushed to unicorn.git as
> commit bc4c412f15a05a37ec40f374239efa83d2dbdb1e
>
> Btw, did you email mongrel-unicorn-owner@rubyforge.org since
> the application/octet-stream attachment got rejected from
> the normal mongrel-unicorn@rubyforge.org list?
>
> I strongly prefer inline patches (which I can pipe directly to "git am"
> w/o leaving my mailer), however setting the attachment MIME type to
> text/plain or text/x-diff should get through.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 4.5.0pre1 - check_client_connection option
@ 2012-11-30  0:07  5% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-11-30  0:07 UTC (permalink / raw)
  To: mongrel-unicorn

Just pushed out the RubyGem for this.

Changes:

The new check_client_connection option allows unicorn to detect
most disconnected clients before potentially expensive
application processing begins.

This feature is useful for applications experiencing spikes of
traffic leading to undesirable queue times, as clients will
disconnect (and perhaps even retry, compounding the problem)
before unicorn can even start processing the request.

To enable this feature, add the following line to a unicorn
config file:

  check_client_connection true

A huge thanks to Tom Burns for implementing and testing this
change in production with real traffic (including mitigating
an unexpected DoS attack).

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Fwd: Maintaining capacity during deploys
       [not found]     <CAHOTMV++otgxdru_oZLXuVuqHF7F4uMwd04O0QZBjxeqFR-=XQ@mail.gmail.com>
@ 2012-11-29 23:05  5% ` Tony Arcieri
  0 siblings, 0 replies; 200+ results
From: Tony Arcieri @ 2012-11-29 23:05 UTC (permalink / raw)
  To: unicorn list

We're using unicornctl restart with the default before/after hook
behavior, which is to reap old Unicorn workers via SIGQUIT after the
new one has finished booting.

Unfortunately, while the new workers are forking and begin processing
requests, we're still seeing significant spikes in our haproxy request
queue. It seems as if after we restart, the unwarmed workers get
swamped by the incoming requests. As far as I can tell, the momentary
loss of capacity we experience translates fairly quickly into a
thundering herd.

We've experimented with rolling restarts at the server level but these
do not resolve the problem.

I'm curious if we could do a more granular application-level rolling
restart, perhaps using TTOU instead of QUIT to progressively dial down
the old workers one-at-a-time, and forking new ones to replace them
incrementally. Anyone tried anything like that before?

Or are there any other suggestions? (short of "add more capacity")

--
Tony Arcieri<div class="gmail_extra"><br><br><div
class="gmail_quote">On Thu, Nov 29, 2012 at 2:50 PM, Tony Arcieri
<span dir="ltr">&lt;<a href="mailto:tony.arcieri@gmail.com"
target="_blank">tony.arcieri@gmail.com</a>&gt;</span>
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex;">We're using
unicornctl restart with the default before/after hook behavior, which
is to reap old Unicorn workers via SIGQUIT after the new one has
finished booting.<div><br></div><div>Unfortunately, while the new
workers are forking and begin processing requests, we're still seeing
significant spikes in our haproxy request queue. It seems as if after
we restart, the unwarmed workers get swamped by the incoming
requests.&nbsp;As far as I can tell, the momentary loss of capacity we
experience translates fairly quickly into a thundering herd.</div>
<div><div><br></div><div>We've experimented with rolling restarts at
the server level but these do not resolve the
problem.</div><div><br></div><div>I'm curious if we could do a more
granular application-level rolling restart, perhaps using TTOU instead
of QUIT to progressively dial down the old workers one-at-a-time, and
forking new ones to replace them incrementally. Anyone tried anything
like that before?</div>
<div><br></div><div>Or are there any other suggestions? (short of "add
more capacity")</div><span class="HOEnZb"><font
color="#888888"><div><br></div>-- <br>Tony Arcieri<br><br>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Tony
Arcieri<br><br>
</div>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Re: Combating nginx 499 HTTP responses during flash traffic scenario
  2012-11-03 22:45  2%             ` Tom Burns
@ 2012-11-05 11:48  0%               ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-11-05 11:48 UTC (permalink / raw)
  To: unicorn list

Tom Burns <tom.burns@jadedpixel.com> wrote:
> On Fri, Nov 2, 2012 at 3:38 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Everything looks good for stock configurations.  The ERROR_XXX_RESPONSE
> > constants should have "HTTP/1.1 " removed if used with this option, too.
> > That requires keeping track of whether or not we've written out
> > "HTTP/1.1 ", yet.
> 
> The patch below tracks if we've sent the start of the response.  It
> also passes that to http_response_write as a parameter with default
> value false.  By doing this, the test suite passes if you change the
> default value for check_client_connection to true, which makes me feel
> a lot better about the patch.

Cool.

> > > The only thing it's missing from your TODO is enforcing
> > > tcp_nodelay/tcp_nopush as I wasn't sure where was the best place to do
> > > the enforcement.
> >
> > It should probably be done inside Unicorn::Configurator#commit!
> 
> So, I still didn't so this, because in commit! we don't know if it's a
> TCP or UNIX socket, and I don't think it'd be nice to force tcp
> configs if the user's using a UNIX socket.  If you disagree I can add
> the check there, it just seemed weird when I looked at it.

Erm, just check if :tcp_* is set in the :listener_opts hash and raise if
there's incompatible settings.  We shouldn't be changing settings behind
users' backs.

> >From d7e249202b0fd5c24f2b98c700e02bf992c6576b Mon Sep 17 00:00:00 2001
> From: Tom Burns <tom.burns@jadedpixel.com>
> Date: Tue, 30 Oct 2012 16:22:21 -0400
> Subject: [PATCH] Begin writing HTTP request headers early to detect
>  disconnected clients

This definitely needs a thorough commit message which explains why this
is a necessary addition, how it works, when it works, and what benefits
it brings.

We can wait until you can report on how this change improves your
situation before fleshing this out.  Real-world results are always
good to have :>

> --- a/lib/unicorn/http_request.rb
> +++ b/lib/unicorn/http_request.rb
> @@ -70,6 +82,15 @@ class Unicorn::HttpParser
>        # an Exception thrown from the parser will throw us out of the loop
>        false until add_parse(socket.kgio_read!(16384))
>      end
> +
> +    # detect if the socket is valid by writing a partial response:
> +    if @@check_client_connection && headers?
> +      @response_start_sent = true
> +      Unicorn::Const::HTTP_RESPONSE_START.each { |c| socket.write(c) }
> +    else
> +      @response_start_sent = false

This else portion is redundant given it's set to false when you reset
the parser (see my final comment below)

> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -524,23 +533,33 @@ class Unicorn::HttpServer
>        Unicorn.log_error(@logger, "app error", e)
>        Unicorn::Const::ERROR_500_RESPONSE
>      end
> +    msg = 'HTTP/1.1 ' + msg unless @request.response_start_sent

String interpolation would be easier to read, here (and slightly
smaller bytecode if it matters :>).

>      client.kgio_trywrite(msg)
>      client.close
>      rescue
>    end
> 
> +  def expect_100_response
> +    if @request.response_start_sent
> +      Unicorn::Const::EXPECT_100_RESPONSE_SUFFIXED
> +    else
> +      Unicorn::Const::EXPECT_100_RESPONSE
> +    end
> +  end
> +
>    # once a client is accepted, it is processed in its entirety here
>    # in 3 easy steps: read request, call app, write app response
>    def process_client(client)
> +    @request.response_start_sent = false

Better to reset this ivar in the HttpParser#reset method
(it's in the C extension, I can change it later).

Random thought:  HttpResponse generation gains even more coupling
with the current Http{Request,Parser} state.  Organizing code is hard :x
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Combating nginx 499 HTTP responses during flash traffic scenario
  @ 2012-11-03 22:45  2%             ` Tom Burns
  2012-11-05 11:48  0%               ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Tom Burns @ 2012-11-03 22:45 UTC (permalink / raw)
  To: unicorn list

On Fri, Nov 2, 2012 at 3:38 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Everything looks good for stock configurations.  The ERROR_XXX_RESPONSE
> constants should have "HTTP/1.1 " removed if used with this option, too.
> That requires keeping track of whether or not we've written out
> "HTTP/1.1 ", yet.

The patch below tracks if we've sent the start of the response.  It
also passes that to http_response_write as a parameter with default
value false.  By doing this, the test suite passes if you change the
default value for check_client_connection to true, which makes me feel
a lot better about the patch.

> > The only thing it's missing from your TODO is enforcing
> > tcp_nodelay/tcp_nopush as I wasn't sure where was the best place to do
> > the enforcement.
>
> It should probably be done inside Unicorn::Configurator#commit!

So, I still didn't so this, because in commit! we don't know if it's a
TCP or UNIX socket, and I don't think it'd be nice to force tcp
configs if the user's using a UNIX socket.  If you disagree I can add
the check there, it just seemed weird when I looked at it.

>
> You'll probably find it more reliable and easier to use "git send-email".

Sadly with 2-factor authentication I think this is a no go.

Let me know what you think!

Cheers,
Tom

>From d7e249202b0fd5c24f2b98c700e02bf992c6576b Mon Sep 17 00:00:00 2001
From: Tom Burns <tom.burns@jadedpixel.com>
Date: Tue, 30 Oct 2012 16:22:21 -0400
Subject: [PATCH] Begin writing HTTP request headers early to detect
 disconnected clients

---
 examples/unicorn.conf.rb       |    6 ++++++
 lib/unicorn/configurator.rb    |    7 +++++++
 lib/unicorn/const.rb           |   12 ++++++++----
 lib/unicorn/http_request.rb    |   21 +++++++++++++++++++++
 lib/unicorn/http_response.rb   |    5 +++--
 lib/unicorn/http_server.rb     |   23 +++++++++++++++++++++--
 test/exec/test_exec.rb         |    7 ++++++-
 test/unit/test_configurator.rb |   12 ++++++++++++
 8 files changed, 84 insertions(+), 9 deletions(-)

diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index 0238043..1f4c9c0 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -46,6 +46,12 @@ preload_app true
 GC.respond_to?(:copy_on_write_friendly=) and
   GC.copy_on_write_friendly = true

+# Enable this flag to have unicorn test client connections by writing the
+# beginning of the HTTP headers before calling the application.  This
+# prevents calling the application for connections that have disconnected
+# while queued.
+check_client_connection false
+
 before_fork do |server, worker|
   # the following is highly recomended for Rails + "preload_app true"
   # as there's no need for the master process to hold a connection
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 89cbf5c..ca84a88 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -45,6 +45,7 @@ class Unicorn::Configurator
       },
     :pid => nil,
     :preload_app => false,
+    :check_client_connection => false,
     :rewindable_input => true, # for Rack 2.x: (Rack::VERSION[0] <= 1),
     :client_body_buffer_size => Unicorn::Const::MAX_BODY,
     :trust_x_forwarded => true,
@@ -454,6 +455,12 @@ class Unicorn::Configurator
     set_int(:client_body_buffer_size, bytes, 0)
   end

+  # When enabled, unicorn will check the client connection by writing
+  # the beginning of the HTTP headers before calling the application.
+  def check_client_connection(bool)
+    set_bool(:check_client_connection, bool)
+  end
+
   # Allow redirecting $stderr to a given path.  Unlike doing this from
   # the shell, this allows the unicorn process to know the path its
   # writing to and rotate the file if it is used for logging.  The
diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index b3d8d71..f0c4c12 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -29,12 +29,16 @@ module Unicorn::Const

   # :stopdoc:
   # common errors we'll send back
-  ERROR_400_RESPONSE = "HTTP/1.1 400 Bad Request\r\n\r\n"
-  ERROR_414_RESPONSE = "HTTP/1.1 414 Request-URI Too Long\r\n\r\n"
-  ERROR_413_RESPONSE = "HTTP/1.1 413 Request Entity Too Large\r\n\r\n"
-  ERROR_500_RESPONSE = "HTTP/1.1 500 Internal Server Error\r\n\r\n"
+  ERROR_400_RESPONSE = "400 Bad Request\r\n\r\n"
+  ERROR_414_RESPONSE = "414 Request-URI Too Long\r\n\r\n"
+  ERROR_413_RESPONSE = "413 Request Entity Too Large\r\n\r\n"
+  ERROR_500_RESPONSE = "500 Internal Server Error\r\n\r\n"
+
   EXPECT_100_RESPONSE = "HTTP/1.1 100 Continue\r\n\r\n"
+  EXPECT_100_RESPONSE_SUFFIXED = "100 Continue\r\n\r\nHTTP/1.1 "

+  HTTP_RESPONSE_START = ['HTTP', '/1.1 ']
   HTTP_EXPECT = "HTTP_EXPECT"
+
   # :startdoc:
 end
diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index a0435d6..9aceb0e 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -22,11 +22,14 @@ class Unicorn::HttpParser

   NULL_IO = StringIO.new("")

+  attr_accessor :response_start_sent
+
   # :stopdoc:
   # A frozen format for this is about 15% faster
   REMOTE_ADDR = 'REMOTE_ADDR'.freeze
   RACK_INPUT = 'rack.input'.freeze
   @@input_class = Unicorn::TeeInput
+  @@check_client_connection = false

   def self.input_class
     @@input_class
@@ -35,6 +38,15 @@ class Unicorn::HttpParser
   def self.input_class=(klass)
     @@input_class = klass
   end
+
+  def self.check_client_connection
+    @@check_client_connection
+  end
+
+  def self.check_client_connection=(bool)
+    @@check_client_connection = bool
+  end
+
   # :startdoc:

   # Does the majority of the IO processing.  It has been written in
@@ -70,6 +82,15 @@ class Unicorn::HttpParser
       # an Exception thrown from the parser will throw us out of the loop
       false until add_parse(socket.kgio_read!(16384))
     end
+
+    # detect if the socket is valid by writing a partial response:
+    if @@check_client_connection && headers?
+      @response_start_sent = true
+      Unicorn::Const::HTTP_RESPONSE_START.each { |c| socket.write(c) }
+    else
+      @response_start_sent = false
+    end
+
     e[RACK_INPUT] = 0 == content_length ?
                     NULL_IO : @@input_class.new(socket, self)
     e.merge!(DEFAULTS)
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index b781e20..9c2bacc 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -18,11 +18,12 @@ module Unicorn::HttpResponse
   CRLF = "\r\n"

   # writes the rack_response to socket as an HTTP response
-  def http_response_write(socket, status, headers, body)
+  def http_response_write(socket, status, headers, body,
response_start_sent=false)
     status = CODES[status.to_i] || status

+    http_response_start = response_start_sent ? '' : 'HTTP/1.1 '
     if headers
-      buf = "HTTP/1.1 #{status}\r\n" \
+      buf = "#{http_response_start}#{status}\r\n" \
             "Date: #{httpdate}\r\n" \
             "Status: #{status}\r\n" \
             "Connection: close\r\n"
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 13df55a..abe27db 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -17,6 +17,7 @@ class Unicorn::HttpServer
                 :listener_opts, :preload_app,
                 :reexec_pid, :orig_app, :init_listeners,
                 :master_pid, :config, :ready_pipe, :user
+
   attr_reader :pid, :logger
   include Unicorn::SocketHelper
   include Unicorn::HttpResponse
@@ -355,6 +356,14 @@ class Unicorn::HttpServer
     Unicorn::HttpParser.trust_x_forwarded = bool
   end

+  def check_client_connection
+    Unicorn::HttpRequest.check_client_connection
+  end
+
+  def check_client_connection=(bool)
+    Unicorn::HttpRequest.check_client_connection = bool
+  end
+
   private

   # wait for a signal hander to wake us up and then consume the pipe
@@ -524,23 +533,33 @@ class Unicorn::HttpServer
       Unicorn.log_error(@logger, "app error", e)
       Unicorn::Const::ERROR_500_RESPONSE
     end
+    msg = 'HTTP/1.1 ' + msg unless @request.response_start_sent
     client.kgio_trywrite(msg)
     client.close
     rescue
   end

+  def expect_100_response
+    if @request.response_start_sent
+      Unicorn::Const::EXPECT_100_RESPONSE_SUFFIXED
+    else
+      Unicorn::Const::EXPECT_100_RESPONSE
+    end
+  end
+
   # once a client is accepted, it is processed in its entirety here
   # in 3 easy steps: read request, call app, write app response
   def process_client(client)
+    @request.response_start_sent = false
     status, headers, body = @app.call(env = @request.read(client))

     if 100 == status.to_i
-      client.write(Unicorn::Const::EXPECT_100_RESPONSE)
+      client.write(expect_100_response)
       env.delete(Unicorn::Const::HTTP_EXPECT)
       status, headers, body = @app.call(env)
     end
     @request.headers? or headers = nil
-    http_response_write(client, status, headers, body)
+    http_response_write(client, status, headers, body,
@request.response_start_sent)
     client.shutdown # in case of fork() in Rack app
     client.close # flush and uncork socket immediately, no keepalive
   rescue => e
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index b30a3d6..1cee2b7 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -871,13 +871,14 @@ EOF
     wait_for_death(pid)
   end

-  def hup_test_common(preload)
+  def hup_test_common(preload, check_client=false)
     File.open("config.ru", "wb") { |fp| fp.syswrite(HI.gsub("HI", '#$$')) }
     pid_file = Tempfile.new('pid')
     ucfg = Tempfile.new('unicorn_test_config')
     ucfg.syswrite("listen '#@addr:#@port'\n")
     ucfg.syswrite("pid '#{pid_file.path}'\n")
     ucfg.syswrite("preload_app true\n") if preload
+    ucfg.syswrite("check_client_connection true\n") if check_client
     ucfg.syswrite("stderr_path 'test_stderr.#$$.log'\n")
     ucfg.syswrite("stdout_path 'test_stdout.#$$.log'\n")
     pid = xfork {
@@ -942,6 +943,10 @@ EOF
     hup_test_common(false)
   end

+  def test_check_client_hup
+    hup_test_common(false, true)
+  end
+
   def test_default_listen_hup_holds_listener
     default_listen_lock do
       res, pid_path = default_listen_setup
diff --git a/test/unit/test_configurator.rb b/test/unit/test_configurator.rb
index c19c427..fc4170e 100644
--- a/test/unit/test_configurator.rb
+++ b/test/unit/test_configurator.rb
@@ -139,6 +139,18 @@ class TestConfigurator < Test::Unit::TestCase
     end
   end

+  def test_check_client_connection
+    tmp = Tempfile.new('unicorn_config')
+    test_struct = TestStruct.new
+    tmp.syswrite("check_client_connection true\n")
+
+    assert_nothing_raised do
+      Unicorn::Configurator.new(:config_file => tmp.path).commit!(test_struct)
+    end
+
+    assert test_struct.check_client_connection
+  end
+
   def test_after_fork_proc
     test_struct = TestStruct.new
     [ proc { |a,b| }, Proc.new { |a,b| }, lambda { |a,b| } ].each do |my_proc|
-- 
1.7.7.5 (Apple Git-26)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 2%]

* Re: Combating nginx 499 HTTP responses during flash traffic scenario
  @ 2012-10-29 21:06  4%       ` Hongli Lai
  0 siblings, 0 replies; 200+ results
From: Hongli Lai @ 2012-10-29 21:06 UTC (permalink / raw)
  To: unicorn list

On Mon, Oct 29, 2012 at 8:41 PM, Eric Wong <normalperson@yhbt.net> wrote:
> I'm not sure, Tom can investigate further.
>
> I do recall haproxy having more intelligent load distribution than
> nginx, so it can send requests to less-busy machines, at least.

It is correct that HAProxy can do more intelligent load distribution.
The "maxconn 1" feature in HAProxy is a bit like global queuing in
Phusion Passenger, or the kernel socket queuing that Unicorn relies
on. Phusion Passenger sends the request to the first worker that
becomes available, and the kernel passes the connection to the first
Unicorn instance that accept()s the socket. Nginx's proxy_module
passes requests immediately and only does round robin.

So to solve this problem we need to add 2 features to the layer that
manages the queue:
1. It must be able to detect an early disconnect.
2. It must be able to remove a client from the queue.

The kernel obviously can't do this, but I'm very curious as to whether
HAProxy supports these two things. I also think it shouldn't be too
hard to implement this in Phusion Passenger 4's new architecture.

-- 
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: Detecting unicorn / defining after_fork after master startup
  2012-07-21 19:42  0%   ` Matt Sanders
@ 2012-07-22 21:20  0%     ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-07-22 21:20 UTC (permalink / raw)
  To: unicorn list

Matt Sanders <matt@modal.org> wrote:
> On Thu, Jul 19, 2012 at 6:13 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Matt Sanders <matt@modal.org> wrote:
> >> 2. Is there an established way for adding an after_fork hook after
> >> Unicorn has already started up?
> >
> > Nothing that I know of.
> >
> >> I'm aware of the ability to do this via the config file but I don't
> >> want my users to have to add something to their unicorn config file.
> >> Is there an equivalent to passenger's
> >> PhusionPassenger.on_event(:starting_worker_process) method?
> >
> > Lately, I've been favoring the following pattern instead:

(top-posting corrected)

> Thanks! I definitely agree in general with that pattern. In this case
> I'd ideally like some activity on the forked child before the next
> request cycle - that is, if the forked child doesn't serve any
> requests for a little bit, I'd still like my behavior to be
> initialized.

Yes, that sucks.  I'll often just throw some requests at the child
just to start something up if I need it, or initiate a separate daemon.

> Do you know of a way to get this behavior without a direct after fork hook?

No, unfortunately not.  I have considered petitioning for something
along the lines of pthread_atfork() in Ruby, but pthread_atfork() itself
also has unfortunate drawbacks (it makes fork() async-signal un-safe,
and thus _Fork() will be introduced in the next version of POSIX).
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Detecting unicorn / defining after_fork after master startup
  2012-07-19 23:13  0% ` Eric Wong
@ 2012-07-21 19:42  0%   ` Matt Sanders
  2012-07-22 21:20  0%     ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Matt Sanders @ 2012-07-21 19:42 UTC (permalink / raw)
  To: unicorn list

Eric,

Thanks! I definitely agree in general with that pattern. In this case
I'd ideally like some activity on the forked child before the next
request cycle - that is, if the forked child doesn't serve any
requests for a little bit, I'd still like my behavior to be
initialized.

Do you know of a way to get this behavior without a direct after fork hook?

Thanks for your help!
-Matt

On Thu, Jul 19, 2012 at 6:13 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Matt Sanders <matt@modal.org> wrote:
>> 1. What is the best way to determine whether the app is indeed running
>> inside a unicorn server?
>
> Unicorn.listener_names will return a non-empty array of listener
> addresses.
>
>> 2. Is there an established way for adding an after_fork hook after
>> Unicorn has already started up?
>
> Nothing that I know of.
>
>> I'm aware of the ability to do this via the config file but I don't
>> want my users to have to add something to their unicorn config file.
>> Is there an equivalent to passenger's
>> PhusionPassenger.on_event(:starting_worker_process) method?
>
> Lately, I've been favoring the following pattern instead:
>
>   def initialize
>     @init_pid = $$
>     ...
>   end
>
>   def initialize_child_fork
>     @init_pid = $$
>     ...
>   end
>
>   def call(env)
>     initialize_child_fork if @init_pid != $$
>     ...
>   end
>
> The overhead is negligible and works regardless of server (even in
> non-HTTP servers somebody may write).  I really don't like code
> that would need to special case for all sorts of servers:
>
>   foo_for_passenger if defined?(PhusionPassenger)
>   foo_for_unicorn if defined?(Unicorn)
>   foo_for_something_fastcgi if defined?(SomethingFastCGI)
>   foo_for_something_scgi if defined?(SomethingSCGI)
>   foo_for_something_else_that_forks if defined?(YetAnotherForkingRackServer)
>   foo_for_something_else if defined?(SomethingElse)
>   ...
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Detecting unicorn / defining after_fork after master startup
  2012-07-19 22:43  4% Detecting unicorn / defining after_fork after master startup Matt Sanders
@ 2012-07-19 23:13  0% ` Eric Wong
  2012-07-21 19:42  0%   ` Matt Sanders
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2012-07-19 23:13 UTC (permalink / raw)
  To: unicorn list

Matt Sanders <matt@modal.org> wrote:
> 1. What is the best way to determine whether the app is indeed running
> inside a unicorn server?

Unicorn.listener_names will return a non-empty array of listener
addresses.

> 2. Is there an established way for adding an after_fork hook after
> Unicorn has already started up?

Nothing that I know of.

> I'm aware of the ability to do this via the config file but I don't
> want my users to have to add something to their unicorn config file.
> Is there an equivalent to passenger's
> PhusionPassenger.on_event(:starting_worker_process) method?

Lately, I've been favoring the following pattern instead:

  def initialize
    @init_pid = $$
    ...
  end

  def initialize_child_fork
    @init_pid = $$
    ...
  end

  def call(env)
    initialize_child_fork if @init_pid != $$
    ...
  end

The overhead is negligible and works regardless of server (even in
non-HTTP servers somebody may write).  I really don't like code
that would need to special case for all sorts of servers:

  foo_for_passenger if defined?(PhusionPassenger)
  foo_for_unicorn if defined?(Unicorn)
  foo_for_something_fastcgi if defined?(SomethingFastCGI)
  foo_for_something_scgi if defined?(SomethingSCGI)
  foo_for_something_else_that_forks if defined?(YetAnotherForkingRackServer)
  foo_for_something_else if defined?(SomethingElse)
  ...
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Detecting unicorn / defining after_fork after master startup
@ 2012-07-19 22:43  4% Matt Sanders
  2012-07-19 23:13  0% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Matt Sanders @ 2012-07-19 22:43 UTC (permalink / raw)
  To: mongrel-unicorn

Hey everyone,

Working on an engine for rails that needs specialized behavior with
forking and I had a couple questions.

1. What is the best way to determine whether the app is indeed running
inside a unicorn server?

Most of the attempts I can find to detect check to see if the main
modules for Unicorn are defined, but this really only checks to see
that Unicorn is present, not that you are using it. I know a lot of
shops may use Unicorn in production but something else for local
development. Is there something equivalent to the IN_PHUSION_PASSENGER
constant that passenger defines?

2. Is there an established way for adding an after_fork hook after
Unicorn has already started up?

I'm aware of the ability to do this via the config file but I don't
want my users to have to add something to their unicorn config file.
Is there an equivalent to passenger's
PhusionPassenger.on_event(:starting_worker_process) method?

Thanks!
-Matt
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: Address already in use
  @ 2012-06-25 23:57  5%     ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-06-25 23:57 UTC (permalink / raw)
  To: unicorn list

Manuel Palenciano Guerrero <mpalenciano@gmail.com> wrote:
> Hi,
> 
> First, thanks Eric, Jérémy and Aaron for replying. I really appreciate it.
> 
> Yes Eric, I can see the line... "inherited addr=/tmp/unicorn.app.sock fd=..."
> 
> here is the full log
> 
> -------------------------------------------------
> I, [2012-06-21T11:40:44.282224 #29212]  INFO -- : inherited addr=/tmp/unicorn.sublimma_staging.sock fd=3
> I, [2012-06-21T11:40:44.282480 #29212]  INFO -- : Refreshing Gem list
> master process ready
> worker=0 ready
> worker=1 ready
> reaped #<Process::Status: pid=28683,exited(0)> worker=0
> reaped #<Process::Status: pid=28684,exited(0)> worker=1
> master complete

Ugh, lack of formatting caused by Rails mucking with Logger is annoying.
Can you add the following to your unicorn config?

  Configurator::DEFAULTS[:logger].formatter = Logger::Formatter.new

(that's from unicorn.bogomips.org/FAQ.html)

> E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener failed addr=/tmp/unicorn.sublimma_staging.sock (in use)

I'm curious if PID=29401 is a worker of pid 29212.  Or you're starting
another master somewhere...

> /var/www/sublimma/staging/shared/bundle/ruby/1.8/gems/unicorn-4.3.0/lib/unicorn/socket_helper.rb:140:in `initialize': Address already in use - /tmp/unicorn.sublimma_staging.sock (Errno::EADDRINUSE)
> -------------------------------------------------
> 
> my unicorn.rb: https://gist.github.com/2991110

Did you share the correct config?  Your config has:

  listen "/tmp/unicorn.app_production.sock"

But your config has: /tmp/unicorn.sublimma_staging.sock
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Sinatra app fails to perform at scale
@ 2012-06-15 18:09  6% Eliot Shepard
  0 siblings, 0 replies; 200+ results
From: Eliot Shepard @ 2012-06-15 18:09 UTC (permalink / raw)
  To: mongrel-unicorn

Hello,

We're attempting to move our set of Rails and Sinatra apps from
Passenger/REE to Unicorn/1.9.3. We've started with one of the Sinatra
apps, which typically sees about 50 req/s. It happens to talk to both
Redis (Resque) and MongoDB GridFS.

The basic setup works fine when tested under ab, but we're having
trouble getting the deploy into production. It performs fine for a
bit, then the nginx write queue fills up and begins returning 502s. A
colleague has posted a more detailed description of the issue and our
setup on ServerFault:
http://serverfault.com/questions/398972/need-to-increase-nginx-throughput-to-an-upstream-unix-socket-linux-kernel-tun

Additional information on the environment
curbed@app1:~$ uname -a
Linux app1 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
curbed@app1:~$ ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
curbed@app1:~$ unicorn -v
unicorn v4.3.1
curbed@app1:~$ nginx -V
nginx version: nginx/1.2.1
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_ssl_module
--with-cc-opt=-Wno-error --with-http_gzip_static_module
--with-http_stub_status_module
--add-module=/home/curbed/src/nginx-modules/nginx-gridfs
--add-module=/home/curbed/src/nginx-modules/ngx_http_redis-0.3.6
--add-module=/home/curbed/src/nginx-modules/headers-more-nginx-module

Kernel tweaks:
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.route.flush = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.somaxconn = 8192
net.netfilter.nf_conntrack_max = 131072

Any suggestions on configuration, kernel tuning, etc. would be
welcomed (here or on SF). Please CC me if you answer through the list.
Thanks for your time.

Eliot

-- 
Eliot Shepard
Head of Tech, Curbed Network
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Triggering OobGC when heap is nearly full
  @ 2012-06-04  4:47  5% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-06-04  4:47 UTC (permalink / raw)
  To: unicorn list

Tony <bascule@gmail.com> wrote:
> We've started using OobGC at my workplace and it's definitely helping,
> however the amount of garbage various requests in our app can generate
> is quite the chunky stew.

Cool! (though I can't say I've ever /liked/ OobGC :x)

> It seems like OobGC configuration is all predicated around a number of
> requests to process before OobGCing. However, REE exposes heap/GC
> stats that could be used to make that decision intelligently at
> runtime.
> 
> Is there any way presently to use some heuristics around the current
> state of the heap to decide when to OobGC, or barring that, a way to
> pass a proc I would write into OobGC that can answer the question
> "should I OobGC?" with true/false rather than relying on a certain
> number of requests?

Not right now, but OobGC is only ~20 lines of code or so it
should be easy to figure out how to add/change.

Btw, I'm still really curious to know how the lazy-sweep GC in 1.9.3
behaves with OobGC, I think 1.9.3+ should make OobGC obsolete.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Re: [Suggestion] Add upstream param to listen statement
  2012-05-23 13:29  6% [Suggestion] Add upstream param to listen statement Manuel Valente
@ 2012-05-23 18:50  5% ` Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-05-23 18:50 UTC (permalink / raw)
  To: unicorn list; +Cc: Manuel Valente

Manuel Valente <manuel.valente@gmail.com> wrote:
> Hi,
> 
> It would be nice to be able to reject all incoming requests to
> unicorn if they do not originate from our upstream http server.
> 
> An additional parameter to the listen statement is perhaps the best
> way to achieve this :
> 
> server.listen(addr, :tries => -1, :delay => 5, :backlog => 128,
> :upstream => '10.0.0.1')
> 
> This param could be a string or an array of IP addresses.

iptables (or whatever firewall module that comes with your OS)
is far more efficient than anything in userspace for rejecting
IPs entirely.

You can also do this in middleware by checking env["REMOTE_ADDR"].
I'll sometimes do something like this to reject certain HTTP
methods (/POST/PUT/DELETE), but let GET/HEAD requests through:

# totally untested code for Rack middleware:
class Rejector
  def new(app, bad_ips)
    @bad_ips = bad_ips
    @app = app
  end

  def call(env)
    case env["REQUEST_METHOD"]
    when "POST", "DELETE", "PUT"
      return [ 403, {}, [] ] if @bad_ips.include?(env["REMOTE_ADDR"])
    end
    @app.call(env)
  end
end

---------- config.ru --------------
require 'set'
require 'rejector'
use Rejector, Set.new("10.0.0.1")
run YourApp.new


Instead of Set, you can also check out rpatricia for netmasks:
http://www.goto.info.waseda.ac.jp/~tatsuya/rpatricia/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* [Suggestion] Add upstream param to listen statement
@ 2012-05-23 13:29  6% Manuel Valente
  2012-05-23 18:50  5% ` Eric Wong
  0 siblings, 1 reply; 200+ results
From: Manuel Valente @ 2012-05-23 13:29 UTC (permalink / raw)
  To: mongrel-unicorn

Hi,

It would be nice to be able to reject all incoming requests to unicorn 
if they do not originate from our upstream http server.

An additional parameter to the listen statement is perhaps the best way 
to achieve this :

server.listen(addr, :tries => -1, :delay => 5, :backlog => 128, 
:upstream => '10.0.0.1')

This param could be a string or an array of IP addresses.

I read the documentation but did not find any way to achieve this.

Best regards,

-- 
Manuel Valente
prixing.fr

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Socket errors and nginx 502 gateway errors
  2012-04-29 19:18  4%     ` Eric Wong
@ 2012-05-02 16:30  0%       ` Pico Aeterna
  0 siblings, 0 replies; 200+ results
From: Pico Aeterna @ 2012-05-02 16:30 UTC (permalink / raw)
  To: unicorn list

Eric,

Thanks for your help.  Was able to isolate the problem to a geoip gem.
 I appreciate the tips and debugging help.

Thanks
Will

On Sun, Apr 29, 2012 at 12:18 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Pico Aeterna <flippedootninja@gmail.com> wrote:
>> reaped #<Process::Status: pid=9103,signaled(SIGIOT=6)> worker=9
>> /var/www/domain/releases/20120427011507/app/controllers/application_controller.rb:172:
>> [BUG] Segmentation fault
>
> OK, something (most likely your app) is causing Ruby to segfault.
> This is completely unrelated to nginx, so it'll be easier to hit
> unicorn directly to diagnose this.
>
> Since you're new to debugging, I'll try to walk you through dealing
> with core dumps:
>
> * Enable core dumps in the same shell process before starting unicorn
>  (ulimit -c unlimited && unicorn -c /path/to/config ...)
>
> * On Linux, /proc/sys/kernel/core_* sysctls control where your core dumps
>  are created.
>
> And when you get a core dump (a "core" or core.$PID file), you can run
> gdb on it:
>
>   gdb /path/to/ruby /path/to/core
>
> Make sure you're using the same ruby binary that unicorn is using.
> Ruby should be compiled with debugging symbols ('-g' or '-ggdb') by
> default, but if not, you'll probably want to recompile.
>
>> 7369  writev(31, [{"GET
>> /widget/looks.js?id=1172821&thumbs=3&source=my_looks&gender=both&align=center&r=1320622693&hash=36189f21135680efbe2d076d0b56bb06
>> HTTP/1.0\r\nX-Forwarded
>> -For: 83.9.20.243, 83.9.20.243\r\nHost: domain.nu\r\nConnection:
>> close\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64)
>> AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5
>> Safari/534.55.3\r\nAccept: */*\r\nReferer:
>> http://www.xxxxx.com/\r\nAccept-Language: pl-PL\r\nAccept-Encoding:
>> gzip, deflate\r\nCookie:
>> __utma=78941577.1641012066.1319789648.1325094074.1325949868.3;
>> __utmz=78941577.1325949868.3.3.utmcsr=xxxx.com|utmccn=(referral)|utmcmd=referral|utmcct=/l.php;
>> __gads=ID=269c5cea8f09aea0:T=1319789648:S=ALNI_MaqwdMaIFS5I3FeqLnv86oivdVDuA;
>> __qca=P0-1390452860-1319789648200\r\n\r\n", 745}], 1) = 7457369
>
>> Using the following I was able to get a 200 response
>>
>> req='GET / HTTP/1.1\r\nHost: domain.com\r\n\r\n'
>> printf "$req" | socat - UNIX:/var/www/domain/shared/sockets/unicorn.sock
>
> OK, I think your printf request needs to more closely match what nginx
> is sending to trigger the segfault.  I would add/remove
> headers/parameters you see above until things start segfaulting.
>
> With socat, you may also want to wait a bit for the response for
> slower endpoints:
>
>  (printf "$req"; sleep $SECONDS) | socat - ...
>
> Otherwise socat will disconnect immediately after its written to
> the socket without waiting for a response.
>
> (a better, but complicated way would be to setup a FIFO as the
>  integration tests in unicorn do).
>
> If you're familiar with Rack::Mock*, you may also want to isolate this
> to just a Rack application.  It's likely the segfault can be triggered
> for your app without unicorn, too.
>
>> Sorry but i'm new to debugging Unicorn.  Hopefully some of this is helpful.
>
> It is helpful.  Debugging unicorn isn't different than debugging
> anything else.  The most important is to understand the problem, and
> that's done by isolating variables to get a small, reproducible test
> case.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Socket errors and nginx 502 gateway errors
  @ 2012-04-29 19:18  4%     ` Eric Wong
  2012-05-02 16:30  0%       ` Pico Aeterna
  0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2012-04-29 19:18 UTC (permalink / raw)
  To: unicorn list

Pico Aeterna <flippedootninja@gmail.com> wrote:
> reaped #<Process::Status: pid=9103,signaled(SIGIOT=6)> worker=9
> /var/www/domain/releases/20120427011507/app/controllers/application_controller.rb:172:
> [BUG] Segmentation fault

OK, something (most likely your app) is causing Ruby to segfault.
This is completely unrelated to nginx, so it'll be easier to hit
unicorn directly to diagnose this.

Since you're new to debugging, I'll try to walk you through dealing
with core dumps:

* Enable core dumps in the same shell process before starting unicorn
  (ulimit -c unlimited && unicorn -c /path/to/config ...)

* On Linux, /proc/sys/kernel/core_* sysctls control where your core dumps
  are created.

And when you get a core dump (a "core" or core.$PID file), you can run
gdb on it:

   gdb /path/to/ruby /path/to/core

Make sure you're using the same ruby binary that unicorn is using.
Ruby should be compiled with debugging symbols ('-g' or '-ggdb') by
default, but if not, you'll probably want to recompile.

> 7369  writev(31, [{"GET
> /widget/looks.js?id=1172821&thumbs=3&source=my_looks&gender=both&align=center&r=1320622693&hash=36189f21135680efbe2d076d0b56bb06
> HTTP/1.0\r\nX-Forwarded
> -For: 83.9.20.243, 83.9.20.243\r\nHost: domain.nu\r\nConnection:
> close\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64)
> AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5
> Safari/534.55.3\r\nAccept: */*\r\nReferer:
> http://www.xxxxx.com/\r\nAccept-Language: pl-PL\r\nAccept-Encoding:
> gzip, deflate\r\nCookie:
> __utma=78941577.1641012066.1319789648.1325094074.1325949868.3;
> __utmz=78941577.1325949868.3.3.utmcsr=xxxx.com|utmccn=(referral)|utmcmd=referral|utmcct=/l.php;
> __gads=ID=269c5cea8f09aea0:T=1319789648:S=ALNI_MaqwdMaIFS5I3FeqLnv86oivdVDuA;
> __qca=P0-1390452860-1319789648200\r\n\r\n", 745}], 1) = 7457369

> Using the following I was able to get a 200 response
> 
> req='GET / HTTP/1.1\r\nHost: domain.com\r\n\r\n'
> printf "$req" | socat - UNIX:/var/www/domain/shared/sockets/unicorn.sock

OK, I think your printf request needs to more closely match what nginx
is sending to trigger the segfault.  I would add/remove
headers/parameters you see above until things start segfaulting.

With socat, you may also want to wait a bit for the response for
slower endpoints:

  (printf "$req"; sleep $SECONDS) | socat - ...

Otherwise socat will disconnect immediately after its written to
the socket without waiting for a response.

(a better, but complicated way would be to setup a FIFO as the
 integration tests in unicorn do).

If you're familiar with Rack::Mock*, you may also want to isolate this
to just a Rack application.  It's likely the segfault can be triggered
for your app without unicorn, too.

> Sorry but i'm new to debugging Unicorn.  Hopefully some of this is helpful.

It is helpful.  Debugging unicorn isn't different than debugging
anything else.  The most important is to understand the problem, and
that's done by isolating variables to get a small, reproducible test
case.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: murdering high-memory workers and auto-scaling
  2012-03-01 23:52  3% murdering high-memory workers and auto-scaling Ben Somers
  2012-03-02  0:12  0% ` Clifton King
@ 2012-03-02  1:07  0% ` Eric Wong
  1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2012-03-02  1:07 UTC (permalink / raw)
  To: unicorn list

Ben Somers <somers.ben@gmail.com> wrote:
> Two ideas, one more controversial than the other.

Neither is really controversal.

> First: auto-killing bloated workers. My current app has some memory
> leakage that wasn't really visible on our older passenger setup, since
> the auto-scaling meant that bloated workers got killed periodically.
> In a perfect world, we'd find and patch all of the leaks, but in the
> meantime (and as a safety net) I'd like to get the bloated workers
> auto-killed. It looks like it'd be simple to add in a bloated-worker
> check at the same point when we check for timeout violations, and it
> could be hidden behind a config setting. Alternately, I could write
> this in a separate script.
> 
> Pros: might be a useful built-in feature, looks easy to implement the killing
> Cons: Getting the memory usage might actually be surprisingly
> difficult. Comparing to passenger's memory management code, where they
> actually use platform-specific system calls, and we might get a
> sizeable quantity of code that we don't want dirtying up the unicorn
> internals. Also, some methods of checking appear to have performance
> risks.

You can try something like the following middleware (totally untested,
but I've done similar things here and there).  I don't know about
non-Linux, but I suspect /proc/#$$/* is likely to have something
similar...

class MemCheckLinux < Struct.new(:app)
  def call(env)
    # a faster, but less-readable version may use /proc/#$$/stat or
    # /proc/#$$/statm but those aren't as human-friendly as
    # /proc/#$$/status
    if /VmRSS:\s+(\d+)\s/ =~ File.read("/proc/#$$/status")
      # gracefully kill ourselves if we exceed ~100M
      Process.kill(:QUIT, $$) if $1.to_i > 100_000
    end
    app.call(env)
  end
end

use MemCheckLinux
run Rack::Lobster.new

Sadly, setrlimit(:RLIMIT_AS) only causes SIGSEGV to get raised,
and Ruby controls that signal for itself.  I sometimes use
setrlimit(:RLIMIT_CPU) + trap(:XCPU) to kill runaway processes.

Apache has long had a similar parameter where you could just
tell a worker to gracefully die after X number of requests.  That'd
also be trivial to implement with middlware using SIGQUIT.

> Second: in my use case, I have webservers running as VMs, sharing a
> physical box with backend utility servers. The util servers run lots
> of very CPU- and memory-hungry jobs, mostly at night; the webservers
> handle requests, mostly in the daytime. Currently, most of these
> webservers are running passenger, which is very polite about not using
> more resources than it needs to handle requests. Unicorn, by contrast
> (and by design) is very resource-greedy, what with the "scale to what
> you can theoretically handle" strategy. If I spin down my number of
> unicorn workers when they're not needed, I free up resources for my
> util servers, which is important. TTOU and TTIN signals give me a
> (very nice) means to write an auto-scaling module outside of unicorn,
> but it might be nice to have it included as an optional component. (I
> expect this will get voted down, as I expect the dev team is not
> interested in it).

If you can make an effort to support it when it breaks, I wouldn't mind
including a script in the examples/ section or as an optional module.

It's definitely not going to ever be the default.  Auto-scaling is hard
(if not impossible) to get right.  In my experience, it always get
things wrong by default or gets configured wrong, making things more
difficult to fix.

Dedicated servers will always be the primary target of unicorn.
(And unicorn of course only scales to server + backend resources,
 nginx handles scaling to client connections :)

> Happy to work on implementing these myself, just wanted to poll to see
> if it'd be worth developing them as part of unicorn proper rather than
> standalone scripts.

If you're willing to help support users of these scripts/modules,
I'd have no reservations about distributing them along with unicorn.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: murdering high-memory workers and auto-scaling
  2012-03-01 23:52  3% murdering high-memory workers and auto-scaling Ben Somers
@ 2012-03-02  0:12  0% ` Clifton King
  2012-03-02  1:07  0% ` Eric Wong
  1 sibling, 0 replies; 200+ results
From: Clifton King @ 2012-03-02  0:12 UTC (permalink / raw)
  To: unicorn list

We use the following at the bottom of a God config. I believe it's in an example
somewhere.

Bloated worker gets sent a QUIT and it will finish processing the request and
exits gracefully. We never really use TTOU and TTIN since the # of workers used
is basically determined by the ram in the machine in question.

```ruby
# unicorn workers
unicorn_worker_memory_limit = 220_000

Thread.new do
  loop do
    begin
      workers = `ps -e -www -o pid,rss,command | grep '[u]nicorn_rails worker'`
      workers.split("\n").each do |line|
        parts = line.split(' ')
        if parts[1].to_i > unicorn_worker_memory_limit
          # tell the worker to die after it finishes serving its request
          ::Process.kill('QUIT', parts[0].to_i)
        end
      end
    rescue Object
      # don't die ever once we've tested this
      nil
    end

    sleep 30
  end
end

```

Clifton

On Mar 1, 2012, at 5:52 PM, Ben Somers wrote:

> Two ideas, one more controversial than the other.
> First: auto-killing bloated workers. My current app has some memory
> leakage that wasn't really visible on our older passenger setup, since
> the auto-scaling meant that bloated workers got killed periodically.
> In a perfect world, we'd find and patch all of the leaks, but in the
> meantime (and as a safety net) I'd like to get the bloated workers
> auto-killed. It looks like it'd be simple to add in a bloated-worker
> check at the same point when we check for timeout violations, and it
> could be hidden behind a config setting. Alternately, I could write
> this in a separate script.
> 
> Pros: might be a useful built-in feature, looks easy to implement the killing
> Cons: Getting the memory usage might actually be surprisingly
> difficult. Comparing to passenger's memory management code, where they
> actually use platform-specific system calls, and we might get a
> sizeable quantity of code that we don't want dirtying up the unicorn
> internals. Also, some methods of checking appear to have performance
> risks.
> 
> Second: in my use case, I have webservers running as VMs, sharing a
> physical box with backend utility servers. The util servers run lots
> of very CPU- and memory-hungry jobs, mostly at night; the webservers
> handle requests, mostly in the daytime. Currently, most of these
> webservers are running passenger, which is very polite about not using
> more resources than it needs to handle requests. Unicorn, by contrast
> (and by design) is very resource-greedy, what with the "scale to what
> you can theoretically handle" strategy. If I spin down my number of
> unicorn workers when they're not needed, I free up resources for my
> util servers, which is important. TTOU and TTIN signals give me a
> (very nice) means to write an auto-scaling module outside of unicorn,
> but it might be nice to have it included as an optional component. (I
> expect this will get voted down, as I expect the dev team is not
> interested in it).
> 
> Happy to work on implementing these myself, just wanted to poll to see
> if it'd be worth developing them as part of unicorn proper rather than
> standalone scripts.
> 
> -ben
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* murdering high-memory workers and auto-scaling
@ 2012-03-01 23:52  3% Ben Somers
  2012-03-02  0:12  0% ` Clifton King
  2012-03-02  1:07  0% ` Eric Wong
  0 siblings, 2 replies; 200+ results
From: Ben Somers @ 2012-03-01 23:52 UTC (permalink / raw)
  To: mongrel-unicorn

Two ideas, one more controversial than the other.
First: auto-killing bloated workers. My current app has some memory
leakage that wasn't really visible on our older passenger setup, since
the auto-scaling meant that bloated workers got killed periodically.
In a perfect world, we'd find and patch all of the leaks, but in the
meantime (and as a safety net) I'd like to get the bloated workers
auto-killed. It looks like it'd be simple to add in a bloated-worker
check at the same point when we check for timeout violations, and it
could be hidden behind a config setting. Alternately, I could write
this in a separate script.

Pros: might be a useful built-in feature, looks easy to implement the killing
Cons: Getting the memory usage might actually be surprisingly
difficult. Comparing to passenger's memory management code, where they
actually use platform-specific system calls, and we might get a
sizeable quantity of code that we don't want dirtying up the unicorn
internals. Also, some methods of checking appear to have performance
risks.

Second: in my use case, I have webservers running as VMs, sharing a
physical box with backend utility servers. The util servers run lots
of very CPU- and memory-hungry jobs, mostly at night; the webservers
handle requests, mostly in the daytime. Currently, most of these
webservers are running passenger, which is very polite about not using
more resources than it needs to handle requests. Unicorn, by contrast
(and by design) is very resource-greedy, what with the "scale to what
you can theoretically handle" strategy. If I spin down my number of
unicorn workers when they're not needed, I free up resources for my
util servers, which is important. TTOU and TTIN signals give me a
(very nice) means to write an auto-scaling module outside of unicorn,
but it might be nice to have it included as an optional component. (I
expect this will get voted down, as I expect the dev team is not
interested in it).

Happy to work on implementing these myself, just wanted to poll to see
if it'd be worth developing them as part of unicorn proper rather than
standalone scripts.

-ben
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 3%]

* release unicorn v4.2.0 soon?
@ 2012-01-23  9:20  7% Eric Wong
  0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2012-01-23  9:20 UTC (permalink / raw)
  To: mongrel-unicorn

I just realized we haven't had a release in a while.  There's nothing
major that needs fixing, but I figure we should release 4.2.0 to
get some changes that have been brewing for a while.

The SSL support is still unaudited, but since Unicorn is only good on
LAN, it should be at least useful enough to detect network corruption.
TCP checksums are weak, so SSL can add an additional layer of
checksumming to the data stream if you're intent on end-to-end data
integrity.  (Rack apps can also support the Content-MD5 header/trailer
without SSL too)

Here are the changes since v4.1.1 in git://bogomips.org/unicorn.git

Brian P O'Rourke (1):
      Detect daemonization via configuration.

Eric Wong (18):
      add GPLv3 option to the license
      http_server: a few more things eligible for GC in worker
      http_server: update comment on tick == 0
      Links: add a link to the UnXF middleware
      add preliminary SSL support
      t: ensure SSL certificates exist on fresh test
      configurator: limit timeout to 30 days
      tests: just use the sha1sum implemented in Ruby
      tests: try to set a shorter path for Unix domain sockets
      bump dependencies
      socket_helper: remove out-of-date comment for TCP_NODELAY
      socket_helper: set SO_KEEPALIVE on TCP sockets
      socket_helper: fix grammerr fail
      quiet possible IOError from SIGUSR1 (reopen logs)
      cleanup exception handling on SIGUSR1
      http: test case for "Connection: TE"
      update tests for rack 1.4.0
      Rakefile: swap freshmeat.net URL for freecode.com

Jeremy Evans (2):
      t0011: fix test under OpenBSD
      test_helper: ensure test client connects to valid address

Yuichi Tateno (1):
      OobGC: force GC.start

In the future, feel free to ping me if you want an official release, I'm
forgetful sometimes and sometimes get completely immersed in other
projects.  "master" on unicorn.git is usually in a releasable state
anyways.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 7%]

Results 1-200 of ~330   | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2012-01-23  9:20  7% release unicorn v4.2.0 soon? Eric Wong
2012-03-01 23:52  3% murdering high-memory workers and auto-scaling Ben Somers
2012-03-02  0:12  0% ` Clifton King
2012-03-02  1:07  0% ` Eric Wong
2012-04-28 21:33     Socket errors and nginx 502 gateway errors Pico Aeterna
2012-04-28 21:59     ` Eric Wong
2012-04-29 16:34       ` Pico Aeterna
2012-04-29 19:18  4%     ` Eric Wong
2012-05-02 16:30  0%       ` Pico Aeterna
2012-05-23 13:29  6% [Suggestion] Add upstream param to listen statement Manuel Valente
2012-05-23 18:50  5% ` Eric Wong
2012-06-04  0:52     Triggering OobGC when heap is nearly full Tony
2012-06-04  4:47  5% ` Eric Wong
2012-06-15 18:09  6% Sinatra app fails to perform at scale Eliot Shepard
2012-06-25 13:02     Address already in use Manuel Palenciano Guerrero
2012-06-25 20:28     ` Eric Wong
2012-06-25 21:03       ` Manuel Palenciano Guerrero
2012-06-25 23:57  5%     ` Eric Wong
2012-07-19 22:43  4% Detecting unicorn / defining after_fork after master startup Matt Sanders
2012-07-19 23:13  0% ` Eric Wong
2012-07-21 19:42  0%   ` Matt Sanders
2012-07-22 21:20  0%     ` Eric Wong
2012-10-29 17:44     Combating nginx 499 HTTP responses during flash traffic scenario Tom Burns
2012-10-29 18:45     ` Eric Wong
2012-10-29 19:27       ` Hongli Lai
2012-10-29 19:41         ` Eric Wong
2012-10-29 21:06  4%       ` Hongli Lai
2012-10-29 21:53       ` Eric Wong
2012-10-30 20:40         ` Tom Burns
2012-10-30 21:37           ` Eric Wong
2012-11-02 17:59             ` Tom Burns
2012-11-02 19:38               ` Eric Wong
2012-11-03 22:45  2%             ` Tom Burns
2012-11-05 11:48  0%               ` Eric Wong
     [not found]     <CAHOTMV++otgxdru_oZLXuVuqHF7F4uMwd04O0QZBjxeqFR-=XQ@mail.gmail.com>
2012-11-29 23:05  5% ` Fwd: Maintaining capacity during deploys Tony Arcieri
2012-11-30  0:07  5% [ANN] unicorn 4.5.0pre1 - check_client_connection option Eric Wong
     [not found]     <mailman.105.1354553259.10424.mongrel-unicorn@rubyforge.org>
     [not found]     ` <CABe3nBB0B3vV+3yTUc=eh9RsV6S00Jy0PqtSks4DQfp_PgsjdA@mail.gmail.com>
     [not found]       ` <20121203211712.GA16488@dcvr.yhbt.net>
     [not found]         ` <CABe3nBC5bMDjupE3gr+OTX=qSeBHwNv3vV4qQXkxSVR_YACEUg@mail.gmail.com>
     [not found]           ` <20121204113041.GA17535@dcvr.yhbt.net>
2012-12-04 11:38  6%         ` Add licenses to gemspec Peter Marsh
2012-12-08  0:17  4% [ANN] unicorn 4.5.0 (final) - check_client_connection option Eric Wong
2013-01-25 10:52  6% No middleware without touching RACK_ENV Lin Jen-Shin (godfat)
2013-01-25 18:39  0% ` Eric Wong
2013-01-28 14:43  0%   ` Lin Jen-Shin (godfat)
2013-01-29  3:21  8% [PATCH] Add -N or --no-default-middleware option Lin Jen-Shin
2013-01-29  3:52  5% ` Eric Wong
2013-01-29  4:02  5%   ` Eric Wong
2013-01-29  4:03  6%   ` Lin Jen-Shin (godfat)
2013-05-12 18:12 13% add HTTP response headers pille
2013-05-12 19:20  5% ` Eric Wong
2013-05-12 20:25  6%   ` pille
2013-05-12 21:37  6%   ` Alejandro Riera
2013-06-13 16:09     HEAD responses contain body Jonathan Rudenberg
2013-06-13 18:22  5% ` Eric Wong
2013-06-13 18:42  0%   ` Jonathan Rudenberg
2013-06-13 19:21  0%     ` Eric Wong
2013-06-13 19:28  0%       ` Jonathan Rudenberg
2013-06-13 19:34  0%         ` Jonathan Rudenberg
2013-07-24  3:11  5% [PATCH] unicorn_forever: new executable to respawn masters Eric Wong
2013-08-20 14:47     A barrage of unexplained timeouts nick
2013-08-20 16:37     ` Eric Wong
2013-08-20 17:27       ` nick
2013-08-20 17:40         ` Eric Wong
2013-08-20 18:11           ` nick
2013-08-20 18:49             ` Eric Wong
2013-08-20 20:03               ` nick
2013-08-20 20:42                 ` Eric Wong
2013-08-20 21:19  5%               ` nick
2013-08-20 21:32  0%                 ` Eric Wong
2013-08-21 13:33  0%                   ` nick
2013-09-29 20:13  4% More " nick
2013-09-30  0:06  0% ` Eric Wong
2013-10-24 11:08  5% Forking non web processes Sam Saffron
2013-10-24 16:17  0% ` Eric Wong
2013-10-26  7:58  5% [PATCH] license: allow all future versions of the GNU GPL Eric Wong
2013-10-26  9:54     ` Hongli Lai
2013-10-26 10:09  5%   ` Eric Wong
2013-11-04  7:57  4% [ANN] unicorn 4.7.0 - minor updates, license tweak Eric Wong
2014-04-21 18:37     [ANN] public-inbox - "archives first" approach to mailing lists Eric Wong
2014-04-21 19:07  5% ` Eric Wong
2014-04-21 19:18  5% [PATCH] add slrnpull.conf example for gmane archives Eric Wong
2014-05-07  8:05     [ANN] unicorn 4.8.3 - the end of an era Eric Wong
2014-05-07  9:30     ` Jérémy Lecour
2014-05-07  9:46  4%   ` Eric Wong
2014-05-08  8:43  4% subscription to new ML probably working Eric Wong
2014-07-03 14:40     Log reopening broken on Rails 4 with config.autoflush_log = false? Cedric Maion
2014-07-03 17:24     ` Eric Wong
2014-07-03 19:28  5%   ` [PATCH] FAQ: add entry for Rails autoflush_log Eric Wong
2014-08-02  7:51     Please move to github Gary Grossman
2014-08-02  8:50  3% ` Eric Wong
2014-08-02 19:07       ` Gary Grossman
2014-08-02 19:33         ` Michael Fischer
2014-08-04  7:22           ` Hongli Lai
2014-08-04  8:48  5%         ` Rack encodings (was: Please move to github) Eric Wong
2014-10-03 11:34     Master hooks needed Bráulio Bhavamitra
2014-10-03 12:22  4% ` Eric Wong
2014-10-03 12:36  0%   ` Valentin Mihov
2014-10-04  0:53  0%   ` Bráulio Bhavamitra
2014-10-04  1:22         ` Eric Wong
2014-10-04  1:35           ` Bráulio Bhavamitra
2014-10-04  1:57             ` Eric Wong
2014-10-04  2:04               ` Bráulio Bhavamitra
2014-10-04  2:25  5%             ` Eric Wong
2014-10-04 23:22  4% [ANN] email submission port (587) enabled Eric Wong
2014-12-03  9:50     No, passenger 5.0 is not faster than unicorn :) Bráulio Bhavamitra
2014-12-03 11:00  4% ` Hongli Lai
2014-12-03 11:05  0%   ` Sam Saffron
2014-12-03 14:10  0%   ` Bráulio Bhavamitra
2015-02-10 17:06  4% [PATCH] ISSUES: add section for bugs in other projects Eric Wong
2015-02-10 17:26  5% ` Eric Wong
2015-03-27 14:32     $USER and $HOME shell variables not set Russell Jennings
2015-03-27 18:34  4% ` Eric Wong
2015-04-22 19:02  6% unicorn 4.8.x-stable branch pushed to git Eric Wong
2015-04-24  3:17  6% [ANN] unicorn 4.9.0 - Rack HTTP server for fast clients and *nix Eric Wong
2015-05-18 21:34  5% [PATCH] FAQ: add note about ECONNRESET errors from bodies Eric Wong
2015-06-04  1:39  2% [PATCH] http_server: remove a few more accessors and constants Eric Wong
2015-06-15 22:56  6% [ANN] unicorn 5.0.0.pre1 - incompatible changes! Eric Wong
2015-06-26 22:18     Does the the environment is preserved on USR2? Bráulio Bhavamitra
2015-06-26 22:31  5% ` Eric Wong
2015-06-26 23:14  0%   ` Eric Wong
2015-06-26 23:38  0%     ` Bráulio Bhavamitra
     [not found]     <3137844.nsIDTWmvl4@debstor>
     [not found]     ` <20150625083118.GA22140@luke.ws.skroutz.gr>
2015-06-25 23:26       ` [DRE-maint] unicorn: native systemd service Eric Wong
2015-06-26 11:41         ` Christos Trochalakis
2015-06-27  3:05  5%       ` Eric Wong
2015-06-27  4:01  4%         ` [RFC] emulate sd_listen_fds for systemd support Eric Wong
2015-06-30  9:20  0%         ` [DRE-maint] unicorn: native systemd service Christos Trochalakis
2015-07-15 22:05  2% [PATCH] doc: remove references to old servers Eric Wong
2015-10-05  2:43  7% [PATCH] doc: update mail archive info Eric Wong
2015-10-27  3:33     [PATCH 0/2] socket inheritance behavior/bug => feature! Eric Wong
2015-10-27  3:33  3% ` [PATCH 2/2] inheriting sockets from UNICORN_FD does not close them Eric Wong
2015-11-01  8:37     [PATCH 0/3] last updates before 5.0 release Eric Wong
2015-11-01  8:37  4% ` [PATCH 3/3] doc updates Eric Wong
2015-11-17 21:45  5% [PATCH] examples: add systemd socket and service files Eric Wong
2015-11-17 22:06  6% [PATCH] add .gitattributes for Ruby method detection Eric Wong
2016-01-07  3:41  2% [PUSHED] various documentation updates Eric Wong
2016-01-08 23:01     if you want to unsubscribe Eric Wong
2016-06-07 14:14  4% ` Eric Wong
     [not found]     <CAL-rKu6CZ731c=uHyZ8+Fg2fhC30e-3-J26XODacUK=YrfX+5Q@mail.gmail.com>
2016-06-07 13:41     ` [PATCH] `unicorn upgrade` script resilience against exceptions Eric Wong
2016-06-07 15:17       ` Jesper Rønn-Jensen
2016-06-07 21:13  3%     ` [PATCH] examples/init.sh: update to reduce upgrade raciness Eric Wong
2016-06-20 20:00  4% [PATCH] examples/logrotate.conf: update example for systemd Eric Wong
2016-08-05 23:49     Please join the Rack Specification discussion for `env['upgrade.websocket']` Bo
2016-08-06  1:01     ` Eric Wong
2016-08-06  3:09  3%   ` Boaz Segev
2016-08-06  4:59  0%     ` Eric Wong
2016-10-20  9:05  4% [PATCH] Add some tolerance (RFC2616 sec. 19.3) Mishael A Sibiryakov
2016-10-20 17:55  5% ` Eric Wong
2016-10-20 20:25  6%   ` Mishael A Sibiryakov
2016-10-20 20:50  6%     ` Eric Wong
2016-10-20 21:03  6%       ` Mishael A Sibiryakov
     [not found]     <20161031-unicorn-5.2.0-released@bogomips.org>
2016-10-31 20:04  4% ` [ANN] unicorn 5.2.0 - Rack HTTP server for fast clients and *nix Eric Wong
2017-02-21 19:19  4% Patch: Add after_worker_exit configuration option Jeremy Evans
2017-02-21 19:43  6% ` Eric Wong
2017-02-21 20:02  6%   ` Jeremy Evans
2017-02-21 20:15  6%     ` Eric Wong
2017-02-21 20:49  8%       ` Jeremy Evans
2017-02-21 21:38  6%         ` Eric Wong
2017-02-21 19:24  8% Patch: Add support for chroot to Worker#user Jeremy Evans
2017-02-21 19:53  6% ` Eric Wong
2017-02-21 20:15  6%   ` Jeremy Evans
2017-02-22 12:02     check_client_connection using getsockopt(2) Simon Eskildsen
2017-02-22 18:33     ` Eric Wong
2017-02-22 20:09       ` Simon Eskildsen
2017-02-23  1:42  4%     ` Eric Wong
2017-02-23  2:42  0%       ` Simon Eskildsen
2017-02-23  0:49  4% Patch: Add after_worker_ready configuration option Jeremy Evans
2017-02-23  2:32  9% ` Eric Wong
2017-02-23  2:34  6%   ` Eric Wong
2017-02-23  3:43  8%   ` Jeremy Evans
2017-02-23  9:23  4%     ` Eric Wong
2017-02-23 20:00  5%       ` Jeremy Evans
2017-02-23 18:46  8% Patch: Add support for chroot to Worker#user V2 Jeremy Evans
2017-04-05  3:44  6% ` Eric Wong
2017-04-05  4:57  6%   ` Jeremy Evans
2017-02-23 18:48     Patch: Fix code example in after_worker_exit documentation Jeremy Evans
2017-02-23 20:21  5% ` Eric Wong
2017-02-23 18:49  7% Patch: Add after_worker_ready configuration option V2 Jeremy Evans
2017-02-23 20:29 13% ` Eric Wong
2017-03-08  7:29 13%   ` Eric Wong
2017-03-08  7:44  9%     ` [PATCH] doc: add version annotations for new features Eric Wong
2017-03-08  6:02  5% [PATCH 0/3] TCP_INFO check_client_connection followups Eric Wong
2017-03-08 10:14  0% ` Simon Eskildsen
2017-03-08 18:44  3% [PATCH] Add worker_exec configuration option Jeremy Evans
2017-03-08 20:02  4% ` Eric Wong
2017-03-09  4:52  6%   ` Jeremy Evans
2017-03-09 13:57  6%     ` Eric Wong
2017-03-09 19:41  5%   ` [PATCH] Add worker_exec configuration option V2 Jeremy Evans
2017-03-10 21:19 11%     ` Eric Wong
2017-03-11  5:26  6%       ` Jeremy Evans
2017-03-11  7:18  6%         ` Eric Wong
2017-03-13 15:32  5%           ` Jeremy Evans
2017-03-13 19:18  6%             ` Eric Wong
2017-03-21 20:13     mail queue breakage :x Eric Wong
2017-03-21 23:10  5% ` Eric Wong
2017-03-21 23:24  0%   ` Eric Wong
2017-03-23  2:48  3% [ANN] raindrops 0.18.0 - real-time stats for preforking Rack servers Eric Wong
2017-03-24  0:28  6% [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix Eric Wong
2017-03-24  0:33  5% [PATCH] doc: note after_worker_exit is also 5.3.0+ Eric Wong
2017-04-01  8:08  6% [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix Eric Wong
2017-04-04 14:08  5% after_worker_exit on murder Simon Eskildsen
2017-04-04 14:32  0% ` Jeremy Evans
2017-04-05  1:19  4% ` Eric Wong
2017-08-07 20:18     Random crash when sending USR2 + QUIT signals to Unicorn process Eric Wong
2017-10-03 14:52  5% ` Xuanzhong Wei
2017-10-03 17:15  0%   ` Eric Wong
2017-10-03 14:57  5% [PATCH] fix GC issue on rb_global_variable array Xuanzhong Wei
2017-12-04 23:42  5% Auto scaling workers with unicorn Sam Saffron
2017-12-05  1:51  0% ` Eric Wong
2017-12-22  3:17  3% [PATCH] tests: cleanup some unused variable warnings Eric Wong
2017-12-23 23:06  5% [PATCH] ISSUES: add a note about Debian BTS interopability Eric Wong
2017-12-23 23:42  4% [ANN] unicorn 5.4.0 - Rack HTTP server for fast clients and Unix Eric Wong
2018-08-14  2:05     KGIO issues under WSL Sam Saffron
2018-08-14  3:14  4% ` Eric Wong
2018-08-20 20:42  6% [PATCH] socket_helper: add hint for FreeBSD users for accf_http(9) Eric Wong
2018-09-13 19:20  5% Support default_middleware configurator method Jeremy Evans
2018-09-13 22:34  5% ` Eric Wong
2018-09-14  0:00  6%   ` Jeremy Evans
2018-09-14 10:56  0%     ` Eric Wong
2018-09-14 15:03  6%       ` Jeremy Evans
2018-09-19  7:39             ` Eric Wong
2018-09-21  0:21  5%           ` Jeremy Evans
2018-12-13  1:07  4% [PATCH] doc/ISSUES: add links to git clone-able mail archives of our dependencies Eric Wong
2018-12-20 22:28  5% [ANN] unicorn 5.5.0.pre1 - Rack HTTP server for fast clients and Unix Eric Wong
2018-12-28 21:45  5% [PATCH] Links: add historical mongrel dev list mirror Eric Wong
2019-07-04 22:01     [PATCH 0/3] http: use gperf for common field memoization Eric Wong
2019-07-04 22:01  5% ` [PATCH 3/3] http: memoize more common fields Eric Wong
2019-12-16 22:34     Traffic priority with Unicorn Bertrand Paquet
2019-12-17  5:12  3% ` Eric Wong
2019-12-18 22:06  0%   ` Bertrand Paquet
2020-01-14  7:46  4% [PATCH] doc: s/bogomips.org/yhbt.net/g Eric Wong
2020-01-14  7:46  2% ` Eric Wong
2020-07-16 10:05  4% [PATCH] Add early hints support Jean Boussier
2020-07-16 10:50  5% ` Eric Wong
2020-07-16 11:41  6%   ` Jean Boussier
2020-07-16 12:16  6%     ` Eric Wong
2020-07-16 12:24  5%       ` Jean Boussier
2020-07-17  1:19  6%         ` Eric Wong
2020-07-20  9:18  6%           ` Jean Boussier
2020-07-20 10:09  6%             ` Eric Wong
2020-07-20 10:27  5%               ` Jean Boussier
2020-07-20 10:55  6%                 ` Eric Wong
2020-07-20 11:53  6%                   ` Jean Boussier
2020-07-20 20:27  6%                     ` Eric Wong
2020-09-08  2:06  4% [PATCH] PATCH: doc: add IMAP/IMAPS mailbox info Eric Wong
2020-12-08 21:47  4% [PATCH] Add rack.after_reply functionality Blake Williams
2020-12-08 22:46  5% ` Eric Wong
2020-12-08 23:48  6%   ` Blake Williams
2020-12-09  9:43  6%     ` Eric Wong
2020-12-09 14:58  6%       ` Blake Williams
2020-12-09 22:18  9%         ` Eric Wong
2020-12-09 23:44  6%           ` Blake Williams
     [not found]     <F6712BF3-A4DD-41EE-8252-B9799B35E618@github.com>
     [not found]     ` <20210311030250.GA1266@dcvr>
     [not found]       ` <7F6FD017-7802-4871-88A3-1E03D26D967C@github.com>
2021-03-12  9:41  2%     ` Potential Unicorn vulnerability Eric Wong
2021-03-12 11:14  5%       ` Dirkjan Bussink
2021-03-12 12:00  0%         ` Eric Wong
2021-03-29 19:06     Bus Error with Unicorn 6.0 on OpenBSD/adJ 6.8 with Ruby 3.0 vtamara
2021-03-30  1:58     ` Eric Wong
2021-03-30  4:00       ` Jeremy Evans
2021-04-04 11:06  4%     ` vtamara
2021-04-04 16:43  5%       ` Jeremy Evans
2021-04-04 20:03  0%         ` Eric Wong
2021-04-07 21:19  0%           ` Bus Error with Unicorn 6.0 on OpenBSD-current (6.9) with Ruby 3.0 in minimal rails application vtamara
2022-08-16 18:44  4% [PATCH] doc: add POP3 archive link, favor AUTH=ANONYMOUS for IMAP Eric Wong
2023-05-06 22:02     unicorn worker being killed issue subashkc1
2023-05-07 23:13     ` Eric Wong
2023-05-10 21:34  4%   ` subashkc1
2023-06-05 10:32  1% [PATCH 00-23/23] start porting tests to Perl5 Eric Wong
2023-06-20 10:46  4% [PATCH] unicorn_http_common.rl: use only ASCII spaces for compatibility EW
2023-06-20 12:28  7% [PATCH] add chunk_response config directive " EW
2023-09-10 20:08  1% [PATCH 00..11/11] more tests to Perl 5 Eric Wong
2024-03-23 19:45  3% [PATCH 0/4] a small pile of patches Eric Wong

Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).