unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 00-23/23] start porting tests to Perl5
@ 2023-06-05 10:32  1% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2023-06-05 10:32 UTC (permalink / raw)
  To: unicorn-public

[-- Attachment #1: Type: text/plain, Size: 4179 bytes --]

Still a lot more work to do, but at least socat is no longer a
test dependency.  Perl5 is installed on far more systems than
socat.

Ruby introduces breaking changes every year and I can't trust
tests to work as they were originally intended, anymore.
Perl 5 doesn't have perfect backwards compatibility, either; but
it's the least bad of any widely-installed scripting language.

Note: that 23/23 introduces a subtle bugfix which changes
behavior for systemd users

Patches are attached to reduce load on SMTP servers.
Some more patches to come as I deal with Ruby 3.x deprecation
warnings :<

Eric Wong (23):
  switch unit/test_response.rb to Perl 5 integration test
  support rack 3 multi-value headers
  port t0018-write-on-close.sh to Perl 5
  port t0000-http-basic.sh to Perl 5
  port t0002-parser-error.sh to Perl 5
  t/integration.t: use start_req to simplify test slighly
  port t0011-active-unix-socket.sh to Perl 5
  port t0100-rack-input-tests.sh to Perl 5
  tests: use autodie to simplify error checking
  port t0019-max_header_len.sh to Perl 5
  test_exec: drop sd_listen_fds emulation test
  test_exec: drop test_basic and test_config_ru_alt_path
  tests: check_stderr consistently in Perl 5 tests
  tests: consistent tcp_start and unix_start across Perl 5 tests
  port t9000-preread-input.sh to Perl 5
  port t/t0116-client_body_buffer_size.sh to Perl 5
  tests: get rid of sha1sum.rb and rsha1() sh function
  early_hints supports Rack 3 array headers
  test_server: drop early_hints test
  t/integration.t: switch PUT tests to MD5, reuse buffers
  tests: move test_upload.rb tests to t/integration.t
  drop redundant IO#close_on_exec=false calls
  LISTEN_FDS-inherited sockets are immortal across SIGHUP

 GNUmakefile                                |   7 +-
 lib/unicorn/http_server.rb                 |  12 +-
 t/README                                   |  21 +-
 t/active-unix-socket.t                     | 113 +++++++
 t/bin/content-md5-put                      |  36 ---
 t/bin/sha1sum.rb                           |  17 --
 t/{t0116.ru => client_body_buffer_size.ru} |   2 -
 t/client_body_buffer_size.t                |  82 ++++++
 t/integration.ru                           | 114 +++++++
 t/integration.t                            | 326 +++++++++++++++++++++
 t/lib.perl                                 | 217 ++++++++++++++
 t/preread_input.ru                         |  21 +-
 t/rack-input-tests.ru                      |  21 --
 t/t0000-http-basic.sh                      |  50 ----
 t/t0002-parser-error.sh                    |  94 ------
 t/t0011-active-unix-socket.sh              |  79 -----
 t/t0018-write-on-close.sh                  |  23 --
 t/t0019-max_header_len.sh                  |  49 ----
 t/t0100-rack-input-tests.sh                | 124 --------
 t/t0116-client_body_buffer_size.sh         |  80 -----
 t/t9000-preread-input.sh                   |  48 ---
 t/test-lib.sh                              |   4 -
 t/write-on-close.ru                        |  11 -
 test/exec/test_exec.rb                     |  57 ----
 test/unit/test_response.rb                 | 111 -------
 test/unit/test_server.rb                   |  31 --
 test/unit/test_upload.rb                   | 301 -------------------
 27 files changed, 891 insertions(+), 1160 deletions(-)
 create mode 100644 t/active-unix-socket.t
 delete mode 100755 t/bin/content-md5-put
 delete mode 100755 t/bin/sha1sum.rb
 rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
 create mode 100644 t/client_body_buffer_size.t
 create mode 100644 t/integration.ru
 create mode 100644 t/integration.t
 create mode 100644 t/lib.perl
 delete mode 100644 t/rack-input-tests.ru
 delete mode 100755 t/t0000-http-basic.sh
 delete mode 100755 t/t0002-parser-error.sh
 delete mode 100755 t/t0011-active-unix-socket.sh
 delete mode 100755 t/t0018-write-on-close.sh
 delete mode 100755 t/t0019-max_header_len.sh
 delete mode 100755 t/t0100-rack-input-tests.sh
 delete mode 100755 t/t0116-client_body_buffer_size.sh
 delete mode 100755 t/t9000-preread-input.sh
 delete mode 100644 t/write-on-close.ru
 delete mode 100644 test/unit/test_response.rb
 delete mode 100644 test/unit/test_upload.rb

[-- Attachment #2: 0001-switch-unit-test_response.rb-to-Perl-5-integration-t.patch --]
[-- Type: text/x-diff, Size: 15667 bytes --]

From 086e397abc0126556af24df77a976671294df2ee Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:30 +0000
Subject: [PATCH 01/23] switch unit/test_response.rb to Perl 5 integration test

http_response_write may benefit from API changes for Rack 3
support.

Since there's no benefit I can see from using a unit test,
switch to an integration test to avoid having to maintain the
unit test if our internal http_response_write method changes.

Of course, I can't trust tests written in Ruby since I've had to
put up with a constant stream of incompatibilities over the past
two decades :<   Perl is more widely installed than socat[1], and
nearly all the Perl I wrote 20 years ago still works
unmodified today.

[1] the rarest dependency of the Bourne shell integration tests
---
 GNUmakefile                |   5 +-
 t/README                   |  24 +++--
 t/integration.ru           |  38 ++++++++
 t/integration.t            |  64 +++++++++++++
 t/lib.perl                 | 189 +++++++++++++++++++++++++++++++++++++
 test/unit/test_response.rb | 111 ----------------------
 6 files changed, 313 insertions(+), 118 deletions(-)
 create mode 100644 t/integration.ru
 create mode 100644 t/integration.t
 create mode 100644 t/lib.perl
 delete mode 100644 test/unit/test_response.rb

diff --git a/GNUmakefile b/GNUmakefile
index 0e08ef0..5cca189 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -86,7 +86,7 @@ $(tmp_bin)/%: bin/% | $(tmp_bin)
 bins: $(tmp_bins)
 
 t_log := $(T_log) $(T_n_log)
-test: $(T) $(T_n)
+test: $(T) $(T_n) test-prove
 	@cat $(t_log) | $(MRI) test/aggregate.rb
 	@$(RM) $(t_log)
 
@@ -141,6 +141,9 @@ t/random_blob:
 
 test-integration: $(T_sh)
 
+test-prove:
+	prove -vw
+
 check: test-require test test-integration
 test-all: check
 
diff --git a/t/README b/t/README
index 14de559..8a5243e 100644
--- a/t/README
+++ b/t/README
@@ -5,16 +5,24 @@ TCP ports or Unix domain sockets.  They're all designed to run
 concurrently with other tests to minimize test time, but tests may be
 run independently as well.
 
-We write our tests in Bourne shell because that's what we're
-comfortable writing integration tests with.
+New tests are written in Perl 5 because we need a stable language
+to test real-world behavior and Ruby introduces incompatibilities
+at a far faster rate than Perl 5.  Perl is Ruby's older cousin, so
+it should be easy-to-learn for Rubyists.
+
+Old tests are in Bourne shell, but the socat(1) dependency was probably
+too rare compared to Perl 5.
 
 == Requirements
 
-* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!)
+* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
+* {Perl 5.14+}[https://www.perl.org/] # your distro should have it
 * {GNU make}[https://www.gnu.org/software/make/]
+
+The following requirements will eventually be dropped.
+
 * {socat}[http://www.dest-unreach.org/socat/]
 * {curl}[https://curl.haxx.se/]
-* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...)
 
 We do not use bashisms or any non-portable, non-POSIX constructs
 in our shell code.  We use the "pipefail" option if available and
@@ -26,9 +34,13 @@ with {dash}[http://gondor.apana.org.au/~herbert/dash/] and
 
 To run the entire test suite with 8 tests running at once:
 
-  make -j8
+  make -j8 && prove -vw
+
+To run one individual test (Perl5):
+
+  prove -vw t/integration.t
 
-To run one individual test:
+To run one individual test (shell):
 
   make t0000-simple-http.sh
 
diff --git a/t/integration.ru b/t/integration.ru
new file mode 100644
index 0000000..6ef873c
--- /dev/null
+++ b/t/integration.ru
@@ -0,0 +1,38 @@
+#!ruby
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+# this goes for t/integration.t  We'll try to put as many tests
+# in here as possible to avoid startup overhead of Ruby.
+
+$orig_rack_200 = nil
+def tweak_status_code
+  $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
+  Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
+  [ 200, {}, [] ]
+end
+
+def restore_status_code
+  $orig_rack_200 or return [ 500, {}, [] ]
+  Rack::Utils::HTTP_STATUS_CODES[200] = $orig_rack_200
+  [ 200, {}, [] ]
+end
+
+run(lambda do |env|
+  case env['REQUEST_METHOD']
+  when 'GET'
+    case env['PATH_INFO']
+    when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+    when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
+    when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+    end # case PATH_INFO (GET)
+  when 'POST'
+    case env['PATH_INFO']
+    when '/tweak-status-code'; tweak_status_code
+    when '/restore-status-code'; restore_status_code
+    end # case PATH_INFO (POST)
+    # ...
+  when 'PUT'
+    # ...
+  end # case REQUEST_METHOD
+end) # run
diff --git a/t/integration.t b/t/integration.t
new file mode 100644
index 0000000..5569155
--- /dev/null
+++ b/t/integration.t
@@ -0,0 +1,64 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+my $srv = tcp_server();
+my $t0 = time;
+my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+
+sub slurp_hdr {
+	my ($c) = @_;
+	local $/ = "\r\n\r\n"; # affects both readline+chomp
+	chomp(my $hdr = readline($c));
+	my ($status, @hdr) = split(/\r\n/, $hdr);
+	diag explain([ $status, \@hdr ]) if $ENV{V};
+	($status, \@hdr);
+}
+
+my ($c, $status, $hdr);
+
+# response header tests
+$c = tcp_connect($srv);
+print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+my $orig_200_status = $status;
+is_deeply([ grep(/^X-R2: /, @$hdr) ],
+	[ 'X-R2: a', 'X-R2: b', 'X-R2: c' ],
+	'rack 2 LF-delimited headers supported') or diag(explain($hdr));
+
+SKIP: { # Date header check
+	my @d = grep(/^Date: /i, @$hdr);
+	is(scalar(@d), 1, 'got one date header') or diag(explain(\@d));
+	eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1;
+	$d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix';
+	my $t = HTTP::Date::str2time($d[0]);
+	ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or
+		diag(explain([$t, $!, \@d]));
+};
+
+# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
+$c = tcp_connect($srv);
+print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
+	'nil header value accepted for broken apps') or diag(explain($hdr));
+
+if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
+	$c = tcp_connect($srv);
+	print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
+
+	$c = tcp_connect($srv);
+	print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	is($status, $orig_200_status, 'original status restored');
+}
+
+
+# ... more stuff here
+undef $ar;
+diag slurp("$tmpdir/err.log") if $ENV{V};
+done_testing;
diff --git a/t/lib.perl b/t/lib.perl
new file mode 100644
index 0000000..dd9c6b7
--- /dev/null
+++ b/t/lib.perl
@@ -0,0 +1,189 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@80x24.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+package UnicornTest;
+use v5.14;
+use parent qw(Exporter);
+use Test::More;
+use IO::Socket::INET;
+use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
+use File::Temp 0.19 (); # 0.19 for ->newdir
+our ($tmpdir, $errfh);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
+	SEEK_SET);
+
+my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
+$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
+open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+
+sub tcp_server {
+	my %opt = (
+		ReuseAddr => 1,
+		Proto => 'tcp',
+		Type => SOCK_STREAM,
+		Listen => SOMAXCONN,
+		Blocking => 0,
+		@_,
+	);
+	eval {
+		die 'IPv4-only' if $ENV{TEST_IPV4_ONLY};
+		require IO::Socket::INET6;
+		IO::Socket::INET6->new(%opt, LocalAddr => '[::1]')
+	} || eval {
+		die 'IPv6-only' if $ENV{TEST_IPV6_ONLY};
+		IO::Socket::INET->new(%opt, LocalAddr => '127.0.0.1')
+	} || BAIL_OUT "failed to create TCP server: $! ($@)";
+}
+
+sub tcp_host_port {
+	my ($s) = @_;
+	my ($h, $p) = ($s->sockhost, $s->sockport);
+	my $ipv4 = $s->sockdomain == AF_INET;
+	if (wantarray) {
+		$ipv4 ? ($h, $p) : ("[$h]", $p);
+	} else {
+		$ipv4 ? "$h:$p" : "[$h]:$p";
+	}
+}
+
+sub tcp_connect {
+	my ($dest, %opt) = @_;
+	my $addr = tcp_host_port($dest);
+	my $s = ref($dest)->new(
+		Proto => 'tcp',
+		Type => SOCK_STREAM,
+		PeerAddr => $addr,
+		%opt,
+	) or BAIL_OUT "failed to connect to $addr: $!";
+	$s->autoflush(1);
+	$s;
+}
+
+sub slurp {
+	open my $fh, '<', $_[0] or die "open($_[0]): $!";
+	local $/;
+	<$fh>;
+}
+
+sub spawn {
+	my $env = ref($_[0]) eq 'HASH' ? shift : undef;
+	my $opt = ref($_[-1]) eq 'HASH' ? pop : {};
+	my @cmd = @_;
+	my $old = POSIX::SigSet->new;
+	my $set = POSIX::SigSet->new;
+	$set->fillset or die "sigfillset: $!";
+	sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
+	pipe(my ($r, $w)) or die "pipe: $!";
+	my $pid = fork // die "fork: $!";
+	if ($pid == 0) {
+		close $r;
+		$SIG{__DIE__} = sub {
+			warn(@_);
+			syswrite($w, my $num = $! + 0);
+			_exit(1);
+		};
+
+		# pretend to be systemd (cf. sd_listen_fds(3))
+		my $cfd;
+		for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
+			my $io = $opt->{$cfd} // next;
+			my $pfd = fileno($io) // die "fileno($io): $!";
+			if ($pfd == $cfd) {
+				fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+			} else {
+				dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
+			}
+		}
+		if (($cfd - 3) > 0) {
+			$env->{LISTEN_PID} = $$;
+			$env->{LISTEN_FDS} = $cfd - 3;
+		}
+
+		if (defined(my $pgid = $opt->{pgid})) {
+			setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
+		}
+		$SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
+		if (defined(my $cd = $opt->{-C})) {
+			chdir $cd // die "chdir($cd): $!";
+		}
+		$old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
+		sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
+		@ENV{keys %$env} = values(%$env) if $env;
+		exec { $cmd[0] } @cmd;
+		die "exec @cmd: $!";
+	}
+	close $w;
+	sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK(old): $!";
+	if (my $cerrnum = do { local $/, <$r> }) {
+		$! = $cerrnum;
+		die "@cmd PID=$pid died: $!";
+	}
+	$pid;
+}
+
+sub which {
+	my ($file) = @_;
+	return $file if index($file, '/') >= 0;
+	for my $p (split(/:/, $ENV{PATH})) {
+		$p .= "/$file";
+		return $p if -x $p;
+	}
+	undef;
+}
+
+# returns an AutoReap object
+sub unicorn {
+	my %env;
+	if (ref($_[0]) eq 'HASH') {
+		my $e = shift;
+		%env = %$e;
+	}
+	my @args = @_;
+	push(@args, {}) if ref($args[-1]) ne 'HASH';
+	$args[-1]->{2} //= $errfh; # stderr default
+
+	state $ruby = which($ENV{RUBY} // 'ruby');
+	state $lib = File::Spec->rel2abs('lib');
+	state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`;
+	state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`;
+	state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http");
+	state $exe = File::Spec->rel2abs('bin/unicorn');
+	my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args);
+	UnicornTest::AutoReap->new($pid);
+}
+
+# automatically kill + reap children when this goes out-of-scope
+package UnicornTest::AutoReap;
+use v5.14;
+
+sub new {
+	my (undef, $pid) = @_;
+	bless { pid => $pid, owner => $$ }, __PACKAGE__
+}
+
+sub kill {
+	my ($self, $sig) = @_;
+	CORE::kill($sig // 'TERM', $self->{pid});
+}
+
+sub join {
+	my ($self, $sig) = @_;
+	my $pid = delete $self->{pid} or return;
+	CORE::kill($sig, $pid) if defined $sig;
+	my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+	$ret == $pid or die "BUG: waitpid($pid) != $ret";
+}
+
+sub DESTROY {
+	my ($self) = @_;
+	return if $self->{owner} != $$;
+	$self->join('TERM');
+}
+
+package main; # inject ourselves into the t/*.t script
+UnicornTest->import;
+Test::More->import;
+# try to ensure ->DESTROY fires:
+$SIG{TERM} = sub { exit(15 + 128) };
+$SIG{INT} = sub { exit(2 + 128) };
+1;
diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb
deleted file mode 100644
index fbe433f..0000000
--- a/test/unit/test_response.rb
+++ /dev/null
@@ -1,111 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2005 Zed A. Shaw
-# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or
-# the GPLv2+ (GPLv3+ preferred)
-#
-# Additional work donated by contributors.  See git history
-# for more information.
-
-require './test/test_helper'
-require 'time'
-
-include Unicorn
-
-class ResponseTest < Test::Unit::TestCase
-  include Unicorn::HttpResponse
-
-  def test_httpdate
-    before = Time.now.to_i - 1
-    str = httpdate
-    assert_kind_of(String, str)
-    middle = Time.parse(str).to_i
-    after = Time.now.to_i
-    assert before <= middle
-    assert middle <= after
-  end
-
-  def test_response_headers
-    out = StringIO.new
-    http_response_write(out, 200, {"X-Whatever" => "stuff"}, ["cool"])
-    assert ! out.closed?
-
-    assert out.length > 0, "output didn't have data"
-  end
-
-  # ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-  def test_response_header_broken_nil
-    out = StringIO.new
-    http_response_write(out, 200, {"Nil" => nil}, %w(hysterical raisin))
-    assert ! out.closed?
-
-    assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted'
-  end
-
-  def test_response_string_status
-    out = StringIO.new
-    http_response_write(out,'200', {}, [])
-    assert ! out.closed?
-    assert out.length > 0, "output didn't have data"
-  end
-
-  def test_response_200
-    io = StringIO.new
-    http_response_write(io, 200, {}, [])
-    assert ! io.closed?
-    assert io.length > 0, "output didn't have data"
-  end
-
-  def test_response_with_default_reason
-    code = 400
-    io = StringIO.new
-    http_response_write(io, code, {}, [])
-    assert ! io.closed?
-    lines = io.string.split(/\r\n/)
-    assert_match(/.* Bad Request$/, lines.first,
-                 "wrong default reason phrase")
-  end
-
-  def test_rack_multivalue_headers
-    out = StringIO.new
-    http_response_write(out,200, {"X-Whatever" => "stuff\nbleh"}, [])
-    assert ! out.closed?
-    assert_match(/^X-Whatever: stuff\r\nX-Whatever: bleh\r\n/, out.string)
-  end
-
-  # Even though Rack explicitly forbids "Status" in the header hash,
-  # some broken clients still rely on it
-  def test_status_header_added
-    out = StringIO.new
-    http_response_write(out,200, {"X-Whatever" => "stuff"}, [])
-    assert ! out.closed?
-  end
-
-  def test_unknown_status_pass_through
-    out = StringIO.new
-    http_response_write(out,"666 I AM THE BEAST", {}, [] )
-    assert ! out.closed?
-    headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/)
-    assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0])
-  end
-
-  def test_modified_rack_http_status_codes_late
-    r, w = IO.pipe
-    pid = fork do
-      r.close
-      # Users may want to globally override the status text associated
-      # with an HTTP status code in their app.
-      Rack::Utils::HTTP_STATUS_CODES[200] = "HI"
-      http_response_write(w, 200, {}, [])
-      w.close
-    end
-    w.close
-    assert_equal "HTTP/1.1 200 HI\r\n", r.gets
-    r.read # just drain the pipe
-    pid, status = Process.waitpid2(pid)
-    assert status.success?, status.inspect
-  ensure
-    r.close
-    w.close unless w.closed?
-  end
-end

[-- Attachment #3: 0002-support-rack-3-multi-value-headers.patch --]
[-- Type: text/x-diff, Size: 1710 bytes --]

From ea0559c700fa029044464de4bd572662c10b7273 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:31 +0000
Subject: [PATCH 02/23] support rack 3 multi-value headers

The first step in adding Rack 3 support.  Rack supports
multi-value headers via array rather than newlines.

Tested-by: Martin Posthumus <martin.posthumus@gmail.com>
Link: https://yhbt.net/unicorn-public/7c851d8a-bc57-7df8-3240-2f5ab831c47c@gmail.com/
---
 t/integration.ru | 1 +
 t/integration.t  | 9 +++++++++
 2 files changed, 10 insertions(+)

diff --git a/t/integration.ru b/t/integration.ru
index 6ef873c..5183217 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -23,6 +23,7 @@ def restore_status_code
   when 'GET'
     case env['PATH_INFO']
     when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ]
+    when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
     when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
     end # case PATH_INFO (GET)
diff --git a/t/integration.t b/t/integration.t
index 5569155..e876c71 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -38,6 +38,15 @@ SKIP: { # Date header check
 		diag(explain([$t, $!, \@d]));
 };
 
+
+$c = tcp_connect($srv);
+print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+($status, $hdr) = slurp_hdr($c);
+is_deeply([ grep(/^x-r3: /, @$hdr) ],
+	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
+	'rack 3 array headers supported') or diag(explain($hdr));
+
+
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
 $c = tcp_connect($srv);
 print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;

[-- Attachment #4: 0003-port-t0018-write-on-close.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4091 bytes --]

From 295a6c616f8840bc04617a377c04c3422aeebddc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:32 +0000
Subject: [PATCH 03/23] port t0018-write-on-close.sh to Perl 5

This doesn't require restarting, so it's a perfect candidate.
---
 t/integration.ru          | 15 +++++++++++++++
 t/integration.t           | 14 +++++++++++++-
 t/lib.perl                |  2 +-
 t/t0018-write-on-close.sh | 23 -----------------------
 t/write-on-close.ru       | 11 -----------
 5 files changed, 29 insertions(+), 36 deletions(-)
 delete mode 100755 t/t0018-write-on-close.sh
 delete mode 100644 t/write-on-close.ru

diff --git a/t/integration.ru b/t/integration.ru
index 5183217..12f5d48 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -18,6 +18,20 @@ def restore_status_code
   [ 200, {}, [] ]
 end
 
+class WriteOnClose
+  def each(&block)
+    @callback = block
+  end
+
+  def close
+    @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
+  end
+end
+
+def write_on_close
+  [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
+end
+
 run(lambda do |env|
   case env['REQUEST_METHOD']
   when 'GET'
@@ -26,6 +40,7 @@ def restore_status_code
     when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
     when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+    when '/write_on_close'; write_on_close
     end # case PATH_INFO (GET)
   when 'POST'
     case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index e876c71..3ab5c90 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -4,6 +4,7 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
 my $t0 = time;
 my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
 
@@ -66,8 +67,19 @@ if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
 	is($status, $orig_200_status, 'original status restored');
 }
 
+SKIP: {
+	eval { require HTTP::Tiny } or skip "HTTP::Tiny missing: $@", 1;
+	my $ht = HTTP::Tiny->new;
+	my $res = $ht->get("http://$host_port/write_on_close");
+	is($res->{content}, 'Goodbye', 'write-on-close body read');
+}
 
 # ... more stuff here
 undef $ar;
-diag slurp("$tmpdir/err.log") if $ENV{V};
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+is_deeply(\@err, [], 'no unexpected errors in stderr');
+is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
 done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index dd9c6b7..12deaf8 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET);
+	SEEK_SET tcp_host_port);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
diff --git a/t/t0018-write-on-close.sh b/t/t0018-write-on-close.sh
deleted file mode 100755
index 3afefea..0000000
--- a/t/t0018-write-on-close.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 4 "write-on-close tests for funky response-bodies"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	unicorn -D -c $unicorn_config write-on-close.ru
-	unicorn_wait_start
-}
-
-t_begin "write-on-close response body succeeds" && {
-	test xGoodbye = x"$(curl -sSf http://$listen/)"
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
-	check_stderr
-}
-
-t_done
diff --git a/t/write-on-close.ru b/t/write-on-close.ru
deleted file mode 100644
index 725c4d6..0000000
--- a/t/write-on-close.ru
+++ /dev/null
@@ -1,11 +0,0 @@
-class WriteOnClose
-  def each(&block)
-    @callback = block
-  end
-
-  def close
-    @callback.call "7\r\nGoodbye\r\n0\r\n\r\n"
-  end
-end
-use Rack::ContentType, "text/plain"
-run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] })

[-- Attachment #5: 0004-port-t0000-http-basic.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3372 bytes --]

From 1bb4362cee167ac7aeec910d3f52419e391f1e61 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:33 +0000
Subject: [PATCH 04/23] port t0000-http-basic.sh to Perl 5

One more socat dependency down...
---
 t/integration.ru      | 16 ++++++++++++++
 t/integration.t       | 11 ++++++++++
 t/t0000-http-basic.sh | 50 -------------------------------------------
 3 files changed, 27 insertions(+), 50 deletions(-)
 delete mode 100755 t/t0000-http-basic.sh

diff --git a/t/integration.ru b/t/integration.ru
index 12f5d48..c0bef99 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -32,6 +32,21 @@ def write_on_close
   [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ]
 end
 
+def env_dump(env)
+  require 'json'
+  h = {}
+  env.each do |k,v|
+    case v
+    when String, Integer, true, false; h[k] = v
+    else
+      case k
+      when 'rack.version', 'rack.after_reply'; h[k] = v
+      end
+    end
+  end
+  h.to_json
+end
+
 run(lambda do |env|
   case env['REQUEST_METHOD']
   when 'GET'
@@ -40,6 +55,7 @@ def write_on_close
     when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ]
     when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ]
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
+    when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
     end # case PATH_INFO (GET)
   when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 3ab5c90..ee22e7e 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -47,6 +47,17 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
 	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
 	'rack 3 array headers supported') or diag(explain($hdr));
 
+SKIP: {
+	eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
+	$c = tcp_connect($srv);
+	print $c "GET /env_dump\r\n" or die $!;
+	my $json = do { local $/; readline($c) };
+	unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
+	unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
+	my $env = JSON::PP->new->decode($json);
+	is(ref($env), 'HASH', 'JSON decoded body to hashref');
+	is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9');
+}
 
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
 $c = tcp_connect($srv);
diff --git a/t/t0000-http-basic.sh b/t/t0000-http-basic.sh
deleted file mode 100755
index 8ab58ac..0000000
--- a/t/t0000-http-basic.sh
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 8 "simple HTTP connection tests"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	unicorn -D -c $unicorn_config env.ru
-	unicorn_wait_start
-}
-
-t_begin "single request" && {
-	curl -sSfv http://$listen/
-}
-
-t_begin "check stderr has no errors" && {
-	check_stderr
-}
-
-t_begin "HTTP/0.9 request should not return headers" && {
-	(
-		printf 'GET /\r\n'
-		cat $fifo > $tmp &
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-}
-
-t_begin "env.inspect should've put everything on one line" && {
-	test 1 -eq $(count_lines < $tmp)
-}
-
-t_begin "no headers in output" && {
-	if grep ^Connection: $tmp
-	then
-		die "Connection header found in $tmp"
-	elif grep ^HTTP/ $tmp
-	then
-		die "HTTP/ found in $tmp"
-	fi
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr has no errors" && {
-	check_stderr
-}
-
-t_done

[-- Attachment #6: 0005-port-t0002-parser-error.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 4875 bytes --]

From 2eb7b1662c291ab535ee5dabf5d96194ca6483d4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:34 +0000
Subject: [PATCH 05/23] port t0002-parser-error.sh to Perl 5

Another socat dependency down...
---
 t/integration.t         | 33 +++++++++++++++
 t/lib.perl              |  9 +++-
 t/t0002-parser-error.sh | 94 -----------------------------------------
 3 files changed, 41 insertions(+), 95 deletions(-)
 delete mode 100755 t/t0002-parser-error.sh

diff --git a/t/integration.t b/t/integration.t
index ee22e7e..503b7eb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -85,6 +85,39 @@ SKIP: {
 	is($res->{content}, 'Goodbye', 'write-on-close body read');
 }
 
+if ('bad requests') {
+	$c = start_req($srv, 'GET /env_dump HTTP/1/1');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
+
+	$c = tcp_connect($srv);
+	print $c 'GET /' or die $!;
+	my $buf = join('', (0..9), 'ab');
+	for (0..1023) { print $c $buf or die $! }
+	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 414 \b!,
+		'414 on REQUEST_PATH > (12 * 1024)');
+
+	$c = tcp_connect($srv);
+	print $c 'GET /hello-world?a' or die $!;
+	$buf = join('', (0..9));
+	for (0..1023) { print $c $buf or die $! }
+	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 414 \b!,
+		'414 on QUERY_STRING > (10 * 1024)');
+
+	$c = tcp_connect($srv);
+	print $c 'GET /hello-world#a' or die $!;
+	$buf = join('', (0..9), 'a'..'f');
+	for (0..63) { print $c $buf or die $! }
+	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
+}
+
+
 # ... more stuff here
 undef $ar;
 my @log = slurp("$tmpdir/err.log");
diff --git a/t/lib.perl b/t/lib.perl
index 12deaf8..7d712b5 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port);
+	SEEK_SET tcp_host_port start_req);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -59,6 +59,13 @@ sub tcp_connect {
 	$s;
 }
 
+sub start_req {
+	my ($srv, @req) = @_;
+	my $c = tcp_connect($srv);
+	print $c @req, "\r\n\r\n" or die "print: $!";
+	$c;
+}
+
 sub slurp {
 	open my $fh, '<', $_[0] or die "open($_[0]): $!";
 	local $/;
diff --git a/t/t0002-parser-error.sh b/t/t0002-parser-error.sh
deleted file mode 100755
index 9dc1cd2..0000000
--- a/t/t0002-parser-error.sh
+++ /dev/null
@@ -1,94 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "parser error test"
-
-t_begin "setup and startup" && {
-	unicorn_setup
-	unicorn -D env.ru -c $unicorn_config
-	unicorn_wait_start
-}
-
-t_begin "send a bad request" && {
-	(
-		printf 'GET / HTTP/1/1\r\nHost: example.com\r\n\r\n'
-		cat $fifo > $tmp &
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-	test xok = x$(cat $ok)
-}
-
-dbgcat tmp
-
-t_begin "response should be a 400" && {
-	grep -F 'HTTP/1.1 400 Bad Request' $tmp
-}
-
-t_begin "send a huge Request URI (REQUEST_PATH > (12 * 1024))" && {
-	rm -f $tmp
-	cat $fifo > $tmp &
-	(
-		set -e
-		trap 'echo ok > $ok' EXIT
-		printf 'GET /'
-		for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
-		do
-			printf '0123456789ab'
-		done
-		printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
-	) | socat - TCP:$listen > $fifo || :
-	test xok = x$(cat $ok)
-	wait
-}
-
-t_begin "response should be a 414 (REQUEST_PATH)" && {
-	grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && {
-	rm -f $tmp
-	cat $fifo > $tmp &
-	(
-		set -e
-		trap 'echo ok > $ok' EXIT
-		printf 'GET /hello-world?a'
-		for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}')
-		do
-			printf '0123456789'
-		done
-		printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
-	) | socat - TCP:$listen > $fifo || :
-	test xok = x$(cat $ok)
-	wait
-}
-
-t_begin "response should be a 414 (QUERY_STRING)" && {
-	grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "send a huge Request URI (FRAGMENT > 1024)" && {
-	rm -f $tmp
-	cat $fifo > $tmp &
-	(
-		set -e
-		trap 'echo ok > $ok' EXIT
-		printf 'GET /hello-world#a'
-		for i in $(awk </dev/null 'BEGIN{for(i=0;i<64;i++) print i}')
-		do
-			printf '0123456789abcdef'
-		done
-		printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n'
-	) | socat - TCP:$listen > $fifo || :
-	test xok = x$(cat $ok)
-	wait
-}
-
-t_begin "response should be a 414 (FRAGMENT)" && {
-	grep -F 'HTTP/1.1 414 ' $tmp
-}
-
-t_begin "server stderr should be clean" && check_stderr
-
-t_begin "term signal sent" && kill $unicorn_pid
-
-t_done

[-- Attachment #7: 0006-t-integration.t-use-start_req-to-simplify-test-sligh.patch --]
[-- Type: text/x-diff, Size: 2556 bytes --]

From 0bb06cc0c8c4f5b76514858067bbb2871dda0d6e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:35 +0000
Subject: [PATCH 06/23] t/integration.t: use start_req to simplify test slighly

Less code is usually better.
---
 t/integration.t | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/t/integration.t b/t/integration.t
index 503b7eb..b7ba1fb 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -20,8 +20,7 @@ sub slurp_hdr {
 my ($c, $status, $hdr);
 
 # response header tests
-$c = tcp_connect($srv);
-print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
 my $orig_200_status = $status;
@@ -40,8 +39,7 @@ SKIP: { # Date header check
 };
 
 
-$c = tcp_connect($srv);
-print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([ grep(/^x-r3: /, @$hdr) ],
 	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -49,8 +47,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
 
 SKIP: {
 	eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
-	$c = tcp_connect($srv);
-	print $c "GET /env_dump\r\n" or die $!;
+	my $c = start_req($srv, 'GET /env_dump');
 	my $json = do { local $/; readline($c) };
 	unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
 	unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -60,20 +57,17 @@ SKIP: {
 }
 
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = tcp_connect($srv);
-print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!;
+$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
 if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
-	$c = tcp_connect($srv);
-	print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!;
+	$c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
 
-	$c = tcp_connect($srv);
-	print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!;
+	$c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	is($status, $orig_200_status, 'original status restored');
 }

[-- Attachment #8: 0007-port-t0011-active-unix-socket.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 6945 bytes --]

From 10c83beaca58df8b92d8228e798559069cd89beb Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:36 +0000
Subject: [PATCH 07/23] port t0011-active-unix-socket.sh to Perl 5

Another socat dependency down...  I've also started turning
FD_CLOEXEC off on a pipe as a mechanism to detect daemonized
process death in tests.
---
 t/active-unix-socket.t        | 117 ++++++++++++++++++++++++++++++++++
 t/integration.ru              |   1 +
 t/t0011-active-unix-socket.sh |  79 -----------------------
 3 files changed, 118 insertions(+), 79 deletions(-)
 create mode 100644 t/active-unix-socket.t
 delete mode 100755 t/t0011-active-unix-socket.sh

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
new file mode 100644
index 0000000..6b5c218
--- /dev/null
+++ b/t/active-unix-socket.t
@@ -0,0 +1,117 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use IO::Socket::UNIX;
+my %to_kill;
+END { kill('TERM', values(%to_kill)) if keys %to_kill }
+my $u1 = "$tmpdir/u1.sock";
+my $u2 = "$tmpdir/u2.sock";
+my $unix_req = sub {
+	my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
+	print $s @_, "\r\n\r\n" or die $!;
+	$s;
+};
+{
+	use autodie;
+	open my $fh, '>', "$tmpdir/u1.conf.rb";
+	print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u1"
+stderr_path "$tmpdir/err1.log"
+EOM
+	close $fh;
+
+	open $fh, '>', "$tmpdir/u2.conf.rb";
+	print $fh <<EOM;
+pid "$tmpdir/u.pid"
+listen "$u2"
+stderr_path "$tmpdir/err2.log"
+EOM
+	close $fh;
+
+	open $fh, '>', "$tmpdir/u3.conf.rb";
+	print $fh <<EOM;
+pid "$tmpdir/u3.pid"
+listen "$u1"
+stderr_path "$tmpdir/err3.log"
+EOM
+	close $fh;
+}
+
+my @uarg = qw(-D -E none t/integration.ru);
+
+# this pipe will be used to notify us when all daemons die:
+pipe(my ($p0, $p1)) or die "pipe: $!";
+fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+
+# start the first instance
+unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+is($?, 0, 'daemonized 1st process');
+chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
+ok(kill(0, $worker_pid), 'worker is kill-able');
+
+
+# 2nd process conflicts on PID
+unicorn('-c', "$tmpdir/u2.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting PID file fails to start');
+
+chomp(my $pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
+
+chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID unchanged');
+
+
+# 3rd process conflicts on socket
+unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
+isnt($?, 0, 'conflicting UNIX socket fails to start');
+
+chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+is($worker_pid, $pid2, 'worker PID still unchanged');
+
+chomp($pidf = slurp("$tmpdir/u.pid"));
+is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
+
+{ # teardown initial process via SIGKILL
+	ok(kill('KILL', delete $to_kill{u1}), 'SIGKILL initial daemon');
+	close $p1;
+	vec(my $rvec = '', fileno($p0), 1) = 1;
+	is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+	is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+	ok(-f "$tmpdir/u.pid", 'pid file stayed after SIGKILL');
+	ok(-S $u1, 'socket stayed after SIGKILL');
+	is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef,
+		'fail to connect to u1');
+	ok(!kill(0, $worker_pid), 'worker gone after parent dies');
+}
+
+# restart the first instance
+{
+	pipe(($p0, $p1)) or die "pipe: $!";
+	fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+	unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
+	is($?, 0, 'daemonized 1st process');
+	chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
+	like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
+
+	chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+	like($pid2, qr/\A\d+\z/, 'worker running');
+
+	ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
+	close $p1;
+	vec(my $rvec = '', fileno($p0), 1) = 1;
+	is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP');
+	is(my $undef = <$p0>, undef, 'process closed pipe writer at exit');
+	ok(!-f "$tmpdir/u.pid", 'pid file gone after SIGTERM');
+	ok(-S $u1, 'socket stays after SIGTERM');
+}
+
+my @log = slurp("$tmpdir/err.log");
+diag("@log") if $ENV{V};
+done_testing;
diff --git a/t/integration.ru b/t/integration.ru
index c0bef99..21f5449 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -57,6 +57,7 @@ def env_dump(env)
     when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ]
     when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
+    when '/pid'; [ 200, {}, [ "#$$\n" ] ]
     end # case PATH_INFO (GET)
   when 'POST'
     case env['PATH_INFO']
diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh
deleted file mode 100755
index fae0b6c..0000000
--- a/t/t0011-active-unix-socket.sh
+++ /dev/null
@@ -1,79 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 11 "existing UNIX domain socket check"
-
-read_pid_unix () {
-	x=$(printf 'GET / HTTP/1.0\r\n\r\n' | \
-	    socat - UNIX:$unix_socket | \
-	    tail -1)
-	test -n "$x"
-	y="$(expr "$x" : '\([0-9][0-9]*\)')"
-	test x"$x" = x"$y"
-	test -n "$y"
-	echo "$y"
-}
-
-t_begin "setup and start" && {
-	rtmpfiles unix_socket unix_config
-	rm -f $unix_socket
-	unicorn_setup
-	grep -v ^listen < $unicorn_config > $unix_config
-	echo "listen '$unix_socket'" >> $unix_config
-	unicorn -D -c $unix_config pid.ru
-	unicorn_wait_start
-	orig_master_pid=$unicorn_pid
-}
-
-t_begin "get pid of worker" && {
-	worker_pid=$(read_pid_unix)
-	t_info "worker_pid=$worker_pid"
-}
-
-t_begin "fails to start with existing pid file" && {
-	rm -f $ok
-	unicorn -D -c $unix_config pid.ru || echo ok > $ok
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "worker pid unchanged" && {
-	test x"$(read_pid_unix)" = x$worker_pid
-	> $r_err
-}
-
-t_begin "fails to start with listening UNIX domain socket bound" && {
-	rm $ok $pid
-	unicorn -D -c $unix_config pid.ru || echo ok > $ok
-	test x"$(cat $ok)" = xok
-	> $r_err
-}
-
-t_begin "worker pid unchanged (again)" && {
-	test x"$(read_pid_unix)" = x$worker_pid
-}
-
-t_begin "nuking the existing Unicorn succeeds" && {
-	kill -9 $unicorn_pid
-	while kill -0 $unicorn_pid
-	do
-		sleep 1
-	done
-	check_stderr
-}
-
-t_begin "succeeds in starting with leftover UNIX domain socket bound" && {
-	test -S $unix_socket
-	unicorn -D -c $unix_config pid.ru
-	unicorn_wait_start
-}
-
-t_begin "worker pid changed" && {
-	test x"$(read_pid_unix)" != x$worker_pid
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "no errors" && check_stderr
-
-t_done

[-- Attachment #9: 0008-port-t0100-rack-input-tests.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 11722 bytes --]

From b4ed148186295f2d5c8448eab7f2b201615d1e4e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:37 +0000
Subject: [PATCH 08/23] port t0100-rack-input-tests.sh to Perl 5

Yet another socat dependency gone \o/
---
 t/bin/content-md5-put       |  36 -----------
 t/integration.ru            |  27 +++++++-
 t/integration.t             |  97 +++++++++++++++++++++++++++-
 t/lib.perl                  |   3 +-
 t/rack-input-tests.ru       |  21 ------
 t/t0100-rack-input-tests.sh | 124 ------------------------------------
 6 files changed, 124 insertions(+), 184 deletions(-)
 delete mode 100755 t/bin/content-md5-put
 delete mode 100644 t/rack-input-tests.ru
 delete mode 100755 t/t0100-rack-input-tests.sh

diff --git a/t/bin/content-md5-put b/t/bin/content-md5-put
deleted file mode 100755
index 01da0bb..0000000
--- a/t/bin/content-md5-put
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# simple chunked HTTP PUT request generator (and just that),
-# it reads stdin and writes to stdout so socat can write to a
-# UNIX or TCP socket (or to another filter or file) along with
-# a Content-MD5 trailer.
-require 'digest/md5'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-
-bs = ENV['bs'] ? ENV['bs'].to_i : 4096
-
-if ARGV.grep("--no-headers").empty?
-  $stdout.write(
-      "PUT / HTTP/1.1\r\n" \
-      "Host: example.com\r\n" \
-      "Transfer-Encoding: chunked\r\n" \
-      "Trailer: Content-MD5\r\n" \
-      "\r\n"
-    )
-end
-
-digest = Digest::MD5.new
-if buf = $stdin.readpartial(bs)
-  begin
-    digest.update(buf)
-    $stdout.write("%x\r\n" % [ buf.size ])
-    $stdout.write(buf)
-    $stdout.write("\r\n")
-  end while $stdin.read(bs, buf)
-end
-
-digest = [ digest.digest ].pack('m').strip
-$stdout.write("0\r\n")
-$stdout.write("Content-MD5: #{digest}\r\n\r\n")
diff --git a/t/integration.ru b/t/integration.ru
index 21f5449..98528f6 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -47,6 +47,29 @@ def env_dump(env)
   h.to_json
 end
 
+def rack_input_tests(env)
+  return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
+  cap = 16384
+  require 'digest/sha1'
+  digest = Digest::SHA1.new
+  input = env['rack.input']
+  case env['PATH_INFO']
+  when '/rack_input/size_first'; input.size
+  when '/rack_input/rewind_first'; input.rewind
+  when '/rack_input'; # OK
+  else
+    abort "bad path: #{env['PATH_INFO']}"
+  end
+  if buf = input.read(rand(cap))
+    begin
+      raise "#{buf.size} > #{cap}" if buf.size > cap
+      digest.update(buf)
+    end while input.read(rand(cap), buf)
+  end
+  [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
+    [ digest.hexdigest ] ]
+end
+
 run(lambda do |env|
   case env['REQUEST_METHOD']
   when 'GET'
@@ -66,6 +89,8 @@ def env_dump(env)
     end # case PATH_INFO (POST)
     # ...
   when 'PUT'
-    # ...
+    case env['PATH_INFO']
+    when %r{\A/rack_input}; rack_input_tests(env)
+    end
   end # case REQUEST_METHOD
 end) # run
diff --git a/t/integration.t b/t/integration.t
index b7ba1fb..8cef561 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,13 +1,16 @@
 #!perl -w
 # Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# this is the main integration test for things which don't require
+# restarting or signals
 
 use v5.14; BEGIN { require './t/lib.perl' };
 my $srv = tcp_server();
 my $host_port = tcp_host_port($srv);
 my $t0 = time;
 my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
-
+my $curl = which('curl');
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
 sub slurp_hdr {
 	my ($c) = @_;
 	local $/ = "\r\n\r\n"; # affects both readline+chomp
@@ -17,6 +20,48 @@ sub slurp_hdr {
 	($status, \@hdr);
 }
 
+my %PUT = (
+	chunked_md5 => sub {
+		my ($in, $out, $path, %opt) = @_;
+		my $bs = $opt{bs} // 16384;
+		require Digest::MD5;
+		my $dig = Digest::MD5->new;
+		print $out <<EOM;
+PUT $path HTTP/1.1\r
+Transfer-Encoding: chunked\r
+Trailer: Content-MD5\r
+\r
+EOM
+		my ($buf, $r);
+		while (1) {
+			$r = read($in, $buf, $bs) // die "read: $!";
+			last if $r == 0;
+			printf $out "%x\r\n", length($buf);
+			print $out $buf, "\r\n";
+			$dig->add($buf);
+		}
+		print $out "0\r\nContent-MD5: ", $dig->b64digest, "\r\n\r\n";
+	},
+	identity => sub {
+		my ($in, $out, $path, %opt) = @_;
+		my $bs = $opt{bs} // 16384;
+		my $clen = $opt{-s} // -s $in;
+		print $out <<EOM;
+PUT $path HTTP/1.0\r
+Content-Length: $clen\r
+\r
+EOM
+		my ($buf, $r, $len);
+		while ($clen) {
+			$len = $clen > $bs ? $bs : $clen;
+			$r = read($in, $buf, $len) // die "read: $!";
+			die 'premature EOF' if $r == 0;
+			print $out $buf;
+			$clen -= $r;
+		}
+	},
+);
+
 my ($c, $status, $hdr);
 
 # response header tests
@@ -111,6 +156,55 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
 }
 
+# input tests
+my ($blob_size, $blob_hash);
+SKIP: {
+	open(my $rh, '<', 't/random_blob') or
+		skip "t/random_blob not generated $!", 1;
+	$blob_size = -s $rh;
+	require Digest::SHA;
+	$blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+
+	my $ck_hash = sub {
+		my ($sub, $path, %opt) = @_;
+		seek($rh, 0, SEEK_SET) // die "seek: $!";
+		$c = tcp_connect($srv);
+		$c->autoflush(0);
+		$PUT{$sub}->($rh, $c, $path, %opt);
+		$c->flush or die "flush: $!";
+		($status, $hdr) = slurp_hdr($c);
+		is(readline($c), $blob_hash, "$sub $path");
+	};
+	$ck_hash->('identity', '/rack_input', -s => $blob_size);
+	$ck_hash->('chunked_md5', '/rack_input');
+	$ck_hash->('identity', '/rack_input/size_first', -s => $blob_size);
+	$ck_hash->('identity', '/rack_input/rewind_first', -s => $blob_size);
+	$ck_hash->('chunked_md5', '/rack_input/size_first');
+	$ck_hash->('chunked_md5', '/rack_input/rewind_first');
+
+
+	$curl // skip 'no curl found in PATH', 1;
+
+	my ($copt, $cout);
+	my $url = "http://$host_port/rack_input";
+	my $do_curl = sub {
+		my (@arg) = @_;
+		pipe(my $cout, $copt->{1}) or die "pipe: $!";
+		open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+		my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
+		close(delete $copt->{1}) or die "close: $!";
+		is(readline($cout), $blob_hash, "curl @arg response");
+		is(waitpid($cpid, 0), $cpid, "curl @arg exited");
+		is($?, 0, "no error from curl @arg");
+		is(slurp("$tmpdir/curl.err"), '', "no stderr from curl @arg");
+	};
+
+	$do_curl->(qw(-T t/random_blob));
+
+	seek($rh, 0, SEEK_SET) // die "seek: $!";
+	$copt->{0} = $rh;
+	$do_curl->('-T-');
+}
 
 # ... more stuff here
 undef $ar;
@@ -120,4 +214,5 @@ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
 is_deeply(\@err, [], 'no unexpected errors in stderr');
 is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
 
+undef $tmpdir;
 done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 7d712b5..ae9f197 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port start_req);
+	SEEK_SET tcp_host_port start_req which spawn);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -193,4 +193,5 @@ Test::More->import;
 # try to ensure ->DESTROY fires:
 $SIG{TERM} = sub { exit(15 + 128) };
 $SIG{INT} = sub { exit(2 + 128) };
+$SIG{PIPE} = sub { exit(13 + 128) };
 1;
diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru
deleted file mode 100644
index 5459e85..0000000
--- a/t/rack-input-tests.ru
+++ /dev/null
@@ -1,21 +0,0 @@
-# SHA1 checksum generator
-require 'digest/sha1'
-use Rack::ContentLength
-cap = 16384
-app = lambda do |env|
-  /\A100-continue\z/i =~ env['HTTP_EXPECT'] and
-    return [ 100, {}, [] ]
-  digest = Digest::SHA1.new
-  input = env['rack.input']
-  input.size if env["PATH_INFO"] == "/size_first"
-  input.rewind if env["PATH_INFO"] == "/rewind_first"
-  if buf = input.read(rand(cap))
-    begin
-      raise "#{buf.size} > #{cap}" if buf.size > cap
-      digest.update(buf)
-    end while input.read(rand(cap), buf)
-  end
-
-  [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ]
-end
-run app
diff --git a/t/t0100-rack-input-tests.sh b/t/t0100-rack-input-tests.sh
deleted file mode 100755
index ee7a437..0000000
--- a/t/t0100-rack-input-tests.sh
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-test -r random_blob || die "random_blob required, run with 'make $0'"
-
-t_plan 10 "rack.input read tests"
-
-t_begin "setup and startup" && {
-	rtmpfiles curl_out curl_err
-	unicorn_setup
-	unicorn -E none -D rack-input-tests.ru -c $unicorn_config
-	blob_sha1=$(rsha1 < random_blob)
-	blob_size=$(count_bytes < random_blob)
-	t_info "blob_sha1=$blob_sha1"
-	unicorn_wait_start
-}
-
-t_begin "corked identity request" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT / HTTP/1.0\r\n'
-		printf 'Content-Length: %d\r\n\r\n' $blob_size
-		cat random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		content-md5-put < random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#size first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /size_first HTTP/1.0\r\n'
-		printf 'Content-Length: %d\r\n\r\n' $blob_size
-		cat random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked identity request (input#rewind first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /rewind_first HTTP/1.0\r\n'
-		printf 'Content-Length: %d\r\n\r\n' $blob_size
-		cat random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#size first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /size_first HTTP/1.1\r\n'
-		printf 'Host: example.com\r\n'
-		printf 'Transfer-Encoding: chunked\r\n'
-		printf 'Trailer: Content-MD5\r\n'
-		printf '\r\n'
-		content-md5-put --no-headers < random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "corked chunked request (input#rewind first)" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'PUT /rewind_first HTTP/1.1\r\n'
-		printf 'Host: example.com\r\n'
-		printf 'Transfer-Encoding: chunked\r\n'
-		printf 'Trailer: Content-MD5\r\n'
-		printf '\r\n'
-		content-md5-put --no-headers < random_blob
-		wait
-		echo ok > $ok
-	) | ( sleep 1 && socat - TCP4:$listen > $fifo )
-	test 1 -eq $(grep $blob_sha1 $tmp |count_lines)
-	test x"$(cat $ok)" = xok
-}
-
-t_begin "regular request" && {
-	curl -sSf -T random_blob http://$listen/ > $curl_out 2> $curl_err
-        test x$blob_sha1 = x$(cat $curl_out)
-        test ! -s $curl_err
-}
-
-t_begin "chunked request" && {
-	curl -sSf -T- < random_blob http://$listen/ > $curl_out 2> $curl_err
-        test x$blob_sha1 = x$(cat $curl_out)
-        test ! -s $curl_err
-}
-
-dbgcat r_err
-
-t_begin "shutdown" && {
-	kill $unicorn_pid
-}
-
-t_done

[-- Attachment #10: 0009-tests-use-autodie-to-simplify-error-checking.patch --]
[-- Type: text/x-diff, Size: 8495 bytes --]

From 3a1d015a3859b639d8e4463e9436a49f4f0f720e Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:38 +0000
Subject: [PATCH 09/23] tests: use autodie to simplify error checking

autodie is bundled with Perl 5.10+ and simplifies error
checking in most cases.  Some subroutines aren't perfectly
translatable and their call sites had to be tweaked, but
most of them are.
---
 t/active-unix-socket.t | 13 +++++++------
 t/integration.t        | 37 +++++++++++++++++++------------------
 t/lib.perl             | 30 +++++++++++++++---------------
 3 files changed, 41 insertions(+), 39 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 6b5c218..1241904 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -4,17 +4,18 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use IO::Socket::UNIX;
+use autodie;
+no autodie 'kill';
 my %to_kill;
 END { kill('TERM', values(%to_kill)) if keys %to_kill }
 my $u1 = "$tmpdir/u1.sock";
 my $u2 = "$tmpdir/u2.sock";
 my $unix_req = sub {
 	my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
-	print $s @_, "\r\n\r\n" or die $!;
+	print $s @_, "\r\n\r\n";
 	$s;
 };
 {
-	use autodie;
 	open my $fh, '>', "$tmpdir/u1.conf.rb";
 	print $fh <<EOM;
 pid "$tmpdir/u.pid"
@@ -43,8 +44,8 @@ EOM
 my @uarg = qw(-D -E none t/integration.ru);
 
 # this pipe will be used to notify us when all daemons die:
-pipe(my ($p0, $p1)) or die "pipe: $!";
-fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+pipe(my $p0, my $p1);
+fcntl($p1, POSIX::F_SETFD, 0);
 
 # start the first instance
 unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
@@ -93,8 +94,8 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 
 # restart the first instance
 {
-	pipe(($p0, $p1)) or die "pipe: $!";
-	fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC
+	pipe($p0, $p1);
+	fcntl($p1, POSIX::F_SETFD, 0);
 	unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join;
 	is($?, 0, 'daemonized 1st process');
 	chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
diff --git a/t/integration.t b/t/integration.t
index 8cef561..af17d51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -5,6 +5,7 @@
 # restarting or signals
 
 use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
 my $srv = tcp_server();
 my $host_port = tcp_host_port($srv);
 my $t0 = time;
@@ -34,7 +35,7 @@ Trailer: Content-MD5\r
 EOM
 		my ($buf, $r);
 		while (1) {
-			$r = read($in, $buf, $bs) // die "read: $!";
+			$r = read($in, $buf, $bs);
 			last if $r == 0;
 			printf $out "%x\r\n", length($buf);
 			print $out $buf, "\r\n";
@@ -54,7 +55,7 @@ EOM
 		my ($buf, $r, $len);
 		while ($clen) {
 			$len = $clen > $bs ? $bs : $clen;
-			$r = read($in, $buf, $len) // die "read: $!";
+			$r = read($in, $buf, $len);
 			die 'premature EOF' if $r == 0;
 			print $out $buf;
 			$clen -= $r;
@@ -130,28 +131,28 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
 
 	$c = tcp_connect($srv);
-	print $c 'GET /' or die $!;
+	print $c 'GET /';
 	my $buf = join('', (0..9), 'ab');
-	for (0..1023) { print $c $buf or die $! }
-	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	for (0..1023) { print $c $buf }
+	print $c " HTTP/1.0\r\n\r\n";
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on REQUEST_PATH > (12 * 1024)');
 
 	$c = tcp_connect($srv);
-	print $c 'GET /hello-world?a' or die $!;
+	print $c 'GET /hello-world?a';
 	$buf = join('', (0..9));
-	for (0..1023) { print $c $buf or die $! }
-	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	for (0..1023) { print $c $buf }
+	print $c " HTTP/1.0\r\n\r\n";
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on QUERY_STRING > (10 * 1024)');
 
 	$c = tcp_connect($srv);
-	print $c 'GET /hello-world#a' or die $!;
+	print $c 'GET /hello-world#a';
 	$buf = join('', (0..9), 'a'..'f');
-	for (0..63) { print $c $buf or die $! }
-	print $c " HTTP/1.0\r\n\r\n" or die $!;
+	for (0..63) { print $c $buf }
+	print $c " HTTP/1.0\r\n\r\n";
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)');
 }
@@ -159,7 +160,7 @@ if ('bad requests') {
 # input tests
 my ($blob_size, $blob_hash);
 SKIP: {
-	open(my $rh, '<', 't/random_blob') or
+	CORE::open(my $rh, '<', 't/random_blob') or
 		skip "t/random_blob not generated $!", 1;
 	$blob_size = -s $rh;
 	require Digest::SHA;
@@ -167,11 +168,11 @@ SKIP: {
 
 	my $ck_hash = sub {
 		my ($sub, $path, %opt) = @_;
-		seek($rh, 0, SEEK_SET) // die "seek: $!";
+		seek($rh, 0, SEEK_SET);
 		$c = tcp_connect($srv);
 		$c->autoflush(0);
 		$PUT{$sub}->($rh, $c, $path, %opt);
-		$c->flush or die "flush: $!";
+		$c->flush or die $!;
 		($status, $hdr) = slurp_hdr($c);
 		is(readline($c), $blob_hash, "$sub $path");
 	};
@@ -189,10 +190,10 @@ SKIP: {
 	my $url = "http://$host_port/rack_input";
 	my $do_curl = sub {
 		my (@arg) = @_;
-		pipe(my $cout, $copt->{1}) or die "pipe: $!";
-		open $copt->{2}, '>', "$tmpdir/curl.err" or die $!;
+		pipe(my $cout, $copt->{1});
+		open $copt->{2}, '>', "$tmpdir/curl.err";
 		my $cpid = spawn($curl, '-sSf', @arg, $url, $copt);
-		close(delete $copt->{1}) or die "close: $!";
+		close(delete $copt->{1});
 		is(readline($cout), $blob_hash, "curl @arg response");
 		is(waitpid($cpid, 0), $cpid, "curl @arg exited");
 		is($?, 0, "no error from curl @arg");
@@ -201,7 +202,7 @@ SKIP: {
 
 	$do_curl->(qw(-T t/random_blob));
 
-	seek($rh, 0, SEEK_SET) // die "seek: $!";
+	seek($rh, 0, SEEK_SET);
 	$copt->{0} = $rh;
 	$do_curl->('-T-');
 }
diff --git a/t/lib.perl b/t/lib.perl
index ae9f197..49632cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -4,6 +4,7 @@
 package UnicornTest;
 use v5.14;
 use parent qw(Exporter);
+use autodie;
 use Test::More;
 use IO::Socket::INET;
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
@@ -14,7 +15,7 @@ our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
-open($errfh, '>>', "$tmpdir/err.log") or die "open: $!";
+open($errfh, '>>', "$tmpdir/err.log");
 
 sub tcp_server {
 	my %opt = (
@@ -62,14 +63,14 @@ sub tcp_connect {
 sub start_req {
 	my ($srv, @req) = @_;
 	my $c = tcp_connect($srv);
-	print $c @req, "\r\n\r\n" or die "print: $!";
+	print $c @req, "\r\n\r\n";
 	$c;
 }
 
 sub slurp {
-	open my $fh, '<', $_[0] or die "open($_[0]): $!";
+	open my $fh, '<', $_[0];
 	local $/;
-	<$fh>;
+	readline($fh);
 }
 
 sub spawn {
@@ -80,8 +81,8 @@ sub spawn {
 	my $set = POSIX::SigSet->new;
 	$set->fillset or die "sigfillset: $!";
 	sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!";
-	pipe(my ($r, $w)) or die "pipe: $!";
-	my $pid = fork // die "fork: $!";
+	pipe(my $r, my $w);
+	my $pid = fork;
 	if ($pid == 0) {
 		close $r;
 		$SIG{__DIE__} = sub {
@@ -94,9 +95,9 @@ sub spawn {
 		my $cfd;
 		for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) {
 			my $io = $opt->{$cfd} // next;
-			my $pfd = fileno($io) // die "fileno($io): $!";
+			my $pfd = fileno($io);
 			if ($pfd == $cfd) {
-				fcntl($io, F_SETFD, 0) // die "F_SETFD: $!";
+				fcntl($io, F_SETFD, 0);
 			} else {
 				dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!";
 			}
@@ -110,9 +111,7 @@ sub spawn {
 			setpgid(0, $pgid) // die "setpgid(0, $pgid): $!";
 		}
 		$SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG);
-		if (defined(my $cd = $opt->{-C})) {
-			chdir $cd // die "chdir($cd): $!";
-		}
+		if (defined(my $cd = $opt->{-C})) { chdir $cd }
 		$old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!";
 		sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!";
 		@ENV{keys %$env} = values(%$env) if $env;
@@ -162,22 +161,23 @@ sub unicorn {
 # automatically kill + reap children when this goes out-of-scope
 package UnicornTest::AutoReap;
 use v5.14;
+use autodie;
 
 sub new {
 	my (undef, $pid) = @_;
 	bless { pid => $pid, owner => $$ }, __PACKAGE__
 }
 
-sub kill {
+sub do_kill {
 	my ($self, $sig) = @_;
-	CORE::kill($sig // 'TERM', $self->{pid});
+	kill($sig // 'TERM', $self->{pid});
 }
 
 sub join {
 	my ($self, $sig) = @_;
 	my $pid = delete $self->{pid} or return;
-	CORE::kill($sig, $pid) if defined $sig;
-	my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+	kill($sig, $pid) if defined $sig;
+	my $ret = waitpid($pid, 0);
 	$ret == $pid or die "BUG: waitpid($pid) != $ret";
 }
 

[-- Attachment #11: 0010-port-t0019-max_header_len.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 5571 bytes --]

From 43c7d73b8b9e6995b5a986b10a8623395e89a538 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:39 +0000
Subject: [PATCH 10/23] port t0019-max_header_len.sh to Perl 5

This was the final socat requirement for integration tests.
I think curl will remain an optional dependency for tests
since it's probably the most widely-installed HTTP client.
---
 GNUmakefile               |  2 +-
 t/README                  |  7 +-----
 t/integration.ru          |  1 +
 t/integration.t           | 43 +++++++++++++++++++++++++++++++---
 t/t0019-max_header_len.sh | 49 ---------------------------------------
 5 files changed, 43 insertions(+), 59 deletions(-)
 delete mode 100755 t/t0019-max_header_len.sh

diff --git a/GNUmakefile b/GNUmakefile
index 5cca189..eab9082 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -125,7 +125,7 @@ $(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore
 t/trash/.gitignore : | t/trash
 	echo '*' >$@
 
-dependencies := socat curl
+dependencies := curl
 deps := $(addprefix t/.dep+,$(dependencies))
 $(deps): dep_bin = $(lastword $(subst +, ,$@))
 $(deps):
diff --git a/t/README b/t/README
index 8a5243e..d09c715 100644
--- a/t/README
+++ b/t/README
@@ -10,18 +10,13 @@ to test real-world behavior and Ruby introduces incompatibilities
 at a far faster rate than Perl 5.  Perl is Ruby's older cousin, so
 it should be easy-to-learn for Rubyists.
 
-Old tests are in Bourne shell, but the socat(1) dependency was probably
-too rare compared to Perl 5.
+Old tests are in Bourne shell and slowly being ported to Perl 5.
 
 == Requirements
 
 * {Ruby 2.0.0+}[https://www.ruby-lang.org/en/]
 * {Perl 5.14+}[https://www.perl.org/] # your distro should have it
 * {GNU make}[https://www.gnu.org/software/make/]
-
-The following requirements will eventually be dropped.
-
-* {socat}[http://www.dest-unreach.org/socat/]
 * {curl}[https://curl.haxx.se/]
 
 We do not use bashisms or any non-portable, non-POSIX constructs
diff --git a/t/integration.ru b/t/integration.ru
index 98528f6..edc408c 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -81,6 +81,7 @@ def rack_input_tests(env)
     when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
     when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+    else '/'; [ 200, {}, [ env_dump(env) ] ]
     end # case PATH_INFO (GET)
   when 'POST'
     case env['PATH_INFO']
diff --git a/t/integration.t b/t/integration.t
index af17d51..c687655 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -1,15 +1,19 @@
 #!perl -w
 # Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
-# this is the main integration test for things which don't require
-# restarting or signals
+
+# This is the main integration test for fast-ish things to minimize
+# Ruby startup time penalties.
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
 my $srv = tcp_server();
 my $host_port = tcp_host_port($srv);
 my $t0 = time;
-my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv });
+my $conf = "$tmpdir/u.conf.rb";
+open my $conf_fh, '>', $conf;
+$conf_fh->autoflush(1);
+my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');
 END { diag slurp("$tmpdir/err.log") if $tmpdir };
 sub slurp_hdr {
@@ -207,7 +211,40 @@ SKIP: {
 	$do_curl->('-T-');
 }
 
+
 # ... more stuff here
+
+# SIGHUP-able stuff goes here
+
+if ('max_header_len internal API') {
+	undef $c;
+	my $req = 'GET / HTTP/1.0';
+	my $len = length($req."\r\n\r\n");
+	my $fifo = "$tmpdir/fifo";
+	POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+	print $conf_fh <<EOM;
+Unicorn::HttpParser.max_header_len = $len
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+	$ar->do_kill('HUP');
+	open my $fifo_fh, '<', $fifo;
+	my $wpid = readline($fifo_fh);
+	like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+	close $fifo_fh;
+	$wpid =~ s/\Apid=// or die;
+	ok(CORE::kill(0, $wpid), 'worker PID retrieved');
+
+	$c = start_req($srv, $req);
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
+
+	$c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
+}
+
+
 undef $ar;
 my @log = slurp("$tmpdir/err.log");
 diag("@log") if $ENV{V};
diff --git a/t/t0019-max_header_len.sh b/t/t0019-max_header_len.sh
deleted file mode 100755
index 6a355b4..0000000
--- a/t/t0019-max_header_len.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 5 "max_header_len setting (only intended for Rainbows!)"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	req='GET / HTTP/1.0\r\n\r\n'
-	len=$(printf "$req" | count_bytes)
-	echo Unicorn::HttpParser.max_header_len = $len >> $unicorn_config
-	unicorn -D -c $unicorn_config env.ru
-	unicorn_wait_start
-}
-
-t_begin "minimal request succeeds" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf "$req"
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-	test xok = x$(cat $ok)
-
-	fgrep "HTTP/1.1 200 OK" $tmp
-}
-
-t_begin "big request fails" && {
-	rm -f $tmp
-	(
-		cat $fifo > $tmp &
-		printf 'GET /xxxxxx HTTP/1.0\r\n\r\n'
-		wait
-		echo ok > $ok
-	) | socat - TCP:$listen > $fifo
-	test xok = x$(cat $ok)
-	fgrep "HTTP/1.1 413" $tmp
-}
-
-dbgcat tmp
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
-	check_stderr
-}
-
-t_done

[-- Attachment #12: 0011-test_exec-drop-sd_listen_fds-emulation-test.patch --]
[-- Type: text/x-diff, Size: 1751 bytes --]

From 5d828a4ef7683345bcf2ff659442fed0a6fb7a97 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:40 +0000
Subject: [PATCH 11/23] test_exec: drop sd_listen_fds emulation test

The Perl 5 tests already rely on this implicitly, and there was
never a point when Perl 5 couldn't emulate systemd behavior.
---
 test/exec/test_exec.rb | 33 ---------------------------------
 1 file changed, 33 deletions(-)

diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 2929b2e..1d3a0fd 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -97,39 +97,6 @@ def teardown
     end
   end
 
-  def test_sd_listen_fds_emulation
-    # [ruby-core:69895] [Bug #11336] fixed by r51576
-    return if RUBY_VERSION.to_f < 2.3
-
-    File.open("config.ru", "wb") { |fp| fp.write(HI) }
-    sock = TCPServer.new(@addr, @port)
-
-    [ %W(-l #@addr:#@port), nil ].each do |l|
-      sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
-
-      pid = xfork do
-        redirect_test_io do
-          # pretend to be systemd
-          ENV['LISTEN_PID'] = "#$$"
-          ENV['LISTEN_FDS'] = '1'
-
-          # 3 = SD_LISTEN_FDS_START
-          args = [ $unicorn_bin ]
-          args.concat(l) if l
-          args << { 3 => sock }
-          exec(*args)
-        end
-      end
-      res = hit(["http://#@addr:#@port/"])
-      assert_equal [ "HI\n" ], res
-      assert_shutdown(pid)
-      assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool,
-                  'unicorn should always set SO_KEEPALIVE on inherited sockets'
-    end
-  ensure
-    sock.close if sock
-  end
-
   def test_inherit_listener_unspecified
     File.open("config.ru", "wb") { |fp| fp.write(HI) }
     sock = TCPServer.new(@addr, @port)

[-- Attachment #13: 0012-test_exec-drop-test_basic-and-test_config_ru_alt_pat.patch --]
[-- Type: text/x-diff, Size: 1667 bytes --]

From 548593c6b3d52a4bebd52542ad9c423ed2b7252d Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:41 +0000
Subject: [PATCH 12/23] test_exec: drop test_basic and test_config_ru_alt_path

We already have coverage for these basic things elsewhere.
---
 test/exec/test_exec.rb | 24 ------------------------
 1 file changed, 24 deletions(-)

diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 1d3a0fd..55f828e 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -265,16 +265,6 @@ def test_exit_signals
     end
   end
 
-  def test_basic
-    File.open("config.ru", "wb") { |fp| fp.syswrite(HI) }
-    pid = fork do
-      redirect_test_io { exec($unicorn_bin, "-l", "#{@addr}:#{@port}") }
-    end
-    results = retry_hit(["http://#{@addr}:#{@port}/"])
-    assert_equal String, results[0].class
-    assert_shutdown(pid)
-  end
-
   def test_rack_env_unset
     File.open("config.ru", "wb") { |fp| fp.syswrite(SHOW_RACK_ENV) }
     pid = fork { redirect_test_io { exec($unicorn_bin, "-l#@addr:#@port") } }
@@ -638,20 +628,6 @@ def test_read_embedded_cli_switches
     assert_shutdown(pid)
   end
 
-  def test_config_ru_alt_path
-    config_path = "#{@tmpdir}/foo.ru"
-    File.open(config_path, "wb") { |fp| fp.syswrite(HI) }
-    pid = fork do
-      redirect_test_io do
-        Dir.chdir("/")
-        exec($unicorn_bin, "-l#{@addr}:#{@port}", config_path)
-      end
-    end
-    results = retry_hit(["http://#{@addr}:#{@port}/"])
-    assert_equal String, results[0].class
-    assert_shutdown(pid)
-  end
-
   def test_load_module
     libdir = "#{@tmpdir}/lib"
     FileUtils.mkpath([ libdir ])

[-- Attachment #14: 0013-tests-check_stderr-consistently-in-Perl-5-tests.patch --]
[-- Type: text/x-diff, Size: 2415 bytes --]

From cd7ee67fc8ebadec9bdd913d49ed3f214596ea47 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:42 +0000
Subject: [PATCH 13/23] tests: check_stderr consistently in Perl 5 tests

The Bourne shell tests did, so lets not let stuff sneak past us.
---
 t/active-unix-socket.t |  5 ++---
 t/integration.t        |  7 ++-----
 t/lib.perl             | 10 +++++++++-
 3 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 1241904..c132dc2 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -20,7 +20,7 @@ my $unix_req = sub {
 	print $fh <<EOM;
 pid "$tmpdir/u.pid"
 listen "$u1"
-stderr_path "$tmpdir/err1.log"
+stderr_path "$tmpdir/err.log"
 EOM
 	close $fh;
 
@@ -113,6 +113,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 	ok(-S $u1, 'socket stays after SIGTERM');
 }
 
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
+check_stderr;
 done_testing;
diff --git a/t/integration.t b/t/integration.t
index c687655..939dc24 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -246,11 +246,8 @@ EOM
 
 
 undef $ar;
-my @log = slurp("$tmpdir/err.log");
-diag("@log") if $ENV{V};
-my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
-is_deeply(\@err, [], 'no unexpected errors in stderr');
-is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+
+check_stderr;
 
 undef $tmpdir;
 done_testing;
diff --git a/t/lib.perl b/t/lib.perl
index 49632cf..315ef2d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,12 +11,20 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port start_req which spawn);
+	SEEK_SET tcp_host_port start_req which spawn check_stderr);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
 open($errfh, '>>', "$tmpdir/err.log");
 
+sub check_stderr () {
+	my @log = slurp("$tmpdir/err.log");
+	diag("@log") if $ENV{V};
+	my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log));
+	is_deeply(\@err, [], 'no unexpected errors in stderr');
+	is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
+}
+
 sub tcp_server {
 	my %opt = (
 		ReuseAddr => 1,

[-- Attachment #15: 0014-tests-consistent-tcp_start-and-unix_start-across-Per.patch --]
[-- Type: text/x-diff, Size: 8017 bytes --]

From 0dcd8bd569813a175ad43837db3ab07019a95b99 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:43 +0000
Subject: [PATCH 14/23] tests: consistent tcp_start and unix_start across Perl
 5 tests

I'll be using Unix sockets more in tests since there's no
risk of system-wide conflicts with TCP port allocation.
Furthermore, curl supports `--unix-socket' nowadays; so
there's little reason to rely on TCP sockets and the conflicts
they bring in tests.
---
 t/active-unix-socket.t | 13 ++++---------
 t/integration.t        | 28 ++++++++++++++--------------
 t/lib.perl             | 30 ++++++++++++++++--------------
 3 files changed, 34 insertions(+), 37 deletions(-)

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index c132dc2..8723137 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -10,11 +10,6 @@ my %to_kill;
 END { kill('TERM', values(%to_kill)) if keys %to_kill }
 my $u1 = "$tmpdir/u1.sock";
 my $u2 = "$tmpdir/u2.sock";
-my $unix_req = sub {
-	my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM);
-	print $s @_, "\r\n\r\n";
-	$s;
-};
 {
 	open my $fh, '>', "$tmpdir/u1.conf.rb";
 	print $fh <<EOM;
@@ -53,7 +48,7 @@ is($?, 0, 'daemonized 1st process');
 chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
 like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
 
-chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $worker_pid = readline(unix_start($u1, 'GET /pid')));
 like($worker_pid, qr/\A\d+\z/s, 'captured worker pid');
 ok(kill(0, $worker_pid), 'worker is kill-able');
 
@@ -65,7 +60,7 @@ isnt($?, 0, 'conflicting PID file fails to start');
 chomp(my $pidf = slurp("$tmpdir/u.pid"));
 is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure');
 
-chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp(my $pid2 = readline(unix_start($u1, 'GET /pid')));
 is($worker_pid, $pid2, 'worker PID unchanged');
 
 
@@ -73,7 +68,7 @@ is($worker_pid, $pid2, 'worker PID unchanged');
 unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join;
 isnt($?, 0, 'conflicting UNIX socket fails to start');
 
-chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
 is($worker_pid, $pid2, 'worker PID still unchanged');
 
 chomp($pidf = slurp("$tmpdir/u.pid"));
@@ -101,7 +96,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 	chomp($to_kill{u1} = slurp("$tmpdir/u.pid"));
 	like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file');
 
-	chomp($pid2 = readline($unix_req->($u1, 'GET /pid')));
+	chomp($pid2 = readline(unix_start($u1, 'GET /pid')));
 	like($pid2, qr/\A\d+\z/, 'worker running');
 
 	ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon');
diff --git a/t/integration.t b/t/integration.t
index 939dc24..b33e3c3 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -70,7 +70,7 @@ EOM
 my ($c, $status, $hdr);
 
 # response header tests
-$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
 my $orig_200_status = $status;
@@ -89,7 +89,7 @@ SKIP: { # Date header check
 };
 
 
-$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0');
+$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([ grep(/^x-r3: /, @$hdr) ],
 	[ 'x-r3: a', 'x-r3: b', 'x-r3: c' ],
@@ -97,7 +97,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ],
 
 SKIP: {
 	eval { require JSON::PP } or skip "JSON::PP missing: $@", 1;
-	my $c = start_req($srv, 'GET /env_dump');
+	my $c = tcp_start($srv, 'GET /env_dump');
 	my $json = do { local $/; readline($c) };
 	unlike($json, qr/^Connection: /smi, 'no connection header for 0.9');
 	unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9');
@@ -107,17 +107,17 @@ SKIP: {
 }
 
 # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com>
-$c = start_req($srv, 'GET /nil-header-value HTTP/1.0');
+$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
 ($status, $hdr) = slurp_hdr($c);
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
 if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
-	$c = start_req($srv, 'POST /tweak-status-code HTTP/1.0');
+	$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked');
 
-	$c = start_req($srv, 'POST /restore-status-code HTTP/1.0');
+	$c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	is($status, $orig_200_status, 'original status restored');
 }
@@ -130,12 +130,12 @@ SKIP: {
 }
 
 if ('bad requests') {
-	$c = start_req($srv, 'GET /env_dump HTTP/1/1');
+	$c = tcp_start($srv, 'GET /env_dump HTTP/1/1');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request');
 
-	$c = tcp_connect($srv);
-	print $c 'GET /';
+	$c = tcp_start($srv);
+	print $c 'GET /';;
 	my $buf = join('', (0..9), 'ab');
 	for (0..1023) { print $c $buf }
 	print $c " HTTP/1.0\r\n\r\n";
@@ -143,7 +143,7 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on REQUEST_PATH > (12 * 1024)');
 
-	$c = tcp_connect($srv);
+	$c = tcp_start($srv);
 	print $c 'GET /hello-world?a';
 	$buf = join('', (0..9));
 	for (0..1023) { print $c $buf }
@@ -152,7 +152,7 @@ if ('bad requests') {
 	like($status, qr!\AHTTP/1\.[01] 414 \b!,
 		'414 on QUERY_STRING > (10 * 1024)');
 
-	$c = tcp_connect($srv);
+	$c = tcp_start($srv);
 	print $c 'GET /hello-world#a';
 	$buf = join('', (0..9), 'a'..'f');
 	for (0..63) { print $c $buf }
@@ -173,7 +173,7 @@ SKIP: {
 	my $ck_hash = sub {
 		my ($sub, $path, %opt) = @_;
 		seek($rh, 0, SEEK_SET);
-		$c = tcp_connect($srv);
+		$c = tcp_start($srv);
 		$c->autoflush(0);
 		$PUT{$sub}->($rh, $c, $path, %opt);
 		$c->flush or die $!;
@@ -235,11 +235,11 @@ EOM
 	$wpid =~ s/\Apid=// or die;
 	ok(CORE::kill(0, $wpid), 'worker PID retrieved');
 
-	$c = start_req($srv, $req);
+	$c = tcp_start($srv, $req);
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds');
 
-	$c = start_req($srv, 'GET /xxxxxx HTTP/1.0');
+	$c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
 	like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails');
 }
diff --git a/t/lib.perl b/t/lib.perl
index 315ef2d..1d6e78d 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -10,8 +10,8 @@ use IO::Socket::INET;
 use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
-our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port start_req which spawn check_stderr);
+our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
+	SEEK_SET tcp_host_port which spawn check_stderr unix_start);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
@@ -55,26 +55,28 @@ sub tcp_host_port {
 	}
 }
 
-sub tcp_connect {
-	my ($dest, %opt) = @_;
-	my $addr = tcp_host_port($dest);
-	my $s = ref($dest)->new(
+sub unix_start ($@) {
+	my ($dst, @req) = @_;
+	my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or
+		BAIL_OUT "unix connect $dst: $!";
+	$s->autoflush(1);
+	print $s @req, "\r\n\r\n" if @req;
+	$s;
+}
+
+sub tcp_start ($@) {
+	my ($dst, @req) = @_;
+	my $addr = tcp_host_port($dst);
+	my $s = ref($dst)->new(
 		Proto => 'tcp',
 		Type => SOCK_STREAM,
 		PeerAddr => $addr,
-		%opt,
 	) or BAIL_OUT "failed to connect to $addr: $!";
 	$s->autoflush(1);
+	print $s @req, "\r\n\r\n" if @req;
 	$s;
 }
 
-sub start_req {
-	my ($srv, @req) = @_;
-	my $c = tcp_connect($srv);
-	print $c @req, "\r\n\r\n";
-	$c;
-}
-
 sub slurp {
 	open my $fh, '<', $_[0];
 	local $/;

[-- Attachment #16: 0015-port-t9000-preread-input.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 3856 bytes --]

From 1b8840d8d13491eecd2fa92e06f73c65eadd33ba Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:44 +0000
Subject: [PATCH 15/23] port t9000-preread-input.sh to Perl 5

Stuffing it into t/integration.t for now so we can save on
startup costs.
---
 t/integration.t          | 32 ++++++++++++++++++++++++---
 t/lib.perl               |  2 +-
 t/preread_input.ru       |  4 +---
 t/t9000-preread-input.sh | 48 ----------------------------------------
 4 files changed, 31 insertions(+), 55 deletions(-)
 delete mode 100755 t/t9000-preread-input.sh

diff --git a/t/integration.t b/t/integration.t
index b33e3c3..f5afd5d 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -7,8 +7,8 @@
 
 use v5.14; BEGIN { require './t/lib.perl' };
 use autodie;
-my $srv = tcp_server();
-my $host_port = tcp_host_port($srv);
+our $srv = tcp_server();
+our $host_port = tcp_host_port($srv);
 my $t0 = time;
 my $conf = "$tmpdir/u.conf.rb";
 open my $conf_fh, '>', $conf;
@@ -209,8 +209,34 @@ SKIP: {
 	seek($rh, 0, SEEK_SET);
 	$copt->{0} = $rh;
 	$do_curl->('-T-');
-}
 
+	diag 'testing Unicorn::PrereadInput...';
+	local $srv = tcp_server();
+	local $host_port = tcp_host_port($srv);
+	check_stderr;
+	truncate($errfh, 0);
+
+	my $pri = unicorn(qw(-E none t/preread_input.ru), { 3 => $srv });
+	$url = "http://$host_port/";
+
+	$do_curl->(qw(-T t/random_blob));
+	seek($rh, 0, SEEK_SET);
+	$copt->{0} = $rh;
+	$do_curl->('-T-');
+
+	my @pr_err = slurp("$tmpdir/err.log");
+	is(scalar(grep(/app dispatch:/, @pr_err)), 2, 'app dispatched twice');
+
+	# abort a chunked request by blocking curl on a FIFO:
+	$c = tcp_start($srv, "PUT / HTTP/1.1\r\nTransfer-Encoding: chunked");
+	close $c;
+	@pr_err = slurp("$tmpdir/err.log");
+	is(scalar(grep(/app dispatch:/, @pr_err)), 2,
+			'app did not dispatch on aborted request');
+	undef $pri;
+	check_stderr;
+	diag 'Unicorn::PrereadInput middleware tests done';
+}
 
 # ... more stuff here
 
diff --git a/t/lib.perl b/t/lib.perl
index 1d6e78d..b6148cf 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -79,7 +79,7 @@ sub tcp_start ($@) {
 
 sub slurp {
 	open my $fh, '<', $_[0];
-	local $/;
+	local $/ if !wantarray;
 	readline($fh);
 }
 
diff --git a/t/preread_input.ru b/t/preread_input.ru
index 79685c4..f0a1748 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,8 +1,6 @@
 #\-E none
 require 'digest/sha1'
 require 'unicorn/preread_input'
-use Rack::ContentLength
-use Rack::ContentType, "text/plain"
 use Unicorn::PrereadInput
 nr = 0
 run lambda { |env|
@@ -13,5 +11,5 @@
     dig.update(buf)
   end
 
-  [ 200, {}, [ "#{dig.hexdigest}\n" ] ]
+  [ 200, {}, [ dig.hexdigest ] ]
 }
diff --git a/t/t9000-preread-input.sh b/t/t9000-preread-input.sh
deleted file mode 100755
index d6c73ab..0000000
--- a/t/t9000-preread-input.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 9 "PrereadInput middleware tests"
-
-t_begin "setup and start" && {
-	random_blob_sha1=$(rsha1 < random_blob)
-	unicorn_setup
-	unicorn  -D -c $unicorn_config preread_input.ru
-	unicorn_wait_start
-}
-
-t_begin "single identity request" && {
-	curl -sSf -T random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
-	test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "single chunked request" && {
-	curl -sSf -T- < random_blob http://$listen/ > $tmp
-}
-
-t_begin "sha1 matches" && {
-	test x"$(cat $tmp)" = x"$random_blob_sha1"
-}
-
-t_begin "app only dispatched twice" && {
-	test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "aborted chunked request" && {
-	rm -f $tmp
-	curl -sSf -T- < $fifo http://$listen/ > $tmp &
-	curl_pid=$!
-	kill -9 $curl_pid
-	wait
-}
-
-t_begin "app only dispatched twice" && {
-	test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )"
-}
-
-t_begin "killing succeeds" && {
-	kill -QUIT $unicorn_pid
-}
-
-t_done

[-- Attachment #17: 0016-port-t-t0116-client_body_buffer_size.sh-to-Perl-5.patch --]
[-- Type: text/x-diff, Size: 8861 bytes --]

From e9593301044f305d4a0e074f77eea35015ca0ec4 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:45 +0000
Subject: [PATCH 16/23] port t/t0116-client_body_buffer_size.sh to Perl 5

While I'm fine with depending on curl for certain things,
there's no need for it here since unicorn has had lazy
rack.input for over a decade, at this point.
---
 t/active-unix-socket.t                     |  1 +
 t/{t0116.ru => client_body_buffer_size.ru} |  2 -
 t/client_body_buffer_size.t                | 83 ++++++++++++++++++++++
 t/integration.t                            | 10 ---
 t/lib.perl                                 | 12 +++-
 t/t0116-client_body_buffer_size.sh         | 80 ---------------------
 6 files changed, 95 insertions(+), 93 deletions(-)
 rename t/{t0116.ru => client_body_buffer_size.ru} (82%)
 create mode 100644 t/client_body_buffer_size.t
 delete mode 100755 t/t0116-client_body_buffer_size.sh

diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t
index 8723137..4e11837 100644
--- a/t/active-unix-socket.t
+++ b/t/active-unix-socket.t
@@ -109,4 +109,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure');
 }
 
 check_stderr;
+undef $tmpdir;
 done_testing;
diff --git a/t/t0116.ru b/t/client_body_buffer_size.ru
similarity index 82%
rename from t/t0116.ru
rename to t/client_body_buffer_size.ru
index fab5fce..44161a5 100644
--- a/t/t0116.ru
+++ b/t/client_body_buffer_size.ru
@@ -1,6 +1,4 @@
 #\ -E none
-use Rack::ContentLength
-use Rack::ContentType, 'text/plain'
 app = lambda do |env|
   input = env['rack.input']
   case env["PATH_INFO"]
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
new file mode 100644
index 0000000..b1a99f3
--- /dev/null
+++ b/t/client_body_buffer_size.t
@@ -0,0 +1,83 @@
+#!perl -w
+# Copyright (C) unicorn hackers <unicorn-public@yhbt.net>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+
+use v5.14; BEGIN { require './t/lib.perl' };
+use autodie;
+my $uconf = "$tmpdir/u.conf.rb";
+
+open my $conf_fh, '>', $uconf;
+$conf_fh->autoflush(1);
+print $conf_fh <<EOM;
+client_body_buffer_size 0
+EOM
+my $srv = tcp_server();
+my $host_port = tcp_host_port($srv);
+my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf);
+my $ar = unicorn(@uarg, { 3 => $srv });
+my ($c, $status, $hdr);
+my $mem_class = 'StringIO';
+my $fs_class = 'Unicorn::TmpIO';
+
+$c = tcp_start($srv, "PUT /input_class HTTP/1.0\r\nContent-Length: 0");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'zero-byte file is StringIO');
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+print $c '.';
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 byte file is filesystem-backed');
+
+
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
+seek($conf_fh, 0, SEEK_SET);
+truncate($conf_fh, 0);
+print $conf_fh <<EOM;
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+$ar->do_kill('HUP');
+open my $fifo_fh, '<', $fifo;
+like(my $wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+	'reloaded w/ default client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, 'class for a 1 byte file is memory-backed');
+
+
+my $one_meg = 1024 ** 2;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte file is FS-backed');
+
+# reload with bigger client_body_buffer_size
+say $conf_fh "client_body_buffer_size $one_meg";
+$ar->do_kill('HUP');
+open $fifo_fh, '<', $fifo;
+like($wpid = readline($fifo_fh), qr/\Apid=\d+\z/a ,
+	'reloaded w/ bigger client_body_buffer_size');
+
+
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $mem_class, '1 megabyte file is now memory-backed');
+
+my $too_big = $one_meg + 1;
+$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $too_big");
+($status, $hdr) = slurp_hdr($c);
+like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid');
+is(readline($c), $fs_class, '1 megabyte + 1 byte file is FS-backed');
+
+
+undef $ar;
+check_stderr;
+undef $tmpdir;
+done_testing;
diff --git a/t/integration.t b/t/integration.t
index f5afd5d..855c260 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -15,16 +15,6 @@ open my $conf_fh, '>', $conf;
 $conf_fh->autoflush(1);
 my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');
-END { diag slurp("$tmpdir/err.log") if $tmpdir };
-sub slurp_hdr {
-	my ($c) = @_;
-	local $/ = "\r\n\r\n"; # affects both readline+chomp
-	chomp(my $hdr = readline($c));
-	my ($status, @hdr) = split(/\r\n/, $hdr);
-	diag explain([ $status, \@hdr ]) if $ENV{V};
-	($status, \@hdr);
-}
-
 my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
diff --git a/t/lib.perl b/t/lib.perl
index b6148cf..2685c3b 100644
--- a/t/lib.perl
+++ b/t/lib.perl
@@ -11,11 +11,12 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD);
 use File::Temp 0.19 (); # 0.19 for ->newdir
 our ($tmpdir, $errfh);
 our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh
-	SEEK_SET tcp_host_port which spawn check_stderr unix_start);
+	SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr);
 
 my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1);
 open($errfh, '>>', "$tmpdir/err.log");
+END { diag slurp("$tmpdir/err.log") if $tmpdir };
 
 sub check_stderr () {
 	my @log = slurp("$tmpdir/err.log");
@@ -25,6 +26,15 @@ sub check_stderr () {
 	is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr');
 }
 
+sub slurp_hdr {
+	my ($c) = @_;
+	local $/ = "\r\n\r\n"; # affects both readline+chomp
+	chomp(my $hdr = readline($c));
+	my ($status, @hdr) = split(/\r\n/, $hdr);
+	diag explain([ $status, \@hdr ]) if $ENV{V};
+	($status, \@hdr);
+}
+
 sub tcp_server {
 	my %opt = (
 		ReuseAddr => 1,
diff --git a/t/t0116-client_body_buffer_size.sh b/t/t0116-client_body_buffer_size.sh
deleted file mode 100755
index c9e17c7..0000000
--- a/t/t0116-client_body_buffer_size.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/bin/sh
-. ./test-lib.sh
-t_plan 12 "client_body_buffer_size settings"
-
-t_begin "setup and start" && {
-	unicorn_setup
-	rtmpfiles unicorn_config_tmp one_meg
-	dd if=/dev/zero bs=1M count=1 of=$one_meg
-	cat >> $unicorn_config <<EOF
-after_fork do |server, worker|
-  File.open("$fifo", "wb") { |fp| fp.syswrite "START" }
-end
-EOF
-	cat $unicorn_config > $unicorn_config_tmp
-	echo client_body_buffer_size 0 >> $unicorn_config
-	unicorn -D -c $unicorn_config t0116.ru
-	unicorn_wait_start
-	fs_class=Unicorn::TmpIO
-	mem_class=StringIO
-
-	test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a zero-byte file should be StringIO" && {
-	> $tmp
-	test xStringIO = x"$(curl -T $tmp -sSf http://$listen/input_class)"
-}
-
-t_begin "class for a 1 byte file should be filesystem-backed" && {
-	echo > $tmp
-	test x$fs_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "reload with default client_body_buffer_size" && {
-	mv $unicorn_config_tmp $unicorn_config
-	kill -HUP $unicorn_pid
-	test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "class for a 1 byte file should be memory-backed" && {
-	echo > $tmp
-	test x$mem_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)"
-}
-
-t_begin "class for a random blob file should be filesystem-backed" && {
-	resp="$(curl -T random_blob -sSf http://$listen/tmp_class)"
-	test x$fs_class = x"$resp"
-}
-
-t_begin "one megabyte file should be filesystem-backed" && {
-	resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
-	test x$fs_class = x"$resp"
-}
-
-t_begin "reload with a big client_body_buffer_size" && {
-	echo "client_body_buffer_size(1024 * 1024)" >> $unicorn_config
-	kill -HUP $unicorn_pid
-	test x"$(cat $fifo)" = xSTART
-}
-
-t_begin "one megabyte file should be memory-backed" && {
-	resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
-	test x$mem_class = x"$resp"
-}
-
-t_begin "one megabyte + 1 byte file should be filesystem-backed" && {
-	echo >> $one_meg
-	resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)"
-	test x$fs_class = x"$resp"
-}
-
-t_begin "killing succeeds" && {
-	kill $unicorn_pid
-}
-
-t_begin "check stderr" && {
-	check_stderr
-}
-
-t_done

[-- Attachment #18: 0017-tests-get-rid-of-sha1sum.rb-and-rsha1-sh-function.patch --]
[-- Type: text/x-diff, Size: 1255 bytes --]

From b47912160f2336dde3901e588cc23fb2c2f8d9dc Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:46 +0000
Subject: [PATCH 17/23] tests: get rid of sha1sum.rb and rsha1() sh function

These are no longer needed since Perl has long included
Digest::SHA
---
 t/bin/sha1sum.rb | 17 -----------------
 t/test-lib.sh    |  4 ----
 2 files changed, 21 deletions(-)
 delete mode 100755 t/bin/sha1sum.rb

diff --git a/t/bin/sha1sum.rb b/t/bin/sha1sum.rb
deleted file mode 100755
index 53d68ce..0000000
--- a/t/bin/sha1sum.rb
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/usr/bin/env ruby
-# -*- encoding: binary -*-
-# Reads from stdin and outputs the SHA1 hex digest of the input
-
-require 'digest/sha1'
-$stdout.sync = $stderr.sync = true
-$stdout.binmode
-$stdin.binmode
-bs = 16384
-digest = Digest::SHA1.new
-if buf = $stdin.read(bs)
-  begin
-    digest.update(buf)
-  end while $stdin.read(bs, buf)
-end
-
-$stdout.syswrite("#{digest.hexdigest}\n")
diff --git a/t/test-lib.sh b/t/test-lib.sh
index e70d0c6..8613144 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -123,7 +123,3 @@ unicorn_wait_start () {
 	# no need to play tricks with FIFOs since we got "ready_pipe" now
 	unicorn_pid=$(cat $pid)
 }
-
-rsha1 () {
-	sha1sum.rb
-}

[-- Attachment #19: 0018-early_hints-supports-Rack-3-array-headers.patch --]
[-- Type: text/x-diff, Size: 4606 bytes --]

From 6ad9f4b54ee16ffecea7e16b710552b45db33a16 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:47 +0000
Subject: [PATCH 18/23] early_hints supports Rack 3 array headers

We can hoist out append_headers into a new method and use it in
both e103_response_write and http_response_write.

t/integration.t now tests early_hints with both possible
values of check_client_connection.
---
 t/integration.ru |  7 +++++++
 t/integration.t  | 47 ++++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 49 insertions(+), 5 deletions(-)

diff --git a/t/integration.ru b/t/integration.ru
index edc408c..dab384d 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -5,6 +5,11 @@
 # this goes for t/integration.t  We'll try to put as many tests
 # in here as possible to avoid startup overhead of Ruby.
 
+def early_hints(env, val)
+  env['rack.early_hints'].call('link' => val) # val may be ary or string
+  [ 200, {}, [ val.class.to_s ] ]
+end
+
 $orig_rack_200 = nil
 def tweak_status_code
   $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200]
@@ -81,6 +86,8 @@ def rack_input_tests(env)
     when '/env_dump'; [ 200, {}, [ env_dump(env) ] ]
     when '/write_on_close'; write_on_close
     when '/pid'; [ 200, {}, [ "#$$\n" ] ]
+    when '/early_hints_rack2'; early_hints(env, "r\n2")
+    when '/early_hints_rack3'; early_hints(env, %w(r 3))
     else '/'; [ 200, {}, [ env_dump(env) ] ]
     end # case PATH_INFO (GET)
   when 'POST'
diff --git a/t/integration.t b/t/integration.t
index 855c260..8433497 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -13,8 +13,16 @@ my $t0 = time;
 my $conf = "$tmpdir/u.conf.rb";
 open my $conf_fh, '>', $conf;
 $conf_fh->autoflush(1);
+my $u1 = "$tmpdir/u1";
+print $conf_fh <<EOM;
+early_hints true
+listen "$u1"
+listen "$host_port" # TODO: remove this requirement for SIGHUP
+EOM
 my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');
+my $fifo = "$tmpdir/fifo";
+POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
 my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
@@ -102,6 +110,26 @@ $c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0');
 is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '],
 	'nil header value accepted for broken apps') or diag(explain($hdr));
 
+my $ck_early_hints = sub {
+	my ($note) = @_;
+	$c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 2 value');
+	is_deeply(['link: r', 'link: 2'], $hdr, 'rack 2 hints match '.$note);
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+	is(readline($c), 'String', 'early hints used a String for rack 2');
+
+	$c = unix_start($u1, 'GET /early_hints_rack3 HTTP/1.0');
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 3');
+	is_deeply(['link: r', 'link: 3'], $hdr, 'rack 3 hints match '.$note);
+	($status, $hdr) = slurp_hdr($c);
+	like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards');
+	is(readline($c), 'Array', 'early hints used a String for rack 3');
+};
+$ck_early_hints->('ccc off'); # we'll retest later
+
 if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') {
 	$c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0');
 	($status, $hdr) = slurp_hdr($c);
@@ -154,6 +182,7 @@ if ('bad requests') {
 # input tests
 my ($blob_size, $blob_hash);
 SKIP: {
+	skip 'SKIP_EXPENSIVE on', 1 if $ENV{SKIP_EXPENSIVE};
 	CORE::open(my $rh, '<', 't/random_blob') or
 		skip "t/random_blob not generated $!", 1;
 	$blob_size = -s $rh;
@@ -232,16 +261,24 @@ SKIP: {
 
 # SIGHUP-able stuff goes here
 
+if ('check_client_connection') {
+	print $conf_fh <<EOM; # appending to existing
+check_client_connection true
+after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
+EOM
+	$ar->do_kill('HUP');
+	open my $fifo_fh, '<', $fifo;
+	my $wpid = readline($fifo_fh);
+	like($wpid, qr/\Apid=\d+\z/a , 'new worker ready');
+	$ck_early_hints->('ccc on');
+}
+
 if ('max_header_len internal API') {
 	undef $c;
 	my $req = 'GET / HTTP/1.0';
 	my $len = length($req."\r\n\r\n");
-	my $fifo = "$tmpdir/fifo";
-	POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
-	print $conf_fh <<EOM;
+	print $conf_fh <<EOM; # appending to existing
 Unicorn::HttpParser.max_header_len = $len
-listen "$host_port" # TODO: remove this requirement for SIGHUP
-after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
 EOM
 	$ar->do_kill('HUP');
 	open my $fifo_fh, '<', $fifo;

[-- Attachment #20: 0019-test_server-drop-early_hints-test.patch --]
[-- Type: text/x-diff, Size: 1737 bytes --]

From 3e6bc9fb589fd88469349a38a77704c3333623e0 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:48 +0000
Subject: [PATCH 19/23] test_server: drop early_hints test

t/integration.t already is more complete in that it tests
both Rack 2 and 3 along with both possible values of
check_client_connection.
---
 test/unit/test_server.rb | 31 -------------------------------
 1 file changed, 31 deletions(-)

diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index fe98fcc..0a710d1 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -23,17 +23,6 @@ def call(env)
   end
 end
 
-class TestEarlyHintsHandler
-  def call(env)
-    while env['rack.input'].read(4096)
-    end
-    env['rack.early_hints'].call(
-      "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload"
-    )
-    [200, { 'content-type' => 'text/plain' }, ['hello!\n']]
-  end
-end
-
 class TestRackAfterReply
   def initialize
     @called = false
@@ -112,26 +101,6 @@ def test_preload_app_config
     tmp.close!
   end
 
-  def test_early_hints
-    teardown
-    redirect_test_io do
-      @server = HttpServer.new(TestEarlyHintsHandler.new,
-                               :listeners => [ "127.0.0.1:#@port"],
-                               :early_hints => true)
-      @server.start
-    end
-
-    sock = tcp_socket('127.0.0.1', @port)
-    sock.syswrite("GET / HTTP/1.0\r\n\r\n")
-
-    responses = sock.read(4096)
-    assert_match %r{\AHTTP/1.[01] 103\b}, responses
-    assert_match %r{^Link: </style\.css>}, responses
-    assert_match %r{^Link: </script\.js>}, responses
-
-    assert_match %r{^HTTP/1.[01] 200\b}, responses
-  end
-
   def test_after_reply
     teardown
 

[-- Attachment #21: 0020-t-integration.t-switch-PUT-tests-to-MD5-reuse-buffer.patch --]
[-- Type: text/x-diff, Size: 3740 bytes --]

From cb826915cdd1881cbcfc1fb4e645d26244dfda71 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:49 +0000
Subject: [PATCH 20/23] t/integration.t: switch PUT tests to MD5, reuse buffers

MD5 is faster, and these tests aren't meant to be secure,
they're just for checking for data corruption.
Furthermore, Content-MD5 is a supported HTTP trailer and
we can verify that here to obsolete other tests.

Furthermore, we can reuse buffers on env['rack.input'].read
calls to avoid malloc(3) and GC overhead.

Combined, these give roughly a 3% speedup for t/integration.t
on my system.
---
 t/integration.ru   | 20 +++++++++++++++-----
 t/integration.t    |  5 ++---
 t/preread_input.ru | 17 ++++++++++++-----
 3 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/t/integration.ru b/t/integration.ru
index dab384d..086126a 100644
--- a/t/integration.ru
+++ b/t/integration.ru
@@ -55,8 +55,8 @@ def env_dump(env)
 def rack_input_tests(env)
   return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT']
   cap = 16384
-  require 'digest/sha1'
-  digest = Digest::SHA1.new
+  require 'digest/md5'
+  dig = Digest::MD5.new
   input = env['rack.input']
   case env['PATH_INFO']
   when '/rack_input/size_first'; input.size
@@ -68,11 +68,21 @@ def rack_input_tests(env)
   if buf = input.read(rand(cap))
     begin
       raise "#{buf.size} > #{cap}" if buf.size > cap
-      digest.update(buf)
+      dig.update(buf)
     end while input.read(rand(cap), buf)
+    buf.clear # remove this call if Ruby ever gets escape analysis
   end
-  [ 200, {'content-length' => '40', 'content-type' => 'text/plain'},
-    [ digest.hexdigest ] ]
+  h = { 'content-type' => 'text/plain' }
+  if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+    cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+    cmd5_bin = cmd5_b64.unpack('m')[0]
+    if cmd5_bin != dig.digest
+      h['content-length'] = cmd5_b64.size.to_s
+      return [ 500, h, [ cmd5_b64 ] ]
+    end
+  end
+  h['content-length'] = '32'
+  [ 200, h, [ dig.hexdigest ] ]
 end
 
 run(lambda do |env|
diff --git a/t/integration.t b/t/integration.t
index 8433497..38a9675 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -27,7 +27,6 @@ my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
 		my $bs = $opt{bs} // 16384;
-		require Digest::MD5;
 		my $dig = Digest::MD5->new;
 		print $out <<EOM;
 PUT $path HTTP/1.1\r
@@ -186,8 +185,8 @@ SKIP: {
 	CORE::open(my $rh, '<', 't/random_blob') or
 		skip "t/random_blob not generated $!", 1;
 	$blob_size = -s $rh;
-	require Digest::SHA;
-	$blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest;
+	require Digest::MD5;
+	$blob_hash = Digest::MD5->new->addfile($rh)->hexdigest;
 
 	my $ck_hash = sub {
 		my ($sub, $path, %opt) = @_;
diff --git a/t/preread_input.ru b/t/preread_input.ru
index f0a1748..18af221 100644
--- a/t/preread_input.ru
+++ b/t/preread_input.ru
@@ -1,15 +1,22 @@
 #\-E none
-require 'digest/sha1'
+require 'digest/md5'
 require 'unicorn/preread_input'
 use Unicorn::PrereadInput
 nr = 0
 run lambda { |env|
   $stderr.write "app dispatch: #{nr += 1}\n"
   input = env["rack.input"]
-  dig = Digest::SHA1.new
-  while buf = input.read(16384)
-    dig.update(buf)
+  dig = Digest::MD5.new
+  if buf = input.read(16384)
+    begin
+      dig.update(buf)
+    end while input.read(16384, buf)
+    buf.clear # remove this call if Ruby ever gets escape analysis
+  end
+  if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i
+    cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']]
+    cmd5_bin = cmd5_b64.unpack('m')[0]
+    return [500, {}, [ cmd5_b64 ] ] if cmd5_bin != dig.digest
   end
-
   [ 200, {}, [ dig.hexdigest ] ]
 }

[-- Attachment #22: 0021-tests-move-test_upload.rb-tests-to-t-integration.t.patch --]
[-- Type: text/x-diff, Size: 12280 bytes --]

From 181e4b5b6339fc5e9c3ad7d3690b736f6bd038aa Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:50 +0000
Subject: [PATCH 21/23] tests: move test_upload.rb tests to t/integration.t

The overread tests are ported over, and checksumming alone
is enough to guard against data corruption.

Randomizing the size of `read' calls on the client side will
shake out any boundary bugs on the server side.
---
 t/integration.t          |  32 ++++-
 test/unit/test_upload.rb | 301 ---------------------------------------
 2 files changed, 27 insertions(+), 306 deletions(-)
 delete mode 100644 test/unit/test_upload.rb

diff --git a/t/integration.t b/t/integration.t
index 38a9675..a568758 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -26,7 +26,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
 my %PUT = (
 	chunked_md5 => sub {
 		my ($in, $out, $path, %opt) = @_;
-		my $bs = $opt{bs} // 16384;
 		my $dig = Digest::MD5->new;
 		print $out <<EOM;
 PUT $path HTTP/1.1\r
@@ -36,7 +35,7 @@ Trailer: Content-MD5\r
 EOM
 		my ($buf, $r);
 		while (1) {
-			$r = read($in, $buf, $bs);
+			$r = read($in, $buf, 999 + int(rand(0xffff)));
 			last if $r == 0;
 			printf $out "%x\r\n", length($buf);
 			print $out $buf, "\r\n";
@@ -46,15 +45,15 @@ EOM
 	},
 	identity => sub {
 		my ($in, $out, $path, %opt) = @_;
-		my $bs = $opt{bs} // 16384;
 		my $clen = $opt{-s} // -s $in;
 		print $out <<EOM;
 PUT $path HTTP/1.0\r
 Content-Length: $clen\r
 \r
 EOM
-		my ($buf, $r, $len);
+		my ($buf, $r, $len, $bs);
 		while ($clen) {
+			$bs = 999 + int(rand(0xffff));
 			$len = $clen > $bs ? $bs : $clen;
 			$r = read($in, $buf, $len);
 			die 'premature EOF' if $r == 0;
@@ -192,8 +191,10 @@ SKIP: {
 		my ($sub, $path, %opt) = @_;
 		seek($rh, 0, SEEK_SET);
 		$c = tcp_start($srv);
-		$c->autoflush(0);
+		$c->autoflush($opt{sync} // 0);
 		$PUT{$sub}->($rh, $c, $path, %opt);
+		defined($opt{overwrite}) and
+			print { $c } ('x' x $opt{overwrite});
 		$c->flush or die $!;
 		($status, $hdr) = slurp_hdr($c);
 		is(readline($c), $blob_hash, "$sub $path");
@@ -205,6 +206,27 @@ SKIP: {
 	$ck_hash->('chunked_md5', '/rack_input/size_first');
 	$ck_hash->('chunked_md5', '/rack_input/rewind_first');
 
+	$ck_hash->('identity', '/rack_input', -s => $blob_size, sync => 1);
+	$ck_hash->('chunked_md5', '/rack_input', sync => 1);
+
+	# ensure small overwrites don't get checksummed
+	$ck_hash->('identity', '/rack_input', -s => $blob_size,
+			overwrite => 1); # one extra byte
+
+	# excessive overwrite truncated
+	$c = tcp_start($srv);
+	$c->autoflush(0);
+	print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 1\r\n\r\n";
+	if (1) {
+		local $SIG{PIPE} = 'IGNORE';
+		my $buf = "\0" x 8192;
+		my $n = 0;
+		my $end = time + 5;
+		$! = 0;
+		while (print $c $buf and time < $end) { ++$n }
+		ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time;
+	}
+	undef $c;
 
 	$curl // skip 'no curl found in PATH', 1;
 
diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb
deleted file mode 100644
index 76e6c1c..0000000
--- a/test/unit/test_upload.rb
+++ /dev/null
@@ -1,301 +0,0 @@
-# -*- encoding: binary -*-
-
-# Copyright (c) 2009 Eric Wong
-require './test/test_helper'
-require 'digest/md5'
-
-include Unicorn
-
-class UploadTest < Test::Unit::TestCase
-
-  def setup
-    @addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1'
-    @port = unused_port
-    @hdr = {'Content-Type' => 'text/plain', 'Content-Length' => '0'}
-    @bs = 4096
-    @count = 256
-    @server = nil
-
-    # we want random binary data to test 1.9 encoding-aware IO craziness
-    @random = File.open('/dev/urandom','rb')
-    @sha1 = Digest::SHA1.new
-    @sha1_app = lambda do |env|
-      input = env['rack.input']
-      resp = {}
-
-      @sha1.reset
-      while buf = input.read(@bs)
-        @sha1.update(buf)
-      end
-      resp[:sha1] = @sha1.hexdigest
-
-      # rewind and read again
-      input.rewind
-      @sha1.reset
-      while buf = input.read(@bs)
-        @sha1.update(buf)
-      end
-
-      if resp[:sha1] == @sha1.hexdigest
-        resp[:sysread_read_byte_match] = true
-      end
-
-      if expect_size = env['HTTP_X_EXPECT_SIZE']
-        if expect_size.to_i == input.size
-          resp[:expect_size_match] = true
-        end
-      end
-      resp[:size] = input.size
-      resp[:content_md5] = env['HTTP_CONTENT_MD5']
-
-      [ 200, @hdr.merge({'X-Resp' => resp.inspect}), [] ]
-    end
-  end
-
-  def teardown
-    redirect_test_io { @server.stop(false) } if @server
-    @random.close
-    reset_sig_handlers
-  end
-
-  def test_put
-    start_server(@sha1_app)
-    sock = tcp_socket(@addr, @port)
-    sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
-    @count.times do |i|
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      sock.syswrite(buf)
-    end
-    read = sock.read.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal length, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-  end
-
-  def test_put_content_md5
-    md5 = Digest::MD5.new
-    start_server(@sha1_app)
-    sock = tcp_socket(@addr, @port)
-    sock.syswrite("PUT / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n" \
-                  "Trailer: Content-MD5\r\n\r\n")
-    @count.times do |i|
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      md5.update(buf)
-      sock.syswrite("#{'%x' % buf.size}\r\n")
-      sock.syswrite(buf << "\r\n")
-    end
-    sock.syswrite("0\r\n")
-
-    content_md5 = [ md5.digest! ].pack('m').strip.freeze
-    sock.syswrite("Content-MD5: #{content_md5}\r\n\r\n")
-    read = sock.read.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal length, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-    assert_equal content_md5, resp[:content_md5]
-  end
-
-  def test_put_trickle_small
-    @count, @bs = 2, 128
-    start_server(@sha1_app)
-    assert_equal 256, length
-    sock = tcp_socket(@addr, @port)
-    hdr = "PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n"
-    @count.times do
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      hdr << buf
-      sock.syswrite(hdr)
-      hdr = ''
-      sleep 0.6
-    end
-    read = sock.read.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal length, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-  end
-
-  def test_put_keepalive_truncates_small_overwrite
-    start_server(@sha1_app)
-    sock = tcp_socket(@addr, @port)
-    to_upload = length + 1
-    sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{to_upload}\r\n\r\n")
-    @count.times do
-      buf = @random.sysread(@bs)
-      @sha1.update(buf)
-      sock.syswrite(buf)
-    end
-    sock.syswrite('12345') # write 4 bytes more than we expected
-    @sha1.update('1')
-
-    buf = sock.readpartial(4096)
-    while buf !~ /\r\n\r\n/
-      buf << sock.readpartial(4096)
-    end
-    read = buf.split(/\r\n/)
-    assert_equal "HTTP/1.1 200 OK", read[0]
-    resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, ''))
-    assert_equal to_upload, resp[:size]
-    assert_equal @sha1.hexdigest, resp[:sha1]
-  end
-
-  def test_put_excessive_overwrite_closed
-    tmp = Tempfile.new('overwrite_check')
-    tmp.sync = true
-    start_server(lambda { |env|
-      nr = 0
-      while buf = env['rack.input'].read(65536)
-        nr += buf.size
-      end
-      tmp.write(nr.to_s)
-      [ 200, @hdr, [] ]
-    })
-    sock = tcp_socket(@addr, @port)
-    buf = ' ' * @bs
-    sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n")
-
-    @count.times { sock.syswrite(buf) }
-    assert_raise(Errno::ECONNRESET, Errno::EPIPE) do
-      ::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) }
-    end
-    sock.gets
-    tmp.rewind
-    assert_equal length, tmp.read.to_i
-  end
-
-  # Despite reading numerous articles and inspecting the 1.9.1-p0 C
-  # source, Eric Wong will never trust that we're always handling
-  # encoding-aware IO objects correctly.  Thus this test uses shell
-  # utilities that should always operate on files/sockets on a
-  # byte-level.
-  def test_uncomfortable_with_onenine_encodings
-    # POSIX doesn't require all of these to be present on a system
-    which('curl') or return
-    which('sha1sum') or return
-    which('dd') or return
-
-    start_server(@sha1_app)
-
-    tmp = Tempfile.new('dd_dest')
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=#{@bs}", "count=#{@count}"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
-    assert $?.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-
-    # small StringIO path
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=1024", "count=1"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/`
-    assert $?.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-  end
-
-  def test_chunked_upload_via_curl
-    # POSIX doesn't require all of these to be present on a system
-    which('curl') or return
-    which('sha1sum') or return
-    which('dd') or return
-
-    start_server(@sha1_app)
-
-    tmp = Tempfile.new('dd_dest')
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=#{@bs}", "count=#{@count}"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    cmd = "curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
-           -isSf --no-buffer -T- " \
-          "http://#@addr:#@port/"
-    resp = Tempfile.new('resp')
-    resp.sync = true
-
-    rd, wr = IO.pipe.each do |io|
-      io.sync = io.close_on_exec = true
-    end
-    pid = spawn(*cmd, { 0 => rd, 1 => resp })
-    rd.close
-
-    tmp.rewind
-    @count.times { |i|
-      wr.write(tmp.read(@bs))
-      sleep(rand / 10) if 0 == i % 8
-    }
-    wr.close
-    pid, status = Process.waitpid2(pid)
-
-    resp.rewind
-    resp = resp.read
-    assert status.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-    assert_match(/expect_size_match/, resp)
-  end
-
-  def test_curl_chunked_small
-    # POSIX doesn't require all of these to be present on a system
-    which('curl') or return
-    which('sha1sum') or return
-    which('dd') or return
-
-    start_server(@sha1_app)
-
-    tmp = Tempfile.new('dd_dest')
-    # small StringIO path
-    assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}",
-                        "bs=1024", "count=1"),
-           "dd #@random to #{tmp}")
-    sha1_re = %r!\b([a-f0-9]{40})\b!
-    sha1_out = `sha1sum #{tmp.path}`
-    assert $?.success?, 'sha1sum ran OK'
-
-    assert_match(sha1_re, sha1_out)
-    sha1 = sha1_re.match(sha1_out)[1]
-    resp = `curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \
-            -isSf --no-buffer -T- http://#@addr:#@port/ < #{tmp.path}`
-    assert $?.success?, 'curl ran OK'
-    assert_match(%r!\b#{sha1}\b!, resp)
-    assert_match(/sysread_read_byte_match/, resp)
-    assert_match(/expect_size_match/, resp)
-  end
-
-  private
-
-  def length
-    @bs * @count
-  end
-
-  def start_server(app)
-    redirect_test_io do
-      @server = HttpServer.new(app, :listeners => [ "#{@addr}:#{@port}" ] )
-      @server.start
-    end
-  end
-
-end

[-- Attachment #23: 0022-drop-redundant-IO-close_on_exec-false-calls.patch --]
[-- Type: text/x-diff, Size: 891 bytes --]

From 841b9e756beb1aa00d0f89097a808adcbbf45397 Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:51 +0000
Subject: [PATCH 22/23] drop redundant IO#close_on_exec=false calls

Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
---
 lib/unicorn/http_server.rb | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 348e745..dd92b38 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -472,10 +472,7 @@ def worker_spawn(worker)
 
   def listener_sockets
     listener_fds = {}
-    LISTENERS.each do |sock|
-      sock.close_on_exec = false
-      listener_fds[sock.fileno] = sock
-    end
+    LISTENERS.each { |sock| listener_fds[sock.fileno] = sock }
     listener_fds
   end
 

[-- Attachment #24: 0023-LISTEN_FDS-inherited-sockets-are-immortal-across-SIG.patch --]
[-- Type: text/x-diff, Size: 3229 bytes --]

From 6ff8785c9277c5978e6dc01cb1b3da25d6bae2db Mon Sep 17 00:00:00 2001
From: Eric Wong <BOFH@YHBT.net>
Date: Mon, 5 Jun 2023 10:12:52 +0000
Subject: [PATCH 23/23] LISTEN_FDS-inherited sockets are immortal across SIGHUP

When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.

I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
---
 lib/unicorn/http_server.rb  | 7 ++++++-
 t/client_body_buffer_size.t | 1 -
 t/integration.t             | 1 -
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index dd92b38..f1b4a54 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -77,6 +77,7 @@ def initialize(app, options = {})
     options[:use_defaults] = true
     self.config = Unicorn::Configurator.new(options)
     self.listener_opts = {}
+    @immortal = [] # immortal inherited sockets from systemd
 
     # We use @self_pipe differently in the master and worker processes:
     #
@@ -158,6 +159,7 @@ def listeners=(listeners)
     end
     set_names = listener_names(listeners)
     dead_names.concat(cur_names - set_names).uniq!
+    dead_names -= @immortal.map { |io| sock_name(io) }
 
     LISTENERS.delete_if do |io|
       if dead_names.include?(sock_name(io))
@@ -807,17 +809,20 @@ def inherit_listeners!
     # inherit sockets from parents, they need to be plain Socket objects
     # before they become Kgio::UNIXServer or Kgio::TCPServer
     inherited = ENV['UNICORN_FD'].to_s.split(',')
+    immortal = []
 
     # emulate sd_listen_fds() for systemd
     sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
     if sd_pid.to_i == $$ # n.b. $$ can never be zero
       # 3 = SD_LISTEN_FDS_START
-      inherited.concat((3...(3 + sd_fds.to_i)).to_a)
+      immortal = (3...(3 + sd_fds.to_i)).to_a
+      inherited.concat(immortal)
     end
     # to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
 
     inherited.map! do |fd|
       io = Socket.for_fd(fd.to_i)
+      @immortal << io if immortal.include?(fd)
       io.autoclose = false
       io = server_cast(io)
       set_server_sockopt(io, listener_opts[sock_name(io)])
diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t
index b1a99f3..3067f28 100644
--- a/t/client_body_buffer_size.t
+++ b/t/client_body_buffer_size.t
@@ -36,7 +36,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!";
 seek($conf_fh, 0, SEEK_SET);
 truncate($conf_fh, 0);
 print $conf_fh <<EOM;
-listen "$host_port" # TODO: remove this requirement for SIGHUP
 after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } }
 EOM
 $ar->do_kill('HUP');
diff --git a/t/integration.t b/t/integration.t
index a568758..bb2ab51 100644
--- a/t/integration.t
+++ b/t/integration.t
@@ -17,7 +17,6 @@ my $u1 = "$tmpdir/u1";
 print $conf_fh <<EOM;
 early_hints true
 listen "$u1"
-listen "$host_port" # TODO: remove this requirement for SIGHUP
 EOM
 my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv });
 my $curl = which('curl');

^ permalink raw reply related	[relevance 1%]

* [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1
  2023-06-02  2:45  0%   ` Jeremy Evans
@ 2023-06-05  9:12  5%     ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2023-06-05  9:12 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> We deprecated Rack::Chunked in Rack 3.0 and plan to remove it in Rack
> 3.1. I agree it would be best to deal with this now, I just wasn't sure
> how you wanted to handle it.  Your patch below to deal with it at the
> server level looks good, though it doesn't appear to remove the Chunked
> usage at line 69 of unicorn.rb.  I recommend that also be removed.

OK.  Also added HEAD and STATUS_WITH_NO_ENTITY_BODY checks...

No tests, yet; they'll be in Perl 5.  (I started rewriting a bunch
of tests in Perl5 last year since tests are where Ruby's yearly
breaking changes are most unacceptable to me).

No trailers for responses yet, either; I didn't realize Rack::Chunked
added special support for that in 2020.

I care deeply about trailers in requests, but never used them
for responses.

--------8<-------
Subject: [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1

Rack::Chunked will be gone in Rack 3.1, so provide a
non-middleware fallback which takes advantage of IO#write
supporting multiple arguments in Ruby 2.5+.

We still need to support Ruby 2.4, at least, since Rack 3.0
does.  So a new (GC-unfriendly) Unicorn::WriteSplat module now
exists for Ruby <= 2.4 users.
---
  v2: remove Rack::Chunk load attempt
      fix arity check for Ruby <= 2.4
      update docs + examples
Interdiff:
  diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1
  index d76d40f..b2c5e70 100644
  --- a/Documentation/unicorn.1
  +++ b/Documentation/unicorn.1
  @@ -176,7 +176,7 @@ As of Unicorn 0.94.0, RACK_ENV is exported as a process\-wide environment
   variable as well.  While not current a part of the Rack specification as
   of Rack 1.0.1, this has become a de facto standard in the Rack world.
   .PP
  -Note the Rack::ContentLength and Rack::Chunked middlewares are also
  +Note the Rack::ContentLength middleware is also
   loaded by "deployment" and "development", but no other values of
   RACK_ENV.  If needed, they must be individually specified in the
   RACKUP_FILE, some frameworks do not require them.
  diff --git a/examples/echo.ru b/examples/echo.ru
  index 14908c5..e982180 100644
  --- a/examples/echo.ru
  +++ b/examples/echo.ru
  @@ -19,7 +19,6 @@ def each(&block)
   
   end
   
  -use Rack::Chunked
   run lambda { |env|
     /\A100-continue\z/i =~ env['HTTP_EXPECT'] and return [100, {}, []]
     [ 200, { 'Content-Type' => 'application/octet-stream' },
  diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
  index c339024..afdf680 100644
  --- a/ext/unicorn_http/unicorn_http.rl
  +++ b/ext/unicorn_http/unicorn_http.rl
  @@ -28,11 +28,15 @@ void init_unicorn_httpdate(void);
   #define UH_FL_TO_CLEAR 0x200
   #define UH_FL_RESSTART 0x400 /* for check_client_connection */
   #define UH_FL_HIJACK 0x800
  -#define UH_FL_RES_CHUNK_OK (1U << 12)
  +#define UH_FL_RES_CHUNK_VER (1U << 12)
  +#define UH_FL_RES_CHUNK_METHOD (1U << 13)
   
   /* all of these flags need to be set for keepalive to be supported */
   #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
   
  +/* we can only chunk responses for non-HEAD HTTP/1.1 requests */
  +#define UH_FL_RES_CHUNKABLE (UH_FL_RES_CHUNK_VER | UH_FL_RES_CHUNK_METHOD)
  +
   static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
   
   /* this is only intended for use with Rainbows! */
  @@ -146,6 +150,9 @@ request_method(struct http_parser *hp, const char *ptr, size_t len)
   {
     VALUE v = rb_str_new(ptr, len);
   
  +  if (len != 4 || memcmp(ptr, "HEAD", 4))
  +    HP_FL_SET(hp, RES_CHUNK_METHOD);
  +
     rb_hash_aset(hp->env, g_request_method, v);
   }
   
  @@ -159,7 +166,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
     if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
       /* HTTP/1.1 implies keepalive unless "Connection: close" is set */
       HP_FL_SET(hp, KAVERSION);
  -    HP_FL_SET(hp, RES_CHUNK_OK);
  +    HP_FL_SET(hp, RES_CHUNK_VER);
       v = g_http_11;
     } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
       v = g_http_10;
  @@ -806,9 +813,9 @@ static VALUE HttpParser_keepalive(VALUE self)
   /* :nodoc: */
   static VALUE chunkable_response_p(VALUE self)
   {
  -  struct http_parser *hp = data_get(self);
  +  const struct http_parser *hp = data_get(self);
   
  -  return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse;
  +  return HP_FL_ALL(hp, RES_CHUNKABLE) ? Qtrue : Qfalse;
   }
   
   /**
  diff --git a/lib/unicorn.rb b/lib/unicorn.rb
  index 8b1cda7..b817b77 100644
  --- a/lib/unicorn.rb
  +++ b/lib/unicorn.rb
  @@ -66,7 +66,6 @@ def self.builder(ru, op)
   
         middleware = { # order matters
           ContentLength: nil,
  -        Chunked: nil,
           CommonLogger: [ $stderr ],
           ShowExceptions: nil,
           Lint: nil,
  diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
  index 342dd0b..0ed0ae3 100644
  --- a/lib/unicorn/http_response.rb
  +++ b/lib/unicorn/http_response.rb
  @@ -12,6 +12,12 @@ module Unicorn::HttpResponse
   
     STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ?
                    Rack::Utils::HTTP_STATUS_CODES : {}
  +  STATUS_WITH_NO_ENTITY_BODY = defined?(
  +                 Rack::Utils::STATUS_WITH_NO_ENTITY_BODY) ?
  +                 Rack::Utils::STATUS_WITH_NO_ENTITY_BODY : begin
  +    warn 'Rack::Utils::STATUS_WITH_NO_ENTITY_BODY missing'
  +    {}
  +  end
   
     # internal API, code will always be common-enough-for-even-old-Rack
     def err_response(code, response_start_sent)
  @@ -40,7 +46,7 @@ def http_response_write(socket, status, headers, body,
         code = status.to_i
         msg = STATUS_CODES[code]
         start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
  -      term = false
  +      term = STATUS_WITH_NO_ENTITY_BODY.include?(code) || false
         buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
               "Date: #{httpdate}\r\n" \
               "Connection: close\r\n"
  diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
  index 4ae4c85..c2ba75e 100644
  --- a/lib/unicorn/socket_helper.rb
  +++ b/lib/unicorn/socket_helper.rb
  @@ -15,7 +15,7 @@ def kgio_tryaccept # :nodoc:
       end
     end
   
  -  if IO.instance_method(:write).arity # Ruby <= 2.4
  +  if IO.instance_method(:write).arity == 1 # Ruby <= 2.4
       require 'unicorn/write_splat'
       UNIXClient = Class.new(Kgio::Socket) # :nodoc:
       class UNIXSrv < Kgio::UNIXServer # :nodoc:
  diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
  index cea9791..fe98fcc 100644
  --- a/test/unit/test_server.rb
  +++ b/test/unit/test_server.rb
  @@ -196,7 +196,7 @@ def test_client_shutdown_writes
       # continue to process our request and never hit EOFError on our sock
       sock.shutdown(Socket::SHUT_WR)
       buf = sock.read
  -    assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last
  +    assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
       next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
       assert_equal 'hello!\n', next_client
       lines = File.readlines("test_stderr.#$$.log")

 Documentation/unicorn.1          |  2 +-
 examples/echo.ru                 |  1 -
 ext/unicorn_http/unicorn_http.rl | 18 ++++++++++++++++++
 lib/unicorn.rb                   |  5 ++---
 lib/unicorn/http_response.rb     | 27 ++++++++++++++++++++++++++-
 lib/unicorn/socket_helper.rb     | 18 ++++++++++++++++--
 lib/unicorn/write_splat.rb       |  7 +++++++
 test/unit/test_server.rb         |  2 +-
 8 files changed, 71 insertions(+), 9 deletions(-)
 create mode 100644 lib/unicorn/write_splat.rb

diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1
index d76d40f..b2c5e70 100644
--- a/Documentation/unicorn.1
+++ b/Documentation/unicorn.1
@@ -176,7 +176,7 @@ As of Unicorn 0.94.0, RACK_ENV is exported as a process\-wide environment
 variable as well.  While not current a part of the Rack specification as
 of Rack 1.0.1, this has become a de facto standard in the Rack world.
 .PP
-Note the Rack::ContentLength and Rack::Chunked middlewares are also
+Note the Rack::ContentLength middleware is also
 loaded by "deployment" and "development", but no other values of
 RACK_ENV.  If needed, they must be individually specified in the
 RACKUP_FILE, some frameworks do not require them.
diff --git a/examples/echo.ru b/examples/echo.ru
index 14908c5..e982180 100644
--- a/examples/echo.ru
+++ b/examples/echo.ru
@@ -19,7 +19,6 @@ def each(&block)
 
 end
 
-use Rack::Chunked
 run lambda { |env|
   /\A100-continue\z/i =~ env['HTTP_EXPECT'] and return [100, {}, []]
   [ 200, { 'Content-Type' => 'application/octet-stream' },
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index ba23438..afdf680 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -28,10 +28,15 @@ void init_unicorn_httpdate(void);
 #define UH_FL_TO_CLEAR 0x200
 #define UH_FL_RESSTART 0x400 /* for check_client_connection */
 #define UH_FL_HIJACK 0x800
+#define UH_FL_RES_CHUNK_VER (1U << 12)
+#define UH_FL_RES_CHUNK_METHOD (1U << 13)
 
 /* all of these flags need to be set for keepalive to be supported */
 #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
 
+/* we can only chunk responses for non-HEAD HTTP/1.1 requests */
+#define UH_FL_RES_CHUNKABLE (UH_FL_RES_CHUNK_VER | UH_FL_RES_CHUNK_METHOD)
+
 static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
 
 /* this is only intended for use with Rainbows! */
@@ -145,6 +150,9 @@ request_method(struct http_parser *hp, const char *ptr, size_t len)
 {
   VALUE v = rb_str_new(ptr, len);
 
+  if (len != 4 || memcmp(ptr, "HEAD", 4))
+    HP_FL_SET(hp, RES_CHUNK_METHOD);
+
   rb_hash_aset(hp->env, g_request_method, v);
 }
 
@@ -158,6 +166,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
   if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
     /* HTTP/1.1 implies keepalive unless "Connection: close" is set */
     HP_FL_SET(hp, KAVERSION);
+    HP_FL_SET(hp, RES_CHUNK_VER);
     v = g_http_11;
   } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
     v = g_http_10;
@@ -801,6 +810,14 @@ static VALUE HttpParser_keepalive(VALUE self)
   return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse;
 }
 
+/* :nodoc: */
+static VALUE chunkable_response_p(VALUE self)
+{
+  const struct http_parser *hp = data_get(self);
+
+  return HP_FL_ALL(hp, RES_CHUNKABLE) ? Qtrue : Qfalse;
+}
+
 /**
  * call-seq:
  *    parser.next? => true or false
@@ -981,6 +998,7 @@ void Init_unicorn_http(void)
   rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0);
   rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0);
   rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0);
+  rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0);
   rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0);
   rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
   rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 1a50631..b817b77 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -66,7 +66,6 @@ def self.builder(ru, op)
 
       middleware = { # order matters
         ContentLength: nil,
-        Chunked: nil,
         CommonLogger: [ $stderr ],
         ShowExceptions: nil,
         Lint: nil,
@@ -75,8 +74,8 @@ def self.builder(ru, op)
 
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
-      # and Zbatery both do.  Users accustomed to the Rack::Server default
-      # middlewares will need ContentLength/Chunked middlewares.
+      # does.  Users accustomed to the Rack::Server default
+      # middlewares will need ContentLength middleware.
       case ENV["RACK_ENV"]
       when "development"
       when "deployment"
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 19469b4..0ed0ae3 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -12,6 +12,12 @@ module Unicorn::HttpResponse
 
   STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ?
                  Rack::Utils::HTTP_STATUS_CODES : {}
+  STATUS_WITH_NO_ENTITY_BODY = defined?(
+                 Rack::Utils::STATUS_WITH_NO_ENTITY_BODY) ?
+                 Rack::Utils::STATUS_WITH_NO_ENTITY_BODY : begin
+    warn 'Rack::Utils::STATUS_WITH_NO_ENTITY_BODY missing'
+    {}
+  end
 
   # internal API, code will always be common-enough-for-even-old-Rack
   def err_response(code, response_start_sent)
@@ -35,11 +41,12 @@ def append_header(buf, key, value)
   def http_response_write(socket, status, headers, body,
                           req = Unicorn::HttpRequest.new)
     hijack = nil
-
+    do_chunk = false
     if headers
       code = status.to_i
       msg = STATUS_CODES[code]
       start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
+      term = STATUS_WITH_NO_ENTITY_BODY.include?(code) || false
       buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
             "Date: #{httpdate}\r\n" \
             "Connection: close\r\n"
@@ -47,6 +54,12 @@ def http_response_write(socket, status, headers, body,
         case key
         when %r{\A(?:Date|Connection)\z}i
           next
+        when %r{\AContent-Length\z}i
+          append_header(buf, key, value)
+          term = true
+        when %r{\ATransfer-Encoding\z}i
+          append_header(buf, key, value)
+          term = true if /\bchunked\b/i === value # value may be Array :x
         when "rack.hijack"
           # This should only be hit under Rack >= 1.5, as this was an illegal
           # key in Rack < 1.5
@@ -55,12 +68,24 @@ def http_response_write(socket, status, headers, body,
           append_header(buf, key, value)
         end
       end
+      if !hijack && !term && req.chunkable_response?
+        do_chunk = true
+        buf << "Transfer-Encoding: chunked\r\n".freeze
+      end
       socket.write(buf << "\r\n".freeze)
     end
 
     if hijack
       req.hijacked!
       hijack.call(socket)
+    elsif do_chunk
+      begin
+        body.each do |b|
+          socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze)
+        end
+      ensure
+        socket.write("0\r\n\r\n".freeze)
+      end
     else
       body.each { |chunk| socket.write(chunk) }
     end
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 8a6f6ee..c2ba75e 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc:
     end
   end
 
+  if IO.instance_method(:write).arity == 1 # Ruby <= 2.4
+    require 'unicorn/write_splat'
+    UNIXClient = Class.new(Kgio::Socket) # :nodoc:
+    class UNIXSrv < Kgio::UNIXServer # :nodoc:
+      include Unicorn::WriteSplat
+      def kgio_tryaccept # :nodoc:
+        super(UNIXClient)
+      end
+    end
+    TCPClient.__send__(:include, Unicorn::WriteSplat)
+  else # Ruby 2.5+
+    UNIXSrv = Kgio::UNIXServer
+  end
+
   module SocketHelper
 
     # internal interface
@@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
         end
         old_umask = File.umask(opt[:umask] || 0)
         begin
-          Kgio::UNIXServer.new(address)
+          UNIXSrv.new(address)
         ensure
           File.umask(old_umask)
         end
@@ -203,7 +217,7 @@ def server_cast(sock)
         Socket.unpack_sockaddr_in(sock.getsockname)
         TCPSrv.for_fd(sock.fileno)
       rescue ArgumentError
-        Kgio::UNIXServer.for_fd(sock.fileno)
+        UNIXSrv.for_fd(sock.fileno)
       end
     end
 
diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
new file mode 100644
index 0000000..7e6e363
--- /dev/null
+++ b/lib/unicorn/write_splat.rb
@@ -0,0 +1,7 @@
+# -*- encoding: binary -*-
+# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
+module Unicorn::WriteSplat # :nodoc:
+  def write(*arg) # :nodoc:
+    super(arg.join(''))
+  end
+end
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 98e85ab..fe98fcc 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -196,7 +196,7 @@ def test_client_shutdown_writes
     # continue to process our request and never hit EOFError on our sock
     sock.shutdown(Socket::SHUT_WR)
     buf = sock.read
-    assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last
+    assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last
     next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
     assert_equal 'hello!\n', next_client
     lines = File.readlines("test_stderr.#$$.log")



^ permalink raw reply related	[relevance 5%]

* Re: Rack 3 Compatibility
  2023-06-02  0:00  4% ` Eric Wong
@ 2023-06-02  2:45  0%   ` Jeremy Evans
  2023-06-05  9:12  5%     ` [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1 Eric Wong
  0 siblings, 1 reply; 191+ results
From: Jeremy Evans @ 2023-06-02  2:45 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

On 06/02 12:00, Eric Wong wrote:
> Jeremy Evans <code@jeremyevans.net> wrote:
> > This takes Eric's patch from December 25, 2022, and includes all
> > necessary test fixes to allow Unicorn tests to pass with both
> > Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for
> > newer curl versions and an OpenBSD test fix.
> > 
> > Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3
> > support.  If further fixes are needed, I'm happy to work on them.
> 
> Isn't a chunk replacement needed for Rack 3.1, also?
> I dunno if I missed anything else in Rack 3.x; and don't want to
> make too many releases if we can do 3.0 and 3.1 in one go.

We deprecated Rack::Chunked in Rack 3.0 and plan to remove it in Rack
3.1. I agree it would be best to deal with this now, I just wasn't sure
how you wanted to handle it.  Your patch below to deal with it at the
server level looks good, though it doesn't appear to remove the Chunked
usage at line 69 of unicorn.rb.  I recommend that also be removed.

Thanks,
Jeremy

> -------8<-------
> Subject: [PATCH] chunk unterminated HTTP/1.1 responses
> 
> Rack::Chunked will be gone in Rack 3.1, so provide a
> non-middleware fallback which takes advantage of IO#write
> supporting multiple arguments in Ruby 2.5+.
> 
> We still need to support Ruby 2.4, at least, since Rack 3.0
> does.  So a new (GC-unfriendly) Unicorn::WriteSplat module now
> exists for Ruby <= 2.4 users.
> ---
>  ext/unicorn_http/unicorn_http.rl | 11 +++++++++++
>  lib/unicorn.rb                   |  4 ++--
>  lib/unicorn/http_response.rb     | 21 ++++++++++++++++++++-
>  lib/unicorn/socket_helper.rb     | 18 ++++++++++++++++--
>  lib/unicorn/write_splat.rb       |  7 +++++++
>  test/unit/test_server.rb         |  2 +-
>  6 files changed, 57 insertions(+), 6 deletions(-)
>  create mode 100644 lib/unicorn/write_splat.rb
> 
> diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
> index ba23438..c339024 100644
> --- a/ext/unicorn_http/unicorn_http.rl
> +++ b/ext/unicorn_http/unicorn_http.rl
> @@ -28,6 +28,7 @@ void init_unicorn_httpdate(void);
>  #define UH_FL_TO_CLEAR 0x200
>  #define UH_FL_RESSTART 0x400 /* for check_client_connection */
>  #define UH_FL_HIJACK 0x800
> +#define UH_FL_RES_CHUNK_OK (1U << 12)
>  
>  /* all of these flags need to be set for keepalive to be supported */
>  #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
> @@ -158,6 +159,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
>    if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
>      /* HTTP/1.1 implies keepalive unless "Connection: close" is set */
>      HP_FL_SET(hp, KAVERSION);
> +    HP_FL_SET(hp, RES_CHUNK_OK);
>      v = g_http_11;
>    } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
>      v = g_http_10;
> @@ -801,6 +803,14 @@ static VALUE HttpParser_keepalive(VALUE self)
>    return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse;
>  }
>  
> +/* :nodoc: */
> +static VALUE chunkable_response_p(VALUE self)
> +{
> +  struct http_parser *hp = data_get(self);
> +
> +  return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse;
> +}
> +
>  /**
>   * call-seq:
>   *    parser.next? => true or false
> @@ -981,6 +991,7 @@ void Init_unicorn_http(void)
>    rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0);
>    rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0);
>    rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0);
> +  rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0);
>    rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0);
>    rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
>    rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
> diff --git a/lib/unicorn.rb b/lib/unicorn.rb
> index 1a50631..8b1cda7 100644
> --- a/lib/unicorn.rb
> +++ b/lib/unicorn.rb
> @@ -75,8 +75,8 @@ def self.builder(ru, op)
>  
>        # return value, matches rackup defaults based on env
>        # Unicorn does not support persistent connections, but Rainbows!
> -      # and Zbatery both do.  Users accustomed to the Rack::Server default
> -      # middlewares will need ContentLength/Chunked middlewares.
> +      # does.  Users accustomed to the Rack::Server default
> +      # middlewares will need ContentLength middleware.
>        case ENV["RACK_ENV"]
>        when "development"
>        when "deployment"
> diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
> index 19469b4..342dd0b 100644
> --- a/lib/unicorn/http_response.rb
> +++ b/lib/unicorn/http_response.rb
> @@ -35,11 +35,12 @@ def append_header(buf, key, value)
>    def http_response_write(socket, status, headers, body,
>                            req = Unicorn::HttpRequest.new)
>      hijack = nil
> -
> +    do_chunk = false
>      if headers
>        code = status.to_i
>        msg = STATUS_CODES[code]
>        start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
> +      term = false
>        buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
>              "Date: #{httpdate}\r\n" \
>              "Connection: close\r\n"
> @@ -47,6 +48,12 @@ def http_response_write(socket, status, headers, body,
>          case key
>          when %r{\A(?:Date|Connection)\z}i
>            next
> +        when %r{\AContent-Length\z}i
> +          append_header(buf, key, value)
> +          term = true
> +        when %r{\ATransfer-Encoding\z}i
> +          append_header(buf, key, value)
> +          term = true if /\bchunked\b/i === value # value may be Array :x
>          when "rack.hijack"
>            # This should only be hit under Rack >= 1.5, as this was an illegal
>            # key in Rack < 1.5
> @@ -55,12 +62,24 @@ def http_response_write(socket, status, headers, body,
>            append_header(buf, key, value)
>          end
>        end
> +      if !hijack && !term && req.chunkable_response?
> +        do_chunk = true
> +        buf << "Transfer-Encoding: chunked\r\n".freeze
> +      end
>        socket.write(buf << "\r\n".freeze)
>      end
>  
>      if hijack
>        req.hijacked!
>        hijack.call(socket)
> +    elsif do_chunk
> +      begin
> +        body.each do |b|
> +          socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze)
> +        end
> +      ensure
> +        socket.write("0\r\n\r\n".freeze)
> +      end
>      else
>        body.each { |chunk| socket.write(chunk) }
>      end
> diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
> index 8a6f6ee..4ae4c85 100644
> --- a/lib/unicorn/socket_helper.rb
> +++ b/lib/unicorn/socket_helper.rb
> @@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc:
>      end
>    end
>  
> +  if IO.instance_method(:write).arity # Ruby <= 2.4
> +    require 'unicorn/write_splat'
> +    UNIXClient = Class.new(Kgio::Socket) # :nodoc:
> +    class UNIXSrv < Kgio::UNIXServer # :nodoc:
> +      include Unicorn::WriteSplat
> +      def kgio_tryaccept # :nodoc:
> +        super(UNIXClient)
> +      end
> +    end
> +    TCPClient.__send__(:include, Unicorn::WriteSplat)
> +  else # Ruby 2.5+
> +    UNIXSrv = Kgio::UNIXServer
> +  end
> +
>    module SocketHelper
>  
>      # internal interface
> @@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
>          end
>          old_umask = File.umask(opt[:umask] || 0)
>          begin
> -          Kgio::UNIXServer.new(address)
> +          UNIXSrv.new(address)
>          ensure
>            File.umask(old_umask)
>          end
> @@ -203,7 +217,7 @@ def server_cast(sock)
>          Socket.unpack_sockaddr_in(sock.getsockname)
>          TCPSrv.for_fd(sock.fileno)
>        rescue ArgumentError
> -        Kgio::UNIXServer.for_fd(sock.fileno)
> +        UNIXSrv.for_fd(sock.fileno)
>        end
>      end
>  
> diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
> new file mode 100644
> index 0000000..7e6e363
> --- /dev/null
> +++ b/lib/unicorn/write_splat.rb
> @@ -0,0 +1,7 @@
> +# -*- encoding: binary -*-
> +# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
> +module Unicorn::WriteSplat # :nodoc:
> +  def write(*arg) # :nodoc:
> +    super(arg.join(''))
> +  end
> +end
> diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
> index 98e85ab..cea9791 100644
> --- a/test/unit/test_server.rb
> +++ b/test/unit/test_server.rb
> @@ -196,7 +196,7 @@ def test_client_shutdown_writes
>      # continue to process our request and never hit EOFError on our sock
>      sock.shutdown(Socket::SHUT_WR)
>      buf = sock.read
> -    assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last
> +    assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last
>      next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
>      assert_equal 'hello!\n', next_client
>      lines = File.readlines("test_stderr.#$$.log")

^ permalink raw reply	[relevance 0%]

* Re: Rack 3 Compatibility
  @ 2023-06-02  0:00  4% ` Eric Wong
  2023-06-02  2:45  0%   ` Jeremy Evans
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2023-06-02  0:00 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: unicorn-public

Jeremy Evans <code@jeremyevans.net> wrote:
> This takes Eric's patch from December 25, 2022, and includes all
> necessary test fixes to allow Unicorn tests to pass with both
> Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for
> newer curl versions and an OpenBSD test fix.
> 
> Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3
> support.  If further fixes are needed, I'm happy to work on them.

Isn't a chunk replacement needed for Rack 3.1, also?
I dunno if I missed anything else in Rack 3.x; and don't want to
make too many releases if we can do 3.0 and 3.1 in one go.

-------8<-------
Subject: [PATCH] chunk unterminated HTTP/1.1 responses

Rack::Chunked will be gone in Rack 3.1, so provide a
non-middleware fallback which takes advantage of IO#write
supporting multiple arguments in Ruby 2.5+.

We still need to support Ruby 2.4, at least, since Rack 3.0
does.  So a new (GC-unfriendly) Unicorn::WriteSplat module now
exists for Ruby <= 2.4 users.
---
 ext/unicorn_http/unicorn_http.rl | 11 +++++++++++
 lib/unicorn.rb                   |  4 ++--
 lib/unicorn/http_response.rb     | 21 ++++++++++++++++++++-
 lib/unicorn/socket_helper.rb     | 18 ++++++++++++++++--
 lib/unicorn/write_splat.rb       |  7 +++++++
 test/unit/test_server.rb         |  2 +-
 6 files changed, 57 insertions(+), 6 deletions(-)
 create mode 100644 lib/unicorn/write_splat.rb

diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index ba23438..c339024 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -28,6 +28,7 @@ void init_unicorn_httpdate(void);
 #define UH_FL_TO_CLEAR 0x200
 #define UH_FL_RESSTART 0x400 /* for check_client_connection */
 #define UH_FL_HIJACK 0x800
+#define UH_FL_RES_CHUNK_OK (1U << 12)
 
 /* all of these flags need to be set for keepalive to be supported */
 #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
@@ -158,6 +159,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len)
   if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) {
     /* HTTP/1.1 implies keepalive unless "Connection: close" is set */
     HP_FL_SET(hp, KAVERSION);
+    HP_FL_SET(hp, RES_CHUNK_OK);
     v = g_http_11;
   } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) {
     v = g_http_10;
@@ -801,6 +803,14 @@ static VALUE HttpParser_keepalive(VALUE self)
   return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse;
 }
 
+/* :nodoc: */
+static VALUE chunkable_response_p(VALUE self)
+{
+  struct http_parser *hp = data_get(self);
+
+  return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse;
+}
+
 /**
  * call-seq:
  *    parser.next? => true or false
@@ -981,6 +991,7 @@ void Init_unicorn_http(void)
   rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0);
   rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0);
   rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0);
+  rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0);
   rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0);
   rb_define_method(cHttpParser, "next?", HttpParser_next, 0);
   rb_define_method(cHttpParser, "buf", HttpParser_buf, 0);
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 1a50631..8b1cda7 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -75,8 +75,8 @@ def self.builder(ru, op)
 
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
-      # and Zbatery both do.  Users accustomed to the Rack::Server default
-      # middlewares will need ContentLength/Chunked middlewares.
+      # does.  Users accustomed to the Rack::Server default
+      # middlewares will need ContentLength middleware.
       case ENV["RACK_ENV"]
       when "development"
       when "deployment"
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 19469b4..342dd0b 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -35,11 +35,12 @@ def append_header(buf, key, value)
   def http_response_write(socket, status, headers, body,
                           req = Unicorn::HttpRequest.new)
     hijack = nil
-
+    do_chunk = false
     if headers
       code = status.to_i
       msg = STATUS_CODES[code]
       start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
+      term = false
       buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
             "Date: #{httpdate}\r\n" \
             "Connection: close\r\n"
@@ -47,6 +48,12 @@ def http_response_write(socket, status, headers, body,
         case key
         when %r{\A(?:Date|Connection)\z}i
           next
+        when %r{\AContent-Length\z}i
+          append_header(buf, key, value)
+          term = true
+        when %r{\ATransfer-Encoding\z}i
+          append_header(buf, key, value)
+          term = true if /\bchunked\b/i === value # value may be Array :x
         when "rack.hijack"
           # This should only be hit under Rack >= 1.5, as this was an illegal
           # key in Rack < 1.5
@@ -55,12 +62,24 @@ def http_response_write(socket, status, headers, body,
           append_header(buf, key, value)
         end
       end
+      if !hijack && !term && req.chunkable_response?
+        do_chunk = true
+        buf << "Transfer-Encoding: chunked\r\n".freeze
+      end
       socket.write(buf << "\r\n".freeze)
     end
 
     if hijack
       req.hijacked!
       hijack.call(socket)
+    elsif do_chunk
+      begin
+        body.each do |b|
+          socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze)
+        end
+      ensure
+        socket.write("0\r\n\r\n".freeze)
+      end
     else
       body.each { |chunk| socket.write(chunk) }
     end
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 8a6f6ee..4ae4c85 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc:
     end
   end
 
+  if IO.instance_method(:write).arity # Ruby <= 2.4
+    require 'unicorn/write_splat'
+    UNIXClient = Class.new(Kgio::Socket) # :nodoc:
+    class UNIXSrv < Kgio::UNIXServer # :nodoc:
+      include Unicorn::WriteSplat
+      def kgio_tryaccept # :nodoc:
+        super(UNIXClient)
+      end
+    end
+    TCPClient.__send__(:include, Unicorn::WriteSplat)
+  else # Ruby 2.5+
+    UNIXSrv = Kgio::UNIXServer
+  end
+
   module SocketHelper
 
     # internal interface
@@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {})
         end
         old_umask = File.umask(opt[:umask] || 0)
         begin
-          Kgio::UNIXServer.new(address)
+          UNIXSrv.new(address)
         ensure
           File.umask(old_umask)
         end
@@ -203,7 +217,7 @@ def server_cast(sock)
         Socket.unpack_sockaddr_in(sock.getsockname)
         TCPSrv.for_fd(sock.fileno)
       rescue ArgumentError
-        Kgio::UNIXServer.for_fd(sock.fileno)
+        UNIXSrv.for_fd(sock.fileno)
       end
     end
 
diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb
new file mode 100644
index 0000000..7e6e363
--- /dev/null
+++ b/lib/unicorn/write_splat.rb
@@ -0,0 +1,7 @@
+# -*- encoding: binary -*-
+# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+
+module Unicorn::WriteSplat # :nodoc:
+  def write(*arg) # :nodoc:
+    super(arg.join(''))
+  end
+end
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index 98e85ab..cea9791 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -196,7 +196,7 @@ def test_client_shutdown_writes
     # continue to process our request and never hit EOFError on our sock
     sock.shutdown(Socket::SHUT_WR)
     buf = sock.read
-    assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last
+    assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last
     next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/"))
     assert_equal 'hello!\n', next_client
     lines = File.readlines("test_stderr.#$$.log")

^ permalink raw reply related	[relevance 4%]

* Re: [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix
  @ 2017-04-02  2:14  6% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2017-04-02  2:14 UTC (permalink / raw)
  To: ruby-talk, unicorn-public
  Cc: Jeremy Evans, Simon Eskildsen, Dylan Thacker-Smith

Eric Wong <e@80x24.org> wrote:
> Anyways, this is a largish release with several new features,
> and no backwards incompatibilities.

Oops :x  Just released Rainbows! 5.1.1 to fix one incompatibility
introduced with unicorn 5.3.0 :x

  https://bogomips.org/rainbows-public/20170402021109.GA29200@dcvr/

Anyways, I no longer promote Rainbows! and there's almost no
users, so I hope the impact is small.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] limit rack version for ruby compatibility
  @ 2016-01-21 20:12  5%           ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2016-01-21 20:12 UTC (permalink / raw)
  To: Adam Duke; +Cc: Aaron Patterson, rack-devel, unicorn-public

Adam Duke <adamduke@twitter.com> wrote:
> Following up on this, it seems to me like keeping the Ruby 2.2.2
> requirement is the right way to go for rack. If the unicorn project
> wants to continue support for older rubies, the unicorn gemspec should
> be changed to limit the rack dependency to '< 2'. If rack 2.0.0 is
> released and there is no limit on the dependency in unicorn's gemspec,
> it seems to me like any deployments that are not running Ruby 2.2.2
> will fail.

I prefer we drop the rack dependency entirely.  We don't use rack
for much.  This seems doable, (totally untested) patch below:

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b0e6bd1..a3eae5e 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -1,9 +1,14 @@
 # -*- encoding: binary -*-
 require 'etc'
 require 'stringio'
-require 'rack'
 require 'kgio'
 
+begin
+  require 'rack'
+rescue LoadError
+  warn 'rack not available, functionality reduced'
+end
+
 # :stopdoc:
 # Unicorn module containing all of the classes (include C extensions) for
 # running a Unicorn web server.  It contains a minimalist HTTP server with just
@@ -32,6 +37,9 @@ module Unicorn
   def self.builder(ru, op)
     # allow Configurator to parse cli switches embedded in the ru file
     op = Unicorn::Configurator::RACKUP.merge!(:file => ru, :optparse => op)
+    if ru =~ /\.ru$/ && !defined?(Rack::Builder)
+      abort "rack and Rack::Builder must be available for processing #{ru}"
+    end
 
     # Op is going to get cleared before the returned lambda is called, so
     # save this value so that it's still there when we need it:
@@ -53,32 +61,33 @@ def self.builder(ru, op)
 
       return inner_app if no_default_middleware
 
+      middleware = { # order matters
+        ContentLength: [],
+        Chunked: [],
+        CommonLogger: [ $stderr ],
+        ShowExceptions: [],
+        Lint: [],
+        TempfileReaper: []
+      }
+
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
       # and Zbatery both do.  Users accustomed to the Rack::Server default
       # middlewares will need ContentLength/Chunked middlewares.
       case ENV["RACK_ENV"]
       when "development"
-        Rack::Builder.new do
-          use Rack::ContentLength
-          use Rack::Chunked
-          use Rack::CommonLogger, $stderr
-          use Rack::ShowExceptions
-          use Rack::Lint
-          use Rack::TempfileReaper if Rack.const_defined?(:TempfileReaper)
-          run inner_app
-        end.to_app
       when "deployment"
-        Rack::Builder.new do
-          use Rack::ContentLength
-          use Rack::Chunked
-          use Rack::CommonLogger, $stderr
-          use Rack::TempfileReaper if Rack.const_defined?(:TempfileReaper)
-          run inner_app
-        end.to_app
+        middleware.delete(:ShowExceptions)
+        middleware.delete(:Lint)
       else
-        inner_app
+        return inner_app
       end
+      Rack::Builder.new do
+        middleware.each do |m, args|
+          use(Rack.const_get(m), *args) if Rack.const_defined?(m)
+        end
+        run inner_app
+      end.to_app
     end
   end
 
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 7b446c2..ec128e4 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -10,10 +10,13 @@
 # is the job of Rack, with the exception of the "Date" and "Status" header.
 module Unicorn::HttpResponse
 
+  STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ?
+                 Rack::Utils::HTTP_STATUS_CODES : {}
+
   # internal API, code will always be common-enough-for-even-old-Rack
   def err_response(code, response_start_sent)
     "#{response_start_sent ? '' : 'HTTP/1.1 '}" \
-      "#{code} #{Rack::Utils::HTTP_STATUS_CODES[code]}\r\n\r\n"
+      "#{code} #{STATUS_CODES[code]}\r\n\r\n"
   end
 
   # writes the rack_response to socket as an HTTP response
@@ -23,7 +26,7 @@ def http_response_write(socket, status, headers, body,
 
     if headers
       code = status.to_i
-      msg = Rack::Utils::HTTP_STATUS_CODES[code]
+      msg = STATUS_CODES[code]
       start = response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze
       buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \
             "Date: #{httpdate}\r\n" \
diff --git a/unicorn.gemspec b/unicorn.gemspec
index 2728373..c3f8267 100644
--- a/unicorn.gemspec
+++ b/unicorn.gemspec
@@ -31,11 +31,11 @@
   # version requirements here.
   s.required_ruby_version = '< 3.0'
 
-  # for people that are absolutely stuck on Rails 2.3.2 and can't
-  # up/downgrade to any other version, the Rack dependency may be
-  # commented out.  Nevertheless, upgrading to Rails 2.3.4 or later is
-  # *strongly* recommended for security reasons.
-  s.add_dependency(%q<rack>)
+  # We do not have a hard dependency on rack, it's possible to load
+  # things which respond to #call.  HTTP status lines in responses
+  # won't have descriptive text, only the numeric status.
+  # s.add_dependency(%q<rack>)
+
   s.add_dependency(%q<kgio>, '~> 2.6')
   s.add_dependency(%q<raindrops>, '~> 0.7')

...  Not touching the untested, ancient old_rails stuff, yet

^ permalink raw reply related	[relevance 5%]

* [ANN] unicorn 5.0.0 - Rack HTTP server for fast clients and *nix
@ 2015-11-01  8:55  3% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2015-11-01  8:55 UTC (permalink / raw)
  To: ruby-talk, unicorn-public

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* http://unicorn.bogomips.org/
* public list: unicorn-public@bogomips.org
* mail archives: http://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

Changes since 4.9.0:

unicorn 5.0.0 - most boring major release. EVER.

An evolutionary dead-end since its announcement[1] nearly six years
ago, this old-fashioned preforker has had enough bugs and missteps
that it's managed to hit version 5!

I wish I could say unicorn 5 is leaps and bounds better than 4, but
it is not.  This major version change allows us to drop some cruft
and unused features which accumulated over the years, resulting in
several kilobytes of memory saved[2]!

Compatibility:

* The horrible, proprietary (:P) "Status:" response header is
  finally gone, saving at least 16 precious bytes in every HTTP
  response.  This should make it easier to write custom HTTP clients
  which are compatible across all HTTP servers.  It will hopefully
  make migrating between different Rack servers easier for new
  projects.

* Ruby 1.8 support removed.  Ruby 1.9.3 is currently the earliest
  supported version.  However, expect minor, likely-unnoticeable
  performance regressions if you use Ruby 2.1 or earlier.  Going
  forward, unicorn will favor the latest version (currently 2.2) of
  the mainline Ruby implementation, potentially sacrificing
  performance on older Rubies.

* Some internal, undocumented features and APIs used by
  derivative servers are gone; removing bloat and slightly lowering
  memory use.  We have never and will never endorse the use of any
  applications or middleware with a dependency on unicorn,
  applications should be written for Rack instead.
  Note: Rainbows! 5.0 will be released next week or so to be
  compatible with unicorn 5.x

New features:

* sd_listen_fds(3) emulation added for systemd compatibility.
  You may now stop using PID files and other process monitoring
  software when using systemd.

-- 
EW

^ permalink raw reply	[relevance 3%]

* Re: Request to follow SemVer/mention it in homepage
  2015-09-29 19:36  0%     ` Eric Wong
@ 2015-09-30 16:04  0%       ` Pirate Praveen
  0 siblings, 0 replies; 191+ results
From: Pirate Praveen @ 2015-09-30 16:04 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wednesday 30 September 2015 01:06 AM, Eric Wong wrote:
> Pirate Praveen <praveen@debian.org> wrote:
>> On २९ सप्टेंबर, २०१५ १:०६:५० [PM] IST, Eric Wong <e@80x24.org> 
>> wrote:
>>> Pirate Praveen <praveen@debian.org> wrote:
>>>> Do you follow Semantic Versioning as defined by semver.org? 
>>>> If yes, can you mention it on the homepage? If not, can you 
>>>> consider it?
> 
> <snip>
> 
>>> I have considered it in the past, but decided against it...
>>> 
>>> unicorn is on the verge of becoming 5.x:
>>> 
>>> This is for internal changes which some weirdo projects 
>>> (notably Rainbows!) rely on; but was never considered public 
>>> API or marketed as such.  I absolutely do not want people 
>>> building server-specific apps when Rack exists.
>> 
>> Can you consider SemVer from 5.x?
> 
> Again, the answer is "no".  I do not want people to depend on 
> unicorn providing a stable API of any sort.  Even if in practice 
> and history it does: the config file directives and internal 
> constant names haven't changed at all in 5-6 years and there are
> no plans to change it.

Can you mention the recommended way of adding unicorn in a Gemfile in
your home page?

gitlab and diaspora has unicorn in their Gemfile, for a user they want
gitlab or diaspora and they don't care if its using unicorn or puma or
passenger.

I need some kind of an official statement about compatibility which I
can show to projects like gitlab and diaspora, so they don't insist on
exact patch release of unicorn. I could patch the Gemfile, but that
adds extra burden on me as a maintainer which I'd like to avoid if
possible.

>>> Regardless of a formalities such as semver, I'll work to
>>> ensure unicorn can be compatible[1] with any Rack apps in
>>> Debian main. Heck, perhaps Rack 0.9.x still works (assuming
>>> the installed Ruby version supports it).
>>> 
>>> Just let us know what real problems you find, they will be 
>>> fixed.
>> 
>> If you can convince gitlab to declare ~> 4.8 and assure it will 
>> work with 4.9, that will solve the issue for now.
>> 
>> See https://gitlab.com/gitlab-org/gitlab-ce/issues/2820
> 
> That's not a real compatibility bug with unicorn itself.  Again, 
> it's wrong of them to have a dependency on any Rack server at all.
> 
>> Also if you can help convinse diaspora upstream to declare a 
>> looser dependency, that will also help.
>> 
>>> Fwiw, Debian is my preferred platform and I can stay 100% 
>>> within email with the Debian BTS + lists.
> 
> I am only doing plain-text email.  I'm done with websites requiring
> login + registration.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWDAf9AAoJEM4fnGdFEsIqYV0P/AmIFATrnxSF4V5Q5v05LT5G
LcVkXCzMhQYnPaVYwiAEi5JLGkiFw2r2Y5kWgF3WLynXHqWmUpGoNLEsG5tZG38V
109RYYKG5VgsGw6HlIoOwpy61MF9/d29+CRDXvLs0q7wc1/CbDEYqnhwLvKKOT9o
xZBD3bXsLuXiRl/BR8xk1q5fy4oS0rNnTU4MWcc8pEKU3KFE0xkXqA5WVS3ENShg
WLfqlUnll2SnQwyR1FzqFJws5OzejKpdNRZK1IOD1ivMd8aBVY25+0u6Q3o/2h/J
dW4ID9QV6R1fCtLybKqkI/fwljlKGs+shY+5lvUqE2KSYe8gwyHm6r1DslprzwM6
QhzLXqBtvDxSxX41sbcr12SoEt7poRztr+8gFXqni+iTkoljgsmODQMWX/ed/9O8
AekqfSrA//K1JfUpGKWHr1+UXhijHggYcKi0eoCJA2GH5ijLu5DIglEuGCQtthl9
z4/aI/na1M7goqBKw1Qtuxc/AHlBAkDcsott5n+4FIQInc5ooiyj54+Sek1AZNWw
/pp5F6z3axPgH3+h26Jf4W8OHl58gUeht9G7sl2j5GwMDkVSOtwlgwhLEYWdZ8Mw
pZcDXl/0VieD6LBFEh3N5MfkP+n3aYHrenCbsxRBh/jn6/+wczoUruNYl0SbSgUh
62qZ2q/7NgvHBdVPtXaB
=F2uT
-----END PGP SIGNATURE-----

^ permalink raw reply	[relevance 0%]

* Re: Request to follow SemVer/mention it in homepage
  2015-09-29  8:00  0%   ` Pirate Praveen
@ 2015-09-29 19:36  0%     ` Eric Wong
  2015-09-30 16:04  0%       ` Pirate Praveen
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2015-09-29 19:36 UTC (permalink / raw)
  To: Pirate Praveen; +Cc: unicorn-public

Pirate Praveen <praveen@debian.org> wrote:
> On २९ सप्टेंबर, २०१५ १:०६:५० [PM] IST, Eric Wong <e@80x24.org> wrote:
> >Pirate Praveen <praveen@debian.org> wrote:
> >> Do you follow Semantic Versioning as defined by semver.org? If yes,
> >> can you mention it on the homepage? If not, can you consider it?

<snip>

> >I have considered it in the past, but decided against it...
> >
> >unicorn is on the verge of becoming 5.x:
> >
> >This is for internal changes which some weirdo projects (notably
> >Rainbows!) rely on; but was never considered public API or marketed
> >as such.  I absolutely do not want people building server-specific
> >apps when Rack exists.
> 
> Can you consider SemVer from 5.x?

Again, the answer is "no".  I do not want people to depend on unicorn
providing a stable API of any sort.  Even if in practice and history
it does: the config file directives and internal constant names
haven't changed at all in 5-6 years and there are no plans to change it.

> >Regardless of a formalities such as semver, I'll work to ensure unicorn
> >can be compatible[1] with any Rack apps in Debian main.
> >Heck, perhaps Rack 0.9.x still works (assuming the installed Ruby
> >version supports it).
> >
> >Just let us know what real problems you find, they will be fixed.
> 
> If you can convince gitlab to declare ~> 4.8 and assure it will work
> with 4.9, that will solve the issue for now.
> 
> See https://gitlab.com/gitlab-org/gitlab-ce/issues/2820

That's not a real compatibility bug with unicorn itself.  Again, it's
wrong of them to have a dependency on any Rack server at all.

> Also if you can help convinse diaspora upstream to declare a looser
> dependency, that will also help.
> 
> >Fwiw, Debian is my preferred platform and I can stay 100% within email
> >with the Debian BTS + lists.

I am only doing plain-text email.  I'm done with websites requiring
login + registration.

^ permalink raw reply	[relevance 0%]

* Re: Request to follow SemVer/mention it in homepage
  2015-09-29  7:36  3% ` Eric Wong
@ 2015-09-29  8:00  0%   ` Pirate Praveen
  2015-09-29 19:36  0%     ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Pirate Praveen @ 2015-09-29  8:00 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public



On २९ सप्टेंबर, २०१५ १:०६:५० [PM] IST, Eric Wong <e@80x24.org> wrote:
>Pirate Praveen <praveen@debian.org> wrote:
>> Do you follow Semantic Versioning as defined by semver.org? If yes,
>> can you mention it on the homepage? If not, can you consider it?
>
>Maybe we follow semver...
>
>No, I'd rather not mention it on the homepage or be formally/officially
>bound to it (see below)
>
>I have considered it in the past, but decided against it...
>
>unicorn is on the verge of becoming 5.x:
>
>This is for internal changes which some weirdo projects (notably
>Rainbows!) rely on; but was never considered public API or marketed
>as such.  I absolutely do not want people building server-specific
>apps when Rack exists.

Can you consider SemVer from 5.x?

>Rack will also hit 2.x soon, but regardless of incompatible changes
>in Rack, unicorn must remain compatible with both 1.x and 2.x for
>practical reasons.
>
>> It will help us a lot in maintain rails applications in debian. We
>> like to keep only one version of any app/lib in debian and if you can
>> guarantee Semantic Versioning rails apps using unicorn could declare
>a
>> looser dependency rather than upto the patch version. Now gitlab
>> defines unicorn ~> 4.8.3 and diaspora defines ~> 4.9.0 If you are
>> following Semantic Versioning they could change it to ~> 4.8 and ~>
>> 4.9 and 4.9.0 will satisfy both. Currently debian has 4.9.0 and it
>> does not match ~> 4.8.3
>
>Tying a Rack app to unicorn is totally, completely wrong and defeats
>the
>point of Rack.
>
>Honestly, I could understand tying an app to thin with it's EM-specific
>API, but unicorn(!?!?)  Even if an app isn't thread-safe, it should
>still work in passenger, thin, or puma/webrick configured for a
>single-thread...
>
>
>Regardless of a formalities such as semver, I'll work to ensure unicorn
>can be compatible[1] with any Rack apps in Debian main.
>Heck, perhaps Rack 0.9.x still works (assuming the installed Ruby
>version supports it).
>
>Just let us know what real problems you find, they will be fixed.

If you can convince gitlab to declare ~> 4.8 and assure it will work with 4.9, that will solve the issue for now.

See https://gitlab.com/gitlab-org/gitlab-ce/issues/2820

Also if you can help convinse diaspora upstream to declare a looser dependency, that will also help.

>Fwiw, Debian is my preferred platform and I can stay 100% within email
>with the Debian BTS + lists.

Glad to know.

>
>[1] I would never say unicorn is the best choice for all apps,
>    but it should at least limp along with sufficiently loose
>    definitions of "working", perhaps with hundreds of inefficient
>    processes in some cases.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

^ permalink raw reply	[relevance 0%]

* Re: Request to follow SemVer/mention it in homepage
  @ 2015-09-29  7:36  3% ` Eric Wong
  2015-09-29  8:00  0%   ` Pirate Praveen
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2015-09-29  7:36 UTC (permalink / raw)
  To: Pirate Praveen; +Cc: unicorn-public

Pirate Praveen <praveen@debian.org> wrote:
> Do you follow Semantic Versioning as defined by semver.org? If yes,
> can you mention it on the homepage? If not, can you consider it?

Maybe we follow semver...

No, I'd rather not mention it on the homepage or be formally/officially
bound to it (see below)

I have considered it in the past, but decided against it...

unicorn is on the verge of becoming 5.x:

This is for internal changes which some weirdo projects (notably
Rainbows!) rely on; but was never considered public API or marketed
as such.  I absolutely do not want people building server-specific
apps when Rack exists.

Rack will also hit 2.x soon, but regardless of incompatible changes
in Rack, unicorn must remain compatible with both 1.x and 2.x for
practical reasons.

> It will help us a lot in maintain rails applications in debian. We
> like to keep only one version of any app/lib in debian and if you can
> guarantee Semantic Versioning rails apps using unicorn could declare a
> looser dependency rather than upto the patch version. Now gitlab
> defines unicorn ~> 4.8.3 and diaspora defines ~> 4.9.0 If you are
> following Semantic Versioning they could change it to ~> 4.8 and ~>
> 4.9 and 4.9.0 will satisfy both. Currently debian has 4.9.0 and it
> does not match ~> 4.8.3

Tying a Rack app to unicorn is totally, completely wrong and defeats the
point of Rack.

Honestly, I could understand tying an app to thin with it's EM-specific
API, but unicorn(!?!?)  Even if an app isn't thread-safe, it should
still work in passenger, thin, or puma/webrick configured for a
single-thread...


Regardless of a formalities such as semver, I'll work to ensure unicorn
can be compatible[1] with any Rack apps in Debian main.
Heck, perhaps Rack 0.9.x still works (assuming the installed Ruby
version supports it).

Just let us know what real problems you find, they will be fixed.

Fwiw, Debian is my preferred platform and I can stay 100% within email
with the Debian BTS + lists.


[1] I would never say unicorn is the best choice for all apps,
    but it should at least limp along with sufficiently loose
    definitions of "working", perhaps with hundreds of inefficient
    processes in some cases.

^ permalink raw reply	[relevance 3%]

* [PATCH] doc: remove references to old servers
@ 2015-07-15 22:05 12% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2015-07-15 22:05 UTC (permalink / raw)
  To: unicorn-public

They'll continue to be maintained, but we're no longer advertising
them.  Also, favor lowercase "unicorn" while we're at it since that
matches the executable and gem name to avoid unnecessary escaping
for RDoc.
---
 Application_Timeouts             |  6 +++---
 KNOWN_ISSUES                     | 14 +++++++-------
 Links                            | 30 ++++++++++++++----------------
 PHILOSOPHY                       |  6 ------
 README                           | 32 ++++++++++++++++----------------
 Sandbox                          |  2 +-
 TUNING                           | 10 +++++-----
 examples/nginx.conf              | 21 ++++++++++-----------
 ext/unicorn_http/unicorn_http.rl |  2 +-
 lib/unicorn.rb                   |  6 +++---
 lib/unicorn/configurator.rb      | 24 ++++++++++--------------
 lib/unicorn/http_server.rb       |  4 ++--
 lib/unicorn/socket_helper.rb     |  8 +++-----
 lib/unicorn/util.rb              |  2 +-
 lib/unicorn/worker.rb            |  4 ++--
 15 files changed, 78 insertions(+), 93 deletions(-)

diff --git a/Application_Timeouts b/Application_Timeouts
index 5f0370d..561a1cc 100644
--- a/Application_Timeouts
+++ b/Application_Timeouts
@@ -4,10 +4,10 @@ This article focuses on _application_ setup for Rack applications, but
 can be expanded to all applications that connect to external resources
 and expect short response times.
 
-This article is not specific to \Unicorn, but exists to discourage
+This article is not specific to unicorn, but exists to discourage
 the overuse of the built-in
 {timeout}[link:Unicorn/Configurator.html#method-i-timeout] directive
-in \Unicorn.
+in unicorn.
 
 == ALL External Resources Are Considered Unreliable
 
@@ -71,7 +71,7 @@ handle network/server failures.
 == The Last Line Of Defense
 
 The {timeout}[link:Unicorn/Configurator.html#method-i-timeout] mechanism
-in \Unicorn is an extreme solution that should be avoided whenever
+in unicorn is an extreme solution that should be avoided whenever
 possible.  It will help catch bugs in your application where and when
 your application forgets to use timeouts, but it is expensive as it
 kills and respawns a worker process.
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
index 1950223..6b80517 100644
--- a/KNOWN_ISSUES
+++ b/KNOWN_ISSUES
@@ -13,7 +13,7 @@ acceptable solution.  Those issues are documented here.
 
 * PRNGs (pseudo-random number generators) loaded before forking
   (e.g. "preload_app true") may need to have their internal state
-  reset in the after_fork hook.  Starting with \Unicorn 3.6.1, we
+  reset in the after_fork hook.  Starting with unicorn 3.6.1, we
   have builtin workarounds for Kernel#rand and OpenSSL::Random users,
   but applications may use other PRNGs.
 
@@ -36,13 +36,13 @@ acceptable solution.  Those issues are documented here.
 
 * Under some versions of Ruby 1.8, it is necessary to call +srand+ in an
   after_fork hook to get correct random number generation.  We have a builtin
-  workaround for this starting with \Unicorn 3.6.1
+  workaround for this starting with unicorn 3.6.1
 
   See http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/36450
 
 * On Ruby 1.8 prior to Ruby 1.8.7-p248, *BSD platforms have a broken
   stdio that causes failure for file uploads larger than 112K.  Upgrade
-  your version of Ruby or continue using Unicorn 1.x/3.4.x.
+  your version of Ruby or continue using unicorn 1.x/3.4.x.
 
 * Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will
   segfault if called after forking.  Upgrade to Ruby 1.9.2 or call
@@ -53,12 +53,12 @@ acceptable solution.  Those issues are documented here.
 
 * Rails 2.3.2 bundles its own version of Rack.  This may cause subtle
   bugs when simultaneously loaded with the system-wide Rack Rubygem
-  which Unicorn depends on.  Upgrading to Rails 2.3.4 (or later) is
+  which unicorn depends on.  Upgrading to Rails 2.3.4 (or later) is
   strongly recommended for all Rails 2.3.x users for this (and security
   reasons).  Rails 2.2.x series (or before) did not bundle Rack and are
   should be unnaffected.  If there is any reason which forces your
   application to use Rails 2.3.2 and you have no other choice, then
-  you may edit your Unicorn gemspec and remove the Rack dependency.
+  you may edit your unicorn gemspec and remove the Rack dependency.
 
   ref: http://mid.gmane.org/20091014221552.GA30624@dcvr.yhbt.net
   Note: the workaround described in the article above only made
@@ -71,9 +71,9 @@ acceptable solution.  Those issues are documented here.
     set :env, :production
     set :run, false
   Since this is no longer an issue with Sinatra 0.9.x apps, this will not be
-  fixed on our end.  Since Unicorn is itself the application launcher, the
+  fixed on our end.  Since unicorn is itself the application launcher, the
   at_exit handler used in old Sinatra always caused Mongrel to be launched
-  whenever a Unicorn worker was about to exit.
+  whenever a unicorn worker was about to exit.
 
   Also remember we're capable of replacing the running binary without dropping
   any connections regardless of framework :)
diff --git a/Links b/Links
index 5a586c1..ad05afd 100644
--- a/Links
+++ b/Links
@@ -1,13 +1,13 @@
 = Related Projects
 
-If you're interested in \Unicorn, you may be interested in some of the projects
+If you're interested in unicorn, you may be interested in some of the projects
 listed below.  If you have any links to add/change/remove, please tell us at
 mailto:unicorn-public@bogomips.org!
 
 == Disclaimer
 
-The \Unicorn project is not responsible for the content in these links.
-Furthermore, the \Unicorn project has never, does not and will never endorse:
+The unicorn project is not responsible for the content in these links.
+Furthermore, the unicorn project has never, does not and will never endorse:
 
 * any for-profit entities or services
 * any non-{Free Software}[http://www.gnu.org/philosophy/free-sw.html]
@@ -15,13 +15,13 @@ Furthermore, the \Unicorn project has never, does not and will never endorse:
 The existence of these links does not imply endorsement of any entities
 or services behind them.
 
-=== For use with \Unicorn
+=== For use with unicorn
 
 * {Bluepill}[https://github.com/arya/bluepill] -
   a simple process monitoring tool written in Ruby
 
 * {golden_brindle}[https://github.com/simonoff/golden_brindle] - tool to
-  manage multiple \Unicorn instances/applications on a single server
+  manage multiple unicorn instances/applications on a single server
 
 * {raindrops}[http://raindrops.bogomips.org/] - real-time stats for
   preforking Rack servers
@@ -29,27 +29,25 @@ or services behind them.
 * {UnXF}[http://bogomips.org/unxf/]  Un-X-Forward* the Rack environment,
   useful since unicorn is designed to be deployed behind a reverse proxy.
 
-=== \Unicorn is written to work with
+=== unicorn is written to work with
 
 * {Rack}[http://rack.github.io/] - a minimal interface between webservers
   supporting Ruby and Ruby frameworks
 
 * {Ruby}[https://www.ruby-lang.org/en/] - the programming language of
-  Rack and \Unicorn
+  Rack and unicorn
 
-* {nginx}[http://nginx.org/] - the reverse proxy for use with \Unicorn
+* {nginx}[http://nginx.org/] - the reverse proxy for use with unicorn
 
-* {kgio}[http://bogomips.org/kgio/] - the I/O library written for \Unicorn
+* {kgio}[http://bogomips.org/kgio/] - the I/O library written for unicorn
+  (deprecated and functionality being mainlined into Ruby)
 
 === Derivatives
 
-* {Green Unicorn}[http://gunicorn.org/] - a Python version of \Unicorn
+* {Green Unicorn}[http://gunicorn.org/] - a Python version of unicorn
 
-* {Rainbows!}[http://rainbows.bogomips.org/] - \Unicorn for sleepy
-  apps and slow clients (historical).
-
-* {yahns}[http://yahns.yhbt.net/] - like Rainbows!, but with fewer options
-  and designed for energy efficiency on idle sites.
+* {yahns}[http://yahns.yhbt.net/] - the complete opposite of unicorn in
+  every imaginable way.  Designed for energy efficiency on idle sites.
 
 === Prior Work
 
@@ -57,4 +55,4 @@ or services behind them.
   unicorn is based on
 
 * {david}[http://bogomips.org/david.git] - a tool to explain why you need
-  nginx in front of \Unicorn
+  nginx in front of unicorn
diff --git a/PHILOSOPHY b/PHILOSOPHY
index 18b2d82..feb83d9 100644
--- a/PHILOSOPHY
+++ b/PHILOSOPHY
@@ -137,9 +137,3 @@ unicorn is highly inefficient for Comet/reverse-HTTP/push applications
 where the HTTP connection spends a large amount of time idle.
 Nevertheless, the ease of troubleshooting, debugging, and management of
 unicorn may still outweigh the drawbacks for these applications.
-
-The {Rainbows!}[http://rainbows.bogomips.org/] aims to fill the gap for
-odd corner cases where the nginx + unicorn combination is not enough.
-While Rainbows! management/administration is largely identical to
-unicorn, Rainbows! is far more ambitious and has seen little real-world
-usage.
diff --git a/README b/README
index bd626e9..dc121d3 100644
--- a/README
+++ b/README
@@ -1,10 +1,10 @@
-= Unicorn: Rack HTTP server for fast clients and Unix
+= unicorn: Rack HTTP server for fast clients and Unix
 
-\Unicorn is an HTTP server for Rack applications designed to only serve
+unicorn is an HTTP server for Rack applications designed to only serve
 fast clients on low-latency, high-bandwidth connections and take
 advantage of features in Unix/Unix-like kernels.  Slow clients should
 only be served by placing a reverse proxy capable of fully buffering
-both the the request and response in between \Unicorn and slow clients.
+both the the request and response in between unicorn and slow clients.
 
 == Features
 
@@ -15,9 +15,9 @@ both the the request and response in between \Unicorn and slow clients.
 * Compatible with Ruby 1.9.3 and later.
   unicorn 4.8.x will remain supported for Ruby 1.8 users.
 
-* Process management: \Unicorn will reap and restart workers that
+* Process management: unicorn will reap and restart workers that
   die from broken apps.  There is no need to manage multiple processes
-  or ports yourself.  \Unicorn can spawn and manage any number of
+  or ports yourself.  unicorn can spawn and manage any number of
   worker processes you choose to scale to your backend.
 
 * Load balancing is done entirely by the operating system kernel.
@@ -33,11 +33,11 @@ both the the request and response in between \Unicorn and slow clients.
 * Builtin reopening of all log files in your application via
   USR1 signal.  This allows logrotate to rotate files atomically and
   quickly via rename instead of the racy and slow copytruncate method.
-  \Unicorn also takes steps to ensure multi-line log entries from one
+  unicorn also takes steps to ensure multi-line log entries from one
   request all stay within the same file.
 
 * nginx-style binary upgrades without losing connections.
-  You can upgrade \Unicorn, your entire application, libraries
+  You can upgrade unicorn, your entire application, libraries
   and even your Ruby interpreter without dropping clients.
 
 * before_fork and after_fork hooks in case your application
@@ -60,15 +60,15 @@ both the the request and response in between \Unicorn and slow clients.
 
 == License
 
-\Unicorn is copyright 2009 by all contributors (see logs in git).
+unicorn is copyright 2009 by all contributors (see logs in git).
 It is based on Mongrel 1.1.5.
 Mongrel is copyright 2007 Zed A. Shaw and contributors.
 
-\Unicorn is licensed under (your choice) of the GPLv2 or later
+unicorn is licensed under (your choice) of the GPLv2 or later
 (GPLv3+ preferred), or Ruby (1.8)-specific terms.
 See the included LICENSE file for details.
 
-\Unicorn is 100% Free Software.
+unicorn is 100% Free Software.
 
 == Install
 
@@ -108,17 +108,17 @@ In RAILS_ROOT, run:
 
   unicorn_rails
 
-\Unicorn will bind to all interfaces on TCP port 8080 by default.
+unicorn will bind to all interfaces on TCP port 8080 by default.
 You may use the +--listen/-l+ switch to bind to a different
 address:port or a UNIX socket.
 
 === Configuration File(s)
 
-\Unicorn will look for the config.ru file used by rackup in APP_ROOT.
+unicorn will look for the config.ru file used by rackup in APP_ROOT.
 
-For deployments, it can use a config file for \Unicorn-specific options
+For deployments, it can use a config file for unicorn-specific options
 specified by the +--config-file/-c+ command-line switch.  See
-Unicorn::Configurator for the syntax of the \Unicorn-specific options.
+Unicorn::Configurator for the syntax of the unicorn-specific options.
 The default settings are designed for maximum out-of-the-box
 compatibility with existing applications.
 
@@ -130,7 +130,7 @@ supported.  Run `unicorn -h` to see command-line options.
 There is NO WARRANTY whatsoever if anything goes wrong, but
 {let us know}[link:ISSUES.html] and we'll try our best to fix it.
 
-\Unicorn is designed to only serve fast clients either on the local host
+unicorn is designed to only serve fast clients either on the local host
 or a fast LAN.  See the PHILOSOPHY and DESIGN documents for more details
 regarding this.
 
@@ -140,6 +140,6 @@ All feedback (bug reports, user/development dicussion, patches, pull
 requests) go to the mailing list/newsgroup.  See the ISSUES document for
 information on the {mailing list}[mailto:unicorn-public@bogomips.org].
 
-For the latest on \Unicorn releases, you may also finger us at
+For the latest on unicorn releases, you may also finger us at
 unicorn@bogomips.org or check our NEWS page (and subscribe to our Atom
 feed).
diff --git a/Sandbox b/Sandbox
index a6c3fe7..997b92f 100644
--- a/Sandbox
+++ b/Sandbox
@@ -1,4 +1,4 @@
-= Tips for using \Unicorn with Sandbox installation tools
+= Tips for using unicorn with Sandbox installation tools
 
 Since unicorn includes executables and is usually used to start a Ruby
 process, there are certain caveats to using it with tools that sandbox
diff --git a/TUNING b/TUNING
index 6a6d7db..247090b 100644
--- a/TUNING
+++ b/TUNING
@@ -1,10 +1,10 @@
-= Tuning \Unicorn
+= Tuning unicorn
 
-\Unicorn performance is generally as good as a (mostly) Ruby web server
+unicorn performance is generally as good as a (mostly) Ruby web server
 can provide.  Most often the performance bottleneck is in the web
 application running on Unicorn rather than Unicorn itself.
 
-== \Unicorn Configuration
+== unicorn Configuration
 
 See Unicorn::Configurator for details on the config file format.
 +worker_processes+ is the most-commonly needed tuning parameter.
@@ -14,7 +14,7 @@ See Unicorn::Configurator for details on the config file format.
 * worker_processes should be scaled to the number of processes your
   backend system(s) can support.  DO NOT scale it to the number of
   external network clients your application expects to be serving.
-  \Unicorn is NOT for serving slow clients, that is the job of nginx.
+  unicorn is NOT for serving slow clients, that is the job of nginx.
 
 * worker_processes should be *at* *least* the number of CPU cores on
   a dedicated server (unless you do not have enough memory).
@@ -58,7 +58,7 @@ See Unicorn::Configurator for details on the config file format.
 * UNIX domain sockets are slightly faster than TCP sockets, but only
   work if nginx is on the same machine.
 
-== Other \Unicorn settings
+== Other unicorn settings
 
 * Setting "preload_app true" can allow copy-on-write-friendly GC to
   be used to save memory.  It will probably not work out of the box with
diff --git a/examples/nginx.conf b/examples/nginx.conf
index a68fe6f..0583c1f 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -1,5 +1,5 @@
 # This is example contains the bare mininum to get nginx going with
-# Unicorn or Rainbows! servers.  Generally these configuration settings
+# unicorn servers.  Generally these configuration settings
 # are applicable to other HTTP application servers (and not just Ruby
 # ones), so if you have one working well for proxying another app
 # server, feel free to continue using it.
@@ -44,8 +44,8 @@ http {
   # click tracking!
   access_log /path/to/nginx.access.log combined;
 
-  # you generally want to serve static files with nginx since neither
-  # Unicorn nor Rainbows! is optimized for it at the moment
+  # you generally want to serve static files with nginx since
+  # unicorn is not and will never be optimized for it
   sendfile on;
 
   tcp_nopush on; # off may be better for *some* Comet/long-poll stuff
@@ -67,10 +67,10 @@ http {
              text/javascript application/x-javascript
              application/atom+xml;
 
-  # this can be any application server, not just Unicorn/Rainbows!
+  # this can be any application server, not just unicorn
   upstream app_server {
     # fail_timeout=0 means we always retry an upstream even if it failed
-    # to return a good HTTP response (in case the Unicorn master nukes a
+    # to return a good HTTP response (in case the unicorn master nukes a
     # single worker for timing out).
 
     # for UNIX domain socket setups:
@@ -132,12 +132,11 @@ http {
       # redirects, we set the Host: header above already.
       proxy_redirect off;
 
-      # set "proxy_buffering off" *only* for Rainbows! when doing
-      # Comet/long-poll/streaming.  It's also safe to set if you're using
-      # only serving fast clients with Unicorn + nginx, but not slow
-      # clients.  You normally want nginx to buffer responses to slow
-      # clients, even with Rails 3.1 streaming because otherwise a slow
-      # client can become a bottleneck of Unicorn.
+      # It's also safe to set if you're using only serving fast clients
+      # with unicorn + nginx, but not slow clients.  You normally want
+      # nginx to buffer responses to slow clients, even with Rails 3.1
+      # streaming because otherwise a slow client can become a bottleneck
+      # of unicorn.
       #
       # The Rack application may also set "X-Accel-Buffering (yes|no)"
       # in the response headers do disable/enable buffering on a
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index a5f069d..046ccb5 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -38,7 +38,7 @@ static VALUE set_maxhdrlen(VALUE self, VALUE len)
   return UINT2NUM(MAX_HEADER_LEN = NUM2UINT(len));
 }
 
-/* keep this small for Rainbows! since every client has one */
+/* keep this small for other servers (e.g. yahns) since every client has one */
 struct http_parser {
   int cs; /* Ragel internal state */
   unsigned int flags;
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 9fdcb8e..b0e6bd1 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -10,10 +10,10 @@ require 'kgio'
 # enough functionality to service web application requests fast as possible.
 # :startdoc:
 
-# \Unicorn exposes very little of an user-visible API and most of its
-# internals are subject to change.  \Unicorn is designed to host Rack
+# unicorn exposes very little of an user-visible API and most of its
+# internals are subject to change.  unicorn is designed to host Rack
 # applications, so applications should be written against the Rack SPEC
-# and not \Unicorn internals.
+# and not unicorn internals.
 module Unicorn
 
   # Raised inside TeeInput when a client closes the socket inside the
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 02f6b6b..4da19bb 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -1,7 +1,7 @@
 # -*- encoding: binary -*-
 require 'logger'
 
-# Implements a simple DSL for configuring a \Unicorn server.
+# Implements a simple DSL for configuring a unicorn server.
 #
 # See http://unicorn.bogomips.org/examples/unicorn.conf.rb and
 # http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
@@ -282,20 +282,19 @@ class Unicorn::Configurator
   #   Setting this to +true+ can make streaming responses in Rails 3.1
   #   appear more quickly at the cost of slightly higher bandwidth usage.
   #   The effect of this option is most visible if nginx is not used,
-  #   but nginx remains highly recommended with \Unicorn.
+  #   but nginx remains highly recommended with unicorn.
   #
   #   This has no effect on UNIX sockets.
   #
-  #   Default: +true+ (Nagle's algorithm disabled) in \Unicorn,
-  #   +true+ in Rainbows!  This defaulted to +false+ in \Unicorn
-  #   3.x
+  #   Default: +true+ (Nagle's algorithm disabled) in unicorn
+  #   This defaulted to +false+ in unicorn 3.x
   #
   # [:tcp_nopush => true or false]
   #
   #   Enables/disables TCP_CORK in Linux or TCP_NOPUSH in FreeBSD
   #
   #   This prevents partial TCP frames from being sent out and reduces
-  #   wakeups in nginx if it is on a different machine.  Since \Unicorn
+  #   wakeups in nginx if it is on a different machine.  Since unicorn
   #   is only designed for applications that send the response body
   #   quickly without keepalive, sockets will always be flushed on close
   #   to prevent delays.
@@ -303,7 +302,7 @@ class Unicorn::Configurator
   #   This has no effect on UNIX sockets.
   #
   #   Default: +false+
-  #   This defaulted to +true+ in \Unicorn 3.4 - 3.7
+  #   This defaulted to +true+ in unicorn 3.4 - 3.7
   #
   # [:ipv6only => true or false]
   #
@@ -387,12 +386,10 @@ class Unicorn::Configurator
   #   and +false+ or +nil+ is synonymous for a value of zero.
   #
   #   A value of +1+ is a good optimization for local networks
-  #   and trusted clients.  For Rainbows! and Zbatery users, a higher
-  #   value (e.g. +60+) provides more protection against some
-  #   denial-of-service attacks.  There is no good reason to ever
-  #   disable this with a +zero+ value when serving HTTP.
+  #   and trusted clients.  There is no good reason to ever
+  #   disable this with a +zero+ value with unicorn.
   #
-  #   Default: 1 retransmit for \Unicorn, 60 for Rainbows! 0.95.0\+
+  #   Default: 1
   #
   # [:accept_filter => String]
   #
@@ -401,8 +398,7 @@ class Unicorn::Configurator
   #   This enables either the "dataready" or (default) "httpready"
   #   accept() filter under FreeBSD.  This is intended as an
   #   optimization to reduce context switches with common GET/HEAD
-  #   requests.  For Rainbows! and Zbatery users, this provides
-  #   some protection against certain denial-of-service attacks, too.
+  #   requests.
   #
   #   There is no good reason to change from the default.
   #
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 0f97516..3dbfd3e 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -7,8 +7,8 @@
 #
 # Users do not need to know the internals of this class, but reading the
 # {source}[http://bogomips.org/unicorn.git/tree/lib/unicorn/http_server.rb]
-# is education for programmers wishing to learn how \Unicorn works.
-# See Unicorn::Configurator for information on how to configure \Unicorn.
+# is education for programmers wishing to learn how unicorn works.
+# See Unicorn::Configurator for information on how to configure unicorn.
 class Unicorn::HttpServer
   # :stopdoc:
   attr_accessor :app, :timeout, :worker_processes,
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 812ac53..df8315e 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -5,14 +5,12 @@ require 'socket'
 module Unicorn
   module SocketHelper
 
-    # internal interface, only used by Rainbows!/Zbatery
+    # internal interface
     DEFAULTS = {
       # The semantics for TCP_DEFER_ACCEPT changed in Linux 2.6.32+
       # with commit d1b99ba41d6c5aa1ed2fc634323449dd656899e9
-      # This change shouldn't affect Unicorn users behind nginx (a
-      # value of 1 remains an optimization), but Rainbows! users may
-      # want to use a higher value on Linux 2.6.32+ to protect against
-      # denial-of-service attacks
+      # This change shouldn't affect unicorn users behind nginx (a
+      # value of 1 remains an optimization).
       :tcp_defer_accept => 1,
 
       # FreeBSD, we need to override this to 'dataready' if we
diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index c7784bd..2f8bfeb 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -1,7 +1,7 @@
 # -*- encoding: binary -*-
 
 require 'fcntl'
-module Unicorn::Util
+module Unicorn::Util # :nodoc:
 
 # :stopdoc:
   def self.is_log?(fp)
diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb
index b3f8afe..6748a2f 100644
--- a/lib/unicorn/worker.rb
+++ b/lib/unicorn/worker.rb
@@ -3,8 +3,8 @@ require "raindrops"
 
 # This class and its members can be considered a stable interface
 # and will not change in a backwards-incompatible fashion between
-# releases of \Unicorn.  Knowledge of this class is generally not
-# not needed for most users of \Unicorn.
+# releases of unicorn.  Knowledge of this class is generally not
+# not needed for most users of unicorn.
 #
 # Some users may want to access it in the before_fork/after_fork hooks.
 # See the Unicorn::Configurator RDoc for examples.
-- 
EW


^ permalink raw reply related	[relevance 12%]

* Re: TAN: Coquelicot
  2015-06-30 23:19  0%       ` TAN: Coquelicot Eric Wong
@ 2015-07-01 11:54  4%         ` Lunar
  0 siblings, 0 replies; 191+ results
From: Lunar @ 2015-07-01 11:54 UTC (permalink / raw)
  To: Eric Wong; +Cc: Hleb Valoshka, unicorn-public, rainbows-public

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

Eric Wong:
> Lunar <lunar@anargeek.net> wrote:
> > Rainbows! is used by Coquelicot:
> > https://coquelicot.potager.org/
> > 
> > The project needs some love, but I know about several installations that
> > are used on a daily basis.
> 
> Interesting!  I'm honestly a bit disappointed it's not a generic Rack
> app and depends on certain server features, but the project seems mostly
> inline with my interests.

In the very first versions, it was a generic Rack app. And then I
discovered that the file that was transmitted was saved in clear before
being given to the handler (where it would get encrypted)…
Finding that Rainbows! could be made to process the incoming bytes
directly really helped in fixing this issue.

The choice of tying it to a single HTTP server is also a conscious
decision to make it easy to install. On Debian, it only requires
`apt-get install coquelicot` and then 3 lines in Apache configuration.

> But if I were to run it, I'd remove all CSS+images+JS and might
> contribute a patch to disable that all, too :)

Why not. :)

-- 
Lunar


^ permalink raw reply	[relevance 4%]

* TAN: Coquelicot
  2015-06-24  9:19  4%     ` Lunar
@ 2015-06-30 23:19  0%       ` Eric Wong
  2015-07-01 11:54  4%         ` Lunar
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2015-06-30 23:19 UTC (permalink / raw)
  To: Lunar; +Cc: Hleb Valoshka, unicorn-public, rainbows-public

Lunar <lunar@anargeek.net> wrote:
> Rainbows! is used by Coquelicot:
> https://coquelicot.potager.org/
> 
> The project needs some love, but I know about several installations that
> are used on a daily basis.

Interesting!  I'm honestly a bit disappointed it's not a generic Rack
app and depends on certain server features, but the project seems mostly
inline with my interests.

But if I were to run it, I'd remove all CSS+images+JS and might
contribute a patch to disable that all, too :)

Anyways, subscribed to the mailing list :>

^ permalink raw reply	[relevance 0%]

* Re: [ANN] unicorn 5.0.0.pre1 - incompatible changes!
  2015-06-23 20:04  6%   ` Eric Wong
  2015-06-23 20:42  0%     ` Damian Janowski
@ 2015-06-24  9:19  4%     ` Lunar
  2015-06-30 23:19  0%       ` TAN: Coquelicot Eric Wong
  1 sibling, 1 reply; 191+ results
From: Lunar @ 2015-06-24  9:19 UTC (permalink / raw)
  To: Eric Wong; +Cc: Hleb Valoshka, unicorn-public, rainbows-public

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

Eric Wong:
> Hleb Valoshka <375gnu@gmail.com> wrote:
> > On 6/16/15, Eric Wong <e@80x24.org> wrote:
> > > ...
> > > * const: drop constants used by Rainbows!
> > 
> > So, does it mean that Rainbows! can use Unicorn 4.x only?
> 
> For now, yes.  However, I am likely to make a maintenance release of
> Rainbows! 5.x to work with unicorn 5.x so it can stay in Debian
> if anybody uses it.
> 
> But as far as I know, nobody has ever used Rainbows! for production in
> its 6 years of existence...

Rainbows! is used by Coquelicot:
https://coquelicot.potager.org/

The project needs some love, but I know about several installations that
are used on a daily basis.

-- 
Lunar


^ permalink raw reply	[relevance 4%]

* Re: [ANN] unicorn 5.0.0.pre1 - incompatible changes!
  2015-06-23 20:04  6%   ` Eric Wong
@ 2015-06-23 20:42  0%     ` Damian Janowski
  2015-06-24  9:19  4%     ` Lunar
  1 sibling, 0 replies; 191+ results
From: Damian Janowski @ 2015-06-23 20:42 UTC (permalink / raw)
  To: Eric Wong; +Cc: Hleb Valoshka, unicorn-public, rainbows-public

On Tue, Jun 23, 2015 at 5:04 PM, Eric Wong <e@80x24.org> wrote:
> But as far as I know, nobody has ever used Rainbows! for production in
> its 6 years of existence...

o/

I have, for a couple of years at least. Then changed back to Unicorn.

^ permalink raw reply	[relevance 0%]

* Re: [ANN] unicorn 5.0.0.pre1 - incompatible changes!
  2015-06-23 19:57  4% ` Hleb Valoshka
@ 2015-06-23 20:04  6%   ` Eric Wong
  2015-06-23 20:42  0%     ` Damian Janowski
  2015-06-24  9:19  4%     ` Lunar
  0 siblings, 2 replies; 191+ results
From: Eric Wong @ 2015-06-23 20:04 UTC (permalink / raw)
  To: Hleb Valoshka; +Cc: unicorn-public, rainbows-public

Hleb Valoshka <375gnu@gmail.com> wrote:
> On 6/16/15, Eric Wong <e@80x24.org> wrote:
> > ...
> > * const: drop constants used by Rainbows!
> 
> So, does it mean that Rainbows! can use Unicorn 4.x only?

For now, yes.  However, I am likely to make a maintenance release of
Rainbows! 5.x to work with unicorn 5.x so it can stay in Debian
if anybody uses it.

But as far as I know, nobody has ever used Rainbows! for production in
its 6 years of existence...

^ permalink raw reply	[relevance 6%]

* Re: [ANN] unicorn 5.0.0.pre1 - incompatible changes!
  2015-06-15 22:56  4% [ANN] unicorn 5.0.0.pre1 - incompatible changes! Eric Wong
@ 2015-06-23 19:57  4% ` Hleb Valoshka
  2015-06-23 20:04  6%   ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Hleb Valoshka @ 2015-06-23 19:57 UTC (permalink / raw)
  To: unicorn-public

On 6/16/15, Eric Wong <e@80x24.org> wrote:
> ...
> * const: drop constants used by Rainbows!

So, does it mean that Rainbows! can use Unicorn 4.x only?

^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 5.0.0.pre1 - incompatible changes!
@ 2015-06-15 22:56  4% Eric Wong
  2015-06-23 19:57  4% ` Hleb Valoshka
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2015-06-15 22:56 UTC (permalink / raw)
  To: unicorn-public

This release finally drops Ruby 1.8 support and requires Ruby 1.9.3
or later.  The horrible "Status:" header in our HTTP response is
finally gone, saving at least 16 precious bytes in every single HTTP
response.

Under Ruby 2.1 and later, the monotonic clock is used for timeout
handling for better accuracy.

Several experimental, unused and undocumented features are removed.

There's also tiny, minor performance and memory improvements from
dropping 1.8 compatibility, but probably nothing noticeable on a
typical real-life (bloated) app.

The biggest performance improvement we made was to our website by
switching to olddoc.  Depending on connection speed, latency, and
renderer performance, it typically loads two to four times faster.

Finally, for the billionth time: unicorn must never be exposed
to slow clients, as it will never ever use new-fangled things
like non-blocking socket I/O, threads, epoll or kqueue.  unicorn
must be used with a fully-buffering reverse proxy such as nginx
for slow clients.

I'll tag 5.0.0 final in a week or so if all goes well

= gem install --pre unicorn
= git clone git://bogomips.org/unicorn.git
= http://unicorn.bogomips.org/

* ISSUES: update with mailing list subscription
* GIT-VERSION-GEN: start 5.0.0 development
* http: remove xftrust options
* FAQ: add entry for Rails autoflush_log
* dev: remove isolate dependency
* unicorn.gemspec: depend on test-unit 3.0
* http_response: remove Status: header
* remove RubyForge and Freecode references
* remove mongrel.rubyforge.org references
* http: remove the keepalive requests limit
* http: reduce parser from 72 to 56 bytes on 64-bit
* examples: add run_once to before_fork hook example
* worker: remove old tmp accessor
* http_server: save 450+ bytes of memory on x86-64
* t/t0002-parser-error.sh: relax test for rack 1.6.0
* remove SSL support
* tmpio: drop the "size" method
* switch docs + website to olddoc
* README: clarify/reduce references to unicorn_rails
* gemspec: fixup olddoc migration
* use the monotonic clock under Ruby 2.1+
* http: -Wshorten-64-to-32 warnings on clang
* remove old inetd+git examples and exec_cgi
* http: standalone require + reduction in binary size
* GNUmakefile: fix clean gem build + reduce build cruft
* socket_helper: reduce constant lookups and caching
* remove 1.8, <= 1.9.1 fallback for missing IO#autoclose=
* favor IO#close_on_exec= over fcntl in 1.9+
* use require_relative to reduce syscalls at startup
* doc: update support status for Ruby versions
* fix uninstalled testing and reduce require paths
* test_socket_helper: do not depend on SO_REUSEPORT
* favor "a.b(&:c)" form over "a.b { |x| x.c }"
* ISSUES: add section for bugs in other projects
* http_server: favor ivars over constants
* explain 11 byte magic number for self-pipe
* const: drop constants used by Rainbows!
* reduce and localize constant string use
* Links: mark Rainbows! as historical, reference yahns
* save about 200 bytes of memory on x86-64
* http: remove deprecated reset method
* http: remove experimental dechunk! method
* socket_helper: update comments
* doc: document UNICORN_FD in manpage
* doc: document Etc.nprocessors for worker_processes
* favor more string literals for cold call sites
* tee_input: support for Rack::TempfileReaper middleware
* support TempfileReaper in deployment and development envs
* favor kgio_wait_readable for single FD over select
* Merge tag 'v4.9.0'
* http_request: support rack.hijack by default
* avoid extra allocation for hijack proc creation
* FAQ: add note about ECONNRESET errors from bodies
* process SIGWINCH unless stdin is a TTY
* ISSUES: discourage HTML mail strongly, welcome nyms
* http: use rb_hash_clear in Ruby 2.0+
* http_response: avoid special-casing for Rack < 1.5
* www: install NEWS.atom.xml properly
* http_server: remove a few more accessors and constants
* http_response: simplify regular expression
* move the socket into Rack env for hijacking
* http: move response_start_sent into the C ext
* FAQ: reorder bit on Rack 1.1.x and Rails 2.3.x
* ensure body is closed during hijack

-- 
EW

^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 4.9.0 - Rack HTTP server for fast clients and *nix
@ 2015-04-24  3:17  3% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2015-04-24  3:17 UTC (permalink / raw)
  To: ruby-talk; +Cc: unicorn-public, Mike Mulvaney

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* http://unicorn.bogomips.org/
* public list: unicorn-public@bogomips.org
* mail archives: http://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml

Changes:

  unicorn 4.9.0 - TempfileReaper support in Rack 1.6

  This release supports the Rack::TempfileReaper middleware found
  in rack 1.6 for cleaning up disk space used by temporary files.
  We also use Rack::TempfileReaper for cleaning up large temporary
  files buffered with TeeInput.  Users on rack 1.5 and earlier
  will see no changes.

  There's also a bunch of documentation/build system improvements.

  This is likely to be the last Ruby 1.8-compatible release, unicorn 5.x
  will require 1.9.3 or later as well as dropping lots of cruft (the
  stupid "Status:" header in responses being the most notable).

  21 changes backported from master:

        ISSUES: update with mailing list subscription
        FAQ: add entry for Rails autoflush_log
        dev: remove isolate dependency
        unicorn.gemspec: depend on test-unit 3.0
        remove RubyForge and Freecode references
        remove mongrel.rubyforge.org references
        examples: add run_once to before_fork hook example
        t/t0002-parser-error.sh: relax test for rack 1.6.0
        switch docs + website to olddoc
        README: clarify/reduce references to unicorn_rails
        gemspec: fixup olddoc migration
        GNUmakefile: fix clean gem build + reduce build cruft
        doc: update support status for Ruby versions
        fix uninstalled testing and reduce require paths
        test_socket_helper: do not depend on SO_REUSEPORT
        ISSUES: add section for bugs in other projects
        explain 11 byte magic number for self-pipe
        Links: mark Rainbows! as historical, reference yahns
        doc: document UNICORN_FD in manpage
        tee_input: support for Rack::TempfileReaper middleware
        support TempfileReaper in deployment and development envs
-- 
EW

^ permalink raw reply	[relevance 3%]

* unicorn 4.8.x-stable branch pushed to git
@ 2015-04-22 19:02  3% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2015-04-22 19:02 UTC (permalink / raw)
  To: unicorn-public; +Cc: Mulvaney, Mike

Only backporting documentation + build/test system fixes, but I'll
probably apply and push the TeeInput patch in:
http://bogomips.org/unicorn-public/m/20150422183808.GA5277@dcvr.yhbt.net.txt

The following changes since commit 7087bb7ed5a1b9d9f24069cb92707d086668b6dc:

  unicorn 4.8.3 - the end of an era (2014-05-07 07:49:19 +0000)

are available in the git repository at:

  git://bogomips.org/unicorn 4.8.x-stable

for you to fetch changes up to 548e1e67d314f6ebd17df37ece0ee20632462f6f:

  doc: document UNICORN_FD in manpage (2015-04-22 18:57:39 +0000)

----------------------------------------------------------------
Eric Wong (19):
      ISSUES: update with mailing list subscription
      FAQ: add entry for Rails autoflush_log
      dev: remove isolate dependency
      unicorn.gemspec: depend on test-unit 3.0
      remove RubyForge and Freecode references
      remove mongrel.rubyforge.org references
      examples: add run_once to before_fork hook example
      t/t0002-parser-error.sh: relax test for rack 1.6.0
      switch docs + website to olddoc
      README: clarify/reduce references to unicorn_rails
      gemspec: fixup olddoc migration
      GNUmakefile: fix clean gem build + reduce build cruft
      doc: update support status for Ruby versions
      fix uninstalled testing and reduce require paths
      test_socket_helper: do not depend on SO_REUSEPORT
      ISSUES: add section for bugs in other projects
      explain 11 byte magic number for self-pipe
      Links: mark Rainbows! as historical, reference yahns
      doc: document UNICORN_FD in manpage

 .document                             |  1 -
 .gitignore                            |  4 +-
 .wrongdoc.yml => .olddoc.yml          |  6 ++-
 Documentation/unicorn.1.txt           |  7 ++++
 FAQ                                   | 10 ++++-
 GNUmakefile                           | 71 +++++++++++++----------------------
 HACKING                               | 31 ++++-----------
 ISSUES                                | 46 +++++++++++++++++++++--
 KNOWN_ISSUES                          | 20 +++++-----
 Links                                 |  5 ++-
 README                                | 10 ++---
 Rakefile                              | 44 ----------------------
 Sandbox                               |  2 +-
 examples/unicorn.conf.rb              | 11 ++++++
 lib/unicorn/configurator.rb           |  2 -
 lib/unicorn/http_server.rb            |  6 ++-
 local.mk.sample                       | 59 -----------------------------
 script/isolate_for_tests              | 31 ---------------
 t/GNUmakefile                         |  6 +--
 t/README                              |  2 +-
 t/t0002-parser-error.sh               |  6 +--
 test/exec/test_exec.rb                |  2 +-
 test/test_helper.rb                   |  4 +-
 test/unit/test_http_parser.rb         |  6 +--
 test/unit/test_http_parser_ng.rb      |  2 +-
 test/unit/test_http_parser_xftrust.rb |  2 +-
 test/unit/test_request.rb             |  2 +-
 test/unit/test_response.rb            |  6 +--
 test/unit/test_server.rb              |  6 +--
 test/unit/test_signals.rb             |  2 +-
 test/unit/test_socket_helper.rb       |  8 ++--
 test/unit/test_upload.rb              |  2 +-
 test/unit/test_util.rb                |  2 +-
 unicorn.gemspec                       | 13 +++----
 34 files changed, 167 insertions(+), 270 deletions(-)
 rename .wrongdoc.yml => .olddoc.yml (85%)
 delete mode 100644 local.mk.sample
 delete mode 100755 script/isolate_for_tests

^ permalink raw reply	[relevance 3%]

* [PATCH 1/2] const: drop constants used by Rainbows!
@ 2015-02-09  9:12 21% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2015-02-09  9:12 UTC (permalink / raw)
  To: unicorn-public

Rainbows! (in maintenance mode) will need to define it's own
constants in the future.  We'll trim down our constant usage in
subsequent commits as we take advantage of Ruby VM improvements.
---
 lib/unicorn/const.rb | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index 26fa62b..e24b511 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -25,14 +25,6 @@ module Unicorn::Const
   MAX_BODY = 1024 * 112
 
   # :stopdoc:
-  # common errors we'll send back
-  # (N.B. these are not used by unicorn, but we won't drop them until
-  #  unicorn 5.x to avoid breaking Rainbows!).
-  ERROR_400_RESPONSE = "HTTP/1.1 400 Bad Request\r\n\r\n"
-  ERROR_414_RESPONSE = "HTTP/1.1 414 Request-URI Too Long\r\n\r\n"
-  ERROR_413_RESPONSE = "HTTP/1.1 413 Request Entity Too Large\r\n\r\n"
-  ERROR_500_RESPONSE = "HTTP/1.1 500 Internal Server Error\r\n\r\n"
-
   EXPECT_100_RESPONSE = "HTTP/1.1 100 Continue\r\n\r\n"
   EXPECT_100_RESPONSE_SUFFIXED = "100 Continue\r\n\r\nHTTP/1.1 "
 
-- 
EW


^ permalink raw reply related	[relevance 21%]

* [PATCH] remove 1.8, <= 1.9.1 fallback for missing IO#autoclose=
@ 2015-02-05 18:01  6% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2015-02-05 18:01 UTC (permalink / raw)
  To: unicorn-public

We're requiring Ruby 1.9.3+, so we can safely depend
on IO#autoclose= being available in 1.9+ and shave off
some bloat.
---
 lib/unicorn/http_server.rb   |  9 +++------
 lib/unicorn/socket_helper.rb | 14 +-------------
 2 files changed, 4 insertions(+), 19 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index e0c0370..8a295f0 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -156,9 +156,6 @@ class Unicorn::HttpServer
 
     LISTENERS.delete_if do |io|
       if dead_names.include?(sock_name(io))
-        IO_PURGATORY.delete_if do |pio|
-          pio.fileno == io.fileno && (pio.close rescue nil).nil? # true
-        end
         (io.close rescue nil).nil? # true
       else
         set_server_sockopt(io, listener_opts[sock_name(io)])
@@ -239,7 +236,7 @@ class Unicorn::HttpServer
     begin
       io = bind_listen(address, opt)
       unless Kgio::TCPServer === io || Kgio::UNIXServer === io
-        prevent_autoclose(io)
+        io.autoclose = false
         io = server_cast(io)
       end
       logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
@@ -453,7 +450,7 @@ class Unicorn::HttpServer
       (3..1024).each do |io|
         next if listener_fds.include?(io)
         io = IO.for_fd(io) rescue next
-        prevent_autoclose(io)
+        io.autoclose = false
         io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
       end
 
@@ -770,7 +767,7 @@ class Unicorn::HttpServer
     inherited = ENV['UNICORN_FD'].to_s.split(',').map do |fd|
       io = Socket.for_fd(fd.to_i)
       set_server_sockopt(io, listener_opts[sock_name(io)])
-      prevent_autoclose(io)
+      io.autoclose = false
       logger.info "inherited addr=#{sock_name(io)} fd=#{fd}"
       server_cast(io)
     end
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 2ecf438..a4247ac 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -5,10 +5,6 @@ require 'socket'
 module Unicorn
   module SocketHelper
 
-    # prevents IO objects in here from being GC-ed
-    # kill this when we drop 1.8 support
-    IO_PURGATORY = []
-
     # internal interface, only used by Rainbows!/Zbatery
     DEFAULTS = {
       # The semantics for TCP_DEFER_ACCEPT changed in Linux 2.6.32+
@@ -36,14 +32,6 @@ module Unicorn
       [ af_name, nil ].pack('a16a240')
     end if RUBY_PLATFORM =~ /freebsd/ && Socket.const_defined?(:SO_ACCEPTFILTER)
 
-    def prevent_autoclose(io)
-      if io.respond_to?(:autoclose=)
-        io.autoclose = false
-      else
-        IO_PURGATORY << io
-      end
-    end
-
     def set_tcp_sockopt(sock, opt)
       # just in case, even LANs can break sometimes.  Linux sysadmins
       # can lower net.ipv4.tcp_keepalive_* sysctl knobs to very low values.
@@ -161,7 +149,7 @@ module Unicorn
         sock.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1)
       end
       sock.bind(Socket.pack_sockaddr_in(port, addr))
-      prevent_autoclose(sock)
+      sock.autoclose = false
       Kgio::TCPServer.for_fd(sock.fileno)
     end
 
-- 
EW


^ permalink raw reply related	[relevance 6%]

* [PATCH 4/4] http: reduce parser from 72 to 56 bytes on 64-bit
    2014-09-22  1:05  7% ` [PATCH 3/4] http: remove the keepalive requests limit Eric Wong
@ 2014-09-22  1:05  8% ` Eric Wong
  1 sibling, 0 replies; 191+ results
From: Eric Wong @ 2014-09-22  1:05 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong

This allows the parser struct to fit in one cache line on
x86-64 systems where cache lines are 64 bytes.

Using 32-bit integer lengths is safe here because these are only for
tracking offsets within the HTTP header buffer.  We can safely limit
HTTP headers and in-memory buffers to be less than 4GB without
anybody complaining.

HTTP bodies continue to use off_t (usually 64-bit, even on 32-bit
systems) sizes and support as much as the OS/hardware can handle.
---
 ext/unicorn_http/unicorn_http.rl | 18 +++++++++---------
 test/unit/test_http_parser_ng.rb |  6 ++++++
 2 files changed, 15 insertions(+), 9 deletions(-)

diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 9372d39..3294357 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -29,27 +29,27 @@ void init_unicorn_httpdate(void);
 /* all of these flags need to be set for keepalive to be supported */
 #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
 
-static size_t MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
+static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
 
 /* this is only intended for use with Rainbows! */
 static VALUE set_maxhdrlen(VALUE self, VALUE len)
 {
-  return SIZET2NUM(MAX_HEADER_LEN = NUM2SIZET(len));
+  return UINT2NUM(MAX_HEADER_LEN = NUM2UINT(len));
 }
 
 /* keep this small for Rainbows! since every client has one */
 struct http_parser {
   int cs; /* Ragel internal state */
   unsigned int flags;
-  size_t mark;
-  size_t offset;
+  unsigned int mark;
+  unsigned int offset;
   union { /* these 2 fields don't nest */
-    size_t field;
-    size_t query;
+    unsigned int field;
+    unsigned int query;
   } start;
   union {
-    size_t field_len; /* only used during header processing */
-    size_t dest_offset; /* only used during body processing */
+    unsigned int field_len; /* only used during header processing */
+    unsigned int dest_offset; /* only used during body processing */
   } s;
   VALUE buf;
   VALUE env;
@@ -74,7 +74,7 @@ static void parser_raise(VALUE klass, const char *msg)
 }
 
 #define REMAINING (unsigned long)(pe - p)
-#define LEN(AT, FPC) (FPC - buffer - hp->AT)
+#define LEN(AT, FPC) (FPC - buffer - (unsigned long)hp->AT)
 #define MARK(M,FPC) (hp->M = (FPC) - buffer)
 #define PTR_TO(F) (buffer + hp->F)
 #define STR_NEW(M,FPC) rb_str_new(PTR_TO(M), LEN(M, FPC))
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 9167845..d5c8d2e 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -11,6 +11,12 @@ class HttpParserNgTest < Test::Unit::TestCase
     @parser = HttpParser.new
   end
 
+  def test_parser_max_len
+    assert_raises(RangeError) do
+      HttpParser.max_header_len = 0xffffffff + 1
+    end
+  end
+
   def test_next_clear
     r = "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
     @parser.buf << r
-- 
EW


^ permalink raw reply related	[relevance 8%]

* [PATCH 3/4] http: remove the keepalive requests limit
  @ 2014-09-22  1:05  7% ` Eric Wong
  2014-09-22  1:05  8% ` [PATCH 4/4] http: reduce parser from 72 to 56 bytes on 64-bit Eric Wong
  1 sibling, 0 replies; 191+ results
From: Eric Wong @ 2014-09-22  1:05 UTC (permalink / raw)
  To: unicorn-public; +Cc: Eric Wong

This was a hack for some event loops such as those found in nginx
and some Rainbows! concurrency models.  Using epoll/kqueue with
one-shot notification (which yahns does) avoids all fairness
problems.
---
 ext/unicorn_http/unicorn_http.rl | 37 ++------------------
 test/unit/test_http_parser_ng.rb | 74 +---------------------------------------
 2 files changed, 3 insertions(+), 108 deletions(-)

diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index ecdadb0..9372d39 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -29,29 +29,6 @@ void init_unicorn_httpdate(void);
 /* all of these flags need to be set for keepalive to be supported */
 #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER)
 
-static unsigned long keepalive_requests = 100; /* same as nginx */
-
-/*
- * Returns the maximum number of keepalive requests a client may make
- * before the parser refuses to continue.
- */
-static VALUE ka_req(VALUE self)
-{
-  return ULONG2NUM(keepalive_requests);
-}
-
-/*
- * Sets the maximum number of keepalive requests a client may make.
- * A special value of +nil+ causes this to be the maximum value
- * possible (this is architecture-dependent).
- */
-static VALUE set_ka_req(VALUE self, VALUE val)
-{
-  keepalive_requests = NIL_P(val) ? ULONG_MAX : NUM2ULONG(val);
-
-  return ka_req(self);
-}
-
 static size_t MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */
 
 /* this is only intended for use with Rainbows! */
@@ -64,7 +41,6 @@ static VALUE set_maxhdrlen(VALUE self, VALUE len)
 struct http_parser {
   int cs; /* Ragel internal state */
   unsigned int flags;
-  unsigned long nr_requests;
   size_t mark;
   size_t offset;
   union { /* these 2 fields don't nest */
@@ -580,7 +556,6 @@ static VALUE HttpParser_init(VALUE self)
   http_parser_init(hp);
   hp->buf = rb_str_new(NULL, 0);
   hp->env = rb_hash_new();
-  hp->nr_requests = keepalive_requests;
 
   return self;
 }
@@ -814,15 +789,13 @@ static VALUE HttpParser_keepalive(VALUE self)
  *    parser.next? => true or false
  *
  * Exactly like HttpParser#keepalive?, except it will reset the internal
- * parser state on next parse if it returns true.  It will also respect
- * the maximum *keepalive_requests* value and return false if that is
- * reached.
+ * parser state on next parse if it returns true.
  */
 static VALUE HttpParser_next(VALUE self)
 {
   struct http_parser *hp = data_get(self);
 
-  if ((HP_FL_ALL(hp, KEEPALIVE)) && (hp->nr_requests-- != 0)) {
+  if (HP_FL_ALL(hp, KEEPALIVE)) {
     HP_FL_SET(hp, TO_CLEAR);
     return Qtrue;
   }
@@ -984,12 +957,6 @@ void Init_unicorn_http(void)
    */
   rb_define_const(cHttpParser, "LENGTH_MAX", OFFT2NUM(UH_OFF_T_MAX));
 
-  /* default value for keepalive_requests */
-  rb_define_const(cHttpParser, "KEEPALIVE_REQUESTS_DEFAULT",
-                  ULONG2NUM(keepalive_requests));
-
-  rb_define_singleton_method(cHttpParser, "keepalive_requests", ka_req, 0);
-  rb_define_singleton_method(cHttpParser, "keepalive_requests=", set_ka_req, 1);
   rb_define_singleton_method(cHttpParser, "max_header_len=", set_maxhdrlen, 1);
 
   init_common_fields();
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index ab335ac..9167845 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -8,7 +8,6 @@ include Unicorn
 class HttpParserNgTest < Test::Unit::TestCase
 
   def setup
-    HttpParser.keepalive_requests = HttpParser::KEEPALIVE_REQUESTS_DEFAULT
     @parser = HttpParser.new
   end
 
@@ -29,25 +28,6 @@ class HttpParserNgTest < Test::Unit::TestCase
     assert_equal false, @parser.response_start_sent
   end
 
-  def test_keepalive_requests_default_constant
-    assert_kind_of Integer, HttpParser::KEEPALIVE_REQUESTS_DEFAULT
-    assert HttpParser::KEEPALIVE_REQUESTS_DEFAULT >= 0
-  end
-
-  def test_keepalive_requests_setting
-    HttpParser.keepalive_requests = 0
-    assert_equal 0, HttpParser.keepalive_requests
-    HttpParser.keepalive_requests = nil
-    assert HttpParser.keepalive_requests >= 0xffffffff
-    HttpParser.keepalive_requests = 1
-    assert_equal 1, HttpParser.keepalive_requests
-    HttpParser.keepalive_requests = 666
-    assert_equal 666, HttpParser.keepalive_requests
-
-    assert_raises(TypeError) { HttpParser.keepalive_requests = "666" }
-    assert_raises(TypeError) { HttpParser.keepalive_requests = [] }
-  end
-
   def test_connection_TE
     @parser.buf << "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: TE\r\n"
     @parser.buf << "TE: trailers\r\n\r\n"
@@ -71,41 +51,11 @@ class HttpParserNgTest < Test::Unit::TestCase
       "REQUEST_METHOD" => "GET",
       "QUERY_STRING" => ""
     }.freeze
-    HttpParser::KEEPALIVE_REQUESTS_DEFAULT.times do |nr|
-      @parser.buf << req
-      assert_equal expect, @parser.parse
-      assert @parser.next?
-    end
-    @parser.buf << req
-    assert_equal expect, @parser.parse
-    assert ! @parser.next?
-  end
-
-  def test_fewer_keepalive_requests_with_next?
-    HttpParser.keepalive_requests = 5
-    @parser = HttpParser.new
-    req = "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n".freeze
-    expect = {
-      "SERVER_NAME" => "example.com",
-      "HTTP_HOST" => "example.com",
-      "rack.url_scheme" => "http",
-      "REQUEST_PATH" => "/",
-      "SERVER_PROTOCOL" => "HTTP/1.1",
-      "PATH_INFO" => "/",
-      "HTTP_VERSION" => "HTTP/1.1",
-      "REQUEST_URI" => "/",
-      "SERVER_PORT" => "80",
-      "REQUEST_METHOD" => "GET",
-      "QUERY_STRING" => ""
-    }.freeze
-    5.times do |nr|
+    100.times do |nr|
       @parser.buf << req
       assert_equal expect, @parser.parse
       assert @parser.next?
     end
-    @parser.buf << req
-    assert_equal expect, @parser.parse
-    assert ! @parser.next?
   end
 
   def test_default_keepalive_is_off
@@ -664,28 +614,6 @@ class HttpParserNgTest < Test::Unit::TestCase
     assert_equal "", @parser.buf
   end
 
-  def test_keepalive_requests_disabled
-    req = "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n".freeze
-    expect = {
-      "SERVER_NAME" => "example.com",
-      "HTTP_HOST" => "example.com",
-      "rack.url_scheme" => "http",
-      "REQUEST_PATH" => "/",
-      "SERVER_PROTOCOL" => "HTTP/1.1",
-      "PATH_INFO" => "/",
-      "HTTP_VERSION" => "HTTP/1.1",
-      "REQUEST_URI" => "/",
-      "SERVER_PORT" => "80",
-      "REQUEST_METHOD" => "GET",
-      "QUERY_STRING" => ""
-    }.freeze
-    HttpParser.keepalive_requests = 0
-    @parser = HttpParser.new
-    @parser.buf << req
-    assert_equal expect, @parser.parse
-    assert ! @parser.next?
-  end
-
   def test_chunk_only
     tmp = ""
     assert_equal @parser, @parser.dechunk!
-- 
EW


^ permalink raw reply related	[relevance 7%]

* unicorn 5 roadmap
@ 2014-05-25  3:52  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2014-05-25  3:52 UTC (permalink / raw)
  To: unicorn-public; +Cc: yahns-public

unicorn is over 5 years old, things are still kicking, but
has not changed much since it was first announced in 2009:
   http://mid.gmane.org/20090211230457.GB22926@dcvr.yhbt.net [1]

So 5 will be the next major version, but there's nothing earth
shattering and probably nothing visible to the majority of users.
Almost all just internal housekeeping:

* (finally) remove Unicorn::HttpParser#reset

* switch to minitest5 (or test-unit2?)

* drop some things that were intended for Rainbows!
  - keepalive_requests, not necessary with the MT/one-shot-based event
    loops like yahns, this is only intended as a DoS mitigation measure for
    single-threaded event loops (nginx) or pure-MT-based servers
  - xftrust (long deprecated)

* cleanup tests, port shell script integration tests to shunit2?
  (or making the tests pure Ruby is probably fine, I trust the stability
  of the Ruby language more today than I did in 2009 with the painful
  1.8->1.9 transition)

* extract terminal-friendly coverage output from yahns into a gem
  and use it for unicorn.

Tangentially related things to do:

* reduce memory usage in Ruby so users may run more unicorns

* improve reverse proxy support in yahns so it may be an
  alternative to nginx for unicorn users

What else?

[1] Supporting the project with only email is probably the only
    reason I haven't given up after all these years.

^ permalink raw reply	[relevance 4%]

* kgio 2.4.0 coming soon
@ 2011-05-05 19:48  7% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-05-05 19:48 UTC (permalink / raw)
  To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw,
	rainbows-talk-GrnCvJ7WPxnNLxjTenLetw

The I/O library used by the latest version Unicorn/Rainbows! got some
OpenBSD portability fixes thanks to Jeremy Evans.  Users of the latest
Rainbows!/Unicorn versions should just be able to update kgio
independently and be on their way if they need OpenBSD fixes.

Summary of changes and reasoning here:

  http://mid.gmane.org/20110505194305.GA29336-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org

Feel free to comment on kgio-TBHja+091Wbby3iVrkZq2A@public.gmane.org[1], reply here, or email me
privately if you're shy.

[1] subscription is required there unfortunately, send a message and
    follow directions to subscribe

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 7%]

* [ANN] kgio library / RubyGem
@ 2010-09-28  3:03  7% Eric Wong
  2010-11-15 18:41  0% ` Eric Wong
  2010-12-22 11:24  0% ` Iñaki Baz Castillo
  0 siblings, 2 replies; 191+ results
From: Eric Wong @ 2010-09-28  3:03 UTC (permalink / raw)
  To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw,
	rainbows-talk-GrnCvJ7WPxnNLxjTenLetw

Hello all,

I've released kgio, a kinder, gentler I/O library for Ruby.  Some of its
features are useful for Unicorn, and all of it is useful for Rainbows!

I intend to make kgio a requirement for both Unicorn and
Rainbows!/Zbatery.  I'm comfortable with the code, but extra testers and
extra eyes to review it would be helpful (it's nearly all C).

There were several factors leading the creation of this library:

  1. The performance loss from Ruby 1.9.2 due to extending exceptions
     with IO::Wait{Read,Writ}able is annoying.  While it's partly
     fixed[1] and fixable[2] for 1.9.3, it could be a while before
     1.9.3 is available.  Unicorn currently does non-blocking accept
     very aggressively.

  2. (Rainbows! only) Slicing partially written strings for
     non-blocking I/O in gets painful with Ruby 1.9, it's much easier
     (at least for me) to do this in C and kgio_trywrite allows
     exactly that, returning the unwritten portion of a string.

  3. (Rainbows! only) It offers hooks for using alternative
     methods to switch execution contexts (experience taken from
     building the FiberSpawn/FiberPool modules).

  4. (philosophical) I've never considered EAGAIN/EINPROGRESS
     "exceptional" conditions when explicitly doing non-blocking
     I/O, and thus unworthy of raising exceptions.


kgio currently passes all tests on on Linux 2.6 (1.9.{1,2},
1.8.{6,7}, rbx 1.1) and FreeBSD 7.0 (1.8.7).

	http://unicorn.bogomips.org/kgio/
	git://git.bogomips.org/kgio.git
	http://git.bogomips.org/cgit/kgio.git

[1] - http://thread.gmane.org/gmane.comp.lang.ruby.core/30854
[2] - http://thread.gmane.org/gmane.comp.lang.ruby.core/31801

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 7%]

* dealing with client disconnects with TeeInput
@ 2009-11-12 10:04  6% Eric Wong
  2009-11-14  1:16  0% ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2009-11-12 10:04 UTC (permalink / raw)
  To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw,
	rainbows-talk-GrnCvJ7WPxnNLxjTenLetw

Foreword: this probably doesn't affect nginx+Unicorn users, which is the
recommended configuration for the vast majority of sites.  It probably
affects Rainbows! users using Thread* or Revactor the most, and probably
some Unicorn users serving Intranet clients directly.

When clients are uploading large files, there's always a good
possibility of them disconnecting before the upload ends.  For other web
app servers it's not much of a problem: they read the entire upload
before attempting to process things; so the app never sees a prematurely
disconnected client.

However Rainbows! and Unicorn have the TeeInput class which allows
real-time processing of uploads as they occur.  Now, we _want_ the
exception to be thrown and application to stop processing the dead
client request immediately.  I've made changes in unicorn.git and
rainbows.git to ensure no EOFError exceptions from the socket are
silenced, not just ones from reading trailers.

However, this means (many more) socket errors will be seen within the
application and any global exception trappers they use will see them as
well.  For Rails (and possibly other frameworks), this can mean very
messy log files with large backtraces.

So, would making a Unicorn::Disconnect < EOFError exception class and
raising it with a short/empty backtrace on EOFErrors be the best way to
go?  That way those global exception trappers can distinguish between
EOFError exceptions raised by Unicorn/Rainbows! itself and other code
that Unicorn/Rainbows does not care about, and log appropriately...

The other option we have is catch/throw.  We can avoid worrying about
the stack trace entirely, and middlewares that opt-in can still capture
and log the disconnect if they want to.  More maintenance overhead for
Rainbows! with all its concurrency models, but this is a situation
where I think catch/throw is appropriate for given the current
middleware/application stacks these days.

Thanks for reading.

-- 
Eric Wong

^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 4.0.0.2.g19f7 prerelease
@ 2011-06-29  7:29  6% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-06-29  7:29 UTC (permalink / raw)
  To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw
  Cc: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw

I just pushed a prerelease gem to RubyGems.org with the following
changes.  Please let us know if there were any other regressions in
4.0.0.

Expect a final 4.0.1 release soonish...

Full git changelog (pushed to git://bogomips.org/unicorn.git):

commit 19f798301ac1884f423640efafb277b071bb5439
Author: Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>
Date:   Wed Jun 29 07:19:32 2011 +0000

    fix per-worker listen directive in after_fork hook
    
    The testcase for this was broken, too, so we didn't notice
    this :<
    
    Reported-by: ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org on the Rainbows! mailing list,
    http://mid.gmane.org/BANLkTi=oQXK5Casq9SuGD3edeUrDPvRm3A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

commit 38672501206c9e64d241e3d8571f70b198f0c1e5
Author: Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>
Date:   Mon Jun 27 20:51:16 2011 +0000

    configurator: truncate timeouts to 32-bit LONG_MAX
    
    IO.select in Ruby can't wait longer than this.  This
    means Unicorn can't support applications that take
    longer than 68 years to respond :(

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Re: Unicorn/Rainbows! mailing list migration
       [not found]     ` <20091029224426.GA12314-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
@ 2009-10-30 22:17  7%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-10-30 22:17 UTC (permalink / raw)
  To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw,
	rainbows-talk-GrnCvJ7WPxnNLxjTenLetw

Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> As I'm sure you've all heard by now, RubyForge will be moving to a
> read-only state and we'll have to migrate our mailing lists somewhere.

OK, it looks like we'll be able to keep our mailing lists on RubyForge.

http://tomcopeland.blogs.com/juniordeveloper/2009/10/things-to-keep-from-rubyforge.html

On the other hand, if anybody just *wants* to move to librelist, let us
know and we can consider it together, too.

Thanks.

-- 
Eric Wong

^ permalink raw reply	[relevance 7%]

* [ANN] unicorn 4.0.1 - regression bugfixes
@ 2011-06-29 19:10  6% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-06-29 19:10 UTC (permalink / raw)
  To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw
  Cc: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw

Changes:

This release fixes things for users of per-worker "listen"
directives in the after_fork hook.  Thanks to ghazel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
for reporting the bug.

The "timeout" configurator directive is now truncated to
0x7ffffffe seconds to prevent overflow when calling
IO.select.

* http://unicorn.bogomips.org/
* mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
* git://bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 1.1.1 - fixing cleanups gone bad :x
       [not found]     ` <20100708080457.GA2345-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
@ 2010-07-11  3:05  7%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-07-11  3:05 UTC (permalink / raw)
  To: mongrel-unicorn-GrnCvJ7WPxnNLxjTenLetw,
	rainbows-talk-GrnCvJ7WPxnNLxjTenLetw

Unicorn::TeeInput constant resolution for Unicorn::ClientError
got broken simplifying code for RDoc.  This affects users
of Rainbows! and Zbatery.

I've also made new minor releases for Rainbows! (0.95.1) and
Zbatery (0.3.1) users to depend on Unicorn 1.1.1

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 7%]

* [RFC] workaround reopen atomicity issues for stdio vs non-stdio
@ 2013-10-20  4:44  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-10-20  4:44 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: rainbows-talk

In multithreaded apps, we must use dup2/dup3 with a temporary
descriptor to reopen log files atomically.  This is the only way
to protect all concurrent userspace access to a file when reopening.

ref: http://bugs.ruby-lang.org/issues/9036
ref: yahns commit bcb10abe53cfb1d6a8ef7daef59eb10ced397c8a
---
 Review of this patch is greatly appreciated.  This doesn't affect most
 unicorn users unless they spawn threads themselves and write to log
 files in their app.  This does affect Rainbows! users who configure
 Rainbows! to use threads, though.


 Also, I guess I should announce yahns on these lists for those not on
 ruby-talk:  http://yahns.yhbt.net/README
 git clone git://yhbt.net/yahns - not for production, yet, but soon
 I can write HTTP servers all day long, really, I just can't stand
 web browsers :P

 lib/unicorn/util.rb | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/util.rb b/lib/unicorn/util.rb
index f84241c..94c4e37 100644
--- a/lib/unicorn/util.rb
+++ b/lib/unicorn/util.rb
@@ -39,7 +39,7 @@ module Unicorn::Util
     to_reopen.each do |fp|
       orig_st = begin
         fp.stat
-      rescue IOError, Errno::EBADF
+      rescue IOError, Errno::EBADF # race
         next
       end
 
@@ -50,8 +50,28 @@ module Unicorn::Util
       end
 
       begin
-        File.open(fp.path, 'a') { |tmpfp| fp.reopen(tmpfp) }
+        # stdin, stdout, stderr are special.  The following dance should
+        # guarantee there is no window where `fp' is unwritable in MRI
+        # (or any correct Ruby implementation).
+        #
+        # Fwiw, GVL has zero bearing here.  This is tricky because of
+        # the unavoidable existence of stdio FILE * pointers for
+        # std{in,out,err} in all programs which may use the standard C library
+        if fp.fileno <= 2
+          # We do not want to hit fclose(3)->dup(2) window for std{in,out,err}
+          # MRI will use freopen(3) here internally on std{in,out,err}
+          fp.reopen(fp.path, "a")
+        else
+          # We should not need this workaround, Ruby can be fixed:
+          #    http://bugs.ruby-lang.org/issues/9036
+          # MRI will not call call fclose(3) or freopen(3) here
+          # since there's no associated std{in,out,err} FILE * pointer
+          # This should atomically use dup3(2) (or dup2(2)) syscall
+          File.open(fp.path, "a") { |tmpfp| fp.reopen(tmpfp) }
+        end
+
         fp.sync = true
+        fp.flush # IO#sync=true may not implicitly flush
         new_st = fp.stat
 
         # this should only happen in the master:
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 4%]

* Re: unicorn simple cgi without rails
  @ 2013-10-10 22:53  6% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-10-10 22:53 UTC (permalink / raw)
  To: unicorn list

nomad Bellcam <damonswirled@hotmail.com> wrote:
> my website is mostly static with some small cgi areas, and i like to
> use ruby for the cgi. when i did my research for the best ruby cgi
> handler for nginx, unicorn figured prominently in my results, and so i
> became interested in trying it. i spent some time reading up on how to
> configure and use it but have been unsuccessful implementing, mostly i
> believe due to the fact that i do not have a rails framework installed
> nor a legitimate rackup config.ru

> my question is this: does it make any sense at all to use unicorn as a
> ruby cgi handler if i am not also using rails?

Yes, but perhaps Rainbows! (a sister project of unicorn) is better
<http://rainbows.rubyforge.org/> since it has less overhead when
waiting for a big CGI to run.  If performance/scalability isn't
an issue, then unicorn is fine for CGI.  unicorn is much easier
to configure as there's less documentation to read :)

> and if there is indeed
> some sense in this idea, how might i go about it? is there a simple
> rackup file that would work for a configuration such as this? i
> couldn't find any information on rackup configs of this sort, and not
> being familiar with rails the terrain simply became to steep at this
> point to continue without some guidance or assurances.

You can check out rack-legacy git://github.com/eric1234/rack-legacy

I've never used rack-legacy, as I've been running the bundled
Unicorn::App::ExecCGI before with cgit <http://git.zx2c4.com/cgit/>.
I haven't used it on anything but cgit, however.

I use the following for serving http://bogomips.org/unicorn.git

(I run this with Rainbows!, but it's fine with unicorn, too)
------------------------ config.ru ------------------------
require "unicorn/app/exec_cgi"
map("http://bogomips.org/") do
  repo = "/path/to/my/git/repos/on/bogomips.org"
  cgit = Unicorn::App::ExecCgi.new("#{repo}/cgit.cgi")

  # Attempts static file serving with Rack::File, first.
  # Fall back to calling cgit if the URL matches .git
  # Otherwise, just return a 404
  r404 = [ 404, [ %w(Content-Length 0), %w(Content-Type text/plain) ], [] ]
  cgit_wrap = lambda { |e| e["PATH_INFO"] =~ %r{\.git} ? cgit.call(e) : r404 }
  run Rack::Cascade.new([ Rack::File.new(repo), cgit_wrap ])
end
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Ruby 2.0 Bad file descriptor (Errno::EBADF)
  2013-08-23  6:47  4%     ` Eric Wong
@ 2013-09-04 19:00  0%       ` Eric Chapweske
  0 siblings, 0 replies; 191+ results
From: Eric Chapweske @ 2013-09-04 19:00 UTC (permalink / raw)
  To: mongrel-unicorn

Eric Wong <normalperson <at> yhbt.net> writes:

> OK, this is really strange; especially since you're only hitting this
> on your legacy app and not a new one.
> 
> I certainly haven't hit this with Ruby 2.0.0 anywhere (neither unicorn
> nor Rainbows!).  I'm fairly certain enough folks are using Ruby 2.0.0 by
> now that we would have more reports if something were amiss.
> 
> Let us know what you find, thanks!


We ran into the same issue. For us, it was because we were executing the 
process using bundle exec.  Bundler doesn't preserve the 1.9 behavior around
FD inheritance. https://github.com/bundler/bundler/issues/2628


_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Ruby 2.0 Bad file descriptor (Errno::EBADF)
  @ 2013-08-23  6:47  4%     ` Eric Wong
  2013-09-04 19:00  0%       ` Eric Chapweske
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2013-08-23  6:47 UTC (permalink / raw)
  To: unicorn list

port <port.himmerland@icloud.com> wrote:
> Eric Wong <normalperson <at> yhbt.net> writes:
> 
> > Did you upgrade to Ruby 2.0.0 before upgrading to unicorn 4.1+?
> 
> we've been running a unicorn 4.6.3 setup on ruby 1.9.3 for quite a while
> with no similar issues. i'll keep digging through our lifecycle config
> to see if i can get closer to the cause.

OK, this is really strange; especially since you're only hitting this
on your legacy app and not a new one.

I certainly haven't hit this with Ruby 2.0.0 anywhere (neither unicorn
nor Rainbows!).  I'm fairly certain enough folks are using Ruby 2.0.0 by
now that we would have more reports if something were amiss.

Let us know what you find, thanks!
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: HTTP streaming and Unicorn timeout
  2013-08-22 15:26  4%   ` Nokan Emiro
  2013-08-22 15:33  4%     ` Hongli Lai
@ 2013-08-22 17:04  0%     ` Eric Wong
  1 sibling, 0 replies; 191+ results
From: Eric Wong @ 2013-08-22 17:04 UTC (permalink / raw)
  To: unicorn list

Nokan Emiro <uzleepito@gmail.com> wrote:
> Thank you for your answer Eric.
> 
> > Using Rack::Timeout with Rainbows! instead of unicorn is probably a
> > better idea.
> 
> 99% of the requests are fast and can be served far below 300 ms.  Only
> a few type of requests need to stream lots of data.  Do you really think that
> unicorn isn't a good choice in this case and I should consider switching to
> something else (Rack::Timeout + Rainbows!, Puma or whatever) just for the
> sake of that 1%?

What Hongli said.  unicorn is only for sending fast responses.  You'll
run into scalability problems very quickly if unicorn is used for any
slow response.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: HTTP streaming and Unicorn timeout
  2013-08-22 15:26  4%   ` Nokan Emiro
@ 2013-08-22 15:33  4%     ` Hongli Lai
  2013-08-22 17:04  0%     ` Eric Wong
  1 sibling, 0 replies; 191+ results
From: Hongli Lai @ 2013-08-22 15:33 UTC (permalink / raw)
  To: unicorn list

Maybe you can use a combination. Use Nginx location blocks to forward
most requests to Unicorn, and forward the streaming URLs to Passenger
Enterprise/Rainbows/Puma/etc.

On Thu, Aug 22, 2013 at 5:26 PM, Nokan Emiro <uzleepito@gmail.com> wrote:
> Thank you for your answer Eric.
>
>> Using Rack::Timeout with Rainbows! instead of unicorn is probably a
>> better idea.
>
> 99% of the requests are fast and can be served far below 300 ms.  Only
> a few type of requests need to stream lots of data.  Do you really think that
> unicorn isn't a good choice in this case and I should consider switching to
> something else (Rack::Timeout + Rainbows!, Puma or whatever) just for the
> sake of that 1%?
>
> u.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying



-- 
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: HTTP streaming and Unicorn timeout
  2013-08-21 16:41  4% ` Eric Wong
@ 2013-08-22 15:26  4%   ` Nokan Emiro
  2013-08-22 15:33  4%     ` Hongli Lai
  2013-08-22 17:04  0%     ` Eric Wong
  0 siblings, 2 replies; 191+ results
From: Nokan Emiro @ 2013-08-22 15:26 UTC (permalink / raw)
  To: unicorn list

Thank you for your answer Eric.

> Using Rack::Timeout with Rainbows! instead of unicorn is probably a
> better idea.

99% of the requests are fast and can be served far below 300 ms.  Only
a few type of requests need to stream lots of data.  Do you really think that
unicorn isn't a good choice in this case and I should consider switching to
something else (Rack::Timeout + Rainbows!, Puma or whatever) just for the
sake of that 1%?

u.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: HTTP streaming and Unicorn timeout
  @ 2013-08-21 16:41  4% ` Eric Wong
  2013-08-22 15:26  4%   ` Nokan Emiro
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2013-08-21 16:41 UTC (permalink / raw)
  To: unicorn list

Nokan Emiro <uzleepito@gmail.com> wrote:
> Hi guys,
> 
> I am working on a Rails app that needs to stream some data to the user and
> I have just found out that Unicorn's timeout feature doesn't respect streaming.
> Content generation is interrupted by Unicorn after the timeout is over.  I
> had to comment out the timeout line in the Unicorn config and use
> Rack::Timeout instead.  This way the streaming isn't interrupted, but the normal
> requests (I mean those that don't use streaming) are forced to finish in the
> defined period of time.
> 
> I'd like to know your opinion about that.  Why is Unicorn cutting off the app
> even if it streams actively to the user?  Is there any better solutions to my
> problem than using Rack::Timeout and switch off timing out in Unicorn?

unicorn isn't appropriate for long-running responses (taking up an
entire process is expensive), it is designed for fast responses to a
fast client able to read it quickly.

Using Rack::Timeout with Rainbows! instead of unicorn is probably a
better idea.

> Is that a bad idea that the timeout counter should restart counting after
> each byte/packet/chunk traverses trough the connection?

That'd be more expensive for the common case of fast responses
(which is what unicorn is designed for)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* [PATCH] unicorn_forever: new executable to respawn masters
@ 2013-07-24  3:11  3% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-07-24  3:11 UTC (permalink / raw)
  To: mongrel-unicorn

Comments/reports of success/failure appreciated.

(Bcc-ing the user who contacted me privately about daemontools :)
--------------------------------8<------------------------------
From: Eric Wong <normalperson@yhbt.net>
Subject: [PATCH] unicorn_forever: new executable to respawn masters

Warning: lightly tested (and not under daemontools/systemd/etc)

This may be useful for daemontools and similar init replacements
which behave badly when the master process is replaced during the
normal SIGUSR2 && SIGQUIT routine.

Usage:

  unicorn_forever EXISTING UNICORN COMMAND-LINE

Example:

  unicorn_forever unicorn -c /path/to/unicorn_config.rb config.ru

It can also be used to keep Rainbows! processes alive as long as
you check for "Rainbows!" constant references in your config file.

  unicorn_forever rainbows -c /path/to/rainbows_config.rb config.ru

Supported signals:

SIGKILL - really kill the unicorn_forever process (unblockable)

SIGSTOP - pause the process, this prevent unicorn_forever from detecting
          or respawning a dead master

SIGTSTP - same as SIGSTOP

SIGCONT - resumes a process stopped by SIGCONT

Those signals above were really implicit to everything, the following
two should be familiar to existing unicorn users.

SIGHUP  - reloads the config (just like regular unicorn).
          This does not touch the existing master process, but allows
	  future masters to be spawned with a different set of listen
	  sockets.

SIGUSR1 - reopens existing log files, this signal is forwarded to the
          regular unicorn master (and thus any workers it has)

All other normal unicorn signals are logged and otherwise ignored.
They are not forwarded to the unicorn master.

To upgrade a unicorn application, just send SIGQUIT (not SIGUSR2) to the
existing master and unicorn_forever will automatically respawn.

There is no way to gracefully upgrade unicorn_forever without losing
connections.  Doing graceful upgrades of unicorn_forever would defeat
the purpose and cause parents (e.g. daemontools) to notice a child
death.

unicorn_forever is probably unnecessary for systemd.  The use of cgroups
with systemd prevents daemons from "escaping" the control of systemd, so
a daemonized unicorn probably remains visible to systemd.

Implementation:

unicorn_forever is stripped down version of unicorn (and the
Unicorn::HttpServer class) which contains enough to:

* parse the config file for listeners (and general validation)
* bind listen sockets
* issue chdir for the working_directory
* set the UNICORN_FD environment variable
* exec the real process (unicorn/rainbows/whatever...)

It does not load nor validate the application.
---
 bin/unicorn_forever    | 126 +++++++++++++++++++++
 lib/unicorn/forever.rb | 289 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 415 insertions(+)
 create mode 100755 bin/unicorn_forever
 create mode 100644 lib/unicorn/forever.rb

diff --git a/bin/unicorn_forever b/bin/unicorn_forever
new file mode 100755
index 0000000..bef3a5f
--- /dev/null
+++ b/bin/unicorn_forever
@@ -0,0 +1,126 @@
+#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
+# -*- encoding: binary -*-
+require 'unicorn'
+require 'unicorn/forever'
+require 'optparse'
+
+rackup_opts = Unicorn::Configurator::RACKUP
+options = rackup_opts[:options]
+
+op = OptionParser.new("", 24, '  ') do |opts|
+  cmd = File.basename($0)
+  opts.banner = "Usage: #{cmd} " \
+                "[ruby options] [#{cmd} options] [rackup config file]"
+  opts.separator "Ruby options:"
+
+  lineno = 1
+  opts.on("-e", "--eval LINE", "evaluate a LINE of code") do |line|
+    eval line, TOPLEVEL_BINDING, "-e", lineno
+    lineno += 1
+  end
+
+  opts.on("-d", "--debug", "set debugging flags (set $DEBUG to true)") do
+    $DEBUG = true
+  end
+
+  opts.on("-w", "--warn", "turn warnings on for your script") do
+    $-w = true
+  end
+
+  opts.on("-I", "--include PATH",
+          "specify $LOAD_PATH (may be used more than once)") do |path|
+    $LOAD_PATH.unshift(*path.split(/:/))
+  end
+
+  opts.on("-r", "--require LIBRARY",
+          "require the library, before executing your script") do |library|
+    require library
+  end
+
+  opts.separator "#{cmd} options:"
+
+  # some of these switches exist for rackup command-line compatibility,
+
+  opts.on("-o", "--host HOST",
+          "listen on HOST (default: #{Unicorn::Const::DEFAULT_HOST})") do |h|
+    rackup_opts[:host] = h
+    rackup_opts[:set_listener] = true
+  end
+
+  opts.on("-p", "--port PORT",
+          "use PORT (default: #{Unicorn::Const::DEFAULT_PORT})") do |p|
+    rackup_opts[:port] = p.to_i
+    rackup_opts[:set_listener] = true
+  end
+
+  opts.on("-E", "--env RACK_ENV",
+          "use RACK_ENV for defaults (default: development)") do |e|
+    ENV["RACK_ENV"] = e
+  end
+
+  opts.on("-N", "--no-default-middleware",
+          "do not load middleware implied by RACK_ENV") do |e|
+    rackup_opts[:no_default_middleware] = true
+  end
+
+  opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
+    rackup_opts[:daemonize] = !!d
+  end
+
+  opts.on("-P", "--pid FILE", "DEPRECATED") do |f|
+    warn %q{Use of --pid/-P is strongly discouraged}
+    warn %q{Use the 'pid' directive in the Unicorn config file instead}
+    options[:pid] = f
+  end
+
+  opts.on("-s", "--server SERVER",
+          "this flag only exists for compatibility") do |s|
+    warn "-s/--server only exists for compatibility with rackup"
+  end
+
+  # Unicorn-specific stuff
+  opts.on("-l", "--listen {HOST:PORT|PATH}",
+          "listen on HOST:PORT or PATH",
+          "this may be specified multiple times",
+          "(default: #{Unicorn::Const::DEFAULT_LISTEN})") do |address|
+    options[:listeners] << address
+  end
+
+  opts.on("-c", "--config-file FILE", "Unicorn-specific config file") do |f|
+    options[:config_file] = f
+  end
+
+  # I'm avoiding Unicorn-specific config options on the command-line.
+  # IMNSHO, config options on the command-line are redundant given
+  # config files and make things unnecessarily complicated with multiple
+  # places to look for a config option.
+
+  opts.separator "Common options:"
+
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts opts.to_s.gsub(/^.*DEPRECATED.*$/s, '')
+    exit
+  end
+
+  opts.on_tail("-v", "--version", "Show version") do
+    puts "#{cmd} v#{Unicorn::Const::UNICORN_VERSION}"
+    exit
+  end
+
+  opts.parse! ARGV
+end
+
+# ARGV[0] is usually "unicorn", but "unicorn_rails" or custom
+# BYOE wrappers work, too
+ru = ARGV[1] || 'config.ru'
+Unicorn::Configurator::RACKUP.merge!(:file => ru, :optparse => op)
+op = nil
+
+if $DEBUG
+  require 'pp'
+  pp({
+    :unicorn_options => options,
+    :daemonize => rackup_opts[:daemonize],
+  })
+end
+Unicorn::Forever.new(options).start.join
diff --git a/lib/unicorn/forever.rb b/lib/unicorn/forever.rb
new file mode 100644
index 0000000..3f6c5fe
--- /dev/null
+++ b/lib/unicorn/forever.rb
@@ -0,0 +1,289 @@
+# -*- encoding: binary -*-
+#
+# Normally the unicorn master process can handle all the restarting,
+# however with init replacements becoming process managers, we may want
+# to use something which never dies and restarts the master.
+class Unicorn::Forever
+  # :stopdoc:
+  include Unicorn::SocketHelper
+
+  attr_accessor :listener_opts, :init_listeners, :config, :logger
+
+  START_CTX = {
+    :argv => ARGV.map { |arg| arg.dup },
+  }
+
+  # We favor ENV['PWD'] since it is (usually) symlink aware for Capistrano
+  # and like systems
+  START_CTX[:cwd] = begin
+    a = File.stat(pwd = ENV['PWD'])
+    b = File.stat(Dir.pwd)
+    a.ino == b.ino && a.dev == b.dev ? pwd : Dir.pwd
+  rescue
+    Dir.pwd
+  end
+
+  def initialize(options = {})
+    @listeners = []
+    @self_pipe = []
+    @new_listeners = []
+    @init_listeners = options[:listeners] ? options[:listeners].dup : []
+    options[:use_defaults] = true
+    @config = Unicorn::Configurator.new(options)
+    @listener_opts = {}
+    @config.commit!(self, :skip => [:listeners])
+    @respawn = true
+    @self_pipe = Kgio::Pipe.new
+    @self_pipe.each { |io| io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
+    # signal queue used for self-piping
+    @sig_queue = []
+  end
+
+  def setup_sighandlers
+    %w(CHLD HUP QUIT TERM INT USR1 USR2 TTIN TTOU WINCH).each do |s|
+      trap(s) { sig_handler(s.to_sym) }
+    end
+  end
+
+  # Runs the thing.  Returns self so you can run join on it
+  def start
+    # no inheriting, just start the default if needed
+    config_listeners = @config[:listeners].dup
+    if config_listeners.empty?
+      config_listeners << Unicorn::Const::DEFAULT_LISTEN
+      @init_listeners << Unicorn::Const::DEFAULT_LISTEN
+      START_CTX[:argv] << "-l#{Unicorn::Const::DEFAULT_LISTEN}"
+    end
+    @new_listeners.replace(config_listeners)
+
+    bind_new_listeners!
+    setup_sighandlers
+    do_exec
+    self
+  end
+
+  # replaces current listener set with +listeners+.  This will
+  # close the socket if it will not exist in the new listener set
+  def listeners=(listeners)
+    cur_names, dead_names = [], []
+    listener_names.each do |name|
+      if ?/ == name[0]
+        # mark unlinked sockets as dead so we can rebind them
+        (File.socket?(name) ? cur_names : dead_names) << name
+      else
+        cur_names << name
+      end
+    end
+    set_names = listener_names(listeners)
+    dead_names.concat(cur_names - set_names).uniq!
+
+    @listeners.delete_if do |io|
+      if dead_names.include?(sock_name(io))
+        IO_PURGATORY.delete_if do |pio|
+          pio.fileno == io.fileno && (pio.close rescue nil).nil? # true
+        end
+        (io.close rescue nil).nil? # true
+      else
+        set_server_sockopt(io, listener_opts[sock_name(io)])
+        false
+      end
+    end
+
+    (set_names - cur_names).each { |addr| listen(addr) }
+  end
+
+  def stdout_path=(path); redirect_io($stdout, path); end
+  def stderr_path=(path); redirect_io($stderr, path); end
+
+  # for Configurator compatibility:
+  def noop(arg = nil); end
+  alias client_body_buffer_size= noop
+  alias client_body_buffer_size noop
+  alias check_client_connection= noop
+  alias check_client_connection noop
+  alias pid= noop
+  alias pid noop
+  alias preload_app= noop
+  alias preload_app noop
+  alias timeout= noop
+  alias timeout noop
+  alias trust_x_forwarded= noop
+  alias trust_x_forwarded noop
+  alias rewindable_input= noop
+  alias rewindable_input noop
+  alias worker_processes= noop
+  alias worker_processes noop
+  alias after_fork= noop
+  alias after_fork noop
+  alias before_fork= noop
+  alias before_fork noop
+  alias before_exec= noop
+  alias before_exec noop
+
+  # add a given address to the +listeners+ set, idempotently
+  # Allows workers to add a private, per-process listener via the
+  # after_fork hook.  Very useful for debugging and testing.
+  # +:tries+ may be specified as an option for the number of times
+  # to retry, and +:delay+ may be specified as the time in seconds
+  # to delay between retries.
+  # A negative value for +:tries+ indicates the listen will be
+  # retried indefinitely, this is useful when workers belonging to
+  # different masters are spawned during a transparent upgrade.
+  def listen(address, opt = {}.merge(listener_opts[address] || {}))
+    address = config.expand_addr(address)
+    return if String === address && listener_names.include?(address)
+
+    delay = opt[:delay] || 0.5
+    tries = opt[:tries] || 5
+    begin
+      io = bind_listen(address, opt)
+      unless Kgio::TCPServer === io || Kgio::UNIXServer === io
+        IO_PURGATORY << io
+        io = server_cast(io)
+      end
+      @logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
+      @listeners << io
+      io
+    rescue Errno::EADDRINUSE => err
+      @logger.error "adding listener failed addr=#{address} (in use)"
+      raise err if tries == 0
+      tries -= 1
+      @logger.error "retrying in #{delay} seconds " \
+                    "(#{tries < 0 ? 'infinite' : tries} tries left)"
+      sleep(delay)
+      retry
+    rescue => err
+      @logger.fatal "error adding listener addr=#{address}"
+      raise err
+    end
+  end
+
+  # reaps all unreaped processes
+  def reap_all
+    begin
+      pid, status = Process.waitpid2(-1, Process::WNOHANG)
+      pid or return
+      @logger.error "reaped #{status.inspect}"
+      @exec_pid = nil if pid == @exec_pid
+    rescue Errno::ECHILD
+      break
+    end while true
+  end
+
+  # Monitors child and receives signals forever (or until SIGKILL is sent).
+  # This handles signals one-at-a-time time.
+  # Send SIGSTOP to this process to prevent it from respawning
+  def join
+    begin
+      IO.select([@self_pipe[0]])
+      @self_pipe[0].kgio_tryread(11)
+      reap_all
+      sig = @sig_queue.shift
+
+      case sig
+      when nil # spurious wakeup
+      when :USR1 # rotate logs
+        @logger.info "unicorn-forever reopening logs..."
+        Unicorn::Util.reopen_logs
+        @logger.info "unicorn-forever done reopening logs"
+
+        # this is the only signal we always forward
+        Process.kill(sig, @exec_pid) if @exec_pid
+      when :HUP
+        # we only implement SIGHUP because we need to be aware of new
+        # sockets from updated configs
+        if @config.config_file
+          load_config!
+        else # exec binary and exit if there's no config file
+          @logger.info "SIGHUP received but config file not defined"
+        end
+      else
+        @logger.info "unhandled signal: SIG#{sig} received"
+        zero = START_CTX[:argv][0]
+        if @exec_pid
+          @logger.info("did you mean to signal `#{zero}' at PID:#@exec_pid?")
+        else
+          @logger.info("`#{zero}' not running")
+        end
+      end
+      unless @exec_pid
+        do_exec
+
+        # throttle, in case the master keeps dying, we don't want to burn
+        # cycles and fill up logs by constant respawning
+        sleep 1
+      end
+    rescue => e
+      Unicorn.log_error(@logger, "forever loop error", e)
+    end while true
+    # We don't go down easily, use SIGKILL
+  end
+
+  def sig_handler(s)
+    @sig_queue << s
+    @self_pipe[1].kgio_trywrite('^') # wakeup ourselves from IO.select
+  end
+
+  def do_exec
+    @exec_pid = fork do # This will become the unicorn master process
+      # Don't let actions on any TTY -forever may have influence us
+      Process.setsid
+
+      listener_fds = Hash[@listeners.map do |sock|
+        # IO#close_on_exec= will be available on any future version of
+        # Ruby that sets FD_CLOEXEC by default on new file descriptors
+        # ref: http://redmine.ruby-lang.org/issues/5041
+        sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
+        [ sock.fileno, sock ]
+      end]
+      ENV['UNICORN_FD'] = listener_fds.keys.join(',')
+      Dir.chdir(START_CTX[:cwd])
+      cmd = START_CTX[:argv]
+
+      # avoid leaking FDs we don't know about, but let before_exec
+      # unset FD_CLOEXEC, if anything else in the app eventually
+      # relies on FD inheritence.
+      (3..1024).each do |io|
+        next if listener_fds.include?(io)
+        io = IO.for_fd(io) rescue next
+        IO_PURGATORY << io
+        io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
+      end
+
+      # exec(command, hash) works in at least 1.9.1+, but will only be
+      # required in 1.9.4/2.0.0 at earliest.
+      cmd << listener_fds if RUBY_VERSION >= "1.9.1"
+      @logger.info "executing #{cmd.inspect} (in #{Dir.pwd})"
+      exec(*cmd)
+    end
+  end
+
+  def load_config!
+    @logger.info "reloading config_file=#{@config.config_file}"
+    @config[:listeners].replace(@init_listeners)
+    @config.reload
+    @config.commit!(self)
+    Unicorn::Util.reopen_logs
+    @logger.info "done reloading config_file=#{@config.config_file}"
+  rescue StandardError, LoadError, SyntaxError => e
+    Unicorn.log_error(@logger,
+        "error reloading config_file=#{@config.config_file}", e)
+  end
+
+  # returns an array of string names for the given listener array
+  def listener_names(listeners = @listeners)
+    listeners.map { |io| sock_name(io) }
+  end
+
+  def redirect_io(io, path)
+    File.open(path, 'ab') { |fp| io.reopen(fp) } if path
+    io.sync = true
+  end
+
+  # This binds any listeners we did NOT inherit from the parent
+  def bind_new_listeners!
+    @new_listeners.each { |addr| listen(addr) }
+    raise ArgumentError, "no listeners" if @listeners.empty?
+    @new_listeners.clear
+  end
+end
-- 
Eric Wong

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 3%]

* Re: [PATCH 1/2] Integration test for --no-default-middleware option
  @ 2013-06-07  9:25  5% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-06-07  9:25 UTC (permalink / raw)
  To: unicorn list; +Cc: Micah Chalmer

Micah Chalmer <micah@micahchalmer.net> wrote:
> This adds an integration test to ensure that the -N option
> continues to function as documented.

Thanks for this fix.  To avoid breaking bisection, I always keep
previously-failing test cases in the same commit as the fix, so I'll
squash your commits together.

Your 2/2 came whitespace mangled, too.  The git-format-patch(1) manpage
has info for popular mailers, and git also comes with a 'send-email'
command for use in conjunction with 'format-patch'.

> +++ b/t/t0300-no-default-middleware.sh
> @@ -0,0 +1,15 @@
> +#!/bin/sh
> +. ./test-lib.sh
> +t_plan 2 "test the -N / --no-default-middleware option"
> +
> +t_begin "setup and start" && {
> +    unicorn_setup
> +    unicorn -N -D -c $unicorn_config fails-rack-lint.ru
> +    unicorn_wait_start

Existing integration tests use hard tabs for indentation,
I'll update your patch to match on my end (my personal preference is
hard tabs, especially for non-Ruby languages).

I'll push out the following unless you have objections:

--------------------------------8<----------------------------------
From: Micah Chalmer <micah@micahchalmer.net>
Subject: [PATCH] Make -N/--no-default-middleware option work

This fixes the -N (a.k.a. --no-defaut-middleware) option, which
was not working.  The problem was that Unicorn::Configurator::RACKUP
is cleared before the lambda returned by Unicorn.builder is run,
which means that checking whether the :no_default_middleware option
was set from the lambda could not detect anything.  This patch copies
it to a local variable that won't get clobbered, restoring the feature.

[ew: squashed test commit into the fix, whitespace fixes]

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 lib/unicorn.rb                   |  6 +++++-
 t/fails-rack-lint.ru             |  5 +++++
 t/t0300-no-default-middleware.sh | 15 +++++++++++++++
 3 files changed, 25 insertions(+), 1 deletion(-)
 create mode 100644 t/fails-rack-lint.ru
 create mode 100644 t/t0300-no-default-middleware.sh

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index f0ceffe..2535159 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -35,6 +35,10 @@ module Unicorn
     # allow Configurator to parse cli switches embedded in the ru file
     op = Unicorn::Configurator::RACKUP.merge!(:file => ru, :optparse => op)
 
+    # Op is going to get cleared before the returned lambda is called, so
+    # save this value so that it's still there when we need it:
+    no_default_middleware = op[:no_default_middleware]
+
     # always called after config file parsing, may be called after forking
     lambda do ||
       inner_app = case ru
@@ -49,7 +53,7 @@ module Unicorn
 
       pp({ :inner_app => inner_app }) if $DEBUG
 
-      return inner_app if op[:no_default_middleware]
+      return inner_app if no_default_middleware
 
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
diff --git a/t/fails-rack-lint.ru b/t/fails-rack-lint.ru
new file mode 100644
index 0000000..82bfb5f
--- /dev/null
+++ b/t/fails-rack-lint.ru
@@ -0,0 +1,5 @@
+# This rack app returns an invalid status code, which will cause
+# Rack::Lint to throw an exception if it is present.  This
+# is used to check whether Rack::Lint is in the stack or not.
+
+run lambda {|env| return [42, {}, ["Rack::Lint wasn't there if you see this"]]}
diff --git a/t/t0300-no-default-middleware.sh b/t/t0300-no-default-middleware.sh
new file mode 100644
index 0000000..c017c16
--- /dev/null
+++ b/t/t0300-no-default-middleware.sh
@@ -0,0 +1,15 @@
+#!/bin/sh
+. ./test-lib.sh
+t_plan 2 "test the -N / --no-default-middleware option"
+
+t_begin "setup and start" && {
+	unicorn_setup
+	unicorn -N -D -c $unicorn_config fails-rack-lint.ru
+	unicorn_wait_start
+}
+
+t_begin "check exit status with Rack::Lint not present" && {
+	test 42 -eq "$(curl -sf -o/dev/null -w'%{http_code}' http://$listen/)"
+}
+
+t_done
-- 
Eric Wong

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 5%]

* [PATCH 2/2] Make -N/--no-default-middleware option work
@ 2013-06-07  3:03  7% Micah Chalmer
  0 siblings, 0 replies; 191+ results
From: Micah Chalmer @ 2013-06-07  3:03 UTC (permalink / raw)
  To: mongrel-unicorn

This fixes the -N (a.k.a. --no-defaut-middleware) option, which
was not working.  The problem was that Unicorn::Configurator::RACKUP
is cleared before the lambda returned by Unicorn.builder is run,
which means that checking whether the :no_default_middleware option
was set from the lambda could not detect anything.  This patch copies
it to a local variable that won't get clobbered, restoring the feature.
---
lib/unicorn.rb | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index f0ceffe..2535159 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -35,6 +35,10 @@ module Unicorn
    # allow Configurator to parse cli switches embedded in the ru file
    op = Unicorn::Configurator::RACKUP.merge!(:file => ru, :optparse => op)

+    # Op is going to get cleared before the returned lambda is called, so
+    # save this value so that it's still there when we need it:
+    no_default_middleware = op[:no_default_middleware]
+
    # always called after config file parsing, may be called after forking
    lambda do ||
      inner_app = case ru
@@ -49,7 +53,7 @@ module Unicorn

      pp({ :inner_app => inner_app }) if $DEBUG

-      return inner_app if op[:no_default_middleware]
+      return inner_app if no_default_middleware

      # return value, matches rackup defaults based on env
      # Unicorn does not support persistent connections, but Rainbows!
-- 
1.8.2

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 7%]

* [PATCH] HttpParser#next? becomes response_start_sent-aware
@ 2013-05-08 23:01  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-05-08 23:01 UTC (permalink / raw)
  To: mongrel-unicorn

This could allow servers with persistent connection support[1]
to support our check_client_connection in the future.

[1] - Rainbows!/zbatery, possibly others
---
 ext/unicorn_http/unicorn_http.rl |  6 ++----
 test/unit/test_http_parser_ng.rb | 17 +++++++++++++++++
 2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 1a8003f..3529740 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -732,10 +732,8 @@ static VALUE HttpParser_parse(VALUE self)
   struct http_parser *hp = data_get(self);
   VALUE data = hp->buf;
 
-  if (HP_FL_TEST(hp, TO_CLEAR)) {
-    http_parser_init(hp);
-    rb_funcall(hp->env, id_clear, 0);
-  }
+  if (HP_FL_TEST(hp, TO_CLEAR))
+    HttpParser_clear(self);
 
   http_parser_execute(hp, RSTRING_PTR(data), RSTRING_LEN(data));
   if (hp->offset > MAX_HEADER_LEN)
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 93c44bb..ab335ac 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -12,6 +12,23 @@ def setup
     @parser = HttpParser.new
   end
 
+  def test_next_clear
+    r = "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
+    @parser.buf << r
+    @parser.parse
+    @parser.response_start_sent = true
+    assert @parser.keepalive?
+    assert @parser.next?
+    assert @parser.response_start_sent
+
+    # persistent client makes another request:
+    @parser.buf << r
+    @parser.parse
+    assert @parser.keepalive?
+    assert @parser.next?
+    assert_equal false, @parser.response_start_sent
+  end
+
   def test_keepalive_requests_default_constant
     assert_kind_of Integer, HttpParser::KEEPALIVE_REQUESTS_DEFAULT
     assert HttpParser::KEEPALIVE_REQUESTS_DEFAULT >= 0
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 4%]

* Re: Fedora Unix socket file location problems
  @ 2013-04-05 21:56  5% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-04-05 21:56 UTC (permalink / raw)
  To: unicorn list; +Cc: David Wilkins

David Wilkins <dwilkins@conecuh.com> wrote:
> Folks,
> 
> It seems that it's pretty common to use /tmp for the directory where
> you store the Unicorn unix: socket file.   I'm a Fedora user and our
> lovable systemd (by default) gives nginx a private /tmp directory (see
> "PrivateTmp=true" in system configuration file example below).
> That's the kind of thing that *could* take a while to figure out.
> 
> Could you put a note somewhere near the listen docs warning Fedora /
> systemd users to locate the socket file somewhere else?  I know it's
> not your problem, but I'll bet that more than a few Fedora users are
> using TCP sockets instead of unix sockets with Unicorn because of
> this.

Thanks, I'll queue up something like the following patch.
Comments/corrections greatly appreciated:

------------------------------8<-------------------------------
From: Eric Wong <normalperson@yhbt.net>
Subject: [PATCH] doc: update documentation for systemd + PrivateTmp users

The PrivateTmp feature of systemd breaks the usage of /tmp for the
shared Unix domain socket between nginx and unicorn, so discourage the
use of /tmp in that case.

While we're at it, use consistent paths for everything and use an
obviously intended-for-user-customization "/path/to" prefix instead
of "/tmp"

ML-Ref: CAKLVLx_t+9zWMhquMWDfStrxS7xrNoGmN0ZDsjSCUE=VxU+oyQ@mail.gmail.com
Reported-by: David Wilkins <dwilkins@conecuh.com>
---
 examples/nginx.conf         |  8 ++++----
 examples/unicorn.conf.rb    |  2 +-
 lib/unicorn/configurator.rb | 10 ++++++++--
 3 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/examples/nginx.conf b/examples/nginx.conf
index 66ac0aa..a68fe6f 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -24,8 +24,8 @@ user nobody nogroup; # for systems with a "nogroup"
 # user nobody nobody; # for systems with "nobody" as a group instead
 
 # Feel free to change all paths to suite your needs here, of course
-pid /tmp/nginx.pid;
-error_log /tmp/nginx.error.log;
+pid /path/to/nginx.pid;
+error_log /path/to/nginx.error.log;
 
 events {
   worker_connections 1024; # increase if you have lots of clients
@@ -42,7 +42,7 @@ http {
   default_type application/octet-stream;
 
   # click tracking!
-  access_log /tmp/nginx.access.log combined;
+  access_log /path/to/nginx.access.log combined;
 
   # you generally want to serve static files with nginx since neither
   # Unicorn nor Rainbows! is optimized for it at the moment
@@ -74,7 +74,7 @@ http {
     # single worker for timing out).
 
     # for UNIX domain socket setups:
-    server unix:/tmp/.sock fail_timeout=0;
+    server unix:/path/to/.unicorn.sock fail_timeout=0;
 
     # for TCP setups, point these to your backend servers
     # server 192.168.0.7:8080 fail_timeout=0;
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index 4042d9c..9dce58a 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -25,7 +25,7 @@
 
 # listen on both a Unix domain socket and a TCP port,
 # we use a shorter backlog for quicker failover when busy
-listen "/tmp/.sock", :backlog => 64
+listen "/path/to/.unicorn.sock", :backlog => 64
 listen 8080, :tcp_nopush => true
 
 # nuke workers after 30 seconds instead of 60 seconds (the default)
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7651093..0d0eac7 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -188,7 +188,7 @@ def before_exec(*args, &block)
   #    # on nginx upstream configuration:
   #    upstream unicorn_backend {
   #      # for UNIX domain socket setups:
-  #      server unix:/path/to/unicorn.sock fail_timeout=0;
+  #      server unix:/path/to/.unicorn.sock fail_timeout=0;
   #
   #      # for TCP setups
   #      server 192.168.0.7:8080 fail_timeout=0;
@@ -229,9 +229,15 @@ def listeners(addresses) # :nodoc:
   #
   #   listen 3000 # listen to port 3000 on all TCP interfaces
   #   listen "127.0.0.1:3000"  # listen to port 3000 on the loopback interface
-  #   listen "/tmp/.unicorn.sock" # listen on the given Unix domain socket
+  #   listen "/path/to/.unicorn.sock" # listen on the given Unix domain socket
   #   listen "[::1]:3000" # listen to port 3000 on the IPv6 loopback interface
   #
+  # When using Unix domain sockets, be sure:
+  # 1) the path matches the one used by nginx
+  # 2) uses the same filesystem namespace as the nginx process
+  # For systemd users using PrivateTmp=true (for either nginx or unicorn),
+  # this means Unix domain sockets must not be placed in /tmp
+  #
   # The following options may be specified (but are generally not needed):
   #
   # [:backlog => number of clients]
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 5%]

* Re: Unicorn hangs on POST request
  2013-04-02 20:25  0%               ` Tom Pesman
@ 2013-04-02 22:36  5%                 ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-04-02 22:36 UTC (permalink / raw)
  To: unicorn list

Tom Pesman <tom@tnux.net> wrote:
> > Probably not, at least it won't improve _consistency_ of performance
> > without changing your app.
> >
> > The problem with buffering in Rainbows!+EM is the buffering happens after
> > the work is distributed to different worker processes (Rainbows!+EM is
> > still single-threaded).
> >
> > I don't believe there's a universal Rainbows! configuration which is
> > good for most apps.  With Rainbows!+EM, the buffering happens in a late
> > stage, after workload is distributed, so you end up with head-of-queue
> > blocking inside each process.
> >
> > All of the modules _without_ "rack.input streaming" will fully buffer
> > responses (similar to how Rainbows!+EM) does it:
> > http://rainbows.rubyforge.org/Summary.html
> >
> > However, CoolioThread{Pool/Spawn} should help you with the work load
> > distribution problem if your app is thread-safe, but these both do body
> > buffering.
> >
> > However, the multithreaded options _with_ "rack.input streaming" still
> > give pretty good performance and let you get around the workload
> > distribution problem of the single-threaded + internal buffering
> > options.  You'll reduce disk seek overhead with input streaming.
> >
> > For reference, nginx+unicorn is single-threaded, but with external
> > buffering (nginx does the buffering, not unicorn).
> 
> Thank you for your extensive response!
> 
> To verify my thoughts on this, if I want to prevent the head-of-queue
> blocking behaviour, I need to take a module without
> rack.input_streaming. But we need to make sure the body is buffered
> before it's given to a worker. On which property do I select the
> correct module?

Erm, no.  You'll have head-of-queue blocking behavior with any
single-threaded Rainbows! option and worker_connections > 1

Unless you rely on Cool.io or EM-specific features in your app,
:EventMachine and :Coolio (and :Epoll/:XEpoll for Linux-only) are the
same from an app-perspective in Rainbows!

Using multi-threaded application dispatch (CoolioThread{Pool,Spawn})
will mitigate and reduce head-of-queue blocking.  Likewise for using
XEpollThreadPool/XEpollThreadSpawn (all the *Epoll* stuff is Linux-only)

Alternatively, you won't get _any_ head-of-queue blocking in a
multi-threaded app, you can use Thread{Pool,Spawn}.

Sorry for the confusion.

> Are Coolio modules still a sensible choice as the Coolio project isn't
> actively maintained?

Fwiw, Cool.io works pretty well in my experience.

Last I checked, EM still had a problem by calling close(2) directly
instead of rb_io_close(), triggering Errno::EBADF.  There's a bug
opened by somebody on the ruby-core team, but I'm not sure if it ever
got resolved...

I can also help fix Cool.io bugs since it's written in C, but I can't
fix EM bugs: C++ is too big for my tiny brain.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Re: Unicorn hangs on POST request
  2013-04-02 17:24  6%             ` Eric Wong
@ 2013-04-02 20:25  0%               ` Tom Pesman
  2013-04-02 22:36  5%                 ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Tom Pesman @ 2013-04-02 20:25 UTC (permalink / raw)
  To: unicorn list

> Probably not, at least it won't improve _consistency_ of performance
> without changing your app.
>
> The problem with buffering in Rainbows!+EM is the buffering happens after
> the work is distributed to different worker processes (Rainbows!+EM is
> still single-threaded).
>
> I don't believe there's a universal Rainbows! configuration which is
> good for most apps.  With Rainbows!+EM, the buffering happens in a late
> stage, after workload is distributed, so you end up with head-of-queue
> blocking inside each process.
>
> All of the modules _without_ "rack.input streaming" will fully buffer
> responses (similar to how Rainbows!+EM) does it:
> http://rainbows.rubyforge.org/Summary.html
>
> However, CoolioThread{Pool/Spawn} should help you with the work load
> distribution problem if your app is thread-safe, but these both do body
> buffering.
>
> However, the multithreaded options _with_ "rack.input streaming" still
> give pretty good performance and let you get around the workload
> distribution problem of the single-threaded + internal buffering
> options.  You'll reduce disk seek overhead with input streaming.
>
> For reference, nginx+unicorn is single-threaded, but with external
> buffering (nginx does the buffering, not unicorn).

Thank you for your extensive response!

To verify my thoughts on this, if I want to prevent the head-of-queue
blocking behaviour, I need to take a module without
rack.input_streaming. But we need to make sure the body is buffered
before it's given to a worker. On which property do I select the
correct module?

Are Coolio modules still a sensible choice as the Coolio project isn't
actively maintained?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Unicorn hangs on POST request
  2013-04-02 11:55  6%           ` Tom Pesman
@ 2013-04-02 17:24  6%             ` Eric Wong
  2013-04-02 20:25  0%               ` Tom Pesman
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2013-04-02 17:24 UTC (permalink / raw)
  To: unicorn list

Tom Pesman <tom@tnux.net> wrote:
> > The request body doesn't seem to be there, presumably because Heroku
> > isn't sending it.
> >
> > Doe heroku fully buffer the request body before sending it to unicorn?
> > nginx fully buffers, and this is why I can only recommend nginx for slow
> > clients.
> >
> > The proxy -> unicorn transfer speed should _never_ be dependent by the
> > client -> proxy transfer speed.
> >
> > 1) client        ------------------ (slow) --------------> proxy
> > 2) proxy (nginx) --- (as fast as the connection goes) ---> unicorn
> >
> > With nginx, 1) and 2) are decoupled and serialized.  This adds latency,
> > but is the only way for multiprocess servers like unicorn to efficiently
> > handle slow clients.
> 
> I've some new information. Heroku buffers the headers of a HTTP
> request but it doesn't buffer the body of POST requests. Because of
> that I switched to Rainbows! and the responsiveness of the application
> increased dramatically.

Wow, the Heroku proxy is completely unsuitable for unicorn because of
the lack of body buffering.  Heroku really needs to do body buffering to
support unicorn properly (perhaps use nginx...).

It's a shame they charge people to deploy unicorn improperly like this.

> Right now I'm using this configuration:
> 
> worker_processes 4
> timeout 15
> preload_app true
> 
> Rainbows! do
>   use :EventMachine
>   worker_connections 50
>   client_max_body_size 5*1024*1024 # 5 megabytes
>   client_header_buffer_size 2 * 1024 # 2 kilobytes
>   timeout 10
> end
> 
> With high load the performance drops, is EventMachine the right choice
> for this situation (Rails application with slow POST requests and with
> up to 50% POST requests)? Will increasing the worker_connections help?

Probably not, at least it won't improve _consistency_ of performance
without changing your app.

The problem with buffering in Rainbows!+EM is the buffering happens after
the work is distributed to different worker processes (Rainbows!+EM is
still single-threaded).

> I'm planning to make a blog post about this and tell Heroku not to
> advise Unicorn for Rails applications but to use Rainbows and suggest
> a correct/optimised configuration.

The easiest solution is to use nginx with unicorn, I still believe this
is the best configuration for most web apps.

I don't believe there's a universal Rainbows! configuration which is
good for most apps.  With Rainbows!+EM, the buffering happens in a late
stage, after workload is distributed, so you end up with head-of-queue
blocking inside each process.

All of the modules _without_ "rack.input streaming" will fully buffer
responses (similar to how Rainbows!+EM) does it:
http://rainbows.rubyforge.org/Summary.html

However, CoolioThread{Pool/Spawn} should help you with the work load
distribution problem if your app is thread-safe, but these both do body
buffering.

However, the multithreaded options _with_ "rack.input streaming" still
give pretty good performance and let you get around the workload
distribution problem of the single-threaded + internal buffering
options.  You'll reduce disk seek overhead with input streaming.

For reference, nginx+unicorn is single-threaded, but with external
buffering (nginx does the buffering, not unicorn).
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Unicorn hangs on POST request
  @ 2013-04-02 11:55  6%           ` Tom Pesman
  2013-04-02 17:24  6%             ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Tom Pesman @ 2013-04-02 11:55 UTC (permalink / raw)
  To: unicorn list

> The request body doesn't seem to be there, presumably because Heroku
> isn't sending it.
>
> Doe heroku fully buffer the request body before sending it to unicorn?
> nginx fully buffers, and this is why I can only recommend nginx for slow
> clients.
>
> The proxy -> unicorn transfer speed should _never_ be dependent by the
> client -> proxy transfer speed.
>
> 1) client        ------------------ (slow) --------------> proxy
> 2) proxy (nginx) --- (as fast as the connection goes) ---> unicorn
>
> With nginx, 1) and 2) are decoupled and serialized.  This adds latency,
> but is the only way for multiprocess servers like unicorn to efficiently
> handle slow clients.

I've some new information. Heroku buffers the headers of a HTTP
request but it doesn't buffer the body of POST requests. Because of
that I switched to Rainbows! and the responsiveness of the application
increased dramatically.

Right now I'm using this configuration:

worker_processes 4
timeout 15
preload_app true

Rainbows! do
  use :EventMachine
  worker_connections 50
  client_max_body_size 5*1024*1024 # 5 megabytes
  client_header_buffer_size 2 * 1024 # 2 kilobytes
  timeout 10
end

With high load the performance drops, is EventMachine the right choice
for this situation (Rails application with slow POST requests and with
up to 50% POST requests)? Will increasing the worker_connections help?

I'm planning to make a blog post about this and tell Heroku not to
advise Unicorn for Rails applications but to use Rainbows and suggest
a correct/optimised configuration.

Thanks!

Tom
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Unicorn on shared apps platform
  2013-02-26 17:26  6%   ` Eric Wong
  2013-02-26 17:30  0%     ` Hongli Lai
@ 2013-02-26 18:18  0%     ` Amol Dev
  1 sibling, 0 replies; 191+ results
From: Amol Dev @ 2013-02-26 18:18 UTC (permalink / raw)
  To: unicorn list

> unicorn isn't for cases where
> you're cramming many apps on one box and trying to squeak by on memory
> usage.


Thanks. The discussion helped a lot.


----- Original Message -----
> From: Eric Wong <normalperson@yhbt.net>
> To: unicorn list <mongrel-unicorn@rubyforge.org>
> Cc: 
> Sent: Tuesday, February 26, 2013 9:26 AM
> Subject: Re: Unicorn on shared apps platform
> 
> Hongli Lai <hongli@phusion.nl> wrote:
>>  On Tue, Feb 26, 2013 at 4:08 PM, Amol Dev <devamol@yahoo.com> wrote:
>>  > We are hosting multiple Rails applications on same server and using
>>  > Passenger + Apache. I can see Unicorn be useful for seamless
>>  > deploys, few questions running it on Rails platform with 50+ rails
>>  > apps:
> 
>>  Rainbows! is designed for a reverse proxy setup
> 
> That's not true, Rainbows! was designed to serve clients directly.
> On the other hand, I do not know if anybody uses Rainbows! that way
> (or at all in production).
> 
> Anyways everything else is accurate.  unicorn isn't for cases where
> you're cramming many apps on one box and trying to squeak by on memory
> usage.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
> 
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Unicorn on shared apps platform
  2013-02-26 17:30  0%     ` Hongli Lai
@ 2013-02-26 17:46  4%       ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-02-26 17:46 UTC (permalink / raw)
  To: unicorn list

Hongli Lai <hongli@phusion.nl> wrote:
> On Tue, Feb 26, 2013 at 6:26 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > That's not true, Rainbows! was designed to serve clients directly.
> > On the other hand, I do not know if anybody uses Rainbows! that way
> > (or at all in production).
> 
> I didn't know that. Is it also the recommended setup though? Are you
> fairly sure there are no exploits in the request parser and that kind
> of stuff, that web servers usually shield for in reverse proxy setups?

Yes, it is recommended to expose Rainbows! directly to clients.

I've tested the parser extensively over many years.
As far as I can tell, there are no exploits...
but also no users, nor warranty :>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: Unicorn on shared apps platform
  2013-02-26 17:26  6%   ` Eric Wong
@ 2013-02-26 17:30  0%     ` Hongli Lai
  2013-02-26 17:46  4%       ` Eric Wong
  2013-02-26 18:18  0%     ` Amol Dev
  1 sibling, 1 reply; 191+ results
From: Hongli Lai @ 2013-02-26 17:30 UTC (permalink / raw)
  To: unicorn list

On Tue, Feb 26, 2013 at 6:26 PM, Eric Wong <normalperson@yhbt.net> wrote:
> That's not true, Rainbows! was designed to serve clients directly.
> On the other hand, I do not know if anybody uses Rainbows! that way
> (or at all in production).

I didn't know that. Is it also the recommended setup though? Are you
fairly sure there are no exploits in the request parser and that kind
of stuff, that web servers usually shield for in reverse proxy setups?

-- 
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Unicorn on shared apps platform
  2013-02-26 15:36  6% ` Hongli Lai
  2013-02-26 16:08  6%   ` Amol Dev
@ 2013-02-26 17:26  6%   ` Eric Wong
  2013-02-26 17:30  0%     ` Hongli Lai
  2013-02-26 18:18  0%     ` Amol Dev
  1 sibling, 2 replies; 191+ results
From: Eric Wong @ 2013-02-26 17:26 UTC (permalink / raw)
  To: unicorn list

Hongli Lai <hongli@phusion.nl> wrote:
> On Tue, Feb 26, 2013 at 4:08 PM, Amol Dev <devamol@yahoo.com> wrote:
> > We are hosting multiple Rails applications on same server and using
> > Passenger + Apache. I can see Unicorn be useful for seamless
> > deploys, few questions running it on Rails platform with 50+ rails
> > apps:

> Rainbows! is designed for a reverse proxy setup

That's not true, Rainbows! was designed to serve clients directly.
On the other hand, I do not know if anybody uses Rainbows! that way
(or at all in production).

Anyways everything else is accurate.  unicorn isn't for cases where
you're cramming many apps on one box and trying to squeak by on memory
usage.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Unicorn on shared apps platform
  2013-02-26 15:36  6% ` Hongli Lai
@ 2013-02-26 16:08  6%   ` Amol Dev
  2013-02-26 17:26  6%   ` Eric Wong
  1 sibling, 0 replies; 191+ results
From: Amol Dev @ 2013-02-26 16:08 UTC (permalink / raw)
  To: unicorn list

Thanks. Did went through Phusion Passenger Enterprise 4 features, you guys did neat work Hongli. However the cost of it is driving us to check on alternate solutions.

Can one signal unicorn to increase workers or just spin new set of workers. Any one tried alicorn? https://github.com/bensomers/alicorn.

Provisioning double memory on deploy can be solved differently by spinning new server that has upgrade and moving load balancer to divert trafic, the netflix way. 


----- Original Message -----
From: Hongli Lai <hongli@phusion.nl>
To: unicorn list <mongrel-unicorn@rubyforge.org>
Cc: 
Sent: Tuesday, February 26, 2013 7:36 AM
Subject: Re: Unicorn on shared apps platform

On Tue, Feb 26, 2013 at 4:08 PM, Amol Dev <devamol@yahoo.com> wrote:
> We are hosting multiple Rails applications on same server and using Passenger + Apache. I can see Unicorn be useful for seamless deploys, few questions running it on Rails platform with 50+ rails apps:
>
> a) How does unicorn scale if app is known to be slow with long running quieres to database? Is there a way to a have master worker spin up new workers aka dynamic workers?

Unicorn is designed for short-running requests. Long-running requests
are best taken care of with app servers designed for that purpose, for
example Rainbows! or Phusion Passenger Enterprise 4. Both app servers
can be used in combination with Apache or Nginx. Rainbows! is designed
for a reverse proxy setup while Phusion Passenger Enterprise 4 can
also integrate with the web server.

As far as I know only Phusion Passenger supports dynamic workers.
Unicorn and Rainbows! are both designed for a static number of
workers. Unicorn and Rainbows! allow modifying the number of workers
during runtime through signals, but not automatically according to
traffic as is done by Phusion Passenger.


> b) Can Unicorn be configured to start with zero worker and master spawning first worker as first request comes in? Very useful to stack multiple apps that are not used all the time.

No, because it does not support spawning processes based on traffic.
As far as I know only Phusion Passenger does this.

> c) Unicorn for zero lost connection (no downtime during deploy and changes) needs new workers spinned up before old workers are sent USR2 signal, does this not mean that one has to provision double memory for servers?

Yes it does. Although it's possible to change this behavior through a
script that sends signals in a certain order.

-- 
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Unicorn on shared apps platform
  @ 2013-02-26 15:36  6% ` Hongli Lai
  2013-02-26 16:08  6%   ` Amol Dev
  2013-02-26 17:26  6%   ` Eric Wong
  0 siblings, 2 replies; 191+ results
From: Hongli Lai @ 2013-02-26 15:36 UTC (permalink / raw)
  To: unicorn list

On Tue, Feb 26, 2013 at 4:08 PM, Amol Dev <devamol@yahoo.com> wrote:
> We are hosting multiple Rails applications on same server and using Passenger + Apache. I can see Unicorn be useful for seamless deploys, few questions running it on Rails platform with 50+ rails apps:
>
> a) How does unicorn scale if app is known to be slow with long running quieres to database? Is there a way to a have master worker spin up new workers aka dynamic workers?

Unicorn is designed for short-running requests. Long-running requests
are best taken care of with app servers designed for that purpose, for
example Rainbows! or Phusion Passenger Enterprise 4. Both app servers
can be used in combination with Apache or Nginx. Rainbows! is designed
for a reverse proxy setup while Phusion Passenger Enterprise 4 can
also integrate with the web server.

As far as I know only Phusion Passenger supports dynamic workers.
Unicorn and Rainbows! are both designed for a static number of
workers. Unicorn and Rainbows! allow modifying the number of workers
during runtime through signals, but not automatically according to
traffic as is done by Phusion Passenger.


> b) Can Unicorn be configured to start with zero worker and master spawning first worker as first request comes in? Very useful to stack multiple apps that are not used all the time.

No, because it does not support spawning processes based on traffic.
As far as I know only Phusion Passenger does this.

> c) Unicorn for zero lost connection (no downtime during deploy and changes) needs new workers spinned up before old workers are sent USR2 signal, does this not mean that one has to provision double memory for servers?

Yes it does. Although it's possible to change this behavior through a
script that sends signals in a certain order.

-- 
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 4.6.2 - HTTP parser fix for Rainbows!
@ 2013-02-26  3:06 14% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-02-26  3:06 UTC (permalink / raw)
  To: mongrel-unicorn

Changes:

This release fixes a bug in Unicorn::HttpParser#filter_body
which affected some configurations of Rainbows!  There is
also a minor size reduction in the DSO.

(this parser fix only affects Ruby 2.0.0 with Rainbows!)

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 14%]

* Re: [ANN] unicorn 4.6.0pre1 - hijacking support!
  @ 2013-02-04 13:40  6% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-02-04 13:40 UTC (permalink / raw)
  To: mongrel-unicorn

Anybody running 4.6.0pre1, yet?  I'm tempted to release 4.6.0 soon.

I just pushed out a couple of *BSD-related test fixes ported
over from rainbows.git (@ git://bogomips.org/rainbows.git )

  commit 9cd8554749a9f120b010c93933d09d2dd27b1280
  Author: Eric Wong <normalperson@yhbt.net>
  Date:   Mon Feb 4 12:39:09 2013 +0000

      tests: "wc -l" portability for *BSDs
      
      On FreeBSD 9.0, "wc -l" emits leading whitespace, so
      filter it through tr -d '[:space:]' to eliminate it.

  commit 2a2163594ea2b515e98fbe9f909bcf90e4c35fe8
  Author: Eric Wong <normalperson@yhbt.net>
  Date:   Mon Feb 4 12:29:00 2013 +0000

      tests: "wc -c" portability for *BSDs
      
      On FreeBSD 9.0, "wc -c" emits leading whitespace, so
      filter it through tr -d '[:space:]' to eliminate it.
      
      This is commit 8a6117a22a7d01eeb5adc63d3152acf435cd3176
      in rainbows.git

  commit 85223902e8229bd460ce0b4ad126f42b1db42a46
  Author: Eric Wong <normalperson@yhbt.net>
  Date:   Mon Feb 4 10:36:18 2013 +0000

      tests: replace non-portable "date +%s" with ruby equivalent
      
      "date +%s" is not in POSIX (it is in GNU, and at least FreeBSD
      9.0, possibly earlier).  The Ruby equivalent should be
      sufficiently portable between different Ruby versions.
      
      This change was automated via:
          perl -i -p -e 's/date \+%s/unix_time/' t/*.sh
      
      This is commit 0ba6fc3c30b9cf530faf7fcf5ce7be519ec13fe7
      in rainbows.git

  commit a09a622b4988b5eee819487c96a4563e71f753f7
  Author: Eric Wong <normalperson@yhbt.net>
  Date:   Mon Feb 4 10:30:25 2013 +0000

      tests: remove utee
      
      POSIX already stipulates tee(1) must be unbuffered.  I think my
      decision to use utee was due to my being misled by a bug in
      older curl where -N did not work as advertised (but --no-buffer
      did).
      
      N.B. we don't use tee in unicorn tests, this just matches
      commit cbff7b0892148b037581541184364e0e91d2a138 in rainbows

  commit 64765b95df06256d39daefdeebde97c874770131
  Author: Eric Wong <normalperson@yhbt.net>
  Date:   Tue Jan 29 21:19:22 2013 +0000

      manpage: update middleware-related documentation
      
      -N/--no-default-middleware needs a corresponding manpage entry.
      
      Additionally, the Rack::Chunked/ContentLength middleware comment
      is out-of-date as of unicorn v4.1.0

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: [PATCH] Add -N or --no-default-middleware option.
  @ 2013-01-29  4:03  4%   ` Lin Jen-Shin (godfat)
  0 siblings, 0 replies; 191+ results
From: Lin Jen-Shin (godfat) @ 2013-01-29  4:03 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list

On Tue, Jan 29, 2013 at 11:52 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Lin Jen-Shin <godfat@godfat.org> wrote:
>> +  opts.on("-N", "--no-default-middleware",
>> +          "no default middleware even if RACK_ENV is development") do |e|
>
> RACK_ENV=deployment also loads middleware, so I think it's more accurate
> with:
>
>         do not load middleware implied by RACK_ENV
>
> This also puts us back within the 80-column limit imposed by "test_help"
> in test/exec/test_exec.rb

Oh right, I only remembered to keep the source within 80-column limit,
but forgot that the output might be longer than source. Thanks for the
better wordings. I updated the patch for Zbatery, but it's too late
for Rainbows though.

> Will sign-off and push that change squashed unless there are objections.

Surely, thanks!

> I also just bumped master to rack 1.5.1, which fixes the Rack::Lint
> test regression in 1.5.0.
>
> Will pull in the "hijack" branch into master soon (and release 4.6.0)
> since all my questions about it have been answered in rack-devel.

I haven't had a chance to read hijack more, looking forward to it :)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* [PATCH] Add -N or --no-default-middleware option.
@ 2013-01-29  3:21 10% Lin Jen-Shin
    0 siblings, 1 reply; 191+ results
From: Lin Jen-Shin @ 2013-01-29  3:21 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Lin Jen-Shin

This would prevent Unicorn from adding default middleware,
as if RACK_ENV were always none. (not development nor deployment)

This should also be applied to `rainbows' and `zbatery' as well.

One of the reasons to add this is to avoid conflicting
RAILS_ENV and RACK_ENV. It would be helpful in the case
where a Rails application and Rack application are composed
together, while we want Rails app runs under development
and Rack app runs under none (if we don't want those default
middleware), and we don't really want to make RAILS_ENV
set to development and RACK_ENV to none because it might be
confusing. Note that Rails would also look into RACK_ENV.

Another reason for this is that only `rackup' would be
inserting those default middleware. Both `thin' and `puma'
would not do this, nor does Rack::Handler.get.run which is
used in Sinatra.

So using this option would make it work differently from
`rackup' but somehow more similar to `thin' or `puma'.

Discussion thread on the mailing list:
http://rubyforge.org/pipermail/mongrel-unicorn/2013-January/001675.html
---
 bin/unicorn    | 5 +++++
 lib/unicorn.rb | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/bin/unicorn b/bin/unicorn
index 9962b58..415d164 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -58,6 +58,11 @@ op = OptionParser.new("", 24, '  ') do |opts|
     ENV["RACK_ENV"] = e
   end
 
+  opts.on("-N", "--no-default-middleware",
+          "no default middleware even if RACK_ENV is development") do |e|
+    rackup_opts[:no_default_middleware] = true
+  end
+
   opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
     rackup_opts[:daemonize] = !!d
   end
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index d96ff91..f0ceffe 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -49,6 +49,8 @@ module Unicorn
 
       pp({ :inner_app => inner_app }) if $DEBUG
 
+      return inner_app if op[:no_default_middleware]
+
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
       # and Zbatery both do.  Users accustomed to the Rack::Server default
-- 
1.8.1.1

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 10%]

* Re: No middleware without touching RACK_ENV
  2013-01-29  2:31  4%       ` Lin Jen-Shin (godfat)
@ 2013-01-29  2:52  0%         ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2013-01-29  2:52 UTC (permalink / raw)
  To: unicorn list

"Lin Jen-Shin (godfat)" <godfat@godfat.org> wrote:
> And should I send patches for rainbows and zbatery as well?

Yes please, thanks!
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: No middleware without touching RACK_ENV
  @ 2013-01-29  2:31  4%       ` Lin Jen-Shin (godfat)
  2013-01-29  2:52  0%         ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Lin Jen-Shin (godfat) @ 2013-01-29  2:31 UTC (permalink / raw)
  To: unicorn list

On Tue, Jan 29, 2013 at 7:21 AM, Eric Wong <normalperson@yhbt.net> wrote:
> "Lin Jen-Shin (godfat)" <godfat@godfat.org> wrote:
>> Speaking to this, I might want to complain about the other severs :P
>> As far as I know, only if you use `rackup' would you get those middleware.
>>
>> That is, `unicorn' is compatible with `rackup'. That's a good thing.
>> But it seems to me that both `thin' and `puma' (and maybe including
>> others, I didn't check) would not do this if you use their command line
>> tools. They even have different options to enable Rack::CommonLogger.
>>
>> Actually this is also the reason why sometimes `thin' and `unicorn'
>> worked differently when people switching from Thin to Unicorn.
>> And you know a lot of people are using Thin in the first place,
>> thus sometimes blaming Unicorn works differently.
>>
>> I am not sure how many people are using `rackup', but according to
>> my observation, not too much. If people are using Sinatra directly,
>> eventually they would launch the server via Rack::Handler.get.run,
>> which does not try to wrap additional middleware as what `rackup'
>> would do if I read correctly.
>>
>> So I am not sure if a lot of people would be expecting this actually.
>> I used to use `rackup' a lot to get consistent behaviour, but it looks to
>> me not many people even know the existence of rackup :(
>
> *sigh*  :<
>
> I kinda wish there was no default middleware, either.  But it's
> too late to change unicorn defaults.
>
> I think your -N/--no-default-middleware patch can be accepted,
> then.  after_app_load is more prone to user error/confusion, I think
> (I believe RAILS_ENV/RACK_ENV should be frozen for the lifetime of
>  the process)

Awesome. Thanks! Hope this would also be helpful for others.

> Can you try send-email again with the --no-default-middleware patch? :)
>
> The git-send-email(1) manpage has several good examples for your email
> provider.  There's also git-imap-send which you can use to stick email
> in your Drafts folder, but I've never needed/used it, though.

Just got managed to send a test mail to myself. I realized that my git is
using /usr/bin/perl instead of /usr/local/bin/perl, so that instead of running
`cpan Net::SMTP::SSL` I have to run `sudo -H /usr/bin/cpan Net::SMTP::SSL`
following instructions from this site:
http://kbase.wincent.com/old/knowledge-base/Installing_Net::SMTP::SSL_for_sending_patches_with_Git_over_secure_SMTP.html

I'll send the patch shortly after updating the commit message.
Feel free to edit it as usual :)
And should I send patches for rainbows and zbatery as well?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: No middleware without touching RACK_ENV
  2013-01-25 10:52  8% No middleware without touching RACK_ENV Lin Jen-Shin (godfat)
@ 2013-01-25 18:39  0% ` Eric Wong
    0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2013-01-25 18:39 UTC (permalink / raw)
  To: unicorn list

"Lin Jen-Shin (godfat)" <godfat@godfat.org> wrote:
> Hi,
> 
> This might be a bit silly, but finally I decided to bring this up.
> 
> We're running a Rails app along with another Rack app in
> the same config.ru, and what I want to do is telling Unicorn
> that we don't want any middleware which Unicorn might
> insert with RACK_ENV=development and RACK_ENV=
> deployment, because we're adding Rack::CommonLogger
> and other middleware by ourselves, while we're using
> RACK_ENV=development for Rails when developing.

Doesn't Rails favor RAILS_ENV over RACK_ENV?  unicorn ignores RAILS_ENV
(unicorn_rails respects RAILS_ENV, but unicorn_rails isn't recommended
for modern (Rack-enabled) Rails)

> That is, we want to use RACK_ENV=development for Rails
> because that's how it used to be, but we also don't want
> those additional middleware got inserted automatically.
> This has no problem with RACK_ENV=production.

Is there a benefit which RACK_ENV=development gives you for Rails
that RAILS_ENV=development does not give you?

Fwiw, the times I worked on Rails, I used customized dev environments
anyways, so I would use RAILS_ENV=dev_local or RAILS_ENV=dev_$NETWORK
for switching between a localhost-only vs networked setup.

> I know this is somehow a spec from Rack, but I guess I
> don't like this behaviour. One workaround would be
> providing a `after_app_load' hook, and we add this to
> the bottom of config.ru:
> 
>   ENV['RACK_ENV_ORIGINAL'] = ENV['RACK_ENV']
>   ENV['RACK_ENV'] = 'none'
> 
> and add this to the unicorn configuration file:
> 
>   after_app_load do |_|
>     ENV['RACK_ENV'] = ENV['RACK_ENV_ORIGINAL']
>   end
> 
> This is probably a stupid hack, but I wonder if after_app_load
> hook would be useful for other cases?

I don't see how this hook has benefits over putting this in config.ru
(or somewhere else in your app, since your app is already loaded...)

I'd imagine users wanting the same app-wide settings going between
different application servers, too.

> Or if we could have an option to turn off this Rack behaviour
> simulation, like:
> 
>     default_middleware false
> 
> That might be more straightforward for what we want. Oh but
> it seems it's not that easy to add this. What about an option
> for unicorn?
> 
>     unicorn -E development -N
> 
> or
> 
>     unicorn -E development --no-default-middleware

I'm against adding more command-line or config options, especially
around loading middleware.  RACK_ENV is already a source of confusion,
and Rails also has/(or had?) it's own middleware loading scheme
independent of config.ru.

> This would need to be applied to `rainbows' and `zbatery', too,
> though. Patches below for consideration:
> (Sorry if Gmail messed the format up, but from last time
> I tried, it doesn't accept attachments :( What's the best way?
> Links to raw patches?)

git format-patch + git send-email is the recommended way if your
mail client cannot be taught to leave formatting alone.  Also this
is easy for sending multiple patches in the same thread.

git format-patch --stdout with a good mail client which will not wrap
messages is also great (especially for single patches).

Attachments (generated using git format-patch) for text/x-diff should be
accepted by Mailman.  Attached patches are a last resort, I prefer
inline patches since I can quote/reply to them inline.

(same patch submission guidelines as git or Linux kernel)

> https://github.com/godfat/unicorn/pull/2
> 
> commit 95de5abf38a81a76af15476d4705713d2d644664
> Author: Lin Jen-Shin <godfat@godfat.org>
> Date:   Fri Jan 25 18:18:21 2013 +0800
> 
>     Add `after_app_load' hook.
> 
>     The hook would be called right after application is loaded.

Commit messages should explain why a change is made/needed, not just
what it does.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* No middleware without touching RACK_ENV
@ 2013-01-25 10:52  8% Lin Jen-Shin (godfat)
  2013-01-25 18:39  0% ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Lin Jen-Shin (godfat) @ 2013-01-25 10:52 UTC (permalink / raw)
  To: unicorn list

Hi,

This might be a bit silly, but finally I decided to bring this up.

We're running a Rails app along with another Rack app in
the same config.ru, and what I want to do is telling Unicorn
that we don't want any middleware which Unicorn might
insert with RACK_ENV=development and RACK_ENV=
deployment, because we're adding Rack::CommonLogger
and other middleware by ourselves, while we're using
RACK_ENV=development for Rails when developing.

That is, we want to use RACK_ENV=development for Rails
because that's how it used to be, but we also don't want
those additional middleware got inserted automatically.
This has no problem with RACK_ENV=production.

I know this is somehow a spec from Rack, but I guess I
don't like this behaviour. One workaround would be
providing a `after_app_load' hook, and we add this to
the bottom of config.ru:

  ENV['RACK_ENV_ORIGINAL'] = ENV['RACK_ENV']
  ENV['RACK_ENV'] = 'none'

and add this to the unicorn configuration file:

  after_app_load do |_|
    ENV['RACK_ENV'] = ENV['RACK_ENV_ORIGINAL']
  end

This is probably a stupid hack, but I wonder if after_app_load
hook would be useful for other cases?

Or if we could have an option to turn off this Rack behaviour
simulation, like:

    default_middleware false

That might be more straightforward for what we want. Oh but
it seems it's not that easy to add this. What about an option
for unicorn?

    unicorn -E development -N

or

    unicorn -E development --no-default-middleware

This would need to be applied to `rainbows' and `zbatery', too,
though. Patches below for consideration:
(Sorry if Gmail messed the format up, but from last time
I tried, it doesn't accept attachments :( What's the best way?
Links to raw patches?)

https://github.com/godfat/unicorn/pull/2

commit 95de5abf38a81a76af15476d4705713d2d644664
Author: Lin Jen-Shin <godfat@godfat.org>
Date:   Fri Jan 25 18:18:21 2013 +0800

    Add `after_app_load' hook.

    The hook would be called right after application is loaded.

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 7651093..332bdbc 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -43,6 +43,9 @@ class Unicorn::Configurator
     :before_exec => lambda { |server|
         server.logger.info("forked child re-executing...")
       },
+    :after_app_load => lambda { |server|
+        server.logger.info("application loaded")
+      },
     :pid => nil,
     :preload_app => false,
     :check_client_connection => false,
@@ -171,6 +174,13 @@ class Unicorn::Configurator
     set_hook(:before_exec, block_given? ? block : args[0], 1)
   end

+  # sets the after_app_load hook to a given Proc object.  This
+  # Proc object will be called by the master process right
+  # after application loaded.
+  def after_app_load(*args, &block)
+    set_hook(:after_app_load, block_given? ? block : args[0], 1)
+  end
+
   # sets the timeout of worker processes to +seconds+.  Workers
   # handling the request/app.call/response cycle taking longer than
   # this time period will be forcibly killed (via SIGKILL).  This
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index aa98aeb..a3b30ee 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -14,6 +14,7 @@ class Unicorn::HttpServer
   # :stopdoc:
   attr_accessor :app, :request, :timeout, :worker_processes,
                 :before_fork, :after_fork, :before_exec,
+                :after_app_load,
                 :listener_opts, :preload_app,
                 :reexec_pid, :orig_app, :init_listeners,
                 :master_pid, :config, :ready_pipe, :user
@@ -716,6 +717,7 @@ class Unicorn::HttpServer
         Gem.refresh
       end
       self.app = app.call
+      config.after_app_load.call(self)
     end
   end






And --no-default-middleware
https://github.com/godfat/unicorn/pull/3

commit e3575db2a36e3ca2acda18bfee97bf95609a9860
Author: Lin Jen-Shin <godfat@godfat.org>
Date:   Fri Jan 25 18:38:52 2013 +0800

    Add -N or --no-default-middleware option.

    This would prevent Unicorn from adding default middleware,
    as if RACK_ENV is always none. (not development nor deployment)

    This should also apply to `rainbows' and `zbatery'.

diff --git a/bin/unicorn b/bin/unicorn
index 9962b58..415d164 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -58,6 +58,11 @@ op = OptionParser.new("", 24, '  ') do |opts|
     ENV["RACK_ENV"] = e
   end

+  opts.on("-N", "--no-default-middleware",
+          "no default middleware even if RACK_ENV is development") do |e|
+    rackup_opts[:no_default_middleware] = true
+  end
+
   opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
     rackup_opts[:daemonize] = !!d
   end
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index d96ff91..f0ceffe 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -49,6 +49,8 @@ module Unicorn

       pp({ :inner_app => inner_app }) if $DEBUG

+      return inner_app if op[:no_default_middleware]
+
       # return value, matches rackup defaults based on env
       # Unicorn does not support persistent connections, but Rainbows!
       # and Zbatery both do.  Users accustomed to the Rack::Server default
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 8%]

* [ANN] unicorn 4.5.0 (final) - check_client_connection option
@ 2012-12-08  0:17  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2012-12-08  0:17 UTC (permalink / raw)
  To: mongrel-unicorn

Changes:

The new check_client_connection option allows unicorn to detect
most disconnected local clients before potentially expensive
application processing begins.

This feature is useful for applications experiencing spikes of
traffic leading to undesirable queue times, as clients will
disconnect (and perhaps even retry, compounding the problem)
before unicorn can even start processing the request.

To enable this feature, add the following line to a unicorn
config file:

      check_client_connection true

This feature only works when nginx (or any other HTTP/1.0+
client) is on the same machine as unicorn.

A huge thanks to Tom Burns for implementing and testing this
change in production with real traffic (including mitigating
an unexpected DoS attack).

ref: http://mid.gmane.org/CAK4qKG3rkfVYLyeqEqQyuNEh_nZ8yw0X_cwTxJfJ+TOU+y8F+w@mail.gmail.com

This release fixes broken Rainbows! compatibility in 4.5.0pre1.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: Combating nginx 499 HTTP responses during flash traffic scenario
  @ 2012-12-04  3:00 13%                   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2012-12-04  3:00 UTC (permalink / raw)
  To: unicorn list

Eric Wong <normalperson@yhbt.net> wrote:
> I fixed up some minor line-wrapping, signed-off, and added your
> quote above to the commit message.  Pushed as
> commit 5c700fc2cf398848ddcf71a2aa3f0f2a6563e87b
> to git://bogomips.org/unicorn.git

One more fix/cleanup to maintain compatibility with Rainbows!

>From 69e6a793d34ff71da7c8ca59962d627e2fb508d8 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Tue, 4 Dec 2012 02:35:26 +0000
Subject: [PATCH] fix const error responses for Rainbows!

Rainbows! relies on the ERROR_XXX_RESPONSE constants of unicorn
4.x.  Changing the constants in unicorn 4.x will break existing
versions of Rainbows!, so remove the dependency on the constants
and generate the error response dynamically.

Unlike Mongrel, unicorn is unlikely to see malicious traffic and
thus unlikely to benefit from making error messages constant.

For unicorn 5.x, we will drop these constants entirely.

(Rainbows! most likely cannot support check_client_connection
 consistently across all concurrency models since some of them
 pessimistically buffer all writes in userspace.  However, the
 extra concurrency of Rainbows! makes it less likely to be
 overloaded than unicorn, so this feature is likely less useful
 for Rainbows!)
---
 lib/unicorn/const.rb         | 10 ++++++----
 lib/unicorn/http_response.rb |  4 ++++
 lib/unicorn/http_server.rb   | 15 +++++++--------
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/lib/unicorn/const.rb b/lib/unicorn/const.rb
index 60a63b1..02e29c7 100644
--- a/lib/unicorn/const.rb
+++ b/lib/unicorn/const.rb
@@ -29,10 +29,12 @@ module Unicorn::Const
 
   # :stopdoc:
   # common errors we'll send back
-  ERROR_400_RESPONSE = "400 Bad Request\r\n\r\n"
-  ERROR_414_RESPONSE = "414 Request-URI Too Long\r\n\r\n"
-  ERROR_413_RESPONSE = "413 Request Entity Too Large\r\n\r\n"
-  ERROR_500_RESPONSE = "500 Internal Server Error\r\n\r\n"
+  # (N.B. these are not used by unicorn, but we won't drop them until
+  #  unicorn 5.x to avoid breaking Rainbows!).
+  ERROR_400_RESPONSE = "HTTP/1.1 400 Bad Request\r\n\r\n"
+  ERROR_414_RESPONSE = "HTTP/1.1 414 Request-URI Too Long\r\n\r\n"
+  ERROR_413_RESPONSE = "HTTP/1.1 413 Request Entity Too Large\r\n\r\n"
+  ERROR_500_RESPONSE = "HTTP/1.1 500 Internal Server Error\r\n\r\n"
 
   EXPECT_100_RESPONSE = "HTTP/1.1 100 Continue\r\n\r\n"
   EXPECT_100_RESPONSE_SUFFIXED = "100 Continue\r\n\r\nHTTP/1.1 "
diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb
index 61563cd..579d957 100644
--- a/lib/unicorn/http_response.rb
+++ b/lib/unicorn/http_response.rb
@@ -17,6 +17,10 @@ module Unicorn::HttpResponse
   }
   CRLF = "\r\n"
 
+  def err_response(code, response_start_sent)
+    "#{response_start_sent ? '' : 'HTTP/1.1 '}#{CODES[code]}\r\n\r\n"
+  end
+
   # writes the rack_response to socket as an HTTP response
   def http_response_write(socket, status, headers, body,
                           response_start_sent=false)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index ef1ea58..aa98aeb 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -519,22 +519,21 @@ class Unicorn::HttpServer
   # if the socket is already closed or broken.  We'll always ensure
   # the socket is closed at the end of this function
   def handle_error(client, e)
-    msg = case e
+    code = case e
     when EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::EINVAL,Errno::EBADF,
          Errno::ENOTCONN
-      Unicorn::Const::ERROR_500_RESPONSE
+      500
     when Unicorn::RequestURITooLongError
-      Unicorn::Const::ERROR_414_RESPONSE
+      414
     when Unicorn::RequestEntityTooLargeError
-      Unicorn::Const::ERROR_413_RESPONSE
+      413
     when Unicorn::HttpParserError # try to tell the client they're bad
-      Unicorn::Const::ERROR_400_RESPONSE
+      400
     else
       Unicorn.log_error(@logger, "app error", e)
-      Unicorn::Const::ERROR_500_RESPONSE
+      500
     end
-    msg = "HTTP/1.1 #{msg}" unless @request.response_start_sent
-    client.kgio_trywrite(msg)
+    client.kgio_trywrite(err_response(code, @request.response_start_sent))
     client.close
     rescue
   end
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 13%]

* Re: Unused Unicorn processes
  2012-08-21  9:11  3%   ` Eric Wong
@ 2012-08-22 18:16  4%     ` Konstantin Gredeskoul
  0 siblings, 0 replies; 191+ results
From: Konstantin Gredeskoul @ 2012-08-22 18:16 UTC (permalink / raw)
  To: unicorn list

Thanks Eric!  Appreciate well thought out answers.

We are actually using Rainbows also, in an project where long
server-side HTTP calls are part of the application design and are
necessary, and it's working out really well.

Our main web app mostly spends it's time in CPU cycles serving web
pages (about 80% of the request time), and 20% in database operations.

Thanks again
K

On Tue, Aug 21, 2012 at 2:11 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Konstantin Gredeskoul <kig@wanelo.com> wrote:
>> Greetings!
>>
>> I have a question on optimal # of unicorn worker processes.
>>
>> We are running Unicorn 4.3.1 + Rails 3.2.6 (without threading), on
>> ruby 1.9.3-p194, hosted on SmartOS/Joyent.
>>
>> At the moment, unicorns are configured to start 30 worker processes. I
>> know this is a lot, and I am going to reduce this number. But in
>> trying to figure out what is a more appropriate number of workers to
>> run, I noticed something interesting that I couldn't explain.
>>
>> If I look at the process table on each machine (see top output below),
>> I notice that some unicorn processes are heavily used (and have
>> accumulated longer CPU times, as well as have grown their RAM usage
>> since boot time), but other processes (at the bottom of the top
>> output) appear to potentially not having been used at all.  There are
>> several processes with RSS size of 143Mb, which I believe is unicorn
>> size before it processes any requests.
>>
>> What I am gathering from this, is that only 16 unicorn processes are
>> actually processing requests, while the rest are just sitting there
>> idle.
>>
>> Is this expected behavior?
>
> It's not _unexpected_ if your load is low.  The OS scheduler does the
> load balancing itself as unicorn uses non-blocking accept().  The
> scheduler may choose busier processes because they're likely to be
> hotter in cache/memory.
>
> So you're probably getting better performance than if you had perfectly
> balanced traffic.
>
> Another possible explanation is something is killing some workers
> (timeout, or crash buts, check stderr logs to confirm).  But even in a
> situation where workers never die, it's perfectly normal to see workers
> that never/rarely serve traffic.
>
>> This Joyent SmartMachine can burst up to 8 cores. Given that our web
>> requests spend only 80% of their time in ruby, I figured we could run
>> 10 unicorn processes for maximum efficiency.  However seeing that 16
>> are actually used I am curious whether 16 is actually a better number.
>
> It depends what your app is waiting on.
>
> The only rule is your machine should not be swap thrashing under
> peak load.
>
> Extra workers won't be detrimental as long as you:
> 1) have enough memory
> 2) have enough backend resources (e.g DB connections)
>
> I know of deployments that run 30 processes/core because the app spends
> much time waiting on slow HTTP/DB requests[1]
>
> But if your app is completely bound by resources local to a machine (no
> external DB/HTTP/memcached/redis/etc requests), then a 1:1 worker:core
> (or even 1:1 worker:disk) relationship will work, too.
>
>
>
> [1] Arguably a multi-threaded or non-blocking server such as Rainbows!
> would be most efficient of machine resources if you're waiting on HTTP
> calls, but the developers decided human time was more important and
> did not want to worry about thread-safety.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying



-- 

Konstantin Gredeskoul
CTO :: Wanelo Inc
cell: (415) 265 1054
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: Unused Unicorn processes
  @ 2012-08-21  9:11  3%   ` Eric Wong
  2012-08-22 18:16  4%     ` Konstantin Gredeskoul
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2012-08-21  9:11 UTC (permalink / raw)
  To: unicorn list

Konstantin Gredeskoul <kig@wanelo.com> wrote:
> Greetings!
> 
> I have a question on optimal # of unicorn worker processes.
> 
> We are running Unicorn 4.3.1 + Rails 3.2.6 (without threading), on
> ruby 1.9.3-p194, hosted on SmartOS/Joyent.
> 
> At the moment, unicorns are configured to start 30 worker processes. I
> know this is a lot, and I am going to reduce this number. But in
> trying to figure out what is a more appropriate number of workers to
> run, I noticed something interesting that I couldn't explain.
> 
> If I look at the process table on each machine (see top output below),
> I notice that some unicorn processes are heavily used (and have
> accumulated longer CPU times, as well as have grown their RAM usage
> since boot time), but other processes (at the bottom of the top
> output) appear to potentially not having been used at all.  There are
> several processes with RSS size of 143Mb, which I believe is unicorn
> size before it processes any requests.
> 
> What I am gathering from this, is that only 16 unicorn processes are
> actually processing requests, while the rest are just sitting there
> idle.
> 
> Is this expected behavior?

It's not _unexpected_ if your load is low.  The OS scheduler does the
load balancing itself as unicorn uses non-blocking accept().  The
scheduler may choose busier processes because they're likely to be
hotter in cache/memory.

So you're probably getting better performance than if you had perfectly
balanced traffic.

Another possible explanation is something is killing some workers
(timeout, or crash buts, check stderr logs to confirm).  But even in a
situation where workers never die, it's perfectly normal to see workers
that never/rarely serve traffic.

> This Joyent SmartMachine can burst up to 8 cores. Given that our web
> requests spend only 80% of their time in ruby, I figured we could run
> 10 unicorn processes for maximum efficiency.  However seeing that 16
> are actually used I am curious whether 16 is actually a better number.

It depends what your app is waiting on.

The only rule is your machine should not be swap thrashing under
peak load.

Extra workers won't be detrimental as long as you:
1) have enough memory
2) have enough backend resources (e.g DB connections)

I know of deployments that run 30 processes/core because the app spends
much time waiting on slow HTTP/DB requests[1]

But if your app is completely bound by resources local to a machine (no
external DB/HTTP/memcached/redis/etc requests), then a 1:1 worker:core
(or even 1:1 worker:disk) relationship will work, too.



[1] Arguably a multi-threaded or non-blocking server such as Rainbows!
would be most efficient of machine resources if you're waiting on HTTP
calls, but the developers decided human time was more important and
did not want to worry about thread-safety.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 3%]

* Re: [PATCH] `kill -SIGTRAP <worker pid>`
  @ 2012-06-25  3:59  6%       ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2012-06-25  3:59 UTC (permalink / raw)
  To: unicorn list; +Cc: Cedric Maion

Cedric Maion <cedric@maion.com> wrote:
> Eric Wong <normalperson@yhbt.net> wrote:
> > SIGKILL timeout is only a last line of defense when the Ruby VM itself
> > is completely broken. Handling SIGTRAP implies the worker can still
> > respond (and /can/ be rescued), so your SIGTRAP handler is worthless if
> > SIGKILL is required to kill a process.
> Sure. But if the VM is responding, being able to get a backtrace is nice.
> And if it's stuck, you won't get anything indeed, but that's still an
> information (in that case, one may eventually want to get a gdb
> backtrace too). No?

Sure it's nice.  But the point is you should've had something around to
handle it in your app anyways if your worker was capable of responding
to SIGTRAP at all.  The SIGKILL logic only exists in the master because
it must run outside of the worker.

> > See http://unicorn.bogomips.org/Application_Timeouts.html
> Yes, I'm well aware of this. However, when you still get rare unicorn
> timeouts, debugging them is not obvious.
> In my case, a server in a loadbalanced farm sometimes sees all it's
> unicorn workers timeout in the same minute (approx once a day at what
> seems a random time) -- other servers are fine. Couldn't correlate this
> with any specific network/disk/misc system/user activity yet.

I might even crank the unicorn timeout sky high and have something
else (per-worker) handling timeouts + debugging/dumping in this case.

I recall some mailing list threads on similar topics over the years,
gmane has excellent archives and I'd start there (and not the Rubyforge
archives): gmane.org/gmane.comp.lang.ruby.unicorn.general

The Rainbows::ThreadTimeout could be used as a starting point for a Rack
middleware to debug with.

	git clone git://bogomips.org/rainbows
	cat lib/rainbows/thread_timeout.rb
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 4.3.0.2.g4551 gem prerelease
@ 2012-04-27 22:00  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2012-04-27 22:00 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Joel Nimety, George, Matt Smith

Can you guys test this out?  Thanks.  I'll make a real 4.3.1 release
tomorrow.

>From RubyGems.org:  gem install --pre unicorn

ChangeLog since v4.3.0:

commit 4551c8ad4d63d4031c618f76d39532b39e88f9be
Author: Eric Wong <normalperson@yhbt.net>
Date:   Fri Apr 27 14:42:38 2012 -0700

    stream_input: call shutdown(2) if a client EOFs on us
    
    In case the Rack app forks before a client upload is complete,
    shutdown(2) the socket to ensure the client isn't attempting to
    read from us (even if it explicitly stopped writes).

commit 04901da5ae0b4655c83be05d24ae737f1b572002
Author: Eric Wong <normalperson@yhbt.net>
Date:   Fri Apr 27 11:48:16 2012 -0700

    http_server: ignore ENOTCONN (mostly from shutdown(2))
    
    Since there's nothing unicorn can do to avoid this error
    on unconnected/halfway-connected clients, ignoring ENOTCONN
    is a safe bet.
    
    Rainbows! has long had this rescue as it called getpeername(2)
    on untrusted sockets
-- 
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: app error: Socket is not connected (Errno::ENOTCONN)
  2012-04-27 18:59  5% ` Eric Wong
@ 2012-04-27 19:33  0%   ` Matt Smith
  0 siblings, 0 replies; 191+ results
From: Matt Smith @ 2012-04-27 19:33 UTC (permalink / raw)
  To: Eric Wong; +Cc: mongrel-unicorn, George, Joel Nimety

On Fri, Apr 27, 2012 at 11:59 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Are the rest of you guys getting these errors intermittently, or only
> once in a while.
>
> Rainbows! has long had a similar patch as below, but I'm a bit concerned
> as to why Joel is getting this multiple times per request...

I usually get several every request as well (in dev). It can be one or
a handful at a time. They seem to be somewhat random as to when they
are thrown.

Thanks for the patch (and the fantastic support, Eric!). If I can help
narrow down the problem please let me know!

It seems the issue is when unicorn is serving static files, at least
for me. In production, Nginx serves my static files so I doubt there
will be any issue. I do have 4.3.0 deployed on Ubuntu 10.04.4 LTS and
I just checked the logs and I see no errors when I make the same
request.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: app error: Socket is not connected (Errno::ENOTCONN)
  @ 2012-04-27 18:59  5% ` Eric Wong
  2012-04-27 19:33  0%   ` Matt Smith
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2012-04-27 18:59 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Joel Nimety, George, Matt Smith

Joel Nimety <jnimety@continuity.net> wrote:
> I'm getting the following errors multiple times per request when using
> 4.3.0.  I do not receive any errors when using 4.2.1.  Please CC me on
> replies, I'm not subscribed to the mailing list.

Multiple times per request?  Yikes.  This can be expected occasionally.

The below patch will just silently ignore them since they're not
errors we can avoid/deal with any other way.  I'd still prefer
exceptions to not get raised at all (they're expensive).

Are the rest of you guys getting these errors intermittently, or only
once in a while.

Rainbows! has long had a similar patch as below, but I'm a bit concerned
as to why Joel is getting this multiple times per request...

>From 04901da5ae0b4655c83be05d24ae737f1b572002 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Fri, 27 Apr 2012 11:48:16 -0700
Subject: [PATCH] http_server: ignore ENOTCONN (mostly from shutdown(2))

Since there's nothing unicorn can do to avoid this error
on unconnected/halfway-connected clients, ignoring ENOTCONN
is a safe bet.

Rainbows! has long had this rescue as it called getpeername(2)
on untrusted sockets
---
 lib/unicorn/http_server.rb |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index f942e2f..14a6f9a 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -507,7 +507,8 @@ class Unicorn::HttpServer
   # the socket is closed at the end of this function
   def handle_error(client, e)
     msg = case e
-    when EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::EINVAL,Errno::EBADF
+    when EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::EINVAL,Errno::EBADF,
+         Errno::ENOTCONN
       Unicorn::Const::ERROR_500_RESPONSE
     when Unicorn::RequestURITooLongError
       Unicorn::Const::ERROR_414_RESPONSE
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 5%]

* Re: Request-URI Too Long from mongrel-unicorn
  2012-04-12  3:45  4%     ` Eric Wong
@ 2012-04-12  8:18  6%       ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2012-04-12  8:18 UTC (permalink / raw)
  To: unicorn list

Eric Wong <normalperson@yhbt.net> wrote:
> Lawrence Pit <lawrence.pit@gmail.com> wrote:
> > Should there be a limit at all in unicorn? Should it not be assumed
> > this is configured at the webserver level, like:
> > 
> > http://wiki.nginx.org/NginxHttpCoreModule#large_client_header_buffers
> 
> There should be a limit in unicorn, it's cheap to enforce and there
> could be corner cases (nginx bugs, internal security probes) where it's
> helpful.  The unicorn parser is also used by servers (Rainbows!) that
> expect untrusted/malicious clients without nginx to protect it.

On the other hand, the _granularity_ of limits may unnecessary.  There
is already a 112K limit on the overall header size (which IMHO is really
huge).  However, this 112K overall limit is tunable on Rainbows! because
Rainbows! is designed to handle hundreds/thousands of clients in one
process.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: Request-URI Too Long from mongrel-unicorn
  @ 2012-04-12  3:45  4%     ` Eric Wong
  2012-04-12  8:18  6%       ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2012-04-12  3:45 UTC (permalink / raw)
  To: unicorn list

Lawrence Pit <lawrence.pit@gmail.com> wrote:
> IE browsers up to and including IEv8 have a max URL length of 2,083
> characters : http://support.microsoft.com/kb/208427. 

Ah, that probably explains why MAX_URI_LENGTH=2083 in WEBrick
(lib/webrick/httprequest.rb).

> There may be other clients (ruby http clients, bots, traffic
> analysers, database clients with limited column spaces to hold URLs,
> etc.) that can't handle long URLs if you take it too far. So
> practically it's advisable to keep your URLs within limits.

Agreed.  I'm really hoping no apps out there depend on the
Mongrel/unicorn internal limits to enforce database column length :)

> Should there be a limit at all in unicorn? Should it not be assumed
> this is configured at the webserver level, like:
> 
> http://wiki.nginx.org/NginxHttpCoreModule#large_client_header_buffers

There should be a limit in unicorn, it's cheap to enforce and there
could be corner cases (nginx bugs, internal security probes) where it's
helpful.  The unicorn parser is also used by servers (Rainbows!) that
expect untrusted/malicious clients without nginx to protect it.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: using unicorn as a local development server
  @ 2012-02-24 23:10  3% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2012-02-24 23:10 UTC (permalink / raw)
  To: unicorn list

Matt Smith <matt@nearapogee.com> wrote:
> Eric Wong wrote:
> > Normally I just write integration tests (sometimes starting unicorn (or
> > zbatery) + hitting it with curl, but often just mocking a Rack env).
> > Unlike most folks that develop apps that run over HTTP, I have a strong
> > aversion to web browsers. I'd rather pipe curl output to "vim -" if I
> > have to look at any text output from the application.
> 
> This is slightly off topic, so I will make this concise, or this can
> be taken offline.
> 
> What you are talking about, Eric, is exactly the workflow I am working
> toward. I am almost there, but still too dependent on the browser. So
> 2 questions:
> 1) Would you share what you use for integrations tests for rails and
> rack apps in general? (rack test, minitest, capybara, etc.)

For Rack apps, I normally use test/unit (minitest in 1.9) + Rack::Mock*.
test/test_watcher.rb in raindrops[1] is a public example of that.

I don't develop a lot of Rack apps, though.  I haven't touched Rails in
ages, but last time I used test/unit and some builtin Rails test
extensions.

If I'm testing within Ruby, I stay with test/unit because it's
bundled/maintained with the latest version(s) of Ruby.  Back in the day,
I know some projects had trouble migrating to Ruby 1.9.1 because rspec
wasn't 1.9-compatible at the time (it is now).


Back to Unicorn (and Rainbows!)
-------------------------------

For testing HTTP servers (or anything that interacts with
non-Ruby-components), I'll use scripts in other languages
(shell/Perl/awk) and external tools (e.g. curl) to shake out potential
bugs.

I worry about "self-cancelling" bugs which can be hidden from tests
because I didn't understand something (often Ruby itself) well enough.
Ruby 1.9 encodings is/was especially confusing to me, so I reached
outside of Ruby in many cases.


[1] - git clone git://bogomips.org/raindrops

[2] - For the few Rack apps I write, almost all of them are APIs or
      targeted at lynx or curl users.  I hate pretty things :)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 3%]

* Re: large uploads
  @ 2011-10-01  4:06  4%         ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-10-01  4:06 UTC (permalink / raw)
  To: unicorn list

John Joseph Bachir <j@jjb.cc> wrote:
> So buffering the entire request and then sending it to the backend is
> the default behavior of HttpUpstreamModule? That's fascinating. It
> almost seems like that's a bad design choice, because it inhibits the
> possibility of parallel work being done as the request comes in (like,
> it's the opposite of how unix pipes work).

Yes, exactly.  It's a design trade-off; if a client is uploading slowly
(whether intentionally or maliciously) the process-per-client model of
Unicorn is /extremely/ inefficient and will easily be shut down
by slow clients.

If you want to be able to process uploads in a Rack application while
the client is uploading, you'd have to ensure your app is thread-safe
and use Rainbows! + ThreadSpawn/ThreadPool.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: large uploads
  @ 2011-09-29 18:47  5% ` Eric Wong
    0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-09-29 18:47 UTC (permalink / raw)
  To: unicorn list

John Joseph Bachir <j@jjb.cc> wrote:
> My application accepts uploads from users, which can be quite huge in
> some cases. This of course requires setting the unicorn timeout to
> something much higher than 60 seconds, more like 10 minutes.
> 
> Are there any drawbacks to doing this, other than the obvious drawback
> of not killing off long-running requests that are illegitimate?

Are your users remote? (outside of your immediate LAN).

The upload speed (and thus timeout) for unicorn should be based on
the nginx <-> unicorn transfer rate.  Unicorn should never talk to users
directly and users can upload slowly to nginx which buffers requests to the
filesystem, first.

> I've googled quite a bit about this and have found surprisingly little
> -- I guess people who have apps that receive uploads just generally
> don't use unicorn?

I have a LAN-only application that regularly processes uploads several
hundreds of megabytes (via PUT) directly to Unicorn.  The local disk I/O
is often the limiting factor since the parallel requests fill up the
kernel buffers and wait on disk I/O.  The Rack application itself
unfortunately needs to seek/rewind so it must be in the filesystem.

Rainbows! with ThreadSpawn/ThreadPool can process uploads without
buffering them to disk first (but the Rack multipart parser may not).
I often stream several hundred megabytes of data directly to apps on
Rainbows! via PUT requests (curl -T), too.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 5%]

* Re: Timeout callback
  2011-09-26  1:42  4% ` Eric Wong
@ 2011-09-26  6:06  0%   ` Christopher Bailey
  0 siblings, 0 replies; 191+ results
From: Christopher Bailey @ 2011-09-26  6:06 UTC (permalink / raw)
  To: unicorn list

Alex, we were having problems with timeouts, and it killing our logs,
making it near impossible to figure out what was causing the timeout,
etc. We too looked into a solution within Unicorn, but as Eric
explains, it's not possible.

What we wound up doing, which may or may not work for you, is to put
an explicit Timeout wrapper around the code we knew caused this (we
are lucky and knew a specific API call/controller action that was
getting the timeouts).  e.g. wrap with:

  Timeout::timeout(...) do

This wound up actually being a far better solution for us, because our
mobile clients that call this API timeout the HTTP request at 20
seconds anyway, so it was pointless to even get to 60.  So, we now
have a far better solution, one where we can time it out ourselves,
handle the exception, and log the timeout/problem (e.g. we create a
Zendesk ticket, etc.).  Anyway, figured I'd mention that in case it
could work in your case as well...

On Sun, Sep 25, 2011 at 6:42 PM, Eric Wong <normalperson@yhbt.net> wrote:
>
> Alex Sharp <ajsharp@gmail.com> wrote:
> >  Would there be any support for a worker-level timeout callback, for
> >  workers that get killed by the master process for violating the
> >  Unicorn::Configurator.timeout setting?
>
> Something like this /cannot/ be done right.  The unicorn timeout uses
> SIGKILL because SIGKILL is a last resort and not
> catchable/blockable/trappable in user space.  (SIGSTOP is in the same
> boat as SIGKILL).
>
> > I'm thinking the method could be on the Unicorn::Configurator class,
> > something like ".at_timeout_exit". My thinking here is I want to be
> > able to invoke caller() and get a backtrace to figure out the code
> > that's resulting in timeouts.
>
> Getting a backtrace relies on Ruby being in a runnable state.
> If user space (and Ruby) is capable of accepting non-SIGKILL/SIGSTOP,
> you could already be using something along the lines of the Timeout
> module in Ruby stdlib, SystemTimeout, or the Rainbows::ThreadTimeout
> middleware.
>
> In other words, you can already use an application-level timeout
> (even around the entire app dispatch) if you could get a backtrace.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying



--
Christopher Bailey
Cobalt Edge LLC
http://cobaltedge.com
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Timeout callback
  @ 2011-09-26  1:42  4% ` Eric Wong
  2011-09-26  6:06  0%   ` Christopher Bailey
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-09-26  1:42 UTC (permalink / raw)
  To: unicorn list

Alex Sharp <ajsharp@gmail.com> wrote:
>  Would there be any support for a worker-level timeout callback, for
>  workers that get killed by the master process for violating the
>  Unicorn::Configurator.timeout setting?

Something like this /cannot/ be done right.  The unicorn timeout uses
SIGKILL because SIGKILL is a last resort and not
catchable/blockable/trappable in user space.  (SIGSTOP is in the same
boat as SIGKILL).

> I'm thinking the method could be on the Unicorn::Configurator class,
> something like ".at_timeout_exit". My thinking here is I want to be
> able to invoke caller() and get a backtrace to figure out the code
> that's resulting in timeouts. 

Getting a backtrace relies on Ruby being in a runnable state.
If user space (and Ruby) is capable of accepting non-SIGKILL/SIGSTOP,
you could already be using something along the lines of the Timeout
module in Ruby stdlib, SystemTimeout, or the Rainbows::ThreadTimeout
middleware.

In other words, you can already use an application-level timeout
(even around the entire app dispatch) if you could get a backtrace.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 4%]

* Re: Rainbows! or unicorn?
  2011-09-16 12:38 11%   ` russell muetzelfeldt
@ 2011-09-16 17:00  7%     ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-09-16 17:00 UTC (permalink / raw)
  To: unicorn list

russell muetzelfeldt <russm-rubyforge@slofith.org> wrote:
> On 16/09/2011, at 9:11 PM, Eric Wong wrote:
> > :ThreadSpawn + worker_connections=1 and the (default) :Base option are
> > almost the same in Rainbows! if you don't want to worry about your app
> > being thread-safe at all.

<snip>

> I passed on Rainbows::Base just because of the "not intended
> for production use" comment in the docs.

Ah, Base and worker_connections=1 (anywhere) is risky for dealing with
untrusted clients directly.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 7%]

* Re: Rainbows! or unicorn?
  2011-09-16 11:11 13% ` Eric Wong
@ 2011-09-16 12:38 11%   ` russell muetzelfeldt
  2011-09-16 17:00  7%     ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: russell muetzelfeldt @ 2011-09-16 12:38 UTC (permalink / raw)
  To: unicorn list

On 16/09/2011, at 9:11 PM, Eric Wong wrote:
> :ThreadSpawn + worker_connections=1 and the (default) :Base option are
> almost the same in Rainbows! if you don't want to worry about your app
> being thread-safe at all.

my reason for intending to run worker_connections=1 isn't avoiding threading issues, but to allow app reloads without leaving around old instances of the app in server processes that are handling a 2+ hour upload. I passed on Rainbows::Base just because of the "not intended for production use" comment in the docs.


> Rainbows! can (and does by default) limit upload sizes
> (client_max_body_size) for handling untrusted clients who may try to
> run you out of space.

ah, that's a good point. I'll be fronting through Pound for SSL, but having the server process limit upload size would be worthwhile.


> Since performance/scalability isn't your concern, it depends on whether
> you trust your clients to not upload until you run out of disk space.

I trust them to not do that on purpose, but that doesn't mean a lot. :/


cheers

Russell


-----
Russell Muetzelfeldt <russm@slofith.org>
Mundus vult decipi, ergo decipiatur.

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 11%]

* Re: Rainbows! or unicorn?
  2011-09-16 10:19 13% Rainbows! or unicorn? russell muetzelfeldt
@ 2011-09-16 11:11 13% ` Eric Wong
  2011-09-16 12:38 11%   ` russell muetzelfeldt
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-09-16 11:11 UTC (permalink / raw)
  To: unicorn list

russell muetzelfeldt <russm-rubyforge@slofith.org> wrote:
> I'm putting together a small web frontend for a client to upload files
> into an existing application. It's trivial - there will never be more
> than a (small) handful of concurrent connections, but I need a
> streaming rack.input for upload progress on files up to 500MB or so. I
> was planning on using Rainbows! with ThreadSpawn and
> worker_connections=1, then noticed that unicorn is also listed as
> having streaming rack.input.

:ThreadSpawn + worker_connections=1 and the (default) :Base option are
almost the same in Rainbows! if you don't want to worry about your app
being thread-safe at all.

> While what I'm doing is pretty much the opposite of the unicorn design
> case, is there any reason in this scenario for me to use Rainbows!, or
> should I just go with unicorn and enough backends to handle a couple
> of concurrent uploads and the minimal other frontend bits?

Rainbows! can (and does by default) limit upload sizes
(client_max_body_size) for handling untrusted clients who may try to
run you out of space.

Since performance/scalability isn't your concern, it depends on whether
you trust your clients to not upload until you run out of disk space.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 13%]

* Rainbows! or unicorn?
@ 2011-09-16 10:19 13% russell muetzelfeldt
  2011-09-16 11:11 13% ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: russell muetzelfeldt @ 2011-09-16 10:19 UTC (permalink / raw)
  To: mongrel-unicorn

I'm putting together a small web frontend for a client to upload files into an existing application. It's trivial - there will never be more than a (small) handful of concurrent connections, but I need a streaming rack.input for upload progress on files up to 500MB or so. I was planning on using Rainbows! with ThreadSpawn and worker_connections=1, then noticed that unicorn is also listed as having streaming rack.input.

While what I'm doing is pretty much the opposite of the unicorn design case, is there any reason in this scenario for me to use Rainbows!, or should I just go with unicorn and enough backends to handle a couple of concurrent uploads and the minimal other frontend bits?

cheers

Russell


-----
Russell Muetzelfeldt <russm@slofith.org>
Mundus vult decipi, ergo decipiatur.

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 13%]

* SSL support pushed out to unicorn.git :x
@ 2011-09-15 22:46  3% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-09-15 22:46 UTC (permalink / raw)
  To: mongrel-unicorn

Consider this a joke until somebody who knows crypto well can review it
(_and_ kgio-monkey[1]).  I know it works with curl (test case included)
and I can't see plain-text when I strace/tcpdump, that's about it :)

Subject: [PATCH] add preliminary SSL support

This will also be the foundation of SSL support in Rainbows!
and Zbatery.  Some users may also want to use this in
Unicorn on LANs to meet certain security/auditing requirements.
Of course, Nightmare! (in whatever form) should also be able to
use it.
---
  The patch is a big, so you can view it here
  http://bogomips.org/unicorn.git/patch?id=ac346b5abc

  [1] - http://bogomips.org/kgio-monkey/
        git clone git://bogomips.org/kgio-monkey.git

  This is absolutely NOT intended to be an endorsement of the current
  certificate authority system.  Don't support or encourage it.

 lib/unicorn/configurator.rb     |   13 +++--
 lib/unicorn/http_server.rb      |    3 +
 lib/unicorn/ssl_client.rb       |    6 ++
 lib/unicorn/ssl_configurator.rb |  104 +++++++++++++++++++++++++++++++++++++++
 lib/unicorn/ssl_server.rb       |   42 ++++++++++++++++
 script/isolate_for_tests        |    1 +
 t/.gitignore                    |    2 +
 t/sslgen.sh                     |   63 +++++++++++++++++++++++
 t/t0600-https-server-basic.sh   |   48 ++++++++++++++++++
 test/unit/test_sni_hostnames.rb |   47 +++++++++++++++++
 10 files changed, 325 insertions(+), 4 deletions(-)
 create mode 100644 lib/unicorn/ssl_client.rb
 create mode 100644 lib/unicorn/ssl_configurator.rb
 create mode 100644 lib/unicorn/ssl_server.rb
 create mode 100755 t/sslgen.sh
 create mode 100755 t/t0600-https-server-basic.sh
 create mode 100644 test/unit/test_sni_hostnames.rb

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 3%]

* Re: Sending ABRT to timeout-errant process before KILL
  2011-09-08 19:13  6% ` Eric Wong
@ 2011-09-09 23:01  0%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-09-09 23:01 UTC (permalink / raw)
  To: unicorn list; +Cc: J. Austin Hughey

Eric Wong <normalperson@yhbt.net> wrote:
> sleeping means they haven't accepted a client connection, yet.  Not
> sleeping while processing a client request.   I'll clarify that in the
> code.

Pushed to git://bogomips.org/unicorn.git
(commit d209910e29d4983f8346233262a49541464252c1)

> 	git clone git://bogomips.org/rainbows
> 	cat rainbows/lib/rainbows/thread_timeout.rb
> 
> This is conceptually similar to "timeout" in the Ruby standard library,
> but does not allow nesting.
> 
> I'll try to clarify more later today if you have questions, in a bit of
> a rush right now.

Did you manage to get anything going based on this?  I should add that
this has the same chance of working as a SIGABRT handler written in
Ruby (but is less intrusive).

Hopefully the following diagram/chart from a previous post can explain
why you can't rely on trappable signal handlers if the Ruby VM is in a
bad state:

  http://mid.gmane.org/20110817201323.GA24581@dcvr.yhbt.net

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Sending ABRT to timeout-errant process before KILL
  @ 2011-09-08 19:13  6% ` Eric Wong
  2011-09-09 23:01  0%   ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-09-08 19:13 UTC (permalink / raw)
  To: unicorn list; +Cc: J. Austin Hughey

"J. Austin Hughey" <jhughey@engineyard.com> wrote:
> The general idea is that I'd like to have some way to "warn" the
> application when it's about to be killed.  I've patched
> murder_lazy_workers to send an abort signal via kill_worker, sleep for
> 5 seconds, then look and see if the process still exists by using
> Process.getpgid.  If so, it sends the original kill command, and if
> not, a rescue block kicks in to prevent the raised error from
> Process.getpgid from making things explode.

The problem with anything other than SIGKILL (or SIGSTOP) is that it
assumes the Ruby VM is working and in a good state.

> I've created a simulation app, built on Rails 3.0, that uses a generic
> "posts" controller to simulate a long-running request.  Instead of
> just throwing a straight-up sleep 65 in there, I have it running
> through sleeping 1 second on a decrementing counter, and doing that 65
> times.  The reason is because, assuming I've read the code correctly,
> even with my "skip sleeping workers" commented line below, it'll skip
> over the process, thus rendering my simulation of a long-running
> process invalid.  However, clarification on this point is certainly
> welcome.  You can see the app here:
> https://github.com/jaustinhughey/unicorn_test/blob/master/app/controllers/posts_controller.rb

(purely for educational purposes, since I'll point you towards another
approach I believe is better)

              Signal.trap(:ABRT) do
                # Write some stuff to the Rails log
                logger.info "Caught Unicorn kill exception!"

If this is the logger that ships with Ruby, it locks a Mutex, so it'll
deadlock if another SIGABRT is received while logging the above
statement (a very small window, admittedly).

                # Do a controlled disconnect from ActiveRecord
                ActiveRecord::Base.connection.disconnect!

Likewise, if AR needs to lock internal structures before disconnecting,
it also must be reentrant.  Ruby's normal Mutex implementation is not
reentrant-safe.


> So it looks like Worker 1 is hitting a strange/false timeout of
> 1315467289 seconds, which isn't really possible as it wasn't even
> running 1315467289 seconds prior to that (which equates to roughly 41
> years ago if my math is right).

You're getting this because you removed the following line:

      0 == tick and next # skip workers that are sleeping

sleeping means they haven't accepted a client connection, yet.  Not
sleeping while processing a client request.   I'll clarify that in the
code.

> Needless to say, I'm a bit stumped at this point, and would sincerely
> appreciate another point of view on this.  Am I going about this all
> wrong?  Is there a better approach I should consider?  And if I'm on
> the right track, how can I get this to work regardless of how many
> Unicorn workers are running?

Since it's an application error, it can be done as middleware.  You can
try something like the Rainbows::ThreadTimeout middleware, it's
currently Rainbows! specific but can easily be made to work with
Unicorn.

	git clone git://bogomips.org/rainbows
	cat rainbows/lib/rainbows/thread_timeout.rb

This is conceptually similar to "timeout" in the Ruby standard library,
but does not allow nesting.

I'll try to clarify more later today if you have questions, in a bit of
a rush right now.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Nightmare! - an nginx alternative for unicorn
@ 2011-09-03 10:52  3% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-09-03 10:52 UTC (permalink / raw)
  To: mongrel-unicorn

This is a slow client buffering layer which may be used instead
of nginx to protect Unicorn from slow clients.

Nightmare! will _never_ beat nginx in raw throughput nor
performance.  It /may/ be easier to setup than nginx and a
suitable alternative to Rainbows! for users who do not wish to
maintain a thread-safe/async-safe Rack application.

Code changes to the existing Unicorn codebase are minimal and
unlikely to impact existing users.  Nightmare! will never be
enabled by default.

Nightmare! supports streaming responses (new in Rails 3.1, but
ancient to Rack) with "lazy buffering" of upstream responses.
It'll stream as much of the response as possible immediately and
then buffer any data that can't.

Userspace memory buffering is kept to a minimum; if it can't fit
into generously-sized skbs (at least on modern Linux), it'll
be buffered to the _filesystem_ (which may be tmpfs or a real FS
with dirty ratios cranked up).

HTTPS support is planned/wired.  Somebody with SSL/crypto
knowledge needs to review
{kgio-monkey}[http://bogomips.org/kgio-monkey/] before it can be
trusted.  V nz n zbaxrl!  Qb abg gehfg zl pbqr!

Since Nightmare! already uses sendfile to serve buffers, a
"try_files" directive will be added to bypass Rack for simple
static file serving.  It will not serve directory indices nor
gzip, Rack already supports those.

Nightmare! internals are still in flux, and likely to remain so.
Like Unicorn internals, do not consider Nightmare itself a
stable development API unless explicitly told otherwise.

****** Use Rack for application logic ******

Nightmare! should work on Ruby 1.8.x and 1.9.x.

Configuration directives are not implemented, yet.   Eventually
it will support operation in standalone mode (so Nightmare! can
talk to Unicorn on different machines).

Extra RubyGems required (in addition to what Unicorn requires):

* kcar - client-side HTTP parser based on the Unicorn one
* sendfile - for sendfile() support
* kgio-monkey - (upcoming, optional) for HTTPS support


I've pushed this up to the "nightmare" branch of
git://bogomips.org/unicorn.git || http://bogomips.org/unicorn.git

Please review the code/tests if you have a chance, thanks for reading!

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 3%]

* Re: Strange quit behavior
  2011-09-01 18:45  0%                 ` Alex Sharp
@ 2011-09-01 19:46  0%                   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-09-01 19:46 UTC (permalink / raw)
  To: unicorn list

Alex Sharp <ajsharp@gmail.com> wrote:
> On Tuesday, August 30, 2011 at 5:33 PM, Eric Wong wrote:
> > Eric Wong <normalperson@yhbt.net (mailto:normalperson@yhbt.net)> wrote:
> > > + trap(sig) do
> > > + @logger.debug("received SIG#{sig}")
> > 
> > I'm even more glad I didn't apply this patch to Unicorn. I completely
> > forgot Logger uses a mutex internally (even though it doesn't need to
> > when writing to a POSIX-compliant file system).
> > 
> > Rainbows! has a similar issue I fixed/worked around:
> > http://mid.gmane.org/20110830233232.GA19633@dcvr.yhbt.net
> 
> So are you saying that Unicorn is affected with the same issue? I'm
> confused -- does the master send USR1 signals to the workers when it
> receives a USR2? I'm not sending a USR1 signal to the master. 

No, Unicorn is not affected.  It /would have/ been, had I applied that
patch.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: Strange quit behavior
  2011-08-31  0:33  4%               ` Eric Wong
@ 2011-09-01 18:45  0%                 ` Alex Sharp
  2011-09-01 19:46  0%                   ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Alex Sharp @ 2011-09-01 18:45 UTC (permalink / raw)
  To: unicorn list

On Tuesday, August 30, 2011 at 5:33 PM, Eric Wong wrote:
> Eric Wong <normalperson@yhbt.net (mailto:normalperson@yhbt.net)> wrote:
> > + trap(sig) do
> > + @logger.debug("received SIG#{sig}")
> 
> I'm even more glad I didn't apply this patch to Unicorn. I completely
> forgot Logger uses a mutex internally (even though it doesn't need to
> when writing to a POSIX-compliant file system).
> 
> Rainbows! has a similar issue I fixed/worked around:
> http://mid.gmane.org/20110830233232.GA19633@dcvr.yhbt.net
So are you saying that Unicorn is affected with the same issue? I'm confused -- does the master send USR1 signals to the workers when it receives a USR2? I'm not sending a USR1 signal to the master. 

- Alex
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: Strange quit behavior
  @ 2011-08-31  0:33  4%               ` Eric Wong
  2011-09-01 18:45  0%                 ` Alex Sharp
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-08-31  0:33 UTC (permalink / raw)
  To: unicorn list

Eric Wong <normalperson@yhbt.net> wrote:
> +      trap(sig) do
> +        @logger.debug("received SIG#{sig}")

I'm even more glad I didn't apply this patch to Unicorn.  I completely
forgot Logger uses a mutex internally (even though it doesn't need to
when writing to a POSIX-compliant file system).

Rainbows! has a similar issue I fixed/worked around:
  http://mid.gmane.org/20110830233232.GA19633@dcvr.yhbt.net

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 4.1.0 - small updates and fixes
@ 2011-08-20  0:42  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-08-20  0:42 UTC (permalink / raw)
  To: mongrel-unicorn

Changes:

* Rack::Chunked and Rack::ContentLength middlewares are loaded
  by default for RACK_ENV=(development|deployment) users to match
  Rack::Server behavior.  As before, use RACK_ENV=none if you want
  fine-grained control of your middleware.  This should also
  help users of Rainbows! and Zbatery.

* CTL characters are now rejected from HTTP header values

* Exception messages are now filtered for [:cntrl:] characters
  since application/middleware authors may forget to do so

* Workers will now terminate properly if a SIGQUIT/SIGTERM/SIGINT
  is received while during worker process initialization.

* close-on-exec is explicitly disabled to future-proof against
  Ruby 2.0 changes [ruby-core:38140]

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://bogomips.org/unicorn.git
* http://unicorn.bogomips.org/NEWS.atom.xml

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [PATCH] Rack::Chunked and ContentLength middlewares by default
@ 2011-08-19 21:07 12% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-08-19 21:07 UTC (permalink / raw)
  To: mongrel-unicorn

This was prompted by the recent thread in
http://thread.gmane.org/gmane.comp.lang.ruby.unicorn.general/1106
starting with Message-ID:
  CAL3dLFrkDix=L-STUnrxy7Wuc4wDZOb05NLbG6HABvFJNGmnmQ@mail.gmail.com

Rainbows! users have also run into this problem a few times, too:
  AANLkTimSuK7-ihgCa00D-fot4U-FbZt2diQFndyN4uit@mail.gmail.com
  2B157204-E5C6-4F5D-98A9-E2A79F9F9765@christophsturm.com

>From 1077961a3f8933c65d39c7e6c9ed6ff3b6b53647 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Fri, 19 Aug 2011 20:47:29 +0000
Subject: [PATCH] Rack::Chunked and ContentLength middlewares by default

This is needed to match the behavior of Rack::Server for
RACK_ENV=(deployment|development), actually.  This won't
affect users of other RACK_ENV values.

This change has minor performance consequences, so users
negatively affected should set RACK_ENV to "none" instead for
full control of their middleware stack.

This mainly affects Rainbows!/Zbatery users since they have
persistent connections and /need/ Content-Length or
Transfer-Encoding:chunked headers.
---
 lib/unicorn.rb |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index bb5409b..b882ce3 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -50,9 +50,14 @@ module Unicorn
       pp({ :inner_app => inner_app }) if $DEBUG
 
       # return value, matches rackup defaults based on env
+      # Unicorn does not support persistent connections, but Rainbows!
+      # and Zbatery both do.  Users accustomed to the Rack::Server default
+      # middlewares will need ContentLength/Chunked middlewares.
       case ENV["RACK_ENV"]
       when "development"
         Rack::Builder.new do
+          use Rack::ContentLength
+          use Rack::Chunked
           use Rack::CommonLogger, $stderr
           use Rack::ShowExceptions
           use Rack::Lint
@@ -60,6 +65,8 @@ module Unicorn
         end.to_app
       when "deployment"
         Rack::Builder.new do
+          use Rack::ContentLength
+          use Rack::Chunked
           use Rack::CommonLogger, $stderr
           run inner_app
         end.to_app
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 12%]

* Re: Unicorn logging in production env
  @ 2011-08-17 10:04  6%       ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-08-17 10:04 UTC (permalink / raw)
  To: unicorn list; +Cc: Serg Podtynnyi

Eric Wong <normalperson@yhbt.net> wrote:
> The the rest of the app should have local timeouts for all
> non-deterministic calls.

If you're lazy, maybe the following middleware works, too, I haven't
ever needed it but wrote (and later rewrote it) as a proof-of-concept:
  http://bogomips.org/rainbows.git/tree/lib/rainbows/thread_timeout.rb

It should work with bare Unicorn if you remove the "Rainbows" references
and the negative threshold handling in "initialize".   If somebody wants
I can split it into it's own gem (should work with WEBrick, Mongrel,
Passenger, and maybe Thin), too).

Unlike the Timeout library in Ruby stdlib, this doesn't allow nested
timeout {} calls so it's simpler/possibly-safer in this regard...

I even posted a request-for-review for the middleware on ruby-talk but
didn't get any responses:
  http://mid.gmane.org/20110421232615.GA27468@dcvr.yhbt.net

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Re: Unicorn vs Apache
@ 2011-07-11 21:43  4% Matt Smith
  0 siblings, 0 replies; 191+ results
From: Matt Smith @ 2011-07-11 21:43 UTC (permalink / raw)
  To: mongrel-unicorn

Thanks Eric and Steve,

I really appreciate the input and time. And thanks for reading through
my typos... You understood exactly what I meant to say.

I have read about Rainbows! before. I will have to give it a shot on
some upcomming projects, as I really enjoy using Unicorn.

Thanks for pointing out rack-pagespeed.

Much thanks,

Matt Smith



_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: Unicorn vs Apache
  @ 2011-07-11 18:45  5% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-07-11 18:45 UTC (permalink / raw)
  To: unicorn list

Matt Smith <matthewcalebsmith@gmail.com> wrote:
> Hello all,
> 
> I have always deploys rails apps with unicorn and nginx as a reverse
> proxy in the past. However, I am working with a new firm and they would
> like to use Apache with mod_pagespeed in front of unicorn.
> 
> We will be deploying a rails 3.1 app with streaming. To me,
> mod_pagespeed does not seem like a magic bullet, as it appears to be a
> collection of best practices which have to be rewritten as a page is
> being served.

note: mod_pagespeed is totally independent if Rails 3.1 streaming.

Keep in mind the most important feature of nginx Unicorn relies on (full
request/response buffering) also defeats Rails 3.1 streaming because of
how it's currently implemented.

> Why not code with the best practices in mind as to not
> have to scan and yet again reprocess the content?

My thoughts exactly :)  But implementing those best practices
isn't cheap, either.  If you have to run mod_pagespeed, the safest
configuration would be:

  nginx <-> mod_pagespeed <-> unicorn

Of course that would introduce latency.  Maybe there's Rack middleware
that works like mod_pagespeed...

> Searching on the web, I find few entries about mod_page speed and rails,
> and relatively few entries about apache with unicorn.

Apache + Unicorn is still unsupported since (as far as anybody knows),
it doesn't fully buffer responses and requests to completely isolate
Unicorn from the harmful effects of slow clients.

> To me it seems that using best practices, with appropriate use of
> caching, and nginx would be a better solution than apache with
> mod_pagespeed.

I agree.

> Questions:
> 
> 1) Does nginx provide any necessary services to unicorn, that apache
> cannot provide?

nginx provides full request + response buffering (all uploads, and large
response bodies that can't fit into socket buffers).  nginx also
provides inexpensive handling of idle keepalive connections; so you
can have a lot of idle connections for little memory usage.

Apache 2.x mpm_event /should/ be as good *if* (and only if) it an
provide the full request/response buffering; but I don't have any
experience with it.  mpm_worker can be fairly efficient with Linux+NPTL
if you can use smaller pthreads stack sizes (and are using 64-bit).

> 2) If apache can provide all necessary services, does it do them as well
> as nginx.

Buffering + keepalive maintenance is the most important thing I care
about, and as far as I know, Apache doesn't do that as well as nginx.

> 3) We would like the fastest user experience possible. Does apache with
> mod_pagespeed hold any weight here over nginx?

The full request/response buffering of nginx also hurts latency a
little, so the user experience is slightly worse under low load.
However, nginx does the best job of keeping things going under high
load.

> 4) Part of a fast user experience is how fast you get the page, correct?
> Seems like nginx has the upper hand here, due to nginx's nio vs apache's
> threading.

Not sure what "nio" means... non-blocking I/O?  Apache can use mpm_event
which is like the non-blocking + event loop nginx uses, but AFAIK the
buffering of requests responses is not like what nginx implements.

> 5) Would it make since to put nginx in front of apache w/mod_pagespeed
> (or visa versa), to use the applicable mod_pagespeed filters? Or is this
> a bad idea to begin with?

nginx <-> mod_pagespeed <-> unicorn should be alright, but of course not
ideal because of the extra copying + latency involved.

For non-Ruby/Rack apps, I've always recommended nginx sit in front of
any Apache 1.3 or 2.x+mpm_prefork apps that are exposed to slow clients,
too.

> 6) Is there anything else I am missing? Are there any specific resources
> I need to look at?

If you're willing to take a risk with Rainbows!, I posted some notes
about making it work with Rails 3.1 streaming here:

http://thread.gmane.org/gmane.comp.lang.ruby.rainbows.general/229

Keep in mind nobody I know of uses Rainbows! in production; but
it /should/ be able to work without nginx (or any proxy).

> 7) Should we go with nginx or apache? (Opinions ok.)

nginx remains the only supported reverse proxy for Unicorn at this
moment.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 5%]

* Re: Unicorn and streaming in Rails 3.1
  @ 2011-06-25 20:33  7% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-06-25 20:33 UTC (permalink / raw)
  To: unicorn list

Xavier Noria <fxn@hashref.com> wrote:
> If it is a real bad idea, is the recommendation
> to Unicorn users that they should just ignore this new feature?

Another thing, Rainbows! + ThreadSpawn/ThreadPool concurrency may do the
trick (without needing nginx at all).  The per-client overhead of
Rainbows! + threads (several hundred KB) is higher than nginx, but still
much lower than Unicorn.

All your Rails code must be thread-safe, though.

If you use Linux, XEpollThreadPool/XEpollThreadSpawn can be worth a
try, too.  The cost of completely idle keepalive clients should be
roughly inline with nginx.

If you want to forgo thread-safety, Rainbows! + StreamResponseEpoll[1]
+ ForceStreaming middleware[2] may also be an option, too (needs nginx).


Keep in mind that I don't know of anybody using Rainbows! for any
serious sites, so there could still be major bugs :)

Rainbows! http://rainbows.rubyforge.org/


[1] currently in rainbows.git, will probably be released this weekend...

[2] I'll probably move this to Rainbows! instead of a Unicorn branch:
    http://bogomips.org/unicorn.git/tree/lib/unicorn/force_streaming.rb?h=force_streaming
    This is 100% untested, I've never run it.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 7%]

* Re: query string max length tweak ?
  2011-06-23 19:31  4%   ` Mohit Chawla
@ 2011-06-24  0:38  4%     ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-06-24  0:38 UTC (permalink / raw)
  To: unicorn list

Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:
> Also, I am new to using unicorn, hence not aware if
> using Rainbows! in this scenario can be achieved/is suggested ? Nice
> to know about limiting the entire size header in future releases ( in
> prod we are using 1.1.4 ! ).

No, Rainbows! is focused on network concurrency.  Unicorn can just use
nginx for stopping large requests (large_client_header_buffers in nginx)
and not be concerned about large headers at all.

> Also, sorry for the multiple html mails before this !

I just made the mailman config on rubyforge kill all HTML :)

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: query string max length tweak ?
  2011-06-23 18:37  4% ` Eric Wong
@ 2011-06-23 19:31  4%   ` Mohit Chawla
  2011-06-24  0:38  4%     ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Mohit Chawla @ 2011-06-23 19:31 UTC (permalink / raw)
  To: unicorn list

Hi,

On Fri, Jun 24, 2011 at 12:07 AM, Eric Wong <normalperson@yhbt.net> wrote:
> You can change the following lines in ext/unicorn/global_variables.h
> and rebuild the parser:
>
>  DEF_MAX_LENGTH(REQUEST_URI, 1024 * 12);
>  ...
>  DEF_MAX_LENGTH(QUERY_STRING, (1024 * 10));
>
> There's currently no way to configure them otherwise, but I can consider
> dropping these per-field limits entirely.  We now have a config option
> to set the maximum header size (only exposed via Rainbows!, though).
>
> This means we would only care about the size of the entire header, and
> not the size of any individual element.  I consider the size of the
> entire header more important anyways.  We currently don't have
> limits on the number of headers that can be sent.

Thanks a lot, yes I was trying out changing only the QUERY_STRING max
length, but that wasn't working. Will change the REQUEST_URI max
length as well. Also, I am new to using unicorn, hence not aware if
using Rainbows! in this scenario can be achieved/is suggested ? Nice
to know about limiting the entire size header in future releases ( in
prod we are using 1.1.4 ! ).

Also, sorry for the multiple html mails before this !
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: query string max length tweak ?
  @ 2011-06-23 18:37  4% ` Eric Wong
  2011-06-23 19:31  4%   ` Mohit Chawla
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-06-23 18:37 UTC (permalink / raw)
  To: unicorn list

Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:
> Hi,
> 
> Is there a way that would allow us to increase the max length allowed
> for query_string ? We have been getting 414s on large GET requests. We
> are running the puppet master under unicorn behind nginx. I am not
> sure of the general implications of this, whether its suggested/safe
> even, but it is somewhat of a necessity. Any suggestions ?

You can change the following lines in ext/unicorn/global_variables.h
and rebuild the parser:

  DEF_MAX_LENGTH(REQUEST_URI, 1024 * 12);
  ...
  DEF_MAX_LENGTH(QUERY_STRING, (1024 * 10));

There's currently no way to configure them otherwise, but I can consider
dropping these per-field limits entirely.  We now have a config option
to set the maximum header size (only exposed via Rainbows!, though).

This means we would only care about the size of the entire header, and
not the size of any individual element.  I consider the size of the
entire header more important anyways.  We currently don't have
limits on the number of headers that can be sent.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 3.7.0 - minor feature update
@ 2011-06-09 20:55  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-06-09 20:55 UTC (permalink / raw)
  To: mongrel-unicorn

Changes:

* miscellaneous documentation improvements
* return 414 (instead of 400) for Request-URI Too Long
* strip leading and trailing linear whitespace in header values

User-visible improvements meant for Rainbows! users:

* add :ipv6only "listen" option (same as nginx)
---
* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [PATCH] configurator: add :ipv6only directive
  @ 2011-06-07 21:05  7% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-06-07 21:05 UTC (permalink / raw)
  To: mongrel-unicorn

Eric Wong <normalperson@yhbt.net> wrote:
> Unicorn itself supports IPv6, too, but nobody uses/needs it.
> I'll add :ipv6only support shortly (probably tomorrow).

Pushed to unicorn.git (c3880bb0cc00821d1715a7dd94b0b76a03a7ace0)

Documentation below:

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index b6ad022..0b84d53 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -281,6 +281,22 @@ class Unicorn::Configurator
   #
   #   Default: +true+ in \Unicorn 3.4+, +false+ in Rainbows!
   #
+  # [:ipv6only => true or false]
+  #
+  #   This option makes IPv6-capable TCP listeners IPv6-only and unable
+  #   to receive IPv4 queries on dual-stack systems.  A separate IPv4-only
+  #   listener is required if this is true.
+  #
+  #   This option is only available for Ruby 1.9.2 and later.
+  #
+  #   Enabling this option for the IPv6-only listener and having a
+  #   separate IPv4 listener is recommended if you wish to support IPv6
+  #   on the same TCP port.  Otherwise, the value of \env[\"REMOTE_ADDR\"]
+  #   will appear as an ugly IPv4-mapped-IPv6 address for IPv4 clients
+  #   (e.g ":ffff:10.0.0.1" instead of just "10.0.0.1").
+  #
+  #   Default: Operating-system dependent
+  #
   # [:tries => Integer]
   #
   #   Times to retry binding a socket if it is already in use
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 7%]

* Re: workers not utilizing multiple CPUs
       [not found]               ` <BANLkTikC5D+0pUDRDgvQbzu=dpwmdKNY=A@mail.gmail.com>
@ 2011-06-01  6:51  0%             ` Nate Clark
  0 siblings, 0 replies; 191+ results
From: Nate Clark @ 2011-06-01  6:51 UTC (permalink / raw)
  To: unicorn list

Thanks for the responses, all.

Eric, you were right, our load was not enough. We had just started on
load testing our app, and I think we started with too many app servers
and not enough load. Once we cranked up the load and used fewer
instances, we're now definitely seeing all CPU cores being utilized. I
was not aware that the kernel would optimize like you described.

Once we did start seeing heavier load, our collectd data and htop were
reporting usage on the virtual cores correctly.
Thanks again, very happy with the results so far,

Nate

On Wed, Jun 1, 2011 at 2:35 PM, Nate Clark <nate@pivotallabs.com> wrote:
>
> Thanks for the responses, all.
> Eric, you were right, our load was not enough. We had just started on load testing our app, and I think we started with too many app servers and not enough load. Once we cranked up the load and used fewer instances, we're now definitely seeing all CPU cores being utilized. I was not aware that the kernel would optimize like you described.
> Once we did start seeing heavier load, our collectd data and htop were reporting usage on the virtual cores correctly.
> Thanks again, very happy with the results so far,
> Nate
> On Tue, May 31, 2011 at 11:55 PM, Clifton King <cliftonk@gmail.com> wrote:
>>
>> Thanks Eric, I had expected that to be the case (we are under light
>> load as of now).
>>
>> On Tue, May 31, 2011 at 10:48 AM, Eric Wong <normalperson@yhbt.net> wrote:
>> > Clifton King <cliftonk@gmail.com> wrote:
>> >> We experience the same problem. I believe the problem has more to do
>> >> with the kernel CPU scheduler than anything else. If you figure put a
>> >> reliable way to spread the load, I'd like to hear it.
>> >
>> > Load not being spread is /not/ a problem unless there are requests that
>> > get stuck in the listen queue.
>> >
>> > If no requests are actually stuck in the queue (light load), the kernel
>> > is right to put requests into the most recently used worker since it can
>> > get better CPU cache behavior this way.
>> >
>> >
>> > == The real problem
>> >
>> > Under high loads (many cores, fast responses), Unicorn currently uses
>> > more resources because of non-blocking accept() + select().  This isn't
>> > a noticeable problem for most machines (1-16 cores).
>> >
>> > Future versions of Unicorn may take advantage of /blocking/ accept()
>> > optimizations under Linux.  Rainbows! already lets you take advantage
>> > of this behavior if you meet the following requirements:
>> >
>> > * Ruby 1.9.x under Linux
>> > * only one listen socket (if worker_connections == 1 under Rainbows!)
>> > * use ThreadPool|XEpollThreadPool|XEpollThreadSpawn|XEpoll
>> >
>> > I haven't had a chance to benchmark any of this on very big machines so
>> > I have no idea how well it actually works compared to Unicorn, only how
>> > well it works in theory :)
>> >
>> >
>> > Blocking accept() under Ruby 1.9.x + Linux should distribute load evenly
>> > across workers in all situations, even in the non-busy cases where load
>> > distribution doesn't matter (your case :).
>> >
>> > [1] - http://rainbows.rubyforge.org/Rainbows/XEpollThreadPool.html
>> >
>> > --
>> > Eric Wong
>> > _______________________________________________
>> > Unicorn mailing list - mongrel-unicorn@rubyforge.org
>> > http://rubyforge.org/mailman/listinfo/mongrel-unicorn
>> > Do not quote signatures (like this one) or top post when replying
>> >
>> _______________________________________________
>> Unicorn mailing list - mongrel-unicorn@rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
>> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: workers not utilizing multiple CPUs
  2011-05-31 15:48  6%       ` Eric Wong
@ 2011-05-31 15:55  0%         ` Clifton King
       [not found]               ` <BANLkTikC5D+0pUDRDgvQbzu=dpwmdKNY=A@mail.gmail.com>
  0 siblings, 1 reply; 191+ results
From: Clifton King @ 2011-05-31 15:55 UTC (permalink / raw)
  To: unicorn list

Thanks Eric, I had expected that to be the case (we are under light
load as of now).

On Tue, May 31, 2011 at 10:48 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Clifton King <cliftonk@gmail.com> wrote:
>> We experience the same problem. I believe the problem has more to do
>> with the kernel CPU scheduler than anything else. If you figure put a
>> reliable way to spread the load, I'd like to hear it.
>
> Load not being spread is /not/ a problem unless there are requests that
> get stuck in the listen queue.
>
> If no requests are actually stuck in the queue (light load), the kernel
> is right to put requests into the most recently used worker since it can
> get better CPU cache behavior this way.
>
>
> == The real problem
>
> Under high loads (many cores, fast responses), Unicorn currently uses
> more resources because of non-blocking accept() + select().  This isn't
> a noticeable problem for most machines (1-16 cores).
>
> Future versions of Unicorn may take advantage of /blocking/ accept()
> optimizations under Linux.  Rainbows! already lets you take advantage
> of this behavior if you meet the following requirements:
>
> * Ruby 1.9.x under Linux
> * only one listen socket (if worker_connections == 1 under Rainbows!)
> * use ThreadPool|XEpollThreadPool|XEpollThreadSpawn|XEpoll
>
> I haven't had a chance to benchmark any of this on very big machines so
> I have no idea how well it actually works compared to Unicorn, only how
> well it works in theory :)
>
>
> Blocking accept() under Ruby 1.9.x + Linux should distribute load evenly
> across workers in all situations, even in the non-busy cases where load
> distribution doesn't matter (your case :).
>
> [1] - http://rainbows.rubyforge.org/Rainbows/XEpollThreadPool.html
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: workers not utilizing multiple CPUs
  @ 2011-05-31 15:48  6%       ` Eric Wong
  2011-05-31 15:55  0%         ` Clifton King
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-05-31 15:48 UTC (permalink / raw)
  To: unicorn list

Clifton King <cliftonk@gmail.com> wrote:
> We experience the same problem. I believe the problem has more to do
> with the kernel CPU scheduler than anything else. If you figure put a
> reliable way to spread the load, I'd like to hear it.

Load not being spread is /not/ a problem unless there are requests that
get stuck in the listen queue.

If no requests are actually stuck in the queue (light load), the kernel
is right to put requests into the most recently used worker since it can
get better CPU cache behavior this way.


== The real problem

Under high loads (many cores, fast responses), Unicorn currently uses
more resources because of non-blocking accept() + select().  This isn't
a noticeable problem for most machines (1-16 cores).

Future versions of Unicorn may take advantage of /blocking/ accept()
optimizations under Linux.  Rainbows! already lets you take advantage
of this behavior if you meet the following requirements:

* Ruby 1.9.x under Linux
* only one listen socket (if worker_connections == 1 under Rainbows!)
* use ThreadPool|XEpollThreadPool|XEpollThreadSpawn|XEpoll

I haven't had a chance to benchmark any of this on very big machines so
I have no idea how well it actually works compared to Unicorn, only how
well it works in theory :)


Blocking accept() under Ruby 1.9.x + Linux should distribute load evenly
across workers in all situations, even in the non-busy cases where load
distribution doesn't matter (your case :).

[1] - http://rainbows.rubyforge.org/Rainbows/XEpollThreadPool.html

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* [PATCH] doc: add Links page to help folks find relevant info
@ 2011-05-23 18:37 10% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-05-23 18:37 UTC (permalink / raw)
  To: mongrel-unicorn

Older announcements on our mailing list could be harder
to find.
---

 I just pushed this out to unicorn.git and
 http://unicorn.bogomips.org/Links.html

 If there are any other project announcements I missed here, please let
 us know, thanks!

 .document |    1 +
 Links     |   53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 0 deletions(-)
 create mode 100644 Links

diff --git a/.document b/.document
index 317e36b..8c7b7cb 100644
--- a/.document
+++ b/.document
@@ -17,3 +17,4 @@ unicorn_1
 unicorn_rails_1
 ISSUES
 Sandbox
+Links
diff --git a/Links b/Links
new file mode 100644
index 0000000..e7f5e60
--- /dev/null
+++ b/Links
@@ -0,0 +1,53 @@
+= Related Projects
+
+If you're interested in \Unicorn, you may be interested in some of the projects
+listed below.  If you have any links to add/change/remove, please tell us at
+mailto:mongrel-unicorn@rubyforge.org!
+
+== Disclaimer
+
+The \Unicorn project is not responsible for the content in these links.
+Furthermore, the \Unicorn project has never, does not and will never endorse:
+
+* any for-profit entities or services
+* any non-{Free Software}[http://www.gnu.org/philosophy/free-sw.html]
+
+The existence of these links does not imply endorsement of any entities
+or services behind them.
+
+=== For use with \Unicorn
+
+* {Bluepill}[https://github.com/arya/bluepill] -
+  a simple process monitoring tool written in Ruby
+
+* {golden_brindle}[https://github.com/simonoff/golden_brindle] - tool to
+  manage multiple \Unicorn instances/applications on a single server
+
+* {raindrops}[http://raindrops.bogomips.org/] - real-time stats for
+  preforking Rack servers
+
+=== \Unicorn is written to work with
+
+* {Rack}[http://rack.rubyforge.org/] - a minimal interface between webservers
+  supporting Ruby and Ruby frameworks
+
+* {Ruby}[http://ruby-lang.org/] - the programming language of Rack and \Unicorn
+
+* {nginx}[http://nginx.org/] - the reverse proxy for use with \Unicorn
+
+* {kgio}[http://bogomips.org/kgio/] - the I/O library written for \Unicorn
+
+=== Derivatives
+
+* {Green Unicorn}[http://gunicorn.org/] - a Python version of \Unicorn
+
+* {Rainbows!}[http://rainbows.rubyforge.org/] - \Unicorn for sleepy
+  apps and slow clients.
+
+=== Prior Work
+
+* {Mongrel}[http://mongrel.rubyforge.org/] - the awesome webserver \Unicorn is
+  based on
+
+* {david}[http://bogomips.org/david.git] - a tool to explain why you need
+  nginx in front of \Unicorn
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 10%]

* [PATCH 1/2] examples/nginx.conf: clarify proxy_buffering for Rails 3.1
  2011-04-27 21:12  4% ` [PATCH 0/2] attempting to clarify docs for streaming Eric Wong
@ 2011-04-27 21:12 10%   ` Eric Wong
  2011-04-27 21:12 13%   ` [PATCH 2/2] configurator: attempt to clarify :tcp_nopush/:tcp_nodelay Eric Wong
  1 sibling, 0 replies; 191+ results
From: Eric Wong @ 2011-04-27 21:12 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Aaron Patterson

I've tested with nginx 1.0.0 and confirmed "proxy_buffering off;"
can cause Unicorn to block on a slow client reading a
large response.  While there's a potential (client-visible)
performance improvement with Rails 3.1 streaming responses, it
can also hurt the server with slow clients.

Rainbows! with (ThreadSpawn or ThreadPool) is probably the best
way to do streaming responses efficiently from all angles (from
a server, client and programmer time perspective).
---
 examples/nginx.conf |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/examples/nginx.conf b/examples/nginx.conf
index 52ec245..9f245c8 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -125,10 +125,15 @@ http {
       proxy_redirect off;
 
       # set "proxy_buffering off" *only* for Rainbows! when doing
-      # Comet/long-poll stuff.  It's also safe to set if you're
-      # using only serving fast clients with Unicorn + nginx.
-      # Otherwise you _want_ nginx to buffer responses to slow
-      # clients, really.
+      # Comet/long-poll/streaming.  It's also safe to set if you're using
+      # only serving fast clients with Unicorn + nginx, but not slow
+      # clients.  You normally want nginx to buffer responses to slow
+      # clients, even with Rails 3.1 streaming because otherwise a slow
+      # client can become a bottleneck of Unicorn.
+      #
+      # The Rack application may also set "X-Accel-Buffering (yes|no)"
+      # in the response headers do disable/enable buffering on a
+      # per-response basis.
       # proxy_buffering off;
 
       proxy_pass http://app_server;
-- 
Eric Wong

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 10%]

* [PATCH 0/2] attempting to clarify docs for streaming
  2011-04-27 18:13  4% Rails streaming example video Eric Wong
@ 2011-04-27 21:12  4% ` Eric Wong
  2011-04-27 21:12 10%   ` [PATCH 1/2] examples/nginx.conf: clarify proxy_buffering for Rails 3.1 Eric Wong
  2011-04-27 21:12 13%   ` [PATCH 2/2] configurator: attempt to clarify :tcp_nopush/:tcp_nodelay Eric Wong
  0 siblings, 2 replies; 191+ results
From: Eric Wong @ 2011-04-27 21:12 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Aaron Patterson

Since streaming responses is becoming a topic nowadays, I figure we
should attempt to clarify configuration options relevant to it.

In general, I think the defaults for both Unicorn and Rainbows!  in
their targeted environments are reasonable and do not need more tuning.

I've pushed out the following two patches to
git://bogomips.org/unicorn.git:

* [PATCH 1/2] examples/nginx.conf: clarify proxy_buffering for Rails
* [PATCH 2/2] configurator: attempt to clarify

Feedback is greatly appreciated as always.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [PATCH 2/2] configurator: attempt to clarify :tcp_nopush/:tcp_nodelay
  2011-04-27 21:12  4% ` [PATCH 0/2] attempting to clarify docs for streaming Eric Wong
  2011-04-27 21:12 10%   ` [PATCH 1/2] examples/nginx.conf: clarify proxy_buffering for Rails 3.1 Eric Wong
@ 2011-04-27 21:12 13%   ` Eric Wong
  1 sibling, 0 replies; 191+ results
From: Eric Wong @ 2011-04-27 21:12 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Aaron Patterson

These options will probably be more important as interest in
streaming responses in Rails 3.1 develops.

I consider the respective defaults for Unicorn (designed to run
behind nginx) and Rainbows! (designed to run standalone) to be
the best choices in their respective environments.
---
 lib/unicorn/configurator.rb |   22 +++++++++++++++-------
 1 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 73869de..bed3abe 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -253,24 +253,32 @@ class Unicorn::Configurator
   #
   # [:tcp_nodelay => true or false]
   #
-  #   Disables Nagle's algorithm on TCP sockets if +true+
+  #   Disables Nagle's algorithm on TCP sockets if +true+.
+  #
+  #   Setting this to +true+ can make streaming responses in Rails 3.1
+  #   appear more quickly at the cost of slightly higher bandwidth usage.
+  #   The effect of this option is most visible if nginx is not used,
+  #   but nginx remains highly recommended with \Unicorn.
   #
   #   This has no effect on UNIX sockets.
   #
-  #   Default: operating system defaults (usually Nagle's algorithm enabled)
+  #   Default: +false+ (Nagle's algorithm enabled) in \Unicorn,
+  #   +true+ in Rainbows!
   #
   # [:tcp_nopush => true or false]
   #
   #   Enables/disables TCP_CORK in Linux or TCP_NOPUSH in FreeBSD
   #
-  #   This is enabled by default as of Unicorn 3.4.  This prevents partial
-  #   TCP frames from being sent out and reduces wakeups in nginx if it is
-  #   on a different machine.  Since Unicorn is only designed for applications
-  #   that send the response body quickly without keepalive, sockets will
-  #   always be flushed on close to prevent delays.
+  #   This prevents partial TCP frames from being sent out and reduces
+  #   wakeups in nginx if it is on a different machine.  Since \Unicorn
+  #   is only designed for applications that send the response body
+  #   quickly without keepalive, sockets will always be flushed on close
+  #   to prevent delays.
   #
   #   This has no effect on UNIX sockets.
   #
+  #   Default: +true+ in \Unicorn 3.4+, +false+ in Rainbows!
+  #
   # [:tries => Integer]
   #
   #   Times to retry binding a socket if it is already in use
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 13%]

* Rails streaming example video
@ 2011-04-27 18:13  4% Eric Wong
  2011-04-27 21:12  4% ` [PATCH 0/2] attempting to clarify docs for streaming Eric Wong
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-04-27 18:13 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: Aaron Patterson

Aaron Patterson gladly provided us with a streaming video which I've
agreed to host:

http://unicorn.bogomips.org/streaming.ogv

If anybody has more Unicorn/Rainbows!-related videos/slides/etc in Free
Software-friendly formats, I'd be more than glad to host or mirror them.

Thanks Aaron!

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: [ANN] unicorn 3.6.0 - small fixes, PRNG workarounds
  @ 2011-04-26 23:01  7%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-04-26 23:01 UTC (permalink / raw)
  To: unicorn list

ghazel@gmail.com wrote:
> On Wednesday, April 20, 2011, Eric Wong <normalperson@yhbt.net> wrote:
> > Changes:
> >
> > Mainly small fixes, improvements, and workarounds for fork() issues
> > with pseudo-random number generators shipped with Ruby (Kernel#rand,
> > OpenSSL::Random (used by SecureRandom and also by Rails).
> >
> > The PRNG issues are documented in depth here (and links to Ruby Redmine):
> >
> >   http://bogomips.org/unicorn.git/commit?id=1107ede7
> >   http://bogomips.org/unicorn.git/commit?id=b3241621
(top-posting corrected)
> Is it possible there is a problem with this change? Since I upgraded
> to 3.6.0 I have encountered two collisions on
> ActiveSupport::SecureRandom.hex(64), which seems very unlikely, since
> it has never happened in the history of my app otherwise.

Oops, the return value of srand shouldn't be relied on, I need to call
Kernel#rand instead.  My attempt to fix things actually made the problem
worse (which is why I pushed upstream Ruby to fix the problem, first :).
The following should fix it (3.6.1 release coming):

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index d70de45..3077b95 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -492,11 +492,11 @@ class Unicorn::HttpServer
   def after_fork_internal
     @ready_pipe.close if @ready_pipe
     self.ready_pipe = nil # XXX Rainbows! compat, change for Unicorn 4.x
-    tmp = srand # http://redmine.ruby-lang.org/issues/4338
+    srand # http://redmine.ruby-lang.org/issues/4338
 
     # The OpenSSL PRNG is seeded with only the pid, and apps with frequently
     # dying workers can recycle pids
-    OpenSSL::Random.seed(tmp.to_s) if defined?(OpenSSL::Random)
+    OpenSSL::Random.seed(rand.to_s) if defined?(OpenSSL::Random)
   end
 
   def spawn_missing_workers
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply related	[relevance 7%]

* Re: Struggling with logrotate and unicorn
  @ 2011-04-12 18:59  4%     ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-04-12 18:59 UTC (permalink / raw)
  To: unicorn list; +Cc: Emmanuel Gomez

Emmanuel Gomez <emmanuel.gomez@gmail.com> wrote:
> On Apr 12, 2011, at 10:58 AM, Eric Wong wrote:
> > Emmanuel Gomez <emmanuel.gomez@gmail.com> wrote:
> >> I have confirmed that logrotate creates the logs with a 0600 umask
> >> and the uid/gid of my unprivileged user (per my logrotate config,
> >> loosely based on the example logrotate.conf from 3.4 or 3.5). 
> > 
> > Did the permissions of the old (rotated) log files change?
> 
> No, they remained owned by root.

Remained owned by root?  Yes that sounds like the problem.

> >> The problem occurs when I send a USR1 signal to the master process
> >> (as root, because the master is running as root) after the logs
> >> have been rotated. As near as I can tell, after that the Unicorn
> >> master chowns the logs to root ownership. Then, the workers attempt
> >> to chown the logs back to ownership by the unprivileged user, which
> >> repeatedly fails, spewing megabytes of errors that look like:
> > 
> > The rotation error handling should probably just exit! the worker
> > and rely on the master to restart it...
> 
> That would probably be better behavior, although in this specific case
> the worker would immediately die on respawn because the log is still
> owned by root and unwriteable by my unprivileged user.
> 
> I did find a resolution: in the after_fork block, I had copied the
> code to switch users from the GitHub blog post on unicorn
> (https://github.com/blog/517-unicorn). I didn't see
> Unicorn::Worker#user, which implements the same code with the addition
> of a call to Unicorn::Util.chown_logs. When I replace the inline
> GitHub-blog-derived code with a call to Unicorn::Worker#user, it
> works. 

Yes, the old versions of Unicorn didn't change users at all.

I always knew user-switching is a pain in the ass to deal with due to
issues like this.   Due to work on Rainbows! (designed to run on port
80) and users starting as root anyways (due to init scripts), I needed
add this feature.

> My understanding is that this (after_fork/Unicorn::Util.chown_logs)
> shouldn't be executed on USR1; I don't see how the
> Unicorn::Util.chown_logs call on after_fork (startup) would make a
> difference w/r/t rotation (much later), but my understanding is
> obviously incomplete, because it works.

It's the rotation that attempts to chown since it thinks (incorrectly)
that it's in the master process.  I'll make that more robust and release
3.6.0 sometime this week with (hopefully) a few other minor
improvements.

> Thanks for your reply, I'm off to comment on the GitHub blog post to
> try to warn others to use Unicorn::Worker#user instead of the example
> code in after_fork.

Thanks, that seems to be a general problem with people relying on
blog/mailing list posts instead of consistently updated documentation.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: [ANN] raindrops updates (mainly for Linux users)
  2011-03-18 18:09  4% [ANN] raindrops updates (mainly for Linux users) Eric Wong
@ 2011-03-23 18:12  0% ` Troex Nevelin
  0 siblings, 0 replies; 191+ results
From: Troex Nevelin @ 2011-03-23 18:12 UTC (permalink / raw)
  To: unicorn list

On 03/18/2011 09:09 PM, Eric Wong wrote:
> I figured I should cross post here.  raindrops (listen queue stats
> package for Unicorn/Rainbows!) has gotten some largish updates over the
> past few weeks:
>
> raindrops 0.5.0 release announcement:
>    http://mid.gmane.org/20110317033547.GA30653@dcvr.yhbt.net
>
> demo site changes (will probably be in 0.6.0 soon):
>    http://mid.gmane.org/20110318094637.GA3526@dcvr.yhbt.net
>
> Feedback would be greatly appreciated (either here or
> raindrops@librelist.com).  There are no backwards-incompatible changes
> for existing users as far as I'm aware of.

This update is really great! Now we can see 'Mean' users awaiting for 
their requests to be served - this information is very useful, because 
we now can calculate avarage wait time for each request.

For example we know that an average request time is 0.15 sec, than if 
our mean queue is 32 requests, each request will be served in 
(32+1)*0.15=4.95 seconds. This information is very useful and even 
NewRelics cannot provide it.

I'm going to build graphs for this data. You can see live stats for 700k 
request per day project running on 2x Xeon E5410 2.33 Ghz (4 cores each):

http://kinokopilka.tv:9292/queued/%2Fvar%2Fwww%2Fkk%2Fcurrent%2Ftmp%2Fsockets%2Funicorn.sock.html
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* [ANN] raindrops updates (mainly for Linux users)
@ 2011-03-18 18:09  4% Eric Wong
  2011-03-23 18:12  0% ` Troex Nevelin
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-03-18 18:09 UTC (permalink / raw)
  To: mongrel-unicorn, rainbows-talk

I figured I should cross post here.  raindrops (listen queue stats
package for Unicorn/Rainbows!) has gotten some largish updates over the
past few weeks:

raindrops 0.5.0 release announcement:
  http://mid.gmane.org/20110317033547.GA30653@dcvr.yhbt.net

demo site changes (will probably be in 0.6.0 soon):
  http://mid.gmane.org/20110318094637.GA3526@dcvr.yhbt.net

Feedback would be greatly appreciated (either here or
raindrops@librelist.com).  There are no backwards-incompatible changes
for existing users as far as I'm aware of.

Thanks for reading!

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: [RFC/PATCH] Bundler/Sandbox documentation updates
  @ 2011-03-10 21:29  6%       ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-03-10 21:29 UTC (permalink / raw)
  To: unicorn list

Lawrence Pit <lawrence.pit@gmail.com> wrote:
> Eric Wong <normalperson@yhbt.net> wrote:
>> I've tried this a few times, and everytime I get something like this:
>>> https://gist.github.com/331b0decab62fd58483c
>> Yikes, it looks like the UNICORN_FD env is getting clobbered somehow.
>> Not sure what could be causing that.
>>
>>> If I revert back to setting the GEM_HOME, GEM_PATH and PATH in the
>>> before_exec it works fine.

> >  Which Bundler version is this with?
>
> Bundler 1.0.10

Odd, I actually used this in my Rainbows! config file to switch an
Isolate deploy to Bundler (1.0.10) yesterday.   Everything for Rainbows!
applies to Unicorn, too).

  # switching from Isolate to Bundler:
  if ENV["GEM_HOME"] =~ %r{/isolate/}
    ENV.delete "GEM_HOME"
    ENV.delete "GEM_PATH"
    # don't need anything else in $PATH for a web server
    ENV["PATH"] = "/home/ew/ruby-1.9.2/bin"

    # START_CTX is considered a stable interface in Unicorn
    start_ctx = Unicorn::HttpServer::START_CTX
    start_ctx[0] = "bundle"

    # it's even possible to use USR2 to switch between Unicorn and
    # Rainbows! without any downtime :)
    start_ctx[:argv] = %w(exec rainbows).concat(start_ctx[:argv])
  end

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* [PATCH] examples/nginx.conf: use try_files directive
@ 2011-01-25 22:09  8% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-01-25 22:09 UTC (permalink / raw)
  To: mongrel-unicorn

I'm lazy but I finally updated from using nginx 0.6.x myself for
http://unicorn.bogomips.org/ and http://bogomips.org/unicorn.git
to 0.8.54...

>From 09afcf2ce9fc89d77b6b282bbf00a78c73741a4b Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Tue, 25 Jan 2011 13:58:29 -0800
Subject: [PATCH] examples/nginx.conf: use try_files directive

This feature is in nginx 0.7.x and 0.8.x and optimized
better than the "if" directive in nginx.conf

ref: http://wiki.nginx.org/Pitfalls
ref: http://wiki.nginx.org/IfIsEvil
---
 examples/nginx.conf |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/examples/nginx.conf b/examples/nginx.conf
index 70d1851..52ec245 100644
--- a/examples/nginx.conf
+++ b/examples/nginx.conf
@@ -98,7 +98,16 @@ http {
     # path for static files
     root /path/to/app/current/public;
 
-    location / {
+    # Prefer to serve static files directly from nginx to avoid unnecessary
+    # data copies from the application server.
+    #
+    # try_files directive appeared in in nginx 0.7.27 and has stabilized
+    # over time.  Older versions of nginx (e.g. 0.6.x) requires
+    # "if (!-f $request_filename)" which was less efficient:
+    # http://bogomips.org/unicorn.git/tree/examples/nginx.conf?id=v3.3.1#n127
+    try_files $uri/index.html $uri.html $uri @app;
+
+    location @app {
       # an HTTP header important enough to have its own Wikipedia entry:
       #   http://en.wikipedia.org/wiki/X-Forwarded-For
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -122,12 +131,7 @@ http {
       # clients, really.
       # proxy_buffering off;
 
-      # Try to serve static files from nginx, no point in making an
-      # *application* server like Unicorn/Rainbows! serve static files.
-      if (!-f $request_filename) {
-        proxy_pass http://app_server;
-        break;
-      }
+      proxy_pass http://app_server;
     }
 
     # Rails error pages
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 8%]

* Re: Thread.current
  2011-01-13  4:26  6%           ` Thread.current Eric Wong
@ 2011-01-13 16:46  0%             ` Jordan Ritter
  0 siblings, 0 replies; 191+ results
From: Jordan Ritter @ 2011-01-13 16:46 UTC (permalink / raw)
  To: unicorn list

For the record, the "clear the Thread.current storage before/after each request" is what I think is a crappy idiom.  YMMV I guess.

cheers,
--jordan

On Jan 12, 2011, at 8:26 PM, Eric Wong wrote:

> Jimmy Soho <jimmy.soho@gmail.com> wrote:
>> Take for example activesupport's usage of Time.zone. Under water this
>> is set in a thread local var. If you set Time.zone in one request, but
>> not in the next request, using unicorn the next request will use the
>> time zone of the previous request. Using rack or mongrel (in
>> multithreaded mode) you don't have this issue perse (though they have
>> other issues then).
>> 
>> Same for the i18n gem and it's usage of the I18n.locale value, which
>> is also set in a thread local var.
>> 
>> So yeah, unfortunately I have to take into account this "crappy idiom"
>> and need to know exactly which thread local vars are set by all the
>> components we use, and determine which of those must be reset before
>> each request.
> 
> You can probably just write a trivial middleware to clear all
> keys in Thread.current before every request.  Or play around with
> Rainbows! with a single-threaded ThreadSpawn:
> 
> cat >> unicorn.conf.rb <<EOF
> Rainbows! do
>  use :ThreadSpawn
>  worker_connections 1
>  keepalive_timeout 0
> end
> EOF
> 
> And then just run "rainbows" instead of "unicorn".
> 
> -- 
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: Thread.current
  @ 2011-01-13  4:26  6%           ` Eric Wong
  2011-01-13 16:46  0%             ` Thread.current Jordan Ritter
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2011-01-13  4:26 UTC (permalink / raw)
  To: unicorn list

Jimmy Soho <jimmy.soho@gmail.com> wrote:
> Take for example activesupport's usage of Time.zone. Under water this
> is set in a thread local var. If you set Time.zone in one request, but
> not in the next request, using unicorn the next request will use the
> time zone of the previous request. Using rack or mongrel (in
> multithreaded mode) you don't have this issue perse (though they have
> other issues then).
> 
> Same for the i18n gem and it's usage of the I18n.locale value, which
> is also set in a thread local var.
> 
> So yeah, unfortunately I have to take into account this "crappy idiom"
> and need to know exactly which thread local vars are set by all the
> components we use, and determine which of those must be reset before
> each request.

You can probably just write a trivial middleware to clear all
keys in Thread.current before every request.  Or play around with
Rainbows! with a single-threaded ThreadSpawn:

cat >> unicorn.conf.rb <<EOF
Rainbows! do
  use :ThreadSpawn
  worker_connections 1
  keepalive_timeout 0
end
EOF

And then just run "rainbows" instead of "unicorn".

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Re: Thread.current
  @ 2011-01-11 23:12  6%       ` Eric Wong
    1 sibling, 0 replies; 191+ results
From: Eric Wong @ 2011-01-11 23:12 UTC (permalink / raw)
  To: unicorn list

Jimmy Soho <jimmy.soho@gmail.com> wrote:
> Hi,
> 
> Some more questions still:
> 
> It seems a worker uses the exact same thread to handle each request.

Correct.

> Is that guaranteed to happen for the lifetime of a worker? Or are
> there cases where a unicorn worker might spin a new thread to handle
> the next requests?

Unicorn itself is always single-threaded and never spawns new threads.

> If the same thread is always used, isn't that a potential issue when
> programmers use thread local variables, which are not reset at the
> next request?  (I know, the usage of thread local variables is not
> recommended, but take a random rails project, go into their $GEM_HOME
> and do grep -r Thread.current . , see what I mean..)

Thats the problem of those libraries/apps, not Unicorn.

They can try Rainbows! using the :ThreadSpawn option which behaves much
like Mongrel 1, but uses Rainbows! in production that I know of.

The Rack +env+ hash is the safe/universal way to store request-local
variables across different web servers.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 3.3.0 - minor optimizations
@ 2011-01-05 23:57  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2011-01-05 23:57 UTC (permalink / raw)
  To: mongrel-unicorn

Certain applications that already serve hundreds/thousands of requests a
second (per-worker) should experience performance improvements due to
Time.now.httpdate usage being removed and reimplemented in C.

There are also minor internal changes and cleanups for Rainbows!

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 3.2.1 - parser improvements for Rainbows!
@ 2010-12-26  8:29 13% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-12-26  8:29 UTC (permalink / raw)
  To: mongrel-unicorn

There are numerous improvements in the HTTP parser for
Rainbows!, none of which affect Unicorn-only users.

The kgio dependency is incremented to 2.1: this should avoid
ENOSYS errors for folks building binaries on newer Linux
kernels and then deploying to older ones.

There are also minor documentation improvements, the website
is now JavaScript-free!

(Ignore the 3.2.0 release, I fat-fingered some packaging things)

Expect a new Rainbows! release in the next day or two with
the improved HTTP parser and Cool.io support.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 13%]

* Re: [ANN] kgio library / RubyGem
  2010-09-28  3:03  7% [ANN] kgio library / RubyGem Eric Wong
  2010-11-15 18:41  0% ` Eric Wong
@ 2010-12-22 11:24  0% ` Iñaki Baz Castillo
  1 sibling, 0 replies; 191+ results
From: Iñaki Baz Castillo @ 2010-12-22 11:24 UTC (permalink / raw)
  To: unicorn list; +Cc: rainbows-talk

2010/9/28 Eric Wong <normalperson@yhbt.net>:
> I've released kgio, a kinder, gentler I/O library for Ruby.  Some of its
> features are useful for Unicorn, and all of it is useful for Rainbows!
>
> I intend to make kgio a requirement for both Unicorn and
> Rainbows!/Zbatery.  I'm comfortable with the code, but extra testers and
> extra eyes to review it would be helpful (it's nearly all C).

Very interesting project :)

A typo in the doc:

---------------------
kgio_wait_readable()
Blocks the running Thread indefinitely until self IO object is writable
---------------------

Should be "readable", right? :)


-- 
Iñaki Baz Castillo
<ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* [ANN] unicorn 3.0.1 - one bugfix for Rainbows!
@ 2010-12-03  1:29 14% Eric Wong
  2010-12-03  1:30 14% ` [ANN] Rainbows! 2.0.1 - upload pipelining fixes Eric Wong
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-12-03  1:29 UTC (permalink / raw)
  To: mongrel-unicorn; +Cc: rainbows-talk

...and only Rainbows!  This release fixes HTTP pipelining for
requests with bodies for users of synchronous Rainbows!
concurrency models.

Since Unicorn itself does not support keepalive nor pipelining,
Unicorn-only users need not upgrade.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 14%]

* [ANN] Rainbows! 2.0.1 - upload pipelining fixes
  2010-12-03  1:29 14% [ANN] unicorn 3.0.1 - one bugfix for Rainbows! Eric Wong
@ 2010-12-03  1:30 14% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-12-03  1:30 UTC (permalink / raw)
  To: rainbows-talk; +Cc: mongrel-unicorn

For HTTP clients living on the edge and pipelining uploads, we
now fully support pipelined requests (as long as the application
consumes each request in its entirety).

Rainbows! is an HTTP server for sleepy Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.

* http://rainbows.rubyforge.org/
* rainbows-talk@rubyforge.org
* git://git.bogomips.org/rainbows.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 14%]

* Re: Dedicated queues for long-running requests with single unicorn master: question and a possible solution
  @ 2010-11-26  0:40  6% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-11-26  0:40 UTC (permalink / raw)
  To: unicorn list

Alexander Dymo <adymo@pluron.com> wrote:
> Hey,
> I have two major types of requests for my app:
> - long-running (10 sec and more, I can differentiate them by url)
> - normal (less than 1 sec)
<snip>
> Here's the possible solution I came up with and it seems to work.
> What do you think about it? Does it have problems I didn't think of?
> Are there better ways to do the same thing?

No chance of speeding up the long requests?  That's the best option.

It may make more sense to other people (non-Rubyists) who could inherit
the system to manage the Unicorn master instances separately and use
separate config files.  Otherwise, I suppose your approach is valid
(haven't tried nor seen it myself).

If the long request is blocking on something external (e.g. making an
HTTP request to an OpenID provider), then Rainbows![1] is worth looking
into.

[1] http://rainbows.rubyforge.org/
    The second section here is probably applicable to you:
    http://rainbows.rubyforge.org/DEPLOY.html

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Re: [ANN] unicorn 3.0.0 - disable rewindable input!
  2010-11-20  2:47  4% [ANN] unicorn 3.0.0 " Eric Wong
@ 2010-11-20 17:50  0% ` Michael Guterl
  0 siblings, 0 replies; 191+ results
From: Michael Guterl @ 2010-11-20 17:50 UTC (permalink / raw)
  To: unicorn list

On Fri, Nov 19, 2010 at 9:47 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Changes:
>
> Rewindable "rack.input" may be disabled via the
> "rewindable_input false" directive in the configuration file.
> This will violate Rack::Lint for Rack 1.x applications, but can
> reduce I/O for applications that do not need a rewindable
> input.
>
> This release updates us to the Kgio 2.x series which should play
> more nicely with other libraries and applications.  There are
> also internal cleanups and improvements for future versions of
> Rainbows!
>
> The Unicorn 3.x series supercedes the 2.x series
> while the 1.x series will remain supported indefinitely.
>
We've been using Unicorn 1.x very successfully for some time with a
Rails 2.3 application.  Would you recommend upgrading to Unicorn 3.x?

Best,
Michael Guterl
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* [ANN] unicorn 3.0.0 - disable rewindable input!
@ 2010-11-20  2:47  4% Eric Wong
  2010-11-20 17:50  0% ` Michael Guterl
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-11-20  2:47 UTC (permalink / raw)
  To: mongrel-unicorn

Changes:

Rewindable "rack.input" may be disabled via the
"rewindable_input false" directive in the configuration file.
This will violate Rack::Lint for Rack 1.x applications, but can
reduce I/O for applications that do not need a rewindable
input.

This release updates us to the Kgio 2.x series which should play
more nicely with other libraries and applications.  There are
also internal cleanups and improvements for future versions of
Rainbows!

The Unicorn 3.x series supercedes the 2.x series
while the 1.x series will remain supported indefinitely.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 3.0.0pre1 - disable rewindable input!
@ 2010-11-17  0:26  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-11-17  0:26 UTC (permalink / raw)
  To: mongrel-unicorn

Rewindable "rack.input" may be disabled via the
"rewindable_input false" directive in the configuration file.
This will violate Rack::Lint for Rack 1.x applications, but
can reduce I/O for applications that do not need it.

There are also internal cleanups and enhancements for future
versions of Rainbows!

Eric Wong (11):
      t0012: fix race condition in reload
      enable HTTP keepalive support for all methods
      http_parser: add HttpParser#next? method
      tee_input: switch to simpler API for parsing trailers
      switch versions to 3.0.0pre
      add stream_input class and build tee_input on it
      configurator: enable "rewindable_input" directive
      http_parser: ensure keepalive is disabled when reset
      *_input: make life easier for subclasses/modules
      tee_input: restore read position after #size
      preread_input: no-op for non-rewindable "rack.input"

See "git log" output for all the gory details.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: [ANN] kgio library / RubyGem
  2010-09-28  3:03  7% [ANN] kgio library / RubyGem Eric Wong
@ 2010-11-15 18:41  0% ` Eric Wong
  2010-12-22 11:24  0% ` Iñaki Baz Castillo
  1 sibling, 0 replies; 191+ results
From: Eric Wong @ 2010-11-15 18:41 UTC (permalink / raw)
  To: mongrel-unicorn, rainbows-talk

Eric Wong <normalperson@yhbt.net> wrote:
> Hello all,
> 
> I've released kgio, a kinder, gentler I/O library for Ruby.  Some of its
> features are useful for Unicorn, and all of it is useful for Rainbows!

OK, it's got a separate mailing list at kgio@librelist.com, now.

Apparently it's garnered some attention and dalli[1] is one of the first
projects to start using it.

Homepage and git repository locations remain the same:

	http://unicorn.bogomips.org/kgio/
	git://git.bogomips.org/kgio.git
	http://git.bogomips.org/cgit/kgio.git

[1] - git://github.com/mperham/dalli

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* [ANN] Unicorn 2.0.0, 1.1.5, 1.0.2 released!
@ 2010-10-28  0:19  5% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-10-28  0:19 UTC (permalink / raw)
  To: mongrel-unicorn

tldr:
2.0.0 - cleanups for Rainbows!, but should be ready for general use
        barring portability issues
1.1.5 - bug fixes
1.0.2 - bug fixes

Please send any and all feedback to the mailing list.  Do not waste
bandwidth with HTML or signature attachments.

email: mongrel-unicorn@rubyforge.org
git: git://git.bogomips.org/unicorn.git

The longer change summaries/release notes:

== unicorn 2.0.0 - mostly internal cleanups

Despite the version number, this release mostly features
internal cleanups for future versions of Rainbows!.  User
visible changes include reductions in CPU wakeups on idle sites
using high timeouts.

Barring possible portability issues due to the introduction of
the kgio library, this release should be ready for all to use.
However, 1.1.x (and possibly 1.0.x) will continue to be
maintained.  Unicorn 1.1.5 and 1.0.2 have also been released
with bugfixes found during development of 2.0.0.

== unicorn 1.1.5

This maintenance release fixes several long-standing but
recently-noticed bugs.  SIGHUP reloading now correctly restores
default values if they're erased or commented-out in the Unicorn
configuration file.  Delays/slowdowns in signal handling since
0.990 are fixed, too.

== unicorn 1.0.2

This is the latest maintenance release of the 1.0.x series.
All users are encouraged to upgrade to 1.1.x stable series
and report bugs there.

Shortlog of changes since 1.0.1:

Eric Wong (8):
      SIGTTIN works after SIGWINCH
      fix delays in signal handling
      Rakefile: don't post freshmeat on empty changelogs
      Rakefile: capture prerelease tags
      configurator: use "__send__" instead of "send"
      configurator: reloading with unset values restores default
      gemspec: depend on Isolate 3.0.0 for dev
      doc: stop using deprecated rdoc CLI options

==
Have fun!

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 5%]

* Re: 502 bad gateway on nginx with recv() failed
  2010-10-23 23:22  0%     ` Eric Wong
@ 2010-10-24  6:00  0%       ` Naresh V
  0 siblings, 0 replies; 191+ results
From: Naresh V @ 2010-10-24  6:00 UTC (permalink / raw)
  To: unicorn list

On 24 October 2010 04:52, Eric Wong <normalperson@yhbt.net> wrote:
> Naresh V <nareshov@gmail.com> wrote:
>> On 23 October 2010 02:44, Eric Wong <normalperson@yhbt.net> wrote:
>> > Naresh V <nareshov@gmail.com> wrote:
>> >> I'm serving the puppetmaster application with its config.ru through
>> >> unicorn - proxied by nginx.
>> >> I'm using unix sockets, 4 workers, and 2048 backlog.
>> >>
>> >> The clients - after their typical "puppet run" - send back a report to
>> >> the master in YAML.
>> >> Some clients whose reports tend to be large (close to 2mb) get a 502
>> >> bad gateway error and error out.
>> >>
>> >> nginx log:
>> >>
>> >> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
>> >> Connection reset by peer) while reading response header from upstream,
>> >> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
>> >> HTTP/1.1", upstream:
>> >> "http://unix:/tmp/.sock:/production/report/nagios", host:
>> >> "puppet:8140"
>> >
>> > Hi Naresh, do you see anything in the Unicorn stderr log file?
>>
>> Hi Eric, I think I caught it:
>>
>> E, [2010-10-22T23:03:30.207455 #10184] ERROR -- : worker=2 PID:10206
>> timeout (60.207392s > 60s), killing
>> I, [2010-10-22T23:03:31.212533 #10184]  INFO -- : reaped
>> #<Process::Status: pid=10206,signaled(SIGKILL=9)> worker=2
>> I, [2010-10-22T23:03:31.214768 #10490]  INFO -- : worker=2 spawned pid=10490
>> I, [2010-10-22T23:03:31.221748 #10490]  INFO -- : worker=2 ready
>>
>> > Is the 2mb report part of the response or request?  Unicorn should
>> > have no problems accepting large requests (Rainbows! defaults the
>> > client_max_body_size to 1mb, just like nginx).
>>
>> It's part of the PUT request, I guess.
>>
>> > It could be Unicorn's internal (default 60s) timeout kicking
>> > in because puppet is slowly reading/generating the 2mb body.
>>
>> I raised the timeout first to 120, then 180 - and I continued to get
>> the 502 (with the logs as above)
>> When I raised it upto 240, puppetd complained:
>
> Interesting.  I'm not familiar with Puppet internals, but is there any
> valid reason it would be taking this long?
>
> Can you tell if the Unicorn worker is doing anything (using up CPU time
> in top) or just blocked on some socket connection to a database or DNS?
> (strace/truss will help).
>
> You should definitely talk to Puppet developers/users about why it's
> taking so long.  HTTP requests taking anywhere near 60s is an eternity,
> I wonder if your Puppet is somehow misconfigured.
>

Yes, puppet-server was mis-configured and went unnoticed. The reports
are supposed to be accepted by another URL and that had to be
configured appropriately.

Sorry for the false alarm.

-Naresh.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: 502 bad gateway on nginx with recv() failed
  2010-10-23  4:48  0%   ` Naresh V
@ 2010-10-23 23:22  0%     ` Eric Wong
  2010-10-24  6:00  0%       ` Naresh V
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-10-23 23:22 UTC (permalink / raw)
  To: unicorn list

Naresh V <nareshov@gmail.com> wrote:
> On 23 October 2010 02:44, Eric Wong <normalperson@yhbt.net> wrote:
> > Naresh V <nareshov@gmail.com> wrote:
> >> I'm serving the puppetmaster application with its config.ru through
> >> unicorn - proxied by nginx.
> >> I'm using unix sockets, 4 workers, and 2048 backlog.
> >>
> >> The clients - after their typical "puppet run" - send back a report to
> >> the master in YAML.
> >> Some clients whose reports tend to be large (close to 2mb) get a 502
> >> bad gateway error and error out.
> >>
> >> nginx log:
> >>
> >> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
> >> Connection reset by peer) while reading response header from upstream,
> >> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
> >> HTTP/1.1", upstream:
> >> "http://unix:/tmp/.sock:/production/report/nagios", host:
> >> "puppet:8140"
> >
> > Hi Naresh, do you see anything in the Unicorn stderr log file?
> 
> Hi Eric, I think I caught it:
> 
> E, [2010-10-22T23:03:30.207455 #10184] ERROR -- : worker=2 PID:10206
> timeout (60.207392s > 60s), killing
> I, [2010-10-22T23:03:31.212533 #10184]  INFO -- : reaped
> #<Process::Status: pid=10206,signaled(SIGKILL=9)> worker=2
> I, [2010-10-22T23:03:31.214768 #10490]  INFO -- : worker=2 spawned pid=10490
> I, [2010-10-22T23:03:31.221748 #10490]  INFO -- : worker=2 ready
> 
> > Is the 2mb report part of the response or request?  Unicorn should
> > have no problems accepting large requests (Rainbows! defaults the
> > client_max_body_size to 1mb, just like nginx).
> 
> It's part of the PUT request, I guess.
> 
> > It could be Unicorn's internal (default 60s) timeout kicking
> > in because puppet is slowly reading/generating the 2mb body.
> 
> I raised the timeout first to 120, then 180 - and I continued to get
> the 502 (with the logs as above)
> When I raised it upto 240, puppetd complained:

Interesting.  I'm not familiar with Puppet internals, but is there any
valid reason it would be taking this long?

Can you tell if the Unicorn worker is doing anything (using up CPU time
in top) or just blocked on some socket connection to a database or DNS?
(strace/truss will help).

You should definitely talk to Puppet developers/users about why it's
taking so long.  HTTP requests taking anywhere near 60s is an eternity,
I wonder if your Puppet is somehow misconfigured.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: 502 bad gateway on nginx with recv() failed
  2010-10-22 21:14  4% ` Eric Wong
@ 2010-10-23  4:48  0%   ` Naresh V
  2010-10-23 23:22  0%     ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Naresh V @ 2010-10-23  4:48 UTC (permalink / raw)
  To: unicorn list

On 23 October 2010 02:44, Eric Wong <normalperson@yhbt.net> wrote:
> Naresh V <nareshov@gmail.com> wrote:
>> Hi,
>>
>> I'm serving the puppetmaster application with its config.ru through
>> unicorn - proxied by nginx.
>> I'm using unix sockets, 4 workers, and 2048 backlog.
>>
>> The clients - after their typical "puppet run" - send back a report to
>> the master in YAML.
>> Some clients whose reports tend to be large (close to 2mb) get a 502
>> bad gateway error and error out.
>>
>> nginx log:
>>
>> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
>> Connection reset by peer) while reading response header from upstream,
>> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
>> HTTP/1.1", upstream:
>> "http://unix:/tmp/.sock:/production/report/nagios", host:
>> "puppet:8140"
>
> Hi Naresh, do you see anything in the Unicorn stderr log file?

Hi Eric, I think I caught it:

E, [2010-10-22T23:03:30.207455 #10184] ERROR -- : worker=2 PID:10206
timeout (60.207392s > 60s), killing
I, [2010-10-22T23:03:31.212533 #10184]  INFO -- : reaped
#<Process::Status: pid=10206,signaled(SIGKILL=9)> worker=2
I, [2010-10-22T23:03:31.214768 #10490]  INFO -- : worker=2 spawned pid=10490
I, [2010-10-22T23:03:31.221748 #10490]  INFO -- : worker=2 ready


> Is the 2mb report part of the response or request?  Unicorn should
> have no problems accepting large requests (Rainbows! defaults the
> client_max_body_size to 1mb, just like nginx).

It's part of the PUT request, I guess.

> It could be Unicorn's internal (default 60s) timeout kicking
> in because puppet is slowly reading/generating the 2mb body.

I raised the timeout first to 120, then 180 - and I continued to get
the 502 (with the logs as above)
When I raised it upto 240, puppetd complained:

#-(1)> puppetd -t -v  --trace
notice: Ignoring --listen on onetime run
info: Caching catalog for nagios
info: Applying configuration version '1287807847'
notice: Finished catalog run in 25.86 seconds
/usr/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:56:in `timeout'
/usr/lib/ruby/1.8/timeout.rb:76:in `timeout'
/usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
/usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
/usr/lib/ruby/1.8/net/http.rb:1050:in `request'
/usr/lib/ruby/1.8/net/http.rb:1037:in `request'
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
/usr/lib/ruby/1.8/net/http.rb:1035:in `request'
/usr/lib/ruby/1.8/net/http.rb:857:in `put'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:90:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:253:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector.rb:64:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:178:in `send_report'
/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:172:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:21:in `lock'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/usr/lib/ruby/1.8/sync.rb:229:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:103:in `with_client'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:37:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:171:in `call'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:171:in `controlled_run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:35:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:114:in `onetime'
/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:88:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:287:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:393:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:287:in `run'
/usr/sbin/puppetd:4
err: Could not run Puppet configuration client: execution expired


> Also, which version of Unicorn and nginx is this?

unicorn (1.1.4) and nginx-0.8.49-1.el5

>> I was getting the same thing earlier when I had unicorn listening on
>> TCP sockets instead of UNIX sockets. And I had a lot of connections in
>> TIME_WAIT:
>>
>> tcp        0      0 127.0.0.1:8141              127.0.0.1:54507
>>      TIME_WAIT   -
>> tcp        0      0 127.0.0.1:8141              127.0.0.1:57322
>>      TIME_WAIT   -
>>
>> Fluctuating all the way from 20 to 800. A quick restart of nginx
>> tended to bring the number down.
>
> Having many TIME_WAIT sockets is normal and expected when you're
> starting/stopping lots of TCP connections.  It's nothing to worry about
> unless you get several thousands of requests/second, and then you should
> apply tcp_tw_reuse/tcp_tw_recycle as mentioned in
> http://unicorn.bogomips.org/TUNING.html (or switch to UNIX domain
> sockets and use nginx for keepalive).
>
> --
> Eric Wong


-Naresh V.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: 502 bad gateway on nginx with recv() failed
  @ 2010-10-22 21:14  4% ` Eric Wong
  2010-10-23  4:48  0%   ` Naresh V
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-10-22 21:14 UTC (permalink / raw)
  To: unicorn list

Naresh V <nareshov@gmail.com> wrote:
> Hi,
> 
> I'm serving the puppetmaster application with its config.ru through
> unicorn - proxied by nginx.
> I'm using unix sockets, 4 workers, and 2048 backlog.
> 
> The clients - after their typical "puppet run" - send back a report to
> the master in YAML.
> Some clients whose reports tend to be large (close to 2mb) get a 502
> bad gateway error and error out.
> 
> nginx log:
> 
> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
> Connection reset by peer) while reading response header from upstream,
> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
> HTTP/1.1", upstream:
> "http://unix:/tmp/.sock:/production/report/nagios", host:
> "puppet:8140"

Hi Naresh, do you see anything in the Unicorn stderr log file?

Is the 2mb report part of the response or request?  Unicorn should
have no problems accepting large requests (Rainbows! defaults the
client_max_body_size to 1mb, just like nginx).

It could be Unicorn's internal (default 60s) timeout kicking
in because puppet is slowly reading/generating the 2mb body.

Also, which version of Unicorn and nginx is this?

> I was getting the same thing earlier when I had unicorn listening on
> TCP sockets instead of UNIX sockets. And I had a lot of connections in
> TIME_WAIT:
> 
> tcp        0      0 127.0.0.1:8141              127.0.0.1:54507
>      TIME_WAIT   -
> tcp        0      0 127.0.0.1:8141              127.0.0.1:57322
>      TIME_WAIT   -
> 
> Fluctuating all the way from 20 to 800. A quick restart of nginx
> tended to bring the number down.

Having many TIME_WAIT sockets is normal and expected when you're
starting/stopping lots of TCP connections.  It's nothing to worry about
unless you get several thousands of requests/second, and then you should
apply tcp_tw_reuse/tcp_tw_recycle as mentioned in
http://unicorn.bogomips.org/TUNING.html (or switch to UNIX domain
sockets and use nginx for keepalive).

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* internal API changes in Unicorn 2.x
@ 2010-10-08 21:37  5% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-10-08 21:37 UTC (permalink / raw)
  To: mongrel-unicorn

Here's a non-exhaustive list of internal API changes for Unicorn 2.x.

This will NOT affect most users, only those who use/rely on the
undocumented API which we've never recommended people to use.

This mainly affects Rainbows!  development.  Nearly all of these changes
are to make life easier for Rainbows! which Unicorn 2.x is intended to
support.

* Util.tmpio -> TmpIO.new

* HttpParser and HttpRequest become the same class
  - not finalized, yet, one name will go away, probably in 3.x
  - the Rack "env" hash and "buf" String buffer belong
    to the C struct since they're always used together
  - the outward API should be mostly compatible with 1.1.x
  - PARSER, BUF, REQ constants removed from HttpRequest,
    the new HttpParser holds all these on a per-socket basis
  - HttpParser#parse method added to work on the env/buf
    elements

* TeeInput.new takes only two parameters, socket and the
  parser/request object

* HttpRequest::LOCALHOST constant removed, use Kgio::LOCALHOST

* RACK_INPUT and REMOTE_ADDR constants moved from Const to HttpRequest
  since they're unused outside of HttpRequest


If there are other apps/libraries who depend on some things we changed
and you really can't have it any other way, we _may_ try to accomodate
you if it's not too much effort.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 5%]

* [ANN] unicorn 2.0.0pre1 - a boring "major" release
@ 2010-10-06  1:53  6% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-10-06  1:53 UTC (permalink / raw)
  To: mongrel-unicorn, rainbows-talk

Changes:

Mostly internal cleanups for future versions of Rainbows! and
people trying out Rubinius.  There are tiny performance
improvements for Ruby 1.9.2 users which may only be noticeable
with Rainbows!.

There is a new dependency on the "kgio" library for kinder,
gentler I/O :)  Please report any bugs and portability issues
with kgio to the Unicorn mailing list[1].

Unicorn 1.1.x users are NOT required nor even encouraged to
upgrade yet.  Unicorn 1.1.x will be maintained for the
forseeable future.

[1] - mongrel-unicorn@rubyforge.org


I'm holding off on supporting massively multicore machines with
a few hundred processors until a later release.  Expect
Rainbows! to see some changes/improvements in the days to come,
too.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Re: Unicorn Signal Handling Shared for Other Servers
  @ 2010-10-04  5:41  3% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-10-04  5:41 UTC (permalink / raw)
  To: unicorn list; +Cc: Ben Curren

Ben Curren <ben@outright.com> wrote:
> I was curious if anyone has attempted to, or is interested, it ripping
> out Unicorn's signal handling and daemon code to make it possible to
> share with other Ruby Servers. I really love have 0 down time deploys
> and would love to have this in workling, thrift or other custom
> servers we have written. I'm more than willing to fork the code and
> take stab at it but wanted to check with the group to see if something
> has already been attempted. I'm not the type who likes to reinvent the
> wheel.

Hi Ben,

I recall some attempts to make libraries around what Unicorn did a year
ago when Unicorn started picking up more press.  I'm not sure how far
any of them got and can't remember the names, but October/November 2009
was the time frame.

Personally, I think the core Ruby methods around Unix APIs are
"just right" and extra wrappers either:

  a) take away needed flexibility
     (no, I don't want umask 0000 with my daemonize())

  b) require as much effort to learn as the Unix APIs
     (Process.daemon() in 1.9, really?)

The Unix APIs themselves are well-documented in manpages, textbooks,
and needless to say, useful outside of Ruby[1].

Ruby already does a great job of hiding the tedious, more painful parts
of the original C APIs from you.  Things like trap(signal, &block),
IO.select, and exceptions (vs writing error check for *everything*)
already make life much easier than before.


For Unicorn (and Rainbows!) development, one huge (maybe the biggest)
factor in trusting it is having integration tests that
start/stop/restart/thrash/nuke the server in various ways a user would.
The test suite hits signal handling logic, directory changes, process
creation/reaping, log file rotation, pid files, etc.  Some parts of the
test suite are ugly (test_exec.rb), but nevertheless effective at
finding bugs.


[1] - I'll admit to being reasonably experienced with Unix programming
      and worked on several daemons before I picked up Ruby.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 3%]

* [ANN] unicorn 1.1.3 - small fixes
@ 2010-08-28 19:57  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-08-28 19:57 UTC (permalink / raw)
  To: mongrel-unicorn

Changes:

This release fixes race conditions during SIGUSR1 log cycling.
This bug mainly affects Rainbows! users serving static files,
but some Rack apps use threads internally even under Unicorn.

Other small fixes:
* SIGTTIN works as documented after SIGWINCH
* --help output from `unicorn` and `unicorn_rails` is more consistent


Unless people scream, there will be no further releases from the 1.0.x
line and all users are encouraged to upgrade to 1.1.x.  I'm still
thinking about how to approach some internal cleanups for 2.x, but 2.x
could be out as early as next week  (but 1.x will continue to be
maintained).

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: running unicorn under rvm
  2010-08-17  3:52  4% ` Eric Wong
  2010-08-17 22:37  0%   ` Joseph McDonald
@ 2010-08-18  1:22  0%   ` Joseph McDonald
  1 sibling, 0 replies; 191+ results
From: Joseph McDonald @ 2010-08-18  1:22 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list

I added
TOPLEVEL_BINDING = binding
to gems/unicorn-1.1.2/bin/unicorn
and it seemed to fix it:

#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
# -*- encoding: binary -*-
require 'unicorn/launcher'
require 'optparse'
# !!!!!!!!!  added the line below:
TOPLEVEL_BINDING = binding
ENV["RACK_ENV"] ||= "development"

I still have no idea why I need to do that.

thanks,
-joe


On Mon, Aug 16, 2010 at 8:52 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Joseph McDonald <superjoe@gmail.com> wrote:
>> unicorn 1.1.2 runs fine under my system ruby, but when trying to run
>> it using rvm, I get:
>>
>>
>> % rvm use 1.8.7
>> info: Using ruby 1.8.7 p299
>>
>> % unicorn
>> /home/joe/.rvm/gems/ruby-1.8.7-p299/gems/unicorn-1.1.2/lib/unicorn/configurator.rb:494:in
>> `eval': undefined local variable or method `host' for main:Object
>> (NameError)
>
> Hi Joseph,
>
> Is there any chance you have an older "unicorn" executable script
> somewhere that gets exposed rvm switches the environment on you?  That's
> the only thing I can think of right now.
>
>> line 494 looks like:
>>
>>  # XXX ugly as hell, WILL FIX in 2.x (along with Rainbows!/Zbatery)
>
> Yes, I still need to work on 2.x and clean up a lot of the
> ugly internal API bits :)
>
>
> --
> Eric Wong
>
> P.S.: At least nowadays I'm (finally) getting some experience
> developing/supporting applications using Unicorn and Rainbows! (or
> Zbatery).  Things like e4d0b226391948ef433f1d0135814315e4c48535 in
> unicorn.git are a direct result of that.
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: running unicorn under rvm
  2010-08-17  3:52  4% ` Eric Wong
@ 2010-08-17 22:37  0%   ` Joseph McDonald
  2010-08-18  1:22  0%   ` Joseph McDonald
  1 sibling, 0 replies; 191+ results
From: Joseph McDonald @ 2010-08-17 22:37 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list

Thanks Eric, see below:

On Mon, Aug 16, 2010 at 8:52 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Joseph McDonald <superjoe@gmail.com> wrote:
>> unicorn 1.1.2 runs fine under my system ruby, but when trying to run
>> it using rvm, I get:
>>
>>
>> % rvm use 1.8.7
>> info: Using ruby 1.8.7 p299
>>
>> % unicorn
>> /home/joe/.rvm/gems/ruby-1.8.7-p299/gems/unicorn-1.1.2/lib/unicorn/configurator.rb:494:in
>> `eval': undefined local variable or method `host' for main:Object
>> (NameError)
>
> Hi Joseph,
>
> Is there any chance you have an older "unicorn" executable script
> somewhere that gets exposed rvm switches the environment on you?  That's
> the only thing I can think of right now.

I'm sure I'm running the right version of unicorn.
what's weird is if I remove the reference to TOPLEVEL_BINDING, it works, i.e.:

 host, port, set_listener, options, daemonize =
                    eval("[ host, port, set_listener, options, daemonize ]")

works, it gets the right port from the config.ru file.

my config.ru file looks like"
#\ -w -p 4452
require './mystuff'
run Sinatra::Application

another strange thing:  I need to add the "./"  in front of mystuff,
but when i'm not using rvm (rvm use system), I don't need to do that.

I'm curious if anyone else is using unicorn with rvm and not having
this problem.  I'm guessing it's something weird with my system, but
not sure.

thanks,
-joe





>
>> line 494 looks like:
>>
>>  # XXX ugly as hell, WILL FIX in 2.x (along with Rainbows!/Zbatery)
>
> Yes, I still need to work on 2.x and clean up a lot of the
> ugly internal API bits :)
>
>
> --
> Eric Wong
>
> P.S.: At least nowadays I'm (finally) getting some experience
> developing/supporting applications using Unicorn and Rainbows! (or
> Zbatery).  Things like e4d0b226391948ef433f1d0135814315e4c48535 in
> unicorn.git are a direct result of that.
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: running unicorn under rvm
  2010-08-17  1:29  4% running unicorn under rvm Joseph McDonald
@ 2010-08-17  3:52  4% ` Eric Wong
  2010-08-17 22:37  0%   ` Joseph McDonald
  2010-08-18  1:22  0%   ` Joseph McDonald
  0 siblings, 2 replies; 191+ results
From: Eric Wong @ 2010-08-17  3:52 UTC (permalink / raw)
  To: Joseph McDonald; +Cc: unicorn list

Joseph McDonald <superjoe@gmail.com> wrote:
> unicorn 1.1.2 runs fine under my system ruby, but when trying to run
> it using rvm, I get:
> 
> 
> % rvm use 1.8.7
> info: Using ruby 1.8.7 p299
> 
> % unicorn
> /home/joe/.rvm/gems/ruby-1.8.7-p299/gems/unicorn-1.1.2/lib/unicorn/configurator.rb:494:in
> `eval': undefined local variable or method `host' for main:Object
> (NameError)

Hi Joseph,

Is there any chance you have an older "unicorn" executable script
somewhere that gets exposed rvm switches the environment on you?  That's
the only thing I can think of right now.

> line 494 looks like:
> 
>  # XXX ugly as hell, WILL FIX in 2.x (along with Rainbows!/Zbatery)

Yes, I still need to work on 2.x and clean up a lot of the
ugly internal API bits :)


-- 
Eric Wong

P.S.: At least nowadays I'm (finally) getting some experience
developing/supporting applications using Unicorn and Rainbows! (or
Zbatery).  Things like e4d0b226391948ef433f1d0135814315e4c48535 in
unicorn.git are a direct result of that.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* running unicorn under rvm
@ 2010-08-17  1:29  4% Joseph McDonald
  2010-08-17  3:52  4% ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Joseph McDonald @ 2010-08-17  1:29 UTC (permalink / raw)
  To: mongrel-unicorn

unicorn 1.1.2 runs fine under my system ruby, but when trying to run
it using rvm, I get:


% rvm use 1.8.7
info: Using ruby 1.8.7 p299

% unicorn
/home/joe/.rvm/gems/ruby-1.8.7-p299/gems/unicorn-1.1.2/lib/unicorn/configurator.rb:494:in
`eval': undefined local variable or method `host' for main:Object
(NameError)

line 494 looks like:

 # XXX ugly as hell, WILL FIX in 2.x (along with Rainbows!/Zbatery)
    host, port, set_listener, options, daemonize =
                    eval("[ host, port, set_listener, options, daemonize ]",
                         TOPLEVEL_BINDING)

any ideas why it would have a problem when running under rvm?

thanks,
-joe
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: unicorn behind apache with file uploads
  @ 2010-08-15 22:56  6% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-08-15 22:56 UTC (permalink / raw)
  To: unicorn list

Surendra Singhi <singhi.surendra@gmail.com> wrote:
> Hi,
> 
> For one of our apps we switched to unicorn from mongrel.
> The unicorn server use to run behind an apache with timeout of 300sec,
> as sometime the customer may do a large file upload.
> 
> After we made the switch we started getting lots of
> Unicorn::ClientShutdown occurred in photos#flash_upload
> bytes_read=2762663 (pr some other number)
> We were not able to recreate it in our tests, but support was getting
> customer emails, that the files upload is not working.
> It finally forced us to switch back to mongrel.

Hi Surendra,

Could you be hitting a timeout in your Apache config?  Can you try
simulating a slow connection to reproduce it?  Apache defaults to 300s,
too, so double check your config and make sure it's not different
anymore.   The other thing could be your kernel listen queue used by
Unicorn is completely full and Apache is just giving up on certain
requests...

> I know a fully buffering reverse proxy like nginx is advised as
> reverse proxy for unicorn, but apache has been serving us well, and we
> don't want to change it.
> Any idea what can be going wrong, or why was unicorn throwing these errors?

While it's generally bad for Unicorn to run behind anything other than
nginx, "generally bad" does not apply to the case of file uploads.

Honestly, when dealing with file uploads from slow clients,
Unicorn is the WORST SERVER YOU COULD EVER PICK.  EVER :)

Really, trust me on this one :)


Use nginx, but if you insist on using Apache, then you're far better off
sticking with Mongrel or trying out Rainbows![1] for uploads.

Rainbows! has the advantage of using Unicorn's prefork and shared
listeners along with your choice of threads/fibers/events/fibers/actors.

[1] - http://rainbows.rubyforge.org/

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* [ANN] 1.9 users: socket_dontwait - MSG_DONTWAIT socket methods
@ 2010-08-09 22:54  6% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-08-09 22:54 UTC (permalink / raw)
  To: mongrel-unicorn

Hi all,

I initially announced[1] this on the Rainbows! mailing list since
Rainbows! is more experimental in nature, but I've been running this for
a few days with real traffic (with Rainbows!) and it seems safe enough
for less crazy people to use :)

If you're using Ruby 1.9.1, then this library includes a fix for
errno getting zeroed and hitting rb_bug() during rb_sys_fail()
This is fixed in Ruby trunk r27401, but 1.9.1 doesn't have this.

This library is a drop-in replacement that reimplements several IO
methods with replacements using MSG_DONTWAIT for BasicSocket.  This
allows us to avoid unnecessary system calls and GVL bouncing.

[1] - http://mid.gmane.org/20100803091847.GC3255@dcvr.yhbt.net

The rest of the README below:

We've reimplemented the +readpartial+, +read_nonblock+,
+write_nonblock+, +read+ and +write+ instance methods normally inherited
from the IO class directly into BasicSocket with socket-specific system
calls and flags.

This library is only intended for Ruby 1.9 and will not build with other
versions of Ruby.  This only supports operating systems with the
non-POSIX MSG_DONTWAIT flag for send(2) and recv(2) syscalls.

This library is considered EXPERIMENTAL.  If successful, we'll see about
getting them integrated into the standard Ruby socket library.

== Features

* Avoid use of fcntl(2) to set O_NONBLOCK in favor of MSG_DONTWAIT when
  using non-blocking I/O.  We _unset_ O_NONBLOCK if we need to block
  and release the GVL instead of relying on select(2).

* Avoids select(2) entirely in favor of blocking I/O when the
  GVL is released.  This allows using file descriptor numbers higher
  than 1023 without overflowing select(2) buffers.

* BasicSocket#read uses recv(2) with MSG_WAITALL to avoid extra system
  calls for larger reads.

* Thread and signal-safe, releases the GVL for all blocking operations
  and retries if system calls are interrupted.

* Includes a 1.9.1-specific workaround to preserve errno after reacquiring
  the GVL.  This is
  {fixed}[http://redmine.ruby-lang.org/repositories/diff/ruby-19?rev=27401]
  in newer versions of Ruby.

* Falls back to line-buffered IO if needed (not recommended).

== Bugs/Caveats

* We ignore taint/$SAFE checks, we'll support it if there's demand,
  but we doubt there is...

* Does not support 1.9 encoding filters.  1.9 defaults all sockets to
  Encoding::BINARY anyways, so this should not be noticeable to code
  that leaves socket encodings untouched.

* Does not support write buffering in userspace.  Ruby defaults all
  sockets to "IO#sync = true", anyways so this does not affect code
  that leaves the default setting untouched.

* Avoid using line-buffered IO on sockets (IO#gets, IO#each_line),
  nearly all of the features of this library are cancelled out when
  the line-buffering fallback is used.

== Install

If you're using a packaged Ruby distribution, make sure you have a C
compiler and the matching Ruby development libraries and headers.
You need Ruby 1.9 to install socket_dontwait.  Previous versions of
Ruby will NOT be supported.

If you use RubyGems:

    gem install socket_dontwait

Otherwise grab the latest tarball from:

http://bogomips.org/socket_dontwait/files/

Unpack it, and run "ruby setup.rb"

== Development

You can get the latest source via git from the following locations:

  git://git.bogomips.org/socket_dontwait.git
  git://repo.or.cz/socket_dontwait.git (mirror)

You may browse the code from the web and download the latest snapshot
tarballs here:

* http://git.bogomips.org/cgit/socket_dontwait.git (cgit)
* http://repo.or.cz/w/socket_dontwait.git (gitweb)

Inline patches (from "git format-patch") to the mailing list are
preferred because they allow code review and comments in the reply to
the patch.

We will adhere to mostly the same conventions for patch submissions as
git itself.  See the Documentation/SubmittingPatches document
distributed with git on on patch submission guidelines to follow.  Just
don't email the git mailing list or maintainer with socket_dontwait
patches.

== Contact/Bug Reports/Feedback/Patches/Pull-Requests

This was originally created for the Rainbows! project (but may be used by
others), so we'll reuse their mailing list at
{rainbows-talk@rubyforge.org}[mailto:rainbows-talk@rubyforge.org].

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 1.1.0 - small changes and cleanups
@ 2010-07-08  8:04  4% Eric Wong
       [not found]     ` <20100708080457.GA2345-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-07-08  8:04 UTC (permalink / raw)
  To: unicorn list

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

This is a small, incremental feature release with some internal
changes to better support upcoming versions of the Rainbows! and
Zbatery web servers.  There is no need to upgrade if you're
happy with 1.0.0, but also little danger in upgrading.

There is one pedantic bugfix which shouldn't affect anyone
and small documentation updates as well.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: Purpose of "Status" header in HTTP responses?
  @ 2010-06-23  9:07  6% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-06-23  9:07 UTC (permalink / raw)
  To: unicorn list; +Cc: Craig Davey

Craig Davey <me@craigdavey.ca> wrote:
> Hi folks
> 
> On line #63 of unicorn/http_response.rb a "Status" header is written
> to the socket. A comment in the code explains that some broken clients
> require this header and unicorn generously accommodates them.
> 
> We’re having the opposite problem. One of our clients using Microsoft
> Windows and ASP haven’t been able to connect to our HTTP API since we
> moved it to unicorn from passenger. They receive the following error
> message when they try to connect to our servers:
> 
> msxml3.dll error '80072f78' server returned an invalid or unrecognized
> response

Hi Craig,

Interesting and strange...

Looking at lib/phusion_passenger/rack/request_handler.rb (blob ad22dfa)
line 94, they also set the Status: header, too (but just the numeric
code, no text).

You can try "proxy_hide_header Status;" in your nginx config
to suppress it.


Another theory: You are running nginx in front of Unicorn, right?

If not (but you really should be), the lack of a Server header may throw
off some clients...

I also don't ever want folks to be forced to reveal they use which
server they use for security concerns, so Unicorn won't ever force the
Server: header on you.  And since nginx overwrites any Server header
Unicorn would set, Unicorn won't bother, either.  However, it's easy to
setup Rack middleware to write anything you want in the Server header.

rainbows.git (unreleased) allows using the Rainbows::ServerToken
middleware, and if you really need it, it should be easy to port to
Unicorn:

  http://git.bogomips.org/cgit/rainbows.git/tree/lib/rainbows/server_token.rb

> Our client thinks this error is caused by the "Status" header that is
> added to responses by unicorn. We don’t know of any other instances
> where this header is causing problems so we’re pretty confused about
> why it’s a problem for them.

Passenger also adds X-Powered-By, but that's completely non-standard and
probably used to get around proxies (like nginx) that overwrite the
standard Server: header.  You can also make middleware (or your app) add
that header, too, and even go as far to make Unicorn pretend to be
Passenger :>

> Does anyone remember why this "Status" header was added to
> HttpResponse? Which broken clients was the change trying to
> accommodate?

I seem to recall some JavaScript libraries relied on it at some point,
and possibly some versions of Firebug.  Maybe some browser plugins do,
too.  Some folks here with more experience on client-side stuff ought
to chime in, since I generally stay away from GUI/DOM things.

However, even with my lack of JS experience (or because of) I realize
it's very easy to fall into the trap of writing JavaScript that relies
on the Status: header.  The Status: header has been with us as a
de-facto standard since the CGI days.  Older cgi.rb-based versions of
Rails set it, too.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* Re: [ANN] unicorn 1.0.0 - yes, this is a real project
  2010-06-17 10:08  5% [ANN] unicorn 1.0.0 - yes, this is a real project Eric Wong
@ 2010-06-18  1:47  0% ` Augusto Becciu
  0 siblings, 0 replies; 191+ results
From: Augusto Becciu @ 2010-06-18  1:47 UTC (permalink / raw)
  To: unicorn list

Wow, awesome. Huge congrats Eric!

On Thu, Jun 17, 2010 at 7:08 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Unicorn is an HTTP server for Rack applications designed to only serve
> fast clients on low-latency, high-bandwidth connections and take
> advantage of features in Unix/Unix-like kernels.  Slow clients should
> only be served by placing a reverse proxy capable of fully buffering
> both the the request and response in between Unicorn and slow clients.
>
> * http://unicorn.bogomips.org/
> * mongrel-unicorn@rubyforge.org
> * git://git.bogomips.org/unicorn.git
>
> Changes:
>
> There are only minor changes since 0.991.0.
>
> For users clinging onto the past, MRI 1.8.6 support has been
> restored.  Users are strongly encouraged to upgrade to the
> latest 1.8.7, REE or 1.9.1.
>
> For users looking towards the future, the core test suite and
> the Rails 3 (beta) integration tests pass entirely under 1.9.2
> preview3.  As of the latest rubinius.git[1], Rubinius support is
> nearly complete as well.
>
> Under Rubinius, signals may corrupt responses as they're being
> written to the socket, but that should be fixable transparently
> to us[4].  Support for the hardly used, hardly documented[2]
> embedded command-line switches in rackup config (.ru) files is
> is also broken under Rubinius.
>
> The recently-released Rack 1.2.1 introduced no compatiblity
> issues[3] in core Unicorn.  We remain compatible with all Rack
> releases starting with 0.9.1 (and possibly before).
>
> [1] tested with Rubinius upstream commit
> cf4a5a759234faa3f7d8a92d68fa89d8c5048f72
> [2] lets avoid the Dueling Banjos effect here :x
> [3] actually, Rack 1.2.1 is broken under 1.8.6.
> [4] http://github.com/evanphx/rubinius/issues/373
>
>
> The Future:
>
> * Bug/compatibility fixes as needed, of course!
> * Scalability for hardware that may be common in 5-10 years
> * Rainbows!  LOTS of Rainbows!
>
> Thanks for reading!
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* [ANN] unicorn 1.0.0 - yes, this is a real project
@ 2010-06-17 10:08  5% Eric Wong
  2010-06-18  1:47  0% ` Augusto Becciu
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-06-17 10:08 UTC (permalink / raw)
  To: ruby-talk ML, mongrel-unicorn

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

There are only minor changes since 0.991.0.

For users clinging onto the past, MRI 1.8.6 support has been
restored.  Users are strongly encouraged to upgrade to the
latest 1.8.7, REE or 1.9.1.

For users looking towards the future, the core test suite and
the Rails 3 (beta) integration tests pass entirely under 1.9.2
preview3.  As of the latest rubinius.git[1], Rubinius support is
nearly complete as well.

Under Rubinius, signals may corrupt responses as they're being
written to the socket, but that should be fixable transparently
to us[4].  Support for the hardly used, hardly documented[2]
embedded command-line switches in rackup config (.ru) files is
is also broken under Rubinius.

The recently-released Rack 1.2.1 introduced no compatiblity
issues[3] in core Unicorn.  We remain compatible with all Rack
releases starting with 0.9.1 (and possibly before).

[1] tested with Rubinius upstream commit
cf4a5a759234faa3f7d8a92d68fa89d8c5048f72
[2] lets avoid the Dueling Banjos effect here :x
[3] actually, Rack 1.2.1 is broken under 1.8.6.
[4] http://github.com/evanphx/rubinius/issues/373


The Future:

* Bug/compatibility fixes as needed, of course!
* Scalability for hardware that may be common in 5-10 years
* Rainbows!  LOTS of Rainbows!

Thanks for reading!

-- 
Eric Wong



^ permalink raw reply	[relevance 5%]

* Re: anything left before 1.0?
  @ 2010-06-16  2:27  3%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-06-16  2:27 UTC (permalink / raw)
  To: unicorn list

Andrew Grim <andrew@kongregate.com> wrote:
> > Let us know if there's anything else missing, pipe up within the next 24
> > hours or so...
> >
> Hey Eric,
> 
> I was hoping to spend some more time debugging myself, but since you
> were going to release 1.0 I thought I'd get your thoughts on this.
> Quick overview, I work for kongregate.com, one of the larger rails
> sites, and one of the only large sites (that I know of personally)
> running ruby 1.9.  We are currently running mostly mongrels, but I've
> got one server testing Unicorn.  Things are mostly great, we are
> seeing nearly 20ms improvement in average response time, which is
> awesome.  Now to the issue.
> 
> SYMPTOM: The master process is killing the workers fairly frequently
> based on the workers timing out.
> 
> CAUSE: I've added some logging to get the backtrace when I send a
> SIGTERM, and it is always stuck on line 68 in http_response.rb:
> 
>       body.each { |chunk| socket.write(chunk) }
> 
> I ran some straces, and here's an example of the last few lines where
> it gets killed:
> 
> 06:50:23.239931 clock_gettime(CLOCK_REALTIME, {1276523423, 239967000})
> = 0 <0.000172>
> 06:50:23.240213 write(12, "HTTP/1.1 200 OK\r\nDate: Mon, 14 J"...,
> 1896) = 1896 <0.000087>
> 06:50:23.242072 write(12, "<!DOCTYPE html PUBLIC \"-//W3C//D"...,
> 166842) = 128000 <0.000107>
> 06:50:23.242230 select(13, NULL, [12], NULL, NULL <unfinished ...>
> 06:51:22.167122 +++ killed by SIGKILL +++
> 
> So it's writing and then (to my understanding) waiting on the socket
> to return, but you can see that for a full 60s it isn't.

Hi Andrew,

The timer starts when the app is initially dispatched, not when writing
starts.  You can check the log output from the master process (which
usually goes to stderr_path) and see the exact time the master process
saw before killing it.

Are you using nginx (or something else) to reverse proxy?  You should
be using nginx :)

> My best guess off-hand is that the large size of the string being
> written to the socket is causing an issue, and I have noticed that it
> is happening primarily on requests that return larger payloads.

That's unlikely.  I suspect the client you're using to hit Unicorn with
is not reading the other end of the socket, so once the kernel buffers
fill up, Unicorn blocks on the write.

Not a real solution, but you can probably hide the problem by increasing
the buffer sizes in the Linux kernel (net.core.wmem_max and
net.ipv4.tcp_wmem sysctls), but the defaults are already very generous.

> At the same time, it isn't that much data, so I'm a little surprised
> it would be an issue.  I am planning on trying to split the body up
> into smaller chunks in a rack middleware or something.

I doubt the middleware would help at all.

> Or I could be totally off.  Just wanted to see if you have any ideas,
> I'm not even sure this is a Unicorn issue, definitely could be ruby
> 1.9 bug too.

I would definitely look at your _client_ (which should be nginx).

You should isolate your client from other requests and strace that, too,
and see if it's reading off the socket at the same time.  I've used
1.9.1 pretty heavily with Rainbows! and large responses myself in
development/testing.

nginx will freeze up badly when running poorly-written Perl code with
the embedded Perl support.  Other than that it's been very solid in my
experience.

> Sorry about the long email, but I appreciate any help you can give.

No problem, let us know what you find out.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 3%]

* Re: Forking off the unicorn master process to create a background worker
  2010-06-15 22:51  0%       ` Russell Branca
@ 2010-06-16  0:06  0%         ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-06-16  0:06 UTC (permalink / raw)
  To: Russell Branca; +Cc: unicorn list

Russell Branca <chewbranca@gmail.com> wrote:
> On Tue, Jun 15, 2010 at 3:14 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Russell Branca <chewbranca@gmail.com> wrote:
> >> Hello Eric,
> >>
> >> Sorry for the delayed response, with the combination of being sick and
> >> heading out of town for a while, this project got put on the
> >> backburner. I really appreciate your response and think its a clean
> >> solution for what I'm trying to do. I've started back in getting the
> >> job queue working this week, and will hopefully have a working
> >> solution in the next day or two. A little more information about what
> >> I'm doing, I'm trying to create a centralized resque job queue server
> >> that each of the different applications can queue work into, so I'll
> >> be using redis behind resque for storing jobs and what not, which
> >> brings me an area I'm not sure of the best approach on. So when we hit
> >> the job queue endpoint in the rack app, it spawns the new worker, and
> >> then immediately returns the 200 ok started background job message,
> >> which cuts off communication back to the job queue. My plan is to save
> >> a status message of the result of the background task into redis, and
> >> have resque check that to verify the task was successful. Is there a
> >> better approach for returning the resulting status code with unicorn,
> >> or is this a reasonable approach? Thanks again for your help.
> >
> > Hi Russell, please don't top post, thanks.
> >
> > If you already have a queue server (and presumably a standalone app
> > processing the queue), I would probably forgo the background Unicorn
> > worker entirely.
> >
> > Based on my ancient (mid-2000s) knowledge of user-facing web
> > applications: the application should queue the job, return 200, and have
> > HTML meta refresh to constantly reload the page every few seconds.
> >
> > Hitting the reload endpoint would check the database (Redis in this
> > case) for completion, and return a new HTML page to stop the meta
> > refresh loop.
> >
> > This means you're no longer keeping a single Unicorn worker idle and
> > wasting it.  Nowadays you could do it with long-polling on
> > Rainbows!/Thin/Zbatery, too, but long-polling is less reliable for
> > people switching between WiFi access points.  The meta refresh method
> > can be a waste of power/bandwidth on the client side if the background
> > job takes a long time, though.
> >
> > I'm familiar at all with Resque or Redis, but I suspect other folks
> > on this mailing list should be able to help you flesh out the details.
> 
> Hi Eric,
> 
> I have a queue server, but I don't have a standalone app processing
> the jobs, because I have a large number of stand alone applications on
> a single server. Right now I've got 12 separate apps running, so if I
> wanted to have a standalone app for each, that would be 12 additional
> applications in memory for handling background jobs. The whole reason
> I want to go with the unicorn worker approach for handling background
> jobs, is so I can fork off the master process as needed, avoid the
> spawning time for a normal rails instance, and only use workers as
> needed. This way I can have just a few workers running at any given
> time, rather than 1 worker for each app. The number of apps is only
> going to increase, but I want to keep the worker pool a constant. I'll
> probably just update status of completion with redis, these jobs won't
> be run by users, this is all background stuff like sending
> notifications, data analysis, feed parsing, etc etc, so I'm planning
> on just having resque initiate a request directly, and then use
> unicorn to process the task in the background.

Ah, so I guess it's a single queue server but multiple queues?  I
guess thats where I got confused with your description.

> I didn't exactly follow what you meant when you were talking about a
> unicorn worker being idle, from the example config.ru you responded
> with earlier on, it looks like I can just spawn a new worker that will
> be outside of the normal worker pool to handle the job. I'm pretty
> sure this will work, I was curious about the best approach for
> returning completion status, but I think just having the worker record
> its status and exit is better than having long polling connections
> open between the job queue and the new unicorn worker.

Yes, having the fork as I made in the example should work.  I haven't
tested it, of course :)  My instincts tell me recording the status and
exiting ASAP is better because it uses less memory.

You should test and experiment with it either way.  You know your apps,
requirements, and Redis/Resque far better than I do :)  Consider
software an evolutionary process, so whatever the "best approach" may
be, another one can usurp it eventually or be completely wrong in a
slightly different setting :)

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: Forking off the unicorn master process to create a background  worker
  2010-06-15 22:14  4%     ` Eric Wong
@ 2010-06-15 22:51  0%       ` Russell Branca
  2010-06-16  0:06  0%         ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Russell Branca @ 2010-06-15 22:51 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn list

On Tue, Jun 15, 2010 at 3:14 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Russell Branca <chewbranca@gmail.com> wrote:
>> Hello Eric,
>>
>> Sorry for the delayed response, with the combination of being sick and
>> heading out of town for a while, this project got put on the
>> backburner. I really appreciate your response and think its a clean
>> solution for what I'm trying to do. I've started back in getting the
>> job queue working this week, and will hopefully have a working
>> solution in the next day or two. A little more information about what
>> I'm doing, I'm trying to create a centralized resque job queue server
>> that each of the different applications can queue work into, so I'll
>> be using redis behind resque for storing jobs and what not, which
>> brings me an area I'm not sure of the best approach on. So when we hit
>> the job queue endpoint in the rack app, it spawns the new worker, and
>> then immediately returns the 200 ok started background job message,
>> which cuts off communication back to the job queue. My plan is to save
>> a status message of the result of the background task into redis, and
>> have resque check that to verify the task was successful. Is there a
>> better approach for returning the resulting status code with unicorn,
>> or is this a reasonable approach? Thanks again for your help.
>
> Hi Russell, please don't top post, thanks.
>
> If you already have a queue server (and presumably a standalone app
> processing the queue), I would probably forgo the background Unicorn
> worker entirely.
>
> Based on my ancient (mid-2000s) knowledge of user-facing web
> applications: the application should queue the job, return 200, and have
> HTML meta refresh to constantly reload the page every few seconds.
>
> Hitting the reload endpoint would check the database (Redis in this
> case) for completion, and return a new HTML page to stop the meta
> refresh loop.
>
> This means you're no longer keeping a single Unicorn worker idle and
> wasting it.  Nowadays you could do it with long-polling on
> Rainbows!/Thin/Zbatery, too, but long-polling is less reliable for
> people switching between WiFi access points.  The meta refresh method
> can be a waste of power/bandwidth on the client side if the background
> job takes a long time, though.
>
> I'm familiar at all with Resque or Redis, but I suspect other folks
> on this mailing list should be able to help you flesh out the details.
>
> --
> Eric Wong
>

Hi Eric,

I have a queue server, but I don't have a standalone app processing
the jobs, because I have a large number of stand alone applications on
a single server. Right now I've got 12 separate apps running, so if I
wanted to have a standalone app for each, that would be 12 additional
applications in memory for handling background jobs. The whole reason
I want to go with the unicorn worker approach for handling background
jobs, is so I can fork off the master process as needed, avoid the
spawning time for a normal rails instance, and only use workers as
needed. This way I can have just a few workers running at any given
time, rather than 1 worker for each app. The number of apps is only
going to increase, but I want to keep the worker pool a constant. I'll
probably just update status of completion with redis, these jobs won't
be run by users, this is all background stuff like sending
notifications, data analysis, feed parsing, etc etc, so I'm planning
on just having resque initiate a request directly, and then use
unicorn to process the task in the background.

I didn't exactly follow what you meant when you were talking about a
unicorn worker being idle, from the example config.ru you responded
with earlier on, it looks like I can just spawn a new worker that will
be outside of the normal worker pool to handle the job. I'm pretty
sure this will work, I was curious about the best approach for
returning completion status, but I think just having the worker record
its status and exit is better than having long polling connections
open between the job queue and the new unicorn worker.


-Russell
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: Forking off the unicorn master process to create a background worker
  @ 2010-06-15 22:14  4%     ` Eric Wong
  2010-06-15 22:51  0%       ` Russell Branca
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-06-15 22:14 UTC (permalink / raw)
  To: Russell Branca; +Cc: unicorn list

Russell Branca <chewbranca@gmail.com> wrote:
> Hello Eric,
> 
> Sorry for the delayed response, with the combination of being sick and
> heading out of town for a while, this project got put on the
> backburner. I really appreciate your response and think its a clean
> solution for what I'm trying to do. I've started back in getting the
> job queue working this week, and will hopefully have a working
> solution in the next day or two. A little more information about what
> I'm doing, I'm trying to create a centralized resque job queue server
> that each of the different applications can queue work into, so I'll
> be using redis behind resque for storing jobs and what not, which
> brings me an area I'm not sure of the best approach on. So when we hit
> the job queue endpoint in the rack app, it spawns the new worker, and
> then immediately returns the 200 ok started background job message,
> which cuts off communication back to the job queue. My plan is to save
> a status message of the result of the background task into redis, and
> have resque check that to verify the task was successful. Is there a
> better approach for returning the resulting status code with unicorn,
> or is this a reasonable approach? Thanks again for your help.

Hi Russell, please don't top post, thanks.

If you already have a queue server (and presumably a standalone app
processing the queue), I would probably forgo the background Unicorn
worker entirely.

Based on my ancient (mid-2000s) knowledge of user-facing web
applications: the application should queue the job, return 200, and have
HTML meta refresh to constantly reload the page every few seconds.

Hitting the reload endpoint would check the database (Redis in this
case) for completion, and return a new HTML page to stop the meta
refresh loop.

This means you're no longer keeping a single Unicorn worker idle and
wasting it.  Nowadays you could do it with long-polling on
Rainbows!/Thin/Zbatery, too, but long-polling is less reliable for
people switching between WiFi access points.  The meta refresh method
can be a waste of power/bandwidth on the client side if the background
job takes a long time, though.

I'm familiar at all with Resque or Redis, but I suspect other folks
on this mailing list should be able to help you flesh out the details.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: Unicorn future plans
  2010-06-14 20:58  0% ` John-Paul Bader
@ 2010-06-14 22:10  3%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-06-14 22:10 UTC (permalink / raw)
  To: unicorn list

John-Paul Bader <hukl@berlin.ccc.de> wrote:
> Hey, 
> 
> your plans sound great except for ignoring FreeBSD ;)
> 
> Seriously many many Servers, including many which I administrate run
> on FreeBSD and they do so very well. I'm not very familiar with the
> internals of the FreeBSD Kernel but I can assure you that it runs
> perfectly on 8 and 16 core machines and I would extremely happy if I
> could continue to use unicorn on them.

Hi John-Paul,

Reread my post carefully, not much is changing for 8-16 core users.
FreeBSD and SMP will continue to be supported.  I wasn't referring
to SMP at all when I was talking about Linux-only bits.

> It should be easier to maintain a FreeBSD version as they (FreeBSD
> developers) tend to aim for more consistency across releases than
> Linux. You said once that you don't run any BSD machines but I'd be
> happy to offer you access to one of my BSD servers for testing.

Thanks for the offer.  I'll keep it in mind if I need it again.

> Furthermore since Mac OS X and FreeBSD share many features like kqueue
> and many Ruby developers are running macs it would make even more
> sense to at least care about BSD even if its not you primary platform. 

I know, I've fixed some things on OSX platforms via FreeBSD (OSX scares
me with its GUI-ness).  Keep in mind kqueue (and epoll) are worthless
for Unicorn itself since Unicorn only handles one client at a time.

However, Rainbows! can already use kqueue/epoll with EventMachine and
Rev.

> I don't want to start a flame war or say that one OS is better than
> another. Linux is certainly great but so is FreeBSD. For the SMP part
> I recommend the following page on freebsd.org:

Again, I wasn't talking about SMP at all.   SMP works fine on current
Unicorn and mid-sized servers.  When going to more cores, SMP itself
is a bottleneck, not the kernel.

With Unicorn 2.x, I'm targeting NUMA hardware that isn't available in
commodity servers yet.  Currently, NUMA makes _zero_ sense in web
application servers, but in case things change in a few years, we'll be
ready.

It may be a chicken-and-egg problem, too.  Hardware manufacturers are
slow to commoditize NUMA because a good chunk of software can't even
utilize SMP effectively :)

> To conclude I can only say that I'm running unicorn on several FreeBSD
> hosts and so far I couldn't be happier with it. As stated before, I'd
> be more than happy to continue using that setup.

Again, don't worry :)  You'll be able to continue using everything
as-is.

> Kindest regards and thanks for all you efforts,

No problem.  Please don't top post in the future.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 3%]

* Re: Unicorn future plans
  2010-06-14 19:46  6% Unicorn future plans Eric Wong
@ 2010-06-14 20:58  0% ` John-Paul Bader
  2010-06-14 22:10  3%   ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: John-Paul Bader @ 2010-06-14 20:58 UTC (permalink / raw)
  To: unicorn list

Hey, 

your plans sound great except for ignoring FreeBSD ;)

Seriously many many Servers, including many which I administrate run on FreeBSD and they do so very well. I'm not very familiar with the internals of the FreeBSD Kernel but I can assure you that it runs perfectly on 8 and 16 core machines and I would extremely happy if I could continue to use unicorn on them.

It should be easier to maintain a FreeBSD version as they (FreeBSD developers) tend to aim for more consistency across releases than Linux. You said once that you don't run any BSD machines but I'd be happy to offer you access to one of my BSD servers for testing.

Furthermore since Mac OS X and FreeBSD share many features like kqueue and many Ruby developers are running macs it would make even more sense to at least care about BSD even if its not you primary platform. 

I don't want to start a flame war or say that one OS is better than another. Linux is certainly great but so is FreeBSD. For the SMP part I recommend the following page on freebsd.org:

http://www.freebsd.org/smp/

And maybe specifically:

http://www.freebsd.org/smp/#resources

To conclude I can only say that I'm running unicorn on several FreeBSD hosts and so far I couldn't be happier with it. As stated before, I'd be more than happy to continue using that setup.

Kindest regards and thanks for all you efforts,

John


On 14.06.2010, at 21:46, Eric Wong wrote:

> Hi all,
> 
> Some of you are wondering about the future of the project, especially
> since we're nearing a 1.0 release.
> 
> == 1.x - bugfixes and Rack-compatibility synchronization
> 
> The 1.x series will focus mainly on bug fixes and compatibility with
> Rack as it evolves.
> 
> If Rack drops the rewindability requirement of "rack.input", Unicorn
> will follow ASAP and allow TeeInput to be optional middleware with
> newer Rack versions.
> 
> Rubinius should be fully-supported soon, as it's already mostly working
> except for a few corner-case things Rubinius doesn't implement (issues
> filed on their bug tracker).
> 
> == 2.x - the fun and crazy stuff
> 
> First off, there'll be internal API cleanups + sync with
> Rainbows!/Zbatery This won't be user visible, but it'll be less ugly
> inside.
> 
> === Scalability improvements
> 
> I don't know if we'll see hundreds/thousands of CPUs in a single
> application server any time soon, but 2.x will have internal changes
> that'll make us more ready for that.  It could be 5-10 years before
> massively multi-core machines are viable for web apps, but it'd
> certainly be fun to max out Unicorn on those.
> 
> I'll be less shy about sacrificing portability for massive scalability.
> Correct me if I'm wrong, but Linux is the only Free kernel that scales
> to monster multicore machines right now, so I'll primarily focus on
> working with features in Linux.
> 
> Currently, 8-16 cores seems to be the sweet spot (and has been for a
> while), and present-day Unicorn handles that fine as-is.
> 
> === Features (for Rainbows!, mainly)
> 
> IPv6 support and SSL will come, too.   These features will mainly be to
> support Rainbows!, though some LANs could benefit from those, too.  I'll
> have to review the MRI code for those, but I'm leaning towards only
> these new features under 1.9.2+.
> 
> Multi-VM (MVM, Ruby 1.9 experimental branch) support will probably
> happen only in Rainbows! rather than Unicorn.  A true fork() is still
> safer in the event of a crash (but less memory-efficient).
> 
> == Miscellaneous
> 
> I have no plans to provide commercial support, but I'll continue
> offering free support on the mailing list (or private email if you're
> shy).
> 
> As some of you may have noticed; I don't endorse commercial products or
> services at all.  This won't change.  However, there will probably be
> more commercial support available from 3rd parties after a 1.0 release.
> 
> 
> Thanks for reading!
> 
> -- 
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
> 

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Unicorn future plans
@ 2010-06-14 19:46  6% Eric Wong
  2010-06-14 20:58  0% ` John-Paul Bader
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-06-14 19:46 UTC (permalink / raw)
  To: mongrel-unicorn

Hi all,

Some of you are wondering about the future of the project, especially
since we're nearing a 1.0 release.

== 1.x - bugfixes and Rack-compatibility synchronization

The 1.x series will focus mainly on bug fixes and compatibility with
Rack as it evolves.

If Rack drops the rewindability requirement of "rack.input", Unicorn
will follow ASAP and allow TeeInput to be optional middleware with
newer Rack versions.

Rubinius should be fully-supported soon, as it's already mostly working
except for a few corner-case things Rubinius doesn't implement (issues
filed on their bug tracker).

== 2.x - the fun and crazy stuff

First off, there'll be internal API cleanups + sync with
Rainbows!/Zbatery This won't be user visible, but it'll be less ugly
inside.

=== Scalability improvements

I don't know if we'll see hundreds/thousands of CPUs in a single
application server any time soon, but 2.x will have internal changes
that'll make us more ready for that.  It could be 5-10 years before
massively multi-core machines are viable for web apps, but it'd
certainly be fun to max out Unicorn on those.

I'll be less shy about sacrificing portability for massive scalability.
Correct me if I'm wrong, but Linux is the only Free kernel that scales
to monster multicore machines right now, so I'll primarily focus on
working with features in Linux.

Currently, 8-16 cores seems to be the sweet spot (and has been for a
while), and present-day Unicorn handles that fine as-is.

=== Features (for Rainbows!, mainly)

IPv6 support and SSL will come, too.   These features will mainly be to
support Rainbows!, though some LANs could benefit from those, too.  I'll
have to review the MRI code for those, but I'm leaning towards only
these new features under 1.9.2+.

Multi-VM (MVM, Ruby 1.9 experimental branch) support will probably
happen only in Rainbows! rather than Unicorn.  A true fork() is still
safer in the event of a crash (but less memory-efficient).

== Miscellaneous

I have no plans to provide commercial support, but I'll continue
offering free support on the mailing list (or private email if you're
shy).

As some of you may have noticed; I don't endorse commercial products or
services at all.  This won't change.  However, there will probably be
more commercial support available from 3rd parties after a 1.0 release.


Thanks for reading!

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Re: [ANN] unicorn 0.990.0 - inching towards 1.0
  @ 2010-06-11 20:37  4% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-06-11 20:37 UTC (permalink / raw)
  To: mongrel-unicorn

Eric Wong <normalperson@yhbt.net> wrote:
> Rubinius updates:
>   but things should generally work unless you
>   need USR2 upgrades.  Feedback and reports would be greatly
>   appreciated as usual.

Log rotation for $stderr and $stdout (but not other
paths) are broken under Rubinius, too.  I only discovered this
while running the Rainbows! integration tests.

ref: http://github.com/evanphx/rubinius/issues/360

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: Working directory and config.ru
  @ 2010-06-10  9:58  8%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-06-10  9:58 UTC (permalink / raw)
  To: unicorn list

Eric Wong <normalperson@yhbt.net> wrote:
> Pierre Baillet <oct@fotonauts.com> wrote:
> > Hi,
> > 
> > I naively expected unicorn to honor the working_directory
> > configuration directive before attempting to locate the config.ru but
> > this does not seem to be the case. Is this behavior wanted ? From the
> > code, I can guess that the configuration file parsing is done much
> > later  than the current config.ru location attempt.
> 
> Hi Pierre,
> 
> Oops, definitely not wanted behavior.  Unicorn should honor
> working_directory with regard to config.ru.  I'll start working
> on a fix shortly, thanks for the report!

I just pushed the following out with a few other cleanups.

Maintaining Rainbows!/Zbatery compatibility was a bit costly in terms of
prettiness.

Overall, my initial hesitation to support "working_directory" last year
was validated by the complexity of maintaining relative path
compatibility across the board.

I'm very glad for the integration test suite (stolen from Rainbows!)
which allows me to write tests in my "native" language :>

There'll be more tests coming when I'm more awake (and I'm actually
developing a real Rack application which will be public Real Soon
Now(TM), but more targeted at Rainbows!/Zbatery :)

>From 4accf4449390c649bce0b1c9d84314d65fd41f8e Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Thu, 10 Jun 2010 08:47:10 +0000
Subject: [PATCH] respect "working_directory" wrt config.ru

Since we added support for the "working_directory" parameter, it
often became unclear where/when certain paths would be bound.

There are some extremely nasty dependencies and ordering issues
when doing this.  It's all pretty fragile, but works for now
and we even have a full integration test to keep it working.

I plan on cleaning this up 2.x.x to be less offensive to look
at (Rainbows! and Zbatery are a bit tied to this at the moment).

Thanks to Pierre Baillet for reporting this.

ref: http://mid.gmane.org/AANLkTimKb7JARr_69nfVrJLvMZH3Gvs1o_KwZFLKfuxy@mail.gmail.com
---
 bin/unicorn                  |    5 +---
 bin/unicorn_rails            |   17 +++++++++---
 lib/unicorn.rb               |    7 ++---
 lib/unicorn/configurator.rb  |   56 ++++++++++++++++++++++++++++++++++++++++++
 lib/unicorn/launcher.rb      |    5 ++-
 t/t0003-working_directory.sh |   53 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 129 insertions(+), 14 deletions(-)
 create mode 100755 t/t0003-working_directory.sh

diff --git a/bin/unicorn b/bin/unicorn
index beece97..2cc10b1 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -107,10 +107,7 @@ opts = OptionParser.new("", 24, '  ') do |opts|
   opts.parse! ARGV
 end
 
-ru = ARGV[0] || "config.ru"
-abort "configuration file #{ru} not found" unless File.exist?(ru)
-
-app = Unicorn.builder(ru, opts)
+app = Unicorn.builder(ARGV[0] || 'config.ru', opts)
 options[:listeners] << "#{host}:#{port}" if set_listener
 
 if $DEBUG
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 2f88bc1..e9cac00 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -107,8 +107,6 @@ opts = OptionParser.new("", 24, '  ') do |opts|
   opts.parse! ARGV
 end
 
-ru = ARGV[0] || (File.exist?('config.ru') ? 'config.ru' : nil)
-
 def rails_dispatcher
   if ::Rails::VERSION::MAJOR >= 3 && ::File.exist?('config/application.rb')
     if ::File.read('config/application.rb') =~ /^module\s+([\w:]+)\s*$/
@@ -127,9 +125,20 @@ def rails_dispatcher
   result || abort("Unable to locate the application dispatcher class")
 end
 
-def rails_builder(daemonize)
+def rails_builder(ru, opts, daemonize)
+  return Unicorn.builder(ru, opts) if ru
+
+  # allow Configurator to parse cli switches embedded in the ru file
+  Unicorn::Configurator::RACKUP.update(:file => :rails, :optparse => opts)
+
   # this lambda won't run until after forking if preload_app is false
+  # this runs after config file reloading
   lambda do ||
+    # Rails 3 includes a config.ru, use it if we find it after
+    # working_directory is bound.
+    ::File.exist?('config.ru') and
+      return Unicorn.builder('config.ru', opts).call
+
     # Load Rails and (possibly) the private version of Rack it bundles.
     begin
       require ::File.expand_path('config/boot')
@@ -179,7 +188,7 @@ def rails_builder(daemonize)
   end
 end
 
-app = ru ? Unicorn.builder(ru, opts) : rails_builder(daemonize)
+app = rails_builder(ARGV[0], opts, daemonize)
 options[:listeners] << "#{host}:#{port}" if set_listener
 
 if $DEBUG
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index c026236..a7b0646 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -33,11 +33,10 @@ module Unicorn
     # Unicorn config).  The returned lambda will be called when it is
     # time to build the app.
     def builder(ru, opts)
-      if ru =~ /\.ru\z/
-        # parse embedded command-line options in config.ru comments
-        /^#\\(.*)/ =~ File.read(ru) and opts.parse!($1.split(/\s+/))
-      end
+      # allow Configurator to parse cli switches embedded in the ru file
+      Unicorn::Configurator::RACKUP.update(:file => ru, :optparse => opts)
 
+      # always called after config file parsing, may be called after forking
       lambda do ||
         inner_app = case ru
         when /\.ru$/
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index e4305c2..0716e64 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -13,6 +13,12 @@ module Unicorn
   # nginx is also available at
   # http://unicorn.bogomips.org/examples/nginx.conf
   class Configurator < Struct.new(:set, :config_file, :after_reload)
+    # :stopdoc:
+    # used to stash stuff for deferred processing of cli options in
+    # config.ru after "working_directory" is bound.  Do not rely on
+    # this being around later on...
+    RACKUP = {}
+    # :startdoc:
 
     # Default settings for Unicorn
     DEFAULTS = {
@@ -51,6 +57,8 @@ module Unicorn
     def reload #:nodoc:
       instance_eval(File.read(config_file), config_file) if config_file
 
+      parse_rackup_file
+
       # working_directory binds immediately (easier error checking that way),
       # now ensure any paths we changed are correctly set.
       [ :pid, :stderr_path, :stdout_path ].each do |var|
@@ -66,6 +74,9 @@ module Unicorn
 
     def commit!(server, options = {}) #:nodoc:
       skip = options[:skip] || []
+      if ready_pipe = RACKUP.delete(:ready_pipe)
+        server.ready_pipe = ready_pipe
+      end
       set.each do |key, value|
         value == :unset and next
         skip.include?(key) and next
@@ -411,5 +422,50 @@ module Unicorn
       set[var] = my_proc
     end
 
+    # this is called _after_ working_directory is bound.  This only
+    # parses the embedded switches in .ru files
+    # (for "rackup" compatibility)
+    def parse_rackup_file # :nodoc:
+      ru = RACKUP[:file] or return # we only return here in unit tests
+
+      # :rails means use (old) Rails autodetect
+      if ru == :rails
+        File.readable?('config.ru') or return
+        ru = 'config.ru'
+      end
+
+      File.readable?(ru) or
+        raise ArgumentError, "rackup file (#{ru}) not readable"
+
+      # it could be a .rb file, too, we don't parse those manually
+      ru =~ /\.ru\z/ or return
+
+      /^#\\(.*)/ =~ File.read(ru) or return
+      warn "ru cli opts: #{$1}"
+      RACKUP[:optparse].parse!($1.split(/\s+/))
+
+      # XXX ugly as hell, WILL FIX in 2.x (along with Rainbows!/Zbatery)
+      host, port, set_listener, options, daemonize =
+                      eval("[ host, port, set_listener, options, daemonize ]",
+                           TOPLEVEL_BINDING)
+
+      warn [ :host, :port, :set_listener, :options, :daemonize ].inspect
+      warn [ ru, host, port, set_listener, options, daemonize ].inspect
+      # XXX duplicate code from bin/unicorn{,_rails}
+      set[:listeners] << "#{host}:#{port}" if set_listener
+
+      if daemonize
+        # unicorn_rails wants a default pid path, (not plain 'unicorn')
+        if ru == :rails
+          spid = set[:pid]
+          pid('tmp/pids/unicorn.pid') if spid.nil? || spid == :unset
+        end
+        unless RACKUP[:daemonized]
+          Unicorn::Launcher.daemonize!(options)
+          RACKUP[:ready_pipe] = options.delete(:ready_pipe)
+        end
+      end
+    end
+
   end
 end
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 5ab04c7..7f3ffa6 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -56,8 +56,9 @@ class Unicorn::Launcher
       end
     end
     # $stderr/$stderr can/will be redirected separately in the Unicorn config
-    Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
-    Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+    Unicorn::Configurator::DEFAULTS[:stderr_path] ||= "/dev/null"
+    Unicorn::Configurator::DEFAULTS[:stdout_path] ||= "/dev/null"
+    Unicorn::Configurator::RACKUP[:daemonized] = true
   end
 
 end
diff --git a/t/t0003-working_directory.sh b/t/t0003-working_directory.sh
new file mode 100755
index 0000000..3faa6c0
--- /dev/null
+++ b/t/t0003-working_directory.sh
@@ -0,0 +1,53 @@
+#!/bin/sh
+. ./test-lib.sh
+t_plan 4 "config.ru inside alt working_directory"
+
+t_begin "setup and start" && {
+	unicorn_setup
+	rtmpfiles unicorn_config_tmp
+	rm -rf $t_pfx.app
+	mkdir $t_pfx.app
+
+	port=$(expr $listen : '[^:]*:\([0-9]\+\)')
+	host=$(expr $listen : '\([^:]*\):[0-9]\+')
+
+	cat > $t_pfx.app/config.ru <<EOF
+#\--daemonize --host $host --port $port
+use Rack::ContentLength
+use Rack::ContentType, "text/plain"
+run lambda { |env| [ 200, {}, [ "#{\$master_ppid}\\n" ] ] }
+EOF
+	# we have --host/--port in config.ru instead
+	grep -v ^listen $unicorn_config > $unicorn_config_tmp
+
+	# the whole point of this exercise
+	echo "working_directory '$t_pfx.app'" >> $unicorn_config_tmp
+
+	# allows ppid to be 1 in before_fork
+	echo "preload_app true" >> $unicorn_config_tmp
+	cat >> $unicorn_config_tmp <<\EOF
+before_fork do |server,worker|
+  $master_ppid = Process.ppid # should be zero to detect daemonization
+end
+EOF
+
+	mv $unicorn_config_tmp $unicorn_config
+
+	# rely on --daemonize switch, no & or -D
+	unicorn -c $unicorn_config
+	unicorn_wait_start
+}
+
+t_begin "hit with curl" && {
+	body=$(curl -sSf http://$listen/)
+}
+
+t_begin "killing succeeds" && {
+	kill $unicorn_pid
+}
+
+t_begin "response body ppid == 1 (daemonized)" && {
+	test "$body" -eq 1
+}
+
+t_done
-- 

Eric Wong (7):
      launcher: get rid of backwards compatibility code
      isolate: don't run isolate in parallel
      Rakefile: only try rake-compiler if VERSION is defined
      Rakefile: isolate to rbx directory
      tests: set NO_PROXY when running tests
      unicorn_rails: use Kernel#warn instead of $stderr.puts
      respect "working_directory" wrt config.ru

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 8%]

* Re: Shared memory between workers
  2010-04-26  8:18  6% Shared memory between workers Iñaki Baz Castillo
@ 2010-04-26 19:03  0% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-04-26 19:03 UTC (permalink / raw)
  To: unicorn list

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> Hi, I plan to build a SIP TCP server (no UDP) based on
> Unicorn/Rainbows! HTTP server. The main different between a SIP server
> and HTTP server are:
> 
> - SIP uses persistent TCP connections, so I should use Rainbows!.
> - For a SIP server it's not valid a simple request-response model.
> Different workers could handle SIP messages (requests and responses)
> belonging to the same SIP session so I need a shared memory between
> all the workers.
> 
> Another option is using EventMachine, perhaps more suitable for this
> purpose by design as it uses a single Ruby process so sharing memory
> is not a problem. In the other side using a single process in a
> multicore server is a pain.
> I would like to use Unicorn/Rainbows as I love its design: by far it's
> the more reliable and efficient Ruby HTTP server and it takes
> advantages of Unix's features.
> 
> I don't want to use a DB server neither MemCache as "shared memory" as
> it would be too slow. Is there any way to share RAM memory between
> different Unicorn/Rainbows! workers in a *safe* way? I could create a
> Hash or Array of SIP sessions into the master process so all the
> workers reuse it, but I don't think it would be safe to access/write
> into it from different Ruby processes. For that I would also need a
> semaphore system (perhaps again a shared variable between all workers
> in order to lock the shared Array/Hash when a worker writes into it).
> 
> Any tip about it? suggstions?

Hi Iñaki,

First off I wouldn't call Memcached slow...  How many requests do you
need out of it and what kind of access patterns are you doing?

If you're on a modern Linux 2.6 box, then anything on the local
filesystem is basically shared memory if you have enough RAM to store
your working set (and it sounds like you do).

If you're willing to be confined to a single box, I've had great
experiences with TokyoCabinet the past few years.  One system I built
shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn
(previously Mongrel) workers.  TokyoCabinet uses mmap and pread/pwrite,
so you can safely share DB handles across any number of native
threads/processes.

Writes in TC need filesystem locking and you can only have one active
writer, but that was still plenty fast enough in my experience.

If fsync() becomes a bottleneck, then for non-critical data either:

  1) disable it (TC lets you)
  2) or use libeatmydata if you choose something that doesn't let
     you disable fsync()

<soapbox>
  My personal opinion is that fsync()/O_SYNC are completely overrated.
  All important machines these days have redundant storage, redundant
  power supplies, ECC memory, hardware health monitoring/alerting and
  backup power.  Any remaining hardware/kernel failure possibilities
  are far less likely than application bugs that corrupt data :)

  But, if you really have critical data, /then/ get an SSD or
  a battery-backed write cache.
</soapbox>

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Shared memory between workers
@ 2010-04-26  8:18  6% Iñaki Baz Castillo
  2010-04-26 19:03  0% ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Iñaki Baz Castillo @ 2010-04-26  8:18 UTC (permalink / raw)
  To: unicorn list

Hi, I plan to build a SIP TCP server (no UDP) based on
Unicorn/Rainbows! HTTP server. The main different between a SIP server
and HTTP server are:

- SIP uses persistent TCP connections, so I should use Rainbows!.
- For a SIP server it's not valid a simple request-response model.
Different workers could handle SIP messages (requests and responses)
belonging to the same SIP session so I need a shared memory between
all the workers.

Another option is using EventMachine, perhaps more suitable for this
purpose by design as it uses a single Ruby process so sharing memory
is not a problem. In the other side using a single process in a
multicore server is a pain.
I would like to use Unicorn/Rainbows as I love its design: by far it's
the more reliable and efficient Ruby HTTP server and it takes
advantages of Unix's features.

I don't want to use a DB server neither MemCache as "shared memory" as
it would be too slow. Is there any way to share RAM memory between
different Unicorn/Rainbows! workers in a *safe* way? I could create a
Hash or Array of SIP sessions into the master process so all the
workers reuse it, but I don't think it would be safe to access/write
into it from different Ruby processes. For that I would also need a
semaphore system (perhaps again a shared variable between all workers
in order to lock the shared Array/Hash when a worker writes into it).

Any tip about it? suggstions?
Thanks a lot.

-- 
Iñaki Baz Castillo
<ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 6%]

* [ANN] unicorn 0.97.1 - fix HTTP parser for Rainbows!/Zbatery
@ 2010-04-19 21:31 11% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-04-19 21:31 UTC (permalink / raw)
  To: mongrel-unicorn

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

This release fixes a denial-of-service vector for derived
servers exposed directly to untrusted clients.

This bug does not affect most Unicorn deployments as Unicorn is
only supported with trusted clients (such as nginx) on a LAN.
nginx is known to reject clients that send invalid
Content-Length headers, so any deployments on a trusted LAN
and/or behind nginx are safe.

Servers affected by this bug include (but are not limited to)
Rainbows! and Zbatery.  This bug does not affect Thin nor
Mongrel, as neither got the request body filtering treatment
that the Unicorn HTTP parser got in August 2009.

The bug fixed in this release could result in a
denial-of-service as it would trigger a process-wide assertion
instead of raising an exception.  For servers such as
Rainbows!/Zbatery that serve multiple clients per worker
process, this could abort all clients connected to the
particular worker process that hit the assertion.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 11%]

* Re: Using HTTP/1.1 and permanent connections
  2010-04-16 20:09  6% ` Eric Wong
@ 2010-04-17 22:53  0%   ` Iñaki Baz Castillo
  0 siblings, 0 replies; 191+ results
From: Iñaki Baz Castillo @ 2010-04-17 22:53 UTC (permalink / raw)
  To: unicorn list

2010/4/16 Eric Wong <normalperson@yhbt.net>:
> Hi Iñaki,
>
> Unicorn won't try to read further requests from the socket (see
> process_client() in lib/unicorn.rb).

Ok, understood now.


> However, you can use Rainbows! without specifying a concurrency model at
> all (or "use :Base").  That way you'll get proper HTTP/1.1 keepalive
> support and you can control keepalive_timeout:
>
> # both of these values are the defaults for unconfigured Rainbows!
> Rainbows! do
>  use :Base
>  keepalive_timeout 5
> end


Great! Thanks a lot.


-- 
Iñaki Baz Castillo
<ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[relevance 0%]

* Re: Using HTTP/1.1 and permanent connections
  @ 2010-04-16 20:09  6% ` Eric Wong
  2010-04-17 22:53  0%   ` Iñaki Baz Castillo
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-04-16 20:09 UTC (permalink / raw)
  To: unicorn list

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> Hi, I know that Unicorn forces TCP disconnection as it's explained at
> the top of lib/unicorn/http_response.rb:
> 
>   # A design decision was made to force the client to not pipeline or
>   # keepalive requests.  HTTP/1.1 pipelining really kills the
>   # performance due to how it has to be handled and how unclear the
>   # standard is.  To fix this the HttpResponse always gives a
>   # "Connection: close" header which forces the client to close right
>   # away.  The bonus for this is that it gives a pretty nice speed boost
>   # to most clients since they can close their connection immediately.
> 
> 
> However I want to try TCP permanent connections from the client (or a
> proxy) to Unicorn. I've tryed to modify same file as above by changing
> at the end:

<snip>

> But of course this is not enough and it fails. Not sure what exactly
> happens, it seems that a Unicorn worker doesn't process requests
> anymore until replying the first response.
> 
> So I would like to know if it's feasible to make Unicorn work in
> persistent mode.
> NOTE: I already know that the current design is really good, better
> than using persistent connections, but I want to experiment with
> persistent connections for other purposes I will explain in a future.

Hi Iñaki,

Unicorn won't try to read further requests from the socket (see
process_client() in lib/unicorn.rb).

However, you can use Rainbows! without specifying a concurrency model at
all (or "use :Base").  That way you'll get proper HTTP/1.1 keepalive
support and you can control keepalive_timeout:

# both of these values are the defaults for unconfigured Rainbows!
Rainbows! do
  use :Base
  keepalive_timeout 5
end

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* [ANN] raindrops - real-time stats for preforking Rack servers
@ 2010-04-09  2:37  5% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-04-09  2:37 UTC (permalink / raw)
  To: mongrel-unicorn

(I announced this yesterday on ruby-talk, but I realized it would
be wise to actually announce it on the list of a server it was
designed for :)

Raindrops is a real time stats package to show statistics for Rack HTTP
servers.  It is designed for preforking servers such as Rainbows! and
Unicorn, but should support any Rack HTTP server under Ruby 1.9, 1.8 and
possibly Rubinius (untested) on platforms supporting POSIX shared memory
and compiled with GCC (for atomic builtins).

Raindrops includes a Struct-like Raindrops::Struct class that may be used
standalone to create atomic counters shared across any number of forked
processes under SMP.

* http://raindrops.bogomips.org/
* raindrops@librelist.com
* git://git.bogomips.org/raindrops.git

== Features

* counters are shared across all forked children and lock-free

* counters are kept on separate cache lines to reduce contention under SMP

* may expose server statistics as a Rack Middleware endpoint
  (default: "/_raindrops")

* middleware displays the number of actively processing and writing
  clients from a single request regardless of which worker process
  it hits.

== Linux-only Extra Features!

* Middleware response includes extra stats for bound TCP and
  Unix domain sockets (configurable, it can include stats from
  other TCP or UNIX domain socket servers).

* TCP socket stats use efficient inet_diag facilities via netlink
  instead of parsing /proc/net/tcp to minimize overhead.
  This was fun to discover and write.

== Install

raindrops requires GCC 4.x (or compatible) or later to support the
atomic builtins (__sync_{add,sub}_and_fetch()).  Atomic operations on
other compilers may be supported if there is demand.

If you're using a packaged Ruby distribution, make sure you have a C
compiler and the matching Ruby development libraries and headers.

If you use RubyGems:

    gem install raindrops

Otherwise grab the latest tarball from:

http://raindrops.bogomips.org/files/

Unpack it, and run "ruby setup.rb"

== Usage (Rainbows!/Unicorn preload_app=false)

If you're using preload_app=false (the default) in your Rainbows!/Unicorn
config file, you'll need to create the global Stats object before
forking.

        require 'raindrops'
        $stats ||= Raindrops::Middleware::Stats.new

In your Rack config.ru:

        use Raindrops::Middleware, :stats => $stats

== Usage (Rainbows!/Unicorn preload_app=true)

If you're using preload_app=true in your Rainbows!/Unicorn
config file, just add the middleware to your stack:

In your Rack config.ru:

        use Raindrops::Middleware

== Usage (Linux-extras)

To get bound listener statistics under Linux, you need to specify the
listener names for your server.  You can even include listen sockets for
*other* servers on the same machine.  This can be handy for monitoring
your nginx proxy as well.

In your Rack config.ru, just pass the :listeners argument as an array of
strings (along with any other arguments).  You can specify any
combination of TCP or Unix domain socket names:

        use Raindrops::Middleware, :listeners => %w(0.0.0.0:80 /tmp/.sock)

See the tests/ and examples/ directory for more examples

== Development

You can get the latest source via git from the following locations:

  git://git.bogomips.org/raindrops.git
  git://repo.or.cz/raindrops.git (mirror)

You may browse the code from the web and download the latest snapshot
tarballs here:

* http://git.bogomips.org/cgit/raindrops.git (cgit)
* http://repo.or.cz/w/raindrops.git (gitweb)

Inline patches (from "git format-patch") to the mailing list are
preferred because they allow code review and comments in the reply to
the patch.

We will adhere to mostly the same conventions for patch submissions as
git itself.  See the Documentation/SubmittingPatches document
distributed with git on on patch submission guidelines to follow.  Just
don't email the git mailing list or maintainer with raindrops patches.

== Contact

All feedback (bug reports, user/development discussion, patches, pull
requests) go to the mailing list: mailto:raindrops@librelist.com

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 5%]

* Re: funky process tree + stillborn masters
  2010-04-08 23:55  4% ` Eric Wong
@ 2010-04-09  1:20 14%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-04-09  1:20 UTC (permalink / raw)
  To: unicorn list

Eric Wong <normalperson@yhbt.net> wrote:
> I assume you're using regular "unicorn" to run your Sinatra apps and not
> "unicorn_rails".  I made some largish cleanups to both for the 0.97.0
> release and and perhaps some bugs slipped into the "_rails" variant.

<snip>

> Jamie Wilkinson <jamie@tramchase.com> wrote:
> > Are the goofy worker processes in the process tree a real problem, or
> > just a red herring?
> 
> Not sure if it's a problem, but with Bundler I assume Rack itself is a
> bundled dependency, but you're starting unicorn_rails out of
> /usr/bin/unicorn_rails which indicates Unicorn is not bundled (and won't
> use the bundled Rack).  Can you ensure your unbundled Rack is the same
> version as the bundled one to be on the safe side?
> 
> I've yet to try bundler 0.9 (and have barely used earlier), but you can
> also try bundling Unicorn and using the bundled bin/unicorn(_rails)
> launchers instead to ensure a consistent environment.
> 
> Unicorn currently ends up (auto)loading "rack/utils" before the
> application is loaded, maybe it could (auto)load it after the app is
> loaded for preload_app users.

Do the following two patches help?  I've also pushed out a few
cleanups to unicorn.git and also put up a prerelease gem at:

  http://unicorn.bogomips.org/files/gems/unicorn-0.97.0.7.g22e3.gem

Shortlog of changes since 0.97.0:

Eric Wong (7):
      tests: fix to run under Ruby 1.9.2dev
      KNOWN_ISSUES: document Array#shuffle / Array#sample issue under 1.9
      unicorn_rails: use TOPLEVEL_BINDING for eval
      unicorn_rails: respect user's encoding in config.ru in 1.9
      unicorn_rails: rename variable for consistency
      bin/*: remove unnecessary listeners variable
      unicorn: load constants after app has loaded

>From e1a72d58add4260feb6da56d9d588270173da74f Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Thu, 8 Apr 2010 17:10:46 -0700
Subject: [PATCH] unicorn_rails: use TOPLEVEL_BINDING for eval

This is to ensure there are no namespace inconsistencies
---
 bin/unicorn_rails |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 37ee027..de2361e 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -148,7 +148,7 @@ def rails_builder(config, daemonize)
     when /\.ru$/
       raw = File.open(config, "rb") { |fp| fp.sysread(fp.stat.size) }
       raw.sub!(/^__END__\n.*/, '')
-      eval("Rack::Builder.new {(#{raw}\n)}.to_app", nil, config)
+      eval("Rack::Builder.new {(#{raw}\n)}.to_app", TOPLEVEL_BINDING, config)
     else
       require config
       Object.const_get(File.basename(config, '.rb').capitalize)
-- 

>From 22e3ed4de0e89b97dac91c95c796eb8a7f93e5de Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Thu, 8 Apr 2010 18:05:43 -0700
Subject: [PATCH] unicorn: load constants after app has loaded

This will help ensure we use the same version of Rack the
application uses and avoid loading conflicting/incompatible
versions.
---
 lib/unicorn.rb |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b63abeb..75ce09d 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -800,15 +800,15 @@ module Unicorn
 
     def build_app!
       if app.respond_to?(:arity) && app.arity == 0
-        # exploit COW in case of preload_app.  Also avoids race
-        # conditions in Rainbows! since load/require are not thread-safe
-        Unicorn.constants.each { |x| Unicorn.const_get(x) }
-
         if defined?(Gem) && Gem.respond_to?(:refresh)
           logger.info "Refreshing Gem list"
           Gem.refresh
         end
         self.app = app.call
+
+        # exploit COW in case of preload_app.  Also avoids race
+        # conditions in Rainbows! since load/require are not thread-safe
+        Unicorn.constants.each { |x| Unicorn.const_get(x) }
       end
     end

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 14%]

* Re: funky process tree + stillborn masters
  @ 2010-04-08 23:55  4% ` Eric Wong
  2010-04-09  1:20 14%   ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-04-08 23:55 UTC (permalink / raw)
  To: unicorn list

Jamie Wilkinson <jamie@tramchase.com> wrote:
> Since upgrading bundler (but applying the RUBYOPT path fixes) we've
> started experiencing intermittent, difficult-to-isolate USR2 restart
> failures.
> 
> After a USR2 signal our process tree winds up looking like this, with
> several master-esque processes listed as children (but without the
> "worker[N]" label):
> 
> app      14402  4.4  0.8 199612 70264 ?        S    14:07   0:04 unicorn_rails master -c config/unicorn.rb -E production -D
> app      14433  0.0  0.8 204540 68504 ?        Sl   14:07   0:00  \_ unicorn_rails worker[0] -c config/unicorn.rb -E production -D
> app      14435  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[1] -c config/unicorn.rb -E production -D
> app      14438  0.0  0.8 199748 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
> app      14440  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[3] -c config/unicorn.rb -E production -D
> app      14442  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[4] -c config/unicorn.rb -E production -D
> app      14445  0.0  0.8 199760 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
> app      14447  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[6] -c config/unicorn.rb -E production -D
> app      14449  0.0  0.8 204780 69272 ?        Sl   14:07   0:00  \_ unicorn_rails worker[7] -c config/unicorn.rb -E production -D

> Sending another USR2 signal will bring a new master into the mix as a
> child, spins up a single child worker of its own (which also resembles
> the "/usr/bin/ruby1.8" master-esque processes), and then fails to
> continue. 

Hi Jamie,

Odd, if I had to guess PIDs 14438 and 14445 are actually worker[2] and
worker[5] based on the PIDs relative to other workers.  So the new
master died right away, which really should've been logged...

> Further USR2 restarts will obviously do nothing, and we're forced to
> either kill -9 the stillborn master or cold-restart all of the
> unicorns. Nothing out of the ordinary is dumped to stderr or stdout

Anything in your before_fork/after_fork hooks?  Since it looks like
you're on a Linux system, can you strace the master while you send
it a USR2 and see if anything strange happens?

Also, can you strace the weird looking processes above and see if
they're doing anything?

> Starting unicorns fresh produces a nice process list where every child
> is listed cleanly as "unicorn_rails worker[N]"
> 
> We only have this issue in one of our applications, on a machine that
> has 1 Rails app & 2 Sinatra apps, all powered by nginx+unicorn. We've
> also only noticed this since upgrading to bundler from bundler08

I assume you're using regular "unicorn" to run your Sinatra apps and not
"unicorn_rails".  I made some largish cleanups to both for the 0.97.0
release and and perhaps some bugs slipped into the "_rails" variant.

Can you try regular "unicorn" with a config.ru for Rails?  I've stolen
this from the Rainbows! FAQ (http://rainbows.rubyforge.org/FAQ.html):

  For Rails 2.3.x and later, the following config.ru will work for you:

    ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
    require "config/environment"
    use Rails::Rack::Static
    run ActionController::Dispatcher.new

  For older versions of Rails, the following config.ru will work:

    ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
    require 'config/boot'
    require 'config/environment'
    require 'unicorn/app/old_rails'
    require 'unicorn/app/old_rails/static' # not needed with Unicorn 0.95+
    use Unicorn::App::OldRails::Static
    run Unicorn::App::OldRails.new

  One thing to watch out for is that RAILS_ENV will not be set in the
  environment for you, thus we set it to match RACK_ENV.

> Are the goofy worker processes in the process tree a real problem, or
> just a red herring?

Not sure if it's a problem, but with Bundler I assume Rack itself is a
bundled dependency, but you're starting unicorn_rails out of
/usr/bin/unicorn_rails which indicates Unicorn is not bundled (and won't
use the bundled Rack).  Can you ensure your unbundled Rack is the same
version as the bundled one to be on the safe side?

I've yet to try bundler 0.9 (and have barely used earlier), but you can
also try bundling Unicorn and using the bundled bin/unicorn(_rails)
launchers instead to ensure a consistent environment.

Unicorn currently ends up (auto)loading "rack/utils" before the
application is loaded, maybe it could (auto)load it after the app is
loaded for preload_app users.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 0.96.1 - leak fix for Rainbows!/Zbatery
@ 2010-02-13 10:34 11% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2010-02-13 10:34 UTC (permalink / raw)
  To: ruby-talk ML, mongrel-unicorn, rainbows-talk

First off, this memory leak doesn't affect Unicorn itself at all
(but it doesn't hurt, either).

Unicorn itself always allocates the HttpParser once and always reuses it
in every sequential request.

This leak only affects applications that repeatedly allocate a new HTTP
parser.  Thus this bug affects _all_ deployments of Rainbows! and
Zbatery.  These servers allocate a new parser for every client
connection to serve clients concurrently, but due to a bug in Unicorn,
never allows the Ruby GC to properly free the memory allocated.

Here's what happened:

  I misread the Data_Make_Struct()/Data_Wrap_Struct()
  documentation and ended up passing NULL as the "free" argument
  instead of -1, causing the memory to never be freed.

  From README.EXT in the MRI source which I misread:
  > The free argument is the function to free the pointer
  > allocation.  If this is -1, the pointer will be just freed.
  > The functions mark and free will be called from garbage
  > collector.

Yes, I suck at reading and can't write code properly as a result.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong



^ permalink raw reply	[relevance 11%]

* Re: stderr_path doesn't work at the moment of being called
  2010-01-05 21:47  4%   ` Eric Wong
@ 2010-01-05 22:50  0%     ` Iñaki Baz Castillo
  0 siblings, 0 replies; 191+ results
From: Iñaki Baz Castillo @ 2010-01-05 22:50 UTC (permalink / raw)
  To: mongrel-unicorn

El Martes, 5 de Enero de 2010, Eric Wong escribió:
> Iñaki Baz Castillo <ibc@aliax.net> wrote:
> > El Martes, 5 de Enero de 2010, Iñaki Baz Castillo escribió:
> > > The second workaround is exactly what Unicorn does when calling to
> > > "stderr_path" ("redirect_io" method). So, why doesn't work for me
> > > without the hacks? do I miss something?
> >
> > By running 'exec "ls -l /proc/$$/fd"' after 'stderr_path' I've realized
> > that the stderr is still pointing to /dev/pts/N. Perhaps the method is
> > called later and now it's just "eval"-ulated?
> 
> The value is stored and then set later.  I avoid changing internal
> configuration variables during evaluation time for several reasons:
> 
> 1. A Unicorn config file is typically the last thing configured for an
>    application, when a user is testing their configuration it's easier
>    to see the error message in the console if it's in the Unicorn
>    config file
> 
> 2. HUP-ing a with a bad config file should not leave the server in an
>    unknown state.  There are still some things that bind immediately
>    (working_directory, all of Rainbows!) for now, but for the most
>    part it should be possible to recover on bad HUPs.
> 
> 3. "working_directory" needs to be factored in for non-absolute paths.
>    I prefer to work with absolute paths, but some people like
>    non-absolute ones for various reasons.  So we always ensure
>    stderr_path is set after "working_directory" is called.

Ok, it makes sense :)
Thanks. 


-- 
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: stderr_path doesn't work at the moment of being called
  @ 2010-01-05 21:47  4%   ` Eric Wong
  2010-01-05 22:50  0%     ` Iñaki Baz Castillo
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2010-01-05 21:47 UTC (permalink / raw)
  To: unicorn list

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Martes, 5 de Enero de 2010, Iñaki Baz Castillo escribió:
> 
> > The second workaround is exactly what Unicorn does when calling to
> > "stderr_path" ("redirect_io" method). So, why doesn't work for me without
> >  the hacks? do I miss something?
> 
> By running 'exec "ls -l /proc/$$/fd"' after 'stderr_path' I've realized that 
> the stderr is still pointing to /dev/pts/N. Perhaps the method is called later 
> and now it's just "eval"-ulated?

The value is stored and then set later.  I avoid changing internal
configuration variables during evaluation time for several reasons:

1. A Unicorn config file is typically the last thing configured for an
   application, when a user is testing their configuration it's easier
   to see the error message in the console if it's in the Unicorn
   config file

2. HUP-ing a with a bad config file should not leave the server in an
   unknown state.  There are still some things that bind immediately
   (working_directory, all of Rainbows!) for now, but for the most
   part it should be possible to recover on bad HUPs.

3. "working_directory" needs to be factored in for non-absolute paths.
   I prefer to work with absolute paths, but some people like
   non-absolute ones for various reasons.  So we always ensure
   stderr_path is set after "working_directory" is called.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
  2009-12-28  3:29  5%           ` Eric Wong
@ 2009-12-28 10:39  0%             ` Iñaki Baz Castillo
  0 siblings, 0 replies; 191+ results
From: Iñaki Baz Castillo @ 2009-12-28 10:39 UTC (permalink / raw)
  To: mongrel-unicorn

El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:

> > The master process would start properly and would notify "success" to
> > grandparent. (so the init script returns 0). But the fact is that all the
> > workers fail to start and are respawned again and again.
> 
> For that particular case there'll be a Unicorn::Configurator#user
> directive.
> 
> But really, there's absolutely no good reason to use user switching in a
> backend application server like Unicorn.
> 
> I only added that feature to support derivative servers like Rainbows!,
> and even then it's debatable since using things like iptables or load
> balancers can be used to redirect port 80 to arbitrary ports anyways.

Well, chaning the running user it's common in most of servers. I've already 
found lots of cases of attacks to Apache servers running some "cool" PHP 
application (so we get exploits in /tmp or/var/tmp as they are the only 
writable paths for "www-data" user running apache).

However it's true that Unicorn approach (worker.user) is different as the 
master process remains as root (but since the master process doesn't listen it 
shouldn't matter).

So, do you mean that there will be a new configuration option called "user" 
(and "group") so also themaster process would run as such user?

Thanks.

-- 
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
  @ 2009-12-28  3:29  5%           ` Eric Wong
  2009-12-28 10:39  0%             ` Iñaki Baz Castillo
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2009-12-28  3:29 UTC (permalink / raw)
  To: unicorn list

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Domingo, 27 de Diciembre de 2009, Eric Wong escribió:
> > > Hi, I've improved it a bit with your suggestions and also simplified
> > > it a lot.  Now there is no a "controller process". Instead, if
> > > "Unicorn.run" raises then master process rescues the exception and
> > > sends USR1 to parent process so it exists with 1.
> > 
> > Hi, I realized Unicorn.run inside the daemonize method doesn't work.
> 
> Strange, it works for me...

"Doesn't work" as in it's not friendly to servers based on Unicorn (like
Rainbows!).

> > However, I've reimplemented much of your idea here so it does not
> > involve sleeping at all, but rather shares a pipe with the grandparent
> > (started by the user) and Unicorn master process.  The added benefit of
> > using a pipe is that there's no fuzzy sleeping involved at or grace
> > periods to worry about, too.
> > 
> > Also pushed out to the "ready_pipe" branch of
> > git://git.bogomips.org/unicorn.git
> > 
> > Let me know how it goes.
> > 
> > If everything looks good, I'll consider merging for 0.96.0.
> 
> It's really awesome! I've tested it and it works great, and avoids the 
> sleeping stuff and a new commandlline option. Congratulations :)

The current version is actually slightly buggy in that it leaks
the pipe descriptor...

> However there is still a not solved case. Let me please explain it with two 
> examples:
> 
> 1) In "after_fork" I set:
>      worker.user("non_existing_user","non_existing_group")
> 
> The master process would start properly and would notify "success" to 
> grandparent. (so the init script returns 0). But the fact is that all the 
> workers fail to start and are respawned again and again.

For that particular case there'll be a Unicorn::Configurator#user
directive.

But really, there's absolutely no good reason to use user switching in a
backend application server like Unicorn.

I only added that feature to support derivative servers like Rainbows!,
and even then it's debatable since using things like iptables or load
balancers can be used to redirect port 80 to arbitrary ports anyways.

> 2) I use "Clogger" in my Rack config.ru. But I set a log file in a path that 
> doesn't exist. Clogger raises due to this (it doesn't try to create the full 
> path).
> Again the master process has notified "success" to grandparent (exis status 0 
> then) but the workers are respawned again and again.

There's really an infinite number of ways to screw things up badly in
workers and cause them to flap.  It's not possible to protect careless
users from all of them, so attempting to do so would be a waste of
effort.

Avoiding user-switching in an app server is a great first step, as it
eliminates an entire class of problems :)

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 5%]

* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
  @ 2009-12-27  1:31  2%       ` Eric Wong
    0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2009-12-27  1:31 UTC (permalink / raw)
  To: unicorn list

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Sábado, 26 de Diciembre de 2009, Iñaki Baz Castillo escribió:
> > El Sábado, 26 de Diciembre de 2009, Eric Wong escribió:
> > > Interesting, I could be tempted to accept this with a few changes...
> > >
> > > Process.detach() spawns a background Thread.  Having running threads
> > > before forking won't fly with certain OS threading libraries (some
> > > versions of FreeBSD, I believe, check MRI Redmine for some open bugs).
> > 
> > Yes, but there is a problem:
> > Imagine I don't use Process.detach and the master process ends when loads
> >  the rack application (due to a typo or due to an error in unicorn conf
> >  file). Then the master process would remain visible/alive but as zombie.
> > If so, when the controller process sends signal 0 to check the master
> >  process it will always receive 'true' (a zombie process is a process which
> >  can receive signals, it's does exist).
> 
> Hi, I've improved it a bit with your suggestions and also simplified
> it a lot.  Now there is no a "controller process". Instead, if
> "Unicorn.run" raises then master process rescues the exception and
> sends USR1 to parent process so it exists with 1.

Hi, I realized Unicorn.run inside the daemonize method doesn't work.
The daemonize method is also used by Rainbows! (and Zbatery).

However, I've reimplemented much of your idea here so it does not
involve sleeping at all, but rather shares a pipe with the grandparent
(started by the user) and Unicorn master process.  The added benefit of
using a pipe is that there's no fuzzy sleeping involved at or grace
periods to worry about, too.

Also pushed out to the "ready_pipe" branch of
git://git.bogomips.org/unicorn.git

Let me know how it goes.

If everything looks good, I'll consider merging for 0.96.0.

>From 5eea32764571b721cd1a89cf9ebfa853c621ac9e Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Sat, 26 Dec 2009 17:04:57 -0800
Subject: [PATCH] exit with failure if master dies when daemonized
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This behavior change also means our grandparent (launched
from a controlling terminal or script) will wait until
the master process is ready before returning.

Thanks to Iñaki Baz Castillo for the initial implementations
and inspiration.
---
 bin/unicorn             |    2 +-
 bin/unicorn_rails       |    2 +-
 lib/unicorn.rb          |    9 +++++++--
 lib/unicorn/launcher.rb |   37 +++++++++++++++++++++++++++++--------
 test/exec/test_exec.rb  |    2 +-
 5 files changed, 39 insertions(+), 13 deletions(-)

diff --git a/bin/unicorn b/bin/unicorn
index 651c2ff..5af021d 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -161,5 +161,5 @@ if $DEBUG
   })
 end
 
-Unicorn::Launcher.daemonize! if daemonize
+Unicorn::Launcher.daemonize!(options) if daemonize
 Unicorn.run(app, options)
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 4a22a8c..b1458fc 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -202,6 +202,6 @@ end
 
 if daemonize
   options[:pid] = rails_pid
-  Unicorn::Launcher.daemonize!
+  Unicorn::Launcher.daemonize!(options)
 end
 Unicorn.run(app, options)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 71d5994..ae05f03 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -25,7 +25,8 @@ module Unicorn
 
   class << self
     def run(app, options = {})
-      HttpServer.new(app, options).start.join
+      ready_pipe = options.delete(:ready_pipe)
+      HttpServer.new(app, options).start.join(ready_pipe)
     end
   end
 
@@ -313,7 +314,7 @@ module Unicorn
     # (or until a termination signal is sent).  This handles signals
     # one-at-a-time time and we'll happily drop signals in case somebody
     # is signalling us too often.
-    def join
+    def join(ready_pipe = nil)
       # this pipe is used to wake us up from select(2) in #join when signals
       # are trapped.  See trap_deferred
       init_self_pipe!
@@ -324,6 +325,10 @@ module Unicorn
       trap(:CHLD) { |sig_nr| awaken_master }
       proc_name 'master'
       logger.info "master process ready" # test_exec.rb relies on this message
+      if ready_pipe
+        ready_pipe.syswrite($$.to_s)
+        ready_pipe.close
+      end
       begin
         loop do
           reap_all_workers
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 1229b84..2d6ad97 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -19,21 +19,42 @@ class Unicorn::Launcher
   #     the directory it was started in when being re-executed
   #     to pickup code changes if the original deployment directory
   #     is a symlink or otherwise got replaced.
-  def self.daemonize!
+  def self.daemonize!(options = {})
     $stdin.reopen("/dev/null")
+    $stdin.sync = true # may not do anything...
 
     # We only start a new process group if we're not being reexecuted
     # and inheriting file descriptors from our parent
     unless ENV['UNICORN_FD']
-      exit if fork
-      Process.setsid
-      exit if fork
+      # grandparent - reads pipe, exits when master is ready
+      #  \_ parent  - exits immediately ASAP
+      #      \_ unicorn master - writes to pipe when ready
 
-      # $stderr/$stderr can/will be redirected separately in the Unicorn config
-      Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
-      Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+      rd, wr = IO.pipe
+      grandparent = $$
+      if fork
+        wr.close # grandparent does not write
+      else
+        rd.close # unicorn master does not read
+        Process.setsid
+        exit if fork # parent dies now
+      end
+
+      if grandparent == $$
+        # this will block until HttpServer#join runs (or it dies)
+        master_pid = rd.sysread(16).to_i
+        exit!(1) unless master_pid > 1
+        exit 0
+      else # unicorn master process
+        options[:ready_pipe] = wr
+        # $stderr/$stderr can/will be redirected separately in the
+        # Unicorn config
+        Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
+        Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+
+        # returns so Unicorn.run can happen
+      end
     end
-    $stdin.sync = $stdout.sync = $stderr.sync = true
   end
 
 end
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index fc0719b..24ba856 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -805,7 +805,7 @@ EOF
       exec($unicorn_bin, "-D", "-l#{@addr}:#{@port}", "-c#{ucfg.path}")
     end
     pid, status = Process.waitpid2(pid)
-    assert status.success?, "original process exited successfully"
+    assert ! status.success?, "original process exited successfully"
     sleep 1 # can't waitpid on a daemonized process :<
     assert err.stat.size > 0
   end
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply related	[relevance 2%]

* Re: Expect 100-continue errors
  2009-12-09 22:16  4% ` Eric Wong
@ 2009-12-09 23:46  0%   ` Jan Dvorak
  0 siblings, 0 replies; 191+ results
From: Jan Dvorak @ 2009-12-09 23:46 UTC (permalink / raw)
  To: mongrel-unicorn

Eric Wong wrote:
> Hi Jan,
>
> Try disabling automatic code reloading under Rails development mode.
>
>   
Thanks, that worked.
> Unicorn (and Rainbows!) are the only Ruby servers I know of that allow
> Rack applications to properly return and continue with 100 responses.
>
> Unicorn calls your Rack app twice in this case: it sees a 100 response,
> writes the 100-continue response and then does app.call(env) again after
> deleting the "Expect: 100-continue" header from env.
>
> I haven't tested this under Rails so I don't know how well it works with
> what Rails does under the covers, but I've done this heavily with bare
> Rack applications.
>
>   
After googling, it seems like rails error, it creates locks but never
properly releases them, and deadlocks upon class reloading.


Thanks,
Jan
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 0%]

* Re: Expect 100-continue errors
  @ 2009-12-09 22:20  6%   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-12-09 22:20 UTC (permalink / raw)
  To: unicorn list

David Palm <dvdplm@gmail.com> wrote:
> On Wed, 09 Dec 2009 22:05:21 +0100, Jan Dvorak wrote:
> > Hello,
> > 
> > i'm trying to implement handling of http Expect: 100-continue with
> > unicorn in my rails app, but when i return [100] as a response
> > (which
>
> Just out of curiosity: what does 100 Continue mean and how can you use
> it (usefully)?

Hi David,

It's described in the HTTP/1.1 spec
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3

curl uses it by default:
http://curl.haxx.se/docs/faq.html#My_HTTP_POST_or_PUT_requests_are

There are several uses of them in the Rainbows! test cases:

$ git clone git://git.bogomips.org/rainbows
$ grep HTTP_EXPECT rainbows/t/*.ru

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 6%]

* Re: Expect 100-continue errors
    @ 2009-12-09 22:16  4% ` Eric Wong
  2009-12-09 23:46  0%   ` Jan Dvorak
  1 sibling, 1 reply; 191+ results
From: Eric Wong @ 2009-12-09 22:16 UTC (permalink / raw)
  To: unicorn list

Jan Dvorak <jan.dvorak@kraxnet.cz> wrote:
> Hello,
> 
> i'm trying to implement handling of http Expect: 100-continue with
> unicorn in my rails app, but when i return [100] as a response (which
> should force unicorn to return 100 status and rerun my app with the same
> connection) i get this error:
> 
> Read error: #<ThreadError: stopping only thread
>     note: use sleep to stop forever>
> /var/lib/gems/1.8/gems/actionpack-2.3.5/lib/action_controller/reloader.rb:31:in
> `lock'

Hi Jan,

Try disabling automatic code reloading under Rails development mode.

Unicorn (and Rainbows!) are the only Ruby servers I know of that allow
Rack applications to properly return and continue with 100 responses.

Unicorn calls your Rack app twice in this case: it sees a 100 response,
writes the 100-continue response and then does app.call(env) again after
deleting the "Expect: 100-continue" header from env.

I haven't tested this under Rails so I don't know how well it works with
what Rails does under the covers, but I've done this heavily with bare
Rack applications.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[relevance 4%]

* Re: Nginx Conf
  2009-11-23  3:57  4%   ` Dylan Stamat
@ 2009-11-24  7:19  0%     ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-11-24  7:19 UTC (permalink / raw)
  To: unicorn list

Dylan Stamat <dstamat@elctech.com> wrote:
> On Nov 22, 2009, at 2:53 PM, Eric Wong wrote:
> > One general thing about the nginx configs I've seen is that they're
> > missing the fail_timeout=0 directive in the "server" lines.
> > 
> > I highly recommend setting it since it's a low cost to try an upstream
> > for nginx, and you can avoid 502 errors in case there's a bug in your
> > app that causes a Unicorn worker to not send a valid HTTP response
> > (including hitting the app timeout).
> 
> Hey Eric,
> 
> Yeah, an nginx.conf in examples/ would be great.
> It's probably going to be the most widely used front for
> Unicorn/Rainbows!, so a sample config with some explanations here and
> there would be awesome.  It was great setting up Unicorn and being
> able to just grab the init.sh out of there!

OK, I'll push out an nginx example in a bit, need to make sure things
are adequately explained and linked.

> In terms of the fail_timeout, I haven't seen a nginx.conf with that
> entry yet!  Maybe I'm looking at configuration files on the wrong
> projects ;) So, since the upstream attempts are cheap, the combination
> of a max_fail of 1 and fail_timeout of 0 is ideal?  If the
> fail_timeout was at 10, and all the upstreams timed out, wouldn't it
> restart at the beginning of the round-robin, and not block... or...
> would it actually not retry on any due to the inoperable state?

max_fails doesn't seem to have any effect when fail_timeout=0.  Setting
max_fails=0 means the same thing as fail_timeout=0 in nginx >=0.6.33
(it would segfault before :x)  I've just been using fail_timeout=0
since what seems like forever in nginx...

Reading ngx_http_upstream_round_robin.c again, it appears seems that
fail_timeout/max_fails is supposed to get ignored if *all* upstreams are
failing.  So if you have a single upstream it looks like you're safe
even if your Unicorn master nukes workers.  On the other hand I remember
this not being the case at some point a while back; an entire cluster
got effectively shut down because of it.  Just set fail_timeout=0
and stop worrying about it :)

-- 
Eric Wong

^ permalink raw reply	[relevance 0%]

* Re: Nginx Conf
  @ 2009-11-23  3:57  4%   ` Dylan Stamat
  2009-11-24  7:19  0%     ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Dylan Stamat @ 2009-11-23  3:57 UTC (permalink / raw)
  To: unicorn list


On Nov 22, 2009, at 2:53 PM, Eric Wong wrote:

> huet bartels <hbartels@i-neda.com> wrote:
>> Thank you all very much for you time.  
>> 
>> I will have a look at the configuration  examples today.
> 
> Sorry for the late reply, I forgot about this thread (and I'm lazy
> about following links).
> 
> One general thing about the nginx configs I've seen is that they're
> missing the fail_timeout=0 directive in the "server" lines.
> 
> I highly recommend setting it since it's a low cost to try an upstream
> for nginx, and you can avoid 502 errors in case there's a bug in your
> app that causes a Unicorn worker to not send a valid HTTP response
> (including hitting the app timeout).
> 
> I actually have this in the Configurator documentation[1]:
> 
>    #    # See http://wiki.nginx.org/NginxHttpUpstreamModule for more details
>    #    # on nginx upstream configuration:
>    #    upstream unicorn_backend {
>    #      # for UNIX domain socket setups:
>    #      server unix:/path/to/unicorn.sock fail_timeout=0;
>    #
>    #      # for TCP setups
>    #      server 192.168.0.7:8080 fail_timeout=0;
>    #      server 192.168.0.8:8080 fail_timeout=0;
>    #      server 192.168.0.9:8080 fail_timeout=0;
>    #    }
> 
> [1] - http://unicorn.bogomips.org/Unicorn/Configurator.html
> 
> We've had fail_timeout=0 deployed to several places (many non-Unicorn
> servers) here and there and have experienced no negative effects
> (we're pretty good about keeping our backends up :)
> 
> If anybody can recommend a better place in the Unicorn docs to put this,
> that'd be great, too...  Maybe I'll drop something in the examples/
> directory.
> 
> -- 
> Eric Wong
> _______________________________________________
> mongrel-unicorn mailing list
> mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn

Hey Eric,

Yeah, an nginx.conf in examples/ would be great.
It's probably going to be the most widely used front for Unicorn/Rainbows!, so
a sample config with some explanations here and there would be awesome.
It was great setting up Unicorn and being able to just grab the init.sh out of there!

In terms of the fail_timeout, I haven't seen a nginx.conf with that entry yet!
Maybe I'm looking at configuration files on the wrong projects ;)
So, since the upstream attempts are cheap, the combination of a max_fail of 1
and fail_timeout of 0 is ideal?  If the fail_timeout was at 10, and all the
upstreams timed out, wouldn't it restart at the beginning of the round-robin, and
not block... or... would it actually not retry on any due to the inoperable state?

Thanks!
==
Dylan Stamat

ELC Technologies (TM)
1921 State Street
Santa Barbara, CA 93101
dstamat@elctech.com

(866)863-7365 Tel
(866)893-1902 Fax

+44 (0)20 7504 1346 Tel - London Office
+44 (0)20 7504 1347 Fax - London Office

http://www.elctech.com

^ permalink raw reply	[relevance 4%]

* Re: rainbows for sleepy/lethargic apps
  2009-11-16 19:30 14% rainbows for sleepy/lethargic apps Dylan Stamat
@ 2009-11-16 21:26 11% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-11-16 21:26 UTC (permalink / raw)
  To: unicorn list; +Cc: rainbows-talk

Dylan Stamat <dstamat@elctech.com> wrote:
> I'm working on a project that has very bad application performance,
> specifically, tons of long running requests due to view and
> (specifically) database contention.
> 
> While I haven't tested our Unicorn setup under load yet, tweak backlog
> counts, etc... I'm wondering if I'd be better off giving Rainbows! a
> shot?  In the Rainbows! AppPool diagram, I'm having a hard time
> understanding the N:P relationship.  When the P threshold is met, what
> happens to N (the client)?

Hi Dylan,

With AppPool, request headers are fully parsed and end up getting queued
up in userspace.  No request bodies are read (since AppPool is only
compatible with the Thread-based concurrency models at the moment).

Rainbows is designed for apps that are intentionally/unavoidably sleepy.
Since I imagine the database is within your control, I'd optimize that
first and fall back to using Unicorn with a higher :backlog.

Since you're hitting database contention, throwing more concurrency at
the database with Rainbows! is probably the wrong thing to do... If your
database is just on a high latency network for whatever reason, then
maybe Rainbows! with Neverblock drivers can help.  Otherwise if your
database is bogged down because of memory/CPU/disk, then you probably
want fewer things hitting it at once.  Can you shard, cache or otherwise
offload DB requests?

With slow views, Unicorn is pretty effective at using all available
cores already with multiple worker processes.  Even with a Ruby VM with
non-crippled native threads (Rubinius maybe, not MRI 1.9.1) I doubt
it'll help with CPU/memory-bound rendering performance.

> Also, with both Unicorn/Rainbows!, is there a explicit timeout number
> set on the clients (and how does that differ between
> Unicorn/Rainbows!)?  For instance, in our Nginx config, I have
> proxy_buffering turned on, with proxy_read_timeout and
> proxy_send_timeout's, set pretty high.

Based on your nginx config, the easiest thing would probably be to set a
high :backlog for now and then work on tuning/optimizing/avoiding your
database ASAP.

Views are harder to optimize, Unicorn itself always uses as much
resources as the OS can give it to render.  A lot of slow views I've
seen hit a lot of memory allocation, so check out tcmalloc (included
with REE) or use 1.9 (possibly with tcmalloc, too).  1.9 has some good
optimizations for allocating shorter strings (should be more effective
on 64-bit) and small hashes/arrays, too.

My personal style is also to use destructive methods as much as possible
(concat/<< vs +/+=, map! vs map, gsub! vs gsub, tr! vs tr, etc...).
Avoid creating lots of OpenStruct, too, excessive use of
define_method/metadef allocates a lot of short-lived objects and
thrashes memory.


The timeout in Unicorn affects the entire request (including all I/O).

The timeout in Rainbows only kicks in when the process is completely
deadlocked.

Since apparently lots of "normal" HTTP clients have ridiculous keepalive
times, Rainbows! will be getting a separate keepalive_timeout directive
soon that only affects reading HTTP headers.  I don't plan enforcing a
timeout when processing request bodies or app responses (those can
probably be done on a more controlled manner in the app/middleware).

> Lastly... ¡You rock Eric!

Thanks, but much of the credit goes to random folks all over who
have helped with this (and projects leading up to this), too :>

-- 
Eric Wong

^ permalink raw reply	[relevance 11%]

* rainbows for sleepy/lethargic apps
@ 2009-11-16 19:30 14% Dylan Stamat
  2009-11-16 21:26 11% ` Eric Wong
  0 siblings, 1 reply; 191+ results
From: Dylan Stamat @ 2009-11-16 19:30 UTC (permalink / raw)
  To: mongrel-unicorn

I'm working on a project that has very bad application performance, specifically, tons of long running requests due to view and (specifically) database contention.

While I haven't tested our Unicorn setup under load yet, tweak backlog counts, etc... I'm wondering if I'd be better off giving Rainbows! a shot?
In the Rainbows! AppPool diagram, I'm having a hard time understanding the N:P relationship.  When the P threshold is met, what happens to N (the client)?

Also, with both Unicorn/Rainbows!, is there a explicit timeout number set on the clients (and how does that differ between Unicorn/Rainbows!)?
For instance, in our Nginx config, I have proxy_buffering turned on, with proxy_read_timeout and proxy_send_timeout's, set pretty high.

Lastly... ¡You rock Eric!
==
Dylan

^ permalink raw reply	[relevance 14%]

* [ANN] unicorn 0.95.0 - we <3 Rainbows! and ponies too
@ 2009-11-15 22:34  9% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-11-15 22:34 UTC (permalink / raw)
  To: ruby-talk ML, mongrel-unicorn; +Cc: ruby-talk

Unicorn is a HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

Mostly internal cleanups and documentation updates.  Irrelevant
stacktraces from client disconnects/errors while reading
"rack.input" are now cleared to avoid unnecessary noise.  If
user switching in workers is used, ownership of logs is now
preserved when reopening worker logs (send USR1 only to the the
master in this case).  The timeout config no longer affects long
after_fork hooks or application startups.

New features include the addition of the :umask option for the
"listen" config directive and error reporting for non-portable
socket options.

No ponies have ever been harmed in our development.

Eric Wong (28):
      unicorn.1: document RACK_ENV changes in 0.94.0
      HACKING: update with "gmake" in examples
      don't nuke children for long after_fork and app loads
      local.mk.sample: steal some updates from Rainbows!
      Load Unicorn constants when building app
      tee_input: fix RDoc argument definition for tee
      Add FAQ
      FAQ: fix links to Configurator docs
      tee_input: better premature disconnect handling
      tee_input: don't shadow struct members
      raise Unicorn::ClientShutdown if client aborts in TeeInput
      tee_input: fix comment from an intermediate commit
      FAQ: additional notes on getting HTTPS redirects right
      configurator: update RDoc and comments in examples
      bump version to 0.95.0pre
      configurator: listen :umask parameter for UNIX sockets
      preserve user/group ownership when reopening logs
      old_rails/static: avoid freezing strings
      old_rails: autoload Static
      const: no need to freeze HTTP_EXPECT
      test_server: ensure stderr is written to before reading
      tee_input: expand client error handling
      replace "rescue => e" with "rescue Object => e"
      socket_helper: do not hide errors when setting socket options
      socket_helper: RDoc for constants
      ClientShutdown: RDoc
      Rakefile: add raa_update task
      tee_input: client_error always raises

-- 
Eric Wong



^ permalink raw reply	[relevance 9%]

* Re: dealing with client disconnects with TeeInput
  2009-11-12 10:04  6% dealing with client disconnects with TeeInput Eric Wong
@ 2009-11-14  1:16  0% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-11-14  1:16 UTC (permalink / raw)
  To: mongrel-unicorn, rainbows-talk

Eric Wong <normalperson@yhbt.net> wrote:
> So, would making a Unicorn::Disconnect < EOFError exception class and
> raising it with a short/empty backtrace on EOFErrors be the best way to
> go?  That way those global exception trappers can distinguish between
> EOFError exceptions raised by Unicorn/Rainbows! itself and other code
> that Unicorn/Rainbows does not care about, and log appropriately...

I actually named it Unicorn::ClientShutdown since I figured the
name would be more descriptive.  Here's what I've pushed out
to unicorn.git:

>From e4256da292f9626d7dfca60e08f65651a0a9139a Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Sat, 14 Nov 2009 00:23:19 +0000
Subject: [PATCH] raise Unicorn::ClientShutdown if client aborts in TeeInput

Leaving the EOFError exception as-is bad because most
applications/frameworks run an application-wide exception
handler to pretty-print and/or log the exception with a huge
backtrace.

Since there's absolutely nothing we can do in the server-side
app to deal with clients prematurely shutting down, having a
backtrace does not make sense.  Having a backtrace can even be
harmful since it creates unnecessary noise for application
engineers monitoring or tracking down real bugs.
---
 lib/unicorn.rb           |    6 ++++
 lib/unicorn/tee_input.rb |   11 ++++++++
 test/unit/test_server.rb |   61 ++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 76 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index a696402..c6c311e 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -8,6 +8,12 @@ autoload :Rack, 'rack'
 # a Unicorn web server.  It contains a minimalist HTTP server with just enough
 # functionality to service web application requests fast as possible.
 module Unicorn
+
+  # raise this inside TeeInput when a client disconnects inside the
+  # application dispatch
+  class ClientShutdown < EOFError
+  end
+
   autoload :Const, 'unicorn/const'
   autoload :HttpRequest, 'unicorn/http_request'
   autoload :HttpResponse, 'unicorn/http_response'
diff --git a/lib/unicorn/tee_input.rb b/lib/unicorn/tee_input.rb
index 69397c0..50ddb5b 100644
--- a/lib/unicorn/tee_input.rb
+++ b/lib/unicorn/tee_input.rb
@@ -135,10 +135,21 @@ module Unicorn
         end
       end
       finalize_input
+      rescue EOFError
+        # in case client only did a premature shutdown(SHUT_WR)
+        # we do support clients that shutdown(SHUT_WR) after the
+        # _entire_ request has been sent, and those will not have
+        # raised EOFError on us.
+        socket.close if socket
+        raise ClientShutdown, "bytes_read=#{@tmp.size}", []
     end
 
     def finalize_input
       while parser.trailers(req, buf).nil?
+        # Don't worry about throw-ing :http_499 here on EOFError, tee()
+        # will catch EOFError when app is processing it, otherwise in
+        # initialize we never get any chance to enter the app so the
+        # EOFError will just get trapped by Unicorn and not the Rack app
         buf << socket.readpartial(Const::CHUNK_SIZE)
       end
       self.socket = nil
diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb
index bbb06da..a7f6a35 100644
--- a/test/unit/test_server.rb
+++ b/test/unit/test_server.rb
@@ -12,11 +12,12 @@ include Unicorn
 
 class TestHandler 
 
-  def call(env) 
-  #   response.socket.write("HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nhello!\n")
+  def call(env)
     while env['rack.input'].read(4096)
     end
     [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']]
+    rescue Unicorn::ClientShutdown => e
+      $stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n")
    end
 end
 
@@ -103,6 +104,62 @@ class WebServerTest < Test::Unit::TestCase
     assert_equal 'hello!\n', results[0], "Handler didn't really run"
   end
 
+  def test_client_shutdown_writes
+    sock = nil
+    buf = nil
+    bs = 15609315 * rand
+    assert_nothing_raised do
+      sock = TCPSocket.new('127.0.0.1', @port)
+      sock.syswrite("PUT /hello HTTP/1.1\r\n")
+      sock.syswrite("Host: example.com\r\n")
+      sock.syswrite("Transfer-Encoding: chunked\r\n")
+      sock.syswrite("Trailer: X-Foo\r\n")
+      sock.syswrite("\r\n")
+      sock.syswrite("%x\r\n" % [ bs ])
+      sock.syswrite("F" * bs)
+      sock.syswrite("\r\n0\r\nX-")
+      "Foo: bar\r\n\r\n".each_byte do |x|
+        sock.syswrite x.chr
+        sleep 0.05
+      end
+      # we wrote the entire request before shutting down, server should
+      # continue to process our request and never hit EOFError on our sock
+      sock.shutdown(Socket::SHUT_WR)
+      buf = sock.read
+    end
+    assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last
+    lines = File.readlines("test_stderr.#$$.log")
+    assert lines.grep(/^Unicorn::ClientShutdown: /).empty?
+    assert_nothing_raised { sock.close }
+  end
+
+  def test_client_shutdown_write_truncates
+    sock = nil
+    buf = nil
+    bs = 15609315 * rand
+    assert_nothing_raised do
+      sock = TCPSocket.new('127.0.0.1', @port)
+      sock.syswrite("PUT /hello HTTP/1.1\r\n")
+      sock.syswrite("Host: example.com\r\n")
+      sock.syswrite("Transfer-Encoding: chunked\r\n")
+      sock.syswrite("Trailer: X-Foo\r\n")
+      sock.syswrite("\r\n")
+      sock.syswrite("%x\r\n" % [ bs ])
+      sock.syswrite("F" * (bs / 2.0))
+
+      # shutdown prematurely, this will force the server to abort
+      # processing on us even during app dispatch
+      sock.shutdown(Socket::SHUT_WR)
+      IO.select([sock], nil, nil, 60) or raise "Timed out"
+      buf = sock.read
+    end
+    assert_equal "", buf
+    lines = File.readlines("test_stderr.#$$.log")
+    lines = lines.grep(/^Unicorn::ClientShutdown: bytes_read=\d+/)
+    assert_equal 1, lines.size
+    assert_match %r{\AUnicorn::ClientShutdown: bytes_read=\d+ true$}, lines[0]
+    assert_nothing_raised { sock.close }
+  end
 
   def do_test(string, chunk, close_after=nil, shutdown_delay=0)
     # Do not use instance variables here, because it needs to be thread safe
-- 
Eric Wong

^ permalink raw reply related	[relevance 0%]

* [ANN] unicorn 0.94.0 - small fixes and new features
@ 2009-11-05 10:06  3% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-11-05 10:06 UTC (permalink / raw)
  To: mongrel-unicorn

Unicorn is a HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

The HTTP parser is fix for oddly-aligned reads of trailers (this
technically affects headers, too, but is highly unlikely due to
our non-support of slow clients).  This allows our HTTP parser
to better support very slow clients when used by other servers
(like Rainbows!).  Fortunately this bug does not appear to lead
to any invalid memory accesses (and potential arbitrary code
execution).

FreeBSD (and possibly other *BSDs) support is improved and and
all the test cases pass under FreeBSD 7.2.  Various flavors of
GNU/Linux remains our primary platform for development and
production.

New features added include the "working_directory" directive in
the configurator .  Even without specifying a
"working_directory", symlink-aware detection of the current path
no longer depends on /bin/sh so it should work out-of-the-box on
FreeBSD and Solaris and not just systems where /bin/sh is dash,
ksh93 or bash.

User-switching support is finally supported but only intended
for use in the after_fork hook of worker processes.  Putting it
in the after_fork hook allows allows users to set things like
CPU affinity[1] on a per-worker basis before dropping
privileges.  The master process retains all privileges it
started with.

The ENV["RACK_ENV"] (process-wide) environment variable is now
both read and set for `unicorn' in the same way RAILS_ENV is
used by `unicorn_rails'.  This allows the Merb launcher to read
ENV["RACK_ENV"] in config.ru.  Other web servers already set
this and there may be applications or libraries that already
rely on this de facto standard.

Eric Wong (26):
      cleanup: avoid redundant error checks for fstat
      test_helper: connect(2) may fail with EINVAL
      GNUmakefile: fix non-portable tar(1) usage
      tests: provide a pure Ruby setsid(8) equivalent
      more portable symlink awareness for START_CTX[:cwd]
      test_signals: avoid portability issues with fchmod(2)
      cleanup error handling and make it less noisy
      Do not override Dir.chdir in config files
      configurator: add "working_directory" directive
      configurator: working_directory is expanded
      configurator: set ENV["PWD"] with working_directory, too
      configurator: working_directory affects pid, std{err,out}_paths
      configurator: update documentation for working_directory
      TODO: remove working_directory bit, done
      Util.reopen_logs: remove needless Range
      worker: user/group switching for after_fork hooks
      Fix autoload of Etc in Worker for Ruby 1.9
      bin/unicorn: allow RACK_ENV to be passed from parent
      tests for RACK_ENV preservation
      http: allow headers/trailers to be written byte-wise
      http: extra test for bytewise chunked bodies
      tee_input: do not clobber trailer buffer on partial uploads
      test_exec: ensure master is killed after test
      Util::tmpio returns a TmpIO that responds to #size
      TODO: remove user-switching bit, done
      unicorn 0.94.0

Wayne Larsen (1):
      bin/unicorn: set ENV["RACK_ENV"] on startup

[1] - Unicorn does not support CPU affinity directly, but it is
      possible to load code that allows it inside after_fork hooks,
      or even just call sched_tool(8).

-- 
Eric Wong

^ permalink raw reply	[relevance 3%]

* Re: workers does not seems to exit when served one client
  @ 2009-11-03 16:56  6% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-11-03 16:56 UTC (permalink / raw)
  To: unicorn list

HaiMing Yin <epaulin@gmail.com> wrote:
> quote from http://unicorn.bogomips.org/:
> 
> {{{
> workers all run within their own isolated address space and only serve
> one client at a time for maximum robustness.
> }}}
> 
> top shows:
> 
> {{{
> 25398 www-data  20   0  268m 137m 3364 S    0  1.7  27:49.41
> unicorn_rails
> 25400 www-data  20   0  266m 135m 3364 S    0  1.7  25:24.31
> unicorn_rails
> 25397 www-data  20   0  265m 134m 3364 S    0  1.7  31:50.72 unicorn_rails
> }}}
> 
> pid are the same since unicorn_rails worker started, and the RES and
> TIME+ column clearly shows that unicorn_rails did not exit after
> served one client.
> 
> What I'm doing wrong?

Hi HaiMing,

You're doing nothing wrong except misunderstanding that phrase.

"one client at a time" means it's not possible to be servicing more
clients than there are worker_processes (the kernel will buffer them).

Basically your workers <=> clients mapping will look like this with
Unicorn:

     unicorn master
     \_ unicorn worker[0]
     |  \_ client[0]
     \_ unicorn worker[1]
     |  \_ client[1]
     \_ unicorn worker[2]
     |  \_ client[2]
     ...
     \_ unicorn worker[M]
        \_ client[M]


Where in other servers (such as Rainbows!) it'll look like this
with multiple clients running under one worker:

     rainbows master
     \_ rainbows worker[0]
     |  \_ client[0,0]
     |  \_ client[0,1]
     |  \_ client[0,2]
     |  ...
     |  \_ client[0,N]
     ...
     \_ rainbows worker[M]
        \_ client[M,0]
        \_ client[M,1]
        \_ client[M,2]
        ...
        \_ client[M,N]

-- 
Eric Wong

^ permalink raw reply	[relevance 6%]

* Unicorn/Rainbows! mailing list migration
@ 2009-10-29 22:44 11% Eric Wong
       [not found]     ` <20091029224426.GA12314-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2009-10-29 22:44 UTC (permalink / raw)
  To: mongrel-unicorn, rainbows-talk; +Cc: meta

As I'm sure you've all heard by now, RubyForge will be moving to a
read-only state and we'll have to migrate our mailing lists somewhere.

== librelist

I'm leaning strongly towards librelist.com.  I've had good, but limited
experiences with it.  I like that librelist has no logins or accounts to
deal with; just email the list you want to join and reply to the
automated message to subscribe.  It also has downloadable archives
via rsync.

librelist is run by Zed Shaw, the same guy who started Mongrel (which
eventually led us to Unicorn).  While I don't believe in giving any
single entity too much power and influence, Zed's done a lot of good for
Mongrel and handed it off gracefully when he didn't want to work on it
anymore.  For these reason, I think librelist will be in good hands for
years to come.

I probably would've picked librelist if it had been around when I
started this RubyForge list in May 2009.  I started Rainbows! more
recently, but continued using RubyForge because I wanted everything
Rainbows!-related on RubyForge.org (oops!).


== Other options

Google Groups is out.  Spam has been a big problem in the few groups
I've joined and I'd rather trust Zed than Google for mailing lists.
Also I don't know of a way to get archives from Google without scraping.

SourceForge - I've had spam problems in the past, mailman...

Savannah - no idea about the quality, but uses mailman...

I've also considered hosting the list ourselves, but that might be
too much work and spreading ourselves too thin at the moment.


Any other list hosting options out there we should consider?  It must be
a mailing list that does not mangle plain-text patches.

Or any votes in favor of migrating to librelist sooner rather than
later?  I'll work on getting the gmane NNTP mirrors moved over as
well.


== Other stuff related to RubyForge going read-only...

Gem hosting will (obviously) be taken care of by our new
Gemcut^WRubygems.org overlords.

The tarballs for non-RubyGems users will probably just be moved to a
files/ section of the websites; it'll be even be easier to find and
download tarballs in automated scripts that way, too.

Since we're using git, our repositories are mirrored everywhere already
and was never centrally-hosted on RubyForge in the first place.

http://rainbows.rubyforge.org/ will probably be redirected to
http://rainbows.bogomips.org/ (same box as the Unicorn website).

Should you ever want to access either of the sites offline, just make
sure you have RDoc 2.4.x ("gem install rdoc" if you're on 1.8) and run
`make doc' or `gmake doc' in the latest source tree.

-- 
Eric Wong

^ permalink raw reply	[relevance 11%]

* re:  multi server failover setup
@ 2009-10-26 19:34  4% Dylan Stamat
  0 siblings, 0 replies; 191+ results
From: Dylan Stamat @ 2009-10-26 19:34 UTC (permalink / raw)
  To: mongrel-unicorn

Eric, have you heard of anybody using the idea you laid out here?
http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31

I really like the approach, and hoping I'm not going to be the first  
one trying it ;)

I noticed that Mr. Wanstrath is using UDS in his deployment here:  http://github.com/blog/517-unicorn
Wondering if they're running with the same or similar strategy as your  
suggestion?

Our application is long-running-request heavy, so we may actually be a  
Rainbows! candidate.

Thanks!
==
Dylan

^ permalink raw reply	[relevance 4%]

* Re: RAILS_ROOT and USR2 restarts
  2009-10-26 16:01  4% ` Eric Wong
@ 2009-10-26 16:13  0%   ` Ian Leitch
  0 siblings, 0 replies; 191+ results
From: Ian Leitch @ 2009-10-26 16:13 UTC (permalink / raw)
  To: unicorn list

Apologies for the noise, I discovered START_CTX[:cwd] and that fixed it.

Solaris sh is indeed the culprit:

$ cd /var/www/apps/production/current
(admin@production:/var/www/apps/production/current)
$ sh -c pwd
/var/www/apps/production/releases/20091026160154

Thanks for creating Unicorn!


2009/10/26 Eric Wong <normalperson@yhbt.net>:
> Ian Leitch <port001@gmail.com> wrote:
>> Hi,
>>
>> I've just started using Unicorn in production behind Nginx with a
>> Rails app, for the most part its been a smooth transition apart from
>> this one issue.
>> When I do an in place restart (USR2), the restart process works but
>> the workers load my old Rails release, not the new one. I'm guessing
>> that because the master process is currently in the old release
>> directory, when I signal USR2 it's just reloading inside the current
>> directory. I'm deploying with Capistrano, so app/current is a symlink
>> to a specific release. I've tried changing to the new release
>> directory in before_work but that made no difference. Is there a
>> callback that fires before the master preloads the app?
>
> Hi Ian,
>
> Unicorn chdirs to the directory returned by `/bin/sh -c pwd` when it was
> originally started.  The output of `pwd` *should* be symlink-aware, but
> then Solaris /bin/sh is weird.  Can you confirm that it's broken?
>
> You can put the following in your config (outside of the hooks):
>
>  app_root = "/var/www/apps/systino_production/current"
>  Dir.chdir(Unicorn::HttpServer::START_CTX[:cwd] = app_root)
>
> I've been pondering adding a "working_directory" directive to
> Configurator, too.  However the START_CTX hash is considered a stable
> interface and I recently documented it in
> http://unicorn.bogomips.org/Unicorn.html
>
> START_CTX even allows you to switch between different installation paths
> for Unicorn[1] and alter command line options that were originally
> passed.  Lots of rope there :>
>
> [1] for different Ruby installs/versions,
>    or even swap in Rainbows! or vice versa
>
> --
> Eric Wong
> _______________________________________________
> mongrel-unicorn mailing list
> mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
>
_______________________________________________
mongrel-unicorn mailing list
mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn

^ permalink raw reply	[relevance 0%]

* Re: RAILS_ROOT and USR2 restarts
  @ 2009-10-26 16:01  4% ` Eric Wong
  2009-10-26 16:13  0%   ` Ian Leitch
  0 siblings, 1 reply; 191+ results
From: Eric Wong @ 2009-10-26 16:01 UTC (permalink / raw)
  To: unicorn list

Ian Leitch <port001@gmail.com> wrote:
> Hi,
> 
> I've just started using Unicorn in production behind Nginx with a
> Rails app, for the most part its been a smooth transition apart from
> this one issue.
> When I do an in place restart (USR2), the restart process works but
> the workers load my old Rails release, not the new one. I'm guessing
> that because the master process is currently in the old release
> directory, when I signal USR2 it's just reloading inside the current
> directory. I'm deploying with Capistrano, so app/current is a symlink
> to a specific release. I've tried changing to the new release
> directory in before_work but that made no difference. Is there a
> callback that fires before the master preloads the app?

Hi Ian,

Unicorn chdirs to the directory returned by `/bin/sh -c pwd` when it was
originally started.  The output of `pwd` *should* be symlink-aware, but
then Solaris /bin/sh is weird.  Can you confirm that it's broken?

You can put the following in your config (outside of the hooks):

  app_root = "/var/www/apps/systino_production/current"
  Dir.chdir(Unicorn::HttpServer::START_CTX[:cwd] = app_root)

I've been pondering adding a "working_directory" directive to
Configurator, too.  However the START_CTX hash is considered a stable
interface and I recently documented it in
http://unicorn.bogomips.org/Unicorn.html

START_CTX even allows you to switch between different installation paths
for Unicorn[1] and alter command line options that were originally
passed.  Lots of rope there :>

[1] for different Ruby installs/versions,
    or even swap in Rainbows! or vice versa

-- 
Eric Wong

^ permalink raw reply	[relevance 4%]

* [ANN] unicorn 0.93.3 - OpenBSD compatibility
@ 2009-10-09 23:28  4% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-10-09 23:28 UTC (permalink / raw)
  To: mongrel-unicorn

Unicorn is a HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

This release fixes compatibility with OpenBSD (and possibly
other Unices with stricter fchmod(2) implementations) thanks to
Jeremy Evans.  Additionally there are small documentation
changes all around.

Eric Wong (12):
      doc: expand on the SELF_PIPE description
      fchmod heartbeat flips between 0/1 for compatibility
      examples/init.sh: remove "set -u"
      configurator: update with nginx fail_timeout=0 example
      PHILOSOPHY: clarify experience other deployments
      PHILOSOPHY: plug the Rainbows! spin-off project
      README: remove unnecessary and extraneous dash
      DESIGN: clarification and possibly improve HTML validity
      README: remove the "non-existent" part
      README: emphasize the "fast clients"-only part
      drop the whitespace cleaner for Ragel->C
      unicorn 0.93.3
-- 
Eric Wong

^ permalink raw reply	[relevance 4%]

* Re: alive.chmod(Time.now.to_i) raises Errno::EINVAL on OpenBSD
  @ 2009-10-09  3:22  3% ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-10-09  3:22 UTC (permalink / raw)
  To: Jeremy Evans; +Cc: mongrel-unicorn

Jeremy Evans <jeremyevans0@gmail.com> wrote:
> On OpenBSD:
> 
> $ ruby -e "File.new('TODO').chmod(Time.now.to_i)"
> -e:1:in `chmod': Invalid argument - TODO (Errno::EINVAL)
>         from -e:1
> 
> This is explained in the man page:
> 
>      int
>      fchmod(int fd, mode_t mode);
> 
>     ...
> 
>      [EINVAL]      mode contains bits other than the file type and those de-
>                    scribed above.
> 
> I think 04777 is the highest allowed mode on OpenBSD.  Time.now.to_i
> is obviously higher than that.

Yikes.  I was worried about the portability of this.  I actually redid
this in the Revactor model of Rainbows! to flip between 0 and 1, I'll do
that in Unicorn, too.

> Here's a diff that should fix the problem.  At the very least it
> allows the workers to start without crashing:
> 
> diff --git a/lib/unicorn.rb b/lib/unicorn.rb
> index ddec8e9..092f500 100644
> --- a/lib/unicorn.rb
> +++ b/lib/unicorn.rb
> @@ -579,13 +579,13 @@ module Unicorn
>          # changes with chmod doesn't update ctime on all filesystems; so
>          # we change our counter each and every time (after process_client
>          # and before IO.select).
> -        t == (ti = Time.now.to_i) or alive.chmod(t = ti)
> +        t == (ti = Time.now.to_i) or (t = ti;
> alive.chmod(alive.stat.mode ^ 0100))

I think the Time.now.to_i bit is overkill (and probably detrimental to
performance on non-x86_64, non-VDSO-enabled, non-GNU/Linux machines).

> There are definitely other ways that will work, as long as the mode is
> kept between 0 and 04777.

Here's what I committed and pushed out, it should work for all *nix
now since it only alternates between 0 and 1, but be sure to let
me know if it doesn't for some reason.  Thanks!

>From 24a1b4c6b5fcb948e4fdee04e286c044d6d45f98 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Thu, 8 Oct 2009 19:56:29 -0700
Subject: [PATCH] fchmod heartbeat flips between 0/1 for compatibility

This removes the Time.now.to_i comparison that was used to avoid
multiple, no-op fchmod() syscalls[1] within the same second.

This should allow us to run on OpenBSD where it can raise EINVAL
when Time.now.to_i is passed to it.

Reported-by: Jeremy Evans <jeremyevans0@gmail.com>

[1] - gettimeofday() from Time.now is not a real syscall on
VDSO-enabled x86_64 GNU/Linux systems where Unicorn is primarily
developed.
---
 lib/unicorn.rb |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index d63567f..13c203a 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -575,13 +575,13 @@ module Unicorn
       nr = 0 # this becomes negative if we need to reopen logs
       alive = worker.tmp # tmp is our lifeline to the master process
       ready = LISTENERS
-      t = ti = 0
 
       # closing anything we IO.select on will raise EBADF
       trap(:USR1) { nr = -65536; SELF_PIPE.first.close rescue nil }
       trap(:QUIT) { alive = nil; LISTENERS.each { |s| s.close rescue nil } }
       [:TERM, :INT].each { |sig| trap(sig) { exit!(0) } } # instant shutdown
       logger.info "worker=#{worker.nr} ready"
+      m = 0
 
       begin
         nr < 0 and reopen_worker_logs(worker.nr)
@@ -595,13 +595,13 @@ module Unicorn
         # changes with chmod doesn't update ctime on all filesystems; so
         # we change our counter each and every time (after process_client
         # and before IO.select).
-        t == (ti = Time.now.to_i) or alive.chmod(t = ti)
+        alive.chmod(m = 0 == m ? 1 : 0)
 
         ready.each do |sock|
           begin
             process_client(sock.accept_nonblock)
             nr += 1
-            t == (ti = Time.now.to_i) or alive.chmod(t = ti)
+            alive.chmod(m = 0 == m ? 1 : 0)
           rescue Errno::EAGAIN, Errno::ECONNABORTED
           end
           break if nr < 0
@@ -614,7 +614,7 @@ module Unicorn
         redo unless nr == 0 # (nr < 0) => reopen logs
 
         ppid == Process.ppid or return
-        alive.chmod(t = 0)
+        alive.chmod(m = 0 == m ? 1 : 0)
         begin
           # timeout used so we can detect parent death:
           ret = IO.select(LISTENERS, nil, SELF_PIPE, timeout) or redo
-- 
Eric Wong

^ permalink raw reply related	[relevance 3%]

* [ANN] unicorn 0.93.2 - more compatible with Rails
@ 2009-10-07  8:58  5% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-10-07  8:58 UTC (permalink / raw)
  To: mongrel-unicorn

Unicorn is a HTTP server for Rack applications designed to take
advantage of features in Unix/Unix-like kernels and only serve fast
clients on low-latency, high-bandwidth connections.  Slow clients should
only be served by placing a reverse proxy capable of fully-buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn@rubyforge.org
* git://git.bogomips.org/unicorn.git

Thanks to Chris Wanstrath for reporting issues with large
POST bodies and for helping me test.

Changes:

Avoid truncated POST bodies with URL-encoded forms in Rails
by switching TeeInput to use read-in-full semantics (only) when
a Content-Length: header exists.  Chunked request bodies
continue to exhibit readpartial semantics to support
simultaneous bidirectional chunking.

The lack of return value checking in Rails to protect against a
short ios.read(length) is entirely reasonable even if not
pedantically correct.  Most ios.read(length) implementations
return the full amount requested except right before EOF.

A ticket has been opened here to track the issue:
  https://rails.lighthouseapp.com/projects/8994/tickets/3343

Also there are some minor documentation improvements.

Eric Wong (8):
      Fix NEWS generation on single-paragraph tag messages
      Include GPLv2 in docs
      doc: make it clear contributors retain copyrights
      TODO: removed Rainbows! (see rainbows.rubyforge.org)
      Document the START_CTX hash contents
      more-compatible TeeInput#read for POSTs with Content-Length
      tests for read-in-full vs readpartial semantics
      unicorn 0.93.2
-- 
Eric Wong

^ permalink raw reply	[relevance 5%]

* Re: POST Body Truncated
  @ 2009-10-07  3:04  3%                   ` Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-10-07  3:04 UTC (permalink / raw)
  To: Chris Wanstrath; +Cc: mongrel-unicorn

Chris Wanstrath <chris@ozmm.org> wrote:
> On Tue, Oct 6, 2009 at 3:52 PM, Eric Wong <normalperson@yhbt.net> wrote:
> 
> > OK, here's a workaround that should work for now.  I have to hit the
> > road in a few minutes but will be back on a computer in a few hours.
> >
> > This only affects older Rails (and I'm supposed to still be supporting
> > 1.2.x!) and its interaction with a wrapped CGI.stdinput somewhere is
> > going bad...
> 
> This works!
> 
> I suppose we should upgrade to a newer Rails :)
> 
> As usual, thanks a million.

Here's a real patch with lots of documentation I just pushed out, still
working on automated test cases.  Can you let me know how it works?
Thanks.

>From 438c99aec2d74489fa89b3a6c60d1fb41bb2f7e6 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Tue, 6 Oct 2009 19:45:05 -0700
Subject: [PATCH] more-compatible TeeInput#read for POSTs with Content-Length

There are existing applications and libraries that don't check
the return value of env['rack.input'].read(length) (like Rails
:x).  Those applications became broken under the IO#readpartial
semantics of TeeInput#read when handling larger request bodies.

We'll preserve the IO#readpartial semantics _only_ when handling
chunked requests (as long as Rack allows it, it's useful for
real-time processing of audio/video streaming uploads,
especially with Rainbows! and mobile clients) but use
read-in-full semantics for TeeInput#read on requests with a
known Content-Length.
---
 lib/unicorn/tee_input.rb |   43 +++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 41 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/tee_input.rb b/lib/unicorn/tee_input.rb
index 96a053a..188e2ea 100644
--- a/lib/unicorn/tee_input.rb
+++ b/lib/unicorn/tee_input.rb
@@ -41,6 +41,26 @@ module Unicorn
       @size = tmp_size
     end
 
+    # call-seq:
+    #   ios = env['rack.input']
+    #   ios.read([length [, buffer ]]) => string, buffer, or nil
+    #
+    # Reads at most length bytes from the I/O stream, or to the end of
+    # file if length is omitted or is nil. length must be a non-negative
+    # integer or nil. If the optional buffer argument is present, it
+    # must reference a String, which will receive the data.
+    #
+    # At end of file, it returns nil or "" depend on length.
+    # ios.read() and ios.read(nil) returns "".
+    # ios.read(length [, buffer]) returns nil.
+    #
+    # If the Content-Length of the HTTP request is known (as is the common
+    # case for POST requests), then ios.read(length [, buffer]) will block
+    # until the specified length is read (or it is the last chunk).
+    # Otherwise, for uncommon "Transfer-Encoding: chunked" requests,
+    # ios.read(length [, buffer]) will return immediately if there is
+    # any data and only block when nothing is available (providing
+    # IO#readpartial semantics).
     def read(*args)
       socket or return @tmp.read(*args)
 
@@ -55,9 +75,9 @@ module Unicorn
         rv = args.shift || @buf2.dup
         diff = tmp_size - @tmp.pos
         if 0 == diff
-          tee(length, rv)
+          ensure_length(tee(length, rv), length)
         else
-          @tmp.read(diff > length ? length : diff, rv)
+          ensure_length(@tmp.read(diff > length ? length : diff, rv), length)
         end
       end
     end
@@ -130,5 +150,24 @@ module Unicorn
       StringIO === @tmp ? @tmp.size : @tmp.stat.size
     end
 
+    # tee()s into +buf+ until it is of +length+ bytes (or until
+    # we've reached the Content-Length of the request body).
+    # Returns +buf+ (the exact object, not a duplicate)
+    # To continue supporting applications that need near-real-time
+    # streaming input bodies, this is a no-op for
+    # "Transfer-Encoding: chunked" requests.
+    def ensure_length(buf, length)
+      # @size is nil for chunked bodies, so we can't ensure length for those
+      # since they could be streaming bidirectionally and we don't want to
+      # block the caller in that case.
+      return buf if buf.nil? || @size.nil?
+
+      while buf.size < length && @size != @tmp.pos
+        buf << tee(length - buf.size, @buf2)
+      end
+
+      buf
+    end
+
   end
 end
-- 
Eric Wong

^ permalink raw reply related	[relevance 3%]

* [ANN] Rainbows! 0.1.0 initial release
@ 2009-10-05 11:00 15% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-10-05 11:00 UTC (permalink / raw)
  To: mongrel-unicorn

Rainbows! Unicorn for slow apps and slow clients

Rainbows! is a HTTP server for Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.  For Rack applications not
heavily bound by slow external dependencies, consider Unicorn instead as
it simpler and easier to debug.

Rainbows! includes previous work from the "gossamer" and "rainbows"
branches of Unicorn and will support additional network concurrency
models.  It is basically for all the things that Unicorn sucks
at :)

* http://rainbows.rubyforge.org/
* rainbows-talk@rubyforge.org
* git://git.bogomips.org/rainbows.git

Changes:

Initial release

This release is currently highly experimental and is still
missing a lot of test coverage.

-- 
Eric Wong

^ permalink raw reply	[relevance 15%]

* Rainbows...
@ 2009-09-16 21:11 12% Eric Wong
  0 siblings, 0 replies; 191+ results
From: Eric Wong @ 2009-09-16 21:11 UTC (permalink / raw)
  To: mongrel-unicorn

Some of you may have noticed the "rainbows" branch of the git
repository.  I may be changing that however and making a small group of
gems (or even one gem) that can get loaded at runtime and monkey patch
parts of Unicorn.

Rainbows will primarily be to support things Unicorn sucks at:

  1. apps that rely on out-of-datacenter network connections
     (CAPTCHA services, OpenID, real-time feed aggregation, etc...)

  2. Comet / long-polling / reverse HTTP

Since the majority of Unicorn use cases (or even requests within an
application) do not need these things, I'm hesitant to make Unicorn
itself more complicated and more difficult to support for the majority
of apps.  Instead, I'm leaning towards putting that burden on Rainbows
for the applications that absolutely need it.

For 1), the apps I've seen that rely on out-of-datacenter network
connections don't use them for the large majority (>= 95%) of HTTP
requests, so the overall application with a few poorly-performing
application endpoints were fine and predictable even with basic Unicorn.

Of course, some folks I know want to make a proxy with Unicorn and
thats why I starting working on it.

As far as concurrency models go, forked workers sharing listener sockets
will continue to be used to better exploit CPU/memory concurrency[1].
However, each worker process will also support Threads or Actors so
there'll be a more flexible M:N mapping of processes:clients instead of
the 1:1 mapping Unicorn uses.

Rev and EventMachine are being considered, too, but mapping those
programming models to TeeInput will be more work...

[1] Even on Ruby implementations without a big global lock for threads,
    forked workers that don't have to share large amounts of memory
    on big SMP boxes, so they'll experience less memory/cache contention
    and should experience better performance as a result.

-- 
Eric Wong

^ permalink raw reply	[relevance 12%]

Results 1-191 of 191 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2009-09-16 21:11 12% Rainbows Eric Wong
2009-10-05 11:00 15% [ANN] Rainbows! 0.1.0 initial release Eric Wong
2009-10-06 21:13     POST Body Truncated Chris Wanstrath
2009-10-06 21:22     ` Eric Wong
2009-10-06 21:26       ` Chris Wanstrath
2009-10-06 21:30         ` Eric Wong
2009-10-06 21:46           ` Chris Wanstrath
2009-10-06 21:50             ` Eric Wong
2009-10-06 21:58               ` Chris Wanstrath
2009-10-06 22:32                 ` Eric Wong
2009-10-06 22:52                   ` Eric Wong
2009-10-06 22:57                     ` Chris Wanstrath
2009-10-07  3:04  3%                   ` Eric Wong
2009-10-07  8:58  5% [ANN] unicorn 0.93.2 - more compatible with Rails Eric Wong
2009-10-08 23:57     alive.chmod(Time.now.to_i) raises Errno::EINVAL on OpenBSD Jeremy Evans
2009-10-09  3:22  3% ` Eric Wong
2009-10-09 23:28  4% [ANN] unicorn 0.93.3 - OpenBSD compatibility Eric Wong
2009-10-26 15:04     RAILS_ROOT and USR2 restarts Ian Leitch
2009-10-26 16:01  4% ` Eric Wong
2009-10-26 16:13  0%   ` Ian Leitch
2009-10-26 19:34  4% multi server failover setup Dylan Stamat
2009-10-29 22:44 11% Unicorn/Rainbows! mailing list migration Eric Wong
     [not found]     ` <20091029224426.GA12314-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2009-10-30 22:17  7%   ` Eric Wong
2009-11-03  9:44     workers does not seems to exit when served one client HaiMing Yin
2009-11-03 16:56  6% ` Eric Wong
2009-11-05 10:06  3% [ANN] unicorn 0.94.0 - small fixes and new features Eric Wong
2009-11-12 10:04  6% dealing with client disconnects with TeeInput Eric Wong
2009-11-14  1:16  0% ` Eric Wong
2009-11-15 22:34  9% [ANN] unicorn 0.95.0 - we <3 Rainbows! and ponies too Eric Wong
2009-11-16 19:30 14% rainbows for sleepy/lethargic apps Dylan Stamat
2009-11-16 21:26 11% ` Eric Wong
2009-11-19  7:49     RE:Nginx Conf huet bartels
2009-11-22 22:53     ` Nginx Conf Eric Wong
2009-11-23  3:57  4%   ` Dylan Stamat
2009-11-24  7:19  0%     ` Eric Wong
2009-12-09 21:05     Expect 100-continue errors Jan Dvorak
2009-12-09 21:54     ` David Palm
2009-12-09 22:20  6%   ` Eric Wong
2009-12-09 22:16  4% ` Eric Wong
2009-12-09 23:46  0%   ` Jan Dvorak
2009-12-26  4:29     "unicorn -D" always returns 0 "success" (even when failed to load) Iñaki Baz Castillo
2009-12-26  6:16     ` Eric Wong
2009-12-26 15:47       ` Iñaki Baz Castillo
2009-12-26 18:23         ` Iñaki Baz Castillo
2009-12-27  1:31  2%       ` Eric Wong
2009-12-27  3:06             ` Iñaki Baz Castillo
2009-12-28  3:29  5%           ` Eric Wong
2009-12-28 10:39  0%             ` Iñaki Baz Castillo
2010-01-05 10:26     stderr_path doesn't work at the moment of being called Iñaki Baz Castillo
2010-01-05 10:32     ` Iñaki Baz Castillo
2010-01-05 21:47  4%   ` Eric Wong
2010-01-05 22:50  0%     ` Iñaki Baz Castillo
2010-02-13 10:34 11% [ANN] unicorn 0.96.1 - leak fix for Rainbows!/Zbatery Eric Wong
2010-04-08 21:21     funky process tree + stillborn masters Jamie Wilkinson
2010-04-08 23:55  4% ` Eric Wong
2010-04-09  1:20 14%   ` Eric Wong
2010-04-09  2:37  5% [ANN] raindrops - real-time stats for preforking Rack servers Eric Wong
2010-04-16 18:49     Using HTTP/1.1 and permanent connections Iñaki Baz Castillo
2010-04-16 20:09  6% ` Eric Wong
2010-04-17 22:53  0%   ` Iñaki Baz Castillo
2010-04-19 21:31 11% [ANN] unicorn 0.97.1 - fix HTTP parser for Rainbows!/Zbatery Eric Wong
2010-04-26  8:18  6% Shared memory between workers Iñaki Baz Castillo
2010-04-26 19:03  0% ` Eric Wong
2010-05-25 18:53     Forking off the unicorn master process to create a background worker Russell Branca
2010-05-26 21:05     ` Eric Wong
2010-06-15 17:55       ` Russell Branca
2010-06-15 22:14  4%     ` Eric Wong
2010-06-15 22:51  0%       ` Russell Branca
2010-06-16  0:06  0%         ` Eric Wong
2010-06-08  9:51     [ANN] unicorn 0.990.0 - inching towards 1.0 Eric Wong
2010-06-11 20:37  4% ` Eric Wong
2010-06-09 12:58     Working directory and config.ru Pierre Baillet
2010-06-09 18:04     ` Eric Wong
2010-06-10  9:58  8%   ` Eric Wong
2010-06-14 19:46  6% Unicorn future plans Eric Wong
2010-06-14 20:58  0% ` John-Paul Bader
2010-06-14 22:10  3%   ` Eric Wong
2010-06-16  0:09     anything left before 1.0? Eric Wong
2010-06-16  1:05     ` Andrew Grim
2010-06-16  2:27  3%   ` Eric Wong
2010-06-17 10:08  5% [ANN] unicorn 1.0.0 - yes, this is a real project Eric Wong
2010-06-18  1:47  0% ` Augusto Becciu
2010-06-23  5:57     Purpose of "Status" header in HTTP responses? Craig Davey
2010-06-23  9:07  6% ` Eric Wong
2010-07-08  8:04  4% [ANN] unicorn 1.1.0 - small changes and cleanups Eric Wong
     [not found]     ` <20100708080457.GA2345-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2010-07-11  3:05  7%   ` [ANN] unicorn 1.1.1 - fixing cleanups gone bad :x Eric Wong
2010-08-09 22:54  6% [ANN] 1.9 users: socket_dontwait - MSG_DONTWAIT socket methods Eric Wong
2010-08-15 15:50     unicorn behind apache with file uploads Surendra Singhi
2010-08-15 22:56  6% ` Eric Wong
2010-08-17  1:29  4% running unicorn under rvm Joseph McDonald
2010-08-17  3:52  4% ` Eric Wong
2010-08-17 22:37  0%   ` Joseph McDonald
2010-08-18  1:22  0%   ` Joseph McDonald
2010-08-28 19:57  4% [ANN] unicorn 1.1.3 - small fixes Eric Wong
2010-09-28  3:03  7% [ANN] kgio library / RubyGem Eric Wong
2010-11-15 18:41  0% ` Eric Wong
2010-12-22 11:24  0% ` Iñaki Baz Castillo
2010-10-03 22:03     Unicorn Signal Handling Shared for Other Servers Ben Curren
2010-10-04  5:41  3% ` Eric Wong
2010-10-06  1:53  6% [ANN] unicorn 2.0.0pre1 - a boring "major" release Eric Wong
2010-10-08 21:37  5% internal API changes in Unicorn 2.x Eric Wong
2010-10-22 19:50     502 bad gateway on nginx with recv() failed Naresh V
2010-10-22 21:14  4% ` Eric Wong
2010-10-23  4:48  0%   ` Naresh V
2010-10-23 23:22  0%     ` Eric Wong
2010-10-24  6:00  0%       ` Naresh V
2010-10-28  0:19  5% [ANN] Unicorn 2.0.0, 1.1.5, 1.0.2 released! Eric Wong
2010-11-17  0:26  4% [ANN] unicorn 3.0.0pre1 - disable rewindable input! Eric Wong
2010-11-20  2:47  4% [ANN] unicorn 3.0.0 " Eric Wong
2010-11-20 17:50  0% ` Michael Guterl
2010-11-25 23:06     Dedicated queues for long-running requests with single unicorn master: question and a possible solution Alexander Dymo
2010-11-26  0:40  6% ` Eric Wong
2010-12-03  1:29 14% [ANN] unicorn 3.0.1 - one bugfix for Rainbows! Eric Wong
2010-12-03  1:30 14% ` [ANN] Rainbows! 2.0.1 - upload pipelining fixes Eric Wong
2010-12-26  8:29 13% [ANN] unicorn 3.2.1 - parser improvements for Rainbows! Eric Wong
2011-01-05 23:57  4% [ANN] unicorn 3.3.0 - minor optimizations Eric Wong
2011-01-08  2:00     Thread.current Jimmy Soho
2011-01-08  3:09     ` Thread.current Curtis j Schofield
2011-01-08  5:54       ` Thread.current Jimmy Soho
2011-01-11 22:52         ` Thread.current Jimmy Soho
2011-01-11 23:12  6%       ` Thread.current Eric Wong
2011-01-11 23:12           ` Thread.current Jordan Ritter
2011-01-12  3:07             ` Thread.current Jimmy Soho
2011-01-13  4:26  6%           ` Thread.current Eric Wong
2011-01-13 16:46  0%             ` Thread.current Jordan Ritter
2011-01-25 22:09  8% [PATCH] examples/nginx.conf: use try_files directive Eric Wong
2011-03-08  4:40     [RFC/PATCH] Bundler/Sandbox documentation updates Eric Wong
2011-03-09 10:39     ` Lawrence Pit
2011-03-10  0:52       ` Eric Wong
2011-03-10  3:30         ` Lawrence Pit
2011-03-10 21:29  6%       ` Eric Wong
2011-03-18 18:09  4% [ANN] raindrops updates (mainly for Linux users) Eric Wong
2011-03-23 18:12  0% ` Troex Nevelin
2011-04-12 16:13     Struggling with logrotate and unicorn Emmanuel Gomez
2011-04-12 17:58     ` Eric Wong
2011-04-12 18:36       ` Emmanuel Gomez
2011-04-12 18:59  4%     ` Eric Wong
2011-04-21  6:56     [ANN] unicorn 3.6.0 - small fixes, PRNG workarounds Eric Wong
2011-04-26 22:38     ` ghazel
2011-04-26 23:01  7%   ` Eric Wong
2011-04-27 18:13  4% Rails streaming example video Eric Wong
2011-04-27 21:12  4% ` [PATCH 0/2] attempting to clarify docs for streaming Eric Wong
2011-04-27 21:12 10%   ` [PATCH 1/2] examples/nginx.conf: clarify proxy_buffering for Rails 3.1 Eric Wong
2011-04-27 21:12 13%   ` [PATCH 2/2] configurator: attempt to clarify :tcp_nopush/:tcp_nodelay Eric Wong
2011-05-05 19:48  7% kgio 2.4.0 coming soon Eric Wong
2011-05-23 18:37 10% [PATCH] doc: add Links page to help folks find relevant info Eric Wong
2011-05-31  9:02     workers not utilizing multiple CPUs Nate Clark
2011-05-31 12:10     ` Lawrence Pit
2011-05-31 12:20       ` Nate Clark
2011-05-31 14:07         ` Clifton King
2011-05-31 15:48  6%       ` Eric Wong
2011-05-31 15:55  0%         ` Clifton King
     [not found]               ` <BANLkTikC5D+0pUDRDgvQbzu=dpwmdKNY=A@mail.gmail.com>
2011-06-01  6:51  0%             ` Nate Clark
2011-06-07  2:28     [PATCH] examples/nginx.conf: add ipv6only comment Eric Wong
2011-06-07 21:05  7% ` [PATCH] configurator: add :ipv6only directive Eric Wong
2011-06-09 20:55  4% [ANN] unicorn 3.7.0 - minor feature update Eric Wong
2011-06-23 16:36     query string max length tweak ? Mohit Chawla
2011-06-23 18:37  4% ` Eric Wong
2011-06-23 19:31  4%   ` Mohit Chawla
2011-06-24  0:38  4%     ` Eric Wong
2011-06-25 16:08     Unicorn and streaming in Rails 3.1 Xavier Noria
2011-06-25 20:33  7% ` Eric Wong
2011-06-29  7:29  6% [ANN] unicorn 4.0.0.2.g19f7 prerelease Eric Wong
2011-06-29 19:10  6% [ANN] unicorn 4.0.1 - regression bugfixes Eric Wong
2011-07-11 16:07     Unicorn vs Apache Matt Smith
2011-07-11 18:45  5% ` Eric Wong
2011-07-11 21:43  4% Matt Smith
2011-08-02 20:09     Strange quit behavior James Cox
2011-08-02 20:34     ` Alex Sharp
2011-08-02 21:54       ` Eric Wong
2011-08-05  4:09         ` Alex Sharp
2011-08-05  4:12           ` Alex Sharp
2011-08-05  8:07             ` Eric Wong
2011-08-17  4:26               ` Alex Sharp
2011-08-17  9:22                 ` Eric Wong
2011-08-31  0:33  4%               ` Eric Wong
2011-09-01 18:45  0%                 ` Alex Sharp
2011-09-01 19:46  0%                   ` Eric Wong
2011-08-16 15:45     Unicorn logging in production env Serg Podtynnyi
2011-08-16 17:30     ` Eric Wong
2011-08-17  9:21       ` Serg Podtynnyi
2011-08-17  9:58         ` Eric Wong
2011-08-17 10:04  6%       ` Eric Wong
2011-08-19 21:07 12% [PATCH] Rack::Chunked and ContentLength middlewares by default Eric Wong
2011-08-20  0:42  4% [ANN] unicorn 4.1.0 - small updates and fixes Eric Wong
2011-09-03 10:52  3% Nightmare! - an nginx alternative for unicorn Eric Wong
2011-09-08  9:04     Sending ABRT to timeout-errant process before KILL J. Austin Hughey
2011-09-08 19:13  6% ` Eric Wong
2011-09-09 23:01  0%   ` Eric Wong
2011-09-15 22:46  3% SSL support pushed out to unicorn.git :x Eric Wong
2011-09-16 10:19 13% Rainbows! or unicorn? russell muetzelfeldt
2011-09-16 11:11 13% ` Eric Wong
2011-09-16 12:38 11%   ` russell muetzelfeldt
2011-09-16 17:00  7%     ` Eric Wong
2011-09-26  1:24     Timeout callback Alex Sharp
2011-09-26  1:42  4% ` Eric Wong
2011-09-26  6:06  0%   ` Christopher Bailey
2011-09-29 18:28     large uploads John Joseph Bachir
2011-09-29 18:47  5% ` Eric Wong
2011-09-29 20:05       ` John Joseph Bachir
2011-09-29 20:54         ` Eric Wong
2011-10-01  3:39           ` John Joseph Bachir
2011-10-01  4:06  4%         ` Eric Wong
2012-02-24 17:43     using unicorn as a local development server Matt Smith
2012-02-24 23:10  3% ` Eric Wong
2012-04-11  8:07     Request-URI Too Long from mongrel-unicorn Nuo Yan
2012-04-11 20:30     ` Eric Wong
2012-04-11 23:43       ` Lawrence Pit
2012-04-12  3:45  4%     ` Eric Wong
2012-04-12  8:18  6%       ` Eric Wong
2012-04-27 14:36     app error: Socket is not connected (Errno::ENOTCONN) Joel Nimety
2012-04-27 18:59  5% ` Eric Wong
2012-04-27 19:33  0%   ` Matt Smith
2012-04-27 22:00  4% [ANN] unicorn 4.3.0.2.g4551 gem prerelease Eric Wong
2012-06-23 16:12     `kill -SIGTRAP <worker pid>` to get a live ruby backtrace + generate backtrace when murdering worker due to timeout Cedric Maion
2012-06-23 16:12     ` [PATCH] `kill -SIGTRAP <worker pid>` to get a live ruby backtrace + generate backtrace when murdered " Cedric Maion
2012-06-23 18:55       ` [PATCH] `kill -SIGTRAP <worker pid>` Eric Wong
2012-06-24 11:05         ` Cedric Maion
2012-06-25  3:59  6%       ` Eric Wong
     [not found]     <mailman.0.1345509654.31187.mongrel-unicorn@rubyforge.org>
2012-08-21  0:44     ` Unused Unicorn processes Konstantin Gredeskoul
2012-08-21  9:11  3%   ` Eric Wong
2012-08-22 18:16  4%     ` Konstantin Gredeskoul
2012-10-30 20:40     Combating nginx 499 HTTP responses during flash traffic scenario Tom Burns
2012-10-30 21:37     ` Eric Wong
2012-11-02 17:59       ` Tom Burns
2012-11-02 19:38         ` Eric Wong
2012-11-03 22:45           ` Tom Burns
2012-11-05 11:48             ` Eric Wong
2012-11-06  3:16               ` Tom Burns
2012-11-06 21:23                 ` Eric Wong
2012-11-29 15:52                   ` Tom Burns
2012-11-29 20:41                     ` Eric Wong
2012-12-04  3:00 13%                   ` Eric Wong
2012-12-08  0:17  4% [ANN] unicorn 4.5.0 (final) - check_client_connection option Eric Wong
2013-01-25 10:52  8% No middleware without touching RACK_ENV Lin Jen-Shin (godfat)
2013-01-25 18:39  0% ` Eric Wong
2013-01-28 14:43       ` Lin Jen-Shin (godfat)
2013-01-28 23:21         ` Eric Wong
2013-01-29  2:31  4%       ` Lin Jen-Shin (godfat)
2013-01-29  2:52  0%         ` Eric Wong
2013-01-29  3:21 10% [PATCH] Add -N or --no-default-middleware option Lin Jen-Shin
2013-01-29  3:52     ` Eric Wong
2013-01-29  4:03  4%   ` Lin Jen-Shin (godfat)
2013-01-29 21:10     [ANN] unicorn 4.6.0pre1 - hijacking support! Eric Wong
2013-02-04 13:40  6% ` Eric Wong
2013-02-26  3:06 14% [ANN] unicorn 4.6.2 - HTTP parser fix for Rainbows! Eric Wong
2013-02-26 15:08     Unicorn on shared apps platform Amol Dev
2013-02-26 15:36  6% ` Hongli Lai
2013-02-26 16:08  6%   ` Amol Dev
2013-02-26 17:26  6%   ` Eric Wong
2013-02-26 17:30  0%     ` Hongli Lai
2013-02-26 17:46  4%       ` Eric Wong
2013-02-26 18:18  0%     ` Amol Dev
2013-03-09 16:02     Unicorn hangs on POST request Tom Pesman
2013-03-09 21:02     ` Eric Wong
2013-03-10 16:22       ` Tom Pesman
2013-03-11 19:49         ` Eric Wong
2013-03-11 22:20           ` Tom Pesman
2013-03-11 23:01             ` Eric Wong
2013-04-02 11:55  6%           ` Tom Pesman
2013-04-02 17:24  6%             ` Eric Wong
2013-04-02 20:25  0%               ` Tom Pesman
2013-04-02 22:36  5%                 ` Eric Wong
2013-04-05 21:16     Fedora Unix socket file location problems David Wilkins
2013-04-05 21:56  5% ` Eric Wong
2013-05-08 23:01  4% [PATCH] HttpParser#next? becomes response_start_sent-aware Eric Wong
2013-06-07  3:03     [PATCH 1/2] Integration test for --no-default-middleware option Micah Chalmer
2013-06-07  9:25  5% ` Eric Wong
2013-06-07  3:03  7% [PATCH 2/2] Make -N/--no-default-middleware option work Micah Chalmer
2013-07-24  3:11  3% [PATCH] unicorn_forever: new executable to respawn masters Eric Wong
2013-08-21 10:28     HTTP streaming and Unicorn timeout Nokan Emiro
2013-08-21 16:41  4% ` Eric Wong
2013-08-22 15:26  4%   ` Nokan Emiro
2013-08-22 15:33  4%     ` Hongli Lai
2013-08-22 17:04  0%     ` Eric Wong
2013-08-23  1:34     Ruby 2.0 Bad file descriptor (Errno::EBADF) Port Himmerland
2013-08-23  2:04     ` Eric Wong
2013-08-23  6:21       ` port
2013-08-23  6:47  4%     ` Eric Wong
2013-09-04 19:00  0%       ` Eric Chapweske
2013-10-10 22:18     unicorn simple cgi without rails nomad Bellcam
2013-10-10 22:53  6% ` Eric Wong
2013-10-20  4:44  4% [RFC] workaround reopen atomicity issues for stdio vs non-stdio Eric Wong
2014-05-25  3:52  4% unicorn 5 roadmap Eric Wong
2014-09-22  1:05     [PATCH 0/4] a few more minor cleanups pushed out Eric Wong
2014-09-22  1:05  7% ` [PATCH 3/4] http: remove the keepalive requests limit Eric Wong
2014-09-22  1:05  8% ` [PATCH 4/4] http: reduce parser from 72 to 56 bytes on 64-bit Eric Wong
2015-02-05 18:01  6% [PATCH] remove 1.8, <= 1.9.1 fallback for missing IO#autoclose= Eric Wong
2015-02-09  9:12 21% [PATCH 1/2] const: drop constants used by Rainbows! Eric Wong
2015-04-22 19:02  3% unicorn 4.8.x-stable branch pushed to git Eric Wong
2015-04-24  3:17  3% [ANN] unicorn 4.9.0 - Rack HTTP server for fast clients and *nix Eric Wong
2015-06-15 22:56  4% [ANN] unicorn 5.0.0.pre1 - incompatible changes! Eric Wong
2015-06-23 19:57  4% ` Hleb Valoshka
2015-06-23 20:04  6%   ` Eric Wong
2015-06-23 20:42  0%     ` Damian Janowski
2015-06-24  9:19  4%     ` Lunar
2015-06-30 23:19  0%       ` TAN: Coquelicot Eric Wong
2015-07-01 11:54  4%         ` Lunar
2015-07-15 22:05 12% [PATCH] doc: remove references to old servers Eric Wong
2015-09-29  6:38     Request to follow SemVer/mention it in homepage Pirate Praveen
2015-09-29  7:36  3% ` Eric Wong
2015-09-29  8:00  0%   ` Pirate Praveen
2015-09-29 19:36  0%     ` Eric Wong
2015-09-30 16:04  0%       ` Pirate Praveen
2015-11-01  8:55  3% [ANN] unicorn 5.0.0 - Rack HTTP server for fast clients and *nix Eric Wong
2016-01-08 18:34     [PATCH] limit rack version for ruby compatibility Adam Duke
2016-01-08 19:18     ` Eric Wong
2016-01-08 21:50       ` Aaron Patterson
2016-01-08 22:37         ` Eric Wong
2016-01-08 23:19           ` Aaron Patterson
2016-01-21 17:12             ` Adam Duke
2016-01-21 20:12  5%           ` Eric Wong
2017-04-01  8:08     [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix Eric Wong
2017-04-02  2:14  6% ` Eric Wong
2023-06-01 18:54     Rack 3 Compatibility Jeremy Evans
2023-06-02  0:00  4% ` Eric Wong
2023-06-02  2:45  0%   ` Jeremy Evans
2023-06-05  9:12  5%     ` [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1 Eric Wong
2023-06-05 10:32  1% [PATCH 00-23/23] start porting tests to Perl5 Eric Wong

Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).