unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [ANN] unicorn 5.3.0 - Rack HTTP server for fast clients and Unix
@ 2017-04-01  8:08  3% Eric Wong
  0 siblings, 0 replies; 7+ results
From: Eric Wong @ 2017-04-01  8:08 UTC (permalink / raw)
  To: ruby-talk, unicorn-public
  Cc: Jeremy Evans, Simon Eskildsen, Dylan Thacker-Smith

unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

Changes:

unicorn 5.3.0

A couple of portability fixes from Dylan Thacker-Smith and
Jeremy Evans since 5.3.0.pre1 over a week ago, but this looks
ready for a stable release, today.

When I started this over 8 years ago, I wondered if this would
just end up being an April Fools' joke.  Guess not.  I guess I
somehow tricked people into using a terribly marketed web server
that cannot talk directly to untrusted clients :x  Anyways,
unicorn won't be able to handle slow clients 8 years from now,
either, or 80 years from now.  And I vow never to learn to use
new-fangled things like epoll, kqueue, or threads :P

Anyways, this is a largish release with several new features,
and no backwards incompatibilities.

Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
  https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170401&x=t

This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:

   https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u

Jeremy Evans contributed several new features.  First he
implemented after_worker_exit to aid debugging:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
  https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170401&x=t#t

And then security-related features to isolate workers.  Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
  https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
  https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170401&x=t#t
  https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170401&x=t#t

Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
  https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170401&x=t#t

There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.

shortlog of changes since v5.2.0 (2016-10-31):

Dylan Thacker-Smith (1):
      Check for Socket::TCP_INFO constant before trying to get TCP_INFO

Eric Wong (30):
      drop rb_str_set_len compatibility replacement
      TUNING: document THP caveat for Linux users
      tee_input: simplify condition for IO#write
      remove response_start_sent
      http_request: freeze constant strings passed IO#write
      Revert "remove response_start_sent"
      t/t0012-reload-empty-config.sh: access ivars directly if needed
      t0011-active-unix-socket.sh: fix race condition in test
      new test for check_client_connection
      revert signature change to HttpServer#process_client
      support "struct tcp_info" on non-Linux and Ruby 2.2+
      unicorn_http: reduce rb_global_variable calls
      oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
      http_request: reduce insn size for check_client_connection
      freebsd: avoid EINVAL when setting accept filter
      test-lib: expr(1) portability fix
      tests: keep disabled tests defined
      test_exec: SO_KEEPALIVE value only needs to be true
      doc: fix links to raindrops project
      http_request: support proposed Raindrops::TCP states on non-Linux
      ISSUES: expand on mail archive info + subscription disclaimer
      test_ccc: use a pipe to synchronize test
      doc: remove private email support address
      input: update documentation and hide internals.
      http_server: initialize @pid ivar
      gemspec: remove olddoc from build dependency
      doc: add version annotations for new features
      unicorn 5.3.0.pre1
      doc: note after_worker_exit is also 5.3.0+
      test_exec: SO_KEEPALIVE value only needs to be true (take #2)

Jeremy Evans (7):
      Add after_worker_exit configuration option
      Fix code example in after_worker_exit documentation
      Add support for chroot to Worker#user
      Add after_worker_ready configuration option
      Add worker_exec configuration option
      Don't pass a block for fork when forking workers
      Check for SocketError on first ccc attempt

Simon Eskildsen (1):
      check_client_connection: use tcp state on linux

-- 
Yes, this release is real despite the date.

^ permalink raw reply	[relevance 3%]

* [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix
@ 2017-03-24  0:28  3% Eric Wong
  0 siblings, 0 replies; 7+ results
From: Eric Wong @ 2017-03-24  0:28 UTC (permalink / raw)
  To: ruby-talk, unicorn-public; +Cc: Jeremy Evans, Simon Eskildsen

unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between unicorn and slow clients.

* https://bogomips.org/unicorn/
* public list: unicorn-public@bogomips.org
* mail archives: https://bogomips.org/unicorn-public/
* git clone git://bogomips.org/unicorn.git
* https://bogomips.org/unicorn/NEWS.atom.xml
* nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn

This is a pre-release RubyGem intended for testing.

Changes:

unicorn 5.3.0.pre1

A largish release with several new features.

Simon Eskildsen contributed heavily using TCP_INFO under Linux
to implement the (now 5 year old) check_client_connection feature:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
  https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170324&x=t

This also led to FreeBSD and OpenBSD portability improvements in
one of our dependencies, raindrops:

  https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u

Jeremy Evans contributed several new features.  First he
implemented after_worker_exit to aid debugging:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
  https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170324&x=t#t

And then security-related features to isolate workers.  Workers
may now chroot to drop access to the master filesystem, and the
new after_worker_ready configuration hook now exists to aid with
chroot support in workers:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
  https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
  https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170324&x=t#t
  https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170324&x=t#t

Additionally, workers may run in a completely different VM space
(nullifying preload_app and any CoW savings) with the new
worker_exec option:

  https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
  https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170324&x=t#t

There are also several improvements to FreeBSD and OpenBSD
support with the addition of these features.

34 changes since 5.2.0 (2016-10-31):

Eric Wong (27):
      drop rb_str_set_len compatibility replacement
      TUNING: document THP caveat for Linux users
      tee_input: simplify condition for IO#write
      remove response_start_sent
      http_request: freeze constant strings passed IO#write
      Revert "remove response_start_sent"
      t/t0012-reload-empty-config.sh: access ivars directly if needed
      t0011-active-unix-socket.sh: fix race condition in test
      new test for check_client_connection
      revert signature change to HttpServer#process_client
      support "struct tcp_info" on non-Linux and Ruby 2.2+
      unicorn_http: reduce rb_global_variable calls
      oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
      http_request: reduce insn size for check_client_connection
      freebsd: avoid EINVAL when setting accept filter
      test-lib: expr(1) portability fix
      tests: keep disabled tests defined
      test_exec: SO_KEEPALIVE value only needs to be true
      doc: fix links to raindrops project
      http_request: support proposed Raindrops::TCP states on non-Linux
      ISSUES: expand on mail archive info + subscription disclaimer
      test_ccc: use a pipe to synchronize test
      doc: remove private email support address
      input: update documentation and hide internals.
      http_server: initialize @pid ivar
      gemspec: remove olddoc from build dependency
      doc: add version annotations for new features

Jeremy Evans (6):
      Add after_worker_exit configuration option
      Fix code example in after_worker_exit documentation
      Add support for chroot to Worker#user
      Add after_worker_ready configuration option
      Add worker_exec configuration option
      Don't pass a block for fork when forking workers

Simon Eskildsen (1):
      check_client_connection: use tcp state on linux

-- 
5.3.0 in a week, maybe?

^ permalink raw reply	[relevance 3%]

* Re: [PATCH] check_client_connection: use tcp state on linux
  2017-03-08 12:06  0%                 ` Simon Eskildsen
@ 2017-03-13 20:16  0%                   ` Simon Eskildsen
  0 siblings, 0 replies; 7+ results
From: Simon Eskildsen @ 2017-03-13 20:16 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

I've put ccc-tcp-v3 in production today--it appears to be working as
expected, still rejecting at the exact same rate as before (100-200
per second for us during steady-state).

On Wed, Mar 8, 2017 at 7:06 AM, Simon Eskildsen
<simon.eskildsen@shopify.com> wrote:
> Patch-set looks great Eric, thanks!
>
> I'm hoping to test this in production later this week or next, but
> we're a year behind on Unicorn (ugh), so will need to carefully roll
> that out.
>
> On Tue, Mar 7, 2017 at 7:26 PM, Eric Wong <e@80x24.org> wrote:
>> Eric Wong <e@80x24.org> wrote:
>>> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>>> > @@ -28,6 +29,7 @@ class Unicorn::HttpParser
>>> >    # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
>>> >    # 2.2+ optimizes hash assignments when used with literal string keys
>>> >    HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
>>> > +  EMPTY_ARRAY = [].freeze
>>>
>>> (minor) this was made before commit e06b699683738f22
>>> ("http_request: freeze constant strings passed IO#write")
>>> but I've trivially fixed it up, locally.
>>
>> And I actually screwed it up, pushed out ccc-tcp-v2 branch
>> with that fix squashed in :x
>>
>> Writing tests, now...

^ permalink raw reply	[relevance 0%]

* Re: [PATCH] check_client_connection: use tcp state on linux
  2017-03-08  0:26  0%               ` Eric Wong
@ 2017-03-08 12:06  0%                 ` Simon Eskildsen
  2017-03-13 20:16  0%                   ` Simon Eskildsen
  0 siblings, 1 reply; 7+ results
From: Simon Eskildsen @ 2017-03-08 12:06 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

Patch-set looks great Eric, thanks!

I'm hoping to test this in production later this week or next, but
we're a year behind on Unicorn (ugh), so will need to carefully roll
that out.

On Tue, Mar 7, 2017 at 7:26 PM, Eric Wong <e@80x24.org> wrote:
> Eric Wong <e@80x24.org> wrote:
>> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>> > @@ -28,6 +29,7 @@ class Unicorn::HttpParser
>> >    # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
>> >    # 2.2+ optimizes hash assignments when used with literal string keys
>> >    HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
>> > +  EMPTY_ARRAY = [].freeze
>>
>> (minor) this was made before commit e06b699683738f22
>> ("http_request: freeze constant strings passed IO#write")
>> but I've trivially fixed it up, locally.
>
> And I actually screwed it up, pushed out ccc-tcp-v2 branch
> with that fix squashed in :x
>
> Writing tests, now...

^ permalink raw reply	[relevance 0%]

* Re: [PATCH] check_client_connection: use tcp state on linux
  2017-03-07 22:50  3%             ` Eric Wong
@ 2017-03-08  0:26  0%               ` Eric Wong
  2017-03-08 12:06  0%                 ` Simon Eskildsen
  0 siblings, 1 reply; 7+ results
From: Eric Wong @ 2017-03-08  0:26 UTC (permalink / raw)
  To: Simon Eskildsen; +Cc: unicorn-public, Aman Gupta

Eric Wong <e@80x24.org> wrote:
> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
> > @@ -28,6 +29,7 @@ class Unicorn::HttpParser
> >    # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
> >    # 2.2+ optimizes hash assignments when used with literal string keys
> >    HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
> > +  EMPTY_ARRAY = [].freeze
> 
> (minor) this was made before commit e06b699683738f22
> ("http_request: freeze constant strings passed IO#write")
> but I've trivially fixed it up, locally.

And I actually screwed it up, pushed out ccc-tcp-v2 branch
with that fix squashed in :x

Writing tests, now...

^ permalink raw reply	[relevance 0%]

* Re: [PATCH] check_client_connection: use tcp state on linux
  @ 2017-03-07 22:50  3%             ` Eric Wong
  2017-03-08  0:26  0%               ` Eric Wong
  0 siblings, 1 reply; 7+ results
From: Eric Wong @ 2017-03-07 22:50 UTC (permalink / raw)
  To: Simon Eskildsen; +Cc: unicorn-public, Aman Gupta

Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
> Here's another update Eric!

Thanks, a few teeny issues fixed up locally (but commented
inline, below).

However, there's a bigger problem which I'm Cc:-ing Aman about
for tmm1/gctools changing process_client in the internal API.

I won't burden him maintaining extra versions, nor will I force
users to use a certain version of unicorn or gctools to match.

Aman: for reference, relevant messages in the archives:

https://bogomips.org/unicorn-public/?q=d:20170222-+check_client_connection&x=t

(TL;DR: I don't expect Aman will have to do anything,
 just keeping him in the loop)

> +++ b/lib/unicorn/http_server.rb
> @@ -558,8 +558,8 @@ def e100_response_write(client, env)
> 
>    # once a client is accepted, it is processed in its entirety here
>    # in 3 easy steps: read request, call app, write app response
> -  def process_client(client)
> -    status, headers, body = @app.call(env = @request.read(client))
> +  def process_client(client, listener)
> +    status, headers, body = @app.call(env = @request.read(client, listener))

I can squash this fix in, locally:

diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 5572e59..74a1d51 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -67,8 +67,8 @@ def self.new(app, interval = 5, path = %r{\A/})
 
   #:stopdoc:
   PATH_INFO = "PATH_INFO"
-  def process_client(client)
-    super(client) # Unicorn::HttpServer#process_client
+  def process_client(client, listener)
+    super(client, listener) # Unicorn::HttpServer#process_client
     if OOBGC_PATH =~ OOBGC_ENV[PATH_INFO] && ((@@nr -= 1) <= 0)
       @@nr = OOBGC_INTERVAL
       OOBGC_ENV.clear

However, https://github.com/tmm1/gctools also depends on this
undocumented internal API of ours; and I will not consider
breaking it for release...

Pushed to the "ccc-tcp" branch @ git://bogomips.org/unicorn
(commit beaee769c6553bf4e0260be2507b8235f0aa764f)

>      begin
>        return if @request.hijacked?
> @@ -655,7 +655,7 @@ def worker_loop(worker)
>          # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
>          # but that will return false
>          if client = sock.kgio_tryaccept
> -          process_client(client)
> +          process_client(client, sock)
>            nr += 1
>            worker.tick = time_now.to_i
>          end


> @@ -28,6 +29,7 @@ class Unicorn::HttpParser
>    # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
>    # 2.2+ optimizes hash assignments when used with literal string keys
>    HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
> +  EMPTY_ARRAY = [].freeze

(minor) this was made before commit e06b699683738f22
("http_request: freeze constant strings passed IO#write")
but I've trivially fixed it up, locally.

> +    def check_client_connection(socket, listener) # :nodoc:
> +      if Kgio::TCPServer === listener
> +        @@tcp_info ||= Raindrops::TCP_Info.new(socket)
> +        @@tcp_info.get!(socket)
> +        raise Errno::EPIPE, "client closed connection".freeze,
> EMPTY_ARRAY if closed_state?(@@tcp_info.state)

(minor) I needed to wrap that line since I use gigantic fonts
(fixed up locally)

^ permalink raw reply related	[relevance 3%]

* [PATCH] http_request: freeze constant strings passed IO#write
@ 2017-02-13 18:27  7% Eric Wong
  0 siblings, 0 replies; 7+ results
From: Eric Wong @ 2017-02-13 18:27 UTC (permalink / raw)
  To: unicorn-public

This ensures we won't have duplicate objects in Ruby 2.0-2.4.
For Ruby 2.5.0dev+, this avoids any duplicate cleanup
introduced as of r57471: https://bugs.ruby-lang.org/issues/13085
---
 lib/unicorn/http_request.rb | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 0c1f9bb..c176083 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -24,10 +24,7 @@ class Unicorn::HttpParser
   NULL_IO = StringIO.new("")
 
   # :stopdoc:
-  # A frozen format for this is about 15% faster
-  # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
-  # 2.2+ optimizes hash assignments when used with literal string keys
-  HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
+  HTTP_RESPONSE_START = [ 'HTTP'.freeze, '/1.1 '.freeze ]
   @@input_class = Unicorn::TeeInput
   @@check_client_connection = false
 
-- 
EW


^ permalink raw reply related	[relevance 7%]

Results 1-7 of 7 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2017-02-13 18:27  7% [PATCH] http_request: freeze constant strings passed IO#write Eric Wong
2017-02-25 14:03     [PATCH] check_client_connection: use tcp state on linux Simon Eskildsen
2017-02-25 16:19     ` Simon Eskildsen
2017-02-25 23:12       ` Eric Wong
2017-02-27 11:44         ` Simon Eskildsen
2017-02-28 21:12           ` Eric Wong
2017-03-01  3:18             ` Eric Wong
2017-03-06 21:32               ` Simon Eskildsen
2017-03-07 22:50  3%             ` Eric Wong
2017-03-08  0:26  0%               ` Eric Wong
2017-03-08 12:06  0%                 ` Simon Eskildsen
2017-03-13 20:16  0%                   ` Simon Eskildsen
2017-03-24  0:28  3% [ANN] unicorn 5.3.0.pre1 - Rack HTTP server for fast clients and Unix Eric Wong
2017-04-01  8:08  3% [ANN] unicorn 5.3.0 " Eric Wong

Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).