about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-10-25ext: switch filter_body to picohttpparser pico
This should open the door for us to process chunked data faster by modifying the buffer in-place rather than copying to a destination buffer.
2021-10-24allow Ruby to deduplicate remaining globals
Most of these are bound to be used in Rack/Rails/apps/gems, (though possibly with different encodings). Give Ruby a chance to deduplicate them, at least.
2021-10-04use EPOLLEXCLUSIVE on Linux 4.5+
While the capabilities of epoll cannot be fully exploited given our primitive design; avoiding thundering herd wakeups on larger SMP machines while below 100% utilization is possible with Linux 4.5+. With this change, only one worker wakes up per-connect(2) (instead of all of them via select(2)), avoiding the thundering herd effect when the system is mostly idle. Saturated instances should not notice the difference if they rarely had multiple workers sleeping in select(2). This change benefits non-saturated instances. With 2 parallel clients and 8 workers on a nominally (:P) 8-core CPU (AMD FX-8320), the uconnect.perl test script invocation showed a reduction from ~3.4s to ~2.5s when reading an 11-byte response body: echo worker_processes 8 >u.conf.rb bs=11 ruby -I lib -I test/ruby-2.5.5/ext/unicorn_http/ bin/unicorn \ test/benchmark/dd.ru -E none -l /tmp/u.sock -c u.conf.rb time perl -I lib -w test/benchmark/uconnect.perl \ -n 100000 -c 2 /tmp/u.sock Times improve less as "-c" increases for uconnect.perl (system noise and timings are inconsistent). The benefit of this change should be more noticeable on systems with more workers (and more cores). I wanted to use EPOLLET (Edge-Triggered) to further reduce syscalls, here, (similar to the old select()-avoidance bet) but that would've either added too much complexity to deduplicate wakeup sources, or run into the same starvation problem we solved in April 2020[1]. Since the kernel already has the complexity and deduplication built-in for Level-Triggered epoll support, we'll just let the kernel deal with it. Note: do NOT take this as an example of how epoll should be used in a sophisticated server. unicorn is primitive by design and cannot use threads nor handle multiple clients at once, thus it it only uses epoll in this extremely limited manner. Linux 4.5+ users will notice a regression of one extra epoll FD per-worker and at least two epoll watches, so /proc/sys/fs/epoll/max_user_watches may need to be changed along with RLIMIT_NOFILE. This change has also been tested on Linux 3.10.x (CentOS 7.x) and FreeBSD 11.x to ensure compatibility with systems without EPOLLEXCLUSIVE. Various EPOLLEXCLUSIVE discussions over the years: https://yhbt.net/lore/lkml/?q=s:EPOLLEXCLUSIVE+d:..20211001&x=t&o=-1 [1] https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
2021-10-04worker_loop: get rid of select() avoidance hack
It doesn't seem to do anything since commit 221340c4ebc15666 (prevent single listener from monopolizing a worker, 2020-04-16).
2021-10-04http_server: get rid of Process.ppid check
It's actually been unnecessary since commit 6f6e4115b4bb03e5 (rework master-to-worker signaling to use a pipe, 2013-12-09)
2021-10-04HACKING: drop outdated information about pandoc
It's been outdated since d9b5943af26d5df5 (doc: replace pandoc-"Markdown" with real manpages, 2019-12-15)
2021-10-04makefile: reduce unnecessary rebuilds
For the most part, header changes shouldn't trigger extconf.rb reruns.
2021-10-04extconf.rb: get rid of unnecessary checks
SIZEOF_*, *2NUM and NUM2* should all be defined by ruby.h and dependencies it pulls in since Ruby 2.0 and possibly earlier. INT_MAX and LLONG_MAX are in limits.h which is POSIX. HAVE_GMTIME_R is already defined by ruby/config.h, so we shouldn't have to check for it, either. Combined, these changes speed up extconf.rb by several seconds.
2021-09-26drop unnecessary IO#close_on_exec=true assignment
Ruby 2.0+ sets FD_CLOEXEC by default on all FDs.
2021-09-26drop Ruby 1.9.3 support, require 2.0+ for now
Ruby 1.9.3 was released nearly a decade ago, so there's probably few (if any) legacy users left, and they can continue using old versions of unicorn. We'll be able to take advantage of some Ruby 2.0+-only features down the road (and hopefully 2.3+). Also, I no longer have a installation of Ruby 1.8 and getting it working probably isn't worth the effort, so 4.x support is gone.
2021-05-01test_util: less excessive encoding tests
Ruby's handling of encodings hasn't changed much in over a decade and these tests haven't failed for me since August 2013: https://yhbt.net/unicorn-public/9af083d7f6b97c0f5ebbdd9a42b58478a6f874b7/s/ So lets take a small step in reducing energy consumption and save potential developers over 10s of CPU time.
2021-03-17unicorn 6.0.0 - no more recycling Rack env v6.0.0
This release allocates a new Rack `env' hash for every request. This is done for safety with internally-(thread|event)-using Rack apps which expect to use `env' after the normal Rack response is complete, but without relying on rack.hijack[1]. Thanks to Dirkjan Bussink <d.bussink@gmail.com> for the patch: https://yhbt.net/unicorn-public/66A68DD8-83EF-4C7A-80E8-3F1F7AB31670@github.com/ The major version is bumped since: 1) there are performance regressions for some simple Rack apps 2) unsupported 3rd-party monkey patches which previously relied on this behavior may be broken (our version of OobGC was). The test suite is also more reliable on multi-core systems and Ruby 3.x. [1] thread from 2017 around rack.hijack safety: https://yhbt.net/unicorn-public/CAAtdryPG3nLuyo0jxfYW1YHu1Q+ZpkLkd4KdWC8vA46B5haZxw@mail.gmail.com/
2021-03-15Merge remote-tracking branch 'origin/master' into v6-wip v6-wip
* origin/master: tests: force blocking I/O for Ruby 3.x
2021-03-15tests: force blocking I/O for Ruby 3.x
Otherwise we get test failures since we use sysread and syswrite in many places
2021-03-13http_request: drop unnecessary #clear call
Since we allocate a new request object for each request, the #clear call is now unnecessary
2021-03-13Allocate a new request for each client
This removes the reuse of the parser between requests. Reusing these is risky in the context of running any other threads within the unicorn process, also for threads that run background tasks. If any other thread accidentally grabs hold of the request it can modify things for the next request in flight. The downside here is that we allocate more for each request, but that is worth the trade off here and the security risk we otherwise would carry to leaking wrong and incorrect data.
2021-03-13bump version for 6.x development
While no incompatible changes to any supported configuration files are planned, some deep internal data structures will change and there are likely some 3rd-party monkey patches broken by this.
2021-03-13test/test_helper: only unlink redirected logs from parent
We don't want at_exit firing in child processes and never wanted it. This is apparently a long standing bug in the tests that only started causing test_worker_dies_on_dead_master failures for me. I assume it's only showing up now for me due to kernel scheduler changes, since I've been using the same 4-core CPU for ~11 years, now.
2020-12-24unicorn 5.8.0 - rack.after_reply support v5.8.0
This release supports env['rack.after_reply'] which allows rack middleware to pass lambdas to be executed after the client connection is closed, matching functionality in Puma. Thanks to Blake Williams for this patch: https://yhbt.net/unicorn-public/9873E53C-04D3-4759-9678-CA17DBAEF7B7@blakewilliams.me/ The top-level of our website is now simpler and no longer redundant with the contents of https://yhbt.net/unicorn/README.html (which contains the old content)
2020-12-24build: publish_doc: remove created.rid and index.html from site
created.rid has no business being published anyways, and index.html is redundant given README.html. Avoid confusing search engines with identical documents at different URLs. Our "homepage" is just a directory listing, now, which should help discoverability of non-HTML docs. Old content is at https://yhbt.net/unicorn/README.html
2020-12-09Add rack.after_reply functionality
This adds `rack.after_reply` functionality which allows rack middleware to pass lambdas that will be executed after the client connection has been closed. This was driven by a need to perform actions in a request that shouldn't block the request from completing but also don't make sense as background jobs. There is prior art of this being supported found in a few gems, as well as this functionality existing in other rack based servers (e.g. Puma). [ew: check if `env' is set in ensure statement] Acked-by: Eric Wong <e@80x24.org>
2020-09-08unicorn 5.7.0 v5.7.0
Relaxed Ruby version requirements for Ruby 3.0.0dev. Thanks to Jean Boussier for testing
2020-09-08PATCH: doc: add IMAP/IMAPS mailbox info
Apparently, an IMAP server happened at some point and somehow stays running. Maybe more users have IMAP clients than NNTP clients...
2020-09-06Update ruby_version requirement to allow ruby 3.0
Ruby just recently bump the master version to 3.0. This requirement bump is necessary to test unicorn against ruby master. [ew: wrap at <80 columns for hackers with poor eyesight] Acked-by: Eric Wong <bofh@yhbt.net>
2020-07-30build: revamp and avoid unnecessary rebuilds
We can limit the amount of Ruby-version-specific code to just the stuff in ext/* and bin/*, reducing I/O traffic and FS + page cache footprint. Furthermore, rely on GNU make behavior to copy all the necessary files so we don't trigger unnecessary extconf.rb invocations just by touching a .rb file in lib.
2020-07-30test_helper: support TAIL= env for watching tests
This can be useful for diagnosing failures, especially since GNU tail supports inotify these days.
2020-07-26unicorn 5.6.0 - early_hints support v5.6.0
This release adds support for the early_hints configurator directive for the 'rack.early_hints' API used by Rails 5.2+. Thanks to Jean Boussier for the patch. Link: https://yhbt.net/unicorn-public/242F0859-0F83-4F14-A0FF-5BE392BB01E6@shopify.com/
2020-07-24configurator: SIGHUP resets early_hints if unset
If a user removes "early_hints" entirely from the config file, a SIGHUP needs to restore the default value. This is consistent with the behavior of all the other configuration variables. Cc: Jean Boussier <jean.boussier@gmail.com>
2020-07-24test_server: test_early_hints: fix test reliability
IO#sysread may only capture the 103 response and return before the server can send the 200. Since we don't support persistent connections, we can just use IO#read to rely on the server giving us an EOF after the 200 is sent. Cc: Jean Boussier <jean.boussier@gmail.com>
2020-07-16Add early hints support
While not part of the rack spec, this API is exposed by both puma and falcon, and Rails use it when available. The 103 Early Hints response code is specified in RFC 8297.
2020-04-27unicorn 5.5.5 v5.5.5
This release fixes a bug for users of multiple listeners setups where a busy listen socket could starve other listeners. Thanks to Stan Hu for reporting and testing. No need to upgrade if you're using a single listen socket. Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
2020-04-16prevent single listener from monopolizing a worker
In setups with multiple listeners, it's possible for our greedy select(2)-avoidance optimization to get pinned on a single, busy listener and starve the other listener(s). Prevent starvation by retrying the select(2)-avoidance optimization if and only if all listeners were active. This should have no effect on the majority of deployments with only a single listener. Thanks to Stan Hu for reporting and testing. Reported-by: Stan Hu <stanhu@gmail.com> Tested-by: Stan Hu <stanhu@gmail.com> Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
2020-03-24unicorn 5.5.4 v5.5.4
One change to improve RFC 7230 conformance in the HTTP parser: https://yhbt.net/unicorn-public/20200319022823.32472-1-bofh@yhbt.net/
2020-03-19http: improve RFC 7230 conformance
We need to favor "Transfer-Encoding: chunked" over "Content-Length" in the request header if they both exist. Furthermore, we now reject redundant chunking and cases where "chunked" is not the final encoding. We currently do not and have no plans to decode "gzip", "deflate", or "compress" encoding as described by RFC 7230. That's a job more appropriate for middleware, anyways. cf. https://tools.ietf.org/html/rfc7230 https://www.rfc-editor.org/errata_search.php?rfc=7230
2020-01-31unicorn 5.5.3 v5.5.3
Documentation updates to switch bogomips.org to yhbt.net since the .org TLD won't be affordable in the near future. There's also a few minor test cleanups.
2020-01-20test_upload: use spawn to simplify redirects
We can start using Ruby 1.9 APIs, nowadays
2020-01-20test_helper: remove unused `chunked_spawn'
It was added nearly 11 years ago in commit 6945342a1f0a4caa ("Transfer-Encoding: chunked streaming input support") but never used.
2020-01-20test/exec/test_exec: bring worker_processes down to 2
My hardware gets worse and worse every year :<
2020-01-20doc: s/bogomips.org/yhbt.net/g
bogomips.org is due to expire, soon, and I'm not willing to pay extortionist fees to Ethos Capital/PIR/ICANN to keep a .org. So it's at yhbt.net, for now, but it will change again to whatever's affordable... Identity is overrated. Tor users can use .onions and kick ICANN to the curb: torsocks w3m http://unicorn.ou63pmih66umazou.onion/ torsocks git clone http://ou63pmih66umazou.onion/unicorn.git/ torsocks w3m http://ou63pmih66umazou.onion/unicorn-public/ While we're at it, `s/news.gmane.org/news.gmane.io/g', too. (but I suspect that'll need to be resynched since our mail "List-Id:" header is changing).
2019-12-20unicorn 5.5.2 v5.5.2
Thanks to Terry Scheingeld, we now workaround a Ruby bug and can now run with taint checks enabled: <https://bugs.ruby-lang.org/issues/14485> <https://bogomips.org/unicorn-public/CABg1sXrvGv9G6CDQxePDUqTe6N-5UpLXm7eG3YQO=dda-Cgg7A@mail.gmail.com/> There's also a few documentation updates and building packages from source is easier since pandoc is no longer a dependency (and I can no longer afford the bandwidth or space to install it). Eric Wong (7): test/benchmark/ddstream: demo for slowly reading clients test/benchmark/readinput: demo for slowly uploading clients test/benchmark/uconnect: test for accept loop speed examples/unicorn@.service: note the NonBlocking flag Merge remote-tracking branch 'origin/ts/tmpio' test_util: get rid of some unused variables in tests doc: replace pandoc-"Markdown" with real manpages Terry Scheingeld (1): tmpio: workaround File#path being tainted on unlink
2019-12-15doc: replace pandoc-"Markdown" with real manpages
Trying to install pandoc on an x86-64 Debian stable system says: > Need to get 15.2 MB of archives. > After this operation, 117 MB of additional disk space will be used. My laptop is on metered Internet nowadays and already low on disk space, so installing pandoc is not realistic. Maybe it wasn't realistic to other hackers with limited resources in the past. There's also dozens of subtly incompatible Markdown flavors out there, most of which can't really handle manpages. Anyways, roff isn't too bad and at least groff is well-documented. Updating the website now requires olddoc 1.8.0 (which is much smaller than pandoc), but I'm the only one with that burden. On the flipside more users can update and read the manpages locally without extra software, since nearly every developer's *nix system has man(1) command, unlike pandoc.
2019-12-15test_util: get rid of some unused variables in tests
Ruby 2.7.0dev warns on them
2019-12-15Merge remote-tracking branch 'origin/ts/tmpio'
* origin/ts/tmpio: tmpio: workaround File#path being tainted on unlink
2019-12-11tmpio: workaround File#path being tainted on unlink
Ruby mistakenly taints the file path, causing File.unlink to fail: https://bugs.ruby-lang.org/issues/14485 Workaround the Ruby bug by keeping the path as a local variable and passing that to File.unlink, instead of the return value of File#path. Link: https://bogomips.org/unicorn-public/CABg1sXrvGv9G6CDQxePDUqTe6N-5UpLXm7eG3YQO=dda-Cgg7A@mail.gmail.com/
2019-07-03examples/unicorn@.service: note the NonBlocking flag
It's racy otherwise when starting simultaneous instanced units. Without specifying NonBlocking=true, systemd will clear the O_NONBLOCK flag every time it starts a new service instance. There's a small window where systemd can clear O_NONBLOCK immediately after it's set by Ruby (or kgio): unicorn@1 |systemd |unicorn@2 ---------------------------+----------------+-------------------- F_SETFL, O_NONBLOCK|O_RDWR | | (not running, yet) |F_SETFL, O_RDWR | |fork | | exec unicorn@2 | accept4(...) # blocks! | | (now started by systemd) | |F_SETFL,O_NONBLOCK|O_RDWR | |accept4(...) non-blocking
2019-05-12test/benchmark/uconnect: test for accept loop speed
In preparation for kgio removal, I want to ensure we can maintain existing performance when swapping kgio_tryaccept for accept_nonblock on Ruby 2.3+ There's plenty of TCP benchmarking tools, but TCP port reuse delays hurt predictability since unicorn doesn't do persistent connections. So this is exclusively for Unix sockets and uses Perl instead of Ruby since I don't want to be bothered with GC unpredictability on the client side.
2019-05-12test/benchmark/readinput: demo for slowly uploading clients
This is intended to demonstrate how badly we suck at dealing with slow clients making uploads. It can help users evaluate alternative fully-buffering reverse proxies, because nginx should not be the only option.
2019-05-12test/benchmark/ddstream: demo for slowly reading clients
This is intended to demonstrate how badly we suck at dealing with slow clients. It can help users evaluate alternative fully-buffering reverse proxies, because nginx should not be the only option. Update the benchmark README while we're at it
2019-05-06unicorn 5.5.1 v5.5.1
This release fixes and works around issues from v5.5.0 (2019-03-04) Stephen Demjanenko worked around a pipe resource accounting bug present in old Linux kernels. Linux 3.x users from 3.16.57 and on are unaffected. Linux 4.x users from 4.9 and on are unaffected. https://bogomips.org/unicorn-public/1556922018-24096-1-git-send-email-sdemjanenko@gmail.com/ Stan Pitucha reported a bug with the old `unicorn_rails' wrapper (intended for Rails 2.x users) which was promptly fixed by Jeremy Evans: https://bogomips.org/unicorn-public/CAJ2_uEPifcv_ux4sX=t0C4zHTHGhqVfcLcSB2kTU3Rb_6pQ3nw@mail.gmail.com/ There's also some doc updates to warn users off `unicorn_rails'; the homepage is now energy-efficient for OLEDs and CRTs; and I'm no longer advertising mailing list subscriptions (because I hate centralization and mail archives are the priority) Eric Wong (3): doc: unicorn_rails: clarify that it is intended for rails <= 2.x doc: stop advertising mailing list subscription doc: switch homepage to dark216 Jeremy Evans (1): unicorn_rails: fix regression with Rails >= 3.x in app build Stephen Demjanenko (1): Rescue failed pipe resizes due to permissions
2019-05-03Rescue failed pipe resizes due to permissions
When running: ``` require 'kgio' require 'raindrops' F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/ Kgio::Pipe.new.each do |io| io.close_on_exec = true if defined?(F_SETPIPE_SZ) begin puts "setting" io.fcntl(F_SETPIPE_SZ, Raindrops::PAGE_SIZE) rescue Errno::EINVAL puts "rescued" rescue => e puts ["FAILED HARD", e].inspect end end end ``` on a few servers to test some Unicorn boot failures I saw: ``` ["FAILED HARD", #<Errno::EPERM: Operation not permitted>] ``` The `EPERM` error gets raised by the Linux kernel if: ``` (too_many_pipe_buffers_hard(pipe->user) || too_many_pipe_buffers_soft(pipe->user)) && !capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN) ``` Given that the resize is not strictly necessary Unicorn should rescue the error and continue booting.