| Date | Commit message (Collapse) |
|---|
|
Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1]. In order to avoid
incompatibilities in the future, I used a Perl script[2] to
prepend `frozen_string_literal: false' to every Ruby file.
Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.
[1] https://bugs.ruby-lang.org/issues/20205
[2] https://yhbt.net/add-fsl.git/74d7689/s/?b=add-fsl.perl
|
|
kgio is an extra download and shared object which costs users
bandwidth, disk space, startup time and memory. Ruby 2.3+
provides `Socket#accept_nonblock(exception: false)' support
in addition to `exception: false' support in IO#*_nonblock
methods from Ruby 2.1.
We no longer distinguish between TCPServer and UNIXServer as
separate classes internally; instead favoring the `Socket' class
of Ruby for both. This allows us to use `Socket#accept_nonblock'
and get a populated `Addrinfo' object off accept4(2)/accept(2)
without resorting to a getpeername(2) syscall (kgio avoided
getpeername(2) in the same way).
The downside is there's more Ruby-level argument passing and
stack usage on our end with HttpRequest#read_headers (formerly
HttpRequest#read). I chose this tradeoff since advancements in
Ruby itself can theoretically mitigate the cost of argument
passing, while syscalls are a high fixed cost given modern CPU
vulnerability mitigations.
Note: no benchmarks have been run since I don't have a suitable
system.
|
|
It's fairly easy given unicorn was designed with synchronous I/O
in mind. The overhead of backtraces from EOFError on
readpartial should be rare given our requirement to only accept
requests from fast, reliable clients on LAN (e.g. nginx or
yet-another-horribly-named-server).
|
|
When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.
I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
|
|
Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
|
|
Most changes are to the tests to avoid uppercase characters in header
keys, which are no longer allowed in rack 3 (to allow for O(1) access).
This also changes a few places where an array of headers was used to
switch to a hash, as a hash is requierd in Rack 3.
Newer versions of curl use a 000 http_code for invalid http codes,
so switch from "42 -eq" to "500 -ne" in the test, as Rack::Lint will
always raise a 500 error.
There is one test that fails on OpenBSD when opening a fifo. This is
unrelated to unicorn as far as I can see, so skip the remaining part
of the test in that case on OpenBSD.
Tests still pass on Rack 2, and presumably Rack 1 as well, though
I didn't test Rack 1.
Co-authored-by: Eric Wong <bofh@yhbt.net>
|
|
This has been irrelevant since Ruby 1.9.0+, actually.
|
|
While unlikely, it's possible for write(2) to return a truncated
value or ENOSPC error if the device is full when writing a tiny
PID file.
As optimism has no place in engineering, use IO#write instead of
IO#syswrite since the former will retry on truncation and raise
a exception on ENOSPC.
|
|
While the capabilities of epoll cannot be fully exploited given
our primitive design; avoiding thundering herd wakeups on larger
SMP machines while below 100% utilization is possible with
Linux 4.5+.
With this change, only one worker wakes up per-connect(2)
(instead of all of them via select(2)), avoiding the thundering
herd effect when the system is mostly idle.
Saturated instances should not notice the difference if they
rarely had multiple workers sleeping in select(2). This change
benefits non-saturated instances.
With 2 parallel clients and 8 workers on a nominally (:P)
8-core CPU (AMD FX-8320), the uconnect.perl test script
invocation showed a reduction from ~3.4s to ~2.5s when
reading an 11-byte response body:
echo worker_processes 8 >u.conf.rb
bs=11 ruby -I lib -I test/ruby-2.5.5/ext/unicorn_http/ bin/unicorn \
test/benchmark/dd.ru -E none -l /tmp/u.sock -c u.conf.rb
time perl -I lib -w test/benchmark/uconnect.perl \
-n 100000 -c 2 /tmp/u.sock
Times improve less as "-c" increases for uconnect.perl (system
noise and timings are inconsistent). The benefit of this change
should be more noticeable on systems with more workers (and
more cores).
I wanted to use EPOLLET (Edge-Triggered) to further reduce
syscalls, here, (similar to the old select()-avoidance bet) but
that would've either added too much complexity to deduplicate
wakeup sources, or run into the same starvation problem we
solved in April 2020[1].
Since the kernel already has the complexity and deduplication
built-in for Level-Triggered epoll support, we'll just let the
kernel deal with it.
Note: do NOT take this as an example of how epoll should be used
in a sophisticated server. unicorn is primitive by design and
cannot use threads nor handle multiple clients at once, thus it
it only uses epoll in this extremely limited manner.
Linux 4.5+ users will notice a regression of one extra epoll FD
per-worker and at least two epoll watches, so
/proc/sys/fs/epoll/max_user_watches may need to be changed along
with RLIMIT_NOFILE.
This change has also been tested on Linux 3.10.x (CentOS 7.x)
and FreeBSD 11.x to ensure compatibility with systems without
EPOLLEXCLUSIVE.
Various EPOLLEXCLUSIVE discussions over the years:
https://yhbt.net/lore/lkml/?q=s:EPOLLEXCLUSIVE+d:..20211001&x=t&o=-1
[1] https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
|
|
It doesn't seem to do anything since commit 221340c4ebc15666
(prevent single listener from monopolizing a worker, 2020-04-16).
|
|
It's actually been unnecessary since commit 6f6e4115b4bb03e5
(rework master-to-worker signaling to use a pipe, 2013-12-09)
|
|
Ruby 1.9.3 was released nearly a decade ago, so there's probably
few (if any) legacy users left, and they can continue using old
versions of unicorn. We'll be able to take advantage of some
Ruby 2.0+-only features down the road (and hopefully 2.3+).
Also, I no longer have a installation of Ruby 1.8 and getting it
working probably isn't worth the effort, so 4.x support is gone.
|
|
This removes the reuse of the parser between requests. Reusing these is
risky in the context of running any other threads within the unicorn
process, also for threads that run background tasks.
If any other thread accidentally grabs hold of the request it can modify
things for the next request in flight.
The downside here is that we allocate more for each request, but that is
worth the trade off here and the security risk we otherwise would carry
to leaking wrong and incorrect data.
|
|
This adds `rack.after_reply` functionality which allows rack middleware
to pass lambdas that will be executed after the client connection has
been closed.
This was driven by a need to perform actions in a request that shouldn't
block the request from completing but also don't make sense as
background jobs.
There is prior art of this being supported found in a few gems, as well
as this functionality existing in other rack based servers (e.g. Puma).
[ew: check if `env' is set in ensure statement]
Acked-by: Eric Wong <e@80x24.org>
|
|
While not part of the rack spec, this API is exposed
by both puma and falcon, and Rails use it when available.
The 103 Early Hints response code is specified in RFC 8297.
|
|
In setups with multiple listeners, it's possible for our greedy
select(2)-avoidance optimization to get pinned on a single, busy
listener and starve the other listener(s).
Prevent starvation by retrying the select(2)-avoidance
optimization if and only if all listeners were active. This
should have no effect on the majority of deployments with only a
single listener.
Thanks to Stan Hu for reporting and testing.
Reported-by: Stan Hu <stanhu@gmail.com>
Tested-by: Stan Hu <stanhu@gmail.com>
Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
|
|
bogomips.org is due to expire, soon, and I'm not willing to pay
extortionist fees to Ethos Capital/PIR/ICANN to keep a .org. So
it's at yhbt.net, for now, but it will change again to
whatever's affordable... Identity is overrated.
Tor users can use .onions and kick ICANN to the curb:
torsocks w3m http://unicorn.ou63pmih66umazou.onion/
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git/
torsocks w3m http://ou63pmih66umazou.onion/unicorn-public/
While we're at it, `s/news.gmane.org/news.gmane.io/g', too.
(but I suspect that'll need to be resynched since our mail
"List-Id:" header is changing).
|
|
Latency from redirects is painful, and HTTPS can protect privacy
in some cases.
|
|
This allows for the equivalent of the
-N/--no-default_middleware command line option to be
specified in the configuration file so it doesn't
need to be specified on the command line every time
unicorn is executed.
It explicitly excludes the use of -N/--no-default_middleware
as an embedded configuration option in the rackup file, by
ignoring the options after ARGV is parsed.
In order to allow the configuration method to work, have
the lambda that Unicorn.builder returns accept two arguments.
Technically, only one argument is needed for the HttpServer
instance, but I'm guessing if the lambda accepts a single
argument, we expect that to be a rack application instead
of a lambda that returns a rack application.
The command line option option to disable default middleware
will take precedence over the unicorn configuration file option
if both are present.
For backwards compatibility, if the lambda passed to
HttpServer accepts 0 arguments, then call it without
arguments.
[ew: fix precedence for arity checking in build_app!
configurator: ensure -N is respected when set in command-line]
|
|
Slowly reducing dependencies on kgio. 'io/wait' is required by
'socket' these days, so it's no extra relocations for .so
loading, either.
|
|
It's not unicorn 6, yet, but we dropped Ruby 1.8 support at
unicorn 5. Stable Ruby 1.9+ releases have always reseeded the
PRNG at fork.
|
|
Ruby trunk started warning about more mismatched indentations
starting around r62836.
|
|
Hijackers may capture and reuse `env' indefinitely, so we must
not use it in those cases for future requests. For non-hijack
requests, we continue to reuse the `env' object to reduce
memory recycling.
Reported-and-tested-by: Sam Saffron <sam.saffron@gmail.com>
|
|
These three cold call sites instruction sequence size by a few
hundred bytes combined since we no longer support Ruby 1.8.6.
The "?/" shorthand is esoteric and no longer avoids allocation
in Ruby 1.9+ (not that this is hot code).
|
|
* origin/worker_exec:
Don't pass a block for fork when forking workers
Add worker_exec configuration option
|
|
This quiets down warnings when run with '-w'
|
|
This reduces the stack depth, making GC more efficient.
|
|
The worker_exec configuration option makes all worker processes
exec after forking. This initializes the worker processes with
separate memory layouts, defeating address space discovery
attacks on operating systems supporting address space layout
randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
Solaris.
Support for execing workers is very similar to support for reexecing
the master process. The main difference is the worker's to_i and
master pipes also need to be inherited after worker exec just as the
listening sockets need to be inherited after reexec.
Because execing working is similar to reexecing the master, this
extracts a couple of methods from reexec (listener_sockets and
close_sockets_on_exec), so they can be reused in worker_spawn.
|
|
This adds a hook that is called after the application has
been loaded by the worker process, directly before it starts
accepting requests. This hook is necessary if your application
needs to gain access to resources during initialization,
and then drop privileges before serving requests.
This is especially useful in conjunction with chroot support
so the app can load all the normal ruby libraries it needs
to function, and then chroot before accepting requests.
If you are preloading the app, it's possible to drop privileges
or chroot in after_fork, but if you are not preloading the app,
the only way to currently do this is to override the private
HttpServer#init_worker_process method, and overriding private
methods is a recipe for future breakage if the internals are
modified. This hook allows for such functionality to be
supported and not break in future versions of Unicorn.
|
|
This option is executed in the master process following all
worker process exits. It is most useful in the case where
the worker process crashes the ruby interpreter, as the worker
process may not be able to send error notifications
appropriately.
For example, let's say you have a specific request that crashes a
worker process, which you expect to be due to a improperly
programmed C extension. By modifying your worker to save request
related data in a temporary file and using this option, you can get
a record of what request is crashing the application, which will
make debugging easier.
Example:
after_worker_exit do |server, worker, status|
server.logger.info "worker #{status.success? ? 'exit' : 'crash'}: #{status}"
file = "request.#{status.pid}.txt"
if File.exist?(file)
do_something_with(File.read(file)) unless status.success?
File.delete(file)
end
end
|
|
HTTPS helps some with reader privacy and Let's Encrypt seems to
be working well enough the past few months.
This change will allow us to reduce subjectAltName bloat in our
TLS certificate over time. It will also promote domain name
agility to support mirrors or migrations to other domains
(including a Tor hidden service mirror).
http://bogomips.org/unicorn/ will remain available for people on
legacy systems without usable TLS. There is no plan for automatic
redirecting from HTTP to HTTPS at this time.
|
|
* add nntp_url to the olddoc website footer
* update legacy support status for 4.x (not 4.8.x)
* update copyright range to 2016
* note all of our development tools are Free Software, too
* remove cgit mention; it may not always be cgit
(but URLs should remain compatible).
* discourage downloading snapshot tarballs;
"git clone" + periodic "git fetch" is more efficient
* remove most mentions of unicorn_rails as that
was meant for ancient Rails 1.x/2.x users
* update path reference to Ruby 2.3.0
* fix nginx upstream module link to avoid redirect
* shorten Message-ID example to avoid redirects
and inadvertant linkage
|
|
The PID of a process can never be zero as kill(2) interprets a '0'
PID arg as "every process in caller's process group", so there's no
risk of the 'nil.to_i => 0' conversion resulting in a truth value
when compared to $$.
|
|
Re-enable and expand on the test case while we're at it for new
Rubies. The bug is now fixed in Ruby 2.3.0dev as of r51576. We
shall assume anybody running a pre-release 2.3.0 at this point is
running a fairly recent snapshot, so we won't bother doing a
finer-grained check in the test for an exact revision number.
|
|
They'll continue to be maintained, but we're no longer advertising
them. Also, favor lowercase "unicorn" while we're at it since that
matches the executable and gem name to avoid unnecessary escaping
for RDoc.
|
|
systemd socket emulation shares FDs across execve, just like
the built-in SIGUSR2 upgrade process in unicorn. Thus it is
easy to support inheriting sockets from systemd.
Tested-by: Christos Trochalakis <yatiohi@ideopolis.gr>
|
|
TCP socket options are now set when inheriting existing sockets from
a parent process. I'm fairly certain all the TCP setsockopt knobs
we use are idempotent and harmless to change.
If anything, the only directive I'd be uncomfortable changing is
shortening the listen(2) (aka :backlog) size, but we've always
changed that anyways since it also applies to UNIX sockets.
Note: removing a configuration knob in a unicorn config file can not
reset the value to the OS-provided default setting. Inherited
sockets must use a new setting to override existing ones.
(or the socket needs to be closed and re-created in the process
launcher before unicorn inherits it).
Noticed-by: Christos Trochalakis <yatiohi@ideopolis.gr>
<20150626114129.GA25883@luke.ws.skroutz.gr>
|
|
Most of these were found by the `linkchecker' package
in Debian.
|
|
Middlewares such as Rack::Lock (used by Rails) break badly unless
the response body is closed on hijack, so we will close it to follow
the lead of other popular Rack servers.
While it's unclear if there's anybody using rack.hijack with unicorn,
we'll try to emulate the behavior of other servers as much as
possible.
ref: https://github.com/ngauthier/tubesock/issues/10
|
|
Unnecessarily exposed accessors and constants take up unnecessary
memory in constant/method tables as well as using extra space in
instruction sequences.
Preforking servers like unicorn are a bloated pigs anyways,
but saving a few hundred bytes here and there can add up and
make them marginally less bad.
|
|
Some process managers such as foreman and daemontools rely on
unicorn not daemonizing, but we still want to be able to process
SIGWINCH in that case.
stdout and stderr may be redirected to a pipe (for cronolog or
similar process), so those are less likely to be attached to a TTY
than stdin. This also allows users to process SIGWINCH when running
inside a regular terminal if they redirect stdin to /dev/null.
Reported-by: Dan Moore <dan@vaporwa.re>
References: <etPan.555b4293.5b47a5b7.e617@danbookpro>
<20150519232858.GA23515@dcvr.yhbt.net>
|
|
kgio_wait_readable is superior for single FDs in that it may use the
ppoll syscall on Linux via Ruby, making it immune to the slowdown
high FDs with select() and the array allocations enforced by the
Ruby wrapper interface.
Note: IO#wait in the io/wait stdlib has the same effect, but as of
2.2 still needlessly checks the FIONREAD ioctl. So avoid needing to
force a new require on users which also incur shared object loading
costs. The longer term plan is to rely entirely on Ruby IO
primitives entirely and drop kgio, but that won't happen until we
can depend on Ruby 2.3 for exception-free accept_nonblock
(which will be released December 2015).
|
|
Literal regexps cost over 450 bytes of memory per-site and
unnecessary use of them costs memory in places where raw execution
speed does not matter. Nowadays, we can rely on String#end_with?
(introduced in 1.8.7) for improved readability, too.
|
|
Literal String#freeze avoids allocations since Ruby 2.1 via the
opt_str_freeze instruction, so we can start relying on it in
some places as Ruby 2.1 adoption increases. The 100-continue
handling is a good place to start since it is an uncommonly-used
code path which benefits from size reduction and the negative
performance impact is restricted to a handful of users.
HTTP_RESPONSE_START can safely live in http_request.rb as its
usage does not cross namespace boundaries
The goal is to eventually eliminate Unicorn::Const entirely.
|
|
Oops, this should've been explained long ago but apparently not.
In response to a comment on
http://www.sitepoint.com/the-self-pipe-trick-explained/
> Does anybody know why both unicorn and foreman read 11 bytes from
> self-pipe?
Unfortunately I couldn't find a way to comment on the site on a
JavaScript-free browser nor does it seem possible without
registering.
Again, anybody can send plain-text mail to:
unicorn-public@bogomips.org
No registration, no real name policy, no terms-of-service, just
plain-text. Feel free to use Tor, mixmaster or any anonymity
service, too.
|
|
In 1.9+ at least, instance variables use less space than constants
in class tables and bytecode, leading to ~700 byte reduction in
bytecode overhead on 64-bit and a reduction in constant table/entries
of the Unicorn::HttpServer class.
|
|
The former is shorter Ruby code and also generates smaller
bytecode.
|
|
unicorn 5 will not support Ruby 1.8 anymore.
Drop mentions of Rubinius, too, it's too difficult to support due to
the proprietary and registration-required nature of its bug tracker.
The smaller memory footprint and CoW-friendly memory allocator in
mainline Ruby is a better fit for unicorn, anyways.
Since Ruby 1.9+ bundles RubyGems and gem startup is faster nowadays,
we'll just depend on that instead of not loading RubyGems.
Drop the local.mk.sample file, too, since it's way out-of-date
and probably isn't useful (I have not used it in a while).
|
|
IO#close_on_exec* methods are available since Ruby 1.9.1. It
allows us to use less bytecode as it requires fewer operands and
avoids constant lookups.
|
|
We're requiring Ruby 1.9.3+, so we can safely depend
on IO#autoclose= being available in 1.9+ and shave off
some bloat.
|