| Date | Commit message (Collapse) |
|---|
|
This adds a hook that is called after the application has
been loaded by the worker process, directly before it starts
accepting requests. This hook is necessary if your application
needs to gain access to resources during initialization,
and then drop privileges before serving requests.
This is especially useful in conjunction with chroot support
so the app can load all the normal ruby libraries it needs
to function, and then chroot before accepting requests.
If you are preloading the app, it's possible to drop privileges
or chroot in after_fork, but if you are not preloading the app,
the only way to currently do this is to override the private
HttpServer#init_worker_process method, and overriding private
methods is a recipe for future breakage if the internals are
modified. This hook allows for such functionality to be
supported and not break in future versions of Unicorn.
|
|
This option is executed in the master process following all
worker process exits. It is most useful in the case where
the worker process crashes the ruby interpreter, as the worker
process may not be able to send error notifications
appropriately.
For example, let's say you have a specific request that crashes a
worker process, which you expect to be due to a improperly
programmed C extension. By modifying your worker to save request
related data in a temporary file and using this option, you can get
a record of what request is crashing the application, which will
make debugging easier.
Example:
after_worker_exit do |server, worker, status|
server.logger.info "worker #{status.success? ? 'exit' : 'crash'}: #{status}"
file = "request.#{status.pid}.txt"
if File.exist?(file)
do_something_with(File.read(file)) unless status.success?
File.delete(file)
end
end
|
|
HTTPS helps some with reader privacy and Let's Encrypt seems to
be working well enough the past few months.
This change will allow us to reduce subjectAltName bloat in our
TLS certificate over time. It will also promote domain name
agility to support mirrors or migrations to other domains
(including a Tor hidden service mirror).
http://bogomips.org/unicorn/ will remain available for people on
legacy systems without usable TLS. There is no plan for automatic
redirecting from HTTP to HTTPS at this time.
|
|
* add nntp_url to the olddoc website footer
* update legacy support status for 4.x (not 4.8.x)
* update copyright range to 2016
* note all of our development tools are Free Software, too
* remove cgit mention; it may not always be cgit
(but URLs should remain compatible).
* discourage downloading snapshot tarballs;
"git clone" + periodic "git fetch" is more efficient
* remove most mentions of unicorn_rails as that
was meant for ancient Rails 1.x/2.x users
* update path reference to Ruby 2.3.0
* fix nginx upstream module link to avoid redirect
* shorten Message-ID example to avoid redirects
and inadvertant linkage
|
|
The PID of a process can never be zero as kill(2) interprets a '0'
PID arg as "every process in caller's process group", so there's no
risk of the 'nil.to_i => 0' conversion resulting in a truth value
when compared to $$.
|
|
Re-enable and expand on the test case while we're at it for new
Rubies. The bug is now fixed in Ruby 2.3.0dev as of r51576. We
shall assume anybody running a pre-release 2.3.0 at this point is
running a fairly recent snapshot, so we won't bother doing a
finer-grained check in the test for an exact revision number.
|
|
They'll continue to be maintained, but we're no longer advertising
them. Also, favor lowercase "unicorn" while we're at it since that
matches the executable and gem name to avoid unnecessary escaping
for RDoc.
|
|
systemd socket emulation shares FDs across execve, just like
the built-in SIGUSR2 upgrade process in unicorn. Thus it is
easy to support inheriting sockets from systemd.
Tested-by: Christos Trochalakis <yatiohi@ideopolis.gr>
|
|
TCP socket options are now set when inheriting existing sockets from
a parent process. I'm fairly certain all the TCP setsockopt knobs
we use are idempotent and harmless to change.
If anything, the only directive I'd be uncomfortable changing is
shortening the listen(2) (aka :backlog) size, but we've always
changed that anyways since it also applies to UNIX sockets.
Note: removing a configuration knob in a unicorn config file can not
reset the value to the OS-provided default setting. Inherited
sockets must use a new setting to override existing ones.
(or the socket needs to be closed and re-created in the process
launcher before unicorn inherits it).
Noticed-by: Christos Trochalakis <yatiohi@ideopolis.gr>
<20150626114129.GA25883@luke.ws.skroutz.gr>
|
|
Most of these were found by the `linkchecker' package
in Debian.
|
|
Middlewares such as Rack::Lock (used by Rails) break badly unless
the response body is closed on hijack, so we will close it to follow
the lead of other popular Rack servers.
While it's unclear if there's anybody using rack.hijack with unicorn,
we'll try to emulate the behavior of other servers as much as
possible.
ref: https://github.com/ngauthier/tubesock/issues/10
|
|
Unnecessarily exposed accessors and constants take up unnecessary
memory in constant/method tables as well as using extra space in
instruction sequences.
Preforking servers like unicorn are a bloated pigs anyways,
but saving a few hundred bytes here and there can add up and
make them marginally less bad.
|
|
Some process managers such as foreman and daemontools rely on
unicorn not daemonizing, but we still want to be able to process
SIGWINCH in that case.
stdout and stderr may be redirected to a pipe (for cronolog or
similar process), so those are less likely to be attached to a TTY
than stdin. This also allows users to process SIGWINCH when running
inside a regular terminal if they redirect stdin to /dev/null.
Reported-by: Dan Moore <dan@vaporwa.re>
References: <etPan.555b4293.5b47a5b7.e617@danbookpro>
<20150519232858.GA23515@dcvr.yhbt.net>
|
|
kgio_wait_readable is superior for single FDs in that it may use the
ppoll syscall on Linux via Ruby, making it immune to the slowdown
high FDs with select() and the array allocations enforced by the
Ruby wrapper interface.
Note: IO#wait in the io/wait stdlib has the same effect, but as of
2.2 still needlessly checks the FIONREAD ioctl. So avoid needing to
force a new require on users which also incur shared object loading
costs. The longer term plan is to rely entirely on Ruby IO
primitives entirely and drop kgio, but that won't happen until we
can depend on Ruby 2.3 for exception-free accept_nonblock
(which will be released December 2015).
|
|
Literal regexps cost over 450 bytes of memory per-site and
unnecessary use of them costs memory in places where raw execution
speed does not matter. Nowadays, we can rely on String#end_with?
(introduced in 1.8.7) for improved readability, too.
|
|
Literal String#freeze avoids allocations since Ruby 2.1 via the
opt_str_freeze instruction, so we can start relying on it in
some places as Ruby 2.1 adoption increases. The 100-continue
handling is a good place to start since it is an uncommonly-used
code path which benefits from size reduction and the negative
performance impact is restricted to a handful of users.
HTTP_RESPONSE_START can safely live in http_request.rb as its
usage does not cross namespace boundaries
The goal is to eventually eliminate Unicorn::Const entirely.
|
|
Oops, this should've been explained long ago but apparently not.
In response to a comment on
http://www.sitepoint.com/the-self-pipe-trick-explained/
> Does anybody know why both unicorn and foreman read 11 bytes from
> self-pipe?
Unfortunately I couldn't find a way to comment on the site on a
JavaScript-free browser nor does it seem possible without
registering.
Again, anybody can send plain-text mail to:
unicorn-public@bogomips.org
No registration, no real name policy, no terms-of-service, just
plain-text. Feel free to use Tor, mixmaster or any anonymity
service, too.
|
|
In 1.9+ at least, instance variables use less space than constants
in class tables and bytecode, leading to ~700 byte reduction in
bytecode overhead on 64-bit and a reduction in constant table/entries
of the Unicorn::HttpServer class.
|
|
The former is shorter Ruby code and also generates smaller
bytecode.
|
|
unicorn 5 will not support Ruby 1.8 anymore.
Drop mentions of Rubinius, too, it's too difficult to support due to
the proprietary and registration-required nature of its bug tracker.
The smaller memory footprint and CoW-friendly memory allocator in
mainline Ruby is a better fit for unicorn, anyways.
Since Ruby 1.9+ bundles RubyGems and gem startup is faster nowadays,
we'll just depend on that instead of not loading RubyGems.
Drop the local.mk.sample file, too, since it's way out-of-date
and probably isn't useful (I have not used it in a while).
|
|
IO#close_on_exec* methods are available since Ruby 1.9.1. It
allows us to use less bytecode as it requires fewer operands and
avoids constant lookups.
|
|
We're requiring Ruby 1.9.3+, so we can safely depend
on IO#autoclose= being available in 1.9+ and shave off
some bloat.
|
|
The monotonic clock is immune to time adjustments so it is not
thrown off by misconfigured clocks. Process.clock_gettime also
generates less garbage on 64-bit systems due to the use of Flonum.
|
|
We implemented barely-advertised support for SSL for two reasons:
1) to detect corruption on LANs beyond what TCP offers
2) to support other servers based on unicorn (never happened)
Since this feature is largely not useful for unicorn itself,
there's no reason to penalize unicorn 5.x users with bloat.
In our defense, SSL support appeared in version 4.2.0 :)
|
|
Replacing the Regexp argument to a rarely-called String#split with a
literal String can save a little memory. The removed Regexp memsize
is 469 bytes on Ruby 2.1:
ObjectSpace.memsize_of(/,/) => 469
Is slightly smaller at 453 bytes on 2.2.0dev (r48474).
These numbers do not include the 40-byte object overhead.
Nevertheless, this is a waste for non-performance-critical code
during the socket inheritance phase. A literal string has less
overhead at 88 bytes:
* 48 bytes for table entry in the frozen string table
* 40 bytes for the object itself
The downside of using a literal string for the String#split argument
is a 40-byte string object gets allocated on every call, but this
piece of code is only called once in a process lifetime.
|
|
This has long been considered a mistake and not documented for very
long.
I considered removing X-Forwarded-Proto and X-Forwarded-SSL
handling, too, so rack.url_scheme is always "http", but that might
lead to compatibility issues in rare apps if Rack::Request#scheme is
not used.
|
|
When daemonizing, it is possible for the grandparent to be
terminated by another process before the master can notify
it. Do not abort the master in this case.
This may fix the following issue:
https://github.com/kostya/eye/issues/49
(which I was notified of privately via email)
|
|
We close SELF_PIPE in the worker immediately, but signal handlers
do not get setup immediately. So prevent workers from erroring out
due to invalid SELF_PIPE.
|
|
This protects us from two problems:
1) we (or our app) somehow called IO#close on one of the sockets
we listen on without removing it from the readers array.
We'll ignore IOErrors from IO#close and assume we wanted to
close it.
2) our SIGQUIT handler is interrupted by itself. This can happen as
a fake signal from the master could be handled and a real signal
from an outside user is sent to us (e.g. from unicorn-worker-killer)
or if a user uses the killall(1) command.
|
|
Signaling using normal kill(2) is preserved, but the master now
prefers to signal workers using a pipe rather than kill(2).
Non-graceful signals (:TERM/:KILL) are still sent using kill(2),
as they ask for immediate shutdown.
This change is necessary to avoid triggering the ubf (unblocking
function) for rb_thread_call_without_gvl (and similar) functions
extensions. Most notably, this fixes compatibility with newer
versions of the 'pg' gem which will cancel a running DB query if
signaled[1].
This also has the nice side-effect of allowing a premature
master death (assuming preload_app didn't cause the master to
spawn off rogue child daemons).
Note: users should also refrain from using "killall" if using the
'pg' gem or something like it.
Unfortunately, this increases FD usage in the master as the writable
end of the pipe is preserved in the master. This limit the number
of worker processes the master may run to the open file limit of the
master process. Increasing the open file limit of the master
process may be needed. However, the FD use on the workers is
reduced by one as the internal self-pipe is no longer used. Thus,
overall pipe allocation for the kernel remains unchanged.
[1] - pg is correct to cancel a query, as it cannot know if
the signal was for a) graceful unicorn shutdown or
b) oh-noes-I-started-a-bad-query-ABORT-ABORT-ABORT!!
|
|
This reduces the window for a non-existent PID for folks who monitor
PIDs (not a great idea anyways). Unfortunately, this change also brings
us back to the case where having a PID later (for other process monitors)
is beneficial but more unicorn releases exist where we write the PID
early.
Thanks to Jimmy Soho for reporting this issue.
ref: <CAHStS5gFYcPBDxkVizAHrOeDKAkjT69kruFdgaY0CbB+vLbK8Q@mail.gmail.com>
This partially reverts 7d6ac0c17eb29a00a5b74099dbb3d4d015999f27
Folks: please monitor your app with HTTP requests rather than checking
processes, a stuck/wedged Ruby VM is still a running one.
|
|
This renables the ability for Ruby 1.8.6 environments to perform reexecs
[ew: clarified this is for 1.8.6,
favor literal {} over Hash.new,
tweaked LISTENERS.map => LISTENERS.each, thanks to Hleb Valoshka
]
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
Thanks to Hongli Lai for noticing my typo. While we're at it, finish up
a halfway-written comment for the EXDEV case
|
|
Ruby 1.9 and later includes IO#autoclose=, so we can use it
and prevent some dead IO objects from hanging around.
|
|
This will preserve mtime on successful renames for comparisions.
While we're at it, avoid writing the new PID until the listeners are
inherited successfully. This can be useful to avoid accidentally
clobbering a good PID if binding the listener or building the app
(preload_app==true) fails
|
|
We do not attempt to write HTTP responses for socket errors if
clients disconnect from us unexpectedly.
Additionally, we do not hide backtraces EINVAL/EBADF errors, since
they are indicative of real bugs which must be fixed.
We do continue to hide hide EOF, ECONNRESET, ENOTCONN, and EPIPE
because clients (even "friendly") ones will break connections due to
client crashes or network failure (which is common for me :P), and
the backtraces from those will cause excessive logging and even
become a DoS vector.
|
|
Once a connection is hijacked, we ignore it completely and leave
the connection at the mercy of the application.
|
|
Rack 1.5.0 (protocol version [1,2]) adds support for
hijacking the client socket (removing it from the control
of unicorn (or any other Rack webserver)).
Tested with rack 1.5.0.
|
|
Rainbows! relies on the ERROR_XXX_RESPONSE constants of unicorn
4.x. Changing the constants in unicorn 4.x will break existing
versions of Rainbows!, so remove the dependency on the constants
and generate the error response dynamically.
Unlike Mongrel, unicorn is unlikely to see malicious traffic and
thus unlikely to benefit from making error messages constant.
For unicorn 5.x, we will drop these constants entirely.
(Rainbows! most likely cannot support check_client_connection
consistently across all concurrency models since some of them
pessimistically buffer all writes in userspace. However, the
extra concurrency of Rainbows! makes it less likely to be
overloaded than unicorn, so this feature is likely less useful
for Rainbows!)
|
|
This patch checks incoming connections and avoids calling the application
if the connection has been closed.
It works by sending the beginning of the HTTP response before calling
the application to see if the socket can successfully be written to.
By enabling this feature users can avoid wasting application rendering
time only to find the connection is closed when attempting to write, and
throwing out the result.
When a client disconnects while being queued or processed, Nginx will log
HTTP response 499 but the application will log a 200.
Enabling this feature will minimize the time window during which the problem
can arise.
The feature is disabled by default and can be enabled by adding
'check_client_connection true' to the unicorn config.
[ew: After testing this change, Tom Burns wrote:
So we just finished the US Black Friday / Cyber Monday weekend running
unicorn forked with the last version of the patch I had sent you. It
worked splendidly and helped us handle huge flash sales without
increased response time over the weekend.
Whereas in previous flash traffic scenarios we would see the number of
HTTP 499 responses grow past the number of real HTTP 200 responses,
over the weekend we saw no growth in 499s during flash sales.
Unexpectedly the patch also helped us ward off a DoS attack where the
attackers were disconnecting immediately after making a request.
ref: <CAK4qKG3rkfVYLyeqEqQyuNEh_nZ8yw0X_cwTxJfJ+TOU+y8F+w@mail.gmail.com>
]
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
In the case where preload_app is true, delay binding new
listeners until after loading the application.
Some applications have very long load times (especially Rails
apps with Ruby 1.9.2). Binding listeners early may cause a load
balancer to incorrectly believe the unicorn workers are ready to
serve traffic even while the app is being loaded.
Once a listener is bound, connect() requests from the load
balancer succeed until the listen backlog is filled. This
allows requests to pile up for a bit (depending on backlog size)
before getting rejected by the kernel. By the time the
application is loaded and ready-to-run, requests in the
listen backlog are likely stale and not useful to process.
Processes inheriting listeners do not suffer this effect, as the
old process should still be capable of serving new requests.
This change does not improve the situation for the
preload_app=false (default) use case. There may not be a
solution for preload_app=false users using large applications.
Fortunately Ruby 1.9.3+ improves load times of large
applications significantly over 1.9.2 so this should be less of
a problem in the future.
Reported via private email sent on 2012-06-29T22:59:10Z
|
|
Since there's nothing unicorn can do to avoid this error
on unconnected/halfway-connected clients, ignoring ENOTCONN
is a safe bet.
Rainbows! has long had this rescue as it called getpeername(2)
on untrusted sockets
|
|
Previously we relied on implicit socket shutdown() from the
close() syscall. However, some Rack applications fork()
(without calling exec()), creating a potentially long-lived
reference to the underlying socket in a child process. This
ends up causing nginx to wait on the socket shutdown when the
child process exits.
Calling shutdown() explicitly signals nginx (or whatever client)
that the unicorn worker is done with the socket, regardless of
the number of FD references to the underlying socket in
existence.
This was not an issue for applications which exec() since
FD_CLOEXEC is always set on the client socket.
Thanks to Patrick Wenger for discovering this. Thanks to
Hongli Lai for the tip on using shutdown() as is done in
Passenger.
ref: http://mid.gmane.org/CAOG6bOTseAPbjU5LYchODqjdF3-Ez4+M8jo-D_D2Wq0jkdc4Rw@mail.gmail.com
|
|
In some cases, EPERM may indicate a real configuration problem,
but it can also just mean the pid file is stale.
|
|
If unicorn doesn't get terminated cleanly (for example if the machine
has its power interrupted) and the pid in the pidfile gets used by
another process, the current unicorn code will exit and not start a
server. This tiny patch fixes that behaviour.
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
No need to duplicate logic here
|
|
It's possible for a SIGUSR1 signal to be received in the
worker immediately before calling IO.select. In that case,
do not clutter logging with IOError and just process the
reopen log request.
|
|
This will also be the foundation of SSL support in Rainbows!
and Zbatery. Some users may also want to use this in
Unicorn on LANs to meet certain security/auditing requirements.
Of course, Nightmare! (in whatever form) should also be able to
use it.
|
|
This prevents the stopping of all workers by SIGWINCH if you're
using a windowing system that will 'exec' unicorn from a process
that's already in a process group.
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
The old comment was confusing. We only zero the tick counter
when forking because application loading can take a long time.
Otherwise, it's always updated.
ref: http://mid.gmane.org/20110908191352.GA25251@dcvr.yhbt.net
|