about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2013-12-09rework master-to-worker signaling to use a pipe
Signaling using normal kill(2) is preserved, but the master now prefers to signal workers using a pipe rather than kill(2). Non-graceful signals (:TERM/:KILL) are still sent using kill(2), as they ask for immediate shutdown. This change is necessary to avoid triggering the ubf (unblocking function) for rb_thread_call_without_gvl (and similar) functions extensions. Most notably, this fixes compatibility with newer versions of the 'pg' gem which will cancel a running DB query if signaled[1]. This also has the nice side-effect of allowing a premature master death (assuming preload_app didn't cause the master to spawn off rogue child daemons). Note: users should also refrain from using "killall" if using the 'pg' gem or something like it. Unfortunately, this increases FD usage in the master as the writable end of the pipe is preserved in the master. This limit the number of worker processes the master may run to the open file limit of the master process. Increasing the open file limit of the master process may be needed. However, the FD use on the workers is reduced by one as the internal self-pipe is no longer used. Thus, overall pipe allocation for the kernel remains unchanged. [1] - pg is correct to cancel a query, as it cannot know if the signal was for a) graceful unicorn shutdown or b) oh-noes-I-started-a-bad-query-ABORT-ABORT-ABORT!!
2013-11-26always write PID file early for compatibility
This reduces the window for a non-existent PID for folks who monitor PIDs (not a great idea anyways). Unfortunately, this change also brings us back to the case where having a PID later (for other process monitors) is beneficial but more unicorn releases exist where we write the PID early. Thanks to Jimmy Soho for reporting this issue. ref: <CAHStS5gFYcPBDxkVizAHrOeDKAkjT69kruFdgaY0CbB+vLbK8Q@mail.gmail.com> This partially reverts 7d6ac0c17eb29a00a5b74099dbb3d4d015999f27 Folks: please monitor your app with HTTP requests rather than checking processes, a stuck/wedged Ruby VM is still a running one.
2013-11-07stream_input: avoid IO#close on client disconnect
This can avoid IOError from being seen by the application, and also reduces points where IO#close may be called. This is a good thing if we eventually port this code into a low-level server like cmogstored where per-client memory space is defined by FD number of a client. Reported-by: Andrew Hobson <ahobson@gmail.com>
2013-11-01construct listener_fds Hash in 1.8.6 compatible way
This renables the ability for Ruby 1.8.6 environments to perform reexecs [ew: clarified this is for 1.8.6, favor literal {} over Hash.new, tweaked LISTENERS.map => LISTENERS.each, thanks to Hleb Valoshka ] Signed-off-by: Eric Wong <normalperson@yhbt.net>
2013-10-29configurator: validate :reuseport for boolean-ess
In case we (and Linux) supports other values in the future, we can update it then. Until now, ensure users only set true or false for this option.
2013-10-26license: allow all future versions of the GNU GPL
There is currently no GPLv4, so this change has no effect at the moment. In case the GPLv4 arrives and I am not alive to approve/review it, the lesser of evils is have give blanket approval of all future GPL versions (as published by the FSF). The worse evil is to be stuck with a license which cannot guarantee the Free-ness of this project in the future. This unfortunately means the FSF can theoretically come out with license terms I do not agree with, but the GPLv2 and GPLv3 will always be an option to all users.
2013-10-25http_server: fixup comments for PID file renaming
Thanks to Hongli Lai for noticing my typo. While we're at it, finish up a halfway-written comment for the EXDEV case
2013-10-25avoid IO_PURGATORY on Ruby 1.9+
Ruby 1.9 and later includes IO#autoclose=, so we can use it and prevent some dead IO objects from hanging around.
2013-10-25support SO_REUSEPORT on new listeners (:reuseport)
This allows users to start an independent instance of unicorn on a the same port as a running unicorn (as long as both instances use :reuseport). ref: https://lwn.net/Articles/542629/
2013-10-24attempt to rename PID file when possible
This will preserve mtime on successful renames for comparisions. While we're at it, avoid writing the new PID until the listeners are inherited successfully. This can be useful to avoid accidentally clobbering a good PID if binding the listener or building the app (preload_app==true) fails
2013-10-20workaround reopen atomicity issues for stdio vs non-stdio
In multithreaded apps, we must use dup2/dup3 with a temporary descriptor to reopen log files atomically. This is the only way to protect all concurrent userspace access to a file when reopening. ref: http://bugs.ruby-lang.org/issues/9036 ref: yahns commit bcb10abe53cfb1d6a8ef7daef59eb10ced397c8a
2013-08-17http_server: improve handling of client-triggerable socket errors
We do not attempt to write HTTP responses for socket errors if clients disconnect from us unexpectedly. Additionally, we do not hide backtraces EINVAL/EBADF errors, since they are indicative of real bugs which must be fixed. We do continue to hide hide EOF, ECONNRESET, ENOTCONN, and EPIPE because clients (even "friendly") ones will break connections due to client crashes or network failure (which is common for me :P), and the backtraces from those will cause excessive logging and even become a DoS vector.
2013-06-07Make -N/--no-default-middleware option work
This fixes the -N (a.k.a. --no-defaut-middleware) option, which was not working. The problem was that Unicorn::Configurator::RACKUP is cleared before the lambda returned by Unicorn.builder is run, which means that checking whether the :no_default_middleware option was set from the lambda could not detect anything. This patch copies it to a local variable that won't get clobbered, restoring the feature. [ew: squashed test commit into the fix, whitespace fixes] Signed-off-by: Eric Wong <normalperson@yhbt.net>
2013-04-05doc: update documentation for systemd + PrivateTmp users
The PrivateTmp feature of systemd breaks the usage of /tmp for the shared Unix domain socket between nginx and unicorn, so discourage the use of /tmp in that case. While we're at it, use consistent paths for everything and use an obviously intended-for-user-customization "/path/to" prefix instead of "/tmp" ML-Ref: CAKLVLx_t+9zWMhquMWDfStrxS7xrNoGmN0ZDsjSCUE=VxU+oyQ@mail.gmail.com Reported-by: David Wilkins <dwilkins@conecuh.com>
2013-02-09http_request: drop conditional assignment for hijack
As far as I can tell, this was never necessary.
2013-02-08http_request: remove FIXME for rack.version clarification
commit a9474624a148fe58e0944664190b259787dcf51e in rack.git
2013-02-08auto-generate Unicorn::Const::UNICORN_VERSION
This DRYs out our code and prevents snafus like the 4.6.0 release where UNICORN_VERSION stayed at 4.5.0 Reported-by: Maurizio De Santis <m.desantis@morganspa.com>
2013-01-29Merge branch 'hijack'
* hijack: ignore normal Rack response at request-time hijack support for Rack hijack in request and response
2013-01-29Add -N or --no-default-middleware option.
This would prevent Unicorn from adding default middleware, as if RACK_ENV were always none. (not development nor deployment) This should also be applied to `rainbows' and `zbatery' as well. One of the reasons to add this is to avoid conflicting RAILS_ENV and RACK_ENV. It would be helpful in the case where a Rails application and Rack application are composed together, while we want Rails app runs under development and Rack app runs under none (if we don't want those default middleware), and we don't really want to make RAILS_ENV set to development and RACK_ENV to none because it might be confusing. Note that Rails would also look into RACK_ENV. Another reason for this is that only `rackup' would be inserting those default middleware. Both `thin' and `puma' would not do this, nor does Rack::Handler.get.run which is used in Sinatra. So using this option would make it work differently from `rackup' but somehow more similar to `thin' or `puma'. Discussion thread on the mailing list: http://rubyforge.org/pipermail/mongrel-unicorn/2013-January/001675.html Signed-off-by: Eric Wong <normalperson@yhbt.net>
2013-01-22ignore normal Rack response at request-time hijack
Once a connection is hijacked, we ignore it completely and leave the connection at the mercy of the application.
2013-01-22support for Rack hijack in request and response
Rack 1.5.0 (protocol version [1,2]) adds support for hijacking the client socket (removing it from the control of unicorn (or any other Rack webserver)). Tested with rack 1.5.0.
2012-12-07unicorn 4.5.0 - check_client_connection option v4.5.0
The new check_client_connection option allows unicorn to detect most disconnected local clients before potentially expensive application processing begins. This feature is useful for applications experiencing spikes of traffic leading to undesirable queue times, as clients will disconnect (and perhaps even retry, compounding the problem) before unicorn can even start processing the request. To enable this feature, add the following line to a unicorn config file: check_client_connection true This feature only works when nginx (or any other HTTP/1.0+ client) is on the same machine as unicorn. A huge thanks to Tom Burns for implementing and testing this change in production with real traffic (including mitigating an unexpected DoS attack). ref: http://mid.gmane.org/CAK4qKG3rkfVYLyeqEqQyuNEh_nZ8yw0X_cwTxJfJ+TOU+y8F+w@mail.gmail.com This release fixes broken Rainbows! compatibility in 4.5.0pre1.
2012-12-04fix const error responses for Rainbows!
Rainbows! relies on the ERROR_XXX_RESPONSE constants of unicorn 4.x. Changing the constants in unicorn 4.x will break existing versions of Rainbows!, so remove the dependency on the constants and generate the error response dynamically. Unlike Mongrel, unicorn is unlikely to see malicious traffic and thus unlikely to benefit from making error messages constant. For unicorn 5.x, we will drop these constants entirely. (Rainbows! most likely cannot support check_client_connection consistently across all concurrency models since some of them pessimistically buffer all writes in userspace. However, the extra concurrency of Rainbows! makes it less likely to be overloaded than unicorn, so this feature is likely less useful for Rainbows!)
2012-11-29unicorn 4.5.0pre1 - check_client_connection option v4.5.0pre1
The new check_client_connection option allows unicorn to detect most disconnected clients before potentially expensive application processing begins. This feature is useful for applications experiencing spikes of traffic leading to undesirable queue times, as clients will disconnect (and perhaps even retry, compounding the problem) before unicorn can even start processing the request. To enable this feature, add the following line to a unicorn config file: check_client_connection true A huge thanks to Tom Burns for implementing and testing this change in production with real traffic (including mitigating an unexpected DoS attack).
2012-11-29check_client_connection: document local-only requirement
In my testing, only dropped clients over Unix domain sockets or loopback TCP were detected with this option. Since many nginx+unicorn combinations run on the same host, this is not a problem. Furthermore, tcp_nodelay:true appears to work over loopback, so remove the requirement for tcp_nodelay:false.
2012-11-29Begin writing HTTP request headers early to detect disconnected clients
This patch checks incoming connections and avoids calling the application if the connection has been closed. It works by sending the beginning of the HTTP response before calling the application to see if the socket can successfully be written to. By enabling this feature users can avoid wasting application rendering time only to find the connection is closed when attempting to write, and throwing out the result. When a client disconnects while being queued or processed, Nginx will log HTTP response 499 but the application will log a 200. Enabling this feature will minimize the time window during which the problem can arise. The feature is disabled by default and can be enabled by adding 'check_client_connection true' to the unicorn config. [ew: After testing this change, Tom Burns wrote: So we just finished the US Black Friday / Cyber Monday weekend running unicorn forked with the last version of the patch I had sent you. It worked splendidly and helped us handle huge flash sales without increased response time over the weekend. Whereas in previous flash traffic scenarios we would see the number of HTTP 499 responses grow past the number of real HTTP 200 responses, over the weekend we saw no growth in 499s during flash sales. Unexpectedly the patch also helped us ward off a DoS attack where the attackers were disconnecting immediately after making a request. ref: <CAK4qKG3rkfVYLyeqEqQyuNEh_nZ8yw0X_cwTxJfJ+TOU+y8F+w@mail.gmail.com> ] Signed-off-by: Eric Wong <normalperson@yhbt.net>
2012-10-11unicorn 4.4.0 - minor updates v4.4.0
Non-regular files are no longer reopened on SIGUSR1. This allows users to specify FIFOs as log destinations. TCP_NOPUSH/TCP_CORK is no longer set/unset by default. Use :tcp_nopush explicitly with the "listen" directive if you wish to enable TCP_NOPUSH/TCP_CORK. Listen sockets are now bound _after_ loading the application for preload_app(true) users. This prevents load balancers from sending traffic to an application server while the application is still loading. There are also minor test suite cleanups.
2012-10-01util: only consider regular files as logs
If a user specifies a non-regular file for stderr_path or stdout_path, we should not attempt to reopen or chown it. This should also allow users to specify FIFOs as log destinations.
2012-08-06do not touch TCP_NOPUSH/TCP_CORK at all by default
On a certain FreeBSD 8.1 installation, explicitly setting TCP_NOPUSH to zero (off) can cause EADDRNOTAVAIL errors and also resets the listen backlog to 5. Enabling TCP_NOPUSH explicitly did not exhibit this issue for the user who (privately) reported this issue. To be on the safe side, we won't set/unset TCP_NOPUSH/TCP_CORK at all, which will leave it off on all current systems.
2012-08-02bind listeners after loading for preload_app users
In the case where preload_app is true, delay binding new listeners until after loading the application. Some applications have very long load times (especially Rails apps with Ruby 1.9.2). Binding listeners early may cause a load balancer to incorrectly believe the unicorn workers are ready to serve traffic even while the app is being loaded. Once a listener is bound, connect() requests from the load balancer succeed until the listen backlog is filled. This allows requests to pile up for a bit (depending on backlog size) before getting rejected by the kernel. By the time the application is loaded and ready-to-run, requests in the listen backlog are likely stale and not useful to process. Processes inheriting listeners do not suffer this effect, as the old process should still be capable of serving new requests. This change does not improve the situation for the preload_app=false (default) use case. There may not be a solution for preload_app=false users using large applications. Fortunately Ruby 1.9.3+ improves load times of large applications significantly over 1.9.2 so this should be less of a problem in the future. Reported via private email sent on 2012-06-29T22:59:10Z
2012-04-29unicorn 4.3.1 - shutdown() fixes v4.3.1
* Call shutdown(2) if a client EOFs on us during upload. We can avoid holding a socket open if the Rack app forked a process during uploads. * ignore potential Errno::ENOTCONN errors (from shutdown(2)). Even on LANs, connections can occasionally be accept()-ed but be unusable afterwards. Thanks to Joel Nimety <jnimety@continuity.net>, Matt Smith <matt@nearapogee.com> and George <lists@southernohio.net> on the mongrel-unicorn@rubyforge.org mailing list for their feedback and testing for this release.
2012-04-27stream_input: call shutdown(2) if a client EOFs on us
In case the Rack app forks before a client upload is complete, shutdown(2) the socket to ensure the client isn't attempting to read from us (even if it explicitly stopped writes).
2012-04-27http_server: ignore ENOTCONN (mostly from shutdown(2))
Since there's nothing unicorn can do to avoid this error on unconnected/halfway-connected clients, ignoring ENOTCONN is a safe bet. Rainbows! has long had this rescue as it called getpeername(2) on untrusted sockets
2012-04-17unicorn 4.3.0 - minor fixes and updates v4.3.0
* PATH_INFO (aka REQUEST_PATH) increased to 4096 (from 1024). This allows requests with longer path components and matches the system PATH_MAX value common to GNU/Linux systems for serving filesystem components with long names. * Apps that fork() (but do not exec()) internally for background tasks now indicate the end-of-request immediately after writing the Rack response. Thanks to Hongli Lai, Lawrence Pit, Patrick Wenger and Nuo Yan for their valuable feedback for this release.
2012-04-12shutdown client socket for apps which fork in background
Previously we relied on implicit socket shutdown() from the close() syscall. However, some Rack applications fork() (without calling exec()), creating a potentially long-lived reference to the underlying socket in a child process. This ends up causing nginx to wait on the socket shutdown when the child process exits. Calling shutdown() explicitly signals nginx (or whatever client) that the unicorn worker is done with the socket, regardless of the number of FD references to the underlying socket in existence. This was not an issue for applications which exec() since FD_CLOEXEC is always set on the client socket. Thanks to Patrick Wenger for discovering this. Thanks to Hongli Lai for the tip on using shutdown() as is done in Passenger. ref: http://mid.gmane.org/CAOG6bOTseAPbjU5LYchODqjdF3-Ez4+M8jo-D_D2Wq0jkdc4Rw@mail.gmail.com
2012-04-11misc documentation spelling fixes
Found via rdoc-spellcheck
2012-03-26unicorn 4.2.1 - minor fix and doc updates v4.2.1
* Stale pid files are detected if a pid is recycled by processes belonging to another user, thanks to Graham Bleach. * nginx example config updates thanks to to Eike Herzbach. * KNOWN_ISSUES now documents issues with apps/libs that install conflicting signal handlers.
2012-03-20log EPERM errors from invalid pid files
In some cases, EPERM may indicate a real configuration problem, but it can also just mean the pid file is stale.
2012-02-29Start the server if another user has a PID matching our stale pidfile.
If unicorn doesn't get terminated cleanly (for example if the machine has its power interrupted) and the pid in the pidfile gets used by another process, the current unicorn code will exit and not start a server. This tiny patch fixes that behaviour. Acked-by: Eric Wong <normalperson@yhbt.net>
2012-01-28unicorn 4.2.0 v4.2.0
The GPLv3 is now an option to the Unicorn license. The existing GPLv2 and Ruby-only terms will always remain options, but the GPLv3 is preferred. Daemonization is correctly detected on all terminals for development use (Brian P O'Rourke). Unicorn::OobGC respects applications that disable GC entirely during application dispatch (Yuichi Tateno). Many test fixes for OpenBSD, which may help other *BSDs, too. (Jeremy Evans). There is now _optional_ SSL support (via the "kgio-monkey" RubyGem). On fast, secure LANs, SSL is only intended for detecting data corruption that weak TCP checksums cannot detect. Our SSL support is remains unaudited by security experts. There are also some minor bugfixes and documentation improvements. Ruby 2.0.0dev also has a copy-on-write friendly GC which can save memory when combined with "preload_app true", so if you're in the mood, start testing Unicorn with the latest Ruby!
2011-12-13cleanup exception handling on SIGUSR1
No need to duplicate logic here
2011-12-13quiet possible IOError from SIGUSR1 (reopen logs)
It's possible for a SIGUSR1 signal to be received in the worker immediately before calling IO.select. In that case, do not clutter logging with IOError and just process the reopen log request.
2011-12-05socket_helper: fix grammerr fail
Oops :x
2011-12-05socket_helper: set SO_KEEPALIVE on TCP sockets
Even LANs can break or be unreliable sometimes and socket disconnect messages get lost, which means we fall back to the global (kill -9) timeout in Unicorn. While the default global timeout is much shorter (60s) than typical TCP timeouts, some HTTP application dispatches take much I/O or computational time (streaming many gigabytes), so the global timeout becomes ineffective. Under Linux, sysadmins are encouraged to lower the default net.ipv4.tcp_keepalive_* knobs in sysctl. There should be similar knobs in other operating systems (the default keepalive intervals are usually ridiculously high, too high for anything). When the listen socket has SO_KEEPALIVE set, the flag should be inherited by accept()-ed sockets.
2011-12-05socket_helper: remove out-of-date comment for TCP_NODELAY
We favor low latency and consistency with the Unix socket behavior even with TCP.
2011-11-15configurator: limit timeout to 30 days
There's no practical difference between a timeout of 30 days and 68 years from an HTTP server standpoint. POSIX limits us to 31 days, actually, but there could be rounding error with floats used in Ruby time calculations and there's no real difference between 30 and 31 days, either... Thanks to Jeremy Evans for pointing out large values will throw EINVAL (on select(2) under OpenBSD with Ruby 1.9.3 and RangeError on older Rubies.
2011-10-04OobGC: force GC.start
[ew: we need to explicitly enable GC if it is disabled and respect applications that disable GC] Acked-by: Eric Wong <normalperson@yhbt.net>
2011-09-15add preliminary SSL support
This will also be the foundation of SSL support in Rainbows! and Zbatery. Some users may also want to use this in Unicorn on LANs to meet certain security/auditing requirements. Of course, Nightmare! (in whatever form) should also be able to use it.
2011-09-14Detect daemonization via configuration.
This prevents the stopping of all workers by SIGWINCH if you're using a windowing system that will 'exec' unicorn from a process that's already in a process group. Acked-by: Eric Wong <normalperson@yhbt.net>
2011-09-09http_server: update comment on tick == 0
The old comment was confusing. We only zero the tick counter when forking because application loading can take a long time. Otherwise, it's always updated. ref: http://mid.gmane.org/20110908191352.GA25251@dcvr.yhbt.net