summary refs log tree commit homepage
AgeCommit message (Collapse)AuthorFilesLines
2012-12-07unicorn 4.5.0 - check_client_connection option v4.5.0Eric Wong2-2/+2
The new check_client_connection option allows unicorn to detect most disconnected local clients before potentially expensive application processing begins. This feature is useful for applications experiencing spikes of traffic leading to undesirable queue times, as clients will disconnect (and perhaps even retry, compounding the problem) before unicorn can even start processing the request. To enable this feature, add the following line to a unicorn config file: check_client_connection true This feature only works when nginx (or any other HTTP/1.0+ client) is on the same machine as unicorn. A huge thanks to Tom Burns for implementing and testing this change in production with real traffic (including mitigating an unexpected DoS attack). ref: This release fixes broken Rainbows! compatibility in 4.5.0pre1.
2012-12-04gemspec: enable licenses metadata attributePeter Marsh1-1/+1
This enables compatibility with metadata scanners such as LicenseFinder[1]. The previously commented-out accessor was commented out in September 2009 when ancient RubyGems were more prevalent. By now (December 2012), those ancient versions of RubyGems are unlikely to be around. [1] [ew: rewritten commit message] Signed-off-by: Eric Wong <>
2012-12-04README: clarify license and copyrightEric Wong1-4/+4
Since Ruby 1.9.3, (Matz) Ruby is licensed under a 2-clause BSDL. Thus we need to clarify we inherited the license terms from Ruby 1.8 to prevent misunderstanding. (The Ruby license change cannot alter the license of other projects automatically) Since we added the GPLv3 as an additional license in 2011, the license terms of unicorn no longer matches Mongrel 1.1.5. This is NOT a change to the unicorn license at all, just a wording clarification.
2012-12-04fix const error responses for Rainbows!Eric Wong3-12/+17
Rainbows! relies on the ERROR_XXX_RESPONSE constants of unicorn 4.x. Changing the constants in unicorn 4.x will break existing versions of Rainbows!, so remove the dependency on the constants and generate the error response dynamically. Unlike Mongrel, unicorn is unlikely to see malicious traffic and thus unlikely to benefit from making error messages constant. For unicorn 5.x, we will drop these constants entirely. (Rainbows! most likely cannot support check_client_connection consistently across all concurrency models since some of them pessimistically buffer all writes in userspace. However, the extra concurrency of Rainbows! makes it less likely to be overloaded than unicorn, so this feature is likely less useful for Rainbows!)
2012-11-29unicorn 4.5.0pre1 - check_client_connection option v4.5.0pre1Eric Wong2-2/+2
The new check_client_connection option allows unicorn to detect most disconnected clients before potentially expensive application processing begins. This feature is useful for applications experiencing spikes of traffic leading to undesirable queue times, as clients will disconnect (and perhaps even retry, compounding the problem) before unicorn can even start processing the request. To enable this feature, add the following line to a unicorn config file: check_client_connection true A huge thanks to Tom Burns for implementing and testing this change in production with real traffic (including mitigating an unexpected DoS attack).
2012-11-29check_client_connection: document local-only requirementEric Wong2-7/+8
In my testing, only dropped clients over Unix domain sockets or loopback TCP were detected with this option. Since many nginx+unicorn combinations run on the same host, this is not a problem. Furthermore, tcp_nodelay:true appears to work over loopback, so remove the requirement for tcp_nodelay:false.
2012-11-29Begin writing HTTP request headers early to detect disconnected clientsTom Burns9-10/+116
This patch checks incoming connections and avoids calling the application if the connection has been closed. It works by sending the beginning of the HTTP response before calling the application to see if the socket can successfully be written to. By enabling this feature users can avoid wasting application rendering time only to find the connection is closed when attempting to write, and throwing out the result. When a client disconnects while being queued or processed, Nginx will log HTTP response 499 but the application will log a 200. Enabling this feature will minimize the time window during which the problem can arise. The feature is disabled by default and can be enabled by adding 'check_client_connection true' to the unicorn config. [ew: After testing this change, Tom Burns wrote: So we just finished the US Black Friday / Cyber Monday weekend running unicorn forked with the last version of the patch I had sent you. It worked splendidly and helped us handle huge flash sales without increased response time over the weekend. Whereas in previous flash traffic scenarios we would see the number of HTTP 499 responses grow past the number of real HTTP 200 responses, over the weekend we saw no growth in 499s during flash sales. Unexpectedly the patch also helped us ward off a DoS attack where the attackers were disconnecting immediately after making a request. ref: <> ] Signed-off-by: Eric Wong <>
2012-11-13tests: remove assert_nothing_raised (part 2)Eric Wong11-247/+166
assert_nothing_raised ends up hiding errors and backtraces, making things harder to debug. Since Test::Unit already fails on uncaught exceptions, there is no need to assert on the lack of exceptions for a successful test run. This is a followup to commit 5acf5522295c947d3118926d1a1077007f615de9
2012-10-11Rakefile: fm_update task updated for HTTPSEric Wong1-2/+2 now requires HTTPS.
2012-10-11unicorn 4.4.0 - minor updates v4.4.0Eric Wong2-2/+2
Non-regular files are no longer reopened on SIGUSR1. This allows users to specify FIFOs as log destinations. TCP_NOPUSH/TCP_CORK is no longer set/unset by default. Use :tcp_nopush explicitly with the "listen" directive if you wish to enable TCP_NOPUSH/TCP_CORK. Listen sockets are now bound _after_ loading the application for preload_app(true) users. This prevents load balancers from sending traffic to an application server while the application is still loading. There are also minor test suite cleanups.
2012-10-01util: only consider regular files as logsEric Wong1-0/+1
If a user specifies a non-regular file for stderr_path or stdout_path, we should not attempt to reopen or chown it. This should also allow users to specify FIFOs as log destinations.
2012-08-06avoid assert_nothing_raised in unit testsEric Wong2-94/+79
It's better to show errors and backtraces when stuff breaks
2012-08-06do not touch TCP_NOPUSH/TCP_CORK at all by defaultEric Wong1-7/+7
On a certain FreeBSD 8.1 installation, explicitly setting TCP_NOPUSH to zero (off) can cause EADDRNOTAVAIL errors and also resets the listen backlog to 5. Enabling TCP_NOPUSH explicitly did not exhibit this issue for the user who (privately) reported this issue. To be on the safe side, we won't set/unset TCP_NOPUSH/TCP_CORK at all, which will leave it off on all current systems.
2012-08-02bind listeners after loading for preload_app usersEric Wong4-2/+49
In the case where preload_app is true, delay binding new listeners until after loading the application. Some applications have very long load times (especially Rails apps with Ruby 1.9.2). Binding listeners early may cause a load balancer to incorrectly believe the unicorn workers are ready to serve traffic even while the app is being loaded. Once a listener is bound, connect() requests from the load balancer succeed until the listen backlog is filled. This allows requests to pile up for a bit (depending on backlog size) before getting rejected by the kernel. By the time the application is loaded and ready-to-run, requests in the listen backlog are likely stale and not useful to process. Processes inheriting listeners do not suffer this effect, as the old process should still be capable of serving new requests. This change does not improve the situation for the preload_app=false (default) use case. There may not be a solution for preload_app=false users using large applications. Fortunately Ruby 1.9.3+ improves load times of large applications significantly over 1.9.2 so this should be less of a problem in the future. Reported via private email sent on 2012-06-29T22:59:10Z
2012-07-28remove Rails-oriented integration testsEric Wong100-1598/+2
It's too much overhead to keep Rails-specific tests working, especially when it's hauling in an ancient version of SQLite3. Since Rails 3 has settled down with Rack and unicorn_rails is unlikely to need changing in the future, we can drop these tests.
2012-04-29unicorn 4.3.1 - shutdown() fixes v4.3.1Eric Wong2-2/+2
* Call shutdown(2) if a client EOFs on us during upload. We can avoid holding a socket open if the Rack app forked a process during uploads. * ignore potential Errno::ENOTCONN errors (from shutdown(2)). Even on LANs, connections can occasionally be accept()-ed but be unusable afterwards. Thanks to Joel Nimety <>, Matt Smith <> and George <> on the mailing list for their feedback and testing for this release.
2012-04-29isolate_for_tests: upgrade to kgio-monkey 0.4.0Eric Wong1-1/+1
Seems to work well enough...
2012-04-27stream_input: call shutdown(2) if a client EOFs on usEric Wong1-1/+5
In case the Rack app forks before a client upload is complete, shutdown(2) the socket to ensure the client isn't attempting to read from us (even if it explicitly stopped writes).
2012-04-27http_server: ignore ENOTCONN (mostly from shutdown(2))Eric Wong1-1/+2
Since there's nothing unicorn can do to avoid this error on unconnected/halfway-connected clients, ignoring ENOTCONN is a safe bet. Rainbows! has long had this rescue as it called getpeername(2) on untrusted sockets
2012-04-17unicorn 4.3.0 - minor fixes and updates v4.3.0Eric Wong2-2/+2
* PATH_INFO (aka REQUEST_PATH) increased to 4096 (from 1024). This allows requests with longer path components and matches the system PATH_MAX value common to GNU/Linux systems for serving filesystem components with long names. * Apps that fork() (but do not exec()) internally for background tasks now indicate the end-of-request immediately after writing the Rack response. Thanks to Hongli Lai, Lawrence Pit, Patrick Wenger and Nuo Yan for their valuable feedback for this release.
2012-04-17tests: set executable bit on integration shell scriptsEric Wong4-0/+0
These should be made executable for ease-of-understanding and consistency, regardless of whether we actually execute them.
2012-04-17http: increase REQUEST_PATH maximum length to 4KEric Wong1-2/+2
The previous REQUEST_PATH limit of 1024 is relatively small and some users encounter problems with long URLs. 4K is a common limit for PATH_MAX on modern GNU/Linux systems and REQUEST_PATH is likely to translate to a filesystem path name. Thanks to Nuo Yan <> and Lawrence Pit <> for their feedback on this issue. ref:
2012-04-12shutdown client socket for apps which fork in backgroundEric Wong4-0/+46
Previously we relied on implicit socket shutdown() from the close() syscall. However, some Rack applications fork() (without calling exec()), creating a potentially long-lived reference to the underlying socket in a child process. This ends up causing nginx to wait on the socket shutdown when the child process exits. Calling shutdown() explicitly signals nginx (or whatever client) that the unicorn worker is done with the socket, regardless of the number of FD references to the underlying socket in existence. This was not an issue for applications which exec() since FD_CLOEXEC is always set on the client socket. Thanks to Patrick Wenger for discovering this. Thanks to Hongli Lai for the tip on using shutdown() as is done in Passenger. ref:
2012-04-12t/ use larger keys for testsEric Wong1-4/+4
This seems required for TLSv1.2 under OpenSSL 1.0.1
2012-04-11misc documentation spelling fixesEric Wong3-5/+5
Found via rdoc-spellcheck
2012-03-26unicorn 4.2.1 - minor fix and doc updates v4.2.1Eric Wong2-2/+2
* Stale pid files are detected if a pid is recycled by processes belonging to another user, thanks to Graham Bleach. * nginx example config updates thanks to to Eike Herzbach. * KNOWN_ISSUES now documents issues with apps/libs that install conflicting signal handlers.
2012-03-24tests: depend on kgio 2.7.4Eric Wong1-1/+1
This latest version of kgio improves portability to FreeBSD-based systems.
2012-03-20log EPERM errors from invalid pid filesEric Wong1-1/+4
In some cases, EPERM may indicate a real configuration problem, but it can also just mean the pid file is stale.
2012-03-20KNOWN_ISSUES: document signal conflicts in libs/appsEric Wong1-0/+5
Jeffrey Yeung confirmed this issue on the mailing list. ref: <>
2012-03-20examples/nginx.conf: use $scheme instead of hard-coded "https"Eric Wong1-3/+3
This adds a little more flexibility to the nginx config, especially as protocols (e.g. SPDY) become more prevalent. Suggested-by: Eike Herzbach <>
2012-03-20examples/nginx.conf: remove redundant wordEric Wong1-1/+1
From: Eike Herzbach <>
2012-02-29Start the server if another user has a PID matching our stale pidfile.Graham Bleach1-1/+1
If unicorn doesn't get terminated cleanly (for example if the machine has its power interrupted) and the pid in the pidfile gets used by another process, the current unicorn code will exit and not start a server. This tiny patch fixes that behaviour. Acked-by: Eric Wong <>
2012-01-28unicorn 4.2.0 v4.2.0Eric Wong2-2/+2
The GPLv3 is now an option to the Unicorn license. The existing GPLv2 and Ruby-only terms will always remain options, but the GPLv3 is preferred. Daemonization is correctly detected on all terminals for development use (Brian P O'Rourke). Unicorn::OobGC respects applications that disable GC entirely during application dispatch (Yuichi Tateno). Many test fixes for OpenBSD, which may help other *BSDs, too. (Jeremy Evans). There is now _optional_ SSL support (via the "kgio-monkey" RubyGem). On fast, secure LANs, SSL is only intended for detecting data corruption that weak TCP checksums cannot detect. Our SSL support is remains unaudited by security experts. There are also some minor bugfixes and documentation improvements. Ruby 2.0.0dev also has a copy-on-write friendly GC which can save memory when combined with "preload_app true", so if you're in the mood, start testing Unicorn with the latest Ruby!
2012-01-28doc: update doc for Ruby 2.0.0dev CoW-friendlinessEric Wong2-4/+4
Ruby 2.0.0dev is the future and includes a CoW-friendly GC, so we shall encourage folks to give Ruby 2.0.0dev a spin.
2012-01-27script/isolate_for_tests: disable sqlite3-ruby for Ruby 2.0.0devEric Wong1-1/+3
We don't need it because we don't test old Rails with bleeding edge Ruby.
2012-01-27disable old Rails tests for Ruby 2.0.0Eric Wong1-1/+4
I doubt anybody would attempt to run ancient, unsupported versions of Rails on the latest (unreleased, even) versions of Ruby...
2012-01-27script/isolate_for_tests: update to kgio 2.7.2Eric Wong1-1/+1
Again, we test with the latest version.
2012-01-24update tests for Rack 1.4.1Eric Wong1-1/+1
Trying to ensure things always work with the latest version.
2012-01-08Rakefile: swap URL for freecode.comEric Wong1-1/+1
2011-12-28update tests for rack 1.4.0Eric Wong1-1/+1
It's the latest and greatest version, so ensure everything works with it.
2011-12-17http: test case for "Connection: TE"Eric Wong1-0/+8
We need to be sure we don't barf on this header.
2011-12-13cleanup exception handling on SIGUSR1Eric Wong1-3/+1
No need to duplicate logic here
2011-12-13quiet possible IOError from SIGUSR1 (reopen logs)Eric Wong1-0/+1
It's possible for a SIGUSR1 signal to be received in the worker immediately before calling In that case, do not clutter logging with IOError and just process the reopen log request.
2011-12-05socket_helper: fix grammerr failEric Wong1-1/+1
Oops :x
2011-12-05socket_helper: set SO_KEEPALIVE on TCP socketsEric Wong1-0/+4
Even LANs can break or be unreliable sometimes and socket disconnect messages get lost, which means we fall back to the global (kill -9) timeout in Unicorn. While the default global timeout is much shorter (60s) than typical TCP timeouts, some HTTP application dispatches take much I/O or computational time (streaming many gigabytes), so the global timeout becomes ineffective. Under Linux, sysadmins are encouraged to lower the default net.ipv4.tcp_keepalive_* knobs in sysctl. There should be similar knobs in other operating systems (the default keepalive intervals are usually ridiculously high, too high for anything). When the listen socket has SO_KEEPALIVE set, the flag should be inherited by accept()-ed sockets.
2011-12-05socket_helper: remove out-of-date comment for TCP_NODELAYEric Wong1-1/+0
We favor low latency and consistency with the Unix socket behavior even with TCP.
2011-12-05bump dependenciesEric Wong2-6/+6
We should always be testing with the newest available versions to watch for incompatibilities, even if we don't /require/ the latest ones to run.
2011-11-15tests: try to set a shorter path for Unix domain socketsEric Wong1-2/+11
We're only allowed 108 bytes for Unix domain sockets. mktemp(1) usually generates path names of reasonable length and we rely on it anyways.
2011-11-15tests: just use the sha1sum implemented in RubyEric Wong2-14/+2
The output of SHA1 command-line tools is too unstable and I'm more comfortable with Ruby 1.9 encoding support than I was in 2009. Jeremy Evans noted the output of "openssl sha1" has changed since I last used it.
2011-11-15test_helper: ensure test client connects to valid addressJeremy Evans1-0/+1
You can listen on, but trying to connect to it doesn't work well on OpenBSD. Acked-by: Eric Wong <>