summary refs log tree commit homepage
AgeCommit message (Collapse)AuthorFilesLines
2015-08-22stream_input: favor String#clear over String#replaceEric Wong1-3/+3
We no longer need Ruby 1.8 compatibility, so use String#clear to reduce argument passing and code size.
2015-07-15doc: remove references to old serversEric Wong15-93/+78
They'll continue to be maintained, but we're no longer advertising them. Also, favor lowercase "unicorn" while we're at it since that matches the executable and gem name to avoid unnecessary escaping for RDoc.
2015-07-15configurator: document net.core.somaxconn sysctl dependencyEric Wong1-0/+5
Linux users are effectively capped to 128 on stock installations and may wonder why connections get rejected with overloaded apps sooner rather than later.
2015-07-08test_exec: disable systemd inheritance testEric Wong1-1/+3
Turns out ruby does have trouble emulating systemd, for now: [ruby-core:69895] https://bugs.ruby-lang.org/issues/11336 When we re-enable this test, we'll only enable it for fixed Rubies. The actual socket inheritance functionality works in any version of Ruby, of course, it's just that emulating systemd won't work until ruby-core fixes mainline Ruby.
2015-07-06unicorn 5.0.0.pre2 - another prerelease! v5.0.0.pre2Eric Wong1-1/+1
There is a minor TCP socket options are now applied to inherited sockets, and we have native support for inheriting sockets from systemd (by emulating the sd_listen_fds(3) function). Dynamic changes in the application to Rack::Utils::HTTP_STATUS codes is now supported, so you can use your own custom status lines. Ruby 2.2 and later is now favored for performance. Optimizations by using constants which made sense in earlier versions of Ruby are gone: so users of old Ruby versions will see performance regressions. Ruby 2.2 users should see the same or better performance, and we have less code as a result. * doc: update some invalid URLs * apply TCP socket options on inherited sockets * reflect changes in Rack::Utils::HTTP_STATUS_CODES * reduce constants and optimize for Ruby 2.2 * http_response: reduce size of multi-line header path * emulate sd_listen_fds for systemd support * test/unit/test_response.rb: compatibility with older test-unit This also includes all changes in unicorn 5.0.0.pre1: http://bogomips.org/unicorn-public/m/20150615225652.GA16164@dcvr.yhbt.net.html
2015-07-05test/unit/test_response.rb: compatibility with older test-unitEric Wong1-1/+1
assert_predicate really isn't that useful even if it seems preferred in another project I work on. Avoid having folks download the latest test-unit if they're on an old version of Ruby (e.g. 1.9.3) which bundled it.
2015-07-05emulate sd_listen_fds for systemd supportEric Wong2-3/+37
systemd socket emulation shares FDs across execve, just like the built-in SIGUSR2 upgrade process in unicorn. Thus it is easy to support inheriting sockets from systemd. Tested-by: Christos Trochalakis <yatiohi@ideopolis.gr>
2015-06-30http_response: reduce size of multi-line header pathEric Wong1-1/+1
This should save over 100 bytes of bytecode overhead due to reduced method dispatch points. The performance difference when this is actually hit could go either way depending on how String#<< and realloc(3) interact, but it's uncommon enough that nobody is expected to notice either way.
2015-06-30reduce constants and optimize for Ruby 2.2Eric Wong2-16/+9
Ruby (MRI) 2.1 optimizes allocations away on String#freeze with literal strings. Furthermore, Ruby 2.2 optimizes away literal string allocations when they are used as arguments to Hash#[] and Hash#[]= Thus we can avoid expensive constant lookups and cache overhead by taking advantage of advancements in Ruby. Since Ruby 2.2 has been out for 7 months, now; it ought to be safe to introduce minor performance regressions for folks using older Rubies (1.9.3+ remains supported) to benefit folks on the latest Ruby. This should recover the performance lost in the "reflect changes in Rack::Utils::HTTP_STATUS_CODES" change in synthetic benchmarks.
2015-06-30reflect changes in Rack::Utils::HTTP_STATUS_CODESEric Wong2-8/+27
Applications may want to alter the message associated with HTTP status codes in Rack::Utils::HTTP_STATUS_CODES. Avoid memoizing status lines ahead-of-time Note: this introduces a minor performance regression, but ought to be unnoticeable unless you're running "Hello world"-type apps.
2015-06-27apply TCP socket options on inherited socketsEric Wong1-2/+3
TCP socket options are now set when inheriting existing sockets from a parent process. I'm fairly certain all the TCP setsockopt knobs we use are idempotent and harmless to change. If anything, the only directive I'd be uncomfortable changing is shortening the listen(2) (aka :backlog) size, but we've always changed that anyways since it also applies to UNIX sockets. Note: removing a configuration knob in a unicorn config file can not reset the value to the OS-provided default setting. Inherited sockets must use a new setting to override existing ones. (or the socket needs to be closed and re-created in the process launcher before unicorn inherits it). Noticed-by: Christos Trochalakis <yatiohi@ideopolis.gr> <20150626114129.GA25883@luke.ws.skroutz.gr>
2015-06-26doc: update some invalid URLsEric Wong8-13/+15
Most of these were found by the `linkchecker' package in Debian.
2015-06-15unicorn 5.0.0.pre1 - incompatible changes! v5.0.0.pre1Eric Wong1-1/+1
This release finally drops Ruby 1.8 support and requires Ruby 1.9.3 or later. The horrible "Status:" header in our HTTP response is finally gone, saving at least 16 precious bytes in every single HTTP response. Under Ruby 2.1 and later, the monotonic clock is used for timeout handling for better accuracy. Several experimental, unused and undocumented features are removed. There's also tiny, minor performance and memory improvements from dropping 1.8 compatibility, but probably nothing noticeable on a typical real-life (bloated) app. The biggest performance improvement we made was to our website by switching to olddoc. Depending on connection speed, latency, and renderer performance, it typically loads two to four times faster. Finally, for the billionth time: unicorn must never be exposed to slow clients, as it will never ever use new-fangled things like non-blocking socket I/O, threads, epoll or kqueue. unicorn must be used with a fully-buffering reverse proxy such as nginx for slow clients. * ISSUES: update with mailing list subscription * GIT-VERSION-GEN: start 5.0.0 development * http: remove xftrust options * FAQ: add entry for Rails autoflush_log * dev: remove isolate dependency * unicorn.gemspec: depend on test-unit 3.0 * http_response: remove Status: header * remove RubyForge and Freecode references * remove mongrel.rubyforge.org references * http: remove the keepalive requests limit * http: reduce parser from 72 to 56 bytes on 64-bit * examples: add run_once to before_fork hook example * worker: remove old tmp accessor * http_server: save 450+ bytes of memory on x86-64 * t/t0002-parser-error.sh: relax test for rack 1.6.0 * remove SSL support * tmpio: drop the "size" method * switch docs + website to olddoc * README: clarify/reduce references to unicorn_rails * gemspec: fixup olddoc migration * use the monotonic clock under Ruby 2.1+ * http: -Wshorten-64-to-32 warnings on clang * remove old inetd+git examples and exec_cgi * http: standalone require + reduction in binary size * GNUmakefile: fix clean gem build + reduce build cruft * socket_helper: reduce constant lookups and caching * remove 1.8, <= 1.9.1 fallback for missing IO#autoclose= * favor IO#close_on_exec= over fcntl in 1.9+ * use require_relative to reduce syscalls at startup * doc: update support status for Ruby versions * fix uninstalled testing and reduce require paths * test_socket_helper: do not depend on SO_REUSEPORT * favor "a.b(&:c)" form over "a.b { |x| x.c }" * ISSUES: add section for bugs in other projects * http_server: favor ivars over constants * explain 11 byte magic number for self-pipe * const: drop constants used by Rainbows! * reduce and localize constant string use * Links: mark Rainbows! as historical, reference yahns * save about 200 bytes of memory on x86-64 * http: remove deprecated reset method * http: remove experimental dechunk! method * socket_helper: update comments * doc: document UNICORN_FD in manpage * doc: document Etc.nprocessors for worker_processes * favor more string literals for cold call sites * tee_input: support for Rack::TempfileReaper middleware * support TempfileReaper in deployment and development envs * favor kgio_wait_readable for single FD over select * Merge tag 'v4.9.0' * http_request: support rack.hijack by default * avoid extra allocation for hijack proc creation * FAQ: add note about ECONNRESET errors from bodies * process SIGWINCH unless stdin is a TTY * ISSUES: discourage HTML mail strongly, welcome nyms * http: use rb_hash_clear in Ruby 2.0+ * http_response: avoid special-casing for Rack < 1.5 * www: install NEWS.atom.xml properly * http_server: remove a few more accessors and constants * http_response: simplify regular expression * move the socket into Rack env for hijacking * http: move response_start_sent into the C ext * FAQ: reorder bit on Rack 1.1.x and Rails 2.3.x * ensure body is closed during hijack
2015-06-10ensure body is closed during hijackEric Wong5-24/+20
Middlewares such as Rack::Lock (used by Rails) break badly unless the response body is closed on hijack, so we will close it to follow the lead of other popular Rack servers. While it's unclear if there's anybody using rack.hijack with unicorn, we'll try to emulate the behavior of other servers as much as possible. ref: https://github.com/ngauthier/tubesock/issues/10
2015-06-06FAQ: reorder bit on Rack 1.1.x and Rails 2.3.xEric Wong1-8/+8
These things were a while ago, and while apps using them still exist, they should not be near the top of the FAQ.
2015-06-06http: move response_start_sent into the C extEric Wong3-6/+35
Combined with the previous commit to eliminate the `@socket' instance variable, this eliminates the last instance variable in the Unicorn::HttpRequest class. Eliminating the last instance variable avoids the creation of a internal hash table used for implementing the "generic" instance variables found in non-pure-Ruby classes. Method entry overhead remains the same. While this change doesn't do a whole lot for unicorn memory usage where the HttpRequest is a singleton, it helps other HTTP servers which rely on this code where thousands of clients may be connected.
2015-06-06move the socket into Rack env for hijackingEric Wong1-2/+3
This avoids the expensive generic instance variable for @socket and exposes the socket as `env["unicorn.socket"]' to the Rack application. As as nice side-effect, applications may access `env["unicorn.socket"]' as part of the API may be useful for 3rd-party bits such as Raindrops::TCP_Info for reading the tcp_info struct on Linux-based systems. Yes, `env["unicorn.socket"]' is a proprietary API in unicorn! News at 11! But then again, unicorn is not the first Rack server to implement `env["#{servername}.socket"]', either...
2015-06-04http_response: simplify regular expressionEric Wong1-1/+1
It's ugly and less-readable to have redundant \z statements, and according to ObjectSpace.memsize_of, this saves 4 bytes on x86-64.
2015-06-04http_server: remove a few more accessors and constantsEric Wong2-29/+28
Unnecessarily exposed accessors and constants take up unnecessary memory in constant/method tables as well as using extra space in instruction sequences. Preforking servers like unicorn are a bloated pigs anyways, but saving a few hundred bytes here and there can add up and make them marginally less bad.
2015-06-01www: install NEWS.atom.xml properlyEric Wong1-0/+1
I just noticed the 4.9.0 release was not properly reflected in the Atom news feed at http://unicorn.bogomips.org/NEWS.atom.xml
2015-06-01http_response: avoid special-casing for Rack < 1.5Eric Wong1-13/+3
Rack 1.4 and earlier will soon die out, so avoid having extra, overengineered code and method dispatch to silently drop support for mis-hijacking with old Rack versions. This will cause improperly hijacked responses in all versions of Rack to fail, but allows properly hijacked responses to work regardless of Rack version. Followup-to: commit fdf09e562733f9509d275cb13c1c1a04e579a68a ("http_request: support rack.hijack by default")
2015-05-29http: use rb_hash_clear in Ruby 2.0+Eric Wong2-3/+19
Calling the function directly avoids the overhead of Ruby method table lookup and global method cache. The only downside is this is now hidden from tracers and cannot be overridden from Ruby, but I doubt anybody cares about that.
2015-05-20ISSUES: discourage HTML mail strongly, welcome nymsEric Wong1-1/+2
HTML email is too likely to be lost, so more strongly discourage it. While we're at it, make it clear we allow anonymous and pseudonymous contributions, unlike many projects nowadays.
2015-05-20process SIGWINCH unless stdin is a TTYEric Wong1-3/+3
Some process managers such as foreman and daemontools rely on unicorn not daemonizing, but we still want to be able to process SIGWINCH in that case. stdout and stderr may be redirected to a pipe (for cronolog or similar process), so those are less likely to be attached to a TTY than stdin. This also allows users to process SIGWINCH when running inside a regular terminal if they redirect stdin to /dev/null. Reported-by: Dan Moore <dan@vaporwa.re> References: <etPan.555b4293.5b47a5b7.e617@danbookpro> <20150519232858.GA23515@dcvr.yhbt.net>
2015-05-18FAQ: add note about ECONNRESET errors from bodiesEric Wong1-0/+9
Thanks to Michael Fischer and Gabe Martin-Dempesy for bringing this to light on the mailing list. Ref: <CABHxtY7Sn5yaiR5a3gDk1G4XySE+UtfuqUTcOSdmwneXLD5rcg@mail.gmail.com> Ref: <FC91211E-FD32-432C-92FC-0318714C2170@zendesk.com> Cc: Michael Fischer <mfischer@zendesk.com> Cc: Gabe Martin-Dempesy <gabe@zendesk.com>
2015-05-16avoid extra allocation for hijack proc creationEric Wong1-5/+11
proc creation is expensive, so merely use a 48-byte generic ivar hash slot for @socket instead.
2015-05-16http_request: support rack.hijack by defaultEric Wong1-23/+9
Rack 1.4 and earlier will soon die out, so avoid having extra code The only minor overhead is assigning two hash slots and the extra hash checks when running ancient versions of Rack, so it is unlikely anybody cares about that overhead with Rack 1.5 and later.
2015-05-16Merge tag 'v4.9.0'Eric Wong0-0/+0
unicorn 4.9.0 - TempfileReaper support in Rack 1.6 This release supports the Rack::TempfileReaper middleware found in rack 1.6 for cleaning up disk space used by temporary files. We also use Rack::TempfileReaper for cleaning up large temporary files buffered with TeeInput. Users on rack 1.5 and earlier will see no changes. There's also a bunch of documentation/build system improvements. This is likely to be the last Ruby 1.8-compatible release, unicorn 5.x will require 1.9.3 or later as well as dropping lots of cruft (the stupid "Status:" header in responses being the most notable). 21 changes backported from master: ISSUES: update with mailing list subscription FAQ: add entry for Rails autoflush_log dev: remove isolate dependency unicorn.gemspec: depend on test-unit 3.0 remove RubyForge and Freecode references remove mongrel.rubyforge.org references examples: add run_once to before_fork hook example t/t0002-parser-error.sh: relax test for rack 1.6.0 switch docs + website to olddoc README: clarify/reduce references to unicorn_rails gemspec: fixup olddoc migration GNUmakefile: fix clean gem build + reduce build cruft doc: update support status for Ruby versions fix uninstalled testing and reduce require paths test_socket_helper: do not depend on SO_REUSEPORT ISSUES: add section for bugs in other projects explain 11 byte magic number for self-pipe Links: mark Rainbows! as historical, reference yahns doc: document UNICORN_FD in manpage tee_input: support for Rack::TempfileReaper middleware support TempfileReaper in deployment and development envs * tag 'v4.9.0': (22 commits) unicorn 4.9.0 - TempfileReaper support in Rack 1.6 support TempfileReaper in deployment and development envs tee_input: support for Rack::TempfileReaper middleware doc: document UNICORN_FD in manpage Links: mark Rainbows! as historical, reference yahns explain 11 byte magic number for self-pipe ISSUES: add section for bugs in other projects test_socket_helper: do not depend on SO_REUSEPORT fix uninstalled testing and reduce require paths doc: update support status for Ruby versions GNUmakefile: fix clean gem build + reduce build cruft gemspec: fixup olddoc migration README: clarify/reduce references to unicorn_rails switch docs + website to olddoc t/t0002-parser-error.sh: relax test for rack 1.6.0 examples: add run_once to before_fork hook example remove mongrel.rubyforge.org references remove RubyForge and Freecode references unicorn.gemspec: depend on test-unit 3.0 dev: remove isolate dependency ...
2015-05-07favor kgio_wait_readable for single FD over selectEric Wong1-1/+1
kgio_wait_readable is superior for single FDs in that it may use the ppoll syscall on Linux via Ruby, making it immune to the slowdown high FDs with select() and the array allocations enforced by the Ruby wrapper interface. Note: IO#wait in the io/wait stdlib has the same effect, but as of 2.2 still needlessly checks the FIONREAD ioctl. So avoid needing to force a new require on users which also incur shared object loading costs. The longer term plan is to rely entirely on Ruby IO primitives entirely and drop kgio, but that won't happen until we can depend on Ruby 2.3 for exception-free accept_nonblock (which will be released December 2015).
2015-04-24unicorn 4.9.0 - TempfileReaper support in Rack 1.6 v4.9.0Eric Wong1-1/+1
This release supports the Rack::TempfileReaper middleware found in rack 1.6 for cleaning up disk space used by temporary files. We also use Rack::TempfileReaper for cleaning up large temporary files buffered with TeeInput. Users on rack 1.5 and earlier will see no changes. There's also a bunch of documentation/build system improvements. This is likely to be the last Ruby 1.8-compatible release, unicorn 5.x will require 1.9.3 or later as well as dropping lots of cruft (the stupid "Status:" header in responses being the most notable). 21 changes backported from master: ISSUES: update with mailing list subscription FAQ: add entry for Rails autoflush_log dev: remove isolate dependency unicorn.gemspec: depend on test-unit 3.0 remove RubyForge and Freecode references remove mongrel.rubyforge.org references examples: add run_once to before_fork hook example t/t0002-parser-error.sh: relax test for rack 1.6.0 switch docs + website to olddoc README: clarify/reduce references to unicorn_rails gemspec: fixup olddoc migration GNUmakefile: fix clean gem build + reduce build cruft doc: update support status for Ruby versions fix uninstalled testing and reduce require paths test_socket_helper: do not depend on SO_REUSEPORT ISSUES: add section for bugs in other projects explain 11 byte magic number for self-pipe Links: mark Rainbows! as historical, reference yahns doc: document UNICORN_FD in manpage tee_input: support for Rack::TempfileReaper middleware support TempfileReaper in deployment and development envs
2015-04-24support TempfileReaper in deployment and development envs 4.x-stableEric Wong1-0/+2
rack 1.6 added a TempfileReaper middleware to cleanup temporary files. Enable it by default for users running rack 1.6 or later to avoid leaving temporary files around.
2015-04-24tee_input: support for Rack::TempfileReaper middlewareEric Wong3-1/+21
Rack::TempfileReaper was added in rack 1.6 to cleanup temporary files. Make Unicorn::TmpIO ducktype-compatible so Rack::TempfileReaper may be used to free up space used by temporary buffer files. Ref: <CY1PR0301MB078011EB5A22B733EB222A45A4EE0@CY1PR0301MB0780.namprd03.prod.outlook.com> Reported-by: Mike Mulvaney <MMulvaney@bna.com>
2015-04-24support TempfileReaper in deployment and development envsEric Wong1-0/+2
rack 1.6 added a TempfileReaper middleware to cleanup temporary files. Enable it by default for users running rack 1.6 or later to avoid leaving temporary files around.
2015-04-24tee_input: support for Rack::TempfileReaper middlewareEric Wong3-1/+21
Rack::TempfileReaper was added in rack 1.6 to cleanup temporary files. Make Unicorn::TmpIO ducktype-compatible so Rack::TempfileReaper may be used to free up space used by temporary buffer files. Ref: <CY1PR0301MB078011EB5A22B733EB222A45A4EE0@CY1PR0301MB0780.namprd03.prod.outlook.com> Reported-by: Mike Mulvaney <MMulvaney@bna.com>
2015-04-22doc: document UNICORN_FD in manpageEric Wong1-0/+7
Due to the prevalence of socket activation in modern init systems, we shall document UNICORN_FD (previously an implementation detail) in the manpage.
2015-04-22Links: mark Rainbows! as historical, reference yahnsEric Wong1-1/+4
Pushing the boundaries of bad marketing :P
2015-04-22explain 11 byte magic number for self-pipeEric Wong1-0/+4
Oops, this should've been explained long ago but apparently not. In response to a comment on http://www.sitepoint.com/the-self-pipe-trick-explained/ > Does anybody know why both unicorn and foreman read 11 bytes from > self-pipe? Unfortunately I couldn't find a way to comment on the site on a JavaScript-free browser nor does it seem possible without registering. Again, anybody can send plain-text mail to: unicorn-public@bogomips.org No registration, no real name policy, no terms-of-service, just plain-text. Feel free to use Tor, mixmaster or any anonymity service, too.
2015-04-22ISSUES: add section for bugs in other projectsEric Wong1-1/+35
This is not anything new, just documenting what has been going on since the beginning. There's been a small number of generic networking (or mm) bugs in the kernel which affect unicorn, but are usually found and fixed with more popular, non-Ruby servers, first. Aside from generic performance problems, I don't think there's ever been a glibc bug which affected unicorn.
2015-04-22test_socket_helper: do not depend on SO_REUSEPORTEric Wong1-1/+1
Older Rubies (2.0) may not define SO_REUSEPORT even if the kernel and libc support it
2015-04-22fix uninstalled testing and reduce require pathsEric Wong12-14/+18
This fixes a bug introduced in commit fe83ead4eae6f011fa15f506cd80cb4256813a92 (GNUmakefile: fix clean gem build + reduce build cruft) which broke clean Ruby installations without an existing unicorn gem installed :x [fixed test/unit/test_http_parser_xftrust.rb for backport]
2015-04-22doc: update support status for Ruby versionsEric Wong9-96/+21
unicorn 5 will not support Ruby 1.8 anymore. Drop mentions of Rubinius, too, it's too difficult to support due to the proprietary and registration-required nature of its bug tracker. The smaller memory footprint and CoW-friendly memory allocator in mainline Ruby is a better fit for unicorn, anyways. Since Ruby 1.9+ bundles RubyGems and gem startup is faster nowadays, we'll just depend on that instead of not loading RubyGems. Drop the local.mk.sample file, too, since it's way out-of-date and probably isn't useful (I have not used it in a while). [reinstate 1.9 version check for listener_fds in backport]
2015-04-22GNUmakefile: fix clean gem build + reduce build cruftEric Wong1-7/+6
Ensure we have a NEWS file for building the gem beforehand. We don't need to polute lib/ with object files, either.
2015-04-22gemspec: fixup olddoc migrationEric Wong1-2/+0
rdoc_options is no longer necesary with olddoc as olddoc can infer document titles and only generates cgit-compatible URLs to source code.
2015-04-22README: clarify/reduce references to unicorn_railsEric Wong1-4/+3
unicorn_rails is an ancient compatibility wrapper for ancient versions of Rails which did not use Rack. Those applications have likely moved on, so stop promoting unicorn_rails.
2015-04-22switch docs + website to olddocEric Wong6-38/+28
wrongdoc was difficult to maintain because of the tidy-ffi dependency and the HTML5 changes in Darkfish could not be handled well by Tidy. olddoc is superior as it generates leaner HTML which loads faster, requires less scrolling and less processing power to render. Aesthetic comparisons are subjective of course but completely unimportant compared to speed and accessibility. The presence of images and CSS on the old (Darkfish-based) site probably set unreasonable expectations as to my ability and willingness to view such things. No more, the new website is entirely simple HTML which renders well with even the wimpiest browser.
2015-04-22t/t0002-parser-error.sh: relax test for rack 1.6.0Eric Wong1-3/+3
This overly zealous test was broken by: rack commit be28c6a2ac152fe4adfbef71f3db9f4200df89e8 ("update HTTP status codes to IETF RFC 7231")
2015-04-22examples: add run_once to before_fork hook exampleEric Wong1-0/+11
There may be code in a before_fork hook which should run only once, document an example using a guard variable since it may not be immediately obvious to all users. Inspired-by: BrĂ¡ulio Bhavamitra <braulio@eita.org.br> http://bogomips.org/unicorn-public/m/20141004015707.GA1951@dcvr.yhbt.net.html
2015-04-22remove mongrel.rubyforge.org referencesEric Wong4-8/+8
mongrel.rubyforge.org has been dead longer than rubyforge.org!
2015-04-22remove RubyForge and Freecode referencesEric Wong3-52/+0
Both sites are gone.
2015-04-22unicorn.gemspec: depend on test-unit 3.0Eric Wong1-0/+1
test-unit 3 and minitest 5 will have equal support status as a bundled gems when Ruby 2.2.0 is released in December 2014. These bundled gems will appear in the user-oriented tarball installations, but do not get installed by "make install" when installing Ruby from SVN or git. test-unit appears to be actively maintained and good at keeping backwards compatibility even on a major version change, so this means no code changes on our end. I am not convinced switching to minitest is worth the effort. Cc: Ken Dreyer <ktdreyer@ktdreyer.com>