about summary refs log tree commit homepage
DateCommit message (Collapse)
2013-11-15try_gzip_static: set no-transform on non-text types
This allows yahns to continue using sendfile when clients request gzipped/bzipped tarballs on my server.
2013-11-15extras/try_gzip_static: account for Rack::Deflater use
Clients may be requesting gzipped files through Rack::Deflater, which will attempt to further compress files.
2013-11-15encoding: binary for everything
This is mainly needed for the regexp in extras/try_gzip_static.rb (POSIX filesystem paths have no encoding, it's just a bag of bytes). Since we host apps of all types and for all (human) languages/encodings, all of our internals must be encoding-agnostic.
2013-11-12exec_cgi: drop chunk/content-length
Leave that up to Rack::Chunked/Rack::ContentLength. Chunking ourselves interacts badly with Rack::Deflater, since Deflater will blindly deflate already-chunked portions.
2013-11-12extras/*: log full backtraces for errors
In case we have bugs, this can help us find bugs in our code.
2013-11-12extras/try_gzip_static: return 404 on ENOTDIR
Some attackers may try /path/to/file/foo where /path/to/file is actually a valid path to a regular file. Of course, requests like this work on dynamic websites, but not static file mappings because Unix directories and files cannot be the same thing.
2013-11-10yahns 0.0.3 - more bugfixes, still working v0.0.3
There are several important bugfixes around graceful shutdown. The shutdown_timeout directive should work more consistently and work even if an app is streaming/trickling slowly to a client. This also plugs an odd FD leak in response buffering. There's also some internal comments/documentation for folks reviewing the strange internals of yahns. Otherwise, it seems to be capable of serving its own website, http://yahns.yhbt.net/README quite well. Rack application authors merely need to write code as if they have a gun to their head if they expect to keep code running on yahns. Again, yahns is extremely intolerant of bugs in the applications it hosts. Otherwise I'm comfortable in the stability of yahns itself. Eric Wong (17): http_client: do not dump backtrace on ETIMEDOUT ensure we close response body if buffering caught up http_response: reorder wbuf_maybe on successful early flush wbuf: document reasoning for the design of these clases build: improve NEWS generation, add atom feed exec_cgi: release pipe/pid sooner SIGWINCH works after SIGUSR2 upgrades shutdown_timeout works around long-running response/apps doc: recommend worker_processes if the app uses SIGCHLD fdmap: simplify IO expiry interface fdmap: document + fix for level-trigger queue_epoll: document epoll concurrency caveats doc: caution users against disabling buffering queue: eliminate :wait_readwrite test: exec_cgi test uses worker_processes test_server: remove unneccessary IO#wait call test: workaround sockets not being binary on rbx
2013-11-10test: workaround sockets not being binary on rbx
ref: https://github.com/rubinius/rubinius/issues/2772
2013-11-10test_server: remove unneccessary IO#wait call
This was failing under Rubinius and not necessary after all ref: https://github.com/rubysl/rubysl-io-wait/issues/2
2013-11-10test: exec_cgi test uses worker_processes
Following our own advice in commit a79a6d8775171ad5cceda9bb3a77946ba60e26ce (doc: recommend worker_processes if the app uses SIGCHLD)
2013-11-07queue: eliminate :wait_readwrite
Simultaneously waiting for both readability and writability is likely unnecessary. Sockets become writable much more commonly, and if we need to read, we'll issue a read the next time it enters yahns_step (after it becomes writable). Also, this makes kqueue easier-to-implement as I don't believe it implements a way to combine EVFILT_READ with EVFILT_WRITE (especially not safely while relying on EV_ONESHOT behavior).
2013-11-07doc: caution users against disabling buffering
Users may be tempted to disable buffering because it improves synthetic benchmark performance. Synthetic benchmarks are just that and not indicative of what evil (or really crippled) clients can do to a server.
2013-11-07queue_epoll: document epoll concurrency caveats
Multithreaded epoll is a tricky interface to use. Fortunately it should be hidden from users and yahns is no different from any other multithreaded server from a user application programming perspective (which requires a lot of careful programming in the first place).
2013-11-07fdmap: document + fix for level-trigger
Our QueueQuitter may race with epoll_wait and GC, potentially triggering memory corruption if a level-trigger object was placed into the ready list and no longer referenced. We'll work around this problem by making the QueueQuitter object a thread-local (on thread failure) to prevent concurency issues where the epoll pointer no longer points to a Ruby object.
2013-11-07fdmap: simplify IO expiry interface
The first argument of the old __expire_for method was completely unused in all but one invocation of it, so it is not worth it. This also simplifies the code a bit too.
2013-11-07doc: recommend worker_processes if the app uses SIGCHLD
If worker_processes are not enabled, our SIGCHLD handler may conflict with one installed by the application. Fortunately it is uncommon in Ruby web apps to rely on SIGCHLD, but it happens and that is bad.
2013-11-07shutdown_timeout works around long-running response/apps
apps/responses may trickle infinitely, so our QueueQuitter may not properly kill epoll worker threads, unfortunately. So we'll start the timer and stop waiting (risking segfaults/ugliness) by closing epoll descriptors
2013-11-07SIGWINCH works after SIGUSR2 upgrades
We must know we're daemonized after a SIGUSR2 upgrade, even if we don't use the pipe to signal to the controlling terminal we've started up.
2013-11-07exec_cgi: release pipe/pid sooner
We don't want to leave extra resources lying around when slow clients read from us and yahns is forced to buffer. yahns delays the close of the request body until the request is fully written so response loggers can have an accurate time of how long it takes.
2013-11-07build: improve NEWS generation, add atom feed
This should hopefully make it easier to share info about new releases.
2013-11-06wbuf: document reasoning for the design of these clases
The choice to jump directly to temporary files on EAGAIN may seem odd to some, document our reasoning for choosing this approach.
2013-11-06http_response: reorder wbuf_maybe on successful early flush
We can use the wbuf_close return value instead to ensure we close tmpio properly and follow the same code path as a normal (:wait_writable-triggering) buffered response would. Add a few tests to ensure we properly close the response body for exec_cgi, where I noticed zombies and started me down this rabbit hole looking for places where the response body was not closed properly.
2013-11-06ensure we close response body if buffering caught up
We may buffer a response partially, but no longer need the buffer if we entered bypass mode. When doing this, we must remember to close the body after using it.
2013-11-06http_client: do not dump backtrace on ETIMEDOUT
Clients may timeout due to SO_KEEPALIVE handling after a few hours. Since we have infinite keepalive, this may happen as we maintain no timers in userspace.
2013-11-06yahns 0.0.2 - barely working! v0.0.2
Lots of bugfixes and tweaks, but everything appears to mostly work for Rack and HTTP. We are self-hosting our site: http://yahns.yhbt.net/README (And if you can't access it, I screwed something up!) There is a yahns-rackup(1) wrapper for quick-starting Rack applications without having to read any documentation, as it works exactly like rackup(1) (just leave "-s/--server" out) New manpages: yahns(1), yahns_config(5), yahns-rackup(1) Eric Wong (110): test/helper: correctly handle exit code in test runner log: workaround atomicity issues for stdio vs non-stdio descriptors tests: improve output to show RUBY_DESCRIPTION and full path tests: support disabling parallelization env test_output_buffering: workaround a rbx bug socket_helper: do not log sizes on rbx for now config: do not set cloexec on stdout/stderr test_wbuf: avoid floating point arg to read_nonblock rework shutdown for systems w/o rb_thread_fd_close SIGUSR2 uses fork + exec again instead of Process.spawn wip for EADDRINUSE failure test_bin: shutdown socket when not inheriting README: update with support status for MRI/RBX server: skip killing acceptors on rbx for now server (cleanup): avoid interning word for log message server: abort loudly if we have old threads running reinstate retry delay for binding new listeners test/helper: Dir.mktmpdir works without blocks rework acceptor thread shutdown (again) server (minor cleanup): use Symbol#to_proc test_server: remove skipped multi-process balance test tests: do not lose exceptions on quit timeouts tests: enforce close_on_exec on all client sockets server: switch abort to raise on BUG config: implement atfork handlers test/server_helper: fix undefined variable favor client timeout if lower than desperate timeout tests: enable checks for desperate client expiry implement + test Expect: 100-continue handling quitter: save one syscall and implement for non-eventfd enforce FD_CLOEXEC on all pipes, including tests test_expect_100: cleanup unused var http_client: reduce the size of the yahns_step method output_buffering handles odd EAGAIN cases fix output buffer corner-case for static files tests: increase mkserver use to reduce LoC fix SIGCHLD w/o workers + PID file renames test_client_expire: disable output buffering in test StreamFile and TmpIO attempt expiry on EMFILE/ENFILE server: avoid metaclass for acceptors support SO_REUSEPORT on new listeners (:reuseport) doc: ignore RDoc for all internal classes rack: leave RACK_ENV unset by us associate private/anonymous queues correctly queue_epoll: remove check for closed descriptor config: disallow defining new, named queues inside app doc: preliminary manpages for yahns(1), yahns_config(5) config: comment for atfork_* hook definitions implement user switching configurator: validate :reuseport for boolean-ess config: working_directory is only called at top-level server: fix out-of-date comment regarding bind/pid order config: reject negative float for client_expire_ratio lower client_body_buffer_size to 8K (from 114K) implement client_timeout for streaming inputs gem: install manpages in the RubyGems package stream_input: use thread-local rbuf to avoid some garbage test output_buffering with hijacked responses test_input: close client when we're done with it allow multiple blocking threads per listen socket acceptor: account for inheriting dead descriptors server: fix incorrect receiver of method call socket_helper: account for undefined options test for binding Unix stream sockets Rack hijack issues EPOLL_CTL_DEL config: raise ArgumentError for consistency tests for SIGTTIN and SIGTTOU use Hash#each instead of Hash#each_pair add test for working_directory config parameter test_unix_socket: remember to close IO when done test for overriding rack.errors destination test_server: improve working_directory test robustness fdmap: prevent possible/theoretical race implement shutdown_timeout and expiry on graceful shutdown doc: fix client_timeout documentation in yahns_config implement before_exec hook comment to explain YAHNS_FD env remove "worker_" prefix from atfork hooks remove arity enforcement for atfork_* hooks allow atfork_* hooks inside app blocks for ease-of-management doc: disambiguate threads: option for listen directive yahns-rackup launcher enable client expiry for non-TCP sockets config: allow Float for timeouts tests: more intelligent waiting for output buffering doc: add HACKING/INSTALL docs + minitest dependency wbuf: bypass buffering if buffers are caught up wbuf: reset FS (sendfile) buffer if caught up wbuf: only enable bypass if we've truncated input and output buffers support tmpdir: arguments tests: further speed up output buffering test test trysendfile usage with UNIX sockets client_expire_generic: drop kgio_write wrapper remove unnecessary map(&:to_s) before Array#join account for truncated/expanded static files http: do not drop Content-Range from response headers worker-less server should not waitpid indiscriminately stream_file: only close FDs we opened ourselves tests: smaller buffer for big header test add extras for exec_cgi and try_gzip_static http_response: fix app-controlled close + test examples: flesh out the example configs a bit README: add disclaimer server: improve shutdown messages extras: add README extras: add autoindex module extras: add proxy_pass Rack app rackup_handler: fix ordering of working_directory worker: avoid double SIGQUIT on unexpected parent death server: log error instead of raising for leftover socks
2013-11-06server: log error instead of raising for leftover socks
This may happen if we get by a forceful exit, so we'll risk triggering some bugs elsewhere, but forceful exits are never pretty...
2013-11-06worker: avoid double SIGQUIT on unexpected parent death
Double SIGQUIT defeats the use of graceful shutdown. We must not watch the worker pipe again for readability after we've initiated graceful shutdown.
2013-11-06rackup_handler: fix ordering of working_directory
We cannot declare apps before working_directory is bound.
2013-11-05extras: add proxy_pass Rack app
This is an ad-hoc reverse proxy solution. This is fully-Rack compatible at the moment, so it's synchronous. This is also only very lightly tested but I don't use it for any important serving, yet.
2013-11-05extras: add autoindex module
Unlike Rack::Directory, this this also avoids tables and CSS for preformatted HTML. This is meant to resemble nginx autoindex and index functionality (combined).
2013-11-05extras: add README
Describe the intent of the extras/ subdirectory.
2013-11-05server: improve shutdown messages
It may be disconcerting to not know how long it takes a server to gracefully exit, so provide that feedback. While we're at it, avoid dropping acceptors if we want a forceful exit.
2013-11-05README: add disclaimer
This should hopefully educate users of the potential negative consequences of using yahns without understanding it. Or scare off some of the willfully ignorant :P
2013-11-05examples: flesh out the example configs a bit
Hopefully this makes things a little easier to pick up.
2013-11-05http_response: fix app-controlled close + test
We allow applications to drop persistent connections, this does not seem forbidden by Rack and gives the app author some control over the lifetime of a connection.
2013-11-02add extras for exec_cgi and try_gzip_static
These applications are what I'll be using to run on yahns on my personal server. Including them here will be helpful for me to find bugs. I've already found some, the following commits were directly the result of playing with these extras: * stream_file: only close FDs we opened ourselves * worker-less server should not waitpid indiscriminately * http: do not drop Content-Range from response headers
2013-11-02tests: smaller buffer for big header test
This was triggering OOM on my 32-bit machine.
2013-11-02stream_file: only close FDs we opened ourselves
We must not directly close descriptors allocated by applications themselves. The only way we may close them is to issue body.close on the response body. A simple comparison we used before may not work because Rack response bodies may be wrapped by middleware, so we use a private subclass of File.
2013-11-02worker-less server should not waitpid indiscriminately
We must not reap processes we did not spawn if we're not using a master/worker configuration. This may break applications which depend on waitpid themselves. However we still trap SIGCHLD currently. In the future, we could switch our notification model mode to be compatible with apps which depend on SIGCHLD or might indiscriminately on children (which might prevent us from knowing about SIGCHLD).
2013-11-02http: do not drop Content-Range from response headers
We parse and use Content-Range, but do not drop it when sending a response since that would confuse clients.
2013-11-02account for truncated/expanded static files
When running a static file server, we must account for filesystem activity outside of our control where files may grow/shrink as they're being served. For truncated files, we must abort any persistent connections downloading a truncated file to avoid confusing clients because the Content-Length header was already set to the big value. We also must ensure (we already did so before this commit, this just adds a test for it) we do not send additional data when a file grows on us after we've started sending the response.
2013-11-01remove unnecessary map(&:to_s) before Array#join
Array#join implies string conversion and unicorn has been doing that for ages.
2013-11-01client_expire_generic: drop kgio_write wrapper
We never use kgio_write for clients, only kgio_trywrite.
2013-11-01test trysendfile usage with UNIX sockets
We didn't cover this before.
2013-11-01tests: further speed up output buffering test
Increase scans of log files and use shorter sleep intervals to get faster notification of successful timeout.
2013-11-01input and output buffers support tmpdir: arguments
This allows users to specify alternative temporary directories in case buffers get too large for one filesystem to handle or to give priority to some clients on certain ports.
2013-11-01wbuf: only enable bypass if we've truncated
The first time we call wbuf_write, we do so because we already hit :wait_writable in the caller. So we can avoid the extra still-highly-likely-to-return-:wait_*) kgio_trywrite by writing to the VFS, first.
2013-11-01wbuf: reset FS (sendfile) buffer if caught up
This can prevent dirty data from being committed, saving the system from unnecessary disk activity.
2013-11-01wbuf: bypass buffering if buffers are caught up
Sometimes buffering can catch up and we no longer need to use the on-disk buffer, so keep trying to flush the data out to the user to avoid VFS activity.
2013-11-01doc: add HACKING/INSTALL docs + minitest dependency
Hopefully it'll be easier for folks to use and contribute. Publish it on the site while we're at it.