huge documentation revamp

14 files changed, 270 insertions, 21 deletions

diff --git a/.document b/.document
index 88e5ce8..5c5efea 100644
--- a/.document
+++ b/.document
@@ -1,6 +1,11 @@
 README
 LICENSE
+DEPLOY
 lib
 rainbows.1
 ChangeLog
 NEWS
+TODO
+FAQ
+TUNING
+vs_Unicorn
diff --git a/DEPLOY b/DEPLOY
new file mode 100644
index 0000000..95526e2
--- /dev/null
+++ b/DEPLOY
@@ -0,0 +1,29 @@
+= Deploying \Rainbows!
+
+== nginx proxying to \Rainbows! or Unicorn
+
+For high-traffic applications, routing slow actions to \Rainbows! with
+nginx is recommended as nginx can serve static files faster and nginx
+can forward fast actions to Unicorn.
+
+          static files
+            |
+      nginx |--> slow actions --> Rainbows!
+            |
+            `--> fast actions --> Unicorn
+
+Be sure to set <tt>proxy_buffering off</tt> in nginx for "slow actions"
+if you have Comet applications (but not for Unicorn).
+
+== \Rainbows! only
+
+For the daring (or low-traffic sites), you should consider deploying
+\Rainbows! in a standalone configuration.  This will be more highly
+recommended as \Rainbows! stabilizes, especially if static file
+performance improves (or you don't need them).
+
+You will need to do this to support things like BOSH or do real-time
+processing of the request body as it is being uploaded.
+
+In this case, haproxy or any similar (non-request-body-buffering) load
+balancer should be used to balance requests between different machines.
diff --git a/FAQ b/FAQ
new file mode 100644
index 0000000..6c15675
--- /dev/null
+++ b/FAQ
@@ -0,0 +1,50 @@
+= Frequently Asked Questions about \Rainbows!
+
+=== Why is \Rainbows! a separate project from Unicorn?
+
+\Rainbows is for the odd, corner-case requests that Unicorn is poorly
+suited for.  More scalable concurrency models introduce additional
+complexity that Unicorn users and developers are uncomfortable with for
+the common cases.
+
+
+=== What complexity?  Threads/events/actors are easy to work with!
+
+Good for you.  Some of us depend on libraries incompatible with those
+models, or are just too lazy to deal with them for the majority of
+requests we service.
+
+
+=== Isn't "rainbows" a branch of Unicorn?
+
+That functionality is now in the Revactor model of \Rainbows!
+
+
+=== What happened to the "gossamer" branch of Unicorn?
+
+It became the ThreadPool model of \Rainbows!
+
+
+=== Which concurrency model should I use?
+
+It depends on your application, libraries, Ruby stack and use cases.
+That's why we support as many concurrency model as we can.  Each model
+has their own strengths and weaknesses in terms of maturity,
+ease-of-debugging, compatibility, performance, and memory usage.
+
+
+=== Should I put \Rainbows! behind nginx to serve slow clients?
+
+It is optional.  You can still use nginx to route certain requests to
+Unicorn and others to \Rainbows!  nginx will always outperform
+\Rainbows! in both pure reverse proxy applications and for serving
+static files,  but \Rainbows! is for hosting applications that are more
+easily-implemented in Ruby than C.
+
+
+=== Should I use \Rainbows! to serve static files?
+
+It depends on the size and amount of static files you're serving.  If
+you're serving a lot of static files (especially large ones), then by
+all means use nginx.  If not, then \Rainbows! is likely a "good enough"
+solution even if nginx will always outperform it in raw throughput.
diff --git a/GNUmakefile b/GNUmakefile
index a8274e7..ba24e03 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -75,6 +75,7 @@ atom = <link rel="alternate" title="Atom feed" href="$(1)" \
 doc: .document NEWS ChangeLog
         for i in $(man1_bins); do > $$i; done
         rdoc -Na -t "$(shell sed -ne '1s/^= //p' README)"
+        install -m644 COPYING doc/COPYING
         install -m644 $(shell grep '^[A-Z]' .document)  doc/
         $(MAKE) -C Documentation install-html install-man
         install -m644 $(man1_paths) doc/
diff --git a/LICENSE b/LICENSE
index c571b10..94b938b 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,4 +1,4 @@
-Rainbows! is copyrighted free software by Eric Wong
+\Rainbows! is copyrighted free software by Eric Wong
 (mailto:normalperson@yhbt.net) and contributors. You can redistribute it
 and/or modify it under either the terms of the
 {GPL2}[http://www.gnu.org/licenses/gpl-2.0.txt] (see link:COPYING) or
diff --git a/README b/README
index def4dce..47cb696 100644
--- a/README
+++ b/README
@@ -1,12 +1,26 @@
-= Rainbows! Unicorn for Comet and slow clients
+= Rainbows! Unicorn for slow apps and slow clients
 
-Rainbows! is a HTTP server for {Rack}[http://rack.rubyforge.org/]
+\Rainbows! is a HTTP server for {Rack}[http://rack.rubyforge.org/]
 applications.  It is based on {Unicorn}[http://unicorn.bogomips.org/],
 but designed to handle applications that expect long request/response
-times and/or slow clients.  It is designed for Comet applications and
-reverse proxy implementations.  For Rack applications not heavily
-bound by external dependencies, consider Unicorn instead as it simpler
-and easier to debug.
+times and/or slow clients.  For Rack applications not heavily bound by
+slow external dependencies, consider Unicorn instead as it simpler and
+easier to debug.
+
+== \Rainbows! is about Diversity
+
+We aim to support as many concurrency models as we can because they all
+suck; differently.
+
+For network concurrency, models we currently support are:
+
+* {:ThreadSpawn}[link:Rainbows/ThreadSpawn.html]
+* {:ThreadPool}[link:Rainbows/ThreadPool.html]
+* {:Revactor}[link:Rainbows/Revactor.html]
+
+We have {more on the way}[link:TODO.html] for handling network concurrency.
+Additionally, we also use multiple processes (managed by Unicorn) for
+CPU/memory/disk concurrency.
 
 == Features
 
@@ -15,23 +29,40 @@ and easier to debug.
 * Built on Unicorn, inheriting its process/socket management features
   such as transparent upgrades and Ruby configuration DSL.
 
-* Like Unicorn, it is able to stream large request bodies off the
-  socket to the application while the client is still uploading.
+* As with Unicorn, it is able to stream large request bodies off the
+  socket to the application while the client is still uploading.  Since
+  \Rainbows! can handle slow clients, this feature is more useful than
+  it is with Unicorn.
 
 * Combines heavyweight concurrency (worker processes) with lightweight
-  concurrency (Actors/Threads), allowing CPU/memory/disk to be scaled
-  independently of client connections.
+  concurrency (Actors or Threads), allowing CPU/memory/disk to be scaled
+  independently of client connections.  Alternative concurrency models
+  (listed in the TODO) will be supported as we find time for them.
+
+== Applications
+
+\Rainbows is for the odd things Unicorn sucks at:
+
+* 3rd-party APIs (to services outside your control/LAN)
+* OpenID consumers (to providers outside your control/LAN)
+* Reverse proxy implementations with editing/censoring
+  (to upstreams outside your control/LAN)
+* Comet
+* BOSH (with slow clients)
+* HTTP server push
+* Long polling
+* Reverse Ajax
 
 == License
 
-Rainbows! is copyright 2009 Eric Wong and contributors.  It is based on
+\Rainbows! is copyright 2009 Eric Wong and contributors.  It is based on
 Mongrel and Unicorn and carries the same license.
 
 Mongrel is copyright 2007 Zed A. Shaw and contributors. It is licensed
 under the Ruby license and the GPL2. See the included LICENSE file for
 details.
 
-Rainbows! is 100% Free Software.
+\Rainbows! is 100% Free Software.
 
 == Usage
 
@@ -41,17 +72,17 @@ In APP_ROOT (where config.ru is located), run:
 
   rainbows
 
-Rainbows! will bind to all interfaces on TCP port 8080 by default.
+\Rainbows! will bind to all interfaces on TCP port 8080 by default.
 
 === Configuration File(s)
 
-Rainbows! will look for the config.ru file used by rackup in APP_ROOT.
+\Rainbows! will look for the config.ru file used by rackup in APP_ROOT.
 
 For deployments, it can use a config file for Unicorn and
-Rainbows!-specific options specified by the +--config-file/-c+
-command-line switch.  Rainbows! accepts all options found in
+\Rainbows!-specific options specified by the +--config-file/-c+
+command-line switch.  \Rainbows! accepts all options found in
 {Unicorn::Configurator}[http://unicorn.bogomips.org/Unicorn/Configurator.html]
-as well as the "Rainbows!" block, so you can have the following in your
+as well as the "\Rainbows!" block, so you can have the following in your
 config file:
 
     Rainbows! do
@@ -71,7 +102,7 @@ the patch.
 We will adhere to mostly the same conventions for patch submissions as
 git itself.  See the Documentation/SubmittingPatches document
 distributed with git on on patch submission guidelines to follow.  Just
-don't email the git mailing list or maintainer with Rainbows! patches.
+don't email the git mailing list or maintainer with \Rainbows! patches.
 
 == Disclaimer
 
diff --git a/Rakefile b/Rakefile
index d2025ce..b9a89c3 100644
--- a/Rakefile
+++ b/Rakefile
@@ -33,7 +33,7 @@ task :news_atom do
     feed :xmlns => "http://www.w3.org/2005/Atom" do
       id! "http://rainbows.rubyforge.org/NEWS.atom.xml"
       title "Rainbows! news"
-      subtitle "Unicorn for Comet and slow clients"
+      subtitle "Unicorn for slow apps and slow clients"
       link! :rel => 'alternate', :type => 'text/html',
             :href => 'http://rainbows.rubyforge.org/NEWS.html'
       updated(new_tags.empty? ? "1970-01-01T00:00:00Z" : new_tags.first[:time])
diff --git a/TODO b/TODO
new file mode 100644
index 0000000..2c90a09
--- /dev/null
+++ b/TODO
@@ -0,0 +1,20 @@
+= TODO items for Rainbows!
+
+We're lazy and pick the easy items to do first, then the ones people
+care about.
+
+* Rev (without Revactor) - since we already use Revactor we might as
+  well support this one so 1.8 users won't be left out.  Doing TeeInput
+  is probably going to get ugly, though...
+
+* EventMachine - much like Rev, but we haven't looked at this one much
+  (our benevolent dictator doesn't like C++).  If we can figure out how
+  to do Rev without Revactor, then this should be pretty easy.
+
+* Fiber support - Revactor already uses these with Ruby 1.9, also not
+  sure how TeeInput can be done with this.
+
+* Omnibus - haven't looked into it, probably like Revactor with 1.8?
+
+* Rubinius Actors - should be like Revactor and easily doable once
+  Rubinius gets more mature.
diff --git a/TUNING b/TUNING
new file mode 100644
index 0000000..9f5f58d
--- /dev/null
+++ b/TUNING
@@ -0,0 +1,31 @@
+= Tuning \Rainbows!
+
+Most of the {tuning notes}[http://unicorn.bogomips.org/TUNING.html]
+apply to \Rainbows! as well.  \Rainbows! is not particularly optimized
+at the moment and is designed for applications that spend large amounts
+of the time waiting on network activity.  Thus memory usage and memory
+bandwidth for keeping connections open are often limiting factors as
+well.
+
+== \Rainbows! configuration
+
+* Don't set +worker_connections+ too high.  It is often better to start
+  denying requests and only serve the clients you can than to be
+  completely bogged down and be unusable for everybody.
+
+* Increase +worker_processes+ if you have resources (RAM/DB connections)
+  available.  Additional worker processes can better utilize SMP, are more
+  robust against crashes and are more likely to be fairly scheduled by
+  the kernel.
+
+== nginx configuration
+
+If you intend to use nginx as a reverse-proxy in front of \Rainbows!  to
+handle Comet applications, make sure you disable proxy response
+buffering in nginx:
+
+  proxy_buffering off;
+
+This can be disabled on a per-backend basis in nginx, so under no
+circumstances should you disable response buffering to Unicorn
+backends, only to \Rainbows! backends.
diff --git a/lib/rainbows/revactor.rb b/lib/rainbows/revactor.rb
index a20e09a..52cebf8 100644
--- a/lib/rainbows/revactor.rb
+++ b/lib/rainbows/revactor.rb
@@ -7,6 +7,20 @@ defined?(Rev::Buffer) or Rev::Buffer = IO::Buffer
 
 module Rainbows
 
+  # Enables use of the Actor model through
+  # {Revactor}[http://revactor.org] under Ruby 1.9.  It spawns one
+  # long-lived Actor for every listen socket in the process and spawns a
+  # new Actor for every client connection accept()-ed.
+  # +worker_connections+ will limit the number of client Actors we have
+  # running at any one time.
+  #
+  # Applications using this model are required to be reentrant, but
+  # generally do not have to worry about race conditions.  Multiple
+  # instances of the same app may run in the same address space
+  # sequentially (but at interleaved points).  Any network dependencies
+  # in the application using this model should be implemented using the
+  # \Revactor library as well.
+
   module Revactor
     require 'rainbows/revactor/tee_input'
 
diff --git a/lib/rainbows/thread_pool.rb b/lib/rainbows/thread_pool.rb
index 62a04a3..a6269cd 100644
--- a/lib/rainbows/thread_pool.rb
+++ b/lib/rainbows/thread_pool.rb
@@ -2,6 +2,18 @@
 
 module Rainbows
 
+  # Implements a worker thread pool model.  This is suited for platforms
+  # where the cost of dynamically spawning a new thread for every new
+  # client connection is too high.
+  #
+  # Applications using this model are required to be thread-safe.
+  # Threads are never spawned dynamically under this model.  If you're
+  # connecting to external services and need to perform DNS lookups,
+  # consider using the "resolv-replace" library which replaces parts of
+  # the core Socket package with concurrent DNS lookup capabilities.
+  #
+  # This model is less suited for many slow clients than the others and
+  # thus a lower +worker_connections+ setting is recommended.
   module ThreadPool
 
     include Base
diff --git a/lib/rainbows/thread_spawn.rb b/lib/rainbows/thread_spawn.rb
index 085da39..36968d8 100644
--- a/lib/rainbows/thread_spawn.rb
+++ b/lib/rainbows/thread_spawn.rb
@@ -1,6 +1,14 @@
 # -*- encoding: binary -*-
 module Rainbows
 
+  # Spawns a new thread for every client connection we accept().  This
+  # model is recommended for platforms where spawning threads is
+  # inexpensive.
+  #
+  # If you're connecting to external services and need to perform DNS
+  # lookups, consider using the "resolv-replace" library which replaces
+  # parts of the core Socket package with concurrent DNS lookup
+  # capabilities
   module ThreadSpawn
 
     include Base
diff --git a/rainbows.gemspec b/rainbows.gemspec
index 61f6fe2..df73ccf 100644
--- a/rainbows.gemspec
+++ b/rainbows.gemspec
@@ -33,7 +33,7 @@ Gem::Specification.new do |s|
 
   s.files = manifest
   s.homepage = %q{http://rainbows.rubyforge.org/}
-  s.summary = %q{Unicorn for Comet and slow clients}
+  s.summary = %q{Unicorn for slow apps and slow clients}
   s.rdoc_options = [ "-Na", "-t", "Rainbows! #{s.summary}" ]
   s.require_paths = %w(lib)
   s.rubyforge_project = %q{rainbows}
diff --git a/vs_Unicorn b/vs_Unicorn
new file mode 100644
index 0000000..827bdb8
--- /dev/null
+++ b/vs_Unicorn
@@ -0,0 +1,48 @@
+= \Rainbows! is like Unicorn, but Different...
+
+While \Rainbows! depends on Unicorn for its process/socket management,
+HTTP parser and configuration language; \Rainbows! is more ambitious.
+
+== Differences from Unicorn
+
+* log rotation is handled immediately in \Rainbows! whereas Unicorn has
+  the luxury of delaying it until the current request is finished
+  processing to prevent log entries for one request to be split across
+  files.
+
+* load balancing between workers is imperfect, certain worker processes
+  may be servicing more requests than others so it is important to not
+  set +worker_connections+ too high.  Unicorn worker processes can never
+  be servicing more than one request at once.
+
+* speculative, non-blocking accept() is not used, this is to help
+  load balance between multiple worker processes.
+
+* HTTP pipelining and keepalive may be used for GET and HEAD requests.
+
+* Less heavily-tested and inherently more complex.
+
+
+== Similarities with Unicorn
+
+While some similarities are obvious (we depend on and subclass of
+Unicorn code), some things are not:
+
+* Does not attempt to accept() connections when pre-configured limits
+  are hit (+worker_connections+).  This will first help balance load
+  to different worker processes, and if your listen() +:backlog+ is
+  overflowing: to other machines in your cluster.
+
+* Accepts the same {signals}[http://unicorn.bogomips.org/SIGNALS.html]
+  for process management, so you can share scripts to manage them (and
+  nginx, too).
+
+* supports per-process listeners, allowing an external load balancer
+  like haproxy or nginx to be used to balance between multiple
+  worker processes.
+
+* Exposes a streaming "rack.input" to the Rack application that reads
+  data off the socket as the application reads it (while retaining
+  rewindable semantics as required by Rack).  This allows Rack-compliant
+  apps/middleware to implement things such as real-time upload progress
+  monitoring.