doc: misc updates

11 files changed, 153 insertions, 116 deletions

diff --git a/.document b/.document
index eef8ef0..127f3b7 100644
--- a/.document
+++ b/.document
@@ -2,9 +2,10 @@ History
 ChangeLog
 LICENSE
 README
+HACKING
+TODO
 lib/mogilefs.rb
 lib/mogilefs/admin.rb
 lib/mogilefs/backend.rb
 lib/mogilefs/client.rb
 lib/mogilefs/mogilefs.rb
-lib/mogilefs/pool.rb
diff --git a/GNUmakefile b/GNUmakefile
index 574e77a..6d5d142 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -91,3 +91,5 @@ publish_doc:
         $(RSYNC) -av doc/ $(RSYNC_DEST)/
         git ls-files | xargs touch
 endif
+
+.PHONY: doc
diff --git a/HACKING b/HACKING
new file mode 100644
index 0000000..575c202
--- /dev/null
+++ b/HACKING
@@ -0,0 +1,33 @@
+= Hacking mogilefs-client for Ruby
+
+* The latest code is available via git:
+  - git://bogomips.org/mogilefs-client.git
+  - git://repo.or.cz/ruby-mogilefs-client
+
+* Follow conventions set in existing code, don't add hard runtime
+  dependencies outside of the standard Ruby library.
+
+* Do not hesitate to send plain-text(-only) email to Eric Wong at
+  mailto:normalperson@yhbt.net about anything not covered in
+  the documentation or mailing list archives.  General MogileFS
+  topics can go to the public mailing list at mailto:mogile@googlegroups.com
+  You may still email Wong directly if you do not trust the corporation
+  that hosts the public mailing list.
+
+* Use "git format-patch" and "git send-email" for sending patches.
+
+* Use "git request-pull" as a guideline for formatting pull-requests.
+
+* Test with the latest upstream MogileFS and Ruby versions.
+
+* Integration tests exist for setting up a fresh MogileFS instance
+  with a single host and device using a SQLite backend if you
+  have "mogdbsetup", "mogilefsd", and "mogstored" in your PATH.
+
+* Setting MOG_TEST_TRACKERS= to a comma-delimited list of trackers
+  can enable additional integration tests against a live tracker
+  (with multiple devices/hosts).
+
+* Tests may be run in parallel using GNU make:
+
+  make -j6 test
diff --git a/README b/README
index d72802d..1c49e5f 100644
--- a/README
+++ b/README
@@ -27,44 +27,16 @@ Then install the RubyGem:
 This library supports Ruby 1.8.7 and later, but Ruby 1.9.3 is
 recommended.  No other libraries are required on the client.
 
-== Use
+== Usage
 
-  # Create a new instance that will communicate with these trackers:
-  hosts = %w[192.168.1.69:6001 192.168.1.70:6001]
-  mg = MogileFS::MogileFS.new(:domain => 'test', :hosts => hosts)
-
-  # Stores "A bunch of text to store" into 'some_key' with a class of 'text'.
-  mg.store_content 'some_key', 'text', "A bunch of text to store"
-
-  # Retrieve data from 'some_key'
-  data = mg.get_file_data 'some_key'
-
-  # Store the contents of 'image.jpeg' into the key 'my_image' with a class of
-  # 'image'.
-  mg.store_file 'my_image', 'image', 'image.jpeg'
-
-  # Store the contents of 'image.jpeg' into the key 'my_image' with a class of
-  # 'image' using an open IO.
-  File.open 'image.jpeg', 'rb' do |fp|
-    mg.store_file 'my_image', 'image', fp
-  end
-
-  # Retrieve the contents of 'my_image' into '/path/to/huge_file'
-  # without slurping the entire contents into memory:
-  File.open '/path/to/huge_file', 'w' do |fp|
-    mg.get_file_data 'my_image', wr
-  end
-
-  # Remove the key 'my_image' and 'some_key'.
-  mg.delete 'my_image'
-  mg.delete 'some_key'
+See MogileFS::MogileFS
 
 == Contact
 
 Feedback (bug reports, user/development dicussion, patches, pull
 requests) are greatly appreciated and handled via email.  We currently
 piggy-back onto the public MogileFS
-{mailing list}[mailto:mogile@googlegroups.com].
+{mailing list}[mailto:mogile@googlegroups.com] for feedback.
 
 If you do not want to deal with the corporate host of the MogileFS
 mailing list, or if you wish to keep your issue secret, feel free to
diff --git a/TODO b/TODO
new file mode 100644
index 0000000..2d93b6d
--- /dev/null
+++ b/TODO
@@ -0,0 +1 @@
+* MogileFS::Admin needs to be fleshed out
diff --git a/lib/mogilefs.rb b/lib/mogilefs.rb
index e34d906..d20f0f9 100644
--- a/lib/mogilefs.rb
+++ b/lib/mogilefs.rb
@@ -1,29 +1,49 @@
 # -*- encoding: binary -*-
-##
-# MogileFS is a Ruby client for Danga Interactive's open source distributed
-# filesystem.
 #
-# To read more about Danga's MogileFS: http://danga.com/mogilefs/
-
+# To read more about \MogileFS, go to http://mogilefs.org/
+#
+# Client usage information is available in MogileFS::MogileFS.
 module MogileFS
   # :stopdoc:
 
   VERSION = '2.2.0'
 
-  ##
-  # Raised when a socket remains unreadable for too long.
-
+  # Standard error class for most MogileFS-specific errors
   class Error < StandardError; end
+
+  # Raised when a socket remains unreadable for too long.
   class UnreadableSocketError < Error; end
+
+  # Raised when a response is truncated while reading
+  # due to network/server # errors)
   class SizeMismatchError < Error; end
+
+  # Raised when checksum verification fails (only while reading deprecated
+  # "bigfiles" from the deprecated mogtool(1).
   class ChecksumMismatchError < RuntimeError; end
+
+  # Raised when a backend is in read-only mode
   class ReadOnlyError < Error
     def message; 'readonly mogilefs'; end
   end
+
+  # Raised when an upload is impossible
   class EmptyPathError < Error; end
+
+  # Raised when we are given an unsupported protocol to upload to.
+  # Currently, the \MogileFS (2.54) server only supports HTTP and
+  # this library is only capable of HTTP.
   class UnsupportedPathError < Error; end
+
+  # Raised when a request (HTTP or tracker) was truncated due to a network or
+  # server error.  It may be possible to retry idempotent requests from this.
   class RequestTruncatedError < Error; end
+
+  # Raised when a response from a server (HTTP or tracker) is not in a format
+  # that we expected (or truncated..
   class InvalidResponseError < Error; end
+
+  # Raised when all known backends have failed.
   class UnreachableBackendError < Error; end
 
   # IO.copy_stream was buggy in Ruby 1.9.2 and earlier
diff --git a/lib/mogilefs/admin.rb b/lib/mogilefs/admin.rb
index b940902..abacfc8 100644
--- a/lib/mogilefs/admin.rb
+++ b/lib/mogilefs/admin.rb
@@ -1,7 +1,7 @@
 # -*- encoding: binary -*-
-##
-# A MogileFS Administration Client
 
+# \MogileFS administration client, this has little real-world usage
+# and is considered a work-in-progress
 class MogileFS::Admin < MogileFS::Client
 
   ##
diff --git a/lib/mogilefs/backend.rb b/lib/mogilefs/backend.rb
index dd775ca..518e67e 100644
--- a/lib/mogilefs/backend.rb
+++ b/lib/mogilefs/backend.rb
@@ -1,14 +1,12 @@
 # -*- encoding: binary -*-
 require 'thread'
 
-##
-# MogileFS::Backend communicates with the MogileFS trackers.
-
+# This class communicates with the MogileFS trackers.
+# You should not have to use this directly unless you are developing
+# support for new commands or plugins for MogileFS
 class MogileFS::Backend
 
-  ##
   # Adds MogileFS commands +names+.
-
   def self.add_command(*names)
     names.each do |name|
       define_method name do |*args|
@@ -17,6 +15,8 @@ class MogileFS::Backend
     end
   end
 
+  # adds idempotent MogileFS commands +names+, these commands may be retried
+  # transparently on a different tracker if there is a network/server error.
   def self.add_idempotent_command(*names)
     names.each do |name|
       define_method name do |*args|
@@ -25,7 +25,7 @@ class MogileFS::Backend
     end
   end
 
-  BACKEND_ERRORS = {}
+  BACKEND_ERRORS = {} # :nodoc:
 
   # this converts an error code from a mogilefsd tracker to an exception:
   #
@@ -160,9 +160,7 @@ class MogileFS::Backend
     end
   end
 
-  ##
   # Performs the +cmd+ request with +args+.
-
   def do_request(cmd, args, idempotent = false)
     request = make_request cmd, args
     @mutex.synchronize do
@@ -199,9 +197,7 @@ class MogileFS::Backend
     end # @mutex.synchronize
   end
 
-  ##
   # Makes a new request string for +cmd+ and +args+.
-
   def make_request(cmd, args)
     "#{cmd} #{url_encode args}\r\n"
   end
@@ -214,10 +210,8 @@ class MogileFS::Backend
     BACKEND_ERRORS[err_snake] || self.class.add_error(err_snake)
   end
 
-  ##
   # Turns the +line+ response from the server into a Hash of options, an
   # error, or raises, as appropriate.
-
   def parse_response(line)
     if line =~ /^ERR\s+(\w+)\s*([^\r\n]*)/
       @lasterr = $1
@@ -231,9 +225,7 @@ class MogileFS::Backend
           "Invalid response from server: #{line.inspect}"
   end
 
-  ##
   # Returns a socket connected to a MogileFS tracker.
-
   def socket
     return @socket if @socket and not @socket.closed?
 
@@ -259,40 +251,39 @@ class MogileFS::Backend
           "couldn't connect to any tracker: #{errors.join(', ')}"
   end
 
-  ##
   # Turns a url params string into a Hash.
-
-  def url_decode(str)
+  def url_decode(str) # :nodoc:
     Hash[*(str.split(/&/).map! { |pair|
       pair.split(/=/, 2).map! { |x| url_unescape(x) }
     } ).flatten]
   end
 
-  ##
-  # Turns a Hash (or Array of pairs) into a url params string.
+  # :stopdoc:
+  # TODO: see if we can use existing URL-escape/unescaping routines
+  # in the Ruby standard library, Perl MogileFS seems to NIH these
+  #  routines, too
+  # :startdoc:
 
-  def url_encode(params)
+  # Turns a Hash (or Array of pairs) into a url params string.
+  def url_encode(params) # :nodoc:
     params.map do |k,v|
       "#{url_escape k.to_s}=#{url_escape v.to_s}"
     end.join("&")
   end
 
-  ##
   # Escapes naughty URL characters.
   if ''.respond_to?(:ord) # Ruby 1.9
-    def url_escape(str)
+    def url_escape(str) # :nodoc:
       str.gsub(/([^\w\,\-.\/\\\: ])/) { "%%%02x" % $1.ord }.tr(' ', '+')
     end
   else # Ruby 1.8
-    def url_escape(str)
+    def url_escape(str) # :nodoc:
       str.gsub(/([^\w\,\-.\/\\\: ])/) { "%%%02x" % $1[0] }.tr(' ', '+')
     end
   end
 
-  ##
   # Unescapes naughty URL characters.
-
-  def url_unescape(str)
+  def url_unescape(str) # :nodoc:
     str.gsub(/%([a-f0-9][a-f0-9])/i) { [$1.to_i(16)].pack 'C' }.tr('+', ' ')
   end
 end
diff --git a/lib/mogilefs/client.rb b/lib/mogilefs/client.rb
index 7268c7c..9a2f34b 100644
--- a/lib/mogilefs/client.rb
+++ b/lib/mogilefs/client.rb
@@ -4,7 +4,6 @@
 # MogileFS::Client is the MogileFS client base class.  Concrete clients like
 # MogileFS::MogileFS and MogileFS::Admin are implemented atop this one to do
 # real work.
-
 class MogileFS::Client
 
   ##
diff --git a/lib/mogilefs/mogilefs.rb b/lib/mogilefs/mogilefs.rb
index adb9451..d942373 100644
--- a/lib/mogilefs/mogilefs.rb
+++ b/lib/mogilefs/mogilefs.rb
@@ -1,25 +1,46 @@
 # -*- encoding: binary -*-
 
-##
-# MogileFS File manipulation client.
-
+# \MogileFS file manipulation client.
+#
+#   Create a new instance that will communicate with these trackers:
+#   hosts = %w[192.168.1.69:6001 192.168.1.70:6001]
+#   mg = MogileFS::MogileFS.new(:domain => 'test', :hosts => hosts)
+#
+#   # Stores "A bunch of text to store" into 'some_key' with a class of 'text'.
+#   mg.store_content('some_key', 'text', "A bunch of text to store")
+#
+#   # Retrieve data from 'some_key' as a string
+#   data = mg.get_file_data('some_key')
+#
+#   # Store the contents of 'image.jpeg' into the key 'my_image' with a
+#   # class of 'image'.
+#   mg.store_file('my_image', 'image', 'image.jpeg')
+#
+#   # Store the contents of 'image.jpeg' into the key 'my_image' with a
+#   # class of 'image' using an open IO object.
+#   File.open('image.jpeg') { |fp| mg.store_file('my_image', 'image', fp) }
+#
+#   # Retrieve the contents of 'my_image' into '/path/to/huge_file'
+#   # without slurping the entire contents into memory:
+#   File.open('/path/to/huge_file', 'w') do |fp|
+#     mg.get_file_data('my_image', fp)
+#   end
+#
+#   # Remove the key 'my_image' and 'some_key'.
+#   mg.delete('my_image')
+#   mg.delete('some_key')
+#
 class MogileFS::MogileFS < MogileFS::Client
-
   include MogileFS::Bigfile
 
-  ##
   # The domain of keys for this MogileFS client.
   attr_accessor :domain
 
-  ##
   # The timeout for get_file_data.  Defaults to five seconds.
-
   attr_accessor :get_file_data_timeout
 
-  ##
   # Creates a new MogileFS::MogileFS instance.  +args+ must include a key
   # :domain specifying the domain of this client.
-
   def initialize(args = {})
     @domain = args[:domain]
 
@@ -34,9 +55,7 @@ class MogileFS::MogileFS < MogileFS::Client
     end
   end
 
-  ##
-  # Enumerates keys starting with +key+.
-
+  # Enumerates keys, limited by optional +prefix+
   def each_key(prefix = "")
     after = nil
 
@@ -50,17 +69,17 @@ class MogileFS::MogileFS < MogileFS::Client
     nil
   end
 
-  ##
-  # Retrieves the contents of +key+.  If +dest+ is specified, +dest+
+  # Retrieves the contents of +key+.  If +dst+ is specified, +dst+
   # should be an IO-like object capable of receiving the +write+ method
-  # or a path name.
-
-  def get_file_data(key, dest = nil, count = nil, offset = nil)
+  # or a path name.  +copy_length+ may be specified to limit the number of
+  # bytes to retrieve, and +src_offset+ can be specified to specified the
+  # start position of the copy.
+  def get_file_data(key, dst = nil, copy_length = nil, src_offset = nil)
     paths = get_paths(key)
     sock = MogileFS::HTTPReader.first(paths, @get_file_data_timeout,
-                                      count, offset)
-    if dest
-      sock.stream_to(dest)
+                                      copy_length, src_offset)
+    if dst
+      sock.stream_to(dst)
     elsif block_given?
       yield(sock)
     else
@@ -70,9 +89,15 @@ class MogileFS::MogileFS < MogileFS::Client
       sock.close if sock && ! sock.closed?
   end
 
-  ##
-  # Get the paths for +key+.
-
+  # Get the paths (URLs as strings) for +key+.  If +args+ is specified,
+  # it may contain:
+  # - :noverify -> boolean, whether or not the tracker checks (default: true)
+  # - :pathcount -> a positive integer of URLs to retrieve (default: 2)
+  # - :zone -> "alt" or nil (default: nil)
+  #
+  # :noverify defaults to false because this client library is capable of
+  # verifying paths for readability itself.  It is also faster and more
+  # reliable to verify paths on the client.
   def get_paths(key, *args)
     opts = {
       :domain => @domain,
@@ -93,18 +118,15 @@ class MogileFS::MogileFS < MogileFS::Client
     (1..res['paths'].to_i).map { |i| res["path#{i}"] }
   end
 
-  ##
-  # Get the URIs for +key+.
-
+  # Get the URIs for +key+ (paths) as URI::HTTP objects
   def get_uris(key, *args)
     get_paths(key, *args).map! { |path| URI.parse(path) }
   end
 
-  ##
   # Creates a new file +key+ in +klass+.  +bytes+ is currently unused.
-  #
-  # The +block+ operates like File.open.
-
+  # Consider using store_file instead of this method for large files.
+  # This requires a block passed to it and operates like File.open.
+  # This atomically replaces existing data stored as +key+ when
   def new_file(key, klass = nil, bytes = 0) # :yields: file
     raise MogileFS::ReadOnlyError if readonly?
     opts = { :domain => @domain, :key => key, :multi_dest => 1 }
@@ -146,20 +168,18 @@ class MogileFS::MogileFS < MogileFS::Client
     end
   end
 
-  ##
   # Copies the contents of +file+ into +key+ in class +klass+.  +file+ can be
   # either a path name (String or Pathname object) or an IO-like object that
   # responds to #read or #readpartial.  Returns size of +file+ stored.
-
+  # This atomically replaces existing data stored as +key+
   def store_file(key, klass, file)
     raise MogileFS::ReadOnlyError if readonly?
 
     new_file(key, klass) { |mfp| mfp.big_io = file }
   end
 
-  ##
-  # Stores +content+ into +key+ in class +klass+.
-
+  # Stores +content+ into +key+ in class +klass+, where +content+ is a String
+  # This atomically replaces existing data stored as +key+
   def store_content(key, klass, content)
     raise MogileFS::ReadOnlyError if readonly?
 
@@ -172,9 +192,7 @@ class MogileFS::MogileFS < MogileFS::Client
     end
   end
 
-  ##
   # Removes +key+.
-
   def delete(key)
     raise MogileFS::ReadOnlyError if readonly?
 
@@ -182,16 +200,12 @@ class MogileFS::MogileFS < MogileFS::Client
     true
   end
 
-  ##
-  # Sleeps +duration+.
-
-  def sleep(duration)
+  # Sleeps +duration+, only used for testing
+  def sleep(duration) # :nodoc:
     @backend.sleep :duration => duration
   end
 
-  ##
   # Renames a key +from+ to key +to+.
-
   def rename(from, to)
     raise MogileFS::ReadOnlyError if readonly?
 
@@ -199,7 +213,6 @@ class MogileFS::MogileFS < MogileFS::Client
     nil
   end
 
-  ##
   # Returns the size of +key+.
   def size(key)
     @backend.respond_to?(:_size) and return @backend._size(domain, key)
@@ -215,10 +228,8 @@ class MogileFS::MogileFS < MogileFS::Client
     MogileFS::PathsSize.call(paths)
   end
 
-  ##
-  # Lists keys starting with +prefix+ follwing +after+ up to +limit+.  If
+  # Lists keys starting with +prefix+ following +after+ up to +limit+.  If
   # +after+ is nil the list starts at the beginning.
-
   def list_keys(prefix = "", after = nil, limit = 1000)
     if @backend.respond_to?(:_list_keys)
       block_given? or return @backend._list_keys(domain, prefix, after, limit)
diff --git a/lib/mogilefs/util.rb b/lib/mogilefs/util.rb
index cdd67d7..6277497 100644
--- a/lib/mogilefs/util.rb
+++ b/lib/mogilefs/util.rb
@@ -1,6 +1,13 @@
 # -*- encoding: binary -*-
 
 module MogileFS::Util
+
+  # MogileFS::Util::StoreContent allows you to roll your own method
+  # of streaming data on an upload (instead of using a string or file)
+  #
+  # Current versions of this library support streaming a IO or IO-like
+  # object to using MogileFS::MogileFS#store_file, so using StoreContent
+  # may no longer be necessary.
   class StoreContent < Proc
     def initialize(total_size, &writer_proc)
       @total_size = total_size