merge large file patch from Andy La Foe

Merging involved "Content-length" => "Content-Length"
capitalizing 'L' as per p4#3627 (aka SVN r433)

Ref: http://rubyforge.org/tracker/index.php?func=detail&aid=13764&group_id=1513&atid=5923
> Submitted By: Andy Lo-A-Foe (arloafoe)
> Category: mogilefs-client
> Summary:
> Store very large files (> 256M) without running out of memory in store_file
>
> Detailed description
>
> This is a patch to the MogileFS::store_file mechanism in order to
> support very large filee stores using HTTPFile. We sometimes have to
> store files of up to 1GB in size. Using chunking is not really an option
> since it has proven to be very unreliable (mogtool) and there is no
> support for it in the current version of this client. This patch
> basically reads 16M chunks at a time and writes them to the tracker
> socket instead of trying to stuff the while file in the StringIO and
> running out of memory. It's probably very rough and the get_file_data
> symmetry patch is not there yet. Feedback appreciated.

2 files changed, 27 insertions, 2 deletions

diff --git a/lib/mogilefs/httpfile.rb b/lib/mogilefs/httpfile.rb
index b795e9b..acfd1e0 100644
--- a/lib/mogilefs/httpfile.rb
+++ b/lib/mogilefs/httpfile.rb
@@ -35,6 +35,11 @@ class MogileFS::HTTPFile < StringIO
   attr_reader :class
 
   ##
+  # The bigfile name in case we have file > 256M
+
+  attr_accessor :bigfile
+
+  ##
   # Works like File.open.  Use MogileFS::MogileFS#new_file instead of this
   # method.
 
@@ -62,6 +67,7 @@ class MogileFS::HTTPFile < StringIO
     @devid = devid
     @klass = klass
     @key = key
+    @bigfile = nil
 
     @dests = dests.map { |(_,u)| URI.parse u }
     @tried = {}
@@ -75,7 +81,20 @@ class MogileFS::HTTPFile < StringIO
   def close
     connect_socket
 
-    @socket.write "PUT #{@path.request_uri} HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n#{string}"
+    file_size = nil
+    if @bigfile
+      # Don't try to run out of memory
+      fp = File.open(@bigfile)
+      file_size = File.size(@bigfile)
+      @socket.write "PUT #{@path.request_uri} HTTP/1.0\r\nContent-Length: #{file_size}\r\n\r\n"
+      while not fp.eof?
+        chunk = fp.read 16384000
+        @socket.write chunk
+      end
+      fp.close
+    else
+      @socket.write "PUT #{@path.request_uri} HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n#{string}"
+    end
 
     if connected? then
       line = @socket.gets
@@ -98,6 +117,7 @@ class MogileFS::HTTPFile < StringIO
     @mg.backend.create_close(:fid => @fid, :devid => @devid,
                              :domain => @mg.domain, :key => @key,
                              :path => @path, :size => length)
+    return file_size if @bigfile
     return nil
   end
 
diff --git a/lib/mogilefs/mogilefs.rb b/lib/mogilefs/mogilefs.rb
index 396a15d..6bcc5b4 100644
--- a/lib/mogilefs/mogilefs.rb
+++ b/lib/mogilefs/mogilefs.rb
@@ -161,7 +161,12 @@ class MogileFS::MogileFS < MogileFS::Client
       if file.respond_to? :read then
         return copy(file, mfp)
       else
-        return File.open(file) { |fp| copy(fp, mfp) }
+        if File.size(file) > (256 * 1024 * 1024) # Bigass file, handle differently
+          mfp.bigfile = file
+          return mfp.close
+        else
+          return File.open(file) { |fp| copy(fp, mfp) }
+        end
       end
     end
   end