about summary refs log tree commit homepage
path: root/lib/mogilefs/mogilefs.rb
DateCommit message (Collapse)
2011-11-08get_uri: object allocation reduction
Using Array#map! instead of Array#map can save us at least one object allocation.
2011-11-08http_reader: remove support for overriding http_method
GET is all we need and we can save some code this way. If we ever need to use HEAD much again, we can use net/http/persistent since our internal HTTP classes are only optimized for large responses.
2011-11-08split deprecated paths_size to its own file
This is only needed for users on old MogileFS servers
2011-11-08get_file_data: avoid exposing users to copy_stream invocation
We'll now accept an optional argument which can be passed to IO.copy_stream directly. This should make life easier on users so they won't be exposed to our internals to make efficient copies of large files.
2011-11-08size/list_keys improvements
We can use the file_info command to get things faster, now.
2011-11-08add support for the file_debug command
This was added in MogileFS 2.45
2011-11-08add file_info command support and test
This is a command added in MogileFS 2.45
2011-11-07avoid circular requires
Ruby 1.9.3 considers them harmful
2011-11-07fix return value for chunked Util::StoreContent users
Using unknown sizes with StoreContent is now supported (but you're probably better off using a pipe or just and object that acts like an IO)
2011-11-07store_file works on IO#read-like methods, too
The readpartial is not in the Rack spec for rack.input objects, but something like IO#read is.
2011-11-07flesh out pipe streaming + chunk for uploads
Of course the backend server needs to support chunking, but the latest Perlbal does.
2011-11-06delete returns true on success
Not some random hash from the parser.
2011-11-05return file size properly when storing file
2011-11-05support chunked uploads via PUT
This can be useful for streaming to a backend (this feature needs tests)
2011-11-05redo IO.copy_stream usage
Avoid deepening stack depth and make it easier to migrate fully to 1.9 in the future (dropping 1.8 support).
2011-11-05make EmptyPathError more descriptive
2011-11-05rename httpfile => http_file
Should be easier to read this way
2011-11-05refactor httpfile and remove layering violation
Splitting calls to backend.create_open + create_close between files is also extremely confusing and error-prone. Hopefully nobody actually depends on some attributes we've removed.
2011-11-04new HTTPReader implementation replaces the http_read_sock method
This is cleaner and replaces redundant code where we would retry paths. MogileFS::MogileFS#size now raises on error instead of returning nil.
2011-11-02remove sysrwloop for copy_stream
We're trying to use as much as we can from Ruby 1.9
2011-11-02mogilefs: don't pass class argument if it's "default"
mogilefsd may not like that
2011-11-02make http_read_sock and friends use new socket API
This will need further refactoring
2009-11-20set encoding: binary for all files
We won't trust Ruby 1.9 String weirdness since data storage is locale-agnostic
2009-04-11Ensure store_{content,file} always returns size
It's useful to know the size of the file we're storing.
2009-03-24Add get_uris API method
This is just like get_paths, but integrates better into Ruby applications that use the parsed-out URI to do operations directly on the URIs.
2009-03-24Really remove all NFS support
This cleans up some of the internal HTTP handling code a bit, too; and does a better job of closing sockets than it did previously.
2009-03-24Compact get_paths output so we don't have nils
I'm not sure if mogilefsd ever returned broken output for this, but just in case we compact the output so users don't have to worry about them.
2009-03-24Unify internal HTTP GET/HEAD methods
This takes advantage of the (ugly) new mogilefs_size Socket attribute to avoid duplicating Content-Length parsing code. I really wish Net::HTTP in Ruby was actually usable...
2009-03-24Respect timeout when doing get_file_data
This adds a sysread_full utility method with configurable timeouts. Individual reads can be timed out as well as the entire sysread_full call.
2009-03-24call IO.select([sock]) if sock is not ready.
2009-02-23Allow 'store_content' to be used with a streamable object
New way to call 'store_content' with a MogileFS::Util::StoreContent allows you to roll your own method of streaming data to mogile on an upload (instead of using a string or file) [ew: this still requires a known content length beforehand] [ew: applied with --whitespace=strip, rewritten subject, 80-column wrapping] Signed-off-by: Eric Wong <normalperson@yhbt.net>
2009-02-07Force binary encoding on all objects
File and StringIO objects need to be opened in binary mode, otherwise they take the default encoding format. Thankfully, Sockets and Tempfile objects seem to be binary by default as of 1.9.1; but it really is a mess to have to deal with FS abstractions that try to deal with encoding crap behind your back...
2009-01-23Remove unnecessary statements and temp variables
Last I checked, the trailing "return" is not optimized away by MRI 1.8. Additionally, remove some useless temporary variables.
2009-01-23allow using the default class to store files
Don't specify an empty class (e.g. "class="), instead just omit the parameter entirely if it is nil.
2009-01-05Mysql: fix get_paths
Oops, this method needs to bypass transformations we in the normal MogileFS::MogileFS code path.
2008-12-30Don't let size(key) get tricked by non-200 OK responses
We never checked the HTTP status code when making the HEAD request. All this HTTP stuff should probably be moved to HTTPFile
2008-12-30Cleanup and document http_get_sock
Correctly fail when we get non-200 HTTP responses and retry on the next URI.
2008-12-23HTTPFile: handle multiple device failover correctly
Previously, when we got multiple destinations to upload to and one of them failed, we failed to correctly retry the next destination. This will set the correct devid and URL.
2008-12-23MogileFS::MogileFS: simplify get_paths
We dropped NFS support, so this can be simplified further.
2008-12-23Remove NFS support
MogileFS 2.x upstream no longer supports it, and it's become a maintenance burden and NFS is a horrible thing anyways. Attempting to use this with servers that support NFS will result in MogileFS::UnsupportedPathError being raised.
2008-12-22HTTPFile: rename bigfile accessor to big_io
"bigfile" is used by the large file support, too. "big_io" also allows an IO object to be sent to us, as well (and not a pathname)
2008-12-22Replace TCPSocket + timeout code with Socket + IO.select
This removes the dependency on unsafe methods used in the Timeout class. Charles makes some good points here: http://blog.headius.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html And even matz agrees: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/294446 Of course, I strongly dislike any unnecessary use of threads, and implementations using native threads to do timeouts makes me even more uncomfortable.
2008-12-18Allow MogileFS::MogileFS to use :db_backend => MogileFS::Mysql
This should complete the integration of the read-only Mysql backend into MogileFS::MogileFS.
2008-12-10initial bigfile read support
Needs more tests, but it seems to work... I seem to have discovered a bug in mogtool which causes it to generate incorrect MD5 checksums when the --gzip flag is used (and --gzip actually just does zlib deflate, not something that gzip(1) can actually decrypt). So right now MD5 checksums are only verified on non-zlib-deflated files.
2008-10-31Move http_get_sock into its own function
We'll be reusing it in the big file module
2008-10-15Avoid double-close on bigfiles
This was leading to ugly "no_temp_file" errors that got converted to exceptions.
2008-10-15list_keys trap exception in this method
2008-10-15Replace the last of the ad-hoc RuntimeError exceptions
The ArgumentErrors happen at initialization time, so I'll keep those as-is
2008-10-15Raise MogileFS::ReadOnly error for readonly instances
Also easier to trap and deal with
2008-10-15Use exceptions for all errors sent to us from the backend