| Date | Commit message (Collapse) |
|---|
|
This allows clients to avoid calling #file_info or #get_uris
immediate after uploading a file to MogileFS. This can speed
things up for cache-using clients with write-through caching.
|
|
Array#pop is faster than Array#shift in most Ruby versions
as the latter may require memmove() of all elements.
Additionally, ensure ordering is correct if a backend
chokes up (for each_file_info, too).
|
|
This allows fast listing of keys and metadata (length,
checksum, devcount, class).
|
|
Blank observed_state (for dead devices) are mapped to nil.
|
|
By avoiding #grep, installations with thousands of hosts/devices
should be sped up considerably. list_fids should be roughly
twice as fast.
This is a followup to commit a309f22f835afe0e6be0e4e2f1a13eaead7434f2
|
|
On an installation with 3000+ domains, get_domains gets
bottlenecked by calling #grep on the keys of a hash. Allow
specifying a "want" directive to filter in data we expect
(instead of grepping blindly)
|
|
While retrying idempotent requests (even on timeouts) would
prevent stale sockets from being noticed, it is better to kill
the socket and immediately propagate the timeout error to the
user. Retrying in a timeout may cause a request/response to
take longer (perhaps _much_ longer) than the timeout configured
by the user.
|
|
* backend/do_request: when a request times out for a slow server,
we now continue retries until we get a socket error, and close the
connection afterward so we don't get interspersed responses
* test: added a test for slow servers
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
Retrying to open a source file that cannot be opened
(repeatedly) and finally raising NoStorageNodesError is
confusing to users.
|
|
This makes things easier-to-read, and admin functions aren't
performance critical so we won't worry about it.
|
|
This is a boolean value on the server and we always try to
make our return values Ruby-friendly
|
|
This feature uses the "set_weight" command to change
device weights and has a part of the MogileFS protocol
for many years, now.
|
|
The ability to create devices from the Ruby API might be
useful to somebody, especially when writing test cases.
|
|
The server sending us a properly-formed "ERR" response is a big
difference than an actual socket or connection error.
|
|
Specifying an invalid domain will raise
MogileFS::Backend::UnregDomain error instead of merely
returning `false'. Only checks for keys in the correct
domain (but non-existent keys) return false.
|
|
list_keys needs to raise UnregDomainError on invalid domains.
This regression was introduced in
commit 0b933fc83b8b519c8e587caa6606dd372dda18af
|
|
Now that checksum support is officially a part of MogileFS, we
can document (and thus encourage) it without risking
compatibility issues.
|
|
This changes the class associated with +key+.
This is _not_ the same as the "update_class" admin command
which actually modifies the class itself.
|
|
It's more future-proof, this way.
|
|
Leave size unchanged, so it's available for easy comparison with
buf.bytesize.
[ew: added test case]
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
:noverify always defaults to _true_ (meaning verification is
_off_ by default). This double negative is unfortunately
a confusing part of the existing API and MogileFS protocol.
|
|
We have lots of timeouts :x
|
|
Most of these exceptions are just too rare to be useful,
and code that cares for exceptions will rescue them
anyways.
|
|
This controls the retry timeout of a failed backend
|
|
This will make debug output quieter.
|
|
This adds an internal :ruby_no_raise flag to the backend
to avoid exceptions.
|
|
Performance with Content-Range uploads sucks either way, so it's
not noticeably worse off on a LAN /without/ persistent
connections.
|
|
The client-provided Content-MD5 may have leading/trailing
whitespace, so we shall strip that before comparing.
We also know any base64-encoded MD5 we generate will have only
trailing whitespace so we can get away with the (slightly)
faster and more-GC-friendly String#rstrip!
|
|
This defaults to 1 hour (which matches the expiry time of a row
in the MogileFS internal tempfile table).
|
|
100% RDoc coverage (not that it's an indicator of _good_
documentation, but we're getting there)
|
|
|
|
This should make documentation easier
|
|
An upload could've taken a long time, ping and try to ensure
socket is valid to minimize (but not completely eliminate) the
chance create_close hits a stale socket (while reading the
response after writing to it) and becomes non-retryable. We
treat create_close specially as its less idempotent than any
other command (even other non-idempotent ones). There may be no
hope of retrying the upload at all if data was streamed and
calling create_close twice will hurt us...
|
|
If we started a connection, make sure it's stopped and the file
descriptor closed. Relying on GC to close file descriptors is
unreliable with some GC implementations.
|
|
Now that pools can be arbitrarily large, using an O(n) check
for object validity can be expensive.
|
|
Instead of hard coding the thresholds, allow
users to change accessors
|
|
Putting this into the MogileFS::Put::* namespace will
make it easier to document.
|
|
This allows us to stream files with a known Content-Length
into MogileFS. This can be useful for streaming an HTTP
download into MogileFS without:
* saving it to the filesystem
* relying on chunked encoding support on the server
|
|
This way, folks can rescue for exceptions we haven't defined,
yet.
|
|
list_fids was being used incorrectly :x
|
|
Unlinking Tempfiles is good practice, so we should
allow/encourage people to use unlinked Tempfiles
with store_file.
|
|
This is the most-compatible way to support largefiles
with the new_file interface. This will unfortunately
generate disk I/O, though...
Also moving the "mog" util to use this with the "tee"
subcommand since it already included its own private
implementation of this before.
|
|
This makes it easier to use an unlinked Tempfile
|
|
This is similar to the "largefile => 1" support in the
Perl MogileFS::Client package. It requires net/http/persistent
to avoid repeatedly setting up and tearing down a socket.
|
|
This returns a new HTTPStream object that behaves
like a writable IO object with the following methods:
* write
* print
* printf
* putc
* puts
* syswrite
* <<
..and also responds to IO.select (for writability)
|
|
TCP keepalives are inexpensive, so we can use them to monitor
whether or not our connection is still alive while uploading.
Remote servers make take an unpredictable amount of time to
actually write out the data we've uploaded (and empty socket
buffers to receive more), so it is extremely difficult to
calculate an effective timeout for select() or poll().
|
|
This reverts commit 11ee57e7aaa8766d13e29d3872bd78d50a3e5263.
Bleh, checksums support is already off by default...
|
|
We won't force this onto the wire for trackers until upstream
MogileFS decides to support it.
|
|
Users will be able to tweak these themselves in OS-dependent
ways. It's probably better to rely on protocol-level timeouts
when the HTTP server should be on a trusted network.
|
|
These allow us to specify Content-MD5 and checksums for use
with the new and improved new_file interface without breaking
existing apps.
|