Ruby mogilefs-client dev/users discussion/patches/bugs/help/...
 help / color / Atom feed
* [PATCH] avoid excessive garbage on uploads with Ruby 2.2+
@ 2017-01-27  2:56 Eric Wong
  0 siblings, 0 replies; only message in thread
From: Eric Wong @ 2017-01-27  2:56 UTC (permalink / raw)
  To: mogilefs-client-public

This is a workaround for <https://bugs.ruby-lang.org/issues/13085>
since we use non-blocking sockets anyways.
---
 lib/mogilefs/socket_common.rb | 12 ++++++++++++
 test/test_cmogstored.rb       | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/lib/mogilefs/socket_common.rb b/lib/mogilefs/socket_common.rb
index 5ba8c10..ad66164 100644
--- a/lib/mogilefs/socket_common.rb
+++ b/lib/mogilefs/socket_common.rb
@@ -56,4 +56,16 @@ def read(size, buf = "", timeout = 5)
   def readpartial(size, buf = "", timeout = 5)
     timed_read(size, buf, timeout) or raise EOFError, "end of file reached"
   end
+
+  # workaround for https://bugs.ruby-lang.org/issues/13085
+  # (excessive garbage from IO#write)
+  # XXX maybe this can be fixed for Ruby 2.5 final, but maybe not:
+  # Update this when Ruby 2.5 is released on 2017-12-25
+  if RUBY_VERSION.to_f >= 2.2 && RUBY_VERSION.to_f <= 2.5
+    def write(buf)
+      # Blocking TCP writes would error out long before one day,
+      # and MogileFS won't allow file creations which take over a day.
+      timed_write(buf, 86400)
+    end
+  end
 end
diff --git a/test/test_cmogstored.rb b/test/test_cmogstored.rb
index 1036429..d112d38 100644
--- a/test/test_cmogstored.rb
+++ b/test/test_cmogstored.rb
@@ -1,6 +1,20 @@
 # -*- encoding: binary -*-
 require "./test/fresh"
 
+# The goal of this is to use a synthetic (non-IO) reader
+# to trigger the read/write loop of IO.copy_stream,
+# bypassing in-kernel mechanisms like sendfile for zero copy,
+# so we wrap the /dev/zero IO object:
+class Zero
+  def initialize
+    @in = File.open('/dev/zero', 'rb')
+  end
+
+  def read(len, buf)
+    @in.read(len, buf)
+  end
+end if File.readable?('/dev/zero')
+
 class Test_cmogstored < Test::Unit::TestCase
   include TestFreshSetup
   alias setup setup_mogilefs
@@ -26,6 +40,24 @@ def test_range_put_new_file
     assert_equal "a\nb\nc\nd\ne\n", client.get_file_data("puts")
   end
 
+  def test_garbage
+    add_host_device_domain
+    client = MogileFS::MogileFS.new :hosts => @hosts, :domain => @domain,
+                                    :timeout => 60
+    nr = 1024 * 1024 * 1024
+    client.new_file('giant', :largefile => :stream,
+                    :content_length => nr) do |io|
+      assert_instance_of MogileFS::NewFile::Stream, io
+      zero = Zero.new
+      before = GC.count
+      wr = IO.copy_stream(zero, io, nr)
+      after = GC.count
+      assert_equal nr, wr
+      assert_in_delta before, after, 1
+    end
+  end if IO.respond_to?(:copy_stream) && defined?(Zero) &&
+         GC.respond_to?(:count) && ENV['TEST_EXPENSIVE'].to_i != 0
+
   def test_stream_new_file
     add_host_device_domain
     client = MogileFS::MogileFS.new :hosts => @hosts, :domain => @domain
-- 
EW


^ permalink raw reply	[flat|nested] only message in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-27  2:56 [PATCH] avoid excessive garbage on uploads with Ruby 2.2+ Eric Wong

Ruby mogilefs-client dev/users discussion/patches/bugs/help/...

Archives are clonable:
	git clone --mirror https://yhbt.net/mogilefs-client-public
	git clone --mirror http://ou63pmih66umazou.onion/mogilefs-client-public

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.file-systems.mogilefs.ruby
	nntp://ou63pmih66umazou.onion/inbox.comp.file-systems.mogilefs.ruby

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git