From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 5E18C20D0E for ; Fri, 27 Jan 2017 02:56:28 +0000 (UTC) From: Eric Wong To: mogilefs-client-public@bogomips.org Subject: [PATCH] avoid excessive garbage on uploads with Ruby 2.2+ Date: Fri, 27 Jan 2017 02:56:28 +0000 Message-Id: <20170127025628.4524-1-e@80x24.org> List-Id: This is a workaround for since we use non-blocking sockets anyways. --- lib/mogilefs/socket_common.rb | 12 ++++++++++++ test/test_cmogstored.rb | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/lib/mogilefs/socket_common.rb b/lib/mogilefs/socket_common.rb index 5ba8c10..ad66164 100644 --- a/lib/mogilefs/socket_common.rb +++ b/lib/mogilefs/socket_common.rb @@ -56,4 +56,16 @@ def read(size, buf = "", timeout = 5) def readpartial(size, buf = "", timeout = 5) timed_read(size, buf, timeout) or raise EOFError, "end of file reached" end + + # workaround for https://bugs.ruby-lang.org/issues/13085 + # (excessive garbage from IO#write) + # XXX maybe this can be fixed for Ruby 2.5 final, but maybe not: + # Update this when Ruby 2.5 is released on 2017-12-25 + if RUBY_VERSION.to_f >= 2.2 && RUBY_VERSION.to_f <= 2.5 + def write(buf) + # Blocking TCP writes would error out long before one day, + # and MogileFS won't allow file creations which take over a day. + timed_write(buf, 86400) + end + end end diff --git a/test/test_cmogstored.rb b/test/test_cmogstored.rb index 1036429..d112d38 100644 --- a/test/test_cmogstored.rb +++ b/test/test_cmogstored.rb @@ -1,6 +1,20 @@ # -*- encoding: binary -*- require "./test/fresh" +# The goal of this is to use a synthetic (non-IO) reader +# to trigger the read/write loop of IO.copy_stream, +# bypassing in-kernel mechanisms like sendfile for zero copy, +# so we wrap the /dev/zero IO object: +class Zero + def initialize + @in = File.open('/dev/zero', 'rb') + end + + def read(len, buf) + @in.read(len, buf) + end +end if File.readable?('/dev/zero') + class Test_cmogstored < Test::Unit::TestCase include TestFreshSetup alias setup setup_mogilefs @@ -26,6 +40,24 @@ def test_range_put_new_file assert_equal "a\nb\nc\nd\ne\n", client.get_file_data("puts") end + def test_garbage + add_host_device_domain + client = MogileFS::MogileFS.new :hosts => @hosts, :domain => @domain, + :timeout => 60 + nr = 1024 * 1024 * 1024 + client.new_file('giant', :largefile => :stream, + :content_length => nr) do |io| + assert_instance_of MogileFS::NewFile::Stream, io + zero = Zero.new + before = GC.count + wr = IO.copy_stream(zero, io, nr) + after = GC.count + assert_equal nr, wr + assert_in_delta before, after, 1 + end + end if IO.respond_to?(:copy_stream) && defined?(Zero) && + GC.respond_to?(:count) && ENV['TEST_EXPENSIVE'].to_i != 0 + def test_stream_new_file add_host_device_domain client = MogileFS::MogileFS.new :hosts => @hosts, :domain => @domain -- EW