Re: Bug#830353: ruby-kgio: FTBFS: ERROR: Test "ruby2.3" failed.

Re: Bug#830353: ruby-kgio: FTBFS: ERROR: Test "ruby2.3" failed.

[Eric Wong]

Lucas Nussbaum <lucas@debian.org> wrote:
> The full build log is available from:
>    http://people.debian.org/~lucas/logs/2016/07/07/ruby-kgio_2.10.0-1_unstable_reb.normal.log
> 
> A list of current common problems and possible solutions is available at
> http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!
> 
> About the archive rebuild: The rebuild was done on EC2 VM instances from
> Amazon Web Services, using a clean, minimal and up-to-date chroot. Every
> failed build was retried once to eliminate random failures.

Thanks for the report.  Can you try the patch below?

I'd be interested to know what non-standard sysctl knobs are set
for kernel socket buffer sizes (in /proc/sys/net/{core,ipv4}/*mem*)

You can also increase the 20M to 30M or whatever (otoh, that
means using more memory during tests :<)

I'm not comfortable calling setsockopt to set SO_{SND,RCV}BUF
knobs since that can trigger VM deadlock bugs on some older
kernels, too.

Anyways, most of the kgio functionality is in Ruby 2.3 nowadays
and kgio can probably go away when support for 2.2 can be
dropped in gems.

----8<----
Subject: [PATCH] test: increase random blob size for giant socket buffers

Socket buffers on some systems may be too big to trigger
partial write/blocking behavior we need for our tests.
Increase it to 20M for now and see if we can fix an FTBFS
on larger machines: https://bugs.debian.org/830353

This should also speed up tests a little since urandom is
slow and reading 10M from it in the first place was lazy.
---
 test/lib_read_write.rb | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/test/lib_read_write.rb b/test/lib_read_write.rb
index 26e7aef..f5c5bf8 100644
--- a/test/lib_read_write.rb
+++ b/test/lib_read_write.rb
@@ -7,7 +7,12 @@ $-w = true
 require 'kgio'
 
 module LibReadWriteTest
-  RANDOM_BLOB = File.open("/dev/urandom") { |fp| fp.read(10 * 1024 * 1024) }
+  RANDOM_BLOB = File.open("/dev/urandom") do |fp|
+    nr = 31
+    buf = fp.read(nr)
+    # get roughly a 20MB block of random data
+    (buf * (20 * 1024 * 1024 / nr)) + (buf * rand(123))
+  end
 
   def teardown
     @rd.close if defined?(@rd) && ! @rd.closed?
-- 
EW

Re: Bug#830353: ruby-kgio: FTBFS: ERROR: Test "ruby2.3" failed.

[Antonio Terceiro]

[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]

Control: retitle -1 ruby-kgio: FTBFS on Amazon EC2: ERROR: Test "ruby2.3" failed.
Control: severity -1 important

On Fri, Jul 08, 2016 at 06:53:29PM +0000, Eric Wong wrote:
> Lucas Nussbaum <lucas@debian.org> wrote:
> > The full build log is available from:
> >    http://people.debian.org/~lucas/logs/2016/07/07/ruby-kgio_2.10.0-1_unstable_reb.normal.log
> > 
> > A list of current common problems and possible solutions is available at
> > http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!
> > 
> > About the archive rebuild: The rebuild was done on EC2 VM instances from
> > Amazon Web Services, using a clean, minimal and up-to-date chroot. Every
> > failed build was retried once to eliminate random failures.
> 
> Thanks for the report.  Can you try the patch below?

Didn't help.

I can't, however, reproduce this locally on my workstation.

I remember hitting this type of issue with kgio whenever I did rebuilds of all
Ruby packages on an Amazon EC2 instance, and then couldn't reproduce it
locally.

If you want to give a closer look I can give you access to the VM I use for
these builds, where I can consistently reproduce this.

> I'd be interested to know what non-standard sysctl knobs are set
> for kernel socket buffer sizes (in /proc/sys/net/{core,ipv4}/*mem*)

$ tail -n 1000 /proc/sys/net/{core,ipv4}/*mem*
==> /proc/sys/net/core/optmem_max <==
20480

==> /proc/sys/net/core/rmem_default <==
212992

==> /proc/sys/net/core/rmem_max <==
16777216

==> /proc/sys/net/core/wmem_default <==
212992

==> /proc/sys/net/core/wmem_max <==
16777216

==> /proc/sys/net/ipv4/igmp_max_memberships <==
20

==> /proc/sys/net/ipv4/tcp_mem <==
2954019 3938693 5908038

==> /proc/sys/net/ipv4/tcp_rmem <==
4096    12582912        16777216

==> /proc/sys/net/ipv4/tcp_wmem <==
4096    12582912        16777216

==> /proc/sys/net/ipv4/udp_mem <==
2954019 3938693 5908038

==> /proc/sys/net/ipv4/udp_rmem_min <==
4096

==> /proc/sys/net/ipv4/udp_wmem_min <==
4096

> You can also increase the 20M to 30M or whatever (otoh, that
> means using more memory during tests :<)
> 
> I'm not comfortable calling setsockopt to set SO_{SND,RCV}BUF
> knobs since that can trigger VM deadlock bugs on some older
> kernels, too.
> 
> Anyways, most of the kgio functionality is in Ruby 2.3 nowadays
> and kgio can probably go away when support for 2.2 can be
> dropped in gems.

In Debian we don't even have 2.2 anymore. The only things using kgio are
unicorn and rainbows. Maybe if their dependency on kgio could be skipped on
Ruby 2.3, we could drop kgio from Debian.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Bug#830353: ruby-kgio: FTBFS: ERROR: Test "ruby2.3" failed.

[Eric Wong]

Antonio Terceiro <terceiro@debian.org> wrote:
> On Fri, Jul 08, 2016 at 06:53:29PM +0000, Eric Wong wrote:
> > Thanks for the report.  Can you try the patch below?
> 
> Didn't help.
> 
> I can't, however, reproduce this locally on my workstation.

Thanks for following up.

> > I'd be interested to know what non-standard sysctl knobs are set
> > for kernel socket buffer sizes (in /proc/sys/net/{core,ipv4}/*mem*)
 
> ==> /proc/sys/net/ipv4/tcp_rmem <==
> 4096    12582912        16777216
> 
> ==> /proc/sys/net/ipv4/tcp_wmem <==
> 4096    12582912        16777216

Yikes at the gigantic default (middle) values.  Anyways, I've
set those locally and notice the problem with my previous patch:
I missed some spots where temporary strings/arrays were created
inside the tests themselves.

> In Debian we don't even have 2.2 anymore. The only things using kgio are
> unicorn and rainbows. Maybe if their dependency on kgio could be skipped on
> Ruby 2.3, we could drop kgio from Debian.

Yes, I still have to check and make sure I don't introduce
performance regressions and such when dropping kgio.
I still need to support Ruby 1.9.3 and 2.0.0 outside of Debian
(or on wheezy LTS/jessie)

The following ought to work (I tested locally with those sysctl
values) but I'll see about making the tests more robust against
larger kernel buffers.

-------8<--------
--- a/test/lib_read_write.rb
+++ b/test/lib_read_write.rb
@@ -7,7 +7,12 @@ $-w = true
 require 'kgio'
 
 module LibReadWriteTest
-  RANDOM_BLOB = File.open("/dev/urandom") { |fp| fp.read(10 * 1024 * 1024) }
+  RANDOM_BLOB = File.open("/dev/urandom") do |fp|
+    nr = 31
+    buf = fp.read(nr)
+    # get roughly a 20MB block of random data
+    (buf * (20 * 1024 * 1024 / nr)) + (buf * rand(123))
+  end
 
   def teardown
     @rd.close if defined?(@rd) && ! @rd.closed?
@@ -369,7 +374,7 @@ module LibReadWriteTest
       @nr += 1
       IO.select(nil, [self])
     end
-    buf = "." * 1024 * 1024 * 10
+    buf = RANDOM_BLOB
     thr = Thread.new { @wr.kgio_write(buf) }
     Thread.pass until thr.stop?
     readed = @rd.read(buf.size)
@@ -385,7 +390,7 @@ module LibReadWriteTest
       @nr += 1
       IO.select(nil, [self])
     end
-    buf = ["." * 1024] * 1024 * 10
+    buf = [ RANDOM_BLOB, RANDOM_BLOB ]
     buf_size = buf.inject(0){|c, s| c + s.size}
     thr = Thread.new { @wr.kgio_writev(buf) }
     Thread.pass until thr.stop?
-- 
EW