JRuby 9.0.4.0 + mogilefs-client

JRuby 9.0.4.0 + mogilefs-client

[伊藤洋也]

Hello.

I have implmemted S3 like API with mogilefs-client + Rails + CRuby,
and it works very well.
When I tried to run mogilefs-client with jruby experimentally, an
following exception was raised.

 * mogilefs-test.rb

```
#!/usr/bin/env ruby

require 'mogilefs'

mogfs = MogileFS::MogileFS.new(
  hosts:  ['127.0.0.1:7001'],
  domain: 'sandbox',
)
mogfs.sleep(1)
puts "ok"
```

````
$ uname -a
Linux vagrant-centos6 2.6.32-573.el6.x86_64 #1 SMP Thu Jul 23 15:44:03
UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ ruby -v
jruby 9.0.4.0 (2.2.2) 2015-11-12 b9fb7aa OpenJDK 64-Bit Server VM
25.65-b01 on 1.8.0_65-b17 +jit [linux-amd64]

$ $ echo '!version' | nc localhost 7001
2.72
.

$ ruby mogilefs-test.rb
MogileFS::UnreachableBackendError: couldn't connect to any tracker:
127.0.0.1:7001 - socket connect timeout (MogileFS::Timeout)
               socket at
/usr/local/rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:362
    dispatch_unlocked at
/usr/local/rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:143
  block in do_request at
/usr/local/rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:243
          synchronize at org/jruby/ext/thread/Mutex.java:151
           do_request at
/usr/local/rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:241
   block in get_paths at
/usr/local/rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:23
            get_paths at
/usr/local/rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/mogilefs.rb:222
                <top> at mogilefs-test.rb:10

# I have also tested on OSX
# Darwin PMAC132S.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul
29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64 i386
MacBookPro11,1 Darwin
#
# $ ruby -v
# jruby 9.0.4.0 (2.2.2) 2015-11-12 b9fb7aa Java HotSpot(TM) 64-Bit
Server VM 25.60-b23 on 1.8.0_60-b27 +jit [darwin-x86_64]
#
# MogileFS::UnreachableBackendError: couldn't connect to any tracker:
127.0.0.1:7001 - socket connect timeout (MogileFS::Timeout)
#                socket at
/Users/hiroya/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:362
#     dispatch_unlocked at
/Users/hiroya/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:143
#   block in do_request at
/Users/hiroya/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:243
#           synchronize at org/jruby/ext/thread/Mutex.java:151
#            do_request at
/Users/hiroya/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:241
#    block in get_paths at
/Users/hiroya/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/backend.rb:23
#             get_paths at
/Users/hiroya/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/mogilefs-client-3.9.0/lib/mogilefs/mogilefs.rb:222
#                 <top> at mogilefs-test.rb:9
#
````

When I use CRuby, it works properly.

```
$ ruby -v
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]

$ ruby mogilefs-test.rb
ok
```

And I investigated the execpetion and the same exception was
reproduced by following code.

 * socket-test.rb

```
#!/usr/bin/env ruby

require 'socket'
require 'io/wait'

sock = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM, 0)

begin
  sock.connect_nonblock(Socket.sockaddr_in(7001, '127.0.0.1'))
  sock.post_init(host, port)
rescue Errno::EINPROGRESS
end

unless sock.wait_writable(5)
  raise 'MogileFS::Timeout, socket connect timeout'
end
```

```
$ ruby socket-test.rb
RuntimeError: MogileFS::Timeout, socket connect timeout
  <top> at socket-test.rb:15
```

The same problem has been issued and discussed on GitHub jruby/jruby
https://github.com/jruby/jruby/issues/1694, but not been resolved yet.

It might not be appropriate to ask you about this problem since it
does not seem to be the problem of mogilfs-client but of jruby, but do
you have any ideas?

Regards.

----------------------------------------------
Hiroya Ito
 * hiroyan@gmail.com
 * https://github.com/hiboma/
 * GMO Pepabo, Inc https://pepabo.com
----------------------------------------------

Re: JRuby 9.0.4.0 + mogilefs-client

[Eric Wong]

伊藤洋也 <hiroyan@gmail.com> wrote:
> The same problem has been issued and discussed on GitHub jruby/jruby
> https://github.com/jruby/jruby/issues/1694, but not been resolved yet.
> 
> It might not be appropriate to ask you about this problem since it
> does not seem to be the problem of mogilfs-client but of jruby, but do
> you have any ideas?

I can help with nearly any userspace <-> kernel issues on GNU/Linux
and maybe FreeBSD.  However, I don't know C++ or Java at all,
so I'm not sure what your JVM or JRuby is doing...

Under Linux, it would be helpful to compare strace output between
C Ruby and JRuby with the test script below (and variations from
making adjustments where I left "XXX" comments).

Does retrying connect_nonblock on EINPROGRESS/EALREADY help?
In other words, something like this:

require 'socket'
require 'io/wait'

sock = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM, 0)
addr = Socket.sockaddr_in(7001, '127.0.0.1')
begin
  # XXX it would be worth trying sock.connect(addr) instead
  # and compare strace output between C Ruby and JRuby, too
  sock.connect_nonblock(addr)
rescue Errno::EINPROGRESS, Errno::EALREADY => e
  warn "waiting on #{e.class}"

  # XXX Maybe you need to comment out this line on JRuby?
  # Ideally you wouldn't need the timeout arg for wait_writable
  sock.wait_writable(0.001)

  retry
rescue Errno::EISCONN
  warn "Success!"
end

unless sock.wait_writable(5)
  raise 'MogileFS::Timeout, socket connect timeout'
end
----------------

And based on the comments on that ticket, it might be a problem
with the JVM or JRuby mishandling the issue...

Regardless of whether I can help, thank you for documenting the
issue publically.

Re: JRuby 9.0.4.0 + mogilefs-client

[Eric Wong]

Ping.  It's been a while.

Did you figure out this problem and try my suggestion?

For reference:

https://bogomips.org/mogilefs-client-public/20151202205343.GA5690@dcvr.yhbt.net/

Any answer (including "I've been too busy") is fine, thanks :)