* [ANN] sleepy_penguin 3.5.0 - Linux I/O events (and more) for Ruby
@ 2017-03-22 7:27 7% Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2017-03-22 7:27 UTC (permalink / raw)
To: ruby-talk, sleepy-penguin
sleepy_penguin provides access to newer, Linux-only system calls to wait
on events from traditionally non-I/O sources. Bindings to the eventfd,
timerfd, inotify, and epoll interfaces are provided. Experimental support
for kqueue on FreeBSD (and likely OpenBSD/NetBSD) is also provided.
* homepage: https://bogomips.org/sleepy_penguin/
* git clone git://bogomips.org/sleepy_penguin.git
* Atom feed: https://bogomips.org/sleepy_penguin/NEWS.atom.xml
* mailing list: sleepy-penguin@bogomips.org
* mail archives: https://bogomips.org/sleepy-penguin/
nntp://news.public-inbox.org/inbox.comp.lang.ruby.sleepy-penguin
https://bogomips.org/sleepy-penguin/new.atom
Changes:
sleepy_penguin 3.5.0
Most notably, kevent and epoll_wait wrappers are now
nestable, so you can write your own event loops inside
somebody elses event loop (not that it's a good idea,
but hey, that's reality, sometimes).
https://bogomips.org/sleepy_penguin/SleepyPenguin/Kqueue.html#method-i-kevent
https://bogomips.org/sleepy_penguin/SleepyPenguin/Epoll.html#method-i-wait
For Linux users, there is now copy_file_range(2) support
for copying regular files:
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#method-c-copy_file_range
There is also a new sendfile wrapper which emulates Linux
sendfile behavior regardless of platform. It will use the
native sendfile(2) syscall on FreeBSD and Linux, at least.
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#method-c-linux_sendfile
Wrappers for the splice(2) and tee(2) syscalls also exist for
Linux users (vmsplice(2) is omitted):
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#method-c-splice
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#method-c-tee
Along with some related constants:
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#F_GETPIPE_SZ
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#F_SETPIPE_SZ
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#F_MORE
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#F_MOVE
https://bogomips.org/sleepy_penguin/SleepyPenguin.html#F_NONBLOCK
In other words, this release merges the useful parts of the old
"io_splice" RubyGem: https://bogomips.org/ruby_io_splice/
Linux 4.5+ epoll users also get EPOLLEXCLUSIVE along existing
constants:
https://bogomips.org/sleepy_penguin/SleepyPenguin/Epoll.html#EXCLUSIVE
Ruby 1.8 and 1.9 support are both gone, Ruby 2.0.0+ is required
for keyword args, now(*).
31 changes since 3.4.1:
README: fix wording: are => is
TODO: add memfd item
epoll: allow :CLOEXEC instead of the long constant
note the epoll/io.rb file is only for Ruby 1.8
support the splice(2) and tee(2) syscalls
implement copy_file_range support for Linux 4.5+
doc: various URL updates (https)
unify rb_gc() handling for out-of-FD conditions
splice: clarification regarding tee() flags
pkg.mk: various updates from other projects
copy_file_range: use correct syscall numbers on x86/x86-64
new API for splice and tee
doc: remove references to IO#pipe_size accessor
remove PIPE_BUF constant definition
copy_file_range: move wrapper to Ruby for keyword arg handling
gemspec: use SPDX-compatible license specifier
implement linux_sendfile support
portability fixes for systems w/o splice, copy_file_range
tests: switch to test-unit
free buffer if pthread_setspecific fails
allow nestable TLS buffers within the same thread
drop unused Ruby 1.8 support code
epoll: add newline to Kernel#warn messages for atomicity
favor require_relative over require
epoll: add EPOLLEXCLUSIVE constant and documentation
kqueue: remove IO#autoclose= and 1.8-related checks
sp_copy: remove dummy 1.8 code for non-native threaded Ruby
build: remove build-time olddoc dependency
copy_file_range: add documentation
doc: tests and examples for changing pipe size in Linux
doc: avoid incorrect links to Epoll::IO
(*) Even ruby 2.1 is unsupported by ruby-core upstream;
but I could be prodded to re-add 1.9.3 support if
somebody really wants it...
^ permalink raw reply [relevance 7%]
* [PATCH 2/2] implement copy_file_range support for Linux 4.5+
2016-03-16 3:13 6% [sleepy_penguin PATCH 0/2] splice/tee/copy_file_range support Eric Wong
@ 2016-03-16 3:13 4% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2016-03-16 3:13 UTC (permalink / raw)
To: sleepy-penguin; +Cc: ruby-io-splice, Eric Wong
Under Linux 4.5, this allows for efficient copies and
is similar to the splice and sendfile system calls.
---
ext/sleepy_penguin/cfr.c | 68 +++++++++++++++++++++++++++++++++++++++++++
ext/sleepy_penguin/extconf.rb | 1 +
ext/sleepy_penguin/init.c | 2 ++
ext/sleepy_penguin/sp_copy.h | 50 +++++++++++++++++++++++++++++++
ext/sleepy_penguin/splice.c | 54 ++--------------------------------
test/test_cfr.rb | 29 ++++++++++++++++++
6 files changed, 153 insertions(+), 51 deletions(-)
create mode 100644 ext/sleepy_penguin/cfr.c
create mode 100644 ext/sleepy_penguin/sp_copy.h
create mode 100644 test/test_cfr.rb
diff --git a/ext/sleepy_penguin/cfr.c b/ext/sleepy_penguin/cfr.c
new file mode 100644
index 0000000..ea17f82
--- /dev/null
+++ b/ext/sleepy_penguin/cfr.c
@@ -0,0 +1,68 @@
+#include "sleepy_penguin.h"
+#include "sp_copy.h"
+#include <unistd.h>
+
+#ifndef HAVE_COPY_FILE_RANGE
+# include <sys/syscall.h>
+# if !defined(__NR_copy_file_range) && \
+ (defined(__x86_64__) || defined(__i386__))
+# define __NR_copy_file_range 285
+# endif /* __NR_copy_file_range */
+#endif
+
+#ifdef __NR_copy_file_range
+static ssize_t my_cfr(int fd_in, off_t *off_in, int fd_out, off_t *off_out,
+ size_t len, unsigned int flags)
+{
+ long n = syscall(__NR_copy_file_range,
+ fd_in, off_in, fd_out, off_out, len, flags);
+
+ return (ssize_t)n;
+}
+# define copy_file_range(fd_in,off_in,fd_out,off_out,len,flags) \
+ my_cfr((fd_in),(off_in),(fd_out),(off_out),(len),(flags))
+#endif
+
+static void *nogvl_cfr(void *ptr)
+{
+ struct copy_args *a = ptr;
+
+ return (void *)copy_file_range(a->fd_in, a->off_in,
+ a->fd_out, a->off_out, a->len, a->flags);
+}
+
+static VALUE rb_cfr(int argc, VALUE *argv, VALUE mod)
+{
+ off_t i, o;
+ VALUE io_in, off_in, io_out, off_out, len, flags;
+ ssize_t bytes;
+ struct copy_args a;
+
+ rb_scan_args(argc, argv, "51",
+ &io_in, &off_in, &io_out, &off_out, &len, &flags);
+
+ a.off_in = NIL_P(off_in) ? NULL : (i = NUM2OFFT(off_in), &i);
+ a.off_out = NIL_P(off_out) ? NULL : (o = NUM2OFFT(off_out), &o);
+ a.len = NUM2SIZET(len);
+ a.flags = NIL_P(flags) ? 0 : NUM2UINT(flags);
+
+again:
+ a.fd_in = rb_sp_fileno(io_in);
+ a.fd_out = rb_sp_fileno(io_out);
+ bytes = (ssize_t)IO_RUN(nogvl_cfr, &a);
+ if (bytes < 0) {
+ if (errno == EINTR)
+ goto again;
+ rb_sys_fail("copy_file_range");
+ } else if (bytes == 0) {
+ rb_eof_error();
+ }
+ return SSIZET2NUM(bytes);
+}
+
+void sleepy_penguin_init_cfr(void)
+{
+ VALUE mod = rb_define_module("SleepyPenguin");
+
+ rb_define_singleton_method(mod, "copy_file_range", rb_cfr, -1);
+}
diff --git a/ext/sleepy_penguin/extconf.rb b/ext/sleepy_penguin/extconf.rb
index 46d1059..53a2810 100644
--- a/ext/sleepy_penguin/extconf.rb
+++ b/ext/sleepy_penguin/extconf.rb
@@ -19,6 +19,7 @@
end
have_type('clockid_t', 'time.h')
have_func('clock_gettime', 'time.h')
+have_func('copy_file_range', 'unistd.h')
have_func('epoll_create1', %w(sys/epoll.h))
have_func('inotify_init1', %w(sys/inotify.h))
have_func('splice', %w(fcntl.h))
diff --git a/ext/sleepy_penguin/init.c b/ext/sleepy_penguin/init.c
index 93e8092..0a1458b 100644
--- a/ext/sleepy_penguin/init.c
+++ b/ext/sleepy_penguin/init.c
@@ -53,6 +53,7 @@ void sleepy_penguin_init_signalfd(void);
#endif
void sleepy_penguin_init_splice(void);
+void sleepy_penguin_init_cfr(void);
static size_t l1_cache_line_size_detect(void)
{
@@ -130,4 +131,5 @@ void Init_sleepy_penguin_ext(void)
sleepy_penguin_init_inotify();
sleepy_penguin_init_signalfd();
sleepy_penguin_init_splice();
+ sleepy_penguin_init_cfr();
}
diff --git a/ext/sleepy_penguin/sp_copy.h b/ext/sleepy_penguin/sp_copy.h
new file mode 100644
index 0000000..83b9554
--- /dev/null
+++ b/ext/sleepy_penguin/sp_copy.h
@@ -0,0 +1,50 @@
+/* common splice and copy_file_range-related definitions */
+
+#ifndef SSIZET2NUM
+# define SSIZET2NUM(x) LONG2NUM(x)
+#endif
+#ifndef NUM2SIZET
+# define NUM2SIZET(x) NUM2ULONG(x)
+#endif
+
+#if defined(HAVE_RB_THREAD_CALL_WITHOUT_GVL) && defined(HAVE_RUBY_THREAD_H)
+/* Ruby 2.0+ */
+# include <ruby/thread.h>
+# define WITHOUT_GVL(fn,a,ubf,b) \
+ rb_thread_call_without_gvl((fn),(a),(ubf),(b))
+#elif defined(HAVE_RB_THREAD_BLOCKING_REGION)
+typedef VALUE (*my_blocking_fn_t)(void*);
+# define WITHOUT_GVL(fn,a,ubf,b) \
+ rb_thread_blocking_region((my_blocking_fn_t)(fn),(a),(ubf),(b))
+
+#else /* Ruby 1.8 */
+/* partial emulation of the 1.9 rb_thread_blocking_region under 1.8 */
+# include <rubysig.h>
+# define RUBY_UBF_IO ((rb_unblock_function_t *)-1)
+typedef void rb_unblock_function_t(void *);
+typedef void * rb_blocking_function_t(void *);
+static void * WITHOUT_GVL(rb_blocking_function_t *func, void *data1,
+ rb_unblock_function_t *ubf, void *data2)
+{
+ void *rv;
+
+ assert(RUBY_UBF_IO == ubf && "RUBY_UBF_IO required for emulation");
+
+ TRAP_BEG;
+ rv = func(data1);
+ TRAP_END;
+
+ return rv;
+}
+#endif /* ! HAVE_RB_THREAD_BLOCKING_REGION */
+
+#define IO_RUN(fn,data) WITHOUT_GVL((fn),(data),RUBY_UBF_IO,0)
+
+struct copy_args {
+ int fd_in;
+ int fd_out;
+ off_t *off_in;
+ off_t *off_out;
+ size_t len;
+ unsigned flags;
+};
diff --git a/ext/sleepy_penguin/splice.c b/ext/sleepy_penguin/splice.c
index d2f9206..2f901a8 100644
--- a/ext/sleepy_penguin/splice.c
+++ b/ext/sleepy_penguin/splice.c
@@ -1,4 +1,5 @@
#include "sleepy_penguin.h"
+#include "sp_copy.h"
#ifdef HAVE_SPLICE
#include <errno.h>
#include <fcntl.h>
@@ -18,13 +19,6 @@ static VALUE sym_EAGAIN;
# define F_GETPIPE_SZ (F_LINUX_SPECIFIC_BASE + 8)
#endif
-#ifndef SSIZET2NUM
-# define SSIZET2NUM(x) LONG2NUM(x)
-#endif
-#ifndef NUM2SIZET
-# define NUM2SIZET(x) NUM2ULONG(x)
-#endif
-
static int check_fileno(VALUE io)
{
int saved_errno = errno;
@@ -33,51 +27,9 @@ static int check_fileno(VALUE io)
return fd;
}
-#if defined(HAVE_RB_THREAD_CALL_WITHOUT_GVL) && defined(HAVE_RUBY_THREAD_H)
-/* Ruby 2.0+ */
-# include <ruby/thread.h>
-# define WITHOUT_GVL(fn,a,ubf,b) \
- rb_thread_call_without_gvl((fn),(a),(ubf),(b))
-#elif defined(HAVE_RB_THREAD_BLOCKING_REGION)
-typedef VALUE (*my_blocking_fn_t)(void*);
-# define WITHOUT_GVL(fn,a,ubf,b) \
- rb_thread_blocking_region((my_blocking_fn_t)(fn),(a),(ubf),(b))
-
-#else /* Ruby 1.8 */
-/* partial emulation of the 1.9 rb_thread_blocking_region under 1.8 */
-# include <rubysig.h>
-# define RUBY_UBF_IO ((rb_unblock_function_t *)-1)
-typedef void rb_unblock_function_t(void *);
-typedef void * rb_blocking_function_t(void *);
-static void * WITHOUT_GVL(rb_blocking_function_t *func, void *data1,
- rb_unblock_function_t *ubf, void *data2)
-{
- void *rv;
-
- assert(RUBY_UBF_IO == ubf && "RUBY_UBF_IO required for emulation");
-
- TRAP_BEG;
- rv = func(data1);
- TRAP_END;
-
- return rv;
-}
-#endif /* ! HAVE_RB_THREAD_BLOCKING_REGION */
-
-#define IO_RUN(fn,data) WITHOUT_GVL((fn),(data),RUBY_UBF_IO,0)
-
-struct splice_args {
- int fd_in;
- int fd_out;
- off_t *off_in;
- off_t *off_out;
- size_t len;
- unsigned flags;
-};
-
static void *nogvl_splice(void *ptr)
{
- struct splice_args *a = ptr;
+ struct copy_args *a = ptr;
return (void *)splice(a->fd_in, a->off_in, a->fd_out, a->off_out,
a->len, a->flags);
@@ -87,7 +39,7 @@ static ssize_t do_splice(int argc, VALUE *argv, unsigned dflags)
{
off_t i = 0, o = 0;
VALUE io_in, off_in, io_out, off_out, len, flags;
- struct splice_args a;
+ struct copy_args a;
ssize_t bytes;
ssize_t total = 0;
diff --git a/test/test_cfr.rb b/test/test_cfr.rb
new file mode 100644
index 0000000..3483c5a
--- /dev/null
+++ b/test/test_cfr.rb
@@ -0,0 +1,29 @@
+# -*- encoding: binary -*-
+require 'test/unit'
+require 'tempfile'
+$-w = true
+require 'sleepy_penguin'
+
+class TestCfr < Test::Unit::TestCase
+ def test_copy_file_range
+ str = 'abcde'
+ size = 5
+ src = Tempfile.new('ruby_cfr_src')
+ dst = Tempfile.new('ruby_cfr_dst')
+ assert_equal 5, src.syswrite(str)
+ src.sysseek(0)
+ begin
+ nr = SleepyPenguin.copy_file_range(src, nil, dst, nil, size, 0)
+ rescue Errno::EINVAL
+ warn 'copy_file_range not supported (requires Linux 4.5+)'
+ warn "We have: #{`uname -a`}"
+ return
+ end
+ assert_equal nr, 5
+ dst.sysseek(0)
+ assert_equal str, dst.sysread(5)
+ ensure
+ dst.close!
+ src.close!
+ end
+end if SleepyPenguin.respond_to?(:copy_file_range)
--
EW
^ permalink raw reply related [relevance 4%]
* [sleepy_penguin PATCH 0/2] splice/tee/copy_file_range support
@ 2016-03-16 3:13 6% Eric Wong
2016-03-16 3:13 4% ` [PATCH 2/2] implement copy_file_range support for Linux 4.5+ Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2016-03-16 3:13 UTC (permalink / raw)
To: sleepy-penguin; +Cc: ruby-io-splice
Linux 4.5 was just released the other day with
copy_file_range(2) support. This is similar to splice(2), but
meant to operate on regular files rather than pipes.
copy_file_range(2) does not have glibc support, yet,
so it's only supported on x86-64 and x86, for now.
Blatantly stealing code from the "io_splice" gem, but omitting
vmsplice support for now since I never figured out a good way to
use it in Ruby without a mmap(2) wrapper...
So with that, I'm planning to slowly phase out the "io_splice"
gem since this one absorbs its functionality. And I will add
vmsplice support here if we end up supporting mmap, somehow.
I also plan to support sendfile as that's another related
syscall to these, but may also support a "bsd_sendfile"
with support for writing header/trailer vectors...
Eric Wong (2):
support the splice(2) and tee(2) syscalls
implement copy_file_range support for Linux 4.5+
.document | 1 +
ext/sleepy_penguin/cfr.c | 68 +++++++++
ext/sleepy_penguin/extconf.rb | 5 +
ext/sleepy_penguin/init.c | 5 +
ext/sleepy_penguin/sp_copy.h | 50 +++++++
ext/sleepy_penguin/splice.c | 328 ++++++++++++++++++++++++++++++++++++++++++
ext/sleepy_penguin/util.c | 3 +
test/test_cfr.rb | 29 ++++
test/test_splice.rb | 254 ++++++++++++++++++++++++++++++++
test/test_splice_eintr.rb | 34 +++++
10 files changed, 777 insertions(+)
^ permalink raw reply [relevance 6%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2016-03-16 3:13 6% [sleepy_penguin PATCH 0/2] splice/tee/copy_file_range support Eric Wong
2016-03-16 3:13 4% ` [PATCH 2/2] implement copy_file_range support for Linux 4.5+ Eric Wong
2017-03-22 7:27 7% [ANN] sleepy_penguin 3.5.0 - Linux I/O events (and more) for Ruby Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/sleepy_penguin.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).