Date | Commit message (Collapse) |
|
Covering my ass from draconian legislation.
|
|
Instead of a malloc where arguments can silently overflow,
use calloc directly to handle the overflow and zeroing for
us.
|
|
We should warn gracefully when we hit IPv7+ or whatever...
|
|
While no user is likely to have enough listeners to trigger an
overflow, just use ruby_xrealloc2 to be safe since it's already
provided by Ruby (and AFAIK reallocarray(3) isn't standardized).
|
|
Given the history of Ruby removing public C APIs, get ahead
of potential incompatibilities by switching to an externally
maintained unordered hash table.
khashl is a newer, more memory-efficient evolution of the
khash hash table adopted by the git SCM and this will be
my first (and likely not last) time using it in a public
codebase.
|
|
OBJ_FREEZE before calling rb_hash_aset was actually preventing
deduplication since Ruby 2.6. This introduces a performance
regression for Ruby 2.5 users, but I expect the majority of
users are on newer versions (I'm on 2.7, which is still ancient).
|
|
Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1]. In order to avoid
incompatibilities in the future, I used the following Perl
script to prepend `frozen_string_literal: false' to every
Ruby file:
use v5.12;
use autodie;
my $usage = 'perl /path/to/script <LIST_OF_RB_FILES>';
my $fsl = "# frozen_string_literal: false\n";
for my $f (@ARGV) {
open my $fh, '<', $f;
my $s = do { local $/; <$fh> } // die "read($f): $!";
next if $s =~ /^#\s*frozen_string_literal:/sm;
# fsl must be after encoding: line if it exists:
if ($s =~ s/^([ \t]*\#[ \t\-\*\#]+encoding:[^\n]+\n)/$1$fsl/sm
# or after the shebang
|| $s =~ s/^(#![^\n]+\n)/$1$fsl/
# or after embedded switches in rackup files:
|| ($f =~ /\.ru$/ &&
$s =~ s/^(#\\[^\n]+\n)/$1$fsl/)
# or prepend as a last resort:
|| (substr($s, 0, 0) = $fsl)) {
open $fh, '>', $f;
print $fh $s;
close $fh;
}
}
Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.
[1] https://bugs.ruby-lang.org/issues/20205
|
|
Hopefully this doesn't cause more breakage and we'll never
actually need a rack 4
|
|
Compiler optimization isn't useful when doing portability checks
for any of the things we care about.
|
|
posix_mq is rarely installed, so don't force users to
have it in order to test or develop raindrops.
|
|
Diskspace and bandwidth are expensive, and we can make
rack+aggregate optional in tests, too.
|
|
Eric Wong <bofh@yhbt.net> wrote:
> I'll squash this in for fork+preload safety.
I forget there are multithreaded servers using this :x
------8<-----
Subject: [PATCH v3] middleware: reuse inet_diag netlink socket
No point in constantly allocating and deallocating FDs (and Ruby
IO objects) when reusing them is supported.
However, we must guard it against parallel callers since
linux_inet_diag.c::diag releases the GVL while calling
sendmsg(2) and recvmsg(2), so it's possible two Ruby threads can
end up crossing streams if the middlware is used with a
multi-threaded server.
AFAIK, Raindrops::Middleware isn't a heavily-trafficked endpoint,
so saving FDs and avoiding thread specific data is preferable
for memory-constrained systems I use.
|
|
I just debugged an issue with our system, I was witnessing the
number of file descriptor in our process grow at an alarming rate
which I mapped to our use of raindrops to report utilisation.
For various reasons we don’t call raindrops from a Rack middleware
but have one process that monitor the socket continuously, and
share that data with the workers.
Since we call tcp_listener_stats every seconds in a process
that doesn't do much else, GC very rarely triggers if at all
which cause `InetDiagSocket` instances to accumulate very
quickly.
Each of those instances holds a file descriptor.
Looking at the raindrops implementation it seems to assume
the GC will take care of regularly closing these sockets, but
I think it’s a bit too bold of an assumption.
[ew: don't close user-passed sockets on exception]
Acked-by: Eric Wong <e@80x24.org>
|
|
Socket#accept and Socket#accept_nonblock return an Addrinfo
object in addition to a client socket. This allows web servers
to avoid having to make getpeername(2) syscalls to get the
same information.
|
|
We don't list `unicorn' as a development dependency since
unicorn currently depends on this project. While unicorn might
drop us as a dependency, don't waste disk space and bandwidth
of potential raindrops hackers who don't have unicorn.
|
|
alloca makes stack usage unpredictable and life difficult for
static analysis tools and compilers. The 46 bytes of
INET6_ADDRSTRLEN is fine to keep on stack, but page size can be
several MB large in some architectures (but typically 4K on
common architectures).
Thus we handle page size-ed allocations via `rb_str_tmp_new'.
`rb_str_tmp_new' has been in public Ruby headers since the 1.9
days and used by the core `zlib', `digest', and `zlib'
extensions, so it should be safe to use (and `rb_str_resize' is
used in many more C extensions).
|
|
This is less code and hopefully smaller binaries.
`rb_io_check_closed' has been in Ruby since the pre-CVS of
decades ago, and it doesn't matter if it's removed or not
in the future since Ruby 3.1+ doesn't see this code path
and calls `rb_io_descriptor' directly.
|
|
Calling `#to_io' is only necessary when we're handling an argument
from user code where the user could pass a non-IO object.
`#to_io' calls are a waste of time when we create the IO object
ourselves (in `Raindrops::InetDiagSock.new').
This allows us to define the `my_fileno' macro for Ruby 3.1+
users to call the new `rb_io_descriptor' function directly
without an extra C stack frame. This also allows us to get
rid of nesting CPP directives inside C functions which (IMHO)
improves readability.
Furthermore, any necessary #to_io calls using `rb_convert_type'
can be replaced with `rb_io_get_io' to decrease code size.
`rb_io_get_io' has been in ruby/io.h since Ruby 1.9.2 and
there's no expectation that it'd be deprecated since it only
deals with opaque `VALUE' types.
|
|
Prefer to use `rb_io_descriptor` in my_fileno if it is available. This
function was introduced in Ruby 3.1. Ruby 3.3 will deprecate access to
all internal fields of `rb_io_t`.
This change is necessary for compatibility with the current
ruby.git master:
https://bugs.ruby-lang.org/issues/19057#note-17
|
|
This project has been 1.9.3+ only for a long time (and likely
going 2.x+-only), so kill some code we don't need.
|
|
Stats for SO_REUSEPORT sockets are now handled properly.
Thanks to Dale Hamel for the patches.
Dale Hamel (2):
Fix queue stats for sockets with SO_REUSEPORT
Fix off by one error in test
|
|
It doesn't affect the test outcome, but it is not the intended behaviour of
the test.
We listen once to find out what port to use, then should bind with SO_REUSEPORT
N-1 times, but we are doing it N times, so we have one extra listener.
|
|
SO_REUSEPORT was introduced in 2013, see https://lwn.net/Articles/542629/.
However, the current logic for calculating socket queue backlog stats
is from 2011, predating the advent of SO_REUSEPORT.
The current strategy was thus written before the notion that the mapping of
INET_ADDR:socket could potentially be 1:N instead of always 1:1.
This causes raindrops to provide invalid socket backlog values when
SO_REUSEPORT is used, which Unicorn supports since 2013. The current behaviour
will set the queue metric to the queue depth of the last inet_diag_msg
it processes matching the INET_ADDR. In practice, this will result in
the backlog being off by a factor N, assuming relatively even
distribution of requests to sockets, which the kernel should guarantee
through consistent hashing.
The fix here is to accumulate the socket queue depth as we iterate
over the the socket diagnostics, rather than reset it each time.
I have a provided a test, but it only checks the queues, not the
accept metrics, as those are not affected by this bug, and it is
not possible to know which of the listeners will be dispatched the
request by the kernel, and thus which should call accept.
|
|
Raindrops may now use file-backed mmap() rather than anonymous
memory. Thanks to KJ Tsanaktsidis for the patch:
https://yhbt.net/raindrops-public/20211125065618.3432-1-ktsanaktsidis@zendesk.com/T/
The documentation is also updated to note our mail archives are now
available via IMAP(S).
|
|
Our server on yhbt.net runs an IMAP(S) which is accessible over
Tor, too.
We no longer advertise a "mailing list" since the subscriber
info is centralized and not easily reproducible by third-parties.
Thus we focus on making our archives easily mirror-able via
IMAP(S), NNTP(S) and HTTP(S).
|
|
Currently, all memory used by Raindrops is mapped as MAP_ANONYMOUS. This
means that although Raindrops counters can be shared between processes
that have forked from each other, it is not possible to share the
counter values with another, unrelated process.
This patch adds support for backing the Raindrops mapping with a file
descriptor obtained from an IO object. The #initialize API has been
enhanced with two new keyword options:
Raindrops.new(size, io: nil, zero: false)
If an instance of IO is provided, then the underlying file descriptor
for that IO will be used to back the memory mapping Raindrops creates.
An unrelated process can then open the same file, and read the counters;
either by mmap'ing the file itself (or using Raindrops to do so), or by
making ordinary seek()/read() calls if performance is not a concern.
Note that the provided IO object _must_ implement #truncate; this is
used to set the size of the file to be right-sized for the memory
mapping that is created.
If the zero argument is passed as true, then the mapping will be zero'd
by Raindrops as part of its initialization. If it's false, then the
Raindrops counters existing in the file will be preserved. This allows
counter values to be persisted (although note that Raindrops makes no
attempt to msync the values, so they are not durable to e.g. system
crashes).
Counter values can easily be shared between processes
in-memory only without touching the disk by passing in a File on a
tmpfs as the io object.
|
|
This release fixes compatibility with GC.compact on Ruby 3.x
when using ListenStats on Linux. The listener stats
functionality is rarely used and does not affect most users
who just have raindrops installed for shared atomic counters.
|
|
index.html and README.html are identical, so get rid of
it. created.rid has no business being public, anyways.
Our homepage is just a directory listing, nowadays,
to improve discoverability of non-HTML docs.
|
|
Tor is dropping v2 .onion in favor of more secure (but less
readable) v3 .onion URLs. Zooko's triangle once again :/
(no, we won't support planet-destroying proof-of-work schemes)
|
|
With GC.compact in Ruby 3.x, Ruby-defined constants need to be
explicitly marked to prevent movement.
Link: https://yhbt.net/kgio-public/CAAvYYt5Z5f2rMuXO5DMpR1-6uRvu_gXKDvqcyoZ+oNcLiTH39g@mail.gmail.com/T/
|
|
This release fixes some warnings on newer Rubies.
We're no longer on bogomips.org since it's due
for expiry and I can't pay extortionists for a .org, so
s/bogomips.org/yhbt.net/ for now, and be prepared to move again
when extortionists move onto extorting the .net TLD.
doc: switch homepage to dark216
ext/raindrops/extconf: fix cflags reset on ancient CC
fixes for newer rubies
replace bogomips.org with yhbt.net
|
|
The expiration for bogomips.org is coming up and I'm not
keen on paying or supporting extortionists.
Not wanting to be beholden to ICANN or any powerful
organizations, .onion sites are available to Tor users:
http://raindrops.ou63pmih66umazou.onion/
http://ou63pmih66umazou.onion/raindrops.git/
http://ou63pmih66umazou.onion/raindrops-public/
(the demo is not yet available via .onion, yet, could be a bit)
|
|
Newer rubies have more warnings
|
|
Oops, but I guess nobody uses i386 anymore...
|
|
Earth Day!
https://80x24.org/olddoc-public/20190422014906.4253-1-e@80x24.org/
|
|
This release fixes Rack 2.x compatibility for the few users of
Raindrops::Middleware
<https://bogomips.org/raindrops/Raindrops/Middleware.html>.
Thanks to Dmytro Shteflyuk for this release.
No need to upgrade unless you use Raindrops::Middleware with
Rack 2.x.
There's also a few minor, inconsequential cleanups.
Dmytro Shteflyuk (1):
Properly override respond_to? in Raindrops::Middleware::Proxy
Eric Wong (2):
Ruby thread compatibility updates
tcp_info: remove unnecessary extconf.h include
|
|
Correct method definition according to Ruby documentation (https://ruby-doc.org/core-2.4.1/Object.html#method-i-respond_to-3F) is:
respond_to?(string, include_all=false) → true or false
Rack started using second argument starting from version 2:
https://github.com/rack/rack/blob/master/lib/rack/body_proxy.rb#L14
If raindrops is used in Rack 2+ applications, an exception is raised:
ArgumentError: wrong number of arguments (2 for 1)
<ROOT>/gems/raindrops-0.18.0/lib/raindrops/middleware/proxy.rb:30:in `respond_to?'
<ROOT>/gems/rack-2.0.3/lib/rack/body_proxy.rb:14:in `respond_to?'
|
|
ruby.h already pulls in extconf.h and has done so since Ruby 1.8.
|
|
Drop vestigial Ruby 1.8 bits, and start using
rb_thread_call_without_gvl on modern Rubies.
|
|
The most notable feature of this release is the addition of
FreeBSD and OpenBSD TCP_INFO support. This includes the
Raindrops::TCP for portably mapping TCP state names to
platform-dependent numeric values:
https://bogomips.org/raindrops/Raindrops.html#TCP
Thanks to Jeremy Evans and Simon Eskildsen on the
unicorn-public@bogomips.org mailing list for inspiring
these changes to raindrops.
There's also a few internal cleanups, and documentation
improvements, including some fixes to the largely-forgotten
Raindrops::Aggreage::PMQ class:
https://bogomips.org/raindrops/Raindrops/Aggregate/PMQ.html
20 changes since 0.17.0:
test_inet_diag_socket: fix Fixnum deprecation warning
TODO: add item for IPv6 breakage
ext: fix documentation for C ext-defined classes
TCP_Info: custom documentation for #get!
TypedData C-API conversion
test_watcher: disable test correctly when aggregate is missing
tcp_info: support this struct under FreeBSD
define Raindrops::TCP hash for TCP states
linux_inet_diag: reduce stack usage and simplify
avoid reading errno repeatedly
aggregate/pmq: avoid false sharing of lock buffers
aggregate/pmq: remove io-extra requirement
aggregate/pmq: avoid File#stat allocation
Merge remote-tracking branch 'origin/freebsd'
Merge remote-tracking branch 'origin/aggregate-pmq'
doc: remove private email support address
doc: update location of TCP_INFO-related stuff
build: avoid olddoc for building the RubyGem
doc: document Raindrops::TCP hash
aggregate/pmq: update version numbers for Ruby and Linux
|
|
Linux is in the 4.x and Ruby is into the 2.x range, by now,
so try to get with the times.
|
|
This is part of the stable API, so we shall document it for
others to use.
|
|
This should make it easier for distros to pick this
without picking up olddoc, too.
We will still use olddoc for generating the website since it
reduces bandwidth costs, and for generating NEWS for our own
builds.
|
|
|
|
Email was never private, and won't further burden myself or
any future maintainers with trying to maintain someone elses'
privacy.
Offering private support is also unfair to readers on public
lists who may get a watered down or improperly translated
summary (if at all).
Instead, encourage the use of anonymity tools and scrubbing of
sensitive information when the sender deems necessary.
|
|
* origin/aggregate-pmq:
aggregate/pmq: avoid File#stat allocation
aggregate/pmq: remove io-extra requirement
aggregate/pmq: avoid false sharing of lock buffers
|
|
* origin/freebsd:
define Raindrops::TCP hash for TCP states
tcp_info: support this struct under FreeBSD
|
|
File#size is available in modern Rubies so the extra syscall
is avoided.
|
|
IO.copy_stream is standard in 1.9+ and can use pread when
given an offset. We do not need to use pwrite with fcntl
locking, actually.
|
|
And rely on frozen string optimizations in Ruby while we're at it.
|