[ANN] cmogstored 1.5.0rc1 - a mogstored alternative

[ANN] cmogstored 1.5.0rc1 - a mogstored alternative

[Eric Wong]

cmogstored is an alternative implementation of the "mogstored" storage
component of MogileFS.  cmogstored is implemented in C and does not use
Perl at runtime.  cmogstored is the only component you need to install
on a MogileFS storage node.

Changes since 1.4.3:

    A bunch of minor changes; most notable is systemd-style socket
    activation support.  This was easy-to-add since we've always had
    socket activation support for nginx-style SIGUSR2 upgrades.

    This places no link or runtime dependency on libsystemd, so the
    LISTEN_FDS and LISTEN_PID environment variables may be used in other
    init systems as well.  While I have my own reservations about
    systemd itself, I also strongly believe in using socket activation
    to prevent downtime.

    Behavior changes:

    Bad Range: headers return 416 responses in more cases for invalid
    ranges (e.g. miscalculated ranges such as "1--1", while
    completely wrong ones (lacking a "bytes=" prefix)) are ignored
    entirely as in nginx.

    Bugfixes:

    There are also some cleanups to avoid dying on OOM in more places
    on weird systems which trigger OOM.  More work on this is ongoing.

    Also updates to the latest gnulib.git
    commit f197c2c9e5e0d12c373f26d5b3211809457bc972
    ("intprops: new public macro EXPR_SIGNED")
    along with a change which fixes a memory leak when people
    build from cmogstored.git using gnulib
    commit c6148bca89e9465fd6ba3a10d273ec4cb58c2dbe
    or later ("mountlist: add me_mntroot field on Linux machines").
    This memory leak did not affect any released tarballs of cmogstored.

    shortlog of changes since 1.4.3:

          doc: use "builder" RubyGem to generate Atom feed
          dev.c: fail gracefully on out-of-memory errors
          do not die on OOM when for mgmt paths
          HACKING: update URLs to reduce redirects
          http: return 416 errors in more cases for bad Ranges
          update .gitignores for latest autotools + gnulib
          Rakefile: remove text-only part from the Atom feed
          support systemd-style socket activation via environment
          set TCP listener options on inherited sockets
          doc: add example systemd config files
          use free_mount_entry from gnulib instead of rolling our own
          fix tmpdir dependency for slow Ruby tests
          doc: publish examples directory to website

Expect 1.5.0 final in a week or so (unless I forget :x)

http://bogomips.org/cmogstored/files/pre/cmogstored-1.5.0rc1.tar.gz
SHA-1   1b301ce5f180f9001275e26fa1c065593cc3569c
SHA-256 f99a9d4186c0b267952ec4806e6ccd333ab4441fe36e41417e4a09ccbd0c3e1e

* homepage: http://bogomips.org/cmogstored/README
* git clone git://bogomips.org/cmogstored.git
* cgit: http://bogomips.org/cmogstored.git
* gitweb: http://repo.or.cz/w/cmogstored.git
* list: cmogstored-public@bogomips.org (subscription optional)
* archives: http://bogomips.org/cmogstored-public/
* nntp://news.public-inbox.org/inbox.comp.file-systems.mogilefs.cmogstored

-- 
EW

Re: [ANN] cmogstored 1.5.0rc1 - a mogstored alternative

[Eric Wong]

David Birdsong <david.birdsong -AT- gmail -DOT- com> wrote:
> I love the LISTEN_FDS option. Would it be possible to specify the FD
> integer as a command line parameter?

Possible, yes; but I'm not a fan of having multiple ways to accomplish
the same thing.  I'd also want to avoid deviating too much from the Perl
mogstored command-line.

Unless somebody beats me to it, I also plan on implementing
LISTEN_FDS/LISTEN_PID support in mogilefsd and Perlbal/mogstored

I already did it in Perl a few weeks ago:

http://public-inbox.org/meta/20150920044320.2338-2-e%4080x24.org/t/

I consider the inherit() function too trivial to implement
 any other way and can be public domain / CC0.

And also wrote about it in ruby-talk:

http://mid.gmane.org/20151030-inherit-fds-w-o-sd_listen_fds-in-pure@ruby

> circus from mozilla is a more modern version of supervisor and it offers
> socket management but provides the socket through a templated command line.

It can't set environment variables?  I suppose you can use env(1) from
the command-line to launch cmogstored, as it always supported the
CMOGSTORED_FD environment.  Unlike LISTEN_FDS, CMOGSTORED_FD
does not validate $$ == LISTEN_PID like LISTEN_FDS does.  It
is comma-delimited, so it can use non-contiguous FDs:

   env CMOGSTORED_FD=3,4 cmogstored ...

[PATCH (cmogstored)] doc: document CMOGSTORED_FDS in the manpage

[Eric Wong]

David Birdsong <david.birdsong -AT- gmail -DOT- com> wrote:
> On Wed, Nov 11, 2015 at 2:41 PM, Eric Wong <e@80x24.org> wrote:
> > It can't set environment variables?  I suppose you can use env(1) from
> > the command-line to launch cmogstored, as it always supported the
> > CMOGSTORED_FD environment.  Unlike LISTEN_FDS, CMOGSTORED_FD
> > does not validate $$ == LISTEN_PID like LISTEN_FDS does.  It
> > is comma-delimited, so it can use non-contiguous FDs:
> >
> >    env CMOGSTORED_FD=3,4 cmogstored ...
> >
> >
> it can set env, but i was thinking that the templating from the socket
> support doesn't extend into env. either way, a simple shell shm could do
> it, so yeah that's probably fine.

The env(1) command-line would be templated, actually::

	env CMOGSTORED_FD=%d,%d cmogstored ...

Just pushed the following out to cmogstored.git, too.  Thanks.

Subject: [PATCH] doc: document CMOGSTORED_FDS in the manpage

This has always been supported internally, and we can't stop
supporting it since we'll be supporting upgrades from old versions
indefinitely.  So document it, as it has some minor advantages over
the LISTEN_{FDS,PID} environment handling of systemd.
---
 cmogstored.x | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/cmogstored.x b/cmogstored.x
index a6b96e6..20eeb36 100644
--- a/cmogstored.x
+++ b/cmogstored.x
@@ -22,6 +22,12 @@ different sockets, the command line or config file must still specify
 which addresses to listen on.  This feature is available in cmogstored
 1.5.0 and later.
 
+CMOGSTORED_FD - a comma-delimited list of integer file descriptors.
+This is used internally for SIGUSR2 upgrades, but may be specified
+at startup (e.g. via env(1)) as it is not tied to LISTEN_PID
+like LISTEN_FDS is.  Unlike LISTEN_FDS, CMOGSTORED_FD does not rely
+on inherited file descriptors being contiguous.
+
 See MALLOC TUNING for environment variables which may affect memory
 usage.
 
-- 
EW