From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-2.0 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, URIBL_BLOCKED shortcircuit=no autolearn=unavailable version=3.3.2 X-Original-To: cmogstored-public@bogomips.org Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 044291FD74; Sun, 7 Sep 2014 20:50:04 +0000 (UTC) Date: Sun, 7 Sep 2014 20:50:04 +0000 From: Eric Wong To: mogile@googlegroups.com Cc: cmogstored-public@bogomips.org, Mikolaj Golub Subject: Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD Message-ID: <20140907205004.GA31234@dcvr.yhbt.net> References: <20140907103748.GA12577@gmail.com> <20140907105555.GA27823@dcvr.yhbt.net> <20140907134429.GB12577@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20140907134429.GB12577@gmail.com> List-Id: +mogile list, since this affects Perl mogstored, too Current thread on cmogstored-public here: http://bogomips.org/cmogstored-public/?r=8fd0f99fe9135b0e5 TL;DR: sidechannel "watch" does not show I/O util on FreeBSD+ZFS with either Perl mogstored or cmogstored. Mikolaj Golub wrote: > On Sun, Sep 07, 2014 at 10:55:55AM +0000, Eric Wong wrote: > > Mikolaj Golub wrote: > > > Hi, > > > > > > I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS: > > > > Hi, is the build dir on ZFS, too? The Makefile tries to get TMPDIR > > to be inside the build directory since /tmp is often RAM-only and > > not visible in iostat... > > > > I just tested this on a FreeBSD 9.x VM before the release without issues > > (UFS, I believe, definitely not ZFS). > > > > Just wondering, can you retry the test on 1.4.0 your system, since the > > actual tests for iostat didn't change, I suspect it's a test environment > > problem... > > Yes, this is an environment issue, not 1.4.1 regression. The same is > on 1.4.0. > > The tests pass on FreeBSD with UFS if mounted using device names. If > e.g. gpt lables are used insted of names the tests will fail. Also > they fail with ZFS. I suppose this is because stat(2) and friends > return zpool device or gptlabel for the device directory, which can't > be found in iostat output. I observed the same issue with mogstored. > > Don't know about a good solution, but it might be enough if it were > possible to specify mapping between mogstored devices (directories) > and devices in iostat output manually in the config. I hate forcing users to configure more things, so that's a last resort and it should be optional. We should get this fixed in Perl mogstored, too; and it's probably easier to it there first since there's more Perl hackers around these parts. I think averaging IO utilization across all unaccounted devices in mogstored is acceptable, since (I assume) it's common to have only one ZFS pool per host with multiple devices. I don't know much about ZFS or other LVM systems, but it probably tries to distribute load across devices internally, making per-physical device output less useful in the first place, right? Finally, the iostat utilization stuff isn't useful or needed by folks on light loads or really fast SSDs. The important part for ZFS everybody needs is generating the disk usage files correctly. I got that working for Ask Bjørn Hansen in 1.0.0 and he hasn't reported any brokeness since :) (rest of your message quoted for mogile list). > > Finally, can you show the "iostat -dx 1 30" output on your build > > dir? > > zhuzha:~% uname -a > FreeBSD zhuzha 10.0-STABLE FreeBSD 10.0-STABLE #15 r269641M: Sun Aug 10 18:45:44 UTC 2014 root@zhuzha:/usr/obj/usr/src/sys/ZHUZHA amd64 > > zhuzha:~% iostat -dx 1 30 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 2.8 14.7 144.3 710.9 1 6.2 7 > ada1 0.0 0.0 0.0 0.0 0 0.1 0 > cd0 0.0 0.0 0.0 0.0 0 0.0 0 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > pass1 0.0 0.0 0.0 0.0 0 0.0 0 > pass2 0.0 0.0 0.0 0.0 0 0.0 0 > pass3 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > ... > > zhuzha:~% mount > zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls) > ... > zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls) > zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls) > ... > > Or this: > > root@marta:~ # uname -a > FreeBSD marta 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > root@marta:~ # iostat -dx 1 30 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > vtbd0 0.8 6.3 16.2 141.9 0 0.8 0 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > vtbd0 0.0 0.0 0.0 0.0 0 0.0 0 > ^C > root@marta:~ # mount > /dev/gpt/martaroot on / (ufs, local) > devfs on /dev (devfs, local, multilabel) > fdescfs on /dev/fd (fdescfs) > > But it works here: > > kopusha:~% uname -a > FreeBSD kopusha.home.net 11.0-CURRENT FreeBSD 11.0-CURRENT #65 r268370: Tue Jul 8 07:07:10 EEST 2014 root@kopusha.home.net:/home/obj/home/golub/freebsd/base/head/sys/KOPUSHA amd64 > > kopusha:~% mount > /dev/ada0s1a on / (ufs, local, journaled soft-updates) > /dev/ada0s1e on /tmp (ufs, local, journaled soft-updates) > /dev/ada0s1f on /usr (ufs, local, journaled soft-updates) > /dev/ada0s1d on /var (ufs, local, journaled soft-updates) > ... > > kopusha:~% iostat -dx 1 30 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 22.2 15.8 218.7 375.2 0 20.5 20 > cd0 0.0 0.0 0.0 0.0 0 0.0 0 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > pass1 0.0 0.0 0.0 0.0 0 0.0 0 >