* cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD
@ 2014-09-07 10:37 Mikolaj Golub
2014-09-07 10:55 ` Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: Mikolaj Golub @ 2014-09-07 10:37 UTC (permalink / raw)
To: cmogstored-public
Hi,
I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS:
FAIL: test/.mgmt/iostat_watch.slowrb
FAIL: test/.mgmt/iostat_watch_multidir.slowrb
===============================================
2 of 107 tests failed
See ./test-suite.log
Please report to cmogstored-public@bogomips.org
===============================================
FAIL: test/.mgmt/iostat_watch.slowrb (exit: 1)
==============================================
Run options: -n test_iostat_watch -v
# Running tests:
TestMgmt#test_iostat_watch = 1.04 s = F
Finished tests in 1.037295s, 0.9640 tests/s, 3.8562 assertions/s.
1) Failure:
test_iostat_watch(TestMgmt) [./test/mgmt.rb:350]:
"666\t-\n" slow mountpoints increases the chance of this test failing.
Expected /^666\t(?-mix:\d+(?:\.\d+)?)\n/ to match "666\t-\n".
1 tests, 4 assertions, 1 failures, 0 errors, 0 skips
FAIL: test/.mgmt/iostat_watch_multidir.slowrb (exit: 1)
=======================================================
Run options: -n test_iostat_watch_multidir -v
# Running tests:
TestMgmt#test_iostat_watch_multidir = 2.04 s = F
Finished tests in 2.043899s, 0.4893 tests/s, 2.9356 assertions/s.
1) Failure:
test_iostat_watch_multidir(TestMgmt) [./test/mgmt.rb:366]:
["666\t-\n", "999\t-\n"] slow mountpoints increases the chance of this test failing.
Expected /^(666|999)\t(?-mix:\d+(?:\.\d+)?)\n/ to match "666\t-\n".
1 tests, 6 assertions, 1 failures, 0 errors, 0 skips
--
Mikolaj Golub
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD
2014-09-07 10:37 cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD Mikolaj Golub
@ 2014-09-07 10:55 ` Eric Wong
2014-09-07 13:44 ` Mikolaj Golub
0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2014-09-07 10:55 UTC (permalink / raw)
To: Mikolaj Golub; +Cc: cmogstored-public
Mikolaj Golub <to.my.trociny@gmail.com> wrote:
> Hi,
>
> I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS:
Hi, is the build dir on ZFS, too? The Makefile tries to get TMPDIR
to be inside the build directory since /tmp is often RAM-only and
not visible in iostat...
I just tested this on a FreeBSD 9.x VM before the release without issues
(UFS, I believe, definitely not ZFS).
Just wondering, can you retry the test on 1.4.0 your system, since the
actual tests for iostat didn't change, I suspect it's a test environment
problem...
Finally, can you show the "iostat -dx 1 30" output on your build dir?
I'll try to take a closer look at this in a day or two.
Thanks for maintaining this port!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD
2014-09-07 10:55 ` Eric Wong
@ 2014-09-07 13:44 ` Mikolaj Golub
2014-09-07 20:50 ` Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: Mikolaj Golub @ 2014-09-07 13:44 UTC (permalink / raw)
To: Eric Wong; +Cc: cmogstored-public
On Sun, Sep 07, 2014 at 10:55:55AM +0000, Eric Wong wrote:
> Mikolaj Golub <to.my.trociny@gmail.com> wrote:
> > Hi,
> >
> > I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS:
>
> Hi, is the build dir on ZFS, too? The Makefile tries to get TMPDIR
> to be inside the build directory since /tmp is often RAM-only and
> not visible in iostat...
>
> I just tested this on a FreeBSD 9.x VM before the release without issues
> (UFS, I believe, definitely not ZFS).
>
> Just wondering, can you retry the test on 1.4.0 your system, since the
> actual tests for iostat didn't change, I suspect it's a test environment
> problem...
Yes, this is an environment issue, not 1.4.1 regression. The same is
on 1.4.0.
The tests pass on FreeBSD with UFS if mounted using device names. If
e.g. gpt lables are used insted of names the tests will fail. Also
they fail with ZFS. I suppose this is because stat(2) and friends
return zpool device or gptlabel for the device directory, which can't
be found in iostat output. I observed the same issue with mogstored.
Don't know about a good solution, but it might be enough if it were
possible to specify mapping between mogstored devices (directories)
and devices in iostat output manually in the config.
>
> Finally, can you show the "iostat -dx 1 30" output on your build
> dir?
zhuzha:~% uname -a
FreeBSD zhuzha 10.0-STABLE FreeBSD 10.0-STABLE #15 r269641M: Sun Aug 10 18:45:44 UTC 2014 root@zhuzha:/usr/obj/usr/src/sys/ZHUZHA amd64
zhuzha:~% iostat -dx 1 30
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
ada0 2.8 14.7 144.3 710.9 1 6.2 7
ada1 0.0 0.0 0.0 0.0 0 0.1 0
cd0 0.0 0.0 0.0 0.0 0 0.0 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
...
zhuzha:~% mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
...
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
...
Or this:
root@marta:~ # uname -a
FreeBSD marta 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
root@marta:~ # iostat -dx 1 30
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
vtbd0 0.8 6.3 16.2 141.9 0 0.8 0
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
vtbd0 0.0 0.0 0.0 0.0 0 0.0 0
^C
root@marta:~ # mount
/dev/gpt/martaroot on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
fdescfs on /dev/fd (fdescfs)
But it works here:
kopusha:~% uname -a
FreeBSD kopusha.home.net 11.0-CURRENT FreeBSD 11.0-CURRENT #65 r268370: Tue Jul 8 07:07:10 EEST 2014 root@kopusha.home.net:/home/obj/home/golub/freebsd/base/head/sys/KOPUSHA amd64
kopusha:~% mount
/dev/ada0s1a on / (ufs, local, journaled soft-updates)
/dev/ada0s1e on /tmp (ufs, local, journaled soft-updates)
/dev/ada0s1f on /usr (ufs, local, journaled soft-updates)
/dev/ada0s1d on /var (ufs, local, journaled soft-updates)
...
kopusha:~% iostat -dx 1 30
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
ada0 22.2 15.8 218.7 375.2 0 20.5 20
cd0 0.0 0.0 0.0 0.0 0 0.0 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
>
> I'll try to take a closer look at this in a day or two.
> Thanks for maintaining this port!
--
Mikolaj Golub
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD
2014-09-07 13:44 ` Mikolaj Golub
@ 2014-09-07 20:50 ` Eric Wong
2014-09-08 13:12 ` Mikolaj Golub
0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2014-09-07 20:50 UTC (permalink / raw)
To: mogile; +Cc: cmogstored-public, Mikolaj Golub
+mogile list, since this affects Perl mogstored, too
Current thread on cmogstored-public here:
http://bogomips.org/cmogstored-public/?r=8fd0f99fe9135b0e5
TL;DR: sidechannel "watch" does not show I/O util on FreeBSD+ZFS
with either Perl mogstored or cmogstored.
Mikolaj Golub <to.my.trociny@gmail.com> wrote:
> On Sun, Sep 07, 2014 at 10:55:55AM +0000, Eric Wong wrote:
> > Mikolaj Golub <to.my.trociny@gmail.com> wrote:
> > > Hi,
> > >
> > > I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS:
> >
> > Hi, is the build dir on ZFS, too? The Makefile tries to get TMPDIR
> > to be inside the build directory since /tmp is often RAM-only and
> > not visible in iostat...
> >
> > I just tested this on a FreeBSD 9.x VM before the release without issues
> > (UFS, I believe, definitely not ZFS).
> >
> > Just wondering, can you retry the test on 1.4.0 your system, since the
> > actual tests for iostat didn't change, I suspect it's a test environment
> > problem...
>
> Yes, this is an environment issue, not 1.4.1 regression. The same is
> on 1.4.0.
>
> The tests pass on FreeBSD with UFS if mounted using device names. If
> e.g. gpt lables are used insted of names the tests will fail. Also
> they fail with ZFS. I suppose this is because stat(2) and friends
> return zpool device or gptlabel for the device directory, which can't
> be found in iostat output. I observed the same issue with mogstored.
>
> Don't know about a good solution, but it might be enough if it were
> possible to specify mapping between mogstored devices (directories)
> and devices in iostat output manually in the config.
I hate forcing users to configure more things, so that's a last resort
and it should be optional.
We should get this fixed in Perl mogstored, too; and it's probably
easier to it there first since there's more Perl hackers around these
parts.
I think averaging IO utilization across all unaccounted devices in
mogstored is acceptable, since (I assume) it's common to have only
one ZFS pool per host with multiple devices.
I don't know much about ZFS or other LVM systems, but it probably tries
to distribute load across devices internally, making per-physical device
output less useful in the first place, right?
Finally, the iostat utilization stuff isn't useful or needed by folks on
light loads or really fast SSDs.
The important part for ZFS everybody needs is generating the disk usage
files correctly. I got that working for Ask Bjørn Hansen in 1.0.0 and
he hasn't reported any brokeness since :)
(rest of your message quoted for mogile list).
> > Finally, can you show the "iostat -dx 1 30" output on your build
> > dir?
>
> zhuzha:~% uname -a
> FreeBSD zhuzha 10.0-STABLE FreeBSD 10.0-STABLE #15 r269641M: Sun Aug 10 18:45:44 UTC 2014 root@zhuzha:/usr/obj/usr/src/sys/ZHUZHA amd64
>
> zhuzha:~% iostat -dx 1 30
> extended device statistics
> device r/s w/s kr/s kw/s qlen svc_t %b
> ada0 2.8 14.7 144.3 710.9 1 6.2 7
> ada1 0.0 0.0 0.0 0.0 0 0.1 0
> cd0 0.0 0.0 0.0 0.0 0 0.0 0
> pass0 0.0 0.0 0.0 0.0 0 0.0 0
> pass1 0.0 0.0 0.0 0.0 0 0.0 0
> pass2 0.0 0.0 0.0 0.0 0 0.0 0
> pass3 0.0 0.0 0.0 0.0 0 0.0 0
> extended device statistics
> ...
>
> zhuzha:~% mount
> zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
> ...
> zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
> zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
> ...
>
> Or this:
>
> root@marta:~ # uname -a
> FreeBSD marta 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
> root@marta:~ # iostat -dx 1 30
> extended device statistics
> device r/s w/s kr/s kw/s qlen svc_t %b
> vtbd0 0.8 6.3 16.2 141.9 0 0.8 0
> extended device statistics
> device r/s w/s kr/s kw/s qlen svc_t %b
> vtbd0 0.0 0.0 0.0 0.0 0 0.0 0
> ^C
> root@marta:~ # mount
> /dev/gpt/martaroot on / (ufs, local)
> devfs on /dev (devfs, local, multilabel)
> fdescfs on /dev/fd (fdescfs)
>
> But it works here:
>
> kopusha:~% uname -a
> FreeBSD kopusha.home.net 11.0-CURRENT FreeBSD 11.0-CURRENT #65 r268370: Tue Jul 8 07:07:10 EEST 2014 root@kopusha.home.net:/home/obj/home/golub/freebsd/base/head/sys/KOPUSHA amd64
>
> kopusha:~% mount
> /dev/ada0s1a on / (ufs, local, journaled soft-updates)
> /dev/ada0s1e on /tmp (ufs, local, journaled soft-updates)
> /dev/ada0s1f on /usr (ufs, local, journaled soft-updates)
> /dev/ada0s1d on /var (ufs, local, journaled soft-updates)
> ...
>
> kopusha:~% iostat -dx 1 30
> extended device statistics
> device r/s w/s kr/s kw/s qlen svc_t %b
> ada0 22.2 15.8 218.7 375.2 0 20.5 20
> cd0 0.0 0.0 0.0 0.0 0 0.0 0
> pass0 0.0 0.0 0.0 0.0 0 0.0 0
> pass1 0.0 0.0 0.0 0.0 0 0.0 0
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD
2014-09-07 20:50 ` Eric Wong
@ 2014-09-08 13:12 ` Mikolaj Golub
2014-09-09 6:49 ` Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: Mikolaj Golub @ 2014-09-08 13:12 UTC (permalink / raw)
To: Eric Wong; +Cc: mogile, cmogstored-public
On Sun, Sep 07, 2014 at 08:50:04PM +0000, Eric Wong wrote:
> I think averaging IO utilization across all unaccounted devices in
> mogstored is acceptable, since (I assume) it's common to have only
> one ZFS pool per host with multiple devices.
>
> I don't know much about ZFS or other LVM systems, but it probably tries
> to distribute load across devices internally, making per-physical device
> output less useful in the first place, right?
Yes.
With ZFS, `zpool iostat' is more useful in most of the cases than
iostat.
ref10-amd64:/home/trociny% zpool iostat 1 30
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
zroot 1.02T 793G 153 13 302K 27.8K
zroot 1.02T 793G 0 0 0 0
zroot 1.02T 793G 0 159 0 240K
zroot 1.02T 793G 0 0 0 0
^C
ref10-amd64:/home/trociny% zpool iostat -v 1 30
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
zroot 1.02T 793G 153 13 302K 27.8K
raidz1 1.02T 793G 153 13 302K 27.8K
ada0p3 - - 69 4 130K 13.1K
ada1p3 - - 66 3 125K 12.9K
ada2p3 - - 68 3 133K 13.1K
ada3p3 - - 66 3 126K 12.8K
---------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
zroot 1.02T 793G 0 0 0 0
raidz1 1.02T 793G 0 0 0 0
ada0p3 - - 0 0 0 0
ada1p3 - - 0 0 0 0
ada2p3 - - 0 0 0 0
ada3p3 - - 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
^C
--
Mikolaj Golub
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD
2014-09-08 13:12 ` Mikolaj Golub
@ 2014-09-09 6:49 ` Eric Wong
0 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2014-09-09 6:49 UTC (permalink / raw)
To: Mikolaj Golub; +Cc: mogile, cmogstored-public
Mikolaj Golub <to.my.trociny@gmail.com> wrote:
> On Sun, Sep 07, 2014 at 08:50:04PM +0000, Eric Wong wrote:
> > I don't know much about ZFS or other LVM systems, but it probably tries
> > to distribute load across devices internally, making per-physical device
> > output less useful in the first place, right?
>
> Yes.
>
> With ZFS, `zpool iostat' is more useful in most of the cases than
> iostat.
OK, I'm not sure what to do w.r.t MogileFS proper. I suspect the
combined bandwidth of a few devices in a zpool is usually high enough
that any IO util% used by MogileFS is not useful at all. There was
already one proposed patch a few years ago about switching to use
await/svctimes for SSDs...
So back to the original issue of cmogstored tests not passing due to
this. I suspect it's OK to relax the tests because it's probably common
to run tests/builds on tmpfs which does not report iostat, either.
So, I'll probably prepare a 1.4.2 release to ease downstream/CI
maintainers lives in a few days (I give it a few days to see if
any other issues pop up).
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-09-09 6:49 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-07 10:37 cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD Mikolaj Golub
2014-09-07 10:55 ` Eric Wong
2014-09-07 13:44 ` Mikolaj Golub
2014-09-07 20:50 ` Eric Wong
2014-09-08 13:12 ` Mikolaj Golub
2014-09-09 6:49 ` Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/cmogstored.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).