* cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD @ 2014-09-07 10:37 Mikolaj Golub 2014-09-07 10:55 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Mikolaj Golub @ 2014-09-07 10:37 UTC (permalink / raw) To: cmogstored-public Hi, I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS: FAIL: test/.mgmt/iostat_watch.slowrb FAIL: test/.mgmt/iostat_watch_multidir.slowrb =============================================== 2 of 107 tests failed See ./test-suite.log Please report to cmogstored-public@bogomips.org =============================================== FAIL: test/.mgmt/iostat_watch.slowrb (exit: 1) ============================================== Run options: -n test_iostat_watch -v # Running tests: TestMgmt#test_iostat_watch = 1.04 s = F Finished tests in 1.037295s, 0.9640 tests/s, 3.8562 assertions/s. 1) Failure: test_iostat_watch(TestMgmt) [./test/mgmt.rb:350]: "666\t-\n" slow mountpoints increases the chance of this test failing. Expected /^666\t(?-mix:\d+(?:\.\d+)?)\n/ to match "666\t-\n". 1 tests, 4 assertions, 1 failures, 0 errors, 0 skips FAIL: test/.mgmt/iostat_watch_multidir.slowrb (exit: 1) ======================================================= Run options: -n test_iostat_watch_multidir -v # Running tests: TestMgmt#test_iostat_watch_multidir = 2.04 s = F Finished tests in 2.043899s, 0.4893 tests/s, 2.9356 assertions/s. 1) Failure: test_iostat_watch_multidir(TestMgmt) [./test/mgmt.rb:366]: ["666\t-\n", "999\t-\n"] slow mountpoints increases the chance of this test failing. Expected /^(666|999)\t(?-mix:\d+(?:\.\d+)?)\n/ to match "666\t-\n". 1 tests, 6 assertions, 1 failures, 0 errors, 0 skips -- Mikolaj Golub ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD 2014-09-07 10:37 cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD Mikolaj Golub @ 2014-09-07 10:55 ` Eric Wong 2014-09-07 13:44 ` Mikolaj Golub 0 siblings, 1 reply; 6+ messages in thread From: Eric Wong @ 2014-09-07 10:55 UTC (permalink / raw) To: Mikolaj Golub; +Cc: cmogstored-public Mikolaj Golub <to.my.trociny@gmail.com> wrote: > Hi, > > I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS: Hi, is the build dir on ZFS, too? The Makefile tries to get TMPDIR to be inside the build directory since /tmp is often RAM-only and not visible in iostat... I just tested this on a FreeBSD 9.x VM before the release without issues (UFS, I believe, definitely not ZFS). Just wondering, can you retry the test on 1.4.0 your system, since the actual tests for iostat didn't change, I suspect it's a test environment problem... Finally, can you show the "iostat -dx 1 30" output on your build dir? I'll try to take a closer look at this in a day or two. Thanks for maintaining this port! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD 2014-09-07 10:55 ` Eric Wong @ 2014-09-07 13:44 ` Mikolaj Golub 2014-09-07 20:50 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Mikolaj Golub @ 2014-09-07 13:44 UTC (permalink / raw) To: Eric Wong; +Cc: cmogstored-public On Sun, Sep 07, 2014 at 10:55:55AM +0000, Eric Wong wrote: > Mikolaj Golub <to.my.trociny@gmail.com> wrote: > > Hi, > > > > I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS: > > Hi, is the build dir on ZFS, too? The Makefile tries to get TMPDIR > to be inside the build directory since /tmp is often RAM-only and > not visible in iostat... > > I just tested this on a FreeBSD 9.x VM before the release without issues > (UFS, I believe, definitely not ZFS). > > Just wondering, can you retry the test on 1.4.0 your system, since the > actual tests for iostat didn't change, I suspect it's a test environment > problem... Yes, this is an environment issue, not 1.4.1 regression. The same is on 1.4.0. The tests pass on FreeBSD with UFS if mounted using device names. If e.g. gpt lables are used insted of names the tests will fail. Also they fail with ZFS. I suppose this is because stat(2) and friends return zpool device or gptlabel for the device directory, which can't be found in iostat output. I observed the same issue with mogstored. Don't know about a good solution, but it might be enough if it were possible to specify mapping between mogstored devices (directories) and devices in iostat output manually in the config. > > Finally, can you show the "iostat -dx 1 30" output on your build > dir? zhuzha:~% uname -a FreeBSD zhuzha 10.0-STABLE FreeBSD 10.0-STABLE #15 r269641M: Sun Aug 10 18:45:44 UTC 2014 root@zhuzha:/usr/obj/usr/src/sys/ZHUZHA amd64 zhuzha:~% iostat -dx 1 30 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 2.8 14.7 144.3 710.9 1 6.2 7 ada1 0.0 0.0 0.0 0.0 0 0.1 0 cd0 0.0 0.0 0.0 0.0 0 0.0 0 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics ... zhuzha:~% mount zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls) ... zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls) zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls) ... Or this: root@marta:~ # uname -a FreeBSD marta 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@marta:~ # iostat -dx 1 30 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b vtbd0 0.8 6.3 16.2 141.9 0 0.8 0 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b vtbd0 0.0 0.0 0.0 0.0 0 0.0 0 ^C root@marta:~ # mount /dev/gpt/martaroot on / (ufs, local) devfs on /dev (devfs, local, multilabel) fdescfs on /dev/fd (fdescfs) But it works here: kopusha:~% uname -a FreeBSD kopusha.home.net 11.0-CURRENT FreeBSD 11.0-CURRENT #65 r268370: Tue Jul 8 07:07:10 EEST 2014 root@kopusha.home.net:/home/obj/home/golub/freebsd/base/head/sys/KOPUSHA amd64 kopusha:~% mount /dev/ada0s1a on / (ufs, local, journaled soft-updates) /dev/ada0s1e on /tmp (ufs, local, journaled soft-updates) /dev/ada0s1f on /usr (ufs, local, journaled soft-updates) /dev/ada0s1d on /var (ufs, local, journaled soft-updates) ... kopusha:~% iostat -dx 1 30 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 22.2 15.8 218.7 375.2 0 20.5 20 cd0 0.0 0.0 0.0 0.0 0 0.0 0 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 > > I'll try to take a closer look at this in a day or two. > Thanks for maintaining this port! -- Mikolaj Golub ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD 2014-09-07 13:44 ` Mikolaj Golub @ 2014-09-07 20:50 ` Eric Wong 2014-09-08 13:12 ` Mikolaj Golub 0 siblings, 1 reply; 6+ messages in thread From: Eric Wong @ 2014-09-07 20:50 UTC (permalink / raw) To: mogile; +Cc: cmogstored-public, Mikolaj Golub +mogile list, since this affects Perl mogstored, too Current thread on cmogstored-public here: http://bogomips.org/cmogstored-public/?r=8fd0f99fe9135b0e5 TL;DR: sidechannel "watch" does not show I/O util on FreeBSD+ZFS with either Perl mogstored or cmogstored. Mikolaj Golub <to.my.trociny@gmail.com> wrote: > On Sun, Sep 07, 2014 at 10:55:55AM +0000, Eric Wong wrote: > > Mikolaj Golub <to.my.trociny@gmail.com> wrote: > > > Hi, > > > > > > I have been observing the following tests fail on FreeBSD 10.0-STABLE, ZFS: > > > > Hi, is the build dir on ZFS, too? The Makefile tries to get TMPDIR > > to be inside the build directory since /tmp is often RAM-only and > > not visible in iostat... > > > > I just tested this on a FreeBSD 9.x VM before the release without issues > > (UFS, I believe, definitely not ZFS). > > > > Just wondering, can you retry the test on 1.4.0 your system, since the > > actual tests for iostat didn't change, I suspect it's a test environment > > problem... > > Yes, this is an environment issue, not 1.4.1 regression. The same is > on 1.4.0. > > The tests pass on FreeBSD with UFS if mounted using device names. If > e.g. gpt lables are used insted of names the tests will fail. Also > they fail with ZFS. I suppose this is because stat(2) and friends > return zpool device or gptlabel for the device directory, which can't > be found in iostat output. I observed the same issue with mogstored. > > Don't know about a good solution, but it might be enough if it were > possible to specify mapping between mogstored devices (directories) > and devices in iostat output manually in the config. I hate forcing users to configure more things, so that's a last resort and it should be optional. We should get this fixed in Perl mogstored, too; and it's probably easier to it there first since there's more Perl hackers around these parts. I think averaging IO utilization across all unaccounted devices in mogstored is acceptable, since (I assume) it's common to have only one ZFS pool per host with multiple devices. I don't know much about ZFS or other LVM systems, but it probably tries to distribute load across devices internally, making per-physical device output less useful in the first place, right? Finally, the iostat utilization stuff isn't useful or needed by folks on light loads or really fast SSDs. The important part for ZFS everybody needs is generating the disk usage files correctly. I got that working for Ask Bjørn Hansen in 1.0.0 and he hasn't reported any brokeness since :) (rest of your message quoted for mogile list). > > Finally, can you show the "iostat -dx 1 30" output on your build > > dir? > > zhuzha:~% uname -a > FreeBSD zhuzha 10.0-STABLE FreeBSD 10.0-STABLE #15 r269641M: Sun Aug 10 18:45:44 UTC 2014 root@zhuzha:/usr/obj/usr/src/sys/ZHUZHA amd64 > > zhuzha:~% iostat -dx 1 30 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 2.8 14.7 144.3 710.9 1 6.2 7 > ada1 0.0 0.0 0.0 0.0 0 0.1 0 > cd0 0.0 0.0 0.0 0.0 0 0.0 0 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > pass1 0.0 0.0 0.0 0.0 0 0.0 0 > pass2 0.0 0.0 0.0 0.0 0 0.0 0 > pass3 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > ... > > zhuzha:~% mount > zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls) > ... > zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls) > zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls) > ... > > Or this: > > root@marta:~ # uname -a > FreeBSD marta 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > root@marta:~ # iostat -dx 1 30 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > vtbd0 0.8 6.3 16.2 141.9 0 0.8 0 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > vtbd0 0.0 0.0 0.0 0.0 0 0.0 0 > ^C > root@marta:~ # mount > /dev/gpt/martaroot on / (ufs, local) > devfs on /dev (devfs, local, multilabel) > fdescfs on /dev/fd (fdescfs) > > But it works here: > > kopusha:~% uname -a > FreeBSD kopusha.home.net 11.0-CURRENT FreeBSD 11.0-CURRENT #65 r268370: Tue Jul 8 07:07:10 EEST 2014 root@kopusha.home.net:/home/obj/home/golub/freebsd/base/head/sys/KOPUSHA amd64 > > kopusha:~% mount > /dev/ada0s1a on / (ufs, local, journaled soft-updates) > /dev/ada0s1e on /tmp (ufs, local, journaled soft-updates) > /dev/ada0s1f on /usr (ufs, local, journaled soft-updates) > /dev/ada0s1d on /var (ufs, local, journaled soft-updates) > ... > > kopusha:~% iostat -dx 1 30 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 22.2 15.8 218.7 375.2 0 20.5 20 > cd0 0.0 0.0 0.0 0.0 0 0.0 0 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > pass1 0.0 0.0 0.0 0.0 0 0.0 0 > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD 2014-09-07 20:50 ` Eric Wong @ 2014-09-08 13:12 ` Mikolaj Golub 2014-09-09 6:49 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Mikolaj Golub @ 2014-09-08 13:12 UTC (permalink / raw) To: Eric Wong; +Cc: mogile, cmogstored-public On Sun, Sep 07, 2014 at 08:50:04PM +0000, Eric Wong wrote: > I think averaging IO utilization across all unaccounted devices in > mogstored is acceptable, since (I assume) it's common to have only > one ZFS pool per host with multiple devices. > > I don't know much about ZFS or other LVM systems, but it probably tries > to distribute load across devices internally, making per-physical device > output less useful in the first place, right? Yes. With ZFS, `zpool iostat' is more useful in most of the cases than iostat. ref10-amd64:/home/trociny% zpool iostat 1 30 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- zroot 1.02T 793G 153 13 302K 27.8K zroot 1.02T 793G 0 0 0 0 zroot 1.02T 793G 0 159 0 240K zroot 1.02T 793G 0 0 0 0 ^C ref10-amd64:/home/trociny% zpool iostat -v 1 30 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- zroot 1.02T 793G 153 13 302K 27.8K raidz1 1.02T 793G 153 13 302K 27.8K ada0p3 - - 69 4 130K 13.1K ada1p3 - - 66 3 125K 12.9K ada2p3 - - 68 3 133K 13.1K ada3p3 - - 66 3 126K 12.8K ---------- ----- ----- ----- ----- ----- ----- capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- zroot 1.02T 793G 0 0 0 0 raidz1 1.02T 793G 0 0 0 0 ada0p3 - - 0 0 0 0 ada1p3 - - 0 0 0 0 ada2p3 - - 0 0 0 0 ada3p3 - - 0 0 0 0 ---------- ----- ----- ----- ----- ----- ----- ^C -- Mikolaj Golub ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD 2014-09-08 13:12 ` Mikolaj Golub @ 2014-09-09 6:49 ` Eric Wong 0 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2014-09-09 6:49 UTC (permalink / raw) To: Mikolaj Golub; +Cc: mogile, cmogstored-public Mikolaj Golub <to.my.trociny@gmail.com> wrote: > On Sun, Sep 07, 2014 at 08:50:04PM +0000, Eric Wong wrote: > > I don't know much about ZFS or other LVM systems, but it probably tries > > to distribute load across devices internally, making per-physical device > > output less useful in the first place, right? > > Yes. > > With ZFS, `zpool iostat' is more useful in most of the cases than > iostat. OK, I'm not sure what to do w.r.t MogileFS proper. I suspect the combined bandwidth of a few devices in a zpool is usually high enough that any IO util% used by MogileFS is not useful at all. There was already one proposed patch a few years ago about switching to use await/svctimes for SSDs... So back to the original issue of cmogstored tests not passing due to this. I suspect it's OK to relax the tests because it's probably common to run tests/builds on tmpfs which does not report iostat, either. So, I'll probably prepare a 1.4.2 release to ease downstream/CI maintainers lives in a few days (I give it a few days to see if any other issues pop up). ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-09-09 6:49 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-09-07 10:37 cmogstored 1.4.1: check: 2 of 107 tests failed on FreeBSD Mikolaj Golub 2014-09-07 10:55 ` Eric Wong 2014-09-07 13:44 ` Mikolaj Golub 2014-09-07 20:50 ` Eric Wong 2014-09-08 13:12 ` Mikolaj Golub 2014-09-09 6:49 ` Eric Wong
Code repositories for project(s) associated with this public inbox https://yhbt.net/cmogstored.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).