mogstored (and cmogstored) threads per device

[Eric Wong <e@80x24.org>]

Currently, mogstored uses 10 threads per device, with a cap of 100
threads total.  This was set in 2006[1] when it was probably rare to
have 10 devices on commodity hardware.  I suspect the 100 cap is
necessary with more devices as IO::AIO does not set a limit on pthread
stacks sizes (and typical pthreads stack sizes are huge by default).

For rotational disks, using multiple OS-level threads as Perl IO::AIO
and cmogstored do is still the only way to get concurrency for all
FS-level ops[2].

cmogstored also uses 10 threads per device, but with no cap.  It will
continue gracefully if the OS limits on thread creation are reached.
On a modern GNU/Linux or FreeBSD, I'm comfortable having no cap in
cmogstored with 20K page stacks (+4K guard stack).

However, cmogstored is completely multi-threaded with no Perl
bottlenecks.  Thus the threads are also subject to less wait time
and contend for more CPU time.

So I'm considering lowering default thread counts in cmogstored.
Maybe conservatively to 8 per-device; but maybe as drastically as
2-4 per-device.  Maybe mogstored can use a lower default, too.

Any thoughts?

Of course, anybody can also set "SERVER aio_threads = XX" via
sidechannel on either server, but the the goal is to lower memory usage
and reduce scheduler overhead out-of-the-box.

I'm also considering shipping the venerable dlmalloc in cmogstored[3].
glibc default arena knobs are completely wrong for cmogstored and
not enough people read the cmogstored manpage or glibc docs for
recommendations on tuning malloc...

[1] git commit de43ded3658311b83dd23f4176b2fc37c8497599 or SVN r390
[2] The POSIX AIO and Linux-kernel-only AIO APIs lack
    analogous interfaces for open, stat, unlink, etc...
[3] http://bogomips.org/cmogstored-public/m/import-and-use-dlmalloc-2.8.6%40s