mogstored (and cmogstored) threads per device

mogstored (and cmogstored) threads per device

[Eric Wong]

Currently, mogstored uses 10 threads per device, with a cap of 100
threads total.  This was set in 2006[1] when it was probably rare to
have 10 devices on commodity hardware.  I suspect the 100 cap is
necessary with more devices as IO::AIO does not set a limit on pthread
stacks sizes (and typical pthreads stack sizes are huge by default).

For rotational disks, using multiple OS-level threads as Perl IO::AIO
and cmogstored do is still the only way to get concurrency for all
FS-level ops[2].

cmogstored also uses 10 threads per device, but with no cap.  It will
continue gracefully if the OS limits on thread creation are reached.
On a modern GNU/Linux or FreeBSD, I'm comfortable having no cap in
cmogstored with 20K page stacks (+4K guard stack).

However, cmogstored is completely multi-threaded with no Perl
bottlenecks.  Thus the threads are also subject to less wait time
and contend for more CPU time.

So I'm considering lowering default thread counts in cmogstored.
Maybe conservatively to 8 per-device; but maybe as drastically as
2-4 per-device.  Maybe mogstored can use a lower default, too.

Any thoughts?

Of course, anybody can also set "SERVER aio_threads = XX" via
sidechannel on either server, but the the goal is to lower memory usage
and reduce scheduler overhead out-of-the-box.

I'm also considering shipping the venerable dlmalloc in cmogstored[3].
glibc default arena knobs are completely wrong for cmogstored and
not enough people read the cmogstored manpage or glibc docs for
recommendations on tuning malloc...

[1] git commit de43ded3658311b83dd23f4176b2fc37c8497599 or SVN r390
[2] The POSIX AIO and Linux-kernel-only AIO APIs lack
    analogous interfaces for open, stat, unlink, etc...
[3] http://bogomips.org/cmogstored-public/m/import-and-use-dlmalloc-2.8.6%40s

Re: mogstored (and cmogstored) threads per device

[dormando]

Wonder if you could autotune it a bit? Start with a low default, then
self-increase if the threads are starved too often. Self-tuning would
disable if started at something other than the default or if 'SERVER
aio_threads = XX' were issued

On Sun, 2 Nov 2014, Eric Wong wrote:

> Currently, mogstored uses 10 threads per device, with a cap of 100
> threads total.  This was set in 2006[1] when it was probably rare to
> have 10 devices on commodity hardware.  I suspect the 100 cap is
> necessary with more devices as IO::AIO does not set a limit on pthread
> stacks sizes (and typical pthreads stack sizes are huge by default).
>
> For rotational disks, using multiple OS-level threads as Perl IO::AIO
> and cmogstored do is still the only way to get concurrency for all
> FS-level ops[2].
>
> cmogstored also uses 10 threads per device, but with no cap.  It will
> continue gracefully if the OS limits on thread creation are reached.
> On a modern GNU/Linux or FreeBSD, I'm comfortable having no cap in
> cmogstored with 20K page stacks (+4K guard stack).
>
> However, cmogstored is completely multi-threaded with no Perl
> bottlenecks.  Thus the threads are also subject to less wait time
> and contend for more CPU time.
>
> So I'm considering lowering default thread counts in cmogstored.
> Maybe conservatively to 8 per-device; but maybe as drastically as
> 2-4 per-device.  Maybe mogstored can use a lower default, too.
>
> Any thoughts?
>
> Of course, anybody can also set "SERVER aio_threads = XX" via
> sidechannel on either server, but the the goal is to lower memory usage
> and reduce scheduler overhead out-of-the-box.
>
> I'm also considering shipping the venerable dlmalloc in cmogstored[3].
> glibc default arena knobs are completely wrong for cmogstored and
> not enough people read the cmogstored manpage or glibc docs for
> recommendations on tuning malloc...
>
> [1] git commit de43ded3658311b83dd23f4176b2fc37c8497599 or SVN r390
> [2] The POSIX AIO and Linux-kernel-only AIO APIs lack
>     analogous interfaces for open, stat, unlink, etc...
> [3] http://bogomips.org/cmogstored-public/m/import-and-use-dlmalloc-2.8.6%40s
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "mogile" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mogile+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

Re: mogstored (and cmogstored) threads per device

[Eric Wong]

dormando <dormando@rydia.net> wrote:
> Wonder if you could autotune it a bit? Start with a low default, then
> self-increase if the threads are starved too often. Self-tuning would
> disable if started at something other than the default or if 'SERVER
> aio_threads = XX' were issued

I've thought of autotuning, but then reliably detecting starvation in
userspace might be more trouble than it's worth (and expensive).
Probably the best way to leave things unchanged: give the kernel as
much work as possible and let the kernel scheduler decide what to do
with it.

Threads (even with 20K stack) are still pretty cheap and epoll interacts
well with the scheduler: extra/excess threads never get woken due to
FIFO ordering of threads in epoll_wait.