From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS6939 173.255.240.0/20 X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,RCVD_IN_BRBL_LASTEXT, URIBL_BLOCKED shortcircuit=no autolearn=no version=3.3.2 X-Original-To: cmogstored-public@bogomips.org Received: from mail.rydia.net (li263-96.members.linode.com [173.255.253.96]) by dcvr.yhbt.net (Postfix) with ESMTP id AFD0D1FA57 for ; Tue, 16 Dec 2014 07:17:49 +0000 (UTC) Received: from dtop.apt (c-50-136-246-16.hsd1.ca.comcast.net [50.136.246.16]) by mail.rydia.net (Postfix) with ESMTPA id 068F46920A9; Mon, 15 Dec 2014 23:17:48 -0800 (PST) Date: Mon, 15 Dec 2014 23:18:42 -0800 (PST) From: dormando X-X-Sender: dormando@dtop To: mogile@googlegroups.com cc: cmogstored-public@bogomips.org Subject: Re: mogstored (and cmogstored) threads per device In-Reply-To: <20141102-revisiting-threads-per-dev@mogile> Message-ID: References: <20141102-revisiting-threads-per-dev@mogile> User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII List-Id: Wonder if you could autotune it a bit? Start with a low default, then self-increase if the threads are starved too often. Self-tuning would disable if started at something other than the default or if 'SERVER aio_threads = XX' were issued On Sun, 2 Nov 2014, Eric Wong wrote: > Currently, mogstored uses 10 threads per device, with a cap of 100 > threads total. This was set in 2006[1] when it was probably rare to > have 10 devices on commodity hardware. I suspect the 100 cap is > necessary with more devices as IO::AIO does not set a limit on pthread > stacks sizes (and typical pthreads stack sizes are huge by default). > > For rotational disks, using multiple OS-level threads as Perl IO::AIO > and cmogstored do is still the only way to get concurrency for all > FS-level ops[2]. > > cmogstored also uses 10 threads per device, but with no cap. It will > continue gracefully if the OS limits on thread creation are reached. > On a modern GNU/Linux or FreeBSD, I'm comfortable having no cap in > cmogstored with 20K page stacks (+4K guard stack). > > However, cmogstored is completely multi-threaded with no Perl > bottlenecks. Thus the threads are also subject to less wait time > and contend for more CPU time. > > So I'm considering lowering default thread counts in cmogstored. > Maybe conservatively to 8 per-device; but maybe as drastically as > 2-4 per-device. Maybe mogstored can use a lower default, too. > > Any thoughts? > > Of course, anybody can also set "SERVER aio_threads = XX" via > sidechannel on either server, but the the goal is to lower memory usage > and reduce scheduler overhead out-of-the-box. > > I'm also considering shipping the venerable dlmalloc in cmogstored[3]. > glibc default arena knobs are completely wrong for cmogstored and > not enough people read the cmogstored manpage or glibc docs for > recommendations on tuning malloc... > > [1] git commit de43ded3658311b83dd23f4176b2fc37c8497599 or SVN r390 > [2] The POSIX AIO and Linux-kernel-only AIO APIs lack > analogous interfaces for open, stat, unlink, etc... > [3] http://bogomips.org/cmogstored-public/m/import-and-use-dlmalloc-2.8.6%40s > > -- > > --- > You received this message because you are subscribed to the Google Groups "mogile" group. > To unsubscribe from this group and stop receiving emails from it, send an email to mogile+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >