Re: Heavy load

[Arkadi Colson <arkadi@smartbit.be>]

On 17/12/19 20:42, Eric Wong wrote:
> Arkadi Colson <arkadi@smartbit.be> wrote:
>>> Any idea? If you need more info, please just ask!
>> We are having about 192 devices spread over about 23 cmogstored hosts.
>> Each device is one disk with one partition...
>>> How many "/devXYZ" devices do you have?  Are they all
>>> on different partitions?
> OK, thanks.  I've only got a single host nowadays with 3
> rotational HDD.  Most I ever had was 20 rotational HDD on a
> host but that place is out-of-business.
>
> Since your build did not include -ggdb3 with debug_info by default;
> I wonder if there's something broken in your build system or
> build scripts...  Which compiler are you using?
>
> Can you share the output of "ldd /path/to/cmogstored" ?
root@mogstore:~# ldd /usr/bin/cmogstored
     linux-vdso.so.1 (0x00007fff6584b000)
     libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007f634a7f2000)
     libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f634a453000)
     /lib64/ld-linux-x86-64.so.2 (0x00007f634ac50000)

> Since this is Linux, you're not using libkqueue, are you?
>
> Also, which Linux kernel is it?

In fact it's a clean debian stretch installation with htis kernel:

root@mogstore:~# uname -a
Linux mogstore 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 
(2019-11-11) x86_64 GNU/Linux

>
> Are you using "server aio_threads =" via mgmt interface?
I don't think so. How can I verify this?
>
> Are you using the undocumented -W/--worker-processes or
> -M (multi-config) option(s?)

I don't think so: Config looks like this:

httplisten  = 0.0.0.0:7500
mgmtlisten  = 0.0.0.0:7501
maxconns    = 10000
docroot     = /var/mogdata
daemonize   = 1
server      = none

>
> Is your traffic read-heavy or write-heavy?
We saw peaks of 3Gb traffic on the newest cmogstore when marking one 
host dead...
> I just tested 10s parallel PUTs and 100s of parallel GETs
> without any problems.