Re: Heavy load

[Arkadi Colson <arkadi@smartbit.be>]

On 18/12/19 18:58, Eric Wong wrote:
> Arkadi Colson <arkadi@smartbit.be> wrote:
>> On 17/12/19 20:42, Eric Wong wrote:
>>> Arkadi Colson <arkadi@smartbit.be> wrote:
>>>>> Any idea? If you need more info, please just ask!
>>>> We are having about 192 devices spread over about 23 cmogstored hosts.
>>>> Each device is one disk with one partition...
>>>>> How many "/devXYZ" devices do you have? Are they all
>>>>> on different partitions?
>>> OK, thanks. I've only got a single host nowadays with 3
>>> rotational HDD. Most I ever had was 20 rotational HDD on a
>>> host but that place is out-of-business.
>>>
>>> Since your build did not include -ggdb3 with debug_info by default;
>>> I wonder if there's something broken in your build system or
>>> build scripts... Which compiler are you using?
>>>
>>> Can you share the output of "ldd /path/to/cmogstored" ?
>> root@mogstore:~# ldd /usr/bin/cmogstored
>>     linux-vdso.so.1 (0x00007fff6584b000)
>>     libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>> (0x00007f634a7f2000)
>>     libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f634a453000)
>>     /lib64/ld-linux-x86-64.so.2 (0x00007f634ac50000)
>>
>>> Since this is Linux, you're not using libkqueue, are you?
> I was hoping for a simple explanation with libkqueue being
> the culprit, but that's not it.
>
> Have you gotten a better backtrace with debug_info? (-ggdb3)
No, no other crashes so far...
>>> Also, which Linux kernel is it?
>> In fact it's a clean debian stretch installation with htis kernel:
>>
>> root@mogstore:~# uname -a
>> Linux mogstore 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2
>> (2019-11-11) x86_64 GNU/Linux
> OK, I don't think there's known problems with that kernel;
> cmogstored is pretty sensitive to OS bugs and bugs in
> emulation layers like libkqueue.
>
>>> Are you using "server aio_threads =" via mgmt interface?
>> I don't think so. How can I verify this?
> You'd have something connecting to the mgmt port (7501 in your
> case) and setting "server aio_threads = $NUMBER".
I'm not sure I'm getting this setting. How can I set or get this 
setting? We did not configure anything like this anywhere. So I presume 
it uses the default setting?
>
>>> Are you using the undocumented -W/--worker-processes or
>>> -M (multi-config) option(s?)
>> I don't think so: Config looks like this:
>>
>> httplisten  = 0.0.0.0:7500
>> mgmtlisten  = 0.0.0.0:7501
>> maxconns    = 10000
>> docroot     = /var/mogdata
>> daemonize   = 1
>> server      = none
> OK
>
>>> Is your traffic read-heavy or write-heavy?
>> We saw peaks of 3Gb traffic on the newest cmogstore when marking one
>> host dead...
> Are you able to reproduce the problem on a test instance with
> just cmogstored?
> (no need for full MogileFS instance, just PUT/GET over HTTP).
I had no time yet to try and reproduce the problem in another 
environment. We try to do it as soon as possible...
> Also, are you on SSD or HDD? Lower latency of SSD could trigger
> some bugs. The design is for high-latency HDD, but it ought to
> work well with SSD, too. I haven't tested with SSD, much,
> unfortunately.

We only use HDD, no SSD

I will come back to you with more information after the next crash with 
more debug info. For me it's OK to put the case on hold for now... By 
the way, thanks a lot already for your help!

BR
Arkadi