Re: Heavy load

[Eric Wong <e@80x24.org>]

Arkadi Colson <arkadi@smartbit.be> wrote:
> Hi
> 
> We are running cmogstored version 1.7.1 compiled and running on Debian 
> stretch. How do I catch backtrace and debugging symbols?

OK, on Linux, you need to make sure core file size is big
enough to hold the entire core dump.  I normally just use
"unlimited" since I have more free disk space than RAM.

Via init scripts, use "ulimit -c unlimited" in the shell
script before spawning cmogstored (from the same shell
process).

With systemd, it's the "LimitCORE=" property (see systemd.exec
manpage).

You can check "Max core file size" in the
"/proc/$PID_OF_CMOGSTORED/limits" file once cmogstored is running
to verify.

AFAIK, core dumps go to the working directory of the process by
default.  For daemonized processes like cmogstored might try to
write to "/" (and fail due to lack of permissions).

I've used the following to get all my core dumps in
/var/tmp/core.$CRASHED_PID:

	echo /var/tmp/core >/proc/sys/kernel/core_pattern
	echo 1 >/proc/sys/kernel/core_uses_pid

But modern kernels have more options for piping core dumps
to processes and such...

cmogstored builds by default with '-ggdb3' in the CFLAGS, which
is the most verbose debug info.  Some installers strip the debug
info, unfortunately.  Running "file /path/to/cmogstored" will
tell you if there's debug_info and if it's stripped
(probably easiest to install the unstripped one)

Once you have a core dump from a crashed process, you can run
gdb against it with an unstripped binary to get lots of info.

  gdb /path/to/cmogstored /var/tmp/core.$CRASHED_PID

And then you can type "bt" to get a backtrace.
You can also use "t 1", "t 2", etc to switch between
threads and get the backtrace of other threads and a
host of other commands.

I'm not an expert in using gdb, as I can usually figure
everything out from just the backtrace.

Hopefully that's enough to get you the info I need to
fix the problem.  But there's a lot of info out there
on debugging C programs on there (and I'm tired atm and
about to pass out).