Re: Segfaults on http_close?

[Eric Wong <e@80x24.org>]

Xiao Yu <xyu@automattic.com> wrote:
> Thanks for the quick response! Sorry about the delay but I ran into a
> couple issues (sorry kinda learning gdb and compiling binaries in
> general on the go here) and have not been able to capture more useful
> logs yet as crashes have seem to have slowed / stopped since
> recompiling and reloading. For the record I recompiled cmogstored with
> the newer RH `devtoolset-9-toolchain` (9.1) and it has not crash
> since.

No worries.  Based on the toolchain change being a success, I
more strongly suspect the tiny patch I sent you guys will fix
the problem with all compilers.

> Also sorry about the lack of useful logs in my initial message but
> neither kernel logs nor messages contained anything interesting around
> the segfaults. Making matters worse, we didn't consistently reload
> cmogstored as various versions of the compiled binary was installed
> across the cluster and didn't really save the debugging symbols from
> each of the compilations so can't really reply with a more useful
> stack trace with the current core dumps.

That's fine.  One thing is I suggest is NOT using --daemonize/-d
flag and instead rely on systemd or something similar that can
capture stderr.

The --daemonize/-d flag (unfortunately) matches Perlbal
mogstored behavior, which causes stderr to be unconditionally
redirected to /dev/null and hides some errors.

> For what it's worth, we also run another cluster with 1.7.3 that we've
> upgraded over the years and have never seen this issue. Those nodes
> are on a different distro (Debian) if that makes any difference.

Good to know and thanks for the data point.  I suspect it
depends on the compiler version/settings, and maybe other
versions of gcc/clang shipped with Debian will still trigger the
problem without my proposed patch to use default stack size.

Thanks again for the report and will look forward to more info
whenever you can provide it.