Re: Segfaults on http_close?

[Xiao Yu <xyu@automattic.com>]

On Wed, Jan 20, 2021 at 8:57 AM Eric Wong <e@80x24.org> wrote:
>
> Xiao Yu <xyu@automattic.com> wrote:
> > Thanks for the quick response! Sorry about the delay but I ran into a
> > couple issues (sorry kinda learning gdb and compiling binaries in
> > general on the go here) and have not been able to capture more useful
> > logs yet as crashes have seem to have slowed / stopped since
> > recompiling and reloading. For the record I recompiled cmogstored with
> > the newer RH `devtoolset-9-toolchain` (9.1) and it has not crash
> > since.
>
> No worries.  Based on the toolchain change being a success, I
> more strongly suspect the tiny patch I sent you guys will fix
> the problem with all compilers.

That's good to know, would it help if I applied the patch and then
compiled with the older compiler to check?

> > Also sorry about the lack of useful logs in my initial message but
> > neither kernel logs nor messages contained anything interesting around
> > the segfaults. Making matters worse, we didn't consistently reload
> > cmogstored as various versions of the compiled binary was installed
> > across the cluster and didn't really save the debugging symbols from
> > each of the compilations so can't really reply with a more useful
> > stack trace with the current core dumps.
>
> That's fine.  One thing is I suggest is NOT using --daemonize/-d
> flag and instead rely on systemd or something similar that can
> capture stderr.
>
> The --daemonize/-d flag (unfortunately) matches Perlbal
> mogstored behavior, which causes stderr to be unconditionally
> redirected to /dev/null and hides some errors.

I noticed the lack of verbosity / logs :) so right now we're running
daemonized but also using syslog to capture logs, it's still pretty
quiet overall with nothing around when the segfaults happened and
pretty much only logs about premature EOFs when timeouts happen and
things like the following when we reload cmogstored:

```
2021-01-14T15:04:15.888261-05:00 host cmogstored[44351]: upgrade
spawned PID:27575
2021-01-14T15:04:15.891551-05:00 host cmogstored[27575]: inherited
0.0.0.0:7501 on fd=4
2021-01-14T15:04:15.891564-05:00 host cmogstored[27575]: inherited
0.0.0.0:7500 on fd=5
```

Would non-daemonized mode and capturing stderr provide more verbosity?

Thanks again, Xiao