Why Unicorn master process memory increasing

Why Unicorn master process memory increasing

[sumit nagariya]

Hi,

I have a ruby service running with unicorn which spawns 20 child
worker. When I start my service, unicorn master starts with approx 520
MB of memory and then master spawn 20 child worker. My service runs
perfectly for 7-8 day, but gradually unicorn master memory is kept
increasing and goes up to 1.3 GB, which apparently is causing OOM
error when unicorn master trying to fork a new child (which now need
1.3 GB of memory to fork new child)and due to increased memory
footprint, memory was not available.

So my concern here is,

As per my understanding, Unicorn master doesn't serve any request, its
sole purpose is to spawn missing child then why master process memory
is increasing?
If there is any memory leak in my app then it should impact only child
workers processes, not master process. I am not able to understand why
my master process memory is increasing?

Here is my server config:

Ruby: 2.1.2

Unicorn: 4.6.3

Unicorn worker count: 20

OS: Debian GNU/Linux 8, Jessie

RAM: 24 GB


-- 
Thanks & Regards
Sumit Nagariya

Re: Why Unicorn master process memory increasing

[Hleb Valoshka]

Hi!

On 4/11/18, sumit nagariya <sumit.nagariya@gmail.com> wrote:
> Here is my server config:
> Ruby: 2.1.2
> Unicorn: 4.6.3
> Unicorn worker count: 20
> OS: Debian GNU/Linux 8, Jessie
> RAM: 24 GB

I believe it's hard to say something without your unicorn configuration file.

Re: Why Unicorn master process memory increasing

[Eric Wong]

Hleb: remember to reply-all, we will never require subscription
to post to this list, so it's likely sumit never saw it.

Hleb Valoshka <375gnu@gmail.com> wrote:
> I believe it's hard to say something without your unicorn configuration file.

sumit: what Hleb asked...

I suspect "preload_app true" in your config is loading something
in your app which is using up your memory.  Sometimes there's
monitoring threads which might fill a queue up or something.

Anything in error logs?  Failing that, strace (on Linux) might
tell you what's going on at the error level.

But yeah, the master process shouldn't be doing anything besides
listening to signals and respawning failed workers.

If you want to check if there's monitoring threads on Linux
systems with /proc mounted, you can see how many threads the
master has:

	ls /proc/$PID_OF_MASTER/task/

There should be two entries for Ruby 1.9-2.5(*), one being the
master PID and one for the timer-thread.  If you have more,
then that's probably some thread doing stuff in the
background...

(*) _maybe_ we can get rid of timer-thread for 2.6...

Re: Why Unicorn master process memory increasing

[sumit nagariya]

Hi Eric,

Sorry for my late response, somehow missed your mail. Just to update I
managed to fix the issue, it was related to GC. We were using
unciron's oob_gc to control gc execution. Although we had correct
configuration for oob_gc but somehow it was not allowing GC to run. I
checked GC stat and found that major & minor GC count is consistent
over the time. I removed oob_gc and now using gctools gem. With
gctools gem master memory is consistent to ~400MB.

Thanks a lot for your help.

Regards,
Sumit

On Thu, Apr 19, 2018 at 8:15 AM, Eric Wong <e@80x24.org> wrote:
> Hleb: remember to reply-all, we will never require subscription
> to post to this list, so it's likely sumit never saw it.
>
> Hleb Valoshka <375gnu@gmail.com> wrote:
>> I believe it's hard to say something without your unicorn configuration file.
>
> sumit: what Hleb asked...
>
> I suspect "preload_app true" in your config is loading something
> in your app which is using up your memory.  Sometimes there's
> monitoring threads which might fill a queue up or something.
>
> Anything in error logs?  Failing that, strace (on Linux) might
> tell you what's going on at the error level.
>
> But yeah, the master process shouldn't be doing anything besides
> listening to signals and respawning failed workers.
>
> If you want to check if there's monitoring threads on Linux
> systems with /proc mounted, you can see how many threads the
> master has:
>
>         ls /proc/$PID_OF_MASTER/task/
>
> There should be two entries for Ruby 1.9-2.5(*), one being the
> master PID and one for the timer-thread.  If you have more,
> then that's probably some thread doing stuff in the
> background...
>
> (*) _maybe_ we can get rid of timer-thread for 2.6...



-- 
Thanks & Regards
Sumit Nagariya