Virtual Memory Usage

Virtual Memory Usage

[Ben Somers]

Hi,

I've been testing unicorn as a replacement for passenger in my
company's Rails apps, and generally am very much liking it, and will
probably go through with the switch (our deployment times are killing
us). I've been running tests on with jmeter against a demo EC2
m1.xlarge running Ubuntu 9.10 (yes, we know it's out od date), with
15GB RAM, 4x2GHz CPU.

The server is behaving fine, with most of the unicorn workers sitting
at about 280MB resident memory, comparable to passenger's workers. But
I've been testing how many workers to run with, and I've noticed that
once I get above three workers, a couple of them get ballooning
virtual memory, jumping up to a little over 2GB. I initially thought
that the server was overloaded and swapping, but swap usage is still
0, and I get this behavior at over 50% free memory. It doesn't seem to
be causing performance problems, but I was wondering if anyone else
has observed this or similar behavior? I'm trying to decide if it's
normal or an item for concern.

-ben
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Virtual Memory Usage

[Eric Wong]

Ben Somers <somers.ben@gmail.com> wrote:
> m1.xlarge running Ubuntu 9.10 (yes, we know it's out od date), with
> 15GB RAM, 4x2GHz CPU.
> 
> The server is behaving fine, with most of the unicorn workers sitting
> at about 280MB resident memory, comparable to passenger's workers. But
> I've been testing how many workers to run with, and I've noticed that
> once I get above three workers, a couple of them get ballooning
> virtual memory, jumping up to a little over 2GB. I initially thought
> that the server was overloaded and swapping, but swap usage is still
> 0, and I get this behavior at over 50% free memory. It doesn't seem to
> be causing performance problems, but I was wondering if anyone else
> has observed this or similar behavior? I'm trying to decide if it's
> normal or an item for concern.

Since you're on Linux, you can run "pmap $PID" to see how chunks of
virtual memory are used by any process.

If a chunk is backed by a regular file, it hardly matters, the kernel
will share that with other processes and won't swap that memory (since
it's already backed by a regular file).

Big "[ anon ]" chunks you see in pmap output are usually not shared, so
potentially more problematic.  It depends on your application and the
libraries it uses, of course.  Some libraries will share anonymous
memory between processes (raindrops does this for example, but only for
small integer counters).

I've seen Unicorn processes using TokyoCabinet and TDB[1] with database
files many gigabytes large with large VM sizes to match.  I also know
both of those database libraries cooperate with the VM subsystem so I
had nothing to worry about as far as memory usage goes :)



In case you (or somebody else) does notice high RSS and you're using the
stock glibc malloc, try setting MALLOC_MMAP_THRESHOLD_=131072 or
similar.  The author of jemalloc once blogged about this behavior with
glibc malloc, the title was "Mr.  Malloc gets schooled" or something
along those lines.  Making the malloc implementation use mmap() more
to reduce memory usage may hurt performance, though.

I always do incremental processing on large datasets so that helps
me avoid issues with unchecked/unpredictable memory growth due to
malloc behavior.


[1] http://bogomips.org/ruby-tdb/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Virtual Memory Usage

[Ben Somers]

Thank you! I did some pmapping, and it looks like they are grabbing a
big [anon] chunk, but it's shared between all the workers for a given
master (both the low- and high-memory workers). Still no swapping.
I'll proceed with my live testing (we're going to run just one
webserver on unicorn for a few weeks), and see how it goes.

-ben

On Thu, Nov 10, 2011 at 3:50 PM, Eric Wong <normalperson@yhbt.net> wrote:
>
> Ben Somers <somers.ben@gmail.com> wrote:
> > m1.xlarge running Ubuntu 9.10 (yes, we know it's out od date), with
> > 15GB RAM, 4x2GHz CPU.
> >
> > The server is behaving fine, with most of the unicorn workers sitting
> > at about 280MB resident memory, comparable to passenger's workers. But
> > I've been testing how many workers to run with, and I've noticed that
> > once I get above three workers, a couple of them get ballooning
> > virtual memory, jumping up to a little over 2GB. I initially thought
> > that the server was overloaded and swapping, but swap usage is still
> > 0, and I get this behavior at over 50% free memory. It doesn't seem to
> > be causing performance problems, but I was wondering if anyone else
> > has observed this or similar behavior? I'm trying to decide if it's
> > normal or an item for concern.
>
> Since you're on Linux, you can run "pmap $PID" to see how chunks of
> virtual memory are used by any process.
>
> If a chunk is backed by a regular file, it hardly matters, the kernel
> will share that with other processes and won't swap that memory (since
> it's already backed by a regular file).
>
> Big "[ anon ]" chunks you see in pmap output are usually not shared, so
> potentially more problematic.  It depends on your application and the
> libraries it uses, of course.  Some libraries will share anonymous
> memory between processes (raindrops does this for example, but only for
> small integer counters).
>
> I've seen Unicorn processes using TokyoCabinet and TDB[1] with database
> files many gigabytes large with large VM sizes to match.  I also know
> both of those database libraries cooperate with the VM subsystem so I
> had nothing to worry about as far as memory usage goes :)
>
>
>
> In case you (or somebody else) does notice high RSS and you're using the
> stock glibc malloc, try setting MALLOC_MMAP_THRESHOLD_=131072 or
> similar.  The author of jemalloc once blogged about this behavior with
> glibc malloc, the title was "Mr.  Malloc gets schooled" or something
> along those lines.  Making the malloc implementation use mmap() more
> to reduce memory usage may hurt performance, though.
>
> I always do incremental processing on large datasets so that helps
> me avoid issues with unchecked/unpredictable memory growth due to
> malloc behavior.
>
>
> [1] http://bogomips.org/ruby-tdb/
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying