Peformance up - using OobGC & GC.disable

Peformance up - using OobGC & GC.disable

[secondlife]

Hi,
I'm working at Cookpad Inc.

- http://cookpad.com/

Cookpad is very famous cooking recipe service in Japan,
and largest rails application in Japan.

I applied GC.disable after fork, and a few changed Unicorn::OobGC.

# unicorn.conf.rb
 after_fork do |server, worker|
  GC.disable if RAILS_ENV == 'production'

# OobGC code
 def process_client(client)
  super(client) # Unicorn::HttpServer#process_client
  if OOBGC_PATH =~ OOBGC_ENV[PATH_INFO] && ((@@nr -= 1) <= 0)
    @@nr = OOBGC_INTERVAL
    OOBGC_ENV.clear
    disabled = GC.enable
    GC.start
    GC.disable if disabled
  end
 end

after applied,
application response time 130% fast!
and CPU usage to half!

- http://bit.ly/rhnVlg (response time)
- http://bit.ly/pgMN06 (CPU usage)

This is OobGC patch, apply please.
- http://bit.ly/roVLtZ (patch)

Also, my rails application have memory leaks ;(
I wrote process killing code because GC.disable create more memory leak.
- https://gist.github.com/1258681

------------------------------------------
Yuichi Tateno
https://github.com/hotchpotch
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Peformance up - using OobGC & GC.disable

[cliftonk]

That looks promising. I'm curious what our app's memory would climb to (at worst).

Clifton
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Peformance up - using OobGC & GC.disable

[Eric Wong]

secondlife <hotchpotch@gmail.com> wrote:
> after applied,
> application response time 130% fast!
> and CPU usage to half!
> 
> - http://bit.ly/rhnVlg (response time)
> - http://bit.ly/pgMN06 (CPU usage)

Thanks for the feedback!  I wonder if 1.9.3 with lazy sweep GC can
provide similar improvements (without needing OobGC or GC.disable,
even).

> This is OobGC patch, apply please.
> - http://bit.ly/roVLtZ (patch)

Thanks! Applied as c7ba76a21c5d00fb5c173cd6aa847442bbc652cb
with the following commit message:

From: Yuichi Tateno <hotchpotch@gmail.com>
Date: Mon, 3 Oct 2011 16:51:19 +0900
Subject: [PATCH] OobGC: force GC.start

[ew: we need to explicitly enable GC if it is disabled
 and respect applications that disable GC]

Acked-by: Eric Wong <normalperson@yhbt.net>

> I wrote process killing code because GC.disable create more memory leak.
> - https://gist.github.com/1258681

I think that could be done entirely as Unicorn-specific Rack middleware
and not hook into Unicorn internal methods like process_client.  Also,
the pid checks are unnecessary since that code never runs in the master.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Peformance up - using OobGC & GC.disable

[secondlife]

> Thanks! Applied as c7ba76a21c5d00fb5c173cd6aa847442bbc652cb> with the following commit message:
Thank you for applied!

> I think that could be done entirely as Unicorn-specific Rack middleware
> and not hook into Unicorn internal methods like process_client.  Also,
> the pid checks are unnecessary since that code never runs in the master.

Thank you for advice. I rewrite my code.
- https://gist.github.com/1258681

------------------------------------------
Yuichi Tateno
https://github.com/hotchpotch
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Peformance up - using OobGC & GC.disable

[Tatsuya Ono]

This solution is awesome. I applied the patch on our staging server
and I've confirmed a significant performance improve. Some
pages(especially complicated ones) responds like 4 times faster than
before.

However it seems that we hit a same issue Yuichi had. When we had an
access on particular page, I can see memory size of worker increases
5-10MB...

> Also, my rails application have memory leaks ;(
> I wrote process killing code because GC.disable create more memory leak.
> - https://gist.github.com/1258681

Beside I want to avoid to use this solution if possible...

I don't actually understand is why GC.disable solution could introduce
more memory leak. If I simplify the problem, the code is something
like bellow:

---------------
GC.disable
(do something)
GC.enable
GC.start
---------------

When the code block finishes, I expect that memory size should be
(almost) equal with the case GC is enabled at begging. But it doesn't
seems so from our experience.

Do anyone know why there could be significant difference on memory
usage because of timing of GC? It might be a question on Ruby rather
than Unicorn, though, I thought even just sharing my experience could
be worth to someone here.


Tatsuya Ono
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Peformance up - using OobGC & GC.disable

[Eric Wong]

Tatsuya Ono <ononoma@gmail.com> wrote:
> I don't actually understand is why GC.disable solution could introduce
> more memory leak. If I simplify the problem, the code is something
> like bellow:
> 
> ---------------
> GC.disable
> (do something)
> GC.enable
> GC.start
> ---------------
> 
> When the code block finishes, I expect that memory size should be
> (almost) equal with the case GC is enabled at begging. But it doesn't
> seems so from our experience.
> 
> Do anyone know why there could be significant difference on memory
> usage because of timing of GC? It might be a question on Ruby rather
> than Unicorn, though, I thought even just sharing my experience could
> be worth to someone here.

Basically, the free(3) function in the C standard library does not
guarantee memory is released back to the kernel (speed vs memory usage
tradeoff).

There was discussion of this on the usp.ruby mailing list starting at
Message-ID: 20110914234917.GA2480@dcvr.yhbt.net

usp.ruby archives are at http://bogomips.org/usp.ruby/archives/2011.mbox.gz
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Peformance up - using OobGC & GC.disable

[Tatsuya Ono]

Thanks Eric for the feedback.

I actually had read that email and I think I understand it. But what I
am experiencing seems a different story. Our rails app uses around
250MB memory usually. After applying this patch and calling
GC.disabled on after_fork, the usage of memory increases on every
request and goes up to 1GB easily.

However, yes, I must say that I need to test more carefully. Let me
come back later. I am going to have some stress test and monitor if
Unicorn introduces swapping on VM with this solution. Hopefully I can
do it tomorrow or later this week.


Tatsuya

On 10 October 2011 22:53, Eric Wong <normalperson@yhbt.net> wrote:
> Tatsuya Ono <ononoma@gmail.com> wrote:
>> I don't actually understand is why GC.disable solution could introduce
>> more memory leak. If I simplify the problem, the code is something
>> like bellow:
>>
>> ---------------
>> GC.disable
>> (do something)
>> GC.enable
>> GC.start
>> ---------------
>>
>> When the code block finishes, I expect that memory size should be
>> (almost) equal with the case GC is enabled at begging. But it doesn't
>> seems so from our experience.
>>
>> Do anyone know why there could be significant difference on memory
>> usage because of timing of GC? It might be a question on Ruby rather
>> than Unicorn, though, I thought even just sharing my experience could
>> be worth to someone here.
>
> Basically, the free(3) function in the C standard library does not
> guarantee memory is released back to the kernel (speed vs memory usage
> tradeoff).
>
> There was discussion of this on the usp.ruby mailing list starting at
> Message-ID: 20110914234917.GA2480@dcvr.yhbt.net
>
> usp.ruby archives are at http://bogomips.org/usp.ruby/archives/2011.mbox.gz
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Peformance up - using OobGC & GC.disable

[Tatsuya Ono]

Yes, you are right, Eric. The usage of memory stops increasing at a
certain point. Besides I do not see any significant page I/O with it.

I give the patch go for our live service without the UnicornKiller. I
will report if we experience any issues occurred in the wild.

Thanks Yuichi again for submitting the patch and sharing your knowledge.

By the way, I tested this with Rails 2.3/Ruby 1.8.7/FreeBSD 8.2


Tatsuya

On 11 October 2011 00:03, Tatsuya Ono <ononoma@gmail.com> wrote:
> Thanks Eric for the feedback.
>
> I actually had read that email and I think I understand it. But what I
> am experiencing seems a different story. Our rails app uses around
> 250MB memory usually. After applying this patch and calling
> GC.disabled on after_fork, the usage of memory increases on every
> request and goes up to 1GB easily.
>
> However, yes, I must say that I need to test more carefully. Let me
> come back later. I am going to have some stress test and monitor if
> Unicorn introduces swapping on VM with this solution. Hopefully I can
> do it tomorrow or later this week.
>
>
> Tatsuya
>
> On 10 October 2011 22:53, Eric Wong <normalperson@yhbt.net> wrote:
>> Tatsuya Ono <ononoma@gmail.com> wrote:
>>> I don't actually understand is why GC.disable solution could introduce
>>> more memory leak. If I simplify the problem, the code is something
>>> like bellow:
>>>
>>> ---------------
>>> GC.disable
>>> (do something)
>>> GC.enable
>>> GC.start
>>> ---------------
>>>
>>> When the code block finishes, I expect that memory size should be
>>> (almost) equal with the case GC is enabled at begging. But it doesn't
>>> seems so from our experience.
>>>
>>> Do anyone know why there could be significant difference on memory
>>> usage because of timing of GC? It might be a question on Ruby rather
>>> than Unicorn, though, I thought even just sharing my experience could
>>> be worth to someone here.
>>
>> Basically, the free(3) function in the C standard library does not
>> guarantee memory is released back to the kernel (speed vs memory usage
>> tradeoff).
>>
>> There was discussion of this on the usp.ruby mailing list starting at
>> Message-ID: 20110914234917.GA2480@dcvr.yhbt.net
>>
>> usp.ruby archives are at http://bogomips.org/usp.ruby/archives/2011.mbox.gz
>> _______________________________________________
>> Unicorn mailing list - mongrel-unicorn@rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
>> Do not quote signatures (like this one) or top post when replying
>>
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying