* Garbage collection outside of request cycle? @ 2010-05-06 20:29 Luke Melia 2010-05-06 20:57 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Luke Melia @ 2010-05-06 20:29 UTC (permalink / raw) To: mongrel-unicorn I've been analyzing our Unicorn-powered Rails app's performance, and have found that garbage collection is a big factor in slow requests. In the interest of avoiding those performance hits while handling requests, would it be possible to have a unicorn worker run garbage collection after handling a request and before waiting for the next one? Would this be a good idea? Cheers, Luke _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Garbage collection outside of request cycle? 2010-05-06 20:29 Garbage collection outside of request cycle? Luke Melia @ 2010-05-06 20:57 ` Eric Wong 2010-05-06 21:12 ` Eric Wong 2010-05-06 23:17 ` Luke Melia 0 siblings, 2 replies; 6+ messages in thread From: Eric Wong @ 2010-05-06 20:57 UTC (permalink / raw) To: unicorn list; +Cc: Luke Melia Luke Melia <luke@lukemelia.com> wrote: > I've been analyzing our Unicorn-powered Rails app's performance, and > have found that garbage collection is a big factor in slow requests. > > In the interest of avoiding those performance hits while handling > requests, would it be possible to have a unicorn worker run garbage > collection after handling a request and before waiting for the next > one? Would this be a good idea? Hi Luke, I made this for one heavyweight app a while back. I guess I should throw this into the examples section, but it won't be the default since it hurts simpler applications that don't generate much garbage. ==> big_app_gc.rb <== # This shouldn't hurt overall performance as long as the server cluster # is at <=50% CPU capacity, and improves the performance of most memory # intensive requests. This serves to improve _client-visible_ # performance (possibly at the cost of overall performance). # # We'll call GC after each request is been written out to the socket, so # the client never sees the extra GC hit it. It's ideal to call the GC # inside the HTTP server (vs middleware or hooks) since the stack is # smaller at this point, so the GC will both be faster and more # effective at releasing unused memory. # # This monkey patch is _only_ effective for applications that use a lot # of memory, and will hurt simpler apps/endpoints that can process # multiple requests before incurring GC. class Unicorn::HttpServer REQ = Unicorn::HttpRequest::REQ alias _process_client process_client undef_method :process_client def process_client(client) _process_client(client) REQ.clear GC.start end end if defined?(Unicorn) -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Garbage collection outside of request cycle? 2010-05-06 20:57 ` Eric Wong @ 2010-05-06 21:12 ` Eric Wong 2010-05-06 23:17 ` Luke Melia 1 sibling, 0 replies; 6+ messages in thread From: Eric Wong @ 2010-05-06 21:12 UTC (permalink / raw) To: unicorn list; +Cc: Luke Melia Eric Wong <normalperson@yhbt.net> wrote: > I guess I should throw this into the examples section Added a few more comments, but the code is still the same: http://unicorn.bogomips.org/examples/big_app_gc.rb http://git.bogomips.org/cgit/unicorn.git/commit/?id=510a48dafc5f7e2cb618d785885395c79570821c -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Garbage collection outside of request cycle? 2010-05-06 20:57 ` Eric Wong 2010-05-06 21:12 ` Eric Wong @ 2010-05-06 23:17 ` Luke Melia 2010-05-13 20:29 ` Luke Melia 1 sibling, 1 reply; 6+ messages in thread From: Luke Melia @ 2010-05-06 23:17 UTC (permalink / raw) To: unicorn list On May 6, 2010, at 4:57 PM, Eric Wong wrote: > I made this for one heavyweight app a while back. > > ==> big_app_gc.rb <== > # This shouldn't hurt overall performance as long as the server cluster > # is at <=50% CPU capacity, and improves the performance of most memory > # intensive requests. This serves to improve _client-visible_ > # performance (possibly at the cost of overall performance). This is exactly the tradeoff I'm looking for. Our app has a large footprint. Thanks for the quick response. I'll report back when I have some data. Cheers, Luke _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Garbage collection outside of request cycle? 2010-05-06 23:17 ` Luke Melia @ 2010-05-13 20:29 ` Luke Melia 2010-05-14 19:02 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Luke Melia @ 2010-05-13 20:29 UTC (permalink / raw) To: Luke Melia; +Cc: unicorn list > On May 6, 2010, at 4:57 PM, Eric Wong wrote: > >> ==> big_app_gc.rb <== >> # This shouldn't hurt overall performance as long as the server cluster >> # is at <=50% CPU capacity, and improves the performance of most memory >> # intensive requests. This serves to improve _client-visible_ >> # performance (possibly at the cost of overall performance). I thought the list might be interested in how this worked for us. I applied the patch to execute GC between each request. I'm using NewRelic to measure the app. Prior to the patch our, we spent about 25% of our aggregate time serving a request in GC and our application was running at around 20-30% CPU load. Our running app shows up as using ~330MB of memory. Applying the patch cut the time spent in GC time to nearly zero and as predicted CPU spiked. Client-perceived responsiveness increased as well. Unfortunately, during our busiest time of the day, CPU load got so high that nginx locked up, so we rolled back the patch. I made a simple change to execute GC once every 5 requests and applied it again. Aggregate time spent in GC reduced to out 10% of total request time. This resulted in a bout a 25% overall improvement in client response time. Big win! CPU maxes out at about 80% with this configuration.. One other thing I did was force GC to execute before_fork, on the theory that with COW, we would want to fork in the tidiest state possible. I have not measured this on it's own to evaluate it's impact. Thanks again for the help and code on this, Eric. Considering how useful this is, perhaps unicorn should have an after_request hook, to avoid the need to monkey-patch? Cheers, Luke -- http://www.lukemelia.com/ _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Garbage collection outside of request cycle? 2010-05-13 20:29 ` Luke Melia @ 2010-05-14 19:02 ` Eric Wong 0 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2010-05-14 19:02 UTC (permalink / raw) To: Luke Melia; +Cc: unicorn list Luke Melia <luke@lukemelia.com> wrote: > > On May 6, 2010, at 4:57 PM, Eric Wong wrote: > > > >> ==> big_app_gc.rb <== # This shouldn't hurt overall performance as > >> long as the server cluster # is at <=50% CPU capacity, and improves > >> the performance of most memory # intensive requests. This serves > >> to improve _client-visible_ # performance (possibly at the cost of > >> overall performance). > > I thought the list might be interested in how this worked for us. I > applied the patch to execute GC between each request. I'm using > NewRelic to measure the app. Prior to the patch our, we spent about > 25% of our aggregate time serving a request in GC and our application > was running at around 20-30% CPU load. Our running app shows up as > using ~330MB of memory. Thanks for the feedback, Luke. Was the original 30% CPU load during peak traffic or normal traffic? > Applying the patch cut the time spent in GC time to nearly zero and as > predicted CPU spiked. Client-perceived responsiveness increased as > well. Unfortunately, during our busiest time of the day, CPU load got > so high that nginx locked up, so we rolled back the patch. Yikes, nginx locking up is rare. It's worth investigating and fixing that from the nginx side if you can reproduce it. > I made a simple change to execute GC once every 5 requests and applied > it again. Aggregate time spent in GC reduced to out 10% of total > request time. This resulted in a bout a 25% overall improvement in > client response time. Big win! CPU maxes out at about 80% with this > configuration.. How many Unicorn workers do you have per-core? I forgot to mention that you might want to run more workers to "hide" GC costs, something like a poor man's concurrent GC. > One other thing I did was force GC to execute before_fork, on the > theory that with COW, we would want to fork in the tidiest state > possible. I have not measured this on it's own to evaluate it's > impact. I doubt it'd help for anything other than the first <=5 requests that hit the worker process. > Thanks again for the help and code on this, Eric. Thank _you_ for actually being willing to run and report back on crazy experiments I come up with :) > Considering how useful this is, perhaps unicorn should have an > after_request hook, to avoid the need to monkey-patch? Given configurability requirements and the ability of such a feature to penalize some apps, I think making the monkey patch into middleware is a nice compromise. The GC.start in the below middleware runs at a slightly deeper stack level, meaning GC will scan more and reap less than the original monkey patch, but I doubt there'll be real difference if your app is already spending enough time in GC to be a problem. I've also pushed the following up to git://git.bogomips.org/unicorn.git Let me know if you get a chance to test it on your app in place of the monkey patch. >From 95b75a5043b34f39ece4f52befb4b3f884dfdd20 Mon Sep 17 00:00:00 2001 From: Eric Wong <normalperson@yhbt.net> Date: Fri, 14 May 2010 18:27:35 +0000 Subject: [PATCH] add Unicorn::OobGC middleware This middleware allows configurable out-of-band garbage collection outside of the normal request/response cycle. It offers configurable paths (to only GC on expensive actions) and intervals to limit GC frequency. It is only expected to work well with Unicorn, as it would hurt performance on single-threaded servers if they have keepalive enabled. Obviously this does not work well for multi-threaded or evented servers that serve multiple clients at once. --- lib/unicorn/oob_gc.rb | 58 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 58 insertions(+), 0 deletions(-) create mode 100644 lib/unicorn/oob_gc.rb diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb new file mode 100644 index 0000000..8dc4dcf --- /dev/null +++ b/lib/unicorn/oob_gc.rb @@ -0,0 +1,58 @@ +# -*- encoding: binary -*- +module Unicorn + + # Run GC after every request, after closing the client socket and + # before attempting to accept more connections. + # + # This shouldn't hurt overall performance as long as the server cluster + # is at <50% CPU capacity, and improves the performance of most memory + # intensive requests. This serves to improve _client-visible_ + # performance (possibly at the cost of overall performance). + # + # We'll call GC after each request is been written out to the socket, so + # the client never sees the extra GC hit it. + # + # This middleware is _only_ effective for applications that use a lot + # of memory, and will hurt simpler apps/endpoints that can process + # multiple requests before incurring GC. + # + # This middleware is only designed to work with Unicorn, as it harms + # keepalive performance. + # + # Example (in config.ru): + # + # require 'unicorn/oob_gc' + # + # # GC ever two requests that hit /expensive/foo or /more_expensive/foo + # # in your app. By default, this will GC once every 5 requests + # # for all endpoints in your app + # use Unicorn::OobGC, 2, %r{\A/(?:expensive/foo|more_expensive/foo)} + class OobGC < Struct.new(:app, :interval, :path, :nr, :env, :body) + + def initialize(app, interval = 5, path = %r{\A/}) + super(app, interval, path, interval) + end + + def call(env) + status, headers, self.body = app.call(self.env = env) + [ status, headers, self ] + end + + def each(&block) + body.each(&block) + end + + # in Unicorn, this is closed _after_ the client socket + def close + body.close if body.respond_to?(:close) + + if path =~ env['PATH_INFO'] && ((self.nr -= 1) <= 0) + self.nr = interval + self.body = nil + env.clear + GC.start + end + end + + end +end -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-05-14 19:02 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-05-06 20:29 Garbage collection outside of request cycle? Luke Melia 2010-05-06 20:57 ` Eric Wong 2010-05-06 21:12 ` Eric Wong 2010-05-06 23:17 ` Luke Melia 2010-05-13 20:29 ` Luke Melia 2010-05-14 19:02 ` Eric Wong
Code repositories for project(s) associated with this public inbox https://yhbt.net/unicorn.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).