* Re: Unused Unicorn processes
[not found] <mailman.0.1345509654.31187.mongrel-unicorn@rubyforge.org>
@ 2012-08-21 0:44 ` Konstantin Gredeskoul
2012-08-21 9:11 ` Eric Wong
0 siblings, 1 reply; 3+ messages in thread
From: Konstantin Gredeskoul @ 2012-08-21 0:44 UTC (permalink / raw)
To: unicorn list
Greetings!
I have a question on optimal # of unicorn worker processes.
We are running Unicorn 4.3.1 + Rails 3.2.6 (without threading), on
ruby 1.9.3-p194, hosted on SmartOS/Joyent.
At the moment, unicorns are configured to start 30 worker processes. I
know this is a lot, and I am going to reduce this number. But in
trying to figure out what is a more appropriate number of workers to
run, I noticed something interesting that I couldn't explain.
If I look at the process table on each machine (see top output below),
I notice that some unicorn processes are heavily used (and have
accumulated longer CPU times, as well as have grown their RAM usage
since boot time), but other processes (at the bottom of the top
output) appear to potentially not having been used at all. There are
several processes with RSS size of 143Mb, which I believe is unicorn
size before it processes any requests.
What I am gathering from this, is that only 16 unicorn processes are
actually processing requests, while the rest are just sitting there
idle.
Is this expected behavior?
This Joyent SmartMachine can burst up to 8 cores. Given that our web
requests spend only 80% of their time in ruby, I figured we could run
10 unicorn processes for maximum efficiency. However seeing that 16
are actually used I am curious whether 16 is actually a better number.
Any suggestions appreciated!
load averages: 2.07, 1.97, 2.09; up 14+01:56:
58 processes: 53 sleeping, 5 on cpu
CPU states: 77.1% idle, 18.2% user, 4.7% kernel, 0.0% iowait, 0.
Memory: 8192M phys mem, 3157M free mem, 16G total swap, 11G free sw
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
24800 wanelo 18 59 0 337M 313M sleep 17:21 2.05% ruby
24788 wanelo 18 59 0 223M 200M cpu/14 16:34 3.18% ruby
24787 wanelo 18 59 0 221M 197M sleep 14:55 1.89% ruby
24776 wanelo 18 59 0 217M 193M cpu/2 18:17 3.45% ruby
24804 wanelo 18 59 0 216M 192M cpu/8 7:54 0.33% ruby
24785 wanelo 18 59 0 211M 187M sleep 14:33 0.75% ruby
24783 wanelo 10 59 0 209M 185M sleep 16:09 2.16% ruby
24775 wanelo 18 59 0 205M 181M sleep 5:24 0.03% ruby
24777 wanelo 10 59 0 205M 181M cpu/11 3:53 1.54% ruby
24799 wanelo 18 59 0 204M 179M sleep 0:42 0.00% ruby
24803 wanelo 18 59 0 202M 177M sleep 2:28 0.35% ruby
24802 wanelo 3 59 0 195M 168M sleep 1:45 0.01% ruby
24778 wanelo 3 59 0 192M 165M sleep 1:06 0.01% ruby
24801 wanelo 3 59 0 189M 162M sleep 0:50 0.01% ruby
24798 wanelo 3 59 0 188M 161M sleep 0:30 0.00% ruby
24797 wanelo 3 59 0 187M 159M sleep 0:19 0.00% ruby
--- unicorns below this line do not appear to be used
24795 wanelo 3 59 0 173M 142M sleep 0:08 0.00% ruby
24792 wanelo 3 59 0 172M 142M sleep 0:07 0.00% ruby
24779 wanelo 3 59 0 172M 143M sleep 0:05 0.00% ruby
24796 wanelo 3 59 0 172M 142M sleep 0:12 0.00% ruby
24790 wanelo 3 59 0 172M 142M sleep 0:06 0.00% ruby
24793 wanelo 3 59 0 172M 142M sleep 0:06 0.00% ruby
24781 wanelo 3 59 0 172M 142M sleep 0:05 0.00% ruby
24791 wanelo 3 59 0 172M 142M sleep 0:04 0.00% ruby
24794 wanelo 3 58 0 172M 142M sleep 0:10 0.00% ruby
24784 wanelo 3 59 0 172M 142M sleep 0:04 0.00% ruby
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Unused Unicorn processes
2012-08-21 0:44 ` Unused Unicorn processes Konstantin Gredeskoul
@ 2012-08-21 9:11 ` Eric Wong
2012-08-22 18:16 ` Konstantin Gredeskoul
0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2012-08-21 9:11 UTC (permalink / raw)
To: unicorn list
Konstantin Gredeskoul <kig@wanelo.com> wrote:
> Greetings!
>
> I have a question on optimal # of unicorn worker processes.
>
> We are running Unicorn 4.3.1 + Rails 3.2.6 (without threading), on
> ruby 1.9.3-p194, hosted on SmartOS/Joyent.
>
> At the moment, unicorns are configured to start 30 worker processes. I
> know this is a lot, and I am going to reduce this number. But in
> trying to figure out what is a more appropriate number of workers to
> run, I noticed something interesting that I couldn't explain.
>
> If I look at the process table on each machine (see top output below),
> I notice that some unicorn processes are heavily used (and have
> accumulated longer CPU times, as well as have grown their RAM usage
> since boot time), but other processes (at the bottom of the top
> output) appear to potentially not having been used at all. There are
> several processes with RSS size of 143Mb, which I believe is unicorn
> size before it processes any requests.
>
> What I am gathering from this, is that only 16 unicorn processes are
> actually processing requests, while the rest are just sitting there
> idle.
>
> Is this expected behavior?
It's not _unexpected_ if your load is low. The OS scheduler does the
load balancing itself as unicorn uses non-blocking accept(). The
scheduler may choose busier processes because they're likely to be
hotter in cache/memory.
So you're probably getting better performance than if you had perfectly
balanced traffic.
Another possible explanation is something is killing some workers
(timeout, or crash buts, check stderr logs to confirm). But even in a
situation where workers never die, it's perfectly normal to see workers
that never/rarely serve traffic.
> This Joyent SmartMachine can burst up to 8 cores. Given that our web
> requests spend only 80% of their time in ruby, I figured we could run
> 10 unicorn processes for maximum efficiency. However seeing that 16
> are actually used I am curious whether 16 is actually a better number.
It depends what your app is waiting on.
The only rule is your machine should not be swap thrashing under
peak load.
Extra workers won't be detrimental as long as you:
1) have enough memory
2) have enough backend resources (e.g DB connections)
I know of deployments that run 30 processes/core because the app spends
much time waiting on slow HTTP/DB requests[1]
But if your app is completely bound by resources local to a machine (no
external DB/HTTP/memcached/redis/etc requests), then a 1:1 worker:core
(or even 1:1 worker:disk) relationship will work, too.
[1] Arguably a multi-threaded or non-blocking server such as Rainbows!
would be most efficient of machine resources if you're waiting on HTTP
calls, but the developers decided human time was more important and
did not want to worry about thread-safety.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Unused Unicorn processes
2012-08-21 9:11 ` Eric Wong
@ 2012-08-22 18:16 ` Konstantin Gredeskoul
0 siblings, 0 replies; 3+ messages in thread
From: Konstantin Gredeskoul @ 2012-08-22 18:16 UTC (permalink / raw)
To: unicorn list
Thanks Eric! Appreciate well thought out answers.
We are actually using Rainbows also, in an project where long
server-side HTTP calls are part of the application design and are
necessary, and it's working out really well.
Our main web app mostly spends it's time in CPU cycles serving web
pages (about 80% of the request time), and 20% in database operations.
Thanks again
K
On Tue, Aug 21, 2012 at 2:11 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Konstantin Gredeskoul <kig@wanelo.com> wrote:
>> Greetings!
>>
>> I have a question on optimal # of unicorn worker processes.
>>
>> We are running Unicorn 4.3.1 + Rails 3.2.6 (without threading), on
>> ruby 1.9.3-p194, hosted on SmartOS/Joyent.
>>
>> At the moment, unicorns are configured to start 30 worker processes. I
>> know this is a lot, and I am going to reduce this number. But in
>> trying to figure out what is a more appropriate number of workers to
>> run, I noticed something interesting that I couldn't explain.
>>
>> If I look at the process table on each machine (see top output below),
>> I notice that some unicorn processes are heavily used (and have
>> accumulated longer CPU times, as well as have grown their RAM usage
>> since boot time), but other processes (at the bottom of the top
>> output) appear to potentially not having been used at all. There are
>> several processes with RSS size of 143Mb, which I believe is unicorn
>> size before it processes any requests.
>>
>> What I am gathering from this, is that only 16 unicorn processes are
>> actually processing requests, while the rest are just sitting there
>> idle.
>>
>> Is this expected behavior?
>
> It's not _unexpected_ if your load is low. The OS scheduler does the
> load balancing itself as unicorn uses non-blocking accept(). The
> scheduler may choose busier processes because they're likely to be
> hotter in cache/memory.
>
> So you're probably getting better performance than if you had perfectly
> balanced traffic.
>
> Another possible explanation is something is killing some workers
> (timeout, or crash buts, check stderr logs to confirm). But even in a
> situation where workers never die, it's perfectly normal to see workers
> that never/rarely serve traffic.
>
>> This Joyent SmartMachine can burst up to 8 cores. Given that our web
>> requests spend only 80% of their time in ruby, I figured we could run
>> 10 unicorn processes for maximum efficiency. However seeing that 16
>> are actually used I am curious whether 16 is actually a better number.
>
> It depends what your app is waiting on.
>
> The only rule is your machine should not be swap thrashing under
> peak load.
>
> Extra workers won't be detrimental as long as you:
> 1) have enough memory
> 2) have enough backend resources (e.g DB connections)
>
> I know of deployments that run 30 processes/core because the app spends
> much time waiting on slow HTTP/DB requests[1]
>
> But if your app is completely bound by resources local to a machine (no
> external DB/HTTP/memcached/redis/etc requests), then a 1:1 worker:core
> (or even 1:1 worker:disk) relationship will work, too.
>
>
>
> [1] Arguably a multi-threaded or non-blocking server such as Rainbows!
> would be most efficient of machine resources if you're waiting on HTTP
> calls, but the developers decided human time was more important and
> did not want to worry about thread-safety.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
--
Konstantin Gredeskoul
CTO :: Wanelo Inc
cell: (415) 265 1054
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-08-22 18:26 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <mailman.0.1345509654.31187.mongrel-unicorn@rubyforge.org>
2012-08-21 0:44 ` Unused Unicorn processes Konstantin Gredeskoul
2012-08-21 9:11 ` Eric Wong
2012-08-22 18:16 ` Konstantin Gredeskoul
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).