From: Troex Nevelin <list@mrtech.ru>
To: unicorn list <mongrel-unicorn@rubyforge.org>
Subject: Re: Unicorn freezes, requests got stuck in the queue most likely
Date: Tue, 28 May 2013 21:56:38 +0300 [thread overview]
Message-ID: <1E724535-5C14-47BA-97C5-01E2E867FF93@mrtech.ru> (raw)
In-Reply-To: <op.wxskbj1maya692@earth.site>
> This worked perfectly for 2 years. Looks like after we upgraded to unicorn 4.4, the normal requests started to get stuck in the queue. That happens randomly, several times per day. When that happens, requests wait for up to 7 seconds to be served. At that time most or all of the workers are available and not doing anything. Unicorn restart fixes the problem.
Monitoring the queue is important and raindrops is all about it,
I have a small nice munin script to keep track on it:
https://github.com/troex/raindrops-watcher-munin
Live example with 8 workers:
http://mon.melkov.net/PeterHost-RU/spb1.melkov.net/raindrops_watcher.html
If your unicorn queue gets higher than something is probably blocking
the app, in my example you can notice on daily graph that at the each
hour beginning it gets a small spike - that's rake task rushing DB, as
a result app becoming slower and the queue grows which can lead to that
clients wait 3-4 seconds while rails log resulting only with 50-100ms
higher average response times.
>From my experience I never had problems between nginx and unicorn,
if it gets stuck than 99% it's some blocker in the app.
As Eric said the harder to trace issues are when worker dies and you
don't get logging from rails app, for that case take a look at previous
post about that problem that helped me to find out the problem in my app:
http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/1269
Also request-log-analytics and new_relic is good way to start tracing
blocks and slowdowns in the app. And raindrops if great for monitoring
the queue before requests get processed by the app.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
prev parent reply other threads:[~2013-05-28 19:03 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-28 11:17 Unicorn freezes, requests got stuck in the queue most likely Alexander Dymo
2013-05-28 17:20 ` Eric Wong
2013-05-28 18:13 ` Alexander Dymo
2013-05-30 0:12 ` Eric Wong
2013-05-28 18:56 ` Troex Nevelin [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/unicorn/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1E724535-5C14-47BA-97C5-01E2E867FF93@mrtech.ru \
--to=list@mrtech.ru \
--cc=mongrel-unicorn@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).