unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
* Is it possible to log better info in timeouts?
@ 2014-01-02  0:41 Carlos Peñas
  2014-01-02  1:31 ` Eric Wong
  0 siblings, 1 reply; 2+ messages in thread
From: Carlos Peñas @ 2014-01-02  0:41 UTC (permalink / raw)
  To: mongrel-unicorn

Hi!

Recently we had an issue with an external service for an app served with
unicorn. And some of our requests died with unicorn's timeout because a
not properly secured call to this external service. We had 20s
timeout but in production with ten worker and a big percent of urls
hitting external services 20 secons of timeout sooner or later brings
down the entire site.

in the logs when this hapens this line is left for debugging:

E, [2013-12-25T23:56:49.363426 #26324] ERROR -- : worker=2 PID:27956 timeout (21s > 20s), killing
E, [2014-12-25T23:56:49.396005 #26324] ERROR -- : reaped #<Process::Status: pid 27956 SIGKILL (signal 9)> worker=2


Is it possible to opt to print to the error log some other info perhaps
the bactrace of the process or the offending request?

Even better whould be possible to hack a watchdog that could log "slow
requests" like mysql slow_queries?

I'm asking after look a bit the code but it's a bit far away for my
skills right now.

Thanks.

-- 
Carlos Peñas San José
carlos@adoptales.es

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: Is it possible to log better info in timeouts?
  2014-01-02  0:41 Is it possible to log better info in timeouts? Carlos Peñas
@ 2014-01-02  1:31 ` Eric Wong
  0 siblings, 0 replies; 2+ messages in thread
From: Eric Wong @ 2014-01-02  1:31 UTC (permalink / raw)
  To: unicorn list; +Cc: Carlos Peñas

Carlos Peñas <theistian@gmx.com> wrote:
> Recently we had an issue with an external service for an app served with
> unicorn. And some of our requests died with unicorn's timeout because a
> not properly secured call to this external service. We had 20s
> timeout but in production with ten worker and a big percent of urls
> hitting external services 20 secons of timeout sooner or later brings
> down the entire site.
> 
> in the logs when this hapens this line is left for debugging:
> 
> E, [2013-12-25T23:56:49.363426 #26324] ERROR -- : worker=2 PID:27956 timeout (21s > 20s), killing
> E, [2014-12-25T23:56:49.396005 #26324] ERROR -- : reaped #<Process::Status: pid 27956 SIGKILL (signal 9)> worker=2
> 
> 
> Is it possible to opt to print to the error log some other info perhaps
> the bactrace of the process or the offending request?

The SIGKILL sent by the master isn't avoidable/trapable in userspace,
the KILL-ed process has no chance to dump a backtrace.

> Even better whould be possible to hack a watchdog that could log "slow
> requests" like mysql slow_queries?

You need to do this in your application.  The timeout in unicorn is a
last resort: http://unicorn.bogomips.org/Application_Timeouts.html
I don't encourage relying on the unicorn timeout, it's a hack for broken
libraries or fatal bugs in Ruby (which are rarer nowadays).
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2014-01-02  1:31 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-02  0:41 Is it possible to log better info in timeouts? Carlos Peñas
2014-01-02  1:31 ` Eric Wong

Code repositories for project(s) associated with this inbox:

	../../../unicorn.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).