unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: cedric.carrard@gmail.com
Cc: unicorn-public@yhbt.net
Subject: Re: Rails 7 - Unicorn 6.1.0 -- reaped with preload_app
Date: Sat, 7 May 2022 17:27:05 +0000	[thread overview]
Message-ID: <20220507172705.GA28579@dcvr> (raw)
In-Reply-To: <CAOYoJW_=iOgik_wgKJNkvzkKJeu1uxOEW96iHXbr+VR99Rf0wg@mail.gmail.com>

Cédric Carrard <cedriccarrard@gmail.com> wrote:
> Hello
> 
> Since the update to rails 7 and ruby 3.1.1 I have errors with unicorn
> 6.1.0 (we use docker-compose, nginx, sidekiq, postgresql and
> unicorn-worker-killer).

You might need something in your before_fork / after_fork hooks
for sidekiq or anything else that might make connections...

more below...

> 1:10280 configurator.rb:49] Unicorn -- reaped #<Process::Status: pid
> 415 exit 1> worker=15

Is there anything else?

> Here is the unicorn configuration:
> 
> require 'semantic_logger'
> 
> worker_processes(File.read('/proc/cpuinfo').scan(/^processor\s*:/).size
> * (ENV['WORKERS_PER_CPU'] || 2).to_i)

Sidenote: Etc.nprocessors is available since Ruby 2.2 and more
portable than reading /proc/cpuinfo.

I also recommend starting with 1-2 workers whenever you're
debugging to help narrow things down.  Some (most) FD
sharing problems appear when going to 1 -> 2, and 2 is usually
as good as dozens (until system memory/FD limits are hit)

> timeout 300
> preload_app true
> listen(ENV["PORT"] || 3000)
> logger SemanticLogger['Unicorn']
> 
> before_fork do |server, _worker|
>   Signal.trap 'TERM' do
>     server.logger.info 'Unicorn master intercepting TERM and sending
> myself QUIT instead'
>     Process.kill 'QUIT', Process.pid
>   end
> 
>   defined?(ActiveRecord::Base) && ActiveRecord::Base.connection.disconnect!

OK, I guess AR is for Postgres.  Any other things that need to
be handled here?  (e.g. sidekiq, sqlite, xapian, memcached, redis, ...).

"lsof -p $PID" on both master and any single worker can be used
to spot inadvertant FD sharing for things that may need to be
shared.

> end
> 
> after_fork do |server, _worker|
>   Signal.trap 'TERM' do
>     server.logger.info 'Unicorn worker intercepting TERM and doing
> nothing. Wait for master to send QUIT'
>   end
> 
>   SemanticLogger.reopen

I assume SemanticLogger is just a filesystem-based logger?
If it establishes a connection, maybe it needed to be
disconnected in before_fork.

>   defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection
> end
> 
> If I remove the preload_app it works (but apparently it is recommended
> to leave this setting)

Recommendations depend on needs vs willingness to make things work :)
Yes, preload_app saves memory and improves startup time; but it
requires deeper app knowledge to use correctly (as you're
finding out).

> Do you have any idea where the problem could be?

Maybe the above helps, but it's really your codebase nobody in
here knows...

I have no idea what SemanticLogger is, but having fewer dependencies
can help you narrow things down.  In other words, try the normal
logger.

If you don't know the app very well (e.g. inherited from someone
else); I'd start by editing out large sections of the codebase
to narrow things down.

      reply	other threads:[~2022-05-07 17:27 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-05-06 13:55 Rails 7 - Unicorn 6.1.0 -- reaped with preload_app Cédric Carrard
2022-05-07 17:27 ` Eric Wong [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220507172705.GA28579@dcvr \
    --to=e@80x24.org \
    --cc=cedric.carrard@gmail.com \
    --cc=unicorn-public@yhbt.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).