unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: subashkc1 <subashkc1@protonmail.com>
To: Eric Wong <e@80x24.org>
Cc: unicorn-public@yhbt.net
Subject: Re: unicorn worker being killed issue
Date: Wed, 10 May 2023 21:34:09 +0000	[thread overview]
Message-ID: <Duci227PYeBVohB90iG1Hc3x5cS11voaHImp0vHOOJDOxvzcBJbGOlhtNAW5lEmM_lzzMbRKslopgr5EtSXv5NxAdLqvIpw73fMbnTwe_i4=@protonmail.com> (raw)
In-Reply-To: <20230507231308.M295161@dcvr>

Hi Eric,

Thanks for you reply, I'll try to answer all questions below

> subashkc1 subashkc1@protonmail.com wrote:
> > I have a weird issue that I'm having trouble debugging. I've
> > created a sample bare bones rails app where there issue is
> > reproducible https://github.com/subashkc/sidekiq_webui_issue.
> That's too big and having https://enterprise.contribsys.com/ as
> a source in your Gemfile means I'm probably not able/allowed
> to inspect those gems.

I've removed the sidekiq pro form gemfile, it also happens with just the sidekiq gem

> Some general questions:
> * how are you starting unicorn and what is the config file?
> (mainly, is `preload_app' enabled? and any hooks?)` preload_app true' is the most common source of problems if
> the rest of your application (including all dependencies)
> isn't equipped to handle fork.

unicorn is started just with `bundle exec unicorn` from a Procfile using foreman, no other config options so its just 1 unicorn worker and its a very bare-bones rails app to reproduce the issue, all I did was add sidekiq gem, and load up the sidekiq webui to see the issue in action
> * What are the RACK_ENV / RAILS_ENV?

It is happening locally, so all env are development

> `unicorn_rails' is actually pointless nowadays, and the plain` unicorn' command is highly recommended.

I'm only using the unicorn gem, not touching unicorn_rails at all

> Use a single worker for debugging, especially if you want to
> strace/truss the process.

Yeah just using 1 unicorn worker with all the default configs, (this repo doesn't have a unicorn config file so all defaults)

> Fwiw, I haven't touched Rails roughly a decade (and barely at
> that). My only web apps for the past 15 years or so are on bare
> Rack and meant to be consumed via curl or w3m (nothing involving
> JavaScript or fancy browsers).
> Maybe others watching this mailbox can chime in....
> > After running the rails app, and load up the sidekiq web UI,
> > if I click on any tabs e.g. busy/queues then the request is
> > served right away with a 200 but it seems there is a 2nd
> > request (can't tell where from) which eventually times out and
> > kills the unicorn worker.
> I suspect a 2nd request comes from some JS code. Check for a
> network debugging tool included with your browser if it has one.

Yeah that's what I can't figure out, I can't see any other requests in the browser network tools

> Or use a single unicorn worker, then strace/truss that worker.
> Something like `strace -p $PID_OF_WORKER -f -v -s5000 -o /tmp/dump`
> (assuming Linux)

I did this, but the output was so verbose I couldn't make any sense of it as I wasn't sure what to look for

> Are you able to reproduce the issue with JS disabled?
> Preferably just via curl to minimize surprises.

Ok, it doesn't happen if I do the same request using curl, just doing `curl localhost:8080/sidekiq/busy` and if I do this right after clicking on the tab in the UI, this request doesn't get served right away, it waits until the request from the UI times out >60s, kills the unicorn worker and then the curl req is served

> > I originally opened the issue with sidekiq repo but Mike
> > (sidekiq owner) wasn't able to find anything wrong with
> > sidekiq code and he suggested it seemed unicorn was double
> > reading the request and the 2nd one dies after 60s timeout.
> Double-reading a request is impossible unless there's something
> very broken in Ruby or your networking stack. There'd be a
> hundreds of reports about it if that were true if Ruby or
> unicorn were double-reading requests.
> Are you going through some firewalls, or caching/scanning
> proxies which might send a request twice? I've seen that before.

No, this is all happening locally on my dev machine, no firewalls or proxies involved

> I have seen some folks accidentally enable multiple loggers,
> so it looks like a single request is happening multiple times.

No, just a very basic rails app with the default logger, haven't enabled or added anything else

> strace is usually my tool for debugging stuff like this; along
> with sprinkling `warn "#{LINE} ..."' statements throughout.
> And then perhaps commenting out lots of code and disabling
> as many dependencies as possible.

Yeah it is a rails app with just the sidekiq gem, no other dependencies and no code to disable, haven't added anything.

Let me know if that gives you some hints or if there are other things that I can try or check

Warm Regards

  reply	other threads:[~2023-05-10 21:34 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-06 22:02 unicorn worker being killed issue subashkc1
2023-05-07 23:13 ` Eric Wong
2023-05-10 21:34   ` subashkc1 [this message]
2023-05-11 19:08     ` Eric Wong
2023-05-12  6:41       ` subashkc1
2023-05-12  8:05         ` Eric Wong
2023-05-13  0:10           ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='Duci227PYeBVohB90iG1Hc3x5cS11voaHImp0vHOOJDOxvzcBJbGOlhtNAW5lEmM_lzzMbRKslopgr5EtSXv5NxAdLqvIpw73fMbnTwe_i4=@protonmail.com' \
    --to=subashkc1@protonmail.com \
    --cc=e@80x24.org \
    --cc=unicorn-public@yhbt.net \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).