Re: unicorn worker being killed issue

[subashkc1 <subashkc1@protonmail.com>]

Hi Eric,

Thanks for you reply, I'll try to answer all questions below



> subashkc1 subashkc1@protonmail.com wrote:
>
> > I have a weird issue that I'm having trouble debugging. I've
> > created a sample bare bones rails app where there issue is
> > reproducible https://github.com/subashkc/sidekiq_webui_issue.
>
>
> That's too big and having https://enterprise.contribsys.com/ as
> a source in your Gemfile means I'm probably not able/allowed
> to inspect those gems.
>

I've removed the sidekiq pro form gemfile, it also happens with just the sidekiq gem

> Some general questions:
>
> * how are you starting unicorn and what is the config file?
> (mainly, is `preload_app' enabled? and any hooks?)` preload_app true' is the most common source of problems if
> the rest of your application (including all dependencies)
> isn't equipped to handle fork.

unicorn is started just with `bundle exec unicorn` from a Procfile using foreman, no other config options so its just 1 unicorn worker and its a very bare-bones rails app to reproduce the issue, all I did was add sidekiq gem, and load up the sidekiq webui to see the issue in action
>
> * What are the RACK_ENV / RAILS_ENV?
>

It is happening locally, so all env are development

> `unicorn_rails' is actually pointless nowadays, and the plain` unicorn' command is highly recommended.

I'm only using the unicorn gem, not touching unicorn_rails at all

>
> Use a single worker for debugging, especially if you want to
> strace/truss the process.

Yeah just using 1 unicorn worker with all the default configs, (this repo doesn't have a unicorn config file so all defaults)

>
> Fwiw, I haven't touched Rails roughly a decade (and barely at
> that). My only web apps for the past 15 years or so are on bare
> Rack and meant to be consumed via curl or w3m (nothing involving
> JavaScript or fancy browsers).
>
> Maybe others watching this mailbox can chime in....
>
> > After running the rails app, and load up the sidekiq web UI,
> > if I click on any tabs e.g. busy/queues then the request is
> > served right away with a 200 but it seems there is a 2nd
> > request (can't tell where from) which eventually times out and
> > kills the unicorn worker.
>
>
> I suspect a 2nd request comes from some JS code. Check for a
> network debugging tool included with your browser if it has one.
>

Yeah that's what I can't figure out, I can't see any other requests in the browser network tools

> Or use a single unicorn worker, then strace/truss that worker.
> Something like `strace -p $PID_OF_WORKER -f -v -s5000 -o /tmp/dump`
> (assuming Linux)

I did this, but the output was so verbose I couldn't make any sense of it as I wasn't sure what to look for

>
> Are you able to reproduce the issue with JS disabled?
> Preferably just via curl to minimize surprises.

Ok, it doesn't happen if I do the same request using curl, just doing `curl localhost:8080/sidekiq/busy` and if I do this right after clicking on the tab in the UI, this request doesn't get served right away, it waits until the request from the UI times out >60s, kills the unicorn worker and then the curl req is served

>
> > I originally opened the issue with sidekiq repo but Mike
> > (sidekiq owner) wasn't able to find anything wrong with
> > sidekiq code and he suggested it seemed unicorn was double
> > reading the request and the 2nd one dies after 60s timeout.
>
>
> Double-reading a request is impossible unless there's something
> very broken in Ruby or your networking stack. There'd be a
> hundreds of reports about it if that were true if Ruby or
> unicorn were double-reading requests.
>
> Are you going through some firewalls, or caching/scanning
> proxies which might send a request twice? I've seen that before.
>

No, this is all happening locally on my dev machine, no firewalls or proxies involved

> I have seen some folks accidentally enable multiple loggers,
> so it looks like a single request is happening multiple times.
>

No, just a very basic rails app with the default logger, haven't enabled or added anything else

> strace is usually my tool for debugging stuff like this; along
> with sprinkling `warn "#{LINE} ..."' statements throughout.
> And then perhaps commenting out lots of code and disabling
> as many dependencies as possible.

Yeah it is a rails app with just the sidekiq gem, no other dependencies and no code to disable, haven't added anything.

Let me know if that gives you some hints or if there are other things that I can try or check

Warm Regards