Connection issues between nginx and unicorn

Connection issues between nginx and unicorn

[Michael Bernstein]

Hello,

We are having a lot of connection issues between nginx and unicorn,
but we don't know exactly where is the issue. Can somebody help us?

Exemple from our nginx logs:
[error] 6#6: *4269348 connect() failed (111: Connection refused) while
connecting to upstream, client: 10.2.2.252, server: _, request: "POST
/api/v1/documents?access_token=....
[error] 6#6: *4269169 recv() failed (104: Connection reset by peer)
while reading response header from upstream, client: 10.2.10.120,
server: _, request: "GET /api/v1/documents/.....
[error] 6#6: *4268896 upstream timed out (110: Operation timed out)
while connecting to upstream, client: 10.2.2.252, server: _, request:
"GET /accounts/.... HTTP/1.1", upstream:
"http://172.20.214.176:80/accounts...", host: "app.clicksign.com",
referrer: "https://app.clicksign.com/..."

############
Unicorn configuration file:

# frozen_string_literal: true

listen 8080
worker_processes 4
timeout 60

preload_app true
GC.respond_to?(:copy_on_write_friendly=) &&
  (GC.copy_on_write_friendly = true)

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) &&
    ActiveRecord::Base.connection.disconnect!

  old_pid = "#{server.config[:pid]}.oldbin"
  if old_pid != server.pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end

  sleep 1
end

after_fork do |_server, _worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for
master to send QUIT'
  end

  defined?(ActiveRecord::Base) &&
    ActiveRecord::Base.establish_connection
end


############
nginx configuration file:

proxy_cache_path /var/cache/nginx/ levels=1:2 keys_zone=assets_cache:10m
                 max_size=1g inactive=60m use_temp_path=off;

resolver kube-dns.kube-system.svc.cluster.local valid=5s;

upstream backend {
  server unicorn;
}

server {
  server_name _;

  server_tokens off;
  keepalive_timeout 5;
  client_max_body_size 100M;
  client_body_buffer_size 30M;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay off;

  gzip on;
  gzip_min_length 1000;
  gzip_proxied any;
  gzip_buffers 16 8k;
  gzip_http_version 1.0;
  gzip_types text/css
             text/plain
             text/javascript
             application/javascript
             application/json
             application/x-javascript
             application/xml
             application/xml+rss
             application/xhtml+xml
             application/x-font-ttf
             application/x-font-opentype
             application/vnd.ms-fontobject
             image/svg+xml
             image/x-icon
             application/rss+xml
             application/atom_xml;

  location /healthcheck {
    return 204;
  }

  location ~* ^/(fonts|webfonts|assets|packs) {
    proxy_redirect off;
    proxy_cache assets_cache;
    proxy_cache_valid 120m;
    proxy_pass http://backend;
    add_header Cache-Control public;
    if ($http_origin ~
'^((https*?:\/\/).*?((\.clicksign)\.(com|me|dev)))($|\/.*$)$' ){
      add_header Access-Control-Allow-Origin $http_origin;
    }
    expires max;
  }

  location / {
    proxy_pass http://backend;
    proxy_redirect off;
    proxy_read_timeout 60;
    proxy_set_header Host $http_host;
  }
}

######

Thanks,

Michael Belfer Bernstein
michael.bernstein@clicksign.com
https://www.clicksign.com | https://www.fluxia.com.br

-- 
A informação contida nesta mensagem e seus anexos é considerada secreta, 
para uso exclusivo de seu destinatário. Caso você não seja o destinatário, 
notifique o remetente e elimine esta mensagem. The information contained in 
this message and respective attachments is secret, to be used exclusively 
by its addressee. If you are not the intended addressee, please notify the 
sender and delete this message.

Re: Connection issues between nginx and unicorn

[Eric Wong]

Michael Bernstein <michael.bernstein@clicksign.com> wrote:
> Hello,
> 
> We are having a lot of connection issues between nginx and unicorn,
> but we don't know exactly where is the issue. Can somebody help us?

Sure, replies in line...

> Exemple from our nginx logs:
> [error] 6#6: *4269348 connect() failed (111: Connection refused) while
> connecting to upstream, client: 10.2.2.252, server: _, request: "POST
> /api/v1/documents?access_token=....
> [error] 6#6: *4269169 recv() failed (104: Connection reset by peer)
> while reading response header from upstream, client: 10.2.10.120,
> server: _, request: "GET /api/v1/documents/.....
> [error] 6#6: *4268896 upstream timed out (110: Operation timed out)
> while connecting to upstream, client: 10.2.2.252, server: _, request:
> "GET /accounts/.... HTTP/1.1", upstream:
> "http://172.20.214.176:80/accounts...", host: "app.clicksign.com",
> referrer: "https://app.clicksign.com/..."

Did any connections ever work?

> ############
> Unicorn configuration file:
> 
> # frozen_string_literal: true
> 
> listen 8080

OK, so that means it's listening to all IP addresses on TCP port 8080;
no problem here.

If you use curl to hit 8080 on the host(s) unicorn is running,
it should work.

> worker_processes 4
> timeout 60
> 
> preload_app true
> GC.respond_to?(:copy_on_write_friendly=) &&
>   (GC.copy_on_write_friendly = true)

Off-topic, GC.copy_on_write_friendly is the default in Ruby 2.0+;
(and IIRC it wasn't in any official Ruby release, just the
Phusion "Enterprise Edition")

<snip>

> ############
> nginx configuration file:
> 
> proxy_cache_path /var/cache/nginx/ levels=1:2 keys_zone=assets_cache:10m
>                  max_size=1g inactive=60m use_temp_path=off;
> 
> resolver kube-dns.kube-system.svc.cluster.local valid=5s;
> 
> upstream backend {
>   server unicorn;

Is "unicorn" an entry in your DNS (via search path) or
an entry in /etc/hosts file?

There's also no reference to port 8080 anywhere in your nginx
config, so I don't know how nginx is supposed to know that
unicorn is on port 8080.

I would expect to see something like:

    server unicorn.internal.example.com:8080;

Or if on the same host:

    server 127.0.0.1:8080;

> }
> 
> server {
>   server_name _;

<snip>

>   location ~* ^/(fonts|webfonts|assets|packs) {
>     proxy_redirect off;
>     proxy_cache assets_cache;
>     proxy_cache_valid 120m;
>     proxy_pass http://backend;
>     add_header Cache-Control public;
>     if ($http_origin ~
> '^((https*?:\/\/).*?((\.clicksign)\.(com|me|dev)))($|\/.*$)$' ){
>       add_header Access-Control-Allow-Origin $http_origin;
>     }
>     expires max;
>   }
> 
>   location / {
>     proxy_pass http://backend;
>     proxy_redirect off;
>     proxy_read_timeout 60;
>     proxy_set_header Host $http_host;
>   }

No references to port 8080 in either location block, either.

I don't think you need 8080 in location block if you have them
in the "upstream backend" block; it's been a while since I've
used nginx... But you need 8080 (and the correct IP address or
DNS entry) somewhere.

Re: Connection issues between nginx and unicorn

[Eric Wong]

Hey, just wondering if things got figured things out, thanks.

cf. https://yhbt.net/unicorn-public/20200818180611.GA18977@dcvr/