Nginx Sock And Rails Envinroment Error

Nginx Sock And Rails Envinroment Error

[Alex Barlow]

Hi There,

Im running an amazon instance with nginx proxying to a unicorn sock.

For some reason, even though i specify the production environment, when being visited by nginx, the site shows errors in  development form.

Interestingly, when running on a port rather than a sock, if i visit that port, the errors are rendered as normal with a 500 page, the same port, throught nginx, shows errors like you do in development.

The app is rails latest (not 3), i run it with unicorn_rails -E production -c /root/pbr/current/config/unicorn.rb -D

and unicorn.rb looks like..

worker_processes (20)
preload_app(true)

Im thinking it could be a permissions problem, the rails directory is in under root, nginx runs as user nginx, but i have given chown permission the the directory?

Cheers, Alex

Please CC alexbarlowis@me.com
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Nginx Sock And Rails Envinroment Error

[Eric Wong]

Alex Barlow <alexbarlowis@me.com> wrote:
> Hi There,
> 
> Im running an amazon instance with nginx proxying to a unicorn sock.
> 
> For some reason, even though i specify the production environment,
> when being visited by nginx, the site shows errors in  development
> form.
> 
> Interestingly, when running on a port rather than a sock, if i visit
> that port, the errors are rendered as normal with a 500 page, the same
> port, throught nginx, shows errors like you do in development.

Hi Alex,

Sorry, I'm having a little trouble following you, so you're saying:

client <- TCP -> Unicorn                         => quiet errors (good)
client <- TCP -> nginx <- TCP -> Unicorn         => verbose errors (bad)
client <- TCP -> nginx <- Unix socket-> Unicorn  => verbose errors (bad)

Are you sure you have nginx configured correctly and pointing to the
right instance of your app?  Mind sharing your nginx config file?

> The app is rails latest (not 3), i run it with unicorn_rails -E
> production -c /root/pbr/current/config/unicorn.rb -D
> 
> and unicorn.rb looks like..
> 
> worker_processes (20)
> preload_app(true)
> 
> Im thinking it could be a permissions problem, the rails directory is
> in under root, nginx runs as user nginx, but i have given chown
> permission the the directory?

nginx error logs should tell you if you have permissions problems.

I usually specify domain sockets as a dot file in /tmp since I know
the directory is world read/writeable on a properly configured system
and path resolution is slightly faster :)

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Nginx Sock And Rails Envinroment Error

[Alex Barlow]

Hi

It goes

client <- TCP -> Unicorn                         => quiet errors (good)
client <- TCP -> nginx <- TCP -> Unicorn         => quiet errors (good)
client <- TCP -> nginx <- Unix socket-> Unicorn  => verbose errors (bad)

Strange i know!

Ill try the socket in /tmp/ see what that does.

The Nginx error logs show nothing error wise really. My Nginx config is...

user  nginx;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;
    gzip  on;

	upstream unicorn_sock {
		server unix:/root/pbr/unicorn.sock;
	}
	
    server {
        listen       80;
        server_name  localhost;
		proxy_set_header Host $host;

        location / {
			proxy_pass  http://unicorn_sock;
        }
    }
}

This is currently throwing out verbose errors, in production (must be, its using asset host and the production database)

Put the sock in /tmp, no difference

On 15 Feb 2010, at 07:30, Eric Wong wrote:

> Alex Barlow <alexbarlowis@me.com> wrote:
>> Hi There,
>> 
>> Im running an amazon instance with nginx proxying to a unicorn sock.
>> 
>> For some reason, even though i specify the production environment,
>> when being visited by nginx, the site shows errors in  development
>> form.
>> 
>> Interestingly, when running on a port rather than a sock, if i visit
>> that port, the errors are rendered as normal with a 500 page, the same
>> port, throught nginx, shows errors like you do in development.
> 
> Hi Alex,
> 
> Sorry, I'm having a little trouble following you, so you're saying:
> 
> client <- TCP -> Unicorn                         => quiet errors (good)
> client <- TCP -> nginx <- TCP -> Unicorn         => verbose errors (bad)
> client <- TCP -> nginx <- Unix socket-> Unicorn  => verbose errors (bad)
> 
> Are you sure you have nginx configured correctly and pointing to the
> right instance of your app?  Mind sharing your nginx config file?
> 
>> The app is rails latest (not 3), i run it with unicorn_rails -E
>> production -c /root/pbr/current/config/unicorn.rb -D
>> 
>> and unicorn.rb looks like..
>> 
>> worker_processes (20)
>> preload_app(true)
>> 
>> Im thinking it could be a permissions problem, the rails directory is
>> in under root, nginx runs as user nginx, but i have given chown
>> permission the the directory?
> 
> nginx error logs should tell you if you have permissions problems.
> 
> I usually specify domain sockets as a dot file in /tmp since I know
> the directory is world read/writeable on a properly configured system
> and path resolution is slightly faster :)
> 
> -- 
> Eric Wong

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Nginx Sock And Rails Envinroment Error

[Eric Wong]

Alex Barlow <alexbarlowis@me.com> wrote:
> Hi
> 
> It goes
> 
> client <- TCP -> Unicorn                         => quiet errors (good)
> client <- TCP -> nginx <- TCP -> Unicorn         => quiet errors (good)
> client <- TCP -> nginx <- Unix socket-> Unicorn  => verbose errors (bad)
> 
> Strange i know!

Hi Alex,

My gut feeling is that somehow nginx is hitting a different instance of
your app when using the Unix socket.  Other than that, I'm confused.
More info/questions below.

> Ill try the socket in /tmp/ see what that does.
> 
> The Nginx error logs show nothing error wise really. My Nginx config is...
> 
> user  nginx;
> worker_processes  1;
> 
> events {
>     worker_connections  1024;
> }
> 
> http {
>     include       mime.types;
>     default_type  application/octet-stream;
> 
>     sendfile        on;
>     keepalive_timeout  65;
>     gzip  on;
> 
> 	upstream unicorn_sock {
> 		server unix:/root/pbr/unicorn.sock;
> 	}
> 	
>     server {
>         listen       80;
>         server_name  localhost;
> 		proxy_set_header Host $host;
> 
>         location / {
> 			proxy_pass  http://unicorn_sock;
>         }
>     }
> }

Nothing strange there, what's the verbosity of the nginx error_log?

Also, anything enlightening in the Rails production.log or Unicorn
stderr?

Which OS are you running?  Maybe there's a platform-specific bug
somewhere, too...

> This is currently throwing out verbose errors, in production (must be,
> its using asset host and the production database)
> 
> Put the sock in /tmp, no difference

Is there another Unicorn instance on the same box that it might be
somehow hitting?

What happens when you try have Unicorn listening on both TCP and a Unix
socket?  Just put both "listen" directives in your config file and point
nginx to the Unix socket.  Then try hitting the Unicorn TCP port
directly, and then also the Unix socket via nginx.

You can also try hitting the Unix socket directly by crafting your
own HTTP request using socat from th shell:

   req='GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'
   printf "$req" | socat - UNIX:/root/pbr/unicorn.sock

P.S.: I might not have a chance to respond again for the next day
      or so due to personal matters.  Maybe somebody else on this
      list can help....

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Nginx Sock And Rails Envinroment Error

[Alex Barlow]

Fixed it!

Rails was considering all requests to be local. As the request was coming from the local ip (from nginx i assume)

i put these headers in nginx

     proxy_set_header  X-Real-IP  $remote_addr;

      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;

      proxy_set_header Host $http_host;

      proxy_redirect false

Works fine now

Alex


On 15 Feb 2010, at 16:19, Eric Wong wrote:

> Alex Barlow <alexbarlowis@me.com> wrote:
>> Hi
>> 
>> It goes
>> 
>> client <- TCP -> Unicorn                         => quiet errors (good)
>> client <- TCP -> nginx <- TCP -> Unicorn         => quiet errors (good)
>> client <- TCP -> nginx <- Unix socket-> Unicorn  => verbose errors (bad)
>> 
>> Strange i know!
> 
> Hi Alex,
> 
> My gut feeling is that somehow nginx is hitting a different instance of
> your app when using the Unix socket.  Other than that, I'm confused.
> More info/questions below.
> 
>> Ill try the socket in /tmp/ see what that does.
>> 
>> The Nginx error logs show nothing error wise really. My Nginx config is...
>> 
>> user  nginx;
>> worker_processes  1;
>> 
>> events {
>>    worker_connections  1024;
>> }
>> 
>> http {
>>    include       mime.types;
>>    default_type  application/octet-stream;
>> 
>>    sendfile        on;
>>    keepalive_timeout  65;
>>    gzip  on;
>> 
>> 	upstream unicorn_sock {
>> 		server unix:/root/pbr/unicorn.sock;
>> 	}
>> 	
>>    server {
>>        listen       80;
>>        server_name  localhost;
>> 		proxy_set_header Host $host;
>> 
>>        location / {
>> 			proxy_pass  http://unicorn_sock;
>>        }
>>    }
>> }
> 
> Nothing strange there, what's the verbosity of the nginx error_log?
> 
> Also, anything enlightening in the Rails production.log or Unicorn
> stderr?
> 
> Which OS are you running?  Maybe there's a platform-specific bug
> somewhere, too...
> 
>> This is currently throwing out verbose errors, in production (must be,
>> its using asset host and the production database)
>> 
>> Put the sock in /tmp, no difference
> 
> Is there another Unicorn instance on the same box that it might be
> somehow hitting?
> 
> What happens when you try have Unicorn listening on both TCP and a Unix
> socket?  Just put both "listen" directives in your config file and point
> nginx to the Unix socket.  Then try hitting the Unicorn TCP port
> directly, and then also the Unix socket via nginx.
> 
> You can also try hitting the Unix socket directly by crafting your
> own HTTP request using socat from th shell:
> 
>   req='GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'
>   printf "$req" | socat - UNIX:/root/pbr/unicorn.sock
> 
> P.S.: I might not have a chance to respond again for the next day
>      or so due to personal matters.  Maybe somebody else on this
>      list can help....
> 
> -- 
> Eric Wong

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Nginx Sock And Rails Envinroment Error

[Eric Wong]

Alex Barlow <alexbarlowis@me.com> wrote:
> Fixed it!
> 
> Rails was considering all requests to be local. As the request was
> coming from the local ip (from nginx i assume)

Awesome.

Good to know, I've been setting these headers for so long that it's
become second nature and I've forgotten about side effects with Rails
error messages :x

> i put these headers in nginx
> 
>      proxy_set_header  X-Real-IP  $remote_addr;

You probably don't need X-Real-IP, actually, it's quite nginx-specific
and X-Forwarded-For covers you.

>       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
> 
>       proxy_set_header Host $http_host;
> 
>       proxy_redirect false
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Nginx Sock And Rails Envinroment Error

[Alex Barlow]

Cool, thanxs for your support and great software.

I get about 40 requests per second on a 1.7 gb instance with the db on the same host. Thats awesome!

Well, i think so..

Alex

On 17 Feb 2010, at 23:35, Eric Wong wrote:

> Alex Barlow <alexbarlowis@me.com> wrote:
>> Fixed it!
>> 
>> Rails was considering all requests to be local. As the request was
>> coming from the local ip (from nginx i assume)
> 
> Awesome.
> 
> Good to know, I've been setting these headers for so long that it's
> become second nature and I've forgotten about side effects with Rails
> error messages :x
> 
>> i put these headers in nginx
>> 
>>     proxy_set_header  X-Real-IP  $remote_addr;
> 
> You probably don't need X-Real-IP, actually, it's quite nginx-specific
> and X-Forwarded-For covers you.
> 
>>      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
>> 
>>      proxy_set_header Host $http_host;
>> 
>>      proxy_redirect false

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying