after_fork and redis

after_fork and redis

[bradford]

I'm using unicorn w/ a rails app.  I have the following in my
environment.rb $redis = MyApplication::Application.config.redis and in
production.rb I have config.redis = Redis.new(host: "localhost").

I've read I'm supposed to $redis = Redis.new(host: "localhost") in
after_fork when preload_app is true.

When I don't do this, each worker/pid seems to have their own redis
instance.  So, why is this needed?  Here's the logs of me printing out
$redis.client.inspect when both $redis = Redis.new in the after_fork
and just $redis = Redis.new in the environment.rb.

No after fork
https://gist.github.com/bc2c2a3bda01c35730e2

After fork
https://gist.github.com/0c96550660d3926ffe16

The only thing different I notice is Connection::TCPSocket:fd is
always 13 w/ the after fork.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: after_fork and redis

[Eric Wong]

bradford <fingermark@gmail.com> wrote:
> I'm using unicorn w/ a rails app.  I have the following in my
> environment.rb $redis = MyApplication::Application.config.redis and in
> production.rb I have config.redis = Redis.new(host: "localhost").
> 
> I've read I'm supposed to $redis = Redis.new(host: "localhost") in
> after_fork when preload_app is true.
> 
> When I don't do this, each worker/pid seems to have their own redis
> instance.  So, why is this needed?  Here's the logs of me printing out
> $redis.client.inspect when both $redis = Redis.new in the after_fork
> and just $redis = Redis.new in the environment.rb.

(Disclaimer: I still haven't used Redis, but similar knowledge applies
to every TCP-based service)

If a Redis client opens a TCP (or any stream) socket connection before
forking, all the children that are forked will share that _same_ local
TCP socket.  Sharing stream sockets across processes/threads is nearly
always a bad idea: data streams become interleaved and impossible to
separate in either direction.

> No after fork
> https://gist.github.com/bc2c2a3bda01c35730e2
> 
> After fork
> https://gist.github.com/0c96550660d3926ffe16
> 
> The only thing different I notice is Connection::TCPSocket:fd is
> always 13 w/ the after fork.

FD shouldn't really matter.  FDs get recycled ASAP once they're closed
(and GC can automatically close them).  What you want is a different
local port for the TCPSocket on every worker process, and the after_fork
hook will give you that.

Btw, you can demonstrate FD recycling with:

  require 'socket'
  loop do
    c = TCPSocket.new("example.com", 80)
    p [ :fileno, c.fileno ]
    p [ :addr, c.addr ] # (local address)
    c.close
    sleep 1 # be nice to the remote server
  end

You'll see the same FD recycled over and over, but c.addr will
have a different local port.

You can see if after_fork is working correctly by checking the output of
tools like `ss' or `netstat':
(something like: ss -tan | grep -F :$PORT_OF_REDIS)

You can also check connections on the Redis server/process itself
using "lsof -p $PID_OF_REDIS_DAEMON"
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: after_fork and redis

[Ben Somers]

>> When I don't do this, each worker/pid seems to have their own redis
>> instance.  So, why is this needed?  Here's the logs of me printing out
>> $redis.client.inspect when both $redis = Redis.new in the after_fork
>> and just $redis = Redis.new in the environment.rb.

Checked through the redis.rb source a little bit, and I believe it's
got code specifically to avoid opening the connection until it's
needed, to prevent some shared connection problems
(https://github.com/redis/redis-rb/commit/d1bc88f85bc9bfbb7aabdfc2700bf4b30c0b0115).
You probably aren't making actual use of the connection until after
app startup, so that's why things are working so far for you. It's still
a good idea to initialize Redis in the after_fork, though, because if
something in your app's startup or initializer code does wind up using
that redis connection, you'd still get the shared connection problem.

> Sharing stream sockets across processes/threads is nearly
> always a bad idea: data streams become interleaved and impossible to
> separate in either direction.

"Nearly always a bad idea" is a bit of an understatement; the
interleaved data streams tend to create catastrophic errors, things
like users unable to log in or (god forbid) logging into other
people's sessions. And they usually manifest as race conditions, so
they're hard to debug if you don't know what to look for.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying