Shared memory between workers

Shared memory between workers

[Iñaki Baz Castillo]

Hi, I plan to build a SIP TCP server (no UDP) based on
Unicorn/Rainbows! HTTP server. The main different between a SIP server
and HTTP server are:

- SIP uses persistent TCP connections, so I should use Rainbows!.
- For a SIP server it's not valid a simple request-response model.
Different workers could handle SIP messages (requests and responses)
belonging to the same SIP session so I need a shared memory between
all the workers.

Another option is using EventMachine, perhaps more suitable for this
purpose by design as it uses a single Ruby process so sharing memory
is not a problem. In the other side using a single process in a
multicore server is a pain.
I would like to use Unicorn/Rainbows as I love its design: by far it's
the more reliable and efficient Ruby HTTP server and it takes
advantages of Unix's features.

I don't want to use a DB server neither MemCache as "shared memory" as
it would be too slow. Is there any way to share RAM memory between
different Unicorn/Rainbows! workers in a *safe* way? I could create a
Hash or Array of SIP sessions into the master process so all the
workers reuse it, but I don't think it would be safe to access/write
into it from different Ruby processes. For that I would also need a
semaphore system (perhaps again a shared variable between all workers
in order to lock the shared Array/Hash when a worker writes into it).

Any tip about it? suggstions?
Thanks a lot.

-- 
Iñaki Baz Castillo
<ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Shared memory between workers

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> Hi, I plan to build a SIP TCP server (no UDP) based on
> Unicorn/Rainbows! HTTP server. The main different between a SIP server
> and HTTP server are:
> 
> - SIP uses persistent TCP connections, so I should use Rainbows!.
> - For a SIP server it's not valid a simple request-response model.
> Different workers could handle SIP messages (requests and responses)
> belonging to the same SIP session so I need a shared memory between
> all the workers.
> 
> Another option is using EventMachine, perhaps more suitable for this
> purpose by design as it uses a single Ruby process so sharing memory
> is not a problem. In the other side using a single process in a
> multicore server is a pain.
> I would like to use Unicorn/Rainbows as I love its design: by far it's
> the more reliable and efficient Ruby HTTP server and it takes
> advantages of Unix's features.
> 
> I don't want to use a DB server neither MemCache as "shared memory" as
> it would be too slow. Is there any way to share RAM memory between
> different Unicorn/Rainbows! workers in a *safe* way? I could create a
> Hash or Array of SIP sessions into the master process so all the
> workers reuse it, but I don't think it would be safe to access/write
> into it from different Ruby processes. For that I would also need a
> semaphore system (perhaps again a shared variable between all workers
> in order to lock the shared Array/Hash when a worker writes into it).
> 
> Any tip about it? suggstions?

Hi Iñaki,

First off I wouldn't call Memcached slow...  How many requests do you
need out of it and what kind of access patterns are you doing?

If you're on a modern Linux 2.6 box, then anything on the local
filesystem is basically shared memory if you have enough RAM to store
your working set (and it sounds like you do).

If you're willing to be confined to a single box, I've had great
experiences with TokyoCabinet the past few years.  One system I built
shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn
(previously Mongrel) workers.  TokyoCabinet uses mmap and pread/pwrite,
so you can safely share DB handles across any number of native
threads/processes.

Writes in TC need filesystem locking and you can only have one active
writer, but that was still plenty fast enough in my experience.

If fsync() becomes a bottleneck, then for non-critical data either:

  1) disable it (TC lets you)
  2) or use libeatmydata if you choose something that doesn't let
     you disable fsync()

<soapbox>
  My personal opinion is that fsync()/O_SYNC are completely overrated.
  All important machines these days have redundant storage, redundant
  power supplies, ECC memory, hardware health monitoring/alerting and
  backup power.  Any remaining hardware/kernel failure possibilities
  are far less likely than application bugs that corrupt data :)

  But, if you really have critical data, /then/ get an SSD or
  a battery-backed write cache.
</soapbox>

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Shared memory between workers

[Iñaki Baz Castillo]

2010/4/26 Eric Wong <normalperson@yhbt.net>:

> Hi Iñaki,
>
> First off I wouldn't call Memcached slow...  How many requests do you
> need out of it and what kind of access patterns are you doing?

Well, I want to do as efficient as possible. This is not the main
purpose (if so I wouldn't use Ruby) but it's important.

For example:
- 8 workers.
- A SIP request arrives and it handled by worker 1.
- The server must store in *shared* memory info about such request
(status, last SIP response sereplied for it....). The request
transaction remains in memory for some seconds after sending the
response.
- Due to a non so unusual SIP issue, a retransmission of the previous
request arrives to the server but now it is handled by worker 2.
- Worker 2 must check the request "branch" id into shared memory, find
the previously created entry, check that a response was already sent
for it, and re-send such response.

Well, in fact this is really common in SIP under UDP. Under TCP
usually there are no retransmissions (but they could be so the worker
must check it).



> If you're on a modern Linux 2.6 box, then anything on the local
> filesystem is basically shared memory if you have enough RAM to store
> your working set (and it sounds like you do).

And what about mounting a filesystem under RAM? :would it be ever faster? :)


> If you're willing to be confined to a single box, I've had great
> experiences with TokyoCabinet the past few years.  One system I built
> shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn
> (previously Mongrel) workers.  TokyoCabinet uses mmap and pread/pwrite,
> so you can safely share DB handles across any number of native
> threads/processes.

I didn't know it. I've just read the doc, installed and tested it.
Really impressive!


> Writes in TC need filesystem locking and you can only have one active
> writer, but that was still plenty fast enough in my experience.

If you use a "on memory database" there wouldn't be filesystem locking, right?


> If fsync() becomes a bottleneck, then for non-critical data either:
>
>  1) disable it (TC lets you)
>  2) or use libeatmydata if you choose something that doesn't let
>     you disable fsync()
>
> <soapbox>
>  My personal opinion is that fsync()/O_SYNC are completely overrated.
>  All important machines these days have redundant storage, redundant
>  power supplies, ECC memory, hardware health monitoring/alerting and
>  backup power.  Any remaining hardware/kernel failure possibilities
>  are far less likely than application bugs that corrupt data :)
>
>  But, if you really have critical data, /then/ get an SSD or
>  a battery-backed write cache.
> </soapbox>

Thanks, but the fact is that I don't need persistent storage. This is,
the data I will need to share between Ruby processes (workers) is
about realtime data, so if the server is stopped and restarted I don't
want to retrieve such information. Well, it could be useful for some
cases however, I need to think about it.
I've seen a method "sync" in the Tokyo Tyrant client API which forces
the server to store/sync the on memory data to the file (if a file is
being used).


So thanks a lot. Definitely I'll learn about TokioCabinet as it seems
a really nice solution for my needs.
Regards.


-- 
Iñaki Baz Castillo
<ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Shared memory between workers

[Gleb Arshinov]

> If you use a "on memory database" there wouldn't be filesystem locking, right?

fsync throughput (a.k.a. commit rate) is easy to test:

  sudo aptitude install sysbench
  sysbench --test=fileio --file-fsync-freq=1 --file-num=1 \
   --file-total-size=16384 --file-test-mode=rndwr run \
   | grep "Requests/sec"

You can then see if any additional tuning (RAM drive, etc.) is even
necessary for your requirements.

Unless something is doing write-back caching in between the app and
the drive this rate will be a function of RPM of your drive, e.g.
~120/s for 7200RPM   We did a bunch of benchmarking when setting up a
couple of DB servers recently and found that consumer-grade drives
(SATA) lie and have write-back caching enabled by default, and you get
1-3k/s.  It's not considered safe to run a database on such
configuration as HDD cache lacks battery backup.  For comparison we
get ~10k/s on a 4 disk RAID 10 SAS array (w/ battery backup unit) with
write-back caching turned on in the controller and off in the drives.

So, the point is that consumer hardware lies and you are quite likely
to get good fsync performance.  Since you don't care about data safety
for your setup this should be fine.

Best regards,

Gleb
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying