Calculating the required value of "ulimit -q"

Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

Hi, I create a mqueue with msgsize 1024 and maxmsg 5000, so in total:

  5120000 bytes

I set for root that value:

  ~# ulimit -q 5120000

However I get Errno::ENOMEM. In order it to work I must increase the ulimit to 
a value closed to 5160000.

Could I know which is the exact algorithm to calculate the required "ulimit -
q" value based on the desired msgsize and maxmsg?
I would like my application to suggest that value if it cannot start due to 
memory limitations when creating the mqueue.

Thanks a lot.


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> Hi, I create a mqueue with msgsize 1024 and maxmsg 5000, so in total:
> 
>   5120000 bytes
> 
> I set for root that value:
> 
>   ~# ulimit -q 5120000
> 
> However I get Errno::ENOMEM. In order it to work I must increase the ulimit to 
> a value closed to 5160000.
> 
> Could I know which is the exact algorithm to calculate the required "ulimit -
> q" value based on the desired msgsize and maxmsg?
> I would like my application to suggest that value if it cannot start due to 
> memory limitations when creating the mqueue.

Hi Iñaki,

Under Linux, "man 2 getrlimit" says this:

  RLIMIT_MSGQUEUE (Since Linux 2.6.8)
        Specifies the limit on the number of bytes that can be allocated
        for POSIX message queues for the real user  ID  of  the  calling
        process.	 This  limit is enforced for mq_open(3).  Each message
        queue that the user creates counts (until it is removed) against
        this limit according to the formula:

        bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) +
                attr.mq_maxmsg * attr.mq_msgsize

        where  attr  is  the  mq_attr  structure specified as the fourth
        argument to mq_open(3).

        The first addend in the formula,	which  includes  sizeof(struct
        msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot
        create an unlimited number of zero-length  messages  (such  mes
        sages nevertheless each consume some system memory for bookkeep
        ing overhead).

I'm not sure about other OSes (info/research/experiences would be
greatly appreciated!).

-- 
Eric Wong

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Jueves, 14 de Enero de 2010, Eric Wong escribió:
> Iñaki Baz Castillo <ibc@aliax.net> wrote:
> > Hi, I create a mqueue with msgsize 1024 and maxmsg 5000, so in total:
> >
> >   5120000 bytes
> >
> > I set for root that value:
> >
> >   ~# ulimit -q 5120000
> >
> > However I get Errno::ENOMEM. In order it to work I must increase the
> > ulimit to a value closed to 5160000.
> >
> > Could I know which is the exact algorithm to calculate the required
> > "ulimit - q" value based on the desired msgsize and maxmsg?
> > I would like my application to suggest that value if it cannot start due
> > to memory limitations when creating the mqueue.
> 
> Hi Iñaki,
> 
> Under Linux, "man 2 getrlimit" says this:
> 
>   RLIMIT_MSGQUEUE (Since Linux 2.6.8)
>         Specifies the limit on the number of bytes that can be allocated
>         for POSIX message queues for the real user  ID  of  the  calling
>         process.	 This  limit is enforced for mq_open(3).  Each message
>         queue that the user creates counts (until it is removed) against
>         this limit according to the formula:
> 
>         bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) +
>                 attr.mq_maxmsg * attr.mq_msgsize
> 
>         where  attr  is  the  mq_attr  structure specified as the fourth
>         argument to mq_open(3).
> 
>         The first addend in the formula,	which  includes  sizeof(struct
>         msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot
>         create an unlimited number of zero-length  messages  (such  mes
>         sages nevertheless each consume some system memory for bookkeep
>         ing overhead).

Great, I was looking in the wrong place :)


> I'm not sure about other OSes (info/research/experiences would be
> greatly appreciated!).

Unfortunatelly I just use Linux :)
 
Thanks a lot.

-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Jueves, 14 de Enero de 2010, Iñaki Baz Castillo escribió:
> El Jueves, 14 de Enero de 2010, Eric Wong escribió:

> > I'm not sure about other OSes (info/research/experiences would be
> > greatly appreciated!).
> 
> Unfortunatelly I just use Linux :)

But I can append that under Linux 64 bits sizeof(struct msg_msg *) is 8 bytes.
:)

-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Viernes, 15 de Enero de 2010, Iñaki Baz Castillo escribió:
> El Jueves, 14 de Enero de 2010, Iñaki Baz Castillo escribió:
> > El Jueves, 14 de Enero de 2010, Eric Wong escribió:
> > > I'm not sure about other OSes (info/research/experiences would be
> > > greatly appreciated!).
> >
> > Unfortunatelly I just use Linux :)
> 
> But I can append that under Linux 64 bits sizeof(struct msg_msg *) is 8
>  bytes.
> 
> :)

I've received a good suggestion in ruby-talk in order to know the underlying 
architecture (32 or 64 bits):

a) 32 bits:    1.size * 8 => 32
b) 64 bits:    1.size * 8 => 64


Could I suggest a feature in Ruby posix_mq to get the required ammount of 
bytes for a queue? It could calculate:

  queue.attr.mq_maxmsg * sizeof(struct msg_msg *) +
  queue.attr.mq_maxmsg * queue.attr.mq_msgsize

I mean a method like this:

  POSIX_MQ::size(maxmsg, msgsize)


Just a suggestion. Regards.


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Viernes, 15 de Enero de 2010, Iñaki Baz Castillo escribió:
> El Viernes, 15 de Enero de 2010, Iñaki Baz Castillo escribió:
> > El Jueves, 14 de Enero de 2010, Iñaki Baz Castillo escribió:
> > > El Jueves, 14 de Enero de 2010, Eric Wong escribió:
> > > > I'm not sure about other OSes (info/research/experiences would be
> > > > greatly appreciated!).
> > >
> > > Unfortunatelly I just use Linux :)
> >
> > But I can append that under Linux 64 bits sizeof(struct msg_msg *) is 8
> >  bytes.
> >
> > :)
> 
> I've received a good suggestion in ruby-talk in order to know the
>  underlying architecture (32 or 64 bits):
> 
> a) 32 bits:    1.size * 8 => 32
> b) 64 bits:    1.size * 8 => 64
> 
> 
> Could I suggest a feature in Ruby posix_mq to get the required ammount of
> bytes for a queue? It could calculate:
> 
>   queue.attr.mq_maxmsg * sizeof(struct msg_msg *) +
>   queue.attr.mq_maxmsg * queue.attr.mq_msgsize
> 
> I mean a method like this:
> 
>   POSIX_MQ::size(maxmsg, msgsize)
> 
> 
> Just a suggestion. Regards.

Hi, let me know what you think about this:

------------------------
/*
* call-seq:
*  POSIX_MQ.size(msgsize, maxmsg) => Fixnum (bytes)
*
* Returns the amount of bytes required to create a mqueue with the provided
* msgsize (bytes) and maxmsg.
*/
static VALUE s_size(VALUE self, VALUE msgsize, VALUE maxmsg)
{
  long int bytes;

  Check_Type(msgsize, T_FIXNUM);
  Check_Type(maxmsg, T_FIXNUM);
  
  bytes = NUM2INT(maxmsg) * (int)sizeof(struct msg_msg *) + NUM2INT(maxmsg) * 
NUM2INT(msgsize);

  return INT2NUM(bytes);
}



void Init_posix_mq_ext(void)
{
  rb_define_singleton_method(cPOSIX_MQ, "size", s_size, 2);
------------------------ 



It works with normal values (it gives the expected result:
(64 bits):

 irb> POSIX_MQ::size(1024, 5000)
 5160000


However it would fail when the arguments are bigger than FIXNUM (which depends 
on each architecture) and also if the operation gives a very big result.
Unfortunatelly I'n not used to deal with such issues.




-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Viernes, 15 de Enero de 2010, Iñaki Baz Castillo escribió:
> > El Jueves, 14 de Enero de 2010, Iñaki Baz Castillo escribió:
> > > El Jueves, 14 de Enero de 2010, Eric Wong escribió:
> > > > I'm not sure about other OSes (info/research/experiences would be
> > > > greatly appreciated!).
> > >
> > > Unfortunatelly I just use Linux :)
> > 
> > But I can append that under Linux 64 bits sizeof(struct msg_msg *) is 8
> >  bytes.
> > 
> > :)
> 
> I've received a good suggestion in ruby-talk in order to know the underlying 
> architecture (32 or 64 bits):
> 
> a) 32 bits:    1.size * 8 => 32
> b) 64 bits:    1.size * 8 => 64
> 
> 
> Could I suggest a feature in Ruby posix_mq to get the required ammount of 
> bytes for a queue? It could calculate:
> 
>   queue.attr.mq_maxmsg * sizeof(struct msg_msg *) +
>   queue.attr.mq_maxmsg * queue.attr.mq_msgsize
> 
> I mean a method like this:
> 
>   POSIX_MQ::size(maxmsg, msgsize)
> 
> 
> Just a suggestion. Regards.

Hi Iñaki,

I don't think it makes much sense since it doesn't (and can't) take
other queues created by the user into account.  Even if mqueuefs is
mounted under Linux, ownerships can change (and the Linux kernel tracks
sizes for the user that created them, and this doesn't change if
ownership changes).

In my experience, trying to be "helpful" with error handling/recovery
that may end up misleading is worse than just showing the error for the
user to handle.  I consider the best way is just to return the errno so
the user can consult OS-specific documentation and investigate the issue.
This is also a "startup" issue that should be quickly noticed/diagnosed
when an application is launched.

Since I realize POSIX message queues are new territory to a lot of folks
(myself included), I've been meaning to get started on an FAQ RDoc that
I can distribute with the gem/tarball and add to the website.

-- 
Eric Wong

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Viernes, 15 de Enero de 2010, Eric Wong escribió:

> I don't think it makes much sense since it doesn't (and can't) take
> other queues created by the user into account.  Even if mqueuefs is
> mounted under Linux, ownerships can change (and the Linux kernel tracks
> sizes for the user that created them, and this doesn't change if
> ownership changes).
> 
> In my experience, trying to be "helpful" with error handling/recovery
> that may end up misleading is worse than just showing the error for the
> user to handle.  I consider the best way is just to return the errno so
> the user can consult OS-specific documentation and investigate the issue.
> This is also a "startup" issue that should be quickly noticed/diagnosed
> when an application is launched.
> 
> Since I realize POSIX message queues are new territory to a lot of folks
> (myself included), I've been meaning to get started on an FAQ RDoc that
> I can distribute with the gem/tarball and add to the website.

Hi Eric.

Perhaps I explained wrong what I mean. I already deal with all the possible 
exceptions posix_mq raises (no enough memory, inavalid values due to user 
permissions, reusage of existing mqueues so atributes cannot be changed and 
so).

What I propose is really simple: a method to determine tha amount of bytes a 
posix queue would require, and it just depends on 3 factors:
- maxmsg
- msgsize
- 32 or 64 bits OS

So the only I propose (read please my last mail in which I attach a working 
code for posix_mq_ext.c) is a method to know how many bytes a queue would 
require, so the API looks like this:

  POSIX_MQ::size(msgsize, maxmsg)

example:

  irb> POSIX_MQ::size(1024, 5000)
  5160000

Why this could be useful? Imagine your program creates a mqueue with a 
required size. Before trying to create the queue it could read the user 
rlimits for mqueues and try to increase them. The following is a fragment of 
my application:
  

  MQUEUE_SIZE = POSIX_MQ::size(1024, 5000)
  
  if ( current_rlimit = Process.getrlimit(12)[1] ) < MQUEUE_SIZE
    log.info "incrementing rlimits for posix msgqueue (currently " +
             "#{current_rlimit} bytes) to #{MQUEUE_SIZE} bytes"
    begin
      Process.setrlimit(12, MQUEUE_SIZE)
    rescue Errno::EPERM
      log.warn "the current user has no permissions to increase " +
			 "rlimits for posix msgqueue"
    end
  end

  MQUEUE_ATTR          = POSIX_MQ::Attr.new
  MQUEUE_ATTR.maxmsg   = 5000
  MQUEUE_ATTR.msgsize  = 1024
  MQUEUE_ATTR.flags    = 0
  MQUEUE_ATTR.curmsgs  = 0

  begin
    @mq = POSIX_MQ.new MQUEUE_NAME, IO::RDONLY | IO::CREAT | IO::EXCL, 
                       00620, MQUEUE_ATTR
  rescue Errno::ENOSYS => e
    fatal "the kernel has no support for posix queue messages, enable it" 
  rescue Errno::EEXIST
    log.warn "posix message queue already exits so cannot change its " +
             "attributes"
    begin
      @mq = POSIX_MQ.new MQUEUE_NAME, IO::RDONLY
    rescue Errno::EACCES => e
      fatal "cannot open the existing posix message due to user permissions"
    end
  rescue Errno::EINVAL
    log.warn "cannot set posix message queue attributes due to user " + 		      
             "permissions, using system default values"
    @mq = POSIX_MQ.new MQUEUE_NAME, IO::RDONLY | IO::CREAT, 00620
  rescue Errno::ENOMEM => e
    fatal "cannot create posix message queue due to current user's rlimits"
  end


It works very well :)


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Viernes, 15 de Enero de 2010, Eric Wong escribió:
> 
> > I don't think it makes much sense since it doesn't (and can't) take
> > other queues created by the user into account.  Even if mqueuefs is
> > mounted under Linux, ownerships can change (and the Linux kernel tracks
> > sizes for the user that created them, and this doesn't change if
> > ownership changes).
> > 
> > In my experience, trying to be "helpful" with error handling/recovery
> > that may end up misleading is worse than just showing the error for the
> > user to handle.  I consider the best way is just to return the errno so
> > the user can consult OS-specific documentation and investigate the issue.
> > This is also a "startup" issue that should be quickly noticed/diagnosed
> > when an application is launched.
> > 
> > Since I realize POSIX message queues are new territory to a lot of folks
> > (myself included), I've been meaning to get started on an FAQ RDoc that
> > I can distribute with the gem/tarball and add to the website.
> 
> Hi Eric.
> 
> Perhaps I explained wrong what I mean. I already deal with all the possible 
> exceptions posix_mq raises (no enough memory, inavalid values due to user 
> permissions, reusage of existing mqueues so atributes cannot be changed and 
> so).

Hi Iñaki,

I just don't believe such a check before trying operations is ever
useful to do in user-level code.  It's analogous to:

* checking free memory before every memory allocation
* checking fd limits every time a socket/file is opened
* checking disk space for every file write
* ... many more things nobody ever checks about

Keep in mind that even if you did _all_ of the possible checks, you're
still open to race conditions because other processes/threads may be
hitting these limits that affect your process.

Since you're hitting these cases, I wonder if your usage of queues are
too big or there's a part of your application that can be redesigned to
work with smaller queues.  Keep in mind that messages stored in the
queue are not swappable under Linux (unlike "normal" memory).

-- 
Eric Wong

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Viernes, 15 de Enero de 2010, Eric Wong escribió:

> I just don't believe such a check before trying operations is ever
> useful to do in user-level code.  It's analogous to:
> 
> * checking free memory before every memory allocation
> * checking fd limits every time a socket/file is opened
> * checking disk space for every file write
> * ... many more things nobody ever checks about
> 
> Keep in mind that even if you did _all_ of the possible checks, you're
> still open to race conditions because other processes/threads may be
> hitting these limits that affect your process.
> 


Yes, I agree. However let me explain why I do such checks:

I'm coding a server which would be runned as root (just the Unicorn master 
process) and workers and my own extra-processes work as different user (non 
privileged).

When running as root, the queue can be created with the appropriate 
attributes, but for this, rlimits must be increased due to typical Linux 
default values (even for root user!).

However it could occur that the queue was already created by a non privileged 
user which runned the server first, so its attributes would be not enough. I 
must warn the user about it.

Also, I cannot unlink the queue when running the server because it would break 
the hot reload system of Unicorn (USR2 + auto QUIT).


> Since you're hitting these cases, I wonder if your usage of queues are
> too big or there's a part of your application that can be redesigned to
> work with smaller queues.  Keep in mind that messages stored in the
> queue are not swappable under Linux (unlike "normal" memory).

For sure I'm dimensioning my server to work with high traffic, and the 
performance tests I do are very cruel :)

However these tests are passed with values of msgsize=1024 and masmsg=5000 
(but I'll decrease the msgsize).

In total they are 5120000 bytes (less than 5 MB of memory). Is it really 
important even if it's not swappable?

Thanks a lot.


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Viernes, 15 de Enero de 2010, Eric Wong escribió:
> 
> > I just don't believe such a check before trying operations is ever
> > useful to do in user-level code.  It's analogous to:
> > 
> > * checking free memory before every memory allocation
> > * checking fd limits every time a socket/file is opened
> > * checking disk space for every file write
> > * ... many more things nobody ever checks about
> > 
> > Keep in mind that even if you did _all_ of the possible checks, you're
> > still open to race conditions because other processes/threads may be
> > hitting these limits that affect your process.
> > 
> 
> Yes, I agree. However let me explain why I do such checks:
> 
> I'm coding a server which would be runned as root (just the Unicorn master 
> process) and workers and my own extra-processes work as different user (non 
> privileged).

Even though it's technically supported, I still consider running Unicorn
as root to be the wrong thing to do...

> When running as root, the queue can be created with the appropriate 
> attributes, but for this, rlimits must be increased due to typical Linux 
> default values (even for root user!).
 
> However it could occur that the queue was already created by a non privileged 
> user which runned the server first, so its attributes would be not enough. I 
> must warn the user about it.

If the queue is already existing, then you shouldn't worry about the
underlying size/memory usage of it.

> Also, I cannot unlink the queue when running the server because it would break 
> the hot reload system of Unicorn (USR2 + auto QUIT).
> 
> 
> > Since you're hitting these cases, I wonder if your usage of queues are
> > too big or there's a part of your application that can be redesigned to
> > work with smaller queues.  Keep in mind that messages stored in the
> > queue are not swappable under Linux (unlike "normal" memory).
> 
> For sure I'm dimensioning my server to work with high traffic, and the 
> performance tests I do are very cruel :)
> 
> However these tests are passed with values of msgsize=1024 and masmsg=5000 
> (but I'll decrease the msgsize).

> In total they are 5120000 bytes (less than 5 MB of memory). Is it really 
> important even if it's not swappable?

The 5M part is probably not, but the maxmsg=5000 is big enough to make
me highly uncomfortable.  Generally, the lower the latency between
systems, the smaller the queue sizes should be.  I consider the 10
message default in Linux to be reasonable, actually.

In my experience, maximizing throughput under high load for benchmarks
can often be detrimental to providing good _average_ case
latency/predictability/failover characteristics.  Having a huge queues
mean your clients won't know to back off/failover if a machine is
overloaded.  There's a good reason things like TCP use exponential
backoff the way they do to keep the Internet usable as a whole.

Keep in mind FIFOs/pipes under Linux have only 64K total in 2.6
(and only 4K in 2.4).  For TCP sockets, they only scale up to 4MB
(by default in 2.6) for high latency (but also high throughput)
connections.

-- 
Eric Wong

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Sábado, 16 de Enero de 2010, Eric Wong escribió:
> Even though it's technically supported, I still consider running Unicorn
> as root to be the wrong thing to do...

Yes, I will also provide instructions to run is at non root user. But note it 
makes thinkgs complex:

- A normal user cannot create a PID file under the standard path /var/run so 
prior tu running the server it's required to mkdir /var/run/unicorn/ with 
appropriate write permissions.

- Same for listening UNIX socket.

- Same for posix mqueue attributes (it's required to edit /etc/security/limits 
and restart the system).

Is there some security issue in running just the master as root? it doesn't 
process requests, and a vulnerable worker (due to the Rack application code) 
cannot kill the master or force it to do something dangerous. Am I wrong?


> > When running as root, the queue can be created with the appropriate
> > attributes, but for this, rlimits must be increased due to typical Linux
> > default values (even for root user!).
> >
> > However it could occur that the queue was already created by a non
> > privileged user which runned the server first, so its attributes would be
> > not enough. I must warn the user about it.
> 
> If the queue is already existing, then you shouldn't worry about the
> underlying size/memory usage of it.

But the problem is that if the queue was created first by a non privileged 
user then its attributes are limited and could be not enough for high load 
under production.


> The 5M part is probably not, but the maxmsg=5000 is big enough to make
> me highly uncomfortable.  Generally, the lower the latency between
> systems, the smaller the queue sizes should be.  I consider the 10
> message default in Linux to be reasonable, actually.
> 
> In my experience, maximizing throughput under high load for benchmarks
> can often be detrimental to providing good _average_ case
> latency/predictability/failover characteristics.  Having a huge queues
> mean your clients won't know to back off/failover if a machine is
> overloaded.  There's a good reason things like TCP use exponential
> backoff the way they do to keep the Internet usable as a whole.
> 
> Keep in mind FIFOs/pipes under Linux have only 64K total in 2.6
> (and only 4K in 2.4).  For TCP sockets, they only scale up to 4MB
> (by default in 2.6) for high latency (but also high throughput)
> connections.

Thanks for so good explanation. Please let me explain why I need so ammount of 
maxmsg:

- Unicorn workers receive a HTTP PUT or DELETE.

- In case the operation success (a document is modified) the worker writes a 
message (upto ~500 Kbytes) in the mqueue, and replies HTTP 200/201 even if the 
mqueue was full. This is important: the HTTP request is NEVER blocked due to 
the mqueue so the HTTP reques processing is fast.

- Other process (processes in fact) reads the mqueue and generates a TCP SIP 
request (similar to HTTP) which is sent in a blocking mode to a SIP server 
which must run on same host or same network (no network congestion and such 
issues).

- In the same process a green thread is reading the SIP responses to match 
them with the sent SIP requests.. Theorically all should go fine (200 OK), if 
not something wrong is hapeening to the SIP server. The SIP request is sent in 
blocking mode since I want the mqueue to do handle the congestion in case of 
too much traffic (I don't want to have a "queue" Array or similar).


In my benchmark I do an excessive approach since the workers write into the 
mqueue for any HTTP request (GET, PUT, DELETE) even if it doesn't success (i.e 
HTTP/1.1 404), so it's really exagerated.
Also the benchmark runs in localhost and is based on apache "ab" command 
(10000 GET requests with concurrency 10), which is really excessive (for such 
traffic nginx could do load balancing with more servers).

Using 2-4 Unicorn workers and maxmsg=5000 all goes ok, just a few errors 
"somtimes" about mqueue full (but I insist: the test is excessive IMHO).

So under "normal" circumstances I expect maxmsg could be safely decreased to 
200-500. Also msgsize could be decreaed (I don't know yet the required size).

Let me a question: if I set msgsize=1024 but messages are always lower than 
512, would I get in performance if I set msgsize=512? this is: does the Posix 
mqueue sent a fixed block of msgsize bytes even if the real message is lower?


Thanks a lot for all your help.

 


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Sábado, 16 de Enero de 2010, Eric Wong escribió:
> > Even though it's technically supported, I still consider running Unicorn
> > as root to be the wrong thing to do...
> 
> Yes, I will also provide instructions to run is at non root user. But note it 
> makes thinkgs complex:
> 
> - A normal user cannot create a PID file under the standard path /var/run so 
> prior tu running the server it's required to mkdir /var/run/unicorn/ with 
> appropriate write permissions.
> 
> - Same for listening UNIX socket.

Both can go into /{var,}/tmp/unicorn or similar...

> - Same for posix mqueue attributes (it's required to edit /etc/security/limits 
> and restart the system).
> 
> Is there some security issue in running just the master as root? it doesn't 
> process requests, and a vulnerable worker (due to the Rack application code) 
> cannot kill the master or force it to do something dangerous. Am I wrong?

No security issues that I know of, but nothing is guaranteed.

On the other hand, the likelyhood of messing up permissions/ownership
is less when running as a regular user without user switching.

> > > When running as root, the queue can be created with the appropriate
> > > attributes, but for this, rlimits must be increased due to typical Linux
> > > default values (even for root user!).
> > >
> > > However it could occur that the queue was already created by a non
> > > privileged user which runned the server first, so its attributes would be
> > > not enough. I must warn the user about it.
> > 
> > If the queue is already existing, then you shouldn't worry about the
> > underlying size/memory usage of it.
> 
> But the problem is that if the queue was created first by a non privileged 
> user then its attributes are limited and could be not enough for high load 
> under production.

Let your users research and decide whether or not to increase those
values on their own based on their hardware requirements/needs.

Applications that insist on modifying kernel/system-wide defaults raises
red flags for sysadmins and operations folks.

> > The 5M part is probably not, but the maxmsg=5000 is big enough to make
> > me highly uncomfortable.  Generally, the lower the latency between
> > systems, the smaller the queue sizes should be.  I consider the 10
> > message default in Linux to be reasonable, actually.
> > 
> > In my experience, maximizing throughput under high load for benchmarks
> > can often be detrimental to providing good _average_ case
> > latency/predictability/failover characteristics.  Having a huge queues
> > mean your clients won't know to back off/failover if a machine is
> > overloaded.  There's a good reason things like TCP use exponential
> > backoff the way they do to keep the Internet usable as a whole.
> > 
> > Keep in mind FIFOs/pipes under Linux have only 64K total in 2.6
> > (and only 4K in 2.4).  For TCP sockets, they only scale up to 4MB
> > (by default in 2.6) for high latency (but also high throughput)
> > connections.
> 
> Thanks for so good explanation. Please let me explain why I need so ammount of 
> maxmsg:
> 
> - Unicorn workers receive a HTTP PUT or DELETE.
> 
> - In case the operation success (a document is modified) the worker writes a 
> message (upto ~500 Kbytes) in the mqueue, and replies HTTP 200/201 even if the 
> mqueue was full. This is important: the HTTP request is NEVER blocked due to 
> the mqueue so the HTTP reques processing is fast.

Have you considered returning a 50x error so a load balancer can throw
requests to another machine in the cluster?  Or just telling your
clients to try again later?

From what I'm seeing, you're setting up your systems to bite off more
than they can chew and using the queue to hide underlying
capacity/performance problems with the backends...

> - Other process (processes in fact) reads the mqueue and generates a TCP SIP 
> request (similar to HTTP) which is sent in a blocking mode to a SIP server 
> which must run on same host or same network (no network congestion and such 
> issues).
> 
> - In the same process a green thread is reading the SIP responses to match 
> them with the sent SIP requests.. Theorically all should go fine (200 OK), if 
> not something wrong is hapeening to the SIP server. The SIP request is sent in 
> blocking mode since I want the mqueue to do handle the congestion in case of 
> too much traffic (I don't want to have a "queue" Array or similar).
> 
> 
> In my benchmark I do an excessive approach since the workers write into the 
> mqueue for any HTTP request (GET, PUT, DELETE) even if it doesn't success (i.e 
> HTTP/1.1 404), so it's really exagerated.
> Also the benchmark runs in localhost and is based on apache "ab" command 
> (10000 GET requests with concurrency 10), which is really excessive (for such 
> traffic nginx could do load balancing with more servers).
> 
> Using 2-4 Unicorn workers and maxmsg=5000 all goes ok, just a few errors 
> "somtimes" about mqueue full (but I insist: the test is excessive IMHO).
> 
> So under "normal" circumstances I expect maxmsg could be safely decreased to 
> 200-500. Also msgsize could be decreaed (I don't know yet the required size).

How long does it take your backends to process 200 messages in the queue?

> Let me a question: if I set msgsize=1024 but messages are always lower than 
> 512, would I get in performance if I set msgsize=512? this is: does the Posix 
> mqueue sent a fixed block of msgsize bytes even if the real message is lower?

Userspace always has to pass a buffer of at least msgsize bytes to
call mq_receive(3).  I suggest reusing the destination buffer for
POSIX_MQ#receive whenever possible (just like with IO#sysread/IO#read)
to avoid the overhead of allocating a new buffer every time (more GC
overhead).

You should double check the underlying kernel implementation on whether
it dynamically allocates space inside the kernel.



The bottom line is that I'm not going to add things that encourage
people to mess with system defaults.  Especially not without forcing
them to understand the underlying issues at hand and seek alternatives
first.

-- 
Eric Wong

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Lunes, 18 de Enero de 2010, Eric Wong escribió:

> > But the problem is that if the queue was created first by a non
> > privileged user then its attributes are limited and could be not enough
> > for high load under production.
> 
> Let your users research and decide whether or not to increase those
> values on their own based on their hardware requirements/needs.
> 
> Applications that insist on modifying kernel/system-wide defaults raises
> red flags for sysadmins and operations folks.

That's right :)


 

> > - Unicorn workers receive a HTTP PUT or DELETE.
> >
> > - In case the operation success (a document is modified) the worker
> > writes a message (upto ~500 Kbytes) in the mqueue, and replies HTTP
> > 200/201 even if the mqueue was full. This is important: the HTTP request
> > is NEVER blocked due to the mqueue so the HTTP reques processing is fast.
> 
> Have you considered returning a 50x error so a load balancer can throw
> requests to another machine in the cluster?  Or just telling your
> clients to try again later?

The point is that it's not required at all. The client upload a document (HTTP 
PUT) and the server stores it so replies HTTP 200. Ok, the client connection 
ends.
Just before sending the HTTP 200, the Unicorn worker writes into the mqueue a 
message. Other process reads it and generates a SIP message which is sent to a 
different server. In case this SIP message fails (mqueue is full or TCP error) 
then it's responsability of the process to send again latter, queue into a 
database or memory and so on. This is, the HTTP connection has been succesful 
and the client must not resend it.


> From what I'm seeing, you're setting up your systems to bite off more
> than they can chew and using the queue to hide underlying
> capacity/performance problems with the backends...

I'm using the mqueue as queue system, rather than just a inter-processes 
communication method. So in case the remote SIP server fails (or TCP 
connection) then que mqueue queues the messages.

There is other approach I'm working right now: use a normal mqueue (msgsize 
20-50) and the reader process inserts the message in an Array. Other thread in 
same process reads from the Array and sends the message via TCP/SIP.

Un this case, the mqueue is just used for inter-process communication and the 
Array as a queue. I still must do some benchmarks to compare both approachs.

 
> > - Other process (processes in fact) reads the mqueue and generates a TCP
> > SIP request (similar to HTTP) which is sent in a blocking mode to a SIP
> > server which must run on same host or same network (no network congestion
> > and such issues).
> >
> > - In the same process a green thread is reading the SIP responses to
> > match them with the sent SIP requests.. Theorically all should go fine
> > (200 OK), if not something wrong is hapeening to the SIP server. The SIP
> > request is sent in blocking mode since I want the mqueue to do handle the
> > congestion in case of too much traffic (I don't want to have a "queue"
> > Array or similar).
> >
> >
> > In my benchmark I do an excessive approach since the workers write into
> > the mqueue for any HTTP request (GET, PUT, DELETE) even if it doesn't
> > success (i.e HTTP/1.1 404), so it's really exagerated.
> > Also the benchmark runs in localhost and is based on apache "ab" command
> > (10000 GET requests with concurrency 10), which is really excessive (for
> > such traffic nginx could do load balancing with more servers).
> >
> > Using 2-4 Unicorn workers and maxmsg=5000 all goes ok, just a few errors
> > "somtimes" about mqueue full (but I insist: the test is excessive IMHO).
> >
> > So under "normal" circumstances I expect maxmsg could be safely decreased
> > to 200-500. Also msgsize could be decreaed (I don't know yet the required
> > size).
> 
> How long does it take your backends to process 200 messages in the queue?

Under normal circunstances is fast enough. But with high traffic (benchmarks) 
Unicorn workers are faster so the mqueue gets full after sometime (depending 
on maxmsg of course).


> > Let me a question: if I set msgsize=1024 but messages are always lower
> > than 512, would I get in performance if I set msgsize=512? this is: does
> > the Posix mqueue sent a fixed block of msgsize bytes even if the real
> > message is lower?
> 
> Userspace always has to pass a buffer of at least msgsize bytes to
> call mq_receive(3).  I suggest reusing the destination buffer for
> POSIX_MQ#receive whenever possible (just like with IO#sysread/IO#read)
> to avoid the overhead of allocating a new buffer every time (more GC
> overhead).

So you mean that I should avoid doing the following, right?:

  loop do
    # Blocking waiting.
    msg = @mq.shift
    SIP.send(msg)
  end

and instead I should do:

  msg = ""
  loop do
    # Blocking waiting.
    msg.replace @mq.shift
    SIP.send(msg)
  end


Am I right?




> The bottom line is that I'm not going to add things that encourage
> people to mess with system defaults.  Especially not without forcing
> them to understand the underlying issues at hand and seek alternatives
> first.

It makes lot of sense.

Thanks a lot. 


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Lunes, 18 de Enero de 2010, Iñaki Baz Castillo escribió:
> So you mean that I should avoid doing the following, right?:
> 
>   loop do
>     # Blocking waiting.
>     msg = @mq.shift
>     SIP.send(msg)
>   end
> 
> and instead I should do:
> 
>   msg = ""
>   loop do
>     # Blocking waiting.
>     msg.replace @mq.shift
>     SIP.send(msg)
>   end
> 
> 
> Am I right?


Humm, for sure using "msg.replace @mq.shift" is a bad idea as I get msg with 
repeated value sometimes (many times...). This occurs because msg variable is 
passed as variable to other method running in a different thread. If I do 
msg.replace then I'm overwritting its value before the method handles it!


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Lunes, 18 de Enero de 2010, Iñaki Baz Castillo escribió:
> El Lunes, 18 de Enero de 2010, Iñaki Baz Castillo escribió:
> > So you mean that I should avoid doing the following, right?:
> >
> >   loop do
> >     # Blocking waiting.
> >     msg = @mq.shift
> >     SIP.send(msg)
> >   end
> >
> > and instead I should do:
> >
> >   msg = ""
> >   loop do
> >     # Blocking waiting.
> >     msg.replace @mq.shift
> >     SIP.send(msg)
> >   end
> >
> >
> > Am I right?
> 
> Humm, for sure using "msg.replace @mq.shift" is a bad idea as I get msg
>  with repeated value sometimes (many times...). This occurs because msg
>  variable is passed as variable to other method running in a different
>  thread. If I do msg.replace then I'm overwritting its value before the
>  method handles it!

I understand your point, I should use something as StringIO to the message 
readed from the mqueue is appended to the StringIO while other thread reads 
from it.

-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Lunes, 18 de Enero de 2010, Iñaki Baz Castillo escribió:
> > El Lunes, 18 de Enero de 2010, Iñaki Baz Castillo escribió:
> > > So you mean that I should avoid doing the following, right?:
> > >
> > >   loop do
> > >     # Blocking waiting.
> > >     msg = @mq.shift
> > >     SIP.send(msg)
> > >   end
> > >
> > > and instead I should do:
> > >
> > >   msg = ""
> > >   loop do
> > >     # Blocking waiting.
> > >     msg.replace @mq.shift
> > >     SIP.send(msg)
> > >   end
> > >
> > >
> > > Am I right?
> > 
> > Humm, for sure using "msg.replace @mq.shift" is a bad idea as I get msg
> >  with repeated value sometimes (many times...). This occurs because msg
> >  variable is passed as variable to other method running in a different
> >  thread. If I do msg.replace then I'm overwritting its value before the
> >  method handles it!
> 
> I understand your point, I should use something as StringIO to the message 
> readed from the mqueue is appended to the StringIO while other thread reads 
> from it.

No actually, pass msg as the argument of POSIX_MQ#shift:

  msg = ""
  loop do
    # Blocking waiting.
    @mq.shift(msg) # this modifies msg
    SIP.send(msg)
  end

The second argument of IO#read and IO#sysread work the same way.

-- 
Eric Wong

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Martes, 19 de Enero de 2010, Eric Wong escribió:

> No actually, pass msg as the argument of POSIX_MQ#shift:
> 
>   msg = ""
>   loop do
>     # Blocking waiting.
>     @mq.shift(msg) # this modifies msg
>     SIP.send(msg)
>   end
> 
> The second argument of IO#read and IO#sysread work the same way.

Ok, however I cannot use it since "SIP.send(msg)" is in fact a diferent thread 
so if I use the same string instance (msg.object_id) forever it can occcur 
that it's modified by @mq before SIP.send wrote it into a TCP socket. 


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: Calculating the required value of "ulimit -q"

[Iñaki Baz Castillo]

El Lunes, 18 de Enero de 2010, Iñaki Baz Castillo escribió:
> I'm using the mqueue as queue system, rather than just a inter-processes 
> communication method. So in case the remote SIP server fails (or TCP 
> connection) then que mqueue queues the messages.
> 
> There is other approach I'm working right now: use a normal mqueue
>  (msgsize  20-50) and the reader process inserts the message in an Array.
>  Other thread in same process reads from the Array and sends the message
>  via TCP/SIP.
> 
> Un this case, the mqueue is just used for inter-process communication and
>  the  Array as a queue. I still must do some benchmarks to compare both
>  approachs.


Ok, more or less solved:

I've coded a queue based on an Array of fixed length (5000 entries as max 
size). When a message is extracted from the mqueue is appended to the Array. 
When the corresponding SIP message is sent (by other thread) such entry is 
shifted from the Array.

I get same performance than using Posix mqueue as queue, but now the mqueue 
just requires maxmsg=100 which is much less than 5000 :)


-- 
Iñaki Baz Castillo <ibc@aliax.net>