listen backlog

listen backlog

[ghazel]

I found this thread
http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html
after googling for the same error located in my nginx error log.

Is there some way to discover the current queue length of the backlog?
Increasing the maximum size to 2048 "worked" for me as well, but
obviously clients sit in the queue for a long time. If I could monitor
queue length I could get some idea of when I have insufficient unicorn
workers before the delay gets very high or requests fail.

Thanks in advance!

-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[Eric Wong]

ghazel@gmail.com wrote:
> I found this thread
> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html
> after googling for the same error located in my nginx error log.
> 
> Is there some way to discover the current queue length of the backlog?
> Increasing the maximum size to 2048 "worked" for me as well, but
> obviously clients sit in the queue for a long time. If I could monitor
> queue length I could get some idea of when I have insufficient unicorn
> workers before the delay gets very high or requests fail.

If you're using Linux, you can use Raindrops[1] to interrogate
via inet_diag or parse /proc/net/{tcp,unix} yourself.

You can run Raindrops separate/standalone script:
http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb

Or use it as middleware (see webpage):
http://raindrops.bogomips.org/

git clone git://git.bogomips.org/raindrops

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[ghazel]

On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> I found this thread
>> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html
>> after googling for the same error located in my nginx error log.
>>
>> Is there some way to discover the current queue length of the backlog?
>> Increasing the maximum size to 2048 "worked" for me as well, but
>> obviously clients sit in the queue for a long time. If I could monitor
>> queue length I could get some idea of when I have insufficient unicorn
>> workers before the delay gets very high or requests fail.
>
> If you're using Linux, you can use Raindrops[1] to interrogate
> via inet_diag or parse /proc/net/{tcp,unix} yourself.
>
> You can run Raindrops separate/standalone script:
> http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb
>
> Or use it as middleware (see webpage):
> http://raindrops.bogomips.org/
>
> git clone git://git.bogomips.org/raindrops

Amazing! I had somehow never heard of it. I use a unix socket for
unicorn (should I not?) so I had to make a copy of
linux-tcp-listener-stats.rb to support that. Now I'm able to get stats
for my unicorn master.

However, there seems to be some problem getting stats on TCP sockets:

$ ruby linux-tcp-listener-stats.rb 127.0.0.1:9000
            address     active     queued
linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': Connection
refused - sendmsg (Errno::ECONNREFUSED)
        from linux-tcp-listener-stats.rb:42
$ grep  0100007F:2328 /proc/net/tcp
   1: 0100007F:2328 00000000:0000 0A 00000000:00000000 00:00000000
00000000   500        0 25307310 1 ffff880134471700 3000 0 0 2 -1

None of the TCP sockets I know to exist seem to work - they all fail
with that error.

Any ideas?

-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[Eric Wong]

ghazel@gmail.com wrote:
> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > ghazel@gmail.com wrote:
> >> I found this thread
> >> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html
> >> after googling for the same error located in my nginx error log.
> >>
> >> Is there some way to discover the current queue length of the backlog?
> >> Increasing the maximum size to 2048 "worked" for me as well, but
> >> obviously clients sit in the queue for a long time. If I could monitor
> >> queue length I could get some idea of when I have insufficient unicorn
> >> workers before the delay gets very high or requests fail.
> >
> > If you're using Linux, you can use Raindrops[1] to interrogate
> > via inet_diag or parse /proc/net/{tcp,unix} yourself.
> >
> > You can run Raindrops separate/standalone script:
> > http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb
> >
> > Or use it as middleware (see webpage):
> > http://raindrops.bogomips.org/
> >
> > git clone git://git.bogomips.org/raindrops
> 
> Amazing! I had somehow never heard of it.

Yeah, I don't have much interest in marketing; that's what users are
for, but it's a chicken-and-egg problem :x

> I use a unix socket for unicorn (should I not?)

UNIX sockets are faster for nginx <-> Unicorn, but Raindrops is slower
dealing with them since Linux doesn't export UNIX sockets stats to
netlink the same way it does for TCP sockets.  So there's always a
tradeoff :x

> so I had to make a copy of
> linux-tcp-listener-stats.rb to support that. Now I'm able to get stats
> for my unicorn master
> 
> However, there seems to be some problem getting stats on TCP sockets:
> 
> $ ruby linux-tcp-listener-stats.rb 127.0.0.1:9000
>             address     active     queued
> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': Connection
> refused - sendmsg (Errno::ECONNREFUSED)
>         from linux-tcp-listener-stats.rb:42
> $ grep  0100007F:2328 /proc/net/tcp
>    1: 0100007F:2328 00000000:0000 0A 00000000:00000000 00:00000000
> 00000000   500        0 25307310 1 ffff880134471700 3000 0 0 2 -1
> 
> None of the TCP sockets I know to exist seem to work - they all fail
> with that error.

Odd, which kernel do you have?  You might want to "modprobe inet_diag"
if you're on an older kernel.  Or you can just comment-out the TCP
code and use the UNIX sockets code.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[ghazel]

On Thu, Dec 2, 2010 at 11:35 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > ghazel@gmail.com wrote:
>> >> I found this thread
>> >> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html
>> >> after googling for the same error located in my nginx error log.
>> >>
>> >> Is there some way to discover the current queue length of the backlog?
>> >> Increasing the maximum size to 2048 "worked" for me as well, but
>> >> obviously clients sit in the queue for a long time. If I could monitor
>> >> queue length I could get some idea of when I have insufficient unicorn
>> >> workers before the delay gets very high or requests fail.
>> >
>> > If you're using Linux, you can use Raindrops[1] to interrogate
>> > via inet_diag or parse /proc/net/{tcp,unix} yourself.
>> >
>> > You can run Raindrops separate/standalone script:
>> > http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb
>> >
>> > Or use it as middleware (see webpage):
>> > http://raindrops.bogomips.org/
>> >
>> > git clone git://git.bogomips.org/raindrops
>>
>> Amazing! I had somehow never heard of it.
>
> Yeah, I don't have much interest in marketing; that's what users are
> for, but it's a chicken-and-egg problem :x
>
>> I use a unix socket for unicorn (should I not?)
>
> UNIX sockets are faster for nginx <-> Unicorn, but Raindrops is slower
> dealing with them since Linux doesn't export UNIX sockets stats to
> netlink the same way it does for TCP sockets.  So there's always a
> tradeoff :x

When does that netlink speed difference matter - only when collecting stats?

>> so I had to make a copy of
>> linux-tcp-listener-stats.rb to support that. Now I'm able to get stats
>> for my unicorn master
>>
>> However, there seems to be some problem getting stats on TCP sockets:
>>
>> $ ruby linux-tcp-listener-stats.rb 127.0.0.1:9000
>>             address     active     queued
>> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': Connection
>> refused - sendmsg (Errno::ECONNREFUSED)
>>         from linux-tcp-listener-stats.rb:42
>> $ grep  0100007F:2328 /proc/net/tcp
>>    1: 0100007F:2328 00000000:0000 0A 00000000:00000000 00:00000000
>> 00000000   500        0 25307310 1 ffff880134471700 3000 0 0 2 -1
>>
>> None of the TCP sockets I know to exist seem to work - they all fail
>> with that error.
>
> Odd, which kernel do you have?  You might want to "modprobe inet_diag"
> if you're on an older kernel.  Or you can just comment-out the TCP
> code and use the UNIX sockets code.

Linux hostname 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008
x86_64 x86_64 x86_64 GNU/Linux

Using "modprobe inet_diag" made the script spit out a new error:

            address     active     queued
linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': NLMSG_ERROR
(RuntimeError)
        from linux-tcp-listener-stats.rb:42

$ sudo modprobe -l inet_diag
/lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko

I dug in to raindrops a little bit, and it seems the error associated
with the NLMSG_ERROR is -2,
"No such file or directory".


-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[Eric Wong]

ghazel@gmail.com wrote:
> On Thu, Dec 2, 2010 at 11:35 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > UNIX sockets are faster for nginx <-> Unicorn, but Raindrops is slower
> > dealing with them since Linux doesn't export UNIX sockets stats to
> > netlink the same way it does for TCP sockets.  So there's always a
> > tradeoff :x
> 
> When does that netlink speed difference matter - only when collecting stats?

Yes, the performance difference is only when collecting stats.

> Using "modprobe inet_diag" made the script spit out a new error:
> 
>             address     active     queued
> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': NLMSG_ERROR
> (RuntimeError)
>         from linux-tcp-listener-stats.rb:42

I don't think I've ever seen NLMSG_ERROR before.  Are you running
this as the same user that originally bound the listener (the user
of the Unicorn master process)?

> $ sudo modprobe -l inet_diag
> /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko
> 
> I dug in to raindrops a little bit, and it seems the error associated
> with the NLMSG_ERROR is -2,
> "No such file or directory".

Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000?  Use
0.0.0.0:9000 if you're bound to that (I had another user on the
raindrops mailing list with that problem).

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[ghazel]

On Fri, Dec 3, 2010 at 1:49 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> Using "modprobe inet_diag" made the script spit out a new error:
>>
>>             address     active     queued
>> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': NLMSG_ERROR
>> (RuntimeError)
>>         from linux-tcp-listener-stats.rb:42
>
> I don't think I've ever seen NLMSG_ERROR before.  Are you running
> this as the same user that originally bound the listener (the user
> of the Unicorn master process)?

Yes. I also tried as root, same error.

>> $ sudo modprobe -l inet_diag
>> /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko
>>
>> I dug in to raindrops a little bit, and it seems the error associated
>> with the NLMSG_ERROR is -2,
>> "No such file or directory".
>
> Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000?  Use
> 0.0.0.0:9000 if you're bound to that (I had another user on the
> raindrops mailing list with that problem).

My unicorn.rb says 127.0.0.1. But just to test I tried fetching stats
for 0.0.0.0:9000 but got the same error.

-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[Eric Wong]

ghazel@gmail.com wrote:
> On Fri, Dec 3, 2010 at 1:49 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > ghazel@gmail.com wrote:
> >> Using "modprobe inet_diag" made the script spit out a new error:
> >>
> >>             address     active     queued
> >> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': NLMSG_ERROR
> >> (RuntimeError)
> >>         from linux-tcp-listener-stats.rb:42
> >
> > I don't think I've ever seen NLMSG_ERROR before.  Are you running
> > this as the same user that originally bound the listener (the user
> > of the Unicorn master process)?
> 
> Yes. I also tried as root, same error.
> 
> >> $ sudo modprobe -l inet_diag
> >> /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko
> >>
> >> I dug in to raindrops a little bit, and it seems the error associated
> >> with the NLMSG_ERROR is -2,
> >> "No such file or directory".
> >
> > Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000?  Use
> > 0.0.0.0:9000 if you're bound to that (I had another user on the
> > raindrops mailing list with that problem).
> 
> My unicorn.rb says 127.0.0.1. But just to test I tried fetching stats
> for 0.0.0.0:9000 but got the same error.

Ah, can you try "modprobe tcp_diag", too?  I can't remember what other
quirks there were for ancient kernels.  Hopefully that works, because
I'd be at a loss otherwise :x

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: listen backlog

[ghazel]

On Sat, Dec 4, 2010 at 3:48 PM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> On Fri, Dec 3, 2010 at 1:49 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > ghazel@gmail.com wrote:
>> >> Using "modprobe inet_diag" made the script spit out a new error:
>> >>
>> >>             address     active     queued
>> >> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats': NLMSG_ERROR
>> >> (RuntimeError)
>> >>         from linux-tcp-listener-stats.rb:42
>> >
>> > I don't think I've ever seen NLMSG_ERROR before.  Are you running
>> > this as the same user that originally bound the listener (the user
>> > of the Unicorn master process)?
>>
>> Yes. I also tried as root, same error.
>>
>> >> $ sudo modprobe -l inet_diag
>> >> /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko
>> >>
>> >> I dug in to raindrops a little bit, and it seems the error associated
>> >> with the NLMSG_ERROR is -2,
>> >> "No such file or directory".
>> >
>> > Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000?  Use
>> > 0.0.0.0:9000 if you're bound to that (I had another user on the
>> > raindrops mailing list with that problem).
>>
>> My unicorn.rb says 127.0.0.1. But just to test I tried fetching stats
>> for 0.0.0.0:9000 but got the same error.
>
> Ah, can you try "modprobe tcp_diag", too?  I can't remember what other
> quirks there were for ancient kernels.  Hopefully that works, because
> I'd be at a loss otherwise :x

Ah! That worked. Thanks!

-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying