Re: EventMachine with thread pool and thread spawn

[Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> On Sat, Aug 24, 2013 at 6:51 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
 
> EM is also not actively maintained. A bunch of patches were
> not reviewed (or rejected) and just sit there for several years...
> This also discourages me putting more effort on it.

Yeah.  I get that feeling, too.  (And also what I've said about being
hooked on ONESHOT nowadays :)

> > Fwiw, nowadays my personal take these days is to avoid EM/libev*-style
> > wrapper library unless they expose (or only have :P) a oneshot interface
> > (EPOLLONESHOT/EV_ONESHOT).  This is *especially* the case (for me) when
> > mixing epoll/kqueue with threads.
> >
> > Standard event-triggered and level-triggered interfaces are both too
> > confusing to me.  Maybe it's just me, though :x
> 
> Since I don't know much about I/O, I don't know what they are :x

kqueue is a queue, it's in the name.  I haven't looked too hard at the
internal details, but the name says much about it :)

I've studied epoll internals a bit and know that epoll is also a queue,
it's just not in the name.

Oneshot behavior basically makes epoll/kqueue behave like a traditional
queue; once you pull a file/socket off the queue, it won't magically
reappear unless you ask epoll/kqueue to watch for it again.

This is significant for multithreaded servers, because it allows the
kernel to handle synchronization (just like accept()).

Level/edge-triggered epoll/kqueue is kind of like a tracklist in a music
player with repeat=on + random=on.  It's fine if you're playing one
track-a-time (like a single-threaded HTTP server), but if you try to
play multiple tracks in different threads, you could end up playing the
_same_ track in two or more threads.

A multi-threaded HTTP server must implement its own synchronization
on top of ET/LT epoll/kqueue to prevent serving the same client
from multiple threads.

> I wonder if I should just go with XEpollThreadPool then? My concern
> is that as Heroku does not buffer the entire request and response,
> we need something which would do this for us. Not sure if XEpollThreadPool
> would be sufficient?

Probably using XEpollThread* is a good choice, but neither buffer (but
you get more I/O concurrency anyways, so maybe it's not so bad)

Plain XEpoll does do full buffering, and you can probably add a
TryDefer-like middleware on top of that, even...

Anyways, it's still infuriating to me that Heroku cannot do full
buffering and still claims to "support" unicorn.

> Another issue is that most of us develop on a Mac, so Mac support
> would be desirable.

I can't officially support non-Free systems, but I can accept
non-intrusive patches.  I'm not nearly as experienced with kqueue,
so maybe you really found a bug in my kqueue API.

Anyways, I've considered unifying a X{Epoll,Kqueue}Thread* interface,
which is why I added kqueue support to sleepy_penguin before I forgot
about it :x

> Because of this I looked into sleepy_penguin,
> realizing that it now has kqueue support. I just sent a patch to
> its mailing list, trying to make it work on Mac. At least all
> tests passed on my computer with that patch.

I didn't get it on the list and can't find it on the archives, either.
http://librelist.com/browser/sleepy.penguin/
Are you sure it went out?  Feel free to Bcc me or send it here.

(Librelist doesn't like plain Cc :<)

It could be a librelist problem, too, but I just sent out a message
considering a move away from librelist and it went through fine:
http://mid.gmane.org/20130825212946.GA32430-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying