Unicorn future plans

Unicorn future plans

[Eric Wong]

Hi all,

Some of you are wondering about the future of the project, especially
since we're nearing a 1.0 release.

== 1.x - bugfixes and Rack-compatibility synchronization

The 1.x series will focus mainly on bug fixes and compatibility with
Rack as it evolves.

If Rack drops the rewindability requirement of "rack.input", Unicorn
will follow ASAP and allow TeeInput to be optional middleware with
newer Rack versions.

Rubinius should be fully-supported soon, as it's already mostly working
except for a few corner-case things Rubinius doesn't implement (issues
filed on their bug tracker).

== 2.x - the fun and crazy stuff

First off, there'll be internal API cleanups + sync with
Rainbows!/Zbatery This won't be user visible, but it'll be less ugly
inside.

=== Scalability improvements

I don't know if we'll see hundreds/thousands of CPUs in a single
application server any time soon, but 2.x will have internal changes
that'll make us more ready for that.  It could be 5-10 years before
massively multi-core machines are viable for web apps, but it'd
certainly be fun to max out Unicorn on those.

I'll be less shy about sacrificing portability for massive scalability.
Correct me if I'm wrong, but Linux is the only Free kernel that scales
to monster multicore machines right now, so I'll primarily focus on
working with features in Linux.

Currently, 8-16 cores seems to be the sweet spot (and has been for a
while), and present-day Unicorn handles that fine as-is.

=== Features (for Rainbows!, mainly)

IPv6 support and SSL will come, too.   These features will mainly be to
support Rainbows!, though some LANs could benefit from those, too.  I'll
have to review the MRI code for those, but I'm leaning towards only
these new features under 1.9.2+.

Multi-VM (MVM, Ruby 1.9 experimental branch) support will probably
happen only in Rainbows! rather than Unicorn.  A true fork() is still
safer in the event of a crash (but less memory-efficient).

== Miscellaneous

I have no plans to provide commercial support, but I'll continue
offering free support on the mailing list (or private email if you're
shy).

As some of you may have noticed; I don't endorse commercial products or
services at all.  This won't change.  However, there will probably be
more commercial support available from 3rd parties after a 1.0 release.


Thanks for reading!

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn future plans

[John-Paul Bader]

Hey, 

your plans sound great except for ignoring FreeBSD ;)

Seriously many many Servers, including many which I administrate run on FreeBSD and they do so very well. I'm not very familiar with the internals of the FreeBSD Kernel but I can assure you that it runs perfectly on 8 and 16 core machines and I would extremely happy if I could continue to use unicorn on them.

It should be easier to maintain a FreeBSD version as they (FreeBSD developers) tend to aim for more consistency across releases than Linux. You said once that you don't run any BSD machines but I'd be happy to offer you access to one of my BSD servers for testing.

Furthermore since Mac OS X and FreeBSD share many features like kqueue and many Ruby developers are running macs it would make even more sense to at least care about BSD even if its not you primary platform. 

I don't want to start a flame war or say that one OS is better than another. Linux is certainly great but so is FreeBSD. For the SMP part I recommend the following page on freebsd.org:

http://www.freebsd.org/smp/

And maybe specifically:

http://www.freebsd.org/smp/#resources

To conclude I can only say that I'm running unicorn on several FreeBSD hosts and so far I couldn't be happier with it. As stated before, I'd be more than happy to continue using that setup.

Kindest regards and thanks for all you efforts,

John


On 14.06.2010, at 21:46, Eric Wong wrote:

> Hi all,
> 
> Some of you are wondering about the future of the project, especially
> since we're nearing a 1.0 release.
> 
> == 1.x - bugfixes and Rack-compatibility synchronization
> 
> The 1.x series will focus mainly on bug fixes and compatibility with
> Rack as it evolves.
> 
> If Rack drops the rewindability requirement of "rack.input", Unicorn
> will follow ASAP and allow TeeInput to be optional middleware with
> newer Rack versions.
> 
> Rubinius should be fully-supported soon, as it's already mostly working
> except for a few corner-case things Rubinius doesn't implement (issues
> filed on their bug tracker).
> 
> == 2.x - the fun and crazy stuff
> 
> First off, there'll be internal API cleanups + sync with
> Rainbows!/Zbatery This won't be user visible, but it'll be less ugly
> inside.
> 
> === Scalability improvements
> 
> I don't know if we'll see hundreds/thousands of CPUs in a single
> application server any time soon, but 2.x will have internal changes
> that'll make us more ready for that.  It could be 5-10 years before
> massively multi-core machines are viable for web apps, but it'd
> certainly be fun to max out Unicorn on those.
> 
> I'll be less shy about sacrificing portability for massive scalability.
> Correct me if I'm wrong, but Linux is the only Free kernel that scales
> to monster multicore machines right now, so I'll primarily focus on
> working with features in Linux.
> 
> Currently, 8-16 cores seems to be the sweet spot (and has been for a
> while), and present-day Unicorn handles that fine as-is.
> 
> === Features (for Rainbows!, mainly)
> 
> IPv6 support and SSL will come, too.   These features will mainly be to
> support Rainbows!, though some LANs could benefit from those, too.  I'll
> have to review the MRI code for those, but I'm leaning towards only
> these new features under 1.9.2+.
> 
> Multi-VM (MVM, Ruby 1.9 experimental branch) support will probably
> happen only in Rainbows! rather than Unicorn.  A true fork() is still
> safer in the event of a crash (but less memory-efficient).
> 
> == Miscellaneous
> 
> I have no plans to provide commercial support, but I'll continue
> offering free support on the mailing list (or private email if you're
> shy).
> 
> As some of you may have noticed; I don't endorse commercial products or
> services at all.  This won't change.  However, there will probably be
> more commercial support available from 3rd parties after a 1.0 release.
> 
> 
> Thanks for reading!
> 
> -- 
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
> 

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn future plans

[Eric Wong]

John-Paul Bader <hukl@berlin.ccc.de> wrote:
> Hey, 
> 
> your plans sound great except for ignoring FreeBSD ;)
> 
> Seriously many many Servers, including many which I administrate run
> on FreeBSD and they do so very well. I'm not very familiar with the
> internals of the FreeBSD Kernel but I can assure you that it runs
> perfectly on 8 and 16 core machines and I would extremely happy if I
> could continue to use unicorn on them.

Hi John-Paul,

Reread my post carefully, not much is changing for 8-16 core users.
FreeBSD and SMP will continue to be supported.  I wasn't referring
to SMP at all when I was talking about Linux-only bits.

> It should be easier to maintain a FreeBSD version as they (FreeBSD
> developers) tend to aim for more consistency across releases than
> Linux. You said once that you don't run any BSD machines but I'd be
> happy to offer you access to one of my BSD servers for testing.

Thanks for the offer.  I'll keep it in mind if I need it again.

> Furthermore since Mac OS X and FreeBSD share many features like kqueue
> and many Ruby developers are running macs it would make even more
> sense to at least care about BSD even if its not you primary platform. 

I know, I've fixed some things on OSX platforms via FreeBSD (OSX scares
me with its GUI-ness).  Keep in mind kqueue (and epoll) are worthless
for Unicorn itself since Unicorn only handles one client at a time.

However, Rainbows! can already use kqueue/epoll with EventMachine and
Rev.

> I don't want to start a flame war or say that one OS is better than
> another. Linux is certainly great but so is FreeBSD. For the SMP part
> I recommend the following page on freebsd.org:

Again, I wasn't talking about SMP at all.   SMP works fine on current
Unicorn and mid-sized servers.  When going to more cores, SMP itself
is a bottleneck, not the kernel.

With Unicorn 2.x, I'm targeting NUMA hardware that isn't available in
commodity servers yet.  Currently, NUMA makes _zero_ sense in web
application servers, but in case things change in a few years, we'll be
ready.

It may be a chicken-and-egg problem, too.  Hardware manufacturers are
slow to commoditize NUMA because a good chunk of software can't even
utilize SMP effectively :)

> To conclude I can only say that I'm running unicorn on several FreeBSD
> hosts and so far I couldn't be happier with it. As stated before, I'd
> be more than happy to continue using that setup.

Again, don't worry :)  You'll be able to continue using everything
as-is.

> Kindest regards and thanks for all you efforts,

No problem.  Please don't top post in the future.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn future plans

[John-Paul Bader]

On 15.06.2010, at 00:10, Eric Wong wrote:

> John-Paul Bader <hukl@berlin.ccc.de> wrote:
>> Hey, 
>> 
>> your plans sound great except for ignoring FreeBSD ;)
>> 
>> Seriously many many Servers, including many which I administrate run
>> on FreeBSD and they do so very well. I'm not very familiar with the
>> internals of the FreeBSD Kernel but I can assure you that it runs
>> perfectly on 8 and 16 core machines and I would extremely happy if I
>> could continue to use unicorn on them.
> 
> Hi John-Paul,
> 
> Reread my post carefully, not much is changing for 8-16 core users.
> FreeBSD and SMP will continue to be supported.  I wasn't referring
> to SMP at all when I was talking about Linux-only bits.


Hey again,


sorry if I misinterpreted your post. My mail could've been shorter by saying that there are FreeBSD supercomputer clusters and that if we get 1000 cores or more FreeBSD will be fast to adapt so don't let it slip out of focus ;)

My offer for a FreeBSD machine account stands and I'd be more than happy to report issues as they occur to help you out if BSD issues occur.

Thanks again,

John
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying