Saturday, September 4, 2010

EON ZFS takes the road to Illumos

By now Oracle's leaked letter and the fate of Opensolaris, source gates closing, is probably well known. It seems that Solaris 11 will be Oracle's sole OS focus, and while it's unknown if it will be open source or not, the opensolaris community and it's users were ignored, abandoned and left facing a nasty dose of hard realities.

The options left for the opensolaris community (including EON ZFS Storage) were bleak. Sun Microsystems was built on great innovation and thanks to a previous Solaris kernel engineer Garrett D'Amore and the open-source effort called illumos, there is new hope. It's stated that illumos will be 100% community driven and owned and many smart people who are committed to opensolaris and ZFS have signed on to breathe life into this effort and free our reliance from Oracle's handouts.

EON is testing the road to illumos and I have compiled the nightly a couple of times. Some build times:
297mins/~5 hrs    Quad core Q6600 @ 2.4Ghz w/2GB RAM
52mins/~1 hr      Dual X5570 @ 2.93Ghz w/4GB RAM
The journey should be interesting ...

29 comments:

Antonio Vieiro said...

Five hours on a Q6600? I also have one and the builds are faster for me.

Did you remember to remove the "l" (lint) option in your illumos.sh/NIGHTLY_OPTIONS? You may not need it (as it's used for checking code integrity only, and does not affect generated binaries).

Andre Lue said...

Hi Antonio,

In both cases they are virtual development instances with 2 and 4GB of RAM respectively so they won't attain true capable speeds since they are not running bare metal. Yes lint option was removed.

What times do you get bare metal on the Q6600?

Morpheus256 said...

Andre,
I'm a little confused as to how illumos fit's into the picture when yesterday it was announced that OpenIndiana will be continuing on the OpenSolaris path. being that Illumos has Nexenta as a partner I'm inclined to believe that Illumos will have a better ZFS implementation given time, but I cant help but wonder what OpenIndiana will bring to the table.

Thoughts?

Just thought I'd mention, I've been happily running EON for almost a year and haven't rebooted once.

Andre Lue said...

Hi Morpheus256,

Illumos will focus on delivering all further core src-gate developments of Opensolaris/ZFS. Most distributions (Openindiana) will most likely build on the Illumos development bits, going forward.

The current release of Openindiana is based on the last public onnv-gates of the Oracle/SUN source code and their road map as well as Schillix (1st Illumos based release) will use Illumos going forward.

Thanks for sharing your thoughts! It's always great to hear feedback from the user community that EON is working as designed and living up to being rock solid and reliable.

Andre Lue said...

Here's a link to the OpenIndiana roadmap

gumpateus said...

Wondering if this is the right place to post this question. I'd like to run a tftp server on EON but I didn't see an included binary for it. Did I miss it, or does it not exist yet? If not ... are there any plans to include it in the future?

Andre Lue said...

Hi Gumpateus,

A tftpd server is not included, you would have to extract the SUN version of the package from the snv_130 DVD or compile and add one to /usr/local.

Alan said...

when will this be out? I have had to switch to open indiana to get some of the stuff back i had to have, (like zil ram disk and importing pools and crap) but I miss eon, it was rock solid and tiny!

Thanks so much for a great product, alan

Andre Lue said...

Hi Alan,

Right now it's hard to say. Still working thru alot of issues with packaging/IPS.

A.W.S. said...

Hello Andre. I've been setting up an EON server over the past few days and have come up with some stuff that might help out the documentation a little on the google site.

Send me a note through: http://anotherlittle.com/contact

if you are interested and I'll put together some stuff for you.

Егор said...

Greetings Andre, thanx for greate job!
Question about EON build script, where i can download latest ?
p.s. with ver 0.58.9 i have fatall error in boot process (svcs: device/local crash)and system can't load. i use sol-nv-b130-x86-dvd.iso for rebuild eon.
p.p.s i need add infiniband support (8 pkg's) for eon.

Егор said...

I find this bug =)
https://defect.opensolaris.org/bz/show_bug.cgi?format=multiple&id=11919

p.s.
when we can see latest build script ? :)

Andre Lue said...

Hi Erop,

The bug ID you linked, is unrelated. The script you are using is current. It sounds as if this is something introduced by the packages added, missing dependency or svc state at boot related to one of the added packages. It is hard to say.

I would suggest booting in verbose and in steps (single, multi, etc) to try and track down the error.

Егор said...

Im not customize build script, (just remove "#" symbol of grub_menu function in makecd.sh). My build machine is virtualbox (32 bit, 1.5Gb ram) now i try:
a) remove /usr/sbin/zfs volinit || exit $SMF_EXIT_ERR_FATAL
from devices-local
b) change build env to 64 bit
p.s.
how boot in "debug mode" ?
p.p.s
Andre, i want help for EON project, but how ?
i can:
-provide mirror for downloads (if need)
-translate documentation to russian
-contribute our builds of glusterfs for eon
maybe anything else ?

Егор said...

remove /usr/sbin/zfs volinit || exit $SMF_EXIT_ERR_FATAL
from devices-local
---- all work!

Andre Lue said...

Hi Erop,

Here's how to boot to specific levels for future use. Add -v for more verbosity.

Thanks for offering your assistance. For starters, I'm sure others would benefit a great deal from a glustferfs howto for EON.

Bang said...

Solaris Express 11 has been released. Is this relevant to EON project anymore?

Andre Lue said...

Hi Bang,

Solaris 11's current legalese does not allow this kind of use and distribution.

Drin said...

Hello Andre,

Any change on publishing the virtual machine image onto which you are building EON on illumos nightly build ?

Morpheus256 said...

I Really haven't wanted to ask this question but I'm going to. Is EON dead? FreeNas is really stepped up it's game from over a year ago when I wouldn't even consider it for ZFS but now they've made some huge advances with version 8. it seems that since oracle has come in, the BSD guys are doing more with ZFS now then any Solaris based project. I really liked EON when I started using it over a year ago, and I'm still running it on my current storage box, it just seems like other projects are making huge advances where we're standing still.

Andre Lue said...

Hi Morpheus256,

EON is still actively being developed. It's a good question. Just balancing a busy schedule. IPS packaging is the Greatest (awful) challenge right now.

As far as I know, FreeNas 8 is ZFS version 14. EON's last release is pool version 22 and zfs version 4, so if the features are like for like there still seems a way to go for FreeNAS ZFS features.

Is there a particular feature you are needing in the current pool:28 zfs:5 version?

ìgbàlonígbàńlò said...

Quick question. I tried to change the hostname via /etc/hosts and /etc/hostname.aggr1 (my bonded nics) but it borked on reboot.

Is there some other place I'm supposed to do this? Thanks

ìgbàlonígbàńlò said...

I want to kerberize the nfsv4 shares, in that regards do you happen to know what kerberos would require to setup eon as a client with host keytabs conf files etc on eon. I imagine there are packages to install, but i find there are krb5-* and solaris optimized(?) ones. Just trying to pick your brain to see How I should proceed with it. Thanks

ìgbàlonígbàńlò said...

Never mind. I wasn't looking hard enough. It's already in the binary kit it seems.

Andre Lue said...

Hi Igbalonigbanlo,

If you have the time, a write up on how to do this would surely benefit others. Feel free to send it to "eonstore AT gmail dot com"

ìgbàlonígbàńlò said...

I've abandoned it for now. The software was missing kadmin. Trying to install via IPS seemed like it was going to bring in a whole slew of packages I don't want to mess with now.

chroot ~/underverse said...

Hi Andre,

it turns out that rpcsec.so.1 is missing according to truss. Not yet there yet but once I do get it working, I'll send the how to.

ìgbàlonígbàńlò said...

My Kerberos experiment so far:

rpcsec.so.1 was missing and can be found in package SUNWrsg.

I unzipped the none archive with

7z e -o /tmp/ ./SUNWrsg/archive/none.7z

this generated a binary called none
I then ran:

cpio -idumBv < none

this extracted two files one to usr/lib/rpcsec.so.1 and the other to usr/lib/amd64/rpcsec.so.1

(similar to http://opensolaris.org/jive/thread.jspa?messageID=492891)

I then copied these files to my symlinked binary kit i.e. under /usr/local/lib and /usr/local/lib/amd64/ respectively.

I ran an ldd to get a list of the dependencies, they were already in the eon image.

So far that's seems to be the only thing missing on the eon side.

I still have issues enforcing a kerberos only nfs share since my client (fedora) throws kernel errors i.e.
gss_create: Pseudoflavor 390003 not found!

a cursory search seems to lay the blame at the feet of statically compiled kernel modules on the client. So I doubt if this has anything to do with eon.

ìgbàlonígbàńlò said...

Continuing the Kerberos experiment:

Seems ldapclient is not installed and as such, the storage server is not pulling down user information from the ldap server. That might be why sshd is not accepting kerberos principals, even though I can kinit as a kerberos user from eon and login to other systems via ssh. Also might explain why the nfsv4 server is not authorizing mounts by those kerberos users. I'll be hunting down which packages have ldapclient and what other config files or binaries it needs to make eon a fully working ldap client.

If I'm missing something and this is already possible, please let me know. Thanks.