For release 0.59.5 based on snv_125 I am thinking of adding 2 more images. A 64-bit CIFS and Samba version without any embedded web server. These images would be able add a web server based on lighttpd, if needed. I'd like to hear your thoughts on this? With the great news that ZFS de-duplication is complete, we can hope to see a SXCE release. Many thanks to Jeff Bonwick, Bill Moore, Mark Maybee, Matt Ahrens, Adam Leventhal, George Wilson, Robin Guo and all the engineers who made this happen.
I would also appreciate your feedback/review in format(pros:, cons:, and other thoughts:), to understand how EON ZFS storage is filling/failing your storage needs, thus far. I may not be able to address all needs but it would be great to understand how EON is working with you, whether it's good, bad or indifferent.
Update for future releases based on feedback:
- shift SUNWhd, S.M.A.R.T and rsync to the core image, instead of the binary kit.
- attempt to add more intel, broadcom nic drivers. (Note: not all drivers can be added because of image size contribution. Also I have no way of testing with certain hardware. There is a mechanism to add your own drivers)
- add 2 new 64-bit CIFS and Samba versions, with NO HTTP service. Will rely on external lighttpd, if HTTP service is needed.
- add COMSTAR, to replace iscsitgtd.
- documentation to help new to opensolaris, beginners and less technical audience. It would help greatly if you, the user would document or do write-ups, based on your experiences and challenges with EON. I could add the write-up to the wiki to help others. Proper credit will be given for your work.
Subscribe to:
Post Comments (Atom)
44 comments:
Hi Andre,
I just converted from 2009.06 to Eon 0.59.4. Thanks for all the effort you've put into it!
The big pro for me is just booting off flash. Lack of a web UI is not a big con for me, although I assume it would be the ultimate goal (i.e. FreeNAS level of ease of use).
My only big open problem right now is spinning down my SATA drives, which I think should be the default. With 2009.06, I had done this via ata.conf, but I don't know how to tweak that file and have it be preserved by updimg.sh
Changing power.conf is not sufficient, at least with my tests.
But as far as features, my bias would be towards bundling critical utilities like SMART monitoring and UPS support by default (e.g. SUNWhd?).
Mark,
Are you modifying?
/platform/i86pc/kernel/drv/ata.conf
If so, add it to /mnt/eon0/.backup and run updimg.sh
If not please give me some details to reproduce it and share the contents of the ata.conf file.
SMART and SUNWhd is available via the binary kit. Would you rather have it embedded?
EON's great!
I was originally going to use FreeNAS for my NAS box but now I'm definitely going with EON. ZFS is the key and I want it running native on OpenSolaris.
I've been testing EON out in a virtual machine and I really like the simplicity of it (once I knew what I was doing (I'm new to OpenSolaris)).
I'm sure I'll have questions and comments once the last of my hardware arrives (tomorrow?!).
Thanks for all the effort, Andre!
Max
PS - I'm looking forward to dedup, too!
Hi Andre,
thanks for the advice on ata.conf - I hadn't actually tried it as I had some weird idea that stuff under /platform/i86pc/kernel was somehow different - but I'm sure that will do the trick.
Do you have any sense whether adding swap space from my pool is likely to keep the disks from spinning down? Or worse, spinning up and down many times a day? Disabling the pool swap was the first thing I disabled after my power.conf tweaks didn't cause the drives to spin down.
Same goes for the SUNWhd stuff in the binary kit - I just hadn't dug through the blog posts yet to learn the steps to incorporate it.
I guess one related question is that pruning down the EON distribution to the bare minimum makes sense if it runs entirely out of ram disk. Are there other options akin to unionfs that would allow just the volatile stuff to be in ram, while the bulk of the distribution comes off flash (read-only), where we tends to have lots of available space? I'm a newbie and just babbling, but I'm sure that isn't a new idea to you and maybe be on your roadmap already.
Again, thanks for all you've done to make EON so useful! If you've still got a hardware wishlist I'd be happy to chip in.
Also, if anyone out there is using SATA port multipliers with something like an Atom mobo that only has 2 SATA ports, I'd love to hear about how well that works.
support the idea of using space on the usb to accommodate binary kit
dimsoft,
You have control of the symlinks/mount points so you could actually link the binary kit to USB vs zpool.
dimsoft,
If you link your binary kit to USB vs zpool, be aware of any high intensive writing/logging programs you may include that may cause early failure of the USB.
Hi Andre,
Thanks for doing a great work!
I'd like to see more drivers come with stock image, so I don't have to add them manually every time.
Web server/interface can wait, not a big deal for me.
Also including binary kit would be nice. It's only a few megs and with memory being dirt cheep these days, I don't see why not have all the helpful tools as part of the base image.
Thanks again,
Dmitry
how you can put ZIL on RAM disk?
dmirty,
The memory used translates to quite a bit more than it seems. For example, the 89Mb image is approx 300Mb uncompressed + runtime memory. It's best for performance reasons to keep EON as small as possible for the GA image. Also, the reason I provided the .purge feature to remove drivers not needed.
The binary kit on zpool allows a nice common core subset appliance + user customizable (blue sky) binary kit. Yes, it does also have some pluses if included in the image but that list could vary quite a bit.
I agree the webserver can wait but many users would disagree and it would be nice for the audience who would like ZFS storage with lower technical engagement. We will keep trying to improve EON.
What drivers do you add?
dimsoft,
It could be done using ramdiskadm but the ZIL device should:
-be placed on stable disks
-only be separate if they are faster than the zpool disks
Yes, ramdisks are faster but are they stable enough? The danger of living on a ramdisk needs careful research. Losing power would be??? Keep in mind we're storing users precious digital data and memories.
Mark,
I'm not sure if adding a ZFS swap or SUNWhd usage will prevent sleep as my test machine is older and doesn't properly support sleep but I'm going to say it should not.
I included powertop to help diagnose what events keep the machine awake. Maybe that will help with your sleep testing.
The .purge feature helps you to remove un-needed drivers. I have looked at making EON smaller using lofiadm and hsfs mounts but encountered pauses in performance that I was unable to figure out. I may revisit it but for now I'm focusing on continuity with IPS, which has lots of shortfalls in my opinion.
I believe the sata port multiplier used in Backblaze is a sil3124 chip and should work.
I appreciate and would accept your donation for development. However, I don't currently have a mechanism in place to accept donations but will update if I get one setup. Thanks for the offer.
Take a look at the MSI board on my Amazon dev list. I haven't tested/purchased it yet but my research says it's a good board. A little pricey but 4 sata ports and a slot to add a 4/8 port sata card. That's 8 to 12 possible disks.
Hi Andre,
I cannot setup 0.59.4 on HP servers. System dont see drivers E200 and P400 RAID controllers. Do you plan add there drivers in your system?
Thank you.
Andre,
thanks for the outstanding work.
I've setup eon 0.59.4 primarily for iscsi target use. Unfortunately the iscsi write speed with the native implementation is extremly slow (seems to be caused by the fix for bug 6770534). So I would be very happy to see the native implementation replaced by comstar, although it has a higher memory footprint.
appeared in open solaris ZFS Deduplication.
when it is in EON?
Is there a howto for building this stuff. I'm not a solaris person, The instructions seem to be scattered all over the place. I just want to build it for the machine I use since I have to be able to use the dvd writer otherwise the machine would be a dedicated storage setup. Thanks for your help.
Andrew,
Do you know what drivers are needed?
dimsoft,
De-duplication available possibly snv_128
igbaloni,
It may help if you read the Wiki or FAQ. In a nutshell you download one of the ISOs, burn it to CD, boot the CD, install it to USB or CF.
I`m newby for solaris and i dont know name of driver.
Controller P400 has chipset LSI.
chipset: LSISAS1078 C1
On E200 I dont know chipset type.
Hi Andre,
Usually it's igb ob bnx drivers, but I believe you've included these to the latest EON release. Personally I think all Intel, Broadcom and nVidia 100M/1G/10G drivers should be included by default, as pretty much all enterprise servers/boards have network chips from these three. I still think that getting one or two GB of RAM if not a big deal at all, but having not to do binary kit and drivers integration every time will be a huge plus.
Since you're thinking of doing two versions - with and without the web browser why not change the approach a little bit and do this - do one version as small as possible (basically the current version without we browser) and the other version with the web browser and most of the network/storage controller drivers included as well as binary kit, Thumper utils, etc.
Please let me know when you have mechanism for donation setup, so I can do my contribution.
Thanks again,
Dmitry
Hi Andre,
I have been trying out EON for a week now and I think it's great! Thank you very much for the effort you have put into this project!
It would be nice if the installer asked for an external email address that should get status messages if a pool gets degraded or a filesystem is filling up.
I like the idea of removing Apache. I would personally have much more use for the stuff in binary kit. Especially rsync and the Thumper smart tool.
Cheers,
Patrik
Andre:
You've managed to create an incredible piece of software. I pulled an old machine out of the closet, a Dell 650: dual Pentium-class Xeon 2.66 GHz, 3 GB of RAM; 4 x 73 GB 15K rpm old Seagate Cheetahs attached via U320 SCSI). Under OpenE, a while back, I got up to about 35 MB/sec (with an Adaptec hardware RAID controller). This time, I just plugged the 4 drives into the onboard SCSI controller, and used EON to create a ZFS RAIDZ. I could then copy files via NFS at an amazing 51 MB/sec, which is incredible considering the age of the hardware.
Pros: EON ZFS is just awesome. It works, as long as you know how to configure it. It's FAST, and ties in quite closely to the latest OpenSolaris developments of ZFS. This is a big thing, because if I'm not mistaken we'll be able to just type zpool upgrade -a to get everything up to revision 21 for deduplication, when that's available---and that will be even more awesome! (This is in contrast to what I understand about Nexenta, which is not running so close to the latest version of OpenSolaris; but correct me if I'm wrong). The other big plus is that it's free (though I'd be willing to contribute something if you set up an easy way to donate, once everything is working; see NFS problem below).
Cons: You have to understand ZFS and OpenSolaris to make this work. It's not that easy for beginners. This is good and bad, but coming from an Ubuntu background, it took me a few hours to make sense of everything. The other big-picture weakness is a lack of documentation; it's kind of assumed that you know and understand ZFS, but if you're just starting out, it's hard to know just what makes ZFS so much better for NAS (and why, for example, EON is far superior to running ZFS-FUSE internally, which I found to be only half as fast). I realize it might not be practical to re-state everything about ZFS and OpenSolaris, but a couple of well-written, well-linked webpages would be good: why ZFS? why OpenSolaris? why EON? It's obvious in hindsight, but hard for beginners.
The other big problem for me personally is getting NFS properly configured. Somehow I had it working for a few minutes, but then it died and never came back. Probably has to do something with the fact that DHCP failed, and maybe I got lucky with the static IP settings for a few minutes before the router did something... But I don't know anything about how to troubleshoot or fix DHCP in OpenSolaris, and if EON does anything differently. So that's my main problem.
Other thoughts: it would be good to have further guidance on optimizing performance. Is there much to be done for tuning, and if so, how? Again, it's probably obvious once you know this, but finding the resources to get started is a challenge for us beginners. It's not obvious, anywhere, if you should use SMB or CIFS, for example; or that NFS could be faster than both. And can you run both at the same time? But overall, a fantastic thing you've created!
how to find and edit .exec at a set EON
Peter,
Thanks for the detailed feedback. You are correct about EON and just running "zpool upgrade -a" when de-duplication becomes available.
I hear you on the documentation. If you have a write-up to share on your challenges in the form of guiding how you overcame your setup challenges, I'd be happy to post user experiences.
Nothing special or different about DHCP in EON vs opensolaris and you can tune anything in EON that you can in opensolaris. Networking as well as ZFS specific.
Basically, if you want to configure or setup anything included in EON you can start by googling how it's done in opensolaris. Thanks again.
hi Andre,
just started playing around with current eon 32bit cifs on an old machine, AMD K7, 1GB RAM, 2x 300GB mirror ZFS, 1x 200 GB for legacy ms-windows - need this approx. 1-2x per month for some old custom hardware.
I´m new to opensolaris, 12 years back I had some solaris and hp-ux experience.
apache can be removed for another version, no webinterface is not a problem.
SMART-monitoring should be included per default.
my big problem the board won´t boot from a kingston 512mb or a pny 4gb thumb drive - strange bios bug.
install.sh doesn´t allow installing on specific partition. so I cannot install on a free partition from the 200 gb windows-hdd, opensolaris has this possibility - will this available sometimes for eon ?
so my first try was do setup.sh, create zpool, add user, passwd chown, chgrp, bla-bla, cp -r /mnt/eon0 /mynewtank, running the updimg.sh for the new copy, replacing x86.eon in the iso, burn, boot, but ...
2 new problems
my zpool is not there - I have to do a forced import manual.
apache-svc is complaining because loghost is named eon in /etc/hosts and not like a named it during setup.sh. this problem should be resolved if I add /etc/hosts to /mynewtank/eon0/.backup ?
re-burning after every change is not my favorite choice, even if it works without the problems stated above.
so another idea, is it possible to install on an thumb drive (yes I can do that, but can´t boot) and copy the thumb drive (dd, clonezilla, etc.) to a free partition an my 200 GB windows-hdd ?
Hi Andre, excellent job on EON. Comstar for iscsi and ntfs-3g would be nice. I use an e-sata drive to backup my data it would be ideal if i could mount that drive in linux and windows if necessary. Or I guess I could convert my linux desktop to opensolaris and use zfs? :)
tlb,
There are no near plans to add the feature to install to a user chosen partition. After your dd attempt mentioned it sounds like you are only missing the installgrub step. You would need to run installgrub -m /path/stage1 /path/stage2 /dev/rdsk/cXdXsX, where cXdXsX (can be seen running format after booted from EON cd) is the target partition. You MUST be certain the target partition is correct or it may result in losing one of your installed OSes.
I have seen cases where certain machines that could boot from USB would not boot from certain thumb drives. In some cases if the thumb drive was plugged in shortly after power on, during the bios test, some worked. Unsure what causes this but its not really related to EON. More of hardware thing.
Regarding, /etc/hosts changes, it's a symlink to /etc/inet/hosts, put this in /mnt/eon0/.backup instead of /etc/hosts. That should also solve the apache-svc going into maintenance.
Jason,
COMSTAR replacing iscsitgtd will be attempted for next release. ntfs-3g will be tabled as a future possible add.
Hi Andre,
Great project and concept. In my particular case, the vanilla EON image lacks many drivers that my server needs (and SXCE / OS installed by default). (a bit too minimal :) ) .
As such, it would easier to build my own eon.iso using the imgsol.sh script than transplant necessary drivers/bins. However I can only locate the 104 version of the script. Can you post / point me in the direction of the latest imgsol.sh script that you have available?
Thank you.
Deep,
Current is the same version as 104 with the only differences being any drivers or packages that have been added or removed. Adding your drivers and packages using the latest SXCE DVD is all that is needed.
Hey Andre!
As I said before, I would find it very helpful to be alerted through email of a degraded pool. My suggestion would be a daily cron job that sends the output of "zpool status -x" to a user definable email address (in "/etc/eon.conf" perhaps?) unless it equals "all pools are healthy". Perhaps also a monthly job that sends a [Still alive and kicking] message regardless of pool states. Andre and everybody else, don't hesitate to tell me why you dislike this idea if you do. It would be very helpful since I am thinking of doing this to my own instance if no one else finds it a desirable feature.
OT: I saw some references in .backup to files which I assume belong to the Samba image although I'm running the CIFS image. No biggie, just thought I'd share.
Cheers,
Patrik
...another option for email alerts would be through a syslog filter I guess. It would provide instant feedback instead of the daily feedback i proposed with cron. The reason I prefered the cron way was that redirecting syslog messages would could become very "chatty" unless implemented carefully and it also didn't feel as KISS as using "zpool status -x".
Does "zpool status -x" return a different exit code than 0 if zpool errors are found? It would have felt even better than comparing string output. The man page for zpool only states a single error code for errors (apart from "invalid command line options") which makes me suspect that a degraded zpool is not really considered an error in the context of zpool exit codes.
Cheers,
Patrik
Patrik,
A mail option is a good idea there just currently isn't a clean solution because sendmail is not included in eon. MUA agents included(mailx, mail) have a dependency on sendmail.
The .backup entries work for both CIFS and Samba versions. The respective entry will only exist on the version you are using so if it does not exist will not pose any problems to your version. You are free, encouraged to customize /mnt/eon0/.files. Part of the reason they reside on the USB/CF and does not require an image update.
Andrew,
I did some research into the P400 controller and it seems it's incompatible. More here
Most documentation can probably be deferred to existing ZFS, OpenSolaris, and OS livecd documentation. However one component which seems to be specific to EON is updimg, and what we are really lacking is a broad picture of which things (assuming install to writeable medium) are persistent automatically and which require updimg to be run. For instance, newly created zfs filesystems and edits to .exec are automatically persistent, while a password change require updimg to be run. For a novice user there is no rhyme or reason to this. Explaining which parts of the filesystem reside in RAM vs disk, and giving a few examples of what does and doesn't require updimg, would help.
Also, putting some persistent link to useful posts in the blog sidebar and wiki would help make sure useful info like "Adding ZFS swap to EON" gets the attention deserved.
hi Andre,
the last days I became a PRO in slices, partitions, fdisk, grub, etc.
I have the wanted config, 2x old windows (w98se, xp) and EON and a part of a raidz-zpool on the same hdd along with other 2x 200 GB for the raidz.
I just ran setup and install.sh to the thumb drive, mounted the thumb drive again and my target (already fdisk'd 'n newfs), copied the files over, installed grub. done.
the machine boots headless from hdd into EON and depending on grub-menu on 2 different floppydisks into w98 oder xp.
I still have the missing automount issue of my zpool. I ran setup, set hostname, updated the image and even booted into this new config before creating my zpool, but after a reboot my zpool isn´t automounted.
automount-prop. is on and OpenSolaris should automount all zpools during boot - that is my understanding ?
yes there are two ways to fix this:
1) using zpool import mypool in .exec, works but this floods the zfs-history-log of my pool.
2) using mount + fstab with legacy-prop. - trying it the next days.
but I prefer a clean modern solution if it's possible.
zpool import says ... is available to import with name or id ..., I don´t need the -f switch.
zpool.cache is not really human readable but I found my zpoolname and hdd-info in it after command zpool create and still after reboot. EON doesn´t use this info ?
for people (yes me too, I have already plans for the 2nd EON - 64bit, 4gb, 3x500GB, 3x1.5TB, dual-intel-lan - after playing around with current old crappy hardware) who want using more than one EON on the same network, at least one has to be renamed e.g. via your setup-script, but even with fixed .backup (/etc/inet/hosts instead of /etc/hosts) after updimg.sh and reboot apache complains, because wrong or missing nodename in hosts.
fixing it is easy but the info should be included faq or gettings started on your google-site or a persistent link here like geoff already said.
ciao,
tlb
tlb,
The zpool not automounting is a known issue. I do not have yet have a fix why it does not automount. I believe it is a race condition. The workaround is uncommenting "zpool import -a" in /mnt/eon0/.exec.
To rename the host, simply add /etc/inet/hosts to .backup. I added to the capability to preserve symlinks and /etc/hosts is a symlink to /etc/inet/hosts, hence it only preserves the symlink. This is fixed, in EON 0.59.5 release.
The info is included in the wiki, FAQ and blog archive posts. There is a learning curve but as with all things, a little reading and patience is needed.
hi,
how to change keyboard in eos and load it map?
thanks!
Marco,
I think this is what you're after
kbd -s
then choose the country/layout.
A couple of questions:
1. When does /mnt/eon0/.exec execute? Could I put in those commands to start NFS there, and so it would automatically run every time?
2. I'm having a bit of trouble using zfs send|receive with eon. On my workstation, as a superuser, I execute:
"zfs send -i mydatapool@today | ssh root@192.168.1.102 zfs receive mybackupdata"
where 192.168.1.102 is the IP address of the EON storage server. If I run as admin@192.168.1.102, then it says "zfs: Command not found". If I try root@192.168.1.102, it doesn't let me login at all, even after I changed the option in /etc/ssh/sshd_config. I'm very confused as to how to get this to work properly.
3. Alternatively, I do have the same zfs filesystem mounted via NFS on my main workstation. I tried to rsync the two filesystems, running "rsync -av /filesystem_original /mnt/remote_data", where /mnt/remote data is the mountpoint for the filesystem on the EON server, connected via NFS. This gave a ton of Permission denied (13) errors for every file copy.
This is very frustrating, since all I want to do is use the EON server to backup my regular filesystems, and it's always kind of painful to figure this out for the first time!
4. IN other news, perhaps you could gain space back for the EON image by dropping CIFS/Samba altogether? Windows 7 has an NFS client by default (just enable in the control panel), so I can completely ignore CIFS/Samba in order to map my data on the EON server. You might free up space to put more of the goodies in the binary kit into such an image.
Thanks, as always. Did the SATA card work out?
Also, do you have a suggestion in terms of using zfs send/receive for backup, vs. rsync?
And is there a simple way to secure the NFS sharing? I've not had too much success coming up with a simple way to do this. Basically I'd like to just have a secure login; encrypting the transmission isn't so important. Thanks!
Peter,
1. /mnt/eon0/.exec gets executed at run level 3. Yes, you can add your commands there or call a script from it. It was designed for that.
2. The admin user does not have /usr/sbin in it's PATH /admin/.bashrc. So you could add /usr/sbin to the PATH= line. To allow root login edit
/etc/ssh/sshd_config
and change
PermitRootLogin no
to
PermitRootLogin yes
Be aware you are removing a layer of security.
3. Permission denied error kind of says it all. I can only suggest starting with a simple NFS setup (Example 2) on opensolaris as it is can get involved. Be patient with it, once you get it working it will run smooth, stable and solid.
4. Lotsa people use the CIFS feature. On yours you can use the purge feature to get rid of it if you wish. On the card, I am sourcing parts to build a little Franken storage and will post more about it when its complete. Thanks again for your contribution.
I think some documentation, and maybe performance status would be a good start to get users using this. I have little knowledge with opensolaris, therefore, if I am looking at products that have things like zfs, dedupe etc, I am looking for somthing that doesn't require me to know opensolaris. If I knew it well, I'd use it. Currently, I am a user of openfiler, and am looking at some alt options.
Roger,
Give the Wiki a look it should have enough to get you started. The enterprise class features included in EON are numerous and can be tweaked quite a bit, so documentation will be a work in progress. User's howto are more than welcome and would help others a great deal.
Peter,
After giving the permission denied error some thought. Try making sure that the user, nobody can write to the NFS share.
Hi Andre,
I'm pretty new to OpenSolaris, but I really wanted to try out ZFS.
Here's my notes so far:
1) My Intel T1080 Core Duo tends to stay in the highest speed P-state. So adding poll-mode to the cpupm line in /etc/power.conf has helped.
I don't know how to maintain this setting over reboot, so I copied it to the root of my pool, and modified /mnt/eon0/.exec to have a line to copy it over and then run /usr/sbin/pmconfig to make it active.
2) I'm using an Itox G5C100-N Mini-ITX board. It's got dual Yukon GigE ports. Highly recommended. $80 on Ebay :)
3) I've installed Binkit 124. I'm using EON 0.60/snv_130 in 32-bit CIFS. My clients are a Mac running 10.5 and a Mac running 10.4.
There seems to be a problem with 10.4 trying to browse CIFS. The only solution is to specify the share directly when trying to connect. Example:
cifs://192.168.2.205/kiddiepool_media
4) Attempts to get afpd to run haven't gone so well. It seems I'm missing libslp.so ? I don't know where to get it.
I also found out I needed to run crle -u -l /usr/local/lib in order to get the binkit's libraries into the search paths.
Thanks!
Hi Hanzelnut,
Thanks for the hardware compatibility feedback.
Regarding, preserving changes to /etc/power.conf, it's already in /mnt/eon0/.backup. So you would just need to run updimg.sh after editing it, to preserve the changes.
I have to release binary kit 130 as it has all the bits needed (libslp) to support the afpd and more. You shouldn't need to run cre to include /usr/local/lib as LD_LIBRARY_PATH should be in /etc/.bash_profile but what you did is perfectly fine.
I'm not sure what's going on with 10.4 and I do not have it to test. Do you notice any messages on the console when you try to connect from the 10.4 OS X machine?
Post a Comment