Benchmarking EON ZFS NAS performance (Part I)

At some point you may want to know if your EON ZFS storage is performing on par with your hardware's specifications/setup or if it is doing otherwise. Using various (prerequisite) binary kit tools we can identify if network and I/O performance are optimal or working within expectations. We will use the following tools for our testing
Iperf (Mac, Win)
IOMeter [Part II]
In most cases your storage is accessed/utilized across a network using a client PC. So testing will involve a source, destination and the network that connects them. The source will be your client PC and the destination will be your EON ZFS storage. This will help identify what the network is capable of, is it working properly or if it is a bottleneck to your storage.

On the EON ZFS storage (destination), connect 2 terminals using ssh or putty sessions from your client (Win XP in this case). In one terminal, run

iperf -s

In the other terminal, run (substitute your interface name for bge0)

dladm show-link bge0 -s -i 1

On the source or client PC we will run repetitions of below, changing TCP window size

iperf -c IP_of_EON -P 1 -i 1 -w 8k -f m -t 20

The above says, run 1 process thread P=1 for a test duration of t=20secs with interval i=1sec updates and a TCP window size w=8 in kilobytes. The results will be formatted in (m) megabytes.

This will produce an output showing the upper bounds of your network between the source and destination. It may also help identify the optimal TCP window size for your network. The output below will show the different transfer rates with each change of the TCP window size from 8k to 64k, on the client on a gigabit network. So lets begin.

C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 8k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.01 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3448 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  25.2 MBytes   211 Mbits/sec
[1912]  1.0- 2.0 sec  24.7 MBytes   207 Mbits/sec
[1912]  2.0- 3.0 sec  25.4 MBytes   213 Mbits/sec
[1912]  3.0- 4.0 sec  24.1 MBytes   202 Mbits/sec
[1912]  4.0- 5.0 sec  23.8 MBytes   200 Mbits/sec
[1912]  5.0- 6.0 sec  24.0 MBytes   201 Mbits/sec
[1912]  6.0- 7.0 sec  25.0 MBytes   210 Mbits/sec
[1912]  7.0- 8.0 sec  25.0 MBytes   210 Mbits/sec
[1912]  8.0- 9.0 sec  25.5 MBytes   214 Mbits/sec
[1912]  9.0-10.0 sec  24.1 MBytes   202 Mbits/sec
[1912] 10.0-11.0 sec  24.8 MBytes   208 Mbits/sec
[1912] 11.0-12.0 sec  25.1 MBytes   211 Mbits/sec
[1912] 12.0-13.0 sec  25.3 MBytes   212 Mbits/sec
[1912] 13.0-14.0 sec  23.1 MBytes   194 Mbits/sec
[1912] 14.0-15.0 sec  23.6 MBytes   198 Mbits/sec
[1912] 15.0-16.0 sec  23.6 MBytes   198 Mbits/sec
[1912] 16.0-17.0 sec  25.4 MBytes   213 Mbits/sec
[1912] 17.0-18.0 sec  25.6 MBytes   215 Mbits/sec
[1912] 18.0-19.0 sec  25.2 MBytes   211 Mbits/sec
[1912] 19.0-20.0 sec  25.0 MBytes   210 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  494 MBytes    207 Mbits/sec

Optional: Run on EON to compare network interface statistics

dladm show-link bge0 -s -i 1
LINK           IPACKETS  RBYTES   IERRORS  OPACKETS    OBYTES      OERRORS
bge0           19447     27628795 0        2432        155854      0
bge0           19089     27123001 0        2382        152546      0
bge0           19638     27886729 0        2450        156898      0
bge0           18652     26493270 0        2326        148962      0
bge0           18454     26205800 0        2301        147362      0
bge0           18561     26358926 0        2315        148258      0
bge0           19268     27372463 0        2404        153954      0
bge0           19379     27533495 0        2418        154850      0
bge0           19687     27977035 0        2457        157346      0
bge0           18681     26527515 0        2330        149218      0
bge0           19155     27208916 0        2390        153058      0
bge0           19419     27593183 0        2423        155170      0
bge0           19542     27757561 0        2438        156130      0
bge0           17840     25347308 0        2226        142562      0
bge0           18148     25776206 0        2264        144994      0
bge0           18353     26072730 0        2290        146658      0
bge0           19657     27919866 0        2452        157106      0
bge0           19756     28065895 0        2465        157858      0
bge0           19390     27542355 0        2419        154914      0
bge0           17889     25407434 0        2236        143362      0
bge0           38        8169     0        1           162         0

With TCP window set to 8K, Max throughput is approx 207Mbps/8bps=25.87MB/s

C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 16k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.02 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3487 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  40.1 MBytes   336 Mbits/sec
[1912]  1.0- 2.0 sec  31.1 MBytes   261 Mbits/sec
[1912]  2.0- 3.0 sec  36.3 MBytes   305 Mbits/sec
[1912]  3.0- 4.0 sec  43.2 MBytes   363 Mbits/sec
[1912]  4.0- 5.0 sec  43.2 MBytes   363 Mbits/sec
[1912]  5.0- 6.0 sec  41.5 MBytes   348 Mbits/sec
[1912]  6.0- 7.0 sec  41.9 MBytes   352 Mbits/sec
[1912]  7.0- 8.0 sec  29.6 MBytes   249 Mbits/sec
[1912]  8.0- 9.0 sec  33.2 MBytes   278 Mbits/sec
[1912]  9.0-10.0 sec  34.4 MBytes   288 Mbits/sec
[1912] 10.0-11.0 sec  37.5 MBytes   315 Mbits/sec
[1912] 11.0-12.0 sec  41.4 MBytes   348 Mbits/sec
[1912] 12.0-13.0 sec  43.0 MBytes   360 Mbits/sec
[1912] 13.0-14.0 sec  30.9 MBytes   259 Mbits/sec
[1912] 14.0-15.0 sec  39.2 MBytes   329 Mbits/sec
[1912] 15.0-16.0 sec  42.9 MBytes   360 Mbits/sec
[1912] 16.0-17.0 sec  42.9 MBytes   360 Mbits/sec
[1912] 17.0-18.0 sec  40.6 MBytes   340 Mbits/sec
[1912] 18.0-19.0 sec  37.2 MBytes   312 Mbits/sec
[1912] 19.0-20.0 sec  24.7 MBytes   207 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  755 MBytes    317 Mbits/sec

With TCP window set to 16K, Max throughput is approx 317Mbps/8bps=39.63MB/s

C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 24k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.02 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3497 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  53.4 MBytes   448 Mbits/sec
[1912]  1.0- 2.0 sec  50.2 MBytes   421 Mbits/sec
[1912]  2.0- 3.0 sec  40.8 MBytes   343 Mbits/sec
[1912]  3.0- 4.0 sec  62.5 MBytes   524 Mbits/sec
[1912]  4.0- 5.0 sec  63.9 MBytes   536 Mbits/sec
[1912]  5.0- 6.0 sec  63.4 MBytes   532 Mbits/sec
[1912]  6.0- 7.0 sec  64.4 MBytes   540 Mbits/sec
[1912]  7.0- 8.0 sec  58.7 MBytes   493 Mbits/sec
[1912]  8.0- 9.0 sec  48.0 MBytes   403 Mbits/sec
[1912]  9.0-10.0 sec  62.2 MBytes   522 Mbits/sec
[1912] 10.0-11.0 sec  54.1 MBytes   454 Mbits/sec
[1912] 11.0-12.0 sec  53.9 MBytes   452 Mbits/sec
[1912] 12.0-13.0 sec  56.8 MBytes   477 Mbits/sec
[1912] 13.0-14.0 sec  57.6 MBytes   483 Mbits/sec
[1912] 14.0-15.0 sec  49.1 MBytes   412 Mbits/sec
[1912] 15.0-16.0 sec  65.0 MBytes   545 Mbits/sec
[1912] 16.0-17.0 sec  64.2 MBytes   539 Mbits/sec
[1912] 17.0-18.0 sec  64.6 MBytes   542 Mbits/sec
[1912] 18.0-19.0 sec  64.0 MBytes   537 Mbits/sec
[1912] 19.0-20.0 sec  57.5 MBytes   483 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1154 MBytes   484 Mbits/sec

With TCP window set to 24K, Max throughput is approx 484Mbps/8bps=60.5MB/s

C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 32k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.03 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3500 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  77.0 MBytes   646 Mbits/sec
[1912]  1.0- 2.0 sec  78.2 MBytes   656 Mbits/sec
[1912]  2.0- 3.0 sec  48.9 MBytes   410 Mbits/sec
[1912]  3.0- 4.0 sec  76.6 MBytes   642 Mbits/sec
[1912]  4.0- 5.0 sec  75.8 MBytes   636 Mbits/sec
[1912]  5.0- 6.0 sec  79.4 MBytes   666 Mbits/sec
[1912]  6.0- 7.0 sec  78.8 MBytes   661 Mbits/sec
[1912]  7.0- 8.0 sec  79.0 MBytes   663 Mbits/sec
[1912]  8.0- 9.0 sec  66.7 MBytes   559 Mbits/sec
[1912]  9.0-10.0 sec  81.0 MBytes   679 Mbits/sec
[1912] 10.0-11.0 sec  76.8 MBytes   644 Mbits/sec
[1912] 11.0-12.0 sec  79.5 MBytes   667 Mbits/sec
[1912] 12.0-13.0 sec  74.1 MBytes   621 Mbits/sec
[1912] 13.0-14.0 sec  72.9 MBytes   612 Mbits/sec
[1912] 14.0-15.0 sec  70.0 MBytes   587 Mbits/sec
[1912] 15.0-16.0 sec  79.1 MBytes   663 Mbits/sec
[1912] 16.0-17.0 sec  79.6 MBytes   668 Mbits/sec
[1912] 17.0-18.0 sec  79.6 MBytes   668 Mbits/sec
[1912] 18.0-19.0 sec  79.4 MBytes   666 Mbits/sec
[1912] 19.0-20.0 sec  77.7 MBytes   652 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1510 MBytes   633 Mbits/sec

With TCP window set to 32K, Max throughput is approx 633Mbps/8bps=79.13MB/s

C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 48k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.05 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3506 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec   101 MBytes   846 Mbits/sec
[1912]  1.0- 2.0 sec   102 MBytes   852 Mbits/sec
[1912]  2.0- 3.0 sec  84.9 MBytes   712 Mbits/sec
[1912]  3.0- 4.0 sec  81.7 MBytes   686 Mbits/sec
[1912]  4.0- 5.0 sec  94.7 MBytes   794 Mbits/sec
[1912]  5.0- 6.0 sec   103 MBytes   860 Mbits/sec
[1912]  6.0- 7.0 sec   102 MBytes   859 Mbits/sec
[1912]  7.0- 8.0 sec   102 MBytes   852 Mbits/sec
[1912]  8.0- 9.0 sec   101 MBytes   850 Mbits/sec
[1912]  9.0-10.0 sec  90.6 MBytes   760 Mbits/sec
[1912] 10.0-11.0 sec  82.6 MBytes   693 Mbits/sec
[1912] 11.0-12.0 sec   101 MBytes   848 Mbits/sec
[1912] 12.0-13.0 sec  85.5 MBytes   717 Mbits/sec
[1912] 13.0-14.0 sec  85.8 MBytes   719 Mbits/sec
[1912] 14.0-15.0 sec  91.5 MBytes   767 Mbits/sec
[1912] 15.0-16.0 sec  91.8 MBytes   770 Mbits/sec
[1912] 16.0-17.0 sec  89.7 MBytes   753 Mbits/sec
[1912] 17.0-18.0 sec   102 MBytes   858 Mbits/sec
[1912] 18.0-19.0 sec  97.9 MBytes   821 Mbits/sec
[1912] 19.0-20.0 sec   100 MBytes   841 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1891 MBytes   793 Mbits/sec

With TCP window set to 48K, Max throughput is approx 793Mbps/8bps=99.13MB/s

C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 64k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.06 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3538 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  98.2 MBytes   823 Mbits/sec
[1912]  1.0- 2.0 sec  66.3 MBytes   556 Mbits/sec
[1912]  2.0- 3.0 sec   101 MBytes   844 Mbits/sec
[1912]  3.0- 4.0 sec  99.8 MBytes   837 Mbits/sec
[1912]  4.0- 5.0 sec  99.5 MBytes   835 Mbits/sec
[1912]  5.0- 6.0 sec  79.7 MBytes   668 Mbits/sec
[1912]  6.0- 7.0 sec  76.6 MBytes   643 Mbits/sec
[1912]  7.0- 8.0 sec  94.1 MBytes   790 Mbits/sec
[1912]  8.0- 9.0 sec   101 MBytes   846 Mbits/sec
[1912]  9.0-10.0 sec   100 MBytes   840 Mbits/sec
[1912] 10.0-11.0 sec  96.7 MBytes   811 Mbits/sec
[1912] 11.0-12.0 sec  96.0 MBytes   805 Mbits/sec
[1912] 12.0-13.0 sec  94.0 MBytes   789 Mbits/sec
[1912] 13.0-14.0 sec  97.2 MBytes   816 Mbits/sec
[1912] 14.0-15.0 sec  98.5 MBytes   827 Mbits/sec
[1912] 15.0-16.0 sec  81.7 MBytes   686 Mbits/sec
[1912] 16.0-17.0 sec  74.6 MBytes   626 Mbits/sec
[1912] 17.0-18.0 sec  79.8 MBytes   669 Mbits/sec
[1912] 18.0-19.0 sec  98.1 MBytes   823 Mbits/sec
[1912] 19.0-20.0 sec  99.7 MBytes   837 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1832 MBytes   768 Mbits/sec

With TCP window set to 64K, Max throughput is approx 768Mbps/8bps=96MB/s.

The important things to note from these tests between a client (Win XP) and EON ZFS NAS server on this network are:

• Possible upper limit near 99.13MBytes/sec (not the theoretical 1000Mbps/125MBytes/sec).
• The optimal TCP window size for this network is probably 48K (should have tested 56K also).
• A larger TCP window size of 64K did not provide the highest transfer rate.

EON ZFS takes the road to Illumos

By now Oracle's leaked letter and the fate of Opensolaris, source gates closing, is probably well known. It seems that Solaris 11 will be Oracle's sole OS focus, and while it's unknown if it will be open source or not, the opensolaris community and it's users were ignored, abandoned and left facing a nasty dose of hard realities.

The options left for the opensolaris community (including EON ZFS Storage) were bleak. Sun Microsystems was built on great innovation and thanks to a previous Solaris kernel engineer Garrett D'Amore and the open-source effort called illumos, there is new hope. It's stated that illumos will be 100% community driven and owned and many smart people who are committed to opensolaris and ZFS have signed on to breathe life into this effort and free our reliance from Oracle's handouts.

EON is testing the road to illumos and I have compiled the nightly a couple of times. Some build times:

297mins/~5 hrs    Quad core Q6600 @ 2.4Ghz w/2GB RAM
52mins/~1 hr      Dual X5570 @ 2.93Ghz w/4GB RAM

The journey should be interesting ...

Mediatomb UPnP server on your EON ZFS NAS

MediaTomb is an open source (GPL) UPnP MediaServer with a nice web user interface. It allows you to stream your digital media across your home network and listen to/watch it on a variety of UPnP compatible devices. Mediatomb is now available for your EON ZFS NAS in the download section (mtombaa, mtombab) and can be added using the following steps. The binary kit MUST be installed for mediatomb to work.

So let's get started by downloading and re-assembling the mediatomb package.
NOTE: The following steps should be performed as user id "root" unless otherwise specified.

cd /tmp
wget -O mtombaa http://sites.google.com/site/eonstorage/downloads/mtombaa?attredirects=0&d=1
wget -O mtombab http://sites.google.com/site/eonstorage/downloads/mtombab?attredirects=0&d=1
cat mtomba[a-z] > mtomb.tgz

Next, let's create the necessary directories and unpack the mediatomb package.

cd /abyss
mkdir mediatomb
cd mediatomb
gzip -dc /tmp/mtomb.tgz | tar -xf -

Let's add the following entries to /mnt/eon0/.exec, so mediatomb will start automatically when the EON ZFS NAS is booted.

# mediatomb section
(cd /opt ; ln -s ../${POOL}/mediatomb .)
[ -x /opt/mediatomb/bin/start_mtomb.sh ] && /opt/mediatomb/bin/start_mtomb.sh

Let's create the mediatomb logging, database and configuration directories. Also get the start up script (start_mtomb.sh)

mkdir etc log
cd bin
wget -O start_mtomb.sh http://sites.google.com/site/eonstorage/downloads/start_mtomb.sh?attredirects=0&d=1
chmod 755 start_mtomb.sh
(cd /opt ; ln -s ../abyss/mediatomb .)

And finally, to start mediatomb.

/opt/mediatomb/bin/start_mtomb.sh

If all went well, you can point a browser to EON NAS http://ip:49152/ and you should see a image similar to the image below. Let the movie streaming begin.

Mediatomb UPnP server on EON ZFS NAS

Compiling rsync and your own binaries for EON

You have probably noticed xsconf.program (eg xsconf.rsync) name in the download section and wondered how it's used? Here's a video that will hopefully answer that and help if you wish to compile your own binaries (eg rsync) or other open source packages. The notes.program name (eg notes.rsync) contains where to get the source code and any notes/tips/edits needed for a successful compile. A pre-requisite for this exercise is a sunstudio 12 and/or gcc compiler. After the compilers are downloaded and installed on a solaris/unix system you can follow the steps in the video. Most likely you will need both sunstudio and gcc, as some code will not properly compile with the Sunstudio compiler. To build the best optimized binaries, I personally always try to use the SunStudio compiler and linker. Enough of the boring details, on with building a 64-bit version of rsync. Happy compiling ...

EON 0.60 ZFS binary kit snv_130 released!

Here it is finally, the long awaited EON 0.60 binary kit release based on snv_130 and other GNU compiled binaries. Hopefully, it was well worth the wait. I tried to complete all the requests for various packages but was unable to integrate them all cleanly (sabnzbd + dependency python) in this release. I will try to get the missed ones at a later time. It is uploaded in 5 parts. You can download bin-130a[a-e] manually and transfer all 5 files to your EON storage for installation or follow the steps below, which assumes you are using wget from an prior installed binary kit 124. So let's retrieve the files. These steps also assume your EON storage can reach the web.

cd /tmp
wget -O binkit-130.tgz http://i.minus.com/1323745623/7Aw7618NGr6qYvgEXEpkWw/dZLxUEf3SOTLh.tgz

If this is the first time using a binary kit, you will need to perform this step. If you have used previous binary kits, you can skip to the next step.

cd /zpool
mkdir local

This step installs the new binary kit to /usr/local which is linked to /zpool_name/local.

cd /usr/local
gzip -dc /tmp/binkit-130.tgz | tar -xf -

This completes the binary kit install and typing "aria2c -v" should confirm if it worked. Below is a screenshot listing some of the binaries included:

Binary Kit 130 summary:

• PHP/libphp 5.3.1 (GD, jpeg, png support)
• Aria2c 1.9.3 (HTTP/HTTPS, FTP, BitTorrent and Metalink) 
• tmux 1.1 (terminal multiplexer)
• screen (terminal multiplexer)
• wget/curl (retrieving files using HTTP, HTTPS and FTP)
• glusterfs 3.0 (clustered storage solution)
  
• stunnel (encrypt TCP connections inside SSL)
  
• socat (multi-purpose relay)
  
• bonnie/iperf (disk / network benchmarking)
  
• nano (editor)
  
• elinks (text browser)
  
• ddclient/inadyn (dynamic DNS clients)
  
• par2 (parity archive volume set)
  

This does not show powertop and top which have to be manually symlinked for now. See the notes regarding top here. The same is needed for powertop. For 32 bit EON version (To check the EON version you are running, "isainfo -kv"):

cd /usr/local/bin
ln -s ./i86/powertop .

For 64 bit EON version:

cd /usr/local/bin
ln -s ./amd64/powertop .

EON on a ZFS mirrored boot/root?

A typical operating system requires a boot disk and certain storage requirements to reside, boot and do what it's designed to do. The OS or boot disk where the operating system resides in a conventional install becomes a possible point of failure. The OS disk failing, would cause an outage and cut-off access to your data and storage appliance. RAID 1 or mirroring of the OS disk is often used to address/counter this type of failure. ZFS has the features to make an OS disk redundant by mirror-ing (RAID 1), providing an added layer of insurance to your storage appliance's uptime.

A design decision with EON, was to make it run from a RAM disk, hence it would not have an associated failure attached to the media(USB, CF or hard disk) it boots from or resides on. Size becomes a disadvantage running from a RAM disk but the trade off is an add layer of insurance, resilience and uptime. EON uses the boot/residence media mostly for out of band task that are not tied to performance and are not frequently used, so the typical 5+ MB/s performance of a USB/CF media is not a problem. After EON is booted, the USB media could be pulled (simulating a boot disk failure) and only the services below would be impacted. The important thing to note is that EON would continue to run, along with all the services enabled. The show goes on and facilitates dynamic repair or correction of the disk failure situation. EON uses the USB/CF media to preserve:

• edits to the configuration files
• edits to the grub menu entries or updating the image
• updates to the zpool.cache
• user optional binaries on the USB/CF media 

A ZFS boot/root was recently requested as a necessary feature/requirement for EON. I explained the resilience already built into the design and the challenges. I however gave it try the other night and succeeded in booting EON on a ZFS boot/root. This would allow the use of redundant USB/CF media. When will it be seen in a release? I'm not sure. It required using steps that were hard coded and needs more testing and refinement before it's ready for general audience and use. So yes, a ZFS boot/root for EON is possible. Here's a ZFS boot/root birth cry screenshot.

EON ZFS Storage 0.60.0 based on snv 130, Sun-set release!

Embedded Operating system/Networking (EON), RAM based live ZFS NAS appliance is released on Genunix! This release marks the end of SXCE releases and Sun Microsystems as we know it! It is dubbed the Sun-set release! Many thanks to Al at Genunix.org for download hosting and serving the Opensolaris community.

New to EON, start here! Upgrading, see how to use transporter.sh

EON Deduplication ZFS storage is available in 32 and 64-bit, CIFS and Samba versions:
EON 64-bit x86 CIFS ISO image version 0.60.0 based on snv_130

• eon-0.600-130-64-cifs.iso
• MD5: 55c5837985f282f9272f5275163f7d7b
• Size: ~93Mb
• Released: Monday 05-April-2010

EON 64-bit x86 Samba ISO image version 0.60.0 based on snv_130

• eon-0.600-130-64-smb.iso
• MD5: bf095f2187c29fb543285b72266c0295
• Size: ~106Mb
• Released: Monday 05-April-2010

EON 32-bit x86 CIFS ISO image version 0.60.0 based on snv_130

• eon-0.600-130-32-cifs.iso
• MD5: e2b312feefbfb14792c0d190e7ff69cf
• Size: ~59Mb
• Released: Monday 05-April-2010

EON 32-bit x86 Samba ISO image version 0.60.0 based on snv_130

• eon-0.600-130-32-smb.iso
• MD5: bcf6dc76bc9a22cff1431da20a5c56e2
• Size: ~73Mb
• Released: Monday 05-April-2010

EON 64-bit x86 CIFS ISO image version 0.60.0 based on snv_130 (NO HTTPD)

• eon-0.600-130-64-cifs-min.iso
• MD5: f5922c98888521e4d8bef10133ccbe40
• Size: ~87Mb
• Released: Monday 05-April-2010

EON 64-bit x86 Samba ISO image version 0.60.0 based on snv_130 (NO HTTPD)

• eon-0.600-130-64-smb-min.iso
• MD5: e74732c41e4b3a9a06f52779bc9f8352
• Size: ~101Mb
• Released: Monday 05-April-2010

New/Changes/Fixes:
- Active Directory integration problem resolved
- Hotplug errors at boot are being worked on and are safe to ignore.
- Updated /mnt/eon0/.exec with new service configuration additions (light, nginx, afpd, and more ...).
- Updated ZFS, NFS v3 performance tuning in /etc/system
- Added megasys driver.
- EON rebooting at grub(since snv_122) in ESXi, Fusion and various versions of VMware workstation. This is related to bug 6820576. Workaround, at grub press e and add on the end of the kernel line "-B disable-pcieb=true"

What's the best pool to build with 3 or 4 disks?

I've been asked many times in variations, "I just started using EON and I'm new to opensolaris, What's the best pool to build with 3 or 4 disks"? I usually answer, it depends! Credit that reflex answer to Prof. Gordon, one of the best Calculus and Differential equations teacher that walked in my time. May the force be with you, wherever you are!

I'll use Richard Elling's research to explain. Let's say I have 500Gb drives, with IOPs (for avg, small, random, cache-miss, read I/O operations per sec) = 70.59 and max media bandwidth of 133Mbytes/s(includes read and write). What can we build?

RAID Type   Disks   Sets   Storage Space   Performance (IOPS)   Max BW(Mbytes/s)
RAIDZ       4       1      3x500=1500Gb    4x70.59/3=94         3x133=399
RAIDZ       3       1      2x500=1000Gb    3x70.59/2=106        2x133=266 (1 spare)
STRIPE      4       1      4x500=2000Gb    4x70.59=282          4x133=532
STRIPE      3       1      3x500=1500Gb    3x70.59=212          3x133=399
MIRROR      2       2      2x500=1000Gb    4x70.59=282          4x133=532

RAIDZ
With 4 disks in the first raidz set, we get higher bandwidth (399Mbytes/s) vs the 3 disk raidz bandwidth (266Mbytes/s), but the 3 disk raidz pool has a higher I/O operations per second capability. Note, as "sets" are added to the 3 disk raidz (3 disks each time) the difference of IOPS between the 4 disk raidz widens. If you exhaust the usable storage space, it will cost 4 or 3 times the cost of a drive for each new "Set", to add or grow the storage. So the 3 drive raidz has a more economical cost per set. This can be repeated to add more "Sets" or more storage and bandwidth as needed. So this is a very flexible choice. The change with 1 additional set would look like.

RAID Type   Disks   Sets   Storage Space   Performance (IOPS)   Max BW(Mbytes/s)
RAIDZ       4       2      3000Gb          188                  798
RAIDZ       3       2      2000Gb          212                  532

Both 4 and 3 disk raidz allows only 1 disk to fail but if all disks had the same probability of failure then the 4 disk raidz pool would have a higher probability of a failure than the 3 disk version.

STRIPE
Has great bandwidth numbers, usable storage and IOPS, but any disk failure would cause the pool to fail and lose ALL your data. Did I mention that good storage is NOT a substitute for a GOOD backup? This pool is not easily expanded when the usable storage is exhausted and offers no data redundancy.

MIRROR
Has great bandwidth numbers, a higher cost per usable storage and allows failure of 2 disks. It has roughly twice the write bandwidth and up to 4 times the read performance as ZFS is capable of reading from all disks in the mirror in parallel. This configuration will most likely provide the best balance of performance and data protection at the expense of disks or usable storage. Expanding or growing this pool when the usable storage is being exhausted, is also simple.


Hopefully this will help architect pools that suits your workload, cost dynamics and growth needs.

Traffic (QoS) control built into your EON ZFS storage

Your EON ZFS storage provides access to a lot of services, such as HTTP, HTTPS, SFTP, Firefly/daapd, AFP(AppleShare) and more. All of these services are available as a network resource. Wouldn't it be nice to be able to control or manage how different systems use these network resources, such as bandwidth?

Project Crossbow provides the controls to manage and virtual-ize network resources. The traffic controls (QoS) can be used to manage by transport (TCP, UDP, SCTP, iSCSI, etc), bandwidth limits, IP address and more.

For example, one could simply limit the amount of bandwidth the HTTP, HTTPS or SSH service can utilize. You could create virtual nics bound to your real interface and provide different levels of service and bandwith to these virtual or real nics. It allows for a very flexible storage setup where you can really manage the traffic and quality of service it delivers.

Let's do a simple 10Mbps bandwidth limit for HTTP via interface bge0 for a flow we will label httpflow. First, we create a flow that matches the HTTP service

flowadm add-flow -l bge0 -a transport=tcp,local_port=80 httpflow

Let's view it

flowadm show-flow

Finally, let's set bandwidth limits

flowadm set-flowprop -p maxbw=10m httpflow

To verify the properties

flowadm show-flowprop

To show traffic usage

flowadm show-usage

Accounting can also be setup to record the usage. Rather than rehash the numerous possibilities, here are 2 links that details this feature fairly well. The first is written by Ben Rockwood and the other can be found here. Traffic control ... out!

Creating a EON ZFS storage self signed certificate

Your EON ZFS storage is available with 3 web server options, apache2(default), lighttpd, and nginx. They are provided with a self signed certificate that I created to simplify configuration and ease getting them started. The proper thing to do is replace it with your own certificate or create a self signed replacement.

Prerequisite(s): This requires the binary kit or an alternate system with a working openssl, to create your own self signed certificate.
The "\" means the command is one continuous line to create the self signed certificate. Feel free to experiment and substitute your own arguments for:
days = set to 365 x 10 = 3650 or ~10 yrs
CN = $HOSTNAME = replace with your own storage hostname
O = Gibraltar Engineering
OU = EON Secure Certificate

openssl req -x509 -nodes -days 3650 -newkey rsa:1024 -keyout cert.key -out cert.pem \
-subj "/C=US/ST=NY/CN=$HOSTNAME/O=Gibraltar Engineering/OU=EON Secure Certificate"

This will produce 2 files cert.key and cert.pem. For Apache2 they are configured and the proper permissions set as follows:

cp cert.pem /etc/apache2/2.2/server.crt
cp cert.key /etc/apache2/2.2/server.key
chmod 400 /etc/apache2/2.2/server.crt
chmod 400 /etc/apache2/2.2/server.key

For lighttpd the pem and key file are concatenated to create a single file

cp cert.pem /your_pool/lighttpd/etc
cat cert.key >> /your_pool/lighttpd/etc/cert.pem
chmod 400 /your_pool/lighttpd/etc/cert.pem

For nginx

cp cert.pem /your_pool/nginx/conf
cp cert.key /your_pool/nginx/conf
chmod 400 /your_pool/nginx/conf/cert.pem
chmod 400 /your_pool/nginx/conf/cert.key

The SUN has set, for the last time!

On January 27th at approximately 9:50am, Oracle completed it's acquisition of Sun Microsystems for 7.4 billion all cash. The price is quite a steal considering Sun's history of innovations. It is also personally quite sad to see one of the greatest engineering companies in my lifetime, like a beautiful descending Sun-set, descend beyond the horizon.

I was introduced to SunOS over 15 years ago when I had to take an electrical engineering class (Feedback Control Systems) that required running simulations on MatLab. They were the early days of the world wide web and the beginning of an engineering love/career. My internships, from there on were Sun and Solaris related. One beautiful summer, I went to The Herff College of Engineering in Memphis to learn and port a Cochlear model written in C by Dr. T Wilson, to Java. Without plans to do so, I made a career on something I learned and loved, as a sidebar tool. That tool was engineered by Sun Microsystems and for this I humbly say, THANKS for the great ride and memories, ... thanks to everyone who made Sun Microsystems what it became, is, was. It will be missed ...

EON ZFS Storage min and nginx web server

In a previous post, lighttpd was used as a add in web server for the EON ZFS min storage (NO HTTPD) users. Another web server option for the ZFS storage (NO HTTPD) image is nginx. This post will show how to easily add HTTPD service using the nginx web server. 

Prerequisite(s): A storage pool must be created and a binary kit must be installed. The binary kit is needed because it contains libpcre, a dynamic library needed by nginx.

Let's get started adding a nginx web server, to our ZFS min (NO HTTPD) storage image. First, get the pre-built nginx version 0.8.32 here. I have also included compilation notes under the nginx section in case you wish to compile and add other features on your own. Transfer the nginx-0,8.32,tgz file to the storage pool via CIFS share, sftp or USB. Here, I start with the file located in /tmp and a storage pool named abyss. Substitute your zpool name for abyss.

cd /abyss
gzip -dc /tmp/nginx-0.8.32.tgz | tar -xf -

Now let's add the automation entries to /mnt/eon0/.exec. Also, execute the commands to create the necessary symlinks. These entries align paths entered in nginx.conf

(cd /usr ; ln -s ../abyss/nginx .)

At this stage, the nginx web server is ready to run, but feel free to customize the /abyss/nginx/conf/nginx.conf and generate your own /abyss/nginx/conf/cert.pem, /abyss/nginx/conf/cert.key file. OpenSSL (part of the binary kit which is a pre-requisite for this post) would be needed to generate your own self signed cert file (cert.pem, cert.key). To start and automate the web server, run and add the following to /mnt/eon0/.exec.

/usr/nginx/sbin/nginx -c /usr/nginx/conf/nginx.conf

EON ZFS Storage min and lighttpd web server

If you run one of the EON ZFS (NO HTTPD) storage images that do not include a web server this post will show how to easily add HTTPD service using the lighttpd or nginx web server. 

Prerequisite(s): A storage pool must be created and a binary kit must be installed. The binary kit is needed because it contains libpcre, a dynamic library needed by lighttpd.

Let's get started adding a lighttpd web server, to our ZFS min (NO HTTPD) storage image. First, get the pre-built lighttpd version 1.4.25 here. I have also included compilation notes under the lighttpd section in case you wish to compile and add other features on your own. Transfer the lighttpd-1.4.25.tgz file to the storage pool via CIFS share, sftp or USB. Here, I start with the file located in /tmp and a storage pool named abyss. Substitute your zpool name for abyss.

cd /abyss
gzip -dc /tmp/lighttpd-1.4.25.tgz | tar -xf -

Now let's add the automation entries to /mnt/eon0/.exec. Also, execute the commands to create the necessary symlinks. These entries align paths entered in lighttpd.conf

(cd /usr ; ln -s ../abyss/lighttpd .)
(cd /var ; ln -s ../abyss/lighttpd .)

At this stage, the lighttpd web server is ready to run, but feel free to customize the /abyss/lighttpd/etc/lighttpd.conf and generate your own /abyss/lighttpd/etc/server.pem file. OpenSSL (part of the binary kit which is a pre-requisite for this post) would be needed to generate your own self signed cert file (server.pem). To start and automate the web server, run and add the following to /mnt/eon0/.exec.

/usr/lighttpd/sbin/lighttpd -f /usr/lighttpd/etc/lighttpd.conf