Iperf (Mac, Win)
IOMeter [Part II]
In most cases your storage is accessed/utilized across a network using a client PC. So testing will involve a source, destination and the network that connects them. The source will be your client PC and the destination will be your EON ZFS storage. This will help identify what the network is capable of, is it working properly or if it is a bottleneck to your storage.
On the EON ZFS storage (destination), connect 2 terminals using ssh or putty sessions from your client (Win XP in this case). In one terminal, run
iperf -sIn the other terminal, run (substitute your interface name for bge0)
dladm show-link bge0 -s -i 1On the source or client PC we will run repetitions of below, changing TCP window size
iperf -c IP_of_EON -P 1 -i 1 -w 8k -f m -t 20The above says, run 1 process thread P=1 for a test duration of t=20secs with interval i=1sec updates and a TCP window size w=8 in kilobytes. The results will be formatted in (m) megabytes.
This will produce an output showing the upper bounds of your network between the source and destination. It may also help identify the optimal TCP window size for your network. The output below will show the different transfer rates with each change of the TCP window size from 8k to 64k, on the client on a gigabit network. So lets begin.
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 8k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.01 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3448 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 25.2 MBytes 211 Mbits/sec [1912] 1.0- 2.0 sec 24.7 MBytes 207 Mbits/sec [1912] 2.0- 3.0 sec 25.4 MBytes 213 Mbits/sec [1912] 3.0- 4.0 sec 24.1 MBytes 202 Mbits/sec [1912] 4.0- 5.0 sec 23.8 MBytes 200 Mbits/sec [1912] 5.0- 6.0 sec 24.0 MBytes 201 Mbits/sec [1912] 6.0- 7.0 sec 25.0 MBytes 210 Mbits/sec [1912] 7.0- 8.0 sec 25.0 MBytes 210 Mbits/sec [1912] 8.0- 9.0 sec 25.5 MBytes 214 Mbits/sec [1912] 9.0-10.0 sec 24.1 MBytes 202 Mbits/sec [1912] 10.0-11.0 sec 24.8 MBytes 208 Mbits/sec [1912] 11.0-12.0 sec 25.1 MBytes 211 Mbits/sec [1912] 12.0-13.0 sec 25.3 MBytes 212 Mbits/sec [1912] 13.0-14.0 sec 23.1 MBytes 194 Mbits/sec [1912] 14.0-15.0 sec 23.6 MBytes 198 Mbits/sec [1912] 15.0-16.0 sec 23.6 MBytes 198 Mbits/sec [1912] 16.0-17.0 sec 25.4 MBytes 213 Mbits/sec [1912] 17.0-18.0 sec 25.6 MBytes 215 Mbits/sec [1912] 18.0-19.0 sec 25.2 MBytes 211 Mbits/sec [1912] 19.0-20.0 sec 25.0 MBytes 210 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 494 MBytes 207 Mbits/secOptional: Run on EON to compare network interface statistics
dladm show-link bge0 -s -i 1 LINK IPACKETS RBYTES IERRORS OPACKETS OBYTES OERRORS bge0 19447 27628795 0 2432 155854 0 bge0 19089 27123001 0 2382 152546 0 bge0 19638 27886729 0 2450 156898 0 bge0 18652 26493270 0 2326 148962 0 bge0 18454 26205800 0 2301 147362 0 bge0 18561 26358926 0 2315 148258 0 bge0 19268 27372463 0 2404 153954 0 bge0 19379 27533495 0 2418 154850 0 bge0 19687 27977035 0 2457 157346 0 bge0 18681 26527515 0 2330 149218 0 bge0 19155 27208916 0 2390 153058 0 bge0 19419 27593183 0 2423 155170 0 bge0 19542 27757561 0 2438 156130 0 bge0 17840 25347308 0 2226 142562 0 bge0 18148 25776206 0 2264 144994 0 bge0 18353 26072730 0 2290 146658 0 bge0 19657 27919866 0 2452 157106 0 bge0 19756 28065895 0 2465 157858 0 bge0 19390 27542355 0 2419 154914 0 bge0 17889 25407434 0 2236 143362 0 bge0 38 8169 0 1 162 0With TCP window set to 8K, Max throughput is approx 207Mbps/8bps=25.87MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 16k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.02 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3487 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 40.1 MBytes 336 Mbits/sec [1912] 1.0- 2.0 sec 31.1 MBytes 261 Mbits/sec [1912] 2.0- 3.0 sec 36.3 MBytes 305 Mbits/sec [1912] 3.0- 4.0 sec 43.2 MBytes 363 Mbits/sec [1912] 4.0- 5.0 sec 43.2 MBytes 363 Mbits/sec [1912] 5.0- 6.0 sec 41.5 MBytes 348 Mbits/sec [1912] 6.0- 7.0 sec 41.9 MBytes 352 Mbits/sec [1912] 7.0- 8.0 sec 29.6 MBytes 249 Mbits/sec [1912] 8.0- 9.0 sec 33.2 MBytes 278 Mbits/sec [1912] 9.0-10.0 sec 34.4 MBytes 288 Mbits/sec [1912] 10.0-11.0 sec 37.5 MBytes 315 Mbits/sec [1912] 11.0-12.0 sec 41.4 MBytes 348 Mbits/sec [1912] 12.0-13.0 sec 43.0 MBytes 360 Mbits/sec [1912] 13.0-14.0 sec 30.9 MBytes 259 Mbits/sec [1912] 14.0-15.0 sec 39.2 MBytes 329 Mbits/sec [1912] 15.0-16.0 sec 42.9 MBytes 360 Mbits/sec [1912] 16.0-17.0 sec 42.9 MBytes 360 Mbits/sec [1912] 17.0-18.0 sec 40.6 MBytes 340 Mbits/sec [1912] 18.0-19.0 sec 37.2 MBytes 312 Mbits/sec [1912] 19.0-20.0 sec 24.7 MBytes 207 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 755 MBytes 317 Mbits/secWith TCP window set to 16K, Max throughput is approx 317Mbps/8bps=39.63MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 24k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.02 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3497 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 53.4 MBytes 448 Mbits/sec [1912] 1.0- 2.0 sec 50.2 MBytes 421 Mbits/sec [1912] 2.0- 3.0 sec 40.8 MBytes 343 Mbits/sec [1912] 3.0- 4.0 sec 62.5 MBytes 524 Mbits/sec [1912] 4.0- 5.0 sec 63.9 MBytes 536 Mbits/sec [1912] 5.0- 6.0 sec 63.4 MBytes 532 Mbits/sec [1912] 6.0- 7.0 sec 64.4 MBytes 540 Mbits/sec [1912] 7.0- 8.0 sec 58.7 MBytes 493 Mbits/sec [1912] 8.0- 9.0 sec 48.0 MBytes 403 Mbits/sec [1912] 9.0-10.0 sec 62.2 MBytes 522 Mbits/sec [1912] 10.0-11.0 sec 54.1 MBytes 454 Mbits/sec [1912] 11.0-12.0 sec 53.9 MBytes 452 Mbits/sec [1912] 12.0-13.0 sec 56.8 MBytes 477 Mbits/sec [1912] 13.0-14.0 sec 57.6 MBytes 483 Mbits/sec [1912] 14.0-15.0 sec 49.1 MBytes 412 Mbits/sec [1912] 15.0-16.0 sec 65.0 MBytes 545 Mbits/sec [1912] 16.0-17.0 sec 64.2 MBytes 539 Mbits/sec [1912] 17.0-18.0 sec 64.6 MBytes 542 Mbits/sec [1912] 18.0-19.0 sec 64.0 MBytes 537 Mbits/sec [1912] 19.0-20.0 sec 57.5 MBytes 483 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1154 MBytes 484 Mbits/secWith TCP window set to 24K, Max throughput is approx 484Mbps/8bps=60.5MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 32k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.03 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3500 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 77.0 MBytes 646 Mbits/sec [1912] 1.0- 2.0 sec 78.2 MBytes 656 Mbits/sec [1912] 2.0- 3.0 sec 48.9 MBytes 410 Mbits/sec [1912] 3.0- 4.0 sec 76.6 MBytes 642 Mbits/sec [1912] 4.0- 5.0 sec 75.8 MBytes 636 Mbits/sec [1912] 5.0- 6.0 sec 79.4 MBytes 666 Mbits/sec [1912] 6.0- 7.0 sec 78.8 MBytes 661 Mbits/sec [1912] 7.0- 8.0 sec 79.0 MBytes 663 Mbits/sec [1912] 8.0- 9.0 sec 66.7 MBytes 559 Mbits/sec [1912] 9.0-10.0 sec 81.0 MBytes 679 Mbits/sec [1912] 10.0-11.0 sec 76.8 MBytes 644 Mbits/sec [1912] 11.0-12.0 sec 79.5 MBytes 667 Mbits/sec [1912] 12.0-13.0 sec 74.1 MBytes 621 Mbits/sec [1912] 13.0-14.0 sec 72.9 MBytes 612 Mbits/sec [1912] 14.0-15.0 sec 70.0 MBytes 587 Mbits/sec [1912] 15.0-16.0 sec 79.1 MBytes 663 Mbits/sec [1912] 16.0-17.0 sec 79.6 MBytes 668 Mbits/sec [1912] 17.0-18.0 sec 79.6 MBytes 668 Mbits/sec [1912] 18.0-19.0 sec 79.4 MBytes 666 Mbits/sec [1912] 19.0-20.0 sec 77.7 MBytes 652 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1510 MBytes 633 Mbits/secWith TCP window set to 32K, Max throughput is approx 633Mbps/8bps=79.13MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 48k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.05 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3506 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 101 MBytes 846 Mbits/sec [1912] 1.0- 2.0 sec 102 MBytes 852 Mbits/sec [1912] 2.0- 3.0 sec 84.9 MBytes 712 Mbits/sec [1912] 3.0- 4.0 sec 81.7 MBytes 686 Mbits/sec [1912] 4.0- 5.0 sec 94.7 MBytes 794 Mbits/sec [1912] 5.0- 6.0 sec 103 MBytes 860 Mbits/sec [1912] 6.0- 7.0 sec 102 MBytes 859 Mbits/sec [1912] 7.0- 8.0 sec 102 MBytes 852 Mbits/sec [1912] 8.0- 9.0 sec 101 MBytes 850 Mbits/sec [1912] 9.0-10.0 sec 90.6 MBytes 760 Mbits/sec [1912] 10.0-11.0 sec 82.6 MBytes 693 Mbits/sec [1912] 11.0-12.0 sec 101 MBytes 848 Mbits/sec [1912] 12.0-13.0 sec 85.5 MBytes 717 Mbits/sec [1912] 13.0-14.0 sec 85.8 MBytes 719 Mbits/sec [1912] 14.0-15.0 sec 91.5 MBytes 767 Mbits/sec [1912] 15.0-16.0 sec 91.8 MBytes 770 Mbits/sec [1912] 16.0-17.0 sec 89.7 MBytes 753 Mbits/sec [1912] 17.0-18.0 sec 102 MBytes 858 Mbits/sec [1912] 18.0-19.0 sec 97.9 MBytes 821 Mbits/sec [1912] 19.0-20.0 sec 100 MBytes 841 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1891 MBytes 793 Mbits/secWith TCP window set to 48K, Max throughput is approx 793Mbps/8bps=99.13MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 64k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.06 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3538 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 98.2 MBytes 823 Mbits/sec [1912] 1.0- 2.0 sec 66.3 MBytes 556 Mbits/sec [1912] 2.0- 3.0 sec 101 MBytes 844 Mbits/sec [1912] 3.0- 4.0 sec 99.8 MBytes 837 Mbits/sec [1912] 4.0- 5.0 sec 99.5 MBytes 835 Mbits/sec [1912] 5.0- 6.0 sec 79.7 MBytes 668 Mbits/sec [1912] 6.0- 7.0 sec 76.6 MBytes 643 Mbits/sec [1912] 7.0- 8.0 sec 94.1 MBytes 790 Mbits/sec [1912] 8.0- 9.0 sec 101 MBytes 846 Mbits/sec [1912] 9.0-10.0 sec 100 MBytes 840 Mbits/sec [1912] 10.0-11.0 sec 96.7 MBytes 811 Mbits/sec [1912] 11.0-12.0 sec 96.0 MBytes 805 Mbits/sec [1912] 12.0-13.0 sec 94.0 MBytes 789 Mbits/sec [1912] 13.0-14.0 sec 97.2 MBytes 816 Mbits/sec [1912] 14.0-15.0 sec 98.5 MBytes 827 Mbits/sec [1912] 15.0-16.0 sec 81.7 MBytes 686 Mbits/sec [1912] 16.0-17.0 sec 74.6 MBytes 626 Mbits/sec [1912] 17.0-18.0 sec 79.8 MBytes 669 Mbits/sec [1912] 18.0-19.0 sec 98.1 MBytes 823 Mbits/sec [1912] 19.0-20.0 sec 99.7 MBytes 837 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1832 MBytes 768 Mbits/secWith TCP window set to 64K, Max throughput is approx 768Mbps/8bps=96MB/s.
The important things to note from these tests between a client (Win XP) and EON ZFS NAS server on this network are:
- Possible upper limit near 99.13MBytes/sec (not the theoretical 1000Mbps/125MBytes/sec).
- The optimal TCP window size for this network is probably 48K (should have tested 56K also).
- A larger TCP window size of 64K did not provide the highest transfer rate.
10 comments:
What hardware are you running EON on?
Hi Joe,
This test was done with EON running on a Dell Optiplex GX620 with 2GB RAM.
Its very useful article!
EON is very powerful and reliability OS for storage but it need good knowledge to tune up and release its full power.
I hope you can post more such cool article to help us troubleshoot performance issue.
Thank!
I am curious what a 10GbE link would do. 999MB/s?
Nice article, I will def be giving EON a try.
My tests shown below peaked at 32k tcp window size. (i'm connected through a 2510G-24 procurve though).
me@fankleboxen:~$ iperf -c 192.168.13.73
-P 1 -i 1 -w 32k -f m -t 20
------------------------------------------------------------
Client connecting to 192.168.13.73, TCP port 5001
TCP window size: 0.06 MByte (WARNING: requested 0.03 MByte)
------------------------------------------------------------
[ 3] local 192.168.13.76 port 52318 connected with 192.168.13.73 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 111 MBytes 927 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 940 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 940 Mbits/sec
[ 3] 3.0- 4.0 sec 112 MBytes 940 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 940 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 940 Mbits/sec
[ 3] 6.0- 7.0 sec 112 MBytes 940 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 940 Mbits/sec
[ 3] 8.0- 9.0 sec 112 MBytes 940 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 940 Mbits/sec
[ 3] 10.0-11.0 sec 112 MBytes 940 Mbits/sec
[ 3] 11.0-12.0 sec 112 MBytes 940 Mbits/sec
[ 3] 12.0-13.0 sec 112 MBytes 940 Mbits/sec
[ 3] 13.0-14.0 sec 112 MBytes 940 Mbits/sec
[ 3] 14.0-15.0 sec 112 MBytes 940 Mbits/sec
[ 3] 15.0-16.0 sec 112 MBytes 940 Mbits/sec
[ 3] 16.0-17.0 sec 112 MBytes 937 Mbits/sec
[ 3] 17.0-18.0 sec 112 MBytes 940 Mbits/sec
[ 3] 18.0-19.0 sec 112 MBytes 940 Mbits/sec
[ 3] 19.0-20.0 sec 112 MBytes 939 Mbits/sec
[ 3] 0.0-20.0 sec 2240 MBytes 939 Mbits/sec
iperf performace fluctuates on the higher tcp window size (56k-64k) for my tests. There'd be times when iperf only shows 785Mbit/s and then the next test will jump to 845Mbit/s. I don't really understand much of network tuning, so I don't really know how it'll affect my connections. I'm happy with getting at least 84% of my gigE connection.
Btw Andre, really doin' a great job here. I've played around with other distros for opensolaris but I've always come back to your EON. Nothing compares when it comes to it's reliability and stableness.
Also, thanks for adding mediatomb. I've installed it and was able to get on the web UI in 10mins! Now I just gotta start loading media files in my storage and fire up my PS3. Good job! and Thank you!
I finally hit a snag with the server. I think this is one of the problems I had before with my NIC card. My post above describes what I get with the rge driver. I was having problems with transferring large files so I changed the driver to gani. I had problems figuring that out before, but now I finally managed to get it working properly. I did the benchmark again, and I am not getting the same results as I had with the rge driver. It doesn't really matter. Because as of now, my transfer is still running, compared to when it was still the rge driver, I would lose my connection within a minute or two. Transfer is half-way thru, and still no hiccup with the connection. Getting a consistent 42-50MB/s on CIFS. I'm a happy camper now!
Hi Glenn,
That's good to hear. The rge driver has always been hit/miss. It's either the opensolaris or gani driver. Good to hear you got it worked out. It would be great and benefit others if you could share a gani driver howto.
Hi Andre,
I finally got me some Intel PCI GIGe NIC. I've reinstalled everything starting from the beginning. I'm not running any production right now, so it doesn't really matter right now. I haven't tested transfers yet with the Intel. I did have it aggregated as I bought 2 NICs.
As for the gani driver, I had to install a Virtualbox opensolaris. Compilation for the gani driver needs sunstudio12u1 or gcc. So I had to get a full opensolaris box for that. I followed the README on the zip archive. After I got everything set up, I just copied the binary at /kernel/drv/amd64/gani to my eon box. I also copied the add_drive.sh from the zip archive to the eon box and ran it. Not all rge entries are updated so I also edited all the rge entries in /etc/name_to_major and /etc/driver_aliases to point to gani. Add all the proper .backup entries, updimg.sh, then reboot. Everything works after that. Also, don't forget to remove the /etc/hostname.rge and /etc/dhcp.rge entries before running updimg.sh.
As of right now, I'm trying to get napp-it to work on this new build. I'm having trouble with the webserver not executing the perl files. Even test-cgi on the cgi-bin directory won't execute. I get (13)Permission Denied on the error log trying to execute test-cgi. Any ideas?
Hi,
want to share my results:
Configuration:
CPU AMD PH II X4 955 3.2G AM3 RT
MEM 2Gx2 GSKILL F3-12800CL9D-4GBRL
MB ASUS M4A88T-M AM3 R (NB: AMD 880G, SB: SB710, NIC: RTL8111E (1GB/s), GPU: ATI Radeon HD 4250 (8 ch), A: VIA VT1708S (VGA/DV-I/HDMI), Micro ATX
6x 3GB SATA, 1x PATA, 6x USB2,
eon-0.600-130-64-smb
iperf@56k TCP window: ~940Mbits/s (avg. 114MB/s), 2.25G in 20s (!)
Copy/write from windows (8GB file), Raid-Z pool of 7200rpm Samsungs (3 yr old 500GB models): avg. 97MB/s!
I can only recommend this...
:) Thomas
Post a Comment