Iperf (Mac, Win)
IOMeter [Part II]
In most cases your storage is accessed/utilized across a network using a client PC. So testing will involve a source, destination and the network that connects them. The source will be your client PC and the destination will be your EON ZFS storage. This will help identify what the network is capable of, is it working properly or if it is a bottleneck to your storage.
On the EON ZFS storage (destination), connect 2 terminals using ssh or putty sessions from your client (Win XP in this case). In one terminal, run
iperf -sIn the other terminal, run (substitute your interface name for bge0)
dladm show-link bge0 -s -i 1On the source or client PC we will run repetitions of below, changing TCP window size
iperf -c IP_of_EON -P 1 -i 1 -w 8k -f m -t 20The above says, run 1 process thread P=1 for a test duration of t=20secs with interval i=1sec updates and a TCP window size w=8 in kilobytes. The results will be formatted in (m) megabytes.
This will produce an output showing the upper bounds of your network between the source and destination. It may also help identify the optimal TCP window size for your network. The output below will show the different transfer rates with each change of the TCP window size from 8k to 64k, on the client on a gigabit network. So lets begin.
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 8k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.01 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3448 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 25.2 MBytes 211 Mbits/sec [1912] 1.0- 2.0 sec 24.7 MBytes 207 Mbits/sec [1912] 2.0- 3.0 sec 25.4 MBytes 213 Mbits/sec [1912] 3.0- 4.0 sec 24.1 MBytes 202 Mbits/sec [1912] 4.0- 5.0 sec 23.8 MBytes 200 Mbits/sec [1912] 5.0- 6.0 sec 24.0 MBytes 201 Mbits/sec [1912] 6.0- 7.0 sec 25.0 MBytes 210 Mbits/sec [1912] 7.0- 8.0 sec 25.0 MBytes 210 Mbits/sec [1912] 8.0- 9.0 sec 25.5 MBytes 214 Mbits/sec [1912] 9.0-10.0 sec 24.1 MBytes 202 Mbits/sec [1912] 10.0-11.0 sec 24.8 MBytes 208 Mbits/sec [1912] 11.0-12.0 sec 25.1 MBytes 211 Mbits/sec [1912] 12.0-13.0 sec 25.3 MBytes 212 Mbits/sec [1912] 13.0-14.0 sec 23.1 MBytes 194 Mbits/sec [1912] 14.0-15.0 sec 23.6 MBytes 198 Mbits/sec [1912] 15.0-16.0 sec 23.6 MBytes 198 Mbits/sec [1912] 16.0-17.0 sec 25.4 MBytes 213 Mbits/sec [1912] 17.0-18.0 sec 25.6 MBytes 215 Mbits/sec [1912] 18.0-19.0 sec 25.2 MBytes 211 Mbits/sec [1912] 19.0-20.0 sec 25.0 MBytes 210 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 494 MBytes 207 Mbits/secOptional: Run on EON to compare network interface statistics
dladm show-link bge0 -s -i 1 LINK IPACKETS RBYTES IERRORS OPACKETS OBYTES OERRORS bge0 19447 27628795 0 2432 155854 0 bge0 19089 27123001 0 2382 152546 0 bge0 19638 27886729 0 2450 156898 0 bge0 18652 26493270 0 2326 148962 0 bge0 18454 26205800 0 2301 147362 0 bge0 18561 26358926 0 2315 148258 0 bge0 19268 27372463 0 2404 153954 0 bge0 19379 27533495 0 2418 154850 0 bge0 19687 27977035 0 2457 157346 0 bge0 18681 26527515 0 2330 149218 0 bge0 19155 27208916 0 2390 153058 0 bge0 19419 27593183 0 2423 155170 0 bge0 19542 27757561 0 2438 156130 0 bge0 17840 25347308 0 2226 142562 0 bge0 18148 25776206 0 2264 144994 0 bge0 18353 26072730 0 2290 146658 0 bge0 19657 27919866 0 2452 157106 0 bge0 19756 28065895 0 2465 157858 0 bge0 19390 27542355 0 2419 154914 0 bge0 17889 25407434 0 2236 143362 0 bge0 38 8169 0 1 162 0With TCP window set to 8K, Max throughput is approx 207Mbps/8bps=25.87MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 16k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.02 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3487 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 40.1 MBytes 336 Mbits/sec [1912] 1.0- 2.0 sec 31.1 MBytes 261 Mbits/sec [1912] 2.0- 3.0 sec 36.3 MBytes 305 Mbits/sec [1912] 3.0- 4.0 sec 43.2 MBytes 363 Mbits/sec [1912] 4.0- 5.0 sec 43.2 MBytes 363 Mbits/sec [1912] 5.0- 6.0 sec 41.5 MBytes 348 Mbits/sec [1912] 6.0- 7.0 sec 41.9 MBytes 352 Mbits/sec [1912] 7.0- 8.0 sec 29.6 MBytes 249 Mbits/sec [1912] 8.0- 9.0 sec 33.2 MBytes 278 Mbits/sec [1912] 9.0-10.0 sec 34.4 MBytes 288 Mbits/sec [1912] 10.0-11.0 sec 37.5 MBytes 315 Mbits/sec [1912] 11.0-12.0 sec 41.4 MBytes 348 Mbits/sec [1912] 12.0-13.0 sec 43.0 MBytes 360 Mbits/sec [1912] 13.0-14.0 sec 30.9 MBytes 259 Mbits/sec [1912] 14.0-15.0 sec 39.2 MBytes 329 Mbits/sec [1912] 15.0-16.0 sec 42.9 MBytes 360 Mbits/sec [1912] 16.0-17.0 sec 42.9 MBytes 360 Mbits/sec [1912] 17.0-18.0 sec 40.6 MBytes 340 Mbits/sec [1912] 18.0-19.0 sec 37.2 MBytes 312 Mbits/sec [1912] 19.0-20.0 sec 24.7 MBytes 207 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 755 MBytes 317 Mbits/secWith TCP window set to 16K, Max throughput is approx 317Mbps/8bps=39.63MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 24k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.02 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3497 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 53.4 MBytes 448 Mbits/sec [1912] 1.0- 2.0 sec 50.2 MBytes 421 Mbits/sec [1912] 2.0- 3.0 sec 40.8 MBytes 343 Mbits/sec [1912] 3.0- 4.0 sec 62.5 MBytes 524 Mbits/sec [1912] 4.0- 5.0 sec 63.9 MBytes 536 Mbits/sec [1912] 5.0- 6.0 sec 63.4 MBytes 532 Mbits/sec [1912] 6.0- 7.0 sec 64.4 MBytes 540 Mbits/sec [1912] 7.0- 8.0 sec 58.7 MBytes 493 Mbits/sec [1912] 8.0- 9.0 sec 48.0 MBytes 403 Mbits/sec [1912] 9.0-10.0 sec 62.2 MBytes 522 Mbits/sec [1912] 10.0-11.0 sec 54.1 MBytes 454 Mbits/sec [1912] 11.0-12.0 sec 53.9 MBytes 452 Mbits/sec [1912] 12.0-13.0 sec 56.8 MBytes 477 Mbits/sec [1912] 13.0-14.0 sec 57.6 MBytes 483 Mbits/sec [1912] 14.0-15.0 sec 49.1 MBytes 412 Mbits/sec [1912] 15.0-16.0 sec 65.0 MBytes 545 Mbits/sec [1912] 16.0-17.0 sec 64.2 MBytes 539 Mbits/sec [1912] 17.0-18.0 sec 64.6 MBytes 542 Mbits/sec [1912] 18.0-19.0 sec 64.0 MBytes 537 Mbits/sec [1912] 19.0-20.0 sec 57.5 MBytes 483 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1154 MBytes 484 Mbits/secWith TCP window set to 24K, Max throughput is approx 484Mbps/8bps=60.5MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 32k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.03 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3500 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 77.0 MBytes 646 Mbits/sec [1912] 1.0- 2.0 sec 78.2 MBytes 656 Mbits/sec [1912] 2.0- 3.0 sec 48.9 MBytes 410 Mbits/sec [1912] 3.0- 4.0 sec 76.6 MBytes 642 Mbits/sec [1912] 4.0- 5.0 sec 75.8 MBytes 636 Mbits/sec [1912] 5.0- 6.0 sec 79.4 MBytes 666 Mbits/sec [1912] 6.0- 7.0 sec 78.8 MBytes 661 Mbits/sec [1912] 7.0- 8.0 sec 79.0 MBytes 663 Mbits/sec [1912] 8.0- 9.0 sec 66.7 MBytes 559 Mbits/sec [1912] 9.0-10.0 sec 81.0 MBytes 679 Mbits/sec [1912] 10.0-11.0 sec 76.8 MBytes 644 Mbits/sec [1912] 11.0-12.0 sec 79.5 MBytes 667 Mbits/sec [1912] 12.0-13.0 sec 74.1 MBytes 621 Mbits/sec [1912] 13.0-14.0 sec 72.9 MBytes 612 Mbits/sec [1912] 14.0-15.0 sec 70.0 MBytes 587 Mbits/sec [1912] 15.0-16.0 sec 79.1 MBytes 663 Mbits/sec [1912] 16.0-17.0 sec 79.6 MBytes 668 Mbits/sec [1912] 17.0-18.0 sec 79.6 MBytes 668 Mbits/sec [1912] 18.0-19.0 sec 79.4 MBytes 666 Mbits/sec [1912] 19.0-20.0 sec 77.7 MBytes 652 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1510 MBytes 633 Mbits/secWith TCP window set to 32K, Max throughput is approx 633Mbps/8bps=79.13MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 48k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.05 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3506 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 101 MBytes 846 Mbits/sec [1912] 1.0- 2.0 sec 102 MBytes 852 Mbits/sec [1912] 2.0- 3.0 sec 84.9 MBytes 712 Mbits/sec [1912] 3.0- 4.0 sec 81.7 MBytes 686 Mbits/sec [1912] 4.0- 5.0 sec 94.7 MBytes 794 Mbits/sec [1912] 5.0- 6.0 sec 103 MBytes 860 Mbits/sec [1912] 6.0- 7.0 sec 102 MBytes 859 Mbits/sec [1912] 7.0- 8.0 sec 102 MBytes 852 Mbits/sec [1912] 8.0- 9.0 sec 101 MBytes 850 Mbits/sec [1912] 9.0-10.0 sec 90.6 MBytes 760 Mbits/sec [1912] 10.0-11.0 sec 82.6 MBytes 693 Mbits/sec [1912] 11.0-12.0 sec 101 MBytes 848 Mbits/sec [1912] 12.0-13.0 sec 85.5 MBytes 717 Mbits/sec [1912] 13.0-14.0 sec 85.8 MBytes 719 Mbits/sec [1912] 14.0-15.0 sec 91.5 MBytes 767 Mbits/sec [1912] 15.0-16.0 sec 91.8 MBytes 770 Mbits/sec [1912] 16.0-17.0 sec 89.7 MBytes 753 Mbits/sec [1912] 17.0-18.0 sec 102 MBytes 858 Mbits/sec [1912] 18.0-19.0 sec 97.9 MBytes 821 Mbits/sec [1912] 19.0-20.0 sec 100 MBytes 841 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1891 MBytes 793 Mbits/secWith TCP window set to 48K, Max throughput is approx 793Mbps/8bps=99.13MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 64k -f m -t 20 ------------------------------------------------------------ Client connecting to 10.72.100.148, TCP port 5001 TCP window size: 0.06 MByte ------------------------------------------------------------ [1912] local 10.72.100.128 port 3538 connected with 10.72.100.148 port 5001 [ ID] Interval Transfer Bandwidth [1912] 0.0- 1.0 sec 98.2 MBytes 823 Mbits/sec [1912] 1.0- 2.0 sec 66.3 MBytes 556 Mbits/sec [1912] 2.0- 3.0 sec 101 MBytes 844 Mbits/sec [1912] 3.0- 4.0 sec 99.8 MBytes 837 Mbits/sec [1912] 4.0- 5.0 sec 99.5 MBytes 835 Mbits/sec [1912] 5.0- 6.0 sec 79.7 MBytes 668 Mbits/sec [1912] 6.0- 7.0 sec 76.6 MBytes 643 Mbits/sec [1912] 7.0- 8.0 sec 94.1 MBytes 790 Mbits/sec [1912] 8.0- 9.0 sec 101 MBytes 846 Mbits/sec [1912] 9.0-10.0 sec 100 MBytes 840 Mbits/sec [1912] 10.0-11.0 sec 96.7 MBytes 811 Mbits/sec [1912] 11.0-12.0 sec 96.0 MBytes 805 Mbits/sec [1912] 12.0-13.0 sec 94.0 MBytes 789 Mbits/sec [1912] 13.0-14.0 sec 97.2 MBytes 816 Mbits/sec [1912] 14.0-15.0 sec 98.5 MBytes 827 Mbits/sec [1912] 15.0-16.0 sec 81.7 MBytes 686 Mbits/sec [1912] 16.0-17.0 sec 74.6 MBytes 626 Mbits/sec [1912] 17.0-18.0 sec 79.8 MBytes 669 Mbits/sec [1912] 18.0-19.0 sec 98.1 MBytes 823 Mbits/sec [1912] 19.0-20.0 sec 99.7 MBytes 837 Mbits/sec [ ID] Interval Transfer Bandwidth [1912] 0.0-20.0 sec 1832 MBytes 768 Mbits/secWith TCP window set to 64K, Max throughput is approx 768Mbps/8bps=96MB/s.
The important things to note from these tests between a client (Win XP) and EON ZFS NAS server on this network are:
- Possible upper limit near 99.13MBytes/sec (not the theoretical 1000Mbps/125MBytes/sec).
- The optimal TCP window size for this network is probably 48K (should have tested 56K also).
- A larger TCP window size of 64K did not provide the highest transfer rate.
