Ed RAID Tests

From GridPP Wiki
Jump to: navigation, search

Perform some benchmarking using bonnie++. Man page for bonnie++ can be found here

dcache.epcc.ed.ac.uk

The specs of the machine are:

  • Proc: 8 x Intel(R) XEON(TM) MP CPU 1.90GHz
  • Cache: 512KB
  • Mem: 16GB
  • Disk specs: ???? RAID 5 configuration

bonnie++

 -s file size
 -n number of files to use in the file creation test
 -m machine name

 # ./bonnie++-1.03a/bonnie++ -d /export/raid01/ -s 32180 -m  dcache -u root
 Using uid:0, gid:0.
 Writing with putc()...done
 Writing intelligently...done
 Rewriting...done
 Reading with getc()...done
 Reading intelligently...done
 start 'em...done...done...done...
 Create files in sequential order...done.
 Stat files in sequential order...done.
 Delete files in sequential order...done.
 Create files in random order...done.
 Stat files in random order...done.
 Delete files in random order...done.
 Version  1.03       ------Sequential Output------ --Sequential Input- --Random-                 
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
 Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
 dcache       32180M 14431  99 39763  85 23434  36 17851  87 91911  46 804.9   9
                     ------Sequential Create------ --------Random Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  16  1877  98 +++++ +++ +++++ +++  1994  99 +++++ +++  5632  99
 dcache,32180M,14431,99,39763,85,23434,36,17851,87,91911,46,804.9,9,16,1877,98,++ +++,+++,+++++,+++,1994,99,+++++,+++,5632,99


Notice that in this test, I did not use multiple threads. 

# ./bonnie++-1.03a/bonnie++ -d /san-storage/scotgrid1/pool/data/ -s 32180 -m dcache -u root
Using uid:0, gid:0.
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dcache       32180M  2276  20  3370  13  7950  37 19897  99 64732  47 353.6   4
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   193   4  2172  15    85   2    62   1  2190  17   154   2
dcache,32180M,2276,20,3370,13,7950,37,19897,99,64732,47,353.6,4,16,193,4,2172,15,85,2,62,1,2190,17,154,2

dd tests

Write cache is enabled in the RAID controller, but only if there is a battery backup available. The battery in the controller that service raid22 needs replacing, so the effect of having write caching on and off can be seen by the results of the following two tests (since the battery is OK in the other controller, serving raid10): 

[root@dcache raid22]# time dd if=/dev/zero of=big-testfile bs=1024k count=5120 
5120+0 records in
5120+0 records out

real    3m52.104s
user    0m0.020s
sys     1m8.950s

This is about 22MB/s. 

[root@dcache raid10]# time dd if=/dev/zero of=big-testfile bs=1024k count=5120
5120+0 records in
5120+0 records out

real    2m4.012s
user    0m0.010s
sys     1m6.300s

This is about 41MB/s. So there is a performance boost with write caching on, but not as great as we are seeing on a test machine, where it is only taking ~35seconds to write a 5GB file to a RAID 5.

Writing to the SAN over NFS, the same tests gives the result: 

[root@dcache scotgrid1]# time dd if=/dev/zero of=big-testfile bs=1024k count=5120
5120+0 records in
5120+0 records out
 
real    4m57.581s
user    0m0.000s
sys     1m23.900s

Although we do not know the setup of the RAID 5 array in the SAN, which may/may not be using write caching.

dpm.epcc.ed.ac.uk

  • Proc: 2*Intel Pentium III (Coppermine) CPU 1GHz
  • Cache: 256KB
  • Mem: 2GB
  • We are currently using NFS to mount 1 volume from the RAID and 2 volumes from the SAN.

Using tiobench to run some benchmarking tests. I had to enable large filesizes in the Makefile. 

# ./tiotest -t 1 -f 8192 -r 4000 -b 4096 -d /storage2
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        8192 MBs |  958.8 s |   8.544 MB/s |   0.2 %  |   8.1 % |
| Random Write   16 MBs |    4.0 s |   3.877 MB/s |   0.2 %  |   4.7 % |
| Read         8192 MBs |  277.6 s |  29.512 MB/s |   0.8 %  |  15.3 % |
| Random Read    16 MBs |    2.1 s |   7.560 MB/s |   1.0 %  |   7.3 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        0.456 ms |      525.375 ms |  0.00000 |   0.00000 |
| Random Write |        0.984 ms |      631.220 ms |  0.00000 |   0.00000 |
| Read         |        0.131 ms |       54.939 ms |  0.00000 |   0.00000 |
| Random Read  |        0.512 ms |        9.519 ms |  0.00000 |   0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        0.294 ms |      631.220 ms |  0.00000 |   0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

It is clear from the above results that writing is >3 times slower than reading into the NFS mounted RAID volume.

dd test

Writing to the RAID 5 volume over NFS using UDP gives: 

[root@dpm dpmmgr]# time dd if=/dev/zero of=big-testfile bs=1024k count=5120
5120+0 records in
5120+0 records out

real    20m20.782s
user    0m0.060s
sys     1m7.740s

Corresponding to a rate of about 4.5MB/s. Presumably using TCP will result in even slower performance.

iperf

Running iperf between dcache and dpm shows that the 

[root@dcache iperf-dir]# ./iperf -c dpm.epcc.ed.ac.uk -p 52000
------------------------------------------------------------
Client connecting to dpm.epcc.ed.ac.uk, TCP port 52000
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  5] local 129.215.175.24 port 43712 connected with 129.215.175.6 port 52000
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   987 MBytes   828 Mbits/sec

Corresponding to about 100MB/s over TCP.

Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Ed_RAID_Tests&oldid=1221"