Difference between revisions of "ZFS-Tests"
From GridPP Wiki
Line 12: | Line 12: | ||
|- | |- | ||
! Property | ! Property | ||
− | | 1 | + | | 1|| 2 || 3|| |
− | | 2 | + | |
− | | 3 | + | |
|- | |- | ||
− | ! Filesystem | + | ! Filesystem |
− | | ZFS | + | | ZFS|| XFS|| Ext4|| |
− | | XFS | + | |
− | | Ext4 | + | |
|- | |- | ||
! Special filesystem config | ! Special filesystem config | ||
− | | xattr=sa, relatime=on | + | | xattr=sa, relatime=on|| || || |
− | | | + | |
− | | | + | |
|- | |- | ||
! Kind of raid | ! Kind of raid | ||
− | | 3xraidz2 | + | | 3xraidz2|| raid60 || raid60 || |
− | | raid60 | + | |
− | | raid60 | + | |
|- | |- | ||
! Number of Disks | ! Number of Disks | ||
− | | 33, 11 per raidz2 | + | | 33, 11 per raidz2|| 33, 11 per raid6|| 33, 11 per raid6|| |
− | | 33, 11 per raid6 | + | |
− | | 33, 11 per raid6 | + | |
|- | |- | ||
! Kind of Disks | ! Kind of Disks | ||
− | | 2TB SAS, 7200k, 6Gbps | + | | 2TB SAS, 7200k, 6Gbps|| 2TB SAS, 7200k, 6Gbps|| 2TB SAS, 7200k, 6Gbps|| |
− | | 2TB SAS, 7200k, 6Gbps | + | |
− | | 2TB SAS, 7200k, 6Gbps | + | |
|- | |- | ||
! Controller | ! Controller | ||
− | | PERC H800 (33xraid0) | + | | PERC H800 (33xraid0) || PERC H800|| PERC H800|| |
− | | PERC H800 | + | |
− | | PERC H800 | + | |
|- | |- | ||
! Controller Cache used | ! Controller Cache used | ||
− | | yes (512 MB) | + | | yes (512 MB)|| yes (512 MB)|| yes (512MB)|| |
− | | yes (512 MB) | + | |
− | | yes (512MB) | + | |
|- | |- | ||
! RAM | ! RAM | ||
− | | 12GB | + | | 12GB|| 12GB|| 12GB|| |
− | | 12GB | + | |
− | | 12GB | + | |
|- | |- | ||
! CPU | ! CPU | ||
− | | 2xE5620, 2.4GHz, HT on | + | | 2xE5620, 2.4GHz, HT on|| 2xE5620, 2.4GHz, HT on|| 2xE6520, 2.4GHz, HT on|| |
− | | 2xE5620, 2.4GHz, HT on | + | |
− | | 2xE6520, 2.4GHz, HT on | + | |
|- | |- | ||
! OS | ! OS | ||
− | | SL6 | + | | SL6|| SL6|| SL6|| |
− | | SL6 | + | |
− | | SL6 | + | |
|- | |- | ||
! normal averaged sequential read | ! normal averaged sequential read | ||
− | | 536MB/s | + | | 536MB/s|| 856MB/s|| 607MB/s|| |
− | | 856MB/s | + | |
− | | 607MB/s | + | |
|- | |- | ||
! normal averaged parallel read | ! normal averaged parallel read | ||
− | | 746MB/s | + | | 746MB/s || 538MB/s|| 533MB/s || |
− | | 538MB/s | + | |
− | | 533MB/s | + | |
|- | |- | ||
! normal averaged sequential large writes | ! normal averaged sequential large writes | ||
− | | 709MB/s | + | | 709MB/s|| 667MB/s|| 348MB/s|| |
− | | 667MB/s | + | |
− | | 348MB/s | + | |
|- | |- | ||
! normal averaged parallel large writes | ! normal averaged parallel large writes | ||
− | | 680MB/s | + | | 680MB/s || 314MB/s|| 140MB/s|| |
− | | 314MB/s | + | |
− | | 140MB/s | + | |
|- | |- | ||
! normal averaged sequential small writes | ! normal averaged sequential small writes | ||
− | | 10MB/s | + | | 10MB/s || 2MB/s|| 2MB/s|| |
− | | 2MB/s | + | |
− | | 2MB/s | + | |
|- | |- | ||
! normal averaged parallel small writes | ! normal averaged parallel small writes | ||
− | | 74MB/s | + | | 74MB/s|| 15MB/s|| 20MB/s|| |
− | | 15MB/s | + | |
− | | 20MB/s | + | |
|- | |- | ||
! degrated averaged sequential read | ! degrated averaged sequential read | ||
− | | 532MB/s | + | | 532MB/s|| 274MB/s|| 266MB/s|| |
− | | 274MB/s | + | |
− | | 266MB/s | + | |
|- | |- | ||
! degrated averaged parallel read | ! degrated averaged parallel read | ||
− | | 707MB/s | + | | 707MB/s || 280MB/s|| 298MB/s || |
− | | 280MB/s | + | |
− | | 298MB/s | + | |
|- | |- | ||
! degrated averaged sequential large writes | ! degrated averaged sequential large writes | ||
− | | 706MB/s | + | | 706MB/s|| 657MB/s|| 298MB/s|| |
− | | 657MB/s | + | |
− | | 298MB/s | + | |
|- | |- | ||
! degrated averaged parallel large writes | ! degrated averaged parallel large writes | ||
− | | 695MB/s | + | | 695MB/s || 319MB/s|| 141MB/s|| |
− | | 319MB/s | + | |
− | | 141MB/s | + | |
|- | |- | ||
! degrated averaged sequential small writes | ! degrated averaged sequential small writes | ||
− | | 10MB/s | + | | 10MB/s || 2MB/s|| 2MB/s|| |
− | | 2MB/s | + | |
− | | 2MB/s | + | |
|- | |- | ||
! degrated averaged parallel small writes | ! degrated averaged parallel small writes | ||
− | | 74MB/s | + | | 74MB/s|| 17MB/s|| 15MB/s|| |
− | | 17MB/s | + | |
− | | 15MB/s | + | |
|- | |- | ||
! rebuild, averaged sequential read | ! rebuild, averaged sequential read | ||
− | | 401MB/s | + | | 401MB/s|| 247MB/s|| 246MB/s|| |
− | | 247MB/s | + | |
− | | 246MB/s | + | |
|- | |- | ||
! rebuild, averaged parallel read | ! rebuild, averaged parallel read | ||
− | | 533MB/s | + | | 533MB/s || 267MB/s|| 274MB/s || |
− | | 267MB/s | + | |
− | | 274MB/s | + | |
|- | |- | ||
! rebuild, averaged sequential large writes | ! rebuild, averaged sequential large writes | ||
− | | - | + | | -|| 367MB/s|| 239MB/s|| |
− | | 367MB/s | + | |
− | | 239MB/s | + | |
|- | |- | ||
! rebuild, averaged parallel large writes | ! rebuild, averaged parallel large writes | ||
− | | - | + | | - || 279MB/s|| 129MB/s|| |
− | | 279MB/s | + | |
− | | 129MB/s | + | |
|- | |- | ||
! rebuild, averaged sequential small writes | ! rebuild, averaged sequential small writes | ||
− | | 6MB/s | + | | 6MB/s || 3MB/s|| 2MB/s|| |
− | | 3MB/s | + | |
− | | 2MB/s | + | |
|- | |- | ||
! rebuild, averaged parallel small writes | ! rebuild, averaged parallel small writes | ||
− | | 31MB/s | + | | 31MB/s|| 17MB/s|| 19MB/s|| |
− | | 17MB/s | + | |
− | | 19MB/s | + | |
|- | |- | ||
|} | |} | ||
Line 160: | Line 102: | ||
<u>Comments:</u><br> | <u>Comments:</u><br> | ||
− | Files for read test where on compressed area of ZFS (root files, tgz files, LSST data files) | + | * for 1) Files for read test where on compressed area of ZFS (root files, tgz files, LSST data files) |
− | ZFS write test during rebuild not comparable since the rebuild was always faster finished then the test (in 2h compared to 21h for hardware raid) | + | * for 1) ZFS write test during rebuild not comparable since the rebuild was always faster finished then the test (in 2h compared to 21h for hardware raid) |
+ | * for 1-4) all read tests based on the same mix of files |
Revision as of 11:45, 30 January 2017
This page is to track different configurations that were tested under different setups, raid systems, and file systems. If one needs to add comments to the own test setup, please use the running number and add a comment to it under the table.
for tool testscript.sh:
- averaged sequential/parallel read: sum of all bytes over all files divided by total time taken
- averaged sequential/parallel write: bytes per file divided by (total time taken divided by total number of files written)
- parallel read/write default: 10 parallel reads and writes for large files, 20 parallel writes for small files
- other default: 5.1 TB used of mixed files to read, 100x54GB and 200,000x131kB files written
Property | 1 | 2 | 3 | |
---|---|---|---|---|
Filesystem | ZFS | XFS | Ext4 | |
Special filesystem config | xattr=sa, relatime=on | |||
Kind of raid | 3xraidz2 | raid60 | raid60 | |
Number of Disks | 33, 11 per raidz2 | 33, 11 per raid6 | 33, 11 per raid6 | |
Kind of Disks | 2TB SAS, 7200k, 6Gbps | 2TB SAS, 7200k, 6Gbps | 2TB SAS, 7200k, 6Gbps | |
Controller | PERC H800 (33xraid0) | PERC H800 | PERC H800 | |
Controller Cache used | yes (512 MB) | yes (512 MB) | yes (512MB) | |
RAM | 12GB | 12GB | 12GB | |
CPU | 2xE5620, 2.4GHz, HT on | 2xE5620, 2.4GHz, HT on | 2xE6520, 2.4GHz, HT on | |
OS | SL6 | SL6 | SL6 | |
normal averaged sequential read | 536MB/s | 856MB/s | 607MB/s | |
normal averaged parallel read | 746MB/s | 538MB/s | 533MB/s | |
normal averaged sequential large writes | 709MB/s | 667MB/s | 348MB/s | |
normal averaged parallel large writes | 680MB/s | 314MB/s | 140MB/s | |
normal averaged sequential small writes | 10MB/s | 2MB/s | 2MB/s | |
normal averaged parallel small writes | 74MB/s | 15MB/s | 20MB/s | |
degrated averaged sequential read | 532MB/s | 274MB/s | 266MB/s | |
degrated averaged parallel read | 707MB/s | 280MB/s | 298MB/s | |
degrated averaged sequential large writes | 706MB/s | 657MB/s | 298MB/s | |
degrated averaged parallel large writes | 695MB/s | 319MB/s | 141MB/s | |
degrated averaged sequential small writes | 10MB/s | 2MB/s | 2MB/s | |
degrated averaged parallel small writes | 74MB/s | 17MB/s | 15MB/s | |
rebuild, averaged sequential read | 401MB/s | 247MB/s | 246MB/s | |
rebuild, averaged parallel read | 533MB/s | 267MB/s | 274MB/s | |
rebuild, averaged sequential large writes | - | 367MB/s | 239MB/s | |
rebuild, averaged parallel large writes | - | 279MB/s | 129MB/s | |
rebuild, averaged sequential small writes | 6MB/s | 3MB/s | 2MB/s | |
rebuild, averaged parallel small writes | 31MB/s | 17MB/s | 19MB/s |
Comments:
- for 1) Files for read test where on compressed area of ZFS (root files, tgz files, LSST data files)
- for 1) ZFS write test during rebuild not comparable since the rebuild was always faster finished then the test (in 2h compared to 21h for hardware raid)
- for 1-4) all read tests based on the same mix of files