Testing Cluster Redo Log Write Performance with dd

  • Note:  This is a destructive test  !
  • Don’t run this test in a production env or at least use newly created disk without any  DATA !
  • Use bs=128k for writing 128k blocks
  • Use oflag=sync to bypass buffer cache ( Oracle uses same flag )

Test Instances one after the other to get an idea about your single instance performance ( run test 3x )

Instance grac41:
# dd if=/dev/zero  of=/dev/asmdisk1_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 61.1947 s, 10.7 MB/s
--> Instance grac41:  81,96 IOPs   Transfer Rate:  10.7 MB/s

Instance grac42:
# dd if=/dev/zero  of=/dev/asmdisk2_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 41.4141 s, 15.8 MB/s
--> Instance grac42:  121.95 IOPs   Transfer Rate:  15.8 MB/s

Instance grac43:
#   dd if=/dev/zero  of=/dev/asmdisk3_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 62.8516 s, 10.4 MB/s
--> Instance grac43:  80,64 IOPs   Transfer Rate:  10.4 MB/s

 

  • We expect a cummulated Write I/O rate from about 280 IOPs for our whole cluster !

Testing all instances by running dd command in parallel to figure out any bottlenecks

[root@grac41 dev]# dd if=/dev/zero  of=/dev/asmdisk1_test bs=128k oflag=sync count=5000 
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 97.604 s, 6.7 MB/s
[root@grac42 ~]# dd if=/dev/zero  of=/dev/asmdisk2_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 97.645 s, 6.7 MB/s
[root@grac43 ~]# dd if=/dev/zero  of=/dev/asmdisk3_test bs=128k oflag=sync count=5000 
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 118.749 s, 5.5 MB/s
--> Clusterwide  IOPs:  15.000 / 118 = 127 IOPs

 

  • IOPs drops from expected rate of 280 IOPs to 127 IOPs
  • There is a Controller and/or Disk bottleneck and needs further review from storage group
  • If suffering “LOG FILE SYNC” Wait  event IOPs sequential write performance is the limiting factor and you should review whether your hardware is scaling well

Reference

One thought on “Testing Cluster Redo Log Write Performance with dd”

  1. # dd if=/dev/zero of=/dev/asmdisk1_test bs=128k oflag=sync count=5000
    5000+0 records in
    5000+0 records out
    655360000 bytes (655 MB) copied, 61.1947 s, 10.7 MB/s
    –> Instance grac41: 81,96 IOPs Transfer Rate: 10.7 MB/s

    I did not get , how u arrived at 81,96 number

Leave a Reply

Your email address will not be published. Required fields are marked *