Testing Cluster Redo Log Write Performance with dd

  • Note:  This is a destructive test  !
  • Don’t run this test in a production env or at least use newly created disk without any  DATA !
  • Use bs=128k for writing 128k blocks
  • Use oflag=sync to bypass buffer cache ( Oracle uses same flag )

Test Instances one after the other to get an idea about your single instance performance ( run test 3x )

Instance grac41:
# dd if=/dev/zero  of=/dev/asmdisk1_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 61.1947 s, 10.7 MB/s
--> Instance grac41:  81,96 IOPs   Transfer Rate:  10.7 MB/s

Instance grac42:
# dd if=/dev/zero  of=/dev/asmdisk2_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 41.4141 s, 15.8 MB/s
--> Instance grac42:  121.95 IOPs   Transfer Rate:  15.8 MB/s

Instance grac43:
#   dd if=/dev/zero  of=/dev/asmdisk3_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 62.8516 s, 10.4 MB/s
--> Instance grac43:  80,64 IOPs   Transfer Rate:  10.4 MB/s

 

  • We expect a cummulated Write I/O rate from about 280 IOPs for our whole cluster !

Testing all instances by running dd command in parallel to figure out any bottlenecks

[root@grac41 dev]# dd if=/dev/zero  of=/dev/asmdisk1_test bs=128k oflag=sync count=5000 
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 97.604 s, 6.7 MB/s
[root@grac42 ~]# dd if=/dev/zero  of=/dev/asmdisk2_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 97.645 s, 6.7 MB/s
[root@grac43 ~]# dd if=/dev/zero  of=/dev/asmdisk3_test bs=128k oflag=sync count=5000 
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 118.749 s, 5.5 MB/s
--> Clusterwide  IOPs:  15.000 / 118 = 127 IOPs

 

  • IOPs drops from expected rate of 280 IOPs to 127 IOPs
  • There is a Controller and/or Disk bottleneck and needs further review from storage group
  • If suffering “LOG FILE SYNC” Wait  event IOPs sequential write performance is the limiting factor and you should review whether your hardware is scaling well

Reference

Leave a Reply

Your email address will not be published. Required fields are marked *