After many months of straggling with hardware stability issues of our new directory service infrastructure we managed to solve them and tried to actually move to using it. What we found out was that our master servers (almost identical with the read-only replicas except for Fibre Channel capabilities and disk controller on-card instead of on-board) had terrible performance compared to the read-only replicas. Talk about taking a day(!!) to create some new sub-string indexes on a 200,000 entries database.

We needed to find a way to consistently measure i/o performance on top of the (solaris) OS (and not just a disk read benchmark) so that we could open a case to our server manufacturer. After some searching we came across PostMark, an i/o benchmarking utility originally from NetApp. It seems that the software is not maintained anymore but you can find the source code on Debian. It’s only one tiny .c file so you just need to run ‘gcc -o postmark postmark.c’ to get things going.

This is the configuration i used in order to simulate a directory server instance. In general, create 20 files ranging from 5 – 30 MB’s each (common index file size), only run reads and appends on them (with a 4/1 ratio) and no creates/deletes:

set location <your location directory>

set size 5000000 30000000

set number 20

set bias read 2

set bias create -1

The results were a bit… terrifying: Our read-only servers were actually three times faster (while having same disks and almost the same controller). It would be a nice idea to always run Postmark on your servers to see what’s happening and how different servers can handle the same i/o load.

Read-Only Replica:
2568.57 megabytes read (91.73 megabytes per second)
572.20 megabytes written (20.44 megabytes per second)

Master Server:
2568.57 megabytes read (25.69 megabytes per second) (1/3!!)
572.20 megabytes written (5.72 megabytes per second) (1/4!!)

Advertisements