Sun along with it’s Java Directory Server offering also provides a set of very useful utilities (as well as a set of LDAP libraries), called Directory Server Resource Kit (DSRK). It is available in binary form for Solaris and Redhat Linux so installation is as easy as running unzip.
The utilities it includes are quite helpful especially if you ‘re running a Java Directory Server (Log analyzer, Search Operations replay tool an so on). The documentation is quite helpful and provides all the necessary usage information. Maybe i ‘ll write a post soon about using them with your Sun DS installation.
One set of utilities included is specifically targeted at benchmarking your LDAP server operations. It is not as extensive as slamd but on the other side it’s very easy to understand and use so it was a good opportunity to use it on our OpenLDAP installation to measure it’s relative performance, especially with different configuration options.
Just as a side note, i am not trying to measure OpenLDAP performance or compare it with other products, i was just trying to find the best configuration for our setup and the software/hardware limits. The client was not even on the same network as the server although i made sure they were on a gigabit path.
License notice: According to the license accompanying DSRK “3.5 You may not publish or provide the results of any benchmark or comparison tests run on the Licensed Software to any third party without Sun’s prior written consent.” i ‘ve removed actual numbers from the post in order to stay out of trouble. If i find that i can include them, this notice will be removed and the numbers added.
All tools are multithreaded and provide command line option to set the number of threads used. In my case i found that i got the best performance when i set the number of threads to be equal to the number of real CPU cores. Also most of the tools allow random searches on the directory by performing substitutions on the provided search filter with random lines from an input file. That way you can simulate real life usage of your directory service.
First tool i used was the authentication rate measurement tool. I tried two setups. In the first the tool would perform a new connection for each bind while in the second it would keep a connection open and try rebinding. I only used one specific bind DN since i only wanted to measure the bind operation speed and not ldap entry searching. In the first case i got ‘a few’ binds/sec (with a single threaded tool run) while in the second case a multithreaded run performed a few thousands binds/sec. Using tcpdump to check out network overhead it became clear that every bind (along with connection establishment and breakup) needed 0,028 seconds with all the network packets transmitted. As a result each thread had a network upper limit of 35 binds/sec which made clear that this type of setup could not achieve extraordinary performance in a connect and bind every time setup.
The second tool was the search rate measurement tool. I always made sure to connect and bind once and use that connection for subsequent searches in order to measure only the LDAP Search speed. What i wanted to see was the impact of different configuration on the BDB and entry caches. Before each run i restarted the ldap server and performed an ldapsearch of the whole tree in order to prime the caches. All the iterations were random searches on the uid attribute requesting all entry attributes. My database holds about 24,000 entries, has a minimum BDB cache of 4MB and a total size of 109MB. The results were interesting:
|BDB cache||Entry cache||Searches/Sec||DBD cache hit ratio id2entry/dn2id/uid/total||OpenLDAP process resident memory size|
BDB cache hit ratios were taken using db_stat -m. I started by keeping entry cache to zero entries and increasing the BDB cache in order to check out the difference in performance. Initial run with 4MBs produced a few thousand searches/sec while a BDB cache of 110MB produced a medium increase while it required more than double memory. You may wonder about the negligible difference in memory usage between 60 and 110MB of BDB cache. The reason is that i only performed searches on the uid attribute so, although the total database size is 109MBs, the sum of the id2entry.bdb, dn2id.dbd and uid.bdb was only 82MB. Since BDB will automatically increase the BDB cache setting by 25%, 60MB was actually 75MB, so increasing the setting to 110MB didn’t have any real effect. Having a 99% BDB cache hit ratio though proved helpful since the search rate increased quite significantly.
The rest of the runs tried to measure the entry cache performance hit. I kept BDB cache to 110MB in order to keep database files on memory and only measure the entry cache speedup. First run was made with an entry cache of 8000 entries (1/3 of the total entries) and the other with an entry cache of 24000 entries (100% of the entries). The entry cache provided a substantial increase (with 24,000 entries entry cache) from the first run and the 110MB BDB cache run. Memory usage increase was significant though since resident size increased to 330MB (compared to 60MB with 4MB BDB cache, almost 6x times more).
The last run tried to use both caches while minimizing memory usage. The server was setup to use 8MB of BDB cache and 8000 entries entry cache. It performed more than the 110MB BDB cache run but less than the entry cache measuring tries. That’s an imporant increase from the initial run and it reaches a large percentage of the results of the combined full BDB/Entry cache with 211MB memory usage (3,5x 60MB and 65% of the 330MB).
It’s obvious that adding more caching can change the search rate substantially although that incurs a large cost in memory usage. Also, adding more cache than your working set will probably have no result and will just use up valuable memory. Since usually memory available is not unlimited (and the machine used as an LDAP server might also perform other duties as well) i would suggest a strategy like the last run. Set BDB cache to 2x or 3x times the minimum required and the entry cache to a reasonable value to hold your working set and not consume all your system memory. If on the other hand your database is small or you have enough memory, increasing the caches can only help in performance (but always try to keep the working set and not the whole database).
The last tool used was the entry modification rate measurement tool. I kept the full BDB/Entry cache settings in order to keep my database on memory and only measure the modification overhead (and not the entry searching cost). The tool was configured to change the userpassword attribute of a random entry with a random 10 character string. Since the entry selection was totally random that meant that database and disk access would also be random and performance would be much smaller than adding entries. In my setup i had the accesslog database, the audit log and logging enabled so an entry modification actually meant multiple disk writes to all the corresponding databases and log files. I made two test runs. One with a single thread and one with 8 threads (always connecting and binding once to minimize network and connection overhead). The results were a bit poor:
- Single threaded: A mods/sec
- 8 threads: B (B more than A) mods/sec
Measuring i/o usage with systat i found that the disk subsystem was quite saturated performing around 250-300 transactions/sec so the above numbers were probably the actual disk limit of the server. I wanted to see what the overhead of the accesslog and audit log was so i temporarily disabled them and run the test again. This time it reached to more than double the number of the plain 8 thread run. The above makes it obvious that adding modules like audit log or accesslog can be helpful for debugging or for increasing the replication performance but it can also hurt the write throughput a lot. As a result, the administrator should always try and keep modules that require disk writes to the minimum.