You are currently browsing the tag archive for the ‘sun’ tag.

Sun along with it’s Java Directory Server offering also provides a set of very useful utilities (as well as a set of LDAP libraries), called Directory Server Resource Kit (DSRK). It is available in binary form for Solaris and Redhat Linux so installation is as easy as running unzip.

The utilities it includes are quite helpful especially if you ‘re running a Java Directory Server (Log analyzer, Search Operations replay tool an so on). The documentation is quite helpful and provides all the necessary usage information. Maybe i ‘ll write a post soon about using them with your Sun DS installation.

One set of utilities included is specifically targeted at benchmarking your LDAP server operations. It is not as extensive as slamd but on the other side it’s very easy to understand and use so it was a good opportunity to use it on our OpenLDAP installation to measure it’s relative performance, especially with different configuration options.

Just as a side note, i am not trying to measure OpenLDAP performance or compare it with other products, i was just trying to find the best configuration for our setup and the software/hardware limits. The client was not even on the same network as the server although i made sure they were on a gigabit path.

License notice: According to the license accompanying DSRK “3.5 You may not publish or provide the results of any benchmark or comparison tests run on the Licensed Software to any third party without Sun’s prior written consent.” i ‘ve removed actual numbers from the post in order to stay out of trouble. If i find that i can include them, this notice will be removed and the numbers added.

All tools are multithreaded and provide command line option to set the number of threads used. In my case i found that i got the best performance when i set the number of threads to be equal to the number of real CPU cores. Also most of the tools allow random searches on the directory by performing substitutions on the provided search filter with random lines from an input file. That way you can simulate real life usage of your directory service.

First tool i used was the authentication rate measurement tool. I tried two setups. In the first the tool would perform a new connection for each bind while in the second it would keep a connection open and try rebinding. I only used one specific bind DN since i only wanted to measure the bind operation speed and not ldap entry searching. In the first case i got ‘a few’ binds/sec (with a single threaded tool run) while in the second case a multithreaded run performed a few thousands binds/sec. Using tcpdump to check out network overhead it became clear that every bind (along with connection establishment and breakup) needed 0,028 seconds with all the network packets transmitted. As a result each thread had a network upper limit of 35 binds/sec which made clear that this type of setup could not achieve extraordinary performance in a connect and bind every time setup.

The second tool was the search rate measurement tool. I always made sure to connect and bind once and use that connection for subsequent searches in order to measure only the LDAP Search speed. What i wanted to see was the impact of different configuration on the BDB and entry caches. Before each run i restarted the ldap server and performed an ldapsearch of the whole tree in order to prime the caches. All the iterations were random searches on the uid attribute requesting all entry attributes. My database holds about 24,000 entries, has a minimum BDB cache of 4MB and a total size of 109MB. The results were interesting:

BDB cache Entry cache Searches/Sec DBD cache hit ratio id2entry/dn2id/uid/total OpenLDAP process resident memory size
4MB zero


81%,87%,94%,91% 62MB
8MB zero


82%,94%,97%,93% 79MB
60MB zero


97%,99%,99%,99% 141MB
110MB zero


99% all 142MB
110MB 8000


119MB 24000






BDB cache hit ratios were taken using db_stat -m. I started by keeping entry cache to zero entries and increasing the BDB cache in order to check out the difference in performance. Initial run with 4MBs produced a few thousand searches/sec while a BDB cache of 110MB produced a medium increase while it required more than double memory. You may wonder about the negligible difference in memory usage between 60 and 110MB of BDB cache. The reason is that i only performed searches on the uid attribute so, although the total database size is 109MBs, the sum of the id2entry.bdb, dn2id.dbd and uid.bdb was only 82MB. Since BDB will automatically increase the BDB cache setting by 25%, 60MB was actually 75MB, so increasing the setting to 110MB didn’t have any real effect. Having a 99% BDB cache hit ratio though proved helpful since the search rate increased quite significantly.

The rest of the runs tried to measure the entry cache performance hit. I kept BDB cache to 110MB in order to keep database files on memory and only measure the entry cache speedup. First run was made with an entry cache of 8000 entries (1/3 of the total entries) and the other with an entry cache of 24000 entries (100% of the entries). The entry cache provided a substantial increase (with 24,000 entries entry cache) from the first run and the 110MB BDB cache run. Memory usage increase was significant though since resident size increased to 330MB (compared to 60MB with 4MB BDB cache, almost 6x times more).

The last run tried to use both caches while minimizing memory usage. The server was setup to use 8MB of BDB cache and 8000 entries entry cache. It performed more than the 110MB BDB cache run but less than the entry cache measuring tries. That’s an imporant  increase from the initial run and it reaches a large percentage of the results of the combined full BDB/Entry cache with 211MB memory usage (3,5x 60MB and 65% of the 330MB).

It’s obvious that adding more caching can change the search rate substantially although that incurs a large cost in memory usage. Also, adding more cache than your working set will probably have no result and will just use up valuable memory. Since usually memory available is not unlimited (and the machine used as an LDAP server might also perform other duties as well) i would suggest a strategy like the last run. Set BDB cache to 2x or 3x times the minimum required and the entry cache to a reasonable value to hold your working set and not consume all your system memory. If on the other hand your database is small or you have enough memory, increasing the caches can only help in performance (but always try to keep the working set and not the whole database).

The last tool used was the entry modification rate measurement tool. I kept the full BDB/Entry cache settings in order to keep my database on memory and only measure the modification overhead (and not the entry searching cost). The tool was configured to change the userpassword attribute of a random entry with a random 10 character string. Since the entry selection was totally random that meant that database and disk access would also be random and performance would be much smaller than adding entries. In my setup i had the accesslog database, the audit log and logging enabled so an entry modification actually meant multiple disk writes to all the corresponding databases and log files. I made two test runs. One with a single thread and one with 8 threads (always connecting and binding once to minimize network and connection overhead). The results were a bit poor:

  • Single threaded: A mods/sec
  • 8 threads: B (B more than A) mods/sec

Measuring i/o usage with systat i found that the disk subsystem was quite saturated performing around 250-300 transactions/sec so the above numbers were probably the actual disk limit of the server.  I wanted to see what the overhead of the accesslog and audit log was so i temporarily disabled them and run the test again. This time it reached to more than double the number of the plain 8 thread run. The above makes it obvious that adding modules like audit log or accesslog can be helpful for debugging or for increasing the replication performance but it can also hurt the write throughput a lot. As a result, the administrator should always try and keep modules that require disk writes to the minimum.

We have a 6.X setup running for quite a while (but not servicing requests yet) which will replace the current 5.X setup. In order to keep data current we have the primary 6.X master acting as a consumer of the 5.X master. We ‘ve run into the same problem many times and it seems that 5.X and 6.X act differently when adding an entry with empty RDN (something like uid=, ou=people, dc=<domain>). 5.X will allow the entry to be added while 6.X will reject it. The end result is that every time an entry like that is added (mostly by mistake) in the 5.X infrastructure, replication to the 6.X servers is halted and we need to initialize all of them from the start.

Seems strange that the servers exhibit such a different behaviour. If there’s a way to make them behave… properly i ‘d be glad to learn about it.

We ‘ve setup a DS6.X farm for the Greek School Network. It includes two datacenters (one for north and one for south Greece), each with one write master and two read-only replicas. The write masters are in a multi-master topology while the read-only replicas all get updated from both masters. The idea is to be able to lose up to one data-center without complete ldap service failure. We ‘ve been playing with the servers for about a month now and my impressions so far are:

  • The web console is far superior to the previous Java console. Much faster, easily accessible by just a web browser and nicely setup.
  • The installation is much harder now and requires far too many commands to get things going. Instead of just a zip with an installer you now have to install the directory server, play with cacao and dscc (directory server control center), add the console in the application server and learn a lot of <whatever>adm commands. Also we found the installation guide to be quite lacking and a bit unclear in a few issues.
  • Security should be tighter with the new server since you now have to worry about a dozen ports instead of just the ldap(s) and console ports. You have a few ports for cacao two ports (http/https) for the console and the usual directory server ports
  • The documentation quality is a bit degraded compared to previous products. Seems like documentation writing is starting to be outsourced to India as well 🙂 I got the impression that the 5.X documentation was written by the engineers themselves while the 6.X was written from an outside partner.
  • Until 6.1 we ‘ve faced quite a lot of stability issues (the servers run Solaris on 64-bit x86). The ldap server would crash for no reason and sometimes replication would fail and replicas would have to be reinitialized. We ‘ve installed 6.2 today and so far so good but i can say i ‘m rather disappointed with the software quality compared to the one i had gotten used to with the 5.X versions.

We haven’t tried to do any stress testing yet to see how the servers behave from a performance point. Online Replica initialization (over a LAN) on the other hand can reach speeds of at least 500 entries/sec (with 256MB import cache) which are quite impressive. I ‘ll try and update this post as things move on.

Ludo posted a nice link to some FS tuning guidelines for a DS


Read the rest of this entry »

About Me

Kostas Kalevras

LinkedIn profile

E-mail:kkalev AT gmail DOT com
My status

More about me...