FreeRADIUS 2.0 has been released after a long and productive development cycle. It’s much more scalable, fast and simple while providing even more powerful features like a policy language, virtual hosting and IPv6 support. More information available on the freeradius website.
We ‘ve setup a DS6.X farm for the Greek School Network. It includes two datacenters (one for north and one for south Greece), each with one write master and two read-only replicas. The write masters are in a multi-master topology while the read-only replicas all get updated from both masters. The idea is to be able to lose up to one data-center without complete ldap service failure. We ‘ve been playing with the servers for about a month now and my impressions so far are:
- The web console is far superior to the previous Java console. Much faster, easily accessible by just a web browser and nicely setup.
- The installation is much harder now and requires far too many commands to get things going. Instead of just a zip with an installer you now have to install the directory server, play with cacao and dscc (directory server control center), add the console in the application server and learn a lot of <whatever>adm commands. Also we found the installation guide to be quite lacking and a bit unclear in a few issues.
- Security should be tighter with the new server since you now have to worry about a dozen ports instead of just the ldap(s) and console ports. You have a few ports for cacao two ports (http/https) for the console and the usual directory server ports
- The documentation quality is a bit degraded compared to previous products. Seems like documentation writing is starting to be outsourced to India as well
I got the impression that the 5.X documentation was written by the engineers themselves while the 6.X was written from an outside partner. - Until 6.1 we ‘ve faced quite a lot of stability issues (the servers run Solaris on 64-bit x86). The ldap server would crash for no reason and sometimes replication would fail and replicas would have to be reinitialized. We ‘ve installed 6.2 today and so far so good but i can say i ‘m rather disappointed with the software quality compared to the one i had gotten used to with the 5.X versions.
We haven’t tried to do any stress testing yet to see how the servers behave from a performance point. Online Replica initialization (over a LAN) on the other hand can reach speeds of at least 500 entries/sec (with 256MB import cache) which are quite impressive. I ‘ll try and update this post as things move on.
Ludo posted a nice link to some FS tuning guidelines for a DS
Updates…
It was the first conference that I almost forgot about using the Internet. Really excellent presentations and top participants. Also, having your room just a few floors over the the conference rooms was a nice change compared to the usual TERENA conference setting (conference held in a university with the hotel usually being over an hour away).
Day one
The conference started with a nice overview of the standards status by Kurt Zeilenga, followed by a talk from Ludovic Poitu on the merits of the upcoming OpenDS. Seems like it’s getting to a pretty mature status and the figures are impressive: 10 times faster than Sun One and they still have not worked on optimizations. From the looks of it, 2008 will be OpenDS year since 1.0 will be released probably before the end of 2007.
Java based LDAP servers got a lot of attention in this conference. Alex Karasulu described Apache view on LDAP roadmap as well as Apache Directory. They envision Directory Services resembling RDBMS offering Triggers, Stored Procedures and views (though the later could be implemented with a strong proxying interface like the one available in OpenLDAP). Ersin Er gave a more detailed presentation on the actual implementation of Stored Procedures in Apache DS. The idea is to offer an API and the ability to write Java code to implement operations while triggers will actually have a stored procedure as the scheduled action. Personally, i feel a little nervous about having code executed inside the server context. Although triggers might be a nice (and less heavy) alternative to things like persistent searches.
Howard Chu, chief architect of OpenLDAP and employee at Symas gave a nice presentation on the status of version 2.4. Benchmark numbers are very, very impressive (150 million entries, 4800 writes/second 32,000 queries/second, only 6 hours load time) while the cn=config and dynamic configuration/loading of everything makes remote configuration a reality and restarts a thing of the past. If only there was a strong configuration GUI for things like configuring Syncrepl. N-way multimaster replication is also now available making OpenLDAP an equal competitor to commercial offerings.
Giovanni Baruzzi gave an enlightening presentation on how to properly design an LDAP Directory Information Tree (DIT). The main idea is: ‘Keep the tree as flat as possible, as deep as needed’. Groups implementation were also discussed in detail: Static groups can work great as long as they are under 80,000 members (at which case an update can take more than 5 minutes) while memberOf is very flexible but poses security risks (write access to the memberOf attribute means that an administrator can add a user to any of the available groups). I wasn’t able to attend the presentation on Apache Directory Studio but i downloaded it later and played with it. Very impressive and long awaited set of tools. It includes a powerful directory browser, an entry/schema editor as well as some Apache Directory specific tools for ACI and configuration editing.
Hilla Reynolds from far away Australia presented an in-house X.500/LDAP infrastructure with advanced features including geographical distribution of data access, concurrent replication (though i believe this is something difficult to achieve without enlarging the directory update time) and chained queries in order to return combined results from multiple sources.
Day two
Second day started with Steven Legg presenting LDAP and XML integration process. Nice work though a bit technical and hard to follow. I ‘d like to see where things will lead, although at this point the only actual implementation is by him. Next were two presentations from Sun employees on LDAP Proxies/Virtual Directories and Scaling Directories.
Following was … my presentation One excellent and lively presentation from Felix Gaehtgens on how to write efficient LDAP applications followed. Keynotes were to keep connections open, parallelize operations through either multiple threaded connections or by using asynchronous reads (though the later involves more effort from the application writer), not using ‘Directory Manager’ to perform all operations and making use of the ProxyAuth mechanism if possible. I am happy that FreeRADIUS already uses most of the above directives in the LDAP module. Lastly, the conference was closed with a presentation from Volker Lendecke on lessons learned from Samba’s LDAP backends. The general conclusion was that various libc functions were broken and samba had to reimplement them correctly.
Some more blogs on the conference:
I will try and find if the conference proceedings will become available online. If that happens I will post the link here.
Got my paper included in the 1st LDAP Conference which will take place in Cologne, Germany between 6-7 September. Seems that all of the right people will be there including Kurt Zeilegna (OpenLDAP founder), Howarch Chu (SYMAS - OpenLDAP), Alex Karasulu (ApacheDS), Ludovic Poitou (Sun, OpenDS). Usually i can only find a few presentations in a conference that i feel i must attend. In this case i cannot find a single presentation not worth it’s time. Seems it will be two very busy days.
Abstracts Page: http://www.guug.de/veranstaltungen/ldapcon2007/abstracts.html
Conference Page: http://www.guug.de/veranstaltungen/ldapcon2007/index.html
I attended the TERENA Networking Conference 2007 held in Lyngy, Denmark (a town just outside Copenhagen). I mostly focused on Security and Single Sign On presentations. Overall, i wasn’t impressed by the presentations quality and could only find a few that were really interesting and could keep the audience .. awake.
It seems that RedIRIS is doing a lot of work in the SSO field, let’s see what the end result will be.
I liked the siLEDAP paper, describing an AJAX based LDAP browser along with a set of Web Services (using REST) to perform read and write LDAP operations. It seems that the author is quite content with the product as it is and would not want to move it forward (I was curious if it could get integrated with LUMS). Also, when asked about why they didn’t use DSML as a transport for the LDAP entries data his answer was… ‘I don’t know’
A couple of other presentations that caught my eye were ‘Internet Voting - a Menace to Society?’ describing a real life internet voting system and ‘Current Network Security threats: DoS, Internet Worms, and Botnets’ which posted results on monitoring of a large unused network for malicious internet traffic.
Found the architecture notes on Varnish, a high performance caching reverse proxy from Poul-Henning Kamp, a veteran FreeBSD kernel developer. Nice description of ways to use the operating system to your benefit and to minimize the costs of memory management.
And the blog where I found the link to the notes
Alan Dekok (FreeRADIUS chief architect) has been playing around with performace and there’s an ongoing discussion on the mailing list about memory management, so this might prove relevant.
On a recent post i pointed out the advantages of moving ldap writes to web services. I also stated that we couldn’t make the current interface available but another was on the works. Well, after a few days of coding i now have that interface available on sourceforge.
I named it LUMS (LDAP User Management Service). It basically provides a set of basic API functions (search, add, delete, modify, rename, change password), written in PHP and a strong configuration language. This API can then be used to create web services (or used in any PHP script to say the truth). The language allows the administrator to define ldap object types along with their corresponding attributes. For each attribute a whole bunch of options is available:
- define it as required, multivalued
- set the attribute type (string,binary,dn,telephone,mail etc)
- define the attribute type. Can be user inserted, constant, auto increment, function created
- allow for attribute uniqueness
- define extra syntax checking functions
Moreover, pre and post operation functions can be defined while the interface takes care of handling non English char-set attribute values. More information is available in the (small) README and configuration comments. Hope people find it useful. It surely still needs work but it works.
Here’s a small snapshot of the configuration to get a basic idea:
A recent post on the Connexitor Blog was about performance and how modern software often lacks it. The post was mostly directed towards Sun and Redhat for stating that they provide a state of the art, fast directory server and how that’s not true compared to the performance numbers of these products against OpenLDAP. Later on it pointed out that modern development usually uses a lot of abstraction layers and ends up creating bloated and heavy products which cannot achieve decent performance.
I must agree on the optimization comments. We try to do the same in FreeRADIUS, adding features as well as optimizing things whenever possible in order to create a flexible yet blazing fast server.
I will take the role of the devil’s advocate on the directory server front though.
Most organizations, companies and universities have a typical user population of much less or at least not exceeding 250,000 entries. The less entries you have, the less interested you are on sheer performance numbers of your directory server. Usually, administrators will take many more factors into consideration than just the number of auths/sec. And, according to Symas own benchmarks FDS can perform 4,500 auths/sec and 48,000 entries/sec for a 250,000 entries database. Numbers that are usually more than enough for the kind of load expected from a user population of that scale.
What administrators will also look out for when choosing a directory server (apart from performance number) are:
- Dynamic configuration through an ldap interface
- ACIs on the DIT
- Configuration console for easy server administration
- Monitoring capabilities
- An included ldap proxy server
- Active Directory Sync included with the directory
- Extensive documentation, preferably in the form of various guides (Administrator’s, Deployment, Tuning, Installation, and so on)
- Ease of installation
- Available plugins, backends, overlays. Features like class of service, roles, referential integrity, attribute uniqueness, password policy.
- Commercial support and training
If someone were to check the above list with both FDS and OpenLDAP he might find out that overall FDS might be better equipped to handle the above requirements. Specifically:
- Draw, although i think that OpenLDAP allows for an even broader range of changes while the server is running
- Draw
- FDS has a Java Console, i am not aware of a console offering for OpenLDAP.
- Both servers provide a monitoring backend. FDS also provides cache hit statistics something that i am not sure OpenLDAP provides.
- A big plus for OpenLDAP. You might end up installing OpenLDAP along side FDS only to provide an ldap proxy functionality (if you are in need of one).
- FDS provides one. I am not aware of something similar for OpenLDAP
- FDS provides a far richer suite of documentation all readily available on it’s website. OpenLDAP documentation is lacking, especially on areas such as fine tuning the server (there are multiple cache memory settings and other configuration directives which are available for tuning) or setting up replication (with examples and use cases). It’s not uncommon to have to look through mailing lists archives in order to find more information on various aspects of OpenLDAP advanced configuration.
- FDS provides a setup utility which allows for easy first time configuration and multiple silent installs. I haven’t seen what CDS has to offer in this area so i cannot comment on this subject.
- Both products are quite feature full. OpenLDAP provides more exotic plugins,overlays and backends. FDS provides the Class of Service and Roles mechanisms which can prove very useful at times (i haven’t found something similar with OpenLDAP).
- Draw if you are willing to use the commercial versions of the products (Redhat DS or CDS).
Overall, i wish there was an rpm for OpenLDAP which included the setup utility of FDS, multiple documentation guides and a strong console interface allowing the administrator to tune almost every aspect of the server, especially things like performance and replication. Or that FDS included an LDAP Proxy, more plugins and performed as well as OpenLDAP :
I don’t intend the above as a flame to OpenLDAP. I actually like the software, how rich it is in available features and we use it in my university as an LDAP Proxy. I am just trying to point out the fact than when you want to call your product the best you have to see beyond performance and take a look at other important issues. I really do hope that both products will improve on the areas they are lacking and we ‘ll end up with a couple of really strong packages.
Update: Symas (Howard Chu) sent me some comments on the above checklist as well as some more thoughts. I ‘ll agree with everything, just have to point out once more than the above refer to installations holding less (or far less) than 250,000 entries.
Symas (big commercial backers and developers for OpenLDAP) posted benchmark numbers on their site comparing OpenLDAP with the latest versions of FDS (Fedora DS), OpenDS (the Java based Directory Server from Sun which is still in the early stages of development) and ApacheDS (which does not seem to be able to handle real world loads).
It seems that OpenLDAP outperforms the rest many times over, with FDS having trouble keeping 1 million entries in 8MB of RAM. Good news for OpenLDAP, bad news for FDS (which needs much more work, especially for large scale cases) and it makes me wonder how Sun ONE would perform (since it shares the same code base with FDS).
In my last post i went through some tips on RADIUS performance. This time i ‘ll look into issues with RADIUS server redundancy. I ‘ll focus on accounting redundancy. You can easily use things like LVS, VRRP, LDAP and SQL replication to achieve radius server and authentication database redundancy. Having a redundant accounting database is a bit trickier. The reason:
- Accounting is several times larger than authentication information and is ever growing and changing (instead of pretty much static data in the authentication database).
- Regardless of it’s size you still want accounting to be synchronized between all your redundant radius servers so that you can perform network wide double login detection, ip and resource management.
So what i would suggest to do is the following:
- Keep the live accounting table (the one holding online users, already discussed in my post on performance), the nas table (holding the radius server clients) and the ip pool table (used by rlm_sqlippool) in a replicated sql database. MySQL does not support Multimaster replication (it would be very difficult to do that anyway) so you must make sure to:
- Perform all sql queries to a single Virtual SQL server IP
- Configure High Availability between all replicated MySQL servers so that only one, working server holds the virtual IP at any time.
- Don’t use modules like rlm_ippool which depend on local files for storage and therefore cannot be easily replicated.
- You could also keep the radacct table replicated and only log to the Virtual IP but i would suggest against it. You can usually tolerate a small delay in accounting synchronization between your multiple radius servers (as long as you keep information like currently logged in users on a replicated table). Just imagine the scenario that replication fails and you have to manually synchronize all MySQL servers with a multimillion row radacct table. Nightmare! Instead just log to a detail file and have radrelay (or the radius server in radrelay mode in recent versions) send accounting to the rest of the redundant servers. You will need one detail file for each redundant server so this solution will not scale if you have more than a couple of radius servers. On the other hand, if you are looking for a multi-server setup, you will need to use an SQL server cluster. The above mechanism will also keep things like rlm_counter synchronized.
The above suggestions assume that you have a redundant permanent store on each radius server (raid-1/5). If you permanently lose data stored on a disk, no replication strategy can provide 100% guarantee. SQL (or RADIUS) server will first store data locally, acknowledge the fact to the client and afterwards replicate data to the rest of the servers. If this data is lost before it is replicated, it is lost forever.
