You are currently browsing the category archive for the ‘Web Services’ category.

In previous posts i pointed out the benefits of using web services (instead of direct LDAP access) to perform LDAP writes. In this post i ‘m going to take the whole discussion a few steps further and propose a mechanism (based on the basic principles of the LDAP Synchronization Protocol used by OpenLDAP for replication) to keep heterogeneous data systems synced (in our case a database with an LDAP server).

There are two, quite different scenarios when writing to an LDAP server (which actually depend on the client performing the updates):

  • The first one is the case where you are using some sort of administration/modification interface (usually a web page) and perform the modifications through it. In this case the user is performing changes online and will usually wait until he gets a positive response from the underlying LDAP server. Even if you use a web services mechanism to perform the LDAP changes the end result will be the same: The web service function will return with a positive (or negative) result along with the corresponding error message if the operation failed at some point. The user is then able to take any necessary actions to make things work (like change a value to meet some constraints, use an attribute value that is not already used if we have a unique restriction on that attribute and so on) or just abandon the operation and move on.  The changes performed are targeted, small,  contained and serialized. Most of the error handling and decisions are performed by the user himself. The web services interface needs just perform the actions and return with the result without worrying about ‘keeping things in sync’.
  • The second scenario is one where you have an external data source of some kind (usually a database) which needs to keep things synchronized with LDAP. Say your main employee source is your payroll system and you need to keep your LDAP synced with that (add people when they join the firm, modify them when needed and delete them when they leave). In this case you will usually create a web services operation interface and perform every operation when needed (call a web service add() operation when you create a user in the payroll system and so on).

The second scenario poses a few serious constraints and problems:

  1. Before enabling the above mechanism the external data source and LDAP need to be in sync. If your LDAP is empty and your payroll system already contains users you need a way to actually populate (usually with a manual offline mechanism) your Directory.
  2. You need a queuing mechanism so that changes are buffered and sent
    1. at a rate that the LDAP server can handle
    2. even if the LDAP server is down for a while
  3. Changes should be sent serialized, in the order they were committed to the data source and you cannot afford to lose a modification (that might break future modifications that depend on a previous one).
  4. An error in an operation should not put the whole synchronization process to a halt and it should be recoverable in most cases.
  5. If somehow you lose a modification you need a mechanism to resync the two parties.

The above complexities make it obvious that just keeping a changelog and performing the operations committed through the web services mechanism is not actually enough. What is needed is a full, robust, synchronization protocol. The OpenLDAP folks have gone through great pains to design and implement such a mechanism, called the LDAP Syncronization Protocol (described in the OpenLDAP Admin Guide and in RFC 4444). It’s a nice move to use the work done and keep the basic principles when designing a mechanism for our case.

First of all we need to define a key used to synchronize the entries. That key is actually application specific. We could have a unique employee id in our payroll system and decide to keep the same value in the employeenumber attribute in our LDAP entries. Obviously, we could have multiple data sources for each entry (for different attributes), in which case we need to define one key for each data source.

Along with that we have to define a new attribute that will hold the data sources for each entry (if we have just one that step could be skipped). The attribute will be multivalued and have one value for every data source (for instance myCompanyEntryDataSource=payroll). We will need to define a ‘friendlyname’ for each data source in our case.

If we keep multiple entry types (for instance students, faculty, employees) that each need special handling we will need to define an attribute to keep that one too (sometimes eduPersonAffiliation will suffice). If our entries are all of the same type we can skip that one too.

Thankfully, we don’t need to change much in our data source. The only thing that would be required is to keep a Global Data Version (GDV). That would just be a global, unique number that will be incremented for every write (add/delete/modify) operation on the database. The GDV could be maintained as a separate resource and be atomically and transaction-ally maintained for every operation committed (it must be incremented only by one on every operation, the increment operation should be atomic and an write operation on our data must always end with an increment on the GDV value).

We now have to define a few more things:

  • Keep the global data version in our LDAP tree base in the form of a multivalued attribute. We keep one value for each data source and distinguish between them by using value tags (based on the data source friendly name we have defined). The GDV could be maintained on our data tree base entry for easy access.
  • Create a PresentEntry() web service operation. That operation will just send all data (that a data source holds and shares with the LDAP directory) for an entry and update (or create if necessary) the entry on the LDAP  accordingly. If the entry is synced no modification will be performed.
  • Add an update timestamp attribute to the schema that will be added on all entries and will be updated by the PresentEntry() operation.

We are now one step before actually defining our synchronization mechanism. The last step is to manually walk our LDAP tree and set the data source attribute accordingly for all the entries that depend on it (if a data source attribute is actually maintained).

Synchronization Process:

  • Call a web service called GetDataVersion(). That will return the GDV for our data source (or null if we have never synchronized). If the GDV is stale (compared to the one maintained in the data source) then we can actually start the synchronization. Otherwise there’s no reason to perform any further action.
  • Call a web service called StartTotalUpdate(DataVersion, UpdateTimestamp). DataVersion is the GDV of the source, while UpdateTimestamp is just the timestamp of our call to StartTotalUpdate(). We use this function in order to signal the start of the synchronization process as well as to achieve isolation. We don’t allow the sync process to run more than once simultaneously. The operation will return:
    • DataCurrent if no action is needed
    • UpdateStarted
    • UpdateInProgress if another update is already in progress
    • NoDataVersion if LDAP does not have a GDV (in the case of initialization)
    • Error in case of an error (like the LDAP server is not available)

    It will also return a TotalUpdateId unique number which will need to be used by all subsequent web service operations (as an extra argument).

  • Call the PresentEntry() operation for all the entries maintained on the data source (which should be synchronized with the LDAP directory). The call will be serialized and the operation on the LDAP could implement a throttling mechanism in order to perform operations with a rate that the LDAP server can handle. WS-ReliableMessaging could also be used to make sure that calls to the operation are ordered and happen only once. Every entry that is updated will also get an update timestamp attribute with the value of the UpdateTimestamp passed in StartTotalUpdate(). If the operation hits an unrecoverable error it will return the error to the caller and the update process will stop.
  • Call a web service called EndTotalUpdate(TotalUpdateId, Boolean success, error message). This operations signals the end of the update process. If the update failed at some point we can signal it and also log a corresponding error message. If it was successful we update the GDV attribute in the base entry and run a post update process that will scan all entries (with a data source attribute corresponding to this source) that have an update timestamp attribute with a value other than the UpdateTimestamp. These entries should just be deleted.

When we initiate a total update and assign a totaupdateid we can also assign an idle timeout and an idle timer. Every call to the PresentEntry() operation will zero the timer. If the timer exceeds the idle timeout then we kill the update process and return a corresponding error to a call for the PresentEntry() or EndTotalUpdate() operations.

The above need minimum changes in the data source and only require coding of the PresentEntry() caller and provider. On the other hand even a single update needs walking through all the data maintained which is not scalable for more than a few hundred entries. If we have already implemented operation specific web service operations (add, change, delete etc) and we are willing to maintain a changelog on our data source we can define a process to only send updates to the LDAP and keep the above process for initial syncing and as a fallback if we lose sync.

In this case we need to create a changelog in the data source that will be updated on every write operation with the following data:

  • GDV: The Global Data Version that corresponds to this operation
  • Entry unique id
  • Entry type
  • Change Type
  • Changed Data

In this case the update process can be defined as follows (quite similar to the total update process actually):

  • Call the GetDataVersion()
  • Call a StartUpdate() service similar to StartTotalUpdate() (with the same arguments, return values and funcionality)
  • Call the operation specific web services.  The services perform the operation, update the update timestamp attribute and return success or error. An optimization would be to only create a delete() operation for entry removal and keep the PresentEntry() operation for the rest of you needs.
  • Call a EndUpdate() service similar to EndTotalUpdate(). The only difference is that in this case there’s no reason to walk through the tree and search for entries that need deletion.

The above process can be used if the LDAP tree has been initialized and the changelog contains enough data to perform synchronization. Otherwise a total update is performed and we fall back to normal changelog based updates.

Since it was always hard to find documentation i ‘ve gathered available noSOAP API documentation source here:

SOAP Arrays

I had a problem with using arrays with certain WS clients. It seems that nuSOAP uses SOAP array which are deprecated in favor of xml sequences according to WS-I. Here’s a small piece of code to create:

  • An array of strings
  • A soap structure
  • An array of the above structure
$server->wsdl->addComplexType(
        'ArrayOfDN',
        'complexType',
        'array',
        $soap_compositor,
        '',
        array(
                'dn' => array('name' => 'dn', 'type' => 'xsd:string',
                        'minOccurs' => '0', 'maxOccurs' => 'unbounded')
        )
);

$server->wsdl->addComplexType(
        'RoleDesc',
        'complexType',
        'struct',
        $soap_compositor,
        '',
        array(
                'cn' => array('name' => 'cn', 'type' => 'xsd:string'),
                'description' => array('name' => 'description', type => 'xsd:string')
        )
);

$server->wsdl->addComplexType(
        'ArrayOfRoleDesc',
        'complexType',
        'array',
        $soap_compositor,
        '',
        array(
                'desc' => array('name' => 'desc', 'type' => 'tns:RoleDesc',
                        'minOccurs' => '0', 'maxOccurs' => 'unbounded')
        )
);

$soap_compositor can have a value of  ‘all’, ‘sequence’ or ‘choice’. In our case the value is sequence.

UTF-8 encoded values (foreign charsets)

Another thing to keep in mind is UTF-8 encoded values (usually in foreign charsets like greek in my situation). The best approach is to use the iconv() php function to transform native charset (eg ISO-8859-7) to UTF-8 for transport and disable nusoap automatic utf8 decoding. You should also make sure to set the http transport charset to UTF-8 instead of the default iso-8859-1. The corresponding values are:

  • var $decode_utf8 = false;  in the nusoap_client and nusoap_server classes
  • var $soap_defencoding = ‘UTF-8′; in the nusoap_base class

Document/literal instead of RPC/Encoded

Modern web service clients usually request web services to be written in document-literal mode. Thus you should call the configureWSDL() and register() functions this way:

$server->configureWSDL(‘ReaderInterface’,”$url”,false,’document’);

$server->register(‘<function name>’,
array(‘argument’ => ‘xsd:string’),
array(‘retval’ => ‘xsd:string’),
false, #namespace
$url . ‘#<function name>’,
‘document’,
‘literal’,
‘<function documentation>’);

In current nuSOAP version there is no way to omit the namespace attribute in the bindings part of the WSDL. That means that you will probably hit on warning such as:

R2716 WSI-BasicProfile ver. 1.0, namespace attribute not allowed in doc/lit for soapbind:body: “SearchUser”
line 370 of file:/C:/temp/NetBeansProjects/testWSDL3/xml-resources/web-service-references/reader/wsdl/<hostname>/ws/reader.php.wsdl

when attempting to parse the WSDL in your client. For now you have to save the WSDL and remove the namespace attributes by hand. Hopefully the nuSOAP people will have it fixed soon.

SOAP faults

One common problem that you may hit on is to get an error when dissecting a soap fault. The usual fault is something like:

“No NamespaceURI, SOAP requires faultcode content to be a QName”

That means that when creating a soap fault with new soap_fault($faultcode, $faultfactor, $faultstring, $faultdetail) the $faultcode should be something like SOAP-ENV:Client or SOAP-ENV:Server instead of just ‘Client’ or ‘Server’ just as it is described in the class prototype in nusoap.php.

We ‘ve been using nuSOAP as a PHP web services framework for quite some time. It’s just a couple of PHP files meaning that you only need to include it in your php code, it’s easy to code and elegant. You just register functions and the framework takes care of creating WSDL (through a ?wsdl binding in your php web service pages) and all the SOAP communication with minimal effort. The problem is that it’s not maintained anymore and thus there’s no real support for the WS-*specification  stack. We ‘re particularly interested in the WS-Security in our case. WSO2 provides a PHP framework that provides all that (based on the Apache Axis2/C code). The API is quite easy to understand, supports using REST style calls and consuming WSDL. The WSDL mode is the easiest to use for writing both the client and the server but requires having the WSDL file ready.

Got my paper included in the 1st LDAP Conference which will take place in Cologne, Germany between 6-7 September. Seems that all of the right people will be there including Kurt Zeilegna (OpenLDAP founder), Howarch Chu (SYMAS – OpenLDAP), Alex Karasulu (ApacheDS), Ludovic Poitou (Sun, OpenDS). Usually i can only find a few presentations in a conference that i feel i must attend. In this case i cannot find a single presentation not worth it’s time. Seems it will be two very busy days.

Abstracts Page: http://www.guug.de/veranstaltungen/ldapcon2007/abstracts.html

Conference Page: http://www.guug.de/veranstaltungen/ldapcon2007/index.html

On a recent post i pointed out the advantages of moving ldap writes to web services. I also stated that we couldn’t make the current interface available but another was on the works. Well, after a few days of coding i now have that interface available on sourceforge.

I named it LUMS (LDAP User Management Service). It basically provides a set of basic API functions (search, add, delete, modify, rename, change password), written in PHP and a strong configuration language. This API can then be used to create web services (or used in any PHP script to say the truth). The language allows the administrator to define ldap object types along with their corresponding attributes. For each attribute a whole bunch of options is available:

  • define it as required, multivalued
  • set the attribute type (string,binary,dn,telephone,mail etc)
  • define the attribute type. Can be user inserted, constant, auto increment, function created
  • allow for attribute uniqueness
  • define extra syntax checking functions

Moreover, pre and post operation functions can be defined while the interface takes care of handling non English char-set attribute values. More information is available in the (small) README and configuration comments. Hope people find it useful. It surely still needs work but it works.

Here’s a small snapshot of the configuration to get a basic idea:

Read the rest of this entry »

We administer the Greek School Network Directory Service which currently contains more than 170,000 entries including school accounts for email and dialup access, teacher accounts and several student accounts (though this service is just starting). Services include email accounts, dialup access, web pages, VoIP and others. User administration is done through a feature-full web administration interface created by CTI at Patras. This administration interface includes powerful features like:

  • Maintaining referential integrity for configurable attributes
  • Maintaining attribute uniqueness
  • Creating attributes based on the value of other attributes (to be exact any php function or expression may be used when computing attribute values which is even more powerful).
  • Performing post operation tasks like creating user directories, sending welcome emails and so on.

Greek School Network is moving towards the e-school framework which apart from the currently available services includes:

  • A web portal (sPortal) for student parents which allows them to view student data like missed classes and others
  • A school administration platform which will move all school operations (student enrollment, classroom management, grading) to the electronic world.

These new services create new sources of information for the existing Directory Service. Parents will obtain accounts in the web portal while the school administration platform will create student accounts for all the Greek students. Allowing these services to administer these entries through plain LDAP poses some serious drawbacks:

  • Each service only has knowledge of it’s own little world (and attributes). The sPortal for instance just needs to create simple parent username/password for access to the Portal. It is not concerned with the fact that the created account might also be entitled to email or VoIP access.
  • There is no way to perform post operation tasks like creating user directories.
  • Each service is given too much power over the Directory Service. There’s almost no control (apart from ACIs) of what is added to the directory and no constraints can be set on the incoming attribute values (ok you have that ability in latest OpenLDAP releases but you get my point).

The way we decided to overcome the above difficulties was by creating a web service interface around the already excisting user interface. The web service uses WSDL and SOAP over HTTP(S) to create a function interface to all abstract operations needed by the external services. Each time a parent has to be created in sPortal for instance the portal will call the CreateParent() functions with appropriate arguments. This function will perform all the necessary checks on the arguments and call the internal object creation function of the user administration interface. That way:

  • We use the same function backend for both the user administration interface and the web services.
  • Complete and configurable logging of all operations is available with much more detail than that provided in LDAP server logs.
  • Referential integrity and attribute uniqueness is already available in the administration interface backend.
  • Computed attributes values are available using any valid php function or expression for computing values. Something like Class of Service but with much more power and control.
  • Pre and Post operation tasks can be performed through the backend (which can call outside scripts or other web services).
  • All operations pass through a single point where we have complete control over what happens and by whom. We can set constraints on attribute values and do extra checks on these values (for instance we can make sure that an incoming certificate actually belongs to the corresponding user entry).
  • Outside services don’t need to have deep knowledge of our entry scheme. They just need to call already defined functions (with the minimum set of arguments) and the web services/backend handles the rest. We are free to change the entry scheme whenever we want, adding or removing computed and static attributes to the ones sent by the web service.
  • We can impose our own entry expiration policy. The EntryDelete() web service function might end up just setting an active=false attribute inside the entry allowing us to decide when to actually delete the entry and/or perform any other tasks necessary.

Unfortunately, we cannot release the administration interface to the general public. We hope to be able to do that in the near future though since we have to create a web service interface like the one described above for the e-University project. In my opinion things are going to move to the realm of web services as far as writes are concerned. DSML is already available in most LDAP server offerings; creating a web service function interface around the actual LDAP operations is the next step forward in my vision. That is the vision of the XML Enabled Directory Internet draft. Although i believe that just translating LDAP operations to XML is not enough. In our case doing that would:

  • Increase web service development time proportionally
  • Negate some of the basic advantages of the current schema. Clients call abstract functions (ParentCreate(),ParentUpdate()) and don’t need to trouble themselves with the underlying database (LDAP), the semantics of LDAP operations or even the complete entry schema (they only need to provide basic user information, the rest, like creating objectclasses or produced attributes is handled by the underlying function interface).

Providing an inteligent backend/library in order for directory administrators to easily produce an abstract operation web services interface to their existing directory infrastructure for outside services is what i would like to have available in the end.

About Me

Kostas Kalevras

LinkedIn profile

E-mail:kkalev AT gmail DOT com
My status

More about me...

a

Follow

Get every new post delivered to your Inbox.