 |
 |
 |
|
|
|
My university has hitherto provided mail directory services via CCSO (the 'qi' server and 'ph' client), and in various text forms for use by the Simeon mailer and on our Web server. We have about 10000 entries giving name, mail alias, student/staff status and department; information derived from administration files. All forms of the database were rebuilt from scratch each night, taking about 5 minutes on a Sun 690/MP612. When we upgraded our mail server (Sun Sparc 10 to Ultra 1) I felt I had the capacity to take this on a bit. My objectives were:
I did not feel able to set up a full X.500 DSA, for these perceived reasons:
The SLAPD LDAP server produced by Michigan University seemed to avoid all those problems, and meets the last two of my four requirements. Access to an X.500 directory was also possible with the assistance of an LDAP->X.500 converter. That service was offered by Ronan Flood of ULCC, who manages the UK portion of the DIT. Vincent Berkhout of Dante told me about the same time that Critical Angle, a US company, were testing an X.500->LDAP converter (the PARADISE X.500 Enabler), and this seemed to fill in the missing element. It is only available for a small range of UNIX versions, but as Solaris 2.5 was one of them this didn't cause us a problem. We placed an order for a copy of the restricted "read-only" version, which only costs $95.
For anyone unfamiliar with it LDAP is a subset of the X.500 protocol designed for PCs which only have access to TCP/IP, or prefer not to run a full OSI stack because of the overheads. It uses the same external data representation (key/value pairs), and the same database schema, but does not transform the data internally into ASN.1. It also runs directly over TCP/IP, rather than over an RFC1006 or full OSI stack.
Initially I obtained a copy of the Michigan SLAPD package (version 3.3). For optimum performance this requires the Berkeley DB package, which has to be compiled with care to ensure that it is thread-safe (the server is multi-threaded rather than multi-processing). The server can then be built along with a good range of database maintenance and query tools, including a bulk loader.
There is an active mailing list for users of the package, to which many of the developers of all the packages discussed here contribute.
Documentation for the package is good. This was a relief after setting up a CCSO system a few years earlier. Database records consist of key-value pairs. The contents of each record type and the data type of each value are defined in configuration files, but checking of input records for compliance with the schema is optional. Values can be binary data as well as text, input as hexadecimal or via files as pure binary. Attribute names are forced to lower-case whenever used.
Structuring your data is fairly simple, as there are a number of standard record types defined in the OID table files, for which samples are supplied. Unless these really are inadequate for your requirements I recommend sticking to them. If not you may find life gets complicated when you want to export the data to an X.500 DSA or use it in Simeon, both of which hold attributes as ASN.1 OIDs.
You also have the choice whether you enable schema compliance checking. Using the unmodified Michigan server enabling this caused the server to crash, but there is a fix for that. If you turn it off, it makes it easy to put any data you like in the database, but you risk not only the problems mentioned above in exporting to an X.500 DSA, but also the possibility that you will unknowingly cause inconsistencies or missing data which will prevent your server running correctly. When I enabled it found I had no 'organization' record!
Deciding what to use as your DN and RDN are both tricky. The DN must be unique, so in most cases "common name, organisation name" will not do. In theory RDNs can be multi-valued, in the format "common-name + unit-name, organisation-name". Unfortunately commonly-used query tools, such as 'web500gw' described below, do not support them, and even if your tools do, if you are linked to the international X.500 DIT someone will encounter the problem.
The only practical solution, and the one we adopted, is to use the common name as the RDN, with the DN in the format "common-name, unit-name, organisation-name". The only problem with this is that if someone changes unit then their record has to be deleted and reinserted with the new DN. To accommodate people adding attributes to their records the one to be changed has to be read, deleted, the saved copy modified with the new unit name and then added. This considerably complicates bulk additions.
The choice of indexes is not so difficult, though perhaps not obvious. The main point is that each index type corresponds to a search operator, and that if a search is done using an operator for which the attribute in question does not have an index, all records matching other criteria in the search will be examined individually. This can be very costly. Thus names fields must have substring and approximate indexes, despite the update time penalties mentioned in the documentation, because those sort of searches will certainly be done on them by someone.
Presence indexes are only required to locate all entries having a particular attribute, whatever its value. Indexing attributes for equality only is most useful where values are enumerated rather than arbitrary, as with 'objectclass' and 'organizationalStatus'.
In reality the costs in space and update time for a set of indexes able to cope with anything in a simple mail directory are quite limited. You can see what indexes we currently use in this copy of our configuration file.
Building a database from scratch takes time. On our Ultra 1 (then 128Mb) it required about 4 hours to create all the files for our 10000 entries, though from messages on the mailing list it is clear it can be reduced greatly if the bulk loader is allowed enough memory. Both for that reason and to allow user updates to the database I did all further updating using the incremental modification tools.
The database worked first time, which should be typical. I built the X.500/Web gateway 'web500gw' which comes with SLAPD. This has one undocumented bug, which fortunately was solved by someone on the mailing list, but then worked too (though some reconfiguration is needed to tailor it for your schema). The only disappointment at this stage was that Simeon seemed unable to do searches reliably, and when they worked hardly any attributes were returned. This took a long time to resolve.
After a few days of testing I installed the PARADISE Enabler. This is obtained by download from the company's FTP site; a copy is supplied specific to your server's host ID. Installation is very simple, and the software worked at once. Ronan Flood assisted by testing it via an X.500 client.
There is one small problem in configuring the Enabler. SLAPD can maintain creation and last modification details for all records in the database, which we felt useful, but it stores the information in attributes defined in the 1993 X.500 standard. However the UK master DSA is a QUIPU database, which conforms to the 1988 version of the standard. It needs to contain, as 'glue', the organisation record for each server in its part of the DIT, and fetches them from its subordinates regularly. It does not recognise the 1993 attributes in the SLAPD and therefore ejects the record, breaking the link. This can be fixed either by turning off change recording in SLAPD or by fudging the X.500 object IDs in the Enabler's OID tables to make the LDAP attributes look like 1988 ones (fortunately equivalents exist). That's the course we took, as shown in our OIDTABLE.AT
There is not much more to say about the Enabler. It has never crashed, hung up or given any other problems. The limited space it gets here is a reflection of this, rather than lack of enthusiasm.
Before the UK DIT authorities will link your server into the tree, you need to register your organisation name with DISC, part of BSI. This is quite simple; you just have to get a form from them, fill in the institution address and name to register, send it back with £176.25 and about 2 weeks later the form comes back duly signed. This ensures that the 'organization' value on your records is unique.
Initially I made the database read-only, only holding basic information which was updated nightly using difference files. However I got frequent operational problems. The server crashed from time to time, and being multithreaded that makes it entirely unavailable. This happened particularly during overnight updates. At that time there was no library of known patches or FAQ; everything depended on a few active people on the mailing list. That support was actually very good, but there were clearly some major problems in the code. After a few weeks the server was crashing on almost every nightly update.
At this point I decided to try switching to Netscape's Directory Server (DS), which by this time was a released product, version 1.0. This is available for a limited range of UNIX platforms as well as several Windows NT ones. The software comes precompiled, and is simple to install.
The biggest initial problem is that if, like us, you took advantage of Netscape's generous free license terms for educational organizations, then the only documentation you have is in HTML form. Netscape recommend the use of one of their Web servers to browse it, but we currently use Apache, which works perfectly well. As well as online help, there are also more discursive manual-style documents, and after a bit of hunting I had something I could print out and sit down with.
Also, unless you take out a support contract, the main forum for reporting or discussing problems is a newsgroup on Netscape's secure news server. As data transfer rates to Netscape's home servers are often slow, and I don't generally use Netscape's news client anyway, this is not as convenient as the mailing list run by Michigan University for their server.
Netscape's DS is clearly based on the Michigan server. Most details of configuring and running both server and utilities are the same; the only notable exception being the holding of access control information in the directory, rather than in the configuration file. This, though not compatible with the format of ACL data in an X.500 DSA, serves a very similar purpose and has significant advantages.
ACLs can be applied to objects at any level in a schema (e.g. organisation, department, or person), and govern every object at that level and below, unless countermanded by a more specific ACL. An ACL defines a level of access which is either granted or denied to a client (authenticated as the subject of the record, or as another specific identity, or anonymous) in respect of one or more attributes in the record it applies to.
The Michigan server has a similar protection scheme, but because ACLs are stored in the configuration file, routine application to many database records is impractical. They can be easily used to permit access to classes or records, or for a few special cases. Indeed, my current use of ACLs in Netscape DS could be implemented quite easily on the Michigan server. Your need for the added power and flexibility in DS will depend on what you plan to keep in the database.
However, the scheme used by Netscape can produce unexpected consequences. In some circumstances database utilities will read all the ACL records in the database during startup, and if large numbers of records contain ACL data this can take some time.
If you want a read-only directory, with all updates done by a single process, then setting ACLs is trivial in either case (though note that the Netscape DS defaults to give no-one any access, so a minimal ACL is essential). To allow people to update their own records is fairly easy with either server, though note that Netscape DS allows you to hold the passwords required for authentication to the server in UNIX 'crypt' format (or various stronger encryptions, limited by US export regulations), whereas the Michigan one only allows plaintext. The 'crypt' format permits passwords to be copied from a UNIX password file, if you want to, though neither (for quite sensible reasons) will actually do validation via UNIX system mechanisms like "/etc/passwd" or NIS. Password attribute values should of course be made "compare-only"!
Netscape DS comes with a set of HTML and CGI scripts to implement a Web interface to the directory. Again, it was clear that if I had been using a Netscape Web server getting this running would be easier, but after some careful reading of the installation notes and Apache documentation I was able to get it going. However, if you already have a Web server on the directory server host, and it isn't a Netscape one, you may find it messy and time-consuming moving files about, because theirs evidently allow multiple document roots and CGI script directories.
The interface is different from that supplied with the Michigan server, particularly in that the latter is a standalone server, written in C, that talks to the client in HTML. The Netscape version is a set of HTML and CGI scripts, with a lot of embedded JavaScript. That means you need Netscape 3 to run it (the dialect supported in Netscape 2 isn't enough), whereas the Michigan one is fine with anything that supports forms.
The look and feel of the interface is also different. The Netscape one looks much smarter, with their distinctive graphical style. Modifying and adding records is not advertised as something dangerous. It does assume a bit of knowledge; for example, about authentication and binding to the server, but then so does the Michigan one.
Its big problem is navigation: the Michigan interface makes it easy to go from looking at the departmental level, to the organisation, the country and on to the world. The Netscape server seems designed for people who are unlikely to have a server they can refer queries to, when those concern other organizations or their staff. As far as I was able to find out, such queries could only be input as either full distinguished names, or LDAP search expressions.
One major advantage of the Netscape DS is that it can talk to Simeon as delivered. This is because it understands the long-form aliases for several attribute names which are always held in the database in their short forms. This can also be fixed by modifying Simeon's OIDTABLE.AT file to use default to using the short forms, but it was useful not having to make that change on all our servers.
Although initially the Netscape DS seemed to be much more stable that the Michigan one, as time went on this changed. Although many of the bugs I encountered when I first used SLAPD were fixed, the server started crashing during nightly updates again. A number of people on the mailing list for SLAPD and the Netscape's newsgroup for DS seemed to be getting similar problems, but for some time no answer seemed to be available.
After some time both an answer and a workround appeared. The problem is in the Berkeley DB library, and a fix has been produced which can be applied to the Michigan SLAPD server. However it has taken somewhat longer for Netscape to release 1.01, which uses the fixed library, and I am currently still using the workround. This is quite simple, and just requires creating only the first record in the database with the offline build tool; after that the server should be started and the database populated using the online maintenance tools. The resulting database occupies about 4 times the space (about 100Mb for 10000 entries), but after several weeks of use the server is still running.
There is little noticeable difference in the performance of the two servers. Although I have not yet formally released the LDAP service for internal use, we are getting a bit of external access. The mail server seems quite able to cope with this, and the total time consumed by LDAP server and PARADISE Enabler together are insignificant (about 3.5 minutes CPU per day). The working sets for the two on our Ultra 1 (now 256Mb) are 1.4Mb and 0.9Mb. Update times for the server are quite acceptable; about 2 to 5 seconds per change, depending on load and the extent of the modification.
I have yet to get the CCSO->LDAP gateway going, am undecided about the use of the 'web500gw' interface as against the Netscape native one (either can be used with DS), and have not fully decided on the choice of server either. Things often don't happen as fast with a commercial product as a PD one, and I was a bit frustrated at having to wait for DS 1.01. However, there was a time when I felt the Michigan server lacked an authoritative source of information and a coherent collection of patches. Such an archive does now exist, and anyone using the Michigan server must get familiar with it. Running SLAPD 3.3 out of the box is not a good idea, in my experience, whereas Netscape DS will be OK provided only that you build the database the right way.
Netscape have made it clear that the X.500 directory in its LDAP incarnation is central to their server strategy: it will be used for mail, for service registration and for storage of security certificates, among others. They are innovating with it, as shown by their use of ACLs and their extensions to the standard schema. However I believe what they are doing currently anticipates rather than supplants developments in the standards.
The most distinctive feature of the Netscape server is the administration interface, which runs as a separate process and can talk to (or maybe has an equivalent on) all their other servers too. The interface is again created with HTML, CGI and JavaScript. It would be most attractive to administrators who don't like messing around on the UNIX command line, editing configuration files by hand and so on. Those who are used to that style of interaction will probably find it interesting rather than essential, but like so much of Netscape's software it certainly shows how to produce a good-looking and effective interface with Web tools.
In summary I would say that either of these LDAP servers can offer a realistic alternative to a full X.500 DSA, and makes lower demands on the host. Both implementations have more problems than the majority of PD software, but with the right information these can be dealt with. However, if I were paying for Netscape DS I would probably have expected it to be more reliable out of the box.
M.B.Caines@wlv.ac.uk18 March 1997