Directory Servers vs Relational Databases

An interesting question was posed on LinkedIn that asked, "If you were the architect of LinkedIn, MySpace, Facebook or other social networking sites and wanted to model the relationships amongst users and had to use LDAP, what would the schema look like?"

You can find the original post and responses at http://www.linkedin.com/answers/technology/software-development/TCH_SFT/296425-23753864

After reading the responses from other LinkedIn members, I felt compelled to add my proverbial $.02.

---------------------

Directory Servers are simply special purpose data repositories. They are great for some applications and not so great for others. You can extend the schema and create a tree structure to model just about any kind of data for any type of application. But just because you "can" do something does not mean that you "should" do it.

The question becomes should you used a directory server or should you use a relational database. For some applications a directory server would be a definite WRONG choice, for others it is clearly the RIGHT one, for yet others, the choice is not so clear. So how do you decide?

Here are some simply rules of thumb that I have found work for me:

1) How often does your data change?

Keep in mind that directory servers are optimized for reads - this oftentimes comes at the expense of write operations. The reason is that directory servers typically implement extensive indexes that are tied to schema attributes (which by the way are tied to the application fields). So the question becomes, how often do these attributes change? If they do so often, then a directory server may not be the best choice (as you would be constantly rebuilding the indexes). If, however, they are relatively static, then a directory server would be a great choice.

2) What type of data are you trying to model?

If your data can be described in an attribute:value pair (i.e., name:Bill Nelson), then a directory server would be a good choice. If, however, your data is not so discrete, then a directory server should not be used. For instance, uploads to YouTube should NOT be kept in a directory server. User profiles in LinkedIn, however, would be.

3) Can your data be modeled in a hierarchical (tree-like) structure?

Directory servers implement a hierarchical structure for data modeling (similar to a file system layout). A benefit of a directory server is the ability to apply access control at a particular point in the tree and have that apply to all child elements in the tree structure. Additionally, you can start searching at a lower (child element) and increase your search performance times (much like selecting the proper starting point for the Unix "find" command). Relational databases cannot do this - you have to search all entries in the table. If your data lends to a hierarchical structure then a directory server might be a good choice.

I am a big fan on directory servers and have architected/implemented projects that sit 100% on top of a directory, 100% on top of relational databases, and a hybrid of both. Directory servers are extremely fast, flexible, scalable, and are able to handle the type of traffic you see on the Internet very well. Their ability to implement chaining, referrals, web services, and a flexible data modeling structure make them a very nice choice to use as a data repository to many applications, but I would not always lead with a directory server for every application.
So how do you decide which is best? It all comes down to the application, itself, and the way you want to access your data.

A site like LinkedIn might actually be modeled pretty well with a directory server as quite a bit of the content is actually static, lends well to an attribute:value pair, and can easily be modeled in a heirarchical structure. The user profiles for a site like facebook or YouTube could easily be modeled in a directory server, but I would NOT attempt to reference the YouTube or facebook uploads or the "what are you working on now" status with a directory server as it is constantly changing.

If you do decide to use a directory server, here are the general steps you should consider for development (your mileage may vary, but probably not too much).

1. Evaluate the data fields that you want to access from your application
2. Map the fields to existing directory server schema (extend if necessary)
3. Build a heirarchical structure to model your data as appropriate (this is called the directory information tree, or DIT)
4. Architect a directory solution based on where your applications reside thorughout the world (do you need one, two, or multiple directories?) and then determine how you want your data to flow through the system (chaining, referrals, replication)
5. Implement the appropriate access control for attributes or the DIT in general
6. Implement an effective indexing strategy to increase performance
7. Test, test, test

Submitted By: Bill Nelson (bill.nelson@gca.net)

Lessons Learned from Enterprise Identity Management Projects

I have been implementing and/or managing identity-related projects for over ten years now and I can say from experience that the biggest problem with any identity management project can be summed up in one word - EXPECTATIONS.

It does not matter whether you are tackling an identity project for compliance, security, or cost-reduction reasons you need to have proper expectations of what can be realistically accomplished within a reasonable timeframe and those expectations need to be shared among all team members and stakeholders.

Projects that fail to achieve a customer's expectations do so because those expectations were either not validated or were not shared between all parties involved. When expectations are set (typically in a statement of work), communicated (periodic reports), and then reset if necessary (change orders), then the customer is much happier with the project results.

Here are a few lessons I have learned over the years. While they have general applicability to major projects, in general, they are especially true of identity-related projects.

1) Projects MUST be implemented in bite-sized chunks.

Identity projects are enterprise-wide projects; you should create an project roadmap that consists of multiple "mini" projects that can demonstrate an immediate ROI. The joke is, "How do you eat an elephant? One bite at a time." To achieve success with identity projects, you should implement them one bite at a time and have demonstrable/measurable success after each bite.

2) The devil is in the data.

Using development/test data that is not representative of production data will kill you in the end and cause undue rework when going into production. Use data that is as close to production as possible.

3) Start with an analysis phase BEFORE scoping the entire project.

I HIGHLY recommend that the first project you undertake is an analysis. That will define the scope for which you can then get a better idea of how to divvy up the project into multiple bite size chunks and then determine how much (and how long) each chunk will take. This allows you to effectively budget both time and money for the project(s).

Note: If a vendor gives you a price for an identity implementation without this, then run the other way. They are trying to simply get their foot in the door without first understanding your environment. If they say that the analysis phase is part of the project pricing, then get ready for an extensive barrage of change orders to the project.

4) Get everyone involved.

Keep in mind that these are enterprise-wide projects that affect multiple business units within your company. The project team should contain representatives from each organization that is being "touched" by the solution. This includes HR, IT, Help Desk, Training, and above all, upper-level management (C-level).

(The following items apply if you are using external resources for project implementation.)

5) Find someone who has "been there and done that".

Ask for references and follow up on them. More and more companies say that they can implement identity-related projects just because they have taken the latest course from the vendor. This is not enough; if training alone could give you the skills to implement the product, then you would have done the project yourself. You need to find someone who knows where the pitfalls are before you hit them.

6) Let the experts lead.

Don't try to manage an identity management project unless you have done so before - more than once. I have been involved with customers who have great project managers that have no experience with identity projects - yet they want to take ownership of the project and manage the resources. This is a recipe for disaster. Let the people who have done the implementation lead the project and allow your project manager to gain the knowledge for future phases.

7) Help build the car, don't just take the keys.

Training takes place before, after, and during the project. Don't expect to simply take "the keys" from the vendor once the project has been completed. You need to have resources actively involved throughout the project in order to take ownership. Otherwise you not be able to support the product (or make changes to it) without assistance from the vendor. Ensure that you have your own team members actively engaged in the project - side by side with the external team. To do this, you have to ensure that they are not distracted by other work-related tasks.

Submitted By: Bill Nelson