1. ORM: a primer
The basic building block of any application written in an OO language such as Java is the object. As such, the application is basically a more or less large collection of interacting objects. This paradigm works relatively well up to a point. It is when such an application is required to deal with something with a completely different worldview, such as a database, that the brown matter definitely hits the revolving propeller-shaped implement. The term Object-Relational impedance mismatch term was coined to represent this difference in worldviews.
The basic purpose of ORM is to allow an application written in an object oriented language to deal with the information it manipulates in terms of objects, rather than in terms of database-specific concepts such as rows, columns and tables. In the Java world, ORM's first appearance was under the form of entity beans.
There are some problems with entity beans:
They are J2EE constructs: as such, they cannot be used in a J2SE application
The fact that they require the implementation of specific interfaces (and life-cycle methods) pollutes the domain model that you are trying to build
They had serious shortcomings in terms of what could be achieved with them (the whole Fast Lane Reader (anti-)pattern issue and others like it)
The first problem does not kill you, but it also does not make you stronger. In fact, the dependency on a container implies that proper unit testing of entity beans is convoluted and difficult. The second problem is where the real pain lies: the programming model and the sheer number of moving parts will make sure that building a moderately complex, working domain model expressed as entity beans becomes a frustrating and tortuous exercise.
Enter transparent persistence: this is an approach to object persistence that asserts that designers and developers should never have to use anything other than POJOs (Plain Old Java Objects), freeing you from the obligation to implement life-cycle methods. The most common frameworks that claim to provide transparent persistence for Java objects today are JDO, Hibernate and TopLink. At this point, I'd like to clarify that I am not about to discuss the great
JDO vs EJBernate 3.0 religious wars, so, don't even think about it.
Hibernate and TopLink are reflection-based frameworks, which basically means that they use reflection to create objects and to access their attributes. JDO on the other hand is a bytecode instrumentation-based framework. While this difference might not seem to be immediately relevant to you, please bear with me: its significance will become apparent in due course.
2. ORM: the 50000 feet view
At a high level, you need to perform the following tasks when using an ORM framework:
Design and code your domain model (as in, the POJO, java bean-like classes that represent the data that your application requires)
Derive your database schema from the domain model (I can hear the protests: again, please bear with me)
Create the metadata describing how the object map to the database and what their relationships are
Assuming that you have sufficiently detailed requirements and use cases, the first step is a well-understood problem with widely accepted techniques available for its solution. As such, we'll consider the first step as a given and not dwell on it.
The second step is more controversial. The easy way to do it is to create a database schema that mimics the domain model: each class maps to its own table, each class attribute maps to a column in the given table, relationships are represented as foreign keys. The problem with this is that the database's performance is highly dependant on how "good" the schema is, and this "straight" way of creating one generates, shall we say, sub-optimal solutions. If you add to that the fact that you will be constrained (by the very nature of the ORM framework that you are using) in terms of the database optimisation techniques that you can use, and that that one-class-one-table approach will tend to generate a disproportionately large number of tables, you realise pretty soon that the schema that you have is, by DBA standards, a nightmare.
The only way to solve this conundrum is to compromise. From both ends of the spectrum. Therefore, using an ORM tool does not really gel with waterfall development, for you'll need to continually revisit your domain model and your database schema. If you're doing it right, changes at the database schema level will only imply changes at the metadata level (more on this later). Obviously, and by the same token, changes at the domain model level will should only imply changes in the metadata and application code, but not on the database (at least not significant changes).
Creating the metadata for mapping your domain model to the database is where it gets interesting. At a high level, the basic construct available to you is something called a mapping. Depending on which framework you use, you might have different types available to you doing all kinds of interesting stuff, but there is a set that is commonly available:
Direct to field
A direct to field mapping is the basic type of mapping that you use when you want to map a class attribute of some basic type such as string directly onto a VARCHAR column. A relationship mapping is the one that you use when you have an attribute of a class that holds a reference to an instance of some other class in your domain model. The most common types of relationship mappings are "one to one", "one to many" or "many to many".
At this juncture, we need an example to illustrate the use of these mappings. Let us consider the domain model for accounts in a bank; you'll need:
A Bank class
An Account class
A Person class
A Transaction class
The relationships between them are as follows:
Bank has a "one to many" relationship with Account, meaning that a bank holds a lot of accounts, but that the accounts can only belong to one bank. This translates into the Bank class having an attribute of type List holding references to its accounts and that the Account class has an attribute of type Bank holding a reference to the owning Bank instance (this reference is commonly called the "back link" in ORM-speak, because it is used to generate the SQL that will populate the list on Bank: something like SELECT * FROM ACCOUNT WHERE BACK_LINK_TO_BANK_INSTANCE = BANK_INSTANCE_PRIMARY_KEY)
Account has a "many to many" relationship with Person, meaning that an account belongs to one or more persons and that a person may have one or more accounts. In code terms this translates into Account having an attribute of type List holding references to instances of Person and Person having an attribute of type List holding references to instances of Account. It should be noted that this relationship has database schema side effects: it normally requires the creation of a relation table that holds the primary keys of Account and Person objects.
Account has a "one to many" relationship with Transaction (see the relationship between Bank and Account)
Transaction has a "one to one" relationship with Account, meaning that a transaction is to be executed against a single account (this is a simplification for the sake of this example). In code terms, this means that Transaction has an attribute of type Account that holds the reference to the Account instance it is to be performed against.
Please note that the terminology that I am about to use is somewhat TopLink-biased, but you should be able to find out the appropriate Hibernate or JDO equivalent without too much trouble. Anyway, once you have figured out the relationships between classes in your domain model, you need to create the metadata to represent them. Prior to Tiger (JDK 5.0), this was typically done via a text file containing something vaguely XML-like describing the mappings (and a lot of other stuff). If you are lucky, you'll have access to a piece of software that facilitates the creation of the metadata file (TopLink provides you with something called the Mapping Workbench, and I understand that there are Eclipse plug-ins for Hibernate).
With TopLink and Hibernate, once you have the metadata file you are away. If you are using JDO, there is an extra step required, which is to instrument your class files (remember that JDO uses bytecode instrumentation), but this is relatively painless, since most JDO implementations provide you with Ant tasks that automate it for you.
3. IRL ORM
What follows is an account of my experience (and exploits) with TopLink. Some of the issues encountered will be, as such, somewhat specific, but I think that most of them are generic enough to hold for most ORM frameworks.
The first problem that you face is documentation. It is not very good, ambiguous, and only covers the basics of the framework and its use. Obviously, this problem can be solved by getting an expert from Oracle (they own TopLink): I guess that that sort of explains why the documentation isn't very good.
The second problem that you face is that if you are doing something real (as in, not just playing around with the tool, but actually building a system with it), you typically have more than one person creating mappings. You would have thought that Oracle would have considered that when creating the Mapping Workbench. They did not. It is designed to be used by one person at a time, and there's no chance that you can use it in a collaborative development environment. Additionally, it represents the mapping project (the Mapping Workbench name for the internal representation of your mapping data) in such a huge collection of files that storing them in a VCS is an exercise in futility. So, mapping your domain model becomes a project bottleneck: only one person at a time can edit the project, after all. As such, the turnaround time for model changes and updates impinges quite a lot on the development teams, since they can play around with the domain model in memory, but they can't actually test their functionality by trying to store stuff to the database.
When you finally get a metadata file that holds what you need to move your development forward, and you run your code, you start receiving angry e-mails from the project DBA, reading something like "What in the name of all that is holy you think you are doing to my database server?"
At first, you don't know what he is talking about: you are only reading a couple of instances from the database, updating something and saving it. You decide to look at the SQL logs, thinking that the guy must be having problems at home or something. That is when you receive the shock of a lifetime: when you read those two instances, the ORM framework is actually reading half of the database into memory, giving the database server the machine equivalent of grand mal in the process.
After some digging around, you realise that that happens because the ORM framework will not, by default, lazy load relationships. This means that if the metadata contains an association between two classes, and you read in one instance of the association's source class, the framework will read in the target of that association as well so as to resolve it. It will then happily proceed to apply the same principle to the class that it just read in (and which you did not ask for, since you weren't even planning to de-reference that association in your code), ad nauseam.
It's "back to the mappings" time, at this point. The ORM framework offers lazy loading, but it can't be blanket-turned on. It needs to be switched on, one association mapping at a time. You remember what I said about the creation of the mappings being a bottleneck? It just got worse: you are now receiving death threats from the people that need to go through your entire domain model and switch it on. You only ask for a couple of them to be turned on, but the mappings people decide to be proactive, and turn it on for every single association that they can find: they don't want to have to go through this stuff again, so, they might as well bite the bullet right here and now and get it over with.
Eventually, you get the new metadata, you run your code, and the net result is that e-mail tone from the DBA goes from angry to downright shitty. There's no pleasing some people, you think while you scan the SQL logs. That's when it hits you. When you are finally able to breathe again, you seriously consider suicide. Since the mappings team decided to be proactive (read: overzealous), all the mappings are now lazily loaded, and the problem is now that every time you (or somebody else, for that matter) follow an object reference or cycle through a list, there's a database hit.
You guessed it: "back to the mappings" time. This time, though, in order to make a call on whether a relationship should be lazily loaded, you need to trawl through all the use cases, involve a bunch of people, and come up with the most likely usage and access scenarios for each of the classes in your domain model. That takes a lot of time and money and the PMO is not amused. Eventually, the problem is sorted to the point that the abuse that you get from the DBA and the mappings team recedes to background-noise level (the occasional "arsehole" muttered as you walk past one of them in the corridor).
This is when the people managing the servers come to you with a purchase order for enough memory to comfortably run SkyNet in. Trying not to let the tick in you left eye show, you politely ask what this is about. For your sins, they tell you. They have been monitoring the performance and memory usage, and they figure from their projections that that is what you need to see you through the next 6 months. You reply that you need to do some due diligence, and that you'll get back to them. After some investigation, you realise that they are right. The problem is that reflection-based ORM frameworks figure out what needs to be flushed to the database (as in, what you created or updated, and what SQL needs to be generated) by comparing a reference copy that they keep against the instance that you modified. As such, and at the best of times, you are looking at having twice as many instances of a class in memory as you think you should have. The problem gets further compounded by the fact that you are using a multithreaded architecture, so, it is a case of each of the threads holding at least twice as much memory as the code running in them directly manipulates.
(Side note 1: It's actually worse than that as far as TopLink is concerned. It actually holds 2 reference copies.)
(Side note 2: JDO is much more efficient in this respect. When they instrument the classes, they add the equivalent of boolean "dirty" attributes for each attribute that you originally had. So, even though the actual class that is being run is somewhat fatter than the original, it goes nowhere near having multiple copies of it in memory)
At a high level, the only way that you can get around this is to actually figure out which classes are read-only from the application standpoint. TopLink will stop its copying process at the point that it finds a reference to a class that is thus marked, which should save you a considerable amount of memory. This basically amounts to identifying which classes are reference data, and which are routinely modified by the application (leaving aside the issue that you need your application to be able to modify its reference data in some way).
You guessed it: meetings with everybody and their dog first, "back to mappings" time later.
And since nothing comes for free, the downside of marking some classes read-only is that, once they get into the ORM framework's cache, they won't be read from the database again.
Well, you say, that is got to be good for performance, right?
Right. But did I mention that we run in a clustered environment (two boxes, two application server instances, live-live)? And that reference data is sometimes modified? If you put all that together, you get some quite dysfunctional scenarios whereby one application server instance's view of the world is quite different from the other, where reference data is concerned. And none of it is predictable, it depends on whether one of them had already read that particular piece of data and the other had not, which processed the reference data change, and whether that night is going to be a full moon.
Oh. Did I mention that the Operations Support team have been on your back for a number of weeks now, basically saying with increasing levels of venom that your application is, from their point of view, unsupportable? Well, they kind of have a point: they are used to fixing problems by firing up SQL*Plus and tweaking a couple of database tables. They just can't do it now: the schema is not human-readable, for it is a reflection of the domain model and the ORM framework's requirements. Adding insult to injury, surrogate keys are used consistently everywhere. This means that every primary (and, by extension, foreign) key is actually a very long number, meaning nothing to anybody unless they know which domain class was being manipulated at the time that the problem occurred. The final nail in your coffin is the fact that, since your domain model uses inheritance liberally (as any self-respecting object model geared towards behaviour and code reuse should), you have spurious table with cryptic looking keys all over the shop. Adding to that, they just can't change some reference data value in the database. The application might never know that it had been changed (see above).
At this point, life in the streets having breakfast out of a bottle wrapped in a brown paper bag starts becoming your definition of heaven.
4. Looking to windward
Using ORM is difficult. However attractive the notion of having your application communicate with the database using a paradigm that it is very familiar with (object construction, destruction and modification), there are a number of downsides. In general, these are the things that you consider before embarking on the ORM bandwagon:
It does not make the project cheaper. If you are lucky, it will cost the same as if you implement your code using straight JDBC and something like the DAO pattern. If project price is your selling point for ORM, YFI.
It is not more performing than carefully tuned JDBC. If performance of the solution is your selling point for ORM, as above.
It is not a silver bullet for technical inadequacies in the project team. If you think that you can dumb down the workforce (as in, hire more inexperienced people because, after all, all they need to know is bog standard Java, none of that JDBC and DBMS skills BS), as above.
On the other hand, there are a number of things that you gain from using ORM:
It makes your code simpler to understand (well, assuming that your domain model is worth a damn). Additionally, if you do it right, you write less of it. This means that maintainability of a system created using ORM should be significantly higher than one created using straight JDBC calls.
It gives you a number of technical options that would be quite complex to code from scratch. I am talking about stuff like lazy loading, predictable SQL generation order (important in deadlock avoidance for clustered applications) and query by example.
It gives you database independence. Your application does not even know what the database is; its only connection to it is via the ORM framework and the JDBC driver. This means that your development environment can be a cheaper version of the production environment (as in, using an open source database for development would save you all that Oracle development licenses cash), and that you can confidently upgrade database versions without expecting to have to modify the application at all.
In short: if you have to make a decision either way, carefully examine the problem, and make informed decisions. Simple, really.