My Case for DTO’s

In many of my posts about Grails and Flex integration, I take for granted that I use Data Transfer Objects to transfer data between my Grails backend and my Flex frontend. Put simply, Data Transfer Object are pure data containing classes different from the domain entity classes used to store data in the backend. I take it for granted because I’m deeply convinced that it’s the best way to do things and so far, experience has never proved me wrong. But I often get this question in comments or by mail (this is for you Martijn): why bother create an entirely separate class structure and copy data from entities to DTO’s and back instead of just using entities?

I’ve expressed my arguments a couple of times across various posts but I thought it would be nice to sum things up in here for future reference.

Where does it come from?

When I first started to work on enterprise applications and ORM-based architectures, it was with a Model-Driven Architecture framework called AndroMDA. AndroMDA was absolutely key in helping me getting started with Spring and Hibernate and I was especially inspired by one paragraph in their “getting started” tutorial, which I quote here:

Data Propagation Between Layers

In addition to the concepts discussed previously, it is important to understand how data propagates between various layers of an application. Follow along the diagram above as we start from the bottom up.

As you know, relational databases store data as records in tables. The data access layer fetches these records from the database and transforms them into objects that represent entities in the business domain. Hence, these objects are called business entities.

Going one level up, the data access layer passes the entities to the business layer where business logic is performed.

The last thing to discuss is the propagation of data between the business layer and the presentation layer, for which there are two schools of thought. Some people recommend that the presentation layer should be given direct access to business entities. Others recommend just the opposite, i.e. business entities should be off limits to the presentation layer and that the business layer should package necessary information into so-called “value objects” and transfer these value objects to the presentation layer. Let’s look at the pros and cons of these two approaches.

The first approach (entities only, no value objects) is simpler to implement. You do not have to create value objects or write any code to transfer information between entities and value objects. In fact, this approach will probably work well for simple, small applications where the the presentation layer and the service layer run on the same machine. However, this approach does not scale well for larger and more complex applications. Here’s why:

  • Business logic is no longer contained in the business layer. It is tempting to freely manipulate entities in the presentation layer and thus spread the business logic in multiple places — definitely a maintenance nightmare. In case there are multiple front-ends to a service, business logic must be duplicated in all these front-ends. In addition, there is no protection against the presentation layer corrupting the entities – intentionally or unintentionally!
  • When the presentation layer is running on a different machine (as in the case of a rich client), it is very inefficient to serialize a whole network of entities and send it across the wire. Take the example of showing a list of orders to the user. In this scenario, you really don’t need to transfer the gory details of every order to the client application. All you need is perhaps the order number, order date and total amount for each order. If the user later wishes to see the details of a specific order, you can always serialize that entire order and send it across the wire.
  • Passing real entities to the client may pose a security risk. Do you want the client application to have access to the salary information inside the Employee object or your profit margins inside the Order object?

Value objects provide a solution for all these problems. Yes, they require you to write a little extra code; but in return, you get a bullet-proof business layer that communicates efficiently with the presentation layer. You can think of a value object as a controlled view into one or more entities relevant to your client application. Note that AndroMDA provides some basic support for translation between entities and value objects, as you will see in the tutorial.

Because of this paragraph, I started writing all my business services with only data transfer objects (what they call “value objects”) as input and output. And it worked great. Yes it did require a little bit of coding, especially as I had not discovered Groovy yet, but it was worth the time, for all the following reasons.

The conceptual argument: presentation/storage impedance mismatch

Object-relational mapping is what Joel Spolsky calls a “Leaky Abstraction“. It’s supposed to hide away the fact that your business entities are in fact stored in a relational database, but it forces you to do all sorts of choices because of that very fact. You have to save data in a certain order in order not to break certain integrity constraints, certain patterns are to be avoided for better query performance, and so on and so forth. So whether we like it or not, our domain model is filled with “relational choices”.

Now the way data is presented involves a whole different set of constraints. Data is very often presented in a master/detail format, which means you first display a list of items, with only a few fields for each item, and possible some of those fields are calculated based on data that is stored in the database. For example, you may store a country code in your database, but you will display the full country name in the list. And then when the user double-clicks an item, he can see all the fields for that item. This pattern is totally different from how you actually store the data.

So even though some of the fields in your DTO’s will be mere copies of their counterparts in the entity, that’s only true for simple String-typed fields. As soon as you start dealing with dates, formatted floats or enum codes, there is some transformation involved, and doing all that transformation on the client-side is not always the best option, especially when you have several user interfaces on top of your backend (a Flex app and an iPhone app for example), in which case you’re better off doing most of these transformations on the server.

In anyway, if you change the way you store data, it should not influence too much the way you present the same data, and vice-versa. This decoupling is very important for me.

The bandwidth argument: load just the data you need

In the master/data use case, when you display the list of items, you just need a subset of the fields from your entities, not all of them. And even though you’re using Hibernate on the backend with lazy-loading enabled, fields are still initialized and transferred over the wire. So if you use entity classes for data transfer, you will end up transferring a whole bunch of data that may never be used. Now it might not be very important for hundreds of records, but it starts being a problem with thousands of records, especially when there is some parsing involved. The less data you transfer the better.

The security argument: show only the data you want to show

Let’s say you’re displaying a list of users, and in the database, each user has a credit card number. Now of course when you display a list of users, you might not want everyone to see the list of credit card numbers. You might want to expose this data only in detail view for certain users with certain privileges. DTO’s allow you to tailor your API to expose just the data you need.

The error-prone argument: argh! Yet another LazyInitializationException!

Of course there are associations between your business entities, and by default, those associations are lazy-loaded, which means they are not initialized until you actually query them. So if you just load a bunch of instances from your entity manager and send them over to your client, the client might end up with null collections. Now of course you can always pay attention, or use some tricks to initialize associations up to a certain level before you send your data, but this process is not automatic and it’s very error-prone. As for using things like dpHibernate, I think it just adds too much complexity and uncontrolled server requests.

The laziness argument: Come on! It’s not that hard!

I think that most of the time, the real reason why people don’t want to use DTO’s is because they’re lazy. Creating new classes, maintaining code that does “almost” the same as existing code, adding some code to service implementation to copy data back and forth, all of that takes time and effort. But laziness has never been a good reason for ditching a design pattern altogether. Yes, sometimes, best practices force us to do more stuff for the sake of maintainability and robustness of our code, and for me the solution is certainly not to shortcut the whole practice, but just to find the best tools to minimize the added work. With its property support and collection closures, Groovy makes both creating, maintaining and feeding DTO’s as simple and fast as it can be. AndroMDA had converters. There are even some DTO-mapping frameworks like Dozer to help you. No excuse for laziness.

For me, all the reasons above largely overcome the added work to maintain a parallel DTO structure.

Now of course, this is a very opinionated topic and you will probably have a different view. So all your comments are welcome as long as they remain constructive and argumented.

Text, Expressivity and Culture-Oriented Programming

Following up on my reflexion about what could software development look like a few years or decades from now, there is this big problem that has been bugging me for years now and that I have never found the time to really tackle: expressivity. In the same way as files appear to me as the biggest obstacle to collaboration, I think the main barrier in the way of expressivity is TEXT.

It’s hard to admit, but we’re still building software like cavemen. We don’t have spoken language, just a bunch of noises, we don’t conceptualize much but we do communicate with a few gestures and more importantly some colored drawings on cave walls. The way I see it, we are not much more advanced than that, but it’s normal, software is still relatively young as a discipline and although it has already changed our lives, we have to imagine that it’s just the beginning. And the good news is that we are headed in the right direction.

man

We started off with most elementary way of storing information and communicating with a machine: zeros and ones. Binary. It was too elementary, more like noises coming out of our mouths, so we started to group bits in octets corresponding to hardware instructions and characters. In fact, we added gestures to noises. Then we grouped instructions into statements and procedures, and we designed a way to translate those into the most elementary form of language that machines could understand. We started drawing on walls. But as procedures multiplied like crazy, we needed to conceptualize some more, talking about classes, objects, methods, properties, and so on. Spoken language was born. And with higher level concepts like services, components and multiple programming languages, we added written language. OK, the analogy is not that good, but you get my point: I’m convinced we’re still very early in the overall evolution of communication with machines, and although this evolution is somewhat slow and creates a lot of inertia, I believe that if we want computers to really expand our capabilities (note that I didn’t say “replace us”), we need to go further in abstraction levels.

So what’s next? Binary, assembly, procedural, object-oriented (yes, and functional, if you want), then what? Model-driven? I’ve tried that, it’s just replacing the constraints of text with the constraints of visual representations. It sure makes it easier to conceptualize, but at some point we’re still translating those visual models into text code, which we have to compile. The roundtrip is just too long. What about domain-specific languages? Well, I’m more into that right now. It looks like communicating is naturally based on languages, collections of concepts that relate with one another to describe what a software is and what it does. So focusing on making it easier to define new languages definitely goes in the right direction. That’s why it’s so linked with meta-programming: instead of statically defining layers upon layers of fixed concepts to describe systems with even higher level abstractions, let’s define the root concepts we will use to describe languages that will allow us to describe our systems. That’s why I’m so interested in Groovy at the moment for internal DSLs, although I prefer the more elegant idea of external DSLs and language workbenches, like Jetbrains MPS does.

All of this evolution makes me think of a video I saw recently, that tried to make String theory more accessible:

Let’s say binary is our dimension 0. Assembly is dimension 1: a line. Procedural programming is dimension 2: a plane. Object-Oriented Programming is then 3rd dimension: a volume. And it’s very hard for us to leave it, as it is our most natural way of seeing things. But that’s where I find this explanation of String theory particulary interesting (although not rigorous as some math geek friends of mine pointed out): there starts a cycle. Dimension 4 is a line again, formed by 2 different Object-Oriented Languages, like Java and C++ for example. Still there? Good! That’s where the fun starts: dimension 5 is a plane composed by all the parallel universes that are created by our own choices. Functional programming and OOP can be considered as forming such a plan. Now what if we could design a way to go directly from one of those paradigms to another one, to fold the plane in the 6th dimension: please welcome language-oriented (or meta-) programming! See the cycle? Now most of us are stuck in the 3rd dimension, and some of us are already experiencing the 6th dimension.

So there we are. Seventh dimension is the line joining the set of all possible timelines starting from our software big bang, which is the binary transistor, to another set of possible timelines, starting from another big bang. Quantum computing can be another option, but it’s a hardware one. What about software? Isn’t virtualization a way to forget about the physical hardware? And there we go 8th dimension: going from binary transistors to quantum computing is one line in the seventh dimension. Choosing to go to virtualization instead creates an branching line in the 8th dimension. Which means that if we want to create our ninth dimension, we need to fold the eighth in order to jump from quantum to virtual computing. And that’s where I locate what I call Culture-Oriented Programming. The third stage of the third 3-stage cycle. The final frontier? The next step? Hooooo… my head hurts.

But wait a second, I only talked about computing here. A virtual reality. What if dimension 10 was the line uniting computing with the real world in the purest possible way. Direct communication between human beings and computers. Who said “scary”?

PS: I didn’t intend this post to be so “theoretical”. I only thought with my keyboard and let my imagination go. But I’d love to know what you think about that crazy analogy? Do you think we are limited to 11 dimensions like in string theory?

PPS: That might be my geekiest post EVER!

AndroMDA, an Open Source MDA toolkit

I gave a quickie at Javapolis today about AndroMDA. It was a first for me, and boy it was hard! My goal with this presentation was to act as a counterbalance to last year’s scripting language hype, to show that yes, you CAN be productive with traditional Java frameworks. It’s just a matter of tools and methodology.

And because I wanted to demonstrate that, I actually challenged myself into developing a full-blown JSF/Spring/Hibernate application in front of the audience in just 15 minutes. And I must admit that I had my eyes bigger than my stomach on this one. I’ve repeated my demo something like ten times, and I only managed to do it once in less than 15 minutes. So that one was risky. And as a matter of fact, I made just a small mistake in my modeling that actually led me to fail showing my running application.

But just for the sake of precision, and in case you need to see it working, I’ve actually attached the final application to this post: TodoList sample application. And here is what you need to do to build it:

  • You have to install Java 5, Maven 2.0.7 (I’ve not tested it with Maven 2.0.8), MySQL 5, JBoss 4.0.5 GA
  • Download mysql-connector-java-bin-5.1.5.jar and copy it to server/default/lib subdirectory of your JBoss installation
  • Set JAVA_HOME, JBOSS_HOME and M2_HOME environment variables accordingly
  • Create a database called ‘todolist’, and allow a user named ‘todolist’ with a password as ‘todolist’ to do everything on this database
  • Run the following commands in the unzipped directory:
mvn -f core/pom.xml andromdapp:schema -Dtasks=create
mvn install -Ddeploy

Please let me know if you have any issue or question related to this sample project. Comments are here for that.

And I’ll probably use this sample application as a starting point for a more complete learning thread so stay tuned.

And by the way, if you need more information about AndroMDA itself, please refer to their official website. The AndroMDA forum is very active as well, and if you can’t wait to get started, you will definitely love the official Getting Started Guide, which exists in Java and .Net versions.