"Standards First" Domain Object Design

Introduction

Our development team is starting to build up some domain objects for the various business objects that we will need and there are a variety of approaches that people seem to take as they start this endeavor. Many of the approaches seem hap-hazard and often capricious with respect to consistency, re-usability, and simplicity. So I decided to talk about my thoughts and maybe solidify my own ideas.

What is Standards First Domain Object Design

My tack has been to use a "Standards first" domain object design - meaning first look to see if there is a well recognized standard on how something should be represented before I venture out on my own. So for instance, lets say we need to pass a date out of the API, what should the XML for that date look like? I would ask myself, Are there standard ways to represent date in XML? For this example there is a standard XML date representation. I kind of expected everyone to follow this approach when I first started, but it seems this is not the way many developers operate. There are a lot of reasons to look to industry standards first, but there are also some good reasons to burn your own trail.

Pros and Cons

So what are the advantages of using the standards when possible? Well, for one, it often saves a bunch of time later (often in situations where you wouldn't anticipate it). For instance, In the last project I worked on, I was responsible for formatting the data coming out of the API and I choose to format times using the XML date time standard. I didn't realize at the time that when I went to parse it in our Android application, that android already had an ISO 3339 time/date parser built into the standard libraries - which works with the XML date time format (the XML format is a subset of ISO 3339). So I got automatic time parsing for free by just choosing the right format. I did not know the parser existing when I choose to use standard XML dates with timezones, but it ended up making all the pieces work together much smoother than it would have if I had chosen a "custom" format. These kinds of "happy accidents" seem to follow when standards are used (and it forms the basis for why computer software expands so rapidly - building on things before using components).

Another big advantage to standards first domain objects would be when the time comes to share your API with 3rd parties. The adoption of an API occurs far faster when the formats look like things that developers have worked with before. Not having to explain a format is something that everyone should strive for. Also, the use of the API is better when the developers can use your data using their standard tools (which were developed to standards) with components that are already written.

OK, OK, standards are great blah, blah - why would you choose to NOT use them? The primary disadvantage to them is that the research takes time. It takes a lot of time to look up standards for the data you are trying to use, figure out which one is the "best" or "most common" format. Often there are competing standards, many times the standards bodies want you to pay for the standards documents, sometimes the "standards" are far too complex for the task at hand. I personally like standards that are freely available (like RFCs), because they seem to be more highly vetted and have far more adoption because of the availability. I also like standards that are easy to read - often not needing explanation because they just make sense. So research is necessary and it take a lot of time.

At what point do I decide that I am spending too much time researching and not enough time coding? It is a tough balance that I struggle with, and every developer situation is unique, so I can't generalize (and there is no right way). But there is a point of diminishing returns when it comes to software architecture research. I tend to strive the answer 3 questions when I design my domain objects/API:

1. Would I want to use it?
And by this I mean, if I were writing my own software for my own company, and I had 10 APIs to chose from, would this be the one I would go with?
2. Does it handle my complexity?
Does it handle the nuances that I may not have time to fully flesh out for this project at hand (thus saving me time in the long run). As an example, the time format (ISO 3339) handles time zones and durations and daylight savings time - things that might require me making a few stabs at to get right.
3. Does it give me room to grow?
Is the structure so strictly defined that I can't build on it, or is it componentized enough that I can use the structure as pieces to my architected objects.

There are a lot of considerations to take into account when designing formats for domain objects and this post merely scratches the surface of a crazy complex issue, but standing on the shoulders of those who came before is definitely in your best interest. And a "standards first" domain object design is a great way to crank out a sustainable and flexible design.

Versioning and Types in REST/HTTP API Resources

There are a variety of ways to type and version the data with REST services, many of which are used successfully. Because of the living nature of APIs, changing versions and changing data types can lead to API designer headaches that saturate much of their time. I am going to discuss the ways to avoid some of these problems by configuring things properly up front.

Dealing with Types

Lets look at a common way REST calls are made:

===>
GET /customer/123 HTTP/1.1
Accept: application/xml
<===
HTTP/1.1 200 OK
Content-Type: application/xml

<customer>
  <name>Neil Armstrong</name>
</customer>

Ok, we have an API that returns a customer - looks good. The problem here is that the API does not return a customer - but rather a generic XML document. When the designer made this particular API, they didn't bother to specify the type of document being returned. Sure there is an API document somewhere that defines the customer XML being returned, and as soon as you call it you are going to see that it is a customer - what is the big deal? Well, the problem is that the API is never static. And as far as REST/HTTP is concerned, sticking a product in the response is perfectly valid (but totally wrong in concept). Wouldn't it be nice if the client could actually validate the information coming back? Wouldn't the client be more stable and predictable if we knew the format of the data that was going to come back? Wouldn't it be great if the server knew which format the client wanted and could give the client what they ask for? Sure it would, so lets change the call to make sure we are asking for what we want:

===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer+xml

<customer>
  <name>Neil Armstrong</name>
</customer>

Here I have substituted the generic xml request MIME type with a vendor specific MIME type. Awesome, now we ask for a customer XML formated customer, and we get a customer XML back from the call - which can even be validated on the client! That should solve our data format problem once and for all, right?

Dealing with Versions

BUT WAIT - there is still an issue! Lets say you decide to change the customer XML because you want to add some really cool new stuff to make that million dollar sale. How does that impact your client programs?
Here is the new flow:

===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer+xml

<customer>
  <firstName>Neil</firstName>
  <lastName>Armstrong</lastName>
  <salutation>Mr.</salutation>
</customer>

This change has just broken all the clients using that resource since they can no longer parse the changed XML properly! Further, there is no way for the client to check the returned version without calling it or asking for a specific version of the API return object - it just always gets the latest format. Sure you could say that your API needs to maintain backward compatibility - but that is not very realistic when you are properly reusing your API across your product line. To demonstrate further, lets say you have 30 applications (and maybe a handful of external companies using the API), all of which are relying on the "customer" REST resource - your choices now are:

1. Keep it backward compatible (and lose the million dollar sale because you couldn't implement cool feature X)
2. Change all 30 applications simultaneously to handle the new data (you likely don't have enough resource to do this and deliver on time)
3. Make the change, breaking the apps you don't have time to upgrade, but get the sale. (of course you will fix the remaining apps in the future, right?)

Basically, you lose no matter which choice you make. So what can we do now to avoid problems with newer versions within the API?

Common Solutions to Versioning

Some common ways of handling this problem are to request a specific version in the call URI:

Method 1 (Put API version in the URI):
http://server:port/api/v2/customer/123

Method 2 (Add a request parameter in the URI):
http://server:port/api/customer/123?version=2

Both of these methods are used to request a specific version of the resource we are looking for. And for many situations, these work fine. But we are munging up the resource identification with the resources representation (breaking an essential REST tenant). We don't want the identifier that we use to find an object to be mingled with the format we are requesting (for reasons discussed in my last post)! But we are smart designers, and this is essentially the same problem we had above, so we immediately recognize that we can solve this with the same solution as above - just make the type requested more specific:

Method 3 (version the request type):
===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer-v1+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer-v1+xml

<customer>
  <name>Neil Armstrong</name>
</customer>

Similarly, the newer clients make a different call since they are aware of the new version:

===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer-v2+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer-v2+xml

<customer>
  <firstName<Neil>/firstName>
  <lastName<Armstrong>/lastName>
  <salutation>Mr.</salutation>
</customer>

Great! Now you have added the changes and made the sale and not broken any of the clients - you are the company hero! As an added bonus, clients and servers can work together to keep your API stable going forward. By requesting a specific type and version in the request header, the server can decide if it is capable of fulfilling the request and inform the client appropriately (i.e. return a HTTP 415 if it can't fulfill the request, or a 301 if it is changing). This is a much more stable and polite way of dealing with API changes for the client.

Conclusion

We can see that the versioning and the typing of the data are not independent concepts for REST APIs - they are in fact the same, since changing either one results in different data being returned. And since REST is built on a very well tested technology like HTTP, lets take advantage of the capabilities of that technology. It is a simply matter to type the responses being generated and it adheres to 'proper' calling conventions, simplifies the URI and solves our versioning and typing issues cleanly.

URI design in REST

Background and the Issue

We just began switching our existing architecture over to a proper SOA design (yeah!) and decided on using REST (yeah!) for the "API". One of the first discussions we got into was what the REST URI's will look like in the new design. We all agree that proper URI design is a core factor in determining the success of REST so the debate is lively and its good to see everyone passionate about the technology.

One of the sticking points is how to return different formats of the Restful "objects" - say XML and JSON. Our desires come down to 3 possible designs all of which are considered acceptable from what I have read. So let me present you the design possibilities before I go into the pros and cons of each:

Method 1 - Use the URI to configure the return type

So to get a customer with ID 123 you would call:
http://server:port/api/customer/123.json (for JSON)
http://server:port/api/customer/123.xml (for XML)

Method 2 - Use a query parameter to set the return format

http://server:port/api/customer/123?format=json
http://server:port/api/customer/123?format=xml

Method 3 - Use the HTTP headers to get the objects

http://server:port/api/customer/123
(GET request using an XML or JSON mime-type in the request header)

One of our architects choose the first for the prototyping we were doing and when I saw it, something didn't sit right about it. So I decided to do a little research -- and it turns out that there are a lot of folks who do it this way, including the Twitter REST API, and the seeming "reference" text on the subject (RESTful Web Services by Leonard Richardson & Sam Ruby).

Even though I found a lot of great references praising this method, that nagging sense that I was doing something wrong kept creeping in, so I figured it was worth the time to analyze the situation a bit more. So I came up with some reasons for an against each design to help me decide on a proper way to go:

Analysis of Method 1

The first solution has several advantages and it is certainly a quick solution to multiple formats.
Pros:

1. Easily testable in a browser - you just type in the format you want.
2. Simple for most to understand - because you explicitly ask for the format you need.

Cons:

1. Can lead to ambiguity in the API. For instance, lets say you needed to return data that required multiple input parameters, like some reporting data. You request could be something like this (I put the parameters in []):
   http://server:port/api/reportdata/[Sales]/[January]
   
   But if you need to specify the format, what does the final URL look like:
   http://server:port/api/reportdata/Sales/January.xml
http://server:port/api/reportdata/Sales.xml/January
http://server:port/api/reportdata.xml/Sales/January
   While this is a sort of a minor problem (you could just say "put it at the end"), it does represent something you will have to think about when writing the calls and something you will have to explain whenever a 3rd party uses your API.
   
2. Requires a separate handler for each URI type that you add - meaning its more difficult to write a generic format parser and reuse it on all the methods. On the receiving side, most languages (.net and java) have annotation type tags that can be placed above code to handle a specific format. So you client has something like:
   
public method URLHandler()
{
[annotation to handle "/customer/[ID].xml"]
doReturnXML();
[annotation to handle "/customer/[ID].json"]
doReturnJSON();
[annotation to handle "/customer/[ID].newformat"]
doReturnNewFormat();
}

   rather than a generic format identifier:
   
public method URLHandler()
{
[annotation to handle "/customer/[ID]"]
format = getFormat(request)
doReturnCustomer(format);
}

   You can certainly do this generic handler, but you end up parsing the type out of the URL which can be tricky if your URI doesn't contain the format or contains the format keywords in the URI.
   





3. Provides some confusion with the return type.
   What are you really requesting, if we watch the wire while the browser makes this request, you see the browser pass an HTTP GET with an accept header of type "HTML" and the function returns you XML or JSON data. This seems wrong to me. From a browser you won't notice this as much, but if you are actually writing a client you will have to understand that you are not asking for the format in the request header - just the URL.





4. Can't rely on the REST hosting framework to enforce the accept formats and deny requests the server cannot fulfill without manually checking the formats. If you are going to be doing this, you probably aren't going to be writing the whole REST framework from the socket up - you are going to be depending on .NET or a Java library to "help" you out with the coding and you just want to slap some tags on a function to do handle it. If you stick the return type in the URI, you are not letting the framework handle the format negotiations for you. So you are essentially manually overriding what part of the framework was designed to handle.





5. Put yourself in a position to really complicate your URI. Lets say you have developed your solution using method 1). Now the team comes along and says, we also want to be able to return the customer with spanish tags to expedite the rollout to Mexico (or whatever wacky format users now want). The logical place to put this with this design would be in the URI so that you aren't mixing methods:
   http://server:port/api/customer/123.json.mx
   or maybe
   http://server:port/api/mx/customer/123.json
   or worse (mix the two designs above)
   http://server:port/api/customer/123.json?language=mx
   This seems to be fighting against something that is already supported in the HTTP header into the URI instead.
   

Analysis of Method 2

The first method has some problems, so lets look at Method 2.

Pros:

1. Again, easily testable in a browser - you just type in the format you want.
2. Again, simple for most to understand - because you explicitly ask for the format you need.

Cons:

1. Still have the issue of confusion with the return type.
   The format query parameter serves as an override to whatever HTTP header request type you have passed.
2. Still have an unnecessarily complicated URI to specify the format you need without relying on the headers:
   http://server:port/api/customer/123?format=xml&language=mx
   

This method seems to fix many of the problems mentioned with method 1 and I feel is also preferrable. The cons are relatively minor and it seems workable from a design perspective.

Analysis of Method 3

Pros:

1. Consistent URIs for all objects regardless of format.
2. Adheres to the expectations of HTTP with regard to requests

Cons:

1. Much more difficult to test than method 1) or 2), but still doable with a plugin in most browsers (Firefox). Since you overriding the browsers default return type (HTML), you must use a plugin to alter the request that gets sent.
2. More training of the staff to understand how HTTP works and why you have to fill out the HTTP request information properly to get the data you want.

Summary

Adhering to the blog title - there is no "right" way to do REST API design. From the analysis, I feel that method 3 is the "best" choice for our architecture and our particular needs. But all 3 have their place and all are used successfully in a variety of architectures. What do you think, what are the pros and cons I didn't think of?

Welcome

I named this blog "There is No Right Way" to remind myself of that fact when it comes to programming. Often there is a general consensus about which way to go, but with the constant flux of technologies and capabilities, even the sure way will change over time. I will attempt to stir up some discussion about programming ideas and situations as I encounter them in my day-to-day programming.