16 May 2012

API Example Using REST

In the last post I talked about the non-RESTfulness of most "REST" api implementations. I will attempt to show a more RESTful implementation and some examples of how this benefits the software. While researching this topic, it became clear (by both lack of examples and poor examples) that this is a tough concept. Before I jump into the example, lets examine why its so hard - because these issues will drive some of the design.

Difficulties with API and REST

An Application Programming Interface (API) is supposed to be:

a set of functions, procedures, methods or classes used by computer programs to request services from the operating system, software libraries or any other service providers running on the computer.
And when creating a traditional API, you come up with methods and data formats to tell people how to "talk" to the system. But one of the primary REST tenets as mentioned by Roy Fielding is his blog is that you NOT have "out-of-band information driving interaction instead of hypertext." To put more simply, you should not pre-define the format of the data or the methods to be used, they should be discovered. But isn't that what an API is? How does one define the methods and the data formats but not have to know the methods or data formats? And why should I follow this "silly rule" anyway?

Well, the silly rule is the brilliance of REST and also what allows for the incredible flexibility that is the internet as we know today. If we know the format or the calls to make ahead of time, and write software against those, the control is no longer in the data providers hands. The control has shifted back from server to client. To understand the problem with this better, imagine that you wrote a custom browser and hard-coded it to a specific news article page with a fixed format and calling it silly that the news site wants to change the url or page format. (This example is stretched a bit far because an API has the understanding of being slightly more permanent than a generic webpage, but it still applies in spirit to RESTful APIs.)

The solution here is that there was a loophole of sorts, baked into the "Rules of REST" - note the last part in the rule: "out-of-band information driving information instead of hypertext." You can tell the client what it is allowed to do, as long as its part of the data (hypertext) being returned. The "media type" can have the information about what is allowed and what format things should be in! This, combined with the specifics of our chosen protocol (HTTP here), should give us all we need. So how do we do that?

REST Example

In the examples in the last post, the user logged in, searched for a product and created an order in both a traditional website and with a traditional API. So lets now make a REST version:

NOTE: I am going to ignore security at the beginning because it doesn't help in understanding the example, but I will discuss what could be done at the end.

Client starts with the root URL www.example.com/api/ and a discoverable media type apiIndex-v1+xml.

GET / www.example.com/api/ HTTP/1.1
Accept: application/vnd.example.coolapp.apiIndex-v1+xml

Response:
HTTP/1.1 200 OK
Content-Type: application/vnd.example.coolapp.apiIndex-v1+xml
<apiIndex>
  <link>
    <description>Customers</description>
    <href>http://www.example.com/api/customers</href>
    <type>application/vnd.example.coolapp.customers-v1+xml</type>
  </link>
  <link>
    <description>Products</description>
    <href>http://www.example.com/api/products</href>
    <type>application/vnd.example.coolapp.products-v1+xml</type>
  </link>
  <link>
    <description>Carts</description>
    <href>http://www.example.com/api/carts</href>
    <type>application/vnd.example.coolapp.carts-v1+xml</type>
  </link>
  <link>
    <description>Orders</description>
    <href>http://www.example.com/api/orders</href>
    <type>application/vnd.example.coolapp.orders-v1+xml</type>
  </link>
  <link>
    <description>Sessions</description>
    <href>http://www.example.com/api/sessions</href>
    <type>application/vnd.example.coolapp.sessions-v1+xml</type>
  </link>
  <link>
    <description>API Help</description>
    <href>http://www.example.com/api/help</href>
    <type>text/html</type>
  </link>
</apiIndex>

Here I have defined a set of valid hyperlinks the client can travel to from the root. It is reasonable to assume that the client could get more stuff or less stuff depending on what the business wanted to provide at the root, or based on what the user is authorized to use. This exchange would be the equivalent of a generic "index.html" page for the web. The client could show the descriptions as buttons or a drop down or whatever makes sense for the client (but it doesn't matter at the API level because it is not providing a page to be rendered, just resources to be processed as the client sees fit).

Lets say the client chooses to view a list of products, but does not know the allowed operations - the client could then perform an HTTP OPTIONS call against the resource to get the list (there are few standards for the format of the response of an OPTIONS call, but I went with a custom response body for the API to show a useful example):

OPTIONS /api/products
Response:
HTTP/1.1 200 OK
ALLOW: HEAD,GET,OPTIONS
Content-Type:application/vnd.example.coolapp.options-v1+xml
Content-Length: 1032
<options>
  <methods>
    <method>
      <type>GET</type>
      <description>Retrieve a list of products</description>
      <parameters>
        <parameter>
          <name>Keyword</name>
          <type>string</type>
          <required>false</required>
        </parameter>
        <parameter>
          <name>MaxResults</name>
          <type>integer</type>
          <required>false</required>
        </parameter>
        <parameter>
          <name>StartIndex</name>
          <type>integer</type>
          <required>false</required>
        </parameter>        
      </method>
    </methods>
  ...
</options>

Now seeing the options and the calling parameters, the client then provides a product search entry form in its UI, and then decides to perform the search against the API:

GET /api/products?Keyword=green%20widget

Response:
HTTP/1.1 200 OK
Content-Type:application/vnd.example.coolapp.product-v1+xml
Content-Length: 1032
<products>
  <resultInformation>
    <totalItems>3923</totalItems>
    <maxResults>15</maxResults>
    <maxBytes>10000</maxBytes>
    <startIndex>1</startIndex>
    <endIndex>15</endIndex>
  </resultInformation>
  <product>
    <id>12343</id>
    <description>389349 green widget with bells</description>
    <price currencyCode="USD">3.56</price>
    <link>
      <description>Product 12343</description>
      <href>http://www.example.com/api/products/12343</href>
      <type>application/vnd.example.coolapp.product-v1+xml</type>
    </link>
  </product>
  <product>
    <id>2343</id>
    <description>2343 green widget copper clad</description>
    <price currencyCode="USD">4.93</price>
    <link>
      <description>Product 2343</description>
      <href>http://www.example.com/api/products/2343</href>
      <type>application/vnd.example.coolapp.product-v1+xml</type>
    </link>
  </product>
  ...
</products>

The client then processes the results and allows the user to add items to a cart. This can be done against a cart resource so that the cart object is validated by the server (minimum quantities, availability, etc) and so that the client can save the cart if it doesn't want to commit to an order just yet:

POST /api/carts
Content-Type:application/vnd.example.coolapp.cart-v1+xml
Content-Length: 1032
<cart>
  <customerId>1343</customerId>
  <lineItems>
    <lineItem>
      <productId>12343</productId>
      <quantity>4</quantity>
    </lineItem>
    <lineItem>
      <productId>39293</productId>
      <quantity>2</quantity>
    </lineItem>
    ...
  </lineItems>
</cart>

Response:
HTTP/1.1 201 Created
Location: /api/carts/323392

Now that the cart is all set, perhaps the client decides to create an order:

POST /api/orders
Content-Type: application/vnd.example.coolapp.order-v1+xml
Content-Length: 2939
<order>
  <customerId>1343</customerId>
  <lineItems>
    <lineItem>
      <productId>12343</productId>
      <quantity>4</quantity>
    </lineItem>
    <lineItem>
      <productId>39293</productId>
      <quantity>2</quantity>
    </lineItem>
    ...
  </lineItems>
</order>

Response:
HTTP/1.1 201 Created
Location: /api/orders/O293920

That is the basic REST API example.

Security Aside

Let me walk through some of the security handling in this example since it can be confusing the first time you work through it. When you hit the index page for the API, the server would likely be expecting a valid session token in the form of an "Authorization" header so that we can authorize access to resources appropriately. If that header is not present, the server could send back a 302 redirect to a session resource (so that the client could submit a username and password to get a valid session) like so:

GET /api HTTP/1.1
Accept: application/vnd.example.coolapp.apiIndex-v1+xml
Response:
HTTP/1.1 302 Moved
Location: http://www.example.com/api/session

At which point the client could do an OPTIONS to learn the valid session options (not shown), then POST credentials to get a new session resource back:

POST /api/session
Content-Type: application/vnd.example.coolapp.sessionCredentials-v1+xml
Content-Length: 100
<sessionCredentials>
  <username>joe</username>
  <password>opensesame</password>
</sessionCredentials>
Response:
HTTP/1.1 201 Created
Location: http://www.example.com/api.session/74823
Content-Type: application/vnd.example.coolapp.session-v1+xml
Content-Length: 203
<session>
  <id>74823</id>
  <token>293929FDAAECD3293FFA32</token>
  <username>joe</username>
</session>

Typically, you could do something similar to all resources, but probably using 401 Not Authorized instead of the redirect since you really want the user to go through the index page instead of "knowing" the URL ahead of time. This is just a quick example of security and there are a lot of security vulnerabilities here that you should be careful to avoid. Plus, the client would have to start over at the index page, using the valid token - which is a little awkward. There are dozens of different ways to handle security, like OAuth, BASIC Authentication, etc, so more research is warranted with this topic, but you get the basic idea.

So now let me talk about some of the nuances with the flow.

Notes about REST APIs

I am going to parallel these comments with my interpretation of the "rules" that Mr. Fielding presented in his REST blog post.

  1. The only thing the client should know ahead of time is the root URL.  It shouldn't know or expect as specific resource to be available. For example, the client can't assume an "order" resource or a "search" resource - those should be provided as choices to the client from the server.

The first thing to note is that although the client only knows the root URL, it still must be coded to handle the link structure that was defined for the root, plus the media types that were presented. It is very difficult to avoid writing something akin to a browser when consuming a REST API. To create a client properly in this example, it should handle the creation or deletion of links on the index page, and gracefully handle any unknown media types that are presented. Otherwise, you are just basically making an RPC call, because you both know the format and nothing can change without both sides changing. That destroys the REST concept and prevents changes from occurring as seamlessly as it should (such as the host deciding to change the URL, or alter the flow through the API.)

  1. "A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types."

The bulk of what can be done or what is available is meant to be part of the object that is returned. You will notice above that I included a special "resultInformation" block in the product search results. This allows the client to present a "1 of X" link in the app, or make multiple requests to cycle over the results from the server (and limit the bandwidth from the server). Also you will notice that the flow through the API does not jump around, for example, already "knowing" the URL of orders. You don't just jump to /api/orders/123, because the server may have moved it, or deleted it, or you may not be authorized to view it. If you want to give the client the ability to view orders, you create an orderView resource that returns a displayable form of the order and lets the client pick from a list. (server is in control what is available).

  1. "A REST API should not contain any changes to the communication protocols aside from filling-out or fixing the details of under specified bits of standard protocols, such as HTTP’s PATCH method or Link header field."

You will notice above that I presented a use of the HTTP OPTIONS method to assist with the consumption of the API, but tried to stay within the bounds of what is "standard" HTTP. In a normal non-REST API, these method signatures are predefined in a document and both sides code to them. If you do that in REST, you couple the client and server so that neither may change without the other changing. You want the server to decide what is presented and make the client smart enough to handle those choices. Again, it is very difficult to write a client that that is not browser-like and is a little unnatural for an API, but gives great power to the server to evolve.

Again, I have been more long-winded with my post that I would like, but hopefully it gives some value to the developers and REST-ees out there. This was a very difficult post for me to come up with because I had to constantly focus on the REST principles and avoid the classic API mindset. I found it very helpful to discuss with colleagues and to compare what a browser does with what we are trying to get out of the API. There are lots of things that I didn't have space to discuss, like versioning, content-negotiation, internationalization, etc that make this a very rich communication platform to build on. I would appreciate any comments or questions about this flow and REST in general. I know that there are things I can do better, and things that likely need fixing (and that everyone is going to do it differently for their purposes since there is no "right way") so be sure and add a comment.

27 April 2012

Putting the 'ST' in REST

Starting to implement REST has led to some interesting discussions and research topics with regards to what it means to be "REST". After reading Roy Fielding's post "REST APIs must be hypertext driven" - I think a walk through of this point might help things (both for my own understanding and for anyone who might be reading).

Unfortunately, Roy's "rant" comes off a little theoretical - and I think that might be one of the reasons why so many people are missing the point he is trying to make. The basic idea is that the API itself gives you all the information for the route your application can go. All the "application" knows is a root address - everything else flows from the choices the user is given and what they choose. In API land, this is a very foreign concept. Normally, when interfacing to an API, the application consuming the API has a structure and pages hardcoded, and those pages have buttons which restrict the flow for the user. So the API is just a wrapper around the data that the application will be using.

Web vs API

The real difference here is how web pages work versus how the typical API consumption works. For example, if you want to place an order on a website, the flow is something like this:

Webpage Application Example:


  • User goes to the root URL (http://www.awesomeproduct.com), and is presented with links to product pages of stuff on sale and a search box of some sort.

  • The user types in "green widget" to the search box and hits the "search" button.

  • The user is redirected to a product search result page with all the green widgets listed.

  • User clicks a product result link and is taken to a product information page with a "add to cart" button.

  • User likes the product, clicks "add to cart" and is taken either to the cart or the general search page

  • For web pages, this all works because there is an understanding that you are using a web browser and are expecting HTML to come back from nearly every resource. Your browser negotiates with the web server and gets an HTML representation of the information to display and displays it as text/images/buttons/forms. The returned page may contain links to other pages or related information and you can choose to browse to that information. The page could conceivably have a form that you fill out to send a message or something.

    Now compare this to an example ordering application consuming a typical "secure" non-REST API:

    Proprietary API Example:


  • User opens a pre-built login form in the application. This page has a username/password pair that get encrypted and sent to a Login method in the API via a socket connection, which returns a session token for subsequent requests.

  • The application then displays a predefined search page to locate products to order. The user enters "green widget" and hits search (passing the token).

  • "green widget" is sent to the SearchProducts API method and it returns a list of products matching "green widget".

  • The application displays the list in a table and the user hits "Add to order" button on the app. This triggers the application to package up a new order object and send it to the API's OrderCreate method.

  • An order is created on the backend and a success message is returned


  • As you can see the flows are very similar - but there are some important differences which will be discussed in a minute.

    Lets go "REST" !!

    This is pretty much how things are done for most applications, but then some ambitious programmer comes along and says "Lets make our API REST so we can use the same API for multiple apps" (or some such reasoning). The sales people chime in and say "Yeah, lets go REST because everyone is clamoring for that buzzword" and so the decision is made to do REST. After the 10 minute "Google Tutorial" on REST, they implement the following BAD implementation:

    Login method:
    GET to www.awesomeproducts.com/api/v1/login?user=x;encryptedpassword=y
    Product search:
    GET to www.awesomeproducts.com/api/v1/products?keyword="green widget"
    Order create:
    POST to www.awesomeproducts.com/api/v1/orders

    The developers of the app then change over their calls from their proprietary calls and responses to the new "REST" versions using XML defined in an XSD and the app is declared REST compliant! The problem here is that it is NOT REST and it is really no better than the previous proprietary non-REST version before. All that was really changed was the formatting of the data going to and from the backend, and the standardization on a communication protocol (HTTP).  Whats worse, is that it gives the appearance of REST, but it is not really REST at all.

    Problems with RE"ST"

    So why is this bad and how is doing it "properly" going to help anyway? Let now add a simple change to the situation to make the problem more clear - lets say management says that they want to display an advertisement for the new and amazing purple widgets after a customer logs in, but before the search. That will surely drive sales and we have to have this new feature yesterday.  So the app designers go to work and change the application to stick in this new advertisement (because changes to the API can't really make an ad show up in the hard-coded flow).

    BUT WAIT - why does the application have to change?!   That is the problem with this implementation of "REST"!  The issue here, is that if it was REST, the application would not have changed to accommodate this new 'page' in the flow.  To put it in perspective, imagine that if Amazon or Google wanted to change their pages, you had to upgrade your browser to see the new pages!!  That would make the web unworkable.  Yet this is what purported 'REST' designers have been doing and the type of thing that prompted Roy Fielding to go off on his well-deserved "rant" post.  With 'proper' REST, you should have been able to insert or change the flow by simply changing the transitions returned from the API server - not changes in the client.  What you have here is what my coworker called "REST without the 'ST'", there are no state transitions being provided by the "REST" server.  Its akin to web pages with hard-coded pages and no way to link between them - you would just have to know all the pages to go to and manually type them in (or have them hard-coded in your browser) - and any changes to the web page would require you to alter the browser.

    The brilliance of REST is not with the format of the data, or the HTTP negotiation - its with the transfer of state control from the client to the server.   You are allowing a third party to not only provide you with information in a format you can understand, but to control the valid paths through the "application" by letting it tell you were you can go - and then letting you choose.  Prior to REST, it was just a program that was hard-coded to known resources on a server and to find new information, you had to re-code the program to access those resources.  After REST, programs let you connect to servers which provide you a list of information(resources) you could go to.  You could then "choose-your-own-adventure" through the resources and, more importantly, the information providers could control your experiences as they saw fit - directing you down whichever back alley your choices lead.  This paradigm shift cannot be understated and is what gives REST its power.

    Aside: If you want to read more about this shift, it goes under the more technical name HATEOAS (Hypermedia As The Engine Of Application State)

    Roy was justifiably upset to see that great idea applied to situations where all that was "REST-a-fied" was the format of the data and some details around the protocol.  Which is really the same as the Proprietary API Example I provided above.  You basically hard code a program to pull resources in a predefined format from fixed resources.  The only thing that was really added was the standardization of format - which does allow others to more easily pick up the API, but its not REST.

    So How do We Do It More RESTful?

    So this is the point where most REST folks (myself included) start getting flak about being "REST purist" and that providing the data in XML via HTTP using a "REST" URL is good enough.  But you miss the entire crux of what makes REST so great - transfer of state control from the client to the server.  So what changes in the above example to make the Proprietary API Example a REST API example?

    Because this post is running long, I will make a new post explaining an example of how to use REST in an API context.