REST Design Philosophies

In my previous post I said that REST is a protocol design technique. Chances are you do not agree or at least not completely. REST can be deceptively simple and frustratingly controversial at the same time. We often find ourselves in the middle of heated debates and are quickly labeled dogmatic RESTafarians or clueless, reckless hackers when expressing an opinion. If you don’t know what I’m talking about, read this well-researched article by Ivan Zuzak or better yet, join a REST discussion group.

Why is REST controversial? Building distributed applications is hard. REST simplifies it by leveraging existing web design patterns and protocols, but many important design tradeoffs remain. Designers resolve tradeoffs based on what they consider important. What is important is not simply a matter of personal opinion, but depends on the kind of application you are building. So instead of best practices everybody agrees on REST has a number of distinct schools of thought or design philosophies; neither inherently good nor bad, neither obviously right nor wrong; each with its champions and detractors.

I’ll attempt to describe four of these design philosophies without claiming any authority in the matter. Others would likely name and describe them differently. I decided to draw quick sketches instead of painting full portraits, capturing the distinctive features and worrying less about getting every detail right.

Hypermedia APIs

This design philosophy admires and tries to replicate the properties of the World Wide Web as a distributed application development platform. Its proponents believe that the phenomenal success of the Web is due to a number of high-level design choices which together define what they call its architectural style. Roy Fielding was the first to describe this architectural style naming it REpresentational State Transfer or REST. For this reason many consider it the only “true” definition of REST.

The single most distinctive feature of this design philosophy is the use of hypermedia or HTTP messages with embedded hyperlinks, hence the name. The links guide clients to discover available resources and operations dynamically. While specific URIs patterns are used, clients are not expected to understand them and are actively discouraged from parsing or building URIs. You can get a better understanding of how Hypermedia APIs are designed and how they work watching Jon Moore build one in this video.

Much of the Hypermedia API discussion focuses on the difficult task of writing complex, large-scale distributed applications and on evolving them over decades. Design attributes like scalability and loose coupling are considered much more important than simplicity and ease of use. This discourages some who claim that while they understand Hypermedia APIs in theory, they find them difficult to use in practice.

Data APIs

If Hypermedia APIs are an architectural style, Data APIs are first and foremost an application integration pattern. Its proponents consider standardization as the most important design goal, believing that exposing the application’s underlying data in a standard format permits easier and more innovative integrations than exposing the application’s complex, cumbersome, and often restrictive business logic. Data APIs are published by organizations wishing to share useful data without giving access to the application which collects it or are added to existing enterprise software to facilitate third-party integrations.

Data APIs use the HTTP methods POST/GET/PUT/DELETE strictly as the CRUD operations Create/Read/Update/Delete. If they expose any application logic at all, they expose it implicitly. For example, if you use a Data API to manipulate purchase orders you obviously expect that products will be shipped and people billed as a result, but this has no direct effect on how the API looks or works. The behavior is simply implied by our shared understanding of what a purchase order is.

The URI structure is standardized and client applications are encouraged to use URI templates for building queries similar to SQL select statements. The format in which the data is returned is also standardized, often based on the ATOM format. For an example of a standardized Data API take a closer look at Microsoft’s Open Data Protocol.

Web APIs

The designers of Web APIs consider achieving large-scale developer adoption as the most important API design challenge. They try to drive adoption by aiming for the widest possible platform support and maintaining a very low barrier of entry.

It is only a slight exaggeration to say that if a feature is not guaranteed to work in Adobe Flash, on an underpowered mobile phone, on a remote Pacific island, through a web proxy, and from behind an ancient firewall from the 90’s, Web API designers won’t use it. If an interface cannot be accessed directly from HTML forms or called by writing less than five lines of JavaScript, Web API designers worry about adoption.

The result is that Web APIs manage to get by using a surprisingly small subset of features. They stick to GET and POST, largely eschew HTTP headers, strongly favor JSON over XML, and rarely use hyperlinks. Information is often passed in URLs, because “URLs are easy to type into the address bar of a browser”. I call them Web APIs because I see this pattern in services offered to developers at large over the Internet (often free of charge) and are primarily used in Web Mashups or equivalent mobile apps. Many of the smaller, simpler APIs listed in the Programmable Web API Directory are Web APIs. Perhaps the best known and most frequently imitated is the Twitter API.

Pragmatic REST APIs

Pragmatists believe that design trade-offs should be resolved based on concrete, project-specific requirements and constraints. They view the indiscriminate application of the same design patterns to widely different problems as a sign of dogmatic thinking, not as disciplined design. Absent specific requirements and constraints, pragmatists prefer to use the simplest possible design. What “simplest” means is again very context dependent. It may mean a design which fits the technologies, frameworks, or tools used the best.

You might argue that Pragmatic REST is not a separate design philosophy since its practitioners freely borrow and mix design approaches from all the other schools. The “pragmatic” label best describes APIs which do not fit neatly into any of the other categories. There are quite a few of these. If you look at the collection of the Google APIs you will discover characteristics of Hypermedia APIs, Data APIs, and Web APIs, but not in their purest forms.

In Lieu of Conclusion

If memory serves me well, the Big REST Debate reached its peak sometime around 2008.  The rhetoric cooled since, largely because the protagonists agreed to disagree, do REST their own way, and mostly ignore each other. But the issue is far from being settled and I doubt it will ever be. When I decided to illustrate the principles and methods of protocol design using REST, I knew that I can’t pretend not to notice the elephant in the room. That’s why I wrote this post.  In my next post I’ll return to talk about REST and protocol design.

Creative Commons Licence
This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 Canada License.

REST and the Art of Protocol Design

RESTful protocol design is a topic more appropriate for a book than a blog post. Despite the catchy title my goal is very modest. I would like to show you that even limited protocol design knowledge can help you understand the essence of REST and the reasoning behind many of its best practices.

I’ll start with a core insight from protocol design, which is that programs need to share pre-existing knowledge to communicate. Without knowing how to separate incoming messages (framing), convert bytes to useful internal representations (decoding), generate correct sequences of messages to accomplish tasks (state machines), and detect and respond to error conditions, programs only see meaningless streams of bytes. Roy Fielding, who coined the term REST, said: “Of course the client has prior knowledge. Every protocol, every media type definition, every URI scheme, and every link relationship type constitutes prior knowledge that the client must know (or learn) in order to make use of that knowledge.”

What method to use to establish the shared pre-existing knowledge is one of the most fundamental protocol design decisions.

The RESTful method is to minimize the need for new (application-specific) knowledge by reusing the existing knowledge of ubiquitous, mature, and stable web protocols. New RESTful application protocols are designed by constraining and further specifying another application protocol, the HTTP protocol. Every REST message is also a valid HTTP message and can be processed with existing HTTP client libraries, browsers, firewalls, web proxies, and server-side web application frameworks. Simple RESTful applications need very little new code at the protocol layer because so much of this code is reused.

It is important to point out that RESTful protocol design fully leverages HTTP as an application protocol, which is fundamentally different from what other HTTP-based protocols do. SOAP is conceived as a protocol layer above HTTP and uses it as a transport protocol. SOAP needs formal WSDL for protocol description and complex code generation tools in part because it does not reuse any of the application protocol features of HTTP. Other protocols like WebDAV extend HTTP with new methods and headers. While such protocol extensions were anticipated in the HTTP specification, many practical difficulties arise when generic HTTP libraries, intermediaries, or frameworks fail to understand the newly added features. WebDAV does not work well over the Internet and failed to see wide-scale adoption. This is why RESTful protocol design avoids the use of HTTP protocol extensions.

A frequently cited example of a carefully designed RESTful protocol is the Atom Publishing Protocol described in RFC 5023. You can tell that it leverages and constrains HTTP as an application protocol after seeing the HTTP specification explicitly referenced 18 times, in sentences like “Any server response or server content modification not explicitly forbidden by this specification or HTTP [RFC2616] is therefore allowed”.

Reusing existing web protocols has many advantages. First among them is a very low barrier of entry and implicit support for a large variety of platforms. Visibility, or the ability of third parties to see and interact with the protocol and provide useful services, is another. Web protocols are extremely stable and resilient, to the point that in 2011 InfoQ could ran the fake news of the release of the HTTP/1.2 as an April’s Fools Day joke. They evolve and combine well thanks to a clever design which fully decouples the concerns of addressing, operations, and representations. Web applications are the best illustration of just how beneficial loose coupling is: you can run all of them in the same generic client, a web browser, without adding or modifying a single line of code.

There is, however, an important difficulty. You cannot use existing protocols as pre-existent knowledge unless you use them precisely as they were meant to be used. Protocol visibility means that server-side frameworks, network intermediaries, and client libraries read and interpret HTTP methods, status codes, and headers and automatically take specific actions, like closing the TCP connection, caching content, or clearing caches. They stop working correctly if you do something unexpected. It is good to know that when people insist on the correct way of doing RESTful protocol design, what they talk about is more than just a personal opinion.

What else do I mean by using web technologies the way they were meant to be used? Ideally, I would like to see all RESTful protocols work with a single generic client like a web browser. Before you jump to any conclusions, let me state that compatibility with browsers is not an actual REST requirement. However, the further away you move from this ideal, the more application-specific knowledge you’ll need to hard-wire into your programs. And, as I said, minimizing application-specific knowledge is at the very core of RESTful protocol design philosophy. This is again good to know, because many common REST best practices directly derive from it.

A practical advice immediately follows from the above discussion, namely that before taking the plunge into RESTful protocol design it is a good idea to study and understand the underlying web technologies. This is not an effortless undertaking. The HTTP 1.1 specification (RFC 2616) alone is 176 pages long, with enough subtleties to justify the publication of several popular books. URIs are not just strings, but also a set of rules, described in the 40 page long RCF 2396. Another 43 pages in RFC 2046 specify the MIME media types used in HTTP requests and responses and we haven’t even talked yet about XML, for which a entire set of specifications is maintained by the W3C. Even the much simpler JSON representation has a formal specification, the 10-page RCF 4627.

I get two strong objections to the above advice.

A frequent objection is this: “Why should I spend time and effort learning these protocol implementation details when I’ve already got code which handles them?” As I’ve already explained in a previous blog post, when I work with existing libraries and frameworks, I work with leaky abstractions, and, as Joel Spolsky said, “…they save us time working, but they don’t save us time learning”. I often find that I need to know the implementation details to use these abstractions correctly.

Another objection is that my advice sounds like coming from someone who completely ignores such fundamental REST principles as stateless interactions, the use of uniform interfaces, or hypertext as the engine of application state (HATEOAS). This is intentional from my part. Not that the principles aren’t important, but they are also powerful abstractions and I find them difficult to understand without concrete examples. If I put building on existing protocols at the very core of my design philosophy, it makes perfect sense to me to start by learning how these protocols work. Since they are built on the same principles, learning them also makes understanding the principles a lot easier. This approach worked well for me personally, but of course, your mileage may vary.

In my next post I’ll continue this topic and talk about various REST design philosophies, slightly different ways designers choose to leverage web protocols depending on which design trade-offs they consider more important.

Creative Commons Licence
This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 Canada License.