REST and the Art of Protocol Design

RESTful protocol design is a topic more appropriate for a book than a blog post. Despite the catchy title my goal is very modest. I would like to show you that even limited protocol design knowledge can help you understand the essence of REST and the reasoning behind many of its best practices.

I’ll start with a core insight from protocol design, which is that programs need to share pre-existing knowledge to communicate. Without knowing how to separate incoming messages (framing), convert bytes to useful internal representations (decoding), generate correct sequences of messages to accomplish tasks (state machines), and detect and respond to error conditions, programs only see meaningless streams of bytes. Roy Fielding, who coined the term REST, said: “Of course the client has prior knowledge. Every protocol, every media type definition, every URI scheme, and every link relationship type constitutes prior knowledge that the client must know (or learn) in order to make use of that knowledge.”

What method to use to establish the shared pre-existing knowledge is one of the most fundamental protocol design decisions.

The RESTful method is to minimize the need for new (application-specific) knowledge by reusing the existing knowledge of ubiquitous, mature, and stable web protocols. New RESTful application protocols are designed by constraining and further specifying another application protocol, the HTTP protocol. Every REST message is also a valid HTTP message and can be processed with existing HTTP client libraries, browsers, firewalls, web proxies, and server-side web application frameworks. Simple RESTful applications need very little new code at the protocol layer because so much of this code is reused.

It is important to point out that RESTful protocol design fully leverages HTTP as an application protocol, which is fundamentally different from what other HTTP-based protocols do. SOAP is conceived as a protocol layer above HTTP and uses it as a transport protocol. SOAP needs formal WSDL for protocol description and complex code generation tools in part because it does not reuse any of the application protocol features of HTTP. Other protocols like WebDAV extend HTTP with new methods and headers. While such protocol extensions were anticipated in the HTTP specification, many practical difficulties arise when generic HTTP libraries, intermediaries, or frameworks fail to understand the newly added features. WebDAV does not work well over the Internet and failed to see wide-scale adoption. This is why RESTful protocol design avoids the use of HTTP protocol extensions.

A frequently cited example of a carefully designed RESTful protocol is the Atom Publishing Protocol described in RFC 5023. You can tell that it leverages and constrains HTTP as an application protocol after seeing the HTTP specification explicitly referenced 18 times, in sentences like “Any server response or server content modification not explicitly forbidden by this specification or HTTP [RFC2616] is therefore allowed”.

Reusing existing web protocols has many advantages. First among them is a very low barrier of entry and implicit support for a large variety of platforms. Visibility, or the ability of third parties to see and interact with the protocol and provide useful services, is another. Web protocols are extremely stable and resilient, to the point that in 2011 InfoQ could ran the fake news of the release of the HTTP/1.2 as an April’s Fools Day joke. They evolve and combine well thanks to a clever design which fully decouples the concerns of addressing, operations, and representations. Web applications are the best illustration of just how beneficial loose coupling is: you can run all of them in the same generic client, a web browser, without adding or modifying a single line of code.

There is, however, an important difficulty. You cannot use existing protocols as pre-existent knowledge unless you use them precisely as they were meant to be used. Protocol visibility means that server-side frameworks, network intermediaries, and client libraries read and interpret HTTP methods, status codes, and headers and automatically take specific actions, like closing the TCP connection, caching content, or clearing caches. They stop working correctly if you do something unexpected. It is good to know that when people insist on the correct way of doing RESTful protocol design, what they talk about is more than just a personal opinion.

What else do I mean by using web technologies the way they were meant to be used? Ideally, I would like to see all RESTful protocols work with a single generic client like a web browser. Before you jump to any conclusions, let me state that compatibility with browsers is not an actual REST requirement. However, the further away you move from this ideal, the more application-specific knowledge you’ll need to hard-wire into your programs. And, as I said, minimizing application-specific knowledge is at the very core of RESTful protocol design philosophy. This is again good to know, because many common REST best practices directly derive from it.

A practical advice immediately follows from the above discussion, namely that before taking the plunge into RESTful protocol design it is a good idea to study and understand the underlying web technologies. This is not an effortless undertaking. The HTTP 1.1 specification (RFC 2616) alone is 176 pages long, with enough subtleties to justify the publication of several popular books. URIs are not just strings, but also a set of rules, described in the 40 page long RCF 2396. Another 43 pages in RFC 2046 specify the MIME media types used in HTTP requests and responses and we haven’t even talked yet about XML, for which a entire set of specifications is maintained by the W3C. Even the much simpler JSON representation has a formal specification, the 10-page RCF 4627.

I get two strong objections to the above advice.

A frequent objection is this: “Why should I spend time and effort learning these protocol implementation details when I’ve already got code which handles them?” As I’ve already explained in a previous blog post, when I work with existing libraries and frameworks, I work with leaky abstractions, and, as Joel Spolsky said, “…they save us time working, but they don’t save us time learning”. I often find that I need to know the implementation details to use these abstractions correctly.

Another objection is that my advice sounds like coming from someone who completely ignores such fundamental REST principles as stateless interactions, the use of uniform interfaces, or hypertext as the engine of application state (HATEOAS). This is intentional from my part. Not that the principles aren’t important, but they are also powerful abstractions and I find them difficult to understand without concrete examples. If I put building on existing protocols at the very core of my design philosophy, it makes perfect sense to me to start by learning how these protocols work. Since they are built on the same principles, learning them also makes understanding the principles a lot easier. This approach worked well for me personally, but of course, your mileage may vary.

In my next post I’ll continue this topic and talk about various REST design philosophies, slightly different ways designers choose to leverage web protocols depending on which design trade-offs they consider more important.

This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 Canada License.