Message Encoding in REST

The basic HTTP message framing described in my last post has some performance issues. You can think of advanced message framing techniques like chunking, compression, and multipart messages as performance optimizations. The HTTP specification calls them encodings because they alter (encode) the message body for transmission and reconstruct it at arrival. All HTTP 1.1 libraries and frameworks support chunkig, most support compression, and many can handle multipart messages too.

Chunking

You need to know the length of the message body to use the Content-Length header. While this is not an issue with static content like HTML files, it creates performance problems with dynamically generated REST messages. Not only that the entire message body has to be buffered in memory, but the network sits unused during message generation and the CPU is idle during transmission. The performance improves significantly if the two steps are combined and message generation or processing is done in parallel with transmission. To support this, a new message length prefixing model, called chunked transfer encoding, was introduced in HTTP 1.1. It is described in RFC 2616, section 3.6.1

Chunked transfer encoding is length prefixing applied not to the entire message body, but to smaller parts of it (chunks). A chunk starts with the length of data in hexadecimal format separated by a CRLF (carriage return and line feed) sequence from the actual chunk data. Another CRLF pair marks the end of the chunk. After a series of chunks a zero-length chunk signals the end of the message. The presence of the Transfer-Encoding header containing the value “chunked” and the absence of the Content-Length header tell the receiver to read the message body in chunks (Figure 1).

Figure 1: Compressed and chunked HTTP response

You don’t need to take any action to enable chunking. Basic message framing is used only for short messages. Longer messages are automatically chunked by HTTP 1.1 implementations. Just remember not to depend on the Content-Length header for message processing because it is not always present. Structure your code so that you can begin processing before receiving the full message body. Use streaming and event-based programming techniques, for example event-based JSON and XML parsers.

Compression

Compression is a data optimization technique used to reduce message sizes, thus network bandwidth usage. It also speeds up message delivery. Compression looks for repeating patterns in data streams and replaces their occurrences with shorter placeholders. How effective this is depends on the type and volume of data. XML and JSON compress quite well, by 60% or more for messages over two kilobytes in length. Experience shows that the generic HTTP compression algorithms are just as effective as specially developed JSON- and XML-aware algorithms.

HTTP compresses the message body before sending it and un-compresses it at arrival. Headers are never compressed. If present, the Content-Length header shows the number of the actual (compressed) bytes sent. Similarly, chunking is applied to the already compressed stream of bytes (Figure 1). Since compression is optional in HTTP 1.1, clients must list the compression algorithms they support in the Accept-Encoding header to receive compressed messages. Likewise, REST services should not send compressed content unless the client is prepared to receive it. Compressed messages are sent with a Content-Encoding header identifying the compression algorithm used.

Client support for HTTP compression is good since well-known public domain algorithms are used. On Android, you can receive gzip compressed messages with the Spring for Android RestTemplate Module. RestKit for Apple iOS is built on top of NSURLConnection, which provides transparent gzip/deflate support for response bodies. On Blackberry you can use HttpConnection with with GZipInputStream. When writing client-side Javascript the XmlHttpRequest implementation decompresses messages automatically.

Multipart messages

As the name suggests, multipart messages contain parts of different types within as a single message body. This helps reduce protocol chattiness, eliminating the need to retrieve each part with a separate HTTP request. In REST this technique is called batching.

Figure 2: Multipart HTTP response

Multipart encoding was originally developed for email attachments (Multipurpose Internet Mail Extensions or MIME) and later extended to HTTP and the Web. It is described in six related RFC documents: RFC 2045, RFC 2046, RFC 2047, RFC 4288, RFC 4289 and RFC 2049. Figure 2 shows a multipart message example. To receive multipart messages, clients must list the multipart formats they support in the Accept header. The Content-Type header identifies the type of multipart message sent (related, mixed, etc.) and also contains the byte pattern used to mark the boundary between message parts. The mime type of each message part is transmitted separately as shown in Figure 2.

Building and parsing multipart messages is difficult unless supported out-of-the box by libraries or frameworks. You cannot expect clients to write code for it from scratch so you must provide alternative ways of retrieving the data. Common alternatives are embedding links into messages to let clients retrieve parts separately or using JSON or XML collections for embedding multiple message parts of the same type.

REST message examples are always shown in documentation with basic message framing because it is easy to read. Nonetheless, many real-life REST protocols use a combination of chunking, compression, and multipart encoding for better performance. When you develop REST clients, remember that compression and multipart encoding are optional. REST services won’t use them unless the client sends them the proper Accept and Accept-Encoding headers. When you design REST services, consider compressing large messages to save bandwidth and using multipart messages to reduce protocol chattiness.

This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 Canada License.