RESTful Web Services

Chapter 1, The Programmable Web and Its Inhabitants

In this chapter we introduce web services in general: programs that go over the Web and ask a foreign server to provide data or run an algorithm. We demonstrate the three common web service architectures: RESTful, RPC-style, and REST-RPC hybrid. We show sample HTTP requests and responses for each architecture, along with typical client code.

Chapter 2, Writing Web Service Clients

In this chapter we show you how to write clients for existing web services, using an HTTP library and an XML parser. We introduce a popular REST-RPC service (the web service for the social bookmarking site del.icio.us) and demonstrate clients written in Ruby, Python, Java, C#, and PHP. We also give technology recommendations for several other languages, without actually showing code. JavaScript and Ajax are covered separately in Chapter 11.

Chapter 3, What Makes RESTful Services Different?

We take the lessons of Chapter 2 and apply them to a purely RESTful service: Amazon’s Simple Storage Service (S3). While building an S3 client we illustrate some important principles of REST: resources, representations, and the uniform interface.

Chapter 4, The Resource-Oriented Architecture

A formal introduction to REST, not in its abstract form but in the context of a specific architecture for web services. Our architecture is based on four important REST concepts: resources, their names, their representations, and the links between them. Its services should be judged by four RESTful properties: addressability, statelessness, connectedness, and the uniform interface.

Chapter 5, Designing Read-Only Resource-Oriented Services

We present a procedure for turning an idea or a set of requirements into a set of RESTful resources. These resources are read-only: clients can get data from your service but they can’t send any data of their own. We illustrate the procedure by designing a web service for serving navigable maps, inspired by the Google Maps web application.

Chapter 6, Designing Read/Write Resource-Oriented Services

We extend the procedure from the previous chapter so that clients can create, modify, and delete resources. We demonstrate by adding two new kinds of resource to the map service: user accounts and user-defined places.

Chapter 7, A Service Implementation

We remodel an RPC-style service (the del.icio.us REST-RPC hybrid we wrote clients for back in Chapter 2) as a purely RESTful service. Then we implement that service as a Ruby on Rails application. Fun for the whole family!

Chapter 8, REST and ROA Best Practices

In this chapter we collect our earlier suggestions for service design into one place, and add new suggestions. We show how standard features of HTTP can help you with common problems and optimizations. We also give resource-oriented designs for tough features like transactions, which you may have thought were impossible to do in RESTful web services.

Chapter 9, The Building Blocks of Services

Here we describe extra technologies that work on top of REST’s big three of HTTP, URI, and XML. Some of these technologies are file formats for conveying state, like XHTML and its microformats. Some are hypermedia formats for showing clients the levers of state, like WADL. Some are sets of rules for building RESTful web services, like the Atom Publishing Protocol.

Chapter 10, The Resource-Oriented Architecture Versus Big Web Services

We compare our architecture, and REST in general, to another leading brand. We think that RESTful web services are simpler, more scalable, easier to use, better attuned to the philosophy of the Web, and better able to handle a wide variety of clients than are services based on SOAP, WSDL, and the WS-* stack.

Chapter 11, Ajax Applications as REST Clients

Here we explain the Ajax architecture for web applications in terms of web services: an Ajax application is just a web service client that runs inside your web browser. That makes this chapter an extension of Chapter 2. We show how to write clients for RESTful web services using XMLHttpRequest and the standard JavaScript library.

Chapter 12, Frameworks for RESTful Services

In the final chapter we cover three popular frameworks that make it easy to implement RESTful web services: Ruby on Rails, Restlet (for Java), and Django (for Python).

Appendix A, Some Resources for REST and Some RESTful Resources

The first part lists interesting standards, tutorials, and communities related to RESTful web services. The second part lists some existing, public RESTful web services that you can use and learn from.

Appendix B, The HTTP Response Code Top 42

Describes every standard HTTP response code (plus one extension), and explains when you’d use each one in a RESTful web service.

Appendix C, The HTTP Header Top Infinity

Does the same thing for HTTP headers. It covers every standard HTTP header, and a few extension headers that are useful for web services.

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

The HTTP method

In this request, the method is “GET.” In other discussions of REST you may see this called the “HTTP verb” or “HTTP action.”

The name of the HTTP method is like a method name in a programming language: it indicates how the client expects the server to process this envelope. In this case, the client (my web browser) is trying to GET some information from the server (www.oreilly.com).

The path

This is the portion of the URI to the right of the hostname: here, http://www.oreilly.com/index.html becomes “/index.html.” In terms of the envelope metaphor, the path is the address on the envelope. In this book I sometimes refer to the “URI” as shorthand for just the path.

The request headers

These are bits of metadata: key-value pairs that act like informational stickers slapped onto the envelope. This request has eight headers: Host, User-Agent, Accept, and so on. There’s a standard list of HTTP headers (see Appendix C), and applications can define their own.

The entity-body, also called the document or representation

This is the document that inside the envelope. This particular request has no entity-body, which means the envelope is empty! This is typical for a GET request, where all the information needed to complete the request is in the path and the headers.

The HTTP response code

This numeric code tells the client whether its request went well or poorly, and how the client should regard this envelope and its contents. In this case the GET operation must have succeeded, since the response code is 200 (“OK”). I describe the HTTP response codes in Appendix B.

The response headers

Just as with the request headers, these are informational stickers slapped onto the envelope. This response has 10 headers: Date, Server, and so on.

The entity-body or representation

Again, this is the document inside the envelope, and this time there actually is one! The entity-body is the fulfillment of my GET request. The rest of the response is just an envelope with stickers on it, telling the web browser how to deal with the document.

The most important of these stickers is worth mentioning separately. The response header Content-Type gives the media type of the entity-body. In this case, the media type is text/html. This lets my web browser know it can render the entity-body as an HTML document: a web page.

There’s a standard list of media types (http://www.iana.org/assignments/media-types/). The most common media types designate textual documents (text/html), structured data documents (application/xml), and images (image/jpeg). In other discussions of REST or HTTP, you may see the media type called the “MIME type,” “content type,” or “data type.”

• http://flickr.com/photos/tags/penguin

• http://api.flickr.com/services/rest/?method=flickr.photos.search&tags=penguin

The main topic of this book is the web service architectures which can be considered RESTful: those which get a good score when judged on the criteria set forth in Roy Fielding’s dissertation. Now, lots of architectures are technically RESTful,^[7]but I want to focus on the architectures that are best for web services. So when I talk about RESTful web services, I mean services that look like the Web. I’m calling this kind of service resource-oriented. In Chapter 3 I’ll introduce the basic concepts of resource-oriented REST, in the context of a real web service: Amazon’s Simple Storage Service. Starting in Chapter 4, I’ll talk you through the defining characteristics of REST, and define a good architecture for RESTful web services: the Resource-Oriented Architecture.

In RESTful architectures, the method information goes into the HTTP method. In resource-oriented architectures, the scoping information goes into the URI. The combination is powerful. Given the first line of an HTTP request to a resource-oriented RESTful web service (“GET /reports/open-bugs HTTP/1.1”), you should understand basically what the client wants to do. The rest of the request is just details; indeed, you can make many requests using only one line of HTTP. If the HTTP method doesn’t match the method information, the service isn’t RESTful. If the scoping information isn’t in the URI, the service isn’t resource-oriented. These aren’t the only requirements, but they’re good rules of thumb.

A few well-known examples of RESTful, resource-oriented web services include:

Whenever I cover unRESTful architectures, as well as architectures that aren’t resource-oriented, I do it with some ulterior motive. In this chapter, I want to put RESTful web services into perspective, against the larger backdrop of the programmable web. In Chapter 2, I’m widening the book’s coverage of real web services, and showing that you can use the same client tools whether or not a service exactly fits my preferred architecture. In Chapter 10, I’m making an argument in a long-running debate about what the programmable web should look like.

An RPC-style web service accepts an envelope full of data from its client, and sends a similar envelope back. The method and the scoping information are kept inside the envelope, or on stickers applied to the envelope. What kind of envelope is not important to my classification, but HTTP is a popular envelope format, since any web service worthy of the name must use HTTP anyway. SOAP is another popular envelope format (transmitting a SOAP document over HTTP puts the SOAP envelope inside an HTTP envelope). Every RPC-style service defines a brand new vocabulary. Computer programs work this way as well: every time you write a program, you define functions with different names. By contrast, all RESTful web services share a standard vocabulary of HTTP methods. Every object in a RESTful service responds to the same basic interface.

The XML-RPC protocol for web services is the most obvious example of the RPC architecture. XML-RPC is mostly a legacy protocol these days, but I’m going to start off with it because it’s relatively simple and easy to explain. Example 1-11 shows a Ruby client for an XML-RPC service that lets you look up anything with a Universal Product Code.

#!/usr/bin/ruby -w
# xmlrpc-upc.rb

require 'xmlrpc/client'
def find_product(upc)
  server = XMLRPC::Client.new2('http://www.upcdatabase.com/rpc')
  begin
    response = server.call('lookupUPC', upc)
  rescue XMLRPC::FaultException => e
    puts "Error: "
    puts e.faultCode
    puts e.faultString
  end
end

puts find_product("001441000055")['description']
# "Trader Joe's Thai Rice Noodles"

An XML-RPC service models a programming language like C. You call a function (lookupUPC) with some arguments (“001441000055”) and get a return value back. The method data (the function name) and the scoping data (the arguments) are put inside an XML document. Example 1-12 gives a sample document.

<?xml version="1.0" ?>
 <methodCall>
  <methodName>lookupUPC</methodName>
  <params>
   <param><value><string>001441000055</string></value></param>
  </params>
 </methodCall>

This XML document is put into an envelope for transfer to the server. The envelope is an HTTP request with a method, URI, and headers (see Example 1-13). The XML document becomes the entity-body inside the HTTP envelope.

POST /rpc HTTP/1.1
Host: www.upcdatabase.com
User-Agent: XMLRPC::Client (Ruby 1.8.4)
Content-Type: text/xml; charset=utf-8
Content-Length: 158
Connection: keep-alive

<?xml version="1.0" ?>
<methodCall>
 <methodName>lookupUPC</methodName>
 ...
</methodCall>

The XML document changes depending on which method you’re calling, but the HTTP envelope is always the same. No matter what you do with the UPC database service, the URI is always http://www.upcdatabase.com/rpc and the HTTP method is always POST. Simply put, an XML-RPC service ignores most features of HTTP. It exposes only one URI (the “endpoint”), and supports only one method on that URI (POST).

Where a RESTful service would expose different URIs for different values of the scoping information, an RPC-style service typically exposes a URI for each “document processor”: something that can open the envelopes and transform them into software commands. For purposes of comparison, Example 1-14 shows what that code might look like if the UPC database were a RESTful web service.

require 'open-uri'
upc_data = open('http://www.upcdatabase.com/upc/00598491').read()
...

Here, the method information is contained in the HTTP method. The default HTTP method is GET, which is equivalent in this scenario to lookupUPC. The scoping information is contained in the URI. The hypothetical service exposes an enormous number of URIs: one for every possible UPC. By contrast, the HTTP envelope is empty: an HTTP GET request contains no entity-body at all.

For another example of a client for an RPC-style service, look back at Example 1-8. Google’s SOAP search API is an RPC-style service that uses SOAP as its envelope format.

A service that uses HTTP POST heavily or exclusively is probably an RPC-style service. Again, this isn’t a sure sign, but it’s a tip-off that the service isn’t very interested in putting its method information in the HTTP method. An otherwise RESTful service that uses HTTP POST a lot tends to move toward a REST-RPC hybrid architecture.

A few well-known examples of RPC-style web services:

This is a term I made up for describing web services that fit somewhere in between the RESTful web services and the purely RPC-style services. These services are often created by programmers who know a lot about real-world web applications, but not much about the theory of REST.

Take another look at this URI used by the Flickr web service: http://www.flickr.com/services/rest?api_key=xxx&method=flickr.photos.search&tags=penguin. Despite the “rest” in the URI, this was clearly designed as an RPC-style service, one that uses HTTP as its envelope format. It’s got the scoping information (“photos tagged ‘penguin’”) in the URI, just like a RESTful resource-oriented service. But the method information (“search for photos”) also goes in the URI. In a RESTful service, the method information would go into the HTTP method (GET), and whatever was leftover would become scoping information. As it is, this service is simply using HTTP as an envelope format, sticking the method and scoping information wherever it pleases. This is an RPC-style service. Case closed.

Except…look at Example 1-15.

GET services/rest?api_key=xxx&method=flickr.photos.search&tags=penguin HTTP/1.1
Host: www.flickr.com

That’s the HTTP request a client makes when remotely calling this procedure. Now it looks like the method information is in the HTTP method. I’m sending a GET request to get something. What am I getting? A list of search results for photos tagged “penguin.” What used to look like method information (“photoSearch()”) now looks like scoping information (“photos/tag/penguin”). Now the web service looks RESTful.

This optical illusion happens when an RPC-style service uses plain old HTTP as its envelope format, and when both the method and the scoping information happen to live in the URI portion of the HTTP request. If the HTTP method is GET, and the point of the web service request is to “get” information, it’s hard to tell whether the method information is in the HTTP method or in the URI. Look at the HTTP requests that go across the wire and you see the requests you’d see for a RESTful web service. They may contain elements like “method=flickr.photos.search” but that could be interpreted as scoping information, the way “photos/” and “search/” are scoping information. These RPC-style services have elements of RESTful web services, more or less by accident. They’re only using HTTP as a convenient envelope format, but they’re using it in a way that overlaps with what a RESTful service might do.

Many read-only web services qualify as entirely RESTful and resource-oriented, even though they were designed in the RPC style! But if the service allows clients to write to the data set, there will be times when the client uses an HTTP method that doesn’t match up with the true method information. This keeps the service from being as RESTful as it could be. Services like these are the ones I consider to be REST-RPC hybrids.

Here’s one example. The Flickr web API asks clients to use HTTP GET even when they want to modify the data set. To delete a photo you make a GET request to a URI that includes method=flickr.photos.delete. That’s just not what GET is for, as I’ll show in Chapter 5. The Flickr web API is a REST-RPC hybrid: RESTful when the client is retrieving data through GET, RPC-style when the client is modifying the data set.

A few well-known examples of REST-RPC hybrid services include:

From a design standpoint, I don’t think anybody sets out to to design a service as a REST-RPC hybrid. Because of the way HTTP works, any RPC-style service that uses plain HTTP and exposes multiple URIs tends to end up either RESTful or hybrid. Many programmers design web services exactly as they’d design web applications, and end up with hybrid services.

The existence of hybrid architectures has caused a lot of confusion. The style comes naturally to people who’ve designed web applications, and it’s often claimed that hybrid architectures are RESTful: after all, they work “the same way” as the human web. A lot of time has been spent trying to distinguish RESTful web services from these mysterious others. My classification of the “others” as REST-RPC hybrids is just the latest in a long line of neologisms. I think this particular neologism is the most accurate and useful way to look at these common but baffling services. If you’ve encountered other ways of describing them (“HTTP+POX” is the most popular at the time of writing), you might want read on, where I explain those other phrases in terms of what I’m saying in this book.

In the previous sections I claimed that all static web sites are RESTful. I claimed that web applications fall into one of the three categories, the majority being REST-RPC hybrids. Since the human web is made entirely of static web sites and web applications, this means that the entire human web is also on the programmable web! By now this should not be surprising to you. A web browser is a software program that makes HTTP requests and processes the responses somehow (by showing them to a human). That’s exactly what a web service client is. If it’s on the Web, it’s a web service.

My goal in this book is not to make the programmable web bigger. That’s almost impossible: the programmable web already encompasses nearly everything with an HTTP interface. My goal is to help make the programmable web better: more uniform, better-structured, and using the features of HTTP to greatest advantage.

All web services use HTTP, but they use it in different ways. A request to a RESTful web service puts the method information in the HTTP method and the scoping information in the URI. RPC-style web services tend to ignore the HTTP method, looking for method and scoping information in the URI, HTTP headers, or entity-body. Some RPC-style web services use HTTP as an envelope containing a document, and others only use it as an unlabelled envelope containing another envelope.

Again, all web services use URIs, but in different ways. What I’m about to say is a generalization, but a fairly accurate one. A RESTful, resource-oriented service exposes a URI for every piece of data the client might want to operate on. A REST-RPC hybrid exposes a URI for every operation the client might perform: one URI to fetch a piece of data, a different URI to delete that same data. An RPC-style service exposes one URI for every processes capable of handling Remote Procedure Calls (RPC). There’s usually only one such URI: the service “endpoint.”

A few, mostly legacy, web services use XML-RPC on top of HTTP. XML-RPC is a data structure format for representing function calls and their return values. As the name implies, it’s explicitly designed to use an RPC style.

Lots of web services use SOAP on top of HTTP. SOAP is an envelope format, like HTTP, but it’s an XML-based envelope format.

Now I’m going to say something controversial. To a first approximation, every current web service that uses SOAP also has an RPC architecture. This is controversial because many SOAP programmers think the RPC architecture is déclassé and prefer to call their services “message-oriented” or “document-oriented” services.

Well, all web services are message-oriented, because HTTP itself is message-oriented. An HTTP request is just a message: an envelope with a document inside. The question is what that document says. SOAP-based services ask the client to stick a second envelope (a SOAP document) inside the HTTP envelope. Again, the real question is what it says inside the envelope. A SOAP envelope can contain any XML data, just as an HTTP envelope can contain any data in its entity-body. But in every existing SOAP service, the SOAP envelope contains a description of an RPC call in a format similar to that of XML-RPC.

There are various ways of shuffling this RPC description around and giving it different labels—“document/literal” or “wrapped/literal”—but any way you slice it, you have a service with a large vocabulary of method information, a service that looks for scoping information inside the document rather than on the envelope. These are defining features of the RPC architecture.

I emphasize that this is not a fact about SOAP, just a fact about how it’s currently used. SOAP, like HTTP, is just a way of putting data in an envelope. Right now, though, the only data that ever gets put in that envelope is XML-RPC-esque data about how to call a remote function, or what’s the return value from such a function. I argue this point in more detail in Chapter 10.

These standards define special XML “stickers” for the SOAP envelope. The stickers are analagous to HTTP headers.

The Web Service Description Language (WSDL) is an XML vocabulary used to describe SOAP-based web services. A client can load a WSDL file and know exactly which RPC-style methods it can call, what arguments those methods expect, and which data types they return. Nearly every SOAP service in existence exposes a WSDL file, and most SOAP services would be very difficult to use without their WSDL files serving as guides. As I discuss in Chapter 10, WSDL bears more responsiblity than any other technology for maintaining SOAP’s association with the RPC style.

The Web Application Description Language (WADL) is an XML vocabulary used to describe RESTful web services. As with WSDL, a generic client can load a WADL file and be immediately equipped to access the full functionality of the corresponding web service. I discuss WADL in Chapter 9.

Since RESTful services have simpler interfaces, WADL is not nearly as necessary to these services as WSDL is to RPC-style SOAP services. This is a good thing, since as of the time of writing there are few real web services providing official WADL files. Yahoo!’s web search service is one that does.

Service-Oriented Architecture

This is a big industry buzzword. I’m not going to dwell on it for two reasons. First, the term is not very well defined. Second, to the extent that it is defined, it means something like: “a software architecture based on the production and consumption of web services.” In this book I talk about the design of individual services. A book on service-oriented architecture should work on a slightly higher level, showing how to use services as software components, how to integrate them into a coherent whole. I don’t cover that sort of thing in this book.

SOAP as a competitor to REST

If you get involved with web service debates you’ll hear this one a lot. You won’t hear it here because it gives the wrong impression. The primary competitors to RESTful architectures are RPC architectures, not specific technologies like SOAP. It is true that basically every SOAP service that now exists has an RPC architecture, but SOAP is just a way of putting a document in an envelope with stickers on it, like HTTP. SOAP is tied to the RPC architecture mainly by historical contingency and the current generation of automated tools.

There is a real tension here, but it’s not one I’ll cover much in this book. Roughly speaking, it’s the tension between services that put their documents in a SOAP envelope and then an HTTP envelope; and services that only use the HTTP envelope.

HTTP+POX

Stands for HTTP plus Plain Old XML. This term covers roughly those services I call REST-RPC hybrid services. They overlap with RESTful designs, especially when it comes to retrieving data, but their basic architecture is RPC-oriented.

I don’t like this term because Plain Old XML is inaccurate. The interesting thing about these services is not that they produce plain old XML documents (as opposed to XML documents wrapped in SOAP envelopes). Some of these services don’t serve XML at all: they serve JSON, plain text, or binary files. No, the interesting thing about these services is their RPC architecture. That’s what puts them in opposition to REST.

STREST

Means Service-Trampled REST. This is another term for REST-RPC hybrid architectures. It’s more accurate than HTTP+POX since it conveys the notion of a RESTful architecture taken over by something else: in this case, the RPC style.

This is a cute acronym but I don’t like it, because it buys into a myth that the only true web services are RPC-style services. After all, the service that trampled your REST was an RPC service. If you think that REST services are real services, it doesn’t make sense to cry “Help! I had some REST but then this Service got into it!” RPC-Trampled REST would be more accurate, but that’s a lousy acronym.

High and low REST

Yet another way of distinguishing between truly RESTful services and the ones I call REST-RPC hybrids. High REST services are just those that adhere closely to the Fielding dissertation. Among other things, they put method information in the HTTP method and scoping information in the URI. Low REST services are presumed to have deviated. Since low REST services tend to deviate from orthodoxy in a particular direction (toward the RPC style), I prefer a more specific terminology.

• http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=restbook&query=jellyfish

• http://search.yahoo.com/search?p=jellyfish

1. Come up with the data that will go into the HTTP request: the HTTP method, the URI, any HTTP headers, and (for requests using the PUT or POST method) any document that needs to go in the request’s entity-body.

2. Format the data as an HTTP request, and send it to the appropriate HTTP server.

3. Parse the response data—the response code, any headers, and any entity-body—into the data structures the rest of your program needs.

Although a web service request is just an HTTP request, any given web service has a logic and a structure that is missing from the World Wide Web as a whole. If you follow the three-step algorithm every time you make a web service request, your code will be a mess and you’ll never take advantage of that underlying structure.

Instead, as a smart programmer you’ll quickly notice the patterns underlying your requests to a given service, and write wrapper methods that abstract away the details of HTTP access. The print_page_titles method defined in Example 2-1 is a primitive wrapper. As a web service gets popular, its users release polished wrapper libraries in various languages. Some service providers offer official wrappers: Amazon gives away clients in five different languages for its RESTful S3 service. That hasn’t stopped outside programmers from writing their own S3 client libraries, like jbucket and s3sh.

Wrappers make service programming easy, because the API of a wrapper library is tailored to one particular service. You don’t have to think about HTTP at all. The downside is that each wrapper is slightly different: learning one wrapper doesn’t prepare you for the next one.

This is a little disappointing. After all, these services are just variations on the three-step algorithm for making HTTP requests. Shouldn’t there be some way of abstracting out the differences between services, some library that can act as a wrapper for the entire space of RESTful and hybrid services?

This is the problem of service description. We need a language with a vocabulary that can describe the variety of RESTful and hybrid services. A document written in this language could script a generic web service client, making it act like a custom-written wrapper. The SOAP RPC community has united around WSDL as its service description language. The REST community has yet to unite around a description language, so in this book I do my bit to promote WADL as a resource-oriented alternative to WSDL. I think it’s the simplest and most elegant solution that solves the whole problem. I show a simple WADL client in this chapter and it is covered in detail in the WADL” section.

There’s also a generic client called ActiveResource, still in development. ActiveResource makes it easy to write clients for many kinds of web services written with the Ruby on Rails framework. I cover ActiveResource at the end of Chapter 3.

• The web site is rooted at http://del.icio.us/ and the web service is rooted at https://api.del.icio.us/v1/. The web site communicates with clients through HTTP, the web service uses secure HTTPS.

• The web site and the web service expose different URI structures. To get your recent bookmarks from the web site, you fetch https://del.icio.us/{your-username}. To get your recent bookmarks from the web service, you fetch https://api.del.icio.us/v1/posts/recent.

• The web site serves HTML documents, and the web service serves XML documents. The formats are different, but they contain the same data.

• The web site lets you see a lot of information without logging in or even having an account. The web service makes you authenticate for every request.

• Both offer features for personal bookmark management, but the web site also has social features. On the web site, you can see lists of URIs other people have bookmarked, lists of people who have bookmarked a particular URI, lists of URIs tagged with a certain tag, and lists of popular bookmarks. The web service only lets you see your own bookmarks.

In the sections that follow, I show you simple del.icio.us clients in a variety of programming languages. All of these clients do exactly the same thing, and it’s worth spelling out what that is. First, they open up a TCP/IP socket connection to port 443 (the standard HTTPS port) on the server at api.del.icio.us. Then they send something like the HTTP request in Example 2-2. Like all HTTP responses, this one has three parts: a status code, a set of headers, and an entity-body. In this case, the entity-body is an XML document.

GET /v1/posts/recent HTTP/1.1
Host: api.del.icio.us
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

The del.icio.us web service sends back something like the HTTP response in Example 2-3, then closes the socket connection.

200 OK
Content-Type: text/xml
Date: Sun, 29 Oct 2006 15:09:36 GMT
Connection: close

<?xml version='1.0' standalone='yes'?>
<posts tag="" user="username">
  <post href="http://www.foo.com/" description="foo" extended="" 
   hash="14d59bdc067e3c1f8f792f51010ae5ac" tag="foo"
   time="2006-10-29T02:56:12Z" />
  <post href="http://amphibians.com/" description="Amphibian Mania" 
   extended="" hash="688b7b2f2241bc54a0b267b69f438805" tag="frogs toads" 
   time="2006-10-28T02:55:53Z" />
</posts>

The clients I write are only interested in the entity-body part. Specifically, they’re only interested in the href and description attributes of the post tags. They’ll parse the XML document into a data structure and use the XPath expression /posts/post to iterate over the post tags. They’ll print to standard output the href and description attribute of every del.icio.us bookmark:

foo: http://www.foo.com/
Amphibian Mania: http://amphibians.com/

To use any of these clients, you’ll need to create a del.icio.us account. Some of the clients hard-code the strings “username” and “password”; you’ll need to substitute your del.icio.us username and password.

• It must support HTTPS and SSL certificate validation. Web services, like web sites, use HTTPS to secure communication with their clients. Many web services (del.icio.us is one example) won’t accept plain HTTP requests at all. A library’s HTTPS support often depends on the presense of an external SSL library written in C.

• It must support at least the five main HTTP methods: GET, HEAD, POST, PUT, and DELETE. Some libraries support only GET and POST. Others are designed for simplicity and support only GET.

  You can get pretty far with a client that only supports GET and POST: HTML forms support only those two methods, so the entire human web is open to you. You can even do all right with just GET, because many web services (among them del.icio.us and Flickr) use GET even where they shouldn’t. But if you’re choosing a library for all your web service clients, or writing a general client like a WADL client, you need a library that supports all five methods. Additional methods like OPTIONS and TRACE, and WebDAV extensions like MOVE, are a bonus.

• It must allow the programmer to customize the data sent as the entity-body of a PUT or POST request.

• It must allow the programmer to customize a request’s HTTP headers.

• It must give the programmer access to the response code and headers of an HTTP response; not just access to the entity-body.

• It must be able to communicate through an HTTP proxy. The average programmer may not think about this, but many HTTP clients in corporate environments can only work through a proxy. Intermediaries like HTTP proxies are also a standard part of the REST meta-architecture, though not one I’ll be covering in much detail.

There are also some features of an HTTP library that make life easier as you write clients for RESTful and hybrid services. These features mostly boil down to knowledge about HTTP headers, so they’re technically optional. You can implement them yourself so long as your library gives you access to request and response HTTP headers. The advantage of library support is that you don’t have to worry about the details.

When you’re writing code against a specific service, you may be able to do without some or all of these features. Ruby’s standard open-uri library only supports GET requests. If you’re writing a client for del.icio.us, there’s no problem, since that web service expects only GET requests. But try to use open-uri with Amazon S3 (which uses GET, HEAD, PUT, and DELETE), and you’ll quickly run into a wall. In the next sections I recommend good HTTP client libraries for some popular programming languages.

Ruby comes with two HTTP client libraries, open-uri and the lower-level net/http. Either can make HTTPS requests if you’ve got the net/https extension installed. Windows installations of Ruby should be able to make HTTPS requests out of the box. If you’re not on Windows, you may have to install net/https separately.^[8]

The open-uri library has a simple and elegant interface that lets you treat URIs as filenames. To read a web page, you simply open its URI and read data from the “filehandle.” You can pass in a hash to open containing custom HTTP headers and open-specific keyword arguments. This lets you set up a proxy, or specify authentication information.

Unfortunately, right now open-uri only supports one HTTP method: GET. That’s why I’ve made some minor modifications to open-uri and made the result available as the rest-open-uri Ruby gem.^[9] I’ve added two keyword arguments to open: method, which lets you customize the HTTP method, and :body, which lets you send data in the entity-body.

Example 2-4 is an implementation of the standard del.icio.us example using the open-uri library (rest-open-uri works the same way). This code parses the response document using the REXML::Document parser, which you’ve seen before.

#!/usr/bin/ruby -w
# delicious-open-uri.rb


require 'open-uri'
require 'rexml/document'

# Fetches a del.icio.us user's recent bookmarks, and prints each one.
def print_my_recent_bookmarks(username, password)
  # Make the HTTPS request.
  response = open('https://api.del.icio.us/v1/posts/recent',
                  :http_basic_authentication => [username, password])

  # Read the response entity-body as an XML document.
  xml = response.read

  # Turn the document into a data structure.
  document = REXML::Document.new(xml)

  # For each bookmark...
  REXML::XPath.each(document, "/posts/post") do |e|
    # Print the bookmark's description and URI
    puts "#{e.attributes['description']}: #{e.attributes['href']}"
  end
end

# Main program
username, password = ARGV
unless username and password
  puts "Usage: #{$0} [username] [password]"
  exit
end
print_my_recent_bookmarks(username, password)

I mentioned earlier that Ruby’s stock open-uri can only make HTTP GET requests. For many purposes, GET is enough, but if you want to write a Ruby client for a fully RESTful service like Amazon’s S3, you’ll either need to use rest-open-uri, or turn to Ruby’s low-level HTTP library: net/http.

This built-in library provides the Net::HTTP class, which has several methods for making HTTP requests (see Table 2-1). You can build a complete HTTP client out of this class, using nothing more than the Ruby standard library. In fact, open-uri and rest-open-uri are based on Net::HTTP. Those libraries only exist because Net::HTTP provides no simple, easy-to-use interface that supports all the features a REST client needs (proxies, HTTPS, headers, and so on). That’s why I recommend you use rest-open-uri.

The Python standard library comes with two HTTP clients: urllib2, which has a file-like interface like Ruby’s open-uri; and httplib, which works more like Ruby’s Net::HTTP. Both offer transparent support for HTTPS, assuming your copy of Python was compiled with SSL support. There’s also an excellent third-party library, Joe Gregorio’s httplib2, which is the one I recommend in general. httplib2 is an excellent piece of software, supporting nearly every feature on my wish list—most notably, transparent caching. Table 2-2 lists the features available in each library.

Example 2-5 is a del.icio.us client that uses httplib2. It uses the ElementTree library to parse the del.icio.us XML.

#!/usr/bin/python2.5
# delicious-httplib2.py
import sys
from xml.etree import ElementTree
import httplib2

# Fetches a del.icio.us user's recent bookmarks, and prints each one.
def print_my_recent_bookmarks(username, password):
    client = httplib2.Http(".cache")
    client.add_credentials(username, password)

    # Make the HTTP request, and fetch the response and the entity-body.
    response, xml = client.request('https://api.del.icio.us/v1/posts/recent')

    # Turn the XML entity-body into a data structure.
    doc = ElementTree.fromstring(xml)

    # Print information about every bookmark.
    for post in doc.findall('post'):
        print "%s: %s" % (post.attrib['description'], post.attrib['href'])

# Main program
if len(sys.argv) != 3:    
    print "Usage: %s [username] [password]" % sys.argv[0]
    sys.exit()

username, password = sys.argv[1:]
print_my_recent_bookmarks(username, password)

The Java standard library comes with an HTTP client, java.net.HttpURLConnection. You can get an instance by calling open on a java.net.URL object. Though it supports most of the basic features of HTTP, programming to its API is very difficult. The Apache Jakarta project has a competing client called HttpClient, which has a better design. There’s also Restlet. I cover Restlet as a server library in Chapter 12, but it’s also an HTTP client library. The class org.restlet.Client makes it easy to make simple HTTP requests, and the class org.restlet.data.Request hides the HttpURLConnection programming necessary to make more complex requests. Table 2-3 lists the features available in each library.

Example 2-6 is a Java client for del.icio.us that uses HttpClient. It works in Java 1.5 and up, and it’ll work in previous versions if you install the Xerces parser (see Java: javax.xml, Xerces, or XMLPull” later in this chapter).

// DeliciousApp.java
import java.io.*;

import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.GetMethod;

import org.w3c.dom.*;
import org.xml.sax.SAXException;
import javax.xml.parsers.*;
import javax.xml.xpath.*;

/**
 * A command-line application that fetches bookmarks from del.icio.us
 * and prints them to standard output.
 */
public class DeliciousApp
{
  public static void main(String[] args)
    throws HttpException, IOException, ParserConfigurationException,
           SAXException, XPathExpressionException
  {        
    if (args.length != 2)
    {
      System.out.println("Usage: java -classpath [CLASSPATH] "
                         + "DeliciousApp [USERNAME] [PASSWORD]");
      System.out.println("[CLASSPATH] - Must contain commons-codec, " +
                         "commons-logging, and commons-httpclient");
      System.out.println("[USERNAME]  - Your del.icio.us username");
      System.out.println("[PASSWORD]  - Your del.icio.us password");
      System.out.println();

      System.exit(-1);
    }

    // Set the authentication credentials.
    Credentials creds = new UsernamePasswordCredentials(args[0], args[1]);
    HttpClient client = new HttpClient();
    client.getState().setCredentials(AuthScope.ANY, creds);

    // Make the HTTP request.
    String url = "https://api.del.icio.us/v1/posts/recent";
    GetMethod method = new GetMethod(url);
    client.executeMethod(method);
    InputStream responseBody = method.getResponseBodyAsStream();

    // Turn the response entity-body into an XML document.
    DocumentBuilderFactory docBuilderFactory =
      DocumentBuilderFactory.newInstance();
    DocumentBuilder docBuilder = 
      docBuilderFactory.newDocumentBuilder();
    Document doc = docBuilder.parse(responseBody);
    method.releaseConnection();

    // Hit the XML document with an XPath expression to get the list
    // of bookmarks.
    XPath xpath = XPathFactory.newInstance().newXPath();        
    NodeList bookmarks = (NodeList)xpath.evaluate("/posts/post", doc,
                                                  XPathConstants.NODESET);

    // Iterate over the bookmarks and print out each one.
    for (int i = 0; i < bookmarks.getLength(); i++)
    {
       NamedNodeMap bookmark = bookmarks.item(i).getAttributes();
       String description = bookmark.getNamedItem("description")
           .getNodeValue();
       String uri = bookmark.getNamedItem("href").getNodeValue();
       System.out.println(description + ": " + uri);
    }

    System.exit(0);
  }
}

The .NET Common Language Runtime (CLR) defines HTTPWebRequest for making HTTP requests, and NetworkCredential for authenticating the client to the server. The HTTPWebRequest constructor takes a URI. The NetworkCredential constructor takes a username and password (see Example 2-7).

using System;
using System.IO;
using System.Net;
using System.Xml.XPath;

public class DeliciousApp {
    static string user = "username";
    static string password = "password";
    static Uri uri = new Uri("https://api.del.icio.us/v1/posts/recent");

    static void Main(string[] args) {
        HttpWebRequest request = (HttpWebRequest) WebRequest.Create(uri);
        request.Credentials = new NetworkCredential(user, password);
        HttpWebResponse response = (HttpWebResponse) request.GetResponse();

        XPathDocument xml = new
	    XPathDocument(response.GetResponseStream());
        XPathNavigator navigator = xml.CreateNavigator();
        foreach (XPathNavigator node in navigator.Select("/posts/post")) {
          string description = node.GetAttribute("description","");
          string href = node.GetAttribute("href","");
          Console.WriteLine(description + ": " + href);
        }
    }
}

PHP comes with a binding to the C library libcurl, which can do pretty much anything you might want to do with a URI (see Example 2-8).

<?php
  $user = "username";
  $password = "password";

  $request = curl_init();
  curl_setopt($request, CURLOPT_URL,
              'https://api.del.icio.us/v1/posts/recent');
  curl_setopt($request, CURLOPT_USERPWD, "$user:$password");
  curl_setopt($request, CURLOPT_RETURNTRANSFER, true);

  $response = curl_exec($request);
  $xml = simplexml_load_string($response);
  curl_close($request);

  foreach ($xml->post as $post) {
    print "$post[description]: $post[href]\n";
  }
?>

If you’re writing a web service client in JavaScript, you probably intend it to run inside a web browser as part of an Ajax application. All modern web browsers implement a HTTP client library for JavaScript called XMLHttpRequest.

Because Ajax clients are developed differently from standalone clients, I’ve devoted an entire chapter to them: Chapter 11. The first example in that chapter is a del.icio.us client, so you can skip there right now without losing the flow of the examples.

This example is a bit different: it doesn’t use a programming language at all. A program called curl is a capable HTTP client that runs from the Unix or Windows command line. It supports most HTTP methods, custom headers, several authentication mechanisms, proxies, compression, and many other features. You can use curl to do quick one-off HTTP requests, or use it in conjunction with shell scripts. Here’s curl in action, grabbing a user’s del.icio.us bookmarks:

$ curl https://username:password@api.del.icio.us/v1/posts/recent
<?xml version='1.0' standalone='yes'?>
<posts tag="" user="username">
 ...
</posts>

I don’t have the space or the expertise to cover every popular programming language in depth with a del.icio.us client example. I can, however, give brief pointers to HTTP client libraries for some of the many languages I haven’t covered yet.

Ruby comes with a standard XML parser library, REXML, that supports both DOM and SAX interfaces, and has good XPath support. Unfortunately, REXML’s internals put it in a strange middle ground: it’s too strict to be used to parse bad XML, but not strict enough to reject all bad XML.

I use REXML throughout this book because it’s the default choice, and because I only deal with well-formed XML. If you want to guarantee that you only deal with well-formed XML, you’ll need to install the Ruby bindings to the GNOME project’s libxml2 library (described in Other Languages” later in this chapter).

If you want to be able to handle bad markup, the best choice is hpricot, available as the hpricot gem. It’s fast (it uses a C extension), and it has an intuitive interface including support for common XPath expressions.

Example 2-9 is an implementation of the del.icio.us client using REXML’s SAX interface.

#!/usr/bin/ruby -w
# delicious-sax.rb
require 'open-uri'
require 'rexml/parsers/sax2parser'

def print_my_recent_bookmarks(username, password)
  # Make an HTTPS request and read the entity-body as an XML document.
  xml = open('https://api.del.icio.us/v1/posts/recent',
             :http_basic_authentication => [username, password])

  # Create a SAX parser whose destiny is to parse the XML entity-body.
  parser = REXML::Parsers::SAX2Parser.new(xml)

  # When the SAX parser encounters a 'post' tag...
  parser.listen(:start_element, ["post"]) do |uri, tag, fqtag, attributes|
    # ...it should print out information about the tag.
    puts "#{attributes['description']}: #{attributes['href']}"
  end

  # Make the parser fulfil its destiny to parse the XML entity-body.
  parser.parse 
end

# Main program.
username, password = ARGV
unless username and password
  puts "Usage: #{$0} [USERNAME] [PASSWORD]"  
  exit
end
print_my_recent_bookmarks(username, password)

In this program, the data isn’t parsed (or even read from the HTTP connection) until the call to SAXParser#parse. Up to that point I’m free to call listen and set up pieces of code to run in response to parser events. In this case, the only event I’m interested in is the start of a post tag. My code block gets called every time the parser finds a post tag. This is the same as parsing the XML document with a tree-style parser, and running the XPath expression “//post” against the object tree. What does my code block do? The same thing my other example programs do when they find a post tag: print out the values of the description and href attributes.

This implementation is faster and much more memory-efficient than the equivalent tree-style implementation. However, complex SAX-based programs are much more difficult to write than equivalent tree-style programs. Pull parsers are a good compromise. Example 2-10 shows a client implementation that uses REXML’s pull parser interface.

#!/usr/bin/ruby -w
# delicious-pull.rb
require 'open-uri'
require 'rexml/parsers/pullparser'

def print_my_recent_bookmarks(username, password)
  # Make an HTTPS request and read the entity-body as an XML document.
  xml = open('https://api.del.icio.us/v1/posts/recent',
             :http_basic_authentication => [username, password])

  # Feed the XML entity-body into a pull parser
  parser = REXML::Parsers::PullParser.new(xml)

  # Until there are no more events to pull...
  while parser.has_next?
    # ...pull the next event.
    tag = parser.pull    
    # If it's a 'post' tag...
    if tag.start_element?
      if tag[0] == 'post'       
        # Print information about the bookmark.
        attrs = tag[1]
        puts "#{attrs['description']}: #{attrs['href']}"
      end
    end
  end
end

# Main program.
username, password = ARGV
unless username and password
  puts "Usage: #{$0} [USERNAME] [PASSWORD]"  
  exit
end
print_my_recent_bookmarks(username, password)

The world is full of XML parsers for Python. There are seven different XML interfaces in the Python 2.5 standard library alone. For full details, see the Python library reference.

For tree-style parsing, the best library is ElementTree. It’s fast, it has a sensible interface, and as of Python 2.5 you don’t have to install anything because it’s in the standard library. On the downside, its support for XPath is limited to simple expressions—of course, nothing else in the standard library supports XPath at all. If you need full XPath support, try 4Suite.

Beautiful Soup is a slower tree-style parser that is very forgiving of invalid XML, and offers a programmatic interface to a document. It also handles most character set conversions automatically, letting you work with Unicode data.

For SAX-style parsing, the best choice is the xml.sax module in the standard library. The PyXML suite includes a pull parser.

Java 1.5 includes the XML parser written by the Apache Xerces project. The core classes are found in the packages javax.xml.*, (for instance, javax.xml.xpath). The DOM interface lives in org.w3c.dom.*, and the SAX interface lives in org.xml.sax.*. If you’re using a previous version of Java, you can install Xerces yourself and take advantage of the same interface found in Java 1.5 (http://xerces.apache.org/xerces2-j/).

There are a variety of pull parsers for Java. Sun’s Web Services Developer Pack includes a pull parser in the javax.xml.stream package.

For parsing bad XML, you might try TagSoup.

The.NET Common Language Runtime comes with a pull parser interface, in contrast to the more typical (and more complex) SAX-style interface. You can also create a full W3C DOM tree using XmlDocument. The XPathDocument class lets you iterate over nodes in the tree that match an XPath expression.

If you need to handle broken XML documents, check out Chris Lovett’s SgmlReader at http://www.gotdotnet.com/Community/UserSamples/.

You can create a SAX-style parser with the function xml_parser_create, and a pull parser with the XMLReader extension. The DOM PHP extension (included in PHP 5) provides a tree-style interface to the GNOME project’s libxml2 C library. You might have an easier time using SimpleXML, a tree-style parser that’s not an official DOM implementation. That’s what I used in Example 2-8.

There’s also a pure PHP DOM parser called DOMIT!.

If you’re using XMLHttpRequest to write an Ajax client, you don’t have to worry about the XML parser at all. If you make a request and the response entity-body is in XML format, the web browser parses it with its own tree-style parser, and makes it available through the responseXML property of the XMLHttpRequest object. You manipulate this document with JavaScript DOM methods: the same ones you use to manipulate HTML documents displayed in the browser. Chapter 11 has more information on how to use responseXML—and how to handle non-XML documents with the responseData member.

There’s a third-party XML parser, XML for <SCRIPT>, which works independently of the parser built into the client’s web browser. “XML for <SCRIPT>” offers DOM and SAX interfaces, and supports XPath queries.

Backup server

You store your data through S3 and don’t give anyone else access to it. Rather than buying your own backup disks, you’re renting disk space from Amazon.

Data host

You store your data on S3 and give others access to it. Amazon serves your data through HTTP or BitTorrent. Rather than paying an ISP for bandwidth, you’re paying Amazon. Depending on your existing bandwidth costs this can save you a lot of money. Many of today’s web startups use S3 to serve data files.

A bucket has one piece of information associated with it: the name. A bucket name can only contain the characters A through Z, a through z, 0 through 9, underscore, period, and dash. I recommend staying away from uppercase letters in bucket names.

As I mentioned above, buckets cannot contain other buckets: only objects. Each S3 user is limited to 100 buckets, and your bucket name cannot conflict with anyone else’s. I recommend you either keep everything in one bucket, or name each bucket after one of your projects or domain names.

An object has four parts to it:

If I wanted to host the O’Reilly web site on S3, I’d create a bucket called “oreilly.com,” and fill it with objects whose keys were “” (the empty string), “catalog,” “catalog/9780596529260,” and so on. These objects correspond to the URIs http://oreilly.com/, http://oreilly.com/catalog, and so on. The object’s values would be the HTML contents of O’Reilly’s web pages. These S3 objects would have their Content-Type metadata value set to text/html, so that people browsing the site would be served these objects as HTML documents, as opposed to XML or plain text.

If S3 was implemented as an object-oriented code library instead of a web service, you’d have two classes S3Bucket and S3Object. They’d have getter and setter methods for their data members: S3Bucket#name, S3Object.value=, S3Bucket#addObject, and the like. The S3Bucket class would have an instance method S3Bucket#getObjects that returned a list of S3Object instances, and a class method S3Bucket.getBuckets that returned all of your buckets. Example 3-1 shows what the Ruby code for this class might look like.

class S3Bucket
  # A class method to fetch all of your buckets.
  def self.getBuckets
  end

  # An instance method to fetch the objects in a bucket.
  def getObjects
  end
  ...
end

class S3Object
  # Fetch the data associated with this object.
  def data
  end

  # Set the data associated with this object.
  def data=(new_value)
  end
  ...  
end

• The list of your buckets (https://s3.amazonaws.com/). There’s only one resource of this type.

• A particular bucket (https://s3.amazonaws.com/{name-of-bucket}/). There can be up to 100 resources of this type.

• A particular S3 object inside a bucket (https://s3.amazonaws.com/{name-of-bucket}/{name-of-object}). There can be infinitely many resources of this type.

Example 3-4 shows an object-oriented class for my first resource, the list of buckets. I’ll call the class for this resource S3::BucketList.

# The bucket list.
class BucketList
  include Authorized

  # Fetch all the buckets this user has defined.
  def get
    buckets = []

    # GET the bucket list URI and read an XML document from it.
    doc = REXML::Document.new(open(HOST).read)

    # For every bucket...
    REXML::XPath.each(doc, "//Bucket/Name") do |e|
      # ...create a new Bucket object and add it to the list.
      buckets << Bucket.new(e.text) if e.text
    end
    return buckets
  end
end

Now my file is a real web service client. If I call S3::BucketList#get I make a secure HTTP GET request to https://s3.amazonaws.com/, which happens to be the URI of the resource “a list of your buckets.” The S3 service sends back an XML document that looks something like Example 3-5. This is a representation (as I’ll start calling it in the next chapter) of the resource “a list of your buckets.” It’s just some information about the current state of that list. The Owner tag makes it clear whose bucket list it is (my AWS account name is evidently “leonardr28”), and the Buckets tag contains a number of Bucket tags describing my buckets (in this case, there’s one Bucket tag and one bucket).

<?xml version='1.0' encoding='UTF-8'?>
<ListAllMyBucketsResult xmlns='http://s3.amazonaws.com/doc/2006-03-01/'>
 <Owner>
  <ID>c0363f7260f2f5fcf38d48039f4fb5cab21b060577817310be5170e7774aad70</ID>
  <DisplayName>leonardr28</DisplayName>
 </Owner>
 <Buckets>
  <Bucket>
   <Name>crummy.com</Name>
   <CreationDate>2006-10-26T18:46:45.000Z</CreationDate>
  </Bucket>
 </Buckets>
</ListAllMyBucketsResult>

For purposes of this small client application, the Name is the only aspect of a bucket I’m interested in. The XPath expression //Bucket/Name gives me the name of every bucket, which is all I need to create Bucket objects.

As we’ll see, one thing that’s missing from this XML document is links. The document gives the name of every bucket, but says nothing about where the buckets can be found on the Web. In terms of the REST design criteria, this is the major shortcoming of Amazon S3. Fortunately, it’s not too difficult to program a client to calculate a URI from the bucket name. I just follow the rule I gave earlier: https://s3.amazonaws.com/{name-of-bucket}.

Now, let’s write the S3::Bucket class, so that S3::BucketList.get will have something to instantiate (Example 3-6).

# A bucket that you've stored (or will store) on the S3 application.
class Bucket
  include Authorized
  attr_accessor :name

  def initialize(name)
    @name = name
  end

  # The URI to a bucket is the service root plus the bucket name.
  def uri
    HOST + URI.escape(name)
  end

  # Stores this bucket on S3. Analagous to ActiveRecord::Base#save,
  # which stores an object in the database. See below in the
  # book text for a discussion of acl_policy.
  def put(acl_policy=nil)
    # Set the HTTP method as an argument to open(). Also set the S3
    # access policy for this bucket, if one was provided.
    args = {:method => :put}
    args["x-amz-acl"] = acl_policy if acl_policy

    # Send a PUT request to this bucket's URI.
    open(uri, args)
    return self
  end

  # Deletes this bucket. This will fail with HTTP status code 409
  # ("Conflict") unless the bucket is empty.
  def delete
    # Send a DELETE request to this bucket's URI.
    open(uri, :method => :delete)
  end

Here are two more web service methods: S3::Bucket#put and S3::Bucket#delete. Since the URI to a bucket uniquely identifies the bucket, deletion is simple: you send a DELETE request to the bucket URI, and it’s gone. Since a bucket’s name goes into its URI, and a bucket has no other settable properties, it’s also easy to create a bucket: just send a PUT request to its URI. As I’ll show when I write S3::Object, a PUT request is more complicated when not all the data can be stored in the URI.

Earlier I compared my S3:: classes to ActiveRecord classes, but S3::Bucket#put works a little differently from an ActiveRecord implementation of save. A row in an ActiveRecord-controlled database table has a numeric unique ID. If you take an ActiveRecord object with ID 23 and change its name, your change is reflected as a change to the database record with ID 23:

SET name="newname" WHERE id=23

The permanent ID of an S3 bucket is its URI, and the URI includes the name. If you change the name of a bucket and call put, the client doesn’t rename the old bucket on S3: it creates a new, empty bucket at a new URI with the new name. This is a result of design decisions made by the S3 programmers. It doesn’t have to be this way. The Ruby on Rails framework has a different design: when it exposes database rows through a RESTful web service, the URI to a row incorporates its numeric database IDs. If S3 was a Rails service you’d see buckets at URIs like /buckets/23. Renaming the bucket wouldn’t change the URI.

Now comes the last method of S3::Bucket, which I’ve called get. Like S3::BucketList.get, this method makes a GET request to the URI of a resource (in this case, a “bucket” resource), fetches an XML document, and parses it into new instances of a Ruby class (see Example 3-7). This method supports a variety of ways to filter the contents of S3 buckets. For instance, you can use :Prefix to retrieve only objects whose keys start with a certain string. I won’t cover these filtering options in detail. If you’re interested in them, see the S3 technical documentation on “Listing Keys.”

  # Get the objects in this bucket: all of them, or some subset.
  #
  # If S3 decides not to return the whole bucket/subset, the second
  # return value will be set to true. To get the rest of the objects,
  # you'll need to manipulate the subset options (not covered in the
  # book text).
  #
  # The subset options are :Prefix, :Marker, :Delimiter, :MaxKeys.
  # For details, see the S3 docs on "Listing Keys".
  def get(options={})
    # Get the base URI to this bucket, and append any subset options
    # onto the query string.
    uri = uri()
    suffix = '?'

    # For every option the user provided...
    options.each do |param, value|      
      # ...if it's one of the S3 subset options...
      if [:Prefix, :Marker, :Delimiter, :MaxKeys].member? :param
        # ...add it to the URI.
        uri << suffix << param.to_s << '=' << URI.escape(value)
        suffix = '&'
      end
    end

    # Now we've built up our URI. Make a GET request to that URI and
    # read an XML document that lists objects in the bucket.
    doc = REXML::Document.new(open(uri).read)
    there_are_more = REXML::XPath.first(doc, "//IsTruncated").text == "true"

    # Build a list of S3::Object objects.
    objects = []
    # For every object in the bucket...
    REXML::XPath.each(doc, "//Contents/Key") do |e|
      # ...build an S3::Object object and append it to the list.
      objects << Object.new(self, e.text) if e.text
    end
    return objects, there_are_more
  end
end

Make a GET request of the application’s root URI, and you get a representation of the resource “a list of your buckets.” Make a GET request to the URI of a “bucket” resource, and you get a representation of the bucket: an XML document like the one in Example 3-8, containing a Contents tag for every element of the bucket.

<?xml version='1.0' encoding='UTF-8'?>  
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Name>crummy.com</Name>
 <Prefix></Prefix>
 <Marker></Marker>
 <MaxKeys>1000</MaxKeys>
 <IsTruncated>false</IsTruncated>
 <Contents>
  <Key>mydocument</Key>
  <LastModified>2006-10-27T16:01:19.000Z</LastModified>
  <ETag>"93bede57fd3818f93eedce0def329cc7"</ETag>
  <Size>22</Size>
  <Owner>
   <ID>
    c0363f7260f2f5fcf38d48039f4fb5cab21b060577817310be5170e7774aad70</ID>
    <DisplayName>leonardr28</DisplayName>
  </Owner>
  <StorageClass>STANDARD</StorageClass>
 </Contents>
</ListBucketResult>

In this case, the portion of the document I find interesting is the list of a bucket’s objects. An object is identified by its key, and I use the XPath expression “//Contents/Key” to fetch that information. I’m also interested in a certain Boolean variable (“//IsTruncated”): whether this document contains keys for every object in the bucket, or whether S3 decided there were too many to send in one document and truncated the list.

Again, the main thing missing from this representation is links. The document lists lots of information about the objects, but not their URIs. The client is expected to know how to turn an object name into that object’s URI. Fortunately, it’s not too hard to build an object’s URI, using the rule I already gave: https://s3.amazonaws.com/{name-of-bucket}/{name-of-object}.

Now we’re ready to implement an interface to the core of the S3 service: the object. Remember that an S3 object is just a data string that’s been given a name (a key) and a set of metadata key-value pairs (such as Content-Type="text/html"). When you send a GET request to the bucket list, or to a bucket, S3 serves an XML document that you have to parse. When you send a GET request to an object, S3 serves whatever data string you PUT there earlier—byte for byte.

Example 3-9 shows the beginning of S3::Object, which should be nothing new by now.

# An S3 object, associated with a bucket, containing a value and metadata.
class Object
  include Authorized

  # The client can see which Bucket this Object is in.
  attr_reader :bucket
  
  # The client can read and write the name of this Object.
  attr_accessor :name

  # The client can write this Object's metadata and value.
  # I'll define the corresponding "read" methods later.
  attr_writer :metadata, :value

  def initialize(bucket, name, value=nil, metadata=nil)
    @bucket, @name, @value, @metadata = bucket, name, value, metadata
  end

  # The URI to an Object is the URI to its Bucket, and then its name.
  def uri
    @bucket.uri + '/' + URI.escape(name)
  end

What comes next is my first implementation of an HTTP HEAD request. I use it to fetch an object’s metadata key-value pairs and populate the metadata hash with it (the actual implementation of store_metadata comes at the end of this class). Since I’m using rest-open-uri, the code to make the HEAD request looks the same as the code to make any other HTTP request (see Example 3-10).

  # Retrieves the metadata hash for this Object, possibly fetching
  # it from S3.
  def metadata
    # If there's no metadata yet...
    unless @metadata
      # Make a HEAD request to this Object's URI, and read the metadata
      # from the HTTP headers in the response.
      begin
        store_metadata(open(uri, :method => :head).meta) 
      rescue OpenURI::HTTPError => e
        if e.io.status == ["404", "Not Found"]
          # If the Object doesn't exist, there's no metadata and this is not
          # an error. 
          @metadata = {}
        else
          # Otherwise, this is an error.
          raise e
        end
      end

    end
    return @metadata
  end

The goal here is to fetch an object’s metadata without fetching the object itself. This is the difference between downloading a movie review and downloading the movie, and when you’re paying for the bandwidth it’s a big difference. This distinction between metadata and representation is not unique to S3, and the solution is general to all resource-oriented web services. The HEAD method gives any client a way of fetching the metadata for any resource, without also fetching its (possibly enormous) representation.

Of course, sometimes you do want to download the movie, and for that you need a GET request. I’ve put the GET request in the accessor method S3::Object#value, in Example 3-11. Its structure mirrors that of S3::Object#metadata.

  # Retrieves the value of this Object, possibly fetching it
  # (along with the metadata) from S3.
  def value
    # If there's no value yet...
    unless @value
      # Make a GET request to this Object's URI.
      response = open(uri)
      # Read the metadata from the HTTP headers in the response.
      store_metadata(response.meta) unless @metadata
      # Read the value from the entity-body
      @value = response.read
    end
    return @value
  end

The client stores objects on the S3 service the same way it stores buckets: by sending a PUT request to a certain URI. The bucket PUT is trivial because a bucket has no distinguishing features other than its name, which goes into the URI of the PUT request. An object PUT is more complex. This is where the HTTP client specifies an object’s metadata (such as Content-Type) and value. This information will be made available on future HEAD and GET requests.

Fortunately, setting up the PUT request is not terribly complicated, because an object’s value is whatever the client says it is. I don’t have to wrap the object’s value in an XML document or anything. I just send the data as is, and set HTTP headers that correspond to the items of metadata in my metadata hash (see Example 3-12).

  # Store this Object on S3.
  def put(acl_policy=nil)

    # Start from a copy of the original metadata, or an empty hash if
    # there is no metadata yet.
    args = @metadata ? @metadata.clone : {}

    # Set the HTTP method, the entity-body, and some additional HTTP
    # headers.
    args[:method] = :put
    args["x-amz-acl"] = acl_policy if acl_policy
    if @value
      args["Content-Length"] = @value.size.to_s
      args[:body] = @value
    end

    # Make a PUT request to this Object's URI.
    open(uri, args)
    return self
  end

The S3::Object#delete implementation (see Example 3-13) is identical to S3::Bucket#delete.

  # Deletes this Object.
  def delete
    # Make a DELETE request to this Object's URI.
    open(uri, :method => :delete)
  end

And Example 3-14 shows the method for turning HTTP response headers into S3 object metadata. Except for Content-Type, you should prefix all the metadata headers you set with the string “x-amz-meta-”. Otherwise they won’t make the round trip to the S3 server and back to a web service client. S3 will think they’re quirks of your client software and discard them.

  private

  # Given a hash of headers from a HTTP response, picks out the
  # headers that are relevant to an S3 Object, and stores them in the
  # instance variable @metadata.
  def store_metadata(new_metadata)    
    @metadata = {}
    new_metadata.each do |h,v| 
      if RELEVANT_HEADERS.member?(h) || h.index('x-amz-meta') == 0
        @metadata[h] = v  
      end
    end
  end
  RELEVANT_HEADERS = ['content-type', 'content-disposition', 'content-range',
                      'x-amz-missing-meta']
end

My S3 library has one feature still to be implemented. I’ve mentioned a few times that S3 lets you sign an HTTP request and give the URI to someone else, letting them make that request as you. Here’s the method that lets you do this: signed_uri (see Example 3-20). Instead of making an HTTP request with open, you pass the open arguments into this method, and it gives you a signed URI that anyone can use as you. To limit abuse, a signed URI works only for a limited time. You can customize that time by passing a Time object in as the keyword argument :expires.

  # Given information about an HTTP request, returns a URI you can
  # give to anyone else, to let them them make that particular HTTP
  # request as you. The URI will be valid for 15 minutes, or until the
  # Time passed in as the :expires option.
  def signed_uri(headers_and_options={}) 
    expires = headers_and_options[:expires] || (Time.now.to_i + (15 * 60))
    expires = expires.to_i if expires.respond_to? :to_i
    headers_and_options.delete(:expires) 
    signature = URI.escape(signature(uri, headers_and_options[:method], 
                                     headers_and_options, nil))
    q = (uri.index("?")) ? "&" : "?"
    "#{uri}#{q}Signature=#{signature}&Expires=#{expires}&AWSAccessKeyId=#{@@public_key}"
  end
end

end # Remember the all-encompassing S3 module? This is the end.

Here’s how it works. Suppose I want to give a customer access to my hosted file at https://s3.amazonaws.com/BobProductions/KomodoDragon.avi. I can run the code in Example 3-21 to generate a URI for my customer.

#!/usr/bin/ruby1.9
# s3-signed-uri.rb
require 'S3lib'

bucket = S3::Bucket.new("BobProductions")
object = S3::Object.new(bucket, "KomodoDragon.avi")
puts object.signed_uri
# "https://s3.amazonaws.com/BobProductions/KomodoDragon.avi
# ?Signature=J%2Fu6kxT3j0zHaFXjsLbowgpzExQ%3D
# &Expires=1162156499&AWSAccessKeyId=0F9DBXKB5274JKTJ8DG2"

That URI will be valid for 15 minutes, the default for my signed_uri implementation. It incorporates my key ID (AWSAccessKeyId), the expiration time (Expires), and the cryptographic Signature. My customer can visit this URI and download the movie file KomodoDragon.avi. Amazon will charge me for my customer’s use of their bandwidth. If my customer modifies any part of the URI (maybe they to try to download a second movie too), the S3 service will reject their request. An untrustworthy customer can send the URI to all of their friends, but it will stop working in 15 minutes.

You may have noticed a problem here. The canonical string usually includes the value of the Date header. When my customer visits the URI you signed, their web browser will surely send a different value for the Date header. That’s why, when you’re generating a canonical string to give to someone else, you set an expiration date instead of a request date. Look back to Example 3-18 and the implementation of canonical_string, where the expiration date (if provided) overwrites any value for the Date header.

What if I want to make an object publicly accessible? I want to serve my files to the world and let Amazon deal with the headaches of server management. Well, I could set an expiration date very far in the future, and give out the enormous signed URI to everyone. But there’s an easier way to get the same results: allow anonymous access. You can do this by setting the access policy for a bucket or object, telling S3 to respond to unsigned requests for it. You do this by sending the x-amz-acl header along with the PUT request that creates the bucket or object.

That’s what the acl_policy argument to Bucket#put and Object#put does. If you want to make a bucket or object publicly readable or writable, you pass an appropriate value in for acl_policy. My client sends that value as part of the custom HTTP request header X-amz-acl. Amazon S3 reads this request header and sets the rules for bucket or object access appropriately.

The client in Example 3-22 creates an S3 object that anyone can read by visiting its URI at https://s3.amazonaws.com/BobProductions/KomodoDragon-Trailer.avi. In this scenario, I’m not selling my movies: just using Amazon as a hosting service so I don’t have to serve movies from my own web site.

#!/usr/bin/ruby -w
# s3-public-object.rb
require 'S3lib'

bucket = S3::Bucket.new("BobProductions")
object = S3::Object.new(bucket, "KomodoDragon-Trailer.avi")
object.put("public-read")

S3 understands four access policies:

There are also fine-grained ways of granting access to a bucket or object, which I won’t cover. If you’re interested, see the section “Setting Access Policy with REST” in the S3 technical documentation. That section reveals a parallel universe of extra resources. Every bucket /{name-of-bucket} has a shadow resource /{name-of-bucket}?acl corresponding to that bucket’s access control rules, and every object /{name-of-bucket}/{name-of-object} has a shadow ACL resource /{name-of-bucket}/{name-of-object}?acl. By sending PUT requests to these URIs, and including XML representations of access control lists in the request entity-bodies, you can set specific permissions and limit access to particular S3 users.

My web service will be a simple notebook: a way of keeping timestamped notes to myself. I’ve got Rails 1.2 installed on my computer, so I can create the notebook service like this:

$ rails notebook    
$ cd notebook

I create a database on my system called notebook_development, and edit the Rails file notebook/config/database.yml to give Rails the information it needs to connect to my database. Any general guide to Rails will have more detail on these initial steps.

Now I’ve created a Rails application, but it doesn’t do anything. I’m going to generate code for a simple, RESTful web service with the scaffold generator. I want my notes to contain a timestamp and a body of text, so I run the following command:

$ ruby script/generate scaffold note date:date body:text
create  app/views/notes
create  app/views/notes/index.rhtml
create  app/views/notes/show.rhtml
create  app/views/notes/new.rhtml
create  app/views/notes/edit.rhtml
create  app/views/layouts/notes.rhtml
create  public/stylesheets/scaffold.css
create  app/models/note.rb
create  app/controllers/notes_controller.rb
create  test/functional/notes_controller_test.rb
create  app/helpers/notes_helper.rb
create  test/unit/note_test.rb
create  test/fixtures/notes.yml
create  db/migrate
create  db/migrate/001_create_notes.rb
route  map.resources :notes

Rails has generated a complete set of web service code—model, view, and controller—for my “note” object. There’s code in db/migrate/001_create_notes.rb that creates a database table called notes with three fields: a unique ID, a date (date), and a piece of text (body).

The model code in app/models/note.rb provides an ActiveResource interface to the database table. The controller code in app/controllers/notes_controller.rb exposes that interface to the world through HTTP, and the views in app/views/notes define the user interface. It adds up to a RESTful web service—not a very fancy one, but one that’s good enough for a demo or to use as a starting point.

Before starting the service I need to initialize the database:

$ rake db:migrate
== CreateNotes: migrating =====================================================
-- create_table(:notes)
   -> 0.0119s
== CreateNotes: migrated (0.0142s) ============================================

Now I can start the notebook application and start using my service:

$ script/server
=> Booting WEBrick...
=> Rails application started on http://0.0.0.0:3000
=> Ctrl-C to shutdown server; call with --help for options

The application I just generated is not much use except as a demo, but it demos some pretty impressive features. First, it’s both a web service and a web application. I can visit http://localhost:3000/notes in my web browser and create notes through the web interface. After a while the view of http://localhost:3000/notes might look like Figure 3-1.

Figure 3-1. The notebook web application with a few entered notes

If you’ve ever written a Rails application or seen a Rails demo, this should look familiar. But in Rails 1.2, the generated model and controller can also act as a RESTful web service. A programmed client can access it as easily as a web browser can.

Unfortunately, the ActiveResource client itself was not released along with Rails 1.2. As of the time of writing, it’s still being developed on the tip of the Rails development tree. To get the code I need to check it out from the Subversion version control repository:

$ svn co http://dev.rubyonrails.org/svn/rails/trunk activeresource_client
$ cd activeresource_client

Now I’m ready to write ActiveResource clients for the notebook’s web service. Example 3-24 is a client that creates a note, modifies it, lists the existing notes, and then deletes the note it just created.

#!/usr/bin/ruby -w
# activeresource-notebook-manipulation.rb

require 'activesupport/lib/active_support'
require 'activeresource/lib/active_resource'

# Define a model for the objects exposed by the site
class Note < ActiveResource::Base
  self.site = 'http://localhost:3000/'
end

def show_notes
  notes = Note.find :all                 # GET /notes.xml
  puts "I see #{notes.size} note(s):"
  notes.each do |note|
    puts " #{note.date}: #{note.body}"
  end
end

new_note = Note.new(:date => Time.now, :body => "A test note")
new_note.save                            # POST /notes.xml

new_note.body = "This note has been modified."
new_note.save                            # PUT /notes/{id}.xml

show_notes

new_note.destroy                         # DELETE /notes/{id}.xml

puts
show_notes

Example 3-25 shows the output when I run that program:

I see 3 note(s):
 2006-06-05: What if I wrote a book about REST?
 2006-12-18: Pasta for lunch maybe?
 2006-12-18: This note has been modified.

I see 2 note(s):
 2006-06-05: What if I wrote a book about REST?
 2006-12-18: Pasta for lunch maybe?

If you’re familiar with ActiveRecord, the object-relational mapper that connects Rails to a database, you’ll notice that the ActiveResource interface looks almost exactly the same. Both libraries provide an object-oriented interface to a wide variety of objects, each of which exposes a uniform interface. With ActiveRecord, the objects live in a database and are exposed through SQL, with its SELECT, INSERT, UPDATE, and DELETE. With ActiveResource, they live in a Rails application and are exposed through HTTP, with its GET, POST, PUT, and DELETE.

Example 3-26 is an excerpt from the Rails server logs at the time I ran my ActiveResource client. The GET, POST, PUT, and DELETE requests correspond to the commented lines of code back in Example 3-24.

"POST /notes.xml HTTP/1.1" 201
"PUT /notes/5.xml HTTP/1.1" 200
"GET /notes.xml HTTP/1.1" 200
"DELETE /notes/5.xml HTTP/1.1" 200 
"GET /notes.xml HTTP/1.1" 200

What’s going on in these requests? The same thing that’s going on in requests to S3: resource access through HTTP’s uniform interface. My notebook service exposes two kinds of resources:

These resources expose GET, PUT, and DELETE, just like the S3 resources do. The list of notes also supports POST to create a new note. That’s a little different from S3, where objects are created with PUT, but it’s just as RESTful.

When the client runs, XML documents are transferred invisibly between client and server. They look like the documents in Example 3-27 or 3-28: simple depictions of the underlying database rows.

<?xml version="1.0" encoding="UTF-8"?>
<notes>
 <note>
  <body>What if I wrote a book about REST?</body>
  <date type="date">2006-06-05</date>
  <id type="integer">2</id>
 </note>
 <note>
  <body>Pasta for lunch maybe?</body>
  <date type="date">2006-12-18</date>
  <id type="integer">3</id>
 </note>
</notes>

<?xml version="1.0" encoding="UTF-8"?>
<note>
 <body>This note has been modified.</body>
</note>

Right now the only ActiveResource client library is the Ruby library, and Rails is the only framework that exposes ActiveResource-compatible services. But nothing’s happening here except HTTP requests that pass XML documents into certain URIs and get XML documents back. There’s no reason why a client in some other language couldn’t send those XML documents, or why some other framework couldn’t expose the same URIs.

Example 3-29 is a Python implementation of the client program from Example 3-24. It’s longer than the Ruby program, because it can’t rely on ActiveResource. It has to build its own XML documents and make its own HTTP requests, but its structure is almost exactly the same.

#!/usr/bin/python
# activeresource-notebook-manipulation.py

from elementtree.ElementTree import Element, SubElement, tostring
from elementtree import ElementTree
import httplib2
import time

BASE = "http://localhost:3000/"
client = httplib2.Http(".cache")

def showNotes():
    headers, xml = client.request(BASE + "notes.xml")
    doc = ElementTree.fromstring(xml)
    for note in doc.findall('note'):
        print "%s: %s" % (note.find('date').text, note.find('body').text)

newNote = Element("note")
date = SubElement(newNote, "date")
date.attrib['type'] = "date"
date.text = time.strftime("%Y-%m-%d", time.localtime())
body = SubElement(newNote, "body")
body.text = "A test note"
 
headers, ignore = client.request(BASE + "notes.xml", "POST",
                                 body= tostring(newNote),
                                 headers={'content-type' : 'application/xml'})
newURI = headers['location']

modifiedBody = Element("note")
body = SubElement(modifiedBody, "body")
body.text = "This note has been modified"

client.request(newURI, "PUT",
               body=tostring(modifiedBody),
               headers={'content-type' : 'application/xml'})

showNotes()

client.request(newURI, "DELETE")

print
showNotes()

• Version 1.0.3 of the software release

• The latest version of the software release

• The first weblog entry for October 24, 2006

• A road map of Little Rock, Arkansas

• Some information about jellyfish

• A directory of resources pertaining to jellyfish

• The next prime number after 1024

• The next five prime numbers after 1024

• The sales numbers for Q42004

• The relationship between two acquaintances, Alice and Bob

• A list of the open bugs in the bug database

Here’s the first point where the ROA builds upon the sparse recommendations of the REST thesis and the W3C recommendations. I propose that a resource and its URI ought to have an intuitive correspondence. Here are some good URIs for the resources I listed above:

URIs should have a structure. They should vary in predictable ways: you should not go to /search/Jellyfish for jellyfish and /i-want-to-know-about/Mice for mice. If a client knows the structure of the service’s URIs, it can create its own entry points into the service. This makes it easy for clients to use your service in ways you didn’t think of.

This is not an absolute rule of REST, as we’ll see in the Name the Resources” section of Chapter 5. URIs do not technically have to have any structure or predictability, but I think they should. This is one of the rules of good web design, and it shows up in RESTful and REST-RPC hybrid services alike.

Let’s consider some edge cases. Can two resources be the same? Can two URIs designate the same resource? Can a single URI designate two resources?

By definition, no two resources can be the same. If they were the same, you’d only have one resource. However, at some moment in time two different resources may point to the same data. If the current software release is 1.0.3, then http://www.example.com/software/releases/1.0.3.tar.gz and http://www.example.com/software/releases/latest.tar.gz will refer to the same file for a while. But the ideas behind those two URIs are different: one of them always points to a particular version, and the other points to whatever version is newest at the time the client accesses it. That’s two concepts and two resources. You wouldn’t link to latest when reporting a bug in version 1.0.3.

A resource may have one URI or many. The sales numbers available at http://www.example.com/sales/2004/Q4 might also be available at http://www.example.com/sales/Q42004. If a resource has multiple URIs, it’s easier for clients to refer to the resource. The downside is that each additional URI dilutes the value of all the others. Some clients use one URI, some use another, and there’s no automatic way to verify that all the URIs refer to the same resource.

Fetching sales/2004/Q4 might get you the same bytestream as fetching sales/Q42004, because they’re different URIs for the same resource: “sales for the last quarter of 2004.” Fetching releases/1.0.3.tar.gz might give you the exact same bytestream as fetching releases/latest.tar.gz, but they’re different resources because they represent different things: “version 1.0.3” and “the latest version.”

Every URI designates exactly one resource. If it designated more than one, it wouldn’t be a Universal Resource Identifier. However, when you fetch a URI the server may send you information about multiple resources: the one you requested and other, related ones. When you fetch a web page, it usually conveys some information of its own, but it also has links to other web pages. When you retrieve an S3 bucket with an Amazon S3 client, you get a document that contains information about the bucket, and information about related resources: the objects in the bucket.

When we talk about “statelessness,” what counts as “state”? What’s the difference between persistent data, the useful server-side data that makes us want to use web services in the first place, and this state we’re trying to keep off the server? The Flickr web service lets you upload pictures to your account, and those pictures are stored on the server. It would be crazy to make the client send every one of its pictures along with every request to flickr.com, just to keep the server from having to store any state. That would defeat the whole point of the service. But what’s the difference between this scenario, and state about the client’s session, which I claim should be kept off the server?

The problem is one of terminology. Statelessness implies there’s only one kind of state and that the server should go without it. Actually, there are two kinds of state. From this point on in the book I’m going to distinguish between application state, which ought to live on the client, and resource state, which ought to live on the server.

When you use a search engine, your current query and your current page are bits of application state. This state is different for every client. You might be on page 3 of the search results for “jellyfish,” and I might be on page 1 of the search results for “mice.” The page number and the query are different because we took different paths through the application. Our respective clients store different bits of application state.

A web service only needs to care about your application state when you’re actually making a request. The rest of the time, it doesn’t even know you exist. This means that whenever a client makes a request, it must include all the application states the server will need to process the request. The server might send back a page with links, telling the client about other requests it might want to make in the future, but then it can forget all about the client until the next request. That’s what I mean when I say a web service should be “stateless.” The client should be in charge of managing its own path through the application.

Resource state is the same for every client, and its proper place is on the server. When you upload a picture to Flickr, you create a new resource: the new picture has its own URI and can be the target of future requests. You can fetch, modify, and delete the “picture” resource through HTTP. It’s there for everybody: I can fetch it too. The picture is a bit of resource state, and it stays on the server until a client deletes it.

Client state can show up when you don’t expect it. Lots of web services make you sign up for a unique string they call an API key or application key. You send in this key with every request, and the server uses it to restrict you to a certain number of requests a day. For instance, an API key for Google’s deprecated SOAP search API is good for 1,000 requests a day. That’s application state: it’s different for every client. Once you exceed the limit, the behavior of the service changes dramatically: on request 1,000 you get your data, and on request 1,001 you get an error. Meanwhile, I’m on request 402 and the service still works fine for me.

Of course, clients can’t be trusted to self-report this bit of application state: the temptation to cheat is too great. So servers keep this kind of application state on the server, violating statelessness. The API key is like the Rails _session_id cookie, a key into a server-side client session that lasts one day. This is fine as far as it goes, but there’s a scalability price to be paid. If the service is to be distributed across multiple machines, every machine in the cluster needs to know that you’re on request 1,001 and I’m on request 402 (technical term: session replication), so that every machine knows to deny you access and let me through. Alternatively, the load balancer needs to make sure that every one of your requests goes to the same machine in the cluster (technical term: session affinity). Statelessness removes this requirement. As a service designer, you only need to start thinking about data replication when your resource state needs to be split across multiple machines.

1. One containing only metadata, like a cover image and reviews, used to advertise the book.

2. An electronic copy of the data in the book, sent to you via HTTP when you pay for it.

If a server provides multiple representations of a resource, how does it figure out which one the client is asking for? For instance, a press release might be put out in both English and Spanish. Which one does a given client want?

There are a number of ways to figure this out within the constraints of REST. The simplest, and the one I recommend for the Resource-Oriented Architecture, is to give a distinct URI to each representation of a resource. http://www.example.com/releases/104.en could designate the English representation of the press release, and http://www.example.com/releases/104.es could designate the Spanish representation.

I recommend this technique for ROA applications because it means the URI contains all information necessary for the server to fulfill the request. The disadvantage, as whenever you expose multiple URIs for the same resource, is dilution: people who talk about the press release in different languages appear to be talking about different things. You can mitigate this problem somewhat by exposing the URI http://www.example.com/releases/104 to mean the release as a Platonic form, independent of any language.

The alternative way is called content negotiation. In this scenario the only exposed URI is the Platonic form URI, http://www.example.com/releases/104. When a client makes a request for that URI, it provides special HTTP request headers that signal what kind of representations the client is willing to accept.

Your web browser has a setting for language preferences: which languages you’d prefer to get web pages in. The browser submits this information with every HTTP request, in the Accept-Language header. The server usually ignores this information because most web pages are available in only one language. But it fits with what we’re trying to do here: expose different-language representations of the same resource. When a client requests http://www.example.com/releases/104, the server can decide whether to serve the English or the Spanish representation based on the client’s Accept-Language header.

A client can also set the Accept header to specify which file format it prefers for representations. A client can say it prefers XHTML to HTML, or SVG to any other graphics format.

The server is allowed to use any of this request metadata when deciding which representation to send. Other types of request metadata include payment information, authentication credentials, the time of the request, caching directives, and even the IP address of the client. All of these might make a difference in the server’s decision of what data to include in the representation, which language and which format to use, and even whether to send a representation at all or to deny access.

It’s RESTful to keep this information in the HTTP headers, and it’s RESTful to put it in the URI. I recommend keeping as much of this information as possible in the URI, and as little as possible in request metadata. I think URIs are more useful than metadata. URIs get passed around from person to person, and from program to program. The request metadata almost always gets lost in transition.

Here’s a simple example of this dilemma: the W3C HTML validator, a web service available at http://validator.w3.org/. Here’s a URI to a resource on the W3C’s site, a validation report on the English version of my hypothetical press release: http://validator.w3.org/check?uri=http%3A%2F%2Fwww.example.com%2Freleases%2F104.en.

Here’s another resource: a validation report on the Spanish version of the press release: http://validator.w3.org/check?uri=http%3A%2F%2Fwww.example.com%2Freleases%2F104.es.

Every URI in your web space becomes a resource in the W3C’s web application, whether or not it designates a distinct resource on your site. If your press release has a separate URI for each representation, you can get two resources from the W3C: validation reports for the English and the Spanish versions of the press release.

But if you only expose the Platonic form URI, and serve both representations from that URI, you can only get one resource from the W3C. That would be a validation report for the default version of the press release (probably the English one). You’ve got no way of knowing whether or not the Spanish representation contains HTML formatting errors. If the server doesn’t expose the Spanish press release as its own URI, there’s no corresponding resource available on the W3C site. This doesn’t mean you can’t expose that Platonic form URI: just that it shouldn’t be the only URI you use.

Unlike humans, computer programs are very bad at dealing with representations they didn’t expect. I think an automated web client should be as explicit as possible about the representation it wants. This almost always means specifying a representation in the URL.

• The external web page that talks about jellyfish: http://www.aloha.com/~lifeguards/jelyfish.html. The main point of this web service, of course, is to present links of this sort.

• A link to a Google-provided cache of the external page (the “Cached” link). These links always have long URIs that point to Google-owned IP addresses, like http://209.85.165.104/search?q=cache:FQrLzPU0tKQJ...

• A link to a directory of pages Google thinks are related to the external page (http://www.google.com/search?q=related:www.aloha.com/~lifeguards/jelyfish.html, linked as “Similar pages”). This is another case of a web service taking a URI as input.

• A set of navigation links that take you to different pages of the “jellyfish” directory: http://www.google.com/search?q=jellyfish&start=10, http://www.google.com/search?q=jellyfish&start=20, and so on.

• Retrieve a representation of a resource: HTTP GET

• Create a new resource: HTTP PUT to a new URI, or HTTP POST to an existing URI (see the POST” section below)

• Modify an existing resource: HTTP PUT to an existing URI

• Delete an existing resource: HTTP DELETE

These three should be familiar to you from the S3 example in Chapter 3. To fetch or delete a resource, the client just sends a GET or DELETE request to its URI. In the case of a GET request, the server sends back a representation in the response entity-body. For a DELETE request, the response entity-body may contain a status message, or nothing at all.

To create or modify a resource, the client sends a PUT request that usually includes an entity-body. The entity-body contains the client’s proposed new representation of the resource. What data this is, and what format it’s in, depends on the service. Whatever it looks like, this is the point at which application state moves onto the server and becomes resource state.

Again, think of the S3 service, where there are two kinds of resources you can create: buckets and objects. To create an object, you send a PUT request to its URI and include the object’s content in the entity-body of your request. You do the same thing to modify an object: the new content overwrites any old content.

Creating a bucket is a little different because you don’t have to specify an entity-body in the PUT request. A bucket has no resource state except for its name, and the name is part of the URI. (This is not quite true. The objects in a bucket are also elements of that bucket’s resource state: after all, they’re listed when you GET a bucket’s representation. But every S3 object is a resource of its own, so there’s no need to manipulate an object through its bucket. Every object exposes the uniform interface and you can manipulate it separately.) Specify the bucket’s URI and you’ve specified its representation. PUT requests for most resources do include an entity-body containing a representation, but as you can see it’s not a requirement.

There are three other HTTP methods I consider part of the uniform interface. Two of them are simple utility methods, so I’ll cover them first.

You saw the HEAD method exposed by the S3 services’s resources in Chapter 3. An S3 client uses HEAD to fetch metadata about a resource without downloading the possibly enormous entity-body. That’s what HEAD is for. A client can use HEAD to check whether a resource exists, or find out other information about the resource, without fetching its entire representation. HEAD gives you exactly what a GET request would give you, but without the entity-body.

The OPTIONS method lets the client discover what it’s allowed to do to a resource. The response to an OPTIONS request contains the HTTP Allow header, which lays out the subset of the uniform interface this resource supports. Here’s a sample Allow header:

Allow: GET, HEAD

That particular header means the client can expect the server to act reasonably to a GET or HEAD request for this resource, but that the resource doesn’t support any other HTTP methods. Effectively, this resource is read-only.

The headers the client sends in the request may affect the Allow header the server sends in response. For instance, if you send a proper Authorization header along with an OPTIONS request, you may find that you’re allowed to make GET, HEAD, PUT, and DELETE requests against a particular URI. If you send the same OPTIONS request but omit the Authorization header, you may find that you’re only allowed to make GET and HEAD requests. The OPTIONS method lets the client do simple access control checks.

In theory, the server can send additional information in response to an OPTIONS request, and the client can send OPTIONS requests that ask very specific questions about the server’s capabilities. Very nice, except there are no accepted standards for what a client might ask in an OPTIONS request. Apart from the Allow header there are no accepted standards for what a server might send in response. Most web servers and frameworks feature very poor support for OPTIONS. So far, OPTIONS is a promising idea that nobody uses.

Now we come to that most misunderstood of HTTP methods: POST. This method essentially has two purposes: one that fits in with the constraints of REST, and one that goes outside REST and introduces an element of the RPC style. In complex cases like this it’s best to go back to the original text. Here’s what RFC 2616, the HTTP standard, says about POST (this is from section 9.5):

What does this mean in the context of REST and the ROA?

If you expose HTTP’s uniform interface as it was designed, you get two useful properties for free. When correctly used, GET and HEAD requests are safe. GET, HEAD, PUT and DELETE requests are idempotent.

The important thing about REST is not that you use the specific uniform interface that HTTP defines. REST specifies a uniform interface, but it doesn’t say which uniform interface. GET, PUT, and the rest are not a perfect interface for all time. What’s important is the uniformity: that every service use HTTP’s interface the same way.

The point is not that GET is the best name for a read operation, but that GET means “read” across the Web, no matter which resource you’re using it on. Given a URI of a resource, there’s no question of how you get a representation: you send an HTTP GET request to that URI. The uniform interface makes any two services as similar as any two web sites. Without the uniform interface, you’d have to learn how each service expected to receive and send information. The rules might even be different for different resources within a single service.

You can program a computer to understand what GET means, and that understanding will apply to every RESTful web service. There’s not much to understand. The service-specific code can live in the handling of the representation. Without the uniform interface, you get a multiplicity of methods taking the place of GET: doSearch and getPage and nextPrime. Every service speaks a different language. This is also the reason I don’t like overloaded POST very much: it turns the simple Esperanto of the uniform interface into a Babel of one-off sublanguages.

Some applications extend HTTP’s uniform interface. The most obvious case is WebDAV, which adds eight new HTTP methods including MOVE, COPY, and SEARCH. Using these methods in a web service would not violate any precept of REST, because REST doesn’t say what the uniform interface should look like. Using them would violate my Resource-Oriented Architecture (I’ve explicitly tied the ROA to the standard HTTP methods), but your service could still be resource-oriented in a general sense.

The real reason not to use the WebDAV methods is that doing so makes your service incompatible with other RESTful services. Your service would use a different uniform interface than most other services. There are web services like Subversion that use the WebDAV methods, so your service wouldn’t be all alone. But it would be part of a much smaller web. This is why making up your own HTTP methods is a very, very bad idea: your custom vocabulary puts you in a community of one. You might as well be using XML-RPC.

Another uniform interface consists solely of HTTP GET and overloaded POST. To fetch a representation of a resource, you send GET to its URI. To create, modify, or delete a resource, you send POST. This interface is perfectly RESTful, but, again, it doesn’t conform to my Resource-Oriented Architecture. This interface is just rich enough to distinguish between safe and unsafe operations. A resource-oriented web application would use this interface, because today’s HTML forms only support GET and POST.

1. Resources

2. Their names (URIs)

3. Their representations

4. The links between them

1. Addressability

2. Statelessness

3. Connectedness

4. A uniform interface

1. Figure out the data set

2. Split the data set into resources

3. Name the resources with URIs

4. Design the representation(s) served to the client

5. Integrate this resource into existing resources, using hypermedia links and forms

6. Consider the typical course of events: what’s supposed to happen?

7. Consider error conditions: what might go wrong?

This is a standard first step in any analysis. Sometimes you get to choose your data set, and sometimes you’re trying to expose data you’ve already got. You may come back to this step as you see how best to expose your data set as resources. I went through the design process two or three times before I figured out that points on a planet needed to be considered distinct from points on any particular map. Even now, the data set is chaotic, just a bundle of ideas. I’ll give it shape when I divide it into resources.

I presented the results of a search operation (“places on Earth called Springfield”) as part of the data set. An RPC-oriented analysis would treat these as actions that the client invokes—remember doGoogleSearch from the Google SOAP service. Compare this to how the Google web site works: in a resource-oriented analysis, ways of looking at the data are themselves pieces of data. If you consider an algorithm’s output to be a resource, running the algorithm can be as simple as sending a GET to that resource.

So far I’ve said nothing about how a web service client can access this data set through HTTP. Right now I’m just gathering everything together in one place. I’m also ignoring any consideration of how these features should be implemented. If I actually planned to provide this service, the features I’ve announced so far would have a profound effect on the structure of my database, and I could start designing that part of the application as well. As it is, I’m going to wave away details of the backend implementation, and press on with the design of the web service.

Predefined one-off resources for especially important aspects of the application.

This includes top-level directories of other available resources. Most services expose few or no one-off resources.

Example: A web site’s homepage. It’s a one-of-a-kind resource, at a well-known URI, which acts as a portal to other resources.

The root URI of Amazon’s S3 service serves a list of your S3 buckets. There’s only one resource of this type on S3. You can GET this resource, but you can’t DELETE it, and you can’t modify it directly: it’s modified only by operating on its buckets. It’s a predefined resource that acts as a directory of child resources (the buckets).

A resource for every object exposed through the service.

One service may expose many kinds of objects, each with its own resource set. Most services expose a large or infinite number of these resources.

Example: Every S3 bucket you create is exposed as a resource. You can create up to 100 buckets, and they can have just about any names you want (it’s just that your names can’t conflict with anyone else’s). You can GET and DELETE these resources, but once you’ve created them you can’t modify them directly: they’re modified only by operating on the objects they contain.

Every S3 object you create is also exposed as a resource. A bucket has room for any number of objects. You can GET, PUT, and DELETE these resources as you see fit.

Resources representing the results of algorithms applied to the data set.

This includes collection resources, which are usually the results of queries. Most services either expose an infinite number of algorithmic resources, or they don’t expose any.

Example: A search engine exposes an infinite number of algorithmic resources. There’s one for every search request you might possibly make. The Google search engine exposes one resource at http://google.com/search?q=jellyfish (that’d be “a directory of resources about jellyfish”) and another at http://google.com/search?q=chocolate (“a directory of resources about chocolate”). Neither of these resources were explicitly defined ahead of time: Google translates any URI of the form http://google.com/search?q={query} into an algorithmic resource “a directory of resources about {query}.”

I didn’t cover this in much detail back in Chapter 3, but S3 also exposes an infinite number of algorithmic resources. If you’re interested, look back to Example 3-7 and the implementation of S3::Bucket#getObjects. Some of S3’s algorithmic resources work like a search engine for the objects in a bucket. If you’re only interested in objects whose names start with the string “movies/”, there’s a resource for that: it’s exposed through the URI https://s3.amazonaws.com/MyBucket?Prefix=movies/. You can GET this resource, but you can’t manipulate it directly: it’s just a view of the underlying data set.

• The list of planets

• Mars

• Earth

• The satellite map of Mars

• The radar map of Venus

• The topographic map of Earth

• The political map of Earth

• 24.9195N 17.821E on Earth

• 24.9195N 17.821E on the political map of Earth

• 24.9195N 17.821E on Mars

• 44N 0W on the geologic map of Earth

• The Cleopatra crater on Venus

• The Ubehebe crater on Earth

• 1005 Gravenstein Highway North, Sebastopol, CA

• The headquarters of O’Reilly Media, Inc.

• The place called Springfield in Massachusetts, in the United States of America, on Earth

• Places on Earth called Springfield

• Container ships on Earth

• Craters on Mars more than 1 km in diameter

• Places on the moon named before 1900

• Places in the United States named Springfield

• Sites of hot springs in Colorado

• Oil tankers or container ships near Indonesia

• Pizza restaurants in Worcester, MA

• Diners near Mount Rushmore

• Areas of high arsenic near 24.9195N 17.821E

• Towns in France with population less than 1,000

A RESTful web service exposes both its data and its algorithms through resources. There’s usually a hierarchy that starts out small and branches out into infinitely many leaf nodes. The list of planets contains the planets, which contain points and places, which contain maps. The S3 bucket list contains the individual buckets, which contain the objects.

It takes a while to get the hang of exposing an algorithm as a set of resources. Instead of thinking in terms of actions (“do a search for places on the map”), you need to think in terms of the results of that action (“the list of places on the map matching a search criteria”). You may find yourself coming back to this step if you find that your design doesn’t fit HTTP’s uniform interface.

1. Use path variables to encode hierarchy: /parent/child

2. Put punctuation characters in path variables to avoid implying hierarchy where none exists: /parent/child1;child2

3. Use query variables to imply inputs into an algorithm, for example: /search?q=jellyfish&start=20

Let’s make URIs for the second class of resource: planets and places on planets. There’s one piece of scoping information here: what planet are we looking at? (Earth? Venus? Ganymede?) This scoping information fits naturally into a hierarchy: the list of planets is at the top, and underneath it is every particular planet. Here are the URIs to some of my planets. I show hierarchy by using the slash character to separate pieces of scoping information.

To identify geographical places by name I’ll just extend the hierarchy to the right. You’ll know you’ve got a good URI design when it’s easy to extend hierarchies by tacking on additional path variables. Here are some URIs to various places on planets:

We’re now deep into web service territory. Sending a GET to one of these URIs invokes a remote operation that takes a variable number of arguments, and can locate a place on a planet to any desired degree of precision. But the URIs themselves look like normal web site URIs you can bookmark, cache, put on billboards, and pass to other services as input—because that’s what they are. Path variables are the best way to organize scoping information that can be arranged hierarchically. The same structure you see in a filesystem, or on a static web site, can correspond to an arbitrarily long list of path variables.

The next resources I need to name are geographic points on the globe, represented by latitude and longitude. Latitude and longitude are tied together, so a hierarchy isn’t appropriate. A URI like /Earth/24.9195/17.821 doesn’t make sense. The slash makes it look like longitude is a subordinate concept to latitude, the way /Earth/Chicago signals that Chicago is part of Earth.

Instead of using the slash to put two pieces of scoping information into a hierarchy, I recommend combining them on the same level of a hierarchy with a punctuation character: usually the semicolon or the comma. I’m going to use a comma to separate latitude and longitude. This yields URIs like the following:

Latitude and longitude can also be used as scoping information to uniquely identify a named place. A human would probably identify Mount Rushmore as /Earth/USA/Mount%20Rushmore or as /v1/Earth/USA/SD/Mount%20Rushmore, but /v1/Earth/43.9,-103.46/Mount%20Rushmore would be more precise.

From a URI design perspective, the interesting thing here is that I’m stuffing two pieces of scoping information into one path variable. The first path variable denotes a planet, and the second one denotes both latitude and longitude. This kind of URI may look a little strange, because not many web sites or services use them right now, but they’re catching on.

I recommend using commas when the order of the scoping information is important, and semicolons when the order doesn’t matter. In this case the order matters: if you switch latitude and longitude, you get a different point on the planet. So I used commas to separate the two numbers. It doesn’t hurt that people already use commas in written language to separate latitude and longitude: URIs should use our existing conventions when possible.

In another case the order might not matter. Consider a web service that lets you mix colors of paint to get the shade you want. If you’re mixing red and blue paint, it doesn’t matter whether you pour the red into the blue or the blue into the red: you get purple either way. So the URI /color-blends/red;blue identifies the same resource as /color-blends/blue;red. I think the semicolon is better than the comma here, because the order doesn’t matter. This is just a typographical convention, but it helps a human being make sense of your web service URIs. The use of the semicolon feeds into an obscure idea called matrix URIs, a way of defining key-value pairs in URIs without using query variables. Some newer standards, like WADL, offer support for matrix URIs. They’re especially useful if you ever need to put key-value pairs in the middle of a hierarchy.

Most web applications don’t store much state in path variables: they use query variables instead. You may have seen URIs like this:

Those URIs would look better without the query variables:

Sometimes, though, query variables are appropriate. Here’s a Google search URI: http://www.google.com/search?q=jellyfish. If the Google web application used path variables, its URIs would look more like directories and less like the result of running an algorithm: http://www.google.com/search/jellyfish.

Both of those URIs would be legitimate resource-oriented names for the resource “a directory of web pages about jellyfish.” The second one doesn’t look quite right, though, because of how we’re socialized to look at URIs. Path variables look like you’re traversing a hierarchy, and query variables look like you’re passing arguments into an algorithm. “Search” sounds like an algorithm. For example, http://www.google.com/directory/jellyfish" might work better than /search/jellyfish.

This perception of query variables is reinforced whenever we use the Web. When you fill out an HTML form in a web browser, the data you input is turned into query variables. There’s no way to type “jellyfish” into a form and then be sent to http://www.google.com/search/jellyfish. The destination of an HTML form is hard-coded to http://www.google.com/search/, and when you fill out that form you end up at http://www.google.com/search?q=jellyfish. Your browser knows how to tack query variables onto a base URI. It doesn’t know how to substitute variables into a generic URI like http://www.google.com/search/{q}.

Because of this precedent, a lot of REST-RPC hybrid services use query variables when it would be more idiomatic to use path variables. Even when a hybrid service happens to expose resources RESTfully, the resources have URIs that make them look like function calls: URIs such as http://api.flickr.com/services/rest/?method=flickr.photos.search&tags=penguin. Compare that URI to the corresponding URI on the human-usable Flickr site: http://flickr.com/photos/tags/penguin.

I’ve managed to avoid query variables so far: every planet, every point on a planet, and every corresponding map is addressable without them. I don’t really like the way query variables look in a URI, and including them in a URI is a good way to make sure that URI gets ignored by tools like proxies, caches, and web crawlers. Think back to the Google Web Accelerator I mentioned in Why safety and idempotence matter” in Split the Data Set into Resources. It never pre-fetches a URI that includes a query variable, because that’s the kind of URI exposed by poorly-designed web applications that abuse HTTP GET. My service won’t abuse GET, of course, but outside applications have no way of knowing that.

But I’ve got one more type of resource to represent—lists of search results—and I’m out of tricks. It doesn’t make sense to keep going down the hierarchy of place, and I can’t keep piling on punctuation just to avoid the impression that my service is running an algorithm. Besides, this last type of resource is the result of running an algorithm. My search algorithm finds places that match map-specific criteria, just as a search engine finds web sites that match the client’s keywords. Query variables are perfectly appropriate for naming algorithmic resources.

The search interface for places can get as complex as I need it to be. I could expose a name query variable for place names and pollutant for sites of high pollution and cuisine for restaurants and all sorts of other query variables. But let’s imagine I’ve got the technology to make it simple. The only query variable I’ll add is show, which lets the client specify in natural language what feature(s) they’re searching for. The server will parse the client’s values for show and figure out what places should be in the list of search results.

In Split the Data Set into Resources” earlier in this chapter, I gave a whole lot of sample search resources: “places on Earth called Springfield,” and so on. Here’s how a client might use show to construct URIs for some of those resources.

Note that all of these URIs are searching the planet, not any particular map.

That’s a lot of details. After all, this is the first place where my fantasy resources come into contact with the real world of HTTP. Even so, my service only supports three basic kinds of URI. To recap, here they are:

The main purpose of any representation is to convey the state of the resource. Remember that “resource state” is just any information about the underlying resource. In this case, the state is going to answer questions like: what does this part of the world look like, graphically? Where exactly is that meteor crater, in latitude and longitude? Where are the nearby restaurants and what are their names? Where are the container ships right now? Representations of different resources will represent different items of state.

The other job of the representation is to provide levers of state. A resource’s representation ought to link to nearby resources (whatever “nearby” means in context): possible new application states. The goal here is connectedness: the ability to get from one resource to another by following links.

This is how web sites work. You don’t surf the Web by typing in URIs one after the other. You might type in one URI to get to a site’s home page, but then you surf by following links and filling out forms. One web page (a “state” of the web site) contains links to other, related web pages (nearby “states”).

Of course, a computer program can’t look at a document and decide which links it wants to follow. It only has the wants the programmer gives it. If a web service includes links in its representations, the representations must also contain machine-readable signals as to where each link leads. A programmer can write his or her client to pick up on those signals and decide which link matches up with the goals of the moment.

These links are the levers of application state. If a resource can be modified with PUT, or it can spawn new resources in response to POST, its representation ought to also expose the levers of resource state. The representation ought to provide any necessary information about what the POST or PUT request should look like. I’m getting a little ahead of myself here, since all the resources in this chapter are read-only. For now, I’ll be creating representations that expose the levers of application state.

The “home page” of my map service is a good place to start, and a good place to introduce the issues behind choosing a representation format. Basically, I want to display a list of links to the planets for which I have maps. What’s a good format for a representation of a list?

There’s always plain text. This representation in Example 5-2 shows one planet per line: the URI and then the name.

http://maps.example.com/Earth Earth
http://maps.example.com/Venus Venus
...

This is simple but it requires a custom parser. I generally think a structured data format is better than plain text, especially as representations get more complex. (Of course, if plain text is what you’re serving, there’s no need to dress it up as something else.) JSON keeps the simplicity of plain text but adds a little bit of structure (see Example 5-3).

[{"url": "http://maps.example.com/Earth", "description": "Earth"},
 {"url": "http://maps.example.com/Venus", "description": "Venus"},
 ...]

The downside is that neither JSON nor plain text are generally considered “hypermedia” formats. Another popular option is a custom XML vocabulary, either with or without a schema definition (see Example 5-4).

<?xml version="1.0" standalone='yes'?>
<planets>
 <planet href="http://maps.example.com/Earth" name="Earth" />
 <planet href="http://maps.example.com/Venus" name="Venus" />
 ...
</planets>

These days, a custom XML vocabulary seems to be the default choice for web service representations. XML is excellent for representing documents, but I think it’s actually pretty rare that you would have to come up with a custom vocabulary. The basic problems have already been solved, and most of the time you can reuse an existing XML vocabulary. As it happens, there’s already an XML vocabulary for communicating lists of links called Atom.

I cover Atom in detail in Chapter 9. Atom will work to represent the list of planets, but it’s not a very good fit. Atom is designed for lists of published texts, and most of its elements don’t make sense in this context—what does it mean to know the “author” of a planet, or the date it was last modified? Fortunately, there’s another good XML language for displaying lists of links: XHTML. Example 5-5 shows one more representation of the planet list, and this is the one I’m actually going to use.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
 <title>Planet List</title>
</head>
<body>

<ul class="planets">
 <li><a href="/Earth">Earth</a></li>
 <li><a href="/Venus">Venus</a></li>
 ...
</ul>

</body>
</html>

It might seem a little odd to use XHTML, a technology associated with the human web, as a representation format for a web service. I chose it for this example because HTML solves many general markup problems and you’re probably already familiar with it. I’d probably choose it for a real web service, for exactly the same reasons. Though it’s human-readable and easy to render attractively, nothing prevents well-formed HTML from being processed automatically like XML. XHTML is also extensible. I turned a generic XHTML list into a list of “planets” using XHTML’s class attribute. This is a simple example of an XHTML microformat: a way of adding semantic meaning to XHTML’s markup tags. I cover some standard microformats in Chapter 9.

What about the maps themselves? What do I serve if someone asks for a satellite map of the Moon? The obvious thing to send is an image, either in a traditional graphics format like PNG or as a SVG scalar graphic. Except for the largest-scale maps, these images will be huge. Is this OK? It depends on the audience for my web service.

If I’m serving clients with ultra-high bandwidth who expect to process huge chunks of map data, then huge files are exactly what they want. But it’s more likely my clients will be like the users of existing map applications like Google and Yahoo! Maps: clients who want smaller-sized maps for human browsing.

If the client asks for a medium-scale hiking map centered around 43N 71W, it’s surely a waste of bandwidth to send a map of the whole world centered around that point. Instead I should send a little bit of a hiking map, centered around that point, along with navigation links that let the client change the focus of the map. Even if the client asks for a detailed map of the whole world, I don’t need to send the entire map: I can send part of the map and let the client fetch the rest as needed.

This is more or less how the online map sites work. If you visit http://maps.google.com/, you get a political map centered on the continental United States: that’s its representation of “a map of Earth.” If you visit http://maps.google.com/maps?q=New+Hampshire, you get a road map centered on Concord, the capital city. In either case, the map is divided into square “tile” images 256 pixels on a side. The client (your web browser) fetches tiles as needed and stitches them together to form a navigable map.

Google Maps splits the globe into a grid of 256-pixel square tiles, pretty much ignoring issues of latitude and longitude, and generates static images for each tile. It does this 10 times, once for every zoom level. This is efficient (though it does use a lot of storage space), but for pedagogical purposes I’ve chosen a conceptually simpler system. I’m assuming my map service can dynamically generate and serve a 256 ×256 image at any scale, centered on any point of latitude and longitude on any map.

When the client requests a point on a map, I’ll serve a hypermedia file that includes a link to a tiny map image (a single, dynamically-generated tile) centered on that point. When the client requests a map of an entire planet, I’ll pick a point on that planet somewhat arbitrarily and serve a hypermedia file that links to an image centered on that point. These hypermedia files will include links to adjacent points on the map, which will include more links to adjacent points, and so on. The client can follow the navigation links to stitch many tiles together into a map of any desired size.

So Example 5-6 is one possible representation of http://maps.example.com/road/Earth. Like my representation of the list of planets, it uses XHTML to convey resource state and to link to “nearby” resources. The resource state here is information about a certain point on the map. The “nearby” resources are nearby in a literal sense: they’re nearby points.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
 <title>Road Map of Earth</title>
</head>
<body>
...
<img class="map" src="/road.2/Earth/images/37.0,-95.png" alt="Map tile"/>
...
<a class="map_nav" href="46.0518,-95.8">North</a>
<a class="map_nav" href="41.3776,-89.7698">Northeast</a>
<a class="map_nav" href="36.4642,-84.5187">East</a>
<a class="map_nav" href="32.3513,-90.4459">Southeast</a>
...
<a class="zoom_in" href="/road.1/Earth/37.0;-95.8">Zoom out</a>
<a class="zoom_out" href="/road.3/Earth/37.0;-95.8">Zoom in</a>
...
</body>
</html>

Now when a client requests the resource “a road map of Earth” at the URI /road/Earth, the representation they get is not an enormous, insanely detailed image that they can’t deal with. It’s a small XHTML document, one that includes links to several other resources.

A human being can just look at this document and know what it means. A computer program doesn’t have that ability; it has to be programmed in advance by someone who can think about a whole class of these documents and write code to find which bits have the meaning. A web service works by making promises that it will serve representations of resources with a certain structure. That’s why my representation is full of semantic cues like “zoom_in” and “Northeast”. Programmers can write clients that pick up on the semantic cues.

The representation of a road map of Earth, given in Example 5-6, has a lot of links in it. Most of these are links to XHTML documents that look a lot like “a road map of Earth” does: they’re representations of points on the map at various zoom levels. The most important link, though, is the one in the IMG tag. That tag’s src attribute references the URI http://maps.example.com/road/Earth.8/images/37.0,-95.png.

This is a new kind of resource, and I haven’t really considered it before, but it’s not hard to figure out what it is. This resource is “an image centered around 37°N 95.8°W on the road map of Earth.” In my service, the representation of that resource will be a 256 ×256 image showing the geographic center of the continental U.S. (see Figure 5-1).

Figure 5-1. A representation for /road/Earth.3/images/37.0,-95.png. Image data courtesy Google Maps.

The image in Figure 5-1 is 256 pixels square, and represents an area of the Earth about 625 miles square. This image is distinct from the representation of “39°N 95.8°W on the road map of Earth.” that would be an XHTML file like the one in Example 5-6. The XHTML file would include this image by reference, and also link to a lot of nearby points on the map.

Here’s another example: if the client requests /road.8/Earth/32.37,-86.30, my service will send an XHTML representation whose IMG tag references /road.8/Earth/images/32.37,-86.30.png (see Figure 5-2). This is a very detailed road map centered on 32.37°N, 86.30°W on Earth.

Figure 5-2. A representation for /road.8/Earth/images/37.0,-95.png. Image data courtesy Google Maps.

That image too is 256 pixels square, but it represents an area of the Earth only a half-mile square. Scale makes the difference.

The important thing here is not the exact setup of the tile system or the precise format of the URIs to the tile resources. What’s important is what I’m putting into my representations. The URI /road/Earth refers to a resource: “a road map of Earth”. You’d expect a pretty big image as the representation of that resource. You’d at least expect one that showed all of Earth. But my service sends an XHTML document that references a 256 ×256 tile image that doesn’t even cover four U.S. states. How can that document be a good representation for “a road map of Earth”?

A representation conveys the state of its resource, but it doesn’t have to convey the entire state of the resource. It just has to convey some state. The representation of “Earth” (coming up in a bit) isn’t the actual planet Earth, and the representation of “a road map of the Earth” can reference just a simple image tile. But this representation does more than that: the XHTML file links this arbitrarily chosen point on the map to other nearby points on the part of the map directly to the north of this tile, the part directly to the east, and so on. The client can follow these links to other resources and piece together a larger picture. The map is made of a bunch of connected resources, and you can get as many graphical tiles as you need by following the links. So in a sense, this representation does convey all the state there is about the road map of Earth. You can get as much of that state as you want by following its links to other resources.

It’s worth repeating here that if my clients actually need detailed multigigabyte maps, there’s no point in me chopping up the state of the map into these tiny tiles. It’d be more efficient to have the representation of /road/Earth?zoom=1 convey the entire state of the map with one huge image. I’ve designed for clients that only really want part of a map, and wouldn’t know what to do with one huge map of the earth if I gave it to them. The clients I have in mind can consume the XHTML files, load the appropriate images, and automatically follow links to stitch together a map that’s as big as necessary. You could write an Ajax client for my web service that worked like the Google Maps application.

I’ve shown representations for the planet list, for maps of the planets, and for points on the maps. But how are you supposed to get from the planet list to, say, the road map of Earth? Presumably you click “Earth” in the planet list, sending a GET request to /Earth, and get back a representation of Earth. This representation includes a bunch of links to maps of Earth. At this point you follow a second link to the road map of Earth. Well, I just described the representation of Earth. My representation of a planet contains whatever useful information I have about the planet, as well as a set of links to other resources: maps of the planet (see Example 5-7).

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head><title>Earth</title></head>
<body>

 <dl class="place">
  <dt>name</dt> <dd>Earth</dd>
  <dt>maps</dt>
   <dd>
    <ul class="maps">
     <li><a class="map" href="/road/Earth">Road</a></li>
     <li><a class="map" href="/satellite/Earth">Satellite</a></li>
     ...
    </ul>
   </dd>
  <dt>type</dt> <dd>planet</dd>
  <dt>description</dt> 
   <dd>
     Third planet from Sol. Inhabited by bipeds so amazingly primitive 
     that they still think digital watches are a pretty neat idea.
   </dd>
 </dl>

</body>
</html>

I’ve chosen to represent places as lists of key-value pairs. Here, the “place” is the planet Earth itself. Earth in this system is a named place, just like San Francisco or Egypt. I’m representing it using the dd tag: HTML’s standard way of presenting a set of key-value pairs. Like any place, Earth has a name, a type, a description, and a list of maps: links to all the resources that map this place.

Why am I representing a planet as a place? Because now my clients can parse the representation of a planet with the same code they use to parse the representation of a place. Example 5-8 is a representation for Mount Rushmore on Earth. You might get this XHTML file back in response to a GET request for /Earth/USA/Mount%20Rushmore.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head><title>Mount Rushmore</title></head>
<body>

<ul class="places">

<li>
<dl class="place">
 <dt>name</dt> <dd>Mount Rushmore</dd>
 <dt>location</dt>
   <dd>
    <a class="coordinates" href="/Earth/43.9,-95.9">43.9°N 95.8°W</a>
   </dd>
 <dt>maps</dt> 
  <dd>
   <ul class="maps">
    <li><a class="map" href="/road/Earth/43.9,-95.9">Road</a></dd>
    <li><a class="map" href="/satellite/Earth/43.9,-95.9">Satellite</a></li>
    ...
   </ul>
  </dd>
 <dt>type</dt> <dd>monument</dd>
 <dt>description</dt> 
  <dd>
   Officially dedicated in 1991. Under the jurisdiction of the 
   <a href="http://www.nps.gov/">National Park Service</a>.
  </dd>
</dl>
</li>
</ul>

</body>
</html>

Rather than serve a map image of Mount Rushmore, or even an XHTML page that links to that image, this representation links to resources I’ve already defined: maps of the geographical point where Mount Rushmore happens to be located. Those resources take care of all the imagery and navigation details. The purpose of this resource is to talk about the state of the place, and what it looks like on a map is just one bit of that state. There’s also its name, its type (“monument”), and its description. The only difference between the representation of a planet and that of a place is that a place has a location in its definition list, and a planet doesn’t. A client can parse both representations with the same code.

You may also have noticed that you don’t have to write a special client for this web service at all. You can use a plain old web browser. Starting at the home page (http://maps.example.com/), you click a link (“Earth”) to select a planet. You get the representation shown in Example 5-7, and you click “Road” to see a road map of Earth. Then you navigate that map by clicking links (“North,” “Zoom out”). My web service is also a web site! It’s not a very pretty web site, because it’s designed to be used by a computer program, but nothing prevents a human from consuming it (or debugging it) with a web browser.

If you get only one thing out of this book, I hope it’s that this idea starts seeming natural to you (assuming it didn’t before). Web services are just web sites for robots. My map service is particularly web site-like: it connects its resources together with hypermedia, the hypermedia representations happen to be HTML documents, and (so far) it doesn’t use any features that web browsers don’t support. But all RESTful resource-oriented web services partake of the nature of the Web, even if you can’t use them with a standard web browser.

Example 5-9 shows one more representation: the representation of a point on the map.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
 <title>43.9°N 103.46°W on Earth</title>
</head>
<body>

<p> 
 Welcome to 
 <a class="coordinates" href="/Earth/43.9,-103.46">43.9°N 
  103.46°W</a> 
 on scenic <a class="place" href="/Earth">Earth</a>.
</p>

<p>See this location on a map:</p>

<ul class="maps">
 <li><a class="map" href="/road/Earth/43.9,-95.9">Road</a></li>
 <li><a class="map" href="/satellite/Earth/43.9,-95.9">Satellite</a></li>
 ...
</ul>

<p>Things that are here:</p>

<ul class="places">
 <li><a href="/Earth/43.9,-95.9/Mount%20Rushmore">Mount Rushmore</a></li>
</ul>

<form id="searchPlace" method="get" action="">
 <p>
  Show nearby places, features, or businesses: 
  <input name="show" repeat="template" /> <input class="submit" />
 </p>
</form>

</body>
</html>

This representation consists entirely of links: links to maps centered around this point, and links to places located at this point. It has no state of its own. It’s just a gateway to other, more interesting resources.

I’ve shown representations for the planet list, for a planet, for points and places on a planet, and for the maps themselves. What about my algorithmic resources, the search results? What’s a good representation of the resource “diners near Mount Rushmore” (/Earth/USA/Mount%20Rushmore?show=diners)? What about “Areas of high arsenic near 24.9195°N 17.821°E” (/Earth/24.9195,17.821?show=arsenic)?

A list of search results is of course associated with the place being “searched,” so a representation of “diners near Mount Rushmore” should link to the place “Mount Rushmore.” That’s a start.

When the client searches in or around a place, they’re searching for more places. Whether the search string is an ambiguous place name (“Springfield”) or a more general description of a place (“diners,” “arsenic”), the results will be places on the map: cities named Springfield, diners, or sites with high arsenic readings. So a list of search results takes one place (“Mount Rushmore”), and associates certain other places (“Joe’s Diner”) with it.

A list of search results, then, can be nothing but a list of links to resources I’ve already defined: named places on the map. If the client is interested in a place, it can follow the appropriate link and find out about its state.

Example 5-10 shows the representation of a set of search results. The search is an attempt to find places called “Springfield” in the United States: its URI would be /Earth/USA?show=Springfield.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head><title>Search results: "Springfield"</title></head>
<body>

<p>
 Places matching <span class="searchterm">Springfield</span> 
 in or around 
 <a class="place" href="/Earth/USA">the United States of America</a>:
</p>

<ul>
 <li>
  <a class="place" href="/Earth/USA/IL/Springfield">Springfield, IL</a>
 </li>
 <li>
  <a class="place" href="/Earth/USA/MA/Springfield">Springfield, MA</a>
 </li>
 <li>
  <a class="place" href="/Earth/USA/MO/Springfield">Springfield, MO</a>
 </li>
 ...
</body>
</html>

This representation is made up almost entirely of links to places. There’s the link to the place that was searched, “the United States of America” (a place of type “country”). There are also links to various places that matched the search criteria. Each of these places is a resource, and exposes a representation that looks like Example 5-8. Each link contains enough scoping information to uniquely identify the Springfield in question.

A client can follow the links in the representations to find information about the places, as well as maps of them. Figure 5-3 shows some of the links you can follow from /Earth/USA?show=Springfield.

Figure 5-3. Where can you go from the list of search results?

Google Maps presents search results by sewing image tiles together to make a large-scale map, and annotating the map with graphical markers. You can write a client for this web service that does the same thing. The first step is to build the large-scale map. The client follows the initial link to /Earth/USA and gets a representation like the one in Example 5-4. This gives the client the address of one graphical tile. The client can get adjacent tiles by following navigation links, and stitch them together into a large-scale tile map of the whole country.

The second step is to stick markers on the map, one for each search result. To find out where a marker should go, the client follows one of the search result links, fetching a representation like the one in Example 5-8. This representation lists the latitude and longitude of the appropriate Springfield.

That’s potentially a lot of link following, but my representations are simple and so are the rules for going from one to another. I’ve spread out my data set over a huge number of resources, and made it easy to find the resource you want by following links. This strategy works on the human web, and it works on the programmable web too.

This step of the design is conceptually simple, but it’s the gateway to where you’re going to spend much of your implementation time: making sure that client requests are correctly turned into responses.

Most read-only resources have a pretty simple typical course of events. The user sends a GET request to a URI, and the server sends back a happy response code like 200 (“OK”), some HTTP headers, and a representation. A HEAD request works the same way, but the server omits the representation. The only main question is which HTTP headers the client should send in the request, and which ones the server should send in the response.

The HTTP response headers are a fairly large toolkit, and most of them don’t apply to this simple service. (For descriptions of the standard HTTP headers, see Appendix C.) In my service, the main HTTP response header is Content-Type, which tells the client the media type of the representation. My media types are application/xhtml+xml for the map representations and search results, and image/png for the map images. If you’ve done any server-side web programming you already know about Content-Type: every HTTP server and framework uses it.

I don’t use HTTP request headers very often. I think it’s best if the client can tweak the representation by tweaking the URI to the resource, rather than tweaking the request headers. But there is one set of headers that I think ought to be built into every HTTP client, every web service, and every service hacker’s brain: the ones that make conditional GET possible.

Last-Modified: Thu, 30 Nov 2006 20:00:51 GMT

GET /road/Earth HTTP/1.1
Host: maps.example.com
If-Modified-Since: Thu, 30 Nov 2006 20:00:51 GMT

I also need to plan for requests I can’t fulfill. When I hit an error condition I’ll send a response code in the 3xx, 4xx, or 5xx range, and I may provide supplementary data in HTTP headers. If they provide an entity-body, it’ll be a document describing an error condition, not a representation of the requested resource (which, after all, couldn’t be served).

I provide a full list of the HTTP response codes in Appendix B, along with examples where you might use each of them. Here are some likely error conditions for my map application:

Why have I decided to design my user accounts differently from those of nearly every existing web service? I have two reasons. First: most web services make you sign up for an account through a web application. Web application design is a well-understood topic and it’s not the topic of this book. Web services are indeed very similar to web applications, but resource creation is one of the places where they differ. The main difference here is that HTML forms currently support only GET and POST. This means web applications must use overloaded POST to convey any unsafe operation. If I tried to cover the typical method of getting a user account, I’d end up skimming the details as not relevant to web services. Treating user accounts as read/write resources means I can demonstrate the new resource-oriented design procedure on a data structure you’re probably familiar with.

Second, I want to show that new possibilities open up when you treat everyday data structures as resources, subject to the uniform interface. Consider an Internet-connected GPS device that ties into my map service. Every hour or so, it annotates Earth (as exposed through my web service) with its current position, creating a record of where the GPS device is over time.

There will be thousands of these devices, and each one should only be able to see its own annotations. The person in charge of programming the device should not be limited to creating a single user account for personal use. Nor should everyone who buys the device have to go to my web site and fill out a form before they can use the device they bought.

Since user accounts are resources, every one of these devices can have its own account on my web service (possibly with a username based on the serial number), and these accounts can be created automatically. They might be created in batches as the devices are manufactured, or each one may create an account for itself when its owner first turns it on.

The end users may never know that they’re using a web service, and they’ll never have to sign up for a key. The device programmer does need to know how our web service works, and needs to write software that can create user accounts. If user accounts are resources, it’s obvious how the device programmer can do this. HTTP’s uniform interface gives most of the answers ahead of time.

Once I start exposing user accounts, I need some way of determining which user, if any, is responsible for a given HTTP request. Authentication is the problem of tying a request to a user. If you want to name a new place on Mars, I need some way of knowing that the new place should be associated with your user account instead of someone else’s. Authorization is the problem of determining which requests to let through for a given user. There are some HTTP requests I’d accept from user A but reject from user B: requests like “DELETE user A” or “GET all of user A’s private places.” In my service, if you authenticate as user A, you’re allowed to manipulate user A’s account, but not anyone else’s.

I’ll have more to say about RESTful modes of authentication and authorization in Chapter 8, but here are the basics. When a web service client makes an HTTP request, it may include some credentials in the HTTP header Authorization. The service examines the credentials, and decides whether they correctly identify the client as a particular user (authentication), and whether that user is actually allowed to do what the client is trying to do (authorization). If both conditions are met, the server carries out the request. If the credentials are missing, invalid, or not good enough to provide authorization, then the server sends a response code of 401 (“Unauthorized”). It sets the WWW-Authenticate response header with instructions about how to send correct credentials in the future.

There are several standard kinds of authentication. The most common are HTTP Basic, HTTP Digest, and WSSE. Some web services implement custom forms of authentication: in Chapter 3 I showed how Amazon S3 implements authentication with a sophisticated request signing mechanism. It doesn’t really matter which authentication mechanism I choose since I’m not actually implementing this service, but let’s say I go with the simplest choice: HTTP Basic authentication.

There’s also the notion of privacy. Given that user A’s list of private annotations can’t be accessed by any other user, the representation of that list still needs to be transmitted over the Internet. The data’s going to go through a lot of computers before it gets to the client. What’s to stop one of those computers from examining the supposedly private list? To solve this problem I’m going to encrypt each HTTP transaction over SSL. In the previous chapter I presented URIs that started with http://maps.example.com/. In this chapter my URIs all start with https://maps.example.com/.

Using HTTPS instead of HTTP prevents other computers from eavesdropping on the conversation between client and server. This is especially important when using HTTP Basic authentication, since that authentication mechanism involves the client sending its credentials in plain text.

Now I’ve got a secure, trusted means of communication between the client and the server. But there’s one more relationship to consider: the relationship between the client software and the human end user. Why should the end user trust the client software with its authentication credentials? Let me ask you a question to clarify the problem. Whenever you log in to a web site, you’re trusting your web browser to send your username and password to that web site, and nowhere else. Why do you trust your browser with that information? How do you know your browser doesn’t have a secret backdoor that broadcasts everything you type to some seedy IRC channel?

There are several possible answers. You might be using an open source browser like Firefox, which has good source control and a lot of people looking at the source code. You might say there’s safety in numbers: that millions of people use your brand of browser and there haven’t been any problems traceable to the browser itself. You might monitor your network traffic to make sure your browser is only sending the data you tell it to send. But most people just take it on faith that their web browser is trustworthy.

That’s the human web. Now imagine I send you a cool new web service client for managing your del.icio.us bookmarks. Do you trust that client with your del.icio.us username and password? Do you trust it as much as you trust your web browser with the same information? Hopefully not! No web service client is as popular as a web browser, and no web service client has as many eyes on the source code. On the human web, we usually ignore the problem by taking a leap of faith and trusting our web browsers. On the programmable web the problem is more obvious. We don’t necessarily trust our own clients with our authentication credentials.

There’s nothing in the HTTP standard to deal with this problem, because it’s a problem between the end user and the client: HTTP lives between the client and the server. Solving this problem requires forgoing all the standard ways of sending authentication information: Basic, Digest, and WSSE don’t work because they require the client to know the credentials. (You can solve it with Digest or WSSE by having a tiny, trusted account manager send encrypted authentication strings to the actual, untrusted client. I don’t know of any web service clients that use this architecture.)

Big names in web services like Google, Amazon, eBay, and Flickr have come up with ways for a client to make web service requests without knowing the actual authentication credentials. You saw a hint of this in Chapter 3: I showed how to sign an Amazon S3 request and give a special URI to someone else, which they could use without knowing your password. I’ll have more to say about this in Chapter 8. For now I just want you to know that there’s a complication on the programmable web you might never have considered. Because there’s not yet any standard way of solving this problem, I’m going to punt on it for now and use HTTP Basic authentication for my services. My users will have to trust their clients as much as they trust their web browsers.

Now that I’ve identified a new data set (user accounts), I’m going to go through the same design procedure I did for the data set I developed in the previous chapter (planets, places on the planets, maps of the planets, and points on the maps). But the procedure from the previous chapter only suffices for read-only resources. This chapter makes it possible for clients to create, modify, and delete resources. So I’ve added two steps to the procedure (steps 4 and 5).

Most sites with user accounts try to associate personal information with your account, like your name or email address. I don’t care about any of that. In my map service, there are only two pieces of information associated with a user account:

Each user account also has some subordinate resources (custom places on planets) associated with it, but I’ll figure that part out later. All I need for now is a way of identifying specific user accounts (a username), and a way for a client to present credentials that tie them to a certain user account (a password).

Since I don’t track any personal information, there’s no reason apart from tradition to even call this a “user account.” I could call it a “password-protected set of annotations.” But I’ll stick to the traditional terminology. This makes it easier to visualize the service, and easier for you to come up with your own enhancements to the user account system.

This was a fairly large step back in Chapter 5, when my data set was large and vague: “planets, places, and maps.” Here the data set is fairly constrained: “user accounts.” I’ll expose each user account as a resource. In terms of the Chapter 5 terminology, these new resources are resources of the second type. They’re the portals through which my service exposes its underlying user objects. Another site might also expose the list of user accounts itself as a one-off resource, or expose algorithmic resources that let a client search the list of users. I won’t bother.

This part is also easy, since I only have one kind of resource. I’ll expose a user account with a URI of the following form: https://maps.example.com/user/{user-name}.

This is the first new step. I skipped it when designing read-only resources, because there was nothing to decide. By definition, read-only resources are the ones that expose no more than the HTTP methods GET, HEAD, and OPTIONS. Now that I’re designing resources that can be created and modified at runtime, I also have PUT, POST, and DELETE to consider.

Even so, this step is pretty simple because the uniform interface is always the same. If you find yourself wishing there were more HTTP methods, the first thing to do is go back to step two, and try to split up your data set so you have more kinds of resources. Only if this fails should you consider introducing an element of the RPC style by making a particular resource support overloaded POST.

To reiterate the example from Chapter 5: if you have resources for “readers,” and resources for “published columns,” and you start thinking “it sure would be nice if there was a SUBSCRIBE method in HTTP,” the best thing to do is to create a new kind of resource: the “subscription.” As HTTP resources, subscriptions are subject to HTTP’s uniform interface. If you decide to forgo the uniform interface and handle subscriptions through overloaded POST on your “reader” resources, defining the interface for those resources becomes much more difficult.

I can decide which bits of the uniform interface to expose by asking questions about intended usage:

From those two questions I get my first result. To create a user account, a client will send a PUT request to the account’s URI. If the answer to the second question was “the server’s in charge of the final URI,” I’d expect my clients to create a user by sending a POST request to some “factory” or “parent” URI. See the Custom Places” section later in this chapter for a case where the answer to the second question is “the server’s in charge.”

My data set comes with no built-in user accounts: every one is created by some client. The obvious next step in this design is to specify how the client is supposed to create a user account.

Let’s go back to Chapter 3 and Amazon S3 for a minute. A client creates an S3 bucket by sending an empty PUT request to the URI of the bucket. The client doesn’t need to send an entity-body in the request, because the bucket has no state other than its name.

To create an S3 object inside a bucket takes a little more work. An S3 object has two bits of state: name and value. The name goes into the URI, the destination of the PUT request. But the value needs to go into the entity-body of the PUT request. S3 will accept any data at all in this entity-body, because the whole point is that the value of an S3 object can be anything, but there needs to be something there: you can’t have an empty object.

Most web services are a little pickier about what goes into the entity-body: it has to be in a certain format and convey certain bits of resource state. My user accounts have two elements of resource state: the username and the password. If a PUT request is going to succeed in creating a user account, it needs to convey both pieces of state. The username is included in the scoping information: any PUT request that creates an account will have that account’s username in the URI. What about the password?

The client will send the new user’s password in an entity-body, as part of a representation. In Chapter 5, I introduced representations as documents the server sends the client: a way for the server to convey the state of a resource. Representations flow the other way, too. They’re how a client suggests changes to the state of a resource. When you PUT an S3 object, the entity-body you send is a representation of the object. The representation you send with a PUT request is an assertion about the new state of a resource.

In Representing the List of Planets” in Chapter 5 I considered several possible representation formats. I looked at plain text, JSON, XML using a made-up vocabulary, and Atom (XML again, but using a preexisting vocabulary). I decided on XHTML, a preexisting XML vocabulary oriented around marking up human-readable documents. In that chapter the question was what format would be most useful when served to the client. Now, the question is how the client should format its proposed state changes. What format makes it easiest for the client to convey a password to the server?

When the state is complex, it’s helpful for the server to accept the same representation format it sends. The client can request a representation with GET, modify the representation, and then PUT it back, committing its changes to the underlying resource state. As we’ll see in Chapter 9, the Atom Publishing Protocol uses this technique effectively. And, of course, S3 serves the representation of an object byte for byte the way it was when the client first PUT it into the system. S3 doesn’t even pretend to know anything about the meaning of the representations it serves.

Here, I’ve only got one item of state (the password), and it’s not one that the server will ever send to the client. Now’s a good time to introduce a representation format for simple cases like these.

<form action="http://www.example.com/color-pairs" method="POST">
 <input name="color1" type="text"/>
 <input name="color2" type="text"/>
</form>

My map service accepts a form-encoded representation when a client tries to create or edit a user. The only pieces of state I’ve associated with a user are its name and password. The name goes into the URI and I’ve decided it can’t change, so my user representations just look like “password={the-password}”. Example 6-2 is hypothetical Ruby code for creating a user account with the map service.

require 'rubygems'
require 'rest-open-uri'
require 'cgi'
require 'uri'
def make_user(username, password)
  open("https://maps.example.com/user/#{URI.escape(username)}", 
       :data => CGI::escape("password=#{password}"), :method => :put)
end

Changing an account’s password is the same as creating the account in the first place. The client sends a PUT request to the account URI, with a new representation of the account (that is, the new password). Of course, no one can change an account’s password without authorization. To modify a user account, a client must also provide an Authorization header that convinces my service it has the right to modify that account. In short, changing a user’s password requires knowing the current password. As I said earlier, my service expects incoming Authorization headers to conform to the HTTP Basic authentication standard.

A DELETE request never requires a representation, but deleting a user from my service will require a proper Authorization header. That is: to delete a user account you must know that user’s password.

A client will GET a user account’s URI to retrieve a representation of a user account, just as a client GETs the URI of a map or a place to retrieve a representation of that map or place. What should the representation of a user account look like?

Right now it won’t look like much, since I’ve only got two pieces of state to convey, and one of them (the password) I don’t want to be sending out. Indeed, in a well-designed system I won’t even have the password to send out. I’ll only have an encrypted version of it, for use in authentication. Once I integrate custom places into this representation, it’ll look better. For now, Example 6-3 is a fairly sparse XHTML document.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
 <title>User homepage for leonardr</title>
</head>
<body>

<p class="authenticated">
 You are currently logged in as 
 <a class="user" href="/user/leonardr">leonardr</a>.
</p>

<p>User homepage for 
   <a class="user" href="/user/leonardr">leonardr</a></p>

<form id="modifyUser" method="put" action="">
 <p>Change your password:
   <input class="password" name="password" /><br />
   <input class="submit" /></p>
</form>

</body>
</html>

Once again I’m using the representation to convey the current resource state, and to help the client drive to other states. I used an HTML form to describe a future PUT request the client might make if it wants to change the user’s password (an item of resource state). Note that there’s no form telling the client how to get a representation, or how to delete this user. It’s taken for granted that you use HTTP GET and DELETE for that. I only need hypermedia for complicated things: links to other resources (so the client knows which URI to GET or DELETE), and descriptions of representations.

If you GET someone else’s user account, you’ll be served a different representation, similar to the one in Example 6-4.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
 <title>User homepage for samruby</title>
</head>
<body>

<p class="authenticated">
 You are currently logged in as 
 <a class="user" href="/user/leonardr">leonardr</a>.
</p>

<p>User homepage for <a class="user" href="/user/samruby">samruby</a></p>

</body>
</html>

This representation has no controls for altering the state of the resource, because the client isn’t authorized to do that: the client authenticated as leonardr and this is samruby’s page. Right now the representation does nothing but confirm to leonardr that a user named samruby exists. If there was no such user, a GET request to /user/samruby would give a status code of 404 (“Not Found”), and the client would be free to create samruby with a PUT request.

In the previous chapter I defined several classes of resource: the list of maps, individual maps, places, and lists of places (that is, lists of search results). None of these are directly relevant to user accounts, but there are a couple of nice features I can add at this point.

One nice feature is to add the “authenticated” message (seen in the two sample representations above) to the representation of every resource. It’ll be displayed whenever the client submits a request with valid credentials. The “authenticated” message is a piece of hypermedia that shows an authenticated client how to retrieve data about its user account. Every resource is now connected to the user account of the user who requested it.

Another nice piece of hypermedia would be one that shows an unauthenticated client how to create a user account. The best place for this bit of hypermedia would be the representation of the list of planets: after all, that’s the service’s “home page.” It already contains links to the other main parts of the service, so it should contain a link to this new part.

Once again, HTML hypermedia isn’t quite up to the job. And once again, I’m going to use XHTML 5, which makes minor changes to HTML, rather than introduce a totally new technology like WADL in the middle of a chapter. Example 6-5 is an XHTML 5 snippet that tells a client how to create a user.

<form id="createUser" method="PUT" template="/user/{username}">
 <p>Username: <input type="text" name="username" /><br />
 <p>Password: <input type="password" name="password" /><br />
 <input class="submit" />
</form>

The hypermedia form talks about the syntax of the PUT request, but it can’t say much about the semantics. A web service client can read the HTML form in Example 6-5, but its understanding is limited. It knows that the form is labelled “createUser” but it doesn’t know what “createUser” means. It knows that if it PUTs a certain representation to a certain URI, the server will probably accept it. It knows what PUT means, because PUT always means the same thing. It knows that the representation should include a “username,” but it doesn’t know a username from an ostrich. It takes a human being—a programmer—to understand what a user is, that “createUser” means “create a user,” what a username is, and all the rest. A programmer needs to set the rules about when and how user accounts are created. This piece of hypermedia does nothing but tell the client how to structure the PUT request when it comes time to “createUser,” whatever that means. It’s a promise from the web service to the client.

Many web services put all of this data up front, in a single WSDL or WADL file, for the ease of the client programmer. This is somewhat contrary to the REST design philosophy because it violates, or at the very least subverts, the principle of connectedness. But in web services, where the client must be programmed in advance, it’s an understandable impulse, and often it doesn’t cause any problems.

Let’s consider what might happen when a client sends a PUT request to /user/leonardr. As is usual with HTTP, the server reads this request, takes some action behind the scenes, and serves a response. I need to decide which numeric response code the response will have, and what HTTP headers and/or entity-body will be provided. I also need to decide how the request will affect resource state: that is, what real-world effects it will have.

It’s not hard to see what happens if all goes well with a PUT request. If there’s no user called “leonardr,” the service creates one with the specified password. The response code is 201 (“Created”), and the Location header contains the URI of the newly created user.

If the user account already exists, the resource state is modified to bring it in line with the client’s proposed new representation. That is, the account’s password is modified. In this case the response code may be 200 (“OK”), and the response entity-body may contain a representation of the user account. Or, since the password change never affects the representation, the response code may be 205 (“Reset Content”) and the response entity-body may be omitted altogether.

PUT requests are the only complicated ones, because they’re the only ones that include a representation. GET and DELETE requests work exactly according to the uniform interface. A successful GET request has a response code of 200 (“OK”) and a representation in the entity-body. A successful DELETE request also has a response code of 200 (“OK”). The server can send an entity-body in response to a successful DELETE, but it would probably contain just a status message: there’s no longer a resource to send a representation of.

A request that creates, modifies, or deletes a resource has more failure conditions than one that just retrieves a representation. Here are a few of the error conditions for this new resource.

The most obvious problem is that the client’s representation might be unintelligible to the server. My server expects a representation in form-encoded format; the client might send an XML document instead. The status code in this case is 415 (“Unsupported Media Type”).

Alternatively, the client might not have provided a representation at all. Or it might have provided a form-encoded representation that’s ill-formed or full of nonsense data. The status code in this case is 400 (“Bad Request”).

Maybe the representation makes sense but it tells the server to put the resource into an inconsistent or impossible state. Perhaps the representation is “password=”, and I don’t allow accounts with empty passwords. The exact status code depends on the error; in the case of the empty password it would probably be 400 (“Bad Request”). In another situation it might be 409 (“Conflict”).

Maybe the client sends the wrong credentials, sends authorization credentials for a totally different user account, or doesn’t send the Authorization header at all. A client can only modify or delete a user if it provides that user’s credentials. The response code in this case is 401 (“Unauthorized”), and I’ll set the WWW-Authenticate header with instructions to the client, giving a clue about how to format the Authorization header according to the rules of HTTP Basic authentication.

If the client tries to create a user that already exists, one possible response code is 409 (“Conflict”). This is appropriate because carrying out the PUT request would put the service’s resources into an inconsistent state: there’d be two user resources with the same username. Another possibility is to treat the PUT request as an attempt to change an existing user’s password without providing any authentication, and send a response code of 401 (“Unauthorized”).

As in the previous chapter, there might be an unspecified problem on the server side: response code 500 (“Internal Server Error”) or 503 (“Service Unavailable”).

A web service client can create any number of places on any of the planets for which I have maps. Custom places will show up in lists of search results, just like the built-in places from the previous chapter. Custom places can have the same data as built-in places: a type (“city”), a name (“Springfield”), coordinates of latitude and longitude (“39.81E 89.64W”), and a textual description (“The capital of Illinois”). Many custom places may share the same coordinates (“My house” and “My current location”), and a custom place may share a location with a built-in place.

Every custom place is associated with some user account. Custom places may be public or private. A private place is visible and modifiable only to someone who provides the credentials for the user account that “owns” the place.

Each custom place will be a resource, just as every built-in place is. I also want to let clients get a list of their custom places. In my design, a user account is just a password-protected list of places, so I won’t be exposing the place list as a separate resource. Instead I’ll expand the “user account” resource so it encompasses a user’s list of places. This is analogous to the way a bucket in Amazon S3 is represented as nothing but a list of objects.

A custom place is clearly a subordinate resource, but subordinate to what? I could reasonably associate it with a user account, a geographic point on some planet, or an enclosing place like a city, country, or planet. Which of these relationships should I capture with my URIs?

I’ve chosen to name custom places much the same way I name built-in places. Each place is associated with a geographic point, and can be accessed with a URI of the form /user/{username}/{planet}/{latitude},{longitude}/{place name}. The new element is {username}, intended to distinguish between different people’s views of the same place: for instance, Sam’s review of Joe’s Diner at /user/samruby/Earth/45.2,-114.2/Joe’s%20Diner and Leonard’s less glowing review at /user/leonardr/Earth/45.2,-114.2/Joe's%20Diner.

A URI like /Earth/USA?show=Joe's+Diner works like it did before: it returns search results for places called “Joe’s Diner,” anywhere in the U.S. The only difference is that now there are more possible places to search: not only the built-in database of places, but each user’s public list of places, and your own private list.

Built-in places are still privileged. As it happens, there’s a Japanese theme park that includes a one-third scale model of Mount Rushmore. If a client creates a custom place called “Mount Rushmore” north of Tokyo, /Earth/Mount%20Rushmore still points to the original in South Dakota. It doesn’t suddenly become ambiguous which “Mount Rushmore” resource that URI refers to. However, /Earth?show=Mount+Rushmore will show both places.

Clients can use GET and HEAD to retrieve representations of built-in places, their own places (whether public or private), and public places created by others. Clients can delete their own places with DELETE, and change the state of their places with PUT.

There are two ways a client might create a map annotation. The client might add a comment to an existing place on the map (“Mount Rushmore”), or it might give a new name to a certain point of latitude and longitude (“the cornfield where I kissed Betty”).

In the first case, the resource being created is “Mount Rushmore (from leonardr’s point of view).” When creating this resource the client shouldn’t have to know exactly where on the map Mount Rushmore is. “Mount Rushmore” is a consensus name and there’s a built-in place by that name. The client can rely on the server to look up the coordinates. In the second case, the resource being created is a brand new place that the server’s never heard of, and the client is responsible for knowing the coordinates.

How can I work this feature into my resource-oriented design? “Mount Rushmore (from leonardr’s point of view)” is a subordinate resource of another resource: the built-in place “Mount Rushmore.” This resource already exists and has a URI: one of them is /Earth/Mount%20Rushmore. If the client wants to reuse the consensus name for a place, it shouldn’t have to look up its location. Instead of figuring out the final URI of the annotation and sending a PUT request to it, the client can send a POST request to the “Mount Rushmore” URI and let the server figure out the ultimate URI.

Similarly, if the client wants to comment on the Alabama capitol building, it can POST to /Earth/USA/AL/State%20capitol instead of figuring out the exact coordinates or street address. Any URI that identifies a built-in place can be the target of a POST request that comments on that place.

What about custom names? What if a client wants to give the name “Mount Rushmore” not to the original in South Dakota, but to the scale model in Imaichi? What if the client wants to create an annotation for “the cornfield where I kissed Betty”?

Here the client must know the latitude and longitude of the place it wants to create. This means it’ll have all the information necessary to create the URI of the new resource: the world, a geographic point on the world, the name of the place, and its own username. The client could make a PUT request to a URI like /user/bob/Earth/42,-93.7/the%20cornfield%20where.... This would work just like creating a user account by sending a PUT request to /user/bob.

Even here, it’s cleaner to use POST. A brand-new place on the map is a subordinate resource: it’s subordinate to some point on the planet, just like a comment on a built-in place is subordinate to a place on the planet. So a client could also put a new place on the map by sending a POST request to /Earth/42,-93.7. It works the same way as a comment on existing places (a POST to /Earth/Mount%20Rushmore), except here the place is identified by latitude and longitude, not by consensus name.

My service will support POST for brand-new places because that’s simpler. The interface will be the same whether you’re adding a brand new place to the planet, or making a comment on some consensus place. Another service might support both methods: PUT to the final URI if the client is willing to figure out that URI, and POST to a parent URI if it’s not.

Finally, note that although I’m using POST, it’s not overloaded POST. Clients of my service use POST only when they want to create a resource “beneath” an existing one. The URI of the new resource (/user/leonardr/Earth/43.9,-103.46/Mount%20Rushmore) may not directly extend the URI of the old (/Earth/Mount%20Rushmore), but the resources have a conceptual relationship.

When the client sticks a pin into a planet and creates a custom place, what information does it need to provide? It must identify a planet and a place on that planet: the spot where the pin goes. The place can be identified either by latitude and longitude, or by reference to a canonical name like “Mount Rushmore.” Call these variables planet, latitude, longitude, and name. The server must know what type of place the client is putting on the map. A place may be public or not, and the client may provide a custom description of the place. The final URI also incorporates a username, but the client is already providing that, in the Authorization header. There’s no need to make the client send that information twice.

These are all key-value pairs. I can have clients represent places the way they represent user accounts: as form-encoded strings. There are no complex data structures here that might call for a JSON or XML representation.

Client requests may choose to send some key-value pairs and omit others. Information that’s in the URI as scoping information doesn’t need to be repeated in the representation. When the client sends a POST to /Earth/43.9,-103.46 it doesn’t need to specify latitude and longitude, because that information’s in the URI. It does need to specify name and type.

When the client sends a POST to /Earth/Mount%20Rushmore it shouldn’t specify latitude, longitude, or name. The client is making a new place based on a well-known existing place, and the new place will inherit the name and location of the existing place. The client may specify a custom type (“national-park,” “political,” “places in North Dakota”) or inherit the default (“monument”).

The client may always choose to omit description and public. My service sets default values for those variables: descriptions are empty by default, and places are public by default.

When the client modifies one of its custom places, anything and everything about the place might change: its name, its location, its type, its description, or its public status. The PUT request that modifies a place can specify the same key-value pairs used to create a place, in any combination. The server will make the appropriate changes, assuming the changes make sense.

Example 6-6 shows a sample HTTP POST request that creates a new custom place. Combined, the form-encoded representation and the scoping information in the URI convey all required states for the new resource. The name and location of the new resource come from the scoping information; its type and description come from the representation. Since the representation doesn’t specify a value for public, the default takes over and this new resource is made public.

POST /Earth/USA/Mount%20Rushmore HTTP/1.1
Host: maps.example.com
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

type=national-park&description=We%20visited%20on%203/5/2005

Most of the work here is already done. In Chapter 5 I defined an XHTML-based representation format for places. Custom places look the same as places from the built-in database.

The only new part is this: when an authenticated client requests a representation of one of its custom places, our service will tack onto the representation some hypermedia showing the client how to edit that place (see Example 6-7). I don’t need to tell clients how to delete the place: the uniform interface takes care of that. But I do need to convey the information I wrote in prose above: that a place is defined by planet, latitude, longitude, and so on.

<form id="modifyPlace" method="PUT" action="">
 <p>Modify this place:</p>

 <p>
  Name: <input name="name" value="Mount Rushmore" type="text" /><br />
  Type: <input name="type" value="national-park" type="text" /><br />
  Position: 
   <input name="latitude" value="43.9" type="text" />,
   <input name="longitude" value="-103.46" type="text" /><br />
  Description:
   <textarea name="description">We visited on 3/5/2005</textarea><br />
  Public?
   <input name="public" type="checkbox" value="on"/>
  <input type="submit" />
 </p>
</form>

The caveats from earlier apply here too. This isn’t valid XHTML 4, though it is valid XHTML 5, because it specifies PUT as its method. Also, a client doesn’t know what to do with this form unless it’s been programmed in advance. Computers don’t know what “modifyPlace” means or what data might be a good value for “latitude.”

Because clients have to be programmed in advance to understand these forms, most of today’s services don’t include a form for modifying a resource in that resource’s representation. They either serve all the forms up front (in a WSDL or WADL file), or they specify them in prose (as I did above) and leave it for the service programmer to figure out. It’s debatable whether it’s really helpful to serve forms along with representations, but serving them is better than just specifying the API in prose and making the programmer implement it.

I’ve got three kinds of integration to do. The first is data integration. When you DELETE a user account, the account’s custom places—everything under /user/{username}—should also be deleted. URIs to these resources used to work, but now they will return a response code of 410 (“Gone”) or 404 (“Not Found”).

The other kinds of integration should be familiar by now. They involve changing the representations of existing resources to talk about the new one. I want search results to link to custom places. I want points on the globe to show how the user can create a custom place at that point. I want to improve my connectedness by connecting “custom place” resources to the resources I defined already.

The rather empty-looking representation of a user’s account, seen in Example 6-3, badly needs some link-based integration. This is the ideal place to list a user’s custom places. I’ll represent the place list with the same XHTML list of links I use to represent search results.

In the service defined in Chapter 5, a client that searched for places called “Mount Rushmore” (/Earth?show=Mount+Rushmore) would only find places from my built-in place database: probably only the “consensus” location of Mount Rushmore in South Dakota. In the new version of the service, there’s likely to be more than one result. In the new version, that search will also return other users’ annotations for Mount Rushmore, and other places that users have named “Mount Rushmore,” like the scale model in Imaichi.

This is the same case as in Chapter 5, where the built-in place database contained more than one “Joe’s diner.” I present search results in a list, each linking to a specific resource. All I’m doing is expanding the search. A search result may be a place in the built-in database, a custom place created by some other user and exposed publicly, or a custom place created by the authenticated user (which may be public or private).

I also need to show the client how to create its own places on the map. Custom places are created as subordinate resources of existing places. The logical thing to do is to put that information in the representations of those places: places with URIs like /Earth/Mount%20Rushmore and /Earth/42,-93.7.

Example 6-8 is a possible representation of /Earth/43.9,-103.46 that brings together most of what I’ve covered in the past two chapters. This representation abounds in hypermedia. It links to a certain point on several different maps, a place from the built-in database, custom places from other users, and a custom place created by the authenticated user. It also has a hypermedia form that will let the authenticated user create a new custom place at these coordinates. Compare this representation to the smaller representation of /Earth/43.9,-103.46 back in Example 5-9.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
 <title>43.9°N 103.46°W on Earth</title>
</head>
<body>

<p class="authenticated">
 You are currently logged in as 
 <a class="user" href="/user/leonardr">leonardr</a>.
</p>

<p> 
 Welcome to 
 <a class="coordinates" href="/Earth/43.9,-103.46">43.9°N 
  103.46°W</a> 
 on scenic <a class="place" href="/Earth">Earth</a>.
</p>

<p>See this location on a map:</p>

<ul class="maps">
 <li><a class="map" href="/road/Earth/43.9,-103.46">Road</a></li>
 <li><a class="map" href="/satellite/Earth/43.9,-103.46">Satellite</a></li>
  ...
</ul>

<p>Places at this location:</p>

<ul class="places">
 <li>
  <a class="builtin" href="Mount%20Rushmore">Mount Rushmore</a>
  System data says: 
  <span class="description">The world's largest sculpture</span>
 </li>

 <li>   
  <a class="custom" href="Mt.%20Rushmore/user1">Mt. Rushmore</a>
  <a class="user" href="/users/user1">user1</a> says: 
  <span class="description">Built on land stolen from the Lakota tribe</span>
 </li>

 <li>  
  <a class="custom" href="Mount%20Rushmore%20Gift%20Shop/user2">
   Mount Rushmore Gift Shop
  </a>
  <a class="user" href="/users/user1">user1</a> says: 
  <span class="description">Best fudge I've ever had</span>
 </li>

 <li> 
  <a class="custom-private" href="Mount%20Rushmore/leonardr">Mt. Rushmore</a>
  You said: <span class="description">We visited on 3/5/2005</span>
 </li>
</ul>

<form id="searchPlace" method="get" action="">
 <p>
  Show nearby places, features, or businesses: 
  <input name="show" repeat="template" /> <input class="submit" />
 </p>
</form>

<form id="createPlace" method="post" action="">
 <p>Create a new place here:</p>

 <p>
  Name: <input name="name" value="" type="text" /><br />
  Type: <input name="type" value="" type="text" /><br />
  Description:
   <textarea name="description"></textarea><br />
  Public?
   <input name="public" type="checkbox" value="on"/>
  <input type="submit" />
 </p>
</form>

</body>
</html>

This new resource, the custom place, mostly works like other resources I’ve already defined. A custom place responds to GET just like a built-in place. It responds to PUT (with a representation consisting of key-value pairs) and DELETE (with no representation) just like “user account” resources do. I only have a couple new edge cases to consider here.

When the client creates a custom place, the response code is 201 (“Created”). This works the same way as users. But it was never possible to cause a user’s URI to change, because I prohibited users from changing their usernames. It’s possible to change the name of a place, or to move one (say, a ship) from one point on the map to another. Either of these actions will change the URI.

When the client modifies a custom place without changing its location, the response code will be 200 (“OK”). If the location changes, the response code will be 301 (“Moved Permanently”) and the Location header will contain the place’s new URI. The client is responsible for updating its data structures to keep track of the new URI. This ties into a debate I’ll revisit in Chapter 8, about whether it’s more important to have URIs that contain useful information, or URIs that never change. My URIs describe a custom place using two pieces of resource state: coordinates and name (/user/leonardr/Earth/43.9,-103.46/Mt.%20Rushmore). If either of those changes, the old URI breaks.

Broken URIs are no fun on the human web, and they’re even less fun on the programmable web. If my custom “place” is a ship or something else that’s constantly moving, it effectively has no permanent URI. This is the single biggest design flaw in my system. If I were exposing this as a real web service, I’d probably give a “permalink” to every place: an alternate URI that doesn’t incorporate any changeable resource state. Since everything about a place can change except the planet it’s on and the person who owns it, these URIs will not look very friendly: my annotation of Mount Rushmore might be accessible from /user/leonardr/Earth/36028efa8. But at least they’ll always refer to the same place.

This new kind of resource introduces new error conditions, but most of them are variations of ones I’ve already covered, so I’ll pass over them quickly. The client might try to move an existing place off of the map by providing an invalid latitude or longitude: the response code is 400 (“Bad Request”), just as it was in a similar case in Chapter 5. The 400 response code is also appropriate when a client tries to create a place without providing all the information the server needs. This is similar to the 400 response code the server sends if the client tells the server to change a user’s password, but doesn’t actually provide the new password.

My service doesn’t allow a single user to define more than one place with the same name at the same coordinates. /user/leonardr/Earth/43.9,-103.46/Mt.%20Rushmore can only identify one place at a time. Suppose a client has two places called “My car,” and makes a PUT request that would move one to the location of the other. My service rejects this request with a response code of 409 (“Conflict”). There’s nothing wrong with moving a place to a certain set of coordinates; it’s just that right now there happens to be another place with that name there. The same 409 response code would happen if the client had two custom places at the same coordinates, and tried to rename one to match the name of the other. In either case, the client is making a syntactically valid request that would put the system’s resources into an inconsistent state. It’s the same as trying to create a user that already exists.

There’s one totally new error condition worthy of attention: the client may try to access a private place created by someone else. There are two possibilities. The first is to deny access with response code 403 (“Forbidden”). The 403 response code is used when the client provides no authentication, or insufficient authentication; the latter certainly applies in this case.

But a response code of 403 is a tacit admission that the resource exists. The server should not be giving out this information. If client A creates a custom place and marks it private, client B should not be able to figure out anything about it, even its name, even by guessing. When revealing the existence of a resource would compromise security, the HTTP standard allows the server to lie, and send a response code of 404 (“Not Found”).

• posts/get: Search your posts by tag or date, or search for a specific bookmarked URI.

• posts/recent: Fetch the n most recent posts by the authenticated user. The client may apply a tag filter: “fetch the n most recent posts that the authenticated user tagged with tag t”.

• posts/dates: Fetch the number of posts by the authenticated user for each day: perhaps five posts on the 12th, two on the 15th, and so on. The client may apply a tag filter here, too.

• posts/all: Fetch all posts for the authenticated user, ever. The client may apply a tag filter.

• posts/update: Check when the authenticated user last posted a bookmark. Clients are supposed to check this before deciding to call the expensive posts/all.

• posts/add: Create a bookmark for a URI. The client must specify a short description. It may choose to specify a long description, a set of tags, and a timestamp. A bookmark may be public or private (the default is public). A client may not bookmark the same URI more than once: calling posts/add again overwrites the old post with new information.

• posts/delete: Deletes a user’s post for a particular URI.

• tags/get: Fetch a list of tags used by the authenticated user.

• tags/rename: Rename one of the authenticated user’s tags. All posts tagged with the old name will now be tagged with the new name instead.

• tags/bundles/all: Fetch the user’s bundles. The resulting document lists the bundles, and each bundle lists the tags it contains.

• tags/bundles/set: Group several tags together into a (possibly new) bundle.

• tags/bundles/delete: Delete a bundle.

• /{username}: Fetch any user’s bookmarks.

• /{username}/{tag}: Fetch any user’s bookmarks, applying a tag filter.

• /tag/{tag-name}: Fetch bookmarks tagged with a particular tag, from all users.

• /url/{URI-MD5}: Fetch the list of users who have bookmarked a particular URI. The {URI-MD5} happens to be the MD5 hash of the URI, but from the average client’s point of view that’s not important: it’s an opaque string of bytes that somehow identifies a URI within the del.icio.us system.

• /recent: Fetch the most recently posted bookmarks, from all users. The del.icio.us home page also shows this information.

I’m not designing these resources in a vacuum: I’m going to implement them in a Rails application. It’s worth taking a brief look at how RESTful applications work in Rails. Unlike some other frameworks, Rails doesn’t let you define your resources directly. Instead, it divides up an application’s functionality into controllers: it’s the controllers that expose the resources. The first path variable in a request URI is used to route Rails to the appropriate controller class. For instance, in the URI /weblogs/4 the “weblogs” designates the controller: probably a class called WeblogController. The “4” designates the database ID of a particular weblog.

In previous versions of Rails, programmers defined RPC-style methods on controllers: methods like rename and delete. To rename a weblog you’d send a GET or an overloaded POST request to /weblogs/4/rename. Rails applications, like most web applications, were REST-RPC hybrids.

In Rails 1.2, programmers define special controller methods that correspond to the methods of HTTP’s uniform interface. For instance, sending a GET to /weblogs triggers the WeblogController’s index method, which is supposed to retrieve a list of the weblogs. Sending a POST to the same URI triggers the WeblogController#create method, which creates a subordinate resource beneath /weblogs: say, a weblog with a URI of /weblogs/4. The Rails controller exposes a resource—“the list of weblogs”—that responds to GET and POST. As you’d expect, when you POST to the “list” resource you get a subordinate resource: a new weblog.

The subordinate resource also supports the uniform interface. If you wanted to rename a weblog in an RPC-style service, you might POST a new name to /weblogs/4/rename. Under a RESTful regime, you PUT a new name to /weblogs/4, triggering the WeblogController#update method. To delete a weblog, you send a DELETE request to its URI, triggering the controller’s WeblogController#destroy method. There’s no need to expose an RPC-style URI /weblogs/4/delete, because HTTP’s uniform interface already knows about deleting.

These two resources, a list and an item in the list, show up all the time. Every database table is a list that contains items. Anything that can be represented as an RSS or Atom feed is a list that contains items. Rails defines a RESTful architecture that makes a simplifying assumption: every resource you expose can be made to fit one of these two patterns. This makes things easy most of the time, but the cost is aggravation when you try to use Rails controllers to expose resources that don’t fit this simple model.

I’m going to define my resources in terms of Rails controllers. These controllers impose constraints on my URI structure and my use of the uniform interface, and I need to design with those constraints in mind. By the time I’m done designing the controllers, I’ll know which resources the controllers expose, which URIs they answer to, and which methods of the uniform interface correspond to which RPC functions from the del.icio.us service. Basically, I’ll have completed steps 2 through 4 of the 9-step procedure from the Turning Requirements into Read/Write Resources” section in Chapter 6: “Split the data set into resources,” “Name the resources with URIs,” and “Expose a subset of the uniform interface.” In Chapter 12 I give a variant of the service design procedure specifically for Rails services.

I’ll only be accessing my Rails application from my local machine. The root URI will be http://localhost:3000/v1. When I give a relative URI below, like /users, understand that I’m talking about http://localhost:3000/v1/users. I only ever plan to write one version of this service, but I’m versioning the URIs, just in case. (When and how to version is discussed in Chapter 8).

Now I’m going to go back to that big list of RPC functions I found in the del.icio.us API and web site, and try to tease some Rails controllers out of it. One obvious controller is one that exposes information about user accounts. In Rails, this would be a class called UsersController. As soon as I say that, a lot of decisions are made for me. Rails sets up a path of least resistance that looks like this:

The user controller exposes a one-off “user list” resource, at the URI /users. It also exposes a resource for every user on the system, at a URI that incorporates the user’s database ID: /users/52 and the like. These resources expose some subset of HTTP’s uniform interface. Which subset? Rails defines this with a programming-language interface in the superclass of all controller classes: ActionController::Base. Table 7-1 shows how the two interfaces line up.

So if I want to let clients create new user accounts, I implement UsersController#create, and my “user list” resource starts calling that method in response to POST requests.

The path of least resistance is pretty good but I have a couple problems with it. First, I don’t want to let clients fetch the list of users, because del.icio.us doesn’t have that feature. (Presumably the del.icio.us administrative interface does have a feature like this.) That’s fine: I don’t have to expose GET on every resource, and I don’t have to define index in every controller. My user list resource, at the URI /users, will only expose the POST method, for creating new users. My user list is a featureless container for user account resources, and the only thing a client can do with it is create a new account. This incorporates functionality like that at https://secure.del.icio.us/register, where you can use your web browser to sign up for a del.icio.us account.

Another problem is that URIs like /users/52 look ugly. They certainly don’t look like http://del.icio.us/leonardr, the URI to my corresponding page on del.icio.us. This URI format is the Rails default because every object in a Rails application’s database can be uniquely identified by its table (“users”) and its ID (“52”). This URI might go away (if user 52 DELETEs her account), but it will never change, because database unique IDs don’t change.

I’d rather expose readable URIs that might change occasionally than permanent URIs that don’t say anything, so I’m going to identify a user using elements of its resource state. I happen to know that users have unique names, so I’m going to expose my “user” resources at URIs like /users/leonardr. Each resource of this type will expose the methods GET, PUT, and DELETE. This incorporates the functionality of the del.icio.us web site’s /{username} “function.” It also incorporates the pages on the web site (I didn’t mention these earlier) that let you edit and delete your own del.icio.us account.

To expose this RESTful interface, I just need to implement four special methods on UsersController. The create method implements POST on the “user list” resource at /users. The other three methods implement HTTP methods on the “user” resources at /users/{username}: show implements GET, update implements PUT, and destroy implements DELETE.

Each user account has a number of subordinate resources associated with it: the user’s bookmarks. I’m going to expose these resources through a second controller class, rooted beneath the “user account” resource.

The base URI of this controller will be /users/{username}/bookmarks. Like the users controller, the bookmarks controller exposes two types of resource: a one-off resource for the list of a user’s bookmarks, and one resource for each individual bookmark.

Rails wants to expose an individual bookmark under the URI /users/{username}/bookmarks/{database-id}. I don’t like this any more than I like /users/{database-id}. I’d like the URI to a bookmark to have some visible relationship to the URI that got bookmarked.

My original plan was to incorporate the target URI in the URI to the bookmark. That way if I bookmarked http://www.oreilly.com/, the bookmark resource would be available at /v1/users/leonardr/bookmarks/http://www.oreilly.com/. Lots of services work this way, including the W3C’s HTML validator. Looking at one of these URIs you can easily tell who bookmarked what. Rails didn’t like this URI format, though, and after trying some hacks I decided to get back on Rails’s path of least resistance. Instead of embedding external URIs in my resource URIs, I’m going to put the URI through a one-way hash function and embed the hashed string instead.

If you go to http://del.icio.us/url/55020a5384313579a5f11e75c1818b89 in your web browser, you’ll see the list of people who’ve bookmarked http://www.oreilly.com/. There’s no obvious connection between the URI and its MD5 hash, but if you know the former you can calculate the latter. It’s certainly better than a totally opaque database ID. And since it’s a single alphanumeric string, Rails handles it with ease. My bookmark resources will have URIs like /v1/users/leonardr/bookmarks/55020a5384313579a5f11e75c1818b89. That URI identifies the time I bookmarked http://www.oreilly.com/ (see Example 7-2).

require 'digest/md5'
Digest::MD5.new("http://www.oreilly.com/").to_s
# => "55020a5384313579a5f11e75c1818b89"

When a user is first created it has no bookmarks. A client creates bookmarks by sending a POST request to its own “bookmark list” resource, just as it might create a user account by sending a POST to the “user list” resource. This takes care of the posts/add and posts/delete functions from the del.icio.us API.

Unlike with the list of users, I do want to let clients fetch the list of a user’s bookmarks. This means /users/{username}/bookmarks will respond to GET. The individual bookmarks will respond to GET, PUT, and DELETE. This means the BookmarksController: index, create, show, update, and delete.

The “bookmark list” resource incorporates some of the functionality from the del.icio.us API functions posts/get, posts/recent, and posts/all.

Bookmarks aren’t the only type of resource that conceptually fits “beneath” a user account. There’s also the user’s tag vocabulary. I’m not talking about tags in general here: I’m asking questions about which tags a particular user likes to use. These questions are handled by the user tags controller.

This controller is rooted at /users/{username}/tags. That’s the “user tag list” resource. It’s an algorithmic resource, generated from the tags a user uses to talk about her bookmarks. This resource corresponds roughly to the del.icio.us tags/get function. It’s a read-only resource: a user can’t modify her vocabulary directly, only by changing the way she uses tags in bookmarks.

The resources at /users/{username}/tags/{tag} talk about the user’s use of a specific tag. My representation will show which bookmarks a user has filed under a particular tag. This class of resource corresponds to the /{username}/{tag} “function” from the web site. It also incorporates some stuff of the del.icio.us API functions posts/get, posts/recent, and posts/all.

The “tag” resources are also algorithmic, but they’re not strictly read-only. A user can’t delete a tag except by removing it from all of her bookmarks, but I do want to let users rename tags. (Tag deletion is a plausible feature, but I’m not implementing it because, again, del.icio.us doesn’t have it.) So each user-tag resource will expose PUT for clients who want to rename that tag.

Instead of PUT, I could have used overloaded POST to define a one-off “rename” method like the del.icio.us API’s tag/rename. I didn’t, because that’s RPC-style thinking. The PUT method suffices to convey any state change, whether it’s a rename or something else. There’s a subtle difference between renaming the tag and changing its state so the name is different, but it’s the difference between an RPC-style interface and a uniform, RESTful one. It’s less work to program a computer to understand a generic “change the state” than to program it to understand “rename a tag.”

A user’s posting history—her calendar— is handled by one more controller that lives “underneath” a user account resource. The posting history is another algorithmically generated, read-only resource: you can’t change your posting history except by posting. The controller’s root URI is /users/{username}/calendar, and it corresponds to the del.icio.us API’s posts/dates function.

I’ll also expose a variety of subresources, one for each tag in a user’s vocabulary. These resources will give a user’s posting history when only one tag is considered. These resources correspond to the del.icio.us API’s posts/dates function with a tag filter applied. Both kinds of resource, posting history and filtered posting history, will expose only GET.

I mentioned earlier that URIs in a social bookmarking system have emergent properties. The URI controller gives access to some of those properties. It’s rooted at /uris/, and it exposes URIs as resources independent from the users who bookmark them.

I’m not exposing this controller’s root URI as a resource, though I could. The logical thing to put there would be a huge list of all URIs known to the application. But again, the site I’m taking for my model doesn’t have any feature like that. Instead, I’m exposing a series of resources at /uris/{URI-MD5}: one resource for each URI known to the application. The URI format is the same as /users/{username}/bookmarks/{URI-MD5} in the user bookmark controller: calculate the MD5 hash of the target URI and stick it onto the end of the controller’s base URI.

These resources expose the application’s knowledge about a specific URI, such as which users have bookmarked it. This corresponds to the /url/{URI-MD5} “function” on the del.icio.us web site.

My last implemented controller reveals another emergent property of the URIs. In this case the property is newness: which URIs were most recently posted.

This controller is rooted at /recent. The top-level “list” resource lists all the recently posted bookmarks. This corresponds to the /recent “function” on the del.icio.us web site.

The sub-resources at /recent/{tag} expose the list of recently posted bookmarks that were tagged with a particular tag. For instance, a client can GET /recent/recipes to find recently posted URIs that were tagged with “recipes”. This corresponds to the /tag/{tag-name} function on the del.icio.us web site.

Again, I’m not going to implement this controller, but I want to design it so you can see I’m not cheating. This controller is rooted at /user/{username}/bundles/. An alternative is /user/{username}/tags/bundles/, but that would prevent any user from having a tag named “bundles”. A client can send a GET request to the appropriate URI to get any user’s “bundle list”. A client can POST to its own bundle list to create a new bundle. This takes care of tags/bundles/all and part of tags/bundles/set.

The sub-resources at /user/{username}/bundles/{bundle} expose the individual bundles by name. These respond to GET (to see which tags are in a particular bundle), PUT (to modify the tags associated with a bundle), and DELETE (to delete a bundle). This takes care of tags/bundles/delete and the rest of tags/bundles/set.

What’s left? I’ve covered almost all the functionality of the original del.icio.us API, but I haven’t placed the posts/update function. This function is designed to let a client avoid calling posts/all when there’s no new data there. Why bother? Because the posts/all function is extremely expensive on the server side. A del.icio.us client is supposed to keep track of the last time it called posts/all, and check that time against the “return value” of posts/update before calling the expensive function again.

There’s already a solution for this built into HTTP: conditional GET. I cover it briefly in Conditional GET later in this chapter and I’ll cover it in more detail in Chapter 8, but in this chapter you’ll see it implemented. By implementing conditional GET, I can give the time- and bandwidth-saving benefits of posts/update to most of the resources I’m exposing, not just the single most expensive one.

I’ve taken an RPC-style web service that was only RESTful in certain places and by accident, and turned it into a set of fully RESTful resources. I’d like to take a break now and illustrate how the two services line up with each other. Tables 7-2 through 7-6 show every social bookmarking operation I implemented, the HTTP request you’d send to invoke that operation on my RESTful web service, and how you’d invoke the corresponding operation on del.icio.us itself.

I think you’ll agree that the RESTful service is more self-consistent, even accounting for the fact that some of the del.icio.us features come from the web service and some from the web site. Table 7-6 is probably the best for a straight-up comparison. There you can distinctly see the main advantage of my RESTful service: its use of the HTTP method to remove the operation name from the URI. This lets the URI identify an object in the object-oriented sense. By varying the HTTP method you can perform different operations on the object. Instead of having to understand some number of arbitrarily-named functions, you can understand a single class (in the object-oriented sense) whose instances expose a standardized interface.

My service also lifts various restrictions found in the del.icio.us web service. Most notably, you can see other peoples’ public bookmarks. Now, sometimes restrictions are the accidental consequences of bad design, but sometimes they exist for a reason. If I were deploying this service commercially it might turn out that I want to add those limits back in. I might not want user A to have unlimited access to user B’s bookmark list. I don’t have to change my design to add these limits. I just have to change the authorization component of my service. I make it so that authenticating as userA doesn’t authorize you to fetch userB’s public bookmarks, any more than it authorizes you to delete userB’s account. Or if bandwidth is the problem, I might limit how often any user can perform certain operations. I haven’t changed my resources at all: I’ve just added additional rules about when operations on those resources will succeed.

Ready for some more code? I’ve split my data set into Rails controllers, and each Rails controller has divided its data set further into one or two kinds of resources. Rails has also made decisions about what my URIs will look like. I vetoed some of these decisions (like /users/52, which I changed to /users/leonardr), but most of them I’m going to let stand.

I’ll implement the controllers as Ruby classes, but what about the URIs? I need some way of mapping path fragments like bookmarks/ to controller classes like BookmarksController. In a Rails application, this is the job of the routes.rb file. Example 7-3 is a routes.rb that sets up URIs for the six controllers I’ll implement later in the chapter.

# service/config/routes.rb
ActionController::Routing::Routes.draw do |map|
  base = '/v1'

  ## The first controller I define is the UsersController. The call to
  ## map.resources sets it up so that all HTTP requests to /v1/users
  ## or /v1/users/{username} are routed to the UsersController class.

  # /v1/users => UsersController
  map.resources :users, :path_prefix => base

  ## Now I'm going to define a number of controllers beneath the
  ## UsersController. They will respond to requests for URIs that start out
  ## with /v1/users/{username}, and then have some extra stuff.
  user_base = base + '/users/:username'

  # /v1/users/{username}/bookmarks => BookmarksController
  map.resources :bookmarks, :path_prefix => user_base

  # /v1/users/{username}/tags => TagsController
  map.resources :tags, :path_prefix => user_base

  # /v1/users/{username}/calendar => CalendarController
  map.resources :calendar, :path_prefix => user_base

  ## Finally, two more controllers that are rooted beneath /v1.

  # /v1/recent => RecentController
  map.resources :recent, :path_prefix => base

  # /v1/uris => UrisController
  map.resources :uris, :path_prefix => base
end

Now I’m committed to defining six controller classes. The code in Example 7-3 determines the class names by tying into Rails’ naming conventions. My six classes are called UsersController, BookmarksController, TagsController, CalendarController, RecentController, and UrisController. Each class controls one or two kinds of resources. Each controller implements a specially-named Ruby method for each HTTP method the resources expose.

There’s one feature I want for my service that isn’t built into Rails or plugins, and there’s another that goes against Rails’s path of least resistance. I’m going to be implementing these features myself. These two items account for much of the tricky code in the service.

This class is the abstract superclass of my six controllers, and it contains most of the common functionality (the rest will go into the ActiveRecord model classes). Example 7-7 starts by defining an action for the single most common operation in this service: fetching a list of bookmarks that meet some criteria.

# app/controllers/application.rb
require 'digest/sha1'
require 'digest/md5'
require 'rubygems'
require 'atom/feed'

class ApplicationController < ActionController::Base

  # By default, show 50 bookmarks at a time.
  @@default_limit = 50

  ## Common actions

  # This action takes a list of SQL conditions, adds some additional
  # conditions like a date filter, and renders an appropriate list of
  # bookmarks. It's used by BookmarksController, RecentController,
  # and TagsController.
  def show_bookmarks(conditions, title, feed_uri, user=nil, tag=nil)
    errors = []

    # Make sure the specified limit is valid. If no limit is specified,
    # use the default.
    if params[:limit] && params[:limit].to_i < 0
      errors << "limit must be >=0"
    end
    params[:limit] ||= @@default_limit
    params.delete(:limit) if params[:limit] == 0  # 0 means "no limit"

    # If a date filter was specified, make sure it's a valid date.
    if params[:date]
      begin        
        params[:date] = Date.parse(params[:date])
      rescue ArgumentError
        errors << "incorrect date format"
      end
    end

    if errors.empty?
      conditions ||= [""]
      
      # Add a restriction by date if necessary.
      if params[:date]
        conditions[0] << " AND " unless conditions[0].empty?
        conditions[0] << "timestamp >= ? AND timestamp < ?"
        conditions << params[:date]
        conditions << params[:date] + 1
      end

      # Restrict the list to bookmarks visible to the authenticated user.
      Bookmark.only_visible_to!(conditions, @authenticated_user)

      # Find a set of bookmarks that matches the given conditions.
      bookmarks = Bookmark.custom_find(conditions, tag, params[:limit])
      
      # Render the bookmarks however the client requested.
      render_bookmarks(bookmarks, title, feed_uri, user)
    else
      render :text => errors.join("\n"), :status => "400 Bad Request"
    end
  end

The show_bookmarks method works like any Rails action: it gets query parameters like limit from params, and verifies them. Then it fetches some data from the database and renders it with a view. A lot of my RESTful action methods will delegate to this method. If the RESTful action specifies no conditions, show_bookmarks will fetch all the bookmarks that match the date and tag filters, up to the limit. Most of my actions will impose additional conditions, like only fetching bookmarks posted by a certain user.

The main difference between show_bookmarks and a traditional Rails action is in the view. Most Rails actions define the view with an ERb template like show.rhtml: a combination of HTML and Ruby code that works like JSP templates or PHP code. Instead, I’m passing the job off to the render_bookmarks function (see Example 7-8). This function uses code-based generators to build the XML and Atom documents that serve as representations for most of my application’s resources.

  # This method renders a list of bookmarks as a view in RSS, Atom, or
  # ActiveRecord XML format. It's called by show_bookmarks
  # above, which is used by three controllers. It's also used
  # separately by UriController and BookmarksController.
  # 
  # This view method supports conditional HTTP GET.
  def render_bookmarks(bookmarks, title, feed_uri, user, except=[])
    # Figure out a current value for the Last-Modified header.
    if bookmarks.empty?
      last_modified = nil
    else
      # Last-Modified is the most recent timestamp in the bookmark list.
      most_recent_bookmark = bookmarks.max do |b1,b2|
        b1.timestamp <=> b2.timestamp
      end
      last_modified = most_recent_bookmark.timestamp
    end
    
    # If the bookmark list has been modified since it was last requested...
    render_not_modified_or(last_modified) do
      respond_to do |format|
        # If the client requested XML, serialize the ActiveRecord
        # objects to XML. Include references to the tags in the
        # serialization.
        format.xml  { render :xml => 
          bookmarks.to_xml(:except => except + [:id, :user_id],
                           :include => [:tags]) }
        # If the client requested Atom, turn the ActiveRecord objects
        # into an Atom feed.
        format.atom { render :xml => atom_feed_for(bookmarks, title, 
                                                   feed_uri, user) }
      end
    end
  end

That method is also where I start handling conditional HTTP requests. I’ve chosen to use the timestamp of the most recent bookmark as the value of the HTTP header Last-Modified.

The rest of the conditional request handling is in the render_not_modified_or function (see Example 7-9). It’s called just before render_bookmarks is about to write the list of bookmarks, and it applies the rules of conditional HTTP GET. If the list of bookmarks has changed since this client last requested it, this function calls the Ruby keyword yield and the rest of the code in render_bookmarks runs normally. If the list of bookmarks hasn’t changed, this function short-circuits the Rails action, sending a response code of 304 (“Not Modified”) instead of serving the representation.

  ## Helper methods

  # A wrapper for actions whose views support conditional HTTP GET.
  # If the given value for Last-Modified is after the incoming value
  # of If-Modified-Since, does nothing. If Last-Modified is before
  # If-Modified-Since, this method takes over the request and renders
  # a response code of 304 ("Not Modified").
  def render_not_modified_or(last_modified)
    response.headers['Last-Modified'] = last_modified.httpdate if last_modified

    if_modified_since = request.env['HTTP_IF_MODIFIED_SINCE']
    if if_modified_since && last_modified &&
        last_modified <= Time.httpdate(if_modified_since)
      # The representation has not changed since it was last requested.
      # Instead of processing the request normally, send a response
      # code of 304 ("Not Modified").
      render :nothing => true, :status => "304 Not Modified"
    else
      # The representation has changed since it was last requested.
      # Proceed with normal request processing.
      yield
    end
  end

Example 7-10 shows one more helper function used in multiple actions. The if_found method makes sure the client specified a URI that corresponds to an object in the database. If given a non-null object, nothing happens: if_found uses yield to return control to the action that called it. If given a null object, the function short-circuits the request with a response code of 404 (“Not Found”), and the action never gets a chance to run.

  # A wrapper for actions which require the client to have named a
  # valid object. Sends a 404 response code if the client named a
  # nonexistent object. See the user_id_from_username filter for an
  # example.
  def if_found(obj)
    if obj
      yield     else
      render :text => "Not found.", :status => "404 Not Found"
      false
    end    
  end

I’ve also implemented a number of filters: pieces of code that run before the Rails actions do. Some Rails filters perform common setup tasks (see Example 7-11). This is the job of authenticate, which checks the client’s credentials. Filters may also check for a problem and short-circuit the request if they find one. This is the job of must_authenticate, and also must_specify_user, which depends on the if_found method defined above. Filters let me keep common code out of the individual actions.

  ## Filters

  # All actions should try to authenticate a user, even those actions
  # that don't require authorization. This is so we can show an
  # authenticated user their own private bookmarks.
  before_filter :authenticate

  # Sets @authenticated_user if the user provides valid
  # credentials. This may be used to deny access or to customize the
  # view.
  def authenticate    
    @authenticated_user = nil
    authenticate_with_http_basic do |user, pass|      
      @authenticated_user = User.authenticated_user(user, pass)
    end
    return true
  end

  # A filter for actions that _require_ authentication. Unless the
  # client has authenticated as some user, takes over the request and
  # sends a response code of 401 ("Unauthorized").  Also responds with
  # a 401 if the user is trying to operate on some user other than
  # themselves. This prevents users from doing things like deleting
  # each others' accounts.
  def must_authenticate    
    if @authenticated_user && (@user_is_viewing_themselves != false)
      return true
    else
      request_http_basic_authentication("Social bookmarking service")      
      return false
    end    
  end

  # A filter for controllers beneath /users/{username}. Transforms 
  # {username} into a user ID. Sends a 404 response code if the user
  # doesn't exist.
  def must_specify_user
    if params[:username]
      @user = User.find_by_name(params[:username])      
      if_found(@user) { params[:user_id] = @user.id }
      return false unless @user
    end
    @user_is_viewing_themselves = (@authenticated_user == @user)
    return true
  end

Finally, the application controller is where I’ll implement my primary view method: atom_feed_for (see Example 7-12). This method turns a list of ActiveRecord Bookmark objects into an Atom document. The controller that wants to serve a list of bookmarks needs to provide a title for the feed (such as “Bookmarks for leonardr”) and a URI to the resource being represented. The resulting document is rich in links. Every bookmark links to the external URI, to other people who bookmarked that URI, and to bookmarks that share tags with this one.

  ## Methods for generating a representation

  # This method converts an array of ActiveRecord's Bookmark objects
  # into an Atom feed.
  def atom_feed_for(bookmarks, title, feed_uri, user=nil)
    feed = Atom::Feed.new
    feed.title = title
    most_recent_bookmark = bookmarks.max do |b1,b2|
      b1.timestamp <=> b2.timestamp
    end
    feed.updated = most_recent_bookmark.timestamp

    # Link this feed to itself
    self_link = feed.links.new
    self_link['rel'] = 'self'
    self_link['href'] = feed_uri + ".atom"

    # If this list is a list of bookmarks from a single user, that user is
    # the author of the feed.
    if user
      user_to_atom_author(user, feed)
    end

    # Turn each bookmark in the list into an entry in the feed.
    bookmarks.each do |bookmark|
      entry = feed.entries.new
      entry.title = bookmark.short_description
      entry.content = bookmark.long_description

      # In a real application, a bookmark would have a separate
      # "modification date" field which was not under the control of
      # the user. This would also make the Last-Modified calculations
      # more accurate.
      entry.updated = bookmark.timestamp      

      # First, link this Atom entry to the external URI that the
      # bookmark tracks.
      external_uri = entry.links.new
      external_uri['href'] = bookmark.uri

      # Now we give some connectedness to this service. Link this Atom
      # entry to this service's resource for this bookmark.
      bookmark_resource = entry.links.new
      bookmark_resource['rel'] = "self"
      bookmark_resource['href'] = bookmark_url(bookmark.user.name, 
                                               bookmark.uri_hash) + ".atom"
      bookmark_resource['type'] = "application/xml+atom"

      # Then link this entry to the list of users who've bookmarked
      # this URI.
      other_users = entry.links.new
      other_users['rel'] = "related"
      other_users['href'] = uri_url(bookmark.uri_hash) + ".atom"
      other_users['type'] = "application/xml+atom"

      # Turn this entry's user into the "author" of this entry, unless
      # we already specified a user as the "author" of the entire
      # feed.
      unless user
        user_to_atom_author(bookmark.user, entry) 
      end

      # For each of this bookmark's tags...
      bookmark.tags.each do |tag|
        # ...represent the tag as an Atom category.
        category = entry.categories.new
        category['term'] = tag
        category['scheme'] = user_url(bookmark.user.name) + "/tags"

        # Link to this user's other bookmarks tagged using this tag.
        tag_uri = entry.links.new
        tag_uri['href'] = tag_url(bookmark.user.name, tag.name) + ".atom"
        tag_uri['rel'] = 'related'
        tag_uri['type'] = "application/xml+atom"

        # Also link to all bookmarks tagged with this tag.
        recent_tag_uri = entry.links.new
        recent_tag_uri['href'] = recent_url(tag.name) + ".atom"
        recent_tag_uri['rel'] = 'related'
        recent_tag_uri['type'] = "application/xml+atom"
      end
    end 
    return feed.to_xml
  end

  # Appends a representation of the given user to an Atom feed or element
  def user_to_atom_author(user, atom)
    author = atom.authors.new
    author.name = user.full_name
    author.email = user.email
    author.uri = user_url(user.name)
  end
end

Example 7-13 shows what kind of Atom representation this method might serve.

<feed xmlns='http://www.w3.org/2005/Atom'>
 <title>Bookmarks for leonardr</title>
 <screen>
 <updated>2007-02-14T02:26:58-05:00</updated>
 <link href="http://localhost:3000/v1/users/leonardr/bookmarks.atom" rel="self"/>
 <author>
  <name>leonardr</name>
  <uri>http://localhost:3000/v1/users/leonardr</uri>
  <email>leonardr@example.com</email>
 </author>
 
 <entry>
  <title>REST and WS-*</title>
  <content>Joe Gregorio's lucid explanation of RESTful principles</content>
  <category term="rest" scheme="http://localhost:3000/v1/users/leonardr/rest"/>
  <link href="http://bitworking.org/news/125/REST-and-WS" rel="alternate"/>
  <link href="http://localhost:3000/v1/users/leonardr/bookmarks/68044f26e373de4a08ff343a7fa5f675.atom" 
   rel="self" type="application/xml+atom"/>
  ...
  <link href="http://localhost:3000/v1/recent/rest.atom"
   rel="related" type="application/xml+atom"/>
  <updated>2007-02-14T02:26:58-05:00</updated>
 </entry>
</feed>

Now I’m ready to show you some specific actions. I’ll start with the controller that makes user accounts possible. In the code in Example 7-14, note the call to before_filter that sets up the must_authenticate filter. You don’t need to authenticate to create (POST) a user account (as whom would you authenticate?), but you must authenticate to modify (PUT) or destroy (DELETE) an account.

class UsersController < ApplicationController  

  # A client must authenticate to modify or delete a user account.
  before_filter :must_authenticate, :only => [:modify, :destroy]

  # POST /users
  def create   
    user = User.find_by_name(params[:user][:name])
    if user
      # The client tried to create a user that already exists.
      headers['Location'] = user_url(user.name)
      render :nothing => true, :status => "409 Conflict"
    else   
      user = User.new(params[:user])
      if user.save
        headers['Location'] = user_path(user.name)
        render :nothing => true, :status => "201 Created"
      else
        # There was a problem saving the user to the database.
        # Send the validation error messages along with a response
        # code of 400.
        render :xml => user.errors.to_xml, :status => "400 Bad Request"
      end
    end
  end

The conventions of RESTful Rails impose a certain structure on UsersController (and, indeed, on the name of the class itself). This controller exposes a resource for the list of users, and one resource for each particular user. The create method corresponds to a POST to the user list. The show, update, and delete methods correspond to a GET, PUT, or DELETE request on a particular user.

The create method follows a pattern I’ll use for POST requests throughout this service. If the client tries to create a user that already exists, the response code is 409 (“Conflict”). If the client sends bad or incomplete data, the ActiveRecord validation rules (defined in the User) model) fail, and the call to User#save returns false. The response code then is 400 (“Bad Request”). If all goes well, the response code is 201 (“Created”) and the Location header contains the URI of the newly created user. All I’ve done in Example 7-15 is put into code the things I said in What’s Supposed to Happen?” and What Might Go Wrong?” earlier in this chapter. I’ll mention this generic control flow again in Chapter 8.

  # PUT /users/{username}
  def update
    old_name = params[:id]
    new_name = params[:user][:name]
    user = User.find_by_name(old_name)

    if_found user do
      if old_name != new_name && User.find_by_name(new_name)
        # The client tried to change this user's name to a name
        # that's already taken. Conflict!
        render :nothing => true, :status => "409 Conflict"      
      else        
        # Save the user to the database.
        user.update_attributes(params[:user])        
        if user.save
          # The user's name changed, which changed its URI.
          # Send the new URI.
          if user.name != old_name
            headers['Location'] = user_path(user.name)
            status = "301 Moved Permanently"
          else
            # The user resource stayed where it was.
            status = "200 OK"
          end
          render :nothing => true, :status => status
        else
          # There was a problem saving the user to the database.
          # Send the validation error messages along with a response
          # code of 400.
          render :xml => user.errors.to_xml, :status => "400 Bad Request"
        end
      end
    end
  end

The update method has a slightly different flow, and it’s a flow I’ll use for PUT requests throughout the service. The general outline is the same as for POST. The twist is that instead of trying to create a user (whose name might already be in use), the client can rename an existing user (and their new name might already be in use).

I send a 409 response code (“Conflict”) if the client proposes a new username that already exists, and a 400 response code (“Bad Request”) if the data validation fails. If the client successfully edits a user, I send not a 201 response code (“Created”) but a simple 200 (“OK”).

The exception is if the client successfully changes a user’s name. Now that resource is available under a different URI: say, /users/leonard instead of /users/leonardr. That means I need to send a response code of 301 (“Moved Permanently”) and put the user’s new URI in the Location header.

The GET and DELETE implementations are more straightforward, as shown in Example 7-16.

  # GET /users/{username}
  def show
    # Find the user in the database.
    user = User.find_by_name(params[:id])
    if_found(user) do
      # Serialize the User object to XML with ActiveRecord's to_xml.
      # Don't include the user's ID or password when building the XML
      # document.
      render :xml => user.to_xml(:except => [:id, :password])
    end
  end

  # DELETE /users/{username}
  def destroy
    user = User.find_by_name(params[:id])
    if_found user do
      # Remove the user from the database.
      user.destroy
      render :nothing => true, :status => "200 OK"
    end
  end
end

There is one hidden detail: the if_found method sends a response code of 404 (“Not Found”) if the user tries to GET or DELETE a nonexistent user. Otherwise, the response code is 200 (“OK”). I have not implemented conditional HTTP GET for user resources: I figured the possible bandwidth savings wasn’t big enough to justify the added complexity.

This is the other main controller in this application (see Example 7-17). It exposes a user’s list of bookmarks and each individual bookmark. The filters are interesting here. This BookmarksController is for displaying a particular user’s bookmarks, and any attempt to see a nonexistent user’s bookmarks should be rebuffed with a stern 404 (“Not Found”). That’s the job of the must_specify_user filter I defined earlier. The must_authenticate filter works like it did in UsersController: it prevents unauthenticated requests from getting through to Rails actions that require authentication. I’ve also got a one-off filter, fix_params, that enforces consistency in incoming representations of bookmarks.

class BookmarksController < ApplicationController
  before_filter :must_specify_user
  before_filter :fix_params
  before_filter :must_authenticate, :only => [:create, :update, :destroy]

  # This filter cleans up incoming representations.
  def fix_params    
    if params[:bookmark]
      params[:bookmark][:user_id] = @user.id if @user
    end 
  end

The rest of BookmarksController is just like UsersController: fairly involved create (POST) and update (PUT) methods, simple show (GET) and delete (DELETE) methods (see Example 7-18). The only difference is that this controller’s list resource responds to GET, so I start with a simple implementation of index. Like many of the Rails actions I’ll define, index and show simply delegate to the show_bookmarks action.

  # GET /users/{username}/bookmarks
  def index
    # Show this user's bookmarks by passing in an appropriate SQL
    # restriction to show_bookmarks.
    show_bookmarks(["user_id = ?", @user.id],
                   "Bookmarks for #{@user.name}", 
                   bookmark_url(@user.name), @user)
  end

  # POST /users/{username}/bookmarks
  def create
    bookmark = Bookmark.find_by_user_id_and_uri(params[:bookmark][:user_id], 
                                                params[:bookmark][:uri])
    if bookmark
      # This user has already bookmarked this URI. They should be
      # using PUT instead.
      headers['Location'] = bookmark_url(@user.name, bookmark.uri)
      render :nothing => true, :status => "409 Conflict"
    else
      # Enforce default values for 'timestamp' and 'public'
      params[:bookmark][:timestamp] ||= Time.now
      params[:bookmark][:public] ||= "1"

      # Create the bookmark in the database.
      bookmark = Bookmark.new(params[:bookmark])
      if bookmark.save
        # Add tags.
        bookmark.tag_with(params[:taglist]) if params[:taglist]

        # Send a 201 response code that points to the location of the
        # new bookmark.
        headers['Location'] = bookmark_url(@user.name, bookmark.uri)
        render :nothing => true, :status => "201 Created"
      else
        render :xml => bookmark.errors.to_xml, :status => "400 Bad Request"
      end
    end
  end

  # PUT /users/{username}/bookmarks/{URI-MD5}
  def update
    bookmark = Bookmark.find_by_user_id_and_uri_hash(@user.id, params[:id])
    if_found bookmark do
      old_uri = bookmark.uri
      if old_uri != params[:bookmark][:uri] && 
          Bookmark.find_by_user_id_and_uri(@user.id, params[:bookmark][:uri])
        # The user is trying to change the URI of this bookmark to a
        # URI that they've already bookmarked. Conflict!
        render :nothing => true, :status => "409 Conflict"
      else
        # Update the bookmark's row in the database.
        if bookmark.update_attributes(params[:bookmark])
          # Change the bookmark's tags.
          bookmark.tag_with(params[:taglist]) if params[:taglist]
          if bookmark.uri != old_uri
            # The bookmark changed URIs. Send the new URI.
            headers['Location'] = bookmark_url(@user.name, bookmark.uri)
            render :nothing => true, :status => "301 Moved Permanently"
          else
            # The bookmark stayed where it was.
            render :nothing => true, :status => "200 OK"
          end
        else
          render :xml => bookmark.errors.to_xml, :status => "400 Bad Request"
        end
      end
    end
  end

  # GET /users/{username}/bookmarks/{uri}
  def show
    # Look up the requested bookmark, and render it as a "list"
    # containing only one item.
    bookmark = Bookmark.find_by_user_id_and_uri_hash(@user.id, params[:id])
    if_found(bookmark) do
      render_bookmarks([bookmark], 
                       "#{@user.name} bookmarked #{bookmark.uri}",
                       bookmark_url(@user.name, bookmark.uri_hash),
                       @user)
    end
  end

  # DELETE /users/{username}/bookmarks/{uri}
  def destroy
    bookmark = Bookmark.find_by_user_id_and_uri_hash(@user.id, params[:id])
    if_found bookmark do
      bookmark.destroy
      render :nothing => true, :status => "200 OK"
    end
  end
end

This controller exposes a user’s tag vocabulary, and the list of bookmarks she’s filed under each tag (see Example 7-19). There are two twists here: the tag vocabulary and the “tag rename” operation.

The tag vocabulary is simply a list of a user’s tags, along with a count of how many times this user used the tag. I can get this data fairly easily with ActiveResource, and format it as a representation with to_xml, but what about security? If you tag two public and six private bookmarks with “ruby,” when I look at your tag vocabulary, I should only see “ruby” used twice. If you tag a bunch of private bookmarks with “possible-acquisition,” I shouldn’t see “possible-acquisition” in your vocabulary at all. On the other hand, when you’re viewing your own bookmarks, you should be able to see the complete totals. I use some custom SQL to count only the public tags when appropriate. Incidentally, this is another resource that doesn’t support conditional GET.

class TagsController < ApplicationController
  before_filter :must_specify_user
  before_filter :must_authenticate, :only => [:update]

  # GET /users/{username}/tags
  def index    
    # A user can see all of their own tags, but only tags used
    # in someone else's public bookmarks.
    if @user_is_viewing_themselves
      tag_restriction = ''
    else
      tag_restriction = " AND bookmarks.public='1'"
    end
    sql = ["SELECT tags.*, COUNT(bookmarks.id) as count" +
           " FROM tags, bookmarks, taggings" +
           " WHERE taggings.taggable_type = 'Bookmark'" +
           " AND tags.id = taggings.tag_id" +
           " AND taggings.taggable_id = bookmarks.id" + 
           " AND bookmarks.user_id = ?" + tag_restriction +
           " GROUP BY tags.name", @user.id]
    # Find a bunch of ActiveRecord Tag objects using custom SQL.
    tags = Tag.find_by_sql(sql)

    # Convert the Tag objects to an XML document.
    render :xml => tags.to_xml(:except => [:id])
  end

I said earlier I’d handle the “tag rename” operation with HTTP PUT. This makes sense since a rename is a change of state for an existing resource. The difference here is that this resource doesn’t correspond to a specific ActiveRecord object. There’s an ActiveRecord Tag object for every tag, but that object represents everyone’s use of a tag. This controller doesn’t expose tags, per se: it exposes a particular user’s tag vocabulary.

Renaming a Tag object would rename it for everybody on the site. But if you rename “good” to “bad,” then that should only affect your bookmarks. Any bookmarks I’ve tagged as “good” should stay “good.” The client is not changing the tag, just one user’s use of the tag.

From a RESTful perspective none of this matters. A resource’s state is changed with PUT, and that’s that. But the implementation is a bit tricky. What I need to do is find all the client’s bookmarks tagged with the given tag, strip off the old tag, and stick the new tag on. Unlike with users or bookmarks, I won’t be sending a 409 (“Conflict”) response code if the user renames an old tag to a tag that already exists. I’ll just merge the old tag into the new one (see Example 7-20).

  # PUT /users/{username}/tags/{tag} 
  # This PUT handler is a little trickier than others, because we
  # can't just rename a tag site-wide. Other users might be using the
  # same tag.  We need to find every bookmark where this user uses the
  # tag, strip the "old" name, and add the "new" name on.
  def update
    old_name = params[:id]
    new_name = params[:tag][:name] if params[:tag]
    if new_name
      # Find all this user's bookmarks tagged with the old name
      to_change = Bookmark.find(["bookmarks.user_id = ?", @user.id], 
                                old_name)
      # For each such bookmark...
      to_change.each do |bookmark|
        # Find its tags.
        tags = bookmark.tags.collect { |tag| tag.name }
        # Remove the old name.
        tags.delete(old_name)
        # Add the new name.
        tags << new_name
        # Assign the new set of tags to the bookmark.
        bookmark.tag_with tags.uniq
      end
      headers['Location'] = tag_url(@user.name, new_name)
      status = "301 Moved Permanently"
    end
    render :nothing => true, :status => status || "200 OK"
  end

  # GET /users/{username}/tags/{tag}
  def show
    # Show bookmarks that belong to this user and are tagged
    # with the given tag.
    tag = params[:id]
    show_bookmarks(["bookmarks.user_id = ?", @user.id], 
                   "#{@user.name}'s bookmarks tagged with '#{tag}'",
                   tag_url(@user.name, tag), @user, tag)
  end
end

Every other controller in my application is read-only. This means it implements at most index and show. Hopefully by now you get the idea behind the controllers and their action methods, so I’ll cover the rest of the controllers briefly.

class CalendarController < ApplicationController
  before_filter :must_specify_user

  # GET /users/{username}/calendar
  def index
    calendar = Bookmark.calendar(@user.id, @user_is_viewing_themselves)
    render :xml => calendar_to_xml(calendar)
  end

  # GET /users/{username}/calendar/{tag}
  def show    
    tag = params[:id]
    calendar = Bookmark.calendar(@user.id, @user_is_viewing_themselves,
                                 tag)
    render :xml => calendar_to_xml(calendar, tag)
  end

  private

  # Build an XML document out of the data structure returned by the
  # Bookmark.calendar method.
  def calendar_to_xml(days, tag=nil)
    xml = Builder::XmlMarkup.new(:indent => 2)
    xml.instruct!
    # Build a 'calendar' element.
    xml.calendar(:tag => tag) do
      # For every day in the data structure...
      days.each do |day|
        # ...add a "day" element to the document
        xml.day(:date => day.date, :count => day.count)
      end
    end
  end
end

# recent_controller.rb
class RecentController < ApplicationController

  # GET /recent
  def index
    # Take bookmarks from the database without any special conditions.
    # They'll be ordered with the most recently-posted first.
    show_bookmarks(nil, "Recent bookmarks", recent_url)
  end

  # GET /recent/{tag}
  def show
    # The same as above, but only fetch bookmarks tagged with a
    # certain tag.
    tag = params[:id]
    show_bookmarks(nil, "Recent bookmarks tagged with '#{tag}'", 
                   recent_url(tag), nil, tag)
  end
end

# uris_controller.rb
class UrisController < ApplicationController
  # GET /uris/{URI-MD5}
  def show
    # Fetch all the visible Bookmark objects that correspond to
    # different people bookmarking this URI.
    uri_hash = params[:id]
    sql = ["SELECT bookmarks.*, users.name as user from bookmarks, users" +
           " WHERE users.id = bookmarks.user_id AND bookmarks.uri_hash = ?",
           uri_hash]
    Bookmark.only_visible_to!(sql, @authenticated_user)
    bookmarks = Bookmark.find_by_sql(sql)

    if_found(bookmarks) do

      # Render the list of Bookmark objects as XML or a syndication feed,
      # depending on what the client requested.
      uri = bookmarks[0].uri
      render_bookmarks(bookmarks, "Users who've bookmarked #{uri}",
                       uri_url(uri_hash), nil)
    end
  end
end

This is the simpler of the two models (see Example 7-24). It has some validation rules, a one-to-many relationship with Bookmark objects, and a few methods (called by the controllers) for validating passwords.

class User < ActiveRecord::Base
  # A user has many bookmarks. When the user is destroyed,
  # all their bookmarks should also be destroyed.
  has_many :bookmarks, :dependent => :destroy

  # A user must have a unique username.
  validates_uniqueness_of :name

  # A user must have a username, full name, and email.
  validates_presence_of :name, :full_name, :email

  # Make sure passwords are never stored in plaintext, by running them
  # through a one-way hash as soon as possible.
  def password=(password)
    super(User.hashed(password))
  end

  # Given a username and password, returns a User object if the
  # password matches the hashed one on file. Otherwise, returns nil.
  def self.authenticated_user(username, pass)
    user = find_by_name(username)
    if user
      user = nil unless hashed(pass) == user.password
    end
    return user
  end

  # Performs a one-way hash of some data.
  def self.hashed(password)
    Digest::SHA1.new(password).to_s
  end
end

This is a more complicated model (see Example 7-25). First, let’s define the relationships between Bookmark and the other model classes, along with some validation rules and a rule for generating the MD5 hash of a URI. We have to keep this information because the MD5 calculation only works in one direction. If a client requests /v1/uris/55020a5384313579a5f11e75c1818b89, we can’t reverse the MD5 calculation. We need to be able to look up a URI by its MD5 hash.

class Bookmark < ActiveRecord::Base
  # Every bookmark belongs to some user.
  belongs_to :user

  # A bookmark can have tags. The relationships between bookmarks and
  # tags are managed by the acts_as_taggable plugin.
  acts_as_taggable

  # A bookmark must have an associated user ID, a URI, a short
  # description, and a timestamp.
  validates_presence_of :user_id, :uri, :short_description, :timestamp

  # The URI hash should never be changed directly: only when the URI
  # changes.
  attr_protected :uri_hash

  # And.. here's the code to update the URI hash when the URI changes.
  def uri=(new_uri)
    super
    self.uri_hash = Digest::MD5.hexdigest(new_uri)
  end

  # This method is triggered by Bookmark.new and by
  # Bookmark#update_attributes. It replaces a bookmark's current set
  # of tags with a new set.
  def tag_with(tags)
    Tag.transaction do
      taggings.destroy_all
      tags.each { |name| Tag.find_or_create_by_name(name).on(self) }
    end
  end

That last method makes it possible to associate tags with bookmarks. The acts_as_taggable plugin allows me to do basic queries like “what bookmarks are tagged with ‘ruby’?” Unfortunately, I usually need slightly more complex queries, like “what bookmarks belonging to leonardr are tagged with ‘ruby’?”, so I can’t use the plugin’s find_tagged_with method. I need to define my own method that attaches a tag restriction to some preexisting restriction like “bookmarks belonging to leonardr.”

This custom_find method is the workhorse of the whole service, since it’s called by the ApplicationController#show_bookmarks method, which is called by many of the RESTful Rails actions (see Example 7-26).

  # This method finds bookmarks, possibly ones tagged with a
  # particular tag.
  def self.custom_find(conditions, tag=nil, limit=nil)
    if tag       
      # When a tag restriction is specified, we have to find bookmarks
      # the hard way: by constructing a SQL query that matches only
      # bookmarks tagged with the right tag.
      sql = ["SELECT bookmarks.* FROM bookmarks, tags, taggings" +
             " WHERE taggings.taggable_type = 'Bookmark'" +
             " AND bookmarks.id = taggings.taggable_id" +
             " AND taggings.tag_id = tags.id AND tags.name = ?",
             tag]
      if conditions
        sql[0] << " AND " << conditions[0]
        sql += conditions[1..conditions.size]
      end
      sql[0] << " ORDER BY bookmarks.timestamp DESC"
      sql[0] << " LIMIT " << limit.to_i.to_s if limit
      bookmarks = find_by_sql(sql)
    else
      # Without a tag restriction, we can find bookmarks the easy way:
      # with the superclass find() implementation.
      bookmarks = find(:all, {:conditions => conditions, :limit => limit,
                              :order => 'timestamp DESC'})
    end    
    return bookmarks
  end

There are two more database-related methods (see Example 7-27). The Bookmark.only_visible_to! method manipulates a set of ActiveRecord conditions so that they only apply to bookmarks the given user can see. The Bookmark.calendar method groups a user’s bookmarks by the date they were posted. This implementation may not work for you, since it uses a SQL function (DATE) that’s not available for all databases.

  # Restricts a bookmark query so that it only finds bookmarks visible
  # to the given user. This means public bookmarks, and the given
  # user's private bookmarks.
  def self.only_visible_to!(conditions, user)
    # The first element in the "conditions" array is a SQL WHERE
    # clause with variable substitutions. The subsequent elements are
    # the variables whose values will be substituted. For instance,
    # if "conditions" starts out empty: [""]...

    conditions[0] << " AND " unless conditions[0].empty?
    conditions[0] << "(public='1'"
    if user
      conditions[0] << " OR user_id=?"
      conditions << user.id
    end
    conditions[0] << ")"

    # ...its value might now be ["(public='1' or user_id=?)", 55].
    # ActiveRecord knows how to turn this into the SQL WHERE clause
    # "(public='1' or user_id=55)".
  end 

  # This method retrieves data for the CalendarController. It uses the
  # SQL DATE() function to group together entries made on a particular
  # day.
  def self.calendar(user_id, viewed_by_owner, tag=nil)    
    if tag
      tag_from = ", tags, taggings"
      tag_where = "AND taggings.taggable_type = 'Bookmark'" +
        " AND bookmarks.id = taggings.taggable_id" +
        " AND taggings.tag_id = tags.id AND tags.name = ?"
    end

    # Unless a user is viewing their own calendar, only count public
    # bookmarks.
    public_where = viewed_by_owner ? "" : "AND public='1'"

    sql = ["SELECT date(timestamp) AS date, count(bookmarks.id) AS count" +
           " FROM bookmarks#{tag_from} " +
           " WHERE user_id=? #{tag_where} #{public_where} " +
           " GROUP BY date(timestamp)", user_id]
    sql << tag if tag

    # This will return a list of rather bizarre ActiveRecord objects,
    # which CalendarController knows how to turn into an XML document.
    find_by_sql(sql)
  end
end

Now you should be ready to start your Rails server in a console window, and start using the web service.

$ script/server

There are three possibilities for making it easy to write clients; they’re more or less the ones I covered in Chapters 2 and 3. The simplest is to publish an English description of the service’s layout. If someone wants to use my service they can study my description and write custom HTTP client code.

Most of today’s RESTful and hybrid web services work this way. Instead of specifying the levers of state in hypermedia, they specify the levers in regular media—English text—which a human must interpret ahead of time. You’ll need a basic natural-language description of your service anyway, to serve as advertisement. You want people to immediately see what your service does and want to use it.

I’ve already got a prose description of my social bookmarking service: it takes up much of this chapter. Example 7-28 is a simple command-line Ruby client for the service, based on that prose description. This client knows enough to create user accounts and post bookmarks.

#!/usr/bin/ruby
#open-uri-bookmark-client.rb
require 'rubygems'
require 'rest-open-uri'
require 'uri'
require 'cgi'

# An HTTP-based Ruby client for my social bookmarking service
class BookmarkClient

  def initialize(service_root)
    @service_root = service_root 
  end

  # Turn a Ruby hash into a form-encoded set of key-value pairs.
  def form_encoded(hash)
    encoded = []
    hash.each do |key, value|
      encoded << CGI.escape(key) + '=' + CGI.escape(value)
    end
    return encoded.join('&')
  end

  # Create a new user.
  def new_user(username, password, full_name, email)
    representation = form_encoded({ "user[name]" => username,
                                    "user[password]" => password,
                                    "user[full_name]" => full_name,
                                    "user[email]" => email })      
    puts representation
    begin
      response = open(@service_root + '/users', :method => :post, 
                      :body => representation)
      puts "User #{username} created at #{response.meta['location']}"
    rescue OpenURI::HTTPError => e
      response_code = e.io.status[0].to_i
      if response_code == 409 # Conflict
        puts "Sorry, there's already a user called #{username}."
      else
        raise e
      end
    end
  end

  # Post a new bookmark for the given user.
  def new_bookmark(username, password, uri, short_description)
    representation = form_encoded({ "bookmark[uri]" => uri,
                                    "bookmark[short_description]" => 
                                    short_description })
    begin
      dest = "#{@service_root}/users/#{URI.encode(username)}/bookmarks"
      response = open(dest, :method => :post, :body => representation,
                      :http_basic_authentication => [username, password])
      puts "Bookmark posted to #{response.meta['location']}"
    rescue OpenURI::HTTPError => e
      response_code = e.io.status[0].to_i
      if response_code == 401 # Unauthorized
        puts "It looks like you gave me a bad password."
      elsif response_code == 409 # Conflict
        puts "It looks like you already posted that bookmark."
      else
        raise e
      end
    end    
  end
end

# Main application
command = ARGV.shift
if ARGV.size != 4 || (command != "new-user" && command != "new-bookmark")
  puts "Usage: #{$0} new-user [username] [password] [full name] [email]"
  puts "Usage: #{$0} new-bookmark [username] [password]" +
    " [URI] [short description]"
  exit
end

client = BookmarkClient.new('http://localhost:3000/v1')
if command == "new-user"
  username, password, full_name, email = ARGV
  client.new_user(username, password, full_name, email)
else
  username, password, uri, short_description = ARGV
  client.new_bookmark(username, password, uri, short_description)
end

One alternative to explaining everything is to make your service like other services. If all services exposed the same representation formats, and mapped URIs to resources in the same way... well, we can’t get rid of client programming altogether, but clients could work on a higher level than HTTP.^[23]Conventions are powerful tools: in fact, they’re the same tools that REST uses. Every RESTful resource-oriented web service uses URIs to designate resources, and expresses operations in terms of HTTP’s uniform interface. The idea here is to apply higher-level conventions than REST’s, so that the client programmer doesn’t have to write as much code.

Take the Rails architecture as an example. Rails is good at gently imposing its design preferences on the programmer. The result is that most RESTful Rails services do the same kind of thing in the same way. At bottom, the job of almost every Rails service is to send and accept representations of ActiveRecord objects. These services all map URIs to Rails controllers, Rails controllers to resources, resources to ActiveRecord objects, and ActiveRecord objects to rows in the database. The representation formats are also standardized: either as XML documents like the one in Example 7-4, or form-encoded key-value pairs like the ones in Example 7-5. They’re not the best representation formats, because it’s difficult to make connected services out of them, but they’re OK.

The ActiveResource library, currently under development, is a client library that takes advantage of these similarities between Rails services. I first mentioned ActiveResource in Chapter 3, where I showed it in action against a very simple Rails service. It doesn’t replace custom client code, but it hides the details of HTTP access behind an interface that looks like ActiveRecord. The ActiveResource/ActiveRecord approach won’t work for all web services, or even all Rails web services. It doesn’t work very well on this service. But it’s not quite fair for me to judge ActiveResource by these standards, since it’s still in development. As of the time of writing, it’s more a promising possiblity than a real-world solution to a problem.

Even when the Ruby ActiveResource client is improved and officially released, it will be nothing more than the embodiment of some high-level design conventions. The conventions are useful: another web service framework might copy these conventions, and then Ruby’s ActiveResource client would work with it. An ActiveResource library written in another language will work with Rails services. But if a service doesn’t follow the conventions, ActiveResource can’t talk to it.

What we need is a general framework, a way for each individual service to tell the client about its resource design, its representation formats, and the links it provides between resources. That will give us some of the benefits of standardized conventions, without forcing all web services to comply with more than a few minimal requirements.

This brings us full circle to the REST notion of connectedness, of “hypermedia as the engine of application state.” I talk about connectedness so much because hypermedia links and forms are these machine-readable conventions for describing the differences between services. If your service only serves serialized data structures that show the current resource state, then of course you start thinking about additional standards and conventions. Your representations are only doing half a job.

We don’t think the human web needs these additional standards, because the human web serves documents full of links and forms, not serialized data structures that need extra interpretation. The links and forms on the human web tell our web browsers how to manipulate application and resource state, in response to our expressed desires. It doesn’t matter that every web site was designed by a different person, because the differences between them are represented in machine-readable format.

The XHTML links and forms in Chapters 5 and 6 are machine-readable descriptions of what makes the fantasy map service different from other services. In this chapter, the links embedded in the Atom documents are machine-readable descriptions of the connections that distinguish this service from others that serve Atom documents. In Chapter 9 I’ll consider three major hypermedia formats that can describe these differences between services: XHTML 4, XHTML 5, and WADL. For now, though, it’s time to take a step back and take a look at REST and the ROA as a whole.

1. Figure out the data set

2. Split the data set into resources

   For each kind of resource:

3. Name the resources with URIs

4. Expose a subset of the uniform interface

5. Design the representation(s) accepted from the client

6. Design the representation(s) served to the client

7. Integrate this resource into existing resources, using hypermedia links and forms

8. Consider the typical course of events: what’s supposed to happen? Standard control flows like the Atom Publishing Protocol can help (see Chapter 9).

9. Consider error conditions: what might go wrong? Again, standard control flows can help.

A URI should never represent more than one resource. Then it wouldn’t be a Universal Resource Identifier. Furthermore, I suggest that every representation of a resource should have its own URI. This is because URIs are often passed around or used as input to other web services. The expectation then is that the URI designates a particular representation of the resource.

Let’s say you’ve exposed a press release at /releases/104. There’s an English and a Spanish version of the press release, an HTML and plain-text version of each. Your clients should be able set the Accept-Language request header to choose an English or Spanish representation of /releases/104, and the Accept request header to choose an HTML or plain-text representation. But you should also give each representation a separate URI: maybe URIs like /releases/104.en, /releases/104.es.html, and /releases/104.txt.

In the bookmarking service from Chapter 7, I exposed two representations of a set of bookmarks: a generic XML representation at /v1/users/leonardr/bookmarks.xml, and an Atom representation at /v1/users/leonardr/bookmarks.atom. I also exposed a canonical URI for the resource at /v1/users/leonardr/bookmarks. A client can set its Accept request header to distinguish between Atom and generic XML representations of /v1/users/leonardr/bookmarks, or it can tweak the URI to get a different representation. Both techniques work, and both techniques are RESTful, but a URI travels better across clients if it specifies a resource and a representation.

It’s OK for a client to send information in HTTP request headers, so long as the server doesn’t make that the only way of selecting a resource or representation. Headers can also contain sensitive information like authentication credentials, or information that’s different for every client. But headers shouldn’t be the only tool a client has to specify which representation is served or which resource is selected.

A GET or HEAD request should be safe: a client that makes a GET or HEAD request is not requesting any changes to server state. The server might decide on its own to change state (maybe by logging the request or incrementing a hit counter), but it should not hold the client responsible for those changes. Making any number of GET requests to a certain URI should have the same practical effect as making no requests at all.

A PUT or DELETE request should be idempotent. Making more than one PUT or DELETE request to a given URI should have the same effect as making only one. One common problem: PUT requests that set resource state in relative terms like “increment value by 5.” Making 10 PUT requests like that is a lot different from just making one. PUT requests should set items of resource state to specific values.

The safe methods, GET and HEAD, are automatically idempotent as well. POST requests for resource creation are neither safe nor idempotent. An overloaded POST request might or might not be safe or idempotent. There’s no way for a client to tell, since overloaded POST can do anything at all. You can make POST idempotent with POST Once Exactly (see Chapter 9).

You can expose the creation of new resources through PUT, POST, or both. But a client can only use PUT to create resources when it can calculate the final URI of the new resource. In Amazon’s S3 service, the URI path to a bucket is /{bucket-name}. Since the client chooses the bucket name, a client can create a bucket by constructing the corresponding URI and sending a PUT request to it.

On the other hand, the URI to a resource in a typical Rails web service looks like /{database-table-name}/{database-ID}. The name of the database table is known in advance, but the ID of the new resource won’t be known until the corresponding record is saved to the database. To create a resource, the client must POST to a “factory” resource, located at /{database-table-name}. The server chooses a URI for the new resource.

POST isn’t just for creating new resources and appending to representations. You can also use it to turn a resource into a tiny RPC-style message processor. A resource that receives an overloaded POST request can scan the incoming representation for additional method information, and carry out any task whatsoever. This gives the resource a wider vocabulary than one that supports only the uniform interface.

This is how most web applications work. XML-RPC and SOAP/WSDL web services also run over overloaded POST. I strongly discourage the use of overloaded POST, because it ruins the uniform interface. If you’re tempted to expose complex objects or processes through overloaded POST, try giving the objects or processes their own URIs, and exposing them as resources. I show several examples of this in Resource Design” later in this chapter.

There are two noncontroversial uses for overloaded POST. The first is to simulate HTTP’s uniform interface for clients like web browsers that don’t support PUT or DELETE. The second is to work around limits on the maximum length of a URI. The HTTP standard specifies no limit on how long a URI can get, but many clients and servers impose their own limits: Apache won’t respond to requests for URIs longer than 8 KB. If a client can’t make a GET request to http://www.example.com/numbers/1111111 because of URI length restrictions (imagine a million more ones there if you like), it can make a POST request to http://www.example.com/numbers?_method=GET and put “1111111” in the entity-body.

If you want to do without PUT and DELETE altogether, it’s entirely RESTful to expose safe operations on resources through GET, and all other operations through overloaded POST. Doing this violates my Resource-Oriented Architecture, but it conforms to the less restrictive rules of REST. REST says you should use a uniform interface, but it doesn’t say which one.

If the uniform interface really doesn’t work for you, or it’s not worth the effort to make it work, then go ahead and overload POST, but don’t lose the resource-oriented design. Every URI you expose should still be a resource: something a client might want to link to. A lot of web applications create new URIs for operations exposed through overloaded POST. You get URIs like /weblog/myweblog/rebuild-index. It doesn’t make sense to link to that URI. Instead of putting method information in the URI, expose overloaded POST on your existing resources (/weblog/myweblog) and ask for method information in the incoming representation (method=rebuild-index). This way, /weblog/myweblog still acts like a resource, albeit one that doesn’t totally conform to the uniform interface. It responds to GET, PUT, DELETE... and also “rebuild-index” through overloaded POST. It’s still an object in the object-oriented sense.

A rule of thumb: if you’re using overloaded POST, and you never expose GET and POST on the same URI, you’re probably not exposing resources at all. You’ve probably got an RPC-style service.

Addressability means that every interesting aspect of your service is immediately accessible from outside. Every interesting aspect of your service has a URI: a unique identifier in a format that’s familiar to every computer-literate person. This identifier can be bookmarked, passed around between applications, and used as a stand-in for the actual resource. Addressability makes it possible for others to make mashups of your service: to use it in ways you never imagined.

In Chapter 4 I compared URIs to cell addresses in a spreadsheet, and to file paths in a command-line shell. The web is powerful in the same way that spreadsheets and command-line shells are powerful. Every piece of information has a structured name that can be used as a reference to the real thing.

Statelessness is the simplifying assumption to beat all simplifying assumptions. Each of a client’s requests contains all application states necessary to understand that request. None of this information is kept on the server, and none of it is implied by previous requests. Every request is handled in isolation and evaluated against the current resource state.

This makes it trivial to scale your application up. If one server can’t handle all the requests, just set up a load balancer and make a second server handle half the requests. Which half? It doesn’t matter, because every request is self-contained. You can assign requests to servers randomly, or with a simple round-robin algorithm. If two servers can’t handle all the requests, you add a third server, ad infinitum. If one server goes down, the others automatically take over for it. When your application is stateless, you don’t need to coordinate activity between servers, sharing memory or creating “server affinity” to make sure the same server handles every request in a “session.” You can throw web servers at the problem until the bottleneck becomes access to your resource state. Then you have to get into database replication, mirroring, or whatever strategy is most appropriate for the way you’ve chosen to store your resource state.

Stateless applications are also more reliable. If a client makes a request that times out, statelessness means the client can resend the request without worrying that its “session” has gone into a strange state that it can’t recover from. If it was a POST request, the client might have to worry about what the request did to the resource state, but that’s a different story. The client has complete control over the application state at all times.

There’s an old joke. Patient: “Doctor, it hurts when I try to scale a system that keeps client state on the server!” Doctor: “Then don’t do that.” That’s the idea behind statelessness: don’t do the thing that causes the trouble.

I covered this in detail near the end of Chapter 4, so I’ll just give a brief recap here. If you say to me, “I’ve exposed a resource at http://www.example.com/myresource,” that gives me no information about what that resource is, but it tells me a whole lot about how I can manipulate it. I know how to fetch a representation of it (GET), I know how to delete it (DELETE), I know roughly how to modify its state (PUT), and I know roughly how to spawn a subordinate resource from it (POST).

There are still details to work out: which of these activities the resource actually supports,^[24]which representation formats the resource serves and expects, and what this resource represents in the real world. But every resource works basically the same way and can be accessed with a universal client. This is a big part of the success of the Web.

The restrictions imposed by the uniform interface (safety for GET and HEAD, idempotence for PUT and DELETE), make HTTP more reliable. If your request didn’t go through, you can keep resending it with no ill effects. The only exception is with POST requests. (See POST Once Exactly” in Chapter 9 for ways of making POST idempotent.)

The power of the uniform interface is not in the specific methods exposed. The human web has a different uniform interface—it uses GET for safe operations, and POST for everything else—and it does just fine. The power is the uniformity: everyone uses the same methods for everything. If you deviate from the ROA’s uniform interface (say, by adopting the human web’s uniform interface, or WebDAV’s uniform interface), you switch communities: you gain compatibility with certain web services at the expense of others.

Imagine the aggravation if instead of hypertext links, web pages gave you English instructions on how to construct the URI to the next page. That’s how most of today’s RESTful web services work: the resources aren’t connected to each other. This makes web services more brittle than human-oriented web sites, and it means that emergent properties of the Web (like Google’s PageRank) don’t happen on the programmable web.

Look at Amazon S3. It’s a perfectly respectable resource-oriented service. It’s addressable, it’s stateless, and it respects the uniform interface. But it’s not connected at all. The representation of the S3 bucket list gives the name of each bucket, but it doesn’t link to the buckets. The representation of a bucket gives the name of each object in the bucket, but it doesn’t link to the objects. We humans know these objects are conceptually linked, but there are no actual links in the representations (see Figure 8-1).

Figure 8-1. We see links, but there are none

An S3 client can’t get from one resource to another by following links. Instead it must internalize rules about how to construct the URI to a given bucket or object. These rules are given in the S3 technical documentation, not anywhere in the service itself. I demonstrated the rules in Resources” in Chapter 3. This wouldn’t work on the human web, but in a web service we don’t complain. Why is that?

In general, we expect less from web services than from the human web. We experience the programmable web through customized clients, not generic clients like web browsers. These customized clients can be programmed with rules for URI construction. Most information on the programmable web is also available on the human web, so a lack of connectedness doesn’t hide data from generic clients like search engines. Or else the information is hidden behind an authentication barrier and you don’t want a search engine seeing it anyway.

The S3 service gets away with a lack of connectedness because it only has three simple rules for URI construction. The URI to a bucket is just a slash and the URI-escaped name of the bucket. It’s not difficult to program these rules into a client. The only bug that’s at all likely is a failure to URI-escape the bucket or object name. Of course, there are additional rules for filtering and paginating the contents of buckets, which I skimmed over in Chapter 3. Those rules are more complex, and it would be better for S3 representations to provide hypermedia forms instead of making clients construct these URIs on their own.

More importantly, the S3 resources have simple and stable relationships to each other. The bucket list contains buckets, and a bucket contains objects. A link is just an indication of a relationship between two resources. A simple relationship is easy to program into a client, and “contains” is one of the simplest. If a client is preprogrammed with the relationships between resources, links that only serve to convey those relationships are redundant.

The social bookmarking service I implemented in Chapter 7 is a little better-connected than S3. It represents lists of bookmarks as Atom documents full of internal and external links. But it’s not totally connected: its representation of a user doesn’t link to that user’s bookmarks, posting history, or tag vocabulary (look back to Figure 7-1). And there’s no information about where to find a user in the service, or how post a bookmark. The client is just supposed to know how to turn a username into a URI, and just supposed to know how to represent a bookmark.

It’s easy to see how this is theoretically unsatisfying. A service ought to be self-describing, and not rely on some auxiliary English text that tells programmers how to write clients. It’s also easy to see that a client that relies on rules for URI construction is more brittle. If the server changes those rules, it breaks all the clients. It’s less easy to see the problems that stem from a lack of connectedness when the relationships between resources are complex or unstable. These problems can break clients even when the rules for URI construction never change.

Let’s go back to the mapping service from Chapter 5. My representations were full of hyperlinks and forms, most of which were not technically necessary. Take this bit of markup from the representation of a road map that was in Example 5-6:

<a class="zoom_in" href="/road.1/Earth/37.0,-95.8" />Zoom out</a>
<a class="zoom_out" href="/road.3/Earth/37.0,-95.8" />Zoom in</a>

Instead of providing these links everywhere, the service provider could put up an English document telling the authors of automated clients how to manipulate the zoom level in the first path variable. That would disconnect some related resources (the road map at different zoom levels), but it would save some bandwidth in every representation and it would have little effect on the actual code of any automated client. Personally, if I was writing a client for this service, I’d rather get from zoom level 8 to zoom level 4 by setting road.4 directly, than by following the “Zoom out” link over and over again. My client will break if the URI construction rule ever changes, but maybe I’m willing to take that risk.

Now consider this bit of markup from the representation of the planet Earth. It’s reprinted from Example 5-7:

 <dl class="place">
  <dt>name</dt> <dd>Earth</dd>
  <dt>maps</dt>
    <ul class="maps">
     <li><a class="map" href="/road/Earth">Road</a></li>
     <li><a class="map" href="/satellite/Earth">Satellite</a></li>
     ...
    </ul>

The URIs are technically redundant. The name of the place indicates that these are maps of Earth, and the link text indicates that there’s a satellite and a road map. Given those two pieces of information, a client can construct the corresponding map URI using a rule like the one for S3 objects: slash, map type, slash, planet name. Since the URIs can be replaced by a simple rule, the service might follow the S3 model and save some bandwidth by presenting the representation of Earth in an XML format like this:

<place name="Earth" type="planet">
 <map type="satellite" />
 <map type="road" />
 ...
</place>

If I was writing a client for this service, I would rather be given those links than have to construct them myself, but it’s up for debate.

Here’s another bit of markup from Example 5-6. These links are to help the client move from one tile on the map to another.

<a class="map_nav" href="46.0518,-95.8">North</a>
<a class="map_nav" href="41.3776,-89.7698">Northeast</a>
<a class="map_nav" href="36.4642,-84.5187">East</a>
<a class="map_nav" href="32.3513,-90.4459">Southeast</a>

It’s technically possible for a client to generate these URIs based on rules. After all, the server is generating them based on rules. But the rules involve knowing how latitude and longitude work, the scale of the map at the current zoom level, and the size and shape of the planet. Any client programmer would agree it’s easier to navigate a map by following the links than by calculating the coordinates of tiles. We’ve reached a point at which the relationships between resources are too complex to be expressed in simple rules. Connectedness becomes very important.

This is where Google Maps’s tile-based navigation system pays off (I described that system back in Representing Maps and Points on Maps” in Chapter 5, if you’re curious). Google Maps addresses its tiles by arbitrary X and Y coordinates instead of latitude and longitude. Finding the tile to the north is usually as easy as subtracting one from the value of Y. The relationships between tiles are much simpler. Nobody made me design my tile system in terms of latitude and longitude. If latitude/longitude calculations are why I have to send navigation links along with every map representation, maybe I should rethink my strategy and expose simpler URIs, so that my clients can generate them more easily.

But there’s another reason why connectedness is valuable: it makes it possible for the client to handle relationships that change over time. Links not only hide the rules about how to build a URI for a given resource, they embody the rules of how resources are related to each other. Here’s a terrifying example to illustrate the point.

A terrifying example

Suppose I get some new map data for my service. It’s more accurate than the old data, but the scale is a little different. At zoom level 8, the client sees a slightly smaller map than it did before. Let’s say at zoom level 8, a tile 256 pixels square now depicts an area three-quarters of a mile square, instead of seven-eigths of a mile square.

At first glance, this has no effect on anything. Latitude and longitude haven’t changed, so every point on the old map is in the same place on the new map. Google Maps-style tile URIs would break at this point, because they use X and Y instead of latitude and longitude. When the map data was updated, I’d have to recalculate all the tile images. Many points on the map would suddenly shift to different tiles, and get different X and Y coordinates. But all of my URIs still work. Every point on the map has the same URI it did before.

In this new data set, the URI /road.8/Earth/40.76,-73.98.png still shows part of the island of Manhattan, and the URI /road.8/Earth/40.7709,-73.98 still shows a point slightly to the north. But the rules have changed for finding the tile directly to the north of another tile. Those two tile graphics are centered on the same coordinates as before, but now each tile depicts a slightly smaller space. They used to be adjacent on the map, but now there’s a gap between them (see Figure 8-2).

Figure 8-2. When clients choose URIs for map tiles: before and after

If a client application finds nearby tiles by following the navigation links I provide, it will automatically adapt to the new map scale. But an application that “already knows” how to turn latitude and longitude into image URIs will suddenly start showing maps that look like MAD Magazine fold-ins.

I made a reasonable change to my service that didn’t change any URIs, but it broke clients that always construct their own URIs. What changed was not the resources but the relationships between them: not the rules for constructing URIs but the rules for driving the application from one state to another. Those rules are embedded in my navigation links, and a client duplicates those rules at its own peril.

And that’s why it’s important to connect your resources to each other. It’s fine to expect your clients to use your rules to construct an initial URI (say, a certain place on the map at a certain zoom level), but if they need to navigate from one URI to another, you should provide appropriate links. As the programmable web matures, connectedness will become more and more important.

• Predefined one-off resources, such as your service’s home page or a static list of links to resources. A resource of this type corresponds to something you’ve only got a few of: maybe a class in an object-oriented system, or a database table in a database-oriented system.

• A large (possibly infinite) number of resources corresponding to individual items of data. A resource of this type might correspond to an object in an object-oriented system, or a database row in a database-oriented system.

• A large (probably infinite) number of resources corresponding to the possible outputs of an algorithm. A resource of this type might correspond to the results of a query in a database-oriented system. Lists of search results and filtered lists of resources fall into this category.

Suppose Alice and Bob are resources in my service. That is, they’re people in the real world, but my service gives them URIs and offers representations of their state. One day Alice and Bob get married. How should this be represented in my service?

A client can PUT to Alice’s URI, modifying her state to reflect the fact that she’s married to Bob, and then PUT to Bob’s URI to say he’s married to Alice. That’s not very satisfying because it’s two steps. A client might PUT to Alice’s URI and forget to PUT to Bob’s. Now Alice is married to Bob but not vice versa.

Instead I should treat the marriage, this relationship between two resources, as a thing in itself: a third resource. A client can declare two people married by sending a PUT request to a “marriage” URI or a POST request to a “registrar” URI (it depends on how I choose to do the design). The representation includes links to Alice and Bob’s URIs: it’s an assertion that the two are married. The server applies any appropriate rules about who’s allowed to get married, and either sends an error message or creates a new resource representing the marriage. Other resources can now link to this resource, and it responds to the uniform interface. A client can GET it or DELETE it (though hopefully DELETEing it won’t be necessary).

HTTP has a synchronous request-response model. The client opens an Internet socket to the server, makes its request, and keeps the socket open until the server has sent the response. If the client doesn’t care about the response it can close the socket early, but to get a response it must leave the socket open until the server is ready.

The problem is not all operations can be completed in the time we expect an HTTP request to take. Some operations take hours or days. An HTTP request would surely be timed out after that kind of inactivity. Even if it didn’t, who wants to keep a socket open for days just waiting for a server to respond? Is there no way to expose such operations asynchronously through HTTP?

There is, but it requires that the operation be split into two or more synchronous requests. The first request spawns the operation, and subsequent requests let the client learn about the status of the operation. The secret is the status code 202 (“Accepted”).

I’ll demonstrate one strategy for implementing asynchronous requests with the 202 status code. Let’s say we have a web service that handles a queue of requests. The client makes its service request normally, possibly without any knowledge that the request will be handled asynchronously. It sends a request like this one:

POST /queue HTTP/1.1
Host: jobs.example.com
Authorization: Basic mO1Tcm4hbAr3gBUzv3kcceP=

Give me the prime factorization of this 100000-digit number:
...

The server accepts the request, creates a new job, and puts it at the end of the queue. It will take a long time for the new job to be completed, or there wouldn’t be a need for a queue in the first place. Instead of keeping the client waiting until the job finally runs, the server sends this response right away:

202 Accepted
Location: http://jobs.example.com/queue/job11a4f9

The server has created a new “job” resource and given it a URI that doesn’t conflict with any other job. The asynchronous operation is now in progress, and the client can make GET requests to that URI to see how it’s going— that is, to get the current state of the “job” resource. Once the operation is complete, any results will become available as a representation of this resource. Once the client is done reading the results it can DELETE the job resource. The client may even be able to cancel the operation by DELETEing its job prematurely.

Again, I’ve overcome a perceived limitation of the Resource-Oriented Architecture by exposing a new kind of resource corresponding to the thing that was causing the problem. In this case, the problem was how to handle asynchronous operations, and the solution was to expose each asynchronous operation as a new resource.

There’s one wrinkle. Because every request to start an asynchronous operation makes the server create a new resource (if only a transient one), such requests are neither safe nor idempotent. This means you can’t spawn asynchronous operations with GET, DELETE, or (usually) PUT. The only HTTP method you can use and still respect the uniform interface is POST. This means you’ll need to expose different resources for asynchronous operations than you would for synchronous operations. You’ll probably do something like the job queue I just demonstrated. You’ll expose a single resource—the job queue—to which the client POSTs to create a subordinate resource—the job. This will hold true whether the purpose of the asynchronous operation is to read some data, to make a calculation (as in the factoring example), or to modify the data set.

Sometimes clients need to operate on more than one resource at once. You’ve already seen this: a list of search results is a kind of batch GET. Instead of fetching a set of resources one at a time, the client specifies some criteria and gets back a document containing abbreviated representations of many resources. I’ve also mentioned “factory” resources that respond to POST and create subordinate resources. The factory idea is easy to scale up. If your clients need to create resources in bulk, you can expose a factory resource whose incoming representation describes a set of resources instead of just one, and creates many resources in response to a single request.

What about modifying or deleting a set of resources at once? Existing resources are identified by URI, but addressability means an HTTP request can only point to a single URI, so how can you DELETE two resources at once? Remember that URIs can contain embedded URI paths, or even whole other URIs (if you escape them). One way to let a client modify multiple resources at once is to expose a resource for every set of resources. For instance, http://www.example.com/sets/resource1;subdir/resource2 might refer to a set of two resources: the one at http://www.example.com/resource1 and the one at http://www.example.com/subdir/resource2. Send a DELETE to that “set” resource and you delete both resources in the set. Send a PUT instead, with a representation of each resource in the set, and you can modify both resources with a single HTTP request.

You might be wondering what HTTP status code to send in response to a batch operation. After all, one of those PUTs might succeed while the other one fails. Should the status code be 200 (“OK”) or 500 (“Internal Server Error”)? One solution is to make a batch operation spawn a series of asynchronous jobs. Then you can send 202 (“Accepted”), and show the client how to check on the status of the individual jobs. Or, you can use an extended HTTP status code created by the WebDAV extension to HTTP: 207 (“Multi-Status”).

The 207 status code tells the client to look in the entity-body for a list of status codes like 200 (“OK”) and 500 (“Internal Server Error”). The entity-body is an XML document that tells the client which operations succeeded and which failed. This is not an ideal solution, since it moves information about what happened out of the status code and into the response entity-body. It’s similar to the way overloaded POST moves the method information out of the HTTP method and into the request entity-body. But since there might be a different status code for every operation in the batch, you’re really limited in your options here. Appendix B has more information about the 207 status code.

In the Resource-Oriented Architecture, every incoming HTTP request has some resource as its destination. But some services expose operations that span multiple resources. The classic example is an operation that transfers money from a checking to a savings account. In a database-backed system you’d use a transaction to prevent the possibility of losing or duplicating money. Is there a resource-oriented way to implement transactions?

You can expose simple transactions as batch operations, or use overloaded POST, but here’s another way. It involves (you guessed it) exposing the transactions themselves as resources. I’ll show you a sample transaction using the account transfer example. Let’s say the “checking account” resource is exposed at /accounts/checking/11, and the “savings account” resource is exposed at /accounts/savings/55. Both accounts have a current balance of $200, and I want to transfer $50 from checking to savings.

I’ll quickly walk you through the requests and then explain them. First I create a transaction by sending a POST to a transaction factory resource:

POST /transactions/account-transfer HTTP/1.1
Host: example.com

The response gives me the URI of my newly created transaction resource:

201 Created
Location: /transactions/account-transfer/11a5

I PUT the first part of my transaction: the new, reduced balance of the checking account.

PUT /transactions/account-transfer/11a5/accounts/checking/11 HTTP/1.1
Host: example.com

balance=150

I PUT the second part of my transaction: the new, increased balance of the savings account.

PUT /transactions/account-transfer/11a5/accounts/savings/55 HTTP/1.1
Host: example.com

balance=250

At any point up to this I can DELETE the transaction resource to roll back the transaction. Instead, I’m going to commit the transaction:

PUT /transactions/account-transfer/11a5 HTTP/1.1
Host: example.com

committed=true

This is the server’s chance to make sure that the transaction doesn’t create any inconsistencies in resource state. For an “account transfer” transaction the server should check whether the transaction tries to create or destroy any money, or whether it tries to move money from one person to another without authorization. If everything checks out, here’s the response I might get from my final PUT:

200 OK
Content-Type: application/xhtml+xml

...
<a href="/accounts/checking/11">Checking #11</a>: New balance $150
<a href="/accounts/savings/55">Savings #55</a>: New balance $250
...

At this point I can DELETE the transaction and it won’t be rolled back. Or the server might delete it automatically. More likely, it will be archived permanently as part of an audit trail. It’s an addressable resource. Other resources, such as a list of transactions that affected checking account #11, can link to it.

The challenge in representing transactions RESTfully is that every HTTP request is supposed to be a self-contained operation that operates on one resource. If you PUT a new balance to /accounts/checking/11, then either the PUT succeeds or it doesn’t. But during a transaction, the state of a resource is in flux. Look at the checking account from inside the transaction, and the balance is $150. Look at it from outside, and the balance is still $200. It’s almost as though there are two different resources.

That’s how this solution presents it: as two different resources. There’s the actual checking account, at /accounts/checking/11, and there’s one transaction’s view of the checking account, at /transactions/account-transfer/11a5/accounts/checking/11. When I POSTed to create /transactions/account-transfer/11a5/, the service exposed additional resources beneath the transaction URI: probably one resource for each account on the system. I manipulated those resources as I would the corresponding account resources, but my changes to resource state didn’t go “live” until I committed the transaction.

How would this be implemented behind the scenes? Probably with something that takes incoming requests and builds a queue of actions associated with the transaction. When the transaction is committed the server might start a database transaction, apply the queued actions, and then try to commit the database transaction. A failure to commit would be propagated as a failure to commit the web transaction.

A RESTful transaction is more complex to implement than a database or programming language transaction. Every step in the transaction comes in as a separate HTTP request. Every step identifies a resource and fits the uniform interface. It might be easier to punt and use overloaded POST. But if you implement transactions RESTfully, your transactions have the benefits of resources: they’re addressable, operations on them are transparent, and they can be archived or linked to later. Yet again, the way to deal with an action that doesn’t fit the uniform interface is to expose the action itself as a resource.

The techniques I’ve shown you are not the official RESTful or resource-oriented ways to handle transactions, asynchronous operations, and so on. They’re just the best ones I could think up. If they don’t work for you, you’re free to try another arrangement.

The larger point of this section is that when I say “anything can be a resource” I do mean anything. If there’s a concept that’s causing you design troubles, you can usually fit it into the ROA by exposing it as a new kind of resource. If you need to violate the uniform interface for performance reasons, you’ve always got overloaded POST. But just about anything can be made to respond to the uniform interface.

By now you probably know that HTTP authentication and authorization are handled with HTTP headers—“stickers” on the HTTP “envelope.” You might not know that these headers were designed to be extensible. HTTP defines two authentication schemes, but there’s a standard way of integrating other authentication schemes into HTTP, by customizing values for the headers Authorization and WWW-Authenticate. You can even define custom authentication schemes and integrate them into HTTP: I’ll show you how that’s done by adapting a small portion of the WS-Security standard to work with HTTP authentication. But first, I’ll cover the two predefined schemes.

GET /resource.html HTTP/1.1
Host: www.example.com

401 Unauthorized
WWW-Authenticate: Basic realm="My Private Data"

#!/usr/bin/ruby
# calculate-base64.rb
USER="Alibaba"
PASSWORD="open sesame"

require 'base64'
puts Base64.encode64("#{USER}:#{PASSWORD}")
# QWxpYmFiYTpvcGVuIHNlc2FtZQ==

GET /resource.html HTTP/1.1
Host: www.example.com
Authorization: Basic QWxpYmFiYTpvcGVuIHNlc2FtZQ==

401 Unauthorized
WWW-Authenticate: Digest realm="My Private Data",
  qop="auth",
  nonce="0cc175b9c0f1b6a831c399e269772661",
  opaque="92eb5ffee6ae2fec3ad71c777531578f"

#!/usr/bin/ruby
# calculate-http-digest.rb
require 'md5'

#Information from the original request
METHOD="GET"
PATH="/resource.html"

# Information from the challenge
REALM="My Private Data"
NONCE="0cc175b9c0f1b6a831c399e269772661",
OPAQUE="92eb5ffee6ae2fec3ad71c777531578f"
QOP="auth"

# Information calculated by or known to the client
NC="00000001"
CNONCE="4a8a08f09d37b73795649038408b5f33"
USER="Alibaba"
PASSWORD="open sesame"

# Calculate the final digest in three steps.
ha1 = MD5::hexdigest("#{USER}:#{REALM}:#{PASSWORD}")
ha2 = MD5::hexdigest("#{METHOD}:#{PATH}")
ha3 = MD5::hexdigest("#{ha1}:#{NONCE}:#{NC}:#{CNONCE}:#{QOP}:#{ha2}")

puts ha3
# 2370039ff8a9fb83b4293210b5fb53e3

GET /resource.html HTTP/1.1
Host: www.example.com
Authorization: Digest username="Alibaba",
  realm="My Private Data",
  nonce="0cc175b9c0f1b6a831c399e269772661",
  uri="/resource.html",
  qop=auth,
  nc=00000001,
  cnonce="4a8a08f09d37b73795649038408b5f33",
  response="2370039ff8a9fb83b4293210b5fb53e3",
  opaque="92eb5ffee6ae2fec3ad71c777531578f"

HTTP/1.1 401 Unauthorized
WWW-Authenticate: WSSE realm="My Private Data", profile="UsernameToken"

#!/usr/bin/ruby
# calculate-wsse-digest.rb
require 'base64'
require 'sha1'

PASSWORD = "open sesame"
NONCE = "EFD89F06CCB28C89",
CREATED = "2007-04-13T09:00:00Z"

puts Base64.encode64(SHA1.digest("#{NONCE}#{CREATED}#{PASSWORD}"))
# Z2Y59TewHV6r9BWjtHLkKfUjm2k=

GET /resource.html HTTP/1.1
Host: www.example.com
Authorization: WSSE profile="UsernameToken"
X-WSSE: UsernameToken Username="Alibaba",
  PasswordDigest="Z2Y59TewHV6r9BWjtHLkKfUjm2k=",
  Nonce="EFD89F06CCB28C89",
  Created="2007-04-13T09:00:00Z"

<wsse:UsernameToken
   xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/xx/secext"
   xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/xx/utility">
   <wsse:Username>Alibaba</wsse:Username>
   <wsse:Password Type="wsse:PasswordDigest">
      Z2Y59TewHV6r9BWjtHLkKfUjm2k=
   </wsse:Password>
   <wsse:Nonce>EFD89F06CCB28C89</wsse:Nonce>
   <wsu:Created>2007-04-13T09:00:00Z</wsu:Created>
</wsse:UsernameToken>

Textual representations like XML documents can be compressed to a fraction of their original size. An HTTP client library can request a compressed version of a representation and then transparently decompress it for its user. Here’s how it works: along with an HTTP request the client sends an Accept-Encoding header that says what kind of compression algorithms the client understands. The two standard values for Accept-Encoding are compress and gzip.

GET /resource.html HTTP/1.1
Host: www.example.com
Accept-Encoding: gzip,compresss

If the server understands one of the compression algorithms from Accept-Encoding, it can use that algorithm to compress the representation before serving it. The server sends the same Content-Type it would send if the representation wasn’t compressed. But it also sends the Content-Encoding header, so the client knows the document has been compressed:

200 OK
Content-Type: text/html
Content-Encoding: gzip

[Binary representation goes here]

The client decompresses the data using the algorithm given in Content-Encoding, and then treats it as the media type given as Content-Type. In this case the client would use the gzip algorithm to decompress the binary data back into an HTML document. This technique can save a lot of bandwidth, with very little cost in additional complexity.

You probably remember that I think different representations of a resource should have distinct URIs. Why do I recommend using HTTP headers to distinguish between compressed and uncompressed versions of a representation? Because I don’t think the compressed and uncompressed versions are different representations. Compression, like encryption, is something that happens to a representation in transit, and must be undone before the client can use the representation. In an ideal world, HTTP clients and servers would compress and decompress representations automatically, and programmers should not have to even think about it. Today, most web browsers automatically request compressed representations, but few programmable clients do.

Conditional HTTP GET allows a server and client to work together to save bandwidth. I covered it briefly in Chapter 5, in the context of the mapping service. There, the problem was sending the same map tiles over and over again to clients who had already received them. This is a more general treatment of the same question: how can a service keep from sending representations to clients that already have them?

Neither client nor server can solve this problem alone. If the client retrieves a representation and never talks to the server again, it will never know when the representation has changed. The server keeps no application state, so it doesn’t know when a client last retrieved a certain representation. HTTP isn’t a reliable protocol anyway, and the client might not have received the representation the first time. So when the client requests a representation, the server has no idea whether the client has done this before—unless the client provides that information as part of the application state.

Conditional HTTP GET requires client and server to work together. When the server sends a representation, it sets some HTTP response headers: Last-Modified and/or ETag. When the client requests the same representation, it should send the values for those headers as If-Modified-Since and/or If-None-Match. This lets the server make a decision about whether or not to resend the representation. Example 8-4 gives a demonstration of conditional HTTP GET.

#!/usr/bin/ruby
# fetch-oreilly-conditional.rb

require 'rubygems'
require 'rest-open-uri'
uri = 'http://www.oreilly.com'

# Make an HTTP request and then describe the response.
def request(uri, *args)
  begin
    response = open(uri, *args)
  rescue OpenURI::HTTPError => e
    response = e.io
  end
    
  puts " Status code: #{response.status.inspect}"
  puts " Representation size: #{response.size}"
  last_modified = response.meta['last-modified']
  etag = response.meta['etag']
  puts " Last-Modified: #{last_modified}"
  puts " Etag: #{etag}"
  return last_modified, etag
end    

puts "First request:"
last_modified, etag = request(uri)

puts "Second request:"
request(uri, 'If-Modified-Since' => last_modified, 'If-None-Match' => etag)

If you run that code once, it’ll fetch http://www.oreilly.com twice: once normally and once conditionally. It prints information about each request. The printed output for the first request will look something like this:

First request:
 Status code: ["200", "OK"]
 Representation size: 41123
 Last-Modified: Sun, 21 Jan 2007 09:35:19 GMT
 Etag: "7359b7-a37c-45b333d7"

The Last-Modified and Etag headers are the ones that make HTTP conditional GET possible. To use them, I make the HTTP request again, but this time I use the value of Last-Modified as If-Modified-Since, and the value of ETag as If-None-Match. Here’s the result:

Second request:
 Status code: ["304", "Not Modified"]
 Representation size: 0
 Last-Modified:
 Etag: "7359b7-a0a3-45b5d90e"

Instead of a 40-KB representation, the second request gets a 0-byte representation. Instead of 200 (“OK”), the status code is 304 (“Not Modified”). The second request saved 40 KB of bandwidth because it made the HTTP request conditional on the representation of http://www.oreilly.com/ actually having changed since last time. The representation didn’t change, so it wasn’t resent.

Last-Modified is a pretty easy header to understand: it’s the last time the representation of this resource changed. You may be able to view this information in your web browser by going to “view page info” or something similar. Sometimes humans check a web page’s Last-Modified time to see how recent the data is, but its main use is in conditional HTTP requests.

If-Modified-Since makes an HTTP request conditional. If the condition is met, the server carries out the request as it would normally. Otherwise, the condition fails and the server does something unusual. For If-Modified-Since, the condition is: “the representation I’m requesting must have changed after this date.” The condition succeeds when the server has a newer representation than the client does. If the client and server have the same representation, the condition fails and the server does something unusual: it omits the representation and sends a status code of 304 (“Not Modified”). That’s the server’s way of telling the client: “reuse the representation you saved from last time.”

Both client and server benefit here. The server doesn’t have to send a representation of the resource, and the client doesn’t have to wait for it. Both sides save bandwidth. This is one of the tricks underlying your web browser’s cache, and there’s no reason not to use it in custom web clients.

How does the server calculate when a representation was last modified? A web server like Apache has it easy: it mostly serves static files from disk, and filesystems already track the modification date for every file. Apache just gets that information from the filesystem. In more complicated scenarios, you’ll need to break the representation down into its component parts and see when each bit of resource state was last modified. In Chapter 7, the Last-Modified value for a list of bookmarks was the most recent timestamp in the list. If you’re not tracking this information, the bandwidth savings you get by supporting Last-Modified might make it worth your while to start tracking it.

Even when a server provides Last-Modified, it’s not totally reliable. Let’s say a client GETs a representation at 12:30:00.3 and sees a Last-Modified with the time “12:30:00.” A tenth of a second later, the representation changes, but the Last-Modified time is still “12:30:00.” If the client tries a conditional GET request using If-Modified-Since, the server will send a 304 (“Not Modified”) response, even though the resource was modified after the original GET. One second is not a high enough resolution to keep track of when a resource changes. In fact, no resolution is high enough to keep track of when a resource changes with total accuracy.

This is not quite satisfactory. The world cries out for a completely reliable way of checking whether or not a representation has been modified since last you retrieved it. Enter the Etag response header. The Etag (it stands for “entity tag”) is a nonsensical string that must change whenever the corresponding representation changes.

The If-None-Match request header is to Etag as the If-Modified-Since request header is to Last-Modified. It’s a way of making an HTTP request conditional. In this case, the condition is “the representation has changed, as embodied in the entity tag.” It’s supposed to be a totally reliable way of identifying changes between representations.

It’s easy to generate a good ETag for any representation. Transformations like the MD5 hash can turn any string of bytes into a short string that’s unique except in pathological cases. The problem is, by the time you can run one of those transformations, you’ve already created the representation as a string of bytes. You may save bandwidth by not sending the representation over the wire, but you’ve already done everything necessary to build it.

The Apache server uses filesystem information like file size and modification time to generate Etag headers for static files without reading their contents. You might be able to do the same thing for your representations: pick the data that tends to change, or summary data that changes along with the representation. Instead of doing an MD5 sum of the entire representation, just do a sum of the important data. The Etag header doesn’t need to incorporate every bit of data in the representation: it just has to change whenever the representation changes.

If a server provides both Last-Modified and Etag, the client can provide both If-Modified-Since and If-None-Match in subsequent requests (as I did in Example 8-4). The server should make both checks: it should only send a new representation if the representation has changed and the Etag is different.

Conditional HTTP GET gives the client a way to refresh a representation by making a GET request that uses very little bandwidth if the representation has not changed. Caching gives the client some rough guidelines that can make it unnecessary to make that second GET request at all.

HTTP caching is a complex topic, even though I’m limiting my discussion to client-side caches and ignoring proxy caches that sit between the client and the server.^[27]The basics are these: when a client makes an HTTP GET or HEAD request, it might be able to cache the HTTP response document, headers and all. The next time the client is asked to make the same GET or HEAD request, it may be able to return the cached document instead of actually making the request again. From the perspective of the user (a human using a web browser, or a computer program using an HTTP library), caching is transparent. The user triggers a request, but instead of making an actual HTTP request, the client retrieves a cached response from the server and presents it as though it were freshly retrieved. I’m going to focus on three topics from the point of view of the service provider: how you can tell the client to cache, how you can tell the client not to cache, and when the client might be caching without you knowing it.

Expires: Tue, 30 Jan 2007 17:02:06 GMT

Cache-Control: max-age=3600

Cache-Control: no-cache

Conditional GET is designed to save the server from sending enormous representations to a client that already has them. Another feature of HTTP, less often used, can save the client from fruitlessly sending enormous (or sensitive) representations to the server. There’s no official name for this kind of request, so I’ve came up with a silly name: look-before-you-leap requests.

To make a LBYL request, a client sends a PUT or POST request normally, but omits the entity-body. Instead, the client sets the Expect request header to the string “100-continue”. Example 8-5 shows a sample LBYL request.

PUT /filestore/myfile.txt HTTP/1.1
Host: example.com
Content-length: 524288000
Expect: 100-continue

This is not a real PUT request: it’s a question about a possible future PUT request. The client is asking the server: “would you allow me to PUT a new representation to the resource at /filestore/myfile.txt?” The server makes its decision based on the current state of that resource, and the HTTP headers provided by the client. In this case the server would examine Content-length and decide whether it’s willing to accept a 500 MB file.

If the answer is yes, the server sends a status code of 100 (“Continue”). Then the client is expected to resend the PUT request, omitting the Expect and including the 500-MB representation in the entity-body. The server has agreed to accept that representation.

If the answer is no, the server sends a status code of 417 (“Expectation Failed”). The answer might be no because the resource at /filestore/myfile.txt is write-protected, because the client didn’t provide the proper authentication credentials, or because 500 MB is just too big. Whatever the reason, the initial look-before-you-leap request has saved the client from sending 500 MB of data only to have that data rejected. Both client and server are better off.

Of course, a client with a bad representation can lie about it in the headers just to get a status code of 100, but it won’t do any good. The server won’t accept a bad representation on the second request, any more than it would have on the first request.

Partial HTTP GET allows a client to fetch only a subset of a representation. It’s usually used to resume interrupted downloads. Most web servers support partial GET for static content; so does Amazon’s S3 service.

Example 8-6 is a bit of code that makes two partial HTTP GET requests to the same URI. The first request gets bytes 10 through 20, and the second request gets everything from byte 40,000 to the end.

#!/usr/bin/ruby
# fetch-oreilly-partial.rb

require 'rubygems'
require 'rest-open-uri'
uri = 'http://www.oreilly.com/'

# Make a partial HTTP request and describe the response.
def partial_request(uri, range)
  begin
    response = open(uri, 'Range' => range)
  rescue OpenURI::HTTPError => e
    response = e.io
  end

  puts " Status code: #{response.status.inspect}"
  puts " Representation size: #{response.size}"
  puts " Content Range: #{response.meta['content-range']}"
  puts " Etag: #{response.meta['etag']}"
end    

puts "First request:"
partial_request(uri, "bytes=10-20")

puts "Second request:"
partial_request(uri, "bytes=40000-")

When I run that code I see this for the first request:

First request:
 Status code: ["206", "Partial Content"]
 Representation size: 11
 Content Range: bytes 10-20/41123
 Etag: "7359b7-a0a3-45b5d90e"

Instead of 40 KB, the server has only sent me the 11 bytes I requested. Similarly for the second request:

Second request:
 Status code: ["206", "Partial Content"]
 Representation size: 1123
 Content Range: bytes 40000-41122/41123
 Etag: "7359b7-a0a3-45b5d90e"

Note that the Etag is the same in both cases. In fact, it’s the same as it was back when I ran the conditional GET code back in Example 8-4. The value of Etag is always a value calculated for the whole document. That way I can combine conditional GET and partial GET.

Partial GET might seem like a way to let the client access subresources of a given resource. It’s not. For one thing, a client can only address part of a representation by giving a byte range. That’s not very useful unless your representation is a binary data structure. More importantly, if you’ve got subresources that someone might want to talk about separately from the containing resource, guess what: you’ve got more resources. A resource is anything that might be the target of a hypertext link. Give those subresources their own URIs.