Grauw’s blog

EVE Online, an MMORPG that I play, will get a new API soon, which they wrote about a few weeks ago on their developers blog.

It has a number of good improvements, such as a move from a session-based authentication system to a stateless authentication based system, and publishing more information than they did before. However, there are also some things that are not so nice, and that could be done better.

CCP claims to go with a REST style approach, which is great because it is surely a better and more simple approach than SOAP or XML-RPC. But even though it falls under a narrow interpretation of REST, if you look at the actual API, it isn’t really a RESTful way to go about it. That is, it looks more like a ‘POST style approach’.

To quickly summarise the basic idea of REST and a RESTful web application in a few words, a REST API uses all the functionality that the HTTP standard provides. Some examples of this are using the GET method for retrieval of data and POST, PUT and DELETE for modifications, and other built-in functions of HTTP are means for authentication, caching, and error reporting. Most RESTful APIs also like to use ‘pretty’ URLs, although this is more of aesthetic value.

Topics:

• Pretty URLs
• Caching
• Authentication
• Content negotiation

Pretty URLs

Part of REST is good URL design. This is actually not really really important, but it’s one of the things that is usually the first things that people notice. Let me just sum up a few considerations:

• URLs point to resources, uniquely identified by one URL (independant of authentication information), which you can GET information from, POST updates to, modify with PUT or remove with DELETE. As the EVE API is read-only, only GET requests are necessary.
• “Good URLs never change”. This is a well-known mantra, which basically says it is good to spend some time on URL design so that you never have to change them again. That way, you can avoid trouble having to change all applications that use the API just because you e.g. start using a different backend and want to change your .aspx extensions to .php.
• If the URLs are pretty, it just looks good on the eyes ^_^. Pretty usually means separating things with slashes and using lower-case. Just look at the examples below. There are also many examples on the web, e.g. Flickr and Last.fm. Bad example is Youtube :).

The current URLs of the EVE API look like this (with POST parameters in parenthesis):

http://myeve.eve-online.com/eve/SkillTree.xml.aspx
http://myeve.eve-online.com/eve/RefTypes.xml.aspx
http://myeve.eve-online.com/map/Sovereignty.xml.aspx
http://myeve.eve-online.com/char/CharacterSheet.xml.aspx (userID, apiKey, characterID)
http://myeve.eve-online.com/char/WalletJournal.xml.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/char/WalletJournal.csv.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/corp/WalletJournal.xml.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/corp/WalletJournal.csv.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/char/WalletTransactions.xml.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/char/WalletTransactions.csv.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/corp/WalletTransactions.xml.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/corp/WalletTransactions.csv.aspx (userID, apiKey, characterID, accountKey, beforeRefID)
http://myeve.eve-online.com/char/AccountBalance.xml.aspx (userID, apiKey, characterID)
http://myeve.eve-online.com/corp/AccountBalance.xml.aspx (userID, apiKey, characterID)

A ‘REST redesign’ of these URLs would look something like this:

http://myeve.eve-online.com/eve/skilltree
http://myeve.eve-online.com/eve/referencetypes
http://myeve.eve-online.com/map/sovereignty
http://myeve.eve-online.com/char/<characterID>/charactersheet
http://myeve.eve-online.com/char/<characterID>/walletjournal
http://myeve.eve-online.com/char/<characterID>/walletjournal/<accountKey>
http://myeve.eve-online.com/char/<characterID>/walletjournal/<accountKey>/<beforeRefID>
http://myeve.eve-online.com/corp/<corpID>/walletjournal
http://myeve.eve-online.com/corp/<corpID>/walletjournal/<accountKey>
http://myeve.eve-online.com/corp/<corpID>/walletjournal/<accountKey>/<beforeRefID>
http://myeve.eve-online.com/char/<characterID>/wallettransactions
http://myeve.eve-online.com/char/<characterID>/wallettransactions/<accountKey>
http://myeve.eve-online.com/char/<characterID>/wallettransactions/<accountKey>/<beforeRefID>
http://myeve.eve-online.com/corp/<corpID>/wallettransactions
http://myeve.eve-online.com/corp/<corpID>/wallettransactions/<accountKey>
http://myeve.eve-online.com/corp/<corpID>/wallettransactions/<accountKey>/<beforeRefID>
http://myeve.eve-online.com/char/<characterID>/accountbalance
http://myeve.eve-online.com/corp/<corpID>/accountbalance

As you can see here, the parameters have been moved into the URI as integral part, and it’s become kind of a selector: “of all the characters, get the one with this ID, and then show me his charactersheet”. For some I made multiple entries; if you would leave the <accountkey> out, you would get the first account (the only account for characters), and if you leave the <beforerefid> out, you just get the first 1000 entries. It’s very intuitive and very ‘hackable’, as-in you can easily change the URL by cutting off parts, and get the results that you expect :).

I also changed the corporation info that took a characterID as the basis for looking up information into a corpID instead. This is because an URL identifies a resource, and given that a character can change corporations, using a character’s ID is not a good basis to identify a corporation. Also, when the current user is not a member of a corp, you may still want to return certain information. One interesting way to find a corp based on a characterID is that a GET to http://myeve.eve-online.com/char/<characterID>/corp could return a 302 Found redirect to the corp’s address.

One URL that is not listed here but which would be a logical continuation of these would be http://myeve.eve-online.com/char/<characterID>, which could for example provide a summary, and link to the pages with detailed information.

However, I must note that a lot of these changes are aesthetic; the EVE API allows the parameters to be added as GET parameters, resulting in a URL like http://myeve.eve-online.com/char/CharacterSheet.xml.aspx?characterID=<characterID>&userID=<userID>&apiKey=<apiKey>. Including the authentication information in this URL is not good (see the section on authentication below), but if you would cut that part off, the remainder is kind an ugly URL but it does the job.

Now let’s get on to the real meat…

Authentication

The EVE API does not use standard HTTP authentication. Instead, it passes the authentication information via POST parameters. It is however better to use the built-in HTTP authentication functionality, primarily because many HTTP applications and stacks have built-in support for it.

For example, authentication information can be directly embedded in URLs, like http://user:pass@example.com/. If you leave out the password, applications will automatically pop up a prompt. There are also many applications that do not allow you to send POST values, like say an RSS reader, if you would ever want to output information in RSS format, but authentication is pretty much always supported.

Other forms of application support that HTTP authentication has and custom authentication has not is proper caching behaviour by caching proxies and local caches. I will explain a little more about that later in the section on caching.

Also, because HTTP stacks and applications have built-in support, it is usually much simpler to use. To illustrate this point, let me show how much easier things get when you use HTTP authentication instead of by passing authentication strings via POST variables, through some examples.

Curl

This is how you have to create both authentications using the unix curl command. HTTP authentication:

curl http://userid:apikey@eve-online.com/eve/SkillTree.xml.aspx

or

curl --user userid:apikey http://eve-online.com/eve/SkillTree.xml.aspx

EVE POST-authentication:

curl http://eve-online.com/eve/SkillTree.xml.aspx --data userid=userid&apikey=apikey

The difference is not too big, but the POST variant involves formatting the userid and apikey using www-form-urlencoding as data. Also, if you leave the password (api key) out, the application will ask the user for a password, something that takes more effort to achieve if you do not use HTTP authentication.

You may think curl is not such an interesting case, but it makes it very easy to cook up a cron job which retrieves a page from eve-online.com periodically, run some ugly regular expression on it, and do whatever the user wants with that.

AJAX XMLHttpRequest

Another example using AJAX XMLHttpRequest. HTTP authentication:

open("GET", "http://eve-online.com/eve/SkillTree.xml.aspx", true, userid, apikey);
send();

EVE POST-authentication:

open("POST", "http://eve-online.com/eve/SkillTree.xml.aspx",);
setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
send("userid="+encodeURIComponent(userid)+"&apikey="+encodeURIComponent(apikey))

Microsoft .NET HttpWebRequest

Another example using the Microsoft .NET HttpWebRequest object. HTTP authentication:

HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create("http://eve-online.com/eve/SkillTree.xml.aspx");
request.Credentials = new NetworkCredential(userid, apikey);
HttpWebResponse response = (HttpWebResponse) request.GetResponse();

EVE POST-authentication:

HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create("http://eve-online.com/eve/SkillTree.xml.aspx");
string postData = "userid="+Uri.EscapeDataString(userid)+"&apikey="+Uri.EscapeDataString(apikey);
UTF8Encoding encoding = new UTF8Encoding();
byte[] byte1 = encoding.GetBytes(postData);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.ContentLength = postData.Length;
Stream stream = request.GetRequestStream();
stream.Write(byte1, 0, byte1.Length);
stream.Close();
HttpWebResponse response = (HttpWebResponse) request.GetResponse();

The HTTP authentication method is not only much much shorter, it also does not require knowledge of www-form-urlencoding, and it can use the built-in CredentialCache to automatically store the authentication information.

The browser

And finally, in your favorite browser (e.g. Firefox), with HTTP authentication you can browse to:

http://eve-online.com/eve/SkillTree.xml.aspx

When you do this, you would get an authentication dialog, and then view the data that it returns. With POST-ed authentication information, this is simply not possible.

Further notes

When using HTTP authentication, there are two methods: Basic and Digest authentication. Basic sends the authentication credentials plain over the wire, Digest uses several methods such as hashes and nonces to protect your credentials against various attacks. For EVE, I would say Basic would suffice, because the api-key system makes the security risk very low. However, note that if CCP at some point would want to switch to from Basic to Digest or vice versa, the HTTP stack will handle that and applications will not have to make any changes.

On a final note, let me point out that although they published the HTTP authentication scheme in a separate specification RFC (no. 2617) than HTTP (no. 2616), it is an integral part of HTTP and not a separate standard.

Caching

By using POST, you effectively disable caching. The EVE API compensates for this by implementing a custom caching-scheme through timeInCache and timeLeftInCache attributes, however that’s a shame because HTTP provides excellent built-in means for caching.

There are several headers for cache control available, e.g. the Last-Modified and Expires headers. In addition to that, an URL that is POSTed to (i.e. modified) will automatically be cleared from the cache, because as it is modified.

All this is also understood by caching proxies. In addition to that, caching proxies also give special treatment to e.g. any request that has authentication headers; these will never be cached by the caching proxies, because it may contain sensitive information, but will still be cached by the user’s local cache according to the caching headers.

Let me give two examples.

First, the skill tree structure. This information is updated very infrequently, and it is not known in advance when it will be updated. The best way to cache this would be by sending a Last-Modified header when it is retrieved, indicating the modification date. Next time that the application tries to retrieve the file, it will add an If-Modified-Since header, so that the server can check if the file has been modified, and if not, send back a short 304 Not Modified response that indicates the cached copy is still good.

Second, the character sheet. This sheet updates almost continuously, however CCP wants these to be cached for six hours after each request to reduce server load. This can be done by using the Expires header, e.g. Expires: Mon, 17 Jun 2007 16:40:23 GMT. This will cause applications to use a cached version until the expiry date is reached.

Now, a big benefit of this is that it really makes the application developer’s life easier, because in many HTTP stacks this caching happens automatically. For example, if you use XMLHttpRequest the browser will use its browser cache to cache your requests. .NET also has caching functionality for its HTTP stack. In other environments, you have to handle the caching yourself because the stack doesn’t implement a cache, but that is no different from today.

Because it’s handled automatically by many stacks, if CCP gets too high a server-load from certain requests, all they have to do is change the caching headers to those requests, and most applications would automatically adapt to the new caching scheme. No change of applications to read attributes and store cached copies would be needed, because that is already done for them by the HTTP stack, and CCP would see an instant reduction in the number of requests.

Content negotiation

Consider these three requests:

GET http://myeve.eve-online.com/char/<characterid>/charactersheet
GET http://myeve.eve-online.com/char/<characterid>/charactersheet.xml
GET http://myeve.eve-online.com/char/<characterid>/charactersheet.csv

In HTTP, the desired format is indicated by sending the Accept header. Browsers send text/html, and other applications can send other Accept headers. Based on that header, the server can send different content.

So, that would mean that if a user would surf to the first URL in their browser, they would get a pretty HTML marked-up version, if an application sends Accept: application/xml then they would get the XML version, and if they send Accept: text/csv, then they would get the CSV version. And if none of the accepted formats, it could either send a 406 Not Acceptable error response, or default to sending e.g. the XML version.

In the real world however, not all applications are nice enough to send a proper Accept header. For example there are unfortunately many RSS readers that do not send Accept: application/rss+xml but text/html instead. In that case, you can fall back to extensions (as shown above in the second and third URL). If the server sees a .rss extension, it finds the MIME type that belongs to the extension and processes the request as if it had been made with a application/rss+xml Accept header.

Fortunately, if you do not really have to consider applications that send an incorrect Accept header, which would be the case for the EVE API, you do not need extension-stuff. Although it’s still nice to have, because it makes it easy to debug with your browser without needing some extension that can change the Accept header for you.

Finally

So, I think that on these points, the EVE API could use some improvement. Some things are more important than others, particularly the HTTP authentication would help make application development easier, because HTTP stacks provide built-in implementations for this, and the same is true for HTTP caching. The pretty URLs and content negotiation are more of like nice-to-have things, but still…

Finally, I’d just like to point to one really great presentation called World of Resources by David Heinemeier Hansson. It’s about REST functionality in Ruby on Rails, but if you ignore the Ruby parts it’s still a really nice talk about REST that touches many interesting points. Here are the slides and the video recording (Quicktime) of the presentation.

I hope that the EVE API will turn into a truly RESTful API in the future.