A few tips for Web API design

Dec 22, 2010
software engineering

During the iteration in spring 2010 that has lead Lokad to release its Forecasting API v3, I have been thinking a lot about how to design proper Web APIs in this age of cloud computing.

Designing a good Web API is very surprisingly hard. Because Remote Procedure Call has been around forever, one might think that designing API is a well-known established practice, and yet, suffering the defects of Forecasting API v1 and v2, we learned the hard way it wasn’t really the case.

In this post, I will try cover some of gotchas that we learned the hard-way about Web API design.

1. Full API stack ownership

When you expose an API on the web, you tend to rely on building-blocks such as XML, SOAP, HTTP, … You must be prepared to accept and to support a full ownership of this stack: in order words, problems are likely to happen at all levels, and you must be ready to address them even if the problem arises in one of those building blocks.

For example, at Lokad, we initially opted for SOAP, and I now believe it was a mistake from Day 1. There is nothing really wrong with SOAP itself (*), the problem was that the Lokad team (myself included) was not ready to support SOAP to in full. Whenever a client was raising a subtle question some obscure XML namespace or SOAP version matters, we were very far from our comfort area. In short, we were relying on a complex SOAP toolkit which proved to be a leaky abstraction.

When it comes to public API, you have to be ready to support all abstraction leaks from the TCP, HTTP, XML, SOAP… That why, in the end, for our Forecasting API v3, we settled for POX (Plain Old Xml) over HTTP with a basic REST philosophy. Now, when an issue arises, we can trully own the problem along with its solution, instead of being helpless facing a leaky abstraction that we don’t have the resource to embrace.

(*) Although there are definitely many gray areas with SOAP (just look at SOAP Exceptions for example).

2. Be minimalistic

Dealing with all the software layers that sit between the client code and the Web API itself is bad enough, if the API itself adds to complexity of its own, the task quickly becomes maddening.

While refactoring our Forecasting API v2 into the Forecasting API v3, I reduced the number of web methods from +20 to 8. Looking back, it was probably one of the best insights I had.

When developing a software library, it is usually convenient for the client developer to benefit from many syntactic sugars that is to say method variants that achieve the purpose following various coding style. For a Web API, the reverse is true: there should be one and only one way to achieve each task.

Redundancy within methods only cause confusion and extra-friction when developing against the API, and while cause an endless stream of questions whether the method X should be favored against the method X* while both achieve essentially the same thing.

3. Idempotence

Network is unreliable. With a low frequency, API calls will fail because of network glitches. The easiest way to deal with those glitches is simply to have retry policies in place: if the call fails, try again. If your API semantic is idempotent, retry policies will integrate seamlessly with your API. If not, each network glitch is likely to wreak havoc within the client app.

For example, coming from a SQL background, most developer might consider having both INSERT (only for new items) and UPDATE (only for existing items) methods. INSERT is a bad choice, because if the method is attempted twice because of a retry, the second call will fail probably triggering some unexpected exception on the client side. Instead you should rather adopt UPSERT, ie UPDATE or INSERT, semantics which play much nicer with retry policies.

4. Explicit input/output limitations

No web request or response should be allowed to be arbitrarily large as it would cause a lot of problem server side to maintain decent performance across concurrent web requests. It means that from Day 1 every message that goes in or out your API is explicitly capped: don’t let your users discover your API limitations through reverse engineering. In practice, it means that no array, list or string should be left unbounded: max length should always be specified.

Then, unless you precisely want to move plain text around, I suggest to keep tight limitation on any string being passed on your API. In the Forecasting API, we have opted for the Regex pattern ^[a-zA-Z0-9]{1,32}$ for all string tokens. Obviously, this is rather inflexible, but again, unless your API is intended for plain text / binary data storage there is no point is supporting the whole UTF-8 range which can prove very hard to debug and trigger SQL-injection-like problems.

Reader Comments (5)

You can always add, but you can never remove Good read December 22, 2010 | Björn Raupach

UPSERT? More commonly known as SET. December 23, 2010 | Matthew Holloway

Nice post. I like #3. I currently have insert and update separated, maybe in v2 of my current project I could combine those. I also agree about the simplicity in #1. I use rest whenever it might fit because all the verbs are already known and documented, error codes are already known and documented, its almost to the point where a client can just make calls and look at the data they get back ;) I try to just have client get/post/etc json objects, it keeps things simple. December 23, 2010 | ryan day

you might want to double-check that regex to make sure it’s anchored at the start of the string as well, or it will just let anything that ends with alphanumerics through. like ^[a-zA-Z0-9]{1,32}$. December 23, 2010 | Chris