I am Joannes Vermorel, founder at Lokad. I am also an engineer from the Corps des Mines who initially graduated from the ENS.

I have been passionate about computer science, software matters and data mining for almost two decades. (RSS - ATOM)


Entries in web (25)


A few tips for Web API design

During the iteration in spring 2010 that has lead Lokad to release its Forecasting API v3, I have been thinking a lot about how to design proper Web APIs in this age of cloud computing.

Designing a good Web API is very surprisingly hard. Because Remote Procedure Call has been around forever, one might think that designing API is a well-known established practice, and yet, suffering the defects of Forecasting API v1 and v2, we learned the hard way it wasn't really the case.

In this post, I will try cover some of gotchas that we learned the hard-way about Web API design.

1. Full API stack ownership

When you expose an API on the web, you tend to rely on building-blocks such as XML, SOAP, HTTP, ... You must be prepared to accept and to support a full ownership of this stack: in order words, problems are likely to happen at all levels, and you must be ready to address them even if the problem arises in one of those building blocks.

For example, at Lokad, we initially opted for SOAP, and I now believe it was a mistake from Day 1. There is nothing really wrong with SOAP itself (*), the problem was that the Lokad team (myself included) was not ready to support SOAP to in full. Whenever a client was raising a subtle question some obscure XML namespace or SOAP version matters, we were very far from our comfort area. In short, we were relying on a complex SOAP toolkit which proved to be a leaky abstraction.

When it comes to public API, you have to be ready to support all abstraction leaks from the TCP, HTTP, XML, SOAP... That why, in the end, for our Forecasting API v3, we settled for POX (Plain Old Xml) over HTTP with a basic REST philosophy. Now, when an issue arises, we can trully own the problem along with its solution, instead of being helpless facing a leaky abstraction that we don't have the resource to embrace.

(*) Although there are definitely many gray areas with SOAP (just look at SOAP Exceptions for example).

2. Be minimalistic

Dealing with all the software layers that sit between the client code and the Web API itself is bad enough, if the API itself adds to complexity of its own, the task quickly becomes maddening.

While refactoring our Forecasting API v2 into the Forecasting API v3, I reduced the number of web methods from +20 to 8. Looking back, it was probably one of the best insights I had.

When developing a software library, it is usually convenient for the client developer to benefit from many syntactic sugars that is to say method variants that achieve the purpose following various coding style. For a Web API, the reverse is true: there should be one and only one way to achieve each task.

Redundancy within methods only cause confusion and extra-friction when developing against the API, and while cause an endless stream of questions whether the method X should be favored against the method X* while both achieve essentially the same thing.

3. Idempotence

Network is unreliable. With a low frequency, API calls will fail because of network glitches. The easiest way to deal with those glitches is simply to have retry policies in place: if the call fails, try again. If your API semantic is idempotent, retry policies will integrate seamlessly with your API. If not, each network glitch is likely to wreak havoc within the client app.

For example, coming from a SQL background, most developer might consider having both INSERT (only for new items) and UPDATE (only for existing items) methods. INSERT is a bad choice, because if the method is attempted twice because of a retry, the second call will fail probably triggering some unexpected exception on the client side. Instead you should rather adopt UPSERT, ie UPDATE or INSERT, semantics which play much nicer with retry policies.

4. Explicit input/output limitations

No web request or response should be allowed to be arbitrarily large as it would cause a lot of problem server side to maintain decent performance across concurrent web requests. It means that from Day 1 every message that goes in or out your API is explicitly capped: don't let your users discover your API limitations through reverse engineering. In practice, it means that no array, list or string should be left unbounded: max length should always be specified.

Then, unless you precisely want to move plain text around, I suggest to keep tight limitation on any string being passed on your API. In the Forecasting API, we have opted for the Regex pattern ^[a-zA-Z0-9]{1,32}$ for all string tokens. Obviously, this is rather inflexible, but again, unless your API is intended for plain text / binary data storage there is no point is supporting the whole UTF-8 range which can prove very hard to debug and trigger SQL-injection-like problems.



9 steps to make sure your startup exists

My uISV isn't even remotely an audience based business - we are on a narrow B2B segment - but since the very beginning, I have invested a lot of efforts to get a decent online presence. So far, every effort that I have pushed to strengthen the online presence was very significantly rewarded. Every week or so, excellent news just pop out of nowhere:

  • A consulting group wants to add the product to its portfolio.

  • A customer sends you a detailed spec of what you should be doing instead, and it happens to be really smart suggestions.

  • A large company wants to know if your product scales up to 1 zillion users, because they are considering buying a zillion licenses.

Nearly one year ago, I had the chance to get my own uISV admitted at the Startup Incubator of Telecom ParisTech. An incubator is a nice place to meet other people that are facing roughly the same sort of problems that you have. To my great surprise, most startups have poor online presence, and even more surprising, most investors seem to have no clue about online presence either.

It's not clear how much it hurts the business; but in my opinion your online presence is the only tangible proof of your company existence for all people who do not happens to be within a 20km radius of your office.

Thus, here are my 9 steps to make sure the company has an online presence: 

  1. No stealth-mode crap, get online, no excuse.

  2. Look & feel should be decent.

  3. Customer benefits come first.

  4. Happy talk has no place on your site.

  5. Decent Google PageRank is required.

  6. English is required.

  7. Public pricing is required.

  8. Blog is required.

  9. Community feedback should be possible.

1. No stealth-mode crap, get online, no excuse
People tend to think too much good of their own ideas. Ideas matters little while execution is everything. Remember that Google was half-a-decade late in the search engine race; idem with Facebook for social networking websites. Stealth development is a game for big players who can sustain years of R&D expenses with no visible returns and then inject millions in marketing once the technology is ready.

2. Look & feel should be decent
Unless you happen to be a graphic designer, don't even try to skin your website yourself: it will look awfully amateurish and turn your customers away. For $100 or less you can get a nice website template. It might not be unique, but it does not matter. There are so many templates available anyway, that 99.99% of your visitors won't even notice that aspect. In 2009, there is no more excuse to have a half-backed website skin.

3. Customer benefits come first
If your visitors can't figure out the benefits of your technology / product / service, why should they actually care about the way it's designed? Many startup fails at actually explain the value of what they are offering, and strongly focus on random technical aspects that happened to be a challenge for the development team.

4. Happy talk has no place on your site
Happy talk is an easy way to fill your website. Ever considered putting a Welcome on our website sentence in your front page? Well, don't. Also, for B2B company, happy talk usually happens with (slightly) more subtle verbiage such as mindless mission statements: our mission is to serve our customer's interests. Make sure that every single word that you put on your website carry a valuable message. If it doesn't, delete the word.

5. Decent Google PageRank is required
Ever googled a company name to end up on the Facebook page of an employee? Well, that sort of things happens when your Google PageRank is just too low. More generally, a decent PageRank ensures that if somebody does a deep market research, your company will appears. I am not even talking about grabbing thousands of visitors through top SERP on strategic keywords; I am just considering the journalist / student / consultant / ... who is trying to figure out all the players of your business niche. If this person can't find you, then you don't exist.

6. English is required
If you happen to be a native English speaker, that one isn't going to be too hard for you. For the rest of us, well, we have to make the effort to get it done nonetheless. The harsh reality is that through English, you can reach roughly 10x more people than what you can through any other languages. It's doesn't mean that you can't do other languages, but English should be a primary focus.

7. Public pricing is required
It's always a bit puzzling to me to notice how people are usually reluctant to display any pricing on their website - especially on B2B websites. Yet, pricing is a vital information for your customers. Software or services can be priced from $1 / month to $10 million / month. Where do you stand? This concern stays valid even for beta products. Displaying a price is a very good signal for your customers: it tells them that you are a real company with a real product under way. Without pricing, you're simply not part of the economic circuit.

8. Blog is required
A company can be long dead while the website is still up and running. Providing some news - any news, anywhere on the website - as long the dates are visible, is the most simple way to prove to your visitors that the company is still up and running. Having a blog, and posting at least once a month is probably the easiest to complete this step. Blogs are dirty cheap and dead simple, no excuse will be considered for not having a blog.

9. Community feedback should be possible
I found that it's always very frustrating not being able to provide feedback about a product, a website, a service whatever. Granted, most web visitors are never giving any feedback, but some are doing it all the time. The feedback provided by those users is gold. Don't neglect your community when setting-up web forums is just a matter of hours. Your forums are likely to have a slow profile, but in my experience, the few early feedbacks that you get can actually make a difference in your business. You should not miss that sort of opportunity.

As a final word, I have already started to collect some data about the '07 and '08 classes of the incubator of Telecom ParisTech. Stay tuned.



Embedding maths in a Google Gadget

Google sites is a great product to increase productivity of businesses of any size. At Lokad, one feature that I am really missing from Google Sites is the ability the embed clean mathematical formula in regular web pages.

Actually, a simple and yet very practical way of achieving a decent mathematical display consists of using a JavaScript ASCII Maths to MathML converter. Yet, Google Sites does not support direct 3rd party script inclusion either (which is probably on of the most painful limitation of Google Sites, but I digress).

Thus, I have decided to spend some time toying with the Google Gadgets to figure out if it was possible to circumvent this limitation through widgets. The resulting gadget can be found here. See also the screenshot below.

The syntax is identical the one of AsciiMathML. Enjoy.

Known limitations:

  • Works poorly with Internet Explorer. Since MathML is not natively supported, you need a MathML addon (it's free though).

  • Gadget preferences can't be changed in syndicated environment such as Google Sites. Which means that you have to edit first the page in Google Sites if you want the change made to the gadget to be persistent.

Custom error page in ASP.NET when database connectivity is lost.

A particularly annoying, yet frequent, issue for your ASP.NET is the loss of database connectivity. Indeed, if your database is hosted on a separate machine (as it is generally advised for performance), then your web application is subject to database downtime.

Database downtimes have several particularities

  • It generates internal server errors.

  • It's not the type of error that can be fixed by be the developer.

  • The problem tends to get solved by itself (think: reboot of the database server)

  • Errors don't get logged (well, assuming that you are logging errors in the database).

Thus, for my own ASP.NET application, I want to display an error page that invites people to try again at a later time whenever a database downtime occurs. In comparison, if a "real" error is encountered, the error gets logged and the customer is invited to contact the support (although, support is also monitoring server side error logs on its own).

Although, ASP.NET makes it very easy to add a generic error page for all internal errors through the <customErrors/> section in the web.config, it's not that simple to have a dedicated page that is selectively displayed for database connectivity issue. Thus, I have decided to come up with my own HttpModule that catches database connectivity error and performs a custom redirect.

using System;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Text;
using System.Web;

namespace Lokad.Web.Modules
public class DatabaseConnectionErrorModule : IHttpModule
public void Init(HttpApplication context)
context.Error += new EventHandler(OnError);

public void Dispose() { }

protected virtual void OnError(object sender, EventArgs args)
HttpApplication application = (HttpApplication) sender;

// The SQL exception might have been wrapped into other exceptions.
Exception exception = application.Server.GetLastError();
while (exception != null && exception as SqlException == null)
exception = exception.InnerException;

if (exception as SqlException != null)
// HACK: no SqlConnection.TryOpen() method.
// Relying on error numbers seems risky because there are
// different numbers that can reflect a connectivity problem.
using (SqlConnection connection = new SqlConnection("foobar"))
catch (SqlException)
application.Response.Redirect("~/MyError.aspx", true);

Finally, add a <httpModules/> section to your web.config, and you're done.

Ps: I have been suggested to use the Global.asax hook. I have discarded this approach, because no matter how I was looking at the problem, Global.asax just looks legacy to me (zero modularity, zero testability, etc...).


Hard times ahead for shopping cart providers

Since my μIVS is providing sales forecasting services for eCommerce, I have been spending a considerable amount of time reviewing the market of eCommerce frameworks and providers. I did come up with a few conclusions that may be of interest for people considering developing business in this area.

The first shocking element when I did start to review the online shopping cart market is the insane amount of competing software solutions. Based on my own market survey, I roughly estimate that the market includes roughly 1000 companies actually trying to provide shopping cart solutions of some sort. More over, in 2007, there was almost one major entrant per month in this market. Microsoft is, most probably, planning to make its own entrance through Microsoft Office Live in 2008.

One can argue that the eCommerce market needs a large amount of specialized solutions to fit all the market niches. But I think the reality is quite the opposite: the best shopping carts are the ones that stick to the mainstream design. As a result, most shopping carts provide nearly identical features : catalog management, check-out process, payment provider integration, ... I can't think of any other software area with so many competitors providing nearly identical products.

I see a few reasons that could probably explain such a situation

  • designing some (naive) e-commerce solution is easy, and can be done in 3 months by an experienced programmer. In addition, there are plenty of open source software packages to get you inspired.

  • getting your first ten customers is relatively easy. Just prospect your neighborhood, and you will probably find a few retailers that would accept to get an online front-end for their existing business.

  • e-commerce is hype and all major media are promising a huge business growth for online transactions in the next few years.

Yet, all those positive elements seem seriously flawed to me. Although, a minimal shopping cart can be designed in 3 months, a practical one needs to support virtually all payment providers and all shipping providers (and probably many accounting, ERP solutions as well, if you wish to catch successful e-commerce owners) . And then, it's not any more a 3 months project, but requires some major development efforts.

Then, as getting your first few customers might be (relatively) easy, because you can leverage your immediate neighborhoods, this approach does not scale at all. Considering the decreasing costs of the e-commerce hosting, I don't think that any e-commerce provider business will be sustainable within a few years with less than a couple of thousands of customers. In order to scale-up on such a market, you need a huge online presence, that will drive huge amount of customers to your website. But considering that most e-commerce provider websites already have a Google PageRank of 7 or above (osCommerce has 8 ), competition is clearly super-tough in this area.

Finally, although the e-commerce market is promised to grow, I suspect that most of the growth of the retail activity is going to be absorbed by a few hundred companies. Beside those leading online retailers, there will probably be some room for a few thousands online retailers operating in niches. Even if we assume that the web can sustain 100.000 profitable web shops worldwide (which looks already quite an optimistic estimate to me), it clearly won't sustain the 1.000 shopping cart providers that currently exists. Thus, I would expect 90% of those companies to either disappear or merge in the next decade. Since it does not cost much to maintain an online business, the process can be quite slow though.

Finally, if the shopping cart software itself may have been an issue in the past to create an online store, it is not anymore (unless you have to deal with millions of customers, because most e-commerce solutions don't scale, but very few online businesses end up with such issues). And the amount of money that needs to be invested in the e-commerce software is now ridiculously low compared to the other - non software related - areas such as creating textual content and marketing.