How we ended up writing our own programming language

About one year ago, my company had the opportunity to expand into an area which was very new for us at the time : pricing optimization for commerce. Pricing optimization is quite different to demand forecasting; the latter being the original focus of Lokad at the beginning of the company’s existence. While demand forecasting fits rather nicely into quantitative frameworks that allow you to decide which forecasting methods are the most suitable for any given task pricing is a much more evasive problem. The idea that profits can be maximized by carrying out a simple analysis of demand elasticity is deceptive. Indeed, pricing is a signal sent to the market; and like with any marketing ingredient, there is not one valid answer to the problem.

One year ago, most of the companies that helped merchants manage their pricing were consulting companies, but I wanted to build a pricing webapp to help businesses deal with their pricing which would go beyond the classic consulting services on pricing. I quickly ruled out the idea of offering a template list of “pricing recipes”. Some competitors were already offering such “pricing recipe” services, and they were desperately inflexible. Merchants needed to be able to “tweak” their pricing logic in many ways. Thus, I started to consider more elaborate user interfaces that would allow merchants to compose their own pricing strategies. After pushing some efforts at mockups, I was ending up with something oddly similar to Microsoft Access “visual” query designer.

This was not a good thing. My limited interactions with this query designer, a decade prior, had left me with the lasting impression of this being just about the worst user experience I have ever had with the “normal” behavior of a product released by Microsoft. While it was supposedly a visual query editor with plenty of very visual buttons, but unless you had some knowledge of SQL or experience in programming, you weren’t going very far with this tool. In the end, anyone using Access was falling back on the non-visual query editor, which quite unfortunately, was a second-class citizen.

Gradually, I came to consider the possibility of going for a programming language instead. With a programming language, we could provide the desirable expressiveness, but also a powerful data environment. Lokad would be hosting the data along with offering a cloud-based execution environment for the pricing logic. This environment would not be constrained by the client’s desktop setup which can be too old, too weak, too messy or downright corrupted.

At first, I considered reusing an existing programming language such as JavaScript or Python. However, this presented two very specific challenges. The first challenge was security. Running server-side client code seemed like a giant vector for entire classes of injection attacks. In theory, it should be possible to sandbox the execution of any program, but my instincts were telling me that the surface attack area was so great we would never be confident enough about not having leaks in our sandbox. So, we would have to leverage disposable VMs for every execution, and it seemed that an endless stream of technical problems was heading our way if we were to implement this.

The second, and in fact bigger, problem is that JavaScript or Python being full-fledged programming languages are also complex languages, which include truckloads of features downright irrelevant for the pricing use cases that I was considering: loops, objects, exceptions, nil references. No matter how much we would try to steer the usage of our future product away from these elements, I felt that they would invariably resurface again and again because some our future users would be familiar with just these languages, and as a result, they would do things the way they were used to doing them before. It is tough to debug a generic programming source code, so the tooling would necessarily end up being complex as well.

This left me with the prospect of inventing a new programming language, and yet this idea was accompanied by all the red flags possible in my mind. Usually, for a software company, inventing its own programming language is a terrible idea. Actually, I had witnessed quite closely three companies who had rolled out their own respective programming languages, and for each one of these companies, the experience was very poor to say the least . Two of them managed to achieve a certain level of success nonetheless, but the ad hoc nature of the language had been a huge hindrance to the process. Moreover, about every single experience I ever had with niche programming languages confirmed that an ad hoc language was a terrible idea.

However, as far pricing and commerce were concerned, a generic programming language was not required, and we were hopeful that through extreme specialization, we could produce a highly specialized language that would, all being well, compare favorably with the mainstream languages, at least for the narrow scope of commerce.

And thus, the Envision programming language was born at Lokad.

Unlike a generic programming language, Envision is designed around a relatively rigid and relatively simple data model that we knew to work well for commerce. My intent was to be able to reproduce all the domain-specific calculations that nearly all merchants were doing in Microsoft Excel, but putting some distance between the logic and the data - but not too much distance either. Indeed, Envision, as the name suggests, is also heavily geared toward data visualization; again, not generic data visualization, but things that matter for commerce.

Envision has no loops, no branches, no nulls, no exceptions, no objects … and it does just fine without them. It is not Turing-complete either, so we do not end up with indefinite execution delays.

Less than one year after starting to write the first line of code for the Envision compiler, we have now secured over a million Euros through multi-year contracts that are to be (almost) completely implemented in Envision. To be transparent, it is not the language that clients are buying, but the service to be built with it. Nevertheless, over the last couple of months, we have been able to deliver all kinds of quantitative optimizations with Envision - not just pricing actually – and within timeframes that we would never had achieved in the past.

There are a two takeaways lessons to be learned from this initiative. First, ultra-specialized languages are still a valid option for vertical niches. While it is a very rough estimate, I would say that with Envision, when dealing with a suitable challenge, we end-up with about 50 times fewer lines of code than when the same logic is implemented in C#, the primary programming language used at Lokad. Yes, using a functional language like F# would already make the code more compact than C#, but it would still be far from being that compact. Also, with Envision, we get more concise code not because of we leverage highly abstract operators, but merely because the language itself is geared towards the exact problem to be addressed.

Second, when introducing a programming language, it should not be half-baked. Not only the language itself needs the good ingredients known to computer science – a context-free grammar written in Backus-Naur form for example; but a good integrated programming environment with code auto-completion and meaningful error messages is also needed. The language is only the first ingredient; it is the tooling around the language that makes the difference in terms of productivity. From the very beginning, we invested in the environment as much as did we invest in the programming language.

Also, having a webapp as the primary execution environment for your language opens up a lot of new possibilities. For example, unless you spend years polishing your syntax before releasing anything, you are bound to make design mistakes. We certainly did. Then, as all the Envision scripts were also hosted by Lokad, it was possible for us to rectify those mistakes, first by fixing the language, and second, by upgrading all the impacted scripts from the entire user base. Sure, it takes time, but better to spend a few days on this early on, as opposed to end up with broken language forever.

I have not delved much into the details of the Envision language itself in this post, but in case you would be interested, I have just published a book about it. The preview gives you access to the entire book too.

Ps: while I credit myself for initiating the Envision project at Lokad, it is actually a colleague of mine, Victor Nicollet, presently the CTO of Lokad, who came up with nearly all the good ideas for the design of this language and who carried about 90% of the implementation effort.