Super-fast flat file parsing in C# and Java with a perfect hash function

At Lokad, (almost) all we do is to crunch flat text files. It’s not that we haven’t tried anything else - we did - many times - and it went poorly. Flat files are ubiquitous, well understood, and they yield very good performance both of the write side and the read side when working under tight budgets. Keep in mind that the files we crunch are frequently generated by our clients, so while ProtoBuf or Cap’n Proto are very cool, asking our clients to deliver such formats would be roughly equivalent asking them to reimplement their in-house Java ERP in Haskell.

A few lessons about pricing B2B apps

My own SaaS company has always been struggling with its own pricing. For a company now selling its own pricing optimization technology for commerce, this was a bit ironic. Well, pricing of software is unfortunately very unlike pricing goods in store, and the experience we acquired working with our retail clients improving their own prices provided little insights about the pricing of Lokad. Since the creation of the company, Lokad has been offering a metered pricing, charging according to the amount of forecasts consumed.

How we ended up writing our own programming language

About one year ago, my company had the opportunity to expand into an area which was very new for us at the time : pricing optimization for commerce. Pricing optimization is quite different to demand forecasting; the latter being the original focus of Lokad at the beginning of the company’s existence. While demand forecasting fits rather nicely into quantitative frameworks that allow you to decide which forecasting methods are the most suitable for any given task pricing is a much more evasive problem.

Dismal IT usually starts with recruitment agencies

In most companies, especially non-tech companies, the IT landscape is dismal: the old ERP (vastly unmaintainable) coexists the new ERP (mostly unmaintainable). Sales figures from the ERP diverges from the ones of the Business Intelligence which also diverge from the ones of the CRM. Systems are slow and unreliable. Major evolutions take years, etc. Technical debt has not been paid for years, and interests are usurious. Root causes of dismal IT are many.

Bitcoin, more thoughts on an emerging currency

Two years ago, I was publishing some first thoughts on Bitcoin. Meantime, Bitcoin has grown tremendously, and I remain an enthusiast observer of those developments. I had originally proposed a vision in 5 stages for the development of Bitcoin with Mining stage Trading stage End-user stage Merchant stage Enterprise stage Back in 2011, I had written that mining was taken care of. Well, since that time, Bitcoin has witnessed an explosion of the hashing power through the development of ASICs, that is, hardware dedicated to the sole purpose of mining Bitcoins.

Thinking Big Data for commerce

As of October 2013, Google Trends indicates that the buzz around Big Data is still growing. Based on my observations of many services company (mostly retailers though), I believe that it’s not all hype, and that indeed Big Data is going to deeply transform those businesses. However, I also believe that Big Data is wildly misunderstood by most, and that most Big Data vendors should not be trusted. Search volume on the term “Big Data” as reported by Google Trends

A buyer’s guide for enterprise software

Through a series of Big Data consulting missions that were overlapping with the entire IT landscape, data being all over the place, I have observed software purchasing processes of many large companies. Being also an enterprise software vendor myself, I have been baffled countless times by broken buying processes that lead smart people routinely choose about the worst price-quality ratio that the market has to offer. In this post, I am trying to gather a survival kit for buying enterprise software.

8 tips to turn your Big Data into Small Data

Hectic times. Looking at the last entry, I realize it has been half a year already since my last post. The Big Data projects I do, and the more I realize how usually scalability aspects for business projects are irrelevant to the point that the quasi-totality of the valuable data crunching processes could actually be run on a smartphone if the proper approaches are taken. Obviously there is no point in actually doing the analysis on a smartphone, this merely illustrating that really it does not take much computational power.

Big Data: choosing the problem before choosing the solution

My company has started several important big data missions, and I am taking here the opportunity publish some insights are are relevant to all those initiatives. A major (and frequent) pitfall of the Big Data projects consists of starting with a solution instead of starting with a problem. In particular, software vendors (Lokad’s included) are pushing their own Big Data recipe which will randomly involve: Hadoop SAP HANA HBase Amazon EC2 Cassandra Windows Azure Storm Node.

A few tips for Big Data projects

At Lokad, we are routinely working on Big Data projects, primarily for retail, but with occasional missions in energy or biotech companies. Big Data is probably going to remain as one of the big buzzword of 2012, along with a big trail of failed projects. A while ago, I was offering tips for Web API design, today, let’s cover some Big Data lessons (learned the hard way, as always).