Custom error page in ASP.NET when database connectivity is lost.

March 21st, 2008

A particularly annoying, yet frequent, issue for your ASP.NET is the loss of database connectivity. Indeed, if your database is hosted on a separate machine (as it is generally advised for performance), then your web application is subject to database downtime.

Database downtimes have several particularities

  • It generates internal server errors.
  • It’s not the type of error that can be fixed by be the developer.
  • The problem tends to get solved by itself (think: reboot of the database server)
  • Errors don’t get logged (well, assuming that you are logging errors in the database).

Thus, for my own ASP.NET application, I want to display an error page that invites people to try again at a later time whenever a database downtime occurs. In comparison, if a “real” error is encountered, the error gets logged and the customer is invited to contact the support (although, support is also monitoring server side error logs on its own).

Although, ASP.NET makes it very easy to add a generic error page for all internal errors through the <customErrors/> section in the web.config, it’s not that simple to have a dedicated page that is selectively displayed for database connectivity issue. Thus, I have decided to come up with my own HttpModule that catches database connectivity error and performs a custom redirect.

using System;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Text;
using System.Web;

namespace Lokad.Web.Modules
{
    public class DatabaseConnectionErrorModule : IHttpModule
    {
        public void Init(HttpApplication context)
        {
            context.Error += new EventHandler(OnError);
        }

        public void Dispose() { }

        protected virtual void OnError(object sender, EventArgs args)
        {
            HttpApplication application = (HttpApplication) sender;

            // The SQL exception might have been wrapped into other exceptions.
            Exception exception = application.Server.GetLastError();
            while (exception != null && exception as SqlException == null)
            {
                exception = exception.InnerException;
            }

            if (exception as SqlException != null)
            {
                try
                {
                    // HACK: no SqlConnection.TryOpen() method.
                    // Relying on error numbers seems risky because there are
                    // different numbers that can reflect a connectivity problem.
                    using (SqlConnection connection = new SqlConnection("foobar"))
                    {
                        connection.Open();
                    }
                }
                catch (SqlException)
                {
                    application.Response.Redirect("~/MyError.aspx", true);
                }
            }
        }
    }

Finally, add a <httpModules/> section to your web.config, and you’re done.

Ps: I have been suggested to use the Global.asax hook. I have discarded this approach, because no matter how I was looking at the problem, Global.asax just looks legacy to me (zero modularity, zero testability, etc…).

My startup at the Incubator of Telecom Paris

March 4th, 2008

For over a year, I have been working on Lokad.com. The project has been growing nicely, and last week, Lokad has been accepted at the Incubator of Telecom Paris. The incubator of Telecom Paris is the largest incubator in France with some nice success stories (such as Netvibes).

Thus, for the next 18 months, Lokad will have nice offices in Paris (in the 14th arrondissement).

For a young company, an incubator is probably the nicest way to smooth all the mundane details (yet critical) that become unavoidable as soon your company grows beyond the stage of the 1-man company. Details include getting an office, a phone, a network connection, a lawyer, a copy machine, a coffee machine, an accountant, a meeting room, …

Since we have a couple of investors too, Lokad is now hiring. In particular, I am looking for top notch developers. Do not hesitate to apply or to forward the link.

Hard times ahead for shopping cart providers

December 22nd, 2007

Since my μIVS is providing sales forecasting services for eCommerce, I have been spending a considerable amount of time reviewing the market of eCommerce frameworks and providers. I did come up with a few conclusions that may be of interest for people considering developing business in this area.

The first shocking element when I did start to review the online shopping cart market is the insane amount of competing software solutions. Based on my own market survey, I roughly estimate that the market includes roughly 1000 companies actually trying to provide shopping cart solutions of some sort. More over, in 2007, there was almost one major entrant per month in this market. Microsoft is, most probably, planning to make its own entrance through Microsoft Office Live in 2008.

One can argue that the eCommerce market needs a large amount of specialized solutions to fit all the market niches. But I think the reality is quite the opposite: the best shopping carts are the ones that stick to the mainstream design. As a result, most shopping carts provide nearly identical features : catalog management, check-out process, payment provider integration, … I can’t think of any other software area with so many competitors providing nearly identical products.

I see a few reasons that could probably explain such a situation

  • designing some (naive) e-commerce solution is easy, and can be done in 3 months by an experienced programmer. In addition, there are plenty of open source software packages to get you inspired.
  • getting your first ten customers is relatively easy. Just prospect your neighborhood, and you will probably find a few retailers that would accept to get an online front-end for their existing business.
  • e-commerce is hype and all major media are promising a huge business growth for online transactions in the next few years.

Yet, all those positive elements seem seriously flawed to me. Although, a minimal shopping cart can be designed in 3 months, a practical one needs to support virtually all payment providers and all shipping providers (and probably many accounting, ERP solutions as well, if you wish to catch successful e-commerce owners) . And then, it’s not any more a 3 months project, but requires some major development efforts.

Then, as getting your first few customers might be (relatively) easy, because you can leverage your immediate neighborhoods, this approach does not scale at all. Considering the decreasing costs of the e-commerce hosting, I don’t think that any e-commerce provider business will be sustainable within a few years with less than a couple of thousands of customers. In order to scale-up on such a market, you need a huge online presence, that will drive huge amount of customers to your website. But considering that most e-commerce provider websites already have a Google PageRank of 7 or above (osCommerce has 8 ), competition is clearly super-tough in this area.

Finally, although the e-commerce market is promised to grow, I suspect that most of the growth of the retail activity is going to be absorbed by a few hundred companies. Beside those leading online retailers, there will probably be some room for a few thousands online retailers operating in niches. Even if we assume that the web can sustain 100.000 profitable web shops worldwide (which looks already quite an optimistic estimate to me), it clearly won’t sustain the 1.000 shopping cart providers that currently exists. Thus, I would expect 90% of those companies to either disappear or merge in the next decade. Since it does not cost much to maintain an online business, the process can be quite slow though.

Finally, if the shopping cart software itself may have been an issue in the past to create an online store, it is not anymore (unless you have to deal with millions of customers, because most e-commerce solutions don’t scale, but very few online businesses end up with such issues). And the amount of money that needs to be invested in the e-commerce software is now ridiculously low compared to the other - non software related - areas such as creating textual content and marketing.

Delete-proof data paging

November 19th, 2007

In order to retrieve a large amount of data from a SQL table, you need to resort to a data paging scheme. Conceptually, a typical paged SQL query looks like (the syntax is approximate and vaguely inspired from MS SQL Server 2005)


SELECT Foo.Bar
FROM Foo
WHERE RowNumber() BETWEEN @Index AND @Index + @PageSize
ORDER BY Foo.Id;

The queries are made iteratively until no rows get returned any more. Yet, this approach fails both if rows are added or deleted in the table Foo during the iteration.

If rows are inserted, then the RowNumbers() will be impacted. Some rows will see their number to be incremented.

  • The newly inserted rows maybe missed.
  • Certain rows are going to be retrieved twice.

In the overall, the situation isn’t bad, because, after all, all rows that were present when the retrieval iteration started will be retrieved.

In the other hand, if rows are deleted, then some rows will get their RowNumbers() decremented. As a consequence, certain rows will never get retrieved. This issue is quite troublesome, because you would not expect deleted rows to prevent the retrieval of other (valid) rows.

One workaround is to this situation would be to add some IsDeleted flag to the table (instead of actually deleting rows, flags would be changed from false to true); and to purge the database only once a while. Yet, this solution has many drawbacks: table schema must be changed and all existing queries that target this table must add the extra condition IsDeleted = false.

A more subtle approach to the problem consists in changing the definition of the iterating index. Instead of using an index that correspond to a generate RowNumbers(), let directly use @IndexId, the greatest identifier retrieved so far. Thus, the paged query becomes


SELECT TOP @PageSize
FROM Foo
WHERE Foo.Id > @IndexId
ORDER BY Foo.Id

With this new approach, we are now certain that no rows will be skipped during the retrieval process. Plus, we haven’t made any change to the table schema.

Tracking file downloads in Google Analytics AND Google Adwords

November 16th, 2007

Google has not one web analytics system, but two of them, namely Google Analytics and Google Adwords. For the average webmaster, this situation is quite a pain, because most of the tracking code must be duplicated. To make the situation worse, Google Adwords does not support any straightforward solution to track file downloads.

For the sake of my own μISV, I have designed the following script that enables both Analytics and Adwords tracking in a single function.

// Tracking.js - Joannes Vermorel, Lokad.com, 2007
function dhtmlLoadScript(url)
{
   var e = document.createElement("script");
   e.src = url;
   e.type="text/javascript";
   document.getElementsByTagName("head")[0].appendChild(e);
}

function lokadTracker(path)
{
   urchinTracker(path);

   google_conversion_id = 10683XXXX; // use your own ID here
   google_conversion_language = "en_US";
   google_conversion_format = "1";
   google_conversion_color = "FFFFFF";
   if (1) {
       google_conversion_value = 1;
   }
   google_conversion_label = "pageview";
   dhtmlLoadScript('http://www.googleadservices.com/pagead/conversion.js');
}

In order to track a file download just embed the script here above, and intercept the onclick event of your download link

<a onclick="lokadTracker('/Foo.zip')" href="Foo.zip" > Foo <a>

Crypt your config files with PowerShell

October 31st, 2007

ASP.Net 2.0 comes with a convenient native support for configuration file encryption. Yet, things are still not that easy for WinForms, Console applications or Windows Services since the aspnet_regiis.exe utility only supports Web Configuration files.

My own μISV has its share of distributed applications which involve securing a few connection strings over several machines. Securing the connection strings through encryption is not an ultimate defense (if the attacker gains executions rights on the local machine, connection strings will get disclosed anyway), but it can still save you a lot of trouble such as involuntary disclosure.

Download crypt-config.zip

I have found a practical way to solve the issue through PowerShell (see the PowerShell team blog for regular tips), namely two functions crypt-config and decrypt-config. The source code comes as single PSH script contains the function definitions.

To get started, extract the PS1 file from the Zip archive, then


PS docs:\> . ($directory + "\crypt-config.ps1") ;

PS docs:\>crypt-config ‘MyConsole.exe’ ’section’;

PS docs:\>decrypt-config ‘MyConsole.exe’ ’section’;

Typically, section will be replaced by connectionStrings. Note that you do not need to add the .config at the end of the configuration file path.

Velib’s from a software engineer viewpoint

October 21st, 2007

The Velib’s are becoming insanely popular in Paris because of the strikes (strikes in public transportations is a national sport in France, a bit like baseball is in the US). Thus, I have been taking my first Velib ride yesterday, a few months after their initial launch.

Velib picture

The Velib both the name of a public bike renting system in Paris but also the name of the bike itself. There are now 10.000 Velib’s in Paris (the figure will increase up to 20.000 at the beginning of 2008). The key idea is that take a Velib from any Velib station and put it back into any other Velib station (it does not have to be the same station).

Velib’s are a bit bulky (17kg), but in overall they are quite nicely designed.

In my opinion, there are two main weaknesses in the current Velib’s system

  • the Velib traffic regulation
  • the software interface of the Velib renting system

The idea of taking/letting the Velib wherever you want is quite nice. Yet, in practice, there are very important daily migrations of Velib’s within Paris. Basically, in the morning you observe that all the Velib are taken (by the people) toward the inner center of Paris. Then, at the end of the day, there is the opposite flux, and the Velib’s get massively migrated to the outer part of Paris again.

For the average user, strong migrations means that that you are having hard-time to actually find a free Velib in your starting area; but also that you are having hard-time again to find a free slot to park your Velib in your arrival. In order to overcome such a situation, the deal with JCDecaux (the company in charge of the Velib system) include some Velib traffic regulation to organize counter-migration of the Velib’s (through special trucks). Yet, I suspect that the initial deal was massively under-estimating the strength in the migrations in Paris.

At this point, I can hope for two things: Paris re-negotiates with JCDecaux another agreement to increase the Velib traffic regulation; and/or JCDecaux upgrades its traffic regular software to anticipate the migration and respond more pro-actively to them.

Also, the software interface to rent your Velib is a pain. The first mistake comes from the fact that there is not one but two display devices: a big color digital screen that displays the main interface and below, a small alphanumeric display that displays some informations related to the credit card processing. Together those two display devices are a real pain, because you are never sure were to look at while waiting for the next instruction.

Then, the total number of keys that have to pressed iteratively on the numeric pad to do a rent-for-the-day operation is completely insane. I have quickly lost the track, but it must be around 50 key operations or so; which takes 10 mins no matter how much familiarity you have with the system (I was yesterday assisted by somehow who did dictate to me the instructions in order to speed-up the process).

Among the things that are plain nuts with the current UI, I think the password management is a design truly born in Hell. You have to choose a password, then confirm your password, then re-enter again your password. Now, you are asked to enter your credit card password; don’t mix the two of them or your going to block your credit card (and get sent back to the starting point). Actually, the whole password thing is completely useless. The credit card should be the default way to perform authentication for those who do not have an RFID pass (the RFID pass comes with the 1-year subscription). That would save half of the operations.

Not sure that the Velib UI would have succeeded against any hallway testing; yet, during the strikes you got the perfect excuse to be late anyway. You can perfectly afford some 20min struggle to rent your Velib.

Securing CruiseControl.Net integration server

August 8th, 2007

CruiseControl.NET is a great open source tool for continuous integration (CI). Yet, the default settings are quite permissive, and unless you’re working on an open source project as well, you might prefer restrict the accesses to your sole team. I have found that securing CruiseControl.Net while keeping a developer-friendly environment is not such an easy task. This post is a summary of the various steps needed to secure your CI server. It should work against CCNET 1.2 and 1.3.

Create a dedicated Windows User for CCNET

There is (probably) no reason for your integration process to run as a administrator on your CI server. Running the CI as an administrator is just asking for more trouble if something goes wrong in the build process. First create a dedicated Windows account, I suggest to name it integration for the sake of simplicity. Then, From Start » Administrative Tools » Services, you can change the properties of the running services named CruiseControl.Net server, in the Log On tab. Just define the newly created account to be used for the CCNET service. You will also probably need to grant some Windows directory permission on root integration directory.

Restrict CCNET remoting access to localhost

Unless you’re having a farm of build servers with a webserver dedicated to reporting the various build statuses, the CCNET remoting endpoint should not be remotely accessible (yet, that’s the default CCNET settings). This behavior can be adjusted by changing the ccnet.exe.config file. Replace the line <channel ref="tcp" port="21234" > with <channel ref="tcp" port="21234" rejectRemoteRequests="true" >. Now, only a local CCNET dashboard instance is able to connect to the CCNET remoting endpoint.

Restrict CCNET Dashboard access to logged users

By default, no access restrictions are put on the CCNET Dashboard. The most simple way of restricting the access to the dashboard panel is to add a windows authentication layer within the ASP.NET application. You can add the following lines to the webdashboard\web.config configuration file to do that:

<authentication mode="Windows" />
<authorization>
  <deny users="?" />
  <allow users="*" />
</authorization>

Re-opening the CCTray status

CCTray does not support any kind of authentication, thus both Remoting and Via Web Dashboard connection methods will fail now that we have purposely put access restrictions. The trick consists in changing again the webdashboard\web.config to allow anonymous access to XmlServerReport.aspx with

<location path="XmlServerReport.aspx">
  <system.web>
    <authorization>
      <allow users="?" />
    </authorization>
  </system.web>
</location>

Then, configure CCTray with Via CruiseControl.Net Dashboard to connect to the URL http://myserver/XmlServerReport.aspx. Note that your build statuses (i.e. “Success” or “Failure”) will be publicly available to anybody, yet it’s not an issue to disclose such a limited information.

RESX utilities open-sourced

July 28th, 2007

Due to popular demand, I have finally open-sourced my RESX utilities. All the content (source code as well as binaries) is now available at resx.sourceforge.net, released under the GPL open-source license.

The release includes RESX Editor a simple yet efficient RESX file editor. It can be very handy if the translator is not too much familiar either with XML or with Visual Studio.

The release also includes Resx2word a RESX to Microsoft Word converter. The converter has been packaged as a command line utility (resx2word.exe and word2resx.exe).

Migrating to a DNS hosting provider

July 17th, 2007

I have recently migrated all the DNS data of Lokad.com toward DnsMadeEasy, a provider specialized with DNS hosting. For a long time, it did not even crossed my mind that such low cost independent service would actually exists.

DNS stands for Domain Name System. In simply words, the DNS converts the domain name address into a IP address. Look for the DNS wikipedia, if you want to know more.

Why do I need to know anything about DNS?

As a webmaster, DNS are most usually completely handled by your hosting provider. If your web requirements are simple, you might not even cross DNS settings, everything being completely handled by the hosting provider.

Yet, if you start to rely on 3rd party hosted services such as blog hosting and forums hosting, you may have to update your DNS settings so that the URL is blog.mycompany.com instead of mycompany.myblogprovider.com. With those 3rd party hosted service, your internet domain becomes a patchwork that includes machines owned by various hosting providers. For example, both blog.lokad.com and forums.lokad.com are hosted by 3rd party companies.

Why should I bother about DNS?

At present time, there are many dirty cheap hosted services. Lokad is still a uISV but relies already on almost a dozen of various hosted services providers. And, in very center of this small network lies the DNS. If the DNS go wrong, then the whole network is going to be in deep trouble.

Until very recently, Lokad was relying on the DNS management provided by a low cost regular hosting provider. The DNS hosting was really reliable, we never encountered any issue at that level. Yet, I was a bit concerned by the fact that the “Lokad network” was dependent on a regular hosting provider to manage all the information related to the other hosting providers. This situation was making this particular hosting provider much more critical to the whole network than it ought to be.

Thus, I have decided to migrate everything to DnsMadeEasy. As a result, I am gaining flexibility for potential further hosting migrations. One might argue that I have simply moved for critical node for my previous hosting provider to DnsMadeEasy. That’s true. But DnsMadeEasy is completely dedicated to DNS management. Thus, they are not competing with other providers that are referred by the DNS.