Level-of-details and the difficulty of software project estimation

All levels of details have to considered in order to estimate a software project accurately.

Numerous books, articles and commentaries have been written about the difficulty of estimating a software project, however, none of them has really sinked into my conscious, even though I have gone through a number of different software projects – that is until recently, when I was faced with a firm deadline for releasing a software package. The following series of examples illustrate how all levels of details have to be considered in order to estimate the scope of a project, they include things such as conceptual mismatches between the high-level view of the project design and the lower-level views or implementation views, potential problems with tools, social interactions, and maintaining a clear head under pressure. Guess it will always be a challenge to keep all of them in proper perspective.

1. Conceptual Mismatches

There was a number of features that I have promised to different people -- my original estimation was that they will take no more than two weeks. They seem to be mostly just wrapping existing functions to provide a nicer interface, one that is a little closer to the theoretical ideal of relational algebra. Since the software was for indexing and querying large data warehouses, this goal seems to be a reasonable target to achieve, especially since I have promised to do it already to my boss and some expected users.

Gosh, was I surprised when I finally sit down to do it. It took more than a month, 10 thousand lines of C++ code, five different new classes, and endless changes to the existing code. Now that I have just finished this project, it is a good time to take a quick look back and see what led me to such optimistic estimation.

The root cause of this problem seems to be that I was thinking too much in terms of relational algebra and forgot about the messy details needed to make things conform to this nice ideal. For example, the relational algebra deals with values in abstract, therefore has no concept of data types, however, the C++ programming language I chosen to implement the software was strongly typed. This means that a large number of functions in the main API has to been devoted to type conversions. These conversion functions are long and full of ugly looking casting functions. They are also the major source of complaints from compilers. Eventually, I have to give up on trying to remove all the warnings about potential loss of data due to type conversion. For example it is possible for a user to enter a variable as signed 32-bit integers, but retrieve them as unsigned 32-bit integers. In which case, all negative values in the original data will be reinterpreted as ridiculously large numbers. Guess, the user should know that they are converting signed integers into unsigned ones, but this is a level-of-detail that was not visible from the high-level view of relational algebra.

2. Problems with Tools

In preparation for the release of the code, I thought it might be a good idea to have some documentation beyond what is in the code (through doxygen). Since I already have a number of different web pages developed for different users, it would be a piece of cake to throw them together with some introductory material. Should be something doable in a week. Boy, was I wrong about this. The first set of problems was related to the HTML editors. Since I am familiar with text editors, I thought it is easiest to use a text editor. Well, the problem a text editor is that I make tons of typos and the text editor I was using offers no help in flagging them or correcting them. Since I decided to try some word processors, like MS Word and Open Office Writer. These programs are happy to take what I give them, but immediately mangle them with their own CSS styling instructions. Every time I tried to spell check a HTML page, it will take me hours to reverse the styling problems that was introduced by the work processor. The cure was worse than the sickness.

In this case, the unexpected problem was that the tools are not well suited for the task at hand -- a level of detail that was unanticipated in the planning stage.

3. Social Grace

Before I open up the released version to everyone on the Internet, I asked a few friends and friendly users to try out the package and its documentation. A number of good suggestions came out that exercise, which seems to be a good thing. However, a couple of little things was missed in the process and cause some more delays.

A user was nice enough to suggest some ideas for a document that help users to start using the package quickly. Great, I went to work right away and got back to the him a day later intending to obtain some feed backs. He was eager to help out and was telling me that he has already found typos and other problems with the web page. He said that he can send me all the changes he has made under MS Word. Based on my previous experience with MS Word, I blurt out that I hated what MS Word does to my HTML file. Well, that made the conversation terminated quickly and I never see the changes he made. Guess I still have a lot to learn about graceful social interactions and taking whatever forms of help I can get.

4. Never Forget the Basics

In my rush to release the code, I made a number of last minute changes without rerunning the examples given the documents. Luck would have it that the example that was broken was the very first one in the quick start document. Well, now that everyone who download the package and try to run it, the first thing it does is to core dump. It probably caused all new users to discard the package and curse me for wasting their time. Thank goodness, there was a bug report about it the very next day the package was posted on the web. The moral of this story, don't forget to try the examples in the documentation, even if you are pressed for time.

Notes:

B. The person who suggested the quick start document for FastBit was Luke Gosink from UC Davis. I am very grateful for his suggestion and numerous other helps in developing the FastBit software and documentation.

C. The person who reported the bug related to the first example in the quick start document was Mark Sordo from Yahoo! Research.