The Worst Unfixed Fixed Inexistent Bug

Wondering eyesWe all know that Delphi is a really great tool. Many consider it #1 in Desktop Windows development. So do I, but perhaps I’m biased. I use it from so many years, and I still have to find something better that it, especially when it comes to talk about complex GUIs, speed of development, speed and size of executable, power and flexibility given to the developer, mainly when we speak about database-related applications.

Pretty nice advertisement, eh?…

Well, many of us think that this is true, especially with the latest versions of it (I mean D2007, D2009) – guys, no religious wars, please! The post is about another thing.

It’s about a thing which, while “inexistent” (note the quotes) from technical POV (perhaps) it was (and still is!) one of the Most Expensive Human Bugs which I found in my entire career…

Some background

One of the most basic (and used) features in an n-Tier application is a so called ‘in-memory DataSet’ which (for ones who doesn’t know) is a TDataSet descendant which holds all its data in memory. Commonly used for caching / buffering, fetching / briefcasing data from various DB servers, as well as stand-alone, file based, data engine etc. Anyway a very used workhorse. And one of the most important features is, of course, speed.

The Genesis of the Thing

Many years ago, (almost ten, I think), Borland (then the owner of Delphi) introduced an in-memory DataSet called TClientDataSet. We all jumped on it but it was sloooooow (and it required a DLL – thing which latter was addressed by having the option to embed the DLL right in the .exe). Dog slow. All DB developers (a huge mass) perhaps would prefer it, but we’re forced to switch to other 3rd party solutions which were much faster. To have an idea, there were (and still are) in-memory datasets which are 10 times faster. (!) (Yep, folks, we did benchmarks as anyone else).

The Domino Effect

But TClientDataSet was a very nice piece of code. And Delphi team built underneath a data access framework (DBExpress – DBX for short) which even if it was also ‘a very nice piece of code’ it relied on TClientDataSet to do CRUD operations with the data in a true RAD way. Of course, you can display/edit/insert the data “by hand”, but…

So, even if DBX was (and is) a solid infrastructure, the final link was a very weak (slow) one, and as we all know, the chain is a weak as it’s weakest link. On the other part, of course, there were a blooming market of 3rd party vendors providing their frameworks (masterpieces of code btw), exploiting this weakness which dragged down the entire official Delphi database access solution. And mind you, this is one of the main Delphi sweet spots…

The Mystery

What amazed me is that TClientDataSet didn’t like at all as a pile of junk. Well thought, nice features, a joy to use theoretically.. Same applied to DBX. Retest it as new versions of Delphi appeared. Looking at Help. Nothing…

After the Borland’s spun off, the Delphi team (which suffered some changes) became more open and we begun to know them. They seemed wery good programmers, thing which increased my surprise. Finally, I “hooked” one of DB engineers in a newsgroup and I showed him the benchmarks. He was very kind and promised that he’ll “have a look”. Indeed after few ours (a very short response time, considering his workload) he spot the “problem”. And the answer was stunning:

Try to set the TClientDataSet’s.LogChanges property to ‘False’. And please re-run the benchmarks and tell me how is going.

…and indeed that was it. After ten years. I looked immediately again at Help. Nothing. In Object Inspector… nothing. Perhaps a Readme.txt… somewhere… Release Notes… Google… Aha! There were some guys who knew about it. But many others who don’t (read the entire thread).

And the bug?

Is in human communications. Such a thing which has such severs impacts on performance should be known by anyone who had an interest in evaluating that solution. Imho, they should made the property ‘published’ (in order to appear in the Object Inspector), to document the property in Help and to specify in a distinctive formating something like “NOTE: The LogChanges property logs all the dataset’s changes in order to allow you to post the things back to the database. But for large datasets it severely affects performance.”

Ok, we are very happy with out 3rd party solution. But for us was an unforgettable lesson that the human factors prevail the technical ones. We should never forget that the ones for which we code are humans and we must guide them how to use our product. And we must be sure that we know the pain points of our user base.

7 thoughts on “The Worst Unfixed Fixed Inexistent Bug

    • Are you interested in more like this?
      Spread the word. After the series of posts related to D2010 we’ll have again such ones.

    • Imho the things are a little bit different. The memory allocation bug was a rather known one and not with so deep impact compared with the one which I described. My post isn’t about a technical bug, is about a _human_ mistake which should be avoided in the future and is a lesson to learn for all of us because we all are prone to such mistakes. A technical bug is easy to fix (is somewhat obvious) but a not so good way to handle things from human point of view is way harder to correct.

      • Of course is a documentation failure. But if instead of:

        – Insert/modify/copy as deleted record on the delta
        – Make the record appear if UpdateStatus is correct
        – When apply updates, merge the Delta with Data packed

        you:

        – Put/remode the actual record to/from data buffer.

        Is pretty obvious you’ll gain a consirable performance
        difference… What the documentation not show is the factor
        of that difference.

  1. There are a few areas where a ClientDataSet shows a poor performance. Most of them were addressed during the years. The Midas Speed Fix from Andreas Hausladen is one of them, allowing a ClientDataSet to have a linear performance when the number of records increase.
    I’ve been profiling DBClient.pas and Provider.pas to find possible bottlenecks. And recently (believe me, I started using ClientDataSet more than 8 years ago!) I found the main bottleneck in a multitier application that uses ClientDataSet and DataSetProviders in a Provider-Resolver construction.
    You can find more about it in my blog here:
    http://alexandrecmachado.blogspot.com/2009/08/make-your-datasnap-application-fly.html

    Best regards

  2. Pingback: RAD Studio 2010 Review #10: – Database Jewels « Wings of Wind Software

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s