Data Warehousing

Not necessarily a data warehousing topic, per se, but it is relevant to Infobright as an open source database.

An interesting read on the open source culture and community - "The Pirate's Dilemma".

http://www.amazon.com/Pirates-Dilemma-Culture-Reinventing-Capitalism/dp/1416532188/ref=sr_1_27?ie=UTF8&s=books&qid=1231168447&sr=1-27

I’m a bit of an old war horse when it comes to project management. Having managed many projects over the years and as my boss would be happy to point out, having plenty of grey hair to show for it.

When I first started my career, projects were managed using the waterfall life cycle management. Of course some more successfully then others. I got quite good at it to tell you the truth. As the practice software engineering improved, iterative development approaches where introduced which I quickly applied to my projects. Following the Microsoft mantra of “get to a shippable state and keep there”; adding function and feature iteratively once a skeleton framework was established.

We apply Agile methods to our development process at Infobright. Applying Agile to the development of a database system is a bit of a struggle since the problems tend to be fairly large and rarely fit in one iteration so defining appropriate test points at the end of an iterations becomes a bit of a challenge. However, the benefits of Agile outweigh many of these struggles and we have been quite successful with its application.

Then I got to thinking about the various data warehouse projects I have worked on. Most of these were pretty mammoth scale projects both in the objective and the number of people involved. All followed some sort of waterfall approach to project management and all had issues around changing requirements, unknowns in source system data, etc. All projects suffered from date slippages and failed features that had to be re-implemented. Just for the record, I wasn’t managing the projects but working on requirements gathering and data warehouse architecture and design.

If you look at Agile and think carefully about what development problems it solves, it seems to me it’s a perfect fit for data warehousing applications. Each iteration can bring one more bit of data/subject area into the warehouse. Business can prioritize data/analysis needs and they can change them every month! The team can apply what it learns about the data over time rather than waiting until its completely implemented before figuring out the data quality is insufficient to meet the business requirement.

So what’s different about Agile relative to other iterative development approaches? Well, several seemingly minor things but they add up. Here’s a few of the more obvious.

• Create a list of business functionality to be delivered by the data warehouse project and have the business prioritize it.
• At the beginning of an iteration, select from a backlog of business functionality the items that deliver the most value to the business and can be completed in a month (if items are too big for a month, break them down into several smaller functional items.)
• Time box the iteration to about a month (we follow calendar months).
• Leave the development team alone during the iteration (i.e. don’t change the task list). If something disastrous happens, scrap the iteration and start over.
• The development team should be cross functional, including developers (ETL and BI), QA, business analysis etc. Keep the team small, around 7 or so. Have the team members sign up to the tasks from the task list. Watch magic happen as the team solves their problems to meet the iteration goals.
• At the end of the iteration, demonstrate the delivered functionality. The software should be shippable, so the business can choose if they would like to implement it into production.
• Repeat the process, starting by re-prioritizing the backlog of business items and selecting the tasks for the next iteration

 Of course there are much more to Agile than I can put into a few bullet points. There are many resources on the web that can help you get started with Agile if you are interested (see http://agilemanifesto.org/ and http://www.agilealliance.org). I just find it curious that data warehouse projects don’t seem to follow the best practices in software engineering. Perhaps people don’t think of data warehousing as software development, since you generally implement using tools (but this is really just another type of programming isn’t it?)