Borrowing from Eric Reis’s book Lean Startup, start your data warehousing project simply. Often, we envision a comprehensive, elaborate solution, but it goes beyond the problem at hand. Or, the problem at hand is complex but you may not see a clear path to the solution. Start small by identifying the smallest possible deliverable that meets the most important requirements but may not address everything the stakeholders have in mind. Build on that solution toward the final vision.
Complex software systems are everywhere in Corporate America. The successful ones start out small and are evolved through careful stewardship. Systems that start out complex often fail because the kitchen sink approach proves difficult to manage and is filled with untenable interactions among the components.
Update
I recently came across an article that refines the Minimum Viable Product (Solution) approach. The refinement advises that the solution be something the users actually want, even if minimum. Users don’t care about a minimum system that doesn’t do anything useful. This is an important nuance and serves to prioritize aspects of the design so that we learn as necessary but keep the users / customers engaged by delivering useful results.
Recent Case Study
In one project, we took on enormous scope, beyond what the users wanted. The thinking was that we would build for the future. Then, we created an ivory tower design. The design was beautiful on paper and on the whiteboard. We checked all the architectural boxes. All along, our stakeholders kept repeating that they needed the solution to load in 20 minutes and Excel was the best visualization tool. The project team of course knew better, and, we went into our development cave for months. When we finally emerged, the load process took hours and had all sorts of pretty reports in a fancy visualization tool. We were miles off the mark. We never actually delivered the solution. Instead, we had to trash much of what we built. In the end, we pivoted to a simpler solution that did what the users wanted, but we did so a year late on a project that was supposed to be about 15 months in the first place.
Keep the Scope Small
There were many lessons learned in our project. The first of which was to keep the scope small. The users don’t care that you’ve designed for the future if you can’t handle the here and now. Identify the smallest viable project, outcome, or product that still works. Sometimes, that aligns with singling out a stakeholder and building for only their requirements. Or, you may need to slice up the business plan and focus on just one aspect. Use the simplest design that delivers value and won’t lead to chaos.
Deliver Fast
Someone once said, to deliver fast, you must fail fast and break things. That’s fine if you’re building a web site to post vacation photos. Mistakes are not really tolerated in a world of SOX compliance and Dodd-Frank. Fail fast if necessary, but do it as early in the process as possible. Don’t deploy to production only to discover that it somehow does not work or worse, that you delivered the wrong thing.
The key to fast delivery is a transparent partnership with your users along with a solid technical delivery team. The project leader must possess a strong penchant for soliciting and acting on users’ feedback. The technical delivery team must be able to slice through the requirements and modularize the delivery into manageable chunks that users can test.
Small is Huge
The Agile methodology attempts to rein in the risk of having to know everything up front. Agile is not always a fit for certain projects. For example, if a system involves calculation, there may not be a “close” result that can be prototyped. The calculation is either right or wrong. Stakeholders usually don’t have the patience for wrong answers. They are often not interested in looking at a system unless it works.
Many times, large projects do not have the luxury of pure agile development. Meaning, a project will not get funded unless there is an estimate. An estimate cannot be credible unless it is based on a design. A design must be based on actual requirements. Requirements must be gathered and documented. And everything gets done as a waterfall.
If you can’t go full-on agile, and waterfall seems too heavy, the best thing to do is to start small. Control the scope. Run the project like a lean startup. Deliver fast. Nominate a benevolent dictator to run the project. Deliver something that’s useful and works. Rinse and repeat until the original vision is fulfilled.
Nominate a Benevolent Dictator
Identify a single person to own the design and deliver the project. Let that person have full control. Now, this person must have the requisite technical and project management skills to deliver. Most importantly, the person must have the courage to stand up to users, upper management, and their own team to keep the project focused and on track to deliver. The benevolent dictator must have the authority and the freedom to do the right thing.
Bottom Line
The agile process is great but not always realistic for large projects. Waterfall can be too slow and accrue risk for each day that you don’t deliver something. Strike a balance by keeping the scope as small as possible and the solution as simple as possible. Be transparent with your stakeholders. Get feedback often and act on it. In the end, you are managing each step toward success.
Does agile fit into your data warehouse delivery framework? When have you had to stand up an go against the grain to deliver a project?