Agile Data Warehouse design
As you might have gathered by now, we’re big fans of Lawrence Corr and Jim Stagnitto’s book Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema (Amazon, eBook). The “BEAM✲ book”, as we call it, has become our AgileBI bible at OptimalBI.
We’ve made a lot of fuss about BEAM✲ on this blog, and taken you through some core parts of the approach, including why you’d build an agile data warehouse, and beam and the agile ba. In this post I’ll take you through another crucial component: modelstorming.
Modelstorming is how you gather business requirements under the BEAM✲ methodology. It’s the one-word version of the tagline of Lawrence and Jim’s book: “Collaborative Dimensional Modeling, from Whiteboard to Star Schema”. Let’s break that down to be clear on all the parts.
Modelstorming is collaborative
This implies you work with, not against, the business stakeholders who have asked you to help. Unfortunately the process of asking what they want, and checking that we delivered it, is often adversarial. If you’ve been on either side of this situation we discuss in An Optimal Introduction to BEAM✲, you should appreciate the value of switching to collaboration.
Modelstorming is requirements-gathering built for the dimensional data models you know and love
Ralph Kimball popularised the dimensional approach to data warehousing, in contrast to the entity-relationship approach best suited for transactional systems, or the third-normal form that end-users of data often prefer. The key benefits of modelling your data dimensionally are: a) data is stored in business-friendly groupings to make it easier to understand, b) this model optimises performance for those queries typically employed in data warehousing, and c) dimensional models are built expecting changes to be made in the future.
If you’re at all familiar with data modelling or data warehousing, you’ll appreciate how these benefits of dimensional models translate directly in to increased value for our stakeholders: a) business-centric, b) fast, c) flexible. Data warehouse developers have flocked to Kimball’s call since the 1990s, and the design patterns for implementing dimensional data warehouses are well-understood.
Yet as Corr and Stagnitto noticed, despite this technical change, “an ever increasing number of BI initiatives [are] stumbling long before they reach the data modelling phase”. In short, a technical revolution like dimensional modelling won’t pay if projects never make it to that phase. They wrote the book to overcome those stumbling blocks, to “revolutionize BI requirements analysis in the same way that dimensional modeling has revolutionized BI database design.” And they latched on to Agile as the way to do it.
This obviously required a lot of brain power, but they started with a core belief that the artefacts (ie documentation) delivered from the agile data warehouse design process must be familiar to the developers building data warehouses. If you re-engineer the requirements-gathering process, only to deliver foreign objects in to the data modelling and build phases, you’ve swapped fire for frying pan. Here’s what those familiar, comfortable artefacts look like. As we say in An Optimal Intro to BEAM✲, there are no surprises here!
Modelstorming starts with a whiteboard
The whiteboard is important because if we show these developer-friendly artefacts to business stakeholders, they’ll get confused and likely run away. We need to keep these artefacts in mind as our targets to please the technies, but introduce a method of populating them that doesn’t scare off non-technical stakeholders. We also need it to be flexible as the modelstorming process is iterative.
Here’s an example from our Optimal Intro to BEAM✲ course, for an event called “Customer Orders Product” in our running case study, “Rick’s Pizza”. This isn’t anybody’s finest work, but it’s approachable to (and even written by!) business stakeholders, and easy to formalise in to an “official” BEAM table above. Starting with the whiteboard is a great way to get conversation and collaboration with your business stakeholders underway.
Modelstorming delivers artefacts that make star schemas easy to validate and build
Together, the documentation produced by modelstorming delivers business requirements that are very close to a star schema. Compared to “I want a report on this” or “give me all the data on that”, you have tailored the conversation with your stakeholders to get them thinking dimensionally without freaking them out. This reduces uncertainty and increases the ease of iteration when you get to the all-important data profiling and schema modelling stages. The BEAM✲ artefacts can also be easily annotated with the results of this validation and build work, leaving an easy-to-use trail for those picking up your work later (a key tenet of agile). Here’s where you get to, a star schema even I can understand! (from p.156 of the book)
tl;dr: Modelstorming = Data Modelling + Brain Storming
Modelstorming is an efficient compromise between developers who just want to deliver great models, and business stakeholders who want to brain dump their wishes. The magic of modelstorming, and the BEAM✲ approach more broadly, is that if those folks each move off their pedestals just a little bit, and get round a whiteboard to collaborate, a huge amount of the conflict that derails BI/DW projects will fizzle away.
Join us to learn more!
We’re so devoted to this way of doing things that we offer our own one-day “Optimal Introduction to BEAM✲” workshop across New Zealand and Australia. And interest in BEAM✲ down here has been more than enough to get Lawrence’s attention: he comes by to teach his three day course at OptimalBI in Wellington you can get information on that here. You can follow the rest of our journey adopting and using BEAM✲ to deliver agile data warehouses for our customers here.
Keep asking better questions (and your stakeholders will love you for it!)
You can book your place on the next Data Driven Requirements Gathering Course here.