It’s the first day of spring here in Wellington, and OptimalHQ is buzzing after our successful first delivery of “An Optimal Introduction to BEAM✲”. We spent two sprints developing the course from scratch, including a running example, and it was great to see it come together on the day.

What is BEAM✲?

BEAM✲ stands for Business Event Analysis & Modelling, and it’s a methodology for gathering business requirements for Agile Data Warehouses and building those warehouses. It was developed by Lawrence Corr (@LawrenceCorr) and Jim Stagnitto (@JimStag), and published in their book Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema (Amazon, eBook). “The BEAM book”, as we call it, has become our AgileBI bible at OptimalBI and we are already using BEAM to build data warehouses.

What is Agile?

Agile grew out of software product development and has natural strengths for any product-focused startup. As a product and services company, we have also successfully applied it outside software development, to research tasks, and even used it to develop our code-free BEAM workshop.
Everybody claims the Agile mantle these days (read this and you’ll think of somebody!) but how do you really qualify? The full manifesto is twelve points long, but boils down to four key statements:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
– quoted from agilemanifesto.org

If you just fell out of your chair, that’s OK, Agile should be a little bit shocking to most of us! Either because we’re so risk-averse it makes us reach for the nearest project planning document in panic, or because we just realised how we’ve used the non-bolded alternatives above to cover our backsides in the past. It’s powerful language and stands in stark contrast to the “Waterfall” approach in which all requirements are specified up-front and mid-development change is strictly controlled.

So what does Agile mean to me?

“Individuals and interactions over processes and tools” implies that if you push a particular process/tool as “the Agile way to do X”, you’re lying. At best, a tool enables your team to successfully use Agile to deliver for your customers.
“Working software over comprehensive documentation” doesn’t mean no documentation. It means that meeting customer needs by delivering something that works is more important than documenting extensively why you failed to deliver anything useful at all.
“Customer collaboration over contract negotiation” means the customer really is always right, unless you convince them to go another way. You talk to them constantly about how things are going, rather than waiting until just before those grim change control board meetings.
“Responding to change over following a plan” means you value the relationship and meeting customer needs over using a piece of paper to dictate the development path.

The most important thing to remember is that Agile ≠ “Ad hoc”. Agile implies a different way of planning, documenting and developing, typically focused on short cycles of work and extensive collaboration with the customer. Agile will only lead to better and faster outcomes, and happier customers, if you commit to its principles. If you use it as an excuse to be shoddy, the wheels come off remarkably quickly.

Why is Agile necessary for BI?

Any vendor will tell you that reliable business intelligence requires flexibility to deal with a changing business environment. If fields in your reports are hard-coded in SQL back to the source data from an operational system, and IT upgrades that system, are you confident your reports will run the next day? Even if they run, are you confident the result can be compared to yesterday’s? When the upgrade breaks the system, how quickly can you resume delivering critical insights?
Business intelligence has held out against the Agile revolution, but the methodology really is a natural fit for our world. In some ways, the BI environment is even less stable than the product development landscape. We must weave diverse data from multiple systems in to coherent insights that deliver business value. Unfortunately, we are rarely stakeholders in the decision-making that changes this challenging environment. AgileBI, enabled by following the BEAM methodology, sets us up to succeed in this volatile environment from the first requirements we gather. Here’s how:

Individuals and interactions: business intelligence is powered by the questions users have about their business, and no tool extracts these automatically
Working software: well-documented data warehouses that take years to deliver will always be out of date, and business users will go elsewhere for insight
Customer collaboration: end-user knowledge about their business is your greatest resource, so don’t use the contract as a barrier to extracting it
Responding to change: if you do all of the above, change comes naturally, and the plan can only fail for weeks rather than months/years on end

In one sentence: AgileBI uses continuous collaboration to deliver information faster, embracing frequent change. Or as a motto: speak with your users in their language, expect that what they want will change.

How does BEAM✲ help me be Agile?

You’re probably thinking “that sounds good, but tough”, and you’re right. The danger of stripping back the comfort-blanket protections of the traditional Waterfall approach is chaos. If we go from talking to users poorly to not at all, things will get worse. We’ll deliver software that never does what they want, and will fall back on the contract to defend ourselves against necessary changes.
Remember: Agile ≠ “Ad hoc”. We need to replace Waterfall with something else, preferably something developed by smart people who realised long ago the problems outlined above, and set about fixing them. For us, BEAM is that something else: a complete methodology that delivers the promises of Agile, fully aware of how tough that is to do in BI/DW. My favourite feature of the BEAM book is that it doesn’t shy away from the tough cases. In fact, these motivate its approach to building a data warehouse: we should expect the edge cases, the missing data, the evolving events that are all features of the real-world systems we pull data from.
The way to find these cases and make sure they inform data warehouse design is to ask end-users about the events that happen in their business (hence Business Event Analysis & Modeling). The question “what things happen in your business?” is a better way to start gathering requirements than “what entities do you want to report on?” Following up with “what other events happen around the event we just discussed?” is better than “does this FACT table share any dimensions with other FACT tables?” Of course, a skilled BA would never ask those questions up front, but part of BEAM’s magic is that it gives business users and developers a common language. Rather than relying on the BA for translation, the BA can facilitate agreement about the answers to the questions prescribed by BEAM methodology.
Here’s a sneak peek at a BEAM event table from our course. Several “data stories” have been transcribed in to what looks like a spreadsheet to the business user. On this first pass, the priority is to capture business information, by following an easy set of questions about this event: “Who does what?” then “when? where? how many? why? and how?”  Little do the business stakeholders know, the distance from this template to a complete dimensional model for the DW developer is decreasing with every new story transcribed. This is what we mean by a simple common language for requirements.

What is An Optimal Introduction to BEAM✲?

We provide a full-day workshop that briefly introduces BEAM✲ and then gets you using it right away to gather business requirements in an extended case study. It doesn’t replace the book but provides an accessible introduction to the concepts and hands-on experience so you can see the value in this way of doing things.
The February 2014 revision of the BEAM book weighs in at 304 pages, and it truly deserves to be called a bible of AgileBI. The first third of the book covers requirements gathering (ie our workshop), the remainder explains the process of turning requirements in to a working data warehouse (our future workshop). When Lawrence runs his three-day workshops, he follows a similar split: one day on requirements, two days on building.
To use BEAM properly, you’ll need to read the book eventually. But as with any new way of doing things, part of the adoption curve is understanding why the new way is better. We believe that for our clients, a brief introduction to BEAM and immediate hands-on practice is the best way to demonstrate its ongoing value. If we can convince practitioners by giving them a taste, we are confident they can take the next step and use BEAM in their organisation.

Tell me more!

Until next time, keep asking better questions Shaun – @shaunmcgirr
You can read Shaun’s blog AgileBI vs BEAM✲ vs Modelstorming.
Shaun blogs about analytics, machine learning and how data solves problems in the real world.