Many companies know the pattern: after a few years, a Data Warehouse gets rebuilt from the ground up. It still happens often, across organizations of every size and industry.
There are two different reasons to rebuild a Data Warehouse.
One is innovation. A genuine technology shift opens capabilities the old system could never provide, for example moving from an on-premise database to elastic cloud storage and compute. The old system worked, but it had reached its limits, and the rebuild on new technology lifts them.
The other is inertia. The system still does roughly what it always did, but every change has become slow and expensive, and the business has grown tired of waiting. Requests pile up faster than the team can work through them. Delivering anything new takes too long.
This article is about the second case. The innovation-driven rebuild is a strategic investment with a clear return. The inertia-driven rebuild is the one to be careful with. It is often launched on new technology and presented as innovation, but new technology rarely removes what caused the inertia in the first place. Unless the underlying causes are found and fixed, the new system inherits them, and the rebuild repeats. It becomes a cycle.
The cycle
It starts with a real problem. The existing Data Warehouse can no longer keep up with business requirements. Delivery timelines stretch. Workarounds pile up. At some point, rebuilding feels like the only viable option. The rebuild happens on new technology, often a new cloud platform, and the early results look promising. Then, over months and years, the same friction returns.
The pattern has five phases:
- “We need a new Data Warehouse.” The current system is too slow to adapt and too expensive to maintain.
- Rebuild on new technology, same approach. The platform changes. The way the warehouse is designed and built does not.
- New requirements take too long. The old problems resurface. The business loses patience and starts building its own solutions outside the architecture.
- Workarounds accumulate. The warehouse team takes shortcuts to keep up with demand. Complexity grows, and shadow solutions multiply.
- Complexity becomes unmanageable. The system is fragile again, and every change carries risk.
Back to step one.
The cycle is not bad luck, and it is not a failure of the people running it. It persists because its root causes are structural.
“We need a new Data Warehouse.”
The decision to rebuild is not made suddenly. It grows over time. But when it is made, the situation behind it tends to look the same: the warehouse has grown so complex that the team can no longer keep pace with the business when implementing new requirements. Frustration and pressure are high on both sides, in the business and in the warehouse team, and the rebuild becomes hard to avoid.
How the warehouse grows too complex to manage
When a warehouse is rebuilt, its complexity is manageable at first. There are few business objects, each with only a few sources, and the KPIs are still simple, calculated from the data of one or a few business objects.
Even so, the implementation already involves a range of tasks:
- logical data modeling, today often with Data Vault
- deriving a physical data model for the target database
- developing and orchestrating the ETL and transformation processes, both for the raw data and for the business logic of calculated KPIs
Two of these take the most experience: the logical data modeling, with its recurring design decisions, and the transformation processes for calculated KPIs.
Today’s automation helps, but it is not enough
When Data Vault models are involved, teams often bring in tools to automate part of the work. But they automate only two of these tasks: generating the physical data model and building the ETL processes for the raw data. For the logical data model, a tool offers suggestions at best, and those still need review and require design decisions.
The transformation processes for calculated KPIs are no different. Mapping the business logic correctly requires a thorough understanding of the data model, and so these transformations are still built by hand.
This mechanizes the warehouse implementation rather than automating it. The repetitive work is handled automatically, while the complex tasks still depend on people. We cover the difference in Automation or mechanization? What real data warehouse automation would take.
Increasing complexity shifts the team’s focus from business to technology
As the warehouse grows, so do the business requirements, and with them the number of business objects and KPIs. The data models become more complex, and the business logic behind the calculated KPIs usually grows more complex along with them.
The more complexity accumulates, the more of the team’s attention goes to the technical side of the warehouse instead of the business requirements it is meant to serve. Development slows, and the whole process starts to feel sluggish.
What accelerates the slowdown
Under pressure, teams sometimes fall back on shortcuts and workarounds to deliver a business requirement on time. Each one raises the overall complexity of the warehouse. If the pressure stays high, the workarounds are never cleaned up, and new ones are added on top. In the worst case, different shortcuts produce KPIs that contradict each other, and the business starts to lose trust in the numbers.
Why new technology does not break the cycle
A warehouse rebuild often comes with new technology: a new platform, a move from on-premise to the cloud, a new query engine. This is what usually gets called modernization or framed as innovation.
But the causes of the slowdown are not technological. The complex data models, the business logic built by hand: neither is removed by changing the platform. The demanding work remains, complexity builds again, and the structural problems return.
Breaking the cycle
The pattern above has a single common thread: every rebuild changes the technology and leaves the way of working intact. That is why a new platform, however capable, returns the organization to step one. The root causes are structural, so the fix has to be structural too.
What breaks the cycle is moving the point of control from the technical level to the conceptual one. Business objects, relationships, and KPIs are defined once, and everything downstream is derived from that definition: the logical data model, and the business logic for calculated KPIs, expressed at the business-object level and derived along with it. The technical implementation is then fully automated instead of only mechanized. The team’s expertise moves up to the conceptual level, to the business objects and KPIs, and away from the technical detail below. We describe how in Your data warehouse as code, with SQL on business objects.
One thing is worth holding onto. The cycle is not a technology problem, so no technology alone will end it. Recognizing that is the first step to not repeating it.