The Problem
The organisation had built its reporting stack on a legacy SQL-based data warehouse that had grown organically over years. What started as a manageable setup had become a bottleneck: executive reports that should take seconds were running for hours, and the business had lost confidence in the numbers.
Data quality issues were endemic. Gaps in historical records meant trend analysis was unreliable. IT teams spent their time firefighting data pipeline failures rather than building new capability. Every month-end close became a stressful manual exercise.
What We Built
Medallion Architecture on Microsoft Fabric
We moved the entire data platform to Microsoft Fabric, implementing a three-layer Medallion architecture — Bronze, Silver, and Gold — that gave each data layer a clear purpose and a clear owner.
- Bronze layer: Raw ingestion from all source systems, preserved exactly as received. Nothing is discarded.
- Silver layer: Cleaned, validated, and conformed data. Business rules applied consistently. Data quality checks run automatically.
- Gold layer: Business-ready aggregations and models, purpose-built for reporting and analytics consumption.
Full Historical Capture
One of the organisation's key requirements was closing the historical gaps. We implemented a backfill strategy using PySpark on Fabric to reprocess years of source data, ensuring the Gold layer contained a complete, unbroken history — something the legacy warehouse had never achieved.
Governance Dashboards
We built a suite of Power BI dashboards specifically for governance and oversight — not just operational reporting. Executive teams gained visibility into data lineage, pipeline health, and platform usage. IT teams received monitoring dashboards to catch issues before they reached end users.
Tech Stack
The Results
Report execution times collapsed from multi-hour runs to under a minute for most workloads. The business now runs month-end close with confidence rather than anxiety. Four governance dashboards give executives and IT full visibility into platform health and data lineage — a first for the organisation.
Key Takeaways
- A Medallion architecture isn't just a pattern — it's a governance framework that makes data ownership explicit at every layer.
- Historical backfill is always harder than it looks, but it's the difference between a reporting platform and a trusted one.
- Governance dashboards should be a first-class deliverable, not an afterthought. Decision-makers need to trust the platform, not just the numbers it produces.