A national logistics operator engaged MinervaDB Support to resolve a problem that quietly taxes almost every scaling operation: business-critical data was scattered across more than ten systems, and no two of them fully agreed. Transport management, warehouse management, order capture, telematics, customer relationship management, and finance each held a piece of the truth, and assembling a single report meant exporting from several places and reconciling the results by hand. This article details how we unified those systems into a Databricks Lakehouse single source of truth, cutting report generation from hours to minutes and giving operations and finance one governed view they could finally trust. The approach reflects the engineering discipline we apply to every database infrastructure engineering engagement.

The Cost of Ten Disagreeing Systems

The operator had grown by adding systems rather than integrating them. A transport management system planned and tracked shipments, a warehouse management system governed inventory and picking, an order-capture platform handled bookings, telematics units streamed vehicle and route data, a CRM held customer records, and a finance system owned invoicing and the general ledger. Each was sensible in isolation, but none shared a common definition of a customer, a shipment, or a delivery date.

The practical consequences showed up every reporting cycle. Producing a single operational and financial view meant analysts exported spreadsheets from each platform, manually keyed cross-references between mismatched identifiers, and spent hours stitching the pieces together before anyone could ask a question. Worse, the answers often disagreed: operations and finance regularly arrived at meetings with different revenue and on-time numbers because each had counted from a different source. The reporting delay was painful, but the erosion of trust in the numbers was the deeper cost.

We have seen this pattern across industries, and the resolution is rarely another integration tool bolted between systems. The durable fix is to land all the data in one governed platform and let that platform become the authoritative source, which is precisely what a lakehouse is designed to provide.

What a Single Source of Truth Means

A single source of truth is not a dashboard or a report. It is an architecture in which every consumer reads from one governed, authoritative copy of each business entity, so a metric means exactly one thing no matter who asks. The Databricks Lakehouse establishes this by unifying data access and storage in a single system, removing the need to create and synchronize copies across platforms (Databricks on the single source of truth). When the raw data, the conformed entities, and the business-ready facts all live in one catalog, the question of which copy is correct simply disappears.

For the logistics operator, that meant agreeing on canonical definitions before writing a single pipeline. We worked with operations and finance to define what counted as a delivered shipment, how on-time was measured, and which identifier was authoritative for a customer. Those definitions then became the contract the lakehouse enforced, rather than assumptions buried in ten different export scripts.

• One authoritative copy of each entity, governed centrally rather than duplicated per system.
• Consistent business definitions enforced in the platform, not re-implemented in every report.
• Full lineage from raw source to published metric, so any number can be traced to its origin.
• A single access and audit surface, so security and compliance are managed in one place.

The Lakehouse Architecture We Built

We organized the lakehouse using the medallion architecture, a layered pattern that progressively improves data quality as it flows from raw to business-ready. Databricks recommends this multi-layered approach specifically for building a single source of truth for enterprise data products (Databricks medallion architecture). The three layers gave each team a clear entry point into shared data.

Each layer lived in a separate schema within a governed catalog, so access could be granted at the appropriate level: engineers and auditors reached bronze, analysts worked in silver, and business users consumed gold. The bronze layer preserved the raw state of every source and served as the immutable record, which meant we could reprocess downstream layers at any time without ever re-reading from the operational systems. That separation is what lets a lakehouse act as the single source of truth while still protecting the live systems from query load.

Ingesting 10+ Systems with Auto Loader

Ten-plus systems meant ten-plus ingestion patterns, from database extracts to CSV drops to streaming telematics. Rather than build a bespoke pipeline for each, we standardized on Auto Loader, which incrementally and efficiently processes new data files as they arrive in cloud storage and tracks which files have already been handled (Databricks Auto Loader). Each source landed its extracts in a dedicated storage path, and a single, parameterized Auto Loader job ingested every path into its bronze table.

# Incrementally ingest each source into its bronze table
(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaLocation", schema_path)
   .option("cloudFiles.inferColumnTypes", True)
   .load(source_path)
   .withColumn("_source_system", lit(system_name))
   .withColumn("_ingested_at", current_timestamp())
 .writeStream
   .option("checkpointLocation", checkpoint_path)
   .trigger(availableNow=True)
   .toTable(f"ops.bronze.{system_name}_raw"))

Two design choices kept the bronze layer reliable. First, we stored incoming data with permissive typing and captured provenance columns such as the source system and ingestion timestamp, so an unexpected schema change upstream never dropped a record. Second, the availableNow trigger let the same streaming code run as efficient incremental batches for the systems that delivered files periodically, while genuinely streaming sources like telematics ran continuously. One ingestion framework therefore covered every source, which made adding the eleventh and twelfth systems a configuration change rather than a new project.

Conforming Data in the Silver Layer

Landing the data was the easy part; making ten systems agree was the engineering. In the silver layer we matched, merged, conformed, and cleansed the bronze data into an enterprise view of each key entity, so that a customer or a shipment was represented once, consistently, regardless of how many systems described it (Databricks on the silver layer). This is where mismatched identifiers were resolved, duplicates removed, and canonical definitions applied.

Resolving identities across systems

The order-capture platform, the CRM, and the finance system each identified customers differently. We built cross-reference tables that mapped every source key to one canonical customer key, then conformed all downstream entities to that key. The same approach unified shipment identifiers across transport and warehouse management, so a shipment could be followed end to end for the first time.

-- Conform shipments from two systems into one enterprise entity
CREATE OR REPLACE TABLE ops.silver.shipments AS
SELECT
    x.canonical_shipment_id        AS shipment_id,
    x.canonical_customer_id        AS customer_id,
    COALESCE(t.promised_date, w.promised_date) AS promised_date,
    COALESCE(t.delivered_at,  w.shipped_at)    AS delivered_at,
    CASE WHEN COALESCE(t.delivered_at, w.shipped_at)
              <= COALESCE(t.promised_date, w.promised_date)
         THEN TRUE ELSE FALSE END AS is_on_time
FROM ops.silver.shipment_xref x
LEFT JOIN ops.bronze.tms_raw t ON t.ship_id  = x.tms_key
LEFT JOIN ops.bronze.wms_raw w ON w.ship_ref = x.wms_key
QUALIFY ROW_NUMBER() OVER (
    PARTITION BY x.canonical_shipment_id
    ORDER BY COALESCE(t._ingested_at, w._ingested_at) DESC) = 1;

The single, canonical definition of on-time delivery lived here, computed once in the silver layer rather than re-derived in every report. The QUALIFY ROW_NUMBER() pattern deduplicated late-arriving and replayed records so each shipment resolved to exactly one current row. We enforced schema, handled nulls, and quarantined records that failed validation into a separate table for review rather than silently discarding them, keeping the conformance process fully auditable.

Modeling the Gold Layer for Reporting

The gold layer is where the data became business-ready. We modeled it as a read-optimized, denormalized dimensional schema, the Kimball-style star design that the medallion pattern recommends for the gold layer because it minimizes joins on the queries reporting runs most (Databricks gold-layer modeling). Narrow fact tables for shipments, deliveries, and invoices sat alongside conformed dimensions for customer, location, and date.

-- Pre-aggregate the metric operations and finance argue about most
CREATE OR REPLACE MATERIALIZED VIEW ops.gold.daily_delivery_kpi AS
SELECT
    d.delivery_date,
    c.region,
    COUNT(*)                              AS shipments,
    SUM(IF(f.is_on_time, 1, 0))    AS on_time_count,
    ROUND(100.0 * AVG(IF(f.is_on_time, 1, 0)), 1) AS on_time_pct,
    SUM(f.invoice_amount)                 AS revenue
FROM ops.gold.fct_delivery f
JOIN ops.gold.dim_customer c ON c.customer_key = f.customer_key
JOIN ops.gold.dim_date     d ON d.date_key     = f.date_key
GROUP BY d.delivery_date, c.region;

For the handful of metrics that drove every executive dashboard, we precomputed the result in materialized views so a dashboard read a small, maintained table instead of recomputing across the full fact history on each load. Because on-time percentage and revenue were both derived here from the same conformed facts, operations and finance now read identical numbers by construction. The disagreement that used to fill meetings was engineered out of the system.

Governance and Trust with Unity Catalog

A single source of truth is only trusted if it is governed. We placed Unity Catalog at the center of the lakehouse to provide one unified view of every data asset, one tool for access management, and one surface for auditing, which is what makes governance seamless across a unified platform (Databricks Unity Catalog governance). Three capabilities mattered most for establishing trust.

• Centralized fine-grained access control, so operations, finance, and audit each saw exactly the data appropriate to a role, managed in one place rather than per system.
• End-to-end lineage, so any number on any dashboard could be traced back through gold, silver, and bronze to the originating source record.
• Consistent definitions and auditability, so the canonical meaning of a metric was enforced by the platform and every access was logged for compliance.

Lineage proved decisive for adoption. When a finance leader questioned an on-time figure, we traced it from the dashboard through the gold materialized view, the silver conformance logic, and back to the exact bronze records and source systems in minutes. Being able to show the provenance of a number, rather than argue about it, is what converted skeptical stakeholders into users of the shared view.

From Hours to Minutes

With conformed silver entities and curated gold tables in place, reporting changed character entirely. Reports that had required hours of manual export and reconciliation now ran in minutes directly against the gold layer, because the heavy work of joining, conforming, and aggregating had already been done once, upstream, rather than repeated by every analyst.

Several factors compounded the speedup. The gold materialized views returned pre-aggregated results instantly for the common dashboard queries. The dimensional model meant most reports resolved with a single layer of joins rather than walking ten source schemas. And because business users queried the gold layer rather than the operational systems, reporting load no longer competed with live operations, so neither slowed the other. The manual stitching step, which had been the single largest consumer of analyst time, was eliminated.

Just as important, the reports finally agreed. Every dashboard drew from the same governed facts, so the morning operational review and the monthly finance close worked from one set of numbers. The shift from hours to minutes was the visible win, but the shift from contested numbers to a trusted shared view was the one that changed how the business made decisions.

Results for Operations and Finance

Measured against the prior fragmented process, the unified lakehouse delivered clear outcomes.

• More than ten operational systems were unified into one governed Databricks Lakehouse single source of truth.
• Report generation fell from hours of manual assembly to minutes running directly against the gold layer.
• Operations and finance reviewed identical governed figures, ending the recurring disputes over revenue and on-time numbers.
• Every published metric carried full lineage back to its source records, making numbers traceable rather than contestable.
• New data sources were onboarded by configuration into the existing ingestion framework instead of bespoke projects.
• Live operational systems were shielded from reporting load, because business queries ran against the lakehouse, not the sources.

The lesson we stress to every operator carrying a sprawl of disconnected systems is that the answer is architectural, not cosmetic. Unifying data into a governed lakehouse, conforming it once into trusted entities, and serving curated facts to every team is what turns ten disagreeing systems into one source of truth. Faster reports are the result; trusted reports are the point.

Key Takeaways

• Agree on canonical business definitions with stakeholders before building pipelines, then let the platform enforce them.
• Use the medallion architecture so data quality improves progressively from raw bronze to business-ready gold.
• Standardize ingestion on Auto Loader so onboarding a new source is configuration, not a new pipeline.
• Conform identities and compute shared metrics once in the silver layer rather than re-deriving them in every report.
• Model the gold layer dimensionally and pre-aggregate hot metrics with materialized views for instant dashboards.
• Govern with Unity Catalog for centralized access, end-to-end lineage, and the auditability that earns stakeholder trust.
• Serve reporting from the lakehouse, not the operational systems, so analytics and operations never compete for resources.

Frequently Asked Questions

What is a single source of truth in a Databricks Lakehouse?

It is an architecture where every consumer reads from one governed, authoritative copy of each business entity. The Databricks Lakehouse unifies storage and access in a single system, so there is no need to create and synchronize copies across platforms, and a metric means exactly one thing regardless of who queries it.

How do you unify data from more than ten different systems?

We land every source into a bronze layer using a standardized Auto Loader framework, conform identities and definitions into enterprise entities in the silver layer, and publish curated facts in the gold layer. Cross-reference tables map mismatched source keys to one canonical key so each customer or shipment is represented once.

Why did report generation drop from hours to minutes?

The heavy work of joining, conforming, and aggregating is done once upstream in silver and gold rather than repeated in every report. Dimensional modeling reduces joins, materialized views pre-aggregate hot metrics, and business users query the lakehouse instead of the operational systems, so reports run fast and without contention.

How does the medallion architecture support a single source of truth?

Bronze preserves raw source data as an immutable, auditable archive and serves as the foundational record. Silver conforms and cleanses it into trusted enterprise entities. Gold curates business-ready facts for reporting. Each layer improves quality progressively, which is exactly the pattern Databricks recommends for an enterprise single source of truth.

What role does Unity Catalog play in establishing trust?

Unity Catalog provides centralized fine-grained access control, end-to-end lineage, and unified auditing. Lineage lets any dashboard number be traced back to its source records, which is what converts skeptical stakeholders into users of the shared view, while consistent governance keeps definitions and security managed in one place.

How are new data sources added after the platform is live?

Because ingestion is standardized on a parameterized Auto Loader framework, onboarding a new source is largely a configuration change: point it at a storage path, register its bronze table, and extend the silver conformance logic. This avoids building a bespoke pipeline for every system and keeps the architecture scalable.

Published by MinervaDB Support · Full-Stack Database Infrastructure Engineering, Operations & Analytics · minervadb.com/