Reliable data engineering, measured in results.

The situation

A US financial-services firm was running its nightly reporting on a tangle of legacy SQL Server stored procedures and SSIS packages. The full ETL cycle took roughly 14 hours, which meant analysts routinely started their day on stale data, and any failure mid-run pushed reporting into the afternoon. Every schema change was a multi-day risk because nothing was tested or version-controlled.

What we did

• Migrated 47 on-premise SQL Server sources into a Delta Lake lakehouse on Azure Databricks, using the medallion (bronze/silver/gold) pattern.
• Rebuilt transformations in PySpark with incremental processing, replacing full-table reloads that were the main cause of the long runtime.
• Orchestrated the whole flow in Azure Data Factory with retries, alerting, and data-quality checks so failures surface immediately instead of silently.
• Right-sized clusters with autoscaling, which is where most of the cost reduction came from.
• Ran a parallel cutover so the old and new pipelines produced identical output before switching off the legacy system — zero data loss.

The result

The nightly cycle now finishes in about 22 minutes — a 97% reduction — so analysts have fresh data before the workday starts. Right-sizing and incremental loads cut compute cost by roughly 60%. Because everything is orchestrated, tested, and version-controlled, schema changes that used to take days now ship in hours.

The situation

An EU healthcare provider needed near-real-time visibility into patient data for operational reporting, but was working with batch processes that lagged by hours. On top of the latency problem, anything touching patient data had to satisfy strict GDPR requirements around access control, lineage, and data residency — which the existing setup couldn't demonstrate.

What we did

• Built a streaming pipeline on Databricks using Structured Streaming and Delta Live Tables, replacing the lagging batch jobs.
• Implemented Unity Catalog for fine-grained access control, full data lineage, and auditability — the backbone of the GDPR story.
• Kept all processing within an EU Azure region to meet data-residency requirements, with row-level security on sensitive fields.
• Delivered 12 Power BI dashboards on a trusted semantic model so clinical and operational teams worked from one set of numbers.

The result

Data now flows end-to-end in under five minutes, turning what was overnight reporting into something close to live. The Unity Catalog foundation gave the compliance team the lineage and access controls they needed to sign off on GDPR, and the 12 dashboards became the single source of truth for day-to-day operational decisions.