Data Lake buildout on AWS using Databricks

Problem

Analytics and ML depended on extracts from operational systems, and deployments were manual.

Work

• Dev, Stage and Prod Databricks workspaces on AWS, provisioned in Terraform: networking, storage and cross-account IAM.
• RBAC over all data through Unity Catalog groups, schemas and grants.
• External connections to warehouse, lakehouse, NoSQL and relational sources.
• 27 Delta Live Tables pipelines, deployed with GitHub Actions and Databricks Asset Bundles; unstructured data cataloged in Unity Catalog.
• Batch and online inference on MLflow Model Registry, Feature Store and Delta Lake, validated on held-out data before promotion. An in-house lead-scoring model replaced a third-party model.
• Applications built on Lakebase.
• Agentic engineering with Genie Code: shipped team skills to every engineer’s AI coding harness, including contractors and consultants across time zones, to enforce pull request standards.
• Pandas-to-PySpark rewrite of a finance pipeline: 55 min to 15 min.
• A pipeline template (incremental and batch loads, SCD Type I and II) covering about 80% of new pipelines.

Result

• One governed platform across three environments, with access controlled in Unity Catalog.
• The same review standards applied to every engineer and every AI assistant, regardless of time zone.