David Callaghan – Senior Solutions Architect

Business users don’t use notebooks. Full stop. And for that reason, most organizations don’t have business users accessing the Databricks UI. This has always been a fundamental flaw in Databricks’ push to democratize data and AI. This disconnect is almost enshrined in the medallion architecture: Bronze is for system accounts, data scientists with notebooks use […]

Databricks recently announced support for MLflow 3.0, which features a range of enhancements that redefine model management for enterprises. Integrated seamlessly into Databricks, MLflow is an open-source platform designed to manage the complete machine learning lifecycle. It provides tools to track experiments, package code into reproducible runs, and share and deploy models. With the launch […]

Model Context Protocol (MCP) is reshaping the way AI agents interface with data and tools, providing a robust framework for standardization and interoperability. As AI continues to permeate business landscapes, MCP offers particular advantages in creating scalable, efficient AI systems. This blog explores what MCP is, its role in the AI landscape, and focuses on […]

“Clean rooms” have emerged as a pivotal data sharing innovation with both Databricks and Snowflake providing enterprise alternatives. Clean rooms are secure environments designed to allow multiple parties to collaborate on data analysis without exposing sensitive details of data. They serve as a sandbox where participants can perform computations on shared datasets while keeping raw […]

I’ve been coding in notebooks for so long, I forgot how much I missed a nice, deployed application. I also didn’t realize how this was limiting my solution space. Then I started working with Databricks Apps. Databricks Apps are designed to extend the functionality of the Databricks platform, providing users with enriched features and capabilities […]

Databricks announced a full, free version of the platform at the Data and AI Summit. While the Free Edition is targeted to students and hobbyists, I also see opportunities where enterprise architects can effectively evangelize Databricks without going through Procurement for a license. Choosing the right platform to manage, analyze, and extract insights from massive […]

Lakebase is Databricks‘ OLTP database and the latest member of its ML/AI offering. Databricks has incorporated various components to support its AI platform, including data components. The Feature Store has been available for some time as a governed, centralized repository that manages machine learning features throughout their lifecycle. Mosaic AI Vector Search is a vector […]

Several breakthrough announcements emerged at DAIS 2025, but the Lakeflow updates around building robust pipelines had the most immediate impact on my current code. Specifically, I can now see a clear path to persisting SCD2 (Slowly Changing Dimension Type 2) tables in the silver layer from mutable data sources. If this sentence resonates with you, […]

By providing AI functions for SQL analysts, Databricks continues to integrate data, ML, and AI across its platform. AI Functions in Databricks SQL are pre-built, easy-to-use functions that incorporate machine learning models into SQL queries. These functions enable data analysts and engineers to leverage AI capabilities without the need for extensive machine learning expertise. By […]

Metadata contention in Unity Catalog can occur in high-throughput Databricks environments, slowing down user queries and impacting performance across the platform. Our Finops strategy shifts left on performance. However, we have found scenarios where clients are still experiencing query slowdowns intermittently and even on optimized queries. As our client’s lakehouse footprint grows, we are seeing […]

Achieving end-to-end lineage in Databricks while allowing external users to access raw data can be a challenging task. In Databricks, leveraging Unity Catalog for end-to-end lineage is a best practice. However, enabling external users to access raw data while maintaining security and lineage integrity requires a well-thought-out architecture. This blog outlines a reference architecture to […]

I wanted to discuss the top 5 mistakes that make your Databricks queries slow as a prequel to some of my FinOps blogs. Premature optimization may or may be the root of all evil, but we can all agree optimization without a solid foundation is not an effective use of time and resources. Predictive optimization […]