Skip to main content

Posts Tagged ‘data pipeline optimization’

Pipelines@1x.jpg

Lakeflow: Revolutionizing SCD2 Pipelines with Change Data Capture (CDC)

Several breakthrough announcements emerged at DAIS 2025, but the Lakeflow updates around building robust pipelines had the most immediate impact on my current code. Specifically, I can now see a clear path to persisting SCD2 (Slowly Changing Dimension Type 2) tables in the silver layer from mutable data sources. If this sentence resonates with you, […]

Istock 2163867912

Top 5 Mistakes That Make Your Databricks Queries Slow (and How to Fix Them)

I wanted to discuss the top 5 mistakes that make your Databricks queries slow as a prequel to some of my FinOps blogs. Premature optimization may or may be the root of all evil, but we can all agree optimization without a solid foundation is not an effective use of time and resources. Predictive optimization […]