Deletion Vectors will be enabled by default in Delta Live Tables (DLTs) for materialized views and streaming tables starting April 28, 2025. Predictive Optimization for DLT maintenance will also be enabled by default. This could provide both cost savings and performance improvements. Our Databricks Practice holds FinOps as a core architectural tenet, but sometimes compliance […]
Blogs from this Author
Unlocking the Future of Enterprise AI: Databricks announces Anthropic Partnership
The recent strategic partnership between Databricks and Anthropic is a big step forward for enabling enterprises to build, deploy, and govern AI agents that reason over proprietary data with accuracy, security, and governance. The landscape of enterprise AI is evolving rapidly, and we’re excited to share how our practice is positioned to help businesses maximize […]
Delta Live Tables and Great Expectations: Better Together
Modern data platforms like Databricks enable organizations to process massive volumes of batch and streaming data—but scaling reliably requires more than just compute power. It demands data observability: the ability to monitor, validate, and trace data through its lifecycle. This blog compares two powerful tools—Delta Live Tables and Great Expectations—that bring observability to life in […]
How Automatic Liquid Clustering Supports Databricks FinOps at Scale
Perficient has a FinOps mindset with Databricks, so the Automatic Liquid Clustering announcement grabbed my attention. I’ve mentioned Liquid Clustering before when discussing the advantages of Unity Catalog beyond governance use cases. Unity Catalog: come for the data governance, stay for the predictive optimization. I am usually a fan of being able to tune the dials […]
SAP and Databricks: Better Together
SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint […]
Databricks on Azure versus AWS
As a Databricks Champion working for Perficient’s Data Solutions team, I spend most of my time installing and managing Databricks on Azure and AWS. The decision on which cloud provider to use is typically outside my scope since the organization has already made it. However, there are occasions when the client uses both hyperscalers or […]
Optimizing Costs and Performance in Databricks: A FinOps Approach
As organizations increasingly rely on Databricks for big data processing and analytics, managing costs and optimizing performance become crucial for maximizing ROI. A FinOps strategy tailored to Databricks can help teams strike the right balance between cost control and efficient resource utilization. Below, we outline key practices in cluster management, data management, query optimization, coding, […]
SAP and Databricks: Better Together
SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint […]
Integrate Salesforce and Databricks
90% of Fortune 500 companies use Salesforce as their Customer Relations Management tool. I have ingested data from Salesforce into almost every database using almost every ETL tool. Every integration tool out there has a Salesforce connector; Salesforce even owns Mulesoft. The integration always worked, but it was rarely smooth. Its just something that you […]
Unity Catalog, the Well-Architected Lakehouse and Operational Excellence
I have written about the importance of migrating to Unity Catalog as an essential component of your Data Management Platform. Any migration exercise implies movement from a current to a future state. A migration from the Hive Metastore to Unity Catalog will require planning around workspaces, catalogs and user access. This is also an opportunity […]
Unity Catalog, the Well-Architected Lakehouse and Performance Efficiency
I have written about the importance of migrating to Unity Catalog as an essential component of your Data Management Platform. Any migration exercise implies movement from a current to a future state. A migration from the Hive Metastore to Unity Catalog will require planning around workspaces, catalogs and user access. This is also an opportunity […]
Unity Catalog, the Well-Architected Lakehouse and Cost Optimization
I have written about the importance of migrating to Unity Catalog as an essential component of your Data Management Platform. Any migration exercise implies movement from a current to a future state. A migration from the Hive Metastore to Unity Catalog will require planning around workspaces, catalogs and user access. This is also an opportunity […]