Posts Tagged ‘Databricks’

Metadata contention in Unity Catalog can occur in high-throughput Databricks environments, slowing down user queries and impacting performance across the platform. Our Finops strategy shifts left on performance. However, we have found scenarios where clients are still experiencing query slowdowns intermittently and even on optimized queries. As our client’s lakehouse footprint grows, we are seeing […]

Achieving end-to-end lineage in Databricks while allowing external users to access raw data can be a challenging task. In Databricks, leveraging Unity Catalog for end-to-end lineage is a best practice. However, enabling external users to access raw data while maintaining security and lineage integrity requires a well-thought-out architecture. This blog outlines a reference architecture to […]

Deletion Vectors will be enabled by default in Delta Live Tables (DLTs) for materialized views and streaming tables starting April 28, 2025. Predictive Optimization for DLT maintenance will also be enabled by default. This could provide both cost savings and performance improvements. Our Databricks Practice holds FinOps as a core architectural tenet, but sometimes compliance […]

Modern data platforms like Databricks enable organizations to process massive volumes of batch and streaming data—but scaling reliably requires more than just compute power. It demands data observability: the ability to monitor, validate, and trace data through its lifecycle. This blog compares two powerful tools—Delta Live Tables and Great Expectations—that bring observability to life in […]

Perficient has a FinOps mindset with Databricks, so the Automatic Liquid Clustering announcement grabbed my attention. I’ve mentioned Liquid Clustering before when discussing the advantages of Unity Catalog beyond governance use cases. Unity Catalog: come for the data governance, stay for the predictive optimization. I am usually a fan of being able to tune the dials […]

Following up on my previous post which highlights different approaches of accessing Oracle Fusion Cloud Apps Data from Databricks, I present in this post details of Approach D, which leverages the Perficient accelerator solution. And this accelerator applies to all Oracle Fusion Cloud applications: ERP, SCM, HCM and CX. As demonstrated in the previous post, […]

Connecting to Oracle Fusion Cloud Applications data from external non-Oracle systems, like Databricks, is not feasible for bulk data operations via a direct connection. However, there are several approaches to making Oracle apps data available for consumption from Databricks. What makes this task less straightforward is the fact that Oracle Fusion Cloud Applications and Databricks […]

SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint […]

As organizations increasingly rely on Databricks for big data processing and analytics, managing costs and optimizing performance become crucial for maximizing ROI. A FinOps strategy tailored to Databricks can help teams strike the right balance between cost control and efficient resource utilization. Below, we outline key practices in cluster management, data management, query optimization, coding, […]

SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint […]

At Dreamforce 2024, Perficient explored the integration of Databricks and Salesforce Data Cloud, focusing on an insurance industry use case. This session showcased data processing, customer engagement, and AI-driven insights, offering real-world value to enterprises. Here’s a comprehensive recap of the session, highlighting the key takeaways and technical depth discussed. Speakers  Two of Perficient’s top […]

For more than a decade, Salesforce CEO Marc Benioff has championed the concept of shoshin—a “beginner’s mind” that allows for continuous innovation and reinvention. As the company now embarks on its “hard pivot” to Agentforce, this philosophy takes on new relevance.   Agentforce isn’t just an extension of Salesforce’s AI capabilities—it’s a reimagining of what AI […]