Skip to main content

Posts Tagged ‘data governance’

Istock 1536191188

Avoiding Metadata Contention in Unity Catalog

Metadata contention in Unity Catalog can occur in high-throughput Databricks environments, slowing down user queries and impacting performance across the platform. Our Finops strategy shifts left on performance. However, we have found scenarios where clients are still experiencing query slowdowns intermittently and even on optimized queries. As our client’s lakehouse footprint grows, we are seeing […]

Why Data Governance Is A Must

Why Do Organizations Need Data Governance?

A well-known fact about data is that it is a crucial Asset in an organization when managed appropriately. Data governance helps organizations manage data appropriately. Some customers say data governance is an optional best practice but not a mandatory implementation strategy. Then, ask your customer a few questions: Is your data reliable or trustworthy? Is your […]

Construction Concept. Residential Building Drawings And Architectural Model,

End-to-End Lineage and External Raw Data Access in Databricks

Achieving end-to-end lineage in Databricks while allowing external users to access raw data can be a challenging task. In Databricks, leveraging Unity Catalog for end-to-end lineage is a best practice. However, enabling external users to access raw data while maintaining security and lineage integrity requires a well-thought-out architecture. This blog outlines a reference architecture to […]

delete

Deletion Vectors in Delta Live Tables: Identifying and Remediating Compliance Risks

Deletion Vectors will be enabled by default in Delta Live Tables (DLTs) for materialized views and streaming tables starting April 28, 2025. Predictive Optimization for DLT maintenance will also be enabled by default. This could provide both cost savings and performance improvements. Our Databricks Practice holds FinOps as a core architectural tenet, but sometimes compliance […]

Istock 179133772

How Automatic Liquid Clustering Supports Databricks FinOps at Scale

Perficient has a FinOps mindset with Databricks, so the Automatic Liquid Clustering announcement grabbed my attention. I’ve mentioned Liquid Clustering before when discussing the advantages of Unity Catalog beyond governance use cases. Unity Catalog: come for the data governance, stay for the predictive optimization. I am usually a fan of being able to tune the dials […]

Handshake

SAP and Databricks: Better Together

SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint […]

Handshake

SAP and Databricks: Better Together

SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint […]

Young developers working together, programming.

A New Era of AI Agents in the Enterprise?

In a move that has sparked intense discussion across the enterprise software landscape, Klarna announced its decision to drop both Salesforce Sales Cloud and Workday, replacing these industry-leading platforms with its own AI-driven tools. This announcement, led by CEO Sebastian Siemiatkowski, may signal a paradigm shift toward using custom AI agents to manage critical business […]

Ai, Machine Learning, Hands Of Robot And Human Touch On Big Data Network, Brain Data Creative In Light Bulb, Science And Artificial Intelligence Technology, Innovation For Futuristic.

Agentic AI: The New Frontier in GenAI

In the rapidly evolving landscape of digital transformation, businesses are constantly seeking innovative ways to enhance their operations and gain a competitive edge. While Generative AI (GenAI) has been the hot topic since OpenAI introduced ChatGPT to the public in November 2022, a new evolution of the technology is emerging that promises to revolutionize how […]

Istock 2148575950

Maximize Your Data Management with Unity Catalog

Databricks Unity Catalog is a unified and open governance solution for data and AI, built into the Databricks Data Intelligence Platform. Unity Catalog offers a comprehensive solution for enhancing data governance, operational efficiency, and technological performance. By centralizing metadata management, access controls, and data lineage tracking, it simplifies compliance, reduces complexity, and improves query performance […]

1632183332264

Risk Management Data Strategy – Insights from an Inquisitive Overseer

We are witnessing a sea-change in the way data is managed by banks and financial institutions all over the world. Data being commoditized and, in some cases, even monetized by banks is the order of the day. Though this seems to be at a stage where some more push is required in terms of adoption […]

Istock 960790462 (1)

Data Lake Governance with Tagging in Databricks Unity Catalog

The goal of Databricks Unity Catalog is to provide centralized security and management to data and AI assets across the data lakehouse. Unity Catalog provides fine-grained access control for all the securable objects in the lakehouse; databases, tables, files and even models. Gone are the limitations of the Hive metadata store. The Unity Catalog metastore […]

Load More