Following up on my previous post which highlights different approaches of accessing Oracle Fusion Cloud Apps Data from Databricks, I present in this post details of Approach D, which leverages the Perficient accelerator solution. And this accelerator applies to all Oracle Fusion Cloud applications: ERP, SCM, HCM and CX.

As demonstrated in the previous post, the Perficient accelerator differs from the other approaches in that it has minimal requirements for additional cloud platform services. The other approaches of extracting data efficiently and in a scalable manner require the deployment of additional cloud services such as data integration/replication services and an intermediary data warehouse. With the Perficient accelerator, however, replication is driven by techniques that are solely reliant on native Oracle Fusion and Databricks. The accelerator consists of a Databricks workflow with configurable tasks to handle the end-to-end process of managing data replication from Oracle Fusion into the silver layer of Databricks tables. When deploying the solution, you get access to all underlying python/SQL notebooks that can be further customized based on your needs.

Why consider deploying the Perficient Accelerator?

There are several benefits to deploying this accelerator as opposed to building data replications from Oracle Fusion from the ground up. Built with automation, the solution is future-proof and enables scalability to accommodate evolving data requirements with ease. The diagram below highlights key considerations.

A Closer Look at How Its Done

In the Oracle Cloud: The Perficient solution leverages Oracle BI Cloud Connector (BICC) which is the preferred method of extracting data in bulk from Oracle Fusion while minimizing the impact to the Fusion application itself. Extracted data and metadata is temporarily made available in the OCI Object Storage buckets for downstream processing. Archival of exported data on the OCI (Oracle Cloud Infrastructure) side is also automatically handled, if required, with purging rules.

In the Databricks hosting cloud:

• Hosted in one of: AWS, Azure or GCP, the accelerator’s workflow job and notebooks are deployed in the Databricks workspace. The Databricks delta tables schema, configuration and log files are all hosted within the Databricks Unity Catalog.
• Notebooks leverage parametrized code to programmatically determine which Fusion view objects get replicated through the silver tables.
• The Databricks workflow triggers the data extraction from Oracle Fusion BICC based on a predefined Fusion BICC job. The BICC job determines which objects get extracted.
• Files are then transferred over from OCI to a landing zone object store in the cloud that hosts Databricks.
• Databricks AutoLoader handles the ingestion of data into bronze Live Tables which store historical insert, update and delete operations relevant to the extracted objects.
• Databricks silver Live Tables are then loaded from bronze via a Databricks managed DLT Pipeline. The silver tables are de-duped and represent the same level of data granularity for each Fusion view object as it exists in Fusion.
• Incremental table refreshes are set up automatically leveraging Oracle Fusion object metadata that enables incremental data merges within Databricks. This includes inferring any data deletion from Oracle Fusion and processing deletions through to the silver tables.

Whether starting small with a few tables or looking to easily scale to hundreds and thousands of tables, the Perficient Databricks accelerator for Oracle Fusion data handles the end-to-end workflow orchestration. As a result, you end up spending less time with data integration and focus efforts on business facing analytical data models.

For assistance with enabling data integration between Oracle Fusion Applications and Databricks, reach out to mazen.manasseh@perficient.com.

Connecting to Oracle Fusion Cloud Applications data from external non-Oracle systems, like Databricks, is not feasible for bulk data operations via a direct connection. However, there are several approaches to making Oracle apps data available for consumption from Databricks. What makes this task less straightforward is the fact that Oracle Fusion Cloud Applications and Databricks exist in separate clouds. Oracle Fusion apps (ERP, SCM, HCM, CX) are hosted on Oracle Cloud while Databricks leverages one of AWS, Azure or Google Cloud. Nevertheless, there are several approaches that I will present in this blog on how to access Oracle Apps data from Databricks.

While there are other means of performing this integration than what I present in this post, I will be focusing on:

1. Methods that don’t require 3^rd party tools: The focus here is on Oracle and Databricks technologies or Cloud services.
2. Methods that scale to large number of objects and high data volumes: While there are additional means of Fusion data extraction such as using REST APIs, OTBI, or BI Publisher, these are not recommended methods for handling large bulk data extracts from Oracle Fusion and are therefore not part of this analysis. One or more of these techniques may still be applied though, when necessary, and may co-exist with the approaches discussed in this blog.

The following diagrams summarize four different approaches on how to replicate Oracle Fusion Apps data in Databricks. Each diagram highlights the data flow, and the technologies applied.

• Approach A: Leverages Oracle Autonomous Data Warehouse and an Oracle GoldenGate Replication Deployment
• Approach B: Leverages Oracle Autonomous Data Warehouse and the standard Delta Sharing protocol
• Approach C: Leverages Oracle Autonomous Data Warehouse and a direct JDBC connection from Databricks.
• Approach D: Leverages a Perficient accelerator solution using Databricks AutoLoader and DLT Pipelines. More information is available on this approach here.

Choosing the right approach for your use case is dependent on the objective of performing this integration and the ecosystem of cloud platforms that are applicable to your organization. For guidance on this, you may reach Perficient by leaving a comment in the form below. Our Oracle and Databricks specialists will connect with you and provide recommendations.

Python Optimization: Improve Code Performance

# List (mutable)
my_list = [1, 2, 3]
# Tuple (immutable, faster)
my_tuple = (1, 2, 3)

# Slow list lookup (O(n))
numbers = [1, 2, 3, 4, 5]
print(3 in numbers)  # Yawn... Slow!

# Fast set lookup (O(1))
numbers_set = {1, 2, 3, 4, 5}
print(3 in numbers_set)  # Blink and you'll miss it! ⚡

# Generator (better memory usage)
def squared_numbers(n):
    for i in range(n):
        yield i * i
squares = squared_numbers(1000000)  # No memory explosion! 💥

# Inefficient
for i in range(10000):
    result = expensive_function()  # Ugh! Repeating this is a performance killer 😩
    process(result)

# Optimized
cached_result = expensive_function()  # Call it once and chill 😎
for i in range(10000):
    process(cached_result)

# Traditional loop (meh...)
squares = []
for i in range(10):
    squares.append(i * i)

# Optimized list comprehension (so sleek! 😍)
squares = [i * i for i in range(10)]

# Inefficient (Creates too many temporary strings 🤯)
words = ["Hello", "world", "Python"]
sentence = ""
for word in words:
    sentence += word + " "

# Optimized (Effortless and FAST 💨)
sentence = " ".join(words)

name = "Alice"
age = 25

# Old formatting (Ew 🤢)
print("My name is {} and I am {} years old.".format(name, age))

# Optimized f-string (Sleek & stylish 😎)
print(f"My name is {name} and I am {age} years old.")

import timeit
print(timeit.timeit("sum(range(1000))", number=10000))  # How fast is your code? 🚀

import cProfile
cProfile.run('my_function()')  # Find bottlenecks like a pro! 🔍

import sys
my_list = [1, 2, 3, 4, 5]
print(sys.getsizeof(my_list))  # How big is that object? 🤔

import gc
large_object = [i for i in range(1000000)]
del large_object  # Say bye-bye to memory hog! 👋
gc.collect()  # Cleanup crew 🧹

from multiprocessing import Pool

def square(n):
    return n * n

with Pool(4) as p:  # Use 4 CPU cores 🏎
    results = p.map(square, range(100))

import threading

def print_numbers():
    for i in range(10):
        print(i)

thread = threading.Thread(target=print_numbers)
thread.start()
thread.join()

SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint in many large enterprises. However, SAP has an opaque data model.  There was always a long painful process to do the glue work required to move the data while recognizing no real value was being realized in that intermediate process. This caused a lot of projects to be delayed, fail, or not pursued resulting in a pretty significant lost opportunity cost for the client and a potential loss of trust or confidence in the system integrator. SAP recognized this and partnered with a small handful of companies to enhance and enlarge the scope of their offering. Databricks was selected to deliver bi-directional integration with their Databricks Lakehouse platform. When I heard there was going to be a big announcement, I thought we were going to hear about a new Lakehouse Federation Connector. That would have been great; I’m a fan.

This was bigger.

Technical details are still emerging, so I’m going to try to focus on what I heard and what I think I know. I’m also going to hit on some use cases that we’ve worked on that I think could be directly impacted by this today. I think the most important takeaway for data engineers is that you can now combine SAP with your Lakehouse without pipelines. In both directions. With governance. This is big.

SAP Business Data Cloud

I don’t know much about SAP, so you can definitely learn more here. I want to understand more about the architecture from a Databricks perspective and I was able to find out some information from the Introducing SAP Databricks post on the internal Databricks blog page.

This is when it really sunk in that we were not dealing with a new Lakeflow Connector; SAP Databricks is a native component in the SAP Business Data Cloud and will be sold by SAP as part of their SAP Business Data Cloud offering. It’s not in the diagram here, but you can actually integrate new or existing Databricks instances with SAP Databricks. I don’t want to get ahead of myself, but I would definitely consider putting that other instance of Databricks on another hyperscaler. In my mind, the magic is the dotted line from the blue “Curated context-rich SAP data products” up through the Databricks stack.

Open Source Sharing

The promise of SAP Databricks is the ability to easily combine SAP data with the rest of the enterprise data. In my mind, easily means no pipelines that touch SAP. The diagram we see with the integration point between SAP and Databricks SAP uses Delta Sharing as the underlying enablement technology.

Delta Sharing is an open-source protocol, developed by Databricks and the Linux Foundation, that provides strong governance and security for sharing data, analytics and AI across internal business units, cloud providers and applications. Data remains in its original location with Delta Sharing: you are sharing live data with no replication. Delta Share, in combination with Unity Catalog, allows a provider to grant access to one or more recipients and dictate what data can be seen by those shares using row and column-level security.

Open Source Governance

Databricks leverages Unity Catalog for security and governance across the platform including Delta Share. Unity Catalog offers strong authentication, asset-level access control and secure credential vending to provide a single, unified, open solution for protecting both (semi- & un-)structured data and AI assets. Unity Catalog offers a comprehensive solution for enhancing data governance, operational efficiency, and technological performance. By centralizing metadata management, access controls, and data lineage tracking, it simplifies compliance, reduces complexity, and improves query performance across diverse data environments. The seamless integration with Delta Lake unlocks advanced technical features like predictive optimization, leading to faster data access and cost savings. Unity Catalog plays a crucial role in machine learning and AI by providing centralized data governance and secure access to consistent, high-quality datasets, enabling data scientists to efficiently manage and access the data they need while ensuring compliance and data integrity throughout the model development lifecycle.

Data Warehousing

Databricks is now a first-class Data Warehouse with its Databricks SQL offering. The serverless SQL warehouses have been kind of a game changer for me because they spin up immediately and size elastically. Pro tip: now is a great time to come up with a tagging strategy. You’ll be able to easily connect your BI tool (Tableau, Power BI, etc) to the warehouse for reporting. There are also a lot of really useful AI/BI opportunities available natively now. If you remember in the introduction, I said that I would have been happy had this only been a Lakehouse Federation offering. You still have the ability to take advantage of Federation to discover, query and govern data from Snowflake, Redshift, Salesforce, Teradata and many others all from within a Databricks instance. I’m still wrapping my head around being able to query Salesforce and SAP Data in a notebook inside Databricks inside SAP.

Mosaic AI + Joule

As a data engineer, I was the most excited about zero-copy, bi-directional SAP data flow into Databricks. This is selfish because it solves my problems, but its relatively short-sighted. The integration between SAP and Databricks will likely deliver the most value through Agentic AI. Lets stipulate that I believe that chat is not the future of GenAI. This is not a bold statement; most people agree with me. Assistants like co-pilots represented a strong path forward. SAP thought so, hence Joule. It appears that SAP is leveraging the Databricks platform in general and MosaicAI in particular to provide a next generation of Joule which will be an AI copilot infused with agents.

As a Databricks Champion working for Perficient’s Data Solutions team, I spend most of my time installing and managing Databricks on Azure and AWS. The decision on which cloud provider to use is typically outside my scope since the organization has already made it. However, there are occasions when the client uses both hyperscalers or has not yet moved to the cloud. It is helpful in those situations to advise the client on the advantages and disadvantages of one platform over another from a Databricks perspective. I’m aware that I am skipping over the Google Cloud Platform, but I want to focus on the questions I am actually asked rather than questions that could be asked. I am also not advocating for one cloud provider over another. I am limiting myself to the question of AWS versus Azure from a Databricks perspective.

Advantages of Databricks on Azure

Databricks is a first-party service on Azure, which means it enjoys deep integration with the Microsoft ecosystem. Identity management in Databricks is integrated with Azure Active Directory (AAD) authentication, which can save time and effort in an area I have found difficult in large, regulated organizations. The same applies to deep integration with networking, private links, and Azure compliance frameworks. The value of this integration is amplified if the client also uses some combination of Azure Data Lake Storage (ADLS), Azure Synapse Analytics, or Power BI. The Databricks integration with these products on Azure is seamless. FinOps gets a boost in Azure for companies with an Azure Consumption Commitment (MACC), as Databricks’ costs can be applied against that number. Regarding cost management, Azure spot VMs can be used in some situations to reduce costs. Azure Databricks and ADLS Gen2/Blob Storage are optimized for high throughput, which reduces latency and improves I/O performance.

Disadvantages of Databricks in Azure

Databricks and Azure are tightly integrated within the Microsoft ecosystem. Azure Databricks uses Azure AD, role-based access control (RBAC), and network security groups (NSGs). These dependencies will require additional and sometimes complex configurations if you want to use a hybrid or multi-cloud approach. Some advanced networking configurations require enterprise licensing or additional manual configurations in the Azure Marketplace.

Advantages of Databricks on AWS

Azure is focused on seamless integration with Databricks, assuming the organization is a committed Microsoft shop. AWS takes the approach of providing more dials to tune in exchange for greater flexibility.  Additionally, AWS offers a broad selection of EC2 instance types, Spot Instance options, and scalable S3 storage, which can result in better cost and performance optimization. Finally, AWS has more instance types than Azure, including more options for GPU and memory-optimized workloads. AWS has a more flexible spot pricing model than Azure. VPC Peering, Transit Gateway, and more granular IAM security controls than Azure make AWS a stronger choice for organizations with advanced security requirements and/or organizations committed to multi-cloud or hybrid Databricks deployments. Many advanced features are released in AWS before Azure. Photon is a good example.

Disadvantages of Databricks in AWS

AWS charges for cross-region data transfers, and S3 read/write operations can become costly, especially for data-intensive workloads. This can result in higher networking costs. AWS also has weaker native BI Integration when you compare Tableau on AWS versus PowerBI on Azure.

Conclusion

Databricks is a strong cloud database on all the major cloud providers. If your organization has already committed to a particular cloud provider, Databricks will work. However, I have been asked about the differences between AWS and Azure enough that I wanted to get all my thoughts down in one place. Also, I recommend a multi-cloud strategy for most of our client organizations for Disaster Recovery and Business Continuity purposes.

Contact us to discuss the pros and cons of your planned or proposed Databricks implementation. We can help you navigate the technical complexities that affect security, cost, and BI integrations.

So, DeepSeek just dropped their latest AI models, and while it’s exciting, there are some cautions to consider. Because of the US export controls around advanced hardware, DeepSeek has been operating under a set of unique constraints that have forced them to get creative in their approach. This creativity seems to have yielded real progress in reducing the amount of hardware required for training high-end models in reasonable timeframes and for inferencing off those same models. If reality bears out the claims, this could be a sea change in the monetary and environmental costs of training and hosting LLMs.

In addition to the increased efficiency, DeepSeek’s R1 model is continuing to swell the innovation curve around reasoning models. Models that follow this emerging chain of thought paradigm in their responses, providing an explanation of their thinking first and then summarizing into an answer, are providing a step change in response quality. Especially when paired with RAG and a library of tools or actions in an agentic framework, baking this emerging pattern into the models instead of including it in the prompt is a serious innovation. We’re going to see even more open-source model vendors follow OpenAI and DeepSeek in this.

Key Considerations

One of the key factors in considering the adoption of DeepSeek models will be data residency requirements for your business. For now, self-managed private hosting is the only option for maintaining full US, EU, or UK data residency with these new DeepSeek models (the most common needs for our clients). The same export restrictions limiting the hardware available to DeepSeek have also prevented OpenAI from offering their full services with comprehensive Chinese data residency. This makes DeepSeek a compelling offering for businesses needing an option within China. It’s yet to be seen if the hyperscalers or other providers will offer DeepSeek models on their platforms (Before I managed to get his published, Microsoft made a move and is offering DeepSeek-R1 in Azure AI Foundry).  The good news is that the models are highly efficient, and self-image hosting is feasible and not overly expensive for inferencing with these models. The downside is managing provisioned capacity when workloads can be uneven, which is why pay-per-token models are often the most cost efficient.

We are expecting that these new models and the reduced prices associated with them will have serious downward pressure on per-token costs for other models hosted by the hyperscalers. We’ll be paying specific attention to Microsoft as they are continuing to diversify their offerings beyond OpenAI, especially with their decision to make DeepSeek-R1 available. We also expect to see US-based firms replicate DeepSeek’s successes, especially given that Hugging Face has already started work within their Open R1 project to take the research behind DeepSeek’s announcements and make it fully open source.

What to Do Now

This is a definite leap forward and progress in the direction of what we have long said is the destination—more and smaller models targeted at specific use cases. For now, when looking at our clients, we advise a healthy dose of “wait and see.” As has been the case for the last three years, this technology is evolving rapidly, and we expect there to be further developments in the near future from other vendors. Our perpetual reminder to our clients is that security and privacy always outweigh marginal cost savings in the long run.

The comprehensive FAQ from Stratechery is a great resource for more information.

SAP Databricks is important because convenient access to governed data to support business initiatives is important. Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint in many large enterprises. However, SAP has an opaque data model.  There was always a long painful process to do the glue work required to move the data while recognizing no real value was being realized in that intermediate process. This caused a lot of projects to be delayed, fail, or not pursued resulting in a pretty significant lost opportunity cost for the client and a potential loss of trust or confidence in the system integrator. SAP recognized this and partnered with a small handful of companies to enhance and enlarge the scope of their offering. Databricks was selected to deliver bi-directional integration with their Databricks Lakehouse platform. When I heard there was going to be a big announcement, I thought we were going to hear about a new Lakehouse Federation Connector. That would have been great; I’m a fan.

This was bigger.

Technical details are still emerging, so I’m going to try to focus on what I heard and what I think I know. I’m also going to hit on some use cases that we’ve worked on that I think could be directly impacted by this today. I think the most important takeaway for data engineers is that you can now combine SAP with your Lakehouse without pipelines. In both directions. With governance. This is big.

SAP Business Data Cloud

I don’t know much about SAP, so you can definitely learn more here. I want to understand more about the architecture from a Databricks perspective and I was able to find out some information from the Introducing SAP Databricks post on the internal Databricks blog page.

This is when it really sunk in that we were not dealing with a new Lakeflow Connector; SAP Databricks is a native component in the SAP Business Data Cloud and will be sold by SAP as part of their SAP Business Data Cloud offering. It’s not in the diagram here, but you can actually integrate new or existing Databricks instances with SAP Databricks. I don’t want to get ahead of myself, but I would definitely consider putting that other instance of Databricks on another hyperscaler. In my mind, the magic is the dotted line from the blue “Curated context-rich SAP data products” up through the Databricks stack.

Open Source Sharing

The promise of SAP Databricks is the ability to easily combine SAP data with the rest of the enterprise data. In my mind, easily means no pipelines that touch SAP. The diagram we see with the integration point between SAP and Databricks SAP uses Delta Sharing is the underlying enablement technology.

Delta Sharing is an open-source protocol, developed by Databricks and the Linux Foundation, that provides strong governance and security for sharing data, analytics and AI across internal business units, clouds providers and applications. Data remains in its original location with Delta Sharing: you are sharing live data with no replication. Delta Share, in combination with Unity Catalog, allows a provider to grant access to one or more recipients and dictate what data can be seen by those shares using row and column-level security.

Open Source Governance

Databricks leverages Unity Catalog for security and governance across the platform including Delta Share. Unity Catalog offers strong authentication, asset-level access control and secure credential vending to provide a single, unified, open solution for protecting both (semi- & un-)structured data and AI assets. Unity Catalog offers a comprehensive solution for enhancing data governance, operational efficiency, and technological performance. By centralizing metadata management, access controls, and data lineage tracking, it simplifies compliance, reduces complexity, and improves query performance across diverse data environments. The seamless integration with Delta Lake unlocks advanced technical features like predictive optimization, leading to faster data access and cost savings. Unity Catalog plays a crucial role in machine learning and AI by providing centralized data governance and secure access to consistent, high-quality datasets, enabling data scientists to efficiently manage and access the data they need while ensuring compliance and data integrity throughout the model development lifecycle.

Data Warehousing

Databricks is now a first-class Data Warehouse with its Databricks SQL offering. The serverless SQL warehouses have been kind of a game changer for me because they spin up immediately and size elastically. Pro tip: now is a great time to come up with a tagging strategy. You’ll be able to easily connect your BI tool (Tableau, PowerBI, etc) to the warehouse for reporting. There are also a lot of really useful AI/BI opportunities available natively now. If you remember in the introduction, I said that I would have been happy had this only been a Lakehouse Federation offering. You still have the ability to take advantage of Federation to discover, query and govern data from Snowflake, Redshift, Salesforce, Teradata and many others all from within a Databricks instance. I’m still wrapping my head around being able to query Salesforce and SAP Data in a notebook inside Databricks inside SAP.

Mosaic AI + Joule

As a data engineer, I was the most excited about zero-copy, bi-directional SAP data flow into Databricks. This is selfish because it solves my problems, but its relatively short-sighted. The integration between SAP and Databricks will likely deliver the most value through Agentic AI. Lets stipulate that I believe that chat is not the future of GenAI. This is not a bold statement; most people agree with me. Assistants like co-pilots represented a strong path forward. SAP thought so, hence Joule. It appears that SAP is leveraging the Databricks platform in general and MosaicAI in particular to provide a next generation of Joule which will be an AI copilot infused with agents.

Recently, the news broke that Optimizely acquired Netspring, a warehouse-native analytics platform.

I’ll admit, I hadn’t heard of Netspring before, but after taking a closer look at their website and capabilities, it became clear why Optimizely made this strategic move.

Simplifying Omnichannel Analytics for Real Digital Impact

Netspring is not just another analytics platform. It is focused on making warehouse-native analytics accessible to organizations of all sizes. As businesses gather more data than ever before from multiple sources – CRM, ERP, commerce, marketing automation, offline/retail – managing and analyzing that data in a cohesive way is a major challenge. Netspring simplifies this by enabling businesses to conduct meaningful analytics directly from their data warehouse, eliminating data duplication and ensuring a single source of truth.

By bringing Netspring into the fold, Optimizely has future-proofed its ability to leverage big data for experimentation, personalization, and analytics reporting across the entire Optimizely One platform.

Why Optimizely Acquired Netspring

Netspring brings significant capabilities that make it a best-in-class tool for warehouse-native analytics.

With Netspring, businesses can:

• Run Product Analytics: Understand how users engage with specific products.
• Analyze Customer Journeys: Dive deep into the entire customer journey, across all touchpoints.
• Access Business Intelligence: Easily query key business metrics without needing advanced technical expertise or risking data inconsistency.

This acquisition means that data teams can now query and analyze information directly in the data warehouse, ensuring there’s no need for data duplication or exporting data to third-party platforms. This is especially valuable for large organizations that require data consistency and accuracy.

Ready to capitalize on these new features? Contact Perficient for a complimentary assessment!

The Growing Importance of Omnichannel Analytics

It’s no secret that businesses today are moving away from single analytics platforms. Instead, they are combining data from a wide range of sources to get a holistic view of their performance. It’s not uncommon to see businesses using a combination of tools like Snowflake, Google BigQuery, Salesforce, Microsoft Dynamics, Qualtrics, Google Analytics, and Adobe Analytics.
How?

These tools allow organizations to consolidate and analyze performance metrics across their entire omnichannel ecosystem. The need to clearly measure customer journeys, marketing campaigns, and sales outcomes across both online and offline channels has never been greater. This is where warehouse-native analytics, like Netspring, come into play.

Why You Need an Omnichannel Approach to Analytics & Reporting

Today’s businesses are increasingly reliant on omnichannel analytics to drive insights. Some common tools and approaches include:

• Customer Data Platforms (CDPs): These platforms collect and unify customer data from multiple sources, providing businesses with a comprehensive view of customer interactions across all touchpoints.
• Marketing Analytics Tools: These tools help companies measure the effectiveness of their marketing campaigns across digital, social, and offline channels. They ensure you have a real-time view of campaign performance, enabling better decision-making.
• ETL Tools (Extract, Transform, Load): ETL tools are critical for moving data from various systems into a data warehouse, where it can be analyzed as a single, cohesive dataset.

The combination of these tools allows businesses to pull all relevant data into a central location, giving marketing and data teams a 360-degree view of customer behavior. This not only maximizes the return on investment (ROI) of marketing efforts but also provides greater insights for decision-making.

Navigating the Challenges of Omnichannel Analytics

While access to vast amounts of data is a powerful asset, it can be overwhelming. Too much data can lead to confusion, inconsistency, and difficulties in deriving actionable insights. This is where Netspring shines – its ability to work within an organization’s existing data warehouse provides a clear, simplified way for teams to view and analyze data in one place, without needing to be data experts. By centralizing data, businesses can more easily comply with data governance policies, security standards, and privacy regulations, ensuring they meet internal and external data handling requirements.

AI’s Role in Omnichannel Analytics

Artificial intelligence (AI) plays a pivotal role in this vision. AI can help uncover trends, patterns, and customer segmentation opportunities that might otherwise go unnoticed. By understanding omnichannel analytics across websites, mobile apps, sales teams, customer service interactions, and even offline retail stores, AI offers deeper insights into customer behavior and preferences.

This level of advanced reporting enables organizations to accurately measure the impact of their marketing, sales, and product development efforts without relying on complex SQL queries or data teams. It simplifies the process, making data-driven decisions more accessible.

Additionally, we’re looking forward to learning how Optimizely plans to leverage Opal, their smart AI assistant, in conjunction with the Netspring integration. With Opal’s capabilities, there’s potential to further enhance data analysis, providing even more powerful insights across the entire Optimizely platform.

What’s Next for Netspring and Optimizely?

Right now, Netspring’s analytics and reporting capabilities are primarily available for Optimizely’s experimentation and personalization tools. However, it’s easy to envision these features expanding to include content analytics, commerce insights, and deeper customer segmentation capabilities. As these tools evolve, companies will have even more ways to leverage the power of big data.

A Very Smart Move by Optimizely

Incorporating Netspring into the Optimizely One platform is a clear signal that Optimizely is committed to building a future-proof analytics and optimization platform. With this acquisition, they are well-positioned to help companies leverage omnichannel analytics to drive business results.

At Perficient, an Optimizely Premier Platinum Partner, we’re already working with many organizations to develop these types of advanced analytics strategies. We specialize in big data analytics, data science, business intelligence, and artificial intelligence (AI), and we see firsthand the value that comprehensive data solutions provide. Netspring’s capabilities align perfectly with the needs of organizations looking to drive growth and gain deeper insights through a single source of truth.

Ready to leverage omnichannel analytics with Optimizely?

Start with a complimentary assessment to receive tailored insights from our experienced professionals.

Connect with a Perficient expert today!
Contact Us

At Dreamforce 2024, Perficient explored the integration of Databricks and Salesforce Data Cloud, focusing on an insurance industry use case. This session showcased data processing, customer engagement, and AI-driven insights, offering real-world value to enterprises.

Here’s a comprehensive recap of the session, highlighting the key takeaways and technical depth discussed.

Speakers 

Two of Perficient’s top experts, Eric Walk (Director, Data Strategy Consulting) and Johnathon Rademacher, JR (Principal, Salesforce Global Operations), led the session.

Both speakers brought years of expertise to the discussion, focusing on helping enterprises become more data-driven with AI and cloud-based technologies.

Business Scenario: Insurance with Real-Time Customer Data 

The session featured a real-world auto insurance scenario. The story centered on Roberta, a customer of Acme Insurance, and her son Ricky, who is flagged for risky driving behaviors. Acme’s use of telematics and a safe-driving tracker, combined with real-time insights from Databricks and Data Cloud, allowed Acme’s customer service team to proactively engage the family.

This outreach not only enhances customer satisfaction but also offers potential savings on insurance premiums.

Technical Integration: Data Cloud and Databricks

Attendees discovered how these technologies work together to:

• Process massive data pipelines in real time, leveraging Databricks’ Lakehouse architecture.
• Use ACID transactions and data governance to maintain data integrity while benefiting from the flexibility of data lakes.
• Drive personalized customer experiences with AI and machine learning models that can be quickly deployed using the Databricks and Salesforce Data Cloud platforms.

Key Features of the Integration 

• Lakehouse Architecture: This hybrid system combines data lakes and warehouses to allow for both structured and unstructured data, enhancing scalability and flexibility.
• Data Harmonization: The integration unifies data from various sources, providing a consistent view across the organization.
• AI Integration with Salesforce: With tools like Einstein GPT, the combined platform makes it easier to derive actionable insights from data, improving both sales and service operations.

AI and Data Cloud Advancements 

Eric and J.R. highlighted Salesforce’s paradigm shift, focusing on how the combination of Data + AI + CRM is set to transform customer relationship management. This includes Salesforce’s Einstein GPT, which leverages large language models and real-time data to automate tasks, deliver insights, and improve customer experience.

The addition of Databricks’ data processing capabilities allows for sophisticated data modeling and activation, giving enterprises the power to engage customers more meaningfully.

Technical Breakdown: Demo Architecture

A major part of the session included a demo showcasing how Databricks and Data Cloud work together. The demo architecture’s key components included:

• Data Ingestion: Bringing in large volumes of telemetry and customer data in real time.
• Data Harmonization: Consolidating disparate data into unified customer profiles, enabling a 360-degree view of the customer.
• Actionable Insights: Using predictive analytics to drive real-time customer engagement, including proactive alerts for risky driving behaviors.

The architecture leveraged Salesforce’s Service Cloud to provide customer support teams with the right tools to manage customer interactions. This holistic platform not only simplifies data management but also accelerates the time it takes to extract actionable insights, making it a key tool for data-driven companies.

PACE Framework 

A significant part of Perficient’s service offerings is their P.A.C.E. Framework, designed to operationalize AI responsibly:

• Policies: Setting guidelines for AI usage.
• Advocacy: Promoting AI adoption.
• Enablement: Offering tools and resources for AI deployment.
• Controls: Ensuring governance and risk management of AI systems.

Final Takeaways: The Future of Data and AI in Customer Engagement

The session closed with an emphasis on the future possibilities of combining Databricks and Data Cloud. Businesses, especially in industries like insurance, can now engage with customers in real time, leveraging AI to deliver personalized and proactive experiences.

Much like how peanut butter and jelly combine to create a classic sandwich, the integration of Databricks and Data Cloud creates a powerful combination that’s greater than the sum of its parts.

Perficient + Salesforce 

We are a Salesforce Summit Partner with more than two decades of experience delivering digital solutions in the manufacturing, automotive, healthcare, financial services, and high-tech industries. Our team has deep expertise in all Salesforce Clouds and products, artificial intelligence, DevOps, and specialized domains to help you reap the benefits of implementing Salesforce solutions.

Missed Dreamforce? 

Don’t worry! Schedule some time with us and let our experts fill you in. And stay tuned to our Salesforce blog for all our post-conference insights.

Databricks has recognized David Callaghan as a Partner Champion. As the first Perficient colleague to receive inclusion in the program, David is paving the way for others to get their footing with the partner.

Program Overview

To be a Databricks Partner champion, one must:

1. Display Thought Leadership
2. Harness Technical Expertise
3. Become Community Leader
4. Demonstrate Innovation

Individuals who show promise and interest are approached with an intensive multi-step program that leads to becoming an official Databricks Partner Champion.  This program recognizes the best and brightest on the Databricks platform and its capabilities. It helps to deepen a preexisting understanding of the platform by offering exclusive trainings and limitless growth potential for individuals daring enough to commit to the program. Those who advance through the rigor are equipped with knowledge like no other and are capable of demonstrating advanced understanding of the Lakehouse and Databricks.

How Did I Get Here?

David Callaghan, a Senior Solutions Architect, is on a mission to bring trusted data to complex regulated industries and has been deep in the Databricks weeds developing innovative solutions that are widely applicable. He participated in a Databricks Architect Panel and presented some of Perficient’s Databricks Accelerators and as a result was approached by program leadership to take the next steps to receive the Partner Champion recognition and be inducted into the program. David has since aided in the development of the Perficient Migration Factory, Databricks Brickbuilder solution and plans to leverage his expertise to shepherd a global team of Partner Champions at Perficient.

“I would like to bolster a global team of Databricks Partner Champions and establish a talent pool that brings diverse strengths to Databricks data and analytics platform to deliver value across enterprises and industries through training and mentorship by Perficient’s Databricks Center of Excellence.”

– David Callaghan, Senior Solutions Architect

What’s Next?

David is spearheading the charge of the development of a new life sciences solution related to one of Perficient’s most successful engagements with Databricks in this space. Our Migration Factory is a unique approach to migrating legacy data platforms into the Lakehouse and has set the tone for new innovative Brickbuilder solutions to be developed by Perficient’s experts.

More About our Partnership

At Perficient, we are a trusted Databricks consulting partner and our passion for creating custom data engineering, data science and advanced analytics knows no limits. With over 50 Databricks certified consultants, we build end-to-end solutions that empower our customers to gain more value from their data.

Learn more about the practice here.

In the bustling city of Tech Ville, where data flows like rivers and companies thrive on insights, there lived a dedicated data engineer named Tara. With over five years of experience under her belt, Tara had navigated the vast ocean of data engineering, constantly learning, and evolving with the ever-changing tides.
One crisp morning, Tara was called into a meeting with the analytics team at the company she worked for. The team had been facing significant delays in processing their massive datasets, which was hampering their ability to generate timely insights. Tara’s mission was clear: optimize the performance of their Apache Spark jobs to ensure faster and more efficient data processing.
The Analysis
Tara began her quest by diving deep into the existing Spark jobs. She knew that to optimize performance, she first needed to understand where the bottlenecks were. she started with the following steps:
1. Reviewing Spark UI: Tara meticulously analyzed the Spark UI for the running jobs, focusing on stages and tasks that were taking the longest time to execute. she noticed that certain stages had tasks with high execution times and frequent shuffling.

2. Examining Cluster Resources: she checked the cluster’s resource utilization. The CPU and memory usage graphs indicated that some of the executor nodes were underutilized while others were overwhelmed, suggesting an imbalance in resource allocation.

                                           
The Optimization Strategy
Armed with this knowledge, Tara formulated a multi-faceted optimization strategy:

1. Data Serialization: she decided to switch from the default Java serialization to Kryo serialization, which is faster and more efficient.
conf = SparkConf().set(“spark.serializer”, “org.apache.spark.serializer.KryoSerializer”)

2. Tuning Parallelism: Tara adjusted the level of parallelism to better match the cluster’s resources. By setting `spark.default.parallelism` and `spark.sql.shuffle.partitions` to a higher value, she aimed to reduce the duration of shuffle operations.
conf = conf.set(“spark.default.parallelism”, “200”)
conf = conf.set(“spark.sql.shuffle.partitions”, “200”)
3. Optimizing Joins: she optimized the join operations by leveraging broadcast joins for smaller datasets. This reduced the amount of data shuffled across the network.
small_df = spark.read.parquet(“hdfs://path/to/small_dataset”)
large_df = spark.read.parquet(“hdfs://path/to/large_dataset”)
small_df_broadcast = broadcast(small_df)
result_df = large_df.join(small_df_broadcast, “join_key”)

4. Caching and Persisting: Tara identified frequently accessed DataFrames and cached them to avoid redundant computations.
df = spark.read.parquet(“hdfs://path/to/important_dataset”).cache()
df.count() – Triggering cache action

5. Resource Allocation: she reconfigured the cluster’s resource allocation, ensuring a more balanced distribution of CPU and memory resources across executor nodes.
conf = conf.set(“spark.executor.memory”, “4g”)
conf = conf.set(“spark.executor.cores”, “2”)
conf = conf.set(“spark.executor.instances”, “10”)

The Implementation
With the optimizations planned, Tara implemented the changes and closely monitored their impact. she kicked off a series of test runs, carefully comparing the performance metrics before and after the optimizations. The results were promising:
– The overall job execution time reduced by 40%.
– The resource utilization across the cluster was more balanced.
– The shuffle read and write times decreased significantly.
– The stability of the jobs improved, with fewer retries and failures.
The Victory
Tara presented the results to the analytics team and the management. The improvements not only sped up their data processing pipelines but also enabled the team to run more complex analyses without worrying about performance bottlenecks. The insights were now delivered faster, enabling better decision-making, and driving the company’s growth.
The Continuous Journey
While Tara had achieved a significant milestone, she knew that the world of data engineering is ever evolving. she remained committed to learning and adapting, ready to tackle new challenges and optimize further as the data landscape continued to grow.
And so, in the vibrant city of Tech Ville, Tara’s journey as a data engineer continued, navigating the vast ocean of data with skill, knowledge, and an unquenchable thirst for improvement.

Dancing with Data

It was but a fortnight into 2024 AC (After Conquest) when the great council gathered to decide who would succeed Perficient’s 2023 Data & AI Summit attendees. Many claims were heard, but only a few were considered.  The council was assembled to prevent a war from being fought over the succession, for all knew the only thing that could tear down the house of Perficient, was itself.

As interesting as it would be to see a hypothetical war fought over annual conference attendees, it is certainly a stretch of the truth for this House of the Dragon fan. Databricks’ Data and AI Summit will be held on June 10^th-13^th at the Moscone Center in San Francisco, CA. Shortly after on June 16^th, this author will be journeying back to Westeros as season two of HBO’s House of the Dragon will be returning to screens.

Meet the Heirs

After much deliberation, the names of the newest attendee council were proclaimed. Five individuals will represent Perficient at this year’s Data and AI Summit.

• Grand Chancellor; Senior Vice President & Data Solutions GM, Santhosh Nair
• Master of Data; Databricks Practice Director, Nick Passero
• Master of Partnerships; Alliance Manager, Kyla Faust
• Master of Coin; Portfolio Specialist, Brian Zielinski
• Master of Commerce; Portfolio Specialist, Al Muse

The council of heirs will set off for San Francisco, California, for a full week of cutting-edge content, networking, and collaboration. The Perficient Council will have the opportunity to discover new use cases and capabilities on the Databricks platform that can be brought back to their loyal subjects (customers) to strengthen Perficient’s ability to serve and deliver quality solutions.

Decrees of the Council

“The hour of Data and AI Summit approaches, and I am afire with anticipation. The secrets of data shall unfold before us, like the ancient tomes of Old Valyria. Let the banners of Perficient and Databricks fly high, for this shall be a gathering remembered in the annals of our time!”

Databricks Practice Director,  Nick Passero

“I am honored to stand among such great minds where the arcane arts of data and AI are revealed—’twill be a journey more thrilling than any battle. Knowledge shall flow as freely as the rivers of the Trident.”

Alliance Manager, Kyla Faust

“The Data and AI Summit approaches, and my excitement burns brighter than a dragon’s flame. This gathering shall echo through time as a beacon of innovation and power.”

Portfolio Specialist, Brian Zielinski

See you there!

The Perficient council of heirs would love to meet you if you will be at this year’s conference! Please reach out to Kyla Faust to organize a meeting with the team.

Check out more information about the Perficient, Databricks practice here.