Integrating SAP and Databricks has typically required a lot of glue. Set up the SAP Data Hub environment, connect to the SAP data, set up a pipeline with Pipeline Modeler, configure the Streaming Analytics Service, setup Kafka or MQTT and receive the streaming data in Databricks with Spark Streaming. Most of these intermediate steps required […]
David Callaghan –
As a solutions architect with Perficient, I bring twenty years of development experience and I'm currently hands-on with Hadoop/Spark, blockchain and cloud, coding in Java, Scala and Go. I'm certified in and work extensively with Hadoop, Cassandra, Spark, AWS, MongoDB and Pentaho. Most recently, I've been bringing integrated blockchain (particularly Hyperledger and Ethereum) and big data solutions to the cloud with an emphasis on integrating Modern Data produces such as HBase, Cassandra and Neo4J as the off-blockchain repository.
Connect with David
Blogs from this Author
Real-time Data Processing: Databricks vs Flink
Real-time data processing is a critical need for modern-day businesses. It involves processing data as soon as it is generated to derive insights and take immediate actions. Databricks Streaming and Apache Flink are two popular stream processing frameworks that enable developers to build real-time data pipelines, applications and services at scale. In this article, we […]
Accelerate and Scale your Event Driven Architecture with GridGain
Are you looking for a way to accelerate and scale your Event Driven Architecture in the cloud? GridGain is here to help. GridGain, built on top of Apache Ignite, is a comprehensive in-memory computing platform that provides distributed caching, messaging, and compute capabilities, with enterprise-grade support. With its performance capabilities, it can increase the overall […]
Harden Databricks with Immuta’s Policy-As-Code Framework
Databricks Databricks provides a powerful, spark-centric, cloud-based analytics platform that enables users to rapidly process, transform and explore data. However, its preconfigured security can be insufficient in regulating or monitoring confidential information due to the flexibility it offers. This can be of particular concern to highly regulated enterprise, such a financial and health-care companies. Policy-as-code […]
Next-Generation Data Cleanrooms with Delta Sharing
Data-driven companies are finding more and more use cases where their internal data could be supplemented with external datasets to deliver more business value. At the same time, there are legitimate data privacy concerns that need to be addressed, particularly among regulated enterprises in the financial and healthcare sector. There are opportunities here for a […]
Delta Sharing for Modern Secure Data Sharing
Delta Lake is an open-source framework under the Linux Foundation used to build Lakehouse architectures. A new project is Delta Sharing, which is an open protocol for secure real-time exchange of large datasets. Databricks provides production-grade implementations of the projects under delta-io, including Databricks Delta Sharing. Understanding the open-source foundation of the enterprise offering can […]
Beyond Encryption: Implementing anonymity algorithms
Businesses and organizations now hold more personal information than ever before. Storing a lot of data may be useful in a variety of ways, such as reporting and analytics, which might expose PII that is linked to the data being analyzed. When data is being transmitted or stored, encryption is useful for protecting it, whereas […]
Beyond Encryption: Protect sensitive data using t-closeness
Businesses and organizations now hold more personal information than ever before. Storing a lot of data may be useful in a variety of ways, such as reporting and analytics, which might expose PII that is linked to the data being analyzed. When data is being transmitted or stored, encryption is useful for protecting it, whereas […]
Beyond Encryption: Protect sensitive data using l-diversity
Businesses and organizations now hold more personal information than ever before. Storing a lot of data may be useful in a variety of ways, such as reporting and analytics, which might expose PII that is linked to the data being analyzed. When data is being transmitted or stored, encryption is useful for protecting it, whereas […]
How will the sale of Watson affect healthcare?
IBM is selling their AI for healthcare, known as Watson Health, to Francisco Partners for an estimated $1 billion. Watson Health is an artificial intelligence system created by IBM. The system is designed to work with healthcare providers and pharmaceutical companies to help improve the efficacy of treatments and reduce healthcare costs. Francisco Partners has […]
cURL to get a JSON makeover
cURL is frequently used by developers working with REST API’s to send and receive data using JSON notation. This has been a common pattern for years, but it has never been seamless. There have been a number of times when I’ve been trying to get a JSON payload to work against an endpoint for a […]
Understanding Searchable Encryption in the Cloud
Many companies struggle with storing PII in the cloud. When you store sensitive data in the cloud, it’s critical to guarantee that it remains private. Encrypting the data before sending it to the cloud storage server is one approach to do this. This will protect your information and ensure that no one can access it. […]