Analytics Articles / Blogs / Perficient https://blogs.perficient.com/category/services/data-intelligence/analytics/ Expert Digital Insights Wed, 19 Nov 2025 15:08:32 +0000 en-US hourly 1 https://blogs.perficient.com/files/favicon-194x194-1-150x150.png Analytics Articles / Blogs / Perficient https://blogs.perficient.com/category/services/data-intelligence/analytics/ 32 32 30508587 Sitecore Content SDK: What It Offers and Why It Matters https://blogs.perficient.com/2025/11/19/sitecore-content-sdk-what-it-offers-and-why-it-matters/ https://blogs.perficient.com/2025/11/19/sitecore-content-sdk-what-it-offers-and-why-it-matters/#respond Wed, 19 Nov 2025 15:08:05 +0000 https://blogs.perficient.com/?p=388367

Sitecore has introduced the Content SDK for XM Cloud-now Sitecore AI to streamline the process of fetching content and rendering it on modern JavaScript front-end applications. If you’re building a website on Sitecore AI, the new Content SDK is the modern, recommended tool for your development team.

Think of it as a specialized, lightweight toolkit built for one specific job: getting content from Sitecore AI and displaying it on your modern frontend application (like a site built with Next.js).

Because it’s purpose-built for Sitecore AI, it’s fast, efficient, and doesn’t include a lot of extra baggage. It focuses purely on the essential “headless” task of fetching and rendering content.

What About the JSS SDK?
This is the original toolkit Sitecore created for headless development.

The key difference is that the JSS SDK was designed to be a one-size-fits-all solution. It had to support both the new, headless Sitecore AI and Sitecore’s older, all-in-one platform, Sitecore XP/XM.

To do this, it had to include extra code and dependencies to support older features, like the “Experience Editor”. This makes the JSS SDK “bulkier” and more complex. If you’re only using Sitecore AI, you’re carrying around a lot of extra weight you simply don’t need.

The Sitecore Content SDK is the modern, purpose-built toolkit for developers using Sitecore AI, providing seamless, out-of-the-box integration with the platform’s most powerful capabilities. This includes seamless visual editing that empowers marketers to build and edit pages in real-time, as well as built-in hooks for personalization and analytics that simplify the delivery and tracking of targeted user experiences. For developers, it provides GraphQL utilities to streamline data fetching and is deeply optimized for Next.js, enabling high-performance features like server-side rendering. Furthermore, with the recent introduction of App Router support (in beta), the SDK is evolving to give developers even more granular control over performance, SEO, bundle sizes, and security through a more modern, modular code structure.

What does the Content SDK offer?

1) App Router support (v1.2)

With version 1.2.0, Sitecore Content SDK introduces App Router support in beta. While the full fledged stable release is expected soon, developers can already start exploring its benefits and work flow with 1.2 version.
This isn’t just a minor update; it’s a huge step toward making your front-end development more flexible and highly optimized.

Why should you care? –
The App Router introduces a fantastic change to your starter application’s code structure and how routing works. Everything becomes more modular and declarative, aligning perfectly with modern architecture practices. This means defining routes and layouts is cleaner, content fetching is neatly separated from rendering, and integrating complex Next.js features like dynamic routes is easier than ever. Ultimately, this shift makes your applications much simpler to scale and maintain as they grow on Sitecore AI.

Performance: Developers can fine-tune route handling with nested layouts and more aggressive and granular caching to seriously boost overall performance, leading to faster load times.

Bundle Size: Smaller bundle size because it uses React Server Components (RSC) to render components. It help fetch and render component from server side without making the static files in bundle.

Security: It helps with security by giving improved control over access to specific routes and content.

With the starter kit applications, this is how app router routing structure looks like:

Approute

 

2) New configs – sitecore.config.ts & sitecore.cli.config.ts

The sitecore.config.ts file, located in the root of your application, acts as the central configuration point for Content SDK projects. It is replacement of the older temp/config file used by the JSS SDK. It contains properties that can be used throughout the application just by importing the file. It contains important properties like sitename, defaultLanguage, edge props like contextid. Starter templates include a very lightweight version containing only the mandatory parameters necessary to get started. Developers can easily extend this file as the project grows and requires more specific settings.

Key Aspects:

Environment Variable Support: This file is designed for deployment flexibility using a layered approach. Any configuration property present in this file can be sourced in three ways, listed in order of priority:

  1. Explicitly defined in the configuration file itself.
  2. Fallback to a corresponding environment variable (ideal for deployment pipelines).
  3. Use a default value if neither of the above is provided.

This layered approach ensures flexibility and simplifies deployment across environments.

 

The sitecore.cli.config.ts file is dedicated to defining and configuring the commands and scripts used during the development and build phases of a Content SDK project.

Key Aspects:

CLI Command Configuration: It dictates the commands that execute as part of the build process, such as generateMetadata() and generateSites(), which are essential for generating Sitecore-related data and metadata for the front-end.

Component Map Generation: This file manages the configuration for the automatic component map generation. This process is crucial for telling Sitecore how your front-end components map to the content structure, allowing you to specify file paths to scan and define any files or folders to exclude. Explored further below.

Customization of Build Process: It allows developers to customize the Content SDK’s standard build process by adding their own custom commands or scripts to be executed during compilation.

While sitecore.config.ts handles the application’s runtime settings (like connection details to Sitecore AI), sitecore.cli.config.ts works in conjunction to handle the development-time configuration required to prepare the application for deployment.

Cli Config

 

3) Component map

In Sitecore Content SDK-based applications, every custom component must be manually registered in the .sitecore/component-map.ts file located in the app’s root. The component map is a registry that explicitly links Sitecore renderings to their corresponding frontend component implementations. The component map tells the Content SDK which frontend component to render for each component receives from Sitecore. When the rendering gets added to any page via presentation, component map tells which frontend rendering should be rendered at the place.

Key Aspects:

Unlike JSS implementations that automatically maps components, the Content SDK’s explicit component map enables better tree-shaking. Your final production bundle will only include the components you have actually registered and use, resulting in smaller, more efficient application sizes.

This is how it looks like: (Once you start creating custom component, you have to add the component name here to register.)

Componentmap

 

4) Import map

The import map is a tool used specifically by the Content SDK’s code generation feature. It manages the import paths of components that are generated or used during the build process. It acts as a guide for the code generation engine, ensuring that any new code it creates correctly references your existing components.
Where it is: It is a generated file, typically found at ./sitecore/import-map.ts, that serves as an internal manifest for the build process. You generally do not need to edit this file manually.
It simplifies the logic of code generation, guaranteeing that any newly created code correctly and consistently references your existing component modules.

The import map generation process is configurable via the sitecore.cli.config.ts file. This allows developers to customize the directories scanned for components.

 

5) defineMiddleware in the Sitecore Content SDK

defineMiddleware is a utility for composing a middleware chain in your Next.js app. It gives you a clean, declarative way to handle cross-cutting concerns like multi-site routing, personalization, redirects, and security all in one place. This centralization aligns perfectly with modern best practices for building scalable, maintainable functions.

The JSS SDK leverages a “middleware plugin” pattern. This system is effective for its time, allowing logic to be separated into distinct files. However, this separation often requires developers to manually manage the ordering and chaining of multiple files, which could become complex and less transparent as the application grew. The Content SDK streamlines this process by moving the composition logic into a single, highly readable utility which can customizable easily by extending Middleware

Middleware

 

6) Debug Logging in Sitecore Content SDK

Debug logging helps you see what the SDK is doing under the hood. Super useful for troubleshooting layout/dictionary fetches, multisite routing, redirects, personalization, and more. The Content SDK uses the standard DEBUG environment variable pattern to enable logging by namespace. You can selectively turn on logging for only the areas you need to troubleshoot, such as: content-sdk:layout (for layout service details) or content-sdk:dictionary (for dictionary service details)
For all available namespaces and parameters, refer to sitecore doc – https://doc.sitecore.com/sai/en/developers/content-sdk/debug-logging-in-content-sdk-apps.html#namespaces 

 

7) Editing & Preview

In the context of Sitecore’s development platform, editing and preview render optimization with the Content SDK involves leveraging middleware, architecture, and framework-specific features to improve the performance of rendering content in editing and preview modes. The primary goal is to provide a fast and responsive editing experience for marketers using tools like Sitecore AI Pages and the Design Library. EditingRenderMiddleware: The Content SDK for Next.js includes optimized middleware for editing scenarios. Instead of a multi-step process involving redirects, the optimized middleware performs an internal, server-side request to return the HTML directly. This reduces overhead and speeds up rendering significantly.
This feature Works out of the box in most environments: Local container, Vercel / Netlify, SitecoreAI (defaults to localhost as configured)

For custom setups, override the internal host with: SITECORE_INTERNAL_EDITING_HOST_URL=https://host
This leverages a Integration with XM Cloud/Sitecore AI Pages for visual editing and testing of components.

 

8) SitecoreClient

The SitecoreClient class in the Sitecore Content SDK is a centralized data-fetching service that simplifies communication with your Sitecore content backend typically with Experience Edge or preview endpoint via GraphQL endpoints.
Instead of calling multiple services separately, SitecoreClient lets you make one organized request to fetch everything needed for a page layout, dictionary, redirects, personalization, and more.

Key Aspect:

Unified API: One client to access layout, dictionary, sitemap, robots.txt, redirects, error pages, multi-site, and personalization.
To understand all key methods supported, please refer to sitecore documentation: https://doc.sitecore.com/sai/en/developers/content-sdk/the-sitecoreclient-api.html#key-methods

Sitecoreclientmethods

9) Built-In Capabilities for Modern Web Experiences

GraphQL Utilities: Easily fetch content, layout, dictionary entries, and site info from Sitecore AI’s Edge and Preview endpoints.
Personalization & A/B/n Testing: Deploy multiple page or component variants to different audience segments (e.g., by time zone or language) with no custom code.
Multi-site Support: Seamlessly manage and serve content across multiple independent sites from a single Sitecore AI instance.
Analytics & Event Tracking: Integrated support via the Sitecore Cloud SDK for capturing user behavior and performance metrics.
Framework-Specific Features: Includes Next.js locale-based routing for internationalization, and supports both SSR and SSG for flexible rendering strategies.

 

10) Cursor for AI development

Starting with Content SDK version 1.1, Sitecore has provided comprehensive “Cursor rules” to facilitate AI-powered development.
The integration provides Cursor with sufficient context about the Content SDK ecosystem and Sitecore development patterns. These set of rules and context helps to accelerate the development. The cursor rules are created for contentsdk with starter application under .cursor folder. This enables the AI to better assist developers with tasks specific to building headless Sitecore components, leading to improved development consistency and speed following same patterns just by providing few commands in generic terms. Example given in below screenshot for Hero component which can act as a pattern to create another similar component by cursor.

Cursorrules

 

11) Starter Templates and Example Applications

To accelerate development and reduce setup time, the Sitecore Content SDK includes a set of starter templates and example applications designed for different use cases and development styles.
The SDK provides a Next.js JavaScript starter template that enables rapid integration with Sitecore AI. This template is optimized for performance, scalability, and best practices in modern front-end development.
Starter Applications in examples

basic-nextjs -A minimal Next.js application showcasing how to fetch and render content from Sitecore AI using the Content SDK. Ideal for SSR/SSG use cases and developers looking to build scalable, production-ready apps.

basic-spa -A single-page application (SPA) example that demonstrates client-side rendering and dynamic content loading. Useful for lightweight apps or scenarios where SSR is not required.

Other demo site to showcase Sitecore AI capabilities using the Content SDK:

kit-nextjs-article-starter

kit-nextjs-location-starter

kit-nextjs-product-starter

kit-nextjs-skate-park

 

Final Thoughts

The Sitecore Content SDK represents a major leap forward for developers building on Sitecore AI. Unlike the older JSS SDK, which carried legacy dependencies, the Content SDK is purpose-built for modern headless architectures—lightweight, efficient, and deeply optimized for frameworks like Next.js. With features like App Router support, runtime and CLI configuration flexibility, and explicit component mapping, it empowers teams to create scalable, high-performance applications while maintaining clean, modular code structures.

]]>
https://blogs.perficient.com/2025/11/19/sitecore-content-sdk-what-it-offers-and-why-it-matters/feed/ 0 388367
Navigating the AI Frontier: Data Governance Controls at SIFIs in 2025 https://blogs.perficient.com/2025/10/13/navigating-the-ai-frontier-data-governance-controls-at-sifis-in-2025/ https://blogs.perficient.com/2025/10/13/navigating-the-ai-frontier-data-governance-controls-at-sifis-in-2025/#comments Mon, 13 Oct 2025 10:57:25 +0000 https://blogs.perficient.com/?p=387652

The Rise of AI in Banking

AI adoption in banking has accelerated dramatically. Predictive analytics, generative AI, and autonomous agentic systems are now embedded in core banking functions such as loan underwriting, compliance including fraud detection and AML, and customer engagement. 

A recent White Paper by Perficient affiliate Virtusa Agentic Architecture in Banking – White Paper | Virtusa documented that when designed with modularity, composability, Human-in-the-Loop (HITL), and governance, agentic AI agents empower a more responsive, data-driven, and human-aligned approach in financial services.

However, the rollout of agentic and generative AI tools without proper controls poses significant risks. Without a unified strategy and governance structure, Strategically Important Financial Institutions (“SIFIs”) risk deploying AI in ways that are opaque, biased, or non-compliant. As AI becomes the engine of next-generation banking, institutions must move beyond experimentation and establish enterprise-wide controls.

Key Components of AI Data Governance

Modern AI data governance in banking encompasses several critical components:

1. Data Quality and Lineage: Banks must ensure that the data feeding AI models is accurate, complete, and traceable.

Please refer to Perficient’s recent blog on this topic here:

AI-Driven Data Lineage for Financial Services Firms: A Practical Roadmap for CDOs / Blogs / Perficient

2. Model Risk Management: AI models must be rigorously tested for fairness, accuracy, and robustness. It has been documented many times in lending decision-making software that the bias of coders can result in biased lending decisions.

3. Third-Party Risk Oversight: Governance frameworks now include vendor assessments and continuous monitoring. Large financial institutions do not have to develop AI technology solutions themselves (Buy vs Build) but they do need to monitor the risks of having key technology infrastructure owned and/or controlled by third parties.

4. Explainability and Accountability: Banks are investing in explainable AI (XAI) techniques. Not everyone is a tech expert, and models need to be easily explainable to auditors, regulators, and when required, customers.

5. Privacy and Security Controls: Encryption, access controls, and anomaly detection are essential. These are all done already in legacy systems and extending it to the AI environment, whether it is narrow AI, machine learning, or more advanced agentic and/or generative AI it is natural to ensure these proven controls are extended to the new platforms. 

Industry Collaboration and Standards

The FINOS Common Controls for AI Services initiative is a collaborative, cross-industry effort led by the FINtech Open-Source Foundation (FINOS) to develop open-source, technology-neutral baseline controls for safe, compliant, and trustworthy AI adoption in financial services. By pooling resources from major banks, cloud providers, and technology vendors, the initiative creates standardized, open-source technology-neutral controls, peer-reviewed governance frameworks, and real-time validation mechanisms to help financial institutions meet complex regulatory requirements for AI. 

Key participants of FINOS include financial institutions such as BMO, Citibank, Morgan Stanley, and RBC, and key Technology & Cloud Providers include Perficient’s technology partners including Microsoft, Google Cloud, and Amazon Web Services (AWS). The FINOS Common Controls for AI Services initiative aims to create vendor-neutral standards for secure AI adoption in financial services.

At Perficient, we have seen leading financial institutions, including some of the largest SIFIs, establishing formal governance structures to oversee AI initiatives. Broadly, these governance structures typically include:

– Executive Steering Committees at the legal entity level
– Working Groups, at the legal entity as well as the divisional, regional and product levels
– Real-Time Dashboards that allow customizable reporting for boards, executives, and auditors

This multi-tiered governance model promotes transparency, agility, and accountability across the organization.

Regulatory Landscape in 2025

Regulators worldwide are intensifying scrutiny of Artificial Intelligence in banking. The EU AI Act, the U.S. SEC’s cybersecurity disclosure rules, and the National Insititute of Standards and Technology (“NIST”) AI Risk Management Framework are shaping how financial institutions must govern AI systems.

Key regulatory expectations include:

– Risk-Based Classification
– Human Oversight
– Auditability
– Bias Mitigation

Some of these, and other regulatory regimes have been documented and summarized by Perficient at the following links:

AI Regulations for Financial Services: Federal Reserve / Blogs / Perficient

AI Regulations for Financial Services: European Union / Blogs / Perficient 

Eu Ai Act Risk Based Approach

The Road Ahead

As AI becomes integral to banking operations, data governance will be the linchpin of responsible innovation. Banks must evolve from reactive compliance to proactive risk management, embedding governance into every stage of the AI lifecycle.

The journey begins with data—clean, secure, and well-managed. From there, institutions must build scalable frameworks that support ethical AI development, align with regulatory mandates, and deliver tangible business value.

Readers are urged to read the links contained in this blog and then contact Perficient, a global AI-first digital consultancy to discuss how partnering with Perficient can help run a tailored assessment and pilot design that maps directly to your audit and governance priorities and ensure all new tools are rolled out in a well-designed data governance environment.

]]>
https://blogs.perficient.com/2025/10/13/navigating-the-ai-frontier-data-governance-controls-at-sifis-in-2025/feed/ 1 387652
AI-Driven Data Lineage for Financial Services Firms: A Practical Roadmap for CDOs https://blogs.perficient.com/2025/10/06/ai-driven-data-lineage-for-financial-services-firms-a-practical-roadmap-for-cdos/ https://blogs.perficient.com/2025/10/06/ai-driven-data-lineage-for-financial-services-firms-a-practical-roadmap-for-cdos/#respond Mon, 06 Oct 2025 11:17:05 +0000 https://blogs.perficient.com/?p=387626

Introduction

Imagine just as you’re sipping your Monday morning coffee and looking forward to a hopefully quiet week in the office, your Outlook dings and you see that your bank’s primary federal regulator is demanding the full input – regulatory report lineage for dozens of numbers on both sides of the balance sheet and the income statement for your latest financial report filed with the regulator. The full first day letter responses are due next Monday, and as your headache starts you remember that the spreadsheet owner is on leave; the ETL developer is debugging a separate pipeline; and your overworked and understaffed reporting team has three different ad hoc diagrams that neither match nor reconcile.

If you can relate to that scenario, or your back starts to tighten in empathy, you’re not alone. Artificial Intelligence (“AI”) driven data lineage for banks is no longer a nice-to-have. We at Perficient working with our clients in banking, insurance, credit unions, and asset managers find that it’s the practical answer to audit pressure, model risk (remember Lehman Brothers and Bear Stearns), and the brittle manual processes that create blind spots. This blog post explains what AI-driven lineage actually delivers, why it matters for banks today, and a phased roadmap Chief Data Officers (“CDOs”) can use to get from pilot to production.

Why AI-driven data lineage for banks matters today

Regulatory pressure and real-world consequences

Regulators and supervisors emphasize demonstrable lineage, timely reconciliation, and governance evidence. In practice, financial services firms must show not just who touched data, but what data enrichment and/or transformations happened, why decisions used specific fields, and how controls were applied—especially under BCBS 239 guidance and evolving supervisory expectations.

In addition, as a former Risk Manager, the author knows that he would have wanted and has spoken to a plethora of financial services executives who want to know that the decisions they’re making on liquidity funding, investments, recording P&L, and hedging trades are based on the correct numbers. This is especially challenging at global firms that operate in in a transaction heavy environment with constantly changing political, interest rate, foreign exchange and credit risk environment.

Operational risks that keep CDOs up at night

Manual lineage—spreadsheets, tribal knowledge, and siloed code—creates slow audits, delayed incident response, and fragile model governance. AI-driven lineage automates discovery and keeps lineage living and queryable, turning reactive fire drills into documented, repeatable processes that will greatly shorten the time QA tickets are closed and reduce compensation costs for misdirected funds. It also provides a scalable foundation for governed data practices without sacrificing traceability.

What AI-driven lineage and controls actually do (written by and for non-tech staff)

At its core, AI-driven data lineage combines automated scanning of code, SQL, ETL jobs, APIs, and metadata with semantic analysis that links technical fields to business concepts. Instead of a static map, executives using AI-driven data lineage get a living graph that shows data provenance at the field level: where a value originated, which transformations touched it, and which reports, models, or downstream services consume it.

AI adds value by surfacing hidden links. Natural language processing reads table descriptions, SQL comments, and even README files (yes they do still exist out there) to suggest business-term mappings that close the business-IT gap. That semantic layer is what turns a technical lineage graph into audit-ready evidence that regulators or auditors can understand.

How AI fixes the pain points keeping CDOs up at night

Faster audits: As a consultant at Perficient, I have seen AI-driven lineage that after implementation allowed executives to answer traceability questions in hours rather than weeks. Automated evidence packages—exportable lineage views and transformation logs—provide auditors with a reproducible trail.
Root-cause and incident response: When a report or model spikes, impact analysis highlights which datasets and pipelines are involved, highlighting responsibility and accountability, speeding remediation and alleviating downstream impact.
Model safety and feature provenance: Lineage that includes training datasets and feature transformations enables validation of model inputs, reproducibility of training data, and enforcement of data controls—supporting explainability and governance requirements. That allows your P&L to be more R&S. (a slogan used by a client that used R&S P&L to mean rock solid profit and loss.)

Tooling, architecture, and vendor considerations

When evaluating vendors, demand field-level lineage, semantic parsing (NLP across SQL, code, and docs), auditable diagram exports, and policy enforcement hooks that integrate with data protection tools. Deployment choices matter in regulated banking environments; hybrid architectures that keep sensitive metadata on-prem while leveraging cloud analytics often strike a pragmatic balance.

A practical, phased roadmap for CDOs

Phase 0 — Align leadership and define success: Engage CRO, COO, and Head of Model Risk. Define 3–5 KPIs (e.g., lineage coverage, evidence time, mean time to root cause) and what “good” will look like. This is often done during a evidence gathering phase by Perficient with clients who are just starting their Artificial Intelligence journey.
Phase 1 — Inventory and quick wins: Target a high-risk area such as regulatory reporting, a few production models, or a critical data domain. Validate inventory manually to establish baseline credibility.
Phase 2 — Pilot AI lineage and controls: Run automated discovery, measure accuracy and false positives, and quantify time savings. Expect iterations as the model improves with curated mappings.
Phase 1 and 2 are usually done by Perficient with clients as a Proof-of-Concept phase to show that the key feeds into and out of existing technology platforms can be done.
Phase 3 — Operationalize and scale: Integrate lineage into release workflows, assign lineage stewards, set SLAs, and connect with ticketing and monitoring systems to embed lineage into day-to-day operations.
Phase 4 — Measure, refine, expand: Track KPIs, adjust models and rules, and broaden scope to additional reports, pipelines, and models as confidence grows.

Risks, human oversight, and governance guardrails

AI reduces toil but does not remove accountability. Executives, auditors and regulators either do or should require deterministic evidence and human-reviewed lineage. Treat AI outputs as recommendations subject to curator approval. This will avoid what many financial services executives are dealing with what is now known as AI Hallucinations.

Guardrails include the establishment of exception processing workflows for disputed outputs and toll gates to ensure security and privacy are baked into design—DSPM, masking, and appropriate IAM controls should be integral, not afterthoughts.

Conclusion and next steps

AI data lineage for banks is a pragmatic control that directly addresses regulatory expectations, speeds audits, and reduces model and reporting risk. Start small, prove value with a focused pilot, and embed lineage into standard data stewardship processes. If you’re a CDO looking to move quickly with minimal risk, contact Perficient to run a tailored assessment and pilot design that maps directly to your audit and governance priorities. We’ll help translate proof into firm-wide control and confidence.

]]>
https://blogs.perficient.com/2025/10/06/ai-driven-data-lineage-for-financial-services-firms-a-practical-roadmap-for-cdos/feed/ 0 387626
Transform Your Data Workflow: Custom Code for Efficient Batch Processing in Talend-Part 1 https://blogs.perficient.com/2025/10/03/transform-data-workflow-custom-code-for-efficient-batch-processing-in-talend-part-1-2/ https://blogs.perficient.com/2025/10/03/transform-data-workflow-custom-code-for-efficient-batch-processing-in-talend-part-1-2/#respond Fri, 03 Oct 2025 07:22:35 +0000 https://blogs.perficient.com/?p=387572

Introduction:

Custom code in Talend offers a powerful way to enhance batch processing efficiently by allowing developers to implement specialized logic that is not available through Talend’s standard components. This can involve data transformations, custom code as per use case and integration with flat files as per specific project needs. By leveraging custom code, users can optimize performance, improve data quality, and streamline complex batch workflows within their Talend jobs.

Understand Batch Processing:

            Batch processing is a method of running high-volume, repetitive data within Talend jobs. The batch method allows users to process a bunch of data when computing resources are available, and with little or no user interaction.

Through batch processing, users gather and retain data, subsequently processing it during a designated period referred to as a “batch window.” This method enhances efficiency by establishing processing priorities and executing data tasks in a timeframe that is optimal.

Here, Talend job takes the total row count from source file then load the data from the flat file, processes it in a batch, provided input through context variable & then write the data into smaller flat files. This implementation made it possible to process enormous amounts of data more precisely and quickly than other implementation.

Batch processing is a method of executing a series of jobs sequentially without user interaction, typically used for handling large volumes of data efficiently. Talend, a prominent and extensively employed ETL (Extract, Transform, Load) tool, utilizes batch processing to facilitate the integration, transformation, and loading of data into data warehouse and various other target systems.

Talend Components:

Key components for batch processing as mention below:

  • tFileInputDelimited, tFileOutputDelimited: For reading & writing data from/to files.
  • tFileRowCount: Reads file row by row to calculate the number of rows.
  • tLoop: Executes a task automatically, based on a loop size.
  • tHashInput, tHashOutput: For high-speed data transfer and processing within a job. tHashOutput writes data to cache memory, while tHashInput reads from that cached data.
  • tFilterRow: For filtering rows from a dataset based on specified.
  • tMap: Use for data transformation which allow to map input data with output data along with use to perform data filtering, complex data manipulation, typecasting & multiple input source join.
  • tJavaRow: It can be used as an intermediate component, and we are able to access the input flow and transform the data using custom Java code.
  • tJava: It has no input or output data flow & can be used independently to Integrate custom Java code.
  • tLogCatcher: It is used in error handling within Talend job for adding runtime logging information. It catches all the exceptions and warnings raised by tWarn and tDie components during Talend job execution.
  • tLogRow: It is employed in error handling to display data or keep track of processed data in the run console.

Workflow with example:

To process the bulk of data in Talend, we can implement batch processing to efficiently process flat file data within a minimal execution time. We can read the flat file data & after execution, we can write it into a chunk of another flat file as a target & we can achieve this without batch processing. But this data flow will take quite a larger execution time to execute. If we use batch processing using the custom code, it takes minimal execution time to write the entire source file data into chunks of files at the target location.

Talend job design

Talend job design

Solution:

  • Read the number of rows in the source flat file using tFileRowCount component.
  • To determine the batch size, subtract the header count from the total row count and then divide the number by the total batch size. Take the whole number nearby which indicates the total number of batch or chunk.

    Calculate the batch / chunk by reducing the header from total row count & then divide the number by the total batch size. Take the whole number nearby which indicates the total number of batch or chunk.

    Calculate the batch size from total row count

  • Now use tFileInputDelimited component to read the source file content. In the tMap component, utilize the sequence Talend function to generate row numbers for your data mapping and transformation tasks. Then, load all of the data into the tHashOutput component, which stores the data into a cache.
  • Iterate the loop based on the calculated whole number using tLoop
  • Retrieve all the data from tHashInput component.
  • Filter the dataset retrieved from tHashInput component based on the rowNo column in the schema using tFilterRow

    Filter the dataset retrieved from tHashInput component based on the rowNo column in the schema using tFilterRow

    Filter the dataset using tFilterRow

  • If First Iteration is in progress & batch size is 100 then rowNo range will be as 1 to 100.
    If Third Iteration is in progress & batch size is 100 then rowNo range will be as 201 to 300.
    For example, if the value of current iteration is 3 then [(3-1=2)* 100]+1 = 201 & [3*100=300]. So final dataset range for the 3rd iteration will be 201 to 300.
  • Finally extract the dataset range between the rowNo column & write it into chunk of output target file using tFileOutputDelimited
  • The system uses the tLogCatcher component for error management by capturing runtime logging details, including warning or exception messages, and employs tLogRow to display the information in the execution console.
  • Regarding performance tuning, we have a tMap component that maps source data to output data, allows for complex data transformation, and offers unique join, first join, and all other join options for looking up data within the tMap component.
  • The temporary data that the tHashInput & tHashOutput components store in cache memory enhances runtime performance.

 

Advantages of Batch Processing:

  • Batch processing can efficiently handle large datasets.
  • It takes minimal time to process the data even after data transformation.
  • By grouping records from a large dataset and processing them as a single unit, it can be highly beneficial for improving performance.
  • With the batch processing, it can easily scale to accommodate growing data volumes.
  • It is particularly useful for operations like generating reports, performing data integration, and executing complex transformations on large datasets.

For more details: Get-started-talend-open-studio-data-integration

Note: Efficient Batch Processing in Talend-Part 2

]]>
https://blogs.perficient.com/2025/10/03/transform-data-workflow-custom-code-for-efficient-batch-processing-in-talend-part-1-2/feed/ 0 387572
Beyond Denial: How AI Concierge Services Can Transform Healthcare from Reactive to Proactive https://blogs.perficient.com/2025/09/24/beyond-denial-how-ai-concierge-services-can-transform-healthcare-from-reactive-to-proactive/ https://blogs.perficient.com/2025/09/24/beyond-denial-how-ai-concierge-services-can-transform-healthcare-from-reactive-to-proactive/#respond Wed, 24 Sep 2025 14:39:32 +0000 https://blogs.perficient.com/?p=387380

The headlines are troubling but predictable. The Trump administration will launch a program next year to find out how much money an artificial intelligence algorithm could save the federal government by denying care to Medicare patients. Meanwhile, a survey of physicians published by the American Medical Association in February found that 61% think AI is “increasing prior authorization denials, exacerbating avoidable patient harms and escalating unnecessary waste now and into the future.”

We’re witnessing the healthcare industry’s narrow vision of AI in action: algorithms designed to say “no” faster and more efficiently than ever before. But what if we’re missing the bigger opportunity?

The Current AI Problem: Built to Deny, Not to Help

The recent expansion of AI-powered prior authorization reveals a fundamental flaw in how we’re approaching healthcare technology. “The more expensive it is, the more likely it is to be denied,” said Jennifer Oliva, a professor at the Maurer School of Law at Indiana University-Bloomington, whose work focuses on AI regulation and health coverage.

This approach creates a vicious cycle: patients don’t understand their benefits, seek inappropriate or unnecessary care, trigger costly prior authorization processes, face denials, appeal those denials, and ultimately either give up or create even more administrative burden for everyone involved.

The human cost is real. Nearly three-quarters of respondents thought prior authorization was a “major” problem in a July poll published by KFF, and we’ve seen how public displeasure with insurance denials dominated the news in December, when the shooting death of UnitedHealthcare’s CEO led many to anoint his alleged killer as a folk hero.

A Better Vision: The AI Concierge Approach

What if instead of using AI to deny care more efficiently, we used it to help patients access the right care more effectively? This is where the AI Concierge concept transforms the entire equation.

An AI Concierge doesn’t wait for a claim to be submitted to make a decision. Instead, it proactively:

  • Educates patients about their benefits before they need care
  • Guides them to appropriate providers within their network
  • Explains coverage limitations in plain language before appointments
  • Suggests preventive alternatives that could avoid more expensive interventions
  • Streamlines pre-authorization by ensuring patients have the right documentation upfront

The Quantified Business Case

The financial argument for AI Concierge services is compelling:

Star Ratings Revenue Impact: A half-star increase in Medicare Star Ratings is valued at approximately $500 per member. For a 75,000-member plan, that translates to $37.5 million in additional funding. An AI Concierge directly improves patient satisfaction scores that drive these ratings.

Operational Efficiency Gains: Healthcare providers implementing AI-powered patient engagement systems report 15-20% boosts in clinic revenue and 10-20% reductions in overall operational costs. Clinics using AI tools see 15-25% increases in patient retention rates.

Cost Avoidance Through Prevention: Utilizing AI to help patients access appropriate care could save up to 50% on treatment costs while improving health outcomes by up to 40%. This happens by preventing more expensive interventions through proper preventive care utilization.

The HEDIS Connection

HEDIS measures provide the perfect framework for demonstrating AI Concierge value. With 235 million people enrolled in plans that report HEDIS results, improving these scores directly impacts revenue through bonus payments and competitive positioning.

An AI Concierge naturally improves HEDIS performance in:

  • Preventive Care Measures: Proactive guidance increases screening and immunization rates
  • Care Gap Closure: Identifies and addresses gaps before they become expensive problems
  • Patient Engagement: Improves medication adherence and chronic disease management

Beyond the Pilot Programs

While government initiatives like the WISeR pilot program focus on “Wasteful and Inappropriate Service Reduction” through AI-powered denials, forward-thinking healthcare organizations have an opportunity to differentiate themselves with AI-powered patient empowerment.

The math is simple: preventing a $50,000 hospitalization through proactive care coordination delivers better ROI than efficiently denying the claim after it’s submitted.

AI Healthcare Concierge Implementation Strategy

For healthcare leaders considering AI Concierge implementation:

  • Phase 1: Deploy AI-powered benefit explanation tools that reduce call center volume and improve patient understanding
  • Phase 2: Integrate predictive analytics to identify patients at risk for expensive interventions and guide them to preventive alternatives
  • Phase 3: Expand to comprehensive care navigation that optimizes both patient outcomes and organizational performance

The Competitive Advantage

While competitors invest in AI to process denials faster, organizations implementing AI Concierge services are investing in:

  • Member satisfaction and retention (15-25% improvement rates)
  • Star rating improvements ($500 per member value per half-star)
  • Operational cost reduction (10-20% typical savings)
  • Revenue protection through better member experience

Conclusion: Choose Your AI Future

The current trajectory of AI in healthcare—focused on denial optimization—represents a massive missed opportunity. As one physician noted about the Medicare pilot: “I will always, always err on the side that doctors know what’s best for their patients.”

AI Healthcare Concierge services align with this principle by empowering both patients and providers with better information, earlier intervention, and more effective care coordination. The technology exists. The business case is proven. The patient need is urgent.

The question isn’t whether AI will transform healthcare—it’s whether we’ll use it to build walls or bridges between patients and the care they need.

The choice is ours. Let’s choose wisely.

]]>
https://blogs.perficient.com/2025/09/24/beyond-denial-how-ai-concierge-services-can-transform-healthcare-from-reactive-to-proactive/feed/ 0 387380
Unlocking the power of Data Enrichment in Collibra for effective Data Governance https://blogs.perficient.com/2025/07/28/unlocking-the-power-of-data-enrichment-in-collibra-a-key-element-in-effective-data-governance/ https://blogs.perficient.com/2025/07/28/unlocking-the-power-of-data-enrichment-in-collibra-a-key-element-in-effective-data-governance/#respond Mon, 28 Jul 2025 09:19:30 +0000 https://blogs.perficient.com/?p=385103

In today’s data-driven landscape, Organizations are not just collecting the Data, they are striving to understand, trust, and maximize its value. One of the critical capabilities that helps achieve the goal is data enrichment, especially when implemented through enterprise-grade governance tools like Collibra.

In this blog, we will explore how Collibra enables data enrichment, why it is essential for effective data governance, and how organizations can leverage it to drive better decision-making.

What is Data Enrichment in Collibra?

Data enrichment enhances the dataset within the Collibra data governance tool by adding business context, metadata, correcting inaccuracies, and governance attributes that help users to understand the data’s meaning, usage, quality, and lineage.

Rather than simply documenting tables/columns, data enrichment enables organizations to transform technical metadata into meaningful, actionable insights, in which this enriched context empowers business and technical users alike to trust the data they are working with and use it confidently for analysis, reporting, and compliance.

How does Data Enrichment work in Collibra?

Data Enrichment

How We Use Data Enrichment in the Banking Domain

In today’s digital landscape, banks manage various data formats (such as CSV, JSON, XML, and tables) with vast volumes of data originating from internal and external sources like file systems, cloud platforms, and databases. Collibra automatically catalogs these data assets and generates metadata.

But simply cataloging data isn’t enough. The next step is data enrichment, where we link technical metadata with business-glossary terms to give metadata meaningful business context and ensure consistent description and understanding across the organization. Business terms clarify what each data element represents from a business perspective, making it accessible not just to IT teams but also to business users.

In addition, each data asset is tagged with data classification labels such as PCI (Payment Card Information), PII (Personally Identifiable Information), and confidential. This classification plays a critical role in data security, compliance, and risk management, especially in a regulated industry like banking.

To further enhance the trustworthiness of data, Collibra integrates data profiling capabilities. This process analyzes the actual content of datasets to assess their structure and quality. Based on profiling results, we link data to data‑quality rules that monitor completeness, accuracy, and conformity. These rules help enforce high-quality standards and ensure that the data aligns with both internal expectations and external regulatory requirements.

An essential feature in Collibra is data lineage, which provides a visual representation of the data flow from its source to its destination. This visibility helps stakeholders understand how data transforms and moves through various systems, which is essential for impact analysis, audits, and regulatory reporting.

Finally, the enriched metadata undergoes a structured workflow-driven review process. This involves all relevant stakeholders, including data owners, application owners, and technical managers. The workflow ensures that we not only produce accurate and complete metadata but also review and approve it before publishing or using it for decision-making.

Example: Enriching the customer data table

  • Database: Vertica Datalake
  • Table: Customer_Details
  • Column: Customer_MailID
  • Business Term: Customer Mail Identification
  • Classification:P II (Personally Identifiable Information)
  • Quality rule: There are no null values in Customer mailID. (Completeness)
  • Linked Polity: GDPR policy for the EU Region
  • Lineage: Salesforce à ETL pipeline à Vertica

Data enrichment in Collibra is a cornerstone of a mature Data Governance Framework; it helps transform raw technical metadata into a living knowledge asset, fueling trust, compliance, and business value. By investing time in enriching your data assets, you are not just cataloging them; you are empowering your organization to make smarter, faster, and more compliant data-driven decisions.

]]>
https://blogs.perficient.com/2025/07/28/unlocking-the-power-of-data-enrichment-in-collibra-a-key-element-in-effective-data-governance/feed/ 0 385103
AI at the Service of Talent: Building Evaluator Agents with Copilot and n8n https://blogs.perficient.com/2025/07/23/ai-at-the-service-of-talent-building-evaluator-agents-with-copilot-and-n8n/ https://blogs.perficient.com/2025/07/23/ai-at-the-service-of-talent-building-evaluator-agents-with-copilot-and-n8n/#respond Thu, 24 Jul 2025 01:03:33 +0000 https://blogs.perficient.com/?p=385075

Abstract

This article explores how artificial intelligence can transform technical evaluation processes in talent acquisition, particularly in the software development field. Through the Dessert & Learn session, two complementary approaches are presented: a basic one using prompts in Copilot for automated code analysis, and a more advanced one using intelligent agents built with n8n and the RAG (Retrieval-Augmented Generation) pattern. These tools demonstrate how bias can be reduced, evaluation accuracy improved, and hiring processes scaled. Additionally, the fundamentals of RAG and n8n are explained, highlighting their potential to build contextualized automation workflows that are replicable across multiple industries.


Introduction

In a world where tech talent is increasingly scarce and competitive, traditional hiring processes face major challenges: subjective evaluations, lack of scalability, unconscious biases, and difficulties in identifying the true technical potential of candidates.

In this context, Artificial Intelligence (AI) emerges as a powerful ally. In this Dessert & Learn session, we explore how to build intelligent evaluator agents using tools like Copilotn8n, and the RAG (Retrieval-Augmented Generation) pattern to transform the way we assess technical talent.

 

The Challenge of Traditional Hiring

In traditional hiring processes—particularly in the software development field—companies face multiple challenges:

  • Subjective evaluations, often based more on intuition than on objective technical criteria.
  • Lack of scalability, as manually reviewing dozens or hundreds of resumes consumes significant time and resources.
  • Difficulty identifying true technical potential, due to poorly structured interviews or misaligned assessments.
  • Unconscious biases, which can undermine fairness in candidate selection.

These challenges not only slow down the hiring process but also increase the risk of poor decisions, negatively impacting productivity, organizational culture, and talent retention. In this context, artificial intelligence emerges as a powerful tool to transform and optimize the recruitment workflow.

From Prompts to Agents: The Automation Journey

Automation begins with something as simple as a prompt in Copilot, but it evolves into autonomous agents capable of reasoning, retrieving contextual information, and making decisions. This journey involves integrating tools like OpenAI, Pinecone, and orchestration platforms such as n8n, enabling the creation of intelligent workflows that analyze, classify, and generate automated reports.

Basic Agent with Copilot

The first step was to design a specific prompt in Copilot to analyze technical development tests. This basic agent evaluates the source code submitted by a candidate, identifying best practices, common mistakes, and compliance with technical criteria. It’s an agile solution for obtaining an initial automated assessment.

Screenshot 2025 07 23 100312

Figure 1. Technical Interviews Agent

This agent is designed to assist in the technical evaluation of candidates applying for .NET developer positions. Its main role is to analyze the responses given during technical interviews and generate a detailed report based on widely accepted principles such as Clean Code, Object-Oriented Programming (OOP), SOLID principles, Entity Framework, software architecture, REST services, software testing (unit, integration, TDD), and experience with both relational and NoSQL databases. Through this analysis, the agent identifies technical strengths, applied best practices, and highlighted areas of knowledge, as well as potential improvement opportunities or topics the candidate should reinforce. This process provides a clear and structured view of the interviewee’s technical readiness, facilitating well-informed decision-making during the selection process. The ultimate goal is to ensure that the candidate meets the organization’s expected technical standards.

Screenshot 2025 07 23 193813

Figure 2. Configuration the Agent on Copilot

Agent with n8n and RAG

In this demonstration, I will showcase an advanced evaluator agent built using n8n and the Retrieval-Augmented Generation (RAG) pattern. This agent goes beyond simple prompts by integrating multiple tools and services to perform intelligent evaluations of software developer profiles.

The agent uses n8n as the orchestration platform, combining OpenAI for language processing, Pinecone for vector-based retrieval of previously indexed CVs or technical documents, and custom logic for decision-making. The workflow includes:

  • Semantic understanding of input (e.g., technical skills, project history)
  • Contextual search across a vector store to enrich the analysis
  • Rule-based scoring and classification (Junior, Mid, Senior)
  • Generation of structured, automated technical reports

This example illustrates how to build scalable and intelligent automation flows capable of interpreting, reasoning, and deciding in real-world hiring processes.

First Flow on n8n

This automated workflow in n8n is designed to process candidate folders based on their technical profile (FullStack React.js, Angular, BackEnd with C#, or Java), stored in Dropbox. Upon execution, the workflow lists all available folders, splits them individually, and routes each one using a Switch Folders node according to the detected stack. It then lists and downloads the corresponding files from each folder, which may include resumes, interview notes, or technical responses. These documents are sent to the Candidates Vectorization module, which uses Azure OpenAI to generate semantic embeddings. The resulting vectors are further processed by a Default Data Loader and text segmentation tools (Token Splitter), optimizing the format for future queries or analysis. This flow enables structured and enriched processing of candidate data, facilitating intelligent searches and contextual evaluations, ultimately supporting more efficient and accurate technical hiring decisions.

Screenshot 2025 07 23 104502

Figure 3. CV’s Flow vectorization

Second Flow on n8n – Candidate Assistant

This flow represents an intelligent agent designed to answer technical questions about candidates categorized by technology stack: FullStack React.js, FullStack Angular, BackEnd with C#, and BackEnd with Java. The process begins with an incoming HTTP request via a webhook. The request fields are then edited and sent to the central AI Agent node, which combines simple memory (for session context) with an Azure OpenAI chat model.

The agent evaluates which specific tool to use based on the mentioned stack and redirects the query to one of four custom tools (one per stack). Each tool is connected to a dedicated vector store containing previously generated embeddings from resumes, interview responses, or technical assessments.

For each stack, there is a dedicated chat model that interacts directly with its respective vector store, enabling contextualized and accurate responses. This modular design allows highly specialized interaction tailored to the candidate’s profile. Finally, the generated response is returned via webhook to the original request source.

This flow enables smart querying of technical information, providing a semantic search system powered by embeddings and language models—ideal for automating the evaluation process in technical talent recruitment.

Screenshot 2025 07 23 104541

Figure 4. Intelligent Agent using RAG architecture

Interactive Web Chat with AI Agent

The image shows a web chat interface connected to an automation flow in n8n via a webhook. The purpose of this chatbot is to assist in technical recruitment processes by allowing users to request detailed information about candidates based on their technology stack or specific skills. In this case, the user inquires about profiles with C# experience, and the agent responds by listing three candidates: Camilo Herrera, Natalia Paredes, and Valentina Ríos, noting that none explicitly mention Entity Framework expertise. When asked for details about Camilo Herrera, the agent replies with a professional summary describing his growth from Junior Developer to Tech Lead, highlighting his experience in maintenance, refactoring, and technical leadership at CloudForge Ltda. (2022–2023). This solution combines natural language processing with retrieval-augmented generation (RAG), enabling contextualized responses based on previously vectorized information. The workflow streamlines the selection process by providing accurate, automated, and easily accessible candidate insights through a user-friendly interface.

Screenshot 2025 07 23 195457

Figure 5. Web Chat Endpoint with Embedded Agent

Conclusions

AI is a powerful ally in technical talent evaluation
Intelligent agents help reduce bias, streamline processes, and improve decision-making during developer selection.

From simple prompts to advanced agents: the journey is progressive
It’s possible to start with basic automated evaluations (like those in Copilot) and gradually evolve toward more robust, scalable solutions with n8n and RAG.

n8n enables complex workflows without sacrificing technical control
By connecting tools like OpenAI, Pinecone, and custom rules, we can build contextual evaluators that generate structured reports and make autonomous decisions.

RAG enables more informed and accurate evaluations
By retrieving relevant information in real time, agents don’t just “respond”—they understand and contextualize, improving the precision of the analysis.

This approach is replicable across multiple domains
While the use case focused on talent evaluation, the same principles apply to support, education, document analysis, healthcare, and more.

Cognitive automation is no longer the future—it’s the present
Implementing intelligent agents is already viable, accessible, and a clear competitive advantage for tech, HR, and analytics teams.

]]>
https://blogs.perficient.com/2025/07/23/ai-at-the-service-of-talent-building-evaluator-agents-with-copilot-and-n8n/feed/ 0 385075
Intelligent Automation in the Healthcare Sector with n8n, OpenAI, and Pinecone https://blogs.perficient.com/2025/07/11/intelligent-automation-in-the-healthcare-sector-with-n8n-openai-and-pinecone/ https://blogs.perficient.com/2025/07/11/intelligent-automation-in-the-healthcare-sector-with-n8n-openai-and-pinecone/#respond Fri, 11 Jul 2025 15:02:36 +0000 https://blogs.perficient.com/?p=384424

Abstract

In today’s digital-first world, healthcare organizations face increasing pressure to modernize operations and improve service delivery. Intelligent automation is no longer a luxury (it’s the foundation for scalable, efficient, and personalized healthcare systems). At Perficient, we’re driving innovation by integrating tools like n8n, Azure OpenAI, and Pinecone to develop smarter, context-aware solutions for the medical field.

This blog explores how we built an automation pipeline that connects clinical data ingestion, semantic search, and conversational interfaces (without the need for complex infrastructure). Using n8n as the orchestration engine, we retrieve medical records, process them through Azure OpenAI to generate embeddings, and store them in Pinecone, a high-performance vector database.

To complete the experience, we added an AI-powered Telegram assistant. This bot interacts with users in real time (patients or staff), answers questions, retrieves medical data, and checks doctor availability by leveraging our semantic layer.

This architecture proves how low-code platforms combined with enterprise AI and vector tools can deliver conversational and data-driven healthcare experiences. Whether you’re a provider, architect, or innovator, this solution offers a real glimpse into the future (where decisions are supported by smart, contextual agents and users get meaningful, accurate answers).

If you need mor information about Chunks and Embeddings, and Vector Databases, you can visit this previous Post:

https://blogs.perficient.com/2025/07/07/turn-your-database-into-a-smart-chatbot-with-openai-langchain-and-chromadb/

Proof of Concept

A simple proof of concept (POC) was developed to demonstrate how an automation and AI-based solution can effectively integrate into real clinical environments. This prototype allows users to quickly and contextually check a patient’s recorded medical visits (including relevant data such as weight, height, consultation date, and clinical notes) and verify whether a healthcare professional is available for a new appointment. The solution, built using visual workflows in n8n and connected to a structured medical database, shows how accurate and helpful responses can be delivered through a channel like Telegram (without the need for complex apps or multiple steps). All of this was achieved by combining tools like Pinecone for semantic search and Azure OpenAI for natural language understanding, resulting in a smooth, user-centered conversational experience.

Creation Embeddings Flow

For the AI assistant to provide useful and contextual responses, all information related to medical appointments and clinical records must be transformed into a format that allows it to truly understand the content (not just read it literally). That’s why the first automation focuses on converting this data into embeddings (numerical representations that capture the meaning of the text).

This process runs automatically every hour, ensuring that any new data (such as a recent appointment or updated clinical note) is quickly processed and indexed. The workflow begins with an API call that retrieves the most recent clinical records or, if it’s the first run, the entire medical history. The data then goes through a processing and cleanup stage before being sent to Azure OpenAI, where the corresponding embeddings are generated.

These vectors are stored in Pinecone, a system specialized in semantic search, allowing the AI assistant to retrieve relevant information accurately (even when the user doesn’t phrase the question exactly as it was recorded).

Thanks to this preparation step, the assistant can respond with specific details about diagnoses, consultation dates, or previously recorded information (all without the user having to manually search through their history). This approach not only improves the user experience, it also ensures that the information is available at the right time and communicated in a natural way.

Imagen1

Figure 1: Creation Embeddings Flow

Once the SQL query is executed and the clinical data is retrieved (including patient name, appointment date, medical notes, professional details, and vital signs), the records go through a transformation process to prepare them for embedding generation. This step converts the content of each appointment into a numerical format (one that the language model can understand), enabling more accurate contextual searches and more relevant responses when the assistant queries the information.

Imagen 2

Figure 2: Clinical Data Embedding Pipeline, SQL Section

The vectorization process was carried out using the Pinecone Vector Store node, which handles the storage of the generated embeddings in a database specifically designed for high-speed semantic searches. This step ensures that the clinical information is organized in a format the assistant can easily query (even when the user’s questions don’t exactly match the original wording), significantly improving the accuracy and usefulness of each response.

Imagen 3

Figure 3: Clinical Data Embedding Pipeline, Creation of the Chunks

Imagen 4

Figure 4: Clinical Data Embedding Pipeline, Creation of the embeddings

Creation Embeddings Flow

This second workflow allows users to interact directly with the system through Telegram, using a conversational assistant connected to language models and external tools. When a message is received, the AI Agent analyzes the request (supported by an Azure OpenAI model and an internal memory that maintains context) and decides what action to take. If the user asks about medical history, the agent queries the vector store stored in Pinecone (via the Medical_History node) to retrieve relevant information. If the request is related to a doctor’s availability, the agent connects to the medical database through the Agenda_Doctors node. Finally, the response is sent back through Telegram in natural language (clear and to the point), allowing for a conversational experience that is agile, helpful, and aligned with the needs of a clinical environment.

Imagen 5

Figure 5: AI-Powered Telegram Assistant for Clinical Queries

This image shows a real example of the assistant working within Telegram. Through a natural conversation, the bot is able to identify the patient by last name, retrieve their full name, and then provide the date and time of their last medical appointment (including the doctor’s name and specialty). All of this happens within seconds and without the need to navigate through portals or forms, demonstrating how the integration of AI, semantic search, and instant messaging can streamline access to clinical information in a fast and accurate way.

Imagen 6

Figure6: Real-Time Patient Query via Telegram Assistant

Conclusions

  • Intelligent automation improves efficiency in clinical environments
    By combining tools like n8n, Azure OpenAI, and Pinecone, it’s possible to build workflows that reduce repetitive tasks and provide faster access to medical information (without constant manual intervention).

  • Vectorizing clinical data enables more accurate queries
    Transforming medical records into embeddings allows for more effective semantic searches (even when users don’t phrase their questions exactly as written in the original text).

  • Conversational assistants offer a natural and accessible experience
    Integrating the workflow into platforms like Telegram lets users interact with the system in an intuitive and direct way (without technical barriers or complex interfaces).

  • Hourly updates ensure information is always current
    Running the embedding process every hour keeps the system in sync with the latest records (which improves the accuracy and relevance of the assistant’s responses).

  • A well-structured POC shows the real value of AI in healthcare
    Even as a prototype, this case demonstrates how artificial intelligence can be applied in a concrete and functional way in the healthcare sector (enhancing both user experience and internal processes).

]]>
https://blogs.perficient.com/2025/07/11/intelligent-automation-in-the-healthcare-sector-with-n8n-openai-and-pinecone/feed/ 0 384424
Turn Your Database into a Smart Chatbot with Azure OpenAI, LangChain, and ChromaDB https://blogs.perficient.com/2025/07/07/turn-your-database-into-a-smart-chatbot-with-openai-langchain-and-chromadb/ https://blogs.perficient.com/2025/07/07/turn-your-database-into-a-smart-chatbot-with-openai-langchain-and-chromadb/#respond Mon, 07 Jul 2025 15:33:22 +0000 https://blogs.perficient.com/?p=384077

Abstract

In recent years, language models and AI have reshaped how we interact with data. Yet, a significant portion of business knowledge still resides in relational databases (highly structured but not exactly user-friendly for those unfamiliar with SQL). So what if you could query that data using natural language, as if you were speaking to a smart assistant?

In this post, we’ll walk through a practical proof of concept (POC) using the well-known Northwind database as our data source. We’ll apply Retrieval-Augmented Generation (RAG) techniques to convert structured data into meaningful, searchable knowledge. By combining tools like LangChain, OpenAI Embeddings, and ChromaDB, we’ll build a system that can answer real business questions about customers, orders, and sales (all without writing a single SQL query).

This hands-on example will demonstrate how to turn raw database records into descriptive text, generate semantic embeddings, and store them in a vector database optimized for intelligent retrieval. Finally, we’ll connect a conversational AI model to deliver precise, context-aware answers in plain language.

Our goal isn’t just to show a technical integration (it’s much more than that). We aim to explore a new way of accessing business data that’s intuitive, scalable, and aligned with the future of intelligent interfaces. If you’ve ever wondered how to bring your data to life through conversational AI, this is the roadmap you’ve been looking for.

Introduction

Relational databases have been the backbone of enterprise systems for decades, offering structured and reliable storage for vast amounts of business information. However, accessing that data still relies heavily on technical users who are comfortable writing SQL queries (which limits its utility for many other roles within an organization). This post aims to demonstrate how we can transform structured data into accessible knowledge through AI-powered conversational interfaces.

To do this, we’ll build a practical proof of concept using the classic Northwind database (a well-known dataset that includes customers, orders, products, and more). Instead of querying the data directly with SQL, we’ll generate readable and semantically rich descriptions that can be interpreted by a language model. These textual fragments will be converted into embeddings using OpenAI, stored in a vector database powered by ChromaDB, and made retrievable through LangChain, all using Python as the orchestration layer.

Why apply embeddings to tabular data? Because it allows us to move beyond the rigid structure of SELECT statements and JOINs (toward a system where users can ask questions like “Which customer bought the most in 1997?” and receive clear, context-aware answers in natural language). This approach does not replace traditional querying techniques (instead, it complements them), making data access more inclusive and aligned with modern AI-driven experiences.

Ultimately, this opens the door to a new way of working with data (one where conversation replaces complexity and insight becomes truly accessible).

What Are Embeddings?

One of the fundamental challenges in working with language models or textual data is representation. Computers operate on numbers, but words have no inherent numerical value. To bridge this gap, we need a way to transform text into a format that machines can understand while still preserving its semantic meaning. This is precisely where embeddings come into play.

Embeddings are dense vector representations of words, sentences, or even entire documents. Each piece of text is mapped to a real-valued vector—often in a space with hundreds or thousands of dimensions—where semantic relationships can be modeled geometrically. Unlike older methods like one-hot encoding, embeddings allow us to capture similarity: words such as “king” and “queen” appear close together, while unrelated terms like “king” and “lettuce” remain far apart.

The real power of embeddings lies in their ability to reflect meaning. This enables models to reason not just about the surface of the text, but about what it implies—unlocking applications in translation, sentiment analysis, document classification, recommendation systems, semantic search, and, crucially, retrieval-augmented conversational agents.

In the context of this blog, we use embeddings to convert structured database records into descriptive, semantically rich text fragments. These fragments are embedded using a model like OpenAI’s and stored in a vector database. When a user asks a question, it too is embedded, and the system retrieves the most relevant fragments to generate a natural-language answer. This technique is part of what’s known as Retrieval-Augmented Generation (RAG).

Embeddings are commonly produced by pre-trained models from providers like OpenAI, Hugging Face, or Cohere. In our case, we rely on OpenAIEmbeddings, which leverage large-scale transformer models trained on diverse, multilingual datasets optimized for semantic understanding.

One of the greatest advantages of embeddings is their ability to generalize. For instance, a user might ask “Who was the top customer in 1997?” and the system can infer related notions like “highest purchase volume” or “most frequent buyer” without needing exact word matches. This goes far beyond traditional keyword-based search.

Modern embeddings are also contextual. In models like BERT, ELMo, or GPT, the vector for a word depends on its surrounding text. The word “bank” in “sat on the bank” and “deposited money in the bank” will generate entirely different embeddings. This dynamic understanding of context is one reason why these models perform so well in complex language tasks.

In our use case, we apply embeddings to fragments derived from SQL queries, effectively transforming structured information into semantically searchable knowledge. This enables a more natural interaction with data, where users don’t need to understand database schemas or SQL syntax to retrieve meaningful insights.

The pipeline involves embedding each text chunk, storing the resulting vectors in a vector store like ChromaDB, and embedding the user’s query to perform a similarity search. The most relevant matches are passed to a language model, which uses them as context to generate an intelligent, context-aware response.

This method not only streamlines access to information but also enhances accuracy by leveraging the semantic proximity between questions and data fragments.

Let’s understand, what is a chunk?

In the context of language models and semantic search, chunks refer to small, meaningful segments of text that have been split from a larger document. Rather than processing an entire file or paragraph at once, the system breaks down the content into manageable pieces (usually a few hundred characters long with some overlap between them). This technique allows the model to better understand and retrieve relevant information during a query.

Chunking is essential when working with long documents or structured data transformed into natural language. It ensures that each piece maintains enough context to be useful while staying within the token limits of the language model. For example, an entire order history from a database might be divided into chunks that describe individual transactions, making it easier for the system to locate and reason over specific details.

This process not only improves the efficiency of embedding generation and similarity search but also enhances the relevance of responses provided by the conversational agent.

LangChain and ChromaDB: Connecting Language Models to Meaningful Data

To build a system where users can ask questions in natural language and receive intelligent, relevant answers, we need more than just a powerful language model (we need a framework that can manage context, memory, and retrieval). That’s where LangChain comes in.

LangChain is an open-source framework designed to help developers integrate large language models (LLMs) with external data sources and workflows. Rather than treating the model as a black box, LangChain provides structured components (like prompt templates, memory modules, agents, and chains) that make it easier to build dynamic, stateful, and context-aware applications. One of its most popular use cases is Retrieval-Augmented Generation (RAG) (where the model uses external knowledge retrieved from a document store to improve its responses).

To make this retrieval process efficient and accurate, LangChain works seamlessly with ChromaDB.

ChromaDB is a lightweight, high-performance vector database optimized for storing and querying embeddings. It enables fast similarity searches (allowing the system to retrieve the most semantically relevant pieces of information based on a user’s query). This makes Chroma ideal for use in search engines, recommendation systems, and conversational agents.

In a LangChain workflow, ChromaDB serves as the brain’s long-term memory. It stores the embedded representations of documents or data fragments, and returns the most relevant ones when queried. These fragments are then injected into the language model as context (resulting in more accurate and grounded responses).

Together, LangChain and ChromaDB bridge the gap between raw data and intelligent conversation.

Proof of Concept: From SQL Rows to Smart Conversations

In this section, we’ll walk through the steps of building a fully functional proof of concept. Our goal: enable users to ask questions in plain language (such as “Which customers placed the most orders in 1997?”) and get accurate answers generated using data from a relational database.

We’ll use the classic Northwind database, which contains tables for customers, orders, products, and more. Instead of querying it directly with SQL, we’ll extract meaningful data, turn it into descriptive text fragments, generate semantic embeddings, and store them in ChromaDB. Then, we’ll use LangChain to retrieve relevant chunks and feed them to OpenAI’s language model (turning structured data into natural conversation).

For this Proof of Concept you must follow the following steps:

Step 0: Environment setup

Before diving into building the intelligent assistant, it’s essential to prepare a clean and isolated development environment. This ensures that all dependencies are properly aligned, and avoids conflicts with other global Python packages or projects. Here’s how to set up everything from scratch.

Create Embedding Depployment

Before we can generate embeddings from our text data, we need to create a deployment for an embedding model within our Azure OpenAI resource. This deployment acts as the gateway through which we send text and receive vector representations in return.

 

Imagen5

Figure 1: Azure OpenAI – Embedding and Chat Model Deployments.

To begin, navigate to your Azure OpenAI resource in the Azure Portal. Select the Deployments tab, then click + Create to initiate a new deployment. Choose the model text-embedding-3-small from the dropdown list (this is one of the most efficient and semantically rich models currently available). Assign a unique name to your deployment—for example, text-embedding-3-small—and ensure you take note of this name, as it will be required later in your code.

 

Imagen6

Figure 2: Selecting the Text Embedding Model (text-embedding-3-small) in Azure OpenAI.

Once deployed, Azure will expose a dedicated endpoint along with your API key. These credentials will allow your application to communicate securely with the deployed model. Be sure to also confirm the API version associated with the model (such as 2024-02-01) and verify that this matches the version specified in your code or environment variables.

By completing this step, you set up the foundation for semantic understanding in your application. The embedding model will convert text into high-dimensional vectors that preserve the meaning and context of the input, enabling powerful similarity search and retrieval capabilities later on in your pipeline.

LLM Model Deployment

Don’t forget to configure the LLM model as well (such as gpt-4.1-mini), since it will be responsible for generating responses during the interaction phase of the implementation.

Imagen7

Figure 3: Deploying the LLM Base Model in Azure OpenAI (gpt-4.1-mini).

Imagen8

Figure 4: Selecting a Chat completion Model from Azure OpenAI Catalog.

To connect your application with the deployed LLM, you will need the endpoint URL and the API key shown in the deployment details. This information is essential for authenticating your requests and sending prompts to the model. In this case, we are using the gpt-4.1-mini deployment with the Azure OpenAI SDK and API key authentication. Once retrieved, these credentials allow your code to securely interact with the model and generate context-aware responses as part of the proof of concept.

Imagen6

Figure 5: Accessing Endpoint and API Key for gpt-4.1-mini Deployment.

The key information we need from this screenshot to correctly configure our code in the Proof of Concept is the following:

  1. Endpoint URL (Target URI)
    (Located under the “Endpoint” section)
    This is the base URL you will use to send requests to the deployed model. It’s required when initializing the client in your code.

  2. API Key
    (Hidden under the “Key” field)
    This is your secret authentication token. You must pass it securely in your code to authorize requests to the Azure OpenAI service.

  3. Deployment Name
    (Shown as “gpt‑4.1‑mini” in the “Deployment info” section)
    You will need this name when specifying which model deployment your client should interact with (e.g., when using LangChain or the OpenAI SDK).

  4. Provisioning Status
    (Shows “Succeeded”)
    Confirms that the deployment is ready to use. If this status is anything other than “Succeeded,” the model is not yet available.

  5. Model Version and Creation Timestamp
    (Optional, for auditing or version control)
    Useful for documentation, debugging, or future migration planning.

Create a requirements.txt file

Start by listing all the libraries your project will depend on. Save the following content as requirements.txt in your project root:

pyodbc
langchain==0.3.25
openai==1.82.0
chromadb==1.0.10
tiktoken
pydantic
langchain-core
langchain-community
langchain-text-splitters
langchain-openai==0.1.8

This file defines the exact versions needed for everything—from LangChain and ChromaDB to the Azure OpenAI integration.

Create a virtual environment

To avoid interfering with other Python installations on your machine, use a virtual environment. Run the following command in your terminal:

python -m venv venv

This creates a dedicated folder called venv that holds a self-contained Python environment just for this project.

Activate the virtual environment

Next, activate the environment:

venv\Scripts\activate

Once activated, your terminal prompt should change to reflect the active environment.

Install dependencies

Now install all required libraries in one go by running:

pip install -r requirements.txt

This will install all the packages listed in your requirements file, ensuring your environment is ready to connect to the Northwind database and work with LangChain and Azure OpenAI. With everything in place, you’re ready to move on to building the assistant—from querying structured data to transforming it into natural, intelligent responses.

Step 1: Azure OpenAI Configuration

Before diving into code, we need to configure the environment so that our application can access Azure OpenAI services securely and correctly. This involves setting three essential environment variables:

  • AZURE_OPENAI_API_KEY (your Azure OpenAI API key)

  • AZURE_OPENAI_ENDPOINT (the full endpoint URL of your Azure OpenAI resource)

  • AZURE_OPENAI_API_VERSION (the specific API version compatible with your deployed models)

These variables are defined directly in Python using the os.environ method:

# --- 1. Environment setup ---
os.environ["AZURE_OPENAI_API_KEY"] = "<ApiKey>"
os.environ["AZURE_OPENAI_ENDPOINT"] = "<Endpoint>"
os.environ["AZURE_OPENAI_API_VERSION"] = "<AzureOpenIAVersion>"

By setting these values, LangChain will know how to connect to your Azure deployment and access the correct models for embeddings and chat completion.

It’s important to ensure that the deployment names used in your code match exactly the ones configured in your Azure portal. With this step complete, you’re now ready to start connecting to your database and transforming structured data into natural language knowledge.

Step 2: Connecting to the database

With the environment ready, the next step is to connect to the Northwind database and retrieve meaningful records. Northwind is a well-known sample dataset that contains information about customers, employees, orders, products, and their relationships. It offers a rich source of structured data for demonstrating how to turn database rows into conversational context.

To begin, we establish a connection with a local SQL Server instance using pyodbc (a Python driver for ODBC-based databases). Once connected, we execute a SQL query that joins several related tables (Orders, Customers, Employees, Order Details, and Products). This query returns detailed records for each order (including the customer who placed it, the salesperson involved, the date, and the specific products purchased with their quantities, prices, and discounts).

By retrieving all of this information in a single query, we ensure that each order contains enough context to be transformed later into meaningful text that a language model can understand.

# --- 2. Database connection ---
conn = pyodbc.connect(
    "DRIVER={ODBC Driver 17 for SQL Server};SERVER=localhost;DATABASE=Northwind;Trusted_Connection=yes;"
)
cursor = conn.cursor()
cursor.execute("""
    SELECT 
        o.OrderID,
        c.CompanyName AS Customer,
        e.FirstName + ' ' + e.LastName AS Salesperson,
        o.OrderDate,
        p.ProductName,
        od.Quantity,
        od.UnitPrice,
        od.Discount
    FROM Orders o
    JOIN Customers c ON o.CustomerID = c.CustomerID
    JOIN Employees e ON o.EmployeeID = e.EmployeeID
    JOIN [Order Details] od ON o.OrderID = od.OrderID
    JOIN Products p ON od.ProductID = p.ProductID
""")
records = cursor.fetchall()

Step 3: Transforming records into text

Relational databases are optimized for storage and querying, not for natural language understanding. To bridge that gap, we need to convert structured rows into readable, descriptive sentences that capture the meaning behind the data.

In this step, we take the SQL query results and group them by order. Each order includes metadata (such as customer name, date, and salesperson) along with a list of purchased products. We format this information into short narratives that resemble human-written descriptions.

For example, an entry might read:
“Order 10250 was placed by ‘Ernst Handel’ on 1996-07-08. Salesperson: Nancy Davolio. Items included: 10 x Camembert Pierrot at $34.00 each with 0% discount.”

By doing this, we make the data semantically rich and accessible to the language model. Instead of dealing with abstract IDs and numeric values, the model now sees contextually meaningful information about who bought what, when, and under what conditions. These text fragments are the foundation for generating accurate embeddings and useful answers later on.

Then, the next part of the code in charge of performing this task, is as follows

# --- 3. Transform records into text ---
from collections import defaultdict

orders_data = defaultdict(list)
for row in records:
    key = (row.OrderID, row.Customer, row.Salesperson, row.OrderDate)
    orders_data[key].append((row.ProductName, row.Quantity, row.UnitPrice, row.Discount))

documents = []
for (order_id, customer, seller, date), items in orders_data.items():
    lines = [f"Order {order_id} was placed by '{customer}' on {date:%Y-%m-%d}. Salesperson: {seller}."]
    lines.append("Items included:")
    for product, qty, price, discount in items:
        lines.append(f" - {qty} x {product} at ${price:.2f} each with {discount*100:.0f}% discount.")
    documents.append(" ".join(lines))

Relational databases are optimized for storage and querying, not for natural language understanding. To bridge that gap, we need to convert structured rows into readable, descriptive sentences that capture the meaning behind the data.

In this step, we take the SQL query results and group them by order. Each order includes metadata (such as customer name, date, and salesperson) along with a list of purchased products. We format this information into short narratives that resemble human-written descriptions.

For example, an entry might read:
“Order 10250 was placed by ‘Ernst Handel’ on 1996-07-08. Salesperson: Nancy Davolio. Items included: 10 x Camembert Pierrot at $34.00 each with 0% discount.”

By doing this, we make the data semantically rich and accessible to the language model. Instead of dealing with abstract IDs and numeric values, the model now sees contextually meaningful information about who bought what, when, and under what conditions. These text fragments are the foundation for generating accurate embeddings and useful answers later on.

Step 4: Splitting text into chunks

Once we have our text-based descriptions, the next challenge is to prepare them for embedding. Language models (and vector databases) perform best when working with manageable segments of text rather than long, unstructured paragraphs.

To achieve this, we break each document into smaller chunks using a character-based splitter. In this case, we set a chunk size of 300 characters with an overlap of 50 characters. The overlap ensures that important information near the edges of a chunk isn’t lost when transitioning between segments.

For example, if an order includes many products or a particularly detailed description, it may span multiple chunks. These overlapping fragments preserve continuity and improve the accuracy of downstream retrieval.

# --- 4. Split texts into chunks ---
splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=50)
docs = splitter.create_documents(documents)

This process not only improves the quality of the embeddings but also enhances retrieval performance later on, when a user asks a question and the system needs to locate the most relevant context quickly.

By preparing clean, consistent input in this way, we’re setting the stage for a robust and semantically aware assistant that understands each order at a granular level.

Step 5: Creating embeddings with Azure OpenAI

With our text chunks ready, the next step is to convert them into numerical representations that capture their semantic meaning. These representations, known as embeddings, allow the system to measure how similar one piece of text is to another—not by exact words, but by meaning.

To generate these embeddings, we use the text-embedding-3-small model deployed on Azure OpenAI. This model transforms each chunk into a high-dimensional vector, where semantically similar chunks are positioned close together in vector space.

# --- 5. Create embeddings with Azure OpenAI ---
embeddings = AzureOpenAIEmbeddings(
    deployment="text-embedding-3-small",  # Deployment correcto para embeddings
    model="text-embedding-3-small",
    api_version="2024-02-01"
)

For instance, two orders that include similar products or are placed by the same customer will produce embeddings that are close in distance. This similarity is what allows the assistant to later retrieve relevant information based on a natural language query, even if the wording is different.

Using Azure OpenAI for embeddings offers both scalability and enterprise-grade integration. It also ensures compatibility with the rest of our LangChain pipeline, as the embeddings can be seamlessly stored and queried within a vector database like Chroma.

This step essentially transforms our structured business data into a format that the language model can reason over—making it a critical part of the entire retrieval-augmented workflow.

Step 6: Storing embeddings in Chroma

Once the embeddings are generated, they need to be stored in a way that allows for fast and accurate retrieval. This is where Chroma comes in—a lightweight, high-performance vector database built specifically for handling semantic search.

Each text chunk, along with its corresponding embedding, is stored in Chroma using a local persistence directory. By doing so, we’re creating a searchable memory that allows the system to quickly find the most relevant fragments when a question is asked.

# --- 6. Store the embeddings in Chroma ---
db = Chroma.from_documents(docs, embeddings, persist_directory="./northwind_db")

Chroma supports similarity search out of the box. When a user submits a query, the system converts that query into its own embedding and searches the database for nearby vectors (in other words, the most semantically related pieces of content).

This design mimics how our own memory works—we don’t recall entire databases, just the most relevant bits based on context and meaning. Storing the embeddings in Chroma gives our assistant the same ability.

By the end of this step, we’ve effectively turned structured business data into a knowledge base that can be queried using natural language, enabling more intelligent and human-like interactions.

Step 7: Configuring the question-answering engine with AzureChatOpenAI

At this stage, we have a searchable knowledge base ready to go. Now it’s time to build the brain of the assistant—the component that takes a user’s question, retrieves the relevant context, and generates a natural, intelligent response.

We use AzureChatOpenAI, a LangChain-compatible wrapper for Azure-hosted GPT models. In this example, we configure it to use the gpt-4.1-mini deployment. This model serves as the core reasoning engine, capable of understanding user queries and formulating answers based on the data retrieved from Chroma.

LangChain’s RetrievalQA chain orchestrates the interaction. When a question is submitted, the process works as follows:

  1. The system converts the query into an embedding.

  2. Chroma searches for the most relevant chunks.

  3. The retrieved chunks are passed as context to the GPT model.

  4. The model generates a concise and informative response.

# --- 7. Configure RetrievalQA with AzureChatOpenAI ---
retriever = db.as_retriever()
llm = AzureChatOpenAI(
    deployment_name="gpt-4.1-mini",  # Deployment LLM
    model="gpt-4",                   # Model LLM
    api_version="2024-02-01",
    temperature=0
)
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)

This architecture is what makes Retrieval-Augmented Generation (RAG) so effective. Rather than relying solely on the model’s training data, it supplements it with real, dynamic business information—allowing it to give accurate and up-to-date answers.

By combining a high-quality language model with focused contextual data, we give our assistant the tools to reason, explain, and even summarize complex order information without writing a single SQL query.

Step 8: Question loop

With everything in place, the final step is to set up a simple interaction loop that allows users to engage with the assistant through natural language. This loop waits for a user’s input, processes the question, retrieves the most relevant data from Chroma, and generates an answer using Azure OpenAI.

The experience is intuitive—users don’t need to know SQL or the structure of the database. Instead, they can simply ask questions like:

  • Which employee achieved the highest total unit sales?
  • Which discounted products were the most frequently sold?
  • Which customer purchased the widest variety of products?
  • Which order had the highest overall value?
  • Who processed the most orders in July 1996?

Behind the scenes, the assistant interprets the question, finds the best-matching entries from the embedded knowledge base, and composes a response based on actual transactional data.

# --- 8. Question loop ---
print("Ask me about any order from the Northwind database.")
while True:
    question = input("\nYour question: ")
    if question.lower() == "exit":
        break
    result = qa(question)
    print("\n💡 Aswer:\n", result["result"])

This approach makes data exploration conversational. It lowers the barrier for interacting with structured information, opening new possibilities for customer support, sales analysis, and executive reporting—all through plain language.

At the end of this loop, we’ve built something more than just a chatbot. It’s a practical proof of concept showing how large language models can bring structured data to life in real time, transforming static records into dynamic, human-centered knowledge.

Northwind Assistant in Action: Sample Questions and Answers

This section showcases the assistant in action. By combining data from the Northwind database with Azure OpenAI and Chroma, we’ve built a system that understands natural language and responds with precise, contextual answers. Instead of writing complex SQL queries, users can now explore business insights simply by asking questions. Below are some example queries and the kind of intelligent responses the assistant is capable of generating.

Imagen1

Figure 6: User prompt asking which discounted products were sold the most.

Imagen2

Figure 7: AI-generated answer showing the top-selling products with discounts applied.

Imagen3

Figure 8: Natural language question about which order had the highest total value.

Imagen4

Figure 9: Response calculating and displaying the order with the highest overall value.

Conclusions

Combining SQL with LLMs unlocks new value from legacy data
By extracting and transforming structured information from a traditional database like Northwind, we demonstrated how even decades-old datasets can become fuel for modern AI-driven insights (without rewriting backend systems).

Semantic search enhances how we ask and answer questions
Using embeddings and a vector store (in this case, ChromaDB), the assistant is able to retrieve contextually relevant chunks of information instead of relying on rigid keyword matches. This allows for more flexible and intuitive interactions.

Natural language becomes the new interface for analytics
Thanks to Azure OpenAI’s chat capabilities, users no longer need to write complex SQL queries to understand data. Instead, they can simply ask questions in plain English (and get coherent answers backed by structured sources).

Modularity and scalability are built into the architecture
Each step of the assistant—data extraction, transformation, embedding, storage, and retrieval—is modular. This makes it easy to extend to new datasets, scale up in the cloud, or integrate into enterprise tools and workflows.

This approach bridges the gap between business users and data
Perhaps most importantly, this proof of concept shows that language models can act as intelligent intermediaries (allowing non-technical users to access meaningful insights from complex databases, instantly and conversationally).

References

Microsoft. (2024).What is Azure OpenAI Service? Microsoft Learn.
https://learn.microsoft.com/en-us/azure/ai-services/openai/overview

Chroma. (2024).ChromaDB: The AI-native open-source embedding database.
https://docs.trychroma.com/

OpenAI. (2024). Text embeddings documentation. OpenAI API Reference.
https://platform.openai.com/docs/guides/embeddings/what-are-embeddings

FutureSmart AI. (2024). ChromaDB: An open-source vector embedding database. FutureSmart AI Blog. https://blog.futuresmart.ai/chromadb-an-open-source-vector-embedding-database

FutureSmart AI. (2023). Master RAG with LangChain: A practical guide. FutureSmart AI Blog. https://blog.futuresmart.ai/master-rag-with-langchain-a-practical-guide

Jurafsky, D., & Martin, J. H. (2023). Speech and Language Processing (3rd ed., Draft). Stanford University. Retrieved from https://web.stanford.edu/~jurafsky/slp3/

]]>
https://blogs.perficient.com/2025/07/07/turn-your-database-into-a-smart-chatbot-with-openai-langchain-and-chromadb/feed/ 0 384077
Intelligent Agents with n8n: AI-Powered Automation https://blogs.perficient.com/2025/07/04/intelligent-agents-with-n8n-ai-powered-automation/ https://blogs.perficient.com/2025/07/04/intelligent-agents-with-n8n-ai-powered-automation/#respond Sat, 05 Jul 2025 00:23:01 +0000 https://blogs.perficient.com/?p=384044

Abstract

We live in a time when automating processes is no longer a luxury, but a necessity for any team that wants to remain competitive. But automation has evolved. It is  no longer just about executing repetitive tasks, but about creating solutions that understand context, learn over time, and make smarter decisions. In this blog, I want to show you how n8n (a visual and open-source automation tool) can become the foundation for building intelligent agents powered by AI.

We will explore what truly makes an agent “intelligent,” including how modern AI techniques allow agents to retrieve contextual information, classify tickets, or automatically respond based on prior knowledge.

I will also show you how to connect AI services and APIs from within a workflow in n8n, without the need to write thousands of lines of code. Everything will be illustrated with concrete examples and real-world applications that you can adapt to your own projects.

This blog is an invitation to go beyond basic bots and start building agents that truly add value. If you are exploring how to take automation to the next level, thiss journey will be of great interest to you.

 

Introduction

Automation has moved from being a trend to becoming a foundational pillar for development, operations, and business teams. But amid the rise of tools that promise to do more with less, a key question emerges: how can we build truly intelligent workflows that not only execute tasks but also understand context and act with purpose? This is where AI agents begin to stand out.

This blog was born from that very need. Over the past few months, I’ve been exploring how to take automation to the next level by combining two powerful elements: n8n (a visual automation platform) and the latest advances in artificial intelligence. This combination enables the design of agents capable of understanding, relating, and acting based on the content they receive—with practical applications in classification, search, personalized assistance, and more.

In the following sections, I’ll walk you through how these concepts work, how they connect with each other, and most importantly, how you can apply them yourself (without needing to be an expert in machine learning or advanced development). With clear explanations and real-world examples built with n8n, this blog aims to be a practical, approachable guide for anyone looking to go beyond basic automation and start building truly intelligent solutions.

What are AI Agents?

An AI agent is an autonomous system (software or hardware) that perceives its environment, processes information, and makes decisions to achieve specific goals. It does not merely react to basic events; it can analyze context, query external sources, and select the most appropriate action. Unlike traditional bots, intelligent agents integrate reasoning and sometimes memory, allowing them to adapt and make decisions based on accumulated experience (Wooldridge & Jennings, 1995; Cheng et al., 2024).

In the context of n8n, an AI agent translates into workflows that not only execute tasks but also interpret data using language models and act according to the situation, enabling more intelligent and flexible processes.

From Predictable to Intelligent: Traditional Bot vs. Context-Aware AI Agent

A traditional bot operates based on a set of rigid rules and predefined responses, which limits its ability to adapt to unforeseen situations or understand nuances in conversation. Its interaction is purely reactive: it responds only to specific commands or keywords, without considering the conversation’s history or the context in which the interaction occurs. In contrast, a context-aware artificial intelligence agent uses advanced natural language processing techniques and conversational memory to adapt its responses according to the flow of the conversation and the previous information provided by the user. This allows it to offer a much more personalized, relevant, and coherent experience, overcoming the limitations of traditional bots. Context-aware agents significantly improve user satisfaction, as they can understand intent and dynamically adapt to different conversational scenarios (Chen, Xu, & Wang, 2022).

Imagen1

Figure 1: Architecture of an intelligent agent with hybrid memory in n8n (Dąbrowski, 2024).

How Does n8n Facilitate the Creation of Agents?

n8n is an open-source automation platform that enables users to design complex workflows visually, without the need to write large amounts of code. It simplifies the creation of intelligent agents by seamlessly integrating language models (such as OpenAI or Azure OpenAI), vector databases, conditional logic, and contextual memory storage.

With n8n, an agent can receive text input, process it using an AI model, retrieve relevant information from a vector store, and respond based on conversational history. All of this is configured through visual nodes within a workflow, making advanced solutions accessible even to those without a background in artificial intelligence.

Thanks to its modular and flexible design, n8n has become an ideal platform for building agents that not only automate tasks but also understand, learn, and act autonomously.

 

Imagen2

Figure 2: Automated workflow in n8n for onboarding and permission management using Slack, Jira, and ServiceNow (TextCortex, 2025).

Integrations with OpenAI, Python, External APIs, and Conditional Flows

One of n8n’s greatest strengths is its ability to connect with external tools and execute custom logic. Through native integrations, it can interact with OpenAI (or Azure OpenAI), enabling the inclusion of language models for tasks such as text generation, semantic classification, or automated responses.

Additionally, n8n supports custom code execution through Python or JavaScript nodes, expanding its capabilities and making it highly adaptable to different use cases. It can also communicate with any external service that provides a REST API, making it ideal for enterprise-level integrations.

Lastly, its conditional flow system allows for dynamic branching within workflows, evaluating logical conditions in real time and adjusting the agent’s behavior based on the context or incoming data.

Imagen3

Figure 3: Basic conversational agent flow in n8n with language model and contextual memory.

This basic flow in n8n represents the core logic of a conversational intelligent agent. The process begins when a message is received through the “When chat message received” node. That message is then passed to the AI Agent node, which serves as the central component of the system.

The agent is connected to two key elements: a language model (OpenAI Chat Model) that interprets and generates responses, and a simple memory that allows it to retain the context or conversation history. This combination enables the agent not only to produce relevant responses but also to remember previous information and maintain coherence across interactions.

This type of architecture demonstrates how, with just a few nodes, it is possible to build agents with contextual behavior and basic reasoning capabilities—ideal for customer support flows, internal assistants, or automated conversational interfaces.

Before the agent can interact with users, it needs to be connected to a language model. The following shows how to configure this integration in n8n.

Configuring the Language Model in the AI Agent

As developers at Perficient, we have the advantage of accessing OpenAI services through the Azure platform. This integration allows us to leverage advanced language models in a secure, scalable manner, fully aligned with corporate policies, and facilitates the development of artificial intelligence solutions tailored to our needs.

One of the fundamental steps in building an AI agent in n8n is to define the language model that will be used to process and interpret user inputs. In this case, we use the OpenAI Chat Model node, which enables the agent to connect with advanced language models available through the Azure OpenAI API.

When configuring this node, n8n will require an access credential, which is essential for authenticating the connection between n8n and your Azure OpenAI service. If you do not have one yet, you can create it from the Azure portal by following these steps:

  • Go to the Azure portal. If you do not yet have an Azure OpenAI resource, create one by selecting “Create a resource“, searching for “Azure OpenAI”, and following the setup wizard to configure the service with your subscription parameters. Then access the implemented resource.
  • Go to url https://ai.azure.com and sign in with your Azure account. Select the Azure OpenAI resource you created and, from the side menu, navigate to the “Deployments” section. There you must create a new deployment, selecting the language model you want to use (for example, GPT 3.5 or GPT 4) and assigning it a unique deployment name. You can also click on the Command-Go to Azure AI Foundry portal option as shown in the image.

Imagen4

Figure 4: Access to the Azure AI Foundry portal from the Azure OpenAI resource.

  • Once the deployment is created, go to “API Keys & Endpoints” to copy the access key (API Key) and the endpoint corresponding to your resource.

Imagen5

Figure 5: Visualization of model deployments in Azure AI Foundry.

Once the model deployment has been created in Azure AI Foundry, it is essential to access the deployment details in order to obtain the necessary information for integrating and consuming the model from external applications. This view provides the API endpoint, the access key (API Key), as well as other relevant technical details of the deployment, such as the assigned name, status, creation date, and available authentication parameters.

This information is crucial for correctly configuring the connection from tools like n8n, ensuring a secure and efficient integration with the language model deployed in Azure OpenAI.

Imagen6

Figure 6: Azure AI Foundry deployment and credentialing details.

  • Step 1:

    In n8n, select the “+ Create new credential” option in the node configuration, and enter the endpoint, the API key, and the deployment name you configured. But first we must create the AI Agent:

Imagen8

Figure 7: Chat with AI Agent.

Step 2:

After the creation of the Agent, the model is added, as shown in the figure above. Within the n8n environment, integration with language models is accomplished through specialized nodes for each AI provider.

To connect our agent with Azure OpenAI services, it is necessary to select the Azure OpenAI Chat Model node in the Language Models section.

This node enables you to leverage the advanced capabilities of language models deployed in Azure, making it easy to build intelligent and customizable workflows for various corporate use cases. Its configuration is straightforward and, once properly authenticated, the agent will be ready to process requests using the selected model from Azure’s secure and scalable infrastructure.

Imagen7

Figure 8: Selection of the Azure OpenAI Chat Model node in n8n.

Step 3:

Once the Azure OpenAI Chat Model node has been selected, the next step is to integrate it into the n8n workflow as the primary language model for the AI agent.

The following image illustrates how this model is connected to the agent, allowing chat inputs to be processed intelligently by leveraging the capabilities of the model deployed in Azure. This integration forms the foundation for building more advanced and personalized conversational assistants in enterprise environments.

Imagen9

Figure 9: Selection of the Azure OpenAI Chat Model node in n8n.

Step 4:

When configuring the Azure OpenAI Chat Model node in n8n, it is necessary to select the access credential that will allow the connection to the Azure service.

If a credential has not yet been created, you can do so directly from this panel by selecting the “Create new credential” option.

This step is essential to authenticate and authorize the use of language models deployed in Azure within your automation workflows.

Imagen10

Figura 10: Selection of the Azure OpenAI Chat Model node in n8n.

Step 5:

To complete the integration with Azure OpenAI in n8n, it is necessary to properly configure the access credentials.

The following screen shows the required fields, where you must enter the API Key, resource name, API version, and the corresponding endpoint.

This information ensures that the connection between n8n and Azure OpenAI is secure and functional, enabling the use of language models deployed in the Azure cloud.

Imagen11

Figure 11: Azure OpenAI credentials configuration in n8n.

Step 6:

After creating and selecting the appropriate credentials, the next step is to configure the model deployment name in the Azure OpenAI Chat Model node.

In this field, you must enter exactly the name assigned to the model deployment in Azure, which will allow n8n to correctly use the deployed instance to process natural language requests. Remember to select the Model of the selected implementation in Azure OpenIA, in this case gtp-4.1-mini:

Imagen12

Figure 12: Configuration of the deployment name in the Azure OpenAI Chat Model node.

Step 7:

Once the language model is connected to the AI agent in n8n, you can enhance its capabilities by adding memory components.

Integrating a memory system allows the agent to retain relevant information from previous interactions, which is essential for building more intelligent and contextual conversational assistants.

In the following image, the highlighted area shows where a memory module can be added to enrich the agent’s behavior.

Imagen13

Figure 13: Connecting the memory component to the AI agent in n8n.

Step 8:

To start equipping the AI agent with memory capabilities, n8n offers different options for storing conversation history.

The simplest alternative is Simple Memory, which stores the data directly in n8n’s internal memory without requiring any additional credentials.

There are also more advanced options available, such as storing the history in external databases like MongoDB, Postgres, or Redis, which provide greater persistence and scalability depending on the project’s requirements.

Imagen14

Figure 14: Memory storage options for AI agents in n8n.

Step 9:

The configuration of the Simple Memory node in n8n allows you to easily define the parameters for managing the conversational memory of the AI agent.

In this interface, you can specify the session identifier, the field to be used as the conversation tracking key, and the number of previous interactions the model will consider as context.

These settings are essential for customizing information retention and improving continuity in the user’s conversational experience.

Imagen15

Figure 15: Memory storage options for AI agents in n8n.

Step 10:

The following image shows the successful execution of a conversational workflow in n8n, where the AI agent responds to a chat message using a language model deployed on Azure and manages context through a memory component.

You can see how each node in the workflow performs its function and how the conversation history is stored, enabling the agent to provide more natural and contextual responses.

Imagen16

Figure 16: Execution of a conversational workflow with Azure OpenAI and memory in n8n.

Once a valid credential has been added and selected, the node will be ready to send requests to the chosen language model (such as GPT 3.5 or GPT 4) and receive natural language responses, allowing the agent to continue the conversation or execute actions automatically.

With this integration, n8n becomes a powerful automation tool, enabling use cases such as conversational assistants, support bots, intelligent classification, and much more.

Integration of the AI agent into the web application through an n8n workflow triggered by a webhook.

Before integrating the AI agent into a web application, it is essential to have a ready-to-use n8n workflow that receives and responds to messages via a Webhook. Below is a typical workflow example where the main components for conversational processing are connected.

For the purposes of this blog, we will assume that both the Webhook node (which receives HTTP requests) and the Set/Edit Fields node (which prepares the data for the agent) have already been created. As shown in the following image, the workflow continues with the configuration of the language model (Azure OpenAI Chat Model), memory management (Simple Memory), processing via the AI Agent node, and finally, sending the response back to the user using the Respond to Webhook node.

Imagen17

Figure 17: n8n Workflow for AI Agent Integration with Webhook.

Before connecting the web interface to the AI agent deployed in n8n, it is essential to validate that the Webhook is working correctly. The following image shows how, using a tool like Postman, you can send an HTTP POST request to the Webhook endpoint, including the user’s message and the session identifier. As a result, the flow responds with the message generated by the agent, demonstrating that the end-to-end integration is functioning properly.

Imagen7

Figure 18: Testing the n8n Webhook with Postman.

    1. Successful Test of the n8n Chatbot in a WebApp: The following image shows the functional integration between a chatbot built in n8n and a custom web interface using Bootstrap. By sending messages through the application, responses from the AI agent deployed on Azure OpenAI are displayed in real time, enabling a seamless and fully automated conversational experience directly from the browser.
      Imagen19
      Figure 19: n8n Chatbot Web Interface Working with Azure OpenAI.
    2. Introductory Text: Before consuming the agent from a web page or an external application, it is essential to ensure that the flow in n8n is activated. As shown in the image, the “Active” button must be enabled (green) so that the webhook works continuously and can receive requests at any time. Additionally, remember that when deploying to a production environment, you must change the webhook URL, using the appropriate public address instead of “localhost”, ensuring external access to the flow.

      Imagen20
      Figure 20: Activation and Execution Tracking of the Flow in n8n.

Conclusions

Intelligent automation is essential for today’s competitiveness

Automating tasks is no longer enough; integrating intelligent agents allows teams to go beyond simple repetition, adding the ability to understand context, learn from experience, and make informed decisions to deliver real value to business processes.

Intelligent agents surpass the limitations of traditional bots

Unlike classic bots that respond only to rigid rules, contextual agents can analyze the flow of conversation, retain memory, adapt to changing situations, and offer personalized and coherent responses, significantly improving user satisfaction.

n8n democratizes the creation of intelligent agents

Thanks to its low-code/no-code approach and more than 400 integrations, n8n enables both technical and non-technical users to design complex workflows with artificial intelligence, without needing to be experts in advanced programming or machine learning.

The integration of language models and memory in n8n enhances conversational workflows

Easy connection with advanced language models (such as Azure OpenAI) and the ability to add memory components makes n8n a flexible and scalable platform for building sophisticated and customizable conversational agents.

Proper activation and deployment of workflows ensures the availability of AI agents

To consume agents from external applications, it is essential to activate workflows in n8n and use the appropriate production endpoints, thus ensuring continuous, secure, and scalable responses from intelligent agents in real-world scenarios.

References

  • Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2), 115–152.
  • Cheng, Y., Zhang, C., Zhang, Z., Meng, X., Hong, S., Li, W., Zhao, J. (2024). Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects. arXiv.
  • Chen, C., Xu, Y., & Wang, Z. (2022). Context-Aware Conversational Agents: A Review of Methods and Applications. IEEE Transactions on Artificial Intelligence, 3(4), 410-425.
  • Zamani, H., Sadoughi, N., & Croft, W. B. (2023). Intelligent Workflow Automation: Integrating Memory-Augmented Agents in Business Processes. Journal of Artificial Intelligence Research, 76, 325-348.
  • Dąbrowski, D. (2024). Day 67 of 100 Days Agentic Engineer Challenge: n8n Hybrid Long-Term Memory. Medium. https://damiandabrowski.medium.com/day-67-of-100-days-agentic-engineer-challenge-n8n-hybrid-long-term-memory-ce55694d8447
  • n8n. (2024). Build your first AI Agent – powered by Google Gemini with memory. https://n8n.io/workflows/4941-build-your-first-ai-agent-powered-by-google-gemini-with-memory
  • Luo, Y., Liang, P., Wang, C., Shahin, M., & Zhan, J. (2021). Characteristics and challenges of low code development: The practitioners’ perspective. arXiv. http://dx.doi.org/10.48550/arXiv.2107.07482
  • TextCortex. (2025). N8N Review: Features, pricing & use cases. Cybernews. https://cybernews.com/ai-tools/n8n-review/
]]>
https://blogs.perficient.com/2025/07/04/intelligent-agents-with-n8n-ai-powered-automation/feed/ 0 384044
Databricks Lakebase – Database Branching in Action https://blogs.perficient.com/2025/07/04/databricks-lakebase-database-branching-in-action/ https://blogs.perficient.com/2025/07/04/databricks-lakebase-database-branching-in-action/#respond Fri, 04 Jul 2025 07:17:16 +0000 https://blogs.perficient.com/?p=383982

What is Databricks Lakebase?

Databricks Lakebase is a Postgres OLTP engine, integrated into Databricks Data Intelligence Platform. A database instance is a compute type that provides fully managed storage and compute resources for a postgres database. Lakebase leverages an architecture that separates compute and storage, which allows independent scaling while supporting low latency (<10ms) and high concurrency transactions.

Databricks has integrated this powerful postgres engine along with sophisticated capabilities that are benefited by Databricks recent acquisition of Neon. Lakebase is fully managed by Databricks, which means no infrastructure has to be provisioned and maintained separately. In addition to traditional OLTP engine, Lakebase comes with below features,

  • Openness: Lakebase are built on open-source standards
  • Storage and compute separation: Lakebase stores data in data lakes in open format. It enables scaling storage and compute independently.
  • Serverless: Lakebase is lightweight, meaning it can scale instantly up and down based on the load. It can scale down to zero, at which the cost of the lakebase is just for the storage  of data only. No compute cost will be applied.
  • Modern development workflow: Branching a database is as simple as branching a code repository. It is done near instantly.
  • Built for AI Agents: Lakebases are designed to support a large number of AI agents. It’s branching and checkpointing capabilities enable AI agents to experiment and rewind to any point in time.
  • Lakehouse Integration: Lakebase make it easy to combine operational, analytical and AI systems without complex ETL pipelines.

In this article, we shall discuss in detail about how database branching feature works in Lakebase.

Database Branching

Database branching is one of the unique features introduced in Lakebase, that enables to branch out a database. It resembles the exact behavior of how code branch could be branched out from an existing branch.

Branching database is beneficial for an isolated test environment or point in time recovery. Lakebase uses copy-on-write branching mechanism to create an instant zero-copy clone of the database, with dedicated compute to operate on that branch. With zero-copy clone, it enables to create a branch of parent database of any size instantly.

The child branch is managed independently of the parent branch. With child isolated database branch, one can perform testing/debugging in the production copy of data. Though both parent and child databases appear separate, physically both instances would be pointing to same data pages. Under the hood, child database will be pointing to the actual data pages which parent is pointing to. When a change occurs in any of the data in child branch, then a new data page will be created with the new changes, and it will be available only to the branch. Any changes done in branch will not reflect in parent branch.

How branching works

The below diagrams represent how database branching works under the hood,

Database Branching

Database Branching Updates

Lakebase in action

Here is the demonstration of how Lakebase instance can be created, branch out an instance and how table changes behave,

To create Lakebase instance, login Databricks and navigate to Compute -> OLTP Database tab -> Click “Create New Instance” button,

Create New Instance 01

Create New Instance Success 02

Click “New Query” to launch SQL Editor for PostgreSQL Database. In current instance, let’s create a new table and add some records.

Instance1 Create Table 03

Instance1 Query Table 04

Let’s create a database branch “pginstance2” from instance “pginstance1”. Goto Compute –> OLTP Database –> Create Database instance

Enter new instance name and expand “Advanced Settings” -> Enable “Create from parent” option -> Enter the source instance name “pginstance1”.

Under “Include data from parent up to”, select “Current point in time” option. Here, we can choose any specific point in time instance too.

Create Instance2 05

Instance2 Create Success 06

Launch SQL Editor from pginstance2 database instance and query tbl_user_profile table

Instance2 Query Table 07

Now, let’s insert new record and update an existing record in the tbl_user_profile table in pginstance2,

Instance2 Update Table 08

Now, let’s switch back to parent database instance pginsntance1 and query tbl_user_profile table. The table in pginsntance1 should still be only 3 records. All the changes done in tbl_user_profile table should be available only in pginstance2.

Instance1 Query Table 09

Conclusion

Database changes that are done in one branch will not impact/reflect in another branch, thereby provide clear isolation of database at scale. Currently Lakebase do not have a feature to merge database branch. However, Databricks is committed and working towards database merge capability in near future.

]]>
https://blogs.perficient.com/2025/07/04/databricks-lakebase-database-branching-in-action/feed/ 0 383982
Monitoring Object Creation/Deletion in Cloud Storage with GCP Pub-Sub https://blogs.perficient.com/2025/07/02/monitoring-object-creation-deletion-in-cloud-storage-with-gcp-pub-sub/ https://blogs.perficient.com/2025/07/02/monitoring-object-creation-deletion-in-cloud-storage-with-gcp-pub-sub/#respond Wed, 02 Jul 2025 09:20:48 +0000 https://blogs.perficient.com/?p=383879

When using cloud-based event-driven systems, it’s essential to respond to changes at the storage level, such as when files are added, modified, or deleted. Google Cloud Platform (GCP) makes this easy by enabling Cloud Storage and Pub/Sub to talk to one another directly. This arrangement lets you send out structured real-time alerts whenever something happens inside a bucket. This configuration is specifically designed to catch deletion occurrences. When a file is deleted from a GCS bucket, a message is sent to a Pub/Sub topic. That subject becomes the main connection, providing alerts to any systems that are listening, such as a Cloud Run service, an external API, or another microservice. These systems can then react by cleaning up data, recording the incident, or sending out alarms. The architecture also takes care of critical backend needs. It employs IAM roles to set limits on who can access what, has retry rules in case something goes wrong for a short time, and links to a Dead-Letter Queue (DLQ) to keep messages that couldn’t be sent even after numerous tries. The whole system stays loosely coupled and strong because it employs technologies that are built into GCP. You can easily add or remove downstream services without changing the initial bucket. This pattern is a dependable and adaptable way to enforce cleanup rules, track changes for auditing, or initiate actions in real-time. In this article, we’ll explain the fundamental ideas, show you how to set it up, and talk about the important design choices that make this type of event notification system work with Pub/Sub to keep everything running smoothly.

Why Use Pub/Sub for Storage Notifications?

Pub/Sub makes it easy to respond to changes in Cloud Storage, like when a file is deleted, without having to connect everything closely. You don’t link each service directly to the storage bucket. Instead, you send events using Pub/Sub. This way, logging tools, data processors, and alarm systems may all work on their own without interfering with each other. The best thing? You can count on it. Even if something goes wrong, Pub/Sub makes sure that events don’t get lost. And since you only pay when messages are delivered or received, you don’t have to pay for resources that aren’t being used. This setup lets you be flexible, respond in real time, and evolve, which is great for cloud-native systems that need to be able to adapt and stay strong.

Architecture Overview

Archi Ovr

Step 1: Create a Cloud Storage Bucket

If you don’t already have a bucket, go to the Cloud Storage console, click ‘Create Bucket’, and follow these steps:
– Name: Choose a globally unique bucket name (e.g., demopoc-pubsub)
– Location: Pick a region or multi-region
– Default settings: You can leave the rest unchanged for this demo

S1

Step 2: Create a Pub/Sub Topic

Go to Pub/Sub in the Cloud Console and:
1. Click ‘Create Topic’
2. Name it something like demo-poc-pubsub
3. Leave the rest as defaults
4. Click Create

Step 3: Create a Pub/Sub Subscription (Pull-based)

  1. Click on the topic gcs-object-delete-topic
    2. Click ‘Create Subscription’
    3. Choose a pull subscription
    4. Name it gcs-delete-sub
    5. Leave other options as default
    6. Click Create

Step 4: Grant Pub/Sub Permission to Publish to the Topic

Go to the IAM permissions for your topic:
1. In the Pub/Sub console, go to your topic
2. Click ‘Permissions’
3. Click ‘Grant Access’
4. Add the GCS service account: service-<project-number-sample>@gs-project-accounts.iam.gserviceaccount.com
5. Assign it the role: Pub/Sub Publisher
6. Click Save

Step 5: Connect Cloud Storage to Pub/Sub via Shell

Open your cloud shell terminal and run:

gcloud storage buckets notifications create gs://my-delete-audit-bucket –topic=gcs-object-delete-topic –event-types=OBJECT_DELETE –payload-format=json

Explanation of gsutil command:

 gs://my-delete-audit-bucket: Your storage bucket

–topic: Pub/Sub topic name

–event-types=OBJECT_DELETE: Triggers only when objects are deleted

–payload-format=json: Format of the Pub/Sub message

 

Step 6: Test the Notification System
Then pull the message from Pub/Sub console.

Expected message payload:
{
“kind”: “storage#object”,
“bucket”: “my-delete-audit-bucket”,
“name”: “test.txt”,
“timeDeleted”: “2025-06-05T14:32:29.123Z”
}

Sample Use Cases

  1. Audit Trail for Object Deletion Use Case: Keep track of every deletion for compliance or internal audits.        How it works:When a file is destroyed, the Cloud Storage trigger sends an event to Pub/Sub. A Cloud Function or Dataflow task subscribes to the topic and sends metadata (such as the file name, timestamp, user, and bucket) to Big-Query.Why it matters:
    Keeps an unchangeable audit trail, helps with compliance (such HIPAA and SOC2), security audits, and internal investigations.

    Enforcement of Data Retention

    Use Case: Stop people from accidentally deleting data or make sure that data is kept for at least a certain amount of time.How it works:

    When an object is deleted, the system checks to see if it should have been deleted based on its name, metadata, and other factors.
    It logs the incident or restores the object from backup (for example, using Nearline or Archive tier) if it finds a violation.

    Why it matters:

    Helps keep data safe from being lost because to accidental or unlawful deletion. Makes sure that data lifecycle policies are followed

    3. Start cleaning jobs downstream

    Use Case: When an object is removed, connected data in other systems should be cleaned away automatically.How it works:

    When you delete a GCS object, a Cloud Function or Dataflow pipeline is triggered by the Pub/Sub message. This job deletes records that are linked to it in Big-Query, Fire-store, or invalidates cache/CDN entries.

    Why it matters:
    Keeps data systems working together. Stops recordings from being left behind or metadata from getting old

    4. Alerts in Real Time

    Use Case: Let teams or monitoring systems know when sensitive or unexpected removals happen.How it works:

    A Cloud Function that listens to Pub/Sub looks at the event. It gives an alert if the deletion meets certain criteria, such as a certain folder or file type.

Why it matters
Allows for a response in real time. Increases the ability to see high-risk operations

Result:

We created a modular, fault-tolerant real-time event-driven pipeline by using a Pub/Sub-based notification system for deleting Cloud Storage objects. When an object is added to or removed from the specified GCS bucket, alert notifications are sent to a Pub/Sub topic. That topic makes sure that the message gets to one or more downstream consumers.

Conclusion

Combining Cloud Storage with Pub/Sub for deleting objects is a basic idea in today’s GCP design. It publishes events to a Pub/Sub topic in almost real time when something is deleted. These events can be used for audit trails, enforcing data policies, automatic cleanups, and even alarms.This method promotes loose coupling by enabling Cloud Storage send events without having to know who the subscribers are. Cloud Functions, Dataflow, and custom applications that subscribers use can handle messages on their own. This makes the system easier to scale and manage.Using pub/sub makes production workflows more organized because it adds reliability, parallelism, retries, and other benefits. If GCP engineers want to design cloud systems that are adaptable, responsive, and ready for the future, they need to be experts at event-driven integration.

]]>
https://blogs.perficient.com/2025/07/02/monitoring-object-creation-deletion-in-cloud-storage-with-gcp-pub-sub/feed/ 0 383879