Implementing Scatter/Gather – Data Aggregation pattern (Fan-out/Fan-in) in Datapower
This article covers one of the very important patterns in SOA – Data aggregation. The data aggregation can be implemented in Datapower in many different ways.
Data aggregation in Datapower can be categorized mainly in three segments:
- Scatter n Gather Pattern
- Fan-out/Fan-in Pattern
- Data/Content Aggregation
Fundamental principle of all 3 patterns remains the same i.e. securing and displaying aggregated information from a single or multiple sources on web interface.
In Scatter and Gather pattern, a single request in the flow can be fanned out (scattered) to multiple backend services or to the same service with different resource URIs or multiple SOAP operations and the results from backend service(s) can be aggregated (gather) into a single response to service consumers.
The perfect example for such pattern could be taken into account of securing insurance or stock quotes, inventory status or booking appointments (doctors’, restaurants’ etc.) or providing 360 degree view of a customer by aggregating data from multiple systems or vendors. This pattern can also be used sometimes when backend exposes single operation for an interaction point e.g. a legacy system might provide available appointments per service provider and you may have to build the service which needs to show appointment for 10-20 service providers. In my experience, we have observed latency and performance issues in building such services. However, after enabling asynchronously option in Results and using event sink action, the latency of Datapower service decreased significantly. Therefore, it is imperative that backend services should be invoked asynchronously from Datapower to results action to achieve better performance in terms of response time.
As depicted in the aforementioned graphic, after receiving response from a backend service provider A, a call was generated for two other backend service providers (B & C) to get data from the response of service provider A. XSLT (transform action) can be used to parse the response from service provider A and build request URLs based on the response from the service A. Aforementioned diagram is specifically used to highlight that have specifically used term in the diagram above to emphasis that the multiple requests can go to the same service provider or multiple service providers. There is a significant performance improvement by using Results Action with asynchronous option than without asynchronous option as the number of URLs for the Backend service providers (B & C) increases because of parallel invocation. One will need to add Event Sink Action after Conditional Action or Results Action or both based upon the requirements and design of the service. The results from service A, B and C etc. can be aggregated using an XSLT action in the processing rule flow.
To demonstrate process flow better in this blog, results action is used, however a XSLT transform action can be used to call multiple backend services (Fan- out) and aggregate (Fan-in) the data in the same action.
