Data & Intelligence

Integrating IBM Integration Bus with DataPower XC10 – (REST APIs)

Introduction

In this article we will discuss about how you can achieve the integration between IBM Integration Bus (IIB) and WebSphere DataPower XC10 (XC10) using the REST APIs functionality of XC10 devices. Since the release of the V9, IIB can now connect to an external data grid cache to improve SOA’s services performance by the use of a eXtreme Scale client. But the good thing about integrating with the XC10 REST APIs is that it can also be achieved by previous and earliest versions of WebSphere Message Broker (WMB) like 7 and 8.

For a complete understanding of Global Cache and External Cache on IIB, please consider the following link:
http://www.ibm.com/developerworks/websphere/library/techarticles/1212_hart/1212_hart.html

If you’re looking for details on how to achieve this integration using the Java APIs MbGlobalMap, please visit:
http://www.ibm.com/developerworks/websphere/library/techarticles/1406_gupta/1406_gupta.html

Also, a detailed explanation of the Side Cache pattern in a SOA architecture is beyond the scope of this article. To learn more about the pattern, please consider the following links:

The requester side caching pattern specification, Part 1: Overview of the requester side caching pattern
http://www.ibm.com/developerworks/webservices/library/ws-rscp1/

Cache mediation pattern specification: an overview
http://www.ibm.com/developerworks/library/ws-soa-cachemed/


Assumptions

Before going ahead, let’s just explain some of terms that you will find from now on during the rest of the article:

Cache Hit

IIB will first query the XC10 cache when receiving a request for a cached service. If the requested data is found on the cache – HTTP Return Code 200 OK received from XC10 API, the request can be fulfilled by the cache, avoiding a backend call.

The flow stops right there by returning the data to the client, no further processing is needed and the most expensive task, which is to call the backend, is avoided.

Cache Miss

On the opposite side, if IIB doesn’t find the requested data on the cache – HTTP Return Code 404 received from XC10 API, it will need to invoke the backend. After the data is retrieved from the backend, it will be inserted into the cache, for the use and speed up of the subsequent calls.

Bottom line is: Cache Hit is faster than Cache Miss. The best performance experience can be achieved when the majority of the requests can be served by the cache. That is the goal point, to have the end-to-end performance maximized by the use of the side cache pattern. And that is done transparently to the client, who assumes it is still hitting the backend.


 

Scenario

For the scope of this article, we will be caching a SOAP Web Service (WS) response “mocked” at SoapUI. To better illustrate the scenario, we are adding delay on the SoapUI mock responses.

The same Caching concept applies to database queries or different types of backends. The most important part is a good understanding of what type of data can be cached – static data vs dynamic data, which will depend on your environment, architecture and service usability.

Proposed Architecture

The below diagram illustrates the proposed architecture to achieve the Side Cache pattern for a SOAP WS using IIB and XC10 REST APIs:

IIBXC10Architecture

 


 

IBM Integration Bus Architecture

Security

Before introducing the message flows and sub-flows that were used to achieve the integration, we will cover two important steps related to the XC10 security on IIB.
By default, XC10 REST APIs does requires HTTP Basic Authentication. So you will need to perform the following configurations on IIB:

Security profile for HTTP Basic Auth

Using the MQSI command console, issue the commands below to register the user credentials and to create a security profile:

$ mqsisetdbparms <BROKERNAME> -n <securityIdName> -u <user> -p <pass>

$ mqsicreateconfigurableservice <BROKERNAME> -c SecurityProfiles -o <securityProfileName> -n “propagation,idToPropagateToTransport,transportPropagationConfig” -v “TRUE,STATIC ID,<securityIdName>”

Attaching the securityProfile to the BAR file

Once the security profile is created, we need to attach it to the BAR file. At this point you should not have your BAR file ready yet, but since it’s a pretty straightforward task, we will cover it now.
Click on the BAR file, on the Manage tab, expand your application and select the Message Flow as shown on the picture. On the Configure properties that appears below, scroll down and set the security profile you just created. IIB will now add the Basic Auth header on the HTTP Requests used on this flow.

IIBXC10Architecture_2

 

Overall MsgFlow

This is an overview of the complete message flow implementing the side cache pattern.

IIBXC10Architecture_3

A brief description of the flow can be:

Data Intelligence - The Future of Big Data
The Future of Big Data

With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.

Get the Guide

The SOAP Input nodes exposes a Web Service interface. Once being called, the subsequent sub-flow SF_CacheQuery will first check if the data requested is cached by hitting the XC10 API (HTTP GET Method). If that is successful, the response is returned to the client immediately and no subsequent processing is done. Otherwise, the Invoke_BE_getCustomer node will call the SOAP Web Service. A Flow Order will first return the response to the Client and, after that the SF_CahceInsert sub-flow will insert this response data into the XC10 cache grid (HTTP POST Method).

Note that neither error handling nor retry logic when calling the XC10 APIs and the SOAP backend was implemented. You will certainly need to improve that and build your own flow and adapt it based on your own needs.

Cache Query SubFlow

As described above, this SubFlow will query the XC10 Cache Grid by sending a GET method to the REST API and using one of the request incoming data as the KeyIdentifier.
You should consider using a small timeout period on every call to the XC10 because we don’t want to cause an overhead processing time to the client in case of connections problems or if the XC10 is not available. For example, in this PoC, I used a timeout of 2 seconds.

IIBXC10Architecture_4

Set CacheQuery Params Compute Node – ESQL

The below ESQL contains the code used to achieve the Query Cache. First we save the incoming request into the variables for further usage.
After that the InputKEY is referenced from the incoming SOAP body. This is the key that will be used to query/insert the data in XC10. As the name suggests, you need to use something that uniquely distinguishes a request from others. If needed, you can even consider concatenating two or more fields and use that as your key.
After that we are ready to Query the XC10 cache by overwriting the HTTP Method to GET and then specifying the RequestURL in the XC10 REST API notation.
A successful response of that is acknowledged by XC10 with a HTTP 200 along with the data previously cached.

–Storing incoming request in case of CacheMiss
SET Environment.Variable.InputMessage = InputRoot.SOAP;

–Getting ID from request which is the KEY for XC10 Cache
DECLARE InputKEY REFERENCE TO InputRoot.SOAP.Body.ns:getCustomerRequest.ID;

–GET – Query Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestLine.Method = ‘GET’;
–XC10 URL for Query Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestURL = ‘http://192.168.122.1:7000/resources/datacaches/’ || CACHENAME || ‘/’ || CACHENAME || ‘/’ || CAST(InputKEY AS CHARACTER);

Note: Since we’re overwriting the HTTPRequest properties from a previous node, make sure you have the Compute Node propertie Compute Scope set to LocalEnvironment and Message:

IIBXC10Architecture_LocalEnv

 

Cache Insert SubFlow

After a Cache Miss, this SubFlow will insert the response data into the XC10 Cache Grid by sending a HTTP POST request method to the REST API and using one of the request incoming data as the KeyIdentifier. The data to be cached is mandatory and is sent as the request payload.

Remembering, you should consider using a small timeout period on every call to the XC10 because we don’t want to cause an overhead processing time to the client in case of connections problems or if the XC10 is not available. For example, in this PoC, I used a timeout of 2 seconds.

IIBXC10Architecture_5

Set CacheInsert Params Compute Node – ESQL

The below ESQL contains the code used to achieve the Insert Cache. First we save the incoming request into the variables for further usage.
After that the InputKEY is referenced from the incoming SOAP body. This is the key that will be used to query/insert the data in XC10. As the name suggests, you need to use something that uniquely distinguishes a request from others. If needed, you can even consider concatenating two or more fields and use that as your key.
After that we are ready to Query the XC10 cache by overwriting the HTTP Method to GET and then specifying the RequestURL in the XC10 REST API notation.
A successful response of that is acknowledged by XC10 with a HTTP 200 along with the data previously cached.

–Getting ID from request which is the KEY for XC10 Cache
DECLARE InputKEY REFERENCE TO Environment.Variable.InputMessage.Body.v1:getCustomerRequest.ID;

–POST – Insert Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestLine.Method = ‘POST’;
–XC10 URL for Insert Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestURL = ‘http://192.168.122.1:7000/resources/datacaches/’ || CACHENAME || ‘/’ || CACHENAME || ‘/’ || CAST(InputKEY AS CHARACTER);

Note: Since we’re overwriting the HTTPRequest properties from a previous node, make sure you have the Compute Node propertie Compute Scope set to LocalEnvironment and Message:

IIBXC10Architecture_LocalEnv

 


 

XC10 REST APIs

It’s beyond the scope of the article to explain all the potential and possibilities that can be achieved by using the XC10 REST APIs.
For a complete reference and usability functions available, please visit:
http://www-01.ibm.com/support/knowledgecenter/SSS8GR_2.5.0/com.ibm.websphere.datapower.xc.doc/tdevrest.html

POST

You can use any HTTP client to interact with the XC10 REST APIs. HTTPRequest nodes are used on IIB, but you can also use cURL or SoapUI for testing purposes. There’s a sample SoapUI project along with the files available for download. cURL commands samples are available at the Appendix chapter.

Insert Data into the XC10 for test

HTTP method: POST
HTTP header: ’Content-type: text/xml;charset=UTF-8’
URI: /resources/datacaches/[grid_name]/[map_name]/[key]

POST data:
’<xml>Sample Data</xml>’

Response:
200

GET

You can use any HTTP client to interact with the XC10 REST APIs. HTTPRequest nodes are used on IIB, but you can also use cURL or SoapUI for testing purposes. There’s a sample SoapUI project along with the files available for download. cURL commands samples are available at the Appendix chapter.

Retrieve (GET) Cache Data from the XC10
HTTP method: GET
URI: /resources/datacaches/[grid_name]/[map_name]/[key]

Returns Data cached previously:
‘<xml>Sample Data</xml>’

If key doesn’t exist, returns HTTP error:
404

Monitoring

XC10 offers native monitoring of for each Data Grid. Using the GUI, just follow Monitor -> Individual Data Grid Overview -> click on your Grid. The below monitoring and performance metrics will appear.

IIBXC10Architecture_6

 


 

Conclusion

In this article we covered how easy it is to implement the Side Cache Pattern on a SOA architecture using IBM Integration Bus and WebSphere DataPower XC10 to speed up performance. This can be used on a variety of backends and not only SOAP Web Services. The XC10 REST APIs is a pretty good interface and provides all necessary functions to integrate pretty straight forward. In our scenario we solved a “slow SOAP Web Service backend” problem by caching the response data into the XC10 Data Grid.


 

Appendix

Insert (POST) Cache Data example using command utility “cURL”

curl -u <user>:<pass> -H ‘Content-type: text/xml;charset=UTF-8’ -X POST -d ‘<xml>Sample Data</xml>’ http://<xc10hostname>/resources/datacaches/IIB_POC/IIB_POC/1020

Retrieve (GET) Cache Data example using command utility “cURL”

curl -u <user>:<pass> -X GET http://<xc10hostname>/resources/datacaches/IIB_POC/IIB_POC/1020

Consider using SoapUI for a more friendly GUI instead of cuRL. There’s a sample SoapUI project along with the files available for download.

 

Troubleshooting Tools

Consider using troubleshooting tools such as NetTools web debugging tool to sit in the middle between IIB and XC10.
Download at: http://sourceforge.net/projects/nettool/files/

 

DOWNLOAD IBM INTEGRATION BUS PROJECT INTERCHANGE SAMPLE CODE
IIB_XC10

DOWNLOAD SOAPUI PROJECTS
IIBXC10_SoapUI

About the Author

More from this Author

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to the Weekly Blog Digest:

Sign Up