Introduction
In this article we will discuss about how you can achieve the integration between IBM Integration Bus (IIB) and WebSphere DataPower XC10 (XC10) using the REST APIs functionality of XC10 devices. Since the release of the V9, IIB can now connect to an external data grid cache to improve SOA’s services performance by the use of a eXtreme Scale client. But the good thing about integrating with the XC10 REST APIs is that it can also be achieved by previous and earliest versions of WebSphere Message Broker (WMB) like 7 and 8.
For a complete understanding of Global Cache and External Cache on IIB, please consider the following link:
http://www.ibm.com/developerworks/websphere/library/techarticles/1212_hart/1212_hart.html
If you’re looking for details on how to achieve this integration using the Java APIs MbGlobalMap, please visit:
http://www.ibm.com/developerworks/websphere/library/techarticles/1406_gupta/1406_gupta.html
Also, a detailed explanation of the Side Cache pattern in a SOA architecture is beyond the scope of this article. To learn more about the pattern, please consider the following links:
The requester side caching pattern specification, Part 1: Overview of the requester side caching pattern
http://www.ibm.com/developerworks/webservices/library/ws-rscp1/
Cache mediation pattern specification: an overview
http://www.ibm.com/developerworks/library/ws-soa-cachemed/
Assumptions
Before going ahead, let’s just explain some of terms that you will find from now on during the rest of the article:
Cache Hit
IIB will first query the XC10 cache when receiving a request for a cached service. If the requested data is found on the cache – HTTP Return Code 200 OK received from XC10 API, the request can be fulfilled by the cache, avoiding a backend call.
The flow stops right there by returning the data to the client, no further processing is needed and the most expensive task, which is to call the backend, is avoided.
Cache Miss
On the opposite side, if IIB doesn’t find the requested data on the cache – HTTP Return Code 404 received from XC10 API, it will need to invoke the backend. After the data is retrieved from the backend, it will be inserted into the cache, for the use and speed up of the subsequent calls.
Bottom line is: Cache Hit is faster than Cache Miss. The best performance experience can be achieved when the majority of the requests can be served by the cache. That is the goal point, to have the end-to-end performance maximized by the use of the side cache pattern. And that is done transparently to the client, who assumes it is still hitting the backend.
Scenario
For the scope of this article, we will be caching a SOAP Web Service (WS) response “mocked” at SoapUI. To better illustrate the scenario, we are adding delay on the SoapUI mock responses.
The same Caching concept applies to database queries or different types of backends. The most important part is a good understanding of what type of data can be cached – static data vs dynamic data, which will depend on your environment, architecture and service usability.
Proposed Architecture
The below diagram illustrates the proposed architecture to achieve the Side Cache pattern for a SOAP WS using IIB and XC10 REST APIs:
IBM Integration Bus Architecture
Security
Before introducing the message flows and sub-flows that were used to achieve the integration, we will cover two important steps related to the XC10 security on IIB.
By default, XC10 REST APIs does requires HTTP Basic Authentication. So you will need to perform the following configurations on IIB:
Security profile for HTTP Basic Auth
Using the MQSI command console, issue the commands below to register the user credentials and to create a security profile:
$ mqsisetdbparms <BROKERNAME> -n <securityIdName> -u <user> -p <pass>
$ mqsicreateconfigurableservice <BROKERNAME> -c SecurityProfiles -o <securityProfileName> -n “propagation,idToPropagateToTransport,transportPropagationConfig” -v “TRUE,STATIC ID,<securityIdName>”
Attaching the securityProfile to the BAR file
Once the security profile is created, we need to attach it to the BAR file. At this point you should not have your BAR file ready yet, but since it’s a pretty straightforward task, we will cover it now.
Click on the BAR file, on the Manage tab, expand your application and select the Message Flow as shown on the picture. On the Configure properties that appears below, scroll down and set the security profile you just created. IIB will now add the Basic Auth header on the HTTP Requests used on this flow.
Overall MsgFlow
This is an overview of the complete message flow implementing the side cache pattern.
A brief description of the flow can be:
The Future of Big Data
With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.
The SOAP Input nodes exposes a Web Service interface. Once being called, the subsequent sub-flow SF_CacheQuery will first check if the data requested is cached by hitting the XC10 API (HTTP GET Method). If that is successful, the response is returned to the client immediately and no subsequent processing is done. Otherwise, the Invoke_BE_getCustomer node will call the SOAP Web Service. A Flow Order will first return the response to the Client and, after that the SF_CahceInsert sub-flow will insert this response data into the XC10 cache grid (HTTP POST Method).
Note that neither error handling nor retry logic when calling the XC10 APIs and the SOAP backend was implemented. You will certainly need to improve that and build your own flow and adapt it based on your own needs.
Cache Query SubFlow
As described above, this SubFlow will query the XC10 Cache Grid by sending a GET method to the REST API and using one of the request incoming data as the KeyIdentifier.
You should consider using a small timeout period on every call to the XC10 because we don’t want to cause an overhead processing time to the client in case of connections problems or if the XC10 is not available. For example, in this PoC, I used a timeout of 2 seconds.
Set CacheQuery Params Compute Node – ESQL
The below ESQL contains the code used to achieve the Query Cache. First we save the incoming request into the variables for further usage.
After that the InputKEY is referenced from the incoming SOAP body. This is the key that will be used to query/insert the data in XC10. As the name suggests, you need to use something that uniquely distinguishes a request from others. If needed, you can even consider concatenating two or more fields and use that as your key.
After that we are ready to Query the XC10 cache by overwriting the HTTP Method to GET and then specifying the RequestURL in the XC10 REST API notation.
A successful response of that is acknowledged by XC10 with a HTTP 200 along with the data previously cached.
–Storing incoming request in case of CacheMiss
SET Environment.Variable.InputMessage = InputRoot.SOAP;–Getting ID from request which is the KEY for XC10 Cache
DECLARE InputKEY REFERENCE TO InputRoot.SOAP.Body.ns:getCustomerRequest.ID;–GET – Query Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestLine.Method = ‘GET’;
–XC10 URL for Query Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestURL = ‘http://192.168.122.1:7000/resources/datacaches/’ || CACHENAME || ‘/’ || CACHENAME || ‘/’ || CAST(InputKEY AS CHARACTER);
Note: Since we’re overwriting the HTTPRequest properties from a previous node, make sure you have the Compute Node propertie Compute Scope set to LocalEnvironment and Message:
Cache Insert SubFlow
After a Cache Miss, this SubFlow will insert the response data into the XC10 Cache Grid by sending a HTTP POST request method to the REST API and using one of the request incoming data as the KeyIdentifier. The data to be cached is mandatory and is sent as the request payload.
Remembering, you should consider using a small timeout period on every call to the XC10 because we don’t want to cause an overhead processing time to the client in case of connections problems or if the XC10 is not available. For example, in this PoC, I used a timeout of 2 seconds.
Set CacheInsert Params Compute Node – ESQL
The below ESQL contains the code used to achieve the Insert Cache. First we save the incoming request into the variables for further usage.
After that the InputKEY is referenced from the incoming SOAP body. This is the key that will be used to query/insert the data in XC10. As the name suggests, you need to use something that uniquely distinguishes a request from others. If needed, you can even consider concatenating two or more fields and use that as your key.
After that we are ready to Query the XC10 cache by overwriting the HTTP Method to GET and then specifying the RequestURL in the XC10 REST API notation.
A successful response of that is acknowledged by XC10 with a HTTP 200 along with the data previously cached.
–Getting ID from request which is the KEY for XC10 Cache
DECLARE InputKEY REFERENCE TO Environment.Variable.InputMessage.Body.v1:getCustomerRequest.ID;–POST – Insert Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestLine.Method = ‘POST’;
–XC10 URL for Insert Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestURL = ‘http://192.168.122.1:7000/resources/datacaches/’ || CACHENAME || ‘/’ || CACHENAME || ‘/’ || CAST(InputKEY AS CHARACTER);
Note: Since we’re overwriting the HTTPRequest properties from a previous node, make sure you have the Compute Node propertie Compute Scope set to LocalEnvironment and Message:
XC10 REST APIs
It’s beyond the scope of the article to explain all the potential and possibilities that can be achieved by using the XC10 REST APIs.
For a complete reference and usability functions available, please visit:
http://www-01.ibm.com/support/knowledgecenter/SSS8GR_2.5.0/com.ibm.websphere.datapower.xc.doc/tdevrest.html
POST
You can use any HTTP client to interact with the XC10 REST APIs. HTTPRequest nodes are used on IIB, but you can also use cURL or SoapUI for testing purposes. There’s a sample SoapUI project along with the files available for download. cURL commands samples are available at the Appendix chapter.
Insert Data into the XC10 for test
HTTP method: POST
HTTP header: ’Content-type: text/xml;charset=UTF-8’
URI: /resources/datacaches/[grid_name]/[map_name]/[key]
POST data:
’<xml>Sample Data</xml>’
Response:
200
GET
You can use any HTTP client to interact with the XC10 REST APIs. HTTPRequest nodes are used on IIB, but you can also use cURL or SoapUI for testing purposes. There’s a sample SoapUI project along with the files available for download. cURL commands samples are available at the Appendix chapter.
Retrieve (GET) Cache Data from the XC10
HTTP method: GET
URI: /resources/datacaches/[grid_name]/[map_name]/[key]
Returns Data cached previously:
‘<xml>Sample Data</xml>’
If key doesn’t exist, returns HTTP error:
404
Monitoring
XC10 offers native monitoring of for each Data Grid. Using the GUI, just follow Monitor -> Individual Data Grid Overview -> click on your Grid. The below monitoring and performance metrics will appear.
Conclusion
In this article we covered how easy it is to implement the Side Cache Pattern on a SOA architecture using IBM Integration Bus and WebSphere DataPower XC10 to speed up performance. This can be used on a variety of backends and not only SOAP Web Services. The XC10 REST APIs is a pretty good interface and provides all necessary functions to integrate pretty straight forward. In our scenario we solved a “slow SOAP Web Service backend” problem by caching the response data into the XC10 Data Grid.
Appendix
Insert (POST) Cache Data example using command utility “cURL”
curl -u <user>:<pass> -H ‘Content-type: text/xml;charset=UTF-8’ -X POST -d ‘<xml>Sample Data</xml>’ http://<xc10hostname>/resources/datacaches/IIB_POC/IIB_POC/1020
Retrieve (GET) Cache Data example using command utility “cURL”
curl -u <user>:<pass> -X GET http://<xc10hostname>/resources/datacaches/IIB_POC/IIB_POC/1020
Consider using SoapUI for a more friendly GUI instead of cuRL. There’s a sample SoapUI project along with the files available for download.
Troubleshooting Tools
Consider using troubleshooting tools such as NetTools web debugging tool to sit in the middle between IIB and XC10.
Download at: http://sourceforge.net/projects/nettool/files/
DOWNLOAD IBM INTEGRATION BUS PROJECT INTERCHANGE SAMPLE CODE
IIB_XC10
DOWNLOAD SOAPUI PROJECTS
IIBXC10_SoapUI
