Perficient IBM blog

Subscribe to Emails

Subscribe to RSS feed

Archives

Follow IBM Technologies on Pinterest

Archive for the ‘Message Broker’ Category

Integrating IBM Integration Bus with DataPower XC10 – (REST APIs)

Introduction

In this article we will discuss about how you can achieve the integration between IBM Integration Bus (IIB) and WebSphere DataPower XC10 (XC10) using the REST APIs functionality of XC10 devices. Since the release of the V9, IIB can now connect to an external data grid cache to improve SOA’s services performance by the use of a eXtreme Scale client. But the good thing about integrating with the XC10 REST APIs is that it can also be achieved by previous and earliest versions of WebSphere Message Broker (WMB) like 7 and 8.

For a complete understanding of Global Cache and External Cache on IIB, please consider the following link:
http://www.ibm.com/developerworks/websphere/library/techarticles/1212_hart/1212_hart.html

If you’re looking for details on how to achieve this integration using the Java APIs MbGlobalMap, please visit:
http://www.ibm.com/developerworks/websphere/library/techarticles/1406_gupta/1406_gupta.html

Also, a detailed explanation of the Side Cache pattern in a SOA architecture is beyond the scope of this article. To learn more about the pattern, please consider the following links:

The requester side caching pattern specification, Part 1: Overview of the requester side caching pattern
http://www.ibm.com/developerworks/webservices/library/ws-rscp1/

Cache mediation pattern specification: an overview
http://www.ibm.com/developerworks/library/ws-soa-cachemed/


Assumptions

Before going ahead, let’s just explain some of terms that you will find from now on during the rest of the article:

Cache Hit

IIB will first query the XC10 cache when receiving a request for a cached service. If the requested data is found on the cache – HTTP Return Code 200 OK received from XC10 API, the request can be fulfilled by the cache, avoiding a backend call.

The flow stops right there by returning the data to the client, no further processing is needed and the most expensive task, which is to call the backend, is avoided.

Cache Miss

On the opposite side, if IIB doesn’t find the requested data on the cache – HTTP Return Code 404 received from XC10 API, it will need to invoke the backend. After the data is retrieved from the backend, it will be inserted into the cache, for the use and speed up of the subsequent calls.

Bottom line is: Cache Hit is faster than Cache Miss. The best performance experience can be achieved when the majority of the requests can be served by the cache. That is the goal point, to have the end-to-end performance maximized by the use of the side cache pattern. And that is done transparently to the client, who assumes it is still hitting the backend.


 

Scenario

For the scope of this article, we will be caching a SOAP Web Service (WS) response “mocked” at SoapUI. To better illustrate the scenario, we are adding delay on the SoapUI mock responses.

The same Caching concept applies to database queries or different types of backends. The most important part is a good understanding of what type of data can be cached – static data vs dynamic data, which will depend on your environment, architecture and service usability.

Proposed Architecture

The below diagram illustrates the proposed architecture to achieve the Side Cache pattern for a SOAP WS using IIB and XC10 REST APIs:

IIBXC10Architecture

 


 

IBM Integration Bus Architecture

Security

Before introducing the message flows and sub-flows that were used to achieve the integration, we will cover two important steps related to the XC10 security on IIB.
By default, XC10 REST APIs does requires HTTP Basic Authentication. So you will need to perform the following configurations on IIB:

Security profile for HTTP Basic Auth

Using the MQSI command console, issue the commands below to register the user credentials and to create a security profile:

$ mqsisetdbparms <BROKERNAME> -n <securityIdName> -u <user> -p <pass>

$ mqsicreateconfigurableservice <BROKERNAME> -c SecurityProfiles -o <securityProfileName> -n “propagation,idToPropagateToTransport,transportPropagationConfig” -v “TRUE,STATIC ID,<securityIdName>”

Attaching the securityProfile to the BAR file

Once the security profile is created, we need to attach it to the BAR file. At this point you should not have your BAR file ready yet, but since it’s a pretty straightforward task, we will cover it now.
Click on the BAR file, on the Manage tab, expand your application and select the Message Flow as shown on the picture. On the Configure properties that appears below, scroll down and set the security profile you just created. IIB will now add the Basic Auth header on the HTTP Requests used on this flow.

IIBXC10Architecture_2

 

Overall MsgFlow

This is an overview of the complete message flow implementing the side cache pattern.

IIBXC10Architecture_3

A brief description of the flow can be:

The SOAP Input nodes exposes a Web Service interface. Once being called, the subsequent sub-flow SF_CacheQuery will first check if the data requested is cached by hitting the XC10 API (HTTP GET Method). If that is successful, the response is returned to the client immediately and no subsequent processing is done. Otherwise, the Invoke_BE_getCustomer node will call the SOAP Web Service. A Flow Order will first return the response to the Client and, after that the SF_CahceInsert sub-flow will insert this response data into the XC10 cache grid (HTTP POST Method).

Note that neither error handling nor retry logic when calling the XC10 APIs and the SOAP backend was implemented. You will certainly need to improve that and build your own flow and adapt it based on your own needs.

Cache Query SubFlow

As described above, this SubFlow will query the XC10 Cache Grid by sending a GET method to the REST API and using one of the request incoming data as the KeyIdentifier.
You should consider using a small timeout period on every call to the XC10 because we don’t want to cause an overhead processing time to the client in case of connections problems or if the XC10 is not available. For example, in this PoC, I used a timeout of 2 seconds.

IIBXC10Architecture_4

Set CacheQuery Params Compute Node – ESQL

The below ESQL contains the code used to achieve the Query Cache. First we save the incoming request into the variables for further usage.
After that the InputKEY is referenced from the incoming SOAP body. This is the key that will be used to query/insert the data in XC10. As the name suggests, you need to use something that uniquely distinguishes a request from others. If needed, you can even consider concatenating two or more fields and use that as your key.
After that we are ready to Query the XC10 cache by overwriting the HTTP Method to GET and then specifying the RequestURL in the XC10 REST API notation.
A successful response of that is acknowledged by XC10 with a HTTP 200 along with the data previously cached.

–Storing incoming request in case of CacheMiss
SET Environment.Variable.InputMessage = InputRoot.SOAP;

–Getting ID from request which is the KEY for XC10 Cache
DECLARE InputKEY REFERENCE TO InputRoot.SOAP.Body.ns:getCustomerRequest.ID;

–GET – Query Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestLine.Method = ‘GET';
–XC10 URL for Query Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestURL = ‘http://192.168.122.1:7000/resources/datacaches/’ || CACHENAME || ‘/’ || CACHENAME || ‘/’ || CAST(InputKEY AS CHARACTER);

Note: Since we’re overwriting the HTTPRequest properties from a previous node, make sure you have the Compute Node propertie Compute Scope set to LocalEnvironment and Message:

IIBXC10Architecture_LocalEnv

 

Cache Insert SubFlow

After a Cache Miss, this SubFlow will insert the response data into the XC10 Cache Grid by sending a HTTP POST request method to the REST API and using one of the request incoming data as the KeyIdentifier. The data to be cached is mandatory and is sent as the request payload.

Remembering, you should consider using a small timeout period on every call to the XC10 because we don’t want to cause an overhead processing time to the client in case of connections problems or if the XC10 is not available. For example, in this PoC, I used a timeout of 2 seconds.

IIBXC10Architecture_5

Set CacheInsert Params Compute Node – ESQL

The below ESQL contains the code used to achieve the Insert Cache. First we save the incoming request into the variables for further usage.
After that the InputKEY is referenced from the incoming SOAP body. This is the key that will be used to query/insert the data in XC10. As the name suggests, you need to use something that uniquely distinguishes a request from others. If needed, you can even consider concatenating two or more fields and use that as your key.
After that we are ready to Query the XC10 cache by overwriting the HTTP Method to GET and then specifying the RequestURL in the XC10 REST API notation.
A successful response of that is acknowledged by XC10 with a HTTP 200 along with the data previously cached.

–Getting ID from request which is the KEY for XC10 Cache
DECLARE InputKEY REFERENCE TO Environment.Variable.InputMessage.Body.v1:getCustomerRequest.ID;

–POST – Insert Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestLine.Method = ‘POST';
–XC10 URL for Insert Cache
SET OutputLocalEnvironment.Destination.HTTP.RequestURL = ‘http://192.168.122.1:7000/resources/datacaches/’ || CACHENAME || ‘/’ || CACHENAME || ‘/’ || CAST(InputKEY AS CHARACTER);

Note: Since we’re overwriting the HTTPRequest properties from a previous node, make sure you have the Compute Node propertie Compute Scope set to LocalEnvironment and Message:

IIBXC10Architecture_LocalEnv

 


 

XC10 REST APIs

It’s beyond the scope of the article to explain all the potential and possibilities that can be achieved by using the XC10 REST APIs.
For a complete reference and usability functions available, please visit:
http://www-01.ibm.com/support/knowledgecenter/SSS8GR_2.5.0/com.ibm.websphere.datapower.xc.doc/tdevrest.html

POST

You can use any HTTP client to interact with the XC10 REST APIs. HTTPRequest nodes are used on IIB, but you can also use cURL or SoapUI for testing purposes. There’s a sample SoapUI project along with the files available for download. cURL commands samples are available at the Appendix chapter.

Insert Data into the XC10 for test

HTTP method: POST
HTTP header: ’Content-type: text/xml;charset=UTF-8’
URI: /resources/datacaches/[grid_name]/[map_name]/[key]

POST data:
’<xml>Sample Data</xml>’

Response:
200

GET

You can use any HTTP client to interact with the XC10 REST APIs. HTTPRequest nodes are used on IIB, but you can also use cURL or SoapUI for testing purposes. There’s a sample SoapUI project along with the files available for download. cURL commands samples are available at the Appendix chapter.

Retrieve (GET) Cache Data from the XC10
HTTP method: GET
URI: /resources/datacaches/[grid_name]/[map_name]/[key]

Returns Data cached previously:
‘<xml>Sample Data</xml>’

If key doesn’t exist, returns HTTP error:
404

Monitoring

XC10 offers native monitoring of for each Data Grid. Using the GUI, just follow Monitor -> Individual Data Grid Overview -> click on your Grid. The below monitoring and performance metrics will appear.

IIBXC10Architecture_6

 


 

Conclusion

In this article we covered how easy it is to implement the Side Cache Pattern on a SOA architecture using IBM Integration Bus and WebSphere DataPower XC10 to speed up performance. This can be used on a variety of backends and not only SOAP Web Services. The XC10 REST APIs is a pretty good interface and provides all necessary functions to integrate pretty straight forward. In our scenario we solved a “slow SOAP Web Service backend” problem by caching the response data into the XC10 Data Grid.


 

Appendix

Insert (POST) Cache Data example using command utility “cURL”

curl -u <user>:<pass> -H ‘Content-type: text/xml;charset=UTF-8′ -X POST -d ‘<xml>Sample Data</xml>’ http://<xc10hostname>/resources/datacaches/IIB_POC/IIB_POC/1020

Retrieve (GET) Cache Data example using command utility “cURL”

curl -u <user>:<pass> -X GET http://<xc10hostname>/resources/datacaches/IIB_POC/IIB_POC/1020

Consider using SoapUI for a more friendly GUI instead of cuRL. There’s a sample SoapUI project along with the files available for download.

 

Troubleshooting Tools

Consider using troubleshooting tools such as NetTools web debugging tool to sit in the middle between IIB and XC10.
Download at: http://sourceforge.net/projects/nettool/files/

 

DOWNLOAD IBM INTEGRATION BUS PROJECT INTERCHANGE SAMPLE CODE
IIB_XC10

DOWNLOAD SOAPUI PROJECTS
IIBXC10_SoapUI

WebSphere Message Broker integration with Netezza using ODBC/JDBC

What is Netezza:

At an extremely high level Netezza is a massively Parallel Process Database Appliance designed to process and analyze large amounts of data. Netezza’s core value is to keep the things simple and accelerating high performance analysis of data to help clients uncover insights into their business. One of the reasons of it’s performance is that it is deeply integrates hardware, software and storage to simplify and accelerate deployment to deliver business analytics value in few days. This data warehouse appliance requires nearly no to minimal tuning and administration – e.g., no storage administration, less database/system administration – further reducing the cost of deploying and total cost of ownership.

WebSphere Message Broker integration with Netezza using ODBC/JDBCIBM Netezza appliances are now part of IBM PureSystems – expert integrated systems with built in expertise, integration by design and a simplified user experience. Part of the PureData family, the Netezza appliance is now known as the PureData System for Analytics. It has the same key design tenets of simplicity, speed, scalability and analytics power that was fundamental to Netezza appliances. With simple deployment, out-of-the-box optimization, no tuning and minimal on-going maintenance, the IBM PureData System for Analytics has the industry’s fastest time-to-value and lowest total-cost-of-ownership.

What is Message Broker:

IBM WebSphere® Message Broker is an Enterprise Service Bus (ESB) built for universal connectivity and transformation in heterogeneous IT environments. WebSphere Message Broker distributes information and data generated by business events in real time to people, applications, and devices throughout your extended enterprise and beyond.

Read the rest of this post »

Tags: , , ,

Posted in Message Broker, News, WMB

Simplify Development = Embrace Patterns

I’m a BIG believer in two things: a) work smarter, not harder; 2) keep things simple — avoid complexity. Complex leads to complicated, complicated leads to misunderstanding, misunderstanding leads to chaos. Nonetheless, my guidelines seem easy enough, right? Well, good news then, they are simple and easy to apply in just about anything we face day to day — except in golf. Ah yes, the wonderful world of golf [which currently holds me hostage]. The place where complex instructions, ideas and or suggestions have license to run rampant — like a wild herd of buffalo feasting on the open range. If anyone tells you differently about golf, run. I started hacking away on fairways and driving ranges across this great country 28 years ago. It’s only recently that I finally feel like a hero when I play. But that feeling came at a high price. Golf IS a complex sport to learn, practice and or play. It requires the highest of all levels in humility and stick-to-itiveness. Love that word, stick-to-itiveness [defined as: the quality that allows someone to continue trying to do something even though it is difficult or unpleasant]. But I digress.

cupcake

Let’s get back to keeping things simple in the blog. What is a pattern, as it applies to WebSphere Message Broker [WMB] and or IIBv9? In short, a pattern is a reusable solution that encapsulates a tested approach to solving a common architecture, design, or deployment task in a particular context. Ever hear of the television show Cupcakes Wars on Food Network? Each show consists of cupcake bakers battling to see who will reign supreme as cupcake baker of the week. In the final bake-off, bakers are required to bake 1,000 cupcakes each. To accomplish this, bakers use commercial cupcake pans, like the one shown to the right. So, how does a cupcake pan apply to patterns in WMB or IIBv9? Easy, the pan is my pattern. The pattern ensures that each artifact [message flows in WMB or IIBv9] are created the same, like a cupcake. Are you now thinking “…what about customization Jason?” Easy, what kind of cupcake does your client like? Vanilla, chocolate or strawberry? Does it need sprinkles and frosting too? In case you’re curious, my fav cupcake is a vanilla, maple syrup and bacon concoction. Read the rest of this post »

eXtreme Scale

Just for curiosity’s sake and this blog, I looked up the definition of ‘cache’ [as used in computer science]. Most sources define it as “…a component that transparently stores data so that future requests for that data can be served faster “.  The part about served faster reminded me of an old adage. It goes like this. Cheap, fast and good; pick two. In other words, say you need something that is cheap and fast; most likely, it will not be good. Kinda like the bread sticks I ordered for dinner earlier this week. Conversely, say you need something that is fast and good, it will not be cheap. Enter WebSphere eXtreme Scale [WXS]; an enterprise caching solution.

eXtreme ScaleIn the SOA world, architects, developers and or administrators are consistently faced with performance challenges. For me, I feel immersed in a never-ending battle of resolving and or improving our client’s performance issues. Many customers have business applications that process thousands, hundreds of thousands, if not billions of transactions daily. Typically, these applications will require high levels of efficiency and near-linear capability. It is here that WXS, a distributed caching platform fits well. It provides an efficient way to increase and streamline system wide performance. WXS is designed to work in heterogeneous environments throughout leading application server platforms and visualization environments.

Read the rest of this post »

IIBv9 – Introduction

Peers, it is time to embrace IBM’s Integration Bus v9 aka IIBv9. Many of us are comfortable developing solutions using WebSphere Message Broker [WMB] version 6, 7 and 8. To date, all the projects I’ve worked on at Perficient, clients still used WMB 6/7. However, IBM recently re-branded WMB as Integration Bus. Much of the changes and or improvements released in WMB version 8,  were significantly updated and improved in IIBv9. Meaning, the word is out. Clients are looking to move away from WMB 6/7/8 and into IIBv9. Let’s get ahead of the migration wave and become the leaders of IIBv9 implementation. No sweat, the easiest way to do that is by informing ourselves.

According to IBM:

IBM® Integration Bus is designed to solve all integration requirements between back-end systems, from simple point-to-point connectivity to sophisticated topological patterns such as ESBs. It has a wide range of built-in connectors that enable integration between web services, files, databases, packaged applications such as SAP and Microsoft® Dynamics CRM, plus a variety of configuration and transformation options such as graphical mapping, Java™, and Microsoft .NET®.

I’ve created a brief 16 minute presentation. It provides a high-level overview of IIBv9; as well as, share new features. Please click on the link below to view the presentation. Feel free to post comments/questions too.

http://youtu.be/6k8fCSDxn90

Message modelling using DFDL in WMB

Introduction

This article addresses the DFDL message modelling feature in IBM® WebSphere® Message Broker (also known as IBM® Integration Bus). It contains step by step instructions for creating a DFDL schema for a flat file using WMB tool-kit.

Data Format Description Language, commonly known as DFDL is an open standard based modelling language which is using in Websphere message broker V8 onwards for modelling data.  DFDL is based on the XML schema and is the recommended way of modelling over the proprietary MRM domain. DFDL is using a subset of the constructs in XML schema to modell the non-XML messages.

DFDL can be used to describe legacy data files,  record-oriented text or scientific and numeric computations. DFDL is more flexible than the MRM modelling and provides better performance.

Read the rest of this post »

Automated Build and Deploy in WebSphere Message Broker Using Ant

Summary:  This article shows a step-by-step approach for setting up an automated build and deploy framework in IBM® WebSphere® Message Broker (also known as IBM® Integration Bus) using Ant, Hudson and SubVersion. This also includes the set of frameworks components for setting up the auto build and deploy for a sample WMB project .

Broker ARchive(BAR) file:

Broker Archive(BAR)  is a deployable container  in a compressed file format which contains a single deployment descriptor (broker.xml), compiled message flows(*.cmf), message set dictionary files(*.xsdzip, *.dictionary), style sheets(.xsl), XSLT files and JAR files. When you unzip the BAR file, the single descriptor file can be found under META_INF folder. The deployment descriptor file has information about configuration properties of the flow and nodes.

Sample contents of a deployment descriptor file(broker.xml):
<Broker>
<CompiledMessageFlow name="com.ibm.wmb.build.SampleMsgFlow1">
...

<ConfigurableProperty uri="com.ibm.wmb.build.SampleMsgFlow1#additionalInstances"/>
<Con
figurableProperty override="DB2_DSN_DEV" uri="com.ibm.wmb.build.SampleMsgFlow1#Compute.dataSource"/>
<ConfigurableProperty override="SAMPLE.DEV.IN" uri="com.ibm.wmb.build.SampleMsgFlow1#MQ_Input.queueName"/>
...
</CompiledMessageFlow >
</Broker>

Step 1: Get Source Code from SCM:

Read the rest of this post »

Message Broker’s ABITSTREAM function

I recently had a unique integration requirement where an external service provider was expecting XML within the SOAP request. One automatically thinks CDATA but in this case the endpoint required the XML to be escaped within a SOAP request.

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://exacttarget.com/wsdl/partnerAPI">
  <SOAP-ENV:Body>
  <ns1:CreateRequest>
  <ns1:Name>XML_Data</ns1:Name>
  <ns1:Value>&lt;order&gt; &lt;ns1:Attributes&gt; &lt;ns1:Name&gt;Transaction_ID&lt;/ns1:Name&gt;&lt;ns1:Value&gt;Transaction_ID goes here&lt;/ns1:Value&gt; &lt;/ns1:Attributes&gt; &lt;/order&gt;
  </ns1:Value>
  </SOAP-ENV:Body>
 </SOAP-ENV:Envelope>

After multiple attempts trying to solve this with XSLT’s a colleague suggested why not move back to ESQL and use the ABITSTREAM function. A few minutes later the below ESQL snippet proves once again why Message Broker is a true Enterprise ESB.

SET soapReq.ns:Objects.ns:Subscribers.ns:Attributes[1].ns:Name='XML_Data';
 SET myMsgBlob = ASBITSTREAM(InputRoot.XMLNSC.Order, 
                                       InputRoot.Properties.Encoding, 
                                       InputRoot.Properties.CodedCharSetId, 
                                       '','','', 
                                       FolderBitStream );
  SET myMsgChar = CAST( myMsgBlob AS CHAR CCSID InputRoot.Properties.CodedCharSetId ENCODING InputRoot.MQMD.Encoding); 
 SET soapReq.ns:Objects.ns:Subscribers.ns:Attributes[1].ns:Value = myMsgChar;

Perficient & IBM – Making an Impact in 2013

Perficient is Showcasing Our Award-Winning IBM SOA, BPM, and Mobile Solutions & Services at IBM’s Impact 2013 Conference

Perficient is a gold sponsor of Impact 2013, and our experts are available in Booth G9 to discuss how customers can implement impactful technology solutions including BPM, service-oriented architecture (SOA), cloud, mobile and WebSphere solutions for BPMS-Practicemajor industries like healthcare, financial services, automotive and retail. Visitors to the booth can learn how Perficient’s IBM solutions and its technology offerings help companies align business and IT objectives, establish optimal business processes, accelerate productivity and reduce costs.


Watch live streaming video from ibmimpact at livestream.com

Perficient is highlighting our IBM Mobility Practice at IBM’s Impact 2013 conference, showcasing how companies, including retailers, can transform their business processes to enable multichannel and mobile commerce initiatives.

Perficient and our clients Target, TBC and Monsanto are demonstrating BPM, SOA, mobile and cloud capabilities in the following sessions:

Learn more about Perficient and our IBM solution capabilities at Impact – www.perficient.com/impact

Get live updates during the events, follow Perficient experts via Twitter @Perficient_IBM and follow the Perficient IBM Technologies blog.

Attending IBM Connect? Get Bold!

Join Perficient at IBM Connect this year. You can download the eBook “Get Bold”, and learn how the enterprise is responding to consumer expectations for engagement via social channels.

Whether you are planning on attending IBM Connect or not, Perficient would like to offer you this valuable eBook. “Get Bold” explores innovative ways businesses are using social media to improve collaboration internally, better understand and measure markets, and reach new customers.

Get Bold Now! 

Perficient is an award-winning Premier IBM solution provider. Our experts are adept at using the latest solutions from IBM for Social Business and Collaboration to create exceptional web experiences that improve customer satisfaction with the right mix of personalization, self-service, and commerce abilities. If you are planning to attend IBM Connect we invite you to learn more about Perficient’s IBM capabilities and attend our Connect 2013 breakout sessions.

Get BoldSession 1238 – Social CMO: Engaging the Consumer
Speakers: Michael Porter, Perficient; Saren Sakurai, Perficient XD

Session 1713 – Connect and Collaborate Globally: Fluor Corporations Social Business Intranet
Speakers: Say Lim, Fluor Corporation; Glenn Kline, Perficient

Session 1184 – Case Study: Saving Money and Lives Through Excellent Web Experiences
Speakers: Denise Hatzidakis, Premier healthcare; Jonathan Distad, Perficient

Stop by our booth and attend one of our sessions to learn more about Perficient and how we can help you meet your IBM solution needs, and don’t forget to request your “Get Bold” eBook.