As REST APIs are taking over the world, json has stood out and become the de facto data format for APIs. It’s important that developers are familiar with json data processing.
A couple of years ago I wrote a blog post discussing Mule Json transformers. Since then, I have seen many new nuances dealing with json in a Mule flow. In this new post, we’ll take another look at json data processing with Mule transformers and Dataweave.
MuleSoft DataSense
One interesting feature of Mule flow is it attempts to “automagically” interpret data format on your behalf. It’s called DataSense. At design time, when you drop a message processor somewhere in the flow, the Anypoint Studio IDE auto senses the inbound and outbound payload, and handles / converts data in the best format it can think of (Mule runtime does the same thing). This great feature can be a double-edged sword. It helps greatly when it works correctly since you don’t have to worry about what’s going on under the hood. However, when it doesn’t work, you are left to scratch your head for a long time.
We’ll look at some examples how DataSense works “automagically”, and when we need to step in and tell Mule what the correct data type should be.
The json transformers
Mule comes with quite a few json transformers. We’ll look at two of them closely in this post: “json to object” and “object to json”.
If you hover over the “json to object” and “object to json” transformers on the studio palette, you will see the following descriptions:
“The JSON to Object Transformer converts a JSON encoded object graph to a Java object” and “The Object to JSON Transformer converts a Java object to a JSON encoded object that can be consumed by other languages, such as JavaScript or Ruby”.
Examples
In all the examples, we assume we will start with a Json payload that looks like below. It’s an array with two entries of map (i.e. pairs of key and value):
[{“course1″:”Introduction to Mule”}, {“course2″:”Advanced Mule”}]
Example 1 – “JSON to Object” the default behavior
See code snippet below, in this example:
- We first set payload to the json string.
- Then use “json-to-object” transformer. By default, this transformer coverts the Json string to an internal JsonData object.
Please note: if the inbound payload is not a well-formed json string, the transformer will throw exception.
3. Now the payload can be accessed like:
- JsonPath: #[json:[0]/course1] – This is a deprecated feature, won’t be supported in 4.0.
- JsonData object: )#[payload.get(0)] – Very uncommon to do it this way. You have to look up JsonData API to figure out what to call.
- Dataweave: mycourse: payload[0].course1 – The payload can also be parsed by Dataweave with array-like syntax. Although payload is JsonData, this is DataSense at work. It “knows” what to do to parse the data.
<set-payload value=“[{“course1″:”Introduction to Mule”}, {“course2″:”Advanced Mule”}]“ doc:name=“Set Payload-json-map-text”/>
<json:json-to-object-transformer doc:name=“JSON to Object-default JsonData”/>
<logger message=“json:[0]/course1=#[json:[0]/course1], payload.get(0)=#[payload.get(0)], json:[1]/course2=#[json:[1]/course2]” level=“INFO” doc:name=“default json to obj”/>
<dw:transform-message doc:name=“Transform Message”>
<dw:set-payload>
<![CDATA[%dw 1.0
%output application/json
—
mycourse: payload[0].course1]]>
</dw:set-payload>
</dw:transform-message>
Example 2 – json to object with return type “java.lang.Object” or “java.util.HashMap[]”
In this example,
- we first set the json payload like in example 1
- then we call “json to object” also like example 1, but we added returnClass=“java.util.HashMap[]”. This way, we control the result payload type instead of getting the default JsonData. Please note the return class type has to be correct for the input json, otherwise the transformer will throw exception.
Please note, you can also use returnClass=“java.lang.Object”, let the transformer figure out the return class. If you do that in this case, it would return java.util.ArrayList. The list contains HashMap entries.
- Now the payload can be parsed easily with MEL, like #[payload[0].course1], #[payload[0][‘course1′]], #[payload[0].’course1’]
- For Dataweave, however, the DataSense can’t figure out the inbound type without some human help (see screen capture below). We need to tell Dataweave the input payload type by setting the input payload meta-data type as a custom data type: choose Java, then collection, then java.util.HashMap
With the meta-data, then Dataweave can parse the payload as an array like “new course1: payload[0].course1”
<set-payload value=“[{“course1″:”Introduction to Mule”}, {“course2″:”Advanced Mule”}]“ doc:name=“Set Payload”/>
<json:json-to-object-transformer returnClass=“java.util.HashMap[]” doc:name=“Copy_of_JSON to Object-java.util.HashMap[]”/>
<!–json:json-to-object-transformer returnClass=“java.lang.Object” doc:name=“Copy_of_JSON to Object-java.lang.Object”/–>
<logger message=“array of map, #[payload[0].course1], #[payload[0][‘course1′]], #[payload[0].’course1’]” level=“INFO” doc:name=“Logger map”/>
<dw:transform-message doc:name=“DW wt input meta-data collection<HashMap>” metadata:id=“80d9dde1-a513-4e33-a1e9-d6d905da18a4”>
<dw:input-payload mimeType=“application/java”/>
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
—
newcouse1: payload[0].course1]]></dw:set-payload>
</dw:transform-message>
<logger message=“payload=#[payload]” level=“INFO” doc:name=“Logger”/>
Example 3 – Object to Json
This example tests the default behavior of “Object to JSON” transformer. It’s nearly the same as example 2, except after we convert payload to HashMap[], we call “Object to JSON” right away, and the payload is converted into a plain string object with a mime type of “application/json”.
Does this example have any practical value? Maybe. If you have a long flow, at some point you need to access the payload using MEL and java map, then at some later stage you need to use dataweave again, instead of going through the troubles with the meta-data manipulation like in example 2, you can just convert it back to a json string (with mime type application/json).
Here is another practical use case. When you use DB connector to query database, the return result is a list of HashMap. If you want to emulate the DB query later (for debugging or test purpose) without connecting to DB, you can use this trick to capture the DB query result as a json array, write out the json array as a string in the log. Capture the string, then you can use that to emulate your DB query. You can modify the values any way you want. It can be a great trick to run different DB query test cases.
<set-payload value=“[{“course1″:”Introduction to Mule”}, {“course2″:”Advanced Mule”}]“ doc:name=“Set Payload”/>
<json:json-to-object-transformer returnClass=“java.util.HashMap[]” doc:name=“JSON to Object-java.lang.Object”/>
<logger message=“array of map, #[payload[0].course1], #[payload[0][‘course1′]], #[payload[0].’course1’]” level=“INFO” doc:name=“array of map”/>
<json:object-to-json-transformer doc:name=“Object to JSON – result-payload-string-wt-mime-app/json”/>
<dw:transform-message doc:name=“DW-same-as-#1”>
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
—
course1: payload[0].course1]]></dw:set-payload>
</dw:transform-message>
<logger message=“payload=#[payload]” level=“INFO” doc:name=“Logger payload”/>
Example 4 – what not to do
Not to beat the dead horse, but what if you try to convert the HashMap payload directly as a string in example 3 and give it an “application/json” mine type?
Unfortunately, our curiosity has gone too far this time, the result payload is:
“{{course1=Introduction to Mule},{course2=Advanced Mule}}“.
Even if the payload has a “application/json” mime type, it almost looks like a json string, if you look closely, it is NOT a valid json structure! In fact, it is not of any known data structures. It is just a nice looking string for human eyes, but the flow can no longer parse it!
<set-payload value=“[{“course1″:”Introduction to Mule”}, {“course2″:”Advanced Mule”}]“ doc:name=“Set Payload”/>
<json:json-to-object-transformer returnClass=“java.util.HashMap[]” doc:name=“JSON to Object-java.util.HashMap[]”/>
<set-payload value=“#[message.payloadAs(java.lang.String)]” mimeType=“application/json” doc:name=“Set Payload-to-string-not-json”/>
Example 5 – Mime type trick
If you just want to parse that sample json data with Dataweave, here is the simplest way to do it.
- We set payload to the same json string. However, we also set the mime type to “application/json”
- Although the payload type is still string, now DW knows how to parse it. I suppose that’s the magic power of DataSense.
Please note that you cannot parse the payload with MEL, JsonPath or anything else, because technically the payload is still a plain string.
<set-payload value=“[{“course1″:”Introduction to Mule”}, {“course2″:”Advanced Mule”}]“ mimeType=“application/json” doc:name=“Set Payload Json-array-mime-app/json”/>
<dw:transform-message doc:name=“DW-works-only-wt-mime-app-json-payload[0].course1”>
<dw:set-payload>
<![CDATA[%dw 1.0
%output application/json
—
course1: payload[0].course1]]>
</dw:set-payload>
</dw:transform-message>
Example 6 – load Json map from a text file
I’ll be remiss not to show another Json processing technique: load json data from a text file, since it’s fairly common we get json data as a text file, or we want to use json data file to drive our emulation or test.
We first load the data file (stream), convert it to string, then “object to json” converts it Java object.
<set-payload value=“#[Thread.currentThread().getContextClassLoader().getResourceAsStream(‘json-map.json’)]” mimeType=“application/json” doc:name=“load-json-map-file”/>
<!—convert stream to string –>
<object-to-string-transformer doc:name=“Object to String”/>
<!—convert json string to internal Java class –>
<json:json-to-object-transformer doc:name=“JSON to Object” returnClass=”java.lang.Object”/>
