In this portion of blog on MuleSoft we will try to demonstrate how we can use the concept of water-marking in polling at regular intervals to check whether any new information was added, available etc. for comparison and processing purposes.
Assumption:
- The developer has fundamental knowledge of how to write code in MuleSoft
- Developer is able to use the latest version of studio and Mule runtime (3.8) on local system for ease of demo
What is polling and water-mark?
Polling is a mechanism by which MuleSoft code is able to check some conditions at a fixed regular interval. Specifically, you can query a data store, source system or even check the value of a field to see if there were any changes from last time.
If there was a non-zero difference between the last reading and the current reading, then the last reading/ observation can be used as a “water-mark” to compare with subsequent readings for identifying the differences. Hence, in this context the water-mark is similar to the water-mark used to measure the rising water levels to predict and control floods in real damns, lakes etc.
How to set up polling and water-marking
To elaborate, I will take a simple case of a database and polling will be done to check the records in the database table. For simplicity, I will assume mySQL is on a local machine as the database. Please see snapshot below for a sample code:
In the code above, I have polled a database (using select query) every 5 seconds to check the records. To do this, I used a where clause and compared the value of the primary key with the last highest value.
Only the first time, the result will fetch all the current records in the table. After that, all subsequent calls will compare and check if any new entry was added into the table. If yes, it will bring and print only the new records.
If there were no changes in the table, you will see a message as below:
This concept is known as watermarking in ESB.
Where can we apply this watermarking and polling concept in real use-case:
- Data synchronization between multiple backend systems: This watermarking strategy can be used to compare and synchronize the data between multiple backend systems. We do not have to rerun the entire set of records between two data stores. Instead, only the delta (difference ) will be compared and sync up will be performed. This is much better for performance and processing overheads.
At Perficient, we have subject matter expertise working with various clients to help solve their business problems leveraging the functionalities with MuleSoft ESB platform. For more information please feel free to contact us.
