Log Miner for Your TIBCO Infrastructure

Nowadays all over the world, IT companies are evolving with the term of ‘Automation’. The term Automation describes ‘Huge time and cost saving effort to the consumers’. Keeping in mind that industries have developed many automation tools. In that ocean, I’m going to talk about a few Log Automation tools in the TIBCO landscape. Tools like Splunk, ELK, and SIEM platform are used widely in all the production boxes. In my project environment we are also using Splunk to extract the logs from the production box, but when comes to SIT and UAT environments tools like SPLUNK would be a costly affair. Therefore, I decided to build a custom log automation tool leveraging the powerful capabilities of TIBCO, which I fondly call ‘Log Miner’. Let’s talk in detail about this implementation.

In my project, we tried to extract the logs manually from the SIT and UAT boxes. Our developers and testers struggled to fetch the logs whenever a defect comes to us. We spent so much time fetching the logs that it affected our productivity. So we discussed the issue and gathered the requirements. Finally, I decided to come up with a simple UI so that new users don’t need to log in complex putty sessions and manually search logs in our Linux environment.

Why Log Miner?

In my project we have two different set of log files that say LogServer (contains DEBUG and INFO details of the applications) and LogError (contains ERROR details of the applications.) Yes, all the projects have this but in this situation each file has more than 0.5 GB and the files are placed in the UNIX server. Whenever a defect arrives, the user has to login to the corresponding environment (computer machines/hostnames) and verify the logs manually by searching all the files using a Unique ID. There are so many lines in Terminal which makes it very difficult to go through, analyze and becomes a very time-consuming process.

What is Log Miner and How Does it Work with TIBCO and Other Open Source Tools?

Log Miner is the combination of various technologies. It creates  a metadata sheet (containing details of possible unique id’s) out of log files and also provides two optimistic ways of search for the users. The options are:

Advance- Search: Allows users to search and get the user specific lines by simply clicking the advance- search button. This search uses a metadata file which is created by Log Miner itself.

Classic Search: Allows users to search and get the details from the all the log files. This search will give the collective matching blocks as a file from the log files and users can download the file by simply clicking the classic search button. Using this, the user doesn’t need to worry about searching all the files because this classic search will do everything for them.

This is how Log Miner works:

Technologies we used….

First and foremost, we used TIBCO BW, ADMIN. Using TIBCO BW I created processes for creating a metadata sheet out of log files and created processes for simple UI for the users to give their input and get the desired results. In TIBCO ADMIN, TIBCO has inbuilt the TOMCAT Server which is used for hosting BW applications. Through TIBCO ADMIN I hosted my application. Users can then access the application by hitting the service by the given URL.

Also, I used the most efficient and powerful scripting language ‘Shell Script’ (as I mentioned earlier log files are placed in UNIX server) in Log Miner to fetch the details from the log files and metadata files as well. Finally, for UI, I used the simple and familiar web technology tool HTML and CSS to create the UI. Those are all the parts of Log Automation. Here are few TIBCO Process and UI screenshots…

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

TIBCO Process

Reading lines from the Logs and passing records as batches.

Process to Create Metadata (as I said earlier all possible Unique ID is called metadata).

Process to Receive the User Request and Send Response.

Simple HTML UI

Home Page

User Search Page

Advance Search Results Page.

After clicking the get lines button, you will get the following:

When users click on the classic search button, files will be downloaded with the requested details.

Ideal Outcome of Automation using TIBCO….

After this implementation, we saw a considerable reduction in time and effort required to debug issues.

I hope you have enjoyed reading this blog on Log Miner using TIBCO. If you have any queries on this implementation leave your queries below and I’ll be glad to assist! looking forward to hearing your comments.