Informatica Intelligent Cloud Services (IICS) now offers a free command line utility that can be used to integrate your ETL jobs into most enterprise release management pipelines. It’s called the Asset Management command line interface (CLI). Version two now allows you to extract an IICS job into a single compressed file. Moving a single standalone artifact from Development through QA and Production using a script makes it easy to incorporate your ETL pipeline into common release management tools. This can have profound implications for the deployment stage of a complex enterprise release cycle. Consider a typical war room scenario in a major release and how we can use the new Informatica tools to improve the deployment procedure.
The War Room
In a war room (aka situation room or command center), key stakeholders gather together to focus on solving a specific, critical problem. A war room for a major enterprise release is typically focused on the coordinated deployment of artifacts such as application code, store and compute resources and database changes. Database changes refer to both structure changes (ex: table definitions) and data migration, which is where Extract Transform and Load (ETL) tools like Informatica come into play. The divide between the teams that have a CI/CD practice like the application development and infrastructure teams and the teams that rely on manual deployments like the database administrators (DBAs) and ETL teams become painfully clear. Using a common frame of reference before the Production deployment is the major source of disconnect. It makes sense to have an integrated release management system before deployment.
There are three concepts that make a good release management system; source code control, a pipeline to coordinate deployment of artifacts and a repository to store versioned instances of these artifacts. Git is the de facto version-control system for tracking changes in source code during software development. Git can also be used to track changes in any set of files. Jenkins is an automation server that executes pipelines related to building, testing and deploying code, usually code stored in a Git repo. Like Git, Jenkins can also be used to deploy any kind of file, even files that are not the subject of building and testing. Finally, a tool like Artifactory can be used to store the binaries that are the target of deployment. This is optional since you can configure your source code tool to store releases. We will not demo it here, but it is a nice tool. There are several approaches that try to insert Informatica at the beginning of the CI/CD process and emulate the same pipeline as a software development project. But Informatica is not a custom Java application, and doesn’t need to be. Invert the process and start from release management.
Build an Informatica release pipeline
To build an Informatica release pipeline, we are going to start by exporting an Informatica project as a single file. This is the key differentiator between using Informatica as a standalone application and integrating Informatica into your enterprise’s release pipeline. For the sake of this blog, we are going to assume that your organization uses GitHub for source control and Jenkins for orchestration. The open source tools are popular enough that you will easily be able to find the equivalent instructions for a different application. I deployed this example using Amazon Web Services but this will work on any public or private cloud.
Create a new repository in GitHub to store the sample Informatica project and deploy to Jenkins.
- Navigate to GitHub and sign in.
- In Your repositories, choose New repository.
- On the navigation bar, choose Create new (+), and then choose New repository.
- In the Create a new repository page, do the following:
- In the Repository name box, enter IICSDemo.
- Select Public.
- Clear the Initialize this repository with a README check box.
- Choose Create repository.
Go to AWS (or another cloud provider) and create a free, barebones Linux-based computer instance. Connect to the new instance with a terminal and execute the following:
mkdir ~/workspace/default && cd "$_" touch README.md git init git add README.md git commit -m "Init IICS demo" git remote add origin https://github.com/
user-name/IICSDemo.git git push -u origin master
You now have a working github repository. In the next step, we will setup Jenkins.
Clone the ec2 instance you created for the IICS CLI and connect to the new instance in another terminal window and execute the following:
sudo yum install docker sudo usermod -a -G docker ec2-user newgrp docker sudo systemctl enable docker sudo systemctl start docker sudo systemctl status docker docker run \ -u root \ --rm \ -d \ -p 8080:8080 \ -p 50000:50000 \ --name jenkins_iics \ -v jenkins-data:/var/jenkins_home \ -v /var/run/docker.sock:/var/run/docker.sock \ jenkinsci/blueocean # get the admin password docker exec -it jenkins_iics bash cat /var/jenkins_home/secrets/initialAdminPassword exit
You use the admin password to log into Jenkins, which will be running on the public url of your EC2 instance on port 8080. Create a new user so you don’t log in with Admin all of the time.
- Click New Item
- Enter ‘IICS Default’ as the item name
- Select Freestyle Project
- Click OK
- Scroll down to the Build section
- Select ‘Execute script’ from the ‘Add Build Step’ drop down.
- In the command box, type echo “Hello, World’
- You can now test the pipeline by clicking Build Now.
- Once the build is complete, click on the Console Output to verify the Hello World message was printed
Being able to execute a script from Jenkins demonstrates that you can integrate with almost any build pipeline. You could script executing processes, moving files, etc. Next, we will integrate Jenkins into GitHub.
Integrate GitHub and Jenkins
You will copying and pasting information from Jenkins to GitHub and vice versa, so have both tabs open.
- In Jenkins,
- scroll down to the Source Code Management section
- Select Git
- From Github repo,
- Cick on the Clone with HTTPS button and paste the url into the Repository URL text box in Jenkins
- Next, connect Jenkins to Github’s checkin events
- Navigate to Setting and click WebHooks
- Add the Jenkins url and append /jenkins/github-webhook (ex http://220.127.116.11/jenkins/github-webhook/)
- Select application/json as the content type
- We just want the push event.
- Click Add Webhook
At this point, we have a README file checked into GitHub and Jenkins running a simple Hello World script. In the next step, we will extract the Default project from IICS into the git repo, which will trigger a Jenkins build.
Setup IICS cli
Sign up for Informatica Intelligent Cloud Service training and get access to a free instance for thirty days. This first part of the url is your pod name (ex: na1.dm-us.informaticacloud.com). Note your region and make sure you save your user name and password. Execute the following commands from the same terminal instance where you configured git.
mkdir ~/tools && cd "$_" get https://github.com/InformaticaCloudApplicationIntegration/Tools/raw/master/IICS%20Asset%20Management%20CLI/v2/linux-x86_64/iics chmod 775 iics ./iics version
You now have an executable instance of the IICS Asset Management command line interface and it should be version two. Let’s see if you can connect to your IICS instance.
./iics list -region us --podHostName na1.dm-us.informaticacloud.com --username firstname.lastname@example.org --password xxxxxxx
You should see something like
Now, export a project into a working directory.
cd ~/tools ./iics export -region us --podHostName na1.dm-us.informaticacloud.com --username email@example.com --password xxxxxxx --zipFilePath ~/workspace/default/default.zip --logLevel info --artifacts Explore/Default.Project
You should now have a zip file called default.zip in the default directory of your workspace. Check it in.
cd ~/workspace/default git add default.zip git commit -m "Push first export from IICS to GitHub" git push -u origin master
Go to GitHub and confirm that you have a new file in your repository. Go to Jenkins and confirm that a new build was triggered on the commit. You have just extracted an Informatica job as a single artifact, checked it into GitHub and executed a Jenkins build. If you build this process into your normal routine, you will find it easier to integrate into larger enterprise projects in the future. In the next war room, you’ll all be speaking the same language.
This is not the fail-fast model usually enabled by Continuous Integration/Continuous Release cycles. ETL project typically do not fall into this category. IICS usually moves key business information from an on-premise source system to a cloud-based data warehousing solution. These ETL project enable fail-fast projects, but they themselves are typically the subject of deep controls. However, you can integrate your Informatica Intelligent Cloud Services jobs into a standard deployment pipeline by using the new Asset Management command line interface (CLI) .