Subscribe via Email

Subscribe to RSS feed

Get R Running over YARN-based MapReduce

Out of the mathematical and statistics language and tools such as SAS, SPSS, Matlab, etc. R language is a pretty good tool which provides the environment and essential packages for statistical computing and graphics. It is free and it offers an open environment and the means to allow users to develop custom package.

In addition to R, there is an IDE called RStudio, which is a powerful and web based user interface for R. It’s also free (the advanced options and support require some licenses),  open source, and works great on Windows, Mac, and Linux. RStudio has a desktop and server version.


We know that if we aim to perform data mining and machine learning from big data in the Hadoop environment, Mahout can be a great option which offers core algorithms like clustering, classifying and collaborative filtering. Compared to Mahout, R has its own advantages in  algorithm and computation speed. Furthermore, Mahout Community has announced that it will reject the new MapReduce algorithm implementations beginning in  May of this year.

Recently GDC China’s big data lab team established the R and RStudio in the Hadoop clustered environments. We obtained some experience and learned lessons on putting these components together. The  steps for doing this are as follows. Read the rest of this post »

Tags: , , , ,

Posted in BI&Database

Two “shipping” ways of using docker

Docker is a platform for developers and sysadmins to develop, ship, and run applications. Ship the “Dockerized” app and dependencies anywhere – to QA, teammates, or the cloud – without breaking anything. It’s one of the most important purposes to use docker.

Two shipping ways of using dockerIn Vernon Stinebaker‘s earlier post “Docker, mobile, and putting things in boxes“, he introduced the background of the Library system, the corresponding mobile applications (Andriod version and iOS version) and the need of dockerization. As one of the team members working in this ‘Library in a box’ project, I mainly focus on the part of “putting things in boxes”.

In this article, I am not going to introduce the detail steps of dockerization. Instead, I am going to introduce two “shipping” ways via using docker.


The target is to dockerize the application. By given the pressure from the project schedule, another team member and I decided that firstly we just used the interaction mode to install all the necessary software, Java 7, Liferay, and mysql to get the infrastructure set up. And then deployed the library application war file into the container. All of these steps were just same as an usual environment preparation and the application deployment. The only difference was that the installation and deployment wass done upon a docker container instead of a virtual machine. As highlighted in below command, the arguments “-t” and “-i” enable you to interact with the container.

$ docker run -t -i ubuntu:trusty/bin/bash

Once this was done, we committed the container as a docker image. At the first two Sprints, we delivered the docker images to the PO (Vernon Stinebaker) using the docker save command.

Read the rest of this post »

Service discovery, segmented networks, and two pieces of Pi

In an earlier blog post I noted that I like things that just work. To this end one of the User Stories I related to the Library in a Box team was “As a mobile app user I would like the library to be found automatically if the library service is available on the network so that I can connect to it without requiring intervention (typing in a computer name or IP address).”

This sounds simple enough, and indeed it should be. But like many things in the world of technology things aren’t as simple as they might first seem.

Read on to share some of our discoveries about challenges with zeroconf networking. Read the rest of this post »

Retrieve and Update Data in EssBase database using java API


Essbase API Overview

 Oracle Essbase Provides a solution of EPM (Enterprise Performance Management) that helps end-user analysts across the enterprise in various departments Such as finance, accounting, Sales and marketing to solve complex calculations and perform ad-hoc data query operations .

We operate Essbase in a client server environment from custom applications written in Java using the Essbase Java API.  The Essbase Java API is installed with the Essbase Client.

API is an interface between the Essbase and client program which manage the transfer of data between server and client. Custom program calls to functions within the API and data is returned from the Essbase servers you connect to Header file that include in

Source Code of custom program contain API functionality and set a library that you link to custom program)

API Functionality

For application programmer is important to include in the program for function to run successfully


Initializing the Essbase API A program should initialize the API before calling other EssbaseAPI function. Below function initialize all internal API variables.

initialize all internal API variables

initialize all internal API variables

Note:  IEssbase.Home object exits solely for initializing an instance of the Java API. You must also tell the API which version of the API you are writing your code

Logging into an Essbase Server

Program prompts the User for user name, Password, Server name and Provider.

User tries to log in with the login credential to that server.

Logging into an Essbase Server

Logging into an Essbase Server

 S_olapServer object for the required server


 Selecting an Active Application

Once user is logged in the server the login function (Esslogin) also return a list of the applications and databases to which the logged in user has access. The program allows the user to select a specific application.

Selecting an Active Application

Selecting an Active Application

If we want any description about a specific application Call setDescription.

And to get active Olap application name call getOlapApplicationName


We  get  return value  zero means  successfully  logged in  and  non zero  means we  get  some error  while login.

setting IEssbase and IEssCubeView value   Null

setting IEssbase and IEssCubeView value Null

Above we are setting IEssbase and IEssCubeView value   Null.


Above we call method signOn (String username, java.lang.String password, Boolean, java.lang.String providerUrl,) to enter in Essbase Server.

We call method OpenCubeView (String arg0, String arg1, String arg2, String arg3) to enter into the application.

call method OpenCubeView  to enter into the application

call method OpenCubeView to enter into the application

SetRepeatMemberNames (Boolean repeatMemberNames) to check whether any member is getting repeat.

SetIncludeSelection (Boolean include Selection) to reflect all changes.

We call updatePropertyValues () method because update will only reflect when we will call this method


We call this performCubeViewoperation ( )Class to display the   members and their values.

call  performCubeViewoperation ( )Class

call performCubeViewoperation ( )Class

Here we are using Gride Class to display member in matrix form. Inside Grid we are using SetSize and SetValue methods.

Gride Class to display member in matrix form

Gride Class to display member in matrix form

Above we are calling createIEssOpRetrieve( ) method to create the operation specification and to get operation object.

create IEss Op Retrieve

Above code counts the Row and column and display the grid in matrix form.

We are using getCountRows ( ) to count the rows and getCountColumns ( ) to count the columns method.

"GetCountRows" to count the rows

“GetCountRows” to count the rows

We are calling getCellContent (int row, int column) method to get the cell content at the specified row, column intersection.

Call IEssGridView () method to display our output in Excel add in grid.

Call getDouble (int columnIndex) to retrieve the value of the designated column in the current row of this Result Set object as a double.

Below example   I retrieve the value of units (12666) and changed into 1266714dBelow its showing changed value


We call creatIEssopUpdate() method to update the  value in database.

Creat IEssop UpdateBelow code counts the updated row and column and display the result

Counts the Updated row and column



Overview of IBM MDM PIM Scripting Concepts


IBM MDM PIM tool is a middleware software component provides robust solutions for Product Information Management. It is designed and implemented based on rich set of MDM technologies and concepts. It enables enterprises to manage single source of reliable and most trusted information. It provides a repository of product information that is accurate, non-redundant, synchronized and most updated. The basic Overview of IBM MDM PIM Scripting Conceptsfunctionality of the MDM PIM objects can be experienced by the UI interface. In today’s business world to meet enterprises goals and provide much more user friendly interface we need to have an environment where we can provide the solutions in dynamic fashion. In MDM PIM we can use scripting mechanism to extend the basic functionality of the MDM PIM objects towards business requirements and needs. Customization and dynamic functionality can be achieved by the Scripting concepts.

Read the rest of this post »

How to implement agile testing on a non-agile project

Often, people think agile testing only can be implemented on an agile project. Actually agile testing is a set of engineering practices, and it can be implemented on any kind of project.

shutterstock_143577256For example, as a typical agile testing practice, test-driven requirements can be leveraged in any kind of project. Even for a traditional waterfall project, we can do test case/script development with requirement development in parallel. All unclear points for preconditions, scenarios, testing data will be found and clarified earlier. It is easy to help a team to make requirements ready as early as possible, and it will avoid a lot of costs caused by bad quality in the requirements.

Then which kind of agile testing practice can be leveraged for any kind of project? It will include but not be limited to the following:

Read the rest of this post »

Would you like a “little rice” with your container?

Sorry, it’s a pun(ny) title. If you haven’t gotten it already, you will soon (ahhh! stop it with the puns already!)

My family, colleagues, and people who attend my Scrum training will all attest to my being a big fan of Apple’s products. I appreciate good design and I like things that “just work”.

But I’m also a gadget-head in general.

Read the rest of this post »

Docker, mobile, and putting things in boxes

Docker and custom mobile application development are both very hot. Recently we decided to run a small internal project to gain some ‘sleeves-up’ insight into Docker as well as how we could deliver containerized versions of applications.

This blog article, along with others to follow from both my colleagues and myself will document some of our learning. We hope they will be of value to others who may not have the time or environment to conduct a similar exercise on their own.

Read the rest of this post »

Master Data and Integration Tools

In today’s business world, the major challenge that enterprises face is to have one source of business data (Master Data) which is non-redundant, more consistent, accurate, complete and error free. Master data of an enterprise could be information about its customers, products, employees, suppliers, business locations, etc. Master data plays an important role in making key business decisions. An organization could have several systems as data sources where each system has its own data model and structure. So there could be more chances with high probability to have inaccurate and redundant data when there are heterogeneous source systems.

Read the rest of this post »

100% code based configuration in Spring MVC

If you’re a JEE developer, you must be bored with XML configuration. Thanks to new features of Spring project, now you can have a 100% code based configuration for a web application. Here are some tips:

Read the rest of this post »

Posted in Web application