Perficient Enterprise Information Solutions Blog

Blog Categories

Subscribe via Email

Subscribe to RSS feed

Posts Tagged ‘Big Data’

Disruptive Scalability

The personal computer, internet, digital music players (think ipods), smart phones, tablets are just a few of the disruptive technologies that have become common place in our lifetime.   What is consistent about these technology disruptions is that they all have changed the way we work, live, and play.  Whole industries have grown up around these technologies.   Can you imagine a major corporation being competitive in today’s Disruptive Scalabilityworld without personal computers?

Big Data is another disruptive technology.    Big Data is spawning its own industry with 100s of startups and every major technology vendor seems to have a “Big Data Offering.”  Soon, companies will need to leverage Big Data to stay competitive.   The Big Data technology disruption in an Enterprise’s data architecture is significant. How we source, integrate, process, analyze, manage, and deliver will evolve and change. Big Data truly is changing everything!   Over the next few weeks I will focusing my blogging on how Big Data is changing our enterprise information architecture.   Big Data’s effect on MDM, data integration, analytics, and overall data architecture will be covered.   Stay-tuned!

Perficient takes Cognos TM1 to the Cloud

IBM Cognos TM1 is well-known as the planning, analysis, and forecasting software that delivers flexible solutions to address requirements across an enterprise, as well as provide real-time analytics, reporting, and what-if scenario modeling and Perficient is well-known for delivering expertly designed TM1 based solutions.

Analytic Projects

Perficient takes Cognos TM1 to the CloudPerhaps phase zero of a typical analytics project would involve our topology experts determining the exact server environment required to support the implementation of a number of TM1 servers (based upon not only industry proven practices, but our own breadth of practical “in the field” experiences). Next would be the procurement and configuration of said environment (and prerequisite software) and finally the installation of Cognos TM1.

It doesn’t stop there

As TM1 development begins, our engineers work closely with internal staff to outline processes for the (application and performance) testing and deployment (of developed TM1 models) but also to establish a maintainable support structure for after the “go live” date. “Support” includes not only the administration of the developed TM1 application but the “road map” to assign responsibilities such as:

  • Hardware monitoring and administration
  • Software upgrades
  • Expansion or reconfiguration based upon additional requirements (i.e. data or user base changes or additional functionality or enhancements to deployed models)
  • And so on…

Teaming Up

Earlier this year the Perficient analytics team teamed up with the IBM Cloud team to offer an interesting alternative to the “typical”: Cognos TM1 as a service in the cloud.

Using our internal TM1 models and colleagues literally all over the country, we evaluated and tested the viability of a fully cloud based TM1 solution.

What we found was, it works and works well, offering unique advantages to our customers:

  • Lowers the “cost of entry” (getting TM1 deployed)
  • Lowers the total cost of ownership (ongoing “care and feeding”)
  • Reduces the level of capital expenditures (doesn’t require the procurement of internal hardware)
  • Reduces IT involvement (and therefore expense)
  • Removes the need to plan for, manage and execute upgrades when newer releases are available (new features are available sooner)
  • (Licensed) users anywhere in world have access form day 1 (regardless of internal constraints)
  • Provides for the availability of auxiliary environments for development and testing (without additional procurement and support)

In the field

Once we were intimate with all of the “ins and outs” of TM1 10.2 on a cloud platform, we were able to to work directly with IBM to demonstrate how a cloud based solution would work to address the specific needs of one of our larger customers. After that, the Perficient team “on the ground” developed and deployed a “proof of concept” using real customer data, and partnered with the customer for the “hands on” evaluation and testing. Once the results were in, it was unanimous: “full speed ahead!””.

A Versatile platform

During the project life-cycle, the cloud environment was seamless; allowing Perficient developers to work (at the client site or remotely) and complete all necessary tasks without issue. The IBM cloud team was available (24/7) to analyze any perceived bottlenecks and, when required, to “tweak” things per the Perficient team’s suggestions, ensuring an accurately configured cloud and a successful, on-time solution delivery.

Bottom Line

Built upon our internal teams experience and IBM’s support, our delivered cloud based solution is robust and cutting edge and infinitely scalable.

Major takeaways

Even given everyone’s extremely high expectations, the project team was delighted and reported back the following major takeaways from the experience:

  • There is no “hardware administration” to worry about
  • No software installation headaches to hold things up!
  • The cloud provided an accurately configured VM -including dedicated RAM and CPU based exactly upon the needs of the solution.
  • The application was easily accessible, yet also very secure.
  • Everything was “powerfully fast” – did not experience any “WAN effects”.
  • 24/7 support provided by the IBM cloud team was “stellar”
  • The managed RAM and “no limits” CPU’s set things up to take full advantage of features like TM1’s MTQ.
  • The users could choose a complete web based experience or install CAFÉ on their machines.

In addition, IBM Concert (provided as part of the cloud experience) is a (quote) “wonderful tool for our user community to combine both TM1 & BI to create intuitive workflows and custom dashboards”.

More to Come

To be sure, you’ll be hearing much more about Concert & Cognos in the cloud and when you do, you can count on the Perficient team for expert delivery.

Web analytics and Enterprise data…

I was looking at the market share of  Google Analytics (GA) and it is definitely on the rise. So I was curious to see the capabilities and what this tool can do. Of course it is a great campaign management tool. It’s been a while since I worked on campaign management.

GA_graphicsI wanted to know all the more now about this tool, off to YouTube and got myself up to speed on the tools capabilities. Right off the bat I noticed campaign management has changed drastically compared to the days when we were sending email blasts or snail mail, junk mail etc. I remember the days when we generated email lists and run it through third-party campaign management tools, blast it out to the world and wait. Once we get enough data (mostly when the purchase the product) to run the results through SAS, we could see the effectiveness. It took more than a month to see any valuable insights.

Fast-track to the social media era, GA provides instant results and the intelligent click-stream data for  tracking campaign management in  real-time. Checkout the YouTube Webinars to see what GA can do in a 45 min.

GA1On a very basic level, GA can track the new visitor, micro conversion (download a newsletter, or add something in a shopping cart), Macro Conversion (buy a product), or is it a returning customer. GA can track the ad-word traffic (how did they get to the website, trigger). It also has a link tag feature– which is very useful to identify the channel (email, referral website etc), linking the traffic to a specific campaign, based on the origination. It has many other features besides cool reports and analytical abilities as well.

There is so much information collected whether the customer buys a product or not. How much of this web analytics data is part of enterprise data. Does historical analysis include this data? Is this data used  for predictive and prescriptive analytics? It is important to ask the following questions to assess what percentage of gathered information is actually used at the enterprise level:

  • How well the organizations integrate this campaign data into enterprise data?
  • Do they collect and manage new prospect information at enterprise level?
  • Does the organization use this tool to enhance their master data?

This may become a Big Data question, depending on the number of Campaigns/ hits and the amount of micro activates the site can offer. Chances are that the data resides  in silo or at a third-party location and the results are not stored in the enterprise data.

SAP HANA and Hadoop – complementary or competitive?

In my last blog post, we learned about SAP HANA… or as I called it, “a database on steroids”. Here is what SAP former CTO and Executive Board Member, Vishal Sikka, told InformationWeek:

SAP HANA and Hadoop – complementary or competitive?“Hana is a full, ACID-compliant database, and not just a cache or accelerator. All the operations happen in memory, but every transaction is committed, stored, and persisted.”

In the same InformationWeek article you can read of how SAP is committed to become the #2 database vendor by 2015.

So, even if HANA is a new technology, it looks like SAP has pretty much bet its future on it. Soon, SAP customers may have SAP ERP, SAP NetWeaver BW, and their entire SAP system landscape sitting on a HANA database.

But if HANA is such a great database, you may wonder, why would SAP HANA need a partnership with Hadoop, or be integrated with Hadoop at all? Can HANA really integrate with Hadoop seamlessly? And, most importantly, are HANA and Hadoop complementary or competitive?

Well, in October 2012, SAP announced the integration of Hadoop into its data warehousing family – why?

The composite answer, in brief, is:

  1. tighter integration – SAP, Hadoop, Cloudera, Hitachi Data Systems, HP, and IBM are all brought together in order to address the ever-growing demands in the Big Data space
  2. analytics scenarios – in order to build more complex and mature analytics scenarios, HANA can be integrated with Hadoop via SAP Sybase IQ, SAP Data Services, or R queries, and include structured AND unstructured Big Data with prior integration and consolidation by Hadoop
  3. in-memory capabilities – some organizations already have existing Hadoop strategies or solutions but cannot do in-memory Big Data without HANA
  4. tailored solutions – by bringing together speed, scale and flexibility, SAP enables customers to integrate Hadoop into their existing BI and Data Warehousing environments in multiple ways, so as to tailor the integration to their very specific needs
  5. transparency for end-users – SAP BusinessObjects Data Integrator allows organizations to read data from Hadoop Distributed File Systems (HDFS) or Hive, and load the desired data very rapidly into SAP HANA or SAP Sybase IQ, helping ensure that SAP BusinessObjects BI users can continue to use their existing reporting and analytics tools
  6. queries federation – customers can federate queries across an SAP Sybase IQ MPP environment using built-in functionality
  7. direct exploration – SAP BusinessObjects BI users can query Hive environments giving business analysts the ability to directly explore Hadoop environments

In short, SAP is looking at a co-exist strategy with Hadoop… NOT a competitive one.

In the next blog post, we’ll look at Hadoop and its position in the Big Data landscape… stay tuned.

Is IT ready for Innovation in Information Management ?

Information Technology (IT) has come a long way from being a delivery organization to an organization part of business innovation strategy, though a lot has to change in the coming years. Depending on the industry and the company culture, IT organization will mostly fall in the operational spectrum and a lot of progressive ones are  gravitating towards innovation. Typically, IT maybe consulted on executing the strategic vision. It is not IT’s role to lead the business strategy but data and information is another story.  IT is uniquely positioned to innovation in Information Management because of their knowledge in data, if they don’t take up that challenge, business will look for outside innovation. Today’s market place offers tools and technologies to business users and they are bypassing IT organizations if they are not ready for the information challenge. A good example will be business users trying out third-party services (cloud), self-service BI tools for slicing and dicing data, cutting down the development cycle. The only way IT can play strategic game is to get into the game.

It is almost impossible for IT not to pay attention to data and just bury their heads in keeping the lights on projects. So I took a stab at the types of products and technologies which is maturing in the last 5 years in the Data Management space. By any means this is not the complete list but it captures the essence.DM_tools_x

Interesting phenomenon is many companies traditionally late to adopt data driven approach are using analytical tools as they become visually appealing and are at a price they can buy. Cloud adoption is another trend which is making the technology deployment and management without a huge IT bottleneck.

The question every IT organization, irrespective of company size, should ask is Are we ready to take on the strategic role in the enterprise? How well they can co-lead the business solution and not just implementing an application after the fact. Data Management is one area IT needs to take the lead in educating and leading innovation to solve business problems. Predictive analytics and Big Data is right on top with all the necessary supporting platforms including Data Quality, Master Data Management and Governance.

It will be interesting to know how many IT organizations leverage the Information Management opportunity.



More on the MDM platform…

Picking up from my earlier blog post, there are two kinds of MDM tool types, one targets specific domain (Customer and Product are the most common ones) and the others follow a multi-domain (Customer, Product, Location, Supplier etc. all in one) strategy. Most of the analysis I found are either for Customer Domain or Product Domain, which includes multi-domain types as well.

So to round-up the top list equitably, I looked at Gartner research as well, thanks to the vendors, most of the reports are in public domain. There is a report from Gartner which you can buy, if you need complete analysis and understanding. Not sure how one gets on the list of these research. But I am assuming, if the market share of a tool is big enough or the technology is way superior, the tool should have made the list. Just a disclaimer, my intention is not to write  research paper but just commentary and some observation.

I looked at 2009, 2011/12 and 2013 magic quadrants for Product and Customer MDM. We see few more companies and some missing ones. Going back to my Forrester slide from 2007 (See my earlier blog), gives us an idea of type of companies approaching MDM and then retreating.

Reading the market news, and from my client experience, most of the medium to large enterprises do fall within the list of vendors we are seeing here. But there are other vendors very much in the market. Also my feeling is that the traditional Data Management software vendors are gaining market share through consolidation and through improved product lines. I am sure market will continue to surprise with new products and services. Microsoft is still playing a low-key in MDM space. Robust MDM from Microsoft will be a game changer.

What is your observation? What is your experience?



Exercising IBM Cognos Framework Manager

In Framework Manager, an expression is any combination of operators, constants, functions, and other components that evaluates to a single value. You can build expressions to create calculation and filter definitions. A calculation is an expression that you use to create a new value from existing values contained within a data item. A filter is an expression that you use to retrieve a specific subset of records. Lets walk though a few simple examples:

Using a Session Parameter

I’ve talked before about session parameters in Framework manager (a session parameter is a variable that IBM Cognos Framework Manager associates with a session, for example user ID and preferred language and you also create your own) in a previous post.

It doesn’t matter if you use a default session parameter or one you’ve created, it’s easy to include a session parameter in your Framework Manager Meta Model.

Here is an example.

In a Query Subject (a query subject is a set of query items that have a relationship and are used to optimize the data being received for reporting); you can click on the Calculations tab and then click Add.

Framework Manager shows the Calculation Definition dialog where you can view and select from the Available Components to create a new Calculation. The Components are separated into 3 types – Model, Functions and Parameters.

I clicked on Parameters and then expanded Session Parameters. Here FM lists all of the default parameters and any I’ve created as well. I selected current_timestamp (to add as my Expression definition (note – FM wraps the expression with the # character to indicate that it’s a MACRO that will be resolved at runtime).

During some additional experimentation I found:

  • You can add a reasonable name for your calculation
  • You may have to (or want to) nest functions within the expression statement (i.e. I’ve added the function “sq” as an example. This function wraps the returned value in single quotes). Hint: the more functions you nest, the slower the performance, so think it thorough).
  • If you’ve got the expression correct (the syntax anyway), the blue Run arrow lights up and you can test the expression and view the results the lower right hand pane of the dialog. Tips will show you errors/Results will show the runtime result of your expression.
  • Finally, you can click OK to save your calculation expression with your Query Subject.















Filtering works the same way as calculations. In my example I’m dealing with parts and inventories. If I’d like to create a query subject that perhaps lists only part numbers with a current inventory count of 5 or less, I can set a filter by clicking on the Filter tab and then Add (just like we just did for the calculation).

This time I can select the column InventoryCount from the Model tab and add it as my Expression definition. From there I can grab the “less than or equal to” operator (you can type it directly or select it from the Function list).














Filter works the same as Calculation as far as syntax and tips (but it does not give you a chance to preview your result or the effect of your filter).

Click OK to save your filter.


Finally, my inventory report is based upon the SQL table named PartInventory which only provides a part number and an inventory count. I’d like to add part descriptions (which are in a table named simply “Part”) to my report so I click on the SQL tab and create a simple join query (joining the tables using PartNo):












To make sure everything looks right, I can click on the tab named Test and then click Test Sample.

You can see that you have a part name for each part number, the session parameter Time Stamp is displayed for each record and only those parts in the database where the inventory count is 5 or less:













By the way, back on the SQL tab, you can:

  • Clear everything (and start over)
  • Enter or Modify SQL directly (remember to click the Validate button to test your code)
  • Insert an additional data source into your Query subject to include data from another source, perhaps an entirely different SQL database.
  • Insert a Macro, For example, you can add inline macro functions to your SQL query.

Here is an example:


Notice the # character to indicate the code within is a function to be resolved within the SQL query.

This code uses a parameter map (I’ve blogged about PM’s in the past) to convert a session parameter (set to a particular vehicle model year) to the name of a particular SQL table column (and include that column of information in my query subject result). So in other words, the database table column included in the query result will be decided at run time.













And our result:








You can see that these are simple but thought-provoking examples of the power of IBM Cognos Framework Manager.

Framework Manager is a metadata modeling tool that drives query generation for Cognos BI reporting. Every reporting project should begin with a solid meta model to ensure success. More to come…

Framework Manager – Creating a Parameter Map

A session parameter is a variable that IBM Cognos Framework Manager associates with a particular session. Examples include (current user name, current active language, current date and time, and others). Parameter maps are a method for substituting different values with different keys.

A parameter map can be thought of as simple data “look-up table”.

Each parameter map has two columns:

  • a key column and
  • a value column (holding the value that the key represents).

In Cognos TM1, Lookup (or mapping) cubes (and dimensions) are common (and I’ve blogged on them before).

So let’s create a simple Framework Manager Parameter Map:

Well, to construct your map, you can:

  • enter the keys and values (for your map) manually,
  • import them from an external file, or
  • base them on query items in your Meta model

– it all depends upon the size and/or complexity of the parameter map you need to build.

Some helpful hints:

  • All parameter map keys must be unique so that the Framework Manager can reliably obtain the correct value!
  • The value of one parameter can be the value of another parameter, so you must enclose the entire value in number signs (#).
  • There is a limit of five levels when nesting parameters in this way.

So let’s look at an example exercise. I chose to use the “source file” method to create my map.

In Framework Manager, right-click in the Parameter Maps icon, then select Create and Parameter Map:












From there, you can enter a name for your parameter map.

Since I am converting (or mapping) (Corvette) part numbers into part descriptions, I’m naming my new parameter map:

“Keen Corvette Restoration Parts”,

and then selecting the option “Manually enter the parameter keys and/or import them from a file”:





















On the Create Parameter Map Wizard dialog, I entered a default value (a value to be used if a key doesn’t have a value in your map) and then clicked on Import File…
















Navigated and selected mysource file (to use a .txt file for import, the values must be separated by tabs and the file must be saved as UTF8 or in the Unicode format. ANSI text files are not supported):















Clicked OK and Framework Manager created my parameter map. It looks good, (it does!) so I clicked on Finish:

















And you can see my new map now existing in my project:













If you double-click on your map, the Parameter Map dialog opens again where you can clear your map, import a new source file (to over lay or add to your map), add new specific keys, export your map or edit it directly.

Next time I will illustrate how to use the new parameter map!

Risks Associated with Niche Big Data Vendors

One of the not so nice parts of the conference is seeing companies that have technology that have been superseded by the new releases of Hadoop, primarily Hive. One company, boasted about the fact it did not have to do full scans and could return SQL queries in seconds on large datasets. The look on the booth attendant’s face was initially startled, when I asked how his company’s technology was different then that that was included in Risks Associated with Niche Big Data VendorsHive/Stinger’s ORCFile. Bottom-line, is that he did not have a good answer other than saying we are significantly faster than “legacy” versions of Hive.

Hadoop market is in the Cambrian Explosion stage where new vendors and solutions are coming to the market at an incredible pace. However, we do know that most will either be acquired or bankrupt within the next few years which adds risk for companies needing to invest in these niche solutions.  Understand the Apache Hadoop roadmap, understand the unique capabilities of the niche provider, and understand the risks with selecting the particular niche vendor before you buy.

Creating Business Logic – in Cognos Framework Manager

One of the goals of the Cognos Framework Manager modeler is to build a model that makes report authoring easier. To accomplish that, you “build in” business logic” (a loosely defined term here) into your meta model.

This business logic can be simple (like simply renaming or hiding a database table column) or quite complex (like modifying SQL commands to return only specific data from a datasource).

Some of the basics you can add to your model are:

  • Stipulating attributes
  • Renaming or hiding database columns
  • Adding prompts
  • Applying filters
  • Creating calculations
  • Adding formatting to data items
  • Using folders and namespaces for grouping information
  • Using shortcuts to include the same information in different places

Let’s explore a few of these.

Stipulating Attributes

A simple example is changing the attribute type of an imported database column. This is done for various reasons, but manly to dictate how Report Studio deals with the data in a report (for example, Report Studio may attempt to sum numeric fields which may, depending upon the database column, be a meaningless number).

An example (from my demo project) would be the ProductID field in the Products table (imported from SQL Server).It is imported as a FACT field (since it is defined as a numeric in the database table). To change this field’s attribute, I:

Select the database field (by clicking on it) and in the Framework Managers “Properties Pane” (hint, if you can’t find the properties pane, go to “view” on the FM menu and click Properties to make it visible), I locate the property “Usage”. There, I can change the attribute:
















Renaming or hiding database columns

This is even easier. If you’d like to rename a database column (to something more “user friendly”) or perhaps hide it (for example, we don’t want to display Salary information to the report authors), you can just:

  • right-click on the database column in the Framework Manager Explorer column and select (from the popup menu) “rename” or
  • back in the properties pane, change the “Is Hidden” property to TRUE (to hide the column)

Applying filters

Filters are neat. For example, my demo includes a database table named Employee. Right now, the meta model includes all records (all employees) in the table. That would include both active and inactive employees. I want to make sure that reports only use active employees, so I can alter the FM query (to apply a query filter) to accomplish this:

  1. Double-click on the table name Employee (in the FM explorer pane).
  2. FM shows me the “Query Subject Definition” dialog.
  3. Edit the SQL (hint: use the Validate button).
  4. Click OK.












And so on. You can see that Cognos Framework Manager makes it easy to build out your meta model with your business logic but to be sure, the real work here is to identify what logic to include and obtain consensus from your business stakeholders on the correctness of the logic.

Until next time…