Perficient Enterprise Information Solutions Blog

Blog Categories


Posts Tagged ‘rules’

Automation of Cognos TM1 Data Loading


In one of my earlier posts I recommended an approach for loading larger amounts of data into TM1 applications, and provided a high-level explanation of this approach, which I call the “File Route” solution.

Some More Thoughts                 

I like this load-strategy because, in following best practice guidelines, it leverages TM1’s proven ETL tool, TurboIntegrator to load data. TurboIntegrator (or TI) is a fast and efficient method for loading rows and rows of data and can be programmed to handle most presumed exception conditions. The following are a few more thoughts on implementing such a solution.

Generally “bulk data load” does not refer to the process of receiving formatted data files from source systems or directly querying a ledger for data, what it does refer to is the need for a simple, robust process for business users to enter lager amounts of data into TM1 without having to type in the data, record by record. For example, when making top-side adjustments to a forecast it may be considered reasonable to expect a user to key-in or edit certain cells in a cube (using a formatted cube view or WebSheet, for example) but if a user must provide actuals for 100’s of accounts and for dozens of cost centers, than data entry isn’t going to work. Here is where a “bulk load” solution comes in handy.

Think of a bulk load process as an assembly line that continuously moves from point A (the user’s desktop) to point B (a location with TM1). Users “drop” chunks of data onto a “receiving point” or “drop box” where an automated, intelligent process receives the “package”, “logs it as a request”, (does some verification) and then indicates to an individual loader process that there is work to be done. Once a loader process has processed the request (loads the data), another process validates the results of the processing request and then notifies the requestor (the user). (During each of these “steps” the process keeps a status object (perhaps a cube) updated with the latest processing information which can be viewed by the user).


More architectural thoughts to keep the following in mind when considering your file route solution:

  • Users will need to adopt a somewhat standard format for submitting their data. “Somewhat” means that there can be some flexibility in the format, but where formats differ, “logical rules” need to be strictly enforced to allow program logic to identify and understand data (as the TM1 Architect, you’ll need a good understanding of all data requirements for this!)
  • You will need to determine the cadence that is used by the solution to check for new data. How often? Every 30 minutes? Every hour? On demand? This will require some experimentation of average file sizes, data “overlaps”, security, etc.
  • Security requirements. For example, how will you restrict users from submitting data on behalf of others? Will you need to restrict WHEN or how often a user can submit data for load? How about maximum file sizes? Etc.
  • User feedback. It is absolutely imperative that you provide a near real time method for monitoring and auditing the status of load requests to each user.
  • Expirations. You will need to consider the idea of “expiring” requests. . Based upon a variety of factors, will a load request ever become “stale” and not need to be loaded?
  • You solution must always load to absorption or input cubes, never to any key application calculation or business process cube.
  • After processing data files, always move them from the drop box folder to an offline location for archival and later (if required) audit.
  • Build your solution as “generic” as possible! Do not “hard code” or “custom link” anything. The process should easily manage multiple load requests and, in a perfect world, have the ability to identify the format of the file (based upon its destination and perhaps other business logic), apply appropriate business logic and then load the data.



There is more to think about when designing and implementing a usable and scalable bulk load solution of course – but this approach works and works well. I’ve used it (or something similar to it) throughout my career with great success. Don’t believe me? Need some specifics? Give me a call.


Cognos TM1 User Experience

The Quandary

I’ve literally written a ton of VBA code over the years to “enhance” my end user’s experience. For example, I’ve often slapped a button on a worksheet or toolbar and used the E_PICK to pop-up a dimension subset editor allowing a selection of particular element and then using that selection to drive changes in the worksheet.  This works well:


Cognos TM1 gives us the E_PICK function which is a macro function, valid only in Microsoft Excel macros and VBA modules. The function calls the TM1 Subset
Editor, listing all elements in the specified dimension. The element name you select in the Subset Editor becomes the return value of the E_PICK function call.


E_PICK(Dimension, Alias, Subset, Element);


What I really want to do from time to time is allow selection and return of multiple elements –which E_PICK does not support.

For example, I have a worksheet that lists accounts down the left of the worksheet and periods across the top. My goal is to allow the user to select from the periods dimension a “period view”. This can be any combination of periods.

A typical “view” selection might include a total year, 4 quarters and 3 months:





I want to permit the user to select all 8 elements and have them returned – preferably in an array.

No such luck with the E_PICK!

A Workable Solution

But, if I change out the E_PICK function with the SUBPICK, I’m (kind of) in business.

Have a look at the SUBPICK:


This is also a macro function, valid only in Microsoft Excel macros and VBA modules. This function calls a dialog box that lists all the elements in the specified subset. The elements
you select are inserted in the active worksheet, starting at the current cell position.


SUBPICK(dimension, subset, vertical)



So I set up that same button, use the SUBPICK, and up comes that same subset editor, exposing the period dimension and this time my user makes the multiple selection and clicks OK.

Now, Cognos TM1 doesn’t understand arrays yet, so it does the next best thing. It returns the selected elements into your Microsoft Excel worksheet – which is nothing more than an “array of cells”.  You can even designate which worksheet (you make that worksheet the active sheet) and which cells (vertical – a column – or horizontal – a row).

So, in my case I spun the following simple, but useful, VBA code:

 ‘ — gotta make sure that it’s the active sheet and pointing to A1

  ‘ — call tm1 subpicker and then set the screen updating off so we can rebuild
  ‘ — the template w/o showing the user the dirty work – it also saves on performance

    sPeriodSelections = Application.Run(“SUBPICK”, sPeriodDimName, “default”, True)
    Application.Cursor = xlWait
    Application.ScreenUpdating = False
    Application.StatusBar = “Configuring Template Periods. Please Wait!” 

Then of course I’d write some clever lines of code to refresh the periods currently residing across the top of my worksheet with the newly selected ones – making sure to then recalculate my sheet to bring back my TM1 data.

A Little Bit More

Of course I soon found that my users only had a handful of “period views” that they really cared about 90% of time and now they were annoyed that they had to “reselect” the same periods each time they used my worksheet.



This function creates a dimension subset consisting of element names found in the active worksheet.

So, making use of this handy little nugget, I implemented a few more lines of code to allow the user to save their favorite period views on the TM1 server for later selection and use.

   lAns = MsgBox(“Save Your Period View To TM1?”, vbYesNo, “Save Period View”)
    If lAns = vbYes Then
       myname = InputBox(“Please Enter A Name To Save Your Period View As!”, “Save”)
       bResult = Application.Run(“SUBDEFINE”, sPeriodDimName, Trim(myname),        Worksheets(sInputWorksheetName).Range(“A1:A999”))
       If bResult = True Then MsgBox “Period View Saved to TM1!”
    End If

I know I know, so I finally ended up getting really creative and allowed the user to:

  • Share their views (save them as a public subset rather than a private subset)
  • Define a “default” period view that would load up upon opening the worksheet
  • Of course check for the existence of a previously saved view of the same name and choose to overwrite or provide an different name

And so on – you get the picture…


On a side note: I’ve reached my goal (although a bit later than I’d planned) of logging 1000+ miles this year.

Here is a pic from last night’s 7 miler:

Reality is merely an illusion, albeit a very persistent one. — Albert Einstein