The Find Command

I come from a systems background in Windows and Linux, maintaining operations and writing programs in C.  Since I bring that perspective to my TM1 work, I’m more comfortable with command line tools than some of my other teammates, so I thought I would share some tips.

As an example, let’s look at a large file of actuals.  A customer’s file is over 1GB in size and has over six million rows, but Excel can only load one million rows.  The file is also too large to be loaded into Notepad++ or other common tools.  To research an issue with this file, I probably only need a small number of records that affect a particular cost center or model number.

With a command like:

find “HOU00884500” actuals.txt >smallfile.txt

I can reduce my large file of actuals down to 414 lines in about two minutes, and then use another tool such as Excel to analyze the results (smallfile.txt).

Example

Here is a more detailed example. The Find command can be used in Windows Command Prompt or Powershell. I’ll use the command prompt window for this example.

Start the Windows Command Prompt.  One way to do this is to click on the search magnifying glass on the task bar and type in “cmd” (without the quotes) and hit <Enter>.

• The default directory will normally be a directory you have write access to, such as “C:\USERS\KPORTER” but you may need to switch to a different drive and path.
• To switch to a different drive, such as the F: drive, just enter the drive letter and : and press <Enter>.
• To go to a particular path, use the CD (change directory) command, such as “CD \SOURCEDATA\EXAMPLE”.  The target directory must already exist or you will get an error.

Running the Find

Now build the command:

find “HOU00884500” actuals.txt >smallfile.txt

In place of the filename actuals.txt, you can drag a file from a Windows Explorer window onto the command prompt window.  This will insert the full path of the file, so it may look like (bold added for emphasis):

find “HOU00884500” C:\DATADIR\actuals.txt

This command will display the results to the screen.  If you want to capture the results in a file to view them in an editor or Excel, use the > command to redirect to another file.  This file will be created in your current directory if you don’t specify otherwise:

find “HOU00884500” C:\DATADIR\actuals.txt > smallfile.txt

When searching TM1 log files, you may have a ” (quote) character that needs to be included in your actual search string.  To do this, make any ” character that is part of the search string a double quote, “”.  This example will search all the events where the Load Data process completed successfully so you can compare execution times over the timespan of the log:

find “Process “”Load Data””: finished” C:\SERVER\Logs\tm1server.log

Wrapping Up

As you can see, there is a lot of flexibility with the tool, but some uses of it can get pretty complicated.  Don’t be intimidated, though. You can get a lot of value from even just these simple uses.

Let me know in the comments if you have any questions or suggestions.  Is there a similar tool available in the Windows GUI?  I know how to search, but I haven’t seen a comparable way to capture the results.

Hopefully many of you have enjoyed my many Cognos TM1 blog posts. Now some of you may be new to TM1 or a novice or an expert at the product. Regardless of your skill level, training is always important to stay up to date with new product changes.

With that said, Perficient’s IBM Business Analytics group will be hosting comprehensive training sessions during the week of August 17th-20th in our Denver, Colorado office.  The following TM1 training course is an intermediate instructor-led course. This offering will teach modelers how to build a complete model in TM1 mainly using the Performance Modeler tool. Through a series of lectures and hands-on exercises, students will learn how to create and customize dimensions and cubes, import and link data, and build and maintain applications.

If you are interested in attending our TM1 training event in Denver, CO, please visit our Perficient Contact Page and in the comments section please add, “Cognos TM1 Training in Denver, CO on August 17-20 2015”, else you can DM me on Twitter at @XxCLEvansxX for further details.

Throughout my years as a technical consultant I have worked with many bright minds, but the majority of

QUBEdocs

the people I have worked with always hated technical documentation.  They thought this task was either something beneath them or just didn’t have the time due to the hundreds of development tasks that had higher importance.  But whatever the case, clients need documentation of the system to help with any new developers get up to speed with the current system.  Now I’ll admit it, I’m not the greatest writer but I do like to write, whether it be blogs, technical documentation, SOW and etc.  So I was impressed with QUBEdocs, as the system provides high quality documentation that is current with development changes with a little turnaround time to setup.

Recently I was able to work with one of our Perficient TM1 Solution Architects “William Masterson” to check out the great new features QUBEdocs has to offer. I found that it helps automatically document a Cognos TM1 solution, as well as provide other great features such as the ability to trace data and metadata flows, and help users understand how a TM1 model hangs together.

Since I did not install or setup the QUBEdocs application, my understanding from Bill was this was not a very difficult task.  Once the application is setup, QUBEdocs generates HTML webpages that documents your model name.  The home page will look like the image below.

Once of the great features that I loved about QUBEdocs was the features that trace data and metadata flows.  To help trace this lineage, QUBEdocs used color coding to distinguish rule flow, process flow, data source flow, metadata updated flow and linked process flow.

Here is a great example of the lineage of how QUBEdocs represents a TM1 process utilizing cube, ODBC source, TI processes and rule.

In addition, to tracing data and metadata flows, the major business benefits QUBEdocs offers is that it significantly decreases the cost of managing your TM1 implementation, adding significant value and reducing ongoing costs. By providing effortless documentation QUBEdocs uses a simple framework for documenting models ‘as you go’, updating documentation via tags in real time, resulting in time and resource savings.

Now there is one caveat with QUBEdocs.  To fully use all the features from QUBEdocs, developers will need to design and develop their TM1 application utilizing and following some small simple formatting in which QUBEdocs will be able to display.  To be honest this is a good thing as it provides a standard and it emphasizes the importance that developers should add more comments when creating rules, TI’s and etc.

For further information on QUBEdocs, check out this overview video.

IBM Cognos TM1 is well-known as an enterprise planning software platform that can transform your entire planning cycle, from target setting and budgeting to reporting, scorecarding, analysis and forecasting [1], and Perficient is well-known for delivering expertly designed TM1-based solutions. At Perficient, we are excited about solving our customers’ business needs. Every Cognos TM1 implementation begins with the question: what problem does our customer need our help to solve? Here at Perficient, we strive to make sure we truly understand customer needs and future needs to help them succeed within their business.

Now with any type of implementation there are many uncertainties for a customer and two of major uncertainties my clients have conveyed to me are the following:

• Does this solution really meet my current and future business needs
• Does my IT team understand the business requirements and can it be reflected in the solution

Now at Perficient we understand our customer’s concerns, and with that said, the Perficient’s Business Analytics team has put together unique approach to implementing IBM Cognos TM1.

Before I get into Perficient’s approach in implementing IBM Cognos TM1, I want first elaborate why customers have these two major uncertainties. Recently, I saw an image on Linkedin “Tire-Swing” and said to myself, “Somebody truly gets it!” This image alone hits the nail on the head and what the Perficient’s Business Analytics team did was to find a solution to address these concerns while implementing a IBM Cognos TM1 solution.

With any type implementation, especially with IBM Cognos TM1, there are many moving parts. The Perficient’s Business Analytics team has found a way to integrate the use of Atlassian JIRA with its IBM Cognos TM1 implementations. Now some of you may already know what Atlassian JIRA is, and you’re probably saying to yourself, “How can a issue tracker software help with my TM1 solution needs?” Well Atlassian JIRA software offers more than just an issue tracker solution and the Perficient’s Business Analytics team has utilized this tool to its full potential.

The Perficient’s Business Analytics team significantly reduces the risk of failure on IBM Cognos TM1 implementation projects and empowers the end client to “own” both the solution and the technology at the end of the very first project utilizing Atlassian JIRA. A major advantage in which we empower our customers is by offering them a dashboard to allow them to see the progress of the project and what major shower stoppers need to be addresses. Below is a sample Atlassian JIRA dashboard of a fictitious company, “Acme.”

Now before anything can be documented in Atlassian JIRA, the business requirements must have been collected by the business users. Once the business requirements have been collected by the business analyst these would than need to be created in JIRA. In addition, use cases would also need to be created in which should back tie to all the business requirements. Below is a sample “User Story Enhancement” or use case in which is created in Atlassian JIRA.

Once the use cases have been created, the next step would be for the TM1 solution architect to start putting together the IBM Cognos TM1 solution. Once the solution has been architected the TM1 solution architect and project manager would document all the components of the build or upgrade to the IBM Cognos TM1 solution within Atlassian JIRA. Now this is the most important piece because all development activities will not only be assigned to developers but also tie back to a use case. This ensures all requirements have been met and developed. Examples of a few Atlassian JIRA “Tasks” would be, “Create Cube, Create TI, Create Web Sheet or etc.”

In addition, to have creating the IBM Cognos TM1 Atlassian JIRA Tasks, users would be able to track the progress of the development and hours spent on a task. This allows the customers to know where things are at, at all times.

Besides from creating the requirements, use cases and development tasks in Atlassian JIRA users also have the ability to create dependencies in which can be assigned to other teams. In addition, future improvements can be created and assigned to the TM1 architect so that estimates can be given, as well as risks and issues.

These are the many benefits utilizing Atlassian JIRA with your next TM1 implementation and Perficient has mastered this approach. Our approach to use Atlassian JIRA with the IBM Cognos TM1 implementation or upgrade solution requires the development and business team to work side by side to achieve a successful project implementation. When clients are empowered to own the IBM Cognos TM1 solution, they also tend to achieve much greater returns on their investment. This is the primary value proposition for selecting Perficient for IBM Cognos TM1 implementation projects.

If you interested in learning more on how Perficient can help with your next IBM Cognos TM1 implementation or IBM Cognos TM1 upgrade utilizing Atlassian JIRA, please visit or Perficient Contact Page and let them know Chris Evans sent you, else you can ping me on Twitter at @XxCLEvansxX

[1] http://www-03.ibm.com/software/products/en/cognostm1

Consider the scenario where you have to convert information held in a “categorical field” into a “collection of flag fields” – found in a transactional source. For example, suppose you have a file of transactions that includes (among other fields) a customer identifier (“who”) and a product identifier (“what”).  This file of transactional data indicates purchases of various energy drinks by unique customer IDs:

Aggregation and Flag Fields

What I need is have a single record per unique customer showing whether or not that customer purchased each of the energy drinks (True or False – not sales amounts) during a period of time. More like:

Doing it in TM1

In Cognos TM1, you’d utilize TurboIntegrator to read the file – record by record – aggregating the data (by customer id) and updating each measure (energy drink) – a reasonable trivial process, but still requiring lines  of script to be written. Something (assuming you perhaps initialized the period being loaded to all False values) like:

MODELER’s Approach

In SPSS MODELER, the SetToFlag node (found in the Field Ops palette) enables you to create the flags you need and to aggregate the data (based on a grouping, or aggregate key fields) at the same time – without writing any script.

In order to have the SetToFlag node populated with the values of the categorical field, the field has to be instantiated so that MODELER knows the values for which to create flag fields. (In practice, this means that your data has to be read in a Type node prior to the SetToFlag node.) The procedure to create flag fields from the file is:

1. Add a SetToFlag node to your stream.
2. Edit the SetToFlag node to set the options for the SetToFlag operation.

Editing the SetToFlag

Once you have added your SetToFlag node, you use the Settings tab to:

1. Select the categorical field to be expanded in flags (I selected “Product”). The Available set value box is populated with its categories (when the data are instantiated).
2. Optionally, add a field name extension for the new flag field’s names, either as suffix or prefix (I left this blank).
3. Select the categories for which one wishes to create flag fields in the Create flag fields list box and move them to Create flag fields (I selected all of the products).
4. True and False value can be changed, if desired (I left it blank – no changed).
5. Optionally, aggregate records by checking the Aggregate keys check box and selecting the appropriate key field(s) – I selected “id”.

Conclusion

If you go back to the TM1 example, you now have a cube loaded that you can view by customer and period to which products each customer purchased within the period:

In SPSS Modeler, the output from the SetToFlag node is similar:

Next up – Data Sampling Patterns!

Data loaded into a TM1 or SPSS model will, in most cases, include files consisting of thousands (or hundreds of thousands) of records. It is not reasonable, given the number of fields and records in files of this size, for you to visually inspect all fields in every record (of every file) for missing or invalid values.

Data Transformations

In TM1, data is usually loaded from an EDW or GL system, so (once the initial testing phase has been completed) the probability that incoming data contains unexpected values (should be) somewhat small. However, the data will (most likely) need to be transformed into a format that the TM1 model can use, or is more optimal for the model’s use. Additionally, if data is being manually input to the model (for example a user entering a (sales) forecast value), then inspecting the data for invalid values is required.

With SPSS Modeler, data may come from sources more “sketchy” – marketing surveys, multiple files merged into one or even manually typed-in data. Obviously, the need for auditing and transforming is critical.

Types of Data Violations

Generally, (with TM1 or SPSS) three different types of data violations may be found during the loading process:

• Values do not comply (with a field’s defined storage type). For example, finding a string value in a numeric field.
• Only a set of values or a range of values are allowed for a field and the incoming value does not exist within the defined set or is greater or less than the acceptable range.
• Undefined (SPSS refers to this as $null$ and has “rules” about how it handles these values) values encountered anywhere (no data!).

Actions

When an invalid value (a value violating one of the three rules) is found, one of five possible actions can be taken:

• Nullify – in SPSS, you can convert the value to undefined ($null$) again, this has a special meaning for SPSS, for TM1, this is not usually a viable option since TM1 does not work well with NULL values.
• Coerce – you can convert the invalid to valid. What invalid and valid is usually determined by the measurement level of the field. In SPSS:
  • Flags; if not True or False, then it is False
  • Nominal, Ordinal; if invalid, it becomes the first member of the sets values
  • Continuous; value less than lower bound-> lower bound, value greater than upper bound -> upper bound
  • Undefined ($null$) -> midpoint of range

In TM1, this process is more typically a “remapping” from one value to another. For example, translating a product’s code used in one system to that products code used in another.

• Discard – delete the record – with SPSS, you can Discard the record, in TM1, you might use an ItemReject function;
• Warn – invalids are reported in a message window in SPSS, minor errors may be written to the message log in TM1.
• Abort – In SPSS, the first invalid value encountered can result in an error and the stream execution aborted, with TM1, processes can be terminated with the ProcessError function.

Handling data Violations

In SPSS Modeler, the Type node enables data checking and transforming. (Checking can also be done using the Types tab in a data source node). To check and transform, you need to specify 1) what the valid values are and 2) the action to be taken:

1. Select the Types tab.
2. Select the field to check and click in the corresponding cell in the Values column.
3. Select Specify values and labels and enter the lower and upper bound.
4. Select the action to take when an invalid value is encountered.

In TM1, checking for and handling (transforming) of invalid values is a little more work.

Using TurboIntegrator processes is the best approach for loading and transforming data in a TM1 model and scripting logic (using predefined functions) is required to evaluate and take action on records being loaded into TM1:

• Select the Data tab.
• Type the script to use the appropriate TM1 functions. Some examples include:
  • Checking for invalid data types – Value_Is_String (does the cell contain a string value?)
  • Checking data against a set of values – DIMIX (does the value exist in a dimension?)
  • Checking data to ensure it is within a range – If (CapitalValue < MaxAllowed & CapitalValue > MinAllowed); (is the value within the allowed range?)
  • Checking for missing data (empty fields) – IF (value) @= ‘’ (is the value empty?)
• Save and Run the process.

Conclusion

In both tools you have similar objectives – load data, ensure that the data is “usable” and if it is not, perform an appropriate action (rather than break the model!). SPSS Modeler allows you to do some checking and transforming by selecting values in dialogs, while Cognos TM1 requires you to use TI scripting to accomplish these basic operations. Both can be straight forward or complex and both can be automated and reused on future datasets.

SPSS Data Audit Node

Once data is imported into SPSS Modeler, the next step is to explore the data and to become “thoroughly acquainted” with its characteristics. Most (if not all) data will contain problems or errors such as missing information and/or invalid values. Before any real work can be done using this data you must assess its quality (higher quality = more accurate the predictions).

Addressing issues of data quality

Fortunately, SPSS Modeler makes it (almost too) easy! Modeler provides us several nodes that can be used for our integrity investigation. Here are a couple of things even a TM1 guy can do.

Auditing the data

After importing the data, do a preview to make sure the import worked and things “look okay”.

In my previous blog I talked about a college using predictive analytics to predict which students might or might not graduate on time, based upon their involvement in athletics or other activities.

From the Variable File Source node, it was easy to have a quick look at the imported file and verify that the import worked.

Another useful option is run a table. This will show if field values make sense (for example, if a field like age contains numeric values and no string values). The Table node is cool – after dropping it into my stream and connecting my source node to it, I can open it up and click run (to see all of my data nicely fit into a “database like” table) or I can do some filtering using the real-time “expression builder”.

The expression builder lets me see all of the fields in my file (along with their level of measurement (shown as Type) and their Storage (integer, real, string). It also gives me the ability to select from SPSS predefined functions and logical operators to create a query expression to run on my data. Here I wanted to highlight all students in the file that graduated “on time”:

You can see the possibilities that the Table node provides – but of course it is not practical to visually inspect thousands of records. A better alternative is the Data Audit node.

The Data Audit node is used to study the characteristics of each field. For continuous fields, minimum and maximum values are displayed. This makes it easy to detect out of range values.

Our old pal measurement level

Remember, measurement level (a fields “use” or “purpose”)? Well the data audit node reports different statistics and graphs, depending on the measurement level of the fields in your data.

For categorical fields, the data audit node reports the number of unique values (the number of categories).

For continuous fields, minimum, maximum mean, standard deviation (indicating the spread in the distribution), and skewness (a measure of the asymmetry of a distribution; if a distribution is symmetric it has a skewness value of 0) are reported.

For typeless fields, no statistics are produced.

“Distribution” or “Histogram”?

The data audit node also produces different graphs for each field (except for typeless fields, no graphs are produced for them) in your file (again based upon the field’s level of measurement).

For a categorical field (like “gender”) the Data Audit Node will display a distribution graph and for a continuous field (for example “household income”) it will display a histogram graph.

So back to my college’s example, I added an audit node to my stream and took a look at the results.

First, I excluded the “ID” field (it is just a unique student identification number and has no real meaning for the audit node). Most of the fields in my example (gender, income category, athlete, activities and graduate on time) are qualified as “Categorical” so the audit node generated distribution graphs, but the field “household income” is a “Continuous” field, so a histogram was created for it (along with the meaningful statistics like Min, Max, Mean, etc.).

Another awesome feature – if you click on the generated graphs, SPSS will give you a close up of the graph along with totals, values and labels.

Conclusion

I’ve talked before about the importance of understanding field measure levels. The fact that the audit data node generates statistics and chart types are derived from the measurement level is another illustration of how modeler uses the approach that measurement level determines the output.

In TM1, you may be used to “integer or string”, in SPSS Modeler, data gets much more interesting. In fact, you will need to be familiar with a concept known as “Field Measurement Level” and the practice of “Data Instantiation”.

In TM1, data is transformed by aggregation, multiplication or division, concatenation or translation, and so on, all based on the “type” of the data (meaning the way it is stored), with SPSS, the storage of a field is one thing, but the use of the field (in data preparation and in modeling) is another. For example if you take (numeric) data fields such as “age” and “zip code”, I am sure that you will agree that age has “meaning” and a statistic like mean age makes sense while the field zip code is just a code to represent a geographical area so mean doesn’t make sense for this field.

So, considering the intended use of a field, one needs the concept of measurement level. In SPSS, the results absolutely depend on correctly setting a field’s measurement level.

Measurement Levels in Modeler

SPSS Modeler defines 5 varieties of measurement levels. They are:

• Flag,
• Nominal,
• Ordinal,
• Continuous and
• Typeless

Flag

This would describe a field with only 2 categories – for example male/female.

Nominal

A nominal field would be a field with more than 2 categories and the categories cannot be ranked. A simple example might be “region”.

Ordinal

An Ordinal field will contain more than 2 categories but the categories represent ordered information perhaps an “income category” (low, medium or high).

Continuous

This measurement level is used to describe simple numeric values (integer or real) such as “age” or a “years of employment”.

Typeless

Finally, for everything else, “Typeless” is just that – for fields that do not conform to any other types –like a customer ID or account number.

Instantiation

Along with the idea of setting measurement levels for all fields in a data file, comes the notion of Instantiation.

In SPSS Modeler, the process of specifying information such as measurement level (and appropriate values) for a field is called instantiation.

Data consumed by SPSS Modeler qualifies all fields as 3 kinds:

• Un-instantiated
• Partially Instantiated
• Fully Instantiated

Fields with totally unknown measurement level are considered un-instantiated. Fields are referred to as partially instantiated if there is some information about how fields are stored (string or numeric or if the fields are Categorical or Continuous), but we do not have all the information. When all the details about a field are known, including the measurement level and values, it is considered fully instantiated (and Flag, Nominal Ordinal, or Continuous is displayed with the field by SPSS).

It’s a Setup

Just as TM1’s TurboIntegrator “guesses” what field (storage) type and use (contents to TM1 developers) based upon a specified fields value (of course you can override these guesses), SPSS data source nodes will initially assign a measurement level to each field in the data source file for you- based upon their storage value (again, these can be overridden). Integer, real and date fields will be assigned a measurement level of Continuous, while strings area assigned a measurement level of Categorical.

This is the easiest method for defining measurement levels – allowing Modeler to “autotype” by passing data through the source node and then manually reviewing and editing any incorrect measurement levels, resulting a fully Instantiated data file.

Having recently completed the course “IBM SPSS Modeler & Data Mining” offered by Global Knowledge, I was looking to find more opportunities to do some modeling with SPSS Modeler. So, when I read in the news recently, about college recruiters using predictive techniques to determine the probability of a particular recruit graduating on time, I thought it would be interesting to explore that idea.

For Example

My college wants to determine if a recruit will graduate on time or not. The institution can draw a sample from its historic data and using this sample, possibly predict if a particular recruit would graduate on time. The sample below gives us an idea such a historical dataset. Typically, a dataset will include a field that indicates the behavior, here: has the student graduated on time? Yes or no.

An Idea – and some cross tabulation!

The college recruiters have a hunch that there is a difference between students who are athletes and students who are not and if a student participates in collegiate activities or not. Based on this hunch, they investigate to see if there might be any differences in graduating on time statistics – by cross tabulating on “athlete”:

In IBM SPSS Modeler, it is very simple to cross tabulate data using the Matrix node. You can simply drop it into your stream, connect it to your source data and set some parameters. For example, I set “Rows” to the field in my file “graduate on time” and “Columns” to “athlete”. (I also went to the “Appearance” tab and clicked-on “Counts” and “Percentage of column” for my “Cross-tabulation cell contents”.

After clicking “Run”, the output is ready for review:

Another Analysis Tool

SPSS Modeler also provides the Distribution Node which lets you show the occurrence of symbolic (non-numeric) values, in this case, “graduated on time” or “athlete”, in our dataset. A typical use of the Distribution node can be to show imbalances in the data (that can be rectified by using a Balance node before creating a model). What I did was use the node to plot “athlete” overlaid with “graduate on time” for an interesting perspective:

Back to the Analysis

Looking at my cross-tabulation output, it appears that 93 % of the non-athlete students did not graduate on time, while for the students who were athletes; only 13 % did not graduate on time. The question now is -can this difference be attributed to chance (because just a sample was drawn) or, does the difference in the sample reflect a true difference in the population of all students?

The Chi-Square test is a statistical test is used to answer this question. This test gives the probability that the difference between athletes and non-athletes can be attributed to chance.

The CHAID Node

CHAID, or Chi-squared Automatic Interaction Detection, is a classification method for building decision trees by using chi-square statistics to identify optimal splits. Again, SPSS Modeler offers the ‘CHAID Node” that can be dropped into a stream and configured. In my exercise, I set my (CHAID) target to “graduate on time” and my predictors to “activities” and “athlete”. My results are presented in the viewer which shows a “tree” to present the data.  The initial node shows the breakdown of graduate on-time vs. not on-time and then modeler broke out the next level as students who did not participate in activities and those who did.

The exercise found the probability (P-Value) to be 0, so the probability is 0 that the difference between students involved in activities vs. those who are not can be attributed to chance.  In other words: there are differences between participating in activities and graduating on-time!

Looking at these results, I concluded that students who do not participate in activities during their college career have a much higher chance of NOT graduating on time (96 %) – vs. those that do participate in activities (3 %).

The next step might be to zoom in on these students that do not participate in groups. Modeler broke down the “tree” into a third level:

Here, modeler tells me that those students that do not participate in activities are both athletes and non-athletes. The non-athletes who do not participate in activities have a slightly better “on time” rate then do the athletes who do not participate in activities.

Conclusion

Of course there is more to a legitimate data mining project, but it clear that IBM SPSS is a handy tool that “fits” for novice to expert level data scientists. More exploration to come!

Serviceability

What is application serviceability?

 “Serviceability (also known as supportability,) is one of the -ilities or aspects (from IBM’s RAS(U) (Reliability, Availability, Serviceability, and Usability)). It refers to the ability of application support personnel to install, configure, and monitor an application, identify exceptions or faults, debug or isolate faults to root cause analysis, and provide maintenance in pursuit of keeping an application current and/or solving a problem and restoring the product into service. Incorporating serviceability facilitating features typically results in more efficient product maintenance and reduces operational costs and maintains business continuity.”

Serviceability Levels

An application serviceability audit should be conducted and have the objective of classifying all support procedures within the application as having either a low, medium or high serviceability level.

Characteristics of Low and medium serviceability procedures are:

• Require manual intervention to initiate
• Require manual intervention during processing
• Require manual intervention to verify (complete)
• Are overly complex
• Are not repeatable without significant manual intervention
• Are “tightly coupled” to other areas or procedures within the application or other applications
• Require specific skillsets or skill levels
• Are not documented

Characteristics of High serviceability procedures are:

• Are fully automated
• Require zero or limited manual intervention
• Are encapsulated within a distinct area of the application
• Are easily repeated if required
• Are documented

Application Procedure Areas

All application serviceability procedures will fall into one of the following areas:

• Data Initialization and Manipulation
• Administrative Maintenance
• Setting Assumptions
• Validation

Data Initialization and Manipulation

Data initialization and Manipulation refers to procedures that clear data from, transform or update data within or load data into an application. All applications absorb data. Data absorption is categorized as one of the following:

• Generic (i.e. the regular refresh of a current period of activity)
• Upon-request (i.e. a one-time request to address a specific need)

Generic Absorption

Because generic Absorption procedures are “expected” – meaning that they occur at predetermined intervals and have expected parameters – all generic absorption procedures should be able to be automated, scheduled and documented. These procedures can then be classified as highly serviceability.

Upon-Request

Upon-Request procedures are usually un-documented and not scheduled and therefore have a low serviceability level; however these requests will be on a case-by-case basis and will usually be performed by a systems administrator, as needed. These requests are understandable outside the consideration of an application serviceability audit.

A simple example of a data initialization and manipulation procedures to be reviewed during a typical audit might be a procedure to maintain application Meta data:

Dimension Metadata File Data Load

1. Get an updated metadata file for the application dimensions from enterprise source system.
2. Verify that the files are in the proper format with the correct name (Dim Internal Order.csv).
3. Place file in the .\MetaData folder on the server.
4. Run the appropriate TurboIntergrator process to update the dimension.

The audit team should review this procedure and provide feedback:

Serviceability Level:

Recommendations:

The generation and loading of external data should be an automated and scheduled process to avoid the manual dependency to initiate, and process and to avoid administrative errors.  An automated process will eliminate errors in file formatting, naming and placement.

Administrative Maintenance

Administrative Maintenance procedures include the management of governance mechanisms within an application to keep the application synchronized with appropriate business processes. These procedures should:

• Be straight-forward and easily understandable
• Well documented
• Easily reversible
• Have minimal impact to system availability during the procedure

The following might be an example of a generic administrative maintenance procedure that would be audited:

Update Input Cubes with Actuals and Open the Forecast

Actual Sales Activity must be pushed to the Input cubes so users can being forecasting. The administrator might:

1. Set run parameters such as last actuals year and month and then run a TurboIntegrator process. Once the process completes, the administrator will inform the users that they can now update forecasts using provided.

The audit team would provide feedback:

Serviceability Level:

Recommendations:

This is a straight forward procedure however it is recommended that a reversal or “back out” procedure be defined. In addition system impact should be calculated for the procedure for current and future system states. Consider automating the update process so that users are either altered with an email or perhaps a visual cue that the forecast is now “input enabled”.

Setting Assumptions

Setting Assumptions refers to setting parameters that control how the system and/or application users should interpret information within the system. An example is the following:

Updating the “Last Actuals” Period

In this simple example, the applications administrator follows a 3 step process:

1. Open file Update Last Actuals Period on the TM1 server (Applications -> Admin).
2. Verify that the dropdown has the correct new Last Actuals Period.
3. Click button Run to update the Last Actuals Period process.

A serviceability audit might respond with the following feedback:

Serviceability Level:

Recommendations:

Consider automating this step as part of the actuals load process.

Validation

Validation procedures are tasks or procedures that verify the authenticity or accuracy of the information within an application or that preparedness of parameters, structures or assumptions are at a sufficient state to proceed.

File Verifications is a typical example. When data files are being manually created and loaded, file formats, names and placements are at risk of error. Meta data files, activity (actuals) files, etc. are all at risk. At serviceability audit would respond with the following feedback:

Serviceability Level:

Recommendations:

File verification should not be required if the process of generating and loading the files are automated. The recommendation is to automate all file processing procedures to eliminate the need for file verifications.

Conclusions

Even an application that is a soundly-constructed may have low serviceability. Some typical actionable items identified by a serviceability audit may be:

• Reduce the level of administrative intervention. Most of the application service requirements should be “lights out”, automated processing.
• Automate all data processes (i.e. file creations, placement, loads and updates to TM1).
• All automated processes should include failure check points that automatically alert the system administrators of intervention needs.
• Create a sequence and dependency diagram. Most of the procedures reviewed during the audit are interconnected (Actuals loading => Actuals (cube to cube) pushes => Forecast initiation => variance reporting, etc.) and may be able to be linked into single scheduled processes. In addition, this document will clarify the general operations of the system as it relates to business processes.
• Create an administrative dashboard showing all application statuses and parameters allowing an administrator to view and adjust each through a single point of reference. This makes it easy to quickly determine the overall health of the application or where intervention may be required before it affects the availability of the application. The use of graphical status indicators is recommended.
• Modify reports to be “As Of” reports showing current states -rather than dependent upon a data push.
• Develop and maintain sufficient application user guides and run books and store them in the TM1 applications folder for easy access.
• Develop an application performance monitoring routine.

If you’ve ever attempted to perform a performance test on a TM1 application, you know that there is not really an effective way to manually create sufficient load on a model. Getting “real users” to execute application operations – over and over again – is nearly impossible. Thankfully, an automated testing product can solve for this.

Automated testing refers to creating and running test “scripts” mechanically, concurrently, for as long as you need to.  There are many tools today that can be used for automating your performance testing. One of the best is HP LoadRunner.

What is LoadRunner?

HP LoadRunner is a performance and test automation product from Hewlett-Packard that can be used for examining system behavior and performance, while generating actual load on your TM1 application.

HP’s description of the product is:

“HP LoadRunner is the industry-standard performance testing product for predicting system behavior and performance. Using limited hardware resources, LoadRunner emulates hundreds or thousands of concurrent users to put the application through the rigors of real-life user loads.”

LoadRunner is made up of 3 main components which are:

• Virtual User Generator or “VUGen” (where LoadRunner creates virtual users for your application using “scripts” that you create)
• The Controller (the “user script runner”)
• Analysis Engine (shows the detailed results)

LoadRunner Objectives

The LoadRunner website goes on to outline typical performance testing objectives as:

• Emulate conditions of controlled load and maximum load on the application environment
• Measure application performance under load (response time, memory, throughput, etc.)
• Check where performance delays occur: network or client delays, CPU performance, I/O delays, database locking, or other issues at the server
• Monitor the network and server resources under load

Plan

A clearly defined routine (a plan) will make sure that your performance test is effective. Just because you have the ability to automate activities doesn’t guarantee that those activities are meaningful, based upon your objectives. It is advisable to clearly define and document even the most obvious information. A “best practice” recommendation is to develop a “performance testing questionnaire template” that can be filled out before the testing begins. The questionnaire will insure that all of the “ingredients” required are available at the appropriate time during the testing routine.

The following are examples of detail provided in a typical questionnaire:

Generalizations

• Define “the team” – Identify application owners and their contact information
• Are there any SLAs in place?
• What are the start and (expected) end dates for the testing?
• Name the environment to perform the test (QA? Production?) – including platform, servers, network, database, Web services, etc.
• What about network specifics – is the application accessible to outside users via the internet? Is there a specific bandwidth allocated to the application? Dedicated or burstable? Is the application hosted internally or externally? Is any load balancing occurring?
• Describe the details of all components of the infrastructure required to support this application (IP Address, Server Name, Software/version, OS (version/SP), CPU, Memory, Disk Sizes, etc.)
• From a client perspective, how does a user access the application? A web browser or other? (Citrix or thick client, etc.) If a browser is used define the supported browser – JAVA, ActiveX?
• Identify the conceptual goals of the performance tests – for example “to support expected user concurrency with acceptable response times.” And then provide quantitative goals – such as “simulate 75 concurrent users due to the nature of the business process which has the worldwide Affiliates (largest group of users) performing updates and reporting in a period of less than 24 hours”.
• Test requirements such as response time, load, memory, CPU utilization, etc. and provide any exiting performance metrics

Application, User and Use-Case Overviews

• What is the application to be tested? What is the purpose of the application (an overview)?(It is recommended that a high level architectural diagram be provided)
• Name the critical vs. non-critical transactions within the application under test
• User behavior, user locations, and application timeframe usability regarding the application under test
• List the business use cases that the application solves for and a brief description of each, then define the business process with in your application that will be included in the scope of the performance test. For example ‘Sales Planning contributors enter, review and adjust a current sales plan.”
• Define the categories of users for your application; identify which business processes are used by these users and state the percentage of the overall system activity this category of user represents.
• Estimate the average and peak number of simultaneous users for each business process (as seen in a normal day of business). Indicate which business processes may experience heavy throughput or are considered mission critical.
• For each business process (selected to be included in the performance test) document the specific user steps required to complete and determine whether the step requires input data.
• For each business process selected to be included in the performance test document number of concurrent users and percentage of total concurrent users that will be assigned to each process.
• Finally, for each remote location to be simulated, document number of concurrent users and percentage of total concurrent users that will be assigned to each process for tat site.

 Preparation

Once the questionnaire is completed, the performance test team will use the information to create virtual or “Vusers” scripts within LoadRunner. Vusers emulate human users interacting with your application. A Vuser script contains the actions that each Vuser performs during scenario execution.

Test

Once the Vuser scripts are constructed, you can emulate load by instructing multiple Vusers to perform tasks simultaneously (load level is set by increasing or decreasing the number of Vusers that perform tasks at the same time).

Presentation

The LoadRunner analysis engine helps you “slice and dice” generated test data in many ways to determine which transactions passed or failed (your test objectives defined in the questionnaire), as well as some potential causes of failure. You can then generate custom reports to present to stakeholders.

Conclusion

Any enterprise level application will be performance tested before production delivery and periodically during its life. Typically you’ll find performance testing is conducted prior to delivery of the application but is not executed thereafter; is recommended that a schedule be established for re-execution of the performance tests, especially for applications that are evolving (new features being added) or experiencing a widening user base.

If you are excited as I am and heard the news IBM Cognos TM1 10.2 is available, you have to read on.  Now with any major upgrade your probably skeptical, I mean when I heard the iPhone 5s was coming I was expecting the world, but all I got was fingerprint recognition that I have had on my IBM laptop before Lenovo owned them.  But don’t you fear I’m hearing to tell you the hype is worth the wait.  IBM Cognos TM1 10.2 has got all the bells and whistles.  Are there something’s that have been missed sure, but there are a lot of new things that your customer will love.  So lets read on as I discuss a few new features within the IBM Cognos TM1 10.2.

So the first thing we hear from our customers, is TM1 mobile?  Lets be honest, we are all mobile, and mobile is what our customers want!  IBM Cognos TM1 10.2.  has a mobile app that connects to an IBM Cognos TM1 Server and coexists with other desktop and web programs that connects to the same server. You can use the Mobile app to do the following tasks:

• Access all planning applications from Cognos TM1 Applications.
• View submission status for all approval nodes.
• Interact with cube views in a grid or chart format.
• Change values on the iPad and submit back to the plan where the data is updated on the Cognos TM1 server.[1]

One great feature I love is the “Conversion tool for Microsoft Excel .xls files.”  I was recently at client where we faced a few issues due to the use of 2007 and with this latest upgrade IBM Cognos TM1 10.2, you can convert Microsoft Excel 2007 or higher worksheets into OpenXML format.

Another awesome upgrade to IBM Cognos TM1 10.2 is the new features to Performance modeler.  Now some may be happy with this, while some of the older coders who loved the old way to create cubes, dimensions and etc the manually way may not.  But IBM has now added a few new things with Performance modeler such as Cube calculation, Feeder Optimization, Data flow diagrams, Data flow diagrams, TurboIntegrator processes triggered from workflow actions, and more!

Another big update to IBM Cognos TM1 10.2 is the Cognos Insight features.  I would recommend going through the full list of checking out all the cool new features!  New Cognos Insight features in Cognos TM1 version 10.2

Now there are way more new features that I have skipped so I recommend you reading more by checking out IBM’s Cognos TM1 10.2.0 new features for more details.  Well hopefully this post has inspired you to take some tome to look at the new IBM Cognos TM1 10.2 features!  I plan on installing it over the weekend and checking them out for myself.

Stay tuned for more fun and exciting blog posts from myself and be sure to check out the other great blog posts found on https://blogs.perficient.com/