Data & Intelligence

Baselines for Performance Testing

baseline

In my last post I introduced the concept of performance and performance testing. Without establishing baselines, performance testing is a waste of time. So what is a baseline and why is it important to performance testing?

Baselines

Measuring performance requires an initial state (how is the application performing at this time with these parameters?) and a next state (how is the application performing at this time with these parameters after this event took place?).

A base line is defined as the “initial state of particular application event”. The marking of significant states (within a series of several changes) and the identification of significant states (within a revision history) is the essence of performance testing.

Sample Baselines

The following are a few examples of valid baselines:

  • The total time in seconds to load 1 (average) year of sales activity into a sales cube
  • The total time in seconds to open a particular view on a particular cube
  • The total time in seconds to refresh a TM1Web report
  • The total time in seconds for the TM1 Server to start

Or:

  • The total memory consumed during a load of 1 (average) year of sales activity into a sales cube
  • The total memory consumed during an open of a particular view on a particular cube
  • The memory consumed during a refresh a TM1Web report
  • The memory consumed during a during a TM1 Server to start-up

The following are critical in establishing valid baselines:

  • Control – baselines must be established in a consistent environment
  • Repeatable – the process of establishing a baseline must be repeatable (and optimally, be automated)
  • Reflect an “Average” – a baseline must be based upon results captured over time at regular intervals
  • Scale – baselines should call out variances that can affect their interpretation, such as changes in the number or type of users, volumes and the patterns of data, security, etc.
  • Common Sense – does the baseline “make sense”? Are we measuring a reasonable event and in a reasonable way?
  • Comparable – Can we compare this baseline to an external baseline to establish perspective? (for example: ”n seconds to refresh a P&L report in TM1 compares to  n minutes to refresh a similar P&L report in some other system.
Data Intelligence - The Future of Big Data
The Future of Big Data

With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.

Get the Guide

What Events to Baseline

Application events that are baseline candidates fall into 1 of the following categories:

  • Application User Focused Events
  • Administrative User Focused Events
  • System Maintenance Focused Events
  • System Infrastructure Focused Events

Application User Focused Events

These are events performed by application business users. Examples would include but are not limited to:

  • Opening and refreshing reports
  • Logging into the application
  • Performing an update of business data
  • Opening cube views
  • Modifying cube views in TM1 Cube Viewer
  • Creating snapshots, slices or active form reports

Administrator User Focused Events

These are events performed by a TM1 application administrative user (not a TM1 Admin). Examples would include but are not limited to:

  • Running specific Chores and or Processes
  • Logging into the application
  • Publishing reports to the TM1 applications folder
  • Modifying security (if appropriate)
  • Exporting and Importing data (to and from files)

System Maintenance Focused Events

These are events performed by a TM1 application administrative user (not a TM1 Admin) or are automated.. Examples would include but are not limited to:

  • Running specific Chores and or Processes
  • Updating dimensions and dimension hierarchies
  • Set application assumption values and settings
  • Loading or clearing data
  • Performing the processes to save data to disk (from memory)
  • Performing Security Refreshes

System Infrastructure Focused Events

These are events are more narrowly focused on the TM1 software itself:

  • Stopping, Starting and Restarting the TM1 server, Excel server and TM1 Admin Server
  • Stopping and restarting web services

 

Next in my performance testing series I will talk about “profiling”. See you then…

 

 

 

About the Author

Mr. Miller is an IBM certified and accomplished Senior Project Leader and Application/System Architect-Developer with over 30 years of extensive applications and system design and development experience. His current role is National FPM Practice Leader. His experience includes BI, Web architecture & design, systems analysis, GUI design and testing, Database modeling and systems analysis, design, and development of Client/Server, Web and Mainframe applications and systems utilizing: Applix TM1 (including TM1 rules, TI, TM1Web and Planning Manager), dynaSight - ArcPlan, ASP, DHTML, XML, IIS, MS Visual Basic and VBA, Visual Studio, PERL, Websuite, MS SQL Server, ORACLE, SYBASE SQL Server, etc. His Responsibilities have included all aspects of Windows and SQL solution development and design including: analysis; GUI (and Web site) design; data modeling; table, screen/form and script development; SQL (and remote stored procedures and triggers) development and testing; test preparation and management and training of programming staff. Other experience includes development of ETL infrastructure such as data transfer automation between mainframe (DB2, Lawson, Great Plains, etc.) systems and client/server SQL server and Web based applications and integration of enterprise applications and data sources. In addition, Mr. Miller has acted as Internet Applications Development Manager responsible for the design, development, QA and delivery of multiple Web Sites including online trading applications, warehouse process control and scheduling systems and administrative and control applications. Mr. Miller also was responsible for the design, development and administration of a Web based financial reporting system for a 450 million dollar organization, reporting directly to the CFO and his executive team. Mr. Miller has also been responsible for managing and directing multiple resources in various management roles including project and team leader, lead developer and applications development director. Specialties Include: Cognos/TM1 Design and Development, Cognos Planning, IBM SPSS and Modeler, OLAP, Visual Basic, SQL Server, Forecasting and Planning; International Application Development, Business Intelligence, Project Development. IBM Certified Developer - Cognos TM1 (perfect score 100% on exam) IBM Certified Business Analyst - Cognos TM1

More from this Author

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to the Weekly Blog Digest:

Sign Up