In my last post I introduced the concept of performance and performance testing. Without establishing baselines, performance testing is a waste of time. So what is a baseline and why is it important to performance testing?
Baselines
Measuring performance requires an initial state (how is the application performing at this time with these parameters?) and a next state (how is the application performing at this time with these parameters after this event took place?).
A base line is defined as the “initial state of particular application event”. The marking of significant states (within a series of several changes) and the identification of significant states (within a revision history) is the essence of performance testing.
Sample Baselines
The following are a few examples of valid baselines:
- The total time in seconds to load 1 (average) year of sales activity into a sales cube
- The total time in seconds to open a particular view on a particular cube
- The total time in seconds to refresh a TM1Web report
- The total time in seconds for the TM1 Server to start
Or:
- The total memory consumed during a load of 1 (average) year of sales activity into a sales cube
- The total memory consumed during an open of a particular view on a particular cube
- The memory consumed during a refresh a TM1Web report
- The memory consumed during a during a TM1 Server to start-up
The following are critical in establishing valid baselines:
- Control – baselines must be established in a consistent environment
- Repeatable – the process of establishing a baseline must be repeatable (and optimally, be automated)
- Reflect an “Average” – a baseline must be based upon results captured over time at regular intervals
- Scale – baselines should call out variances that can affect their interpretation, such as changes in the number or type of users, volumes and the patterns of data, security, etc.
- Common Sense – does the baseline “make sense”? Are we measuring a reasonable event and in a reasonable way?
- Comparable – Can we compare this baseline to an external baseline to establish perspective? (for example: ”n seconds to refresh a P&L report in TM1 compares to n minutes to refresh a similar P&L report in some other system.
What Events to Baseline
Application events that are baseline candidates fall into 1 of the following categories:
- Application User Focused Events
- Administrative User Focused Events
- System Maintenance Focused Events
- System Infrastructure Focused Events
Application User Focused Events
These are events performed by application business users. Examples would include but are not limited to:
- Opening and refreshing reports
- Logging into the application
- Performing an update of business data
- Opening cube views
- Modifying cube views in TM1 Cube Viewer
- Creating snapshots, slices or active form reports
Administrator User Focused Events
These are events performed by a TM1 application administrative user (not a TM1 Admin). Examples would include but are not limited to:
- Running specific Chores and or Processes
- Logging into the application
- Publishing reports to the TM1 applications folder
- Modifying security (if appropriate)
- Exporting and Importing data (to and from files)
System Maintenance Focused Events
These are events performed by a TM1 application administrative user (not a TM1 Admin) or are automated.. Examples would include but are not limited to:
- Running specific Chores and or Processes
- Updating dimensions and dimension hierarchies
- Set application assumption values and settings
- Loading or clearing data
- Performing the processes to save data to disk (from memory)
- Performing Security Refreshes
System Infrastructure Focused Events
These are events are more narrowly focused on the TM1 software itself:
- Stopping, Starting and Restarting the TM1 server, Excel server and TM1 Admin Server
- Stopping and restarting web services
Next in my performance testing series I will talk about “profiling”. See you then…
