The Challenges of measuring performance in software delivery
Surveys on software development metrics are nothing new. This topic has been a source of discussion for decades with little change to the dichotomy of findings. To quote just one example:
A recent global survey of over 150 CIOs found that while over 75% of them recognized a strong business need to measure performance of their IT organizations, less than 1/3 of those same IT organizations actually measured their performance.
The above is definitely consistent to what I find when speaking to many executives in the IT space and is fairly uniform regardless of industry, size and maturity level of development methodology. The only exceptions to this being fairly small IT organizations (less than 50). These smaller organizations ‘tend’ to have a better handle on the performance and measurement of their organizations – although one could easily argue this has more to do with the fact that these smaller organizations have an easier time getting their arms around their organization’s day to day activities. In other words – in smaller organizations, even subjective evaluations coupled with a small amount of sampling metrics, seem to reasonably predict how their organizations are fairing.
Certainly this is not true of all small IT organizations and for the majority of IT organizations of any moderate size, subjective evaluations of performance at the executive management levels are often quite far off from the actual inner workings of how things are actually ‘getting done’.
Furthermore – over 70% of mid-size to large IT organizations currently engage in some sort of multi-shore engagements and 90% of IT organizations are evaluating it as part of their strategic planning. Yet results from a recent Forrester Research survey in 2009, less than ½ of IT organizations are ‘satisfied’ with the results they are getting from their multi-shore efforts.
So if the majority of IT leadership agrees that metrics are important, why are so few actually measuring?
Based on my own observations through the years, I’ve heard a wide range of answers:
- Lack of time and / or money
- Lack of discipline, priority or perceived incremental ROI
- Lack of effective tools
- Not knowing what metrics to standardize on
- Lack of industry metrics to compare to / apples-to-oranges across industries
- Procrastination (always ‘working on it’)
- Inability to project meaningful metrics at an executive summary level
But all of this really boils down to a matter of prioritization. IT folks are generally pretty good problem solvers given the opportunity and priority. The key is to integrate measurement as part of the development lifecycle so metrics become a natural output of that lifecycle.
In the next post of this series, we’ll explore how Agile methodology can enable more accurate and timely measurements.
