Welcome to the Managing BI Performance Problem series. This will be a 4 (or possibly more) part series on how to tackle BI performance problems. This first part is the most important. I am fond of saying, “A good manager is worth three good developers. However, three great developers cannot make up for one bad manager.” Ergo, it is important that the right direction is set from the beginning. Given the nature of this post a lot of the lessons here can be used for almost any application performance problem, however, future installments will deal with BI specific topics.
As a consultant I’ve seen juggernauts of industry and management be reduced to prattling amateurs when faced with application performance issues. Managing performance problems can be very different than managing development or support of an application and as such many managers are ill prepared to deal with a major performance problem when one arises. Here are some tips to avoid confusion and delay.
1. Understand and accept what can create a performance problem.
When faced with an application performance problem most managers fall back on what had worked for them before, be that database indexing or code optimization. While that approach at times can lead to good results, as often as not they fall short. Reason for that is that performance problems can be far more complex and require more than a one tiered approach.
Application performance problems can reside in any of the following:
1.) The Code/Configuration
2.) The Database
3.) The Network
4.) The Server Hardware
5.) The Vendor’s software
And most likely
6.) Any combination of any of the above.
It is important to realize and accept that a performance problem can be a very complicated thing that can span multiple groups each with their own manager and their own goals. Understand that a performance problem can be a manifestation of issues within several different areas, prepare to take action accordingly.
2. Create a Performance “SWAT” team. Eliminate “Blame storming” and “Not it!” mentalities.
You can ask a fisherman why his boat isn’t as fast as it should be, but it’d be more effective to ask the ship builder. Many companies make the mistake of asking their developers to become DBAs or their DBAs to optimize configuration or expect them to suddenly be networking/hardware experts. In some cases it might be the case that resources have the skill and access to jump between roles. However, a concerted effort should be made to secure the experts in the areas outlined above instead of overloading resources in areas they might not be entirely proficient in. Speed should be the main focus of the team, as such time wasted trying to learn new technology is opportunity wasted.
Once a team is assembled it is of vital importance to eliminate any “blame storming” or “not it” mentalities that might exist. Left to their own devices many “SWAT” teams quickly denigrate to finger pointing, fault finding and cover up. Set the tone from the beginning that the focus of the group is not to lay blame or chastise, but to fix the problem. People do not perform well, if at all, when they feel threatened.
3. Make good application performance a measurable goal for all
It does little good to assemble a team of people to solve a problem when they are not measured by, and held accountable for, their ability to solve the problem. Doing so will only lead to team members peeling off to work on things that they are being measured on and are being held account for. Therefore, performance should be an organizational goal held by any group that has a stake in performance.
4. Create a Post-Mortem with actionable findings
This, actually, is one step most companies already do. The concept of a “post-mortem” is generally well held. However, many fail when it comes to following through. I have sat in post-mortems where the findings were, “We didn’t follow what we learned in the last post mortem.” Findings from post mortems should be worked into employee and manager annual performance plans. Again, care should be taken to not lay blame, lest you alienate your partners.
5. Listen to your team and enable to them to be proactive
The best way to fix a performance problem is to not have one in the first place. Good resources often will be able to pick out things that might cause performance problems before they happen. Foster a culture where such concerns can be voiced and are quickly acknowledged by bringing in all parties that might be able to help. Many managers deal with possible performance problems by simply telling their resources to do their job better. Such an approach only tells your resources that at best you don’t take the issue seriously, or at worst you don’t trust them and their work. Give your resources access to other team members’ time to try and fix performance problems before they occur.
6. Performance test early, often and thoroughly
Even before their prototypes are built engineers performance test their components. Chefs taste their soup as they cook it. Musicians record and listen to themselves well before performing. For some reason this level of care is generally ignored when it comes to software development, where many applications undergo performance testing a few weeks before go live, if at all.
There are plenty of good software/service options out there to create performance tests. But even when companies employ them many either employ them too late or only account for “happy path” usage. Developers will hate it because it might point to non-issues but consistent performance testing early on in the development cycle can point to unforeseen problems in enough time for developers to address the root cause.
Leaving performance testing until the end of a project is similar to only tasting a dish when it’s almost finished. You’re not really testing it’s ok, you’re hoping it’s ok and hope is no performance strategy.
