If Sir Isaac Newton were stating the laws of computer systems performance, his first law would surely have been: The graph of performance continues in a straight line unless the force of some external event causes it to change.
Not knowing what changed is a serious impediment to problem diagnosis.
How does performance suddenly become “abnormal”? Of course, the answer is, it doesn’t–at least, not on its own. There is always an external cause. So to fix a performance problem, we must find the cause–usually more of something, such as:
- Increased processing volumes
- More data in a database
- More customers
- A new application competing for resources
- Increased competition from existing applications on the same servers
- Increased interference from other traffic on the network.
Tracking Growth Patterns
Growth is not always sudden or dramatic. Sudden workload growth can produce sudden changes in performance; gradual growth tends to produce a corresponding gradual decline in performance levels.
Any computer system may gradually reach a point where memory size, processor speed, I/O processing and communications overhead are interacting in a way which was not predicted when an earlier performance model was evaluated. Regular monitoring should reveal a corresponding gradual decline in performance levels.
Environmental Changes
If nothing in the workload grew noticeably, then the explanation must be that some component of the application or its environment became less efficient. Typical examples of environmental changes that create performance problems are:
- A new version of the application, or a fix to critical application routines, is installed, changing the way hardware or systems software resources are used.
- A database or index becomes sufficiently disorganized that normally efficient processing is seriously degraded.
- Previously matched software components become mismatched when system software is upgraded. For example, a new version of database software may handle certain types of request differently, and previous default settings used by the client need to be changed accordingly.
- Changes to system software parameters (like cache sizes, scheduling priorities, or available threads) can affect the performance of some applications directly. Changes in the environment can invalidate the previous settings of these system parameters. For example, after a new application has been added to a system, settings that were suitable for a low volume of activity may now be causing excessive queuing of service requests.
- Changes in the hardware or software environment can consume resources previously available. For example, new applications, users, or devices may be suddenly dumping large volumes of data onto a shared network.
- A network device may develop a fault that reduces the effective bandwidth of the network.
Monitor External Changes
In the Apdex community, we tend to focus on the numerical aspects of performance. But successful performance management demands more than merely collecting, reporting, and tracking measurement data. The ideal performance management process is one that also systematically tracks all external changes, because they are the source of most performance problems.
This is the first of a series of posts I plan to write on the subject of “Performance Principles.” I encourage members of the Apdex community to contribute comments about practical applications of these principles from their own experience.