I’m writing a series of posts about Generalizing Apdex. This is #14. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
The current Apdex specification defines an attractively simple scoring rule in which measurements falling within the zones Satisfied, Tolerating, and Frustrated receive scores of 1, ½, and 0 respectively. Enhancements have been proposed that would involve retaining the core approach to scoring while allowing for more than two targets (or thresholds), thereby creating more than three zones, and using finer scoring gradations (but still between 0 and 1) for rating measurements that fall within those zones.
In my previous posts in this series, I have stated that the notion of classifying all measurements into one of three performance zones is a core feature of Apdex that should be retained, because three-category classification schemes are common to many measurement and reporting domains. I have also made the case for allowing more thresholds, and thereby defining each performance zone as the union of one or more distinct performance intervals. For the conclusion of that discussion, see Generalizing the Apdex Thresholds.
In this post I will consider whether the Apdex formula itself should be generalized. In particular, should Apdex-G accommodate other scoring rules within the Tolerating Zone–and if so, how?
I’m writing a series of posts about Generalizing Apdex. This is #13. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
In my previous post in this series, I reviewed the first part of section §5 Reporting in the Apdex-G spec, introducing Interval Notation and proposing a new comma-separated format for the Uniform Output File. For the full story, see Configurable Reporting in Apdex-G. In this post, I cover the remainder of section §5.
First I will explore the question of what data should be made available to an independent reporting tool that queries an Apdex-based data analysis service. The current spec requires tools to report Apdex scores in Uniform Output format, but does not require any identifiers or context to be supplied with those scores. This strikes me as an omission, especially since section §3.2 Report Groups contains mandatory rules for defining report groups, which are “the foundation for an Apdex calculation”.
I’m writing a series of posts about Generalizing Apdex. This is #12. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
In my previous post in this series, I introduced a conceptual model for Apdex-G. Compared to the current Apdex model, the key addition is the Performance Interval, a range of values within the measurement domain that shares a common Satisfaction Level of ‘Satisfied’, ‘Tolerating’, or ‘Frustrated’. Thresholds now form the boundaries of performance intervals, rather than Performance Zones–which are now defined as a collection (union) of one or more performance intervals. For the full story, see Generalizing the Apdex Thresholds.
This generalization allows Apdex-G to accommodate arbitrary orderings of satisfaction levels within the measurement domain, including discontinuous performance zones. But it also complicates Apdex reporting. The current Apdex model, which employs just two thresholds, T and F, makes it easy to report the threshold(s) alongside the value of an Apdex score. And doing so is a mandatory requirement of Apdex:
All Apdex values are calculated with a particular target threshold, T. The value of T must be clearly displayed in association with the Apdex score.
– Apdex spec, section §5 Reporting the Index, introduction
How can we retain this essential Apdex feature in Apdex-G, which must accommodate reporting on arbitrary alignments of performance intervals?
I’m writing a series of posts about Generalizing Apdex. This is #11. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
In my previous post in this series, I discussed non-monotonic metrics. These are metrics for which improvement in the metric does not always always correspond to an improvement in quality. To accommodate such metrics, the Apdex-G specification must support six distinct orderings of the three Apdex performance zones (Satisfied, Tolerating, and Frustrated). It should also accept discontinuous zone definitions. For the full story, see Configurable Zone Alignment in Apdex-G,
These requirements introduce two complications, both involving the current spec’s use of just two thresholds, T and F, to define the three Apdex zones. In this post, I will review:
I’m writing a series of posts about Generalizing Apdex. This is #10.
In my post on Separately Configurable Thresholds in Apdex-G, I began work on a general description of Apdex zones and thresholds. The language I proposed would allow Apdex to be used to report on metrics for which “quality” increases or decreases monotonically with the value of the metric. Response times and VOIP MOS scores are typical monotonic metrics: smaller response times are always better, higher MOS scores are always better. Many metrics have this property. Arthur Schneiderman writes:
Properties of good metrics
The first requirement for a good metric is that it should be a reliable proxy for stakeholder satisfaction. In other words, improvement in the metric should link directly to improved stakeholder satisfaction. This linkage should be clear and uncomplicated. It should also be what mathematicians call monotonic—i.e., improvement in the metric should always produce improved stakeholder satisfaction …
I’m writing a series of posts about Generalizing Apdex. This is #8. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
To compute an Apdex score for a set of response-time measurements, you count how many measurements fall into each of three performance zones. Apdex calls these zones Satisfied, Tolerating, and Dissatisfied but, as Neil Gunther observed, the labels … are unimportant. You could just as easily call them Good, Bad and UglyGUNT09.
Indeed, while investigating other potential applications of Apdex in other measurement domains, I found many other examples of measured outcomes being classified using three categories. Of course, many other labels are being used for those categories. For example:
I’m writing a series of posts about Generalizing Apdex. This is #6.
The current Apdex specification is entirely focused on application response time, specifically on the response time of Tasks and Task Chains. However, people have already adopted the Apdex method as a convenient way to report on other metrics, including network “turns”, bandwidth, and VOIP quality scores. These adaptations demonstrate the strength and adaptability of the core Apdex concepts. But because such ad hoc extensions fall outside the standard, they are likely to be implemented in inconsistent ways. By generalizing the Apdex spec, we aim to rectify that situation, bringing a wide class of metrics in a wide range of measurement domains within the scope of the standard.
Which Apdex features are candidates to be generalized? To answer that question, I will review the current Apdex specification. For clarity, I will use the section symbol (§) when referring to specific sections or subsections of the spec. During this pass, my goal is only to identify the major aspects of the current standard to be reviewed and reworked, not to address the actual structure or language of the generalized spec. It will be much easier to produce precise language once the concepts are clear.
I’m writing a series of posts about Generalizing Apdex. This is #4.
I ended my earlier post on Core Apdex Qualities with this observation: “I believe that a core benefit of Apdex is its usefulness as a management indicator (such as a Key Performance Indicator, or KPI). I don’t think the current spec really captures that aspect properly, so I’m going to devote my next post to that subject”.
Since then I have done extensive research on the subject of performance indicators, resulting in a collection of relevant papers, articles, and blog posts–see my later post on Metrics and Performance Indicators: A Bibliography. I have also been extending my original list of 20 references, appending several newer discoveries of relevant online resources as comments. But much as I enjoy doing research, at some point I have to stop and draw some conclusions, however tentative. So here’s a bit more detail about the role of Apdex as a (key) performance indicator.
I’m writing a series of posts about Generalizing Apdex. This is #3.
In my previous post on Core Apdex Qualities, I set out to enumerate the essential characteristics of the Apdex metric. What aspects of the Apdex method, and the metric it produces, would make it useful for reporting measurements in domains other than the application response time, the focus of the Apdex specification today?
The Apdex goals already identified in section 2.1 of the spec are certainly a good starting point, but I felt that to properly assess where a generalized Apdex metric might potentially be applied, I needed to establish a general picture of the use of metrics and performance indicators in management. This is easier said than done. How, for example, can we discuss the applicability of an Apdex metric as a Key Performance Indicator (KPI), when no standard definition of that concept exists?
If you spend a couple of weeks sorting through Google’s ~4 million references to key performance indicators, two things will happen. First, you will see that confusion abounds; there is no generally accepted framework we can use to position Apdex. Second, you will begin to assemble your own short list of useful reading that will eventually reveal some broad conclusions about the field of management metrics. This post records my own experience doing this, with potential applications of a generalized Apdex standard in mind.
Today the Apdex specification is entirely focused on application response time. Its first paragraph defines Apdex as “a method for calculating and reporting a metric of transactional application response time in the form of an index with a value of 0 to 1.” But in reality, the Apdex method is much more widely applicable, and a more appropriate description is already spelled out in the first paragraph of the Wikipedia article on Apdex:
(Apdex) defines a standard method for reporting and comparing the performance of software applications in computing. Its purpose is to convert measurements into insights about user satisfaction, by specifying a uniform way to analyze and report on the degree to which measured performance meets user expectations. – Referenced on May 7, 2010
Of course, there’s a good reason why Wikipedia reflects the broader view. Over the years, the idea of generalizing the Apdex standard to apply in other domains has been discussed periodically within the Apdex community. So when I created the Apdex wiki description in 2007, I already had those discussions in mind.
