I’m writing a series of posts about Generalizing Apdex. This is #15. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now ready to go back and fill in those less contentious paragraphs. Over the next week, I plan to post updated drafts of each section of the Apdex-G spec, beginning today with Section [1] Introduction.
Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G. Continue reading Apdex-G Section [1] Introduction
I’m writing a series of posts about Generalizing Apdex. This is #14. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
The current Apdex specification defines an attractively simple scoring rule in which measurements falling within the zones Satisfied, Tolerating, and Frustrated receive scores of 1, ½, and 0 respectively. Enhancements have been proposed that would involve retaining the core approach to scoring while allowing for more than two targets (or thresholds), thereby creating more than three zones, and using finer scoring gradations (but still between 0 and 1) for rating measurements that fall within those zones.
In my previous posts in this series, I have stated that the notion of classifying all measurements into one of three performance zones is a core feature of Apdex that should be retained, because three-category classification schemes are common to many measurement and reporting domains. I have also made the case for allowing more thresholds, and thereby defining each performance zone as the union of one or more distinct performance intervals. For the conclusion of that discussion, see Generalizing the Apdex Thresholds.
In this post I will consider whether the Apdex formula itself should be generalized. In particular, should Apdex-G accommodate other scoring rules within the Tolerating Zone–and if so, how?
I’m writing a series of posts about Generalizing Apdex. This is #13. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
In my previous post in this series, I reviewed the first part of section §5 Reporting in the Apdex-G spec, introducing Interval Notation and proposing a new comma-separated format for the Uniform Output File. For the full story, see Configurable Reporting in Apdex-G. In this post, I cover the remainder of section §5.
First I will explore the question of what data should be made available to an independent reporting tool that queries an Apdex-based data analysis service. The current spec requires tools to report Apdex scores in Uniform Output format, but does not require any identifiers or context to be supplied with those scores. This strikes me as an omission, especially since section §3.2 Report Groups contains mandatory rules for defining report groups, which are “the foundation for an Apdex calculation”.
I’m writing a series of posts about Generalizing Apdex. This is #12. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
In my previous post in this series, I introduced a conceptual model for Apdex-G. Compared to the current Apdex model, the key addition is the Performance Interval, a range of values within the measurement domain that shares a common Satisfaction Level of ‘Satisfied’, ‘Tolerating’, or ‘Frustrated’. Thresholds now form the boundaries of performance intervals, rather than Performance Zones–which are now defined as a collection (union) of one or more performance intervals. For the full story, see Generalizing the Apdex Thresholds.
This generalization allows Apdex-G to accommodate arbitrary orderings of satisfaction levels within the measurement domain, including discontinuous performance zones. But it also complicates Apdex reporting. The current Apdex model, which employs just two thresholds, T and F, makes it easy to report the threshold(s) alongside the value of an Apdex score. And doing so is a mandatory requirement of Apdex:
All Apdex values are calculated with a particular target threshold, T. The value of T must be clearly displayed in association with the Apdex score.
– Apdex spec, section §5 Reporting the Index, introduction
How can we retain this essential Apdex feature in Apdex-G, which must accommodate reporting on arbitrary alignments of performance intervals?
I’m writing a series of posts about Generalizing Apdex. This is #11. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
In my previous post in this series, I discussed non-monotonic metrics. These are metrics for which improvement in the metric does not always always correspond to an improvement in quality. To accommodate such metrics, the Apdex-G specification must support six distinct orderings of the three Apdex performance zones (Satisfied, Tolerating, and Frustrated). It should also accept discontinuous zone definitions. For the full story, see Configurable Zone Alignment in Apdex-G,
These requirements introduce two complications, both involving the current spec’s use of just two thresholds, T and F, to define the three Apdex zones. In this post, I will review:
I’m writing a series of posts about Generalizing Apdex. This is #10.
In my post on Separately Configurable Thresholds in Apdex-G, I began work on a general description of Apdex zones and thresholds. The language I proposed would allow Apdex to be used to report on metrics for which “quality” increases or decreases monotonically with the value of the metric. Response times and VOIP MOS scores are typical monotonic metrics: smaller response times are always better, higher MOS scores are always better. Many metrics have this property. Arthur Schneiderman writes:
Properties of good metrics
The first requirement for a good metric is that it should be a reliable proxy for stakeholder satisfaction. In other words, improvement in the metric should link directly to improved stakeholder satisfaction. This linkage should be clear and uncomplicated. It should also be what mathematicians call monotonic—i.e., improvement in the metric should always produce improved stakeholder satisfaction …
I’m writing a series of posts about Generalizing Apdex. This is #9.
Previously I analyzed the current Apdex specification, asking the question Which Apdex Features Can Be Generalized? and answering with a list of topics to be worked on. I began by establishing working assumptions about document structure and terminology:
Separate Core Apdex Rules from Domain-Specific Rules
We should separate the core Apdex method into its own specification, and create distinct spec documents that contain any rules associated with domain-specific applications of the Apdex method. I will refer to the domain-specific components of the spec as addenda.
Apdex Document Names
I will refer to the new core specification as Apdex-G. This document will contain definitions and notation for all the core Apdex features we decide to generalize, such as targets, zones, and scoring. If I need to refer to a domain-specific addendum, I will use a suffix appropriate to their particular domain, such as Apdex-R for response time, Apdex-V for VOIP quality, Apdex-S for service quality, Apdex-N for network performance, etc.
As I have begun work on the details (see Generalizing the Apdex Language and Separately Configurable Thresholds in Apdex-G), a mental framework for the structure and content of each document has begun to emerge. This post aims to solidify that framework by laying out draft templates for Apdex-G and the addenda.
I’m writing a series of posts about Generalizing Apdex. This is #8. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
To compute an Apdex score for a set of response-time measurements, you count how many measurements fall into each of three performance zones. Apdex calls these zones Satisfied, Tolerating, and Dissatisfied but, as Neil Gunther observed, the labels … are unimportant. You could just as easily call them Good, Bad and UglyGUNT09.
Indeed, while investigating other potential applications of Apdex in other measurement domains, I found many other examples of measured outcomes being classified using three categories. Of course, many other labels are being used for those categories. For example:
I’m writing a series of posts about Generalizing Apdex. This is #7. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].
In my previous post, I reviewed the Apdex specification systematically, asking Which Apdex Features Can Be Generalized? That exercise showed that creating a general version of the spec involves more than generalizing the rules for implementing the features of Apdex; the language describing those features must be generalized too.
The current spec is grounded in a conceptual model of a specific measurement domain. Its language refers to users of an application being productive based on the responsiveness of a human-computer interface. We encounter this as soon as we begin reading Section §1, shown in the left-hand column below. In the new generalized specification documents, that terminology belongs in the domain-specific addendum, Apdex-R, which will deal with how to apply the Apdex method to response time measurements.
In Apdex-G we need abstract language describing only the properties of a metric. If more concrete language is needed to clarify an explanation, if must appear in an illustrative example. The only exception is text in Apdex-G describing the origins of Apdex.
I’m writing a series of posts about Generalizing Apdex. This is #6.
The current Apdex specification is entirely focused on application response time, specifically on the response time of Tasks and Task Chains. However, people have already adopted the Apdex method as a convenient way to report on other metrics, including network “turns”, bandwidth, and VOIP quality scores. These adaptations demonstrate the strength and adaptability of the core Apdex concepts. But because such ad hoc extensions fall outside the standard, they are likely to be implemented in inconsistent ways. By generalizing the Apdex spec, we aim to rectify that situation, bringing a wide class of metrics in a wide range of measurement domains within the scope of the standard.
Which Apdex features are candidates to be generalized? To answer that question, I will review the current Apdex specification. For clarity, I will use the section symbol (§) when referring to specific sections or subsections of the spec. During this pass, my goal is only to identify the major aspects of the current standard to be reviewed and reworked, not to address the actual structure or language of the generalized spec. It will be much easier to produce precise language once the concepts are clear.
