I’m writing a series of posts about Generalizing Apdex. This is #20. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now going back to fill in those less contentious paragraphs, posting updated drafts of each section of the Apdex-G spec. The first five are Section [1] Introduction, Section [2] Index Overview, Section [3] Calculation Inputs, Section [4] Calculating the Index, and Section [5] Reporting; today I continue with Section [6] References.

Summary

In this section, I have retained the original references from Apdex section §6, added four new ones, and introduced headings to show which sections the references relate to.

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G.

Next Steps …

This completes my second draft. I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G. I am postponing work on Section [7] Glossary until I have completed a draft of Apdex-R. If I want to record any new terminology, or edit previous definitions, I will amend my previous post, An Extensible Apdex Glossary.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

I’m writing a series of posts about Generalizing Apdex. This is #19. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now going back to fill in those less contentious paragraphs, posting updated drafts of each section of the Apdex-G spec. The first four are Section [1] Introduction, Section [2] Index Overview, Section [3] Calculation Inputs, and Section [4] Calculating the Index; today I continue with Section [5] Reporting.

Summary

In this section, I have generalized and reorganized the material in Apdex section §5 as follows:

• The introduction is a rewording of the current spec.
• Section [5.1] corresponds to Apdex section §5.1. I have numbered the four rules.
• Section [5.1.1] corresponds to Apdex section §5.3, describing general cases. Placing it here improves the flow.
• Section [5.2] corresponds to the last part of Apdex section §5.1 on Uniform Output, revised and extended. These extensions were discussed in two previous posts — Configurable Reporting in Apdex-G and Report Groups and Quality Ratings in Apdex-G.
• Section [5.3] corresponds to Apdex section §5.2. I have generalized the language, and tried to enumerate the possible classes of errors in a general way. This section was discussed briefly with the first draft.
• Section [5.4] corresponds to Apdex section §5.4. I have generalized the language.

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G.

Public Comment

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

I’m writing a series of posts about Generalizing Apdex. This is #18. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now going back to fill in those less contentious paragraphs, posting updated drafts of each section of the Apdex-G spec. The first three are Section [1] Introduction, Section [2] Index Overview, and Section [3] Calculation Inputs; today I continue with Section [4] Calculating the Index.

Summary

The first draft of this section of Apdex-G was posted as Configurable Scoring in Apdex-G. In this section, I have generalized and reorganized the material in Apdex section §4 as follows:

• The introduction generalizes the language in the current spec.
• Section [4.1] corresponds to Apdex section §4.1. I have generalized the language, removing references to response times. Since the first draft, I have revised the references to earlier sections, making them more complete.
• Section [4.1.1] corresponds to the last paragraph of Apdex section §4.1. I have rewritten this section, generalizing the language and adding the second paragraph. I am not sure if the third paragraph is required.
• Section [4.1.2] is new. It reflects the discussion of alternative scoring methods published with the first draft, and extends the conclusion with a rule for the implementation of a sliding scale.
• Section [4.2] corresponds to Apdex section §4.2. I have generalized the language, and tried to enumerate the possible classes of errors in a general way. This section was discussed briefly with the first draft.

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G.

Public Comment

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

I’m writing a series of posts about Generalizing Apdex. This is #17. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now going back to fill in those less contentious paragraphs. Over the next week, I plan to post updated drafts of each section of the Apdex-G spec. The first two are Section [1] Introduction and Section [2] Index Overview; today I continue with Section [3] Calculation Inputs.

Summary

In this section of Apdex-G, I have generalized and reorganized the material in Apdex section §3 as follows:

• The introduction repeats the current spec, adding a note about the possibility of a domain specific addendum.
• Section [3.1] corresponds to Apdex section §3.1. I have generalized the language, and removed the discussion of Web response time measurements, which will be reviewed later for inclusion in Apdex-R.
• Section [3.2] corresponds to Apdex section §3.2. I have rewritten this section to improve the flow, added the Measurement Subtype filter (to accommodate the distinction between Task and Task Chain in the current spec, and later in Apdex-R, for example).
• Section [3.3] corresponds to Apdex section §3.4. I reversed the order of sections §3.3 and §3.4, to improve the flow of ideas. For the same reason, I moved content about report group size in §3.2 to [3.3].
• Section [3.4] corresponds to Apdex section §3.3, generalizing response time terminology, and adding a paragraph about adjusting thresholds to compensate for differences between tools.

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G.

Public Comment

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

I’m writing a series of posts about Generalizing Apdex. This is #16. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now going back to fill in those less contentious paragraphs. Over the next week, I plan to post updated drafts of each section of the Apdex-G spec. I began yesterday, with Section [1] Introduction; today I continue with Section [2] Index Overview.

Summary

In this section of Apdex-G, I have generalized and reorganized the material in Apdex section §2 as follows:

• The introduction contains a brief description of the reasons for reducing a set of measurements to an index.
• Section [2.1] corresponds to Apdex section §2.1. I have generalized the language, and made the last two goals (comparability among tools and applications) more realistic. This section was originally discussed in Core Apdex Qualities.
• Section [2.2] corresponds to Apdex section §2.2. I have added definitions for some terms used but not defined in the current Apdex spec, and introduced Measurement Domain and Tool Creator.
• Section [2.3] replaces all the material currently found in Apdex section §2.3, which will be reviewed later for inclusion in Apdex-R.
• Section [2.4] replaces all the material currently found in Apdex section §2.4, which will be reviewed later for inclusion in Apdex-R.
• Section [2.5] presents the most significant changes in section [2]. The difference is illustrated in Figure 1 below. First, section [2.5] defines three Satisfaction Levels. Next, section [2.5.1] defines Thresholds as the boundaries of Performance Intervals, which are ranges of values within the measurement domain that share a common satisfaction level. Finally, section [2.5.2] defines Performance Zones as a collection (union) of one or more performance intervals. The arguments for these proposals were presented in Configurable Zone Alignment in Apdex-G and Generalizing the Apdex Thresholds. The extensive discussion of threshold relationships at the end of section §2.5 will be reviewed later for inclusion in Apdex-R.
• Section [2.6] corresponds to Apdex section §2.6.

Figure 1. Conceptual Models of Apdex and Apdex-G

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G.

Open Issues and Public Review

I am undecided on some text formatting questions. As a result, this draft deliberately illustrates different possibilities, rather than adopting single a consistent style throughout:

• I am unsure whether to capitalize and/or bold Apdex terminology (a) when it is first introduced, and (b) when it is referenced later. In section [2.5] above I experimented with using bold face for terms of art that are referenced within definitions.
• I am considering using glossary-style definitions when introducing terms within the body of the document. This approach, illustrated in [2.2] and [2.5] above, allows definitions in the glossary to be made identical to their counterparts in the earlier sections of the spec.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.experimenting

I’m writing a series of posts about Generalizing Apdex. This is #15. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now ready to go back and fill in those less contentious paragraphs. Over the next week, I plan to post updated drafts of each section of the Apdex-G spec, beginning today with Section [1] Introduction.

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G.

Summary

In this section of Apdex-G, I have reorganized the material in Apdex section §1 as follows:

• The introduction contains a brief generic description of Apdex, originally discussed in Generalizing the Apdex Language.
• Section [1.1] is largely new in Apdex-G. It introduces the separation of Apdex specifications into generic and domain-specific documents. This was originally discussed in Which Apdex Features Can Be Generalized?
• Section [1.2] corresponds to Apdex section §1.1, apart from material that has been moved to [1.3]
• Section [1.3] summarizes Alliance goals and points to the Alliance website for more details. It replaces material currently found in Apdex sections §1.1 and §1.2.

Parts of the current spec, shown on the left below, have been rearranged to show correspondences with the Apdex-G proposal:

Public Review

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

I’m writing a series of posts about Generalizing Apdex. This is #14. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

The current Apdex specification defines an attractively simple scoring rule in which measurements falling within the zones Satisfied, Tolerating, and Frustrated receive scores of 1, ½, and 0 respectively. Enhancements have been proposed that would involve retaining the core approach to scoring while allowing for more than two targets (or thresholds), thereby creating more than three zones, and using finer scoring gradations (but still between 0 and 1) for rating measurements that fall within those zones.

In my previous posts in this series, I have stated that the notion of classifying all measurements into one of three performance zones is a core feature of Apdex that should be retained, because three-category classification schemes are common to many measurement and reporting domains. I have also made the case for allowing more thresholds, and thereby defining each performance zone as the union of one or more distinct performance intervals. For the conclusion of that discussion, see Generalizing the Apdex Thresholds.

In this post I will consider whether the Apdex formula itself should be generalized. In particular, should Apdex-G accommodate other scoring rules within the Tolerating Zone--and if so, how?

I am going to approach that question through the process of reviewing section §4 of the current spec (shown in the left-hand column below), and proposing text for the corresponding paragraphs of Apdex-G (shown on the right).

Describing the Formula

In composing the introductory paragraph and section [4.1] of Apdex-G, in addition to removing references to response times, my goal was to reduce the amount of repitition in the spec. Rather than describing concepts that have already been defined in earlier sections, I decided to simply refer to the appropriate descriptions or definitions. In this case, the relevant paragraphs should all be found in section [3] Apdex Inputs. I have not yet drafted these, but when I do, I will make sure they fit properly with section [4].

Pros and Cons of Configurable Scoring

One way to describe an Apdex score is as the weighted proportion of satisfactory samples in a report group. Samples in the Satisfied Zone have weights of 1, those in the Frustrated Zone have weights of 0. I see no reason why those weights should change. But today, all samples in the Tolerating Zone have weights of ½, regardless of their values.

Alan Ackers and Neil Gunther GUNT09 discuss further partitioning the Tolerating Zone into sub-zones, to make the Apdex score more responsive to the distribution of tolerating samples. Similar ideas, like scoring measurements within the Tolerating zone on a sliding scale between 1 and 0, have been raised during informal discussions of Apdex at conferences like CMG. All such proposals assign progressively higher weights to samples approaching the Satisfied zone, lower weights approaching the Frustrated Zone. Ackers and others have described some pros and cons of this idea, which I summarize below:

In practice, for most measurement domains, if the average of all weights used (which must be values between 1 and 0) remains ½, then I think applying any graduated scoring system within the Tolerating zone would be unlikely to make much difference to the result of an Apdex calculation. In which case, the current method of using a constant weight of ½ for all samples in the Tolerating Zone is preferable, because it is much simpler to compute.

On the other hand, we do want Apdex-G to be applicable to any measurement domain that uses a three-category classification scheme. And allowing an Addendum the option of specifying a more general rule for scoring samples within the Tolerating Zone does not seem to undermine any fundamental characteristic of Apdex.

Therefore I propose that Apdex-G adopt the current Apdex formula as the standard scoring method, but also allow an Addendum to substitute an alternative scoring function (such as a sliding scale) for the factor Tolerating_Count/2 in the standard formula. In the draft spec for Apdex-G (continued below), I have included that option as section [4.1.2], following a note on the standard formula that I have carried over from the current Apdex spec, with some clarifying wording changes.

Errors and Exceptions

Not all measurements reflect normal operation of the system or process under study. And because the intent of an Apdex index is to track the degree to which a system under study is achieving designated targets, errors and exceptions cannot be ignored. Indeed, they should be counted whenever possible, because they usually indicate that a system or process failed to achieve a target performance level.

Apdex-G provides some general guidelines for handling exceptions, which may be refined and extended by an addendum to cover domain-specific errors and exceptions. The discussion in the current spec will be reviewed for inclusion in Apdex-R.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

References

GUNT09

The Apdex Index Revealed Neil J. Gunther, February 2009. [CMG MeasureIT]

AKER08

Discussion: Is Apdex Sharp Enough? Alan Ackers, December 2008. [Apdex Symposium presentation]

I’m writing a series of posts about Generalizing Apdex. This is #13. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

In my previous post in this series, I reviewed the first part of section §5 Reporting in the Apdex-G spec, introducing Interval Notation and proposing a new comma-separated format for the Uniform Output File. For the full story, see Configurable Reporting in Apdex-G. In this post, I cover the remainder of section §5.

First I will explore the question of what data should be made available to an independent reporting tool that queries an Apdex-based data analysis service. The current spec requires tools to report Apdex scores in Uniform Output format, but does not require any identifiers or context to be supplied with those scores. This strikes me as an omission, especially since section §3.2 Report Groups contains mandatory rules for defining report groups, which are “the foundation for an Apdex calculation”.

The Importance of Context

In The Big Book of Key Performance Indicators, Eric Peterson writes (on page 8) that “Key performance indicators are always rates, ratios, averages or percentages; they are never raw numbers. Raw numbers are valuable to web analytics reporting to be sure, but because they don’t provide context, are less powerful than key performance indicators,” and (on page 11), “Raw numbers are not key performance indicators. I know that many smart people disagree with me on this point but, well, they’re wrong” PETE06.

Among those who do disagree is Brian Clifton, author of the book Advanced Web Metrics with Google Analytics. In his blog, Measuring Success, he writes that “a KPI is not always an average, ratio or percentage – sometimes raw numbers are better” CLIF08.

Having surveyed the field of (Key) Performance Indicators, I am not surprised to find people disagreeing about what is and is not a KPI. But why not have both derived metrics and raw numbers? A derived metric like Apdex is ideal for tracking progress against targets, while the underlying data, like sample counts, supply important context.

Today Apdex recognizes this (see section §5.2 below) by demanding an asterisk when the sample count is below 100, but this does not always substitute for knowing the actual sample count. When comparing a pair of Apdex scores, 0.95 is “excellent” while 0.90 is merely “good” (see section §5.4 below) — the first is clearly better, right? But what if those two scores were based on the measurements of customer satisfaction shown in Table 1 below?

A management dashboard probably needs to draw attention to the 500 frustrated customers of application A, despite its higher Apdex score. Size does matter, for two reasons. Whenever we measure something that varies, we first need enough samples to establish that any apparent effect we observe in a sample is truly a property of the real world, and not simply a chance occurrence. Second, we need to determine the magnitude of that effect. In the field of statistics, these two aspects are called ’statistical significance’ and ‘practical (or scientific) significance’, and the two are frequently confused by scientists and non-scientists alike ZILI08, ZILI09.

Supporting Management Dashboards

Neil Gunther’s paper, The Apdex Index Revealed GUNT09 lists a possible criticism of Apdex for not specifying how to present results for “hundreds of applications”:

Actually, the Apdex spec deliberately choses to address this (in sections §5 and §5.1) by leaving the form of any “report” of Apdex scores to be determined by the tool doing the reporting. I think this is the right place to draw the line. Apdex is primarily focused on the data analysis aspect of reporting, not on data visualization. But the spec should support visualization tools by supplying the data they need.

Visualization tools need to implement natural ways to highlight the most important results, and suppress visual clutter. Sorting the lowest Apdex scores to the top could be one way. Another might be to sort by the number of samples underlying an Apdex score. That would distinguish scores based on 20,000 samples from those based on 200 samples, so that problems indicated in the larger group can be given a higher priority. Other possibilities include sorting displays by frustrated sample count, by application, by user, by a timestamp, or by report group name. I am proposing that the Apdex-G spec extend the Uniform Output format to supply these kinds of identifiers.

Neil suggests a display format (shown on the right) in which many Apdex result are lined up side-by-side for easy comparison. I agree with this direction–it is typical of many dashboard displays–but it can be improved upon. It devotes too much ink to the colored bars, and not enough to the Apdex scores (the black lines). This aspect of data visualization is a key design feature for dashboards. Edward Tufte TUFT03 and Stephen Few FEW10 both emphasize the crucial importance of maximizing the data-ink ratio, namely:

[5.1.2] Report Group Section

Section [5.1.2] is a new section that specifies an extension of the Uniform Output spec defined previously in section [5.1.1]. The additional elements, labeled 2-15, summarize the report group and distribution of samples underlying the Apdex metric. At this stage I am undecided on whether some or all of these fields should be made mandatory within the Uniform Output. Once we decide which elements are to be optional or mandatory, I believe we can design the fields of the Uniform Output Header Record to describe the contents of a file. This will allow any tool processing a Uniform Output file to identify the presence or absence of optional fields.

Because this proposal extends the current spec, there is no corresponding text in section §5. So the two columns below show the current section §3.2 Report Groups on the left, and the draft proposal for section [5.1.2] on the right.

[5.3] Describing General Cases

I have reversed the order in which I review sections §5.3 and §5.2. in this post, to eliminate a forward reference. Eventually I will make the same change in the Apdex-G spec, but for now it’s probably less confusing to retain the current spec’s numbering.

Because Apdex-G has generalized the definition and format of thresholds, I propose that Apdex-G should support only generic references to thresholds, rather than the specific forms defined in the current spec. I propose to drop the example of subscripted format because it is not consistent with the defined Apdex-G terminology. It may be included in Apdex-R.

[5.2] Sample Size

To adapt the current section §5.2 for the Apdex-G spec, I have made the langauge consistent with the notation for general thresholds defined in [5.3], and allowed for an addendum to override the definition of a ’small group’ of samples.

[5.4] Quality Ratings

To adapt the current section §5.4 for the Apdex-G spec, I have:

• introduced the term “Quality Ratings” to describe this feature
• clarified the layout and wording of the introductory paragraphs
• made the notation in the table consistent with the notation defined for general thresholds in [5.3]
• allowed for an addendum to override the color specifications if appropriate

The actual rating band values are those defined in Apdex today, which I have carried forward into Apdex-G. I have left open for further discussion the question of whether an addendum may specify domain-specific values to override these.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

References

CLIF08

A KPI is not always an average, ratio or percentage – sometimes raw numbers are better, Brian Clifton, 2008. [Measuring Success, June 2008]

FEW04

Elegance Through Simplicity Stephen Few, 2004. [Perceptual Edge, October 16, 2004]

FEW10

Perceptual Edge Stephen Few, 2010. [Perceptual Edge Website]

GUNT09

The Apdex Index Revealed, Neil J. Gunther, February 2009. [CMG MeasureIT]

PETE06

The Big Book of Key Performance Indicators, Eric T. Peterson, 2006. [1.1Mb pdf]

TUFT03

Executive Dashboards, Edward Tufte and others, 2003-2009. [Discussion thread on "Ask E.T."]

ZILI08

The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives Stephen T. Ziliak and Deirdre N. McCloskey, University of Michigan Press, 2008. [Amazon Books]

ZILI09

The Cult of Statistical Significance Stephen T. Ziliak and Deirdre N. McCloskey, Joint Statistical Meetings, Washington, DC, August 3, 2009. [179Kb pdf]

I’m writing a series of posts about Generalizing Apdex. This is #12. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

In my previous post in this series, I introduced a conceptual model for Apdex-G. Compared to the current Apdex model, the key addition is the Performance Interval, a range of values within the measurement domain that shares a common Satisfaction Level of ‘Satisfied’, ‘Tolerating’, or ‘Frustrated’. Thresholds now form the boundaries of performance intervals, rather than Performance Zones–which are now defined as a collection (union) of one or more performance intervals. For the full story, see Generalizing the Apdex Thresholds.

This generalization allows Apdex-G to accommodate arbitrary orderings of satisfaction levels within the measurement domain, including discontinuous performance zones. But it also complicates Apdex reporting. The current Apdex model, which employs just two thresholds, T and F, makes it easy to report the threshold(s) alongside the value of an Apdex score. And doing so is a mandatory requirement of Apdex:

How can we retain this essential Apdex feature in Apdex-G, which must accommodate reporting on arbitrary alignments of performance intervals?

Holding to Einstein’s Razor

The short answer is, of course, that we can’t make all Apdex-G scores as simple to display and read as the current Apdex metric. In the immortal words of Albert Einstein, we should “make everything as simple as possible, but no simpler”. Interestingly, those words are just a shortened version of what Einstein actually said, which turns out to be particularly relevant in this context:

This principle is sometimes called Einstein’s razor by people who view it as a counterpoint to Occam’s razor, namely “entities must not be multiplied beyond necessity”. Both principles can be applied to Apdex-G. First, I believe I have followed Occam’s razor when extending Apdex. Apdex-G does introduce a new entity–a performance interval–but that is necessary to accommodate arbitrary alignments of satisfaction levels. Second, we must follow Einstein’s razor to ensure that when reporting an Apdex-G score, we do not “surrender the adequate representation” of the configuration of those performance intervals.

Introducing Interval Notation

Clearly, Apdex-G must define a standard notation for describing any perfomance interval. This notation must show the lower and upper thresholds that define the interval, and whether or not they are contained within the interval. For example, in the current Apdex spec, we could describe the tolerating zone (Z_t say), using the following inequality:

T < Z_t ≤ F

where in this example the symbols T and F signify threshold values. The choice of inequalities, < and ≤, show that T, the lower threshold, is not contained within the zone, while F, the upper threshold, is included.

Interval notation ZONAW1, SOSMW1, WIKIW1 is an alternative way convey the same information without using inequality symbols; Roberta Bloom has created a handy one-page summary BLOO07. For the Apdex tolerating zone we would write:

Z_t: (T, F]

The round bracket (or parenthesis) signifies exclusion from the interval, the square bracket signifies inclusion. So we could describe the Apdex index like this:

Apdex: [0.00, 1.00]

To describe a tolerating zone composed of two intervals (for example, Metric Type #7, Symmetric Process Control in Table 4 of my previous post), we would write something like this:

Z_t: (t₁, t₂] ∪ (t₃, t₄]

where the t_i (i=1..4) are threshold values, and the mathematical “cup” symbol ∪ signifies the union of the two intervals. The full description of Metric Type #7 would be:

Z_s: (t₂, t₃]
Z_t: (t₁, t₂] ∪ (t₃, t₄]
Z_f: (−∞, t₁] ∪ (t₄, ∞)

This choice of round and square brackets in this example follow the Apdex-G rule that “Each Threshold lies within one and only one Performance Interval. Unless otherwise specified by a domain-specific Addendum, a Threshold lies within the Performance Interval for which it acts as the upper bound”. The final bracket is round by convention; in this notation, infinity symbols are always accompanied by round brackets.

I propose that we adopt interval notation to describe performance intervals and performance zones within the Apdex-G Uniform Output spec. Granted, the 3-line example given above is a lot more cumbersome than the current Apdex requirement of a simple suffix displaying a single T value. But note that:

• the example illustrates how to report a more complex metric type that employs five performance intervals
• an addendum (like Apdex-R) can specify simpler alternative notation (like the current subscript method) if the metric type warrants it
• in addition to requiring that all tools support the Uniform Output specification, the spec allows tools to use other ways to display Apdex scores

These matters are addressed in Section §5 of the Apdex spec, which is shown in the left-hand column in the following paragraphs. On the right are my proposals for the corresponding portions of Apdex-G.

General Reporting Rules

I have generalized these rules, tightened up the language a little, and removed the last sentence. It has been incorporated into the introduction to section [5.1.1].

[5.1.1] Uniform Output

First some housekeeping: In Apdex-G, I have numbered the principal subsections, and removed the optional subscripted format for output. This format is not applicable in Apdex-G, and will be reviewed later for inclusion in Apdex-R.

To support Apdex-G’s more complex performance zones, comprising an unknown number of performance intervals, we have to abandon the fixed format specification for Uniform Output provided in the current spec. I propose that we replace it with a more flexible comma-separated values (CSV) file. For compatibility with existing Apdex implementations, we can retain the current fixed format as an option in the Apdex-R spec.

There is no universally accepted standard specification for CSV files, but to provide a degree of rigor in the Apdex-G specification, we can incorporate the public domain specification for CSV files published as RFC 4180 RFC4180.

Section [5.1.2] is a placeholder for a new proposal which would permit (or maybe require) tools to supply additional Report Group identifiers in Uniform Output files.

More on Section [5] Reporting …

I debated whether to call this post configurable reporting, because the configuration options that most affect reporting are the selection of Apdex-G performance intervals and thresholds. But the rules for reporting do have to accommodate those options, which in turn forces us to introduce more flexibility throughout the reporting specifications. And we have more decisions to consider. This post has addressed the first half of the reporting area; in a further post, I will draft the remaining paragraphs of Apdex-G section [5]:

[5.1.2] Report Group Section The current spec requires tools to report Apdex scores in Uniform Output format, but does not require any identifiers or context to be reported with those scores. This strikes me as an omission. I will explore the question of what data should be made available to an independent reporting tool that queries an Apdex-based data analysis service, and discuss the relationship of those sections to section [3.2] Report Groups.

[5.2] Sample Size We must consider whether we need any configuration options in the rules for Apdex sample sizes.

[5.3] Describing General Cases This section of the current spec can be retained in Apdex-G with minor modifications.

[5.4] Quality Ratings We must consider whether we need any configuration options for the display of Apdex ratings or grades.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

References

BLOO07

Interval Notation Roberta Bloom, 2007. [pdf]

RFC4180

Common Format and MIME Type for Comma-Separated Values (CSV) Files Y. Shafranovich, 2005. [RFC 4180, The Internet Society, October 2005]

SOSMW1

The Real Number Line, Interval Notation and Set Notation Helmut Knaust, S.O.S. MATHematics [S.O.S. MATHematics Website]

WIKIW1

Interval (mathematics) [Wikipedia]

ZONAW1

Interval Notation Zonaland Education. [Zonaland Website]

I’m writing a series of posts about Generalizing Apdex. This is #11. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

In my previous post in this series, I discussed non-monotonic metrics. These are metrics for which improvement in the metric does not always always correspond to an improvement in quality. To accommodate such metrics, the Apdex-G specification must support six distinct orderings of the three Apdex performance zones (Satisfied, Tolerating, and Frustrated). It should also accept discontinuous zone definitions. For the full story, see Configurable Zone Alignment in Apdex-G,

These requirements introduce two complications, both involving the current spec’s use of just two thresholds, T and F, to define the three Apdex zones. In this post, I will review:

1. The relationship between thresholds and zones
2. The number of thresholds required

The Relationship Between Thresholds and Zones

To explain the first problem, the Table 1 below lists the six possible orderings of three continuous zones, grouping them according to whether low or high data values are preferred. Reading from left to right, each row shows the alignment of the three Apdex performance zones, low to high, within the measurement domain. It also shows where the thresholds T and F would be placed in Apdex-G, to delimit the zones.

Explanatory Notes

1. When the Satisfied Zone is above (to the right of) the Frustrated Zone, higher values are preferred, and vice versa.
2. These numbers refer to the previous post. They were used when describing the different types of metrics.
3. These types were identified in the previous post.
4. The Monotonic Decreasing metric type corresponds to the present Apdex spec.
5. For each metric type, Low Zone < Middle Zone < High Zone
6. I use the notation [T] and [F] to denote the Apdex-G versions of the two thresholds.
7. In each row I placed the thresholds so that they are always adjacent to their “corresponding” zones. [T] is always adjacent to the Tolerating Zone; [F] is always adjacent to the Frustrated Zone. Indeed, in each row there is only one sensible way to position a pair of thresholds having those labels; reversing their positions always separates one of the thresholds from its corresponding zone.
8. In the current Apdex spec, thresholds are defined to fall within the lower of two adjacent zones. For example, T is the upper bound of the Satisfied Zone, and the Tolerating Zone is defined as “Greater than T to F”. There is an argument for reversing this convention, so that a threshold falls within the higher of the two zones. Then T would fall within the Tolerating Zone, which (to me) seems more natural. However, whether or not we retain or change that convention has no bearing on the placement of thresholds in this table.

Problems with Using [T] and [F]

In the current Apdex spec, T and F have natural meanings: above T begins the Tolerating Zone, above F begins the Frustrated Zone. But when we shuffle the zones, and continue to use T and F as delimiters, we can’t hold onto these natural meanings. Even though I have assigned the roles of [T] and [F] in the only possible way in each row, the result still requires their meanings to vary from row to row. Sometimes they mark the preferred end of their corresponding zone, sometimes the opposite.

As a result, if we adopt the approach shown in Table 1, there will be no simple and natural way to describe the roles of [T] and [F] in Apdex-G. The only ways would be either to reproduce Table 1, or my explanation of why [T] and [F] occupy those positions in Table 1, or both. Since our aim is to keep Apdex-G as simple as possible, these are not attractive propositions.

An alternative approach would be to use thresholds called [S], [T], and [F], picking two as appropriate for each metric type, so as to retain the natural relationship between threshold names and roles that exists in the current spec. Table 2 illustrates how this idea might be applied:

This approach has the merit of removing any confusion over threshold roles: above [S] is the Satisfied Zone, above [T] is the Tolerating Zone, above [F] is the Frustrated Zone. That much is natural and easy to specify. However, we have now introduced a complication into the Apdex-G spec: which pair of thresholds is appropriate for any given metric type? To answer that, we would still need to include a version of Table 2 in the spec. And if we have to do that, does this solution even qualify as a “generalization” of Apdex?

Generic Threshold Names

I therefore conclude that it would be better not to introduce the names [S], [T], and [F] when defining the purpose of thresholds in Apdex-G. Instead we should use generic threshold names that contain no implied references to the names of the Apdex performance zones.

An addendum (like Apdex-R, for response time metrics) can still specify domain-specific threshold names (like [T] and [F]) to replace the generic ones. But in Apdex-G, we would use names like [LT] and [HT], signifying the Low and High Threshold respectively, or completely neutral threshold labels like [T1] and [T2]. This last approach seems to work best when describing discontinuous zones, which I discuss next. But first, here’s Table 3, showing this conclusion:

The Number of Thresholds

As I discussed in my previous post, to support typical process control applications, Apdex-G must accommodate bifurcated performance zones. The classic control diagram is symmetric, with both the Yellow/Tolerating and Red/Frustrated Zones being split equally on either side of the central Green/Satisfied Zone.

We can also imagine cases in which measurements of a process to be reported by Apdex are strongly skewed, so that the Frustrated Zone occurs on only one side of the target value. Both positive and negative skews are common in real-world measurement data, when the process or application being measured has a natural lower or upper bound. For example:

• timings are often postively skewed, because times cannot be less than zero, but may have no upper bound
• bandwidth data may be negatively skewed, because network technology imposes an upper limit, but no lower limit

With those possibilities in mind, Table 4 shows the three types of process control metrics. As in Tables 1-3, reading from left to right, each row shows the alignment of the Apdex performance zones, low to high, within the measurement domain. The only difference is that the Frustrated and Tolerating Zones are now split into two regions, one above and one below the Satisfied Zone. And in the case of the Frustrated Zone, one of those regions may be absent.

Explanatory Notes

1. To enumerate the different types of metrics and zone alignments, these numbers continue the sequence begun in Table 3.
2. These names reflect the likely shape of the data distribution for each metric type.
3. The names and colors reflect the example.
4. I use the abbreviation “Thr” for “Threshold” in this table.
5. For each metric type, Low Red Zone < Low Yellow Zone < Green Zone < High Yellow Zone < High Red Zone

Introducing Performance Intervals

There are basically two ways to get from my earlier draft of section [2.5] of Apdex-G to a version that accommodates the nine zone alignments listed in Tables 3 and 4. Either we have to add detail, enumerating the nine metric types, or we have to be less specific about the exact composition of the three performance zones, the number of thresholds, and their names. I much prefer the second approach, because any generic rule specified in Apdex-G can be further refined and focused by a domain-specific Addendum like Apdex-R.

I believe we can retain the essential feature of Apdex–three distinct performance zones–while permitting more flexibility in their alignment. As illustrated in the diagrams below, we can do that by introducing an additional concept: a performance interval, which is defined to be a range of values within the measurement domain that share a common satisfaction or quality level (i.e. ‘Satisfied’, ‘Tolerating’, or ‘Frustrated’). Thresholds now define the boundaries of the performance intervals, not the performance zones, which are now simply a collection (union) of one or more intervals. The Apdex-G diagram illustrates Metric Type #7, Symmetric Process Control measurements.

This is not an entirely new concept. First, the proposed performance intervals can be defined as mathematical intervals, which will be convenient when we work on section [5] Reporting. Second, the current performance zones are already identical to performance intervals, so that the current model is just a trivial case of the general model. But by allowing subdivision of the measurement domain into an arbitrary number of intervals, then collecting those intervals into three performance zones, we permit Apdex-G to support all the different metric types we discussed above–and more if the need arises.

Drafting New Definitions

Below I propose language to describe these concepts. The left hand column shows how the current Apdex spec describes performance zones and thresholds; draft descriptions of the corresponding concepts in Apdex-G appear on the right. My goal in this draft is to define the new rules correctly, so that they are both precise and general. And since the concepts to be described in section [2.5.1] must also be spelled out systematically in section [7], the Glossary, I have chosen to work out the details using the Glossary format. When those definitions are finalized, I will edit them into a narrative format to create section [2.5.1].

Next Steps …

The definitions above may be clumsy in places. Nevertheless, I will add them to the Extensible Apdex Glossary once this is posted. I expect that refinements will emerge as the complete document takes shape. Note also that the list of proposed Apdex-G definitions above appears in an order that eliminates most forward references, to maximize its readability. In the Glossary, of course, terms will be listed be in alphabetical order.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.