Sir Harold Jeffreys’ (tail area) one-liner: Saturday night comedy (b)

.

This headliner appeared before, but to a sparse audience, so Management’s giving him another chance… His joke relates to both Senn’s post (about alternatives), and to my recent post about using (1 – β)/α as a likelihood ratio--but for very different reasons. (I’ve explained at the bottom of this “(b) draft”.)

 ….If you look closely, you’ll see that it’s actually not Jay Leno who is standing up there at the mike, (especially as he’s no longer doing the Tonight Show) ….

.

It’s Sir Harold Jeffreys himself! And his (very famous) joke, I admit, is funny. So, since it’s Saturday night, let’s listen in on Sir Harold’s howler joke* in criticizing the use of p-values.

“Did you hear the one about significance testers rejecting H₀ because of outcomes H₀ didn’t predict?

‘What’s unusual about that?’ you ask?

What’s unusual is that they do it when these unpredicted outcomes haven’t even occurred!”

Much laughter.

[The actual quote from Jeffreys: Using p-values implies that “An hypothesis that may be true is rejected because it has failed to predict observable results that have not occurred. This seems a remarkable procedure.” (Jeffreys 1939, 316)]

I say it’s funny, so to see why I’ll strive to give it a generous interpretation.

We can view p-values in terms of rejecting H₀, as in the joke: There’s a test statistic D such that H₀ is rejected if its observed value d₀ reaches or exceeds a cut-off d* where Pr(D > d*; H₀) is small, say .025.
           Reject H₀ if Pr(D > d₀; H₀) < .025.
The report might be “reject H₀ at level .025″.
Example:  H₀: The mean light deflection effect is 0. So if we observe a 1.96 standard deviation difference (in one-sided Normal testing) we’d reject H₀ .

Now it’s true that if the observation were further into the rejection region, say 2, 3 or 4 standard deviations, it too would result in rejecting the null, and with an even smaller p-value. It’s also true that H₀ “has not predicted” a 2, 3, 4, 5 etc. standard deviation difference in the sense that differences so large are “far from” or improbable under the null. But wait a minute. What if we’ve only observed a 1 standard deviation difference (p-value = .16)? It is unfair to count it against the null that 1.96, 2, 3, 4 etc. standard deviation differences would have diverged seriously from the null, when we’ve only observed the 1 standard deviation difference. Yet the p-value tells you to compute Pr(D > 1; H₀), which includes these more extreme outcomes! This is “a remarkable procedure” indeed! [i]

So much for making out the howler. The only problem is that significance tests do not do this, that is, they do not reject with, say, D = 1 because larger D values might have occurred (but did not). D = 1 does not reach the cut-off, and does not lead to rejecting H_0. Moreover, looking at the tail area makes it harder, not easier, to reject the null (although this isn’t the only function of the tail area): since it requires not merely that Pr(D = d₀ ; H₀ ) be small, but that Pr(D > d₀ ; H₀ ) be small. And this is well justified because when this probability is not small, you should not regard it as evidence of discrepancy from the null. Before getting to this ….

1.The joke talks about outcomes the null does not predict–just what we wouldn’t know without an assumed test statistic, but the tail area consideration arises in Fisherian tests in order to determine what outcomes H₀ “has not predicted”. That is, it arises to identify a sensible test statistic D.

In familiar scientific tests, we know the outcomes that are ‘more extreme’ from a given hypothesis in the direction of interest, e.g., the more patients show side effects after taking drug Z, the less indicative Z is benign, not the other way around. But that’s to assume the equivalent of a test statistic. In Fisher’s set-up, one needs to identify a suitable measure of accordance, fit, or directional departure. Improbability of outcomes (under H₀) should not indicate discrepancy from H₀ if even less probable outcomes would occur under discrepancies from H₀. (Note: To avoid confusion, I always use “discrepancy” to refer to the parameter values used in describing the underlying data generation; values of D are “differences”.)

2. N-P tests and tail areas: Now N-P tests do not consider “tail areas” explicitly, but they fall out of the desiderata of good tests and sensible test statistics. N-P tests were developed to provide the tests that Fisher used with a rationale by making explicit the alternatives of interest—even if just in terms of directions of departure.

In order to determine the appropriate test and compare alternative tests “Neyman and I introduced the notions of the class of admissible hypotheses and the power function of a test. The class of admissible alternatives is formally related to the direction of deviations—changes in mean, changes in variability, departure from linear regression, existence of interactions, or what you will.” (Pearson 1955, 207)

Under N-P test criteria, tests should rarely reject a null erroneously, and as discrepancies from the null increase, the probability of signaling discordance from the null should increase. In addition to ensuring Pr(D < d*; H₀) is high, one wants Pr(D > d*; H’: μ₀ + γ) to increase as γ increases.  Any sensible distance measure D must track discrepancies from H₀.  If you’re going to reason, “the larger the D value, the worse the fit with H₀,” then observed differences must occur because of the falsity of H₀ (in this connection consider Kadane’s howler).

3. But Fisher, strictly speaking, has only the null distribution, along with an implicit interest in tests with sensitivity toward implicit departures. To find out if H₀ has or has not predicted observed results, we need a sensible distance measure. (Recall Senn’s post: “Fisher’s alternative to the alternative”, just reblogged yet again.**)

Suppose I take an observed difference d₀ as grounds to reject H₀ on account of its being improbable under H₀, when in fact larger differences (larger D values) are more probable under H₀. Then, as Fisher rightly notes, the improbability of the observed difference was a poor indication of underlying discrepancy. This fallacy would be revealed by looking at the tail area; whereas it is readily committed with accounts that only look at the improbability of the observed outcome d₀ under H₀.

4. Even if you have a sensible distance measure D (tracking the discrepancy relevant for the inference), and observe D = d, the improbability of d under H₀ should not be indicative of a genuine discrepancy, if it’s rather easy to bring about differences even greater than observed, under H₀. Equivalently, we want a high probability of inferring H₀ when H₀ is true. In my terms, considering Pr(D < d*;H₀) is what’s needed to block rejecting the null and inferring alternative H’ when you haven’t rejected it with severity (where H’ and H₀ exhaust the parameter space). In order to say that we have “sincerely tried”, to use Popper’s expression, to reject H’ when it is false and H₀ is correct, we need Pr(D < d*; H₀) to be high.

5. Concluding remarks:

The rationale for the tail area for Fisher, as I see it, is twofold: to get the right direction of departure, but also to ensure Pr(test T does not reject H₀; H₀ ) is high.

If we don’t already have an appropriate distance measure D, then we don’t know which outcomes we should regard as those H₀ does or does not predict–so Jeffreys’ quip can’t even be made out. That’s why Fisher looks at the tail area associated with any candidate for a test statistic. Neyman and Pearson make alternatives explicit in order to arrive at relevant test statistics. (For N-P, the “tail area” falls out of the rejection region; they actually criticize Fisher for not justifying his use of it.)

If we have an appropriate D, then Jeffreys’ criticism is equally puzzling because considering the tail area does not make it easier to reject H₀ but harder. Harder because it’s not enough that the outcome be improbable under the null, outcomes even greater must be improbable under the null. And it makes it a lot harder (leading to blocking a rejection) just when it should: because the data could readily be produced by H₀ [ii].

Either way, Jeffreys’ criticism, funny as it seems, collapses.

When an observation leads to rejecting the null in a significance test, it is because of that outcome—not because of any unobserved outcomes. Considering other possible outcomes that could have arisen is essential for determining (and controlling) the capabilities of the given testing method. In fact, understanding the properties of our testing tool T just is to understand what T would do under different outcomes, under different conjectures about what’s producing the data.

Feb 22 addition(b): The relevance to Senn’s post is pretty obvious, as considering the tail area is a way Fisher ensures a sensible (directional) test statistic. The connection to my post about using  (1 – β)/α as a likelihood ratio is less direct.  If you use this in a Bayesian computation you’ll get a higher posterior probability for the non-null than if you used the observed data point. In this case, considering the tail area (for the power) really would make it easier to find evidence against the null. But it comes from using these terms in an entirely unintended way. Moreover, power is a lousy “distance” (between data and hypothesis) measure. That’s what I was trying to bring up in the post that went off topic.