Evaluation: try to break the claim

Why the two counts differ

Both counts use the same statistic and the same observed value, 0.5. They differ in the set of patterns the null hypothesis treats as equally likely. If whole tasks may reverse there are 16, and 5 give T ≥ 0.5: the record itself, three ties that reverse D together with one of A, B or C, and the reversal of D alone, which gives 1. If every row could reverse on its own there would be 65,536; splitting tasks produces many values near zero that the whole-task null never produces, so the observed value looks rarer: 2,517 of 65,536.

Neither count is chosen by which is smaller, and the two are not ordered in general. Here the whole-task count is larger; with two tasks whose differences are (2, −1) and (2, −1) it is smaller, 1/4 against 3/8. The design decides. This one says the tasks were fixed and whole tasks may reverse; it says nothing that lets trials inside a task reverse separately. The identical signs within each task could not establish that, and neither could the shared task labels.

What goes wrong otherwise can be counted. Suppose the whole-task null holds and the record is equally likely to be any of its 16 whole-task reversals. In 5 of them the single-row count is at most 2,517 of 65,536, so a rule that rejects when the counted fraction is 0.05 or less would reject in 5 of these 16 equally likely records, although the null holds for every one of them. The whole-task count never falls below 1/16, so the same rule never rejects with four tasks. (The 0.05 is used only to show the miscalibration; this page sets no threshold.)

Copies and fresh trials

Choose Rows copied. Every recorded row appears twice, the copy under a new trial id. The task means, T and the whole-task count are unchanged, because a copy reverses with its task. Counted as if all 32 rows could reverse independently, the tail becomes 15,033,173 of 2³²2 to the power 32 (≈ 0.00350), about 11 times smaller, from no new observation. A copy cannot reverse apart from its source; in every possible outcome it equals its source. A fresh trial is different: a new run on the same task can show how variable that task is. It is still another trial of the same task, not another task.

The claim, revised

On these four fixed tasks, the variant’s equal-task mean difference is 0.5: three tasks favour it on every trial and one favours the baseline on every trial. Under the stated whole-task null, 5 of 16 reversals give a value at least this large. That is not evidence against the null, and it is not evidence of no difference or of equivalence: with four tasks, no record can give less than 1/16.

Which evidence comes next depends on the target. If the claim stays about these four tasks, more tasks would change the set that T averages rather than inform it. Fresh trials on the same four, meaning new runs rather than copies and collected under a stated repeated-trial design, can estimate each task’s expected difference and its variability more closely. They add no tasks, and the whole-task count still cannot fall below 1/16.

If the claim is meant to reach other tasks, that is a different target. First say which tasks it covers and how new tasks will be chosen from them. Tasks chosen under that design, not more trials of these four, are then the evidence, and the task remains the unit that reverses. Copies add nothing to either question.

Definitions

d_t,i = v_t,i − b_t,i

The paired difference for task t and trial i: variant minus baseline, higher is better. D_t is task t’s list of differences.

m_t = (1/n_t) Σ_i d_t,i

Task t’s mean difference over its n_t rows (here n_t = 4).

T(d) = (1/J) Σ_t m_t

The target, the equal-task mean over J = 4 tasks: (1/4)(1 + 1 + 1 − 1) = 0.5. Equal n_t guarantee that the row mean (1/N) Σ d equals it for any scores. Unequal counts weight the tasks differently, though the two values can still coincide: two tasks with 1 and 2 rows and every d = 1 give 1 for both.

g_εd = (ε₁D₁, …, ε_JD_J), ε ∈ {−1, +1}^J

A whole-task reversal: each sign multiplies a task’s entire list. These form a group of 2^J elements, the identity included. Single-row reversals give each of the N rows its own sign: 2^N elements. A reordering of tasks is in neither group; it leaves T unchanged and reverses nothing.

p = #{ε : T(g_εd) ≥ T(d)} / 2^J

The inclusive upper tail: every group element counts once, the identity and every tie included, even when several give the same value.

When the count is valid. The whole-task null says d and g_εd have the same joint distribution for every ε. Under a null of that kind, a test that rejects when p ≤ α rejects with probability at most α. Ties and the discreteness of the count can make it conservative, so exact counting does not make the rejection rate equal α. Independent task lists, each distributed like its own reversal, are sufficient. They need not share a distribution, and the rows inside a task may depend on each other. Independence is not necessary either. Two weaker statements do not suffice. A zero mean is not symmetry: a difference of +1/2 with probability 2/3 and −1 otherwise has mean zero and is not symmetric. Symmetry of each task separately is not joint invariance: if all tasks share one fair sign, each task is symmetric, but reversing one task alone produces data that never occur.

The validity result is from Hemerik and Goeman (2018; arXiv:1411.7565v3, §2.1 Definition 1 and Theorem 1, and §2.2) and Koning and Hemerik (2024; arXiv:2202.00967v3, §2: Example 1 gives the sign-flipping group, their test is defined with the inclusive ≥ tail, and Theorem 1 bounds its size). Applying it to the whole-task group, and every number on this page, is this page’s own derivation and calculation, not a result reported by either paper.

Background, if needed: Probability basics and Random variables.

from collections import Counter
from fractions import Fraction

# One paired row per (task, trial): (baseline, variant), higher is better.
record = {(task, trial): (0, 1) for task in "ABC" for trial in "1234"}
record |= {("D", trial): (1, 0) for trial in "1234"}


def differences_by_task(record):
    """d = variant - baseline, grouped by task id only, never by similar scores."""
    tasks = {}
    for (task, _trial), (baseline, variant) in record.items():
        tasks.setdefault(task, []).append(variant - baseline)
    return tasks


def parts(tasks, unit):
    """Each reversible unit's share of T = (1/J) * (sum of the task means)."""
    J = len(tasks)
    if unit == "task":
        return [Fraction(sum(d), len(d)) / J for d in tasks.values()]
    return [Fraction(x, len(d)) / J for d in tasks.values() for x in d]


def reference_counts(parts):
    """How many sign patterns give each value of T; 2 ** len(parts) patterns in all."""
    counts = Counter({Fraction(0): 1})
    for c in parts:
        step = Counter()
        for total, k in counts.items():
            step[total + c] += k
            step[total - c] += k
        counts = step
    return counts


def upper_tail(parts):
    """Inclusive: patterns with T(g d) >= T(d), the identity and every tie counted."""
    counts, observed = reference_counts(parts), sum(parts)
    at_least = sum(k for value, k in counts.items() if value >= observed)
    return at_least, counts[observed], 2 ** len(parts)


def show(label, tail):
    at_least, ties, total = tail
    print(f"{label}: {at_least} of {total} patterns, {ties} of them ties")


tasks = differences_by_task(record)
T = sum(parts(tasks, "task"))
print("T =", T)
print({str(v): k for v, k in sorted(reference_counts(parts(tasks, "task")).items())})
show("whole tasks", upper_tail(parts(tasks, "task")))
show("single rows", upper_tail(parts(tasks, "row")))

# Copy every row under a new trial id: no new run, so no new information.
copied = record | {(task, trial + " copy"): pair for (task, trial), pair in record.items()}
copied_tasks = differences_by_task(copied)
print("copied: T =", sum(parts(copied_tasks, "task")))
show("copied, whole tasks", upper_tail(parts(copied_tasks, "task")))
show("copied, rows as if independent", upper_tail(parts(copied_tasks, "row")))

assert T == Fraction(1, 2) == sum(parts(copied_tasks, "task"))
assert upper_tail(parts(tasks, "task")) == upper_tail(parts(copied_tasks, "task")) == (5, 4, 16)
assert upper_tail(parts(tasks, "row")) == (2517, 1820, 2 ** 16)
assert upper_tail(parts(copied_tasks, "row"))[::2] == (15033173, 2 ** 32)