∇Continuous Function

Transformer Systems Lab

Evaluation and falsification station

Carry the Speculative checkpoint into a claim test: choose the slice you expect to break first, reveal aggregate and slice evidence, then save the bounded claim.

station 09local LabCase v1prediction firstclaim boundary
Start EvaluationBack to SpeculationOpen Route Cockpit
Selected claimFaster served long-context behavior is acceptableclaim + scenario + slice + metric + threshold + provenance + counterexample
Aggregatelooks highcan hide the failing slice
Slicemiddle evidencebreaks the headline first
Boundaryscope narrowsnext test is explicit
01Object

one scoped claim card

02Gap

benchmark result is not proof

03Prediction

choose the first break

04Manipulation

slice, metric, threshold, provenance

05Evidence

aggregate, slice, interval, counterexample

06Invariant

claim survives only under its boundary

07Next

assemble a defensible systems claim

Carry-in contract

A claim is not understood until the learner knows how to break it.

The route already produced memory, context, serving, and speculative boundaries. Evaluation converts those observations into one scoped claim with a declared metric, slice, threshold, uncertainty, and next counterexample.

Carried objectSpeculative checkpoint: target correctness and speedup remain separate gates.
PredictionWhich boundary will break first: target correctness, context use, SLO, robustness, or none?
EvidenceAggregate score, slice score, interval, latency, provenance, and counterexample.
Next moveUse the bounded verdict to assemble one defensible systems claim.

Operable Station 09

Evaluation and falsification

Checking local routeNo prediction yet
  1. 01Text to one update
  2. 02Tokens and position
  3. 03Attention routing
  4. 04RoPE phase
  5. 05KV memory
  6. 06Long-context pressure
  7. 07Serving and decoding
  8. 08Speculative decoding
  9. 09Evaluation and falsification
  10. 10Capstone systems claim

Carry-in from Speculative Decoding

A speedup is not a product claim until a slice can try to break it.

Speculative decoding separated target correctness from target-step savings. This station asks whether a bounded claim survives the declared scenario, metric, threshold, provenance, and counterexample.

Prior route objectChecking saved Speculative history.
Serving historyChecking saved Serving history.
Long-context historyChecking saved Long Context history.
KV historyChecking saved KV history.

Selected claim object

Faster served long-context behavior is acceptable

claim:ai-lab/transformers/evaluation-falsification#faster-served-long-context-acceptable

A claim is operable only when the system under test, slice, metric, threshold, provenance, and failure boundary remain attached.

Prediction checkpoint

Which boundary will break this claim first?

Manipulation instrument

Change one claim into a bounded attempt to break it.

This is a deterministic local witness. It does not certify a model; it teaches how claim scope moves when a slice, perturbation, threshold, or provenance flag changes.

Choose a likely failure boundary before evidence.

Route spine

After evaluation, the route stops making headlines and starts making claims.

The learner now has enough pieces to say exactly what survived, what failed, what remains uncertain, and what the next systems claim is allowed to mean.

  1. 01Serving and decoding

    TTFT, TPOT, cache reads, sample support

    ready
  2. 02Speculative decoding

    draft-target speedup and correctness

    ready
  3. 03Evaluation and falsification

    claim, slice, metric, counterexample

    active

Route memory

Save the bounded claim, then carry it forward.

The station writes a browser-local checkpoint with the claim, first prediction, observed boundary, slice, counterexample, uncertainty, and next capstone move.

Open Study Memory