Tutorial: adaptive stopping

A fixed trial budget has to be set in advance, and set for the least precise participant you can imagine — so everyone else is over-tested. If inference costs a fraction of a millisecond you can instead refit between trials and stop each participant when their own estimate is precise enough.

In simulation this saves about 74% of trials at a target of 0.35 on the posterior SD, and the saving grows as the target tightens. Nobody is measured less precisely; the saving comes entirely from the participants who never needed the full budget.

You can try this without writing anything: the live task runs exactly this loop in your browser.

State the target

A target is a statement about precision, not about trial count. Either a maximum posterior SD on some parameters, or a minimum confidence in a construct.

import grintools as gt

# stop when the correlations are known to within an SD of 0.35
decision = gt.stop_on_precision(fit, sd_max=0.35, params=["rho"])
decision.stop           # True / False
print(decision.summary())
#   stop = True  (combine='all')
#       sd:rho_0         0.171 vs 0.350  [met]

decision.checks         # one dict per target: value, threshold, met, reachable
decision.blocked_by     # the checks that are not met yet

library(grin)

# stop when the correlations are known to within an SD of 0.35
decision <- grin_stop_on_precision(fit$result, sd_max = 0.35,
                                   params = c("rho"))
decision$stop          # TRUE / FALSE
decision$checks        # one row per target: value, threshold, met, reachable
decision$blocked_by    # the targets not met yet

// In a browser task, the same rule is three lines: refit, average the
// posterior SDs you care about, compare to your target.
const out = await model.predict({ counts, trials });
const rhoSD = out.std.slice(8, 12);
const stop = rhoSD.reduce((a, b) => a + b, 0) / 4 <= 0.35;

Put it in the trial loop

Accrue in blocks rather than single trials. The posterior barely moves on one response, and every stimulus needs a few observations before the fit means anything.

import numpy as np, grintools as gt

counts = np.zeros((4, 4), dtype=int)
BLOCK, MIN_TRIALS, MAX_TRIALS = 16, 40, 400

for block in range(MAX_TRIALS // BLOCK):
    for _ in range(BLOCK):
        s = present_random_stimulus()      # your experiment
        r = collect_response()             # your experiment
        counts[s, r] += 1

    n = counts.sum()
    if n < MIN_TRIALS:
        continue                           # too early to ask

    fit, con = gt.infer(counts)
    if gt.stop_on_precision(fit, sd_max=0.35, params=["rho"]).stop:
        print(f"target reached after {n} trials")
        break

library(grin)

counts <- matrix(0L, 4, 4)
BLOCK <- 16; MIN_TRIALS <- 40; MAX_TRIALS <- 400

for (block in seq_len(MAX_TRIALS %/% BLOCK)) {
  for (i in seq_len(BLOCK)) {
    s <- present_random_stimulus()   # your experiment
    r <- collect_response()          # your experiment
    counts[s, r] <- counts[s, r] + 1L
  }

  n <- sum(counts)
  if (n < MIN_TRIALS) next

  fit <- grin_infer(counts)
  d <- grin_stop_on_precision(fit$result, sd_max = 0.35, params = c("rho"))
  if (isTRUE(d$stop)) {
    message("target reached after ", n, " trials")
    break
  }
}

import { loadModelCached } from "./assets/js/grin-model.js";

const model = await loadModelCached("./assets/models/cm");
const counts = [[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]];
const BLOCK = 16, MIN_TRIALS = 40, MAX_TRIALS = 400;
let n = 0;

while (n < MAX_TRIALS) {
  for (let i = 0; i < BLOCK; i++) {
    const s = presentRandomStimulus();     // your experiment
    const r = await collectResponse();     // your experiment
    counts[s][r] += 1;
    n += 1;
  }
  if (n < MIN_TRIALS) continue;

  const trials = counts.map((row) => row.reduce((a, b) => a + b, 0));
  const out = await model.predict({ counts, trials });
  const rhoSD = out.std.slice(8, 12);
  if (rhoSD.reduce((a, b) => a + b, 0) / 4 <= 0.35) break;
}

Always set a ceiling

Some participants will not reach any given target, because the design does not contain the information — particularly for the correlations, and particularly at high accuracy. Without a maximum you are running an experiment with no defined end.

Report the ceiling and how many participants hit it. A saving computed over a cohort where the slowest were quietly dropped is not a saving.

Two caveats worth knowing

Targets on correlations inherit a calibration issue. The correlation intervals run narrower than nominal, so a target on rho is met slightly sooner than it should be. Either pass calibrated=True / calibrated = TRUE, or set the target a little tighter than you actually want.

Stopping on a construct probability is different from stopping on precision. A probability near one half can mean the data are uninformative, and more trials will fix that — or that the true answer sits near a boundary, and they will not. A precision target is the safer default.

Next: steer stimulus levels into the informative range, or go back to all tutorials.

State the target

Put it in the trial loop

Always set a ceiling

Two caveats worth knowing