Human Oversight Design

1. Hypothesis

Risk-tiered sampling plus anomaly triggers will outperform universal manual review.

2. Experiment design

Design: workflow simulation and field pilot with decision audit; baseline, active comparison, calibrated AI condition, user-controlled condition, failure or handback scenario and delayed follow-up Methods: workflow experiments; task-allocation trials; simulation; incident analysis; ethnography; decision audits; controlled handover tests; pre-registered analysis; active comparison; subgroup and accessibility analysis; delayed retention or longitudinal follow-up; adverse-effect capture; reproducibility testing; participant debrief Independent variables: sampling rate; risk tier; anomaly threshold; reviewer load Dependent variables: failure detection; false alarms; review burden; harm Confounders: baseline capability; prior exposure; age; motivation; context; technology access; implementation fidelity Measures: detection recall; precision; review minutes; escaped incidents Success criteria: Statistically and practically meaningful improvement; preserved or improved human skill and agency; acceptable burden; no disproportionate subgroup harm; reproducible performance; effective handback, correction and recovery. Failure conditions: No meaningful benefit; gains mask reduced understanding or skill; dependency, fatigue, stress or exclusion exceeds benefit; effects fail to transfer or persist; user control is ineffective; correction or handback fails.

3. Seed result / current evidence

DEFENSIBLE SEED RESULT — NON-EMPIRICAL. The current evidence supports Human Oversight Design as a testable research proposition. Problem basis: Current approaches to human oversight design are fragmented, poorly calibrated or insufficiently measured, making it difficult to distinguish real benefit from substitution, novelty or surveillance effects. Directional expectation: If the hypothesis is supported, the intervention condition should improve failure detection; false alarms; review burden; harm while avoiding material deterioration in independence, confidence calibration or delayed performance. Proposed observations: detection recall; precision; review minutes; escaped incidents. Seed data profile: Evidence Strength 10/100; Confidence 25/100; Maturity 20/100; Overall Health 36/100; Novelty 78/100; Strategic Importance 95/100. Evidence boundary: No validated results yet.; experiments 0, studies 0, participants 0. This is suitable for protocol formation and baseline comparison, not as a finding of effect.

4. Seed conclusion

DEFENSIBLE SEED CONCLUSION — PROVISIONAL. Human Oversight Design warrants structured testing because the CSV identifies a defined problem, falsifiable hypothesis, measurable outcomes and relevant literature foundations. The present position is that “Risk-tiered sampling plus anomaly triggers will outperform universal manual review.” is plausible and decision-relevant, but unvalidated. Proceed to controlled testing against the stated success and failure conditions. Confirm, narrow or reject this seed after effect sizes, uncertainty, subgroup outcomes, adverse effects, persistence and handback performance are observed.

Prior-art search performed before starting

PRIOR-ART SEED BASELINE — PARTIAL. The CSV records these literature domains: Joint cognitive systems; levels of automation; team cognition; high-reliability organisations; human-in-the-loop safety research. It also records: NIST AI RMF — https://www.nist.gov/itl/ai-risk-management-framework; ISO human-centred AI — https://www.iso.org/; OECD AI Principles — https://oecd.ai/; UNESCO AI Ethics — https://www.unesco.org/. Evidence register status: “Seeded; authoritative source register refreshed; empirical evidence not yet ingested”. This is defensible as a starting prior-art inventory, but not as proof of a completed systematic search because search dates, databases, exact queries, reviewer, result counts, screening decisions, claim mapping and a replayable receipt are absent.

Prior-art material named: Existing literature: Joint cognitive systems; levels of automation; team cognition; high-reliability organisations; human-in-the-loop safety research. References: NIST AI RMF — https://www.nist.gov/itl/ai-risk-management-framework; ISO human-centred AI — https://www.iso.org/; OECD AI Principles — https://oecd.ai/; UNESCO AI Ethics — https://www.unesco.org/

Critical gap / next action

Create and attach a dated prior-art search log; lock the protocol; execute the proposed study; link raw data and analysis; then replace the results and conclusion placeholders with evidence-bounded findings.