Format Tolerance

The meeting began with a number everyone liked and a question nobody could answer. Output had risen. Complaints had fallen. But Mateo Silva asked what had happened to judgement, agency and the ability to continue when the system was unavailable.

Mateo Silva, a multilingual service designer in Wellington, had been asked to decide whether Format Tolerance belonged in a real programme or in another folder of attractive ideas. The immediate problem was practical: Current approaches to format tolerance are fragmented, poorly calibrated or insufficiently measured, making it difficult to distinguish real benefit from substitution, novelty or surveillance effects. People affected by the decision could feel the consequences, but the organisation lacked a defensible way to separate benefit from novelty, substitution or hidden burden.

Instead of beginning with a product demonstration, Mateo Silva put the research question on the wall: “Which input and output adaptations most improve performance for non-dominant formats and languages?” That changed the room. The question did not assume the technology worked. It asked where, for whom and under what conditions it might work. The proposed hypothesis was equally testable: Explicit multi-format training and user-controlled format preferences will reduce exclusion more than automatic normalisation. Around it, the team marked assumptions in amber and evidence in blue. Almost everything was amber.

The first trial was designed as a scenario, not reported as a completed experiment. Participants would move through a baseline condition, an active comparison, a calibrated AI condition and a user-controlled condition. They would also face a failure or handback moment, because a system that works only when nothing goes wrong is not a safe system. The organising lens was the Capability–Access–Representation–Equity model applied to Format Tolerance. Measures would cover immediate performance, understanding, confidence calibration, retained capability, burden, equity and the ability to recover without the tool.

In the scenario, the most revealing moment came after the apparent success. A participant completed the task quickly, then could not explain why the answer was sound. Another worked more slowly but spotted a subtle error and corrected it without assistance. Mateo Silva refused to call either result decisive. The contrast showed why the study needed repeated measures, subgroup analysis, delayed follow-up and participant debriefing rather than a single productivity score. It also exposed the risk of measurement intrusion, which could make a polished outcome look more trustworthy than it was.

The team then tested the human boundary. Participants could reduce assistance, inspect sources, challenge recommendations and withdraw. People using non-dominant formats, different languages or accessibility settings were not treated as exceptions to be averaged away. Their experience was part of construct validity. The success gate remained demanding: Statistically and practically meaningful improvement; preserved or improved human skill and agency; acceptable burden; no disproportionate subgroup harm; reproducible performance; effective handback, correction and recovery. The failure gate was explicit too: No meaningful benefit; gains mask reduced understanding or skill; dependency, fatigue, stress or exclusion exceeds benefit; effects fail to transfer or persist; user control is ineffective; correction or handback fails.

By the end of the workshop, Mateo Silva had no miracle result to announce. That was the point. The evidence state was recorded as “PARTIAL — structured hypothesis seed; empirical evidence not yet bound”. The story could therefore demonstrate the decision pressure and the proposed research design, but it could not claim efficacy, causation or generalisability. The next legitimate step was clear: Complete primary-source review for Format Tolerance; appoint owner; define benchmark, comparison and measures; convene affected-user and expert review; pre-register protocol; establish handback, adverse-effect and recovery tests.

A commercial pathway still emerged, but in the right order. If the research survives pre-registration, ethical review, comparative testing and replication, format tolerance assessment module could become a useful module for public services. Services could include assessment, implementation support, training, certification and assurance. Policy makers could use the resulting measures to ask for evidence of agency, equity, handback and retained skill rather than accepting output uplift alone.

Later, Mateo Silva returned to the original dashboard. It still showed green. Now the team understood that green was not a finding; it was an invitation to investigate. For Format Tolerance, progress would mean building a traceable chain from question to protocol, participant experience, raw data, analysis, limitation and decision. The strongest closing lesson was simple: human capability must remain visible even when the machine makes performance look effortless.