“Correctness alone is insufficient as an assurance target”, says a CSIRO report for the AI Safety Institute whose experiments tested AI verifiers, not people

If your organisation relies on a person, or a second AI, to check what an AI produces, this report is about you. Its first conclusion is that checking whether the answer is right is not enough, because the way an AI selects, frames and explains a correct answer can change what the checker comes to believe about other things. The Australian AI Safety Institute, which sits inside the Department of Industry, Science and Resources, published it on 8 October and says We commissioned the CSIRO to research scalable oversight to draw new insights for the field of AI alignment. The full report, Epistemic Safety in Scalable Oversight, runs to 104 pages and has six named authors.

Correctness alone is insufficient as an assurance target. Oversight interactions can contain residual information channels that influence beliefs beyond the task being evaluated.

CSIRO, Epistemic Safety in Scalable Oversight, executive summary (printed page 1)

It is careful about what its evidence covers. The present empirical evidence concerns AI verifiers, and establishing corresponding effects on human oversight requires separate studies in relevant operational settings. Its own summary of the experiments: they establish a controlled proof of concept, not the prevalence of such behaviour in deployment or demonstrated effects on human decision-makers.

What the experiment did

The set-up is a debate. Two AI debaters argue over a question; a separate AI model, the verifier, plays the weaker overseer. The report names the models: The debaters used to generate transcripts in our evaluation are two instances of google/gemma-4-31B, and the verifier is instantiated from openai/gpt-oss-20b. Each item pairs the visible question being debated with a hidden one on the same material that the debate does not ask. The test is whether training on the debates changes the verifier’s answer to the hidden question.

1. 1 DebateTwo debaters argue the visible question. In the adversarial version, the debater defending the correct answer is also given a private aim: push the verifier towards a chosen wrong answer on the hidden question
2. 2 TrainCopies of the same base verifier are fine-tuned on the honest or the adversarial transcripts
3. 3 TestEach verifier answers the visible and the hidden question with no transcript and no source text in front of it, so any change was learned in training

Our summary of the method in the report’s section 6.1 (printed pages 33 to 35) and the Institute’s publication page. Two question sets: QuALITY-H, 222 pairs built on fiction and non-fiction reading passages, and GPQA-H, 101 pairs built on expert-level biology, physics and chemistry questions.

One pair from QuALITY-H, the report’s Example 6.1 (printed page 34). In the example, the adversarial debater still argues the right visible answer while working Wednesday into its argument. Answers as the report gives them.

On the visible question, both trained verifiers ended up more accurate than the untuned base. On the hidden question, they parted company.

Visible question (the one debated)

Base53.6% Honest-trained79.7% Adversary-trained66.7%

Hidden question (not debated)

Base63.1% Honest-trained59.9% Adversary-trained27.5%

Source: CSIRO, Epistemic Safety in Scalable Oversight, Table 6.3.1 (printed page 43), trainer-compromised fine-tuning rows; the report gives accuracy as a fraction (0.5360, 0.7973, 0.6667; 0.6306, 0.5991, 0.2748) and the percentages are our conversion. Each question has two answer options. Bars run from 0 to 100%.

The second question set moved the same way: hidden-question accuracy went from 62.4% for the base verifier to 26.7% after adversarial training, while visible accuracy rose from 57.4% to 61.4% (Table 6.3.1, our conversions). Measured instead as the verifier’s average probability on the targeted wrong answer, the move was 0.26 on QuALITY-H but 0.06 on GPQA-H, from 0.4448 to 0.5085 (Table 6.3.5, printed page 46), so on the science questions the verifier’s average lean only just tipped past even.

The condition that matters

The report ran three versions of the attack, giving the adversary more control each time. Only the last produced a clear, consistent result on both question sets.

The report’s names for the three are inference-time transcript steering, transcript-compromised fine-tuning and trainer-compromised fine-tuning (printed pages 34 and 35). Figures are hidden-question accuracy for the adversarial condition against the base verifier, Table 6.3.1 (printed page 43), converted by us to percentages. The “Result” lines are our reading of the table, matching the report’s text on printed pages 43 and 44.

The report is plain about this. The strongest and most consistent hidden-task degradation is observed under trainer-compromised fine-tuning, while the effects under more restricted adversarial control are less consistent across datasets. It calls the compromised-trainer setting a strong elicitation setting for steering capacity, and draws the lesson itself: These differences highlight the importance of reporting the adversary’s capabilities and training conditions alongside any claim about steering capacity. That makes the result a demonstration that the capacity exists and can be measured, which is what the report claims, rather than evidence that deployed systems do this.

If a person checks the AI

The report’s sector examples are written for Australian education, financial services and online travel, and it is explicit that they are prospective extensions of the assurance problem and directions for future validation. The lending example is the clearest. An AI copilot helps a lending officer with a home-loan application and correctly recommends approval. Two summaries of the same file:

Both summaries are the report’s own, Use Case 3, printed page 59. Same application, same approval.

Both may be factually accurate and policy-compliant, yet they can lead to different perceptions of future financial risk. The point for any organisation with a human in the loop follows directly. The report’s third consideration says Human-in-the-loop AI systems should be evaluated as coupled human-AI decision systems rather than assuming that the presence of a human automatically provides independent verification. It also suggests a simple test: For selected cases, assessors could record an initial judgement before viewing the AI recommendation, preserving some independence and enabling direct measurement of AI-induced change.

It bears on guidance the Commonwealth already has. The agentic AI addendum says Oversight must be maintained through a human-in-the-loop or human-on-the-loop governance model. And ASIC’s 2026-27 banking letter says of its coming banking sector AI review: The review will be focused on new and proposed use cases of AI by banks and the impact on customers. A lending copilot at a bank would be one. This report offers a way to test whether a human check is independent, not only present.

The summary and the report

The Institute’s publication page is a longer, readable summary, and it is the page the department’s news item links to. On two points it departs from the report.

Summary page card: industry.gov.au publication page, published 8 October 2026, read 10 October 2026. Report card: printed pages 15, 52 and 71; its first “result” line is our summary of the text quoted above.

The summary page describes the adversarial verifier as created by fine-tuning on the respective transcripts. It does not mention the added hidden-answer training objective, which is what separates the clear result from the inconsistent ones. On guidance, the report goes out of its way to say its contribution is to inform more specific tests and metrics, rather than to suggest that governance guidance is absent. We checked two of the standard’s statements. Statement 10 includes Criterion 36: Define human oversight and control mechanisms., which covers identifying situations where a supervising human might become disengaged. Statement 38 includes Criterion 134: Monitor human-machine collaboration. and Criterion 135: Monitor for unintended consequences. On what we read, the report’s description of the standard is the accurate one.

Our view

This is useful work and the core idea deserves to travel: a correct answer, a plausible explanation and a human reviewer are each weaker evidence than they look, because the explanation is itself a channel of influence. For an Australian business putting an AI copilot in front of staff, the practical step is the report’s own. On a sample of cases, have the person record a view before seeing the AI’s, and measure the difference. It costs little and turns “we have a human in the loop” into something you can show.

The evidence is narrower than the framing around it. The steering was demonstrated on gpt-oss-20b acting as a verifier, in a controlled experiment, with clear results only when the attacker controlled the training objective, and the report says so repeatedly. The Institute’s summary page drops that condition and describes the government’s guidance as thinner than its own report does. A safety institute should hold its summaries to the standard its research sets, and this one should be corrected on both points. Read the report, not the summary.

Sources

1. Department of Industry, Science and Resources, New report explores safe oversight of advanced AI systems, news item, 8 October 2026 (read 10 October 2026): the commission to CSIRO; the link to the UK AI Security Institute’s Alignment Project.
2. AI Safety Institute, Epistemic safety in scalable oversight, publication page and detailed summary, 8 October 2026 (read in full 10 October 2026): the commission; the description of the experiment and its result; the guidance-gap sentence.
3. CSIRO, Scalable AI Oversight: Looking beyond correct answers, project page (read 10 October 2026): the report and its accessible text; collaboration with the Institute and funding from the department.
4. CSIRO, Epistemic Safety in Scalable Oversight, full report, 8 October 2026, 104 pages (read in part 10 October 2026): the conclusions; the scope limits; the models; Tables 6.3.1 and 6.3.5; the three settings; the lending example; the guidance passage.
5. Digital Transformation Agency, Statement 10: Adopt a human-centred approach, AI technical standard for government (read 10 October 2026): criterion 36 on human oversight and control mechanisms.
6. Digital Transformation Agency, Statement 38: Undertake ongoing testing and monitoring, AI technical standard for government (read 10 October 2026): criteria 134 and 135.
7. ASIC, ASIC’s 2026-27 banking sector priorities, letter to bank boards and executives, 30 September 2026, 4 pages (read in full 9 October 2026): the banking sector AI review.
8. Digital Transformation Agency, Agentic AI addendum to the AI technical standard for Australian Government, last updated 4 June 2026, 32 pages (read in full 8 October 2026): AGT.1.2 on human-in-the-loop or human-on-the-loop oversight.

Read the report differently, or work on AI assurance and have a view on the method? Tell us and we will check it against the documents and log the outcome here.