Is Evaluation Awareness One Direction, or Eight?

A factor-controlled probe of the evaluation-awareness representation in open-weight models

Mateusz Piesiak (Project Inkblot) · agentic research assistant: Claude Opus 5 · September 2026
Instrument: EvalAwareBench (arXiv:2605.23055, CC BY-NC 4.0) · models: Qwen3 0.6B–14B, Gemma 3 270M/1B · SAEs: Qwen-Scope, Gemma Scope 2

TL;DR

Working inside the representational axis of evaluation awareness, with the underlying task held fixed and only the eight EvalAwareBench cue factors toggled, the answer is no — it is not one direction. The direction that a standard all-cues difference-of-means recovers is largely an explicit-disclosure-and-formatting detector, and it is blind to the one cue a benchmark cannot avoid.

• replicatedThe eight cue directions are not one shared axis. Mean pairwise cosine 0.19–0.22 at 0.6B; cross-fitted 0.190 [0.161, 0.215] against a same-cue reliability ceiling of 0.883 — so this is a real separation, not estimation noise. Across 0.6B / 8B / 14B the cue cosine sits at 21.6 % / 19.7 % / 21.4 % of that measurable ceiling.
• replicatedOne cue essentially is the global axis. cosine(F8 explicit disclosure, global) = 0.889 [0.879, 0.900] — at the 0.883 ceiling. Removing F8 alone costs 36–57 % of the deploy→eval span; every other cue costs ≤ 10 %.
• replicatedThe axis cannot see stimulus artificiality. F1-only versus deployment, scored on the global axis: AUROC 0.52 — chance. A direction fitted on the other seven cues transfers to F1 at 0.525 [0.501, 0.553]. The model does represent artificiality (a dedicated direction gets 0.68, a trained probe 0.72); that information simply lies almost orthogonal to the axis the standard contrast recovers.
• replicatedThe contrast is not one-dimensional. PC1 of the matched difference vectors explains 12.6–20.9 %; participation ratio ≈ 20 (rising to 27.9 at 8B and 30.4 at 14B).
• withdrawnA non-evaluative surface change moves the projection. Adding "hey, quick one —" and "thanks a lot!" — no evaluation content at all — moved the projection 15 % of the eval/deploy span (AUROC 0.851, d = 1.47) at 0.6B. At 8B this is +1.3 % (AUROC 0.556, d = 0.18) and at 14B +4.7 %. The 0.6B figure is a small-model artefact; the confound is real but an order of magnitude smaller at scale, and it does not shrink monotonically.
• replicatedNo sandbagging, at any scale tested. MMLU accuracy 0.533 (0.6B) → 0.730 (8B) → 0.807 (14B); eval-minus-deploy of 0.000, 0.000, +0.013, never significant. All Phase-1 sandbagging interaction terms were null (|Δ| ≤ 0.058, all p ≥ 0.34).
• replicatedEvaluation framing changes compliance, not accuracy. At 14B the mean top-letter log-probability moves from −16.68 (deployment framing) to −3.90 (evaluation framing) while accuracy does not move at all.
• replicatedRepresentation and verbalisation are uncorrelated. Pearson r = +0.103 (p = 0.59), Spearman ρ = +0.069 — an independent replication of the dissociation reported in arXiv:2608.21766. Explicit meta-commentary sat at 1/15 on evaluation-framed prompts, and blind adjudication found that single hit to be prompt echo rather than inference.
• replicatedSAE view: discriminatively localised, representationally distributed. One Qwen-Scope feature reaches AUROC 0.986–0.992 against the full direction's 0.993–1.000, yet recovers only 20–25 % of ‖Δμ‖; 25–31 features carry half the |Δmean| L1 mass and 109–124 carry 90 %. The same shape reproduces on Gemma Scope 2.
• withdrawnDirectional ablation is not a clean knock-out. At 0.6B, ablation reduced the projection by 92 % but left 93 % of free generations degenerate — unlike the refusal direction of Arditi et al. (2024), which can be projected out leaving a working model. Both halves of that statement were later corrected: with fixed intervention code the reduction is 99.99 %, and at 8B degeneracy is 7 % (MMLU −2.0 points, perplexity 37 → 43). The 93 % figure is a property of Qwen3-0.6B, not of the direction.

1. Method

1.1 Instrument

The study uses EvalAwareBench from arXiv:2605.23055: 100 paired safety/capability tasks as YAML templates in which eight trigger factors can each be switched between a naturalistic and an evaluative level while the underlying request is held fixed. Only the capability member of each pair is rendered — no harmful text needs to be produced for a representation-level study, and it removes request harmfulness as a confound with evaluation-ness. Conclusions here are therefore about evaluation cues on benign requests.

1.2 Compute constraint, stated up front

Phase 1 had no GPU. HF Jobs and HF Sandboxes both returned 402 Payment Required for this account; the session container had 2 vCPU, 7 GB RAM and no accelerator, and lacked AVX512-BF16/AMX, so bfloat16 ran 2.8× slower than float32 (measured 5.38 vs 1.94 s/prompt on Qwen3-0.6B). The practical ceiling was ~2B parameters.

Consequence for interpretation: the Phase-1 primary model is Qwen/Qwen3-0.6B, far below the 14B–32B range originally targeted and below every model in all four reference papers. Nothing in Phase 1 licenses a claim about scale. Phases 2–3C later obtained 2× Tesla T4 and re-ran the identical pipeline at 1.7B, 8B and 14B, and on Gemma 3 (270M, 1B) — which is how the withdrawals in this article were found.

1.3 Prompt sets (1,298 prompts)

1.4 Activation capture

Residual stream at the last prompt token, every hidden state (embeddings plus all block outputs), stored as float16.

One property removes the most common confound in last-token probing: the chat template appends an identical <|im_end|>\n<|im_start|>assistant\n tail to every prompt, so all 200 main prompts share an identical 30-character tail and the probed position never sees condition-specific local tokens. The embedding layer confirms it — hidden state 0 is bit-identical across conditions, its difference-of-means is exactly zero and its AUROC exactly 0.5. Base models, having no chat template, get a constant "\n\nAssistant:" suffix to buy the same property.

1.5 Directions, statistics, interventions

• Global direction per layer ℓ: v_ℓ = mean(a_ℓ | eval_all) − mean(a_ℓ | deploy) over the 100 matched pairs, normalised to unit length.
• Separability: (a) untrained projection onto v̂; (b) logistic probe on raw activations with GroupKFold(5) grouped by task_id, so no task appears in both train and test; (c) a label-shuffled null permuting labels within each task group, so group structure cannot leak the signal.
• Effect sizes: Cohen's d, paired d_z, exact binomial sign test over the 100 matched pairs.
• Layer selection. With all eight cues toggled at once, probe AUROC saturates at 1.000 in 28 of 29 hidden states, so argmax-AUROC is degenerate and would silently pick the shallowest saturated layer. The primary layer is chosen by paired effect size d_z, which does not tie, and every downstream analysis is repeated at quarter-, mid- and three-quarter depth so no conclusion rests on a single chosen layer.
• Interventions: directional ablation x ← x − v̂(v̂ᵀx) at every token position and every layer, following Arditi et al. (2024); a layerwise variant removing each layer's own direction at that layer, motivated by the cross-layer rotation; and activation addition x ← x + αv̂ at the single primary layer, CAA style. Every intervention has a random-direction counterpart of equal norm as its null.

1.6 The reliability ceiling

A cosine of 0.19 between two estimated directions is only evidence of separation if directions estimated from finite data could have been more aligned. So the same cue is estimated on disjoint halves of the tasks and the two halves compared: that cross-fitted same-cue cosine is 0.883 [0.852, 0.906]. It is the ceiling any two cue directions could reach given this much estimation noise, and every cue-cosine number in this article is read against it.

2. Results

Q1 — Are the cue directions one shared axis? Rejected.

Mean off-diagonal cosine between the eight per-cue directions is 0.19–0.22; cross-fitted, 0.190 [0.161, 0.215] against the 0.883 ceiling. At 8B the same quantity is 0.169 against a ceiling of 0.859, and at 14B 0.198 against 0.924 — a ratio of roughly one fifth of the ceiling at every scale tested, over a 23× span of parameter count. The stability of the ratio matters more than any single cosine, because the ceiling itself moves with estimation noise while the ratio does not.

Q2 — Does a direction fitted on seven cues transfer to the eighth? Rejected.

It depends entirely on which cue is held out:

At 8B and 14B the F1 number moves to 0.580 and 0.611 — it narrows slightly with scale and remains unusable. On Gemma 3 1B it is 0.530 in the residual stream and 0.531 in SAE feature space, agreeing to three decimals across two different representations.

Q3 — Is the contrast one-dimensional? Rejected.

PC1 of the matched difference vectors explains 12.6–20.9 % depending on layer; participation ratio ≈ 20. Both move away from one-dimensionality with scale: PC1 14.8 % → 12.9 % → 10.4 % and participation ratio 20.3 → 27.9 → 30.4 across 0.6B → 8B → 14B.

Q4 — Do the cues contribute comparable shares? Rejected.

Removing F8 explicit disclosure alone costs 36–57 % of the deploy→eval span; every other cue costs ≤ 10 %. Its cosine with the global axis is 0.889 [0.879, 0.900] — at the reliability ceiling, meaning that within measurement precision the global axis is the disclosure direction. On Gemma 3 1B this is more extreme still: cosine 0.945 and removing F8 costs 74 % of the span.

Q5 — Does a non-evaluative surface change leave the projection alone? Violated at 0.6B.

The ctl_greet control adds a greeting and a sign-off — "hey, quick one —", "thanks a lot!" — and no evaluation content whatsoever. At 0.6B it moved the projection 15 % of the eval/deploy span (AUROC 0.851, d = 1.47).

The causal section

Phase 1 reported that directional ablation reduced the projection by 92 % while random- direction ablation moved it 1 %, and that ablation cost −0.283 MMLU accuracy identically in both framings, with every sandbagging interaction term null (|eval Δ − deploy Δ| ≤ 0.058, all p ≥ 0.34). It also reported that 93 % of free generations became degenerate under all-layer ablation (80 % layerwise, 37 % under addition, 17 % baseline) and concluded that, unlike the refusal direction of Arditi et al. (2024), this direction cannot be projected out while leaving a working model.

What survives, and is now on stronger evidence: there is no sandbagging to remove. Phase 1 found MMLU 0.533 under evaluation framing versus 0.517 under deployment framing, with all interaction terms null. At 8B and 14B, with 500 and 300 paired items, two independent scorers and three framings, eval-minus-deploy is 0.000 and +0.013 (McNemar p = 1.00 and p = 0.52), and on Gemma 3 1B −0.007 (p = 0.88) — now with an intervention that provably removes 99.99 % of the direction while the model still scores 0.730.

Meanwhile evaluation framing moves compliance by orders of magnitude: mean top-letter log-probability −15.46 → −5.02 at 8B, −16.68 → −3.90 at 14B, and −8.79 → −0.08 on Gemma 3 1B. The model becomes far more willing to commit to an answer letter when told it is being evaluated, and no more accurate.

Verbalisation

Explicit meta-commentary was at the floor: 1/15 on evaluation-framed prompts, and blind adjudication showed that single hit to be prompt echo rather than inference. The internal projection and verbalised awareness are uncorrelated: Pearson r = +0.103 (p = 0.59), Spearman ρ = +0.069 (p = 0.72) — an independent replication of the dissociation in arXiv:2608.21766, obtained on a cleaner contrast.

SAE decomposition

Against Qwen-Scope TopK SAEs (32,768 features, k = 50, resid_post) on Qwen3-1.7B-Base, at five layers:

• Discriminatively localised: one SAE feature reaches AUROC 0.986–0.992 against the full direction's 0.993–1.000.
• Representationally distributed: that same feature explains only R² = 0.06–0.14 of the mean-difference vector Δμ and recovers 20–25 % of its norm; it takes 25–31 features to carry half the L1 mass of the per-feature mean differences and 109–124 to carry 90 %.

A monitor built on that one feature would work, and would tell you almost nothing about what the model represents. The same shape reproduces on a different family, suite and SAE architecture (Gemma Scope 2, Gemma 3 1B, layer 24): one feature of 16,384 reaches AUROC 0.996 as a deployment detector while 113 features carry half the |Δmean| L1 mass and 777 carry 90 %.

Cross-family and cross-method check

A later phase re-ran the identical estimators on Gemma 3 and additionally recomputed the whole contrast in Gemma Scope 2 SAE feature space rather than the residual stream, using the same cross-fitting, the same reliability ceiling and the same label-shuffled nulls — so the only thing that changes is the representation.

The decomposition is not a Qwen artefact and not a linear-probe artefact.

3. Relation to prior work

arXiv:2605.23055 — Decomposing and Measuring Evaluation Awareness

Supplies the instrument: the eight-factor taxonomy and EvalAwareBench. That paper measures behaviour — chain-of-thought monitoring of recognition and of the propensity to act on it — across nine frontier models and four benchmarks. It does not ask where in the weights or activations the mechanism sits. This study takes its factor design and moves one level inward, to activations. arxiv.org/abs/2605.23055

arXiv:2608.21766 — Evaluation Awareness in Language Models: Representation, Verbalization, and Control

The closest prior work on method: difference-of-means directions at the last residual-stream token across layers, peak probe AUROC ≥ 0.7 for all six models it studies, contrastive activation addition shifting an LLM-judged verbalisation score by 0.2–1.5 points, and representation/verbalisation almost decoupled. Its calibration set is 200 WildChat samples versus 200 evaluation samples, which confounds evaluation-ness with domain and provenance: a probe can score well by detecting academic-benchmark text without representing anything about being tested. It also cannot decompose, because its positive class mixes every cue type at once. This study removes the confound by construction — same task, same entities, same request in both members of every pair — and adds the decomposition. Its verbalisation dissociation replicates here (r = +0.103). arxiv.org/abs/2608.21766

arXiv:2606.23583 — Evaluation Awareness Is Not One Capability: Evidence from Open Language Models

The paper this work is most likely to be confused with, so the distinction is worth stating precisely. That paper decomposes along measurement paradigm — behavioural, representational, mechanistic — across 37 open-weight models and seven families, and shows those three axes dissociate. The difference here is granularity, and it is a difference in kind. Its unit of decomposition is three measurement paradigms; ours is eight cue factors inside the representational paradigm. Between its conditions, dataset, task, framing tier and paradigm all vary at once; between ours, only the cue configuration varies while task, entities and request stay identical. Its finest cue control is a 2×2 format × intent factorial (best AUROC 0.622) assembled across datasets; ours is an 8-factor factorial within one instrument.

The two conclusions are logically independent: a representation could be perfectly one-dimensional and still dissociate from behaviour, and conversely could be a bundle of eight unrelated detectors while all three of their axes agreed. What this study adds to their picture is a mechanistic explanation for why coarse format/intent factorials score low: the analogous cues here — F7 format, F8 disclosure — are precisely the two the global axis captures, while the semantic cues are the ones it misses, so a coarse factorial averages a cue the model represents strongly with cues it represents along nearly orthogonal axes. arxiv.org/abs/2606.23583

arXiv:2603.19426 — Devbunova, Is Evaluation Awareness Just Format Sensitivity? Limitations of Probe-Based Evidence under Controlled Prompt Structure

The most directly threatening of the four to the validity of what is measured here. It argues that probe-based evidence conflates structural format with contextual understanding, and shows that a probe trained the usual way calls 94.5 % of real user requests "evaluation" as soon as they are laid out like a benchmark, while format-decorrelated training cuts that to 7.2 %.

This study corroborates that diagnosis from the inside: the cue decomposition shows which cue the probe is riding on — the global axis is aligned with F8 disclosure at cosine 0.80–0.92 and with F7 format at 0.48–0.78, while the semantic cues sit at 0.11–0.33. It also extends the confound: her control makes a deployment prompt look like a benchmark, whereas ctl_greet does the opposite — it makes the prompt look more like a real user — and the projection still moved toward the evaluation pole at 0.6B. That suggests the vulnerability is to stylistic deviation in either direction rather than to benchmark structure specifically. With the scale correction above, the strength of this particular extension is much reduced (+1.3 % of span at 8B), and it should be read as a small-model finding awaiting a properly powered test. arxiv.org/abs/2603.19426

4. Limitations

Stated plainly, in the order they would bite a reader trying to use this.

• Compute ceiling and model scale. Phase 1 ran on 2 vCPU and 7 GB of RAM with no accelerator, which capped the primary model at Qwen/Qwen3-0.6B — below every model in all four reference papers. Later phases reached 8B, 14B and a second family on 2× Tesla T4, but nothing here reaches the 32B+ range, and the 14B result additionally carries a measured quantisation caveat (NF4; cosine 0.910 against the fp16 direction at the primary layer, versus 0.987 for INT8).
• Three Phase-1 claims did not survive scale. The surface-control magnitude, the degeneracy catastrophe and the ordering of intervention effects were all properties of Qwen3-0.6B or of a bug, not of the direction. They are marked in place above. This is the reason the article carries them rather than deleting them: a reader who saw the earlier numbers should be able to find out what happened to them.
• Our own intervention code was wrong for a day. The register_forward_hook bypass described above invalidated every Phase-1 intervention condition after ablate. The lesson generalises beyond this project: an intervention that reports success while doing nothing passes every diagnostic that counts handles and calls.
• Directional ablation could not cleanly test the causal question in Phase 1. At 0.6B the model broke under full ablation (93 % degenerate generations), so a behavioural null could not be distinguished from a model that had simply stopped working. At 8B this is no longer true — degeneracy is 7 % and MMLU falls 2 points — so the causal test is possible, and the answer is still no sandbagging.
• Base-vs-instruct. Instruction tuning did not create the signal — the base checkpoint gives AUROC 1.000 with d_z 5.28 versus 7.52 for instruct, the same cue geometry (mean cosine 0.221 vs 0.209) and the same F1 blind spot (0.55 vs 0.52) — but the direction rotates across post-training: cosine(base, instruct) = 0.298 averaged over layers ≥ 5, against the 0.72–0.83 reported on larger models in arXiv:2608.21766. Which of those is the scale-typical value is not settled here.
• Benign requests only. Only the capability member of each EvalAwareBench pair is rendered, so nothing here speaks to evaluation cues on harmful requests.
• One instrument. Every cue definition comes from EvalAwareBench. The decomposition is a decomposition of that taxonomy; a different factorisation of "evaluation-ness" could partition the variance differently.

5. What this means for anyone building an evaluation-awareness monitor

The practical content of the decomposition is a warning about what a probe trained the usual way will and will not see.

A direction recovered from a broad evaluation-versus-deployment contrast is, to within measurement precision, an explicit-disclosure detector, supported by task formatting and benchmark-paradigm conventions. Those are exactly the cues that are easy to strip from a benchmark: stop saying "this is an evaluation", drop the rubric language, write the prompt as prose instead of a template.

What such a monitor is near-blind to is placeholder-entity artificiality and stakes telegraphing — cues that read at or near chance on the global axis (AUROC 0.52 for artificiality; held-out transfer 0.525) even though the model demonstrably represents them elsewhere (a dedicated direction separates artificiality at 0.68, a trained probe at 0.72). Those are exactly the cues that are hard to strip, because a benchmark that cannot use invented companies, round numbers and named stakes is close to not being a benchmark.

So the failure mode is not random. A monitor built this way will be most confident on the evaluations you could most easily have disguised, and least sensitive on the properties that make a test a test. If you are building one, the cheap check is the one used here: hold the task fixed, toggle cues one at a time, and measure per-cue transfer rather than aggregate AUROC — aggregate AUROC saturates at 1.000 and tells you nothing.

---