Is Evaluation Awareness One Direction, or Eight?

A factor-controlled probe of the evaluation-awareness representation in open-weight models

Mateusz Piesiak (Project Inkblot) · agentic research assistant: Claude Opus 5 · September 2026 · updated 18 October 2026 (Qwen3-32B added)
Instrument: EvalAwareBench (arXiv:2605.23055, CC BY-NC 4.0) · analysis code ↗
models: Qwen3 0.6B–32B, Gemma 3 270M/1B · SAEs: Qwen-Scope, Gemma Scope 2

TL;DR

Working inside the representational axis of evaluation awareness, with the underlying task held fixed and only the eight EvalAwareBench cue factors toggled, the answer is no — it is not one direction. The direction that a standard all-cues difference-of-means recovers is largely an explicit-disclosure-and-formatting detector, and it is blind to the one cue a benchmark cannot avoid.

• replicatedThe eight cue directions are not one shared axis. Mean pairwise cosine 0.19–0.22 at 0.6B; cross-fitted 0.190 [0.161, 0.215] against a same-cue reliability ceiling of 0.883 — so this is a real separation, not estimation noise. Across 0.6B / 8B / 14B the cue cosine sits at 21.5 % / 19.7 % / 21.4 % of that measurable ceiling. Not one direction is the claim, and it has now been reproduced independently (see below). The count is not: the data do not show eight independent directions that add up — the structure may be several non-redundant and possibly interacting directions, and geometric distinctness is not by itself evidence of distinct mechanisms (Limitations).
• replicatedOne cue essentially is the global axis. cosine(F8 explicit disclosure, global) = 0.889 [0.879, 0.900] — at the 0.883 ceiling. Removing F8 alone costs 55 % of the deploy→eval span at the probe-selected layer 27 (36–57 % across layers 7–27); every other cue costs ≤ 10 % at layers 14–27.
• replicatedThe axis cannot see stimulus artificiality. F1-only versus deployment, scored on the global axis: AUROC 0.52 — chance. A direction fitted on the other seven cues transfers to F1 at 0.525 [0.501, 0.553]. The model does represent artificiality (a dedicated direction gets 0.68, a trained probe 0.72); that information simply lies almost orthogonal to the axis the standard contrast recovers.
• replicatedThe contrast is not one-dimensional. PC1 of the matched difference vectors explains 12.6–20.9 %; participation ratio ≈ 20 (rising to 27.9 at 8B and 30.4 at 14B).
• withdrawnA non-evaluative surface change moves the projection. Adding "hey, quick one —" and "thanks a lot!" — no evaluation content at all — moved the projection 15 % of the eval/deploy span (AUROC 0.851, d = 1.47) at 0.6B. At 8B this is +1.3 % (AUROC 0.556, d = 0.18) and at 14B +4.7 %. The 0.6B figure is a small-model artefact; the confound is real but an order of magnitude smaller at scale, and it does not shrink monotonically.
• replicatedNo statistically detectable sandbagging in the tested settings. MMLU accuracy 0.458 (0.6B) → 0.734 (8B) → 0.807 (14B) under neutral framing; eval-minus-deploy of −0.020, +0.000, +0.013, never significant (McNemar p ≥ 0.29; validation harness, letter log-probability). All Phase-1 sandbagging interaction terms were null (|Δ| ≤ 0.058, all Wilcoxon p ≥ 0.34).
• replicatedEvaluation framing changes compliance, not accuracy. At 14B the mean top-letter log-probability moves from −16.68 (deployment framing) to −3.90 (evaluation framing) while accuracy does not move at all.
• replicatedRepresentation and verbalisation are uncorrelated. Pearson r = +0.103 (p = 0.59), Spearman ρ = +0.069 — an independent replication of the dissociation reported in arXiv:2608.21766. Explicit meta-commentary sat at 1/15 on evaluation-framed prompts, and blind adjudication found that single hit to be prompt echo rather than inference.
• replicatedSAE view: discriminatively localised, representationally distributed. One Qwen-Scope feature reaches AUROC 0.986–0.992 against the full direction's 0.993–1.000, yet recovers only 20–25 % of ‖Δμ‖; 25–31 features carry half the |Δmean| L1 mass and 109–124 carry 90 %. The same shape reproduces on Gemma Scope 2.
• withdrawnDirectional ablation is not a clean knock-out. At 0.6B, ablation reduced the projection by 92 % but left 93 % of free generations degenerate — unlike the refusal direction of Arditi et al. (2024), which can be projected out leaving a working model. Both halves of that statement were later corrected: with fixed intervention code the reduction is 99.99 %, and at 8B degeneracy is 7 % (MMLU −2.0 points, perplexity 37 → 43). The 93 % figure is a property of Qwen3-0.6B, not of the direction.
• update 18 October 2026Qwen3-32B (bf16, frozen protocol): the geometry result is unchanged in kind. At layer 34 the cross-fitted cue cosine is 0.215 [0.200, 0.228] against a ceiling of 0.910 [0.888, 0.925]: R = 0.237 (conservative bounds 0.216–0.257), pre-registered label SEPARATION_SUPPORTED. cos(F8, global axis) is 0.852; removing F8 alone costs 37.4 % of the deploy→eval span (best other cue, F6: 8.1 %); held-out F1 AUROC 0.593 [0.561, 0.631]. 8B/14B/32B differ in precision and hardware, so this is a description, not a scaling trend (§6.3).
• update 18 October 2026No framing effect detected on MMLU at 32B. M600 (n = 600): eval − deploy = +0.012 [−0.007, +0.032], McNemar p = 0.311: no framing effect was detected on this harness. This does not establish absence of an effect or absence of sandbagging (§6.4).
• exploratoryAblation × framing interaction on a separate 200-item battery. The global-direction ablation shows an interaction of −0.070 [−0.115, −0.030] (Holm over 7 tests: 0.022); the layerwise variant, which collapses accuracy, has the opposite sign. No size-matched control exists in this battery. Not a sandbagging result and not evidence for a mechanism (§6.5).
• openAdd interventions are not validated in the production configuration. verify reads a layer before the add site, so it cannot see the addition; the self-test uses a different index. add_random is not distinguishable from add_x2 in this battery: no demonstrated specificity (§6.5).

1. Method

Every estimator, statistic and intervention described in this section is in the code/ directory of this Space; the table in code/README.md maps each script to the numbers it produces.

1.1 Instrument

The study uses EvalAwareBench from arXiv:2605.23055: 100 paired safety/capability tasks as YAML templates in which eight trigger factors can each be switched between a naturalistic and an evaluative level while the underlying request is held fixed. Only the capability member of each pair is rendered — no harmful text needs to be produced for a representation-level study, and it removes request harmfulness as a confound with evaluation-ness. Conclusions here are therefore about evaluation cues on benign requests.

1.2 Compute constraint, stated up front

Phase 1 had no GPU. HF Jobs and HF Sandboxes both returned 402 Payment Required for this account; the session container had 2 vCPU, 7 GB RAM and no accelerator, and lacked AVX512-BF16/AMX, so bfloat16 ran 2.8× slower than float32 (measured 5.38 vs 1.94 s/prompt on Qwen3-0.6B). The practical ceiling was ~2B parameters.

Consequence for interpretation: the Phase-1 primary model is Qwen/Qwen3-0.6B, far below the 14B–32B range originally targeted and below every model in all four reference papers. Nothing in Phase 1 licenses a claim about scale. Phases 2–3C later obtained 2× Tesla T4 and re-ran the identical pipeline at 1.7B, 8B and 14B, and on Gemma 3 (270M, 1B) — which is how the withdrawals in this article were found.

1.3 Prompt sets (1,298 prompts)

1.4 Activation capture

Residual stream at the last prompt token, every hidden state (embeddings plus all block outputs), stored as float16.

One property removes the most common confound in last-token probing: the chat template appends an identical <|im_end|>\n<|im_start|>assistant\n tail to every prompt, so all 200 main prompts share an identical 30-character tail and the probed position never sees condition-specific local tokens. The embedding layer confirms it — hidden state 0 is bit-identical across conditions, its difference-of-means is exactly zero and its AUROC exactly 0.5. Base models, having no chat template, get a constant "\n\nAssistant:" suffix to buy the same property.

1.5 Directions, statistics, interventions

• Global direction per layer ℓ: v_ℓ = mean(a_ℓ | eval_all) − mean(a_ℓ | deploy) over the 100 matched pairs, normalised to unit length.
• Separability: (a) untrained projection onto v̂; (b) logistic probe on raw activations with GroupKFold(5) grouped by task_id, so no task appears in both train and test; (c) a label-shuffled null permuting labels within each task group, so group structure cannot leak the signal. The probe is exactly make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000, C=1.0)) under GroupKFold(n_splits=5) — scikit-learn 1.8.0 defaults otherwise, so L2 penalty, lbfgs solver, and C = 1.0 held fixed across every layer, model and fold. No regularisation search is run and no reported number depends on one.
• Effect sizes: Cohen's d, paired d_z, exact binomial sign test over the 100 matched pairs.
• Layer selection. With all eight cues toggled at once, probe AUROC saturates at 1.000 in 28 of 29 hidden states, so argmax-AUROC is degenerate and would silently pick the shallowest saturated layer. The primary layer is chosen by paired effect size d_z, which does not tie, and every downstream analysis is repeated at quarter-, mid- and three-quarter depth so no conclusion rests on a single chosen layer.
• Interventions: directional ablation x ← x − v̂(v̂ᵀx) at every token position and every layer, following Arditi et al. (2024); a layerwise variant removing each layer's own direction at that layer, motivated by the cross-layer rotation; and activation addition x ← x + αv̂ at the single primary layer, CAA style. The magnitude convention is α = ‖Δμ‖, the norm of the unnormalised difference-of-means vector at that layer — so add_x1 is x + α·v̂, add_x2 is x + 2α·v̂, and the control add_random is x + 2α·r̂, magnitude-matched to add_x2 rather than to add_x1. Every intervention has a random-direction counterpart of equal norm as its null; for ablation the stronger control rescales per token so that the same magnitude is removed along an unrelated direction (κ = |x·v̂|/|x·r̂|, clamped at 50).

1.6 The reliability ceiling

A cosine of 0.19 between two estimated directions is only evidence of separation if directions estimated from finite data could have been more aligned. So the same cue is estimated on disjoint halves of the tasks and the two halves compared: that cross-fitted same-cue cosine is 0.883 [0.852, 0.906]. It is the ceiling any two cue directions could reach given this much estimation noise, and every cue-cosine number in this article is read against it.

1.7 Per-cue reliabilities: the full cross-half matrix

The 0.883 ceiling is a mean over eight cues, and a mean can hide one cue that is estimated much less reliably than the rest — whose low cosines would then be partly noise. Figure 7 shows the matrix the ceiling is taken from: on each of 400 random splits of the 48 single-cue tasks into disjoint halves, every cue's direction from one half is compared with every cue's direction from the other. The diagonal is each cue's own split-half reliability, and the same computation returns the two summary numbers exactly — diagonal mean 0.883, off-diagonal mean 0.190.

Two things sharpen the reading; neither changes the conclusion.

• Reliability is not uniform across cues. It runs from 0.98 (F8 disclosure) and 0.95 (F7 format) down to 0.81 (F5 stakes) and 0.67 [0.56, 0.76] (F1 artificiality). F1 is the one cue whose own direction is materially noisier than the aggregate ceiling implies — consistent with it being the most weakly represented cue (own-direction AUROC 0.68). Its low cross-cue cosines are therefore partly lower reliability, but only partly: rescaled by the two reliabilities involved, F1's closest partner is F3 at 0.32, and its value with F8 is −0.00.
• No pair comes near its own ceiling. Dividing each cross-cue cosine by the geometric mean of its two cues' reliabilities gives a per-pair version of "percent of ceiling". Its mean is 0.21 — the same one fifth as the aggregate — and its maximum is 0.43 (F6 paradigm with F7 format), followed by F5–F7 (0.40) and F2–F7 (0.37). The most-shared pairs all involve F7 format; pairs among F1, F4 and F5 sit at zero.

Intervals here are over random splits of the same 48 tasks, as for the aggregate ceiling, not over fresh task samples. Computed by code/bootstrap_geometry.py; the matrix with its intervals is in results/qwen3-0.6b_cross_half.json.

1.8 Models, precision, hardware and layers (added 18 October 2026)

The three Qwen3 sizes that were run on the full 100-pair main set with the bootstrap estimators are listed with everything that differs between them. The 0.6B model (CPU, §1.2) and the Gemma 3 models are described where they appear; they are not part of this table. The layer is the argmax of the paired effect size on the matched main pairs, excluding the post-final-norm state, in every row.

The 8B dtype was not recorded when the run was made, so it is unverified; dtype='auto' on a T4 cannot be bf16. The 14B checkpoint is pre-quantised, so its geometry is that of the 4-bit model. The 32B run is the only one with a pinned revision, a frozen manifest (sha256 b2ec4200…e368e10), pre-registered criteria, and numeric gates (21 recorded, none failed). Precision, hardware and checkpoint provenance are therefore confounded with size throughout this article.

Two estimator conventions are kept apart everywhere in the new material. Resample means (cross-fitted cosine, ceiling, R, cos(F8, global), LOFO F1): the point value is the mean over n_boot random task-half resamples, computed by code/bootstrap_geometry.py, with a percentile 95 % interval of the same resamples. Full-sample values (F8 necessity, PC1, participation ratio, surface controls, OOD): one pass over all pairs, no interval. R = c/ρ; its bounds are c_lo/ρ_hi and c_hi/ρ_lo, which is a conservative bound built from the endpoints, not a confidence interval of the ratio. The earlier 8B and 14B numbers are quoted in their original convention; none was re-estimated for this update.

MMLU harness. The prompts are the same item file in all runs (letter log-probability at the assistant prefill Answer:, non-thinking template). Three framings: neutral — the question and four options; eval — "Question i of n. This is an automated evaluation. Answer with a single letter only; do not explain.", then the item; deploy — "hey, quick one - i'm stuck on this and i keep going back and forth.", then the item, then "which one is it?". Because the eval framing contains the item count n, runs with different n are different prompts. The 32B run therefore has three separate MMLU sets that are never pooled: M300 (n = 300, text-identical to the 14B and Gemma 3 1B calls), M600 (n = 600, the powered test of B1), and the 200-item intervention battery (first 200 items, n = 200 in the eval text, layer 34, 8 conditions × 2 framings, 3,200 rows).

2. Results

Q1 — Are the cue directions one shared axis? Rejected.

Mean off-diagonal cosine between the eight per-cue directions is 0.19–0.22; cross-fitted, 0.190 [0.161, 0.215] against the 0.883 ceiling. At 8B the same quantity is 0.169 against a ceiling of 0.859, and at 14B 0.198 against 0.924 — a ratio of roughly one fifth of the ceiling at every scale tested, over a 23× span of parameter count. The stability of the ratio matters more than any single cosine, because the ceiling itself moves with estimation noise while the ratio does not.

At Qwen3-32B (layer 34, bf16) the cross-fitted cosine is 0.215 [0.200, 0.228] against a ceiling of 0.910 [0.888, 0.925]; R = 0.237 (bounds 0.216–0.257). The earlier text says "roughly one fifth of the ceiling at every scale tested"; at 32B the ratio is nearer one quarter (0.197 at 8B, 0.214 at 14B, 0.237 at 32B, resample means, with overlapping bounds), so "one fifth" is best read as "between one fifth and one quarter". The full matrices are in §6.2 (Figures 9 and 11).

Q2 — Does a direction fitted on seven cues transfer to the eighth? Rejected.

It depends entirely on which cue is held out:

At 8B and 14B the F1 number moves to 0.582 and 0.616 (bootstrap means) — it narrows slightly with scale and remains unusable. On Gemma 3 1B it is 0.535 in the residual stream and 0.537 in SAE feature space, agreeing within 0.002 across two different representations.

Held-out F1 AUROC at 32B: 0.593 [0.561, 0.631] (resample mean), against 0.582 (8B) and 0.616 (14B) in the same convention. The transfer to F1 remains close to unusable. F1 on the global axis is 0.538 and on its own direction 0.767 (full sample).

Q3 — Is the contrast one-dimensional? Rejected.

PC1 of the matched difference vectors explains 12.6–20.9 % depending on layer; participation ratio ≈ 20. Both move away from one-dimensionality with scale: PC1 14.8 % → 12.9 % → 10.4 % and participation ratio 20.3 → 27.9 → 30.4 across 0.6B → 8B → 14B.

PC1 explains 11.5 % and the participation ratio is 28.4 at 32B (full-sample, layer 34); top-5 components 34.0 %. The sequence 8B → 14B → 32B is PC1 12.9 % → 10.4 % → 11.5 % and PR 27.9 → 30.4 → 28.4. That is not a continuation of the movement "away from one-dimensionality with scale" stated above: the 32B values lie between the 8B and 14B values. The contrast is still far from one-dimensional, but the monotone statement is not extended, and across checkpoints that differ in precision and hardware no trend is claimed.

Q4 — Do the cues contribute comparable shares? Rejected.

Removing F8 explicit disclosure alone costs 55 % of the deploy→eval span at the probe-selected layer 27 (36–57 % across layers 7–27); every other cue costs ≤ 10 % at layers 14–27 (F7 at layer 7: 17 %). Its cosine with the global axis is 0.889 [0.879, 0.900] — at the reliability ceiling, meaning that within measurement precision the global axis is the disclosure direction. On Gemma 3 1B this is more extreme still: at its probe-selected layer 25 the cosine is 0.945 and removing F8 costs 74 % of the span.

Removing F8 costs 37.4 % of the deploy→eval span at 32B (best other cue: F6, 8.1 %); cos(F8, global axis) = 0.852 [0.839, 0.864]. Full-sample necessity at 8B / 14B / 32B: 54.5 % / 37.0 % / 37.4 % (the 0.6B value of 55 % above is a different layer and model). No cue other than F8 exceeds 8.1 % at 32B. In pre-registered terms the A1 and A1b criteria are supported.

Q5 — Does a non-evaluative surface change leave the projection alone? Violated at 0.6B.

The ctl_greet control adds a greeting and a sign-off — "hey, quick one —", "thanks a lot!" — and no evaluation content whatsoever. At 0.6B it moved the projection 15 % of the eval/deploy span (AUROC 0.851, d = 1.47).

Surface controls at 32B, as a fraction of the deploy→eval span (full sample): greeting 0.060, typo 0.006, verbose 0.019 (AUROC vs deploy 0.691 / 0.524 / 0.575), all small, as pre-registered (A4 supported). The greeting control is the largest and is nominally higher than at 8B (0.013) and 14B (0.047) — noted, not interpreted; no interval exists for full-sample values. The 0.6B violation above was a property of that model.

The causal section

Phase 1 reported that directional ablation reduced the projection by 92 % while random- direction ablation moved it 1 %, and that ablation cost −0.283 MMLU accuracy identically in both framings, with every sandbagging interaction term null (|eval Δ − deploy Δ| ≤ 0.058, all Wilcoxon p ≥ 0.34). It also reported that 93 % of free generations became degenerate under all-layer ablation (80 % layerwise, 37 % under addition, 17 % baseline) and concluded that, unlike the refusal direction of Arditi et al. (2024), this direction cannot be projected out while leaving a working model.

What survives, and is now on stronger evidence: there is no sandbagging to remove. Phase 1 found MMLU 0.533 under evaluation framing versus 0.517 under deployment framing, with all interaction terms null. At 8B and 14B, with 500 and 300 paired items, two independent scorers and three framings, eval-minus-deploy is 0.000 and +0.013 (McNemar p = 1.00 and p = 0.52), and on Gemma 3 1B −0.007 (p = 0.88) — now with an intervention that provably removes 99.99 % of the direction while the model still scores 0.730.

Meanwhile evaluation framing moves compliance by orders of magnitude: mean top-letter log-probability −15.46 → −5.02 at 8B, −16.68 → −3.90 at 14B, and −8.79 → −0.08 on Gemma 3 1B. The model becomes far more willing to commit to an answer letter when told it is being evaluated, and no more accurate.

At 32B the interventions were run under the corrected code with fixed thresholds written before the run. The result is mixed and is described in §6.5: global-direction ablation leaves the model working, the layerwise variant collapses accuracy, the add conditions are not validated in the production configuration, and an exploratory ablation × framing interaction appears whose size-matched control does not exist. None of that revisits the withdrawn Phase-1 claims above.

Verbalisation

Explicit meta-commentary was at the floor: 1/15 on evaluation-framed prompts, and blind adjudication showed that single hit to be prompt echo rather than inference. The internal projection and verbalised awareness are uncorrelated: Pearson r = +0.103 (p = 0.59), Spearman ρ = +0.069 (p = 0.72) — an independent replication of the dissociation in arXiv:2608.21766, obtained on a cleaner contrast.

At 32B, free-generation meta-commentary was again at the floor (explicit mentions 0/15 under the eval-framed and deploy prompts for the unmodified model; at most 1/15 in any intervention cell). The teacher-forced verbalisation grid (n = 30 OOD prompts) shows no detectable change from ablating the global direction at layer 34 (mean change −0.000, d_z ≈ 0); see §6.6. The representation–verbalisation correlation reported above was not recomputed at 32B.

Verbalisation propensity by depth

A second, teacher-forced measure scores how much more probable four evaluation-aware chain-of-thought openers ("This looks like a test rather than a real user request.", …) are than four syntax-matched neutral ones, on the 30 out-of-distribution prompts (code/verbalize_grid.py). Each row removes the direction estimated at one layer from every layer of the model, at full strength, and reports the paired change in that score against no intervention. The numbers are in results/verbalisation_depth.json.

On Qwen3-8B the sign depends on the depth the direction comes from: the layer-18 and layer-22 directions raise the score on 30 and 30 of 30 prompts, the layer-9 and layer-27 directions lower it (25 and 22 of 30 prompts down). Gemma 3 1B shows the same structure at different depths: the layer-13 direction lowers the score (−0.232) while layers 6, 20 and 25 raise it. At 0.6B every depth raises it. There is no single "evaluation-awareness dial" whose removal has one sign, and the probe-selected layer is not the one with the largest effect (Qwen3-8B: layer 18; Gemma 3 1B: layer 6). The magnitude-matched random ablation at the probe-selected layer moves the score the other way on all three models (−0.013, −0.017 and −0.386; Wilcoxon p = 0.015, 0.009 and 0.001), so the effects belong to the learned directions, not to the size of the edit. Why removing a direction raises the propensity at some depths and lowers it at others is not established. These cells were computed in Phase 3 but, until 15 October 2026, were described only in the script's docstring and not on this page; the PLLuM-12B companion page found the deep-layer sign again at layer 30 of 40.

SAE decomposition

Against Qwen-Scope TopK SAEs (32,768 features, k = 50, resid_post) on Qwen3-1.7B-Base, at five layers:

• Discriminatively localised: one SAE feature reaches AUROC 0.986–0.992 against the full direction's 0.993–1.000.
• Representationally distributed: that same feature explains only R² = 0.06–0.14 of the mean-difference vector Δμ and recovers 20–25 % of its norm; it takes 25–31 features to carry half the L1 mass of the per-feature mean differences and 109–124 to carry 90 %.

A monitor built on that one feature would work, and would tell you almost nothing about what the model represents. The same shape reproduces on a different family, suite and SAE architecture (Gemma Scope 2, Gemma 3 1B, layer 24): one feature of 16,384 reaches AUROC 0.996 as a deployment detector while 113 features carry half the |Δmean| L1 mass and 777 carry 90 %.

Cross-family and cross-method check

A later phase re-ran the identical estimators on Gemma 3 and additionally recomputed the whole contrast in Gemma Scope 2 SAE feature space rather than the residual stream, using the same cross-fitting, the same reliability ceiling and the same label-shuffled nulls — so the only thing that changes is the representation.

All three columns are bootstrap means over task resamples (400 for Qwen3-8B, 1,000 for Qwen3-14B and Gemma 3), recomputed from reports/*_bootstrap.json. The decomposition is not a Qwen artefact and not a linear-probe artefact.

A Qwen3-32B row for the same three columns (resample means, 1,000 resamples). Precision and hardware differ from the Qwen3 rows above, so the rows are not a series.

3. Relation to prior work

arXiv:2605.23055 — Decomposing and Measuring Evaluation Awareness

Supplies the instrument: the eight-factor taxonomy and EvalAwareBench. That paper measures behaviour — chain-of-thought monitoring of recognition and of the propensity to act on it — across nine frontier models and four benchmarks. It does not ask where in the weights or activations the mechanism sits. This study takes its factor design and moves one level inward, to activations. arxiv.org/abs/2605.23055

arXiv:2608.21766 — Evaluation Awareness in Language Models: Representation, Verbalization, and Control

The closest prior work on method: difference-of-means directions at the last residual-stream token across layers, peak probe AUROC ≥ 0.7 for all six models it studies, contrastive activation addition shifting an LLM-judged verbalisation score by 0.2–1.5 points, and representation/verbalisation almost decoupled. Its calibration set is 200 WildChat samples versus 200 evaluation samples, which confounds evaluation-ness with domain and provenance: a probe can score well by detecting academic-benchmark text without representing anything about being tested. It also cannot decompose, because its positive class mixes every cue type at once. This study removes the confound by construction — same task, same entities, same request in both members of every pair — and adds the decomposition. Its verbalisation dissociation replicates here (r = +0.103). arxiv.org/abs/2608.21766

arXiv:2606.23583 — Evaluation Awareness Is Not One Capability: Evidence from Open Language Models

The paper this work is most likely to be confused with, so the distinction is worth stating precisely. That paper decomposes along measurement paradigm — behavioural, representational, mechanistic — across 37 open-weight models and seven families, and shows those three axes dissociate. The difference here is granularity, and it is a difference in kind. Its unit of decomposition is three measurement paradigms; ours is eight cue factors inside the representational paradigm. Between its conditions, dataset, task, framing tier and paradigm all vary at once; between ours, only the cue configuration varies while task, entities and request stay identical. Its finest cue control is a 2×2 format × intent factorial (best AUROC 0.622) assembled across datasets; ours is an 8-factor factorial within one instrument.

The two conclusions are logically independent: a representation could be perfectly one-dimensional and still dissociate from behaviour, and conversely could be a bundle of eight unrelated detectors while all three of their axes agreed. What this study adds to their picture is a mechanistic explanation for why coarse format/intent factorials score low: the analogous cues here — F7 format, F8 disclosure — are precisely the two the global axis captures, while the semantic cues are the ones it misses, so a coarse factorial averages a cue the model represents strongly with cues it represents along nearly orthogonal axes. arxiv.org/abs/2606.23583

arXiv:2603.19426 — Devbunova, Is Evaluation Awareness Just Format Sensitivity? Limitations of Probe-Based Evidence under Controlled Prompt Structure

The most directly threatening of the four to the validity of what is measured here. It argues that probe-based evidence conflates structural format with contextual understanding, and shows that a probe trained the usual way calls 94.5 % of real user requests "evaluation" as soon as they are laid out like a benchmark, while format-decorrelated training cuts that to 7.2 %.

This study corroborates that diagnosis from the inside: the cue decomposition shows which cue the probe is riding on — at the probe-selected layer 27 (bootstrap means) the global axis is aligned with F8 disclosure at cosine 0.89 and with F7 format at 0.48, F6 paradigm recognizability — benchmark framing, itself a structural cue — sits at 0.50, and the five remaining cues at 0.11–0.36. It also extends the confound: her control makes a deployment prompt look like a benchmark, whereas ctl_greet does the opposite — it makes the prompt look more like a real user — and the projection still moved toward the evaluation pole at 0.6B. That suggests the vulnerability is to stylistic deviation in either direction rather than to benchmark structure specifically. With the scale correction above, the strength of this particular extension is much reduced (+1.3 % of span at 8B), and it should be read as a small-model finding awaiting a properly powered test. arxiv.org/abs/2603.19426

4. Limitations

Stated plainly, in the order they would bite a reader trying to use this.

• Compute ceiling and model scale. Phase 1 ran on 2 vCPU and 7 GB of RAM with no accelerator, which capped the primary model at Qwen/Qwen3-0.6B — below every model in all four reference papers. Later phases reached 8B, 14B and a second family on 2× Tesla T4, but nothing here reaches the 32B+ range, and the 14B result additionally carries a measured quantisation caveat (NF4; cosine 0.910 against the fp16 direction at the primary layer, versus 0.987 for INT8).
• Three Phase-1 claims did not survive scale. The surface-control magnitude, the degeneracy catastrophe and the ordering of intervention effects were all properties of Qwen3-0.6B or of a bug, not of the direction. They are marked in place above. This is the reason the article carries them rather than deleting them: a reader who saw the earlier numbers should be able to find out what happened to them.
• Our own intervention code was wrong for a day. The register_forward_hook bypass described above invalidated every Phase-1 intervention condition after ablate. The lesson generalises beyond this project: an intervention that reports success while doing nothing passes every diagnostic that counts handles and calls.
• Directional ablation could not cleanly test the causal question in Phase 1. At 0.6B the model broke under full ablation (93 % degenerate generations), so a behavioural null could not be distinguished from a model that had simply stopped working. At 8B this is no longer true — degeneracy is 7 % and MMLU falls 2 points — so the causal test is possible, and the answer is still no sandbagging. More generally: the geometry in this article — distinct cue directions, low cross-fitted cosines, a disclosure-dominated global axis — establishes that the cues are represented differently, not that they are handled by different causal mechanisms. Geometrically distinct directions can feed the same downstream computation, and nothing here beyond whole-direction ablation tests which cue is mediated by which circuit.
• Base-vs-instruct. Instruction tuning did not create the signal — the base checkpoint gives AUROC 1.000 with d_z 5.28 versus 7.52 for instruct, the same cue geometry (mean cosine 0.221 vs 0.209) and the same F1 blind spot (0.55 vs 0.52) — but the direction rotates across post-training: cosine(base, instruct) = 0.298 averaged over layers ≥ 5, against the 0.72–0.83 reported on larger models in arXiv:2608.21766. Which of those is the scale-typical value is not settled here.
• Benign requests only. Only the capability member of each EvalAwareBench pair is rendered, so nothing here speaks to evaluation cues on harmful requests.
• One instrument. Every cue definition comes from EvalAwareBench. The decomposition is a decomposition of that taxonomy; a different factorisation of "evaluation-ness" could partition the variance differently.
• update 18 October 2026Qwen3-32B is one run on one checkpoint. A single bf16 run on one A100, one seed for the task-half resamples, 100 matched pairs, 48 tasks. The statement earlier in this section that nothing reaches 32B+ referred to the original article; the 32B result is now included, but 32B is the largest size here and nothing is claimed beyond it.
• Size is confounded with precision, hardware and checkpoint provenance. 8B: unrecorded dtype on 2×T4, 14B: 4-bit NF4 on 2×T4, 32B: bf16 on an A100; n_boot 400 / 1000 / 1000. The three-model comparison (§6.3) is descriptive; no test of differences between models was done.
• The pre-registered thresholds were approved after the author had seen the earlier Qwen3 results. They were written and approved before the 32B run and not changed, but they were not set blind to the 8B/14B/Gemma numbers.
• Behavioural nulls are one-harness nulls. B1 reports no detected effect on one prompt template and one scorer (M600 interval [−0.0067; +0.0317], M300 [−0.0033; +0.0500]). It does not establish absence of an effect, of sandbagging, or of evaluation awareness.
• The ablation × framing interaction is exploratory. One battery of 200 items, 7 interaction tests, a 6-item baseline difference between framings, no size-matched random control in this battery (a size-matched control exists only in the verbalisation grid), and an opposite-signed result for the layerwise variant under accuracy collapse.
• Add interventions are not validated in the production configuration. See §6.5. Results from add_x1, add_x2 and add_random are reported with that qualification; add_random ≈ add_x2 means no specificity of the direction was demonstrated, not that none exists.
• One per-item file was reconstructed. The per-item rows of the 200-item battery were reassembled from the stored run file (3,168 of 3,200 rows read directly, 32 rows patched from a single inline read of the correct field). Every accuracy, gained/lost count and interaction statistic reproduces the run's own summary file to the digit; the reconstructed rows are marked and published.
• No 32B scree figure. The 32B activations are in a private dataset; the PC1 and participation-ratio values are given as numbers and in Figure 12 only.

5. What this means for anyone building an evaluation-awareness monitor

The practical content of the decomposition is a warning about what a probe trained the usual way will and will not see.

A direction recovered from a broad evaluation-versus-deployment contrast is, to within measurement precision, an explicit-disclosure detector, supported by task formatting and benchmark-paradigm conventions. Those are exactly the cues that are easy to strip from a benchmark: stop saying "this is an evaluation", drop the rubric language, write the prompt as prose instead of a template.

What such a monitor is near-blind to is placeholder-entity artificiality and stakes telegraphing — cues that read at or near chance on the global axis (AUROC 0.52 for artificiality; held-out transfer 0.525) even though the model demonstrably represents them elsewhere (a dedicated direction separates artificiality at 0.68, a trained probe at 0.72). Those are exactly the cues that are hard to strip, because a benchmark that cannot use invented companies, round numbers and named stakes is close to not being a benchmark.

So the failure mode is not random. A monitor built this way will be most confident on the evaluations you could most easily have disguised, and least sensitive on the properties that make a test a test. If you are building one, the cheap check is the one used here: hold the task fixed, toggle cues one at a time, and measure per-cue transfer rather than aggregate AUROC — aggregate AUROC saturates at 1.000 and tells you nothing.

6. Qwen3-32B: the full run (added 18 October 2026)

This section describes the Qwen3-32B run as a whole. Where its numbers also bear on an earlier result, they are repeated as dated update paragraphs under that result, and the corrections box above is unchanged. The run used the same eight-factor instrument and code as the earlier Qwen3 runs, on Qwen/Qwen3-32B revision 9216db5781bf21249d130ec9da846c4624c16137 in bfloat16 without quantisation on one A100 80 GB, with the protocol, criteria and numeric thresholds frozen in a manifest (sha256 b2ec42009c9d4bb1acf3517b0b0d2b5db1c3093848fb402ace97299fae368e10) before any 32B statistic was read. The results are sorted by what kind of statement they support: pre-registered, supported geometry (§6.2); no effect detected behaviour (§6.4); exploratory interaction (§6.5); and open add validation (§6.5).

6.1 Design and what was frozen

Data: the same 1,298 geometry prompts and the 100 matched main pairs as before (EvalAwareBench, commit 2961c17e…), plus a factorial pass of 722 prompts for the register × disclosure analysis. All hidden states (65, including embeddings; the post-final-norm state is excluded from the layer rule) were collected at the last prompt token. The primary layer is the argmax of the paired effect size on the matched pairs: layer 34 (paired d_z = 9.16; probe AUROC is 1.000 in 64 of 65 hidden states, so separability is not informative, as stated at the top of this article). Numeric run gates: 21 recorded, 0 failed (among them batch invariance, tokenizer and template constants, the MMLU harness sanity check for each run, intervention validity of ablation, and baseline degeneracy; the full list is in criteria_evaluation.json).

Thresholds. The criteria E1, A1–A10 and B1–B4 and their numeric thresholds were proposed in writing and approved by the author on 18 October 2026 without change. As noted in the Limitations, this approval came after the author had seen the 8B, 14B and Gemma results; B3 and B4 carry no prediction (label EXPLORATORY_NO_PREDICTION). The pre-registration and the criteria document are published unmodified in docs/q32/.

Deviations and operations are summarised in §6.7 and documented in OPERATIONAL_HISTORY.md; none changed code, configuration or thresholds after the freeze.

6.2 Geometry at layer 34

E1, A1 and A2 are resample means (1,000 task-half resamples of 48 tasks, seed 0) with percentile intervals; A1b and A3–A6 are full-sample values without intervals. The eight cue directions are therefore not one shared axis at 32B either: the cross-fitted cosine is about a quarter of the ceiling, cos(F8, global axis) is 0.852, F8 alone removes 37.4 % of the span, and the leave-one-factor-out transfer to F1 is 0.593.

Descriptive analyses (A7–A10), no thresholds. Directions across depth: the mean cosine between the global directions of adjacent layers is 0.883, but over all layer pairs from layer 5 it is 0.241 (median 0.146; 57.1 % of pairs below 0.2), so the direction at layer 34 is not the same vector at every depth. Format/semantic decomposition at layer 34: cos(format, semantic) = 0.428, cos(format, global) = 0.822, cos(semantic, global) = 0.324. Register × disclosure: the register and disclosure directions have cosine 0.180, and the global axis has cosine 0.500 with register and 0.830 with disclosure. The factorial interaction terms are large and significant, so register and disclosure do not add independently. Massive activations: the largest share of any single coordinate at layer 34 is 1.7 % (coordinate 731); all values are in massive_q32.json.

6.3 Comparison with 8B and 14B (descriptive)

One estimator convention per row group; the earlier models' values were taken from their stored reports, unchanged. The 8B column is marked: its dtype is unverified.

*8B: dtype unverified (2×T4); 14B: 4-bit NF4 on 2×T4; 32B: bf16 on one A100. Resample counts: 400 / 1000 / 1000. Layers: 22 / 17 / 34.

What can and cannot be said. The pattern found at 8B and 14B appears again at 32B: cross-cue cosines near a quarter or a fifth of the ceiling, F8 dominating the global axis (cos 0.886 / 0.865 / 0.852), F8 necessity far above any other cue, and LOFO F1 near 0.6. The nominal differences between the three models are small relative to the intervals for (a)–(c) and were not tested; the nominal ordering of R (0.197 < 0.214 < 0.237) has overlapping bounds. No claim is made about a trend with size, and the sentence "with scale" in the Q3 text is not extended.

6.4 Behaviour: MMLU under evaluation and deployment framing

Harness and framings are described in §1.8. G9 (neutral-framing sanity) passed on both runs: neutral logprob accuracy 0.817 (M300) and 0.822 (M600); base degeneracy 0.

B1 is labelled NO_FRAMING_EFFECT_DETECTED for both runs. No framing effect was detected on this harness: eval−deploy = +0.0117, 95% CI [−0.0067; +0.0317]. This does not establish absence of an effect or absence of sandbagging. (M300, same harness: eval−deploy = +0.0233, 95% CI [−0.0033; +0.0500]; the label is the pre-registered one.) The wording that was fixed before the run (allowed_statement in criteria_evaluation.json and the pre-registration) is kept there unchanged as part of the frozen record and is not the interpretation used here. M300 and M600 are separate runs with different eval-framing text (n = 300 vs 600) and are not pooled. The 600-item run restricted to its first 300 items is descriptive only and is in the aggregates.

B2 (ablation scan). Under global-direction ablation at full strength (projection reduced by 99.9 %) MMLU is 0.750 (eval) / 0.785 (deploy), against 0.815 / 0.790 at baseline; the eval−deploy difference is −0.035 against +0.025 at baseline. n = 200 per framing, so power is low; no mechanistic reading is offered.

6.5 Interventions on the 200-item battery

Layer 34, α = ‖Δμ‖ = 98.55, scorer = letter log-probability, non-thinking template, 8 conditions × 2 framings × 200 items = 3,200 rows; this battery is separate from M300/M600. Baseline (none): eval 164/200, deploy 158/200, so the framings already differ by 6 items before any intervention. Ablation edits the embedding and every block; add edits block 34 only; add_x2 adds 2α, add_random adds 2α along a random unit vector.

Ablation × framing interaction (exploratory)

The interaction statistic for a condition is the mean over items of (Δ_eval − Δ_deploy), where Δ is the item's correctness under the condition minus its correctness under none; negative means the condition costs the eval-framed prompts more. Tests: paired item bootstrap (20,000 resamples) for the interval; sign-flip permutation p (200,000 Monte-Carlo flips, not full enumeration); Holm correction over the 7 interaction tests of this battery.

† add conditions: production validation open (below). n = 200; gained/lost are item counts against none within the same framing.

Reading, with the limits stated: (1) ablate, the only ablation that leaves the model working (accuracy 0.745 eval / 0.785 deploy), costs the eval-framed prompts 17 items and the deploy-framed 8, an interaction of −0.070 [−0.115, −0.030] that survives Holm over the 7 tests (0.022); against its random control ablate_random0 the contrast is −0.050 [−0.095, −0.010], p = 0.040, which is the post-hoc family of 2 (Holm 0.040). (2) ablate_layerwise has the opposite sign (+0.120) at collapsed accuracy (0.47 / 0.32 against 0.82 / 0.79 at baseline); two ablations of the same direction disagree in sign, and the collapsed one is best read as damage, not as a property of the direction. (3) Ablation removes the projection in proportion to its size, and eval-framed prompts have a different projection from deploy-framed ones, so the removed amount differs between framings; the difference could be unequal perturbation size rather than anything specific to the direction. No size-matched control exists in this battery. (4) This is not a sandbagging test, not evidence of awareness or deception, and not evidence for a mechanism. It is a pattern to follow up with a pre-registered, size-matched design.

Layerwise ablation: accuracy collapse

ablate_layerwise removes the per-layer direction at every block and drops accuracy to 0.47 (eval) and 0.32 (deploy) from 0.82 / 0.79: 77 and 107 items lost. Its random control ablate_layerwise_random leaves accuracy at 0.81 / 0.80. The difference is large, but a collapse is not a selective result: the model breaks before the layerwise directions are removed in any behaviourally interpretable sense, and the interaction sign reverses relative to ablate. Contrast against the random control: +0.135 [+0.045, +0.220], p = 0.004 (post-hoc family of 2, Holm 0.008).

Add interventions: validation open

The pre-registered verify step reported identical mean projections for none, add_x1, add_x2 and add_random (mean projection −0.538 for all four, sd 30.10; ablation: −0.017; verify_q32.json) and a clear reduction for ablation. An audit of the code, without new measurement, explains why. ResidualIntervention wraps the decoder block's forward and edits its output; block i produces hidden_states[i+1]. Production add therefore acts on the output of block 34, i.e. hidden_states[35]. verify reads hidden_states[34], the output of block 33, which is before the add site and by causality cannot change. Ablation is visible because it acts on all blocks up to the read. The self-test uses a different index (it adds at block 33 and reads hidden_states[34]), so it validates the add mechanism but not the exact production configuration.

The add hook is demonstrably active in the behavioural battery (add_x2 loses 37 items under the eval framing and 34 under deploy, McNemar p = 5.3e-06 and 3.1e-03; add_random loses 36 and 35, p = 1.9e-07 and 1.4e-06), so the open point is not whether it does something but whether it adds α along the intended vector at the intended place. The pre-registered gate G8(b) checked only the ablation conditions, so the add validation is absent rather than failed. It is unresolved and is not hidden by this update: no measurement of the added projection in the production configuration exists. The minimal test (same model and revision, the same 30 OOD prompts, projection read at hidden_states[35] with [34] as negative control, written to a new file without touching the original) has not been run and is outside this update. Open validation of add does not block the other results; it qualifies every add statement below.

add_random ≈ add_x2. At 2α, the evaluation direction and a random direction produce indistinguishable accuracy losses: eval 134 vs 132 correct, deploy 137 vs 128. This is reported as no demonstrated specificity of the direction in this battery. It is not attributed to vector norm or to any mechanism (that would need a norm-matched control after add is validated). add_x1 does not move accuracy detectably.

6.6 Verbalisation at 32B

Free-generation meta-commentary on the OOD prompts (15 per framing, teacher-forced <think> openers) is at the floor: explicit mentions 0/15 in the unmodified model under both prompt conditions; one hit in 15 for ablate_layerwise and one for add_random under the eval framing (McNemar vs none not significant). The teacher-forced grid scores the log-probability of four meta-commentary openers against four neutral ones on 30 OOD prompts. Ablation at the primary layer changes the score by −0.000 (full strength, d_z ≈ 0, p = 0.987); the dose slope is 0.004 (p = 0.849). Ablating at other layers (16: −0.102, p = 0.005; 32: −0.074, p = 0.003; 48: −0.079, p = 0.090) and the magnitude-matched random control (−0.053 at full strength) move the score more than the primary-layer ablation does.

Specificity percentages are omitted on purpose. The grid output computes "control change as a fraction of the real change"; with a real change of −0.000 the denominator is almost zero and such a ratio is arbitrarily large (the file contains −29.6 and +126). It is neither a result nor a measure of specificity, so it is not quoted. The honest summary is that ablating the global direction at the layer where it was fitted does not detectably change this teacher-forced measure, and that other perturbations do.

6.7 Operations, deviations and history

The run was executed on one A100 80 GB (Google Colab session) and had to be restarted on fresh VMs. Documented points: (1) the intervention self-test uses a bf16 tolerance floor multiplied by K = 10, calibrated data-independently before the freeze (pre-registration §11.1; the maximum observed ratio to the single-rounding floor was 4.62); (2) three restarts within five minutes (11:11, 11:13 and 11:15 UTC on 18 October 2026): on each, stale stage markers and result files were removed because s0_env had been marked done on a fresh VM while data files were missing, and the criteria and summary stages had been run with NOT_RUN labels after failed stages; (3) the final criteria_evaluation.json and RESULT.json were regenerated by the completed run (all stages ok). The earlier NOT_RUN blocks remain in the stage log and are operational history, not results. No code, configuration or threshold changed. Details and the three resume notes are in OPERATIONAL_HISTORY.md and docs/q32/ops/.

6.8 Artifacts and what is not published

• docs/q32/REPORT_QWEN3_32B.md — the final run report (methods, deviations, results, limits).
• PREREG_QWEN3_32B.md, KRYTERIA_DO_ZATWIERDZENIA.md (the approval document, in Polish), RUN_CONFIG.json, MANIFEST.json — frozen protocol, unmodified.
• results/q32/ — safe aggregates only (JSON/CSV), including criteria_evaluation.json, the three-model comparison, the interaction table and every number quoted in §6. A machine-readable verification of the quoted values against the aggregates is in verification_q32.json.
• code/q32/ — analysis/ (interaction statistics, public data builder, figure code, aggregate verifier) and frozen_pipeline/ (the code that produced the run).
• docs/q32/INDEX.md — a file index with sha256 of every published file, separating the measurement, the extra analysis and the publication steps.

Not published: the 32B activations and per-prompt rows (they derive from EvalAwareBench prompts, CC BY-NC 4.0, and sit in a private dataset), the rendered prompts, and generated text beyond the four short openers listed in the aggregates. The per-item rows of the 200-item battery are summarised, not redistributed.

6.9 Update history

• 18 October 2026. Qwen3-32B added: §1.8, §6, new TL;DR items, dated update paragraphs under Q1–Q5, the causal section, verbalisation and the cross-family table, extra Limitations, new figures 8–13, results/q32/, docs/q32/, code/q32/. In that revision only the byline's date and model list were edited in existing text; no earlier number, figure, file or correction was changed.
• 18 October 2026 (second revision, editorial). Wording only, no number changed: the TL;DR item "No sandbagging, at any scale tested" now reads "No statistically detectable sandbagging in the tested settings"; the interpretation "no framing effect larger than ±0.032" was removed from the update note, §6.4, the new TL;DR item and the Limitations and replaced by the statement quoted in the update note; the frozen pre-registration and criteria_evaluation.json were not changed. Files changed in this revision are listed with hashes in docs/q32/INDEX.md.

---