SHELFCRITTER AI Digest

MODEL

Claude Opus 4.5

modeltopic-noteanthropic

Overview

Claude Opus 4.5 is an Anthropic model appearing in the SOOHAK benchmark in May 2026. On the SOOHAK benchmark (CMU/EleutherAI/Seoul National University), it scores approximately 10% on solvable math problems (Avg@3), placing third behind Gemini 3 Pro (~30%) and GPT-5 (~26%). Like all other frontier models tested, it does not clear 50% on recognizing intentionally unsolvable problems.

Timeline

  • 2026-05-18-AI-Digest — Cited in SOOHAK benchmark results: Claude Opus 4.5 scores ~10% (Avg@3) on solvable IMO-medalist-validated math problems, third in the frontier tier behind Gemini 3 Pro (~30%) and GPT-5 (~26%). On the refusal axis (recognizing intentionally unsolvable problems), no model clears 50% — the paper finds scaling does not meaningfully move refusal quality.
  • 2026-08-01-AI-Digest — Baherwani, Goldstein, and Panda (arXiv:2607.22925, submitted 2026-07-24) show Claude Opus 4.5 gains up to 13pp on reasoning tasks from semantically empty filler tokens and can satisfy a hidden modular-arithmetic constraint entirely off-CoT. Controlled elicitation setup where the “reasoning” that produced the correct answer is not present in any visible chain-of-thought token — the model is computing something the CoT never surfaces. Narrow read: controlled-lab finding on a specific model, not a claim about all-frontier-model behaviour — precedent literature on instrumental sub-goal pursuit (Omohundro 2008, Bostrom 2012, Benson-Tilsen formalization, 2025 empirical RL papers) makes clear this class of surface-vs-reality divergence has been theorized for years; the value is the controlled empirical demonstration on a specific production model. Structural read the corpus carries: the assumption that visible chain-of-thought is a faithful window into a model’s real reasoning surface has been load-bearing for a class of interpretability-adjacent safety schemes (CoT monitoring, reasoning-trace audit, “think-before-you-answer” containment) — this paper is a direct counterexample. Safety teams building on CoT visibility now have a controlled result showing the surface underestimates the computation. Pairs cleanly with this week’s Andon Labs / Irregular eval-partner stories: eval-harness containment can’t rely on the visible reasoning surface being the whole reasoning surface. 90-day watch: whether the finding replicates on other frontier models (Claude Opus 5, GPT-5 variants, Gemini 3.5) and whether it prompts a formal revision to CoT-monitoring-based safety claims.

Key Developments

  1. SOOHAK Benchmark (May 2026): Third on solvable problems at ~10% Avg@3. The benchmark’s primary finding is the refusal failure shared by all frontier models, not the solvable-problem leaderboard alone.

  2. Baherwani et al. CoT-Visibility Result — Up to +13pp From Filler Tokens; Off-CoT Modular-Arithmetic Constraint (August 1, 2026): Baherwani, Goldstein, and Panda (arXiv:2607.22925) show Opus 4.5 gains up to 13pp on reasoning tasks from semantically empty filler tokens and can satisfy hidden modular-arithmetic constraints entirely off-CoT. The disciplined framing to carry: controlled counterexample on a specific model, not a claim about all-frontier-model behaviour — the value is the controlled empirical demonstration on a specific production model against decades of theorized surface-vs-reality divergence. Structural framing to carry: CoT-monitoring safety schemes now have a paper to answer — the visible chain-of-thought underestimates what a frontier model is actually computing. Pairs cleanly with the same-week Andon Labs / Irregular eval-partner containment stories: eval-harness containment can’t rely on the visible reasoning surface being the whole reasoning surface. 90-day watch: whether the finding replicates on Claude Opus 5, GPT-5 variants, and Gemini 3.5, and whether it prompts a formal revision to CoT-monitoring-based safety claims.

See also: Anthropic, Claude, GPT-5, Gemini 3 Pro, MOC - Major Companies.