Where is the plateau??

Some
thoughts on
Astra
saturating
ARC

AGI

3

diegobit
04 Sep 2026

Some thoughts on Astra saturating ARC-AGI-3

ARC-AGI-* has always been my favourite AI benchmark. I love the idea of testing AI models on tasks designed to be easily solvable by humans without specialized prior knowledge: only logic, intuition, and the ability to learn by observation. Each task is like a little, abstract and pure puzzle game.

OpenAI just released GPT-6 Astra, and its ARC-AGI-3 score jumped from ~8% with GPT-5.6 Sol (Opus 5 was at 30%) to 99% with a provider-specific harness, otherwise 63% with the standardized one! That is a lot! Astra also needs less steps than the human baseline, and it is interesting that the model solved some games while developing custom symbolic representations (eg. like chess notation).

I have two hypotheses about OpenAI's (and in part also other labs) success with their models, and both have to do with the kind of signals they are tuning their training towards: good, less-cheatable goals.

  1. Premise: the biggest step of the last year was using a kind of reinforcement learning with "verifiable feedback": problems with a definite, hard-to-get solution, where different reasoning trajectories can be explored, with a clear reward for the ones that eventually reach a verifiably correct solution. In a sense, this turns reasoning into a search problem.

  2. Pointing the model towards being good at computer use (on which OpenAI is spending a lot of effort) might be a sort of broader intelligence test. Computer use is interactive by nature, with discrete steps, some intuitive controls, and some definitely less intuitive ones.

  3. The industry has recently been moving away from caring about cost per token, and more towards measuring cost per task. Reasoning models doing agentic tasks have to act on some environment, discover how it works, think about what to do, make tests, learn, etc. A pricier model that solves the task with a fraction of the tokens of a cheaper model may turn out to be cheaper overall! A model that learns and solves a new task more efficiently is intuitively a more "intelligent" model.

This is an important point to me, because I like the definition of intelligence as the skill-acquisition efficiency: just being able to solve a task is not that important, as we might have given that capacity to a model manually (willingly or not). The ability to quickly learn a completely new task is instead real generalization, and a real path towards AGI.

Where is the plateau?? Exciting times ahead!