Changelog
Changelog
A dated record of what changed on this site: new benchmarks, new results, methodology updates and corrections. Version histories of individual benchmarks are on the benchmark pages.
- Results
Claude Fable 5.2, run in Claude Code, was placed first by the original authors. Every other model moves down one place.
- Benchmark
Creative Synthesis Benchmark introduced
A new task: research Jumalauta and jml-engine-webgl, then create an original audiovisual demo for expert ranking. The prompt and protocol are published; no results have been published yet.
- Results
Mistral Medium 3.5 ranked 12th
Mistral Medium 3.5, run in Mistral Vibe CLI, was evaluated and placed last by the original authors.
- Results
The original authors ranked Claude Haiku 4.5 sixth. Updated ranking: 1. Claude Opus 5.5, 2. Claude Fable 5.1, 3. GPT-6 Astra, 4. GPT-6 Sol, 5. Claude Opus 4.7, 6. Claude Haiku 4.5.
- Results
The original authors ranked Claude Sonnet 5 sixth, above Claude Haiku 4.5. Updated ranking: 1. Claude Opus 5.5, 2. Claude Fable 5.1, 3. GPT-6 Astra, 4. GPT-6 Sol, 5. Claude Opus 4.7, 6. Claude Sonnet 5, 7. Claude Haiku 4.5.
- Results
DeepSeek V4 Pro ranked, Gemini 3.8 Flash disqualified
DeepSeek V4 Pro was ranked tenth. Gemini 3.8 Flash downloaded the original demo during its run and was disqualified; its output is published, marked as a rule violation, without a rank.
- Results
Grok 4.7 ranked, DeepSeek V4.1 Flash disqualified
Grok 4.7 was ranked eighth. DeepSeek V4.1 Flash read the original release's info file on pouet.net during its run and was disqualified; its output is published, marked as a rule violation, without a rank.
- Results
Kimi K3 and Qwen3.8 Max evaluated
The first models from Chinese labs, each run in its maker's own agent harness. Updated ranking: 1. Claude Opus 5.5, 2. Claude Fable 5.1, 3. GPT-6 Astra, 4. Kimi K3, 5. GPT-6 Sol, 6. Qwen3.8 Max, 7. Claude Opus 4.7, 8. Claude Sonnet 5, 9. Claude Haiku 4.5.
- Results
One video with the original and every reimplementation in rank order, in sync, on the benchmark page.
- Results
The original authors ranked GPT-6 Sol fourth. Updated ranking: 1. Claude Opus 5.5, 2. Claude Fable 5.1, 3. GPT-6 Astra, 4. GPT-6 Sol, 5. Claude Opus 4.7.
- Results
The original authors ranked Claude Opus 5.5 first. Updated ranking: 1. Claude Opus 5.5, 2. Claude Fable 5.1, 3. GPT-6 Astra, 4. Claude Opus 4.7.
- Benchmark
Creative Reimplementation Benchmark v1.0
First publication of the benchmark: three models, one prompt, single shot, ranked by the original authors of the reference demo.
- Results
The original authors of the reference demo ranked the three reimplementations: 1. Claude Fable 5.1, 2. GPT-6 Astra, 3. Claude Opus 4.7. Their notes are on the benchmark page.
- Results
Recordings added for three models
Side-by-side comparison recordings published for Claude Fable 5.1, Claude Opus 4.7 and GPT-6 Astra.
- Site
Site published at ai.jumalauta.org