JumalAIta IntelligenceIndependent evaluation

Benchmark 02 / Creative intelligence

Creative Synthesis Benchmark

v2.0PreliminaryUpdated · 0 models · Expert-ranked

Research Jumalauta. Learn its demo engine. Create an original demo. Each model uses jml-engine-webgl to make a new Jumalauta demo. Experts rank the works on how well each one works as a new Jumalauta demo and how well it uses JML Engine.

Measures How a model discovers a creative tradition, learns an unfamiliar engine and turns that understanding into an original audiovisual work with a distinct identity.

Generation
Single shot
Required engine
jml-engine-webgl
Creative task
Original demo
Evaluation
Expert ranking

01 / Leaderboard

Leaderboard

v2.0 · Expert-ranked

No results have been published yet.

02 / Key takeaways

Key takeaways

    No findings have been published yet.

    03 / Recordings

    Recordings

    No recordings have been published yet.

    04 / The task

    The task

    The model must work out what makes a good Jumalauta demo and how to build one with jml-engine-webgl. It researches multiple productions, learns the engine and develops its own concept, visual material, scenes and audiovisual structure. There is no single reference work to reproduce and no prescribed theme, effect list or visual style.

    Deliverable
    A complete, original browser demo with sound
    Runtime
    jml-engine-webgl
    Supporting material
    Source, assets, run instructions, research notes and asset credits
    Reference to reproduce
    None
    Music selection
    One of eight supplied tracks, chosen by the model

    Research resources

    05 / Prompt

    Prompt

    The benchmark prompt, verbatim
    Make a new Jumalauta demo with Jumalauta's own engine, JML Engine: https://github.com/jumalauta/jml-engine-webgl. Use the engine and example revisions pinned in research-sources.json.
    
    A demo here means what it means on the demoscene: a real-time audiovisual production that runs full screen from start to finish without interaction, its visuals computed live and choreographed to the music. Not a slideshow, a web page, a video or an interactive toy.
    
    Jumalauta is a Finnish demogroup with more than 600 productions since 2000: https://demozoo.org/groups/1297/. Watch many of them, old and new, across platforms, and read their titles, texts and credits. The engine examples in research-sources.json show how the engine is used; they do not define the group. Work out for yourself what makes a production a Jumalauta demo, then make a new one: one the group could release under its own name at a party. A name or a logo does not make it one, and neither does technical polish on its own. Do not make a demo about Jumalauta or a tribute to it.
    
    Learn the engine properly from its source, documentation and examples, and use it well. Its loader, scenes, animation, timing and rendering run the whole demo; custom shaders and generated assets go through it. No separate renderer and no prerecorded video.
    
    rules:
    - music: exactly one track from music-catalog.json, in full and unchanged, and no other audio. The demo lasts as long as the track. Include the credits from the track's credits file
    - everything else is yours: concept, title, texts, visuals, effects and synchronization. Do not copy, port or reskin an existing production
    - it runs in a browser, starts with one click and plays to the end on its own
    - play it through with audio before you finish, and fix what you find
    - work on your own; do not ask for direction
    
    Leave a README with how to run it, the track you chose and the productions you studied.
    
    Experts familiar with Jumalauta rank the results on two things: how well it works as a new Jumalauta demo, and how well it uses JML Engine.
    Inputmusic-catalog.jsonThe same eight-track catalogue, source descriptions, licenses, durations and hashes for every model
    Inputmusic/Eight MP3 files, each with credits, source-based notes and explicitly estimated timing data
    Inputresearch-sources.jsonFixed engine, example collection and 15 example-submodule revisions

    This is the complete prompt, reproduced verbatim. Every model receives exactly this text together with the listed inputs.

    To test another model, copy the prompt and follow the shared conditions in the evaluation protocol.

    The benchmark kit contains the prompt, licensed music pack, fixed source revisions, input checksums and the shared Docker runner with external network capture. The kit is not yet public.

    06 / Evaluation protocol

    Evaluation protocol

    The generation ceiling increased from 240 to 480 minutes after the first local round. That round retains its original 240-minute run records. Report the budget version with results; do not pool rounds with different ceilings as if their resource conditions were identical.

    Every model receives the prompt reproduced on this page and the same music catalogue, track files and source-revision manifest listed above in a fresh Docker workspace, with access to the engine and example repositories. Research is part of the run. No scene plan, style recipe or prewritten demo is supplied. The model may inspect documentation and source, run examples, study recordings, create assets, test and revise autonomously. There is one run per model and no human creative direction after the initial prompt.

    Before the first evaluated run, select and record a common engine commit and an example-collection commit, including the linked demo submodule revisions. Give every model in that evaluation round access to those same revisions. Record the environment, browser, available research and asset tools, harness configuration and any access failures. Preserve each run's transcript, submitted source, dependency lockfiles, assets and instructions. The kit locks all input hashes before evaluated runs and copies the manifest outside the model's workspace.

    Monitored Docker environment

    Use the shared benchmark runner for every evaluated run. Its base image is Debian 13 with Node.js 26, Python 3 with NumPy and Pillow, FFmpeg, ImageMagick, Git and common build tools. The model's own harness is installed with its official installer. No browser is preinstalled; the model may install one. The model works as an unprivileged user with passwordless sudo and internet access inside its container, with factory-fresh harness settings and only its login restored.

    Only the fresh task workspace is shared from the host. The kit's tools, other submissions, host configuration, Docker socket and monitoring files are not mounted into the model's container. Permissions are auto-approved inside the container. The shared runner enforces the 480-minute limit and records the harness version, image identity, prompt hash, kit and runner revisions, elapsed time, exit status and redacted transcript outside the writable task workspace.

    An independent network-capture sidecar starts before the harness. The model's container joins its network namespace without network administration or raw-packet capabilities. Capture records include DNS, destination addresses, TLS hostnames where visible and plaintext HTTP requests. The model cannot edit the saved log or control the sidecar through its process or mount namespace. Missing or incomplete capture must be investigated before an entry can be accepted as a monitored run.

    HTTPS paths and encrypted content are not visible to this network capture. Research performed by a provider-side tool may only appear in the transcript. Inspect both sources alongside the submission; network logs alone do not prove originality. Accessing demo archives and examples is allowed here, so the reimplementation benchmark's forbidden-source list does not apply. The restriction is on copying the creative work into the submission, not on researching it.

    Eligibility and originality

    Run the submitted project using its instructions and inspect the actual rendering path. The JML engine must load the demo and drive its scenes, timing and rendering. Custom code and shaders may extend that path. A token engine import or an independent application launched from its player does not meet the requirement.

    Check the source, assets, research notes and run transcript against the examples the model studied. Look beyond identical files: compare the central concept, distinctive scene construction, ordering, transitions and synchronization. A renamed or recoloured production, a close reconstruction, or a collage of copied scenes violates the originality rule. Standard API usage, general graphics techniques and disclosed supporting resources are not evidence of copying by themselves. Assess substantial creative reuse with specific source or scene comparisons, rather than a similarity percentage alone. Uncertain similarity is not proof of copying.

    Document rule violations separately from creative judgment. Confirmed violations of the required engine or originality rules are disqualified and receive no rank; retain the submitted output and publish the reason with evidence. Record playback failures and missing assets as observed. Do not repair a model's submitted work before evaluating or recording it.

    Expert ranking

    Experts familiar with Jumalauta demos rank eligible entries as complete audiovisual works on two things: how well each one works as a new Jumalauta demo, and how well it uses JML Engine. Their judgment considers how convincingly the demo expresses its own idea within that creative context, how substantially it differs from the examples, and how its imagery, sound, rhythm, progression and ending work together. Execution includes the effectiveness of the effects, readability, continuity, playback reliability and synchronization. These are considerations for a holistic ranking, not independent scores or a weighted checklist.

    The prompt deliberately supplies no definitive account of Jumalauta's style. Understanding that context and making creative decisions from it are part of the task. A familiar logo, a collection of standard effects or a polished technical showcase cannot establish that understanding on its own. Research notes help reviewers inspect the model's choices; the finished work is the evidence for its creative quality.

    Watch each submitted demo from beginning to end with its own soundtrack under the same playback conditions. Publish the ordinal ranking and the experts' actual reasons, with concrete moments from the works where useful. No numerical scores, sub-scores or confidence intervals are produced. Evaluator identities and panel details remain unpublished until supplied; this protocol does not claim that an evaluation has already taken place.

    Shared music pack

    Every model receives the same eight-track pack and chooses exactly one track. All tracks are by Kevin MacLeod and licensed under CC BY 4.0. The composer's usage guidance permits YouTube use with attribution. The pack includes the original MP3 files, track source URLs, required credits, measured durations and SHA-256 checksums. Preserve its license evidence with the kit.

    Track and listening page File duration, rounded Publisher BPM
    Harmful or Fatal 5:31 135
    Rock Hybrid 2:10 116
    Jet Fueled Vixen 2:40 100
    In a Heartbeat 3:38 130
    Blue Scorpion 3:18 130
    Industrial Cinematic 4:04 110
    The Complex 4:28 104
    Cool Hard Facts 2:17 116

    The catalogue's character and instrumentation descriptions are based on the composer's metadata, not on listening by the benchmark preparer. The accompanying timing files contain approximate spectral-flux onset candidates and one-second RMS energy measurements. They explicitly disclose that they have not received a listening review. They are not verified beats, downbeats or musical section boundaries, and prescribe no visual concept or scene sequence. Every model receives the same information and may analyze the audio further with its own tools.

    Native audio perception is not assumed. Record the actual harness's listening support and the audio tools used. Programmatic checks can confirm loading and clock progression, but cannot establish perceived sound quality. Experts listen to the finished work. This benchmark measures creation using supplied musical context, not unaided music perception.

    The composer uses Content ID. Include the supplied attribution in each demo and video description and resolve any claims before publication. A license permits use but does not guarantee the absence of automated claims. No test upload or claim clearance is asserted here.

    Record the model's chosen track ID and file hash with its result. Do not force different tracks across models or rerun a model because its choice matches another entry. Models must not substitute music or add generated audio. Experts judge how the chosen music and visuals work together; they do not attribute the supplied composition's quality to the model.

    Recordings and reproducibility

    Record each complete output with its own audio, from the initial start through its ending, including visible failures. Keep capture conditions consistent and publish the browser, viewport and capture settings with the evaluation round. Record actual runtime behaviour; do not replace missing visuals or sound in editing. There is no original production to put beside the output for a fidelity comparison.

    Retain the source revisions, run details and final artifacts needed to reproduce each result. Record cost, token use, elapsed time and tool calls when available. Report missing information as unpublished. A change to the prompt, fixed research inputs or evaluation method requires a new major benchmark version; results from different task versions must not be presented as directly comparable.

    Publication status

    Prompt
    Published verbatim
    Research resources
    Published
    Recordings
    Not yet published
    Ranking
    Not yet published
    Evaluators' notes
    Not yet published
    Scores
    None, by design
    Evaluators
    Experts familiar with Jumalauta demos
    Number, names and roles of evaluators
    Not published
    Harness names, run dates, cost and tokens
    Not yet published
    Benchmark kit
    Not yet public

    Shared conditions

    Generation
    Single shotOne initial prompt. The model may research, test and revise within its run, without human follow-up.
    Required engine
    jml-engine-webglThe engine must load and play the demo and control its scenes, timing and rendering.
    Creative task
    Original demoExisting demos are research material. Copying, reskinning or reconstructing one is not permitted.
    Evaluation
    Expert rankingExperts judge complete works. No numerical scores or weighted rubric.
    Reasoning effort
    HighExtra-high and max settings are excluded.
    Harness setup
    VanillaDefault configuration of the model maker's own agent harness, as in the general methodology.
    Runs per model
    OneNo retries or best-of-N selection. Autonomous iteration within the run is allowed.
    Research access
    Permitted and requiredRead engine documentation and source, inspect multiple Jumalauta demos and study their audiovisual output.
    Soundtrack
    Shared eight-track music packEvery model chooses one complete track from the same licensed collection. The chosen track is recorded.
    Music context
    Source descriptions and audio measurementsPublisher descriptions, tempo metadata and estimated onsets and energy are supplied. Native audio listening is not assumed.
    Run environment
    Monitored Docker containerShared benchmark runner, fresh harness state, external network capture and saved transcript.
    Duration
    Full supplied soundtrackThe demo must cover its chosen track from beginning to end, at the original speed.
    Time limit
    480 minutesThe shared runner stops the container at the deadline and records the timeout.
    Cost and token budgets
    Not specifiedRecord actual resource use where available.

    07 / Limitations

    Limitations

    Results are not yet published
    No model results, recordings or expert findings have been published for this benchmark. Evaluator identities and panel size have not yet been published. The kit has been prepared, but no paid model run or expert evaluation is claimed.
    Expert judgment
    The ranking reflects the experts' judgment of whole works. It is subjective, does not measure the distance between entries and is not a general measure of intelligence. No blind evaluation is claimed.
    Public examples and prior exposure
    The engine and examples are public and may be present in training data. Research notes and source inspection can expose copying but cannot prove that every creative choice is novel.
    Research and tool access
    Repository contents, web availability and tool capabilities can change. Runs must record their source revisions and available tools. Different access to browsing, audiovisual inspection or asset generation can affect the result. Shared music notes support agents without audio input, but do not reproduce the full experience of hearing the music.
    Choice of soundtrack
    Models may choose different tracks and durations. The ranking judges the finished demo and its musical interpretation, not the composition quality of the supplied music. This does not isolate visual quality under an identical soundtrack; the selected track must be disclosed with each result.
    One open-ended task
    One run per model does not measure run-to-run variance. The selected soundtrack fixes the demo's duration. Runs share a 480-minute ceiling, but token and cost budgets are not fixed; this is not a comparison at equal token use or cost.

    08 / FAQ

    FAQ

    Can every model actually listen to the soundtrack?
    No. Access to an audio file does not imply native audio understanding in an agent harness. Every model receives the same publisher descriptions and programmatic timing estimates. Runs disclose whether the model could listen or only analyze the file programmatically. Experts listen to the finished work when judging it.
    Why is the soundtrack supplied?
    The task measures demo creation and audiovisual synchronization. Every model receives the same eight-track collection and chooses a track for its concept. The CC BY 4.0 music can be distributed with the demo and used in recordings with the required attribution. Music generation is outside the task; choosing and interpreting a track are creative decisions.
    What makes this different from Creative Reimplementation?
    Creative Reimplementation asks the model to interpret one supplied work. This task asks it to research a creative tradition and an engine, then make a new work of its own. Similarity to an existing production is not the objective.
    Why not define the Jumalauta style in the prompt?
    Discovering that context is part of the task. A prescribed motif list would supply part of the answer and encourage surface imitation. Experts assess whether the model's own creative choices show understanding, using the complete demo as evidence.
    What may be reused?
    The engine and its bundled resources, API patterns and general techniques are permitted. Credited utilities, fonts and shared identity assets may support the work. Music is supplied by the benchmark. Existing demo-specific scenes, effects, visual assets, scene sequences and synchronization must not be submitted as a new production.
    Does working code guarantee a good ranking?
    No. Correct engine use makes the entry eligible for consideration. Experts rank the complete creative work, including its identity, composition, sound, pacing and execution. Technical complexity or visual polish alone does not determine the order.

    09 / Version history

    Version history

    1. v2.0Preliminary
      • Rewrote the prompt. It defines a demo, widens the research to the group's whole output and names the two ranking criteria: the work as a new Jumalauta demo and its use of JML Engine. Results are not comparable with version 1 runs.
    2. v1.1Preliminary
      • Increased the generation ceiling from 240 to 480 minutes. The first local round retains its original 240-minute records.
    3. v1.0Preliminary
      • Initial task, prompt and expert-ranking protocol. No results published yet.

    10 / How to cite

    How to cite

    Cite the benchmark together with its version. Results from different major versions are not comparable.

    BibTeX
    @misc{jumalaita2026creativesynthesis,
      author = {{JumalAIta Intelligence}},
      title  = {Creative Synthesis Benchmark},
      year   = {2026},
      url    = {https://ai.jumalauta.org/benchmarks/creative-synthesis/},
      note   = {Version 2.0, preliminary}
    }

    The principles that apply to every JumalAIta Intelligence benchmark are described in the general methodology. Changes to this site are recorded in the changelog.