Audio and captions

Audio is a separate track. A Go PCM mixer builds the full bed in memory (music, sound effects, voiceover), writes a WAV, and ffmpeg muxes it onto the video. Because it never touches renderFrame, adding audio leaves every frame byte-identical.

The audio object

{ "audio": { "silent": true } }

The audio object takes music, sting, vo, musicGain, and silent. While you author, silent: true is common: it skips music and sound entirely. The mixer runs at 44100 Hz, loops music under the video, ducks it under a voiceover, and applies a soft limiter.

Automatic sound design

The most useful mode derives sound cues from the scene's own timing:

{ "audio": { "auto": true } }

With auto, the scene reads its own layers and stings and places effects deterministically:

• a whoosh on every cut, at the cutting layer's start
• a reveal on every sting, at the sting's t

Cue times are a pure function of the JSON, so audio stays reproducible and frames stay snap-identical. You get a scored video without hand-placing a single sound.

The SFX library

Every sound is synthesized from parameters, not downloaded. The cue library is Cuelume v0.1.2 (MIT), ported verbatim: 14 interaction cues, each a small spec (noise or a tone, a filter, an envelope) that renderCue walks sample by sample at 44.1kHz with a seeded PRNG for the noise. Same spec, same seed, same bytes, on any machine, with no licence obligation and no network.

make audio        # bake every cue into assets/sfx/ (additive; --force to re-bake)
make sfx-check    # is each cue the SHAPE its role claims?

The cues: chime, sparkle, droplet, bloom, whisper, tick, press, release, toggle, success, error, page, loading, ready. whoosh and reveal are the engine's own role names (they alias whisper and chime), which is what the automatic mode reaches for. Nothing is committed; assets/sfx/ is gitignored and self-heals from make audio.

Do not hand-edit the specs in core/audio-kit.mjs. They were hand-ported once and drifted from the real library on 7 of 14 cues, which is a difference you hear and cannot debug from the JSON.

Typing

A typing layer emits one press per revealed character automatically. core/layers/text.js reveals character i at start + (i+1)/cps, so the cue is that same expression — change the copy or the speed and the clicks follow. Variation comes from dynamics (a deterministic ±12% gain per index), because a real keyboard varies in force, not in identity. keyClicks: false silences it; keyGain sets the level.

make sfx-check exists because a downloaded file named click was once 19.6 seconds long and the typing stacked it into a drone. It decodes each WAV and measures how long the signal is actually audible (not how long the file is), then holds it to a cap derived from the cue's role.

Captions

Auto-time a script into burned-in subtitles:

make captions D=formats/scene/my-video.json TEXT="your script here"

Each caption's duration is proportional to its word count, computed deterministically, and written into the scene's captions array. Pair it with "captionMode": "pop" for the muted-social look, where words punch in one at a time. The three modes are sentence, word, and pop.