Audio-visual highlight: three RoPE-aligned checkpoints

Twenty clips from the avhl test-200 list (test_0902_clean3), run through last.ckpt of three runs that add cross-attention RoPE time alignment (xattn_rope_time_align). Inference: SB flow, t_start 0.02, 10 Euler steps, the same settings as the other test-200 arms. Chips give waveform audio PSNR of the generated clip minus pass-through (source vs target), so a positive value means the edit moved the audio closer to the target than doing nothing.

Checkpoint (last.ckpt)	Single · 8	Two · 6	Failure · 6	All · 20
0902 + RoPE alignσ 0.01 · shared t · ep104, ~6.5k steps	-0.812/8	-1.092/6	-5.330/6	-2.254/20
shared-t + RoPE alignσ 1 · shared t · ep103, ~6.4k steps	+0.312/8	-0.371/6	-1.911/6	-0.564/20
sigma1 sept + RoPE alignσ 1 · separate t · ep174, ~10.9k steps	+1.065/8	+0.555/6	-3.310/6	-0.4010/20

Mean audio PSNR gain over pass-through (dB); small number = clips with a positive gain. With 6–8 clips per group this shows direction only. An earlier 84-clip run found about ±0.4 dB of noise from sample choice alone, so it doesn't rank the checkpoints. The runs were trained from scratch on 2026-09-11 and are at different steps.

Large single object

8 clips

One visible instance, highlight box covering 64–79% of the frame, detector IoU ≥ 0.6.

harp

c4qetxfWTlM_000030

1 visible
box 79% of frame
match

Sourceinput · pass-through 19.0 dB

Targethighlighted ground truth

0902 + RoPE align-1.43 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-0.63 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+0.28 dBσ 1 · separate t · ep174, ~10.9k steps

water

GBf5DgubSuE_000030

1 visible
box 75% of frame
match

Sourceinput · pass-through 17.0 dB

Targethighlighted ground truth

0902 + RoPE align+0.15 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align+2.22 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+1.14 dBσ 1 · separate t · ep174, ~10.9k steps

banjo

io94WhQJZ5k_000095

1 visible
box 73% of frame
match

Sourceinput · pass-through 21.5 dB

Targethighlighted ground truth

0902 + RoPE align-1.01 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-1.42 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+2.01 dBσ 1 · separate t · ep174, ~10.9k steps

grand piano

4qaPtkgKSNE_000074

1 visible
box 72% of frame
match

Sourceinput · pass-through 16.3 dB

Targethighlighted ground truth

0902 + RoPE align-3.30 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-1.81 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-1.72 dBσ 1 · separate t · ep174, ~10.9k steps

female vocalist

I4_A-NlyDIk_000030

1 visible
box 67% of frame
match

Sourceinput · pass-through 18.9 dB

Targethighlighted ground truth

0902 + RoPE align-1.44 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-1.07 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+0.17 dBσ 1 · separate t · ep174, ~10.9k steps

piano

HmfCGoJGMiY_000160

1 visible
box 66% of frame
match

Sourceinput · pass-through 14.5 dB

Targethighlighted ground truth

0902 + RoPE align-0.71 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-2.54 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-0.15 dBσ 1 · separate t · ep174, ~10.9k steps

motorcycle

N-LO3sRsxyw_000030

1 visible
box 64% of frame
match

Sourceinput · pass-through 17.4 dB

Targethighlighted ground truth

0902 + RoPE align+3.27 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align+8.33 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+6.91 dBσ 1 · separate t · ep174, ~10.9k steps

flowing water

Q4OqL84-pK0

1 visible
box 64% of frame
match

Sourceinput · pass-through 31.4 dB

Targethighlighted ground truth

0902 + RoPE align-2.01 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-0.57 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-0.15 dBσ 1 · separate t · ep174, ~10.9k steps

Large two objects

6 clips

Two visible instances; the largest two-object highlights in the test-200 set.

tools

heAGQ0YfuwQ

2 visible
box 77% of frame
match

Sourceinput · pass-through 24.3 dB

Targethighlighted ground truth

0902 + RoPE align-3.45 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-2.99 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-2.04 dBσ 1 · separate t · ep174, ~10.9k steps

water

IdjAYKYjepE_000222

2 visible
box 41% of frame
match

Sourceinput · pass-through 28.5 dB

Targethighlighted ground truth

0902 + RoPE align-3.95 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-0.29 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+0.01 dBσ 1 · separate t · ep174, ~10.9k steps

tabla

eiPwk5lxnuQ_000167

2 visible
box 40% of frame
match

Sourceinput · pass-through 18.6 dB

Targethighlighted ground truth

0902 + RoPE align+0.39 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-1.20 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+1.78 dBσ 1 · separate t · ep174, ~10.9k steps

birds

KpNcKnxBGlA

2 visible
box 37% of frame
match

Sourceinput · pass-through 21.4 dB

Targethighlighted ground truth

0902 + RoPE align+2.11 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align+3.32 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+3.02 dBσ 1 · separate t · ep174, ~10.9k steps

individuals

afJFNGBfNgU_000070

2 visible
box 35% of frame
match

Sourceinput · pass-through 15.1 dB

Targethighlighted ground truth

0902 + RoPE align-1.37 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-0.80 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+0.22 dBσ 1 · separate t · ep174, ~10.9k steps

cymbal

mV8UjOGRhO8_000070

2 visible
box 27% of frame
match

Sourceinput · pass-through 22.9 dB

Targethighlighted ground truth

0902 + RoPE align-0.26 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-0.24 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align+0.33 dBσ 1 · separate t · ep174, ~10.9k steps

Failure cases

6 clips

Picked before this run: the six clips where the earlier sigma1_sept best checkpoint lost most audio PSNR against pass-through.

vehicle engine

i-CUmPO9ad4

1 visible
box 37% of frame
match
earlier Δ -6.6 dB

Sourceinput · pass-through 16.7 dB

Targethighlighted ground truth

0902 + RoPE align-4.60 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-2.59 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-4.39 dBσ 1 · separate t · ep174, ~10.9k steps

large metal object

plz9-OYCLYU_000049

1 visible
box 50% of frame
match
earlier Δ -6.3 dB

Sourceinput · pass-through 24.5 dB

Targethighlighted ground truth

0902 + RoPE align-5.89 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-4.14 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-5.66 dBσ 1 · separate t · ep174, ~10.9k steps

gong

RVOpQ5tOChY_000085

1 visible
box 17% of frame
match
earlier Δ -6.2 dB

Sourceinput · pass-through 27.5 dB

Targethighlighted ground truth

0902 + RoPE align-4.76 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-4.74 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-4.81 dBσ 1 · separate t · ep174, ~10.9k steps

water

eW7_vMFcGRA_000379

1 visible
box 31% of frame
match
earlier Δ -5.4 dB

Sourceinput · pass-through 30.9 dB

Targethighlighted ground truth

0902 + RoPE align-7.27 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align+2.58 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-1.55 dBσ 1 · separate t · ep174, ~10.9k steps

water

yc7g0cGr4Rc_000075

1 visible
box 85% of frame
global
earlier Δ -3.9 dB

Sourceinput · pass-through 26.5 dB

Targethighlighted ground truth

0902 + RoPE align-4.22 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-2.33 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-2.86 dBσ 1 · separate t · ep174, ~10.9k steps

cattle

QDEBsXt1iAo_000216

1 visible
box 10% of frame
mismatch
earlier Δ -3.8 dB

Sourceinput · pass-through 37.1 dB

Targethighlighted ground truth

0902 + RoPE align-5.21 dBσ 0.01 · shared t · ep104, ~6.5k steps

shared-t + RoPE align-0.23 dBσ 1 · shared t · ep103, ~6.4k steps

sigma1 sept + RoPE align-0.57 dBσ 1 · separate t · ep174, ~10.9k steps

Media: h264 256×256 at 24 fps with AAC audio re-encoded from the inference outputs, 8 s per clip. Box coverage and verdict come from the highlight visual check (hlvis_verdicts.csv); object count comes from GroundingDINO.