KilometerAudio: Probing Multimodal Audio-Visual Understanding in Hour-Long Walking Tour Videos

Benchmark & Task Design

In the KilometerAudio task, a model receives a full hour-long walking-tour video (1–3 hours long, more than 70 minutes on average) including its audio track, together with a question and five candidate answer options, out of which only one is correct. All questions were collected with human raters and verified by our research team to require processing both the audio and visual modalities:

• Sound sources in space: Identifying what sound can be heard when passing a specific visual landmark or street location (often from an out-of-view source).
• Temporal audio localisation: Pinpointing when an acoustic event (e.g., an ambulance siren, bell, or announcement) occurs within an hour-long recording.
• Counting & multi-hop audio-visual reasoning: Counting acoustic events or using an audio cue to locate visual evidence (e.g., counting church bell chimes to infer the time of day).
• Speech & music recognition in the wild: Recognising songs performed by street buskers or understanding spoken announcements grounded in the visual scene.

Figure 2: Comparison with existing multiple-choice audio-video benchmarks. KilometerAudio features the longest videos (72.2 minutes on average) and focuses exclusively on audio-visual events; Video-MME (Long) has videos up to 1 hour long, but only a small fraction of its questions focus on audio events.

The validation split (8 questions on 6 videos) reuses the audio-related questions of the hour-long video QA benchmark from the 2024 and 2025 editions. All 500 test questions are new, and only 3 of the 218 test videos appeared in previous editions. No training split is provided, enforcing zero-shot evaluation. The evaluation metric is top-1 accuracy.

ECCV 2026 Challenge Results & Key Findings

The table below summarises the awarded entries on the KilometerAudio test set compared to the random baseline:

^†Best-performing entry overall, not eligible for monetary prizes due to Google affiliation.

• Agentic audio-visual indexing (Team IUCV, Winner — 0.782): Proposed AVRA, a training-free audio-visual reasoning agent built on Gemini 3.7 Flash. For each video, a question-independent index is pre-computed with frozen models—447 AudioSet event classes from BEATs, object detections, and multilingual OCR—allowing the agent to localise candidate time windows, inspect raw audio and video directly, and derive an answer before reading the options.
• Multi-modal verification under a strict frame budget (Team WW11, Runner-up — 0.752): Combined Gemini 3.1 Pro planning (capped at 48 frames/crops per question) with offline Whisper speech transcripts, CLAP sound-event retrieval, and Qwen3-Omni audio verification, updating answers only when at least two independent modalities agreed.
• Self-evolving tool compilation (Team CloudAI-evolve, Best-performing — 0.864): Used an agentic pipeline with signal-processing tools and Gemini reasoning to explore the validation set and a small unlabelled subset of test questions to compile a task-solving strategy, reaching 86.4% accuracy.
• Tool-augmented agents vs. standalone models: While expensive agentic pipelines with dedicated audio-indexing tools reached 0.782–0.864, the Best Generalist single-model entry (Team Wind_Rain_Tower, Qwen3.8-Max with speech transcription) scored 0.506—showing that hour-long audio-visual reasoning is tractable for tool-orchestrating agents but remains an open challenge for standalone multimodal models.

Citation

@misc{kitashov2026perceptiontest2026challenge,
  title={Perception Test 2026: Challenge Summary and Extension to City-scale Audio-Visual Reasoning},
  author={Fedor Kitashov and Jo{\~a}o Carreira and Shiry Ginosar and
          Dima Damen and Andrew Zisserman and Viorica P{\u{a}}tr{\u{a}}ucean},
  year={2026},
  eprint={2610.12081},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.12081}
}