AISHELL8-FISHEYE: A Fisheye Audio-Visual Dataset for Target Speaker Extraction with Distortion-Aware Baselines

2026-11-09来源 arXiv eess.ASEnglish
arXiv:2611.09368v1 Announce Type: new Abstract: Fisheye cameras offer wide visual coverage but distort facial and lip geometry, challenging audio-visual target speaker extraction. We introduce AISHELL8-FISHEYE, a 47.03-hour corpus of 94 Mandarin speakers with synchronized audio and paired fisheye and normal-camera videos. Speaker-disjoint splits and synthetic two-speaker mixtures support reproducible evaluation under matched acoustic conditions. We also establish a baseline that combines a lip-reading encoder with a frozen distortion-aware branch transferred from fisheye head pose estimation. On fisheye video, adding this branch to an adapted lip encoder improves scale-invariant signal-to-distortion ratio improvement from 10.47 to 11.84 dB. The branch does not improve performance with a frozen lip encoder, highlighting the importance of visual adaptation. A substantial gap to paired normal-camera inputs remains. The corpus and baseline resources support further research on speech extraction from distorted visual cues.

原文:https://www.a8q9.cn/en/article/aishell8-fisheye-a-fisheye-audio-visual-dataset-for-target-speaker-extraction-wi-2d58de.html

相关文章