Content

  1. A Demo of Direct Sound Estimation (DSE) on MISP2023 Training Set
  2. Ablation Results of SuPseudo Training on MISP2023 Evaluation Set (SE Model: FARNET)
  3. Comparison with Other Front-end Systems on MISP2023 Evaluation Set

    4. Notes:

    [ Detailed description of the MISP2023 corpus/Challenge can be found at ICASSP 2024 MISP Challenge. ]

    [ Sorry for that the demos below are all in Mandarin. ]

Ⅰ. A Demo of Direct Sound Estimation (DSE) on MISP2023 Training Set

Utterance ID 1: S072-R03_S072073_C01_I1_Far-112160_112296

close-talk speech signal Spectrogram for Near
GSS signal Spectrogram for GSS
DSE signal
(i.e., pseudo-label) Spectrogram for GSS

Utterance ID 2: S080-R03_S079080081_C02_I0_Far-088488_088608

close-talk speech signal Spectrogram for Near
GSS signal Spectrogram for GSS
DSE signal
(i.e., pseudo-label) Spectrogram for GSS

Utterance ID 3: S158-R08_S158159160_C02_I0_Far-030616_030860

close-talk speech signal Spectrogram for Near
GSS signal Spectrogram for GSS
DSE signal
(i.e., pseudo-label) Spectrogram for GSS

Utterance ID 4: S274-R52_S274275276277278279_C06_I1_Far-079404_079700

close-talk speech signal Spectrogram for Near
GSS signal Spectrogram for GSS
DSE signal
(i.e., pseudo-label) Spectrogram for GSS

Utterance ID 5: S305-R55_S303304305_C03_I0_Far-011072_011344

close-talk speech signal Spectrogram for Near
GSS signal Spectrogram for GSS
DSE signal
(i.e., pseudo-label) Spectrogram for GSS

Ⅱ. Ablation Results of SuPseudo Training on MISP2023 Evaluation Set (SE Model: FARNET)

Utterance ID 1: S121-R04_S120121122_C07_I0_Far-126992_127472

GSS signal
(ID: B1) Spectrogram for Near
supervised training on simulated data
(ID: M2) Spectrogram for Supervised
SuPseudo training on real-recorded data
(ID: M3) Spectrogram for Pseudo-Supervised
SuPseudo training on both simulated and real-recorded data
(ID: M4d) Spectrogram for SuPseudo

Utterance ID 2: S126-R04_S126127_C09_I1_Far-001864_001988

GSS signal
(ID: B1) Spectrogram for Near
supervised training on simulated data
(ID: M2) Spectrogram for Supervised
SuPseudo training on real-recorded data
(ID: M3) Spectrogram for Pseudo-Supervised
SuPseudo training on both simulated and real-recorded data
(ID: M4d) Spectrogram for SuPseudo

Utterance ID 3: S126-R04_S126127_C09_I1_Far-069436_069736

GSS signal
(ID: B1) Spectrogram for Near
supervised training on simulated data
(ID: M2) Spectrogram for Supervised
SuPseudo training on real-recorded data
(ID: M3) Spectrogram for Pseudo-Supervised
SuPseudo training on both simulated and real-recorded data
(ID: M4d) Spectrogram for SuPseudo

Utterance ID 4: S129-R04_S128129_C12_I2_Far-026540_026836

GSS signal
(ID: B1) Spectrogram for Near
supervised training on simulated data
(ID: M2) Spectrogram for Supervised
SuPseudo training on real-recorded data
(ID: M3) Spectrogram for Pseudo-Supervised
SuPseudo training on both simulated and real-recorded data
(ID: M4d) Spectrogram for SuPseudo

Utterance ID 5: S131-R04_S130131_C02_I2_Far-010336_010920

GSS signal
(ID: B1) Spectrogram for Near
supervised training on simulated data
(ID: M2) Spectrogram for Supervised
SuPseudo training on real-recorded data
(ID: M3) Spectrogram for Pseudo-Supervised
SuPseudo training on both simulated and real-recorded data
(ID: M4d) Spectrogram for SuPseudo

Utterance ID 6: S349-R59_S346347348349_C06_I1_Far-025836_026288

GSS signal
(ID: B1) Spectrogram for Near
supervised training on simulated data
(ID: M2) Spectrogram for Supervised
SuPseudo training on real-recorded data
(ID: M3) Spectrogram for Pseudo-Supervised
SuPseudo training on both simulated and real-recorded data
(ID: M4d) Spectrogram for SuPseudo

Ⅲ. Comparison with Other Front-end Systems on MISP2023 Evaluation Set

Utterance ID 1: S126-R04_S126127_C09_I1_Far-049840_050152

1st channel of far-field microphone array signal Spectrogram for Near
GSS [1]
(widely used front-end system) Spectrogram for Near
GSS+MEASE [2]
(challenge baseline) Spectrogram for GSS
XMU-system [3]
(third-ranked system) Spectrogram for GSS
GSS+FARNET+SuPseudo training
(ours) Spectrogram for GSS

Utterance ID 2: S127-R04_S126127_C09_I1_Far-056348_056596

1st channel of far-field microphone array signal Spectrogram for Near
GSS [1]
(widely used front-end system) Spectrogram for Near
GSS+MEASE [2]
(challenge baseline) Spectrogram for GSS
XMU-system [3]
(third-ranked system) Spectrogram for GSS
GSS+FARNET+SuPseudo training
(ours) Spectrogram for GSS

Utterance ID 3: S128-R04_S128129_C12_I2_Far-147124_147768

1st channel of far-field microphone array signal Spectrogram for Near
GSS [1]
(widely used front-end system) Spectrogram for Near
GSS+MEASE [2]
(challenge baseline) Spectrogram for GSS
XMU-system [3]
(third-ranked system) Spectrogram for GSS
GSS+FARNET+SuPseudo training
(ours) Spectrogram for GSS

Utterance ID 4: S189-R11_S189190191192_C10_I2_Far-051932_052228

1st channel of far-field microphone array signal Spectrogram for Near
GSS [1]
(widely used front-end system) Spectrogram for Near
GSS+MEASE [2]
(challenge baseline) Spectrogram for GSS
XMU-system [3]
(third-ranked system) Spectrogram for GSS
GSS+FARNET+SuPseudo training
(ours) Spectrogram for GSS

Utterance ID 5: S349-R59_S346347348349_C06_I1_Far-084992_085588

1st channel of far-field microphone array signal Spectrogram for Near
GSS [1]
(widely used front-end system) Spectrogram for Near
GSS+MEASE [2]
(challenge baseline) Spectrogram for GSS
XMU-system [3]
(third-ranked system) Spectrogram for GSS
GSS+FARNET+SuPseudo training
(ours) Spectrogram for GSS

References

  1. Raj, D., Povey, D., Khudanpur, S. (2023) GPU-accelerated Guided Source Separation for Meeting Transcription. Proc. Interspeech 2023, 3507-3511, doi: 10.21437/Interspeech.2023-42
  2. Wu S, Wang C, Chen H, et al. The multimodal information based speech processing (misp) 2023 challenge: Audio-visual target speaker extraction[C]//ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024: 8351-8355.
  3. Luo L, Li T, Li L, et al. The XMUSpeech system for audio-visual target speaker extraction in MISP 2023 challenge[C]//2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). IEEE, 2024: 39-40.