After a brief discussion of the problem arising from the processing and modeling of multiple stream (multi-channel, multi-sensor) signals, we will discuss a few statistical structures (such as multi-stream HMM and asynchronous HMM) that can accommodate multiple (asynchnonous) observation streams (possibly exhibiting different frame rates). Indeed, it will be shown on different speech recognition and multimodal fusion tasks that it might sometimes be a good idea to be able to ``desynchronize'' the streams in order to maximize their joint likelihood. Different applications in speech recognition, such as multi-band and multi-stream speech processing, will be discussed. Finally, multimodal applications significantly benefiting from this multi-stream paradigm will also be discussed, including audio-visual speech recognition and modeling of human interaction in meetings (by modeling the joint behaviours of participants through multiple audio and visual features).