Abstract Nowadays the huge amount of video material stored in multimedia repositories makes the search and retrieval of such content a very slow and usually difficult task. The existing video abstraction systems aim to ease the multimedia access problem by providing short versions of the original content which ease the search and navigation process by reducing the time spent in content browsing. There are many video abstraction system variations providing different kinds of output (video skims, keyframe-based summaries, etc). This paper presents a generic video abstraction architecture which aims characterize the stages required for a generic abstraction process as well as the definition of the theoretical aspects and requirements for the modelling of video abstraction systems which is a first step before building abstraction systems with specific characteristics.