In this paper, we propose a framework that can be used for detecting relevant changes in highly dynamic scenes, where the background has several changing elements. To establish a clear distinction between what is relevant and what is not is a very challenging task. Therefore, we first categorize the changes into two main classes called ordinary changes and relevant changes. Detected changes are considered as irrelevant if they are recurrent elements and changes pertaining on the dynamic background of the scene. The proposed framework makes use of a set of orthogonal linear transforms to capture spatiotemporal signatures of local ordinary change patterns and subsequently employ them in the detection of relevant changes. The use of this framework is demonstrated in a variety of videos with highly dynamic backgrounds including lakes, pools, and roads. Compared to existing methods reported on the same test videos, the proposed framework detects the relevant changes more accurately.