This paper describes a system that can automatically annotate videos and illustrates its application to tennis games. A unified apparatus is proposed, cast in a Bayesian reasoning framework. This is supported by a cognitive memory architecture that allows the system to store raw video data at the lowest cognitive level and its semantic annotation with increasing levels of abstraction up to determining the score of a game. Also embedded in the system is a set of mechanisms to detect anomalies caused by a change of domain in the input data. Once an anomaly is detected, transfer learning methods are triggered to adapt the knowledge to new domains, such as new sport modalities. We also present a generic framework for rule induction that is crucial in the context of an adaptive annotation system.