This research presents a large scale system for detection of visual concepts and annotation of images. The system is composed of two parts: feature extraction and classification/ annotation. The feature extraction part provides global and local descriptions of the images in the form of numerical vectors. Using these numerical descriptions, we train a classifier, a predictive clustering tree (PCT), to produce annotations for unseen images. PCTs are able to handle target concepts that are organized in a hierarchy, i.e., perform hierarchical multi-label classification. To improve the classification performance, we construct ensembles (bags and random forests) of PCTs.

We evaluate our system on two different databases: IRMA database which contains medical images and the image database from the ImageCLEF@ICPR 2010 photo annotation task which contains general images. The extensive experiments conducted on the benchmark databases show that our system has very high predictive performance and can be easily scaled to large amounts of visual concepts and data. In addition, our approach is very general: it can be easily extended with new feature extraction methods, and it can thus be easily applied to other domains, types of images and other classification schemes. Furthermore, it can handle arbitrarily sized hierarchies organized as trees or directed acyclic graphs.