Image segmentation is often defined as a partitioning of pixels or image blocks into homogeneous groups. These groups are characterized by a prototypical vector in feature space, e.g., the space of Gabor filter responses, by a prototypical histograms of features or by pairwise dissimilarities between image blocks. For all three data formats cost functions have been proposed to measure distortion and, thereby, to encode the quality of a partition. Learning in image segmentation can be defined as the inference of prototypical descriptors of segments like codebook vectors or average feature probability within a segment. Contrary to classification or regression, the empirical risk of image segmentation is often composed of sums of dependent random variables like in Normalized Cut, Pairwise Clustering or k-means clustering with smoothness constraints. One of the core challenges for machine learning is to discover what kind of information can be learned from these data sources assuming MRF cost functions as image models. The validation procedure for image segmentations strongly depends on this issue. I will demonstrate the learning and validation issue in the context of image analysis based on color and texture features.