This talk addresses the problem of building models for category-level recognition. The starting point is a set of local invariant features which have been shown to support the robust recognition of specific objects and scenes. In the category-level object recognition context, it is no longer sufficient to use individual features, and it becomes necessary to model intra-class variations, to select discriminant features, and to model spatial relations. Furthermore, it is important to use shape information, as in many cases objects of a given category are different in grey-level appearance, but similar in shape. This leads to a part-based approach to category-level recognition that I will illustrate with several examples, including feature selection, local affine-invariant part models, and contour-based shape description. This is joint work with Gyuri Dorko, Frederic Jurie, Svetlana Lazebnik and Jean Ponce.