In the "structural" paradigm for visual pattern recognition, or what some call "strong" pattern recognition, one is not satisfied with simply assigning a class label to an input object, but instead we aim at finding exactly which parts of the template object correspond to which parts of the scene. This is a much harder problem in principle, because it is inherently combinatorial on the number of parts (features) involved, both in the template object and in the scene. This talk describes a summary of our research efforts in setting this as a mathematical optimization problem and solving it efficiently by exploiting geometric constraints. The key insight involves encoding geometric constraints as conditional independency assumptions in a probabilistic graphical model. Due to some geometric facts, it is possible to show that such models are very well behaved: they allow for exact probabilistic inference in polynomial time. The result is a unified framework for structural visual pattern recognition that is able to handle in a principled way a variety of problems, including point pattern matching in its many instances: invariant to translations, isometries, scalings, affine or projective transformations. Attributed graph matching problems, such as matching road networks, can also be solved within such framework. Limitations and future directions will be discussed.