In this talk I will discuss a number of vignettes on scaling optimization and inference. Despite arising from very different contexts (graphical models inference, convex optimization, neural networks), they all share a common design pattern - a synchronization mechanism in the form of a parameter server. It formalizes the notion of decomposing optimization problems into subsets and reconciling partial solutions. I will discuss some of the systems and distribution issues involved in building such a system.