In this course I will discuss how exponential families, a standard tool in statistics, can be used with great success in machine learning to unify many existing algorithms and to invent novel ones quite effortlessly. In particular, I will show how they can be used in feature space to recover Gaussian Process classification for multiclass discrimination, sequence annotation (via Conditional Random Fields), and how they can lead to Gaussian Process Regression with heteroscedastic noise assumptions.