The digitization of our daily lives has led to an explosion in the collection of data by governments, corporations, and individuals. Protection of confidentiality of this data is of utmost importance. However, knowledge of statistical properties of private data can have significant societal benefit, for example, in decisions about the allocation of public funds based on Census data, or in the analysis of medical data from different hospitals to understand the interaction of drugs. I will start by introducing two application scenarios, privacy-preserving data analysis and privacy-preserving data publishing. I will show how in simple models background knowledge can lead to severe breaches of privacy in both applications, and I will describe how proper modeling of background knowledge can avoid privacy breaches. I will outline first algorithmic steps towards privacy-preserving data analysis and data publishing with background knowledge, and I will conclude with open problems.