Closed-form, interpretable mathematical models have been instrumental for advancing our understanding of the world; with the data revolution, we may now be in a position to uncover new such models for many systems from physics to the social sciences. However, to deal with increasing amounts of data, we need “machine scientists” that are able to extract these models automatically from data. Here, we will introduce the main ideas of Bayesian inference and model selection, and discuss a Bayesian machine scientist, which establishes the plausibility of models using explicit approximations to the exact marginal posterior over models and establishes its prior expectations about models by learning from a large empirical corpus of mathematical expressions. Within this approach, the space of models is explored using Markov chain Monte Carlo. We will show that this approach uncovers accurate models for synthetic and real data and provides out-of-sample predictions that are more accurate than those of existing approaches and of other nonparametric methods. We will also use this approach to discuss how models are not always learnable from data and that, in some situations, no algorithm will ever be able to learn the correct model from the data alone.