Learning sparse structure is useful in many applications. For example, gene regulatory networks are sparsely connected since each gene is typically only regulated by a small number of other genes. In this case factor analysis models with sparse loading matrices have been used to uncover the regulatory network from gene expression data. In this talk I will examine the performance of sparsity priors, such as mixture and L1 priors, by calculating learning curves for Bayesian PCA in the limit of large data dimension. This allows us to address a number of questions e.g. how well can we estimate sparsity using the marginal likelihood when the prior is not well-matched to the data generating process?