In this paper we present several ideas based on learning deep generative models from high-dimensional, richly structured sensory input. We will exploit the following two key properties: First, we show that deep generative models can be learned efficiently from large amounts of unlabeled data. Second, they can be discriminatively fine-tuned using the standard backpropagation algorithm. Our results reveal that the learned high-level feature representations capture a lot of structure in the unlabeled input data, which is useful for subsequent discriminative tasks, such as classification or regression, even though these tasks are unknown when the deep generative model is being trained.