Given a dictionary Π and a signal ξ=Πx generated by a few linearly independent columns of Π, classical sparse recovery theory deals with the problem of uniquely recovering the sparse representation x of ξ. In this work, we consider the more general case where ξ lies in a low-dimensional subspace spanned by a few columns of Π, which are possibly linearly dependent. In this case, x may not unique, and the goal is to recover any subset of the columns of Π that spans the subspace containing ξ. We call such a representation x subspace-sparse. We study conditions under which existing pursuit methods recover a subspace-sparse representation. Such conditions reveal important geometric insights and have implications for the theory of classical sparse recovery as well as subspace clustering.