In this work we wish to answer the questions: (1) how much can we trust the list of genes and the biological functions found in a predictive signature and (2) how do common feature selection methods compare to each other in this regard? We propose a rigorous framework to assess the accuracy, the stability and the interpretability of a feature selection method and compare 8 common feature selection methods as well as ensemble feature selection variants on three breast cancer datasets. Results highlight the very low robustness of most existing methods, including ensemble methods, and raise a warning about the over interpretation of published signatures in terms of genes and biological processes.