Machine learning has achieved unprecedented results in a variety of application areas. Medical science has always been an area of high importance for AI applications due to its high social potential impact. Machine learning models are now able to reliably diagnose cancer from medical imaging and to assist physicians providing better care to their patients more efficiently. The question is how much can we trust these models? Recently, deep neural networks have been shown to be vulnerable to adversarial attacks where a designed fake input can lead to misclassification. The US Food and Drug Administration is currently reviewing its policy on accepting machine learning models in medical devices and diagnostics due a recent case of a failed cancer diagnostic model. Hence, machine learning models are not only expected to perform accurately, but they have to adhere to strict criteria on model performance, bias, and ongoing maintenance. Most importantly in critical domains, like medicine, the model has to be able to explain its decision-making process. I will present recent advances in building machine learning models that are robust to adversarial attacks and can explain their outputs.