I will discuss our recent work on interactive machine learning in two closely related setups: (i) interactive intent modeling for information discovery and (ii) knowledge elicitation on features for improving predictive modelling given limited high-dimensional data. Both setups require balancing between exploration and exploitation in the interactions, and interactive modelling of the user which can be formulated as experimental design or multiarmed bandit problems. I will also discuss extensions to multimodal interfaces, including mind reading, and to inferring more advanced cognitive models from data with Approximate Bayesian Computation.