Online learning to rank for information retrieval (IR) aims to enable search systems to learn directly from interactions with their users. In our recent work, we explore formulations based on reinforcement learning to allow systems to continuously adapt to changes in user behavior and preferences. A major challenge in this setting is to correctly interpret user interactions as feedback for learning. User interactions can be heavily biased by the presentation of search results (e.g., layout, ordering). Randomized data collection can address some of these issues, but can itself affect user interactions. In this talk I present some of the solutions developed for the IR setting, and discuss open issues and ongoing work in this area.