Reinforcement learning has proved to be a powerful method for robotic control. In this talk, drawing on examples from autonomous helicopter flight, quadruped robot control and autonomous driving, I'll describe some of the challenges we've faced in applying RL algorithms to various control problems, such as (i) Problems where the reward function is exceedingly difficult to specify by hand, and must itself be learned, (ii) Safe exploration, where one wishes to explore without damaging the robot, and (iii) Learning high performance controllers even if we have only an extremely inaccurate model of our robot's dynamics. Using apprenticeship learning - in which we learn by watching an expert demonstration - as a unifying theme, I'll also describe a few algorithms for addressing these challenges.