Many popular optimization algorithms, like the Levenberg-Marquardt algorithm (LMA), use heuristic-based “controllers” that modulate the behavior of the op- timizer during the optimization process. For example, in the LMA a damping parameter λ is dynamically modified based on a set of rules that were developed using various heuristic arguments. Here we show that a modern reinforcement learning technique utilizing a very simple state space can dramatically improve the performance of general purpose optimizers, like the LMA, on problems where the number of function evaluations allowed is constrained by a budget. Results are given on both classical non-linear optimization problems as well as a difficult computer vision task. Interestingly the controllers learned for a particular opti- mization domain work well on other optimization domains. Thus, the controller appeared to have extracted optimization rules that were not just domain specific but generalized across a range of optimization domains.