Future intelligent robots will need to interact with uncertain and changing environments. One key aspect to allow robotic agents to adapt to such situations is to enable them to learn multiple solution strategies to one problem, such that the agent can remain flexible and employ alternative solutions even if the preferred solution is no longer viable. We propose a unifying framework that allows the use of hierarchical policies and which can, thus, learn multiple solutions at once.