In this paper, we investigate the role of heterogeneity added to the Voter Model. In our model, voters are equipped with an individual inertia to change opinion, which depends on the persistence time of a voter’s current opinion. In the simplest case, inertia-values are linearly increasing with persistence time by a slope μ, but our findings are qualitatively valid for other monotonously increasing functions, too. Here, in contrast to the Voter Model, voters change their individual behavior over time and the system builds up heterogeneity. The unexpected outcome of this dynamics is a non-monotonous development of average consensus times TC on the slope μ. Up to a value μc, TC decreases systematically with increasing μ, i.e. systems with higher average inertia reach the final attractor state faster. For slopes larger than μc, consensus times increase and can exceed the reference time of the Voter Model. These results are obtained only by considering a heterogeneity of voters that evolves through the described ageing of the voters, as we find monotonously increasing consensus times in a control setting of homogeneous inertia values. In the paper, we present the dynamical equations for a simplified mean field model, that give insight into the complex dynamics leading to the observed slower-is-faster effect.