Confidence intervals reflect our uncertainty about a parameter of interest, and models reflect our assumptions about the context of the data. Some of these assumptions may be justified by background knowledge, but others will be rather arbitrary. Statistics text books advise that before assuming a model we should check that it gives a good fit to the data (by using goodness-of-fit tests or graphical diagnostics). But does a well-fitting model necessarily mean a good confidence interval? Looking at the robustness of confidence limits to model choice suggests some rather basic questions about our use of models and assumptions in statistics.