What are the main factors that limit behavioral performance in decision making? In most models, behavioral performance is set primarily by the amount of neural noise. This factor can indeed account for a vast array of experimental results, including Weber’s law. I will argue instead that behavioral performance is constrained by a combination of two factors: suboptimal inference and variability in the stimulus. I will demonstrate how suboptimal inference can masquerade as neural noise in olfactory processing in rodents. In this particular case, the suboptimal inference is the result of the animal wrongly assuming that the task is not stationary when it in fact is. This faulty assumption leads the animal to learn on each trial, causing extra variability. I will also show that Weber’s law could be the mere consequence of the statistics of natural sensory inputs, as opposed to neural noise and log nonlinearities. In summary, it is far from clear that neural noise limits performance in decision making. Instead, the brain is most likely limited by the computational complexity of the inference it performs and the quality of the data it receives.