In performance display advertising, bidders compete on behalf of advertisers for ad impressions, that is, the opportunity to display relevant ads on a publisher website. We consider bidding on behalf of online retailers who buy ad impressions hoping to realize value only from purchases attributed from clicks. The bidder has a two stage problem. In the first stage, the bidder has to select a small subset from a large selection of ads, with the selected ads most likely to lead to purchases. In the second stage, the bidder has to estimate the purchase probability of the selected ads, which can then be used to create bid values. The challenge in the first stage is that a model optimized for purchases also needs to be (near) optimal for clicks, due to the click attribution constraint. The challenge in the second stage is that true probability of purchases is extremely small, and is difficult to accurately model. We propose a ranking model, followed by a calibration method, to sequentially address the two stage problem. We describe how ordinal ranking is a natural fit for the ad selection problem and how to learn a single model by optimizing for purchases, while being (near) optimal for clicks. We then propose a calibration method, which comprises of a novel non-uniform binning technique for empirical probability estimation, in conjunction with calibration functions such as isotonic and polynomial regression and Platt scaling. We provide empirical results on logged events from a major ad network, that demonstrate the superiority of ordinal model over binary classifiers for ranking ads and the superiority of our proposed calibration technique over traditional uniform binning based calibration technique.