Similarity-based learning assumes one is given similarities between samples to learn from, and can be considered a special case of graph-based learning where the graph is given and fully-connected. Such problems arise frequently in computer vision, bioinformatics, and problems involving human judgment. We will review the field of similarity-based classification and describe the main problems encountered in adapting standard algorithms for this problem, including different approaches to approximating indefinite similarities by kernels. We will motivate why local methods lessen the indefinite similarity problem, and show that a kernelized linear interpolation and local kernel ridge regression can be profitably applied to such similarity-based classification problems by framing them as weighted nearest-neighbor classifiers. Eight real datasets will be used to compare state-of-the-art methods and illustrate the open challenges in this field.