Language models have become indispensable tools for scholars to tackle information overload, enabling them to efficiently discover and comprehend the most relevant research literature. In this talk, we will present a thorough overview of language modeling techniques specifically tailored for the scientific domain. We will begin by delving into the foundations of prevalent language models for science, outlining their essential components and training methodologies. Subsequently, we will explore domain adaptation techniques, which enable these models to be fine-tuned for specialized scientific fields. Next, we will examine evaluation methodologies and investigate specific natural language processing (NLP) tasks pertinent to scientific documents. Furthermore, we will discuss representation learning methods for scientific papers and address the challenges associated with evaluating these methods. To conclude the talk, we will highlight the existing challenges and open problems in the field of scientific language modeling, and suggest promising directions for future research endeavours.