Analyzing and detecting events from ubiquitous sensors across the city has been an important goal in recent years. Different techniques that are able to automatically detect events by monitoring urban sensor’s data have been efficiently applied in several smart cities to improve the citizens everyday life. However, the analysis of such voluminous data streams often interferes with several constraints that arise in smart cities scenarios. For example it is impossible to hire human oracles that will monitor each data stream continuously to provide knowledge to these models and to annotate past instances. Thus, the development of novel techniques is required in order to build efficient supervised learning models that will be able to cope with urban data deluge. Our approach makes the following contributions: (i) we formulate the problem of building supervised learning models efficiently by incorporating streaming input from urban data, and (ii) we present a novel framework that is able to cope with the restrictions that arise in the event detection of streaming urban data, requiring labels from carefully selected instances.