Sensor based activity recognition is a critical component of mobile phone based applications aimed at driving detection. Current methodologies consist of hand-engineered features input into discriminative models, and experiments to date have been restricted to small scale studies of O(10) users. Here we show how convolutional neural networks can be used to learn features from raw and spectrogram sensor time series collected from the phone accelerometer and gyroscope. While with limited training data such an approach under performs existing models, we show that convolutional neural networks outperform currently used discriminative models when the training dataset size is sufficiently large. We also test performance of the model implemented on the Android platform and we validate our methodology using sensor data collected from over 2000 mobile phone users.