It is believed that violation of or conformity to expectancy when listening to music is one of the main sources of musical emotion. To address this, we test a new way of building feature vectors and representing features within the vector for the machine learning approach to continuous emotion tracking systems. Instead of looking at the absolute values for specific features, we concentrate on the average value of that feature across the whole song and the difference between that and the absolute value for a particular sample. To test this “relative” representation, we used a corpus of popular music with continuous labels on the arousalvalence space. The model consists of a Support Vector Regression classifier for each axis, with one feature vector for each second of a song. The relative representation, when compared to the standard way of building feature vectors, gives a 10% improvement on average (and up to 25% improvement for some models) on the explained variance for both the valence and arousal axes. We also show that this result is