In the present study the Geneva Emotional Music Scale (GEMS-25) and its German offshoot, the GEMS-28-G were tested for measurement invariance across different types of musical stimuli. Additionally, the comparability of scores across the different language versions was checked. While alternative scales are often based on general dimensional or categorical emotion theories and are thus "stimulus-neutral", the domain-specific likert-type emotion scale GEMS is designed to especially capture the emotions evoked when listening to music. Within the study, an online survey was administered (n = 245) using a stimuli set of 20 excerpts from musical pieces. By analyzing the data with structural equation modeling (SEM), we tried to verify the reliability of the scales in terms of measurement invariance towards popular/classic music as well as towards the genre of electroacoustic music, employing the latter as an extreme case of a "non-conventional musical style". We subsequently also tested for measurement invariance across languages. Concerning music styles, measurement invariance of the original GEMS-25 was achieved only at the "configural level", while the GEMS-28-G could reach at least "weak factorial invariance". This demonstrates that only for the German version the contextual meaning of the construct remains constant across different musical genres with a reasonable fit. Nevertheless, researchers should be cautious when comparing GEMS factor scores achieved with very heterogenic musical styles in future studies, regardless in which language.