Background Despite numerous studies providing evidence for positive effects of physical activity and physical fitness, evidence for association between physical fitness and health-related quality of life (HRQoL) in young adults is limited. The aim of the present cross-sectional study was to investigate the association of cardiorespiratory and muscular fitness with HRQoL from the perspective of its physical and mental components among young adult Finnish males. Methods The sample consisted of 754 men, with the mean age of 26 years (SD 6.7 years), who participated in the military refresher training. HRQoL was measured using the Finnish RAND 36-item health survey. Cardiorespiratory fitness was determined by a bicycle ergometer test, and muscular fitness by various tests measuring maximal strength and muscular endurance. Logistic regression modelling was used to compare low, moderate and high physical and mental component of HRQoL scores to the respective levels of muscular and cardiorespiratory fitness. Results The findings of the adjusted (age, educational level, marital status, employment status, smoking, use of alcohol and BMI) analysis showed that cardiorespiratory and muscular fitness are positively associated with both physical and mental components of HRQoL. In terms of the physical component of HRQoL, even a moderate fitness level was positively associated with better HRQoL. In terms of the mental component of HRQoL, the impact was seen only in the group with the highest fitness level. Conclusions The findings suggest a positive contribution of physical fitness to mental health and highlight the importance of both muscular and cardiorespiratory fitness in the promotion of HRQoL. Even lighter forms of physical activity that result in moderate physical fitness could contribute to the physical component of HRQoL. In terms of the mental component of HRQoL, higher levels of physical fitness may be needed to gain higher levels of HRQoL among young males.