Background: The present study aimed to investigate how subjective self-reported stress is associated with objective heart rate variability (HRV)-based stress and recovery on workdays. Another aim was to investigate how physical activity (PA), body composition, and age are associated with subjective stress, objective stress, and recovery. Methods: Working-age participants (n = 221; 185 women, 36 men) in this cross-sectional study were overweight (body mass index, 25.3–40.1 kg/m2 ) and psychologically distressed (≥3/12 points on the General Health Questionnaire). Objective stress and recovery were based on HRV recordings over 1–3 workdays. Subjective stress was assessed by the Perceived Stress Scale. PA level was determined by questionnaire, and body fat percentage was assessed by bioelectrical impedance analysis. Results: Subjective stress was directly associated with objective stress (P = 0.047) and inversely with objective recovery (P = 0.046). These associations persisted after adjustments for sex, age, PA, and body fat percentage. Higher PA was associated with lower subjective stress (P = 0.037). Older age was associated with higher objective stress (P < 0.001). After further adjustment for alcohol consumption and regular medication, older age was associated with lower subjective stress (P = 0.043). Conclusions: The present results suggest that subjective self-reported stress is associated with objective physiological stress, but they are also apparently affected by different factors. However, some of the found associations among these overweight and psychologically distressed participants with low inter-individual variation in PA are rather weak and the clinical value of the present findings should be studied further among participants with greater heterogeneity of stress, PA and body composition. However, these findings suggest that objective stress assessment provides an additional aspect to stress evaluation. Furthermore, the results provide valuable information for developing stress assessment methods.