It is not known whether decrease in physical activity (PA) is associated with binge drinking among former athletes. The purpose of this study was to investigate the reciprocal associations between PA and use of alcohol among former athletes and controls at four time points. Furthermore, we examined whether there were longitudinal latent profiles related to use of alcohol, smoking and PA during the follow-up. Finnish male former elite athletes (n = 1633) and matched controls (n = 1099) questionnaire-reported their PA, alcohol consumption and smoking at four time points in 1985, 1995, 2001 and 2008. Former athletes were more physically active and smoked less than controls, but in all profiles smoking decreased during the follow-up. Former athletes consumed alcohol significantly more compared to controls in 1985, especially if their athletic career had ended suddenly by sports injury. At other time points, no differences were seen. Five latent profiles were found, and there were significant differences between former athletes and controls in the probabilities to belong to four of them. PA decreased in four of five profiles, while alcohol consumption decreased or increased in some profiles. But PA did not predict later alcohol consumption at any time point. Cross-lagged path model indicated that the mutual associations of alcohol use and PA were weak at most. Although risk of excessive alcohol consumption may increase in individuals, whose athletic career has ended suddenly by sports injury, overall PA and alcohol affected each other’s development only modestly among former athletes and controls during the 23-year follow-up.