Background: Hereditary factors, including single genetic variants and family history, can be used for targeting colorectal cancer (CRC) screening, but limited data exist on the impact of polygenic risk scores (PRS) on risk-based CRC screening. Methods: Using longitudinal health and genomics data on 453,733 Finnish individuals including 8801 CRC cases, we estimated the impact of a genome-wide CRC PRS on CRC screening initiation age through population-calibrated incidence estimation over the life course in men and women. Results: Compared to the cumulative incidence of CRC at age 60 in Finland (the current age for starting screening in Finland), a comparable cumulative incidence was reached 5 and 11 years earlier in persons with high PRS (80–99% and >99%, respectively), while those with a low PRS (< 20%) reached comparable incidence 7 years later. The PRS was associated with increased risk of post-colonoscopy CRC after negative colonoscopy (hazard ratio 1.76 per PRS SD, 95% CI 1.54–2.01). Moreover, the PRS predicted colorectal adenoma incidence and improved incident CRC risk prediction over non-genetic risk factors. Conclusions: Our findings demonstrate that a CRC PRS can be used for risk stratification of CRC, with further research needed to optimally integrate the PRS into risk-based screening.