Background: There are no prospective studies that would have compared the relationships of different types of physical activity (PA) and sedentary behavior (SB) with academic skills among children. We therefore investigated the associations of different types of PA and SB with reading and arithmetic skills in a follow-up study among children. Methods: The participants were 186 children (107 boys, 79 girls, 6–8 yr) who were followed-up in Grades 1–3. PA and SB were assessed using a questionnaire in Grade 1. Reading fluency, reading comprehension and arithmetic skills were assessed using standardized tests at the end of Grades 1–3. Results: Among all children more recess PA and more time spent in SB related to academic skills were associated with a better reading fluency across Grades 1–3. In boys, higher levels of total PA, physically active school transportation and more time spent in SB related to academic skills were associated with a better reading fluency across the Grades 1–3. Among girls, higher levels of total PA were related to worse arithmetic skills across Grades 1–3. Moreover, total PA was directly associated with reading fluency and arithmetic skills in Grades 1–3 among girls whose parents had a university degree, whereas these relationships were inverse in girls of less educated parents. Conclusions: Total PA, physically active school transportation and SB related to academic skills may be beneficial for the development of reading skills in boys, whereas factors that are independent of PA or SB may be more important for academic skills in girls.