Recent studies indicate that air pollutants not only increase the risk of cardiovascular and respiratory diseases but also have a negative impact on the developing brain. Exposure to airborne particulate matter (PM) and nitrogen dioxide (NO2) may lead to disruption of neural development by interfering with critical maturation processes. In this study, we assessed the impact of prenatal and early life PM10 and NO2 exposure on diffusion Magnetic Resonance Imaging (dMRI) structural measures: fractional anisotropy (FA), mean diffusivity (MD), and fixel-based analysis (FBA) on a population of 425 10- to 13-year-old children with attention deficit hyperactivity disorder (ADHD, n = 116), a sensitive, at-risk population, and typically developing children (TD, n = 309) from the NeuroSmog study. Unlike traditional voxel-based methods, FBA allows identification of distinct fiber bundles within voxels. We show that early life exposure to NO2 was associated with lower global FA and higher MD measures. However, despite having a large sample size and using state-of-the-art techniques, we found no significant fixel-level associations. Notably, we found no evidence that individuals with ADHD are more susceptible to the effects of air pollution. Combined with other studies, our results suggest that dMRI measures are the brain outcomes most consistently affected by air pollution.