Recently, it has been shown that atomic structure determination with X-ray fluorescence holography (XFH) can be hindered by matrix effects, i.e. beam attenuation and indirect excitation. The analysis was limited to the monochromatic regime. In this work, the description of matrix effects is extended to the polychromatic case. It is shown that matrix effects affect the element sensitivity of white-beam XFH by introducing distortions in the holographic signal which may lead to spurious maxima in the reconstructed image. For high energies of the X-ray beam it is found that the effect of beam attenuation is very weak and indirect excitation mainly contributes to the distortions. A correction for matrix effects is proposed in the high-energy range, which allows one to remove the distortions and retrieve pure element-sensitive information. Numerical model calculations are performed to visualize the reduction of element sensitivity and its implications on local structure imaging.