This study explores nonlinear optical effects in diamond crystals doped with nitrogen-vacancy (NV) centers, focusing on the optical Kerr effect in the infrared spectral region. By employing the Z-scan technique with 230 fs laser pulses at a wavelength of 1032 nm, we investigate the nonlinear refractive index of high-purity and NV-containing diamond crystals at NV concentrations of 0.3 ppm and 4.5 ppm. The results demonstrate a pronounced anisotropy in the nonlinear refractive index, which exhibits a four-fold rotational symmetry, with its amplitude reduced by the NV concentration. The reduction in non-linear susceptibility is linked to the negative contribution of NV centers, modeled as two-level systems upon a strong, laser field, red-detuned relative to their resonance frequency. We believe these findings provide new insights into the interplay between NV centers and the nonlinear optical properties of diamond, offering potential pathways for tunable nonlinear optics applications.