We point out that a successful inflationary magnetogenesis could be realised if we break the local U(1) gauge symmetry during inflation. The effective electric charge is fixed as a fundamental constant, which allows us to obtain an almost scale-invariant magnetic spectrum avoiding both the strong coupling and back reaction problems. We examine the corrections to the primordial curvature perturbation due to these stochastic electromagnetic fields and find that, at both linear and non-linear orders, the contributions from the electromagnetic field are negligible compared to those created from vacuum fluctuations. Finally, the U(1) gauge symmetry is restored at the end of inflation.