Permutation testing and covariate/confound inclusion

I have a data set with two groups where participants in both groups were collected from numerous sites. I performed randomised permutation testing between the groups however I was asked if I included site as a covariate/confound.

Is it necessary to control for site? Intuitively, I would have thought that the randomisation between groups would mean that you are in an indirect way controlling for this? With enough permutations then you could assume that there will be a mixing of different sites? Is this incorrect? Would it be more appropriate to use another technique that explicitly controls for site?