How to get random element from CharacterSet in Swift

I have an array of characters the user has already entered. I would like to get a random character that the user has not already entered from a combined list of all of the alphanumeric characters, the punctuation characters and the symbols. I have looked through the documentation but can't find a way to get a random element from a CharacterSet, which seems odd... I feel like I am missing something obvious.

func newRandomCharacter() -> Character {

let validCharacters: CharacterSet = .alphanumerics.union(.punctuationCharacters).union(.symbols)

var setOfUsedCharacters = CharacterSet()
// usedCharacters: [Character]
usedCharacters.forEach { setOfUsedCharacters.insert(charactersIn: String($0)) }

let setOfUnusedCharacters = validCharacters.subtracting(setOfUsedCharacters)

return setOfUnusedCharacters.randomElement() <- ???

}