Unique Random File Selector Generator

I have a directory with a large number of files. I don't want to keep all the file names in memory, but I want to randomly get a subset of these files using a generator.

I can do this using information found in the post "Best way to choose a random file from a directory", but I would like to make sure that my generator never returns the same file twice.

The methods I can think of still creates a file list to compare against (Create a list of already used file names and return if not in list) and would take longer to execute the more times the generator yielded results.

Is there a way, if I create an array of numbers equal to the number of files in the directory, that when I randomly pop a value from the array, I could get the file at that position? (I think this would take up significantly less memory than an array of strings)