Error when removing random item from list

I'm getting an error when I try to run this that says:

ValueError: list.remove(x): x not in list

This error, however, does not always occur. I think that is because it only happens when the random number that comes back has already been removed but I don't know for sure or how to fix it. All help greatly appreciated.

def main():
    numbers = [1,2,3,4,5,6,7,8,9,10,]
    count = 0

    while count <= 3:  #how many numbers to remove
        count += 1
        remove(numbers)

print (numbers)
done = input ("Enter to exit.")

def remove(numbers):

    import random
    randnum = random.randint(1,10)
    numbers.remove(int(randnum))


main()