Python: Number guessing game

I'm really not sure how to go about this but I have this weird issue that I am not very sure on how to fix. Here is the concept of my game, the computer randomly generates a number from 1-100 and the player has to guess that number. If the number they guess is higher or lower the computer tells them so. I added some code to make sure that the guess that the user enters is a number, but for some reason, it only works for their first guess. I don't really know how to else to explain it. But for example, if I guess the number 20 as my first guess, it would output too high or too low depending on the randomly generated number, but after that, If I input a bunch of random letters it would give me an error that the guess could not be compared to the randomly generated number, I'm really not sure on how to go about this or if I even explained my problem that well. You can try running the code yourself and inputting a number for your first guess and a random bunch of letters as your second to see if that would help. Again, I am very much a beginner and have only been coding in Python for about a week. Thanks again to anyone that is taking their time out of their day to read this and help me out!

import random

x = random.randint(1, 100)
guess = input("Guess the number")

while guess.isnumeric() == True:

    if x > int(guess):
        print("Too low, guess again")
        guess = input("Guess the number")

    if x < int(guess):
        print("Too high, guess again")
        guess = input("Guess the number")

    if x == int(guess):
        print ("That is correct!")
        break

if guess.isnumeric() == False:
        print("Please enter a valid number")
        guess = input("Guess the number")