tf.estimator shuffle - random seed?

When I repeatedly run tf.estimator.LinearRegressor the results are slightly different each time. I'm guessing that's because of the shuffle=True here:

input_fn = tf.estimator.inputs.numpy_input_fn(
    {"x": x_train}, y_train, batch_size=4, num_epochs=None, shuffle=True)

Which is fine as far as it goes, but when I try to make it deterministic by seeding the random number generators in both np and tf:

np.random.seed(1)
tf.set_random_seed(1)

The results are still slightly different each time. What am I missing?