Unable to reproduce randomness with tensorflow and numpy combined?

I have a project in which I cannot reproduce random numbers when I use numpy in combination with tensorflow. In the beginning of all my tests, I set

tf.set_random_seed(seed)
np.random.seed(seed)

I have been debugging, and when I use numpy and no TF, all results are reproducible. When I add the TF code, the random numbers stop being reproducible. When I use both TF and numpy, I get the following results:

1. TF variables are initialized to the same value every time (OK)
2. When I use np.random.RandomState() with a set seed instead of direct calls to np.random.uniform(), np.random.normal(), etc, results are reproducible (OK)
3. When I use direct calls to np.random.uniform(), np.random.normal(), etc, results are not reproducible (NOT OK)

The difference between 2 and 3 makes me think that TF must be using numpy internally somewhere in order to generate random numbers. This sounds a bit strange and unexpected. I have only 1 main thread, so the difference is definitely not caused because of race conditions. Furthermore, even if TF uses np.random, this should not change the random numbers that I observe between runs in my project since the sequence of querying the random number generation is always the same.

What is even more strange is that the particular piece of TF code which makes results non-reproducible is computing and applying gradients, where I would not expect any random number generation to be needed. Note that I am comparing only the sampled random numbers rather than results from the network (since TF has some non-deterministic operations) and these random numbers are not affected in any way by results produced from training the net.

Sorry I am unable to post my code, but it is just too big and reducing it to a smaller sample will likely make the problem go away. Thus any suggestions how to debug further are welcome.