vendredi 14 juillet 2017

Why does Tensorflow shuffle_batch repeat examples and How to work around?

I am attempting to read in my data once with a string_input_producer(num_epochs=1) and then shuffle and batch the data. According to my unit test, I can successfully shuffle the data, but not batch it to my needs. Meaning, if I set the shuffle_batch to only output one batch that is the entire size of its input, then the data is shuffled and there are no repeats of any examples in the data. However, once I decide I want more than one batch, the shuffle_batch begins to repeat data, same if I increase the number of epochs in the string_input_producer. I do not know how to stop it from doing this.

What I want:

After reading in my data with arbitrary number of epochs, I want to shuffle the data by examples in an epoch, split it into an arbitrary number of batches, and ensure that there are no repeats of examples. However, if epcohs > 1, then there should be a repetition of examples over all the batches equal to the number of epochs, and each batch group for an epoch should still have only unique examples. How can accomplish this in tensorflow?

Script Depicting Dilemma:

My code is in part of a larger project, so I am providing a small script that depicts the repetition of examples in shuffle_batch. And by making this script, I have encountered something even more strange. In the script below, shuffle_batch repeats examples even if the batch size is equivalent to the input size. I included allow_smaller_final_batch because I require it in my program. I tried to mimic my setup in my actual code exactly with this script. The print out check mimics how my unit test operates to ensure the data matches, advice on this check are welcomed.

from __future__ import print_function
import numpy as np
import tensorflow as tf # r1.2

a = []
b = []
record_count = 5
batch_count = 1

for i in range(record_count):
    a.append(range((i-1) * 5, i*5))
a = np.asarray(a)

print("original numpy")
for row in a:
    print(row)

record = tf.train.shuffle_batch([a],
            batch_size=record_count,
            capacity=record_count,
            min_after_dequeue=record_count-1,
            num_threads=1,
            enqueue_many=True,
            allow_smaller_final_batch=True,
        )

init_op = tf.group(tf.global_variables_initializer(),
                   tf.local_variables_initializer())
with tf.Session() as sess:
    sess.run(init_op)
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)

    for i in range(batch_count):
        rec = sess.run(record)
        b.append(rec)

    coord.request_stop()
    coord.join(threads)

    b = np.vstack(b)

print("\nnumpy retrieved from sess.run()")
for row in b:
    print(row)

print("Check if a & b are 1 to 1, but b is shuffled")
print("shape equal? ", a.shape == b.shape)
print("number of elements equal? ", a.size == b.size)

u1, uc1 = np.unique(a, return_counts=True)
u2, uc2 = np.unique(b, return_counts=True)

print("Unique elements are equal? ", np.array_equal(u1, u2))
print("Unique elements counts are equal? ", np.array_equal(uc1, uc2))

print("b not equal to original a", not np.array_equal(a, b))


Example output from script:

original numpy
[-5 -4 -3 -2 -1]
[0 1 2 3 4]
[5 6 7 8 9]
[10 11 12 13 14]
[15 16 17 18 19]

numpy retrieved from sess.run()
[0 1 2 3 4]
[-5 -4 -3 -2 -1]
[-5 -4 -3 -2 -1]
[5 6 7 8 9]
[15 16 17 18 19]
Check if a & b are 1 to 1, but b is shuffled
shape equal? True
number of elements equal? True
Unique elements are equal? False
Unique elements counts are equal? False
b not equal to original a? True

Desired output in this case:

original numpy
[-5 -4 -3 -2 -1]
[0 1 2 3 4]
[5 6 7 8 9]
[10 11 12 13 14]
[15 16 17 18 19]

numpy retrieved from sess.run()
[10 11 12 13 14]
[15 16 17 18 19]
[0 1 2 3 4]
[5 6 7 8 9]
[-5 -4 -3 -2 -1]
Check if a & b are 1 to 1, but b is shuffled
shape equal? True
number of elements equal? True
Unique elements are equal? True
Unique elements counts are equal? True
b not equal to original a True




Aucun commentaire:

Enregistrer un commentaire