mercredi 10 janvier 2018

Why python read tiff images slower when using random generator to retrieve the image paths

I am trying to write a python program to randomly read some tiff images from a large sample tiff images. Interestingly, I found that python tends to read the tiff images (with floating point values) much slower if we use random generator to generate indices and obtain the list of the image paths, as compared to use hard code random indices to obtain the image paths and read the tiff images.

import datetime
import matplotlib.pyplot as plt
import numpy

def read_in_seq(image_filenames, indices):
    return [ plt.imread(image_filenames[index]) for index in indices ]

image_filenames = []

for index in range(15000):
    image_filenames.append("/tmp/%05d" % index + ".tiff")

# This is generated from numpy.random.choice(15000, 100) but hard coded the values here
indices=[
  3885,   901,  6233,  7234, 10195,  2204,   469,  2906, 12114, 13515, 12977, 5201,
  8829, 11537,  5400,  9633, 10744, 12991,  2593,  3046,  5103,  1901,  8831, 12454,
  9779,  4714, 10839,  8702,  8537,  2136,  5095,  9006, 13293,  9933,  3584, 10818,
  8594, 11032,  3705,   435,  6679,  8349,  6930,  9741, 12933,  3231,  1849,  7871,
 11752,  8361,  3094,  2229, 14303,  2006,  5554,  1492, 14817, 12690, 10648, 14631,
  6401,  6181,  4401,  7222,  9881,  8381,  7603, 11374, 12702,  6881, 11868, 10967,
 14508, 12930,  3542,  1197,  8387, 11253,  1802, 14732,  7419, 11994,  6083,  8846,
  5370,  4276, 13953, 14409,  8197,  8956,  4717,  3262,  2314, 12527,  5394, 12495,
  6708,  9724,   740, 10416]

print(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f') + ": Normal input read started with size=" + str(len(indices)))
output = read_in_seq(image_filenames, indices) # takes 0.8 seconds
print(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f') + ": Normal input read completed with size=" + str(len(output)))

indices = numpy.random.choice(15000, 100)
print(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f') + ": Random input read started with size=" + str(len(indices)))
output = read_in_seq(image_filenames, indices) # takes ~3 seconds
print(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f') + ": Random input read completed with size=" + str(len(output)))

Here is the output:

2018-01-10 15:30:46.170487: Normal input read started with size=100
2018-01-10 15:30:46.943557: Normal input read completed with size=100
2018-01-10 15:30:46.943718: Random input read started with size=100
2018-01-10 15:30:49.858074: Random input read completed with size=100

All 15000 tiff images are identical, ~3MB each. As you can see, normal input read with hard coded random indices for 100 tiff images out of 15000 tiff images only takes 0.8 seconds. However, it takes almost 3 seconds when we use the indices generated from the random generator (e.g. numpy.random).

On the other hand, if we modify the above code to read 100 png images out of 15000 images. The time to read the images using the hard coded random generated indices is almost the same as the indices generated by numpy.random.

for index in range(15000):
    image_filenames.append("/tmp/%05d" % index + ".png")

Please note that the time measure to read the tiff images does not count the time taken by numpy.random (only count the time to read the image read_in_seq).

Let's assume we can only use single thread, please can someone explain why python read tiff images slower when using random generator to retrieve the image paths (as compared to hard coded random indices to retrieve the image paths)? e.g. is it related to CPU floating point support, hard-drive seek, OS design or something else?




Aucun commentaire:

Enregistrer un commentaire