I need to create many large numpy arrays (4e6, 100) with random numbers from a standard normal distribution, which I'm trying to speed up. I tried to generate different parts of the arrays using multiple cores but I'm not getting the expected speed improvements. Is there something I'm doing wrong, or am I wrong to expect speed improvements in this way?
from numpy.random import default_rng
from multiprocessing import Pool
from time import time
def rng_mp(rng):
return rng.standard_normal((250000, 100))
if __name__ == '__main__':
n_proc = 4
rngs = [default_rng(n) for n in range(n_proc)]
rng_all = default_rng(1)
start = time()
result = rng_all.standard_normal((int(1e6), 100))
print(f'Single process: {time() - start:.3f} seconds')
start = time()
with Pool(processes=n_proc) as p:
result = p.map_async(rng_mp, rngs).get()
print(f'MP: {time() - start:.3f} seconds')
# Single process: 1.114 seconds
# MP: 2.634 seconds
Aucun commentaire:
Enregistrer un commentaire