I want to split m*n
elements (e.g., 1, 2, ..., m*n
) into n
group randomly and evenly such that each group has m
random elements. Each group will process k
(k>=1
) elements at one time from its own group and at the same speed (via some synchronization mechanism), until all group has processed all their own elements. Actually each group is in an independent process/thread.
I use numpy.random.choice(m*n, m*n, replace=False)
to generate the permutation first, and then index the permuted result from each group.
The problem is that when m*n
is very large (e.g., >=1e8
), the speed is very slow (tens of seconds or minutes).
Is there any faster/lazier way to do this? I think maybe this can be done in a lazier way, which is not generating the permuted result in the first time, but generate a generator first, and in each group, generate k elements at each time, and its effect should be identical to the method I currently use. But I don't know how to achieve this lazy way. And I am not sure whether this can be implemented actually.
Aucun commentaire:
Enregistrer un commentaire