I want to sample ~10⁷ times from a population of ~10⁷ integers without replacements and weights, each time picking 10 elements. I have timed two approaches (python3 and numpy) in the following script. Both approaches seem painfully slow to me, do you see a way of speeding it up?
import numpy as np
import random
@profile
def test_choices():
population = list(range(10**7))
weights = np.random.uniform(size=10**7)
np_weights = np.array(weights)
def numpy_choice():
np_w = np_weights / sum(np_weights)
c = np.random.choice(population, size=10, replace=False, p=np_w)
def python_choice():
c = []
while len(c) < 10:
c += random.choices(population=population, weights=weights, k=10 - len(c))
c = list(set(c))
for i in range(10**1):
numpy_choice()
python_choice()
add_weight = np.random.uniform()
random_element = random.randint(0, 10**7)
weights[random_element] += add_weight
np_weights[random_element] += add_weight
test_choices()
With the timer result:
Line # Hits Time Per Hit % Time Line Contents
==============================================================
24 10 20720062.0 2072006.2 56.6 numpy_choice()
25 10 15593925.0 1559392.5 42.6 python_choice()
Aucun commentaire:
Enregistrer un commentaire