jeudi 21 juillet 2022

Multiprocessing sub processes with one shared input (stdin)

i ran into a speed problem while using multiprocessing sub processes. My Goal is to have multiple sub processes which run parallel with the multiprocessing package.

    process1 = multiprocessing.Process(
        target=controller.runSubprocess, args=(3,))
    jobs.append(process1)
    process1.start()
    process2 = multiprocessing.Process(
        target=controller.runSubprocess, args=(4,))
    jobs.append(process2)
    process2.start()
    process3 = multiprocessing.Process(
        target=controller.runSubprocess, args=(5,))
    jobs.append(process3)
    process3.start()
    for p in jobs:
        p.join()

But i also need to feed these sub processes input via random integers in my case. For that i have a class that can create the next integer via a next() function. I got this to work using Popen and writing to stdin as long as the subprocess doesn't have a returncode.

        prng = CliffPRNG(mode) # my class for random integers
        args = [DIEHARDER, GENERATOR_NUMBER, f'-d{self.testNumber}']
        dieharderTestProc = subprocess.Popen(args, stdout=subprocess.PIPE,
                                             stdin=subprocess.PIPE)
        while dieharderTestProc.returncode is None:
            dieharderTestProc.stdin.write(
                struct.pack('>I', prng.next()))
            dieharderTestProc.poll()

All of this is working fine but it is too slow for my needs. I think one problem is that for every sub process I'm starting, the random integers are created separately, even tough once for all processes would be enough. Does anyone have an idea how i can use just one stream of integers and the sub processes just take the integer when they need them? I hope i somehow described my Problem well enough. I'm not just looking for your solution, i would also appreciate ideas i can look into. Because at the moment i am pretty lost and don't really know how to tackle this Problem. Thanks for your time, let me know if you need more information :)




Aucun commentaire:

Enregistrer un commentaire