lundi 4 mai 2020

Generating a random sample of BigInt values

For a game implemented in JavaScript, I need to produce a random list of n unique numbers in the range [0, N) where N may be greater than Number.MAX_SAFE_INTEGER. This poses three important challenges:

  • Memory cost cannot be O(N), as we wouldn't have enough RAM. This means we cannot use a modified Fisher-Yates shuffle algorithm to take n shuffled values from an array of N elements. (JS arrays are limited to 32-bit indexes, but we'd run out of RAM long before that.)
  • Execution time should be as close as possible to O(n) and not O(N). I expect n to be relatively small (< 100).
  • Most Math operations use double-precision floating point values and will not work with numbers this big.

I found a solution to the first two challenges in Kim-Hung Li's reservoir sampling algorithm L. To summarize, the idea behind this algorithm is that we shuffle the first n numbers to form an initial reservoir, then we progressively overwrite some of it with random numbers taken from the rest of the range. We use a geometric progression to determine how many numbers to skip on each iteration.

Assume that sampleSize is a number and populationSize is a BigInt in the following code:

// Creates an array initialized with [0n, 1n, ... BigInt(count-1)].
function bigRange (count) {
    const array = Array(count);
    for (let i = 0; i < count; i += 1) {
        array[i] = BigInt(i);
    }
    return array;
}

function bigSample (sampleSize, populationSize) {
    const reservoir = shuffle(bigRange(sampleSize));
    let record = BigInt(sampleSize);
    let weight = exp(log(random()) / sampleSize);

    while (true) {
        const skipCount = floor(log(random()) / log(1 - weight));
        record += BigInt(skipCount);

        if (record >= populationSize) {
            return reservoir;
        }

        reservoir[floor(random() * sampleSize)] = record;
        record += BigInt(1);
        weight *= exp(log(random()) / sampleSize);
    }
}

This code, however, will not work for arbitrarily large numbers as the precision of the floating point numbers is limited. As the populationSize grows, non-significant bits of the floating point numbers are eventually used and we no longer have uniform distribution, then we get into unsafe integer territory, and finally it's possible for skipCount to become Infinity. Knowing this, I'm left wondering...

  • At which point does this function become unreliable?
  • Is there a way for me to improve this algorithm to compensate for JavaScript's limitations?
  • Are there any better alternatives?

Note that my knowledge of probability is limited, and I wrote this code after a week spent digging through scientific papers I barely understood. I do have an alternative approach, but it would be much less efficient and would require mixing random number generation with the puzzle generation logic.




Aucun commentaire:

Enregistrer un commentaire