mercredi 25 septembre 2019

Shuffling large memory-mapped numpy array

I have an array of dimension (20000000, 247) of size around 30 GB in a .npy file. I have 32 GB available memory. I need to shuffle the data along rows. I have opened the file in mmap_mode. However, if I try anything other than in-place modification, for example np.random.permutation or creating a random.sampled array of indices p and then returning array[p], I get MemoryError. I have also tried shuffling the in chunks and then try stacking the chunks to build the full array, but MemoryError. The only solution I have found till now is loading the file in mmap_mode = 'r+' and then doing np.random.shuffle. However, it takes forever (it has been 5 hours still it's getting shuffled).

Current code:

import numpy as np
array = np.load('data.npy',mmap_mode='r+')
np.random.seed(1)
np.random.shuffle(array)

Is there any faster method to do this without breaking the memory constraint?




Discrepancy between 2 functions [PHP]

I have 2 functions, in first I'am generation random number and setting probability. In second functions cheking the generated number is in certain range. And if its in certain range setting value to variable.

            function chance($input = array()) {
                $number  = rand(0, array_sum($input) * 10);
                $starter = 0;
                $b       = 3;
                foreach ($input as $key => $val) {
                    $starter += $val * 10;
                    if ($number <= $starter) {
                        $ret = $key;
                        break;
                    }

                }
                return $ret;
                //return $b;;
            }

            function in_range($number, $min, $max, $inclusive = FALSE) {
                if (is_int($number) && is_int($min) && is_int($max)) {
                    return $inclusive ? ($number >= $min && $number <= $max) : ($number > $min && $number < $max);
                }

                return FALSE;
            }

            $array = array(
                rand(0, 9885) => 81.9,
                rand(9886, 9985) => 15.1,
                rand(9986, 9993) => 1,
                rand(9994, 9997) => 0.09,
                rand(9998, 9999) => 0
            );
    $rolledNumber = chance($array);
                $coinsWon = 0;
if (in_range($rolledNumber, 0, 9885) == true) {
        $coinsWon = 5;
                } elseif (in_range($rolledNumber, 9886, 9985) == true) {
                    $coinsWon = 10;
                } elseif (in_range($rolledNumber, 9986, 9993) == true) {
                    $coinsWon = 50;
                }

After a few tests found that there are dismaches. For example:

Getting number 9884, and the value of coinswon is 10, but should be 5. Also I thing that the probability options its not working properly, because for 9986, 9993 numbers I've set 1% probability, but numbers in this range are generated frequently




Exploit Software Insecurity with Bash

I have an executable C program on a linux machine. The program is intended to reveal a "secret" once the password is entered. I found an exploit by using the backup function of the code which is not requiring me to enter the (correct) password. Also, I know where the backup is saved and that the backup name is generated randomly using the time as a seed. However, I have have difficulties writing a shell script which outputs the secret into the terminal.

My attempt was to make the C program create a backup. The problem however, is writing the random number generator in bash using the time as seed to get the file name of the backup. Also, using echo to output to the terminal does not seem to work.

Below is the code of the c program:

#include <stdio.h> 
#include <string.h> 
#include <unistd.h> 
#include<stdlib.h> 
#include <time.h>
// Execute any shell command
void execute(char *cmd)
{
execl("/bin/bash", "bash", "-p", "-c", cmd, NULL);
}
void sanitise(char *password)
{
int i,j;
char tmp[15];
// remove non-alphabet characters from passwords
j=0;
for(i=0; i < 15; ++i)
if(password[i] >= 'a' && password[i] <= 'z') { tmp[j]=password[I];
++j;
} else break; tmp[j] = '\0';
strcpy(password, tmp); }
int authenticate(char *str) {
char stored_password[15]="";
char pass[15];
char path[128] = "/etc/comp2700/bob/password"; int I;
FILE *fpp; int auth=0;
fpp = fopen(path, "r");
if(fpp == NULL) {
printf("Password file %s not found\n", path);
exit(1); }
fgets(stored_password, 15, fpp); sanitise(pass);
strcpy(pass, str); sanitise(pass);
if(strcmp(stored_password,pass) == 0) auth=1;
else {
 auth=0;
}
fclose(fpp);
return auth; }

int main(int argc, char* argv[], char *envp[]) {
unsigned int seed=time(0); 
// use current time as the seed for therandom number generator 
char user[16];
int r;
char command[128];
int choice=0;
if(argc < 4) {
printf("Usage: %s choice user password\n", argv[0]);
printf("To display the secret, use choice=1, e.g. \n\n %s 1 bob password\n\n", argv[0]); 
printf("To backup the secret, use choice=2, e.g., \n\n %s 2 bob password\n\n", argv[0]);
return 0; }
strcpy(user, argv[2]); printf("Welcome, %s!\n", user);
srand(seed); // change the seed for random number generator r = rand();
// generate a random number
choice = atoi(argv[1]);
if(choice == 1) { if(!authenticate(argv[3])) {
printf("Wrong password.\n");
return 0; }
execute("/bin/cat /etc/comp2700/bob/secret"); }
else if(choice == 2) {
// invoke the backup.sh script to copy the secret to the backup folder.
// the filename is randomly generated
sprintf(command, "/home/bob/Public/backup.sh f%d", r); execute(command);
}
else printf("Wrong choice.\n");
return 0; }

File: backup.sh
#!/bin/bash -p
/bin/echo "Copying secret..."
/bin/cat /etc/comp2700/bob/secret > /home/bob/Public/backup/$1

It would really help me if someone could tell me how I can get a shell script to output the secret to the terminal.

Thank you so much in advance!




Generate copula-correlated samples with specified marginals in Python

I have N random variables (X1,...,XN) each of which is distributed over a specific marginal (normal, log-normal, Poisson...) and I want to generate a sample of p joint realizations of these variables Xi, given that the variables are correlated with a given Copula, using Python 3. I know that R is a better option but i want to do it in Python.

Following this method I managed to do so with a Gaussian Copula. Now I want to adapt the method to use a Archimedean Copula (Gumbel, Frank...) or a Student Copula. Ath the beginning of the Gaussian copula method, you draw a sample of p realizations from a multivariate normal distribution. To adapt this to another copula, for instance a bivariate Gumbel, my idea is to draw a sample from the joint distribution of a bivariate Gumbel, but I am not sure on how to implement this.

I have tried using several Python 3 packages : copulae, copula and copulas all provide the noption to fit a particular copula to a dataset but do not allow to draw a random sample from a given copula.

Is there a package that does what I'm looking for, and if not, can you provide some algorithmic insight on how to draw multivariate random samples from a given Copula with uniform marginals ?

Thanks.




Display an item from a list of items

I have a sample:

link

I want the item with the "item-extra" class to be randomly displayed in the item list.

Now, the code rearranges all the elements.

I want the rest to remain fixed and only the one with the "item-extra" class will be moved to each page reload.

var ul = document.querySelector('ul');
for (var i = ul.children.length; i >= 0; i--) {
  ul.appendChild(ul.children[Math.random() * i | 0]);
}
.item-extra {
  color: red;
}
<ul>
  <li>item 1</li>
  <li>item 2</li>
  <li>item 3</li>
  <li>item 4</li>
  <li>item 5</li>
  <li class="item-extra">item extra</li>
</ul>



mardi 24 septembre 2019

Checking if theres two pairs in card hand Python

I am trying to check if there is two pairs in a hand that is random. I have it right now where it'll print one pair so it prints the number of occurrences of that card so if there are 2 twos it'll be 2 x 2 so the first number is the occurrence then the second number is the card number and then to print one pair.

How do I make it where it'll print two pairs instead so checking in a hand of 5 if there is let's say for example 2 x 2 and 2 x 5 so a pair of 2 and 5 then to print out "two pairs".

I added in numbers = cards.count(card) and for the if statement below it then numbers == 2 So that if there is one pair and one pair it prints two pairs and the probability of getting it.

def twopair():
    count = 0
    while True:
        cards = []
        for i in range(5):
            cards.append(random.choice([1,2,3,4,5,6,7,8,9,10,11,12,13]))
        stop = False
        for card in cards:
            number = cards.count(card) # Returns how many of this card is in your hand
            numbers = cards.count(card)
            print(f"{number} x {card}")
            if(number == 2 and numbers == 2):
                print("One Pair")
                stop = True
                break
        if stop:
            break
        else:
            count += 1
    print(f'Count is {count}')



Python probability of a pair

I am trying to make a poker game where it would check if it is a pair or three of a kind or four of a kind.

I am trying to figure out where to insert a while loop. if I should put it in front of the for card in set(cards): statement or the for i in range(5): I want to keep printing 5 cards until it shows either a pair, 3 of a kind, or 4 of a kind.

Then what I want to do is to print the probability of printing one of those options

import random
def poker():
    cards = []
    count = 0
    for i in range(5):
        cards.append(random.choice([1,2,3,4,5,6,7,8,9,10,11,12,13]))
        print(cards)
    for card in set(cards):
        number = cards.count(card) # Returns how many of this card is in your hand
        print(f"{number} x {card}")
        if(number == 2):
            print("One Pair")
            break
        if(number == 3):
            print("Three of a kind")
            break
        if(number == 4):
            print("Four of a kind")
            break