I have about a million files in my directory, and their number is likely to grow. For machine learning, I would like to randomly sample from those files without replacement. How can I do this very quickly? os.listdir(path) is too slow for me.
dimanche 5 novembre 2023
How can I randomize existing byte array?
I create an array of bytes.
array = bytearray(random.randint(1, 8192))
Now how can I randomize each elements of the array?
Just like,
// with Java
var array = new byte[ThreadLocalRandom.current().nextInt(1, 8192)];
ThreadLocalRandom.current().nextBytes(array);
samedi 4 novembre 2023
Randomization code with unequal allocation ratio
I have 60 patients, 5 cohorts where in each there are 4 female and 8 male. In each cohort the allocation ratio for female is (3:1) so 3 treatment and 1 placebo. And for male is (6:2) so 6 treatment and 2 placebo. Cohort 1 has the 150 mg dose Cohort 2 has 200 mg Cohort 3 has 250 mg Cohort 4 is "optional treatment" Cohort 5 is "optional treatment"
I need to create a randomization code in a way that the final output looks like below:
## Cohort 1 - 150 mg ##
## Female ##
| Randomization number | Treatment |
|---|---|
| 101 | Treatment |
| 102 | Placebo |
| 103 | Treatment |
| 104 | Treatment |
## Cohort 1 - 150 mg ##
## Female ##
| Treatment | Count |
|---|---|
| Treatment | 3 |
| Placebo | 1 |
Cohort 1 - 150 mg
## Male ##
| Randomization number | Treatment | | -------- | -------------- | | 105 | Treatment | | 106 | Treatment | | 107 | Treatment | | 108 | Placebo | | 109 | Treatment | | 110 | Treatment | | 111 | Treatment | | 112 | Placebo | ## Cohort 1 - 150 mg ## ## Male ##
| Treatment | Count |
|---|---|
| Treatment | 6 |
| Placebo | 2 |
## Cohort 2 - 200 mg ##
## Female ##
| Randomization number | Treatment |
|---|---|
| 201 | Treatment |
| 202 | Treatment |
| 203 | Treatment |
| 204 | Placebo |
## Cohort 2 - 200 mg ##
## Female ##
| Treatment | Count |
|---|---|
| Treatment | 3 |
| Placebo | 1 |
Cohort 2 - 200 mg
## Male ##
| Randomization number | Treatment |
|---|---|
| 205 | Placebo |
| 206 | Treatment |
| 207 | Treatment |
| 208 | Placebo |
| 209 | Treatment |
| 210 | Treatment |
| 211 | Treatment |
| 212 | Treatment |
## Cohort 2 - 200 mg ##
## Male ##
| Treatment | Count |
|---|---|
| Treatment | 6 |
| Placebo | 2 |
. . . Up to cohort 5:
## Cohort 5 - Optional dose ##
## Female ##
| Randomization number | Treatment |
|---|---|
| 501 | Treatment |
| 502 | Treatment |
| 503 | Placebo |
| 504 | Treatment |
## Cohort 5 - Optional dose ##
## Female ##
| Treatment | Count |
|---|---|
| Treatment | 3 |
| Placebo | 1 |
Cohort 5 - Optional dose
## Male ##
| Randomization number | Treatment |
|---|---|
| 505 | Treatment |
| 506 | Treatment |
| 507 | Treatment |
| 508 | Treatment |
| 509 | Treatment |
| 510 | Treatment |
| 511 | Placebo |
| 512 | Placebo |
## Cohort 1 - Optional dose ##
## Male ##
| Treatment | Count |
|---|---|
| Treatment | 6 |
| Placebo | 2 |
then I need to create a pdf output, an excel output and the sas output. In addition, I want a pdf output in a way that each randomization number for each male and female and the allocated treatment/placebo be printed separately on each pdf page.
I also need the same exact output but with the following randomization number (where in the title for each page, a "replacement" will be added):
Female cohort 1: 181 182 183 184 Male cohort 1: 185 186 187 188 189 190 191 192
Female cohort 2: 281 282 283 284 Male Cohort 2: 285 286 287 288 289 290 292 292
up to cohort 5: 581,...,585 (female) 585,...,592 (male)
Which I am having hard time creating them as I am not familiar with SAS programming at all. This is what I worked on so far but the problem is I could only do it for the equal number of male and female in each cohort and I cannot separate them in a way that in one pdf page I get female and the next page male (like the above output) and also having hard time creating the randomization number as above.
options ls=132 ps=58 nodate nonumber; title1 "RANDOMIZATION SCHEDULE"; title2 "A Randomized, Double-Blind, Placebo-Controlled, MAD Study";
%let seed=9683473;
data rannum(keep=seed);
do i=1 to 5;
rannum=ranuni(&seed);
seed=int(rannum*1000000);
output;
end; run;
proc sql noprint;
select seed
into :seed1 - :seed5
from rannum;
quit;
proc format;
value sex 1='Female'
2='Male';
value treat 1='Placebo'
2='150 mg'
3='200 mg'
4='250 mg'
5='optional dose'
6='optional dose';
run;
proc plan seed=&seed1;
factors sex=2 treat=12 /noprint;
output out=data1
sex nvals=(1 2) random
treat nvals=(1 1 2 2 2 2 2 2)
random;
run;
proc plan seed=&seed2;
factors sex=2 treat=12 /noprint;
output out=data2
sex nvals=(1 2) random
treat nvals=(1 1 3 3 3 3 3 3)
random;
run;
proc plan seed=&seed3;
factors sex=2 treat=12 /noprint;
output out=data3
sex nvals=(1 2) random
treat nvals=(1 1 4 4 4 4 4 4)
random;
run;
proc plan seed=&seed4;
factors sex=2 treat=12 /noprint;
output out=data4
sex nvals=(1 2) random
treat nvals=(1 1 5 5 5 5 5 5)
random;
run;
proc plan seed=&seed5;
factors sex=2 treat=12 /noprint;
output out=data5
sex nvals=(1 2) random
treat nvals=(1 1 6 6 6 6 6 6)
random;
run;
%macro combine(dataset);
%do i=1 %to 5;
%let dataset=data&i;
proc sort data=&dataset; by sex;
data &dataset;
set &dataset;
cohort=&i;
subid=%eval(&i.00)+(_n_);
%end; %mend;
%combine
data combine;
set data1 data2 data3 data4 data5
run;
proc sort; by cohort sex;
print data=combine noobs double
proc
uniform split='*';
var sex subid treat;
id cohort;
by cohort;
label subid="Randomization number"
treat="TREATMENT";
format treat treat.;
run;
vendredi 3 novembre 2023
How can I make my pygame shapes remain on the screen?
I have this program where I want to make this shape randomly appear and remain within this area 11 times. I've managed to make the shape randomly appear in this area 11 times, but I'm struggling to find a way for the shapes to remain. The circles appear and then disappear right away.
count = 0
while running:
num_flowers = 11
while count <= num_flowers:
flowers_x = random.randrange(640)
flowers_y = random.randrange(350, 470)
pygame.draw.circle(screen, (255, 255, 255), (x + flowers_x, y + flowers_y), 5)
count += 1
How can I use openssl rand to generate more than 2GiB of data?
Can I use the "openssl rand" command to generate more than 2GiB of data?
openssl rand 2147483647 | pv > /dev/null
2.00GiB 0:00:01 [1.45GiB/s] [ <=>
openssl rand 2147483648
rand: Value "2147483648" outside integer range
rand: Use -help for summary.
The limit of 2GiB seems to be arbitrary.
Thank you!
PS: I'm using this command
openssl enc -e -chacha20 -k $(pwgen -ync 40 1) -pbkdf2 -in /dev/zero | openssl enc -e -aes-256-ctr -k $(pwgen -ync 40 1) -pbkdf2 | pv > /dev/null
to generate an endless stream of random data. It's useful for example to prepare the hard-drive for the encryption.
jeudi 2 novembre 2023
Program to read a file and generate a random number
Let me explain the problem first. The below shows the generation of a random number from a text file having a book.
* **Seed Initialization**: The PRNG (Pseudorandom Number Generator) is initialized with a given seed. This dictates the starting position within the text. For instance, with a seed of 1,000, the process would start from the 1,000th character in the text.
* **Pair Formation**:
* To generate pairs of letters, start at the initial seed position and move forward by a predetermined step (e.g., 100 characters). This ensures the letters in each pair are independent of one another.
* After picking the first letter, skip the step interval to select the second letter, forming your first pair.
* If both letters in a pair are same , the pair is discarded and not used in the process.
* Continue this process to create subsequent pairs. For our purpose, this is done 64 times, resulting in 32 letter pairs.
* **Binary Representation**:
* For each of the 32 pairs, determine the relationship between the letters based on their ASCII values.
* If the first letter has a greater ASCII value than the second, assign a "1" to a new column for that pair. Otherwise, assign a "0". This assignment is based on the premise that there's a 50% chance for either scenario.
* **Probability Assignment**:
* Each pair is then given a decreasing probability value. The first pair gets a probability of 1/2, the second pair 1/4, the third pair 1/8, and so on. This pattern continues for all pairs.
* **Random Number Generation**:
* Utilize the assigned probabilities and the binary representation (1 or 0) from the earlier step to compute a pseudorandom number. This can be achieved using the formula:
This method effectively utilizes the text to produce pseudorandom numbers based on the initialized seed and predetermined step value.
To assist with understanding and visualization, these two attached pictures can help complement this description.
Here's my code for the same. My main issue is when char2 is an empty string, the seed resets to 1000 which may cause the random number to be repeated. I tried using continue but it becomes an infinite loop. I was hoping to get some help in that part. I get a type error if I don't that if statement.
from statistics import mean
class WarAndPeacePseudoRandomNumberGenerator:
def __init__(self,seed=1000,step = 100):
self.seed = seed
self.step = step
self.fileName = "war-and-peace.txt"
self.file = open(self.fileName, 'r') # open a file for reading
self.file.seek(0) # start from 0
def readCharacters(self,seed,step):
#print(self.seed)
self.file.seek(self.seed)
char1 = self.file.read(1)
self.file.seek(self.seed+self.step)
char2 = self.file.read(1)
return char1, char2
def generateList(self):
binarylist = []
pairs_generated = 0
while pairs_generated < 32:
char1, char2 = self.readCharacters(self.seed, self.step)
if not char2:
# If char2 is empty (end of file), reset the seed and step to start over
self.seed = 1000
self.step += 100
continue
if char1 == char2:
self.step += 100 # Increase the step by 100 places
self.file.seek(self.seed + self.step)
char2 = self.file.read(1) # Read char2 again with the updated step
continue
if ord(char1) > ord(char2):
binarylist.append(1)
else:
binarylist.append(0)
pairs_generated += 1
self.step += 100
self.seed = self.seed + self.step
return binarylist
def closeFile(self):
if self.file is not None:
self.file.close()
def random(self):
randomNumber = 0.0
probDenom= 2
binaryList = self.generateList()
for i in range (len(binaryList)):
randomNumber = randomNumber + binaryList[i] * (1/probDenom)
probDenom = probDenom * 2
return randomNumber
def main():
randomNumberList = []
prng2 = WarAndPeacePseudoRandomNumberGenerator(12345) # With seed
r1 = prng2.random() # Random number 1
r2 = prng2.random() # Random number 2
prng2.closeFile() # Closing the file
prng = WarAndPeacePseudoRandomNumberGenerator() # Without seed value
for i in range(10): # Generating 10000 random numbers
r = prng.random() #
randomNumberList.append(r) # Appending the random number obtained to the list
prng.closeFile() # Closing the file
median = mean(randomNumberList)
minimum = min(randomNumberList)
maximum = max(randomNumberList)
print("Random numbers generated with seed 12345: {} {}".format(r1, r2))
print("\nAfter generating 10000 random numbers")
print("\nMedian: {} ".format(median))
print("\nMinimum: {} ".format(minimum))
print("\nMaximum: {} ".format(maximum))
print("random num: {}".format(randomNumberList[1:5])) # testing
if __name__ == '__main__':
main()
how to replace every element of a zeros numpy array with a random number with uniform distribution between -1 and 1 including both -1 and 1 in python
I want to generate a numpy array with a specific shape filled with zeros. then after some operations replace every single element of the array with a random uniform distribution number between [-1, 1] including both -1 and 1.
I tried this code below but it seems as discussed Here [low, high) (includes low, but excludes high this code does not include 1 ever.
import numpy as np
np.random.seed(0)
# the shape must be one of the following.
# the_shape_of_array = (5)
# the_shape_of_array = (8, 5)
the_shape_of_array = (3, 8, 5)
my_array = np.zeros(the_shape_of_array)
# Doing some operations with the array
length_of_shape_of_my_array = len(my_array.shape)
if length_of_shape_of_my_array == 1:
for element_index, _ in enumerate(my_array):
my_array[element_index] = np.random.uniform(-1, 1)
elif length_of_shape_of_my_array == 2:
for row_index, row in enumerate(my_array):
for element_index, _ in enumerate(row):
my_array[row_index][element_index] = np.random.uniform(-1, 1)
elif length_of_shape_of_my_array == 3:
for block_index, block in enumerate(my_array):
for row_index, row in enumerate(block):
for element_index, _ in enumerate(row):
my_array[block_index][row_index][element_index] = np.random.uniform(
-1, 1
)
print(my_array)
I also tried 2 * np.random.rand() - 1 instead np.random.uniform(-1, 1) but it was the same.
what should I do?
I tried the code that I sent and I expect the code to generate random values including both -1 and 1.
Inscription à :
Articles (Atom)