mercredi 8 juillet 2020

Pyspark - how to generate same random numbers for each float value of a DataFrame column?

I created a function to generate random values in a given range for DF columns.

test_df = self.spark.createDataFrame([(1, 'metric1', 10.5), (2, 'metric2', 20.7), (3, 'metric3', 30.1)], ['id', 'metric', 'score'])

def generate_rand_value(col: Column) -> Column:
    lower = col - (col * RANGE)
    upper = col + (col * RANGE)
    return random.uniform(lower, upper)

Then I decided to modify it to generate a fixed number for each column value in a range using seed:

def generate_fixed_rand_value(column: Column):
    random.seed(5)
    return random.randint(column, 10)

This results in an error: TypeError: int() argument must be a string, a bytes-like object or a number, not 'Column'. What will be the right way to call the function for generating fixed float numbers for each column value? Or maybe there is a more suitable approach for that?

I call the function like this:

def parse_cols(df, cols: list):
    for col_name in cols:
        df = df.withColumn(col_name, generate_fixed_rand_value(F.col(col_name)))
    return df



Aucun commentaire:

Enregistrer un commentaire