I created a function to generate random values in a given range for DF columns.
test_df = self.spark.createDataFrame([(1, 'metric1', 10.5), (2, 'metric2', 20.7), (3, 'metric3', 30.1)], ['id', 'metric', 'score'])
def generate_rand_value(col: Column) -> Column:
lower = col - (col * RANGE)
upper = col + (col * RANGE)
return random.uniform(lower, upper)
Then I decided to modify it to generate a fixed number for each column value in a range using seed:
def generate_fixed_rand_value(column: Column):
random.seed(5)
return random.randint(column, 10)
This results in an error: TypeError: int() argument must be a string, a bytes-like object or a number, not 'Column'. What will be the right way to call the function for generating fixed float numbers for each column value? Or maybe there is a more suitable approach for that?
I call the function like this:
def parse_cols(df, cols: list):
for col_name in cols:
df = df.withColumn(col_name, generate_fixed_rand_value(F.col(col_name)))
return df
Aucun commentaire:
Enregistrer un commentaire