mardi 11 juillet 2017

SparkSql Random using column as seed

I have a Dataset with the following columns:

Id, seed

my dataset has 10M per 100k rows and every 100k rows the seed change. I would like to add a column "random" as follows:

 data.withColumn("random",rand(col("seed")))

the problem is that function org.apache.spark.sql.functions.rand accept only a value not a column. Is there a way to generate randoms with sparksql and a column as seed?

Thanks




Aucun commentaire:

Enregistrer un commentaire