My question is similar to this post:
How to select a same-size stratified sample from a dataframe in Apache Spark?
I have a dataframe in Spark 2, scala, as shown below where clients have between 3 to tens of products. I would like to create a new test dataframe that will have all the clients in the original dataframe but with only 1 randomly sampled product for each client.
client_id product_id| rating|
+--------+--------------+--------------------+
|67778705|1.............|1...................|
|67778705|2.............|2...................|
|67778705|3.............|1...................|
|67778705|4.............|2...................|
|67778705|5.............|1...................|
|68950272|2.............|1...................|
|68950272|6.............|1...................|
|68950272|8.............|2...................|
I need to get the SAME dataframes on different sessions (to be able to reproduce the results).
I try to use this code:
import org.apache.spark.sql.functions.row_number
import org.apache.spark.sql.expressions.Window
val testDF = DF.orderBy("client_id", "product_cluster_id").
withColumn("random_column",rand(seed=1L)).
withColumn("row_num",row_number().over(Window.partitionBy("client_id").
orderBy("random_column"))).filter("row_num" === 1).cache()
I expect that using this code in different applications I get the same samples, e.g.
client_id product_id| rating|
+--------+--------------+--------------------+
|67778705|4.............|2...................|
|68950272|2.............|1...................|
But chosen product_ids are quite different in different applications. Any advice?
Aucun commentaire:
Enregistrer un commentaire