jeudi 7 décembre 2017

How to get the same result of sampling the dataframe in Apache Spark?

My question is similar to this post:

How to select a same-size stratified sample from a dataframe in Apache Spark?

I have a dataframe in Spark 2, scala, as shown below where clients have between 3 to tens of products. I would like to create a new test dataframe that will have all the clients in the original dataframe but with only 1 randomly sampled product for each client.

client_id product_id| rating|

+--------+--------------+--------------------+

|67778705|1.............|1...................|

|67778705|2.............|2...................|

|67778705|3.............|1...................|

|67778705|4.............|2...................|

|67778705|5.............|1...................|

|68950272|2.............|1...................|

|68950272|6.............|1...................|

|68950272|8.............|2...................|

I need to get the SAME dataframes on different sessions (to be able to reproduce the results).

I try to use this code:

import org.apache.spark.sql.functions.row_number
import org.apache.spark.sql.expressions.Window

val testDF = DF.orderBy("client_id", "product_cluster_id").
    withColumn("random_column",rand(seed=1L)).
    withColumn("row_num",row_number().over(Window.partitionBy("client_id").
    orderBy("random_column"))).filter("row_num" === 1).cache()

I expect that using this code in different applications I get the same samples, e.g.

client_id product_id| rating|

+--------+--------------+--------------------+

|67778705|4.............|2...................|

|68950272|2.............|1...................|

But chosen product_ids are quite different in different applications. Any advice?




Aucun commentaire:

Enregistrer un commentaire