mardi 11 février 2020

Random sample from column of ArrayType Pyspark

I've a column in a Pyspark dataframe with a structure like

Column1
[a,b,c,d,e]
[c,b,d,f,g,h,i,p,l,m]

I'd like to return another column with a random selection of each array in each row, the amount specified in the function.

So something like data.withColumn("sample", SOME_FUNCTION("column1", 5)) returning:

sample
[a,b,c,d,e]
[c,b,h,i,p]

Hopefully avoiding a python UDF, feel like there should be a function available??




Aucun commentaire:

Enregistrer un commentaire