vendredi 7 septembre 2018

Creating a column of random numbers with mean based on another column using python

I would like to be able to basically pass a column into the np.random.normal() function.

I have the following;

def calc_z(w,S,a1,a2,yt1,yt2):

    mu = w * S
    print 'Mu' , mu
    sigma = mt.sqrt(0.5)
    z = np.array(np.random.normal(mu,sigma))
    u = [a1,a2,z]
    yt = [yt1,yt2,1]
    thetaset = np.random.rand(len(u))
    m = [i for i in range(len(u))]

    max_iter = 30

#Calculate E-step
    for i in range(max_iter):

        print 'Iteration:', i
        print 'z:', z
        print 'thetaset', thetaset

        devLz = eq6(var,w,S,z,yt,u,thetaset,m)
        dev2Lz2 = eq9(var,thetaset,u)

#Calculate M-Step
        z = z - (devLz / dev2Lz2)
        w = lambdaw * z

        for i in range(len(thetaset)):

            devLTheta = eq7(yt,u,thetaset,lambdatheta)
            dev2LTheta2 = eq10(thetaset,u,lambdatheta)           

            thetaset = thetaset - (devLTheta / dev2LTheta2)

    return float(z)

calc_z_udf = udf(calc_z,FloatType())

data.show()

data = data.withColumn('z', calc_z(data['w'],data['Org_Depth_Diff_S'],data['proximity_rank_a1'],data['cotravel_count_a2'],data['cotravel_yt1'],data['proximity_yt2']))

But when I pass S in, the np.random.normal function doesn't like being passed a column and gives me the following error;

Traceback (most recent call last):
  File "/home/taylorr2/PySparkLatent3.py", line 125, in <module>
    data = data.withColumn('z', calc_z(data['w'],data['Org_Depth_Diff_S'],data['proximity_rank_a1'],data['cotravel_count_a2'],data['cotravel_yt1'],data['proximity_yt2']))
  File "/home/taylorr2/PySparkLatent3.py", line 90, in calc_z
    z = np.array(np.random.normal(mu,sigma))
  File "mtrand.pyx", line 1282, in mtrand.RandomState.normal (numpy/random/mtrand/mtrand.c:6920)
ValueError: setting an array element with a sequence.

I am trying to think of a way to get this function to accept this value or do this a different way.

Thanks!




Aucun commentaire:

Enregistrer un commentaire