vendredi 8 octobre 2021

How do I get newly randomized weights with each repeat of CV for a RandomizedSearchCV

I want to tune my Hyperparameter for the sklearn MLPRegressor. I do already a CV like so:

rkf = RepeatedKFold(n_splits=5, n_repeats=5, random_state=42)
clf = RandomizedSearchCV(MLPRegressor(), params, cv=rkf, scoring='neg_mean_absolute_error', n_iter=10000, n_jobs=-1)
clf.fit(normalizedFeaturesTrain, targetTrain)

So I do already 5 different random splits to get a more consistent result. My Problem is that the result also depends on the random weight initialization of the MLPRegressor (for example r2 is between 0.9 and 0.96 even with 5x 5 Splits). I tried to define a RandomState (np.random.RandomState(42)) for the MLPRegressor but each repeat of the RepeatedKFold and even each iteration of the RandomizedSearchCV uses the same state because I assume that the fit function is not actually called or something like that.

I had the same problem with the actual validation of my model, despite I also used a np.random.RandomState the weight stayed the same for all splits and repeats, but I've overcame the problem by changing this code:

rkf = RepeatedKFold(n_splits=5, n_repeats=10, random_state=1337)
result = cross_validate(regressor, normalizedFeaturesTrain, targetTrain, cv=rkf, scoring=scoring, n_jobs=-1)

to this code:

state = np.random.RandomState(1337)
for x in 10:
    kf = KFold(n_splits=5, shuffle=True, random_state=state.randint(0, high=2147483647))
    regressor.random_state = state.randint(0, high=2147483647)
    result = cross_validate(regressor, normalizedFeaturesTrain, targetTrain, cv=kf, scoring=scoring, n_jobs=-1)

So as you can see, for each iteration not only the KFold split is randomized but also the model weight initialization. I could even do a inner RepeatedKFold instead of KFold to validate each random weight initialization with x random KFold splits if I wish.

But now again my question, how can I accomplish a similar approach for my RandomizedSearchCV? So that the weights for the regressor are newly randomized for each n_repeat (would be the best) or for each n_split (would be ok) or at least for each n_iter (better then nothing)? Because like I said, I testet it with providing a RandomState for the MLPRegressor() but the random_state stays the same over all repeats of my CV and even over all n_iter in general.

The general idea is from: https://stats.stackexchange.com/questions/263999/is-random-state-a-parameter-to-tune

I suggest you select a random state value at random and use it for all your experiments. Alternatively you could take the average accuracy of your models over a random set of random states.

Edit: I'm aware that I could do something like:

state = np.random.RandomState(42)
for x in 10000:
   rkf = RepeatedKFold(n_splits=5, n_repeats=5, random_state=42)
   clf = RandomizedSearchCV(MLPRegressor(random_state=state.randint(0, high=2147483647)), params, cv=rkf, 
   scoring='neg_mean_absolute_error', n_iter=1, n_jobs=-1)
   clf.fit(normalizedFeaturesTrain, targetTrain)

So run a loop n_iter times with only one iteration and change the random state in that loop for the MLPRegressor(). But is there really no better way?




Aucun commentaire:

Enregistrer un commentaire