Shuffling specific element within a nested list of lists

I'm trying to randomly shuffle one element within a nested list of lists.

E.g. I have:

list1=[[['a', 'b', 'c', 1], ['a', 'b', 'c', 2]], [['a', 'b', 'c', 3], ['a', 'b', 'c', 4]], [['a', 'b', 'c', 5], ['a', 'b', 'c', 6]]]

I want to randomly shuffle the 3rd element of each sub-list (the float) while keeping the rest of the list entries and their structure within this nested list intact. E.g. as such:

list1=[[['a', 'b', 'c', 1], ['a', 'b', 'c', 6]], [['a', 'b', 'c', 4], ['a', 'b', 'c', 2]], [['a', 'b', 'c', 3], ['a', 'b', 'c', 5]]]

So far, I've come up with the following:

import random

list1 = [[['a', 'b', 'c', 1], ['a', 'b', 'c', 2]], [['a', 'b', 'c', 3], ['a', 'b', 'c', 4]], [['a', 'b', 'c', 5], ['a', 'b', 'c', 6]]]

vals = [x[3] for line in list1 for x in line]
shuffled_vals = random.sample(vals,len(vals))

counter = 0
for i in range(len(list1)):
    for j in range(len(list1[i])):
        list1[i][j][3] = shuffled_vals[counter]
        counter += 1

Although this works as intendend, I'd be curious if there is a more elegant / Pythonic solution. Also, I'm not sure how well this scales - the list I intend to use it on will contain several million entries.

Any tips for improving this code (to a more Pythonic or more efficient one) are greatly appreciated.