How do I randomly shuffle a subset of the elements of an existing matrix in Matlab?

I have a matrix of values ranging from 0 to 3 in Matlab. I want to randomly shuffle the elements of the matrix, but only within the cells that have values in the range 1 -3 (so only within a subset of the whole matrix). Is there a way to do that? Thanks.