Python: Generate Lists with Random Unique Numbers outside a given Dataset Pattern

I have the following lists (as a simple dataset in a CSV file). Some lists contain consecutive, non-repetitive integer numbers and others contain non-repetitive unique numbers that follow a pattern:

sample-list-1.csv

N = 5

list1 = [1,  2,  3,  4,  5]
list2 = [2,  3,  4,  5,  6]
list3 = [1,  7, 13, 19, 25]
list4 = [7, 13, 19, 25, 31]

and so on... each list is in a separate row and each number is in a separate column - no headers or index in the csv

At a closer look, the numbers in the lists may represent a matrix, which is totally correct, but with N+1... i.e, a 6x6 matrix:

6 x 6 matrix

1   2   3   4   5   6
7   8   9   10  11  12
13  14  15  16  17  18
19  20  21  22  23  24
25  26  27  28  29  30
31  32  33  34  35  36

My goal is to generate another list (or lists) that contain 'N' non-consecutive, non-repetitive, unique/random numbers which do not follow the patterns in the original list Dataset. The N numbers should, of course, be among the 36 numbers from the matrix.
An error margin of maximum 2 consecutive numbers in a row or 2 continuous numbers (column-vice) can also be taken into consideration... but the other numbers in the generated list should follow the above rules.

So far, I'm able to import the CSV file and able to generate a list of random, non-repetitive numbers with the simple Code:

import random
import csv

N = 5
pick = random.sample(range(1, (N*N)+1), N-1)
#print("pick:", pick)

with open('sample-list-1.csv', newline = '') as f:
    reader = csv.reader(f)
    csvdata = list(reader)

csvdata_list = [int(x) for x in csvdata[2]]
#print("csvdata_list:", csvdata_list)

for n in pick:
    for i in csvdata_list:
        if n == i:
            print("n=", n, "i=", i)

And I'm able to do some simple comparisons with the original Dataset and the random number list using for loops...
However, my current method only iterates through a single list/pattern at a time, and I wish to iterate through multiple lists and their patterns to generate uniqueness in the new lists.

Any ideas on how I can achieve true randomness outside the patterns in the original Dataset using Code?
Thanks in advance!