jeudi 4 mars 2021

Uniformly spreading Kinesis partition key algorithm

I have an Apache Flink app running on AWS Kinesis Data Analytics that writes high volume of data to Kinesis stream. I want to distribute the data evenly across the shards. Whats the best algorithm for KinesisPartitioner to generate appropriate shard keys? I understand that eg UUID.randomUUID can be quite slow, especially when using older JVM versions. I've heard Mersenne Twister and Murmur2/3 mentioned in the context, but don't know much about it...




Aucun commentaire:

Enregistrer un commentaire