jeudi 7 avril 2016

Need help generating new text using a random number generator and transition rates

Hello so right now I'm writing a program that basically takes the words in a text file and creates list of structs for each unique word pair. For example the following text: "a b a c a d" would have the following unique word pairs a-b, b-a, a-c, c-a, and a-d. For each unique word pair struct I then create a probability for the transition to the second word in a pair given the first word in a pair. For example a-b would have a 33% transition rate to b if we have the word a, and so from b, we have b-a which has a 100% transition rate.

The second part of my program is to create a "new document" by using the transition rates to see which word should be printed out next. So going from my example above, given the word a, there's a 33% chance of going to b or c, or d and so on. If you go from a to d, then the program terminates since there are no more transitions.

I've successfully done the first part of finding the transition rates, however for the second part my program does not seem to be selecting words properly based on their transition rates. Here are the functions that I'm currently using to generate text:

struct word_pair {
        char* word_one;
        char* word_two;
        int count;
        double p; 
};

char* get_next_word(struct word_pair** pairs, int num_pairs, char* word) {
    int count = 0;
    char* last_word = malloc(200*sizeof(char));
    last_word[0] = '\0';

    while (count != num_pairs) {

        int r = strcmp(word, pairs[count]->word_one);
        if (r == 0) {
            double random_number = rand() / (double)RAND_MAX;
            if ((pairs[count]->p)>=random_number) {
                last_word = pairs[count]->word_two;
                return last_word;
            }
            else {
                last_word = pairs[count]->word_two;
            }
        }
        count++;
    }
    return last_word;
}


void generate_text(struct word_pair** pairs, int num_pairs) {
    int count = 0;
    char* start_word = malloc(200*sizeof(char));
    start_word = pairs[0]->word_one;
    printf("%s ", start_word);

    while (count != 999) {
        start_word = get_next_word(pairs, num_pairs, start_word);
        int r = strcmp("", start_word);
        if (r == 0) {
            break;
        }

        printf("%s ", start_word);
        count++;
    }
}

Basically my function generate_text takes in an array of structs that contain word pairs, where the field p in each struct is the transition probability from word one to word two. Then using pairs[0]->word_one as a starting point, use the function get_next_word to determine which word to transition to.

The get_next_word function takes in the same array of structs as well as the word to transition from. It then goes through all the structs (word pairs) and see's if pairs[count]->word_one is a match to the word to transition from given as an input to the function. If it is, a random number is generated, and if the transition percentage of the word pair is higher or equal to the random number generated, that word is selected to be the next word. If the loop goes through all the structs without one being selected, then it just returns `last_word.

I've done quite a bit of testing and it seems that my program biases the last unique pair in the array of structs where the first word matches the input word for the get_next_word function. For example if I have the input words "a b a c... a aa a bb a cc... a zz" That means there are 49 transitions from the word "a", meaning the program should output just the words "a zz" 1/49 of the time, however I've done ~100 tests and it's more like 1/3 of the time, meaning something is definitely wrong with my program, most likely in selecting the transition words. Can anyone point out where I'm going wrong? Also I've seeded my random number generator with srand(time(NULL) + clock())




Aucun commentaire:

Enregistrer un commentaire