use Algorithm::NGram; my $ng = Algorithm::NGram->new(ngram_width => 3); # use trigrams # feed in text $ng->add_text($text1); # analyze $text1 $ng->add_text($text2); # analyze $text2 # feed in arbitrary sequence of tokens $ng->add_start_token; $ng->add_tokens(qw/token1 token2 token3/); $ng->add_end_token; my $output = $ng->generate_text;
This is a module for analyzing token sequences with n-grams. You can use it to parse a block of text, or feed in your own tokens. It can generate new sequences of tokens from what has been fed in.
Create a new n-gram analyzer instance.
This is the "window size" of how many tokens the analyzer will keep track of. A ngram_width of two will make a bigram, a ngram_width of three will make a trigram, etc...
Returns token window size (e.g. the "n" in n-gram)
Returns n-gram table
Splits a block of text up by whitespace and processes each word as a token. Automatically calls
add_start_token() at the beginning of the text and
add_end_token() at the end.
Adds an arbitrary list of tokens.
Adds the "start token." This is useful because you often will want to mark the beginnings and ends of a token sequence so that when generating your output the generator will know what tokens start a sequence and when to end.
Adds the "end token." See
Generates an n-gram frequency table. Returns a hashref of N => tokens => count, where N is the number of tokens (will be from 2 to ngram_width). You will not normally need to call this unless you want to get the n-gram frequency table.
After feeding in text tokens, this will return a new block of text based on whatever text was added.
Generates a new sequence of tokens based on whatever tokens have previously been fed in.
Given a list of tokens, will pick a possible token to come next.
Returns a hashref of the counts of tokens that follow a sequence of tokens.
Serializes a sequence of tokens for use as a key into the n-gram table. You will not normally need to call this.
Returns the tokens and n-gram (if one has been generated) in a string
Deserializes a string and returns an
Mischa Spiegelmock, <email@example.com>
Copyright 2007 by Mischa Spiegelmock
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.