Search::Tools::TokenList - a bunch of tokens from a Tokenizer
use Search::Tools::Tokenizer; my $tokenizer = Search::Tools::Tokenizer->new(); my $tokens = $tokenizer->tokenize('quick brown red dog'); # use like an iterator while ( my $token = $tokens->next ) { # token isa Search::Tools::Token } # iterate the other way while ( my $token = $tokens->prev ) { # ... } # fetch a particular token my $token = $tokens->get_token( $position ); # reset the iterator $tokens->reset; # get the current iterator position my $pos = $tokens->pos; # set the iterator position $tokens->set_pos( $pos + 1 ); # how many tokens originally? my $num = $tokens->num; # treat like array push( @{ $tokens->as_array }, $new_token ); # now how many tokens? my $len = $tokens->len; # $len != $num # get all the hot tokens my $heat = $tokens->get_heat; # get all the matches to the regex in Tokenizer my $matches = $tokens->get_matches; # just the number of matches my $num_matches = $tokens->num_matches;
A TokenList is an object containing Tokens. You may treat it like an iterator or an array, and call methods on it to get/set attributes.
Most of Search::Tools::TokenList is written in C/XS so if you view the source of this class you will not see much code. Look at the source for Tools.xs and search-tools.c if you are interested in the internals, or look at Search::Tools::TokenListPP.
See Search::Tools::TokenListUtils for other methods available on this class.
This class inherits from Search::Tools::Object. Only new or overridden methods are documented here.
Get the next Token.
Get the previous Token.
Get the iterator position.
Set the iterator position.
Same as calling:
$tokens->set_pos(0);
The number of Tokens in the internal AV (array).
The number of Tokens initially parsed by the Tokenizer. This is the same value as len() unless you alter the TokenList via as_array().
Returns an array ref to the internal AV (array) of tokens. If you alter the array, it will alter the len() value but not the num() value.
Prints internal XS attributes to stderr.
Returns an array ref to the internal AV (array) of positions with is_hot() set by the original Tokenizer. This method will return an empty list unless you have passed a heat_seeker to the tokenize() method. See Search::Tools::Tokenizer.
Returns an array ref to the internal AV (array) of sentence start positions for each position in get_heat().
Returns an array ref of all the Tokens with is_match() set. The array is constructed at the time you call the method so if you alter the array it will not affect the TokenList object, but if you alter a Token in the array it will affect the Token in the TokenList object.
Like calling:
my $num = scalar @{ $tokens->matches };
Returns the Token at position. If position is invalid returns undef.
Peter Karman <karman@cpan.org>
<karman@cpan.org>
Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
bug-search-tools at rt.cpan.org
You can find documentation for this module with the perldoc command.
perldoc Search::Tools
You can also look for information at:
RT: CPAN's request tracker
http://rt.cpan.org/NoAuth/Bugs.html?Dist=Search-Tools
AnnoCPAN: Annotated CPAN documentation
http://annocpan.org/dist/Search-Tools
CPAN Ratings
http://cpanratings.perl.org/d/Search-Tools
Search CPAN
http://search.cpan.org/dist/Search-Tools/
Copyright 2009 by Peter Karman.
This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Search::Tools, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Search::Tools
CPAN shell
perl -MCPAN -e shell install Search::Tools
For more information on module installation, please visit the detailed CPAN module installation guide.