Text::Identify::BoilerPlate - Remove repeated text
Version 0.3.1
Finds boilerplate text (lines that are repeated across documents) in a list of plain text files.
use Text::Identify::BoilerPlate; my @files = ('file1', 'file2', 'file3'); rem_boilerplate(\@files, { min_dupl => 4, ignore_digits => 0 });
New files are written, containing everything but the boilerplate text.
rem_boilerplate() takes two arguments: A reference to a list of files to be processed, and a reference to a hash of options.
rem_boilerplate()
The options are:
min_dupl
The minimum number of thimes a line has to occur to be considered boilerplate (default: 3). Can be either an integer or a percentage ('50%') of the number of files processed. Minimum value: 2.
ignore_digits
Lines only seperated by differences in digits will be considered duplicates (default: yes).
suffix
Added to the new files (default: 'content').
only_headers_and_footers
Only sets consecutive lines of duplicates at the start and end of documents are considered boilerplate (default: yes).
digest
Lines will be replaced by a MD5 digest during duplicate compilation, saving memory (default: no).
log
Nname of the log file, where deleted lines are recorded; if set to false, no log will be created (default: './text-identify-boilerplate.log').
Lars Nygaard, <lars.nygaard@inl.uio.no>
<lars.nygaard@inl.uio.no>
The program needs extensive testing and tweaking before the simple algorithm can give consistently high-quality results.
Please report any bugs or feature requests to bug-text-identify-boilerplate@rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Text-Identify-BoilerPlate. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
bug-text-identify-boilerplate@rt.cpan.org
Copyright 2005 Lars Nygaard, all rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Text::Identify::BoilerPlate, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Text::Identify::BoilerPlate
CPAN shell
perl -MCPAN -e shell install Text::Identify::BoilerPlate
For more information on module installation, please visit the detailed CPAN module installation guide.