selurl - Normalise and validate URIs for harvesting
Selurl selects and normalises URIs on basis of both general practice (hostname lowercasing, portnumber substsitution etc.) and Combine-specific handling (aplpying config_allow, config_exclude, config_serveralias and other relevant config settings).
The Config settings catered for currently are:
maxUrlLength - the maximum length of an unnormalised URL allow - Perl regular to identify allowed URLs exclude - Perl regular expressions to exclude URLs from harvesting serveralias - Aliases of server names sessionids - List sessionid markers to be removed
A selurl object can hold a single URL and has methods to obtain its subparts as defined in URI.pm, plus some methods to normalise and validate it in Combine context.
Currently, the only schemes supported are http, https and ftp. Others may or may not work correctly. For one thing, we assume the scheme has an internet hostname/port.
clone() will only return a copy of the real URI object, not a new selurl.
URI URI-escapes the strings fed into it by new() once. Existing percent signs in the input are left untouched, which implicates that:
(a) there is no risk of double-encoding; and
(b) if the original contained an inadvertent sequence that could be interpreted as an escape sequence, uri_unescape will not render the original input (e.g. url_with_%66_in_it goes whoop) If you know that the original has not yet been escaped and wish to safeguard potential percent signs, you'll have to escape them (and only them) once before you offer it to new().
A problem with URI is, that its object is not a hash we can piggyback our data on, so I had to resort to AUTOLOAD to emulate inheritance. I find this ugly, but well, this *is* Perl, so what'd you expect?
To install Combine::UA, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Combine::UA
CPAN shell
perl -MCPAN -e shell install Combine::UA
For more information on module installation, please visit the detailed CPAN module installation guide.