uniq-files - Report or omit duplicate file contents
This document describes version 0.11 of uniq-files (from Perl distribution App-UniqFiles), released on 2015-09-03.
% uniq-files [options] <files>...
List all files which do no have duplicate contents:
% uniq-files *
List all files which have duplicate contents:
% uniq-files -d *
List number of occurences of contents for each file:
% uniq-files -c *
Given a list of filenames, will check each file size and content for duplicate content. Interface is a bit like the
uniq Unix command-line program.
* marks required options.
Set path to configuration file.
Can be specified multiple times.
Set configuration profile to use.
Do not use any configuration file.
Do not read environment for default options.
Choose output format, e.g. json, text.
Set output format to json.
When outputing as JSON, strip result envelope.
By default, when outputing as JSON, the full enveloped result is returned, e.g.:
The reason is so you can get the status (1st element), status message (2nd element) as well as result metadata/extra result (4th element) instead of just the result (3rd element). However, sometimes you want just the result, e.g. when you want to pipe the result for more post-processing. In this case you can use `--naked-res` so you just get:
Whether to return each file content's number of occurence.
1 means the file content is only encountered once (unique), 2 means there is one duplicate, and so on.
Can be specified multiple times.
Display help message and exit.
If set to 0, uniqueness will be determined solely from file size. This can be quicker but might generate a false positive when two files of the same size are deemed as duplicate even though their content are different.
Whether to return duplicate items.
Can be set to either 0, 1, 2.
If set to 2 (the default), will only return the first of duplicate items. For example: file1 contains text 'a', file2 'b', file3 'a'. Only file1 will be returned because file2 is unique and file3 contains 'a' (already represented by file1).
If set to 1, will return all the the duplicate items. From the above example: file1 and file3 will be returned.
If set to 0, duplicate items will not be returned.
Display program's version and exit.
Alias for --noreport-unique --report-duplicate=1.
Alias for --report-unique --report-duplicate=0.
This script has shell tab completion capability with support for several shells.
To activate bash completion for this script, put:
complete -C uniq-files uniq-files
in your bash startup (e.g.
~/.bashrc). Your next shell session will then recognize tab completion for the command. Or, you can also directly execute the line above in your shell to activate immediately.
It is recommended, however, that you install shcompgen which allows you to activate completion scripts for several kinds of scripts on multiple shells. Some CPAN distributions (those that are built with Dist::Zilla::Plugin::GenShellCompletion) will even automatically enable shell completion for their included scripts (using
shcompgen) at installation time, so you can immadiately have tab completion.
To activate tcsh completion for this script, put:
complete uniq-files 'p/*/`uniq-files`/'
in your tcsh startup (e.g.
~/.tcshrc). Your next shell session will then recognize tab completion for the command. Or, you can also directly execute the line above in your shell to activate immediately.
It is also recommended to install
shcompgen (see above).
For fish and zsh, install
shcompgen as described above.
Specify additional command-line options
This script can read configuration file, which by default is searched at
/etc/uniq-files.conf (can be changed by specifying
--config-path). All found files will be read and merged.
To disable searching for configuration files, pass
Configuration file is in the format of IOD, which is basically INI with some extra features.
You can put multiple profiles in a single file by using section names like
[profile=SOMENAME]. Those sections will only be read if you specify the matching
List of available configuration parameters:
check_content (see --no-check-content) count (see --count) files (see --files) format (see --format) naked_res (see --naked-res) report_duplicate (see --report-duplicate) report_unique (see --no-report-unique)
Please visit the project's homepage at https://metacpan.org/release/App-UniqFiles.
Source repository is at https://github.com/perlancar/perl-App-UniqFiles.
Please report any bugs or feature requests on the bugtracker website https://rt.cpan.org/Public/Dist/Display.html?Name=App-UniqFiles
When submitting a bug or request, please include a test-file or a patch to an existing test-file that illustrates the bug or desired feature.
This software is copyright (c) 2015 by email@example.com.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.