=encoding utf-8
=head1 DISCLAIMER
Note that updating these docs is an ongoing process and some perl5 idioms
might not have been translated yet into correct perl6 idiom. My bad. Sorry.
(Feedback welcome)
=head1 NAME
Text::CSV - comma-separated values manipulation routines
=head1 SYNOPSIS
use Text::CSV;
# Read whole file in memory
my @aoa = csv(in => "data.csv"); # as array of arrays
my @aoh = csv(in => "data.csv",
headers => "auto"); # as array of hashes
# Write array of arrays as csv file
csv(in => @aoa, out => "file.csv", sep => ";");
my @rows;
# Read/parse CSV
my $csv = Text::CSV.new;
my $fh = open "test.csv", :r, chomp => False;
while (my @row = $csv.getline($fh)) {
@row[2] ~~ m/pattern/ or next; # 3rd field should match
@rows.push: @row;
}
$fh.close;
# and write as CSV
$fh = open "new.csv", :w;
$csv.say($fh, $_) for @rows;
$fh.close;
=head1 DESCRIPTION
Text::CSV provides facilities for the composition and decomposition of
comma-separated values. An instance of the Text::CSV class will combine
fields into a C<CSV> string and parse a C<CSV> string into fields.
The module accepts either strings or files as input and support the use of
user-specified characters (or sequences thereof) for delimiters, separators,
and escapes.
In all following documentation, C<WIP> stands for "Work In Progress" and
C<NYI> for "Not Yet Implemented". The goal is to get rid of all of those.
=head2 Embedded newlines
B<Important Note>: The default behavior is to accept only UTF-8 characters.
But you still have the problem that you have to pass a correct line to the
L</parse> method, which is more complicated from the usual point of usage:
my $csv = Text::CSV.new;
for lines() : eager { # WRONG!
$csv.parse($_);
my @fields = $csv.fields;
}
this will break for several reasons. The default open mode is to C<chomp>
lines, which will also remove the newline sequence if that sequence is not
(part of) the newline at all. As the C<for> might read broken lines: it
does not care about the quoting. If you need to support embedded newlines,
the way to go is to B<not> pass L<C<eol>|/eol> in the parser (it accepts
C<\n>, C<\r>, B<and> C<\r\n> by default) and then
my $csv = Text::CSV.new;
my $io = open $file, :r, chomp => False;
while (my $row = $csv.getline($io)) {
my @fields = @$row;
}
=head2 Binary data
For now, Text::CSV only accepts Unicode. Binary data is planned.
=head1 SPECIFICATION
While no formal specification for CSV exists, RFC 4180 I<1>) describes the
common format and establishes C<text/csv> as the MIME type registered with
the IANA. RFC 7111 I<2> adds fragments to CSV.
Many informal documents exist that describe the C<CSV> format. "How To: The
Comma Separated Value (CSV) File Format" I<3>) provides an overview of the
C<CSV> format in the most widely used applications and explains how it can
best be used and supported.
1) http://tools.ietf.org/html/rfc4180
2) http://tools.ietf.org/html/rfc7111
3) http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm
The basic rules are as follows:
B<CSV> is a delimited data format that has fields/columns separated by the
comma character and records/rows separated by newlines. Fields that contain
a special character (comma, newline, or double quote), must be enclosed in
double quotes. However, if a line contains a single entry that is the empty
string, it may be enclosed in double quotes. If a field's value contains a
double quote character it is escaped by placing another double quote
character next to it. The C<CSV> file format does not require a specific
character encoding, byte order, or line terminator format.
=over 2
=item *
Each record is a single line ended by a line feed (ASCII/C<LF>=C<0x0A>) or
a carriage return and line feed pair (ASCII/C<CRLF>=C<0x0D 0x0A>), however,
line-breaks may be embedded.
=item *
Fields are separated by commas.
=item *
Allowable characters within a C<CSV> field include C<0x09> (C<TAB>) and the
inclusive range of C<0x20> (space) through C<0x7E> (tilde). In binary mode
all characters are accepted, at least in quoted fields.
=item *
A field within C<CSV> must be surrounded by double-quotes to contain a
separator character (comma).
=back
Though this is the most clear and restrictive definition, Text::CSV is way
more liberal than this, and allows extension:
=over 2
=item *
Line termination by a single carriage return is accepted by default
=item *
The separation, escape, and escape can be any valid Unicode sequence.
=item *
A field in C<CSV> must be surrounded by double-quotes to make an embedded
double-quote, represented by a pair of consecutive double-quotes, valid.
You may additionally use the sequence C<"0> for representation of a NULL
byte. Using C<0x00> is just as valid.
=item *
Several violations of the above specification may be lifted by passing some
options as attributes to the object constructor.
=back
=head1 METHODS
=head2 version
X<version>
Returns the current module version.
=head2 new
X<new>
Returns a new instance of class Text::CSV. The attributes are described by
the optional named parameters
my $csv = Text::CSV.new(attributes ...);
The following attributes are available:
=over 4
=item eol
X<eol>
my $csv = Text::CSV.new(eol => "\r\n");
$csv.eol(Str);
my $eol = $csv.eol;
The end-of-line string to add to rows for L</print> or the record separator
for L</getline>.
When not set in a B<parser> instance, the default behavior is to accept
C<\n>, C<\r>, and C<\r\n>, so it is probably safer to not specify C<eol> at
all. Passing C<Str> or the empty string behave the same.
As perl6 interprets C<\r\n> as a single grapheme in input, it is dissuaded
to use C<\r\n> as C<eol> when parsing. Please choose C<Str> instead.
When not passed in a B<generating> instance, records are not terminated at
all, so it is probably wise to pass something you expect. A safe choice for
C<eol> on output is either C<\n> or C<\r\n>.
Common values for C<eol> are C<"\012"> (C<\n> or Line Feed), C<"\015\012">
(C<\r\n> or Carriage Return, Line Feed), and C<"\015"> (C<\r> or Carriage
Return).
=item sep
X<sep>
=item sep_char
X<sep_char>
=item sep-char
X<sep-char>
=item separator
X<separator>
my $csv = Text::CSV.new(sep => ";");
$csv.sep("\x[ff0c]"); # FULLWIDTH COMMA
my $sep = $csv.sep;
The sequence used to separate fields, by default a comma: (C<,>). This
sequence is required and cannot be disabled.
The separation sequence can not be equal to the quote sequence, the
escape sequence or the newline sequence.
See also L</CAVEATS>
=item quote
X<quote>
=item quote_char
X<quote_char>
=item quote-char
X<quote-char>
my $csv = Text::CSV.new(quote => "'");
$csv.quote("\x[ff02]); # FULLWIDTH QUOTATION MARK
$csv.quote(Str);
my $quo = $csv.quote;
The sequence to quote fields containing blanks or binary data, by default
the double quote character (C<">). A value of C<Str> disables quotation
(for simple cases only).
The quotation sequence can not be equal to the separation sequence or the
newline sequence.
See also L</CAVEATS>
=item escape
X<escape>
=item escape_char
X<escape_char>
=item escape-char
X<escape-char>
my $csv = Text::CSV.new(escape => "\\");
$csv.escape("\x[241b]"); # SYMBOL FOR ESCAPE
$csv.escape(Str);
my $esc = $csv.escape;
The sequence to escape certain characters inside quoted fields.
The C<escape> defaults to being the double-quote mark (C<">). In other
words the same as the default sequence for L<C<quote>|/quote>. This means
that doubling the quote mark in a field escapes it:
"foo","bar","Escape ""quote mark"" with two ""quote marks""","baz"
If you change L<C<quote>|/quote> without changing C<escape>, C<escape> will
still be the double-quote (C<">). If instead you want to escape
L<C<quote>|/quote> by doubling it you will need to also change C<escape> to
be the same as what you have changed L<C<quote>|/quote> to.
The escape sequence can not be equal to the separation sequence.
=item binary
X<binary>
WIP: Default is True. Non-UTF-8 real binary (Blob) does not yet parse.
Opening the resource with encoding utf8-c8 is most likely the way to go.
my $csv = Text::CSV.new(:binary);
$csv.binary(False);
my $bin = $csv.binary;
If this attribute is C<True>, you may use binary data in quoted fields,
including line feeds, carriage returns and C<NULL> bytes. (The latter could
be escaped as C<"0>.) By default this feature is on.
Note that valid Unicode (UTF-8) is not considered binary.
=item strict
X<strict>
my $csv = Text::CSV.new(:strict);
$csv.strict(False);
my $flg = $csv.strict;
If set to True, any row that parses to a different number of columns than
the previous row will cause the parser to throw error 2014.
=item auto_diag
X<auto_diag>
=item auto-diag
X<auto-diag>
my $csv = Text::CSV.new(auto_diag => False);
$csv.auto_diag(True);
my $a-d = $csv.auto_diag;
Set this attribute to a number between C<1> and C<9> causes L</error_diag>
to be automatically called in void context upon errors. The value C<True>
evaluates to C<1>.
In case of error C<2012 - EOF>, this call will be void.
If C<auto_diag> is set to a numeric value greater than C<1>, it will C<die>
on errors instead of C<warn>.
=item diag_verbose
X<diag_verbose>
=item diag-verbose
X<diag-verbose>
my $csv = Text::CSV.new(diag_verbose => 1);
$csv.diag_verbose(2);
my $d-v = $csv.diag_verbose;
Set the verbosity of the output triggered by C<auto_diag>.
WIP: Does not add any information yet.
=item blank_is_undef
X<blank_is_undef>
=item blank-is-undef
X<blank-is-undef>
my $csv = Text::CSV.new(:blank_is_undef);
$csv.blank_is_undef(False);
my $biu = $csv.blank_is_undef;
Under normal circumstances, C<CSV> data makes no distinction between quoted-
and unquoted empty fields. These both end up in an empty string field once
read, thus
1,"",," ",2
is read as
[ "1", "", "", " ", "2" ]
When I<writing> C<CSV> files with L<C<always_quote>|/always_quote> set, the
unquoted I<empty> field is the result of an undefined value. To enable this
distinction when I<reading> C<CSV> data, the C<blank_is_undef> attribute
will cause unquoted empty fields to be set to C<Str>, causing the above
to be parsed as
[ "1", "", Str, " ", "2" ]
=item empty_is_undef
X<empty_is_undef>
=item empty-is-undef
X<empty-is-undef>
my $csv = Text::CSV.new(:empty_is_undef);
$csv.empty_is_undef(False);
my $eiu = $csv.empty_is_undef;
Going one step further than L<C<blank_is_undef>|/blank_is_undef>, this
attribute causes all empty fields to return as C<Str>, so
1,"",," ",2
is read as
[ 1, Str, Str, " ", 2 ]
Note that this effects only fields that are originally empty, not fields
that are empty after stripping allowed whitespace. YMMV.
=item allow_whitespace
X<allow_whitespace>
=item allow-whitespace
X<allow-whitespace>
my $csv = Text::CSV.new(:allow_whitespace);
$csv.allow_whitespace(False);
my $a-w = $csv.allow_whitespace;
When this option is set to C<True>, the whitespace (C<TAB>'s and C<SPACE>'s)
surrounding the separation sequence is removed when parsing. If either
C<TAB> or C<SPACE> is one of the three major sequences L<C<sep>|/sep>,
L<C<quote>|/quote>, or L<C<escape>|/escape> it will not be considered
whitespace.
Now lines like:
1 , "foo" , bar , 3 , zapp
are parsed as valid C<CSV>, even though it violates the C<CSV> specs.
Note that B<all> whitespace is stripped from both start and end of each
field. That would make it I<more> than a I<feature> to enable parsing bad
C<CSV> lines, as
1, 2.0, 3, ape , monkey
will now be parsed as
[ "1", "2.0", "3", "ape", "monkey" ]
even if the original line was perfectly acceptable C<CSV>.
=item allow_loose_quotes
X<allow_loose_quotes>
=item allow-loose-quotes
X<allow-loose-quotes>
my $csv = Text::CSV.new(:allow_loose_quotes);
$csv.allow_loose_quotes(False);
my $alq = $csv.allow_loose_quotes;
By default, parsing unquoted fields containing L<C<quote>|/quote>'s like
1,foo "bar" baz,42
would result in parse error 2034. Though it is still bad practice to allow
this format, we cannot help the fact that some vendors make their
applications spit out lines styled this way.
If there is B<really> bad C<CSV> data, like
1,"foo "bar" baz",42
or
1,""foo bar baz"",42
there is a way to get this data-line parsed and leave the quotes inside the
quoted field as-is. This can be achieved by setting C<allow_loose_quotes>
B<AND> making sure that the L<C<escape>|/escape> is I<not> equal to
L<C<quote>|/quote>.
=item allow_loose_escapes
X<allow_loose_escapes>
=item allow-loose-escapes
X<allow-loose-escapes>
my $csv = Text::CSV.new(:allow_loose_escapes);
$csv.allow_loose_escapes(False);
my $ale = $csv.allow_loose_escapes;
Parsing fields that have L<C<escape>|/escape> sequences that escape
characters that do not need to be escaped, like:
my $csv = Text::CSV.new(escape_char => "\\");
$csv.parse(q{1,"my bar\'s",baz,42});
would result in parse returning C<False> with reason 2025. Though it is bad
practice to allow this format, this attribute enables you to treat all
escape sequences equal.
=item allow_unquoted_escape
X<allow_unquoted_escape>
=item allow-unquoted-escape
X<allow-unquoted-escape>
my $csv = Text::CSV.new(:allow_unquoted_escape);
$csv.allow_unquoted_escape(False);
my $aue = $csv.allow_unquoted_escape;
A backward compatibility issue where L<C<escape>|/escape> differs from
L<C<quote>|/quote> prevents L<C<escape>|/escape> to be in the first
position of a field. If L<C<quote>|/quote> is equal to the default C<"> and
L<C<escape>|/escape> is set to C<\>, this would be illegal:
1,\0,2
Setting this attribute to C<True> might help to overcome issues with
backward compatibility and allow this style.
=item always_quote
X<always_quote>
=item always-quote
X<always-quote>
my $csv = Text::CSV.new(:always_quote);
$csv.always_quote(False);
my $f = $csv.always_quote;
By default the generated fields are quoted only if they I<need> to be. For
example, if they contain the separator sequence. If you set this attribute
to C<1> then I<all> defined fields will be quoted. (undefined (C<Str>)
fields are not quoted, see L</blank_is_undef>). This makes it quite often
easier to handle exported data in external applications. (Poor creatures
who are better to use Text::CSV. :)
=item quote_empty
X<quote_empty>
=item quote-empty
X<quote-empty>
my $csv = Text::CSV.new(:quote_empty);
$csv.quote_empty(False);
my $q-s = $csv.quote_empty;
By default the generated fields are quoted only if they I<need> to be. An
empty defined field does not need quotation. If you set this attribute to
C<True> then empty defined fields will be quoted. See also
L<C<always_quote>|/always_quote>.
=item quote_space
X<quote_space>
=item quote-space
X<quote-space>
my $csv = Text::CSV.new(:quote_space);
$csv.quote_space(False);
my $q-s = $csv.quote_space;
By default, a space in a field would trigger quotation. As no rule exists
this to be forced in C<CSV>, nor any for the opposite, the default is
C<True> for safety. You can exclude the space from this trigger by setting
this attribute to C<False>.
=item escape_null
X<escape_null>
=item quote-null
X<quote-null>
my $csv = Text::CSV.new(:escape_null);
$csv.escape_null(False);
my $q-n = $csv.escape_null;
By default, a C<NULL> byte in a field would be escaped. This option enables
you to treat the C<NULL> byte as a simple binary character in binary mode
(the C<< binary => True >> is set). The default is C<True>. You can prevent
C<NULL> escapes by setting this attribute to C<False>.
=item quote_binary
X<quote_binary>
=item quote-binary
X<quote-binary>
my $csv = Text::CSV.new(:quote_binary);
$csv.quote_binary(False);
my $q-b = $csv.quote_binary;
By default, all "unsafe" bytes inside a string cause the combined field to
be quoted. By setting this attribute to C<False>, you can disable that
trigger for bytes >= C<0x7F>. (WIP)
=item keep_meta
X<keep_meta>
=item keep-meta
X<keep-meta>
my $csv = Text::CSV.new(:keep_meta);
$csv.keep_meta(False);
my $k-m = $csv.keep_meta_info;
By default, the parsing of input records is as simple and fast as possible.
However, some parsing information - like quotation of the original field -
is lost in that process. Setting this flag to true enables retrieving that
information after parsing with the methods L</meta_info>, L</is_quoted>,
and L</is_binary> described below. Default is C<False> for ease of use.
If C<keep-meta> is set to C<True>, the returned fields are not of type
C<Str> but of type L<C<CSV::Field>|/CSV::Field>.
=item types
NYI
A set of column types; the attribute is immediately passed to the L</types>
method.
=item callbacks
X<callbacks>
See the L</Callbacks> section below.
=back
To sum it up,
$csv = Text::CSV.new;
is equivalent to
$csv = Text::CSV.new(
eol => Nil, # \r, \n, or \r\n
sep => ',',
quote => '"',
escape => '"',
binary => True,
auto_diag => False,
diag_verbose => 0,
blank_is_undef => False,
empty_is_undef => False,
allow_whitespace => False,
allow_loose_quotes => False,
allow_loose_escapes => False,
allow_unquoted_escape => False,
always_quote => False,
quote_space => True,
escape_null => True,
quote_binary => True,
keep_meta => False,
verbatim => False,
types => Nil,
callbacks => Nil,
});
For all of the above mentioned flags, an accessor method is available where
you can inquire the current value, or change the value
my $quote = $csv.quote;
$csv.binary(True);
It is not wise to change these settings halfway through writing C<CSV> data
to a stream. If however you want to create a new stream using the available
C<CSV> object, there is no harm in changing them.
If the L</new> constructor call fails, an exception of type C<CSV::Diac> is
thrown with the reason like the L</error_diag> method would return:
my $e;
{ $csv = Text::CSV.new(ecs_char => ":") or
CATCH { default { $e = $_; }}
}
$e and $e.message.say;
The message will be a string like
"INI - Unknown attribute 'ecs_char'"
=head2 print
X<print>
$status = $csv.print($io, $fld, ... );
$status = $csv.print($io, ($fld, ...));
$status = $csv.print($io, [$fld, ...]);
$status = $csv.print($io, @fld );
$csv.column_names(%fld.keys); # or use a subset
$status = $csv.print($io, %fld );
Similar to L</combine> + L</string> + L</print>, but much more efficient.
It takes an IO object and any number of arguments interpreted as fields.
The resulting string is immediately written to the C<$io> stream.
NYI: no fields in combination with L</bind_columns>, like
$csv.bind_columns(\($foo, $bar));
$status = $csv.print($fh);
A benchmark showed this order of preference, but the difference is within
noise range:
my @data = ^20;
$csv.print($io, @data ); # 2.6 sec
$csv.print($io, [ @data ]); # 2.7 sec
$csv.print($io, ^20 ); # 2.7 sec
$csv.print($io, \@data ); # 2.8 sec
=head2 say
X<say>
Is the same a sL</print> where L<C<eol>|/eol> defaults to C<$*OUT.nl>.
$status = $csv.say($io, $fld, ... );
$status = $csv.say($io, ($fld, ...));
$status = $csv.say($io, [$fld, ...]);
$status = $csv.say($io, @fld );
$csv.column_names(%fld.keys); # or use a subset
$status = $csv.say($io, %fld );
=head2 combine
X<combine>
$status = $csv.combine(@fields);
$status = $csv.combine($fld, ...);
$status = $csv.combine(\@fields);
This method constructs a C<CSV> row from C<@fields>, returning success or
failure. Failure can result from lack of arguments or an argument that
contains invalid data. Upon success, L</string> can be called to retrieve
the resultant C<CSV> string. Upon failure, the value returned by L</string>
is undefined and L</error_input> could be called to retrieve the invalid
argument. (WIP)
=head2 string
X<string>
$line = $csv.string;
This method returns the input to L</parse> or the resultant C<CSV> string
of L</combine>, whichever was called more recently. If L<C<eol>|/eol> is
defined, it is added to the string.
=head2 getline
X<getline>
@row = $csv.getline($io);
@row = $csv.getline($io, :meta);
@row = $csv.getline($str);
@row = $csv.getline($str, :meta);
This is the counterpart to L</print>, as L</parse> is the counterpart to
L</combine>: it parses a row from the C<$io> handle or C<$str> using the
L</getline> method associated with C<$io> (or the internal temporary IO
handle used to read from the string as if it were an IO handle) and parses
this row into an array. This array is returned by the function or C<Array>
for failure. When C<$io> does not support C<getline>, you are likely to hit
errors.
NYI: When fields are bound with L</bind_columns> the return value is a
reference to an empty list.
=head2 getline_all
=head2 getline-all
X<getline_all>
X<getline-all>
@rows = $csv.getline_all($io);
@rows = $csv.getline_all($io, :meta);
@rows = $csv.getline_all($io, $offset);
@rows = $csv.getline_all($io, $offset, :meta);
@rows = $csv.getline_all($io, $offset, $length);
@rows = $csv.getline_all($io, $offset, $length, :meta);
This will return a list of L<getline($io)|/getline> results.
If C<$offset> is negative, as with C<splice>, only the last C<abs($offset)>
records of C<$io> are taken into consideration.
Given a CSV file with 10 lines:
lines call
----- ---------------------------------------------------------
0..9 $csv.getline_all($io) # all
0..9 $csv.getline_all($io, 0) # all
8..9 $csv.getline_all($io, 8) # start at 8
- $csv.getline_all($io, 0, 0) # start at 0 first 0 rows
0..4 $csv.getline_all($io, 0, 5) # start at 0 first 5 rows
4..5 $csv.getline_all($io, 4, 2) # start at 4 first 2 rows
8..9 $csv.getline_all($io, -2) # last 2 rows
6..7 $csv.getline_all($io, -4, 2) # first 2 of last 4 rows
=head2 getline_hr
=head2 getline-hr
X<getline_hr>
X<getline-hr>
The L</getline_hr> and L</column_names> methods work together to allow you
to have rows returned as hashes instead of arrays. You must invoke
L</column_names> first to declare your column names.
$csv.column_names(< code name price description >);
%hr = $csv.getline_hr($str, :meta);
%hr = $csv.getline_hr($io);
say "Price for %hr<name> is %hr<price> \c[EURO SIGN]";
L</getline_hr> will fail if invoked before L</column_names>.
=head2 getline_hr_all
=head2 getline-hr-all
X<getline_hr_all>
X<getline-hr-all>
@rows = $csv.getline_hr_all($io);
@rows = $csv.getline_hr_all($io, :meta);
@rows = $csv.getline_hr_all($io, $offset);
@rows = $csv.getline_hr_all($io, $offset, :meta);
@rows = $csv.getline_hr_all($io, $offset, $length);
@rows = $csv.getline_hr_all($io, $offset, $length, :meta);
This will return a list of L<getline_hr($io)|/getline_hr> results.
=head2 parse
X<parse>
$status = $csv.parse($line);
This method decomposes a C<CSV> string into fields, returning success or
failure. Failure can result from a lack of argument or improper format in
the given C<CSV> string. Upon success, invoke L</fields> or L</strings> to
get the decomposed fields. Upon failure these methods shall not be trusted
to return reliable data.
NYI: You may use the L</types> method for setting column types. See
L</types>' description below.
=head2 fragment
X<fragment>
This function implements L<RFC7111|http://tools.ietf.org/html/rfc7111>
(URI Fragment Identifiers for the text/csv Media Type).
my @rows = $csv.fragment($io, $spec);
In specifications, C<*> is used to specify the I<last> item, a dash (C<->)
to indicate a range. All indices are C<1>-based: the first row or column
has index C<1>. Selections can be combined with the semi-colon (C<;>).
When using this method in combination with L</column_names>, the returned
array will be a list of hashes instead of an array of arrays. A disjointed
cell-based combined selection might return rows with different number of
columns making the use of hashes unpredictable.
$csv.column_names(< Name Age >);
my @rows = $csv.fragment($io, "row=3;8");
Note that for C<col="..">, the column names are the names for I<before> the
selection is taken to make it more consistent with reading possible headers
from the first line of the CSV datastream.
$csv,column_names(< foo bar >); # WRONG
$csv.fragment($io, "col=3");
would set the column names for the first two columns that are then skipped
in the fragment. To skip the unwanted columns, use placeholders.
$csv.column_names(< x x Name >);
$csv.fragment($io, "col=3");
WIP: If the L</after_parse> callback is active, it is also called on every
line parsed and skipped before the fragment.
=over 2
=item row
row=4
row=5-7
row=6-*
row=1-2;4;6-*
=item col
col=2
col=1-3
col=4-*
col=1-2;4;7-*
=item cell
In cell-based selection, the comma (C<,>) is used to pair row and column
cell=4,1
The range operator (C<->) using C<cell>s can be used to define top-left and
bottom-right C<cell> location
cell=3,1-4,6
The C<*> is only allowed in the second part of a pair
cell=3,2-*,2 # row 3 till end, only column 2
cell=3,2-3,* # column 2 till end, only row 3
cell=3,2-*,* # strip row 1 and 2, and column 1
Cells and cell ranges may be combined with C<;>, possibly resulting in rows
with different number of columns
cell=1,1-2,2;3,3-4,4;1,4;4,1
Disjointed selections will only return selected cells. The cells that are
not specified will not be included in the returned set, not even as
C<Str>. As an example given a C<CSV> like
11,12,13,...19
21,22,...28,29
: :
91,...97,98,99
with C<cell=1,1-2,2;3,3-4,4;1,4;4,1> will return:
11,12,14
21,22
33,34
41,43,44
Overlapping cell-specs will return those cells only once, So
C<cell=1,1-3,3;2,2-4,4;2,3;4,2> will return:
11,12,13
21,22,23,24
31,32,33,34
42,43,44
=back
L<RFC7111|http://tools.ietf.org/html/rfc7111> does B<not> allow different
types of specs to be combined (either C<row> I<or> C<col> I<or> C<cell>).
Passing an invalid fragment specification will croak and set error 2013.
Using L</colrange> and L</rowrange> instead of L</fragment> will allow you
to combine row- and column selecting as a grid.
=head2 colrange
X<colrange>
my Int @range = ^5, 5..9;
$csv.colrange(@range);
$csv.colrange("0-4;6-10");
my @range = $csv.colrange;
Set or inspect the column ranges. When passed as an array of C<Int>, the
indexes are 0-based. When passed as a string, the syntax of the range is as
defined by L<RFC7111|/fragment> and thus 1-based.
=head2 rowrange
X<rowrange>
$csv.rowrange("1;16-*");
my @r = $csv.rowrange;
Set or inspect the row ranges. Only supports L<RFC7111|/fragment> style.
Indexes are 1-based.
=head2 column_names
=head2 column-names
X<column_names>
X<column-names>
Set the "keys" that will be used in the L</getline_hr> calls. If no keys
(column names) are passed, it will return the current setting as a list.
$csv.column_names(< code description price >);
my @names = $csv.column_names;
L</column_names> accepts a list of strings (the column names) or a single
array with the names. You can pass the return value from L</getline> too:
$csv.column_names($csv.getline($io));
L</column_names> does B<no> checking on duplicates at all, which might lead
to unexpected results. As perl6 does not accept undefined keys in a hash,
passing just types will lead to fail later on.
$csv.column_names(Str, "", "name"); # Will FAIL becaus of Str
$csv.column_names(< code name count name >); # will drop the second column
%hr = $csv.getline_hr($io);
=head2 header
Parse the CSV header and set C<sep> and encoding.
my @hdr = $csv.header($fh).column-names;
$csv.header($fh, sep-set => [ ";", ",", "|", "\t" ]);
$csv.header($fh, munge-column-names => "fc");
The first argument should be a file handle.
Assuming that the file opened for parsing has a header, and the header does
not contain problematic characters like embedded newlines, read the first
line from the open handle then auto-detect whether the header separates the
column names with a character from the allowed separator list.
If any of the allowed separators matches, and none of the I<other> allowed
separators match, set L<C<sep>|/sep> to that separator for the current
CSV_XS instance and use it to parse the first line, map those to lowercase,
and use that to set the instance L</column_names>:
my $csv = Text::CSV.new;
my $fh = open "file.csv";
$csv.header($fh);
while (my $row = $csv.getline_hr($fh)) {
...
}
If the header is empty, contains more than one unique separator out of the
allowed set, contains empty fields, or contains identical fields (after
folding), it will croak with error 1010, 1011, 1012, or 1013 respectively.
If the header contains embedded newlines or is not valid CSV in any other
way, this method will throw an exception.
A successful call to C<header> will always set the L<C<sep>|/sep> of the
C<$csv> object. This behavior can not be disabled.
=head3 return value
On error this method will throw an exception.
On success, this method will return the instance.
=head3 Options
=over 2
=item sep-set
$csv.header($fh, sep_set => [ ";", ",", "|", "\t" ]);
The list of legal separators defaults to C<[ ";", "," ]> and can be changed
by this option.
Multi-byte sequences are allowed, both multi-character and Unicode. See
L<C<sep>|/sep>.
=item munge-column-names
This option offers the means to modify the column names into something that
is most useful to the application. The default is to map all column names
to fold case.
$csv.header($fh, munge-column-names => "lc");
The following values are available:
fc - fold case
lc - lower case
uc - upper case
none - do not change
&cb - supply a callback
$csv.header($fh, munge-column-names => { "column_".$col++ });
=item set-column-names
$csv.header($fh, :set-column-names);
The default is to set the instances column names using L</column_names> if
the method is successful, so subsequent calls to L</getline_hr> can return
a hash. Disable setting the header can be forced using a false value for
this option like C<:!set-column-names>.
=back
=head2 bind_columns
=head2 bind-columns
X<bind_columns>
X<bind-columns>
NYI!
Takes a list of scalar references to be used for output with L</print> or
to store in the fields fetched by L</getline>. When you do not pass enough
references to store the fetched fields in, L</getline> will fail with error
C<3006>. If you pass more than there are fields to return, the content of
the remaining references is left untouched.
$csv.bind_columns(\$code, \$name, \$price, \$description);
while ($csv.getline($io)) {
print "The price of a $name is \x[20ac] $price\n";
}
To reset or clear all column binding, call L</bind_columns> with the single
undefined argument like C<Array>. This will also clear column names.
$csv.bind_columns(Array);
If no arguments are passed at all, L</bind_columns> will return the list of
current bindings or C<Array> if no binds are active.
=head2 eof
X<eof>
$eof = $csv.eof;
If L</parse> or L</getline> was used with an IO stream, this method will
return C<True> if the last call hit end of file, otherwise it will return
C<False>. This is useful to see the difference between a failure and end
of file.
If the last L</parse> or L</getline> finished just before end of file, the
B<next> L</parse> or L</getline> will fail and set C<eof>.
That implies that if you are I<not> using L</auto-diag>, an idiom like
while (my @row = $csv.getline ($fh)) {
# ...
}
$csv.eof or $csv.error_diag;
will I<not> report the error. You would have to change that to
while (my @row = $csv.getline ($fh)) {
# ...
}
$csv.error_diag.error and $csv.error_diag;
=head2 types
X<types>
NYI!
$csv.types(@types);
my @types = $csv.types;
This method is used to force a type for all fields in a column. For
example, if you have an integer column, two columns with doubles and a
string column, then you might do a
$csv.types(Int, Num, Num, Str);
You can unset column types by doing a
$csv.types(Array);
=head2 row
X<row>
CSV::Row $row = $csv.row;
Returns the last row parsed. See L<C<CSV::Row>|/CSV::Row>
=head2 fields
X<fields>
CSV::Field @fields = $csv.fields;
This method returns the input to L</combine> or the resultant decomposed
fields of a successful L</parse> or L</getline>, whichever was called more
recently. The fields are still of type L<C<CSV::Field>|/CSV::Field> and
thus feature attributes.
=head2 strings
X<strings>
@fields = $csv.strings;
This method returns the input to L</combine> or the resultant decomposed
fields of a successful L</parse> or L</getline>, whichever was called more
recently. The fields are simplified to Str entries from
L<C<CSV::Field>|/CSV::Field>, so no attributes ate available.
NYI: If types are used, the fields should comply to the types.
=head2 meta_info
X<meta_info>
=head2 meta-info
X<meta-info>
$csv.meta_info(True);
my $km = $csv.keep_meta;
This methods sets or inquires the default setting for keeping meta-info on
fields. See L<C<CSV::Field>|/CSV::Field>.
=head2 is_quoted
X<is_quoted>
=head2 is-quoted
X<is-quoted>
my $quoted = $csv.is_quoted($column_idx);
Where C<$column_idx> is the (zero-based) index of the column in the last
result of L</parse> or L</getline>, even if C<meta> was false on that last
invocation.
This returns C<True> if the data in the indicated column was enclosed in
L<C<quote_char>|/quote_char> quotes. This might be important for fields
where content C<,20070108,> is to be treated as a numeric value, and where
C<,"20070108",> is explicitly marked as character string data.
Also see L<C<CSV::Field>|/CSV::Field>.
=head2 is_binary
X<is_binary>
=head2 is-binary
X<is-binary>
NYI/WIP: utf8-c8
my $binary = $csv.is_binary($column_idx);
Where C<$column_idx> is the (zero-based) index of the column in the last
result of L</parse> or L</getline>, even if C<meta> was false on that last
invocation.
Also see L<C<CSV::Field>|/CSV::Field>.
=head2 is_missing
X<is_missing>
=head2 is-missing
X<is-missing>
NYI
my $missing = $csv.is_missing($column_idx);
Where C<$column_idx> is the (zero-based) index of the column in the last
result of L</parse> or L</getline>, even if C<meta> was false on that last
invocation.
while (my @row = $csv.getline_hr($fh, :meta)) {
$csv.is_missing(0) and next; # This was an empty line
}
When using L</getline_hr>, it is impossible to tell if the parsed fields
are undefined because they where not filled in the C<CSV> stream or because
they were not read at all, as B<all> the fields defined by L</column_names>
are set in the hash-ref. If you still need to know if all fields in each
row are provided, you should enable L<C<keep_meta>|/keep_meta> so you can
check the flags.
=head2 status
X<status>
$status = $csv.status;
This method returns success (or failure) of the last invoked L</combine> or
L</parse> call.
=head2 error_input
X<error_input>
=head2 error-input
X<error-input>
$bad_argument = $csv.error_input;
This method returns the erroneous argument (if it exists) of L</combine>,
L</parse>, or L</getline>, whichever was called most recent. If the last
invocation was successful, C<error_input> will return C<Str>.
=head1 DIAGNOSTICS
Errors are transported internally using the L<C<CSV::Diag>|/CSV::Diag>
class. Text::CSV will return that object when it fails, so it can be
caught, but on non-fatal failures, like parse returning C<False>, one can
use the methods to inquire the internal status.
=head2 CSV::Diag
The method is created with the widest possible use in mind, serving both
the mindset of perl6 as well as the direct approach of the old module. It
is immutable: it is created with all available error parameters known at
the time of failure, and the cannot be changed afterwards.
my CSV::Diag $d .= new(
error => 0,
message => "",
pos => 0,
field => 0,
record => 0,
buffer => Str
);
If only C<error> is given, the message is set accordingly if it is a known
error value.
The object can be used in many contexts:
=over 2
=item void context
CSV::Diag.new(error => 2034, buffer => q{ "",}, pos => 1);
will print
EIF - Loose unescaped quote : error 2034 @ record 1, field 1, position 2
"?",
which is what happens when L<C<auto_diag>|/auto_diag> is C<True> and you
parse illegal CSV:
Text::CSV.new(:auto_diag).parse(q{ "",});'
EIF - Loose unescaped quote : error 2034 @ record 1, field 1, position 2
"?",
=item numeric context
Will return the error code
my Int $e;
{ fail CSV::Diag(error => 2034);
CATCH { default { $e = +$_; }}
}
# $e is now 2034
=item string context
Will return the error message
my Str $e;
{ fail CSV::Diag(error => 2034);
CATCH { default { $e = ~$_; }}
}
# $e is now "EIF - Loose unescaped quote"
=item list context
All of the 6 items can be retrieved as a list or positional:
{ fail CSV::Diag(error => 2034);
CATCH { default { $_[0].say; }}
}
The indices are chosen to be compatible with the old API
$e[0] = error number
$e[1] = error message
$e[2] = error position in buffer
$e[3] = field number
$e[4] = record number
$e[5] = errror buffer
=item hash context
All of the 6 items can be retrieved as a hash entry
{ fail CSV::Diag(error => 2034);
CATCH { default { $_<errno>.say; }}
}
The keys are chosen to be compatible with the old API.
$e<errno> = error number
$e<error> = error message
$e<pos> = error position in buffer
$e<field> = field number
$e<recno> = record number
$e<buffer> = errror buffer
=back
The CSV::Diag is also used by this Text::CSV method
=head2 error_diag
X<error_diag>
=head2 error-diag
X<error-diag>
$csv.error_diag;
$error_code = +$csv.error_diag;
$error_str = ~$csv.error_diag;
($cde, $str, $pos, $rec, $fld) = $csv.error_diag;
This function returns the diagnostics of the most recent error.
If called in void context, this will print the internal error code and the
associated error message along with the record number, the position of the
failure and the buffer of failure with an eject symbol at that position:
$csv.parse(q{ "",})
$csv.error_diag;
will print
EIF - Loose unescaped quote : error 2034 @ record 1, field 1, position 2
"?",
If called in list context, this will return the error code, the error
message, the location within the line that was being parsed, the record
number, and the buffer itself in that order. Their values are 1-based. The
position currently is index of the character at which the parsing failed in
the current record. The record number the index of the record parsed by the
csv instance. The field number is the index of the field the parser thinks
it is currently trying to parse.
If called as C<+$csv.error_diag> or C<$csv.error_diag.Num>, it will return
the error code. If called as C<~$csv.error_diag> or C<$csv.error_diag.Str>
it will return the error message.
=head2 record_number
X<record_number>
$recno = $csv.record_number;
Returns the records parsed by this csv instance. This value should be more
accurate than C<$.> when embedded newlines come in play. Records written by
this instance are not counted.
=head2 set_diag
X<set_diag>
$csv.set_diag(0);
$csv.set_diag(2025, pos => 12, fieldno => 4, recno => 99);
Use to (re)set the diagnostics if you are dealing with errors.
=head1 CSV::Field
The fields are internally represented as CSV::Field objects. Any methods
that directly or indirectly supports the C<meta> attribute controls
weather the returned fields will be of this CSV::Field type or that the
fields are simplified to a simple basic type.
If the fields are represented/returned as CSV::Field, it supports these
methods:
=head2 new
my CSV::Field $f .= new;
my $f = CSV::Field.new("foo");
my $f = CSV::Field.new(1);
Instantiate a new field. Optionally takes a C<Cool>.
=head2 Bool
?$f and say "The field is true";
$f.Bool and say "This field is true too";
Return the boolean value of the field. As CSV is text-only be design, this
will also return C<False> for C<"0">, where perl6 sees C<"0"> as C<True>.
=head2 text
=head2 Str
$str = ~$f;
$str = $f.Str;
$str = $f.text;
Return the string representation of the field.
=head2 Buf
$buf = $f.Buf;
Return the field data as "utf8-c8" encoded Buf.
=head2 Numeric
$i = +$f;
$i = $f.Int;
$i = $f.Num;
Return the numeric representation of the field.
=head2 gist
$f.gist.say;
Will show a complete compressed representation of the field with properties.
C<Q> is quoted, C<q> is unquoted. Likewise for C<B/b> for binary, C<8/7>
for Unicode-ness and C<M/m> for missing.
A field that was parsed as C<,cat,> would return
qb7m:"cat"
A field parsed as C<,"Ħēłĺº",> would return
QB8m:"Ħēłĺº"
=head2 add
$f.add($str);
Accepts a Str to be added to this field.
=head2 set_quoted
$f.set_quoted;
Set the fact the the field was/is quoted.
=head2 is_quoted
$f.is_quoted and say "The field was quoted in CSV";
Is C<True> when the parsed field was quoted in the source.
=head2 undefined
$f.undefined and say "The field is undefined";
Returns C<True> when the field is undefined. As CSV is all about strings,
the various options that allow interpreting empty fields as undefined make
this a required method.
=head2 is_binary
WIP: utf8-c8
$f.is_binary and say "Do we need a Buf instead?";
Returns true if the field has data beyond ASCII.
=head2 is_utf8
$f.is_utf8 or say "The field is empty or pure ACII";
Returns C<True> if the field is beyond ASCII, but still valid UTF-8.
=head2 is_missing
WIP
$f.is_missing and fail;
Returns C<True> if the field is missing.
=head1 CSV::Row
X<CSV::Row>
This class is a wrapper over the current row of fields just to add
convenience methods.
This is the only argument to callbacks.
The fields in C<CSV::Row> are always of class L<C<CSV::Field>|/CSV::Field>
and thus contain all meta-information, even if the C<Text::CSV> attribute
L<C<meta_info>|/meta_info> is C<False>.
=head2 methods
=over 2
=item new
my CSV::Row $row .= new;
my CSV::Row $row .= new(csv => $csv, fields => @f.map({ CSV::Field.new(*) });
=item csv
The current Text::CSV object related to this row.
=item elems
Return the number of fields in the row.
=item fields
The fields (CSV::Field items) this row consist of.
=item push
$row.push(CSV::Field.new(1));
$row.push(1);
$row.push("foo");
$row.push($another-row);
Pushing simple things onto the row will extend the row by converting these
to CSV::Field objects.
Pushing a CSV::Row onto the row will extend that row with the fields of the
row being pushed.
=item pop
my CSV::Field $f = $row.pop;
=item Str
my $str = $row.Str;
$io.say(~$row);
The stringification of the CSV::Row object is like invoking the
L<C<string>|/string> method. This only works if there is a Text::CSV
object known to the CSV::Row instance.
=item hash
my %h = $row.hash;
Returns the hash with C<.csv>'s L<C<column_names>|/column_names> as keys
and the C<.text> of each matching C<fields> entry as values.
=item strings
my @l = $row.strings;
Returns the C<.text> part of each entry in C<.fields>.
=back
The row allow direct indexing and iteration as well as hash addressing when
L</column_names> are set.
my $field = $row[1];
my $field = $row<foo>;
The last parsed row of a L<Text::CSV>|/Text::CSV> can be acquired using
my CSV::Row $row = $csv.row;
################################################################################
=head1 FUNCTIONS
=head2 csv
X<csv>
This is an high-level function that aims at simple (user) interfaces. This
can be used to read/parse a C<CSV> file or stream (the default behavior) or
to produce a file or write to a stream (define the C<out> attribute). It
returns an array- or hash-reference on parsing or the
numeric value of L</error_diag> on writing. When this function fails you
can get to the error using the class call to L</error_diag>
my $aoa = csv(in => "test.csv") or
die Text::CSV.error_diag;
This function takes the arguments as key-value pairs. This can be passed as
a list or as an anonymous hash:
my $aoa = csv(in => "test.csv", sep => ";");
my $aoh = csv(in => $fh, :headers);
The arguments passed consist of two parts: the arguments to L</csv> itself
and the optional attributes to the C<CSV> object used inside the function
as enumerated and explained in L</new>.
If not overridden, the default option used for CSV is
auto_diag => 1
The option that is always set and cannot be altered is
binary => 1
=head3 in
X<in>
Used to specify the source. C<in> can be a file name (e.g. C<"file.csv">),
which will be opened for reading and closed when finished, a file handle
(e.g. C<$fh> or C<FH>), a reference to a glob (e.g. C<\*ARGV>), the glob
itself (e.g. C<*STDIN>), or a reference to a scalar (e.g. C<\q{1,2,"csv"}>).
When used with L</out>, C<in> should be a reference to a CSV structure (AoA
or AoH) or a Callable (Sub, Routine, Code, or Block) that returns an
array-reference or a hash-reference. The code-ref will be invoked with no
arguments.
my $aoa = csv(in => "file.csv");
open my $fh, "<", "file.csv";
my $aoa = csv(in => $fh);
my $csv = [[qw( Foo Bar )], [ 1, 2 ], [ 2, 3 ]];
my $err = csv(in => $csv, out => "file.csv");
The C<in> attribute supports a wide range of types, all of which can be
combined with the use of C<fragment>:
=over 2
=item Str
csv(in => "file.csv")
A plain string is interpreted as a file name to be opened for parsing.
=item IO
my $io = open "file.csv", :r;
csv(in => $io);
Parse from the already opened data stream.
=item Capture
csv(in => \("str,ing"));
Parse from a single string.
=item Array of Strings
csv(in => ["a,b\n1,2\n3,4\n"]);
csv(in => ["a,b", "1,2", "3,4"]);
Parse from the String(s)
=item Array of Data
csv(in => [[<a b>], [1, 2], [3, 4]]);
csv(in => [{:a(1), :b(2)}, {:a(3), :b(4)}]);
Use the data as provided
=item Sub, Routine
sub provider {
@data.elems == 0 and return False;
return @data.pop;
}
csv(in => &provider);
While the providing Routine returns a data row, use that as is. Stop when
the provider returns C<False>.
=item Callable, Block
csv(in => { $sth.fetch });
While the providing Callable returns a data row, use that as is. Stop when
the provider returns C<False>.
=item Supply
my $supply = Supply.from-list(@data);
csv(in => $supply);
Fetch data rows from the supply.
=item Channel
my $ch = Channel.new;
start {
$ch.send($_) for @data;
$ch.close;
}
csv(in => $ch);
Fetch data from the Channel.
=item Iterator
csv(in => @data.iterator);
Fetch data rows from the iterator.
=item Any
csv(in => Any);
is a shortcut/fallback for
csv(in => $*IN);
=back
=head3 out
X<out>
In output mode, the default CSV options when producing CSV are
eol => "\r\n"
The L</fragment> attribute is ignored in output mode.
C<out> can be a file name (e.g. C<"file.csv">), which will be opened for
writing and closed when finished, a file handle (e.g. C<$fh> or C<FH>), a
Channel, or a Supply.
csv(in => sub { $sth.fetch }, out => "dump.csv");
csv(in => { $sth.fetchrow_hashref }, out => "dump.csv",
headers => $sth.{NAME_lc});
When a code-ref is used for C<in>, the output is generated per invocation,
so no buffering is involved. This implies that there is no size restriction
on the number of records. The C<csv> function ends when the coderef returns
C<False>.
=over 2
=item Str:U
my $str = csv(in => $in, out => Str);
Returns a single String of CSV data.
=item Str:D
my $str = csv(in => $in, out => "out.csv");
Writes the data as CSV to the named file.
=item Array:U
my $aoa = csv(in => $in, out => Array);
Returns an Array of Arrays.
=item Hash:U
my $aoh = csv(in => $in, out => Hash);
Returns an Array of Hashes
=item IO:D
my $io = open "file.csv", :w;
csv(in => $in, out => $io);
Writes the data as CSV to the IO handle.
=item Callable, Block, Sub, Routine
my @d;
csv(in => $in, out => { @d.push: $_ });
Passes the data rows to the Callable
=item Channel:U
my $ch = csv(in => $in, out => Channel, :!meta);
react {
whenever $ch -> \row {
@d.push: row;
LAST { done; }
}
}
Writes the data rows into a new Channel, which is returned.
=item Channel:D
my $ch = Channel.new;
my $pr = start {
react {
whenever $ch -> \row {
@d.push: row;
LAST { done; }
}
}
}
csv(in => $in, out => $ch);
await $pr;
Writes the data rows into the existing Channel.
=item Supplier:D
my $sup = Supplier.new;
$sup.Supply.tap (-> \row { @d.push: row; });
csv(in => $in, out => $sup, :!meta);
Writes the data rows into the Supplier.
=item Supply:U
my $sup = csv(in => $in, out => Supply, :!meta);
$ch.tap (-> \row { @d.push: row; });
Writes the data rows into a new Supply.
=back
=head3 encoding
X<encoding>
If passed, it should be an encoding accepted by the C<:encoding()> option
to C<open>. There is no default value.
If C<encoding> is set to the literal value C<"auto">, the method L</header>
will be invoked on the opened stream to check if there is a BOM and set the
encoding accordingly. This is equal to passing True in the option
L<C<detect-bom>|/detect-bom>.
=head3 detect-bom
X<detect-bom>
X<detect_bom>
NYI for various reasons. Also see L</is-binary>
If C<detect-bom> is given, the method L</header> will be invoked on the
opened stream to check if there is a BOM and set the encoding accordingly.
C<detect_bom> can be abbreviated to C<bom>.
This is the same as setting L<C<encoding>|/encoding> to C<"auto">.
Note that as L</header> is invoked, its default is to also set the headers.
=head3 headers
X<headers>
If this attribute is not given, the default behavior is to produce an array
of arrays.
If C<headers> is supplied, it should be an Array of column names, a Bool, a
Hash, a Callable, or a literal flag: C<auto>, C<lc>, C<uc>, or C<skip>.
=over 2
=item skip
When C<skip> is used, the header will not be included in the output.
my $aoa = csv(in => $fh, headers => "skip");
=item auto
If C<auto> is used, the first line of the C<CSV> source will be read as the
list of field headers and used to produce an array of hashes.
my $aoh = csv(in => $fh, headers => "auto");
=item lc
If C<lc> is used, the first line of the C<CSV> source will be read as the
list of field headers mapped to lower case and used to produce an array of
hashes. This is a variation of C<auto>.
my $aoh = csv(in => $fh, headers => "lc");
=item uc
If C<uc> is used, the first line of the C<CSV> source will be read as the
list of field headers mapped to upper case and used to produce an array of
hashes. This is a variation of C<auto>.
my $aoh = csv(in => $fh, headers => "uc");
=item Bool
If C<True> is passed, the method L</header> will be invoked with the
default options on the opened stream to check if there is a BOM and set the
encoding accordingly, detect and set L<C<sep>|/sep>, L<C<eol>|/eol> and
column names.
=item Callable
If a Callable is used, the first line of the C<CSV> source will be read as
the list of mangled field headers in which each field is passed as the only
argument to the coderef. This list is used to produce an array of hashes.
my $i = 0;
my $aoh = csv(in => $fh, headers => { $^h.lc ~ $i++ });
this example is a variation of using C<lc> where all headers are forced to
be unique by adding an index.
=item ARRAY
If C<headers> is an Array, the entries in the list will be used as field
names. The first line is considered data instead of headers.
my $aoh = csv(in => $fh, headers => [< Foo Bar >]);
=item HASH
If C<headers> is a Hash, this implies C<auto>, but header fields for that
exist as key in the Hash will be replaced by the value for that key. Given
a CSV file like
post-kode,city,name,id number,fubble
1234AA,Duckstad,Donald,13,"X313DF"
using
csv (headers => %{ "post-kode" => "pc", "id number" => "ID" }, ...
will return an entry like
{ pc => "1234AA",
city => "Duckstad",
name => "Donald",
ID => "13",
fubble => "X313DF",
}
=back
See also L<C<munge-column-names>|/munge-column-names> and
L<C<set-column-names>|/set-column-names>.
=head3 munge-column-names
X<munge-column-names>
X<munge_column_names>
If C<munge-column-names> is set, the method L</header> is invoked on the
opened stream with all matching arguments to detect and set the headers.
C<munge-column-names> can be abbreviated to C<munge>.
=head3 key
X<key>
If passed, will default L<C<headers>|/headers> to C<"auto"> and return a
hashref instead of an array of hashes.
my $ref = csv(in => "test.csv", key => "code");
with test.csv like
code,product,price,color
1,pc,850,gray
2,keyboard,12,white
3,mouse,5,black
will return
{ 1 => {
code => 1,
color => 'gray',
price => 850,
product => 'pc'
},
2 => {
code => 2,
color => 'white',
price => 12,
product => 'keyboard'
},
3 => {
code => 3,
color => 'black',
price => 5,
product => 'mouse'
}
}
=head3 fragment
X<fragment>
Only output the fragment as defined in the L</fragment> method. This option
is ignored when I<generating> C<CSV>. See L</out>.
Combining all of them could give something like
use Text::CSV qw( csv );
my @aoh = csv(
in => "test.txt",
encoding => "utf-8",
headers => "auto",
sep_char => "|",
fragment => "row=3;6-9;15-*",
);
say @aoh[15]{Foo};
=head3 sep-set
X<sep-set>
X<sep_set>
X<seps>
If C<sep-set> is set, the method L</header> is invoked on the opened stream
to detect and set L<C<sep>|/sep> with the given set.
C<sep-set> can be abbreviated to C<seps>.
Note that as L</header> is invoked, its default is to also set the headers.
=head3 set_column_names
X<set_column_names>
If C<set_column_names> is passed, the method L</header> is invoked on the
opened stream with all arguments meant for L</header>.
=head2 Callbacks
Callbacks enable actions triggered from the I<inside> of Text::CSV.
While most of what this enables can easily be done in an unrolled loop as
described in the L</SYNOPSIS> callbacks, can be used to meet special
demands or enhance the L</csv> function.
All callbacks except C<error> are called with just one argument: the
current L<CSV::Row>.
=over 2
=item error
X<error>
$csv.callbacks(error => { $csv.SetDiag(0) });
the C<error> callback is invoked when an error occurs, but I<only> when
L</auto_diag> is set to a true value. This callback is invoked with the
values returned by L</error_diag>:
my ($c, $s);
sub ignore3006 (Int $err, Str $msg, Int $pos, Int $recno, Int $fldno) {
if ($err == 3006) {
# ignore this error
($c, $s) = (Str, Str);
Text::CSV.SetDiag(0);
}
# Any other error
return;
} # ignore3006
$csv.callbacks(error => \&ignore3006);
$csv.bind_columns(\$c, \$s);
while ($csv.getline($fh)) {
# Error 3006 will not stop the loop
}
=item after_parse
X<after_parse>
X<after-parse>
sub add-new (CSV::Row $r) { $r.fields.push: "NEW"; }
$csv.callbacks(after_parse => &add-new);
while (my @row = $csv.getline($fh)) {
@row[-1] eq "NEW";
}
This callback is invoked after parsing with L</getline> only if no error
occurred.
The return code of the callback is ignored.
sub add_from_db (CSV::Row $r) {
$sth.execute($r[4]);
push $r.fields: $sth.fetchrow_array;
} # add_from_db
my $aoa = csv(in => "file.csv", callbacks => {
after_parse => &add_from_db });
my $aoa = csv(in => "file.csv", after_parse => {
$sth.execute($^row[4]); $^row.fields.push: $sth.fetchrow_array; });
=item before_print
X<before_print>
X<before-print>
my $idx = 1;
$csv.callbacks(before_print => { $^row[0] = $idx++ });
$csv.print(*STDOUT, [ 0, $_ ]) for @members;
This callback is invoked before printing with L</print> only if no error
occurred.
The return code of the callback is ignored.
sub max_4_fields (CSV::Row $r) {
$r.elems > 4 and $r.splice (4);
} # max_4_fields
csv(in => csv(in => "file.csv"), out => *STDOUT,
callbacks => { before print => \&max_4_fields });
csv(in => csv(in => "file.csv"), out => *STDOUT,
before print => { $^row.elems > 4 and $^row.splice(4) });
This callback is not active for L</combine>.
=back
=head3 Callbacks for csv
The L</csv> allows for some callbacks that do not integrate in internals
but only feature the L</csv> function. XXX: Is this still true?
csv(in => "file.csv",
callbacks => {
after_parse => { say "AFTER PARSE"; }, # first
after_in => { say "AFTER IN"; }, # second
on_in => { say "ON IN"; }, # third
},
);
csv(in => $aoh,
out => "file.csv",
callbacks => {
on_in => { say "ON IN"; }, # first
before_out => { say "BEFORE OUT"; }, # second
before_print => { say "BEFORE PRINT"; }, # third
},
);
=over 2
=item filter
X<filter>
This callback can be used to filter records. It is called just after a new
record has been scanned. The callback will be invoked with the current
L</CSV::Row> and should return C<True> for records to accept and C<False>
for records to reject.
csv (in => "file.csv", filter => {
$^row[2] ~~ /a/ && # third field should contain an "a"
$^row[4].chars > 4 # length of the 5th field minimal 5
});
csv (in => "file.csv", filter => "not_blank");
csv (in => "file.csv", filter => "not_empty");
csv (in => "file.csv", filter => "filled");
If the filter is used to I<alter> the content of a field, make sure that
the sub, block or callable returns true in order not to have that record
skipped:
filter => { $^row[1].text .= uc }
will upper-case the second field, and then skip it if the resulting content
evaluates to false. To always accept, end with truth:
filter => { $^row[1].text .= uc; 1 }}
B<Predefined filters>
Given a file like (line numbers prefixed for doc purpose only):
1:1,2,3
2:
3:,
4:""
5:,,
6:, ,
7:"",
8:" "
9:4,5,6
=over 2
=item not_blank
=item not-blank
Filter out the blank lines
This filter is a shortcut for
filter => { $^row.elems > 1 or
$^row[0].defined && $^row[0] ne "" or
$^row[0].is-quoted }
With the given example, line 2 will be skipped.
=item not_empty
=item not-empty
Filter out lines where all the fields are empty.
This filter is a shortcut for
filter => { $^row.first: { .defined && $_ ne "" }}
A space is not regarded being empty, so given the example data, lines 2, 3,
4, 5, and 7 are skipped.
=item filled
Filter out lines that have no visible data
This filter is a shortcut for
filter => { $^row.first: { .defined && $_ ~~ /\S/ }}
This filter rejects all lines that I<not> have at least one field that does
not evaluate to the empty string.
With the given example data, this filter would skip lines 2 through 8.
=back
=item after_in
X<after_in>
X<after-in>
This callback is invoked for each record after all records have been parsed
but before returning the reference to the caller.
This callback can also be passed as an attribute to L</csv> without the
C<callbacks> wrapper.
=item before_out
X<before_out>
X<before-out>
This callback is invoked for each record before the record is printed.
This callback can also be passed as an attribute to L</csv> without the
C<callbacks> wrapper.
=item on_in
X<on_in>
X<on-in>
This callback acts exactly as the L</after_in> or the L</before_out> hooks.
This callback can also be passed as an attribute to L</csv> without the
C<callbacks> wrapper.
=back
=head1 EXAMPLES
=head2 Reading a CSV file line by line:
my $csv = Text::CSV.new(:auto_diag);
my $fh = open "file.csv", :r, chomp => False;
while (my @row = $csv.getline($fh)) {
# do something with @$row
}
$fh.close;
=head3 Reading only a single column
my $csv = Text::CSV.new(:auto_diag);
my $fh = open "file.csv", :r, :!chomp;
# get only the 4th column
my @column = $csv.getline_all($fh).map(*[3]);
$fh.close;
with L</csv>, you could do
my @column = csv(in => "file.csv", fragment => "col=4").map(*[0]);
or
my @column = csv(in => "file.csv", fragment => "col=4").map(*.flat);
=head2 Parsing CSV strings:
my $csv = Text::CSV.new(:keep_meta);
my $sample_input_string =
q{"I said, ""Hi!""",Yes,"",2.34,,"1.09","\x[20ac]",};
if ($csv.parse($sample_input_string)) {
my @field = $csv.fields;
for ^@field.elems -> $col {
my $quo = $csv.is_quoted($col) ? $csv.{quote_char} : "";
printf "%2d: %s%s%s\n", $col, $quo, $field[$col], $quo;
}
}
else {
print STDERR "parse failed on argument: ",
$csv.error_input, "\n";
$csv.error_diag;
}
=head2 Printing CSV data
=head3 The fast way: using L</print>
An example for creating C<CSV> files using the L</print> method:
my $csv = Text::CSV.new(eol => $*OUT.nl);
open my $fh, ">", "foo.csv" or die "foo.csv: $!";
for 1..10 -> $x {
$csv.print($fh, [ $x, ~$x ]) or $csv.error_diag;
}
close $fh or die "$tbl.csv: $!";
=head3 The slow way: using L</combine> and L</string>
or using the slower L</combine> and L</string> methods:
my $csv = Text::CSV.new;
open my $csv_fh, ">", "hello.csv" or die "hello.csv: $!";
my @sample_input_fields = (
'You said, "Hello!"', 5.67,
'"Surely"', '', '3.14159');
if ($csv.combine(@sample_input_fields)) {
print $csv_fh $csv.string, "\n";
}
else {
print "combine failed on argument: ",
$csv.error_input, "\n";
}
close $csv_fh or die "hello.csv: $!";
=head2 Rewriting CSV
Rewrite C<CSV> files with C<;> as separator character to well-formed C<CSV>:
use Text::CSV qw( csv );
csv(in => csv(in => "bad.csv", sep_char => ";"), out => *STDOUT);
=head2 Dumping database tables to CSV
Dumping a database table can be simple as this (TIMTOWTDI):
my $dbh = DBI.connect(...);
my $sql = "select * from foo";
# using your own loop
open my $fh, ">", "foo.csv" or die "foo.csv: $!\n";
my $csv = Text::CSV.new(eol => "\r\n");
my $sth = $dbh.prepare($sql); $sth.execute;
$csv.print($fh, $sth.{NAME_lc});
while (my $row = $sth.fetch) {
$csv.print($fh, $row);
}
# using the csv function, all in memory
csv(out => "foo.csv", in => $dbh.selectall_arrayref($sql));
# using the csv function, streaming with callbacks
my $sth = $dbh.prepare($sql); $sth.execute;
csv(out => "foo.csv", in => { $sth.fetch });
csv(out => "foo.csv", in => { $sth.fetchrow_hashref });
Note that this does not discriminate between "empty" values and NULL-values
from the database, as both will be the same empty field in CSV. To enable
distinction between the two, use L<C<quote_empty>|/quote_empty>.
csv(out => "foo.csv", in => { $sth.fetch }, :quote_empty);
If the database import utility supports special sequences to insert C<NULL>
values into the database, like MySQL/MariaDB supports C<\N>, use a filter
or a map
csv(out => "foo.csv", in => { $sth.fetch },
on_in => { $_ //= "\\N" for @$_[1] }); # WIP
while (my @row = $sth.fetch) {
$csv.print($fh, @row.map({ * // "\\N" }));
}
these special sequences are not recognized by Text::CSV_XS on parsing the
CSV generated like this, but map and filter are your friends again
while (my @row = $csv.getline($io)) {
$sth.execute(@row.map({ $_ eq "\\N" ?? Nil !! $_ }));
}
csv(in => "foo.csv", filter => { 1 => {
$sth.execute(@{$_[1]}.map({ $_ eq "\\N" ?? Nil !! $_ }); False; }});
=head2 The examples folder
For more extended examples, see the F<examples/> C<1>) sub-directory in the
original distribution or the git repository C<2>).
1. http://repo.or.cz/w/Text-CSV.git?a=tree;f=examples
2. http://repo.or.cz/w/Text-CSV.git
The following files can be found there:
=over 2
=item csv-check
X<csv-check>
This is a command-line tool to check the C<CSV> file and report on its
content.
TODO
=item csv2xls
X<csv2xls>
A script to convert C<CSV> to Microsoft Excel.
TODO
=item csvdiff
X<csvdiff>
A script that provides colorized diff on sorted CSV files, assuming first
line is header and first field is the key. Output options include colorized
ANSI escape codes or HTML.
TODO
=back
################################################################################
=head1 CAVEATS
=head2 Microsoft Excel
The import/export from Microsoft Excel is a I<risky task>, according to the
documentation in C<Text::CSV::Separator>. Microsoft uses the system's list
separator defined in the regional settings, which happens to be a semicolon
for Dutch, German and Spanish (and probably some others as well). For the
English locale, the default is a comma. In Windows however, the user is
free to choose a predefined locale, and then change I<every> individual
setting in it, so checking the locale is no solution.
A lone first line with just
sep=;
will be recognized and honored: it will set L<C<sep>|/sep> to C<;> and skip
that line.
=head1 TODO / WIP / NYI
=over 2
=item Real binary data
The solution would be a working C<utf8-c8> encoding.
=item BOM detection
There is no working solution yet for detection of BOM on the L</header>
method. Besides that, not all encodings are supported in perl6.
=item on-in and before-print callbacks
The L</on-in> callback currently is an alias for L</after-parse> if the
latter is not specified.
=item Examples
Convert the perl5 example/tool files to perl6 versions
=item Metadata and CSV for the web
L<Metadata Vocabulary for Tabular Data|http://w3c.github.io/csvw/metadata/>
(a W3C editor's draft) could be an example for supporting more metadata.
W3C's work L<CSV on the Web: Use Cases and
Requirements|http://w3c.github.io/csvw/use-cases-and-requirements/index.html>
is almost finished and worth looking at.
=item Cookbook
Write a document that has recipes for most known non-standard (and maybe
some standard) C<CSV> formats, including formats that use C<TAB>, C<;>,
C<|>, or other non-comma separators.
Examples could be taken from W3C's L<CSV on the Web: Use Cases and
Requirements|http://w3c.github.io/csvw/use-cases-and-requirements/index.html>
=back
=head1 DIAGNOSTICS
Still under construction ...
This section describes the error codes that are used in perl5's module
L<Text::CSV_XS>, and several of these errors are either not applicable in
perl6 or changed slightly. Once all of the API is finished, this section
will be cleaned up. The intention of the error coded however remains.
If an error occurs, L<C<< $csv.error_diag >>|/error_diag> can be used to
get information on the cause of the failure. Note that for speed reasons
the internal value is never cleared on success, so using the value returned
by L</error_diag> in normal cases - when no error occurred - may cause
unexpected results.
If the constructor failed, the cause will be thrown as an Exception that
represents L</error_diag>.
The C<< $csv.error_diag >> method is automatically invoked upon error when
the contractor was called with L<C<auto_diag>|/auto_diag> set to C<True>.
Errors can be (individually) caught using the L</error> callback.
The errors as described below are available. I have tried to make the error
itself explanatory enough, but more descriptions will be added. For most of
these errors, the first three capitals describe the error category:
=over 2
=item *
INI
Initialization error or option conflict.
=item *
ECR
Carriage-Return related parse error.
=item *
EOF
End-Of-File related parse error.
=item *
EIQ
Parse error inside quotation.
=item *
EIF
Parse error inside field.
=item *
ECB
Combine error.
=item *
EHR
Hash parse related error.
=item *
EHK
Errors related to hooks/callbacks.
=item *
CSV
Errors related to the csv function.
=back
And below should be the complete list of error codes that can be returned:
=over 2
=item *
1001 "INI - separator is equal to quote- or escape sequence"
X<1001>
The L<separation sequence|/sep_char> cannot be equal to L<the quotation
sequence|/quote_char> or to L<the escape sequence|/escape_char>, as this
would invalidate all parsing rules.
=item *
1002 "INI - allow_whitespace with escape_char or quote_char SP or TAB"
X<1002>
Using the L<C<allow_whitespace>|/allow_whitespace> attribute when either
L<C<quote_char>|/quote_char> or L<C<escape_char>|/escape_char> is equal to
C<SPACE> or C<TAB> is too ambiguous to allow.
=item *
1003 "INI - \r or \n in main attr not allowed"
X<1003>
Using default L<C<eol>|/eol> sequences in either
L<separation sequence|/sep_char>, L<quotation sequence|/quote_char>, or
L<escape sequence|/escape_char> is not allowed.
=item *
1004 "INI - callbacks should be undefined or a hashref"
X<1004>
The L<C<callbacks>|/Callbacks> attribute only allows one to be undefined or
a hash reference.
=item *
1010 "INI - the header is empty"
X<1010>
The header line parsed in the L</header> is empty.
=item *
1011 "INI - the header contains more than one valid separator"
X<1011>
The header line parsed in the L</header> contains more than one (unique)
separator character out of the allowed set of separators.
=item *
1012 "INI - the header contains an empty field"
X<1012>
The header line parsed in the L</header> is contains an empty field.
=item *
1013 "INI - the header contains nun-unique fields"
X<1013>
The header line parsed in the L</header> contains at least two identical
fields.
=item *
2010 "ECR - QUO char inside quotes followed by CR not part of EOL"
X<2010>
When L<C<eol>|/eol> has been set to anything but the default, like
C<"\r\t\n">, and the C<"\r"> is following the B<second> (closing)
L<C<quote_char>|/quote_char>, where the characters following the C<"\r"> do
not make up the L<C<eol>|/eol> sequence, this is an error.
=item *
2011 "ECR - Characters after end of quoted field"
X<2011>
Sequences like C<1,foo,"bar"baz,22,1> are not allowed. C<"bar"> is a quoted
field and after the closing double-quote, there should be either a new-line
sequence or a separation sequence.
=item *
2012 "EOF - End of data in parsing input stream"
X<2012>
Self-explaining. End-of-file while inside parsing a stream. Can happen only
when reading from streams with L</getline>, as using L</parse> is done on
strings that are not required to have a trailing L<C<eol>|/eol>.
=item *
2013 "INI - Specification error for fragments RFC7111"
X<2013>
Invalid specification for URI L</fragment> specification.
=item *
2021 "EIQ - NL char inside quotes, binary off"
X<2021>
Sequences like C<1,"foo\nbar",22,1> are allowed only when the binary option
has been selected with the constructor.
=item *
2022 "EIQ - CR char inside quotes, binary off"
X<2022>
Sequences like C<1,"foo\rbar",22,1> are allowed only when the binary option
has been selected with the constructor.
=item *
2023 "EIQ - QUO sequence not allowed"
X<2023>
Sequences like C<"foo "bar" baz",qu> and C<2023,",2008-04-05,"Foo, Bar",\n>
will cause this error.
=item *
2024 "EIQ - EOF cannot be escaped, not even inside quotes"
X<2024>
The escape sequence is not allowed as last item in an input stream.
=item *
2025 "EIQ - Loose unescaped escape"
X<2025>
An escape sequence should escape only characters that need escaping.
Allowing the escape for other characters is possible with the attribute
L</allow_loose_escapes>.
=item *
2026 "EIQ - Binary character inside quoted field, binary off"
X<2026>
Binary characters are not allowed by default. Exceptions are fields that
contain valid UTF-8, that will automatically be upgraded if the content is
valid UTF-8. Set L<C<binary>|/binary> to C<1> to accept binary data.
=item *
2027 "EIQ - Quoted field not terminated"
X<2027>
When parsing a field that started with a quotation sequence, the field is
expected to be closed with a quotation sequence. When the parsed line is
exhausted before the quote is found, that field is not terminated.
=item *
2030 "EIF - NL char inside unquoted verbatim, binary off"
X<2030>
=item *
2031 "EIF - CR char is first char of field, not part of EOL"
X<2031>
=item *
2032 "EIF - CR char inside unquoted, not part of EOL"
X<2032>
=item *
2034 "EIF - Loose unescaped quote"
X<2034>
=item *
2035 "EIF - Escaped EOF in unquoted field"
X<2035>
=item *
2036 "EIF - ESC error"
X<2036>
=item *
2037 "EIF - Binary character in unquoted field, binary off"
X<2037>
=item *
2110 "ECB - Binary character in Combine, binary off"
X<2110>
=item *
2200 "EIO - print to IO failed. See errno"
X<2200>
=item *
3001 "EHR - Unsupported syntax for column_names"
X<3001>
=item *
3002 "EHR - getline_hr called before column_names"
X<3002>
=item *
3003 "EHR - bind_columns and column_names fields count mismatch"
X<3003>
=item *
3004 "EHR - bind_columns only accepts refs to scalars"
X<3004>
=item *
3006 "EHR - bind_columns did not pass enough refs for parsed fields"
X<3006>
=item *
3007 "EHR - bind_columns needs refs to writable scalars"
X<3007>
=item *
3008 "EHR - unexpected error in bound fields"
X<3008>
=item *
3009 "EHR - print_hr called before column_names"
X<3009>
=item *
3010 "EHR - print_hr called with invalid arguments"
X<3010>
=item *
3100 "EHK - Unsupported callback"
X<3100>
=item *
5000 "CSV - Unsupported type for in"
X<5000>
=item *
5001 "CSV - Unsupported type for out"
X<5001>
=back
=head1 SEE ALSO
Modules in perl5:
L<IO::File>, L<IO::Handle>, L<IO::Wrap>, L<Text::CSV_XS>, L<Text::CSV>,
L<Text::CSV_PP>, L<Text::CSV::Encoded>, L<Text::CSV::Separator>,
L<Text::CSV::Slurp>, L<Spreadsheet::CSV> and L<Spreadsheet::Read>;
=head1 AUTHOR
H.Merijn Brand F<E<lt>h.m.brand@xs4all.nlE<gt>> wrote this based on the
features provided by perl5's Text::CSV_XS.
Liz Mattijsen helped in getting the best out of perl6.
=head1 COPYRIGHT AND LICENSE
Copyright (C) 2014-2017 H.Merijn Brand. All rights reserved.
This library is free software; you can redistribute and/or modify it under
the same terms as Perl itself.
=cut
=for elvis
:ex:se gw=75|color guide #ff0000: