IN ORDER TO BECOME FAMILIAR WITH THE DecisionTree MODULE
========================================================
(1) First run the scripts
construct_dt_and_classify_one_sample_case1.pl
construct_dt_and_classify_one_sample_case2.pl
construct_dt_and_classify_one_sample_case3.pl
construct_dt_and_classify_one_sample_case4.pl
as they are. The first script is for the purely symbolic case, the
second for a case that involves both numeric and symbolic features, the
third for the case of purely numeric features, and the last for the
case when the training data is synthetically generated by the script
generate_training_data_numeric.pl
Next, try to modify the test sample in these scripts and see what
classification results you get for the new test samples.
(2) The second and the third scripts listed above use the training file
`stage3cancer.csv'. The first script named above uses the training
file `training.dat'. And the last script named above uses the training
data file `training.csv'. These training files serve as examples for:
stage3cancer.csv: Example of a CSV training data file with both
symbolic and numeric features. The class label
in this case corresponds to the label `pgstat' in
the first record at the top. This implies that
the class label is in the column whose column index
is 2. (Note that column indexing is zero based.
That is, the index of the first column is 0.)
training.csv : Example of a CSV training data file for the
purely numeric case. Contains two classes, each
a Gaussian distribution in 2D. The parameters of
the two Gaussians are in the file:
`param_numeric.txt'
training.dt : Example of a `.dat' file.
Note again that you can use a CSV file for the cases of purely symbolic
data, purely numeric data, or a mixture of the two. However, a `.dat'
file can only be used for purely symbolic data.
There are two additional training data files in the directory:
training2.csv
training3.csv
These are similar to the file `training.csv' in the sense that
they both contain two classes, each a 2D Gaussian distribution.
The first, `training2.csv' was generated by the script
`generate_training_data_numeric.pl ' using the parameter file
param_numeric_strongly_overlapping_classes.txt
and the second, `training3.csv' was generated by the same script
using the parameter file
param_numeric_extremely_overlapping_classes.txt
Study the training datafiles carefully. Now create your own
datafiles that follow the formatting guidelines in these data files
and create scripts similar to those in Item (1) above for
working with your data.
(3) So far we have talked about classifying one test data record at a time.
You can place multiple test data records in a disk file and classify
them all in one go. To see how that can be done, execute the following
two command lines in the `examples' directory:
classify_test_data_in_a_file_numeric.pl training4.csv test4.csv out4.csv
classify_test_data_in_a_file_symbolic.pl training4.dat test4.dat out4.dat
Each of these two scripts constructs the decision tree from the data in
the first argument file and then uses it to classify the data in the
second argument file. The computed class labels are deposited in the
third argument file.
In general, the test data files should look identical to the training
data files. Of course, for real-world test data, you will not have the
class labels for the test samples. You are still required to reserve a
column for the class label, which now must be just the empty string ""
for each data record. For example, the test data supplied in the
following two calls through the files test4_no_class_labels.csv and
test4_no_class_labels.dat does not mention class labels:
classify_test_data_in_a_file_numeric.pl training4.csv test4_no_class_labels.csv out4.csv
classify_test_data_in_a_file_symbolic.pl training4.dat test4_no_class_labels.dat out4.dat
For bulk classification, the output file can also be a `.txt' file. In
that case, you will see white-space separate results in the output
file. When you mention a `.txt' file for the output, you can control
the extent of information placed in the output file by setting the
variable $show_hard_classifications in the script. If this variable is
set, the output will show only the most probable class for each test
data record.
> TO REMIND THE READER AGAIN, IF YOUR TRAINING DATA USES JUST NUMERIC
> FEATURES OR A MIXTURE OF NUMERIC AND SYMBOLIC FEATURES, YOU MUST USE
> CSV FILES FOR TRAINING AND TEST DATA.
(4) Let's now talk about how you can deal with features that, statistically
speaking, are not so "nice". We are talking about features with
heavy-tailed distributions over large value ranges. As mentioned in
the HTML based API for this module, such features can create problems
with the estimation of the probability distributions associated with
them. As mentioned there, the main problem that such features cause
is with deciding how best to sample the value range.
Beginning with Version 2.22, you have two options in dealing with such
features. You can choose to go with the default behavior of the
module, which is to sample the value range for such a feature over a
maximum of 500 points. Or, you can supply an additional option to the
constructor that sets a user-defined value for the number of points to
use. The name of the option is "number_of_histogram_bins". The
following script
construct_dt_for_heavytailed.pl
shows an example of a DecisionTree constructor with the
"number_of_histogram_bins" option.
(5) To see an example of what to do when your CSV file has double-quoted
values (that are allowed to include commas) for fields and when you
are dealing with CSV files with tens of thousands of records, play
with the script
dt_example_involving_csv_cleanup.pl
Read the comment block in this script for the importance of explicitly
setting the constructor parameter 'number_of_histogram_bins' when you
are dealing with CSV training files with tens of thousands of records.
===========================================================================
FOR USING A DECISION TREE CLASSIFIER INTERACTIVELY
Starting with Version 1.6 of the module, you can use the DecisionTree
classifier in an interactive mode. In this mode, after you have
constructed the decision tree, the user is prompted for answers to the
questions regarding the feature tests at the nodes of the tree. Depending
on the answer supplied by the user at a node, the classifier takes a path
corresponding to the answer to descend down the tree to the next node, and
so on. To get a feel for using a decision tree in this mode, examine the
script
classify_by_asking_questions.pl
Execute the script as it is and see what happens.
===========================================================================
EVALUATING THE CLASS DISCRIMINATORY POWER OF YOUR TRAINING DATA
Given a training data file that contains data records and the associated
class labels, one often wants to know the quality of the data in the file.
In other words, one wants to know if a training data file contains
sufficient information to discriminate between the different classes
mentioned in the file.
Starting with Version 2.2 of the DecisionTree module, you can now run a
10-fold cross-validation test on your training data to find out how much
class-discriminatory information is contained in the data. The following
two scripts in the Examples directory:
evaluate_training_data1.pl
evaluate_training_data2.pl
As these scripts show, the following class
EvalTrainingData
defined in the main DecisionTree module file makes it straightforward to
evaluate the class discriminatory power your data (as long as it resides in
a `.csv' file.) This new class is is a subclass of the DecisionTree class
in the module file.
Both the `evaluate' scripts mentioned above are identical in terms of the
usage logic shown. The first is specifically for the training data file
`stage3cancer.csv' and second for the training data files `training.csv',
`training2.csv', and `training3.csv'. The latter three data files contain
two Gaussian classes that are increasingly overlapping. You can see for
yourself the decreasing quality of the training data as you evaluate first
the training file `training.csv', then the training file `training2.csv',
and finally the training file `training3.csv'.
===========================================================================
USING THE DT INTROSPECTION CLASS
Starting with Version 2.3, you can ask the DTIntrospection class of the
module to explain the classification decisions made at the different nodes
of the decision tree. An instance of this class can also show as to which
nodes of the tree are directly affected by a given training sample. A node
is affected directly by a training sample if the sample falls directly in
the portion of the feature space corresponding to that node. Yet another
thing that an instance of this class can show is how the influence of a
given training sample propagates in the decision tree.
Perhaps the most important bit of information you are likely to seek
through DT introspection is the list of the training samples that fall
directly in the portion of the feature space that is assigned to a node.
However, note that, when training samples are non-uniformly distributed in
the underlying feature space, it is possible for a node to exist even when
there are no training samples in the portion of the feature space assigned
to the node. That is because the decision tree is constructed from the
probability densities estimated from the training data. When the training
samples are non-uniformly distributed, it is entirely possible for the
estimated probability densities to be non-zero in a small region around a
point even when there are no training samples specifically in that region.
(After you have created a statistical model for, say, the height
distribution of people in a community, the model may return a non-zero
probability for the height values in a small interval even if the community
does not include a single individual whose height falls in that interval.)
That a decision-tree node can exist even where there are no training
samples in the portion of the feature space assigned to that node is an
indication of the generalization abilities of decision-tree-based
classification.
See the following three scripts in the Examples directory for how to carry
out DT introspection:
introspection_in_a_loop_interactive.pl
introspection_show_training_samples_at_all_nodes_direct_influence.pl
introspection_show_training_samples_to_nodes_influence_propagation.pl
The first script places you in an interactive session in which you will be
asked for the node number you are interested in. Subsequently, you will be
asked for whether or not you are interested in specific questions that
introspection can provide answers for. The second script descends down the
decision tree and shows for each node the training samples that fall
directly in the portion of the feature space assigned to that node. The
third script shows for each training sample how it affects the
decision-tree nodes either directly or indirectly through the
generalization achieved by the probabilistic modeling of the data.
===========================================================================
GENERATING SYNTHETIC TRAINING AND TEST DATA
Starting with Version 1.6, you can use the module itself to generate
synthetic training and test data. See the script
generate_training_and_test_data_numeric.pl
generate_training_and_test_data_symbolic.pl
for how to generate training data for the decision-tree classifier for the
purely numeric case and for the purely symbolic case. The data is
generated according to the information placed in a parameter file in each
case. These files must follow certain rules regarding the declaration of
the classes, the features, the possible values for the features, etc. An
example of such a parameter file for the numeric case is:
param_numeric.txt
and for the symbolic case:
param_symbolic.txt