The job has options for specifying Hadoop connection details and input/output paths. It also allows
control over the number of map tasks that actually get executed via the max-split-size option (this
sets dfs.block.size) as Hadoop's default of 64Mb may not be appropriate for batch learning tasks,
depending on data characteristics. The classifier job, covered in the next instalment of this series,
has a pre-processing option to create a set of randomly shuffled input data chunks, which gives
greater control over the number and size of the data sets processed by the mappers. The ARFF
header job also has a set of options for controlling how the CSV input file gets parsed and
processed. It is possible to specify attribute (column) names directly or have them read from a
"names" file (one attribute name per line; not to be confused with the C4.5 ".names" file format)
stored on the local file system or in HDFS.
As other Weka Hadoop jobs use the ARFF job internally, and it is not necessary to repeat it for
subsequent jobs that process the same data set, it is possible to prevent the job from executing by
providing a path to an existing ARFF header (in or out of HDFS) to use.
The image below shows what the job produces for the UCI hypothyroid dataset. Given the
configuration for this job shown above, the header gets stored as
/users/mhall/output/arff/hypothyroid.arff in HDFS. It also gets displayed by the TextViewer in the
Knowledge Flow. "Class" is the last of the actual data attributes and the ones that occur after that
are the summary meta attributes that correspond to each of the nominal or numeric attributes in the
data.
我的内容管理
展开
