Thursday, 21 June 2012

Java problems refering to unknown classes.

There is a problem/misunderstanding generated because of a change in the hadoop api at some point, which affects the configuration of the job quite a lot.

If you download any example from the internet, make sure that the libraries being imported contain the string "mapreduce" instead of "mapred". The later one will bring you trouble as it is the deprecated version.

Another way to recognize the version we are using from the code is to check that the job configuration resembles this one:
 
    // This three lines should be very similar
    Configuration conf = new Configuration();
    Job job = new Job(conf, "word count");
    job.setJarByClass(WordCount.class);  <--- This line is VERY important, so that
                                                                        hadoop knows what to execute
  

    // This is less important as I think is is pretty much the same. but make sure
    // your version is similar to this:

    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);


Hope this helps,

Jesus

No comments:

Post a Comment