Creating the jar file to execute in hadoop.

If you are like me, that you have not really been exposed at using java programs directly from the console, you will find it a bit difficult at the beginning.

Luckily IDEs like Eclipe and Netbeans generate executable jars for us and that is pretty much what we need to execute our programs.

In eclipse you have to explicitly generate the jar file of your project, and for this you can have a look at some tutorials on the internet.

Under Netbeans (as it is my case) everytime you execute or build your project, a new jar file is generated under the "dist" subfolder of your project.

I recommend you copy this folder to wherever you want so you can include it in the hadoop jar execution command. I.e something like this:

In the context of the wordcounter example:

bin/hadoop jar dist/WordCount.jar /collections/gutenberg outputfolder

Where:
- "bin/hadoop" is the "hadoop" program located in my "bin" subfolder
- jar tells the hadoop program, that we want to run a jar file/program
- "dist/WordCount.jar" is our wordcount program, located under the "dist" subfolder
- "/collections/gutenberg"   the input file located in the DFS (i.e. hosted in the cluster storage)

- "outputfolder"  The folder where the output is gonna be written to.

Hope this helps

Cheers,

Jesus

Java problems refering to unknown classes.

There is a problem/misunderstanding generated because of a change in the hadoop api at some point, which affects the configuration of the job quite a lot.

If you download any example from the internet, make sure that the libraries being imported contain the string "mapreduce" instead of "mapred". The later one will bring you trouble as it is the deprecated version.

Another way to recognize the version we are using from the code is to check that the job configuration resembles this one:
 
    // This three lines should be very similar
    Configuration conf = new Configuration();
    Job job = new Job(conf, "word count");
    job.setJarByClass(WordCount.class);  <--- This line is VERY important, so that
                                                                        hadoop knows what to execute
  

    // This is less important as I think is is pretty much the same. but make sure
    // your version is similar to this:

    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);


Hope this helps,

Jesus

"Unknown host kairali" problem under windows.

If you are having any problem in which only the "kairali" name is mentioned instead of the full address "kairali.dcs.gla.ac.uk" of the master node. This is due to some problem resolving the host.

(Solution 1, under windows. (thanks to James)).

- Edit the file: "C:\Windows\System32\drivers\etc\hosts"
- And add the line: "kairali              kairali.dcs.gla.ac.uk" (without quotes)
- Now reboot the computer.

Comment 1: This should solve the problem. Although I have experienced it coming back after a while. Rebooting the computer again solves the issue. I know this is not especially convenient, but we haven't found any other fix yet. Try it out, you might be luckier than me.

Comment 2: In my opinion it might be worth it to just use linux when you want to work with the cluster. If your machine is powerful enough you could have a small virtualized copy of linux, that you can use under windows for this purpose.

Welcome!

Hi everybody,

I thought it would be a good idea if we post any problems, and possible solutions in some sort of blog, in an attempt to keep a common source of knowledge and troubleshooting, for all of us to share with respect to Hadoop.

I'll start by posting the problems I have found under windows.

Cheers!