Procedure to upgrade Hadoop cluster.

Upgrading a HADOOP cluster is easy but a risky task. You need to follow the steps accurately, otherwise you may end up in data loss.

PLAN OF ACTION: 

Right now, my Hadoop cluster is installed with version "1.0.1".

I need to upgrade it to version "1.2.1".

The safe and accurate procedure is as follows:

STEP 1: Stop map-reduce, to stop all process that is using DFS.

bin/hadoop stop-mapred.sh

STEP 2: Run fsck command:

bin/hadoop fsck / -files -blocks -locations > dfs-fsck-1.log

Fix the FS errors. For more details on fixing FS errors, read the following blog post:

http://www.maninmanoj.com/2015/04/how-to-find-fs-errors-in-hdfs-and-fix-it.html

STEP 3 : Run the below command to get a complete namespace of the file system.

bin/hadoop dfs -lsr / > dfs-lsr-1.log

STEP 4: Run report command to create the details of data nodes in the cluster.

bin/hadoop dfsadmin -report > dfs-report-1.log

STEP 5: Stop and restart HDFS cluster, in order to create an up-to-date namespace checkpoint of the old version. Copy the following checkpoint files into a backup directory:

dfs.name.dir/edits
dfs.name.dir/image/fsimage

STEP 6: Stop DFS cluster.

bin/stop-dfs.sh

STEP 7: Now, install new version of hadoop. For example, the new version will be of folder name "/usr/local/hadoop2". We need to do this on all datanodes. Make sure the conf files mapred-site.xml, core-site.xml, hdfs-site.xml, slaves, masters as same as that of old versions conf file. You can use following link for reference:

http://www.maninmanoj.com/2015/03/building-multi-node-hadoop-cluster.html

STEP 8: Now, start the new hadoop namenode using "-upgrade" option as below:

bin/hadoop-daemon.sh start namenode -upgrade

You can check the status of upgrade using the following command:

bin/hadoop dfsadmin -upgradeProgress status.

Once completed, you will get the below message:

From the web interface the message will be as follows:

STEP 9: Run lsr command and compare with dfs-lsr-1.log
bin/hadoop dfs -lsr / > dfs-lsr-0.log

STEP 10: Start DFS cluster.
bin/start-dfs.sh

STEP 11: Run report command and compare with  dfs-report-1.log to ensure all data nodes previously belonging to the cluster are up and running.

bin/hadoop dfsadmin -report > dfs-report-0.log

STEP 12: Run fsck and compare with dfs-fsck-1.log. These files should be identical, unless the fsck reporting format has changed in the new version.

bin/hadoop fsck / -files -blocks -locations > dfs-fsck-0.log

STEP 13: Start map-reduce.

bin/start-mapred.sh

STEP 14: If everything looks good finalize the upgrade using below command:

/bin/hadoop dfsadmin -finalizeUpgrade

Once done, the web interface will show the new version, as below:

If you see anything wrong, stop everything and navigate to bin directory of old hadoop version and start the cluster.

Kool :)