Monday 7 December 2015

Hadoop: Export and import hive tables

SCENARIO:

We need to move a table named "daily_input" from cluster whose master server is "hadoop-1-test-master" (source) to "hadoop-2-test-master" (master of destination cluster).

PROCEDURE:

STEP 1 :EXPORT THE TABLE TO HDFS OF CURRENT CLUSTER

Run the following command in hive shell of hadoop-1-test-master:
---------
EXPORT TABLE daily_input TO '/tmp/backup/table-source/';
---------

The above command will export the table "daily_input" to HDFS of source cluster inside the folder named "/tmp/backup1/table-source/".

STEP 2: COPY FROM SOURCE HDFS TO DESTINATION HDFS:

Run the below command in destination cluster:
------------------
hadoop distcp -skipcrccheck -update hftp://hadoop-1-test-master.maninmanoj.com:50070/tmp/backup/table-source hdfs://hadoop-2-test-master.maninmanoj.com:8020/tmp/backup/table-destination
--------------------

The above command will copy the exported table from source "hadoop-1-test-master" to destination "hadoop-2-test-master" HDFS folder "/tmp/backup/table-destination"

STEP 3: IMPORT THE TABLES:

Run the following command in the destination hive shell:

------------
IMPORT TABLE daily_input FROM '/tmp/backup/table-destination';
-----------

The above command will import the table "daily_input" to destination hive table.

IMPORTANT: No need to manually create the table in the destination, it will be automatically created during import.

Reference : http://www.ericlin.me/hive-export-import-command-transfering-data-between-hive-instances

No comments:

Post a Comment

Note: only a member of this blog may post a comment.