- Download the correct version of hadoop-openstack integration lib from here or use the attached jar
- Place it to the folder containing hadoop libraries on every node (if CDH is used then this folder would be /opt/cloudera/parcels/CDH/lib/hadoop)
- Add lines from configuration.xml to core-site.xml configuration file (if CLoudera Manager is used, go to YARN configuration, search for YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml and add the lines to the section, then restart cluster). Note that you need to specify correct parameters specific to your environment: PROVIDER, AUTH-URL, REGION-NAME, TENANT-NAME
- Test the integration using commands like below:
Copy data from HDFS to Swift:
hadoop distcp -D fs.swift.service.<PROVIDER>.username=<username> -D fs.swift.service.<PROVIDER>.password=<api-key> -update -p <PATH-ON-HDFS> swift://<CONTAINER-NAME>.<PROVIDER>/<OBJECT-NAME>
Copy data from Swift to HDFS:
hadoop distcp -D fs.swift.service.<PROVIDER>.username=<username> -D fs.swift.service.<PROVIDER>.password=<api-key> -update -p swift://<CONTAINER-NAME>.<PROVIDER>/<OBJECT-NAME> <PATH-ON-HDFS>
Note: Parameters fs.swift.service.<PROVIDER>.username=<username> and fs.swift.service.<PROVIDER>.password=<api-key> can be added to core-site.xml config file, and once added there is no need to specify them via -D options in hadoop distcp
command.
- Copying 100GB data from HDFS to Swift: ~43 mins
You can use Oozie workflow to perform regular backup from HDFS to Swift.
Please follow the below steps to enable the workflow on your environment.
- Clone the repo to your environment and cd to workflows/hdfs-to-swift
- Edit hdfs-to-swift.properties file and specify parameters specific to your environment
- Edit sourcelist.txt file and specify HDFS directories to backup
- Edit distcp.sh file and specify parameters specific to your environment (edit lines 4, 5, 7 and 12)
- Copy workflow directory to HDFS:
hdfs dfs -put -f workflows /user/$USER
- Run coordinator job as below:
oozie job -oozie http://<oozie-server-hostname>:11000/oozie -run -config workflows/hdfs-to-swift/hdfs-to-swift.properties
- Monitor the coordinator and workflow jobs via HUE or Oozie WebUI.