• Download the correct version of hadoop-openstack integration lib from here or use the attached jar
• Place it to the folder containing hadoop libraries on every node (if CDH is used then this folder would be /opt/cloudera/parcels/CDH/lib/hadoop)
• Add lines from configuration.xml to core-site.xml configuration file (if CLoudera Manager is used, go to YARN configuration, search for YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml and add the lines to the section, then restart cluster). Note that you need to specify correct parameters specific to your environment: PROVIDER, AUTH-URL, REGION-NAME, TENANT-NAME
• Test the integration using commands like below:

Copy data from HDFS to Swift:

hadoop distcp -D fs.swift.service.<PROVIDER>.username=<username> -D fs.swift.service.<PROVIDER>.password=<api-key> -update -p <PATH-ON-HDFS> swift://<CONTAINER-NAME>.<PROVIDER>/<OBJECT-NAME>

Copy data from Swift to HDFS:

hadoop distcp -D fs.swift.service.<PROVIDER>.username=<username> -D fs.swift.service.<PROVIDER>.password=<api-key> -update -p swift://<CONTAINER-NAME>.<PROVIDER>/<OBJECT-NAME> <PATH-ON-HDFS> 

Note: Parameters fs.swift.service.<PROVIDER>.username=<username> and fs.swift.service.<PROVIDER>.password=<api-key> can be added to core-site.xml config file, and once added there is no need to specify them via -D options in hadoop distcp command.

• Copying 100GB data from HDFS to Swift: ~43 mins

You can use Oozie workflow to perform regular backup from HDFS to Swift.

Please follow the below steps to enable the workflow on your environment.

• Clone the repo to your environment and cd to workflows/hdfs-to-swift
• Edit hdfs-to-swift.properties file and specify parameters specific to your environment
• Edit sourcelist.txt file and specify HDFS directories to backup
• Edit distcp.sh file and specify parameters specific to your environment (edit lines 4, 5, 7 and 12)
• Copy workflow directory to HDFS:

hdfs dfs -put -f workflows /user/$USER

• Run coordinator job as below:

oozie job -oozie http://<oozie-server-hostname>:11000/oozie -run -config workflows/hdfs-to-swift/hdfs-to-swift.properties

• Monitor the coordinator and workflow jobs via HUE or Oozie WebUI.