how to run this pipeline ? about aws-data-pipeline HOT 1 CLOSED

Hi there,

You'd have to provision the AWS resources before running the pipeline, so you have to run Terraform first and to do that, the values in terraform.tfvars will need to be filled in. After the resources are provisioned, run the docker-compose file.

I was learning Terraform during this project, so if you are doing the same I would suggest you to try out creating resources in the AWS portal first, then replicate the action with Terraform, following roughly my steps in main.tf. Be careful about the EMR and Redshift costs, always delete them after you are done (until you get the pipeline to delete/pause them for you automatically).

from aws-data-pipeline.