This is a Singer target that puts data in an S3 bucket. Data piped to the target must follow the Singer spec. Currently this target supports no state parameter, it simply copies all injected data to S3.

Based on the Python Singer.io target template

Provided the following configuration is passed to the target

{
  "bucket_name": "do-da-airflow-files-stg",
  "buffer_size": 10
}

one can target-s3 singer records like

echo '{"type": "RECORD", "stream": "json-paths", "record": {"ke1": "val1", "key2": "val2"}}' |\
target-s3

and, provided bucket do-da-airflow-files-stg exists, a new S3 object will be created in the bucket

# on json-paths/2019-04-29T12:18:57.335226
{"ke1": "val1", "key2": "val2"}

The configurations in the file can be overriden with the command line parameter --overrides, which takes configuration overrides in a JSON string and applies them over the passed config file (if any). So

target-s3 -c config.conf --overrides '{"buffer_size": 1000}'

will override the buffer_size parameter for that particular execution.

To install the target-s3 utility as a system command, create and activate a Python3 virtual environment

$ cd target-foobar
$ python3 -m venv ~/.virtualenvs/target-s3
$ source ~/.virtualenvs/target-s3/bin/activate

and install the package

$ pip install -e .

If you prefer not to install the package, you can still run the target with python3 target_s3/__init__.py