An AWS Lambda function for collating and transferring QSP batch data from CloudWatch to a remote FTP server for further processing.

This project forms part of the ois-tuxedo-stack set of services and infrastructure. It replaces a set of cron jobs that were previously used and functions by collating batch data generated from multiple sources before transferring the resulting data file to a remove FTP server.

During normal operation, the ORDERS service in ois-tuxedo writes comma-separated batch data to a local log file whose path is specified in the QSPFILE environment variable (typically set in configuration files in ois-tuxedo-configs). Many ORDERS services may be active in the same environment—distributed across one or more EC2 instances—and log data for those services will be exported via CloudWatch agent to the same log group, one log stream per EC2 instance. The Lambda function filters data in the log group for a given time period—taking all log streams into consideration—and collates the resulting data before transfer to a remote FTP server.

The following Lambda function environment variables are required:

The Lambda function is triggered by an EventBridge schedule defined in the infrastructure branch of the ois-tuxedo-stack project.

At execution time, the function operates on data generated for the previous day by default. To manually trigger the Lambda function for a specific date, invoke the function using a test event that is configured with a single field named date using the date format DD-MM-YYYY. For example:

{
  "date": "31-01-2022"
}