TwinGraph provides a Python-based high-throughput container orchestration framework for simulation, predictive modeling and optimization workflows. It supports asynchronous multi-host computing, dynamic directed-acyclic-graph (DAG) pipelines, and recording of custom workflow artifacts and attributes within a graph database, for repeatability and auditability.
TwinGraph is used by adding decorators to Python functions to record attributes associated with these functions, such as inputs/outputs, source code and compute platform in a TinkerGraph or Amazon Neptune database. It is also optionally a graph orchestrator using Celery in the backend and runs the decorated functions on a chosen compute (AWS Batch, AWS Lambda, Amazon EKS) and container orchestrator (Kubernetes, Docker Compose) in an asynchronous manner.
TwinGraph scales to hundreds of thousands of containerized compute tasks in a number of different compute nodes/hosts; communication of information between tasks is handled through message queues in an event-driven workflow chain. An example architectural flow of information is shown in Figure 1:
Figure 1: Overall Information Flow.
There are a number of key capabilities outlined in the following Figure 2 for TwinGraph. Instead of domain specific language (DSL) for specifying pipelines, TwinGraph uses algorithms defined in native Python (loops, conditionals & recursion) to define the control flow, i.e. dependencies of graph structure on intermediate outputs and runtime stochasticity.
Figure 2: Challenges in Designing Orchestrators
The examples highlighted in the next section provide an overview of the capabilities of TwinGraph.
- Linux
- Amazon Linux 2, Centos 7, RHEL, Fedora, Ubuntu, etc.
- Other Operating Systems
- Please use the Docker Compose installation option
There are two API calls within TwinGraph - component and pipeline - information about these can be found here.
There are a series of examples located in the examples/orchestration_demos folder. These are designed to show how TwinGraph can be used through the two exposed Python decorator APIs, 'component' and 'pipeline' to orchestrate complex workflows on heterogeneous compute environments. The summary and links to the individual Readmes is given below. Before running AWS service specific examples (Demo 1-) please ensure aws-cli is configured, and run update_credentials.py. To stop running Celery tasks, use stop_and_delete.py.
Example | Description |
---|---|
Demo 1 | Demonstrate graph tracing capability with local compute |
Demo 2 | Building a Docker container & graph tracing with containerized (Docker) compute |
Demo 3 | Automatically including git history in traced attributes |
Demo 4 | Using Celery to dispatch tasks and perform asynchronous computing |
Demo 5 | Using Celery to dispatch and run containerized tasks on Kubernetes (MiniK8s, EKS) |
Demo 6 | Using Celery to dispatch and run containerized tasks on AWS Batch |
Demo 7 | Using Celery to dispatch tasks to AWS Lambda |
Demo 8 | Using Celery to dispatch multiple tasks to different platforms/environments |
Demo 9 | Using Amazon Neptune instead of TinkerGraph for graph tracing |
Demo 10 | Running multiple pipelines together |
- RHEL/Centos/Fedora/AL2:
sudo yum install make git docker python
- Ubuntu/Debian:
sudo apt-get install -y make git python3 python-is-python3 apt-transport-https curl gnupg-agent ca-certificates software-properties-common unzip python3-pip
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt install -y docker-ce docker-ce-cli containerd.io
- Clone the repository:
git clone https://github.com/aws-samples/twingraph.git
make install
make docker_containers_poetry
Note that this step might indicate some missing packages on some distributions, please remedy this by installing any missing base packages as indicated during the installation.
For users of Kubernetes and Amazon EKS, please also install kubectl and eksctl separately, and optionally Kubernetes dashboard for monitoring and debugging.
make uninstall
- The packages AWS-CLI and poetry virtual environment have to be removed manually:
poetry env remove <PATH-TO-PYTHON-EXE>
-
Install and ensure Docker with the Compose tool is running:
docker info
-
Run Docker Compose, and open an interactive shell:
docker compose up -d
- Note: When running for the first time,
compose
might show error in the first line as the image does not exist yet, but this will be remedied automatically by the script building/pulling the container images.
- Note: When running for the first time,
-
Open an interactive shell to run the code, or develop within the container using VSCode remote development extension.
docker exec -it twingraph-twingraph-1 bash
-
Once completed, you can bring down all the containers:
docker compose down
-
Note: When using Compose, TwinGraph API decorator variables need to point to Redis, Gremlin server and TwinGraph hosts - examples included in the provided Dockerfile should run without any changes with the provided Compose script; key in 'tinkergraph-server' instead of 'localhost' when using the Gremlin visualizer in the host field.
-
If using the provided Dockerfile without Compose, please replace the ENV vars in lines 3-5 with localhost or the host where the other containers are running.
If you have any issues with the quick installation, and are not able to use Docker Compose, follow instructions here for manually installing required packages and core modules of TwinGraph.
The following ports need to be forwarded to visualize on the local/client machine on a web browser (e.g. on a local browser such as Chrome or Edge, go to localhost:PORT). For more details on how to port forward using various tools and a list of additional ports to forward, please see here.
PORT | Description |
---|---|
3000 | Gremlin Visualizer Viewing Port - Open localhost:3000 |
3001 | Gremlin Visualizer Data Backend |
Security Best Practice: When opening connections from a remote host (i.e. Linux machine running TwinGraph) to local client, ensure that security best practices are followed and the remote machine is not open to unrestricted access to the public (0.0.0.0/0). If using AWS resources, please consult with your security team for any concerns and follow AWS guidelines here.
The known issues and limitations can be found here.
Tests can be run using the PyTest system, with the following command:
cd tests
pytest
Note that AWS credentials need to be configured prior to running tests for AWS Batch and AWS Lambda, and Kubernetes config has to be done prior to Kubernetes tests.
We welcome all contributions to improve the code, identify bugs and adopt best development and deployment practices. Please be sure to run the tests prior to commits in the repo. Rules and instructions for contributing can be found here.
Tools used in this framework:
AWS Resources:
- Amazon Web Services (AWS)
- Amazon EC2
- AWS Lambda
- Amazon Elastic Container Service (ECS)
- Amazon Elastic Kubernetes Service (EKS)
- Amazon EKS Blueprints
- AWS Batch
- Amazon Neptune
- Amazon SQS
This open source framework was developed by the Autonomous Computing Team within Amazon Web Services (AWS) Worldwide Specialist Organization (WWSO), led by Vidyasagar Ananthan and Satheesh Maheswaran, with contributions from Cheryl Abundo on Amazon EKS Blueprints, David Sauerwein on applications, and Ross Pivovar for initial testing and identifying issues. The authors acknowledge and thank Adam Rasheed for his leadership and support to our team, and to Alex Iankoulski for his detailed guidance and contributions on Docker tools and expertise in reviewing the code.