modeo-data / airbyte_serverless Goto Github PK

Airbyte made simple

AirbyteServerless is a simple tool to manage Airbyte connectors, run them locally or deploy them in serverless mode.

Airbyte is a must-have in your data-stack with its catalog of open-source connectors to move your data from any source to your data-warehouse.

To manage these connectors, Airbyte offers Airbyte-Open-Source-Platform which includes a server, workers, database, UI, orchestrator, connectors, secret manager, logs manager, etc.

AirbyteServerless aims at offering a lightweight alternative to Airbyte-Open-Source-Platform to simplify connectors management.

abs is the CLI (command-line-interface) of AirbyteServerless which facilitates connectors management.

pip install airbyte-serverless

abs create my_first_connection --source="airbyte/source-faker:0.1.4" --destination="bigquery:my_project.my_dataset" --remote-runner "cloud_run_job"

1. Docker is required. Make sure you have it installed.
2. source param can be any Public Docker Airbyte Source (here is the list). We recomend that you use faker source to get started.
3. destination param must be one of the following:
   • print (default value if not set)
   • bigquery
   • contributions are welcome to offer more destinations 🤗
4. remote-runner param must be cloud_run_job. More integrations will come in the future. This remote-runner is only used if you want to run the connection on a remote runner and schedule it.
5. The command will create a configuration file ./connections/my_first_connection.yaml with initialized configuration.
6. Update this configuration file to suit your needs.

abs run my_first_connection

1. This will launch an Extract-Load Job from the source to the destination.
2. The run commmand will only work if you have correctly edited ./connections/my_first_connection.yaml configuration file.
3. If you chose bigquery destination, you must:
   • have gcloud installed on your machine with default credentials initialized with the command gcloud auth application-default login.
   • have correctly edited the destination section of ./connections/my_first_connection.yaml configuration file. You must have dataEditor permission on the chosen BigQuery dataset.
4. Data is always appended at destination (not replaced nor upserted). It will be in raw format.
5. If the connector supports incremental extract (extract only new or recently modified data) then this mode is chosen.

You may not want to copy all the data that the source can get. To see all available streams run:

abs list-available-streams my_first_connection

If you want to configure your connection with only some of these streams, run:

abs set-streams my_first_connection "stream1,stream2"

Next run executions will extract selected streams only.

abs remote-run my_first_connection

2. The remote-run commmand will only work if you have correctly edited ./connections/my_first_connection.yaml configuration file including the remote_runner part.

1. This command will launch an Extract-Load Job like the abs run command. The main difference is that the command will be run on a remote deployed container (we use Cloud Run Job as the only container runner for now).
2. If you chose bigquery destination, the selected service account must be bigquery.dataEditor on the target dataset and have permission to create some BigQuery jobs in the project.

abs schedule-remote-run my_first_connection "0 * * * *"

⚠️ THIS IS NOT IMPLEMENTED YET

$ abs --help
Usage: abs [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  create                  Create CONNECTION
  list                    List created connections
  list-available-streams  List available streams of CONNECTION
  remote-run              Run CONNECTION Extract-Load Job from remote runner
  run                     Run CONNECTION Extract-Load Job
  run-env-vars            Run Extract-Load Job configured by environment...
  set-streams             Set STREAMS to retrieve for CONNECTION (STREAMS...

Any contribution is more than welcome 🤗!

• Add a ⭐ on the repo to show your support
• Raise an issue to raise a bug or suggest improvements
• Open a PR! Below are some suggestions of work to be done:
  • implements a scheduler
  • improve secrets management (use secret manager)
  • implement the get_logs method of BigQueryDestination
  • add a new destination connector (Cloud Storage?)
  • add more remote runners such compute instances.
  • implements vpc access
  • implement optional post-processing (replace, upsert data at destination instead of append?)

• Big kudos to Airbyte for all the hard work on connectors!
• The generation of the sample connector configuration in yaml is heavily inspired from the code of octavia CLI developed by airbyte.