Add to Docs: How continuous deployment with Astronomer set up works, including CI builds/integration with github about astronomer HOT 8 CLOSED

@rbradk So for now, without any further CLI modifications, they can follow the steps I listed above (edited to show <release_name>).

Would you agree @cwurtz?

from astronomer.

If @cwurtz gives a thumbs up, then I think we just point them to their own CI/CD tools docs. There really shouldn't be anything Astro specific here, just standard docker stuff.

from astronomer.

@rbradk the details will differ between CI/CD products, but at a high level, the process is basically the same.

1. Install the astronomer platform, using EE docs. One component of the installation is a private docker registry. The registry is exposed at registry.baseDomain, where baseDomain is the domain the user launched the platform at. When new images are pushed to this registry, they get deployed to the cluster (this is how the CLI works under the hood).
2. Set up a CI/CD pipeline using whatever tool you'd like. Most tools will have some sort of documentation on how to trigger the pipeline from an event from github, like a new tag/release, or push to a branch. In this pipeline, after any tests have been executed, you just need to run a few docker commands. A couple examples:

• https://circleci.com/docs/2.0/building-docker-images/
• https://documentation.codeship.com/pro/builds-and-configuration/image-registries/#custom--self-hosted-registry

Images should be tagged with this format registry.baseDomain/<release_name>/airflow:<version>, and the user/password will be generated when the platform is initially installed.

@andscoop @tedmiston @cwurtz feel free to add anything to this.

from astronomer.

Probably be best to have them download the CLI in their CI build as the registry is currently password protected, and will eventually have checks if the user has permission to deploy. They can do:

astro auth login -d {their domain}
astro airflow deploy {release_name}

Where {their domain} is their base domain, and {release_name} is their kubernetes release name. The CLI will handle the rest.

edit:
Actually thinking about it, we'll need to modify the CLI so the username/pass can be passed to the CLI via a flag, as now it prompts for that info

from astronomer.

We have a customer in Slack who has started down the path of running it on CircleCI already. I spoke with him about a few weeks ago. (Omitting the name here since this repo is public.)

Edit: I just sent him a follow up message to see how it went and if there any pain points.

from astronomer.

I just started a branch for CI deploy docs with some basic information.

https://github.com/astronomerio/astronomer-ee/compare/ci-deploy

Here's what I wrote so far:

Deploy via CI/CD

You can deploy DAGs with continuous integration / continuous deployment via the Astronomer CLI.

While this guide focuses on CircleCI, a similar setup can be accomplished using Travis CI, Jenkins, Codeship, TeamCity, GitLab CI/CD, or any other CI/CD system.

For background information and best practices on CI/CD, we recommend reading the article An Introduction to CI/CD Best Practices from DigitalOcean.

TODO: copy discussion from #75

...

If we all agree, I'm thinking of pasting in what @cwurtz and @schnie wrote above as well.

To @cwurtz point, we could consider wiring up the current CLI with something like expect or pass creds in with a pipe/redirect like this pulled from environment variables today with potentially no change to the CLI.

from astronomer.

@schnie what needs to be true to close this issue out?

from astronomer.

Greg's recommendation is the way to go right now

from astronomer.