Scalable automated testing and deployment of upstream (core), plugin and theme updates on Pantheon sites. This repository uses these open source tools:
- Terminus for interacting with Pantheon on the command line
- CircleCI for running jobs in the cloud
- WP-CLI for interacting with WordPress on the command line
- Drush for interacting with Drupal on the command line
- Behat for integration and acceptance testing
- WordHat for WordPress-specific functionality for common testing scenarios in Behat
- Google Lighthouse for performance testing
- BackstopJS for visual regression testing
- Slack for communication
While this script has worked well for us your mileage may vary. This repository is a proof of concept example of advanced Pantheon platform usage and is provided without warranty or direct support.
Issues and questions may be filed in GitHub but their resolution is not guaranteed.
This script will loop through all sites in sites-to-auto-update.json
and:
- Authenticate with Terminus via machine token
- Delete the multidev environment
auto-update
- Recreate the multidev environment
auto-update
- Deletion and recreation is done to clear any existing changes and pull the latest database/files from the live environment
- This is opt-out per site and should only be disabled if using Solr or multidev creation takes an excessively long time due to large file and database sizes
- Switch the multidev environment
auto-update
to git mode - Check for and apply Pantheon upstream updates
- WordPress or Drupal core updates are managed in the default upstream
- Custom upstream updates will be applied if using a custom upstream
- Switch the multidev environment
auto-update
to SFTP mode - Check for and apply WordPress plugin or Drupal module updates, if available
- Check for and apply WordPress theme updates, if available
- If no updates are available the script will stop here
- Run a visual regression test between the live environment and the multidev environment
- If the visual regression test fails the script will stop here and post a link to the visual regression report in Slack
- Merge the multidev environment with the dev environment
- Deploy the dev environment to the test environment
- Create a backup of the test environment
- This is opt-out per site and should only be disabled if backups take an excessively long time due to large file and database sizes and you have regularly backups scheduled via another method.
- Create a backup of the live environment
- This is opt-out per site and should only be disabled if backups take an excessively long time due to large file and database sizes and you have regularly backups scheduled via another method.
- Deploy the test environment to the live environment
- Post a success message to Slack
- Test failures will also be reported to Slack
- Create a new GitHub repository
- Don't fork this repository, instead clone it and change the remote URL to the GitHub repository above and push the code there
- Update
backstop.template.json
to meet your needs- For example, you may want to tweak things like
viewport
- For example, you may want to tweak things like
- Create a CircleCI project
- It should be linked to your GitHub repository
- Add environment variables to CircleCI for the following:
MULTIDEV
: The multidev name to use for applying/testing updates. Defaults toauto-update
CIRCLE_TOKEN
: A CircleCI API token with access to the CircleCI project created above.TERMINUS_MACHINE_TOKEN
: A Pantheon Terminus machine token with access to all the sites you plan to update.SLACK_HOOK_URL
: The Slack incoming webhook URLSLACK_CHANNEL
: The Slack channel to post notifications toSLACK_USERNAME
: The username to post to Slack with
- Add an SSH key to Pantheon and to the CircleCI project
- Edit
sites-to-auto-update.json
and set the following key for each JSON object for sites you wish to auto update. The order is important do not change the order of the keys in the object or skip keys.SITE_UUID
: The site UUID, which can be acquired withterminus site:list
or is in the dashboard URL for the site.SITE_NAME
: The site machine name, which can be acquired withterminus site:list
.CREATE_BACKUPS
:0
or1
to determine if a backup of the site is made before deployment. You may want to disable backups for sites where a backup takes an excessively long time due to large file and database sizes and you have regularly backups scheduled via another method.RECREATE_MULTIDEV
:0
or1
to determine if the multidev is deleted and recreated with each run.LIVE_URL
: The preferred custom domain (full URL) for the live environment to use in automated testing or set to0
to use the default Pantheon hosted URL,
- Backups of sites with large a large database or media files are taking too long and timing out on Circle CI
- Sites with a Solr Index fail visual regression tests as the multidev created doesn't have items indexed in Solr
backstop-crawl
on sites with many (~100+) URLs are taking too long and timing out on Circle CI.- This can be alleviated with a custom
SITE_NAME.backstop.json
orSITE_NAME.backstop-config.js
file.
- This can be alleviated with a custom
- If automated testing fails and
RECREATE_MULTIDEV
is set to0
then a multidev with updates applied will persist causing subsequent update checks to return no updates and prevent automated tests from running again. This cycle will break when there are further updates released to be applied. - Composer managed sites are not supported. For Composer managed sites a service like dependencies.io, which will create pull requests for Composer updates, is useful.
backstop.template.json
is used as a generic template for backstop-crawl
when creating backstop.json
files dynamically.
For each site a custom template in the form of SITE_NAME.backstop.json
or SITE_NAME.backstop-config.js
, where SITE_NAME
is the machine name for the Pantheon site, can be also be created and will take precedence over the template above. The former is a hard coded Backstop JS file and the latter is a JavaScript file that must export valid JSON contents of a Backstop JS file.
The custom templates above are most useful for sites with many URLs where a crawl will timeout. In this case manually specifying which URLs to test is needed.
Behat testing is currently only supported for WordPress sites. To have Behat tests run for a site create Behat feature files in the directory tests/behat/<site-name>/features
, replacing <site-name>
with the Pantheon site machine name.
This workflow assumes that the Dev and Test environments on Pantheon are always in a shippable state as the script will automatically deploy changes from Dev to Test and Live if the visual regression tests of updates pass.
All work that isn't ready for deployment to production should be kept in a Pantheon multidev environment, on a separate git branch.
Scalability relies on the number of CircleCI workers available. If workers are available to process jobs in parallel this script will scale very well.
- Initial beta release