This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

The monitoring automation currently hardcodes namespace helix and package helix-services. It should be possible to support monitoring of actions deployed in other namespaces/packages.

We are now part of the CircleCI Windows beta (docs here: https://github.com/CircleCI-Public/windows-preview-docs) and should have a build stage that runs on Windows to rule out compatibility issues.

The monitoring job / code introduced by adobe/helix-status#46 is copied over across 15 repos! A change would require a lot of PRs. This code / job(s) could moved into a CircleCI Orb. A new release of the Orb would be detected by renovate and PRs would come automatically.

cc @rofe

The smoke tests as defined in https://github.com/adobe/helix-continuous/blob/master/.circleci/config.yml are behaving like this:

• construct a hlx-cli with the master of everything + the branch of the PR being tested (in any module)
• run the smoke tests in project-helix.io with this "patched" cli

Experience showed that this is not enough to not break the cli: when aggregating a branch of some module in the cli, this might break something completely unrelated or not visible by the simple / basic project-helix.io smoke tests.

The proposal here is to execute the hlx-cli tests as part of the smoke tests suite.

Helix Pages smoke tests validate (see adobe/helix-pages#8) that if you make a change in Helix Pages code, it does not break existing site. Currently this "validation" (hlx deploy / publish on a test domain) uses the latest released hlx cli.

The goal here is to integrate these smoke tests into helix-continuous smoke tests, i.e. when we change something inside hlx cli then Helix Pages smoke tests are run against the "work in progress" hlx cli, similar to what we do with project-helix.io.

But this comes with some difficulties:

• helix-continuous is currently responsible for building a "patched" hlx cli (a cli that integrates the changes being made by a developer) - this "patched" hlx cli needs to be "given" to Helix Pages smoke tests. Not sure how we could do that.
• Helix Pages smoke tests have been built with the idea of simplicity in terms of visualisation of the execution (check https://circleci.com/workflow-run/6a600e23-d940-4336-9c62-3cf7e0a7cb53) but also performances: the smoke tests executed runs in parallel and the UI can display what is going on which makes it easy to understand what / where it failed. We should avoid another remote call of the job (the orb we add to each project is already doing a remote call to trigger helix-continuous smoke tests) where the execution would be hidden in a bash script and makes it hard for a developer to trace.

I do not know yet how we could implement this, some researches / thinking need to be done.

Orb configs should always be validated

@kptdobe

With adobe/helix-cli#1200, the pipeline is not part of the cli anymore: either the project specify the version it needs or the latest is installed into the project.
When working on a helix-pipeline branch, the smoke tests run with the assumption the cli can be patched with the specified pipeline branch. This is not the case work anymore and new logic needs to be added in each test project to handle a custom pipeline:

• the project-helix.io test must install the specified helix-pipeline version if it is provided via the parameters
• helix-cli tests must accept a pipeline version as a parameter. See adobe/helix-cli#1242

for example: https://app.circleci.com/pipelines/github/adobe/helix-continuous/6599/workflows/ce079025-b524-41e5-9550-b2096cfb8e0c/jobs/28845

The CircleCi config can be optimised to run certain jobs in parallel like the test helix-cli test and project-helix tests.

As an enhancement to #38, it should be possible to provide the monitor URL as a parameter instead of constructing it from the package name and version.

Examples: helix-bot, helix-bot-action-config and helix-bot-action-login all use a URL pattern that is different from the rest of the Helix services.

The code (https://github.com/adobe/helix-continuous/blob/master/.circleci/orbs/helix-smoke-tests/orb.yml#L62-L117) required to start a remote workflow and wait for the execution result has started to be copied in various places:

• https://github.com/adobe/helix-publish/blob/master/.circleci/config.yml#L47-L106
• https://github.com/adobe/helix-word2md/blob/master/.circleci/config.yml#L37-L114

We should move that code inside an orb to make it easily reusable.

If you create a new branch and PR in helix-cli, the build job (smoke tests) is then executed. The branch is well received as a parameter (debug msg shows the branch name) but tests are running against the helix-cli master version. Reason: the helix-cli is the wrapping project and the initial git clone does not consider the GDM_MODULE_BRANCHES parameter which is only handled in the GDM (which does not take care of the wrapping project)...

To trigger a build manual:

curl \
	--user 20f56f120a30755c27e3579463196936298d543b: \
    --header "Content-Type: application/json" \
    --silent \
    --data "{\"build_parameters\": {\"GDM_MODULE_BRANCHES\": { \"helix-cli\": \"v0.2.1\"}}}" \
    --request POST "https://circleci.com/api/v1.1/project/github/adobe/helix-continuous/tree/master"

This call will trigger the build workflow and is supposed to run the smoke tests with helix-cli v0.2.1 (and all the rest master). This is definitively not supposed to work since v0.2.1 is... from July 2018!

See https://circleci.com/gh/adobe/helix-continuous/364, execution is successful while I expected it to fail very early.

Solution: the initial git clone helix-cli must check if GDM_MODULE_BRANCHES contains helix-cli and use the provided branch / tag name.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

From @trieloff:

This workflow failed: https://circleci.com/gh/adobe/helix-pipeline/1468?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link
but the smoke test that got triggered actually succeeded: https://circleci.com/gh/adobe/helix-continuous/1311

I could find some other occurrences in the CircleCI job history.

Currently the CI executes the smoke tests in more than 4mins. This includes the clone of all the repositories (master branch or specified branch), npm install and test execution. Naturally, the npm install of the cloned repo is the bottleneck. Some researches could be made to improve the execution.

The smoke tests currently builds a work-in-progress version of the helix-cli with the code of the PR being worked on. This is great to detect early integration issues. The tests run are:

• project-helix.io tests (which are pretty outdated): https://github.com/adobe/helix-continuous/blob/master/.circleci/config.yml#L85-L139
• helix-cli tests (with custom cli): https://github.com/adobe/helix-continuous/blob/master/.circleci/config.yml#L141-L169

And that's it.
helix-pages project is our main final "service" and its own smoke tests usually reveal integration issues. But this happens late at the end of the chain.
The proposal here is to integrate those test, i.e. run the helix-pages smoke tests with the work-in-progress version of the helix-cli. Like this, we should know early if a PR breaks of our final service.
The task will require to adjust the helix-pages to run with a custom helix-cli exec.

The command currently hardcodes the helix namespace. It would be great if one could specify a different one (e.g. helix-index as in https://github.com/adobe/helix-index-big-tree/blob/master/.circleci/config.yml)

When dealing with non-backward compatible changes, the smoke tests fail until the project-helix.io code has been adjusted. This is done in a branch and running the tests against the current helix-cli branch (where the backward comp change is occurring) and this project-helix.io branch makes more sense to prepare the new version and validate that everything works as expected.

The process would look like this:

• I change something in helix-cli (or one of the other project)
• If the smoke tests fail for a valid reason (non-backward comp changes), I configure the CI part of helix-cli to define which branch in project-helix.io contains the upgraded project. Smoke tests should become green again.
• once ready, I merge project-helix.io branch into master.
• I change the CI part of helix-cli back to master: smoke tests should be green again

CI part of helix-cli (or other) to be changed has to be determined but it should be around the place we call the orb that runs the smoke tests: https://github.com/adobe/helix-cli/blob/master/.circleci/config.yml#L243-L247

@tripodsan WDYT ?

This job https://circleci.com/gh/adobe/helix-pipeline/4611 triggers a new helix-continuous pipeline but does not get the workflow id (null) which generates a broken observation task and does not give the link to the correct workflow.

New pipeline triggered. See API https://circleci.com/api/v2/pipeline/a27c7aa3-1f7f-4292-999c-47517d932dbd
Waiting now for the workflow to start.
Smoke tests workflow - i.e "build" workflow - triggered. See API https://circleci.com/api/v2/workflow/null

To debug error or check workflow sate, you can check the workflow UI here - https://circleci.com/workflow-run/null

Waiting now for workflow execution.
Smoke tests https://circleci.com/api/v2/workflow/null finished with status "null"
jq: error (at smoke_result.json:0): Cannot iterate over null (null)

In order to check if building and deploying actions really work, we should add a test to the CI to do this.

the test should:

1. run hlx clean && hlx build && hlx deploy
2. request the html action,
3. request the static action

See adobe/helix-home#74

#66 addressed only half of the problem: the helix-cli part has not been done!

• helix-cli tests must accept a pipeline version as a parameter. See adobe/helix-cli#1242

must be implemented!

Smoke tests regularly fires the famous "API rate limit exceeded". Github token must be added.

@trieloff can you enable it ?

The smoke tests are currently being run with a constructed helix-cli which takes the master branch from all the projects, except for the project that triggered the tests which normally passes its working branch.
While this has the advantage to make sure the helix-cli from master will always work with its dependencies, it leads to a lot of tests failures when introducing breaking changes that requires work in multiple modules.
To fix that, the proposal here is to change the helix-cli construction to the following:

• git clone helix-cli
  • use master branch by default
  • use branch passed as a parameter (case if helix-cli is the one being changed)
• build helix-cli
• only git clone and link any project being passed as a parameter
  • the linking must be done in sub dependencies too

The built helix-cli should then be:

• either helix-cli master "patched" with dep_project/branch passed as parameter
• or helix-cli/branch

cc @tripodsan

Description
The smoke test fails in CircleCI with the following output:

  1) project-helix.io renders properly
       Root ("/" / index.html) is rendered using project htl scripts:

      AssertionError [ERR_ASSERTION]: nav in index (summary_html.htl) is included via ESI include
      + expected - actual

      -false
      +true
      
      at Context.it (test/smoke/test.hlx.up.js:51:12)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:189:7)

  2) project-helix.io renders properly
       README.html page is rendered using project htl scripts:

      AssertionError [ERR_ASSERTION]: nav in README (summary_html.htl) is included via ESI include
      + expected - actual

      -false
      +true
      
      at Context.it (test/smoke/test.hlx.up.js:59:12)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:189:7)

  3) project-helix.io renders properly
       content assets in index.html check:

      AssertionError [ERR_ASSERTION]: no assets found
      + expected - actual

      -false
      +true
      
      at Context.it (test/smoke/test.hlx.up.js:82:12)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:189:7)

Seems like there is some problem with ESI includes? For what it's worth, the related commit is adobe/helix-cli@6ba5a88

To Reproduce
See here for first failure (all subsequent runs are failing with the same output):
https://circleci.com/gh/adobe/helix-cli/8016

Expected behavior
The smoke test succeeds.