saucelabs / saucectl-run-action Goto Github PK

View Code? Open in Web Editor NEW

9.0 4.0 6.0 676 KB

GitHub Action for running saucectl test suites.

Home Page: http://opensource.saucelabs.com/saucectl-run-action/

JavaScript 99.56% Shell 0.44%

github action saucectl saucelabs cypress testcafe puppeteer playwright

saucectl-run-action's Introduction

Saucectl Run Action from Sauce Labs

This action installs saucectl and launches tests.
You can use it to run your tests on Sauce Labs !

Usage

⚠️ Avoid being throttled by GitHub. Be sure to provide GITHUB_TOKEN through the env field, or you may face an API rate limit exceeded error.

- uses: saucelabs/saucectl-run-action@v4
  env:
    GITHUB_TOKEN: ${{ github.token }}
  with:
    # Which version of saucectl.
    # Default: latest
    saucectl-version: v0.123.0

    # Sauce Labs Credentials.
    sauce-username: ${{ secrets.SAUCE_USERNAME }}
    sauce-access-key: ${{ secrets.SAUCE_ACCESS_KEY }}

    # Install saucectl, but don't run it.
    # Default: false
    skip-run: false

    # Relative path under $GITHUB_WORKSPACE to use as the new working directory.
    working-directory: ""

    # An environment variable key-value pair that may be referenced in the tests executed by this command.
    # Currently not supported by espresso/xcuitest.
    env: |
      MY_FIRST_VAR=VALUE
      MY_SECOND_VAR=VALUE

    # Specifies an alternative configuration file for this execution.
    # Default: .sauce/config.yml
    config-file: .sauce/myconfig.yml

    # Specifies the Sauce Labs data center through which tests will run.
    # Valid values are us-west-1 or eu-central-1.
    region: us-west-1

    # Controls how many suites run in parallel.
    # Default: 1
    concurrency: 1

    # Global timeout that limits how long saucectl can run in total.
    # Supports duration values like '10s', '30m' etc.
    timeout: 5m

    # Identifies an active Sauce Connect tunnel to use for secure connectivity to the Sauce Labs cloud.
    tunnel-name: ""

    # Identifies the Sauce Labs user who created the specified tunnel, which is required if the user running the tests did not create the tunnel.
    tunnel-owner: ""
    
    # How long to wait for the specified tunnel to be ready. Supports duration values like '10s', '30m' etc.
    tunnel-timeout: 30s

    # Specifies a test suite to execute by name rather than all suites defined in the config file.
    select-suite: ""

    # Includes the contents of the suite's console.log in the output of the command regardless of the test results. By default, the console log contents are shown for failed test suites only.
    # Default: false
    show-console-log: false

    # Launches tests without awaiting outcomes; operates in a fire-and-forget manner.
    async: false

saucectl-run-action's People

Contributors

Stargazers

Watchers

Forkers

friggahel approved-3rd-party-actions freshactions acm0x

saucectl-run-action's Issues

Playwright Custom Reporter not running when integrating with SauceLabs

I have a Custom Reporter that I'm running for my Playwright tests.
It is defined in the Playwright configuration file and I'm feeding it to the .sauce/config.yml for SauceLabs to use.

When running the Playwright tests on my local without SauceLabs, the reporter is properly fired off and running as expected.
However, when I run the Playwright tests against SauceLabs using the saucectl CLI and the saucectl-run-action GitHub Action, it is not working and seems to be using another reporter.

I see that there is a reporter object in the Playwright SauceLabs Configuration YAML:
https://docs.saucelabs.com/web-apps/automated-testing/playwright/yaml/#reporters

Could this be what is overriding my Custom Reporter?

My Custom Reporter ./testrail/reporter/testrail.reporter.ts looks something like:

import { Reporter } from '@playwright/test/types/testReporter';

class TestRailReporter implements Reporter {
  
  onBegin(config, suite): void {
    // logic
  }

  onTestBegin(test, result): void {
    // logic
  }

  onTestEnd(test, result): void {
    // logic
  }

  onEnd(result): void | Promise<void> {
    // logic
  }
}

export default TestRailReporter;

In my playwright.sauce.config.ts, I have the following reporter being used:

const config: PlaywrightTestConfig = {
  ...
  reporter: './testrail/reporter/testrail.reporter.ts',
  ...
}

In my .sauce/config.yml, I have the following:

...
playwright:
  version: package.json
  configFile: playwright.sauce.config.ts
...

Any feedback would be greatly appreciated as I really need this Custom Reporter to be executed while running my Playwright tests in SauceLabs.

Thanks.

Running saucectl-run-action in GitHub Actions for Playwright tests using "env" in the YAML configuration is not working properly for "macOS"

I am currently using the saucectl-run-action GitHub Action to run my Playwright tests in SauceLabs.
I am passing env variables from the GitHub Actions like so:

 - name: Run saucectl with Playwright tests
   uses: saucelabs/saucectl-run-action@v2
   with:
      sauce-username: ${{ secrets.SAUCE_USERNAME }}
      sauce-access-key: ${{ secrets.SAUCE_ACCESS_KEY }}
      region: ${{ secrets.SAUCE_REGION }}
      config-file: .sauce/config.yml
      concurrency: 10
   env: |
      NODE_ENV=dev # This is just an example value.
      ...

All my tests are running with the environment variables perfectly fine against Windows machines in SauceLabs.
However, when I run the same setup against macOS, the environment variables are not being passed in correctly. Even the same test suite is failing for macOS, while it passes for Windows. Here is the suites portion of the .sauce/config.yml:

suites:
  - name: "FOOBAR - Firefox"
    platformName: "Windows 11"
    screenResolution: "1440x900"
    testMatch: [ tests\/foobar\/(.*).ts ]
    params:
      browserName: "firefox"
      project: "firefox"

  - name: "FOOBAR - MacOS Chromium"
    platformName: "macOS 12"
    screenResolution: "1440x900"
    testMatch: [ tests\/foobar\/(.*).ts ]
    params:
      browserName: "chromium"
      project: "chromium"

While debugging the .sauce/config.yml, I found something very interesting. I tried the following:

env:
  environment: $NODE_ENV # should print "dev" from the GitHub Action snippet above - "dev" for macOS as expected
  Environment: $NODE_ENV # should print "dev" from the GitHub Action snippet above - "undefined" for macOS
  NODE_ENV: $NODE_ENV # should print "dev" from the GitHub Action snippet above - "undefined" for macOS
  ...

When running the above .sauce/config.yml against the Windows machine, the expected "dev" being printed for all environment variables above worked as expected.
However, when running the above .sauce/config.yml against the macOS, it only prints the expected "dev" value for process.env.environment (lower-case) and NOT process.env.Environment and process.env.NODE_ENV.
I am not sure why the values would be different as they are all getting their values from the same source - $NODE_ENV from the GitHub Workflow file. It seems like macOS is only working for environment variables that are lower-case.

This issue seems to be specific to the GitHub Action as it does not occur when I run it from my local. When I run the following command, for both Windows and macOS it works:

npx saucectl run -e  NODE_ENV=dev

Thanks!

Add some contribution guidelines

To complete the setup it would be good to have some contribution guidelines explaining what kind of contributions you would like to see / accept. Feel free to copy&paste them from other projects.

Failure with "RequestError [HttpError]: fetch failed" when running UI tests

Expected Behavior

Action does not fail with code 500 or retries automatically when failed to fetch (for example 3 retries with backoff).

Actual Behavior

Starting UI tests via saucectl-run-action fails once per ~15 runs with the following message:

/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:7814
    throw new requestError.RequestError(error.message, 500, {
          ^

RequestError [HttpError]: fetch failed
    at /home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:7814:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async selectCompatibleVersion (/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:16408:20)
    at async saucectlInstall (/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:16428:19)
    at async run (/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:16763:9) {
  status: 500,
  request: {
    method: 'GET',
    url: 'https://api.github.com/repos/saucelabs/saucectl/releases',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-rest.js/19.0.4 octokit-core.js/4.0.5 Node.js/20.8.1 (linux; x64)',
      authorization: 'token [REDACTED]'
    },
    request: { hook: [Function: bound bound register] }
  }
}

Node.js v20.8.1

We run the action the following way:

      - name: Run tests
        uses: saucelabs/saucectl-run-action@v4
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Steps to Reproduce the Problem

In our case - run multiple jobs (15+) in parallel, at least one will fail with this issue

Specifications

Version: saucectl-run-action@v4
Platform: Ubuntu 22.04

RequestError [HttpError]: API rate limit exceeded

Since last week we are starting to get some API rate limit exceeded errors when using this action.

This is the error:

Run saucelabs/saucectl-run-action@v2
/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:7761
      const error = new requestError.RequestError(toErrorMessage(data), status, {
                    ^

RequestError [HttpError]: API rate limit exceeded for xxx.xxx.xxx.xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)
    at /home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:7761:21
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async selectCompatibleVersion (/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:16336:22)
    at async saucectlInstall (/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:16351:21)
    at async run (/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:16676:10) {
  status: 403,
  response: {
    url: 'https://api.github.com/repos/saucelabs/saucectl/releases',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
      connection: 'close',
      'content-length': '276',
      'content-security-policy': "default-src 'none'; style-src 'unsafe-inline'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Mon, 23 Jan 2023 10:33:45 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'Varnish',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-media-type': 'github.v3; format=json',
      'x-ratelimit-limit': '60',
      'x-ratelimit-remaining': '0',
      'x-ratelimit-reset': '1674471352',
      'x-ratelimit-resource': 'core',
      'x-ratelimit-used': '60',
      'x-xss-protection': '1; mode=block'
    },
    data: {
      message: "API rate limit exceeded for xxx.xxx.xxx.xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
      documentation_url: 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'
    }
  },
  request: {
    method: 'GET',
    url: 'https://api.github.com/repos/saucelabs/saucectl/releases',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-rest.js/19.0.4 octokit-core.js/4.0.5 Node.js/16.13.0 (linux; x64)'
    },
    request: { hook: [Function: bound bound register] }
  }
}

The error states that Authenticated requests get a higher rate limit, but I think we are already using authenticated requests or, at least this is how we are starting the Action:

      - name: Run PR Android test
        uses: saucelabs/saucectl-run-action@v2
        with:
          sauce-username: ${{ secrets.SAUCE_USERNAME }}
          sauce-access-key: ${{ secrets.SAUCE_ACCESS_KEY }}
          testing-environment: ''
          select-suite: 'PR_TEST'

Are we doing something wrong? How is it possible that we are already hitting the API limit? And why the error is stating that Authenticated requests get a higher rate limit?

Automate Release Process

To avoid the bus factor it would be good to have automated releases or ones triggered through GitHub Actions. Similar to other packages. You can find an OSS account for publishing in our 1Password app. Releases should ideally auto-generate changelogs. I found release-it being very useful for that.

Feature request: Set saucelabs-cli stdout/stderr as gh-action output

Would it be possible to set the output of the saucelabs-cli as the output of the github action as shown here?

My usecase is that I want to access the output and analyze the logs in the following step to decide if I want to fail the job or not, because there might be failures due to a dependency that I have no control over and therefore I don't want to consider this job failing.

arm64 support

There's no better time to introduce new architecture into the Saucectl Action!

We ran builds on cloud-based arm64 Macs and got the "Undefined: browser_download_url" error. The reason for this is that the arm64 architecture is not recognized by the run-action, though the underlying saucectl release has it available.

I've added a small fix to alleviate the immediate problem of the availability of ctl on arm64 mac runners, but from what I understand, the arm64 filename is not following the naming convention for 32-bit and 64-bit here. If that was not intentional, then I'll update PR with new name.

PR: #26

Node12 usage is deprecated

Now that Github Actions enforces node 16, it needs to be taken into use here.

Error: Unable to resolve action `saucelabs/saucectl-run-action@v2`, unable to find version `v2`

I tried updating the version of the action in my workflow from v1 to v2 to get rid of nodejs compatibility warnings but now the workflow fails with:

Error: Unable to resolve action saucelabs/saucectl-run-action@v2, unable to find version v2