saucelabs / saucectl-run-action Goto Github PK
View Code? Open in Web Editor NEWGitHub Action for running saucectl test suites.
Home Page: http://opensource.saucelabs.com/saucectl-run-action/
GitHub Action for running saucectl test suites.
Home Page: http://opensource.saucelabs.com/saucectl-run-action/
I tried updating the version of the action in my workflow from v1 to v2 to get rid of nodejs compatibility warnings but now the workflow fails with:
Error: Unable to resolve action saucelabs/saucectl-run-action@v2
, unable to find version v2
I am currently using the saucectl-run-action
GitHub Action to run my Playwright
tests in SauceLabs
.
I am passing env
variables from the GitHub Actions like so:
- name: Run saucectl with Playwright tests
uses: saucelabs/saucectl-run-action@v2
with:
sauce-username: ${{ secrets.SAUCE_USERNAME }}
sauce-access-key: ${{ secrets.SAUCE_ACCESS_KEY }}
region: ${{ secrets.SAUCE_REGION }}
config-file: .sauce/config.yml
concurrency: 10
env: |
NODE_ENV=dev # This is just an example value.
...
All my tests are running with the environment variables perfectly fine against Windows
machines in SauceLabs
.
However, when I run the same setup against macOS
, the environment variables are not being passed in correctly. Even the same test suite is failing for macOS
, while it passes for Windows
. Here is the suites
portion of the .sauce/config.yml
:
suites:
- name: "FOOBAR - Firefox"
platformName: "Windows 11"
screenResolution: "1440x900"
testMatch: [ tests\/foobar\/(.*).ts ]
params:
browserName: "firefox"
project: "firefox"
- name: "FOOBAR - MacOS Chromium"
platformName: "macOS 12"
screenResolution: "1440x900"
testMatch: [ tests\/foobar\/(.*).ts ]
params:
browserName: "chromium"
project: "chromium"
While debugging the .sauce/config.yml
, I found something very interesting. I tried the following:
env:
environment: $NODE_ENV # should print "dev" from the GitHub Action snippet above - "dev" for macOS as expected
Environment: $NODE_ENV # should print "dev" from the GitHub Action snippet above - "undefined" for macOS
NODE_ENV: $NODE_ENV # should print "dev" from the GitHub Action snippet above - "undefined" for macOS
...
When running the above .sauce/config.yml
against the Windows
machine, the expected "dev"
being printed for all environment variables above worked as expected.
However, when running the above .sauce/config.yml
against the macOS
, it only prints the expected "dev"
value for process.env.environment
(lower-case) and NOT process.env.Environment
and process.env.NODE_ENV
.
I am not sure why the values would be different as they are all getting their values from the same source - $NODE_ENV
from the GitHub Workflow file. It seems like macOS
is only working for environment variables that are lower-case.
This issue seems to be specific to the GitHub Action as it does not occur when I run it from my local. When I run the following command, for both Windows
and macOS
it works:
npx saucectl run -e NODE_ENV=dev
Thanks!
To avoid the bus factor it would be good to have automated releases or ones triggered through GitHub Actions. Similar to other packages. You can find an OSS account for publishing in our 1Password app. Releases should ideally auto-generate changelogs. I found release-it being very useful for that.
Since last week we are starting to get some API rate limit exceeded
errors when using this action.
This is the error:
Run saucelabs/saucectl-run-action@v2
/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:7761
const error = new requestError.RequestError(toErrorMessage(data), status, {
^
RequestError [HttpError]: API rate limit exceeded for xxx.xxx.xxx.xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)
at /home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:7761:21
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async selectCompatibleVersion (/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:16336:22)
at async saucectlInstall (/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:16351:21)
at async run (/home/runner/work/_actions/saucelabs/saucectl-run-action/v2/dist/main/index.js:16676:10) {
status: 403,
response: {
url: 'https://api.github.com/repos/saucelabs/saucectl/releases',
status: 403,
headers: {
'access-control-allow-origin': '*',
'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
connection: 'close',
'content-length': '276',
'content-security-policy': "default-src 'none'; style-src 'unsafe-inline'",
'content-type': 'application/json; charset=utf-8',
date: 'Mon, 23 Jan 2023 10:33:45 GMT',
'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
server: 'Varnish',
'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
'x-content-type-options': 'nosniff',
'x-frame-options': 'deny',
'x-github-media-type': 'github.v3; format=json',
'x-ratelimit-limit': '60',
'x-ratelimit-remaining': '0',
'x-ratelimit-reset': '1674471352',
'x-ratelimit-resource': 'core',
'x-ratelimit-used': '60',
'x-xss-protection': '1; mode=block'
},
data: {
message: "API rate limit exceeded for xxx.xxx.xxx.xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
documentation_url: 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'
}
},
request: {
method: 'GET',
url: 'https://api.github.com/repos/saucelabs/saucectl/releases',
headers: {
accept: 'application/vnd.github.v3+json',
'user-agent': 'octokit-rest.js/19.0.4 octokit-core.js/4.0.5 Node.js/16.13.0 (linux; x64)'
},
request: { hook: [Function: bound bound register] }
}
}
The error states that Authenticated requests get a higher rate limit
, but I think we are already using authenticated requests or, at least this is how we are starting the Action:
- name: Run PR Android test
uses: saucelabs/saucectl-run-action@v2
with:
sauce-username: ${{ secrets.SAUCE_USERNAME }}
sauce-access-key: ${{ secrets.SAUCE_ACCESS_KEY }}
testing-environment: ''
select-suite: 'PR_TEST'
Are we doing something wrong? How is it possible that we are already hitting the API limit? And why the error is stating that Authenticated requests get a higher rate limit?
To complete the setup it would be good to have some contribution guidelines explaining what kind of contributions you would like to see / accept. Feel free to copy&paste them from other projects.
There's no better time to introduce new architecture into the Saucectl Action!
We ran builds on cloud-based arm64 Macs and got the "Undefined: browser_download_url" error. The reason for this is that the arm64
architecture is not recognized by the run-action, though the underlying saucectl
release has it available.
I've added a small fix to alleviate the immediate problem of the availability of ctl
on arm64 mac runners, but from what I understand, the arm64 filename is not following the naming convention for 32-bit
and 64-bit
here. If that was not intentional, then I'll update PR with new name.
PR: #26
I have a Custom Reporter that I'm running for my Playwright tests.
It is defined in the Playwright configuration file and I'm feeding it to the .sauce/config.yml
for SauceLabs to use.
When running the Playwright tests on my local without SauceLabs, the reporter is properly fired off and running as expected.
However, when I run the Playwright tests against SauceLabs using the saucectl
CLI and the saucectl-run-action
GitHub Action, it is not working and seems to be using another reporter.
I see that there is a reporter object in the Playwright SauceLabs Configuration YAML:
https://docs.saucelabs.com/web-apps/automated-testing/playwright/yaml/#reporters
Could this be what is overriding my Custom Reporter?
My Custom Reporter ./testrail/reporter/testrail.reporter.ts
looks something like:
import { Reporter } from '@playwright/test/types/testReporter';
class TestRailReporter implements Reporter {
onBegin(config, suite): void {
// logic
}
onTestBegin(test, result): void {
// logic
}
onTestEnd(test, result): void {
// logic
}
onEnd(result): void | Promise<void> {
// logic
}
}
export default TestRailReporter;
In my playwright.sauce.config.ts, I have the following reporter being used:
const config: PlaywrightTestConfig = {
...
reporter: './testrail/reporter/testrail.reporter.ts',
...
}
In my .sauce/config.yml
, I have the following:
...
playwright:
version: package.json
configFile: playwright.sauce.config.ts
...
Any feedback would be greatly appreciated as I really need this Custom Reporter to be executed while running my Playwright tests in SauceLabs.
Thanks.
Action does not fail with code 500 or retries automatically when failed to fetch (for example 3 retries with backoff).
Starting UI tests via saucectl-run-action
fails once per ~15 runs with the following message:
/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:7814
throw new requestError.RequestError(error.message, 500, {
^
RequestError [HttpError]: fetch failed
at /home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:7814:11
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async selectCompatibleVersion (/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:16408:20)
at async saucectlInstall (/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:16428:19)
at async run (/home/github/actions-runner/_work/_actions/saucelabs/saucectl-run-action/v4/dist/main/index.js:16763:9) {
status: 500,
request: {
method: 'GET',
url: 'https://api.github.com/repos/saucelabs/saucectl/releases',
headers: {
accept: 'application/vnd.github.v3+json',
'user-agent': 'octokit-rest.js/19.0.4 octokit-core.js/4.0.5 Node.js/20.8.1 (linux; x64)',
authorization: 'token [REDACTED]'
},
request: { hook: [Function: bound bound register] }
}
}
Node.js v20.8.1
We run the action the following way:
- name: Run tests
uses: saucelabs/saucectl-run-action@v4
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Would it be possible to set the output of the saucelabs-cli as the output of the github action as shown here?
My usecase is that I want to access the output and analyze the logs in the following step to decide if I want to fail the job or not, because there might be failures due to a dependency that I have no control over and therefore I don't want to consider this job failing.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.