c-hive / gha-remove-artifacts Goto Github PK
View Code? Open in Web Editor NEWGitHub Action to customize artifact cleanup
License: MIT License
GitHub Action to customize artifact cleanup
License: MIT License
This affects any other (often implicit!) use of the GITHUB_TOKEN
within the same repo workflows: actions/runner-images#1740 (comment).
Looking at the log https://github.com/ansible/pylibssh/runs/1232157025?check_suite_focus=true and many similar ones before, it seems like the action keeps re-fetching all the Check Runs that's been cleaned up before.
I think the fix could be having some setting like "don't go deeper than X workflows back, or Y days back".
Otherwise, the script will keep running until the timeout-minutes
is reached.
In addition, an improvement can be done, that if the error is because of quota exhausted, the scripts end without more re-tries.
Run c-hive/gha-remove-artifacts@v1
Maximum artifact age: 1 second ( created before 2020-12-22T03:06:20+00:00 )
Skipping recent artifact (id: 32210712, name: main.pdf).
Skipping recent artifact (id: 32209832, name: main.pdf).
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/369426095/artifacts?per_page=100, number of total global retries: 0
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/369385267/artifacts?per_page=100, number of total global retries: 1
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/369384675/artifacts?per_page=100, number of total global retries: 2
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/369384676/artifacts?per_page=100, number of total global retries: 3
Retrying after 2233 seconds!
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/369385262/artifacts?per_page=100, number of total global retries: 4
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/367763523/artifacts?per_page=100, number of total global retries: 5
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/367763521/artifacts?per_page=100, number of total global retries: 6
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/367228857/artifacts?per_page=100, number of total global retries: 7
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/367228856/artifacts?per_page=100, number of total global retries: 8
Retrying after 2233 seconds!
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/367219273/artifacts?per_page=100, number of total global retries: 9
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/358928910/artifacts?per_page=100, number of total global retries: 10
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/367219272/artifacts?per_page=100, number of total global retries: 11
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/281637202/artifacts?per_page=100, number of total global retries: 12
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/358928909/artifacts?per_page=100, number of total global retries: 13
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/358920131/artifacts?per_page=100, number of total global retries: 14
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/358920130/artifacts?per_page=100, number of total global retries: 15
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/281637203/artifacts?per_page=100, number of total global retries: 16
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/279989747/artifacts?per_page=100, number of total global retries: 17
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/279989748/artifacts?per_page=100, number of total global retries: 18
Retrying after 2233 seconds!
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/279979882/artifacts?per_page=100, number of total global retries: 19
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/279753471/artifacts?per_page=100, number of total global retries: 20
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/279753470/artifacts?per_page=100, number of total global retries: 21
Retrying after 2233 seconds!
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/279979881/artifacts?per_page=100, number of total global retries: 22
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/259416276/artifacts?per_page=100, number of total global retries: 23
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/259416271/artifacts?per_page=100, number of total global retries: 24
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/259144589/artifacts?per_page=100, number of total global retries: 25
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/259144590/artifacts?per_page=100, number of total global retries: 26
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/257299798/artifacts?per_page=100, number of total global retries: 27
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/257299799/artifacts?per_page=100, number of total global retries: 28
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/257263229/artifacts?per_page=100, number of total global retries: 29
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/257263228/artifacts?per_page=100, number of total global retries: 30
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/249299537/artifacts?per_page=100, number of total global retries: 31
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/245602670/artifacts?per_page=100, number of total global retries: 32
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/215573331/artifacts?per_page=100, number of total global retries: 33
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/245602669/artifacts?per_page=100, number of total global retries: 34
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/215573330/artifacts?per_page=100, number of total global retries: 35
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/214789373/artifacts?per_page=100, number of total global retries: 36
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/214789372/artifacts?per_page=100, number of total global retries: 37
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/249299536/artifacts?per_page=100, number of total global retries: 38
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/215547805/artifacts?per_page=100, number of total global retries: 39
Retrying after 2233 seconds!
Request quota exhausted for request DELETE /repos/{owner}/{repo}/actions/artifacts/{artifact_id}, number of total global retries: 40
Retrying after 2233 seconds!
Request quota exhausted for request GET https://api.github.com/repos/wanzhiguo/blockchain-por/actions/runs/215547806/artifacts?per_page=100, number of total global retries: 41
Retrying after 2232 seconds!
Error: The operation was canceled.
thank for your helpful action, can you add support for keeping a configurable number of recently published artifacts?
For example:
steps:
- name: Remove old artifacts
uses: c-hive/gha-remove-artifacts@v1
with:
age: '1 month'
keep-recent: 7 # keep the number of recently published artifacts
skip-tags: true
so that even if those 7 recently published artifacts are older than 1 month, the action will not delete them.
I've created this yaml file:
name: "Delete old artifacts"
on:
schedule:
- cron: "0 * * * *" # every hour
jobs:
delete-artifacts:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: c-hive/gha-remove-artifacts@v1
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_ACCESS_TOKEN }}
age: '7 days'
skip-tags: true
When the job runs, I get this error:
Error while requesting tags: RequestError [HttpError]: Not Found
at /home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:11376:23
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async Job.doExecute (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:14132:18) {
name: 'HttpError',
status: 404,
headers: {
'access-control-allow-origin': '*',
'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
connection: 'close',
'content-encoding': 'gzip',
'content-security-policy': "default-src 'none'",
'content-type': 'application/json; charset=utf-8',
date: 'Mon, 30 Mar 2020 22:05:16 GMT',
'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
server: 'GitHub.com',
status: '404 Not Found',
'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
'transfer-encoding': 'chunked',
vary: 'Accept-Encoding, Accept, X-Requested-With',
'x-accepted-oauth-scopes': '',
'x-content-type-options': 'nosniff',
'x-frame-options': 'deny',
'x-github-media-type': 'github.v3; format=json',
'x-github-request-id': '0401:29F9:8DB5F:EBC3D:5E826D1C',
'x-oauth-scopes': 'workflow',
'x-ratelimit-limit': '5000',
'x-ratelimit-remaining': '4996',
'x-ratelimit-reset': '1585606782',
'x-xss-protection': '1; mode=block'
},
request: {
method: 'GET',
url: 'https://api.github.com/repos/***/test-repo/tags?per_page=100&ref=tags',
headers: {
accept: 'application/vnd.github.v3+json',
'user-agent': 'octokit-action.js/2.0.0 octokit-core.js/2.4.2 Node.js/12.13.1 (Linux 5.0; x64)',
authorization: 'token [REDACTED]'
},
request: { hook: [Function: bound bound register] }
},
documentation_url: 'https://developer.github.com/v3/repos/#list-tags'
}
(node:2466) UnhandledPromiseRejectionWarning: TypeError: (s || "").replace is not a function
at escapeData (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:3435:10)
at Command.toString (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:3429:35)
at issueCommand (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:3392:30)
at Object.issue (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:3396:5)
at error (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:5557:15)
at Object.setFailed (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:5531:5)
at /home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:680:8
(node:2466) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2466) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Maximum artifact age: 7 days ( created before 2020-03-23T22:05:15+00:00 )
I have this action setup to run on a cron schedule every day, but it's been failing with an HTTP error recently:
https://github.com/beeb/awsbck/actions/runs/4238612972/jobs/7365872273
About versioning see:
v1.0.0
and v1
tags have to be created.
We should update the project's readme since it's cloned from a bare-bone project. Apart from the Conventions
and Licence
chapters, I'd propose the followings:
How the hell do I get past this? Frustrating
Errors
Request quota exhausted for request GET https://api.github.com/repos/dschinkel/we-do-tdd/actions/runs?per_page=100, number of total global retries: 1
Retrying after 0 seconds!
Error: HttpError: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
(node:1535) UnhandledPromiseRejectionWarning: HttpError: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
at /home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:[11](https://github.com/dschinkel/we-do-tdd/runs/6875740490?check_suite_focus=true#step:2:11)413:23
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async Job.doExecute (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:14169:18)
(node:1535) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1535) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:1535) UnhandledPromiseRejectionWarning: HttpError: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
at /home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:114[13](https://github.com/dschinkel/we-do-tdd/runs/6875740490?check_suite_focus=true#step:2:13):23
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
Request quota exhausted for request DELETE /repos/{owner}/{repo}/actions/artifacts/{artifact_id}, number of total global retries: 70
Retrying after 3424 seconds!
gha-remove-artifacts
is retrying way too many times and then causing this problem. How can we limit retries with this plugin?
Workflow
name: Remove-Old-Github-Action-Artifacts
on:
push:
branches:
- master
schedule:
# Every day at 4am
- cron: '0 4 * * *'
jobs:
remove-old-artifacts:
runs-on: ubuntu-latest
timeout-minutes: 2
steps:
- name: Remove artifacts older than 30 seconds
uses: c-hive/gha-remove-artifacts@v1
with:
age: '60 seconds'
See the outcome of actions/toolkit#334
This might be required for using plugins (and therefore #3)
Currently, the action is being tested by running it in the CI. The CI creates artifacts which are then removed by subsequent runs, see:
gha-remove-artifacts/.github/workflows/ci.yml
Lines 48 to 79 in bf5f1f9
Ideally, the action could be run locally (provided a personal access token) on a selected repo for testing purposes (make sure it's a dry run and doesn't actually remove artifacts).
This could either be achieved by conditions and a local env setup, or perhaps via https://github.com/nektos/act.
There's an extra branch because the pages aren't normalized for some reason:
Lines 91 to 105 in 7dcce22
Waiting for the output of this on-going discussion: https://github.com/octokit/rest.js/issues/1632
with:
age: '1 month'
# Optional inputs
# skip-tags: true
skip-recent: 5
the skip-recent
options specific the number of artifacts, rather than commits. So, suspose my action generates more than one artifacts per commit (even more, not a constant number. It depends. ), while I want to keep recently 3 commits' artifacts, a new option which might be called skip-recent-commit
might be handy.
Although there're separate keys per repo and 5k available requests at a time, so hopefully a daily build will work just fine. Still, would be nice to at least get proper errors.
https://github.com/octokit/plugin-throttling.js/
actions/toolkit#106 and https://github.com/actions/toolkit/blob/fc9958ec6a57d1b08e9a387c49971faecd0f0405/packages/github/src/github.ts#L18 might be useful on how to use plugins.
Step 1:
skip-tags
option for now (see #1) (resolved by #7)age
option must workrun-action
step in the CI works without any errorsStep 2:
for await (const { data: ... } of octokit.paginate.iterator
syntax to avoid nesting and make execution more dynamicAPI resources:
My task started failing on October 21. Is there some limit I need to be aware of?
Log:
(This continues until the time limit is reached)
Run c-hive/gha-remove-artifacts@v1
Maximum artifact age: 1 month ( created before 2020-09-29T01:34:46+00:00 )
Abuse detected for request GET https://api.github.com/repos/<org>/<repo>/actions/runs/247833771/artifacts?per_page=100, retry count: 0
Retrying after 60 seconds!
Abuse detected for request GET https://api.github.com/repos/<org>/<repo>/actions/runs/247824105/artifacts?per_page=100, retry count: 1
Retrying after 60 seconds!
Abuse detected for request GET https://api.github.com/repos/<org>/<repo>/actions/runs/247824104/artifacts?per_page=100, retry count: 2
Retrying after 60 seconds!
YML file:
name: Remove old artifacts
on:
schedule:
# Every day at 1am
- cron: '0 1 * * *'
jobs:
remove-old-artifacts:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Remove old artifacts
uses: c-hive/gha-remove-artifacts@v1
with:
age: '1 month'
# Optional inputs
# skip-tags: true
# skip-recent: 5
In case requests fail, at least an error code should be visible. See also #42
To figure out that "14 days" would be acceptable input, I needed to go into the java script and see where the configuration is applied.
It is clear from the README.md that "1 month" is acceptable, but no discussion is present on what other formats are available.
I believe it is pretty standard case when GITHUB_TOKEN
will be defined globally for all steps, but currently this plugin will fail in such cases with this error:
Error: Error: [@octokit/auth-action] `GITHUB_TOKEN` variable is set on both `env:` and `with:`. Use either the one or the other. See https://github.com/octokit/auth-action.js#createactionauth
It seems that making `GITHUB_TOKEN optional input parameter to fix the issue
Regards
Hey folks, I'm fairly new to workflows and wanna understand why running this workflow couldn't purge all the artifacts from my github quota?
Screenshot of the workflow which gets initated on [push] to master
Here's my workflow file :
name: Remove old artifacts
on:
push:
branches: [master]
jobs:
remove-old-artifacts:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Remove old artifacts
uses: c-hive/gha-remove-artifacts@v1
with:
age: "1 day"
and here's a screenshot of my github storage space:
Can y'all please help me figure out what exactly am I doing wrong?
AFAICT if there's an error somewhere in the code then it won't be reflected on the CI's interface currently(i.e. the run-action
job doesn't reflect the fail with the red X icon even if e.g. there's an unhandled promise rejection). I'd propose using core.setFailed()
to explicitly mark a run as failed. Perhaps there are some other debug utility functions we may take advantage of.
It would be nice to a skip-recent variant which keeps a number of artifacts per branch. That way, you could create multiple branches without invalidating the artifacts of the others branches.
Action is failing consistently for the past few weeks, regardless of when it's ran or what other tasks are ongoing.
Skipping recent artifact (id: 216039564, name: uberjar).
Error: HttpError: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
(node:1556) UnhandledPromiseRejectionWarning: HttpError: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
at /home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:11413:23
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async Job.doExecute (/home/runner/work/_actions/c-hive/gha-remove-artifacts/v1/dist/index.js:14169:18)
The specific workflow we're using:
name: Remove old artifacts
on:
schedule:
- cron: '0 22 * * *'
jobs:
remove-old-artifacts:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Remove old artifacts
uses: c-hive/gha-remove-artifacts@v1
with:
age: '1 day'
skip-tags: true
skip-recent: 5
Perhaps there has been a change to secondary rate limits?
Since artifacts are automatically cleaned up after 3 months, the pagination of workflow runs can be stopped after this timestamp is reached.
Add an optional parameter that disables the removal of release artifacts. Turned out to be more difficult than I thought as you cannot seem to get tags based on a commit hash:
We could ask for all tags can keep a list of SHAs to compare. E.g. https://api.github.com/repos/c-hive/team-contribution-calendar/tags
Until now the action was working perfectly, but today we got the following error when it tried to run in our scheduled workflow:
##[error]Error: [@octokit/auth-action] 'GITHUB_ACTION' environment variable is not set. @octokit/auth-action is meant to be used in GitHub Actions only.
Our configuration:
uses: c-hive/gha-remove-artifacts@v1
with:
age: '14 days'
skip-tags: true
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.