actions / actions-sync Goto Github PK

Hello, I want to synchronize an action from an organization to my self-hosted GitHub enterprise. This action repo requires authorization, how can I synchronize it? I don't find a place to set credential.

./actions-sync sync --cache-dir "cache" --destination-token "ghp_12345" --destination-url "https://github.*****.cn" --repo-name "-Shared-Actions/Action_:DevOps/Action_"
pulling -Shared-Actions/Action_ to cache/DevOps/Action_ ...
could not pull -Shared-Actions/Action_, the repository may require authentication or does not exist

Hello,
The release binary for linux_amd64 depends on glibc version 2.32.
This prevent us to run it on Linux distros older then Ubuntu-22.04, RHEL-9, etc.
When we run it we get the error message: actions-sync: /lib64/libc.so.6: version 'GLIBC_2.32' not found (required by actions-sync)

There is a lot of productions systems running older LTS Linux distributions.
Could you please compile your release binary for older Linux distros?

When I sync a repo, all branches are get pulled. This means that all branches consequently get their push event triggered on the (GHES) destination side. For some repos this means you involuntarily/perhaps unknowingly set off potentially quite a large backlog of workflows - the repo I just synced contained ~50 branches, ~50 releases, and ~20 workflows, all on: [push]. I suppose I won't be using my workers for anything else for the rest of today... 😢

Ideas

• Would it make sense to only sync the main branch by default?
• Should there be some way to allow specifying "branches to sync, per repo-name" (i.e. extending the upstream_org/upstream_repo:destination_org/dstination_repo syntax)?

Running:
actions-sync push --cache-dir ./cache --destination-token=XYZ --destination-url=https://ghe-test.mycompany.com --repo-name=actions/setup-go

Yields:
error syncing repository 'actions/setup-go': failed to push to repo: https://ghe-test.mycompany.com/actions/setup-go.git: authorization failed

Note that the token has repo, workflow, and site admin scopes (re-confirmed this multiple times)

When synchronizing a repository with default branch master to GitHub Enterprise Server, the default branch is wrongly set the first branch in the list of available branches after synchronization.

Repositories with default branch main do have the correct default branch after synchronization.

Ideally, the actions-sync should be able to handle any case. Otherwise we will need to adjust the default branch after synchronization.

While we are trying with 32-bit bundle of this actions-sync tool,
it fails in fetching the organization.

Customer facing errors:
error creating github repository actions/setup-node: error creating organization actions: POST https://example.com/api/v3/admin/organizations: 422 Organization name is not available []

error creating github repository synced-actions/setup-ruby: error creating organization synced-actions: POST https://example.com/api/v3/admin/organizations: 404 Not Found []

Internal details when run with extra logs locally:
While fetching the organization, we get the plan

   org.plan.private_repos: 999999999999,

and it exceeds 32 bit INT (2147483647). So fails during unmarshal with below error.
json: cannot unmarshal number 999999999999 into Go struct field Plan.plan.private_repos of type int

If we try with 64 bit bundle (gh_202009231612_linux_amd64.tar.gz), it's working fine.

Running the actions-sync tool from a CronJob in K8s always results in an exception, while running the exact same command on local machine (Macbook Pro) always succeeds.

The tried command:

/bin/actions-sync sync  --cache-dir=/tmp/ --destination-token <token> --source-url=https://github.com --destination-url=https://github-<enterprise-url> --repo-name-list "Azure/setup-kubectl:actions/setup-kubectl"

Results in (inside cronjob):

fetching * refs for Azure/setup-kubectl ...
syncing `actions/setup-kubectl`
error creating github repository `actions/setup-kubectl`: error creating organization actionsss: POST https://github-<enterprise-url>/api/v3/admin/organizations: 422 Validation Failed [{Resource:Organization Field:login Code:custom Message:login is not available} {Resource:Organization Field:admins Code:missing_field Message:}]

On Macbook machine:

fetching * refs for Azure/setup-kubectl ...
syncing `actions/setup-kubectl`
successfully synced `actions/setup-kubectl`

Any idea what causes this discrepancy?

Running the actions-sync tool from a CronJob in K8s always results in an exception, while running the exact same command on local machine (Macbook Pro) always succeeds.

Command I tried:

/bin/actions-sync sync --cache-dir=/tmp/ --destination-token <token> --source-url=https://github.com --destination-url=https://github-<enterprise-url> --repo-name-list "Azure/setup-kubectl:actions/setup-kubectl"

Results (inside CronJob):

fetching * refs for Azure/setup-kubectl ...
syncing `actions/setup-kubectl`
error creating github repository `actions/setup-kubectl`: error creating organization actions POST https://github-<enterprise-url>/api/v3/admin/organizations: 422 Validation Failed [{Resource:Organization Field:login Code:custom Message:login is not available} {Resource:Organization Field:admins Code:missing_field Message:}]

On Macbook machine with exact same command:

fetching * refs for Azure/setup-kubectl ...
syncing `actions/setup-kubectl`
successfully synced `actions/setup-kubectl`

One thing I noticed is that it succeeds on the CronJob only when the organisation and repo are not existing before, so it creates them and works, but if org/repo are existing it fails.

Used version: https://github.com/actions/actions-sync/releases/tag/v202009231612

Any idea what causes this discrepancy?

Right now, if an organization does not exist on the GHES appliance and we try to sync an action from GitHub.com to the GHES appliance, it will error out with a 404 Not Found since the user or organization does not exist. To remediate this, we can use the GHES API to create an organization with the exact same name as the owner of the GitHub.com repository. Even if the owner of the GitHub.com repository is a regular user and not an organization, we should still create it as an organization since its easier for individual users to create organizations via the API than other user accounts.

What is the required scope to run actions-sync for destination fine-grained PAT?

We're manually importing actions into our GHES, so the first thing i did as a good CI/CD user is try to create a repository where I could maintain the list of actions we sync and periodically trigger the update.

I found the first thing i had to do is download and execute this manually. Surely this is a good candidate for an action?

You'd obviously then have to bootstrap that action, but then it could keep itself up to date.

I believe actions-sync can only download actions from public repos in github.com. What was the reason to only include public repos and not private repos? It would be great to hear if you are considering private repos in your roadmap.

Running actions-sync version always returns GHES Actions Sync v0.2. Instead, it needs to return the actual version. We can do this by setting ldflags in the .goreleaser.yml and then using those values in the verision CLI command.

https://goreleaser.com/customization/builds/

Currently actions-sync tool doesn't support the syncing the release assets.
This is tracking issue to see if more customer upvotes, then we will prioritise accordingly.

We have seen some customers wanted to sync the release assets as well into their GHES instance.
From technical standpoint, we need to make lot of effort to support granular asset downloads. looking at the current library we use (github.com/go-git/go-git) and assets are considered non-native to git and significant changes needed.

from github.com

from GHES, when synced, we don't see release assets

In every release you provide a corresponding gh_202211070205_checksums.txt which includes checksums of each tar.gz file.

But with which algorithm do you calculate them?
COuld you maybe document that in the Readme?

Thanks in advance.

Thank you 🙇‍♀ for wanting to create an issue in this repository. Before you do, please ensure you are filing the issue in the right place. Issues should only be opened on if the issue relates to code in this repository.

• If you have found a security issue please submit it here
• If you have questions about writing workflows or action files, then please visit the GitHub Community Forum's Actions Board
• If you are having an issue or question about GitHub Actions then please contact customer support

If your issue is relevant to this repository, please delete this text and continue to create this issue. Thank you in advance.

For those using action-sync for GHES with large repo-name-list-files, it would be nice to organize the file with comments.

Hi
It will be nice to have option to disable actions inside synced action, because mostly they won't work after sync

We are currently using actions-sync to keep our GHES actions org in-sync with the public actions org.

Could we please have a flag that allows us to just sync the entire actions org so that we do not need to enumerate every repository in the actions org?

At the moment our Actions workflow looks like:

name: actions-sync

on:
  workflow_dispatch:
    inputs:
      token:
        description: 'Personal access token with repo permissions to GitHub.example.com'
        required: true

jobs:
  actions-org:
    runs-on: [self-hosted, ubuntu-latest]
    steps:
      - name: setup
        run: |
          wget https://github.com/actions/actions-sync/releases/download/v202009231612/gh_202009231612_linux_amd64.tar.gz
          tar -xf gh_202009231612_linux_amd64.tar.gz
      - name: sync
        run: |
          mkdir -p cache
          # can't sync actions/cache, actions/upload-release-asset as it has "been retired and cannot be reused"
          ./bin/actions-sync sync \
            --cache-dir "cache" \
            --destination-token "${{ github.event.inputs.token }}" \
            --destination-url "https://github.example.com" \
            --repo-name-list "\
                              actions/.github,\
                              actions/actions-sync,\
                              actions/add-to-project,\
                              actions/boost-versions,\
                              actions/checkout,\
                              actions/container-action,\
                              actions/container-toolkit-action,\
                              actions/create-release,\
                              actions/delete-package-versions,\
                              actions/deploy-pages,\
                              actions/download-artifact,\
                              actions/example-services,\
                              actions/first-interaction,\
                              actions/github,\
                              actions/github-script,\
                              actions/go-versions,\
                              actions/hello-world-docker-action,\
                              actions/hello-world-javascript-action,\
                              actions/heroku,\
                              actions/http-client,\
                              actions/humans.txt,\
                              actions/javascript-action,\
                              actions/jekyll-build-pages,\
                              actions/labeler,\
                              actions/node-versions,\
                              actions/publish-action,\
                              actions/python-versions,\
                              actions/runner,\
                              actions/setup-dotnet,\
                              actions/setup-elixir,\
                              actions/setup-go,\
                              actions/setup-haskell,\
                              actions/setup-java,\
                              actions/setup-node,\
                              actions/setup-python,\
                              actions/setup-ruby,\
                              actions/stale,\
                              actions/starter-workflows,\
                              actions/toolkit,\
                              actions/typescript-action,\
                              actions/upload-artifact,\
                              actions/versions-package-tools,\
                              actions/virtual-environments,\
                              actions/virtual-environments-packages"

👋 This issue is to track the move over to using main as the default branch for this repo. We’d love your team's help in completing this transition.

Do not remove your old default branch, customers are going to be using it. We will be sending messages out about these changes, but if you want to message in your repository, that's fine as well.

• Create a main branch.
• You might need to rebase any pull requests you want to merge before changing the default branch.
• Change the default branch in settings to main.
• Update any documentation in this repo to refer to the new branch name, although using the version tag is still preferred.
• Check that this Action works correctly for users who have a repository with a custom default branch name.
• Close this issue and celebrate 🎉

We are aiming to complete this work by July 17th July 24th.

Quick Note - This whole thing could simply be that at one time I thought that actions-sync created the organization if it did not exist, then created the repository, again if it did not exist, then finally push the repo. Just to REALLY sort things out, I event did the command: gh-org-admin-promote -u ghe-admin -o actions.

Error

json: cannot unmarshal array into Go value of type github.Repository

Command

actions-sync sync --cache-dir 'c:\temp' --destination-token 'abc123' --destination-url 'http://<serverFQDN>' --repo-name docker/bake-action

Environment

Windows 11 23H2 (OS Build 22631.3007)
PowerShell 7.4.1
Go v1.21.6
Chocolatey v2.2.2

Note: I am using this to test the process. I have a classic token which as the following rights: admin:enterprise, admin:org, repo, site_admin, workflow

Narrative

I have the latest version of actions-sync saved in my c:\program files\dotnet directory. This directory is in the path. I am using PowerShell 7.4.1 to execute the command as noted above. I am attempting to sync from my Windows workstation to a GitHub Version 3.10.5 instance.

Expectations

sync the repository of my choice from https://github.com to my instance using actions-sync and the sync command. If organization does not exist, the tool creates the organization then creates the repository and pushes the fetched repository to the newly created organization and repository. Could be me, but I thought earlier versions did this?

Steps and Outcomes

Note: I am using this to test the process. I have a classic token which as the following rights: admin:enterprise, admin:org, repo, site_admin, workflow

1. Issue Action Sync Command
   • command: actions-sync sync --cache-dir 'c:\temp' --destination-token 'abc123' --destination-url 'http://myserver' --repo-name docker/bake-action
   • result: error creating github repository docker/bake-action: error creating organization docker: GET https://myserver/api/v3/admin/organizations: 404 Not Found []

At this point, I go and create the docker organization in my instance and re-issue the command

1. Issue Action Sync Command
   • command: actions-sync sync --cache-dir 'c:\temp' --destination-token 'abc123' --destination-url 'http://myserver' --repo-name docker/bake-action
   • result: error creating github repository docker/bake-action: error creating repository docker/bake-action: json: cannot unmarshal array into Go value of type github.Repository

At this point I create the bake-action repository in the docker organization and re-issue the command

1. Issue Action Sync Command
   • command: actions-sync sync --cache-dir 'c:\temp' --destination-token 'abc123' --destination-url 'http://myserver' --repo-name docker/bake-action
   • result:
     fetching * refs for docker/bake-action ...
     not using impersonation for the requests
     syncing docker/bake-action
     Existing repo docker/bake-action
     successfully synced docker/bake-action

We are running actions-sync in a GitHub workflow and are using an application access token instead of a PAT (https://github.com/peter-murray/workflow-application-token-action).
Unfortunately even though the target organization exists, the actions-sync fails, because it can't call client.Users.get(), because an application cannot access that endpoint (

Would you accept a PR to alter the getOrCreateGithubRepo function (

Related: #14

In commands where a source/destination URL can be present, clients can provide any support git url. When using HTTP the HTTP protocol, if https is not used then clients will encounter errors such as those reported in #96.

To prevent these kinds of issues, we should force the use of https for all flags (when provided) and return an error back to clients if URLs use http

We run actions-sync with a service account, the account has permissions on the main actions repo but we also use it for syncing actions for teams.

What we tried to do is have someone on the team create the empty repo in the desired org, provide the service account write access to the repo and then run sync. However this fails with messages like:

error creating github repository `XXX/YYY`: error creating repository XXX/YYY: POST https://org/api/v3/orgs/XXX/repos: 403 You need admin access to the organization before adding a repository to it. []
##[error]Process completed with exit code 1.

Because the logic is trying to use that token to query if the repo exists via the github api, which it doesn't have permissions to do.

To reduce the permissions needed here the first check should really be direct to the repo to see if we can simply use it rather than have to check and create it.

Hi!
Thank you for always providing excellent service.
I got this error when I was syncing.

% go run . sync --cache-dir ./tmp --destination-token $GHE_TOKEN --destination-url https://ours.ghe.address --repo-name peter-evans/create-pull-request:example/create-pull-request
fetching * refs for peter-evans/create-pull-request ...
syncing `exampe/create-pull-request`
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12f25f7]

goroutine 1 [running]:
github.com/go-git/go-git/v5/plumbing/protocol/packp.(*ReportStatus).Error(0x0, 0xc0000bc000, 0x168d470)
        /Users/pyama/src/github.com/actions/actions-sync/vendor/github.com/go-git/go-git/v5/plumbing/protocol/packp/report_status.go:31 +0x37
github.com/go-git/go-git/v5.(*Remote).PushContext(0xc0000c8ca8, 0x168f350, 0xc0000bc000, 0xc0003303c0, 0x0, 0x0)
        /Users/pyama/src/github.com/actions/actions-sync/vendor/github.com/go-git/go-git/v5/remote.go:189 +0x825
github.com/actions/actions-sync/src.syncWithCachedRepository(0x168f350, 0xc0000bc000, 0xc000523c78, 0xc000278000, 0xc0001936b0, 0x27, 0x168e0a0, 0x199ebd0, 0x0, 0x0)
        /Users/pyama/src/github.com/actions/actions-sync/src/push.go:206 +0x2ef
github.com/actions/actions-sync/src.PushWithGitImpl(0x168f350, 0xc0000bc000, 0xc000523c78, 0x7ffeefbfee92, 0x43, 0xc000196280, 0x168e0a0, 0x199ebd0, 0x8, 0xc0000de090)
        /Users/pyama/src/github.com/actions/actions-sync/src/push.go:110 +0x2d4
github.com/actions/actions-sync/src.PushManyWithGitImpl(0x168f350, 0xc0000bc000, 0xc005013c78, 0xc000593f30, 0x1, 0x1, 0xc000196280, 0x168e0a0, 0x199ebd0, 0x199ebd0, ...)
        /Users/pyama/src/github.com/actions/actions-sync/src/push.go:81 +0xb4
github.com/actions/actions-sync/src.Push(0x168f350, 0xc0000bc000, 0xc0005ddc78, 0x0, 0x0)
        /Users/pyama/src/github.com/actions/actions-sync/src/push.go:76 +0x16d
github.com/actions/actions-sync/src.Sync(0x168f350, 0xc0000bc000, 0x1970360, 0x0, 0x0)
        /Users/pyama/src/github.com/actions/actions-sync/src/sync.go:33 +0x16a
github.com/actions/actions-sync/cmd.glob..func4(0x1969ea0, 0xc0001a0200, 0x0, 0x8)
        /Users/pyama/src/github.com/actions/actions-sync/cmd/root.go:77 +0x96
github.com/spf13/cobra.(*Command).execute(0x1969ea0, 0xc0001a0100, 0x8, 0x8, 0x1969ea0, 0xc0001a0100)
        /Users/pyama/src/github.com/actions/actions-sync/vendor/github.com/spf13/cobra/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x1969420, 0x1969ea0, 0x1, 0x1)
        /Users/pyama/src/github.com/actions/actions-sync/vendor/github.com/spf13/cobra/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/pyama/src/github.com/actions/actions-sync/vendor/github.com/spf13/cobra/command.go:887
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /Users/pyama/src/github.com/actions/actions-sync/vendor/github.com/spf13/cobra/command.go:880
github.com/actions/actions-sync/cmd.Execute(0x168f350, 0xc0000bc000, 0xc00018bf78, 0x1007a85)
        /Users/pyama/src/github.com/actions/actions-sync/cmd/root.go:98 +0x1b8
main.main()
        /Users/pyama/src/github.com/actions/actions-sync/main.go:13 +0x3d
exit status 2

I think that peter-evans/create-pull-request has a some problem but I don't resolve problem.

Please tell me why this problem occurs.

Just dropping a suggestion here to allow GHEC admins to use this tool to sync to a GHEC owned organization so that Actions allowlists could point to internally curated Actions via these docs...

It looks like this would require support for the .com API versioning as well as the DNS split out for https://api.github.com vs the GHES URL mechanism.

Hej there!
Using the actions-sync I have to add the option --destination-token on the command line. But I don't feel too comfortable typing PATs/passwords on a command line, because they are available in the command history.

Could you think of making this parameter optional? I.e. if it's not provided on the command line, the action-sync command would ask for it in its first step?

I am setting up GitHub Actions Sync between my GitHub enterprise and GitHub.

I downloaded the actions sync tool and tried to sync a repo as mentioned in
https://docs.github.com/en/[email protected]/admin/github-actions/managing-access-to-actions-from-githubcom/manually-syncing-actions-from-githubcom#prerequisites

Observed that the action-sync works when the organization does not exist, it creates one. But when trying to sync to an organization that already exists, if fails.

See below for comparison:

Here, the org synced-actions already exists and has been created prior as directed in the above doc:

Note: I am an owner in the org, so we can rule out a permissions problem

admin@ghe-primary:~/actions-sync/bin$ ./actions-sync sync --cache-dir "/tmp/cache" --destination-token "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --destination-url "https://my-github-enterprise.com" --repo-name "docker/build-push-action:synced-actions/build-push-action"
fetching * refs for docker/build-push-action ...
syncing synced-actions/build-push-action
error creating github repository synced-actions/build-push-action: error creating organization synced-actions: POST https://my-github-enterprise.com/api/v3/admin/organizations: 422 Validation Failed [{Resource:Organization Field:login Code:custom Message:login is not available} {Resource:Organization Field:admins Code:missing_field Message:}]

In this scenario, the org synced-actions-new was not pre-created manually and the sync works

admin@ghe-primary:~/actions-sync/bin$ ./actions-sync sync --cache-dir "/tmp/cache" --destination-token "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --destination-url "https://my-github-enterprise.com" --repo-name "docker/build-push-action:synced-actions-new/build-push-action"
pulling docker/build-push-action to /tmp/cache/synced-actions-new/build-push-action ...

fetching * refs for docker/build-push-action ...
syncing synced-actions-new/build-push-action
Created organization synced-actions-new (admin: user)
Created repo synced-actions-new/build-push-action
successfully synced synced-actions-new/build-push-action

There is no official binary for darwin-arm64. This creates an entry barrier for darwin-arm64 users who want to quickly test out the tool.

Please provide an official release for darwin-arm64.

When we run the sync command with one of the inputs of repo-name, repo-name-list, or repo-name-list-file, it currently attempts to sync every single repository in the cache-dir AND the newly added repository. Rather than doing this, shouldn't it only care about the repositories from the relevant flag?

I have the following error while syncing this GitHub public action

pulling OctopusDeploy/install-octopus-cli-action to /tmp/actions/OctopusDeploy_install-octopus-cli-action ...                                                    
reference delta not found

I am using the latest version of the actions-sync binary.