Code Monkey home page Code Monkey logo

repo-sync's Introduction

Repo Sync

All Contributors

Keep a pair of GitHub repos in sync.

How it Works

This project uses GitHub Actions workflows to keep pairs of git repos in sync. It runs on a schedule (every 15 minutes by default). Shortly after changes are made to the default branch of repo A, the Actions workflow runs on repo B and generates a pull request including the recent changes from repo A. If more changes are made to repo A before the pull request is merged, those changes will be added to the existing pull request. The same is true in the opposite direction: changes made to repo B will eventually get picked up by the workflow in repo A.

Features

  • One-way or two-way sync
  • Sync between a private and public repo
  • Sync between two private repos
  • Sync between two public repos
  • Sync from a third-party repo to a Github repo
  • Uses Github Actions and a flexible scheduled job. No external service required!

Requirements

Installation

Step 1. Set up Secrets

GitHub Secrets are variables stored on your GitHub repository that are made available in the GitHub Actions environment. There are two (2) required secrets on each repo. Go to Settings > Security > Secrets and variables > Actions > New repository secret on your repo page and add the following secrets:

SOURCE_REPO

The shorthand name or URL of the repo to sync.

  • If the source repo is a public GitHub repo, use a shorthand name like owner/repo.
  • If the source repo is a private GitHub repo, specify an HTTPS clone URL in the format https://<access_token>@github.com/owner/repo.git that includes an access token with repo and workflow scopes. You can generate a token in your Github settings.
  • If the source repo is not hosted on GitHub, specify an HTTPS URL that includes pull access credentials.

INTERMEDIATE_BRANCH

The name of the temporary branch to use when creating a pull request, for example, repo-sync. You can use whatever name you like, but do NOT use the name of a branch that already exists, as it will be overwritten.

Step 2. Create Actions workflow files

Create a file .github/workflows/repo-sync.yml in both repositories and add the following content:

name: Repo Sync

on:
  schedule: 
  - cron: "*/15 * * * *" # every 15 minutes. set to whatever interval you like

jobs:
  repo-sync:
    name: Repo Sync
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: repo-sync/github-sync@v2
      name: Sync repo to branch
      with:
        source_repo: ${{ secrets.SOURCE_REPO }}
        source_branch: main
        destination_branch: ${{ secrets.INTERMEDIATE_BRANCH }}
        github_token: ${{ secrets.GITHUB_TOKEN }}
    - uses: repo-sync/pull-request@v2
      name: Create pull request
      with:
        source_branch: ${{ secrets.INTERMEDIATE_BRANCH }}
        destination_branch: main
        github_token: ${{ secrets.GITHUB_TOKEN }}

To set up two-way sync, create these files in both repositories. To set up one-way sync, you only need to do this step in the target (destination) repository.

Step 3. Watch the pull requests roll in!

There is no step 3! Once you commit to the repo, the workflows run on the schedule you specified in the workflow file. When repo-sync finds changes, it creates a pull request. If an unmerged repo-sync pull request already exists on the destination repo, repo-sync updates the existing PR.

Advanced configuration

The workflow file is fully customizable allowing for advanced configurations.

cron

The default cron is every 15 minutes. This can be easily adjusted by changing the cron string.

Manual events

Instead of triggering workflows using the cron scheduler, you can set up manual events to trigger the workflow when the source repo changes.

Workflow steps

You can add or remove workflow steps to meet your needs. For example, you might remove the "Create pull request" to commit directly, or you could add a "Merge pull request" step.

One-Way Syncs

For one-way syncs, set up the workflow in the target repository only as described above.

Customize pull request

You can customize the PR/s title, body, label, reviewer, assingee, and milestone by setting environment variables as explained at repo-sync/pull-request.

Use SSH clone URL and deploy keys

You can use a SSH clone URL and specify an SSH_PRIVATE_KEY environment variable instead of using the https clone URL.

Contributors โœจ

Thanks goes to these wonderful people (emoji key):

Wei He
Wei He

๐ŸŽจ ๐Ÿ’ป ๐Ÿ“–
Zeke Sikelianos
Zeke Sikelianos

๐Ÿ“– ๐Ÿค”

This project follows the all-contributors specification. Contributions of any kind welcome!

repo-sync's People

Contributors

allcontributors[bot] avatar dependabot[bot] avatar je-hal avatar jonathanwoollett-light avatar jsoref avatar rsese avatar sanscontext avatar shati-patel avatar smirnoal avatar wei avatar zeke avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

repo-sync's Issues

guidance: Your two repos must share a commit history.

The README mentions:

Your two repos must share a commit history.

I'm confused as to how to set this up exactly; could you provide some details/help for this? Especially for two repos that are already setup (and have commits in them) - I'd love to know how to integrate this to get my repos syncing.

Thanks

Storing Configuration Variables

There are two ways to use sync, unidirectional and bidirectional.

Unidirectional owner/repoA:master โ†’ owner/repoB:master

Configuration variables can be put directly in workflow yaml.
ps. setup for bidirectional flow (using secrets) will also work with unidirectional flow.

Unidirectional setup

repoB/.github/workflows/sync.yml

name: Sync repository

on:
  schedule:
  - cron: "*/15 * * * *"

jobs:
  repo-sync:
    name: Sync repository
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - uses: wei/github-sync@v1
      name: Sync repository to branch
      env:
        SOURCE_REPO: "owner/repoA"
        SOURCE_BRANCH: "master"
        DESTINATION_BRANCH: "from-repoA"
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        args: $SOURCE_REPO $SOURCE_BRANCH:$DESTINATION_BRANCH
    - uses: wei/pull-request@v1
      name: Create pull request
      env:
        SOURCE_BRANCH: "from-repoA"
        DESTINATION_BRANCH: "master"
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Bidirectional owner/repoA:master โ‡„ owner/repoB:master

For bidirectional use-cases, because master branches are synced, the workflow yaml file will be the same between repoA and repoB. Because of this, the values of those variables need to be kept separately from the codebase. However, there is no easy way to store key value pairs that is tied to a repository except for the recently introduced Secrets. Therefore, that is the route I've chosen for now.

Bidirectional setup

Add the following secrets

repoA/settings/secrets

SOURCE_REPO=owner/repoB
SOURCE_BRANCH=master
DESTINATION_BRANCH=from-repoB
PR_DESTINATION_BRANCH=master

repoB/settings/secrets

SOURCE_REPO=owner/repoA
INTERMEDIATE_BRANCH=from-repoA

repoA/.github/workflows/sync.yml AND repoB/.github/workflows/sync.yml

name: Sync repository

on:
  schedule:
  - cron: "*/15 * * * *"

jobs:
  repo-sync:
    name: Sync repository
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - uses: wei/github-sync@v1
      name: Sync repository to branch
      env:
        SSH_PRIVATE_KEY: ${{ secrets.SOURCE_REPO_PRIVATE_KEY }}
        SOURCE_REPO: ${{ secrets.SOURCE_REPO }}
        SOURCE_BRANCH: "master"
        INTERMEDIATE_BRANCH: ${{ secrets.INTERMEDIATE_BRANCH }}
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        args: $SOURCE_REPO $SOURCE_BRANCH:$INTERMEDIATE_BRANCH
    - uses: wei/pull-request@v1
      name: Create pull request
      env:
        SOURCE_BRANCH: ${{ secrets.INTERMEDIATE_BRANCH }}
        DESTINATION_BRANCH: "master"
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Any feedback/suggestions on this flow and using secrets to store config variables?

Two GitHub Actions

Is it actually required that both repositories need to have the GitHub Actions script for this to work?
I would believe so if updates are being carried out to each one, but this question is aimed towards one-way updates.

Stuck at "Set the GITHUB_TOKEN environment variable."

Setup job is

GITHUB_TOKEN Permissions
Contents: read
Metadata: read
Packages: read

And the error occurs there:

Run repo-sync/github-sync@v2
with:
source_repo: ***
source_branch: ***
destination_branch: ***
/usr/bin/docker run --name ghcrioreposyncgithubsyncv230_da0760 --label ed866e --workdir /github/workspace --rm -e "INPUT_SOURCE_REPO" -e "INPUT_SOURCE_BRANCH" -e "INPUT_DESTINATION_BRANCH" -e "INPUT_GITHUB_TOKEN" -e "INPUT_SYNC_TAGS" -e "GITHUB_TOKEN" -e "SYNC_TAGS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/homeassistant-core-mirror/homeassistant-core-mirror":"/github/workspace" ghcr.io/repo-sync/github-sync:v2.3.0 "" ":***"
Set the GITHUB_TOKEN environment variable.

External Events section links to missing section on docs.github.com

Heroku Buildpack Setup

Using two Heroku buildpacks for (1) project in a subdirectory (2) create-react-app

heroku buildpacks:clear -a github-repo-sync
heroku buildpacks:set https://github.com/timanovsky/subdir-heroku-buildpack -a github-repo-sync
heroku buildpacks:add mars/create-react-app -a github-repo-sync
heroku config:set PROJECT_PATH=sync -a github-repo-sync

Sync between Bitbucket and Github

My team is planning to migrate from Bitbucket to Github and we found the github-sync project that does exactly what we need, however, we need to sync between Bitbucket and Github.

Would it be possible to add this feature?

Thanks

One-way repo-sync?

Hi there,

the README says in section Features that one-way syncs are possible, but this case is not mentioned on the rest of the page.

What would I have to do for a one-way sync? Just adding the workflow file in one repo does not seem to work.

Thanks

Use Github Pages for hosting

Github Actions' GITHUB_TOKEN has push permission to gh-pages but it currently does not trigger a page build. Therefore, for now we are using heroku for hosting.

Workflow trigger considerations

Sample workflow file

name: "Sync wei/pull:master to pull"

on:
  schedule:
  - cron: "*/60 * * * *"
  
jobs:
  repo-sync:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - uses: repo-sync/github-sync@v1
      name: repo-sync
      env:
        SOURCE_REPO: "wei/pull"
        SOURCE_BRANCH: "master"
        DESTINATION_BRANCH: "pull"
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        args: $SOURCE_REPO $SOURCE_BRANCH:$DESTINATION_BRANCH

As discussed, the workflow file will reside in the destination repository (for our use-case, a workflow file will exist in each repository). The workflow in the destination repository has write access through the built-in secrets.GITHUB_TOKEN.

Cron is a way to allow maximum flexibility in the server/host and permission of the source repo. Repository dispatch event would be a great way to trigger the workflow when source repository sees a push action, but the request data and format of Github's built-in webhooks is quite different from what Repository dispatch event call takes in, therefore, a middleware/webhook proxy/or an action could to be setup to support this use-case in the future. But for now we are opting for a cron job every 15 minutes. If no updates were found, the job will succeed silently without taking action.

Framework Discussions

Are there any framework preferences as far as frontend project goes? Perhaps a boilerplate that Github uses to kickstart a project?

Also which backend web application / routing framework do you guys prefer?

Thanks!

help needed with proper setup

Hello,

I'm trying to setup synchronisation between a private and a public repositories, and facing the following issue:

  1. Public repo receives some commits, pub_sha1, pub_sha2
  2. repo-sync in the private repo creates a pull request for them
  3. pull request merged but with different checksums - priv_sha1, priv_sha2
  4. repo-sync in the public repo creates a pull request for them, named "Initial commit"

this is unexpected, because changes are already there (with different checksums, though)

I think it could be a problem with my setup and I'm missing some part, could you please help?

There was an error creating the workflow

After authenticating and filling out the form, I see the following message:

There was an error creating the workflow, please manually create it in zeke/test-private.

And I see this in the browser console:

main.9cb8cd7a.chunk.js:69 Error: 404 error making request put https://api.github.com/repos/zeke/test-private/contents/.github/workflows/repo-sync.yml: "Not Found"
    at new t (Requestable.js:110)
    at Requestable.js:434

Here's a screenshot:

Screen Shot 2019-09-09 at 8 09 18 PM

I will try to recreate this error locally in development.

cc @wei

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.