If you have a long list of repos created manually, it's easy to add duplicates :)

Then turbolift would complain every time an action would be executed.

I can provide a fix for that if you would be interested

eg:

turbolift clone --repos allrepos.txt

this would enable the following workflow:

turbolift clone --repos allrepos.txt

# identify repos that require the change
grep -r ... work/* > repos.txt

# now do the work on just that subset of repos in repos.txt 
turbolift foreach ...

The installation instructions could be simplified greatly if we use Homebrew for distribution. Per discussions, it would be good to set up a private tap and configure goreleaser to keep it up to date.

We need to ensure that MacOS distribution works without Gatekeeper problems; I don't think that signing/notarizing the binaries should be required for brew installs, but this needs verification.

CodeQL is a convenient code parser.
It can be added as a CI check and provide some relevant feedback on code style and best practices.

Would be good to add it as a check in Github Actions.

at the moment all prs commits etc are sequential.
For large numbers, adding parallel processing could improve the user experience by accelerating the feedback loop.

On large repo lists, one often has to run this kind of workflow:

turbolift clone
turbolift forearch <insert here the command to change the repo>

then, to validate the changes, they would run something like this:

turbolift foreach git diff
<frantically scroll through thousands of lines to figure out whether all repositories are successfully changed>
<then, trusting your eye sight, you stage and commit>
turbolift commit

It would be nice to have a way to commit those step-by-step with a visual validation from the user, introducing a --incremental.

The workflow would look like:

turbolift commit -i
-- Repository: REPOSITORY_NAME_1
<git diff on this specific repo>
<prompt for the user to confirm with a y/N>
-- Repository: REPOSITORY_NAME_2
...

We will log in the end the repo list that WASN'T updated, the user can decide what to do with it then.

turbolift create-prs expects the PR description to come from README.md

Would be nice to support a user-specified file instead so that README.md can be used for other purposes (eg: describing the process how to use turbolift)

I've recently had a couple of situations where new information has come to light only after mass-raising a bunch of PRs. That new information would be useful to include in the PR description for the benefit of those that haven't yet go around to reviewing the PR, but turbolift (as far as I can tell) doesn't keep a track of the PRs it has raised. This isn't necessarily a problem, but it does mean that the only interaction I have with the PR stage is just the creation stage and I have to live with whatever description I had for ever more.

I'm thinking there could be a different option, or perhaps a flag, that executes code similar to that for create-pr, but is able to update the description of the PR given the content of local README.md file. The logic might be something like:

1. For repo in repos.txt, try to create PR. If there's no PR, no problem, job done.
2. If it fails because a PR already exists, try and update the description. I don't think it even has to be clever enough to decide if it needs to, but that might be nice.

I get the following error when running foreach sed with turbolift 2:

`$ turbolift foreach sed -i '' 's/foo/bar/g' README.md

Error: unknown shorthand flag: 'i' in -i
Usage:
turbolift foreach -- SHELL_COMMAND [flags]
Flags:
-h, --help help for foreach
Global Flags:
-v, --verbose verbose output
2021/05/21 11:49:08 unknown shorthand flag: 'i' in -i`

A temporary workaround is to encapsulate the sed command in double quotes, but I think this is a bug in the way parameters are handled. (also some of the README examples do not work either)

Would help with usability of #11

At the end of a PR run, it would be great if the links were printed from the PRs that were created. So the results could look like this:

$ turbolift create-prs
  OK   Reading campaign data
  OK   Pushing changes in org/repo-1 to origin
  OK   Creating PR in org/repo-1
  OK   PR created: https://github.com/org/repo-1/pull/163
  OK   Pushing changes in org/repo-2 to origin
  OK   Creating PR in org/repo-2
  OK   PR created: https://github.com/org/repo-2/pull/163
  OK   Pushing changes in org/repo-3 to origin
  OK   Creating PR in org/repo-3
  OK   PR created: https://github.com/org/repo-3/pull/163
  OK   turbolift create-prs completed (3 OK, 0 skipped)

Love the tool, and find it very useful!

This feature is to be ported from the existing internal implementation of the tool. See the README and internal codebase to help maintain parity.

This feature creates a working directory for a 'campaign'. The user will run:

turbolift init --name some_campaign_name

And as a result a directory named some_campaign_name will be created with some skeleton contents:

• A README.md file, which will serve as a readme for the campaign and will also be used as the title/description for created PRs
• A .gitignore file covering a work/ directory
• An empty work directory
• A repos.txt file for the user to populate
• A .turbolift file containing a version number, v2.

As-is, turbolift users will end up with large numbers of forks in their personal org. This is annoying, and cleaning these up is a chore.

We should have a turbolift cleanup feature to delete a user's forks.

We should discuss safety aspects, e.g. should we avoid deleting forks if:

• the user's PR is still open for this campaign
• the user has other PRs open from their fork to upstream
• ... other reasons?

Some combination of warnings/prompts may help.

Description

It would be very helpful to add --auto-merge flag to turbolift create-prs command.

Motivation

In case of huge amount of repos it would be very convenient to enable auto-merge (if the repo has such function of course), so if all of the checks are green, then PR would be merged automatically.

As part of #75 I saw some limitations in the logging because the logger is instantiated in the command itself.
That limits for instance the use of logs within functions (unless we're injecting the logger dependency within the function signature).

It would be nice to move to a global singleton logger instead.

We should consider borrowing from https://github.com/Skyscanner/argocd-progressive-rollout/tree/main/.github/workflows

e.g. via a --draft flag which would cause all PRs to be created as drafts.

I had a txt file that had an empty line in the end. I think the tools should be able to filter empty lines so that you could group repositories by org or something.

This feature is to be ported from the existing internal implementation of the tool. See the README and internal codebase to help maintain parity.

This feature applies gh create-pr for each working copy. The user will run:

turbolift create-prs

And as a result gh create-pr will be run for each working copy.

Note that:

• The README.md file should be used to derive the PR title (first line heading) and description (rest of file)
• If a working copy has no changes compared with upstream, the gh command will fail. This is OK. A future enhancement could be to track reasons for failures to display to the user, but as-is we should keep it simple.
• We should rate limit creation of PRs to avoid generating too much load (particularly load on shared CI systems)

The current state of repos.txt is the authoritative list - this command should only act on repos that are listed there and which are not commented out.

That's actually a feature of a git commit -a, however it's unexpected for a turbolift user, as it states that you don't have to do anything. Currently workaround is to turbolift foreach -- git add -A, but maybe that could be part of the commit command by default?

As a user when I run turbolift I would want to predefined settings to be used from a config file instead of typing it all

When running large campaigns it's possible you need to experiment changes with a subsection of your repositories. Currently you need to delete or comment out the repositories you don't want to be taken into account before running any turbolift command.
It would be useful to provide a command line parameter to specify the file to be used as a repository list instead of copy pasting the content into the repos.txt file.
The repos.txt file would continue being the default value.

turbolift clone --repos=test-repos.txt
turbolift foreach --repos=test-repos.txt rm somefile

or

turbolift --repos=test-repos.txt clone
turbolift --repos=test-repos.txt foreach rm somefile

This feature is to be ported from the existing internal implementation of the tool. See the README and internal codebase to help maintain parity.

This feature clones all repositories that are listed in repos.txt

turbolift clone

And as a result all listed repos will be cloned into a work/ORG_NAME/REPO_NAME directory. Any lines in the repos.txt file prefixed by a # should be considered as a comment and ignored.

For now we should wrap the gh command to create a fork of the original repository into the user's personal org and then clone it.

The current internal implementation of the tool uses git worktree for local space efficiency, but we should probably avoid this for simplicity's sake: just a regular fork+clone should suffice.

The specific wording should be:

By approving this PR, you are confirming that you have adequately and effectively reviewed this change.

This should appear in the README.md for new campaigns.

This came up during the Turbolift post mortem, having the ability to file a PR on small % of the repositories first before expanding this to all repositories.

Pushing to git and PR creation can both cause CI workflows to start (both are part of the create-prs subcommand). Particularly in an enterprise context, this could put pressure on shared infrastructure.

We should put in place reasonable measures to prevent this, e.g. one or more of:

• opt-in rate limiting (e.g. a fixed pause between creation)
• automatic rate limiting of PR creation when a certain number of repos is involved.
• using check status (number of PRs with pending checks) as a signal to slow down rate of creation. This would be a fairly involved feature, which may be too much for an initial implementation.

https://github.com/cli/cli/releases/tag/v2.18.0, released yesterday, seems to behave slightly differently and breaks turbolift create-prs.

I get:

...
  OK   Pushing changes in myorg/myrepo to origin
 FAIL  Creating PR in myorg/myrepo: Error: exit status 1. Stderr: aborted: you must first push the current branch to a remote, or use the --head flag

Downgrading gh to 2.17.0 seems to be a temporary workaround and works flawlessly.

Turbolift currently runs multiple commands in series over the list of repositories, displaying stdout/stderr to the user but ignoring exit codes.

This can make it difficult for the user - they have to scour logs for failures.

We should collect errors (i.e. exit code != 0) and display a suitably formatted list at the end of execution.

Break the dependency on execing the git command, and use https://github.com/go-git/go-git instead.

This feature request adds a lot of complexity to this project but here it is anyway

PROBLEM:
Suppose you have a constant list of 10 repos you want to use Turbolift for. Each and every campaign you create will result in a new clone of the repos. This takes up unnecessary space on a dev's local drive, plus it's slow to download them over and over.

FEATURE REQUEST:
Allow to define a localpath in repos.txt which indicates a local clone of a repository to be worked on. Use this clone instead of creating a new one in the work directory for every new campaign

ALTERNATIVE APPROACH:
Allow to define a repos.txt (or core_repos.txt) in the root of a turbolift directory (this also invents this concept). These core_repos are what you expect to work on most of the time, and thus shouldn't re-clone everytime. There would also be a work directory beside the core_repos.txt.

The file structure would look something like this:

/turbolift_project
--core_repos.txt
--/work
----/repo1
----/repo2
--/campaign_1
----repos.txt
----/work
------/repo3
--/capmpaign_2
----/repos.txt
----/work
------/repo4

This feature is to be ported from the existing internal implementation of the tool. See the README and internal codebase to help maintain parity.

This feature performs a git commit for each working copy that has been cloned by #2. The user will run:

turbolift commit --message "some message"

And as a result git commit will be run with the given message. Note that only working copies with staged changes should be affected - i.e. no empty commits should be created, and no files should be committed that the user hasn't explicitly staged (using git add) already.

The current state of repos.txt is the authoritative list - this command should only act on repos that are listed there and which are not commented out.

Turbolift creates a fork for each project it needs to interact with.
Being Turbolift an excellent tool to handle maintenance tasks in the projects where you work usually it doesn't make much sense to fork these repositories where you already have write access.
It would be better if Turbolift could use the original repository if possible and fallback to a fork if not.

One thing we could think about doing is turbolift status.

Some of the useful things is could log would be:

• How many repos have had a PR created
• How many PRs are in each of the open/closed/merged states
• How many PRs have checks failures or have review rejections

As well as listing numbers, we could also consider a --verbose or --list flag which lists each the PRs in each category. The listing would ideally include a URL pointing to the each PR.

Reading the code, it seems it's the current user in the terminal session, but I wonder if it's possible to configuring turbolift to use a machine account instead.

It would be cool to be able to update the PR description once they are open.

Some repositories might not want to get some turbolift updates.
As a repository we should have the option to opt-out and block turbolift requests.

Before committing to a change (pun absolutely intended), it is sometimes useful to validate that the changes that were made actually came out sensibly in practise.

I would like to see a configuration/environment option to commands like turbolift commit and turbolift create-prs (i.e. any write-like commands) to run them in a dry run mode. For example, the dry run commit could be a report of each git diff, and for create-prs it might be a summary of the git diff against the latest default branch.

Personal forks of repos can become seriously outdated in between uses of the tool, meaning that it's easy to accidentally create PRs against very old versions of the target repos. This can result in merge conflicts when creating PRs.

I believe that the simplest way to prevent this would be to amend turbolift clone so that it effectively does:

• Create a fork
• Clone the forked repo locally
• Create a new local branch
• New: Pull changes from upstream/main or upstream/master into the local branch, so that any changes made by the user are against the latest available version of upstream

I think this is likely an issue in the spinner library; interrupting the turbolift process while performing work can cause the cursor to be disabled.

Currently the PR description is taken from the README file of the project. When running large campaigns it's possible you need to create PRs with different descriptions.
It would be useful to provide a command line parameter to specify the file to be used for the description instead of copy pasting the content in the REDME file.
The REDME file would continue being the default value.

turbolift create-prs --description prs/description1.md

This feature is to be ported from the existing internal implementation of the tool. See the README and internal codebase to help maintain parity.

This feature runs a shell command against every working copy. e.g.:

turbolift foreach echo "$USER" >> foo.txt

Would append the user's name to a file named foo.txt in each working copy. A shell should be used for execution so that placeholder substitution etc work properly.

We might want to consider a more unix-native syntax, with -- separating the turbolift command from the subcommand. e.g.:

turbolift foreach -- echo "$USER" >> foo.txt

We should only consider serial execution at this time. We should also ignore the exit code (though we might want to do something with it in a future enhancement).

The current state of repos.txt is the authoritative list - this command should only act on repos that are listed there and which are not commented out.

For expediency the tool currently execs gh as a subprocess for several operations with GitHub.

We should break this dependency so that turbolift has no dependencies on external programs.

It would be nice to have an option to do a shallow clone for quicker clones. This would allow us to grab for example, the last 30 commits per repo instead of the entire commit history

# maybe --shallow-clone which could default to 50 last commits
turbolift clone --shallow-clone
# provide the clone depth which could default to 50 or -1 to account for the default deep clone
turbolift clone --depth=50

https://www.perforce.com/blog/vcs/git-beyond-basics-using-shallow-clones
https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

Example

git clone -–depth [depth] [remote-url]

Use case: You created a campaign, file lots of PRs, find that this was not successful, and want to cancel everything.
Feature: be able to close all PRs generated by a campaign.

It would be good to help users understand how useful their PRs were.

To do this, we could take advantage of the 👍/👎 reactions on PRs:

• as part of the PR description boilerplate, encourage PR receivers to set a reaction
• as part of #18, collate the counts of each kind of reaction and include this in the status output

When doing a turbolift clone in a windows environment, it reports the following error:

Executing: git [checkout -b C:\src\turbolift\campaignx] in work/xxx/xxxxxx fatal: 'C:\src\turbolift\campaignx' is not a valid branch name.
It does actually manage to clone the repositories, but when then trying turbolift create-prs it reports the following error:

Executing: git [push -u origin C:\src\turbolift\campaign] in work/xxx/xxxxxxx fatal: invalid refspec 'C:\src\turbolift\campaign'