We are restructuring our code into multipackage repository, so we move around a lots of files eg. from

/app/templates/foo.hbs -> /packages/foo-package/app/templates/foo.hbs

We want to use downstream readonly repos with the subpackages.
The problem we have encountered is that renames are not followed by splitsh-lite, so our history in downstream repos is limited only to commits after a rename.

Reproduction script:

#!/bin/sh

git init split-repo
cd split-repo
mkdir foo

echo "\n:: Creating history"
for i in 1 2 3
do
    echo "$i" > foo/counter.txt
    git add foo
    git commit -m "Updated counter to $i"
done

# split before moving
echo "\n:: Splitting before rename"
splitsh-lite  --prefix foo/ --target refs/heads/before-rename --progress --debug

# move to subfolder
echo "\n:: Moving foo to packages/foo"
mkdir -p packages/foo
git mv foo/counter.txt packages/foo/counter.txt
git commit -m "Moved foo to packages folder"

echo "4" > packages/foo/counter.txt
git add packages
git commit -m "Updated counter to 4"

# split after moving
echo "\n:: Splitting after rename"
splitsh-lite  --prefix packages/foo/ --target refs/heads/after-rename --progress --debug

Script output:

:: Creating history
[master (root-commit) 33bc3fe] Updated counter to 1
 1 file changed, 1 insertion(+)
 create mode 100644 foo/counter.txt
[master cc7a422] Updated counter to 2
 1 file changed, 1 insertion(+), 1 deletion(-)
[master 61d2845] Updated counter to 3
 1 file changed, 1 insertion(+), 1 deletion(-)

:: Splitting before rename
2017/09/08 12:09:18 Splitting refs/heads/master
2017/09/08 12:09:18   From "foo/" to "ROOT"
2017/09/08 12:09:18 Processing commit: 33bc3fee1b11369771b07a3cdfcb784936ee6466
2017/09/08 12:09:18   parents:
2017/09/08 12:09:18   newparents:
2017/09/08 12:09:18   tree is: 073c7953b8056de57b9849a18c5d5e0ddbff0315
2017/09/08 12:09:18   copy commit "33bc3fee1b11369771b07a3cdfcb784936ee6466" "073c7953b8056de57b9849a18c5d5e0ddbff0315" ""
2017/09/08 12:09:18   newrev is: 5772eae3dc9330aaeba9e4b698598eeee9aa210e
2017/09/08 12:09:18 Processing commit: cc7a422d6dc42902037613d7f41a00ff6da136a0
2017/09/08 12:09:18   parents: 33bc3fee1b11369771b07a3cdfcb784936ee6466
2017/09/08 12:09:18   newparents: 5772eae3dc9330aaeba9e4b698598eeee9aa210e
2017/09/08 12:09:18   tree is: ba4285a8b7086fc3c91d2b9208047ed37d94cf05
2017/09/08 12:09:18   copy commit "cc7a422d6dc42902037613d7f41a00ff6da136a0" "ba4285a8b7086fc3c91d2b9208047ed37d94cf05" "5772eae3dc9330aaeba9e4b698598eeee9aa210e"
2017/09/08 12:09:18   newrev is: f8fcd6f8d8f04c6ced25ddc67870f3752dc83ffe
2017/09/08 12:09:18 Processing commit: 61d28451244d86fc2af159b05584d6548fe62f0c
2017/09/08 12:09:18   parents: cc7a422d6dc42902037613d7f41a00ff6da136a0
2017/09/08 12:09:18   newparents: f8fcd6f8d8f04c6ced25ddc67870f3752dc83ffe
2017/09/08 12:09:18   tree is: 9e7921bfc371b1f1d2f8031c7cac0276a93329f2
2017/09/08 12:09:18   copy commit "61d28451244d86fc2af159b05584d6548fe62f0c" "9e7921bfc371b1f1d2f8031c7cac0276a93329f2" "f8fcd6f8d8f04c6ced25ddc67870f3752dc83ffe"
2017/09/08 12:09:18   newrev is: 6037894597c3ee5df0583afcfa634c51c2becb65
3 commits created, 3 commits traversed, in 6ms
6037894597c3ee5df0583afcfa634c51c2becb65

:: Moving foo to packages/foo
[master 61bc4ed] Moved foo to packages folder
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename {foo => packages/foo}/counter.txt (100%)
[master f742fae] Updated counter to 4
 1 file changed, 1 insertion(+), 1 deletion(-)

:: Splitting after rename
2017/09/08 12:09:18 Splitting refs/heads/master
2017/09/08 12:09:18   From "packages/foo/" to "ROOT"
2017/09/08 12:09:18 Processing commit: 33bc3fee1b11369771b07a3cdfcb784936ee6466
2017/09/08 12:09:18   parents:
2017/09/08 12:09:18   newparents:
2017/09/08 12:09:18 Processing commit: cc7a422d6dc42902037613d7f41a00ff6da136a0
2017/09/08 12:09:18   parents: 33bc3fee1b11369771b07a3cdfcb784936ee6466
2017/09/08 12:09:18   newparents:
2017/09/08 12:09:18 Processing commit: 61d28451244d86fc2af159b05584d6548fe62f0c
2017/09/08 12:09:18   parents: cc7a422d6dc42902037613d7f41a00ff6da136a0
2017/09/08 12:09:18   newparents:
2017/09/08 12:09:18 Processing commit: 61bc4ed159b49269d5ccbf66b5303f45ff5779c4
2017/09/08 12:09:18   parents: 61d28451244d86fc2af159b05584d6548fe62f0c
2017/09/08 12:09:18   newparents:
2017/09/08 12:09:18   tree is: 9e7921bfc371b1f1d2f8031c7cac0276a93329f2
2017/09/08 12:09:18   copy commit "61bc4ed159b49269d5ccbf66b5303f45ff5779c4" "9e7921bfc371b1f1d2f8031c7cac0276a93329f2" ""
2017/09/08 12:09:18   newrev is: 178368081122bcc9ef05febf3223431e25562901
2017/09/08 12:09:18 Processing commit: f742faef4b4a536768685b500d80a1426449058d
2017/09/08 12:09:18   parents: 61bc4ed159b49269d5ccbf66b5303f45ff5779c4
2017/09/08 12:09:18   newparents: 178368081122bcc9ef05febf3223431e25562901
2017/09/08 12:09:18   tree is: 2cf1bff97ea95b5774d3040e6386ee8cf51dcbfb
2017/09/08 12:09:18   copy commit "f742faef4b4a536768685b500d80a1426449058d" "2cf1bff97ea95b5774d3040e6386ee8cf51dcbfb" "178368081122bcc9ef05febf3223431e25562901"
2017/09/08 12:09:18   newrev is: b32d1051c7deadbb8fdb222c0c4bb9ee17ed7f46
2 commits created, 5 commits traversed, in 3ms
b32d1051c7deadbb8fdb222c0c4bb9ee17ed7f46

Inspecting history of created branches shows that after rename history is not followed:

> git checkout before-rename
> git log --oneline

6037894 (HEAD -> before-rename) Updated counter to 3
f8fcd6f Updated counter to 2
5772eae Updated counter to 1

> git checkout after-rename
>git log --oneline

b32d105 (HEAD -> after-rename) Updated counter to 4
1783680 Moved foo to packages folder

@fabpot Is there a way to preserve history (ie. follow renames)?

The sha1 can be used to create a branch or to push the commits to a new repository.

I'm confused at how I can push the commits to a new repository?

In the DrupalCon New Orleans 2016: The Symfony Monolith Repository video, you are using splitsh, but its not the version released here and appears to have a lot more features than this splitsh/lite, just curious if that more advanced splitsh is going to be released as well, or made available as a service or something.

When I try to use the --target=some-new-branch option I get this error:

The target is not a valid Git reference

My understanding was that the target option took a branch name that does not exist and that it would create the branch after the split. Is this incorrect? Is target instead supposed to be used to add new commits to an existing split stored in an existing branch?

Cheers and thanks!

Hi,

Would love to run this on docker and share the cache file between containers.

Regards

I don't find the way, to automatically put the commit on a branch.

If "test-splitsh" doesn't exist

# does not create the branch
splitsh-lite --prefix inc/library --target=heads/test-splitsh --progress
# The target is not a valid Git reference
splitsh-lite --prefix inc/library --target=test-splitsh --progress

If "test-splitsh" exist

# does not update the branch
splitsh-lite --prefix inc/library --target=heads/test-splitsh --progress
# The target is not a valid Git reference
splitsh-lite --prefix inc/library --target=test-splitsh --progress

Did anyone figure out how to merge PRs to a many-repo as discussed in the linked slides?

git merge -s subtree many-pr-branch

results in fatal: refusing to merge unrelated histories

If you do so, what will be the first version ? 1.0.0 ? 0.0.1 ?

I plan to create a debian package, it would be useful to know in advance the versionning scheme :)

In testing splitsh-lite I was comparing the resulting split with what git subtree split produces and noticed that the resulting hashes are different at the end of the split. In the tests for splitsh-lite, I noticed that the hashes should be the same regardless of which tool you use. Unsure how to provide useful information here as the repository I'm using isn't publicly available. When I run splitsh-lite it gives this output:

472 commits created, 103972 commits traversed, in 8.962s

But when I count the commits with git rev-list --count it is actually 466. When comparing the logs of the two splits, the git subtree split has all 472 commits. The logs are identical until the last 49 commits.

My guess is that maybe some of the commits resulted in no changes somehow, which prevents the commit from taking place? In the end, maybe this isn't really a problem unless we (for whatever reason) switch to git subtree split?

Please let me know if I can provide any other information as I know this report is likely lacking.

The fastest way to get started is to download a binary for your platform.

Where is it?
I'd like to use Travis-CI to split my monolytic repository. Do you have an example of how to do it?

Dear all,

since this great binary is used in laravel-framework it would be very nice when you update to a newer go version.

https://www.cvedetails.com/vulnerability-list/vendor_id-14185/product_id-29205/version_id-1035689/Golang-GO-1.13.8.html

Very thanks for this great project !

Regards,
Stefan

Sometimes you might want to modify a commit message in a split because it touches both open source and company-specific functionality.

Would be great to have a message rewriter script that can filter messages.

Hi folks

First, I want to congratulate for this initiative.

Secondly, I want to request a less cryptic README. I installed sliptsh/lite and is working.

But I don't understand what I need to do after splitting.

I create a new branch in Drupal Console project to explain what I want to do; the branch is https://github.com/hechoendrupal/DrupalConsole/tree/splitsh

Recently we started to split in our project to separate language amount other pieces. For instance, the Japanese-language had a new repository https://github.com/hechoendrupal/drupal-console-ja

What I did was copy inside the DrupalConsole branch language in folder console-ja.

What I need to know is how I could split using something like --prefix-console-js, but I don't understand how I could push the repo after the split in the proper target.

Let me know if I wasn't clear enough

If you want to learn more about monorepo vs manyrepos, watch this 4-minutes lightning talk I gave at dotScale... or the longer version from DrupalCon.

There is no links :(

I'm not sure if this is the place to ask for some examples while trying to split and push a sub packe to a new repo. Maybe an examples could eventually be added to the main readme.

I'm trying to build my own monorepo https://github.com/symmono/symmono with for example a sub package framework-bundle https://github.com/symmono/framework-bundle.

On the main branch (master) i'm running

# splitsh, trying to split the master branch FrameworkBundle to it's own repo
git splitsh --prefix=packages/Symmono/Bundle/FrameworkBundle

# sha1
0 commits created, 0 commits traversed, in 0s
c5a195e73c6e2317f44fae252c1d2ae3beb323e7

1. Is there a command to see whats in that sha1?

Now i'm trying to push it to the framework-bundle repo

# at this moment the framework-bundle is an empty repo
# command 1 with respone
git push [email protected]:symmono/framework-bundle.git c5a195e73c6e2317f44fae252c1d2ae3beb323e7

fatal: c5a195e73c6e2317f44fae252c1d2ae3beb323e7 cannot be resolved to branch

# command 2 with respone
git push [email protected]:symmono/framework-bundle.git c5a195e73c6e2317f44fae252c1d2ae3beb323e7:master

error: unable to push to unqualified destination: master
The destination refspec neither matches an existing ref on the remote nor
begins with refs/, and we are unable to guess a prefix based on the source ref.
error: failed to push some refs to '[email protected]:symmono/framework-bundle.git'

Can't get my head around it how to split and push it to a new repo. And how to push to a branch and is it possible with splitsh to use tags.

2. How does the --target flag works?

# branch split example
# when reading readme the --target flag shoud automatically create a branch for the split by passing a branch name, when running below command om main master monorepo
git splitsh --prefix=packages/Symmono/Bundle/FrameworkBundle --target=origin/0.1
0 commits created, 0 commits traversed, in 8ms
c5a195e73c6e2317f44fae252c1d2ae3beb323e7

# trying to push to branch
git push [email protected]:symmono/framework-bundle.git c5a195e73c6e2317f44fae252c1d2ae3beb323e7 0.1
error: src refspec 0.1 does not match any.
error: failed to push some refs to '[email protected]:symmono/framework-bundle.git'

Above split and push commends are apparently not the way to go.

3. How could i split and push the sha1 to an new repo.
4. Is it possible to split and push with tags?

Some extra questions

5. Do you always split of the master branch, when or why to split a branch.
6. When working with tags, for example v2.1.2 should you work on branch 2.1 and consider the master branch as new released branch. In this example for v2.2?

-_-||

You came to written golang now .
You do not love PHP anymore.

Otherwise, this project should written with PHP .

https://github.com/Automattic/jetpack/tree/master/modules has a complex structure. To split individual modules, you'd need to pick a set of files and directories from all over the tree.

Target module is significant enough to be worth some hassle. Any advice here?

When I create a monorepo by combining multiple repositories using git subtree add --prefix=.. and then re-split the prefix using splish-lite --prefix= it creates only one commit in the branch, which is the merge commit created by git subtree add --prefix= and all other history is lost. When I use git subtree split --prefix= that preserves the history. Am I getting something wrong? Maybe it's because git subtree add creates a merge commit and splitsh-lite ignores it?

I've got a monorepo with hundreds of standalone project-root folders which I'm converting to an equal number of individual branches (each branch representing the root of each standalone project currently in the monorepo)

Currently, my "best approach" is running subtree split in parallel: specifying each folder in the root of the monorepo and the corresponding branch name. e.g.:

for d in *; do
    # Will print */ if no directories are available
    #echo $d
    git subtree split -P "$d" -b "$d" &
done

The issue with this approach is that it pegs the CPU for approximately 8 hours and is woefully inefficient.

I would like to be able to run one instance of subtree split in a manner that is capable of multiple target branches for each commit it parses (dependent on the paths affected by each commit): rather than multiple instances of subtree split each focusing on it's own path pattern.

One possible idea would be to allow RegEx parsing of the pattern with capture groups; and allow use of the RegEx capture group value within the target branch name.

Hi, I was wondering how you would set this up together with lerna. Any help would be super appreciated!

I ran into an issue where the hashes are different between this lib and a standard subtree split. This is causing issues when trying to push the changes to a child repos, while using git subtree push works fine.

$ git version
git version 2.34.1

$ splitsh-lite --version
splitsh-lite version v1.0.1

This issue can be reproduced via this script using https://github.com/crystal-manyrepos/two (but also reproduces when using this repo for example):

mkdir mono
cd mono/
git init
touch README.md
git add -A
git commit -m "Initialize mono repo"

git subtree add --prefix=libs/two/ [email protected]:crystal-manyrepos/two.git master

splitsh-lite --scratch --prefix=libs/two && git subtree split --prefix=libs/two

More details below, but it seems like splitsh isn't gathering all the expected commits which results in the hash being different in the end. I'm not using --squash so I don't think it's related to #15. I actually also tried with that Docker image in that issue and same problem.

Output:

Initialized empty Git repository in /home/george/dev/git/crystal-many/mono/.git/
[master (root-commit) 1a613f0] Initialize mono repo
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 README.md
git fetch [email protected]:crystal-manyrepos/two.git master
remote: Enumerating objects: 20, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 20 (delta 3), reused 19 (delta 2), pack-reused 0
Unpacking objects: 100% (20/20), 4.01 KiB | 373.00 KiB/s, done.
From github.com:crystal-manyrepos/two
* branch            master     -> FETCH_HEAD
Added dir 'libs/two'
1 commits created, 5 commits traversed, in 6ms
c8f28643a1f9cb160a075e1672addd1a30d63d6b
88ce77827ecd614fcca6e2bbb6983999daab2367

Debug output:

$ splitsh-lite --prefix=libs/two/ --scratch --debug
2021/12/12 10:25:17 Splitting refs/heads/master
2021/12/12 10:25:17   From "libs/two/" to "ROOT"
2021/12/12 10:25:17 Processing commit: 1a613f0abf4fd5ce57a6d7ec2c9e70bef0c61127
2021/12/12 10:25:17   parents:
2021/12/12 10:25:17   newparents:
2021/12/12 10:25:17 Processing commit: 6e5fa9f03d5350ebee79b6899de1072a3dd126e9
2021/12/12 10:25:17   parents:
2021/12/12 10:25:17   newparents:
2021/12/12 10:25:17 Processing commit: 686027d84b670d9fa7861fe9b16c24a79428a2d4
2021/12/12 10:25:17   parents: 6e5fa9f03d5350ebee79b6899de1072a3dd126e9
2021/12/12 10:25:17   newparents:
2021/12/12 10:25:17 Processing commit: 88ce77827ecd614fcca6e2bbb6983999daab2367
2021/12/12 10:25:17   parents: 686027d84b670d9fa7861fe9b16c24a79428a2d4
2021/12/12 10:25:17   newparents:
2021/12/12 10:25:17 Processing commit: 868052990df6cddf09cc35c4bcbf63c4ade5c66c
2021/12/12 10:25:17   parents: 1a613f0abf4fd5ce57a6d7ec2c9e70bef0c61127 88ce77827ecd614fcca6e2bbb6983999daab2367
2021/12/12 10:25:17   newparents:
2021/12/12 10:25:17   tree is: 774bd095bf0c560acd754e6351f389fe2d1451f0
2021/12/12 10:25:17   copy commit "868052990df6cddf09cc35c4bcbf63c4ade5c66c" "774bd095bf0c560acd754e6351f389fe2d1451f0" ""
2021/12/12 10:25:17   newrev is: c8f28643a1f9cb160a075e1672addd1a30d63d6b
1 commits created, 5 commits traversed, in 4ms
c8f28643a1f9cb160a075e1672addd1a30d63d6b

Git log (master):

$ git log
commit 868052990df6cddf09cc35c4bcbf63c4ade5c66c (HEAD -> master)
Merge: 1a613f0 88ce778
Author: George Dietrich <[email protected]>
Date:   Sun Dec 12 10:24:46 2021 -0500

Add 'libs/two/' from commit '88ce77827ecd614fcca6e2bbb6983999daab2367'

git-subtree-dir: libs/two
git-subtree-mainline: 1a613f0abf4fd5ce57a6d7ec2c9e70bef0c61127
git-subtree-split: 88ce77827ecd614fcca6e2bbb6983999daab2367

commit 1a613f0abf4fd5ce57a6d7ec2c9e70bef0c61127
Author: George Dietrich <[email protected]>
Date:   Sun Dec 12 10:24:45 2021 -0500

Initialize mono repo

commit 88ce77827ecd614fcca6e2bbb6983999daab2367
Author: George Dietrich <[email protected]>
Date:   Sat Dec 11 23:19:21 2021 -0500

Bump component versions

commit 686027d84b670d9fa7861fe9b16c24a79428a2d4
Author: George Dietrich <[email protected]>
Date:   Sat Dec 11 22:39:52 2021 -0500

Add .summary method

commit 6e5fa9f03d5350ebee79b6899de1072a3dd126e9
Author: George Dietrich <[email protected]>
Date:   Sat Dec 11 22:39:10 2021 -0500

Initial commit

Git log, branch created via splitsh:

$ splitsh-lite --prefix=libs/two/ --scratch --target=refs/heads/splitsh
1 commits created, 5 commits traversed, in 6ms
c8f28643a1f9cb160a075e1672addd1a30d63d6b

$ git co splitsh

$ git log
commit c8f28643a1f9cb160a075e1672addd1a30d63d6b (HEAD -> splitsh)
Author: George Dietrich <[email protected]>
Date:   Sun Dec 12 10:24:46 2021 -0500

Add 'libs/two/' from commit '88ce77827ecd614fcca6e2bbb6983999daab2367'

git-subtree-dir: libs/two
git-subtree-mainline: 1a613f0abf4fd5ce57a6d7ec2c9e70bef0c61127
git-subtree-split: 88ce77827ecd614fcca6e2bbb6983999daab2367

Git log, branch created via subtree split:

$ git subtree split -P libs/two -b subtree-split
Created branch 'subtree-split'
88ce77827ecd614fcca6e2bbb6983999daab2367

$ git co subtree-split

$ git log
commit 88ce77827ecd614fcca6e2bbb6983999daab2367 (HEAD -> subtree-split)
Author: George Dietrich <[email protected]>
Date:   Sat Dec 11 23:19:21 2021 -0500

Bump component versions

commit 686027d84b670d9fa7861fe9b16c24a79428a2d4
Author: George Dietrich <[email protected]>
Date:   Sat Dec 11 22:39:52 2021 -0500

Add .summary method

commit 6e5fa9f03d5350ebee79b6899de1072a3dd126e9
Author: George Dietrich <[email protected]>
Date:   Sat Dec 11 22:39:10 2021 -0500

Initial commit

For now we have monolithic repository with added subtree (~45k commits) repository using squash option, like

Add 'package/' from commit 'hash'
git-subtree-dir: package
git-subtree-mainline: hash
git-subtree-split: hash

Any of --git values creates ~3k commits only

../go/splitsh-lite --prefix=package --target=heads/subtree_go --scratch --progress --git="<1.8.2"
3115 commits created, 97916 commits traversed, in 4.132s
45edc1a3dc374172538b3ed41964cbaedcc19c66

../go/splitsh-lite --prefix=package --target=heads/subtree_go --scratch --progress --git="<2.8.0"
3115 commits created, 97916 commits traversed, in 4.246s
240c137ba401897ef2a7e224af6e05895673dabf

../go/splitsh-lite --prefix=package --target=heads/subtree_go --scratch --progress
3164 commits created, 97916 commits traversed, in 5.313s
c0d65fd26c485144a42fcf073422289a0dba5822

Is it possible to attach created commits to subtree history to be able to push changes back into repository without conflicts and forced push?

This is named like #20, but a different bug, as far as we know.

The commit neos/flow-development-collection@882be01 results in neos/flow@4d563e6 right now, using the subtree split. Splitting using splitsh it ends up in kdambekalns/flow@79cae3e. The latter looks exactly like the commit from the development collection…

Looking further, there is no difference between the results, but the hashes differ. To maximise confusion, GitHub actually shows a difference, locally Git says it's the same (https://github.com/neos/flow/compare/3.0...kdambekalns:3.0-splitsh?expand=1 vs. git diff 3.0..3.0-splitsh after cloning kdambekalns/flow)

The split was done using splitsh-lite --prefix=TYPO3.Flow --origin=origin/3.0 --target=refs/heads/3.0-splitsh

I don't know if it's related to big sur or the new apple M1 chip but I get a segmentation fault when executing the binary.

→ ./splitsh-lite
[1]    79703 segmentation fault  ./splitsh-lite

→ lldb ./splitsh-lite
(lldb) target create "./splitsh-lite"
Current executable set to './splitsh-lite' (x86_64).
(lldb) run
Process 79368 launched: './splitsh-lite' (x86_64)
Process 79368 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xf1)
    frame #0: 0x0000000004051805 splitsh-lite`main + 933
splitsh-lite`main:
->  0x4051805 <+933>: movl   $0xf1, 0xf1
    0x4051810 <+944>: retq
    0x4051811 <+945>: int3
    0x4051812 <+946>: int3
Target 0: (splitsh-lite) stopped.
(lldb)

Hi all,

Has anyone figured out how to combine mono/manyrepos with GitHub pull requests and CI? For my non-monorepo projects, each PR automatically gets built by Travis, which helps a lot with screening of pull requests.

I've got CI for all the manyrepos, but nothing in the monorepo where all PRs are received. Are there any tricks to automatically migrate monorepo PRs/branches to manyrepo PRs/branches? Any smart git hacks?

Cheers,
Aslak

Hello,

I am trying to install splitsh-lite in windows 10. I read #18 and followed the manual instructions until the last step when an error is thrown:

In cmd:

go get github.com/splitsh/lite

github.com/libgit2/git2go

C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lgit2
collect2.exe: error: ld returned 1 exit status

In MINGW64:
$ go build -o splitsh-lite github.com/splitsh/lite

github.com/libgit2/git2go

C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lgit2
collect2.exe: error: ld returned 1 exit status

Could you help me or give me some guidelines in order to build splitsh-lite?

Thank you!

Thanks for splitsh-lite.

I have a question. Can I use splitsh-lite to create a new monorepo from many minirepos?

My use case is that we manage our code out of a lot repos which we then bring in via composer. I would like to establish a new monorepo that contains the base application as well as all our internally developed dependencies.

e.g.

.git/
app/
dep1/
dep2/
depN/

Hello,

This is a question: is splitsh available as a github action ?

I would like to know if splitsh can be easily implemented as an action which will update the readonly repositories on commit event on the main/master branch of the mono repository ? I guess that can be done anyway as a shell script called from a github action but if splitch is already packaged as a github action that would make the configuration more concise.

Thanks,

I did not find any binary for windows. I faced few errors when I tried to install manually

Seeing the last release is still from 2017, Go module support is still not merged, PRs are not being looked at… I wonder if this is still supported/maintained?

We switched to it for the Neos project because it was just so fast… and now we'll switch to git-filter-repo as that works fine on ARM64, is just way easier to use (read: no need to compile anything yourself against ageing libraries) and about as fast. And supported…

• If this project is no longer maintained: Thanks for the past, it served us well!
• If it's still alive: Good luck for the future!

Either way, it would be nice to clarify the state, so people can get their expectations right.

It possible to compile splitsh on Windows using MinGW.
Follow these steps to prepare MSYS2 enviroment:
libgit2/git2go#307
After you will be able to follow standart installation processin MSYS2 bash console:
https://github.com/splitsh/lite#installation

I would like to push the tag along with the commit when pushing to a slave repo.

Right now what I do is run splitsh-lite --prefix=lib, take the SHA1 of the commit and push the commit to the master branch of the slave repo:

git push <slave_remote> <SHA1_commit>:master

can I attach a tag to the generated commit and push it along?

While this is already awesome (in terms of speed, minus issue #20), it is rather useless without being able to also split tags… at least for us.

Instead of building tagging ourselves, is there anything else we could do, except hope? Maybe even a timeframe for the non-light version?

Or maybe we'll just email you as suggested in the readme and let you split our stuff :)

Hi,

I'm getting syntax errors when I try to get or build splitsh. These are the messages:

→ go get github.com/splitsh/lite
# github.com/splitsh/lite
src/github.com/splitsh/lite/main.go:87: syntax error: unexpected range, expecting {
src/github.com/splitsh/lite/main.go:93: non-declaration statement outside function body
src/github.com/splitsh/lite/main.go:97: non-declaration statement outside function body
src/github.com/splitsh/lite/main.go:101: non-declaration statement outside function body
src/github.com/splitsh/lite/main.go:105: non-declaration statement outside function body
src/github.com/splitsh/lite/main.go:109: non-declaration statement outside function body
src/github.com/splitsh/lite/main.go:110: syntax error: unexpected }

My go version is

→ go version
go version go1.3.3 linux/amd64
``

Unfortunately I don't have any clue about Go, so I don't know what to do at all. :(

Best regards

Suggest: create home brew package

I am getting the following error when compiling splitsh on github. here is my script:

+ go get github.com/splitsh/lite
cannot find package "github.com/libgit2/git2go/v34" in any of:
	/opt/hostedtoolcache/go/1.20.11/x64/src/github.com/libgit2/git2go/v34 (from $GOROOT)
	/home/runner/work/google-cloud-php/google-cloud-php/.split/go/src/github.com/libgit2/git2go/v34 (from $GOPATH)

mkdir $1
export GOPATH=$1/go

go env -w GO111MODULE=off
go get -d github.com/libgit2/git2go
cd $GOPATH/src/github.com/libgit2/git2go
git checkout next
git submodule update --init
make install

go get github.com/splitsh/lite
go build -o $1/splitsh-lite github.com/splitsh/lite

My github action is using ubuntu-latest. I think the issue has to do with the new tag for v2.0.0. I am actively debugging it, and plan to either follow the guide in #64 or try to roll it back.

I just wanted to submit this issue here also, in case it's helpful to someone else!

Please can someone give me a hint as to why i would be seeing this?

• It is happening for multiple packages of my monorepo.
• The SHA1s given are present in their relevant manyrepo.
• This process has been functioning very well (thanks!) for a number of months before today

It is quite possible that i have deleted a tag from the manyrepo between the last "sync" but without an understanding of this tool/subtrees I'm a little lost (no experience of Go)

It would be cool create some online service that can integrates with Github by webhooks (listen push events) and automatically split repositories by provided config (for example .split.yaml)

Hi,

I have the following error when I try to run the binary for the first time.
This error does not occur when I run the app again.

Any idea?

unexpected fault address 0x7f7641ae4008
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f7641ae4008 pc=0x4d3cd9]

goroutine 1 [running]:
runtime.throw(0x65d770, 0x5)
        /usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc42004b8c0 sp=0xc42004b8a0
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:297 +0x28c fp=0xc42004b910 sp=0xc42004b8c0
github.com/boltdb/bolt.(*Cursor).search(0xc42004bab8, 0xc42000c560, 0x14, 0x20, 0x4)
        /go/src/github.com/boltdb/bolt/cursor.go:255 +0x69 fp=0xc42004b9d8 sp=0xc42004b910
github.com/boltdb/bolt.(*Cursor).seek(0xc42004bab8, 0xc42000c560, 0x14, 0x20, 0x0, 0x0, 0x42bd12, 0x9552c0, 0x7f7641bd0000, 0x0, ...)
        /go/src/github.com/boltdb/bolt/cursor.go:159 +0xb1 fp=0xc42004ba28 sp=0xc42004b9d8
github.com/boltdb/bolt.(*Bucket).Bucket(0xc4200a40f8, 0xc42000c560, 0x14, 0x20, 0xc42004bb20)
        /go/src/github.com/boltdb/bolt/bucket.go:112 +0x108 fp=0xc42004bae8 sp=0xc42004ba28
github.com/boltdb/bolt.(*Tx).Bucket(0xc4200a40e0, 0xc42000c560, 0x14, 0x20, 0xc42001c0c0)
        /go/src/github.com/boltdb/bolt/tx.go:101 +0x4f fp=0xc42004bb20 sp=0xc42004bae8
github.com/splitsh/lite/splitter.(*cache).getHead.func1(0xc4200a40e0, 0x667400, 0xc4200a40e0)
        /go/src/github.com/splitsh/lite/splitter/cache.go:80 +0x72 fp=0xc42004bbd8 sp=0xc42004bb20
github.com/boltdb/bolt.(*DB).View(0xc42009e000, 0xc42004bc50, 0x0, 0x0)
        /go/src/github.com/boltdb/bolt/db.go:629 +0x9f fp=0xc42004bc28 sp=0xc42004bbd8
github.com/splitsh/lite/splitter.(*cache).getHead(0xc42001a930, 0x667a08)
        /go/src/github.com/splitsh/lite/splitter/cache.go:85 +0x66 fp=0xc42004bc78 sp=0xc42004bc28
github.com/splitsh/lite/splitter.(*state).pushRevs(0xc420058230, 0xc420014a50, 0x0, 0x0)
        /go/src/github.com/splitsh/lite/splitter/state.go:538 +0x235 fp=0xc42004bd08 sp=0xc42004bc78
github.com/splitsh/lite/splitter.(*state).walker(0xc420058230, 0x6675b0, 0xc420058230, 0xed16c37a9)
        /go/src/github.com/splitsh/lite/splitter/state.go:169 +0xf3 fp=0xc42004bd78 sp=0xc42004bd08
github.com/splitsh/lite/splitter.(*state).split(0xc420058230, 0x0, 0x0)
        /go/src/github.com/splitsh/lite/splitter/state.go:120 +0xa5 fp=0xc42004be28 sp=0xc42004bd78
github.com/splitsh/lite/splitter.Split(0xc420092140, 0xc42001c500, 0x0, 0x0)
        /go/src/github.com/splitsh/lite/splitter/config.go:51 +0x93 fp=0xc42004be68 sp=0xc42004be28
main.main()
        /go/src/github.com/splitsh/lite/main.go:103 +0x1ed fp=0xc42004bf88 sp=0xc42004be68
runtime.main()
        /usr/local/go/src/runtime/proc.go:185 +0x20a fp=0xc42004bfe0 sp=0xc42004bf88
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42004bfe8 sp=0xc42004bfe0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 6 [runnable]:
main.main.func1(0xc42001c540, 0xc42001c500)
        /go/src/github.com/splitsh/lite/main.go:97 +0x78
created by main.main
        /go/src/github.com/splitsh/lite/main.go:100 +0x50e

The v1.0.1 release file contains a subdirectory named . whichs breaks automated extraction via composer/Phar (see composer/composer#6992).

Could you please re-release the package or fix it in a newer version?

Thanks in advance!

Hello,

Splitsh-lite store already parsed commits on splitsh.db.
But how can I share the same history between two different places.

For example

• Bob run :
  splitsh-lite --prefix=library --target=heads/test-splitsh --progress
  and then store it on a repository :
  git push [email protected]:/library test-splitsh:master

• John wants to update the library :
  git fetch [email protected]:/library master:test-splitsh
  But he cannot run splitsh update, because it will rewrite all commits :
  splitsh-lite --prefix=library --target=heads/test-splitsh --progress

How can it be done ? Should all splitsh executions be performed from the same place?

@fabpot : How do you do for spliting symfony/symfony ?
I guess you do not use the lite version. ?

Thanks in advance

I'm using the latest Ubuntu LTS:

➜ lsb_release -a                                                                                                                     
LSB Version:	core-11.1.0ubuntu2-noarch:printing-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

In apt I found only dev and 28 version:

➜  sudo apt search libgit2
...
libgit2-28/focal,now 0.28.4+dfsg.1-2 amd64 [installed,auto-removable]
  low-level Git library

libgit2-dev/focal 0.28.4+dfsg.1-2 amd64
  low-level Git library (development files)
...

After install libgit2 from repositories I've got:

➜  lite git:(init-go-mod) go run main.go              
# github.com/libgit2/git2go
../../pkg/mod/github.com/libgit2/[email protected]/git_dynamic.go:10:3: error: #error "Invalid libgit2 version; this git2go supports libgit2 v0.27"
   10 | # error "Invalid libgit2 version; this git2go supports libgit2 v0.27"
      |   ^~~~~

hi,

have you considered remembering the config for the subrebos using a config file, similarly to git subrepo which creates a .gitrepo file located at the root of each configured subrepo ?

this way each new contributor who clones the main repo can use the subrepos right away without any further configuration.

I have GPG signed commits in the monorepo but the commits in the read-only splits are not signed. Is it a missing feature or am I doing something wrong?

I'm trying to build a subtree of the Oro Platform. However when processing certain prefixes, splitsh-lite fails.

Steps to reproduce

git clone https://github.com/orocrm/platform.git
cd platform
splitsh-lite --prefix=src/Oro/Bundle/FilterBundle --debug

Debug output

2016/07/14 20:58:19 Splitting refs/heads/master
2016/07/14 20:58:19   From "src/Oro/Bundle/FilterBundle" to "ROOT"
2016/07/14 20:58:19 Processing commit: aaa7ab6635cc69cf40121d4cd7c5e5dcffccb524
2016/07/14 20:58:19   parents:
2016/07/14 20:58:19   newparents:
The requested type does not match the type in the ODB

Whether the bug occurs seems to depend on the prefix (e.g. src/Oro/Bundle/NoteBundle works as expected).

I tried to use splitsh-lite to split the monolithic development repositories of Neos CMS into the readonly package repositories, but I end up with different commit hashes compared to the current approach using git subtree split.

Versions used (ubuntu 14.04), splitsh-lite compiled today according to instructions in readme

$ git --version
git version 1.9.1
$ go version
go version go1.6 linux/amd64

To reproduce:

$ git clone https://github.com/neos/neos-development-collection.git neos-dev
$ cd neos-dev
$ splitsh-lite --prefix=TYPO3.Neos --target=refs/heads/split-neos-master-splitsh --origin=origin/master --git '<2.8.0' --scratch
7197 commits created, 9984 commits traversed, in 4.09s
e494289e6d34edec5b35a0add75ce87e73b344c2
$ git subtree split --prefix="TYPO3.Neos" --branch=split-neos-master-subtree origin/master
Created branch 'split-neos-master'
c3034f367ecb1e4f5b4e5a36b6fe15d0be06cabd
$ git rev-list --count split-neos-master-splitsh
7182
$ git rev-list --count split-neos-master-subtree
7182
$ git log --oneline --topo-order split-neos-master-splitsh | tail -n 5
fd5d14c * FLOW3: (Component) Removed support for setter injection via generic "setProperty()" method. * FLOW3: (Component) Setter injection is now also possible via a "inject" method, e.g. injectSomeDependency() (instead of setSomeDependency). All methods whose name starts with "inject" will be included into the autowiring procedure. Therefore dependencies don't need to be configured explicitly in the Components.ini when "inject" methods are used. Resolves #319. * FLOW3: (Component) Added autowiring support for "inject" setter methods. Addresses #319. * Smarty: Added check against multiple definition of "SMARTY_DIR" * Testing, TYPO3, PhoneBookTutorial: Refactored to the new "inject*" methods in the action controllers. Original-Commit-Hash: 14ceca885be55807c8584116b7eeabfa1b710622
cccdefc * FLOW3: (MVC) Improved the directory structure and naming of views. They now have the name of the controller in their own name. Fixes #314. * Updated the views in other packages accordingly. * Fixed whitespace and inline doc issues in classes I came across. Original-Commit-Hash: 8919dc4a43eb1eb0a5756f27b9cf123ef5bbc960
5ad77ca * FLOW3: (Package) Moved the Components.conf file from the Classes/ directory of each package to a new Configuration/ directory and renamed the file to Components.ini. Closes #309. * FLOW3: Updated the documentation accordingly. Original-Commit-Hash: bbae6513443e2297eec210c08e3ddf9aacf8d37c
6cba34a * FLOW3: Fixed whitespaces and inline comments where I came across * FLOW3: (MVC) Added two new exceptions: NoSuchController and InvalidController (for Dispatcher) * FLOW3: (MVC) Added an initial implementation of a router for web requests. Tests are still in the Web Request Builder test case but will be refactored soon. * TYPO3: A bit of cleanup and reviving of the Page controller * Testing: Changed "POST" to "post" in MainLayout.html. That fixes #264, but in general it's too early to look at errors of this kind because the new Testrunner will still change a lot. Original-Commit-Hash: 984b039f4a2a82d1cbee3e89a47c3b43f9752c3c
a558e62 Subversion reorganisation, part 5:  * Moved some directories again to create a structure agreed upon by the RnD committee Original-Commit-Hash: 22915448b2c9da021015c59f680cecde4e071849
$ git log --oneline --topo-order split-neos-master-subtree | tail -n 5
f370041 * FLOW3: (Component) Removed support for setter injection via generic "setProperty()" method. * FLOW3: (Component) Setter injection is now also possible via a "inject" method, e.g. injectSomeDependency() (instead of setSomeDependency). All methods whose name starts with "inject" will be included into the autowiring procedure. Therefore dependencies don't need to be configured explicitly in the Components.ini when "inject" methods are used. Resolves #319. * FLOW3: (Component) Added autowiring support for "inject" setter methods. Addresses #319. * Smarty: Added check against multiple definition of "SMARTY_DIR" * Testing, TYPO3, PhoneBookTutorial: Refactored to the new "inject*" methods in the action controllers. Original-Commit-Hash: 14ceca885be55807c8584116b7eeabfa1b710622
84d63de * FLOW3: (MVC) Improved the directory structure and naming of views. They now have the name of the controller in their own name. Fixes #314. * Updated the views in other packages accordingly. * Fixed whitespace and inline doc issues in classes I came across. Original-Commit-Hash: 8919dc4a43eb1eb0a5756f27b9cf123ef5bbc960
f9aa0c9 * FLOW3: (Package) Moved the Components.conf file from the Classes/ directory of each package to a new Configuration/ directory and renamed the file to Components.ini. Closes #309. * FLOW3: Updated the documentation accordingly. Original-Commit-Hash: bbae6513443e2297eec210c08e3ddf9aacf8d37c
644d9a2 * FLOW3: Fixed whitespaces and inline comments where I came across * FLOW3: (MVC) Added two new exceptions: NoSuchController and InvalidController (for Dispatcher) * FLOW3: (MVC) Added an initial implementation of a router for web requests. Tests are still in the Web Request Builder test case but will be refactored soon. * TYPO3: A bit of cleanup and reviving of the Page controller * Testing: Changed "POST" to "post" in MainLayout.html. That fixes #264, but in general it's too early to look at errors of this kind because the new Testrunner will still change a lot. Original-Commit-Hash: 984b039f4a2a82d1cbee3e89a47c3b43f9752c3c
a558e62 Subversion reorganisation, part 5:  * Moved some directories again to create a structure agreed upon by the RnD committee Original-Commit-Hash: 22915448b2c9da021015c59f680cecde4e071849

As you can see in the git log snippets, the first commit is the same, but from the second commit on the hashes differ. I'm not sure what further information to include or how to analyze this problem further. Do you have any idea what could cause this?