jyknight / llvm-git-migration Goto Github PK

zip-downstream-fork.py assumes an umbrella respository with submodules in places corresponding to the monorepo, i.e.::

umbrella
compiler-rt
llvm

It should handle other mappings, particularly the default nested layout expected by cmake::

umbrella
llvm
tools
clang
projects
compiler-rt

The easiest way to do this would be to have the user provide a mapping file from submodule path to monorepo path, for example::

llvm -> llvm
llvm/tools/clang -> clang
llvm/projects/compiler-rt -> compiler-rt

This would allow flexibility to handle third-party submodules as well, potentially mapping them to specific places in the monorepo. The default mapping would be simply::

clang -> clang
compiler-rt -> compiler-rt
llvm -> llvm

If the umrella repository used for zipping doesn't contain submodules for all projects in the monorepo, projects not included will not appear in the zipped history. This may or may not be what the user wants, since the zipped branch presumably should represent what downstream actually works on.

Think about adding an option to include other subprojects at the most recent branch point of the umbrella submodules.

There are commits in the official monorepo (https://github.com/llvm/llvm-project.git) which seem to map to more than one svn revision.

Specifically, git commit 2f2aeb9b has the following log:

Creating release_33 branch

llvm-svn: 181277
llvm-svn: 181274
llvm-svn: 181273

find_svnrev returns 181273 for this commit.

Our local repos have a branch off of f9abf237 in the llvm mirror's clang repository (https://github.com/llvm-mirror/clang.git). find_svnrev returns 181274 for this commit. Thus, when migrate-downstream-fork encounters f9abf237, the lookup in svn_to_newrev fails because there's no entry for 181274.

To reproduce:

# Get the migration scripts
git clone https://github.com/jyknight/llvm-git-migration.git

# Set up a downstream repo
git clone https://github.com/llvm-mirror/clang.git downstream_clang
cd downstream_clang
git checkout -b downstream_branch f9abf23757
git commit --allow-empty -m "Downstream commit"
cd ..

# Create the migration repo
# Taken from the script at https://llvm.org/docs/Proposals/GitHubMove.html#basic-migration
mkdir migrate-repo
cd migrate-repo
git init
git remote add upstream/monorepo https://github.com/llvm/llvm-project.git
git remote add upstream/split/clang https://github.com/llvm-mirror/clang.git
git remote add local/split/clang ../downstream_clang
git fetch --all

# Run the script
../llvm-git-migration/migrate-downstream-fork.py \
    refs/remotes/local/split/downstream_branch \
    refs/tags \
    --new-repo-prefix=refs/remotes/upstream/monorepo \
    --old-repo-prefix=refs/remotes/upstream/split \
    --source-kind=merge-split

The solution may just be as easy as having find_svnrev pull all matches and return them as a list.

I tried the migration script on our out-of-tree llvm clone.
Got some problems due to not handling
header_kind == 'encoding'
here.

I made a simple workaround locally to just do "pass" instead of raising the exception, just to be able to move forward with evaluation of the monorepo conversion. Not sure if that is an appropriate solution to incorporate here.

If a submodule contains another submodule, zip-downstream-fork.py should recurse and find all submodules. This is important to handle the standard nested layout (clang inside llvm/tools/clang, etc.).

When I pass --revmap-out to migrate-downstream-fork.py git-fast-import ends up aborting with the error "Unsupported command get-mark :99."

Mark 99 exists in the fast-import crash dump. The get-mark command appears just after a tag command.

Currently zip-downstream-fork.py doesn't update any tags or other refs pointing to commits referenced by submodules. This is because the tool only rewrites commits in the umbrella repository, not the subproject commits themselves.

The tool could update the revmap to point the submodule-referenced commits to the corresponding rewritten umbrella commit. Then fast_filter_branch.py's ref rewriting should naturally process tags and other refs correctly.

When testing this script on our out-of-tree llvm clone I ended up triggering this exception.

Maybe this should be a warning instead, and then the heuristic could be to just continue (as the code below is using "candidate_upstreams[0]"?
Or do you think that I am in deep trouble if doing it like that?

What is the intended workflow for using this tool to migrate projects to their new official git repositories that have not become part of the monorepo (like lnt or test-suite)?

Currently zip-downstream-fork.py discards bits of trees that aren't submodule updates and entire commits updating only such trees. It should instead place these trees under a user-specified subdirectory. Then the tool would not have to discard commits.