Current problem
The size of this repository is much larger than it should be.
$ du -sh terraform-google-gcloud
1.3G terraform-google-gcloud
$ time git clone https://github.com/terraform-google-modules/terraform-google-gcloud.git
Cloning into 'terraform-google-gcloud'...
remote: Enumerating objects: 636, done.
remote: Total 636 (delta 0), reused 0 (delta 0), pack-reused 636
Receiving objects: 100% (636/636), 1.23 GiB | 11.51 MiB/s, done.
Resolving deltas: 100% (292/292), done.
Updating files: 100% (62/62), done.
git clone 10.06s user 9.24s system 17% cpu 1:52.93 total
It takes almost 2 minutes to clone from GitHub with a 150mbps connection.
After cloning I investigated the cause and it looks like google-cloud-sdk.tar.gz
has been added to the repository in multiple different versions, both for Linux and Darwin.
git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest | tail -50
6ef2f3411f67 4.9KiB main.tf
398085a14170 5.0KiB main.tf
8e5ef607508e 6.0KiB main.tf
3d845d8691ea 6.0KiB main.tf
4f3606e5e4f3 6.1KiB main.tf
a1e9f2110c7d 6.2KiB main.tf
318d0ea031f7 6.3KiB main.tf
1e560423e5df 6.8KiB main.tf
cc752fe0a126 6.9KiB main.tf
548021d21412 7.4KiB main.tf
6258b131c2d9 7.5KiB main.tf
e83e8397560a 8.9KiB main.tf
261eeb9e9f8b 11KiB LICENSE
d64569567334 11KiB LICENSE
d363c8a22ba7 844KiB cache/darwin/jq
f48b0ca92144 3.8MiB cache/linux/jq
75fd8ba99519 21MiB cache/darwin/google-cloud-sdk.tar.gz
8c9828a79cc1 21MiB cache/darwin/google-cloud-sdk.tar.gz
0de79388f3ff 22MiB cache/darwin/google-cloud-sdk.tar.gz
8e63c3b02a91 22MiB cache/darwin/google-cloud-sdk.tar.gz
a7e4b6be7829 22MiB cache/darwin/google-cloud-sdk.tar.gz
63002ef2c81d 22MiB cache/darwin/google-cloud-sdk.tar.gz
14171ecf8ace 22MiB cache/darwin/google-cloud-sdk.tar.gz
6afd38c96116 22MiB cache/darwin/google-cloud-sdk.tar.gz
b78769c44c12 22MiB cache/darwin/google-cloud-sdk.tar.gz
cc42739faf5f 28MiB cache/linux/google-cloud-sdk.tar.gz
c5f94c7cf7a2 28MiB cache/linux/google-cloud-sdk.tar.gz
8d0ac84483b2 29MiB cache/linux/google-cloud-sdk.tar.gz
6f3ced3b0c2d 29MiB cache/linux/google-cloud-sdk.tar.gz
2be0be2e1ead 29MiB cache/linux/google-cloud-sdk.tar.gz
3b288442db09 29MiB cache/linux/google-cloud-sdk.tar.gz
d9ae2e931b9e 29MiB cache/linux/google-cloud-sdk.tar.gz
113c1d621257 29MiB cache/linux/google-cloud-sdk.tar.gz
b042b74c3ce9 29MiB cache/linux/google-cloud-sdk.tar.gz
4a7bff9d634c 46MiB cache/darwin/google-cloud-sdk.tar.gz
d7d8e990a635 47MiB cache/darwin/google-cloud-sdk.tar.gz
cbed5f49838e 47MiB cache/darwin/google-cloud-sdk.tar.gz
9e243acd0dce 47MiB cache/darwin/google-cloud-sdk.tar.gz
ea809a943941 47MiB cache/darwin/google-cloud-sdk.tar.gz
d69e3b0586ee 47MiB cache/darwin/google-cloud-sdk.tar.gz
b3a6db909e8a 48MiB cache/darwin/google-cloud-sdk.tar.gz
8215801b7a25 48MiB cache/darwin/google-cloud-sdk.tar.gz
ef4529f27bab 53MiB cache/linux/google-cloud-sdk.tar.gz
16fb075994b2 54MiB cache/linux/google-cloud-sdk.tar.gz
3805942c637d 54MiB cache/linux/google-cloud-sdk.tar.gz
d7a7fe100c77 54MiB cache/linux/google-cloud-sdk.tar.gz
80d8c8e72df4 54MiB cache/linux/google-cloud-sdk.tar.gz
d4efd180bd7e 54MiB cache/linux/google-cloud-sdk.tar.gz
ca7c1d74d521 54MiB cache/linux/google-cloud-sdk.tar.gz
1cc4ef259c43 55MiB cache/linux/google-cloud-sdk.tar.gz
Suggested solution
Stop adding the large "cache files" to the repository by adding the cache directory to .gitignore
:
echo cache/ >> .gitignore
Fix history
Caution: this will rewrite history, so if this is done all clones needs to be re-cloned.
I'd like to suggest that you might want to consider cleaning up the historic commits as well.
Example using bfg (first get a bare/mirror repo):
$ git clone --mirror https://github.com/terraform-google-modules/terraform-google-gcloud
Cloning into bare repository 'terraform-google-gcloud.git'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 691 (delta 0), reused 0 (delta 0), pack-reused 690
Receiving objects: 100% (691/691), 1.46 GiB | 11.68 MiB/s, done.
Resolving deltas: 100% (313/313), done.
$ bfg --strip-blobs-bigger-than 1M --no-blob-protection terraform-google-gcloud.git
Using repo : /private/tmp/terraform-google-gcloud.git
Scanning packfile for large blobs: 691
Scanning packfile for large blobs completed in 72 ms.
Found 41 blob ids for large blobs - biggest=57160066 smallest=3953824
Total size (unpacked)=1586445370
Found 0 objects to protect
Found 46 commit-pointing refs : HEAD, refs/heads/bugfix/gcloud_delete_cyclic_dependencies, refs/heads/create-pull-request/patch-gcloud-version, ...
Protected commits
-----------------
You're not protecting any commits, which means the BFG will modify the contents of even *current* commits.
This isn't recommended - ideally, if your current commits are dirty, you should fix up your working copy and commit that, check that your build still works, and only then run the BFG to clean up your history.
Cleaning
--------
Found 140 commits
Cleaning commits: 100% (140/140)
Cleaning commits completed in 1,476 ms.
Updating 45 Refs
----------------
Ref Before After
-------------------------------------------------------------------------
refs/heads/bugfix/gcloud_delete_cyclic_dependencies | 8dce5f1b | 2e945858
refs/heads/create-pull-request/patch-gcloud-version | fb79cf05 | ec76a6e8
refs/heads/feature/ordering | 7863d696 | 6cf49db5
refs/heads/feature/skip_download | 8a12e54e | d2dfb194
refs/heads/fix/triggers | aca620bd | c425121b
refs/heads/master | f4c9d56c | 99f3206f
refs/heads/release-v0.5.0 | 366035af | f55d423a
refs/heads/release-v0.5.1 | 893a9512 | 2e7e1aee
refs/pull/1/head | 7b1ea507 | 19cfb315
refs/pull/11/head | 9d65bec4 | 4df50447
refs/pull/12/head | 4d633f38 | 1982a584
refs/pull/13/head | a21a482c | 2d9c7a89
refs/pull/14/head | c69ca283 | 91539319
refs/pull/15/head | fd7feaf4 | 3b21da95
refs/pull/16/head | 4a824c26 | 61173d26
...
Updating references: 100% (45/45)
...Ref update completed in 283 ms.
Commit Tree-Dirt History
------------------------
Earliest Latest
| |
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
D = dirty commits (file tree fixed)
m = modified commits (commit message or parents changed)
. = clean commits (no changes to file tree)
Before After
-------------------------------------------
First modified commit | 7b1ea507 | 19cfb315
Last dirty commit | 8b6ccc89 | 1c1c069c
Deleted files
-------------
Filename Git id
---------------------------------------------------------------------
google-cloud-sdk.tar.gz | ea809a94 (47.2 MB), b042b74c (29.3 MB), ...
jq | f48b0ca9 (3.8 MB)
In total, 318 object ids were changed. Full details are logged here:
/private/tmp/terraform-google-gcloud.git.bfg-report/2020-04-07/08-20-16
BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive
$ cd terraform-google-gcloud.git
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Enumerating objects: 691, done.
Counting objects: 100% (691/691), done.
Delta compression using up to 8 threads
Compressing objects: 100% (597/597), done.
Writing objects: 100% (691/691), done.
Selecting bitmap commits: 131, done.
Building bitmaps: 100% (105/105), done.
Total 691 (delta 330), reused 108 (delta 0), pack-reused 0
The bare repo is now less than 500 KB:
du -sh terraform-google-gcloud.git
496K terraform-google-gcloud.git
A clone of the cleaned up bare repo shows less than 2 MB:
$ du -sh terraform-google-gcloud.cleaned
1.6M terraform-google-gcloud.cleaned