sinbad / lfs-folderstore Goto Github PK
View Code? Open in Web Editor NEWgit-lfs custom transfer adapter which simply uses a folder as the remote LFS media store (e.g. a shared NAS folder)
git-lfs custom transfer adapter which simply uses a folder as the remote LFS media store (e.g. a shared NAS folder)
The NAS idea is really good approach .
Not sure if anyone just like me , feel FTP/SFTP connection easy than NAS connection to global developer for sharing.
config FTP/SFP specific uid/pwd in config.
[Ubuntu install]
1.sudo apt install -y git
2.sudo apt-get install -y git-lfs
3.wget https://github.com/sinbad/lfs-folderstore/releases/download/v1.0.0/lfs-folderstore-linux-amd64-v1.0.0.zip
4.sudo unzip -j lfs-folderstore-linux-amd64-v1.0.0.zip lfs-folderstore-linux-amd64/lfs-folderstore -d /usr/bin
5.sudo chmod +x /usr/bin/lfs-folderstore
[lfs-mount-via ftp]
1.sudo apt-get install -y curlftpfs
2.sudo mkdir /mnt/lfs-db
3.sudo curlftpfs -o allow_other lfsuser:[email protected] /mnt/lfs-db
[git clone]
export GIT_SSL_NO_VERIFY=1
export GIT_LFS_SKIP_SMUDGE=1
git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://github.com/XXXXXX/LFS_TEST.git mylfs
Cloning into 'mylfs'...
Username for 'https://github.com': XXXXXX
Password for 'https://[email protected]': *******
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 9 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (9/9), done.
cd mylfs
git config --add lfs.customtransfer.lfs-folder.path lfs-folderstore
git config --add lfs.customtransfer.lfs-folder.args "/mnt/lfs-db"
git config --add lfs.standalonetransferagent lfs-folder
git lfs pull
Git LFS: (3 of 3 files) 115.94 MB / 115.94 MB
Hi,
I'm trying to compile from scratch as the macOS binary does not seem to be working. However, when running go build
, it ask for a go.mod
file.
I don't have any experience with Go, but after some googling I have figured out a workaround
go mod init github.com/sinbad/lfs-folderstore
go mod tidy
I'm assuming that the first line of code creates a simple go.mod
and the second line writes all the dependency of this repo to go.mod
. But I'm not sure. Can someone please verify?
Reference: https://golang.org/doc/tutorial/getting-started
Regards,
Kenyon
I am not sure if you are maintaining this anymore but it could be super helpful for all of my CSVs for doing data science work.
However, I am running in to
EOFoading LFS objects: 100% (51/51), 283 MB | 0 B/s
error: failed to push some refs to 'github.com:langdon/my-private-repo.git'
I am running on linux and have tried existing and new repos with no luck.
Any ideas?
In reading the instructions, I was surprised by the step to set up a remote. My mental model for the intended use case was the ability to use git's excellent version control capabilities locally, in a repo that has large files. (For example, a git repo to track personal notes, stored as a collection of markdown files and images.)
After research and reflection, I believe the reason is because git lfs
doesn't actually push the files to the LFS repository until you push your commits to a remote. I inferred this from reading this git LFS doc.
If one doesn't want to use a remote server at all, I believe one can simply create a "local remote" by making a folder, doing git init --bare
, and setting that folder as the remote, and pushing to it.
What happens if we never push, because we're purely using git
for version control? Based on reading that same LFS doc, I believe what would happen is that the file would get stored in the LFS cache, in .git/lfs/
, at commit time. It would not be pushed to the folder we designated for our LFS store. As soon as we push our commits, the files get pushed from the LFS cache to the LFS store.
I tried this out, and it seemed to work as I expected.
Does my understanding sound about right? Have I missed anything significant?
If I'm right: do you think it is worth saying something in the README about why we need to set up the remote, to help other users who are surprised as I initially was? (Maybe you can just link to this issue for the explanation if I'm not too far off. ๐)
Hi,
I've been using lfs-folderstore for a few weeks and I ran into an issue last week. I was using it on a Gitlab.com repository for some time and it worked fine. One day my co-worker did a push and I pulled a bit sooner than the files where synced and got an error. Since then I can't get a working pull.
Here is our setup :
We have a comon directory on Mega which is synced on our computers.
We both have git for windows installed and git lfs and lfs-folderstore installed too.
We do our push to Gitlab.com, lfs-folderstore is set to the synced Mega folder.
$ git config -l http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt http.sslbackend=openssl diff.astextplain.textconv=astextplain filter.lfs.clean=git-lfs clean -- %f filter.lfs.smudge=git-lfs smudge -- %f filter.lfs.process=git-lfs filter-process filter.lfs.required=true core.autocrlf=true core.fscache=true core.symlinks=false core.editor="C:\\Program Files\\Microsoft VS Code\\Code.exe" --wait pull.rebase=false credential.helper=manager-core credential.https://dev.azure.com.usehttppath=true init.defaultbranch=master user.name=Tarrke [email protected] filter.lfs.clean=git-lfs clean -- %f filter.lfs.smudge=git-lfs smudge -- %f filter.lfs.process=git-lfs filter-process filter.lfs.required=true credential.helper=manager pull.rebase=true rebase.autostash=true merge.autostash=true core.repositoryformatversion=0 core.filemode=false core.bare=false core.logallrefupdates=true core.symlinks=false core.ignorecase=true [email protected]:exposuregames/lotts.git remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master lfs.customtransfer.lfs-folder.path=lfs-folderstore lfs.customtransfer.lfs-folder.args=/F/MegaSyncLOTTS/ lfs.standalonetransferagent=lfs-folder
What happens:
When my co-worker pushes and pulls, no problems. When I pull I got :
Error downloading object: Content/RLContent/Alana/Alana_fbm/Std_Eye_R_Pbr_Normal.uasset (800e187): Smudge error: Error downloading Content/RLContent/Alana/Alana_fbm/Std_Eye_R_Pbr_Normal.uasset (800e187a618a9975f9b4f9761b8797ea997575bb72bc98d611b16dbc13d01dd8): error transferring "800e187a618a9975f9b4f9761b8797ea997575bb72bc98d611b16dbc13d01dd8": [5] Error creating temp file for "F:\MegaSyncLOTTS\80\0e\800e187a618a9975f9b4f9761b8797ea997575bb72bc98d611b16dbc13d01dd8": open F:\UE Project\lotts.ter.git\lfs\tmp\800e187a618a9975f9b4f9761b8797ea997575bb72bc98d611b16dbc13d01dd8.tmp: The file exists.
Errors logged to F:\UE Project\lotts.ter.git\lfs\logs\20210221T205034.8349272.log
Sometimes the temporary file linked exists, sometimes not. Even if I remove the temporary files I get this. I can't pull, so I can't push either to the repository.
`Tarrke@Korriban MINGW64 /f/UE Project/lotts.ter (master)
$ GIT_TRACE=1 git pull origin master
21:09:22.456514 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
21:09:22.460506 git.c:444 trace: built-in: git pull origin master
21:09:22.464519 run-command.c:664 trace: run_command: git merge-base --fork-point refs/remotes/origin/master master
21:09:22.519399 run-command.c:664 trace: run_command: git fetch --update-head-ok origin master
21:09:22.551270 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
21:09:22.556233 git.c:444 trace: built-in: git fetch --update-head-ok origin master
21:09:22.569302 run-command.c:664 trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '''exposuregames/lotts.git''''
21:09:24.676901 run-command.c:664 trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
21:09:24.720565 run-command.c:664 trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
21:09:24.744857 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
21:09:24.747817 git.c:444 trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
From gitlab.com:exposuregames/lotts
Errors logged to F:\UE Project\lotts.ter.git\lfs\logs\20210221T210930.4433789.log
Use git lfs logs last
to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: Content/RLContent/Alana/Alana_fbm/Std_Eye_R_Pbr_Normal.uasset: smudge filter lfs failed
error: could not detach HEAD`
Thanks so much for this helper, it's been of great use to me and my small teams. On Windows it is a seamless process, and makes working with videos so much easier.
For Macs...I saw that you had mentioned testing it with success...what is the installation process? I have added the path to the downloaded "darwin" release using "sudo nano /etc/paths" but still receive an error when pushing as if it can't find the helper.
Is there something missing? Wanted to ask this in case others had a similar issue when installing for Mac, as it might be a welcome addition to the readme. :)
Thanks again for everything!
Hi,
I cant get "git lfs migrate import" comand to work. I need to move files from my repo to LFS. Is this becuase I use the standalonetransferagent? If yes, Is there a similar command that works with standalonetransferagent?
Hi!
I tried using lfs-folderstore using the Readme instructions, but I get an error.
I used a simple test repo with a single file.
git version 2.25.1
git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)
lfs-folderstore build with 'go build'
git init
git lfs install
git add .
git commit -m "."
git config --add lfs.customtransfer.lfs-folder.args PATH
git config --add lfs.customtransfer.lfs-folder.path lfs-folderstore
git config --add lfs.standalonetransferagent lfs-folder
git push --set-upstream origin master
EOFoading LFS objects: 100% (1/1), 43 MB | 0 B/s
error: failed to push some refs to 'https://github.com/sbrt/test.git'
Using github LFS works fine.
Any guess what might be causing this, other than version incompatibilities?
Hey there,
Thanks for writing this nifty tool. I tried setting things up for my game as you described in the readme, but am getting the following error when executing git push origin master
git push origin master โโฏ
Uploading LFS objects: 100% (1033/1033), 2.4 GB | 205 MB/s, done.
Enumerating objects: 5327, done.
Counting objects: 100% (5327/5327), done.
Delta compression using up to 16 threads
Compressing objects: 100% (5324/5324), done.
Writing objects: 100% (5327/5327), 17.38 MiB | 8.17 MiB/s, done.
Total 5327 (delta 2668), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2668/2668), done.
remote: error: GH008: Your push referenced at least 40 unknown Git LFS objects:
remote: d2d36233d085db6ebd5aa073a3fff219f1c67e0e7a2243da1b2f761cdc404290
remote: 8ba285d22695b67ff5fcde886abf789d06680ee564f9f079d5d2e7cbab35931c
remote: 80ee32ab0756c44afc3cb8e2e0f138c9f78dc687387e5bc653c2c6cc3f033510
remote: ...
remote: Try to push them with 'git lfs push --all'.
To https://github.com/myGameUrl.git
! [remote rejected] master -> master (pre-receive hook declined)
Any ideas why this could be happening? I did try to push them with git lfs push --all
, but it makes no difference.
All the files are in my lfs-folder as expected.
I would like to use hard-linking instead of copies to link my LFS files (hard disk space is unfortunately at a premium for me in my particular application). From the README, you mention that there might be file integrity concerns regarding the hard-linking approach, and I was wondering if you could elaborate what those issues might be? I'd like to understand more before I look into changes that I might make to (my own fork of) lfs-folderstore
in order to hard-link instead of copy (if this is possible?).
Some background:
lfs-folderstore
to save my large files to a Dropbox folder. Dropbox is synchronizing those files in the background.I'd love to hear any intuition that you might have on why hard-linking would be a bad idea, or why it was a bad idea when you made lfs-folderstore
Thanks!
PS: This is a brilliant program. It "just worked" from the get-go. Brilliant!
Hi, I don't know if this is an issue or a configuration problem on my side. I follow the steps in the section "Cloning a repo" but when I run the command "git reset --hard master" got this error:
Error downloading object: C0139.MP4 (ee6d455): Smudge error: Error downloading C0139.MP4 (ee6d4556b6bfff6d338ce58f4f52ed349127bcf1b1e0772ccc0061fe957d3881): Error transferring "ee6d4556b6bfff6d338ce58f4f52ed349127bcf1b1e0772ccc0061fe957d3881": [3] Cannot stat "/Users/diego/Desktop/git_bkp/git_storage/lfs/ee/6d/ee6d4556b6bfff6d338ce58f4f52ed349127bcf1b1e0772ccc0061fe957d3881": stat /Users/diego/Desktop/git_bkp/git_storage/lfs/ee/6d/ee6d4556b6bfff6d338ce58f4f52ed349127bcf1b1e0772ccc0061fe957d3881: no such file or directory
Looking into the LFS storage folder I found the name of the file is "4556b6bfff6d338ce58f4f52ed349127bcf1b1e0772ccc0061fe957d3881" and no "ee6d4556b6bfff6d338ce58f4f52ed349127bcf1b1e0772ccc0061fe957d3881, so if I change the name of the file in the storage and add "ee6d" at the beginning, it works.
Thanks!
I get the following error when trying to pull on any machine where the repo is not on the C: drive:
Error downloading object: 4.jpg (ea25c1b): Smudge error: Error downloading 4.jpg (ea25c1b7291fb0908c26c56c784fa35e419d91d9d1e9607981ad0198206ab51b): Failed to copy downloaded file: cannot replace "Q:\<pathBlabla>\\.git\\lfs\\objects\\ea\\25\\ea25c1b7291fb0908c26c56c784fa35e419d91d9d1e9607981ad0198206ab51b" with "C:\\Users\\DAVIDF~1\\AppData\\Local\\Temp\\lfsfolderstore176468347": rename C:\Users\DAVIDF~1\AppData\Local\Temp\lfsfolderstore176468347 Q:\<pathBlabla>\.git\lfs\objects\ea\25\ea25c1b7291fb0908c26c56c784fa35e419d91d9d1e9607981ad0198206ab51b: The system cannot move the file to a different disk drive.
is there a chance to set the temp folder to the same drive that the repo is or make this work across drives?
!!Need help!!
git config --add lfs.customtransfer.lfs-folder.args "C:/path/to/your/folder"
'git push origin master' breaks if the path has spaces in between example "C:/path with space/to/your/folder"
I'm relatively new to git, been using it for the last few months, and I'm completely new to LFS. I'm working on learning game dev with Unity and had LFS configured for github but quickly exceeded the storage limit. I am trying to accomplish having my code repo point to github while the lfs repo is on a local folder where Dropbox will backup the large files. Your solution seems to fit my needs perfectly, but I can't get it to work.
Here are some relevant details:
Windows 10
git v2.21.0.windows.1
lfs v2.7.1
code base (lfs pointers) - github.com
lfs files - d:dropbox\dev\lfs\sandbox
I can't get the git remote add folderremote <url>
command to take. I read that remotes can use file://
and tried file:////d:/dropbox/dev/lfs/sandbox
, however, git lfs env
displays it as Endpoint=https://d////Dropbox/Dev/LFS/Sandbox (auth=none)
followed by SSH=d:///Dropbox/Dev/LFS/Sandbox
. Then when I push I get an error saying ssh: Could not resolve hostname d: ...
.
When I take the file://
out then LFS config usually reports Endpoint=<unknown> (auth=none)
and Endpoint (folderremote)=<unknown> (auth=none)
.
This leads to another point of confusion for me, the lfs.<url>.standalonetransferagent
command. I assume for a file location it would look something like lfs.d:/dropbox/dev/lfs/sandbox.standalonetransferagent=lfs-folder
.
What am I missing?
While typing this I thought about trying to use the Dropbox URL for my remote URL. I'll test that next.
lfs-folderstore/service/service.go
Lines 190 to 194 in 1f14699
When I try to push I get an error saying Cannot create dir "D:\LFSPath\b8\69\": mkdir D:\LFSPath\b8: Cannot create a file when that file already exists.
I believe the error comes from the code above. As seen from the part of the git trace log below, 2 files with hashes starting with b8
begin to be processed simultaneously before any of them is completed. Therefore the first one is completed successfully while the second one is not. But when 2 files with hashes starting with the same two letters are tried to be uploaded after one of them is completed, there is no error. That is why I think it is because they begin to be processed simultaneously. But I try to push again the error disappears.
14:25:44.592584 trace git-lfs: xfer[lfs-folderstore]: Received upload request for b870643b7cd0586c672b60551ab009757df14cc90b155e2f777c66617c45a0a7
14:25:44.599585 trace git-lfs: xfer[lfs-folderstore]: Received upload request for b8699ae114979db009d1bae81fcfda8e9ebf8c5577a449cd17143c05d5c28a4d
14:25:44.604589 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"complete","oid":"b8699ae114979db009d1bae81fcfda8e9ebf8c5577a449cd17143c05d5c28a4d","error":{"code":14,"message":"Cannot create dir \"D:\\\\LFSPath\\\\b8\\\\69\": mkdir D:\\LFSPath\\b8: Cannot create a file when that file already exists."}}
14:25:44.604589 trace git-lfs: xfer[lfs-folderstore]: Received upload request for ed2aa2b7ddc90258333d767499c2b02cbcdfcbaed3f1a3de3ef62a1c895f2db9
14:25:44.609586 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"complete","oid":"5ba2abfcb8cceb9067b1f72eedf4b1f4f55e7dd58cd7190356e8499259a2bf46"}
14:25:44.610585 trace git-lfs: xfer[lfs-folderstore]: Received upload request for 031d367026ddf265c70764075ea403979e804e641a0746174a4bcbe65439a2b4
14:25:44.847643 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"e144b75053622a9a3f6ffb305e7bcb3fc9b9f2b4fdf74f24d74f246d1c61f769","bytesSoFar":618828,"bytesSinceLast":618828}
14:25:44.847643 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"8e67c9186c7a16d1e092eb5d5bb1dfe888acfcddb06c8b977a634cf8702fef41","bytesSoFar":589340,"bytesSinceLast":589340}
14:25:44.848645 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"05ff58f0ec43dacee595480219512ee73c3da664143cdc3239271c2cba8d3392","bytesSoFar":589000,"bytesSinceLast":589000}
14:25:44.848645 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"c8660108abd9f5b531021045f6db2cb2c51f76ecf2191493e6e56cfa970210f9","bytesSoFar":617555,"bytesSinceLast":617555}
14:25:44.849644 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"7161c96ca48a606ba9ec3aa0be272945e1f6cd3b3801a37fbcc8bb513b043260","bytesSoFar":587105,"bytesSinceLast":587105}
14:25:44.849644 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"b870643b7cd0586c672b60551ab009757df14cc90b155e2f777c66617c45a0a7","bytesSoFar":569586,"bytesSinceLast":569586}
14:25:44.853645 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"ed2aa2b7ddc90258333d767499c2b02cbcdfcbaed3f1a3de3ef62a1c895f2db9","bytesSoFar":535317,"bytesSinceLast":535317}
14:25:44.854645 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"progress","oid":"031d367026ddf265c70764075ea403979e804e641a0746174a4bcbe65439a2b4","bytesSoFar":534677,"bytesSinceLast":534677}
14:25:44.872374 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"complete","oid":"e144b75053622a9a3f6ffb305e7bcb3fc9b9f2b4fdf74f24d74f246d1c61f769"}
14:25:44.872374 trace git-lfs: xfer[lfs-folderstore]: Received upload request for ad82ea138ddd7d2f0c8f1aef711f22fdc31e778521c5a7ce73a0ceb19656482c
14:25:44.873892 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"complete","oid":"8e67c9186c7a16d1e092eb5d5bb1dfe888acfcddb06c8b977a634cf8702fef41"}
14:25:44.873892 trace git-lfs: xfer[lfs-folderstore]: Received upload request for 4be5fc87ec70133d03f65d474eb9166bce340531448bbcf8fb2c7527bdaff2d3
14:25:44.875892 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"complete","oid":"05ff58f0ec43dacee595480219512ee73c3da664143cdc3239271c2cba8d3392"}
14:25:44.875892 trace git-lfs: xfer[lfs-folderstore]: Received upload request for f223709caf2529c54e66fcea539a02ef8286cdf74f23babe80b562507066bc7d
14:25:44.877893 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"complete","oid":"c8660108abd9f5b531021045f6db2cb2c51f76ecf2191493e6e56cfa970210f9"}
14:25:44.877893 trace git-lfs: xfer[lfs-folderstore]: Received upload request for b561b788c0b528e5985448db08c6cf1e7d13da66df2f6e909e3fc3b1e62f5860
14:25:44.879891 trace git-lfs: xfer[lfs-folderstore]: Sent message {"event":"complete","oid":"b870643b7cd0586c672b60551ab009757df14cc90b155e2f777c66617c45a0a7"}
Running go build
complains that there is no go.mod
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.