Large file uploads causing 500 error? about lfs-test-server HOT 3 CLOSED

Hey,

I'm not sure exactly what's going on here, but here are some things that would help in tracking it down. First, it's helpful if you can include the entire output of running the client with GIT_TRACE=1, GIT_TRANSFER_TRACE=1, and GIT_CURL_VERBOSE=1. Those three environment variables include all of the relevant info that the client can produce.

If your files are larger than 4 GiB and you're on Windows, you're probably running into the Git problem that you linked to where it corrupts large files. You can check to see if those files which you expect should be larger are of the appropriate size when you look at the appropriate file under .git/lfs/objects. Two of the reasons you could get a 500 error are a hash mismatch or a size mismatch, and you can check the hash of those files with sha256sum, which is likely included with Git for Windows.

The final possibility is that a resume is going on and you have large files; it looks like the server doesn't handle 64-bit integers in Range headers, which I will send a patch to fix in a moment.

from lfs-test-server.

It is a VERY LARGE log. That snippit is from me running with the flags set. Both the client and server are on the same machine in this test.

GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push origin master

Some files are close to 4GB, but some that errored are only 1GB.
These files do show as resuming in several cases so I was going to clean up the troubled files and see if it was a size or hash issue.

Right now the details are long but can be shared at a location if needed. This is the beginning, what I posted first was toward the end where it quit.

User@localhost MINGW64 /q/Movies1 (master)
$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push origin master
12:52:45.126790 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
12:52:45.133807 git.c:440 trace: built-in: git push origin master
12:52:45.140801 run-command.c:663 trace: run_command: unset GIT_PREFIX; 'git-receive-pack '''//localhost/Git
Repo/Movies1''''
12:52:45.254728 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:52:45.265730 git.c:440 trace: built-in: git receive-pack //localhost/GitRepo/Movies1
12:52:45.372734 run-command.c:663 trace: run_command: .git/hooks/pre-push origin file://localhost/GitRepo/Mov
ies1
12:52:45.493847 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:52:45.497842 git.c:703 trace: exec: git-lfs pre-push origin file://localhost/GitRepo/Movies1
12:52:45.497842 run-command.c:663 trace: run_command: git-lfs pre-push origin file://localhost/GitRepo/Movies
1
12:52:45.548844 trace git-lfs: exec: git 'version'
12:52:45.630895 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '
-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
12:52:45.804868 trace git-lfs: exec: git 'config' '-l'
12:52:45.846840 trace git-lfs: pre-push: refs/heads/master e40981e9baf1a48ec73585201f85cb53859a0a22 refs/heads/master c5
b52604abad58ec2f2705c3b478e744c6aa27d5
12:52:46.134572 trace git-lfs: tq: running as batched queue, batch size of 100
12:52:46.135587 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
12:52:46.789040 trace git-lfs: tq: sending batch of size 100
12:52:46.790049 trace git-lfs: api: batch 100 files
12:52:46.790049 trace git-lfs: creds: git credential fill ("http", "localhost:8080", "")
12:52:46.817126 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:52:46.822122 git.c:440 trace: built-in: git credential fill
12:52:46.824278 run-command.c:663 trace: run_command: 'git credential-manager get'
12:52:46.928118 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:52:46.932118 git.c:703 trace: exec: git-credential-manager get
12:52:46.932118 run-command.c:663 trace: run_command: git-credential-manager get
12:52:47.459232 trace git-lfs: Filled credentials for http://localhost:8080/
12:52:47.479206 trace git-lfs: HTTP: POST http://localhost:8080/objects/batch

POST /objects/batch HTTP/1.1
Host: localhost:8080
Accept: application/vnd.git-lfs+json; charset=utf-8
Authorization: Basic * * * * *
Content-Length: 9182
Content-Type: application/vnd.git-lfs+json; charset=utf-8
User-Agent: git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)

from lfs-test-server.

After some trial and error it seems it is file related. it kept failing on certain files on upload to the LFS system with the 500 error I showed. When I check the files everything is fine and the files are not corrupt or anything. So after battling it for awhile I switched to using PowerShell on windows and leveraging PoSH-Git with it.

So far the ones that appeared to have issues are working as expected and are not giving me an error. Through Powershell I am adding and committing 1 file at a time to try to narrow down the pattern but the ones that seemed to fail are successfully committing. And when I do a "git lfs ls-files" those files appear. So it could be a Bash issue on Windows.

This is what I am doing to upload the entire set of untracked files (for posterity).

#Commit all working files
(get-gitstatus).working | foreach-object {get-childitem $_ -recurse -force} | where {(-not $_.psiscontainer) -and ($_.name -notlike "*thumbs.db" -and $_.Extension -ne ".dat" -and $_.Extension -ne ".ini" -and $_.Extension -ne ".partial" -and $_.Fullname -notlike "*/bin/*" -and $_.fullname -notlike "*/obj/*")} | foreach {
$file=$_;
Write-Host "Working file: $($File.Fullname)"
git add "$($file.Fullname)"
git commit -m "`"$($file.Fullname)`""
#git push origin master 2>&1 | write-Output
$resultpush = git push origin master  2>&1 | write-Output
$resultpush
if ($resultpush.Length -ge 2) {
    if ($resultpush[-2].tostring().Contains("Fatal error: Server error:") -or $resultpush[-1].tostring().Contains("error: failed")) {
        Write-Host "Failure Detected, reversing commit for File: $($file.Fullname)" -ForegroundColor Red
        
        git reset origin/master
        git status
    }
}
}

*** Edit
In my testing I forgot I was doing the commit but not a push. The push to LFS is when I was having the issue. But it appears it is either a true file issue since it is not happening all the time or a server capacity issue since I was running several commits at once when I had the most issues. I also updated the above Powershell code to handle the failures and continue inserting. While this causes a LOT of commits since it is doing one file at a time, you don't have the issue of one file failing the entire folder. It also leverages the Posh-Git module for Powershell for the working files filter.

from lfs-test-server.