Comments (13)
I recovered from this problem by:
- Moving my my_repo.git repo out of the way
- Closing the problem PROJECT-3118-Update branch in my Hg repository
- Re-cloning the my_repo.git repo from my Hg repo
- Replacing my config file in the newly-cloned repo with the old one that was moved out of the way
- Running git fetch --all to get all remote refs and re-pulling my active remote branches
So far, no issues. It doesn't fix the problem but it lets me work normally until it occurs again.
from git.
I am getting the same issue. Please let me know if there is any additional information that I can provide about the state of the repository that would help with a fix.
from git.
@rpearl I haven't had this problem occur since I closed the problem branch in Mercurial and re-cloned. I have no confidence that it won't happen again but at least recovery from the problem seems to be fairly straightforward. Well, as long as the problem branch is not one of your main development lines anyway. :)
from git.
This seems to be the same issue as dusty-phillips/gitifyhg#88 -- i.e. a git garbage collection (gc) will remove objects that are referenced by the marks file, because for some of them there simply is no git ref referencing them.
It's kind of nasty :-/
from git.
Actually, I take that back (sorry for the confusion): this issue is different, though possibly related. In the bug I linked, the mark is present but the sha-1 it refers to is not known to GAP due to gc. In the issue here, the mark table somehow got truncated. I just experienced a similar situation. First, I did a pull from a hg remote, and got an error fatal: object not found: fe5fd3da2cbd210213816c90fe6afba6bd093ea9
At this point, I might have fixed it by carefully hand-editing the marks files. Unfortunately, I made another git pull
right away then got this error: fatal: mark :20574 not declared
When I looked at the marks files, I noticed two things: marks-hg listed mark :20574, referencing a hg commit SHA-1. But the last mark in marks-git was :19089, so it look as if it was truncated? My last pull would have imported maybe 10 commits, so that alone can't account for discrepancy.
A manual check discovered a git commit matching the hg commit corresponding to mark :20574 (finding it was easy because that commit happened to be the head of a hg branch).
from git.
EDIT: Disregard this comment, it was before I had a better idea of your issue.
I'm trying to understand what is happening here, but it's not clear from those reports. What I would do if I were you is modify the script to do some checks before and after you do the git push
.
First, I would create back ups of the marks-git and marks-hg files before running the git push
command.
If a crash occurs, save both the back up and the current marks in a directory and post them here to do some forensic analysis.
Finally, restore the back up marks, and everything should keep working.
If all else fails, remove the marks files, and run git fetch
again.
They are located in "$repo/hg/origin/marks-*".
My guess is that some git push
commands are crashing at certain point, and then the marks are messed up, and then everything else fails.
from git.
All right. I've managed to analyze better the output, and the issue I see is that in your script you don't check the exit status of the commands, so you continue with the push, even if the fetch failed, that's why there's so much output in the reports which was confusing.
Now, in the first crash the failure is that the mark '544029' cannot be found by git fast-import
, which presumably means it's missing in the 'marks-git' file, why that is happening is very important, but I will come back to that.
In the 4th one fast-import can't find mark '544213' which comes after '544029' and it's trying to update the branch PROJECT-3118-Update, specifically, the notes. The 5th one is trying to do exactly the same. The 6th one is trying to update many branches (including PROJECT-3118-Update), but fails after trying to to find '544029', the same one as before. The 7ht is trying '544235', again after '544029', and again trying to update PROJECT-3118-Update.
The conclusion so far is that the commits marks of PROJECT-3118-Update are somehow screwed up, so if there's any change to that branch, the fetch fails.
What is interesting is the manner in which they are screwed, it seems the commits are marked correctly in marks-hg, but not in mark-git, which means that at some point there was a crash in git fetch
and git fast-import
didn't update marks-git, but git remote-hg
did update marks-hg. I cannot see how this could happen in recent versions of remote-hg, which leads me to believe that you were running and old version when the crash happens, and everything has been screwed up since.
Another interesting bit is this:
fatal: corrupt mark line: :330340 912c63cfcbc91
Which means that something went really really wrong, and the marks-git file was truncated. However, if that was truly the case, nothing would ever work again. And from the report it seems the next push did actually succeed, so maybe somebody was in the middle of writing it, and there's something weird that that machine's file-system. But the previous git fetch
failed, so the file shouldn't be updated.
Maybe somebody was using remote-hg on that repository at the same time the script was running.
Either way you mark files are screwed up, and it doesn't really make sense to try to analyze the errors you get after you fix them. The easiest way is to simply remove the "$repo/hg/origin/marks-*" files, which would cause the next git fetch
to take a long time, since it would have to basically import everything again. However, I can probably write a script that would synchronize the files manually if you really want to avoid that (some, maybe a lot, of commits would need to be re-imported regardless).
If after the mark files are fixed you get them desynchronized again, then there's definitely a problem worth looking into, but until that happens my money is on the fact that you were running an old version of the script.
from git.
I've written a script that should synchronize marks-hg
with marks-git
, and report if there are issues or not (back up "marks-hg" just in case).
Also, I've checked the code more thoroughly, and now I see that a crash in git fetch
could trigger the issue that you see, even in the newer versions of the script.
Maybe this patch would solve the issue for you:
--- a/git-remote-hg.py
+++ b/git-remote-hg.py
@@ -1255,12 +1255,10 @@ def main(args):
die('unhandled command: %s' % line)
sys.stdout.flush()
+ marks.store()
+
def bye():
- if not marks:
- return
- if not is_tmp:
- marks.store()
- else:
+ if is_tmp:
shutil.rmtree(dirname)
atexit.register(bye)
Unfortunately this might cause issues if git push
crashes, which could be even worst.
I'll spend some time trying to reproduce crashes and their subsequent issues to test the code thoroughly, but my guess is that Git core would need to be updated, it's not enough to fix the remote helper script.
from git.
Ok, I've done a bunch of tests and I've narrowed down the issues.
First of all, you need to fix the marks. The easiest is to simply remove $repo/hg/origin/marks-*, and run git fetch
, that will have to re-import the repository. Alternatively, you can use the tool I wrote: git-marks-check. If you run it like git marks-check origin -f -v
it will fix the marks files, and then you need to run git fetch
again, but only a smaller subset of changes would need to be re-imported.
https://gist.github.com/felipec/10551806
That should get you up and running, however, the issue might appear again, to avoid the synchronization problem you need to do two things:
- You need the change I suggested in a previous comment, I've already pushed the change, so if you use the absolute latest you would be fine. Or you can apply it manually:
bd830d4
- You need a fixed
git push
. I've pushed the changes to my master branch, but I suppose you are using upstream Git, and if so, you need to manually restore the marks-git file ifgit push
fails.
Something like this perhaps:
my_push () {
cp hg/origin/marks-git{,.bak} &&
git --bare push "$@" ||
mv hg/origin/marks-git{.bak,}
}
my_push --quiet origin branches/the_dev_branch_2
my_push --quiet origin branches/the_main_branch
Finally, you don't need to specify --bare on every operation, you can tell Git it's a bare clone with git config core.bare true
, or even better at clone time with git clone --bare
.
So my suggested script would be:
#!/bin/bash
export PATH=$PATH:/usr/local/bin
cd /var/git/the_repo.git
my_push () {
cp hg/origin/marks-git{,.bak} &&
git push "$@" ||
mv hg/origin/marks-git{.bak,}
}
# Get the entire repository from the remote
git fetch --quiet origin
# Push branches we're working on back to the remote repo
my_push --quiet origin branches/the_dev_branch_2
my_push --quiet origin branches/the_main_branch
Additionally, you might want to run git marks-check origin
at the end of the script to make sure everything is OK.
from git.
I have not followed the recent changes discussed here, so the following might be obsolete. But: With the version of git-remote-hg in git.git's next branch, the following is an easy way to trigger a fast-import crash which then garbles the marks file:
# Create a multi-head repository
hg init hgrepo
cd hgrepo
echo a > a && hg add a && hg commit -m a
echo b > b && hg add b && hg commit -m b
hg update 0
echo c > c && hg add c && hg commit -m c
cd ..
# Clone it via remote-hg
git clone "hg::hgrepo" gitrepo
cd gitrepo
git gc --prune=now
git pull
The problem here is that some commits are not reachable from any user-visible ref, hence can be pruned, which then leads to "gaps" in the marks tables. There are more ways to trigger this, but the above (a hg branch with multiple heads) seemed to me to be the simplest way to trigger the issue.
from git.
I've been using 93d24ea branch fc/master and not upstream git during and since this issue was reported. Haven't had the issue again since we closed the problem PROJECT-3118-Update hg branch (bookmark, whatever) so I don't think I have much to contribute as far as ensuring the problem is fixed. I'll definitely update and build/install your latest master, is there anything else I can do to assist with reproducing or ensuring the latest patch fixes this issue?
from git.
@fingolfin Yeah, I see the same problem with git-fc, but I think that's a separate issue.
from git.
@shaunlessman The best you can do is just keep running the sync and if a problem happens report it back here, however, if you don't ensure the marks are correct (either with my script or by removing the files), any report won't be useful, so please do that.
I'll mark this as closed as I think this particular issue is fixed, reopen if you have problems.
from git.
Related Issues (20)
- Something is wrong with git-stage docs HOT 2
- Crash during push to Hg repo HOT 2
- planet bzr repos cannot be cloned HOT 6
- push failed when git branch is different than bzr branch name HOT 2
- Trying to Parse email as int. HOT 1
- Push to new launchpad branch using git-remote-bzr HOT 4
- bzrlib.errors.NoSuchFile error while cloning openobject-addons on Launchpad HOT 1
- Cannot push when using https authentication with git-remote-hg HOT 5
- git-remote-hg mysteriously fails HOT 2
- Cloning hg repo from Octave project crashes fast-import HOT 5
- rebase or merge latest git release HOT 2
- Git commits show up in the "draft" phase in BitBucket HOT 2
- Closed branches can cause synchronization issues after gc
- Link to git-remote-hg.py in the wiki is wrong HOT 2
- Worked for a while, now fetch fails with "mark not declared"
- Sorry, flubbed an enter key press... please delete.
- git-remote-hg doesn't fetch HOT 2
- Bookmarks on named branches HOT 1
- uses obsolete python2 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from git.