Comments (24)
I restored multiple times my 30000 emails without any problems on Mac OSX, Linux (and few times on windows).
The problem seems to be on Windows. I need to investigate.
Thanks for reporting the problem.
from gmvault.
Let me know if there's anything I can do to help!
from gmvault.
@Halu I need to profile the program and it will take some time.
In the mean time, you can use the --restart option for restore. It should really be called --resume as it will resume the restore near the last email you restored.
from gmvault.
Perhaps another hint at what is going wrong: when continueing using the --restart option, gmvault fails within the first 50 emails, again with the same MemoryError, as if it buffers all the emails it already has restored anyway. Restarting the gmvault-shell has no effect.
from gmvault.
Ok thanks, I don't understand your problem but I am profiling Gmvault to remove memory fragmentation issues.
I should have a new version for testing ready next week. Keep your email db it is safe.
Will you be ready to test this new version for me next week ?
Thanks,
Guillaume
from gmvault.
Sure, I'll be happy to help testing. Thanks for the quick responses!
from gmvault.
@Halu I have prepared v1.6-dev available from here http://bit.ly/JtfbvD.
In this version I have fixed some memory fragmentation issues but I am not convinced that you were affected by this issue. Please install the new version and test it in debug mode.
$> gmvault restore [email protected] --debug --restart
If it fails again send me by email guillaume((dot))aubert((at))gmail((dot))com the log file that should be under C:\Users\MyUser and called gmvault.log. Please zip it before to send it if it is big.
Waiting for your results now. Many thanks for your help.
from gmvault.
@Halu Did you have time to do a test ? I will close the issue in the mean time because other people could test the fix for me.
from gmvault.
It did not work alas, it failed on one specific email which I sent you as an email attachment to you 6 days ago :) It is unclear to me though if it still is a memory issue or that I'm encountering another bug...
from gmvault.
@Halu Ooups yes sorry. Just to be sure. Now it doesn't eat all your memory but fails for one email. There is something in your email that is special and I need to analyse it (probably a special char but I need to understand who did put it).
In the mean time, you could go in your email db and remove the failing email (it should start with the gmail id a long number that looks like a timestamp). Move the .meta and .eml.gz file somwhere else and you could continue the restore.
from gmvault.
Deleting the 'faulty' .eml.gz/meta-files does help, it indeed continues the restore when I retry after deleting. However, even though the hanging does not eat my memory anymore, I noticed that the gmv_cmd.exe does hog the CPU: it keeps at a steady 50% on my dualcore cpu while it hangs, which makes me think the hanging on certain emails may somehow still be related to the previous memory errors, except that the memory is now not eaten anymore :) I'll continue restoring now whilst deleting those few mails that give raise to trouble, as it is a fine workaround.
from gmvault.
@Halu If the emails are not confidential please consider sending them to me. I will make a test suite with them.
For a future version, I will profile Gmvault to see how to fix the issues (in imaplib) most probably
from gmvault.
v1.6beta, OSX 10.6, I'm getting this issue too. Got up to 22GB of virtual memory then the Mac freezes totally as it runs out of hard drive space.
If I delete/move the problematic email, is there an easy way for me to set the upload going again from the failed point? Is there any danger of creating duplicates?
from gmvault.
Hi,
I am working on that issue but sometimes emails contain "crap" and they do
not respect the standards. I will try to find a solution but the problem seems to be in the lower layers (SSL).
Once you have removed the faulty email, you can use the option --resume
that will restart around where it stopped.
Could you send me the faulty email if it isn't personnal ?
I would like to do a test with it.
Thanks,
Guillaume
On Wed, Jun 13, 2012 at 11:09 PM, bommy <
[email protected]
wrote:
v1.6beta, OSX 10.6, I'm getting this issue too. Got up to 22GB of virtual
memory then the Mac freezes totally as it runs out of hard drive space.If I delete/move the problematic email, is there an easy way for me to set
the upload going again from the failed point? Is there any danger of
creating duplicates?
Reply to this email directly or view it on GitHub:
#42 (comment)
from gmvault.
Thank you. How would I identify and open the email to see if it's anything
personal or not?
Screenshot of the failure state attached. I'm actually trying to re-do the
process on my work computer (Win 7) as the internet connection is much
faster here, so would be interesting to see if the same happens. I'm
redownloading all the emails rather than transferring my existing backup
files from Mac to PC.
Tom
On 14 June 2012 08:01, Guillaume Aubert <
[email protected]
wrote:
Hi,
I am working on that issue but sometimes emails contain "crap" and they do
not respect the standards. I will try to find a solution.
Once you have removed the faulty email, you can use the option --resume
that will restart around where it stopped.
Could you send me the faulty email if it isn't personnal ?
I would like to do a test with it.Thanks,
GuillaumeOn Wed, Jun 13, 2012 at 11:09 PM, bommy <
[email protected]wrote:
v1.6beta, OSX 10.6, I'm getting this issue too. Got up to 22GB of virtual
memory then the Mac freezes totally as it runs out of hard drive space.If I delete/move the problematic email, is there an easy way for me to
set
the upload going again from the failed point? Is there any danger of
creating duplicates?
Reply to this email directly or view it on GitHub:
#42 (comment)
Reply to this email directly or view it on GitHub:
#42 (comment)
from gmvault.
Sorry, attachment bounced, uploaded here - http://bit.ly/tomwhitakerpublic
Tom
---------- Forwarded message ----------
From: Tom Whitaker [email protected]
Date: 14 June 2012 09:42
Subject: Re: [gmvault] MemoryError on large restore (#42)
To: Guillaume Aubert <
[email protected]
Thank you. How would I identify and open the email to see if it's anything
personal or not?
Screenshot of the failure state attached. I'm actually trying to re-do the
process on my work computer (Win 7) as the internet connection is much
faster here, so would be interesting to see if the same happens. I'm
redownloading all the emails rather than transferring my existing backup
files from Mac to PC.
Tom
On 14 June 2012 08:01, Guillaume Aubert <
[email protected]
wrote:
Hi,
I am working on that issue but sometimes emails contain "crap" and they do
not respect the standards. I will try to find a solution.
Once you have removed the faulty email, you can use the option --resume
that will restart around where it stopped.
Could you send me the faulty email if it isn't personnal ?
I would like to do a test with it.Thanks,
GuillaumeOn Wed, Jun 13, 2012 at 11:09 PM, bommy <
[email protected]wrote:
v1.6beta, OSX 10.6, I'm getting this issue too. Got up to 22GB of virtual
memory then the Mac freezes totally as it runs out of hard drive space.If I delete/move the problematic email, is there an easy way for me to
set
the upload going again from the failed point? Is there any danger of
creating duplicates?
Reply to this email directly or view it on GitHub:
#42 (comment)
Reply to this email directly or view it on GitHub:
#42 (comment)
from gmvault.
@bommy. What operation were you doing ? sync or restore ?
If it is a restore it might be due to a problematic email and I will explain you how to remove it from the database.
from gmvault.
It was a restore, thanks @gaubert.
from gmvault.
@bommy so you should have the email id in the logs of Gmvault. It is the last message that has been restored. For example
[2012-06-14 12:24]:CRITICAL:gmvault:Restore email with id 1293213852590002613
The id here is 1293213852590002613
after in the terminal of your mac, you can do:
$>cd ~/gmvault-db (or where you saved your emails)
$>find . -name "1293213852590002613*"
./db/2009-01/1293213852590002613.meta
./db/2009-01/1293213852590002613.eml.gz
Move the files to quarantine:
$>mv ./db/2009-01/1293213852590002613* ./quarantine
Then the email is 1293213852590002613.eml.gz
$>cd ./quarantine
$>gunzip 1293213852590002613.eml.gz
$>cat 1293213852590002613.eml
To see the content or you can use any editor.
Thanks for the help.
from gmvault.
I am having the MemoryError problem as well while trying to restore a large mailbox. The restore will work for awhile and then I will get this:
Restore email with id 1354135464925655217.
Restore email with id 1354136403316027783.
Restore email with id 1354136977391387211.
Restore email with id 1354142573072803336.
Restore email with id 1354144124929707583.
Restore email with id 1354144265707605388.
Restore email with id 1354146493429344036.
Restore email with id 1354146887922003500.
Restore email with id 1354148204224160982.
Restore email with id 1354148609343052803.
Restore email with id 1354148784008556488.
Error: .
=== Exception traceback ===
Traceback (most recent call last):
File "gmv_cmd.py", line 515, in run
File "gmv_cmd.py", line 423, in _restore
File "gmvault.pyc", line 1032, in restore
MemoryError
=== End of Exception traceback ===
If i use the --restart switch it will work for awhile again, but after about a half hour it will fail with the MemoryError.
I've been trying to restore this mailbox for 2 days and I am now getting a lot of heat for it not being done. Please advise on what I should do.
Thanks
from gmvault.
@3formit What version are you using ? v1.6-beta ?
from gmvault.
Correct 1.6 beta
from gmvault.
@3formit I have version 1.7 in alpha available here: http://bit.ly/JtfbvD.
It should solve the problem and quarantine the emails that create the issues. For some emails, Gmail refuse to ingest them and sometimes it creates in the lower layers (ssl, socket) and infinite loop. I should have fixed it.
Please have a try and let me know.
In case of problems use --debug and send me the gmvault.log file (available in $HOME/gmvault.log for lin and mac and %HOME%/gmvault.log)
from gmvault.
@3formit did you have time to try it and was it better with 1.70alpha.
Thanks,
zoobert
from gmvault.
Related Issues (20)
- https://buhle00#330
- Installation failure HOT 5
- How to install 1.9.2 (beta) or upgrade 1.9.1 to 1.9.2(beta) under Win10
- see https://github.com/gaubert/gmvault/blob/ee915c2799927a30e95963a9c406b415f4825cce/src/gmv/credential_utils.py#L38-L58 HOT 1
- Are deletions propagated? HOT 1
- "This app is blocked" HOT 4
- Gmail claims Im using 50GB, backup is 17GB HOT 1
- email from Google: [Action Required] Migrate your OAuth out-of-band flow to an alternative method before Oct. 3, 2022 HOT 23
- uth
- [Question] Full Documentation of gmvault_default.conf? HOT 1
- Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token HOT 1
- Error 400: invalid_request at installation/setup HOT 1
- ImportError: cannot import name 'Feature' from 'setuptools' HOT 1
- Windows 10 Uninstall
- improvement: Allow export to pst file HOT 2
- All chats being deleted HOT 1
- Access blocked: Gmvault’s request is invalid - Mac OSX HOT 1
- Chocolatey package version of Gmvault is out of date. Can you please update it with new package attached here? HOT 3
- Invalid request error returned when attempting to connect HOT 4
- Build fails on 2 separate Ubuntu 22.04 with pip and sudo python3 setup.py install HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gmvault.