Comments (8)
Hi João,
It looks as if you may have hit some limit on your server, or maybe it's timing out. I'd need to look more carefully at this and I'm afraid I'm not likely to manage that in the near future.
If you need a temporary fix, can I suggest splitting your inbox into folders, e.g. by year, running the program against each folder, and then (if you really want an inbox that large!) recombining them again?
Best,
Quentin
from imapdedup.
Hello Quentin,
How do you suggest that I split the inbox?
Thanks!
from imapdedup.
Well, there are ways you could script it, but I would just use an email program to create a new folder, select all the messages in one year, and move them over. Then do the next year...
Depending on your email client, you may be able to do something clever with smart mailboxes to make the selection process easier...
from imapdedup.
Thanks for the tips!
from imapdedup.
I ran into this as well doing an inbox with 300K+ messages. (Don't ask..) First run was great it deleted 100K dupes & I was excited but there were still dupes showing up in roundcube so I figured I'd run it again but I'd get that EOF error on the same fetch headers line. I changed (RFC822.HEADER) to (BODY.PEEK[HEADER]) and it worked again for 1 run. Then the dreaded EOF error every run after. So I edited (BODY.PEEK[HEADER]) back to (RFC822.HEADER) and it worked.. For one run.. Until I let it sit awhile & it worked again.. For 1 run then EOF. By that time it was clear something funky was up so I decided to dig deeper to try & narrow it down. While I did many things including adjust MAXLINE and wrap the IMAP commands in try/except hoping it'd continue (it doesn't) it wasn't until I enabled debugging with imaplib.Debug & m.debug = True I finally got a big clue as to what was going on:
35:55.56 BYE response: Server shutting down.
So yeah umm seems the remote server is shutting down mid session? That'd explain why it works after editing (time passed allowing the server to be online again) And note it happened on folder with only 39 messages.. I had changed to another folder with fewer messages to try & narrow down the issue. I thought it was a fluke but was able to reproduce this shutting down bit multiple times.
Many guesses as to what is up from corrupt messages on server to overloading server to bug in python imaplib to who knows but clearly there's an issue, just can't say it's in IMAPdedup (in fact it's not in that I get similar issues with other programs/scripts) beyond maybe it'd be helpful if it better handled & recovered.
Btw not sure about OP but in my case this is all on InMotion shared business hosting which is Dovecot:
- OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE NAMESPACE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
EDIT: Ok seems maybe that's syslog rate limiting in that post so maybe unrelated & weird coincidence..
Little searching & maybe it's rate limiting:
"server dovecot: imap([email protected]): Server shutting down. in=7140 out=70598"
https://www.howtoforge.com/community/threads/server-dovecot-imap-account-tld-com-server-shutting-down-in-7140-out-70598.74887/
If that's the case maybe need option to limit max # of messages it does at a time and/or add sleeps in the loop to help?
from imapdedup.
1150749 others in INBOX
30:37.28 > LCJD5 FETCH 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100 (RFC822.HEADER)
30:38.69 last 0 IMAP4 interactions:
30:38.69 > LCJD6 LOGOUT
30:38.69 last 0 IMAP4 interactions:
Traceback (most recent call last):
File "imapdedup.py", line 324, in
main(sys.argv[1:])
File "imapdedup.py", line 321, in main
process(options, mboxes)
File "imapdedup.py", line 248, in process
ms = check_response(server.fetch(message_ids, '(RFC822.HEADER)'))
File "/root/daily_build/64_23/4.3.4/SysUtil/Python-2.7.5-cross/install_path_full/lib/python2.7/imaplib.py", line 443, in fetch
File "/root/daily_build/64_23/4.3.4/SysUtil/Python-2.7.5-cross/install_path_full/lib/python2.7/imaplib.py", line 1070, in _simple_command
File "/root/daily_build/64_23/4.3.4/SysUtil/Python-2.7.5-cross/install_path_full/lib/python2.7/imaplib.py", line 899, in _command_complete
imaplib.abort: command: FETCH => socket error: EOF
I turned the imaplib debug on.
I get that INBOX has huge amount of mails but fetching result in socket error EOF. Anyone has any insights?
from imapdedup.
Mmm. Do you have access to the server logs?
The imaplib source says that '"abort" exceptions imply the connection should be reset, and
the command re-tried.'
So perhaps that's what we should do (if anyone who can test this would like to submit a pull request!)
I guess your mail server may be very heavily loaded and timing out trying to do this even for 100 messages. However, you may be asking for problems with any IMAP server if you keep more than a million messages in a single mailbox! Not to mention using a lot of RAM on your local machine if you do manage to download even their headers...
from imapdedup.
Thanks for the info. I reduced the chunksize to 1 and script ran. although if it again aborts, I will try to add the re connect part in the script. Will comment if that works.
Although I wish it does not abort . Have been at it for quite some time now.
from imapdedup.
Related Issues (20)
- Add option to use an admin user account for authentication HOT 7
- Execute in All Folders HOT 9
- Error using -P with wrapper: /bin/sh: 1: local: not in a function HOT 1
- Python3 compatibility HOT 2
- Thank You! HOT 1
- Syntax error on line 287 ?? HOT 3
- Not an issue at all!! HOT 1
- mailbox list with / delimiter gives error on recursive processing HOT 1
- IndexError: list index out of range HOT 4
- Too many values to unpack HOT 2
- Folders with a large number of messages cause an error HOT 6
- expunge messages? HOT 3
- Add option to exclude `Date:` from `--checksum` mode
- Getting syntax error on first run HOT 1
- Adapt to mboxes? HOT 1
- Thank you! HOT 1
- ssl issue? HOT 1
- Pyton version changing results?
- Using sed to quote each line for processing with xargs HOT 1
- Criteria for choosing the "one true" message?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from imapdedup.