Non-UTF-8 attachments are messed up after being downloaded by Offlineimap3 about offlineimap3 HOT 13 CLOSED

Thanks for spotting that @jishac, I should have checked before posting here. :(
Let me go back to the user and check.

from offlineimap3.

With the latest version of Offlineimap3 in Debian Testing (offlineimap3 0.0~git20210225.1e7ef9e+dfsg-3 according to https://tracker.debian.org/pkg/offlineimap3) I still encounter the bug. (I re-downloaded from the IMAP server one faulty message and it is still messed up.)

Below an extract of the text of one such message (although I do not think this can really help…) with the correct utf-8 encoding. The characters replaced by � are mostly ’ é à è. In mutt they are by default displayed as � because the text is not recognized as utf-8 but as windows-1252, but if I change the encoding I obtain what follows.

*La direction du CNRS a d�cid� de publier en 2021 les classementsdes 
jurys d�admissibilit� aux concours de charg�s de recherchepar ordre 
alphab�tique*, alors qu'ils �taient communiqu�s jusqu�en 2020par ordre 
de m�rite.

Le Sgen CFDT Recherche^EPST s�oppose � cette d�cision qui m�prise les 
r�gles jusque-l� admises par l�ensemble de la communaut� et d�nature le 
jugement par les pairs r�alis� par les sections du comit� national. 
Cette absence de transparence instille le doute sur l�impartialit� du 
jury d�admission et met � mal la confiance que tous et notamment les 
candidats portent sur l�organisation des concours au CNRS�!

Can I do anything to help?

from offlineimap3.

Not sure if this is related but another Debian user has reported non utf-8 decoding error. Looks like #56 has not completely fixed the utf-8 issue.

https://bugs.debian.org/985663

@thekix do you think these two are same issue or do you want me to open a new issue for the Debian bug?
@jishac fyi.

from offlineimap3.

The stack trace at

https://bugs.debian.org/985663

is a bit suspicious since it lists

  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 262, in getmessage
    retval = file.read()

which is the old code base. Is there a chance that both old and new are installed?

from offlineimap3.

With the latest version of Offlineimap3 in Debian Testing (offlineimap3 0.0~git20210225.1e7ef9e+dfsg-3 according to https://tracker.debian.org/pkg/offlineimap3) I still encounter the bug. (I re-downloaded from the IMAP server one faulty message and it is still messed up.)

@Elvith I've had a few messages that were permanently altered by the old code base when uploading a message to the server in this very way (utf-8 encoding when the email states another). Are you able to diagnose the raw message stored on the server (by downloading it out of band for example)? If it is mangled on the server, then a manual correction may be needed.

from offlineimap3.

@Elvith I've had a few messages that were permanently altered by the old code base when uploading a message to the server in this very way (utf-8 encoding when the email states another). Are you able to diagnose the raw message stored on the server (by downloading it out of band for example)? If it is mangled on the server, then a manual correction may be needed.

Not sure about what you mean. What I did is :

1. Kill offlineimap
2. Remove (rm -r) the directory on my computer where offlineimap downloads emails
3. Start offlineimap again.
   Then it re-created the missing directory, and re-downloaded all missing emails from the server. And the faulty email was still messed up.

Is there another test I might do?

from offlineimap3.

If your email provider has a web interface, most of the time you can download a single raw message. Some providers make it hard (Outlook Web Mail comes to mind) but most I've seen have some way of doing it. Then inspect or open that email in your viewer of choice (mutt, Thunderbird, etc). If you do not have a web interface, then you could enable debugging in offlineimap to capture the raw message, but I would not recommend doing that for the entire mailbox as you would generate a lot of output.

Basically, by trying to capture what the server has stored as the message we can determine if offlineimap is having an issue or if the problem exists in the source message stored on the server.

from offlineimap3.

OK. I just looked at the “source” of the faulty message on my webmail interface; it is still scrambled.

But if I understand correctly how offlineimap synchronizes the messages on my computer and on the server, this just means that the scrambled version on my computer, scrambled by offlineimap when downloaded, replaced at some point the original version on the server. This does not mean that the original version on the server was also scrambled and therefore this does not mean that offlineimap is not having an issue. Or does it?

And if this is correct, then in fact this message does not prove anything, since it is dated from March 22, before the upgrade of offlineimap on my debian… It was scrambled on my computer before the upgrade, the scrambled version was uploaded/synchronized on the server before the upgrade, and therefore the message will remain scrambled, upgrade or not. No upgrade will miraculously “un-scramble” it.
What I would need in order to claim that offlineimap is still buggy is a new scrambled email, received after the upgrade. Correct?

from offlineimap3.

OK. I just looked at the “source” of the faulty message on my webmail interface; it is still scrambled.

But if I understand correctly how offlineimap synchronizes the messages on my computer and on the server, this just means that the scrambled version on my computer, scrambled by offlineimap when downloaded, replaced at some point the original version on the server. This does not mean that the original version on the server was also scrambled and therefore this does not mean that offlineimap is not having an issue. Or does it?

And if this is correct, then in fact this message does not prove anything, since it is dated from March 22, before the upgrade of offlineimap on my debian… It was scrambled on my computer before the upgrade, the scrambled version was uploaded/synchronized on the server before the upgrade, and therefore the message will remain scrambled, upgrade or not. No upgrade will miraculously “un-scramble” it.

Correct, no upgrade will miraculously “un-scramble” the original message as the server copy is affected, very likely in exactly the manner you outlined (see note at end). Knowing the old offlineimap behavior, one could try and write a tool to seek out such messages based on a heuristic, and try to correct them. What I did in my case is manually correct any message that was "important".

What I would need in order to claim that offlineimap is still buggy is a new scrambled email, received after the upgrade. Correct?

Correct, seeing the issue on a new message after the upgrade would be a bug, but the new handling should be robust to encoding types so you should not experience any.

PS: It can be hard to assign blame for scrambled/manged messages. I have seen scrambled/mangled email that arrived to my inbox. Usually rare for non-spam messages (but possible). That said, if the sever copy of the March 22 message has a "X-OfflineIMAP" header, then offlineimap uploaded that copy and is likely to blame.

from offlineimap3.

PS: It can be hard to assign blame for scrambled/manged messages. I have seen scrambled/mangled email that arrived to my inbox. Usually rare for non-spam messages (but possible). That said, if the sever copy of the March 22 message has a "X-OfflineIMAP" header, then offlineimap uploaded that copy and is likely to blame.

Interesting. No, it does not have that header, but the local copy does not have it as well. I guess I must have parametrized my offlineimap in such a way that this header is not written, or maybe it is the default behavior.

Anyway, thank you for your precious input, I will keep an eye on new emails in the forthcoming weeks and I will report here in case of scrambled messages. 👍

from offlineimap3.

Hi @Elvith

can we close this issue?

Thanks a lot.
kix

from offlineimap3.

Yes, the problem seems solved, I have not had any issue since my previous message. Thank you very much!

from offlineimap3.

Side note: the "X-OfflineIMAP" header is only set when the server does not support the UID PLUS IMAP extension. This header allows offlineimap to get the assigned UID once the email is uploaded in this case.

However, adding a new header on each uploaded email might be useful to track what's was done by offlineimap. This might help a lot to debug issues.

from offlineimap3.