[BUG] Orphaned documents / missing original + archived due to different unicode encoding of special chars about paperless-ngx HOT 3 CLOSED

0x00E4 is the utf16 encoding of ä. 061 + 0308 is the correct utf8 decomposition for the same character.

Basically your filesystem is using utf16, so you should figure out why and fix that.

from paperless-ngx.

I must openly admit: my encoding knowledge is very limited. But maybe you can make sense out of my thoughts / wild guesses:

• To my understanding the code points for 8859-1 and UTF-16 for äöüÄÖÜ are the same. So the encoding could come from there.
• I sometimes just copy & paste text from the PDF to the file name - maybe from there
• The newest affected file is from around 2020-03-20.
• I checked both server and container locale and both return UTF-8
• Checking with a new file I can confirm that both database and file system use the UTF-8 encoding

So, maybe some historical error.

The correct way to resolve it, if I understand it right is to rename all files to the UTF-8 convention and update the database as well to reflect that?

from paperless-ngx.

That's what I would do. Strings in Python are utf8 and we're not doing anything odd with encoding. Values copied from a PDF might not be UTF-8, I guess an OCR program wouldn't know what encoding, though I would assume everything at this point is UTF-8

from paperless-ngx.