FVWM logs warnings `Invalid byte sequence during conversion from UTF-8 to ISO-8859-1` about fvwm3 HOT 11 CLOSED

Hi @farblos

Thanks. This should not be happening. Can you send me through a proposed patch, and I'll take a look.

from fvwm3.

@ThomasAdam: Nah, my diagnosis was wrong. I really get these warnings, but they are not related to zero-termination of strings. I think the string semantics in that region of code is simply (ptr, length) and not (zero-terminated ptr).

Will continue to poke around with gdb ...

from fvwm3.

Ok, the story (more lilkely) goes like this:

• I use Style * NoIcon in my config file
• Accordingly (?), the icon font never gets loaded for any window, and fw->icon_font always equals NULL for all windows
• However, EWMH_WMIconName gets called, anyway, and tries to convert the icon name encoding
• As there is no icon font which could provide the character set, FiconvSetupConversion picks FLCIconvDefaultCharset as default, which is for whatever reason ISO-8859-1 in my FVWM
• But the emdash in the Firefox title does not convert from UTF-8 to ISO-8859-1 and, hence, the warning

Any hints on why FLCIconvDefaultCharset is on ISO-8859-1? Ultimately it seems to be related to the search for a "best charset" in FlocaleInit_X_Charset not finding a match for encoding UTF-8, but I lost track at that point.

(At least during this journey I came across BugOpts TransliterateUtf8 true, which silences the warnings as a work-around.)

from fvwm3.

@mikeandmore -- is this something you've time to look at? I know you did some work on font handling recently.

from fvwm3.

Sure. I may have some time this weekend or next weekend to take a look.

I think we simply shouldn't convert to ISO-8859-1 under XFT because that was mostly used for old xfonts.

from fvwm3.

send a PR #860 .

@ThomasAdam Does that look right to you?

from fvwm3.

@mikeandmore Thanks. Unfortunately your patch does not make the warnings go away. But maybe I did something wrong ...

git clone https://github.com/fvwmorg/fvwm3.git
cd fvwm3
sh autogen.sh 
./configure
make
git remote add mikeandmore https://github.com/mikeandmore/fvwm3.git
git fetch mikeandmore 
git cherry-pick 0a9a1c212b2b8fd04e3325c5fd7405391169d0e7
make clean
make

The resulting binary should have XFT support:

[fvwm3]$ ./fvwm/fvwm3 -version
fvwm3 1.0.7 (1.0.6a-17-g01908e29)
with support for:  XPM, PNG, SVG, Shape, XShm, SM, Bidi text, XRandR, XRender, XCursor, XFT, NLS

fvwm3 comes with NO WARRANTY, to the extent permitted by law. You may
redistribute copies of fvwm under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

git log shows your commit:

commit 1cfb38c81433b2e906f0e26279f8e09e060cf5ef (HEAD -> main)
Author: Mike Qin <[email protected]>
Date:   Sun Jun 25 10:46:32 2023 -0400

    we don't need iconv under XFT

commit 01908e2930bb24f9cc098941470a417c374f62c2 (origin/main, origin/HEAD)
Author: Thomas Adam <[email protected]>
Date:   Mon Dec 9 00:22:08 2019 +0000

    FvwmPerl: fix regexp warning for special chars
[...]

Started fvwm3 directly from the development directory, double-checking:

[fvwm3]$ nc -U /tmp/fvwm_mfl.sock
{"connection_profile":{"version":"1.0.7","version_info":"1.0.6a-17-g01908e29"}}
set echo
echo $[version.num]
{"echo":{"message":"1.0.7"}}
echo $[version.info]
{"echo":{"message":"1.0.6a-17-g01908e29"}}
echo $[version.line]
{"echo":{"message":"fvwm3 1.0.7 (1.0.6a-17-g01908e29)"}}

But when I open a new tab in firefox I still get the warnings in the log:

[1687899731.167629] CMD_Echo: 1.0.7
[1687899738.559748] CMD_Echo: 1.0.6a-17-g01908e29
[1687899749.815249] CMD_Echo: fvwm3 1.0.7 (1.0.6a-17-g01908e29)
[1687899797.373485] convert_charsets: [fvwm][convert_charsets]: WARNING -

[1687899797.373513] convert_charsets: Invalid byte sequence during conversion from UTF-8 to ISO-8859-1
[1687899801.408411] convert_charsets: [fvwm][convert_charsets]: WARNING -

[1687899801.408443] convert_charsets: Invalid byte sequence during conversion from UTF-8 to ISO-8859-1

If you spot sth I may have done wrong during the build process pls let me know.

from fvwm3.

Will try to debug this tomorrow ...

from fvwm3.

Doh, I'm pretty sure that I haven't used an Xft font as default font during my first tests ... will repeat test tomorrow.

from fvwm3.

@farblos Yes, you need to use Xft font. Anytime you use a xfont, fvwm needs to convert into ISO-8859-1 I believe.

from fvwm3.

@mikeandmore Tested again: With your patch and with an Xft default font (haven't tested finer granularity) the warnings are indeed gone. Without your patch and with an Xft default font I get the warnings.

And if I understood you correctly (modulo your slight uncertainty), for legacy xfonts you will always get these warnings for characters that cannot be converted to ISO-8859-1.

So if @ThomasAdam oks this patch as well we can close this issue...

from fvwm3.