Variables named with modifiers (like \hat) are incorrectly saved about jld2.jl HOT 5 OPEN

wsshin commented on June 2, 2024

Variables named with modifiers (like \hat) are incorrectly saved

from jld2.jl.

Comments (5)

JonasIsensee commented on June 2, 2024 1

I'm afraid, this isn't really a problem restricted to JLD2 but one more generally with Unicode.
There appear to be two different unicode representations of the "same" visual symbol
and julia generates a different version depending on how you create it.

julia> :â
:â

julia> string(:â)
"â"

julia> string(:â) == "â"
false

julia> codeunits(string(:â))
2-element Base.CodeUnits{UInt8, String}:
 0xc3
 0xa2

julia> codeunits("â")
3-element Base.CodeUnits{UInt8, String}:
 0x61
 0xcc
 0x82

julia> Symbol(string(:â)) == :â
true

julia> string(Symbol("â")) == "â"
true

julia> Symbol("â") == :â
false

julia> string(:â) == "â"
false

from jld2.jl.

JonasIsensee commented on June 2, 2024 1

The place to edit is here, I think:

JLD2.jl/src/groups.jl

Line 25 in 30dd578

function lookup_offset(g::Group, name::AbstractString)

The correct function to compare strings is Unicode.is_equal_normalized() and one could consider always adding a normalization step prior to saving (or loading the whole thing) with Unicode.normalize()

from jld2.jl.

wsshin commented on June 2, 2024

Thanks @JonasIsensee. I reported the issue to JuliaLang/julia.

from jld2.jl.