Comments (3)
Hi @mchristianl ,
thank you for this report.
I have tracked the problem down to here:
JLD2.jl/src/data/custom_serialization.jl
Line 46 in a0d0730
BigInt
s are mutable and have a custom finalizer (for freeing the memory of the GMP library).
The conversion function copies the memory of mutable structs to a pre-allocated one in order to allow recursively nested structures.
This is preciscely the problem here. The finalizer may be triggered when GC runs.
from jld2.jl.
Defining the following avoids the problem but I hope that there's a more general solution.
function JLD2.jlconvert(::JLD2.ReadRepresentation{BigInt,JLD2.CustomSerialization{String,JLD2.Vlen{String}}}, f::JLD2.JLDFile, ptr::Ptr, header_offset::JLD2.RelOffset)
JLD2.rconvert(BigInt, JLD2.jlconvert(JLD2.ReadRepresentation{String, JLD2.Vlen{String}}(), f, ptr, header_offset))
end
from jld2.jl.
I can confirm that this fixes the symptoms. Thank you @JonasIsensee ! 🥳
Meanwhile I was using the following improvised workaround serializing a Array{Rational{BigInt}}
into two Array{Int64}
and two Vector{UInt64}
"buffers". I am not sure if this is always true for various GMP configurations.
function serialize_bigrat(A0::Array{Rational{BigInt}})
num_last = Array{Int64}(undef,size(A0))
den_last = Array{Int64}(undef,size(A0))
j = 0; for i in eachindex(A0); num_last[i] = (j += abs(A0[i].num.size)); end
j = 0; for i in eachindex(A0); den_last[i] = (j += abs(A0[i].den.size)); end
num_buf = Vector{UInt64}(undef,num_last[end])
den_buf = Vector{UInt64}(undef,den_last[end])
for i in eachindex(A0)
x = A0[i]
@assert -100 ≤ x.num.size ≤ 100 i # safety check
@assert -100 ≤ x.den.size ≤ 100 i # safety check
num_last[i] = copysign(num_last[i],x.num.size)
unsafe_copyto!(pointer(num_buf,abs(num_last[i]) - abs(x.num.size) + 1),x.num.d,abs(x.num.size))
unsafe_copyto!(pointer(den_buf,abs(den_last[i]) - abs(x.den.size) + 1),x.den.d,abs(x.den.size))
end
return (num_last,den_last,num_buf,den_buf)
end
function deserialize_bigrat(num_last::Array{Int64},den_last::Array{Int64},num_buf::Vector{UInt64},den_buf::Vector{UInt64})
@inline function deserialize_BigInt(x_last,x_buf,i)
n = x_last[i]
s = abs(n) - abs(i > 1 ? x_last[i-1] : 0)
x = BigInt(nbits=64*s)
@assert x.alloc >= s
x.size = copysign(s,n)
unsafe_copyto!(x.d,pointer(x_buf,abs(n)-s+1),s)
x
end
A1 = Array{Rational{BigInt}}(undef,size(num_last))
for i in eachindex(num_last)
A1[i] = deserialize_BigInt(num_last,num_buf,i) // deserialize_BigInt(den_last,den_buf,i)
end
return A1
end
from jld2.jl.
Related Issues (20)
- File saved in Julia 1.10.0-beta3 cannot be loaded in Julia 1.10.0-rc1 with Random.Xoshiro HOT 5
- Don't map back to Julia composite type HOT 2
- Optional plain HDF5 data mapping without reconstruction of data source type HOT 10
- Dict of mutable struct reconstruction fails with newer JLD2 versions HOT 1
- Out of disk space throws generic Bus Error HOT 3
- jldopen(filename, "w";) will fail in win7 HOT 2
- Saving vector of dictionaries initialized as #undef gives "This should not have happened" HOT 2
- error when serializing expression containing `Int128` HOT 4
- Possible bug with version 0.4.42 HOT 6
- Error loading file saved with v0.4.41 on v0.4.43 HOT 1
- JLD2 writes zeros to first 3440 bytes in file HOT 4
- Cannot read custom struct with Dict subfield just written to jld file. HOT 2
- Reconstruction of UnionAll type parameters
- Add flag to disallow committing structs
- File created with earlier JLD2 version can't be opened with version 0.4.38 or later HOT 5
- JLD2 is failed to precompile on the latest nightly HOT 1
- How to print or show every group name of a JLD2 file HOT 2
- Manually flush written data to file HOT 2
- Saving/Reloading a struct with more than 255 fields leads to a "missing field" warning & corrupted loads HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jld2.jl.