hi,

Are you still using Docile/Lexicon for documentation. If so
I would like to add your project to Projects using Docile / Lexicon
if that is fine with you.

Thanks.

I am trying to set the read-only flag on an environment, but the following code

env = create()
setindex!(env,0x20000,:Flags)

throws

ERROR: UndefVarError: warn not defined
Stacktrace:
 [1] setindex!(::Environment, ::UInt32, ::Symbol) at .julia/packages/LMDB/MBmCS/src/env.jl:123
 [2] top-level scope at REPL[131]:1

The other approach I tried was set!(env,0x20000), which throws

ERROR: Code[22]: Invalid argument
Stacktrace:
 [1] set!(::Environment, ::UInt32) at .julia/packages/LMDB/MBmCS/src/env.jl:87
 [2] top-level scope at REPL[144]:1

I'm brand new to Julia, so I'm not sure how to go about debugging this.

(using julia version 1.3.1)

Here in the MDBValue constructor, if val <: Number, won't the pointer to the array created be invalid after the constructor returns? In that case, that memory region may contain garbage by the time the data gets stored in the database.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Hi! Just wanted to start by saying thanks for the package -- it's been super useful :)

Gitlab CI

I have a project on Gitlab and I'm trying to add LMDB to it. For some reason, it fails to build only on CI. I can use it locally just fine though and it obviously builds in travis. This is part of the build.log:

$ cat /root/.julia/packages/LMDB/MBmCS/deps/build.log
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   122    0   122    0     0   1087      0 --:--:-- --:--:-- --:--:--  1089

100  140k    0  140k    0     0   228k      0 --:--:-- --:--:-- --:--:--  228k
[ Info: Attempting to create directory /shield-ai/general/.julia/packages/LMDB/MBmCS/deps/downloads
[ Info: Downloading file https://github.com/LMDB/lmdb/archive/LMDB_0.9.22.tar.gz
[ Info: Done downloading file https://github.com/LMDB/lmdb/archive/LMDB_0.9.22.tar.gz
[ Info: Attempting to create directory /shield-ai/general/.julia/packages/LMDB/MBmCS/deps/src
[ Info: Attempting to create directory /shield-ai/general/.julia/packages/LMDB/MBmCS/deps
[ Info: Directory /shield-ai/general/.julia/packages/LMDB/MBmCS/deps already exists
[ Info: Attempting to create directory /shield-ai/general/.julia/packages/LMDB/MBmCS/deps/usr/lib
ERROR: LoadError: IOError: could not spawn setenv(`make -j8 -C /root/.julia/packages/LMDB/MBmCS/deps/src/lmdb-LMDB_0.9.22/libraries/liblmdb liblmdb.so`,
...
...
no such file or directory (ENOENT)

I created a MWE repository, and here's one of the CI runners that failed to build if you want to see a fuller log. Any chance you have some thoughts?

build.log on my local machine

Installing LMDB results in the following errors on OS X. (Yosemite OS X 10.10.3/XCode 6.3). I was surprised to see it building an so and not a dylib, but I'm no expert.

gcc -fPIC -shared -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast ../src/wrapper.c /Users/sebastian/.julia/v0.4/LMDB/deps/src/lmdb-0.9.14/mdb.c /Users/sebastian/.julia/v0.4/LMDB/deps/src/lmdb-0.9.14/midl.c -o liblmdbjl.so -I/Users/sebastian/.julia/v0.4/LMDB/deps/src/lmdb-0.9.14
gcc: warning: couldn’t understand kern.osversion ‘14.3.0
mkdir -p ../usr/lib
cp liblmdbjl.so ../usr/lib
rm *.so
=============================================[ ERROR: LMDB ]==============================================

LoadError: Provider BinDeps.BuildProcess failed to satisfy dependency liblmdbjl
while loading /Users/sebastian/.julia/v0.4/LMDB/deps/build.jl, in expression starting on line 38

==========================================================================================================

After the fixes applied by @DilumAluthge I agree that tagging the next release as 1.0 would be a good idea. Are there any opinions or suggestions for breaking interface changes that should happen before such a release?

it will be good to have a new release tagged with the latest set of changes. @wildart ?

As described in https://discourse.julialang.org/t/ann-plans-for-removing-packages-that-do-not-yet-support-1-0-from-the-general-registry/ we are planning on removing packages that do not support 1.0 from the General registry. This package has been detected to not support 1.0 and is thus slated to be removed. The removal of packages from the registry will happen approximately a month after this issue is open.

To transition to the new Pkg system using Project.toml, see https://github.com/JuliaRegistries/Registrator.jl#transitioning-from-require-to-projecttoml.
To then tag a new version of the package, see https://github.com/JuliaRegistries/Registrator.jl#via-the-github-app.

If you believe this package has erroneously been detected as not supporting 1.0 or have any other questions, don't hesitate to discuss it here or in the thread linked at the top of this post.

It should be possible to use any bits type (including composite types) as keys or values as they have linear representaion in memory.

It seems like a minor change to MDBValue constructor can enable that.

Is it possible to access a database using parallel julia processes? I'm running into problems with the code below (could be that I'm messing something up). Other than one process, the others seem to hang (perhaps waiting for db access?):

# restart processes
if nprocs() > 1
    rmprocs(workers()) # remove all worker processes
end
wpids = addprocs(2) # add processes

println("Spawned ", nprocs(), " processes, ", nworkers()," workers")
println("Proc IDs: ", procs())

# load LMDB on all processes
@everywhere using LMDB

# create a sample database 
nsamples = 100
dbname = "simpleseq.db"
if !isdir(dbname)
     mkdir(dbname)
end

# the data are just {1:1, 2:2 ... }
create() do env
    #put!(env, :Flags, LMDB.NOSYNC)
    open(env, dbname)
    for i=1:nsamples
        start(env) do txn
            open(txn) do dbi
                open(txn, dbi) do cur
                    insert!(cur, string(i), string(i))    
                end
                commit(txn)
            end
        end
    end
end

# load up the functions (see below for specification)
@everywhere using ParHelperFuncs

# the following single process call works
miniBatchSum([1,2,3], dbname)

# the following (which does it in parallel) does not work
# we generate some ids to split across the nodes
# each node will process sample_size values
# the ids are put into proc_idxs
sample_size = 10;
idxs = randperm(nsamples);
idxs = idxs[1:(nworkers()*sample_size)]
proc_idxs = Any[]
st_idx = 1;
en_idx = sample_size;
for i=1:nworkers()
    push!(proc_idxs, idxs[st_idx:en_idx]);
    st_idx = en_idx+1;
    en_idx = en_idx+sample_size;
end

# spawn and run across all worker nodes
k = 1;
remrefs = Array(Any, nworkers());
for proc in workers()
    println("Remote call to: ", proc);
    remrefs[k] = remotecall(proc, miniBatchSum, proc_idxs[k], dbname); 
    k += 1;
end

# collect the results
k = 1;
results = Array(Any, nworkers());
for k = 1:length(remrefs)
    wait(remrefs[k]); 
    results[k] = fetch(remrefs[k]);
    k += 1;
end

And in ParHelperFuncs.jl

module ParHelperFuncs

using LMDB

export getSamplesFromDb, miniBatchSum

# we pull samples from the database
function getSamplesFromDb(env, idxs::Array{Int})
    txn = start(env)
    dbi = open(txn)
    xs = Int[]
    for idx in idxs
        key = string(idx)
        val = get(txn, dbi, key, String);
        val = int(val)
        push!(xs, val)
    end
    return xs
end

# we 
function miniBatchSum(idxs, dbname::String)
    # open the database 
    println("Opening ", dbname)
    env = create()
    open(env, dbname)
    xs = getSamplesFromDb(env, idxs)
    close(env)

    # the cost is the sum of all the values we get from the db
    cost = 0.0
    for x in xs
        cost += x;
    end 

    return cost
end

end

Try to remove C wrapper code to get better grip on LMDB structures. This update will target Julia v0.4.

Just tried to do the first example from the docs

signal (11): Segmentation fault
mdb_txn_begin at /home/zenna/.julia/v0.3/LMDB/deps/usr/lib/liblmdbjl.so (unknown line)
mdb_txn_start at /home/zenna/.julia/v0.3/LMDB/deps/usr/lib/liblmdbjl.so (unknown line)
start at /home/zenna/.julia/v0.3/LMDB/src/txn.jl:21
julia_start_20503 at (unknown line)
jl_apply_generic at /usr/bin/../lib/x86_64-linux-gnu/julia/libjulia.so (unknown line)
anonymous at no file:3
unknown function (ip: 1799251468)
jl_f_top_eval at /usr/bin/../lib/x86_64-linux-gnu/julia/libjulia.so (unknown line)
eval_user_input at REPL.jl:53
jlcall_eval_user_input_20022 at (unknown line)
jl_apply_generic at /usr/bin/../lib/x86_64-linux-gnu/julia/libjulia.so (unknown line)
anonymous at task.jl:95
jl_handle_stack_switch at /usr/bin/../lib/x86_64-linux-gnu/julia/libjulia.so (unknown line)
julia_trampoline at /usr/bin/../lib/x86_64-linux-gnu/julia/libjulia.so (unknown line)
unknown function (ip: 4199613)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 4199667)
unknown function (ip: 0)
Segmentation fault (core dumped)

This definition of convert is type piracy, since LMDB doesn't own the Ref type:

Among the usual problems with type piracy, this also causes ambiguities for convert(Any, ::Ref{MDB_val}). Caught in the PkgEval run for JuliaLang/julia#43671, see: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/3656ddf_vs_1db8b8f/LMDB.primary.log

I'd recommend just removing this definition.

As also noticed in #24 I get an error about warn() not being defined when it is used to show a warning ;-)

julia> using LMDB

julia> env = create()
Environment is created

julia> env[:mapsize] = 52428800
ERROR: UndefVarError: warn not defined
Stacktrace:
 [1] setindex!(env::Environment, val::UInt32, option::Symbol)
   @ LMDB ~/.julia/packages/LMDB/g7vek/src/env.jl:114
 [2] setindex!(env::Environment, val::Int64, option::Symbol)
   @ LMDB ~/.julia/packages/LMDB/g7vek/src/env.jl:118
 [3] top-level scope
   @ REPL[4]:1

(@v1.7) pkg> st LMDB
      Status `~/.julia/environments/v1.7/Project.toml`
  [11f193de] LMDB v0.2.1

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, haswell)

Edit: yes, I should be using :MapSize, but that's not the point here :-)

I'm having problems building LMDB. The repo for LMDB seems to have moved, rendering build.jl outdated. From the Symas LMDB website, I found a github site.

I changed the url in build.jl to

url = "https://github.com/LMDB/lmdb/archive/LMDB_$(version).tar.gz"

but am now encountering another error:

Path /home/haroldsoh/.julia/v0.3/LMDB/deps/src/mdb-mdb was not created successfully (Tried to run `tar xzf /home/haroldsoh/.julia/v0.3/LMDB/deps/downloads/LMDB_0.9.14.tar.gz --directory=/home/haroldsoh/.julia/v0.3/LMDB/deps/src` )
while loading /home/haroldsoh/.julia/v0.3/LMDB/deps/build.jl, in expression starting on line 41

Provide iterator interface for cursor objects

Hey @wildart I don't know if this is because I'm a noob or not but installing using
pkg> add https://github.com/wildart/LMDB.jl.git#v0.1.0 and then I run Pkg.build("LMDB") and I get an error saying that:
ERROR: LoadError: IOError: could not spawn ldconfig -p: no such file or directory (ENOENT)

I can't tell if this a backwards compatibility issue or if I am missing a base package or not?

The documentation https://wildart.github.io/LMDB.jl/api/LMDB/ source URLs are file URLs, like this: file:///home/art/.julia/v0.4/LMDB/src/env.jl Also, they don't seem to contain line number information.