Code Monkey home page Code Monkey logo

leveldbjni's Introduction

LevelDB JNI

Description

LevelDB JNI gives you a Java interface to the LevelDB C++ library which is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values..

Getting the JAR

Just add the following jar to your java project: leveldbjni-all-1.8.jar

Using as a Maven Dependency

You just need to add the following dependency to your Maven POM:

<dependencies>
  <dependency>
    <groupId>org.fusesource.leveldbjni</groupId>
    <artifactId>leveldbjni-all</artifactId>
    <version>1.8</version>
  </dependency>
</dependencies>

By using the leveldbjni-all dependency, you get the OS specific native drivers for all supported platforms.

If you want to use only one or some but not all native drivers, then directly use the OS specific dependency instead of leveldbjni-all. For example to use Linux 64 bit, use this dependency:

<dependencies>
  <dependency>
    <groupId>org.fusesource.leveldbjni</groupId>
    <artifactId>leveldbjni-linux64</artifactId>
    <version>1.8</version>
  </dependency>
</dependencies>

If you have the leveljni native driver DLL/SO library already separately installed e.g. by a package manager (see issue 90), then you could depend on the Java "launcher" without the JAR containing the OS specific native driver like this:

  <dependency>
    <groupId>org.fusesource.leveldbjni</groupId>
    <artifactId>leveldbjni</artifactId>
    <version>1.8</version>
  </dependency>

Lastly, another project unrelated to this project separately provides a (less mature) pure Java implementation of LevelDB, see dain/leveldb. Note that both that and this project share the same Maven artefact for the Level DB API interface (org.iq80.leveldb:leveldb-api).

API Usage:

Recommended Package imports:

import org.iq80.leveldb.*;
import static org.fusesource.leveldbjni.JniDBFactory.*;
import java.io.*;

Opening and closing the database.

Options options = new Options();
options.createIfMissing(true);
DB db = factory.open(new File("example"), options);
try {
  // Use the db in here....
} finally {
  // Make sure you close the db to shutdown the 
  // database and avoid resource leaks.
  db.close();
}

Putting, Getting, and Deleting key/values.

db.put(bytes("Tampa"), bytes("rocks"));
String value = asString(db.get(bytes("Tampa")));
db.delete(bytes("Tampa"));

Performing Batch/Bulk/Atomic Updates.

WriteBatch batch = db.createWriteBatch();
try {
  batch.delete(bytes("Denver"));
  batch.put(bytes("Tampa"), bytes("green"));
  batch.put(bytes("London"), bytes("red"));

  db.write(batch);
} finally {
  // Make sure you close the batch to avoid resource leaks.
  batch.close();
}

Iterating key/values.

DBIterator iterator = db.iterator();
try {
  for(iterator.seekToFirst(); iterator.hasNext(); iterator.next()) {
    String key = asString(iterator.peekNext().getKey());
    String value = asString(iterator.peekNext().getValue());
    System.out.println(key+" = "+value);
  }
} finally {
  // Make sure you close the iterator to avoid resource leaks.
  iterator.close();
}

Working against a Snapshot view of the Database.

ReadOptions ro = new ReadOptions();
ro.snapshot(db.getSnapshot());
try {
  
  // All read operations will now use the same 
  // consistent view of the data.
  ... = db.iterator(ro);
  ... = db.get(bytes("Tampa"), ro);

} finally {
  // Make sure you close the snapshot to avoid resource leaks.
  ro.snapshot().close();
}

Using a custom Comparator.

DBComparator comparator = new DBComparator(){
    public int compare(byte[] key1, byte[] key2) {
        return new String(key1).compareTo(new String(key2));
    }
    public String name() {
        return "simple";
    }
    public byte[] findShortestSeparator(byte[] start, byte[] limit) {
        return start;
    }
    public byte[] findShortSuccessor(byte[] key) {
        return key;
    }
};
Options options = new Options();
options.comparator(comparator);
DB db = factory.open(new File("example"), options);

Disabling Compression

Options options = new Options();
options.compressionType(CompressionType.NONE);
DB db = factory.open(new File("example"), options);

Configuring the Cache

Options options = new Options();
options.cacheSize(100 * 1048576); // 100MB cache
DB db = factory.open(new File("example"), options);

Getting approximate sizes.

long[] sizes = db.getApproximateSizes(new Range(bytes("a"), bytes("k")), new Range(bytes("k"), bytes("z")));
System.out.println("Size: "+sizes[0]+", "+sizes[1]);

Getting database status.

String stats = db.getProperty("leveldb.stats");
System.out.println(stats);

Getting informational log messages.

Logger logger = new Logger() {
  public void log(String message) {
    System.out.println(message);
  }
};
Options options = new Options();
options.logger(logger);
DB db = factory.open(new File("example"), options);

Destroying a database.

Options options = new Options();
factory.destroy(new File("example"), options);

Repairing a database.

Options options = new Options();
factory.repair(new File("example"), options);

Using a memory pool to make native memory allocations more efficient:

JniDBFactory.pushMemoryPool(1024 * 512);
try {
    // .. work with the DB in here, 
} finally {
    JniDBFactory.popMemoryPool();
}

Building

See also releasing.md:

Prerequisites

Supported Platforms

The following worked for me on:

  • OS X Lion with X Code 4
  • CentOS 5.6 (32 and 64 bit)
  • Ubuntu 12.04 (32 and 64 bit)
  • apt-get install autoconf libtool

Build Procedure

Then download the snappy, leveldb, and leveldbjni project source code:

wget http://snappy.googlecode.com/files/snappy-1.0.5.tar.gz
tar -zxvf snappy-1.0.5.tar.gz
git clone git://github.com/chirino/leveldb.git
git clone git://github.com/fusesource/leveldbjni.git
export SNAPPY_HOME=`cd snappy-1.0.5; pwd`
export LEVELDB_HOME=`cd leveldb; pwd`
export LEVELDBJNI_HOME=`cd leveldbjni; pwd`

Compile the snappy project. This produces a static library.

cd ${SNAPPY_HOME}
./configure --disable-shared --with-pic
make

Patch and Compile the leveldb project. This produces a static library.

cd ${LEVELDB_HOME}
export LIBRARY_PATH=${SNAPPY_HOME}
export C_INCLUDE_PATH=${LIBRARY_PATH}
export CPLUS_INCLUDE_PATH=${LIBRARY_PATH}
git apply ../leveldbjni/leveldb.patch
make libleveldb.a

Now use maven to build the leveldbjni project.

cd ${LEVELDBJNI_HOME}
mvn clean install -P download -P ${platform}

Replace ${platform} with one of the following platform identifiers (depending on the platform your building on):

  • osx
  • linux32
  • linux64
  • win32
  • win64
  • freebsd64

If your platform does not have the right auto-tools levels available just copy the leveldbjni-${version}-SNAPSHOT-native-src.zip artifact from a platform the does have the tools available then add the following argument to your maven build:

-Dnative-src-url=file:leveldbjni-${verision}-SNAPSHOT-native-src.zip

Build Results

  • leveldbjni/target/leveldbjni-${version}.jar : The java class file to the library.
  • leveldbjni/target/leveldbjni-${version}-native-src.zip : A GNU style source project which you can use to build the native library on other systems.
  • leveldbjni-${platform}/target/leveldbjni-${platform}-${version}.jar : A jar file containing the built native library using your currently platform.

leveldbjni's People

Contributors

ayappanec avatar chirino avatar davsclaus avatar dawnbreaks avatar fordguo avatar hsn10 avatar jerrydlamme avatar jsherman1 avatar junhe77 avatar lsb avatar ostinru avatar rgrzywinski avatar tabish121 avatar vorburger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

leveldbjni's Issues

support for Power platform

All the bundled package (osx, linux32,linux64,win32, win64) is for x86 platform.Can the support be enabled for Power platform. i mean a separate directory for linux64 on Power-Platform.Because the problem we are encountering is the artifact from maven repository is not working in Power Platform since the libleveldbjni.so is built for x86 platform. How this can resolved?

allow iteration without copying values

When iterating a range where you only need the keys (e.g. for deletion) it would be better to iterate without copying every value from the native slice (lots of garbage being created).

Currently JniDBIterator.next forces the value part to be always copied. Please provide a different mechanism that avoids this (e.g. an option for iterator or some lazy loading mechanism).

Scala problems

As documented in this stackoverflow post I am having several problems to interact with the library in scala. Anyway, I am not sure if this bug is scala specific or not.

How to use the leveldbjni

I have a try to use it,but always fail, So Do you have any examples to explaining how to use the leveldbjni for beginner??
Thank you!

UnsatisfiedLinkError in window xp sp3

JniDBFactory.factory.open(dbPath, options);
my os is window xp sp3 version,Could U help me!!! i use leveldbjni-1.8.jar (maven leveldbjni-win32 ),it throw exception,as follow:
Exception in thread "main" java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni32-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, C:\Documents and Settings\Justic\Local Settings\Temp\leveldbjni-32-1-55597694747868012.8: 找不到指定的程序。]
at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
at org.fusesource.leveldbjni.JniDBFactory.(JniDBFactory.java:48)
at com.my.jar.one.leveldb.LevelHello.main(LevelHello.java:25)

How could i fix it!!! thank U for your help!!!!

Bug on NativeBuffer offset

Here's a diff.

@@ -99,7 +99,7 @@ class NativeBuffer extends NativeObject {
super(NativeBufferJNI.malloc(length));
this.capacity = length;
this.retained = new AtomicInteger(1);

  •    write(0, data, 0, length);
    
  •    write(0, data, offset, length);
    
    }

Build fails on Linux 64


T E S T S

Running org.fusesource.leveldbjni.test.DBTest
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.424 s
[INFO] Finished at: 2015-01-05T14:42:24-08:00
[INFO] Final Memory: 14M/205M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test (default-test) on project leveldbjni-linux64: There are test failures.
[ERROR]
[ERROR] Please refer to /home/xinyu/Documents/ldb_proj/leveldbjni/leveldbjni-linux64/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]


Looking at the test failure output:
# A fatal error has been detected by the Java Runtime Environment:
3 #
4 # SIGSEGV (0xb) at pc=0x00007f0e7e3397b8, pid=2323, tid=139700776519424
5 #
6 # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build 1.7.0_71-b14)
7 # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode linux-amd64 compressed oops)
8 # Problematic frame:
9 # C [libleveldbjni-64-99-master-SNAPSHOT-8162654786618215726.so+0x2a7b8] leveldb::MemTableIterator::key() const+0x18

Can you help me with this?
Thanks in advance.

IOError while performing random reads on Windows

Hi All,

I am running some tests doing random reads with LevelDBJni on Windows 2008 server. I am running into this exception stack trace consistently when I try to perform random reads. Any pointers would be really helpful.

Some of investigation that I have done:

  1. The disk didn't run out of space, still there is lot of available space.
  2. This is not specific to one machine, it is reproducible everywhere.

org.iq80.leveldb.DBException: IO error: D:\temp\leveldb\randomreadwrite\000785.sst: Could not create random access file.
at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:78)
at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:68)
at com.clearwell.persistentmap.leveldb.LevelDBPersistentMap.get(LevelDBPersistentMap.java:72)
at com.clearwell.persistentmap.leveldb.RandomReadWriteTest.searchData(RandomReadWriteTest.java:88)
at com.clearwell.persistentmap.leveldb.RandomReadWriteTest.testCreateAndSearch(RandomReadWriteTest.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:21)
at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:194)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:53)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:185)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:684)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:391)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: D:\temp\leveldb\randomreadwrite\000785.sst: Could not create random access file.
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:307)
at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:300)
at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:293)
at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:73)

EDIT: Attaching a unit test which reproduces this issue consistently.
https://gist.github.com/bin01/5447282

Windows x64 IO error when opening database

When opening the database with the factory.open(File file, Options options) method on Windows 7 x64, I get the following IOError:

org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: C:\path\without\spaces\to\db\MANIFEST-000005: The handle is invalid.

    at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
    at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
    at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
...

When I looked at MANIFEST-000005 under Notepad++ it looks like some sort file full of nulls and control characters and a bit of text, leveldb.BytewiseComparator. Not sure what that has to do with it, but I can't find a workaround at the moment. The DB is in a gitignored path on my local disk, no spaces. I'm using leveldbjni-all, version 1.7(bundle) out of Maven. Older versions do not fix the issue.

iterator.seek with value higher than anything in db

When I do iterator.seek(byte[] b) with b greater than anything in the database I would expect the cursor to be beyond the last value in the database such that I can still do a hasPrev(). However, this will yield null.

Is this expected behavior (and I should test for that occurrence myself) or not?

Unable to build tag 1.2 with leveldb version 1.4.0

Hi

I tried to build tag 1.2 locally for 64 version. But it having errors and cannot continue further.

[[email protected] fusesource-leveldbjni-12]$ cd leveldbjni-linux64/
[[email protected] leveldbjni-linux64]$ mvn clean deploy -Dleveldb=cd ../../leveldb; pwd -Dsnappy=cd ../../snappy-1.0.3; pwd
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building leveldbjni-linux64 1.2
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ leveldbjni-linux64 ---
[INFO] Deleting file set: /app/test/fusesource-leveldbjni-12/leveldbjni-linux64/target (included: [*], excluded: [])
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ leveldbjni-linux64 ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /app/test/fusesource-leveldbjni-12/leveldbjni-linux64/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ leveldbjni-linux64 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-hawtjni-plugin:1.5:build (default) @ leveldbjni-linux64 ---
[INFO] Extracting /root/.m2/repository/org/fusesource/leveldbjni/leveldbjni/1.2/leveldbjni-1.2-native-src.zip to /app/test/fusesource-leveldbjni-12/leveldbjni-linux64/target/native-build-extracted
[INFO] executing: /bin/sh -c ./configure --disable-ccache --prefix=/app/test/fusesource-leveldbjni-12/leveldbjni-linux64/target/native-build/target --with-leveldb=/app/test/leveldb --with-snappy=/app/test/snappy-1.0.3
[INFO] executing: /bin/sh -c make install
[INFO] /bin/sh ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/app/test/leveldb/include -I/app/j2sdk/include -I/app/j2sdk/include/linux -c -o leveldbjni.lo test -f 'src/leveldbjni.cpp' || echo './'src/leveldbjni.cpp
[INFO] libtool: compile: g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/app/test/leveldb/include -I/app/j2sdk/include -I/app/j2sdk/include/linux -c src/leveldbjni.cpp -fPIC -DPIC -o .libs/leveldbjni.o
[INFO] /bin/sh ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/app/test/leveldb/include -I/app/j2sdk/include -I/app/j2sdk/include/linux -c -o leveldbjni_stats.lo test -f 'src/leveldbjni_stats.cpp' || echo './'src/leveldbjni_stats.cpp
[INFO] libtool: compile: g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/app/test/leveldb/include -I/app/j2sdk/include -I/app/j2sdk/include/linux -c src/leveldbjni_stats.cpp -fPIC -DPIC -o .libs/leveldbjni_stats.o
[INFO] /bin/sh ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/app/test/leveldb/include -I/app/j2sdk/include -I/app/j2sdk/include/linux -c -o leveldbjni_structs.lo test -f 'src/leveldbjni_structs.cpp' || echo './'src/leveldbjni_structs.cpp
[INFO] libtool: compile: g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/app/test/leveldb/include -I/app/j2sdk/include -I/app/j2sdk/include/linux -c src/leveldbjni_structs.cpp -fPIC -DPIC -o .libs/leveldbjni_structs.o
[INFO] /app/test/leveldb/include/leveldb/slice.h: In function 'leveldb::Slice
getNativeSliceFields(JNIEnv_, jobject, leveldb::Slice_)':
[INFO] /app/test/leveldb/include/leveldb/slice.h:81: error: 'const char_ leveldb::Slice::data_' is private
[INFO] src/leveldbjni_structs.cpp:242: error: within this context
[INFO] /app/test/leveldb/include/leveldb/slice.h:82: error: 'size_t leveldb::Slice::size_' is private
[INFO] src/leveldbjni_structs.cpp:243: error: within this context
[INFO] /app/test/leveldb/include/leveldb/slice.h: In function 'void setNativeSliceFields(JNIEnv_, jobject, leveldb::Slice_)':
[INFO] /app/test/leveldb/include/leveldb/slice.h:81: error: 'const char_ leveldb::Slice::data_' is private
[INFO] src/leveldbjni_structs.cpp:250: error: within this context
[INFO] /app/test/leveldb/include/leveldb/slice.h:82: error: 'size_t leveldb::Slice::size_' is private
[INFO] src/leveldbjni_structs.cpp:251: error: within this context
[INFO] make: *** [leveldbjni_structs.lo] Error 1
[INFO] rc: 2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.526s
[INFO] Finished at: Wed May 02 16:43:50 EST 2012
[INFO] Final Memory: 4M/15M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.fusesource.hawtjni:maven-hawtjni-plugin:1.5:build (default) on project leveldbjni-linux64: build failed: org.apache.maven.plugin.MojoExecutionException: make based build failed with exit code: 2 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

leveldbjni build failing

Hi,
I am trying to build leveldbjni following the directions and I get a build error pretty late in. leveldb and snappy build and leveldbjni mostly builds, but when I try to build linux64, it seems like it is missing files. How can this be resolved?

[...]

[INFO] --- maven-hawtjni-plugin:1.5:build (default) @ leveldbjni-linux64 ---
Downloading: http://repo.fusesource.com/nexus/content/groups/public-snapshots/org/fusesource/leveldbjni/leveldbjni/99-master-SNAPSHOT/leveldbjni-99-master-SNAPSHOT-native-src.zip
Downloading: https://oss.sonatype.org/content/repositories/public/org/fusesource/leveldbjni/leveldbjni/99-master-SNAPSHOT/leveldbjni-99-master-SNAPSHOT-native-src.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.704s
[INFO] Finished at: Thu Apr 19 11:39:28 PDT 2012
[INFO] Final Memory: 12M/981M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.fusesource.hawtjni:maven-hawtjni-plugin:1.5:build (default) on project leveldbjni-linux64: build failed: org.apache.maven.plugin.MojoExecutionException: Requested download does not exist. Could not find artifact org.fusesource.leveldbjni:leveldbjni:zip:native-src:99-master-SNAPSHOT in fusesource.nexus.snapshot (http://repo.fusesource.com/nexus/content/groups/public-snapshots)
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.fusesource.leveldbjni -DartifactId=leveldbjni -Dversion=99-master-SNAPSHOT -Dclassifier=native-src -Dpackaging=zip -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.fusesource.leveldbjni -DartifactId=leveldbjni -Dversion=99-master-SNAPSHOT -Dclassifier=native-src -Dpackaging=zip -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]
[ERROR] org.fusesource.leveldbjni:leveldbjni:zip:99-master-SNAPSHOT
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR] fusesource.nexus.snapshot (http://repo.fusesource.com/nexus/content/groups/public-snapshots, releases=true, snapshots=true),
[ERROR] sonatype-nexus (https://oss.sonatype.org/content/repositories/public, releases=true, snapshots=true),
[ERROR] fusesource-releases (http://repo.fusesource.com/nexus/content/groups/public, releases=true, snapshots=false),
[ERROR] fusesource-snapshots (http://repo.fusesource.com/nexus/content/groups/public-snapshots, releases=false, snapshots=true),
[ERROR] central (http://repo1.maven.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Crash when iterating

The error output is below:

Compiled method (nm) 18828440 7291 n 0 org.fusesource.leveldbjni.internal.NativeIterator$IteratorJNI::Valid (native)
total in heap [0x00007f98a70bd310,0x00007f98a70bd698] = 904
relocation [0x00007f98a70bd430,0x00007f98a70bd490] = 96
main code [0x00007f98a70bd4a0,0x00007f98a70bd698] = 504

Last element missing when iterating backwards

I have attached a simple test case which exhibits this behavior. Version is 1.8.

public class TestIterator {

    static final int COUNT = 4;

    public static void main(String[] args) throws IOException {
        Options options = new Options().createIfMissing(true);
        DB db = factory.open(new File("example"), options);

        try {
            DBIterator it = null;
            StringBuffer sb = new StringBuffer();

            for (byte i = 1; i <= COUNT; i++) {
                db.put(new byte[] { i }, new byte[] {});
            }

            sb.append("Forward iterator : ");
            it = db.iterator();
            for (it.seekToFirst(); it.hasNext();) {
                sb.append(it.next().getKey()[0] + " ");
            }

            sb.append("\nBackward iterator: ");
            it = db.iterator();
            for (it.seekToLast(); it.hasPrev();) {
                sb.append(it.prev().getKey()[0] + " ");
            }

            System.out.println(sb.toString());
        } finally {
            if (null != db)
                db.close();
        }
    }
}

Output:

Forward iterator : 1 2 3 4 
Backward iterator: 3 2 1 

SIGSEV on seekToLast, next, peekPrev

DBIterator it = db.iterator())
it.seekToLast();
it.next();
it.peekPrev();

causes

# JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libleveldbjni-64-1-7258230860415032470.7+0x2aef8] leveldb::MemTableIterator::key() const+0x8

SIGSEGV when creating iterator after DB was closed

DB db = JniDBFactory.factory.open(new File("test/jnitest"), options);
db.close();
db.iterator();

I am well aware that this is obviously wrong ;) However, this code results in the JVM core dumping:

SIGSEGV (0xb) at pc=0x0000000176d29df6, pid=7051, tid=4867

JRE version: 7.0_17-b02
Java VM: Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode bsd-amd64 compressed oops)
Problematic frame:
C [libleveldbjni-64-1.6.jnilib+0x1df6] Java_org_fusesource_leveldbjni_internal_NativeDB_00024DBJNI_NewIterator+0x36

I'd rather get an exception telling me that I tried to access the DB object after it was closed. JVM errors are very hard to track down and it took some time until I was able to find the source of the problem.

upgrade to latest LevelDB binaries

Hi,

LevelDB 1.5.0 was release a few months ago. Any chance that there will be a new release of the JNI bindings with this?

Thanks,
Viktor

leveldbjni 1.7 build failure

Enviroment:
java version "1.7.0_25"
OpenJDK Runtime Environment (fedora-2.3.10.3.fc19-i386)
OpenJDK Server VM (build 23.7-b01, mixed mode)

Apache Maven 3.0.5 (rNON-CANONICAL_2013-03-12_12-47_mockbuild; 2013-03-12 13:47:10+0100)
Maven home: /usr/share/maven
Java version: 1.7.0_25, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.i386/jre
Default locale: it_IT, platform encoding: UTF-8
OS name: "linux", version: "3.9.5-301.fc19.i686", arch: "i386", family: "unix"

Dependences:
JAVA
hawtjni 1.6
org.iq80.leveldb:leveldb-api 0.6
C
leveldb 1.9.0
snappy 1.1.0

hi
build fails with the following errors
any ideas?
with 1.2 version i havent this problem
thanks in advance

[INFO] ------------------------------------------------------------------------
[INFO] Building leveldbjni-linux32 1.7
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ leveldbjni-linux32 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory ~/rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ leveldbjni-linux32 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-hawtjni-plugin:1.6:build (default) @ leveldbjni-linux32 ---
[INFO] Extracting ~/rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni/target/leveldbjni-1.7-native-src.zip to /rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build-extracted
[INFO] executing: /bin/sh -c ./configure --disable-ccache --prefix=
/rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/target --with-leveldb=/usr --with-snappy=/usr
[INFO] executing: /bin/sh -c make install
[INFO] CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh ~/rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/autotools/missing --run aclocal-1.12 -I m4
[INFO] ~/rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/autotools/missing: line 51: aclocal-1.12: command not found
[INFO] WARNING: 'aclocal-1.12' is missing on your system. You should only need it if
[INFO] you modified 'acinclude.m4' or 'configure.ac'. You might want
[INFO] to install the Automake and Perl packages. Grab them from
[INFO] any GNU archive site.
[INFO] cd . && /bin/sh /rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/autotools/missing --run automake-1.12 --foreign --ignore-deps
[INFO] /rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/autotools/missing: line 51: automake-1.12: command not found
[INFO] WARNING: 'automake-1.12' is missing on your system. You should only need it if
[INFO] you modified 'Makefile.am', 'acinclude.m4' or 'configure.ac'.
[INFO] You might want to install the Automake and Perl packages.
[INFO] Grab them from any GNU archive site.
[INFO] CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/autotools/missing --run autoconf
[INFO] configure.ac:36: warning: LT_INIT was called before AM_PROG_AR
[INFO] aclocal.m4:73: AM_PROG_AR is expanded from...
[INFO] configure.ac:36: the top level
[INFO] configure.ac:36: warning: AC_PROG_LIBTOOL was called before AM_PROG_AR
[INFO] aclocal.m4:73: AM_PROG_AR is expanded from...
[INFO] configure.ac:36: the top level
[INFO] /bin/sh ./config.status --recheck
[INFO] running CONFIG_SHELL=/bin/sh /bin/sh ./configure --disable-ccache --prefix=
/rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/target --with-leveldb=/usr --with-snappy=/usr --no-create --no-recursion
[INFO] configure: WARNING: unrecognized options: --disable-ccache
[INFO] checking build system type... i686-pc-linux-gnu
[INFO] checking host system type... i686-pc-linux-gnu
[INFO] checking target system type... i686-pc-linux-gnu
[INFO] checking for g++... g++
[INFO] checking whether the C++ compiler works... yes
[INFO] checking for C++ compiler default output file name... a.out
[INFO] checking for suffix of executables...
[INFO] checking whether we are cross compiling... no
[INFO] checking for suffix of object files... o
[INFO] checking whether we are using the GNU C++ compiler... yes
[INFO] checking whether g++ accepts -g... yes
[INFO] checking for a BSD-compatible install... /usr/bin/install -c
[INFO] checking how to print strings... printf
[INFO] checking for gcc... gcc
[INFO] checking whether we are using the GNU C compiler... yes
[INFO] checking whether gcc accepts -g... yes
[INFO] checking for gcc option to accept ISO C89... none needed
[INFO] checking for a sed that does not truncate output... /usr/bin/sed
[INFO] checking for grep that handles long lines and -e... /usr/bin/grep
[INFO] checking for egrep... /usr/bin/grep -E
[INFO] checking for fgrep... /usr/bin/grep -F
[INFO] checking for ld used by gcc... /usr/bin/ld
[INFO] checking if the linker (/usr/bin/ld) is GNU ld... yes
[INFO] checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
[INFO] checking the name lister (/usr/bin/nm -B) interface... BSD nm
[INFO] checking whether ln -s works... yes
[INFO] checking the maximum length of command line arguments... 1572864
[INFO] checking whether the shell understands some XSI constructs... yes
[INFO] checking whether the shell understands "+="... yes
[INFO] checking how to convert i686-pc-linux-gnu file names to i686-pc-linux-gnu format... func_convert_file_noop
[INFO] checking how to convert i686-pc-linux-gnu file names to toolchain format... func_convert_file_noop
[INFO] checking for /usr/bin/ld option to reload object files... -r
[INFO] checking for objdump... objdump
[INFO] checking how to recognize dependent libraries... pass_all
[INFO] checking for dlltool... no
[INFO] checking how to associate runtime and link libraries... printf %s\n
[INFO] checking for ar... ar
[INFO] checking for archiver @file support... @
[INFO] checking for strip... strip
[INFO] checking for ranlib... ranlib
[INFO] checking for gawk... gawk
[INFO] checking command to parse /usr/bin/nm -B output from gcc object... ok
[INFO] checking for sysroot... no
[INFO] checking for mt... no
[INFO] checking if : is a manifest tool... no
[INFO] checking how to run the C preprocessor... gcc -E
[INFO] checking for ANSI C header files... yes
[INFO] checking for sys/types.h... yes
[INFO] checking for sys/stat.h... yes
[INFO] checking for stdlib.h... yes
[INFO] checking for string.h... yes
[INFO] checking for memory.h... yes
[INFO] checking for strings.h... yes
[INFO] checking for inttypes.h... yes
[INFO] checking for stdint.h... yes
[INFO] checking for unistd.h... yes
[INFO] checking for dlfcn.h... yes
[INFO] checking for objdir... .libs
[INFO] checking if gcc supports -fno-rtti -fno-exceptions... no
[INFO] checking for gcc option to produce PIC... -fPIC -DPIC
[INFO] checking if gcc PIC flag -fPIC -DPIC works... yes
[INFO] checking if gcc static flag -static works... no
[INFO] checking if gcc supports -c -o file.o... yes
[INFO] checking if gcc supports -c -o file.o... (cached) yes
[INFO] checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes
[INFO] checking whether -lc should be explicitly linked in... no
[INFO] checking dynamic linker characteristics... GNU/Linux ld.so
[INFO] checking how to hardcode library paths into programs... immediate
[INFO] checking whether stripping libraries is possible... yes
[INFO] checking if libtool supports shared libraries... yes
[INFO] checking whether to build shared libraries... yes
[INFO] checking whether to build static libraries... no
[INFO] checking how to run the C++ preprocessor... g++ -E
[INFO] checking for ld used by g++... /usr/bin/ld
[INFO] checking if the linker (/usr/bin/ld) is GNU ld... yes
[INFO] checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
[INFO] checking for g++ option to produce PIC... -fPIC -DPIC
[INFO] checking if g++ PIC flag -fPIC -DPIC works... yes
[INFO] checking if g++ static flag -static works... no
[INFO] checking if g++ supports -c -o file.o... yes
[INFO] checking if g++ supports -c -o file.o... (cached) yes
[INFO] checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
[INFO] checking dynamic linker characteristics... (cached) GNU/Linux ld.so
[INFO] checking how to hardcode library paths into programs... immediate
[INFO] checking the archiver (ar) interface... ar
[INFO] configure: JAVA_HOME was set, checking to see if it's a JDK we can use...
[INFO] checking if '/usr/lib/jvm/java-1.7.0' is a JDK... yes
[INFO] which: no javac in (/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/local/sbin:/usr/sbin:
/.local/bin:
/bin)
[INFO] checking pthread.h usability... yes
[INFO] checking pthread.h presence... yes
[INFO] checking for pthread.h... yes
[INFO] checking leveldb/db.h usability... yes
[INFO] checking leveldb/db.h presence... yes
[INFO] checking for leveldb/db.h... yes
[INFO] checking sys/errno.h usability... yes
[INFO] checking sys/errno.h presence... yes
[INFO] checking for sys/errno.h... yes
[INFO] checking whether build environment is sane... yes
[INFO] checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
[INFO] checking whether make sets $(MAKE)... yes
[INFO] checking that generated files are newer than configure... done
[INFO] configure: creating ./config.status
[INFO] configure: WARNING: unrecognized options: --disable-ccache
[INFO]
INFO version 1.7
[INFO] Prefix.........: ~/rpmbuild/BUILD/leveldbjni-leveldbjni-1.7/leveldbjni-linux32/target/native-build/target
[INFO] C Compiler.....: gcc -g -O2 -I/usr/include -I/usr/lib/jvm/java-1.7.0/include -I/usr/lib/jvm/java-1.7.0/include/linux
[INFO] Linker.........: /usr/bin/ld -lleveldb -L/usr -lsnappy -L/usr -release 1.7
[INFO]
[INFO] /bin/sh ./config.status
[INFO] config.status: creating Makefile
[INFO] config.status: creating src/config.h
[INFO] config.status: executing libtool commands
[INFO] /bin/sh ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/usr/include -I/usr/lib/jvm/java-1.7.0/include -I/usr/lib/jvm/java-1.7.0/include/linux -c -o leveldbjni.lo test -f 'src/leveldbjni.cpp' || echo './'src/leveldbjni.cpp
[INFO] libtool: compile: g++ -DHAVE_CONFIG_H -I. -I./src -g -O2 -I/usr/include -I/usr/lib/jvm/java-1.7.0/include -I/usr/lib/jvm/java-1.7.0/include/linux -c src/leveldbjni.cpp -fPIC -DPIC -o .libs/leveldbjni.o
[INFO] src/leveldbjni.cpp: In function 'void Java_org_fusesource_leveldbjni_internal_NativeDB_00024DBJNI_ResumeCompactions(JNIEnv_, jclass, jlong)':
[INFO] src/leveldbjni.cpp:426:35: error: 'class leveldb::DB' has no member named 'ResumeCompactions'
[INFO](%28leveldb::DB *%29%28intptr_t%29arg0)->ResumeCompactions();
[INFO] ^
[INFO] src/leveldbjni.cpp: In function 'void Java_org_fusesource_leveldbjni_internal_NativeDB_00024DBJNI_SuspendCompactions(JNIEnv_, jclass, jlong)':
[INFO] src/leveldbjni.cpp:435:35: error: 'class leveldb::DB' has no member named 'SuspendCompactions'
[INFO](%28leveldb::DB *%29%28intptr_t%29arg0)->SuspendCompactions();
[INFO] ^
[INFO] make: *** [leveldbjni.lo] Error 1
[INFO] rc: 2
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Skipping leveldbjni-project
[INFO] This project has been banned from the build due to previous failures.
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] leveldbjni-project ................................ SUCCESS [0.001s]
[INFO] leveldbjni ........................................ SUCCESS [4.529s]
[INFO] leveldbjni-linux32 ................................ FAILURE [14.818s]
[INFO] leveldbjni-all .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

maven build failing

I'm following the build description from the website step by step and I'm getting the following maven build error on a ubuntu 64bit vm :

...
[INFO] Reactor Summary:
[INFO]
[INFO] leveldbjni-project ................................ SUCCESS [1.193s]
[INFO] leveldbjni ........................................ SUCCESS [8.553s]
[INFO] leveldbjni-linux64 ................................ FAILURE [0.296s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.266s
[INFO] Finished at: Mon Mar 05 16:57:42 CET 2012
[INFO] Final Memory: 14M/34M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.fusesource.hawtjni:maven-hawtjni-plugin:1.5:build (default) on project leveldbjni-linux64: build failed: org.apache.maven.plugin.MojoExecutionException: Extracted package did not look like it contained a native source build. -> [Help 1]
...

Can you please help me with that?

Thanks,
Dan

Why CDDL?

I'd like to start with saying this project rocks. I was up and running with leveldb in no time.

However if I actually wanted to use this project for anything real I'd have reservations about the CDDL. Both projects this depends on have much better known and permissive licenses. LevelDb is new Berkley and hawtjni is apache 2. Would it be possible to go with either of these instead of CDDL?

Add OSGi meta-data to the jar

I would like to use leveldbjni in an OSGi environment. I was able to add the OSGi meta-data to a pre-built leveldbjni jar and make it work correctly. However, it would be great if this could be applied upstream. I would have applied it directly here, but the build is a little different than most java projects because of the native code, so I wasn't sure how to correctly modify the build for this.

I did, however, create a servicemix-bundle that adds the meta-data after the fact. It is likely the logic there can be used directly on the upstream project, without the shade part:

vivosys/servicemix4-bundles@8857fc4

If this can be added in the project, then I won't bother submitting the servicemix patch as it will be obsolete.

"java.lang.AssertionError: This object has been deleted" when running tests in Maven

Hi,

I have this very simple test case:

public class LevelDBJniTest {

    @Test
    public void test() throws IOException {
        Options options = new Options();
        options.createIfMissing(true);

        Util.emptyDirectory(new File("test/jnitest"));

        DB db = JniDBFactory.factory.open(new File("test/jnitest"), options);

        for (int i = 0; i < 1024 * 1024; i++) {
            byte[] key = ByteBuffer.allocate(4).putInt(i).array();
            byte[] value = ByteBuffer.allocate(4).putInt(-i).array();

            db.put(key, value);

            assertTrue(Arrays.equals(db.get(key), value));
        }

        db.close();
    }

}

It works fine if I run it from Eclipse, but fails when run using mvn:test from the command line on the very first invocation of JniDB.put. Here is the command line:

test(edu.kit.aifb.ats.index.lsm.leveldb.test.LevelDBJniTest) Time elapsed: 53 sec <<< FAILURE!
java.lang.AssertionError: This object has been deleted
at org.fusesource.leveldbjni.internal.NativeObject.assertAllocated(NativeObject.java:60)
at org.fusesource.leveldbjni.internal.NativeBuffer$Allocation.release(NativeBuffer.java:109)
at org.fusesource.leveldbjni.internal.NativeBuffer.delete(NativeBuffer.java:250)
at org.fusesource.leveldbjni.internal.NativeDB.put(NativeDB.java:249)
at org.fusesource.leveldbjni.internal.JniDB.put(JniDB.java:108)
at org.fusesource.leveldbjni.internal.JniDB.put(JniDB.java:91)
at edu.kit.aifb.ats.index.lsm.leveldb.test.LevelDBJniTest.test(LevelDBJniTest.java:32)

Running leveldbjni code from the command line (outside of mvn:test) works fine, though. I can only guess that the Maven test environment somehow interferes.

When I execute the JUnit runner directly without mvn:test, the test also works:

mvn exec:java -Dexec.mainClass="org.junit.runner.JUnitCore" -Dexec.arguments="edu.kit.aifb.ats.index.lsm.leveldb.LevelDBJniTest"
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building ats-index-leveldb 0.1.0-reorg-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1.jbossorg-3:java (default-cli) @ ats-index-leveldb >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1.jbossorg-3:java (default-cli) @ ats-index-leveldb <<<
[INFO]
[INFO] --- exec-maven-plugin:1.2.1.jbossorg-3:java (default-cli) @ ats-index-leveldb ---
JUnit version 4.10
.
Time: 8,154

OK (1 test)

This is on MacOS X Lion with leveldbjni 1.5 and Java 7:

java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Any ideas?

how to release the lock

Hi,

I am using leveldbjni with clojure, and open a levedb with a specific directory, and then if I open the leveldb again without restart, it will prevent me from opening the db, says that the db is held by process no matter I repair or destroy the levedb for that directory.

Any suggestion to release the lock and open the db again?

Thanks

Julius

Update readme file - Better explain java vs native drivers

If you use maven and include the -all JAR (from 1.8 onwards) you get the java driver out of the box.

If you want to use the linux64 driver you need to add that as a explicit maven dependency. Out of the box the drivers has provided and hence

The documentation should be updated to better explain that.

UnsatisfiedLinkError under Ubuntu 12.04.2

When I try to use leveldbjni (leveldbjni-all-1.7.jar) under Ubuntu 12.04.2, there seems to be a problem loading the native library from inside the JAR. I get the following Exception:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.7 in java.library.path, no leveldbjni-1.7 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-1889838573919501963.7: /tmp/libleveldbjni-64-1-1889838573919501963.7: failed to map segment from shared object: Operation not permitted]
    at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
    at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
    at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
    at Test.main(Test.java:13)

My test class:

import org.fusesource.leveldbjni.JniDBFactory;
import org.iq80.leveldb.DB;
import org.iq80.leveldb.DBIterator;
import org.iq80.leveldb.Options;
import java.io.File;

public class Test {
    public static void main(String[] args) throws Exception {
        DB db = JniDBFactory.factory.open(new File("./db"), new Options());
        System.out.println("Opened: "+db);
    }
}

Steps to reproduce:

javac -cp leveldbjni-all-1.7.jar Test.java
java -cp .:./leveldbjni-all-1.7.jar Test

This test case works as expected on the following systems:

  • OSX 10.8.5
  • Ubuntu 12.04.1
  • Fedora release 15 (Lovelock)
  • Amazon Linux AMI release 2013.03

I have only been able to reproduce this on Ubuntu 12.04.2.

Can't load dlls on Windows 2012. Can't find msvc dlls.

JniDBFactory.factory fails to initialize because it can't load the dll.
The problem is that the windows64 dll depends on msvcp100.dll and msvcr100.dll. These two dlls are included on Win7 (c:\Windows\System32) but they are not included on Win2012.

UnsatisfiedLinkError in win32

I have tried this library, both version 1.6 and 1.5, on a Winxp, 32-bit machine and get the following exception at runtime on the factory.open(new File("example"), options) method from the example.

Exception in thread "main" java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni32-1.6 in java.library.path, no leveldbjni-1.6 in java.library.path, no leveldbjni in java.library.path, C:\Documents and Settings\planck0\Local Settings\Temp\leveldbjni-32-1.6.dll: The specified procedure could not be found]
    at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:184)
    at org.fusesource.hawtjni.runtime.Library.load(Library.java:142)
    at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
    at levelDBConnection.OpenDB(levelDBConnection.java:33)
    at testRunClass.main(testRunClass.java:31)

I've run the same code on both linux and windows 7 (64 bit) and it runs fine, just not on winxp 32 bit.

Error when opening paths with unicode characters

Hello,

I'm using leveldbjni 1.7 as Akka persistence backend and I found the following problem when the path contains unicode characters (Félix) in this case.

Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: C:\Users\Félix\AppData\Roaming\Coinffeine\journal\LOCK: Could not lock file.
    at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) ~[leveldbjni-1.7.jar:1.7]
...

Maybe I'm doing something wrong or there is a workaround you know of.

build failed on max os, cannot find headers for leveldb/db.h

I have done exactly as the readme.

two problems:

  1. [INFO] configure: WARNING: unrecognized options: --disable-ccache

[INFO] checking leveldb/db.h usability... no
[INFO] checking leveldb/db.h presence... no
[INFO] checking for leveldb/db.h... no
[INFO] configure: error: cannot find headers for leveldb

These are the output of the command:
mvn clean install -P download -P osx

[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.fusesource.leveldbjni:leveldbjni:jar:99-master-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-jar-plugin @ line 168, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] leveldbjni-project
[INFO] leveldbjni
[INFO] leveldbjni-osx
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building leveldbjni-project 99-master-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ leveldbjni-project ---
[INFO]
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ leveldbjni-project ---
[INFO] Installing /Users/zhuhaidong/projects/leveldb-jni/pom.xml to /Users/zhuhaidong/.m2/repository/org/fusesource/leveldbjni/leveldbjni-project/99-master-SNAPSHOT/leveldbjni-project-99-master-SNAPSHOT.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building leveldbjni 99-master-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ leveldbjni ---
[INFO] Deleting file set: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target (included: [**], excluded: [])
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ leveldbjni ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ leveldbjni ---
[INFO] Compiling 23 source files to /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/classes
[INFO]
[INFO] --- maven-hawtjni-plugin:1.9:generate (default) @ leveldbjni ---
[INFO] Analyzing classes...
[INFO] Generating...
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/leveldbjni.cpp
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/leveldbjni_stats.h
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/leveldbjni_stats.cpp
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/leveldbjni_structs.h
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/leveldbjni_structs.cpp
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/hawtjni.h
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/hawtjni.c
[INFO] Wrote: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/generated-sources/hawtjni/native-src/windows/stdint.h
[INFO] Done.
[INFO]
[INFO] --- maven-bundle-plugin:2.0.1:manifest (bundle-manifest) @ leveldbjni ---
[WARNING] Warning in manifest for org.fusesource.leveldbjni:leveldbjni:jar:99-master-SNAPSHOT : Superfluous export-package instructions: [org.fusesource, org]
[INFO]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ leveldbjni ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ leveldbjni ---
[INFO] Compiling 1 source file to /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.4.3:test (default-test) @ leveldbjni ---
[INFO] Surefire report directory: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/surefire-reports


T E S T S

There are no tests to run.

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ leveldbjni ---
[INFO] Building jar: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT.jar
[INFO]
[INFO] --- maven-jar-plugin:2.3.1:test-jar (default) @ leveldbjni ---
[INFO] Building jar: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-hawtjni-plugin:1.9:package-source (default) @ leveldbjni ---
[INFO] Building zip: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT-native-src.zip
[INFO]
[INFO] --- maven-source-plugin:2.2.1:jar-no-fork (attach-sources) @ leveldbjni ---
[INFO] Building jar: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ leveldbjni ---
[INFO] Installing /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT.jar to /Users/zhuhaidong/.m2/repository/org/fusesource/leveldbjni/leveldbjni/99-master-SNAPSHOT/leveldbjni-99-master-SNAPSHOT.jar
[INFO] Installing /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/pom.xml to /Users/zhuhaidong/.m2/repository/org/fusesource/leveldbjni/leveldbjni/99-master-SNAPSHOT/leveldbjni-99-master-SNAPSHOT.pom
[INFO] Installing /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT-tests.jar to /Users/zhuhaidong/.m2/repository/org/fusesource/leveldbjni/leveldbjni/99-master-SNAPSHOT/leveldbjni-99-master-SNAPSHOT-tests.jar
[INFO] Installing /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT-native-src.zip to /Users/zhuhaidong/.m2/repository/org/fusesource/leveldbjni/leveldbjni/99-master-SNAPSHOT/leveldbjni-99-master-SNAPSHOT-native-src.zip
[INFO] Installing /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT-sources.jar to /Users/zhuhaidong/.m2/repository/org/fusesource/leveldbjni/leveldbjni/99-master-SNAPSHOT/leveldbjni-99-master-SNAPSHOT-sources.jar
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building leveldbjni-osx 99-master-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ leveldbjni-osx ---
[INFO] Deleting file set: /Users/zhuhaidong/projects/leveldb-jni/leveldbjni-osx/target (included: [**], excluded: [])
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ leveldbjni-osx ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/zhuhaidong/projects/leveldb-jni/leveldbjni-osx/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ leveldbjni-osx ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-hawtjni-plugin:1.9:build (default) @ leveldbjni-osx ---
[INFO] Extracting /Users/zhuhaidong/projects/leveldb-jni/leveldbjni/target/leveldbjni-99-master-SNAPSHOT-native-src.zip to /Users/zhuhaidong/projects/leveldb-jni/leveldbjni-osx/target/native-build-extracted
[INFO] executing: /bin/sh -c ./configure --disable-ccache --prefix=/Users/zhuhaidong/projects/leveldb-jni/leveldbjni-osx/target/native-build/target --with-leveldb=/Users/zhuhaidong/project/leveldb --with-snappy=/Users/zhuhaidong/project/snappy-1.0.5 --with-universal
[INFO] configure: WARNING: unrecognized options: --disable-ccache
[INFO] checking build system type... x86_64-apple-darwin12.5.0
[INFO] checking host system type... x86_64-apple-darwin12.5.0
[INFO] checking target system type... x86_64-apple-darwin12.5.0
[INFO] checking for g++... g++
[INFO] checking whether the C++ compiler works... yes
[INFO] checking for C++ compiler default output file name... a.out
[INFO] checking for suffix of executables...
[INFO] checking whether we are cross compiling... no
[INFO] checking for suffix of object files... o
[INFO] checking whether we are using the GNU C++ compiler... yes
[INFO] checking whether g++ accepts -g... yes
[INFO] checking for a BSD-compatible install... /usr/bin/install -c
[INFO] checking how to print strings... printf
[INFO] checking for gcc... gcc
[INFO] checking whether we are using the GNU C compiler... yes
[INFO] checking whether gcc accepts -g... yes
[INFO] checking for gcc option to accept ISO C89... none needed
[INFO] checking for a sed that does not truncate output... /usr/bin/sed
[INFO] checking for grep that handles long lines and -e... /usr/bin/grep
[INFO] checking for egrep... /usr/bin/grep -E
[INFO] checking for fgrep... /usr/bin/grep -F
[INFO] checking for ld used by gcc... /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld
[INFO] checking if the linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU ld... no
[INFO] checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm
[INFO] checking the name lister (/usr/bin/nm) interface... BSD nm
[INFO] checking whether ln -s works... yes
[INFO] checking the maximum length of command line arguments... 196608
[INFO] checking whether the shell understands some XSI constructs... yes
[INFO] checking whether the shell understands "+="... yes
[INFO] checking how to convert x86_64-apple-darwin12.5.0 file names to x86_64-apple-darwin12.5.0 format... func_convert_file_noop
[INFO] checking how to convert x86_64-apple-darwin12.5.0 file names to toolchain format... func_convert_file_noop
[INFO] checking for /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld option to reload object files... -r
[INFO] checking for objdump... no
[INFO] checking how to recognize dependent libraries... pass_all
[INFO] checking for dlltool... no
[INFO] checking how to associate runtime and link libraries... printf %s\n
[INFO] checking for ar... ar
[INFO] checking for archiver @file support... no
[INFO] checking for strip... strip
[INFO] checking for ranlib... ranlib
[INFO] checking for gawk... no
[INFO] checking for mawk... no
[INFO] checking for nawk... no
[INFO] checking for awk... awk
[INFO] checking command to parse /usr/bin/nm output from gcc object... ok
[INFO] checking for sysroot... no
[INFO] checking for mt... no
[INFO] checking if : is a manifest tool... no
[INFO] checking for dsymutil... dsymutil
[INFO] checking for nmedit... nmedit
[INFO] checking for lipo... lipo
[INFO] checking for otool... otool
[INFO] checking for otool64... no
[INFO] checking for -single_module linker flag... yes
[INFO] checking for -exported_symbols_list linker flag... yes
[INFO] checking for -force_load linker flag... yes
[INFO] checking how to run the C preprocessor... gcc -E
[INFO] checking for ANSI C header files... yes
[INFO] checking for sys/types.h... yes
[INFO] checking for sys/stat.h... yes
[INFO] checking for stdlib.h... yes
[INFO] checking for string.h... yes
[INFO] checking for memory.h... yes
[INFO] checking for strings.h... yes
[INFO] checking for inttypes.h... yes
[INFO] checking for stdint.h... yes
[INFO] checking for unistd.h... yes
[INFO] checking for dlfcn.h... yes
[INFO] checking for objdir... .libs
[INFO] checking if gcc supports -fno-rtti -fno-exceptions... no
[INFO] checking for gcc option to produce PIC... -fno-common -DPIC
[INFO] checking if gcc PIC flag -fno-common -DPIC works... yes
[INFO] checking if gcc static flag -static works... no
[INFO] checking if gcc supports -c -o file.o... yes
[INFO] checking if gcc supports -c -o file.o... (cached) yes
[INFO] checking whether the gcc linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) supports shared libraries... yes
[INFO] checking dynamic linker characteristics... darwin12.5.0 dyld
[INFO] checking how to hardcode library paths into programs... immediate
[INFO] checking whether stripping libraries is possible... yes
[INFO] checking if libtool supports shared libraries... yes
[INFO] checking whether to build shared libraries... yes
[INFO] checking whether to build static libraries... no
[INFO] checking how to run the C++ preprocessor... g++ -E
[INFO] checking for ld used by g++... /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld
[INFO] checking if the linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU ld... no
[INFO] checking whether the g++ linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) supports shared libraries... yes
[INFO] checking for g++ option to produce PIC... -fno-common -DPIC
[INFO] checking if g++ PIC flag -fno-common -DPIC works... yes
[INFO] checking if g++ static flag -static works... no
[INFO] checking if g++ supports -c -o file.o... yes
[INFO] checking if g++ supports -c -o file.o... (cached) yes
[INFO] checking whether the g++ linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) supports shared libraries... yes
[INFO] checking dynamic linker characteristics... darwin12.5.0 dyld
[INFO] checking how to hardcode library paths into programs... immediate
[INFO] checking the archiver (ar) interface... ar
[INFO] configure: JAVA_HOME was set, checking to see if it's a JDK we can use...
[INFO] checking if '/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home' is a JDK... no
[INFO] configure: javac was on your path, checking to see if it's part of a JDK we can use...
[INFO] checking if '/System/Library/Frameworks/JavaVM.framework/Versions/Current' is a JDK... yes
[INFO] checking pthread.h usability... yes
[INFO] checking pthread.h presence... yes
[INFO] checking for pthread.h... yes
[INFO] checking leveldb/db.h usability... no
[INFO] checking leveldb/db.h presence... no
[INFO] checking for leveldb/db.h... no
[INFO] configure: error: cannot find headers for leveldb
[INFO] rc: 1
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] leveldbjni-project ................................ SUCCESS [0.358s]
[INFO] leveldbjni ........................................ SUCCESS [10.958s]
[INFO] leveldbjni-osx .................................... FAILURE [4.637s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16.401s
[INFO] Finished at: Wed Sep 25 18:07:59 CST 2013
[INFO] Final Memory: 13M/255M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.fusesource.hawtjni:maven-hawtjni-plugin:1.9:build (default) on project leveldbjni-osx: build failed: org.apache.maven.plugin.MojoExecutionException: ./configure failed with exit code: 1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :leveldbjni-osx

Problem opening my leveldb database

After using leveldb in my application for a few days now i started seeing this exception

Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /filarkiv/database/journal/ 000119.sst: Cannot allocate memory at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) ~[leveldbjni-all-1. 8.jar:1.8] at org.fusesource.leveldbjni.internal.NativeIterator.checkStatus(NativeIterator.java:121) ~[level dbjni-all-1.8.jar:1.8] at org.fusesource.leveldbjni.internal.NativeIterator.next(NativeIterator.java:157) ~[leveldbjni-a ll-1.8.jar:1.8] at org.fusesource.leveldbjni.internal.JniDBIterator.next(JniDBIterator.java:100) ~[leveldbjni-all -1.8.jar:1.8]

I have 1 leveldb running with 100mb cache and the files take 32mb ram.
I have 7 other leveldb stores which i open on demand every other hour with 10mb cache. Its seems random which leveldb gets this error.

Heap has around 1gb of ram left.

If i restart the computer, start the java process it might work for a while.

I open leveldb like this:

if (db == null) {
            Options options = new Options();
            options.cacheSize(cacheSizeInMB * 1048576); // MB cache
            options.createIfMissing(true);
            if(!new File(directory).exists())
                new File(directory).mkdirs();
            try {
                db = factory.open(new File(directory), options);
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }

close it like this: "db.close();"

Compression algorithm

Hey Guys,

We are using leveldbjni and very successfully , thanks for you work.
Yesterday tried to turn on the compression option but looks like
the implementation is not using any. Do you think we need some
special build of native lib for that ?

Thanks in advance.

@chirino

JVM crashes with SIGBUS signal on opening DB from Java code

Hi Leveldb JNI folks,

Recently we encountered one JVM crash issue when using leveldbjni 1.8. Bellow is some information from hs_err_pid.log. According to the log, it seems that JVM crashed when the app tried to open DB files.

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f3512b19990, pid=6732, tid=139865935472384
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x89990]  memcpy+0x320
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.fusesource.leveldbjni.internal.NativeDB$DBJNI.Open(Lorg/fusesource/leveldbjni/internal/NativeOptions;Ljava/lang/String;[J)J+0
j  org.fusesource.leveldbjni.internal.NativeDB.open(Lorg/fusesource/leveldbjni/internal/NativeOptions;Ljava/io/File;)Lorg/fusesource/leveldbjni/internal/NativeDB;+22
j  org.fusesource.leveldbjni.JniDBFactory.open(Ljava/io/File;Lorg/iq80/leveldb/Options;)Lorg/iq80/leveldb/DB;+24

Do you have any idea what could be the cause of JVM crash? It's appreciated if you could share your opinions.

Thanks,
Zhijie

SIGSEGV on JVM termination

I am writing a program that uses several LevelDB databases (some of them in different threads). Everything works as expected but after the main method finishes execution and the JVM starts to shut down I get this:

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fd24b1e0ea8, pid=12289, tid=140541201016576

JRE version: 7.0_21-b02
Java VM: OpenJDK 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
Problematic frame:
V [libjvm.so+0x6b2ea8] Monitor::ILock(Thread*) [clone .part.73]+0x18

Memory leak with pooled memory

We have been experiencing memory leaks when using pooled memory. The problem resides in NativeBuffer.Pool.create() (along with its interaction with the delete() and release() methods). Consider the case where there are two create() methods called before the corresponding delete()s are called (as is the case in NativeDB.put()) and the total size of the key-value is less than the chunk size (but each of the key and value individually are smaller than the chunk size). It is possible for the first create() to create an allocation (via NativeBuffer.Pool.allocate()) but the second create() to trigger remaining < size and have delete() called.

The memory leak occurs because delete() calls allocation.release() which will not actually release the memory because the NativeBuffer from the first create() still has a reference to it but yet self is set to 0! This means that when the reference counter finally gets to zero that nothing is actually free'd since the pointer (self) isn't pointing to the allocated memory any more.

At this point, I'd wager that the fix is to simply move self = 0 into else if block above where it is now.

Memory Pool clarification

The Readme file mentions a memory pool:

"Using a memory pool to make native memory allocations more efficient"

It is not entirely clear for me how to interpret this and I coudn't find the answers easily in the code either. How does it make memory allocations more efficient? What performance gain can I expect and at what (memory) cost? What is the input unit.. bytes, kilobytes (1024 * 512 = 512K?)? How do I determine its optimal value, should it be equal or somehow relate to block-size, cache-size, or what is the general idea?

Concurrent updates with LevelDb JNI

When we try to do the bulk update with level db, it appears that is not thread-safe and breaks when updates are made to its API concurrently. Please suggest what can be done for this.

leveldbjni-all dependencies are redundant

I'm using leveldbjni-all as a dependency from a gradle build. The pom.xml for leveldbjni-all lists all the other packages (leveldb-api, linux/osx/win32, etc) as a dependency, but it also includes them in the leveldbjni-all.jar (as expected). In the end, this yields to a redundant inclusion of these artifacts in the project.

I'm no maven expert, but I imagine this would be fixed by changing the dependencies in pom.xml to scope = provided (see http://maven.apache.org/pom.html#Dependencies).

Freebsd

Woud you please make it works on Freebsd? I tried but found it failed when saving UTF-8.

get() on a nonexistent key works through exception

I'm trying to work with leveldbjni-linux64-1.2-SNAPSHOT and found that get() on a nonexistent value works through

checkStatus(DBJNI.Get(self, options, keySlice, result.pointer()));

that throws exception if value was not found. Exceptions are usually not too good for performance and get() miss in my case is not an exceptional case.

The following test works 2700 ms with exceptions and 600ms after exceptions were removed in case of missed get():

import org.fusesource.leveldbjni.JniDBFactory;
import org.fusesource.leveldbjni.*;
import org.iq80.leveldb.*;

import java.io.File;
import java.io.IOException;
import java.util.*;

import static org.fusesource.leveldbjni.JniDBFactory.asString;
import static org.fusesource.leveldbjni.JniDBFactory.bytes;

public class Test {
    static final DBFactory factory = JniDBFactory.factory;

    public static void main(String[] args) throws Exception {
        Options options = new Options().createIfMissing(true);

        options.cacheSize(120 * 1024 * 1024);
        options.writeBufferSize(8 * 1024 * 1024);
        options.paranoidChecks(false);
        options.verifyChecksums(false);
        options.blockSize(4096);

        DB db = factory.open(new File("gagaga"), options);

        byte[] key = new byte[] {0, 0, 0, 0, -84, -19, 0, 5, 116, 0, 4, 107, 101, 121, 48};

        long start = System.currentTimeMillis();

        for(int i = 0; i < 1000000; i++) {
            db.get(key);
        }

        long time = System.currentTimeMillis() - start;

        System.out.println("test tool " + time + "ms");

        db.close();

    }
}

The "patch" is:

--- a/leveldbjni/src/main/java/org/fusesource/leveldbjni/internal/NativeDB.java
+++ b/leveldbjni/src/main/java/org/fusesource/leveldbjni/internal/NativeDB.java
@@ -287,8 +287,22 @@ public class NativeDB extends NativeObject {
         assertAllocated();
         NativeStdString result = new NativeStdString();
         try {
-            checkStatus(DBJNI.Get(self, options, keySlice, result.pointer()));
-            return result.toByteArray();
+            long s = DBJNI.Get(self, options, keySlice, result.pointer());
+
+            NativeStatus status = new NativeStatus(s);
+            try {
+                if(status.isOk()) {
+                    return result.toByteArray();
+                }
+
+                if(status.isNotFound()) {
+                    return null;
+                }
+                
+                throw new DBException(status.toString(), status.isNotFound());
+            } finally {
+                status.delete();
+            }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.