diennea / blobit Goto Github PK

View Code? Open in Web Editor NEW

35.0 35.0 12.0 1.11 MB

BlobIt - a Distributed Large Object Storage

Home Page: https://blobit.org

License: Apache License 2.0

Java 96.93% Shell 1.98% CSS 1.08%

blob-storage bookkeeper distributed java storage

blobit's People

Contributors

Stargazers

Watchers

Forkers

taurus3g codelipenghui rudy2steiner yhilem apurbad hamadodene dmercuriali marvincai gauravashok misselvexu stevenlumt syk-coder

blobit's Issues

Bookie - Write ledger header

We need to write an entry to the ledger before adding it to the list of active ledgers to work around a possible fault case in BookKeeper when there is no entry on the Bookie and one Bookie goes down.

See diennea/herddb#572 for more information

Error while sending reply message to PooledUnsafeDi rectByteBuf(freed) while talking to null

20-03-17-12-02-15       java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error java.util.concurrent.ExecutionException: java.lang.Exception: herddb.network.netty.NettyChannel$3@5bdb86aa: error while sending reply m
essage to PooledUnsafeDirectByteBuf(freed) while talking to null
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error java.util.concurrent.ExecutionException: java.lang.Exception: herddb.network.netty.NettyChannel$3@5bdb86aa: error while sending reply message to PooledUnsafeDi
rectByteBuf(freed) while talking to null
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at magnews.gateway.archiver.Archiver.run(Archiver.java:319)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.lang.RuntimeException: Error java.util.concurrent.ExecutionException: java.lang.Exception: herddb.network.netty.NettyChannel$3@5bdb86aa: error while sending reply message to PooledUnsafeDirectByteBuf(freed) while talki
ng to null
        at herddb.network.Channel.sendMessageWithPduReply(Channel.java:95)
        at herddb.client.RoutedClientSideConnection.prepareQuery(RoutedClientSideConnection.java:347)
        at herddb.client.RoutedClientSideConnection.executeScan(RoutedClientSideConnection.java:699)
        at herddb.client.HDBConnection.executeScan(HDBConnection.java:345)
        at herddb.jdbc.HerdDBPreparedStatement.executeQuery(HerdDBPreparedStatement.java:74)
        at org.blobit.core.cluster.HerdDBMetadataStorageManager.listDeletableLedgers(HerdDBMetadataStorageManager.java:306)
        at org.blobit.core.cluster.ClusterObjectManager.gcBucket(ClusterObjectManager.java:430)
        at org.blobit.core.cluster.ClusterObjectManager.access$000(ClusterObjectManager.java:64)
        at org.blobit.core.cluster.ClusterObjectManager$BucketHandleImpl.gc(ClusterObjectManager.java:96)
        at magnews.gateway.store.blobit.BlobitMessageStore.endQueueCleanups(BlobitMessageStore.java:199)
        at magnews.gateway.archiver.Archiver$QueueArchiverTask.run(Archiver.java:283)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        ... 3 more
Caused by: java.lang.Exception: herddb.network.netty.NettyChannel$3@5bdb86aa: error while sending reply message to PooledUnsafeDirectByteBuf(freed)
        at herddb.network.netty.NettyChannel$3.messageSent(NettyChannel.java:208)
        at herddb.network.netty.NettyChannel$1.operationComplete(NettyChannel.java:142)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:502)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:476)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:415)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:540)
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:533)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:114)
        at io.netty.util.concurrent.PromiseCombiner.tryPromise(PromiseCombiner.java:170)
        at io.netty.util.concurrent.PromiseCombiner.finish(PromiseCombiner.java:159)
        at io.netty.handler.codec.MessageToMessageEncoder.writePromiseCombiner(MessageToMessageEncoder.java:139)
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:119)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:708)
        at io.netty.channel.AbstractChannelHandlerContext.access$1700(AbstractChannelHandlerContext.java:56)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1102)
        at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1149)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1073)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:338)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        ... 1 more
Caused by: io.netty.channel.ExtendedClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)

.

Enable BookieID and upgrade BookKeeper to 4.12+

We need to:

Upgrade to Apache BookKeeper 4.14.3
Fix code useing BookieId instead of BookieSocketAddress
Add new configuration parameter server.bookkeeper.bookieid.enabled, disabled by default in "embedded" case

See https://github.com/diennea/herddb/pull/733/files as reference

Concurrent writers automatically closed without being used

When using 10 or more concurrent writers 2 or more writers get closed just after one insertion

BlobIt filesystem

We can create a layer over BlobIt in order to create a virtual file system, with these features:

a FS-tree-like structure (introduce the concept of "directory")
-- directory listing
-- create directory
-- remove directory
information about file size, disk size (virtual and real), possibly per directory (without needing a full scan)
fileExists
getFileInfo (type, size, lastmodification)
setLastModified
renameFile into same directory
renameFile into other directory (move)
delete file
read file (partial reads, offset+length)
overwrite file
truncate file to length (pre-allocate??)
write a file (full write of the whole blob)
streaming write without knowing the length in advance (not yet supported by BlobIt)
streaming write in "append mode" of a byte[] (something like 'concat', not yet supported by BlobIt)
-- streaming write in "append mode" without knowing the length in advance (not yet supported by BlobIt)

We expect that one Filesystem resides on a single Bucket.
We can add one or more metadata tables in order to store directory metadata, this will ease lookups and queries

No need for ACLs, file locks or other POSIX features

Support write with offset and length

Support a method on ObjectManager to write only a portion of a given byte array:

On ObjectManager:

public Future<String> put(String bucketId, byte[] data, int offset, int len);

Add an API to list named Blobs

It would be very useful an API to list named blobs, that is blobs with a custom name.

This API will use HerdDB SQL queries, so we can have an SQL-LIKE syntax.

Bucket.listObject(ObjectFilter filter, Consumer<ObjectMetadata> sink)

with filters built like ObjectFilter.nameLike("%myname%")

We will use this API on the CLI

License

Add license reference in pom like

  <licenses>
    <license>
      <name>Apache License, Version 2.0</name>
      <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
      <distribution>repo</distribution>
    </license>
  </licenses>

BloBIT Can be used in Linux OS and Any CLI is available to use it?

Hello,
Im looking to use this application to upload and download packages as part of an application development.
Is this application having API's?
Is this application having CLI? if yes, please provide commands to install and use it
Is this application can be installed and run in Linux OS (Ubuntu,CentOS)?

It would be helpful, If the above information is known.

Thanks

Cannot perform release on 0.3 due to error in NamesAPITest.java:[150,30] no suitable method found for equals(byte[],int,int,byte[],int,int)

INFO] [ERROR] /Users/enrico.olivelli/dev/blobit/target/checkout/blobit-core/src/test/java/org/blobit/core/common/NamesAPITest.java:[150,30] no suitable method found for equals(byte[],int,int,byte[],int,int)
[INFO] method java.lang.Object.equals(java.lang.Object) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(long[],long[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(int[],int[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(short[],short[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(char[],char[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(byte[],byte[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(boolean[],boolean[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(double[],double[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(float[],float[]) is not applicable
[INFO] (actual and formal argument lists differ in length)
[INFO] method java.util.Arrays.equals(java.lang.Object[],java.lang.Object[]) is not applicable
[INFO] (actual and formal argument lists differ in length)

BKLedgerClosedException: Attempt to write to a closed ledger

In our blobit integration (In EmailSuccess.com), we encountered the exception "Attempt to write to a closed ledger" for a few seconds.
Can you help us fix this problem?

Caused by: org.blobit.core.api.ObjectManagerRuntimeException: org.blobit.core.api.ObjectManagerException: org.apache.bookkeeper.client.BKException$BKLedgerClosedException: Attempt to write to a closed ledger
        at org.blobit.core.cluster.BucketWriter.lambda$writeBlob$0(BucketWriter.java:237)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
        ... 3 more
Caused by: org.blobit.core.api.ObjectManagerException: org.apache.bookkeeper.client.BKException$BKLedgerClosedException: Attempt to write to a closed ledger
        ... 7 more
Caused by: org.apache.bookkeeper.client.BKException$BKLedgerClosedException: Attempt to write to a closed ledger
        at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83)
        at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251)
        at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91)
        at org.apache.bookkeeper.client.LedgerHandleAdv$1.safeRun(LedgerHandleAdv.java:249)
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        ... 1 more

CLI Command to describe a blob

It would be useful for testing a new CLI command that given a BLOB it describes it:

Basic info (size, number of entries, entry size...)
BK location info (which bookies are storing the blob segments??)