bobris / btdb Goto Github PK

Is it expected to improve. Because in fact, under big data, loading all keys will make the usage scenario very limited.

I am currently looking for a KV Store. It is precisely because of the limited memory and hard disk space that 20 billion keys need to be stored, but the value is not large, and the value does not need to be indexed.

my key about MaxLength 50 characters string (dynamic length).
and the value MaxLength about 50 bytes.

The production environment does not have huge memory. It is possible that the data storage area can only use memory below 512mb.

There seems to be a leak of ObjectTypeDescriptor instances in the EventStoreManager, please see #32 for a repro test.

Currently it seems useful to me to have property with both primary and secondary key, but throws Secondary key creating error. Please verify following if this is something that should work.

public class UserNotice2
{
    [PrimaryKey(1)]
    public ulong UserId { get; set; }
    [PrimaryKey(2)]
    [SecondaryKey("NoticeId")]
    public ulong NoticeId { get; set; }
}

public interface IUserNotice2Table
{
    void Insert(UserNotice2 un);
    IEnumerator<UserNotice2> ListByNoticeId(AdvancedEnumeratorParam<ulong> noticeId);
}

[Fact]
public void UserNotice2Works()
{
    using (var tr = _db.StartTransaction())
    {
        var creator = tr.InitRelation<IUserNotice2Table>("UserNotice2");
        var table = creator(tr);
        table.Insert(new UserNotice2 { UserId = 1, NoticeId = 2 });
        var en = table.ListByNoticeId(new AdvancedEnumeratorParam<ulong>(EnumerationOrder.Ascending, 2, KeyProposition.Included, 3, KeyProposition.Excluded));
        Assert.True(en.MoveNext());
        Assert.Equal(2u, en.Current.NoticeId); // throws Secondary key creating error
        tr.Commit();
    }
    ReopenDb();
    var db = (ObjectDB)_db;
    using (var tr = _db.StartTransaction())
    {
        var relationInfo = db.RelationsInfo.CreateByName((IInternalObjectDBTransaction)tr, "UserNotice2", typeof(IUserNotice2Table));
        Assert.Equal(1u, relationInfo.ClientTypeVersion);
    }
}

Is it possible to change internal behaviour of the EnsureClientTypeNotNull method? In the way that it throws different type of exception so we can easily filter it out in the Visual Studio because it is not really an exception at least in one specific scenario. In the EnumerateSingletonTypes there is an empty catch block with comment "Ignore imposibility to create type". In the other cases it is really an exception though.

Reason:

I would like to know when there is an exception thrown but not in this case.

Workaround:

Edit condition in the Exception Settings. I did not really test it - just found it, however, it would be still quite annoying to set this in every solution.

Create Massive test suite

Does BTDB compile and run in a UWP App?

Sometimes when .NET 4.5 is mentioned as the target, a project will work on UWP and sometimes not.

So it would be nice if you could add a note to the main page one way or the other.

If the answer is no, then perhaps re-title this issue as "UWP Porting"

Implement Range key counting

3.5.0.0 version failed to load data stored by 3.4.0.0 version

when loading object

    [StoredInline]
    public class NetServerPackageItem
    {
        public string SourceFile { get; set; }
        public string Version { get; set; }
        public string Comment { get; set; }
        public BlobLocation BlobLocation { get; set; }
        public DateTime UploadStartTime { get; set; }
        public DateTime UploadFinishedTime { get; set; }
        public ulong UserId { get; set; }
        public string UserName { get; set; }
        public double Progress { get; set; }
        public bool Finished { get; set; }
        public bool Preferred { get; set; }
    }

    [StoredInline]
    public class BlobLocation
    {
        public string Prefix { get; set; }
        public string Name { get; set; }
        public ulong EncryptKeyId { get; set; }
    }

ReadNativeObject() (in bold below) gets parent object from cache (as index 0) - because it has different DB context object cache than during save? And then the byte content of BlobLocation is not readed and when interpreting as consecutive DateTime fields fails

static void Loader_NetServerPackageItem_1(
BTDB.ODBLayer.IInternalObjectDBTransaction arg0,
BTDB.ODBLayer.DBObjectMetadata arg1,
BTDB.StreamLayer.AbstractBufferedReader arg2,
object arg3)
{
Gmc.Inachis.InspireProcessing.NetServerPackageDeployment.Db.NetServerPackageItem local0 // index 0
ldarg.3
castclass Gmc.Inachis.InspireProcessing.NetServerPackageDeployment.Db.NetServerPackageItem
stloc.0
BTDB.FieldHandler.IReaderCtx local1 // index 1
ldarg.0
ldarg.2
callvirt BTDB.FieldHandler.IReaderCtx GetReaderCtx(BTDB.StreamLayer.AbstractBufferedReader reader)
stloc.1
ldloc.0
ldarg.2
call string ReadString()
call void set_SourceFile(string value)
ldloc.0
ldarg.2
call string ReadString()
call void set_Version(string value)
ldloc.0
ldarg.2
call string ReadString()
call void set_Comment(string value)
ldloc.0
ldloc.1
callvirt object ReadNativeObject()
isinst Gmc.Cloud.Infrastructure.Host.BlobLocation //is not set - it is NetServerPackageItem
call void set_BlobLocation(Gmc.Cloud.Infrastructure.Host.BlobLocation value)
ldloc.0
ldarg.2
call System.DateTime ReadDateTime()
call void set_UploadStartTime(System.DateTime value)
ldloc.0
ldarg.2
call System.DateTime ReadDateTime()
call void set_UploadFinishedTime(System.DateTime value)

If you store an event with a dictionary property containing recursive value type (applies to transitive relationships as well), an exception will be thrown during event deserialization.

Repro example:

class ObjectValidatedEvent 
{
  public IList<ErrorInfo> Errors { get; set }
}

class ErrorInfo
{
  public IDictionary<string, IList<ErrorInfo>> PropertyErrors { get; set; }
}

Known workaround:

class ErrorInfo : ErrorInfoBase
{
  public IDictionary<string, IList<ErrorInfoBase>> PropertyErrors { get; set; }
}

Implement Range key erase

When I accidently call transation.Store() on KeyValuePair, instead of on his value. I get Verification Exception, operation could destabilize runtime. It's difficult to find the right spot where tr.Store on struct is called.

When I use 3 primary keys and relation ListById() it throws EndOfStreamException.
Please see following case:

public class UserNotice
{
    [PrimaryKey(1)]
    public ulong CompanyId { get; set; }
    [PrimaryKey(2)]
    public ulong UserId { get; set; }
    [PrimaryKey(3)]
    public ulong NoticeId { get; set; }
}

public interface IUserNoticeTable
{
    void Insert(UserNotice un);
    IEnumerator<UserNotice> ListById(ulong companyId, ulong userId, AdvancedEnumeratorParam<ulong> param);
}

[Fact]
public void UserNoticeWorks()
{
    using (var tr = _db.StartTransaction())
    {
        var creator = tr.InitRelation<IUserNoticeTable>("UserNotice");
        var userNoticeTable = creator(tr);
        userNoticeTable.Insert(new UserNotice { CompanyId = 10, UserId = 20, NoticeId = 30 });
        userNoticeTable.Insert(new UserNotice { CompanyId = 10, UserId = 20, NoticeId = 31 });
        userNoticeTable.Insert(new UserNotice { CompanyId = 10, UserId = 20, NoticeId = 32 });
        var un = userNoticeTable.ListById(10, 20, new AdvancedEnumeratorParam<ulong>(EnumerationOrder.Ascending));
        Assert.True(un.MoveNext());
        Assert.Equal(30u, un.Current.NoticeId); // EndOfStreamException
        // ...
    }
}

It seems to me that this should pass.

When I accidently call transation.Store() on KeyValuePair, instead of on his value. I get Verification Exception, operation could destabilize runtime. It's difficult to find the right spot where tr.Store on struct is called.

Implement Key erase

Hello. I've stumbled on your DB while researching LMDB usage from .NET and have a couple of questions:

1. LMDB has very good recovery story. If it crashes before write transaction is finished the DB remains in consistent state. How about BTDB? How is the data structured in the DB and how does it behave under crash conditions? Is limitation, similar to LMDB, just a single writer too?
2. Have you compared BTDB speed with LMDB?

Add Multithreaded tests

Limit Sector cache size and start deallocating long unused sector

Allocation tree more levels (now just 1) 8MB max

See PR #73 for a repro case.

I have UT case as below, and it throw exception.
Workaround is use EnumerationOrder.Descending, or evaluate the listing result before deleting

      public class Room
        {
            [PrimaryKey(1)]
            [SecondaryKey("CompanyId")]
            public ulong CompanyId { get; set; }

            [PrimaryKey(2)]
            public ulong Id { get; set; }

            public string Name { get; set; }
        }
      public interface IRoomTable
        {
            ulong CompanyId { get; set; }
            void Insert(Room room);
            IEnumerator<Room> ListById(AdvancedEnumeratorParam<ulong> param);
            void RemoveById(ulong id);
        }


       [Fact]
        public void EnumerateAndDeleteTest()
        {
            using (var tr = _db.StartTransaction())
            {
                var creator = tr.InitRelation<IRoomTable>("Room");

                var rooms = creator(tr);
                rooms.CompanyId = 1;
                rooms.Insert(new Room { Id = 10, Name = "First 1" });
                rooms.Insert(new Room { Id = 20, Name = "Second 1" });

                var oen = rooms.ListById(new AdvancedEnumeratorParam<ulong>(EnumerationOrder.Ascending));
                while (oen.MoveNext())
                {
                    rooms.RemoveById(oen.Current.Id);
                }
            }
        }

I've used BTDB in two projects already. I had experienced some problems, but after sending log of commands to the author, they were solved next day! Also like the speed of BTDB, which is several times faster than relational databases on the same data :-)