Huge data folder size overhead in multiple overwrite workload about dbreeze HOT 7 CLOSED

Hi.
Repack your data into an empty table to get initial size (select from one table and insert into another, use byte[] level for copying) - it will be your "vacuum" and will restore the table size.
Then try to disable "Technical_SetTable_OverwriteIsNotAllowed" (just don't call it) and sort your batch ascending by key in memory before insert (you can look integrated sorter - tran.RandomKeySorter - read docu).
If the speed of the batch insert is not satisfactory, then try to use "Technical_SetTable_OverwriteIsNotAllowed" after batch is sorted in memory.
If size will be again too big, then algorithm must be rethinked.

from dbreeze.

Hi,
thanks for quick response!

A couple of notes:

• Technical_SetTable_OverwriteIsNotAllowed we do not use from the beginning (actually I've read about it today when digging doc, so this option was always in its default state).
• Keys are always in ascending order. Actually, we use DBreeze as complimentary DB engine to SQL Server, so our keys are values from ID column from corresponding SQL Table and have identity / auto-increment field type (1,1) (so grow always ascending).

Can you slightly explain (or point to the source code or doc) how engine decide when to overwrite or remove/insert instead and how removing implemented if there are other records following deleted row exists (update in the middle)?

from dbreeze.

Am I understand correctly that there are no any additional actions for removing in middle (due to performance reasons), so if we need such behavior we have to implement manually, i.e. read tail, remove it and store update with unmodified parts? Similar to the logic of RandomKeySorter as you pointed.

from dbreeze.

May be you can try to write a small program emulating your behavior of the insert/insert/update logic, pointing the performance or size problem. For me is very difficult to talk without such examples.
If you want we can go on discussion on our native language.

from dbreeze.

Thank you for the answer!

I dig source code a bit and can say that I'm probably right in my assumptions. I think we can close for now. We rethink our save logic and if the issue arises again then we can reopen it.

from dbreeze.

Remove in DBreeze is only a "logical" operation, data stays physically inside the file, only search keys are rewritten.

from dbreeze.

Thanks for clarification.

from dbreeze.