when detect folder based on local KB, the client will raise bad_json-xxxx.txt about ldb HOT 7 CLOSED

Hello @superkaiy, thank you for reporting this issue. You are correct; a new version of LDB will be released soon, and we will take your suggestion into consideration, along with many other improvements and new features. However, a missing sector is not expected behavior, so I recommend that you double-check the consistency of your knowledge base. Please don't hesitate to ask any other questions or make any additional requests.

from ldb.

@mscasso-scanoss Thanks for your replication. I double-confirmed about my KB

1. tide up my knowledge base, and tide up the storage
2. do mining process again
   then I scanned my local target folder on the first time, it will throw the similar exceptions as following:
   return an invalid JSON (JSONDecodeError - xxx): Expecting ',' delimiter: line 1 column 8439 (char 8438)
   but when I tried to scan the folder at the second time, it became normal without no exceptions.
   There is some loss data in the first output data, and the second output data is completed.
   Could you help to give some suggestions for me to follow up the exceptions?

1st trial run:
tide up KB and do mining

1. 1st scanning
   

2. 2nd scanning
   

2nd trial run:
tide up KB and do mining
1st scanning and 2nd scanning

And I found another confused things during scanning process, the table url data will increase during scanning process.
before scanning process, the data in url table is as following:

but after scanning process, the data in url table increased as following:


And the data in table oss/url:

from ldb.

@mscasso-scanoss , I found the cause why the table url data will increase during scanning process. We fetch the data via ldb_fetch_recordset function, and ldb sector is opend with mode 'r+', it will try to new a sector or a file when it does not exist

from ldb.

@superkaiy Hi, may I ask how did you tide up your KB when you have encountered a "cannot read LDB sector" error?

from ldb.

@DRong1121 For it's just on the phase of trial run, I delete whole database via command rm -rf <ldb root>/<database name>.
I guess it is necessary to build up a replication mechanism or version control management on the KB

from ldb.

Hello @superkaiy, thank you very much for your feedback. I wanted to let you know that the release of ldb will be published soon, and your suggestions will be taken into consideration. Please feel free to test it out and take a look at the latest changes on this branch: https://github.com/scanoss/ldb/tree/ldb_upgrade.

from ldb.

@superkaiy Hi, may I ask how did you tide up your KB when you have encountered a "cannot read LDB sector" error?

I meet a "cannot read LDB sector" error only during scanning process, and I found a weird thing that the some ldb sector will be re-written during the scanning process. I guess the exception is related to the re-written operations for there is no exception during mining process. so maybe change the mode "r+" to "r" in ldb_open function will be helpful to you.

from ldb.