Comments (7)
Hello @superkaiy, thank you for reporting this issue. You are correct; a new version of LDB will be released soon, and we will take your suggestion into consideration, along with many other improvements and new features. However, a missing sector is not expected behavior, so I recommend that you double-check the consistency of your knowledge base. Please don't hesitate to ask any other questions or make any additional requests.
from ldb.
@mscasso-scanoss Thanks for your replication. I double-confirmed about my KB
- tide up my knowledge base, and tide up the storage
- do mining process again
then I scanned my local target folder on the first time, it will throw the similar exceptions as following:
return an invalid JSON (JSONDecodeError - xxx): Expecting ',' delimiter: line 1 column 8439 (char 8438)
but when I tried to scan the folder at the second time, it became normal without no exceptions.
There is some loss data in the first output data, and the second output data is completed.
Could you help to give some suggestions for me to follow up the exceptions?
1st trial run:
tide up KB and do mining
2nd trial run:
tide up KB and do mining
1st scanning and 2nd scanning
And I found another confused things during scanning process, the table url data will increase during scanning process.
before scanning process, the data in url table is as following:
but after scanning process, the data in url table increased as following:
And the data in table oss/url:
from ldb.
@mscasso-scanoss , I found the cause why the table url data will increase during scanning process. We fetch the data via ldb_fetch_recordset function, and ldb sector is opend with mode 'r+', it will try to new a sector or a file when it does not exist
from ldb.
@superkaiy Hi, may I ask how did you tide up your KB when you have encountered a "cannot read LDB sector" error?
from ldb.
@DRong1121 For it's just on the phase of trial run, I delete whole database via command rm -rf <ldb root>/<database name>
.
I guess it is necessary to build up a replication mechanism or version control management on the KB
from ldb.
Hello @superkaiy, thank you very much for your feedback. I wanted to let you know that the release of ldb will be published soon, and your suggestions will be taken into consideration. Please feel free to test it out and take a look at the latest changes on this branch: https://github.com/scanoss/ldb/tree/ldb_upgrade.
from ldb.
@superkaiy Hi, may I ask how did you tide up your KB when you have encountered a "cannot read LDB sector" error?
I meet a "cannot read LDB sector" error only during scanning process, and I found a weird thing that the some ldb sector will be re-written during the scanning process. I guess the exception is related to the re-written operations for there is no exception during mining process. so maybe change the mode "r+" to "r" in ldb_open function will be helpful to you.
from ldb.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ldb.