Comments (10)
FWIW the rational behind that was that the application should be entirely responsible for what is contained in the snapshot
data file, and possibly come up with its own way to ensure data integrity. YMMV
from raft.
Sometimes we receive issues where one of the snapshot files seems to cause issues, and it's difficult to pinpoint where it went wrong. If we know if what we read from disk is the same as what we received from the application it could already eliminate a potential source of error. That was a bit my reasoning, don't know if you think it makes sense.
from raft.
Do we have more specific data about the issues we saw? Like a copy of the data
directory of the broken deployment (or at least of the snapshot file)
from raft.
Yes, we have one. Not sure if I can share though (if you're interested), will check.
from raft.
It's okay not to share, I wouldn't have time to check it, just wondering if you have more details about what's wrong with the file, e.g. if you used a debugger to understand where SQLite fails when opening it (assuming that's what's happening).
from raft.
A dump of the db reveals errors like this:
sqlite> SELECT id, instance_id, name, creation_date, stateful, description, expiry_date FROM instances_snapshots WHERE instance_id = 14 AND id > 2360 AND id < 2600;
2368|14|snapshot-65|2021-04-17 19:00:08.875615999+00:00|0||2021-05-17 12:00:08.874972385-07:00
2380|14|snapshot-66|2021-04-18 01:00:17.732227872+00:00|0||2021-05-17 18:00:17.731749164-07:00
Error: database disk image is malformed
and
sqlite> pragma integrity_check;
*** in database main ***
On tree page 126 cell 65: Rowid 2359 out of order
On tree page 43 cell 106: Rowid 71835 out of order
On tree page 43 cell 97: Rowid 71291 out of order
On tree page 43 cell 91: Rowid 70960 out of order
On tree page 43 cell 90: Rowid 70911 out of order
On tree page 43 cell 79: Rowid 70272 out of order
On tree page 43 cell 76: Rowid 70097 out of order
On tree page 43 cell 51: Rowid 63037 out of order
On tree page 43 cell 50: Rowid 62740 out of order
On tree page 354 cell 51: Rowid 68748 out of order
row 66 missing from index sqlite_autoindex_instances_snapshots_1
wrong # of entries in index sqlite_autoindex_instances_snapshots_1
row 66 missing from index sqlite_autoindex_storage_volumes_snapshots_2
row 66 missing from index sqlite_autoindex_storage_volumes_snapshots_1
row 93 missing from index sqlite_autoindex_storage_volumes_snapshots_2
--- continues ----
row 241 missing from index sqlite_autoindex_storage_volumes_snapshots_2
row 243 missing from index sqlite_autoindex_storage_volumes_snapshots_2
row 248 missing from index sqlite_autoindex_storage_volumes_snapshots_2
from raft.
This is when trying to load the latest snapshot, the oldest snapshot is fine and is also +- 0.7MB bigger (5.5MB vs 4.8MB, not necessarily relevant info).
from raft.
Interesting that a couple of rows where returned, so the file is not complete garbage. I guess the database schema is not corrupted? SELECT * FROM sqlite_master WHERE type='table'
or something like that.
from raft.
Yes, that runs fine.
from raft.
Checksum has been added as part of #207.
from raft.
Related Issues (20)
- missing getrandom on centos 7 HOT 1
- [question] Usage of raft_add, raft_remove, and raft_assign HOT 2
- [question] When is the first leader elected? HOT 2
- recvAppendEntries: Assertion `r->state == RAFT_FOLLOWER || r->state == RAFT_CANDIDATE` failed HOT 9
- [question] Forwarding request to leader HOT 8
- Potential use-after-free in handling of raft_transfer HOT 5
- Fix the 32-bit CI HOT 1
- can't build on m1 mac (`<linux/xxxxx.h>` is missing) HOT 3
- v1.x RFC: pull based approach HOT 5
- raft_start(): io: closed segment xxx is past last snapshot xxx HOT 4
- [question] Understanding how to add new servers HOT 15
- ./configure fails when no external dependency is found HOT 4
- Assertion: src/uv_truncate.c:168: UvTruncate: Assertion `index < uv->append_next_index' failed. HOT 3
- src/replication.c:457: getRequest: Assertion `req->type == type' failed. HOT 3
- Segment writes blocked when taking a snapshot. HOT 3
- src/log.c:87: refsTryInsert: Assertion `next_slot->term != term' failed. HOT 9
- Jepsen: Another truncate-related assertion failure
- [suggestion] add lock/unlock function pointers to raft_io (and use them) HOT 1
- `uvOsFallocateEmulation` taking too long time HOT 1
- CI: `xfs` test failures HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from raft.