Comments (4)
vrtmrz
commented on July 17, 2024
2
Thank you for your report!
This problem has been caused by here: (new:2, recycled:2607, cached:46)
To less traffic, Self-hosted LiveSync splits the documents into multiple chunks and transfers only new things. However, checking existent from the local database is a bit heavy if there are a lot of them. And worse, this slows down according to their number.
To avoid this, Self-hosted LiveSync can cache recently used chunks. The initial value is configured in a little conservative value, but we should enlarge this in such a situation.
Please try setting Memory cache size (by total items) to 300000, and Memory cache size (by total characters) to 100 once. (Yes, this can be controlled by the count and total amount, for such a case).
I think that your environment could accept these values.
from obsidian-livesync.
firstunicorn
commented on July 17, 2024
Thanks for the reply! It seems it does help a bit, now it's like 10 seconds for one direction instead of 10-15 (or it's just a placebo? lol).
Also, I did experiments with caching in the past and it feels like it doesn't make any difference with any combination of parameters... the only thing that really worked at first is switching to "Periodic and on Events" for every 4-second interval sync mode with batch and now what helps a bit is only removing CouchDB on a hoster side and create it again with discarding local database, but only for short period of time.
Also, I noticed that right now I have 32000 chunks (or docs?)... and if I delete DB and reset everything, it shows just 6844 docs, which still seems a lot considering I have only 250 files (if this will help - a total size of the Vault is about 100MB with all attachments and images)... Maybe I'm mixing some terms about docs/chunks, so here the some logs after deleting and resetting everything:
Logs
5/2/2024, 9:56:44 PM->Cache initialized 300 / 250000000000
5/2/2024, 9:56:44 PM->loading plugin
5/2/2024, 9:56:44 PM->Self-hosted LiveSync v0.23.3 0.23.3
5/2/2024, 9:56:44 PM->xxhash for plugin initialised
5/2/2024, 9:56:44 PM->Waiting for ready...
5/2/2024, 9:56:44 PM->Cache initialized 10 / 1000000000
5/2/2024, 9:56:44 PM->Cache initialized 300000 / 100000000
5/2/2024, 9:56:44 PM->Newer xxhash has been initialised
5/2/2024, 9:56:44 PM->Opening Database...
5/2/2024, 9:56:44 PM->Database info
5/2/2024, 9:56:44 PM->{
"doc_count": 6844,
"update_seq": 6844,
"db_name": "Obsidian Vault-anon-livesync-v2-indexeddb",
"auto_compaction": false,
"adapter": "indexeddb"
}
5/2/2024, 9:56:44 PM->Database is now ready.
5/2/2024, 9:56:45 PM->Log window opened
5/2/2024, 9:56:46 PM->Initialize and checking database files
5/2/2024, 9:56:46 PM->Checking deleted files
5/2/2024, 9:56:46 PM->Checking expired file history
5/2/2024, 9:56:52 PM->There are no old documents
5/2/2024, 9:56:52 PM->Checking expired file history done
5/2/2024, 9:56:52 PM->Collecting local files on the storage
5/2/2024, 9:56:52 PM->Collecting local files on the DB
5/2/2024, 9:56:52 PM->Collecting local files on the DB: 25
5/2/2024, 9:56:52 PM->Collecting local files on the DB: 50
5/2/2024, 9:56:52 PM->Collecting local files on the DB: 75
5/2/2024, 9:56:52 PM->Collecting local files on the DB: 100
5/2/2024, 9:56:53 PM->Collecting local files on the DB: 125
5/2/2024, 9:56:53 PM->Collecting local files on the DB: 150
5/2/2024, 9:56:53 PM->Collecting local files on the DB: 175
5/2/2024, 9:56:53 PM->Collecting local files on the DB: 200
5/2/2024, 9:56:54 PM->Collecting local files on the DB: 225
5/2/2024, 9:56:54 PM->Collecting local files on the DB: 250
5/2/2024, 9:56:54 PM->Opening the key-value database
5/2/2024, 9:56:54 PM->Updating database by new files
5/2/2024, 9:56:54 PM->UPDATE DATABASE: Nothing to do
5/2/2024, 9:56:54 PM->UPDATE STORAGE: Nothing to do
5/2/2024, 9:56:57 PM->Initialized, NOW TRACKING!
5/2/2024, 9:56:57 PM->Cache initialized 300000 / 100000000
5/2/2024, 9:56:57 PM->Modifying callback of the save command
5/2/2024, 9:56:57 PM->Additional safety scan..
5/2/2024, 9:56:57 PM->There are no conflicted files
5/2/2024, 9:56:57 PM->Additional safety scan done
5/2/2024, 9:57:19 PM->OneShot Sync begin... (sync)
5/2/2024, 9:57:20 PM->Looking for the point last synchronized point.
5/2/2024, 9:57:21 PM->Replication paused
5/2/2024, 9:57:22 PM->Replication activated
5/2/2024, 9:57:26 PM->↑0 ↓2 (6446)
5/2/2024, 9:57:27 PM->The request may have failed. The reason sent by the server: 404: Object Not Found
5/2/2024, 9:57:27 PM->{"error":"not_found","reason":"missing"}
5/2/2024, 9:58:25 PM->Replication closed
In summary, the less number of chunks/docs - the faster everything works, especially for large .MD files. Also, right now, after some experiments, I noticed that it syncs much faster on manual syncing with "Disable all automatic". So weird...
from obsidian-livesync.
firstunicorn
commented on July 17, 2024
Also, after increasing the size of the cache I noticed there is less stuck with the syncing process, which happens sometimes; but it doesn't seem to really affect the speed of syncing (in case when it is not stuck, of course).
from obsidian-livesync.
Related Issues (20)
- Reducing Time for _changes Request in CouchDB HOT 1
- Run replicate now automatically? HOT 2
- Renaming or deleting notes cause sync to be stuck, showing ⏳ Working read storage processes and/or 🛫 Pending read storage processes HOT 6
- Notes with quoted strings in their name are synced as if they are deleted HOT 2
- FileWatching STORAGE -> DB (plain) updates lead to lag on mobile devices like iPad HOT 4
- Default ignore pattern settings for hidden files sync doesn't work HOT 2
- 能否增加中文支持 HOT 6
- no sync when couchdb sits behind a /location
- Remote database is 7 times larger than the vault HOT 1
- Some hidden files fail to sync and may cause sync paused HOT 4
- [Feature request] Can we customize the HTTP request header?
- CORS errors in remote database connection HOT 1
- Click on the upper right corner but cannot close HOT 1
- Live sync won't start HOT 27
- The request may have failed. The reason sent by the server: 404: HOT 2
- Ask for "Reload Obsidian" when it is safe to do HOT 5
- Feature Request: Canarie and LTS versions of the plugin HOT 2
- English translation and explanation of settings HOT 1
- Remote Configuration Error HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from obsidian-livesync.