Abstract Sync occasionally seems to hang on one or more devices, b

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Sync occasionally hangs, and takes a long time to recover,about vrtmrz/obsidian-livesync

Comments (16)

vrtmrz commented on July 17, 2024 1

Sorry for being late! And, am also sorry for the misdirected guidance. In retrospect, it seems that I did not have a deep understanding of what caused the situation.

At v0.22.7, I have fixed some logic to fetch chunks from the remote.
However, the fix contained a misdesign. Finally, it became stable at v0.22.9.

Since v0.22.8, our LiveSync and CouchDB handle chunks a bit efficiently. And the inefficient chunks are deleted on rebuilding. Hence, rebuilding also contributes to improving the situation. However, that version still has some bugs. Please upgrade to v0.22.9.

I am so relieved to hear that your environments have gotten better again! I hope that this explanation makes sense and that you will allow me to close the Issue.

from obsidian-livesync.

vrtmrz commented on July 17, 2024

Thank you for notifying this issue!
This may caused by Fetch chunks on demand. This feature requires serverside work.
When we enable this feature, we synchronise only notes and then fetch the chunks which are actually needed from the note.
This only notes needs some effort.

Disabling Fetch chunks on demand may prevent the synchronisation hang-ups. However, a complete solution would be desirable.

from obsidian-livesync.

zedtools commented on July 17, 2024

I played around with it but it appears that disabling Fetch chunks on demand did not help that much. I disabled this setting, then restarted both the livesync server and the Obsidian client.

It also looks like restarting the server triggered the synchronisation hang-up.

After restarting, I made a change to a small test file (Test.md, 10 bytes long), and I noticed it was stuck again. Interestingly, I noticed the client was using a lot of CPU while it was stuck like this – this is an 11th gen i7:

It was not like this the entire time – the CPU usage went up and down a bit on both the client and the server.

This time it took about 7 minutes for the first sync to happen on my PC, after which it was fine. My other devices took a similar time (5-7 minutes) for their first sync.

So it seems to take about half the time compared to before, but the first sync is still pretty slow. Anything else I can try?

from obsidian-livesync.

vrtmrz commented on July 17, 2024

Thank you for the detail!
This suggests that Self-hosted LiveSync missed the checkpoint and had traced the changes from the beginning of the remote database (Only read and check, no items are transferred. Therefore we would not observe any from the UI).
At v0.22.6, both checkpoints which are local and remote are used. This prevents missing checkpoints.

With this fix, the first synchronisation after the update will take a little longer (about the time reported in this issue, if assumptions are correct). However, it will stabilise afterwards.

Would you mind if I ask you to check the behaviour and issue have been fixed, please?

from obsidian-livesync.

zedtools commented on July 17, 2024

Thanks for the update.

I restarted the couchdb server a couple of times today and Livesync did not hang, so I guess that is not a reliable way of reproducing the issue.

I did notice that I was running 0.20.7 of the plugin (I just realised Obsidian does not auto update plugins), so I updated it manually. Both my Windows PC and Mac hung with the sync after the update, but recovered as you said it would.

Once all my devices recover, I will monitor this to see if it happens again.

from obsidian-livesync.

JamborJan commented on July 17, 2024

Hi @vrtmrz ,

I have a similar issue. Syncing is only happening once in a while and devices need a huge amount of time to catch up. I have installed the latest version of your plugin today. Can you tell me which logs and information are useful for you? I would like to help you as much as possible to ensure you can track down the issue. I really like your extension and rely on it quite a lot. Thank you.

PS: on desktop I often see stuff like this:

Does it mean sync is somehow paused? While I see this and clearly have a diff between two devices, the log says:

22/02/2024, 00:05:25->Replication completed
22/02/2024, 00:05:32->OneShot Sync begin... (sync)
22/02/2024, 00:05:32->Replication completed
22/02/2024, 00:05:59->OneShot Sync begin... (sync)
22/02/2024, 00:05:59->Replication completed

Even after the sync catches up once in a while, it takes a long time afterwards that changes are synced. A minute or multiple minutes. My CoucDB is 1.2 GB and has 31'548 documents. The LXC running the CouchDB server is mostly bored. Seldom the CPu shortly spikes to 50%.

from obsidian-livesync.

vrtmrz commented on July 17, 2024

@JamborJan
Thank you for updating the plug-in! Looking at the information you provided, the synchronisation indicates it has balanced, as you have pointed out.

The next possibility might be a mismatch of the local database and storage.
We can verify files by Verify and repair all files -> Verify all button on the Hatch pane, in the settings dialogue.
When pressing that button, all files are compared with the local database, and if not matched, we can choose which one to use.

If all files are matched on all devices, there might be a mismatch between one of the local databases and the remote database. In this case, we should rebuild the database with the device which has the most certain files.

from obsidian-livesync.

JamborJan commented on July 17, 2024

I did the Verify and repair step and the result is still that changes are not synced in a useful amount of time. Specifically: a change is still not replicated after a lot of minutes. Am I getting you right, that I should click the button for triggering the Overwrite remote database action?

from obsidian-livesync.

vrtmrz commented on July 17, 2024

Thank you for your try!
If no mismatch has been found on every (it is possible that if a device has the latest file, it is not reflected in the local database) device, we have to rebuild the database.

Rebuilding can performed by Rebuild button.

Thank you for your patience and cooperation!

from obsidian-livesync.

JamborJan commented on July 17, 2024

It's still not better. My expectation would be that a change in a file would be replicated to a secondary device in a couple of seconds. I mostly work on my desktop but need to add notes or review something on my mobile phone. I open the app on the phone and nothing is happening. Both devices think they are in sync but they are clearly not. When I keep the mobile phone app open after a long time (hours maybe) something is happening.

Can I provide any logs or more structured details that would help analyze the problem?

from obsidian-livesync.

JamborJan commented on July 17, 2024

FYI. I had to do the follwoing:

Decoupled all devices and ensured one is the primary device having all latest data
Updated all devices to the latest LiveSync version 0.22.8 (I was already one hotfix version late)
Deleted CouchDB
Setup CouchDB new from master device
Synced secondary device again

Now it is working again as expected.

from obsidian-livesync.

zedtools commented on July 17, 2024

I have been running with Fetch chunks on demand disabled, and have not had any issues since v0.22.6.

Should I leave this option disabled because of the large number of files in my vault? I just updated to v0.22.9 and then enabled Fetch chunks on demand, but the initial sync after this change is taking some time.

The client does not seem to be using much of the CPU, though the couchdb server is.

from obsidian-livesync.

zedtools commented on July 17, 2024

This hang happened again with the latest version (0.22.12). Some background on the context:

It has been working fine since my last comment above.
Last night (on the 14th) I noticed all my devices hung again when I opened Obsidian and it tried to sync. After about 15 minutes, they each finished syncing and worked normally.
I have been running with Fetch chunks on demand – not sure if this makes a difference.
I checked my couchdb server, and noticed it had been automatically updated the night before (on the 13th).

I have a Docker environment set up where it updates containers automatically, and it looks like it updated couchdb (from 3.3.2 to 3.3.3), and this may have caused the issue.

Is this expected behaviour?

from obsidian-livesync.

zedtools commented on July 17, 2024

I found the couchdb logs and it looks like it crashed from a SIGTERM just before the update... not sure if this explains it?

couchdb logs

2024-03-13 21:24:11.152	
[notice] 2024-03-13T10:24:11.151652Z nonode@nohost <0.1559.158> acaeaa35e9 obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/ 204 ok 1
	
	
2024-03-13 21:24:11.390	
[notice] 2024-03-13T10:24:11.390382Z nonode@nohost <0.1559.158> 29e2b32465 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/ 200 ok 33
	
	
2024-03-13 21:24:11.408	
[notice] 2024-03-13T10:24:11.405499Z nonode@nohost <0.1559.158> 485ba100c5 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/ 200 ok 4
	
	
2024-03-13 21:24:11.422	
[notice] 2024-03-13T10:24:11.422500Z nonode@nohost <0.1559.158> 8fcee976cc obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/_design/replicate? 204 ok 11
	
	
2024-03-13 21:24:11.443	
[notice] 2024-03-13T10:24:11.443011Z nonode@nohost <0.1559.158> 5bde864057 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_design/replicate? 304 ok 8
	
	
2024-03-13 21:24:11.451	
[notice] 2024-03-13T10:24:11.451054Z nonode@nohost <0.1559.158> 89b5a30cd1 obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/obsydian_livesync_version? 204 ok 1
	
	
2024-03-13 21:24:11.462	
[notice] 2024-03-13T10:24:11.462117Z nonode@nohost <0.1559.158> 16b09fef55 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/obsydian_livesync_version? 304 ok 3
	
	
2024-03-13 21:24:11.470	
[notice] 2024-03-13T10:24:11.470045Z nonode@nohost <0.1559.158> ebf7c64179 obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/_local/obsydian_livesync_milestone? 204 ok 1
	
	
2024-03-13 21:24:11.479	
[notice] 2024-03-13T10:24:11.479341Z nonode@nohost <0.1559.158> 98d2f23bf2 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_local/obsydian_livesync_milestone? 304 ok 3
	
	
2024-03-13 21:24:11.488	
[notice] 2024-03-13T10:24:11.488256Z nonode@nohost <0.1559.158> 3241f5f21c obsidian.home.com 192.168.1.254 undefined OPTIONS / 204 ok 1
	
	
2024-03-13 21:24:11.495	
[notice] 2024-03-13T10:24:11.494821Z nonode@nohost <0.1559.158> 3b4751386c obsidian.home.com 192.168.1.254 obsidianuser GET / 200 ok 2
	
	
2024-03-13 21:24:11.503	
[notice] 2024-03-13T10:24:11.503267Z nonode@nohost <0.1559.158> 3bcf5202ba obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/_local/B1x2mC53febVgPzEmEaWOw%3D%3D? 204 ok 1
	
	
2024-03-13 21:24:11.512	
[notice] 2024-03-13T10:24:11.511782Z nonode@nohost <0.1559.158> 6106d02dfb obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_local/B1x2mC53febVgPzEmEaWOw%3D%3D? 304 ok 3
	
	
2024-03-13 21:24:11.521	
[notice] 2024-03-13T10:24:11.520640Z nonode@nohost <0.1559.158> ec6a0bf8ff obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/ 200 ok 3
	
	
2024-03-13 21:24:11.530	
[notice] 2024-03-13T10:24:11.529532Z nonode@nohost <0.1559.158> 201d56f919 obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/_changes?style=all_docs&filter=replicate%2Fpull&since=108574-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYHh0odcoBi7QXKihVmyJbp6HCbksQBJhgYg9R9u0FU9sEHmiUlJSZZp6NqyAL6BLdw&limit=50 204 ok 1
	
	
2024-03-13 21:24:11.541	
[notice] 2024-03-13T10:24:11.540364Z nonode@nohost <0.1559.158> 8089dd872c obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_changes?style=all_docs&filter=replicate%2Fpull&since=108574-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYHh0odcoBi7QXKihVmyJbp6HCbksQBJhgYg9R9u0FU9sEHmiUlJSZZp6NqyAL6BLdw&limit=50 304 ok 5
	
	
2024-03-13 21:24:11.553	
[notice] 2024-03-13T10:24:11.552730Z nonode@nohost <0.1559.158> 429bdcd92c obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_local/B1x2mC53febVgPzEmEaWOw%3D%3D? 304 ok 3
	
	
2024-03-13 21:24:11.564	
[notice] 2024-03-13T10:24:11.563890Z nonode@nohost <0.1559.158> 251e9c387f obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/ 200 ok 3
	
	
2024-03-13 21:24:11.572	
[notice] 2024-03-13T10:24:11.572366Z nonode@nohost <0.1559.158> 047b9c6322 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/ 200 ok 3
	
	
2024-03-13 21:24:11.584	
[notice] 2024-03-13T10:24:11.584216Z nonode@nohost <0.1559.158> 8f6927bf6c obsidian.home.com 192.168.1.254 obsidianuser GET / 200 ok 2
	
	
2024-03-13 21:24:11.587	
[notice] 2024-03-13T10:24:11.587341Z nonode@nohost <0.1595.158> 2b0a213c90 obsidian.home.com 192.168.1.254 obsidianuser GET / 200 ok 2
	
	
2024-03-13 21:24:11.592	
[notice] 2024-03-13T10:24:11.592130Z nonode@nohost <0.1595.158> ea961396a1 obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/_local/amEhkelueWmQgxmxfdNN0g%3D%3D? 204 ok 1
	
	
2024-03-13 21:24:11.597	
[notice] 2024-03-13T10:24:11.596674Z nonode@nohost <0.1595.158> e468083c2a obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_local/B1x2mC53febVgPzEmEaWOw%3D%3D? 304 ok 3
	
	
2024-03-13 21:24:11.603	
[notice] 2024-03-13T10:24:11.603157Z nonode@nohost <0.1595.158> a37b1ba1a5 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_local/amEhkelueWmQgxmxfdNN0g%3D%3D? 304 ok 3
	
	
2024-03-13 21:24:11.606	
[notice] 2024-03-13T10:24:11.606118Z nonode@nohost <0.1559.158> 9e06ea9435 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/ 200 ok 4
	
	
2024-03-13 21:24:11.614	
[notice] 2024-03-13T10:24:11.614006Z nonode@nohost <0.1559.158> 3caa0b50aa obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/_changes?style=all_docs&heartbeat=30000&filter=replicate%2Fpull&since=108574-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYHh0odcoBi7QXKihVmyJbp6HCbksQBJhgYg9R9u0FU9sEHmiUlJSZZp6NqyAL6BLdw&limit=50 204 ok 2
	
	
2024-03-13 21:24:11.617	
[notice] 2024-03-13T10:24:11.616610Z nonode@nohost <0.1595.158> c92cecc5f9 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_local/amEhkelueWmQgxmxfdNN0g%3D%3D? 304 ok 3
	
	
2024-03-13 21:24:11.623	
[notice] 2024-03-13T10:24:11.622319Z nonode@nohost <0.1595.158> a8475f00f5 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_changes?style=all_docs&heartbeat=30000&filter=replicate%2Fpull&since=108574-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYHh0odcoBi7QXKihVmyJbp6HCbksQBJhgYg9R9u0FU9sEHmiUlJSZZp6NqyAL6BLdw&limit=50 304 ok 3
	
	
2024-03-13 21:24:11.630	
[notice] 2024-03-13T10:24:11.630028Z nonode@nohost <0.1595.158> 5d71acff28 obsidian.home.com 192.168.1.254 obsidianuser GET /obsidian/_local/B1x2mC53febVgPzEmEaWOw%3D%3D? 304 ok 3
	
	
2024-03-13 21:24:11.637	
[notice] 2024-03-13T10:24:11.636475Z nonode@nohost <0.1595.158> 128f5c33e5 obsidian.home.com 192.168.1.254 undefined OPTIONS /obsidian/_changes?style=all_docs&feed=longpoll&heartbeat=30000&filter=replicate%2Fpull&since=108574-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYHh0odcoBi7QXKihVmyJbp6HCbksQBJhgYg9R9u0FU9sEHmiUlJSZZp6NqyAL6BLdw&limit=50 204 ok 1
	
	
2024-03-13 21:43:53.421	
[info] 2024-03-13T10:43:53.415687Z nonode@nohost <0.33.0> -------- SIGTERM received - shutting down
	
	
2024-03-13 21:43:53.421	

	
	
2024-03-13 21:43:53.421	
[info] 2024-03-13T10:43:53.416022Z nonode@nohost <0.33.0> -------- SIGTERM received - shutting down
	
	
2024-03-13 21:43:53.421	

	
	
2024-03-13 21:43:53.924	
[error] 2024-03-13T10:43:53.921060Z nonode@nohost <0.523.0> -------- gen_server <0.523.0> terminated with reason: killed
	
	
2024-03-13 21:43:53.924	
  last msg: redacted
	
	
2024-03-13 21:43:53.924	
     state: {state,#Ref<0.2839434048.1790574594.26395>,couch_replicator_doc_processor,nil,<<"_replicator">>,#Ref<0.2839434048.1790443522.26396>,nil,[],true}
	
	
2024-03-13 21:43:53.924	
    extra: []
	
	
2024-03-13 21:43:53.924	
[error] 2024-03-13T10:43:53.921214Z nonode@nohost <0.523.0> -------- gen_server <0.523.0> terminated with reason: killed
	
	
2024-03-13 21:43:53.924	
  last msg: redacted
	
	
2024-03-13 21:43:53.924	
     state: {state,#Ref<0.2839434048.1790574594.26395>,couch_replicator_doc_processor,nil,<<"_replicator">>,#Ref<0.2839434048.1790443522.26396>,nil,[],true}
	
	
2024-03-13 21:43:53.924	
    extra: []
	
	
2024-03-13 21:43:53.924	
[error] 2024-03-13T10:43:53.921620Z nonode@nohost <0.523.0> -------- CRASH REPORT Process  (<0.523.0>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_multidb_changes,init,['Argument__1']}, ancestors: [<0.469.0>,couch_replicator_sup,<0.386.0>], message_queue_len: 0, links: [], dictionary: [{io_priority,{system,<<"shards/80000000-ffffffff/_replicator.17002002...">>}}], trap_exit: true, status: running, heap_size: 2586, stack_size: 29, reductions: 1132941
	
	
2024-03-13 21:43:53.924	
[error] 2024-03-13T10:43:53.921764Z nonode@nohost <0.523.0> -------- CRASH REPORT Process  (<0.523.0>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_multidb_changes,init,['Argument__1']}, ancestors: [<0.469.0>,couch_replicator_sup,<0.386.0>], message_queue_len: 0, links: [], dictionary: [{io_priority,{system,<<"shards/80000000-ffffffff/_replicator.17002002...">>}}], trap_exit: true, status: running, heap_size: 2586, stack_size: 29, reductions: 1132941

from obsidian-livesync.

vrtmrz commented on July 17, 2024

Sorry for being late! It seems that the CouchDB was force-terminated by an Out-of-memory killer.
Fetch chunks on demand act in two phases.

Synchronise only notes using design documents (The filter definition in CouchDB. Parameter replicate%2Fpull of _chages on the log is so).
fetch only the chunks actually used from the synchronised notes.

I think that it has occurred at phase 1.

I have an idea to make it a bit more performant. (I am not sure about the actual effects yet). It will be added in the next release.

from obsidian-livesync.

zedtools commented on July 17, 2024

Thanks for the update.

For now I have two of my devices with Fetch chunks on demand enabled, and two with it disabled. If it happens again I will post here.

from obsidian-livesync.

Sync occasionally hangs, and takes a long time to recover about obsidian-livesync HOT 16 OPEN

Comments (16)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent