Use change notifications instead of polling about client HOT 50 CLOSED

The onwcloud server (as well as the clients) should just watch for changes on the local filesystems, to make them compatible with all methods to access the filesystem. Then trigger syncs upon changes.

A helper application that will set up filesystem-watches locally and remotely (and triggers efficient and rock solid unison syncs appropriately) is:
https://launchpad.net/sucsynct

To make notifications work even with plain webservers (non-ssh enabled) they may be tunneled (idealy directly as stdin and stdout shell pipes) with something like:
https://cgi-shell.binaervarianz.de (maybe a FastCGI version of it)

from client.

I concur, this is a major issue. However, only for Windows and Mac because owncloud for Linux can already be compiled with inotify support. [1]

Unfortunately, QT's QFileSystemWatcher class (QT 4.7) does not help, as it does not work properly. [1]
[2] hints that also for QT 5.1 there won't be better support.
Searching for a way out I found [3]. But it seems to be a lot of work especially since owncloud for Windows needs to be cross-compiled... And this would only apply for Windows.

However, this is a major drawback (cf. [4]) and should be solved soon IMHO.

[1] http://gitorious.org/owncloud/mirall/blobs/b66f8fba16124c18a43a52dcc1337d20d161801a/src/mirall/folderwatcher.h
[2] http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html
[3] http://qualapps.blogspot.de/2010/05/understanding-readdirectorychangesw.html
[4] http://bugs.owncloud.org/thebuggenie/owncloud/issues/oc-1533

from client.

Is there any documentation on how to build this client version with inotify support? The one in the repositories for ubuntu doesn't seem to be compiled with it (the log there shows a scan every 10 seconds) - and in the comment above there is just one file linked, is it in a branch of some sort? Isn't gitorious the old code repository, shouldn't it be on github?

from client.

Even though I never did it, I guess, you have to pass -DUSE_INOTIFY=ON to cmake (cf. [5])
The linked file [1] is indeed old. But it is the only explaination I found why inotify is used instead of QFileSystemWatcher.

[5] http://dragotin.wordpress.com/2012/03/22/csync-and-mirall-development-setup/#comment-265

from client.

I've tried and abandoned a few versions of owncloud for similar reasons. I've got about 14k files and it spikes the client and server CPUs pretty hard.

from client.

+1

Having thousands of files and syncing all the time is cpu and bandwidth consuming :)

from client.

+1
Currently my system takes around 8 hours to check that no files where changed :)

from client.

I've just updated my Linux client from 1.1.1 to 1.1.3 and it seems it behaves better in terms of not polling all the time. Any clue if it was updated to support inotify?

from client.

@nukeador I think you might have suffered from another bug in 1.1.1, happy that it works better with 1.1.3

We are working on better inotify support on all platforms. Please stay tuned.

from client.

+1 as well! With 13.000 files in 606 folders it takes more than 30 secs at 100% CPU, and that every minute.. :-( Very annoying! This is holding me from implementing ownCloud on larger scale..

from client.

Have to test yet with 1.2 (beta), but there this should be implemented already as far as I heard!

from client.

I have tested it with 1.2 beta1 on Windows.. Inotify works for new/changed files on client-side, but unfortunately not for files changed on server-side. So it still needs to poll the server to check for changes. Which is way too inefficient.

from client.

+1
We are trying to implement owncloud as an alternative to dropbox in our university research group, but at the moment this is not possible due to the spiking CPU usage of the windows client. I have ~10000 files in ~1000 folders, and running the owncloud 1.1.4 client my CPU load looks like this:

from client.

I read my own comment and statement, however, I think I was mistaken. The problem exists, but I don't think my explanation is correct. With the new client filesystem notifier, there shouldn't be a need for periodic polling. Maybe something which will be fixed in the next beta, after confirming that the new notifier works reliable enough? Or perhaps create an option to manually initiate this polling as an integrity check? And/or able to schedule it (perhaps also trigger on user inactivity)? I think it should be at least configurable.

from client.

I agree. The current beta 1.2 windows client still causes periodic CPU spikes. I am REALLY hoping that this is fixed soon as that's a deal breaker (power consumption, irritating fan noise variance, slow systems), and I really like what I've seen from owncloud. If there was a way to make this sync every 10 minutes instead of every minute, it might be bearable based on the duration of scanning my file set.

I did notice that changing the CPU priority of the owncloud process to lowest did reduce the impact on my system. However, it also increased the time over which the spike occurs. Obviously no free lunch.

from client.

I am also using the 1.2 beta client. I set the polling interval to 5 minutes (localPollInterval=300000 in owncloud.cfg) which works. BUT there still is a further problem - whenever a change in a file is detected by owncloud (e.g. I open a spreadsheet and openoffice/excel writes a .lock file or I close a spreadsheet and the .lock file is deleted) a 'syncing spike' is caused.
So whenever any file is changed/created/deleted, the whole cloud directory seems to be synced, which at this point is very resource ineffective for large amounts of files and folders.

from client.

CPU load is much better in 1.2 stable client for Windows. Good job, devs!

edit: Sadly this statement was made prematurely - I still have the problems described above. Please keep working on this.

from client.

I have the same Problem even with owncloud 1.2 I have 24GB of files with 40k Objects (Files and Folder) in it. CPU power always on 100 to 120 %. My Battery lasts 1 Hour with the owncloud client on ;). Theres absolutly no way of getting it in a productive Environment. Sorry if i doubled some notes here. But its very important to change this. Thank you so much.

Beside that. Very good job devs 👍

from client.

This is a major problem for me too, on a Windows machine trying to sync a 50GB documents folder made up by a few thousand files. Constant mountains on the CPU graph. The only real disappointment from this awesome software so far! Really hoping this gets fixed asap XD inotify-win may be a good project for source insight?

from client.

Note to self: Skip local run if no inotify event came in between to sync runs. It should be possible to construct the tree from db.

from client.

While waiting the above issue to be resolved, is there a wait to set the polling interval on windows (I could not find a owncloud.cfg)? So that one could just poll for changes a few times per day for large syncfolders. Or perhaps a way to launch it as a one shot (from the windows scheduler) that stops when done?

from client.

@Boran The owncloud.cfg on windows is in C:\Users(Username)\AppData\Local\ownCloud\owncloud.cfg

from client.

@ctd1500: Thanks.
Setting the two intervals to (for example) 5 mins does work in the sense that nothing is written to the log
localPollInterval=3000000
remotePollInterval=3000000
That is until a file is changed.
==> Then it goes into constant activity, walking all files, repeatedly.....

Would be nice if there was to start ownload, sync, and stop again (oneshot mode)

Anyway I dont want to prolong this thread, but am providing some log entries below should it help later..

An example of the log entries, here one see INO.txt being walked and no change found.

03-02 17:23:25:246 Sync state changed for folder "owncloud2-common" : "Success"
03-02 17:23:25:246 * "owncloud2-common" Poll timer enabled with 2998379 milliseconds
03-02 17:23:25:246 <===================================== sync finished for "owncloud2-common"
03-02 17:23:25:452 XX slotScheduleFolderSync: folderQueue size: 0
03-02 17:23:27:240 * event notification enabled
03-02 17:23:27:773 virtual void Mirall::WatcherThread::run() Change detected in "C:/Users/myuser/owncloud2-common" from Mirall::WatcherThread(0x40d9ad8)
03-02 17:23:27:775 * Pending events for "C:/Users/myuser/owncloud2-common" will be processed after events stop for 1000 milliseconds ( "17:40:07" ). 1 events until now )
03-02 17:23:28:769 * Processing of event queue for "C:/Users/myuser/owncloud2-common"
03-02 17:23:28:769 * Notify 1 change items for "C:/Users/myuser/owncloud2-common"
03-02 17:23:28:769 ** Changed was notified on ("C:/Users/myuser/owncloud2-common")
03-02 17:23:28:769 * "owncloud2-common" Poll timer disabled
03-02 17:23:28:769 Schedule folder "owncloud2-common" to sync!
03-02 17:23:28:769 XX slotScheduleFolderSync: folderQueue size: 1
03-02 17:23:28:769 ==> load folder icon "owncloud-framed"
03-02 17:23:28:770 Returning configured owncloud url: "https://myhost:8008/owncloud/remote.php/webdav/"
03-02 17:23:28:770 Folder in overallStatus Message: Mirall::ownCloudFolder(0x38d1c58) with name "owncloud2-common"
03-02 17:23:28:773 Sync state changed for folder "owncloud2-common" : "Sync Running"
03-02 17:23:28:773 *** Start syncing url to ownCloud: "ownclouds://myhost:8008/owncloud/remote.php/webdav/owncloud2-common"
03-02 17:23:28:773 Returning configured owncloud url: "https://myhost:8008/owncloud/"
03-02 17:23:28:776 starting to sync QThread(0x5e7858) QThread(0x40b1ac8)
03-02 17:23:28:777 ==> returning exclude file path: "C:/Program Files (x86)/ownCloud/sync-exclude.lst"
03-02 17:23:28:777 ==== added CSync exclude List: "C:/Program Files (x86)/ownCloud/sync-exclude.lst"
03-02 17:23:28:781 virtual void Mirall::WatcherThread::run() Change detected in "C:/Users/myuser/owncloud2-common" from Mirall::WatcherThread(0x40d9ad8)
03-02 17:23:28:781 virtual void Mirall::WatcherThread::run() Change detected in "C:/Users/myuser/owncloud2-common" from Mirall::WatcherThread(0x40d9ad8)
03-02 17:23:28:781 virtual void Mirall::WatcherThread::run() Change detected in "C:/Users/myuser/owncloud2-common" from Mirall::WatcherThread(0x40d9ad8)
03-02 17:23:28:788 virtual void Mirall::WatcherThread::run() Change detected in "C:/Users/myuser/owncloud2-common" from Mirall::WatcherThread(0x40d9ad8)
03-02 17:23:28:790 * csync thread started
03-02 17:23:28:790 * event notification disabled
03-02 17:23:28:790 >===================================== sync started for "owncloud2-common"
03-02 17:23:28:794 FolderWatcher::changeDetected when eventsEnabled() -> ignore
03-02 17:23:28:795 FolderWatcher::changeDetected when eventsEnabled() -> ignore
03-02 17:23:28:796 FolderWatcher::changeDetected when eventsEnabled() -> ignore
03-02 17:23:28:797 #### Update start #################################################### >>
03-02 17:23:28:797 csync_ftw: Incoming read_from_db-Flag for C:/Users/myuser/owncloud2-common: 0
03-02 17:23:28:797 csync_ftw: .csync_journal.db excluded
03-02 17:23:28:797 FolderWatcher::changeDetected when eventsEnabled() -> ignore
03-02 17:23:28:798 csync_ftw: .csync_journal.db.ctmp excluded
03-02 17:23:28:798 csync_vio_stat: Win32: STAT-inode for C:/Users/myuser/owncloud2-common/INO.txt: 256002
03-02 17:23:28:798 csync_ftw: Uniq ID read from Database: INO.txt -> 512f62b45b29b9.42234363
03-02 17:23:28:798 csync_ftw: walk: C:/Users/myuser/owncloud2-common/INO.txt
03-02 17:23:28:798 csync_walker: file: C:/Users/myuser/owncloud2-common/INO.txt
03-02 17:23:28:798 _csync_detect_update: file: INO.txt - hash 37354583044946145, mtime: 1360920257
03-02 17:23:28:799 _csync_detect_update: time compare: 1360920257 <-> 1360920257, md5: 512f62b45b29b9.42234363 <-> 512f62b45b29b9.42234363
03-02 17:23:28:799 _csync_detect_update: file: INO.txt, instruction: INSTRUCTION_NONE

from client.

+1
currently unusable with more than 4k-5k file/folder
mac client 1.2.1

anyway congratulations for the work done

from client.

Syncing 640MB (3784 files in 222 directories) with ownCloud Client 1.2.5. This takes up a whole core every few seconds. Is there anything i can do to help resolve this issue?

from client.

@ccoenen We need some sort of API on the server that we can connect to and which pings us as soon as something changes on the server (including shared directories). Maybe ask on the list how you can help there? We should also officially open an issue in the core about this. Mind raising one?

from client.

@ccoenen The other optimization opportunity is to not always scan the local tree, but store its state and rely on the file watchers to send a notification. While that sounds like an obvious optimization, it depends on reliable file watching mechanism (at least on Linux inotify is not gueranteed to work) and we need to iron out more basic issues before starting to optimize.

from client.

How about relying on the unreliable notify for a while, later performing a sweep (the machanism that's currently running every other minute)? This way, you'd get your changed files synced quickly 99% of the time, but at the benefit of less CPU Cycles burned. The full sweep at longer intervals can clean up after the not-so-reliable mechanism.

I have to emphasize that this is not really my area of expertise, so i might miss something important.

from client.

+1

Windows client 1.2.5 (unsurprisingly) exhibits exactly the same behavior as described above.

I was trying to set up a sync between a home server and two computers to simplify data management of all my data. While the setup technically worked, the periodic CPU utilization and cooling fan speedup (and, on a laptop, increased battery drain) forced me to abandon the use of OwnCloud (at least for now).

I would love to see this issue corrected!

from client.

+1

Windows client 1.2.5 (x64)

I observe exactlly the same behavior as described above, even with <1000 files and only ~50MB; my battery drops veeery fast.

I'd love to use owncloud as my one and only sync-solution; keep up the good work!

from client.

1.3.0 windows: still using a lot of cpu-cycles every 30 seconds.

Another note: I regularly deactivate the client whenever run a game, for example. It would always create a noticable lag every 30 seconds as well.

from client.

Same issue here on Mac OS X 10.8.4 with ownCloud Client 1.3.0. The fans of my retina MBP are running all 10 seconds and I got 100% cpu usage.

I'd like to see this issue fixed cause it's very irritating to here the fans all the time but I like ownCloud and want to say "Goody Bye" to Dropbox ;-)

Edit: Clicking the "Reset" button for all my sync directories seams to solve the issue.
Edit2: Saddly, my first edit is wrong. I detected, that after resetting the syncs, owncloud stops syncing until the next client restart. After the restart. All issues are back again. Furthermore I got a lot of conflicts file when working in a sync directory.

from client.

1.3.0 windows:

Same Problem here. OwnCloud is eating up my CPU every 2 minutes, even if nothing was changed.

This is one of the most important "bugs" to fix.

07-08 23:35:58:821 void Mirall::CSyncThread::startSync() Sync started 
[...]
07-08 23:38:05:940 void Mirall::CSyncThread::startSync() Sync started

from client.

+1. Mac OS X 10.8.4 / ownCloud 1.3.0 and still CPU spikes every few minutes. Very, very annoying - makes ownCloud unusable!

from client.

CPU Issue not fixed in 1.4 Beta1. 20k Files syncd. now it eats up my CPU. Again. Maybe in the final.

from client.

Would somebody please look at this issue? It's one of the oldest tickets in here, affects every single user. It's a major pain in the back, and it's labeled "1 - Backlog"?

The fact that there's a few duplicates of this one as well plus the number of people commenting in here indicates that it is not a low priority problem with sync. A regular ticket has 10 comments tops, most of them only have 2-3 comments. There's 22 people in here that have taken the time to write up their specific use case or problem.

I would like to ask you kindly to put this on your near-term roadmap.

from client.

@ccoenen We are actually about to address this with 1.4. The labels are not very up-to-date I agree.

from client.

\o/ About time.. looking forward to moving back to Owncloud :)

from client.

I'd like to apologize for the "demading" tone of my previous comment. Thanks for looking into this! It's very important to me. I'm looking forward to the upcoming version!

from client.

In 1.4, the poll is just an HTTP request to check that nothing changed on the server.
There is still a full sync hapenning every 5 minutes to be sure we catch all the changes that may not been seen by the file system watcher.

from client.

What's the status on this - still a work in progress? Still needs to do a full sync every 5 minutes?

from client.

@fragtion absolutely not. Every fife minutes, we do one PROPFIND on the server and a poll through the local filesystem. No full sync.

from client.

Even on the newest client (1.4 on OSX) I'm still seeing 100% cpu usage.

This is doubly irritating on OSX, since there is a hacky (but functional) filesystem notification method in fsevents. (If used properly, you never have to do a scan - you can catch up with changes that were missed when the client was not running.)

9800 files, 2100 directories.

from client.

@disconn3ct Same here on MacOS... :(

from client.

It's definitely a lot better on Windows now, haven't noticed a problem! Hope the devs can help you guys with Mac too, cause this rocks :)

from client.

Hi guys,

is there an update on the use of inotify to traverse only parts of the tree?

If I look at the logs of the client 1.6 I still see the polling every 30sec for general changes on the filesystem and if there is only one detected change (etag), the client traverses the whole tree which sucks and takes a long time as I have 124K files ..

In summary my real-time syncing takes about 5-10 minutes (thanks to SSD and i7).

07-14 12:44:27:886 !!! Mirall::CheckQuotaJob created for  QUrl( "xxx" )  querying "/" 
07-14 12:44:40:777 * Polling "ownCloud" for changes. (time since last sync: 30 s) 
07-14 12:44:40:777 !!! Mirall::RequestEtagJob created for  QUrl( "xxx" )  querying "" 
07-14 12:44:41:168 * Compare etag  with previous etag:  false 
07-14 12:44:58:272 !!! Mirall::CheckQuotaJob created for  QUrl( "xxx" )  querying "/" 
07-14 12:45:10:777 * Polling "ownCloud" for changes. (time since last sync: 60 s) 
07-14 12:45:10:777 !!! Mirall::RequestEtagJob created for  QUrl( "xxx" )  querying "" 
07-14 12:45:11:124 * Compare etag  with previous etag:  false 
07-14 12:45:28:770 !!! Mirall::CheckQuotaJob created for  QUrl( "xxx" )  querying "/" 
07-14 12:45:40:777 * Polling "ownCloud" for changes. (time since last sync: 90 s) 
07-14 12:45:40:777 !!! Mirall::RequestEtagJob created for  QUrl( "xxx" )  querying "" 
07-14 12:45:41:168 * Compare etag  with previous etag:  false 

from client.

The 30 seconds polling is only for server-side change, we don't look at the file system there.
We have a fille system watcher that track the changes. But it is not reliable so we still force a full scan of the local directory every 5 minutes. You can increase this time with the forceSyncInterval option in the mirall config file (and set a number of milliseconds)

from client.

This issue is the only thing That keeps me away Form own cloud.

from client.

@pixelpoesie you might notice that this issue is closed (and in this case, that more or less means, implemented). What exactly about it is it that keeps you away? If you have any suggestions on what to do better you probably should report a new issue.

from client.

@pixelpoesie it's fine now. There might be room for improvement, but since 1.4 or 1.5 i don't notice the syncer anymore (which is probably the highest kind of praise for this type of software).

Before that, i used to turn it off in certain situations to save disk-io, CPU cycles and battery. Haven't done this in a while.

from client.