Comments (8)
When I used to wait maybe 30-40 seconds for a monthly review, it now takes 10 seconds.
Still 10 seconds? Sounds like we still have room for improvement ;)
I assume that you are not running a debug build, if you are that makes a huge difference in performance.
I want it to be fast enough so a yearly summary won't be close to the per-request 30s timeout for most computers.
My previous db size was 264MB, and its now 213MB, that's an 8% decrease in size. I checked the logs, the bucket sizes and everything seems to match. Is a decrease of this size expected? Furthermore, does aw-server-rust use sqlite by default compared to peewee for aw-server?
Interesting, I had no idea. Peewee is a wrapper around sqlite and now we just use raw sqlite, not sure what the difference might be actually.
Might be a good idea to put a notice for users to hard-refresh their page in the release notes once aw-server-rust becomes the default. Some of the buttons weren't responding for me and I was getting 404's. Only once I did a hard reset, did web-ui function correctly.
Huh, did not know that either. Good point.
Some people may not be aware that aw-server and aw-server-rust store events in different databases(I definitely wasn't), so if they use them interchangeably they will lose some events. This obviously won't happen when aw-server-rust becomes the default, but in the meanwhile, possibly consider adding a disclaimer to the release
Did the automatic database migration work for you or did you manually export from aw-server-python and import again in aw-server-rust?
I don't think the migration has been tested very thoroughly in the "wild", would be interesting to hear if anyone have had any bugs with that before making it the default.
Looking at the logs, there seems to be an issue with sending events from firefox-watcher:
Yeah, the ERROR and WARN is not an issue because it's handled by the CORS fairing afterwards, I've tried to suppress that log and asked upstream how to be able to do so but no luck yet.
from aw-server-rust.
About the DB size that could also simply be because of database fragmentation. If you import everything in order afterwards there will be less fragmentation.
from aw-server-rust.
Regarding the annoying logs, that's a duplicate of the following issue:
from aw-server-rust.
Still 10 seconds? Sounds like we still have room for improvement ;)
I assume that you are not running a debug build, if you are that makes a huge difference in performance.
I want it to be fast enough so a yearly summary won't be close to the per-request 30s timeout for most computers.
I'm running 0.92 release, with nothing extra. The biggest reason for the delay is the "Loading of the Browser Domains". It takes a disproportionate amount of time for some reason. I reran it, and the entire page for the month loaded within the first 5 seconds (Top applications, Window Titles, Categories), and then it took 25 more seconds for the Browser Domains to be displayed(literally 5 times more). Same issue occurs in aw-server, so I'm wondering it this is an issue on the aw-webui front.
@ErikBjare mentioned in ActivityWatch/activitywatch/issues/445 that he plans on merging the window and browser queries together. Would that fix this issue?
About the DB size that could also simply be because of database fragmentation. If you import everything in order afterwards there will be less fragmentation.
I think another reason might be the change in some types: since SQLITE now uses 2 Integers instead of 1 Datetime + 1 Decimal to show events. Too lazy to do the math and find the size of each type but even if there's a 1 byte difference when you have 500k+ events, it adds up really quickly. Either way, the space savings are another thing to promote in future releases when it becomes the default :)
Did the automatic database migration work for you or did you manually export from aw-server-python and import again in aw-server-rust?
I don't think the migration has been tested very thoroughly in the "wild", would be interesting to hear if anyone have had any bugs with that before making it the default.
So I just did a pkill aw-
, then launched aw-server-rust + all watchers
, so aw-server-rust did the migration automatically. I have maybe 1 million events, I think it took maybe 5-10 minutes(very rough estimates, I don't actually remember). Does the code account for a user suddenly shutting off aw-server-rust, and what happens in that case?
You said the migration hasn't been tested thoroughly in the 'wild'. What are your thoughts on something like this to test it:
If there are some tests for integrity in place (I remember the logs indicated the buckets were successfully imported), you could have users run the migration, capture the output in a file (easiest way might be to give them a script they can run locally), and post it in a 'Migration` Issue on ActivityWatch. In the issue we can ask for 5 reports(more would be even better) for each OS platform. I think given the amount of people using AW finding 15 people willing to do this would be very straightforward. 5 for each platform should good enough to find most of the big issues regarding the migration.
Other tests the users could do(above would of course be the most important one), is:
- Launch aw-webui play around with it, see if everything works
- Run aw-server and aw-server-rust side by side, and see if there is any glaring issues(to run side by side perhaps have aw-server-rust migration happen on --testing).
from aw-server-rust.
I'm running 0.92 release, with nothing extra. The biggest reason for the delay is the "Loading of the Browser Domains". It takes a disproportionate amount of time for some reason. I reran it, and the entire page for the month loaded within the first 5 seconds (Top applications, Window Titles, Categories), and then it took 25 more seconds for the Browser Domains to be displayed(literally 5 times more). Same issue occurs in aw-server, so I'm wondering it this is an issue on the aw-webui front.
aw-server can only do one query at a time, so the first 5 seconds is usually the window query and then the browser query runs. So if the browser query takes 25 seconds thats 25-5=20 seconds. So 4 times more I guess which is still very bad.
@ErikBjare mentioned in ActivityWatch/activitywatch/issues/445 that he plans on merging the window and browser queries together. Would that fix this issue?
I was the one to fix that and is on master now so it will be available in next release. The time for all queries will be faster because they are now a single query (both the window query and browser query fetched events from the afk and window buckets which was unnecessary), but that also means that the time to only get the window titles will be longer since the result for both will arrive at the same time.
So I just did a pkill aw-, then launched aw-server-rust + all watchers, so aw-server-rust did the migration automatically. I have maybe 1 million events, I think it took maybe 5-10 minutes(very rough estimates, I don't actually remember). Does the code account for a user suddenly shutting off aw-server-rust, and what happens in that case?
Damn, that's a lot of events! I have around 250k on my laptop (which is not my primary computer, but anyway). Takes a little less than a minute to import though.
Would be possible to let the webui start first though and then import buckets and possibly have a "ready" api which can give a reason for why aw-server is not ready yet in the web-ui.
You said the migration hasn't been tested thoroughly in the 'wild'. What are your thoughts on something like this to test it:
If there are some tests for integrity in place (I remember the logs indicated the buckets were successfully imported), you could have users run the migration, capture the output in a file (easiest way might be to give them a script they can run locally), and post it in a 'Migration` Issue on ActivityWatch. In the issue we can ask for 5 reports(more would be even better) for each OS platform. I think given the amount of people using AW finding 15 people willing to do this would be very straightforward. 5 for each platform should good enough to find most of the big issues regarding the migration.
Other tests the users could do(above would of course be the most important one), is:
- Launch aw-webui play around with it, see if everything works
- Run aw-server and aw-server-rust side by side, and see if there is any glaring issues(to run side by side perhaps have aw-server-rust migration happen on --testing).
Yeah, shouldn't be too hard to test it on all platforms and to test it by some different people with different data in their buckets.
The issue would only be if an issue occurs that the user might not want to provide their buckets to show what broke the import. But I'm not sure what kind of data could possibly break that so that might not even be an issue.
from aw-server-rust.
@johan-bjareholt it's interesting that you have a lot less events than me, since I would imagine you started tracking with AW long before I did. My current buckets contain data starting from August last year
from aw-server-rust.
@nicolae-stroncea Not that strange considering that it's from my laptop that is not my primary computer. I'll likely have more events on my personal desktop and likely the most on my work desktop.
from aw-server-rust.
I just checked my work desktop, it has around 1.4 million events and I've been running activitywatch on it for about 3 years.
from aw-server-rust.
Related Issues (20)
- running get_events from aw_client returns events in reverse order HOT 1
- Persist settings to server HOT 8
- Pins cpu usage at 100% HOT 4
- Major data loss -- server doesn't appear to write data to disk properly HOT 2
- Called `Option::unwrap()` on a `None` value HOT 2
- No log file found on startup
- Error loading common words in "Uncategorized events"
- No rule to make target 'build' HOT 1
- "poisoned lock: another task failed inside" HOT 6
- This is really a great project. I would like to help you with some functionalities. Please let me know where to start. HOT 4
- database upgrade causes all data lost HOT 4
- Encryption at rest for database files HOT 1
- Error: No db found in sync folder HOT 3
- Possible database corruption and 100% CPU after Windows updates
- aw-sync old libssl dependency HOT 2
- Missing license file in mpsc_requests-0.3.3.crate HOT 8
- Suggestion: Regex-based inbound data filters HOT 4
- Add `!local` special hostname when creating buckets (for aw-server-python parity) HOT 1
- Add more options to aw-sync daemon
- Suggestion: Optimizing the Database
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aw-server-rust.