how do you deal with HUGE data? about memacs HOT 6 CLOSED

probably pulling the data needed from sqlite directly and recreating the org file just for the needed days would probably be more functional.

That's an interesting idea. It's not exactly related to this, but I have a prototype Org indexer package for Emacs that indexes Org entries into a SQLite database, based on John Kitchin's work he published on his blog. If you'd find it helpful in some way, let me know and I'll post it.

from memacs.

from memacs.

@JayDugger You can see the code here: https://github.com/alphapapa/helm-org-rifle/tree/org-rifle/sandbox It's very messy, pre-prototype code, but maybe you can get something out of it. It does work, but it's not organized to work as a package with user-facing commands, so you have to eval the code manually.

Note as well that much of it is directly copied from John Kitchin's blog post. I also haven't touched it for a while, so take it for what it's worth. :)

from memacs.

P.S. I guess the place to start is here, which appears to be approximately where I left off: https://github.com/alphapapa/helm-org-rifle/blob/org-rifle/sandbox/org-rifle-indexer1.el

from memacs.

from memacs.

Hi @kidd,

Well, my answer won't make you happy I guess.

My memacs agenda is slow.

Stats on my memacs files alone:

  710528 headings in   1507643 total lines
      1859 task headings
      708669 non-task headings

  217 open tasks:
      TODO:      108
      STARTED:   107
      WAITING:   2

  1642 finished tasks:
      CANCELLED: 1
      DONE:      1641

... where those task headings within memacs files are part of archived very old files and not related to my daily workflows any more.

Read about (early) performance measurements and optimizations: https://github.com/novoid/Memacs/blob/master/docs/performance.org

Probably more of interest is https://github.com/novoid/Memacs/blob/master/docs/FAQs_and_Best_Practices.org which describes some mitigation stuff I did.

The work of Kitchin where he moved things into SQL is quite interesting. I did not try this on my own. I rarely use my memacs agenda and therefore I accept the bad performance.

I'm going to close the issue for now. Please report back on any insights - others may want to know.

from memacs.