Unlimited history about clipster HOT 10 OPEN

As has been pointed out, history size can be set very large (and could probably be made to be unlimited relatively easily by ading config to disable the 'pruning' code).

I'd be interested to know at what history size the current code starts to be noticeably slower (to a human) - if that number is suitably small then I might consider an alternative history backend. I suspect however that the number will be surprisingly large! (for a bit of fun, have a look at https://computers-are-fast.github.io/)

Hi, sorry also for the late reply - looks like I missed your message.

I don't use Windows (and therefore Ditto) much any longer, but as I've left it, my Ditto database measures at over 7 GB. Suffice to say, keeping it all in RAM would be an unreasonable misuse in resources. The 7 GB figure comes after periodic cleanups for deleting screenshots or other very large (over several MB) clips. The database is currently at 195539 clips, which puts them at a mean size of 37508 bytes per clip (incl. metadata such as search indexes), which to me seems not unreasonable.

Some of the larger clips are things like:

• log files
• terminal session transcripts
• entire source files
• program outputs
• chat logs

Again, IMO, not unreasonable.

So, I do still think there is a use case here :)

from clipster.

Just to follow up on this - I've done some rough and ready speed testing.

The 'baseline' speed for the clipster client on my laptop is 0.1secs - i.e if I tell it to output the history when the daemon isn't running, and it exits immediately.

If I query a 'small' history this adds about 0.025secs to the process.

If I generate a 500MB history file, made up of 50,000 10KB entries, it still only adds 0.025secs.

The only thing that is 'slow' is if I then do clipster -o -n 10000 and even then the total time is under a second.

I can't easily test a larger history as I start to run out of memory!

So I'm reluctant to accept that clipster is slowing down for you due to history size - unless of course the history has got slow large that it's starting to move into swap or something?

There are other bugs open related to performance (like #74) which I need to dig into - this might also be the cause of your slowdowns.

from clipster.

The stalebot was actually added in August last year with a view to tweaking it to do minimal tidying. Unfortunately it never fired, so I forgot about it until this last week when it 'woke up'. Unfortunately I was away so missed the pending stale events to stop it before it happened.

I generally try to apply tags to 'pending' issues which haven't been fully resolved and had configured stalebot to ignore some of these - but unfortunately missed a couple, which is why so many 'useful' issues got closed. I've now reopened all of these so they can continue to receive input.

from clipster.

Is this so different than, say, set history size to 50000000?

What I would find more useful is a handy way to specify "output all entries" with -n since writing 50000000 is a bit cumbersome at the CLI.

from clipster.

@memeplex I've just taken a look at the code for this, and since the value of -nis just used to slice the history list, I've very slightly modified the code so that -n can now take 0 and negative numbers.

clipster -o -n 0 - return entire history.

clipster -o -n " -5" - return all but first 5 items in history. (this odd syntax is due to an argparse bug, fixed in some releases: https://code.google.com/archive/p/argparse/issues/37

Changes are in: 7ff78f1 (plus docs changes in a later commit).

from clipster.

@CyberShadow apologies for not replying to your original issue sooner... was sure I had!

While the idea of a DB backend is appealing, it does add an extra layer of complexity and dependencies to the code, which I'm not convinced is entirely justified.

As has been pointed out, history size can be set very large (and could probably be made to be unlimited relatively easily by ading config to disable the 'pruning' code).

I'd be interested to know at what history size the current code starts to be noticeably slower (to a human) - if that number is suitably small then I might consider an alternative history backend. I suspect however that the number will be surprisingly large! (for a bit of fun, have a look at https://computers-are-fast.github.io/)

from clipster.

I've been wanting to update this issue for a while now, since clipster has turned slow for me, presumably because of my history size.
Unfortunately my ssd died a few days ago, and now I don'd know what my size was.
All I know is that it was less than 50000 entries, as that is what my history was capped at.

from clipster.

I'd be interested to know at what history size the current code starts to be noticeably slower (to a human) - if that number is suitably small then I might consider an alternative history backend.

Mine is 38M, and very slow.
Getting the output of clipster -o -n 0 -0 | fzf --read0 --no-sort --reverse --preview='echo {}' | sed -ze 's/\n$//' | clipster takes a few seconds, and pasting even longer.
I am in swap though.

from clipster.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from clipster.

@mrichar1 Please don't use stale bots, they do more harm than good. Not just in terms of value they bring to the particular project, but they generally erode public trust and willingness to participate in open-source software development.
See the discussions here: https://www.google.com/search?q=stale+bot+site%3Aycombinator.com

from clipster.