Comments (16)
What about a generated module? Could generate the entries and some query functions to scan them and delete the module when the cache expires perhaps.
from leveled.
Do you have any pointers to an example of doing that sort of thing? I'd like to try and understand the idea better.
from leveled.
Sure I have a library that does part of it.. http://github.com/DeadZen/goldrush
Should not be too hard to make it store and scan data internally, I would have to make a PoC
from leveled.
I'll work on it now, can't do anything else with AWS down atm .. ;p
from leveled.
OK. I'm still a bit unsure on what your suggesting, so I'm intrigued.
Just to be clear, that the snapshot is taken of the cache for the 2i query so that the query will provide a view of the database at the point in time that the query was initiated. I don't need to stream any new events that arrive after initiation of the query into the snapshot.
The Ledger Cache is an ets ordered set, and it gets flushed occasionally (after it has pushed itself to the Penciller). So by extracting from the set at the point of the query, the Bookie is free to flush the set, without any concern that there may be a clone pointing at the ets table expecting it to still be there.
The extraction at the moment is done by doing a tab2list, and this takes about 1ms. However, i will have N/3 nodes doing this concurrently - so if there's a lot of 2i queries all those 1ms CPU hits will add up. All I was intending to do was to iterate over the ets ordered set to extract only the results relevant to the query at the snapshot - on the basis that a few calls to ets:next will be cheaper than an ets:tab2list.
from leveled.
Ok I was actually going to ask if that was the case with needing to stream new events.
That's good for sure.
from leveled.
Will close this for now. Hopefully over the next few days will have some proper basho_bench results to compare with leveldb
from leveled.
In single-threaded tests, this does OK. But less so when running multiple parallel queries.
There is some possible concern about the cost of starting a query - i.e. what is the cost of a query on a near empty database with limited results to return? Is there too big an overhead of snapshotting?
from leveled.
How do I reproduce this test?
from leveled.
There's a basho_bench script I'm building for 2i. It is a work in progress at the moment. I will upload it tomorrow evening.
from leveled.
Here's the current state of the script (sorry forgot about posting this)
https://github.com/martinsumner/basho_bench/blob/mas-nhsload/src/basho_bench_driver_riakc_pb.erl
I'm going to do some more work on this Thursday.
There is a bunch of optimisation work I've undertaken which is all merged into this branch:
https://github.com/martinsumner/leveled/tree/mas-pushmem-i46/src
This has really about improving the performance,not under query load, but under the update-related load when adding more indexes.
The big differences come from this commit:
2b0ec1d#diff-25d902c24283ab8cfbac54dfa101ad31
There is a bit of a convoluted history around what made the difference. There are two parts - one allowing for slots that contain index entries to have twice as many keys (which reduces the pace of merge activity in the tree), and secondly implementing that in a way which reduces needless object copying by the beam.
Part of the issue is that I'd made a rookie mistake in assuming that all large objects are passed as references, not just large binaries. So there has been a general history of performance-related issues from message passing large objects between processes in leveled.
Prior to this, running a 6-hour test with multiple 2i index updates for every object, led to the beam occupying 40% of RAM by the end of the test. Now the same test and the beam ends up with just 10% of RAM - with all that saved memory being used in the page cache, and presumably saved CPU time in garbage collection
from leveled.
I've rolled through a number of ideas to try and improve this, and they all hit the buffers at some stage.
One thing to note is that although the manifest is small - it must be copied very aggressively for 2i queries. When volume testing at 200 2i queries per second, there are nearly 5000 manifest copies being generated every second. Each individual version of the manifest will be copied thousands of times.
Although the manifest is small, this seems wasteful. Instead, for the first snapshot for any given process a separate worker should be started - and subsequent snapshots for that manifest should use the same worker.
from leveled.
the change to share manifests across snapshots didn't make a huge difference the first time I tested .. but I was busy rolling a lot of changes at the time.
I will re-test this. Also considering a broader overhaul of actor organisation within the Penciller. Perhaps L0 and the manifest should live under a dedicated actor for handling those things - and all queries (including mainstream non-clone queries) are handled by runners which only has the levelzero index. So there will be a permanent runner (using runner as another actor name - as in bookie's runner) for non-snapshot requests, and runner clones for query requests. The actor holding the L0 sate will have to support snapshotting of request and handling multi-version (of both L0 and the manifest).
from leveled.
Tried to have a background accumulator of the penciller memory which could be use din queries ... but this didn't help.
https://github.com/martinsumner/leveled/tree/mas-pmemacc-i34/src
Quick volume test with 2i showed a significant reduction in throughput when compared to master
from leveled.
I'm of the opinion that I should defer work on this until we can test Leveled end-to-end with Riak in OTP 20.0 (and perhaps 20.0 ++ the I/O scheduler changes). Time might be a healer.
Actual testing hasn't actually highlighted this as a major issue. Also despite lots of thinking I'm devoid of any ideas to resolve it that don't have nasty side effects.
I'm moving this to the back-burner for now
from leveled.
When testing in OTP20/21 there appears to be significant speed-ups, especially when running ct tests.
In the target environment (NHS/Riak) things are ultimately slowed by disk - so enhancements here don't appear to be relevant. Also 2i query response times show no issues.
Won't fix - for now.
from leveled.
Related Issues (20)
- Inker delete_confirmed call not cast HOT 1
- Selective sync HOT 2
- Microseconds LMD HOT 1
- Recalc and revolving objects HOT 1
- confirm_delete race HOT 2
- Redact server state on crash HOT 2
- Adjust defaults to reflect higher vnode counts HOT 2
- Handle hash of 0 correctly in leveled_cdb HOT 1
- Experiment with zstd HOT 1
- Memory utilisation for large stores HOT 2
- Snapshot time
- Outdated deps preventing build HOT 3
- Potential Issue re delete post-compaction HOT 1
- Use standard erlang logger HOT 1
- Rebuilding ledger - but first 50K gone HOT 1
- leveled_monitor - sample_period HOT 1
- Penciller work backlog HOT 2
- Never stalling is an issue HOT 1
- Backup timeouts HOT 1
- term_to_binary/1 may change - and it has in OTP 26 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from leveled.