Bulk read/write API about kv-storage HOT 7 OPEN

This discussion is a bit split across here and #57 (comment), but I'll repeat from there that the main tradeoff for adding new features is ensuring that they benefit real-world apps.

It's very clear that KV storage, being only a subset of IndexedDB by design, is less capable and doesn't have bulk operations at this time. The question is, how much of a problem is that for real-world apps? Are there real-world apps that want to perform simple key-value storage operations, but also perform them so often and in such volume, that the current API causes noticeable user-facing slowdowns? Our conjecture was that apps that need that kind of transaction volume would rather use IndexedDB directly, or would face that transaction volume in rare-enough instances that dropping down and using the backingStore getter would be sufficient. We thought this was pretty clear from the evidence of the various IDB wrapper libraries that are popular today, and don't support batched read/write in this way.

But we might be wrong! Do you have a site that's using, e.g., a different IndexedDB wrapper library today, and wants to move to KV storage but can't because that'd cause user-noticeable performance problems from de-batching? That would be invaluable information for evolving the API, and judging whether increasing the complexity and implementation cost is worth it because of how it makes the web better.

I'll also note that it's important to focus on the problem statement, and the solution might not look exactly like changing the web-developer-facing API. For example, in #57 (comment) Firefox developers suggest that one path forward might be took the same web-dev-facing API, but optimize it so that it operates "as fast as" batch operations. But, all that talk is premature without having some examples of actual customers (not synthetic benchmarks) who would benefit from such an upgrade.

from kv-storage.

The synthetic benchmark above is just an extract of what an app is doing on update, actually reduced since the original could take more than 2 seconds. As the experience shows (Best Practices for Using IndexedDB), instead of storing a huge state object as a single entry apps should break it down into small atomic things. While one could argue such activities are rare, but it doesn't mean the users should wait for seconds instead of milliseconds. Even the much more primitive WebExtensions/chrome-extensions API provides a way to perform bulk operations.

from kv-storage.

Ah, great! Could you point me to the app?

from kv-storage.

It's a chrome extension I'm playing with - a 4k array of rule objects (built from a json on update) should be loaded each time the background script resumes after being unloaded - that is almost on every tab navigation. The array is inside IndexedDB, each rule a separate entry. I'm using getAllKeys to quickly get 4000 urls in just 15ms, find the matching ones, then read the full objects for each one. There must be lots of similar use cases - in advanced [web] apps like editors or games that need to load/save the state.

from kv-storage.

Ah, hmm, I see. I'm not sure the web standards process is the best way to add APIs for Chrome extensions; asking Gecko and WebKit to implement something, which would only be used in Chrome, is not great form. Generally, it's best to add Chrome extension APIs through the Chrome extension API process, not the web platform standards process.

Still, I believe you that it's possible there's web apps that might benefit from this. Let's keep this open to track the feature request, in the hopes of finding a web app to work with on this sort of feature addition with us.

Maybe one useful thing to ask, is are there other parts of the IndexedDB API you are using (e.g. indices, transactions, multiple separate object stores, ...), or are you just using it as a key/value store?

from kv-storage.

I'm not sure the web standards process is the best way to add APIs for Chrome extensions

The features of IndexedDB I'm using aren't specific to extensions API so it's not pertinent to the message I was trying to convey: I believe there are lots of web apps that [will] need to perform bulk operations. I've mentioned the set/get methods of extensions API just to show a possible way to expose the putMany/getMany functionality.

other parts of the IndexedDB API

I'm using an additional index indeed and a second store, although it's not crucial - just a minor touch.

from kv-storage.

We made a monstrosity which could be a great example of a "large scale application" in real life:

• It works offline;
• We only save "updates" in our database;
• It's controlled by state manager;
• After update we put whole database state object into localStorage;
• When user returns, we take database state object from localStorage and pop it into state manager;
• It worked;
• We reached 5MB size limit for some clients;
• We didn't have time to invest into indexedDB;
• I found out about kv:storage, used it as a drop in replacement, fixed a couple of bugs (added few awaits);
• It works.

Averegish~ how long it takes.

But we don't even care, because we don't even wait for it to finish. State manager handles it. No visible lag. Everyone is happy.

I probably sound like an asshole (I probably am), but this is how stuff works. We need to deliver good results and these results are good enough because UI handles it.

I'm not sure what you are trying to achieve with kv:storage, but it looks like another library that wraps indexedDB with API that looks like localStorage. If you are creating a standard, wouldn't it be more logical to create something like NoSQL database with offline support? Other libraries could extend it, but they can't create low level APIs like those, they must use them to achieve something.

from kv-storage.