Datasets up to 1GB? about nedb HOT 15 CLOSED

Hey,
Glad to hear that. I especially like the phrase "little brother of MongoDB", I'll reuse it :) Concerning handling large (1 GB+) databases, I don't think I'll be able to support it in the near future, since it would be a lot of work, I am pretty busy these days and I need to finish indexing first (I hope it will be tomorrow).

That said, this is still something I would love for NeDB to support one day, so I would like to know more about your usecase. What kind of documents are you planning to store ? A lot of small documents or a few huge ones ? Could you also tell me more about your node webkit application, because I can't think of an example of a desktop app needing to store 1GB+ of data.

I will leave this issue open in case someone wants to contribute a PR.
Cheers,
Louis

from nedb.

I'm working on desktop RSS reader (as replacement for google reader, will be opensource) and I'm looking for way to store articles/blogposts locally. So everything depends how many and how active feeds someone subscribes to. Assuming articles would be kept on disk for one year (for history search), and someone subscribes to 50 feeds, and on every feed would appear one 30KB article every day database grows to about 0,5GB. Ok, so maybe 1GB is not easily achivable, but in edge case it could happen :), and hundreds of megs are not hard to imagine.

NeDB is also very nice to use as file format for desktop applications and not have to invent your own way of storing data. Then people will probably came with 10x more freeky use cases you expect :)

Keep up good work :)
Jakub

from nedb.

Thanks for the details. I was thinking about how I would implement such a feature and it struck me that for this particular usecase (not that many documents, less than 20k, but every document can be pretty big), you could try to it git style. You hash (with SHA-1 for example) every article that comes in, and store it in a file whose name is the hash.

That way, the only data you need to put in your database is the hash (along with the article's metadata), and you only read the articles you need when the user requests them. I could certainly implement such a behaviour, what do you think?

from nedb.

Btw, here is an article that details why I said 'Git style' : http://git-scm.com/book/en/Git-Internals-Git-Objects

from nedb.

Thanks for the tip, it sounds reasonable indeed. I will try now simply storing everything in nedb and migrate to git-style in the future if it turns out to be necessary (I don't plan to store so large articles history just yet).
But the main reason I want to have it all in one file is convenience, so I don't have to invent my own storage logic (I know git-style is very simple and straight-forward :) ), and that one file simply looks clean.

from nedb.

Since it is a lot of work, I'll close this until more people ask for it :)

from nedb.

Funny, i am building a Desktop-RSS-App with node-webkit, too :) For now i used vanilla localStorage and the WebSQL Database, which is quite dirty for this stuff.

You asked for it: i second szwacz. NeDB provides a real crossplatform solution for node-webkit, given that it can handle at least ~1GB of data :)

from nedb.

I'll need some time to see how to do that, and I guess some more upvotes :) But that's definitely something I'd like nedb to support.

That said, an RSS reader is a special case : the only big fields are the articles themselves. So what you could do, since article test doesn't change, is put every article in a file whose name is the article's MD5 hash and use nedb to store the hash and all the metadata (title, date, author and so on). That's similar to the way Git operates and that would keep you going cleanly before nedb itself supports big datasets.

from nedb.

It would be great if there was support for not loading the entire db to RAM. I'm considering using nedb in a line of business node-webkit application and the size of the data varies depending on the client. While the chance that they'll amass a lot of data is slim, there's still a possibility.

from nedb.

After year and a half there is still no better in-preocess database than NEDB, and I finally had to do something with the memory-hogness of my app.
@louischatriot Thanks for bringing up the idea of git objects! This turned out to be the easiest way.
Maybe someone else will find it useful: https://github.com/szwacz/scattered-store

from nedb.

@szwacz:

there is still no better in-process database than NEDB

Sorry for changing the subject (slightly), but I am surprised by this statement. Did you mean that there is no better pure-javascript in-process database?

SQLite and LevelDB both run in-process. If you're including them in your comparison, I'm curious by what metric you consider NeDB better (performance, scalability, security, usability, etc). I'm not intending to be disparaging or negative; I'm genuinely curious. I've watched NeDB for some time now and have looked at and fiddled with some of the code -- and I do like it, for a pure-Javascript DB -- but I have a hard time seeing it as being in the same league as C/C++ database engines like SQLite and LevelDB.

from nedb.

@szwacz thanks :) And good to see your work on this topic !

@prust yes of course, JS on a VM cannot compete with C, nor should it, the usecases are totally different :)

from nedb.

@prust Yes, pure-javascript'ness is the criterion I had also in mind.

from nedb.

Can NeDB handle large databases now? I'm a student and i'm working on an assignment for my DBMS class.

The topic is compare the performance of a traditional relational database to a new NoSQL database (Such as NeDB). Each of the databases must contain at least 100m records.

from nedb.

@hungpn2212: From the readme:

NeDB is not intended to be a replacement of large-scale databases such as MongoDB, and as such was not designed for speed. That said, it is still pretty fast on the expected datasets, especially if you use indexing. On a typical, not-so-fast dev machine, for a collection containing 10,000 documents

NeDB's performance, especially at scale, is not representative of typical NoSQL databases. I would strongly encourage you to use a more typical NoSQL database to get a more useful comparison. MongoDB is one of the most popular in the node community (Cassandra or Riak would be interesting points of comparison as well, though their performance characteristics may be quite different, I don't know off the top of my head)

from nedb.