Code Monkey home page Code Monkey logo

level-auto-index's Introduction

level-auto-index

Automatic secondary indexing for leveldb and subleveldown.

npm install level-auto-index

level badge npm Build Status dependencies Status devDependencies Status

Usage

var level = require('level')
var AutoIndex = require('level-auto-index')
var sub = require('subleveldown')
var keyReducer = AutoIndex.keyReducer

var db = level()

var posts = sub(db, 'posts', {valueEncoding: 'json'})
var idx = {
  title: sub(db, 'title'),
  length: sub(db, 'length'),
  tag: sub(db, 'tag')
}

// add a title index
posts.byTitle = AutoIndex(posts, idx.title, keyReducer('title'))

// add a length index
// append the post.id for unique indexes with possibly overlapping values
posts.byLength = AutoIndex(posts, idx.length, function (post) {
  return post.body.length + '!' + post.id
})

// Create multiple index keys on an index
posts.byTag = AutoIndex(posts, idx.tag, function (post) {
    if (!post || !post.tags || !Array.isArray(post.tags)) return
    return post.tags.map(function (tag) {
      return [tag, post.id].join('!')
    })
  }, { multi: true })

posts.put('1337', {
  id: '1337',
  title: 'a title',
  body: 'lorem ipsum',
  tags: [ 'foo', 'bar', 'baz' ]
}, function (err) {
  if (err) throw err

  posts.byTitle.get('a title', function (err, post) {
    if (err) throw err
    console.log('get', post)
    // => get: { id: '1337', title: 'a title', body: 'lorem ipsum' }

    posts.del('1337', function (err) {
      if (err) throw err
      posts.byTitle.get('a title', function (err) {
        console.log(err.name)
        // => NotFoundError
      })
    })
  })

  posts.byLength.createReadStream({
    start: 10,
    end: 20
  }).on('data', console.log.bind(console, 'read'))
  // => read { key: '1337', value: { id: '1337', title: 'a title', body: 'lorem ipsum' } }

  posts.byLength.createKeyStream({
    start: 10,
    end: 20
  }).on('data', console.log.bind(console, 'key'))
  // => key 1337

  posts.byLength.createValueStream({
    start: 10,
    end: 20
  }).on('data', console.log.bind(console, 'value'))
  // => value { id: '1337', title: 'a title', body: 'lorem ipsum' }
})

API

AutoIndex(db, idb, reduce, opts)

Automatically index a db level into the idb level using a reduce function that creates the index key. The db and idb levels should be in isolated key namespaces, either by being two different levels or mafintosh/subleveldown partitions. The db hook is mutated by hypermodules/level-hookdown to set up the prehooks used for indexing. Only db keys are stored as values to save space and reduce data redundancy.

Secondary returns an AutoIndex level that helps prune old index values, and automatically looks up source documents from db as you access keys on the AutoIndex level.

The reduce functions get the value of the put or batch operations. Make sure that this value has everything you need to create your index keys.

function reducer (value) {
  var idxKey = value.foo + '!' + value.bar
  return idxKey
}

Available opts:

{
  multi: false // Reducer returns an array of keys to associate with the primary key
}

Multi-key index's are for when you you want to write multiple index entries into an index. This is useful for 'tag' fields, where a document may have n tags per document, and you would like to index documents by 'tag'. When creating a multi-key index, your reducer must return an array of keys to index by.

AutoIndex#get(key, opts, cb)

Get the value that has been indexed with key.

AutoIndex#create{Key,Value,Read}Stream(opts)

Create a readable stream that has indexes as keys and indexed data as values.

AutoIndex#manifest

A level manifest that you can pass to multilevel.

AutoIndex.keyReducer(string)

A shortcut reducer for simplistic key indexing. You might need more than this.

function keyReducer (reducerString) {
  function keyRdc (value) {
    return value[reducerString]
  }
  return keyRdc
}

For a higher level api for creating secondary indexes see hypermodules/level-idx.

AutoIndex.keyReducer(string)

A shortcut reducer for simplistic multi-key indexing. You might need more than this.

function multiKeyReducer (multiFieldName, primaryKeyFieldName) {
  return function multiKeyrdc (document) {
    if (!document || !document[multiFieldName] || !Array.isArray(document[multiFieldName])) return
    return document[multiFieldName].map(function (tag) {
      return [tag, document[primaryKeyFieldName]].join('!')
    })
  }

AutoIndex#db

The level instance that we are indexing.

AutoIndex#idb

The level instance that we are using for the index.

See Also

  • hypermodules/level-hookdown - Simple levelup hooks implemented using instance method override of arbitrary levelups.
  • hypermodules/level-idx - Another high-level API for creating secondary leveldb indexes using level-auto-index and level-hookdown.

This module is a variant of

but aimed at decoupling the index storage fromt the indexd db and also being compatable with subleveldown. It came out of the work trying to make level-secondary compatable with subleveldown and level-sublevel. That work lives here: github.com/bcomnes/level-secondary/commit/9b2f914e53.

level-auto-index's People

Contributors

bcomnes avatar greenkeeper[bot] avatar greenkeeperio-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

level-auto-index's Issues

An in-range update of memdown is breaking the build 🚨

Version 1.2.5 of memdown just got published.

Branch Build failing 🚨
Dependency memdown
Current Version 1.2.4
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As memdown is β€œonly” a devDependency of this project it might not break production or downstream projects, but β€œonly” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this πŸ’ͺ

Status Details
  • ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 8 commits.

  • 1abc47b 1.2.5
  • 90e51b1 Merge pull request #69 from Level/greenkeeper/initial
  • f0b4b76 split up zuul on multiple matrix entries for each browser + test on latest only
  • 2610905 :arrow_up: bump abstract-leveldown
  • b9eb981 Update README.md
  • 9131113 docs(readme): add Greenkeeper badge
  • fb4a9cb chore(travis): whitelist greenkeeper branches
  • fe61dcd chore(package): update dependencies

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

create{Key/Read}Stream use the wrong keys

According to the documentation createKeyStream and createValueStream should use the indices as keys and the indexed values as values. However in reality it uses the keys of the original db on which the index has been created.

const autoLevelIdx  = require("level-auto-index");
const level         = require("level-mem");
async function printStream(text, stream, print) {
    await new Promise((res) => {
        stream.on("data", e => text += " | " + print(e));
        stream.on("end", res);
    });
    console.log(text);
}
(async function() {
    const db     = level();
    const byTitle = autoLevelIdx(db, level(), val => val);
    await Promise.all([db.put("A", "X"), db.put("B", "Y"), db.put("C", "Z")]);
    await printStream("ReadStream", byTitle.createReadStream(),  ({key, value}) => `${key}: ${value}`);
    await printStream("KeyStream",  byTitle.createKeyStream(),   (key) => key);
})();

According to the documentation this code should output:

ReadStream | X: X | Y: Y | Z: Z
KeyStream | X | Y | Z

In reality it generates the following output:

ReadStream | A: X | B: Y | C: Z
KeyStream | A | B | C

One-to-many index support

Does level-auto-index support aggregating multiple keys/object under a single index lookup?

For example, a chat app:
A message belongs to a room
I want an index by which I can look up all the messages in the same room

I have not been able to figure out how to do that with level-auto-index

promises support

Hi! I found this module recently and I think its great!
I was using level with promises (async/await) and then I try to use AutoIndex. But quickly noticed something went wrong with level-hookdown, I get this output after a db.put(...):

node_modules/level-hookdown/index.js:60
      cb(err)
TypeError: cb is not a function

So I try calling the level methods using callbacks and everything works.
Would you consider a PR for handling promises support? Maybe you could give some hints on the level-auto-index code base, where to start and that.

Thanks in advance!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.