Code Monkey home page Code Monkey logo

json-mask's Introduction

JSON Mask Build Status NPM version js-standard-style

This is a tiny language and an engine for selecting specific parts of a JS object, hiding/masking the rest.

var mask = require('json-mask');
mask({ p: { a: 1, b: 2 }, z: 1 }, 'p/a,z'); // {p: {a: 1}, z: 1}

The main difference between JSONPath / JSONSelect and this engine is that JSON Mask preserves the structure of the original input object. Instead of returning an array of selected sub-elements (e.g. [{a: 1}, {z: 1}] from example above), it filters-out the parts of the object that you don't need, keeping the structure unchanged: {p: {a: 1}, z: 1}.

This is important because JSON Mask was designed with HTTP resources in mind, the structure of which I didn't want to change after the unwanted fields were masked / filtered.

If you've used the Google APIs, and provided a ?fields= query-string to get a Partial Response, you've already used this language. The desire to have partial responses in my own Node.js-based HTTP services was the reason I wrote JSON Mask.

For express users, there's an express-partial-response middleware. It will integrate with your existing services with no additional code if you're using res.json() or res.jsonp(). And if you're already using koa check out the koa-json-mask middleware.

This library has no dependencies. It works in Node as well as in the browser.

Note: the 1.5KB (gz), or 4KB (uncompressed) browser build is in the /build folder.

Syntax

The syntax is loosely based on XPath:

  • a,b,c comma-separated list will select multiple fields
  • a/b/c path will select a field from its parent
  • a(b,c) sub-selection will select many fields from a parent
  • a/*/c the star * wildcard will select all items in a field

Take a look at test/index-test.js for examples of all of these and more.

Grammar

     Props ::= Prop | Prop "," Props
      Prop ::= Object | Array
    Object ::= NAME | NAME "/" Prop
     Array ::= NAME "(" Props ")"
      NAME ::= ? all visible characters except "\" ? | EscapeSeq | Wildcard
  Wildcard ::= "*"
 EscapeSeq ::= "\" ? all visible characters ?

Examples

Identify the fields you want to keep:

var fields = 'url,object(content,attachments/url)';

From this sample object:

var originalObj = {
  id: 'z12gtjhq3qn2xxl2o224exwiqruvtda0i',
  url: 'https://plus.google.com/102817283354809142195/posts/F97fqZwJESL',
  object: {
    objectType: 'note',
    content:
      'A picture... of a space ship... launched from earth 40 years ago.',
    attachments: [
      {
        objectType: 'image',
        url: 'http://apod.nasa.gov/apod/ap110908.html',
        image: { height: 284, width: 506 }
      }
    ]
  },
  provider: { title: 'Google+' }
};

Here's what you'll get back:

var expectObj = {
  url: 'https://plus.google.com/102817283354809142195/posts/F97fqZwJESL',
  object: {
    content:
      'A picture... of a space ship... launched from earth 40 years ago.',
    attachments: [
      {
        url: 'http://apod.nasa.gov/apod/ap110908.html'
      }
    ]
  }
};

Let's test that:

var mask = require('json-mask');
var assert = require('assert');

var maskedObj = mask(originalObj, fields);
assert.deepEqual(maskedObj, expectObj);

Escaping

It is also possible to get keys that contain ,*()/ using \ (backslash) as escape character.

{
  "metadata": {
    "labels": {
      "app.kubernetes.io/name": "mysql",
      "location": "WH1"
    }
  }
}

You can filter out the location property by metadata(labels(app.kubernetes.io\/name)) mask.

NOTE: In JavaScript String you must escape backslash with another backslash:

var fields = 'metadata(labels(app.kubernetes.io\\/name))'

Partial Responses Server Example

Here's an example of using json-mask to implement the Google API Partial Response

var http = require('http');
var url = require('url');
var mask = require('json-mask');
var server;

server = http.createServer(function(req, res) {
  var fields = url.parse(req.url, true).query.fields;
  var data = {
    firstName: 'Mohandas',
    lastName: 'Gandhi',
    aliases: [
      {
        firstName: 'Mahatma',
        lastName: 'Gandhi'
      },
      {
        firstName: 'Bapu'
      }
    ]
  };
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify(mask(data, fields)));
});

server.listen(4000);

Let's test it:

$ curl 'http://localhost:4000'
{"firstName":"Mohandas","lastName":"Gandhi","aliases":[{"firstName":"Mahatma","lastName":"Gandhi"},{"firstName":"Bapu"}]}

$ # Let's just get the first name
$ curl 'http://localhost:4000?fields=lastName'
{"lastName":"Gandhi"}

$ # Now, let's just get the first names directly as well as from aliases
$ curl 'http://localhost:4000?fields=firstName,aliases(firstName)'
{"firstName":"Mohandas","aliases":[{"firstName":"Mahatma"},{"firstName":"Bapu"}]}

Note: a few more examples are in the /example folder.

Command Line Interface - CLI

When installed globally using npm i -g json-mask you can use it like:

json-mask "<fields>" <input> [<output>]

Examples

Stream from online resource:

curl https://api.myjson.com/bins/krrxw | json-mask "url,object(content,attachments/url)"

Read from file and write to output file:

json-mask "url,object(content,attachments/url)" input.json > output.json

Read from file and print redirect to file:

json-mask "url,object(content,attachments/url)" input.json > output.json

CDN

unpkg

  • https://unpkg.com/json-mask/build/jsonMask.js
  • https://unpkg.com/json-mask/build/jsonMask.min.js

License

MIT

json-mask's People

Contributors

antialiasis avatar dependabot-preview[bot] avatar fixe avatar luisfarzati avatar merrihew avatar mochja avatar nemtsov avatar nunofgs avatar pke avatar willfarrell avatar yamiodymel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

json-mask's Issues

Unable to figure out how to extract fields from unknown keys

It's pretty easy to work with known key names, but I have an object, which is a combination of readdir and fs.stat

{
    'index.html': {
        dev: 11111111,
        mode: 11111,
        nlink: 1,
        uid: 111,
        gid: 11,
        rdev: 1,
        blksize: 1111,
        ino: 1111111,
        size: 11111,
        blocks: 111,
        atime: Sun Sep 07 2014 19:31:39 GMT-0700 (PDT),
        mtime: Sun Sep 07 2014 14:58:54 GMT-0700 (PDT),
        ctime: Sun Sep 07 2014 14:58:54 GMT-0700 (PDT) }
    },
    "file1.txt"
    {
        dev: ...
    }
}

I've tried all possible things to extract only mtime and atime from the object, */*(mtime,atime)

Is it impossible? I couldn't find an example that covers this.

Thanks

How does Grammar "Object" connects with an Array?

The parser/lexer presents the following grammar as mentioned in README.

Props  ::= Prop   | Prop "," Props
Prop   ::= Object | Array
Object ::= NAME   | NAME "/" Object
Array  ::= NAME "(" Props ")"
NAME   ::= ? all visible characters ?

Have to say it's pretty helpful since I'm also implementing something the same in Golang.

type Sentence struct {
	Props []Prop `@@ ( "," @@ )*`
}

// Object | Array
type Prop struct {
	Array  *Array  `@@`
	Object *Object `| @@`
}

// @Ident | @Ident "/" Object
type Object struct {
	Name   string  `@Ident`
	Object *Object `( "/" @@ )*`
}

// @Ident "(" Prop ( "," Prop )* ")"
type Array struct {
	Name  string `@Ident`
	Prop  Prop `"(" @@ ( "," @@ )* ")"`
}

But there's a problem, once the Lexer goes into Object, it treats the rest of the things as Object.

Because something connects behind the Object can only be an Object, It can't be an Array.

 Object ::= NAME | NAME "/" Object

With this rule the following syntax will not work because C(D,E) is an Array

but something connects behind an Object can only be an Object.

A/B/C(D,E)

But Lexer sees

Object[A/
    Object[B/
        Object[C(D,E)]    โ† unexpected token "(" because it's not an Object syntax
    ]
]

EDIT: Or is A/B/C(D,E) an incorrect usage?

JIT support

For high performance solution it necessary.
Example:

const maskFn = compileMask('p/a,z');
maskFn({ p: { a: 1, b: 2 }, z: 1 }) // {p: {a: 1}, z: 1}

Mask not working as expected

I have the following json

{
  "tom": 3,
  "data1": {
    "first": "Hello",
    "second": {"third": "salude", "fourth": "nvm"}
   },
  "data2": {
    "first": "Bye",
    "second": {"third": "Cheers", "fourth": "nvm"}
   }
}

And I'm applying the follwing mask on it.
"data1(first,second/third),data2(first,second/third)"

I expected to see the,

{
  "data1": {
    "first": "Hello",
    "second": {"third": "salude"}
   },
  "data2": {
    "first": "Bye",
    "second": {"third": "Cheers"}
   }
}

but I'm only getting

{
  "data1": {
    "first": "Hello",
    "second": {"third": "salude"}
   }
}

Can you help me out?

Get all fields minus specific ones?

Instead of selecting specific fields from an object, is it possible to get all fields minus specific ones?

For example, something like this:

const readmeExample = { p: { a: 1, b: 2 }, z: 1 };
mask(readmeExample, 'p(-b),z'); // {p: {a: 1}, z: 1}

Haven't really thought through the syntax specifically, but am wondering if this is already possible or if there's any interesting in supporting something like this?

Getting params from a nested JSON without grouping the params

Lets say we have a JSON like below and I want to grab the language and experience under skills.
{
"name": "john",
"age": "30",
"skills": [{
"language": "Java",
"experience": "5"
},
{
"language": "CSharp",
"experience": "10"
}
]
}

If I group thee fields i request together like below i seem to be getting both language and experience
var fields = 'skills(experience,language)';

Response
{"skills":[{"experience":"5","language":"Java"},{"experience":"10","language":"C
Sharp"}]}

But If I don't group the fields and request them individually like below I only seem to get only the last requested parameter (in this case language). Ideally I should have got both experience and language

var fields = 'skills(experience),skills(language)';

Response
{"skills":[{"language":"Java"},{"language":"CSharp"}]}

Kindly take a look

a/(b,c) kills extraction of following keys

Hello, either I make a mistake in writing down the mask or there is a bug

I have the following json structure:

req_body = {
  title: 'This is the title',
  items: [
    {
      caption: 'Answer 1',
      open: true
    }, {
      caption: 'Answer 2',
      open: false
    }
  ],
  shuffle: true,
  number: 42,
  categories: [1, 2, 3, 4],
  should_not: 'be shown'
};

and my mask is called like this

console.log(mask(req_body, "title,items/(caption,open),number,categories,shuffle"));

what i expect would be the whole object from above except for the last key 'should_not'

What i get is

{ title: 'This is the title',
  items: 
   [ { caption: 'Answer 1', open: true },
     { caption: 'Answer 2', open: false } ] }

It seems that after items/(caption,open) the filtering stops, because if I change the order of the filter to

console.log(mask(req_body, "title,number,categories,shuffle,items/(caption,open)"));

it changes to the expected output:

{ title: 'This is the title',
  number: 42,
  categories: [ 1, 2, 3, 4 ],
  shuffle: true,
  items: 
   [ { caption: 'Answer 1', open: true },
     { caption: 'Answer 2', open: false } ] }

Json mask adding extra fields to the output while using * in the fields filter

"MyFileAttribute": [ {"attrib0": "eljeelj",
"attrib1": "qer",
"attrib2": DB",
"attrib3": "some Val",
"file": "some Val",
"size": 347,
"Platform": "some plateform"
}]
for the above json array when we use * in the fields filter we get
"MyFileAttribute": [
{
"__cachedRelations": {},
"__data": {
"attrib0": "eljeelj",
"attrib1": "qer",
"attrib2": DB",
"attrib3": "some Val",
"file": "some Val",
"size": 347,
"Platform": "some plateform",

    }
  }

While filtering using * it is adding "__cachedRelations": {}, and copying the filtered fields within the __data field attribute.

Incorrect masking

var mask = require('json-mask');

var originalObj = {
  id: 'z12gtjhq3qn2xxl2o224exwiqruvtda0i',
  url: 'https://plus.google.com/102817283354809142195/posts/F97fqZwJESL',
  object: {
    objectType: 'note',
    content: 'A picture... of a space ship... launched from earth 40 years ago.',
    attachments: [{
      objectType: 'image',
      url: 'http://apod.nasa.gov/apod/ap110908.html',
      image: {height: 284, width: 506}
    }]
  },
  provider: {title: 'Google+'}
};

var fields = '*/content';

var masked_data = mask(originalObj, fields);

console.log(JSON.stringify(masked_data));

Output:

{"object":{"content":"A picture... of a space ship... launched from earth 40 years ago."},"provider":{}}

The key "provider" should not be present in the output.

Nested arrays masking?

@nemtsov what do you think about nested arrays masking?

I trying to fix some issues in current version of json-mask and faced with this question. I can't find example for similar behaviour in google API.

Example:

let obj = {
  a: [
    { b: 0 },
    [{ b: 1 }],
    [[{ b: 2 }]],
    [[[{ b: 3 }]]]
  ]
}

Mask: a/b

Result (should json-mask fall in to nested arrays in the masking process?):

{
  a: [
    { b: 0 },
    [{ b: 1 }],
    [[{ b: 2 }]],
    [[[{ b: 3 }]]]
  ]
}

NOT mask?

It'd be nice to have the ability to use json-mask to say what fields should NOT be returned. For instance, if I have some JSON where every field except one should be included.

Is this possible currently? I didn't see anything in the API docs for it.

ES Module support

Great stuff, found this by accident, is it possible to support ES Module output for builds?

An Empty First param followed by a Valid param gives back empty response

Say I have a JSON like below
{
"name": "john",
"age": "30",
"skills": [{
"language": "Java",
"experience": "5"
},
{
"language": "CSharp",
"experience": "10"
}
]
}

When I try to ask for a field followed by an empty string like below i get response
'name,'

Response
{"name":"john"}

But when i have a empty string followed by a field that i want to grab i get an empty response.
Ideally we would have expected a response similar to above scenario

Response
{}

Kindly check.

Using dot instead of slash to specify nested properties

Hi

is it possible to access nested properties using dot instead of slash?

so instead of

mask({ p: { a: 1, b: 2 }, z: 1 }, 'p/a,z'); // {p: {a: 1}, z: 1}

would be

mask({ p: { a: 1, b: 2 }, z: 1 }, 'p.a,z'); // {p: {a: 1}, z: 1}

JsonMask '*' not returning 0 values

Thanks for your awesome plugin! Which i have used heavily in my app and node/hapi plugins. One problem that I noticed is '*' doesn't return all values.

JsonMask = require 'json-mask'
JsonMask { id:1, name: 'Bar 100', hits:0 }, '*'    # => { id: 1, name: 'Bar 100' } missing hits as it's 0
JsonMask { id:1, name: 'Bar 100', hits:0 }, 'hits'   # => { hits: 0 }

Here is my (odme)[https://github.com/tectual/odme] as a real life example, where you can see (here)[https://github.com/tectual/odme/blob/master/src/base.coffee] at line 151 the mask method is trying to get all attributes. But it fails.

Array of objects issue

Based on a simple ExpressJS-drive API, if my results are returned as an array of objects like so:

[
{
  // object
}

{
  // object
}

{
 // object
}
]

And each object contains a "username" field and that's all I want to return from each object, I get NULL from json-mask. Basically if I run http://localhost:3000/users/?fields=username I get NULL, but if I pass say, http://localhost:3000/users/?fields=0(username) THEN I get just the username from the first object in the array. I tried using a wildcard, http://localhost:3000/users/?fields=*(username) but with no luck. Based on the examples I 100% understand how to implement and use, but with the array of objects use-case (which is how my API is already structure) do you have any suggestions? Maybe I'm just being completely dense and I'm missing something.

Unexpected mask.filter behavior

let masked = mask({ 
  a: [
    { b: { name: 'foo' } },
    { b: 1 },
    { b: function noop() {} },
    { b: 0 },
    { b: null },
    { b: undefined }
  ]
}, 'a/b/name')

assert.deepEqual(masked, {
  a: [
    { b: { name: 'foo' } },   // ok
    {},                       // why empty? should be { b: 1 }
    { b: { name: 'noop' } },  // we really want to dive in to function object?
    { b: 0 },                 // ok
    { b: null },              // ok
    {}                        // ok
  ]
})

Mask not working after sub selection part

Hello,
I have encountered an issue processing following JSON:

const exampleObject = {
    rootObjA: {
        nestedArrayA: [
            {
                firstPropToKeep: 123,
                secondPropToKeep: 321
            },
            {
                firstPropToKeep: 435,
                secondPropToKeep: 111
            }
        ]
    },
    rootObjB: {
        nestedFieldB: "some value i want to remove"
    },
    rootArrayA: [
        {
            nestedProp: 321
        },
        {
            nestedProp: 888
        }
    ]
}

Given following functionally equal masks:

const workingMask = 'rootArrayA/*,rootObjA(nestedArrayA/*)';
const notWorkingMask = 'rootObjA(nestedArrayA/*),rootArrayA/*';

I expect following output for both of them:

const expectedObj = {
    rootObjA: {
        nestedArrayA: [
            {
                firstPropToKeep: 123,
                secondPropToKeep: 321
            },
            {
                firstPropToKeep: 435,
                secondPropToKeep: 111
            }
        ]
    },
    rootArrayA: [
        {
            nestedProp: 321
        },
        {
            nestedProp: 888
        }
    ]
}

But for some reason I encounter two different results:

const filteredObjWorking = mask(exampleObject, workingMask);
assert.deepEqual(filteredObjWorking, expectedObj); // fine
/**
{
  "rootArrayA": [
    {
      "nestedProp": 321
    },
    {
      "nestedProp": 888
    }
  ],
  "rootObjA": {
    "nestedArrayA": [
      {
        "firstPropToKeep": 123,
        "secondPropToKeep": 321
      },
      {
        "firstPropToKeep": 435,
        "secondPropToKeep": 111
      }
    ]
  }
}
 */

const filteredObjNotWorking = mask(exampleObject, notWorkingMask);
assert.deepEqual(filteredObjNotWorking, expectedObj); // error
/**
{
  "rootObjA": {
    "nestedArrayA": [
      {
        "firstPropToKeep": 123,
        "secondPropToKeep": 321
      },
      {
        "firstPropToKeep": 435,
        "secondPropToKeep": 111
      }
    ]
  }
}
*/

Could you please help me with that? It seems like a bug. The problem is that whatever is defined in mask after 'a(b,c) sub-selection' part for some reason is skipped.
Regards

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.