If you have a json like this:

{
    "nested": {
        "a": "test"
    },
    "nested2": {
        "a": "test2"
    }
}

and you want to use EachKey with the following paths:

paths := [][]string{
    []string{"nested", "a"},
    []string{"nested2", "a"},
}

it returns the correct value only for the first path. For the second one, it returns an empty byte slice.

(Disclaimer: trying to keep suggestions concise and to the point, add IMHO to everything:) )

I'm too lazy to fork etc. anyways, validate the values:

func TestMoreInvalid(t *testing.T) {
    if _, _, e := GetBoolean([]byte(`{"c": txyz}`), "c"); e == nil {
        t.Errorf("Invalid value")
    }
    if _, _, e := GetBoolean([]byte(`{"c": fxyz}`), "c"); e == nil {
        t.Errorf("Invalid value")
    }
    if v, _, _, _ := Get([]byte(`{"c": "15\u00f8C"}`), "c"); !bytes.Equal(v, []byte("15°C")) {
        t.Errorf("Invalid value %s", v)
    }
    if v, _, _, _ := Get([]byte(`{"c": "\\\""}`), "c"); !bytes.Equal(v, []byte(`\"`)) {
        t.Errorf("Invalid value %s", v)
    }
}

Make the tests to table driven, it will be easier to add new ones. e.g.

    type T struct {
        In   string
        Path []string
        Out  string
    }

    var tests = []T{
        {`{"a": { "b": 1}, "c": 2}`, []string{"a", "b"}, `1`},
        {`{"a": { "b":{"c":"d" }}}`, []string{"a", "b", "c"}, `d`},
        // ...
    }

    for i, test := range tests {
        v, _, _, err := Get([]byte(test.In), test.Path...)
        if err != nil {
            t.Errorf("%02d expected %v got error %v", i, test.Out, err)
            continue
        }
        if !bytes.Equal(v, []byte(test.Out)) {
            t.Errorf("%02d expected %v got %v", i, test.Out, string(v))
        }
    }

PS: Run go vet on your code, your tests have invalid use of t.Errorf.

Is there any way to access the dataType's at all?

jsonparser.ObjectEach(value, func(key []byte, value []byte, dataType jsonparser.ValueType, offset int) error {

}

I want to be able to switch on the dataType but the ValueType doesn't appear to be exported?

Suppose we have following code:

    result := []Foo{}
    err := jsonparser.ArrayEach(data, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
        if err != nil {
            return
        }
        bars, err := extractBars(value)
        if err != nil {
            return
        }
        append(result, Foo{FieldOne: jsonparser.GetString(value, "target"), Bars: bars})
    })

We have no options to stop iterate over data even if our function extractBars will return an error.

I think there are two possible ways of fixing this issue.

1. If we don't care about backward compatibility
Just change cb signature to cb func(value []byte, dataType ValueType, offset int, err error) error.

2. If we respect backward compatibility
Then we could add counterpart for both there functions and name these like IterArray, IterKeys or something like this.

P.S.: In both cases I could contribute with PR.

I'm dealing with a large, flat json payload that has 100+ keys in a map.

I hit panics when using EachKey with the full list of json keys:

goroutine 1 [running]:
panic(0x4e8ac0, 0xc42000a0f0)
    /home/niek/go17/src/runtime/panic.go:500 +0x1a1
github.com/buger/jsonparser.EachKey(0xc421480000, 0x11ac, 0x11ac, 0xc420050e30, 0xc4200da000, 0xaf, 0x100, 0x0)
    /home/niek/workspace/src/github.com/buger/jsonparser/parser.go:237 +0x6a6
  ...snipped...

The problem code seems here:
https://github.com/buger/jsonparser/blob/master/parser.go#L236

If I'm reading it right, there is a limit of 64 keys per EachKey lookup before the int64 bitmask overflows.

Limit should probably be documented and code should return an error rather than panic.

Extra allocations are done when using an empty bytes.Buffer here in createInsertComponents(). bytes.Buffer uses a normal grow strategy of successively doubling in size. This could be avoided by counting the length of the needed keys and the final set value and starting with a Buffer that is about the right size.

It is not very efficient to fetch array elements using EachKey if you know the index of the element you want to fetch.

Not sure how that is possible in current implementation, would be good to complete this lib

I would like to use the lib for a project hosted on Google App Engine. Unfortunately, Google App Engine does not support the unsafe package.

Is it possible to refactor to not have to use the unsafe package? This would open jsonparser up for developers on Google App Engine.

Thanks!

It would be really nice if this package provided a Set(fieldName string, value []byte) or Set(fieldName string, value json.RawMessage) function. Right now we are processing huge amount of big json-line files. Each line has to be unmarshalled as map[string]json.RawMessage, then update just one filed and finally marshal it again. It would be really nice if we could update a value without unmarshalling!
What do you think?

I tried to run token := jsonparser.Get(responseData, "token") but got an error says "multiple-value jsonparser.Get() in single-value context".
I had to run token, _, _, err := jsonparser.Get(responseData, "token") for it to work.

I would like to know the number of Keys in an Object. currently I use ObjectEach and int++ to count it which is of course very slow. Is there a better way of doing it?

First, thank you for this package!

What do you think about adding a new helper function: GetInt(data []byte, keys ...string) (val int64, offset int, err error)?
I think this is a frequent use-case, so this function would be quite useful.

Also to increase the performances of GetNumber and GetInt, you could copy the strconv package from the stdlib into your package and update the signatures of ParseFloat and ParseInt to use []byte instead of string.
It would avoid allocating memory and make your package even faster.

I think this is the kind of optimization that is acceptable in a performance-oriented package like yours.

What do you think?

I guess you want to run the benchmarks on the same machine.
link to the project: a8m/djson

Thanks for this great project 😃

hi
I am following the exception when dealing with json containing special characters

Example json:
{"name":"test","sql":"select `name`\ from tables"}

Panic:
Value looks like Number/Boolean/None, but can't find its end: ',' or '}' symbol

Currently we have :

EachKey
ArrayEach
ObjectEach

ArrayEach and ObjectEach almost do same function, and have same parameters, but the function declarations/def are bit different

Suggest overriding EachKey with KeyEach (or override to EachObject and EachArray whatever sounds more logial) and make the function declaration/parameters as close as possible

Would be greate to have this marked as to-do, so whenever I have time i'll get back to it

@buger hello, nice package idea, keep up a good work!

Just tried example from readme and got a couple of errors, here is code to reproduce:

package main

import (
    "fmt"

    "github.com/buger/jsonparser"
)

func main() {

    data := []byte(`{
  "person": {
    "married": true
  },
  "company": {
    "name": "Acme",
    "size": 109
  }
}`)

    // If the key doesn't exist it will throw an error
    var size float64 = 0
    if value, _, err := jsonparser.GetNumber(data, "company", "size"); err == nil {
        size = value
    } else {
        fmt.Println(err.Error())
    }
    fmt.Println(size)

    var married bool
    if value, _, err := jsonparser.GetBoolean(data, "person", "married"); err == nil {
        married = value
    } else {
        fmt.Println(err.Error())
    }
    fmt.Println(married)
}

Output:

strconv.ParseFloat: parsing "109\n  ": invalid syntax
0
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/buger/jsonparser.GetBoolean(0xc820058070, 0x61, 0x70, 0xc82003bf28, 0x2, 0x2, 0x0, 0x26, 0x0, 0x0)
    /private/var/www/different/go/gopath/src/github.com/buger/jsonparser/parser.go:286 +0x28a
main.main()
    /private/var/www/different/go/gopath/src/github.com/centrifugal/parse.go:30 +0x2a9
exit status 2

Seems like

1. some values need proper trimming
2. I think there must be v[0] on this line

Also example is broken, instead of

// If the key doesn't exist it will throw an error
size := 0
if value, _, err := jsonparser.GetNumber(data, "company", "size"); err != nil {
  size = value
}

There must be sth like this I suppose:

// If the key doesn't exist it will throw an error
var size float64 = 0
if value, _, err := jsonparser.GetNumber(data, "company", "size"); err == nil {
  size = value
}

Also I found confusing return in tests

Sorry for not sending pr!

The following code panics. It should err.

All credit for finding the bug goes to the excellent go-fuzz library.

package main

import (
	"fmt"

	"github.com/buger/jsonparser"
)

func main() {
	data := "{}\"\":"
	_, err := jsonparser.GetString([]byte(data), "type")
	if err != nil {
		fmt.Println(err.Error())
	}
}

The issue came from email.
Here is example payload https://gist.github.com/buger/c16a314968186c8ef857a215487127f1 (vk.com API)

Failing command: jsonparser.Get(data, " Name","Pid","Photo")

Hi, according to documentation the Get API should return when key is not found. But I found that it is not the case for the following example:

package main

import "fmt"
import "github.com/buger/jsonparser"

func main() {
        data := []byte(`{
  "person": {
    "avatars": [
      { "url": "https://a.b.com", "type": "thumbnail" }
    ]
  }
}`)

       // ask for non-existing key in avatars sub-structure
        keys := []string{"person", "avatars", "[0]", "bla"}
        val, dType, _, err := jsonparser.Get(data, keys...)
        fmt.Println(string(val), dType, err)
}

it returns

{ "url": "https://a.b.com", "type": "thumbnail" } object <nil>

Could you please fix the problem.
Thanks,
Valentin.

When Geting strings that contains characters within the upper part of Basic Multilingual Plane (BMP): {"a": "some \uFF11 string"} this will fail silently (return "", nil).

A JSON text containing the two characters \n (bytes 0x5C and 0x6E) is not translated to a new line (byte 0x0A).

This is explained for example in http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf in page 4: "There are two-character escape sequence representations of some characters." with the following list: ", , /, \b, \f, \n, \r and \t

Could you add a msgPack support?)

Just suggestion, it will be awesome,
the same functional but for msgpack;)

Here is a code to reproduce

package main

import (
	"github.com/buger/jsonparser"
	"fmt"
)

func main() {
	body := []byte(`["api",["Application programming interface"],["In computer programming, an application programming interface (API) is a set of subroutine definitions, protocols, and tools for building application software."],["https://en.wikipedia.org/wiki/Application_programming_interface"]]`)
	jsonparser.ArrayEach(body, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
		a, err := jsonparser.GetString(value, "[0]")
		if err == nil {
			fmt.Println(a)
		} else {
			fmt.Println(err)
		}
	})
}

The JSON comes directly from wikipedia. The first element of the array is not an array so I get why "Key path not found" is correct in that case but the other 3 are arrays with single element which is a string. I get "Unknown value type" for those.

Hello,

I have the following JSON and I want to be able to parse it using jsonparser. The standard JSON unmarshalling is crazy slow!

["A9DDB0",{"altitude":14675,"heading":105,"vert_rate":-1408,"speed":328},{"sharecode":"newclientdev","local_ip":"192.168.1.100"},1481276460]

I can't seem to get to the elements by index at all.

I've tried using

jsonparser.ArrayEach(rawJSON, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
		fmt.Printf("%v:%v \n", offset, dataType)
})

Which does iterate them all but it doesn't seem to say the index, am I missing something there? Also then how would I access say the altitude from the second element?

I'm using jsonparser, to parse a json structure with a more or less unknown structure. I have no way of knowing if a ValueType number is a float or an int, and thus if I should call ParseFloat or ParseInt.

would it make sense to split the ValueType number into Integer and Float?

for now I will solve this by running trough the bytearray and check for a "." char, but since the tokenizer already went over this it would be nice if this would be supported from the lib itself.

thank you for helping if you reply to this

We need to make sure that it parse any json.

Does anyone knows good source of json specs?

jsonparser.GetInt(v,"value") v={"value":1.9863169E7}
error :0 Value looks like Number/Boolean/None, but can't find its end: ',' or '}' symbol

When Get() is called on a key that does not exist in the JSON, it currently returns NotExist and an error. I personally would much prefer it return NotExist and no error. That way, one can differentiate between the three possible cases: key found, key not found, and malformed JSON/other error. It doesn't seem like simply not finding a key is really an "error".

I can make a PR updating all of the tests. Would that be OK?

I want to first thank you, @buger, for your work on this library. Looking up a few JSON key paths in large JSON blobs is a significant bottleneck in a project I'm working on, and your library could give us a big speedup without changing our data format.

Unfortunately, I've discovered an issue in Get(): when searching for a key, Get() may locate that key outside the current JSON object. Here is an example test case that breaks (written using check.v1):

package jsonparser_test

import (
    "github.com/buger-jsonparser"
    . "gopkg.in/check.v1"
    "testing"
)

func (s *JsonParserTests) TestJsonParserSearchBleed(c *C) {
    killer := []byte(`{
      "parentkey": {
        "childkey": {
          "grandchildkey": 123
        },
        "otherchildkey": 123
      }
    }`)

    var jtype int

    _, jtype, _, _ = jsonparser.Get(killer, "childkey")
    c.Assert(jtype, Equals, jsonparser.NotExist) // fails, returns data parentkey.childkey

    _, jtype, _, _ = jsonparser.Get(killer, "parentkey", "childkey", "otherchildkey")
    c.Assert(jtype, Equals, jsonparser.NotExist) // fails, returns data parentkey.otherchildkey
}

// Boilerplate
func Test(t *testing.T) { TestingT(t) }
type JsonParserTests struct{}
var _ = Suite(&JsonParserTests{})

The issue is that Get() uses bytes.Index() to find the next key it's looking for, but only validates it by checking that it is surrounded by double quotes and followed by a colon. In particular, it does not check whether it has crossed an unmatched sequence of braces, which would indicate transitioning into another JSON object level.

I don't have a great suggestion as to how to fix this, sadly. Best of luck.

Hi,

how I can get the value of this key "result" from
[[{"result":true}]]

thank you!

Hello, i was wondering if there is a way to remove html escaped characters while parsing a json as standard lib does. i can't find this in documentation.

package main

import (
	"encoding/json"
	"fmt"

	"github.com/buger/jsonparser"
)

func main() {
	e := []byte(`{ "url": "https:\/\/example.com\/" }`)
	url, _, _, _ := jsonparser.Get(e, "url")
	var parsed map[string]interface{}
	json.Unmarshal(e, &parsed)

	fmt.Println("with standard json:", parsed["url"].(string))
	fmt.Println("with jsonparser:", string(url))
}

output

with standard json: https://example.com/
with jsonparser: https:\/\/example.com\/

Escapes can appear in both keys and values in json. The code linked to just compares the raw byte values. See eg https://tools.ietf.org/html/rfc7159#section-8.3

Hello,

A quick example below.

package main

import (
	"strconv"
	"fmt"

	"github.com/buger/jsonparser"
)

func main() {
	var err error
	js := []byte(`{"key1":1, "key2":2,"key3":3,"key4":4}`)

	for n := 0; n < 3; n++ {
		nstr := strconv.FormatInt(int64(n), 10)
		key := "key"+nstr
		js = jsonparser.Delete(js, key)
		js, err = jsonparser.Set(js, []byte(nstr), key)
		if err != nil {
			panic(err)
		}
	}

	// prints { "key3":3,"key4":4,"key0":0,"key1":1,"key2":2}
	fmt.Println(string(js))

	for n := 0; n < 3; n++ {
		js = jsonparser.Delete(js, "key4")
		js, err = jsonparser.Set(js, []byte("four"), "key4")
		if err != nil {
			panic(err)
		}
	}

	// prints { "key3":3,"key0":0,"key1":1,"key2":2fourfour,"key4":four}
	fmt.Println(string(js))
}

Following code would first result in an err, then would panic:

package main

import (
	"fmt"
	"github.com/buger/jsonparser"
)

func main() {
	data := `{"id":`

	jsonparser.EachKey([]byte(data), func(idx int, value []byte, vt jsonparser.ValueType, currentErr error) {
		if currentErr != nil {
			fmt.Printf("Error: %v\n", currentErr)
			return
		}
		fmt.Printf("idx: %d value: %s\n", idx, string(value))
	}, []string{"id"}, []string{"method"}, []string{"params"})
}

Maybe this is similar to #100?

Indexing with Set when the path does not already exist is not supported. For example, if you attempt the example in the docs:

jsonparser.Set([]byte(`{}`), []byte(`"http://github.com"`), "person", "avatars", "[0]", "url")

returns

{"person":{"avatars":{"[0]":{"url":"http://github.com"}}}}

[
{
"first_name": "Admin",
"last_name": "Account",
"role": "org_user",
"id": 1
},
{
"first_name": "Joe",
"last_name": "Smith",
"role": "admin",
"id": 2
}
]

I've tried ArrayEach and EachKey but [0] doesn't find anything and I can't seem to find a clear example.

Any pointers would be greatly appreciated. If push comes to shove I'll wrap the response in a dictionary.

I'm working on allegro/marathon-consul#140. I need to handle JSON events with size from 200B to 8MB. There are 30 types of events but I'm interesting only with two. I need fast way to extract one field (eventType). I think I can do it fast with bytes.Contains but it turns out it's slower than JSON parsing. Then I try jsonparser and it's also slower.

Here are my benchmarks: https://gist.github.com/janisz/9db4d50fc09f2eba81781af8dbc26a03
Maybe I'm doing something wrong.

I see support was added for this, that's great and thanks.

I can't however seem to get it to work properly.

Consider the following JSON.

{"metadata":["5666473ds52ca",50.11,-2.12,117.476220,false,false,"3.6.185",8]}

I want to read out the 5666473ds52ca value and am using the following code but it returns "Key path not found". Any idea what I am doing wrong here?

code, err := jsonparser.GetString(packet.payload, "metdata", "[0]")
if err != nil {
		log.Println(err)
	}

When trying to parse a slightly malformed JSON object, trying to access a value with a path of more than one component triggers an apparently inrecoverable internal runtime error within the parser code.
I would have expected to obtain a parse error and have an opportunity to handle it gracefully within my own code.

Example code:

package main

import (
	"bufio"
	"fmt"
	"os"

	"github.com/buger/jsonparser"
)

type Entry struct {
	Timestamp    string
	PktsToClient int64
}

var evekeys = [][]string{
	[]string{"timestamp"},
	[]string{"flow", "pkts_toclient"},
}

func main() {

	e := Entry{}
	scanner := bufio.NewScanner(os.Stdin)

	for scanner.Scan() {
		json := scanner.Bytes()
		jsonparser.EachKey(json, func(idx int, value []byte, vt jsonparser.ValueType, err error) {
			switch idx {
			case 0:
				if err == nil {
					e.Timestamp = string(value[:])
				}
			case 1:
				if err == nil {
					e.PktsToClient, _ = jsonparser.ParseInt(value[:])
				}
			}
		}, evekeys...)

		fmt.Println(e)
	}
}

Example input:

{"timestamp":"2017-03-06T09:10:09.002473+0000","flow_id":42,"event_type":"flow","src_ip":"0.0.0.0","src_port":23,"dest_ip":"0.0.0.0","dest_port":23,"proto":"UDP","aemuse":1638400,"reassembly_memuse":12332832},"detect":{"alert":0},"app_layer":{"flow":{"http":0,"ftp":0,"smtp":0,"tls":0,"ssh":0,"imap":0,"msn":0,"smb":0,"dcerpc_tcp":0,"dns_tcp":0,"failed_tcp":0,"dcerpc_udp":0,"dns_udp":0,"failed_udp":0},"tx":{"http":0,"smtp":0,"tls":0,"dns_tcp":0,"dns_udp":0}},"flow_mgr":{"closed_pruned":0,"new_pruned":0,"est_pruned":0,"bypassed_pruned":0,"flows_checked":0,"flows_notimeout":0,"flows_timeout":0,"flows_timeout_inuse":0,"flows_removed":0,"rows_checked":65536,"rows_skipped":65536,"rows_empty":0,"rows_busy":0,"rows_maxlen":0},"dns":{"memuse":0,"memcap_state":0,"memcap_global":0},"http":{"memuse":0,"memcap":0}}}

Example run:

$ cat problem.json  | ./jsonparser-bug
panic: runtime error: index out of range

goroutine 1 [running]:
panic(0x49f920, 0xc42000a140)
	/usr/lib/go-1.7/src/runtime/panic.go:500 +0x1a1
github.com/buger/jsonparser.EachKey(0xc42008a000, 0x32d, 0x1000, 0xc42003de78, 0x517e80, 0x2, 0x2, 0x200)
	/home/satta/golang/src/github.com/buger/jsonparser/parser.go:292 +0xe66
main.main()
	/home/satta/golang/src/github.com/satta/jsonparser-bug/main.go:39 +0x139

Is the benchmark Time/op unit in ms? microseconds? nanoseconds? It's not clear from the README.
Anyway, great job in providing a good JSON parser for Go 👍

Hello!

Would it be possible to add the possibility to set the value for each element in the array like this:
jsonparser.Set(jsonBody, []byte("\"\""), "issuesData", "issues", "[]", "extraFields", "[]", "html")

Thanks in advance,
Nighthawk

I think I have root cause outlined in comments below. Here is a nice repro case:

This code should print entries for both "potato_id" and "created" keys

package main

import (
        "fmt"
        "github.com/buger/jsonparser"
)

func main() {

        json := []byte(`{"potato_id": "6", "created": "2016-07-19 15:23:00"}`)

        paths := [][]string{
                []string{"potato_id"},
                []string{"created"},
        }

        jsonparser.EachKey(json, func(pathIdx int, value []byte, dtype jsonparser.ValueType, err error) {
                fmt.Printf("found %s -> %s\n", paths[pathIdx], string(value))
        }, paths...)
}

But I only see potato_id:

nieksand$ ./bugrepro 
found [potato_id] -> 6

But if I comment out the "potato_id" path, then I suddenly see the "created" key.

...snip...
        paths := [][]string{
//                []string{"potato_id"},
                []string{"created"},
        }
...snip...

nieksand$ ./bugrepro 
found [created] -> 2016-07-19 15:23:00

Unless I'm misunderstanding the purpose of EachKey, I would expect the original program to print both entries.

How to parse this ?
[{"resultNumber": "2"}]
use ArrayEach ?

thx~

Would it be possible to use the parser to get the JSON's Structure?

I know there are existing packages out there, but all that I tried can't keep the original JSON's Structure's order. I.e., the output structure are all sorted by keys, which is both a blessed and cursed at the same time.

Since this parser keeps the original order, it seems to be the only candidate that is able to do it.

Thanks

The ArrayEach function will not give an error if the parsing of one of the array elements fails.
The parse err is passed to the callback function, and that's it. (As stated in #53 the callback function is powerless to act on this error)

parser.go#L585

if e != nil {
    break
}

It just breaks out of the iterating loop and just returns as usual. I would expect the ArrayEach function to also signal an error in this case.

EDIT: and yes there are ways to bypass this in your code but still seems like a bug

can make a PR if required...

jsonparser.Delete([]byte(`{"a": {"b": 1}, "b": 2}`), "b")

Expect:
output: {"a": {"b": 1}}

Actual:
output: {"a": {}

Under what license this project is?

Not sure what is going on, but Get doesn't work for such simple JSON:

package main

import (
	"fmt"

	"github.com/buger/jsonparser"
)

func main() {
	json := []byte(`{"fiz":"fuz","foo":{"bar":"baz"}}`)

	value, _, _, err := jsonparser.Get(json, "fiz", "bar")

	if err != nil {
		panic(err)
	}

	fmt.Println(string(value))
	// baz
}

It should return an error instead

Consider "[\n]". This is a valid empty Array in JSON, but it is parsed as having one element, with dataType Unknown.

The following test demonstrates the problem:

func TestParseArrayWithWhitespace(t *testing.T) {
	data := []byte("[\n]")
	_, _ = ArrayEach(data, func(value []byte, dataType ValueType, offset int, err error) {
		t.Errorf("ArrayEach([]byte(%q), ...) called callback with an empty array: value=[]byte(%q), dataType=%v, offset=%d, err=%v", data, value, dataType, offset, err)
	})
}

This test should pass, but instead it fails:

--- FAIL: TestParseArrayWithWhitespace (0.00s)
	parser_test.go:1130: ArrayEach([]byte("[\n]"), ...) called callback with an empty array: value=[]byte(""), dataType=unknown, offset=2, err=Unknown value type