Hello,

Since https://github.com/ohler55/ojg/releases/tag/v1.10.4-develop we are running into an error while Marshal-ing a struct which includes the time.Time

A very tiny reproduce-able case

type MinimalStyle struct {
	CreatedAt time.Time
}

func TestJSON(t *testing.T) {
	testCase := MinimalStyle{
		CreatedAt: time.Unix(0, 0),
	}
	_, err := oj.Marshal(testCase)
	if err != nil {
		t.Fatalf("Marshal error: %s", err)
	}
}

The stacktrace that was shown in the debugger seems to indicate the error originates from this line of code:

Callstack:

Regards,
Gusted

In particular, "Bénédicte" is not parsed correctly.

Hi! We're going to start using this project as a dependency to reviewpad. It is our policy to try to sponsor all open source projects that we use. Is there a way to do this? FYI: We typically do GitHub sponsors.

When using the Pop() method of an alt.Builder on nested objects, I would expect to close the last object and move one level up. But it doesn't seem to be the case.

Code is probably better to explain that. I have modified a test:

func TestBuilderObject(t *testing.T) {
	var b alt.Builder

	err := b.Object()
	tt.Nil(t, err, "b.Object()")
	b.Pop()
	v := b.Result()
	tt.Equal(t, map[string]interface{}{}, v)

	b.Reset()
	tt.Nil(t, b.Result(), "b.Result() after reset")

	err = b.Object()
	tt.Nil(t, err, "first b.Object()")
	err = b.Value(true, "a")
	tt.Nil(t, err, "b.Value(true, a)")

	err = b.Object("b")
	tt.Nil(t, err, "second b.Object()")
	err = b.Value(false, "c")
	tt.Nil(t, err, "b.Value(false, c)")

	b.Pop()
	err = b.Value(nil, "d")
	tt.Nil(t, err, "b.Value(nil, d)")
	b.PopAll()

	v = b.Result()
	tt.Equal(t, map[string]interface{}{"a": true, "b": map[string]interface{}{"c": false}, "d": nil}, v)
}

And running the test gives:

$ go test -run TestBuilderObject
--- FAIL: TestBuilderObject (0.00s)
    util.go:31:
        expect: (map[string]interface {}) map[a:true b:map[c:false] d:<nil>]
        actual: (map[string]interface {}) map[a:true b:map[c:false d:<nil>]]
        github.com/ohler55/ojg/tt.Equal @ /home/nono/dev/ojg/tt/equal.go:26
        github.com/ohler55/ojg/alt_test.TestBuilderObject @ /home/nono/dev/ojg/alt/builder_test.go:67

Is it a bug or the expected behavior?

Hello!

I am very interested in ojg repo, and want import it as an util into my repo.
But I find that result type is always []interface{} even if the result should be a single value(like int/float/string..), I want to know why it is designed like this.
And because I don't have enough time to read the source code, I want to know if it is safe for me to parse the result as follows:
(1) Always treat the result as []interface{}
(2) if I need single value, for example: string, try to parse the first element of the result to string.
(3) if I need slice value, try to parse the whole result([]interface{}) to a typed slice(for example: []int64), if that failed, try to parse the first element of the result to the typed slice.

Anyway, thanks for your help！

func Test_parse(t *testing.T) {
	//jsonStr := `{"foo-bar": "test"}`

	_, err := jp.ParseString(`foo-bar`)
	if err != nil {
		t.Error(err)
	}
}

parse error at 4 in foo-bar

foo\-bar "foo-bar" and "foo\-bar" not work too.

I found - not in tokenMap, can supported it ?

oj.Unmarshal doesn't seem to use custom unmarshaling rules defined by implementing the json.Unmarshaler interface.

Simple playground example: https://play.golang.org/p/-cbMOSel_C6

Command github.com/ohler55/ojg/cmd/oj lacks package documentation.

Looking to use this library for querying a directory of 'Thing Descriptions'
The vocabulary of a TD includes an attribute named "@type". Is it possible to query such attribute?

According to the json-path-comparision, the filter expression with bracket notation and current object literal (for example: "$[?(@['@type']=="sensor")]" is supported by this library but this returns an empty result.

What did work was the bracket notation without the object literal, eg $[?(@['type']=="sensor)].

Btw, many thanks for this library. It just seems to work without the usual head banging :). Great job!

Hello @ohler55. Awesome library, we looked at several JSON path implementations and your work is just beautiful.

We came across a problem compiling Grafana with ojg as a dependency on 32-bit CPU due to usage of math.MaxInt64.

Simple example which reproduces this:

cd cmd/oj
GOOS=linux GOARCH=arm go build
# github.com/ohler55/ojg/jp
../../jp/get.go:414:8: constant 9223372036854775807 overflows int
../../jp/get.go:975:8: constant 9223372036854775807 overflows int
../../jp/node.go:263:8: constant 9223372036854775807 overflows int
../../jp/node.go:538:8: constant 9223372036854775807 overflows int
../../jp/parse.go:356:13: constant 9223372036854775807 overflows int
../../jp/parse.go:363:13: constant 9223372036854775807 overflows int
../../jp/slice.go:28:10: constant 9223372036854775807 overflows int

Related: golang/go#23086

I think in most cases here it's safe to switch using math.MaxInt – but I am not sure at the moment whether it's safe to switch everywhere throughout a library, maybe there is a need to explicitly use int64 type (for example, for BigLimit boundary).

When unmarshaling or recomposing to a struct, if a key matching a private field is found, the private field is attempted to be set resulting in a panic:

reflect: reflect.Value.Set using value obtained using unexported field

Playground demonstration: https://go.dev/play/p/r6oam713GJo

(*Parser).Parse function called will crash when the first parameter buf is "{\"\"\"\":0", the error trace is as follows

panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/ohler55/ojg/oj.(*Parser).add(0xc000096cf0, 0x10d5ca0, 0x1192120)
	/workspace/ojg/oj/parser.go:558 +0x107
github.com/ohler55/ojg/oj.(*Parser).parseBuffer(0xc000096cf0, 0xc000096cd0, 0x7, 0x20, 0x1, 0x1114140, 0x20)
	/workspace/ojg/oj/parser.go:538 +0x4d0a
github.com/ohler55/ojg/oj.(*Parser).Parse(0xc000096cf0, 0xc000096cd0, 0x7, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1114140, ...)
	/workspace/ojg/oj/parser.go:87 +0x4a5
github.com/ohler55/ojg/oj.ParseString(0xc000096e78, 0x7, 0x0, 0x0, 0x0, 0x1fa79b70, 0x1fa79b7000000000, 0x60370d9b, 0xc000096e98)
	/workspace/ojg/oj/oj.go:27 +0xee
github.com/ohler55/ojg.Fuzz(0xa321000, 0x7, 0x7, 0x3)
	/workspace/ojg/fuzz.go:6 +0x87
go-fuzz-dep.Main(0xc000096f70, 0x1, 0x1)
	go-fuzz-dep/main.go:36 +0x1b8
main.main()
	github.com/ohler55/ojg/go.fuzz.main/main.go:15 +0x52
exit status 2

When a JSONPath query uses wildcard or union indexing (.e.g $.array[*] or $.array[0,1]), array elements appear to be matched in reverse order. Here is a minimal reproducer:

bash$ /bin/cat bug_test.go 
package test

import (
        "testing"

        "github.com/stretchr/testify/assert"
        "github.com/ohler55/ojg/jp"
        "github.com/ohler55/ojg/oj"
)

func TestCase(t *testing.T) {
    obj, _ := oj.ParseString(`{
        "a":[
            {"x":1,"y":2,"z":3},
            {"x":2,"y":4,"z":6}
        ]
    }
    `)

    x, _ := jp.ParseString("$.a[*].y")
    ys := x.Get(obj)
    assert.Equal(t, []interface{}{2, 4}, ys)
}

bash$ go test
--- FAIL: TestCase (0.00s)
    bug_test.go:22: 
                Error Trace:    bug_test.go:22
                Error:          Not equal: 
                                expected: []interface {}{2, 4}
                                actual  : []interface {}{4, 2}
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,4 +1,4 @@
                                 ([]interface {}) (len=2) {
                                - (int) 2,
                                - (int) 4
                                + (int64) 4,
                                + (int64) 2
                                 }
                Test:           TestCase
FAIL
exit status 1
FAIL    _/mnt/Data/work/code/jp-bug     0.002s

There may be a bug of oj.ParseString:

func TestOjgParse(t *testing.T) {
	v, err := oj.ParseString("1,2,3\n")
	fmt.Println(v)
	fmt.Println(err)
	fmt.Println("----------")
	v, err = oj.ParseString("1,2,a\n")
	fmt.Println(v)
	fmt.Println(err)
}

It outputs:

=== RUN   TestOjgParse
1
<nil>
----------
<nil>
unexpected character 'a' at 1:5
--- PASS: TestOjgParse (0.00s)
PASS

the string '1,2,3\n' isn't to be a JSON either, but ParseString didn't return an error.

Hello, and thank you for this great library!

I'm trying to apply multiple JSONPaths in a JSON and I'm having a few difficulties, please help me out

Thanks a lot!

Use case

I'm using multiple JSONPaths to select/deselect parts of my datastructure that need processing (modifying, filtering, etc).

It basically looks like this:

type JsonPathSelector struct {
    select []jp.Expr
    deselect []jp.Expr
}

When it runs, it wraps all the matched JSON subtrees in "tags" that tells whenever that subtree is enabled or disabled:

type JsonPathSelectorTag struct {
    value   any
    selected bool
}

For instance, I use it like this:

input := map[string]any{
    "foo": []any{1, 2, 3},
    "bar": 4,
}
filter := {
   select: ["$.foo[0]"],
   deselect: ["$.foo"],
}
output := JsonPathFilter.AddTags(input)

expectedOutput := map[string]any{
    "foo": JsonPathSelectorTag{
        selected: false,
        value: []any{
            JsonPathSelectorTag{
                selected: true,
                value: 1,
            }, 
            2, 
            3
        },
    },
    "bar": 4,
}

Problem

My current implementation uses jp.Expr.Modify() serially to add tags:

func (f *JsonPathSelector) AddTags(data any) (any, error) {
	for _, expr := range f.select {
		data = expr.Modify(data, func(element any) (altered any, changed bool) {
			return JsonFilterTag{element, true}, true
		})
	}
	for _, expr := range f.deselect {
		data = expr.Modify(data, func(element any) (altered any, changed bool) {
			return JsonFilterTag{element, false}, true
		})
	}
	return data, nil
}

Unfortunately, since it modifies the data at each step, the processing order affects the output.

E.g., if $.a.b is executed before $.a both subtrees are tagged.
However if the order is reversed, $.a is tagged, but $.a.b (since .b won't match the JsonFilterTag)

Possible Solutions

I can think of 2 ways to work around this issue, and either one works fine with me. Or maybe there is something else I'm missing?

Supporting "Tag" objects:

Everything would work for me if JSONPath somehow ignored JsonPathSelectorTag. It seems reasonably to add a special structure where I can attach arbitrary data and that is also ignored by JsonPath

type Tag struct {
    contents   any
    tag any
}

This maps very well with my current implementation and seems easier to implement in the library

Supporting operations on multiple paths simultaneously

type MultiExpr []jp.Expr

multiExpr = ["$.foo[0]", "$.foo"]
data = multiExpr.Modify(data, func(element any, exprIndex []int) (altered any, changed bool) {
	return element, false
})

This seems a bit more complicate to implement, but also more flexible.
Note that Modify() would need an extra argument to specify which expressions matched each element.

Give this code:

	expr, err := jp.ParseString("$.vals[-3:]")
	if err != nil {
		panic(err)
	}

	result := expr.Get(map[string]interface{}{
		"vals": []int{0, 10, 20, 30, 40, 50},
	})
	fmt.Printf("result: %v\n", result)

the result is: [50 40 30]，but it should be [30 40 50]，evaluated at https://jsonpath.com/

version: v1.17.5

Look at the following. The results are clearly wrong. I am not sure if this is intended behaviour, but if so, it isn't documented.

package main

import (
	"fmt"

	"github.com/ohler55/ojg/oj"
)

func main() {
	type a struct {
		A string `json:"a"`
	}

	type b struct {
		B string `json:"b"`
	}

	aBytes, _ := oj.Marshal(a{
		A: "a",
	})
	bBytes, _ := oj.Marshal(b{
		B: "b",
	})

	fmt.Println(string(bBytes))
	fmt.Println(string(aBytes))

	//Output:
	//{"b":"b"}
	//{"b":"b"}
}

Does the JSONPath implementation within the OJG library support extracting multiple values from different keys within the same JSONPath query? I haven't been able to find a way to do this using the library, unfortunately. I've got a quick example I can provide if that helps.

Below is some example JSON data that contains information about an AWS S3 bucket.

{
		"accountId": "192547438240",
		"additionalDetails": {
		  "ResourceDescription": {
			"Identifier": "elasticbeanstalk-us-east-1-192547438240",
			"Properties": {
			  "BucketName": "elasticbeanstalk-us-east-1-192547438240",
			  "RegionalDomainName": "elasticbeanstalk-us-east-1-192547438240.s3.us-east-1.amazonaws.com",
			  "DomainName": "elasticbeanstalk-us-east-1-192547438240.s3.amazonaws.com",
			  "WebsiteURL": "http://elasticbeanstalk-us-east-1-192547438240.s3-website-us-east-1.amazonaws.com",
			  "DualStackDomainName": "elasticbeanstalk-us-east-1-192547438240.s3.dualstack.us-east-1.amazonaws.com",
			  "Arn": "arn:aws:s3:::elasticbeanstalk-us-east-1-192547438240"
			}
		  },
		  "TypeName": "AWS::S3::Bucket"
		},
		"region": "ap-northeast-3",
		"resource": {
		  "Identifier": "elasticbeanstalk-us-east-1-192547438240",
		  "Properties": "{\"BucketName\":\"elasticbeanstalk-us-east-1-192547438240\"}"
		},
		"type": "AWS::S3::Bucket"
	  }

I've written the following JSONPath query to extract this data:

$.additionalDetails.ResourceDescription.Properties[WebsiteURL,DualStackDomainName]

I've found using an online JSONPath tool (jsonpath.com) that this query when run against the previously provided sample data produces the following result:

[
  "http://elasticbeanstalk-us-east-1-192547438240.s3-website-us-east-1.amazonaws.com",
  "elasticbeanstalk-us-east-1-192547438240.s3.dualstack.us-east-1.amazonaws.com"
]

However, when using the OJG library I'm not getting any results back from this JSONPath query. Below is the sample code I'm using:

package main

import (
	"fmt"
	"reflect"

	"github.com/ohler55/ojg/jp"
	"github.com/ohler55/ojg/oj"
)

func main() {
	obj, _ := oj.ParseString(`{
		"accountId": "192547438240",
		"additionalDetails": {
		  "ResourceDescription": {
			"Identifier": "elasticbeanstalk-us-east-1-192547438240",
			"Properties": {
			  "BucketName": "elasticbeanstalk-us-east-1-192547438240",
			  "RegionalDomainName": "elasticbeanstalk-us-east-1-192547438240.s3.us-east-1.amazonaws.com",
			  "DomainName": "elasticbeanstalk-us-east-1-192547438240.s3.amazonaws.com",
			  "WebsiteURL": "http://elasticbeanstalk-us-east-1-192547438240.s3-website-us-east-1.amazonaws.com",
			  "DualStackDomainName": "elasticbeanstalk-us-east-1-192547438240.s3.dualstack.us-east-1.amazonaws.com",
			  "Arn": "arn:aws:s3:::elasticbeanstalk-us-east-1-192547438240"
			}
		  },
		  "TypeName": "AWS::S3::Bucket"
		},
		"region": "ap-northeast-3",
		"resource": {
		  "Identifier": "elasticbeanstalk-us-east-1-192547438240",
		  "Properties": "{\"BucketName\":\"elasticbeanstalk-us-east-1-192547438240\"}"
		},
		"type": "AWS::S3::Bucket"
	  }`)

	//
	x, _ := jp.ParseString("$.additionalDetails.ResourceDescription.Properties[WebsiteURL,DualStackDomainName]")
	ys := x.Get(obj)

	fmt.Println("ys =", ys)
	fmt.Println("reflect.TypeOf(ys) = ", reflect.TypeOf(ys))
}

Below is the output from running this program:

➜  test-ojg-jsonpath go build main.go
➜  test-ojg-jsonpath ./main
ys = []
reflect.TypeOf(ys) =  []interface {}
➜  test-ojg-jsonpath

Hi, I'm using your API to parse some json, hitting an error with
data like this:
"ECCENTRICITY": 8.26e-05,

the error is "invalid number at x:xxxxxx"

curious if that is an invalid number or if its a bug in the parsing? if I'm understanding the JSON spec, I think its a valid number but really not sure.

I'm using github.com/ohler55/ojg v1.2.0

many thanks.

Add a key alignment option or make it the default for the pretty format.

    {
      "colors": [
        { "color": "black",   "hex": "#000", "rgb": [ 0,   0,   0   ] },
        { "color": "red",     "hex": "#f00", "rgb": [ 255, 0,   0   ] },
        { "color": "yellow",  "hex": "#ff0", "rgb": [ 255, 255, 0   ] },
        { "color": "green",   "hex": "#0f0", "rgb": [ 0,   255, 0   ] },
        { "color": "cyan",    "hex": "#0ff", "rgb": [ 0,   255, 255 ] },
        { "color": "blue",    "hex": "#00f", "rgb": [ 0,   0,   255 ] },
        { "color": "magenta", "hex": "#f0f", "rgb": [ 255, 0,   255 ] },
        { "color": "white",   "hex": "#fff", "rgb": [ 255, 255, 255 ] }
      ]
    }

JSON Input:
{
"foo": "bar"
}

JSONPath Expression:
$[?(@.foo == "bar")]

Expected results (from Jayway or Gatling, courtesy of http://jsonpath.herokuapp.com/):
[
{
"foo" : "bar"
}
]

Results from calling expr.Get():
nil

Description

Zero values are being excluded from an expression's Get results. It seems this comes down to this line. However, this will also be the case for a few other code paths. The result is incorrect according to JSONPath.

I'm wondering if this is intentional or maybe it could be optional somehow. As it stands, the library isn't suitable for a use case it's otherwise perfect for. Thanks for your contribution to the community!

Reproduction

package test

import (
	"testing"

	"github.com/ohler55/ojg/jp"
	"github.com/stretchr/testify/assert"
)

type Bool struct {
	Value bool
}

func TestFalseValues(t *testing.T) {
	trueBool := Bool{true}
	falseBool := Bool{false}

	exp, err := jp.ParseString(`$.Value`)
	if err != nil {
		assert.NoError(t, err)
		return
	}

	trueResults := exp.Get(trueBool)
	assert.Len(t, trueResults, 1)
	if len(trueResults) == 1 {
		assert.Equal(t, true, trueResults[0])
	}

	falseResults := exp.Get(falseBool)
	assert.Len(t, falseResults, 1)
	if len(falseResults) == 1 {
		assert.Equal(t, false, falseResults[0])
	}
}

Expected

ok

Actual

--- FAIL: TestFalseValues (0.00s)
    simple_test.go:32: 
        	Error Trace:	simple_test.go:31
        	Error:      	"[]" should have 1 item(s), but has 0
        	Test:       	TestFalseValues
FAIL
FAIL	simple_test	0.010s
FAIL

I'm trying to parse the JSONPath $[?(@.price < 10)] in code by passing it to:

expr := jp.MustParseString("$[?(@.price < 10)]")

and reading each fragment in expr. But I'm unable to parse the filter expression in the path.

I know that filter expressions are first declared as Equation objects in code and passed into Filter objects, and the Filter objects encode these Equation objects in an internal data structure. But so far, I haven't found a way to convert the Filter objects back to Equation objects. Is there a way to do this?

When I parse an incomplete json like {"time":"2021-08-16 12:12:15","a":"5","b":"5" by oj.Parse it return a nil result and a nil error.

I'm unsure if this is expected behavior, so I wanted to ask before figuring out recomposers. By default, the standard library's json.Decode function will turn JSON numbers into float64s (ref) for unmarshals to map[string]interface{}, while OjG looks like it tries parsing as an int, and falls back to a float if unsuccessful.

jsoniter follows the same behavior of the standard library, and our code has later type checks on certain fields making any migration a bit more involved than a drop-in. See the below code snippet for details:

func TestUnmarshal(t *testing.T) {
	j := []byte(`{"plain":1, "full_decimal":1.0,
			"negative":-1, "full_negative":-1.0,
			"zero":0,  "full_zero":0.0}`)

	var stdlib, ojg map[string]interface{}
	err := json.Unmarshal(j, &stdlib)
	err2 := oj.Unmarshal(j, &ojg)

	if err != nil || err2 != nil {
		panic("oh no")
	}

	for key := range stdlib {
		if stdlib[key] != ojg[key] {
			stdType := reflect.TypeOf(stdlib[key])
			ojgType := reflect.TypeOf(ojg[key])
			if stdType != ojgType {
				t.Errorf("mismatch between stdlib type '%v' and ojg type '%v' on key '%v'", stdType.String(), ojgType.String(), key)
			}
		}
	}
}
/* returns:
mismatch between stdlib type 'float64' and ojg type 'int64' on key 'plain'
mismatch between stdlib type 'float64' and ojg type 'int64' on key 'negative'
mismatch between stdlib type 'float64' and ojg type 'int64' on key 'zero'
*/

i'm getting panic on bad jsonpath expression syntax: $[*].{}, expecting to get error not panic :)

json example:

[
  {
    "Name": "sds-sds",
    "BackendName": "sd",
    "ID": "i-0sdsd44c0",
    "PublicIP": "23.23.23.23",
    "PrivateIP": "12.12.2.2",
    "Status": "Running",
    "Type": "r5d.large"
  }
]

// example of struct
type Instance struct {
  ID          string
  BackendName string
  Name        string
  Type        string
  Status      string
  PrivateIP   string
  PublicIP    string
}

items := make([]*Instance, 0)

...

expr, err := jp.ParseString("$[*].{}")
if err != nil {
	return err
}

out, err := oj.Marshal(items), &oj.Options{Indent: 2, KeyExact: true})
if err != nil {
	return err
}

panic: reflect: call of reflect.Value.CanInterface on zero Value

goroutine 1 [running]:
reflect.Value.CanInterface(...)
        /usr/local/opt/go/libexec/src/reflect/value.go:1005
github.com/ohler55/ojg/jp.Expr.reflectGetChild(0xc0004f22c0, 0x3, 0x4, 0x2f01e60, 0xc0004e43f0, 0xc000494878, 0x2, 0x4, 0x2e6760f, 0x4)
        /Users/shareed2k/go-workspace/pkg/mod/github.com/ohler55/[email protected]/jp/get.go:1145 +0x38d
github.com/ohler55/ojg/jp.Expr.Get(0xc0004f22c0, 0x3, 0x4, 0x3102880, 0xc0004db700, 0x4, 0x0, 0x0)
        /Users/shareed2k/go-workspace/pkg/mod/github.com/ohler55/[email protected]/jp/get.go:66 +0x66fe
github.com/shareed2k/honey/pkg/place/printers.Print(0xc00035fca0, 0x4, 0x4)
        /Users/shareed2k/golang_projects/honey/pkg/place/printers/printer.go:61 +0x759
github.com/shareed2k/honey/pkg/place/operations.Find(0x370d060, 0xc0000520a0, 0xc0004db440, 0x2, 0x2, 0x7ffeefbff1ed, 0x5, 0x0, 0x7ffeefbff1f5, 0x10, ...)
        /Users/shareed2k/golang_projects/honey/pkg/place/operations/find.go:75 +0x2d2
github.com/shareed2k/honey/cmd.glob..func1(0x441e5a0, 0xc000443d60, 0x0, 0x5, 0x0, 0x0)
        /Users/shareed2k/golang_projects/honey/cmd/root.go:44 +0xad
github.com/spf13/cobra.(*Command).execute(0x441e5a0, 0xc00004e250, 0x5, 0x5, 0x441e5a0, 0xc00004e250)
        /Users/shareed2k/go-workspace/pkg/mod/github.com/spf13/[email protected]/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x441e5a0, 0x1046ab7, 0x43b0c80, 0xc000000180)
        /Users/shareed2k/go-workspace/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/shareed2k/go-workspace/pkg/mod/github.com/spf13/[email protected]/command.go:895
github.com/shareed2k/honey/cmd.Execute()
        /Users/shareed2k/golang_projects/honey/cmd/root.go:53 +0x3b
main.main()
        /Users/shareed2k/golang_projects/honey/main.go:10 +0x25

Hi,
Thank you for this contribution, it's very helpful :)

I'm looking for a method that for a given rule is returning a list of the full paths it appears in the json.
For example, my json is:
{ "name": "John Smith", "male": true, "age": 35, "address": "New York", "null": null, "metadata": { "date": "2022-04-01T00:00:00.000Z", "count": "5", "name": "John Smith" }, "test": { "name": "John Smith" } }

and the rule is: "$.*.name"

So I want to have something like this: ["$.metadata.name","$.test.name"]

Is there any method that does this?

Thanks.

Currently Set() will insert any new values in lists and maps if they don't exist. Is there a way to call Set() so it only replaces existing matching items?

For example, this code:

expr, err := jp.ParseString("$..int")
expr.Set(document, 10)

with this document:

{
    "map": {
        "int": 4
    }
}

should only replace the value of "int" in the nested map:

{
    "map": {
        "int": 10
    }
}

Right now, Set() would insert a key-value pair of "int" and 10 in every matching location:

{
    "map": {
        "int": 10
    }
    "int": 10
}

When I need to unmarshal a date like this:

"2021-12-31"

I get:

"can only recompose a date.Date from a map[string]interface{}, not a string"

I Know that JsonPath syntax is not very strictly defined 😓, but is it possible to select weird field names (containing ' and \)?

My preliminary tests suggest it doesn't work. Am I doing something wrong?

func TestEscaping(t *testing.T) {
	data := map[string]any{
		`\`: "ok",
	}
	expr, err := jp.ParseString(`$['\\']`)
	assert.NoError(t, err)
	assert.Equal(t, []any{"ok"}, expr.Get(data))
}

Thanks!

Hello Mr Ohler!
first i would like to thank you for your amazing work on this parser, my team is very surprised with all the advantages we got by adopting this lib!

I would like to comment that we observed a point in the code where the order of the elements (in an array) is not being preserved, but I cannot say whether this behavior is correct or unexpected.

I saw that a few days ago you published a new release (v1.17.3) with a similar fix, but correcting the Filter, so I thought it was worth opening a ticket to comment on the same problem in Wildcard.

The section that seems unexpected to me is in file jp/get.go, on line 206 (tag v1.17.3). There, the x.reflectGetWild(tv) function is called and returns an inverted list of elements (inside it there is a comment saying that this is the desired behavior). It seems to me that that function is correct, it just needs to change the direction that the loop of the 206 line is traversed, in a similar way to the fix made in the Filter case.

Does what I'm proposing make sense?

oj.Marshal fails with the following error when called on structs with embedded pointer structs: reflect: NumField of non-struct type *<embedded-struct-type>. It works fine if the embedded struct isn't a pointer.

Playground example: https://play.golang.org/p/jow7wUE3nNV

Why are some characters being expanded (as the default output)?

{
  "6": {
    "morgan": "ZILFER",
    "summary": "This code is used to describe the zilfer.",
    "body": [
      "<p><b>Note:</b> Some of these zilfer types and accounts are only applicable to this platypus.</p>"
    ]
  }
}

When I run oj without switches, I get

{
  "6": {
    "morgan": "ZILFER",
    "summary": "This code is used to describe the zilfer.",
    "body": [
      "\u003cp\u003e\u003cb\u003eNote:\u003c/b\u003e Some of these zilfer types and accounts are only applicable to this platypus.\u003c/p\u003e"
    ]
  }
}

How do I stop it (without using sen)?

I didn't see anything mentioned in the documentation, but is it possible to evaluate the json path on a reader, or can you only evaluate the path on a byte array/string? I took a look at the cli code and it looks like the whole file must be read into memory first before it evaluates the expression on it. Is there any way around this?

Thanks

The following queries provide results that do not match those of other implementations of JSONPath
(compare https://cburgmer.github.io/json-path-comparison/):

• $[1:3]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["second", "third"]
  

  Actual output:

  [
    "second",
    "third",
    "forth"
  ]
  

• $[0:5]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "second", "third", "forth", "fifth"]
  

  Actual output:
  NOT_FOUND

• $[1:10]
  Input:

  [
    "first",
    "second",
    "third"
  ]
  

  Expected output:

  ["second", "third"]
  

  Actual output:
  NOT_FOUND

• $[-4:-4]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  []
  

  Actual output:

  [
    4
  ]
  

• $[-4:-3]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  [4]
  

  Actual output:

  [
    4,
    5
  ]
  

• $[-4:2]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  []
  

  Actual output:

  [
    4
  ]
  

• $[-4:3]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  [4]
  

  Actual output:

  [
    4,
    5
  ]
  

• $[:2]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "second"]
  

  Actual output:

  [
    "first",
    "second",
    "third"
  ]
  

• $[-4:]
  Input:

  [
    "first",
    "second",
    "third"
  ]
  

  Expected output:

  ["first", "second", "third"]
  

  Actual output:
  NOT_FOUND

• $[0:3:1]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "second", "third"]
  

  Actual output:

  [
    "first",
    "second",
    "third",
    "forth"
  ]
  

• $[0:4:2]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "third"]
  

  Actual output:

  [
    "first",
    "third",
    "fifth"
  ]
  

• $[]
  Input:

  {
    "": 42,
    "''": 123,
    "\"\"": 222
  }
  

  Expected output:

  NOT_SUPPORTED
  

  Actual output:

  [
    {
      "": 42,
      "\"\"": 222,
      "''": 123
    }
  ]
  

• $..[1].key
  Input:

  {
    "k": [
      {
        "key": "some value"
      },
      {
        "key": 42
      }
    ],
    "kk": [
      [
        {
          "key": 100
        },
        {
          "key": 200
        },
        {
          "key": 300
        }
      ],
      [
        {
          "key": 400
        },
        {
          "key": 500
        },
        {
          "key": 600
        }
      ]
    ],
    "key": [
      0,
      1
    ]
  }
  

  Expected output (in any order as no consensus on ordering exists):

  [200, 42, 500]
  

  Actual output:

  [
    42
  ]
  

• $..key
  Input:

  {
    "object": {
      "key": "value",
      "array": [
        {
          "key": "something"
        },
        {
          "key": {
            "key": "russian dolls"
          }
        }
      ]
    },
    "key": "top"
  }
  

  Expected output (in any order as no consensus on ordering exists):

  ["russian dolls", "something", "top", "value", {"key": "russian dolls"}]
  

  Actual output:

  [
    "something",
    "top",
    "value",
    {
      "key": "russian dolls"
    }
  ]
  

• $.store..price
  Input:

  {
    "store": {
      "book": [
        {
          "category": "reference",
          "author": "Nigel Rees",
          "title": "Sayings of the Century",
          "price": 8.95
        },
        {
          "category": "fiction",
          "author": "Evelyn Waugh",
          "title": "Sword of Honour",
          "price": 12.99
        },
        {
          "category": "fiction",
          "author": "Herman Melville",
          "title": "Moby Dick",
          "isbn": "0-553-21311-3",
          "price": 8.99
        },
        {
          "category": "fiction",
          "author": "J. R. R. Tolkien",
          "title": "The Lord of the Rings",
          "isbn": "0-395-19395-8",
          "price": 22.99
        }
      ],
      "bicycle": {
        "color": "red",
        "price": 19.95
      }
    }
  }
  

  Expected output (in any order as no consensus on ordering exists):

  [12.99, 19.95, 22.99, 8.95, 8.99]
  

  Actual output:

  [
    19.95
  ]
  

• $[?(1==1)]
  Input:

  [
    1,
    3,
    "nice",
    true,
    null,
    false,
    {},
    [],
    -1,
    0,
    ""
  ]
  

  Error:

  unexpected character ']' at 1:40
  

For reference, the output was generated by the program in https://github.com/cburgmer/json-path-comparison/tree/master/implementations/Golang_github.com-ohler55-ojg.

Hello,

It would be nice to have a KeepKeyOrder option such that key order will always be preserved.

row2 := oj.MustParseString(`{"b":0, "a":1, "c":2}`, &ojg.Options{KeepKeyOrder: true})

This could be done by using [][2]any (a list of key/value pairs) instead of map[string]any to store objects.

Is it possible ? if yes, and if someone can point me to the right direction I could try to submit a pull request for that.

Thank you for your answer :)

Adrien

Hello,
I try to remove the nth element of an array.

I have this jsonPath:

$.services[0]

And my initial resource looks like:

{
  "services": [
    {
      "name": "service 1",
    },
    {
      "name": "service 2",
    }
  ]
}

I have my initial resource as a byte[] in b, and then I do this:

	compiledPath, _ := jp.ParseString("$.services[0]")

	var resourceAsMap interface{}
	_ = json.Unmarshal(b, &resourceAsMap)

	_ = compiledPath.Del(resourceAsMap)

(I removed the error handling for lisibility here)

I expected to have this after the Delete:

{
  "services": [
    {
      "name": "service 2",
    }
  ]
}

But instead of this I have:

{
  "services": [
    null,
    {
      "name": "service 2",
    }
  ]
}

When debugging I see that the null comes from here:

My question is, does the Del function should set the element to nil in the case of an array, or removing it?
Thank you in advance for your answer!

I'm looking for something like your alternative for:
b, err := json.Marshal(data)

But I can't find it (perhaps its there, but I just didn't see it)

Can you point me to something like this?

First of all, thanks for the awesome library. I just wanted to report that when oj.Unmarshal is called on a struct and fails, it provides an unhelpful error message. Here's a minimal example:

type Query struct {
	Level  string
	Query  map[string]interface{}
	Expand bool
	Limit  int
}

func main() {
	queryJSON := `{
	"Level": "Series",
	"Query": {},
	"Expand": false,
	"Limit": true
}`

	var query Query
	err := oj.Unmarshal([]byte(queryJSON), &query)
	if err != nil {
		fmt.Println(err)
	}
}

The error here is that a bool is attempted to be unmarshaled onto an int. Here's an example of the error message I get: &{ map[Query:0xc000060050 main/Query:0xc000060050]}.

Playground: https://play.golang.org/p/ytyPl0SF_no

I tried this program: https://go.dev/play/p/De9ncvFbK6S:

package main

import (
	"fmt"

	"github.com/ohler55/ojg/jp"
	"github.com/ohler55/ojg/oj"
)

func main() {
	obj, err := oj.ParseString(`[{"name": "Alice", "age": 20}, {"age": 30}, {"name": "Bob"}]`)
	if err != nil {
		panic(err)
	}

	// Define the JSONPath expression to select objects without a "name" field.
	x, err := jp.ParseString("[?([email protected])]")
	if err != nil {
		panic(err)
	}

	ys := x.Get(obj)
	fmt.Println(ys)
}

It won't panic, but it also just returns all the objects regardless the filter. Is it possible to achieve what I need with your library? Thank you.

When encountering errors at multi-byte/unicode tokens, the ojg parser reports an error message with the wrong character.

$ echo '→' | oj
*-*-* unexpected character 'â' at 1:1

I'd like to either display the correct unicode character, or omit it:

*-*-* unexpected character '→' at 1:1

Note: this same bug shows up in the "encoding/json" library and in Firefox JS, but is correct in Chrome JS.

Here is Chrome's error message:

> JSON.parse('→')

VM83:1 Uncaught SyntaxError: Unexpected token → in JSON at position 0
    at JSON.parse (<anonymous>)
    at <anonymous>:1:6

The JSON is like {"key-1": "value"}, and my code is p, err := jp.ParseString("$.key-1"), then the err is not nil

Rarely, and under high load situations, we see a panic due to a concurrent map access. I'm not entirely sure our code is innocent here, but the panic does occur within ojg version 1.9.5. For reference, here's the relevant part of the stacktrace:

fatal error: concurrent map read and map write

goroutine 9084894 [running]:
runtime.throw(0x188b125, 0x21)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc004719200 sp=0xc0047191d0 pc=0x436e42
runtime.mapaccess2_faststr(0x160cf00, 0xc001bc3d10, 0xc00071ae38, 0x4, 0x44ed4e, 0xc00185b680)
	/usr/local/go/src/runtime/map_faststr.go:116 +0x47c fp=0xc004719270 sp=0xc004719200 pc=0x4150ac
github.com/ohler55/ojg/jp.Expr.Get(0xc000708a00, 0x4, 0x4, 0x160cf00, 0xc001bc3d10, 0x0, 0x19, 0x185af01)
	/builds/foo/bar/baz/.tmp/pkg/mod/github.com/ohler55/[email protected]/jp/get.go:62 +0x64ad fp=0xc004719920 sp=0xc004719270 pc=0x132c11d

<omitted calling code>

Our usage looks like this:

var extractor = mustParseExpr("$.foo.bar.baz")

func mustParseExpr(toBeParsed string) (x jp.Expr) {
	x, err := jp.ParseString(toBeParsed)
	if err != nil {
		panic(err)
	}
	return x
}

func doWork(msg map[string]interface{}) {
    vals := extractor.Get(msg)
    // additional code omitted
}

oj.Marshal fails with the following error when called on structs with embedded interface: reflect: NumField of non-struct type <embedded-interface-type>.

I think this is related to this #71

input jsonRequestString:
{"sessionId":0,"data":{"type":"request_device_set","info_id":"","data":{"sessionId":0,"data":{"WDRC_6_GT":[0.00014944501139803467,0.00015953007112451845]}},"device_id":""}}

requestObj, error := oj.ParseString(jsonRequestString)

output requestObj:
map[data:map[data:map[data:map[WDRC_6_GT:[0.0019242805885176385 0.002054137614082889]] sessionId:0] device_id: info_id: type:request_device_set] sessionId:0]]

values in WDRC_6_GT error

Perhaps this isn't possible, I didn't find it in the documentation

I have a struct that has to be converted to a json []byte array
What is the way to do this (if possible)
(i tried to understand if this is somewhere in the jp part, but didn't find a solution.)

also if possible: an array of structs...

When invoking oj.Marshal or oj.Unmarshal, any custom JSON [Un]Marshalers are expected to be invoked. This behavior is seen in the standard library's json package. Below is a code snipped showing the issue:

type Decimal struct {
	value *big.Int
	exp   int32
}

func (d Decimal) MarshalJSON() ([]byte, error) {
	return []byte(fmt.Sprintf("\"%d,%d\"", d.value, d.exp)), nil
}

type TestStruct struct {
	Outer  bool   `json:"outer"`
	Decimal Decimal `json:"decimal"`
}

func TestMarshall_ArrCompatibility(t *testing.T) {

	strct := []TestStruct{{
		Outer:   true,
		Decimal: Decimal{big.NewInt(5), 2},
	}}

	stdBuf, err := json.Marshal(strct)
	tt.Nil(t, err)

	ojBuf, err := oj.Marshal(strct)
	tt.Nil(t, err)


	tt.Equal(t, true, strings.Contains(string(stdBuf), "\"decimal\":\"5,2\""))
	if !strings.Contains(string(ojBuf), "\"decimal\":\"5,2\"") {
		t.Fatal("Mashaled JSON did not invoke custom Marshaler.  Expected: \"decimal\":\"5,2\" to be present.  Got: ", string(ojBuf))
	}
}

which returns the following:

Mashaled JSON did not invoke custom Marshaler.  Expected: "decimal":"5,2" to be present.  Got:  [{"decimal":{},"outer":true}]

Thank you for this library that allows me to import json data smoothly.
But I still have a problem that when using jsonpath similar to [?(@.key0 == "value0")] to extract json data concurrently, it will operate jp.Script.stack concurrently. is it to save memory to put rhe "s.stack" on the structure? In order to enable concurrent access, is it possible to put "s.stack" on the stack?

Hey!
Are you able to calculate aggregates with ojg?
If so, could you please provide an example of an average of a field (using the command-line tool if possible)?
Thanks!