itchyny / gojq Goto Github PK
View Code? Open in Web Editor NEWPure Go implementation of jq
License: MIT License
Pure Go implementation of jq
License: MIT License
This was caught out by a test failure on mips64:
--- FAIL: TestCliRun (0.86s)
--- FAIL: TestCliRun/multiply_strings (0.00s)
cli_test.go:87: standard output:
(
"""
... // 8 identical lines
"abcabc"
null
- null
+ ""
"""
)
FAIL
FAIL github.com/itchyny/gojq/cli 0.872s
The result of int(math.Inf(1))
1 depends on the architecture, and is not correct in general to do.
On s390x / ppc64le (big-endian), it returns 9223372036854775807, which would have failed as well, would it not be for the limit condition.
A simple way to fix this is to check:
math.IsNaN(cnt - cnt)
Which catches both infinite (positive / negative) as well as NaN
.
Works fine with pipe after a single-level deep object:
# echo '{"foo":123}' | jq '{"xxx": . } | select(true)'
{
"xxx": {
"foo": 123
}
}
# echo '{"foo":123}' | gojq '{"xxx": . } | select(true)'
{
"xxx": {
"foo": 123
}
}
Parse of jq expression fails with more levels:
# echo '{"foo":123}' | jq '{ "qq": {"xxx": . } | select(true) }'
{
"qq": {
"xxx": {
"foo": 123
}
}
}
# echo '{"foo":123}' | gojq '{ "qq": {"xxx": . } | select(true) }'
gojq: invalid query: { "qq": {"xxx": . } | select(true) }
{ "qq": {"xxx": . } | select(true) }
^ unexpected "|" (expected "}")
Hi,
I'm having difficulty reading how to use the library. I can use gojq in the shell just fine but I thought I'd dive into how you managed to do some things but I can't manage.
For instance:
echo '[{"how":"does", "this":"work"},{"how":"does", "this":"work"}]' | gojq '.[]' { "how": "does", "this": "work" } { "how": "does", "this": "work" }
I tried just following the example in the docs to this:
body :=
[{"how":"does", "this":"work"},{"how":"does", "this":"work"}]`
query, err := gojq.Parse(".[]")
if err != nil {
log.Error(err)
}
input := []interface{}{body}
iter := query.Run(input)
for {
v, ok := iter.Next()
if !ok {
break
}
if err, ok := v.(error); ok {
log.Error(err)
}
ioutil.WriteFile("tester.json", []byte(fmt.Sprint(v)), 0644)
fmt.Printf("%#v\n", v)
}`
But this just prints out the same object.
Is there anyone who can offer some suggestions on what I'm doing wrong and if time permits where I should be looking in the package for why it doesn't work.
$ echo '{}' | jq '"05" | tonumber'
5
$ echo '{}' | gojq '"05" | tonumber'
gojq: invalid character '5' after top-level value: "05"
Support to parse and render YAML format would be a very big plus if compared to jq project that is having difficulties in implementing it and probably will never do.
A good example of what I have in mind is the yq python wrapper to jq but with some extra parameters options to trigger the YAML parser instead of JSON (or maybe just a translator).
Parsing/rendering of other formats like XML, TOML, HCL would also make this project really appealing.
I don't really think the jq behaviour is better. I like gojq's error better but I thought I should report the difference:
$ echo '{}' | jq '{"category": 4, "ding": 3} | add'
7
$ echo '{}' | gojq '{"category": 4, "ding": 3} | add'
gojq: add cannot be applied to: object ({"category":4,"ding":3})
Big flat struct on input like {app_name,app_propertie1,...}
and query like .|group_by(.app_name)
resulting abnormal memory consumption.
Would it be an idea to provide a default moduleloader by making this moduleloader public?
Thanks for the awesome tool!
Is there a particular reason for creating the release binaries as .zip
for macOS and .tar.gz
for Linux? And would there be any chance of changing this to use .tar.gz
for both?
I'm writing a script that pulls down gojq as part of CI, and I have to write a special case to work around this inconsistency.
If $HOME
is unset, gojq exits with error $HOME is not defined
I realise this situation is uncommon (I'm running gojq as a CLI tool inside an AWS Lambda) and have worked around it but thought it might be worth reporting anyhow...
Thanks for the tool, works great!
It would be nice to be able to filter each value in an array or object through a shell command
There is an issue in jq, which is popular but hasn't yet made it into the library. Perhaps it would be a lot easier to implement with golang :)
Hi,
I'm trying to install gojq on Mint 19.3, that comes with golang 1.10 in the repositories.
Here is the verbose output:
env GO111MODULE=on go get -v github.com/itchyny/gojq/cmd/gojq
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
golang.org/x/sys/internal/unsafeheader
github.com/pbnjay/strptime
github.com/mattn/go-runewidth
github.com/fatih/color/vendor/golang.org/x/sys/unix
golang.org/x/sys/unix
github.com/pkg/errors
github.com/alecthomas/participle/lexer
gopkg.in/yaml.v3
github.com/lestrrat-go/strftime
github.com/alecthomas/participle
github.com/itchyny/gojq
github.com/fatih/color/vendor/github.com/mattn/go-isatty
github.com/mattn/go-isatty
github.com/itchyny/go-flags
github.com/fatih/color/vendor/github.com/mattn/go-colorable
# github.com/itchyny/gojq
go/src/github.com/itchyny/gojq/func.go:606:16: undefined: strings.ReplaceAll
go/src/github.com/itchyny/gojq/func.go:639:24: undefined: strings.ReplaceAll
go/src/github.com/itchyny/gojq/func.go:1574:7: undefined: strings.ReplaceAll
github.com/fatih/color
github.com/hokaccha/go-prettyjson
The source has been downloaded to $GOPATH/src/, but I think gojq has not been built as it's not in $GOPATH/bin.
~$ go get -u github.com/itchyny/gojq/cmd/gojq
# github.com/itchyny/gojq/cli
Workspaces/go/src/github.com/itchyny/gojq/cli/error.go:81:12: er.Position undefined (type participle.Error has no field or method Position)
Workspaces/go/src/github.com/itchyny/gojq/cli/error.go:86:47: er.Position undefined (type participle.Error has no field or method Position)
Workspaces/go/src/github.com/itchyny/gojq/cli/error.go:87:37: er.Position undefined (type participle.Error has no field or method Position)
Workspaces/go/src/github.com/itchyny/gojq/cli/error.go:91:45: er.Position undefined (type participle.Error has no field or method Position)
Workspaces/go/src/github.com/itchyny/gojq/cli/error.go:92:29: er.Position undefined (type participle.Error has no field or method Position)
Counting with jq:
$ time cat test2.json | jq '[.Bills[].Revenue[].Amount] | length'
777
real 0m0,096s
user 0m0,094s
sys 0m0,009s
Counting with gojq
$ time cat test2.json | gojq '[.Bills[].Revenue[].Amount] | length'
777
real 0m0,078s
user 0m0,082s
sys 0m0,018s
Updating with jq:
$ time cat test2.json | jq '.Bills[].Revenue[].Amount |= "TEST"' > /dev/null
real 0m0,131s
user 0m0,131s
sys 0m0,009s
Updating with gojq:
$ time cat test2.json | gojq '.Bills[].Revenue[].Amount |= "TEST"' > /dev/null
real 0m3,256s
user 0m5,475s
sys 0m0,676s
Updating when using map (gojq)
$ time cat test2.json | gojq '.Bills |= map(.Revenue |= map(.Amount = "TEST"))' > /dev/null
real 0m0,163s
user 0m0,227s
sys 0m0,017s
It looks like it's getting slower the more data is in the parent nodes. Even if those aren't updated / touched.
In can provide the test file but I'd rather not share it publicly.
Would it be possible to add an eval() function?
Steps to reproduce:
On an empty directory run
go mod init test
go get github.com/itchyny/gojq
The command fails with
go: found github.com/itchyny/gojq/cmd/gojq in github.com/itchyny/gojq v0.12.0
go get: github.com/itchyny/[email protected] requires
github.com/itchyny/[email protected]: invalid version: unknown revision 12a293722290
Running the tests on the master branch also fails with the same error
$ echo '{}' | gojq '{test1: 1, test2: 2,}'
gojq: invalid query: {test1: 1, test2: 2,}
{test1: 1, test2: 2,}
^ unexpected "," (expected "}")
$ echo '{}' | jq '{test1: 1, test2: 2,}'
{
"test1": 1,
"test2": 2
}
I don't know to what amount you want to keep compatibility with the original jq? We have a fairly large codebase of jq files where trailing comma's in objects are used quite a lot.
Use the following test json in jq: {"test":"test1","student":[{"name":"Luffy","age":17},{"name":"Pikachu","age":1},{"name":"Gintoki","age":25}]}
Enter the command (.student | .[] | select( .name == "Pikachu" ) | .age ) = 100
Will get the following result
But in gojq, you will get error: gojq: gojq: invalid path against: object ({"age":1,"name":"Pikac ...)
This is due to the inconsistency between the values of args [0] and env.paths.top () in the function at github.com/itchyny/gojq.(*env).pathEntries at execute.go. index has not changed since the last round of execution.
How to get the complete json string after modifying the value of json?
Hi man! Thank you for your job.
It would be great to document how numerics (i.e. float and decimals) are decoded (float64? int? int64? depending on value?). This will make it easier to test for return types on client side.
cat | gojq 'path(.[] | select(.id | startswith("x")) | .id)' <<EOF
[{"id": "xy"}, {"id": "yx"}]
EOF
cat | gojq 'path(.a[path(.b)[0]])' <<EOF
{"a":{"b":0}}
EOF
How do I differentiate between array with single element vs a single element from the Iterator of a query ?
package main
import (
"fmt"
"log"
"github.com/itchyny/gojq"
)
func main() {
query, err := gojq.Parse(".tags[].name")
if err != nil {
log.Fatalln(err)
}
input := map[string]interface{}{
"kind": "do#droplets",
"href": "/droplets/123",
"tags": []interface{}{
map[string]interface{}{
"name": "hello",
},
map[string]interface{}{
"name": "goodbye",
},
},
}
iter := query.Run(input)
var result []interface{}
for {
v, ok := iter.Next()
if !ok {
break
}
if err, ok := v.(error); ok {
log.Fatalln(err)
}
if v != nil {
result = append(result, v)
}
}
fmt.Println(result)
}
Output: [hello goodbye]
Now, if I modify the input,
package main
import (
"fmt"
"log"
"github.com/itchyny/gojq"
)
func main() {
query, err := gojq.Parse(".tags[].name")
if err != nil {
log.Fatalln(err)
}
input := map[string]interface{}{
"kind": "do#droplets",
"href": "/droplets/123",
"tags": []interface{}{
map[string]interface{}{
"name": "hello",
},
},
}
iter := query.Run(input)
var result []interface{}
for {
v, ok := iter.Next()
if !ok {
break
}
if err, ok := v.(error); ok {
log.Fatalln(err)
}
if v != nil {
result = append(result, v)
}
}
fmt.Println(result)
}
Output : [hello]
Output in both cases is an array. Now, if I change the path and set it to .href
, I still get an array:
package main
import (
"fmt"
"log"
"github.com/itchyny/gojq"
)
func main() {
query, err := gojq.Parse(".href")
if err != nil {
log.Fatalln(err)
}
input := map[string]interface{}{
"kind": "do#droplets",
"href": "/droplets/123",
"tags": []interface{}{
map[string]interface{}{
"name": "hello",
},
},
}
iter := query.Run(input)
var result []interface{}
for {
v, ok := iter.Next()
if !ok {
break
}
if err, ok := v.(error); ok {
log.Fatalln(err)
}
if v != nil {
result = append(result, v)
}
}
fmt.Println(result)
}
Output: [/droplets/123]
I want to be able to differentiate between Iterators which is an array of one element vs an Iterator that should be giving me a single value. Is there a good way of handling this using this library ?
I just tried this out and wanted to merge two objects but it's giving me different results compared to jq
jq -s 'reduce .[] as $item({}; . * $item)' <(echo '{ "a": "a" }') <(echo '{ "b": "b" }')
{
"a": "a",
"b": "b"
}
❯ gojq -s 'reduce .[] as $item({}; . * $item)' <(echo '{ "a": "a" }') <(echo '{ "b": "b" }')
{
"a": "a"
}
{
"b": "b"
}
gojq removes backslashes from double quoted string keys.
gojq 0.11.2 (rev: HEAD/go1.15.2)
1 - Setup a file with a JSON containing keys that include doublequotes and require backslashes. Example:
{
"aliasColors": {
"{statuscode=\"200\"} 200": "green",
"{statuscode=\"404\"} 404": "semi-dark-purple",
"{statuscode=\"500\"} 500": "dark-red"
}
}
2 - Run gojq in file.
gojq '.' issue.json
{
"aliasColors": {
"{statuscode=\"200\"} 200": "green",
"{statuscode=\"404\"} 404": "semi-dark-purple",
"{statuscode=\"500\"} 500": "dark-red"
}
}
{
"aliasColors": {
"{statuscode="200"} 200": "green",
"{statuscode="404"} 404": "semi-dark-purple",
"{statuscode="500"} 500": "dark-red"
}
}
When grouping large quantities gojq's version of group_by is considerably slow then jq:
leon@polly:~$ time cat testdata.json | jq 'group_by(.age)' > /dev/null
real 0m0,153s
user 0m0,157s
sys 0m0,000s
leon@polly:~$ time cat testdata.json | gojq 'group_by(.age)' > /dev/null
real 0m17,528s
user 0m23,949s
sys 0m1,187s
Saw #67, and as I mentioned, having a repl is a great thought. As a next step to this, utilising https://github.com/jesseduffield/gocui or something similar to create a ui would be awesome. Being able to do cat file.json | gojq --interactive
so we can construct the query will make life so much easier. Possibly with completion, but that feels like it would be a lot more work.
Hi, would it make sense to export something similar to goqj/cli.emptyError
as gojq.EmptyError
and maybe also add a check for it in the op execution so that it does not need to be checked for in the Iter
loop?
Im using it to implement something similar to empty
and select
but feels a bit strange.
Hi, was digging around trying to learn more about the gojq query AST type and noticed panics for some queries when I tried to print them using log.Printf("var: %#+v\n", var)
(the lv
snippet in vscode). The panic happens because the expression has no operator and panics in func (Operator) String()
.
Example:
package main
import (
"log"
"github.com/itchyny/gojq"
)
func main() {
q, err := gojq.Parse("a")
if err != nil {
panic(err)
}
log.Printf("q: %#+v\n", q)
}
Would it make sense that queries that parses successfully should be formattable like this? but I understand the panic make sense to have also. Maybe a new operator enum OpNone
etc?
jq and yq in one binary executable
For anyone looking for a solution to get the uniques without changing the order:
echo '[{"id": 3}, {"id": 1}, {"id": 5}, {"id": 3}]' | gojq -L=. 'import "jqs/jq-lib" as L; (unique_by(.id), L::unique_by(.id))'
[
{
"id": 1
},
{
"id": 3
},
{
"id": 5
}
]
[
{
"id": 3
},
{
"id": 1
},
{
"id": 5
}
]
def unique_by(f):
reduce .[] as $x ({lookup: {}, vals: []};
($x|f) as $xf
| ($xf|tostring) as $s
| if (.lookup[$xf|type] // {}) | has($s) then
.
else
(.lookup[$xf|type][$s] = $x)
| (.vals |= (. + [$x]))
end
)
| [.vals[]]
;
$ echo '[{"id": 3}, {"id": 1}, {"id": 5}, {"id": 3}]' | gojq -L=. 'import "jqs/jq-lib" as L; (group_by(.id), L::group_by(.id))'
[
[
{
"id": 1
}
],
[
{
"id": 3
},
{
"id": 3
}
],
[
{
"id": 5
}
]
]
[
[
{
"id": 3
},
{
"id": 3
}
],
[
{
"id": 1
}
],
[
{
"id": 5
}
]
]
def group_by(f):
reduce .[] as $x ([];
($x|f) as $xf
| ($xf|tostring) as $s
| (to_entries | map(select(.value.key == $s).key) | first) as $i
| if $i == null then
(. + [{key: $s, vals: [$x]}])
else
.[$i].vals |= (.+ [$x])
end
)
| [.[].vals]
;
I'm compiling a jq expression with this method: https://pkg.go.dev/github.com/itchyny/gojq#Compile
query, _ := gojq.Parse(".foo | ..")
code, _ := gojq.Compile(query)
for i:=0;i<10;i++ {
go func() {
code.Run(input)
}()
}
is the above safe with goroutines?
Is there any way to cancel on-going executions?
Hello! Would it be possible to add support for own "internal" functions written in go when using gojq as a library?
I had a quick look at the code and it seemed not that far away but maybe i'm missing something? If you think it's a good i would even be happy to try implement it if you give some pointers.
Hi, i wonder if it would be possible to support go types in the input data structure that implements a array or object interface.
Something like this
type JsonArray interface {
Length() int
Index(index i) interface{}
}
type JsonObject interface {
Length() int
Property(name string) interface{}
}
type CustomObject struct{}
func (CustomObject) Length() int { return 1 }
func (CustomObject) Property(name string) interface{} {
switch name {
case "a":
return 123
default:
return nil
}
}
func Test() {
query, _ := gojq.Parse(".foo.a")
input := map[string]interface{}{"foo": CustomObject{}}
iter := query.Run(input)
v, ok := iter.Next() // v would be 123
}
$ echo '{}' | jq '# dont do this stuff anymore'
{}
$ echo '{}' | gojq '# dont do this stuff anymore'
gojq: invalid query: # dont do this stuff anymore
# dont do this stuff anymore
^ unexpected "<EOF>" (expected <index> | "." | "." | ".." | "null" | "true" | "false" | <ident> | <variable> | <moduleident> | "{" | "[" | <number> | "+" | "-" | <format> | <string> | " " | "if" | "try" | "reduce" | "foreach" | "break" | "(" | "label")
It seem to make sense to expose an interface such as the following:
input := []byte("[{"foo":42}, {"foo":43"}]")
query, _ := gojq.Parse(". | map(.foo)")
var output []byte
output, _ = query.Run(input)
fmt.Printf("%v\n", output) // prints `[42, 43]`
This would allow user to operate on (valid) json bytes. I looked around the source and seems this is not quite straightforward as of today. Any recommendations?
Hi!
I was doing some custom functions that felt like good fit to be try
:able on error. Did some quick experimentation by exporting gojq.exitCodeError
as gojq.ExitCodeError
that would enable a custom function to do this:
func customFn(c interface{}, a []interface{}) interface{} {
return &gojq.ExitCodeError{
Value: ...,
}
}
Seems to work as expected.
Would that kind of functionality make sense? maybe should be done by checking if errors implement a tryable/catchable-interface somehow?
Another related question. I tried to understad what the significance of exitCodeError.code = 5
has, just a random constant for tests or something else?
Hi, something similar to gofmt
would be nice and make it possible to write IDE extensions that does auto formatting. I guess one tricky bit would be to allow preserves some formatting like some new lines etc?
Vet and lint might be nice also but haven't thought so much about what kind of errors they could find.
$ time cat faker.txt | jq 'length'
14927
real 0m0,161s
user 0m0,157s
sys 0m0,013s
$ time cat faker.txt | gojq 'length'
14927
real 0m0,135s
user 0m0,144s
sys 0m0,034s
time cat faker.txt | jq 'add' > /dev/null
real 0m0,286s
user 0m0,284s
sys 0m0,009s
time cat faker.txt | gojq 'add' > /dev/null
real 0m31,324s
user 0m46,734s
sys 0m0,245s
jq
has the command-line option -L <directory>
, as well as the include
directive (not sure how useful the <metadata>
option to it is, however):
o -Ldirectory / -L directory:
Prepend directory to the search list for modules. If this option is used then no builtin search
list is used. See the section on modules below.
include RelativePathString [<metadata>];
Imports a module found at the given path relative to a directory in a search path as if it were
included in place. A ".jq" suffix will be added to the relative path string. The module's symbols are
imported into the caller's namespace as if the module's content had been included directly.
The optional metadata must be a constant jq expression. It should be an object with keys like "home-
page" and so on. At this time jq only uses the "search" key/value of the metadata. The metadata is
also made available to users via the modulemeta builtin.
It would be great if gojq
could produce colorized output even when --yaml-output
is specified.
same as jq 1859
gojq: compile error: function not defined: keys_unsorted/0
jq has at least one very awkward default behavior.
.[]
and keys
sort things in very different ways
.[]
enumerates values based on the keys input order
keys
enumerates values alphabetically https://jqplay.org/jq?q=keys&j={%22foo%22%3A%201%2C%20%22abc%22%3A%202%2C%20%22abcd%22%3A%203%2C%20%22Fe%22%3A%204}
jq 1.5 introduced keys_unsorted
which enumerates values the same way that .[] does.
https://jqplay.org/jq?q=keys_unsorted&j={%22foo%22%3A%201%2C%20%22abc%22%3A%202%2C%20%22abcd%22%3A%203%2C%20%22Fe%22%3A%204}
Without the --arg option, it is difficult to safely pass strings from outside to gpjq queries. Why not?
@csv
, @json
etc
Hi, i've thought about adding support for hex, octal and binary number literals. Is that something you would be interested in?
I understand if you don't want to add extensions and non-compatible things to gojq compared to jq. Would an option be to make such things disable by default? but maybe messy to do with some features?
Hey,
I have an array in the form of []map[string]interface{}
I would like to run a query such .[] | .id
but it fails with:
expected an object but got: array ([{"id":"fb4ae77f-144d- ...)
what should I pass to query.Run
in order to execute gojq over an array ?
Thanks
Not sure if this has to do with reduce
, but jq
works with this, whereas gojq
can't parse it. FWIW, using a locally defined function def pass: .;
instead of the reduce
will parse.
# cat x.jq
. as $rec | foreach . as $c ({};
$c | . +
reduce(.) as $k ({}; .) |
. += {"qq": "baz"} )
# echo '{"foo": "bar"}' | jq -f x.jq
{
"foo": "bar",
"qq": "baz"
}
# echo '{"foo": "bar"}' | ./gojq -f x.jq
gojq: invalid query: x.jq:2
2 | $c | . +
^ unexpected "+" (expected ")")
I am trying to run jq -n '[inputs] | add'
over my json and it gives an error function not defined: inputs/0
Hi, I noticed some integer arithmetics and some functions seems to convert float
Example:
$ go run cmd/gojq/main.go -n '(18281289274965207791/2)*2'
18281289274965207000
$ go run cmd/gojq/main.go -n '18281289274965207791 | floor'
18281289274965207000
Looking at the code is seems both big.Int / int
and big.Int / big.Int
has this issue.
Not sure floor
is a good example but could be "passthru" i guess? but some others like sin
etc might be harder?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.