Issuing make clean on my system results in deletion of the following files:

else@butters ~/src/jq-1.1 % make -Bnd | grep 'Must remake target' | sed 's/.*`\(.*\)'\''.*/\1/' | grep -v 'all'
lexer.gen.c
lexer.gen.h
parser.gen.c
jv_utf8_tables.gen.h
jv_unicode.c
version.gen.h
main.c
jq

However, main.c and jv_unicode.c must not be deleted.

Not a major feature request, but would be really cool if the tool will suport color highlighting at some point.

This is a standard option in tools like grep and awk, and most notably would allow for jq scripts with:

!/path/to/jq -f

Actually, come to think, that would also require # comments...

Hey,

Awesome project; really love it!

More of an fyi, but under Fedora 13, I couldn't run the precompiled version; it complained "GLIBC_2.14 not found". Compiling locally works great.

I have many "yes, please"s to add to other feature requests; will add them there.

Hitting the following error on master:

$ echo {} | ./jq '{foo: "s"}'

error: syntax error, unexpected QQSTRING_START
{foo: "s"}
      ^
1 compile error

Used git bisect to narrow it down to this commit: afec254

I've got a JSON file that looks something like this:

{ "results" : { "groups" : {"alpha" : { "items": {
   "apple" : { "attributes": { "class": "fruit" } },
   "pear" : { "attributes" : { "class": "fruit" }},
   "dog" : {"attributes" : {"class": null }}
}},
"beta": {"items": {
  "banana" : { "attributes": { "class": "fruit" } }
}}}}}

I want to find keys of items where class is null. I cannot figure out how to do this using jq.

Is there any one can give me a brief instruction to build jq for Windows?

Thanks,
Tien,

The current version (c0a2f1a) won't compile on OS X ML 10.82. To solve the issue update bison/yacc to 2.5 or higher (since OS X comes with 2.3) and do a touch for parser.y and lexer.l, which will rebuild lexer & parser. This is to prevent outdated sources contained in the project.

I'm suggesting a new operator ~. It takes two Arrays x and y, and return a Array [z| z=xi for all yi=true].

Example:
<[1,2,3,4,5] ~ [1,null,1,null]
>[1,3]

Or maybe a set?

<[1,2,3,4,5] ~ [1,null,1,null]
>1
>3

<[[1,2,3,4,5] ~ [1,null,1,null]]
>[1,3]

Steps to reproduce

Run

jq '.subcommands //= [{name}]'

on this input:

{
  "name": "eval-file"
}

Expected result

{
  "subcommands": [
    {
      "name": "eval-file"
    }
  ],
  "name": "eval-file"
}

Actual result

{
  "subcommands": [
    {
      "name": null
    }
  ],
  "name": "eval-file"
}

Note that running jq '.subcommands = [{name}]' produces the expected output.

I'm using the latest 64bit build for Linux: jq version 1.2

PS: A jq --version utility would be helpful.

How can I escape "$" char?
I'm trying to parse the youtube apis, getting this media$group
This work

  curl 'https://gdata.youtube.com/feeds/api/standardfeeds/IT/most_recent_Sports/?v=2&alt=json' | jq' .feed.entry[] | {title}'

This is not working:

  curl 'https://gdata.youtube.com/feeds/api/standardfeeds/IT/most_recent_Sports/?v=2&alt=json' | jq' .feed.entry[] |  {media$group}'

Hey, I know there are good arguments not to do this, but...

My first use for jq is to parse live logs that I record in json. jq is super awesome. However...

Because I'm running it on logs, my logs all begin with '[', and then each line is something like:
{"date":"2012-10-22 00:02:45","level":"INFO","controller":"services","method":"get_my_stores_and_ads","message":"begin_execution","params":["917i","3ktDWc3TppvuBHGYDRUw3C"],"post":false,"version":"917i","user_id":"1979"},

So, each of my logs has no closing ']'. I know this not the problem of jq; you should be able to expect complete json. And I'm not even sure of a best solution. Perhaps if a file didn't parse, jq could retry adding a ']' on the end? Anyway, if you see this as a common use case, something to consider.

I have looked at a few source files for your current software. I have noticed that some checks for return codes are missing.

Would you like to add more error handling for return values from functions like the following?

• malloc ⇒ block_compile, jv_invalid_with_msg
• realloc ⇒ path_push

Input {"foo": 42, "bar": "less interesting data", "baz": "sdkjhsdf"}
Output {"foo": 42, "baz": "sdkjhsdf"}

The jq expression would cite bar, but not foo or baz. Possibly overloading the minus char somehow.

when I do this:
cat file.json | jq '.'
the jq exit when it met an unexpected line in file.json, can you add some features the skip this or log down the error when parsing a stream ?
I want it parse the whole file although there are some corrupt lines in the file.

Perhaps I'm missing it, but I can't see a way to filter based on a value containing a string. Something like

jq 'select(.msg.indexOf('Me") != -1)'

Input
[
{"msg": "Pick Me!", "date": "2012-10-22 04:42:05"},
{"msg": "Not this one","date": "2012-10-22 08:39:53"}
]

Output
{"msg": "Pick Me!", "date": "2012-10-22 04:42:05"},

Suggest you have this code (from selectjson)

{
    "name": {
        "first": "Lloyd",
        "last": "Hilaiel"
    },
    "favoriteColor": "yellow",
    "languagesSpoken": [
        {
            "lang": "Bulgarian",
            "level": "advanced"
        },
        {
            "lang": "English",
            "level": "native",
            "preferred": true
        },
        {
            "lang": "Spanish",
            "level": "beginner"
        }
    ],
    "seatingPreference": [
        "window",
        "aisle"
    ],
    "drinkPreference": [
        "whiskey",
        "beer",
        "wine"
    ],
    "weight": 172
}

And you want to select all languag names you can do that with
.languagesSpoken[].lang

If you want to select only the preferred ones you can do something like
[.languagesSpoken[] | if .preferred then .lang else 0 end] but that is imo not satisfying since the list now contains a placeholder. I suggest <> a subquery syntax.

exp1<exp2> would be equivalent to [exp1[] | if exp2 then . else **Nothing** end]

languagesSpoken<.preferred>.lang would be "English"

Since it can only be applied to a array i suggest it should be languagesSpoken<.preferred> rather than ``languagesSpoken[]<.preferred>`

You should offer this as a node module. It would be super helpful there.

Add a function to base64 encode and decode a string result. Some serialization packages such as Jackson, a popular Java package, include base64 encoding for some data fields.

I think grep users are used to this kind of command line:
grep [OPTIONS] PATTERN [FILE...]
I'd found very useful to have something like this, as many of my json file are already downloaded (I know, I can always use cat |).
Anyway, given that the syntax of jq is:
jq [options]
Doesn't look that hard nor breaking to implement.

First, I really like jq; thank you. I'm trying to use it for some large-ish data analysis, but running into a few snags. This is probably more my misunderstanding than an issue.

I'm trying to do a wordcount: just count how many times each line occurs in a text file. So for an input file like this:

one
two
three
two
three
three

Produce output like this (order not important):

one 1
three 3
two 2

My strategy is to use group_by() (please let me know if there's a better approach). Since group_by() only works on arrays (is there a reason it can't take a filter?), first I need to make my raw text into a JSON array. I couldn't get a single filter to work, so I ended up with jq -R followed by jq -s (as combining both gives me what I don't want):

$ < test.txt jq -R '.' | jq -s '.' 
[
  "one",
  "two",
  "three",
  "two",
  "three",
  "three"
]

Now I can use group_by():

< test.txt jq -R '.' | time jq -s -r 'group_by(.)[] | "\(.[0]) \(length)"'

which works. For small inputs. If I try 10,000 random strings generated like so:

$ tr -cd '\n[:alnum:]' < /dev/urandom | head -10000 > test.txt

The above filter runs in 1.6 seconds. 100,000 strings takes 143 seconds. 1 hour later, I'm still waiting for 1 million strings -- and I'd like to do a lot more than that! For comparison, the equivalent awk:

awk '{h[$0]++} END {for (line in h) {print line " " h[line]}}'

counts 100,000 lines in 0.2 seconds.

I gather group_by() works by sorting, but sort itself seems pretty fast:

$ < test-1M.txt jq -R '.' | time jq -s -r 'sort[]' > /dev/null
7.96user 0.15system 0:10.55elapsed 76%CPU (0avgtext+0avgdata 751648maxresident)k

So, I'm probably doing something dumb and triggering some N^2 behavior or something. Ideas?

I found myself today wishing I could pull the keys out of an object as an array. If you could add that, it'd be mighty useful!

I tried this:

$ jq '.data[1].target_platform' < mata.txt 

error: Invalid character
.data[1].target_platform
               ^
error: syntax error, unexpected IDENT, expecting $end
.data[1].target_platform
                ^^^^^^^^
2 compile errors

The input is a vanilla JSON file with entries like:

{
    "target_country_code": [ "US" ],
    "day": "2012-09-21",
    "campaign_name": "Best Game Ever campaign-11",
    "spend": 663,
    "bundle_id": "com.foo.best_game",
    "target_manufacturer": [ "Samsung" ],
    "clicks": 12,
    "downloads": 6,
    "campaign_id": 11,
    "app_id": "12341",
    "currency": "USD",
    "impressions": 30,
    "target_platform": [ "Nexus S", "阿里云" ]
}

i downloaded the pre-built 64-bit binary and got this error when I executed it:

./jq: /lib/libc.so.6: version `GLIBC_2.14' not found (required by ./jq)

would be great to know the version tags, so that it can be used in something like lets say homebew.

I'm trying to make productive use of jq (cool tool) with input like the following. I am trying to extract the corresponding object's "dns_name" for any object having a specified "state". I failed to figure out any way to do this from the tutorial and manual. Any ideas?

{
  "i-09xxxxxx": {
    "id": "i-09xxxxxx",
    "keyname": "blah",
    "dns_name": "ec2-REDACTED.compute-1.amazonaws.com",
    "created_at": "Sat Aug 18 18:31:48 UTC 2012",
    "state": "running",
    "tags": "{\"node-type\"=>\"bastion\", \"Name\"=>\"bastion\"}",
    "private_dns_name": "ip-REDACTED.ec2.internal"
  },
  "i-0a8bxxxx": {
    "id": "i-0a8bxxxx",
    "keyname": "jblaine",
    "dns_name": null,
    "created_at": "Wed Dec 26 20:32:15 UTC 2012",
    "state": "terminated",
    "tags": "{\"Created-By\"=>\"Puppet\"}",
    "private_dns_name": null
  },
  "i-cedxxxxx": {
    "id": "i-cedxxxx",
    "keyname": "foo",
    "dns_name": null,
    "created_at": "Thu Sep 20 05:36:41 UTC 2012",
    "state": "stopped",
    "tags": "{\"Name\"=>\"graph-inserter-2\"}",
    "private_dns_name": null
  }
}

I would like to have an option (-cc ?) that actually minifies the output. Strip whitespace, print integer-ish numbers as integer, possibly specify float precision.

Please provide pregenerated static html docs in your release tarballs, or provide a way to generate them at install time.

Not every machine will have a internet access, and installing an extra daemon for docs is often not an option.

(1). minor bug in the docs, for "," (comma) operator:

jq '[.user, .projects[]]'
Input   {"user":"stedolan", "projects": ["jq", "wikiflow"]}
Output  "stedolan"
       "jq"
        "wikiflow"

There should be a dot before the query's "[" (otherwise, the query is identical to the example for array construction, but with different results), like this:

jq '.[.user, .projects[]]'

(2). Alternative operator - //
The second paragraph begins

This is useful for providing defaults: .foo or 1

It should be:

This is useful for providing defaults: .foo // 1

When STDIN is whitespace-separated list of JSON objects, could you add a flag to implicitly turn that into a single, top-level JSON array?

As I read from the main page and the docs, this tool really has the potential to be as useful as sed or grep. I downloaded it and played with it for a while, but I found rather cumbersome to check for the docs online.

One of the things I loved while teaching myself those unix tools was the man page. I think that will definitely will give you a shot in making it more friendly and popular.

Cheers and keep up the good work.

I was hoping to use jq to manipulate json-ld (http://json-ld.org) but it complains about @.

cat paris_marley.json | jq '@subject'
error: Invalid character
@subject
^

Throwing "grep -v null" on the end of the command is easy enough, but in many (most?) situations where i'm doing a query on a field name, I don't want records where the query would result in a non-match on the record.

Could we get a command-line flag to spit out tab- or comma-separated values instead of concatenated json objects, csv/tsv-quoting cells as necessary?

It would make jq a real power house for reports and spreadsheets generation.

I have a ton of almost-json dumps dumped into Amazon s3 by KissMetrics, which has a rather inclusive view about what ".json" means.

Could jq gracefully accept octally quoted input like "\342\200\231" (and the like as) synonymous with "\u00e2\u0080\u0099" and silently output syntactically correct json at the output stage?

It's not json, but all javascript engines grok it, and as such, it occasionally rears its ugly head like here.

I have a large JSON file and just want to see the keys in the root object. Then I could dive into each key and get it's key, etc...

I'm either using it incorrectly, or string interpolation appears to be broken. See below.

<tmp]$ echo {"foo":42} | jq '@(the input was %(.))'
error: Invalid character
@(the input was %(.))
^
error: syntax error, unexpected IDENT
@(the input was %(.))
^^^^^
error: Invalid character
@(the input was %(.))
^
3 compile errors

I accidentally discovered an assertion failure when running jq with the following input:

$ echo -e '{ "foo": 42\n}' | jq --slurp .
jq: jv.c:280: jv_array_length: Assertion `jv_get_kind(j) == JV_KIND_ARRAY' failed.
Aborted

It'd be great to be able to slice arrays. Given the program

.[2..4]

and the input

[42,43,44,45,46,47]

result in the output

44
45
46

For the mac, you should add to the brew repository:

http://mxcl.github.com/homebrew/

I might well be reading the docs wrong, but I don't think there's a way to perform the following operation. Given the input

{"foo": "bar", "baz": 10}
{"foo": "blap", "baz": 11}
{"foo": "bar", "baz": 12}

Get only the objects where .foo == "bar", resulting in the output:

{"foo": "bar", "baz": 10}
{"foo": "bar", "baz": 12}

Currently the syntax for conditionals, requires an else case. If this were changed to allow only an if case and to not output a value if the case that doesn't match then the program which supplies the above could be:

./jq 'if .foo == "bar" then . end'

My current work around involves using the else case to output a sentinel value which I grep away before piping back into jq again.

$ (some input) | ./jq '.details[] | select(.health != "ok")'
error: Invalid character
.details[] | select(.health != "ok")
                            ^
1 compile error

Seems valid to me...

I attempted to pipe some JSON from cURL to jq but got the aforementioned parse error. After validating the JSON with JSONLint and some initial confusion I noticed that the JSON data had the UTF-8 representation of the BOM character (U+FEFF), the three bytes 0xEF 0xBB 0xBF, at the beginning. If I stripped them out before piping to jq, everything worked.

I haven't tried if the same happens with, say, UTF-16 (little or big endian) encoded data, but either way, jq needs to handle the BOM correctly.

Please, add the jq-1.2 tag on github

Very important for packaging in different systems. as everybody might not want to install it directly to /usr/local/bin.

Given this JSON:

[
    {
        "my-field": "val1",
        "other_field": "valA"
    },
    {
        "my-field": "val2",
        "other_field": "valB"
    }
]

this works:

$ ./jq '[.[]|{other_field}]' < test.json
[
  {
    "other_field": "valA"
  },
  {
    "other_field": "valB"
  }
]

but this does not:

$ ./jq '[.[]|{my-field}]' < test.json
error: syntax error, unexpected '-', expecting '}'
[.[]|{my-field}]
        ^
1 compile error

I think it should since the JSON is valid. Maybe there is a way around this?

I'm getting syntax errors from the Twitter examples (and everytime I try to use underscores):

curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true | jq '.results[] | {from_user, text, urls: [.entities.urls[].url]}'

error: Invalid character
.results[] | {from_user, text, urls: [.entities.urls[].url]}
^
error: syntax error, unexpected IDENT, expecting '}'
.results[] | {from_user, text, urls: [.entities.urls[].url]}
^^^^
2 compile errors
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5970 100 5970 0 0 11969 0 --:--:-- --:--:-- --:--:-- 23050

(Mac OSX Mountain Lion 10.8.2, Swedish keyboard layout)

Could you please rename the source tarball to be versioned, like jq-1.1.tgz (or jq-1.1.tar.gz) so package managers can distinguish versions. This is more or less mandatory to get in to the standard package repositories of several distros (Gentoo in this case).

~$ echo {} | jq '2-1'
error: syntax error, unexpected LITERAL, expecting $end
2-1
 ^^
1 compile error

~$ echo {} | jq '-1+2'
1

~$ jq --version
jq version 1.2

Subtracting 1 seems to the only way to get the last item in a list: echo [1,2,3] | jq '.[length-1]'