jzelinskie / faq Goto Github PK
View Code? Open in Web Editor NEWFormat Agnostic jQ -- process various formats with libjq
License: Apache License 2.0
Format Agnostic jQ -- process various formats with libjq
License: Apache License 2.0
As mentioned in #11 Drone Cloud has a 32-bit Arm build/test environment as well as arm64 and x86. This issue to track adding a Drone configuration (.drone.yml) and explore this build environment for faq.
While reviewing this project to figure out if it does what I'm looking for, I came across this line in the README:
How do you pronounce "faq"? Fuck you.
It strikes me as very hostile. The README is the first contact many users have with a project, and this one is telling me "fuck you" within the first few paragraphs. As it stands, I wouldn't recommend this project to anyone, because I'd have to explain that line in the README. I'm sure that there are plenty of people that turned away immediately and didn't say anything.
With separators auto-guessing and optional overrides:
https://en.wikipedia.org/wiki/Comma-separated_values
faq -f json -o csv < file.json
Hello @jzelinskie
I've been working with the JSON to Bencode conversion, but there are things that don't seem to make sense to me.
I was looking at how your tool handled things such as null values or booleans when converting to bencode (that doesn't have those types) and this is what came up.
$ faq -n -f json -o bencode '[1,null,2]'
l1:11:2e
# Why are integers converted to strings??? And I thought that null values would be converted to an empty string, for instance. Here they just disappear... which would mess up the length of the list.
$ faq -n -f json -o bencode '{a:null}'
de
# Entire pairs disappearing, although it could make some degree of sense
$ faq -n -f json -o bencode '[true,false]'
li1ei0ee
# Ok now that is something I expected
Now... is there a way to choose how these conversions take place, for instance how a false
in JSON would convert to Bencode, for instance as an empty string rather than a 0?
Thanks in advance
This is going to require adding a bunch of encoders and moving some of the logic around syntax reimplementing the lbjq print flags in Go.
There used to be a 0.0.8 tag but it seems to have vanished... Can somebody please explain why this is?
From jq
manpage:
· -e / --exit-status:
Sets the exit status of jq to 0 if the last output values was neither false nor null,
1 if the last output value was either false or null, or 4 if no valid result was ever
produced. Normally jq exits with 2 if there was any usage problem or system error, 3
if there was a jq program compile error, or 0 if the jq program ran.
I'm only filing this because the README states that they are:
$ faq
dyld: Library not loaded: /usr/local/opt/jq/lib/libjq.1.dylib
Referenced from: /Users/aat/Library/Caches/hermit/pkg/faq-0.0.7/faq
Reason: image not found
It would be great if we could configure the output format explicitly. Would be really useful for simple conversions too.
Trying to build an RPM using https://raw.githubusercontent.com/chancez/faq-rpm/master/faq.spec and running into the following:
/usr/bin/make -j56 SKIP_VALIDATE=true FAQ_LINK_STATIC=false 'GO_LD_FLAGS=-compressdwarf=false -linkmode external -extldflags "-v"'
/usr/bin/make test
make[1]: Entering directory `/builddir/build/BUILD/faq-0.0.6'
scripts/test.sh
go: downloading github.com/sirupsen/logrus v1.3.0
go: downloading github.com/spf13/cobra v0.0.3
go: downloading golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading github.com/globalsign/mgo v0.0.0-20180615134936-113d3961e731
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/Azure/draft v0.16.0
go: downloading github.com/BurntSushi/toml v0.3.1
go: downloading github.com/clbanning/mxj v0.0.0-20180418195244-1f00e0bf9bac
go: downloading github.com/ghodss/yaml v1.0.0
go: downloading github.com/alecthomas/chroma v0.6.2
go: downloading github.com/spf13/pflag v1.0.3
go: downloading github.com/zeebo/bencode v0.0.0-20180308174530-d522839ac797
go: downloading github.com/dlclark/regexp2 v1.1.6
go: downloading github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
go: downloading golang.org/x/sys v0.0.0-20181128092732-4ed8d59d0b35
go: downloading github.com/jbrukh/bayesian v0.0.0-20161210175230-bf3f261f9a9c
# github.com/jzelinskie/faq/pkg/jq
_cgo_export.c: In function 'goJQErrorHandler':
_cgo_export.c:29:3: error: expected specifier-qualifier-list before 'jv'
jv p1;
^
_cgo_export.c:32:8: error: 'struct <anonymous>' has no member named 'p1'
_cgo_a.p1 = jv;
^
FAIL github.com/jzelinskie/faq/pkg/faq [build failed]
=== RUN TestBencodeMarshal
=== RUN TestBencodeMarshal/#00
--- PASS: TestBencodeMarshal (0.00s)
--- PASS: TestBencodeMarshal/#00 (0.00s)
=== RUN TestBencodeUnmarshal
=== RUN TestBencodeUnmarshal/#00
--- PASS: TestBencodeUnmarshal (0.00s)
--- PASS: TestBencodeUnmarshal/#00 (0.00s)
=== RUN TestSVG
--- PASS: TestSVG (0.00s)
PASS
ok github.com/jzelinskie/faq/pkg/formats 0.024s
make[1]: *** [test] Error 2
make[1]: Leaving directory `/builddir/build/BUILD/faq-0.0.6'
make: *** [all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.ktd4Or (%build)
RPM build errors:
bogus date in %changelog: Mon Feb 14 2019 Chance Zibolski <[email protected]> 0.0.6-1
Bad exit status from /var/tmp/rpm-tmp.ktd4Or (%build)
Finish: rpmbuild faq-0.0.6-4.el7.src.rpm
Finish: build phase for faq-0.0.6-4.el7.src.rpm
ERROR: Exception(/tmp/faq-0.0.6-4.el7.src.rpm) Config(epel-7-x86_64) 1 minutes 26 seconds
INFO: Results and/or logs in: /home/builder/rpmbuild/results/epel-7-x86_64
INFO: Cleaning up build root ('cleanup_on_failure=True')
Start: clean chroot
Finish: clean chroot
ERROR: Command failed:
goreleaser would vastly improve the workflow around cutting releases.
If it can support cgo and embedding the version flag, it could probably replace the Makefile.
Because libjq is relatively new, many Linux distributions don't have it packaged and linkable.
Ideally, I can figure out the right cgo incantations to embed a version of libjq into a totally portable Linux amd64 binary.
/var/tmp/portage/app-misc/faq-0.0.2/work/faq-0.0.2/src /var/tmp/portage/app-misc/faq-0.0.2/work/faq-0.0.2
github.com/jzelinskie/faq/formats/json.go:7:2: cannot find package "github.com/alecthomas/chroma/quick" in any of:
/var/tmp/portage/app-misc/faq-0.0.2/work/faq-0.0.2/src/github.com/jzelinskie/faq/vendor/github.com/alecthomas/chroma/quick (vendor tree)
/usr/lib/go/src/github.com/alecthomas/chroma/quick (from $GOROOT)
/var/tmp/portage/app-misc/faq-0.0.2/work/faq-0.0.2/src/github.com/alecthomas/chroma/quick (from $GOPATH)
Hi! Great work! Learn a lot about cgo and jq internals from this project.
I start my own project to use jq programs in Go projects with ability to cache compiled jq state. So here are some findings of jq_start
behavior:
jq_start does two important things:
That stack is freed by a jq_teardown call that will call the same jq_reset. But input jv is already freed by the deferred call ;) That is why you need to use jv_copy here: https://github.com/jzelinskie/faq/blob/master/pkg/jq/exec.go#L237
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".
$ curl -s https://www.govinfo.gov/bulkdata/CFR/2020/title-14/CFR-2020-title14-vol2.xml | faq
Error: failed to encode as: invalid attribute key label: #text - due to attributes not being prefixed
Usage:
...
This works...
$ curl -s https://www.govinfo.gov/content/pkg/CFR-2020-title14-vol2/xml/CFR-2020-title14-vol2-sec91-205.xml | faq
I think it's as simple as
git clone https://github.com/jzelinskie/faq
cd faq
docker build .
to build using the supplied Dockerfile; but I couldn't find where the generated binary by the build was supposed to go.
Hello.
First of all, thanks a lot for the tool.
I am, however, encountering problems when dealing with data encoded with bencode
.
It's a problem I've come across time and again and hopefully one you can address.
From what I've seen you've interpreted the string length as the number of bytes the string is encoded as, which should be fine.
Since, I guess, the original specs of the format, if we can call them that, were less than crystal clear as to what string length meant, there are many implementations around interpreting the string length as the character count, in Unicode terms the count of codepoints present in the string.
Could you create a variant of the bencode
format supported by faq
that matches the variant interpretation of string length described above? It would make my life a lot easier dealing with these sorts of systems.
Just as a reference, faq would encode (arguably correctly) the JSON { "a": "à" }
as the bencode-d d1:a2:àe
, while the variant format would encode it as d1:a1:àe
, assuming UTF-8 encoded strings.
Thanks in advance.
Hi!
I would love to try out this project, it looks amazing!!
Unfortunately, in the builds/release section, I can only find Linux and Mac builds (and I don't know how to satisfy the JQ headers requirement).
I also tried installing the normal way using GO:
go install github.com/jzelinskie/faq/cmd/faq@latest
package github.com/jzelinskie/faq/cmd/faq
imports github.com/jzelinskie/faq/internal/faq
imports github.com/jzelinskie/faq/internal/jq: build constraints exclude all Go files in ...\go\pkg\mod\github.com\jzelinskie\[email protected]\internal\jq
Thank you for your consideration!
This is an inconsistency between the two programs that I'd like to fix.
is it possible to output selected fields using a custom format string like this?
jq '.[] | "foo \(.FIELDNAME2) bar \(.FIELDNAME5)" ...'
I spent a little time trying to see if I could generically support asn.1.
I found that supporting it would probably mean actually having schemas for various things that are encoded in that format (e.g. CAs, private keys). I think supporting files like these would be a huge boon to debugging, but also complicate the project to the point where it almost seems like a different goal: representing everything as JSON (or more generically "objects").
vbatts@infra1:~$ curl -s https://api.github.com/repos/flatcar-linux/manifest/releases | faq -f json '.[0].tag_name'
"v2492.0.0"
vbatts@infra1:~$ curl -s https://api.github.com/repos/flatcar-linux/manifest/releases | faq -f json -r '.[0].tag_name'
Error: failed to encode as pretty: invalid character 'v' looking for beginning of value
Usage:
faq [flags] [filter string] [files...]
[...]
vbatts@infra1:~$ curl -s https://api.github.com/repos/flatcar-linux/manifest/releases | jq -r '.[0].tag_name'
v2492.0.0
faq
seems to be expected raw values to act a certain way?
faq prints a confusing error message when it failed to auto-detect the input format:
$ jsonnet -y /dev/stdin <<< '[{x: 1, y: self.x} + {x: 10}, {z: 2}]' | faq
Error: no supported format found named auto
Usage:
faq [flags] [filter string] [files...]
Flags:
-a, --ascii-output force output to be ascii instead of UTF-8
-C, --color-output colorize the output (default true)
-c, --compact compact instead of pretty-printed output
-f, --format string input format (default "auto")
-h, --help help for faq
-m, --maintain-format maintain original format (don't output JSON)
-M, --monochrome-output monochrome (don't colorize the output)
-r, --raw output raw strings, not JSON texts
-S, --sort-keys sort keys of objects on output
-t, --tab use tabs for indentation
It would be very handy and helpful if faq
were available through Homebrew for macOS.
Hello, I started using faq
but I found what I think is an issue.
When using raw
output mode, faq
breaks trying to pretty encode the output. I would have expected that raw
output would override all enhancements to the output, but it does not seem the case.
How to reproduce:
$ echo '"hello world"' | faq --input-format=json -r "."
Error: failed to encode as pretty: invalid character 'h' looking for beginning of value
Adding --pretty-output=false
to the above command makes it work:
$ echo '"hello world"' | faq --input-format=json -r --pretty-output=false "."
hello world
Is this the expected --raw-output
behaviour?
For the full build log see https://cloud.drone.io/vielmetti/faq/12/1/2 based on the Drone Cloud configuration at https://github.com/vielmetti/faq/blob/vielmetti-drone/.drone.yml
=== RUN TestRunFaq/null_flag_input_simple_program
--
109 | faq.test: execute.c:1071: jq_compile_args: Assertion `jv_get_kind(args) == JV_KIND_ARRAY' failed.
I suspect that this is because I'm doing a Ubuntu-based build (using the "golang" Docker image as a base) and because jq in that image is 1.5 version.
In the future, I'd like to have ARM for darwin and linux built into Github Actions.
faq
should work just fine on armhf (32-bit Arm, Raspberry Pi), once it's built. A request in several parts:
dep
, specific version requirements for go
, and any library compatibility issues along the way.This needs to be a linker flag that we use in the Makefile
Right now the faq.Flags
type is a grab bag of config values and a lot of things read and write to it.
Ideally it would only be used by the pkg/command
package, and we would instead call faq.RunFaq
with something more specific, only containing options needed after command line handling, and preferrably, all the basic flag/conf validation can be done before faq.RunFaq
Any consderation of support for .ini files, the somewhat standardized format often used for config files?
I see a Go library https://github.com/go-ini/ini that could be useful, and there's a proof of concept (in Ruby) called dupervisor
or dv
https://github.com/kigster/dupervisor that provides a 3-way yaml/json/ini tricoder but no filter/manipulation.
jq
has the following flag as an option:
o -f filename / --from-file filename:
Read filter from the file rather than from a command line, like awk's -f option. You can also use '#' to make comments.
faq
should have something similar.
It will be tricky to find ones that are perfectly compatible with the standard library but options do exist (e.g. jsoniter).
faq
fails with this error:
Error: failed to encode as pretty: xml.Decoder.Token() - XML syntax error on line 1: expected attribute name in element
when fed this valid XML/RSS file:
<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
<title>Read Rust - All</title>
<description><![CDATA[All posts on Read Rust]]></description>
<link>https://readrust.net/</link>
<atom:link rel="self" href="https://readrust.net/all/feed.rss"/>
<lastBuildDate>Tue, 22 Oct 2019 13:00:00 +0000</lastBuildDate>
<item>
<guid isPermaLink="false">79f41426-ad58-4eac-9adb-88b27c2a63ba</guid>
<pubDate>Wed, 9 Oct 2019 00:00:00 +0000</pubDate>
<title>How I handle errors in Rust</title>
<link>https://blog.kiani.io/blog/how-i-handle-errors-in-rust/</link>
<dc:creator>Ashkan Kiani</dc:creator>
<description><![CDATA[derive_more is a crate which has many proc macros, amongst which is a macro for deriving From for structs, enums, and newtypes. From is the basic mechanism for using ? ergonomically in a function which returns Result<T, Error>. Almost everything I write has the derive_more crate as a dependency, and the following pattern for handling errors.]]></description>
</item>
</channel>
</rss>
The error seems to be caused by not handling the angle brackets in Result<T, Error>
inside the CDATA.
gojq is a pure Go libjq implementation.
It does have documented incompatibilities, but it's worth leaving an issue open to see if the community is more interested in 100% compatibility or attempting to avoid the complication of linking against C.
Because I hacked this together, I didn't really structure it with unit tests yet.
Hi, thanks for maintaining a great project!
Would you mind if I contributed this feature?
Context:
I wanted to use faq for manipulating a YAML stream, as well as I use jq for manipulating NDJSON(Or line-delimited JSON).
A YAML stream would look like this. It is basically multiple YAML docs separated by ---
:
$ jsonnet -y /dev/stdin <<< '[{x: 1, y: self.x} + {x: 10}, {z: 2}]'
---
{
"x": 10,
"y": 10
}
---
{
"z": 2
}
...
Current it doesn't work, obviously, as faq supports a YAML array in a single YAML file only, rather than a YAML stream:
$ jsonnet -y /dev/stdin <<< '[{x: 1, y: self.x} + {x: 10}, {z: 2}]' | faq
Error: no supported format found named auto
Usage:
faq [flags] [filter string] [files...]
Flags:
-a, --ascii-output force output to be ascii instead of UTF-8
-C, --color-output colorize the output (default true)
-c, --compact compact instead of pretty-printed output
-f, --format string input format (default "auto")
-h, --help help for faq
-m, --maintain-format maintain original format (don't output JSON)
-M, --monochrome-output monochrome (don't colorize the output)
-r, --raw output raw strings, not JSON texts
-S, --sort-keys sort keys of objects on output
-t, --tab use tabs for indentation
Note that the error message Error: no supported format found named auto
is rather confusing as it doesn't correctly express the fact that faq was just unable to auto-detect the format.
I'll open an another dedicated issue for that!
This would allow people to debug/convert files like x.509 certificates.
Here's a potentially useful library: https://godoc.org/golang.org/x/crypto/cryptobyte
When I send an integer value, it seems to be decoded correctly as it shows as a json number :
$ echo i12e | faq -f bencode -o json .
12
But when I ask that to be re-encoded as bencode, it gets translated into a string:
$ echo i12e | faq -f bencode -o bencode .
2:12
$ echo i12e | faq -f bencode -o bencode . | faq -f bencode -o json .
"12"
I do not speak go, so I am unsure wether the problem is here or in the bencode go library used by faq.
$ faq -v
github.com/jzelinskie/faq/cmd/faq version 0.0.7
github.com/Azure/draft v0.16.0
github.com/BurntSushi/toml v0.3.1
github.com/alecthomas/chroma v0.8.2
github.com/clbanning/mxj/v2 v2.5.5
github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
github.com/dlclark/regexp2 v1.2.0
github.com/ghodss/yaml v1.0.0
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
github.com/go-xmlfmt/xmlfmt v0.0.0-20191208150333-d5b6f63a941b
github.com/jbrukh/bayesian v0.0.0-20200318221351-d726b684ca4a
github.com/sirupsen/logrus v1.8.0
github.com/spf13/cobra v1.1.3
github.com/spf13/pflag v1.0.5
github.com/zeebo/bencode v1.0.0
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
golang.org/x/sys v0.0.0-20200413165638-669c56c373c4
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221
gopkg.in/yaml.v2 v2.4.0
howett.net/plist v0.0.0-20201203080718-1454fab16a06
If we remove the rootCommand out of the main package, it can be imported in developer scripts to generate bash completion and documentation.
like this:
$ curl 'https://www.destructoid.com/?mode=atom' | faq .
Error: failed to jsonify file at /dev/stdin: `xml.Decoder.Token() - xml: encoding "iso-8859-1" declared but Decoder.CharsetReader is nil`
Would be very useful, especially if modification were supported as with jq
. All types should be supported, but binary would output XML or JSON to a terminal.
Property list
Introduction to Property Lists
Go library to use? go-plist
Hello @jzelinskie
I've been trying to install faq on aws ec2 linux and when executing make all it shows me a couple of errors as following:
/usr/bin/ld: cannot find -ljq
/usr/bin/ld: cannot find -lonig
[ec2-user@ip-10-0-134-98 faq]$ ls
cmd Dockerfile docs go.mod go.sum internal LICENSE Makefile pkg README.md
[ec2-user@ip-10-0-134-98 faq]$ make all
golint ./...
go test ./...
# github.com/jzelinskie/faq/internal/jq
/usr/bin/ld: cannot find -ljq
/usr/bin/ld: cannot find -lonig
collect2: error: ld returned 1 exit status
FAIL github.com/jzelinskie/faq/internal/faq [build failed]
ok github.com/jzelinskie/faq/pkg/objconv (cached)
make: *** [test] Error 2
[ec2-user@ip-10-0-134-98 faq]$
The very first example doesn't work
$ echo '{"hello":{"world":"whats up"},"with":"you"}' | faq
Error: no arguments provided
Usage:
faq [flags] [filter string] [files...]
[...]
It also doesn't work with [...] | faq -
.
I would expect the following output
$ echo '{"hello":{"world":"whats up"},"with":"you"}' | faq
{
"hello": {
"world": "whats up"
},
"with": "you"
}
I think the problem came in with commit 78b0985.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.