For the full build log see https://cloud.drone.io/vielmetti/faq/12/1/2 based on the Drone Cloud configuration at https://github.com/vielmetti/faq/blob/vielmetti-drone/.drone.yml

=== RUN TestRunFaq/null_flag_input_simple_program
--
109 | faq.test: execute.c:1071: jq_compile_args: Assertion `jv_get_kind(args) == JV_KIND_ARRAY' failed.

I suspect that this is because I'm doing a Ubuntu-based build (using the "golang" Docker image as a base) and because jq in that image is 1.5 version.

As mentioned in #11 Drone Cloud has a 32-bit Arm build/test environment as well as arm64 and x86. This issue to track adding a Drone configuration (.drone.yml) and explore this build environment for faq.

like this:

$ curl 'https://www.destructoid.com/?mode=atom' | faq .

Error: failed to jsonify file at /dev/stdin: `xml.Decoder.Token() - xml: encoding "iso-8859-1" declared but Decoder.CharsetReader is nil`

From jq manpage:

       ·   -e / --exit-status:

           Sets the exit status of jq to 0 if the last output values was neither false nor  null,
           1  if the last output value was either false or null, or 4 if no valid result was ever
           produced. Normally jq exits with 2 if there was any usage problem or system  error,  3
           if there was a jq program compile error, or 0 if the jq program ran.

Hello, I started using faq but I found what I think is an issue.

When using raw output mode, faq breaks trying to pretty encode the output. I would have expected that raw output would override all enhancements to the output, but it does not seem the case.

How to reproduce:

$ echo '"hello world"' | faq --input-format=json -r "."
Error: failed to encode as pretty: invalid character 'h' looking for beginning of value

Adding --pretty-output=false to the above command makes it work:

$ echo '"hello world"' | faq --input-format=json -r --pretty-output=false "."
hello world

Is this the expected --raw-output behaviour?

While reviewing this project to figure out if it does what I'm looking for, I came across this line in the README:

How do you pronounce "faq"? Fuck you.

It strikes me as very hostile. The README is the first contact many users have with a project, and this one is telling me "fuck you" within the first few paragraphs. As it stands, I wouldn't recommend this project to anyone, because I'd have to explain that line in the README. I'm sure that there are plenty of people that turned away immediately and didn't say anything.

faq prints a confusing error message when it failed to auto-detect the input format:

$ jsonnet -y /dev/stdin <<< '[{x: 1, y: self.x} + {x: 10}, {z: 2}]' | faq
Error: no supported format found named auto
Usage:
  faq [flags] [filter string] [files...]

Flags:
  -a, --ascii-output        force output to be ascii instead of UTF-8
  -C, --color-output        colorize the output (default true)
  -c, --compact             compact instead of pretty-printed output
  -f, --format string       input format (default "auto")
  -h, --help                help for faq
  -m, --maintain-format     maintain original format (don't output JSON)
  -M, --monochrome-output   monochrome (don't colorize the output)
  -r, --raw                 output raw strings, not JSON texts
  -S, --sort-keys           sort keys of objects on output
  -t, --tab                 use tabs for indentation

It will be tricky to find ones that are perfectly compatible with the standard library but options do exist (e.g. jsoniter).

I spent a little time trying to see if I could generically support asn.1.

I found that supporting it would probably mean actually having schemas for various things that are encoded in that format (e.g. CAs, private keys). I think supporting files like these would be a huge boon to debugging, but also complicate the project to the point where it almost seems like a different goal: representing everything as JSON (or more generically "objects").

I think it's as simple as

git clone https://github.com/jzelinskie/faq
cd faq
docker build .

to build using the supplied Dockerfile; but I couldn't find where the generated binary by the build was supposed to go.

The very first example doesn't work

$ echo '{"hello":{"world":"whats up"},"with":"you"}' | faq
Error: no arguments provided                                                                                                                                                          
Usage:                                                                                                                                                                                
  faq [flags] [filter string] [files...]                                                                                                                                              
[...]

It also doesn't work with [...] | faq -.
I would expect the following output

$ echo '{"hello":{"world":"whats up"},"with":"you"}' | faq     
{                                                                                          
  "hello": {                                                                               
    "world": "whats up"                                                                    
  },                                                                                       
  "with": "you"                                                                            
}                                                                                          

I think the problem came in with commit 78b0985.

This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".

There used to be a 0.0.8 tag but it seems to have vanished... Can somebody please explain why this is?

is it possible to output selected fields using a custom format string like this?

jq '.[] | "foo \(.FIELDNAME2) bar \(.FIELDNAME5)" ...'

Hello @jzelinskie

I've been trying to install faq on aws ec2 linux and when executing make all it shows me a couple of errors as following:
/usr/bin/ld: cannot find -ljq
/usr/bin/ld: cannot find -lonig

[ec2-user@ip-10-0-134-98 faq]$ ls
cmd  Dockerfile  docs  go.mod  go.sum  internal  LICENSE  Makefile  pkg  README.md
[ec2-user@ip-10-0-134-98 faq]$ make all
golint ./...
go test ./...
# github.com/jzelinskie/faq/internal/jq
/usr/bin/ld: cannot find -ljq
/usr/bin/ld: cannot find -lonig
collect2: error: ld returned 1 exit status
FAIL    github.com/jzelinskie/faq/internal/faq [build failed]
ok      github.com/jzelinskie/faq/pkg/objconv   (cached)
make: *** [test] Error 2
[ec2-user@ip-10-0-134-98 faq]$

Hi! Great work! Learn a lot about cgo and jq internals from this project.

I start my own project to use jq programs in Go projects with ability to cache compiled jq state. So here are some findings of jq_start behavior:

jq_start does two important things:

• it calls jq_reset that free everything on the stack
• it pushes input jv pointer to the stack

That stack is freed by a jq_teardown call that will call the same jq_reset. But input jv is already freed by the deferred call ;) That is why you need to use jv_copy here: https://github.com/jzelinskie/faq/blob/master/pkg/jq/exec.go#L237

Hello.

First of all, thanks a lot for the tool.
I am, however, encountering problems when dealing with data encoded with bencode.
It's a problem I've come across time and again and hopefully one you can address.
From what I've seen you've interpreted the string length as the number of bytes the string is encoded as, which should be fine.
Since, I guess, the original specs of the format, if we can call them that, were less than crystal clear as to what string length meant, there are many implementations around interpreting the string length as the character count, in Unicode terms the count of codepoints present in the string.
Could you create a variant of the bencode format supported by faq that matches the variant interpretation of string length described above? It would make my life a lot easier dealing with these sorts of systems.

Just as a reference, faq would encode (arguably correctly) the JSON { "a": "à" } as the bencode-d d1:a2:àe, while the variant format would encode it as d1:a1:àe, assuming UTF-8 encoded strings.

Thanks in advance.

In the future, I'd like to have ARM for darwin and linux built into Github Actions.

gojq is a pure Go libjq implementation.
It does have documented incompatibilities, but it's worth leaving an issue open to see if the community is more interested in 100% compatibility or attempting to avoid the complication of linking against C.

jq has the following flag as an option:

       o   -f filename / --from-file filename:

           Read filter from the file rather than from a command line, like awk's -f option. You can also use '#' to make comments.

faq should have something similar.

This is an inconsistency between the two programs that I'd like to fix.

Because I hacked this together, I didn't really structure it with unit tests yet.

goreleaser would vastly improve the workflow around cutting releases.
If it can support cgo and embedding the version flag, it could probably replace the Makefile.

Right now the faq.Flags type is a grab bag of config values and a lot of things read and write to it.

Ideally it would only be used by the pkg/command package, and we would instead call faq.RunFaq with something more specific, only containing options needed after command line handling, and preferrably, all the basic flag/conf validation can be done before faq.RunFaq

faq fails with this error:

Error: failed to encode as pretty: xml.Decoder.Token() - XML syntax error on line 1: expected attribute name in element

when fed this valid XML/RSS file:

<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>Read Rust - All</title>
    <description><![CDATA[All posts on Read Rust]]></description>
    <link>https://readrust.net/</link>
    <atom:link rel="self" href="https://readrust.net/all/feed.rss"/>
    <lastBuildDate>Tue, 22 Oct 2019 13:00:00 +0000</lastBuildDate>
    <item>
      <guid isPermaLink="false">79f41426-ad58-4eac-9adb-88b27c2a63ba</guid>
      <pubDate>Wed, 9 Oct 2019 00:00:00 +0000</pubDate>
      <title>How I handle errors in Rust</title>
      <link>https://blog.kiani.io/blog/how-i-handle-errors-in-rust/</link>
      <dc:creator>Ashkan Kiani</dc:creator>
      <description><![CDATA[derive_more is a crate which has many proc macros, amongst which is a macro for deriving From for structs, enums, and newtypes. From is the basic mechanism for using ? ergonomically in a function which returns Result<T, Error>. Almost everything I write has the derive_more crate as a dependency, and the following pattern for handling errors.]]></description>
    </item>
  </channel>
</rss>

The error seems to be caused by not handling the angle brackets in Result<T, Error> inside the CDATA.

Hello @jzelinskie

I've been working with the JSON to Bencode conversion, but there are things that don't seem to make sense to me.
I was looking at how your tool handled things such as null values or booleans when converting to bencode (that doesn't have those types) and this is what came up.

$ faq -n -f json -o bencode '[1,null,2]'
l1:11:2e

# Why are integers converted to strings??? And I thought that null values would be converted to an empty string, for instance. Here they just disappear... which would mess up the length of the list.

$ faq -n -f json -o bencode '{a:null}'
de

# Entire pairs disappearing, although it could make some degree of sense

$  faq -n -f json -o bencode '[true,false]'
li1ei0ee

# Ok now that is something I expected

Now... is there a way to choose how these conversions take place, for instance how a false in JSON would convert to Bencode, for instance as an empty string rather than a 0?

Thanks in advance

Hi, thanks for maintaining a great project!

Would you mind if I contributed this feature?

Context:

I wanted to use faq for manipulating a YAML stream, as well as I use jq for manipulating NDJSON(Or line-delimited JSON).

A YAML stream would look like this. It is basically multiple YAML docs separated by ---:

$ jsonnet -y /dev/stdin <<< '[{x: 1, y: self.x} + {x: 10}, {z: 2}]'
---
{
   "x": 10,
   "y": 10
}
---
{
   "z": 2
}
...

Current it doesn't work, obviously, as faq supports a YAML array in a single YAML file only, rather than a YAML stream:

$ jsonnet -y /dev/stdin <<< '[{x: 1, y: self.x} + {x: 10}, {z: 2}]' | faq
Error: no supported format found named auto
Usage:
  faq [flags] [filter string] [files...]

Flags:
  -a, --ascii-output        force output to be ascii instead of UTF-8
  -C, --color-output        colorize the output (default true)
  -c, --compact             compact instead of pretty-printed output
  -f, --format string       input format (default "auto")
  -h, --help                help for faq
  -m, --maintain-format     maintain original format (don't output JSON)
  -M, --monochrome-output   monochrome (don't colorize the output)
  -r, --raw                 output raw strings, not JSON texts
  -S, --sort-keys           sort keys of objects on output
  -t, --tab                 use tabs for indentation

Note that the error message Error: no supported format found named auto is rather confusing as it doesn't correctly express the fact that faq was just unable to auto-detect the format.
I'll open an another dedicated issue for that!

Hi!

I would love to try out this project, it looks amazing!!

Unfortunately, in the builds/release section, I can only find Linux and Mac builds (and I don't know how to satisfy the JQ headers requirement).

I also tried installing the normal way using GO:
go install github.com/jzelinskie/faq/cmd/faq@latest

package github.com/jzelinskie/faq/cmd/faq
        imports github.com/jzelinskie/faq/internal/faq
        imports github.com/jzelinskie/faq/internal/jq: build constraints exclude all Go files in ...\go\pkg\mod\github.com\jzelinskie\[email protected]\internal\jq

Thank you for your consideration!

It would be very handy and helpful if faq were available through Homebrew for macOS.

This is going to require adding a bunch of encoders and moving some of the logic around syntax reimplementing the lbjq print flags in Go.

Any consderation of support for .ini files, the somewhat standardized format often used for config files?

I see a Go library https://github.com/go-ini/ini that could be useful, and there's a proof of concept (in Ruby) called dupervisor or dv https://github.com/kigster/dupervisor that provides a 3-way yaml/json/ini tricoder but no filter/manipulation.

If we remove the rootCommand out of the main package, it can be imported in developer scripts to generate bash completion and documentation.

Because libjq is relatively new, many Linux distributions don't have it packaged and linkable.

Ideally, I can figure out the right cgo incantations to embed a version of libjq into a totally portable Linux amd64 binary.

Would be very useful, especially if modification were supported as with jq. All types should be supported, but binary would output XML or JSON to a terminal.

Property list
Introduction to Property Lists

Go library to use? go-plist

This needs to be a linker flag that we use in the Makefile

faq should work just fine on armhf (32-bit Arm, Raspberry Pi), once it's built. A request in several parts:

1. Document the manual process for building on 32-bit Arm for Raspberry Pi. This may include explicit instructions for dealing with dep, specific version requirements for go, and any library compatibility issues along the way.
2. Once the manual process is simplified enough, incorporate it into the CI/CD system build process and emit armhf (32-bit) binaries for easy consumption.

When I send an integer value, it seems to be decoded correctly as it shows as a json number :

$ echo i12e | faq -f bencode -o json .
12

But when I ask that to be re-encoded as bencode, it gets translated into a string:

$ echo i12e | faq -f bencode -o bencode .
2:12
$ echo i12e | faq -f bencode -o bencode . | faq -f bencode -o json .
"12"

I do not speak go, so I am unsure wether the problem is here or in the bencode go library used by faq.

$ faq -v
github.com/jzelinskie/faq/cmd/faq version 0.0.7
	github.com/Azure/draft v0.16.0
	github.com/BurntSushi/toml v0.3.1
	github.com/alecthomas/chroma v0.8.2
	github.com/clbanning/mxj/v2 v2.5.5
	github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
	github.com/dlclark/regexp2 v1.2.0
	github.com/ghodss/yaml v1.0.0
	github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
	github.com/go-xmlfmt/xmlfmt v0.0.0-20191208150333-d5b6f63a941b
	github.com/jbrukh/bayesian v0.0.0-20200318221351-d726b684ca4a
	github.com/sirupsen/logrus v1.8.0
	github.com/spf13/cobra v1.1.3
	github.com/spf13/pflag v1.0.5
	github.com/zeebo/bencode v1.0.0
	golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
	golang.org/x/sys v0.0.0-20200413165638-669c56c373c4
	golang.org/x/term v0.0.0-20201117132131-f5c789dd3221
	gopkg.in/yaml.v2 v2.4.0
	howett.net/plist v0.0.0-20201203080718-1454fab16a06

vbatts@infra1:~$ curl -s https://api.github.com/repos/flatcar-linux/manifest/releases | faq -f json '.[0].tag_name'
"v2492.0.0"
vbatts@infra1:~$ curl -s https://api.github.com/repos/flatcar-linux/manifest/releases | faq -f json -r '.[0].tag_name'
Error: failed to encode as pretty: invalid character 'v' looking for beginning of value
Usage:
  faq [flags] [filter string] [files...]
[...]
vbatts@infra1:~$ curl -s https://api.github.com/repos/flatcar-linux/manifest/releases | jq -r '.[0].tag_name'
v2492.0.0

faq seems to be expected raw values to act a certain way?

• make faq-linux-amd64
• make faq-darwin-amd64
• make dist/bash-completion

$ curl -s https://www.govinfo.gov/bulkdata/CFR/2020/title-14/CFR-2020-title14-vol2.xml | faq
Error: failed to encode as: invalid attribute key label: #text - due to attributes not being prefixed
Usage:
...

This works...

$ curl -s https://www.govinfo.gov/content/pkg/CFR-2020-title14-vol2/xml/CFR-2020-title14-vol2-sec91-205.xml | faq

This would allow people to debug/convert files like x.509 certificates.

Here's a potentially useful library: https://godoc.org/golang.org/x/crypto/cryptobyte

Trying to build an RPM using https://raw.githubusercontent.com/chancez/faq-rpm/master/faq.spec and running into the following:

/usr/bin/make -j56 SKIP_VALIDATE=true FAQ_LINK_STATIC=false 'GO_LD_FLAGS=-compressdwarf=false -linkmode external -extldflags "-v"'
/usr/bin/make test
make[1]: Entering directory `/builddir/build/BUILD/faq-0.0.6'
scripts/test.sh
go: downloading github.com/sirupsen/logrus v1.3.0
go: downloading github.com/spf13/cobra v0.0.3
go: downloading golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading github.com/globalsign/mgo v0.0.0-20180615134936-113d3961e731
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/Azure/draft v0.16.0
go: downloading github.com/BurntSushi/toml v0.3.1
go: downloading github.com/clbanning/mxj v0.0.0-20180418195244-1f00e0bf9bac
go: downloading github.com/ghodss/yaml v1.0.0
go: downloading github.com/alecthomas/chroma v0.6.2
go: downloading github.com/spf13/pflag v1.0.3
go: downloading github.com/zeebo/bencode v0.0.0-20180308174530-d522839ac797
go: downloading github.com/dlclark/regexp2 v1.1.6
go: downloading github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
go: downloading golang.org/x/sys v0.0.0-20181128092732-4ed8d59d0b35
go: downloading github.com/jbrukh/bayesian v0.0.0-20161210175230-bf3f261f9a9c
# github.com/jzelinskie/faq/pkg/jq
_cgo_export.c: In function 'goJQErrorHandler':
_cgo_export.c:29:3: error: expected specifier-qualifier-list before 'jv'
   jv p1;
   ^
_cgo_export.c:32:8: error: 'struct <anonymous>' has no member named 'p1'
  _cgo_a.p1 = jv;
        ^
FAIL	github.com/jzelinskie/faq/pkg/faq [build failed]
=== RUN   TestBencodeMarshal
=== RUN   TestBencodeMarshal/#00
--- PASS: TestBencodeMarshal (0.00s)
    --- PASS: TestBencodeMarshal/#00 (0.00s)
=== RUN   TestBencodeUnmarshal
=== RUN   TestBencodeUnmarshal/#00
--- PASS: TestBencodeUnmarshal (0.00s)
    --- PASS: TestBencodeUnmarshal/#00 (0.00s)
=== RUN   TestSVG
--- PASS: TestSVG (0.00s)
PASS
ok  	github.com/jzelinskie/faq/pkg/formats	0.024s
make[1]: *** [test] Error 2
make[1]: Leaving directory `/builddir/build/BUILD/faq-0.0.6'
make: *** [all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.ktd4Or (%build)
RPM build errors:
    bogus date in %changelog: Mon Feb 14 2019 Chance Zibolski <[email protected]> 0.0.6-1
    Bad exit status from /var/tmp/rpm-tmp.ktd4Or (%build)
Finish: rpmbuild faq-0.0.6-4.el7.src.rpm
Finish: build phase for faq-0.0.6-4.el7.src.rpm
ERROR: Exception(/tmp/faq-0.0.6-4.el7.src.rpm) Config(epel-7-x86_64) 1 minutes 26 seconds
INFO: Results and/or logs in: /home/builder/rpmbuild/results/epel-7-x86_64
INFO: Cleaning up build root ('cleanup_on_failure=True')
Start: clean chroot
Finish: clean chroot
ERROR: Command failed: 

It would be great if we could configure the output format explicitly. Would be really useful for simple conversions too.

With separators auto-guessing and optional overrides:
https://en.wikipedia.org/wiki/Comma-separated_values

faq -f json -o csv < file.json

I'm only filing this because the README states that they are:

$ faq
dyld: Library not loaded: /usr/local/opt/jq/lib/libjq.1.dylib
  Referenced from: /Users/aat/Library/Caches/hermit/pkg/faq-0.0.7/faq
  Reason: image not found