UTF-8 crash - `Assertion failed: (utf8_validate(str))` about jo HOT 5 CLOSED

Understood, and thanks for piping up. Your issue actually surfaced a bug in how jo handles reads from stdin, and I'll push out a PR to fix it in short order.

Also, if I understand correctly, you're basically trying to convert a JSON Lines document into a JSON array. That's actually quite easily done in one Bash line, no jo or loops necessary:

( readarray -t lines < full-dataset.json ; IFS=, ; echo "[${lines[*]}]" ) > single-array.json

(The use of a subshell is just so I don't have to clean up lines and reset IFS.)

from jo.

I can replicate both your issues on macOS. jo on Linux handles bad-inference.json just fine:

linux$ jo -a < bad-inference.json
[{"timezone":"Africa/Algiers","cities":["Abou el Hassan"...

but treats utf-8-crash.json as a string:

linux$ jo -a < utf-8-crash.json
["{\"timezone\":\"Asia/Kolkata\",\"cities\":[\"Abhayāpuri\",\"Abohar\",...

So something's hinky here, but the stopgap solution is already documented in the jo man page:

jo normally treats value as a literal string value, unless it begins with one of the following characters:

value   action
──────────────────────────────────────────────────────────────────────────
@file   substitute the contents of file as-is
%file   substitute the contents of file in base64-encoded form
:file   interpret the contents of file as JSON, and substitute the result

jo -a :utf-8-crash.json and jo -a :bad-inference.json work exactly as advertised on both Linux and macOS.

from jo.

That may well be, but that's not how this data is being passed to jo in my use case. It's only written to a file so that was able to pass provide the data here. The data is being piped to jo and both examples are just a single record from a big pile of similar ones.

That said, I'd be happy to ingest the data via a tempfile on disk, but because it isn't a valid json file (it's a json object per line), jo refuses to ingest it.

Here's what the loop which generates the json looks like, to give you a better idea of my use case:

while IFS="|" read -r timezone cities; do
	jo timezone="$timezone" \
	   cities="$(while read -rd ";" city; do echo "$city"; done <<<"${cities};" | jo -a)"
done < "$sqlite_out" | jo -a > "$json_out"

This problem is already solved (using a very ugly string concatenating loop), so I do have a working stopgap solution. I'm not here because I can't solve the problem, I just figured you might be interested in the issue. And truthfully I would love to be able to use jo for this kind of work, it makes it so much more convenient.

from jo.

@nikcorg we’ve just merged a patch by @gromgit which addresses reading long lines in jo. Would you like to see if this fixes your use-case?

from jo.

Yes it does. Thank you both very much :-)

from jo.