json-pull
A streaming JSON pull parser in C.
Does the world really need another JSON parser? Maybe not. But what distinguishes this one is the desire to let you read through gigantic files larger than you can comfortably fit in memory and still find the parts you need.
It builds a parse tree, but you can tear down as much of the tree as you don't need while it is still in the middle of parsing.
Setup
You can create a json_pull
parser object
by calling json_begin_file()
to begin reading from a file
or json_begin_string()
to begin reading from a string.
You can also read from anything you want by calling json_begin()
with your own read()
function that copies UTF-8 bytes
from your input stream.
Reading full JSON trees
The simplest form is json_read_tree()
, which reads a complete
JSON object from the stream, or NULL if there was an error or on end of file.
Call json_free()
on the object when you are done with it.
You can continue calling json_read_tree()
to read additional objects
from the same stream. This is not standard JSON, but is useful for something like
the Twitter filter stream that contains a series of JSON objects separated by
newlines, without a wrapper array that contains them all. (The previous parse tree
will automatically have json_free()
called on it to prevent memory leaks, unless
you have called json_disconnect()
to disconnect it from the parser.)
In addition, extra commas at the top level are no longer flagged as an error, so a series of comma-separated items can be read without their container array.
Reading JSON streams
With something like GeoJSON, where it is common to have a large wrapper array that contains many smaller items that are often useful to consider separately, you can read the stream one token at a time.
The normal form is json_read()
, which returns the next complete
object from the stream. This can be a single string, number, true
,
false
, or null
, or it can be an array or hash that
contains other primitive or compound objects.
Note that each array or hash will be returned following all the objects that it contains.
The outermost object will be the same one that json_read_tree()
would
have returned, and you can tell that it is the outer object because its
parent
field is null.
You can call json_free()
on each object as you are finished with it,
or wait until the end and call json_free()
on the outer object
which will recursively free everything that it contains. Freeing an object before
its container is complete also removes it from its parent array or hash so that
there are not dangling references left to it.
If you want to keep an individual JSON object around for later processing
while allowing its parents and siblings to be freed, json_disconnect()
will detach an object from its container. You should call json_free()
later to free the detached object once you are finished with it.
Reading JSON streams with callbacks
If you are outputting a new stream as you read instead of just looking for the
sub-objects that interest you, you also need to know when arrays and hashes begin,
not just when they end, so you can output the opening bracket or brace. For this
purpose there is an additional streaming reader function,
json_read_separators()
, which takes an additional argument for
a function to call when brackets, braces, commas, and colons are read.
Other object types and the closing of arrays and hashes are still sent through
the normal return value.
The types that can be sent to the callback function are
JSON_ARRAY
, JSON_HASH
, JSON_COMMA
,
and JSON_COLON
.
Cleanup
If there was an error while parsing, the parser will have returned NULL before
all the containers were closed. You will probably want to call json_free()
on the root
elemement of the parser, which should contain the full parse
tree so far, to avoid leaking memory.
Shutdown
To free the parser object, call json_end()
on it.
Object format
JSON objects are represented as the C struct json_object
.
It contains a type
field to indicate its type, a parent
pointer to the container that encloses it, and additional fields per type.
Types JSON_NULL
, JSON_TRUE
, and JSON_FALSE
have no additional data.
Strings have type JSON_STRING
, with null-terminated UTF-8 text
in string
and length in length
.
Numbers have type JSON_NUMBER
, with value in number
,
and also preserve the original representation of the number
in string
and length in length
.
Arrays have type JSON_ARRAY
. There are length
elements in the array,
and the elements are in array
.
Hashes have type JSON_HASH
. There are length
key-value pairs,
and the keys are in keys
and the values in values
.
Parser format
The parser object has two fields of public interest: error
is a string
describing any errors found in the JSON, and line
is the current line number
being read from the input to make it easier to find the errors.
The root
field points to the outer object of the current parse tree.
Utility function
There is a function json_hash_get()
that looks up the JSON object hash value
corresponding to a C string hash key in a JSON hash object. If the object specified is
NULL or not a JSON hash or has no matching key, it returns NULL.
Another function json_stringify()
converts a JSON parse tree back to its
string representation. The caller is responsible for calling free()
on the
return value once it is no longer needed.
Test program
The jsoncat
program reads JSON from the standard input or from
named files and pretty-prints it to the standard output.
Normally it uses the callback interface to avoid memory overhead. It has three options to exercise different aspects of the library:
-t
reads the whole tree and then prints it.-i
reads incrementally, but still keeps it all in memory.-s
reads the file into a string before parsing it with callbacks.
GeoJSON feature extraction
The geojson2nd
program reads JSON from the standard input input or from named files
and outputs each object where object.type === "Feature'
to the standard output on
a separate line. The resulting stream can then conveniently be used as input to
the ndjson-cli tools.