Any reason not to yield each output value? about jq.py HOT 6 CLOSED

Eh, Python is the land of generators. I totally expect Python programmers to get that jq is a generator.

Besides that, if a given jq program will be outputting lots of values, then using a generator will be more "online" than collecting them into an array and outputting the array.

from jq.py.

master now has a new API that supports producing an iterator for the output.

from jq.py.

Mainly that people tend to understand lists better than generators, and they're easier to deal with. If there's a compelling reason to use generators instead, then I'm open to suggestions.

from jq.py.

@mwilliamson I have to completely agree with @nicowilliams .
Python really shines in generators, iterators (very well done in Python 2.x, even more in Python 3.x), what allows to process huge streams of data with really small amount of memory needed just for processing part of the task (something on level of one record transformation).

I accept, that generators are not what beginners understand from very first lesson, but it is definitely a feature, which makes Python really strong tool and it is not so complex as it looks at the first moment.

As @nicowilliams has mentioned, in case someone would process large (or possibly indefinite) stream of json data, generators are the only way to go.

from jq.py.

I've been playing with exposing a generator in a separate branch (see the tests). One issue I've come across is that potentially many instances of jq_state are required for the same program since many executions could be on-going at one time for the same program (since we no longer guarantee that the execution is completed in one go).

Is there a cheap way to copy a compiled program (i.e. copy jq_state immediately after jq_compile)? cc @nicowilliams

from jq.py.

@mwilliamson libjq should be reentrant, so multiple jq_state instances should be no problem. We don't have a way to copy a jq_state instance, but so what, that's just an optimization that we can add.

from jq.py.