marrow / cinje Goto Github PK

• Inline (single-line) constructs.
  • Single-line untranslated code.
  • Comments, including double-hash exclusion from the resulting stream.
  • Flush. Ensure actual buffer contents are in fact flushed on demand. (And not when the buffer is assumed empty: direct buffer manipulation is frowned upon, but not impossible.)
  • Text transformations. Ensure escaping happens when expected, etc.
  • Inline embedding of another template.
• Block (nesting scope) constructs.
  • Conditionals: if, elif, and else.
  • Loops, both while and for, with an optional else run-off-the-end chained block.
  • Context managers.
  • Exception handling.
  • Template wrapping.

Notably pip installation directly from the Git repo for those not needing to modify or contribute, and a brief outline of how to add Git-based dependencies to the application setup.py.

Reminder: Update the Marrow project blueprint README template with this information, too.

For examples:

: use myapp.templates.master:master_template
: using foo.bar:baz

Currently only one directive attempts to split anything on an expression, the cinje.inline.text processor when handling %{expr arg} formatted string replacements. This is broadly useful, notably for the : use and : using directives amongst others, and better aligns with the documentation.

• Extract the cinje.inline.text "expression split" code into the Line class. (As a companion for the partitioned attribute.)
• Update cinje.inline.use directive to utilize the expression split.
• Update cinje.block.using directive to utilize the expression split.

Simple test case:

# encoding: cinje

: def tmpl

A sample of Unicode™ text.

Without the ™ symbol, all is well. With it, a substantial traceback is produced:

/Users/amcgregor/Projects/cegid/cinje/cinje/encoding.pyc in cinje_decode(input, errors, final)
     18 def cinje_decode(input, errors='strict', final=True):
     19         if not final: return '', 0
---> 20         output = transform(bytes(input).decode('utf8', errors))
     21         return output, len(input)
     22 

/Users/amcgregor/Projects/cegid/cinje/cinje/encoding.pyc in transform(input)
     12 def transform(input):
     13         #__import__('pudb').set_trace()
---> 14         translator = Context(input)
     15         return '\n'.join(str(i) for i in translator.stream)
     16 

/Users/amcgregor/Projects/cegid/cinje/cinje/util.pyc in __init__(self, input)
    449 
    450         def __init__(self, input):
--> 451                 self.input = Lines(input.decode('utf8') if isinstance(input, bytes) else input)
    452                 self.scope = 0
    453                 self.flag = set()

/Users/amcgregor/Projects/cegid/cinje/cinje/util.pyc in __init__(self, input, Line)
    397 
    398                 else:
--> 399                         self.source = list(self.Line(i + 1, j) for i, j in enumerate(input.split('\n')))
    400                         self.buffer = deque(self.source)
    401 

/Users/amcgregor/Projects/cegid/cinje/cinje/util.pyc in <genexpr>((i, j))
    397 
    398                 else:
--> 399                         self.source = list(self.Line(i + 1, j) for i, j in enumerate(input.split('\n')))
    400                         self.buffer = deque(self.source)
    401 

/Users/amcgregor/Projects/cegid/cinje/cinje/util.pyc in __init__(self, number, line, scope, kind)
    318         def __init__(self, number, line, scope=None, kind=None):
    319                 if isinstance(line, str):
--> 320                         line = line.decode('utf-8')
    321 
    322                 self.number = number

/Users/amcgregor/Projects/cegid/rita/.venv/lib/python2.7/encodings/utf_8.pyc in decode(input, errors)
     14 
     15 def decode(input, errors='strict'):
---> 16     return codecs.utf_8_decode(input, errors, True)
     17 
     18 class IncrementalEncoder(codecs.IncrementalEncoder):

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2122' in position 19: ordinal not in range(128)

Looks like a double decoding: util.py:451, util.py:320 (incorrect isinstance check if normalized str has been imported from compat)

• Conditionals: if, elif, and else.
• Loops, both while and for, with an optional else run-off-the-end chained block.
• Context managers.
• Exception handling.

Allow and adapt to async def usage. This ticket, vs. #9, is more about integration into an asyncio environment and less about inverting the template rendering call tree.

Thinking something like an extruded _⊓_⊓_ corrugated tin sheet to match the meanings of the name.

Can be used by a wide range of programmers' editors.

• Basic flag capture. Use of Python 3 function return type annotations will be treated as a comma separated list of flags to add, or, with the presence of an exclamation mark prefix, remove flags from the cinje translation context.

Some ideas for flags:

• Unbuffered. Instead of yielding at breaks and at the end (if the buffer contains anything), with buffer resets and all that nonsense, skip the buffer allocation and local-scope method assignment. Yield at each contiguous block of text, and "yield from" (or fallback equivalent) all nested template use. (Prefix goes from __w(( to yield "".join((.)

https://github.com/youtube/spitfire/tree/master/tests/benchmarks

https://github.com/youtube/spitfire

Looking for ideas on how to do this one. Stop-gap measure: include a "source line number" comment on every generated line originating in the source file? We are decorating each produced function, so in theory we can actually grab the exception itself and manipulate it. Comments are stripped from bytecode in optimized execution modes, so we'd need to store line number mapping somewhere else…

External References

• Stack Overflow Question

The following example produces strangeness:

: def template
    : if False
        Hi.

This generates the following code (pre-1.0):

def template(*, _escape=_escape, _bless=_bless, _args=_args):
    if False:
        _buffer = []
        __w, __ws = _buffer.extend, _buffer.append

        __ws('            Hi.\n')

    yield "".join(_buffer)

The _buffer allocation being itself conditional results in problems. Lame, naive solution: have all template functions call ensure_buffer immediately.

Note: Unlike #2, this ticket is more about the inversion of the template include pattern and less about the pure integration of cinje into an asyncio environment, though such integration does need to be kept in mind.

Currently the yield mechanics are used to identify an insertion point for additional content in a template, creating a "wrapping" template function. This gives you an entry point of the most specific template, which then typically dives into its wrapper, then deeper into whatever that is wrapped by internally, etc., etc.

The inverse approach, where you have a "layout" template which you populate with content blocks, is also extremely useful. Because templates are first-class functions, you can easily pass individual, or whole lists of template functions around, using functools.partial to bind values to them for use at render-time. This is a start, as during template processing the layout template would descend into each content block to render, blocking the overall process. Instead, when a child template is encountered, if it's an async def it can be sent up to the reactor for processing. In the resulting HTML insert a placeholder or marker (a la the animated bars on Facebook posts as they load), finally streaming the layout template, then individual content blocks, as MIME multipart. Should also optionally be able to execute the template more classically, without async deferral.

Streaming chunks to the browser as completed can be accomplished via MIME multipart AJAX, via something like mpAjax. A rough p-code example using Futures:

def layout(reactor, *blocks):
    """Mock boostrap row/column layout."""

    # Prepare some content.
    _buffer = []
    _tasks = []

    for block in blocks:
        if not _buffer or not block:
            if _buffer:
                _buffer.extend(('</div><div class="row">\n', ))

            _buffer.extend(('<div class="row">\n', ))

            if not block:
                continue

        _tasks.append(reactor.submit(block))
        _buffer.extend(('<div class="placeholder" data-await="', id(_tasks[-1]), '"></div>\n'))

    return (''.join(_buffer), _tasks)


def render_page():
    identifier = "gc0p4Jq0M2Yt08jU534c0p"

    response.content_type = "multipart/mixed; boundary=" + identifier

    page, content = layout(executor, [
            "some render function, first column",
            "some other render function, second column",
            None,
            "lastly a full width footer",
        ])

    yield page

    for chunk in as_completed(content):
        yield '--' + identifier + '\nIdentifier: ' + str(id(chunk)) + '\n\n' + chunk.result()

    yield '--' + identifier + '--\n'

Python 3.7 has removed this as an acceptable behaviour, as such, cinje breaks on Python 3.7 as demonstrated after adding 3.7 to the Travis-CI test rig, and exploring explosions locally (which are either hilariously non-existent or egregiously verbose exception chains).

This allows us to execute the encoding import and registration on Python startup.

An example approach can be seen in the setup.py for pytest-cov.

They have no effect, an only clutter the generated code.

Hey there,

Seems not to run on python 3.9:

$ python benchmark.py
Traceback (most recent call last):
  File "/home/ybon/Code/py/cinje/example/benchmark.py", line 445, in <module>
    import bigtable
  File "/home/ybon/Code/py/cinje/example/bigtable.py", line 106
    __gzmapping__ = b"eJxjYGJkAANGMEBmMDAxMTAjCWETAwsRlmBgAAAPKwBL"
                                                                  ^
SyntaxError: unexpected EOF while parsing

or am I missing something ?

Thanks :)

https://gist.github.com/amcgregor/edddabd010d3b2f21cc21add58ff0a03

• Pure non-interactive command-line, optionally with coloured output.
  • Useful when combined with Tox to gather stats across many Python runtimes.
• "Terminal GUI" operation with an urwid or similar curses interface for exploring the benchmarks:
  • Available engines in a list/table, with non-installed engines available for installation, but hidden.
  • Ability to view example template and code to render the template.
  • Sort, the usual. Export in a variety of formats and otherwise replicate the CLI options.
• Should probably be a separate package or tool. ;)

According to nedbat on IRC, coverage.py version 4.0 adds the capability to write plugins. We should totally do this.

Some resources to review to facilitate this:

• syn-include
• new-filetype-scripts

WebCore tutorial step using Bless #{} will return:

SyntaxError: decoding with 'cinje' codec failed (AssertionError: Unable to identify handler for line; this should be impossible!)

Test case (based on WebCore tutorial)
https://www.irccloud.com/pastebin/raw/fJubH7dP

Where currently you require multiple lines with explicit flow control statements:

: for item in items
	: if item % 2 == 0
		: continue
	: end

Permit inclusion of filtering conditions (one or more) in the form:

: for item in items if item % 2

There are several avenues for translation to plain Python for these. Unwrap and reconstruct the multiple-line form, or alternatively wrap in a generator comprehension:

for item in (i for i in items if item % 2):

Simplest test case:

# encoding: cinje

: def testcase
    : for i in ('hello', 'world')
        ${i}

Ignoring default imports, optimizations, and line mapping, the transformed result is currently:

def testcase():
    for i in ('hello', 'world'):
        _buffer = []
        __w, __ws = _buffer.extend, _buffer.append

        __w((
            '\t\t',
            _escape(i),
            '\n'
        ))

    yield "".join(_buffer)

Note the dangerous location of buffer construction. If the loop were empty, it would not be called, but the code transformer considers the buffer prepared after that point regardless. In the empty test case, the subsequent yield will explode as _buffer would be undefined. Additionally, the buffer is overwritten on each iteration here.

Temporary workaround: include some emitted content prior to a function's first iterator. An HTML comment works well.

Simple fix: force buffer construction after function declaration.