Raising an issue for a subsequent PR.

the compiler generates scripts like:

    if (false) {
      $t = $.dom.create("sly",false,false);
      $n = $.dom.push($n,$t);
    }
    yield $.call($.template().lib["link"], [$n, {"text": content["title"], "href": "https://www.adobe.com", }]);
    if (false) {
    }
    if (false) {
      $n = $.dom.pop($n);
    }

it should be easy to sanitize the commands and remove all if (false) blocks.

for example iterating over HTMLCollection treats it as object.

(note, the spec is a bit asymmetric in regards to iteration over "maps": https://github.com/adobe/htl-spec/blob/master/SPECIFICATION.md#226-list, otherwise the distinction between arrays and objects would not be needed)

It would be nice if the HTL engine supports data-sly-use for loading additional binding.

the following should be supported:

• loading nodejs modules with default exports¹
• loading nodejs modules with sup-exports
• invoking construtors of exported classes

¹ use functions as mentioned in spec would need to be the default export.

Format:

<sly data-sly-use.<variable>="${<reference> [@ export=<export>, options]}" />

• reference is a valid require id that can be resolved to a module. note that it is the job of the runtime to ensure the module is available on the module path. a list of all referenced modules will be available via compiler.references()
• export is an sub-expression that addresses an exported function or class
• options are an optional comma separated list of arguments to pass to an exported function or constructor.

Examples:

Create a helper that reports the current time

clock.js

module.export = () => {
  time() {
    return new Date();
  }
}

example.htl

<sly data-sly-use.clock=${'./clock.js')/>
The time is ${clock.time}.

more advanced clock with format

clock.js

module.export = class Clock() {
  constructor(opts) {
    this.fmt = opts.format;
  }
  time() {
    return new Date().format(this.fmt); // i wish :-)
  }
}

example.htl

<sly data-sly-use.clock=${'./clock.js', @ format="hh:mm")/>
The time is ${clock.time}.

use external module

example.htl

<sly data-sly-use.mom=${'moment')/>
The time is ${mom.format}.

use exported function

helper.js

class Foo() {
...
}
class Bar() {
...
}
module.export = {
  Foo,
  Bar,
}

example.htl

<sly data-sly-use.foo=${'./helper.js', @ export=Foo)/>

having a single attribute like:

date = '2019-04-02T08:44:26.000Z';

with:

<time datetime="${date }">${date}</time>

escapes the attribute value like:

<time datetime="2019-04-02T08&#x3a;44&#x3a;26.000Z">2019-04-02T08:44:26.000Z</time>

even with context=unsafe.

similar with:

<time data-sly-attribute.datetime="${date @ context = 'unsafe' }">${date}</time>

see https://github.com/adobe/htl-spec/blob/master/SPECIFICATION.md#2211-unwrap

We are now part of the CircleCI Windows beta (docs here: https://github.com/CircleCI-Public/windows-preview-docs) and should have a build stage that runs on Windows to rule out compatibility issues. This would allow us to retire the AppVeyor build.

When there is something wrong in the htl template (like trying to access it.dummy.property which does not exist for exampl), the error log you get in the whole Petridish setup (and probably the same in the production setup) is something like:

[hlx] error: Error while rendering the resource: TypeError: Cannot convert undefined or null to object
    at Function.keys (<anonymous>)
    at lengthOf (../project-helix/modules/helix-helpx/.hlx/build/html.js:295:53)
    at ../project-helix/modules/helix-helpx/.hlx/build/html.js:442:28
    at Generator.next (<anonymous>)
    at onFulfilled (../project-helix/modules/helix-helpx/node_modules/co/index.js:65:19)
    at ../project-helix/modules/helix-helpx/node_modules/co/index.js:54:5
    at new Promise (<anonymous>)
    at co (../project-helix/modules/helix-helpx/node_modules/co/index.js:50:10)
    at Runtime.run (../project-helix/modules/helix-helpx/node_modules/@adobe/htlengine/src/runtime/Runtime.js:47:12)
    at run (../project-helix/modules/helix-helpx/.hlx/build/html.js:305:20)
    at main (../project-helix/modules/helix-helpx/.hlx/build/html.js:482:12)
    at getContextPath.catch.then (../project-helix/modules/helix-helpx/.hlx/build/html.js:240:19)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:182:7)

Which gives not hint on what is wrong in the template. It is crucial for development to output a meaningful error that helps locating the issue.

Hello,

I recently cloned this project and installed the dependencies then attempted to run the tests when I noticed there are a number of tests failing. I am on Windows 7 x64 OS Node v8.9.3. I have attached a log of the test output. Looking forward to using this library!
adobe-htl-test-output-fail.log

injecting a type that is not handled by the dom factory, produces:

TypeError: Failed to execute 'appendChild' on 'Node': parameter 1 is not of type 'Node'.

xss_api#sanitizeUrl does not correctly sanitize URLs.

Sample HTL:

<a href="${url}">XSS Link</a>

Executed with URL:

javascript:alert(0)

Expected Result:

<a href="javascript&#x3a;alert&#x28;0&#x29;">XSS Link</a>

(or some other form of encoding or omission)

Actual Result:

<a href="javascript:alert(0)">XSS Link</a>

Feature support for the data-sly-template and data-sly-call functions.

Raising an issue for a subsequent PR.

as @stevengill mentioned in adobe/helix-helpx#9 (comment), we should the following license header for the source files that originate from Adobe:

Copyright 2018 Adobe. All rights reserved.
This file is licensed to you under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License. You may obtain a copy
of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under
the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR REPRESENTATIONS
OF ANY KIND, either express or implied. See the License for the specific language
governing permissions and limitations under the License.

@trieloff @davidnuescheler ok ?

at Hackathon #1 we've decided to use the Airbnb JavaScript Coding Style for the Helix project's code base, based on this survey and documented in Helix's README.

we should either apply the airbnb coding style in the entire helix code base or review the decision made at the hackathon.

Problem

when using a use-class, the compiler generates a call statement using the filename of the class:

<div data-sly-use.help="${'./src/helper.js'}">${help.title}</div>

const help = yield $.use("./src/helper.js", {});

The Runtime then requires and executes the module. while this allows of dynamic use class loading, it makes it compiled script dependent on the existence of the use class.

Suggestion

add a compiler flag: requireUseClasses, that:

• force constant use class references
• adds require statements to the compiled script

example:

const use0 = require('./src/helper.js');
.
.
.
const help = yield $.use(use0, {});

Hi there,

I'm unable to get htlengine working. By following the base instructions in the readme (running cli.js against simple2) I receive the following error:

$ node src/cli.js test/simple2.html
module.js:549
    throw err;
    ^

Error: Cannot find module '@adobe/htlengine/src/runtime/Runtime'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/aledeg/Development/htlengine/out.js:15:17)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

So I thought okay that makes sense, the generated out-file is meant to be used from somewhere where htlengine is a dependency. So I created a minimum-possible testcase which uses the same setup/architecture as in these test files ( you can find this here: https://github.com/alexanderdegre/htlengine_test ) which fails with the following error:

/Users/aledeg/Development/htlengine_test/index.js:12
    return service(resource);
           ^

TypeError: service is not a function
    at doThing (/Users/aledeg/Development/htlengine_test/index.js:12:12)
    at Object.<anonymous> (/Users/aledeg/Development/htlengine_test/index.js:18:14)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:191:16)
    at bootstrap_node.js:612:3

Interestingly, when I modify htlengine's test files (cli.js, out.js etc) so that way the test data and template are "hardcoded" like in my own testcase I also end up with exactly the same error.

Also interestingly, despite this, running yarn test from htlengine's root folder passes successfully.

I'm not sure where or how my testcase or my environment differs that prevents my output JS from being evaluated and turned into html. Perhaps somebody here can offer some insight?

Also which versions of the following are you using? Perhaps I can adjust to the same versions and give it another try.

nvm version: 0.33.4
node version: 10.10.0
npm version: 6.4.1
antlr version: stable 4.7.1 (via brew)

Thank you so much for the efforts!

https://github.com/Adobe-Marketing-Cloud/htl-spec/releases/tag/1.4

the current compiler allows to define 1 variable name for the runtime globals. but it is not possible to have more external globals. the globals space used by the scripts should not be mixed with the external globals.

the runtime.globals should be read-only. and the runtTimeGlobal name should default to global (or window :-)

see #22

Problem

the use class file reference in data-sly-use is not relative to the .htl file.

eg, with a baseDir of myProject:

myProject/
└── src/
    ├── html.htl
    └── helper.js

the following use class can't be loaded:

src/html.htl

<div data-sly-use.help="${'./helper.js'}"></div>

Suggest

follow the same rule as node with require. i.e. ./ is relative to the source file. with no ./, it will be loaded as node_module. also see #105

the goal is to improve the current resource handling from simply reading the file to a more pluggable mechanism.

for example, the Runtime could have a ResourceHandler delegation.

xss_api.js currently uses Caja-HTML-Sanitizer for filterHTML(). As suggested by @lkrapf here, DOMPurify would be a more proven alternative with a very active community (while Caja-HTML-Sanitizer only has 48 commits and was last updated 3 years ago).

I propose to replace it with DOMPurify.

@trieloff can you remove the appveyor check? they are still running on PRs.

So that we can debug HTL with hlx up (see adobe/helix-cli#162)

Currently in the call_spec.txt, the test to validate "sightly call function can call other templates" is working, but when the same is done by having the templates loaded using data-sly-use, it is not working
in the templateLib.html added the below snippet

<template data-sly-template.foo="${@ a}"><div data-sly-use.lib="templateLib1.html" data-sly-call="${lib.bar @ a=a}"></div></template>

created a new file templateLib1.html under template_spec folder with the below code

<template data-sly-template.bar="${@ a}"><section>${a}</section></template>

and if we test with the below snippet

<div data-sly-use.lib="template_spec/templateLib.html" data-sly-call="${lib.foo @ a=123}"></div>

we are getting error
`

Error: Template cannot be defined in another template
at JSCodeGenVisitor.visit (...\htlengine\src\compiler\JSCodeGenVisitor.js:146:15)
at Start.accept (...\htlengine\src\parser\commands\Command.js:29:13)
at commands.forEach (...\htlengine\src\compiler\JSCodeGenVisitor.js:76:9)
at Array.forEach ()
at JSCodeGenVisitor.process (...\htlengine\src\compiler\JSCodeGenVisitor.js:75:14)
at Compiler.compile (...\htlengine\src\compiler\Compiler.js:235:8)
at

`

this fails atm:

### string constant test
#
<div>${'\\\"\'\u{1f600}'}</div>
===
<div>\"'😀</div>
#
###

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>whitesource/merge-confidence:beta)

The Compiler methods are currently all synchronous. Async methods would be more appropriate.

furthermore, fs.*Sync methods are used throughout the code base incl. tests. Those should be avoided by using async/await or Promise based variants provided by fs-extra instead.

Raising an issue for a subsequent PR to add feature support for the data-sly-resource block.

according to the call spec, the element should remain and the contents be replaced by the template.

so:

<template data-sly-template.one>blah</template>
<div data-sly-call="${one}"></div>

should output

<div>blah</div>

but the htlengine removes the outer element.

Is there a way for us to make it implicit, so that all properties within it just become the global scope within HTL? That way {it.request.path} would become simply {request.path}.

Decouple the I/O handling from the actual engine. this keeps the code much simpler since it doesn't need to deal with files.

Suggest:

• clean up API and remove:
  • Compiler.compileToString()
  • Compiler.compileFile()
  • Compiler.withOutputDirectory()
  • Compiler.withOutputFile()
• remove cli.js
• remove main.js

With #53 the XSS escaping introduced ends up being a bit too zealous and removes all:

• src attributes on images
• href attributes on links

At present, identifiers which contain a colon (:) will cause a syntax error to throw when running compiled code.

As a replacement, I suggest we replace the : with $ during compilation for all variable names (include the provided globals); as the $ is allowed in JavaScript but not the HTL grammar. It is unlikely to conflict with any other variables within the compiled JS.

The current markup context for variables output to HTML text is html. The HTL spec is not 100% clear about this. For use cases, where the source data is assumed to be safe it would be benefitial to change the default markup context to unsafe, i.e. to bypass any xss sanitizer.

regression of:
#53

for example:

<!-- include ${content.nav}.summary.html-->
<esi:include src="${content.nav}.summary.html" />

produces:

<!-- include &#x2f;SUMMARY.summary.html-->
<esi:include src=".summary.html" />

expected:

<!-- include &#x2f;SUMMARY.summary.html-->
<esi:include src="/SUMMARY.summary.html" />

It should be possible to use the HTLEngine as a pipeline processor like https://unified.js.org/.

In a text processing pipeline, the syntax tree could be provided as global variable, which then can be transformed using HTL. for example

var processor = unified()
  .use(markdown)
  .use(toc)
  .use(htlengine)
  .use(html)

<h1>Table of Contents</h1>
<ul data-sly-list="${dom.headings}">
  <li>${item}</li>
</ul>

Considerations

• output to attribute values can easily be replaced in the dom
• output to text-nodes should not allow construction of tags, so the injected dom can just be, well injected. i.e., ```<${tagname} class="..."/>` wold not be allowed.

the runtime main function is not async, thus not delegating errors that happen before the script is executed.

A couple of things:

I think the update from 3.2.7 to 3.2.8 should have had a major version bump.

We have our project structure set up in a way that mimics the use of Java classes. For example, we have a simple file loader that outputs SVG code inline so it can be styled.

<sly data-sly-use.svg="${'com.companyname.project.InlineSvgUtility' @ svgPath=properties.logo}">
    ${logo.svgItem @ context='unsafe'}
</sly>

We then used the withUseDirectory method of the runtime to load modules from the 'modules/' folder in the root e.g.

const fse = require('fs-extra')
const path = require('path')

class InlineSvgUtility {
  async use() {
    const assetsPath = 'assets/icons/'

    if (!this.svgPath) {
      return {
        svgItem: null,
      }
    }

    const data = await fse.readFile(path.join(assetsPath, this.svgPath), 'utf-8')

    return {
      svgItem: data,
    }
  }
}

module.exports = InlineSvgUtility

So this sits in src/modules/com.companyname.project.InlineSvgUtility.js, and was working happily. In 3.2.8 the withUseDirectory seems to effectively do nothing.

This line seems to be the cause.

6d00698#diff-59e501da46af5081071db46e7c50a748R243

If the path doesn't start with './' (e.g. '../'), it will attempt to resolve files relative to the folder the Compiler script is in, rather than the baseDir. For example, this will throw a "MODULE_NOT_FOUND" error.

<sly data-sly-use.svg="${'../modules/com.companyname.project.InlineSvgUtility' @ svgPath=properties.logo}">
    ${logo.svgItem @ context='unsafe'}
</sly>

I can fix it by traversing up 6 or so folders out of the node_modules, but that's not ideal.

<sly data-sly-use.svg="${'../../../../../../modules/com.companyname.project.InlineSvgUtility' @ svgPath=properties.logo}">
    ${logo.svgItem @ context='unsafe'}
</sly>

example:

<div data='"foo","bar"'/>

expected:
<div data='&quot;foo&quot;,&quot;bar&quot;'/>

actual:
<div data='&quot;foo","bar"'/>

It would be useful if the child nodes of the body, or a node in general could be injected via dom.

so eg:

<!-- add all child nodes of body, since DIV cannot have a BODY element.
<div>${document.body}</div>

and also:

<div>${document.body.firstElementChild.childNodes}</div>

In my index.js file (which is referenced by using data-sly-use in my HTL file),

If I try to do the following:

import PropTypes from 'prop-types';
import classnames from 'classnames';

I get the following error when attempting to compile:

(node:4052) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 3): SyntaxError: Unexpected token import
(node:4052) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

However, if I convert the same thing to require statements:

const PropTypes = require('prop-types');
const classnames = require('classnames');

It executes just fine. It seems some ES6 things are working (like const, async await, etc.) but imports are broken. I'm using node v8.9.3 on Windows 7 x64.

implement formatting as dates or numbers
https://github.com/adobe/htl-spec/blob/master/SPECIFICATION.md#122-format

adding some tests with larger files reveals:

  HTML Parsing
    ✓ parses simple.htm
    ✓ parses 400kb.htm (114ms)
    ✓ parses 700kb.htm (152ms)

  HTML Parsing and Processing
    ✓ parses and processes simple.htm (155ms)
    ✓ parses and processes 400kb.htm (16616ms)
    ✓ parses and processes 700kb.htm (26194ms)

it seems that the HTML parser is reasonably fast, but the processing (i.e. converting the HTML DOM into the HTL command stream is slow.

use the semantic-release module to automate releases

Would be beneficial to be able to provide the code generation template that wraps the transpiled code.

Would it be possible for you to provide some examples on how to use this module, as I am a Front End developer and have landed on this page in search of compiling Sightly/HTL templates with javascript/json object to generate HTML.

I am writing a HTL template that receives a data object with information about the page, and a component to include.

data passed to compiler:

{
  component: {
    path: 'src/components/button/button.htl',
    data: {
      text: 'This is a button'
    }
  },
  page: {
    title: 'This is the title'
  }
}

button.htl

<template data-sly-template.data="${@ data}">
    <button>${data.text}</button>
</template>

template.htl

<!DOCTYPE html>
<html lang="en">
  <title>${page.title}</title>
  <body>
      <h1>${page.title}</h1>
     <!-- this fails -->
      <div data-sly-use.lib="${component.path @ data=component.data}"/>      
    
      <!-- this also fails -->
      <div data-sly-use.lib="${component.path}" data-sly-call="${lib.data @ data=component.data}"/>

      <!-- this works --> 
      <div data-sly-use.lib="src/components/button/button.htl" data-sly-call="${lib.data @ data=component.data}"/>
  </body>
</html>

When I try to use a dynamic value, I get this error:

<template data-sly-template.data="${@ data}">
^

SyntaxError: Unexpected token '<'
    at Module._compile (internal/modules/cjs/loader.js:892:18)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
    at Module.load (internal/modules/cjs/loader.js:812:32)
    at Function.Module._load (internal/modules/cjs/loader.js:724:14)
    at Module.require (internal/modules/cjs/loader.js:849:19)
    at require (internal/modules/cjs/helpers.js:74:18)

    at Runtime.use (/Users/david/Dev/7-eleven/aem-frontend/frontend/node_modules/@adobe/htlengine/src/runtime/Runtime.js:110:17)
    at /Users/david/Dev/7-eleven/aem-frontend/frontend/build/output.js:64:25
    at Generator.next (<anonymous>)
    at onFulfilled (/Users/david/Dev/7-eleven/aem-frontend/frontend/node_modules/co/index.js:65:19)

So the path to the component is correct.

When I use a static value, I get this.

<!DOCTYPE html>
<html lang="en">
  <title>This is the title</title>
  <body>
      <h1>This is the title</h1>
      <div>src/components/button/button.htl</div>
    <button>This is a button</button>

  </body>
</html>

Is this supported? If not, is there another way I can include a HTL file and pass data to it without using use?

The spec states that the use statement can be used to load external templates (last sentence of the section). The following example does not work:

<div data-sly-use.lib="templateLib.html" data-sly-call="${lib.one}"></div>

where templateLib.html is

<template data-sly-template.one>blah</template>

You get an error like: SyntaxError: Unexpected token <

Could a maintainer/core contributor please reach out? I'm looking to use this and have a few questions.

twitter: @CodeByAlex
website: codebyalex.com

or please respond to this issue

@trieloff @tripodsan @dominique-pfister