martinohmann / hcl-rs Goto Github PK

I noticed when using dts that floats are mangled when outputting hcl. I found this:

Would it be an option not to do that and instead preserve the type?

TemplateExpr::evaluate right now will return literal $${ for $${ sequence in input.

The following does not fail to parse, but fails the assertion because $${escaped2} is split into a $ literal and the interpolation ${escaped2}:

let input = "$${escaped1} ${unescaped} $${escaped2} $$ESCAPED_SHELL_VAR\n$SHELL_VAR";
let parsed: Template = input.parse().unwrap();
let expected = Template::from_iter([
    Element::from("${escaped1} "),
    Element::from(Interpolation::new(Ident::new("unescaped"))),
    Element::from(" ${escaped2} $$ESCAPED_SHELL_VAR\n$SHELL_VAR"),
]);

assert_eq!(parsed, expected);

This is a parser bug.

This is a followup to #242.

Right now this fails both asserts because it roundtrips to an Expression::Template since the ${ in the string literal is not escaped when it is encoded:

use hcl_edit::expr::Expression;

let expr = Expression::from("${foo}");
let encoded = expr.to_string();
assert_eq!(encoded, "\"$${foo}\"");
let parsed: Expression = encoded.parse().unwrap();
assert_eq!(parsed, expr);

This also applies to hcl-rs. The fix is similar to #247, but we can now also take the opportunity to do the escaping directly in the parser code of hcl-edit.

I ran into a seemingly complex edge case where deserialisation fails when an untagged enum is used together with deserialize_with. In the following example:

#![allow(unused)]

use serde::Deserialize;
use serde::Deserializer;
use serde_json;
use anyhow;
use hcl;

pub fn de_transform<'de, D>(deserializer: D) -> Result<Vec<String>, D::Error>
    where
        D: Deserializer<'de>,
{
    let expressions: Result<Vec<hcl::Expression>, _> = Deserialize::deserialize(deserializer);

    if let Err(ref e) = expressions {
        eprintln!("{:?}", e)
    }

    Ok(expressions?.iter().map(|expr| format!("Parsed expression: {}", expr.to_string())).collect::<Vec<_>>())
}

#[derive(Deserialize, Debug, Clone)]
struct A {
    #[serde(deserialize_with = "de_transform")]
    pub value_a: Vec<String>
}

#[derive(Deserialize, Debug, Clone)]
struct B {
    #[serde(deserialize_with = "de_transform")]
    pub value_b: Vec<String>
}

#[derive(Deserialize, Debug, Clone)]
#[serde(untagged)]
enum Test {
    A(A),
    B(B),
}

#[derive(Deserialize, Debug, Clone)]
struct AX {
    pub value_a: Vec<String>
}

#[derive(Deserialize, Debug, Clone)]
struct BX {
    pub value_b: Vec<String>
}

#[derive(Deserialize, Debug, Clone)]
#[serde(untagged)]
enum TestX {
    AX(AX),
    BX(BX),
}


fn main() -> anyhow::Result<()> {
    let data = r#"
        value_a = [ident.prop1.subprop1, ident.prop2.subprop2]
    "#;

    let get_body = || {
        let body: Result<hcl::Body, _> = hcl::from_str(&data);
        body
    };

    let body_1 = get_body();
    // Will print body as expected
    println!("{:?}", body_1);

    let de_result_1: Result<A, _> = hcl::from_body(body_1?);
    // Will print struct as expected
    println!("{:?}", de_result_1);

    let de_result_2: Result<TestX, _> = hcl::from_body(get_body()?);
    // Will match enum case AX as expected
    println!("{:?}", de_result_2);

    // Will fail
    let de_result_3: Result<Test, _> = hcl::from_body(get_body()?);
    println!("{:?}", de_result_3);

    Ok(())
}

de_transform function will fail with:

Message { msg: "unknown variant `${ident.prop1.subprop1}`, expected one of `Null`, `Bool`, `Number`, `String`, `Array`, `Object`, `TemplateExpr`, `Variable`, `Traversal`, `FuncCall`, `Parenthesis`, `Conditional`, `Operation`, `ForExpr`, `Raw`", location: None }

I haven't been able to find a workaround unfortunately

let _: Body = crate::from_str("a = b[\"c\"]").unwrap() fails to deserialize the index access operator.

Expected behaviour is to deserialize this just fine.

Message { msg: "unknown variant `Index`, expected one of `Null`, `Bool`, `Number`, `String`, `Array`, `Object`, `TemplateExpr`, `VariableExpr`, `ElementAccess`, `FuncCall`, `SubExpr`, `Raw`", location: None }

In #159 the idea came up to also support evaluating expressions partially, returning information about missing variables instead of failing.

Using hcl-rs = "0.6.5" I am trying to produce the output

module "foo" {
    name = var.name
}

with the block macro. But nothing I have tried can produce this output, at most it always quotes the var.name which is not what I need. I have tried:

    hcl::block!(
        module "foo" {
            // name = var.name // error: recursion limit reached
            name = "var.name" // Produces `name = "var.name"` (as expected)
            name = (RawExpression::new("var.name")) // Produces `name = "var.name"`
            // name = RawExpression::new("var.name") // error: recursion limit reached
        }
    )

And cannot see anything other way to get this to work. Is this currently possible with the macros?

Using the builder does work as expected:

    Block::builder("module")
            .add_label("foo")
            .add_attribute(("name", RawExpression::new("var.name")))
            .build()

But ends up being a lot noisier than the macros.

Is hcl-rs fully feature compatible with the original HCL implementation?

I'm looking into having references like in terraform.

resource aws_something my_something {
  name = "something"
}

output foo {
  value = resource.aws_something.my_something
}

Is this something you already thought about and can share your thoughts? Is this something that you think is in scope of this library or should be external (extra crate inside or outside of this repo)?

Right now, Identifiers can be created from any string value and will fail to format if they contain invalid characters.

The type constructors should be charged to make it impossible to create invalid Identifier instances. This would remove the need to validate identifiers during formatting. The identifier validation is the only fallible operation (apart from write errors) that can happen during formatting. If it is removed, we can ensure that to_string really cannot fail.

• Deprecate Identifier::new and Identifier::from in order to eventually remove them in a future release
• Add Identifier::new_unchecked to allow deliberate bypassing of the validation for users that ensure identifier validity by other means. It should be marked with a warning to indicate that invalid identifiers may produce invalid HCL.
• Implement FromStr and TryFrom for `Identifier
• Move validation from format module into the FromStr impl

Is there a way to deserialize the following HCL block:

// outer_block {
  value {
    sub_value = 1
  }
// }

into the following structs?

struct Value {
  sub_value: i32
}

struct OuterBody {
  value: Value
}

...

let result = hcl::from_str::<OuterBody>(input)

I tried wrapping the value field in hcl::ser::Block, but it doesn't seem to work.

Hi,
I've noticed you're trying a rewrite in nom, I'm wondering if the plan would be to also include position spans for each of the parsed AST nodes (like in the Go implementation).
Perhaps by using something like nom_locate

Hello there,
I use VsCodium as my editor, and it would be cool to see this extension in https://open-vsx.org/

Before I grab the source and start working with it, I want to know if serialization is even considered "in scope" for this crate.

This fails to format because _ is not in XID_START, but it should still be allowed:

locals {
  _foo = "bar"
}

In terraform this is valid.

#[test]
fn parsed_body_attribute_has_span_information() {
    use hcl_edit::Span;
    
    let body = hcl_edit::parser::parse_body("foo = bar").unwrap();
    let first_attr = body.into_attributes().next().unwrap();
    assert!(first_attr.key.span().is_some());
}

Is this assertion correct? (it fails)

If this is confirmed and still open in about 2 weeks then i'll provide a PR with a fix attempt

Despite the copious signage, I was brave (foolish?) enough to try out hcl-edit.

I'm happy to report that it did exactly what I wanted it to and would have been easier than the alternatives. So, thank you!

I did find one issue where hcl-edit doesn't correctly parse comments starting with //:

Error:    --> HCL parse error in line 112, column 51
    |
112 |   ami                  = "ami-5f6495430e7781fe5" // Ubuntu 20.04 LTS
    |                                                   ^---
    |
    = invalid expression; expected `"`, `[`, `{`, `-`, `!`, `(`, `_`, `<`, letter or digit

I was able to work around this for my use case by just using # for the comments.

I started playing with hcl_edit to do some parsing.

For pretty error messages I want to display the location with some context like you do with hcl_edit::parser::error::Location. The closest I found was - for example - hcl_edit::structure::Block::span() to find out where the Block is defined.

Have I overlooked a way to do this right now or should I do something alike to the Location struct?

Consider adding BlockLabel::as_str(&self) to allow easy borrowing of the inner value. PR #144

The HCL Native Syntax Specification defines the structural, expression and template sub-languages. Right now, all expression types are part of the structure module. This should be fixed by moving them into a new expression module.

This would be a breaking change, but I'll try to ensure backwards compatibility for at least one minor release by still exposing the expression types in the structure module, but mark them as deprecated. While doing so, the Identifier type should probably also move to the crate root since it's neither structure nor expression-specific.

Given the following HCL:

resource "test" "testing" {
  ids      = [module.instance.0.id]
}

It leads to this panic:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: InvalidDigit }', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/hcl-rs-0.8.2/src/parser/mod.rs:342:32

It should be possible to serialize custom types to HCL blocks. Right now it's only possible to produce labeled and unlabeled blocks via the hcl::structure::Block type.

Internally, it is already possible to create HCL blocks from custom types, but the public API for this is not implemented and documented yet. Some work-in-progress is already present in the marker-types branch.

Follow-up to #184.

Consider a terraform-like environment where local variables can be defined in locals blocks and where locals can reference each other. One way to resolve those references would be to evaluate values in a loop, every time topologically sorting the graph of references and evaluating more missing values.

Variables are referenced through an object local, e.g. local.variable_name. Right now if the variable is not defined in the evaluation context, the library will return the following error:

The problem is, it's not clear what expression triggered the error. In the following code:

locals {
  var_combined = "${local.var_a}-${something.var_a}"
  var_a = "a"
}

the error will be something like:

        EvalError(
            "var_combined",
            Error {
                inner: ErrorInner {
                    kind: NoSuchKey(
                        "var_a",
                    ),
                    expr: Some(
                        TemplateExpr(
                            QuotedString(
                                "${local.var_a}-${something.var_a}",
                            ),
                        ),
                    ),
                },
            },
        )

with this error it seems to be impossible to tell whether the var_a key is missing from local object, or from some other object in code.

would it be possible to include the expression that produced the object itself in the error? or somehow else keep track of the source?

Hi! I'm just wondering if evaulating expressions are supported or even in scope for hcl-rs?

Currently, I want to serialize a HCL config with expressions into a file, but seems like the value just returns a literal string of the raw expression.

Is expression support going to be in hcl-rs?

Provided code snippet causes panic.

let str = r#"
  locals {
    test = {
      a = b// this comment breaks the parser
      c = d // but this one doesn't
    }
  }
"#;

let _: Result<hcl::Body, _> = hcl::from_str(str);

tested on version 0.16.3

In the following code snippet, I have been unable to figure out how to have the hcl::block! macro output the text aws_role.myrole.name without quoting it.

I have scoured the docs, tried many things, but it eludes me. Can you give a pointer? Thank you!

let role_resource = "myrole";
hcl::block!(
    resource "aws_iam_instance_profile" "myrole" {
        name = "myrole-instance-profile"
        role = (format!("aws_role.{}.name", role_resource)
    }
)

Hello, thank you for the awesome library!

I tried to look around but couldn't find this: is it possible to define nested identifiers and objects when evaluating with hcl::eval::Context?

For example, I'd like to evaluate the value of the following property:

  stages = stage.project_test.id

So I would need to make stage object available to context, then I would need to add project_test property, etc. Pretty much the same way it's done in terraform

// ser.rs
#[derive(Deserialize)]
pub struct Project {
    pub proj_type: String,
    pub spec: Option<PathBuf>,
    pub dockerfile: Option<PathBuf>,
    pub scripts: Option<Vec<Script>>,
}

#[derive(Deserialize)]
pub struct Script {
    pub name: String,
    pub command: String,
}

// function to load HCL config here...

// conf.hcl
project "a" {
    proj_type = "generic"
    spec = "./test.spec"
}

returns error

Message { msg: "invalid type: string \"./test.spec\", expected option", location: None }

I am currently trying to write something which relies on using terraforms jsonencode.

From what I can tell this is currently not supported/possible, is this correct?

So what I was trying to do was basically wrapping a block into a jsonecode block and therefore be able to write everything in rust/hcl-rs without having to resort to raw string inputs in attributes (which also mess up escaping and intendation)

I am parsing HCL files that contain comments to be able to update some blocks/attributes, but I would like to keep comments in tact after parsing -> serializing back to text.

Maybe I have missed it, if so please feel free to link me to the documentation.

Thank you!

With the following code:

    #[test]
    fn heckheckheck() {
        use hcl::Block;

        let contents = r#"block1 "label0" "label1" { inner = true }"#;
        let block: Block = hcl::from_str(contents).unwrap();
    }

receives this error:

running 1 test
thread 'config::tests::heckheckheck' panicked at crates/config/src/config.rs:132:52:
called `Result::unwrap()` on an `Err` value: Message("missing field `identifier`")

when, it should be successful with the following structure:

Block {
   identifier: Identifier(block1),
   labels: ["label0", "label1"],
   body: ... # with attribute "inner" ~> true
}

Since it derives Deserialize, I guess the implementation is wrong or is this intended behaviour?

Here's a heavily reduced version of a terraform variable definition:

variable "test" {
  level1 = map(object({
    level2 = map(object({
      level3 = map(object({
        level4 = map(object({
        }))
      }))
    }))
  }))
}

On my system the above takes ~24 seconds to parse.

Removing a level by commenting it out:

variable "test" {
  level1 = map(object({
    level2 = map(object({
      level3 = map(object({
        # level4 = map(object({
        # }))
      }))
    }))
  }))
}

Takes about ~1.77 seconds to parse

Removing one additional level (level3) and parsing this file drops to ~0.455 seconds to parse.

It seems that there is something that goes quadratic when having multiple function calls nested. We have a bunch of terraform that heavily uses this to make sure the type of the variable is correct:

variable "network_integration" {
  description = "Map of networking integrations between accounts"
  type = map(object({
    friendly_name = string,
    vpcs = map(object({
      id           = string
      cidr         = string
      region       = string
      description  = string
      subnets      = map(string)
      route_tables = map(string)
      security_groups = map(object({
        id = string
        rules = map(object({
          direction   = string
          protocol    = string
          from_port   = string
          to_port     = string
          description = string
        }))
      }))
    }))
    additional_propagated_vpcs   = list(string)
    additional_static_vpc_routes = list(string)
  }))
  default = {}
}

As a full example (which this takes ~130 seconds to parse).

The following code panics in debug mode and causes a deadlock in release mode:

use hcl::{Expression, FuncCall};

let expr = Expression::from(
    FuncCall::builder("foo")
        .arg(Expression::from_iter([("bar", FuncCall::new("baz"))]))
        .build(),
);

let result = hcl::format::to_string(&expr).unwrap();

assert_eq!(result, "foo({\"bar\" = baz()})")

This is most likely caused by an invalid formatter state caused by compact_mode.

This was discovered while debugging #82.

Using an input file such as:

output "sometest" { value = {
  name = module.instance.*.tags.Name
  id   = module.instance.*.id
} }

When attempting to read it, will lead to this error:

Error: invalid type: string ".*", expected unit

The splat operator is used heavily in our terraform configuration for outputs.

hcl-edit roundtrip on Body:

• "foo = \"bar\"" => "foo = \"bar\"\n" (0 != 1 newline)
• "foo = \"bar\"\n" => "foo = \"bar\"\n" (1 == 1 newline)
• "foo = \"bar\"\n\n" => "foo = \"bar\"\n\n" (2 == 2 newlines)

For the first case I would expect there to be no newline in the result.

There seems to be a static newline here

Our terraform codebase contains several declarartions of the form:

resource "foo" "inline" {
  role = var.role_name
  name = var.policy_name
  policy = (var.policy != "" ?
    var.policy :
    templatefile("path/to/template", {
      foo = bar
    })
  )
}

This is parsed correctly with the terraform binaries and with go-hcl, but hcl-rs will return an error such as:

ignored parsing error:   --> HCL parse error in line 10, column 36
   |
10 |   policy = (var.policy != "" ?
   |                               ^---
   |
   = invalid expression; expected `"`, `[`, `{`, `-`, `!`, `(`, `_`, `<`, letter or digit

I forked this repo and created a failing test case on a branch. It seems that in deserialize_any some extra context would need to be passed along in the case of parenthesis, as it's legal to have multiline expressions within.

Please let me know if there's anything I can do to help or any extra information you need.

Right now the following fails to deserialize with an error, but it would be nice to support it so that users can create custom types that can leverage the Expression type as attribute value:

#[derive(serde::Deserialize, Debug, PartialEq)]
struct Config {
    op: Expression,
}

let expected = Config {
    op: Expression::from(BinaryOp::new(1, BinaryOperator::Plus, 1)),
};

let result: Config = hcl::from_str("op = 1 + 1").unwrap();

assert_eq!(result, expected);

Right now, template expressions are subject to double-escaping which produces invalid HCL when formatted.

The following tests are failing altough they should pass:

expect_format(
    Attribute::new("a", TemplateExpr::from("${\"b\"}")),
    "a = \"${\"b\"}\"\n",
);

expect_format(value!({ a = "${\"b\"}" }), "{\n  \"a\" = \"${\"b\"}\"\n}");

Right now this test fails because a gets quoted, although it is a valid identifier and the quotes could be omitted:

let value = hcl::value!({ foo = { a = "b", "with space" = "c" } });
let expected = "foo = {\n  a = \"b\"\n  \"with space\" = \"c\"\n}\n";
assert_eq!(hcl::to_string(&value).unwrap(), expected);

  left: `"foo = {\n  \"a\" = \"b\"\n  \"with space\" = \"c\"\n}\n"`
 right: `"foo = {\n  a = \"b\"\n  \"with space\" = \"c\"\n}\n"`

In formatted HCL, identifiers are used as object keys where possible. We could support this via a formatter options, e.g. prefer_ident_keys which is enabled by default, but can be disabled if desired.