Code Monkey home page Code Monkey logo

hocon's Introduction

Hocon

HOCON data parser for Erlang/OTP.

Spec

HOCON spec for reference: https://lightbend.github.io/config/

Divergence from Spec and Caveats

  • The forbidden character @ is allowed in unquoted strings.
  • Value concatenation is not allowed in keys e.g. a b c : 42 is invalid.
  • Newlines do not prevent concatenation.
    • String (All below are parsed to the same result, which is #{key => <<"value1value2">>})
      • "key=value1 value2"
      • "key=value1value2"
      • "key=\"value1\" \"value2\""
      • "key=value1\nvalue2"
    • Array (#{key => [1,2,3,4]})
      • key=[1, 2] [3, 4]
      • key=[1, 2]\n[3, 4]
      • key=[1,2,3,4]
    • Object (#{key => #{a => 1,b => 2}}):
      • key={a: 1} {b: 2}
      • key={a: 1}\n{b: 2}
      • key={a=1, b=2}
  • url()/file()/classpath() includes are not supported
  • Immediate quote before triple-quote is invalid sytax.
    • """"a"""" is invalid because there are 4 closing quotes instead of three. See workarounds below.
  • Multiline strings allow indentation (spaces, not tabs). If ~\n (or ~\r\n) are the only characters following the opening triple-quote, then it's a multiline string with indentation:
    • The first line ~\n is discarded.
    • The closing triple-quote can be either """ or ~""" (~ allows the string to end with " without escaping).
    • Indentation is allowed but not required for empty lines.
    • Indentation level is determined by the least number of leading spaces among the non-empty lines.
    • If the closing triple-quote takes the whole line, it's allowed to be indented less than other lines, but if it's indented more than other lines, the spaces are treated as part of the string.
    • Backslash is NOT a escape character.
    • If a string has three consecutive quotes, there are two workarounds:
      • Make use of string concatenation, and only escape the triple-quotes. e.g.
        a = """~
                line1
            ~"""
            "line2\"\"\"\n"
            """~
                line3
            ~"""
        
      • Use normal string with escape sequence. For example: a = "line1\nline2\"\"\"\nline3\n"

Schema

HOCON schema (hocon_schema) is a type-safe data validation framework. which is compatible to basho cuttlefish

See more information in SCHEMA.md

Test Data

Files in sample-configs are collected from https://github.com/lightbend/config/tree/v1.4.1

TODO

  • Upload to hex.pm

Reference

License

Apache License 2.0, see LICENSE.

Authors

EMQX team

hocon's People

Contributors

6293 avatar emqplus avatar hjianbo avatar iequ1 avatar k32 avatar keynslug avatar kianmeng avatar kjellwinblad avatar lafirest avatar mbaechler avatar mononym avatar paulozulato avatar qzhuyan avatar sdzx-1 avatar sergetupchiy avatar sstrigler avatar terry-xiaoyu avatar thalesmg avatar turtledeng avatar x1001100011 avatar zhongwencool avatar zmstone avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hocon's Issues

string number concatenation

c=2
x:{ a = b${c}
d = e}

above input is parsed as #{c => 2,x => #{a => [<<"b">>,2],d => <<"e">>}}.
I believe it should either raise an error
or be smart and format c as string before performing a concatenation.

recursive substitution reference

in our implementation, variable substitution happens after values are merged.
below examples will cause dead-loop.

var1=x
var2=${var1}y
var1=${var2}z

or simply

self=a
self=${self}b

“a.b = c ” does not work in the list

I created a file that reads: a: [ {b.c = 1} ] , When I executed hocon:load/1, I got it: {ok,#{<<"a">> => [#{<<"b">> => 1}]}} , It looks like c is missing

When I change the contents of the file to: a: {b.c = 1} and executed hocon:load/1 again, I got the correct result: {ok,#{<<"a">> => #{<<"b">> => #{<<"c">> => 1}}}}

It looks like the content of the comments in the file affects the `hocon:load/1` function

I created a file that reads:

a: [
    {b:1},
    # {}
    {c:2}
]

When I executed hocon:load/1, I got it:

{ok,#{<<"a">> => [#{<<"b">> => 1,<<"c">> => 2}]}}

This is wrong and b and c should not be in a map

When I delete comments in a file:

a: [
    {b:1},
    {c:2}
]

and executed hocon:load/1 again, I got the correct result:

{ok,#{<<"a">> => [#{<<"b">> => 1},#{<<"c">> => 2}]}}

It looks like the content of the comments in the file affects the hocon:load/1 function

How to add any key to map() by fields/1

Hi, Guys

I create fields/1 function like this:

fields(headers) ->
    [ {'Accept',            #{type => binary(), default => <<"application/json">>}}
    , {'Accept-Encoding',   #{type => binary(), default => <<"gzip, deflate">>}}
    , {'Accept-Language',   #{type => binary(), default => <<"en-US">>}}
    , {'Authorization',     #{type => binary(), nullable => true}}
    , {'Cache-Control',     #{type => binary(), default => <<"no-cache">>}}
    , {'Connection',        #{type => binary(), default => <<"keep-alive">>}}
    , {'Content-Type',      #{type => binary(), default => <<"application/json">>}}
    , {'Keep-Alive',        #{type => binary(), default => <<"timeout=5">>}}
    ];

I have defined some http headers, how should I update this method if I want to let users freely add other header?

substitute included vars with abs path

file a:

foo={x:1}{include b}
bar={x:2}

file b:

y: ${bar.x}

this causes {unresolved,['foo.bar.x']} exception.
according to the spec:

Substitutions in included files are looked up at two different paths; first, relative to the root of the included file; second, relative to the root of the including configuration.

_

_

unicode

We have atom to list, and list to atom conversions in the code base.
They should be replaced with unicode APIs https://erlang.org/doc/man/unicode.html

And test cases to validate unicode (and update README in this regard)

substitution lookup error

file a

foo.x: 45
y: 2

it can be parsed as expected: #{foo => #{x => 45},y => 2}

file b

foo={x: 45}
y: 2

it is also parsed as expected: #{foo => #{x => 45},y => 2}

however, if we change the second line to y: ${foo}, file a results in exception {unresolved,[foo]} (but not file b)

Hocon schema does not support referencing external type

I want to reference the external type in my schema file, like this:

-type url() :: emqx_http_lib:uri_map().

-reflect_type([ permission/0
              , action/0
              , url/0
              ]).

-typerefl_from_string({url/0, emqx_http_lib, uri_parse}).

-export([ structs/0
        , fields/1
        ]).

structs() -> ["http"].

fields(http) ->
    [ {url, #{type => url()} }
    ];

But I got error:

escript: exception error: undefined function emqx_http_lib:uri_map/0
  in function  emqx_authz_schema:url/0 (emqx_authz_schema.erl, line 7)
  in call from emqx_authz_schema:fields/1 (emqx_authz_schema.erl, line 26)
  in call from hocon_schema:map_field/4 (hocon_schema.erl, line 436)
  in call from hocon_schema:do_map_union/4 (hocon_schema.erl, line 541)
  in call from hocon_schema:map_field/4 (hocon_schema.erl, line 452)
  in call from hocon_schema:do_map_array/4 (hocon_schema.erl, line 552)
  in call from hocon_schema:map_field/4 (hocon_schema.erl, line 462)

Is there any way I can use external type?

dump schema JSON

schema can be dumped to markdown (hocon_schema_doc.erl)
we need to dump a JSON format of the same info in markdowns

Convert Erlang term back to HOCON

When generating erlang terms from hocon files, the hocon schema transforms the values according to the type, which is good.

For example if we specified fun (type) -> ip_port(); the value will be converted from binary() to ip_port() after all validation passed.

But if we tried to convert the generated Erlang term to JSON using jsx:encode(Term), it fails because ip_port() is of type tuple().

I would appreciate if hocon schema supports converting the Erlang term back to json (or hocon), and do the inverse operation of tyeprefl:from_string/2, i.e. converting the ip_port() back to binary().

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.