When running the encoder test suite, toml-test -encoder panics during the multiline-string test. I took a look at the output of my encoder output.txt and it looks fine to me. Here is the error:

$HOME/go/bin/toml-test -encoder encode.bat multiline-string
panic: BUG: Expected valid escape code after , but got '\r'.

[recovered]
panic: BUG: Expected valid escape code after , but got '\r'.

goroutine 1 [running]:
github.com/BurntSushi/toml.parse.func1(0xc042039330)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/parse.go:46 +0xf 2
panic(0x5996a0, 0xc04200e270)
C:/Go/src/runtime/panic.go:489 +0x2dd
github.com/BurntSushi/toml.(*parser).bug(0xc0420081c0, 0x5dfacc, 0x2f, 0xc042038 e20, 0x1, 0x1)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/parse.go:83 +0x1 02
github.com/BurntSushi/toml.(*parser).replaceEscapes(0xc0420081c0, 0xc0420e0059, 0x7, 0x7, 0x3)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/parse.go:534 +0x 2f5
github.com/BurntSushi/toml.(*parser).value(0xc0420081c0, 0x6, 0xc0420e0059, 0x7, 0x5, 0xc0420e003e, 0x15, 0x101, 0xc042039208)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/parse.go:167 +0x 21a2
github.com/BurntSushi/toml.(*parser).topLevel(0xc0420081c0, 0x12, 0xc0420e0000, 0x0, 0x4)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/parse.go:134 +0x 8ac
github.com/BurntSushi/toml.parse(0xc0420e0000, 0x1a1, 0xc0420081c0, 0x699a20, 0x c04200e280)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/parse.go:62 +0x3 8b
github.com/BurntSushi/toml.Decode(0xc0420e0000, 0x1a1, 0x590e20, 0xc04200e0c0, 0 x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/decode.go:115 +0 x2e3
github.com/BurntSushi/toml.DecodeFile(0xc042018120, 0x5e, 0x590e20, 0xc04200e0c0 , 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml/decode.go:133 +0 x13e
main.loadToml(0xc042018120, 0x5e, 0x0, 0x0, 0x1, 0x0)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml-test/results.go: 184 +0x91
main.runValidTest(0xc042048400, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, .. .)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml-test/results.go: 123 +0x517
main.runTestByName(0xc042048400, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, . ..)
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml-test/main.go:176 +0x4ef
main.main()
C:/Users/david.taylor/go/src/github.com/BurntSushi/toml-test/main.go:115 +0x75f

I'm implementing support for toml-test and have found what seems like an inconsistency in the expectations for a few of the tests - specifically for the table arrays.

With non-table arrays, it is expected that the JSON output specifies array for type, ala the example in /tests/valid/arrays.json where this TOML:

strings = ["a", "b", "c"]

...would output this JSON:

{
  "strings": {
    "type": "array",
    "value": [
      {"type": "string", "value": "a"},
      {"type": "string", "value": "b"},
      {"type": "string", "value": "c"}
    ]
  }
}

However, with the table arrays, the expected JSON output is showing not to expect a specified type at all, ala the example in /tests/valid/table-array-one.json where this TOML:

[[people]]
first_name = "Bruce"
last_name = "Springsteen"

...is expecting this JSON:

{
  "people": [
    {
      "first_name": {"type": "string", "value": "Bruce"},
      "last_name": {"type": "string", "value": "Springsteen"}
    }
  ]
}

Since the resulting data structure is is that people is array like any other (an array of objects), I would expect people be typed like all other arrays, like this:

{
  "people": {
    "type": "array",
    "value": [
      {
        "first_name": {"type": "string", "value": "Bruce"},
        "last_name": {"type": "string", "value": "Springsteen"}
      }
    ]
  }
}

What's the reason for the different expectations?

Hey,

With the creation of the toml-lang org and this test resource as the de facto testing resource for parsers, could this repository be moved to that organization? Then this could be upgraded as the TOML spec is bumped. It'd make for better "discovery" and "authority" of this resource.

Thoughts?

I'd like to request that Test.cmpTOML be exported. I'm wanting to compare the output of two different TOML implementations, and this one method is the only one I'm lacking access to.

I'm happy to submit a PR if you're okay with the idea.

When testing my encoder the raw-multiline-string test fails with:

$ $HOME/go/bin/toml-test -encoder encode.bat raw-multiline-string
Test: raw-multiline-string (valid)

Values for key 'firstnl' differ. Expected value is
This string has a ' quote character. (string), but your encoder produced This string has a ' quote character. (string).

0 passed, 1 failed

The output output.txt of my encoder looks good to me. It looks like the test suite is maybe expecting the firsnl string to begin with a newline?

Right now the tests consider capital base prefixes invalid.

$ grep -r 0B tests/invalid
tests/invalid/integer.multi:capital-bin = 0B0
tests/invalid/integer-capital-bin.toml:capital-bin = 0B0

However, the spec doesn't say they are invalid. What it says is:

Non-negative integer values may also be expressed in hexadecimal, octal, or binary.
In these formats, leading + is not allowed and leading zeros are allowed (after the prefix).
Hex values are case-insensitive.

All programming languages I can remember support upper case prefixes, so it's also against the common practice.

$ python3
Python 3.9.5 (default, May 14 2021, 00:00:00) 
>>> 0XFF
255

 perl -e '$x = 0XFF; print "$x\n"'
255

$ node
Welcome to Node.js v14.17.0.
> 0XFF
255

$ ocaml
        OCaml version 4.12.0

# 0XFF ;;
- : int = 255

I'm investigating some TOML libraries in Haskell, and I'm not completely sure what version of TOML they support, but I'm pretty sure they support v0.4.0 or v0.5.0. I'd be open to upgrading them to v1.0.0, but in the meantime, it would be great if I could run toml-test -version v0.4.0 on them to quickly see how they conform to spec.

Alternatively, if you want to avoid maintaining old versions' tests in perpetuity, you could add releases for old versions of toml-test (or add docs somewhere containing github commit hashes) that clearly indicate what version of the spec they adhere to.

Trying this:

class Test
{
	public string FirstName { get; set;}
	public string Surname { get; set;}
}
class Enc
{
	public Test[] Stuff {get; set;}
}

...

Test t1 = new Test { FirstName = "Bill", Surname = "Gates" };
Test t2 = new Test { FirstName = "Barbara", Surname = "Streisand" };
Test[] t = new[] { t1, t2 };
Enc e = new Enc {Stuff = t};
Console.WriteLine(Toml.WriteString(e));

Getting:

[[Stuff]]
FirstName = "Bill"
Surname = "Gates"
[[Stuff]]
FirstName = "Barbara"
Surname = "Streisand"

With an empty line as the very first line of the output. Is it possible not to produce it?

Could I provide you with updates to bring this project's compliance up to the pending TOML v1.0.0?

Also, what is the best way to give you attribution if we port your tests to other projects?

My impression from the spec is that the only banned characters in table names are the ']' and the '.' characters. Quoting from the spec:

Name your tables whatever crap you please, just don't use a dot. 

Reading the TOML spec, I can see:

Arrays are square brackets with values inside. Whitespace is ignored. Elements are separated by commas. Arrays can contain values of the same data types as allowed in key/value pairs. Values of different types may be mixed.

However, the first test in the invalid folder is testing if mixed types is allowed. I am just wondering why that's the case?

In #13, the integer range was adjusted to make Javascript happy, but the tests no longer reflect the real range specific in the TOML specification.

Maybe this is a one-off thing that doesn't matter too much, but maybe we need language specific tests to fix it for real. I'm not sure what the best implementation is.

I'd love for toml-test to become the official test suite for TOML implementations. If you're into that idea, we could move it to the toml-lang org on GitHub so people know it's the real deal. Once it's updated for the imminent 1.0 release, I'd like to create an easy way for implementors to get their implementation's test results published on the upcoming TOML marketing website so others can see which implementations are production ready.

The usage example shows passing flags to the parser command as extra command-line arguments:

Usage: toml-test-v1.0.0-linux-amd64 parser-cmd [ parser-cmd-flags ]

And it also contains the following note:

Any positional arguments are use as the parser-cmd; to pass flags, remember to
stop toml-test's flag parsing with --

   $ toml-test-v1.0.0-linux-amd64 -- my-parser -x -y

Unfortunately this only seems to work consistently for single-character flags. Longer flags seem to be checked for validity against toml-test's own flags.

As an example, the following command is accepted:

$ ./toml-test-v1.0.0-linux-amd64 -- /bin/false -x

But this one is not:

$ ./toml-test-v1.0.0-linux-amd64 -- /bin/false --enable-x
toml-test-v1.0.0-linux-amd64: unknown flag: "--enable-x"

Even stranger, the options are allowed through or not depending on whether they match one of toml-test's options. For example, toml-test has an option --skip, and thus the following is accepted as an option to pass to the decoder to be run:

$ ./toml-test-v1.0.0-linux-amd64 -- /bin/false --skip

But if we change one letter, it is no longer accepted:

$ ./toml-test-v1.0.0-linux-amd64 -- /bin/false --skop
toml-test-v1.0.0-linux-amd64: unknown flag: "--skop"

The spec is pretty simple but there definitely awaits a fair number of tests to catch all of the semantics.

There is no tests/invalid-encoder directory anymore in this repository and using toml-test with the -encoder option therefore fails. A quick workaround is to create the directory tests/invalid-encoder after checking out toml-test locally or in a CI workflow.

Having a simple invalid encoder test like a top-level array or just an empty file like .gitkeep might be useful to preserve the directory.

Um, this doesn't seem accurate any more:

Here's the main page:

I would love to sign up to create some additional tests for the things that are missing. Is there anything specific you would like first?

I see there are different tags corresponding to different TOML versions I suspect. Perhaps a v0.4.0 branch could be added for us to work on together until we get it polished enough to become the main. I have forked and created a v0.4.0 branch and will add the updated tests, if any to it before submitting as a PR.

I'm testing a TOML encoder on testcase valid/array-table-array-string-backslash.
The encode under test outputs

[[foo]]
bar = "\"{{baz}}\""

while the testsuite expects

foo = [ { bar="\"{{baz}}\""} ]

Clearly these two TOML encodings are equivalent.
However, the toml-test tool reports failure:

Type mismatch for key 'foo'. Expected array but got []map[string]interface {}.

I believe this happens because the golden TOML parser uses different types for inline arrays versus arrays of tables. An inline array is parsed into a value of type []interface{}. But a double-bracketed array of tables is parsed into a value of type []map[string]interface{}. These are incompatible types according to the Go typesystem. As a result, the toml-test tool reports failure even though the answer is correct.

There should be a test that this is invalid:

a = [{ b = 1 }]
[a.c]
foo = 1

toml-lang/toml#908

All places that use the term "keygroup" should be modified to use the term "table" instead.

That file has the following lines:

end_esc = "String does not end here\" but ends here\\"
lit_end_esc = 'String ends here\'

lit_multiline_end = '''There is no escape\'''

That would be patently invalid in every programming language that supports breaking long lines with backslashes. For the sake of argument, let's assume that TOML set out to be different and check the spec.

What the spec says is:

When the last non-whitespace character on a line is an unescaped \,
it will be trimmed along with all whitespace (including newlines) up to the next non-whitespace character or closing delimiter.

In those examples, the last non-whitespace character on each line is not \, it's a quote! Thus they don't really match the requirement of that passage and shouldn't be valid at all.

I believe that this test should be marked invalid, given the current spec.

Two reasons:

1. Keys cannot contain a # character.

2. Name your tables whatever crap you please, just don't use #, ., [ or ].

   Not sure if this means to extend to keys as well, or if things like

   a.b.c = 2

   are valid in lieu of

   [a.b]
   c = 2

   If the first example is supposed to be valid, then there's no problem. If, however, that's supposed to be disallowed, I think the test is invalid for containing a . as well as a #.

The tests include the following expectation:

FAIL valid/string-nl
     Values for key 'nl_end' don't match:
       Expected:     value
     
       Your encoder: value\n

     input sent to parser-cmd:
       nl_mid = "val\nue"
       nl_end = """value\n"""

That goes against the spec that requires that the leading newline be trimmed, but expressly states that all other whitespace should be kept:

Multi-line basic strings are surrounded by three quotation marks on each side and allow newlines.
A newline immediately following the opening delimiter will be trimmed.
All other whitespace and newline characters remain intact.

According to the TOML spec:

Float values -0.0 and +0.0 are valid and should map according to IEEE 754.

The TOML file valid/float/zero.toml has for example:

f3 = -0.0

but the corresponding JSON has

  "f3": {
    "type": "float",
    "value": "0"
  },

I think the values here should have the corresponding signs as the TOML file.

The current test gives:

1987-07-05T17:45:56.6Z

and expects:

1987-07-05T17:45:56.600000Z

I expected nanosecond precision to be preserved and survive a round time without the value changing.

Should this behavior be enforced by toml-test or left as a implementation detail for the user to decide?

Hello,

tests/invalid/key/multiline.toml seems to imply that newline characters in quoted keys are invalid, but I can’t find such a restriction in the spec. In 1.0.0’s Keys section we can even find:

Quoted keys follow the exact same rules as either basic strings or literal strings and allow you to use a much broader set of key names.

Shouldn’t this testcase be converted into a valid one?
Thanks!

MacOS Terminal.app does not support true colors, which causes my black background terminal to show "FAIL" as basically invisible. Maybe toml-test could inspect $TERM and use zli.Red if the terminal doesn't support true color? Or maybe use Color256 with 224, which seems close enough? (#ffd7d7; https://www.ditig.com/256-colors-cheat-sheet)

Using the GOPROXY environment variable is crucial in many environments like GitLab CI with private projects or internal infrastructures.

To reproduce it, run it in the official Go image:

docker run -it --rm golang:1.16

Set the GOPROXY environment variable:

export GOPROXY=direct

Simple download and build:

go get -u github.com/BurntSushi/toml-test

Output:

go: downloading github.com/BurntSushi/toml-test v1.0.0
go get: github.com/BurntSushi/[email protected] updating to
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected]: invalid version: unknown revision 20a94d639b8e

This affects all projects around the world that depends on the toml project. For example:

go get -u github.com/mgechev/revive

Output:

go get: github.com/BurntSushi/[email protected] updating to
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected] requires
        github.com/BurntSushi/[email protected]: invalid version: unknown revision 20a94d639b8e

To resolve that, the go.mod file must be patched:

go 1.16
 
require (
    github.com/BurntSushi/toml v0.3.2-0.20210704081116-ccff24ee4463
    // no_term branch, which doesn't depend on x/term and x/sys
    zgo.at/zli v0.0.0-20210619044753-e7020a328e59
)

replace github.com/BurntSushi/toml => github.com/BurntSushi/toml v0.4.0

Hello,

I am not a Go user, but I use this amazing test library to test the toml parser in Nim.

Some time ago, if I did go get -u -v github.com/BurntSushi/toml-test, the toml-test executable used to get installed as $GOPATH/bin/toml-test.

But now if I do go get -u -v github.com/BurntSushi/toml-test@master, no executable is getting installed.

But if I do go get -u -v github.com/BurntSushi/toml-test@9767d201, I see the executable updating in $GOPATH/bin/ once again.

I am on go 1.16.5.

Has something changed in this repo that's disabling the executable compilation by default?

Just an issue to remind. Will try to make time to create a pull-request for this, but since I'm quite busy lately I don't expect it in the next few weeks.

I believe the treatment of 64 bit integers in the test suite is solid, but I think testing floating point values has holes.

It is partially mitigated by the fact that all values are encoded as JSON strings. This means that we can interpret all floating point values as Go does and perform a proper equality test.

However, different languages may have different floating point representations. Which means that some languages might produce different values---which poses problems for an equality test.

In practice, this may not be so much of an issue since floating point conversion is typically well-handled by the language being used. However, it may be worth it to come up with tests for sufficiently small and large numbers that are not reliably represented in practice.

Overall, I'm not sure. And I have only a passing familiarity with the inner workings of floating point representation to know if this is a real issue or not.

TOML allows null characters in strings with \0, but JSON forbids it. This means we cannot naturally test whether TOML parsers handle null characters correctly.

Ideas?

The only thing I can think of is coming up with a convention for representing a null character in JSON without actually writing \0. Then, toml-test could be aware of this and make the appropriate substitution.

Perhaps such a convention could be a character that is illegal in TOML but legal in JSON, so as not to interfere with testing legitimate strings in TOML.

For example, toml-test <my_decoder> table-empty will run the invalid test, but not the valid one. Is there a way to specify this, or run both?

For example, tests/valid/exponent-part-float.json represents floats without a trailing .0, but tests/valid/float-exponent.json represents floats with a trailing .0.

See https://github.com/mojombo/toml#array-of-tables

Since my parser combinator library is LL(1) by default, it thought the space between 27 and # was the date / time separator.

foo = 1979-05-27 # Comment

See toml-rs/toml#269

I'm assuming others might run into this and it'd be good to upstream the test case.

9223372036854776000 is outside the range for signed 64-bit integer. The 0.2.0 spec requires a minimum of 64-bits for integers, the 0.3 spec clarifies this that integers should have range (−9,223,372,036,854,775,808 to 9,223,372,036,854,775,807)

I used this when developing the TOML parser for Nim, and it was very helpful. But now the spec has updated a while back and this still hasn't been bumped to 0.5.0. Any plans to do it?

I'm trying to incorporate this test suite into a Go testing framework without using the toml-test binary interface.

I'd like to step through the tests individually using the Go testing.Run API. Currently, Runner.Run() will run all tests and then let you step through the results (see BurntSushi/toml for example).

With the current API, if a test causes a panic, no clues are offered as to which test caused the panic. It would be helpful and, I think, better fit in testing.Run if the Runner API was extended to offer a way to step through the tests one at a time.

I'm in the process of writing a parser, which I know should fail most of the tests. It's somehow managing to crash your test binary though.

⊙  toml-test ./toml_test                                                                 Julian@air ●
panic: interface conversion: interface is float64, not string

goroutine 1 [running]:
main.result.cmpValues(0x422f364e, 0x5, 0x0, 0x0, 0x1, ...)
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/json.go:122 +0x56b
main.result.cmpMaps(0x422f364e, 0x5, 0x0, 0x0, 0x1, ...)
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/json.go:40 +0x3cf
----- stack segment boundary -----
main.result.cmpJson(0x422f364e, 0x5, 0x0, 0x0, 0x1, ...)
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/json.go:19 +0x1c2
main.result.cmpMaps(0x422f364e, 0x5, 0x0, 0x0, 0x1, ...)
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/json.go:61 +0xaff
----- stack segment boundary -----
main.result.cmpJson(0x422f364e, 0x5, 0x0, 0x0, 0x1, ...)
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/json.go:19 +0x1c2
main.runValidTest(0x422f364e, 0x5, 0x422f364e, 0x5, 0x0, ...)
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/results.go:124 +0x7d2
main.runAllTests(0x0, 0x7fff5fbfdaaa)
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/main.go:146 +0x667
main.main()
        /Users/Julian/.local/share/go/src/github.com/BurntSushi/toml-test/main.go:95 +0xa4

goroutine 2 [syscall]:
created by runtime.main
        /usr/local/Cellar/go/1.0.3/src/pkg/runtime/proc.c:221

goroutine 9 [finalizer wait]:
created by runtime.gc
        /usr/local/Cellar/go/1.0.3/src/pkg/runtime/mgc0.c:882

It's not pushed yet, and I can try to narrow down what is happening, but I thought I'd give you a heads up in case it's immediately obvious to you where this might be going wrong.

#4 contains a hack that separates arguments by spaces to achieve this behavior. Instead, I'd like to allow -- to be used so that any arguments after it are passed directly to the parser being tested.

According to the spec:

Under that, and until the next key or EOF are the key/values of that key group. Keys are on the left of the equals sign and values are on the right. Keys start with the first non-whitespace character and end with the last non-whitespace character before the equals sign. Key/value pairs within key groups are unordered.

This also means that valid/key-special-chars.toml is invalid.

foo = """
foo "bar" qux
"""

should parse foo into a string with foo "bar" qux in it. Also variations of this. "Compliant" Python implementations do not understand this construct correctly.

Is it possible to stream the output instead of waiting until all the tests are complete? That way, you can see the progress, especially with --verbose, to see progress of passing tests

https://github.com/BurntSushi/toml-test/blob/f910e151d1b14d94b1e8a4264db0814fb03520d9/tests/valid/datetime.toml#L2-L3

Is supposed to equate to

https://github.com/BurntSushi/toml-test/blob/f910e151d1b14d94b1e8a4264db0814fb03520d9/tests/valid/datetime.json#L3-L4

But the timezones are different, on milliseconds the time is wrong too.

This is causing problems with encoding tests -encoder. I think there's another bug with the decoder tests which means this is currently passing even though the encoded date doesn't match the data in JSON.

Hi,

I am currently packaging this software in fedora. During the review, I was signaled that the following files don't have a license header.

• json.go
• main.go
• results.go

It is non blocking but perceived as a good practice.

Can you add a header? A simple one stating the license would be OK.

From the TOML project home page:

You may omit the local offset and let the parser or host application decide that information. A good default is to use the host machine's local offset.

1979-05-27T07:32:00
1979-05-27T00:32:00.999999

The date time in datetime-malformed-no-z.toml is 1987-07-05T17:45:00 which is valid.

Hi, thanks for making this test suite!

I'm currently implementing a TOML parser in V and I use this test suite to validate it.

I've come across an oddity in valid/string/raw-multiline.json.

I believe that the following two test values should start with a \n char:
"This string\nhas ' a quote character\nand more than\none newline\nin it."
Should be (note the leading \n)
"\nThis string\nhas ' a quote character\nand more than\none newline\nin it."

and
"This string has a ' quote character."
Should be (note the leading \n)
"\nThis string has a ' quote character."

The TOML looks like:

oneline = '''This string has a ' quote character.'''
firstnl = '''
This string has a ' quote character.'''
multiline = '''
This string
has ' a quote character
and more than
one newline
in it.'''

both firstnl and multiline starts with a line-break?
Since these are multiline literal strings the leading \n should've been included?

Hi,

The current release of toml-test is quite old (2013) and some improvements have been done since. Can you make a new release please? This will allow me to update the package in fedora.

I wanted to use the test suite to test my work-in-progress TOML library, but the build fails with this error for me:

$ go build
go: github.com/BurntSushi/[email protected]: invalid version: unknown revision 20a94d639b8e