Dumping broken in way too many ways about nimyaml HOT 5 CLOSED

Quick heads up, this has now been released in NimYAML 2.1.0. This took far longer than anticipated because of a regression in Nim 2.0.2, for which a workaround is now in place.

Apart from the various fixes regarding output style, I also added a new yaml/style module with pragmas that let you customize the output style of certain fields in the object types you serialize – see new Quickstart example. I hope that together with the general dumper options, this gives enough control over output style for most use-cases.

I did some updates to the documentation and hope that things are now clearer for both new and existing users. Feedback is always welcome.

from nimyaml.

Though the canonical output is certainly not pretty, it is valid YAML and the only problem with it is that NimYAML cannot parse it (which, admittedly, is bad). Can you explain what you think is „broken“ here?

Possibly canonical is a bad term here. The YAML specification, in version 1.2.1, named this explanatory format:

This format uses only flow collections, double-quoted scalars, and explicit tags for each node.

The revised YAML version 1.2.2 doesn't mention or use this format anymore.

I inherited the term canonical form for this format from PyYAML but did not realize that it isn't a term from the specification, which uses the term canonical form only for representation of scalars. In any case, it's a format designed for debugging and inspection, not for production use.

The configuration you get via Dumper() is meant to be the default.

dumper.presentation.quoting = sqUnset # Supposed to be the default, but if unset by library user, everything is force quoted.

It would be the default if you used the default values (via Dumper()), however you initialized the dumper with minimalDumper() which does set this to sqJson.

So there are three implementation issues I see here:

• Parser fails to load flow-style collection nodes with verbatim tags
• there is a problematic interaction with condenseFlow and indentationStep which does cause invalid indentation in some configurations
• Presenter sometimes emits block-style scalars in flow style which is forbidden

Also did you try using Dumper() and were there problems with that? If yes, please elaborate on those. If not, there seems to be a documentation problem because that is the first thing a caller should try, all other options are mostly for edge cases where you need a specific layout. If that isn't clear I should make it clearer.

Does this sum up your complaints?

Thanks for reporting!

from nimyaml.

Though the canonical output is certainly not pretty, it is valid YAML and the only problem with it is that NimYAML cannot parse it (which, admittedly, is bad). Can you explain what you think is „broken“ here?

Possibly canonical is a bad term here. The YAML specification, in version 1.2.1, named this explanatory format:

This format uses only flow collections, double-quoted scalars, and explicit tags for each node.

The revised YAML version 1.2.2 doesn't mention or use this format anymore.

I inherited the term canonical form for this format from PyYAML but did not realize that it isn't a term from the specification, which uses the term canonical form only for representation of scalars. In any case, it's a format designed for debugging and inspection, not for production use.

I remember, that the tags stuff was already some particularly YAML specific stuff, so I had the thought, this might be the case here, as well, but when I further inspected the output, it was utterly clear to me, that this would never ever be a type of output anyone would use in a scenario, where the user gets to see it. Therefore, it was natural to me to (apparently falsely) assume, that this must be either some intermediate step during the serialisation, where you did not apply the last step by serialising it as actual YAML or at best it could be some form of debug output. Otherwise, have never seen this before and I don't see any use case for it, other than behind the scenes technical exchange of data, which is never seen by any human eyes.

however you initialized the dumper with minimalDumper() which does set this to sqJson.

Realised this, after posting the issue. Indeed!

The configuration you get via Dumper() is meant to be the default.

It would be the default if you used the default values (via Dumper())

Also did you try using Dumper() and were there problems with that?

...

If not, there seems to be a documentation problem because that is the first thing a caller should try, all other options are mostly for edge cases where you need a specific layout. If that isn't clear I should make it clearer.

Like a hughmonguous one.

All I found was this Dumping API Example.

Now, after looking at this page again, I realised, that Dumper is mentioned several times there.
However, I usually just look at examples, omitting walls of text, because examples always tell me more than a million words of explaining it. (For example, UNIX-like manuals are exactly the opposite of what should be happening, but that's another story...)

So, just checked out "Dumping API" & saw this canonicalDumper, which seemed to me, like the "default" & didn't think about the actual "Dumper" object, which was mentioned before.

All the rest was me checking out NimYAML's source code, manually.

It'd be nice to have a big documentation example of the "default" Dumper object & then some big example of a highly customised canonicalDumper() or something like that.

Of course, I'd be willing to add this documentation!

Does this sum up your complaints?

With your work, it's never a complaint, it's always trying to help you progress with your project. 🙂

I think, this is now "resolved" as in, I got the message & fully understand what you're saying.

I'm closing this issue & if you say, I should work on examples for the documentation, then I'll open another issue.

Thank you very much, as always. 😃

from nimyaml.

Can you tell me how you generated the Unquoted JSON stuff? I have trouble reproducing the error that emits block scalars in flow style.

Regarding the documentation, well, the first example in the Quickstart on the main page uses Dumper() but I agree that it should be mentioned in the migration guide more explicitly. Also I should probably not show canonicalDumper() as prominently or at least explain what it is and what it's used for. I will come up with something after the fixes.

In case you're interested why you got additional newlines: When you set indentationStep to 1 you will always get newlines, because there is not enough space in the indentation for - (sequence item indicator + space). The problem was that in your case the indentation was doubled (which is why you set 1), that is now fixed and you can use an indentationStep of 2 and get proper indentation.

from nimyaml.

In case you're interested why you got additional newlines: When you set indentationStep to 1 you will always get newlines, because there is not enough space in the indentation for - (sequence item indicator + space). The problem was that in your case the indentation was doubled (which is why you set 1), that is now fixed and you can use an indentationStep of 2 and get proper indentation.

Indeed. I am currently tweaking my Dumper configuration and came to the same conclusion.

Regarding the documentation, well, the first example in the Quickstart on the main page uses Dumper() but I agree that it should be mentioned in the migration guide more explicitly. Also I should probably not show canonicalDumper() as prominently or at least explain what it is and what it's used for. I will come up with something after the fixes.

I see, now. The thing is, that I didn't remember how lazy you made the defaults, so I was oriented towards "migrating" from my lazy configuration, to the "new" lazy configuration, so I just omitted the "Quickstart" section, as I expected no specific configuration there, which, I feared, would've not been the lazy dumping I want. Lazy, as in no tags, no nothing. Just the slim YAML.

Can you tell me how you generated the Unquoted JSON stuff? I have trouble reproducing the error that emits block scalars in flow style.

Tried to reproduce it, but couldn't work it out. I might look at the file history, if enough history is there. If it's very important to you, you can try it yourself. It is definitely a mixture out of my initial configuration.

import pkg/yaml

var dumper = minimalDumper()
# https://github.com/flyx/NimYAML/blob/854d33378e2b31ada7e54716439a4d6990460268/yaml/presenter.nim#L69-L80
dumper.presentation.containers = cBlock # Without it, broken output, too.
dumper.presentation.outputVersion = ovNone
dumper.presentation.newlines = nlLF
dumper.presentation.indentationStep = 1 # Have to set 1, to get an indentation of 2 spaces.
dumper.presentation.condenseFlow = false # Must be false to not break output.
dumper.presentation.suppressAttrs = false
dumper.presentation.directivesEnd = deNever
dumper.presentation.quoting = sqUnset # Supposed to be the default, but if unset by library user, everything is force quoted.
dumper.serialization.tagStyle = tsNone
dumper.serialization.handles = @[]
dumper.dump(config, fStream)

Just change some of the fields' values & change minimalDumper to canonicalDumper & perhaps you will find it. I think, it should be easier for you to figure out, since you know the ins & outs of this, while I just ran a trial & error session.

Nevermind. Just checked file history timestamps & wasn't difficult to find. Thanks Visual Studio Code Local History!

  var dumper = minimalDumper()
  # https://github.com/flyx/NimYAML/blob/854d33378e2b31ada7e54716439a4d6990460268/yaml/presenter.nim#L69-L80
  # it.presentation.containers = cBlock
  dumper.presentation.outputVersion = ovNone
  dumper.presentation.newlines = nlLF
  dumper.presentation.indentationStep = 1
  dumper.presentation.condenseFlow = false
  dumper.presentation.suppressAttrs = false
  dumper.presentation.directivesEnd = deNever
  dumper.presentation.quoting = sqUnset
  dumper.serialization.tagStyle = tsNone
  dumper.serialization.handles = @[]
  dumper.dump(config, fStream)

Product of above code.

{
 version: appVersion,
 contexts: [
  {
   dateFormat: "yyyy-MM-dd",
   timeFormat: "HH:mm",
   timeZone: Europe/Oslo,
   zoom: {
    authentication: [
     {
      mail: [email protected],
      userID: ZoomUserID,
      accountID: "",
      clientID: "",
      clientSecret: ""
     }
    ],
    patternKeywordsYes: [
     {
      statement: OR,
      keywords: [
       MeetupTopicKeyword,
       Second Keyword,
       Juggernaut
      ]
     }
    ],
    patternKeywordsNo: [
     {
      statement: AND,
      keywords: [
       NotAllowedKeyword,
       Nonsense,
       Test
      ]
     }
    ]
   },
   mail: {
    enable: false,
    mailSender: {
     mail: [email protected],
     serverSMTP: smtps.example.com,
     portSMTP: 465,
     user: username,
     password: password,
     startTLS: false
    },
    mailReceiver: {
     mails: [
      [email protected],
      [email protected],
      [email protected],
      [email protected],
      [email protected]
     ],
     headerFrom: Zoomer <[email protected]>,
     subjectTpl: "Invitation to {zoom.TOPIC} on {zoom.START_DATE}",
     bodyTpl: |-
      You are invited to {zoom.TOPIC} at {zoom.START_TIME}!
      
      Please, join via the following link:
      
      {zoom.URL}
      
      
      
      
      This E-Mail was automatically generated and sent. Please do not reply.
    },
    schedule: [
     {
      tType: DAYS,
      amount: 7
     },
     {
      tType: DAYS,
      amount: 3
     },
     {
      tType: HOURS,
      amount: 1
     },
     {
      tType: MINUTES,
      amount: 15
     }
    ]
   }
  }
 ]
}

from nimyaml.