Since this behavior is incompatible with other language's json parser, we can't use Go to run these tests.

I propose that a better test would be to use a json schema validation library in some other language like python or typescript. Allowing us to use the json schema definition file as the source of truth.

The problem

As pointed out by @Nayonika1, certain questions have a choice between them for the student but there's no way of encoding that in the schema.

This implies that we can not ensure that the sum of weights for each year is indeed 75 (as they must be for accuracy).

To solve this problem the schema must be modified to encode this information. Currently there are two potential solutions.

Solution by me

We add a new key of "or" to each year which is an array of topic name for which the current topic had a choice with.
For example, if we had topic A and B which had a choice in year 2022, we can encode that as

"topics: [
  {
    "name": "A",
    "weights": [{
      "year": 2022,
      "weight": 7.5,
      "or": ["B"]
    }]
  },
  {
    "name": "B",
    "weights": [{
      "year": 2022,
      "weight": 7.5,
      "or": ["A"]
    }]
  },
]

This encodes the information accurately without any ambiguity. However, it will be hard for new members to create these files which is an explicit goal for this project.
Also, without using a format like jsonp it will not be very error prone.

Solution by @deepanshu0598

"topics: [
  {
    "name": "A",
    "weights": [{
      "year": 2022,
      "weight": 7.5,
      "optional": true
    }]
  },
  {
    "name": "B",
    "weights": [{
      "year": 2022,
      "weight": 7.5,
      "optional": true
    }]
  },
]

This solution trades accuracy for feasibility.

Should we even?

Currently, it is not clear if this level accuracy is critical for us or not. And as such, it is decided to not make any changes.

Future

In case a solution is found which is accurate enough without trading feasibility and ease of creation by new members, the schema may be changed.

It would be beneficial to make sure that these changes are backward compatible.