[BUG] YAML runner appears to skip all checks if the returned value is null about connectedhomeip HOT 4 CLOSED

First, not all constraint checks are skipped for null. Only the ones that list is_null_allowed=True, which are:

• type constraints
• minLength/maxLength constraints
• minValue/maxValue constraints
• notValue constraint
• anyOf constraint

This is certainly correct behavior for minValue/maxValue and minLength/maxLength checks if the attribute is nullable. Spec constraints always mean "if the value is not null". See section "7.18.3. Constraint & Value", subsection "7.18.3.6. Nullable in Range", which says:

If data is nullable then null SHALL be a valid value.

So if you have a nullable int with constraint 1-3, the valid values are: null, 1, 2, 3. If you have a nullable string with some length constraints, null is a valid value (and in fact a zero-length string is also allowed, per that same spec section, no matter what the constraints say).

I suspect anyOf is probably correct as well, for the same reason.

I'm not sure what the story here is for non-nullable attributes for which the server nevertheless returns null.

For the notValue bit... that part looks odd to me. We do have some YAML using notValue: null and at first glance those would pass even if the value is in fact null. Checking that now.

from connectedhomeip.

And yes, I just confirmed that notValue: null does not fail if the value is in fact null. So we need to fix that.

from connectedhomeip.

#32871 should fix notValue: null.

from connectedhomeip.

Thanks, Boris. Ok, sounds like the runner is correct, and the test just isn't properly accounting for this case. I'll close this issue and handle it in the test itself.

from connectedhomeip.