[TC-OO-2.3] - Test Case failing due to OffWaitTime attribute value mismatch and throwing unexpected error during execution about matter-test-scripts HOT 8 CLOSED

To clarify - I am now writing about the OffWaitTime. In step 11b both the TargetValue and Start Value are 300. The 15% tolerance is 0 here. In a perfect world the OnWithTimedOff(0x01, 300, 300) command and read of the OffWaitTime happens at the same time. As we are not living in a perfect world and there is some latency between the command and attribute read PIXIT.OO.MaxCommunicationTurnaround is stated in the test plan:

The value of OffWaitTime has to be approximately equal to 0x012c. The tolerance is PIXIT.OO.MaxCommunicationTurnaround.

from matter-test-scripts.

@Ashwinigrl is this still an issue in the latest TH release ?

from matter-test-scripts.

@raju-apple TC-OO-2.3 Test case is verified with latest TH image(Version: TH Fall2023 Sha: 3f84bff5) still facing the same issue.

PICS file used:
On-Off Cluster Test Plan .txt

Reference log:
Tc-OO-2.3.pdf
TC-OO-2.3.log

Endpoint used: 1

from matter-test-scripts.

I am getting the same error, the value of OffWaitTime is usually 151 in my case. Let's focus on steps from 11a to 12b:

• 11a:Sends OffWithEffect command to DUT
  At this point the DUT should enter Delayed Off state and OffWaitTime should start countdown from 300 ds to 0 ds.

• 11b:Reads OnOff attribute from DUT
  First attribute read.

• 11b:Reads OnTime attribute from DUT
  Second attribute read.

• 11b:Reads OffWaitTime attribute from DUT
  Third attribute read.

• Wait 10000ms
  Wait 10 seconds.

• 12a:Sends OnWithTimedOff command to DUT
  This command should not affect a countdown.

• 12b:Reads OnOff attribute from DUT
  Fourth attribute read.

• 12b:Reads OnTime attribute from DUT
  Fifth attribute read.

• 12b:Reads OffWaitTime attribute from DUT
  Finally read of the OffWaitTime that is assumed to happen exactly after 10 seconds from OffWithEffect command being called.

So, there are 5 attribute reads and 1 command (so 6 commands in total) executed between start of the OffWaitTime countdown and read of the OffWaitTime attribute. I suspected that each attribute read and command adds some delay, so I started commenting out the intermediate steps, one by one:

It clearly shows that execution of each command/attribute read takes about 5 - 10 ds. I do believe that those delay is mostly caused by TH processing rather than DUT/network. I am getting similar results both using chip-lighting-app running on TH as well as my real DUT. I had no issues with this test on TH 2.6.

Apart from this issue I also think that the 15% tolerance is improperly applied to many steps in this test. For example steps from 10d to 10e:

• 10d:Sends OnWithTimedOff command to DUT

    - label: "10d:Sends OnWithTimedOff command to DUT"
      command: "OnWithTimedOff"
      PICS: OO.S.C42.Rsp
      arguments:
          values:
              - name: "OnOffControl"
                value: 1
              - name: "OnTime"
                value: 300
              - name: "OffWaitTime"
                value: 300

• 10e:Reads OnTime attribute from DUT

    - label: "10e:Reads OnTime attribute from DUT"
      command: "readAttribute"
      attribute: "OnTime"
      PICS: OO.S.A4001 && OO.S.C42.Rsp
      response:
          constraints:
              minValue: 255
              maxValue: 345

Why the maxValue is 345? Of course this is 300 + 15% * 300, but any value above 300 indicates a bug. The OnTime attribute can only decrement as time goes by and it was set to 300 in step 10e.

Also the absolute value of the tolerance shrinks after as the time goes by, that seems counterlogical. For example the tolerance of OffWaitTime at step 11b, so when the countdown starts, is 45 ds, in step 12b, so after 10 seconds and a few commands executed, the tolerance is 30 ds. Why?

BTW, the PIXIT.OO.MaxCommunicationTurnaround from On/Off PICS seems not to work.

from matter-test-scripts.

@LookAtThisTree

• YAML does not currently support PIXIT. In our script, PIXIT.OO.MaxCommunicationTurnaround is not utilized; instead, we follow the expected value with a tolerance of +-15%. For instance, if the expected value is 300, we use 300+-15%, resulting in a range of 255 to 345. The same principle applies to step 12b, where if the expected value is 200, the range becomes 170 to 230.

• In Step 10e, the maxValue is currently set at 345, which is incorrect. It should not exceed 300, and we will rectify this in the script.

@Raju, we are considering making PIXIT.OO.MaxCommunicationTurnaround a configurable variable that can be set and used across all steps. However, this would entail using the same tolerance value for all steps. Could we discuss this further? This issue appears to be similar to what we encounter with CC scripts.

from matter-test-scripts.

we follow the expected value with a tolerance of +-15%. For instance, if the expected value is 300, we use 300+-15%, resulting in a range of 255 to 345. The same principle applies to step 12b, where if the expected value is 200, the range becomes 170 to 230.

As I mentioned in my post above I believe that the way how 15% tolerance is applied here is incorrect. The 15% tolerance in time-dependent test are intended to give some wiggle room due to timer inaccuracy (both TH and DUT), network latency, etc. Naturally the longer the test takes the bigger tolerance is needed, as inaccuracy cumulate. The way the tolerance is applied here - simply 15% of expected value - results in shrinking tolerance over time. The expected value is a timer that counts down.

Let's imagine step 12c:

After 19,9s, TH reads OnOff attribute , OnTime attribute and OffWaitTime attribute from DUT

This is 29,9 seconds after the OnWithTimedOff(0x01, 300, 300) command has been send and the OffWaitTime started countdown. The expected value is 1. Do you think that 15% of 1, so 1 +/- 0 is a good tolerance here?

Similar problem has already been reported here https://github.com/CHIP-Specifications/chip-test-plans/issues/2332

from matter-test-scripts.

@LookAtThisTree I understand your concern about the accuracy of the tolerance calculation. To address this issue, the proposed solution involves using the formula 'tolerance = (TargetValue - StartValue) ±15%.' To provide an example, when applying this formula to step 11b, the calculation (300-0) ± 15% yields a range of 245-345. However, focusing specifically on step 12b, where the calculation is (300-200) ± 15%, the result is 15, leading to a range of 185-215. It's worth noting that if the range is small for a particular condition, the tolerance will also be small. Please let us know your preference in this regard.

from matter-test-scripts.

Raised new Issue project-chip/certification-tool#117. Hence closing it.

from matter-test-scripts.