project-chip / matter-test-scripts Goto Github PK

Description:

• [TC-SWTCH-2.2]
• In TC-SWTCH-2.2 Manual YAML script has to be updated as per Verification step document .

Yaml-Reference:

https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_SWTCH_2_2.yaml

From chip-test-scripts created by xilizhang1: CHIP-Specifications/chip-test-scripts#687

Summary Title:

step13: Verify that the TH successfully connects to the DUT. But the reference log value is "False"

Description:

Steps to reproduce:

Use two Raspberry Pi for simulation testing, one as the reference DUT and the other as the TH (both Raspberry Pi version: v2.8.1-official)

Logs:

============DUT log=========
[1696661554.926464][2843:2843] CHIP:IM: Received Read request
[1696661554.926549][2843:2843] CHIP:DMG: ReadRequestMessage =
[1696661554.926582][2843:2843] CHIP:DMG: {
[1696661554.926608][2843:2843] CHIP:DMG: AttributePathIBs =
[1696661554.926638][2843:2843] CHIP:DMG: [
[1696661554.926664][2843:2843] CHIP:DMG: AttributePathIB =
[1696661554.926695][2843:2843] CHIP:DMG: {
[1696661554.926722][2843:2843] CHIP:DMG: Endpoint = 0x0,
[1696661554.926749][2843:2843] CHIP:DMG: Cluster = 0x31,
[1696661554.926780][2843:2843] CHIP:DMG: Attribute = 0x0000_0001,
[1696661554.926806][2843:2843] CHIP:DMG: }
[1696661554.926831][2843:2843] CHIP:DMG:
[1696661554.926855][2843:2843] CHIP:DMG: ],
[1696661554.926882][2843:2843] CHIP:DMG:
[1696661554.926908][2843:2843] CHIP:DMG: isFabricFiltered = true,
[1696661554.926930][2843:2843] CHIP:DMG: InteractionModelRevision = 1
[1696661554.926952][2843:2843] CHIP:DMG: },
[1696661554.927024][2843:2843] CHIP:DMG: IM RH moving to [GeneratingReports]
[1696661554.927146][2843:2843] CHIP:DMG: Building Reports for ReadHandler with LastReportGeneration = 0 DirtyGeneration = 0
[1696661554.927178][2843:2843] CHIP:DMG: RE:Run Cluster 31, Attribute 1 is dirty
[1696661554.927197][2843:2843] CHIP:DMG: Reading attribute: Cluster=0x0000_0031 Endpoint=0 AttributeId=0x0000_0001 (expanded=0)
[1696661554.927224][2843:2843] CHIP:DMG: AccessControl: checking f=1 a=c s=0x000000000001B669 t= c=0x0000_0031 e=0 p=a
[1696661554.927259][2843:2843] CHIP:DMG: AccessControl: allowed
[1696661554.932586][2843:2843] CHIP:DL: Current connected network: "EDT-liz"
[1696661554.933233][2843:2843] CHIP:DMG: Sending report (payload has 76 bytes)...
[1696661554.933450][2843:2843] CHIP:EM: <<< [E:41228r S:43006 M:134093789 (Ack:50834063)] (S) Msg TX to 1:000000000001B669 [219D] --- Type 0001:05 (IM:ReportData)
[1696661554.933522][2843:2843] CHIP:IN: (S) Sending msg 134093789 on secure session with LSID: 43006
[1696661554.933777][2843:2843] CHIP:DMG: OnReportConfirm: NumReports = 0
[1696661554.933825][2843:2843] CHIP:DMG: ReportsInFlight = 0 with readHandler 0, RE has no more messages
[1696661554.935324][2843:2843] CHIP:DMG: IM RH moving to [AwaitingDestruction]
[1696661554.935382][2843:2843] CHIP:DMG: All ReadHandler-s are clean, clear GlobalDirtySet

=========================TH log======
[1696661554.888709][3777:3779] CHIP:TOO: Endpoint: 0 Cluster: 0x0000_0031 Attribute 0x0000_0001 DataVersion: 3665788991
[1696661554.888807][3777:3779] CHIP:TOO: Networks: 1 entries
[1696661554.888863][3777:3779] CHIP:TOO: [1]: {
[1696661554.888896][3777:3779] CHIP:TOO: NetworkID: 53485F54455354
[1696661554.888924][3777:3779] CHIP:TOO: Connected: FALSE
[1696661554.888949][3777:3779] CHIP:TOO: }
[1696661554.889156][3777:3779] CHIP:EM: <<< [E:41228i S:13574 M:50834064 (Ack:134093789)] (S) Msg TX to 1:0000000000000001 [219D] --- Type 0000:10 (SecureChannel:StandaloneAck)
[1696661554.889209][3777:3779] CHIP:IN: (S) Sending msg 50834064 on secure session with LSID: 13574
[1696661554.889389][3777:3779] CHIP:EM: Flushed pending ack for MessageCounter:134093789 on exchange 41228i
[1696661554.889688][3777:3777] CHIP:CTL: Shutting down the commissioner
[1696661554.889730][3777:3777] CHIP:CTL: Stopping commissioning discovery over DNS-SD
[1696661554.889760][3777:3777] CHIP:CTL: Shutting down the controller
[1696661554.889790][3777:3777] CHIP:IN: Expiring all sessions for fabric 0x1!!

Additional Info:

Reference the Main spec"Version 1.2-prerelease-ipr, 2023-09-05 15:21:34 -0700: Draft" 11.8.5.4 section
Connected Field
This field SHALL indicate the connected status of the associated network, where "connected" means
currently linked to the network technology (e.g. Associated for a Wi-Fi network, media connected
for an Ethernet network).

We want to confirm that this correct value should be False or TRUE ?

2.We not only see this problem on the Raspberry Pi, but also on other chip devices. Is this an SDK issue?

From chip-test-scripts created by KishokG: CHIP-Specifications/chip-test-scripts#677

Description:

• TC-DGWIFI-1.1: Event list step has been changed from step 7 to 5. So the script has to be updated for the same.

Test Plan Reference:

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#685

• [TC-DISHALM-3.x] Updated Reset Command at step 3a.

PFA Script screenshot :

PFA Verification Document

• [TC-DISHALM-3.x] Pics should be updated in Script

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#690

Description :
Test Script should be updated as per verification steps.

PFA UI screenshot :

PFA VS screenshot :

Description:

The test-plan has been updated for the TC-BOOL-1.1, Hence the Yaml script needs to be updated as per the Test-plan.

• Step 7 has been moved to step 5 and the step 5 & 6 we desent to the 6 & 7 respectively.

Test-plan Reference: https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/booleanstate.adoc

Script Reference https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_BOOL_1_1.yaml

Description:

• TC-ACT-2.2: Step 4c,4d and 4e has to be updated as per VS doc.
• TC-ACT-3.2: Step 2 has to be updated as per VS doc.
• TC-DRLK-3.2: Comment has to be updated for all the steps as per VS doc.

Description:
Script has to be updated for below mentioned test cases as per VS doc changes:

• TC-CNET-4.5: Step 1 Command has to be updated
• TC-CNET-4.11: Step 5 and 16 Command has to be updated
• TC-CNET-4.20: Step 3 Command has to be updated

From chip-test-scripts created by Ashwinigrl: CHIP-Specifications/chip-test-scripts#686

Description:

TC-DESC-2.1 :- In Step 1b, 2 Script has to be updated as per VS document

Steps to reproduce:

Step 1b, 2

Yaml-Reference:

https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_DESC_2_1.yaml

Additional Info:

Version: TH Fall2023
Sha: 3f84bff5

Summary Title:

[TC-CNET-4.4] UI manual prompt does not match chip-tool command format

Description:

UI Prompt for step 1 shows as

actual chip-tool usage:
Usage: ./chip-tool networkcommissioning add-or-update-wi-fi-network Ssid Credentials destination-id endpoint-id-ignored-for-group-commands [--paa-trust-store-path] [--cd-trust-store-path] [--commissioner-name] [--commissioner-nodeid] [--use-max-sized-certs] [--only-allow-trusted-cd-keys] [--trace_file] [--trace_log] [--trace_decode] [--trace-to] [--ble-adapter] [--storage-directory] [--commissioner-vendor-id] [--Breadcrumb] [--timedInteractionTimeoutMs] [--busyWaitForMs] [--suppressResponse] [--repeat-count] [--repeat-delay-ms] [--timeout]

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#682

Description:
[TC-DA-1.8] - Script has to be updated as per Verification Step document for step 27(as per VS)

Additional Info:
Version: TH Fall2023
Sha: 3f84bff5

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#689

Description
[TC-ALOGIN-12.1] - Precondition should be updated as per Verification Document.

PFA UI screenshot :

PFA VS screenshot :

From chip-test-scripts created by KishokG: CHIP-Specifications/chip-test-scripts#676

Description:

• Pre-Condtions has to be updated for all the test cases
• TC-CNET-4.3: Step 8 Command and log has to be added
• TC-CNET-4.5: Step 1 to 3 command and step 3 PICS has to be updated
• TC-CNET-4.6: Step 1 to 3 command has to be updated
• TC-CNET-4.9: Step 4 Command has to be updated
• TC-CNET-4.10: Step 4 Command has to be updated
• TC-CNET-4.15: Step 2 and 3 Command has to be updated
• TC-CNET-4.16: Step 2 and 3 Command has to be updated

From chip-test-scripts created by Ashwinigrl: CHIP-Specifications/chip-test-scripts#657

Description:

[TC-CC-8.1] - Steps 2d and 2e failing due to CurrentHue attribute value mismatch.

Steps to reproduce:

Step 2d, 2e

Screen Shot:

Logs:

TC-CC-8.1.log

Additional Info:

DUT : All-clusters-app
Endpoint : 1
PICS used :
Color Control Cluster Test Plan.zip
On-Off Cluster Test Plan.zip

Environment :

Version: TH Fall2023
Sha: 27090a17

From chip-test-scripts created by bzbarsky-apple: CHIP-Specifications/chip-test-scripts#663

Description:

These tests are using RemoveFabric to have a fabric remove itself, then expecting a success response. The spec does not guarantee a success response in this situation, or indeed any response.

Steps to reproduce:

Run Test_TC_CADMIN_1_4 or Test_TC_CADMIN_1_6 or Test_TC_CADMIN_1_16 against an implementation that does not manage to send out a response before the fabric is removed, which is perfectly well allowed by the spec.

See also project-chip/connectedhomeip#29284

From chip-test-scripts created by KishokG: CHIP-Specifications/chip-test-scripts#681

Description:
Below mentioned test cases has to be changed from UI- Semi automated to UI- Manual or Automated. Because these all test cases has to be run in interactive mode and we can't able to run using interactive mode in UI if it is semi-automated test cases.

Test cases that require review:

• TC-SMOKECO-2.2
• TC-SMOKECO-2.3
• TC-SMOKECO-2.4
• TC-SMOKECO-2.5
• TC-SMOKECO-2.6

From chip-test-scripts created by KishokG: CHIP-Specifications/chip-test-scripts#679

Description:

• TC-FAN-1.1: Step 5 has to be added in YAML script as per test plan.

Script Snapshot:

Test Plan Reference:

From chip-test-scripts created by KishokG: CHIP-Specifications/chip-test-scripts#661

Description:

• TC-DRLK-2.10: Step 5a and 5e command has to be updated.
• TC-DRLK-2.1: Step 2b, step number has to be updated in script as per test plan.

Script Snapshot:

Test plan Reference:

Summary Title:

For TC-DRLK-3.2, the expected outcome in the verifications sheet doesn't show the right commandID values for steps 14, 16 and 17

Description:

For TC-DRLK-3.2, step 14, the expected commandID is 0x1b. The verification sheet shows 0x1c. Here is the spec screenshot for the same showing 0x1b.

The same for step 16 and step 17. The expected outcome for these steps are 0x22 and 0x24.

Steps to reproduce:

Please execute steps 14, 16 and 17 with chip-tool and compare the commandIDs that are shown in the log with the Spec and the Verification Document expected outcome for that step. Observe that the commandIDs shown by the chip tool logs and Specs match but the same is different from what is shown in the Verification Document expected outcome.

From chip-test-scripts created by bzbarsky-apple: CHIP-Specifications/chip-test-scripts#666

Description:

See project-chip/connectedhomeip#29375

The test expects fabric info entries to come in a particular order, but the spec does not require that.

From chip-test-scripts created by Survensa: CHIP-Specifications/chip-test-scripts#511

Description:

When all the test case in Access-Control-cluster is executed using PICS in UI, the below mentioned test cases are gets failed.

• TC-ACL-2.3
• TC-ACL-2.4
• TC-ACL-2.7
• TC-ACL-2.8
• TC-ACL-2.9

If the same test cases are executed separately it passed with same PICS uploaded in UI.

Scenario 1:
Executing all test-cases at same time : TC-ACL-2.7,TC-ACL-2.8 in UI is got stuck individually.

Logs: ACL- All tc's_stuck up 2.7.log , ACL- without 2.7_stuck up 2.8.log
Report: ACL- All tc's_stuck up 2.7.pdf ACL- without 2.7_stuck up 2.8.pdf

Scenario 2:
Executing all other test cases except the above one : TC-ACL-2.3, TC-ACL-2.4 and TC-ACL-2.9 are failed.

Log: ACL-without 2.7,2.8.log
Report: ACL- without 2.7, 2.8.pdf

Scenario 3:
Executing all the above test cases individually, the same is passed.

Logs: ACL- 2.3.log, ACL- 2.8.log, ACL- 2.9.log, ACL- 2.4.log, ACL- 2.7.log
Report: ACL- 2.7.pdf, ACL- 2.8.pdf, ACL- 2.9.pdf , ACL- 2.3.pdf, ACL- 2.4.pdf

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#684

Description:
[TC-TCTL-3.2] - Test script should be updated as per VS document by adding the Additional comment as "After a few seconds, please send the below mentioned command to read TemperatureSetpoint attribute from TH to DUT" for step 6, 8 and 11

[TC-TCTL-3.3] - Test script should be updated as per VS document by adding the Additional comment as "After a few seconds, please send the below mentioned command to read SelectedTemperatureLevel attribute from TH to DUT" for step 5 and 8.

Additional Info:
Version: TH Fall2023
Sha: 3f84bff5

Description:

Script has to be updated for below changes:
TC-DRLK-3.2: Command and log has been updated in VS doc. So script has to be updated for the same.
TC-ACT-3.2: Step 0 and 1 Command, comment and log has to be updated.

From chip-test-scripts created by KishokG: CHIP-Specifications/chip-test-scripts#678

Description:

• Test step 19 - Log has been updated in VS doc. So the script has to be updated for the same.

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#691

Description :
[TC-IDM-4.2] Step 6 - Script should be updated as per verification Document

PFA UI Screenshot :

PFA VS screenshot :

From chip-test-scripts created by Ashwinigrl: CHIP-Specifications/chip-test-scripts#638

Description:

[TC-OO-2.3] - Steps 12b and 21b failing due to OffWaitTime attribute value mismatch and throwing unexpected error during middle of the execution.

Steps to reproduce:

Step 12b, 21b

Snapt Shot:

Logs:

TC-OO-2.3.log

Environment :

TH Version: TH Fall2023
Sha: 22538eb7

Additional Info:

• DUT : All-clusters-app
• Endpoint : 1
• PICS used : On-Off Cluster Test Plan.zip

From chip-test-scripts created by Ashwinigrl: CHIP-Specifications/chip-test-scripts#680

Description:

TC-MOD-1.1 : In Step 2 Cluster Revision value has to be updated as per VS document

Steps to reproduce:

Step 2

Script Screen Shot:

VS Document Screen Shot:

From chip-test-scripts created by CoekaertsJasper: CHIP-Specifications/chip-test-scripts#674

Summary Title:

SVE 1.2 TC-IDM-10.1 regular Python script fails to account for duplicate values. DeviceBasicComposition script does catch the errors.

Description:

When reading attributes the IDM-10.1 Python script does not check for duplicate values in global attributes. This allows DUTS with these errors to pass. On the other hand, the DeviceBasicComposition version of the script does catch duplicate values correctly.

Steps to reproduce:

Run TC_IDM_10_1.py on a DUT that contains duplicate values in a global attribute. It will not catch the error and PASS the test case. 

Logs:

TC_IDM_10-1_Lock_Python_DeviceBasicComposition_log_FAIL.txt
TC_IDM_10-1_Lock_Python_log_PASS.txt

Additional Info:

/

From chip-test-scripts created by Ashwinigrl: CHIP-Specifications/chip-test-scripts#683

Description:

TC-MOD-3.4 : Step 6 is verifying the current Mode attribute value should be same as onmode attribute value after Physically power cycle the device. As default value of OnMode attribute is NULL. While verifying manually Precondition step has been added for setting the OnMode to any mode, values listed in the SupportedModes entries In Verification Step document So it is passing in Manual Execution . When we try to execute in UI, Script is failing in Step 6 because of CurrentMode and OnMode attribute value is mismatching.

Steps to reproduce:

Step 6

Logs:

TC-MOD-3.4.txt

Additional Info:

Raised Test plan Issue for the same :- https://github.com/CHIP-Specifications/chip-test-plans/issues/3504
Once the Test plan is fixed script has to be updated

Environment :

Version: TH Fall2023
Sha: 3f84bff5

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#693

Description :
[TC-OPSTATE-2.2] Step 16 and 17 - PICS should be updated in script has per VS

PFA Script screenshot :

PFA VS screenshot :

Test-Plan PR that updated PICS in these steps : https://github.com/CHIP-Specifications/chip-test-plans/commit/a0d33b87c34cc7851cb7254ffef12f280fe27bc5

From chip-test-scripts created by ericzijian1994: CHIP-Specifications/chip-test-scripts#667

Description:

In TC-DGGEN-2.1, the scripts and test plan with eventTrigger are inconsistent. And the DUT no longer requires step 9c to trigger an unknown function.

    - label:
          "Step 9c: When configured TRUE, Node has been configured with one or
          more test event triggers by virtue of the internally programmed non
          zero EnableKey value"
      runIf: IsExpectedValue
      PICS: DGGEN.S.C00.Rsp
      command: "TestEventTrigger"
      arguments:
          values:
              - name: "EnableKey"
                value: TEST_EVENT_ENABLE_KEY
              - name: "EventTrigger"
                value: TEST_EVENT_TRIGGER_KEY

In addition, the script has some descriptive errors at line 479: Step 10c should be Step 10d

Steps to reproduce:

./chip-tool generaldiagnostics test-event-trigger hex:00112233445566778899aabbccddeeff 0x0000000000000003 1 0

Additional Info:

Test plan: general_diagnostics.adoc

Description:

[TC-G-2.2 ] - In Step 1c Test step label has to be updated as per Test plan change

Steps to reproduce:

Step 1c

Yaml Reference:

https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_G_2_2.yaml

Test Plan Reference:

https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/Groups.adoc#tc-g-1-1-global-attributes-with-dut_server

Description:

Prompt from Step 15 uses node ID 1 which is used for TH_CR1 in previous steps.

    - label: "Step 15: TH_CR3 starts a commissioning process with DUT_CE"
      PICS: CADMIN.S
      verification: |
          On TH_CR3 send the below command

          Below is the example when using chip tool as controller (considering 36177160937 as the manual code generated by 1st controller)
          ./chip-tool pairing code 1 36177160937 --commissioner-name gamma

          verify you got the following message in the TH_CR3(chip-tool) log
          CHIP:SC: PASESession timed out while waiting for a response from the peer. Expected message type was 33
          CHIP:TOO: Secure Pairing Failed
          CHIP:TOO: Pairing Failure: ../../third_party/connectedhomeip/src/protocols/secure_channel/PASESession.cpp:324: CHIP Error 0x00000032: Timeout
      disabled: true

command should use nodeID other than 1 (TH_CR1) and 2 (TH_CR2).
eg. ./chip-tool pairing code 3 36177160937 --commissioner-name gamma

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#648

Description:
Secure channel - Script missed to add the PICS_MCORE_ICD for below mentioned test cases.

• TC-SC-4.1
• TC-SC-4.3

Test-Plan Reference :
https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/securechannel.adoc#tc-sc-4-1-commissionable-node-discovery-dut_commissionee

From chip-test-scripts created by bzbarsky-apple: CHIP-Specifications/chip-test-scripts#675

The script has a manual step telling the user to verify something about the event, but it's skipping the manual "Shutdown DUT." step, so there is no event to verify anything about.

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#688

Description :
[TC-WNCV-5.1] - Script should be updated by removing the PICS for 1a (PICS: WNCV.C.Afffc) Due to the PICS ( {PICS_CA_FEATUREMAP}.Read)) at 1a, Test-step is skipping in UI .

PFA UI Screenshot :

PFA Test-Plan screenshot :

PFA script screenshot :

Additional Information :
Version: TH Fall2023
Sha: 3f84bff5

Test-Plan updated : https://github.com/CHIP-Specifications/chip-test-plans/pull/3464

From chip-test-scripts created by Rajashreekalmane: CHIP-Specifications/chip-test-scripts#692

Description :

1. Notes/Testing Considerations Need to be updated.
2. Script Should be updated as Per Verification step Document.
   PFA Script screenshot
   

PFA Verification Steps Doc screenshot:

Description:

YAML script has to be updated for below test cases:

1. TC-ACE-1.5: Step 8 and 10 label has to be updated
2. TC-BRBINFO-2.2: Pre-condition step 4 has to be updated
3. TC-DGWIFI-3.2: Step 2 PICS has to be updated
4. TC-CNET-4.5: Step 2 PICS has to be updated
5. TC-CNET-4.9: Step 9 PICS has to be updated.
6. TC-CNET-4.10: Step 9 PICS has to be updated.
7. TC-DRLK-2.10: Step 5e Test step has to be updated(User status and User type)

Reproduction steps

Step 29 of TC-CADMIN-1.26 verbiage is incorrect:

Instructions to operator state
"Verify TH_CR1 logs for windowstatus to be 1".

This should instead read: "Verify TH_CR1 logs for windowstatus to be 2"

The example chip-tool output below is correct and shows a windowStatus attribute value of 2.

Bug prevalence

Consistent

GitHub hash of the SDK that was being used

5ccb774e7e5fe7ead3ff85bec1cda31a459c9c37

Platform

raspi

Platform Version(s)

No response

Type

Test Improvement

Testing

YAML tested

(Optional) If manually tested please explain why this is only manually tested

No response

Anything else?

No response