edgexfoundry / blackbox-testing Goto Github PK

Support (logging, notifications, scheduling, rules engine)
Export Client
Possibly device services and/or DS SDK

On the Schedule API, the runOnce field is used and set as if it was a long (as if it was in milliseconds). Now, this is a boolean and should be set to true or false.
In test #271 of core-metadata on schedule.

Post to localhost:48081/api/v1/schedule

{"origin":1471806386919,"name":"at midnight4","frequency":2, "cron":"0 15 10 * * ?", "runOnce":1471824000}

include better visualization/dashboard of test results (including look up and display of historical test results). Candidate tools include Telegraf + Grafana + InfluxDB, DataDog, Prometheus.

In the Delhi branch, the scheduler image name needs to be corrected:

arm64_env.sh = docker-support-scheduler-go-arm64
env.sh = docker-support-scheduler-go

Both are currently missing the "go"

According issue edgexfoundry/edgex-go#481,
change
tests[\"Response time is less than \"+data.responseTime] = responseTime < 200
to
tests[\"Response time is less than \"+data.responseTime+\"ms\"] = responseTime < data.responseTime

I think we need to create device data and all of the dependent entries that go with devices (profile, addressable, etc) for the coredata tests. Here's the use case:

By default in the core-data service the "MetaDataCheck" flag is set to false. When set to true, this flag causes core-data to call core-metadata in order to validate that the device Id/Name exists in the database. If it does not exist, a 404 is returned.

When the flag is set to its default (false) and I run the event tests from the core-data collection, I get 78 pass, 0 fail.

When the flag is set to true and I do the same, I get 48 pass, 7 fail. This is because there isn't any device data specified in the DataDumps/coredata folder that I can use to populate the database. Thus every call to core-metadata is returning "not found".

https://github.com/edgexfoundry/blackbox-testing/tree/master/bin/postman-test/DataDumps/coredata

For example: https://jenkins.edgexfoundry.org/view/blackbox-testing/job/blackbox-testing-cavium-arm64-master/2/console

In particular this line in the console logs: �[1BThe mongoSeed variable is not set. Defaulting to a blank string.
I believe the fix is to add an entry for mongoSeed in blackbox-testing/bin/arm64_env.sh

Assigned to blackbox testing as IoTech owns. Move developer documentation to GitHub (versus having it on the Wiki) so that it is maintained, updated and versioned more like code.

Using master's version, tests 340 and 350 in command_error_4xx tests fail. The Delhi version of these tests run green against the same code base. Looks like test materially changed.

Test 340, Delhi:

{
  "name": "340 http://localhost:48081/api/v1/command/id/:id",
  "event": [
    {
      "listen": "test",
      "script": {
        "type": "text/javascript",
        "exec": [
          "/*",
          " * Test Case:  /command - 409 Error - DEL ",
          " * Version: Alpha",
          " * @Author: Tata Elxsi",
          " *",
          " */",
          " ",
          "//Test Case for status : 409",
          "tests[\"Status code is 409\"] = responseCode.code === 409;",
          "//Test response time",
          "tests[\"Response time is less than \"+data.responseTime+\"ms\"] = responseTime < data.responseTime;"
        ]
      }
    }
  ],
  "request": {
    "method": "DELETE",
    "header": [],
    "body": {
      "mode": "raw",
      "raw": ""
    },
    "url": {
      "raw": "{{baseUrl}}/api/v1/command/id/{{commandDelRequestIderr}}",
      "host": [
        "{{baseUrl}}"
      ],
      "path": [
        "api",
        "v1",
        "command",
        "id",
        "{{commandDelRequestIderr}}"
      ]
    },
    "description": "Remove the Command designated by database generated id. ServiceException (HTTP 503) for unknown or unanticipated issues. NotFoundException (HTTP 404) if no Command is found with the provided id. DataValidationException (HTTP 409) if the Command is still associated to a Device Profile."
  },
  "response": []
},

Same test in master:

{
  "name": "340 http://localhost:48081/api/v1/command/id/:id",
  "event": [
    {
      "listen": "test",
      "script": {
        "id": "eed610b6-f5d2-4443-b924-d4fd322cb66f",
        "exec": [
          "/*",
          " * Test Case:  /command - 409 Error - DEL ",
          " * Version: Alpha",
          " * @Author: Tata Elxsi",
          " *",
          " */",
          " ",
          "//Test Case for status : 409",
          "tests[\"Status code is 409\"] = responseCode.code === 409;",
          "//Test response time",
          "tests[\"Response time is less than \"+data.responseTime+\"ms\"] = responseTime < data.responseTime;"
        ],
        "type": "text/javascript"
      }
    },
    {
      "listen": "prerequest",
      "script": {
        "id": "355c096f-7da1-4ac7-b787-be1fdf6931e0",
        "exec": [
          "var baseUrl = pm.environment.get(\"baseUrl\");",
          "",
          "pm.sendRequest(baseUrl+\"/api/v1/command/name/cooling point\", function (err, res) {",
          "    if (err) {",
          "        console.log(err);",
          "    } else {",
          "        pm.environment.set(\"commandDelRequestIderr\", res.json()[0].id);",
          "    }",
          "});"
        ],
        "type": "text/javascript"
      }
    }
  ],
  "request": {
    "method": "DELETE",
    "header": [],
    "body": {
      "mode": "raw",
      "raw": ""
    },
    "url": {
      "raw": "{{baseUrl}}/api/v1/command/id/{{commandDelRequestIderr}}",
      "host": [
        "{{baseUrl}}"
      ],
      "path": [
        "api",
        "v1",
        "command",
        "id",
        "{{commandDelRequestIderr}}"
      ]
    },
    "description": "Remove the Command designated by database generated id. ServiceException (HTTP 503) for unknown or unanticipated issues. NotFoundException (HTTP 404) if no Command is found with the provided id. DataValidationException (HTTP 409) if the Command is still associated to a Device Profile."
  },
  "response": []
},

The Delhi version makes a call to /api/v1/command/id/... passing an invalid static id. This results in the expected 409 result.

The master version first calls /api/v1/command/name/cooling point to obtain an id. It then attempts to use that id in a subsequent call to /api/v1/command/id/....

If the command collection in Mongo is empty, the first call returns a null result, the subsequent call to /api/v1/command/id/ (without an actual id parameter) returns a 404 and the test fails.

If the command collection in Mongo contains the data populated by core-metadata-importer, the first all returns a valid id, the subsequent call to /api/v1/command/id uses that valid id. This results in the data being deleted, the call returning a 200, and the test failing.

A similar issue exists with test 350.

In the California CI path, non-ARM blackbox tests are pulling the postman/newman image with the latest tag. This is causing the same problem as described in the following issue:

#61

Update the non-ARM environment shell script to reference the postman/newman_ubuntu1404:3.10.0 image

https://jenkins.edgexfoundry.org/view/blackbox-testing/job/blackbox-testing-centos7-blackbox-4c-2g-california/45/
and
https://jenkins.edgexfoundry.org/view/blackbox-testing/job/blackbox-testing-cavium-arm64-california/51/consoleText
@tsconn23 has some initial thoughts on the origins of the failure. Specifically the use of mongoimport

[INFO] Init postman test data .
Info: Initiating EdgeX Test
tee: ./postman-test/scriptLogs/edgex31-07-2018_150014.log: No such file or directory
/bin/bash: mongoimport: command not found
Info: /etc/newman/DataDumps/coredata/eventDb.json data imported
/bin/bash: mongoimport: command not found
Info: /etc/newman/DataDumps/coredata/readingDb.json data imported
/bin/bash: mongoimport: command not found
Info: /etc/newman/DataDumps/coredata/valueDescriptorDb.json data imported
Info: Initiating Coredata Test.```

I have just rebased from master and re-imported the core-data collection for running Postman tests. It appears the changes we've made to the service endpoint return codes as specified in the RAML for the Delhi release have not been accounted for in the tests. We need to do this for all services prior to the Delhi release. In the screenshot below, all 503s were replaced with 500s but the tests are still expecting 503s.

1.Invalid object ID
There are many code snippets will check id
just like
if !bson.IsObjectIdHex(id) { return ErrInvalidObjectId }
https://github.com/edgexfoundry/edgex-go/blob/master/core/data/clients/mongo-client.go#L343

if we pass ID is Invalid format , will occur error , like below test data
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/data/deviceData.json#L24

so I will modify test data to meet valid bson format

2.test script assert content-type is application/json that was error
like below test script , it is assert content-type is application/json,
but api haven’t set content-type application/json , and RAML also not define ,
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-metadata.postman_collection.json#L641

so I will remove the assertion in test script

3.Test script can’t correct verify response body
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-metadata.postman_collection.json#L2954
tests[\"Response Body is Empty\"] = responseBody === \"[]\";
can't verify response is empty array ,
so replace with
var jsonData = JSON.parse(responseBody);\n tests[\"Response Body is Empty\"] = jsonData.length === 0;

4.Test data error … invalid OperatingState “enabled” , invalid AdminState "locked"
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/DataDumps/command/deviceDb.json
constant should be uppercase

5.incorrect test parameter opState , adminState
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-metadata.postman_collection.json#L913
"url": "{{baseUrl}}/api/v1/device/{{InvalidId}}/opstate/{{opState}}" , opState real value is xxx ,in go code will occur verify error .
so it should change to valid data .

here are 4 test error in blackbox-testing ,
it's happen when post new event , post new reading , put reading
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-data.postman_collection.json#L361
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-data.postman_collection.json#L929
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-data.postman_collection.json#L293
https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-data.postman_collection.json#L1068

these testing all validate the associated value descriptor is non-existent ,
but after these PR edgexfoundry/edgex-go#247 , it can't validate value descriptor , because configuration.ValidateCheck is false .

so , do we still test the behavior of associated value descriptor is non-existent ?

When reviewing the daily blackbox tests job, I noticed the job is failing due to the following error message repeated over and over in the logs.

Starting blackbox-testing_volume_1 ... �[32mdone�[0m
�[1BPulling postman (postman/newman_ubuntu1404:)...
latest: Pulling from postman/newman_ubuntu1404
Digest: sha256:a94def5e27e103a1e13cc3b18c8d5a47e83af4f2bdd0ec1adb7a4e69409dd613
Status: Downloaded newer image for postman/newman_ubuntu1404:latest

  error: Newman >= v4 does not support the v1 collection format

  Use the Postman Native app to export collections in the v2 format

Example log here:
https://logs.edgexfoundry.org/production/vex-yul-edgex-jenkins-1/blackbox-testing-centos7-blackbox-4c-2g-california/51/console.log.gz

Example build here:
https://jenkins.edgexfoundry.org/view/blackbox-testing/job/blackbox-testing-centos7-blackbox-4c-2g-california/51/

See issue edgexfoundry/edgex-go#904

We need to remove all blackbox tests that verify support-logging endpoints dealing with labels since the API endpoints will be eliminated. These would be GET/DELETEs. Tests can be removed at any time. I'm not sure when the above issue will be assigned/completed, but it will be during the Edinburgh cycle.

edgexfoundry/edgex-go#877

Instead of setting docker images to use when running blackbox tests inline within the docker-compose file, these need to be provided from environment shell scripts that reference the appropriate image for a given architecture when run through the CI pipeline. Currently the postman image is set inline in the docker-compose file. It works for non-ARM CI, but breaks in ARM.

This test is obsolete once the following PR is merged into edgex-go:

edgexfoundry/edgex-go#617

This is because a change had been made whereby if the specified limit to the number of log messages (say, 1000) is greater than the configured ReadMaxLimit (say, 100) then the parameter supplied by the user will be changed to agree with ReadMaxLimit. Therefore an error to say there are more logs available than the user specified will never be thrown.

Create tests for performance measurements (the "Pi Tests" as this is the target platform) and provide reports on how close the system comes to achieving those goals

When I done the blackbox-testing for Delhi, I found some script cannot detect failed test cases.
For example,

If the result is not 423, it is still passed. Some errors were passed by this kind of script, so there are some real issues we haven't recorded.

In this test, the following should be indicated as failed but passed
core-data

core-metadata


core-command

It might be better to review other test scripts one by one.

OS version: ubuntu 16.04
EdgeX version: 1.0.0
Environment details: Docker container
Steps to reproduce:

1. start up EdgeX
2. send put command with a body which has an invalid formatting string /valuedescriptor of Core Data API
   for example, with the following request URL and body:

Method: PUT
URL: http://localhost:48080/api/v1/valuedescriptor
Body:
  {
  "name":"luminousIntensity",
  "min":"-200",
  "max":"200",
  "type":"F",
  "uomLabel":"degreecel23",
  "defaultValue":"0",
  "formatting":"%",
  "labels":["NHCO2","hvac"]
  }

Actual results: Got returned http status code 400 BadRequest

Currently, the expected http status code in the script for this test is 409.
However, according to the RAML: https://github.com/edgexfoundry/edgex-go/blob/master/api/raml/core-data.raml#L671
The expected http status code should be 400.

I am creating this issue to track changes that will need to be made to the core-data blackbox tests based on the following issue:
edgexfoundry/edgex-go#811

I have already found one:
"name": "95 http://localhost:48080/api/v1/event "

The following check in this test is invalid since IDs are now strings, the content of which for an event should always be based on a UUID (36 chars).

"    //Test body value",
"    tests[\"Is Body Conatins Id\"] = responseBody.length === 24;",

Rather than enumerating all of the specific changes that need to be made, I hope the forthcoming PR will be indicative.

To be consistent, the current Newman ARM image used for blackbox tests needs to be version 3.10.0. Currently in Nexus, we have nexus3.edgexfoundry.org:10004/postman-newman-arm64:3.9.4

A new ARM image needs to be built based on 3.10.0

Please add a LICENSE file to the root of this repo per https://wiki.edgexfoundry.org/pages/viewpage.action?pageId=21823866

The following PR in the edgex-go master branch contains changes to the RAML which I'm sure will impact blackbox tests.

edgexfoundry/edgex-go#423

We'll need changes to the support-logging tests in the master branch and how they handle/check status codes according to this new RAML. Once the test definitions have been updated, please assign @brandonforster for verification.

This is the first of many PRs for this effort.

In accordance with the following PR from developer-scripts

edgexfoundry/developer-scripts#85

We'd like to start using Consul v1.3.1 for Edinburgh, so this needs to be included with the blackbox test functionality

The following test is failing due to the URL below being sent to the metadata service

• 224 GET http://localhost:48081/api/v1/deviceservice/{{getDeviceServiceById}}

The key shown in the URL above is not present in the data json file and appears to be populated via javascript. The {{getDeviceServiceById}} value relies on a previous test 210 which has an error. I can duplicate the issue by doing the following:

1.) Clean Mongo
2.) Run the core-metadata-importer collection
3.) Under "Collections" on the left, expand the deviceservice folder and double click test 210
4.) In the right hand panel, change the URL to the following: {{baseUrl}}/api/v1/deviceservice/name/Common room thermostat deviceservice
5.) Hit the "Send" button
6.) At the bottom, "Test Results" will have the following error
There was an error in evaluating the test script: TypeError: Cannot read property '$ref' of undefined

In the console output, I see a successful response with the device service properties followed by the TypeError message. I would try to address this myself, but I'm unsure what the approach should be.

Update any tests applicable to the core-metadata/addressable endpoint based on changes to address this issue in edgex-go

edgexfoundry/edgex-go#881

As soon as 0.7.0 and 0.6.1 artifacts are available in staging we will need to update the tags here to point at them.

re: version revs in edgex-go

Import and clean test data by REST APIs instead of MongoDB scripts.

According https://github.com/edgexfoundry/edgex-go/blob/master/api/raml/core-metadata.raml#L621
postman should be modified api assertion at https://github.com/edgexfoundry/blackbox-testing/blob/master/bin/postman-test/collections/core-metadata.postman_collection.json#L2429

Assertion 409 should be change to 400.

OS version: ubuntu 16.04
EdgeX version: 1.0.0
Environment details: Docker container
Steps to reproduce:

1. start up EdgeX
2. send post command with a body which has an invalid formatting string /valuedescriptor of Core Data API
   for example, with the following request URL and body:
   Method: POST
Body: { "name":"co2", "min":"12", "max":"15", "type":"F", "uomLabel":"degreecel", "defaultValue":"0", "formatting":"%", "labels":["NHCO2","hvac"] }

Actual results: Got returned http status code 400 BadRequest
RAML: https://github.com/edgexfoundry/edgex-go/blob/master/api/raml/core-data.raml#L657

Current test script:

tests["Status code is 409"] = responseCode.code === 409;
tests["Response time is less than 200ms"] = responseTime < 200; 
var contentTypeHeaderExists = responseHeaders.hasOwnProperty("Content-Type");
tests["Has Content-Type"] = contentTypeHeaderExists;
if (contentTypeHeaderExists) { 
    tests["Content-Type is text/plain"] = responseHeaders["Content-Type"].has("text/plain");
}

The following test needs to have the verbiage in its "Response body is correct" check changed.

The verbiage has been changed in the following EdgeX-Go PR
edgexfoundry/edgex-go#604

Rather than "Payload too large", it should instead look for "than configured limit".

According to https://github.com/edgexfoundry/edgex-go/blob/master/api/raml/core-command.raml ,

• /device/{id}/opstate/{opState}
• /device/name/{name}/opstate/{opState}
• /device/{id}/adminstate/{adminState}
• /device/name/{name}/adminstate/{adminState}
  These four apis no need check response content-type

core-metadata

The APIs for Schedule put/post have start, end, and frequency parameters. These are typically converted to integers or time in the code, but the API calls for these to be passed as strings in JSON. In the API calls, they are sent as numbers.
For example, Test 276 on Schedule Error 4xx

localhost:48081/api/v1/schedule

{"id":"57bb49e6555e5218873e5a1b","start":1503360000,"end":1503390000,"frequency":60000, "cron":"0 0 13 * * :"}

Per this morning's Core Working Group call, I am going to add the existing PDF documentation to a "docs" folder off the root of this repo so that they are easy to find for devs coming into the project.

As a future enhancement, Andy and team will update the PDFs with any items that have changed since their initial creation.

• arm64_env.sh
• env.sh

Images pointed to 0.7.0 need to be pointed to 1.0.0

Also the scheduler image name needs to be corrected:

• arm64_env = docker-support-scheduler-go-arm64
• env = docker-support-scheduler-go

This needs to be done soon as the PR to update edgex-go has been created.

Will require Arm environment provided by Cavium (or others)

In the Core Metadata data,

The device with ID: '57fe0a4ce4b060e2df0aa10f' (line 510 of deviceDb.json) references the device service with ID: '57fddb0de4b060e2df0aa105' (line 402 of deviceserviceDb.json) which referenced an addressable with ID: '57fdda80e4b060e2df0aa102' which doesn't exist in addressableDb.json.

In the Core Data data there are a couple of issues:

1.  The syntax for some of the reading references is incorrect which is causing the ID for the reading to be nil when I query in Go.  For example, at line 500 of eventDb.json, the reading reference looks like this:
   

"readings": [
{
"_id": {
"$oid": "57e59a71e4b0ca8e6d6d4cbb"
},
"$ref": "reading"
}
]

The correct syntax, which is done in most of the file, is:

{
"$ref": "reading",
"$id": {
"$oid": "57e59a71e4b0ca8e6d6d4cc0"
}
}

The ID parameter needs to be "$id" and NOT "_id". The "$id" field of a Mongo DBRef is what actually holds the "_id" field for the referenced object. The "$id" field for the incorrect objects is nil which is causing my query to fail.

2.  Also, there is an event that references a reading that doesn't exist.  Event: 57ba04a1189b95b8afcdaed1  (line 694 of eventDb.json) references reading with id 57e745efe4b0ca8e6d711ef1 and this reading is not in the data.
   

These will be the daily images built from master that are pushed to staging.

On many of the tests – specifically many around device and device service, the use of lowercase operating state and admin state are still being used. Per the decision of the community to follow Google Code Styles, all the enums are now supposed to be uppercase. The code was changed in all the services to reflect this, but the tests are still using lowercase which will cause false errors.

Both @JPWKU and I strongly recommend updating the blackbox test docker-compose file to version 3 from version 2. The rationale is two-fold:

1.) The California 0.6.1 release docker-compose uses version 3 as will releases going forward
2.) The new ARM builders we are provisioning will be running recent versions of docker CE and docker-compose. We would like to avoid some of the incompatibility issues we've had in the past that required manual action during the release.

There's a test that fails in the core-data collection which calls the following URL as an HTTP DELETE

localhost:48080/api/v1/event/removeold/age/1000000000000

It fails because the controller returns 0 since no events were deleted. No events were deleted because in the DataDumps/coredata/eventDB.json, none of the events have a specified age.