adobe / athena Goto Github PK

• Move config.js at repo root level.
• Allow for arbitrary directory structure when collecting tests.
• Automatically detect fixtures when a specific test is passed through the -t option.

Recommended solution: handle CLI with yargs instead of commander.

ensure that every child ref from parrent is overwritten with same model and nothing else

🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet.
We recommend using:

• CircleCI
• Travis CI
• Buildkite
• CodeShip
• Azure Pipelines
• TeamCity
• Buddy
• AppVeyor
  But Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.

• Write and validate schema for tests.
• Write and validate schema for suites.

Tests should not present a dependency to a suite.

Suite model is potentially wrong. A suite should look like:

.....
scenario:
given: >
....
then: >
....
when: >
....
tests:
   - ref: "name"
      version: "x.x"
      pottential overriding fields (must respect test model)

Please change the entity parsing logic to refer tests inside suites not the other way around

Could you please take some time in the next few days to make changes to some terminology in your repos and content as much as is possible:

• Whitelist/blacklist to Allowed List and Blocked List (or Approve List/Deny List - some software uses this instead) respectively. Google and many developers are formalizing allowlist and blocklist. You might want to lobby for those terms to be used in the UI.
• Master/Slave to master and replica (or subordinate, if that makes more sense) respectively.

If you cannot remove the term because the writing for example reflects the UI or the code, please make a note and send me an email to [email protected] so we can bring it that team’s attention. Thanks for your efforts in this matter.

In order to speed up the writing and the debugging of Atena tests, a script that translates between cURL commands would be ideal. This assumes a stable enough and predictable model, as suggested here: #3.

Create unit test collection for the framework in order to test several critical parts.

Currently, when fixtures are defined and loaded, there is no system of skipping the already loaded fixtures resulting in potential conflicts.

Feature

• Provide support for running Athena as a manager node.
• Allow Athena agents to join an already set-up cluster using an access token generated by the manager node.
• Use the athena CLI to manage the cluster and delegate performance tests to all agents inside the cluster.
• Provide support to the Athena manager to aggregate the results from all working nodes and store them inside a local database (sqlite?).
• Allow the Athena manager to report the tests result inside a CI/CD environment.

• Improve the existing fixtures support.

Currently, the passing tests simply show up in a simple list. A flag (e.g. --verbose) should be provided so that each expected item within the passing test can be inspected. Examples where this is useful:

• test passes under multiple conditions that are discernible from the assertions
• report information can be further used (part of the response, the result of test fixtures, etc)

Expected Behaviour

Ability to be injected arround k8 pod and on server around a docker image. Start tests and intercept outbound traffic.

Actual Behaviour

N/A

Reproduce Scenario (including but not limited to)

N/A

Steps to Reproduce

N/A

Platform and Version

0.X

Sample Code that illustrates the problem

Known issue

Logs taken while reproducing problem

N/A

Ensure registration of pattern and context with overriding for referenced perf runs

In order to understand the state of a failing test, a list of all the failing assertions should be provided. See an example of faulty reporting below:

• test file:

    expect(response).to.have.header("pragma", "no-cache"),
    expect(response).to.have.header("accept-ranges", "bytes"),
    expect(response).to.have.header("X-First-Bad-Header", "this_header_enjoys_espresso"),
    expect(response).to.have.header("X-Second-Bad-Header", "this_value_drinks_only_decaf")

• output:

     AssertionError: expected header X-First-Bad-Header with value undefined to match this_header_enjoys_espresso

Need to register perfRun in perfPattern and map configs hooks and functions

By offering a plugin as part of the base platform, please also consider a more keyword-based / YAML-structured alternative to what is currently offered. While clearly flexible, the code-heavy model can be a source of inconsistency (e.g. numerous equivalent ways to express the same thing, arbitrary function calls). I raise this issue because many aspects related to API testing are pretty standard (there are only so many types of entities attached to a request or expected from a response).

The coding capabilities need not be removed from this alternative, but certain aspects look repetitive enough and can be addressed case by case (for instance, checking multiple headers requires the same lengthy call many times). I understand that this means adding custom logic where out-of-the-box functionalities already took care of things, yet I consider this feature valuable as long as it does not eat too much development time.

What do you think is the proper way to include this functionality?

The url should be set using an env variable

Expected Behaviour

http://localhost:9200 should be the default if env variable ELASTICSEARCH_URL is missing

Actual Behaviour

Elasticsearch url is hardcoded to http://localhost:9200

Expected Behaviour

Json array stays as array when removing empty elements

Actual Behaviour

Json array is transformed in map when removing empty elements

Reproduce Scenario (including but not limited to)

Steps to Reproduce

Set examples/performance/perfRun.yaml to

name: "apiKey"
version: "1.0"
description: ""
engine: autocannon
type: perfRun
config:
  url: https://api-gateway-qe-d-ue1.adobe.io/test
  connections: 2000
  duration: 30
  method: GET
  requests: # Array of objects.
    - headers:
        host: keepalive-with-upstream-test-stage-service-1.adobe.io
      method:
      path:
    - headers:
        host: baseline-performance-testing-stage-service-1.adobe.io
hooks:
  onInit:
  onDestroy:
  onRequest:

and run athena

Write a comprehensive integration test suite that goes through all functionalities exposed to the user.

The command line arguments given when running performance tests in cluster mode are not propagated to ManagerNode and are discarded.

Expected Behaviour

When issuing a command to master to run performance tests from a given path, the tests located at the given path are executed.

Actual Behaviour

When issuing a command to master to run performance tests from a given path, the tests that are located at the default path are used instead.

Steps to Reproduce

node athena.js cluster --init --addr 0.0.0.0
node athena.js cluster --join --addr 0.0.0.0:5000 --token abcd

node athena.js cluster --run --performance --tests gw-tests/performance/upstream-directive/baseline

pm2 logs athena-agent
pm2 logs athena-manager

Expected Behaviour

Add REST API for starting up functional/performance tests. Since these might be a long running process present corelationId to client so he can later query for the status of the job.

Actual Behaviour

Reproduce Scenario (including but not limited to)

Steps to Reproduce

Platform and Version

Sample Code that illustrates the problem

Logs taken while reproducing problem