Is your feature request related to a problem? Please describe.

Monterey currently does not have any method of loading indices and data from a file to OpenSearch, which limits the ability for functional test developers to setup custom test environments. It also lacks a method of unloading data from OpenSearch, making it difficult to tear down the test environment and return OpenSearch to a clean state. The lack of these two features increases the difficulty of porting over existing OpenSearch Dashboards tests that require specific test environments and constrains the testing capabilities of the Monterey framework.

Describe the solution you'd like

Monterey should be able to parse the contents of a file containing JSONs that represent indices and data elements, and load and unload this parsed data through the use of the OpenSearch REST API. Ideally, this should be implemented as Cypress custom commands using the Cypress readFile and request functions to smoothly integrate the loading and unloading functionality into the existing Monterey framework.

Describe alternatives you've considered

One alternative is to directly use the osd-opensearch-archiver package, which is the existing data file loading and unloading implementation in the OpenSearch Dashboards testing framework. While utilizing this package would have the advantage of providing a solution that is already tailored to Monterey's specific needs, osd-opensearch-archiver has dependencies on another OpenSearch Dashboards package as well as on an Elasticsearch package, which in turn have dependencies on other packages. Handling the dependencies for these packages would increase the overhead for maintaining the Monterey framework, as well as generally increase the complexity of the project; thus, I find the osd-opensearch-archiver package approach to be non-ideal if there exists a simpler solution, especially for the initial proof of concept stages of Monterey.

Additional context
N/A

Describe the bug

When searching for a specific option in a EuiComboBox field and clicking on the desired option, the element sometimes enters an invalid state rather than select the option. The invalid state is visually indicated through a red line along the bottom of the input field.

This bug happens inconsistently, and usually results in the test case failing due to preventing Cypress from performing subsequent interactions with UI elements (e.g. Cypress being unable to press the submit button since the option was not properly selected). It is likely caused by a race condition between the combo box re-rendering after the selection of the option and the combo box checking whether it is in the invalid state.

To Reproduce

This bug is difficult to reproduce at-will due to it occurring inconsistently. The current best method would be:

1. (Optional) add .only() to the 'Explore the data in the dashboard' test suite in the demo_spec.js file (e.g. it.only('Explore the data in the dashboard',... )
2. Open the Cypress Test Runner (e.g. npx cypress open) and run the demo_spec.js test
3. Rerun the test until the error occurs, most frequently when selecting Gnomehouse in the Manufacturer field or when selecting day_of_week in the Add filter field input

Expected behavior

Cypress should consistently be able to type into the input field of the combo box, find the desired option, and click on the option to select it without the combo box entering an invalid state.

OpenSearch Version

1.0.0-SNAPSHOT

Dashboards Version

1.0.0

Additional Context

The EuiComboBox is a option list UI element that allows users to search for options and select multiple options if wanted. The OpenSearch Dashboards interface uses combo boxes to handle functionality that potentially requires searching a large number of options, such as in the Add filter panel.

When developing OpenSearch Dashboards functional tests in Cypress, the standard approach to select an option in a combo box has been the following series of actions:

1. Open the combo box if it is not already visible (e.g. click the add Filter button)
2. Click on the input field and type the keyword of interest (e.g. click on the Field input and type day_of_week)
3. Ensure that the list has properly updated by asserting that the number of visible options is equal to the expected amount (e.g. assert that there are two options remaining after searching for day_of_week)
4. Click on the desired option (e.g. click on the day_of_week option)

The bug of entering the invalid state has been occurring in step 4, after clicking the button. The EuiComboBox source code contains the following comments and code about the invalid state:

// Visually indicate the combobox is in an invalid state if it has lost focus but there is text entered in the input.
// When custom options are disabled and the user leaves the combo box after entering text that does not match any
// options, this tells the user that they've entered invalid input.
const markAsInvalid =
      isInvalid ||
      ((hasFocus === false || isListOpen === false) && searchValue);

The above indicates that the invalid state is being caused by the list being closed or losing focus while there is still a searchValue in the input. This suggests that in the context of the bug, there is likely a race condition created when the option is clicked and the option list closes:

• In the first scenario, the combo box updates with the option selection first, clearing the searchValue and preventing the invalid state from being true.
• In the second scenario, the combo box checks the invalid state first and finds that the list is closed and the searchValue is filled, concluding that the invalid state is true.

Assuming that this evaluation of the bug's cause is correct, the fix will likely involve addressing this race condition and ensuring that the option selection completes before checking the invalid state.

• List all UI Components
• Evaluate all supported operation on components (Mouse, Keyboards events, setValue, getValue)
• Gap analysis if don't support of flaky

As a feasibility POC, this issue is to create a test plugin to consume the monterey library. Since currently there is no page module library in monterey yet, we will first need to add a simple resuable function in monterey and then call it inside a test plugin.

• Define and proposal best practices to organize test cases by group of ui component
• Automation test cases for all supported UI components

• run test in Jenkins - P1
• run test in Github Action - P2
• launch OpenSearch and OpenSearch Dashboards from snapshot P2
• launch OpenSearch and OpenSearch Dashboards from build-from-source P3
• run test against local OpenSearch and OpenSearch Dashboards P1
• run test against remote OpenSearch and OpenSearch Dashboards P2
• generate test artifacts and upload
  • snapshot/video P1
  • test result summary (Junit) P2
  • code coverage (nice to have) P3

• Start Date: 2021-08-27
• RFC PR: (leave this empty)

Summary

Build a set of Cypress-based test fixture handling functions to allow for importing and deleting OpenSearch indices and their corresponding test data.

Basic example

//Sample Cypress spec file importing and removing test fixture data
describe('Sample test suite', () => {
  
  // At the start of the test suite
  before(() => {
    // remove the indices from OpenSearch in case they already exist
    cy.clearJSONMapping('path/to/fixtures/mappings.json.txt')

    // import the indices in the given file
    cy.importJSONMapping('path/to/fixtures/mappings.json.txt')

    // import the documents that will populate the newly added indices
    cy.importJSONDoc('path/to/fixtures/data.json.txt')
  })
  
  // At the end of the test suite
  after(() => {
    // delete the indices and documents to clean up the OpenSearch state
    cy.clearJSONMapping('path/to/fixtures/mappings.json.txt')
  })
  
  it('Test using the imported indices', () => {
    // perform the rest of the test cases that will use the imported data
    ...
  }) 
})

Motivation

Many functional tests in the existing OpenSearch Dashboards framework expect OpenSearch to contain specific indices and documents, which are pre-packaged in test fixture files and imported using the OpenSearch REST API. These test fixture files contain the required OpenSearch state for a test category (e.g. for all of the dashboard tests) or for a specific test file (e.g. for the _date_nanos.js test), with the indices and documents being stored in separate files as JSONs. The test files utilize the osd-opensearch-archiver package to handle the test fixtures, which has the functionality to load (import) and unload (remove) the data contained in the fixture files.

In order to minimize the overhead of migrating functional tests from the current framework, Monterey should have the same functionality to handle the existing test fixtures. Without using the fixture files, developers would either need to hard-code OpenSearch REST API calls into test files to import the desired indices or redesign the test cases to work with test data already available in the OpenSearch instance. The test fixture handling implementation in Monterey should be able to load and unload fixture files using the OpenSearch REST API, and be callable or runnable from test files to set up and tear down the OpenSearch test environment as needed.

Detailed design

I decided to implement the fixture handling functionality using Cypress custom commands to make the loading and unloading functions easily accessible for test files to directly call. The proposed set of three custom commands will be able to import a fixture file containing indices (cy.importJSONMapping), import a fixture file containing documents (cy.importJSONDoc), and remove the indices contained within a fixture file (cy.clearJSONMapping).

Test fixture specifications

In the existing Selenium-based framework, the two types of fixtures files are mappings.json files that contain all of the indices and their corresponding mappings and settings, and data.json.gz files that contain the documents. In both files, each index or document JSON is separated by two newline characters (\n\n).

For Monterey’s implementation of the fixture handling functionality, these files have been changed to mappings.json.txt and data.json.txt respectively. data.json.gz had to be unzipped since the cy.readFile command does not support reading gzip’d files. As well, both files have end with a .txt extension to prevent cy.readFile from automatically parsing the .json files as a single JSON object and failing.

mappings.json.txt

The index JSONs inside of mappings.json.txt can have the following structure:

{
  "type": "index",
  "value": {
    "aliases": {
      ...
    },
    "mappings": {
      ...
    },
    "settings": {
      ...
    }
  }
}

The allowable contents of the aliases, mappings, and settings fields are detailed in the OpenSearch create index API documentation.

data.json.txt

The document JSONs inside of data.json.txt can have the following structure:

{
  "type": "doc",
  "value": {
    "id": ...,
    "index": ...,
    "source": {
      ...
    }
}

The allowable contents of the id and index fields are detailed in the OpenSearch bulk API documentation. The source field should contain the document data to be indexed.

General command structure

All of the added Cypress custom commands perform three general operations needed to handle a test fixture file:

1. Read in the test fixture file and separate the content into JSON strings
2. Extract relevant information from the index or document JSON objects
3. Send a OpenSearch REST API request with the desired operation and data

The code below is a sample custom command that demonstrates the general flow of work, and will be referenced in the following subsections.

Cypress.Commands.add('SampleFixtureHandler', (filepath, ...) => {
  
  // 1) read the test fixture file in as a string
  // and split the string into separate JSONs
  cy.readFile(filepath, 'utf8').then((str) => {
    const JSONStrArray = str.split('\n\n')
    
    // 2) extract and format the contents of each JSON
    JSONStrArray.forEach((element) => {
      const JSONContent = sampleJSONProcesser(JSON.parse(element))
      
      // 3) send the OpenSearch REST API request
      cy.request(...).then((response) => {
    
      })
    })
  })
})

1) Reading the fixture file

The first step when handling test fixture files is to read and process them. The functions use the Cypress API command cy.readFile to read the fixture file at a specific path and return its contents as a string. Afterwards, the string gets split by the chosen delimiter between the JSONs inside the file, two newlines characters (\n\n). The returned array of JSON strings is then processed in future steps.

2) Extracting JSON information

Once the function has the parsed array of JSON strings, the second step is to iterate through each JSON and extract the relevant information. After the JSON has been turned into a JSON object with JSON.parse, the fields needed for the import or remove operation are reformatted into a request body and/or header which is the example above would be done in the dummy function sampleJSONProcessor.

3) Send OpenSearch REST API request

The final step is to send the formatted JSON index or document information to OpenSearch with a network request using the cy.request command. The request’s method, URL, header, and body are set based on the type of operation being completed and the extracted JSON data itself. The default base URL to send requests to is localhost:9200, the default host and port of OpenSearch.

Specific command implementations

cy.importJSONMapping

Cypress.Commands.add('importJSONMapping', (filepath, hostname = 'localhost', port = '9200') => {
  // helper function for parsing index JSONs
  const parseIndex = ({ index, settings, mappings, aliases }) => {
    
    // organize the "settings", "mappings", and "aliases" fields inside a "body" field
    return { index, body: { settings, mappings, aliases } }
  }

  cy.readFile(filepath, 'utf8').then((str) => {
    const strSplit = str.split('\n\n')
    strSplit.forEach((element) => {
      const json = JSON.parse(element)
      
      // only send the OpenSearch create index API request
      // if the JSON represents an index
      if (json.type === 'index') {
        const indexContent = parseIndex(json.value)
        
        // send the OpenSearch create index API request
        // to create an index with the same name as the "index" field of the JSON
        // using the specifications of the formatted "body" field
        cy.request({ method: 'PUT', url: `${hostname}:${port}/${indexContent.index}`, body: indexContent.body, failOnStatusCode: false }).then((response) => {

        })
      }
    })
  })
})

The cy.importJSONMapping function (as defined above) reads in a mappings.json.txt file, and for each index JSON parses its information and sends an OpenSearch create index API request. The request is sent to the specified host name and port, and should not fail the test case if the response’s status code is not 200 (such as when the new index already exists in OpenSearch).

cy.clearJSONMapping

Cypress.Commands.add('clearJSONMapping', (filename, hostname = 'localhost', port = '9200') => {
  cy.readFile(filename, 'utf8').then((str) => {
    const strSplit = str.split('\n\n')
    strSplit.forEach((element) => {
      const json = JSON.parse(element)
      
      // only send the OpenSearch delete index API request
      // if the JSON represents an index
      if (json.type === 'index') {
        const index = json.value.index
        
        // send the OpenSearch delete index API request
        // to delete the index with the same name as the "index" field of the JSON 
        cy.request({ method: 'DELETE', url: `${hostname}:${port}/${index}`, failOnStatusCode: false }).then((response) => {
        })
      }
    })
  })
})

The cy.clearJSONMapping function (as defined above) reads in a mappings.json.txt file, and for each index JSON parses its information and sends an OpenSearch delete index API request. The request is sent to the specified host name and port, and should not fail the test case if the response’s status code is not 200 (such as when the index to be deleted does not already exist in OpenSearch).

Since the delete index API only requires the index’s name to delete it, the settings, mappings, and aliases fields do not need to be parsed. As well, deleting the index consequently deletes all of the documents stored inside it, so there is no current need for a function that bulk removes documents when cleaning up the state of OpenSearch.

cy.importJSONDoc

Cypress.Commands.add('importJSONDoc', (filename, hostname = 'localhost', port = '9200', bulkMax = 1600) => {
  
  // local helper function for parsing document JSONs
  const parseDocument = ({ value: { id, index, source } }) => {
    
    // format the action (id and index) and source into two NDJSONs and combine them
    // following the OpenSearch bulk API format
    const actionData = { index: { _id: id, _index: index } }
    const oneLineAction = JSON.stringify(actionData).replace('\n', '')
    const oneLineSource = JSON.stringify(source).replace('\n', '')
    const bulkOperation = `${oneLineAction}\n${oneLineSource}`
    return bulkOperation
  }

  // local helper function for sending a bulk API network request,
  // joining all of the combined action and source NDJSON strings
  // into one large string for the request body.
  const sendBulkAPIRequest = (bodyArray, hostname, port) => {
    cy.request({ headers: { 'Content-Type': 'application/json' }, method: 'POST', url: `${hostname}:${port}/_bulk`, body: `${bodyArray.join('\n')}\n`, timeout: 30000 }).then((response) => {

    })
  }

  cy.readFile(filename, 'utf8').then((str) => {
    
    // keep track of how many JSONs have been processed
    let readJSONCount = 0
    
    // create a nested array to store combined action and source NDJSON strings
    // before they get sent in a request
    const bulkLines = [[]]
    str.split('\n\n').forEach((element) => {
      
      // store the processed combined action and source NDJSON string in the nested array
      bulkLines[0].push(parseDocument(JSON.parse(element)))

      // update the read counter
      readJSONCount++
      
      // if the number of combined action and source NDJSON strings stored
      // in the nested array is equal to the set threshold
      if (readJSONCount % bulkMax === 0) {
        // Pop the nested array and use it as the body of a bulk API request
        sendBulkAPIRequest(bulkLines.pop(), hostname, port)
        
        // add a new nested array to store future NDJSONs
        bulkLines.push([])
      }
    })

    // if there still exists NDJSONs to be sent to OpenSearch
    if (bulkLines.length > 0) {
      // Pop the nested array and use the remaining NDJSONs in a bulk API request
      sendBulkAPIRequest(bulkLines.pop(), hostname, port)
    }
    
    // refresh all indices to ensure that the imported document data is accessible
    cy.request({ method: 'POST', url: `${hostname}:${port}/_all/_refresh` }).then((response) => {

    })
  })
})

The cy.importJSONDoc function (as defined above) reads in a data.json.txt file, and for each document JSON parses and stores its information as a newline-delimited JSON (NDJSON) string. After the number of locally stored NDJSONs exceeds a threshold (by default 1600), the documents are compiled into one NDJSON and sent in an OpenSearch bulk API request. After all of the document JSONs have been read and imported, a final OpenSearch refresh API request is made to ensure that the imported document data is available for searching.

The NDJSON strings representing each document JSON are stored and sent in batches since bulk requests are more efficient than making index requests for individual documents (read more here). The default batch size of 1600 was selected due to being the largest batch size that did not start to significantly impact the bulk import performance on my test environment, and different batch sizes may be preferrable based on the environment the tests and OpenSearch are running in.

The method of using nested arrays to store the NDJSON strings and popping them when their size meets the batch size threshold is intended to reduce the amount of strings stored in local memory at a time, which is relevant to a scalability issue that is detailed in the “How this scales” section further below.

Usage in test files

The proper time to use the test fixture loading and unloading commands is at the start and end of test files or groups of test files. For example, the code below is from the cypress/integration/dashboard/dashboard_filtering_spec.js test file, and uses the commands to import the needed indices and documents before any tests run and remove the added indices after all tests finished.

// cypress/integration/dashboard/dashboard_filtering_spec.js
describe('dashboard filtering', () => {
  
  // before running any of the tests in the file 
  before(() => {
  
    // remove the indices from OpenSearch in case they already exist
    cy.clearJSONMapping('cypress/fixtures/dashboard/data/mappings.json.txt')

    // import the indices needed for the file's test cases
    cy.importJSONMapping('cypress/fixtures/dashboard/opensearch_dashboards/mappings.json.txt')
    cy.importJSONMapping('cypress/fixtures/dashboard/data/mappings.json.txt')

    // import the documents that will populate the newly added indices
    cy.importJSONDoc('cypress/fixtures/dashboard/opensearch_dashboards/data.json.txt')
    cy.importJSONDoc('cypress/fixtures/dashboard/data/data.json.txt')
  })

  // at the end of the test file
  after(() => {
  
    // delete the indices and their data to clean up state of OpenSearch
    // NOTE: the opensearch_dashboards/mappings.json.txt is not being cleared
    // since it contains the specifications for the ".kibana_1" index,
    // which if deleted using the current fixture handling functionality cannot
    // properly be re-imported
    cy.clearJSONMapping('cypress/fixtures/dashboard/data/mappings.json.txt')
  })
  
  it('Buffer test for importing test environment data', () => {

  })

  describe('Adding and removing filters from a dashboard', () => {
    // perform the rest of the test cases that will use the imported test data
    ...

The cy.importJSONDoc command should only be called with a specific data.json.txt file after cy.importJSONMapping had been called with the corresponding mappings.json.txt file to ensure that the imported documents have indices to be indexed into.

As well, the cy.clearJSONMapping command should not be called on any mappings.json.txt files that contains a .kibana_* index with * representing any integer. The current functionality is unable to properly re-import any deleted .kibana_* indices and future iterations will have to address this issue.

The Buffer test for import test environment data test case above exists to prevent the high workload being performed in the before() block from slowing down future tests. The before() block exists in the context of the first test case of the file, and I sometimes experienced that first test case slowing down dramatically after the completion of the before(). My solution was to have an empty test case as a buffer to prevent the before() block from impacting the performance of the test file.

Drawbacks

One drawback of this design is that implementing the test fixture loading and unloading functionality as Cypress custom commands couples it to the Cypress framework, as does using Cypress API commands such as cy.readFile and cy.request. This means that this implementation would not be supported for use with any other test driver (such as Selenium) or context besides the Cypress framework.

Another downside of this design is that it does not yet contain all of the test fixture handling capabilities that osd-opensearch-archiver does, only the core functionality needed to successful import and remove the test fixture files. Future iterations of the test fixturing handling functionality will have to continue to expand their functionality, such as being able to save new fixture files, edit existing fixture files, and properly clean up the .kibana indices in OpenSearch.

A third downside is the design’s scalability issues on the size of the test fixture file, which would require putting limits on the size of test fixture files. This issue is further discussed in the “How this scales” section below.

Alternatives

The main alternative design is to try and adapt the original osd-opensearch-archiver package to be usable in Cypress. The main advantage of using osd-opensearch-archiver directly is that it would reduce the amount of research time spent to understand the package’s inner mechanisms in order to rebuild the functionality from scratch, as has been done for the Cypress custom commands. As well, the package was already built for the Selenium-based testing framework, and having a general purpose fixture handling library that could run on both Cypress and Selenium would improve the portability of the Monterey platform.

A major blocker for using the osd-opensearch-archiver package, however, is Cypress’ lack of native support for async/await calls in its code. osd-opensearch-archiver heavily utilizes async/await and thus would require developers to find a potential workaround for using async/await in between Cypress commands, or call osd-opensearch-archiver through a Node script using the cy.exec command and lose access to log messages and other debugging capabilities.

Another impediment of utilizing osd-opensearch-archiver is its dependency on local packages in the OpenSearch Dashboards repository and on the Elasticsearch library (as can be seen in the package’s package.json file). The only local package osd-opensearch-archiver is directly dependent on is @osd/dev-utils; however @osd/dev-utils is dependent on @osd/utils, which in turn is dependent on @osd/config-schema. Using osd-opensearch-archiver will either require refactoring its code to remove all usages of @osd/dev-utils and replace them with internal functionality, or to keep all of these local packages in the Monterey framework, adding additional bloat to the code base. The dependency on the Elasticsearch library should also be considered for removal to keep Monterey framework decoupled from any Elasticsearch or Kibana implementations, which would require replacing any usage of the Elasticsearch Node.js client with local functionality (or the OpenSearch Node.js client when it is developed).

Adoption strategy

Since the Monterey framework is still in its early stages of development and does not yet have many test files prepared, it should be easy to adopt this test fixture handling functionality. Any new Cypress functional tests that are adapted from the existing Selenium-based framework can incorporate the Cypress custom commands to replace code in the original test file that used osd-opensearch-archiver to load and unload test fixture files.

For new Monterey-specific test files, developers can decide whether to use test fixtures or a different approach to preparing the state of OpenSearch, and in the later case, the use of the fixture handling commands would be unnecessary.

How this scales

The current implementation of the test fixture handling custom commands has a scalability issue due to the use of cy.readFile. cy.readFile reads in the entire test fixture file at once synchronously rather than as an asynchronous data stream. This works fine for relatively small files but can cause issues once files get large enough due to the amount of local memory being used to hold the file’s contents as a string.

An example of this is while running the dashboard_filtering_spec.js Cypress test file using the Electron browser. When trying to import the data.json.txt fixture file in the cypress/fixtures/dashboard/data directory of Monterey (which is ~66.4MB large), the test hangs on the execution of cy.readFile for an extended period of time, after which it timeouts on the first attempted bulk API request to OpenSearch.

Table 1 below shows the results of experimenting with different fixture files sizes when running the loading and unloading sections of the dashboard_filtering_spec.js test file in different browsers and running modes. The tests were run in Cypress 7.6.0 locally (MacBook Pro 2019, Big Sur 11.4), and each browser/file size combination was ran three times and averaged. The elements highlighted in red are the largest fixture file sizes that the specific browser and running mode ran three times without failing.

Table 1: data.json.txt Fixture File Size vs Runtime of Successful Fixture Loading and Unloading

From the table above, running cy.importJSONDoc on the Electron and Chrome browsers had upper limits on the size of data.json.txt at which point the began to command failed. More experimentation will need to be done running the fixture handling custom commands on CI environments with more powerful hardware specs to determine if the issue is with the browsers themselves running out of memory, or with running the tests locally.

The osd-opensearch-archiver implementation of this functionality does not have this issue since it reads the fixture files as a stream, reading the file chunk by chunk until it reaches the separator between JSONs, at which point it parses the JSON. This approach avoids reading in the entire file at once and needing to store its entirety in memory.

The main implications of this scalability issue are that either the test fixture file size will have to be capped to have this fixture handling functionality be supported with all three browser types (requiring the dashboard_filtering_spec.js test file to be modified to use a smaller fixture file), or that the Electron browser will not be supported in the Monterey project when running tests that utilize test fixture files. A long term development goal should be to fix this scalability issue, most likely by replacing the cy.readFile API call with JavaScript read and write streams to process the test fixture file in chunks.

How we teach this

Since this test fixture handling functionality already existed in OpenSearch Dashboards test files, the main way to teach it would be through documentation detailing the difference between the new Cypress custom commands and osd-opensearch-archiver, as well as sample code demonstrating the use cases of the functions.

Unresolved questions

We are looking example tests to demonstrate following capability and best practices

• One fail test case won't block other test cases unless there is issue
• test cases result doesn't depends on previous test cases execution states
  • be able to index its test data before execution
  • be able to navigate to orignal url/page as expected
• test cases could clean up post execution
  • be able to delete used test data after execution
• test cases could run on different os environment Mac/Linux