avivbenchorin / monterey Goto Github PK
View Code? Open in Web Editor NEWFunctional testing framework to automate OpenSearch-Dashboards
License: MIT License
Functional testing framework to automate OpenSearch-Dashboards
License: MIT License
Is your feature request related to a problem? Please describe.
Monterey currently does not have any method of loading indices and data from a file to OpenSearch, which limits the ability for functional test developers to setup custom test environments. It also lacks a method of unloading data from OpenSearch, making it difficult to tear down the test environment and return OpenSearch to a clean state. The lack of these two features increases the difficulty of porting over existing OpenSearch Dashboards tests that require specific test environments and constrains the testing capabilities of the Monterey framework.
Describe the solution you'd like
Monterey should be able to parse the contents of a file containing JSONs that represent indices and data elements, and load and unload this parsed data through the use of the OpenSearch REST API. Ideally, this should be implemented as Cypress custom commands using the Cypress readFile
and request
functions to smoothly integrate the loading and unloading functionality into the existing Monterey framework.
Describe alternatives you've considered
One alternative is to directly use the osd-opensearch-archiver
package, which is the existing data file loading and unloading implementation in the OpenSearch Dashboards testing framework. While utilizing this package would have the advantage of providing a solution that is already tailored to Monterey's specific needs, osd-opensearch-archiver
has dependencies on another OpenSearch Dashboards package as well as on an Elasticsearch package, which in turn have dependencies on other packages. Handling the dependencies for these packages would increase the overhead for maintaining the Monterey framework, as well as generally increase the complexity of the project; thus, I find the osd-opensearch-archiver
package approach to be non-ideal if there exists a simpler solution, especially for the initial proof of concept stages of Monterey.
Additional context
N/A
When searching for a specific option in a EuiComboBox field and clicking on the desired option, the element sometimes enters an invalid
state rather than select the option. The invalid
state is visually indicated through a red line along the bottom of the input field.
This bug happens inconsistently, and usually results in the test case failing due to preventing Cypress from performing subsequent interactions with UI elements (e.g. Cypress being unable to press the submit button since the option was not properly selected). It is likely caused by a race condition between the combo box re-rendering after the selection of the option and the combo box checking whether it is in the invalid
state.
This bug is difficult to reproduce at-will due to it occurring inconsistently. The current best method would be:
.only()
to the 'Explore the data in the dashboard' test suite in the demo_spec.js
file (e.g. it.only('Explore the data in the dashboard',...
)npx cypress open
) and run the demo_spec.js
testGnomehouse
in the Manufacturer field or when selecting day_of_week
in the Add filter
field inputCypress should consistently be able to type into the input field of the combo box, find the desired option, and click on the option to select it without the combo box entering an invalid state.
1.0.0-SNAPSHOT
1.0.0
The EuiComboBox is a option list UI element that allows users to search for options and select multiple options if wanted. The OpenSearch Dashboards interface uses combo boxes to handle functionality that potentially requires searching a large number of options, such as in the Add filter
panel.
When developing OpenSearch Dashboards functional tests in Cypress, the standard approach to select an option in a combo box has been the following series of actions:
add Filter
button)Field
input and type day_of_week
)day_of_week
)day_of_week
option)The bug of entering the invalid
state has been occurring in step 4, after clicking the button. The EuiComboBox source code contains the following comments and code about the invalid state:
// Visually indicate the combobox is in an invalid state if it has lost focus but there is text entered in the input.
// When custom options are disabled and the user leaves the combo box after entering text that does not match any
// options, this tells the user that they've entered invalid input.
const markAsInvalid =
isInvalid ||
((hasFocus === false || isListOpen === false) && searchValue);
The above indicates that the invalid
state is being caused by the list being closed or losing focus while there is still a searchValue
in the input. This suggests that in the context of the bug, there is likely a race condition created when the option is clicked and the option list closes:
searchValue
and preventing the invalid
state from being true.invalid
state first and finds that the list is closed and the searchValue
is filled, concluding that the invalid
state is true.Assuming that this evaluation of the bug's cause is correct, the fix will likely involve addressing this race condition and ensuring that the option selection completes before checking the invalid state.
As a feasibility POC, this issue is to create a test plugin to consume the monterey library. Since currently there is no page module library in monterey yet, we will first need to add a simple resuable function in monterey and then call it inside a test plugin.
Build a set of Cypress-based test fixture handling functions to allow for importing and deleting OpenSearch indices and their corresponding test data.
//Sample Cypress spec file importing and removing test fixture data
describe('Sample test suite', () => {
// At the start of the test suite
before(() => {
// remove the indices from OpenSearch in case they already exist
cy.clearJSONMapping('path/to/fixtures/mappings.json.txt')
// import the indices in the given file
cy.importJSONMapping('path/to/fixtures/mappings.json.txt')
// import the documents that will populate the newly added indices
cy.importJSONDoc('path/to/fixtures/data.json.txt')
})
// At the end of the test suite
after(() => {
// delete the indices and documents to clean up the OpenSearch state
cy.clearJSONMapping('path/to/fixtures/mappings.json.txt')
})
it('Test using the imported indices', () => {
// perform the rest of the test cases that will use the imported data
...
})
})
Many functional tests in the existing OpenSearch Dashboards framework expect OpenSearch to contain specific indices and documents, which are pre-packaged in test fixture files and imported using the OpenSearch REST API. These test fixture files contain the required OpenSearch state for a test category (e.g. for all of the dashboard tests) or for a specific test file (e.g. for the _date_nanos.js
test), with the indices and documents being stored in separate files as JSONs. The test files utilize the osd-opensearch-archiver
package to handle the test fixtures, which has the functionality to load (import) and unload (remove) the data contained in the fixture files.
In order to minimize the overhead of migrating functional tests from the current framework, Monterey should have the same functionality to handle the existing test fixtures. Without using the fixture files, developers would either need to hard-code OpenSearch REST API calls into test files to import the desired indices or redesign the test cases to work with test data already available in the OpenSearch instance. The test fixture handling implementation in Monterey should be able to load and unload fixture files using the OpenSearch REST API, and be callable or runnable from test files to set up and tear down the OpenSearch test environment as needed.
I decided to implement the fixture handling functionality using Cypress custom commands to make the loading and unloading functions easily accessible for test files to directly call. The proposed set of three custom commands will be able to import a fixture file containing indices (cy.importJSONMapping
), import a fixture file containing documents (cy.importJSONDoc
), and remove the indices contained within a fixture file (cy.clearJSONMapping
).
In the existing Selenium-based framework, the two types of fixtures files are mappings.json
files that contain all of the indices and their corresponding mappings and settings, and data.json.gz
files that contain the documents. In both files, each index or document JSON is separated by two newline characters (\n\n
).
For Monterey’s implementation of the fixture handling functionality, these files have been changed to mappings.json.txt
and data.json.txt
respectively. data.json.gz
had to be unzipped since the cy.readFile
command does not support reading gzip’d files. As well, both files have end with a .txt
extension to prevent cy.readFile
from automatically parsing the .json
files as a single JSON object and failing.
mappings.json.txt
The index JSONs inside of mappings.json.txt
can have the following structure:
{
"type": "index",
"value": {
"aliases": {
...
},
"mappings": {
...
},
"settings": {
...
}
}
}
The allowable contents of the aliases, mappings, and settings fields are detailed in the OpenSearch create index API documentation.
data.json.txt
The document JSONs inside of data.json.txt
can have the following structure:
{
"type": "doc",
"value": {
"id": ...,
"index": ...,
"source": {
...
}
}
The allowable contents of the id
and index
fields are detailed in the OpenSearch bulk API documentation. The source field should contain the document data to be indexed.
All of the added Cypress custom commands perform three general operations needed to handle a test fixture file:
The code below is a sample custom command that demonstrates the general flow of work, and will be referenced in the following subsections.
Cypress.Commands.add('SampleFixtureHandler', (filepath, ...) => {
// 1) read the test fixture file in as a string
// and split the string into separate JSONs
cy.readFile(filepath, 'utf8').then((str) => {
const JSONStrArray = str.split('\n\n')
// 2) extract and format the contents of each JSON
JSONStrArray.forEach((element) => {
const JSONContent = sampleJSONProcesser(JSON.parse(element))
// 3) send the OpenSearch REST API request
cy.request(...).then((response) => {
})
})
})
})
The first step when handling test fixture files is to read and process them. The functions use the Cypress API command cy.readFile
to read the fixture file at a specific path and return its contents as a string. Afterwards, the string gets split by the chosen delimiter between the JSONs inside the file, two newlines characters (\n\n
). The returned array of JSON strings is then processed in future steps.
Once the function has the parsed array of JSON strings, the second step is to iterate through each JSON and extract the relevant information. After the JSON has been turned into a JSON object with JSON.parse
, the fields needed for the import or remove operation are reformatted into a request body and/or header which is the example above would be done in the dummy function sampleJSONProcessor
.
The final step is to send the formatted JSON index or document information to OpenSearch with a network request using the cy.request
command. The request’s method, URL, header, and body are set based on the type of operation being completed and the extracted JSON data itself. The default base URL to send requests to is localhost:9200
, the default host and port of OpenSearch.
cy.importJSONMapping
Cypress.Commands.add('importJSONMapping', (filepath, hostname = 'localhost', port = '9200') => {
// helper function for parsing index JSONs
const parseIndex = ({ index, settings, mappings, aliases }) => {
// organize the "settings", "mappings", and "aliases" fields inside a "body" field
return { index, body: { settings, mappings, aliases } }
}
cy.readFile(filepath, 'utf8').then((str) => {
const strSplit = str.split('\n\n')
strSplit.forEach((element) => {
const json = JSON.parse(element)
// only send the OpenSearch create index API request
// if the JSON represents an index
if (json.type === 'index') {
const indexContent = parseIndex(json.value)
// send the OpenSearch create index API request
// to create an index with the same name as the "index" field of the JSON
// using the specifications of the formatted "body" field
cy.request({ method: 'PUT', url: `${hostname}:${port}/${indexContent.index}`, body: indexContent.body, failOnStatusCode: false }).then((response) => {
})
}
})
})
})
The cy.importJSONMapping
function (as defined above) reads in a mappings.json.txt
file, and for each index JSON parses its information and sends an OpenSearch create index API request. The request is sent to the specified host name and port, and should not fail the test case if the response’s status code is not 200 (such as when the new index already exists in OpenSearch).
cy.clearJSONMapping
Cypress.Commands.add('clearJSONMapping', (filename, hostname = 'localhost', port = '9200') => {
cy.readFile(filename, 'utf8').then((str) => {
const strSplit = str.split('\n\n')
strSplit.forEach((element) => {
const json = JSON.parse(element)
// only send the OpenSearch delete index API request
// if the JSON represents an index
if (json.type === 'index') {
const index = json.value.index
// send the OpenSearch delete index API request
// to delete the index with the same name as the "index" field of the JSON
cy.request({ method: 'DELETE', url: `${hostname}:${port}/${index}`, failOnStatusCode: false }).then((response) => {
})
}
})
})
})
The cy.clearJSONMapping
function (as defined above) reads in a mappings.json.txt
file, and for each index JSON parses its information and sends an OpenSearch delete index API request. The request is sent to the specified host name and port, and should not fail the test case if the response’s status code is not 200 (such as when the index to be deleted does not already exist in OpenSearch).
Since the delete index API only requires the index’s name to delete it, the settings
, mappings
, and aliases
fields do not need to be parsed. As well, deleting the index consequently deletes all of the documents stored inside it, so there is no current need for a function that bulk removes documents when cleaning up the state of OpenSearch.
cy.importJSONDoc
Cypress.Commands.add('importJSONDoc', (filename, hostname = 'localhost', port = '9200', bulkMax = 1600) => {
// local helper function for parsing document JSONs
const parseDocument = ({ value: { id, index, source } }) => {
// format the action (id and index) and source into two NDJSONs and combine them
// following the OpenSearch bulk API format
const actionData = { index: { _id: id, _index: index } }
const oneLineAction = JSON.stringify(actionData).replace('\n', '')
const oneLineSource = JSON.stringify(source).replace('\n', '')
const bulkOperation = `${oneLineAction}\n${oneLineSource}`
return bulkOperation
}
// local helper function for sending a bulk API network request,
// joining all of the combined action and source NDJSON strings
// into one large string for the request body.
const sendBulkAPIRequest = (bodyArray, hostname, port) => {
cy.request({ headers: { 'Content-Type': 'application/json' }, method: 'POST', url: `${hostname}:${port}/_bulk`, body: `${bodyArray.join('\n')}\n`, timeout: 30000 }).then((response) => {
})
}
cy.readFile(filename, 'utf8').then((str) => {
// keep track of how many JSONs have been processed
let readJSONCount = 0
// create a nested array to store combined action and source NDJSON strings
// before they get sent in a request
const bulkLines = [[]]
str.split('\n\n').forEach((element) => {
// store the processed combined action and source NDJSON string in the nested array
bulkLines[0].push(parseDocument(JSON.parse(element)))
// update the read counter
readJSONCount++
// if the number of combined action and source NDJSON strings stored
// in the nested array is equal to the set threshold
if (readJSONCount % bulkMax === 0) {
// Pop the nested array and use it as the body of a bulk API request
sendBulkAPIRequest(bulkLines.pop(), hostname, port)
// add a new nested array to store future NDJSONs
bulkLines.push([])
}
})
// if there still exists NDJSONs to be sent to OpenSearch
if (bulkLines.length > 0) {
// Pop the nested array and use the remaining NDJSONs in a bulk API request
sendBulkAPIRequest(bulkLines.pop(), hostname, port)
}
// refresh all indices to ensure that the imported document data is accessible
cy.request({ method: 'POST', url: `${hostname}:${port}/_all/_refresh` }).then((response) => {
})
})
})
The cy.importJSONDoc
function (as defined above) reads in a data.json.txt
file, and for each document JSON parses and stores its information as a newline-delimited JSON (NDJSON) string. After the number of locally stored NDJSONs exceeds a threshold (by default 1600), the documents are compiled into one NDJSON and sent in an OpenSearch bulk API request. After all of the document JSONs have been read and imported, a final OpenSearch refresh API request is made to ensure that the imported document data is available for searching.
The NDJSON strings representing each document JSON are stored and sent in batches since bulk requests are more efficient than making index requests for individual documents (read more here). The default batch size of 1600 was selected due to being the largest batch size that did not start to significantly impact the bulk import performance on my test environment, and different batch sizes may be preferrable based on the environment the tests and OpenSearch are running in.
The method of using nested arrays to store the NDJSON strings and popping them when their size meets the batch size threshold is intended to reduce the amount of strings stored in local memory at a time, which is relevant to a scalability issue that is detailed in the “How this scales” section further below.
The proper time to use the test fixture loading and unloading commands is at the start and end of test files or groups of test files. For example, the code below is from the cypress/integration/dashboard/dashboard_filtering_spec.js
test file, and uses the commands to import the needed indices and documents before any tests run and remove the added indices after all tests finished.
// cypress/integration/dashboard/dashboard_filtering_spec.js
describe('dashboard filtering', () => {
// before running any of the tests in the file
before(() => {
// remove the indices from OpenSearch in case they already exist
cy.clearJSONMapping('cypress/fixtures/dashboard/data/mappings.json.txt')
// import the indices needed for the file's test cases
cy.importJSONMapping('cypress/fixtures/dashboard/opensearch_dashboards/mappings.json.txt')
cy.importJSONMapping('cypress/fixtures/dashboard/data/mappings.json.txt')
// import the documents that will populate the newly added indices
cy.importJSONDoc('cypress/fixtures/dashboard/opensearch_dashboards/data.json.txt')
cy.importJSONDoc('cypress/fixtures/dashboard/data/data.json.txt')
})
// at the end of the test file
after(() => {
// delete the indices and their data to clean up state of OpenSearch
// NOTE: the opensearch_dashboards/mappings.json.txt is not being cleared
// since it contains the specifications for the ".kibana_1" index,
// which if deleted using the current fixture handling functionality cannot
// properly be re-imported
cy.clearJSONMapping('cypress/fixtures/dashboard/data/mappings.json.txt')
})
it('Buffer test for importing test environment data', () => {
})
describe('Adding and removing filters from a dashboard', () => {
// perform the rest of the test cases that will use the imported test data
...
The cy.importJSONDoc
command should only be called with a specific data.json.txt
file after cy.importJSONMapping
had been called with the corresponding mappings.json.txt
file to ensure that the imported documents have indices to be indexed into.
As well, the cy.clearJSONMapping
command should not be called on any mappings.json.txt
files that contains a .kibana_*
index with *
representing any integer. The current functionality is unable to properly re-import any deleted .kibana_*
indices and future iterations will have to address this issue.
The Buffer test for import test environment data
test case above exists to prevent the high workload being performed in the before()
block from slowing down future tests. The before()
block exists in the context of the first test case of the file, and I sometimes experienced that first test case slowing down dramatically after the completion of the before().
My solution was to have an empty test case as a buffer to prevent the before()
block from impacting the performance of the test file.
One drawback of this design is that implementing the test fixture loading and unloading functionality as Cypress custom commands couples it to the Cypress framework, as does using Cypress API commands such as cy.readFile
and cy.request.
This means that this implementation would not be supported for use with any other test driver (such as Selenium) or context besides the Cypress framework.
Another downside of this design is that it does not yet contain all of the test fixture handling capabilities that osd-opensearch-archiver
does, only the core functionality needed to successful import and remove the test fixture files. Future iterations of the test fixturing handling functionality will have to continue to expand their functionality, such as being able to save new fixture files, edit existing fixture files, and properly clean up the .kibana
indices in OpenSearch.
A third downside is the design’s scalability issues on the size of the test fixture file, which would require putting limits on the size of test fixture files. This issue is further discussed in the “How this scales” section below.
The main alternative design is to try and adapt the original osd-opensearch-archiver
package to be usable in Cypress. The main advantage of using osd-opensearch-archiver
directly is that it would reduce the amount of research time spent to understand the package’s inner mechanisms in order to rebuild the functionality from scratch, as has been done for the Cypress custom commands. As well, the package was already built for the Selenium-based testing framework, and having a general purpose fixture handling library that could run on both Cypress and Selenium would improve the portability of the Monterey platform.
A major blocker for using the osd-opensearch-archiver
package, however, is Cypress’ lack of native support for async/await
calls in its code. osd-opensearch-archiver
heavily utilizes async/await
and thus would require developers to find a potential workaround for using async/await
in between Cypress commands, or call osd-opensearch-archiver
through a Node script using the cy.exec
command and lose access to log messages and other debugging capabilities.
Another impediment of utilizing osd-opensearch-archiver
is its dependency on local packages in the OpenSearch Dashboards repository and on the Elasticsearch library (as can be seen in the package’s package.json file). The only local package osd-opensearch-archiver
is directly dependent on is @osd/dev-utils
; however @osd/dev-utils
is dependent on @osd/utils
, which in turn is dependent on @osd/config-schema
. Using osd-opensearch-archiver
will either require refactoring its code to remove all usages of @osd/dev-utils
and replace them with internal functionality, or to keep all of these local packages in the Monterey framework, adding additional bloat to the code base. The dependency on the Elasticsearch library should also be considered for removal to keep Monterey framework decoupled from any Elasticsearch or Kibana implementations, which would require replacing any usage of the Elasticsearch Node.js client with local functionality (or the OpenSearch Node.js client when it is developed).
Since the Monterey framework is still in its early stages of development and does not yet have many test files prepared, it should be easy to adopt this test fixture handling functionality. Any new Cypress functional tests that are adapted from the existing Selenium-based framework can incorporate the Cypress custom commands to replace code in the original test file that used osd-opensearch-archiver
to load and unload test fixture files.
For new Monterey-specific test files, developers can decide whether to use test fixtures or a different approach to preparing the state of OpenSearch, and in the later case, the use of the fixture handling commands would be unnecessary.
The current implementation of the test fixture handling custom commands has a scalability issue due to the use of cy.readFile
. cy.readFile
reads in the entire test fixture file at once synchronously rather than as an asynchronous data stream. This works fine for relatively small files but can cause issues once files get large enough due to the amount of local memory being used to hold the file’s contents as a string.
An example of this is while running the dashboard_filtering_spec.js
Cypress test file using the Electron browser. When trying to import the data.json.txt
fixture file in the cypress/fixtures/dashboard/data
directory of Monterey (which is ~66.4MB large), the test hangs on the execution of cy.readFile
for an extended period of time, after which it timeouts on the first attempted bulk API request to OpenSearch.
Table 1 below shows the results of experimenting with different fixture files sizes when running the loading and unloading sections of the dashboard_filtering_spec.js
test file in different browsers and running modes. The tests were run in Cypress 7.6.0 locally (MacBook Pro 2019, Big Sur 11.4), and each browser/file size combination was ran three times and averaged. The elements highlighted in red are the largest fixture file sizes that the specific browser and running mode ran three times without failing.
Table 1: data.json.txt
Fixture File Size vs Runtime of Successful Fixture Loading and Unloading
|-------------- | Test Runner (seconds, including "after all" block) | ----| | |---------------- | Headless (seconds, excluding "after all" block) | ----------------| | |
---|---|---|---|---|---|---|
Electron (89) | Chrome (92) | Firefox (78) | Electron (89) | Chrome (92) | Firefox (78) | |
1MB | 15.58 | 16.03 | 17.99 | 15.11 | 15.57 | 14.76 |
10MB | 25.85 | 16.83 | 18.69 | 24.01 | 17.63 | 16.39 |
20MB | 39.18 | 17.58 | 20.87 | 35.81 | 22.63 | 19.38 |
30MB | 46.86 | N/A | N/A | 49.01 | N/A | N/A |
35MB | 54.13 | N/A | N/A | failed 1st time failed 2nd time failed 3rd time |
N/A | N/A |
40MB | 63.95 1st time failed 2nd time failed 3rd time |
28.96 | 30.20 | failed 1st time failed 2nd time failed 3rd time |
30.22 | 25.85 |
60MB | N/A | 42.43 | 35.52 | N/A | 36.52 | 32.54 |
80MB | N/A | 43.74 | 39.36 | N/A | 43.11 | 40.07 |
100MB | N/A | failed 1st time 40.50 2nd time 40.12 3rd time |
44.48 | N/A | failed 1st time failed 2nd time failed 3rd time |
45.96 |
200MB | N/A | 68.24 | 72.78 | N/A | N/A | 75.66 |
From the table above, running cy.importJSONDoc
on the Electron and Chrome browsers had upper limits on the size of data.json.txt
at which point the began to command failed. More experimentation will need to be done running the fixture handling custom commands on CI environments with more powerful hardware specs to determine if the issue is with the browsers themselves running out of memory, or with running the tests locally.
The osd-opensearch-archiver
implementation of this functionality does not have this issue since it reads the fixture files as a stream, reading the file chunk by chunk until it reaches the separator between JSONs, at which point it parses the JSON. This approach avoids reading in the entire file at once and needing to store its entirety in memory.
The main implications of this scalability issue are that either the test fixture file size will have to be capped to have this fixture handling functionality be supported with all three browser types (requiring the dashboard_filtering_spec.js
test file to be modified to use a smaller fixture file), or that the Electron browser will not be supported in the Monterey project when running tests that utilize test fixture files. A long term development goal should be to fix this scalability issue, most likely by replacing the cy.readFile
API call with JavaScript read
and write
streams to process the test fixture file in chunks.
Since this test fixture handling functionality already existed in OpenSearch Dashboards test files, the main way to teach it would be through documentation detailing the difference between the new Cypress custom commands and osd-opensearch-archiver,
as well as sample code demonstrating the use cases of the functions.
We are looking example tests to demonstrate following capability and best practices
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.