Intermittent Cypress test failures about oscal-editor-deployment HOT 8 CLOSED

Okay so we're now in a slightly better place. We have merged a few PRs today and we haven't seen any test failures on main yet. I do not like our "fix" which is really more of a hotfix. Primarily we fixed a bug caused by a recent PR merged on oscal-react-library which did not have corresponding updates to the end-to-end tests (we need to retro that and see if we can catch similar things in the future). Additionally, we significantly increased the waits before timing out across the application; like in some cases up to a minute. This is not acceptable in the long-term but it gets us over this immediate hurdle.

Initially, things indicate that it is likely the backend that is operating slowly. We are seeing that even locally doing a curl http://localhost:8080/oscal/v1/catalogs takes ~15s due to the validation failures that we are seeing (super unfortunate) and other processing (TODO: make sure logging isn't in the "hot path" for actually sending the response). We have issues like EasyDynamics/oscal-rest#55 to start returning "partial" objects but we need to look into that more.

I do not want to close this yet; I would like to keep it open for a few more days. @tuckerzp please keep monitoring for failures on the workflow on the main branch and on PRs; keep a tally and report them separately on Monday so we can see if this is actually getting better.

In the meantime, we can resume merging PRs on this repository and elsewhere that we held until after the tests were reliably passing again.

from oscal-editor-deployment.

keep monitoring for failures on the workflow on the main branch and on PRs

Passing Workflows

This is an attempt to collect all passing workflow runs. I may have missed some.

Failures

fix(workflow): use ubuntu-latest in build/test/deploy workflow

from oscal-editor-deployment.

Status Update

This task is between 20% - 75% done. The percentage could change very drastically depending on input from team, if something pops up that I've been missing, etc. It is also a bit tricky to assign a value of completeness to looking into an issue.

Details

I have been a bit stuck trying to find where the issue lies. I am still looking for some strand to pull to begin fixing the problem. That could happen very quickly or may take a bit.

from oscal-editor-deployment.

These Broken Pipe errors are interesting -- and I am curious which side the failure is happening on. Is Cypress closing the connection quickly because the timeout was already hit and it's moving on? Or is some server-side timeout being hit and the pipe is being closed?

from oscal-editor-deployment.

3/10/23 Sprint Planning - Moved from Sprint 42 to Sprint 43

from oscal-editor-deployment.

Manually triggered https://github.com/EasyDynamics/oscal-editor-deployment/actions/runs/4419699144/jobs/7748456949 to pick up EasyDynamics/oscal-rest-service#178

from oscal-editor-deployment.

Okay I am now feeling a lot better about this. Here are some of the changes that have been made to support this:

1. Waiting between starting the container and running the tests - #228
2. Increasing the timeout to ~60s (I don't love this but it's helpful to make sure it's not actually errors in the tests, we need to set better performance goals and then can tighten this a bit) - #218
3. Using a better base image - #221
4. General workflow refactoring - #226
5. Preventing logging constraint validation errors from slowing down responses - EasyDynamics/oscal-rest-service#182

The next step we might be able to take is to temporarily disable constraint validation in the OSCAL REST Service until issues with the catalog validation are resolved.

from oscal-editor-deployment.

Okay and the fixes that were just merged in #232 should be the last thing needed for us to stop seeing intermittent failures in the tests. We haven't seen any on the main branch that I am aware of this week. I am going to close this and we can either reopen (or create a new issue) if we see failures again.

from oscal-editor-deployment.