Comments (7)
I am using:
cypress: 12.16.0
@cypress/grep: 3.1.5
sorry-cypress: 2.5.1
cypress-cloud: 1.8.2
Existing in repo test case is very simple and does not describe real world usage:
- 5d7f36d#diff-6bd6fd4f71bae5e0b20dfcc2a86a659b57a657f546d2bdf033b7825d985f9376R1-R17
- 5d7f36d#diff-5ba646cdcf865c7d8146e78da174190a6cc056645ad7ee916c4df28844b4ed0aR1-R5.
There is only one spec file and it satisfies grepTags.
But if we have two spec files and one of them does not contain any case that satisfies grepTags then run will fail.
For example if we have following configuration where we have two files each with different tag:
// testTagA.spec.js
it('My test', { tags: ['@tagA'] } () => {
// ...
});
// testTagB.spec.js
it('My test', { tags: ['@tagB'] } () => {
// ...
});
then if we run it like this:
npx cypress-cloud --env grepTags=@tagB ...
Then result will be like this:
Running: cypress/e2e/testTagA.spec.js (1/2)
@cypress/grep: filtering using tag(s) "@tagB"
Can't run because no spec files were found.
We searched for specs matching this glob pattern:
> /home/.../projects/frontend/cypress/e2e/testTagA.spec.js
WARNING Cypress runner failed with message: "Could not find Cypress test run results"
WARNING The following spec files will be marked as failed:
- /home/.../projects/frontend/cypress/e2e/testTagA.spec.js
Reporting results and artifacts in background...
I think thats because cy2 implementation tried to run all specs in one cypress run
.
So when one of specs is filtered out by @cypress/grep there is still another one that will be run.
But cypress-cloud works differently.
It runs every spec file in separate cypress run
command.
Thus, there is only one spec file per run so when @cypress/grep filters out that single file, cypress fails with an error that there is no test cases that satisfy spec and tag filtering.
Cypress programmed to fail if it can't find any test case in entire run.
So if, instead of one big run with all specs, we create many runs with one spec, then all runs that don't match the grep pattern will fail.
I can't use @cypress/grep after I moved from cy2 to cypress-cloud.
This issue makes @cypress/grep incompatible with cypress-cloud at the moment which is very sad for me.
@cypress/grep is crucial for big projects with many tests.
I would be very happy if this issue will be resolved and @cypress/grep will become compatible again.
Thanks in advance for your time.
from cypress-cloud.
@Rikkun-1 thanks for reporting that issue - good catch, we'll issue a fix soon
from cypress-cloud.
I am using: cypress: 12.16.0 @cypress/grep: 3.1.5 sorry-cypress: 2.5.1 cypress-cloud: 1.8.2
Existing in repo test case is very simple and does not describe real world usage:
- 5d7f36d#diff-6bd6fd4f71bae5e0b20dfcc2a86a659b57a657f546d2bdf033b7825d985f9376R1-R17
- 5d7f36d#diff-5ba646cdcf865c7d8146e78da174190a6cc056645ad7ee916c4df28844b4ed0aR1-R5.
There is only one spec file and it satisfies grepTags. But if we have two spec files and one of them does not contain any case that satisfies grepTags then run will fail.
For example if we have following configuration where we have two files each with different tag:
// testTagA.spec.js it('My test', { tags: ['@tagA'] } () => { // ... }); // testTagB.spec.js it('My test', { tags: ['@tagB'] } () => { // ... });then if we run it like this:
npx cypress-cloud --env grepTags=@tagB ...
Then result will be like this:
Running: cypress/e2e/testTagA.spec.js (1/2) @cypress/grep: filtering using tag(s) "@tagB" Can't run because no spec files were found. We searched for specs matching this glob pattern: > /home/.../projects/frontend/cypress/e2e/testTagA.spec.js WARNING Cypress runner failed with message: "Could not find Cypress test run results" WARNING The following spec files will be marked as failed: - /home/.../projects/frontend/cypress/e2e/testTagA.spec.js Reporting results and artifacts in background...
I think thats because cy2 implementation tried to run all specs in one
cypress run
. So when one of specs is filtered out by @cypress/grep there is still another one that will be run.But cypress-cloud works differently. It runs every spec file in separate
cypress run
command. Thus, there is only one spec file per run so when @cypress/grep filters out that single file, cypress fails with an error that there is no test cases that satisfy spec and tag filtering. Cypress programmed to fail if it can't find any test case in entire run. So if, instead of one big run with all specs, we create many runs with one spec, then all runs that don't match the grep pattern will fail.I can't use @cypress/grep after I moved from cy2 to cypress-cloud. This issue makes @cypress/grep incompatible with cypress-cloud at the moment which is very sad for me. @cypress/grep is crucial for big projects with many tests.
I would be very happy if this issue will be resolved and @cypress/grep will become compatible again. Thanks in advance for your time.
I want to add a very important note to this comment.
I was wrong, the @cypress/grep plugin is not completely incompatible with cypress-cloud.
I forgot that I added the grepOmitFiltered option to my config file a long time ago.
Adding this will not only skip tests that don't match the tags, but it will remove those tests from the run entirely, resulting in the error described above.
@cypress/grep plugin works fine if you don't use grepFilterSpecs, grepOmitFiltered options.
Tests that don't match the grep pattern will simply be marked as skipped, and the runner will be fine even if all tests in the file are skipped.
P.S.
However, grepOmitFiltered is a very important option.
With it, those files that do not fit at all are completely discarded.
Without it, fetching a small set of tests with the filter is too slow, because instead of discarding completely unsuitable files, it first launches the browser and then skips all the cases for each individual file that usually would be discarded with grepOmitFiltered.
from cypress-cloud.
That one was tricky!
First, the issue is only relevant when grepFilterSpecs
is activated, grepOmitFiltered
doesn't really matter.
@cypress/grep
mutates config
object - it modifies config.specPattern
to include only the specs that match the tags. The order of plugins in setupNodeEvents
matters, consider this:
async setupNodeEvents(on, config) {
await currents(on, config);
require("cypress-terminal-report/src/installLogsPrinter")(on);
return require("@cypress/grep/src/plugin")(config);
},
await currents(on, config);
runs first and uses the originalconfig.specPattern
- it will include all the files defined in the original cypress configuration@cypress/grep
runs afterwise, mutating the config, and excluding certain specscypress-cloud
will runcypress
for spec files that should have been filtered, cypress activates@cypress/grep
, excludes the intended file and fails because it has nothing to run
As a workaround, make sure to run require("@cypress/grep/src/plugin")(config);
before await currents(on, config);
, for example:
async setupNodeEvents(on, config) {
require("cypress-terminal-report/src/installLogsPrinter")(on);
require("@cypress/grep/src/plugin")(config);
return currents(on, config);
}
Not sure what to do with it because cypress-cloud
is compatible, it's more of a configuration issue and probably a note in the documentation would be sufficient. @Rikkun-1 please let me know if the workaround worked for you.
from cypress-cloud.
@Rikkun-1 awesome!
I updated to documentation with a reference to this issue: https://github.com/currents-dev/cypress-cloud/blob/main/.github/README.md#usage-with-cypressgrep
from cypress-cloud.
from cypress-cloud.
Sorry for the delay, @agoldis.
I was busy with work.
Your workaround works great.
I ran into another issue with @cypress/grep when tried to test it with grepFilterSpecs but it's easy to solve with this comment cypress-io/cypress#27216 (comment).
Thanks a lot for your workaround, it's really important for test suite management.
I think this issue can be closed.
from cypress-cloud.
Related Issues (20)
- Can't use --headless on CLI. HOT 1
- Using CJS syntax in EMS setup fails HOT 5
- When launching 2 cypress-cloud run command for a same pipeline by specifying a tag, the second tagged run is not run.
- Glob patterns not working for --spec parameter HOT 1
- Advice on how best to avoid clashes between `cloudPlugin` and our own setup? HOT 1
- Question: Why not seeing the current test on 'npx cypress-cloud run' command HOT 4
- Executing requirements before execution of all spec files HOT 1
- Mochawesome.json is always deleted for each spec file ran HOT 2
- CSR-674 exit with code 1 when no specs detected
- "--record" flag is always being set to false HOT 4
- [CSR-791] cypress-cloud is not compatible with cypress 13.x HOT 93
- Override specPattern with CLI Argument HOT 7
- Use correct word when using --help flag
- "TypeError: Expected a finite number" after finishing a run HOT 3
- Cypress 13, CypressError: `cy.task('currents:test:after:run')` failed HOT 4
- Rename project HOT 3
- Cypress binaries HOT 3
- Cypress binaries from currents isn't found anymore HOT 1
- Errors while running parallel tests with cypress-cloud HOT 4
- Upload docker images with old cypress binaries HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cypress-cloud.