to pass grep via env parameter

Hi @bahmutov thanks for this amazing plugin!

I set it up, tagged a few suites (describe blocks) and when running the tests in CI with --record --parallel none of the tests runs (I use the Cypress Dashboard). Everything seems to work fine when parallel mode is disabled.

describe('my suite', { tags: ['@smoke'] }, () => {
   it('not ran', () => {})
   it('not ran either', () => {})
})

I assume that individual tests rely on a shared stack that is in scope but since they are distributed across multiple machines the stack is empty. Likely the custom it function doesn't find any tag and therefore skips the tests.

If my assumption is correct then I'd be blocked by this issue but if you can think of a solution I'd be happy to contribute with a fix.

Is it somehow possible to get the context/describe configuration from within the custom it? I think in Cypress one can use Cypress.config but I am not sure if that can be used there.

After the last update of this plugin, if you have another parameter such as scrollBehavior in your test, you get the following error:

No overload matches this call.
  Overload 1 of 6, '(title: string, config: TestConfigOverrides, fn?: Func | undefined): Test', gave the following error.
    Argument of type '{ scrollBehavior: false; }' is not assignable to parameter of type 'TestConfigOverrides'.
      Property 'tags' is missing in type '{ scrollBehavior: false; }' but required in type 'TestConfigOverrides'.
  Overload 2 of 6, '(title: string, config: TestConfigOverrides, fn?: AsyncFunc | undefined): Test', gave the following error.
    Argument of type '{ scrollBehavior: false; }' is not assignable to parameter of type 'TestConfigOverrides'.
      Property 'tags' is missing in type '{ scrollBehavior: false; }' but required in type 'TestConfigOverrides'.ts(2769)

In order to fix it, you have to add an empty tag property to make it work.

it('should do... in your app', { tags: [], scrollBehavior: false }, () => {

@bahmutov please, could you have a look? Thanks in advance!

Not sure

so we can tag and grep them

THE ISSUE
Given the following spec file

describe('Main page', () => {
  it('do a flip', () => {...})

  describe('', () => {
    it('do a roll', () => {...})
  })
})

Launch
cypress run --env grep="Main page do a roll"

Expected result:
cypress runs "do a roll" test, "do a flip" is pending

Actual result:
both tests are pending

THE CULPRIT
When determining nameToGrep here suiteStack names need to be filtered after mapping.

    const nameToGrep = suiteStack
      .map((item) => item.name)
      .filter(name => !!name) // suggested fix
      .concat(name)
      .join(' ')

otherwise, empty string yields an additional space character to the name, which is kinda unexpected (resulting in double space)

WHY?

1. Currently, there are some workarounds involving describe block, e.g. adding two interceptors with different handlers for the same request (this is exactly how I encountered the issue)
2. I'm creating grep string automatically, so this additional space is really annoying

Hi Gleb!

I'm loving the grepTags option; this is a really great and something I've been looking for an elegant solution to for quite some time! 🎉

One issue I'm encountering when running filtered tests is what happens with before blocks. Specifically, if there is a spec file that has no tests to be run, the code in the before block is still run, meaning that test setup steps are executed even though no test will actually be run.

I know that using before instead of beforeEach would generally be considered an anti-pattern, but regrettably there are circumstances with testing our application that make using before necessary in some instances.

I'm curious to know your thoughts on whether or not this package an be extended so that it doesn't run code from before blocks when it isn't needed.

When using from the console Cypress.grep(..., null, 5) it executes 25 tests, seems there is double loop somewhere

This issue contains a list of Renovate updates and their statuses.

This repository currently has no open or pending branches.

• Check this box to trigger a request for Renovate to run again on this repository

Issue:
When using a configFile that contains the desired grepTags, and attempting to utilize it.only() in order to debug tests, if the test case has a matching grep it opens but is skipped with zero output.

Desired result:
it.only() with a matching grepTag should run the test case.

Information to recreate:
cypress version 7.30
cypress-grep version 2.5.2

cypressGrep.json (configFile with grepTags):

{
    "baseUrl": "https://github.com/cypress-io/cypress-grep",
    "env": {
        "grepTags": "foo"
    }
}

login.spec.js (test file with correct tag):

describe('Login Tests', () => {
    
    it.only('Open github',{ tags:['foo'] }, () => {
        cy.visit('/');
    });
});

Command:

npx cypress open --env configFile=cypressGrep

Result:

Cypress: version: 3.7.0
Cypress grep: 2.5.1
Node: 12.18.0
Run command:

./node_modules/.bin/cypress run --env grep=A --spec '**/**/test.spec.js'

my Test file:

describe('A', () => {
  it('B', { tags: '@test' }, () => {
    console.log('in B');
  });
  it('C', { tags: ['@test', '@smoke'] }, () => {
    console.log('in C');
  });
});

Error log:

 1) An uncaught error was detected outside of a test:
     Uncaught TypeError: fn.call is not a function

This error originated from your test code, not from Cypress.

When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.

Cypress could not associate this error to any specific test.

We dynamically generated a new test to display this failure.
      at context.describe.context.context (http://localhost:9090/__cypress/runner/cypress_runner.js:27045:10)

For large spec files, I find it easier to use it.only that to try and write a grep that will only match a specific test. However, it seems that burn can't be used unless a non-empty grep or grepTag is also provided.

Could we make burn work without grep? When using it.only, the burn would apply to that test only and the others would be skipped.

I have installed cypress-grep plugin as a dev dependency so that I can run a cucumber scenario a specific number of times to check its flakiness. These are the changes I have made:

1. yarn add -D cypress-grep which adds this dependency in package.json
2. Made this entry require('cypress-grep')() in cypress/support > index.js
3. Optionally, also made the following below entry here cypress/plugins > index.js

module.exports = (on, config) => {
  require('cypress-grep/src/plugin')(config)
}

4. Now I am trying to run a scenario (title contains "hello") inside a feature file multiple times to check its flakiness cypress/integration/folderA/sample.feature. For this, I am executing the following command:
   npx cypress run --spec "cypress/integration/folderA/sample.feature" --env grep=hello,burn=10

But every time the command runs, I get the following error:

The error was:

Error: Webpack Compilation Error
./cypress/integration/folderA/sample.feature 1:15
Module parse failed: Unexpected token (1:15)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> Feature: Feature file name
| 
|   Background: Sample background title

This occurred while Cypress was compiling and bundling your test code. This is usually caused by:

- A missing file or dependency
- A syntax error in the file or one of its dependencies

Fix the error in your code and re-run your tests.

I am not sure what this error is trying to tell me and how do I fix it.
P.S: I have even referred this but it only talks about spec (.js) files and not feature (.feature) files: https://glebbahmutov.com/blog/burning-tests/

Right now only works with it(name, callback)

Here is a reproducible program.

  it("Does not do much!", { tags: "@smoke" }, () => {
    expect(true).to.equal(true);
  });

  it("Does not do much either!, but shouldn't run", { tags: ["@smoke", "@unstable"] }, () => {
      expect(true).to.equal(true);
    }
  );
});

describe("Tests with before block", () => {
  it("Test within another describe block", { tags: "@smoke" }, () => {
    expect(true).to.equal(true);
  });

  it("Test within another describe, should not run", { tags: ["@smoke", "@unstable"] }, () => {
      expect(true).to.equal(true);
    }
  );
});

I would like to only run @smoke tags but not if the same test is tagged with @unstable.
I tried various combinations like grepTags=@smoke -@unstable or different combinations, couldn't get it work.
What is a right way to do this ?

One specific issue is that the before block runs for tests which are not supposed to run.

describe('My First Test', () => {

  it('Does not do much! @smoke', () => {
    expect(true).to.equal(true)
  })

  it("Does not do much either!, but shouldn't run", () => {
    expect(true).to.equal(true)
  })

})

describe('Tests with before block', () => {

  before('You know, setting up fixtures', () => {
    cy.visit('http://google.com')
    cy.log('setting up fixtures')
  })

  it('Tests with the before block', () => {
    expect(true).to.equal(true)
  })

  it("Tests with the before block, again", () => {
    expect(true).to.equal(true)
  })

})

Strangely the commands of the before block doesn't show up in the command log.
Here if I grep with @smoke the tests in second describe block won't run but the before block does. Is there any way to skip them as well ?

When trying to check types

If the user is already skipping a test ...

We're trying out this very neat library and stumbled upon an issue based on this snippet:

describe('test suite', { tags: ['@validation'] }, () => {
  it('test 1', () => {
    //do something in test 1
  });

  it('test 2', () => {
    //do something in test 2
  });
});

The moment we exclude multiple tests based on the @validation tag by using grepTags=-@validation, the Cypress run is never completed. The first test is still ignored (see screenshot) but after that, nothing happens anymore.

When having only one ignored test, this is not an issue and the Cypress run is correctly completed.

Like the other properties of the default TestConfigOverrides interface 'tags' property should be optional, too. To allow defining other configs without the need to define a tag.

See https://github.com/bahmutov/cypress-grep#devtools-console

We could expose it as a method on the global Cypress object. Something that would set the grep and re-run the tests

Cypress.grep('my test title')
Cypress.grepTags('tag1+tag2') // will be parsed

I have a spec file:

describe('Update ad', () => {

    before(() => {
        
    });

    beforeEach(() => {
        
    });

    it('A', { tags: ['@ad'] }, () => {
        
    });
    
    it('B', { tags: ['@core', '@ad', '@staging'] }, () => {
       
    });
});

I run it with this command:

$ cypress run --browser chrome --headless --env grepTags=@staging

The point is to run only the second test on the staging environment.

The problem is that before() block does not execute in this scenario.

When I add tag @staging to the first test as well:

describe('Update ad', () => {

    before(() => {
        
    });

    beforeEach(() => {
        
    });

    it('A', { tags: ['@ad', '@staging'] }, () => {
        
    });
    
    it('B', { tags: ['@core', '@ad', '@staging'] }, () => {
       
    });
});

the before() block gets executed.

When I reorder the test cases, so that B goes before A:

describe('Update ad', () => {

    before(() => {
        
    });

    beforeEach(() => {
        
    });

    it('B', { tags: ['@core', '@ad', '@staging'] }, () => {
       
    });

    it('A', { tags: ['@ad'] }, () => {
        
    });
});

the before() block also gets executed.

I find this a strange behaviour, I expect the before() block to get executed no matter how my tests are ordered; skipping some tests should not be a problem either.

please need selected tests for typescript

Hi there,

cypress-grep still does not work for me. It seems that "grep" part is ignored.
I'd like to ask you whether your solution is built up on macha "-- grep" pattern, and whether Jest might be the problem that 'grep' is ignored.

Here is what I'm trying to do:

command:

npx cypress run --env grepTags="@shallow" --spec cypress/specs/ui/<spec-name>

spec example:

describe('some description', () => {
  it('test name', {tags: '@shallow'}, () => {})
  it('test name 2' () => {})
  it('test name 3' () => {})
  it('test name 4' () => {})
})

It always runs all tests. My expectation would be that it will run only the first test "test name".

Thanks a lot for any helps.

It is confusing to use the same parameter --grep= to both search the test title and tags. I think a much cleaner use case would be:

• use grep to search the title, or -grep to invert the title search
• use grep-tags to only match the explicit test tags specified in the test config object. The tags can be inverted, space is the OR and + character is the AND

Imagine the tests like this:

it('helps', () => {...})

it('works', { tags: 'smoke' }, () => {...})

it('works more', { tags: ['smoke', 'slow'] }, () => {...})

Then we could filter tests like:

# run just the first test
$ npx cypress --env grep=hello
# run all tests without hello in their title
$ npx cypress --env grep=-hello
# run the test with title "works more"
$ npx cypress --env grep="works more"

# run all tests with tag "smoke"
$ npx cypress --env grep-tags=smoke
# run all tests with tag "smoke" or "slow"
$ npx cypress --env grep-tags="smoke slow"
# run all tests with tag "smoke" AND "slow"
$ npx cypress --env grep-tags=smoke+slow
$ npx cypress --env grep-tags="smoke +slow"
# run all tests with tag "smoke" but without tag "slow"
$ npx cypress --env grep-tags=smoke-slow
$ npx cypress --env grep-tags="smoke -slow"

Cypress mentions:
specify() is identical to it(), so choose whatever terminology works best for you

However, when not listing a grepTag on the specify test block, the test is not skipped.
Whereas, the it test block is skipped.

probably should error ...

We have setup test pipeline in a way that, different jobs run different tags, but if a developer/qa forgets to tag a test, it will fall through cracks. We want to run another job at the end which will run all tests that are not tagged.

https://github.com/bahmutov/cypress-each

Need to copy it.each and describe.each methods if present. If we register cypress-grep after cypress-each plugin, these methods are lost

bahmutov/cypress-each#12

Right now we only support the describe

By reading the docs, it's not clear that it tries to run all spec files, even if some of them do not match the grepped word.

I have experimented with the Cypress examples, those created automatically when opening Cypress for the first time.
I ran npx cypress run --env grep=blur, and although there’s only one match for blur (on the cypress/integration/examples/actions.spec.js file, which successfully executed), all other specs were “run” as well (as pending).

Maybe that's the expected behavior, but better make it clear on the docs.

By the way, that's a great lib, thanks!

Cc @bahmutov.

As cypress has support Mocha bdd syntax would be interesting to add it in this lib.
https://docs.cypress.io/guides/references/bundled-tools#Mocha

Not sure when this stopped working but I've created a replica of my repo here where you can check the implementation.

Typescript users don't wanna be adding // @ts-ignore or triple-slash reference imports everywhere.
If there was a better integration / instruction for TS users, the adoption of this useful tool would skyrocket, I reckon.
Thanks for your work!

Hi Team,

I installed cypress-grep in my project and I am getting the said error.

Steps that I followed:

1. ./pnpm i -D cypress-grep
2. I have this line at line 1 require('cypress-grep')(); of file /support/index.js
3. I have this line in /plugins/index.js
module.exports = (on, config) => { require('cypress-grep/src/plugin')(config)
4. I am using only 1 spec file out of 10 that I have. In that 1 spec file, I am tagging only 2 tests out of 23 other tests.
   it('C5769093 - User should be able to turn on/off sharing for the action plan', { tags: '@smokeTest' }, function () {
it('C5769092 - User should be able to open "Share Action Plan" drawer via "Share" button on the builder view', { tags: '@smokeTest' }, function () {
5. scripts in package.json "e2e.run.smokeTest": "cypress run --headless --browser chrome --env grepTags=@smokeTest",
6. from command line I am running this command ./pnpm run e2e.run.smokeTest

It's a big project with lot of dependencies and devDependencies. Most related to cypress, I have the following installed
"cypress": "8.5.0", "cypress-grep": "^2.5.3", "cypress-multi-reporters": "1.1.22", "cypress-pipe": "^2.0.0", "cypress-wait-until": "^1.7.1",

for example -@tag1 to skip all tests with @tag1 in their title

For example to run tests with tag 1 OR tag 2, perhaps something like

grep=tag1,tag2 # tag1 OR tag2
grep=tag1+tag2 # tag1 AND tag2

Do not consider more complex grammars like tag1,tag2+tag3, too much

Right now having grep + it.only crashes the test runner

Probably comma-separated test titles

1. "test:cypress-ui-regression": "ng run convosight-web-client:cypress-run -- --record --key random --browser chrome --parallel --ci-build-id $BITBUCKET_BUILD_NUMBER --group ui-tests --spec 'cypress/tests/ui/**/*'",
2. "test:cypress-ui-smoke": "ng run convosight-web-client:cypress-run -- --record --key Random-env grepTags=@pullRequest --browser chrome --parallel --ci-build-id $BITBUCKET_BUILD_

For both the command it's filtering the tests based on tag @pullrequest but we wanted to filter it only for the second npm command and not for the first. When we run the first command it should run all the test cases even with tag @pullrequest.

FYI, We have kept grepTags in cypress.json under env.

Issue:
When using a multiple configuration file with configFile and grepTags it doesn't work properly filtering the tag and applying ENV.

Desired result:
Execute tests with the env variable and filter scenarios with tags at same execution

Information to recreate:
cypress version 7.30
cypress-grep version 2.5.2

Need a project multiple configuration files to use --env configFile=ENV
https://docs.cypress.io/api/plugins/configuration-api#Switch-between-multiple-configuration-files

I have uploaded a project to reproduce the issue:
https://github.com/BrunoNelli/grep_issue

Maybe a TodoMVC?

https://github.com/bahmutov/cypress-grep/blob/935230324806047b49348e1c447ac4250303048a/cypress/integration/describe-tags-spec.js#L21

Should this comment be --env instead of --eng?

To pre-filter specs that only have tagged tests

We need a better structure of test tags - where we find all suites and tests with their tags

Blocked by bahmutov/find-test-names#5

for better testing against any edge cases

For me it executed all the specs under integration rather the tag given. I followed below steps. Could you please let me know where I have done wrong?

1. Install grep

2. Package.json
   "smoke:qa": "cypress run --env grepTags="Smoke" --browser chrome",

3. index.js
   require('cypress-grep')

4. Added the Smoke tag for a spec
   describe('ID - 343 : Verify Navigate to "Assisted Service" Page, Heading & Grid Fields',{tags:'Smoke'}, function () {

5. Execute the below command;
   npm run smoke:qa

This is a great plugin! I loved mocha's test tagging and glad it has a complementary plugin for cypress 🙏

What I find a bit confusing though is this case:

describe('Screen A', { tags: ['@smoke', '@screen-a'] }, () => {
  it('should navigate to Screen B successfully', { tags: ['@screen-b'] }, () => {
    // do something that eventually sends the page to screen b.
  }); 
}); 

if I run cypress run --env grepTags="@smoke+@screen-b" I'd expect it to run the test, because I assume that describe adds some context to the test, and tags would have been included in this context (similar to how beforeEach of an ancestor describe is run for all of its descendant its ).

However, this didn't work, and looking a bit at the code it looks like tags are calculated separately for describe and it, and only if the describe contains tags matching the current grep it'll try running all of the its inside it.

So, this works:

describe('Screen A', { tags: ['@smoke', '@screen-a'] }, () => {
  it('should navigate to Screen B successfully', { tags: ['@smoke', '@screen-b'] }, () => {
    // do something that eventually sends the page to screen b.
  }); 
}); 

However, it feels odd to me... why would I need to specify @smoke twice? I know it makes things a bit more explict, and explicitness is good, but this is a contrived example. What if I have multiple describes (2-3-4 aren't that far-fetched), and some of them contain tags, I'd then have to specify all the tags, all the way down, everywhere I have at least one additional tag. This makes things harder to maintain if I'm having to always have to keep them in sync.

Changing to this behavior would mean that describes are no longer "skipped" in advance, instead, they're run as is, and their tags would be collected along the way till we get to the it and then calculate if that it should run according to all of the tags.

If this change is desired then I'd be happy to implement it :)