Code Monkey home page Code Monkey logo

gemini's Introduction

Gemini

npm Build Status Coverage Status Join the chat at https://gitter.im/gemini-testing/gemini Stories on waffle.io

Gemini is a utility for regression testing the visual appearance of web pages.

Gemini allows you to:

  • Work with different browsers:

    • Google Chrome (tested in latest version)
    • Mozilla Firefox (tested in latest version)
    • IE8+
    • Opera 12+
  • Test separate sections of a web page

  • Include the box-shadow and outline properties when calculating element position and size

  • Ignore some special case differences between images (rendering artifacts, text caret, etc.)

  • Gather CSS test coverage statistics

Gemini was created at Yandex and is especially useful to UI library developers.

Quick start

Installing

npm install -g gemini
npm install -g selenium-standalone
selenium-standalone install

Configuring

Put the .gemini.js file in the root of your project:

module.exports = {
    rootUrl: 'http://yandex.ru',
    gridUrl: 'http://127.0.0.1:4444/wd/hub',

    browsers: {
        chrome: {
            desiredCapabilities: {
                browserName: 'chrome'
            }
        }
    }
};

Writing tests

Write a test and put it in the gemini folder in the root of your project:

gemini.suite('yandex-search', (suite) => {
    suite.setUrl('/')
        .setCaptureElements('.home-logo')
        .capture('plain');
});

Saving reference images

You have written a new test and should save a reference image for it:

gemini update

Running tests

Start selenium-standalone in a separate tab before running the tests:

selenium-standalone start

Run gemini tests:

gemini test

Dependencies

Required software:

  1. WebDriver server implementation. There are several options:

    • Selenium Server — for testing in different browsers. Launch with the selenium-standalone start command (if you will get error like "No Java runtime present, requesting install." you should install Java Development Kit (JDK) for your OS.).

    • ChromeDriver — for testing in Google Chrome. Launch with the chromedriver --port=4444 --url-base=wd/hub command.

    • PhantomJS — launch with the phantomjs --webdriver=4444 command.

    • Cloud WebDriver services, such as SauceLabs or BrowserStack

  2. Compiler with support for C++11 ([email protected] or higher). This is a png-img requirement. Compiling on Windows machines requires the node-gyp prerequisites.

Installing

To install the utility, use the npm install command:

npm install -g gemini

Global installation is used for launching commands.

Configuring

Gemini is configured using a config file at the root of the project. Gemini can use one of the following files:

  • .gemini.conf.js
  • .gemini.conf.json
  • .gemini.conf.yml
  • .gemini.js
  • .gemini.json
  • .gemini.yml

Let's say we want to run our tests only in the locally installed PhantomJS.

In this case, the minimal configuration file will only need to have the root URL of your web app and the WebDriver capabilities of PhantomJS: For example,

rootUrl: http://yandex.com
browsers:
  PhantomJS:
    desiredCapabilities:
      browserName: phantomjs

Also, you need to run PhantomJS manually in WebDriver mode:

phantomjs --webdriver=4444

If you are using a remote WebDriver server, you can specify its URL with the gridUrl option:

rootUrl: http://yandex.com
gridUrl: http://selenium.example.com:4444/wd/hub

browsers:
  chrome:
    desiredCapabilities:
      browserName: chrome
      version: "45.0"

  firefox:
    desiredCapabilities:
      browserName: firefox
      version: "39.0"

You can also set up each browser to have its own node:

rootUrl: http://yandex.com

browsers:
  chrome:
    gridUrl: http://chrome-node.example.com:4444/wd/hub
    desiredCapabilities:
      browserName: chrome
      version: "45.0"

  firefox:
    gridUrl: http://firefox-node.example.com:4444/wd/hub
    desiredCapabilities:
      browserName: firefox
      version: "39.0"

Other configuration options

See the details of the config file structure and available options.

Writing tests

Each of the blocks that are being tested may be in one of the determined states. States are tested with the help of chains of step-by-step actions declared in a block's test suites.

For example, let's write a test for a search block at yandex.com:

gemini.suite('yandex-search', function(suite) {
    suite.setUrl('/')
        .setCaptureElements('.search2__input')
        .capture('plain')
        .capture('with text', function(actions, find) {
            actions.sendKeys(find('.search2__input .input__control'), 'hello gemini');
        });
});

We are creating a new test suite yandex-search, assuming that we will capture the .search2__input element from the root URL http://yandex.com. We know that the block has two states:

  • plain — right after the page is loaded
  • with text — with the hello gemini text inserted into .search2__input .input__control

States are executed one after another in the order in which they are defined, without the browser reloading in between.

See the details of test creation methods.

Using CLI

To complete the test creation procedure, you need to take reference shots using the following command:

gemini update [paths to test suites]

To launch a test (to compare the current state of a block with a reference shot), use the command:

gemini test [paths to test suites]

See the details of interaction with CLI and available options.

GUI

You can use the Gemini graphical user interface instead of the command line. It is located in the gemini-gui package and must be installed additionally:

npm install -g gemini-gui

GUI advantages:

  • Handy preview of reference shots

  • Clear real-time demonstration of the differences between a reference shot and the current state of a block

  • Easy to update reference shots

Plugins

Gemini can be extended with plugins. You can choose from the existing plugins or write your own. To use a plugin, install and enable it in your .gemini.yml:

system:
  plugins:
    some-awesome-plugin:
      plugin-option: value

HTML report

To see the difference between the current state of a block and a reference picture more clearly, use the HTML reporter - plugin for gemini. This plugin produces HTML report, which displays reference image, current image and differences between them, for each state in each browser. When all tests are completed, you will see a link to HTML report.

Programmatic API

To use Gemini in your scripts or build tools, you can use the experimental programmatic API.

Events

To learn more about all events in Gemini, see the events documentation.

gemini's People

Contributors

4exova avatar apostol2 avatar arikon avatar catwithapple avatar davidchin avatar dmitriy-kiselyov avatar dudagod avatar egavr avatar gitter-badger avatar greenkeeperio-bot avatar hatroman avatar heel avatar j0tunn avatar kvmamich avatar leonsabr avatar levonet avatar lo1tuma avatar miripiruni avatar mishaberezin avatar niedzielski avatar realzoberg avatar rostik404 avatar saulis avatar scf2k avatar seth2810 avatar sevinf avatar sipayrt avatar swinx avatar tormozz48 avatar up73k avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gemini's Issues

Expose gemini as global

Current code with explicit var gemini = require('gemini'); has following problems:

  • requires local gemini installation.
  • requires gemini to run from local installation.

This can be solved if we'll expose gemini as global. We should also try to keep current API with explicit require working.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Image Diff

It would be great if you give to users ability to reconfigure some steps in gemini by external packages/scripts, for example

  • Image comparison
  • Reporters

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Error message found: "StateError: TypeError: 'undefined' is not a function (evaluating 'value.match(regex)')"

Hi all,

I'm new to Gemini, start using it last week. But It fails to gather screen from the site I'd like to test AsiaWebDirect with error message "StateError: TypeError: 'undefined' is not a function (evaluating 'value.match(regex)')". I've tried other site such as google.com and yandex.com as in example, it works perfectly.

Thank you in advance for any help. If you need any further information, please feel free to let me know.

Shell output error message

...
 > CALL eval("__gemini.prepareScreenshot([\"#web-body > div.global_pink-header\"], {\"coverage\":false})") 
 > POST /session/:sessionID/execute {"script":"return __gemini.prepareScreenshot([\"#web-body > div.global_pink-header\"], {\"coverage\":false});","args":[]}
 > RESPONSE eval("__gemini.prepareScreenshot([\"#web-body > div.global_pink-header\"], {\"coverage\":false})") {"error":"JS","message":"TypeError: 'undefined' is not a function (evaluating 'value.match(regex)')\n    at parseBoxShadow (:136)\n    at getElementRect (:118)\n    at getScreenshotRect (:73)\n    at prepareScreenshotUnsafe (:19)\n    at prepareScreenshot (:9)\n    at anonymous (:1)\n    at La (phantomjs://webpage.evaluate():13)\n    at phantomjs://webpage.evaluate():13\n    at phantomjs://webpage.evaluate():13\n    at phantomjs://webpage.evaluate():14\n    at phantomjs://webpage.evaluate():14"}
✘ awd-search state_one [phantomjs]
StateError: TypeError: 'undefined' is not a function (evaluating 'value.match(regex)')
    at parseBoxShadow (:136)
    at getElementRect (:118)
    at getScreenshotRect (:73)
    at prepareScreenshotUnsafe (:19)
    at prepareScreenshot (:9)
    at anonymous (:1)
    at La (phantomjs://webpage.evaluate():13)
    at phantomjs://webpage.evaluate():13
    at phantomjs://webpage.evaluate():13
    at phantomjs://webpage.evaluate():14
    at phantomjs://webpage.evaluate():14
    at /Volumes/Macintosh HD 2/TestAutomation/Gemini/node_modules/gemini/lib/browser/index.js:165:37
    at _fulfilled (/Volumes/Macintosh HD 2/TestAutomation/Gemini/node_modules/gemini/node_modules/q/q.js:787:54)
    at self.promiseDispatch.done (/Volumes/Macintosh HD 2/TestAutomation/Gemini/node_modules/gemini/node_modules/q/q.js:816:30)
    at Promise.promise.promiseDispatch (/Volumes/Macintosh HD 2/TestAutomation/Gemini/node_modules/gemini/node_modules/q/q.js:749:13)
    at /Volumes/Macintosh HD 2/TestAutomation/Gemini/node_modules/gemini/node_modules/q/q.js:557:44
    at flush (/Volumes/Macintosh HD 2/TestAutomation/Gemini/node_modules/gemini/node_modules/q/q.js:108:17)
    at process._tickCallback (node.js:419:13)
 > CALL quit() 
 > DELETE /session/:sessionID 

Software version:

gemini --version

0.9.5

phantomjs --version

1.9.8

Configuration files:
.gemini.yml

rootUrl: http://www.asiawebdirect.com/
gridUrl: http://localhost:4444/wd/hub
browsers:
  phantomjs: phantomjs

awd.suite

var gemini = require('gemini');

gemini.suite('awd-search', function(suite) {
    suite.setUrl('/')
        .setCaptureElements('#web-body > div.global_pink-header')
        .capture('state_one');
});

Long pages crop in Chrome and Opera

Tested on a few pages with long layout. Took for example: https://github.com/bem/gemini
Specified CSS selector covers full page (body element also fits).
Simple test:

var gemini = require('gemini');
gemini.suite('crop', function(suite) {
    suite
        .setUrl('/bem/gemini')
        .setCaptureElements('.wrapper')
        .capture('initial');
});

Use Selenium Grid with PhantomJS 1.1.1, Opera 12.16, Firefox 25.0, Chrome 33, IE 9 and IE10.

On the resulting screenshots the Chrome one was cropped by the monitor dimensions and Opera as well. All other browsers show expected results.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

CLI options conflict

UI Screenshot testing utility
Compare current state with previous

Usage:
  gemini test [OPTIONS] [ARGS]


Options:
  -h, --help : Help
  -c CONFIGFILE, --config=CONFIGFILE : Config file
  -r ROOTURL, --root-url=ROOTURL : Override root url
  -g GRIDURL, --grid-url=GRIDURL : Override grid url
  -s SCREENSHOTSDIR, --screenshots-dir=SCREENSHOTSDIR : Override screenshots dir
  --debug : Turn on debugging output to the console
  --coverage : Turn on CSS coverage report
  --grep=GREP : Execute suits matching the pattern only
  -r REPORTERS, --reporter=REPORTERS : 

Arguments:
  TESTFILES : Paths to files or directories, containing tests

Option shortcut -r is declared twice in test command.
Now it's parsed as reporter but I think root url is preferred for this shortcut.

stateError:null with Internet Explorer8

Hi! I'm trying to run tests on my virtual machine with WinXP and IE8. When test is running i can see that IE opening required page and all looks good but the test fails with "stateError: null".

Human friendly error on reference file inexistance

This error

Error: Command failed: gm compare: Unable to open file (/Users/arikon/projects/bem/webrebels-gemini-demo/gemini/screens/button/plain/firefox.png) [No such file or directory].

   at ChildProcess.exithandler (child_process.js:637:15)
   at ChildProcess.EventEmitter.emit (events.js:98:17)
   at maybeClose (child_process.js:743:16)
   at Process.ChildProcess._handle.onexit (child_process.js:810:5)

Could look like this:

Reference file `gemini/screens/button/plain/firefox.png` does not exist.
Run `gemini gather <test-suite-file.js>` to create it.

Tests doesn't start

OSx, latest gemini install.

.gemini.xml:

rootUrl: http://google.com
gridUrl: http://localhost:4444/wd/hub
browsers:
  phantomjs: phantomjs

gemini/test.js:

var gemini = require('gemini');

console.log('test runs');

gemini.suite('body', function(suite) {
    suite
        .setUrl('/')
        .setCaptureElements('body')
        .capture('initial');
});
phantomjs --webdriver=4444
PhantomJS is launching GhostDriver...
[INFO  - 2014-07-11T08:47:03.288Z] GhostDriver - Main - running on port 4444

Run:

$ gemini gather
test runs
Total: 0 Passed: 0 Failed: 0 Skipped: 0

.focus(<element>) action

We are often need to focus on an element without click. Currently, it can be done by sending TAB key to previous element. This is not always convinient. Alternative is using sendKeys with empty string, but in user code it looks counter-intuitive. We may add new action focus(<element>) which will just call sendKeys with empty string.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

No way to get screenshots of elements that larger than viewport using SauceLabs

When I'm trying to get screenshot of large element I get this error:

StateError: Failed to capture the element because it is positioned outside of the captured body. Most probably you are trying to capture an absolute positioned
element which does not make body height to expand. To fix this place a tall enough <div> on the page to make body expand.
Element position: 4, 0; size: 991, 740. Page screenshot size: 1014, 686.

With small elements everything is fine.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

`update` command

Should work as follows:

  • by default, gather new images only for missing states
  • if passed --force flag, update all images.

gather should be deprecated after this command is implemented.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Can't find css selectors with spaces

If I set a capture element in my test a class attribute with space in name:

.setCaptureElements('.examle__class with space')

Gemini can't find it:

StateError: Can not find element with css selector: .examle__class with space

Expected, that complicated class with spaces will be processed successfully.

Version output not working

$ gemini --version

UI Screenshot testing utility

Usage:
  gemini COMMAND [OPTIONS] [ARGS]
  gemini [OPTIONS] [ARGS]

Commands:
  gather : Gather screenshots of all states
  test : Compare current state with previous

Options:
  -h, --help : Help
  --version : Show version

Round floating point coordiantes to capture maximum area

Currently Math.round is used to for rounding floating point coodinates when cropping image. This leads sometimes leads to incoreectly cropped image. Solution is to use floor for coordinates and ceil thus capturing the maximal area.

Opera 12 screenshots are bigger than others

For some reason opera12 screenshots are bigger.
This isn't big problem for gathering and testing, but it's problem for comparing with chrome version.

Debug info:

// ie9
{"locationInBody":{"left":50,"top":50},"locationInViewport":{"left":50,"top":50},"bodyHeight":230,"coverage":null,"canHaveCaret":false,"viewportHeight":740,"cropSize":{"height":130,"width":300}}
// opera 12
{"locationInBody":{"left":47,"top":47},"locationInViewport":{"left":47,"top":47},"bodyHeight":230,"coverage":null,"canHaveCaret":true,"viewportHeight":658,"cropSize":{"height":136,"width":306}}

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Show errors in case of incorrect path to test suite

I store my tests in a non-root directory and don’t want to write a full path to it. So, I created a symbolic link to my tests in the root folder and run tests with simple command like: gemini test folder_with_tests/test.js

I didn’t get any errors, but non of tests passed:

$ gemini test /symlink_folder/test.js
Total: 0 Passed: 0 Failed: 0 Skipped: 0

In case without opening slash:

$ gemini test symlink_folder/test.js
Error: Cannot find module 'gemini'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (absolute/path/to/suite/test.js:1:76)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

Also noted that in case of given a totally wrong path — there is no any error messages:

$ gemini gather any_folder_name/any_file_name.some_extension
Total: 0 Passed: 0 Failed: 0 Skipped: 0

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Test pages with async initialization

For example, Yandex Maps API loads itself asynchronously. Currently, there is no way to determine if initialization is finished and the only way to test such pages is actions.wait(some timeout) in callback. We need a better way to support async initialization. One possible solution is:

  1. Mark the suite as using async load:

    suite.loadsAsync()
  2. Inside app code, call some callback to signify that initialization is finished:

    loadSomethingAsync(function() {
       __gemini.start();
    })

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Как запустить тесты на phantomjs?

На такой конфигурации просит с меня gridUrl:

rootUrl: http://localhost/projects/project-name/
browsers:
    - name: phantomjs

Текст ошибки:

$ gemini gather
Error: Field "gridUrl" is required for using non-phantomjs browsers
To fix: Specify selenium grid URL in your config file or use only

Код файла gemini/test.js:

var gemini = require('gemini');
gemini.suite('b-social-list', function(suite) {
    suite
        .setUrl('main.html')
        .setCaptureElements('.b-social-list')
        .before(function(actions, find) {
            this.item = find('.b-social-list__text');
        })
        .capture('plain')
        .capture('hovered', function(actions, find) {
            actions.mouseMove(this.item);
        })
        .capture('pressed', function(actions, find) {
            actions.mouseDown(this.item);
        })
        .capture('clicked', function(actions, find) {
            actions.mouseUp(this.item);
        });
});

Без указания browsers тоже самое, webdriver запускаю:

$ phantomjs --webdriver=4444
PhantomJS is launching GhostDriver...
[INFO  - 2014-05-17T18:52:52.734Z] GhostDriver - Main - running on port 4444

За основу брал https://github.com/bem/bem-components/tree/feature/gemini

ReferenceError: window is not defined at Object.

Hi!
I installed gemini and tried to run examples from tutorial and got the error:
"ReferenceError: window is not defined at Object. (\gemini\node_modules\gemini\lib\browser\client-scripts\gemini.coverage.js:129:3) at Module._compile (module.js:456:26) at Object.Module._extensions..js (module.js:474:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.require (module.js:364:17) at require (module.js:380:17) at requireWithNoCache (\node_modules\gemini\lib\gemini.js:18:18) at Array.forEach (native) at ~\node_modules\gemini\lib\gemini.js:60:26"

Am i doing smth wrong?

Better html report

Suites in report should be collapsable and non-failed tests should be collapsed by default. File should have total number of executed, passed and failed tests.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.