Comments (21)
@ms88privat : Ok then you confirm that it is actually reading image from the disk is the slow path! This is pretty odd to me. Image size on disk is around 27kb which I expect fs.readSync
should be quite fast on this.
I need to spend more time on this and doing some benchmarks to understand what is going on
from differencify.
maybe related to this? jestjs/jest#2925
from differencify.
Hi @ms88privat,
Would you please tell me how long it takes to run your test?
Yeah, being a bit slow is totally expected behavior, as it needs to launch browser, load the page, go through steps, capture screenshot, store it and compare it. All these processes take a significant amount of time. I've already tried to optimised this process by storing the captured image in memory instead of hard disk and using the fastest image comparison library.
On my local machine, I have 36 tests that run in 126 seconds. I managed to run same tests on docker which took around 92 seconds to complete (3 seconds per test) which is quite acceptable for me.
You should also consider your test scenario as well. what are your steps and how long it takes to reach the point that you would want to capture the screenshot. You should always optimize that as well.
I am happy to help more when I know more details about your issue.
Nima
from differencify.
@ms88privat : Please let me know node and jest version you are running your test on
from differencify.
Are you using jest as the test runner? I'm on node 7.6.0 and jest 20.0.4.
https://github.com/NimaSoroush/differencify/blob/master/examples/jest-example.test.js#L22
Here in your example, the one test has 40 seconds timeout. Why did you choose this? At my place it nearly takes 20+ seconds for this test, thats why this option is necessary.
But I did some research and it is just about the file system access. I didn't use your library for it and I didn't use puppeteer
or image compare. It is just loading the images into the buffer which takes all the time. If I start the same script without jest environment (just via node CLI), it is super fast.
Let me try upgrading jest. 3 sec per test would be fine, but 20+ sec is not :)
from differencify.
Upgrading jest didn't solve it.
from differencify.
So it isn't related to your library. I'm just curios if it is working fast enough on your side. But I didn't believed so because of the extreme timeout option you did set in your example. That's why I opened this issue. But if it is working on your side, feel free to close this one
from differencify.
Hi @ms88privat:
Thanks for providing more info. I am wondering why on jest env should cause this issue for you. As I said I am not seeing this issue in my local env and docker. I will close this issue but lets keep posting here till we understand what is the root cause as it might be the case for other peoples.
I will also update the extreme timeout on jest example. That was my arbitrary number when I was developing this package.
Nima
from differencify.
I see, it looks like the resolution makes the difference. Please try using "1920x1080" instead of your default 800x600 or so.
But this shouldn't be an issue anyway because the same code via node CLI is super fast.
from differencify.
What is the image size you getting by "1920x1080"
?
I am actually trying { width: 1600, height: 1200 }
and it is pretty fast.
Can you try it with .capture({ omitBackground: true })
it will remove the useless background and makes the image a bit lighter
from differencify.
@ms88privat : Would you please run you test with debug
enabled? Basically:
const differencify = new Differencify({ debug: true });
this should give you the timestamp of each step execution. Please post the outcome here
from differencify.
So it looks like the test itself is "fast" but the whole test case did take 30s. It is really slow after the tab closes and before the image comparison (confirms my suspicions about the file reading step). The image itself is 26KB.
I created a fresh app with "create-react-app". Node and everything else is up2date.
My test setup looks like this:
import Differencify from 'differencify';
const differencify = new Differencify({ debug: true });
describe('tests differencify', () => {
it('validate github page appear correctly', async () => {
await differencify
.init()
.resize({ width: 1600, height: 1200 })
.goto('http://localhost:3000/')
.capture()
.toMatchSnapshot()
.close()
.end();
});
});
Here the debug test output:
The timestamps looking normal, but it did stop the time at least for 15 seconds in between..
from differencify.
Just tested the same code on coworker's computer. Same result :(
It really get's stuck in between. Which OS are you on? We are working on macOS.
But yeah, its a problem with Jest somehow... Just reading an image files does also not work, without using your library at all. Thanks anyway! 👍
from differencify.
I just saw the same pattern in your test-fail.png in the readme? Only one test, which takes 17s, but the whole test suite took almost 40s?
from differencify.
Ok, Now I see where does the problem come from. It is actually the image comparison part that takes a long time. Just tried to make all my tests fail locally and yeah, it takes a long time for every single test to be compared with the original screenshot.
I already have some ideas around this. I am using Jimp as the image processing lib which uses pixelmatch for image comparison. Jimp also does perceptual hash comparison of generated images which I guess will be the most time-consuming bit.
I will do a benchmark on this and will update the result here.
from differencify.
Ok, here is my the result on running first test of jest-example.test.js
Disabling perceptual hash comparison
Disabling perceptual hash comparison and no writing diff image to disk
So basically it is writing to disk that takes a long time in case of failure. As you might already know if you pass saveDifferencifiedImage: false
to differencify then it wont write the diff image to disk and we are doing this as we are running out tests in CI and Docker. we only enable that flag when there is a failure on tests. you might want to do the same
from differencify.
As an improvement, I am happy to put perceptual hash comparison
behind a config flag. Maybe:
const differencify = new Differencify({ compareDistance: true });
from differencify.
Meantime we can look into how to improve image write to disk functionality. I just played with some Jimp config and I managed to reduce Original run
's time from 46.537s
to 14.794s
by sacrificing image quality.
I can put this under config flag too. I will put a PR for this
from differencify.
Thanks for further investigating this. But for me, if I just import Jimp
and do Jimp.read
inside jest, it also takes so much time. But when I execute the same line of code Jimp.read
or the whole stuff and invoke it via node
directly without jest, it is super fast, even in best quality!
from differencify.
@ms88privat : Interesting! Would you be able to verify the same behavior with pixelmatch using with/without Jest?
from differencify.
pixelmatch is fast on both. But for pixelmatch to work I need to read the images to compare, which is slow again on jest...
from differencify.
Related Issues (20)
- Multiple teststoMatchSnapShots
- An in-range update of eslint-plugin-import is breaking the build 🚨 HOT 7
- An in-range update of jimp is breaking the build 🚨 HOT 7
- A mismatchThreshold of zero will always fail.
- Tests pass even when error occurs in tests HOT 1
- An in-range update of eslint-plugin-jsx-a11y is breaking the build 🚨 HOT 2
- Running w/out Jest prevents consistent filenames HOT 4
- An in-range update of eslint is breaking the build 🚨 HOT 7
- An in-range update of babel-eslint is breaking the build 🚨 HOT 1
- An in-range update of jsdoc-to-markdown is breaking the build 🚨 HOT 1
- Differencify appends indexes to a unique generated snapshots' names. HOT 3
- Feature request: partial page comparison HOT 1
- An in-range update of eslint is breaking the build 🚨 HOT 5
- An in-range update of jsdoc-to-markdown is breaking the build 🚨 HOT 1
- An in-range update of eslint-plugin-import is breaking the build 🚨 HOT 4
- Issue with mockRequest(); HOT 1
- An in-range update of babel-eslint is breaking the build 🚨 HOT 1
- An in-range update of jsdoc is breaking the build 🚨 HOT 1
- runtime issues with Puppeteer 8.0
- ReferenceError: GlobalOptions is not defined HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from differencify.