pulyaevskiy / test-coverage Goto Github PK

Runs tests in Dart VM and collects coverage data.

License: BSD 3-Clause "New" or "Revised" License

Dart 100.00%

dart-vm dartlang

test-coverage's Introduction

test_coverage

A simple command-line tool to collect test coverage information from Dart VM tests. It is useful if you need to generate coverage reports locally during development.

Usage

Add dev dependency to your pubspec.yaml:

dev_dependencies:
  test_coverage: ^0.4.2

Run pub get to install. Then, in the root of your project run:

pub run test_coverage

Result is saved in coverage/lcov.info. If you have lcov tool installed on your system (for Mac it's brew install lcov) you can generate coverage reports using genhtml command:

genhtml -o coverage coverage/lcov.info
# Open in the default browser (mac):
open coverage/index.html

Integrations

Resulting coverage/lcov.info file is ready to be consumed by Codecov command-line tool, so no extra step is needed.

This library was not tested with coveralls yet.

Generating badge image

Coverage badge SVG image is automatically generated and saved to coverage_badge.svg in your project root directory. You can add it to Git and use in README.md on Github as follows:

![Coverage](https://raw.githubusercontent.com/{you}/{repo}/master/coverage_badge.svg?sanitize=true)

If you don't need it you can pass --no-badge flag when running test coverage.

Known limitations

This library was created to run Dart VM tests. It has not been tested and likely won't work for Dart code targeting web platform (compiled to JavaScript). There is no need to use this tool for Flutter as it allows collecting coverage information with flutter test --coverage.

How it works

The tool performs following steps:

1. Generates `test/.test_coverage.dart` file (a "test all" script).

Scans your test/ directory to find all *_test.dart files and creates test/.test_coverage.dart which imports all found test files and runs corresponding main() of all tests functions within the same file (and process), which simplifies collection of coverage information.

It is recommended to add this file to your .gitignore.

Below is an example of test/.test_coverage.dart:

// Auto-generated by test_coverage. Do not edit by hand.
// Consider adding this file to your .gitignore.

import 'some_test.dart' as some_test;
import 'nested/other_test.dart' as other_test;
import 'some_other_test.dart' as some_other_test;

void main() {
  some_test.main();
  other_test.main();
  some_other_test.main();
}

2. Runs the tests

Following command is used to run the tests:

dart --pause-isolates-on-exit --enable_asserts --enable-vm-service \
  test/.test_coverage.dart

3. Collects and formats coverage information

When test execution is completed the tool uses functionality of the coverage package to collect and format coverage report.

Feel free to file feature requests and bug reports at the issue tracker.

4. Minimal coverage percentage

Set minimal coverage percentage to pass min-coverage percentage value of coverage.

5. See test output

print-test-output to show test output.

Show output of tests

Feel free to file feature requests and bug reports at the issue tracker.

test-coverage's People

Contributors

Stargazers

Watchers

Forkers

vitusortner coybit minikin jogboms rafaldziuryk gustavo-henrique-em bobjackman wilburx9 shyndman monagjr higorlapacw matanlurey bsutton shcherbuk96 morristech jeuler dellgreen platform-platform oscardelgado featzima quirijngb gmpassos appsolutely-wonderful jjagg masseelch f3ath civts driky bradyt augusttoko laurensdewaele vojislavjanjic opreageorgealexandru vedadom mralioo hmtanbir danik-tro csmizzle belchaa hemant15kumar pedrorufino yoto1118 mehmetalisarac wustrong ectuser sathik-basha voschezang len4i deetonn anmarchese willianbl99 aditya-botsync ic21b023 cristianms felahgs niketh-boreddy datnguye misiker101 anderdobo genhoni

test-coverage's Issues

How to coverage report for plugins like sqflite

I can run sqflite tests using flutter run "test file path" but I need to include test in coverage report. How should I do that. any code samples appreciated

Fails intermittently on CI

This is a pretty helpful package, however I am facing an issue where it fails intermittently. Here is an example: https://github.com/mobxjs/mobx.dart/runs/560804368?check_suite_focus=true

Any hints ?

Never completes

pub run test_coverage
|

The cursor keeps blinking at this location and there is only ever output if one or more tests fail.
If all tests succeed (checked with pub run test), pub run test_coverage just hangs there forever.

Dart SDK version: 2.11.0-213.1.beta (beta) (Unknown timestamp) on "linux_x64"

option to not regenerate the `.test_coverage.dart`

... would be nice.

I have my own code that generates this file and with such an option this could be used instead.

Unhandled exception

I'm testing the package https://github.com/bsutton/dshell

pub run test_coverage
Precompiling executable... (1.1s)
Precompiled test_coverage:test_coverage.
Found 54 test files.
Generated test-all script in test/.test_coverage.dart. Please make sure it is added to .gitignore.
Unhandled exception:
Tests failed with exit code 255
#0      runTestsAndCollect (package:test_coverage/src/functions.dart:122:5)
<asynchronous suspension>
#1      main (file:///<home>/.pub-cache/hosted/pub.dartlang.org/test_coverage-0.4.3/bin/test_coverage.dart:60:9)
#2      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#3      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

Add a way to specify the output file

It would be awesome if we could specify where we want the lcov.info file should be saved.

Null-safety support

Hi! What's the status on supporting the null-safety within this package?

Coverage fails intermittently on CircleCI with exit code 255

Unhandled exception:
Tests failed with exit code 255
#0      runTestsAndCollect (package:test_coverage/src/functions.dart:120:5)
<asynchronous suspension>
#1      main (file:///root/.pub-cache/hosted/pub.dartlang.org/test_coverage-0.4.0/bin/test_coverage.dart:51:9)
#2      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:305:32)
#3      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:174:12)

We're encountering this exception semi-frequently on our CircleCI builds. Example: https://app.circleci.com/jobs/github/mobxjs/mobx.dart/1944

If you check out our pipelines, many of the failures are the same. Any insight?

Thanks!

Need option to run test_coverage without concurrency

We are facing an issue when we try to run test_coverage because of concurrency.
Do we have an option to run 'pub run test_coverage' without concurrency?

Thanks,
Naresh

Option --no-badge does not work

If I try the following,

git clone https://github.com/pulyaevskiy/test-coverage.git
cd test-coverage/test/stub-package
rm coverage_badge.svg
pub get
pub run test_coverage --no-badge

Then the coverage_badge.svg file is back.

$ dart --version
Dart VM version: 2.7.1 (Thu Jan 23 13:02:26 2020 +0100) on "macos_x64"

run command `flutter pub run test_coverage` frozen in terminal.

When execute the flutter pub run test_coverage, it aways frozen in terminal.

yaml: test_coverage: ^0.3.0

Test run failed with error.

Hi everyone, I'm facing an issue while running the test, you can see below the error I got in terminal while running the
"dart --pause-isolates-on-exit --enable_asserts --enable-vm-service
test/.test_coverage.dart"
command. The .test_coverage.dart file is generated in test folder but couldn't run the test I,e step 2.

My yaml is
environment:
sdk: ">=2.7.0 <3.0.0"

dependencies:
flutter:
sdk: flutter

cupertino_icons: ^1.0.1
dashbook: 0.0.9
coverage: 0.14.2

dev_dependencies:
test_coverage: 0.5.0
flutter_test:
sdk: flutter

My terminal===

kapil@kapil:/Documents/projects/mobile$ flutter pub run test_coverage
Found 2 test files.
Generated test-all script in test/.test_coverage.dart. Please make sure it is added to .gitignore.
^Ckapil@kapil:/Documents/projects/mobile$ dart --pause-isolates-on-exit --enable-vm-service=NNNN test/.test_coverage.dart
Observatory listening on http://127.0.0.1:35921/NfMoCsqD-L0=/
../../../Music/flutter/packages/flutter_test/lib/src/_goldens_io.dart:9:8: Error: Not found: 'dart:ui'
import 'dart:ui';
^
../../../Music/flutter/packages/flutter_test/lib/src/accessibility.dart:8:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui;
^
../../../Music/flutter/packages/flutter_test/lib/src/animation_sheet.dart:6:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui;
^
../../../Music/flutter/packages/flutter_test/lib/src/binding.dart:7:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui;
^
../../../Music/flutter/packages/flutter_test/lib/src/frame_timing_summarizer.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui';
^
../../../Music/flutter/packages/flutter_test/lib/src/image.dart:7:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui;
^
../../../Music/flutter/packages/flutter_test/lib/src/matchers.dart:7:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui;
^
../../../Music/flutter/packages/flutter_test/lib/src/matchers.dart:8:8: Error: Not found: 'dart:ui'
import 'dart:ui';
^
../../../Music/flutter/packages/flutter_test/lib/src/test_pointer.dart:10:1: Error: Not found: 'dart:ui'
export 'dart:ui' show Offset;
^
../../../Music/flutter/packages/flutter_test/lib/src/window.dart:6:8: Error: Not found: 'dart:ui'
import 'dart:ui' hide window;
^

Exclude files

Hi!

I'm using this pub run test_coverage --exclude="**.g.dart" to exclude generated file. Seem that it doesn't work.

Any idea?

NoSuchMethodError: The getter 'iterator' was called on null.

It was working fine for the past builds but suddenly stopped working yesterday.

This is my yaml config: https://github.com/xsahil03x/super_enum/blob/master/.github/workflows/push.yml

Here's the full log.

2019-12-02T20:17:38.8214424Z Installed executable test_coverage.
2019-12-02T20:17:38.8273896Z Warning: Pub installs executables into $HOME/.pub-cache/bin, which is not on your path.
2019-12-02T20:17:38.8274333Z You can fix that by adding this to your shell's config file (.bashrc, .bash_profile, etc.):
2019-12-02T20:17:38.8274438Z 
2019-12-02T20:17:38.8274753Z   export PATH="$PATH":"$HOME/.pub-cache/bin"
2019-12-02T20:17:38.8274814Z 
2019-12-02T20:17:38.8274926Z Activated test_coverage 0.4.0.
2019-12-02T20:17:39.1662717Z Found 1 test files.
2019-12-02T20:17:39.1773272Z Generated test-all script in test/.test_coverage.dart. Please make sure it is added to .gitignore.
2019-12-02T20:17:39.6580057Z Unhandled exception:
2019-12-02T20:17:39.6580900Z NoSuchMethodError: The getter 'iterator' was called on null.
2019-12-02T20:17:39.6581025Z Receiver: null
2019-12-02T20:17:39.6581115Z Tried calling: iterator
2019-12-02T20:17:39.6597401Z #0      Object.noSuchMethod (dart:core-patch/object_patch.dart:51:5)
2019-12-02T20:17:39.6597793Z #1      new List.from (dart:core-patch/array_patch.dart:52:19)
2019-12-02T20:17:39.6597925Z #2      new VM._fromJson (package:vm_service/vm_service.dart:6420:19)
2019-12-02T20:17:39.6598049Z #3      VM.parse (package:vm_service/vm_service.dart:6363:32)
2019-12-02T20:17:39.6598208Z #4      createServiceObject (package:vm_service/vm_service.dart:69:34)
2019-12-02T20:17:39.6598324Z #5      VmService._processResponse (package:vm_service/vm_service.dart:1862:28)
2019-12-02T20:17:39.6598455Z #6      VmService._processMessageStr (package:vm_service/vm_service.dart:1842:7)
2019-12-02T20:17:39.6598584Z #7      VmService._processMessage (package:vm_service/vm_service.dart:1793:7)
2019-12-02T20:17:39.6598710Z #8      _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10)
2019-12-02T20:17:39.6598835Z #9      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
2019-12-02T20:17:39.6598963Z #10     _DelayedData.perform (dart:async/stream_impl.dart:591:14)
2019-12-02T20:17:39.6599070Z #11     _StreamImplEvents.handleNext (dart:async/stream_impl.dart:707:11)
2019-12-02T20:17:39.6599200Z #12     _PendingEvents.schedule.<anonymous closure> (dart:async/stream_impl.dart:667:7)
2019-12-02T20:17:39.6599327Z #13     _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
2019-12-02T20:17:39.6599897Z #14     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
2019-12-02T20:17:39.6600336Z #15     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:116:13)
2019-12-02T20:17:39.6600706Z #16     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:173:5)
2019-12-02T20:17:39.6918847Z ##[error]Process completed with exit code 255.

considering 'dart_test.yaml'

Would be nice if the test-coverage package consider the config-file of test-Package (https://pub.dev/packages/test). Its named dart_test.yaml and is typacally in the root folder. There the user can be define a name pattern of the test files. If the user dont use the default-pattern for the test-files, your test-coverage will not find the test-files.

would be great if this both packages will be more closer :)

alternative: make the file-pattern editable.

issue in resolving dependencies... Because test_coverage

[contentstack_utils] dart pub get
Resolving dependencies...
charcode 1.2.0 (1.3.1 available)
meta 1.3.0 (1.4.0 available)
Got dependencies!
exit code 0

[tensify] dart pub get
Resolving dependencies...
Because test_coverage >=0.3.0 depends on glob ^1.1.7 and dartdoc 0.45.0 depends on glob ^2.0.0, test_coverage >=0.3.0 is incompatible with dartdoc 0.45.0.
And because no versions of dartdoc match >0.45.0 <0.46.0, test_coverage >=0.3.0 is incompatible with dartdoc ^0.45.0.
So, because tensify depends on both dartdoc ^0.45.0 and test_coverage ^0.5.0, version solving failed.
exit code 1

Add option to supply all source to Observatory

Coverage doesn't detect files that haven't been covered if their is no route to this files from the test files.
The workaround is to add imports of all sources files to a test file so that the Dart Observatory can then see them.
This is a known issue and workaround is discussed here:

flutter/flutter#27997 (comment)

Fortunately this project already constructs a test file dynamically so this file should be able to be used if option is set

Could not run tests with Observatory enabled. Try setting a different port with --port option.

Issue

Could not run tests with Observatory enabled. Try setting a different port with --port option.

Error Log

Bad state: Could not run tests with Observatory enabled. Try setting a different port with --port option.
#0 runTestsAndCollect (package:test_coverage/src/functions.dart:107:5)

#1 main (file:///Users/QXZ12DI/project/test_coverage-0.4.1/bin/test_coverage.dart:51:9)
#2 _startIsolate. (dart:isolate-patch/isolate_patch.dart:299:32)
#3 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

Details

QXZ12DI@LSCN886414 test_coverage-0.4.1 % flutter pub get
Running "flutter pub get" in test_coverage-0.4.1...                 2.0s
QXZ12DI@LSCN886414 test_coverage-0.4.1 % pub run test_coverage . false
Found 3 test files.
Generated test-all script in test/.test_coverage.dart. Please make sure it is added to .gitignore.
Unhandled exception:
Bad state: Could not run tests with Observatory enabled. Try setting a different port with --port option.
#0      runTestsAndCollect (package:test_coverage/src/functions.dart:107:5)
<asynchronous suspension>
#1      main (file:///Users/QXZ12DI/project/test_coverage-0.4.1/bin/test_coverage.dart:51:9)
#2      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#3      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)
QXZ12DI@LSCN886414 test_coverage-0.4.1 % code .
QXZ12DI@LSCN886414 test_coverage-0.4.1 % dart --version
Dart SDK version: 2.10.3 (stable) (Tue Oct 27 14:44:30 2020 +0100) on "macos_x64"
QXZ12DI@LSCN886414 test_coverage-0.4.1 % flutter doctor --version
Flutter 1.22.3 • channel unknown • unknown source
Framework • revision 8874f21e79 (5 weeks ago) • 2020-10-29 14:14:35 -0700
Engine • revision a1440ca392
Tools • Dart 2.10.3
QXZ12DI@LSCN886414 test_coverage-0.4.1 %

Option --no-badge still does not work

It looks like the ArgParser was updated to have the negate option but its not being used to prevent the badge file from being output.

https://github.com/pulyaevskiy/test-coverage/blob/master/bin/test_coverage.dart#L67

Related to: #15

Give this repo to someone who can continue maintenance of it

@pulyaevskiy I think it's important that this repo is maintained and you must be very busy. I suggest putting out some feelers to the community and see if anyone is willing to take over as a maintainer.

`Dart 2.10.5 is incompatible with your dependencies' SDK constraints` even setting `sdk: '>=2.10.5 <3.0.0'`

francesco@francesco-yoga720:~/projects/yak_packages/yak_runner$ pub run test_coverage
Dart 2.10.5 is incompatible with your dependencies' SDK constraints. Please run "pub get" again.

environment:
  sdk: '>=2.10.5 <3.0.0'

dependencies: 
  meta: ^1.2.4

dev_dependencies:
  test: ^1.15.7
  mockito: ^4.1.3
  test_coverage: ^0.4.2

[✓] Flutter (Channel master, 1.26.0-18.0.pre.156, on Linux, locale en_US.UTF-8)
    • Flutter version 1.26.0-18.0.pre.156 at /home/francesco/snap/flutter/common/flutter
    • Framework revision 45508985b1 (10 hours ago), 2021-02-02 22:46:04 -0500
    • Engine revision 2c144c3eeb
    • Dart version 2.12.0 (build 2.12.0-282.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /home/francesco/Android/Sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_SDK_ROOT = /home/francesco/Android/Sdk
    • Java binary at: /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~20.04-b01)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • CHROME_EXECUTABLE = /snap/bin/chromium

[✓] Linux toolchain - develop for Linux desktop
    • clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
    • cmake version 3.10.2
    • ninja version 1.8.2
    • pkg-config version 0.29.1

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/linux#android-setup for detailed instructions).

[✓] Connected device (2 available)
    • Linux (desktop) • linux  • linux-x64      • Linux
    • Chrome (web)    • chrome • web-javascript • Chromium 88.0.4324.96 snap

! Doctor found issues in 1 category.

Build Issue while generating coverage

While building using pub get throughs error output:

using

test: ^1.14.4
test_coverage: ^0.4.3

[package_name] dart pub get
Resolving dependencies...
Because test_coverage ^0.4.0 depends on coverage ^0.13.0 and test >=1.16.6 depends on coverage ^1.0.1, test_coverage ^0.4.0 is incompatible with test >=1.16.6.
So, because contentstack_utils depends on both test ^1.16.8 and test_coverage ^0.4.3, version solving failed.
exit code 1

lcov dependency discontinued

is the package going to be upgraded? since some of it's dependencies have been discontinued in pub. e.g lcov

Command hang in the void

When running code_coverage, the command hang after printing:

Generated test-all script in test/.test_coverage.dart. Please make sure it is added to .gitignore.

Nothing happens even after waiting a really long time.

I ran that command on a new project.

Dart SDK version: 2.12.1 (stable) (Wed Mar 10 10:18:47 2021 +0100) on "windows_x64"
Windows 10
Powershell

Is that a known problem ? Any idea what I could check to try and fix the issue ?

Mock class doesn't show in coverage report

I have an issue every time when I run the flutter test --coverage and genhtml -o coverage coverage/lcov.info. After generating the coverage report to analyze it, I found all the Mock classes (Remote data source) do not cover by the library.

Example:

class MockRegisterUserDataSource extends Mock
    implements RegisterUserDataSource {}

void main() {
  MockRegisterUserDataSource mockRegisterUserDataSource;
  RegisterUserRepositoryImp repository;

  setUp(() {
    mockRegisterUserDataSource = MockRegisterUserDataSource();
    repository =
        RegisterUserRepositoryImp(dataSource: mockRegisterUserDataSource);
  });

test('should create an user', () async {
      // arrange
      final userModel = UserModel(email: email, displayName: displayName);
      final user = User(email, displayName);
      when(mockRegisterUserDataSource.createAccount(any, any))
          .thenAnswer((_) async => userModel);
      // act
      final result = await repository.createAccount(email, displayName);
      // assert
      verify(mockRegisterUserDataSource.createAccount(email, displayName));
      verifyNoMoreInteractions(mockRegisterUserDataSource);
      expect(result, equals(Right(user)));
    });
}

And, this is the LCOV - code coverage report.