I'm experimenting with template coverage and playing with the covered gem examples.

If I modify it to use erubi I don't get any coverage:

require 'erubi'
require 'erb'
require 'tilt'
$LOAD_PATH.unshift File.expand_path('../../../lib', __dir__)

template_path = File.expand_path("template.erb", __dir__)

ENV['COVERAGE'] ||= 'PartialSummary'
require 'covered/policy/default'

$covered.enable


@items = ["Cats", "Dogs", "Chickens"]
# template = ERB.new(File.read(template_path)).tap do |template|
# 	template.filename = template_path
# end
# puts template.result(binding)

erubi = Erubi::Engine.new(File.read(template_path), filename: 'template.erb')
puts eval(erubi.src, binding, 'template.erb')

# tilt = Tilt::ErubiTemplate.new(template_path)
# puts tilt.render(self)

$covered.disable

$covered.call($stdout)

$ ruby erb/coverage.rb
	Cats
	Dogs
	Chickens


* 0 files checked; 0/0 lines executed; 100.0% covered.

If you run the tilt variant:

require 'erubi'
require 'erb'
require 'tilt'
$LOAD_PATH.unshift File.expand_path('../../../lib', __dir__)

template_path = File.expand_path("template.erb", __dir__)

ENV['COVERAGE'] ||= 'PartialSummary'
require 'covered/policy/default'

$covered.enable


@items = ["Cats", "Dogs", "Chickens"]
# template = ERB.new(File.read(template_path)).tap do |template|
# 	template.filename = template_path
# end
# puts template.result(binding)

# erubi = Erubi::Engine.new(File.read(template_path), filename: 'template.erb')
# puts eval(erubi.src, binding, 'template.erb')

tilt = Tilt::ErubiTemplate.new(template_path)
puts tilt.render(self)

$covered.disable

$covered.call($stdout)

$ ruby erb/coverage.rb
	Cats
	Dogs
	Chickens

Traceback (most recent call last):
	17: from erb/coverage.rb:26:in `<main>'
	16: from /Users/edzhelyov/rb/covered/lib/covered/policy.rb:91:in `disable'
	15: from /Users/edzhelyov/rb/covered/lib/covered/capture.rb:51:in `disable'
	14: from /Users/edzhelyov/rb/covered/lib/covered/wrapper.rb:60:in `disable'
	13: from /Users/edzhelyov/rb/covered/lib/covered/source.rb:60:in `disable'
	12: from /Users/edzhelyov/rb/covered/lib/covered/wrapper.rb:60:in `disable'
	11: from /Users/edzhelyov/rb/covered/lib/covered/persist.rb:101:in `disable'
	10: from /Users/edzhelyov/rb/covered/lib/covered/wrapper.rb:60:in `disable'
	 9: from /Users/edzhelyov/rb/covered/lib/covered/wrapper.rb:60:in `disable'
	 8: from /Users/edzhelyov/rb/covered/lib/covered/wrapper.rb:60:in `disable'
	 7: from /Users/edzhelyov/rb/covered/lib/covered/wrapper.rb:60:in `disable'
	 6: from /Users/edzhelyov/rb/covered/lib/covered/capture.rb:80:in `disable'
	 5: from /Users/edzhelyov/rb/covered/lib/covered/capture.rb:70:in `flush'
	 4: from /Users/edzhelyov/rb/covered/lib/covered/capture.rb:70:in `each_slice'
	 3: from /Users/edzhelyov/rb/covered/lib/covered/capture.rb:70:in `each'
	 2: from /Users/edzhelyov/rb/covered/lib/covered/capture.rb:71:in `block in flush'
	 1: from /Users/edzhelyov/rb/covered/lib/covered/files.rb:43:in `mark'
/Users/edzhelyov/rb/covered/lib/covered/coverage.rb:99:in `mark': index -3 too small for array; minimum: 0 (IndexError)

Could you give me any hints on why this is happening?

I have a project that instantiates a few FIFOs using generate for.

covered score crashes with
ERROR! Referencing undefined named block hierarchy (read_st) in module iob_fifo_async, file iob_i2s_tdm_pp.v, line 747

Removing the generate and instantiating a single FIFO it works

I can provide more info if this is issue is to be worked on.

Thanks

@bryanmacfarlane I wanted to try to give you some feedback regarding some of the new features. Maybe you have some suggestions or ideas about how I can implement it better.

include directive

Using the include directive to add new configurations is unintuitive. I'm glad that it finally is possible, but it's also tricky to get right.

I ended up having to add these lines to almost all my projects to get the include directives to work correctly:

https://github.com/ioquatix/covered/blob/aca32de87dc224db458ceeebaeb2c3c110e5e893/.github/workflows/development.yml#L21-L22

It's not obvious why these can't just couldn't be env: nil and experimental: false or something like that. It's also not obvious why they need to exist - it seems just to work around the original implementation of include trying to modify existing matrix entries which is not the expected default behaviour IMHO.

Then, I found out if I don't have experimental: true or experimental: false, it fails because https://github.com/ioquatix/covered/blob/aca32de87dc224db458ceeebaeb2c3c110e5e893/.github/workflows/development.yml#L8 is undefined. So all my include directives must have this, even if it's the same as the default in the matrix.

Ultimately, I find the way include to work simply more complicated and tricky than travis. I wish it was simpler and encouraged a more "canonical" approach for these kinds of situations.

Inputs

The covered gem has access to the travis_job_id when running on Travis which is all Coveralls needs to hook up the jobs. I don't think travis_job_id is a secret. But in order to work for GitHub actions, it does require this GITHUB_TOKEN which is not available by default.

I understand the need for security, but I also find it quite tricky to implement.

I tried changing env: COVERAGE=PartialSummary,Coveralls by appending GITHUB_TOKEN=${{secrets.GITHUB_TOKEN}} but it didn't work. I also tried adding a with: github-token: ${{secrets.GITHUB_TOKEN}} to the run block, but that didn't work. Finally, I found this works:

https://github.com/ioquatix/covered/blob/aca32de87dc224db458ceeebaeb2c3c110e5e893/.github/workflows/development.yml#L50

However it's cumbersome and exposes the environment to all invocations of the command even if it isn't necessary.

I don't know why the GITHUB_TOKEN secret is required by coveralls, or why travis_job_id, a public identifier, was sufficient when running on travis. In any case, I don't really want to modify all my tasks like this. Maybe GITHUB_TOKEN should just be part of the general ENV? Even with the official coveralls client, it's cumbersome at best (you need to define the with: github-token: ... as outlined above). Maybe it's just an implementation detail of coveralls?

Given that GITHUB_TOKEN is a secret, I think ideally we don't want to expose that to the unit tests (could be leaked easily). So, I could add another build step like the official Coveralls GitHub Action, but it's still cumbersome. Maybe there is some kind of github_job_id that should be public and sufficient for Coveralls to make the connection it needs?

cc @nickmerwin

Is this by design, or an oversight? this leads to drastically different results than simplecov when files have a lot of requires

The README mentions support for measuring coverage of view templates. I was hoping that this could be applied to measure coverage of ERB templates on a Rails project, but my experiments in collecting coverage using covered on a small Rails app haven't been conclusive.

Is there some form of documentation that can be added for this use case? A sample application and sample report output would be welcome if that's easier to produce than putting instructions into words.

Got the following error when running no specs:

An error occurred in an `after(:suite)` hook.
Failure/Error: packer.flush

Encoding::UndefinedConversionError:
  "\x83" from ASCII-8BIT to UTF-8
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/persist.rb:84:in `write'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/persist.rb:84:in `flush'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/persist.rb:84:in `block in save!'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/persist.rb:75:in `open'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/persist.rb:75:in `save!'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/persist.rb:109:in `disable'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/wrapper.rb:60:in `disable'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/source.rb:60:in `disable'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/wrapper.rb:60:in `disable'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/capture.rb:51:in `disable'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/policy.rb:91:in `disable'
# /Users/choubacha/.rvm/gems/ruby-2.6.8@v3/gems/covered-0.14.0/lib/covered/rspec.rb:66:in `block (2 levels) in <top (required)>'

Top 0 slowest examples (0 seconds, 0.0% of total time):

Finished in 0.05224 seconds (files took 7.27 seconds to load)
0 examples, 0 failures, 1 error occurred outside of examples

Thank you for your work on having better coverage reports!

Maybe i overlooked it but i could not find any hints on how to produce an output on the README page.
To make it easier to implement it would be helpful.

Beste wishes, tee