Annotations and ignore logic not working on Linux about homebrew-test-bot HOT 12 CLOSED

We'll want to remove

when this is sorted.

from homebrew-test-bot.

No idea here. I wonder if it's something else about the environment rather than the OS i.e. the Linux runner/GitHub Actions runner has some environment variables set that are interfering?

from homebrew-test-bot.

Yea, I'm pretty stumped as well. However, I think my hypothesis about what's going on is correct, judging from what's happening on the Linux runner in this PR: Homebrew/homebrew-core#89567

It has a whole bunch of formulae to test, but it tests bison first. The test fails and test-bot exits. It never even makes it to the code that produces annotations (or else we would see them, and we'd see duplicated output in the CI log). What it should be doing is logging the failure and moving on to test other formulae.

from homebrew-test-bot.

CC @Homebrew/brew for ideas.

from homebrew-test-bot.

This is your problem:

Error: No such file or directory @ realpath_rec - /home/runner

Probably from:

https://github.com/Homebrew/brew/blob/07f14c32e9996196661bdc20830f48797079c463/Library/Homebrew/utils/github/actions.rb#L29

from homebrew-test-bot.

This is because of https://github.com/Homebrew/homebrew-core/blob/master/.github/workflows/tests.yml#L182-L190. We would need to change GITHUB_WORKSPACE going into this.

I can fix this but I wonder if we can just make it use GitHub's container system somehow (where it'll work out of the box). @dawidd6 I assume you've tried that already and the answer is it doesn't work? Does setting container to a blank value on macOS error?

from homebrew-test-bot.

This is your problem:

Error: No such file or directory @ realpath_rec - /home/runner

Probably from:

https://github.com/Homebrew/brew/blob/07f14c32e9996196661bdc20830f48797079c463/Library/Homebrew/utils/github/actions.rb#L29

I suspected that too, but IIRC I've seen that error on CI runs well before I started using GitHub::Actions::Annotation.

from homebrew-test-bot.

I suspected that too, but IIRC I've seen that error on CI runs well before I started using GitHub::Actions::Annotation.

I'll need to see an example run of this to tell. I've not seen that error until recently and GITHUB_WORKSPACE being set to a directory that won't exist in the Docker container makes sense as a cause.

from homebrew-test-bot.

Does setting container to a blank value on macOS error?

Yep

from homebrew-test-bot.

Does setting container to a blank value on macOS error?

Yep

This works for me: https://github.com/Bo98/workflow-test/blob/main/.github/workflows/test.yml

from homebrew-test-bot.

Interesting. Looks like it was changed a year ago:
actions/runner#265

So I guess we can now drop a lot of code in our workflows and just use jobs.<id>.container, nice.

from homebrew-test-bot.

I'll need to see an example run of this to tell.

Here are some examples: https://github.com/Homebrew/homebrew-core/runs/4120630533?check_suite_focus=true#step:7:33

https://github.com/Homebrew/homebrew-core/runs/4160230870?check_suite_focus=true#step:7:195

https://github.com/Homebrew/homebrew-core/runs/3638699455?check_suite_focus=true#step:7:33

The ENOENT seems to happen on an audit failure, which probably means that this is triggered by the annotations code.

On the other hand, it also didn't kill the test-bot run, unlike what's happening now. But I suppose that’ll be because the error is happening behind a test call in those examples. Now the error is in test-bot code.

from homebrew-test-bot.