Question regarding tests about mir HOT 15 CLOSED

One of the approaches I used for dmrC was to source tests from various places and adapt them. The issue of course is sometimes the tests have their own licences. Do you care about that?

Thank you for the proposal. Yes, I am interesting in C tests and I believe your experience will be useful to find such tests.

Currently, I am using some tests from GCC testsuite. They are small and currently very useful to find and debug C2MIR translator. Bigger tests are not useful for me right now as some regular C11 features are not implemented yet and there are still a lot of bugs in the translator/MIR-interpreter/MIR-generator.

When I succeed with the small tests, I'll need bigger ones (probably in 1-2 months). So I am open to any tests proposals.

The license issue can be tricky one.

It would be nice to have also tests which check C11 conformance and contains wrong C code too. Unfortunately, such conformant tests (like perennial/plumhall) are not free.

from mir.

Perhaps I should do this in a separate repo so that there is no pollution re the license.

from mir.

Hi, I started this at:

https://github.com/dibyendumajumdar/mir-testing

It is on a branch called tests.

I copied across the dmrC tests which are in turn sourced from various places.
Some are small tests and some are bigger.

It should be easy for you to run the tests if you would like to: instructions are in README.

Regards

from mir.

I added more tests from other sources. A key criteria is that the test must be runnable. So every test has to have a main function. The test success is determined by using a combination of return code from the test and any stdout output.

from mir.

Here are some stats:

Total tests run 447
Successful tests 230

Not bad

from mir.

Thank you. I will look at this later when I resume my work on the project.

from mir.

@vnmakarov I merged the latest updates. The current results are:

Total tests run 447
Successful tests 337

Pretty good improvement!
Thanks

from mir.

Total tests run 447
Successful tests 337

Pretty good improvement!

Thanks for the tests. I worked on them and they are pretty useful.
Although I found some troubles with them:

• some tests (from TESTED) check errors and are supposed to be failed
• some tests expect wrong output, gcc and clang fail on them too
• some tests expect specific undefined C behaviour
• some tests check GCC extensions (binary constants, case ranges, extended ternary operators etc)

But in general they are useful and I will add them to repository after their analysis.

Meanwhile I've just added about 600 GCC tests I am using myself. They can be called by make c2mir-full-test.

from mir.

Although I found some troubles with them:

* some tests (from  TESTED) check errors and are supposed to be failed

* some  tests expect wrong output, gcc and clang fail on them too

* some tests expect specific undefined C behaviour

* some tests check GCC extensions (binary constants, case ranges, extended ternary operators etc)

Yes perhaps as I have not checked all of the failures, and also many tests I just picked from other projects. The ones I use for dmrC are a smaller subset; those are included as well of course.

But in general they are useful and I will add them to repository after their analysis.

Meanwhile I've just added about 600 GCC tests I am using myself. They can be called by make c2mir-full-test.

Cool. I hope this won't cause a license issue.
Are all these tests passing?

I am very impressed by your project and am eagerly waiting to start using it.

from mir.

So looks like passes those 655 gcc tests! Wonderful!

from mir.

So looks like passes those 655 gcc tests! Wonderful!

They are only tests which are passing but there are a few tests from gcc I use which are not passing. I will add them when they start passing. It is easier this way not to introduce new regressions.

from mir.

Cool. I hope this won't cause a license issue.

I am not a lawyer and can not give advices to other people but for me it is obvious that it does not violate the GPL license (and its spirit).

from mir.

Hi, I have deleted the mir-testing repo because I had trouble merging changes from your repo. In any case looks like you are adding the tests to mir so no point having the testing repo.

from mir.

Hi, I have deleted the mir-testing repo because I had trouble merging changes from your repo. In any case looks like you are adding the tests to mir so no point having the testing repo.

Sorry for creating a conflict. I really appreciate your work and collecting a lot of C tests in mir-test repo. Mine tests from gcc-torture were not enough.

I found that Andrew Chambers and especially LACC tests have much better coverage of C standard. LACC tests revealed a lot of problems with LLVM to MIR translator. Basically, I did not implement composite data operators yet (insert and extract value) which are generated for small structures for some ABIs. Implementing this functionality will take some time. So currently I switched off LLVM2MIR translator testing in Makefile.
As for C2MIR compiler I've managed to fix all LACC tests. The next big step will be achieving a bootstrap for C2MIR.

from mir.

Hi, I have deleted the mir-testing repo because I had trouble merging changes from your repo. In any case looks like you are adding the tests to mir so no point having the testing repo.

Sorry for creating a conflict. I really appreciate your work and collecting a lot of C tests in mir-test repo. Mine tests from gcc-torture were not enough.

No worries I am glad that it was helpful.

I found that Andrew Chambers and especially LACC tests have much better coverage of C standard. LACC tests revealed a lot of problems with LLVM to MIR translator. Basically, I did not implement composite data operators yet (insert and extract value) which are generated for small structures for some ABIs. Implementing this functionality will take some time. So currently I switched off LLVM2MIR translator testing in Makefile.
As for C2MIR compiler I've managed to fix all LACC tests. The next big step will be achieving a bootstrap for C2MIR.

That's great news.

from mir.