malsyned / pfstest Goto Github PK

It would be nice if there was some way to do something like this:

const char *edge_names = {
    [EDGE_NONE] = "EDGE_NONE",
    [EDGE_RISING] = "EDGE_RISING",
    [EDGE_FALLING] = "EDGE_FALLING",
};
assert_that("", the_enum(edge_names, edge_get_edge(test_edge)), 
                is_the_enum(edge_names, EDGE_RISING));

and get output like this:

test_edge.c:should_detect_rising FAIL
    Location: test_edge.c:33
    Failed assertion
    Expected: the enum EDGE_RISING
    Actual:   the enum EDGE_NONE

• Nothing verifies the correct return values of print_char, print_nv_string
• No return value comes from the other primitive formatters
• Add test suites for standard output formatters?

(See Issue 3 regarding module naming)

A number of macros in pfstest-mock.h capture FILE (by way of PFSTEST_NV_FILE) and LINE. No self-test exists to make sure that these are captured and passed to pfstest_fail() correctly, which can happen (and the same bug did happen once in assert-that.h). A test exists in test-assert-that.c named should_print_explanation_on_failed_assertion, but I'm not wild about how this bakes in a lot of checks of the standard_reporter's output format. It'd be nicer to have the message_spy save these off somewhere for later comparison.

The colorizers are 2-element vtables. They're only taking up 8 bytes of RAM on AVR8 right now, but I try to keep every possible thing out of RAM.

For example, pfstest_matcher_print and pfstest_value_print, which take the reporter first and then the object.

Stop the proliferation of void do_nothing(void) {}

The AVR8 tools include a 4-byte array containing "EMTY" that is replicated across the span between the heap start and the stack. The template for it should be stored in program memory but isn't.

When gcc is called with -g3, it outputs #define lines in addition to # lines and C source code lines. pycparser's lexer does not handle #define lines at all. They would need to be stripped out like comments or handled in some similar way to make sure they are tolerated and ignored.

the_char should print, for example, newlines as '\n', but it prints them as '
' instead

Every test for failure in test-values-and-matchers.c assumes that assert_that outputs text of the form

Failed assertion
Expected: the short -32768
Actual:   the short 32767

which means that the text "Failed assertion", "Expected: ", and "Actual: " appears in a string in about half of the tests in values-and-matchers.c. This tight coupling is against the spirit of testing separate units separately.

Since test-assert-that.c already verifies that assert_that works by invoking pfstest_matcher_matches(), pfstest_matcher_print(), and pfstest_value_print(), we can use these API calls directly in test-values-and-matchers.c to make these tests cleaner. For example:

test(the_short_should_print_itself)
{
    const pfstest_pg_ptr char *expected =
        pfstest_pg_str("the short 32767");

    pfstest_value_print(message_spy, the_short(32767));

    assert_that("shorts print themselves",
                the_string(captured_output), matches_the_pg_string(expected));
}

test(the_short_should_print_negative_shorts)
{
    const pfstest_pg_ptr char *expected =
        pfstest_pg_str("the short -32768");

    pfstest_value_print(message_spy, the_short(-32768));

    assert_that("negative shorts print themselves",
                the_string(captured_output), matches_the_pg_string(expected));
}

test(equal_shorts_should_pass)
{
    assert_true("is_the_short matches equal shorts",
                the_bool(pfstest_matcher_matches(is(the_short(32767)),
                                                 the_short(32767))));
}

test(unequal_shorts_should_fail)
{
    assert_false("is_the_short fails unequal shorts",
                 the_bool(pfstest_matcher_matches(is(the_short(32767)),
                                                  the_short(-32768))));
}

The current implementation of the array-backed allocator was developed for PIC18 and has only been tested on that platform. Make it a generic plugin and test it on x86-64 and AVR GCC at least.

• Modeled after my Unity/CMock one
• Make it faster by allowing some test binaries to share object files?
• Fix limitations of mock generation that were forced by CMock's behavior?

My best guess is that the way to integrate PFSTest into an IDE is to create a solution with one project file per test binary, plus a project file that creates a library out of the files in pfstest/src. I haven't learned how to do such a thing with Visual Studio projects and solutions yet.

Another possibility might be to provide some tools for automatically generating a Visual Studio project or solution for each desired test binary. They are, after all, plain text files.

it would also be advantageous to have some way to compile and run all of the test binaries. Possibly a simple batch script or PowerShell script that invokes Visual Studio's command line arguments for automated building.

Use ANSI color escapes to call attention to passes, failures, ignores, and run result.

Mostly this is about floating point matching and one-dimensional arrays for built-in types, I think.

I'm finding it cumbersome to write
assert_that("", the_bool(...), is_the_bool(true));
I would prefer
assert_true("", the_bool(...))

• It's shorter
• It more accurately reflects the fact that they also handle statistic gathering (failure count, etc)

Parse C headers and output functions that call pfstest_mock_invoke with the appropriate arguments.

the_enum only supports enumerations whose values start at 0 and are all grouped together in a contiguous range. It's possible that is all I want to support through that convenience value, and more complex enums might be the province of custom values and matchers, but if there is a nice-looking way to support noncontiguous enums, I'm open to having it included in the project.

Any errors that happen in automock.py will result in Python stack traces being shown, which is not ideal.

Most of the patterns in in the target-template will actually match all targets with the same class (actually buildprefix). It would make this distinction clearer if a separate template was created for classes and run on each class ( $(sort $(foreach target,$(TARGETS),$(call target-class,$(target)))) ) instead of on each target.

In the outermost tag.

I changed file filtering to operate only on the basname of files because Visual C++ was using different paths to access those files than my GCC Makefile was. However, I've recently moved the VC++ solution such that this may no longer be an issue, and I was uncomfortable with the basenaming to begin with.

it is legal in C to declare the same function twice, and at least stdio.h does so for vscanf and others. automock.py should generate only header line for each function, regardless of how many times they are declared.

Integer output requires uintptr_t, intmax_t, uintmax_t and ULONG_MAX to be defined, and platform-c89 gets these by including <stdint.h>, which is a C99 header.

the_string and the_memory copy the buffer that is passed to them into a freshly allocated new buffer. Why have I done this? I think only to support this use pattern:

static char *make_me_a_string(void)
{
    char stack_string[] = "foo";
    return stack_string;
}

test(foo) {
    assert_that(the_string(make_me_a_string()), is_the_string("foo"));
}

However, is this pattern really likely? And if it is, wouldn't the user be better advised to pfstest_alloc the buffer requiring dynamic extent themselves?

It isn't useful to have multiple before() and after() functions defined in a single file. The order they are executed can't be guaranteed, and having to name them, and name them something different in every file since they're external identifiers, is cumbersome and inconsistent with how other unit test frameworks operate.

Replace this with simple setup() and teardown() functions which are defined statically and which each of the tests maintain a pointer to so the core can access them.

A function pointer is not a pointer to an object, and so isn't guaranteed to be convertable to a void* and back again without loss of information. automock should notice pointers to functions and treat them as opaque data instead of pointers.

Bring all identifiers in line with https://www.gnu.org/software/libc/manual/html_node/Reserved-Names.html

Embedded devices, communicating results over a serial port, may only wish to report in a machine-readable form. The standard user-centric terminal output formatters would be dead-weight in that case, potentially taking up ROM space without adding value.

For example, many tests in test-xml.c use is(the_pointer(NULL)) or is(the_bool(pfstest_true))

Calls to the pfstest-mock API are all over the core, which feels like a layering violation to me. Add a simple plugin API to pfstest-core that pfstest-mock can be registered with to break the cycle. Maybe a simple pfstest_list_t of plugins.

It would be nice if freeing memory notified you that the buffer had been overrun. It would be even nicer if it provided the file and line where the memory was allocated. Alternatively, a quick introduction to Valgrind could be provided.

These functions aren't part of core functionality and might be useless or in the way on non-hosted C environments: pfstest_main, pfstest_arguments_parse, pfstest_print_usage, pfstest_start, stdout_print_char.

I believe this enables removing fflush from the platform header.

Now that mocks are generated on a per-class basis, there might be use cases for per-class MOCK_CPPFLAGS and AUTOMOCK_ARGS.

(also, shouldn't it be AUTOMOCK_CPPFLAGS?)

Since it is not possible to provide information of what mock invocations occurred during the verification phase, it would be very useful as a command line option to print every invocation as it occurs, so that the list can be visually checked against what is expected in the event of a surprise failure.

Sometimes, it's unavoidable that a mock needs to have side-effects.

I think a pretty light-weight and satisfying solution would be something like this:

void my_callback(void *);
some_type *my_callback_data;
do_side_effect(my_callback, my_callback_data, some_expectation);

Where a side-effect function is a void func(void *). In this design, the side-effect function wouldn't get access to any of the arguments to the call it is side-effecting, but it could use the void* to hold a counter if it needs to have a different side-effect on different calls.

I don't know quite how I would want to handle mocking of varargs yet, but I suspect it would be possible. Perhaps a varargs combinator:

when(mock_func1, arg_that(is_the_int(5)), varargs_that(is_the_string("foo"), is_the_int(4), NULL));

Plus some way to package varargs from a mock.

va_start(__pfstest_ap, __pfstest_mock_arg_0);
[...]
the_varargs(__pfstest_ap)
[...]
va_end(__pfstest_ap);

I haven't tried these or thought through whether they are actually implementable.

Should be [-v|-x] but doesn't list the -x

In the simulator, if the two tests in tests\test-values-and-matchers.c dealing with int_arrays are not registered, the self-test suite passes on PIC18. If they are enabled, though, stack weirdness happens and the program restarts unexpectedly.

This 33-byte table could and should be stored in program memory.

Both for end users and for my education, put guard bytes at the top of the stack and verify that they're still there at the end of testing.

If matches_the_pg_string_test() was rewritten to simply "return true", the test suite would still pass and many of the tests in the self-test suite could be in error. While it is easy to catch errors in this test while adding new self-tests (greens would show up when the developer is expecting reds), refactoring in pfstest-matchers.c could result in this function breaking silently until the problem is discovered while adding new self-tests, if ever.

The project files for Visual Studio and MPLAB 8 are often falling behind in their file lists compared to the GNU Makefile, which is more frequently tested. This could be guarded against by automatically generating them from the source lists stored in src/src.mk and tests/selftestsrc.mk.

I haven't checked out what issues might exist in supporting const qualified types. Ones I can think of right now include:

• Consistent, useful boxing conventions for values (indirect once and store as pointer-to-const-pointer?)
• Accepting pointers to them as arguments to mocks
• Matching them against other const data (the_const_memory, is_the_const_memory)
• Matching non-const data against them (the_memory, matches_the_const_memory)
• Matching them against non-const data? Is this even useful? (the_const_memory, matches_the_memory)
• Providing generic equality matcher functions like is_the_whatever_matcher

Hamcrest has the following matchers that I may want to draw inspiration from. Note that there is some overlap here with Unity's matchers.

array containing an element
array containing certain elements in order
object whose value is in an array (thing that an any() combinator can do this)
allOf()
anyOf()
Attach a different message string
every item in an array matches one matcher
is anything
is equal (this is the is_the_ suite)
is null (is_the_pointer(NULL) -- but maybe a convenience wrapper?)
stringContains
stringEndsWith
stringStartsWith
substring
isCloseTo
isNaN
comparesEqualTo
greaterThan
greaterThanOrEqualTo
lsssThan
lessThanOrEqualTo
isBlankString (only whitespace) }
isEmptyString ("") }-- these two also match NULL
isEqualIgnoringCase
isEqualIgnoringWhitespace
matchesPattern (regex)
stringContainsInOrder

Is the number of failures a poor choice for the return value? Aren't return values > 128 reserved, and > 255 not necessarily supported? What do Unity, jUnit, Python unittest, etc. return on failure?

pycparserext's setup.py indicates that it is MIT licensed.