klemens-morgenstern / metal.test Goto Github PK

View Code? Open in Web Editor NEW

41.0 10.0 4.0 313 KB

Deprecated, superseded by https://github.com/metal-ci/test

Home Page: https://github.com/metal-ci/test

License: GNU Lesser General Public License v3.0

C++ 86.54% C 8.51% Python 2.79% CMake 2.16%

metal debugger light-weight-testing openocd testing gdb calltrace coverage arm arduino bare-metal

metal.test's Introduction

About

This framework provides facilities for automated execution of remote code. The core idea is, to utilize the debugger for automated testing, so that it's featured can be used to obtain information.

At the current state it provides functionality for:

I/O Forwarding
Code Coverage
Unit testing
Call tracing
Profiling
Function Stubbing at link-time

This is based on plugin-system, so that it can be extended with functionality for your system. The above features are also done by plugins that are provided by the framework. Since it is built do use the debugger, it can be used for every case a debugger can, which includes the following applications:

local execution
remote server (gdbsever)
openocd
qemu

When used with the right plugins, an application run via openocd or qemu behaves as if it were run locally and can be integrated into any automated testing.

It currently only supports the gdb, but lldb support is planned for a future version.

Tool Overview

metal.runner

The debug-runner is the core module of the toolest, it allows automated execution of the debugger. This can be used with plugins, so that custom functionality can be added to breakpoints. The main application is for the automated executing tests on remote targets, such as embedded platforms. It comes with two plugins for embedded targets:

I/O (similar to semi-hosting)
Exit-Code

When used with these two plugins an embedded application can be run on target, but behaves as if it were run locally. In addition calltrace & backend provide part of the functionality as part as metal.runner plugins.

metal.unit

We provide a very light weight test backend, for easy use with our metal.runner. When used without the debugger, it will only yield a binary result, while it will provide very detailed information through the debugger. This means that it can be used with the same assertions

metal.calltrace

The calltrace provides a way to assert a certain call sequence for functions. Combined with the metal.runner it can be used to log function calls and add profiling.

metal.serial

The serial library provides a light-weight testing tool for environments that do not provide access for a debugger. It outputs the minimum amount of data from the target device, reads the rest from the source files and provides more detailed report that way. E.g. it provides a printf-alternative that does the actual formatting on the host side, thus providing formatted output without required heap usage on the target.

Documentation

The current master Documentation can be found in the wiki.

Test results

Branches	Build	Tests coverage
Develop:
Master:

Dependency

This library requires boost 1.64.

metal.test's People

Contributors

Stargazers

Watchers

Forkers

integral-io chemingway krish2487 duanluong

metal.test's Issues

Licence?

Excellent looking framework! Are you able to add a licence file so people can start using it?

Thanks :-)

metal.test code location

To provide a very light-weight way to run tests without the debugger, we would want to provide an address location on the lowest level to reproduce where the error occured. Idea: put a label into the function and reproduce the address location like this: (arm)

void sink(void * location, bool condition);

#define COUNTER() __COUNTER__
#define IMPL(Condition, CNT, TYPE)  \
        asm("METAL_SERIAL_" #TYPE "_" #CNT ); \
        extern void* METAL_SERIAL_ ## TYPE ## _ ## CNT; \
        sink(&METAL_SERIAL_ ## TYPE ## _ ## CNT, Condition);

#define IMPL2(Condition, CNT, Type) IMPL(Condition, CNT, Type)
#define METAL_ASSERT(Condition) IMPL2(Condition, __COUNTER__, ASSERT)

void foo()
{
    METAL_ASSERT(true);
    METAL_ASSERT(42);
}

If that works, we output the location of the symbol through the test sink and can then reproduce where the test must've been in the binary.

Union breakpoint

For smaller devices that only provide two breakpoints the current approach might be problematic, because when using the test-lib, the _exit-code breakpoint and the syscalls (for gcov) in conjunction we already need three. There should be a way to remedy that and reduce them to one. The most obvious way is to add a METAL_BREAKPOINT macro. When compiled in unified mode (or whatever I'll call that) it enters a call to metal_trigger_breakpoint which exists ONCE in the binary and which contains the breakpoint. Then the debugger calls step-return and calls the breakpoint it's in.

Licence?

Excellent looking framework! Are you able to add a LICENCE file so people can start using it in their own projects.

Thanks 😄

Deprecation notice - see https://github.com/metal-ci/test

After major refactoring, the new version will be found at https://github.com/metal-ci/test

Licence?

Excellent looking library! Can you please add a licence file so people can start using it in their own projects?

Thanks

Reduce verbosity of python version?

Hi,

I've come back to this after working on some other stuff, and I have now ported a lot of my tests over to use metal.test, and I like how quick and easy it makes the edit/test cycle. However, is there any way to turn off printing every comparison made? If I am running a comparison in a loop, the screen rapidly fills with lines. I had a quick look in metal-unit.py, but couldn't see a parameter for this.

I suspect this also slows test execution down, as a breakpoint is hit every time a METAL_ASSERT macro is used? If the breakpoint could be disabled unless an error occured, I think this would speed things up.

...Timeout... error when using JLinkClGDB.exe

Hi,

Thanks for this tool, I think it fills quite a unique space and I enjoy how standalone it is. I am trying to use it with a JLink using the JLink GDB Server instead of OpenOCD, but although it runs the server sucessfully and seems to connect to the port, I get a timeout error from metal.runner.

As far as I can see from the JLink log, the code has flashed correctly, and the breakpoint in metal_func_stub in metal_newlib_syscalls.c is hit. Grepping the sources, the string ...Timeout... is from metal.runner

My command line is
metal.runner --config-file="../test/metal_runner.cfg" --init-script="../test/test_script_gdb" --exe=bin/executable_unittest.elf

metal_runner.log

Starting run

=thread-group-added,id="i1"~"GNU gdb (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 8.0.50.20171128-git\n"~"Copyright (C) 2017 Free Software Foundation, Inc.\n"~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\nand \"show warranty\" for details.\n"~"This GDB was configured as \"--host=x86_64-linux-gnu --target=arm-none-eabi\".\nType \"show configuration\" for configuration details."~"\nFor bug reporting instructions, please see:\n"~"<http://www.gnu.org/software/gdb/bugs/>.\n"~"Find the GDB manual and other documentation resources online at:\n<http://www.gnu.org/software/gdb/documentation/>.\n"~"For help, type \"help\".\n"~"Type \"apropos word\" to search for commands related to \"word\"...\n"~"Reading symbols from bin/executable_unittest.elf..."~"done.\n"GDB Version "GNU gdb (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 8.0.50.20171128"
GDB Toolset ""
Config      "This GDB was configured as "--host=x86_64-linux-gnu --target=arm-none-eabi""
0-target-select remote localhost:2331
=thread-group-started,id="i1",pid="42000"=thread-created,id="1",group-id="i1"~"main () at ../test/test_nand_flash.c:25\n"~"25\t    system_init();\n"*stopped,frame={addr="0x0000d70e",func="main",args=[],file="../test/test_nand_flash.c",fullname="/mnt/c/Users/chris/repos/apollo3_spi_nand/test/test_nand_flash.c",line="25"},thread-id="1",stopped-threads="all"0^connected1-interpreter-exec console "monitor reset"
@"Resetting target\r\n"Resetting target\r\n1^done2-interpreter-exec console "monitor halt"
2^done3-interpreter-exec console load
~"Loading section .text, size 0x3a94 lma 0xc000\n"3+download,{section=".text",section-size="14996",total-size="659923"}3+download,{section=".text",section-sent="14996",section-size="14996",total-sent="14996",total-size="659923"}~"Loading section .ARM.exidx, size 0x8 lma 0xfa94\n"3+download,{section=".ARM.exidx",section-size="8",total-size="659923"}~"Loading section .data, size 0xa8 lma 0xfa9c\n"3+download,{section=".data",section-size="168",total-size="659923"}~"Loading section .heap, size 0xc00 lma 0xfb44\n"3+download,{section=".heap",section-size="3072",total-size="659923"}~"Loading section .stack, size 0x800 lma 0xfb44\n"3+download,{section=".stack",section-size="2048",total-size="659923"}~"Start address 0xc000, load size 20292\n"~"Transfer rate: 9908 KB/sec, 4058 bytes/write.\n"3^done4-interpreter-exec console "monitor reset"
@"Resetting target\r\n"Resetting target\r\n4^done5-interpreter-exec console "monitor halt"
5^done6-interpreter-exec console "monitor halt"
6^done7-interpreter-exec console continue
~"Continuing.\n"7^running*running,thread-id="all"...Timeout...
Exited with code: -1

metal_runner_jlink.log

[Linked as very long]

metal_runner.cfg

dbg=arm-none-eabi-gdb
other="/mnt/c/Program Files (x86)/SEGGER/Jlink/JLinkGDBServerCL.exe" -If SWD -Speed 4000 -Device AMA3B1KK-KBR
remote=localhost:2331
timeout=10
log=metal_runner.log
debug=true
other-log="metal_runner_jlink.log"
lib="metal-newlib-syscalls metal-unit metal-exitcode"

test_script_gdb

monitor reset
monitor halt
load
monitor reset
monitor halt
continue

klemens-morgenstern / metal.test Goto Github PK

metal.test's Introduction

About

Tool Overview

Documentation

Test results

Dependency

metal.test's People

Contributors

Stargazers

Watchers

Forkers

metal.test's Issues

metal_runner.log

metal_runner.cfg

test_script_gdb

Recommend Projects

Recommend Topics

Recommend Org