Code Monkey home page Code Monkey logo

typeart's Introduction

TypeART · License Coverage Status

What is TypeART?

TypeART [TA18; TA20] is a type and memory allocation tracking sanitizer based on the LLVM compiler toolchain for C/C++ (OpenMP) codes. It consists of an LLVM compiler pass plugin for instrumentation, and a corresponding runtime to track memory allocations during the execution of a target program.

TypeART instruments heap, stack and global variable allocations with a callback to our runtime. The callback consists of (1) the memory address, (2) the type-layout information of the allocation (built-ins, user-defined structs etc.) and (3) number of elements.

Why use it?

TypeART provides type-related information of allocations in your program to help verify some property, and to help generate diagnostics if it doesn't hold.

For instance, low-level C-language APIs use void-pointers as generic types. Often, the user must specify its type and length manually. This can be error-prone. Examples for type unsafe APIs include the Message-Passing Interface (MPI), checkpointing libraries and numeric solver libraries. With TypeART, it is straightforward to verify that a void -pointer argument to an API is, e.g., a type T array with length n.

Use Case: MUST - A dynamic MPI correctness checker

MUST [MU13] is a dynamic MPI correctness checker to, e.g., detect deadlocks or a mismatch of MPI datatypes of the sending and receiving process, see its project page.

MUST relies on intercepting MPI calls for its analysis. As a consequence, though, MUST is unaware of the effective type of the allocated void* buffers used for the low-level MPI API. To that end, TypeART was developed to track memory (de-)allocation relevant to MPI communication. With TypeART, MUST can check for type compatibility between the type-less MPI communication buffer and the declared MPI datatype.

Type checking for MPI calls

Consider the MPI function MPI_Send(const void* buffer, int count, MPI_Datatype datatype, ...). Without TypeART, MUST cannot check (1) if the buffer argument is compatible with the declared MPI_Dataype and (2) if the count argument exceeds the buffer allocation size:

// TypeART tracks this allocation (memory address, type and size):
double* array = (double*) malloc(length*sizeof(double));
// MUST intercepts this MPI call, and asks TypeARTs runtime library for type information:
//   1. Is the first argument of type double (due to MPI_DOUBLE)?
//   2. Is the allocation at least of size *length*? 
MPI_Send((void*) array, length, MPI_DOUBLE, ...)

MUST and TypeART also handle MPI derived datatypes with complex underlying data structures, see our MPI Demo. For more details, see our publications, or download the current release of MUST (1.8 or higher has TypeART integrated) on its project page.

Table of Contents

1. Using TypeART

Making use of TypeART consists of two phases:

  1. Compile your code with Clang/LLVM (version >= 10) using the TypeART LLVM pass plugins to (1) serialize static type information to a file and (2) instrument all relevant allocations. See Section 1.1.
  2. Execute the target program with a runtime library (a client based on the TypeART runtime) to accept the callbacks to do some useful analysis with our interface based on the static type information. See Section 1.2.

1.1 Compiling a target code

Our LLVM compiler pass plugins instrument allocations and also serialize the static type layouts of these allocations to a yaml file (default name types.yaml). To that end, we provide compiler wrapper scripts around Clang and MPI to apply TypeART in the bin folder of the TypeART installation prefix. By default, the wrappers instrument heap, stack and global allocations. The MPI-wrappers also filter allocations that are not passed to an MPI call, see Section 1.1.4.

Note: Currently, the compilation process has to be serialied, e.g., make -j 1, due to extraction and consistency of type information per translation unit.

1.1.1 Building with TypeART

A typical compile invocation may first compile code to object files and then link with any libraries:

# Compile:
$> clang++ -O2 $(COMPILE_FLAGS) -c code.cpp -o code.o
# Link:
$> clang++ $(LINK_FLAGS) code.o -o binary

With TypeART, the recipe needs to be changed to, e.g., use our provided compiler wrapper, as we rely on the LLVM opt (optimizer) tool to load and apply our TypeART passes to a target code:

# Compile, replace direct clang++ call with wrapper of the TypeART installation:
$> typeart-clang++ -O2 $(COMPILE_FLAGS) -c code.cpp -o code.o
# Link, also with the wrapper:
$> typeart-clang++ $(LINK_FLAGS) code.o -o binary

In particular, the wrapper script does the following:

  1. Compile the code down to LLVM IR, and pipe the output to the LLVM opt tool. (Keeping your original compile flags)
  2. Apply heap instrumentation with TypeART through opt.
  3. Optimize the code with your -Ox flag using opt.
  4. Apply stack and global instrumentation with TypeART through opt.
  5. Pipe the final output to LLVM llc to generate the final object file.
  6. Finally, link the TypeART runtime library using your linker flags.

Note: We instrument heap allocations before any optimization, as the compiler may throw out type information of these allocations (for optimization reasons).

Wrapper usage in CMake build systems

For plain Makefiles, the wrapper replaces the GCC/Clang compiler variables, e.g., CC or MPICC. For CMake, during the configuration, it is advised to disable the wrapper temporarily. This is due to CMake executing internal compiler checks, where we do not need TypeART instrumentation:

# Temporarily disable wrapper with environment flag TYPEART_WRAPPER=OFF for configuration:
$> TYPEART_WRAPPER=OFF cmake -B build -DCMAKE_C_COMPILER=*TypeART bin*/typeart-clang 
# Compile with TypeART now:
$> cmake --build build --target install
MPI wrapper generation

For MPI, we rely on detecting the vendor to generate wrappers with appropriate environment variables to force the use of the Clang/LLVM compiler. We support detection for OpenMPI, Intel MPI and MPICH based on mpi.h symbols, and use the following flags for setting the Clang compiler:

Vendor Symbol C compiler env. var C++ compiler env. var
Open MPI OPEN_MPI OMPI_CC OMPI_CXX
Intel MPI I_MPI_VERSION I_MPI_CC I_MPI_CXX
MPICH MPICH_NAME MPICH_CC MPICH_CXX
Internal wrapper invocation

For reference, the wrapper script executes the following (pseudo):

# Compile: 1.Code-To-LLVM | 2.TypeART_HEAP | 3.Optimize | 4.TypeART_Stack | 5.Object-file 
$> clang++ $(COMPILE_FLAGS) $(EMIT_LLVM_IR_FLAGS) code.cpp | opt $(TYPEART_PLUGIN) $(HEAP_ONLY_FLAGS) | opt -O2 -S | opt $(TYPEART_PLUGIN) $(STACK_ONLY_FLAGS) | llc $(TO_OBJECT_FILE)
# Link:
$> clang++ $(LINK_FLAGS) -L$(TYPEART_LIBPATH) -ltypeartRuntime code.o -o binary

1.1.2 Options for TypeART passes

For modification of the pass behavior, we provide several options.

Flag Default Description
typeart - Invoke TypeART pass through LLVM opt
typeart-types types.yaml Serialized type layout information of user-defined types. File location and name can also be controlled with the env variable TYPEART_TYPE_FILE.
typeart-heap true Instrument heap allocations
typeart-stack false Instrument stack and global allocations. Enables instrumentation of global allocations.
typeart-global false Instrument global allocations (see --typeart-stack).
typeart-stats false Show instrumentation statistic counters
typeart-call-filter false Filter stack and global allocations. See also Section 1.1.4
typeart-call-filter-str *MPI_* Filter string target (glob string)
typeart-filter-pointer-alloca true Filter stack alloca of pointers (typically generated by LLVM for references of stack vars)
Example invocations
Pre-requisites
  1. Loading TypeART plugins with opt:
    TYPEART_PLUGIN=-load $(PLUGIN_PATH)/typeartTransformPass.so
  2. Input of opt is LLVM IR, e.g.:
    # Pipe LLVM IR to console:
    $> clang++ -O1 -g -Xclang -disable-llvm-passes -S -emit-llvm -o - example.cpp
Examples
  • Heap-only instrumentation (with stats):
    opt $(TYPEART_PLUGIN) -typeart -typeart-stats
  • Stack- and global-only instrumentation (no stats):
    opt $(TYPEART_PLUGIN) -typeart -typeart-heap=true -typeart-stack
  • Stack- and global-only instrumentation (with default filtering for MPI):
    opt $(TYPEART_PLUGIN) -typeart -typeart-heap=false -typeart-stack -typeart-call-filter
  • Filtering w.r.t. non-standard target API:
    opt $(TYPEART_PLUGIN) -typeart -typeart-heap=false -typeart-stack -typeart-call-filter -typeart-call-filter-str=MY_API*
  • Combined instrumentation (with filtering):
    opt $(TYPEART_PLUGIN) -typeart -typeart-stack -typeart-call-filter

1.1.3 Serialized type information

After instrumentation, the file types.yaml contains the static type information. Each user-defined type layout is extracted and an integer type-id is attached to it. Built-in types (e.g., float) have pre-defined ids and byte layouts.

The TypeART instrumentation callbacks use the type-id. The runtime library correlates the allocation with the respective type (and layout) during execution. Consider the following struct:

struct s1_t {
  char a[3];
  struct s1_t* b;
}

The TypeART pass may write a types.yaml file with the following content:

- id: 256            // struct type-id
  name: struct.s1_t
  extent: 16         // byte size
  member_count: 2
  offsets: [ 0, 8 ]  // byte offsets from struct start
  types: [ 0, 10 ]   // member type-ids (0->char, 10->pointer)
  sizes: [ 3, 1 ]    // member (array) length

1.1.4 Filtering allocations

To improve performance, a translation unit-local (TU) data-flow filter for global and stack variables exist. It follows the LLVM IR use-def chain. If the allocation provably never reaches the target API, it can be filtered. Otherwise, it is instrumented.

Consider the following example.

extern foo_bar(float*); // No definition in the TU 
void bar(float* x, float* y) {
  *x = 2.f; // x is not used after
  MPI_Send(y, ...);
}
void foo() {
  float a = 1.f, b = 2.f, c = 3.f;
  bar(&a, &b);
  foo_bar(&c);
}
  1. The filter can remove a, as the aliasing pointer x is never part of an MPI call.
  2. b is instrumented as the aliasing pointer y is part of an MPI call.
  3. c is instrumented as we cannot reason about the body of foo_bar.

1.2 Executing an instrumented target code

To execute the instrumented code, the TypeART runtime library (or a derivative) has to be loaded to accept the callbacks. The library also requires access to the types.yaml file to correlate the type-id with the actual type layouts. To specify its path, you can use the environment variable TYPEART_TYPE_FILE, e.g.:

$> export TYPEART_TYPE_FILE=/shared/types.yaml
# If the TypeART runtime is not resolved, LD_LIBRARY_PATH is set:
$> env LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(TYPEART_LIBPATH) ./binary

An example for pre-loading a TypeART-based library in the context of MPI is found in the demo, see Section 1.3.

1.3 Example: MPI demo

The folder demo contains an example of MPI-related type errors that can be detected using TypeART. The code is compiled with our instrumentation, and executed by preloading the MPI-related check library implemented in tool.c. The check library uses the TypeART runtime query interface. It overloads the required MPI calls and checks that the passed void* buffer is correct w.r.t. the MPI derived datatype.

To compile and run the demo targets:

  • Makefile
    # Valid MPI demo:
    $> MPICC=*TypeART prefix*/bin/typeart-mpicc make run-demo
    # Type-error MPI demo:
    $> MPICC=*TypeART prefix*/bin/typeart-mpicc make run-demo_broken
  • CMake, likewise:
    $> TYPEART_WRAPPER=OFF cmake -S demo -B build_demo -DCMAKE_C_COMPILER=*TypeART prefix*/bin/typeart-mpicc 
    $> cmake --build build_demo --target run-demo
    $> cmake --build build_demo --target run-demo_broken

2. Building TypeART

TypeART requires LLVM version >= 10 and CMake version >= 3.20.

2.1 Optional software requirements

  • MPI library: (soft requirement) Needed for the MPI compiler wrappers, tests, the demo, our MPI interceptor library, and for logging with our TypeART runtime library within an MPI target application.
  • OpenMP-enabled Clang compiler: Needed for some tests.

Other smaller, external dependencies are defined within the externals folder (depending on configuration options), see Section 2.2.1 (Runtime). They are automatically downloaded during configuration time (internet connection required).

2.2 Building

TypeART uses CMake to build, cf. GitHub CI build file for a complete recipe to build. Example build recipe (debug build, installs to default prefix ${typeart_SOURCE_DIR}/install/typeart)

$> git clone https://github.com/tudasc/TypeART
$> cd TypeART
$> cmake -B build
$> cmake --build build --target install --parallel

2.2.1 CMake configuration: Options for users

Binaries (scripts)
Option Default Description
TYPEART_MPI_WRAPPER ON Install TypeART MPI wrapper (mpic, mpic++). Requires MPI.
Runtime
Option Default Description
TYPEART_ABSEIL ON Enable usage of btree-backed map of the Abseil project (LTS release) for storing allocation data.
TYPEART_PHMAP OFF Enable usage of a btree-backed map (alternative to Abseil).
TYPEART_SOFTCOUNTERS OFF Enable runtime tracking of #tracked addrs. / #distinct checks / etc.
TYPEART_LOG_LEVEL_RT 0 Granularity of runtime logger. 3 is most verbose, 0 is least.
Runtime thread-safety options

Default mode is to protect the global data structure with a (shared) mutex. Two main options exist:

Option Default Description
TYPEART_DISABLE_THREAD_SAFETY OFF Disable thread safety of runtime
TYPEART_SAFEPTR OFF Instead of a mutex, use a special data structure wrapper for concurrency, see object_threadsafe
LLVM passes
Option Default Description
TYPEART_SHOW_STATS ON Passes show compile-time summary w.r.t. allocations counts
TYPEART_MPI_INTERCEPT_LIB ON Library to intercept MPI calls by preloading and check whether TypeART tracks the buffer pointer
TYPEART_MPI_LOGGER ON Enable better logging support in MPI execution context
TYPEART_LOG_LEVEL 0 Granularity of pass logger. 3 is most verbose, 0 is least
Testing
Option Default Description
TYPEART_TEST_CONFIG OFF Enable testing, and set (force) logging levels to appropriate levels for test runner to succeed
TYPEART_CODE_COVERAGE OFF Enable code coverage statistics using LCOV 1.14 and genhtml (gcovr optional)
TYPEART_LLVM_CODE_COVERAGE OFF Enable llvm-cov code coverage statistics (llvm-cov and llvm-profdata required)
TYPEART_ASAN, TSAN, UBSAN OFF Enable Clang sanitizers (tsan is mutually exclusive w.r.t. ubsan and asan as they don't play well together)

References

[TA18] Hück, Alexander and Lehr, Jan-Patrick and Kreutzer, Sebastian and Protze, Joachim and Terboven, Christian and Bischof, Christian and Müller, Matthias S. Compiler-aided type tracking for correctness checking of MPI applications. In 2nd International Workshop on Software Correctness for HPC Applications (Correctness), pages 51–58. IEEE, 2018.
[TA20] Hück, Alexander and Protze, Joachim and Lehr, Jan-Patrick and Terboven, Christian and Bischof, Christian and Müller, Matthias S. Towards compiler-aided correctness checking of adjoint MPI applications. In 4th International Workshop on Software Correctness for HPC Applications (Correctness), pages 40–48. IEEE/ACM, 2020.
[MU13] Hilbrich, Tobias and Protze, Joachim and Schulz, Martin and de Supinski, Bronis R. and Müller, Matthias S. MPI Runtime Error Detection with MUST: Advances in Deadlock Detection. In Scientific Programming, vol. 21, no. 3-4, pages 109–121, 2013.

typeart's People

Contributors

ahueck avatar alexanderlinne avatar jplehr avatar jprotze avatar sebastiankreutzer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

typeart's Issues

Overflow in TypeResolution when calculating base type (undefined behavior)

Querying type info has an overflow of an unsigned type in getTypeInfo when resolving internal types of structs.

Trigger example

struct Datastruct {
  int start;
  double middle;
  float end;
};  
__typeart_alloc((const void*)&data, 257, 1);
  typeart_status status = typeart_get_type((const void*)&data.middle, &id_result, &count_check);

LLVM Type for leave_scope callback out of sync with CallbackInterface.h

LLVM pass uses i32 for the stack allocation counter.
However, CallbackInterface.h uses size_t: void __typeart_leave_scope(size_t alloca_count);

Defect

This is closely related to #40

#include <stddef.h>
void __typeart_leave_scope(size_t alloca_count);
int main(void) {
  __typeart_leave_scope(0);
  return 0;
}

Applying TypeART to this code (at no optim. settings), the pass crashes:

declare dso_local void @__typeart_leave_scope(i64) #1

Call parameter type does not match function signature!
  %__ta_counter_load = load i32, i32* %__ta_alloca_counter
 i64  call void @__typeart_leave_scope(i32 %__ta_counter_load)
in function main

🔬 Tracking issue for MPI type checking

The MPI interceptor library does not currently check the type and element count when checking a buffer used as an argument to a MPI call.

Steps

  • Research what MPI types exist and how they can be inspected
  • Implementation
    • Add type checking for builtin MPI types -> b8295b9
    • Add the ability to run tests using mpic++ as a compiler and mpirun and FileCheck for execution and verification -> 82c6a64, 2408cac
    • Add type checking for custom MPI types
    • Add a README explaining whether and how any given MPI function is typechecked

Unresolved Questions

Research

All MPI types that may be encountered can generally be divided into two categories: builtin MPI types and custom MPI types. Both are represented using a common C++ type MPI_Datatype. A list of builtin MPI types for C/C++ can be found here.
For checking a custom MPI type, the functions MPI_Type_get_envelope and MPI_Type_get_contents may be used. MPI_Type_get_envelope allows to retrieve the function which was used to create a given type and the number of arguments used while MPI_Type_get_contents allows to retrieve the actual arguments passed to the function.

CI-ext skip condition wrong

The commit message payload [ci skip] does not lead to skipping the CI extended test runner.

Fix

Change || to &&: "!contains(..., '[ci skip]') || !contains(..., '[ci ext skip]')"
to "!contains(..., '[ci skip]') && !contains(..., '[ci ext skip]')"

Generate scripts in build directory

The script templates "scrips/*sh.in" should stay in the build directory.
Multi-build projects would each overwrite the respective generated script, leading to unexpected side-effects during execution (wrong TA build executed).

Side effect

Need to look at lit test suite to use the updated script path.

Printing stats during runtime destruction can lead to weird bugs

Currently, the collected stats are serialized and printed when the programs exits, during the destruction of the runtime global.
However, when RuntimeSystem::~RuntimeSystem is called, the LLVM streams llvm::outs()/errs() have already been destroyed.
The logging system depends on these functions. If it is used at this point, the program crashes.

This can lead to problems in certain scenarios. For example, consider the 24_threads_type_check.cpp test.
Both the AccessRecorder and the test itself use atomics. Consequently, some of the functions implemented in the <atomic> are instrumented, if not filtered out.
During the serialization of the softcounters, one of these instrumented functions is called, which causes __typeart_on_alloc_stack to be invoked.
This in turn prompts an entry into the log, crashing the program.

For now, I'm adding a check that will stop the callback functions from being triggered from within the runtime's destructor.
In the future, we might want to think about avoiding this issue altogether, by printing the softcounter stats at an earlier time.
I can think of two possible options:

  1. Add an API call, such as typeart_finalize, which the application has to call explicitly.
  2. Automatically insert this call at the end of main (of course this will no work if the program returns via exit(0) or similar).

We may also choose to leave the code as is and be mindful of potential future issues.

GlobalValue filtering based on linkage rework

Filtering based on linkage, e.g., GlobalValue::ExternalLinkage, is too inclusive.

int i; // @i = dso_local global i32 0, align 4, !dbg !6 -> ExternalLinkage 
int ii = 0; // @ii = dso_local global i32 0, align 4, !dbg !0 -> ExternalLinkage
extern const int eci; //  @eci = external dso_local constant i32, align 4 -> ExternalLinkage
static int si; //  @si = internal global i32 0, align 4, !dbg !12

Tracking issue for compiler wrapper

Write a compiler wrapper that intercepts the compilation process, and transforms the compiler invocation to apply TypeART.
Targets are multi-file Makefile projects and CMake.

Usage example

Instead of using the Clang compiler directly, replace it with our wrappers:
cmake -DCMAKE_C_COMPILER=typeart-clang -DCMAKE_CXX_COMPILER=typeart-clang++ ...

TODO

  • Make scripts work in MPI context 303724e
  • Find required executables through CMake: clang/clang++/opt/llc 3a4c1d6
  • Disable script with (environment) flag during CMake config time eab5701
  • Test against typeart-bench (CI) codes
  • lit tests
  • wrapper need to respect sanitizer flag, if TypeART was build with them

Issues

CMake

  • mpicc-wrapper fails to set variables with CMakes find_config(MPI).
    message(STATUS "LIBS ${MPI_C_LIBRARIES}")
    message(STATUS ${MPI_C_COMPILE_OPTIONS})
    message(STATUS ${MPI_C_COMPILE_OPTIONS})
    message(STATUS ${MPI_C_LINK_FLAGS})
    message(STATUS ${MPIEXEC})  # this is set
    
    • This works instead: target_link_libraries(${target} PUBLIC MPI::MPI_C)

TypeART build using libc++

TypeART currently breaks when building with libc++.

Issues

  • Missing header unordered_map in IRPath.h and TypeARTFunctions.h
  • std::map internal type reference, see issue #58

TypeInterface and related refactoring

This is likely backward compatibiliy breaking change.

TypeInterface

The type id numbering should be reworked (enum typeart_builtin_type_t ) to handle new types (see also #62) and be robust w.r.t. code changes.

Where to put a new type?

Currently: TA_PTR is 10 and TA_NUM_VALID_IDS is 11.
Potentially:

  • TA_PTR is 254, TA_UNKNOWN_TYPE is 255 and User-def. types start at 256 (as before).
  • TA_NUM_VALID could be "concrete LLVM type" + 1 (e.g., TA_PPC_FP128 + 1 or TA_NEW_TYPE + 1)

Needs (some) backward compat. in case a new type is added.

TypeDB & TypeManager

Both dependent on TypeInterface refactoring.

TypeDB

  • BuiltinNames/Sizes should be generated (or checked) dependent on TypeInterface.
  • Query functions should be checked after refactoring.
  • What about isUserDefinedType w.r.t. TA_USER_DEF flag etc.?

TypeManager

  • Adapt getOrRegisterType, see also reserveNextId

🔬 Tracking issue for array cookie support

This is a tracking issue for the support of array cookies in TypeART.

Steps

  • Research when and how array cookies are added
  • Add test cases for cases where array cookies are expected to be added -> 65684d7, 22d8b4f
  • Implementation
    • Finding array cookies -> d40d567
    • Add array cookie data to existing data structures -> 7781f78
    • Consider the padding of array cookies when calculating array element counts -> 0817a3a
    • In case an array cookie is present, instrument the actual starting address of the array instead of the originally allocated address -> 650b1a3
    • Add instrumentation for array cookies

Unresolved Questions

  • Should array cookies be instrumented in any way and if so, how? -> for now, only to correctly calculate array sizes

Array Cookie research

Array cookies are a size_t value saving the allocated length of an array. These cookies are allocated under certain conditions when operator new is used to allocate an array.

=> When are array cookies created?

According to [1], an array cookie is not allocated if either of

  • "the element type T has a trivial destructor […] and the usual (array) deallocation function […] does not take two arguments" or
  • "the new operator being used is ::operator new [](size_t, void*)"

In Clang this is implemented in CGCXXABI::requiresArrayCookie using:

=> How are array cookies created?

According to [1]:

  • array cookies always have size sizeof(size_t)
  • if align is the maximum alignment of size_t and an element of the array and padding is the maximum of sizeof(size_t} and align bytes:
    • "The space allocated for the will be the space required by the array itself plus padding bytes"
    • "The cookie will be stored in the sizeof(size_t) bytes immediately preceding the array
      In Clang this is implemented in ItaniumCXXABI::InitializeArrayCookie.

[1] https://itanium-cxx-abi.github.io/cxx-abi/abi.html#array-cookies

Tracking issue for demo folder

The TypeART demo is not properly maintained currently.

Current status

  • Makefile broken w.r.t. compiling and deprecated MPI symbols
  • CMakeLists untested/out of date?
  • Out of date scripts

Issues for release 1.6

  • Update Makefile to compile examples with TypeART
  • Remove deprecated apply script
  • Fix deprecations for OpenMPI 4 or higher

Issues (future)

  • Update CMakeLists to build examples
  • Add demo to CI system
  • Add README.md for demo explanation

Directly calling callback functions causes a crash

A crash during the compilation is caused by explicit/direct calls to the various callback functions:

#include <stddef.h>
void __typeart_leave_scope(int alloca_count);
int main(void) {
  __typeart_leave_scope(0);
  return 0;
}

Potential fix

In TypeARTFunctions.cpp, check if the function is already declared in the module.

Rough fix outline
TAFunctionDeclarator::make_function(...)  {
  ...
  const auto do_make = [&](auto& name, auto f_type) {
    const bool has_f = m.getFunction(name) != nullptr;
    auto fc          = m.getOrInsertFunction(name, f_type);
    if (has_f) {
      auto fcc = fc.getCallee();
      return dyn_cast<Function>(fc.getCallee()->stripPointerCasts());
    }
    ...

Scripts require better command line arg parsing

Currently, optimization flags are passed as third argument in run.sh and apply.sh, which requires passing a placeholder for $ta_more_args:

target=$1
ta_more_args=${2:-""}
optimize=${3:-" "}

Must be called with run.sh file.c ' ' -O3.

Fix

Use proper parsing code to detect if we pass an optim flag at any point -O1 ... -O3

Establish consistent code style

Currently, the code style is an arbitrary mix of camel case and snake case.

General

Apply clang-tidy to enforce proper naming conventions, see clang-tidy identifier naming.

Style

A good style seems to be Googles configuration (with some modifications), see Google cloud cpp clang-tidy:

  - { key: readability-identifier-naming.NamespaceCase,          value: lower_case }
  - { key: readability-identifier-naming.ClassCase,              value: CamelCase  }
  - { key: readability-identifier-naming.StructCase,             value: CamelCase  }
  - { key: readability-identifier-naming.TemplateParameterCase,  value: CamelCase  }
  - { key: readability-identifier-naming.FunctionCase,           value: aNy_CasE  }
  - { key: readability-identifier-naming.VariableCase,           value: lower_case }
  - { key: readability-identifier-naming.ClassMemberCase,        value: lower_case }
  - { key: readability-identifier-naming.ClassMemberSuffix,      value: _          }
  - { key: readability-identifier-naming.PrivateMemberSuffix,    value: _          }
  - { key: readability-identifier-naming.ProtectedMemberSuffix,  value: _          }
  - { key: readability-identifier-naming.EnumConstantCase,         value: CamelCase }
  - { key: readability-identifier-naming.EnumConstantPrefix,       value: k         }
  - { key: readability-identifier-naming.ConstexprVariableCase,    value: CamelCase }
  - { key: readability-identifier-naming.ConstexprVariablePrefix,  value: k         }
  - { key: readability-identifier-naming.GlobalConstantCase,       value: CamelCase }
  - { key: readability-identifier-naming.GlobalConstantPrefix,     value: k         }
  - { key: readability-identifier-naming.MemberConstantCase,       value: CamelCase }
  - { key: readability-identifier-naming.MemberConstantPrefix,     value: k         }
  - { key: readability-identifier-naming.StaticConstantCase,       value: CamelCase }
  - { key: readability-identifier-naming.StaticConstantPrefix,     value: k         }
  - { key: readability-implicit-bool-conversion.AllowIntegerConditions,  value: 1   }
  - { key: readability-implicit-bool-conversion.AllowPointerConditions,  value: 1   }

Modifications

  • (Class) Member suffix _ -> m_
  • Method case: camelCase
TODO
  • Evaluate out if these modfications make sense.
  • Extend/complete list of modifications as necessary

Improve allocation filter handling

Currently only one allocation filter strategy is supported

The filter should be outlined to a base class, and different strategies inhereit from a common interface (see feat/sc20):

  • Base class, for a common interface
  • Runtime selectable with argument
  • Whole-program call-graph filter

Library/Executable file naming scheme

Going forward, the typeart prefix should be used for executables (our scripts) and eventually the LLVM passes and runtime library.

  • rename scripts: run-typeart.sh and apply-typeart.sh
  • rename passes: typeartpass and meminstfinderpass

Extract static source information for runtime target code introspection

Currently, TypeART uses __builtin_return_address(0) to find the address location of where the allocation occurred

Alternatively, (i) during compilation each allocation is tagged with a unique ID, (ii) the id's are passed as arguments to the callbacks, and (iii) the ID and other useful (debug) information are serialised for look-ups in the runtime

The serialisation should happen as json/yaml, see preliminary implementation in feat/sc20

Scripts with optimization flag apply heap instrumentation twice

The desired workflow with the optimization flag is to apply TypeART using opt like TA heap > optim flag > TA stack.

However, in the last step stack and (again) heap are instrumented.

Fix

Add -typeart-no-heap flag to last step for both run.sh and apply.sh

Filter keeps allocations when passed to callback or mem operation

If, say, an integer for the size of an array is passed to a malloc call, the filter will keep the value, as malloc is a decl call.
decl calls are unknown to the filter, hence, it keeps the aforementioned integer.

Example

extern double* a;
void foo() {
  int n = ...; // n is not filtered
  a = (double*)malloc(sizeof(double) * n);
}

Solution

Need to whitelist all known allocation and deallocation calls.
Also applies to the TypeART callback functions.

TypeART-rt -- onAllocStack may push nullptr

If onAllocStack is called with a nullptr (should not happen), doAlloc will not create an entry in the memory map.
However, the stack vector will, which is errorneous.

doAlloc should be restructured to return an enum return code, which the onAlloc functions (heap, stack, global) can use for corner case handling.

Filter allocas with no direct source relation

Currently, all allocas, which are not filtered by TypeART, are instrumented

Add an additional condition that we need a direct source relation of the alloca (e.g., a name; line/col. number)

Logger may interleave lines in multi-threaded context

Happens with MPI_LOGGER=OFF.

Fix

  • Change:
    llvm::errs() << (LEVEL) << " " << LOG_BASENAME_FILE << ":" << __func__ << ":" << __LINE__ << ": " << MSG << "\n";
  • To:
    std::string s;
    llvm::raw_string_ostream rso(s);
    rso << (LEVEL) << LOG_BASENAME_FILE << ":" << __func__ << ":" << __LINE__ << ":" << MSG << "\n";
    // log rso.str()...

Ninja build does not work for coverage target

Building TypeART with ninja works but executing the (lcov) coverage targets fails (e.g., lcov-make).

Most likely this behavior comes from differences between make and ninja w.r.t. commands in add_custom_target, see https://gitlab.kitware.com/cmake/cmake/-/issues/21778

The respective targets have to be re-worked for ninja-compat.

Update

Ninja doesn't like the lcov-make target, in particular, both the fragments --no-extenal and -b ${CMAKE_SOURCE_DIR}.
What seems to work is:

add_custom_target(
  lcov-make
  COMMAND ${LCOV_COMMAND} ${GCOV_TOOL} ${GCOV_WORKAROUND} -c -d ${CMAKE_BINARY_DIR} -o typeart.coverage
  COMMAND ${LCOV_COMMAND} --remove typeart.coverage '${CMAKE_BINARY_DIR}/*' -o typeart.coverage
  # need to remove externals:
  COMMAND ${LCOV_COMMAND} --remove typeart.coverage '/usr/*' -o typeart.coverage
  COMMAND ${LCOV_COMMAND} --remove typeart.coverage '*/llvm/*' -o typeart.coverage
)

Thread-safe / multi-threaded TypeART runtime implementation

To be able to handle multi-threaded, e.g., hybrid MPI+OpenMP, applications, the TypeART runtime needs to be made thread-safe.

Most likely also some careful inspection of the instrumentation function interfaces, and the LLVM pass implementation is needed here to account for thread-local storage variables, etc.

OpenMP

Some points apply to general thread-safety.

Runtime

Passes

  • Support basics for data-flow filter with omp microtask indirection
    • Add data-flow tracking w.r.t. omp tasks (e.g., task_alloc)

Support InvokeInst in analysis

The MemOpVisitor currently only supports CallInst for (heap) allocations.

In some codes, InvokeInst may be used by the compiler for heap allocations, see InvokeInst LangRef

InvokeInst and CallInst have the common base class CallBase, which can be used to unify the collected list of MallocData.

Support user-defined allocation functions

TypeART should be configurable for user-defined allocation (and deallocation) functions

Consider the AMG multi grid solver.

AMG uses macros for memory allocation, e.g., a calloc-like implementation:

#define hypre_CTAlloc(type, count) \
( (type *)hypre_CAlloc((unsigned int)(count), (unsigned int)sizeof(type)) )

In turn the called function looks approximately like this:

char* hypre_CAlloc(int count, int elt_size) {
  char* ptr;
  int size = count * elt_size;

  ptr = calloc(count, elt_size);

  return ptr;
}

The defect:

Assume a call like:

 double* a = hypre_CTAlloc(double, num_variables);
  1. Our static pass is not aware of this macro, it instead finds the C calloc.
  2. Thus, it deduces the char* type for that calloc, which is put into our runtime.
  3. The subsequent cast of the char* returned by hypre_CAlloc to double is not found by our current implementation.

Improve runtime AccessRecorder

Currently, the recorder does not track distinct types or scalar vs. array alloc

Consider the Recorder from feat/sc20 (Counter). It tracks distinct types and corresponding counts as well as distinction between scalar vs. array allocations.

Potential improvements:

  • Output format for easier parsing
  • Better TypeART runtime memory overhead estimation (especially w.r.t. non std::map implementations of the pointer map)

Update CI workflows

The current workflows (CI, CI-ext) are sequential. This could be changed by using multi-job workflows, saving some execution time.

Pre-requisites

To avoid too much code/steps duplication, recent composite actions can be used depending on implementation state, see 1. Next Steps for Fully Functioning Composite Actions and 2. composite-run-steps-actions

  • Action for setup of Clang/LLVM/OpenMPI package
    • Can this include the environment setup for the LLVM executables?
    • Can we set GITHUB_ENV values in a composite action?

Side effects

Artifacts are generated per job, e.g., 2 files are generated for the CI workflow: 1. coverage html report and 2. lulesh build/run data.

CI

Parallel Jobs:

  1. Build for coverage/testing

OMP Extension

Build matrix to test different thread-safety levels for the TypeART runtime libary.

  • Combine coverage of all levels (safeptr, mutex, unsafe)

CI-ext

Parallel Jobs with and without thread-safety enabled:

  1. test-bench
  • Dividing the subprojects probably too much overhead
  1. AD test-bench
  2. OMP test-bench

Segfault when (thread) CounterStats is called with empty vector

The segfault happens, if the Runtime is instantiated (and then discarded) by calling only the API that does not mutate the AccessCounter during execution, e.g.,

int main(int argc, char** argv) {
  const void* ret_check=NULL;
  const void* addr = 1;
  typeart_get_return_address(addr, &ret_check);
  return 0;
}

Defect

CounterStats::create -> vals is empty, the following code causes the segfault:

    Counter min  = *std::min_element(vals.begin(), vals.end());
    Counter max  = *std::max_element(vals.begin(), vals.end());

Enhance the interceptor library with MPI type checking

Currently, the MPI interceptor is only useful to verify proper tracking of the buffer passed to MPI calls in a (TypeART) target code.

Simple MPI type checking w.r.t. basic MPI datatypes should be implemented to have a more thorough test insight.
This can later be extended to derived datatypes.

For reference, see the demo tool

Refine attributes of the callback functions

See https://llvm.org/docs/LangRef.html

Change pointer argument from ReadOnly to ReadNone(?):

readonly
On an argument, this attribute indicates that the function does not write through this pointer argument, even though it may write to the memory that the pointer points to.
If a readonly function writes memory visible to the program, or has other side-effects, the behavior is undefined. If a function writes to a readonly pointer argument, the behavior is undefined.

readnone
On an argument, this attribute indicates that the function does not dereference that pointer argument, even though it may read or write the memory that the pointer points to if accessed through other pointers.
If a readnone function reads or writes memory visible to the program, or has other side-effects, the behavior is undefined. If a function reads from or writes to a readnone pointer argument, the behavior is undefined.

Mark functions as (and also readnone?):

nofree
This function attribute indicates that the function does not, directly or indirectly, call a memory-deallocation function (free, for example). As a result, uncaptured pointers that are known to be dereferenceable prior to a call to a function with the nofree attribute are still known to be dereferenceable after the call (the capturing condition is necessary in environments where the function might communicate the pointer to another thread which then deallocates the memory).

Invoke -- malloc for array of struct has wrong bitcast/type identification

Defect

The following C++ allocation is wrongly identified by TypeART as int64 (LLVM 10)

  struct S1 {
    int x;
    virtual ~S1() = default;
  };
  try {
    S1* ss = new S1[2];
  } catch (...) {
  }

LLVM/TypeART Generated Code

  %5 = invoke i8* @_Znam(i64 40) #4
          to label %6 unwind label %16

6:                                                ; preds = %0
  call void @__typeart_alloc(i8* %5, i32 3, i64 5)
  %7 = bitcast i8* %5 to i64*
  store i64 2, i64* %7, align 16
  %8 = getelementptr inbounds i8, i8* %5, i64 8
  %9 = bitcast i8* %8 to %struct.S1*
  %10 = getelementptr inbounds %struct.S1, %struct.S1* %9, i64 2
  br label %11

Two bitcasts are generated.
TypeART erroneously takes the first one as the candidate to extract the type (i32 3 -> i64).

AccessRecorder does not increment maxHeapAlloc correctly in leaky code

Currently, maxHeapAlloc is only incremented when a free heap operation triggers in a target code.

Hence, in a code without free, the max heap is incorrectly printed:

int main(void) {
  for (int i = 1; i <= 6; ++i) {
    // 6 heap alloc, max heap (concurrently) 6
    double* d = (double*)malloc(sizeof(double));
  }
  return 0;
}

-> Prints: Max. Heap Allocs 0 but expeced is 6.

Fix

AccessRecorder calculates max. for every heap allocation "just in time", instead of waiting for a free.

CI condition for installing LLVM libc++ wrong

Is:

      - name: Install libc++
        if: matrix.libcxx
        run: sudo apt-get install --no-install-recommends libc++-10-dev libc++abi-10-dev

should be:

      - name: Install libc++
        if: matrix.preset.libcxx
        run: sudo apt-get install --no-install-recommends libc++-10-dev libc++abi-10-dev

AD Lulesh CI segfaults for thread-safe safe-ptr map

Applies to thread-safe codes with ENABLE_SAFEPTR=ON.

Lulesh appears to finish execution. Crashes when the TypeART softcounters are (almost?) printed, e.g.:

2021-04-05T18:39:18.1763435Z 3: [fv-az196-541:05429] Signal: Segmentation fault (11)
2021-04-05T18:39:18.1764428Z 3: [fv-az196-541:05429] Signal code: Address not mapped (1)
2021-04-05T18:39:18.1765347Z 3: [fv-az196-541:05429] Failing at address: 0x8
2021-04-05T18:39:18.1766476Z 3: [fv-az196-541:05429] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fbf69ce33c0]
2021-04-05T18:39:18.1768594Z 3: [fv-az196-541:05429] [ 1] /home/runner/work/tudasc-typeart/tudasc-typeart/install/typeart/lib/libtypeart-rt.so(_ZN2sf28contention_free_shared_mutexILj36ELb0EE4lockEv+0x4f)[0x7fbf6a1e934f]
2021-04-05T18:39:18.1771641Z 3: [fv-az196-541:05429] [ 2] /home/runner/work/tudasc-typeart/tudasc-typeart/install/typeart/lib/libtypeart-rt.so(_ZN7typeart17AllocationTracker10doFreeHeapEPKvS2_+0x22)[0x7fbf6a1e6932]
2021-04-05T18:39:18.1774331Z 3: [fv-az196-541:05429] [ 3] /home/runner/work/tudasc-typeart/tudasc-typeart/install/typeart/lib/libtypeart-rt.so(__typeart_free+0x96)[0x7fbf6a1e76c6]
2021-04-05T18:39:18.1775733Z 3: [fv-az196-541:05429] [ 4] lulesh2.0[0x438fb3]
2021-04-05T18:39:18.1776592Z 3: [fv-az196-541:05429] [ 5] lulesh2.0[0x426fed]
2021-04-05T18:39:18.1777631Z 3: [fv-az196-541:05429] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0x49a27)[0x7fbf69b25a27]
2021-04-05T18:39:18.1778850Z 3: [fv-az196-541:05429] [ 7] /lib/x86_64-linux-gnu/libc.so.6(on_exit+0x0)[0x7fbf69b25be0]
2021-04-05T18:39:18.1780136Z 3: [fv-az196-541:05429] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfa)[0x7fbf69b030ba]
2021-04-05T18:39:18.1781189Z 3: [fv-az196-541:05429] [ 9] lulesh2.0[0x404b9e]
2021-04-05T18:39:18.1782073Z 3: [fv-az196-541:05429] *** End of error message ***
2021-04-05T18:39:18.1782994Z 3: [fv-az196-541:05432] *** Process received signal ***
2021-04-05T18:39:18.1783956Z 3: [fv-az196-541:05432] Signal: Segmentation fault (11)
2021-04-05T18:39:18.1784937Z 3: [fv-az196-541:05432] Signal code: Address not mapped (1)
2021-04-05T18:39:18.1785865Z 3: [fv-az196-541:05432] Failing at address: 0x70
2021-04-05T18:39:18.1786986Z 3: [fv-az196-541:05432] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fb936acd3c0]
2021-04-05T18:39:18.1789170Z 3: [fv-az196-541:05432] [ 1] /home/runner/work/tudasc-typeart/tudasc-typeart/install/typeart/lib/libtypeart-rt.so(_ZN2sf28contention_free_shared_mutexILj36ELb0EE4lockEv+0x4c)[0x7fb936fd334c]
2021-04-05T18:39:18.1792198Z 3: [fv-az196-541:05432] [ 2] /home/runner/work/tudasc-typeart/tudasc-typeart/install/typeart/lib/libtypeart-rt.so(_ZN7typeart17AllocationTracker10doFreeHeapEPKvS2_+0x22)[0x7fb936fd0932]
2021-04-05T18:39:18.1794941Z 3: [fv-az196-541:05432] [ 3] /home/runner/work/tudasc-typeart/tudasc-typeart/install/typeart/lib/libtypeart-rt.so(__typeart_free+0x96)[0x7fb936fd16c6]
2021-04-05T18:39:18.1796325Z 3: [fv-az196-541:05432] [ 4] lulesh2.0[0x438fb3]
2021-04-05T18:39:18.1797200Z 3: [fv-az196-541:05432] [ 5] lulesh2.0[0x426fed]
2021-04-05T18:39:18.1798238Z 3: [fv-az196-541:05432] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0x49a27)[0x7fb93690fa27]
2021-04-05T18:39:18.1799427Z 3: [fv-az196-541:05432] [ 7] /lib/x86_64-linux-gnu/libc.so.6(on_exit+0x0)[0x7fb93690fbe0]
2021-04-05T18:39:18.1800702Z 3: [fv-az196-541:05432] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfa)[0x7fb9368ed0ba]
2021-04-05T18:39:18.1801755Z 3: [fv-az196-541:05432] [ 9] lulesh2.0[0x404b9e]
2021-04-05T18:39:18.1802624Z 3: [fv-az196-541:05432] *** End of error message ***

Refactor instrumentation handling

The current implementation uses lambda functions for each memory operation

Each lambda (i) collects information for the callback (arguments), and (ii) adds the instrumentation callback at the correct position in the code.

In reference to #4 w.r.t. InvokeInst handling, this should be extracted and divided into two phases for easier maintainability:

  1. Per BasicBlock, the information for the callback is collected (and stored appropriately in a list)
  2. A Strategy is used to execute the instrumentation, per BB, using the collected information of phase 1

The default strategy is then to add the callback right after the memory operation (as is the case now).

Install scripts (run/apply)

Currently, these scripts are put into the build folder with appropriate (build-folder related) paths settings to the plugins.

Improvement

These should also be added to the install target.

  • Set install path (instead of build related)
  • Refactor script to put placeholder vars (e.g., @foo@) at central location (e.g., at top)

Runtime exits without types.yaml even if not needed

Assuming a target application does not use any user-defined types (or TypeART filters all related allocations) no types.yaml file is ever created.
However, the runtime lib requires a valid file path to types.yaml and otherwise exits.

Potential fix

Only exit if the user specified the environment variable TA_TYPE_FILE for the file location and it does not exist.
If defaultTypeFileName is not found (types.yaml), print a warning.

TODO

In turn, the runtime needs to handle the case of a callback for a user-defined type gracefully.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.