jpenuchot / ctbench Goto Github PK

Trying to use cmake 3.22.1 to run

cmake --preset release

per the installation instructions returns

CMake Error: Could not read presets from /z/ctbench: Unrecognized "version" field

openjournals/joss-reviews#5165

Is it possible to set sciplot up so that it can be pulled from a submodule or similar without requiring a system-wide install?

openjournals/joss-reviews#5165

Some GCC flags worth checking out:

-ftime-report
-ftime-report-details
-time[=file]

Seems to be comparable to Clang's time trace, so adapting ctbench to it might be pretty easy.

https://devblogs.microsoft.com/cppblog/introducing-c-build-insights/

I own zero Windows machine at the moment, so any help is welcome.

Thanks @JolyLoic for pinging me about that :)

• CLI
• Demangling

Adding a predicate that matches events with a given parent. The matching could be done by name, by pointer, or simply by using other predicates to make it modular.

The main sub-issue for this one will be to keep track of event arborescence.

nlohmann::json has a nice interface but performance doesn't scale for what we're doing (processing hundreds of megabytes of JSON trace events).

Running ctbench-grapher-plot through perf shows that nlohmann::json-related calls are the ones that account for most overhead.

ctbench only supports variable size benchmarks, support for single size benchmarks for A/B comparisons would be nice too. It would require work on everything from the CMake API to grapher core data structures, and possibly the CLI too.

Just figured that some configs I wrote in https://github.com/JPenuchot/rule-of-cheese/ are generic enough to be provided as examples. I'd like them to be baked in a default config directory, and maybe implement a lookup directory list for configurations.

To fix for later:

• boost is missing
• sciplot should be specified as sciplot-git (the regular sciplot package uses catch2-v2, which can't be installed with the new version)

One way to easily add (limited) support for GCC would be to measure compiler execution time instead of relying on internal profiling data. This could be useful to compare metaprogram performance scaling across a variety of compilers.

The main issues are:

• How to measure compiler execution time from CMake
• How to adapt current data wrangling code for a new kind of format

And so far my favorite solutions are:

• Adding compiler execution time measurement to the clang time-trace wrapper
• Use the wrapper to generate a file in the same format as time-trace files, but only with compiler execution time data

The compare_by plotter saves plots with filenames generated using the key itself. This becomes problematic when the symbols grow large enough for filenames to be too long.

The issue is known and currently has no other solution than generating smaller symbols at the moment, eg. benchmark driver functions with names that do not depend on benchmark instantiation size.

Solutions for this problem are welcome. I'm open to having different name shortening strategies, generating index files to help find plot files, and others. I will address this issue if it becomes really problematic.

Having an up-to-date documentation built upon the main branch would be a nice plus. Having it running outside of Github would be even nicer.

It'd be handy if you made a note in the readme that running

sudo apt install nlohmann-json3-dev

installs a necessary prerequisite. Perhaps there are others as well?

openjournals/joss-reviews#5165

Compiler execution time measurement through ttw seems to be broken, the time values it reports aren't coherent with Clang's time trace profiler. More works needs to be done to measure compiler execution more precisely, and Hyperfine seems to have that figured out so it's worth having a look and reproduce their measurement method in ttw.

Hey, please find my review of your tool below:

• Regarding the manuscript I don't have major comments, other than that there are many typos/grammatical errors, especially in the summary.

• I found the installation of the package a bit tedious:

  • there is no description how to install on Mac/Windows. The latter can I guess be excused, but if the package is not easily installable/available on Mac this excludes a significant fraction of the research community,
  • to install I needed to use Docker and reproduce what you did here and additionally google how to run ./vcpkg/vcpkg install sciplot fmt on an ARM. I think it would be easiest to provide a Dockerfile that does the exact installation with all dependencies in the example folder (and have a GitHub action that tests on Mac M1).

• I think providing install instructions using conda would be helpful and also make it straight-forward to use for Mac users. I think all required dependencies are on there.

• I haven't done much C++ development in the last year but seeing how much traction Meson is gaining, I find it unusual that it isn't provided as alternative build system. Would that be easy to offer? Also, is the CMake 3.25 dependency really necessary?

• I think providing a self-contained example with exact instructions how the visualisations are created is helpful. I have probably overlooked something, but it's not 100% clear how you made them from a quick look.

• Create a CONTRIBUTING file with how-to's on contributing new code.

Cheers,
Simon

Turning graph saving off makes ctbench-grapher-plot execution a lot shorter (a several seconds vs several minutes). Generating graphs using ROOT instead of Sciplot might be a good way to cut graph generation times by a significant factor.

• CI
• Test infrastructure
• Tests implementation

Maybe: add action for tests compiled with GCC?

I'm having a hard time finding a good way to organize the documentation into something that's clear and concise. I'm currently not satisfied by the way JSON configuration values are documented as I feel like users have to dig way too much just to find JSON configuration information.

Additionally, the docs aren't clear about what's internal or not.

I'm leaving this issue here as long as I'm not satisfied with way the docs are presented, any suggestion or critics are welcome.

Took note from @marcorubini's idea of using a multi tree visualization strategy to display multiple clang traces on a single graph. At the moment this is only a reminder issue for later.

I'm on Ubuntu 22.04, which is not that old, and finding that cmake 3.22.1 is too old to meet your CMakeLists.txt requirements. Can these be relaxed?

Furthermore, CMakeLists.txt requires 3.25 while CMakePresets.json requires 3.23. Can you make these consistent?

openjournals/joss-reviews#5165

If I edit CMakeLists.txt to read:

cmake_minimum_required(VERSION 3.22)

and then run

cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -GNinja ..
ninja

I see

/z/ctbench/grapher/lib/grapher/plotters/debug.cpp:1:10: fatal error: llvm/Support/raw_ostream.h: No such file or directory
    1 | #include <llvm/Support/raw_ostream.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/z/ctbench/grapher/grapher-utils.cpp:1:10: fatal error: llvm/Support/CommandLine.h: No such file or directory
    1 | #include <llvm/Support/CommandLine.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

Is it possible to remove this LLVM dependency in favour of using the C++ STL instead? If not, is it possible to make the LLVM linking mechanism more robust?

openjournals/joss-reviews#5165

I'm not a CMake expert but maybe someone would be able to help making ctbench installable and usable with find_package. The end goal would be to end up with something that's easy to handle with makepkg (AUR), conan, and vcpkg.