knitschi / cmakeprojectframework Goto Github PK

On windows, the version number can be included in the binary. CMake alows us to do this, so we should do it.

https://stackoverflow.com/questions/6693100/how-to-generate-windows-dll-versioning-information-with-cmake

We should update all third party components to the latest stable versions before the release is published.

Components:

• Jenkins
• Default build configurations. (compiler/Visual Studio)
• CMake
• python
• conan
• sphinx

cmake 3.19 introduced the preset .json files for the same purpose for which I introduced my config.cmake files. Switching to default cmake solution would probably improve acceptance in the user base.

This ticket is about examining if cmakes preset files can replace the custom solution in CPFCMake and CPFBuildscripts and what would need to be done to implement it. We also have not yet removed the duplication of configuration information between the package manager configuration and the local configuration files.

The current solution that pushes the install results to a git repostiry does not work well.

• Github complains about checked-in binary files in the github pages.
• Cloning the result repositories from my NAS takes too long because of lacking main memory.

The publish / archive step in the build-pipeline needs to be customizable. So this is what we need to do:

• Find a way to enable injecting a custom archive / publish mechanic into the build-script.
• Implement one mechanic that pushes the results to a private conan-server. It also needs to copy the result to the web-server.
• Implement one mechanic that pushed the documentation to the Knitschi.github.io repository and creates a github release that contains the rest of the package. (https://help.github.com/en/articles/about-releases)
• Remove the updateResultsRepository.cmake script from CPFCMake.
• Give clients more controll over the install targets if neccessary. (Allow adding installed components to a global install_release target, let the user define custom install prefixes per install component)

Currently cpf projects do not work well when opened with Visual Studios CMake mode. We get CMake errors and in the target overview the targets have no files attached. We should try to make that VS-feature work for cpf projects.

This would allow sharing build-artefacts between developers.

Allowing the selection of the compiler-configuration after CMake's generate step causes multiplie headaches:

1. CMake does not allow generator expressions in the OUTPUT argument of add_custom_command(). CPFCMake makes heavy use of that function and working around that limitation requires a lot of complicated and error prone CMake code and brings an performance overhead.
2. The CMake code gets more complicated because config dependent paths in the build-tree must use harder to read generator expressions.
3. The Conan Package Manager's support for multi-config packages is lacking.
4. Creating multi-config packages is harder because the results of multiple builds need to be composed into one package which requires lifting the package creation from the build level up to the build-job level where artefacts of multiple builds are available.

The advantage of allowing multiple configurations is that Visual Studio users are used to being able to select the build configuration in the UI. But practice showed that you can get used to the alternative workflow where you just open one IDE instance for each configuration that you want to build.

In order to remove the above mentioned problems I decided to change the CPFCMake and CPFBuildscripts to only allow one configuration at a time when using the Visual Studio Generator.

TODO:

• Remove the CompilerConfig from the cpfCIBuildConfigurations.json file.
• Remove the config option from 3_Make.py.

The documentation is not completely worked out right now. This should be finished when all features for the 1.0 Release are done.

We should consider the following:

• Choose a template that is also readable on phones. The "Read-the-Docs" template that is used by conan seemed to work.
• Maybe rework the api doc structure by splitting it into smaller peaces.
• Make sure CPFCMake#27 is implemented.
• Add Boost cmakemodules to the list of similar projects if boost has made the change to CMake before our release.
• Add more details to the similar projects list about what exactly the similarities are.
• Document the compatibility guarantees. -> There are none for now.
• Fill empty pages.
• Re-Iterate over everything to get a consistent style.
• Document the lack of the multi-config feature.
• Acquisition via package manager?

Currently we do not verify that the created packages work for consumers. We should try to add custom targets that test if created packages.

• Developer binary library packages must be consumed by a generated project and test compiled. How will we get the dependencies in this case? This test will discover missing public headers.
• Testexecutables must be run on a clean machine. This will require a stand-alone package type for library packages.
• IT should be possible to test debian install packages, because the should be able to install system libraries.
• Do we need to add special deployment tests slaves to infrastructure, that are reset after running the tests?
• Can we do it all locally with containers, so developers can easily run the tests on their machines?

The currently enforced structure <build-project-dir>/src/<module_package_dir> does not allow having multiple module-packages in one module repository.
We also have the constraint that all modules in a deployment-package must be at the same version.
Additionally we want to follow the doctrine that the commits in the repository are the single source for all versions.

All constraints together limit us to only allowing one module per deployment-package. This ticket is about removing this limitation.

We want to extend the cpfAddPackages() funktion to allow package definitions in the packages.cmake file in the form

set( CPF_PACKAGES
    EXTERNAL <deployment-package-dir>/x/y/<module_package_dir>
    # or
    EXTERNAL <single-package-dir>
)

In the second scenario the <deployment-package-dir> and the <module-package-dir> have the same name.
It shall be possible to have an arbitrary number of directories in between the <deployment-package-dir> and the <module_package_dir>.

The name of the <deployment-package-dir> shall be the only source for the deployment-package-name. This is the name that will be used in package managers to retrieve a binary package and as a part of binary archive files. This will also be the name of the CMake export set and the targetnamespace for the all targets that are exported by the module packages.

The name of the <module-package-dir shall be the only source for the module-package-name. This is the name of the main target in a module-package. It also appears as the first directory in the C++ #include statements.

This ticket should also implement the mechanism that derives the deployment-package-name and the module-package-name from the relative path in the CPF_PACAKGES variable and propagates them to the install commands etc. We also need to make sure that package config files are only generated once per deployment-package.

The following issues need to be done before the release.

• #13
• #14
• #15
• #17
• #2
• #9?
• Knitschi/CPFCMake#53
• Knitschi/CPFCMake#51
• Knitschi/CPFCMake#58
• Knitschi/CPFBuildscripts#8

The major C++ package manager are.

• hunter
• conan
• vcpckg

We could also consider making CPFBuildscripts available via pip.
First we should support hunter because this is what I am most familiar with.

We currently use the term package for two different things which will lead to confusion. The work for this ticket is to go through all source-files of the CPF and make sure that comments, function-names, variable-names etc. consistently use the name convention the is described below.

The host-repository that contains root CMakeLists file will be called the deployment-repository.
The toplevel-cmake-project will be called the deployment-project.
The binary archives that can be obtained via package managers are called deployment-packages.
The cmake-projects that are defined in the formerly package CMakeLists file are called module-projects.
The collection of source files that define one production binary target and additional test-targets etc. and that were formerly known as package are now called module-package.
The repository that holds one or multiple module-packages but no top-level CMakeLists file is called the module-repository.

We need automated tests, that run the pipeline with all the underlying tools. Currently we only have a view unit tests for low-level python and cmake code. As these tests will test the functionality of multiple packages at once we should put them in their own package CPFTests. This package can also contain utility functionality like test driver code or a project generator. The CPFTests package should create an extra custom target to run the integration tests.

Tests should

• Build existing test projects.
• Check target dependency functionality.
• Test config options.
• Test buildscript options.

Random Questions:

• Each package needs to specify its own dependency requirement. (packages and compatible versions). Should this be defined in the package or should the package manager have this information?
• Who should do automated compatibility checks? The project while "building" or the CI-job after building the project?
• Should the package only hold a list with dependencies and the CI-job would find the list of compatible versions? This has the advantage that the repository itself must not be changed after building.
• If the responsibiliy of compatibility checking lies with the CI-job, we could remove the abi/api-checker targets from the CPFCMake.
• If the package-manager is the instance that knows about package compatiblities, we need that list for each package manager.
• If the build checks for compatibility, it needs some generic package manager interface to get older package versions.

Implementing this ticket is not possible if the conan-profile file does not allow defining arbitrary cmake variables. Before I work on this, I have to check that this is possible.

I imagine the following setup:

1. The file that stores the build-configuration is changed from the xxx.config.cmake file to the conan-profile file. The 1_Configure.py script must be changed to support writing new options into that file via command line.
2. CPFBuildscript will get the new build-step script 2_GetDependencies.py. The numbers of the following steps need to be incremented accordingly. I should also consider dropping the numbers completely. The GetDependencies.py script will then run the conan install step which downloads or builds the dependencies and creates the conanbuildinfo.cmake file in the build-tree. This file is used to inject the configuration into the CMake generate step.

Problems:

• Can the cmake generator be added to the conan-profile and use?
• What happens to the cmake toolchain files? I think they are required for cross building.
• This change will add a strong dependency to the conan package manager. It could be later be removed by extending the functionality behind the 2_GetDependencies.py to support other package managers.
• Can this mechanism be extended to allow the quich switch between conan-packages and git-submodules?

Currently it is not clear how we handle incrementing the CPF version for clients.

Problems:

• The CPF components do not verify that depended on components have the correct versions. The jenkinsjob uses the buildscript interface. CPFBuildscripts uses the CPFCMake interface. The components should check for the presence of compatible version. In the case of the CPFBuildscripts -> CPFCMake dependency that would be an argument to merge the two packages into one.

• The build-infrastructure can only build jobs of the current cpf version. Upgrading the CPF version can introduce a barrier for building older revisions. The CPF should therefore implement a mechanism that allows running build-slaves that are able to run older versions of the build-job.

• The post-receive triggers use the parameter interface of the job. This may cause problems when we push to branches that use an older CPF version.

Currently the tools that are used by the CPFCMake pipeline are installed by the dockerfiles of the linux containers or require manual installation for windows slaves.

This is problematic because:

• Developers have to install all tools manually.
• When setting up the infrastructure tools have to be installed manually. This could also be solved by adding a container solution for Windows build-salves.
• It creates a cyclic dependency between CPFCMake which depends on CPFMachines to provide the correct build environment.

To get rid of these problems we should create conan packages for all required dependencies. This will also kill the "cyclic" dependency that is created by CPFCMake requireing the tools that are provided by CPFCMake.

After this, developers should only need to install git, python, and possibly the IDE.

Motivation
Initially the python and cmake code where seperated into two packages. The CMake code however expects
the directory structure that is enforced by the python scripts. It is therefore not practical to use CPFCMake without CPFBuildscripts. By putting both packages into one, client projects would only need to add one package instead of two.

Currently I am no sure if the simplification is worth the trouble. An alternative could be to turn the python scripts into cmake scripts to get rid of the python dependency. But that would probably decrease the usability.

Advantages:

• The user only needs to add one submodule to the project.
• It feels like they belong together because they can not really be used seperately.
• Currently all tests from CPFTests could then be moved to CPFCMake.

Disadvantages:

• A lot of work.