The lib-common is an Intersec libc extension targeting Linux environments. It provides most of the basic tools a C programmer can expect such as:
-
a string manipulation library
-
containers like dynamic vectors, hash tables, hash maps, lists…
-
bitmask manipulation functions
-
endianness conversion primitives
-
…
It also contains higher level libraries:
-
asynchronous event loop
-
http client and server implementations
-
RPC communication channels
-
optimized allocators for our use cases
-
log file management
-
tracing and reporting primitives
-
program argument parser
-
ASN1 parser / writer
-
A prometheus client
-
…
It is also the home of our IOP library. The IOP (Intersec Object Packer) is a framework to serialize structured data to use in communications protocols, data storage… IOP is language independent. IOP objects are encoded using TLV (Tag-Length-Value) packers/un-packers. The IOP concept is similar to the Google Protocol Buffers.
The IOP library is used to transmit data over the network in a safe manner. It deals with data integrity checking, retro-compatibility issues,… They are also used to exchange data between different languages or to provide a generic interface to store and load C data on disk.
This section offers basic instructions for building the lib-common.
The lib-common is mainly written in C, with some extensions coming from clang features. So its main dependencies are:
- glibc
-
used extensively, in particular to interface with various Linux kernel facilities
- llvm
-
base compiler toolchain
- libclang-dev
-
base c and cpp compiler development library
- clang
-
base c and cpp compiler
- waf
-
build system
- poetry
-
managing Python dependencies
- openssl
-
various cryptography requirements
- flex
-
IOP syntax lexer at https://github.com/westes/flex
- gperf
-
GNU library for perfect hash function generation at https://www.gnu.org/software/gperf
Most of dependencies can be installed using your own Linux distribution packages, with the exception of waf which is usually not packaged.
The lib-common build system uses waf. If you want to understand how it works, you should read the Waf Book, but this is not necessary if you just want to use it.
All the lib-common specificities and custom rules are defined in waf
tools,
located in the build/waftools
directory.
The main entry point of the build system is the wscript
file located at
top-level of the repository. It loads the Intersec tools, defines the main
configure
and build
functions, and recurses in sub-directories (that
contain their own wscript
or wscript_build
files).
The waftools and wscript files are supposed to be readable and documented when needed.
waf
can be downloaded as a single executable or built from sources, as
described in the waf documentation.
The simplest approach is to download the waf executable and
install it in a directory already in your PATH environment, for example
/usr/local/bin
:
$ cd /usr/local/bin $ wget https://waf.io/waf-2.0.x $ ln -s waf-2.0.x waf $ chmod +x waf-2.0.x $ ./waf
waf
versions 2.0.8 and 2.0.18 are known to work fine, running the latest 2.0.x
waf
should also work fine.
You need to install the following packages to build the lib-common:
$ sudo apt-get install clang libclang-dev llvm-dev python3-dev python3-pip flex gperf pkg-config exuberant-ctags libxml2-dev libssl-dev
Optionally, you can also install the following packages for a complete installation:
$ sudo apt-get install exuberant-ctags valgrind
In order to run the tests, the following packages also need to be installed:
$ sudo apt-get install smitools snmp-mibs-downloader
You need to install the following packages to build the lib-common:
$ sudo dnf install clang clang-devel llvm-devel flex gperf libxml2-devel openssl-devel python3-devel
Optionally, you can also install the following packages for a complete installation:
$ sudo dnf install ctags-etags valgrind-devel
In order to run the tests, the following packages also need to be installed:
$ sudo dnf install libsmi diffutils
Poetry is used to install Python dependencies used on compilation (Cython), and in tests. To install Poetry, follow the instructions on the official documentation: https://python-poetry.org/docs/#osx—linux—bashonwindows-install-instructions
First of all, you have to configure your project, by running in the top-level directory:
$ waf configure
By default, it will use gcc. You can also use clang by defining the two
environment variables CC
and CXX
:
$ CC=clang CXX=clang++ waf configure
If the configuration step triggers no error, then you are ready to build, which just consists in running:
$ waf or $ waf build
You can run it from a sub-directory in order to build only the targets defined in this directory and its sub-directories (and its dependencies).
All the available targets can be listed with this command:
$ waf list
It is possible to build only a specific target, or a list of targets, by running, from anywhere in the repository:
$ waf --targets=target1,target2
Other commands are listed with waf --help
. Here they are:
-
waf check
: run the tests of the current directory (defined in theZFile
) and in its sub-directories. The following variants also exist (cfwaf --help
for the details):fast-check
,www-check
,selenium
,fast-selenium
. -
waf tags
: generate tags using ctags. -
waf etags
: generate tags for emacs using ctags. -
waf pylint
: run pylint checks on committed python files. -
old-gen-files-detect
: detect old files generated by a previous build system run. -
old-gen-files-delete
: delete the files detected by the previous command. -
coverage-start
: start a coverage session (requires coverage profile). This resets the coverage counters. After running this command, you can run some code and use thecoverage-end
command to produce a coverage report. Note that this is done when configuring the project. -
coverage-end
: end a coverage session and produce a report.
The following environment variables can be used at the configuration phase:
P
(string)-
Specify the desired compilation profile (default, debug, release, …). The complete list of available profiles is defined in
build/waftools/backend.py
, variablePROFILES
. If not specified, the default profile isdefault
. NOCHECK
(boolean)-
The build-system doesn’t run "check" targets, which are:
-
clang check of c files.
-
linters on js/ts files. You may want to set it to speedup the build. You can also bypass the checks thanks to the
nocheck
parameter of task-generators, which can beTrue
to bypass the checks of all the source files, or a list of files to not check.
-
NOASSERT
(boolean)-
If set, the assertions and debug-related code won’t be compiled. By default, assertions are disabled in
release
andmem-bench
profiles, and enabled in all the others. NO_DOUBLE_FPIC
(boolean)-
If set, the compilation will be faster, but the produced binaries will be larger and the runtime performances will be affected. Cf.
build/waftools/backend.py
for the details. This is ignored in release profile. FAKE_VERSIONS
(boolean)-
If set, the version files are generated with fake (and constant) data, so that changing of git revision does not trigger a re-link of all the binaries. This is a huge gain of time, but it’s not possible to know the revision of the binaries that are built with this flag. This is ignored in release profile.
SHARED_LIBRARY_SANITIZER
(boolean)-
If set, the shared libraries will also be compiled with the sanitizer specified by the profile. You will have to use
LD_PRELOAD
or use a process that is compiled with the same sanitizer to load the compiled shared libraries with this option. This is only available for profiles that use sanitizers, i.e. asan or tsan.
An important part of the lib-common uses the blocks clang extension, that is not supported by gcc.
Because of this, we have put in place a two-phase build of some files
(named foo.blk
instead of foo.c
) that are pre-compiled using a special
tool named clang-rewrite-blocks
. gcc is then used to produce the final
object code.
clang-rewrite-blocks
is compiled and used automatically by the build-system.
See src/`clang-rewrite-blocks
/README.adoc` for more information about
clang-rewrite-blocks
.
In the spirit of open source software, everyone is welcome to contribute to this project!
The best way to get involved is to just show up and make yourself heard. We pride ourselves on having a very friendly and encouraging culture. Whether you’re a user, writer, designer, developer, architect, devops, system administrator, advocate, project manager, or just someone with an idea about how to improve the project, we welcome your participation. In return, you’ll get to use better software that we built together as a community.
Thanks in advance for helping to make this project a success!
Copyright © 2005-2020 by Intersec SA and the individual contributors to lib-common.
Licensed under the Apache License, Version 2.0 (the "License"). You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Development of the lib-common is led and sponsored by Intersec.