fluent-package-builder (formerly known as td-agent-builder, it was renamed at Aug, 2021) is a new build system for td-agent which aims to replace the traditional build system omnibus-td-agent since it has several problems due to Omnibus's limitations.

NOTE: Discussed why re-branding is required Rebranding td-agent-builder

• Use system libraries: e.g. openssl
• Remove libraries for 3rd party gems: e.g. postgresql
• Remove embedded directory by omnibus
  • Use /opt/td-agent/bin/fluent-cat instead of /opt/td-agent/embedded/bin/fluent-cat
• Update core components: ruby, jemalloc and more
• Add td-agent-apt-source deb package for maintaining apt-line and keyring
  • Download it and install by apt install, then you can install td-agent via apt install td-agent.
  • td-agent-apt-source deb package is created to install td-agent easily. Currently it points to td-agent 4 apt-line.

See also this issue for omnibus problems.

• Any host OS that is supported by Docker
  • Debian buster or Ubuntu 18.04 are recommended
• Docker
• Ruby 2.5 or later
• Bundler 2.2.0 or later
• Git

• Windows OS (10 Pro or 2019 are verified)
• Docker Desktop for Windows
  • You need to switch to "Windows containers" before using it.
• RubyInstaller 2.5 or later.
• Bundler 2.2.0 or later
• Git for Windows

After installed above software, you need to enable additional features from powershell (as admin).

dism.exe /online /enable-feature /featurename:Microsoft-Hyper-V /all /norestart
dism.exe /online /enable-feature /featurename:Containers /all /norestart

Then restart Windows.

• macOS 10.15 (Catalina)
• Ruby 2.5 or later
• Bundler 2.2.0 or later
• Git

% rake yum:build

By default, yum repositories for following platforms will be built under td-agent/yum/repositories/ directory:

• CentOS 6 (x86_64)
• CentOS 7 (x86_64)
• CentOS 8 (x86_64)

You can choose target platforms by YUM_TARGETS environment variable like this:

% rake yum:build YUM_TARGETS="centos-6,centos-7,centos-8"

You can find other supported platforms under td-agent/yum directory.

You can also build packages for AArch64 platforms like this:

% rake yum:build YUM_TARGETS="centos-8-aarch64"

But if you use older GNU/Linux platforms (e.g. Ubuntu 18.04 or before) as your host OS, you need to copy qemnu-aarch64-static into the base directory of the target:

% export TARGET_BASE="centos-8"
% sudo apt install qemu-user-static
% cd /path/to/fluent-package-builder
% cp /usr/bin/qemu-aarch64-static td-agent/yum/${TARGET_BASE}
% rake yum:build YUM_TARGETS="${TARGET_BASE}-aarch64"

You can also build packages for ppc64le platform like this:

% rake yum:build YUM_TARGETS="centos-8-ppc64le"

But if you use older GNU/Linux platforms (e.g. Ubuntu 18.04 or before) as your host OS, you need to copy qemnu-ppc64le-static into the base directory of the target:

% export TARGET_BASE="centos-8"
% sudo apt install qemu-user-static
% cd /path/to/fluent-package-builder
% cp /usr/bin/qemu-ppc64le-static td-agent/yum/${TARGET_BASE}
% rake yum:build YUM_TARGETS="${TARGET_BASE}-ppc64le"

% rake apt:build

By default, apt repositories for following platforms will be built under td-agent/apt/repositories/ directory:

• Debian 10 "Buster" (x86_64)
• Ubuntu 20.04 LTS "Focal Fossa" (x86_64)
• Ubuntu 18.04 LTS "Bionic Beaver" (x86_64)
• Ubuntu 16.04 LTS "Xenial Xerus" (x86_64)

You can choose target platforms by APT_TARGETS environment variable like this:

% rake apt:build APT_TARGETS="debian-buster,ubuntu-bionic"

You can find other supported platforms under td-agent/apt directory.

You can also built packages for AArch64 platforms like this:

% rake apt:build APT_TARGETS="ubuntu-bionic-arm64"

But if you use older GNU/Linux platforms (e.g. Ubuntu 18.04 or before) as your host OS, you need to copy qemnu-aarch64-static into the base directory of the target:

% export TARGET_BASE="ubuntu-bionic"
% sudo apt install qemu-user-static
% cd /path/to/fluent-package-builder
% cp /usr/bin/qemu-aarch64-static td-agent/apt/${TARGET_BASE}
% rake apt:build APT_TARGETS="${TARGET_BASE}-arm64"

% rake msi:build

A td-agent-${version}-x64.msi package will be built under td-agent/msi directory.

You can use with MSYS2 for C extension gem building.

MSI included Ruby can detect MSYS2 environment. So, you can install C extension included gem with MSYS2.

e.g.)

Prepare C extension gem building environment:

cmd> ridk install 2
...
cmd> ridk install 3

Install gem via ridk exec td-agent-gem install:

cmd> ridk exec td-agent-gem install winevt_c

% rake dmg:selfbuild

A td-agent-${version}.dmg package will be built under td-agent/dmg directory.

GitHub Actions' built package is ready to run on macOS 10.15 (Catalina).

Be sure to permit to be authrized for for assistive access. In System Preferences > Security & Privacy > Privacy > Accessibility, you should permit Terminal.app there.

NOTE: Since authorization is at the application level on Terminal.app, it allows any script run from Terminal.app to perform GUI scripting.

• Edit td-agent/config.rb to choose Ruby & Fluentd versions
• Edit Gemfile and update .lock files
  • cd td-agent && rake lockfile:update
• Bump up the versions of rpm & deb packages by the following command:

% cd td-agent
% rake version:update
% git diff  # Check the change log
% git commit -a

• Build packages

% rake apt:build
% rake yum:build
% rake msi:build

It assumes that Gemfile works with Bundler's multiplatform feature, so bundler must be 2.2.0 or later.