fluent / fluent-package-builder Goto Github PK
View Code? Open in Web Editor NEWtd-agent (Fluentd) Building and Packaging System
License: Apache License 2.0
td-agent (Fluentd) Building and Packaging System
License: Apache License 2.0
Although we don't changes Dockerfile for Windows, docker build
on GitHub Actions fails since yesterday.
https://github.com/clear-code/td-agent-builder/runs/736916721
Install MSYS2 and MINGW development toolchain succeeded
_____ _ _____ _ _ _ ___
| __ \ | | |_ _| | | | | | |__ \
| |__) | _| |__ _ _ | | _ __ ___| |_ __ _| | | ___ _ __ ) |
| _ / | | | '_ \| | | | | | | '_ \/ __| __/ _` | | |/ _ \ '__/ /
| | \ \ |_| | |_) | |_| |_| |_| | | \__ \ || (_| | | | __/ | / /_
|_| \_\__,_|_.__/ \__, |_____|_| |_|___/\__\__,_|_|_|\___|_||____|
__/ | _
|___/ _|_ _ __ | | o __ _| _ _
| (_) | |^| | | |(_|(_)\^/_>
re-exec error: exit status 1: output: time="2020-06-04T02:31:07Z" level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" error="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" importFolderPath="C:\\ProgramData\\docker\\tmp\\hcs403266603" path="\\\\?\\C:\\ProgramData\\docker\\windowsfilter\\2961a824bef1e7df5eedfe4d3e33fde61d6396f0abf0b24601adc418cda42486"
hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)
rake aborted!
Command failed with status (1): [docker build --tag td-agent-windows-x64 ms...]
D:/a/td-agent-builder/td-agent-builder/td-agent/Rakefile:902:in `run_docker'
D:/a/td-agent-builder/td-agent-builder/td-agent/Rakefile:822:in `block (2 levels) in define'
D:/a/td-agent-builder/td-agent-builder/td-agent/Rakefile:812:in `block (2 levels) in define'
Tasks: TOP => msi:dockerbuild
(See full trace by running task with --trace)
https://docs.fluentd.org/installation/install-by-deb install steps depends on deprecated apt-key.
apt-key will be removed Debian 11(bullseye) or later.
provide keyring package and use it.
https://docs.fluentd.org/installation/install-by-deb uses apt-key
We got the following error on building debian package:
E: td-agent: dir-or-file-in-opt opt/td-agent/
E: td-agent: dir-or-file-in-opt opt/td-agent/bin/
E: td-agent: dir-or-file-in-opt opt/td-agent/bin/bundle
E: td-agent: dir-or-file-in-opt opt/td-agent/bin/bundler
Unfortunately, they can't be suppressed by package's .override file.
I think we have two options.
--lintian-opts
of debuild
command$ cat ~/.lintian/profiles/td-agent/ubuntu.profile
Profile: td-agent/ubuntu
Extends: ubuntu/main
Disable-Tags: dir-or-file-in-opt
Currently, our CI is broken for building aarch64 packages.
This is because tzdata version requirements mismatch:
https://travis-ci.com/github/fluent-plugins-nursery/td-agent-builder/jobs/367744595#L120
td-agent is released every 3 months (maybe), so it may be a case that it was already fixed in upstream.
For evaluation purpose, nightly package is available.
Wait until new release or build on yourown.
Failed to start td-agent 4.0.0 on CentOS 8 and 7.
# cat /var/log/messages
Aug 13 18:18:00 ip-10-105-44-194 fluentd[15556]: <jemalloc>: Unsupported system page size
Aug 13 18:18:00 ip-10-105-44-194 fluentd[15556]: <jemalloc>: Unsupported system page size
Aug 13 18:18:00 ip-10-105-44-194 fluentd[15556]: [FATAL] failed to allocate memory
# yum install http://packages.treasuredata.com.s3.amazonaws.com/4/redhat/8/aarch64/td-agent-4.0.0-1.el8.aarch64.rpm -y
# systemctl start td-agent
Please let me know if any of this information is missing, as I may be missing some of it.
ref: https://www.clear-code.com/blog/2020/5/5.html (written in Japanese)
rake apt:build APT_TARGETS="ubuntu-bionic-arm64" failed
my host is ubuntu bionic on x86_64 platform
Follow the instructions in Readme to build
export TARGET="ubuntu-bionic-arm64"
sudo apt install qemu-user-static
cd /path/to/td-agent-builder
cp /usr/bin/qemu-aarch64-static td-agent/apt/${TARGET}
rake apt:build APT_TARGETS=${TARGET}
build successfully
build failed
We migrate CI for AArch64 from GitHub Actions to Travis CI to reduce build time.
But artifacts aren't uploaded yet.
We should upload them like we did it at GitHub Actions.
Currently, we use the same UpgradeCode for MSI package.
This should be valid but td-agent 3 and 4 will not be incomplatible due to upgrading Ruby version and such incompatible changes.
How do we treat it?
Not to make deb package regression such as #145 , it is useful to introduce piuparts.
ref. https://wiki.debian.org/piuparts
Consider using windows-docker-machine :
https://github.com/StefanScherer/windows-docker-machine
Default configuration file contains inappropriate path.
Fixed path by default
For example, /var/log/td-agent/td
exists.
# This section matches events whose tag is td.DATABASE.TABLE
<match td.*.*>
@type tdlog
@id output_td
apikey YOUR_API_KEY
auto_create_table
<buffer>
@type file
path /var/log/td-agent/buffer/td
</buffer>
---INITIAL INSTALL OF TD-AGENT4---
ubuntu@ip-10-0-1-167:~$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent4.sh | sh % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 851 100 851 0 0 4052 0 --:--:-- --:--:-- --:--:-- 4071
==============================
td-agent Installation Script
==============================
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3157 100 3157 0 0 33585 0 --:--:-- --:--:-- --:--:-- 33585
OK
Hit:1 http://us-west-2.ec2.ports.ubuntu.com/ubuntu-ports bionic InRelease
Hit:2 http://us-west-2.ec2.ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
Hit:3 http://us-west-2.ec2.ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
Get:4 http://packages.treasuredata.com/4/ubuntu/bionic bionic InRelease [7692 B]
Hit:5 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
Get:6 http://packages.treasuredata.com/4/ubuntu/bionic bionic/contrib arm64 Packages [682 B]
Fetched 8374 B in 0s (23.5 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
td-agent
0 upgraded, 1 newly installed, 0 to remove and 27 not upgraded.
Need to get 9843 kB of archives.
After this operation, 57.3 MB of additional disk space will be used.
Get:1 http://packages.treasuredata.com/4/ubuntu/bionic bionic/contrib arm64 td-agent arm64 4.0.0-1 [9843 kB]
Fetched 9843 kB in 1s (15.6 MB/s)
Selecting previously unselected package td-agent.
(Reading database ... 60331 files and directories currently installed.)
Preparing to unpack .../td-agent_4.0.0-1_arm64.deb ...
Unpacking td-agent (4.0.0-1) ...
Setting up td-agent (4.0.0-1) ...
Adding system user `td-agent' (UID 112) ...
Adding new group `td-agent' (GID 116) ...
Adding new user `td-agent' (UID 112) with group `td-agent' ...
Creating home directory `/var/lib/td-agent' ...
Conffile /etc/td-agent/td-agent.conf has been modified. Remain untouched.
Created symlink /etc/systemd/system/multi-user.target.wants/td-agent.service → /lib/systemd/system/td-agent.service.
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Installation completed. Happy Logging!
ubuntu@ip-10-0-1-167:~$ sudo /etc/init.d/td-agent start
sudo: /etc/init.d/td-agent: command not found
ubuntu@ip-10-0-1-167:~$ sudo /etc/init.d/td-agent status
sudo: /etc/init.d/td-agent: command not found
--- UNINSTALL OF TD-AGENT 4---
ubuntu@ip-10-0-1-167:~$ sudo apt remove td-agent
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
td-agent
0 upgraded, 0 newly installed, 1 to remove and 27 not upgraded.
After this operation, 57.3 MB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 67477 files and directories currently installed.)
Removing td-agent (4.0.0-1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
ubuntu@ip-10-0-1-167:~$
--- FAILED INSTALL OF TD-AGENT 3---
ubuntu@ip-10-0-1-167:~$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 851 100 851 0 0 4297 0 --:--:-- --:--:-- --:--:-- 4297
==============================
td-agent Installation Script
==============================
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3157 100 3157 0 0 42662 0 --:--:-- --:--:-- --:--:-- 42662
OK
Get:1 http://packages.treasuredata.com/3/ubuntu/bionic bionic InRelease [7692 B]
Hit:2 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
Get:3 http://packages.treasuredata.com/3/ubuntu/bionic bionic/contrib arm64 Packages [513 B]
Hit:4 http://us-west-2.ec2.ports.ubuntu.com/ubuntu-ports bionic InRelease
Hit:5 http://us-west-2.ec2.ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
Hit:6 http://us-west-2.ec2.ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
Fetched 8205 B in 5s (1576 B/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
td-agent is already the newest version (4.0.0-1).
0 upgraded, 0 newly installed, 0 to remove and 27 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up td-agent (4.0.0-1) ...
Updating conffile /etc/td-agent/td-agent.conf ...
cp: cannot stat '/etc/td-agent/td-agent.conf.tmpl': No such file or directory
dpkg: error processing package td-agent (--configure):
installed td-agent package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
td-agent
E: Sub-process /usr/bin/dpkg returned an error code (1)
Installation incompleted. Check above messages.
The template path in postinst script should be /opt/td-agent/share/td-agent.conf.tmpl instead of /etc/td-agent/td-agent.conf.tmpl.
On Windows, c:/var/log/td-agent/buffer/fd is created
c:/opt/td-agent/var/log/td-agent/buffer/td is used
We need to update td-agent.conf.tmpl.
We can remove unnecessary files from package to reduce the installed size.
One example is /opt/td-agent/lib/ruby/gems/2.4.0/cache
.
In addition, I noticed rpm package is 2x bigger than deb.
The size of binary is larger. For example, jemalloc and ruby are different.
$ ls -ahl /opt/td-agent/lib/
total 5.5M
drwxr-xr-x 4 root root 4.0K Feb 18 02:45 .
drwxr-xr-x 7 root root 4.0K Feb 18 02:45 ..
-rw-r--r-- 1 root root 912K May 7 14:45 libjemalloc.a
-rw-r--r-- 1 root root 912K May 7 14:45 libjemalloc_pic.a
lrwxrwxrwx 1 root root 16 May 7 14:45 libjemalloc.so -> libjemalloc.so.2
-rw-r--r-- 1 root root 552K May 7 14:45 libjemalloc.so.2
lrwxrwxrwx 1 root root 16 May 7 14:45 libruby.so -> libruby.so.2.7.1
lrwxrwxrwx 1 root root 16 May 7 14:45 libruby.so.2.7 -> libruby.so.2.7.1
-rw-r--r-- 1 root root 3.2M May 7 14:45 libruby.so.2.7.1
drwxr-xr-x 2 root root 4.0K Feb 18 02:45 pkgconfig
drwxr-xr-x 6 root root 4.0K Feb 18 02:45 ruby
$ ls -ahl /opt/td-agent/lib/
total 65M
drwxr-xr-x. 4 root root 191 Apr 13 13:15 .
drwxr-xr-x. 7 root root 72 Apr 13 13:15 ..
-rwxr-xr-x. 1 root root 25M May 8 2020 libjemalloc.a
lrwxrwxrwx. 1 root root 16 May 8 2020 libjemalloc.so -> libjemalloc.so.2
-rwxr-xr-x. 1 root root 5.8M May 8 2020 libjemalloc.so.2
-rwxr-xr-x. 1 root root 25M May 8 2020 libjemalloc_pic.a
lrwxrwxrwx. 1 root root 16 May 8 2020 libruby.so -> libruby.so.2.7.1
lrwxrwxrwx. 1 root root 16 May 8 2020 libruby.so.2.7 -> libruby.so.2.7.1
-rwxr-xr-x. 1 root root 9.6M May 8 2020 libruby.so.2.7.1
drwxr-xr-x. 2 root root 44 Apr 13 13:15 pkgconfig
drwxr-xr-x. 6 root root 67 Apr 13 13:15 ruby
Of course, other extension gems are also bigger.
Chocolatey is a modern package management system (like apt or yum on GNU/Linux) for Windows: https://chocolatey.org/
Most packages for this system is implemeted as a wrapper of an original installer (.msi or .exe).
I'm also planning to use same way to implement td-agent package for it.
rake msi:build
fails to build.
rake msi:build
succeeds.
ref. #24 (comment)
Fetching fluent-plugin-parser-winevt_xml-0.2.2.gem
Successfully installed mini_portile2-2.4.0
ERROR: Error installing C:/td-agent-4.0.0.rc1/td-agent/downloads/plugin_gems/26-fluent-plugin-windows-eventlog-0.5.4.gem:
The last version of nokogiri (>= 1.10, < 1.12) to support your Ruby & RubyGems was 1.10.9. Try installing it with `gem install nokogiri -v 1.10.9` and then running the current command again
nokogiri requires Ruby version >= 2.3, < 2.7.dev. The current ruby version is 2.7.1.83.
rake aborted!
We have package scripts for Debian that are comes from omnibus-td-agent:
Some of them aren't needed in the new build system since they are for avoiding Omibius's limitations (maybe). I want to remove needless codes and keep it simple.
I compared dependency of librdkafka in both omnibus-td-agent and td-agent-builder.
I found td-agent-builder version lacks some dependencies:
omnibus-td-agent:
$ ldd ./embedded/lib/ruby/gems/2.4.0/gems/rdkafka-0.7.0/ext/librdkafka.so:
linux-vdso.so.1 => (0x00007ffbffffe000)
libm.so.6 => /lib64/libm.so.6 (0x00007efbf77ae000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007efbf7591000)
libssl.so.1.0.0 => /opt/td-agent/embedded/lib/libssl.so.1.0.0 (0x00007efbf7325000)
libcrypto.so.1.0.0 => /opt/td-agent/embedded/lib/libcrypto.so.1.0.0 (0x00007efbf6eec000)
libz.so.1 => /opt/td-agent/embedded/lib/libz.so.1 (0x00007efbf6cd5000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007efbf6ad1000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efbf68b5000)
librt.so.1 => /lib64/librt.so.1 (0x00007efbf66ad000)
libc.so.6 => /lib64/libc.so.6 (0x00007efbf62df000)
/lib64/ld-linux-x86-64.so.2 (0x00007efbf7ddd000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007efbf60c6000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007efbf5e8f000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007efbf5c42000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007efbf5959000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007efbf5726000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007efbf5522000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007efbf5312000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007efbf510f000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007efbf4f0b000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007efbf4ce4000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007efbf4a82000)
td-agent-builder:
$ ldd ./lib/ruby/gems/2.4.0/gems/rdkafka-0.7.0/ext/
linux-vdso.so.1 => (0x00007ffbffffe000)
liblz4.so.1 => /lib64/liblz4.so.1 (0x00007efbf78b2000)
libm.so.6 => /lib64/libm.so.6 (0x00007efbf75b0000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007efbf730c000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007efbf709a000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007efbf6c37000)
libz.so.1 => /lib64/libz.so.1 (0x00007efbf6a21000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007efbf681d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efbf6601000)
librt.so.1 => /lib64/librt.so.1 (0x00007efbf63f9000)
libc.so.6 => /lib64/libc.so.6 (0x00007efbf602b000)
/lib64/ld-linux-x86-64.so.2 (0x00007efbf7ddd000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007efbf5dde000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007efbf5af5000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007efbf58f1000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007efbf56be000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007efbf54ae000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007efbf52aa000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007efbf5091000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007efbf4e6a000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007efbf4c08000)
Following dependencies are missing:
In addition, CentOS 8 package lacks more dependencies:
$ ldd lib/ruby/gems/2.4.0/gems/rdkafka-0.7.0/ext/librdkafka.so
linux-vdso.so.1 (0x00007ffc00100000)
liblz4.so.1 => /lib64/liblz4.so.1 (0x00007efbfefa9000)
libm.so.6 => /lib64/libm.so.6 (0x00007efbfec27000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007efbfe983000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007efbfe6ef000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007efbfe211000)
libz.so.1 => /lib64/libz.so.1 (0x00007efbfdffa000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007efbfddf6000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efbfdbd6000)
librt.so.1 => /lib64/librt.so.1 (0x00007efbfd9cd000)
libc.so.6 => /lib64/libc.so.6 (0x00007efbfd60a000)
/lib64/ld-linux-x86-64.so.2 (0x00007efbff4d6000)
Currently it requires to register/unregister manually:
https://docs.fluentd.org/installation/install-by-msi#step-4-run-td-agent-as-windows-service
Specific APT_TARGET doesn't finish to build.
There is no workaround for specific container currently.
rake apt:build APT_TARGETS="ubuntu-focal-arm64"
rake apt:build APT_TARGETS="ubuntu-focal-arm64" success.
(It is able to skip lintian process if container's lintian is problematic version.)
rake apt:build process stalls.
Currently we get following lintian errors & warnings:
E: td-agent changes: bad-distribution-in-changes-file unstable
E: td-agent: embedded-library opt/td-agent/lib/ruby/2.4.0/x86_64-linux/psych.so: libyaml
E: td-agent: extended-description-is-empty
E: td-agent: no-copyright-file
W: td-agent: binary-without-manpage usr/bin/td
W: td-agent: binary-without-manpage usr/sbin/td-agent
W: td-agent: binary-without-manpage usr/sbin/td-agent-gem
W: td-agent: maintainer-script-needs-depends-on-adduser postinst
W: td-agent: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:60
W: td-agent: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:63
W: td-agent source: missing-debian-source-format
W: td-agent source: no-debian-copyright
W: td-agent: package-has-unnecessary-activation-of-ldconfig-trigger
W: td-agent: possibly-insecure-handling-of-tmp-files-in-maintainer-script postinst:62
W: td-agent source: unused-build-dependency-on-cdbs
W: td-agent: unusual-interpreter opt/td-agent/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/templates/Executable #!<%=
W: td-agent: unusual-interpreter opt/td-agent/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/templates/Executable.bundler #!<%=
W: td-agent: unusual-interpreter opt/td-agent/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/templates/Executable.standalone #!<%=
W: td-agent: uses-dpkg-database-directly postinst (line 37)
W: td-agent: uses-dpkg-database-directly postinst (line 40)
N: 360 tags overridden (224 errors, 136 warnings)
In addition, we are now suppressing more errors/warning by overriding lintian tags:
$ cat td-agent/debian/td-agent.lintian-overrides
td-agent: package-contains-vcs-control-file
td-agent: windows-devel-file-in-package
td-agent: dir-or-file-in-build-tree
td-agent: ruby-script-but-no-ruby-dep
td-agent: script-not-executable
td-agent: python-script-but-no-python-dep
td-agent: executable-not-elf-or-script
td-agent: missing-dep-for-interpreter
td-agent: gz-file-not-gzip
td-agent: no-upstream-changelog
td-agent: library-not-linked-against-libc
td-agent: wrong-path-for-interpreter
td-agent: duplicate-font-file
td-agent: jar-not-in-usr-share
td-agent: incorrect-libdir-in-la-file
We should ensure to address them by proper ways.
AWS provides Amazon Linux 2 images for amd64/arm64v8: https://hub.docker.com/_/amazonlinux
The problem is Amazon Linux 2 is similar to CentOS 7 but not the same.
amazon-linux-extras
to install ruby 2.4/2.6. No centos-release-scl
./etc/redhat-release
. Maybe, fixed distribution/distribution_version is enoughThis build system currently supports only modern distributions:
But at least CentOS 6 should be also supported since we have users who use it.
Everything works when using the example windows_eventlog config but not when im using my own or the example windows_eventlog2 config.
plugin versions:
winevt_c' 0.8.1
fluent-plugin-windows-eventlog 0.7.0
Tried on Windows 10 and Windows Server 2019
Errorlog:
C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': 126: The specified module could not be found. - C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/winevt_c-0.8.1/lib/winevt/winevt.so (LoadError)
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/winevt_c-0.8.1/lib/winevt.rb:4:in `rescue in <top (required)>'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/winevt_c-0.8.1/lib/winevt.rb:1:in `<top (required)>'
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `require'
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `rescue in require'
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:39:in `require'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluent-plugin-windows-eventlog-0.7.0/lib/fluent/plugin/in_windows_eventlog2.rb:1:in `<top (required)>'
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/registry.rb:102:in `block in search'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/registry.rb:99:in `each'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/registry.rb:99:in `search'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/registry.rb:44:in `lookup'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin.rb:155:in `new_impl'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/plugin.rb:105:in `new_input'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:312:in `add_source'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:158:in `block in configure'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:152:in `each'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:152:in `configure'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/engine.rb:105:in `configure'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/engine.rb:80:in `run_configure'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/supervisor.rb:551:in `run_supervisor'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/command/fluentd.rb:330:in `<top (required)>'
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from C:/opt/td-agent/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from C:/opt/td-agent/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/bin/fluentd:8:in `<top (required)>'
from C:/opt/td-agent/bin/fluentd:23:in `load'
from C:/opt/td-agent/bin/fluentd:23:in `<main>'
Until bundler 2.2.0?, Automultiplatform is not work as expected, so we can not
use correct .lock file for each build environment
Don't remove .lock file and use proper .lock file separately.
Currently, we remove .lock file
This issue was fixed in upstream, but not released yet.
rubygems/bundler#7580
After 4.0.1 release, users can use with MSYS2 to build C extension bundled gem.
On td-agent 4.0.0, there is a bug due to lack of *.dll.a:
#130
Currently, installation testcases should execute package installation only.
We should do more installation testcases.
e.g) Serverspec.
td-agent vX.Y.Z represents a curated collection of GEMs that have been tested prior to their release.
Our project needs to add some additional plugins to support our requirements with out invalidating the integrity of the release.
Our CI pipeline takes the release, constructs a Gemfile.lock and then tries to install additional GEMS using td-agent-gem to ensure we don't replace any embedded/tested GEMS.
It is desirable if td-agent could create the lock file and publish it with the release.
https://github.com/treasure-data/omnibus-td-agent/tree/master/templates/package-scripts/td-agent/rpm
was removed, but there may be a possibility that it is not migrated properly.
rake apt:build stalls because of lintian's check with encrypted zip files
rake apt:build
succeeds without stalling.
rake apt:build
stalls. It is caused by stopped lintian check process.
184023 pts/0 S+ 0:09 lintian --profile td-agent/ubuntu td-agent_3.7.0-1_amd64.changes
185063 pts/0 S+ 0:00 unzip -t /tmp/lintian-pool-cdSp7Kwv7Q/pool/t/td-agent/td-agent_3.7.0-1_amd64_binary/unpacked/opt/td-agent/lib/ruby/gems/2.4.0/gems/rubyzip-1.3.0/test/data/zipWithEncryption.zip
17:07
|-containerd-+-containerd-shim-+-build.sh---debuild-+-lintian --profi---unzip
| | | `-tee
| | `-9*[{containerd-shim}]
| `-24*[{containerd}]
This situation occurs on Ubuntu 20.04(focal)
It seems that this is the bug of lintian.
It was already reported as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=935292 and fixed in lintian/2.67.0.
Workaround is:
td-agent 4 should include Ruby 2.6.
When I tried to install a C extension of Ruby on Windows, I got the following error:
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lx64-msvcrt-ruby270
collect2.exe: error: ld returned 1 exit status
make: *** [Makefile:261: bson_native.so] 繧ィ繝ゥ繝シ 1
make failed, exit code 2
Gem files will remain installed in C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/bson-4.9.4 for inspection.
Results logged to C:/opt/td-agent/lib/ruby/gems/2.7.0/extensions/x64-mingw32/2.7.0/bson-4.9.4/gem_make.out
The cause is that td-agent-builder removes libx64-msvcrt-ruby270.dll.a in RubyInstaller.
Build "ubuntu-bionic-arm64" target failed, with logs "175 errors, 61 warnings" at last.
I aslo checked td-agent/apt/repositories/ directory, it's empty.
The builder (container) is still running at last.
my vps host is ubuntu bionic on x86_64 platform, 1CPU and 1 GB RAM.
Follow the instructions in Readme to build
export TARGET_BASE="ubuntu-bionic"
sudo apt install qemu-user-static
cd /path/to/td-agent-builder
cp /usr/bin/qemu-aarch64-static td-agent/apt/${TARGET_BASE}
rake apt:build APT_TARGETS="${TARGET_BASE}-arm64"
Build successfully
Build failed
I check the builder logs using "docker logs -f 860a5cd80e74" and find there are some ruby extensions failed.
At last, it failed with "175 errors, 61 warnings"
This is the log file dumped by "docker logs 860a5cd80e74 > td-agent-builder.log "
td-agent-builder.log
rake apt:build doesn't finish.
rake apt:build APT_TARGETS="ubuntu-focal-arm64"
lintian doesn't stall.
build process stalls because of lintian.
dh_builddeb
dpkg-deb: building package 'td-agent' in '../td-agent_3.7.1-1_arm64.deb'.
dpkg-deb: building package 'td-agent-dbgsym' in 'debian/.debhelper/scratch-space/build-td-agent/td-agent-dbgsym_3.7.1-1_arm64.deb'.
Renaming td-agent-dbgsym_3.7.1-1_arm64.deb to td-agent-dbgsym_3.7.1-1_arm64.ddeb
dpkg-genbuildinfo
dpkg-genchanges >../td-agent_3.7.1-1_arm64.changes
dpkg-genchanges: info: including full source code in upload
dpkg-source --after-build .
dpkg-buildpackage: info: full upload (original source is included)
Now running lintian --debug --verbose --profile td-agent/ubuntu td-agent_3.7.1-1_arm64.changes ...
pstree on host:
├─dockerd─┬─docker-containe─┬─docker-containe─┬─build.sh─┬─debuild─┬─lintian─┬─nproc
│ │ │ │ │ │ └─{lintian}
│ │ │ │ │ ├─tee───{tee}
│ │ │ │ │ └─{debuild}
│ │ │ │ └─{build.sh}
│ │ │ └─8*[{docker-containe}]
│ │ └─18*[{docker-containe}]
│ └─19*[{dockerd}]
ref. #62
rake yum:build A brief survey:
https://packages.ubuntu.com/search?suite=focal&arch=arm64&searchon=names&keywords=lintian
lintian 2.62 is used.
https://bugs.launchpad.net/ubuntu/+source/lintian
It seems that it is not already known issue.
https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=both;repeatmerged=on;src=lintian
It seems that it is not already known issue even though archive.(I've expected something to do with 2.62-2.76)
UPDATE: 2020/11/01, This issue will be fixed in Ubuntu 21.04. (lintian 2.92.0 or later version)
C:\opt\td-agent>dir C:\opt\td-agent\opt\td-agent\share
ドライブ C のボリューム ラベルは Windows です
ボリューム シリアル番号は 8E7B-B5D5 です
C:\opt\td-agent\opt\td-agent\share のディレクトリ
2020/07/09 16:51 <DIR> .
2020/07/09 16:51 <DIR> ..
2020/07/09 16:41 27 td-agent-ruby.conf
2020/07/09 16:41 2,508 td-agent.conf.tmpl
2 個のファイル 2,535 バイト
2 個のディレクトリ 115,490,639,872 バイトの空き領域
Currently, we still use td-agent 3.6.0 based versions.
We should follow and upgrade using software versions.
treasure-data/omnibus-td-agent#243
Ubuntu 20.04 LTS will be released in 2020/4/24.
We should also support this version of Ubuntu release.
@ashie How about including jemalloc in td-agent?
jemalloc changes internal memory management, so using same version is better for trouble-shooting.
Since td-agent-builder is based on omnibu-td-agent, it installs most of files into /opt. But it's not recommended on Debian:
dir-or-file-in-opt
errors can't be suppress by package's .lintian-overrides file.
Currently we suppress it forcedly by user's lintian configuration because a lot of such errors make it hard that we find other errors or warnings, but it's not desired way.
We should consider put files into standard locations on GNU/Linux.
In addition, C:\opt isn't familiar on Windows.
Because of gem dependency, td-agent requires tzinfo 2.x but , travis gem requires tzinfo 1.x.
As a result, CI can't work as expected.
CI on Travis works as expected.
CI on Travis always fails.
We need to wait until activesupport 6.1 release.
Automultiplatform again
rubygems/bundler#7580
was merged, it will be shipped as 6.1.0. we wait https://github.com/rails/rails/milestone/69 6.1.0 milestone.
We need Gemfile integration for simplicity
maintain Single Gemfile and Gemfile.lock for simplicity
There are two Gemfile depend on environment.
gemfiles/linux/Gemfile
gemfile/windows/Gemfile
After bundler 2.2.0 is released, auto multiplatform is enabled, so we can start to integrate Gemfile.
Currently, gem installation is proceeded by gem install
and these files:
https://github.com/fluent-plugins-nursery/td-agent-builder/blob/master/td-agent/core_gems.rb
https://github.com/fluent-plugins-nursery/td-agent-builder/blob/master/td-agent/plugin_gems.rb
Recent rubygems can handle Gemfile with gem install --file
:
https://github.com/rubygems/rubygems/blob/6d167c1df771c7ef3f2f7c4fb12c4416c0a384b6/lib/rubygems/install_update_options.rb#L136-L151
Using gem install --file Gemfile
is one of the way to migrate Gemfile based installation.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.