net-ssh / bcrypt_pbkdf-ruby Goto Github PK

With the planned 1.1 we might be dropping older rubies on windows. (2.0 and 2.1) while those EOL 3 years ago, we might want to bump major version.
We probably still compile/run on very old rubies, just our windows binary precompiler doesn't not build for them.

One thing is that our docs in net-ssh are documenting < 2.0, so bumping the version might be more trouble than worth.

https://github.com/net-ssh/net-ssh/blob/fc91300d7ee6299ba2fe4039cc8085aec9edb7f3/lib/net/ssh/authentication/ed25519_loader.rb#L27

In the last few days, Github updated their Windows 2022 (windows-latest) runner, and it looks like something in there has broken the bcrypt_pbkdf build.

The build error I'm seeing is:

current directory:
C:/hostedtoolcache/windows/Ruby/3.2.4/x64/lib/ruby/gems/3.2.0/gems/bcrypt_pbkdf-1.1.0/ext/mri
make.exe DESTDIR\= sitearchdir\=./.gem.20240516-4380-2nneud
sitelibdir\=./.gem.20240516-4380-2nneud
generating bcrypt_pbkdf_ext-x64-mingw-ucrt.def
compiling bcrypt_pbkdf.c
compiling bcrypt_pbkdf_ext.c
compiling blowfish.c
compiling explicit_bzero.c
explicit_bzero.c: In function 'explicit_bzero':
explicit_bzero.c:18:9: error: implicit declaration of function 'bzero'
[-Wimplicit-function-declaration]
   18 |         bzero(p, n);
      |         ^~~~~
explicit_bzero.c:18:9: warning: incompatible implicit declaration of built-in
function 'bzero' [-Wbuiltin-declaration-mismatch]
make: *** [Makefile:248: explicit_bzero.o] Error 1

I'm seeing this in the integration tests for rails/tailwindcss-rails.

• Passing build from May 8: https://github.com/rails/tailwindcss-rails/actions/runs/8998333655/job/24718244606
• Failing build from this morning: https://github.com/rails/tailwindcss-rails/actions/runs/9092067481/job/25058810906#step:4:564

The runner images are described here:

• Passing: https://github.com/actions/runner-images/blob/win22/20240421.1/images/windows/Windows2022-Readme.md
• Failing: https://github.com/actions/runner-images/blob/win22/20240514.3/images/windows/Windows2022-Readme.md

I'm sorry I'm not able to narrow this down any further today.

Hi,

we're experiencing an issue when bcrypt_pbkdf-ruby is built on a bitnami docker image (bitnami/ruby:2.5.8-prod-debian-10-r236), it compiles fine but fails at runtime with:

symbol lookup error: /app/lib/bcrypt_pbkdf_ext.so: undefined symbol: SHA512Init

However if compiled with the "standard" ruby image (ruby:2.5.8) it works fine.

Both docker images use the same OpenSSL version (OpenSSL 1.1.1d 10 Sep 2019) and same GCC version (version 8.3.0 (Debian 8.3.0-6))

In order to reproduce it I created two Dockerfiles:

bitnami/ruby:2.5.8-prod-debian-10-r236

FROM bitnami/ruby:2.5.8-prod-debian-10-r236

RUN apt-get update \
        && apt-get install -y --no-install-recommends --no-install-suggests \
                            build-essential \
                            git \
                            openssh-client \
        && gem install bundler -v 1.17.3

WORKDIR /app

RUN git clone https://github.com/net-ssh/bcrypt_pbkdf-ruby.git /app \
    && git checkout v1.0.1 \
    && bundle

RUN rake compile
RUN rake test

CMD ["/bin/bash"]

ruby:2.5.8

FROM ruby:2.5.8

RUN apt-get update \
	 && apt-get install -y --no-install-recommends --no-install-suggests \
                             build-essential \
                             git \
                             openssh-client \
         && gem install bundler -v 1.17.3

RUN mkdir /app

WORKDIR /app

RUN git clone https://github.com/net-ssh/bcrypt_pbkdf-ruby.git /app \
         && git checkout v1.0.1 \
         && bundle

RUN rake compile
RUN rake test

CMD ["/bin/bash"]

When I inspect the built "bcrypt_pbkdf_ext.so" using nm on the bitnami container I get:

root@5a6e512b11bb:/app/lib# nm -gD bcrypt_pbkdf_ext.so
0000000000002491 T Blowfish_decipher
0000000000001b1f T Blowfish_encipher
0000000000002ee7 T Blowfish_expand0state
0000000000003061 T Blowfish_expandstate
0000000000002e03 T Blowfish_initstate
0000000000002e64 T Blowfish_stream2word
0000000000001a9a T Init_bcrypt_pbkdf_ext
                 U SHA512Final
                 U SHA512Init
                 U SHA512Update
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w __cxa_finalize
                 w __gmon_start__
0000000000001295 T bcrypt_hash
0000000000001504 T bcrypt_pbkdf
00000000000037ea T blf_cbc_decrypt
000000000000363d T blf_cbc_encrypt
0000000000003317 T blf_dec
00000000000034d9 T blf_ecb_decrypt
0000000000003375 T blf_ecb_encrypt
00000000000032b9 T blf_enc
000000000000327d T blf_key
000000000000985b T crypto_hash_sha512
000000000000980f T crypto_hash_sha512_final
0000000000009649 T crypto_hash_sha512_init
0000000000009690 T crypto_hash_sha512_update
0000000000003b1b T explicit_bzero
                 U memcpy
                 U memset
                 U rb_cObject
                 U rb_define_class_under
                 U rb_define_module
                 U rb_define_singleton_method
                 U rb_num2ulong
                 U rb_str_new
                 U rb_string_value_ptr
                 U ruby_xfree
                 U ruby_xmalloc

On the "standard" ruby container I get:

root@ec3d13ecaef3:/app/lib# nm -gD bcrypt_pbkdf_ext.so
0000000000001c80 T Blowfish_decipher
0000000000001880 T Blowfish_encipher
0000000000002160 T Blowfish_expand0state
0000000000002260 T Blowfish_expandstate
0000000000002080 T Blowfish_initstate
00000000000020c0 T Blowfish_stream2word
0000000000001810 T Init_bcrypt_pbkdf_ext
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w __cxa_finalize
                 w __gmon_start__
0000000000001270 T bcrypt_hash
00000000000013c0 T bcrypt_pbkdf
0000000000002750 T blf_cbc_decrypt
0000000000002610 T blf_cbc_encrypt
0000000000002460 T blf_dec
0000000000002560 T blf_ecb_decrypt
00000000000024b0 T blf_ecb_encrypt
0000000000002410 T blf_enc
00000000000023e0 T blf_key
0000000000005d30 T crypto_hash_sha512
0000000000005c50 T crypto_hash_sha512_final
0000000000005a20 T crypto_hash_sha512_init
0000000000005a60 T crypto_hash_sha512_update
0000000000002900 T explicit_bzero
                 U memcpy
                 U memset
                 U rb_cObject
                 U rb_define_class_under
                 U rb_define_module
                 U rb_define_singleton_method
                 U rb_num2ulong
                 U rb_str_new
                 U rb_string_value_ptr
                 U ruby_xfree
                 U ruby_xmalloc

The problem seems to be with the inline functions SHA512Final, SHA512Init, SHA512Update defined in https://github.com/net-ssh/bcrypt_pbkdf-ruby/blob/v1.0.1/ext/mri/sha2.h

Inside the bitnami image, those inline methods end up as undefined symbols while inside the "standard" ruby image the inlining seems to work properly.

My simple fix was to declare those inline functions as inline static (see v1.0.1...at-point:fix-inline-static)

As reported on gentoo's bug tracker, 32 bit x86 compiling is broken https://bugs.gentoo.org/629164

Including up to the latest tag

Installing bcrypt_pbkdf 1.0.1 (was 1.1.0.rc1) with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/var/tmp/portage/net-analyzer/metasploit-9999/work/all/vendor/ruby/2.5.0/gems/bcrypt_pbkdf-1.0.1/ext/mri
/usr/bin/ruby25 -r ./siteconf20200914-252-mtxmyv.rb extconf.rb
creating Makefile

current directory:
/var/tmp/portage/net-analyzer/metasploit-9999/work/all/vendor/ruby/2.5.0/gems/bcrypt_pbkdf-1.0.1/ext/mri
make "DESTDIR=" clean

current directory:
/var/tmp/portage/net-analyzer/metasploit-9999/work/all/vendor/ruby/2.5.0/gems/bcrypt_pbkdf-1.0.1/ext/mri
make "DESTDIR="
compiling bcrypt_pbkdf.c
compiling bcrypt_pbkdf_ext.c
compiling blowfish.c
compiling explicit_bzero.c
compiling hash_sha512.c
linking shared-object bcrypt_pbkdf_ext.so
/usr/lib/gcc/i686-pc-linux-gnu/9.3.0/../../../../i686-pc-linux-gnu/bin/ld:
bcrypt_pbkdf.o: in function `bcrypt_pbkdf':
bcrypt_pbkdf.c:(.text+0x221): undefined reference to `SHA512Update'
/usr/lib/gcc/i686-pc-linux-gnu/9.3.0/../../../../i686-pc-linux-gnu/bin/ld:
bcrypt_pbkdf.c:(.text+0x2b1): undefined reference to `SHA512Update'
/usr/lib/gcc/i686-pc-linux-gnu/9.3.0/../../../../i686-pc-linux-gnu/bin/ld:
bcrypt_pbkdf.c:(.text+0x2c3): undefined reference to `SHA512Update'
/usr/lib/gcc/i686-pc-linux-gnu/9.3.0/../../../../i686-pc-linux-gnu/bin/ld:
bcrypt_pbkdf.c:(.text+0x32c): undefined reference to `SHA512Update'
collect2: error: ld returned 1 exit status
make: *** [Makefile:259: bcrypt_pbkdf_ext.so] Error 1

Can you please remove the inline definitions as suggested in the gentoo bug?

Hey there @mfazekas,

It's been a while since the 1.1 RC release came out and we've been shipping that in Chef Infra Client without any problems. Any chance this release could get pushed up to Rubygems as full release?

How to reproduce:

1. Create a new rails app: rails new app_name
2. add the following gems in the Gemfile:
   gem 'net-sftp', '>4.0.0'
   gem 'ed25519', '>1.2.0'
   gem 'bcrypt_pbkdf', '~>1.1.0'
3. bundle
4. rails c

The output:
C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in require': cannot load such file -- bcrypt_pbkdf_ext (LoadError) from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies.rb:332:in block in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies.rb:299:in load_dependency'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies.rb:332:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bcrypt_pbkdf-1.1.0-x64-mingw32/lib/bcrypt_pbkdf.rb:5:in rescue in '
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bcrypt_pbkdf-1.1.0-x64-mingw32/lib/bcrypt_pbkdf.rb:1:in <main>' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:60:in block (2 levels) in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:55:in each' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:55:in block in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:44:in each' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:44:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler.rb:176:in require' from C:/Temp/test-gem/config/application.rb:7:in '
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command/actions.rb:22:in require_application!' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command/actions.rb:14:in require_application_and_environment!'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/commands/console/console_command.rb:101:in perform' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in run'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in invoke_command' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in dispatch'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command/base.rb:69:in perform' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command.rb:48:in invoke'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/commands.rb:18:in <main>' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' from bin/rails:4:in '
C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in require': cannot load such file -- 3.0/bcrypt_pbkdf_ext (LoadError) from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies.rb:332:in block in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies.rb:299:in load_dependency'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.3/lib/active_support/dependencies.rb:332:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bcrypt_pbkdf-1.1.0-x64-mingw32/lib/bcrypt_pbkdf.rb:3:in '
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:60:in block (2 levels) in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:55:in each'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:55:in block in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:44:in each'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler/runtime.rb:44:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bundler-2.3.3/lib/bundler.rb:176:in require'
from C:/Temp/test-gem/config/application.rb:7:in <main>' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command/actions.rb:22:in require_application!'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command/actions.rb:14:in require_application_and_environment!' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/commands/console/console_command.rb:101:in perform'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in run' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in invoke_command'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in dispatch' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command/base.rb:69:in perform'
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/command.rb:48:in invoke' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/railties-6.1.7.3/lib/rails/commands.rb:18:in '
from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' from C:/Ruby/3.0.5-1/lib/ruby/gems/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require'
from bin/rails:4:in `'

The gem does not build the "bcrypt_pbkdf_ext.so" file for ruby 3.0.
Any idea how to do this?

With the release of Ruby 3.0 it should get added to the list Ruby releases to cross compile for.