Error 400 on SERVER: Could not find data item levent_parent_upload_dir in any Hiera data file and no default supplied about hiera-eyaml HOT 2 CLOSED

problem solved.
default gem install uses the wrong gem environment. puppet uses a different one.
use /opt/puppet/bin/gem to instal hiera-eyaml instead of default /usr/local/bin/gem

from hiera-eyaml.

Encrypt/decrypt locally on Puppet master works well here however when called from remote node 'Error 400' is displayed. Already reinstalled multiple times with no lucky. Appreciate feedback.

Environment version:
# puppet --version
4.3.1
# hiera --version
3.0.5

Content of "/etc/puppetlabs/code/hiera.yaml":

---
:backends:
- eyaml

:hierarchy:
- "hieradata/nodes/%{::trusted.certname}"
- "modules/%{calling_module}/hieradata/%{::osfamily}"
- "modules/%{calling_module}/hieradata/%{osfamily}"
- "modules/%{calling_module}/hieradata/defaults"
- "modules/%{calling_module}/hieradata/%{calling_module}"
- hieradata/defaults

:eyaml:
:datadir : /etc/puppetlabs/code/environments/%{environment}
:pkcs7_private_key : /etc/puppetlabs/puppet/ssl/keys/private_key.pkcs7.pem
:pkcs7_public_key : /etc/puppetlabs/puppet/ssl/keys/public_key.pkcs7.pem

All gems installed on Puppet master (output for "/opt/puppetlabs/bin/gem list"):

*** LOCAL GEMS ***

bigdecimal (1.2.4)
colored (1.2)
command_line_reporter (3.3.5)
cri (2.6.1)
deep_merge (1.0.1)
facter (3.1.3)
faraday (0.9.1)
faraday_middleware (0.9.1)
faraday_middleware-multi_json (0.0.6)
hiera (3.0.5)
hiera-eyaml (2.1.0)
highline (1.6.19)
hocon (0.9.3)
io-console (0.4.3)
json (1.8.1)
log4r (1.1.10)
mime-types (1.25.1)
minitar (0.5.4)
minitest (4.7.5)
multi_json (1.11.2, 1.11.0)
multipart-post (2.0.0)
net-ssh (2.9.2)
pe-razor-client (1.1.0)
psych (2.0.5)
puppet (4.3.1)
puppet_forge (2.1.1)
r10k (2.1.1)
rake (10.1.0)
rdoc (4.1.0)
rest-client (1.6.9)
rugged (0.21.4)
semantic_puppet (0.1.0)
stomp (1.3.3)
test-unit (2.1.7.0)
trollop (2.1.2)

Plain text content of eyaml data file:

---
lab::file : ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAbf+YrGDeecEz2ImV1Zf2ynJtls7cDQrLwmobpQxkSPqbQSK+0320V9x6NRfnPWGNnR3JqyK+wcXuaaafoSHYpvAezi8f1iLAD0kojdjBUrI5uFeCxNysYAEM32Zx+hAKH9rtsvI5y8MbkUyXGmDs4/4BSwiUHQYKv1XG2Ui0L2VRFefomsQGzCKSmpzja6hyLUa/w48Oml6qScFBcXmJBIESQhnHKKia9T9qAZNDZOydoG6vWWaCyboB9X0u6XtZsmuXZy96x7pdmHXx7qUqbaA3e/BCqDwg5VLqWjOf2YwHZbwLTU1GSTZe47XC8w3vjPB+5b5svGjEUHlbGLhlmDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBp8QjQCbByhkw2tFadyXSGgBAOIHk4+qcDdymNDmKJ17Vq]

Decription performed successfully locally on Puppet Master

1. by 'hiera' command line:

# hiera lab::file calling_module=ehiera environment=production
hahaha

2. by 'eyaml edit' tool:

lab::file : DEC(1)::PKCS7[hahaha]!

3. by 'puppet apply' locally on Puppet Master:

# puppet apply -v tests/init.pp calling_module=ehiera environment=production
Notice: /Stage[main]/Ehiera::Create_file/File[/tmp/hahaha]/ensure: created
Notice: Applied catalog in 1.07 seconds

Error when called from remote node:

# puppet agent -t -v
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item lab::file in any Hiera data file and no default supplied at /etc/puppetlabs/code/environments/production/modules/ehiera/manifests/params.pp:13:13 on node automation01-cb2f46d3-9378-4f87-9f3c-1f402efd41c6.arq.lab

from hiera-eyaml.