charmed-osm / layer-sshproxy Goto Github PK

The proxy charm code is run with Python 2, which causes UTF-8 encoding issues. Calling action_set with e.g. { 'output': u'This is a test! \u25cf äöüß æøå ÄÖÜ ÆØÅ' } leads to an exception:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u25cf' in position 16: ordinal not in range(128)
This is particularly a problem when "output" comes from the stdout of an external program. Once it contains non-ASCII characters, setting the output in action_set fails.

To reproduce the issue:
cmd = ['./action-set']
values = { 'output': u'This is a test! \u25cf äöüß æøå ÄÖÜ ÆØÅ' }
for k, v in list(values.items()):
cmd.append('{}={}'.format(k, v))
subprocess.check_call(cmd)

Using this "action-set" replacement script:
#!/bin/bash
while [ $# -gt 0 ] ; do
echo "Argument: $1"
shift
done

Output when run with Python 2:
Traceback (most recent call last):
File "./t3", line 74, in
cmd.append('{}={}'.format(k, v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u25cf' in position 16: ordinal not in range(128)

Output when run with Python 3:
Argument: output=This is a test! ● äöüß æøå ÄÖÜ ÆØÅ

Using '.encode("utf-8")' creates behaviour being incompatible between Python 2 and 3
(here: values = { 'output': u'This is a test! \u25cf äöüß æøå ÄÖÜ ÆØÅ'.encode('utf-8') }):
Python 2:
Argument: output=This is a test! ● äöüß æøå ÄÖÜ ÆØÅ
Python 3:
Argument: output=b'This is a test! \xe2\x97\x8f \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f \xc3\xa6\xc3\xb8\xc3\xa5 \xc3\x84\xc3\x96\xc3\x9c \xc3\x86\xc3\x98\xc3\x85'

sshproxy.py lacks an "import traceback". When something goes wrong in verify_ssh_credentials(), the call to traceback.format_exc() leads to an exception.

Example:
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed Traceback (most recent call last):
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed File "/var/lib/juju/agents/unit-simulamet-oai-epc-b-spgw-c-aa-0/charm/hooks/config-changed", line 22, in
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed main()
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed File "/usr/local/lib/python3.5/dist-packages/charms/reactive/init.py", line 74, in main
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed bus.dispatch(restricted=restricted_mode)
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 390, in dispatch
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed _invoke(other_handlers)
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 359, in _invoke
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed handler.invoke()
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 181, in invoke
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed self._action(*args)
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed File "/var/lib/juju/agents/unit-simulamet-oai-epc-b-spgw-c-aa-0/charm/reactive/sshproxy.py", line 81, in ssh_configured
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed (verified, output) = charms.sshproxy.verify_ssh_credentials()
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed File "lib/charms/sshproxy.py", line 109, in verify_ssh_credentials
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed tb = traceback.format_exc()
unit-simulamet-oai-epc-b-spgw-c-aa-0: 12:50:02 DEBUG unit.simulamet-oai-epc-b-spgw-c-aa/0.config-changed NameError: name 'traceback' is not defined

Trying to get the value of a parameter with action_get() fails with an exception, since the call to the program action-get fails with exit code 1. However, there is no information about what went wrong with the call: subprocess.CalledProcessError: Command '['action-get', 'cassandra-server-ip', '--format=json']' returned non-zero exit status 1

Is there any possibility to get some information about what is going wrong with "action-get"?

My Charm:

action.yaml:
...
configure-hss:
description: Configures the HSS
params:
cassandra-server-ip:
description: Cassandra Server IP
data-type: string
default: 172.16.6.129
required:
- cassandra-server-ip
...

The code:
...
@when('actions.configure-cassandra')
...
def configure_cassandra():
cassandraServerIP = action_get('cassandra-server-ip')
...

VNF:
...
# ====== Juju Charm ===============================================
vdu-configuration:
juju:
charm: hsscharm
initial-config-primitive:
- seq: '1'
name: config
parameter:
- name: ssh-hostname
value: <rw_mgmt_ip>
- name: ssh-username
value: nornetpp
- name: ssh-password
value:
- seq: '2'
name: prepare-cassandra-hss-build
parameter:
- name: hss-git-repository
value: https://github.com/OPENAIRINTERFACE/openair-cn-cups.git
- name: hss-git-commit
value: develop
- seq: '3'
name: configure-cassandra
parameter:
- name: cassandra-server-ip
value: 172.16.6.129
...

Juju log:
2019-11-18 10:26:06 DEBUG update-status Traceback (most recent call last):
2019-11-18 10:26:06 DEBUG update-status File "/var/lib/juju/agents/unit-simulamet-oai-epc-b-hss-aa-0/charm/hooks/update-status", line 22, in
2019-11-18 10:26:06 DEBUG update-status main()
2019-11-18 10:26:06 DEBUG update-status File "/usr/local/lib/python3.5/dist-packages/charms/reactive/init.py", line 74, in main
2019-11-18 10:26:06 DEBUG update-status bus.dispatch(restricted=restricted_mode)
2019-11-18 10:26:06 DEBUG update-status File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 390, in dispatch
2019-11-18 10:26:06 DEBUG update-status _invoke(other_handlers)
2019-11-18 10:26:06 DEBUG update-status File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 359, in _invoke
2019-11-18 10:26:06 DEBUG update-status handler.invoke()
2019-11-18 10:26:06 DEBUG update-status File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 181, in invoke
2019-11-18 10:26:06 DEBUG update-status self._action(*args)
2019-11-18 10:26:06 DEBUG update-status File "/var/lib/juju/agents/unit-simulamet-oai-epc-b-hss-aa-0/charm/reactive/hsscharm.py", line 197, in configure_cassandra
2019-11-18 10:26:06 DEBUG update-status cassandraServerIP = action_get('cassandra-server-ip')
2019-11-18 10:26:06 DEBUG update-status File "/usr/local/lib/python3.5/dist-packages/charmhelpers/core/hookenv.py", line 82, in wrapper
2019-11-18 10:26:06 DEBUG update-status res = func(*args, **kwargs)
2019-11-18 10:26:06 DEBUG update-status File "/usr/local/lib/python3.5/dist-packages/charmhelpers/core/hookenv.py", line 974, in action_get
2019-11-18 10:26:06 DEBUG update-status action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
2019-11-18 10:26:06 DEBUG update-status File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
2019-11-18 10:26:06 DEBUG update-status **kwargs).stdout
2019-11-18 10:26:06 DEBUG update-status File "/usr/lib/python3.5/subprocess.py", line 708, in run
2019-11-18 10:26:06 DEBUG update-status output=stdout, stderr=stderr)
2019-11-18 10:26:06 DEBUG update-status subprocess.CalledProcessError: Command '['action-get', 'cassandra-server-ip', '--format=json']' returned non-zero exit status 1
2019-11-18 10:26:06 ERROR juju.worker.uniter.operation runhook.go:113 hook "update-status" failed: exit status 1

The issue appears in 100% of the cases, i.e. I can reliably reproduce it. Also, trying to manually invoke the funtion (i.e.: juju run-action simulamet-oai-epc-b-hss-aa/0 configure-cassandra cassandra-server-ip=1.2.3.4) leads to the same issue.

Note, that my VNF applies the Juju charm as part of the VDU configuration. I am not sure whether this is important for the issue.

It would be best practice to accept shell commands as they would run in a shell. To that end, sshproxy should automatically handle any escaping required.

This is a particular issue when using pipes, sed, grep, awk, etc.

Hello,

I am unable to run the the lastest build of the sshproxy in Juju repository (within an Open Source MANO deployment) because of the following import error:

File
"/var/lib/juju/agents/unit-flfh-test-gui-flhfilter-b-0/charm/reactive/sshproxy.py",
line 18, in <module>
from charmhelpers.core.hookenv import (
  ImportError: cannot import name 'unitdata'

From the charm-helpers github repo, I see that unitdata is available under the charmhelpers.core package, not the charmhelpers.core.hookenv one [1]. Also, if I manually import the unitdata file from charmhelpers.core in a Python interactive shell, it works (while the *.hookenv version doesn't).

Do you have any suggestion about how to solve this issue?

[1] https://github.com/juju/charm-helpers/blob/master/charmhelpers/core/unitdata.py

The initial version is a mess, security-wise. Instead of passing a private SSH key in plaintext, the charm should automatically generate a SSH key, and expose the public key that can be read and placed on any machine(s) that the unit should be able to connect to.

I am not sure whether this issue is caused by layer-sshproxy or another part of OSM.

When making some tests with creating and removing NSs with VNFs using Charms, I noticed that at least sometimes the Juju models do not get cleaned up:

$ juju list-models
Controller: osm

Model                                  Cloud/Region         Type  Status     Machines  Units  Access  Last connection
5308bf0e-15e6-4398-b6fa-a1254ef0f9a5   localhost/localhost  lxd   available         1      1  admin   3 hours ago
5e23b013-4d48-4771-966c-e9e72efdbef5   localhost/localhost  lxd   available         2      2  admin   3 hours ago
82fdfb47-d5c9-4281-a0c3-783e4259ba54*  localhost/localhost  lxd   available         4      4  admin   5 seconds ago
cf15deb2-7b15-4fb4-80b0-f6c026565151   localhost/localhost  lxd   available         2      2  admin   27 seconds ago
controller                             localhost/localhost  lxd   available         1      -  admin   just now
d39d59de-90ec-49af-99f7-1100e7bdf7e1   localhost/localhost  lxd   available         2      2  admin   2 hours ago
default                                localhost/localhost  lxd   available         0      -  admin   2020-03-04

The setup here is just running one NS with one VNF instance, but the old Juju models created for already-deleted NS/VNF instances seem to not get disposed. Tested with OSM ReleaseSEVEN.