tiny-pilot / ansible-role-tinypilot Goto Github PK

Now that Raspbian 64-bit is available, we should make sure the role is compatible.

A user reported a successful install, but the HID driver is failing:

tiny-pilot/tinypilot#906

One issue I know is that we're installing the HID driver for the wrong architecture, so we need to add a check on this play to make sure the architecture is armv7l and not aarch64.

On Ubuntu 20.10, I get the following error when running the this role:

TASK [mtlynch.tinypilot : install TinyPilot pre-requisite packages] ************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "No package matching 'python-pip' is available"}

I believe this is because python-pip was removed from Ubuntu (and Debian) repos due to Python 2's deprecation. Should this be changed to python3-pip?

As we explore adding Janus and H264/WebRTC integration, I'd like to start with just a manual implementation to get Janus and uStreamer configured correctly.

• Create proof of concept - Done in #172
• TinyPilot dev team replicates proof of concept build

I'm not sure if this is a Janus bug or a uStreamer bug, but the uStreamer Janus plugin won't compile unless we manually fix a C #include line in a Janus header.

It looks like Janus expects clients to have the janus header directory directly included when a plugin compiles, whereas uStreamer expects to only have the include/ dir included and expects Janus headers to be under <janus/whatever.h>, which seems reasonable.

If TinyPilot doesn't have internet access when booting, Flask fails with dns.resolver.NoResolverConfiguration: Resolver configuration could not be read or specified no nameservers..

I thought setting Restart=always would do the trick:

https://github.com/mtlynch/ansible-role-tinypilot/blob/a4e195ab6ee349a2ab2d66ded1255d6d1d0c1784/templates/tinypilot.systemd.j2#L17

But it looks like it's more complicated:

https://serverfault.com/a/845473

Several browsers have bugs that prevent them from rendering MJPEG properly, and uStreamer has two separate workarounds:

• advance_headers: For Chrome/Blink
• dual_final_frames: For Safari/WebKit

We're using both workarounds, but we actually shouldn't have to. For example, Chrome should work without dual_final_frames, but if we omit that option in TinyPilot, there's a delay in rendering the latest frame. It looks like it's related to nginx.

If I change remote-screen.html in TinyPilot to use a uStreamer URL or /stream?advance_headers=1, Chrome renders the final frame with a delay. I set up a test where TinyPilot is using /stream?advance_headers=1 through nginx and I have a separate browser window rendering /stream?advance_headers=1 directly from uStreamer (no nginx proxy).

You can see that the nginx version has a delay before rendering the final frame at 0.000s, whereas the direct uStreamer URL renders it immediately.

We should figure out how nginx is changing the HTTP communication for MJPEG and update our nginx proxy so that browsers render MJPEG properly without excessive uStreamer workarounds:

https://github.com/tiny-pilot/ansible-role-tinypilot/blob/65b4840902ae0ff4781fcecaa79f5656172fd577/tasks/nginx.yml#L48L50

When performing a system update, the privileged update script is hard-wired to Github (https://raw.githubusercontent.com).

I’m wondering whether it would make sense to change this URL to a host that we control. The URL is distributed to devices and it might stay there for years (if people are lazy with system updates), which makes it very hard if not impossible to ever change anything about the URL.

Potential scenarios:

• We want to rename get-tinypilot.sh, or move it into a subfolder. In this case, we would have to keep the original script around, to guarantee backwards compatibility. This is the same problem that we already have with the quick-install script.
• We move the repo to Gitlab
• Github is down

The most simple solution I can think of is to create a redirect route in gatekeeper that’s like https://gk.tinypilotkvm.com/get-tinypilot.sh, which issues a 307 redirect to https://raw.githubusercontent.com/tiny-pilot/tinypilot/master/get-tinypilot.sh. That way we wouldn’t have to mirror the script itself.

It would even be possible to version that URL, like https://gk.tinypilotkvm.com/get-tinypilot.sh?version=2.0. Not sure what this could be useful for, but it demonstrates that we might be more flexible.

They're pretty easy, just these commands:

sudo vcgencmd measure_temp
sudo vcgencmd get_throttled

The update-overhaul no longer makes use of the /opt/tinypilot-updater directory and is now removed. However, the update-video-settings script heavily relies on this directory because it already contained the virtualenv and ansible-role-ustreamer which was used to update only the ustreamer settings on the device.

We no longer keep a local version of ansible roles that were used during the TinyPilot installation, so it's not easy to rerun a subset of our Ansible tasks to update video settings.

Users are hitting an issue where they're running TinyPilot Pro and accidentally follow update instructions for the free edition of TinyPilot, the web interface fails to load.

The issue is that TinyPilot Pro has two .conf files in nginx: one for port 443, and one for port 80. The free edition has only one, so when the user downgrades from Pro to Free, there's a stray .conf file that causes nginx to fail to start.

The free installer should include a step before the nginx include_role that removes the cat /etc/nginx/sites-enabled/tinypilot.http.conf file that the Pro version placed there.

Related: tiny-pilot/tinypilot#412

Once we migrate the keyboard and mouse settings to Flask configuration settings in the TinyPilot app, the KEYBOARD_PATH and MOUSE_PATH variables become obsolete, so we should remove them.

This is blocked on: tiny-pilot/tinypilot#738

On some laptops, the laptop doesn't output the display to an external monitor unless the user presses the KEY_SWITCHVIDEOMODE key. On Dell laptops, this is Fn+F8:

I ran evtest to dump output from a Dell laptop's keyboard, and this was what I saw when I pressed Fn+F8:

Event: time 1644874377.083934, -------------- SYN_REPORT ------------
Event: time 1644874381.374129, type 4 (EV_MSC), code 4 (MSC_SCAN), value 8b
Event: time 1644874381.374129, type 1 (EV_KEY), code 227 (KEY_SWITCHVIDEOMODE), value 1
Event: time 1644874381.374129, -------------- SYN_REPORT ------------
Event: time 1644874381.374152, type 4 (EV_MSC), code 4 (MSC_SCAN), value 8b
Event: time 1644874381.374152, type 1 (EV_KEY), code 227 (KEY_SWITCHVIDEOMODE), value 0

Full logs: https://gist.github.com/mtlynch/8544fb605bb20cddf7b05ca0c98e1e2b#file-dell-xps-native-keyboard-log

So, the Fn+F8 combination maps to KEY_SWITCHVIDEOMODE, which has a code of 227 on the keyboard.

I tried evtest from the TinyPilot USB device, and KEY_SWITCHVIDEOMODE does not appear as a supported code: https://gist.github.com/mtlynch/8544fb605bb20cddf7b05ca0c98e1e2b#file-tinypilot-virtual-keyboard-log

I'm not sure if there's something we can change in the USB HID descriptor to support this code. It's possible that USB keyboards simply can't send KEY_SWITCHVIDEOMODE, because I don't see it documented in the USB HID spec.

In order for us to offer an H264 option in the web UI, we need to install Janus by default.

In ansible-role-ustreamer, we can drop the ustreamer_install_janus fact altogether.

We should add an automatic action that performs the same function as the Github action we created for the tinypilot -> tinypilot-pro repo:

tiny-pilot/tinypilot#838

The last tasks seems to fail:
TASK [mtlynch.tinypilot : create TinyPilot virtualenv] *************************
fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/opt/tinypilot/venv/bin/pip3 list --format=freeze", "msg": "[Errno 2] No such file or directory: b'/opt/tinypilot/venv/bin/pip3': b'/opt/tinypilot/venv/bin/pip3'", "rc": 2}

Full output:
root@raspberrypi:/home/pi# curl -sS https://raw.githubusercontent.com/mtlynch/tinypilot/master/quick-install | bash -

• set -e
• set -u
• '[' -f /etc/nginx/sites-enabled/KVMPi.conf ']'
  ++ mktemp -d
• pushd /tmp/tmp.yQ9opggT79
  /tmp/tmp.yQ9opggT79 /home/pi
• sudo apt-get install -y libssl-dev python3-dev python3-venv
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  python3-dev is already the newest version (3.7.3-1).
  python3-venv is already the newest version (3.7.3-1).
  libssl-dev is already the newest version (1.1.1d-0+deb10u3+rpt1).
  The following package was automatically installed and is no longer required:
  rpi-eeprom-images
  Use 'sudo apt autoremove' to remove it.
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
• python3 -m venv venv
• . venv/bin/activate
  ++ deactivate nondestructive
  ++ '[' -n '' ']'
  ++ '[' -n '' ']'
  ++ '[' -n /bin/bash -o -n '' ']'
  ++ hash -r
  ++ '[' -n '' ']'
  ++ unset VIRTUAL_ENV
  ++ '[' '!' nondestructive = nondestructive ']'
  ++ VIRTUAL_ENV=/tmp/tmp.yQ9opggT79/venv
  ++ export VIRTUAL_ENV
  ++ _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  ++ PATH=/tmp/tmp.yQ9opggT79/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  ++ export PATH
  ++ '[' -n '' ']'
  ++ '[' -z '' ']'
  ++ _OLD_VIRTUAL_PS1=
  ++ '[' 'x(venv) ' '!=' x ']'
  ++ PS1='(venv) '
  ++ export PS1
  ++ '[' -n /bin/bash -o -n '' ']'
  ++ hash -r
• pip install wheel==0.34.2
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
  Collecting wheel==0.34.2
  Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
  Installing collected packages: wheel
  Successfully installed wheel-0.34.2
• pip install ansible==2.9.10
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
  Collecting ansible==2.9.10
  Collecting cryptography (from ansible==2.9.10)
  Using cached https://www.piwheels.org/simple/cryptography/cryptography-3.0-cp37-cp37m-linux_armv7l.whl
  Collecting PyYAML (from ansible==2.9.10)
  Using cached https://www.piwheels.org/simple/pyyaml/PyYAML-5.3.1-cp37-cp37m-linux_armv7l.whl
  Collecting jinja2 (from ansible==2.9.10)
  Using cached https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl
  Collecting six>=1.4.1 (from cryptography->ansible==2.9.10)
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
  Collecting cffi!=1.11.3,>=1.8 (from cryptography->ansible==2.9.10)
  Using cached https://www.piwheels.org/simple/cffi/cffi-1.14.1-cp37-cp37m-linux_armv7l.whl
  Collecting MarkupSafe>=0.23 (from jinja2->ansible==2.9.10)
  Using cached https://www.piwheels.org/simple/markupsafe/MarkupSafe-1.1.1-cp37-cp37m-linux_armv7l.whl
  Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->ansible==2.9.10)
  Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
  Installing collected packages: six, pycparser, cffi, cryptography, PyYAML, MarkupSafe, jinja2, ansible
  Successfully installed MarkupSafe-1.1.1 PyYAML-5.3.1 ansible-2.9.10 cffi-1.14.1 cryptography-3.0 jinja2-2.11.2 pycparser-2.20 six-1.15.0
• echo '[defaults]
  roles_path = /tmp/tmp.yQ9opggT79
  interpreter_python = /usr/bin/python3
  '
• ansible-galaxy install mtlynch.tinypilot

• downloading role 'tinypilot', owned by mtlynch
• downloading role from https://github.com/mtlynch/ansible-role-tinypilot/archive/master.tar.gz
• extracting mtlynch.tinypilot to /tmp/tmp.yQ9opggT79/mtlynch.tinypilot
• mtlynch.tinypilot (master) was installed successfully
• adding dependency: mtlynch.ustreamer
• adding dependency: geerlingguy.nginx
• downloading role 'ustreamer', owned by mtlynch
• downloading role from https://github.com/mtlynch/ansible-role-ustreamer/archive/master.tar.gz
• extracting mtlynch.ustreamer to /tmp/tmp.yQ9opggT79/mtlynch.ustreamer
• mtlynch.ustreamer (master) was installed successfully
• downloading role 'nginx', owned by geerlingguy
• downloading role from https://github.com/geerlingguy/ansible-role-nginx/archive/2.8.0.tar.gz
• extracting geerlingguy.nginx to /tmp/tmp.yQ9opggT79/geerlingguy.nginx
• geerlingguy.nginx (2.8.0) was installed successfully

• echo '- hosts: localhost
  connection: local
  become: true
  become_method: sudo
  roles:
  • role: mtlynch.tinypilot'
• ansible-playbook -i localhost, install.yml

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [mtlynch.ustreamer : collect universal required apt packages] *************
ok: [localhost]

TASK [mtlynch.ustreamer : collect Raspberry Pi OS specific required apt packages] ***
ok: [localhost]

TASK [mtlynch.ustreamer : collect Debian-specific required apt packages] *******
skipping: [localhost]

TASK [mtlynch.ustreamer : collect Ubuntu-specific required apt packages] *******
skipping: [localhost]

TASK [mtlynch.ustreamer : install uStreamer pre-requisite packages] ************
ok: [localhost]

TASK [mtlynch.ustreamer : create ustreamer group] ******************************
ok: [localhost]

TASK [mtlynch.ustreamer : create ustreamer user] *******************************
ok: [localhost]

TASK [mtlynch.ustreamer : create uStreamer folder] *****************************
ok: [localhost]

TASK [mtlynch.ustreamer : get uStreamer repo] **********************************
ok: [localhost]

TASK [mtlynch.ustreamer : clean repository if needed] **************************
skipping: [localhost]

TASK [mtlynch.ustreamer : enable OpenMax IL acceleration on Pi OS] *************
ok: [localhost]

TASK [mtlynch.ustreamer : build uStreamer] *************************************
ok: [localhost]

TASK [mtlynch.ustreamer : fix uStreamer folder permissions] ********************
ok: [localhost]

TASK [mtlynch.ustreamer : install uStreamer as a service] **********************
ok: [localhost]

TASK [mtlynch.ustreamer : enable systemd uStreamer service file] ***************
ok: [localhost]

TASK [geerlingguy.nginx : Include OS-specific variables.] **********************
ok: [localhost]

TASK [geerlingguy.nginx : Define nginx_user.] **********************************
ok: [localhost]

TASK [geerlingguy.nginx : include_tasks] ***************************************
skipping: [localhost]

TASK [geerlingguy.nginx : include_tasks] ***************************************
skipping: [localhost]

TASK [geerlingguy.nginx : include_tasks] ***************************************
included: /tmp/tmp.yQ9opggT79/geerlingguy.nginx/tasks/setup-Debian.yml for localhost

TASK [geerlingguy.nginx : Update apt cache.] ***********************************
ok: [localhost]

TASK [geerlingguy.nginx : Ensure nginx is installed.] **************************
ok: [localhost]

TASK [geerlingguy.nginx : include_tasks] ***************************************
skipping: [localhost]

TASK [geerlingguy.nginx : include_tasks] ***************************************
skipping: [localhost]

TASK [geerlingguy.nginx : include_tasks] ***************************************
skipping: [localhost]

TASK [geerlingguy.nginx : Remove default nginx vhost config file (if configured).] ***
ok: [localhost]

TASK [geerlingguy.nginx : Ensure nginx_vhost_path exists.] *********************
ok: [localhost]

TASK [geerlingguy.nginx : Add managed vhost config files.] *********************
ok: [localhost] => (item={'listen': '80 default_server', 'server_name': 'tinypilot', 'root': '/opt/tinypilot', 'index': 'index.html', 'extra_parameters': 'proxy_buffers 16 16k;\nproxy_buffer_size 16k;\nproxy_set_header Host $host;\nproxy_set_header X-Forwarded-For $remote_addr;\nproxy_http_version 1.1;\n\nlocation /socket.io {\n proxy_pass http://tinypilot;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection "Upgrade";\n}\nlocation /state {\n proxy_pass http://ustreamer;\n}\nlocation /stream {\n proxy_pass http://ustreamer;\n}\nlocation / {\n proxy_pass http://tinypilot;\n}\nlocation ~* ^/.+\.(html|js|js.map|css|jpeg|png|ico)$ {\n root "/opt/tinypilot/app/static";\n}\n'})

TASK [geerlingguy.nginx : Remove managed vhost config files.] ******************
skipping: [localhost] => (item={'listen': '80 default_server', 'server_name': 'tinypilot', 'root': '/opt/tinypilot', 'index': 'index.html', 'extra_parameters': 'proxy_buffers 16 16k;\nproxy_buffer_size 16k;\nproxy_set_header Host $host;\nproxy_set_header X-Forwarded-For $remote_addr;\nproxy_http_version 1.1;\n\nlocation /socket.io {\n proxy_pass http://tinypilot;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection "Upgrade";\n}\nlocation /state {\n proxy_pass http://ustreamer;\n}\nlocation /stream {\n proxy_pass http://ustreamer;\n}\nlocation / {\n proxy_pass http://tinypilot;\n}\nlocation ~* ^/.+\.(html|js|js.map|css|jpeg|png|ico)$ {\n root "/opt/tinypilot/app/static";\n}\n'})

TASK [geerlingguy.nginx : Remove legacy vhosts.conf file.] *********************
ok: [localhost]

TASK [geerlingguy.nginx : Copy nginx configuration in place.] ******************
ok: [localhost]

TASK [geerlingguy.nginx : Ensure nginx service is running as configured.] ******
ok: [localhost]

TASK [mtlynch.tinypilot : enable dwc2 driver in boot config] *******************
ok: [localhost]

TASK [mtlynch.tinypilot : enable dwc2 driver in modules] ***********************
ok: [localhost]

TASK [mtlynch.tinypilot : copy HID initializer script] *************************
ok: [localhost]

TASK [mtlynch.tinypilot : install HID initializer as a service] ****************
ok: [localhost]

TASK [mtlynch.tinypilot : enable systemd HID initializer service file] *********
ok: [localhost]

TASK [mtlynch.tinypilot : install TinyPilot pre-requisite packages] ************
ok: [localhost]

TASK [mtlynch.tinypilot : create tinypilot group] ******************************
ok: [localhost]

TASK [mtlynch.tinypilot : create tinypilot user] *******************************
ok: [localhost]

TASK [mtlynch.tinypilot : enable passwordless sudo for shutdown command] *******
ok: [localhost]

TASK [mtlynch.tinypilot : create TinyPilot folder] *****************************
ok: [localhost]

TASK [mtlynch.tinypilot : get TinyPilot repo] **********************************
ok: [localhost]

TASK [mtlynch.tinypilot : create TinyPilot virtualenv] *************************
fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/opt/tinypilot/venv/bin/pip3 list --format=freeze", "msg": "[Errno 2] No such file or directory: b'/opt/tinypilot/venv/bin/pip3': b'/opt/tinypilot/venv/bin/pip3'", "rc": 2}

PLAY RECAP *********************************************************************
localhost : ok=35 changed=0 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0

TinyPilot updates should only take about 5 minutes, but if the process hangs and runs forever, nothing stops them, and the hanging process will block future update attempts.

We should add TimeoutSec (and maybe KillMode) to the TinyPilot updater service systemd config so that systemd kills any stuck update processes that last longer than 15 minutes.

Pi 64-bit as an architecture labeled aarch64, which doesn't match the check here:

https://github.com/mtlynch/ansible-role-tinypilot/blob/5329a9b5fd18c7254f2b654c462ac9026c831204/tasks/main.yml#L4

We need to adjust that check and make sure the rest of the process is compatible with RasPi OS 64-bit.

Reference: tiny-pilot/tinypilot#83 (comment)

The Janus-uStreamer proof of concept Docker images is currently hardcoded to use Voyager's uStreamer settings:

It should be pretty easy to make it work with Voyager or Hobbyist. I'd recommend adjusting start.sh so that uStreamer's parameters are specified by "$@" instead of hardcoded.

That way, when the user launches the Docker container, they can pass in parameters like:

docker run \
  --privileged \
  --network host \
  --name janus-ustreamer \
  mtlynch/ustreamer-janus:2022-02-02 \
  -f 30 \
  --host 127.0.0.1 \
  -p8001 \
  --h264-sink tinypilot::ustreamer::h264 \
  --h264-sink-rm \
  --h264-sink-mode 777 \
  --format uyvy \
  --encoder omx \
  --persistent \
  --dv-timings \
  --workers 3 \
  --drop-same-frames 30

That way, Docker passes the flags to start.sh, and start.sh passes those flags to uStreamer.

Then, we update the instructions in the README to show how to launch for Hobbyist or Voyager:

https://github.com/tiny-pilot/tinypilot/tree/experimental/h264#run

If you use TinyPilot on a system that needs an HTTP proxy to access external services, some TinyPilot features won't work because they're not proxy-aware.

• Uploading log files from System > Logs will fail
• Performing updates from System > Update will fail

We could add an Ansible variable for setting the HTTP proxy and then, if it's set, add it to the systemd services for the TinyPilot web app and TinyPilot updater.

It's currently fairly tedious to change uStreamer settings and apply them. You can edit the ustreamer.service file directly, but any changes will get overwritten by the next update. The current way to persist settings is:

1. Set values in ustreamer.service
2. systemctl daemon-reload && sudo service ustreamer restart
3. Set the same variables in /home/tinypilot/settings.yml so that they'll persist through the next update

We should add a convenience script that reads settings from /home/tinypilot/settings.yml (similar to how quick-install does it) and then run a playbook with only ansible-role-ustreamer.

The playbook should:

• Skip fact gathering to save time (gather_facts: no)
• Run only tags related to ustreamer's systemd settings
  • We'll have to edit ansible-role-ustreamer to add those tags

The script should support a quiet mode so that we can call it programmatically from the frontend later.

Update: 2021-03-24

The desired flow for this script should be:

1. User updates /home/tinypilot/settings.yml with their desired uStreamer settings
2. User runs /opt/tinypilot-privileged/update-video-settings
3. update-video-settings (via Ansible) updates uStreamer's systemd file, reloads it, and restarts the uStreamer service

The script should only support changes to uStreamer role variables that affect uStreamer's runtime flags in the systemd file. We shouldn't try to support every possible uStreamer role variable.

Also, we want to execute the uStreamer role, not the TinyPilot role. We really just want to execute the systemd play and then the two handlers come after it to reload and restart ustreamer. I'm hoping we can get this script to complete in <3 seconds.

I'm thinking the call to Ansible can be something like this:

readonly TINYPILOT_ROLE_NAME="mtlynch.ustreamer"
echo "- hosts: localhost
  connection: local
  become: true
  become_method: sudo
  gather_facts: no
  roles:
    - role: ${TINYPILOT_ROLE_NAME}" > update-settings.yml

ansible-playbook \
  update-settings.yml \
  --inventory localhost, \
  --extra-vars "@/home/tinypilot/settings.yml" \
  --extra-vars "@/home/ustreamer/config.yml" \
  --tags "config"

Recovering settings from install

We have to work around this hackery because it means we lose a clean record of how uStreamer was provisioned. I think we can work around that by updating quick-install so that instead of setting those to transient environment variables, it appends the appropriate role variables in /home/tinypilot/settings.yml.

The file init-usb-gadget currently lives in the TinyPilot repo:

https://github.com/mtlynch/tinypilot/blob/33d3cd5630ea17c4ce02eb436f0cdf436d72ac0a/scripts/usb-gadget/init-usb-gadget

But there's an elevation of privilege risk here because the file is writable by the unprivileged tinypilot user, but it's executed as root during startup, so if an attacker compromised tinypilot, they'd compromise the whole system.

To remediate this, we should move the init-usb-gadget back to this repo and add an Ansible play where it places the file in /opt/tinypilot-privileged/init-usb-gadget with mode 700.

We'll also need to update references to /opt/tinypilot/scripts/usb-gadget/init-usb-gadget in both this repo and the tinypilot repo.

It’s not always 100% clear where functionality should go, either to the ansible repo or the app one. So for example:

• When do we create privileged scripts and implement the logic around it in a shell script, vs. when do we just add a sudoer’s entry for a specific command and implement the other logic in Python?
• How do we go about repetition between the two repos, e.g. when certain parameters are needed in both repositories? Example: configfs paths for mass storage, that we have both in the app and in ansible.
• Does the fact that we have mock scripts for local development justify single-line privileged scripts, only for the sake of mocking them out? Or is it okay to just use debug toggles for that? What is our general approach towards dev-prod-parity?

It’s probably hard to come up with a definite answer, but at least we could try writing up some guidelines that help us make consistent decisions.

PRs/Issues during which that discussion came up already:

• https://github.com/tiny-pilot/tinypilot-pro/issues/414#issuecomment-1070520327
• https://github.com/tiny-pilot/tinypilot-pro/pull/281#pullrequestreview-819399824

Apple devices don't implement the HID spec correctly, so it fails to recognize the TinyPilot keyboard in pre-boot. The upstream Linux kernel has a workaround, but it hasn't made its way into Raspbian yet.

A reader on my blog shared instructions for compiling custom builds of the kernel modules and confirmed that they work on his Pi Zero + Apple devices:

Instructions

git clone https://github.com/raspberrypi/linux/
git checkout 4e5d621498df # ("overlays: Reduce Pi 4 vc4 CMA size to 320MB")
get the patch from https://0bin.net/paste/m6fB-sxx#ZF8Jp5ArPkDJS-W3ax5BROkPtkKi+hUH9F/mhgSL+Z1
git apply 0001-usb-gadget-f_hid-optional-SETUP-SET_REPORT-mode.patch # apply patch

CONFIG_LOCALVERSION="-v7l-MY_CUSTOM_KERNEL" # set some environment variable
KERNEL=kernel

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig # default config pi zero
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules_prepare # prepare build environment for modules
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- M=drivers/usb/gadget/function/
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- M=drivers/usb/gadget/legacy/

mkdir -p apple-keyboard-fix
cp ./drivers/usb/gadget/legacy/g_hid.ko apple-keyboard-fix/ # grab the two modules
cp ./drivers/usb/gadget/function/usb_f_hid.ko apple-keyboard-fix/

On the actual device/pi zero:

sudo cp /home/pi/apple-keyboard-fix/g_hid.ko /lib/modules/5.10.52+/kernel/drivers/usb/gadget/legacy/g_hid.ko
sudo cp /home/pi/apple-keyboard-fix/usb_f_hid.ko /lib/modules/5.10.52+/kernel/drivers/usb/gadget/function/usb_f_hid.ko

I've been using the instructions from https://www.isticktoit.net/?p=1383 where you ned to add the following to the script /usr/bin/isticktoit_usb, after subclass configuration:

echo 1 > functions/hid.usb0/no_out_endpoint

I've uploaded both compiled files to my google drive, if you want to skip the instructions above:
g_hid.ko (for kernel 5.10.52+): https://drive.google.com/file/d/17yx2yxFjIjcXplfqRHKXlQAKavysMoeA/view?usp=sharing
usb_f_hid.ko (for kernel 5.10.52+): https://drive.google.com/file/d/10xkzEgTdCPS0M4D1TMv80tlvLI5sY2Mz/view?usp=sharing

Our fix

We should compile our own build for the Pi 4 and host it on a public TinyPilot URL.

This Ansible role should include logic that if the system is a Raspberry Pi with a kernel version without the workaround, we should download our custom kernel module builds and replace the built-in modules on the device.

I'm not sure if we need to replace both modules or only one.

The update script currently just grabs the latest HEAD version of TinyPilot from the master branch, but it would be better for it to support upgrading to a specific release tag.

We should add a parameter to files/update:

-r, --release=VERSION Update TinyPilot to a specific release version or commit ID

If it simplifies the implementation, we can use just -r or just --release instead of supporting both.

Specifying a --release version should set the TINYPILOT_INSTALL_VARS environment variable before calling quick-install:

TINYPILOT_INSTALL_VARS="tinypilot_repo_branch=${VERSION}"

(tinypilot_repo_branch has a misleading name, but it can be used for commits, release tags, or branches)

We're currently hardcoding the uStreamer role variables in tasks/ustreamer.yml:

https://github.com/tiny-pilot/ansible-role-tinypilot/blob/77700969294ae0d166f66e55f094ede20a2e72d0/tasks/ustreamer.yml#L6L9

It would be more flexible to define those variables in defaults/main.yml because that would let role clients override them and we'd avoid implicitly duplicating the values in tasks/nginx.yml:

https://github.com/tiny-pilot/ansible-role-tinypilot/blob/77700969294ae0d166f66e55f094ede20a2e72d0/tasks/nginx.yml#L14L21

As part of H264 integration, we need to integrate a Janus server and configure it to work with TinyPilot's nginx configuration.

Ideally, we'll use the existing bitsy-ai/ansible-role-janus-gateway Ansible role. Let me know if we are, and I'll create a local fork.

We'll also have to update the target uStreamer version to the latest, as we're on 3.26, which precedes Janus/H264 support.

Janus should be off by default with an Ansible variable that controls whether or not to install it.

Blocked on

• ansible-role-tinypilot: Research: Create a proof of concept Janus-uStreamer-TinyPilot integration
• ansible-role-ustreamer: Add support for building Janus plugin

Janus takes 20-30m to build from source on a Raspberry Pi device. Now that we have a pre-built Janus binary, we should install that instead of compiling it from scratch at install time.

The file permissions task reports a change on repeated runs even though nothing should have actually changed:

https://github.com/mtlynch/ansible-role-tinypilot/blob/21514df253b1fa9e6e6d390ef907f5544b2135eb/tasks/main.yml#L65

As a hack, I've placed changed_when: false to pass idempotence tests, but we should figure out why it's reporting a change here.

A user reported the following failure in the TinyPilot updater:

TASK [ansible-role-tinypilot : save whether the HID module should be patched] ***
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'checksum'\n\nThe error appears to be in '/opt/tinypilot-updater/ansible-role-tinypilot/tasks/install_usb_gadget.yml': line 90, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: save whether the HID module should be patched\n ^ here\n"}

The failure is here:

The user mentioned that they had just performed system upgrades, and the update worked after rebooting. It looks like if there's a reboot pending a kernel update, the usb_f_hid.ko won't be there, which causes the play to fail.

The control flow for executing a system update currently goes like this:

1. TinyPilot app triggers Updater systemd service
2. Updater systemd service executes scripts/update-service
3. update-service script calls update privileged script
4. update privileged script downloads and executes quick-install script
5. update-service writes result into JSON file, to communicate back to the TinyPilot app.

The new flow that is described in the UPDATE-WORKFLOW document doesn’t apply to Community: the situation in Community is simpler, since we always download the latest version, so we don’t have to save and pass down the desired version. Effectively, we need to change step 4:

1.-3. (just as before, see above)
4. (new) update privileged script downloads get-tinypilot.sh script and executes it
5. (just as before, see above)

Note: since the quick-install script needs to stay where it is for backwards-compatibility, we could also consider to always go through it, instead of changing the update privileged script. In this case, this issue would be obsolete. Pro: just one single way how it’s being done. Con: an unnecessary indirection just for legacy reasons. Note, the latter (the Con) will become more apparent in Pro, since we need to pass down additional parameters.

⚠️⚠️⚠️
Please hold off on merging until the second 2.4.2 milestone is complete.
(See the note of the second milestone.)
⚠️⚠️⚠️

In addition to our current way of installing the tinypilot folder via git, we need to offer an --extra-vars option so that the tinypilot folder can be created from a local Debian package. (The default way is still like it is right now, though.)

The new option could be tinypilot_install_source:

• If it’s null, then default to git
• If it’s a local path to a Debian package, use that package

The Debian package will be bundled into the tarball, so the new update mechanism is supposed to consume this option.

• Scratch implementation

It looks like they've made some useful changes recently, so we should upgrade to 1.0.4 (or whatever is latest and greatest), since 2.5.1 will be the first release where we officially support H264 over WebRTC.

The voltage log collector shows voltage warnings that occurred at any time, but we really only care about the ones that happened on the current boot. It should be possible to filter the messages by changing the journalctl invokation to:

sudo journalctl --boot --lines=all -xe | grep -i "voltage"

We also might not need the -xe flag. I haven't tested whether it's necessary to get the voltage logs. If we do, we should switch to the verbose versions of the flags for better clarify:

-x --catalog Add message explanations where available
-e --pager-end Immediately jump to the end in the pager

Testing note

An easy way to force TinyPilot to emit a low voltage warning is to connect its power cable to a PC's USB port instead of a real 3 Amp USB adaptor.

Several users are reporting that uStreamer fails after they perform an upgrade:

/opt/ustreamer/ustreamer: error while loading shared libraries: libjpeg.so.8: cannot open shared object file: No such file or directory

This prevents uStreamer from launching and therefore hides the remote screen.

Workaround

Reinstall the microSD from a pre-made image.

Other observations

• All affected users are using HDMI to USB dongles (their logs show --encoder hw --format jpeg)
• This seems to happen after an update
• This isn't introduced in 2.3.2 as some users reported it in 2.3.1
• It might be related to this change: tiny-pilot/ansible-role-ustreamer#48

Example user logs

• https://logs.tinypilotkvm.com/5umcSogn
• https://logs.tinypilotkvm.com/GSBFYqnh
• https://logs.tinypilotkvm.com/wAg5TV20

Sometimes, customers file support requests saying that TinyPilot doesn't work on their device, and it takes a few back-and-forths to discover that they're running it on unsupported hardware.

We should flag a warning in the debug logs if we detect that the running hardware does not match compatible hardware (Pi 4B or Pi Zero series). Only Pi 4B is officially supported, but Pi Zero should also at least work out of the box.

Depends on tiny-pilot/tinypilot#1220

The collect-debug-logs script currently prompts the user for input.

We need to support a -q (quiet) switch for that script that skips any user prompts and allows it to be executed programmatically.

Edit: Changed --silent to -q

The action of collecting logs itself generates a lot of log spew:

Mar 23 14:40:06 tinypilot sudo[2515]: tinypilot : TTY=unknown ; PWD=/opt/tinypilot ; USER=root ; COMMAND=/opt/tinypilot-privileged/collect-debug-logs -q
Mar 23 14:40:06 tinypilot sudo[2515]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2528]:     root : TTY=unknown ; PWD=/opt/ustreamer ; USER=root ; COMMAND=/usr/bin/journalctl -xe
Mar 23 14:40:07 tinypilot sudo[2528]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2528]: pam_unix(sudo:session): session closed for user root
Mar 23 14:40:07 tinypilot sudo[2532]:     root : TTY=unknown ; PWD=/opt/ustreamer ; USER=root ; COMMAND=/usr/bin/journalctl -u tinypilot
Mar 23 14:40:07 tinypilot sudo[2532]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2532]: pam_unix(sudo:session): session closed for user root
Mar 23 14:40:07 tinypilot sudo[2538]:     root : TTY=unknown ; PWD=/opt/ustreamer ; USER=root ; COMMAND=/usr/bin/journalctl -u tinypilot-updater
Mar 23 14:40:07 tinypilot sudo[2538]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2538]: pam_unix(sudo:session): session closed for user root
Mar 23 14:40:07 tinypilot sudo[2543]:     root : TTY=unknown ; PWD=/opt/ustreamer ; USER=root ; COMMAND=/usr/bin/journalctl -u ustreamer
Mar 23 14:40:07 tinypilot sudo[2543]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2543]: pam_unix(sudo:session): session closed for user root
Mar 23 14:40:07 tinypilot sudo[2546]:     root : TTY=unknown ; PWD=/opt/ustreamer ; USER=root ; COMMAND=/usr/bin/journalctl -u nginx
Mar 23 14:40:07 tinypilot sudo[2546]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2546]: pam_unix(sudo:session): session closed for user root
Mar 23 14:40:07 tinypilot sudo[2548]:     root : TTY=unknown ; PWD=/opt/ustreamer ; USER=root ; COMMAND=/usr/bin/tail -n 100 /var/log/nginx/error.log
Mar 23 14:40:07 tinypilot sudo[2548]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2548]: pam_unix(sudo:session): session closed for user root
Mar 23 14:40:07 tinypilot sudo[2550]:     root : TTY=unknown ; PWD=/opt/ustreamer ; USER=root ; COMMAND=/usr/bin/tail -n 30 /var/log/nginx/access.log
Mar 23 14:40:07 tinypilot sudo[2550]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 23 14:40:07 tinypilot sudo[2550]: pam_unix(sudo:session): session closed for user root
Mar 23 14:40:07 tinypilot sudo[2515]: pam_unix(sudo:session): session closed for user root

That's 24 lines, and they're not really useful, and they get longer every time we add someting to collect-debug-logs. We only show the last 200 lines of the tinypilot logs, so each log retrieval eats up 12% of the real estate.

We have some duplication in the privileged scripts, especially in pro. For example, the path to the USB gadget / configfs is spelled out in multiple files, or the procedure for (re-)initializing the gadget. It would be clearer, if there was just one source of truth for those values, and defining functions for common procedures could make the scripts more expressive.

An idea would be to pull out a script file with common constants and procedures (e.g. lib.sh) that can be source’d in the other scripts. Example:

#!/bin/bash

readonly USB_GADGET_PATH="/sys/kernel/config/usb_gadget/g1"

initialize_gadget() {
  ls /sys/class/udc > "${USB_GADGET_PATH}/UDC"
  chmod 777 /dev/hidg0
  chmod 777 /dev/hidg1
}

Then, in eject-mass-storage:

source lib.sh # or whatever the file name is

# ...

if "${is_force_mode}"; then
  echo '' > "${USB_GADGET_PATH}/UDC"
  trap initialize_gadget EXIT
fi

app_settings.cfg contains user-configurable data, so we should be storing it in a directory intended for data like /home/tinypilot.

We're currently storing it in /opt/tinypilot, which is creating problems because the Debian installer expects to remove that directory cleanly, and to migrate from Community to Pro post-update-overhaul, we need to blow away the /opt/tinypilot directory.

We could do something to gracefully move the file from /opt/tinypilot to /home/tinypilot if it already exists, but I think it's too complex to be worth it given how few people have likely made configuration changes in this file.

Splitting off from tiny-pilot/tinypilot#538 (comment)

TinyPilot's nginx configuration disables caching for static resources. I originally did this because it put the app into an inconsistent state after the user applied updates (e.g., they'd reload after an update and have different versions of various static files). But we realized that it has the unintended consequence of forcing the browser to redownload the same file dozens of times for a single page load on Chrome (and possibly other browsers).

We should revise the cache policy so that updates don't throw the frontend into an inconsistent state but we're also not causing bizarre browser behavior of redownloading the same file.

Because we already install uStreamer directly on the device, it's complicated to use a Docker image that has uStreamer built in as well. The proof of concept did this to avoid the hassle of sharing memory between the host system and the Docker container, but we should be able to share memory.

uStreamer and the uStreamer Janus plugin need to share memory for uStreamer's --h264-sink option to work. I believe it's possible to achieve this by mounting /dev/shm in Janus' Docker container with Docker's --volume flag, so we'd do something like this:

docker run \
  --privileged \
  --network host \
  --volume /dev/shm:/dev/shm \
  --name janus \
  mtlynch/janus-image-we-will-build

Update: 2022-02-09

Per discussion, we can't compile the uStreamer Janus plugin on a system unless Janus is already installed.

So we'll keep the uStreamer compilation steps so that we create the uStreamer Janus plugin within the container, but we'll remove the part of start.sh where we launch uStreamer as a service within the container. Instead, we'll launch uStreamer on the host system and share the /dev/shm path (as above) so that the container's uStreamer Janus plugin can read shared memory from uStreamer on the host system.

This play:

https://github.com/mtlynch/ansible-role-tinypilot/blob/a4e195ab6ee349a2ab2d66ded1255d6d1d0c1784/tasks/main.yml

Needs a regexp to pattern match the line to update. Otherwise it keeps creating duplicate entries for the tinypilot user.

I could be wrong here, but a fresh setup (on a raspberry pi) using the ansible role fails for me, and I believe it's a versioning problem between the role and the tinypilot codebase.

I have ended up without the settings file APP_SETTINGS_FILE, and neither can I see that environment variable in the systemd service file, although both of these things seem to now be required by the version of tinypilot itself that is installed.

It seems like the role hasn't been updated on Galaxy since 2020, and the resulting setup of using it (which pulls the latest code from tinypilot) ends up with no HiD devices, and fails to boot.

I appreciate that your docs say to install it directly from this repo rather than Galaxy, so maybe some better signposting might help? Or obviously just release a new version to Galaxy

After having merged both tiny-pilot/tinypilot#1046 and #214, we can remove the use of git for TinyPilot code. For example:

• ansible-role-tinypilot/tasks/main.yml

  Lines 124 to 135 in 19100f2

  	- name: get TinyPilot repo
  	# Don't escalate privileges because only the directory owner can use git.
  	# Issue: https://github.com/tiny-pilot/tinypilot/issues/963
  	become: no
  	git:
  	repo: "{{ tinypilot_repo }}"
  	dest: "{{ tinypilot_dir }}"
  	version: "{{ tinypilot_repo_branch }}"
  	accept_hostkey: yes
  	when: tinypilot_debian_package_path == None
  	notify:
  	- restart TinyPilot service

• ansible-role-tinypilot/files/collect-debug-logs

  Lines 68 to 70 in 19100f2

  	print_info "Checking TinyPilot version..."
  	cd /opt/tinypilot && \
  	printf "TinyPilot version: %s %s\n" "$(git describe --tags --always)" "$(git rev-parse --short HEAD)" >> "${LOG_FILE}"

Hi,

i just tested tinypilot on bullseye and ran into a few problems:

python-pip requirement by ansible must be replaced by python3-pip
libjpeg8-dev is replaced by libjpeg9-dev for ustreamer
dnspython version to old (just used dnspython without version which worked fine)

besides that tinypilot itself works like charm under bullseye.

Would be happy to see native compatibility.

Thanks!
regards

We currently copy over init-usb-gadget and remove-usb-gadget, but we never restart the usb-gadget service in response to those changes.

This doesn't cause any problems now because we reboot after install or update, but it would be cleaner if the Ansible role intelligently tracked when a restart is required.

Note that we don't want to restart the usb-gadget service if a reboot is pending to load the USB gadget module because then the usb-gadget service will fail to start.

A user reported that they have their external host set up to forward traffic from tinypilot.example.com:1234 to their internal tinypilot URL at https://tinypilot.

The problem is that when they visit https://tinypilot.example.com:1234, it redirects them to https://tinypilot.example.com/login (dropping the port).

I believe it's due to this line in the nginx.conf:

https://github.com/mtlynch/ansible-role-tinypilot/blob/a4e195ab6ee349a2ab2d66ded1255d6d1d0c1784/tasks/nginx.yml#L30

Apparently the $host variable drops the port number. I asked the user to try $http_host instead, but they reported that it returned this error:

400 Bad Request, The plain HTTP request was sent to HTTPS port

But then if they hit enter to request the exact same URL, it worked. I'm not sure why that behavior's happening, so we need to investigate further.

The collect-debug-logs script script dumps different TinyPilot-related debug logs to stdout. In addition, it prints status messages about what logs it's retrieving, which also used to go to stdout (until #100).

The problem was that for scripts retrieving the logs programmatically, the status messages cluttered the actual logs. I tried to fix this in a hacky way in #100, but now the status messages are cluttering the TinyPilot log itself whenever the user retrieves the logs through the web interface.

We should revert #100 and replace it with an implementation that has a bash function that prints the progress messages to stdout, but when -q is passed, doesn't print any status messages. Error messages should still go to stderr.

When running quick-install for the first time on a device running raspbian lite (kernel versions >= 5.10 and < 5.15), I get the following error:

RUNNING HANDLER [ansible-role-tinypilot : start usb-gadget service] ******************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to start service usb-gadget: Job for usb-gadget.service failed because the control process exited with error code.\nSee \"systemctl status usb-gadget.service\" and \"journalctl -xe\" for details.\n"}

It fails to run the start usb-gadget service handler introduced by #155

Seeing as we're only enabling the the OTG gadget at boot, the usb-gadget cannot be started until the device is rebooted.

Possible solutions:

1. Simply ignore errors on the handler (i.e. start usb-gadget on best effort). This allows a fresh-install to continue and allows a update-install to actually start the gadget without a reboot.
2. Remove the handler and always rely on a reboot. This isn't ideal for when you simply want to update the software.

Using stock Raspberry Pi OS Lite, the systemd service fails to start with this error:

Jul 28 19:37:10 raspberrypi systemd[5414]: tinypilot.service: Failed at step EXEC spawning /opt/tinypilot/venv/bin/python: Permission denied
Jul 28 19:37:10 raspberrypi systemd[1]: tinypilot.service: Main process exited, code=exited, status=203/EXEC

Alas, I am not knowledgable enough with systemd to figure out why it hates this.

After a reboot, the symlinks point to a no longer existing path under /tmp, so the failure changes to:

Jul 28 19:40:06 raspberrypi systemd[561]: tinypilot.service: Failed to execute command: No such file or directory
Jul 28 19:40:06 raspberrypi systemd[561]: tinypilot.service: Failed at step EXEC spawning /opt/tinypilot/venv/bin/python: No such file or directory

After removing the symlink and manually creating one directly to /usr/bin/python3, things seem to work:

root@raspberrypi:/var/log# rm /opt/tinypilot/venv/bin/python
root@raspberrypi:/var/log# ln -s /usr/bin/python3 /opt/tinypilot/venv/bin/python