ros-realtime / ros-realtime-rpi4-image Goto Github PK

Motivation

Right now, we're only installing ROS 2 galactic, we should be able to install any ROS 2 distro available for focal.

Acceptance Criteria

• Parametrized the image
• Check if other ROS 2 versions works properly

Hi,

I want to install some amount of stuff which will lead to an output image larger than 4GB. The downloaded raspberry pi image is 4GB, and then while I'm installing some packages in my phase1-target script I get:

[...]
Selecting previously unselected package linux-modules-extra-5.15.0-83-generic.
Preparing to unpack .../21-linux-modules-extra-5.15.0-83-generic_5.15.0-83.92_arm64.deb ...
Unpacking linux-modules-extra-5.15.0-83-generic (5.15.0-83.92) ...
dpkg: error processing archive /tmp/apt-dpkg-install-xpMDy0/21-linux-modules-extra-5.15.0-83-generic_5.15.0-83.92_arm64.deb (--unpack):
 cannot copy extracted data for './lib/modules/5.15.0-83-generic/kernel/drivers/media/test-drivers/vivid/vivid.ko' to '/lib/modules/5.15.0-83-generic/kernel/drivers/media/test-drivers/vivid/vivid.ko.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
[...]

It seems that the image is not resized after download and that we're not able to install everything we want.

Is it possible to add a step to resize this image before executing scripts ?

Teusner

The docs right now is all in README.md, which is aimed at people who work in this repo, rather than the users. I can split the docs into two parts: a part aimed at developers and a part aimed at the users.

The build status image on the readme is also broken following the organization move as it linked to my repo. This can easily be fixed as well.

I noticed the humble release tag is tagged to a point before we merged the humble build scripts. We should probably fix it with the "final" release.

cc: @LanderU @carlossvg

CMake toolchain needs set(PYTHON_SOABI cpython-38-aarch64-linux-gnu) to enable for cross-compilation for Python ROS. This is kind of problematic to implement because the variable must be set to the target Python version. Right now the toolchain file is global for the whole builder.

This requires a bit more thought. Ideally the ros-rt-img tool will generate a cmake toolchain file dynamically. I'll have to think about it a bit more.

Hello,
I'm using your real time patch on my Raspberry 3B+. The installation didn't give me any problem.

The problem came when I tried to create a new user "x". Now, I'm not able to access to the gpio pins anymore. Get the privilege to access the pins is not a big deal, if not that when running my scripts with user "x" the real time is not happening anymore. It does happened only if I enter in sudo mode (e.g, sudo -s).

How can I set the new user "x" to be set similar as the sudo, giving at least the possibility to work in real time?

Thank you

From #23 (comment)

• create a directory for the ROS 2 scripts: ros2_build_profiles/galactic, ros2_build_profiles/rolling
• Same for rt-focal, i,e: ubuntu_build_profiles/rt-focal, ubuntu_build_profiles/focal (for no RT version)
• Rename ros2-rt-rpi4 to something generic so we can put here different script types. i.e. ros2-rt-utils, rt-configuration-scripts, rt-configuration-tools, etc. Here we can add scripts to isolate CPUs, CPU affinities, etc
• Rename phase1-* scripts to something more descriptive. i.e host-build and host-post-build.

Right now build.py is really just a hard coded script to build the focal RT image. We want to turn it into a proper command line utility, capable of building even outside the rootdir of this repo (for custom images).

Some immediate feature requests:

• Move qemu-user-static and pause_after configurations into the command line tool.
• Create command line shortcuts to cleanup the host system if something goes wrong
• Create command line shortcuts to chroot into the target image for testing

The builder command need to support a more seamless cross compile workflow. Need to do this before ROScon 2022 if the talk is accepted.

The build process takes about 10-15 minutes on CI without compressing the image. The resulting image is then 8GB, which very large. Github releases have a 2GB/file limit, so we should go under that. When I tried to xz the image, If I recall, I got it to about 1.7GB, which would be below that limit if I'm not wrong. In that case, we can post these images directly to Github releases (only for tags tho). Not sure how Github will feel once we start building more images tho.

That said, the xz process takes up at least 15 minutes on CI without including the build process, as GH's CI runner is only 2 cores when I checked. This means a build would be more than 30 minutes. GH CI only offers 2000 minutes/month, which means <100 builds/month. This might be good enough, especially if the build is limited to tags ans master branch only (depends on the number of images)? Although I suspect this repository will experience more burst of activities, which might be annoying to deal with. Still we might be able to start here for now.

One idea is to have a self-hosted runner running somewhere. This will allow unlimited time. I might be able to set something up on my home server if we don't abuse it... Coupled with the 2GB/file limit, it might be an okay solution.

I noticed the real-time linux documentation are being changed over the last week and this has appeared on https://wiki.linuxfoundation.org/realtime/documentation/known_limitations

We should do this as well.

It looks like the new changes to check the modified files and trigger the build are not working as expected.

Once we consider the project is ready, we will do an official release. Here is a list of things we are considering doing for the official release.

• Create new release tag and host the image here => https://github.com/ros-realtime/ros-realtime-rpi4-image/releases
• Create one or several tutorials that we can refer to in order to show how the image and the tools work
• Update Real-Time Linux guides => https://real-time-working-group.readthedocs.io/en/latest/Guides/Real-Time-Operating-System-Setup/Real-Time-Linux/rt_linux_index.html#real-time-linux
• Create https://discourse.ros.org/ announcement
• Discuss how we can further popularize this work: ROSCon talk, ROSCon workshop talk, add references in ROS 2 docs, etc
• Add image hash for integrity

Motivation

In order to execute RT applications, our ubuntu user should have access to (allocate memory, cpu affinity, thread priority).

Acceptance Criteria

• Include limits file for ubuntu user
• Test if limits are properly configure

TL;DR: Ubuntu by default has a real-time service with a rtpriority of 99 (maximum possible) that can cause scheduling latency of around ~150us. We should remove the multipath-tools package to remove that service, as it seems not necessary.

cc: @carlossvg @LanderU

The Problem

As a part of my benchmarking and testing for some blog posts I'm writing, I've noticed that the maximum latency of the Raspberry Pi 4 when running cyclictest while running stress-ng -c 4 is around 250us. This is actually quite high, as if you have a process that supposed to happen at 1kHz, you lost 25% of your compute time. I then figured I will trace the system via ftrace to see what's causing the problem:

# stress-ng -c 4 # (In a separate terminal)
# trace-cmd start -p wakeup_rt cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0 -D 60s
  plugin 'wakeup_rt'                                                                                               
# /dev/cpu_dma_latency set to 0us                                                                                  
policy: fifo: loadavg: 3.48 1.36 0.69 8/173 12907                                          
                                                                                                                   
T: 0 (12904) P:80 I:200 C: 149982 Min:     15 Act:   29 Avg:   24 Max:     313     
T: 1 (12905) P:80 I:200 C: 149801 Min:     16 Act:   26 Avg:   54 Max:     434                 
T: 2 (12906) P:80 I:200 C: 149613 Min:     16 Act:   68 Avg:   37 Max:     280           
T: 3 (12907) P:80 I:200 C: 149425 Min:     18 Act:   92 Avg:   68 Max:     266   
# trace-cmd stop
# trace-cmd show
# tracer: wakeup_rt                                                                                                
#                                                                                                                  
# wakeup_rt latency trace v1.1.5 on 5.4.140-rt64                                                                   
# --------------------------------------------------------------------                                             
# latency: 400 us, #345/345, CPU#1 | (M:preempt_rt VP:0, KP:0, SP:0 HP:0 #P:4)                                     
#    -----------------                                                                                             
#    | task: cyclictest-12905 (uid:0 nice:0 policy:1 rt_prio:80)                                                   
#    -----------------                                                                                             
#                                                                                                                  
#                    _------=> CPU#                                                                                
#                   / _-----=> irqs-off                                                                            
#                  | / _----=> need-resched                                                                        
#                  || / _---=> hardirq/softirq                                                                     
#                  ||| / _--=> preempt-depth                                                                       
#                  ||||| / _--=> preempt-lazy-depth                                                                
#                  |||||| / _-=> migrate-disable                                                                   
#                  ||||||| /     delay                                                                             
# cmd     pid      |||||||| time   |  caller                                                                       
#     \   /        ||||||||   \    |  /                                                                            
stress-n-12898     1dN.h4..    1us :    12898:120:R   + [001]   12905: 19:R cyclictest 
[omitted for brevity]
stress-n-12898     1d...3..   57us : cpu_have_feature <-__switch_to
multipat-1456      1d...3..   58us : finish_task_switch <-__schedule
[omitted for brevity]
multipat-1456      1d...3..  382us : update_curr_rt <-put_prev_task_rt
multipat-1456      1d...3..  383us : update_rt_rq_load_avg <-put_prev_task_rt
multipat-1456      1d...3..  384us : pick_next_task_stop <-__schedule
multipat-1456      1d...3..  384us : pick_next_task_dl <-__schedule
multipat-1456      1d...3..  385us : pick_next_task_rt <-__schedule
multipat-1456      1d...3..  389us : __schedule <-schedule
multipat-1456      1d...3..  389us :     1456:  0:S ==> [001]   12905: 19:R cyclictest

Note that with ftrace enabled, the entire system is slower, which explains why I'm seeing a latency of ~434us as measured by cyclictest and 400us as measured by wakeup_rt. However, the above log shows that we spent most of the time in multipat-1456, which we can find out is:

$ ps -e -o pid,class,rtprio,comm | grep 1456
   1456 RR      99 multipathd

This is a process scheduled with SCHED_RR with a priority of 99. No wonder it can get ahead of cyclictest when being scheduled, which I set to a priority of 80 (as this is common for RT applications). A look at the Ubuntu docs, it seems like multipathd is something that is involved with multi-path storage. While I'm not familiar with what it is, I don't think it is relavent to the RT base image for robotics (or even other RT applications).

Bonus picture: here's some kernelshark visualization of multipathd (blue) jumping in front of cyclictest (green). The two black lines before the blue bar (multipathd) are Marker A and Marker B respectively, which has a delta of 0.000197482s (197us). This is expected as cyclictest is constructed to wake up every 200us. This periodicity is kept very well (as indicted by the regular appearance of the green lines), until the appearance of multipathd, which steals the CPU by 139us.

Testing the solution

So I disabled multipathd and verified that it is dead:

# systemctl mask multipathd
# systemctl stop multipathd
# ps aux | grep multipath
ubuntu     13061  0.0  0.0   7692   616 pts/2    S+   17:10   0:00 grep --color=auto multipath

Then I started cyclictest with stress-ng again

# stress-ng -c 4 # (In a separate terminal)
# cyclictest --mlockall --smp --priority --interval=200 --distance=0 -D 15m

policy: fifo: loadavg: 6.79 6.17 4.06 5/166 13260           

T: 0 (13103) P:80 I:200 C:4499997 Min:     13 Act:   18 Avg:   20 Max:     132
T: 1 (13104) P:80 I:200 C:4499803 Min:     13 Act:   16 Avg:   19 Max:     115
T: 2 (13105) P:80 I:200 C:4499606 Min:     13 Act:   21 Avg:   20 Max:     136
T: 3 (13106) P:80 I:200 C:4499435 Min:     13 Act:   16 Avg:   19 Max:     138

This is significantly better result than the 250us nominal that I saw with multipathd running.

After a bit of investigation, multipathd is coming from the Ubuntu base image. I'm not sure why it is installed on the Raspberry Pi image, as it is definitely not on my desktop system. We can remove this package on image building to prevent this service from being installed.

root@rpi4-image-build:/# apt list --installed | grep multi

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                               

multipath-tools/now 0.8.3-1ubuntu2 arm64 [installed,local]

See https://wiki.linuxfoundation.org/realtime/documentation/technical_basics/sched_rt_throttling.

Basically, the default /proc/sys/kernel/sched_rt_runtime_us means that there's a chance that the kernel would throttle/sleep a RT process once it uses up 950ms out of 1s of CPU time. This could result in deadline misses. The feature was implemented before the widespread availability of SMP and thus needed to avoid complete system lockup/denial-of-service. With multicore systems (especially with isolcpu), this shouldn't be needed.

Basically, we need a startup service that writes -1 to /proc/sys/kernel/sched_rt_runtime_us.

FYI: @carlossvg @LanderU

The focal-rt-ros2 image is built with Ubuntu 20.04.3 with a bunch of basic features (rootfs, install script):

• The RT kernel, based on 5.4.140-rt64 (although there's a minor bug where the deb filename doesn't match the content from the rt kernel repo).
• Pinned cpu frequency via /etc/default/cpu-frequency.
• Removed snap and fwupd as they uses more ram than needed, although maybe snap is something we want to keep as some people ship code via snap?
• Installed ros galactic
• Installed some misc utility packages
• Automatically source the galactic setup.sh for bash.

I made these up. They're not necessarily the best choices. We need to figure out what we want to include in the default. We probably need:

• Sane /etc/security/limits.conf defaults for RT.
• Optional isolcpus settings similar to what the reference system suggests, either as a compile time configuration or maybe via something similar to the ssh file that raspberry pi OS make you touch in the boot partition to enable ssh, or maybe a command line utility.

Not sure if there are anything else we need, but we can discuss here. If we want to remove something from what I made up, that's fine as well.

If I want to cross-compile a ros2 package from sources available on GitHub in a custom image, I suppose I need to add a phase2-host script.

In this script I first clone all the repositories to be cross-compiled in the process using vcs (from the vcs-tools package), and then it seems a little bit fuzzy when I want to automatically install ros2 dependencies using rosdep as the script is executed as root ...

Here is the part of the script which is causing me troubles :

vcs import < repos.yaml
rosdep update
rosdep install --from-paths src --ignore-src --rosdistro humble -r -y
colcon build --cmake-force-configure --cmake-args -DCMAKE_TOOLCHAIN_FILE=$CMAKE_TOOLCHAIN_FILE

I get the following warn:

+ rosdep update
reading in sources list data from /etc/ros/rosdep/sources.list.d
Warning: running 'rosdep update' as root is not recommended.
  You should run 'sudo rosdep fix-permissions' and invoke 'rosdep update' again without sudo.
[...]

And then nothing is installed as on my system everything is already setup ( I have already rosdep install everything needed on my system as I already have compiled the code somewhere else ...)

+ rosdep install --from-paths src --ignore-src --rosdistro humble -r -y
#All required rosdeps installed successfully

But then, when colcon is trying to build the workspace, I get:

+ colcon build --cmake-force-configure --cmake-args -DCMAKE_TOOLCHAIN_FILE=[...]/data/toolchain.cmake
[...]
--- stderr: my_pkg                                                                                               
CMake Error at CMakeLists.txt:14 (find_package):
  By not providing "Findament_cmake.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "ament_cmake", but CMake did not find one.

  Could not find a package configuration file provided by "ament_cmake" with
  any of the following names:

    ament_cmakeConfig.cmake
    ament_cmake-config.cmake
[...]

I think packages (ament_cmake, and others ...) are installed user space and not accessible root space (but maybe I'm wrong) and so the compile process which is launched as root is not able to find the required packages.

How can I get this working ?

I see that the image is compressed by the pigz from the original 3.9G to 1.5G, however it looks like that @carlossvg produces a release using xz compression which is only 932 MB

Before compression:
total 3.9G
-rw-r--r-- 1 runner docker 4.0G Sep 27 13:43 ubuntu-22.04.1-rt-ros2-arm64+raspi.img
+ cd out
+ echo 'After compression:'
+ pigz ubuntu-22.04.1-rt-ros2-arm64+raspi.img
After compression:
+ ls -lh .
total 1.5G

• Add ubuntu user to the tracing group => https://gitlab.com/ros-tracing/ros2_tracing#tracing
• Fix LTTng kernel tracer installation failure => See below
• Add ros2_tracing and dependencies => https://github.com/ros-realtime/ros-realtime-rpi4-image/blob/master/focal-rt-ros2/rootfs/setup/phase1.sh#L61

LTTng kernel tracer installation failure

Note: This issue it's probably might be related to https://github.com/ros-realtime/linux-real-time-kernel-builder but I decided to report it here so we are able to validate the fix with the resulting image.

The installation of the LTTng kernel tracer fails with the following error message:

sudo apt-get install lttng-modules-dkms
Reading package lists... Done
Building dependency tree       
Reading state information... Done
lttng-modules-dkms is already the newest version (2.12.5-1ubuntu2~20.04.1).
0 upgraded, 0 newly installed, 0 to remove and 110 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Setting up lttng-modules-dkms (2.12.5-1ubuntu2~20.04.1) ...
Removing old lttng-modules-2.12.5 DKMS files...

------------------------------
Deleting module version: 2.12.5
completely from the DKMS tree.
------------------------------
Done.
Loading new lttng-modules-2.12.5 DKMS files...
Building for 5.4.140-rt64
Building initial module for 5.4.140-rt64
ERROR (dkms apport): kernel package linux-headers-5.4.140-rt64 is not supported
Error! Bad return status for module build on kernel: 5.4.140-rt64 (aarch64)
Consult /var/lib/dkms/lttng-modules/2.12.5/build/make.log for more information.
dpkg: error processing package lttng-modules-dkms (--configure):
 installed lttng-modules-dkms package post-installation script subprocess returned error exit status 10
Errors were encountered while processing:
 lttng-modules-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

Checking the log:

cat /var/lib/dkms/lttng-modules/2.12.5/build/make.log
DKMS make.log for lttng-modules-2.12.5 for kernel 5.4.140-rt64 (aarch64)
Fri Mar  4 16:10:56 CET 2022
make: Entering directory '/usr/src/linux-headers-5.4.140-rt64'
  CC [M]  /var/lib/dkms/lttng-modules/2.12.5/build/lttng-ring-buffer-client-discard.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.5/build/lib/ringbuffer/ring_buffer_backend.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.5/build/tests/probes/lttng-test.o
/var/lib/dkms/lttng-modules/2.12.5/build/probes/Kbuild:166: Files ./fs/btrfs/*.h not found. Probe "btrfs" is disabled. Use full kernel source tree to enable it.
/var/lib/dkms/lttng-modules/2.12.5/build/probes/Kbuild:177: Files ./fs/ext4/*.h not found. Probe "ext4" is disabled. Use full kernel source tree to enable it.
/var/lib/dkms/lttng-modules/2.12.5/build/probes/Kbuild:208: File ./drivers/base/regmap/trace.h not found. Probe "regmap" is disabled. Need Linux 4.1+ kernel source tree to enable it.
  CC [M]  /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-sched.o
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[2]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.5/build/tests/probes/lttng-test.o] Error 2
make[2]: *** Deleting file '/var/lib/dkms/lttng-modules/2.12.5/build/tests/probes/lttng-test.o'
make[1]: *** [scripts/Makefile.build:519: /var/lib/dkms/lttng-modules/2.12.5/build/tests] Error 2
make[1]: *** Waiting for unfinished jobs....
  CC [M]  /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-irq.o
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[1]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.5/build/lttng-ring-buffer-client-discard.o] Error 2
make[1]: *** Deleting file '/var/lib/dkms/lttng-modules/2.12.5/build/lttng-ring-buffer-client-discard.o'
  CC [M]  /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-timer.o
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[2]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.5/build/lib/ringbuffer/ring_buffer_backend.o] Error 2
make[2]: *** Deleting file '/var/lib/dkms/lttng-modules/2.12.5/build/lib/ringbuffer/ring_buffer_backend.o'
make[1]: *** [scripts/Makefile.build:519: /var/lib/dkms/lttng-modules/2.12.5/build/lib] Error 2
  CC [M]  /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-kmem.o
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[2]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-sched.o] Error 2
make[2]: *** Deleting file '/var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-sched.o'
make[2]: *** Waiting for unfinished jobs....
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[2]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-irq.o] Error 2
make[2]: *** Deleting file '/var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-irq.o'
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[2]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-timer.o] Error 2
make[2]: *** Deleting file '/var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-timer.o'
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[2]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-kmem.o] Error 2
make[2]: *** Deleting file '/var/lib/dkms/lttng-modules/2.12.5/build/probes/lttng-probe-kmem.o'
make[1]: *** [scripts/Makefile.build:519: /var/lib/dkms/lttng-modules/2.12.5/build/probes] Error 2
make: *** [Makefile:1765: /var/lib/dkms/lttng-modules/2.12.5/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.4.140-rt64'

uname -a
Linux ubuntu 5.4.140-rt64 #1 SMP PREEMPT_RT Sat Oct 16 17:57:52 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Required for #7

A few things that needs to be done:

1. Standardize the script names and the paths (like the rootfs directory).
2. Right now the builder is called via sudo ./builder/main.sh focal-ros2-rt/vars.sh. The idea is to change it so we can call: sudo ./builder/main.sh focal-rt-base ros-galactic custom-stuff, where each argument is a directory. The builder then applies the rootfs and scripts in order, allowing for some sort of overlay. Not sure if this will be the final structure, I have to experiment with the code a little bit to make sure it is not spaghetti.

A minor thing to consider:

1. Consider changing CI such that it only builds when build scripts get changed and not when READMEs gets changed to conserve Github actions limit.

After that, we can re-evaluate and see if we can do some additional longer term improvements (so not for now):

1. Make the builder runnable from any directory, so one can bring their own custom overlay. This likely requires replacing the current Makefile with some sort of python program.

cc: @LanderU @carlossvg

@shuhaowu reported some CPU frequency changes. A possible explanation could be something related with power management. We should review all the related configurations. From the top of my head these are some of the things that we should take into account:

• Apparently, power issues could cause a kernel log message which might affect the latency measurements
  - https://pimylifeup.com/raspberry-pi-low-voltage-warning/
  - https://forums.raspberrypi.com/viewtopic.php?t=224493
• dwc_otg has to be disabled
  - https://community.emlid.com/t/rt-kernel-unusual-dwc-otg-high-8k-s-interrupt-rate-using-quite-some-cpu/1002
  - https://www.osadl.org/Single-View.111+M5c03315dc57.0.html
  - https://www.osadl.org/monitoring/patches/r7s3/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch.html
• CPU idle
  • https://wiki.linuxfoundation.org/realtime/documentation/howto/debugging/no-cpu-idle
  • A Checklist for Writing Linux Real-Time Applications: Kernel configuration
  • Enea Linux Real-Time Guide (2.8.2 CPU Power States and 3.2.2 Optimize Power Save) https://linux.enea.com/4.0/documentation/book-enea-linux-realtime-guide.pdf

When I worked on this builder originally, I was able to use it to cross compile some libraries and built it into the image. This can be done via a custom phase2-outside script and a cmake-toolchain file with the right SYSROOT settings. However, this is not the only use case. We should consider the following usecases:

1. Cross compile binaries for integration into the custom image. This is already feasible as described above, but requires more documentation and testing.
2. Cross compile binaries so it can be uploaded to an already running Raspberry Pi. This would be useful for developing code for RT applications. There is technically https://github.com/ros-tooling/cross_compile, but this tool appears to be using qemu-user-static only. While I haven't tried that tool myself, from my experience with qemu-user-static, the compilation times could be 20-30x slower than using a cross compiler, which is frequently unacceptable. Since this project produces a SYSROOT, we can stop the builder from unmounting the loop device, which would allow the user to cross compile any packages, at least for the raspberry pi.

It would be trivial to "hack together" something to achieve 2 (you can simply stop the builder before it unmounts), but to make it a "friendly" tool will require some more thoughts. For example, we may need to create a program that allows the builder here to run from any directory (See #16). We might also consider having a default cmake-toolchain file, as well as any necessary colcon mixins and what not.

This issue is so we can discuss and document the progress.

cc: @carlossvg @LanderU

Hi,
I was running the official Raspberry Pi OS on my Raspberry Pi 400, and it can connect to 5G WiFi. However, after switching to ros-realtime-rpi4-image, it can only connect to 2.4G WiFi. Moreover, after I install ubuntu-desktop, it could not find WiFi adapter.

After was merged now it's possible to remove boot/firmware paths files.

See https://github.com/ros-realtime/ros-realtime-rpi4-image/blob/master/focal-rt-ros2/rootfs/setup/phase1.sh#L41

Even after I do apt-get purge it is still there

$ sudo apt-get purge linux-image-5.15.74-rt53-raspi
$ sudo reboot
$ apt list --installed | grep linux-image
linux-image-5.15.0-1027-raspi/jammy-updates,jammy-security,now 5.15.0-1027.29 arm64 [installed,automatic]
linux-image-raspi/jammy-updates,jammy-security,now 5.15.0.1027.24 arm64 [installed]

$ uname -a
root@ubuntu:/home/ubuntu# uname -a
Linux ubuntu 5.15.74-rt53-raspi #1 SMP PREEMPT_RT Mon Jan 16 13:25:26 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

This is because /boot/firmware/vmlinuz is still RT kernel

$ ls -la /boot/firmware/vmlinuz*
-rwxr-xr-x 1 root root 10320014 May  6 11:03 /boot/firmware/vmlinuz
-rwxr-xr-x 1 root root 10338087 May  6 10:57 /boot/firmware/vmlinuz.bak

$ cp /boot/vmlinuz-5.15.0-1027-raspi /boot/firmware/vmlinuz
$ cp /boot/initrd.img-5.15.0-1027-raspi /boot/firmware/initrd.img
$ sudo reboot
$ uname -a
Linux ubuntu 5.15.0-1027-raspi #29-Ubuntu SMP PREEMPT Mon Apr 3 10:12:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

even after that

ubuntu@ubuntu:~$ apt list --installed | grep rt53
linux-libc-dev/now 5.15.74-rt53-raspi-1 arm64 [installed,local]

ubuntu@ubuntu:~$ sudo apt-get install --reinstall linux-libc-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Reinstallation of linux-libc-dev is not possible, it cannot be downloaded.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$ wget http://ports.ubuntu.com/pool/main/l/linux/linux-libc-dev_5.15.0-25.25_arm64.deb
$ sudo dpkg -i linux-libc-dev_5.15.0-25.25_arm64.deb
$ apt list --installed | grep linux-libc
$ linux-libc-dev/jammy,now 5.15.0-25.25 arm64 [installed,upgradable to: 5.15.0-71.78]
```

IMAGE_URL="https://cdimage.ubuntu.com/releases/20.04.3/release/ubuntu-20.04.3-preinstalled-server-arm64+raspi.img.xz"

Suggested url:

https://old-releases.ubuntu.com/releases/focal/ubuntu-20.04.3-preinstalled-server-arm64+raspi.img.xz

We have been told there is a tool, packer, which can be used to build images. We could evaluate packer and see if we could leverage an existing tool to create our ROS 2 image builder.

@shuhaowu Are you familiar with this tool?

Resources:

• https://www.packer.io/
• https://medium.com/@source4learn/how-to-build-a-custom-raspberry-pi-os-image-using-packer-da83be261687
• https://linuxhit.com/build-a-raspberry-pi-image-packer-packer-builder-arm/
• https://github.com/solo-io/packer-plugin-arm-image

Hi! I have a setup running on my pi4 and it works. I am wondering if anybody tried running this on pi5?

We have lots of docs about the image builder, but no docs on how to use the image.

This should be addressed. I'll do it as I work through the ROScon presentations.

Hi,

I was wondering if it is possible to also build the rt-image for Raspberry Pi 3?
Running ROS 2 humble based on ubuntu 22.04 server seems to work very well although its not fully supported for the rpi 3 (especially with a faster ssd and a some swap).

Hi,

I am assuming ubuntu 20.04 is currently used for the rpi4 image
In this post the ubuntu devs explain how interacting with the GPIO pins has changed since 21.04:
https://ubuntu.com/tutorials/gpio-on-raspberry-pi#1-overview

I'd like to use 22.04 going forward so I don't have to rewrite my code which uses GPIO pins. Is 22.04 already currently being looked at?

Once we release the image, I suggest that we at least submit a lightning talk to ROScon 2022. Not sure who wants to do it, I'm happy to do it if no one else wants, or we can do it as a group.

I don't think we need a full-length talk, as there's not that much to talk about. However, if we get cross-compile to work nicely and documented, we might be able to fill something longer.

cc: @carlossvg

Example: https://github.com/ros-realtime/ros-realtime-rpi4-image/blob/master/focal-rt/rootfs/etc/profile.d/99-source-ros.sh

We should keep the released image names consistent and the compression algorithms consistent. Looking at the current pre-releases, it seems like the the xz algorithm is more efficient than 7z.

The release tags should also be consistent. We should keep these constant so the documentation (#51) can refer to a consistent image.

Since we are updated to Ubuntu 22.04.1, I can create a new release soon.

Having the demo nodes preinstalled would make it easier to write the docs for beginners (see #51). This is the same node that the ROS installation guide gets you to run.

are you sure both the username and password is "ubuntu", i got wrong password

As I'm testing the image for #4, I see that the CPU governor is set to ondemand after I flashed the image. This shouldn't be the case, as setup/phase1.sh disables the ondemand.service startup file and ros-realtime/linux-real-time-kernel-builder@483090d adds performance to always be enabled? Also, this script should be started on boot (and it is as far as I can tell) should also set it.

Not quite sure what's going on. Will have to double check later. Not the highest priority item right now.

ubuntu@ubuntu:~$ uname -a
Linux ubuntu 5.4.140-rt64 #1 SMP PREEMPT_RT Sat Oct 16 17:57:52 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@ubuntu:~$ sudo systemctl status ondemand
● ondemand.service - Set the CPU Frequency Scaling governor
     Loaded: loaded (/lib/systemd/system/ondemand.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
ubuntu@ubuntu:~$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
ondemand
ondemand
ondemand
ondemand

cc: @LanderU since you're testing it as well so it's just a FYI.