gitbls / sdm Goto Github PK
View Code? Open in Web Editor NEWRaspberry Pi SD Card Image Manager
License: MIT License
Raspberry Pi SD Card Image Manager
License: MIT License
sdm looks very interesting. I have a few customizations I do/need that I'm not clear on how to best handle them with sdm
My base server is a Centos 7 box. It looks like I'll have to set up sdm on one of my Pi devices.
Thanks for your work on this project, and for addressing my questions.
I'd like to install docker, and pull down a few images, prior to the system being available for the first time. Per specific pi instructions, the most reliable way of getting it up and running is with curl -sSL https://get.docker.com | sh
, but when I attempt to run this at various different phases, it fails. This type of setup also doesn't seem to clearly fit into the "apps" install scripts.
I suspect the right time to run this and the subsequent image fetching is after first-boot, but the custom-phase-script interface doesn't seem to allow for this customization. Reading through the docs, I'm guessing using a --bootscripts
setting is the right procedure, but it doesn't seem to be spelled out. Roughly:
nspawn
, manually create and edit a script in /usr/local/sdm/1piboot
called 0-docker.sh
per the --bootscripts flag.What's not clear to me is if the --bootscripts
flag can allow me to pass in a file during burn, which it will subsequently run, or if I pass in an argument of a name of a script that already exists, so that it will run it properly. Any chance you could clarify please?
Trying to figure out how to add a bash script to the image and make it run after all the rest of the processes are done. Any help or examples greatly appreciated.
Alan Bell
hi @gitbls
Just testing on a fresh image after re-running EZ installer again (over top of my previous version). Now with version 8.6, I got the following error. It looks harmless, but thought I'd report it. I checked the /mnt/sdm/usr/local/sdm/local-plugins
directory when sdm dropped me to a root shell and the directory does exist and it is empty.
* Mount IMG '/home/user/Downloads/2023-05-03-raspios-bullseye-arm64.img.test'
mount: /dev/loop11p2 mounted on /mnt/sdm.
mount: /dev/loop11p1 mounted on /mnt/sdm/boot.
* Unmount IMG for --extend
umount: /mnt/sdm/boot unmounted
umount: /mnt/sdm unmounted
* Extend IMG by 200MB...
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.283942 s, 739 MB/s
* Resize partition 2 of '/home/user/Downloads/2023-05-03-raspios-bullseye-arm64.img.test
* Remount IMG to resize the file system
* Mount IMG '/home/user/Downloads/2023-05-03-raspios-bullseye-arm64.img.test'
mount: /dev/loop11p2 mounted on /mnt/sdm.
mount: /dev/loop11p1 mounted on /mnt/sdm/boot.
* Resize the file system
% (Ignore on-line resizing message)
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/loop11p2 is mounted on /mnt/sdm; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/loop11p2 is now 1061858 (4k) blocks long.
chmod: cannot access '/mnt/sdm//usr/local/sdm/local-plugins/*': No such file or directory
* Start Configuration
> Command Line: /usr/local/bin/sdm --customize
/home/user/Downloads/2023-05-03-raspios-bullseye-arm64.img.test --l10n --restart --bootset
serial=0,boot_wait=1,camera=0,i2c=0,onewire=1,blanking=0,overscan=0,boot_behaviour=B4
--regen-ssh-host-keys --user user --disable piwiz --showapt --ssh service --extend --xmb 200
* Host Information
Hostname: hostname
uname: Linux hostname 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64
x86_64 x86_64 GNU/Linux
os-release Name: "Ubuntu"
os-release Version: "22.04.3 LTS (Jammy Jellyfish)"
sdm Version: V8.6
* IMG Information
Date: 2023-05-03
RasPiOS Version: 11
os-release Version: "11 (bullseye)"
% sdm will try to use systemd-nspawn on this 'x86-64' system
If this does not work please post a GitHub issue
> IMG '/home/user/Downloads/2023-05-03-raspios-bullseye-arm64.img.test' has 656616 1K-blocks free
> Copy sdm to /usr/local/sdm in the IMG
I want to copy 3 files in different target directories and included the new copyfile plugin 3 times. Only the last files ends up in the image. If I delete copyfile call 2 and 3, leaving only the first, this file ends up in the image, so my call seems to be correct.
The logs show all three calls of copyfiles with the different files .bash_profile
, .xinitrc
and postcardscanner.service
, but it says that it copies always postcardscanner.service
> Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase 1 with arguments:
'from=./files/.bash_profile|to=/home/pi/.bash_profile|mkdirif=yes'
* Plugin copyfile: Start Phase 1
> Plugin copyfile: Copy 'postcardscanner.service' from /etc/sdm/assets/copyfile to
'/etc/systemd/system/postcardscanner.service'
* Plugin copyfile: Complete Phase 1
> Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase 1 with arguments:
'from=./files/.xinitrc|to=/home/pi/.xinitrc|mkdirif=yes'
* Plugin copyfile: Start Phase 1
> Plugin copyfile: Copy 'postcardscanner.service' from /etc/sdm/assets/copyfile to
'/etc/systemd/system/postcardscanner.service'
* Plugin copyfile: Complete Phase 1
> Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase 1 with arguments:
'from=./files/postcardscanner.service|to=/etc/systemd/system/postcardscanner.service|mkdirif=yes
'
* Plugin copyfile: Start Phase 1
> Plugin copyfile: Copy 'postcardscanner.service' from /etc/sdm/assets/copyfile to
'/etc/systemd/system/postcardscanner.service'
* Plugin copyfile: Complete Phase 1
* ```
The docs say
> Multiple --plugin switches can be used on the command line. This includes specifying the same plugin multiple times
The apps plugin is explicitly mentioned as supported here, but copyfile is not mentioned. Is this a bug or does copyfile just not support multiple calls?
I've just now been playing with setting some things in --burn (having previously done all modifications in --customize, and then burning using another image burning product). This is for an image intended for a pi 3a+ with a pimoroni fan shim. Currently, in either burn case, I install their fan control software in a post-boot customization script of my own. Then I run some stress testing to see how the arm frequency and temperature vary under stress. Besides using the fan, I want to increase the system's temp_soft_limit to 70 in /boot/config.txt (the default being to start throttling at 60 degrees). I had successfully used --bootadd temp_soft_limit:70 during --customize to set that setting. Didn't seem to be happening when I tried the same parameter during --boot - the throttling was happening at 60 degrees, so I checked and the line DID NOT appear in /boot/config.txt.
Here are excerpts from the two scripts I used, the first to do --bootadd in --customiize, the second to do it in --burn:
sudo sdm \
--batch \
--extend --xmb 512 \
--customize \
--wpa /home/pi/Documents/sdm/myfiles/wpa_supplicant.conf \
--L10n \
--reboot 5 \
--disable piwiz,bluetooth \
--regen-ssh-host-keys \
--user pi --password-pi mypassword \
--ssh service \
--plugin apps:"apps=figlet hdparm mc nmap screen stress|name=apps" \
--plugin apps:"apps=passwordsafe|name=xapps" \
--aptcache 192.168.0.118 \
--apt-dist-upgrade \
--swap 1024 \
--csrc /home/pi/Documents/sdm/csrc \
--plugin pl-plugin \
--lxde-config lxterminal:/home/pi/Documents/sdm/myfiles/lxterminal.conf --plugin graphics \
--bootset boot_splash=1,boot_behaviour=B1 \
--host pi3a83 \
--bootadd temp_soft_limit:70 \
/home/pi/Downloads/bullseye64.img
IMGDIR=~/Downloads
CSRC=/home/pi/Documents/sdm/csrc
time sudo sdm \
--batch \
--extend --xmb 512 \
--customize \
--wpa /home/pi/Documents/sdm/myfiles/wpa_supplicant.conf \
--L10n \
--reboot 5 \
--disable piwiz,bluetooth \
--regen-ssh-host-keys \
--user pi --password-pi mypassword \
--ssh service \
--plugin apps:"apps=figlet hdparm mc nmap screen stress|name=apps" \
--plugin apps:"apps=passwordsafe|name=xapps" \
--aptcache 192.168.0.118 \
--apt-dist-upgrade \
--swap 1024 \
--csrc $CSRC \
--plugin pl-plugin \
--lxde-config lxterminal:/home/pi/Documents/sdm/myfiles/lxterminal.conf --plugin graphics \
--bootset boot_splash=1,boot_behaviour=B1 \
$IMGDIR/bullseye64.img
# --plugin runatboot:"user=pi|sudoswitches=-H|script=/home/pi/bin/refresh_pi3a|output=/home/pi/runatboot_refresh_pi3a.log" \
# --host pi3a83 \
# --bootadd temp_soft_limit:70 \
time sudo sdm \
--burn /dev/sdc \
--expand-root \
--csrc $CSRC \
--plugin pl-plugin \
--host pi3a83b \
--bootadd temp_soft_limit:70 \
$IMGDIR/bullseye64.img
$ sudo ln -s /usr/local/sdm/sdm /usr/bin/sdm
$ sudo sdm
/usr/bin/sdm: line 606: /usr/bin/sdm-cparse: No such file or directory
/usr/bin/sdm: line 607: iswsl: command not found
? No command specified (--aptmaint --burn, --customize, --explore, --mount)
This issue originates in sdm
line 605: src=$(dirname $0)
can be replaced by e.g. src=$(dirname "$(realpath "$0")")
.
I downloaded the Bullseye xz file from the official website and extract to my hdd.
Then I tried to extend the partition:
$ sudo sdm --extend ~/Scaricati/2023-05-03-raspios-bullseye-arm64.img
* Extend IMG by 2048MB...
2048+0 Record in
2048+0 registra
2147483648 byte ( 2,1 GB, 2,0 GiB) copiato, 0,557759 s, 3,9 GB/s
* Resize partition 2 of '/home/mark/Scaricati/2023-05-03-raspios-bullseye-arm64.img
* Remount IMG to resize the file system
* Mount IMG '/home/mark/Scaricati/2023-05-03-raspios-bullseye-arm64.img'
mount: /dev/loop24p2 mounted on /mnt/sdm.
mount: /dev/loop24p1 mounted on /mnt/sdm/boot.
* Resize the file system
% (Ignore on-line resizing message)
resize2fs 1.47.0 (5-Feb-2023)
The filesystem is already 1010688 (4k) blocks long. Nothing to do!
umount: /mnt/sdm/boot unmounted
umount: /mnt/sdm unmounted
As the output says, it did not resize the partition.
I also tried to add the same command along with --customize
but the behavior is the same.
Actually, it adds the desired space after the partition. The img file grows and inspecting it with the Ubuntu Disk tool it appears as an "empty space".
Am I doing it wrong?
Received the following error:
% sdm will try to use systemd-nspawn on this 'x86-64' system
If this does not work please post a GitHub issue
/usr/local/sdm/sdm-cmdsubs: line 215: [: !=: unary operator expected
> Disable unneeded RasPiOS firstboot service in cmdline.txt; both --expand-root and
--regen-ssh-host-keys set
Took a quick look at line 215, and its just missing double quotes around the variable $bbootadd
- presumably the variable was an empty string.
[ $bbootadd != "" ] && bootadd="$bbootadd"
Probably should quote all variables in test commands. ;)
I downloaded the latest RaspianOS image from https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64.img.xz and tried to burn it to an SD card using sdm. However, the process does not run through cleanly, as there are errors on the console after the actual burning has finished:
$ sdm --burn /dev/mmcblk0 2022-09-22-raspios-bullseye-arm64.img
% hostname not specified with --host; will not be written
* Burn '2022-09-22-raspios-bullseye-arm64.img' to '/dev/mmcblk0'...
dd if=2022-09-22-raspios-bullseye-arm64.img of=/dev/mmcblk0 status=progress bs=16M iflag=direct
4261412864 bytes (4.3 GB, 4.0 GiB) copied, 186 s, 23.0 MB/s
254+1 records in
254+1 records out
4265607168 bytes (4.3 GB, 4.0 GiB) copied, 185.808 s, 23.0 MB/s
> Set new disk ID 'ce5cc681' on '/dev/mmcblk0'
Disk identifier changed from 0x63ee4f38 to 0xce5cc681.
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or partx(8).
Syncing disks.
> Disk ID ce5cc681 set successfully with sfdisk
mount: /mnt/sdm.167869: mount(2) system call failed: File exists.
mount: /mnt/sdm.167869/boot: mount point does not exist.
> Set new Disk ID 'ce5cc681' in /etc/fstab and /boot/cmdline.txt
sed: can't read /mnt/sdm.167869/etc/fstab: No such file or directory
sed: can't read /mnt/sdm.167869/boot/cmdline.txt: No such file or directory
sed: can't read /mnt/sdm.167869/boot/cmdline.txt: No such file or directory
grep: /mnt/sdm.167869/boot/cmdline.txt: No such file or directory
sed: can't read /mnt/sdm.167869/boot/cmdline.txt: No such file or directory
sed: can't read /mnt/sdm.167869/boot/cmdline.txt: No such file or directory
> RasPiOS firstboot service not disabled; at least one of --expand-root and --regen-ssh-host-keys not set
% IMG '2022-09-22-raspios-bullseye-arm64.img' is not sdm-enhanced
Logs and configuration updates will not be written
* Storage '/dev/mmcblk0' is ready
umount: /mnt/sdm: target is busy.
The Raspberry doesn't boot either, I guess the image wasn't flashed correctly.
Here I read:
sudo sdm --mount raspios-image.img
OR
sudo sdm --mount /dev/sdX
Mounts the IMG file (first example) or SD Card (second example) onto the running system. This enables you to manually and easily copy files from the running RasPiOS system into the IMG.
I'm not sure to understand.
This command is intended to be used on the development machine (where the IMG file is located) and it will hardly be a RasPiOS system. It can, but usually it would be any common distro like Ubuntu, Debian, Arch... RasPiOS is the target os in my opinion.
Hence why you assume the running system is the RasPiOS?
Upon https://github.com/gitbls/sdm/wiki/Command-Details I have read the following:
sudo /usr/local/sdm/sdm --burnfile customized-for-myhostname.img --host myhostname raspios-image.img
Burns the IMG file to the specified SD Card Image and sets the hostname. The customized IMG file must be burned to an SD Card to be used.
Does that mean that will generate an output image file from customized-for-myhostname.img
into raspios-image.img
Into which sd card will be burned? I fail to understand it.
I’m building an image wherein I’d like the user to be able to go through this flow if they’ve moved to an unfamiliar network. Basically if a familiar network is available, great. Otherwise, launch into the sdm-cportal experience. Is this possible via one of the scripts at the moment, or could you point me in the direction of what I might investigate to accomplish this?
I use SDM with the following switches:
user_name=mironibox
sudo sdm --customize \
(...)
--rclocal "exec 1>/home/$user_name/rc.local.log 2>&1 # send stdout and stderr from rc.local to a log file" \
--rclocal "set -x # tell sh to display commands before execution" \
--rclocal "bash /home/$user_name/hotspot.sh" \
The resulting /etc/rc.local in the image is:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
printf "My IP address is %s\n" "$_IP"
fi
exec 1>/home/mironibox/rc.local.log 2>&1
exit 0
Issues:
Maybe both issues are connected.
So I’ve been experimenting with sdm per my previous issue #25 filed, and have yet to be able to test the output. While the image is burned successfully, the post-burn scripts that do things like edit hostnames fail with an error about read-only disk partitions. I’ll follow-up with the actual error text, but just wanted to summarize first to see if there’s something obvious I should be doing.
Awesome project, thank you,
I have just installed and setup a complete new zone with it and it all appears to work fine apart from the lack of reverse lookup zones.
Is this correct of did I fluff something?
When runningsdm
I supply a cron file via --cron-d foo
. I also specify a user via --user
. Running ls -la foo
I see:
-rw------- 1 pi pi 405 Jul 7 2023 /etc/cron.d/foo
Why is the file owner pi
? Is this a user account that is always created on the device even though I specify something else via --user
(I do indeed see pi
in /etc/passwd
)? When investing why my task isn't running I see in /var/log/syslog
(ignore the incorrect datetime)
May 3 03:13:01 raspberrypi cron[411]: (*system*foo) WRONG FILE OWNER (/etc/cron.d/foo)
files in /etc/cron.d/
require a user associated with each task as far as I can tell. I was initially trying with the user specified via --user
. Also the same error message when trying with user pi
in each task of the cron file.
Reading Debian's man cron 8
I see
/etc/crontab and the files in /etc/cron.d must be owned by root, and must not be group- or other-writable.
Running sudo chown root:root /etc/cron.d/foo
seemed to fix my issue. I can change the file owner on boot but I'm guessing this is something sdm should be doing if my understanding is correct.
Hi! Thanks for the work on sdm, it's proving to be a great help in consistently creating images. I am interested in running a script on the first boot for my use case. I see the documentation in Command-Details.md states
--bootscripts
— Directs sdm-firstboot to run the boot scripts in 1piboot/*.sh.
When I couldn't get this to work by creating a directory next to where I was running sdm
, I dug in further and saw here that the --bootscripts
flag will
Run the scripts /usr/local/sdm/1piboot/0*-*.sh during first boot
After some more digging I see in Custom-Phase-Script.md
that
If a Custom Phase Script wants to run a script at boot time, even if --bootscripts is not specified, the Custom Phase script should put the script in the IMG in /etc/sdm/0piboot and named 0*-*.sh (e.g., 010-customize-something.sh).
This is actually exactly what I wanted to begin with, it just took a while to find this information. I am using a Plugin but it seems to work with one as well. Now my plugin will copy the scripts that I want into the appropriate directory. This information is not present in the plugin page however so it wouldn't be obvious from looking at that alone.
I'm not sure what the best way to rectify all this is, it just appears to be confusing as is and hard to find details on scripts at first boot. There may even be somewhere this is spelled out better, but it hasn't been easy to find. Perhaps a doc called first-boot-scripts
could clarify this?
I read through the documentation and was testing out the sample command as shown in https://github.com/gitbls/sdm/wiki/Example:-Commands. However, I cannot build a customized image, because of the following error:
* Mount IMG '2022-09-22-raspios-bullseye-arm64.img.xz'
losetup: 2022-09-22-raspios-bullseye-arm64.img.xz: Warning: file does not fit into a 512-byte sector; the end of the file will be ignored.
mount: /mnt/sdm: special device /dev/loop27p2 does not exist.
? Error mounting IMG '2022-09-22-raspios-bullseye-arm64.img.xz'
I'm running this on Ubuntu 22.04 (64 bit).
Note: I changed the command from the example a bit by removing irrelevant switches and adding switches the script reported as mandatory:
removed:
--apps
--user
--nowpa
Hello @gitbls
i was coming across sdm from a couple days and find it very useful. Thanks for your work
currently I'm facing a issue with the copyfile plugin. Unfortunately it does not copy any of my files.
Here my sdm command:
sudo sdm --customize \
--batch \
--L10n \
--restart \
--disable piwiz \
--host myhostname\
--plugin apps:"apps=samba incron vim tree" \
--plugin copyfile:"from=/root/s2c-builder/src/incron.allow|to=/etc|chown=root:incron" \
--plugin copyfile:"from=/root/s2c-builder/src/test.txt|to=/root" \
--plugin samba:smbconf=/root/s2c-builder/src/smb.conf \
--regen-ssh-host-keys \
--user "${PI_USER}" \
--password-user "${PI_PASSWORD}" /root/s2c-builder/images/2023-05-03-raspios-bullseye-armhf-lite.img)"
The history looks like this. (Phase 0 and post-install)
2023-09-06 07:22:30 * Start Configuration
2023-09-06 07:22:30 > Command Line: /usr/local/bin/sdm --customize --batch --L10n --restart --disable piwiz --host
2023-09-06 07:22:30 scan2cloud --plugin apps:apps=samba incron vim tree --plugin
2023-09-06 07:22:30 copyfile:from=/root/s2c-builder/src/incron.allow|to=/etc/|chown=root:incron --plugin
2023-09-06 07:22:30 copyfile:from=/root/s2c-builder/src/test.txt|to=/root --plugin
2023-09-06 07:22:30 samba:smbconf=/root/s2c-builder/src/smb.conf --regen-ssh-host-keys --user myuser
2023-09-06 07:22:30 --password-user mystroongpassword /root/s2c-builder/images/2023-05-03-raspios-bullseye-armhf-lite.img
2023-09-06 07:22:30 * Host Information
....
2023-09-06 07:22:30 * Start Phase 0 image customization
2023-09-06 07:22:30 *+Phase 0 sdm configuration
2023-09-06 07:22:30 *+Phase 0 Network configuration
2023-09-06 07:22:30 *+Phase 0 User configuration
2023-09-06 07:22:30 *+Phase 0 System configuration
2023-09-06 07:22:30 *+Phase 0 Raspberry Pi-specific configuration
2023-09-06 07:22:30 > Create user 'anykey' home directory /home/anykey
2023-09-06 07:22:30 > Run Plugins Phase '0'
2023-09-06 07:22:30 > Run Plugin 'apps' (/mnt/sdm/usr/local/sdm/plugins/apps) Phase 0 with arguments: 'apps=samba
2023-09-06 07:22:30 incron vim tree'
2023-09-06 07:22:30 * Plugin apps: Start Phase 0
2023-09-06 07:22:30 > Plugin apps: Keys/values found:
2023-09-06 07:22:30 apps: samba incron vim tree
2023-09-06 07:22:30 > Plugin apps [default]: Saved apps list: samba incron vim tree
2023-09-06 07:22:30 * Plugin apps: Complete Phase 0
2023-09-06 07:22:30 > Run Plugin 'copyfile' (/mnt/sdm/usr/local/sdm/plugins/copyfile) Phase 0 with arguments:
2023-09-06 07:22:30 'from=/root/s2c-builder/src/incron.allow|to=/etc/|chown=root:incron'
2023-09-06 07:22:30 * Plugin copyfile: Start Phase 0
2023-09-06 07:22:30 > Plugin copyfile: Keys/values found:
2023-09-06 07:22:30 from: /root/s2c-builder/src/incron.allow
2023-09-06 07:22:30 to: /etc/
2023-09-06 07:22:30 chown: root:incron
2023-09-06 07:22:30 > Run Plugin 'copyfile' (/mnt/sdm/usr/local/sdm/plugins/copyfile) Phase 0 with arguments:
2023-09-06 07:22:30 'from=/root/s2c-builder/src/test.txt|to=/root'
2023-09-06 07:22:30 * Plugin copyfile: Start Phase 0
2023-09-06 07:22:30 > Plugin copyfile: Keys/values found:
2023-09-06 07:22:30 from: /root/s2c-builder/src/test.txt
2023-09-06 07:22:30 to: /root
2023-09-06 07:22:30 > Run Plugin 'samba' (/mnt/sdm/usr/local/sdm/plugins/samba) Phase 0 with arguments:
2023-09-06 07:22:30 'smbconf=/root/s2c-builder/src/smb.conf'
2023-09-06 07:22:30 * Plugin samba: Start Phase 0
2023-09-06 07:22:30 > Plugin samba: Keys/values found:
2023-09-06 07:22:30 smbconf: /root/s2c-builder/src/smb.conf
2023-09-06 07:22:30 > Plugin samba: Copy samba smbconf '/root/s2c-builder/src/smb.conf' into IMG
2023-09-06 07:22:30 /etc/sdm/assets/samba-smb.conf
2023-09-06 07:22:30 * Plugin samba: Complete Phase 0
2023-09-06 07:22:30 * Phase 0 Completed
...
2023-09-06 07:26:39 > Run Plugins Phase 'post-install'
2023-09-06 07:26:39 > Run Plugin 'apps' (/usr/local/sdm/plugins/apps) Phase post-install with arguments:
2023-09-06 07:26:39 'apps=samba incron vim tree'
2023-09-06 07:26:39 * Plugin apps: Start Phase post-install
2023-09-06 07:26:39 * Plugin apps: Complete Phase post-install
2023-09-06 07:26:39 > Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase post-install with arguments:
2023-09-06 07:26:39 'from=/root/s2c-builder/src/incron.allow|to=/etc/|chown=root:incron'
2023-09-06 07:26:40 * Plugin copyfile: Start Phase post-install
2023-09-06 07:26:40 * Plugin copyfile: Complete Phase post-install
2023-09-06 07:26:40 > Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase post-install with arguments:
2023-09-06 07:26:40 'from=/root/s2c-builder/src/test.txt|to=/root'
2023-09-06 07:26:40 * Plugin copyfile: Start Phase post-install
2023-09-06 07:26:40 * Plugin copyfile: Complete Phase post-install
2023-09-06 07:26:40 > Run Plugin 'samba' (/usr/local/sdm/plugins/samba) Phase post-install with arguments:
2023-09-06 07:26:40 'smbconf=/root/s2c-builder/src/smb.conf'
2023-09-06 07:26:41 * Plugin samba: Start Phase post-install
2023-09-06 07:26:41
2023-09-06 07:26:41 > Plugin samba: *** samba service notes ***
2023-09-06 07:26:41 * Plugin samba: Ensure that /etc/samba/smb.conf is as you want it
2023-09-06 07:26:41 * Plugin samba: Add users and passwords using 'sudo smbpasswd'
2023-09-06 07:26:41 * Plugin samba: The service should be enabled when the system boots; review the system logs
2023-09-06 07:26:41 for possible errors
2023-09-06 07:26:41
2023-09-06 07:26:41 * Plugin samba: Complete Phase post-install
2023-09-06 07:26:41 %%%
2023-09-06 07:26:41 % IMG '/root/s2c-builder/images/2023-05-03-raspios-bullseye-armhf-lite.img' is 100% full (10308 1K-blocks free)
2023-09-06 07:26:41 % Review /etc/sdm/apt.log in the image for insufficient disk space errors
2023-09-06 07:26:41 % If needed use --extend --xmb nnnn with --customize to create more space
2023-09-06 07:26:42 %%%
2023-09-06 07:26:42 * Customization complete
2023-09-06 07:26:42 * Batch Mode exit
Curently installed Version
sdm V8.6
Hope this helps to sort it out.
I have just found this and have been browsing the documentation. In particular I'm interested in installing python packages. Is there a method for "pip3 install..."?
Hi there! sdm has been working great to setup many pi but would appreciate some help/direction with using custom phase scripts. I have had success adding python libraries to an image with a custom script but need some help adding a custom service and enabling it on an image.
I think the enable service call in phase 1 is working but the service file itself seems to be failing to be copied across to the systemd/system folder in the image so the enable call is failing to find the service to enable. When mounting and investigating the image I can find no trace of the file, I am sure I am missing a step or have misinterpreted whats required.
Currently I am trying to copy the service file in phase 0 with:
cp /usr/local/sdm/sdm-custom-asset/my.service /mnt/sdm/etc/systemd/system
and enabling the service in phase 1 with:
sudo systemctl enable my.service
Anymore help you could provide would be great as you were fantastically helpful with my last request!
I was attempting to us https://github.com/gitbls/sdm/blob/master/Docs/Captive-Portal.md, but I get the following errors
sdm: unrecognized option '--nowpa'
? Unrecognized plugin 'wifconfig'
Should this be working or is it part of the upcoming release?
Am I able to rename the default pi user to something else using sdm --customize ?
Hi we spoke on the raspi forums about using sdm for setting a static IP among other parameters for a raspberry pi image.
I am aiming to use an image setup with the needed libraries (pi-gen or creating an image from an SD install) to be able to burn to about 20 SD cards with each image having a shared wifi network/password with ssid broadcast off, and each having a different:
-unique hostname
-unique username/password
-static IP
This is with the aim of automating the setup needed to create a connected network for these 20 pis. Any help with using sdm for this goal would be very much appreciated!
Here is a the command I am running:
sudo sdm --customize --network:"netman=nm|wifissid=Netgear96|wifipassword=xxxx|wificountry=US|noipv6" --L10n --restart --disable piwiz --regen-ssh-host-keys --user trg --password-user xxxx --password-same y --autologin --xmb 2048 --plugin copyfile:"filelist=/home/alan/Documents/raspios/filestore/trg/filelist.txt" 2023-05-03-raspios-bullseye-armhf-lite.img
two things are happening
I am pretty sure this has got to be something I am misunderstanding about the sdm so I added the command above in hopes you can educate me.
The simple answer is no.
If I (or someone else) wants to port sdm here's a list of the packages sdm relies on:
This package provides systemd's tools for nspawn and container/VM management:
- systemd-nspawn
- systemd-machined and machinectl
- systemd-importd
- systemd-portabled and portablectl
QEMU is a fast processor emulator: currently the package supports ARM, CRIS, i386, M68k (ColdFire), MicroBlaze, MIPS, PowerPC, SH4, SPARC and x86-64 emulation. By using dynamic translation it achieves reasonable speed while being easy to port on new host CPUs.
This package provides the user mode emulation binaries, built statically. In this mode QEMU can launch Linux processes compiled for one CPU on another CPU.
qemu-user-static package will register binary formats which the provided emulators can handle, so that it will be possible to run foreign binaries directly.
Support for extra binary formats
The binfmt_misc kernel module, contained in versions 2.1.43 and later of the Linux kernel, allows system administrators to register interpreters for various binary formats based on a magic number or their file extension, and cause the appropriate interpreter to be invoked whenever a matching file is executed. Think of it as a more flexible version of the #! executable interpreter mechanism.This package provides an 'update-binfmts' script with which package maintainers can register interpreters to be used with this module without having to worry about writing their own init.d scripts, and which sysadmins can use for a slightly higher-level interface to this module.
disk partition manipulator
GNU Parted is a program that allows you to create, destroy, resize, move, and copy disk partitions. This is useful for creating space for new operating systems, reorganizing disk usage, and copying data to new hard disks.This package contains the binary and manual page. Further documentation is available in parted-doc.
Parted currently supports DOS, Mac, Sun, BSD, GPT, MIPS, and PC98 partitioning formats, as well as a "loop" (raw disk) type which allows use on RAID/LVM. It can detect and remove ASFS/AFFS/APFS, Btrfs, ext2/3/4, FAT16/32, HFS, JFS, linux-swap, UFS, XFS, and ZFS file systems. Parted also has the ability to create and modify file systems of some of these types, but using it to perform file system operations is now deprecated.
I've run the commands from the quick start example to produce a bootable SD card. When I boot from the card, I am dropped into a 'set your keyboard' dialog. The documentation says that this should not happen. Am I doing something incorrectly?
Attached is a script (run as root) to reproduce the problem.
error.txt
I had been working on a series of scripts to help reproduce my many custom configurations when RaspiOS bookworm hits, but I just ran across sdm and thought I shouldn't be re-inventing any wheels, and yours might be a LOT easier ;-) So far it's all making sense to me and I think I'll be able to do everything I want, though perhaps with some custom scripting. This here is my first trial run on bullseye - an attempt to do everything I can just with the command line - and I'm having some trouble with apt-cacher-ng.
Here's my command line:
sudo sdm --extend --xmb 512 --customize --wpa /etc/wpa_supplicant/wpa_supplicant.conf --L10n --restart --disable piwiz --regen-ssh-host-keys --user pi --password-pi mypassword --ssh service --poptions apps,xapps --apps "figlet hdparm mc nmap screen stress" --xapps "passwordsafe" --aptcache 192.168.0.118 --apt-dist-upgrade --disable bluetooth --swap 1024 /mnt/backup/bullseye64.img
NOTE: sdm is running on "pi8", a pi4-8GB with a 240GB SSD, running 64-bit bullseye. I'm already a fan of apt-cacher-ng, so this machine was ALREADY set up as my apt-cacher-ng server. On this and all of the other client pi's, I add the following line to 02proxy:
Acquire::http { Proxy "http://pi8.local:3142"; }
And have had no trouble at all with pi8 or any of my other pi's.
Here, I am (I hope correctly) requesting that sdm set up this image as an apt-cacher-ng client, and it seems to be trying to use my CURRENT cache during its own update and install processes??? I think so because no matter how I code the --aptcache line (with pi8.local, or 192.168.0.118), the error messages reference pi8.local (see below).
l also noted while nosing around within the sdm shell that you are constructing the apt proxy line differently than I am (see yours at the very end of the pasted text below:
Here's part of the sdm output and what I did in the shell:
`2023-08-11 14:18:15 apt update
W: Failed to fetch http://deb.debian.org/debian/dists/bullseye/InRelease Could not resolve 'pi8.local'
W: Failed to fetch http://security.debian.org/debian-security/dists/bullseye-security/InRelease Could not resolve 'pi8.local'
W: Failed to fetch http://deb.debian.org/debian/dists/bullseye-updates/InRelease Could not resolve 'pi8.local'
W: Failed to fetch http://archive.raspberrypi.org/debian/dists/bullseye/InRelease Could not resolve 'pi8.local'
W: Some index files failed to download. They have been ignored, or old ones used instead.
2023-08-11 14:18:23 apt install --no-install-recommends --yes figlet hdparm mc nmap screen stress
E: Failed to fetch http://deb.debian.org/debian/pool/main/f/figlet/figlet_2.2.5-3_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/h/hdparm/hdparm_9.60%2bds-1_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/libl/liblinear/liblinear4_2.3.0%2bdfsg-5_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/l/lua5.3/liblua5.3-0_5.3.3-1.1%2bb1_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/libu/libutempter/libutempter0_1.2.1-2_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/l/lua-lpeg/lua-lpeg_1.0.2-1_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/m/mc/mc-data_4.8.26-1.1_all.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/m/mc/mc_4.8.26-1.1_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/n/nmap/nmap-common_7.91%2bdfsg1%2breally7.80%2bdfsg1-2_all.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/n/nmap/nmap_7.91%2bdfsg1%2breally7.80%2bdfsg1-2_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/s/screen/screen_4.8.0-6_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/s/stress/stress_1.0.4-7_arm64.deb Could not resolve 'pi8.local'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2023-08-11 14:18:26 apt install --no-install-recommends --yes passwordsafe
E: Failed to fetch http://deb.debian.org/debian/pool/main/q/qrencode/libqrencode4_4.1.1-1_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/w/wxwidgets3.0/libwxbase3.0-0v5_3.0.5.1%2bdfsg-2_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/w/wxwidgets3.0/libwxgtk3.0-gtk3-0v5_3.0.5.1%2bdfsg-2_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/x/xerces-c/libxerces-c3.2_3.2.3%2bdebian-3_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/liby/libyubikey/libyubikey0_1.13-6_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/y/yubikey-personalization/libykpers-1-1_1.20.0-3_arm64.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/passwordsafe/passwordsafe-common_1.12.0%2bdfsg-1_all.deb Could not resolve 'pi8.local'
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/passwordsafe/passwordsafe_1.12.0%2bdfsg-1_arm64.deb Could not resolve 'pi8.local'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2023-08-11 14:18:28 apt -y dist-upgrade
2023-08-11 14:18:30 apt --yes autoremove
root@sdm:/etc# cd apt
root@sdm:/etc/apt# cd apt.conf.d
root@sdm:/etc/apt/apt.conf.d# ls
01autoremove 01autoremove-kernels 02proxy 20listchanges 20packagekit 50raspi 70debconf
root@sdm:/etc/apt/apt.conf.d# cat 02*
Acquire::http::proxy "http://pi8.local:3142";
`
On my Windows 11 machine, sdm V8.6 does not work anymore in wsl.
sudo sdm --customize --locale de_DE.UTF-8 2023-05-03-raspios-bullseye-armhf-lite.img
* Mount IMG '2023-05-03-raspios-bullseye-armhf-lite.img'
mount: /dev/loop0p2 ist auf /mnt/sdm eingehängt.
mount: /dev/loop0p1 ist auf /mnt/sdm/boot eingehängt.
chmod: Zugriff auf '/mnt/sdm//usr/local/sdm/local-plugins/*' nicht möglich: Datei oder Verzeichnis nicht gefunden
* Start Configuration
> Command Line: /usr/local/bin/sdm --customize --locale de_DE.UTF-8
2023-05-03-raspios-bullseye-armhf-lite.img
* Host Information
Hostname: xxx
uname: Linux xxx 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC
2021 x86_64 x86_64 x86_64 GNU/Linux
os-release Name: "Ubuntu"
os-release Version: "22.04.1 LTS (Jammy Jellyfish)"
sdm Version: V8.6
* IMG Information
Date: 2023-05-03
RasPiOS Version: 11
os-release Version: "11 (bullseye)"
% No username specified with the --user switch; Using 'pi'
% sdm will try to use systemd-nspawn on this 'x86-64' system
If this does not work please post a GitHub issue
> IMG '2023-05-03-raspios-bullseye-armhf-lite.img' has 294328 1K-blocks free
> Copy sdm to /usr/local/sdm in the IMG
* Start Phase 0 image customization
*+Phase 0 sdm configuration
*+Phase 0 Network configuration
*+Phase 0 User configuration
*+Phase 0 System configuration
*+Phase 0 Raspberry Pi-specific configuration
* Phase 0 Completed
* Enter image '2023-05-03-raspios-bullseye-armhf-lite.img' for Phase 1
Failed to open system bus: No such file or directory
umount: /mnt/sdm/boot ausgehängt
umount: /mnt/sdm ausgehängt
V7.17 does work in wsl. I installed V7.17 by downloading EZsdmInstaller
, changing master
in src to 48ca10bbbfad0dfa0774962ad13da5e96d8f5c20
and executing it.
Hi, I'm trying to make this run on arch linux and I get an error when there are calls to systemd --version
. Changing them to systemctl --version
seems to do the job. When trying on debian it seems that both commands output the same info, so I'm wondering if there is any issue to use systemctl instead of systemd command ?
Hi,
Loving this project so far!
Would it be possible to add a feature to populate .ssh/authorized_keys
with entries as part of the customize / burn process?
Maybe like specifying the path to a local authorized_keys
file which is copied into the image, similar to how wpa_supplicant.conf
is done
First of all: Thank you for this very nice piece of software! The possibilities and documentation are awesome.
I would like to run sdm in a GitHub Action, to have a customized pi image build automatically. This works already fine.
sdm is installed with the EZsdmInstaller
script, which downloads always the most recent commit in the master branch:
Line 60 in 210897b
This is a problem for me, because breaking changes like in sdm V8.x or new introduced bugs can break my build. This would not have to happen, as the build would continue to work fine using the old sdm version. To have a reliable auto build, I need to have the control when to update sdm, which means I need to be able to install a specific version of sdm. Another use case is testing something with a different version of sdm, for example to check for bugs.
Without any changes to sdm, I have this workaround:
EZsdmInstaller
at a specific commitmaster
in the download link with the id of the specific commit using for example sedSo the feature request is a clean way for installing a specific version. There are multiple ways to achieve this. For example, the installer script could have a parameter to specify a commit id replacing master
. Adding tags to the release commits would even allow to specify a version instead of a commit id.
Please see this feature request just as something to start thinking, I don't see a high priority as I listed an easy workaround.
Thank you 😃
Title says it all. See https://forums.raspberrypi.com/viewtopic.php?t=325703 for details.
It doesn't appear to cause data corruption, but it is disconcerting to see in the system log!
Since v9 I'm now prompted to enter keyboard layout when the pi first boots, despite customizing with:
`--plugin L10n:"keymap=us|locale=en_US.UTF-8|timezone=America/New_York"
I'm probably missing something, but can't find any other keyboard options in the docs. Any help would be appreciated!
I ran this customize command:
sudo sdm --customize --wpa wpa_supplicant.conf --disable piwiz --L10n --rename-pi mm --password-pi secret --hotspot hotspot.cfg --ssh service --regen-ssh-host-keys --host mmdwatrous --apt-dist-upgrade 2023-05-03-raspios-bullseye-armhf-lite.img
With this in my hotspot.cfg file:
config="routed"
But I see this in my logs
* Start Access Point (hotspot) installation and configuration ' Unrecognized Access Point configuration type 'routed"
This is being executed here: https://github.com/gitbls/sdm/blob/56017520b915f7d6ee32db3b38e957a617f5d352/sdm-hotspot#L138C3-L138C3
Based on the error message, the variable "hotconfig" is properly set to 'routed', but the conditional is still false for some reason.
I'm running WSL2 Debian:
systemctl --version systemd 252 (252.12-1~deb12u1) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
And I know it doesn't list --fstab as something that occurs in --burn, but I sure wish it did - I have specific, not general, fstab lines that I would like to do only when burning specific pi's.
When customizing an image using the command below I get a warning about not supported locales:
$ sudo sdm --customize \
--os raspios \
--l10n \
--locale en_US.UTF-8 \
--disable bluetooth,piwiz,triggerhappy \
--user pi \
--password-same y \
--password-pi "$user_password" \
--password-root "$user_password" \
--rootpwd \
--host raspibox \
--nowpa \
--loadlocal wifi \
--apssid raspibox \
--poptions apps \
--apps @raspibox-apps \
--dtoverlay hifiberry-dac \
--ssh service \
--plugin samba:"" \
--xmb 4048 \
--showapt \
2022-09-22-raspios-bullseye-raspibox-arm64.img
(...)
Reloading keymap. This may take a short while
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = "en_US.UTF-8",
LC_ALL = (unset),
LC_PAPER = "de_CH.UTF-8",
LC_NUMERIC = "de_CH.UTF-8",
LC_IDENTIFICATION = "de_CH.UTF-8",
LC_MEASUREMENT = "de_CH.UTF-8",
LC_NAME = "de_CH.UTF-8",
LC_TELEPHONE = "de_CH.UTF-8",
LC_ADDRESS = "de_CH.UTF-8",
LC_MONETARY = "de_CH.UTF-8",
LC_TIME = "de_CH.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
(...)
The same warning is repeated several times.
I'm running this on Ubuntu 22.04 LTS (64 bit) to customize the latests RaspianOS image found at https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64.img.xz.
--bootset
has the option camera
, but this does not enable the camera in a current image, probably because they changed the way this is done in config.txt
.
In 2023-05-03-raspios-bullseye-armhf-lite
, there is now the option camera_auto_detect=0
, which needs to be changed to 1
.
The docs https://www.raspberrypi.com/documentation/computers/config_txt.html#camera_auto_detect say that this option is on by default, but at least for the mentioned lite image this is not the case, it is set to 0.
Of course there are several possible workarounds for sdm like using a first boot script, but I had to debug why my camera wasn't working at first although I specified --bootset camera=1
.
When attempting to pass any 'value' with a colon as a plugin argument, the sdm argument parser wrongly splits
So, with an sdm-switch like this
--runonly plugins --plugin sms-imprint=plugin:"BASE_URL=https\://somwhere.com"
the parses uses the second : as the separator between plugin name and arguments and subsequently fails to find sms-imprint
because it is looking for a file called sms-imprint=plugin:"BASE_URL=https
. The \:
is escaped because for parameter parsing, IFS="|:" read ...
is used and this function requires escaping.
The culprit seems to be operation p="${p%:*}"
in sdm
and sdm-cparse
splitting the arguments off at the last :
in the string. It looks like using p="${p%%:*}"
instead solves the problem at the 'cost' that :
cannot appear in the module path or name any more - but I don't see why this would be an issue.
Better still would be to use `IFS=':' read -a plug <<< "$p" because it would allow consistent escaping of the special separation character.
Next issue is in smd-cparse.runplugins()' where the code
IFS="~" read -a plugs <<< "$plugins"apparently expands the plugin-string and _strips_
https:to
https:. Double escaping can solve the issue here (
https\:->
https:`) but that then fails then in other places.
For now, my work-around will be to use two parameters: one for the protocol and one for the rest of the url after the :
.
Perhaps some bash magician out there could help to solve this in a more convenient way? Thanks a lot in advance!
Hello, I'd first like to say I'm really enjoying SDM and have been using it since ~April to further update some of my homelab Ansible automation, mainly with a Debian Bookworm image from https://raspi.debian.net/. Unfortunately, with updates from 8.6 to a 9.x branch, a few things that were previously functional have broken.
Before 9.x, it looks like /etc/resolv.conf was taken by default from the host. Now it is not populated, resulting in no updates / app installation possible, unless you copy it in in a phase0 plugin.
* Mount IMG '20230612_raspi_3_bookworm.img' mount: /dev/loop0p2 mounted on /mnt/sdm. mount: /dev/loop0p1 mounted on /mnt/sdm/boot. chmod: cannot access '/mnt/sdm//usr/local/sdm/local-plugins/*': No such file or directory * Start Configuration > Command Line: /usr/local/bin/sdm --customize --apt-dist-upgrade --nopassword 20230612_raspi_3_bookworm.img * Host Information Hostname: overseer uname: Linux overseer 6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux os-release Name: "Debian GNU/Linux" os-release Version: "12 (bookworm)" sdm Version: V8.6 * IMG Information Date: RasPiOS Version: 12 os-release Version: "12 (bookworm)" % No username specified with the --user switch; Using 'pi' % sdm will try to use systemd-nspawn on this 'x86-64' system If this does not work please post a GitHub issue > IMG '20230612_raspi_3_bookworm.img' has 1074480 1K-blocks free > Copy sdm to /usr/local/sdm in the IMG * Start Phase 0 image customization *+Phase 0 sdm configuration *+Phase 0 Network configuration *+Phase 0 User configuration *+Phase 0 System configuration *+Phase 0 Raspberry Pi-specific configuration > Create user 'pi' home directory /home/pi * Phase 0 Completed * Enter image '20230612_raspi_3_bookworm.img' for Phase 1 * Start Phase 1 image customization > IMG '20230612_raspi_3_bookworm.img' has 1074460 1K-blocks free at start of Phase 1 image customization *+Phase 1 sdm configuration > Configure and enable sdm FirstBoot service (sdm-firstboot) *+Phase 1 Network/WiFi/Bluetooth configuration > Enable SSH service via /boot/ssh *+Phase 1 User configuration > Add user 'pi' usermod: group 'spi' does not exist usermod: group 'i2c' does not exist usermod: group 'gpio' does not exist > Set /home/pi files ownership to 'pi:pi' > Add user 'pi' to sudoers.d /usr/local/sdm/sdm-phase1: line 244: /etc/sudoers.d/010_pi-nopasswd: No such file or directory chmod: cannot access '/etc/sudoers.d/010_pi-nopasswd': No such file or directory > Skip password processing per --nopassword *+Phase 1 System and Boot Configuration > System journal configuration not modified > Start 'apt update' > IMG '20230612_raspi_3_bookworm.img' has 1074436 1K-blocks free at start of 'apt update' > IMG '20230612_raspi_3_bookworm.img' has 916144 1K-blocks free at end of 'apt update' * Phase 1 post-app installation/configuration > Start 'apt dist-upgrade' > IMG '20230612_raspi_3_bookworm.img' has 916144 1K-blocks free at start of 'apt dist-upgrade' > IMG '20230612_raspi_3_bookworm.img' has 510040 1K-blocks free at end of 'apt dist-upgrade' > Start 'apt autoremove' > IMG '20230612_raspi_3_bookworm.img' has 510040 1K-blocks free at start of 'apt autoremove' > IMG '20230612_raspi_3_bookworm.img' has 510040 1K-blocks free at end of 'apt autoremove' > IMG '20230612_raspi_3_bookworm.img' has 510040 1K-blocks free at end of Phase 1 image customization * Phase 1 Completed > IMG '20230612_raspi_3_bookworm.img' has 510040 1K-blocks free at end of image customization * Enter Shell Command Prompt > 'exit' to exit back to host system root@sdm:/# cat /etc/resolv.conf # Generated by NetworkManager search lan nameserver 1.1.1.1 nameserver 10.2.16.1 root@sdm:/# exit * Customization complete umount: /mnt/sdm/boot unmounted umount: /mnt/sdm unmounted
* Mount IMG '20230612_raspi_3_bookworm.img' mount: /dev/loop0p2 mounted on /mnt/sdm. mount: /dev/loop0p1 mounted on /mnt/sdm/boot. * Start Configuration > Command Line: /usr/local/bin/sdm --customize --apt-dist-upgrade 20230612_raspi_3_bookworm.img * Host Information Hostname: overseer uname: Linux overseer 6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux os-release Name: "Debian GNU/Linux" os-release Version: "12 (bookworm)" sdm Version: V9.3 * IMG Information Name: 20230612_raspi_3_bookworm.img Date: RasPiOS Version: 12 os-release Version: "12 (bookworm)" % Configure chroot for sdm on your 'x86-64' computer * sdm will run more slowly due to qemu emulation % No plugins are selected > IMG '20230612_raspi_3_bookworm.img' has 1074524 1K-blocks (1.1GB, 1.0GiB) free > Copy sdm to /usr/local/sdm in the IMG * Start Phase 0 image customization > No Plugins specified with --plugin * Phase 0 Completed * Enter image '20230612_raspi_3_bookworm.img' for Phase 1 * Start Phase 1 image customization //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1064300 1K-blocks (GB, GiB) free at start of Phase 1 image customization > Configure and enable sdm FirstBoot service (sdm-firstboot) > Start 'apt update' //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1064296 1K-blocks (GB, GiB) free at start of 'apt update' //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1060912 1K-blocks (GB, GiB) free at end of 'apt update' > No Plugins specified with --plugin * Phase 1 post-app installation/configuration > Start 'apt dist-upgrade' //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1060912 1K-blocks (GB, GiB) free at start of 'apt dist-upgrade' //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1060912 1K-blocks (GB, GiB) free at end of 'apt dist-upgrade' > Start 'apt autoremove' //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1060912 1K-blocks (GB, GiB) free at start of 'apt autoremove' //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1060912 1K-blocks (GB, GiB) free at end of 'apt autoremove' //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1060912 1K-blocks (GB, GiB) free at end of Phase 1 image customization * Phase 1 Completed > No Plugins specified with --plugin > Run graphics post-install configuration > Enable SSH > Plugin phase1: Enable SSH service via /boot/ssh //usr/local/sdm/sdm-cparse: line 195: python3: command not found //usr/local/sdm/sdm-cparse: line 196: python3: command not found > IMG '20230612_raspi_3_bookworm.img' has 1060912 1K-blocks (GB, GiB) free at end of image customization * Enter Shell Command Prompt 'exit' to exit back to host system root@overseer:/# cat /etc/resolv.conf cat: /etc/resolv.conf: No such file or directory root@overseer:/#
Hi. I have been using sdm with raspbian for a while now, and I am interested in using it to customize an ubuntu image. I see here and here that it is not supported. Do you have any pointers about what would need to be changed with sdm in order to support this or if other tools exist for this space? My team would potentially be interested in contributing to expand sdm for this use case
Hi, Benn -- I hope this finds you well
I have been tinkering with sdm-cportal
and have made a few adjustements, namely I have edited the links and HTTPServer to use port 9090 instead of 80 (since I already have a web-server bound to port 80, 8080, and 8888).
With those adjustments, most of the captive portal works.
I am able to "Start the captive portal" and fill in the form successfully.
'testinputs' returns success.
However, when I attempt the next step, I find that sdm-cportal
depends on an sdm
configuration file, /etc/sdm/local-1piboot.conf
I wanted to bring this to your attention since I know it was a goal of sdm-cportal
to be usable outside of sdm
, though it seems there is a small dependency issue.
Looking at that file, however, everything is commented out, so I'm going to see what happens if I just touch
that file and try again.
Below is the Python traceback for the issue
----------------------------------------
Exception occurred during processing of request from ('10.1.1.246', 60992)
Traceback (most recent call last):
File "/usr/lib/python3.9/socketserver.py", line 316, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python3.9/socketserver.py", line 347, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/home/pi/./sdm/sdm-cportal", line 231, in __init__
super().__init__(*args, **kwargs)
File "/usr/lib/python3.9/socketserver.py", line 720, in __init__
self.handle()
File "/usr/lib/python3.9/http/server.py", line 427, in handle
self.handle_one_request()
File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
method()
File "/home/pi/./sdm/sdm-cportal", line 364, in do_GET
self._setstop(pd, False)
File "/home/pi/./sdm/sdm-cportal", line 268, in _setstop
writel10n(pd, self.pd.keymap, self.pd.locale, self.pd.timezone)
File "/home/pi/./sdm/sdm-cportal", line 225, in writel10n
with open("/etc/{}/local-1piboot.conf".format(pd.facname), 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/sdm/local-1piboot.conf'
----------------------------------------
Hi there,
Firstly, this application looks awesome, and is extremely well documented.
I've spent some time reading through the docs (and watched the youtube demo), and I've got a couple questions before I try this myself.
1. How "easily" are you able to do version control of configuration for an image?
One thing I'm a bit unclear on is how someone may be able to make a custom image based on RaspiOS and store all the things necessary minus the image itself in git?
I see there's examples where you go directly into the image using functionality such as --explore
, however I imagine any changes here are specific to said image?
I guess the --explore
is more of an option, rather than the main way to do config...
Use case, or what I want to do:
Basically I've got a bunch of custom stuff, or stuff I'd like to be quite obvious in a repo for someone else trying to use SDM to build a customer image. I get the impression I can do most of the above, however the impression I get is that most stuff is through command switches (of which they are extensive - nice!), however I guess as someone who in general understands where stuff goes in linux, can I simply do something like:
mv repo-folder/root/etc/* sdm/root/etc/*
I realize with that statement I'm horribly over-simplifying things, but I guess I'm trying to understand whether SDM can be configured to install all the relevant applications, and then I copy existing config files, or append to numerous configs through the OS?
Or am I not thinking far ahead enough?
For reference I've spent some time trying to evaluate different options, whether that be https://github.com/RPi-Distro/pi-gen or simply modify the Pi's firstboot.sh
script to point to a bunch of other scripts and copy necessary files, and install applications.
2. How easily can I script building of an image?
Perhaps this is the tl;dr question of all of the above stated in question 1.
3. How well is the WSL support currently working?
I imagine the answer will be "try it".
I guess a semi related question - Does SDM work in Docker? Again this links back to my scriptibility question,
Perhaps for CICD purposes?
Thanks in advance,
Matt
$ sudo sdm --password-pi --host frambuesa --expand-root --burn /dev/mmcblk0 2022-09-22-raspios-bullseye-arm64.img
* Burn '2022-09-22-raspios-bullseye-arm64.img' to '/dev/mmcblk0'...
dd if=2022-09-22-raspios-bullseye-arm64.img of=/dev/mmcblk0 status=progress bs=16M iflag=direct
4265607168 bytes (4.3 GB, 4.0 GiB) copied, 286 s, 14.9 MB/s
254+1 records in
254+1 records out
4265607168 bytes (4.3 GB, 4.0 GiB) copied, 363.394 s, 11.7 MB/s
cat: /sys/block/mmcblk0/mmcblk02/size: No such file or directory
> Expand Root: Expand partition from 0.0GB (0.0GiB) to 55.5GB (59.6GiB)
Checking that no-one is using this disk right now ... OK
Disk /dev/mmcblk0: 59.49 GiB, 63864569856 bytes, 124735488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x63ee4f38
Old situation:
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 532479 524288 256M c W95 FAT32 (LBA)
/dev/mmcblk0p2 532480 8331263 7798784 3.7G 83 Linux
/dev/mmcblk0p2:
New situation:
Disklabel type: dos
Disk identifier: 0x63ee4f38
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 532479 524288 256M c W95 FAT32 (LBA)
/dev/mmcblk0p2 532480 124735487 124203008 59.2G 83 Linux
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).
Syncing disks.
> Set new disk ID '828d21d2' on '/dev/mmcblk0'
sfdisk: unrecognized option '--disk-id'
Try 'sfdisk --help' for more information.
? sfdisk error setting diskid 1
This error originates in sfdisk
- apparently, some versions of it do not admit the '--disk-id' option.
$ sfdisk --help
Usage:
sfdisk [options] <dev> [[-N] <part>]
sfdisk [options] <command>
Display or manipulate a disk partition table.
Commands:
-A, --activate <dev> [<part> ...] list or set bootable (P)MBR partitions
-d, --dump <dev> dump partition table (usable for later input)
-J, --json <dev> dump partition table in JSON format
-g, --show-geometry [<dev> ...] list geometry of all or specified devices
-l, --list [<dev> ...] list partitions of each device
-F, --list-free [<dev> ...] list unpartitioned free areas of each device
-r, --reorder <dev> fix partitions order (by start offset)
-s, --show-size [<dev> ...] list sizes of all or specified devices
-T, --list-types print the recognized types (see -X)
-V, --verify [<dev> ...] test whether partitions seem correct
--delete <dev> [<part> ...] delete all or specified partitions
--part-label <dev> <part> [<str>] print or change partition label
--part-type <dev> <part> [<type>] print or change partition type
--part-uuid <dev> <part> [<uuid>] print or change partition uuid
--part-attrs <dev> <part> [<str>] print or change partition attributes
<dev> device (usually disk) path
<part> partition number
<type> partition type, GUID for GPT, hex for MBR
Options:
-a, --append append partitions to existing partition table
-b, --backup backup partition table sectors (see -O)
--bytes print SIZE in bytes rather than in human readable format
--move-data[=<typescript>] move partition data after relocation (requires -N)
-f, --force disable all consistency checking
--color[=<when>] colorize output (auto, always or never)
colors are enabled by default
-N, --partno <num> specify partition number
-n, --no-act do everything except write to device
--no-reread do not check whether the device is in use
--no-tell-kernel do not tell kernel about changes
-O, --backup-file <path> override default backup file name
-o, --output <list> output columns
-q, --quiet suppress extra info messages
-w, --wipe <mode> wipe signatures (auto, always or never)
-W, --wipe-partitions <mode> wipe signatures from new partitions (auto, always or never)
-X, --label <name> specify label type (dos, gpt, ...)
-Y, --label-nested <name> specify nested label type (dos, bsd)
-G, --show-pt-geometry deprecated, alias to --show-geometry
-L, --Linux deprecated, only for backward compatibility
-u, --unit S deprecated, only sector unit is supported
-h, --help display this help
-v, --version display version
Available output columns:
gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S Start-C/H/S
bsd: Slice Start End Sectors Cylinders Size Type Bsize Cpg Fsize
sgi: Device Start End Sectors Cylinders Size Type Id Attrs
sun: Device Start End Sectors Cylinders Size Type Id Flags
For more details see sfdisk(8).
$ sfdisk --version
sfdisk from util-linux 2.34
hi @Benn,
Great project mate. Lots of work has gone into this and it works really well.
I was tinkering with sdm to generate standardised images for different types of projects, so rather than have a custom image for each project type, I am configuring defaults for everything in standardised images, and then more specific customisations while burning.
I came across an error when specifying the --expand-root
option to --burn
. The full command was:
sudo sdm --burn /dev/mmcblk2 \
~/Downloads/2023-05-03-raspios-bullseye-arm64.img \
--expand-root \
--restart \
--bootset serial=0,boot_wait=1,camera=0,i2c=0,onewire=1,blanking=0,overscan=0,boot_behaviour=B4 \
--regen-ssh-host-keys \
--hostname xxx
The error was:
cat: /sys/block/mmcblk2/mmcblk22/size: No such file or directory
The path should be: /sys/block/mmcblk2/mmcblk2p2/size
to denote the second partition on the mmcblk2 device. Full log output below for context. The partition table for the root partition does get expanded fully. I checked using fdisk, and the last sector matches the size of the 16G card. But the ext4 filesystem doesn't get expanded to fill the partition, and retains its original size from the image file.
Just noticed another error too where the need for firstboot is confused. See following full log output.
* Burn '/home/clarkd/Downloads/2023-05-03-raspios-bullseye-arm64.img' to '/dev/mmcblk2'...
dd if=/home/clarkd/Downloads/2023-05-03-raspios-bullseye-arm64.img of=/dev/mmcblk2 status=progress bs=16M iflag=direct
4622123008 bytes (4.6 GB, 4.3 GiB) copied, 250 s, 18.5 MB/s
275+1 records in
275+1 records out
4622123008 bytes (4.6 GB, 4.3 GiB) copied, 348.221 s, 13.3 MB/s
cat: /sys/block/mmcblk2/mmcblk22/size: No such file or directory
> Expand Root: Expand partition from 0.0GB (0.0GiB) to 10.5GB (11.3GiB)
> Set new disk ID '???' on '/dev/mmcblk2'
Disk identifier changed from 0x??? to 0x???.
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
> Disk ID ??? set successfully with sfdisk
mount: /dev/mmcblk2p2 mounted on /mnt/sdm.
mount: /dev/mmcblk2p1 mounted on /mnt/sdm/boot.
> Set new Disk ID '???' in /etc/fstab and /boot/cmdline.txt
> Flush accumulated burn log messages
% sdm will try to use systemd-nspawn on this 'x86-64' system
> Disable unneeded RasPiOS firstboot service in cmdline.txt; both --expand-root and
--regen-ssh-host-keys set
> Disable regenerate_ssh_host_keys service; First Boot will run it instead
Failed to disable unit: Unit file regenerate_ssh_host_keys.service does not exist.
The --customize
command used to generate the image follows.
sudo sdm --customize \
~/Downloads/2023-05-03-raspios-bullseye-arm64.img \
--l10n \
--bootset serial=0,boot_wait=1,camera=0,i2c=0,onewire=1,blanking=0,overscan=0,boot_behaviour=B4 \
--user clarkd \
--disable piwiz \
--showapt \
--ssh service \
--extend --xmb 200
When I first boot, the following error prints out three times:
"[FAILED] Failed to start Access point and authentication server for Wi-Fi and Et"
I am trying to get the hotspot and WiFi to work at the same time. I previously accomplished this using this process: https://forums.raspberrypi.com/viewtopic.php?t=211542#p1355569 (https://github.com/idev1/rpihotspot). Some of the approach in sdm
is similar, but I'm not sure exactly what's missing. One difference is that sdm
references eth0 instead of wlan0
. I'm not sure if that means it will only work if I connect to a wired network connection.
Some more debugging data:
root@mmdwatrous:/etc/sdm# ./0piboot/040-hostapd-enable.sh
Synchronizing state of hostapd.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable hostapd
Synchronizing state of dnsmasq.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable dnsmasq
This doesn't bring up uap0
root@mmdwatrous:/etc/sdm# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 28 bytes 3318 (3.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 28 bytes 3318 (3.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.68.131 netmask 255.255.255.0 broadcast 192.168.68.255
inet6 fe80::4246:82d6:7319:dc0b prefixlen 64 scopeid 0x20<link>
ether b8:27:eb:ca:c3:8a txqueuelen 1000 (Ethernet)
RX packets 7909 bytes 1358009 (1.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 721 bytes 131783 (128.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@mmdwatrous:/etc/sdm# systemctl status hostapd
● hostapd.service - Access point and authentication server for Wi-Fi and Ethernet
Loaded: loaded (/lib/systemd/system/hostapd.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sat 2023-09-16 16:35:22 CDT; 2s ago
Docs: man:hostapd(8)
Process: 4197 ExecStart=/usr/sbin/hostapd -B -P /run/hostapd.pid -B $DAEMON_OPTS ${DAEMON_CONF} (code=exited, statu>
CPU: 45ms
root@mmdwatrous:/etc/sdm# systemctl status dnsmasq
● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2023-09-16 15:29:12 CDT; 1h 6min ago
Main PID: 397 (dnsmasq)
Tasks: 1 (limit: 414)
CPU: 887ms
CGroup: /system.slice/dnsmasq.service
└─397 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -r /run/dnsmasq/resolv.conf -7 /etc/dnsmasq>
Sep 16 15:29:09 mmdwatrous dnsmasq[397]: read /etc/hosts - 5 addresses
Sep 16 15:29:11 mmdwatrous dnsmasq[397]: no servers found in /run/dnsmasq/resolv.conf, will retry
Sep 16 15:29:12 mmdwatrous systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.
Sep 16 15:29:26 mmdwatrous dnsmasq[397]: reading /run/dnsmasq/resolv.conf
Sep 16 15:29:26 mmdwatrous dnsmasq[397]: using nameserver 75.75.75.75#53
Sep 16 15:29:26 mmdwatrous dnsmasq[397]: using nameserver 75.75.76.76#53
Sep 16 16:14:27 mmdwatrous dnsmasq[397]: no servers found in /run/dnsmasq/resolv.conf, will retry
Sep 16 16:14:34 mmdwatrous dnsmasq[397]: reading /run/dnsmasq/resolv.conf
Sep 16 16:14:34 mmdwatrous dnsmasq[397]: using nameserver 75.75.75.75#53
Sep 16 16:14:34 mmdwatrous dnsmasq[397]: using nameserver 75.75.76.76#53
I know you're busy with the current release. When you get around to working on hotspot, I'm ready to test. If I make progress before then, I'll report back on this issue.
Hey! I saw that the installation script uses apt sadly which isn't available on arch (The host pc) so my question is do you have a port for arch Linux? or do we need to modify the install script ourselves?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.