Disclaimer: I am not an expert at linux, but I think this might be interesting to let

Error mounting after successful burn rpi5 NVMe,about gitbls/sdm

Comments (32)

gitbls commented on September 7, 2024 1

But anyway now it is all working as expected. Any suggestion how I can set another partition on that disk using the burn plugin that expands to fill the whole free space? It's not fully clear from the docs to me, only how to expand root. I have about 630GB left that I want to configure to the last byte, but would like to be able to do it without calculating all clusters or guessing etc.

Glad that you got the cparams file created. Not sure what's going on there.

Re creating a "last partition" that fills the disk: There's no way to do that at the moment, but there should be. I've got the parted plugin open to add GPT disk support, so will look into adding this "makes perfect sense" feature.

from sdm.

gitbls commented on September 7, 2024 1

Hi, me again. xD

It was good I read this because I was thinking of forking and sending 1 or 2 requests. There are things I could provide knowledge wise that could speed up the "burn" process by hundreds of % if you are interested? (rsync instead of dd, like I do in my script)

I also saw you are doing some unnecessary operations that I could help out with if you'd like. (partition resizing and stuff)

But you should def make the changes you have in mind first because I might need to move a few things around to do what I have in mind.

Please don't hijack other threads...please start a new Issue (or Discussion if you'd prefer) where we can chat about this.

I'm interested in your code to understand the code complexity, tradeoffs, possible issues, etc. as well as proven performance data. I'd much prefer a script that I can download/use on the side than something you've integrated into sdm.

Thx.

from sdm.

gitbls commented on September 7, 2024 1

Excellent! And...apologies for the duffs/hassles along the way, and thank you for sticking with me on getting this working for you.

from sdm.

gitbls commented on September 7, 2024

"Is the device name maybe incorrect somehow? device /dev/nvme0n12" This should actually be /dev/nvme0n1p2, referring to the 2nd partition. This should be fixed in V11.2, What version of sdm are you running (sdm --version)?

If your sdm is earlier than 11.2, please update to 11.2 (the latest) and try again. If you're already running 11.2, then let's hope the nvme adapter and device that will soon be on my workbench (coincidentally arriving this afternoon Seattle time) actually work so this can be sorted.

The dmesg errors (AER and PCIe bus errors) are interesting, but I suspect that they're related to sdm not using the correct partition name. We should only worry about those once sdm can correctly and consistently burn and mount the nvme drive.

from sdm.

paulromeijn commented on September 7, 2024

I will check the version asap, but I installed it yesterday following the quick start curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash command from the pi. So it should be a very recent version. Will update when I check it.

from sdm.

gitbls commented on September 7, 2024

If you installed yesterday you've got the latest. Hopefully my nvme adapter and disk arrive...and work...so I can nail whatever this is. Will update this later today.

from sdm.

gitbls commented on September 7, 2024

I was reviewing some code and found the problem. Will be fixed in the next release.

If you want to fix it now, you can sudoedit /usr/local/sdm/sdm-cparse. Line 1401 is:

        [[ "$dmimg" =~ "mmcblk" ]]  || [[ "$dimg" =~ "nvme0n1" ]] && p1="p1" && p2="p2"

Change it to be

        [[ "$dmimg" =~ "mmcblk" ]]  || [[ "$dmimg" =~ "nvme0n1" ]] && p1="p1" && p2="p2"

that is, change dimg to dmimg

Since test done in several locations it will be centralized, but this is the only code that's broken. Pshew!

from sdm.

paulromeijn commented on September 7, 2024

Great, that did the trick! This now seems to work as expected using the flag --expand-root. I do have version 11.2, btw, now confirmed.

> Set hostname 'myhostname'
> First System Boot Custom Boot Scripts disabled
> First System Boot automatic restart enabled
* Burn Completed
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted
* Storage '/dev/nvme0n1' is ready

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0  58.2G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0  57.7G  0 part /
nvme0n1     259:0    0 223.6G  0 disk
├─nvme0n1p1 259:5    0   512M  0 part
└─nvme0n1p2 259:6    0 223.1G  0 part

One other thing I tried did not go as expected, but that might be me using it wrong. I tried again using parted as a burn-plugin to add a partition and resizing the root and the added one:
sudo sdm --burn /dev/nvme0n1 --hostname myhostname--burn-plugin parted:"rootexpand=122880|addpartition=102400,ext4" --regen-ssh-host-keys 2023-12-11-bookworm-arm64-lite-my-custom-image.img

But this ends in:

* Run Post-Burn plugins
> Run Plugins Phase 'burn-complete'
> Run Plugin 'parted' (/usr/local/sdm/plugins/parted) Phase burn-complete with arguments: 'rootexpand=122880|addpartition=102400,ext4|burndev=/dev/nvme0n1|burnfilefile=|imgtype=Device'
/usr/local/sdm/plugins/parted: line 19: /etc/sdm/sdm-readparams: No such file or directory
/usr/local/sdm/plugins/parted: line 221: plugin_getargs: command not found
? Plugin 'parted' exited with failure

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0  58.2G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0  57.7G  0 part /
nvme0n1     259:0    0 223.6G  0 disk
├─nvme0n1p1 259:3    0   512M  0 part
└─nvme0n1p2 259:4    0     2G  0 part

Root is not expanded because parted is called and new partition not created, as expected from the specification. But am I using it wrong or could this be related?

from sdm.

gitbls commented on September 7, 2024

I just found and fixed a few bugs in the emerging 11.3 version, but I've never seen these errors:

/usr/local/sdm/plugins/parted: line 19: /etc/sdm/sdm-readparams: No such file or directory
/usr/local/sdm/plugins/parted: line 221: plugin_getargs: command not found

Only the first error is interesting, since the first error results in the 2nd error.

A couple of questions so I can repro it here:

What is the host hardware (where sdm is running)?
What is the host OS?

And a note for the future: The first several lines of /etc/sdm/history in the customized IMG or burned SSD/SD Card are super-helpful, especially the first lines to "Plugins selected:" for these types of issues. For plugin problems, helpful to include up to the first report of free disk space.

from sdm.

paulromeijn commented on September 7, 2024

A bit of delay because of our timezones.

The host hardware is a Raspberry Pi 5 Model B Rev 1.0 (revision d04170) 8Gb with a pimoroni bottom nvme hat. The Nvme drive is a Patriot P310 240gb. It's running Linux version 6.1.0-rpi8-rpi-2712 ([email protected]) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25).

I am redoing a test from scratch: a customized latest version of rapberry os lite. Two tests: 1) set burn plugin at burn command and 2) set it at customization command:

Test 1
Customization without parted plugin parameters, I will set that at burn time:
$ sudo sdm --customize --plugin user:"adduser=testuser|prompt" --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart 2023-12-11-bookworm-arm64-lite-test-custom.img

And then burn with parted parameters:
$ sudo sdm --burn /dev/nvme0n1 --hostname testpi --burn-plugin parted:"rootexpand=122880|addpartition=102400,ext4" --regen-ssh-host-keys 2023-12-11-bookworm-arm64-lite-test-custom.img

Which does not set partitions as I expect and the following output:

* Burn '2023-12-11-bookworm-arm64-lite-test-custom.img' (2.7GB, 2.6GiB) to '/dev/nvme0n1'...
dd if=2023-12-11-bookworm-arm64-lite-test-custom.img of=/dev/nvme0n1 status=progress bs=16M iflag=direct
2634022912 bytes (2.6 GB, 2.5 GiB) copied, 25 s, 105 MB/s
163+1 records in
163+1 records out
2738880512 bytes (2.7 GB, 2.6 GiB) copied, 26.6617 s, 103 MB/s
> Set new disk ID '8ec64463' on '/dev/nvme0n1'
Disk identifier changed from 0x4e639091 to 0x8ec64463.

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
> Disk ID 8ec64463 set successfully with sfdisk
* Mount device '/dev/nvme0n1'
mount: /dev/nvme0n1p2 mounted on /mnt/sdm.
mount: /dev/nvme0n1p1 mounted on /mnt/sdm/boot/firmware.
> Set new Disk ID '8ec64463' in /etc/fstab and /boot/firmware/cmdline.txt
> Flush accumulated burn log messages
> Force --no-expand-root for parted burn-plugin
> Force --regen-ssh-host-keys for --no-expand-root
> Disable unneeded RasPiOS firstboot service and /etc/init.d/resize2fs_once
> Disable regenerate_ssh_host_keys service; First Boot will run it instead
> Set hostname 'testpi'
> First System Boot Custom Boot Scripts disabled
> First System Boot automatic restart enabled
* Burn Completed
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted
* Run Post-Burn plugins
> Run Plugins Phase 'burn-complete'
> Run Plugin 'parted' (/usr/local/sdm/plugins/parted) Phase burn-complete with arguments: 'rootexpand=122880|addpartition=102400,ext4|burndev=/dev/nvme0n1|burnfilefile=|imgtype=Device'
/usr/local/sdm/plugins/parted: line 19: /etc/sdm/sdm-readparams: No such file or directory
/usr/local/sdm/plugins/parted: line 221: plugin_getargs: command not found
? Plugin 'parted' exited with failure

$  lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0  58.2G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0  57.7G  0 part /
nvme0n1     259:0    0 223.6G  0 disk
├─nvme0n1p1 259:1    0   512M  0 part
└─nvme0n1p2 259:2    0     2G  0 part

Test 2: set burn plugin at customization:
Customize image and set burn plugin parted:
$ sudo sdm --customize --plugin user:"adduser=testuser|prompt" --plugin L10n:host --plugin disables:piwiz --burn-plugin parted:"rootexpand=122880|addpartition=102400,ext4" --regen-ssh-host-keys --restart 2023-12-11-bookworm-arm64-lite-test-custom.img

Gives output:

** IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has already been customized. Redo? [y/n]? y
* Start Configuration
> Command Line: /usr/local/bin/sdm --customize --plugin user:adduser=testuser|prompt --plugin
  L10n:host --plugin disables:piwiz --burn-plugin
  parted:rootexpand=122880|addpartition=102400,ext4 --regen-ssh-host-keys --restart
  2023-12-11-bookworm-arm64-lite-test-custom.img
* Host Information
   Hostname: unipi-sd
   uname: Linux unipi-sd 6.1.0-rpi8-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25)
  aarch64 GNU/Linux
   os-release Name: Debian GNU/Linux 12 (bookworm)
   Version: 12 (bookworm)
   sdm Version: V11.2
* IMG Information
   Name: 2023-12-11-bookworm-arm64-lite-test-custom.img
   Date: 2023-12-11
   RasPiOS Version: 12
   RasPiOS Architecture: 64-bit aarch64
   os-release Version: 12 (bookworm)
* Plugins selected:
   * user
       Args: adduser=testuser|prompt
   * L10n
       Args: host
   * disables
       Args: piwiz
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 117476 1K-blocks (0.1GB, 0.1GiB) free
> Copy sdm to /usr/local/sdm in the IMG
* Start Phase 0 image customization
> Run Plugins Phase '0'
> Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments:
  'adduser=testuser|prompt'
* Plugin user: Start Phase 0
> Plugin user: Keys/values found:
   adduser: testuser
   prompt:
% Plugin user: User 'testuser' home directory '/home/testuser' already exists
* Plugin user: Complete Phase 0
> Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host'
* Plugin L10n: Start Phase 0
> Plugin L10n: Keys/values found:
   host:
> Plugin L10n: Read L10n configuration from host system
> Plugin L10n: Load Localization (L10N) settings from running system
> Plugin L10n:   Keymap:       us
> Plugin L10n:   Locale:       en_GB.UTF-8
> Plugin L10n:   Timezone:     Europe/Madrid
> Plugin L10n:   WiFi Country:
> Plugin L10n: Save L10n configuration
* Plugin L10n: Complete Phase 0
> Run Plugin 'disables' (/mnt/sdm/usr/local/sdm/plugins/disables) Phase 0 with arguments:
  'piwiz'
* Plugin disables: Start Phase 0
> Plugin disables: Keys/values found:
   piwiz:
* Plugin disables: Complete Phase 0
* Phase 0 Completed
* Enter image '2023-12-11-bookworm-arm64-lite-test-custom.img' for Phase 1
* Start Phase 1 image customization
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 117472 1K-blocks (0.1GB, 0.1GiB) free at start of Phase 1 image customization
> Configure and enable sdm FirstBoot service (sdm-firstboot)
> Start 'apt update'
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 117472 1K-blocks (0.1GB, 0.1GiB) free at start of 'apt update'
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 117476 1K-blocks (0.1GB, 0.1GiB) free at end of 'apt update'> Run Plugins Phase '1'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase 1 with arguments:
  'adduser=testuser|prompt'
* Plugin user: Start Phase 1
% Plugin user: User 'testuser' already in /etc/passwd
* Plugin user: Complete Phase 1
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase 1 with arguments: 'host'
* Plugin L10n: Start Phase 1
% Plugin L10n: Keymap will be set during FirstBoot
> Plugin L10n: Disable keyboard-setup service; will be re-enabled during FirstBoot
> Plugin L10n: Set locale to 'en_GB.UTF-8'
Generating locales (this might take a while)...
  en_GB.UTF-8... done
Generation complete.
> Plugin L10n: Set timezone to 'Europe/Madrid'
* Plugin L10n: Complete Phase 1
> Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase 1 with arguments: 'piwiz'
* Plugin disables: Start Phase 1
* Plugin disables: Complete Phase 1
* Phase 1 post-app installation/configuration
> Start 'apt upgrade'
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 114532 1K-blocks (0.1GB, 0.1GiB) free at start of 'apt upgrade'
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 114532 1K-blocks (0.1GB, 0.1GiB) free at end of 'apt upgrade'
> Start 'apt autoremove'
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 114532 1K-blocks (0.1GB, 0.1GiB) free at start of 'apt autoremove'
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 114532 1K-blocks (0.1GB, 0.1GiB) free at end of 'apt autoremove'
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 114528 1K-blocks (0.1GB, 0.1GiB) free at end of Phase 1 image customization
* Phase 1 Completed
> Run Plugins Phase 'post-install'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase post-install with arguments:
  'adduser=testuser|prompt'
* Plugin user: Start Phase post-install
* Plugin user: Complete Phase post-install
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase post-install with arguments: 'host'
> Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase post-install with arguments:
  'piwiz'
* Plugin disables: Start Phase post-install
> Plugin disables: Disable running piwiz and userconfig on first boot per --disable piwiz
* Plugin disables: Complete Phase post-install
> Run graphics post-install configuration
> Plugin sdm-phase1: No known Display Manager found
> Plugin sdm-phase1: Set RasPiOS to B1:'console no login' for First Boot
> Enable SSH
> Plugin phase1: Enable SSH service via /boot/ssh
> IMG '2023-12-11-bookworm-arm64-lite-test-custom.img' has 114528 1K-blocks (0.1GB, 0.1GiB) free at end of image customization
* Enter Shell Command Prompt

The burn without any parameters linked to parted:
$ sudo sdm --burn /dev/nvme0n1 --hostname testpi --regen-ssh-host-keys 2023-12-11-bookworm-arm64-lite-test- custom.img

Gives the output:

* Burn '2023-12-11-bookworm-arm64-lite-test-custom.img' (2.7GB, 2.6GiB) to '/dev/nvme0n1'...
dd if=2023-12-11-bookworm-arm64-lite-test-custom.img of=/dev/nvme0n1 status=progress bs=16M iflag=direct
2738880512 bytes (2.7 GB, 2.6 GiB) copied, 26 s, 105 MB/s
163+1 records in
163+1 records out
2738880512 bytes (2.7 GB, 2.6 GiB) copied, 26.7771 s, 102 MB/s
> Set new disk ID '0893561a' on '/dev/nvme0n1'
Disk identifier changed from 0x4e639091 to 0x0893561a.

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
> Disk ID 0893561a set successfully with sfdisk
* Mount device '/dev/nvme0n1'
mount: /dev/nvme0n1p2 mounted on /mnt/sdm.
mount: /dev/nvme0n1p1 mounted on /mnt/sdm/boot/firmware.
> Set new Disk ID '0893561a' in /etc/fstab and /boot/firmware/cmdline.txt
> Flush accumulated burn log messages
> Force --no-expand-root for parted burn-plugin
> Force --regen-ssh-host-keys for --no-expand-root
> Disable unneeded RasPiOS firstboot service and /etc/init.d/resize2fs_once
> Disable regenerate_ssh_host_keys service; First Boot will run it instead
> Set hostname 'testpi'
> First System Boot Custom Boot Scripts disabled
> First System Boot automatic restart enabled
* Burn Completed
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted
* Run Post-Burn plugins
> Run Plugins Phase 'burn-complete'
> Run Plugin 'parted' (/usr/local/sdm/plugins/parted) Phase burn-complete with arguments: 'rootexpand=122880|addpartition=102400,ext4|burndev=/dev/nvme0n1|burnfilefile=|imgtype=Device'
/usr/local/sdm/plugins/parted: line 19: /etc/sdm/sdm-readparams: No such file or directory
/usr/local/sdm/plugins/parted: line 221: plugin_getargs: command not found
? Plugin 'parted' exited with failure

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0  58.2G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0  57.7G  0 part /
nvme0n1     259:0    0 223.6G  0 disk
├─nvme0n1p1 259:5    0   512M  0 part
└─nvme0n1p2 259:6    0     2G  0 part

from sdm.

paulromeijn commented on September 7, 2024

I have also rerun the test to set burnplugin at customization with a fresh copy of clean raspberry os image (uncustomized, just in case). I set the burnplugin sudo sdm --customize --plugin user:"adduser=testuser|prompt" --plugin L10n:host --plugin disables:piwiz --burn-plugin parted:"rootexpand=122880|addpartition=102400,ext4" --regen-ssh-host-keys --restart 2023-12-11-bookworm-arm64-lite-test.img and then I check the contents of the files in the errors that appeared in previous output.
Output:

* Start Configuration
> Command Line: /usr/local/bin/sdm --customize --plugin user:adduser=testuser|prompt --plugin
  L10n:host --plugin disables:piwiz --burn-plugin
  parted:rootexpand=122880|addpartition=102400,ext4 --regen-ssh-host-keys --restart
  2023-12-11-bookworm-arm64-lite-test.img
* Host Information
   Hostname: unipi-sd
   uname: Linux unipi-sd 6.1.0-rpi8-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25)
  aarch64 GNU/Linux
   os-release Name: Debian GNU/Linux 12 (bookworm)
   Version: 12 (bookworm)
   sdm Version: V11.2
* IMG Information
   Name: 2023-12-11-bookworm-arm64-lite-test.img
   Date: 2023-12-11
   RasPiOS Version: 12
   RasPiOS Architecture: 64-bit aarch64
   os-release Version: 12 (bookworm)
* Plugins selected:
   * user
       Args: adduser=testuser|prompt
   * L10n
       Args: host
   * disables
       Args: piwiz
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 337076 1K-blocks (0.3GB, 0.3GiB) free
> Copy sdm to /usr/local/sdm in the IMG
* Start Phase 0 image customization
> Run Plugins Phase '0'
> Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments:
  'adduser=testuser|prompt'
* Plugin user: Start Phase 0
> Plugin user: Keys/values found:
   adduser: testuser
   prompt:
> Plugin user: Create user 'testuser' home directory '/home/testuser' so available for other
  plugins
* Plugin user: Complete Phase 0
> Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host'
* Plugin L10n: Start Phase 0
> Plugin L10n: Keys/values found:
   host:
> Plugin L10n: Read L10n configuration from host system
> Plugin L10n: Load Localization (L10N) settings from running system
> Plugin L10n:   Keymap:       us
> Plugin L10n:   Locale:       en_GB.UTF-8
> Plugin L10n:   Timezone:     Europe/Madrid
> Plugin L10n:   WiFi Country:
> Plugin L10n: Save L10n configuration
* Plugin L10n: Complete Phase 0
> Run Plugin 'disables' (/mnt/sdm/usr/local/sdm/plugins/disables) Phase 0 with arguments:
  'piwiz'
* Plugin disables: Start Phase 0
> Plugin disables: Keys/values found:
   piwiz:
* Plugin disables: Complete Phase 0
* Phase 0 Completed
* Enter image '2023-12-11-bookworm-arm64-lite-test.img' for Phase 1
* Start Phase 1 image customization
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 337032 1K-blocks (0.3GB, 0.3GiB) free at start of Phase 1 image customization
> Configure and enable sdm FirstBoot service (sdm-firstboot)
> Start 'apt update'
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 337028 1K-blocks (0.3GB, 0.3GiB) free at start of 'apt update'
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 221928 1K-blocks (0.2GB, 0.2GiB) free at end of 'apt update'
> Run Plugins Phase '1'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase 1 with arguments:
  'adduser=testuser|prompt'
* Plugin user: Start Phase 1
> Plugin user: Add user 'testuser'
> Plugin user: Add groups to user 'testuser':
  dialout,cdrom,floppy,audio,video,plugdev,users,adm,sudo,users,input,netdev,spi,i2c,gpio
Password for testuser:
> Plugin user: Enable sudo for user 'testuser'
> Plugin user: Copy skel files for user 'testuser'
> Plugin user: Set user 'testuser' home directory '/home/testuser' owner to 'testuser:testuser'
> Plugin user: Set password for user 'testuser'
* Plugin user: Complete Phase 1
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase 1 with arguments: 'host'
* Plugin L10n: Start Phase 1
% Plugin L10n: Keymap will be set during FirstBoot
> Plugin L10n: Disable keyboard-setup service; will be re-enabled during FirstBoot
Removed "/etc/systemd/system/sysinit.target.wants/keyboard-setup.service".
> Plugin L10n: Set locale to 'en_GB.UTF-8'
Generating locales (this might take a while)...
  en_GB.UTF-8... done
Generation complete.
> Plugin L10n: Set timezone to 'Europe/Madrid'
* Plugin L10n: Complete Phase 1
> Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase 1 with arguments: 'piwiz'
* Plugin disables: Start Phase 1
* Plugin disables: Complete Phase 1
* Phase 1 post-app installation/configuration
> Start 'apt upgrade'
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 216996 1K-blocks (0.2GB, 0.2GiB) free at start of 'apt upgrade'
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 114580 1K-blocks (0.1GB, 0.1GiB) free at end of 'apt upgrade'
> Start 'apt autoremove'
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 114580 1K-blocks (0.1GB, 0.1GiB) free at start of 'apt autoremove'
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 114580 1K-blocks (0.1GB, 0.1GiB) free at end of 'apt autoremove'
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 114580 1K-blocks (0.1GB, 0.1GiB) free at end of Phase 1 image customization
* Phase 1 Completed
> Run Plugins Phase 'post-install'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase post-install with arguments:
  'adduser=testuser|prompt'
* Plugin user: Start Phase post-install
* Plugin user: Complete Phase post-install
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase post-install with arguments: 'host'
> Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase post-install with arguments:
  'piwiz'
* Plugin disables: Start Phase post-install
> Plugin disables: Disable running piwiz and userconfig on first boot per --disable piwiz
* Plugin disables: Complete Phase post-install
> Run graphics post-install configuration
> Plugin sdm-phase1: No known Display Manager found
> Plugin sdm-phase1: Set RasPiOS to B1:'console no login' for First Boot
> Enable SSH
> Plugin phase1: Enable SSH service via /boot/ssh
> IMG '2023-12-11-bookworm-arm64-lite-test.img' has 114580 1K-blocks (0.1GB, 0.1GiB) free at end of image customization
* Enter Shell Command Prompt
  'exit' to exit back to host system

root@sdm:~# nano /usr/local/sdm/plugins/parted

Then line 18-19-20 is:

function loadparams() {
    source $SDMPT/etc/sdm/sdm-readparams
}

Trying that referenced file root@sdm:~# nano /etc/sdm/sdm-readparams gives:

#!/bin/bash
#
# This file is sourced by other sdm scripts
#
function readparams() {
    #
    # Restores the global variables from /etc/sdm/cparams
    #
    local rpifun value cf="$SDMPT/etc/sdm/cparams" snolq
    if [ -f $cf ]
    then
        while IFS=":" read -r rpifun value
        do
            if [[ ! $rpifun =~ ^\ *# && -n $rpifun ]] # skip comment and malformed lines
            then
                # Cant' use stripquotes, it's in sdm-cparse, and can't source it w/o processing this first ($sdmdir)
                #value="${value%%\#*}"    # Del EOL comments
                value="${value%"${value##*[^[:blank:]]}"}"  # Del trailing spaces/tabs
                snolq="${value#\"}"        # Get string without open quote
                if [ "$snolq" != "$value" ] # if lq was there, then update string and also del close quote
                then
                    value=$snolq
                    value="${value%\"}"
                fi
                snolq="${value#\'}"        # Get string without open quote
                if [ "$snolq" != "$value" ] # Ditto double quote comment
                then
                    value=$snolq
                    value="${value%\'}"
                fi
                printf -v "$rpifun" "%s" "$value"     #eval "${rpifun}=\"$value\""

            fi
        done < $cf
    fi
    [ -f $SDMPT/$sdmdir/sdm-cparse ] && source $SDMPT/$sdmdir/sdm-cparse || echo "? Internal error: File '$SDMPT/$sdmdir/sd>    return
}
# Call the function after we've been sourced
readparams

So they are present in the image. But somehow code doesn't find it? I don't know how to debug this so not sure how for example $SDMPT/etc/sdm/sdm-readparams will evaluate.

Hope this and previous helps?

from sdm.

gitbls commented on September 7, 2024

Thanks for the detailed details 😂 I've got enough info to nail it now.

from sdm.

paulromeijn commented on September 7, 2024

Haha yeah I figured you know better than me what to filter out

from sdm.

gitbls commented on September 7, 2024

Just did a test with 11.2 (which you're using). Wasn't able to replicate the problem with "/etc/sdm/sdm-readparams: No such file or directory", but did run into another issue I fixed yesterday. Reran the same test with 11.3 and all good.

Now I just need to finish up 11.3 for you 🥴Still chasing a couple of other features/bugs, but should have it done in the next day or two, worst case.

from sdm.

paulromeijn commented on September 7, 2024

Too bad that it doesn't reproduce, but fortunately I can work around it. I burned the image without the --expand-root flag and manually expanded the primary partition+fs and created the extra partition+fs. Seems to work.

Could it be due to us using different brand NVMe hats? Also tried setting/unsetting dtparam=pciex1_gen=3 in boot config.txt just to rule it out, but no difference.

Nonetheless, I am grateful for such a great tool being here in the first place.

from sdm.

gitbls commented on September 7, 2024

I believe these are all fixed in 11.3, per testing on my NVME drive. Please test with your configuration. Thx!

from sdm.

paulromeijn commented on September 7, 2024

I just tested on a new Rpi5 4Gb, now with a kingston NV2 1TB drive and another pimoroni base hat using SDM 11.3 and still have the issue:

Run Plugin 'parted' (/usr/local/sdm/plugins/parted) Phase burn-complete with arguments: 'rootexpand=122880|addpartition=102400,ext4|burndev=/dev/nvme0n1|burnfilefile=|imgtype=Device'
/usr/local/sdm/plugins/parted: line 19: /etc/sdm/sdm-readparams: No such file or directory
/usr/local/sdm/plugins/parted: line 221: plugin_getargs: command not found
? Plugin 'parted' exited with failure

But it is another issue than the one I started with, so I could close this one because now that does work. As I'm still with the pimoroni, maybe that has something to do with it?

from sdm.

gitbls commented on September 7, 2024

I'm not worried about whether it's this issue or another issue. This one is open, so let's just continue here.

I need to see the /etc/sdm/history file from in the IMG that you're burning. You can either post it here or email it to me (email on https://github.com/gitbls/sdm/README.md). My suspicion is that there's something in the way your IMG has been customized, but in addition to the history file, please also shoot me the commands you're using to customize and burn. Please feel free to redact passwords as necessary, but leave everything else intact.

Thx!

from sdm.

gitbls commented on September 7, 2024

Ah! Never mind...I was in my best thinking place (shower 😂) and realized exactly what the problem is. 🥴

from sdm.

gitbls commented on September 7, 2024

On host systems running sdm where sdm was NOT used to burn the host system disk, there is an error in the installer. This will be corrected shortly in the default download. The issue is that the installer doesn't do these steps, which burn-plugins rely on.

You can correct your system so that the parted plugin works with the following. It assumes that you put sdm in the default location (/usr/local/sdm). If you've changed that, modify these commands appropriately:

sudo mkdir -p /etc/sdm
sudo cp /usr/local/sdm/sdm-readparams /etc/sdm/readparams
sudo cat >| /etc/sdm/cparse <<EOF
sdmdir:/usr/local/sdm
src:/usr/local/sdm
plugindebug:0
EOF

Please LMK if this fixes it.

Apologies for the delay in nailing this for you 😳

from sdm.

paulromeijn commented on September 7, 2024

The first two lines ok. But then I'm unfamiliar with the syntax. I execute it line by line and there is an error -bash: /etc/sdm/cparse: Permission denied
I do use sudo cat + the rest

from sdm.

gitbls commented on September 7, 2024

The first two lines ok. But then I'm unfamiliar with the syntax. I execute it line by line and there is an error -bash: /etc/sdm/cparse: Permission denied I do use sudo cat + the rest

Sorry, was apparently in too much of a hurry. Try this.

After you type the cat command, you should be prompted with > . Type each of the 3 lines in turn, then type EOF (on a line by itself), and the cat command will complete.

sudo rm -rf /etc/sdm
sudo mkdir -p /etc/sdm
sudo cp -a /usr/local/sdm/sdm-readparams /etc/sdm
sudo cat > /etc/sdm/cparams <<EOF
sdmdir:/usr/local/sdm
src:/usr/local/sdm
plugindebug:0
EOF

from sdm.

paulromeijn commented on September 7, 2024

Still get permission denied:

~ $ sudo rm -rf /etc/sdm
~ $ sudo mkdir -p /etc/sdm
~ $ sudo cp -a /usr/local/sdm/sdm-readparams /etc/sdm
~ $ sudo cat > /etc/sdm/cparams <<EOF
> sdmdir:/usr/local/sdm
> src:/usr/local/sdm
> plugindebug:0
> EOF
-bash: /etc/sdm/cparams: Permission denied

So I outputted the file first to home dir and then copied it to /etc/sdm manually. Now the cparams is in that folder.
The image was customized with the burn parameter --burn-plugin parted:"rootexpand=307200" and then burned without --root-expand:
$ sudo sdm --burn /dev/nvme0n1 --hostname testpi --regen-ssh-host-keys 2023-12-11-bookworm-arm64-lite-custom.img

And success! 😎 Nice

umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted
* Storage '/dev/nvme0n1' is ready
~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0    58G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0  57.4G  0 part /
nvme0n1     259:0    0 931.5G  0 disk
├─nvme0n1p1 259:3    0   512M  0 part
└─nvme0n1p2 259:4    0   302G  0 part

Not sure why cat has no permissions while using sudo?

But anyway now it is all working as expected. Any suggestion how I can set another partition on that disk using the burn plugin that expands to fill the whole free space? It's not fully clear from the docs to me, only how to expand root. I have about 630GB left that I want to configure to the last byte, but would like to be able to do it without calculating all clusters or guessing etc.

--edit: (re-reading the docs)
Or would the following command do the above and expand the extra partition by setting size as 0?
--burn-plugin parted:"rootexpand=307200|addpartition=0,ext4"

--edit2:
I tried customizing using --burn-plugin parted:"rootexpand=307200|addpartition=0,ext4,primary" and then burn, but that leads to:

(--snip--)
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted
> Plugin parted: Make primary partition 3 start: 634497024s end: 634494975s
Error: Can't have the end before the start! (start sector=634497024 length=-2048)
? parted mkpart error 1
? Plugin 'parted' exited with failure
~$  lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0    58G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0  57.4G  0 part /
nvme0n1     259:0    0 931.5G  0 disk
├─nvme0n1p1 259:3    0   512M  0 part
└─nvme0n1p2 259:4    0   302G  0 part

So I haven't figured it out yet.

from sdm.

gitbls commented on September 7, 2024

Confirmed. Adding a last partition with a size of 0 will fill the whole free space in the next release.

Of course, if there's another partition specified after a 0-length partition, it will fail miserably, since there will be no room for it.

from sdm.

UnconnectedBedna commented on September 7, 2024

Hi, me again. xD

It was good I read this because I was thinking of forking and sending 1 or 2 requests.
There are things I could provide knowledge wise that could speed up the "burn" process by hundreds of % if you are interested? (rsync instead of dd, like I do in my script)

I also saw you are doing some unnecessary operations that I could help out with if you'd like. (partition resizing and stuff)

But you should def make the changes you have in mind first because I might need to move a few things around to do what I have in mind.

from sdm.

gitbls commented on September 7, 2024

But anyway now it is all working as expected. Any suggestion how I can set another partition on that disk using the burn plugin that expands to fill the whole free space? It's not fully clear from the docs to me, only how to expand root. I have about 630GB left that I want to configure to the last byte, but would like to be able to do it without calculating all clusters or guessing etc.

--edit: (re-reading the docs) Or would the following command do the above and expand the extra partition by setting size as 0? --burn-plugin parted:"rootexpand=307200|addpartition=0,ext4"

This is fixed in the just-released V11.4. Please test. Thx!

from sdm.

paulromeijn commented on September 7, 2024

Ok interesting, I tried installing latest version using curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash

Resulted in this:

(--snip---)
Downloading 'plugins/wsdd' to '/usr/local/sdm/plugins/wsdd'
Downloading '1piboot/1piboot.conf' to '/usr/local/sdm/1piboot/1piboot.conf'
* Create link for sdm: /usr/local/bin/sdm
bash: line 177: /etc/sdm/cparse: Permission denied
* Ensure required packages are installed: systemd-container qemu-user-static binfmt-support parted
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
systemd-container is already the newest version (252.19-1~deb12u1).
qemu-user-static is already the newest version (1:7.2+dfsg-7+deb12u3).
binfmt-support is already the newest version (2.2.2-2).
parted is already the newest version (3.5-3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

* Invoke sdm with 'sudo sdm'

  Complete sdm documentation: https://github.com/gitbls/sdm/tree/master/Docs

$ sudo sdm --version
sdm V11.4

Note the line bash: line 177: /etc/sdm/cparse: Permission denied
So here also no access to the cparams file. Not sure why though, my installation here is pretty clean and basic afaik.

--edit: using sudo curl ~~ to install same result. Not sure if sudo is necessary there though

from sdm.

gitbls commented on September 7, 2024

Ok interesting, I tried installing latest version using curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash

Resulted in this:
...
Note the line bash: line 177: /etc/sdm/cparse: Permission denied So here also no access to the cparams file. Not sure why though, my installation here is pretty clean and basic afaik.

--edit: using sudo curl ~~ to install same result. Not sure if sudo is necessary there though

Pretty sure I understand what's happening, but what is the host OS and version that you're installing on?
Thx

from sdm.

paulromeijn commented on September 7, 2024

Hardware is Raspberry Pi 5 Model B Rev 1.0 (4GB)

$ cat /proc/version
Linux version 6.1.0-rpi8-rpi-2712 ([email protected]) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25)

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

from sdm.

gitbls commented on September 7, 2024

OK, thx. Please download the latest EZsdmInstaller and rerun it. It will properly create cparse, and I'm off to read about "here documents" and sudo 😕

from sdm.

paulromeijn commented on September 7, 2024

ok that went well now, great!

(---)
Downloading '1piboot/1piboot.conf' to '/usr/local/sdm/1piboot/1piboot.conf'
* Create link for sdm: /usr/local/bin/sdm
* Ensure required packages are installed: systemd-container qemu-user-static binfmt-support parted
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
systemd-container is already the newest version (252.19-1~deb12u1).
qemu-user-static is already the newest version (1:7.2+dfsg-7+deb12u3).
binfmt-support is already the newest version (2.2.2-2).
parted is already the newest version (3.5-3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

* Invoke sdm with 'sudo sdm'

  Complete sdm documentation: https://github.com/gitbls/sdm/tree/master/Docs

Will now try the image burning again

from sdm.

paulromeijn commented on September 7, 2024

ok that's looking good, great work!

(----)
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

* Storage '/dev/nvme0n1' is ready


$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0    58G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0  57.4G  0 part /
nvme0n1     259:0    0 931.5G  0 disk
├─nvme0n1p1 259:4    0   512M  0 part
├─nvme0n1p2 259:5    0   302G  0 part
└─nvme0n1p3 259:6    0   629G  0 part

from sdm.

Error mounting after successful burn rpi5 NVMe about sdm HOT 32 CLOSED

Comments (32)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent