lectrode / xs-update-manjaro Goto Github PK

This may be implemented as part of refactoring the manual updates section of code.

It should probably fall back to manually downloading it (like it does now) in the event that the pacman-static package is removed from repos.

Since the script runs as root, it uses su to run send-notify as each user that is logged in. Latest update in dbus broke this function.

Needs more research, but may be able to fix by finding the DBUS_SESSION_BUS_ADDRESS environment variable for each user session and manually setting it when executing the send-notify command.

pacman-mirrors no longer supports -g

In addition, it also no longer supports OnlyCountry. In order to retain the ability to have a country specified, I'll be adding a new setting to the update script called str_mirrorCountry. I'll also change the default command of pacman-mirrors -g to pacman-mirrors --geoip. (Setting str_mirrorCountry will use -c Country,Country instead of --geoip)

Thank you.
I have installed "xs-update-manjaro" on at least three machines without any problems at all.
Here is my script.txt if you want it.

xs-lectrode-install.txt

Problems:

1. Package from official repo that needs to be rebuilt will be unnecessarily re-installed
2. Package from AUR with same name as one in repo that needs to be rebuilt may be installed
3. A binary package from AUR won't actually be rebuilt, and may not remove dependencies on old python

Solutions:

• Ensure only foreign packages (pacman -Qqm) are rebuilt
• Restrict aur helper to only packages from AUR
• Check if packages to build changed, instead of successful build/install (prevent infinite loop caused by 1,3)

Currently, script will update the main system packages, then download and update everything else.

Should download as much as possible before beginning changes to the system to reduce the window in which catastrophic failure (like a power outage or unintentional shutdown) may cause an unbootable/malfunctioning system.

Goals:

• Like-settings together (sort by category instead of type)
• Readability

(Needed for updating self)

• Better config validation
• Scalable naming scheme
• Maintain limited backward compatibility

When a kernel is EOL, updates will automatically remove it in order to allow updates to proceed. However, if that's the only kernel installed, the system will be unbootable after restarting.

Script should at least have an option to make sure the latest LTS kernel is installed to avoid these issues.

due to upstream changes, critical packages such as archlinux-keyring and manjaro-keyring are now detected as orphan packages.

Implementing this will require the following:

• Settings overhaul

  • Scalable settings
  • Backward compatibility
  • Better config validation

• An update mechanism, either:

  • Update directly from git repo
  • Create a package for AUR

I'm wary of creating a package for the AUR for the following reasons:

• Anything to automate updates is generally frowned upon in the Arch community
• This package is primarily for Manjaro (unofficially), so adding to AUR may seem out of place

This will likely check for updates directly from this repository.

Splitting this out from #45, as this is effectively a separate option and separate implementation.

This is a prerequisite of #45, but it's basically already implemented (not released yet).

some installs started with linux-lts<=1:5.4-4 or linux-latest<=5.10-1 packages, which are no longer provided by manjaro (and possibly other distros). removing the kernels these metapackages installed requires removing these packages.

if these packages are still provided by upstream providers (any configured repos, or the AUR), they will not be automatically removed (like in the case of Garuda Linux)

note: removing linux-lts-headers with sudo pacman -Rdd resulted in an orphaned linx510-headers. in cases like these, the lts kernel packages will need to be set to explicitly installed before the meta package is removed

According to this, @ and _ symbols are allowed in package names.

Currently there are zero packages in either the main repos or the AUR that contain an @ symbol.

However, there are plenty of packages with _ in them.

This is mainly for fixing package database issues - no other functions match the package name with regex.

Forum thread:
https://forum.manjaro.org/t/maintainer-pgp-keys-expired/76545/43?u=lectrode

Normally, refreshing the package signature keys is something you need to do on a regular basis to make sure your system successfully updates. However, If a user refreshes their package keys currently, the key of one of the core maintainers of manjaro gets updated to an "expired" status, preventing further updates of core packages (like manjaro-keyring and kernel packages).

I've deployed a hotfix to temporarily disable refreshing the keys until this is resolved.

If python is updated, pikaur breaks. Being an AUR package, pikaur currently relies on itself for updating itself - which is not possible if it is broken.

likely use pikaur to re-install pikaur:

git clone https://github.com/actionless/pikaur.git
cd pikaur
python3 ./pikaur.py -S pikaur

Obviously this solution would only be needed if pikaur is installed and AUR Helpers are enabled in this script

if the network connection drops before pikaur is executed, it will either get stuck in an infinite loop or crash after exceeding the max recursion depth

workaround: perform basic internet connection test just before running pikaur

For example:

error: could not open file /var/lib/pacman/local/libreoffice-still-6.2.8-4/desc: No such file or directory
error: could not open file /var/lib/pacman/local/libreoffice-still-6.2.8-4/files: No such file or directory

solution:

touch /var/lib/pacman/local/libreoffice-still-6.2.8-4/desc
touch /var/lib/pacman/local/libreoffice-still-6.2.8-4/files
pacman -Syy libreoffice-still --overwrite=*

Quick check for these error messages (discovers either, not both):

pacman -Qo pacman

Finds all errors:

pacman -Dk

New option (disabled by default) to enabling the script to automatically reboot if it detects a reboot is required.

This may be accompanied by a 2 minute delay if logged-in users are detected, during which a notification stating the computer will automatically restart in X:XX every 15 or 20 seconds. Notification time may be adjustable. Delay may also be adjustable.

Currently, package signature keys are refreshed every time the script is run (if enabled), which avoids this issue. The new default will have keys manually updated once a month by default, which should avoid this issue.

However, I could potentially go one step further and refresh the keys upon detecting this error:

error: package-name: signature from "packager" is unknown trust

Only issue with this is that the script can only detect the error after attempting to install packages, so it would need to parse it's own log to check...

More info:

• source

Per this thread, there are more known applications that can potentially require updates. Currently, the script only checks for kernel or systemd updates. This is usually fine, as the kernel and/or systemd is updated frequent enough that the listed packages are unlikely to be updated by themselves.

Will likely either add those packages to the hard-coded list to check, or possibly look into needrestart^AUR to see if there is a more uniform way applications let the system know they need a restart.

script should force pikaur to be from the AUR for the following:

1. script uses pikaur to test ability to install from AUR
2. pikaur from 3rd party repo may not be built against the correct versions of dependencies (i.e. python)

This is particularly important when script relies on pikaur to rebuild packages (like python packages for the 3.10 update)

Flatpak supports removing unused/orphan packages with the following:

flatpak uninstall --unused -y

This should be one of the cleanup operations that this script performs

This would be primarily for AUR packages (repo packages should be taken care of by distro devs). Package should be rebuilt/reinstalled if specified dependencies are updated.

Currently the script uses both wget and curl, depending on what is being downloaded.

I need to review the individual cases and probably standardize on using curl.

I should also make sure that the chosen solution can follow redirects if need be.

It would be nice to support both, while falling back to one or the other if either are not installed, but not strictly necessary, since a lot of packages like git already depend on curl (with seemingly few that depend on wget).

Hello,

Most of what you've done is working properly also on Archlinux, would be an issue for you to make it compatible for both OS ?

Only things which are not working to make it compatible with Arch are:

• manjaro-keyring manjaro-system which are not required on Arch L#714
• pacman-mirrors update which doesn't exist on Arch, workaround is just to disable that feature on Arch.

Are you open to PR to improve that ?

With the abandonment of apacman, an alternative optional AUR helper is needed.

Requirements:

• Run as root
• Non-interactive installation/update
• Handle recursive dependencies

Preferred:

• Stable
• Automatically import and use pgp keys to verify packages

As stated in the title, replacing the /etc/NetworkManager/system-connections folder with a symlink to an alternate location causes issues. Not only that, but managing the location of NetworkManager connections should be out of the scope of this script.

This has been a part of this script for quite some time. There was a time when it was necessary to ensure custom changes were re-implemented after a networkmanager update. However, at this time it does more harm than good and there is a better solution for keeping the system-connections folder on a separate partition, namely, mounting folder to folder as shown below.

permanent fstab entry:
/destination/dir /etc/NetworkManager/system-connections none defaults,bind 0 0

temporary command (needs to be run at every startup)
sudo mount --bind "/destination/dir" "/etc/NetworkManager/system-connections"

There are some tasks that do not need to be run very often. Specifically:

pacman-mirrors

pacman-key --refresh-keys

pikaur -S --needed --noconfirm --devel

While this will not make it into the 3.0 release (nearly complete), at some point I want to add options to specify frequency of the aforementioned tasks to improve efficiency and reduce bandwidth consumption.

The default frequencies for these will likely be at most once per week.

Currently, the script will restart itself as a background process (required if starting as a service).
NOTE: It does not continuously run, it only runs for the duration that it is checking for / installing updates.

At some point I might want to add an option to stay running in foreground (for instances where it is run directly). Script would still default to running in background.

Hey there,

it'd be cool if this script would support the functionality of this script as well, so that there's "one to rule them all".

This means:

• automatically search for updates at shutdown (of snaps, flatpaks, repo packages, aur packages
• update at shutdown only, not at reboots
• use systemd to ensure that the update isn't killed after a timeout (default 1min30)
• graphical integration with progress bars using plymouth (only if Plymouth is present)
• only apply updates when battery is charged to 50% and connected to power (notebook only)

This can be accomplished by using a systemd service that
a) is started at startup
b) has it's dependency services declared
c) prevents those from being stopped at shutdown before the script ends
A specific implementation of such service can be found at the repo above

Also, it's required to override a systemd script to prevent the script from being killed while running. This has the disadvantage of the system potentially hanging endlessly, should there be any other way, recommendations are always welcome.

The updater would have to support snaps as well, to be completely on function parity with the script above.
This could be done by using pamac to update the system. In this case, the progress bar implementation for plymouth is already existing in the script above.

Should I have missed anything, feel free to get in touch! :)

Just something simple to let people know they can do whatever they want with this script as long as it doesn't lock me out of using my own creation

The update script should optionally not download or install updates when the only connection is mobile/data (most cellular services do not have actual unlimited data). This would still be allowed by default for security reasons, as well as to avoid impacting existing devices.

Preliminary search shows that networks provided by cellular carriers will have a qmi folder under the system network location.
example:

$ [[ -d "/sys/class/net/wwan0/qmi" ]] && echo this is a mobile data network

This is not the case on wired or wifi connections.

I should be able to use this in conjunction with the following to narrow down whether or not the only active connection is mobile.
ip --brief addr

better method:
cat /sys/class/net/wwan0/operstate

I'll also need to exclude loopback and anbox connections. All relevant network connections should have a device subfolder.

[[ -d "/sys/class/net/wlan0/device" ]] && echo not a loopback or bridge connection

At some point I want to re-write and re-format the ReadMe to make it easier to understand and have a more logical progression.

This will also hopefully make it easier to learn important related information in quick succession.

Part of replacing the AUR helper apacman with pikaur.

pikaur cannot import and use pgp keys during the install. Workaround: pass --skippgpcheck for packages that need it.

There are 3 potential solutions for this:

1. pass --skippgpcheck when installing all AUR packages (not preferred)
2. detect when a package needs pgp key imported and attempt to handle that (more involved than I'd like, and does not support other flags)
3. provide ability to specify packages and required flags in configuration file (currently preferred)

The 3rd option is the most preferred, as it has less potential to fail, and I can support other flags as well, such as --nocheck for libc++.

The way that timestamps are currently compared is a simple difference calculation between two yyyymmdd integers. Basic math expects all digits to reach 9.

last_keys_update=20200419
last_keys_update=20200501

Per this manjaro gitlab issue, manjaro may be moving away from pacman-mirrors to the brand new shiny-mirrors

I'll probably wait a bit before implementing this depending on how quickly manjaro adopts it, just to let it mature a bit.

Once I do implement support for it, I'll need to make sure config specific to refreshing mirrors is generic enough to apply to both the old and the new mirrors solution.

Support for pacman-mirrors will likely not go away in the foreseeable future. manjaro devs would need to drop it from the repos before I even consider dropping that support.

1. Add option to shutdown instead of reboot Done

2. Add option to always perform the requested action, regardless of updates Done

3. Environment variable to disable fork to background
-This is already available - just call the script with "foreground" as a launch parameter (needs to be added to ReadMe)

4. specify reboot or shutdown as launch parameter to perform that action when script is finished
-Might be part of a larger project at some point, but for now separate configs can be specified with the xs_autoupdate_conf environment variable

Using X11 for dbus-daemon autolaunch was disabled at compile time, set your DBUS_SESSION_BUS_ADDRESS instead

Workaround:
Installing dbus-x11 fixed this issue for me

Further testing:
I have not been able to reproduce this since initially creating this error.

Status:
This is not a priority as the latest script will (by default) automatically reboot if notifications stop working in order to prevent the user from having to guess when it is safe to restart or shutdown.

Info:

• Source 1
• Source 2

AUR python packages often require rebuild after python 3.x minor version updates:
3.7 -> 3.8
3.8 -> 3.9
etc

This is separate from issue #15 in that this will be more automated in nature, and will not require the user to manually specify a bunch of python packages.

This can be manually done with the following (3.8 -> 3.9):

pikaur -S --noconfirm $(pacman -Qqo "/usr/lib/python3.8/site-packages")

• Script will check if folder exists for previous 3.x python versions, and if packages "own" the site-packages subfolder, AUR frequency timestamps will be reset to allow AUR updates/rebuilds
• A bit later, if AUR updates are still configured to proceed, script will check /usr/lib/python3.x paths, and rebuild any packages that own files contained in the respective site-packages folder of each, excluding the current version of python

If main updates are stuck, say so in update notification

New version of script will update flatpak packages by default. This can be toggled with bool_updateFlatpak in settings.

• Fix notifications on phosh
• Use "device" instead of "computer" when applicable
• Detect kernel update for automatic restart

some installs started with short-term support kernels, or kernels that are otherwise EOL. potential issues caused by this:

• unbootable system after updating all other packages, but not the kernels
• possibility of updates conflicting with unsupported kernels (thus preventing successful system updates)
• security concerns, as the EOL kernels are not being patched against newer vulnerabilities

kernel management will have these modes available:

• auto (default) - script detects desired kernel branches (re-detects upon manual kernel changes)
• manual branch selection (via config: kernel_branch_str)
• none: no kernel management - left completely to user to handle

in addition to managing kernels based on branches, script will also support "fallback" kernels - i.e. 4.19 and 5.4 would be fallback to 5.10 (lts+2), or 5.13 being fallback to 5.14 (stable+1)

• edge: latest kernel available (including experimental)
• stable: latest stable
• oldstable: any stable kernel that isn't the latest and is not a fallback to the latest
• lts: latest lts
• oldlts: any LTS kernel older than the latest that is not a fallback to the latest lts

branches will be specified in a separate config setting as follows:
kernel_branch_str=stable,lts+1,oldlts

Considerations:

• the currently running kernel will never be removed
• kernel running on last script run will not be removed (prevent removing known working kernel on first boot after installing new one)
• this will also deal with installed kernel modules and headers (i.e. linux##-headers and linux##-acpi_call)

A while back, I had selected 2 particular icons to display for warnings and critical messages. Since then, those icons have changed. The script needs to use generic icons that will always convey the level of urgency intended despite differing icon themes.

config:

zflag:package1,package2=--skippgpcheck

If either package are not installed, both are skipped.

Script needs to be changed to iterate through the packages instead of testing both at the same time

(Part of Config overhaul)

str_cleanLevel will be replaced with four new settings:

cln_1enable_bool=1
cln_aurpkg_bool=1
cln_aurbuild_bool=1
cln_paccache_num=0

cln_1enable_bool - Enable/Disable all cleanup
cln_aurpkg_bool - Enable/Disable deleting built AUR packages
cln_aurbuild_bool - Enable/Disable deleting AUR package build directories

cln_paccache_num - Number of previous versions of packages from official repos to keep (-1 to disable)

One of the goals of this script is to allow a system that has not been powered on in quite a while to successfully update.

One of the potential issues is package signature expiration. If the local signatures are expired, updated keyrings (which contain updated signatures) cannot be installed.

Solution: temporarily disable signature checks for only keyring packages, and only if they are too old to be updated normally, and only if normal update fails immediately prior.

Signature checks are there to protect against MITM (man in the middle) attacks. Disabling these to update these packages is a security risk. However, in my opinion, running software that has not been updated in over 1.5 years poses an even bigger security risk.

Anyone who updates their system at least once a year will not need to worry about this. It will also have an option to disable.

For example:

:: python-pyqt5 and pyqt5-common are in conflict. Remove pyqt5-common? [y/N] 
error: unresolvable package conflicts detected
error: failed to prepare transaction (conflicting dependencies)
:: python-pyqt5 and pyqt5-common are in conflict

python-pyqt5 was known to have replaced pyqt5-common (5.13.2-1) in an update in December of 2019.

This would probably be implemented with a static list of packages to remove/install.

Notifications do not appear on KDE - look into potential solution

It appears the following syntax works:

DISPLAY=:0 su user -c "notify-send message"

The current command in the script is essentially:

DISPLAY=:0 su user -c "dbus-launch notify-send message"

dbus-launch was specifically added to address #1

Upon further testing, it appears that including dbus-launch now only works in Xfce.
Excluding dbus-launch currently works with Xfce (tested with both dbus and dbus-x11), KDE, and Gnome.

Compiling packages can take quite a while. However, if the system needs a reboot, programs can be finicky at best until the system is rebooted.

Generally the system does not need to reboot for AUR package updates. These should be delayed until next run.

(There will likely be an option to continue with AUR packages anyway - disabled by default)

Recent changes to apacman expose a bug (in apacman) that allows a potentially undefined variable (width) to be used in an expression, causing that script to crash. This affects any situation where stty does not return a numeric value, such as when it is run without <stdin>, or in a non-tty environment.

This can be replicated as shown below

$ yes|sudo apacman -Syu
stty: 'standard input': Inappropriate ioctl for device
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
:: Starting full system upgrade...
 there is nothing to do
:: Synchronizing aur database...
stty: 'standard input': Inappropriate ioctl for device
/usr/sbin/apacman: line 576: *6/10: syntax error: operand expected (error token is "*6/10")

The two instances of stty: 'standard input': Inappropriate ioctl for device are when stty fails to retrieve the width of the current terminal interface. This does not cause the crash by itself (as evident that the script continued to run after the first instance). The script only crashes when it then tries to use an undefined width variable in a mathematical expression: infolen="$(($width*6/10))". Since width is undefined, the expression becomes infolen="$((*6/10))", which is invalid and crashes the script.

A bug was created for this a while ago. There has, unfortunately, been no response from authors of apacman. In the meantime I'm going to have to work around this issue by forcing the script to use a dummy stty script that will always return a numerical value.